crop row line detection with auxiliary segmentation task | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper crop row line detection with auxiliary segmentation task conference paper first online: 12 october 2023 pp 162–175 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) crop row line detection with auxiliary segmentation task download book pdf download book epub igor ferreira da costa  orcid: orcid.org/0009-0004-9921-07259 & wouter caarls  orcid: orcid.org/0000-0001-9069-23789  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 551 accesses 1 citation abstract autonomous robots for agricultural tasks have been researched to great extent in the past years as they could result in a great improvement of field efficiency. navigating an open crop field still is a great challenge; rtk-gnss is a excellent tool to track the robot’s position, but it needs precise mapping and planning while also being expensive and signal dependent. as such, onboard systems that can sense the field directly to guide the robot are a good alternative. those systems detect the rows with adequate image techniques and estimate the position by applying algorithms to the obtained mask, such as the hough transform or linear regression. in this paper, a direct approach is presented by training a neural network model to obtain the position of crop lines directly from an rgb image. while, usually, the camera in such systems are looking down to the field, a camera near the ground is proposed to take advantage of tunnels formed between rows. a simulation environment for evaluating both the model’s performance and camera placement was developed and made available in github, and two datasets to train the models are proposed. the results are shown across different resolutions and stages of plant growth, indicating the system’s capabilities and limitations. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001; the national council for scientific and technological development cnpq under project number 314121/2021-8; and fundação de apoio a pesquisa do rio de janeiro (faperj) apq1 program e-26/010.001551/2019. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others curved and straight crop row detection by accumulation of green pixels from images in maize fields article 03 january 2017 integrating machine vision-based row guidance with gps and compass-based routing to achieve autonomous navigation for a rice field weeding robot article 16 november 2019 enhancing visual autonomous navigation in row-based crops with effective synthetic data generation article open access 11 june 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. agroecology agricultural geography agriculture agronomy computer vision crop waste 1 introduction the use of robotics in agriculture presents the possibility of improving the efficiency of the field, and at the same time increasing the production and quality of the crop. autonomous robots are a good tool to deal with simple but long lasting tasks, such as monitoring the state of plants growth [1, 2], health and presence of plagues or invasive species [3, 4]. real-time kinematics with global navigation satellite system (rtk-gnss) can provide, in open fields, an accurate position of the robot which can be used to navigate [4, 5]. while precise, this system does not describe the crop field itself, thus requiring precise mapping and route planning. rtk-gnss can also be expensive to deploy [5, 6] in scale and be vulnerable to an unreliable or weak signal [1]. crops are not always straight and plain, varying widely in topology due to the place’s geography, which makes navigation even harder [5], see fig. 1. as such, onboard solutions for navigation gained traction and different techniques and sensors have been tested and developed. onboard systems, such as lidar [4, 7], depth cameras and rgb cameras [6, 8], make the robot sense the field directly, allowing it to navigate between plantation rows accordingly. in order to correctly navigate the field, an algorithm to detect the rows and spaces inbetween must be used. convolutional neural networks (cnns) have already been successful at distinguishing crops from lanes [9, 10]. in a similar way, this work employs a cnn, which is trained in images extracted from simulations and previous tests of the robot. given a segmented image, techniques such as linear regression [5, 11], can be applied to extract steering commands to guide the robot. this work aims to extract directly from the image, using a cnn, the steering directions, leveraging the model’s capacity of feature extraction to guide the robot more directly. the previously developed robot is a differential drive design, developed to work across soybean and cotton fields, fig. 2. the original hardware has three logitech c270 cameras for navigation, one up top looking down and one in each front wheel, two intel realsense d435i rgb-d cameras used for plant analysis and two gps modules, all controlled by an intel nuc7i5bnk running ubuntu 18.04 lts and ros melodic. fig. 1. uneven and curved field. full size image fig. 2. the robot roaming in a test field full size image for this work, simulations and robot control are performed by the nvidia jetson agx orin developer kit. this new single board computer (sbc) is replacing the robot’s onboard computer, improving the available computational power, now running ubuntu 20.04 lts with ros noetic. 2 related work deep learning methods for crop field navigation have already been researched and tested. ponnambalam et al. [5] propose a resnet50 to generate a mask of the field using an rgb camera. this work addresses the changing aspects of the field across the season, such as shadows, changes of colours, bare patches or grass across the field by a multi-roi approach with the generated mask. xaud, leite and from [10], applied a flir thermal camera to navigate within sugarcane fields. the thermal camera yielded images with more visibility and better discernible features, especially in poor lighting and weather conditions. the authors tested a simple cnn model with both cameras to compare the performance of the rgb image against the thermal camera with good results in favor of the latter. silva et al. [9] compared the same training dataset with and without the depth information of a rgb-d camera applying the u-net deep learning model to identify the rows of plantation. rgb only images end up being selected due to rgb-d not increasing the results seemingly. the obtained masks were post processed by a triangular scanning algorithm to retrieve the line used for control. similarly to ponnambalam [5], martins et al. [11] applied a multi-roi approach with sliding windows on a mask generated by the excess of green method. this is also the current robot’s method of row navigation and one possible direct comparison which shows good results with smaller plant size scenarios, but limitations are observed in testing. this work attempts to bridge the last gap in all previous examples by obtaining directly the row line or the row path, that way commands are sent straight to the controller and not derived from a mask by a post process algorithm. the mask, in this instance, will be leveraged as an auxiliary task [12] to improve the training model and allow the use of further techniques on the mask itself. one of the biggest challenges faced previously were situations where the soil was occluded by leaves. with a bottom camera close to the ground, a similar navigation system will be tested to face this challenge using the developed tools. 3 approach the controller, which actuates the robot wheels, receives two parameters, an x value that describes the vertical offset of a given line and \(\theta \), the angle between the line and the vertical, see fig. 3. for each frame captured by the robot’s rgb-camera, the aim is to obtain a given line and pass both x and \(\theta \) values to the controller. during the testing phase, the effects of using a segmentation job as an auxiliary task will also be evaluated. a second approach will also test a bottom camera near the ground which creates a “tunnel” and “walls” effect by the camera’s positioning amongst the rows, see fig. 3. fig. 3. top and bottom camera line values given to the controller full size image 3.1 data annotation and preprocessing the image data previously obtained was separated in groups by weather condition and plant growth. these are rgb images with 640\(\,\times \,\)480 resolution and 8bpc of color depth. one first python script was made to annotate line coordinates for each image with opencv and another script built a mask for segmentation using the line and various color filters. after the process, this base mask was loaded into js-segment annotator tool [13] for cleaning and small corrections, see fig. 4. fig. 4. masks are generated by algorithm and then cleaned to improve quality full size image the line was defined with two variable x values and two fixed y values, the line will always be defined by coordinates in the form of \(\left( x_0, 0\right) \) and \(\left( x_1,480\right) \), even if x falls outside of the image size. this special case is caused by lines that start or finish in the sides of the image. the top view dataset was separated in three sections: training, validation and testing, with an split of \(70\%\), \(20\%\) and \(10\%\), respectively. each section contains \(50\%\) of images derived from the simulation and also samples of every type from the original image groups. the botton view dataset was only composed by simulated data due to lack of good images for training among the original data. 3.2 network model a modified network model based on the deeplabv3+ [14] was extracted from keras code examples. the deeplabv3+ employs an encoder-decoder structure and this variation has resnet50 [15] as backbone for early feature extraction by the encoder. the resnet50 was obtained in the keras applications library pretrained with imagenet weights. a second head, known here as line head, was added alongside deeplabv3+ responsible for learning the line directly. its composition is made of a flatten layer, two fully interconnected layers with 25 and 20 neurons each and a output layer. the input is taken from the same tensor that deeplab uses for the atrous convolution section of its encoder, see fig. 5. the interconnected layers apply relu as activation function while the output layer uses a sigmoid. fig. 5. multitask proposed model for mask and line generation full size image a custom loss function (1) was established due the multitask nature of the model, it combine the mask’s sparse categorical cross-entropy and the root mean squared error (rmse) used for the two values, \(x_0\) and \(x_1\), needed to define the line. $$\begin{aligned} loss = ce _\textrm{mask} + rmse _{x_0} + rmse _{x_1} \end{aligned}$$ (1) training is performed in up to 500 epochs with early stopping configured. the adam optimizer was chosen with a learning rate of \(10^{-5}\) and training stopped if bad values (nan) were detected. the best and the final weights are kept to predict the test images and be used in the simulation. two variants of the model were tested, the full proposed model and without the deeplabv3+ decoder alongside the line head, evaluating the impact of the auxiliary task. each variant was tested with 3 image resolutions: \(128\,\times \,128\) for faster inference times, \(512\,\times \,512\) for better image quality and detail, and \(256\,\times \,256\) as a middle ground. all listed configurations were tested for the top and the bottom cameras. 4 simulation and test setup a simulation for testing was elaborated using the gazebo-sim package from ros, and made avaliable at https://github.com/ifcosta/cnn_line. plant models were designed in the blender software based on real image data and a simplified version of the robot was modeled with two cameras, one at a height of 1,8 m looking downward in front of the robot and one just 0,15 m above the ground looking in front slightly upward. images obtained with the simulations can be automatically annotated using simulation tools and filters (fig. 6). fig. 6. simplified robot model built for the simulations full size image the simulated field is built with five stages of plant growth, each stage has six plant rows that are 50 cm apart and 10 m long, with the plants 25 cm apart. there are multiple models of plants for each stage, each plant in a given row is selected randomly in the stage of growth pool. there is also a random component to the plant placement; 15% of given positions have a missing plant and every plant in the row has a small variance in position, see fig. 7. fig. 7. field modelled for testing with five growth stages full size image each model was tested by crossing the field in both directions across all growth stages and the error between the theoretical perfect line and also the mean absolute error of the robot position were both recorded. the robot also starts every run with a small deviation from the center, randomly pointing inward or outward, and has a small error component for the applied velocity as defined by $$\begin{aligned} error _{t+1} = \left( 1 \theta \right) \times error _{t} + normal \left( \mu , \sigma \right) , \end{aligned}$$ (2) with \(\theta =0.25\), \(\mu =0\) and \(\sigma =0.01\). the tests were designed this way so the model has to be able to compensate for actuation errors while correcting the starting deviation correctly during the run. 5 results 5.1 training phase concerning the models trained for the bottom camera, both line-only and the variant with an auxiliary task achieved similar results with regards to the line’s combined loss function, see fig. 8. the larger resolution with auxiliary task is the worst performing model by the final loss value and had its training stopped due to nan values found. both smallest resolution models were the most unstable, while the best model, in general, was the medium resolution one. also, the auxiliary tasks had little to no discernible impact except for the high resolution which had a negative one. fig. 8. bottom camera line loss dotted line mark the minimum value obtained full size image meanwhile, the top camera, in fig. 9, has better training results than the bottom one. this camera position also exhibits a small but positive effect to all resolutions when using an auxiliary task. the lower resolution had the worst performance, like with the bottom camera, but the high and medium resolution had comparable training performance with this camera placement. the top camera was considerably better during the training phase for all resolutions in regards to the model’s loss. fig. 9. top camera line loss dotted line mark the minimum value obtained full size image the bottom camera data set is smaller and composed only by simulation derived images, which could be impacting its results. the way that the line was modeled implied that it had a higher variability at the top of the screen with the bottom camera, i.e. \(x_0\), and that could lead to the higher loss observed. this volatility does not happen with the top camera and to the \(x_1\) value of the bottom one. as it will be shown later, the bottom camera model can still work with this behavior, but a different training approach for the bottom line might bring better results. 5.2 simulation runs alongside the model’s tests, a control run was executed with ros transforms to calculate the theoretical perfect line and evaluate the controller performance. these runs have a perfect line, but still have a processing delay comparable to the model’s inference times. for the top camera, there is another available comparison, which is the multi-roi with excess of green mask [11]. this algorithm is not designed to be used on the bottom camera and its processing time depends of the number of masked pixels inside the sliding window. as such, the larger growth stage, having a greater amount of segmented and processed pixels due to the size of and density of the plants, takes longer to process. it also depends on the cpu’s single thread processing power, being less suitable for the jetson’s hardware, see fig. 10, so this algorithm can be slower depending in plant size and shape while the cnn based algorithms does not get influenced by it. the growth stage 1 and 2, which has the smaller plants, had similar results shown by fig. 11 growth stage 1. also the growth stage 3 and 4 with the medium sized plants, fig. 12, displayed similar traits. evaluating the robot’s position inside the plantation row can assess the entire performance of the system, i.e. model and controller. figures 11 and 12 show that only the low resolution without auxiliary task model got consistently bad results while the high resolution with auxiliary task got them only in the growth stage 1 for the top camera. fig. 10. processing time for all algorithms, from image reception to command sent. the multi-roi is split for each growth stage to show the speed decrease observed with bigger plants full size image fig. 11. robot position deviation from the center of row’s path growing stage 1 full size image fig. 12. robot position deviation from the center of row’s path growing stage 3 full size image figures 12 and 13 also display the weaker result of the multi-roi approach from growth stage 3 onward, possibly due to the high processing time needed and the lack of soil visibility. the fifth growth stage marks failures for all top camera models, while the bottom still has good results, in line with the previous scenarios, see fig. 13. fig. 13. robot position deviation from the center of row’s path growing stage 5 full size image as such we can conclude that the bottom camera is better suited to this dense environment, with the medium and high resolution displaying the best results. 6 discussion the proposed approaches displayed good results in the simulation, in special the medium resolution, which was the most consistent. meanwhile, the auxiliary task showed potential, improving results for the small and medium resolution, but with problems at the highest. overall, with those datasets and this simulated environment, the medium resolution with auxiliary task had the best performance, followed closely by the medium and high resolutions without the auxiliary task. the camera near the ground delivered comparable results to the top camera in most scenarios and excellent ones when the ground was occluded by the plant leaves. the tunnel formed was enough to guide the robot during the simulations, but tests with objects and grass on the robot’s path need to be evaluated. also, tests in a real environment would improve the dataset and test the model’s true capabilities. it is important to note that the controllers used in this work were not modified from those already in place in the robot. however, with the developed simulation environment other techniques could be applied to better actuate the robot with the given line from the network. jesus et al. [16] proposed a method leveraging reinforcement learning to navigate an indoor area, reaching goals without hitting obstacles. the same kind of controller could be used in our work, with the line being used as input to the controller agent. 7 conclusion we presented a neural network model to extract crop lines from rgb images. the network input can be taken from either a top or a bottom camera, and the loss function was augmented with an auxiliary segmentation task. with the simulated environment created, the top and the bottom cameras were evaluated in three image resolutions with and without the auxiliary task proposed. all networks configurations trained successfully and were able to guide the robot in the simulation with varying degrees of success while using the data set created with the built environment. when assessing the overall performance of the entire system, both the top and bottom cameras achieved less than 50 mm of deviation from the center of the path in all but one configuration in the first four growth stages. however, the fifth growth stage resulted in failures for all top configurations, with errors exceeding 90 mm for most configurations, while the bottom camera remained below 50 mm. the processing time of all configurations kept below 70 ms, notably the medium resolution which kept between 30 and 40 ms, comparable to the low resolution model. the auxiliary task had a positive impact on medium and low resolutions but negatively impacted the high resolution configuration. the bottom camera approach is beneficial to scenarios were the ground is mostly occluded by dense foliage, but can have issues in some configurations containing smaller plants. with the gpu computing performance available, a fully integrated model could be feasible. this way, the best camera for each situation would not need to be chosen, as both cameras could feed directly into the model, combining channel wise or just as a larger image. other types of cameras, such as thermal or rgb-d, can be evaluated to take advantage of ground’s temperature as it is occluded by bigger plants or the depth perception possible due to the tunnel seen by the bottom camera. other types of network models could also be evaluated, both for the line head and for the auxiliary task used. references ahmadi, a., nardi, l., chebrolu, n., stachniss, c.: visual servoing-based navigation for monitoring row-crop fields. in: 2020 ieee international conference on robotics and automation (icra) (2020) google scholar  nakarmi, a.d., tang, l.: within-row spacing sensing of maize plants using 3d computer vision. biosys. eng. 125, 54–64 (2014) article  google scholar  mccool, c.s., et al.: efficacy of mechanical weeding tools: a study into alternative weed management strategies enabled by robotics. ieee robot. automat. lett. 1 (2018) google scholar  barbosa, g.b.p.: robust vision-based autonomous crop row navigation for wheeled mobile robots in sloped and rough terrains. dissertação de mestrado em engenharia elétrica, pontifícia universidade católica do rio de janeiro, rio de janeiro (2022) google scholar  ponnambalam, v.r., bakken, m., moore, r.j.d., gjevestad, j.g.o., from, p.j.: autonomous crop row guidance using adaptive multi-roi in strawberry fields. sensors 20(18), 5249 (2020) article  google scholar  ahmadi, a., halstead, m., mccool, c.: towards autonomous visual navigation in arable fields (2021) google scholar  shalal, n., low, t., mccarthy, c., hancock, n.: orchard mapping and mobile robot localisation using on-board camera and laser scanner data fusion part a: tree detection. comput. electron. agric. 119, 254–266 (2015) article  google scholar  english, a., ross, p., ball, d., upcroft, b., corke, p.: learning crop models for vision-based guidance of agricultural robots. in: 2015 international conference on intelligent robots and systems (iros) (2015) google scholar  de silva, r., cielniak, g., wang, g., gao, j.: deep learning-based crop row following for infield navigation of agri-robots (2022) google scholar  xaud, m.f.s., leite, a.c., from, p.j.: thermal image based navigation system for skid-steering mobile robots in sugarcane crops. in: 2019 international conference on robotics and automation (icra). ieee (2019) google scholar  martins, f.f., et al.: sistema de navegação autônoma para o robô agrícola soybot. in: procedings do xv simpósio brasileiro de automação inteligente. sba sociedade brasileira de automática (2021) google scholar  liebel, l., körner, m.: auxiliary tasks in multi-task learning (2018) google scholar  tangseng, p., wu, z., yamaguchi, k.: looking at outfit to parse clothing (2017) google scholar  chen, l.-c., zhu, y., papandreou, g., schroof, f., adam, h.: encoder-decoder with atrous separable convolution for semantic image segmentation (2018) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: ieee conference on computer vision and pattern recognition (cvpr), pp. 382–386 (2016) google scholar  jesus, j.c., bottega, j.a., cuadros, m.a., gamarra, d.f.: deep deterministic policy gradient for navigation of mobile robots in simulated environments. in: 2019 19th international conference on advanced robotics (icar), pp. 362–367 (2019) google scholar  download references author information authors and affiliations pontifícia universidade católica do rio de janeiro, rio de janeiro, rj, brazil igor ferreira da costa & wouter caarls authors igor ferreira da costaview author publications search author on:pubmed google scholar wouter caarlsview author publications search author on:pubmed google scholar corresponding authors correspondence to igor ferreira da costa or wouter caarls . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da costa, i.f., caarls, w. (2023). crop row line detection with auxiliary segmentation task. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_11 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords autonomous robot deep learning agricultural robot publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature binary flying squirrel optimizer for feature selection | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper binary flying squirrel optimizer for feature selection conference paper first online: 12 october 2023 pp 51–64 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) binary flying squirrel optimizer for feature selection download book pdf download book epub luiz fernando merli de oliveira sementille9, douglas rodrigues9, andré nunes de souuza9 & … joão paulo papa9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 532 accesses abstract bio-inspired optimization algorithms aim to address the most diverse problems without the need for derivatives, and they are independent of the shape of the search space. the flying squirrel optimizer belongs to the family of bio-inspired algorithms and simulates the movement of flying squirrels from tree to tree in search of food. this paper proposes a binary version of the flying squirrel optimizer for feature selection problems. to elucidate the performance of the proposed algorithm, we employed six other well-known bio-inspired algorithms for comparison purposes in sixteen benchmark datasets widely known in the literature. furthermore, we employ the binary flying squirrel optimizer in selecting gas concentrations to identify faults in power transformers. the results expressed that binary flying squirrell optimizer can either find compact feature sets or improve classification effectiveness, corroborating its robustness. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a differential squirrel search algorithm chapter © 2021 eight bio-inspired algorithms evaluated for solving optimization problems chapter © 2018 squirrel search optimizer for solving economic load dispatch problem chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms bioinspired robotics continuous optimization discrete optimization learning algorithms optimization 1 introduction finding the most cost-efficient route for transporting goods, the ideal amount of raw material to manufacture a product, or even the best way to drill oil wells are just a few examples of decision-making many companies face. in these cases and several other complex problems, mathematical optimization can be used to find feasible and optimal solutions. through mathematical models, optimization aims for the optimal solution among the feasible candidates in a search space guided by the objective function and the problem’s constraints. given the nature of the objective function, classical mathematical optimization algorithms can be inefficient due to the function’s dependence on being continuous and differentiable in the search space. furthermore, classical algorithms may lose performance in cases where the function is multimodal. bioor nature-inspired algorithms emerge as an elegant alternative for solving complex optimization problems, mitigating the disadvantages of classical algorithms. through the idea that natural processes are essentially optimal, many algorithms were created following metaphors of nature, such as particle swarm optimization [10], bat algorithm [19], flower pollination algorithm  [18], whale optimization algorithm [13], butterfly optimization algorithm [2], and jellyfish search [4], among others. this group of bio-inspired algorithms offers near-optimal solutions in a reasonable time, an alternative to solving np-hard combinatorial problems. artificial intelligence has benefited from using bio-inspired algorithms to optimize neural networks, support vector machines, or optimum-path forest models, among many others. in addition to hyperparameter adjustments to minimize misclassification, the dimensionality of datasets can negatively affect the classifier’s performance during training, increasing the computational burden and even decreasing accuracy. one way to solve this problem is through dimensionality reduction techniques, i.e., selecting the most relevant features for the classification task. in other words, we want to remove irrelevant and redundant features to reduce the computational cost required during training and maximize the classifier’s hit rate. over the last few years, bio-inspired algorithms have successfully addressed feature selection problems, for they can obtain good solutions in a reasonable time, even if the problem is complex. the literature is vast and with promising results. nakamura et al. [14] proposed the binary bat algorithm (bba) for feature selection. although ba is effective for such a purpose, it lacks efficiency compared to other metaheuristics. rodrigues et al. [16] introduced the binary cuckoo search (bcs) to the same context, and rodrigues et al. [16] designed a binary-valued flower pollination algorithm (bfpa) to select the most critical sensors to person identification using electroencephalogram signals. this paper proposes a binary version of the flying squirrel optimizer (fso) for feature selection, called bfso. the fso was proposed by azizyan et al. [3] inspired by flying squirrels moving from one tree to another in search of food. this work’s main contributions lie in using a wrapper-based approach employing bfso and the naive bayes classifier for the feature selection task. however, any other supervised classifier can be used. we considered 16 benchmark datasets to evaluate the proposed approach’s performance. the performance of the bfso was also validated to select relevant gas concentrations to identify faults in power transformers. in short, this paper figures the following contributions: to propose the binary flying squirrel optimizer; to evaluate the proposed approach for feature selection; and to employ bfso for fault diagnosis in power transformers using gas concentration. the remainder of this paper is organized as follows. section 2 presents the theoretical background concerning the flying squirrel optimizer, while sects. 3 and 4 discuss the methodology and experimental results, respectively. section 5 states conclusions and future works. 2 flying squirrel optimizer flying squirrel optimizer [3] is a flying squirrel swarm-based algorithm whose population is given by \(\mathcal{x} = \{\textbf{x}_1,\textbf{x}_2 \ldots , \textbf{x}_m \}\), where m is the number of squirrels and \(\textbf{x}_i\in \re ^n\) denotes a single possible solution. the proposed model follows two movements squirrels perform in their pursuit of food. the first simulates the awkward gait of squirrels when they are on the ground. this motion is modeled as random walks following a normal distribution \(\textbf{r} \sim \mathcal {n}(\mu ,\,\sigma ^{2})\) in which \(\mu \) is the mean position of all flying squirrels and \(\sigma \) is given as follows: $$\begin{aligned} \sigma = -\ln \left( 1 \frac{1}{\sqrt{t + 2}} \right) ^2, \end{aligned}$$ (1) where t corresponds to the current iteration. in the second movement, the squirrels fly from one tree to another through a lévy flight distribution, as follows: $$\begin{aligned} l(\lambda , s, \alpha ) = \frac{\lambda \cdot \vargamma (\lambda )\cdot \text{ sin }(\lambda )}{\pi }\cdot \frac{\alpha }{s ^{1+\lambda }},\quad |s|\rightarrow \infty , \end{aligned}$$ (2) where \(\vargamma (\lambda )\) stands for the gamma function with index \(\lambda \), \(\alpha \) is a control parameter for the tail distribution (\(\alpha = 1\)), and s is the step size. according to mantegna [12], for large steps \(s \gg s_0 > 0\), s can be computed through a linear transformation as follows: $$\begin{aligned} s = \frac{\psi }{\eta ^{\lambda -1}}, \end{aligned}$$ (3) where \(\psi \) and \(\eta \) are drawn from a gaussian distribution with zero mean and standard deviations \(\sigma _\psi \) and \(\sigma _\eta \) computed as follows $$\begin{aligned} \sigma _\psi = \left[ \frac{\vargamma (1+\lambda )}{\lambda \vargamma ((1+\lambda )/2)}\frac{\sin (\frac{\pi \lambda }{2})}{2^{(\lambda -1)/2}}\right] , \sigma _\eta = 1. \end{aligned}$$ (4) in this algorithm, \(\lambda \) grows linearly at each iteration, increasing the length of steps of the lévy flight. moreover, \(\lambda \) is computed as follows: $$\begin{aligned} \lambda = \beta + (2 \beta ) * ((t + 1) / t), \end{aligned}$$ (5) in which \(\beta \) is a user-configured parameter, and t corresponds to the maximum number of iterations. the position of each squirrel is updated as follows: $$\begin{aligned} \textbf{x}_i(t+1) = \textbf{x}_i(t) + l(\lambda ,s,\alpha ) \cdot \textbf{r} \otimes (\textbf{x}_i(t) \textbf{x}_{best}), \end{aligned}$$ (6) where \(\textbf{x}_{best}\) is the best solution found so far, and \(\otimes \) denotes the pointwise multiplication. 2.1 algorithmic analysis algorithm 1 describes how fso works in more detail. lines 1–4 initialize each possible solution (squirrel) with random values uniformly distributed within [0, 1]. the fitness value of each solution is set to a great value in line 4. this loop takes \(\theta (mn)\) operations. the main loop in lines 6–24 controls the fso mechanism, taking \(\theta (t)\) steps. as mentioned earlier, squirrels follow two main movements. the parameter \(\sigma \) of the first (eq. 1) is computed in line 7, and variable \(\lambda \), which concerns the second movement, is obtained in line 8. the inner loop in lines 10–19 computes a new position for each squirrel (lines 13–14), evaluates its fitness value (line 15), and updates its position if it has a better solution (lines 16–19). these steps take \(\theta (mn)\) calculations. last but not least, lines 20–24 update the best solution whithin the swarm. this loop takes o(mn) operations. the overall complexity is given by \(\theta (tmno)\) operations in which \(\theta (o)\) represents the complexity to compute the fitness function. algorithm 1 flying squirrel optimizer full size image 3 methodology table 1 presents the datasets used to evaluate the robustness of bfso in the context of feature selection. note that for this work, we used publicly available datasets in the uci repositoryfootnote 1 and six datasets, i.e., the last six items presented in table 1, which will be used to validate the bfso in the context of identifying faults in power transformers. the data used to form these six data sets were obtained from iec tc10 [1] and scientific papers [5]. one can observe the different scenarios, i.e., datasets with varying size, from different domains, and with a wide range concerning the number of features. each dataset was split into training, validation, and testing sets using a 2 : 1 : 1 ratio. the pipeline of the proposed approach is depicted in fig. 1, in which fig. 1a ilustrates the optimization process to find the best subswt of features and fig. 1b ilustrates the final evaluation step, where the selected features are employed to classify the test subset. table 1. datasets used in the comparative study.full size table fig. 1. proposed approach pipeline: (a) feature optimization and (b) final evaluation. full size image at each iteration, the agent’s position in the search space is updated, and the fitness value is computed by evaluating the objective function. given that agents move through the search space with real values, we propose to use a transfer function to convert real to binary values before evaluating the objective function. the transfer function is given as follows: $$\begin{aligned} \textbf{x}_i & = & \left\{ \begin{array}{ll} 1\quad &{} \text{ if } t(\textbf{x}_i)>\phi \text{, } \\ 0&{} \text{ otherwise } \end{array}\right. \end{aligned}$$ (7) in which \(\textbf{x}_{i}\) represents the i-th agent, \(\phi \sim \mathcal {u}(0, 1)\), and \(t(\cdot )\) stands for a transfer function described as follows: $$\begin{aligned} t(\textbf{z}) = \frac{1}{1 + e^{-\textbf{z}}}. \end{aligned}$$ (8) after applying the transfer function, a binary vector will originate a new subset, selecting only the representative characteristics of the original set. this process is accomplished by multiplying the binary vector that represents the agent’s position by the feature vector of the data set, i.e., the position of the vector that has the value 1 indicates that the characteristic corresponding to that position in the feature vector will be selected to compose the new subset. on the other hand, a 0 value indicates the feature will not be selected. therefore, with a new training and validation subsets in hand, the classifier is trained on the new training set and classifies the samples of the new validation set. the classifier’s hit rate is a fitness value to guide the optimization process. the feature subset that achieves the highest accuracy in this process will be stored and used later to train the classifier once again and classify the test subset to determine the real accuracy of the model. we used naïve bayes classifier in this paper, for it is parameterless and fast for trainingfootnote 2. regarding fso running parameters, we used the same recommended by azizyan et al. [3], i.e., \(\beta =0.5\), and the number of agents and iterations were set to 30 and 60, respectively. once metaheuristics are stochastic processes, we performed 20 independent runs to allow the calculation of the mean and standard deviation. besides, different seeds were generated for each run so that distinct samples constituted the division of training, validation, and test subsets. to provide robust analysis, we chose five widely known bio-inspired algorithmsfootnote 3 for comparison purposes with the bfso: binary aquila optimizer (bao) [11]; binary bat algorithm (bba) [14]; binary firefly algorithm (bfa) [6]; binary flower pollination algorithm (bfpa) [16]; and binary particle swarm optimization (bpso) [9]. it is worth noting that the values of the hyperparameters of each algorithm are the same proposed by their respective authors. 4 experimental results table 2 presents the f1-score obtained using the methodology described in sect. 3 with regard to the final evaluation, as illustrated in fig. 1b. the values are the mean and standard deviation for each of the seventeen benchmark datasets. note that the values highlighted in bold represent the best results for each dataset. bfso achieved the highest mean values in twelve datasets, followed by bfa, which obtained the highest mean values in arcene and basehock datasets. bao obtained the highest average value only in the wine set. the highest mean value in the pcmac set was obtained in the original set, i.e., without feature selection, indicated in the table using the naïve bayes (nb) classifier. table 2. results concerning the test set over all datasets.full size table figure 2 illustrates the average number of features selected for the arcene, basehock, coil20, orl, segment, and spambase datasets. bfso selected the lowest number of features in the spambase set compared to the other bio-inspired algorithms and was the second-best technique in the orl and coil20 datasets, second only to bao. furthermore, it is worth the caveat of the size of the standard deviation of the bao in these datasets. in the arcene and basehock datasets, bfso was not among the best feature selection algorithms. fig. 2. average selected features considering each metaheuristic algorithm on: (a) arcene (b) basehock (c) coil20 (d) orl (e) segment datasets. (f) spambase datasets. full size image moreover, we performed the friedman test [7, 8] with a significance of 0.05 (\(5\%\)) to assess whether the average f1-score obtained using each bio-inspired algorithm is similar, i.e., null hypothesis \(h_0\) for this test. then we employed the nemenyi post-hoc test [15]. figure 3c illustrates the test performed for the coil20 dataset, where it can be seen that bfso performed similarly to bfpa. in the segment set, illustrated in fig. 3e, we notice the similarity in the performance of bfso with bfpa and bao. finally, in the spambase set, illustrated in fig. 3f, note the similarity between bfso, bfpa, bao, and bpso. fig. 3. nemenyi test on: (a) arcene (b) basehock (c) coil20 (d) orl (e) segment (f) spambase datasets. full size image additionally, the wilcoxon signed-rank test [17] with a significance of 0.05 (\(5\%\)) was performed to validate the data obtained more robustly. table 3 presents the test result considering f1-score metric obtained during the 20 independent runs. the symbol \(=\) indicates that the result obtained after feature selection using the bio-inspired algorithms performed statistically similar to that obtained using the original dataset, i.e., we accept the null hypothesis \(h_0\). in contrast, the symbol \(\ne \) indicates that the results were statistically different, i.e., we rejected the null hypothesis. one may observe that bfso performed statistically similar to the nb only in the caltech101 set. based on the bfso results shown in table 2 together with the wilcoxon test obtained, bfso obtained the best result by significantly increasing the classifier’s hit rate when removing degrading features. table 3. results concerning the wilcoxon signed-rank test set over all datasets.full size table following, we tested the performance of bio-inspired algorithms in dissolved gas analysis (dga) datasets for the task of gas concentration selection for fault identification in power transformers. table 4 presents the mean values and the standard deviation for each of the six datasets. also, note that the top performers are highlighted in bold. nb obtained the highest average values in the datasets 1069_5gt, 1069_7gt, 1143_5gte, and 1143_7gte. bba and bfa obtained the highest mean values in the 1086_5ge and 1086_7ge datasets, respectively. figure 4 illustrates the mean of the selected features for the datasets 1069_5gt, 1069_7gt, 1086_5ge, 1086_7g, and 1143_5gte, and 1143_7gte. it is important to note that the bio-inspired algorithms selected, on average, half of the features of the original set, i.e., even if there is a little loss in the hit rate, the computational cost for training can compensate in these cases, as well as the cost for feature extraction. fig. 4. average selected features considering each metaheuristic algorithm on: (a) 1069_5gt (b) 1069_7gt (c) 1086_5ge (d) 1086_7ge (e) 1143_5gte (f) 1143_7gte datasets. full size image table 5 present the wilcoxon signed-rank test on dga datasets. all bio-inspired algorithms performed similarly to nb statistically except in the 1143_5gte set and the bpso in the 1143_7gte set. the difference in the hit rate obtained by the nb to the bio-inspired algorithms was insignificant, which further emphasizes the advantage of training in reduced datasets. table 4. average f1-score values considering the comparison among the bio-inspired algorithms on dga datasets.full size table table 5. wilcoxon signed-rank test for the bio-inspired algorithms in comparison to naíve bayes classifier on dga datasets.full size table 5 conclusions in this work, we propose the binary version of the flying squirrel optimizer. considering the feature selection task, we validated its robustness and performance on sixteen benchmark datasets. next, we employ the bfso to select gaseous concentrations for fault identification in power transformers. for bfso performance comparison purposes, we used the bioinspired algorithms bao, bba, bcs, bfa, bfpa, and bpso. the results showed that the bfso can greatly reduce the set of features in all used datasets. furthermore, in some cases, it achieved better predictive performance than the other bioinspired algorithms used for comparison. thus, the performance of the bfso demonstrated in the feature selection task makes it a viable tool for its performance and low complexity. for future work, one idea would be to use chaotic maps for initializing the flying squirrels and employ opposition-based learning further to improve the exploration and exploitation of the algorithm. notes 1.https://archive.ics.uci.edu/. 2.it is worthy to say that any other supervised classifier can be used. we recommend models that figure a reasonably efficient training step, for the fitness function might be evaluated several times during the optimization process. 3.the algorithms used for comparison purposes and fso are part of opytimizer library, which contains several implementations of metaheuristics in python. the opytimizer library is available in: https://github.com/gugarosa/opytimizer. references iec 60599:2022 mineral oil-filled electrical equipment in service guidance on the interpretation of dissolved and free gases analysis. iec, geneva, switzerland, 4 edn. (2022) google scholar  arora, s., singh, s.: butterfly optimization algorithm: a novel approach for global optimization. soft. comput. 23(3), 715–734 (2019). https://doi.org/10.1007/s00500-018-3102-4 article  google scholar  azizyan, g., miarnaeimi, f., rashki, m., shabakhty, n.: flying squirrel optimizer (fso): a novel si-based optimization algorithm for engineering problems. iranian j. optimiz. 11(2), 177–205 (2019) google scholar  chou, j.s., truong, d.n.: a novel metaheuristic optimizer inspired by behavior of jellyfish in ocean. appl. math. comput. 389, 125535 (2021) mathscinet  math  google scholar  equbal, m.d., khan, s.a., islam, t.: transformer incipient fault diagnosis on the basis of energy-weighted dga using an artificial neural network. turk. j. electr. eng. comput. sci. 26(1), 77–88 (2018) article  google scholar  falcón, r., almeida, m., nayak, a.: fault identification with binary adaptive fireflies in parallel and distributed systems. in: proceedings of the ieee congress on evolutionary computation, pp. 1359–1366. ieee (2011) google scholar  friedman, m.: the use of ranks to avoid the assumption of normality implicit in the analysis of variance. j. am. stat. assoc. 32(200), 675–701 (1937) article  math  google scholar  friedman, m.: a comparison of alternative tests of significance for the problem of m rankings. ann. math. stat. 11(1), 86–92 (1940) article  mathscinet  math  google scholar  kennedy, j., eberhart, r.c.: a discrete binary version of the particle swarm algorithm. in: ieee international conference on systems, man, and cybernetics, vol. 5, pp. 4104–4108 (1997) google scholar  kennedy, j., eberhart, r.: particle swarm optimization. in: proceedings of icnn 1995-international conference on neural networks, vol. 4, pp. 1942–1948. ieee (1995) google scholar  li, l., pan, j.s., zhuang, z., chu, s.c.: a novel feature selection algorithm based on aquila optimizer for covid-19 classification. in: shi, z., zucker, j.d., an, b. (eds.) intelligent information processing xi, pp. 30–41. springer international publishing, cham (2022). https://doi.org/10.1007/978-3-031-03948-5_3 chapter  google scholar  mantegna, r.n.: fast, accurate algorithm for numerical simulation of lévy stable stochastic processes. phys. rev. e 49, 4677–4683 (1994) article  google scholar  mirjalili, s., lewis, a.: the whale optimization algorithm. adv. eng. softw. 95, 51–67 (2016). https://doi.org/10.1016/j.advengsoft.2016.01.008 article  google scholar  nakamura, r.y.m., pereira, l.a.m., costa, k.a., rodrigues, d., papa, j.p., yang, x.s.: bba: a binary bat algorithm for feature selection. in: 2012 25th sibgrapi conference on graphics, patterns and images, pp. 291–297 (2012) google scholar  nemenyi, p.: distribution-free multiple comparisons. princeton university (1963) google scholar  rodrigues, d., et al.: bcs: a binary cuckoo search algorithm for feature selection. in: ieee international symposium on circuits and systems, pp. 465–468 (2013) google scholar  wilcoxon, f.: individual comparisons by ranking methods. biometrics bull. 1(6), 80–83 (1945) article  google scholar  yang, x.s.: flower pollination algorithm for global optimization. in: international conference on unconventional computing and natural computation, pp. 240–249. springer (2012). https://doi.org/10.1007/978-3-031-03948-5_3 yang, x.s., gandomi, a.h.: bat algorithm: a novel approach for global engineering optimization. eng. comput. (2012) google scholar  download references author information authors and affiliations são paulo state university, bauru, sp, brazil luiz fernando merli de oliveira sementille, douglas rodrigues, andré nunes de souuza & joão paulo papa authors luiz fernando merli de oliveira sementilleview author publications search author on:pubmed google scholar douglas rodriguesview author publications search author on:pubmed google scholar andré nunes de souuzaview author publications search author on:pubmed google scholar joão paulo papaview author publications search author on:pubmed google scholar corresponding author correspondence to douglas rodrigues . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de oliveira sementille, l.f.m., rodrigues, d., de souuza, a.n., papa, j.p. (2023). binary flying squirrel optimizer for feature selection. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_4 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords flying squirrel optimizer metaheuristic feature selection publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature investigating methods to detect off-topic essays | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper investigating methods to detect off-topic essays conference paper first online: 30 january 2025 pp 346–357 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) investigating methods to detect off-topic essays download book pdf download book epub joyce m. silva  orcid: orcid.org/0009-0002-9696-62319, rafael t. anchiêta  orcid: orcid.org/0000-0003-4209-90139, rogério f. de sousa  orcid: orcid.org/0000-0003-4589-61579 & … raimundo s. moura  orcid: orcid.org/0000-0002-1558-383010  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 384 accesses 1 citation abstract automated essay scoring is one of the most important educational applications of natural language processing. it helps teachers with automatic assessments, providing a cheaper, faster, and more deterministic approach than humans when scoring essays. nevertheless, off-topic essays pose challenges in this area, causing an automated grader to overestimate the score of an essay that does not adhere to a proposed topic. thus, detecting off-topic essays is important for dealing with unrelated text responses to a given topic. this paper explored approaches based on handcrafted features to feed supervised machine-learning algorithms, tuning a bert model, and prompt engineering with a large language model. we assessed these strategies in a public corpus of portuguese essays, achieving the best result using a fine-tuned bert model with a 75% balanced accuracy. furthermore, this strategy was able to identify low-quality essays. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others autograder: a feature-based quantitative essay grading system using bert chapter © 2023 label embedding based scoring method for secondary school essays chapter © 2023 designing essay questions for effective automatic scoring chapter © 2025 1 introduction automated essay scoring (aes) is the computer technology that evaluates and scores written prose [27]. it aims to provide computational models for automatically grading essays or with minimal involvement of humans [21]. this research area began with page [21] in 1966 with the project essay grader system, which, according to ke and ng [15], remains since then. in brazil, most of the work for the automatic essay grading task focuses on the high school national exam (enem exame nacional do ensino médio) [5]. enem is used to assess the quality of high school education and as an admission test for most public and private universities. it provides an essay test and requires the production of a text in the dissertation-argumentative genre on a specific prompt (topic). each enem essay is graded according to five traits (competencies) detailed below. 1. adherence to the formal written norm of portuguese. 2. conform to the argumentative text genre and the proposed topic (prompt) to develop a text using knowledge from different areas. 3. select, relate, organize, and interpret data and arguments to defend a point of view. 4. usage of argumentative linguistic structures. 5. develop an intervention proposal to solve the problem in question. although existing works attempt to grade an essay considering the five traits, they do not approach off-topic essays, i.e., essays that do not adhere to the expected prompt [1, 9, 20]. off-topic essays are related to the second trait of the enem exam, and according to higgins et al. [12], they may be classified into two types: 1. unexpected topic. possibly well-written essays that do not address the expected topic. 2. bad-faith. essays that mainly consist of text copied from the prompt or with irrelevant musings, such as purposely inserted chunks of text unrelated to the topic and the essay itself. these essay types are challenging and considered a problem for automated essay scoring systems, as a well-written essay that does not address the proposed topic may receive an overestimated score from an automated grader because of linguistic features, such as text structure and surface [23]. moreover, kabra et al. [14] has shown that different state-of-the-art automated essay scoring methods fail to provide a low score for poo-quality essays. more than that, off-topic essays occur infrequently, making them difficult to detect automatically. for example, in the 2022 enem exam, of the 2,355,395 produced essays, only 31,734 were off-topic, that is, 1.34%footnote 1. we investigated three strategies for detecting off-topic essays to tackle the above-mentioned challenges. the first is based on handcrafted features for feeding supervised machine learning algorithms to detect off-topic essays, while the second adopts a fine-tuned bert model. the last approach explores prompt engineering, examining whether the language proficiency retained by a large language model is useful in detecting off-topic essays. these methods were evaluated using the essay-br corpus [19]. the fine-tuned bert model achieved the best result with 75% balanced accuracy. the rest of the paper is organized as follows: in sect. 2, we briefly present related work. section 3 details the corpus used to evaluate and compare the approaches. in sect. 4, we detailed the developed strategies. in sect. 5, we reported and analyzed the achieved results. finally, sect. 6 concludes the paper by indicating future directions. 2 related work due to the difficulty of dealing with off-topic essays and the lack of a large corpus of off-topic essays, most works for the portuguese language focused only on on-topic essays [9, 20]. so, there are a few works on this theme; we briefly discuss them here. passero et al. [23] presented a systematic literature review on automatically detecting off-topic essays. they identified five papers, all of them for the english language. the studies dealt with off-topic essays, mainly using textual features, such as latent semantic analysis [17], n-grams, and latent dirichlet allocation [4], among others, to classify an essay as onor off-topic. from this study, the authors found that the approaches had high error rates, and the studies mostly used artificial essay sets for validation. passero et al. [22] adapted the five studies identified by passero et al. [23] to portuguese. the authors evaluated the adaptations in a corpus of 2,164 essaysfootnote 2. as this corpus has only 12 off-topic essays, the authors used the following strategy to use random on-topic essays as off-topic ones. for each set of n essays of a prompt (negative or on-topic examples), n essays are randomly selected from other prompts (positive or off-topic examples). the adapted work of beigman klebanov et al. [2] achieved the best result. it detects off-topic essays by estimating the topicality of each word in the essay by comparing its occurrence in essays from the same prompt to essays from other prompts. pinho et al. [25] compared several supervised machine learning methods to classify an essay as onor off-topic. to compare the classifiers, they used a private corpus of 1,320 essays, 230 of which were off-topic. a convolutional neural network achieved the best result with 89.4% accuracy with three-fold cross-validation. in the following section, we present the corpus used in this study. 3 corpus we used the essay-br corpus [19] to evaluate and compare the developed strategies. that corpus contains 4,570 argumentative essays graded according to the five traits of the enem exam. out of the total number of essays, 82 are graded with a score of zero, i.e., 1.79%, a similar value to the number of essays scored with zero in the 2022 enem exam. the corpus is organized according to table 1. table 1. the number of on-topic and off-topic essays in the corpus.full size table to understand why these 82 essays received a zero score, we asked two linguists with experience in grading enem essays to evaluate them. for that, we provided the essays and prompts, where each reviewer evaluated all essays independently, following the enem criteriafootnote 3. the reviewers agreed that all 82 essays did not follow the expected prompt and received a zero score for not adhering to it. table 1 presents the number of on-topic and off-topic essays in the corpus for each set. as we can see, the corpus is very unbalanced, indicating the need to apply sampling strategies to adjust the class distribution of the corpus. in what follows, we detail our approaches to handling off-topic essay detection. 4 off-topic detection strategies due to the unbalancing of the corpus, we first evaluated overand under-sampling methods. for oversampling strategies, we analyzed approaches for generating synthetic data, such as smote (synthetic minority over-sampling technique) [6] and adasyn (adaptive synthetic sampling) [11]. also, our second oversampling strategy uses random on-topic essays as off-topic ones. we selected 1.543 on-topic essays from the training set and switched their prompts to transform them into off-topic ones. this way, we balanced the training set, resulting in 1.599 on- and off-topic essays. for undersampling, we used a random undersampling strategy from the imbalanced-learn library [18]. it is important to note that these sampling methods were only applied to the training set. after balancing the corpus, we developed two methods to detect off-topic essays, as depicted in fig. 1. in addition to these methods, we checked whether a large language model may detect off-topic essays. in the following subsections, we detail these approaches. fig. 1. approaches to detect off-topic essays. full size image 4.1 features we manually extracted six features to detect off-topic essays. we extracted these features by comparing each paragraph of an essay to each paragraph of its corresponding prompt. we then compute the cosine similarity for each compared paragraph pair using the extracted feature values, applying eq. 1, where \(\overrightarrow{u} \cdot \overrightarrow{v}\) is the dot product of the two vectors. $$\begin{aligned} cos(\theta ) = \frac{\overrightarrow{u} \cdot \overrightarrow{v}}{\vert \overrightarrow{u}\vert \vert \overrightarrow{v}\vert } \end{aligned}$$ (1) finally, we calculate the arithmetic mean among all cosine similarity values to obtain a similarity between an essay and its prompt. we describe the features in what follows. boolean frequency (bf). this feature assigns one value if a word in an essay occurs in its corresponding prompt. otherwise, a zero value will be assigned, according to eq. 2. $$\begin{aligned} bf_{i, j} = 1 \text { if } i \text { occurs in } j \text { and } 0 \text { otherwise} \end{aligned}$$ (2) term frequency (tf). this feature computes how often a word in an essay occurs in its corresponding prompt. it is the relative frequency of a word in an essay within a prompt, as presented in eq. 3. $$\begin{aligned} tf_{i, j} = \frac{f_{i, j}}{\displaystyle \sum _{i^{'} \in j} f_{i^{'}, j}} \end{aligned}$$ (3) term frequency-inverse document frequency (tf-idf). according to eq. 4, this feature assigns the tf-idf value to each word in an essay for the analyzed prompt, where \(tf_{i, j}\) is the term frequency of i in j, \(df_i\) is the number of prompts containing i, and n is the total number of prompts. $$\begin{aligned} w_{i, j} = tf_{i, j} \times \log \left( \frac{n}{df_{i}} \right) \end{aligned}$$ (4) cosine of word embeddings (cos). to calculate the cosine similarity between an essay paragraph and a prompt paragraph, we got the embeddings for the words, computed the average of the word embeddings for each paragraph, and calculated the cosine similarity between these vectors. we used a 300-dimensional glove model pre-trained for portuguese to get embedding values [10]. word mover’s distance (wmd). this feature assesses the distance between two documents even when they have no words in common [16]. it measures the dissimilarity between two text documents as the minimum amount of distance that embedded words of one document need to “travel” to reach the embedded words of another document. it is important to notice that wmd is a distance function, i.e., the lower the distance value is, the more similar the documents are. to get the wmd distance, we first tokenized and removed stopwords of the paragraphs using the natural language toolkit (nltk) [3]; next, we got the embeddings for the words of the paragraphs; finally, to get the wmd distance, we used the method from the gensim library [30] that receives a paragraph pair encoded as word embeddings as input and returns the wmd value. sentence transformers (st). it is a modification of the pre-trained bert network [8] that uses siamese and triplet network structures to derive semantically meaningful sentence embeddings that can be compared using cosine-similarity [26]. for this feature, we used a multilingual pre-trained modelfootnote 4 to get the embedding values for each paragraph pair. then, we computed the cosine similarity between these vectors. we evaluated some classifiers to assess our approach after the feature extraction step. we used support vector machines (svm), naïve bayes (nb), decision tree (dt), multilayer perceptron (mlp), random forest (rf) and xgboost (xgb) from the scikit-learn library [24]. 4.2 bert we fine-tuned the bertimbau model [29] to detect off-topic essays. for that, we adopt a method developed by de sousa et al. [28] in which the essays and prompts are used to tune the model. in this way, the model may identify whether an essay adheres to a prompt. we used the large version of the bertimbau model with the parameters presented in table 2 to tune the model. table 2. training parameters for the bertimbau large model.full size table one can see that we employed a linear layer to make predictions. this layer takes the bert’s hidden size (768 inputs) as input and output of size 1 to predict whether the essay is onor off-topic. besides, we used six epochs to train the model and chose the best model through the lowest error in the validation set. we computed the loss using the binary cross-entropy with logits loss. this loss function combines a sigmoid layer and the binary cross-entropy loss in one class, which is proper for binary classification problems. 4.3 prompt engineering we investigated whether the language proficiency retained by a large language model may identify an off-topic essay. for that, we used gemini 1.5 profootnote 5, a multimodal large language model developed by google. this model has demonstrated remarkable capabilities on various tasks and has gained significant attention in diverse domainsfootnote 6. furthermore, this model has a free-of-charge option with limits of 15 requests per minute, 1 million tokens per minute, and 1,500 requests per day, which is sufficient for our experiments. our prompt design is straightforward. we instructed gemini to inform whether an essay is onor off-topic based on a prompt. in this way, we provided delimiters, such as <prompt> and “‘essay”’ for our inputs to distinguish them, as depicted in fig. 2. fig. 2. a simple prompt asking to classify the input essay based on a prompt. full size image as shown in the above figure, we instructed the model to classify an essay as onor off-topic given an essay and prompt as input. in what follows, we detail the obtained results. 5 results and analysis we evaluated our approaches using the test set of the essay-br corpus [19]. we obtained the best results with the undersampling strategy, as shown in table 3. table 3. results with undersampling strategy.full size table our feature-based approach achieved the best result with the random forest classifier, and the fine-tuned bertimbau model [29] achieved the best overall result. to our surprise, the gemini model did not identify off-topic essays. as this model reports that it can detect essays that do not adhere to a topic, we expected that it achieved good results for this task. after assessing the approaches, we performed a feature selection to identify the most important features in our feature-based strategy. to calculate the importance of each feature, we computed the gini importance (fig. 3), which measures the impurity of a node in a decision tree with a more substantial weight given to the most important features. fig. 3. importance of each feature. full size image from this figure, we can see that the tf-idf contributes the least to the accuracy of the model. at the same time, the features that contribute most to accuracy are the sentence transformer, the cosine of word embeddings, and the word mover’s distance. this result is in line with the study of huang et al. [13], in which the approach developed by the authors achieved good results in detecting off-topic essays. based on the feature selection, we removed the tf-idf feature and re-evaluated our feature-based approach. with this, our results in recall (off-topic) and balanced accuracy metrics improved a little. on the other hand, the results in the recall (on-topic) and f-score metrics were slightly worse, as shown in table 4. table 4. results after feature selection.full size table we also investigated the results obtained by the bertimbau model through a confusion matrix, presented in table 5. from this table, we can see a high number of false negatives (27), justifying the low precision value in the off-topic class. table 5. confusion matrix of the bertimbau approach.full size table we further analyzed the false negative predictions, performing an error analysis to understand the misclassifications. with this, we realized that out of twenty-seven false negatives, only four were good essays with scores greater than 800. the remaining essays had grades below 300, with issues such as textual genre (six essays), poor and unstructured text (fifteen essays), and tangent to the topic (two essays). our analysis revealed that the strategy of tunning bertimbau with essays and prompts makes the automatic classification more rigorous than human review. despite that, this result may be a contribution, as different state-of-the-art automated essay scoring methods fail to provide a low score for low-quality essays [14]. however, it is important to analyze the extent to which bertimbau is strict with poor-quality essays. a deeper inspection of such essays remains necessary for future work. the developed methods are publicly available at https://github.com/liara-ifpi/off-topic-essay. 6 conclusion and future work this paper investigated methods for detecting off-topic essays, such as handcrafted features to feed supervised machine-learning algorithms, the fine-tuned bertimbau model, and the gemini language model. these strategies were evaluated in a brazilian portuguese corpus, showing some learned lessons, challenges, and contributions. as a learned lesson, we highlight the potential of the sentence-transformer as a feature to identify off-topic essays. as a challenge, we may mention the need to create a corpus with more off-topic essays to improve the precision of classifiers. finally, one of the contributions of our experiments is the potential of the fine-tuned bertimbaul model to score or identify low-quality essays. future work includes joining the sentence-transformer feature into the bertimbau model, investigating unsupervised learning strategies, as very recent work has achieved good results for the english language [7], and building a more balanced corpus. notes 1.https://www.gov.br/inep/pt-br/acesso-a-informacao/dados-abertos/microdados/enem. 2.https://github.com/gpassero/uol-redacoes-xml. 3.https://download.inep.gov.br/publicacoes/institucionais/avaliacoes_e_exames_da_educacao_basica/a_redacao_no_enem_2023_cartilha_do_participante.pdf. 4.https://huggingface.co/sentence-transformers/distiluse-base-multilingual-cased-v1. 5.https://gemini.google.com/. 6.https://storage.googleapis.com/deepmind-media/gemini/gemini_v1_5_report.pdf. references amorim, e., veloso, a.: a multi-aspect analysis of automatic essay scoring for brazilian portuguese. in: proceedings of the student research workshop at the 15th conference of the european chapter of the association for computational linguistics, pp. 94–102. association for computational linguistics, valencia, spain (2017) google scholar  beigman klebanov, b., flor, m., gyawali, b.: topicality-based indices for essay scoring. in: proceedings of the 11th workshop on innovative use of nlp for building educational applications, pp. 63–72. association for computational linguistics, san diego, ca (2016) google scholar  bird, s., klein, e., loper, e.: natural language processing with python: analyzing text with the natural language toolkit. o’reilly media, inc. (2009) google scholar  blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3(jan), 993–1022 (2003) google scholar  caseli, h.m., nunes, m.g.v. (eds.): processamento de linguagem natural: conceitos, técnicas e aplicações em português. bpln, 2 edn. (2024). isbn 978-65-00-95750-1 google scholar  chawla, n.v., bowyer, k.w., hall, l.o., kegelmeyer, w.p.: smote: synthetic minority over-sampling technique. j. artif. intell. res. 16, 321–357 (2002) article  math  google scholar  das, s.d., vadi, y.a., yadav, k.: transformer-based joint modelling for automatic essay scoring and off-topic detection. in: proceedings of the 2024 joint international conference on computational linguistics, language resources and evaluation (lrec-coling 2024), pp. 16751–16761. elra and iccl, torino, italia (2024) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186. association for computational linguistics, minneapolis, minnesota (2019) google scholar  fonseca, e., medeiros, i., kamikawachi, d., bokan, a.: automatically grading brazilian student essays. in: villavicencio, a., et al. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 170–179. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_18 chapter  google scholar  hartmann, n., fonseca, e., shulby, c., treviso, m., silva, j., aluísio, s.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: proceedings of the 11th brazilian symposium in information and human language technology, pp. 122–131. sociedade brasileira de computação, uberlândia, brazil (2017) google scholar  he, h., bai, y., garcia, e.a., li, s.: adasyn: adaptive synthetic sampling approach for imbalanced learning. in: 2008 ieee international joint conference on neural networks (ieee world congress on computational intelligence), pp. 1322–1328. ieee, hong kong (2008) google scholar  higgins, d., burstein, j., attali, y.: identifying off-topic student essays without topic-specific training data. nat. lang. eng. 12(2), 145–159 (2006) article  google scholar  huang, p., li, l., wu, c., zhang, x., liu, z.: a study of sentence-bert based essay off-topic detection. in: proceedings of the 4th international conference on computing, networks and internet of things, pp. 515–519. association for computing machinery, xiamen, china (2023) google scholar  kabra, a., bhatia, m., singla, y.k., jessy li, j., ratn shah, r.: evaluation toolkit for robustness testing of automatic essay scoring systems. in: proceedings of the 5th joint international conference on data science & management of data (9th acm ikdd cods and 27th comad), pp. 90–99. association for computing machinery, bangalore, india (2022) google scholar  ke, z., ng, v.: automated essay scoring: a survey of the state of the art. in: proceedings of the 28th international joint conference on artificial intelligence, pp. 6300–6308. aaai press, macao, china (2019) google scholar  kusner, m., sun, y., kolkin, n., weinberger, k.: from word embeddings to document distances. in: proceedings of the 32nd international conference on machine learning, pp. 957–966. proceedings of machine learning research, pmlr, lille, france (2015) google scholar  landauer, t.k., foltz, p.w., laham, d.: an introduction to latent semantic analysis. discourse process. 25(2–3), 259–284 (1998) article  math  google scholar  lemaître, g., nogueira, f., aridas, c.k.: imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. j. mach. learn. res. 18(17), 1–5 (2017) math  google scholar  marinho, j.c., anchiêta, r.t., moura, r.s.: essay-br: a brazilian corpus of essays. in: xxxiv simpósio brasileiro de banco de dados: dataset showcase workshop. sbbd 2021, pp. 53–64. sbc, online (2021) google scholar  marinho, j.c., c., f., anchiêta, r.t., moura, r.s.: automated essay scoring: an approach based on enem competencies. in: anais do xix encontro nacional de inteligência artificial e computacional, pp. 49–60. sbc, campinas, brazil (2022) google scholar  page, e.b.: the imminence of... grading essays by computer. the phi delta kappan 47(5), 238–243 (1966) google scholar  passero, g., ferreira, r., dazzi, r.l.s.: off-topic essay detection: a comparative study on the portuguese language. revista brasileira de informática na educação 27(03), 177–190 (2019) article  math  google scholar  passero, g., ferreira, r., haendchen filho, a., dazzi, r.: off-topic essay detection: a systematic review. in: proceedings of the xxviii brazilian symposium on computers in education, pp. 51–60. sbc, recife, brazil (2017) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  pinho, c.m.d.a., gaspar, m.a., sassi, r.j.: aplicação de técnicas de inteligência artificial para classificação de fuga ao tema em redações. educação em revista 40, e39773 (2024) google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 3982–3992. association for computational linguistics, hong kong, china (2019) google scholar  shermis, m.d., barrera, f.d.: exit assessments: evaluating writing ability through automated essay scoring. in: annual meeting of the american educational research association, pp. 1–30. eric, new orleans, la (2002) google scholar  de sousa, r.f., marinho, j.c., neto, f.a.r., anchiêta, r.t., moura, r.s.: piln at propor: a bert-based strategy for grading narrative essays. in: proceedings of the 16th international conference on computational processing of portuguese vol. 2, pp. 10–13. association for computational linguistics, santiago de compostela, galicia/spain (2024) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  řehůřek, r., sojka, p.: software framework for topic modelling with large corpora. in: proceedings of lrec 2010 workshop new challenges for nlp frameworks, pp. 46–50. university of malta, valletta, malta (2010) google scholar  download references acknowledgments the authors are grateful to fundação de amparo à pesquisa do estado do piauí (fapepi) and virtex telecom for supporting this work. author information authors and affiliations laboratory of artificial intelligence, robotics, and automation, federal institute of piauí, picos, pi, brazil joyce m. silva, rafael t. anchiêta & rogério f. de sousa laboratory of natural language processing, federal university of piauí, teresina, pi, brazil raimundo s. moura authors joyce m. silvaview author publications search author on:pubmed google scholar rafael t. anchiêtaview author publications search author on:pubmed google scholar rogério f. de sousaview author publications search author on:pubmed google scholar raimundo s. mouraview author publications search author on:pubmed google scholar corresponding author correspondence to joyce m. silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, j.m., anchiêta, r.t., de sousa, r.f., moura, r.s. (2025). investigating methods to detect off-topic essays. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_24 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automated essay scoring off-topic essays bert publish with us policies and ethics profiles rafael t. anchiêta view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a modified nsga-do for solving multiobjective optimization problems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a modified nsga-do for solving multiobjective optimization problems conference paper first online: 28 november 2021 pp 126–139 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a modified nsga-do for solving multiobjective optimization problems download book pdf download book epub jussara gomes machado  orcid: orcid.org/0000-0001-6899-053910, matheus giovanni pires  orcid: orcid.org/0000-0003-1998-030410, fabiana cristina bertoni  orcid: orcid.org/0000-0001-7759-460x10, adinovam henriques de macedo pimenta  orcid: orcid.org/0000-0001-9931-576311 & … heloisa de arruda camargo  orcid: orcid.org/0000-0002-5489-730612  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 755 accesses abstract this paper presents a novel multiobjective genetic algorithm, named modified non-dominated sorting genetic algorithm distance oriented (mnsga-do), which aims to adjust the nsga-do selection operator to improve its diversity when applied to continuous multiobjective optimization problems. in order to validate this new genetic algorithm, we carried out a performance comparison among it and the genetic algorithms nsga-ii and nsga-do, regarding continuous multiobjective optimization problems. to this aim, a set of standard benchmark problems, the so-called zdt functions, was applied considering the quality indicators generational distance, inverted generational distance and hypervolume as well as a time evaluation. the results demonstrate that mnsga-do overcomes nsga-ii and nsga-do in almost all benchmarks, obtaining more accurate solutions and diversity. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a comprehensive survey on nsga-ii for multi-objective optimization and applications article 17 june 2023 deep surrogate network enhancement of non-dominated sorting genetic algorithms for improved multi-objective optimization chapter © 2024 improved nsga-ii for minimum weight minimum connected dominating set problem chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms continuous optimization discrete optimization genetic techniques optimization calculus of variations and optimization 1 introduction multiobjective optimization is an area of multiple-criteria decision making, concerning mathematical optimization problems involving more than one objective function to be optimized simultaneously [12]. multiobjective optimization has been used to many fields of science and engineering, which provides multiple solutions representing the trade-offs among objectives in conflict. genetic algorithms have been applied to several classes of multiobjective optimization problems (mop) and have been shown to be promising for solving such problems efficiently [2, 8,9,10, 13, 16], along with the advantage of evaluating multiple potential solutions in a single iteration because they deal with a population of solutions. given the diversity of existing genetic algorithms for solving mop, named multiobjective genetic algorithms (moga), it is necessary to be aware of their benefits and drawbacks. thereby, this work performs a comparative study among the well-known moga nsga-ii [6], a modification of it, the nsga-do [15] and the novel mnsga-do (modified non-dominated sorting genetic algorithm distance oriented), designed as a adjustment of the nsga-do. the use of crowding distance by nsga-ii as the selection criterion can prioritize individuals that are farther from the optimal front. a selection process which guide the solutions to converge towards the ideal points along the pareto front, as proposed by nsga-do, have been developed as an alternative to enhance a diversity of solutions. however, the nsga-do was developed based on problems with discrete search space. so, when applied to continuous problems, it does not guarantee the assignment of solutions to all the ideal points what might concentrate its set of solutions in specific regions. another disadvantage is the amount of ideal points, which is proportional to the size of pareto front. this imbalance between the quantity of ideal points and the number of solutions that need to be selected makes the convergence time-consuming. in order to overcome these failures, we proposed the modified non-dominated sorting genetic algorithm distance oriented (mnsga-do). this moga calculates the length of pareto front and then estimates a partition considering the number of solutions to be found, setting the coordinates of the ideal points. in addition, mnsga-do assigns one solution to each ideal point, what ensures a diversity of solutions. to perform the proposed comparative analysis, the zdt family of functions [20] was selected, because it is a broad and popular set of test functions for benchmarking the performance of multiobjective optimization methods. for each nsga, a study of its convergence and distribution of solutions along the pareto front was performed by applying the quality indicators generational distance, inverted generational distance and hypervolume, according to the literature. in order to complement the analysis, the optimal pareto front were visually compared to the boundaries computed by the three moga. finally, we carried out an analysis of runtime and a statistical evaluation of the results achieved. 2 multiobjective optimization problems according to [12], a multiobjective optimization problem (mop) can be defined as follows: $$\begin{aligned} \begin{array}{ll} minimize \ \ &{} f(x)= (f_1(x),...,f_m (x))^t \\ subject \ to \ \ &{} g_j (x) \ge 0, j=1,...,j \\ &{} h_k(x) = 0, k=1,...,k\\ &{} x \in \varomega \end{array} \end{aligned}$$ (1) where j and k are the numbers of inequality and equality constraints, respectively. \(\varomega = \vargamma _{i=1}^{n} [a_i,b_i] \subseteq \mathbb {r}^n\) is the space, \(x=(x_1,...,x_n)^t \in \varomega \) is a candidate solution. \(f:\varomega \longrightarrow \mathbb {r}^m\) constitutes m conflicting objective functions and \(\mathbb {r}^m\) is called the objective space. the attainable objective set is defined as \(\vartheta = \{f(x)\mid x\in \varomega , g_j(x) \ge 0, h_k(x) = 0\}\), for \(j\in \{1,...,j\}\) and \(k\in \{1,...k\}\). \(x^1\) is said to dominate \(x^2\) (denoted as \(x^1 \preceq x^2\)) if and only if \(f_i(x^1) \le f_i(x^2)\) for every \(i \in \{1,...,m\}\) and \(f_l(x^1) < f_l(x^2)\) for at least one index \(l \in \{1,...,m\}\). a solution \(x^*\) is pareto-optimal to (1) if there is no other solution \(x \in \varomega \) such that \(x \preceq x^*\). \(f(x^*)\) is then called a pareto-optimal (objective) vector. the set of all pareto-optimal solutions is called the pareto-optimal set (ps). therefore, the set of all pareto-optimal vectors, \(ef = \{f(x) \in \mathbb {r}^m \mid x \in ps\}\), is called the efficient front (ef) [14]. 3 multiobjective genetic optimization genetic algorithms are suitable to solving multiobjective optimization problems because they deal simultaneously with a set of possible solutions (or a population). this allows to find several members of the pareto optimal set in a single run of the algorithm, instead of having to perform a series of separate runs. additionally, genetic algorithms are less susceptible to the shape or continuity of the pareto front [3]. moga are usually designed to meet two often conflicting goals: convergence, viewed as minimizing the distances between solutions and the ef, and diversity, which means maximize the spread of solutions along the ef. balancing convergence and diversity becomes much more difficult in many-objective optimization [12]. moga are population-based approaches which initiate with a randomly created population of individuals. then, the algorithm starts an iterative process that creates a new population at each generation, by the use of operators which simulate the process of natural evolution: selection, crossover and mutation. among of all moga approaches, the literature shows that the nsga-ii is one of the most used for solving multiobjective optimization problems [9, 10, 16]. recently, a modification on the nsga-ii was proposed by [15], seeking to improve the diversity of the set of non-dominated solutions. this new moga was called non-dominated sorting genetic algorithm distance oriented (nsga-do). in order to refine the nsga-do algorithm, we proposed an adjustment on it. a brief description of these approaches, which are evaluated in this work, is presented in sects. 3.1, 3.2 and 3.3, depicting only the procedures used by the selection operator, that is the point at which the three algorithms differ. a detailed mathematical formulation is left to the references cited. 3.1 nsga-ii the nsga-ii algorithm is based on an elitist dominance sorting. for each solution i, contained in the population of solutions, two values are calculated: \(nd_i\), the number of solutions that dominate solution i; and \(u_i\), the set of solutions that are dominated by solution i. solutions with \(nd_i = 0\) are contained in the \(f_1\) front (pareto front). then, for each solution j in \(u_i\), the \(nd_j\) is decremented for each i \(\prec \) j, where i \(\in f_1\). if \(nd_j = 0\), then solution j belongs to the next front, in this case, \(f_2\). this procedure is repeated until all solutions are classified in a front. this procedure consists of classifying the solutions of a set m in different fronts \(f_1,f_2,...,f_f\) according to the dominance degree of such solutions. to guarantee diversity at the front, nsga-ii employs an estimate of the density of solutions that surround each individual in the population. thus, the average distance of the two solutions adjacent to each individual is calculated for all objectives by crowding distance selection operator. the suitability of each solution (individual) i is determined by the following values: \(rank_i =\) f, the \(rank_i\) value is equal to the number of the \(f_f\) front to which it belongs; and \(crowdist_i\), the crowding distance value of i. thus, in the dominance sorting process, a solution i is more suitable than a solution j if: i has a ranking lower than j, that is, \(rank_i < rank_j\); or if both solutions have the same ranking and \(rank_i\) has a higher crowding distance value. offspring population and current generation population are combined and the individuals of the next generation are set by dominance sorting process. the new generation is filled by each front subsequently until the number of individuals reach the current population size. 3.2 nsga-do as nsga-do is based on nsga-ii, the way it works is similar. the difference between them is due to the selection operator that begins with the estimation of the ideal points in the pareto front \(f_1\). in order to find the ideal points, the algorithm calculates the length of the pareto front and then estimates an uniform partition, setting the coordinates of the ideal points. thereafter, the selection of solutions to be inserted in the next generation instead of considering the crowding distance as in nsga-ii, considers the tournament distance between each solution of a certain front f and the calculated ideal points, with the aim to enhance the diversity of the solutions. in order to improve understanding of the difference selection criterion of the closest solution to an ideal point (nsga-do) instead of the crowding distance (nsga-ii), consider the following example, presented by [15] and illustrated in fig. 1. it is possible to observe two representations of the pareto front, formed by the solutions belonging to the fronts f1 and f2. fig. 1. difference between nsga-ii and nsga-do selection operator [15]. full size image solutions belonging to the front f1 dominate the solutions belonging to the front f2 and are not dominated by any other solution, composing the pareto-optimal front. the black dots (i1, i2, i3, i4 and i5) represent the ideal points and the gray dots (f1s1, f1s2, f1s3, f1s4, f1s5, f2s1, f2s2, f2s3 and f2s4) represent the solutions found by a moga. the f1 and f2 are the conflicting objectives to be optimized. assuming that eight solutions would be selected for the next generation of the moga, nsga-ii and nsga-do would form different solution sets. initially, all solutions belonging to the front f1 (f1s1, f1s2, f1s3, f1s4, f1s5) would be selected, because the number of solutions on this front is less than the size of population (f\(_{\text {1 size}}<\) n). up to this point, the nsga-ii solution set is the same as the nsga-do solution set, both of which are: $$\begin{aligned} \mathrm{s}_{\text {nsga-ii}} = \mathrm{s}_{\text {nsga-do}} = \{\mathrm{f}_{1}\mathrm{s}_{1}, \mathrm{f}_{1}\mathrm{s}_{2},\mathrm{f}_{1}\mathrm{s}_{3}, \mathrm{f}_{1}\mathrm{s}_{4}, \mathrm{f}_{1}\mathrm{s}_{5}\} \end{aligned}$$ three solutions will be selected from f2 front (n f\(_{\text {1 size}}\)). the crowding distance operator from nsga-ii choose solutions with greater distance from its neighbors, in this case, solutions s1 and s4, because they are located at the extremity of front. solution s2 would be the last one to be chosen because its crowding distance is greater than the one of s3. on the other hand, the solutions chosen by nsga-do are those closest to the ideal points. among the solutions of front f2 the ranking of choice would be s3, s4 and s2, because they have the lowest distance to an ideal point. therefore, the final solution sets are: $$\begin{aligned} \mathrm{s}_{\text {nsga-ii}} = \{\mathrm{f}_{1}\mathrm{s}_{1}, \mathrm{f}_{1}\mathrm{s}_{2}, \mathrm{f}_{1}\mathrm{s}_{3}, \mathrm{f}_{1}\mathrm{s}_{4}, \mathrm{f}_{1}\mathrm{s}_{5}, \mathrm{f}_{2}\mathrm{s}_{1}, \mathrm{f}_{2}\mathrm{s}_{4}, \mathrm{f}_{2}\mathrm{s}_{2}\}\\ \mathrm{s}_{\text {nsga-do}} = \{\mathrm{f}_{1}\mathrm{s}_{1}, \mathrm{f}_{1}\mathrm{s}_{2}, \mathrm{f}_{1}\mathrm{s}_{3}, \mathrm{f}_{1}\mathrm{s}_{4}, \mathrm{f}_{1}\mathrm{s}_{5}, \mathrm{f}_{2}\mathrm{s}_{3}, \mathrm{f}_{2}\mathrm{s}_{4}, \mathrm{f}_{2}\mathrm{s}_{2}\} \end{aligned}$$ 3.3 mnsga-do modified non-dominated sorting genetic algorithm distance oriented (mnsga-do) is an extension of nsga-do, with the purpose to improve its performance. in the way nsga-do works, it estimates ideal points in the pareto front based on an uniform partition, proportional to its size, and for each solution of a certain front f, the distance from the ideal points is calculated. then, these distances are sorted, selecting the s solutions associated with the smallest distances, where s is the number of individuals to reach the current population size. in this way, if the s smallest distances are associated with a single ideal point, the selected solutions will be concentrated in a single region of the front, harming diversity. instead of estimating ideal points based on an uniform partition proportional to the size of pareto front, mnsga-do calculates an uniform partition based on the number of solutions to be found, setting the coordinates of the ideal points in the pareto front \(f_1\), what decrease convergence time by reducing the number of points when calculating distances. to improve the approximation to the pareto front, mnsga-do uses the number of ideal points equal to twice the number of solutions to be selected (2s). then, it calculates the distances from solutions in f to ideal points, starting from the ends to the center of the pareto front, switching the selection between the two ends. moreover, mnsga-do does not allow the repetition of the same solution for different points. when a solution has already been associated with a point, mnsga-do do not considers this solution to the other ideal points. thus, mnsga-do guarantee the assignment of one different solution to each ideal point, avoiding the concentration of solutions in some of them and ensuring the diversity of the solutions. mnsga-do selection operator is illustrated in algorithm 1. 4 experimental setup for our experiments, the zdt family of functions [20] was selected, because it is a widely used set of test functions for benchmarking the performance of multiobjective optimization methods. all of the zdt functions contain two objectives and a particular feature that is representative of a real world optimization problem that could cause difficulty in converging to the pareto-optimal front [18], as showed in table 1. table 1. zdt family characteristics.full size table in this study, 5 out of 6 zdt functions were considered (zdt{1–4} and zdt6) and for each one, 30 runs were conducted using each moga, in order to ensure the results were not biased based upon the initial population. we performed our experiments on 2.40 ghz pc with 8 gb ram and operating system 64 bits. the implementations of mogas were done in the framework jmetal 5 [7] and assumed the following parameters: codification: the chromosomes \(ci=(ci_1,ci_2,...ci_n)\) are encoded as a vector of floating point numbers in which each component of the vector is a variable of the problem. initial population and stopping criterion were defined following a grid search procedure [4]. the values tested for the population size were 50, 100, 150 and 200. for the number of fitness function evaluations were considered values 10000, 20000, 25000 and 30000. after these tests, initial population was generated randomly considering population size of 100 individuals and stopping criterion was defined as 25000 fitness function evaluations, in all problems. fitness function: the fitness function for mop is the objective function f(x) to be minimized. the goal of solving a mop is to find the pareto-optimal set or at least a set of solutions close to pareto-optimal set. intermediate population: based on [5, 11, 17], we have used binary tournament selection, simulated binary crossover (sbx) and polynomial mutation. the crossover probability and distribution index were respectively defined as 0.9 and 30. similarly, the mutation probability was set as 1/n, where n is the number of problem variables, and the mutation distribution index was 20. the study of convergence and distribution of solutions along the pareto front was performed to the mogas by applying three well known quality indicators, generational distance (gd), inverted generational distance (igd) and hypervolume (hv) [1]. the first one measure the convergence and the other indicators can simultaneously measure the convergence and diversity of obtained solutions. in the following section, the performance of the nsga-ii, nsga-do and mnsga-do on zdt test problems is investigated and the results are presented based on the mentioned quality indicators. 5 experimental results as stated in sect. 4, we have selected five functions from zdt family to evaluate nsga-ii, nsga-do and mnsga-do algorithms: zdt{1–4} and zdt6. according to the moga configurations presented in sect. 4 and regarding zdt functions mentioned, we applied the quality indicators gd, igd and hv, which make a comparison between the set of points sampled from the pareto-optimal (set mathematically calculated) and the set of points found by the moga to be evaluated. thirty design variables \(x_i\) were chosen to zdt{1–4} and ten to zdt6. each design variable ranged in value from 0 to 1, except to zdt4, which the variables ranged from −5 to 5. the numerical results of gd, igd and hypervolume are showed in tables 4, 5 and 6, respectively. the values highlighted in gray means the best for each zdt function and sd column shows standard deviation. according to the results, for the quality indicator gd, which only evaluates convergence, nsga-do found the best results. for both quality indicators igd and hv, which simultaneously measure the convergence and diversity, the mnsga-do had the best results, except zdt3 problem, in which the nsga-ii was slightly better. this happens because both nsga-do and mnsga-do consider the entire extension of pareto front for the definition of ideal points, not taking into account the gaps of the discontinuous zdt3 problem. thereby, ideal points are assigned in infeasible regions of the search space, thus compromising their effectiveness. in table 2, the execution time of the algorithms is presented in seconds. nsga-ii performed better than mnsga-do. nsga-do achieved the worst results, with execution time much higher than the other two multiobjective genetic algorithms. table 2. execution time results.full size table as the result of a two-objective pareto optimization study is a set of points on a curve (the pareto front), we plotted the last fronts from each algorithm with the aim of visualising the results variation. hence, a visual comparison between the algorithms is performed based on the pareto fronts. figures 2, 3, 4, 5 and 6 illustrate the pareto-optimal, nsga-ii, mnsga-do and nsga-do fronts of each zdt function. the x and y axes of all figures represent objectives \(f_1\) and \(f_2\), respectively. after analysing the fronts, we can observe that the nsga-ii and the mnsga-do fronts are very similar comparing to pareto-optimal front. nsga-ii fronts shows some gaps, what demonstrates that the nsga-ii fronts diversity is worst than mnsga-do. on the other side, the nsga-do front is very different from pareto-optimal front. this problem can be attributed to the process of individuals selection to reach the current population size, which consider the increasing sorting of the distances from the solutions to the ideal points. such sorting can add individuals from a single region of the front in the new population. these individuals, combined with others, generate new individuals with characteristics similar to theirs, leading to agglomeration of solutions in regions of the objective space. in addition, as the nsga-do algorithm does not balance the number of solutions selected by each ideal point, one same point can select various solutions and another none. in order to verify if there is statistical difference among the mogas results, we have applied the wilcoxon signed-rank test [19], with level of significance = 0.05. the statistical results are shown in table 3, in which one – symbol means the null hypothesis was accepted, and one \(\blacktriangle \) or \(\triangledown \) symbol means the null hypothesis was rejected. the \(\blacktriangle \) symbol indicate that the algorithm from the line was significantly better than the algorithm from the column, and the \(\triangledown \) symbol indicate the opposite. each –, \(\blacktriangle \) or \(\triangledown \) symbols refers to a function, that is, the first symbol refers to zdt1 function, the second symbol refers to zdt2 function, and so on. table 3. statistical comparison on quality indicators gd, igd and hv.full size table based on table 3, when considering quality indicator gd, the nsga-do was significantly better than mnsga-do and nsga-ii in almost all functions, only to the zdt6 function there was not statistical difference among them. between mnsga-do and nsga-ii, mnsga-do was better than nsga-ii to zdt1 and zdt2 functions, and nsga-ii was better in zdt3 and zdt4 functions. regarding quality indicator igd, mnsga-do and nsga-ii were significantly better than nsga-do in all functions. take into account only mnsga-do and nsga-ii, mnsga-do was better in almost all functions, except to the zdt3 function. considering quality indicator hv, mnsga-do and nsga-ii were significantly better than nsga-do in all functions, once more. comparing mnsga-do and nsga-ii, they are similar to zdt3 function, but mnsga-do was markedly better in all others. finally, evaluating execution time, mnsga-do and nsga-ii were significantly better than nsga-do in all the problems. regarding mnsga-do and nsga-ii, they are similar to zdt1–3, but mnsga-do was worse than nsga-ii to zdt4 and zdt6. fig. 2. pareto-optimal, nsga-ii, mnsga-do and nsga-do from zdt1 function. full size image fig. 3. pareto-optimal, nsga-ii, mnsga-do and nsga-do from zdt2 function. full size image table 4. generational distance results.full size table table 5. inverted generational distance results.full size table table 6. hypervolume results.full size table fig. 4. pareto-optimal, nsga-ii, mnsga-do and nsga-do from zdt3 function. full size image fig. 5. pareto-optimal, nsga-ii, mnsga-do and nsga-do from zdt4 function. full size image fig. 6. pareto-optimal, nsga-ii, mnsga-do and nsga-do from zdt6 function. full size image 6 conclusion in this paper we have presented a novel multiobjective genetic algorithm and made a comparison with nsga-ii and nsga-do in solving multiobjective optimization problems. in addition, we investigated their performance when applied to the popular zdt family benchmark, by analysing the convergence and diversity of these mogas based on three quality indicators, gd, igd and hv. according to statistical results, nsga-do was significantly better considering the quality indicator gd, while mnsga-do was significantly better taking account the quality indicators igd and hv. analysing the pareto fronts from mogas, it was possible to verify that nsga-do pareto front solutions are concentrated in a few points, mainly in zdt1, zdt2, zdt3 and zdt6 functions. this result can be explained by nsga-do individuals selection process, which allow one ideal point select various solutions and another none, leading to agglomeration of solutions in regions of the objective space. on the other side, the mnsga-do fronts has better convergence and diversity of the solutions in comparison with nsga-ii. the nsga-do, proposed to improve the distribution of solutions along the pareto front, fails to overcome the crowding distance of the nsga-ii in continuous problems. as nsga-do was developed based on discrete problems, the gaps in pareto front are not important, what harms its performance in this class of problems. in this context, mnsga-do appears as an alternative in solving continuous multiobjective optimization problems and its results demonstrate its feasibility and efficiency. in future works, we aim to apply mnsga-do in solving multiobjective discrete optimization problems and compare it with nsga-ii and nsga-do by analysing the convergence and diversity as done in this work. moreover, we are going to investigate the adjustment of mnsga-do regarding problems with discontinuous fronts. references audet, c., bigeon, j., cartier, d., le digabel, s., salomon, l.: performance indicators in multiobjective optimization. eur. j. oper. res. 292(2), 397–422 (2021). https://doi.org/10.1016/j.ejor.2020.11.016 article  mathscinet  math  google scholar  bora, t.c., mariani, v.c., dos santos coelho, l.: multi-objective optimization of the environmental-economic dispatch with reinforcement learning based on non-dominated sorting genetic algorithm. appl. therm. eng. 146, 688–700 (2019). https://doi.org/10.1016/j.applthermaleng.2018.10.020 article  google scholar  chang, k.h.: e-design: computer-aided engineering design. elsevier sci. (2015). https://doi.org/10.1016/c2009-0-63076-2 article  google scholar  coroiu, a.m.: tuning model parameters through a genetic algorithm approach. in: 2016 ieee 12th international conference on intelligent computer communication and processing (iccp), pp. 135–140 (2016). https://doi.org/10.1109/iccp.2016.7737135 deb, k., jain, h.: an evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. ieee trans. evol. comput. 18(4), 577–601 (2014) article  google scholar  deb, k., agrawal, s., pratab, a., meyarivan, t.: a fast elitist non-dominated sorting genetic algorithm for multi-objective optimization: nsga-ii. in: international conference parallel problem solving from nature, vol. 1917 (2000) google scholar  durillo, j.j., nebro, a.j.: jmetal: a java framework for multi-objective optimization. adv. eng. softw. 42(10), 760–771 (2011) article  google scholar  george, t., amudha, t.: genetic algorithm based multi-objective optimization framework to solve traveling salesman problem. in: sharma, h., govindan, k., poonia, r.c., kumar, s., el-medany, w.m. (eds.) advances in computing and intelligent systems. ais, pp. 141–151. springer, singapore (2020). https://doi.org/10.1007/978-981-15-0222-4_12 chapter  google scholar  guerrero, c., lera, i., juiz, c.: genetic algorithm for multi-objective optimization of container allocation in cloud architecture. j. grid comput. 16(1), 113–135 (2017). https://doi.org/10.1007/s10723-017-9419-x article  google scholar  hamdy, m., nguyen, a.t., hensen, j.l.m.: a performance comparison of multi-objective optimization algorithms for solving nearly-zero-energy-building design problems. energy build. 121, 57–71 (2016). https://doi.org/10.1016/j.enbuild.2016.03.035 article  google scholar  ishibuchi, h., imada, r., setoguchi, y., nojima, y.: performance comparison of nsga-ii and nsga-iii on various many-objective test problems. in: 2016 ieee congress on evolutionary computation, pp. 3045–3052. ieee (2016) google scholar  li, k., deb, k., zhang, q., kwong, s.: an evolutionary many-objective optimization algorithm based on dominance and decomposition. ieee trans. evol. comput. 19(5), 694–716 (2015) article  google scholar  maghawry, a., hodhod, r., omar, y., kholief, m.: an approach for optimizing multi-objective problems using hybrid genetic algorithms. soft. comput. 25(1), 389–405 (2020). https://doi.org/10.1007/s00500-020-05149-3 article  google scholar  miettinen, k.: nonlinear multiobjective optimization. kluwer academic publishers (1999) google scholar  pimenta, a., camargo, h.: nsga-do: non-dominated sorting genetic algorithm distance oriented. in: ieee international conference fuzzy system, pp. 1–8 (2015). https://doi.org/10.1109/fuzz-ieee.2015.7338080 saborido, r., ruiz, a.b., bermúdez, j.d., vercher, e., luque, m.: evolutionary multi-objective optimization algorithms for fuzzy portfolio selection. appl. soft comput. 39, 48–63 (2016). https://doi.org/10.1016/j.asoc.2015.11.005 article  math  google scholar  seada, h., deb, k.: u-nsga-iii: a unified evolutionary optimization procedure for single, multiple, and many objectives: proof-of-principle results. in: gaspar-cunha, a., henggeler antunes, c., coello, c.c. (eds.) emo 2015. lncs, vol. 9019, pp. 34–49. springer, cham (2015). https://doi.org/10.1007/978-3-319-15892-1_3 chapter  google scholar  priya, v., umamaheswari, k.: enhanced continuous and discrete multi objective particle swarm optimization for text summarization. clust. comput. 22(1), 229–240 (2018). https://doi.org/10.1007/s10586-018-2674-1 article  google scholar  wilcoxon, f.: individual comparisons by ranking methods. biometrics bull. 1(6), 80–83 (1945). https://doi.org/10.2307/3001968 article  google scholar  zitzler, e., deb, k., thiele, l.: comparison of multiobjective evolutionary algorithms: empirical results. evol. comput. 8(2), 173–195 (2000). https://doi.org/10.1162/106365600568202 article  google scholar  download references author information authors and affiliations state university of feira de santana, feira de santana, 44036-900, ba, brazil jussara gomes machado, matheus giovanni pires & fabiana cristina bertoni adventist university centre of são paulo, são paulo, 05858-001, sp, brazil adinovam henriques de macedo pimenta federal university of são carlos, são carlos, 13565-905, sp, brazil heloisa de arruda camargo authors jussara gomes machadoview author publications search author on:pubmed google scholar matheus giovanni piresview author publications search author on:pubmed google scholar fabiana cristina bertoniview author publications search author on:pubmed google scholar adinovam henriques de macedo pimentaview author publications search author on:pubmed google scholar heloisa de arruda camargoview author publications search author on:pubmed google scholar corresponding author correspondence to matheus giovanni pires . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper machado, j.g., pires, m.g., bertoni, f.c., de macedo pimenta, a.h., de arruda camargo, h. (2021). a modified nsga-do for solving multiobjective optimization problems. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_9 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multiobjective genetic algorithm multiobjective optimization nsga-ii nsga-do publish with us policies and ethics profiles heloisa de arruda camargo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature intelligent agents for observation and containment of malicious targets organizations | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper intelligent agents for observation and containment of malicious targets organizations conference paper first online: 28 november 2021 pp 48–63 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) intelligent agents for observation and containment of malicious targets organizations download book pdf download book epub thayanne frança da silva10, matheus santos araújo10, raimundo juracy campos ferro junior10, leonardo ferreira da costa11, joão pedro bernardino andrade11 & … gustavo augusto lima de campos10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 752 accesses abstract the problem addressed in this work is an extension of the cooperative multi-robot observation of multiple moving targets problem (cmommt). the scenario remains the same, but the targets are structured as an organization to achieve the highest possible percentage of exploration of the environment and avoid robots. targets can be organized as hierarchy, holarchy, team, and coalition, but they can also be unorganized. our work seeks to apply computer vision to assist robots in classifying the target team’s organizational structure faced with a group of malicious target agents. thus, robots can select the most appropriate strategy among the containment strategies implemented for each organizational structure or continue with the method proposed by literature for cases where the targets are not organized. the results showed that our approach had satisfactory results since, in luck, robots have a 20% chance of hitting the structure (hierarchy, holarchy, team, coalition, or random). our approach had an accuracy of 63.28%. the containment strategies obtained satisfactory results in the robots’ performance regarding the depreciation of the percentage of the environment explored by the targets (peet) compared to the previous approach for robots. however, for the average number of observed targets (anot), the previous strategy was better. the new organizational approach to targets in cmommt was better than random in the desired exploration of the desired environment. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others distributed assignment with limited communication for multi-robot multi-target tracking article 17 may 2019 an integer linear programming model for fair multitarget tracking in cooperative multirobot systems article 18 april 2018 cooperative observation of smart target agents chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. multiagent systems organization theory organized crime organizational theory robotics social robotics 1 introduction the observation of moving targets is an essential multi-robot application in mas that still presents numerous open challenges, including the effective coordination of robots [11]. the cmommt addresses the scenario where a group of agents, called robots, seek to retain within an “observation band” a maximum number of agents from the opposite group, called targets [14, 16]. in one of the reformulations of cmommt, cooperative target observation (cto) [13, 18] proposed targeting strategies based on structures to improve the performance of this team in the cto. the results showed that the strategies of [18] were better for target performance compared to the proposed strategy [13]. thus, in our approach, we implement four organizational paradigms for multi-agent systems [10], in the target team in cmommt, in a scenario where targets seek not only to walk randomly but explore the environment while avoiding robots. research has shown that it is possible to recognize patterns in images captured by satellites [21], infrared cameras [6], security cameras [3, 4], etc. furthermore, there are researches in the area of multi-agent simulation that made use of computer vision in the simulation image to locate agents in the environment [15]. thus, this research applied computer vision to help classify the four organizational paradigms modeled for the targets (hierarchy, holarchy, team, and coalition) and the random model through images captured from the environment. thus, if we model the organizational structures in the targets so that communication between them depends on the location of these agents with their coordinators, sub-coordinators, or group partners, we can apply computer vision to classify the organizational paradigms present in the targets. with this information, robots can now select an appropriate containment approach for each of the four organizations developed or remain in the strategy proposed by [14] if the targets are walking randomly. this article aims to develop a strategy for the decision-making of artificial robot agents, aiming to maximize the observation metric and minimize the exploration of the environment by target agents organized to explore and perform malicious actions in the background. this paper is organized into four more sections. section 2 presents the literature review. section 3 describes the approaches used by targets and robots, materials, and methods. section 4 shows the experiment and the results. finally, sect. 5 concludes the paper with final observations and future research. 2 a literature review the cmommt problem, initially described in [14], is defined in a simple, two-dimensional polyhedral spatial region, with inputs/outputs containing two teams of agents, the targets and the robots. the team of robots has \( 360^{\circ } \) observation sensors. this team’s objective is to maximize the collective time during which each target in the environment is monitored by at least one robot during the simulation time. we say that a robot is tracking a target when the target is within the robot’s sensory field of observation. some reformulations of this problem have been created, such as fcmommt [8], p-cmommt [2, 9]. the cto is another reformulation of the cmommt [13]. the main difference between these two problems is that in the cto, the targets provide information about their location. the observers’ objectives remain the same as that of the robots. the targets were modeled with random movements, the target team being superior in numbers to the observer team, but the observers are faster. in [1], it was observed that observers at the cto work in a hierarchical structure with the k-means and fuzzy c-means algorithm. thus, [18] proposed three strategies so that the targets were as organized as the observers, hierarchy with k-means, the hierarchy with fuzzy c-means, and holarchy, and a strategy with neural networks. the results showed that the method based on the organizational paradigms was superior to the cto literature approach. [19] proposed an approach to classify the organizational structure of a group of target mobile agents that are continuously monitored by a smaller group of mobile observer agents in the cto problem, a reformulation of the cmommt. the approach considers that the group of target agents can be organized according to eight different paradigms. these agents communicate through the exchange of messages whose contents are performative of the speech act. this approach proved effective in comparison with the dummy classifier that simulated human logic based on the frequency strategy. however, it was observed in this approach that the organizational strategy adopted for the targets, targets, and observers provided their location information to each other in the cto. this type of scenario, where opposing teams exchange information when their goals neutralize each other, is not a realistic scenario. thus, in the cmommt scenario, where the environment is partially observable, the targets and robots (observers) do not contribute with their locations, the simulation becomes more realistic. in [5, 15], computer vision was applied to the images captured by agents to help achieve their goals. [5] presented the development of a robotic multi-agent system, called smart, in which there are two groups of agents, hardware and software agents, that work cooperatively. the hardware agents are robots with three and four legs and an ip camera that captures images of the scene where the cooperative task takes place. [15] presented a behavior-based approach for maintaining robot formation. the robot’s objective is to circulate through the environment, keeping a relative position between them and avoiding shock. based on these concepts of an image capture agent, the use of the computer vision technique, and the limitation of target communication to maintain the organization in cmommt, we can assist robots in classifying target structures through the application of computer vision in the simulation scenario, such as [5, 15]. 3 the approach for observing and containment of malicious targets organizations in this article, we present an extension to the cmommt problem. the scenario remains the same, but the targets are structured as an organization aiming to achieve the highest possible percentage of scenario exploration while avoiding robots. robots continue with the surveillance objective, seeking to maximize the vision on the targets and minimize the effect of the organized targets on the environment through strategies to contain the exploration of the targets. in the original cmommt problem, robots were only concerned with observing targets. however, this approach is not sufficient for robots in this extension of the cmommt problem. the targets are intended to explore the environment, and they are organized as hierarchy, holarchy, team, or coalition. therefore, the proposed approach for robots in this new cmommt problem is using image classification to select the containment strategy to minimize the exploration of targets organized in the scenario. considering the assumption that there is a drone in the scenario capable of capturing images of both the target team and the robots, it is possible to generate a set of examples, label them and train a classification system that can recognize patterns in movement in each organizational paradigm present in the targets to classify the structure. thus, through classification, the work proposes four associated containment strategies to solve the problem. 3.1 target strategy targets, in this extension of the cmommt problem, see the environment in quadrants. there are four quadrants: upper right, upper left, lower left, and lower right. among the paradigms raised by [10], hierarchy, holarchy, team, and coalition were selected for this work. because, according to [10], starting from these four, it is possible to generate the others. the subsections below detail each organizational structure modeled on the target team. hierarchy. in this organizational structure, only the two-level hierarchy was considered. at the level above, there is a target responsible for calculating the quadrant that contains the smallest percentage of exploration closest to it, based on the information obtained by the targets, to request the movement of this team to this particular quadrant at each time interval, called of coordinator. the targets that inform the area explored by them and perform the action requested by the coordinator are called subordinates. therefore, subordinates must remain within the message range to receive the information, as communication is carried out through the speech act. the state machine that demonstrates target team communication is shown in fig. 1. in the initial state (\(q_{0}\)), the coordinator (c), belonging to the target team (t), requests that all subordinates (s) tell you the coordinates of your current state. in state \(q_{1}\), subordinates report their status to the coordinator. finally, in the state (\(q_{2}\)), the coordinator requests all targets to move towards the goal calculated by it. as the communication is not continuous but occurs every period after reaching the final state, the state machine will only restart again in the initial state at the time determined to exchange messages from the target team. fig. 1. hierarchy communication state machine. full size image holarchy. in holarchy, targets have been separated into two holon structured as a simple hierarchy, each containing a sub-coordinator who has authority over the subordinates of his holon. our approach has only been tested with this simpler holarchy. there is a general coordinator that performs the same calculation process as the hierarchy coordinator. however, this one only has access to the sub-coordinators. therefore, the coordinator transmits the message to the sub-coordinators, and these send the message to their subordinates. hierarchy communication state machine. at each given time, the coordinator informs the sub-coordinators of their environment analysis based on information obtained by all members of this paradigm (coordinator, sub-coordinator, and subordinates). the sub-coordinators request an action from their subordinates, who are within the sub-coordinator’s speech act range, based on the analysis of the general coordinator. the coordinator’s message is only received if the sub-coordinators are within the coordinator’s message transmission range. likewise, the subordinates of each holon must be within range of their sub-coordinator’s message transmission. figure 2 shows the state machine of the target team communication in the holarchy. in the initial state (\(q_{0}\)), the coordinator (c), belonging to the target team (t), requests that all sub-coordinators (sc) ask all their subordinates to inform them of the coordinates of their current status. in the \(q_{1}\) state, sub-coordinators request this information from subordinates. in the state \(q_{2}\) and \(q_{3}\), the subordinates report their status to their sub-coordinators, and these report the status of their subordinates to the coordinator, respectively. in the next state, the coordinator asks the sub-coordinators to ask their subordinates to go towards the goal calculated by the coordinator. finally, in the \(q_{5}\) state, the subordinates perform the action forwarded by their sub-coordinator. finally, in the final state (\(q_{0}\)), the machine is shut down until the next communication period between the target team. fig. 2. holarchy communication state machine. full size image team. in this organizational paradigm, all members are at the same level and divided into four groups containing the same number of members. each group is sent to a region of the environment (quadrant) to accomplish its objective. each sub-team member must remain within a certain radius to maintain communication and must stay in the area assigned to their sub-team. as shown in fig. 3, in the initial state, (\(q_{0}\)), the targets of each sub-team report the exploration rate around them to their teammates. in the final state, (\(q_{1}\)), each sub-team will ask its members to go to the coordinates of the target that obtained the lowest exploration rate around it in that sub-team. fig. 3. team communication state machine. full size image coalition. on our problem, there are two coalitions, the members located on the right in the scene form a coalition, and those found on the left form another coalition. as targets are placed randomly in the environment, each coalition can have a different number of members. communication, just like in the team, takes place at a certain point in time, and the members of each coalition must stay within a certain radius to enable communication. the two coalitions tend to be separated from each other, as each is allocated to a specific area. however, the targets of each coalition must be close to each other to carry out decision-making. in fig. 4, the communication state machine is presented. in the initial state (\(q_{0}\)), each target of each coalition informs the exploration state of the region around them. in the \(q_{1}\) state, coalitions request that targets belonging to their coalition go to the target coordinates of each coalition that has the lowest exploitation rate in its surroundings. 3.2 robots strategy computer vision. initially, a survey of classic cnn models was carried out to evaluate each model for classification of organizational paradigms and the random model. mobilenetsv2 was selected because it is a simple but efficient convolutional neural network. in order to carry out the classification in real-time simulation, a light network is needed for this task. fig. 4. coalition communication state machine. full size image other cnn models can be used. but, as our objective is to evaluate the benefits of using simulation scenario images with cnn for agents’ decision-making, only mobilenetsv2 was used in this paper. for our proposed model, which uses netlogo simulation scenario images, we train our dataset to 100 epochs using the adam optimizer and an initial learning rate of \(1e-3\). in this work, we test the lot size valued at 32. \(80\%\) from our image bank was used for the training step and \(20\%\) for the validation step. in the training stage, we obtained \(74.56\%\) accuracy. in the validation step, the accuracy of \(72.14\%\) was obtained. during the simulation, images of the scenario are sent to classify the organizational structure of the targets. four containment strategies for the robot team are proposed to deal with the organizational structure that targets may adopt. the following subsections describe these strategies. strategy against hierarchy. in this type of organization, in which the targets are submitted to a higher-level agent and where decision-making depends on the latter’s endorsement, the containment strategy adopted was to disable the coordinator’s communication with the targets. as only the coordinator requests an action and the agents’ act of speaking represents in the real world the human speech itself, as soon as the hierarchical structure is detected, the robots move between the targets in order to see which one is communicating an action. thus, the robots will disable this agent’s communication, and the targets will be incapacitated, as they will not be able to reach their goals because the agent in charge of transmitting information about their goals has been disabled. however, robots continue to look for more coordinators, as they do not have the details on how the hierarchy is structured, two levels or more. strategy against holarchy. in holarchy, it is necessary to disable the general coordinator and the sub-coordinators. for however much the sub-coordinators are submissive to the general coordinator, they have authority over the targets of their group. if so, the holarchy could turn into coalitions, and targets could still achieve their goals. therefore, the containment strategy adopted for this structure is to disable the communication of all targets that have authority over other targets. strategy against the team. in the team’s case, the targets are independent of each other but cooperate to achieve their goals; that is, they explore the area allocated to their team to maximize the percentage of the region explored by the targets. as there is no essential agent in this structure that its suppression disables, as all targets are at the same level, the strategy adopted was an extension of the approach proposed by [14]. in this extension, robots have two behaviors, that of seeking sub-times and that of disrupting the communication of targets under observation. as the targets of each subteam are close to each other, if the force field proposed by [14] were used to repel all other robots, only four robots would be trying to disable communication, while the other eight would be idle. thus, a conditional was added to this force field. it now allows a maximum of two robots and repels the others when it exceeds this margin. so, instead of just one robot trying to disable the communication of six targets, there are now two for this purpose in each sub-team. the four remaining robots are responsible for looking for more sub-teams, as the robot team does not know how many sub-teams there are in the simulation scenario. strategy against the coalition. as the number of members in each coalition can be unequal and there are no agents with authority over others, the containment strategy was an extension of the method proposed by [14] as well. they are seeking, as well as in the team, to break communication between the coalition targets, as there are not enough robots to disable all targets to minimize the exploration of the scenario. strategy overview. thus, the robot approach consists of them, every 200 steps of time on the netlogo platform, sending an image of the simulation scenario to the jupyter notebook. then, the classification of the organizational paradigm by mobilenetv2 is performed, and the result is returned to the robots team. thus, from the returned response, the most suitable containment strategy is selected. the simulation only returns to processing when the jupyter notebook returns the classification value performed by cnn. 3.3 materials and methods the netlogo [20] platform was selected for scenario simulation, as [1, 18, 19] used to simulate a reformulation of the cmommt problem, the cto. this platform was also chosen for its integrality with the jupyter notebook [12] platform used in this research. mobilenetv2 [17] was loaded by keras [7], an open-source neural network library written in python. in the generation of the image bank, the data augmentation technique was used to supplement our dataset. in addition, the use of this technique simulates a drone flying over the scene and capturing images of the robot team and targets from various angles and positions. our image bank was generated from the simulation scenario images. the starting position of each target and robot is random, and the choice of the following position is based on the strategy adopted by each team. seeking to diversify the image bank, the target communication range sensor can vary between 5, 10, 15, 20, and 25 netlogo distance units. thus, targets can be further away or closer depending on the communication range setting. initially, the image bank contains 750 images of each organizational paradigm and the random model for [14] targets, in which there are 150 images for each communication range sensor configuration, totaling 3750 images. the photos were captured manually in the most diverse positions, rotations, and transitions from one quadrant to another for better learning of the model. after capturing the 3750 images of the scenery, the data augmentation technique was applied, which generated our final image bank with 41, 250 imagesfootnote 1. figure 5 shows the simulation environment with the targets structured as a hierarchy, where the “arrows” agents are the robots and the “people” agents are the targets. according to the robot closest to them, the colors of the target agents are responsible for their observation. there is a quadrant division to aid in viewing target movements and image processing. fig. 5. netlogo simulation example. full size image 4 experiments and results 4.1 test settings the configured parameters were based on the settings used by [13]. targets and robots are in a rectangular field with dimensions 150 by 150 units; there are 1500 interaction steps per simulation; observer speed is 1 step at each interaction step; target speed is 0.9 steps at each interaction step; sensor range is 25 units; communication range varies between 5, 10, 15, 20 and 25 units; 24 targets; 12 robots; in the case of hierarchy there is an extra target, called coordinator; in the case of holarchy there are three extra targets, called general coordinator and sub-coordinators. in order to evaluate the performance of the robots, it was configured for targets with the highest speed and range of vision defined by [13]. well, this is the most challenging scenario for the team of configured robots today. thus, if satisfactory results were obtained in this scenario, in better scenes for robots, the performance tends to be acceptable as well. 4.2 result of the classification of organizational structures and the random model for our proposed model, we train our dataset for 1000 epochs with the lot size value of 32. our model was tested to classify images that were not included in training or validation sets. our test suite consists of 3750 images, with 750 from each organizational paradigm and random model. fig. 6. confusion matrix. full size image as we can see in the confusion matrix presented in fig. 6, our model obtained an accuracy of 0.6328. if we were to consider robots drawing lots to predict the paradigms, the probability of getting it right would be \(20\%\), while our model gets a \(63.28\%\) hit odds rate. analyzing the classification of each paradigm, we can see that the random model and the team obtained a considerable rate of false positives between them. this could be due to the team paradigm spreading the four subteams across the four quadrants of the scenario. for high team communication range values (20 or 25), where targets can move further between their sub-team members, the team structure begins to resemble the random model. some images of the organizational paradigm of the hierarchy were classified as the coalition. as in the coalition, the number of members in each coalition can be unbalanced, to the point where there are 22 targets in one coalition and only two in another. thus, in these cases, the hierarchy can resemble the coalition and vice versa, especially if the communication range is high, as the targets spread further across the environment. holarchy was also classified as a coalition due to the fact that they have a common characteristic; there are two target groups in the scenario. however, while one has the same amount of members, the other may contain unbalanced values. therefore, for scenarios where there were the same amount of members or approximate, the model could misclassify these two paradigms. figure 7 presents the test step rating report, which contains the accuracy and recall for each organizational and random model. the model had the best accuracy for holarchy; that is, when it predicts that targets are arranged as a holarchy, it is correct at \(93\%\) of the time. in the case of recall, the results were close, with the exception of the coalition. however, all were above \(50\%\). that is, our model correctly identifies around \(60\%\) most of the target-structured models. fig. 7. test step classification report. full size image 4.3 results of achieving goals for each team one hundred times, the models with the robots were run using or not the classification approaches for selection of the containment strategy. after that, the average of their values was calculated. tables 1, 2, 3, 4, and 5 present the results of the percentage of environment explored by targets (peet) and the average number of observed targets (anot) with robots using or not the classifications with the containment strategies for each range of target communication. table 1. result with communication range equal to 5.full size table table 2. result with communication range equal to 10.full size table table 3. result with communication range equal to 15.full size table table 4. result with communication range equal to 20.full size table table 5. result with communication range equal to 25.full size table note that in scenarios where robots used these strategies, targets had the lowest peet for any communication range when compared to scenarios where robots did not use them. this means that robots that used classification approaches with containment strategies were more efficient in preventing the progress of exploration of the environment by the targets. in hierarchy and holarchy, it was observed in the behavior of the targets that when agents with authority over them were disqualified, the targets remained around the last location passed by their coordinators. in the team and coalition, the behavior of the agents who departed from communication with their team or coalition was also to walk around the last location where they had communication with their sub-team or coalition. in some cases, it was possible for this isolated agent to meet again with his group, while the rest continued with the objective of exploring the environment. the results for the anot for each strategy are not significantly different. but, the hierarchy and holarchy were observed because the targets are only circling in the last location passed by their coordinators or sub-coordinators when these are disabled by robots, which facilitates the observation of the robot team. however, in the case of team and coalition, while some robots seek to break communication, others seek to locate other teams and coalitions across the scene. thus, the number of robots for observation of targets is lower than the scenario where robots did not use classification approaches. a containment strategy for the random structure was not proposed, as the movement of targets in this structure is unpredictable. so the robots remained in the approach proposed by [14]. however, it was evaluated along with the paradigms in order to compare the achievement of the goals with the targets being organized or not. regarding the avoidance of observers, the random strategy was the best since the average number of observed targets (anot) was lower for this strategy; that is, the targets were a little better at avoiding the robots since in this strategy, the targets were more spread out, while the organizational targets are closer to communicate. for the purpose of exploring the environment, organizational strategies were better, as they reached higher percentages of exploration, except for hierarchy and holarchy that obtained better results from a communication reach equal to or greater than 15 and 10, respectively. because the targets in these strategies are organized for this purpose, as they focus on regions that have not been explored so far, while the random one can repeat areas already explored more than once. in addition, team and coalition obtained the best results for exploration, as they do not have an organizational structure as restricted as hierarchy and holarchy. thus, they have more freedom to spread out across the environment in order to reach their goal. the random model is not affected by the change in communication range, as this strategy does not use communication; the targets just walk randomly. 5 conclusion the contribution given by this research is to show how the use of computer vision enables the classification of organizational structures in multi-agent simulations, which until now had not been proposed. in addition to introducing new strategies and objectives to targets and robots in the cmommt problem. our classification approach showed satisfactory results when compared to drawing lots since the model obtained an accuracy of \(63.28\%\). our containment approaches also showed promising results for the robots team in terms of exploration of the environment, allowing them to minimize the percentage of the territory explored by the targets and thus avoid further damage to the scenarios that the targets could cause. in the anot issue, hierarchy and holarchy had the best results, while the team and coalition containment strategies performed less than the performance obtained by robots that did not use the containment strategies but used the method proposed by [14]. furthermore, in the case of targets, our distance-to-target-based organizational approach for communication was adequate for the targets’ objective of exploring the environment compared to the random strategy. however, it is not suitable for the targets’ other goal, avoiding robots. for future work, we intend to implement other simulation scenarios, for example, with obstacles, in order to evaluate our approach, even in the real world. in addition to implementing more robust frameworks to analyze the performance of our model in complex environments. we intend to evaluate the approach with other cnn models to select the best model to be used in terms of accuracy and processing time. finally, we want to examine the impact of the wrong choice of containment approaches, in addition to implementing the containment strategies for each new organizational paradigm implemented and improving those already implemented. notes 1.the dataset is available at the following link: https://drive.google.com/drive/folders/1pwjdrzp23st4qzszf_wnyquehot9qcdq?usp=sharing. references andrade, j.p., oliveira, r., da silva, t.f., maia, j.e.b., de campos, g.a.: organization/fuzzy approach to the cto problem. in: 2018 7th brazilian conference on intelligent systems (bracis), pp. 444–449. ieee (2018) google scholar  araújo, m.s., et al.: cooperative observation of smart target agents. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12320, pp. 77–92. springer, cham (2020). https://doi.org/10.1007/978-3-030-61380-8_6 chapter  google scholar  arroyo, r., yebes, j.j., bergasa, l.m., daza, i.g., almazán, j.: expert video-surveillance system for real-time detection of suspicious behaviors in shopping malls. expert syst. appl. 42(21), 7991–8005 (2015) article  google scholar  bennett, m.k., younes, n., joyce, k.: automating drone image processing to map coral reef substrates using google earth engine. drones 4(3), 50 (2020) article  google scholar  cena, c.g., cardenas, p.f., pazmino, r.s., puglisi, l., santonja, r.a.: a cooperative multi-agent robotics system: design and modelling. expert syst. appl. 40(12), 4737–4748 (2013) article  google scholar  cho, c., kim, j., kim, j., lee, s.-j., kim, k.j.: detecting for high speed flying object using image processing on target place. clust. comput. 19(1), 285–292 (2016). https://doi.org/10.1007/s10586-015-0525-x article  google scholar  chollet, f., et al.: keras (2015). https://github.com/fchollet/keras ding, y., he, y.: flexible formation of the multi-robot system and its application on cmommt problem. in: 2010 2nd international asia conference on informatics in control, automation and robotics (car 2010), vol. 1, pp. 377–382. ieee (2010) google scholar  ding, y., zhu, m., he, y., jiang, j.: p-cmommt algorithm for the cooperative multi-robot observation of multiple moving targets. in: 2006 6th world congress on intelligent control and automation, vol. 2, pp. 9267–9271. ieee (2006) google scholar  horling, b., lesser, v.: a survey of multi-agent organizational paradigms. knowl. eng. rev. 19(4), 281–316 (2004) article  google scholar  khan, a., rinner, b., cavallaro, a.: cooperative robots to observe moving targets. ieee trans. cybern. 48(1), 187–198 (2016) article  google scholar  kluyver, t., et al.: jupyter notebooks a publishing format for reproducible computational workflows. in: loizides, f., schmidt, b. (eds.) positioning and power in academic publishing: players, agents and agendas, pp. 87–90. ios press (2016) google scholar  luke, s., sullivan, k., panait, l., balan, g.: tunably decentralized algorithms for cooperative target observation. in: proceedings of the fourth international joint conference on autonomous agents and multiagent systems, pp. 911–917 (2005) google scholar  parker, l.e.: cooperative motion control for multi-target observation. in: proceedings of the 1997 ieee/rsj international conference on intelligent robot and systems. innovative robotics for real-world applications. iros 1997, vol. 3, pp. 1591–1597. ieee (1997) google scholar  payá, l., gil, a., reinoso, o., ballesta, m., \(\tilde{n}\)eco, r.: behaviour-based multi-robot formations using computer vision. in: proceedings of the 6th iasted international conference on visualization, imaging and image processing (2006) google scholar  robin, c., lacroix, s.: multi-robot target detection and tracking: taxonomy and survey. auton. robot. 40(4), 729–760 (2015). https://doi.org/10.1007/s10514-015-9491-7 article  google scholar  sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv 2: inverted residuals and linear bottlenecks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4510–4520 (2018) google scholar  frança da silva, t., et al.: smart targets to avoid observation in cto problem. in: proceedings of the 18th international conference on autonomous agents and multiagent systems, pp. 1958–1960 (2019) google scholar  silva, t., araújo, m., junior, r.f., costa, l., andrade, j., campos, g.: classifying organizational structures on targets in the cooperative target observation. in: anais do xvii encontro nacional de inteligência artificial e computacional, pp. 718–729. sbc (2020) google scholar  tisue, s., wilensky, u.: netlogo: a simple environment for modeling complexity. in: international conference on complex systems, vol. 21, pp. 16–21. boston, ma (2004) google scholar  vargas-cuentas, n.i., roman-gonzalez, a., mantari, a.a., muñoz, l.a.: chagas disease study using satellite image processing: a bolivian case. acta astronaut. 144, 216–224 (2018) article  google scholar  download references author information authors and affiliations state university of ceara, fortaleza, brazil thayanne frança da silva, matheus santos araújo, raimundo juracy campos ferro junior & gustavo augusto lima de campos federal university of ceara, fortaleza, brazil leonardo ferreira da costa & joão pedro bernardino andrade authors thayanne frança da silvaview author publications search author on:pubmed google scholar matheus santos araújoview author publications search author on:pubmed google scholar raimundo juracy campos ferro juniorview author publications search author on:pubmed google scholar leonardo ferreira da costaview author publications search author on:pubmed google scholar joão pedro bernardino andradeview author publications search author on:pubmed google scholar gustavo augusto lima de camposview author publications search author on:pubmed google scholar corresponding author correspondence to thayanne frança da silva . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper da silva, t.f., araújo, m.s., ferro junior, r.j.c., da costa, l.f., andrade, j.p.b., de campos, g.a.l. (2021). intelligent agents for observation and containment of malicious targets organizations. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_4 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-agent system cooperative multi-robot observation of multiple moving targets organization computational vision containment strategies publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature merging traditional feature extraction and deep learning for enhanced hop variety classification: a comparative study using the ufop-hvd dataset | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper merging traditional feature extraction and deep learning for enhanced hop variety classification: a comparative study using the ufop-hvd dataset conference paper first online: 12 october 2023 pp 307–322 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) merging traditional feature extraction and deep learning for enhanced hop variety classification: a comparative study using the ufop-hvd dataset download book pdf download book epub pedro castro  orcid: orcid.org/0000-0001-5006-65089, gabriel fortuna  orcid: orcid.org/0000-0002-4209-936610, pedro silva  orcid: orcid.org/0000-0002-5525-61219, andrea g. c. bianchi  orcid: orcid.org/0000-0001-7949-11889, gladston moreira  orcid: orcid.org/0000-0001-7747-59269 & … eduardo luz  orcid: orcid.org/0000-0001-5249-15599  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 510 accesses abstract accurately identifying plant species and varieties is crucial across various disciplines, such as biology, medicine, and agronomy. while species identification is challenging, variety identification presents an even greater difficulty. conventional identification methods, although effective, often require specialized and costly equipment, making them less accessible. in this work, we explore the problem of hop variety classification, comparing traditional feature extraction methods with deep learning approaches using the ufop-hvd dataset. we address two research questions: whether traditional techniques can achieve competitive results given the limited number of images and whether combining traditional techniques and deep learning can improve the current state-of-the-art. our findings indicate that traditional techniques yield competitive results for hop variety identification, offering advantages such as interpretability, reduced computational costs, and potential integration into mobile devices. moreover, we introduce an ensemble method that improves the accuracy from 77.16% to 81.90%, establishing a new state-of-the-art for the ufop-hvd dataset. these results demonstrate the potential of merging traditional methods with deep learning for challenging hop variety classification tasks, providing an initial baseline for future research. supported by universidade federal de ouro preto, (fapemig, grants apq-01518-21, apq-01647-22), capes and (cnpq, grants 307151/2022-0, 308400/2022-4). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others differential evolution fpa-svm for target classification in foliage environment using device-free sensing chapter © 2020 deep learning: as the new frontier in high-throughput plant phenotyping article 18 march 2022 a deep learning-based approach for the identification of selected species of genus euphorbia l. article 19 may 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. agronomy categorization horticulture machine learning plant science statistical learning 1 introduction accurately identifying plant species and varieties holds significant value across numerous disciplines. within biology, it serves as both a subject of investigation and an instrumental tool for teaching botany and promoting biodiversity conservation among high school and undergraduate students [20]. in medicine, plant identification enhances the quality and safety of herbal remedies, consequently reducing poisoning and intoxication incidents [8]. forensic experts utilize this knowledge to bolster criminal investigations and support drug enforcement efforts [14]. in agronomy, the precise identification of plant varieties can optimize planting efficiency [5], as well as aid in detecting weeds detrimental to crop development. numerous methods exist for the identification of plant species and varieties, such as gas chromatography [1], mass spectrometry [18], electrochemical fingerprinting [12], molecular analysis [21], among other techniques. although these methods effectively assist in recognizing plant varieties, they necessitate specialized and costly equipment, often utilized within laboratory settings. the application of machine learning for plant recognition through image analysis has garnered interest due to its user-friendliness and accessibility [4]. in the machine learning approach, the only required equipment is a device capable of capturing plant images (or their components), such as a smartphone or a standard camera, which individuals can utilize without prior experience in plant classification. although some researchers have demonstrated the feasibility of classifying plant species using computer vision with remarkable accuracy, a more challenging subclass of the problem persists: distinguishing varieties within the same species. plants of the same species often exhibit strikingly similar appearances across leaves, stems, and flowers [19]. nevertheless, in certain applications, it is crucial to classify plant varieties, as exemplified by hops used in beer production. hops impart flavor, aroma, and bitterness; depending on the variety, the beverage’s characteristics are significantly affected. chemical processes differentiate hop varieties in the industry, but these methods are expensive and time-consuming. in this context, researchers have made available a dataset, the ufop-hvd [6]footnote 1, containing images of 12 hop varieties. in [7], deep learning techniques, specifically convolutional networks, were explored for hop variety classification. however, the dataset is limited, containing only 1,592 images (4,159 hop leaves). as a result, two research questions emerge: rq1, given the limited number of images in the ufop-hvd dataset, which may not favor deep learning techniques, is it possible to achieve better or more competitive results using traditional feature extraction techniques? and rq2, can the combination of traditional techniques and deep learning improve the current state-of-the-art for the ufop-hvd dataset? the experimental findings of this study indicate that traditional techniques yield competitive results and offer several advantages, such as easier interpretability, reduced computational costs, and the potential for seamless integration into mobile devices, compared to deep learning techniques. nevertheless, merging traditional methods with deep learning has shown potential to yield significant advancements in classification. this was particularly evident in the tested hop varieties dataset, where an improvement in accuracy was observed from 77.16% to 81.90%, as demonstrated by our experimental findings. given that, the dataset under investigation is relatively new, this work also establishes an initial baseline for the problem, encompassing traditional and deep learning approaches. 2 related works and dataset humulus lupulus l., commonly referred to as hops, is a climbing plant whose flowers are a principal ingredient in beer production worldwide. hops impart flavor, bitterness, and aroma to the beverage [3], and play an important role in stabilization. over 250 cataloged varieties of this plant exist [17], with distinguishing features including alpha-acids, beta-acids, and essential oils [25]. these components endow each beer with unique characteristics, potentially influencing its classification. numerous methods exist for identifying hop varieties based on their acids and essential oils [11, 13, 24]. however, these techniques may be inaccessible or unavailable to farmers. machine learning and computer vision-based methods offer promising alternatives, given the extensive literature on plant species recognition [4]. for hops, the focus is on classifying the variety, which is a taxonomic level below species. as [19] states, intraspecific varieties share common genotype or phenotype traits. in the case of hops, visually observable morphological features are quite similar, as demonstrated in fig. 1. to address this problem with computer vision, brazilian researchers have provided an image dataset to facilitate the development of machine-learning techniques for the classification of hop varieties. in [7], an end-to-end method for hop variety classification using the ufop-hvd was proposed. the researchers investigated several cnn architectures: resnets, efficientnets, and inceptionnets. they explored image classification with and without leaf segmentation, with and without data augmentation, and proposed an ensemble architecture called multi-cropped-full, which combined six models. for the leaf-only classification problem, termed “cropped classification” by the authors, a resnet50 architecture with a 50% dropout achieved the best results, attaining 78% accuracy. the ensemble model, multi-cropped-full, utilized multiple leaves from the same image and the entire image as input, achieving 81% accuracy. with the application of data augmentation techniques, the multi-cropped-full model reached 95% accuracy. in the present study, we explore traditional machine learning techniques using the dataset in the “cropped” configuration, wherein each detected leaf image (an image that can have multiple hop leaves) is transformed into a new input for the problem. in this cropped evaluation format, only the main leaf should be considered. given the limited number of samples, it is hypothesized that traditional machine-learning techniques should yield results comparable to those reported in [7]. 2.1 ufop hop varieties dataset the ufop hop varieties dataset (ufop-hvd) [6] comprises 1,592 images of young and adult hop leaves across 12 varieties. the images were captured without controlling lighting, focus, distance, or angle and have resolutions ranging from 1,040\(\,\times \,\)520 to 4,096\(\,\times \,\)3,072. the dataset allocates 70% of the images for training, 15% for validation, and 15% for testing, establishing a protocol to be adhered to by authors. representative samples from the dataset can be viewed in fig. 1. fig. 1. examples of the 12 hop varieties used in this work: (a) cascade; (b) centennial; (c) cluster; (d) comet; (e) hallertau mittelfrueh; (f) nugget; (g) saaz; (h) sorachi ace; (i) tahoma; (j) triple pearl; (k) triumph; (l) zeus. full size image each image in the dataset is accompanied by an xml file that specifies the plant variety and includes one or more leaves labeled using bounding boxes. these bounding boxes are defined by x and y coordinates for the upper-left and lower-right corners. leaves that are not sharp or are significantly occluded have been excluded by the dataset author to maintain data quality. table 1 presents the number of images per class for each set. the photographs were captured using motorola moto g7, samsung galaxy a11, and apple iphone 11 devices. table 1. class/variety division of training, validation, and test set size.full size table 3 methodology in this study, hop classification is not conducted using the entire image. instead, only the main leaf, which possesses the largest area among all leaves, is utilized. as illustrated in fig. 2, the main leaf is marked with a red bounding box, while the remaining leaves are outlined in yellow. the initial step involves evaluating the area of all leaves in the image and extracting the largest one. subsequently, the cropped image is resized such that its longest side measures l pixels. this parameter is further investigated in sect. 4. this standardization process is essential to account for varying image resolutions. lastly, the image is converted to grayscale in preparation for feature extraction algorithms. fig. 2. pre-processing containing the steps of cropping the main leaf, resizing, and converting to grayscale. full size image 3.1 proposed method figure 3 showcases the method, unifying all steps. the original image is pre-processed through cropping, resizing, and grayscale conversion. next, glcm, lbp, daisy, and kase feature extractors are applied. daisy and kase descriptors are normalized between 0 and 1, and fed to the bag of visual words to create feature vectors. these vectors are normalized and concatenated into one, then sent to the svm classifier. fig. 3. full proposed method with pre-processing, feature extraction, and classification steps. full size image to assess the impact of using traditional techniques on convolutional networks, a simple ensemble [10] method for combining the outputs of both is evaluated. in this study, the ensemble sums the predictions of the models to generate a final prediction, as demonstrated in fig. 4. fig. 4. ensemble applied to the outputs of the proposed method and cropped base convolutional network. in this example, the proposed method indicated cluster as the most likely class, while cropped base favored centennial. in the summation of the predictions, the cluster class prevailed. full size image 3.2 traditional machine learning techniques feature extraction is vital for image classification in machine learning, as it helps mitigate errors caused by angle, lighting, and scale variations. this study investigates five robust feature extraction techniques resilient to these variations and employs a support vector machine (svm) with radial basis function (rbf) for the classification step. gray level co-occurrence matrix (glcm). [15] proposes a texture feature extraction method based on second-order statistics. a p \(\times \) p matrix p is generated, where p represents the number of possible image intensities. pairs of intensities are counted based on location parameters, such as distance and angle. then, the co-occurrence matrix is normalized so that the sum of all its elements is 1. this matrix is usually sparse, and there are a few ways to better adapt it for classification. haralick et al. [15] suggested 28 attributes generated from the matrix p, and those applied in this work are contrast, dissimilarity, homogeneity, angular second moment, energy, and correlation. the scalars obtained as a result form a vector of attributes. local binary pattern. the local binary pattern (lbp) is a texture analysis feature extractor [22]. for each image pixel, it generates a value representing the pixel and its surroundings by determining the distance and number of neighbors. a binary code is formed by comparing the central pixel with its neighbors, and then converted to a decimal value. the original image pixels are replaced by these numerals, forming a new matrix that can be used directly as a feature or converted to a histogram as an attribute vector. parameters, such as increasing the radius to the central pixel, can be adjusted. to reduce possible variations, [23] proposed a uniformity and circularity model. uniform patterns have a maximum of 2-bit inversions, and circularity allows codes with the same shifted sequence to be marked as equivalent, reducing the number of combinations. daisy. [28] introduces a local region descriptor generator initially designed for stereo matching, but applicable for classification. it works with histograms of oriented gradients and can be applied to entire images or fragments around points of interest. the algorithm defines a central pixel c and calculates the gradient histogram in o orientations. r rings are generated around c with radius \(r_i\). for each ring \(r_i\), h points are uniformly distributed from 0 to 360\(^\circ \), and histograms with o directions are calculated. the total number of histograms h is \(r \times h + 1\). the gaussian filter size increases proportionally with \(r_i\). histograms are independently normalized and concatenated, resulting in a feature vector of size \(h \times o\). kase. [2] is a method for detecting regions of interest and generating rotation and scale invariant descriptors. it involves building image gradients at various scales, calculating hessian determinants on each scale, and using a sliding window to find local maxima for identifying points of interest. descriptors are created for rectangular regions centered on each point, using first-order derivatives in horizontal and vertical directions. the 64-attribute descriptor is made rotation invariant by calculating the main orientation in a local neighborhood and rotating the descriptor towards that angle. bag of visual words. direct use of daisy and kase descriptors in classifiers presents challenges due to varying input sizes and local rotation and scale invariance. the bag of visual words (bovw) [9, 29] method overcomes these issues by grouping similar descriptors and generating a histogram representing a visual word vocabulary. this creates a constant-size feature vector suitable for classifiers. clustering algorithms, specifically k-means, are used to group similar descriptors in this work. 3.3 deep learning techniques in this study, the cropped base convolutional networks outlined in [7] were replicated, employing the blocks from inceptionv3 [26], resnet50 [16], and efficientnetb3 [27] architectures. the purpose of reproducing these networks is to facilitate a comparison with traditional feature extraction and classification methods. 4 experiments and results to address the research questions posed in this study, the experiments are divided into two groups. first, we investigate the feature extraction techniques with an svm classifier, followed by tests with the re-implemented convolutional networks from the proposed work in [7], under the same setup, using ufop-hvd data. next, we evaluate the combination (ensemble) of multiple techniques. the evaluation metric employed is accuracy. in addition to experimenting with the models, this section also presents data on the interpretability of the base models. the source code for loading and preparing the data from the database is available at https://anonymous.4open.science/r/anonymous-8291. 4.1 rq1: can better or more competitive results be achieved using traditional feature extraction techniques? table 2 displays the classification accuracy using features generated by glcm. each row corresponds to a different configuration, and the ‘id’ column identifies the configuration for reference. for all configurations, the co-occurrence angles of the pixels were 0\(^\circ \), 45\(^\circ \), 90\(^\circ \), 135\(^\circ \), 180\(^\circ \), 225\(^\circ \), 270\(^\circ \), and 315\(^\circ \). the parameter l, introduced earlier, corresponds to the length of the longest side of the resized image and can assume values of 400, 500, and 600 pixels. these values determine the model instances, which will be named l400, l500, and l600, respectively. the variable parameter was the distance between the co-occurring pixels. it can be observed that l400 and l500 instances benefited from the use of longer distances, while the l600’s performance declined under these conditions. the accuracy contribution from greater distances suggests the need to further increase them to investigate the gain limit. table 2. accuracy of glcm feature extractor for different image sizes.full size table table 3 shows the classification results with lbp attributes. the variable parameter was the radius length, and the number of points was set as 8 times the radius value. the feature extraction method was the uniform pattern. l400 and l600 instances performed better with larger radii, indicating potential exploration of increasing these parameters. l500, which achieved the best accuracy for the attribute, did not exhibit noticeable patterns concerning the radius. lbp demonstrated the best individual performance among all extractors. table 3. accuracy of lbp feature extractor for different image sizes.full size table the daisy feature extractor has the most adjustable parameters and, consequently, more available configurations. the accepted values for the descriptor radius were 9, 18, and 36, which may contain 2 or 3 rings, as shown in table 4. additionally, for each instance, a different step parameter was used: 25, 33, and 40 for l400, l500, and l600, respectively. all settings employed a fixed number of 8 orientations and 16 histograms. besides the daisy descriptors parameterization, the number of k-means clusters, which form the basis for bag of visual words, varied between 25, 50, 100, and 200 clusters. l500 performed better with larger clusters, while l400 and l600 performed optimally with 100. table 4. top 10 accuracy of daisy feature extractor for different image sizes.full size table lastly, table 5 presents the accuracy for attributes obtained with kase. the modified parameters were the number of clusters for the k-means algorithm and the presence or absence of contrast enhancement. when contrast enhancement was applied, a factor of 2 was used. the l500 instance achieved the best performance. in all cases, increased contrast contributed to an improvement in the hit rate. this is possible because the number of regions of interest detected by kase increases, generating more descriptors that can help better discriminate one leaf from another. moreover, all instances performed better with more clusters. it is crucial to investigate whether increasing the contrast factor or the number of clusters may lead to an even greater gain in accuracy. table 5. accuracy of kase feature extractor for different image sizes.full size table combined attributes. to explore different image features, the attributes generated by glcm, lbp, daisy, and kase were combined into a single feature vector. tests were conducted on the validation set using the svm classifier with the parameter \(c = 30\). several combinations were performed with each extractor configuration. however, to reduce the total number of solutions, a heuristic was followed. initially, all pairs composed of glcm and lbp attributes were tested. then, the pair with the best accuracy was chosen for the next step, where it was combined with the remaining features (daisy and kase). the tests were executed again, and this time, the tuple with the highest hit rate was chosen as the base model. this process was performed for the l400, l500, and l600 instances, and the base models returned are named mb400, mb500, and mb600, respectively. table 6 displays the chosen configuration of each attribute for the base models, ensuring maximum performance. the total attributes of the final feature vector are also recorded. table 6. base models generated from concatenating glcm, lbp, daisy, and kase attributes.full size table the best models on the validation were executed on the test set to obtain the final accuracy. table 7 presents these accuracies and compares them with those achieved by the cropped base models from the work of [7]. in [7], the cropped bases were applied to the validation set. therefore, here we rerun them on the test set for comparison purposes. cropped base also uses the main cropped leaf at a \(300\,\times \,300\) resolution. it has variants with the inceptionv3, resnet50, and efficientnetb3 neural networks, subject to a dropout rate of 0.2, 0.5, or 0.8, as seen in parentheses in the table 7 ‘model’ column. mb400 and mb600 models outperformed inceptionv3 with dropout rates of 0.2 and 0.5. mb500 only surpassed inceptionv3 with a dropout rate of 0.2. however, it is worth noting the significant reduction in the proposed models’ size in mb. the base models are at least 10 times smaller than the cnn-base model, making them suitable for use on devices with limited storage and memory capacity. table 7. comparison of the accuracy of the base models with the cropped bases applied to the test set.full size table 4.2 rq2: can the combination of techniques improve the current state-of-the-art? the final test setup focuses on the ensemble of the proposed models and the cropped bases. table 8 displays the accuracies of the ensemble models in the ‘model’ column with mb400, mb500, mb600, and their combinations. it can be observed that the ensembles improved the accuracy of all cropped bases. the integration with mb400 showed greater gains in the convolutional networks with dropout rates of 0.2 and 0.5. the network achieving the highest success rate was efficientnetb3, with a dropout rate of 0.5, increasing from 77.16% to 81.90% accuracy when its predictions were combined with the outputs of the mb400 and mb500 models simultaneously. it is also noted that the union of base models can enhance performance, as is the case with mb400 and mb600, which produced a 68.10% accuracy together. table 8. accuracy of an ensemble of the proposed method and the cropped base models.full size table figure 5 presents the confusion matrix of the ensemble of models mb400, mb500, and efficientnetb3 (dropout rate of 0.5). a significant improvement in many varieties is observed. cascade achieved 100% accuracy, and five other varieties had an accuracy of at least 85%. despite some improvement, nugget had the lowest performance, while cluster remained among the worst. fig. 5. confusion matrix of ensemble of models mb400, mb500 and efficientnetb3 (dropout of 0.5). full size image 4.3 interpretability this study used variable permutation to evaluate the importance of glcm, lbp, kase, and daisy features for the mb400, mb500, and mb600 models. lbp was the most significant feature, contributing over 50% to the final prediction (see fig. 6). glcm also played a substantial role, while mb500 relied more on daisy and mb600 on kase. mb400 had minimal influence from kase and daisy. analyzing the feature importance by class revealed potential bottlenecks and opportunities for improving the models, such as removing or replacing kase attributes for mb400, increasing lbp configurations for mb500, and adjusting glcm parameters for mb600. fig. 6. importance of glcm, lbp, kase, and daisy features for each base model: (a) mb400; (b) mb500; (c) mb600. full size image 5 conclusion in this study, we examine the problem of hop variety classification, comparing traditional feature extraction methods with deep learning approaches. we also introduce an ensemble method, achieving a new state-of-the-art for the ufop-hvd dataset (improving the accuracy from 77.16% to 81.90%). our findings indicate that it is possible to develop competitive methods for this problem using traditional techniques and handcrafted features. the three proposed methods have a small footprint, making them suitable for devices with limited memory capacity, such as iot devices for agriculture or even smartphones. this characteristic could facilitate their deployment in the field. it is worth noting that the images in the dataset were captured using three different sensors (3 cell phones) in an uncontrolled environment. moreover, the data comprises numerous classes, and the leaf images exhibit highly similar morphological features, which renders classification more challenging. despite these factors, the results presented here demonstrate robustness. among the limitations of this study, one is the manual annotation of the bounding boxes. it would be worthwhile to explore techniques for automatic object detection. in addition, potential areas for model improvement were presented through attribute permutation and the calculation of the importance of each attribute. consequently, it is expected that future work will yield even better results by fine-tuning each feature extraction technique. notes 1.available at https://doi.org/10.6084/m9.figshare.14933178. references afifi, s.m., el-mahis, a., heiss, a.g., farag, m.a.: gas chromatography-mass spectrometry-based classification of 12 fennel (foeniculum vulgare miller) varieties based on their aroma profiles and estragole levels as analyzed using chemometric tools. acs omega 6(8), 5775–5785 (2021) article  google scholar  alcantarilla, p.f., bartoli, a., davison, a.j.: kaze features. in: fitzgibbon, a., lazebnik, s., perona, p., sato, y., schmid, c. (eds.) eccv 2012. lncs, vol. 7577, pp. 214–227. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-33783-3_16 chapter  google scholar  astray, g., gullón, p., gullón, b., munekata, p.e., lorenzo, j.m.: humulus lupulus l. as a natural source of functional biomolecules. appl. sci. 10(15), 5074 (2020) google scholar  azlah, m.a.f., chua, l.s., rahmad, f.r., abdullah, f.i., wan alwi, s.r.: review on techniques for plant leaf classification and recognition. computers 8(4), 77 (2019) google scholar  bhattarai, u., karkee, m.: a weakly-supervised approach for flower/fruit counting in apple orchards. comput. ind. 138, 103635 (2022) article  google scholar  castro, p., luz, e., moreira, g.: dataset for hop varieties classification. data brief 38, 107312 (2021) article  google scholar  castro, p.h.n., moreira, g.j.p., da silva luz, e.j.: an end-to-end deep learning system for hop classification. ieee latin america trans. 20(3), 430–442 (2021) google scholar  chen, s., et al.: a renaissance in herbal medicine identification: from morphology to dna. biotechnol. adv. 32(7), 1237–1244 (2014) article  google scholar  csurka, g., dance, c., fan, l., willamowski, j., bray, c.: visual categorization with bags of keypoints. in: workshop on statistical learning in computer vision, eccv, pp. 1–2 (2004) google scholar  dietterich, t.g.: ensemble methods in machine learning. in: kittler, j., roli, f. (eds.) mcs 2000. lncs, vol. 1857, pp. 1–15. springer, heidelberg (2000). https://doi.org/10.1007/3-540-45014-9_1 chapter  google scholar  duarte, l.m., adriano, l.h.c., de oliveira, m.a.l.: capillary electrophoresis in association with chemometrics approach for bitterness hop (humulus lupulus l.) classification. electrophoresis 39(11), 1399–1409 (2018) google scholar  fan, b., et al.: development of an electrochemical technology for ten clematis spp varieties identification. int. j. electrochem. sci. 15, 10212–10220 (2020) article  google scholar  farag, m.a., mahrous, e.a., lübken, t., porzel, a., wessjohann, l.: classification of commercial cultivars of humulus lupulus l. (hop) by chemometric pixel analysis of two dimensional nuclear magnetic resonance spectra. metabolomics 10(1), 21–32 (2014) google scholar  ferri, g., alù, m., corradini, b., beduschi, g.: forensic botany: species identification of botanical trace evidence using a multigene barcoding approach. int. j. legal med. 123(5), 395–401 (2009) article  google scholar  haralick, r.m., shanmugam, k., dinstein, i.h.: textural features for image classification. ieee trans. syst. man cybern. smc 3(6), 610–621 (1973) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  healey, j.: the hops list: 265 beer hop varieties from around the world (2016) google scholar  jakab, a., héberger, k., forgács, e.: comparative analysis of different plant oils by high-performance liquid chromatography-atmospheric pressure chemical ionization mass spectrometry. j. chromatogr. a 976(1–2), 255–263 (2002) article  google scholar  jenks, m.a.: plant nomenclature. purdue university-department of horticulture and landscape architecture, disponível (2011) google scholar  killermann, w.: research into biology teaching methods. j. biol. educ. 33(1), 4–9 (1998) article  google scholar  lotti, c., ricciardi, l., rainaldi, g., ruta, c., tarraf, w., de mastro, g.: morphological, biochemical, and molecular analysis of l. open agric. j. 13(1) (2019) google scholar  ojala, t., pietikäinen, m., harwood, d.: a comparative study of texture measures with classification based on featured distributions. pattern recogn. 29(1), 51–59 (1996) article  google scholar  ojala, t., pietikainen, m., maenpaa, t.: multiresolution gray-scale and rotation invariant texture classification with local binary patterns. ieee trans. pattern anal. mach. intell. 24(7), 971–987 (2002) article  math  google scholar  shellie, r.a., poynter, s.d., li, j., gathercole, j.l., whittock, s.p., koutoulis, a.: varietal characterization of hop (humulus lupulus l.) by gc-ms analysis of hop cone extracts. j. sep. sci. 32(21), 3720–3725 (2009) google scholar  steenackers, b., de cooman, l., de vos, d.: chemical transformations of characteristic hop secondary metabolites in relation to beer properties and the brewing process: a review. food chem. 172, 742–756 (2015) article  google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: international conference on machine learning, pp. 6105–6114. pmlr (2019) google scholar  tola, e., lepetit, v., fua, p.: daisy: an efficient dense descriptor applied to wide-baseline stereo. ieee trans. pattern anal. mach. intell. 32(5), 815–830 (2009) article  google scholar  yang, j., jiang, y.g., hauptmann, a.g., ngo, c.w.: evaluating bag-of-visual-words representations in scene classification. in: proceedings of the international workshop on workshop on multimedia information retrieval, pp. 197–206 (2007) google scholar  download references author information authors and affiliations computing department, federal university of ouro preto (ufop), ouro preto, mg, 35402-163, brazil pedro castro, pedro silva, andrea g. c. bianchi, gladston moreira & eduardo luz brazuca lúpulos, petrópolis, rj, 25610-080, brazil gabriel fortuna authors pedro castroview author publications search author on:pubmed google scholar gabriel fortunaview author publications search author on:pubmed google scholar pedro silvaview author publications search author on:pubmed google scholar andrea g. c. bianchiview author publications search author on:pubmed google scholar gladston moreiraview author publications search author on:pubmed google scholar eduardo luzview author publications search author on:pubmed google scholar corresponding author correspondence to eduardo luz . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper castro, p., fortuna, g., silva, p., bianchi, a.g.c., moreira, g., luz, e. (2023). merging traditional feature extraction and deep learning for enhanced hop variety classification: a comparative study using the ufop-hvd dataset. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_21 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hop variety plant recognition deep learning ensemble cnn publish with us policies and ethics profiles gladston moreira view author profile eduardo luz view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an enhanced tsp-based approach for active debris removal mission planning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an enhanced tsp-based approach for active debris removal mission planning conference paper first online: 28 november 2021 pp 140–154 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) an enhanced tsp-based approach for active debris removal mission planning download book pdf download book epub joão batista rodrigues neto10 & gabriel de oliveira ramos10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 786 accesses 4 citations abstract the extensive exploration of the low earth orbit (leo) has created a dangerous spacial environment, where space debris has threatened the feasibility of future operations. in this sense, active debris removal (adr) missions are required to clean up the space, deorbiting the debris with a spacecraft. adr mission planning has been investigated in the literature by means of metaheuristic approaches, focused on maximizing the amount of removed debris given the constraints of the spacecraft. the state-of-the-art approach uses an inver-over and maximal open walk algorithms to solve this problem. however, that approach fails to deal with large instances and duration constraints. this work extends the state of the art, increasing its performance and modeling all the constraints. experimental results evidence the improvements over the original approach, including the ability to run for scenarios with thousands of debris. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others survey on research and development of on-orbit active debris removal methods article 24 september 2020 spacecraft motion planning based on the twin delayed deep deterministic policy gradient algorithm chapter © 2025 french law approach around the topic “legal implications/aspects of active debris removal (adr”) chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. continuous optimization disposal technology and management optimization operations research and decision theory separation science theory and algorithms for application domains 1 introduction in the1960 s, space exploration began to boost the development of new technologies feasible through the use of satellites distributed in orbits. among these orbits, low earth orbit (leo) was widely used for satellite networks with a large number of objects per service in orbit [20]. over time, these satellites became depreciated, lose communication or got out of control, thus becoming space debris. a high population of debris represents a hazard to the operating structures in orbit, since they are objects out of control and at high speed [12]. according to some predictive models, a sufficiently large population of debris will increase the probability of collisions and, therefore, increase the debris population again, thus making this population increase recursively for many years. this phenomenon is known as kessler syndrome and may cause the collapse of the leo, rendering it useless for years [12]. in fact, the literature already pointed out that the debris population in leo has already reached a critical point [16], and now measures to mitigate this situation are needed. while there are documented techniques that would stabilize the debris population, for now, the only approach capable of reducing it consists in active debris removal (adr) missions [16]. these missions aim to clean up orbit space by forcing the re-entry of certain debris as performed by specific spacecraft. due to the limited resources and time to perform the rendezvous maneuvers and the debris population size, the selection of the debris to be removed has became a hard combinatorial problem [3, 4, 7]. a basic adr should consider the bounds of cost and duration, dependent position values of the moving debris along time [3, 4], and also prioritizing the removal of the most threatener debris first, thus increasing the benefits of such missions [17]. various approaches to this problem have been proposed in the literature [7, 9, 13]. however, most of these works share the same limitations: small instance sizes, unbounded approaches, and non-time-dependent modeling. in this work, we propose an enhanced genetic algorithm to optimize adr mission planning. our approach builds upon the work by [9], improving its performance through a novel combination of genetic operators. the final algorithm resembles the original inver-over genetic operator, with modifications to its reversing strategy. moreover, a new variant of the k-opt algorithm is implemented using a stochastic approach. finally, the open-walk algorithm is improved with one additional constraint. through extensive experiments, our approach yielded better solutions with instances larger than the previous largest ones [9]. 2 literature review in order to solve the adr optimization problem, a few methods have been approached and documented in the last 10 years. exact solutions were used by braun et al. [2] with brute force, and branch and bound variations by [3, 14, 19]. however, in both classes of methods can only be applied to small instances. approximate solutions were implemented using simulated annealing [4, 7], reinforcement learning [25], and genetic algorithms [18, 24]. nonetheless, all of these works were also tested only on small instances. on the other hand, a few approximate methods considered bigger instances. barea et al. [1] used a linear programming method, which has a high complexity as drawback. yang et al. [26] used a greedy heuristic, but requires instance-dependent parameters. ant colony optimization was used in [13, 21, 27], but ignoring mission constraints, strongly simplifying the cost dynamics to reduce the complexity, or even leaving mission duration unbounded. finally, izzo et al. [9] and kanazaki et al. [11] used genetic algorithms, though both did not model all the necessary mission constraints. generally speaking, the majority of the works do not prioritize debris by hazard, consider bigger instances, model the time-dependence, or bound the cost or the mission duration, meaning that most works fail to fully meet the adr mission requirements. building against this background, in this work we introduce the enhanced inver-over operator to deal with large instances, and an enhanced maximal open walk algorithm to model the cost and duration constraints while prioritizing the most threatener debris. 3 problem formulation an adr problem is the combinatorial problem of finding the correct sequence to rendezvous maneuvers towards debris in order to maximize the mission profit given some constraints. in this sense, adr can be seen as a complex variant of the travelling salesman problem (tsp), where one wants to find the minimum weight path in a dynamic complete graph, where the debris are the cities and the dynamic transference trajectories are the edges. the dynamicity is due to the time-dependent cost of the transference, so the correct generalized version of the tsp will be a time-dependent tsp (tdtsp). this work will make use of the integer linear tdtsp problem formulation by [7]. hereafter, we will follow the notation typically used in the literature [7]. let \(v = \{1, ..., n\}\) be the set of \(n\) debris. the distance tensor is represented by \(c = (c_{ijtm})\), where \(c_{ijtm}\) is the cost of transfer from debris \(i\) at epoch \(t\) to debris \(j\) at epoch \(t+m\). also, let \(x = (x_{ijtm})\) be a binary tensor, where \(x_{ijtm} = 1\) indicates that this transference is part of the solution and \(x_{ijtm} = 0\) otherwise. the \(n_t\) possible epochs of departure and arrival are discretized following \(n_t \ge n+1\) and \(m \le n_t-(n-1)\). usually, in order to grant some freedom at the mission planning, \(n_t\) is far larger than \(n\), while \(m\) limits the maximum duration of the transfers. along these lines, the problem of finding the optimized route can be modelled as finding the \(x\) matrix that minimizes the total cost with due respect to the constraints, which can be formulated as follows. $$\begin{aligned} \min _x \sum _{i=1}^{n} \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} c_{ijtm} x_{ijtm} \end{aligned}$$ (1) $$\begin{aligned} \text {s.t.} \sum _{i=1}^{n} \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} x_{ijtm} = n \end{aligned}$$ (2) $$\begin{aligned} \sum _{i=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} x_{ijtm} = 1 \qquad \qquad j = 1, ..., n \end{aligned}$$ (3) $$\begin{aligned} \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} x_{ijtm} = 1 \qquad \qquad i = 1, ..., n \end{aligned}$$ (4) $$\begin{aligned} \sum _{j=1}^{n} \sum _{t=2}^{n_t} \sum _{m=1}^{m} tx_{ijtm} \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} tx_{ijtm} = \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} mx_{ijtm} \qquad \qquad i = 2, ..., n \end{aligned}$$ (5) $$\begin{aligned} \sum _{i=1}^{n} \sum _{j=1}^{n} \sum _{t=1}^{n_t} \sum _{m=1}^{m} mx_{ijtm} \le n_t \end{aligned}$$ (6) in the above formulation eq. (1) represents the objective function and eqs. (2)–(6) represent the problem constraints, to which we will refer simply as constraints (2)–(6) hereafter. constraint (2) guarantees that the solution tensor has all the \(n\) debris. constraints (3) and (4) ensure that there is no loops in the solution (one departure and one arrival transfer for each debris). constraint (5) enforces the transfer duration. finally, constraint (6) limits the total mission duration. as the result, this formulation has a \(\mathcal {o}(2^n)\) search space, using \(n^4\) binary decision variables and \(3n 1\) constraints. 3.1 orbit transfer the trip between one debris and another require impulses (\(\vardelta v\)) of the thrusters to change the orbit of the spaceship. low thrust propulsion systems can perform this maneuvers efficiently. however, they require the optimization of the trajectories to make the mission time available [5]. determining a minimal fuel transfer trajectory between two debris is a complex optimization problem in general case. thus, major works simplify this task by using a generic transfer strategy [4]. the major used transfer strategies are the hohmann and lambert transfers. since this work’s scope does not focus on the orbital transfer optimization problem, the mechanics of the transfers will be briefly described. in [6], hohmann is described as a minimum two-impulse elliptic maneuver to transfer from coplanar orbits. hohmann transfer is a high thrust transfer. since debris are not always in co-planar orbits it is also presented a variation of the hohmann transfer, namely the edelbaum transfer, which is a three-impulse bi-elliptic transfer that allows transferences between non-coplanar orbits. in [5], lambert is described as two-impulse trajectory to transfer from coplanar orbits given a certain transference duration. also, it is possible to make use of the \(j_2\) gravitational earth perturbation to wait for the natural alignment of orbital planes, saving some fuel but increasing the mission duration [5, 7]. finally, the cost of a transference between two debris relies on the mass of the spacecraft, since the thrusters consume propellant mass at each impulse, as the mission goes on the cost of the transfers became lower due to the mass lost in the previous maneuvers. so, it is also possible to optimize the removal sequence taking in account the resultant masses of the objects [3]. 4 proposed approach in this work, the adr problem is approached with an improved heuristic solution, similar to the method used by [9] with the inver-over operator in a genetic algorithm and the maximal open walk algorithm. the inver-over operator optimizes the total cost of the mission with a local search strategy through genetic operations on the individuals. the maximal open walk algorithm constraints the path. this work enhances this solution with a greedier implementation of the algorithms. moreover, to avoid the local optimum, a stochastic 2-opt is proposed to improve the solution, creating new connections. finally, the most rewardable open walk is extracted from the best individual as the final solution. as this is a bi-optimization problem, the complexity was divided in two stages. in the first part, the effectiveness of a solution will be given by the total cost of the removal sequence, calculated with the edelbaum transfer [6] (eqs. (7) and (8)), a consistent and reliable cost approximation for cheap orbital maneuvers. in the second part, the effectiveness of a solution became the removed threat of the leo, calculated with the sum of radar cross section (rcs) area of the removed debris. the rcs area is an abstraction of the size of the debris and is widely used for the threat calculation in the literature, being a measure about how much detectable an object is for a ground radar. $$\begin{aligned} \vardelta {v}&= \sqrt{v^2_0 2v_0v_f \cos {\frac{\pi }{2} \vardelta {i}} + v^2_f} \end{aligned}$$ (7) $$\begin{aligned} \cos {\vardelta {i}}&= \cos {i_1} \cos {i_2} + \sin {i_1} \sin {i_2} (\cos {\varomega _1} \cos {\varomega _2} + \sin {\varomega _1} \sin {\varomega _2}) \end{aligned}$$ (8) $$\begin{aligned} t&= \frac{1}{2} \sqrt{\frac{4\pi ^2 a^3}{\mu }} \end{aligned}$$ (9) also, in order to minimize the complexity of the problem, major works in the literature have assumed a few dynamics simplifications. in this work, the following assumptions were made with the same purpose: the time dependence of the problem is relaxed by the correlation explored in [9], where an optimal solution can remain optimal up to 50 d, depending on the size of the instance. since the edelbaum transfer is an optimized variation of the hohmann transfer [6], the duration of the transference arcs can be computed using kepler’s third law of planetary motion (eq. (9)), which measures the orbital transference duration of an object (spacecraft) between two orbits (debris). the transfer cost also depends on mass of the spacecraft, that will decrease during the mission, where the fuel mass will be consumed. moreover, the gravitational effects of the earth on the spacecraft also influence the transfer cost. in this work, the masses of the objects and the orbital perturbation effects are neglected in the cost transfer dynamics. there are more steps of the rendezvous process to remove a debris, and each step take some time to be performed [15]. in this work the duration of the mission will be given by the sum of the duration of the transferences, the other stages will be neglected. 4.1 inver-over the inver-over operator [22] is a unary genetic operator that resembles characteristics of mutation and crossover at the same time. the evolution of an individual is based on simple population-driven inversions and recombinations of genes. this is a well established operator, know by its good performance with larger instances [22]. in this work a different version on the algorithm is used, mixing the original implementation with the [9] implementation. this work allows array cyclic inversions to happen, inversions that include the section of the last to the first gene. however, the next gene pick depends of the previous order of the selected genes for inversion. this inversion process is analogous to a crossover operator. furthermore, it is stated that to avoid the local optima, a process analogous to the mutation operator has to be used to create new connections that do not exist in population [22]. however, these mutations are not greedy and usually delay the convergence process, so for this implementation it was removed. the implemented algorithm pseudo-code is sketched in algorithm 1. 4.2 stochastic 2-opt the 2-opt optimization [8] is a tsp local search algorithm that adjusts the routing sequence greedily with simple inversions. the main idea is to break the route in two paths and reconnect it invertedly, if it improves the fitness so the inversion is kept in the solution. unfortunately, this is an exact algorithm with a complexity of \(\mathcal {o}(n^2)\), so a lot of solutions evaluations need to be performed in order to improve the solution. nonetheless, there are other methods to improve the 2-opt performance, such as search parallelism and the lin and kernighan technique [10]. the present work made use of a stochastic approach for the algorithm. the stochastic implementation relies on the observations that even with a \(n^2\) search space, the actual number of improvements performed by the algorithm is roughly \(n\) [10]. so, with a random exploration of the space, there could be more chances of finding a profitable move. in order to keep control of the algorithm run time, a termination condition is used to limit the exploration. also, a set of explored moves prevents duplicated evaluations. the implemented algorithm pseudo-code is sketched in algorithm 2. 4.3 maximal open walk the maximal open walk proposed by [9] as “city selection", is a separated algorithm that searches for the contiguous part of a hamiltonian path with the maximal cumulative value limited to some constraint. the path is the optimal solution found, while the value and constraint are respectively, the threat, given by the rcs area, and the total cost. in this work, another constraint is added to this problem, the duration of the open walk, calculated with kepler’s third law. the implemented algorithm pseudo-code is sketched in algorithm 3. 5 experimental evaluation the objective of these experiments is to evaluate the performance of the approach, understand the improvement gain by each technique and find some optimal parameters. to preserve the comparability, all the runs used 20000 fixed iterations as the termination-condition of the inver-over algorithm, a 100 individuals population, the original method runs used 0.05 as \(ri\) (mutation probability), the constrained runs were performed with a cost constraint of 1000 m/s and a time constraint of 1 year. the data about the debris were extracted from the satellite catalog (satcat), a catalogue of all the objects on the earth orbit, maintained by the united states space command (usspacecom). the following instances of the problem were extracted: iridium-33, cosmos-2251 and fengyun-1c, with respectively 331, 1048 and 2653 debris. data was collected at respectively 11-jun-2021 00:06 utc, 13-jun-2021 22:06 utc and 13-jun-2021 22:06 utc. to preserve the comparability of some results, back-propagated instances were generated inputting the actual instances in a sgpd4 orbital propagator [23] that backtracked the debris positions back to 01-jan-2015 at 00:00 utc, the same epoch of the instances used by [9]. unfortunately this process is not very precise, though still feasible. all the debris in the clouds were considered, including the ones that will decay during the mission time. for the sake of clearness, the time (min) values on the experiments are concerned to the computation time taken for the run, and std. dev. is the abbreviation for standard deviation. all experiments were conducted on a public online machine with a 2.30ghz cpu and 12.69 gb of ram. also, for all the experiments, the statistical data results out of 10 independent runs. 5.1 back-propagated instances the experiments performed with the back-propagate debris are intended to provide comparative results to the work of [9] and guide the definition of the parameters. the inver-over algorithm implemented in this work differs from the original implementation on two points, each point will be tested separately to show the advantages and justify its usage in this approach. fig. 1. iridium-33 convergence results full size image fig. 2. cosmos-2251 convergence results full size image the changes made on the inversion implementation in this work aims to improve the convergence of the algorithm, to do so, this work approach is more population driven and less random mutated. to analyse the performance of the changes, the instances iridium-33 and cosmos-2251 were each run twice for 10 times, the first runs with the [9] implementation, and the other ones with this work implementation. the figs. 1 and 2 demonstrate the improvement of the convergence, for a better visualization, just the first 4000 iterations were drawn. it is possible to notice a considerably change on the shape of the curve and the taken computation time in table 1, making this works implementation convergence better. for the record, the majority of runs in each instance got the same final result, indicating that, for small and medium sized instances, the fast convergence does not deteriorate the result. to deviate from the local optima, the stochastic 2-opt (s2opt) is used in this work. parametric tests were conducted to analyse its performance when it matters to time and achieved result. iridium-33 and cosmos-2251 were submitted to 9 different runs, running 10 times each, with a different combination of two parameters: how often does the s2opt runs and with how many iterations at each time. all individuals of the population were processed at each s2opt iteration. table 1. inversion resultsfull size table since the search area of the s2opt is large, a range of possible attempts should be chosen, being neither too small, so no improvement move is found, or too big, so almost the whole search space is tested, turning it into a exact solution. in this work, the chosen range is from 10,000 to 1,000,000 attempts, while the parameters are equally spaced discrete values where its configuration do not underflow or overflow the range. the parameters per run are the following: run 1: at each 100 main iterations, run s2opt with 100 iterations. run 2: at each 100 main iterations, run s2opt with 500 iterations. run 3: at each 100 main iterations, run s2opt with 1000 iterations. run 4: at each 500 main iterations, run s2opt with 100 iterations. run 5: at each 500 main iterations, run s2opt with 500 iterations. run 6: at each 500 main iterations, run s2opt with 1000 iterations. run 7: at each 1000 main iterations, run s2opt with 100 iterations. run 8: at each 1000 main iterations, run s2opt with 500 iterations. run 9: at each 1000 main iterations, run s2opt with 1000 iterations. fig. 3. iridium-33 stochastic 2-opt results full size image fig. 4. cosmos-2251 stochastic 2-opt results full size image analysing the results in figs. 3 and 4 it is possible to state that due to the small size of the iridium-33 debris, all the runs achieved the optimal solution. also, the number of s2opt iterations is directly proportional to the computation time. and finally, the runs 4, 7 an 8 have the better performances ratios, with low values of cost and time, among these, run 7 is the best one. also, to preserve the idea of a competitive evolution, the following experiments use s2opt with an elitist strategy. this time, instead of running the s2opt for the whole population at each s2opt iteration, just the better individuals will be improved. parametric tests were performed with 5 different sizes of elites, running 10 times each. to preserve the elitist characteristic, the elite group should not be greater than 30% of the population. in these experiments, iridium-33 will be discarded, since its small size does not help to fully analyse the performance of the algorithm. here, the tests are set with the same parameters of the previous run 7 (best run), at each 1000 main iterations, run s2opt with 100 iterations. table 2. elitist resultsfull size table analysing the results in table 2 it is possible to state that the elitist improvement of the best 5 individuals at each iteration is the wise strategy to follow, having a lower cost with a little bit more computational time taken. to emphasises the importance of each technique, ablation experiments were conducted for this solution. it is important to state that the elitism is applied on the s2opt, so there is no possible scenario using elitism without s2opt. table 3, summarizes the results of each combination. the experiments were performed on instance cosmos-2251, with 10 runs each. the s2opt parameters are 100 runs at each 1000 main iterations, and the elitist parameter is 5. table 3. ablation resultsfull size table with the ablation experiments, it is possible to understand how each technique of our approach affects the final results. it is clear that the conjunction of the techniques improves the found solutions. 5.2 actual instances the experiments performed with the actual instances are intended to provide results for the present status of the debris clouds. the runs were performed using the best parameters found in the previous experiments. also, for a comparative result, the original implementation was used with the actual instances, and its solutions, inputted to the enhanced maximal open walk of this work. for each instance were performed 10 runs, for the sake of clarity, the settings are: enhanced inversion implementation, with s2opt running 100 iterations for the best 5 individuals at each 1000 main iterations, with a cost constraint of 1000 m/s and a time constraint of 1 year. table 4. final resultsfull size table the results on both sections of table 4 are from the same runs, the results at the bottom are given by the maximal open walk applied to the optimized path at the top. being the missions objective: clear the maximum possible area are under the cost and time constraints [17], our approach focused on making an optimized use of the mission resources to outperforms other approaches. cleaning more area, even if the mission duration and cost are bigger, bounded to time and cost constraints, of course. summarizing, in the average of the runs, the enhanced approach decreased the cost by 26.51% and the computation time by 35.56%. also, when constrained, the solutions of the enhanced approach produced paths that performs significantly better than the original approach solutions, cleaning 12.82% more area under the same constraints. this is possible by a better usage of the constrained resources, increasing the cost by 5.28% and the mission duration by 1.06%. finally, it is possible to notice that the most profitable mission is the enhanced iridium-33, cleaning way more area with the same constraints. 6 conclusions the exploration of tsp approaches when dealing with space debris have considerably evolved the tdtsp problem modeling and its available solutions. however, it still lacks from approaches that fulfill the adr mission requirements with a feasible performance and big instance sizes. this work proposed an enhanced method as a strong candidate to future approaches on tsp based adr mission plannings. using the inver-over as a fast convergence algorithm to deal with the greedy search throught fast inversions, the s2opt as a solution diversity creator to deviate the search from local optima, our method found optimized solutions in a feasible time out of large datasets. real world instances were used to evaluate this approach performance and execute parametric tests, the retrieved results were considerably better than the original approach by [9]. however, this approach does not model the time dependence of the problem, meaning that the produced solutions may not be good solutions in a real scenario, since it does not consider the moving dynamics of the debris. also, this approach first optimizes the cost of the solution, and later chooses the most rewardable sub-path, so it is not a fully bi-optimization algorithm. meaning that solutions with a good cost versus reward ratio could be missed. as future work, we would like to implement a time-dependent removal sequence to produce a complete solution for the adr mission planning problem. we also plan to implement lin and kernighan’s algorithm to improve the convergence of local search heuristic. the optimization of the transference cost with the consideration of body masses and \(j_2\) effect on the transfers represents another interesting direction. finally, we would like to improve the approximation of the mission duration with a more suitable equation for the edelbaum (rather than hohmann) transfer duration. references barea, a., urrutxua, h., cadarso, l.: large-scale object selection and trajectory planning for multi-target space debris removal missions. acta astronautica 170, 289–301 (2020). https://doi.org/10.1016/j.actaastro.2020.01.032 article  google scholar  braun, v., et al.: active debris removal of multiple priority targets. adv. space res. 51(9), 1638–1648 (2013) google scholar  cerf, m.: multiple space debris collecting mission-debris selection and trajectory optimization. j. optim. theor. appl. 156, 761–796 (2013). https://doi.org/10.1007/s10957-012-0130-6 article  mathscinet  math  google scholar  cerf, m.: multiple space debris collecting mission: optimal mission planning. j. optim. theor. appl. 167(1), 195–218 (2015). https://doi.org/10.1007/s10957-015-0705-0 article  mathscinet  math  google scholar  koblick, d., et al.: low thrust minimum time orbit transfer nonlinear optimization using impulse discretization via the modified picard-chebyshev method. comput. model. eng. sci. 111(1), 1–27 (2016). https://doi.org/10.3970/cmes.2016.111.001 edelbaum, t.n.: propulsion requirements for controllable satellites. ars j. 31(8), 1079–1089 (1961). https://doi.org/10.2514/8.5723 federici, l., zavoli, a., colasurdo, g.: a time-dependent tsp formulation for the design of an active debris removal mission using simulated annealing (2019) google scholar  flood, m.m.: the traveling-salesman problem. oper. res. 4(1), 61–75 (1956). https://doi.org/10.1287/opre.4.1.61 article  mathscinet  math  google scholar  izzo, d., getzner, i., hennes, d., simões, l.f.: evolving solutions to tsp variants for active space debris removal. in: proceedings of the 2015 annual conference on genetic and evolutionary computation, pp. 1207–1214. gecco ’15, acm (2015) google scholar  johnson, d., mcgeoch, l.a.: the traveling salesman problem: a case study in local optimization (2008) google scholar  kanazaki, m., yamada, y., nakamiya, m.: trajectory optimization of a satellite for multiple active space debris removal based on a method for the traveling serviceman problem. in: 2017 21st asia pacific symposium on intelligent and evolutionary systems (ies), pp. 61–66 (2017). https://doi.org/10.1109/iesys.2017.8233562 kessler, d.j., cour-palais, b.g.: collision frequency of artificial satellites: the creation of a debris belt. j. geophys. res. space phys. 83(a6), 2637–2646 (1978). https://doi.org/10.1029/ja083ia06p02637 article  google scholar  li, h., baoyin, h.: optimization of multiple debris removal missions using an evolving elitist club algorithm. ieee trans. aerospace electron. syst. 56(1), 773–784 (2020). https://doi.org/10.1109/taes.2019.2934373 article  google scholar  li, h.y., baoyin, h.x.: sequence optimization for multiple asteroids rendezvous via cluster analysis and probability-based beam search. sci. chin. technol. sci. 64(1), 122–130 (2020). https://doi.org/10.1007/s11431-020-1560-9 article  google scholar  liou, j.c.: an active debris removal parametric study for leo environment remediation. adv. space res. 47(11), 1865–1876 (2011). https://doi.org/10.1016/j.asr.2011.02.003 article  google scholar  liou, j.c., johnson, n.: instability of the present leo satellite populations. adv. space res. 41(7), 1046–1053 (2008). https://doi.org/10.1016/j.asr.2007.04.081 article  google scholar  liou, j.c., johnson, n., hill, n.: controlling the growth of future leo debris populations with active debris removal. acta astronautica 66(5), 648–653 (2010). https://doi.org/10.1016/j.actaastro.2009.08.005 article  google scholar  liu, y., yang, j., wang, y., pan, q., yuan, j.: multi-objective optimal preliminary planning of multi-debris active removal mission in leo. sci. chin. inform. sci. 60(7), 1–10 (2017). https://doi.org/10.1007/s11432-016-0566-7 article  google scholar  madakat, d., morio, j., vanderpooten, d.: biobjective planning of an active debris removal mission. acta astronautica 84, 182–188 (2013). https://doi.org/10.1016/j.actaastro.2012.10.038 article  google scholar  seumahu, e.s.: exploration of the equatorial leo orbit for communication and other applications, pp. 217–228. springer, netherlands, dordrecht (1996). https://doi.org/10.1007/978-94-011-5692-9_25 stuart, j., howell, k., wilson, r.: application of multi-agent coordination methods to the design of space debris mitigation tours. adv. space res. 57(8), 1680–1697 (2016). https://doi.org/10.1016/j.asr.2015.05.002 article  google scholar  tao, g., michalewicz, z.: inver-over operator for the tsp. in: eiben, a.e., bäck, t., schoenauer, m., schwefel, h.-p. (eds.) ppsn 1998. lncs, vol. 1498, pp. 803–812. springer, heidelberg (1998). https://doi.org/10.1007/bfb0056922 chapter  google scholar  vallado, d., crawford, p.: sgp4 orbit determination, pp. 18–21 (2008). https://doi.org/10.2514/6.2008-6770 wang, d., li, l., chen, l.: an efficient genetic algorithm for active space debris removal planning. in: 2019 ieee congress on evolutionary computation (cec), pp. 514–521 (2019). https://doi.org/10.1109/cec.2019.8790081 yang, j., hou, x., hu, y.h., liu, y., pan, q.: a reinforcement learning scheme for active multi-debris removal mission planning with modified upper confidence bound tree search. ieee access 8, 108461–108473 (2020) article  google scholar  yang, j., hu, y.h., liu, y., pan, q.: a maximal-reward preliminary planning for multi-debris active removal mission in leo with a greedy heuristic method. acta astronautica 149, 123–142 (2018). https://doi.org/10.1016/j.actaastro.2018.05.040 article  google scholar  zhang, t., shen, h., li, h., li, j.: ant colony optimization based design of multiple-target active debris removal mission (2018). https://doi.org/10.2514/6.2018-2412 download references acknowledgements we thank the anonymous reviewers for their constructive feedback. this work was partially supported by fapergs (grant 19/2551-0001277-2). author information authors and affiliations graduate program in applied computing, universidade do vale do rio dos sinos, são leopoldo, rs, brazil joão batista rodrigues neto & gabriel de oliveira ramos authors joão batista rodrigues netoview author publications search author on:pubmed google scholar gabriel de oliveira ramosview author publications search author on:pubmed google scholar corresponding authors correspondence to joão batista rodrigues neto or gabriel de oliveira ramos . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper rodrigues neto, j.b., de oliveira ramos, g. (2021). an enhanced tsp-based approach for active debris removal mission planning. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_10 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords time-dependent traveling salesman problem space debris active debris removal genetic algorithm publish with us policies and ethics profiles gabriel de oliveira ramos view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a conversational agent to support hospital bed allocation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a conversational agent to support hospital bed allocation conference paper first online: 28 november 2021 pp 3–17 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a conversational agent to support hospital bed allocation download book pdf download book epub débora c. engelmann  orcid: orcid.org/0000-0002-6090-829410,11, lucca dornelles cezar  orcid: orcid.org/0000-0001-6312-152910, alison r. panisson  orcid: orcid.org/0000-0002-9438-550812 & … rafael h. bordini  orcid: orcid.org/0000-0001-8688-990110  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 865 accesses 12 citations abstract bed allocation in hospitals is a critical and important problem, and it has become even more important since last year because of the covid-19 pandemic. in this paper, we present an approach based on intelligent-agent technologies to assist hospital staff in charge of bed allocation. as part of this work, we developed a web-based simulation of hospital bed allocation system integrated with a chatbot for interaction with the user. as a core component in our approach, an intelligent agent uses the feedback of a plan validator to check if there are any flaws in a user-made allocation, communicating any detected problems to the user using natural language through the chatbot. thus, our resulting application not only validates bed allocation plans but also interacts with hospital professionals using natural language communication, including giving explainable suggestions of better alternative allocations. we evaluated our approach with professionals responsible for bed allocation in two local hospitals and a doctor who provides consultancy to another local hospital. the version of the system reported in this paper addresses all the suggestions made by the specialists who evaluated its previous version. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others optimizing hospital bed allocation for coordinated medical efficiency and quality improvement article 21 october 2024 better medical efficiency by means of hospital bed management optimization—a comparison of artificial intelligence techniques chapter © 2023 optimum utilization of bed resources in hospitals: a stochastic approach chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. e-health health care multiagent systems health care management practice and hospital management 1 introduction bed allocation is a challenge faced by hospitals because hospital beds are scarce, and when poorly managed, it can lead to long queues or chaos in emergency rooms [9]. this is even more critical considering that developing countries face growing financial constraints making planning and efficient allocation of hospital beds increasingly difficult [11, 20]. also, the area responsible for bed allocation needs to be concerned with several restrictions during bed allocation, such as the type of medical speciality, whether the patient is surgical or clinical, gender, and age of the patient [17]. this makes bed management an essential part of the planning and controlling of the operational capacity, and an activity that demands the efficient use of available resources [18]. effective management of hospital beds has been the focus of much research such as the imbeds model that uses artificial neural networks and multi-attribute value theory for decision making [9]; statistical and data mining approaches [19]; an optimisation model based on evolutionary algorithms is used for bed allocation in [13]; and also literature reviews have been carried out [2, 11]. although all that work seeks to improve bed management, they do not provide easy interaction nor decision support so that the professional in charge retains full control over allocations. keeping professionals in charge is important, given that this is a domain of sensitive decision-making, hence there is much resistance to replacing human operators with automated systems. in domains like healthcare, a mixed-initiative system, which supports human-computer interaction, becomes not just useful [10] but essential. in this domain, wrong decisions can lead to the loss of lives. that is why it is important that intelligent systems can support the decisions being made, but a human must make the final decisions. we have developed an intelligent agent to validate bed allocation plans and to communicate with hospital professionals using natural language. although humans still make the final decisions, an intelligent agent checks if all allocation rules are being complied with and warn the user if they are not. to make this possible, we use plan validation techniques and a chatbot that provides the agent with the ability to communicate with the human operator through natural language. our approach performs the validation of the bed allocation plan using a planning domain built based on the allocation rules used in each of the local hospitals that cooperated with our work. to use different rules for other hospitals, it is only necessary to adapt the planning domain and the way the planning problem is generated. we developed our own pddl plan validator in java, so that it interacts better with the chatbot developed using the jason platform [4]. we also created an automated generator of linear programming models for generating optimal bed allocations, which we can be used to provide suggestions of better alternative allocations when requested by the human operators, for example when the chatbot detected errors in the allocations they created themselves. we can generate optimal allocations much quicker with a solver such as glpk than with a pddl planner. we have evaluated our approach with professionals responsible for bed allocation in two local hospitals. also, the approach was evaluated by a doctor who provides consultancy to a local hospital. with their feedback, we were able to assess the efficacy and acceptance of our approach by those professionals; they considered our approach extremely useful and usable in the daily routine of the hospital. the version of the chatbot described in this paper in fact addresses all the suggestions made by those professionals to increase the usefulness of the chatbot for their work. the remainder of this paper is structured as follows. section 2 shows the bed allocation scenario and our approach. section 3 presents the results obtained in the evaluation of our approach. section 4 explains the improvements made. section 6 provides some conclusions. 2 materials and methods 2.1 eliciting bed allocation rules we interviewed a person responsible for bed allocation at hospital conceição in porto alegre rs, brazil. this professional (referred to as ba1) has more than seven years experience in this work. our objective was to understand the real scenario of hospital bed allocation. fig. 1. bed allocation rules. full size image hospital conceição is the largest unit of the conceição hospital group (ghc). it offers all the specialities of a general hospital in its outpatient clinic, as well as an emergency room and inpatients. it maintains a medical emergency service with doors open 24 h a day and has 784 beds [8]. for contextualisation, in brazil there are three types of health systems: private, where the patient pays; private health insurance, where the patient pays monthly and this plan covers the costs of the hospital; and sus is the national healthcare system that is funded by the federal government. 80% of the population uses sus [17]. the hospital conceição only receives sus patients. ba1 is part of a department of the hospital called nir (portuguese acronym for internal regulation centre). nir is an administrative unit that monitors the patient from arrival at the hospital, during the hospitalisation process, and internal and external movement until discharge from the hospital. nir has a manual created in cooperation with the health ministry to guide sus managers and better conduct the process of creation and running of nir units [12]. the nir has full control over hospital beds. the function of the operational nurse (ba1) is the real-time management of free beds. ba1 is responsible for authorising new admissions from the requested reservations, the exchanges, and the blocking required according to the demand and availability. ba1 currently uses a locally developed system that has a feature for bed management. this system only shows which patients need hospitalisation and which beds are currently available. the rules for bed allocation are in a document outside the system, so the person responsible for allocation needs to look at the document, or in case of ba1, memorised all those rules. bed allocation errors often occur because some rule is overlooked. some errors can cause delays in a patient’s accommodation since it is necessary to wait for a new allocation, besides all the unnecessary patient movement. based on the information obtained from ba1 and the nir manual [12], we created a diagram with the main bed allocation rules (see fig. 1). the diagram shows a large number of rules that need to be considered for an adequate allocation, prioritising good care, privacy, and the psychological state of patients. when there is overcrowding in the hospital, some rules may be ignored. in contrast, others can never be relaxed – for example, placing a patient who needs isolation in a non-isolation area. fig. 2. bed allocation approach. full size image 2.2 developing an approach to bed allocation we propose an approach for helping the professionals ensure that all bed allocation rules have been followed (see fig. 2). in our approach, healthcare professionals inform which patient and which bed they want to allocate the patient for the system to validate the allocation in accordance to a particular set of rules. this is done as follows: our system, based on the allocation information (bed and patient), automatically generates a pddl (planning domain definition language) [1] problem file and a pddl plan file; the particular hospital rules in question are represented as a pddl domain specification. the part of the system with patient and hospital information we call “web simulator”,footnote 1 because it is implemented as a web system to simulate a hospital information system. our web simulator was developed to facilitate interaction with the user when testing our approach. figure 3 shows the simulator bed-allocation screen. fig. 3. web simulator – bed allocation screen. full size image our approach also has an intelligent bdi agent constructed with jason [4], which has access to the pddl problem file and a pddl plan file and uses them to validate the allocation, checking if any rules are being broken.footnote 2 to do this validation, the jason agent originally used the val validator [7, 10], which outputs a latex report saved to a folder that will be analysed by the agent; we later mention how this has been reworked to improve performance. after the report is saved, the agent reads and processes that file. val is a plan validator that in our case checks if any bed allocation rule has been broken in the user allocation. val receives three files as input: 1) the plan file containing the information of the allocation made; 2) the problem file contains the current world state and the goals that should be achieved; 3) the domain filefootnote 3 that describes the rules that must be followed to perform the allocation. fig. 4. web simulator – bed allocation screen with chatbot. full size image this domain file has predicates that describe the characteristics of beds and patients. the actions contained in the domain file are responsible for determining the behaviour according to the real hospital rules. after analysing the validation report, our agent creates the response to the user, saying whether or not the plan is valid, and if it is not valid, it also tells the user which rule was being broken. one of the limitation of val, which led to the development we explain later, is that it stops the validation as soon as a broken rule is found, so if there are more broken rules, our agent would still report only the first one found. this response is sent to the user through a chatbot using the dialogflow platform,footnote 4 which in addition to responding, asks whether the user wants to confirm the allocation (see fig. 4). if the answer is positive, the chatbot itself triggers the routine on the system responsible for completing the allocation. if the answer is no, the chatbot just cancels the allocation, leaving the system’s validation history saved. when the user requests the chatbot to “check if my allocation is valid”, an intention is triggered on dialogflow; we call it “get validation result”. this intention calls through the webhook a function in cloud functions that searches in the database the last validation carried out by our jason intelligent agent and returns to dialogflow the answer elaborated by it. in addition to answering, the chatbot also asks if the user wants to “confirm the allocation”. for the chatbot to wait for the user’s response, we use dialogflow contexts. for the integration between the web simulator, dialogflow, and firestore, we use cloud functionsfootnote 5 that runs on a nodejs platform. furthermore, to integrate the intelligent agent with cloud functions, we use an api developed in asp.net core.footnote 6 the information returned by the api, as well as the representation of the plan and problem, are saved in the database so that the chatbot can access it when prompted. once this data is saved, the simulator issues a warning informing the user that it can already request the validation results to the chatbot. 3 evaluation we evaluated our approach in three phases. the first phase was carried out with two professionals responsible for bed allocation at hospital conceição; one is the same professional who previously informed us of the bed allocation rules. the second phase was with two professionals also responsible for bed allocation but at hospital são lucas da pucrs (hsl). finally, we evaluated our chatbot with the help of one of the doctors who assisted in the construction of the nir manual [12]. for all evaluations, we fed the system with simulated data about beds, doctors, and patients. based on the data in the system, we asked that professionals use the simulator to check out the simulated hospital situation and ask the chatbot to validate the bed allocation they created and then evaluate the feedback that the chatbot gave. furthermore, we performed a semi-structured interview in order to collect the feedback of these professionals about the use of the system. all professionals signed a consent form for participation. 3.1 first phase we evaluated the approach with each professional individually. the first professional (bd1), who informed us about the allocation rules, has 7 years of experience in bed allocation. the second professional has more than 4 years experience in bed allocation. we highlight some points reported by those professionals: chatbot and allocation rules: interaction with chatbot is extremely easy. the information about bed allocation problems that the chatbot points out are very useful and really what they use in practice. they agree that it is not viable a system that allocates alone without a final decision made by a human operator. they would be willing to use our system in their daily activities. suggestions: consider patients’ priorities. the validation performed by the system is useful, but it would be even more valuable if it suggested how to correct the allocation if any rules were broken, providing allocation suggestions that do not break the rules. 3.2 second phase we then evaluated our approach with two professionals at the same time. both professionals have more than one year of experience in the bed allocation function. we also asked questions related to the allocation process to understand the differences between the reality of this hospital and hospital conceição. the main difference is that hsl serves the three types of health systems. we highlight some important considerations reported in this interview: chatbot and allocation rules: they would not like a system that allocates beds autonomously, and prefer one that only make recommendations when requested. they believe that the agent can help them in the daily routine. they are willing to use our system in their daily activities if it remains not necessary to talk much to the robot or chatbot. suggestions: their current system does not distinguish between sus beds, private health insurance, and private; it would be important that the chatbot had this knowledge. considering the rules of patient prioritisation is extremely important to them. they suggested that the system should give priority to the relocation of a patient released from the icu rather than other patients who need a bed of that type. interpret natural language written texts in the patient’s evolution to retrieve relevant information. generate warnings when a patient has been in the icu more than 8 days or more than 30 days in the same bed. based on the patients’ discharge plan for the following day and the scheduling of procedures, the agent could advise if beds will be missing. tell when a patient was discharged for more than 30 min but the bed has not yet been vacated. tell when a bed is interdicted for more than 24 h. the agent should also knows the business rules (regarding private health insurance) so that it can validate them too. 3.3 third phase we performed an evaluation with the doctor who graduated over 10 years ago and has more than a year of experience as consultant medical doctor. we highlight some important considerations reported in this interview: chatbot and allocation rules: the rules are well in line with what are current practices. it is feasible, and it is even necessary to reduce the variability in the conduction of bed allocation processes. they informed us that they would use our system, because this type of software would make the process very fast, in addition to the accuracy and availability to work 24 h a day, seven days a week. suggestions: define the bed typology and not ask the doctor to define it, but validate the typology. suggest an allocation to the user. not that the agent should make an allocation alone, but it would be good to give suggestions that someone would confirm. offer options of frequent-asked questions, instead of user typing or talking to the chatbot all the time. when there is a list of patients in the emergency room waiting for a bed and a bed that is suitable for any of these emergency patients becomes vacant, the system should warn the operator. generate a bed availability forecast, for example, for the next six hours according to the discharge forecasts contained in the system. 4 improvements based on the evaluation some suggestions made during the evaluation were considered of major importance and, in order to achieve them, we have carried out some improvements in our system. to facilitate communication between the jason agent and dialogflow, we started using dial4jacafootnote 7 [5, 6], a general integration between jacamo and dialogflow that has recently become available. to solve the limitation imposed by the val validator, which did not return a complete list with all the allocation problems that the plan has, we implemented a new pddl validator that will be presented next. moreover, to enable our agent to make optimised suggestions for bed allocations, we implemented an optimal bed allocation technique using linear programming, which will also be presented in more detail below. 4.1 bed allocation optimisation in order to allow our agent to make optimal suggestions, we developed an approach to bed allocation based on linear programming. after a linear programming problem is generated for a particular bed allocation instance, we use the glpk solver to find an optimal solution.footnote 8 the optimisation tries to allocate as many patients as possible while attempting to decrease as much as possible the distance of critical patients to the office where nurses are based. the bed allocation optimisation program takes the database’s restrictions and converts them into three types of linear restrictions: equality, relative equality, and negation. consider for example the following constraint. $$\begin{aligned} \texttt {(2*q112[p])+(abs(speciality[p]-2)/2) <= 2;} \end{aligned}$$ the equality constraint above requires that every patient allocated to the bedroom has a specific characteristic. in this case, we want the patient to have the speciality 2. q112[p] is a boolean value equal to 1 if patient p is in the bedroom; gender [p] is an integer value referring to the patient’s speciality, hospitals will usually have specific rooms for specific types of specialities. the left side of the sum results in 2 if the patient has been allocated to that bedroom; otherwise, it results in 0. the right-hand side results in 0 if the patient’s speciality is 2; otherwise, it results in some number greater than zero. the result of the sum must be less than or equal to 2, being the only case in which this does not happen when the patient is allocated to the bedroom, but the speciality is not 2. $$\begin{aligned} \texttt {(q112[p1]+q112[p2])+(abs(gender[p1]-gender[p2])/2) <= 2;} \end{aligned}$$ relative equality is used to require, for example, that if two patients are allocated to the same bedroom, they must have the same characteristic (in the case above, the same gender). the sum’s left-hand side has a maximum value of 2 when the two patients are in room 112. the right-hand side results in a value greater than 0 when the two patients’ genders are different. the restriction would only be violated when the two patients are in the same bedroom, but the genders are different, which must not be allowed. $$\begin{aligned} \texttt {(q112[p]) abs(isolation[p]-1) <= 0;} \end{aligned}$$ negation is used for example when we require that every patient allocated to the room does not have a specific characteristic. if the patient is in the room, but the isolation characteristic is 1, this will result in \(1-0 = 1\), which does not comply with the restriction. if the patient is not allocated to the room or the isolation value is not 1, the result is less than or equal to zero, according to the bed rules. if the room is not designed for isolation, it will not allow patients that must be isolated under the normal circumstances under which the optimiser normally operates. in this work, we used the glpsol solver of glpkfootnote 9 (gnu linear programming kit), which is a free open source software for solving linear programming problems. one of the implemented algorithms is the simplex method where, after assembling a geometrical figure, the program chooses a point and recursively chooses the relative maximum (or minimum) point, approaching the global optimal result at each iteration. glpsol allows the user to set certain limits, such as a time limit. when the limit is reached, the process returns the best result found so far. this allows the program to generate some possibly useful allocation suggestions without having to arrive at the optimal result, which can be an extremely time-consuming process in some instances. we have already integrated the optimiser with jason platform, allowing us to execute tests with our complete structure. figure 5 presents the result of an allocation made using the optimiser.footnote 10 fig. 5. optimiser output in web simulator full size image 4.2 plan validator when real-world problems can be modelled in a planning language, it is possible to use a plan validator for example to tell the human operator whether a given plan is feasible or not [10]. behnke et al. [3] define plan validation as “the task of determining whether a plan is a solution to a given planning problem”. a plan validator can be used in a wide range of applications. the application that interests us is the validation of bed allocation plans prepared by the user. we developed a new plan validatorfootnote 11 using java to facilitate the integration with jason. like val, our plan validator also receives three pddl files as input: a file containing the domain, a file containing the problem, and a file containing the plan to be validated. the domain file establishes some basic rules, such as the types of objects and possible actions. the actions are generally divided into three parts, the parameters, the preconditions, and the effects. in this case, the action of allocating a patient to a bed requires that the patient is not allocated to another bed and that the bed is empty. all other bed allocation constraints are also modelled as preconditions. the effects generated by applying this action are that the patient is now allocated, the room occupied, and the patient is allocated in that room. problem files use domain rules to determine a particular problem, making all objects (e.g., patients and beds) and objectives explicit (e.g., all patients must be allocated). for validators, a plan file is also necessary, which is simply a set of actions to be applied sequentially, which lead to the objective of the problem. given these files, the validation process is straightforward, checking if each action is applicable and then apply its effects. an action is only applicable if the types of the parameters are correct (e.g., the action “aloc bed3” does not work if the action aloc requires a patient type) and if the preconditions have been met. if any action is not applicable, the plan is considered flawed. if all actions are applicable, but the problem’s objective is not satisfied, then the plan is also considered to be flawed. the plan only satisfies the problem if all actions are applicable and the objective is satisfied after applying the last action. a characteristic of the hospital domain is that the actions are not ordered. given a suitable set of allocations, any order of execution will solve the problem, which, in general, would not be the case in other domains. this means that for us it is particularly useful to detect all errors in the plan and not just the first. in our plan validator, the user has the possibility of printing the plan and the validation result in the terminal through the plantest function. the user still has the possibility, through the valout(“filename”) function, to obtain a latex file that generates a pdf with the validation report,footnote 12 the result can also be analysed by our intelligent agent to give an answer to the user in natural language. as the validator was designed so far thinking about a specific type of problem, we have made, for now, certain restrictions to facilitate the implementation of an initial version of the software. in our initial version, we have some limitations that do not negatively affect the results of the tasks for which we are using the validator, but that need to be resolved to make our validator available to the research community to use it. although the pddl language is modular, some specific options are practically universally accepted, however not all of them have been implemented yet. in total, our validator currently supports three options: “strips”, which allows for actions to add or remove effects, and is required in our validator; “typing”, which allows the use of types and sub-types; and “equality”, which allows the use of equality comparisons. another actual limitation is that the plan validator was made to be used in automatically generated problems. it considers that the pddl files are semantically and syntactically correct. we have not yet implemented any pre-processing to look for errors automatically, but we developed a callable method to perform this check. the current version of our plan validator is available on github.footnote 13 5 related work oliveira et al. [13] developed an iterative simulation-optimisation approach using an evolutionary algorithm to optimise hospital-bed allocation. they applied their approach using datasets from datasus of minas gerais, brazil, where the public health system assists nearly 80% of the patients. grübler et al. [9] proposed a model called imbeds to allocate patients to beds. imbeds is a hybrid model that aids in the bed selection process, using some techniques that work together to manage a waiting list of emergency patients and scheduled patients. the model uses an artificial neural network (ann) and multiattribute value theory (mavt), a technique used for decision making and conflict resolution in projects with multiple criteria, using contextual information about patients and beds. they evaluated the model in a hospital in porto alegre, brazil, that receives only patients from private insurance plans. in teow et al. [19], the authors extracted data from a hospital in singapore and applied statistical and data mining approaches to identify the patterns behind bed overflow. their main objective was to help the hospital devise strategies to reduce bed overflow and improve patient care. differently from the work described above, not only does our approach suggest and validate bed allocations through optimisation techniques but it also supports natural language communication using chatbots to interact with the users. we believe our application makes progress towards existing needs in the application of ai systems in domains like healthcare, in which agents provide natural language explanations, e.g., allocation failure in our scenario, supporting users’ decision making rather than being the decision makers. 6 conclusions this paper presented an approach to support hospital staff during the process of bed allocation. although there is some work in the literature that intends to make completely automated allocations, this area of application has resistance to replacing human operators with automated systems. therefore, in our approach, we only seek to support the decision-making of human professionals. our approach was built based on the expertise of professionals in the field, mainly based on an interview we conducted with a professional responsible for bed allocation in a local hospital. we evaluated our approach at hospital conceição and hsl, the latter having practices with significant differences since it serves not only sus but also private patients. also, we interviewed a doctor that provides consultancy to a local hospital in the area of bed allocation. among the points highlighted by the professionals about our approach, it has been mentioned that it is easy to interact with the chatbot and the information contained in the validation of allocation are very useful for their routine. some good ideas have also been raised to improve our system so that it performs useful tasks in the bed-allocation routine. as our approach was developed based only on the existing practices at hospital conceição, the analysis made by the professionals of hsl yielded many ideas for our agent to be more useful in its planned use in those hospitals. through this research, we created a domain knowledge and developed a planning domain for pddl and htn planners that can also be useful for other projects that involve automated planning and plan validation. as future work, we intend to investigate argumentation techniques [14,15,16] to implement more interactive agents and explain the allocation suggestions as requested by the interviewees. we believe that with the use of argumentation techniques, intelligent agents will be able to reason about beds and patients’ relations, thus providing useful explanations that will help the users by saying why a particular suggestion is being made. we intend to integrate our approach with the hospitals’ information systems, which will allow us to make several queries as suggested by the experts during the evaluation phase. moreover, we aim to use natural language models to allow the chatbot to answer questions about a patient’s evolution. notes 1.the web simulator code is available at https://github.com/smart-pucrs/bed-allocation-simulator. 2.the agent code is available at https://github.com/smart-pucrs/jason_assistant_to_bed_allocation. 3.the domain file with plans and problems examples are available at https://github.com/smart-pucrs/hospital-domain-pddl. 4.https://dialogflow.com/. 5.https://cloud.google.com/functions/. 6.https://www.asp.net/core/overview/aspnet-vnext. 7.https://github.com/smart-pucrs/dial4jaca. 8.https://en.wikibooks.org/wiki/glpk. 9.http://winglpk.sourceforge.net/. 10.all patient data in our tests are fictitious. 11.the validator code is available at https://github.com/smart-pucrs/pddl-plan-validator. 12.https://github.com/smart-pucrs/pddl-plan-validator/blob/main/latexoutputvalidator.pdf. 13.https://github.com/smart-pucrs/pddl-plan-validator. references aeronautiques, c., et al.: pddl— the planning domain definition language. technical report (1998) google scholar  ahmadi-javid, a., seyedi, p., syam, s.s.: a survey of healthcare facility location. comput. oper. res. 79, 223–263 (2017) article  mathscinet  google scholar  behnke, g., höller, d., biundo, s.: this is a solution! (... but is it though?) verifying solutions of hierarchical planning problems. in: 27th international conference on automated planning and scheduling, pp. 20–28 (2017) google scholar  bordini, r.h., hübner, j.f., wooldridge, m.: programming multi-agent systems in agentspeak using jason. wiley, hoboken (2007) book  google scholar  engelmann, d., et al.: dial4jaca – a demonstration. in: dignum, f., corchado, j.m., de la prieta, f. (eds.) paams 2021. lncs (lnai), vol. 12946, pp. 346–350. springer, cham (2021). https://doi.org/10.1007/978-3-030-85739-4_29 chapter  google scholar  engelmann, d., et al.: dial4jaca – a communication interface between multi-agent systems and chatbots. in: dignum, f., corchado, j.m., de la prieta, f. (eds.) paams 2021. lncs (lnai), vol. 12946, pp. 77–88. springer, cham (2021). https://doi.org/10.1007/978-3-030-85739-4_7 chapter  google scholar  fox, m., howey, r., long, d.: validating plans in the context of processes and exogenous events. in: 20th national conference on artificial intelligence and the seventeenth innovative applications of artificial intelligence conference, pp. 1151–1156 (2005) google scholar  ghc: hospital conceição (2019). https://www.ghc.com.br/default.asp?idmenu=unidades&idsubmenu=1 grübler, m.d.s., da costa, c.a., righi, r., rigo, s., chiwiacowsky, l.: a hospital bed allocation hybrid model based on situation awareness. comput. inform. nurs. 36, 249–255 (2018) article  google scholar  howey, r., long, d., fox, m.: val: automatic plan validation, continuous effects and mixed initiative planning using pddl. in: 16th international conference on tools with artificial intelligence, pp. 294–301 (2004) google scholar  matos, j., rodrigues, p.p.: modeling decisions for hospital bed management a review. in: 4th international conference on health informatics, pp. 504–507 (2011) google scholar  ministério da saúde: manual de implantação e implementação : núcleo interno de regulação para hospitais gerais e especializados. technical report, ministério da saúde, secretaria de atenção à saúde, departamento de atenção hospitalar e de urgência (2017) google scholar  e oliveira, b., de vasconcelos, j., almeida, j., pinto, l.: a simulation-optimisation approach for hospital beds allocation. int. j. med. inform. 141, 104174 (2020) google scholar  panisson, a.r., bordini, r.h.: knowledge representation for argumentation in agent-oriented programming languages. in: 2016 brazilian conference on intelligent systems, bracis (2016) google scholar  panisson, a.r., bordini, r.h.: towards a computational model of argumentation schemes in agent-oriented programming languages. in: ieee/wic/acm international joint conference on web intelligence and intelligent agent technology (wi-iat) (2020) google scholar  panisson, a.r., engelmann, d., bordini, r.h.: engineering explainable agents: an argumentation-based approach. in: international workshop on engineering multi-agent systems (emas) (2021) google scholar  pinto, l.r., de campos, f.c.c., perpétuo, i.h.o., ribeiro, y.c.n.m.b.: analysis of hospital bed capacity via queuing theory and simulation. in: proceedings of the winter simulation conference 2014, pp. 1281–1292. ieee (2014) google scholar  proudlove, n.c., gordon, k., boaden, r.: can good bed management solve the overcrowding in accident and emergency departments? emerg. med. j. 20, 149–155 (2003) article  google scholar  teow, k.l., el-darzi, e., foo, c., jin, x., sim, j.: intelligent analysis of acute bed overflow in a tertiary hospital in singapore. j. med. syst. 36, 1873–1882 (2012) article  google scholar  tsai, j.c.h., et al.: adjusting daily inpatient bed allocation to smooth emergency department occupancy variation. in: healthcare, vol. 8, p. 78. multidisciplinary digital publishing institute (2020) google scholar  download references acknowledgements this research was partially funded by cnpq and capes – finance code 001. author information authors and affiliations school of technology, pucrs, porto alegre, rs, brazil débora c. engelmann, lucca dornelles cezar & rafael h. bordini dibris, university of genoa, genoa, italy débora c. engelmann department of computing, ufsc, araranguá, sc, brazil alison r. panisson authors débora c. engelmannview author publications search author on:pubmed google scholar lucca dornelles cezarview author publications search author on:pubmed google scholar alison r. panissonview author publications search author on:pubmed google scholar rafael h. bordiniview author publications search author on:pubmed google scholar corresponding author correspondence to débora c. engelmann . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper engelmann, d.c., cezar, l.d., panisson, a.r., bordini, r.h. (2021). a conversational agent to support hospital bed allocation. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_1 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hospital bed allocation intelligent agent pddl plan validation linear programming chatbot publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving particle swarm optimization with self-adaptive parameters, rotational invariance, and diversity control | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving particle swarm optimization with self-adaptive parameters, rotational invariance, and diversity control conference paper first online: 28 november 2021 pp 218–233 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) improving particle swarm optimization with self-adaptive parameters, rotational invariance, and diversity control download book pdf download book epub matheus vasconcelos10, caio flexa10, igor moreira10, reginaldo santos10 & … claudomiro sales10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 736 accesses abstract particle swarm optimization (pso) algorithms are swarm intelligence methods that are effective in solving optimization problems. however, current techniques have some drawbacks: the particles of some pso implementations are sensible to their input hyper-parameters, lack direction diversity in their movement, have rotational variance, and might prematurely converge due to rapid swarm diversity loss. this article addresses these issues by introducing rotationally invariant attractive and repulsive expanded pso (ri-ar-xpso) and rotationally invariant semi-autonomous expanded pso (ri-saxpso) as improvements of rotationally invariant semi-autonomous pso (ri-sapso) and expanded pso (xpso). their swarm behavior was evaluated with classic functions in the literature and their accuracy was tested with the congress on evolutionary computation (cec) 2017 optimization problems, in whose results a statistical significance test was applied. the results obtained attest that strategies such as diversity control, automatic hyper-parameter adjustment, directional diversity, and rotational invariance improve performance without accuracy loss when adequately implemented. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others diversity measurement in different pso variants applied to global optimization and classical engineering problems chapter © 2025 a survey on particle swarm optimization algorithm chapter © 2023 different variants of particle swarm optimization algorithms and its application: a review chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms continuous optimization diversity-oriented synthesis learning algorithms optimization stochastic learning and adaptive control 1 introduction heuristic algorithms are becoming increasingly more robust means to find suitable solutions to complex problems. current approaches range from low-level heuristics—designed for specific issues—to complex, more general hyper-heuristics [4]. particle swarm optimization (pso) is an effective, swarm-intelligence technique based on the collective behavior of fish shoals and bird flocks used to solve optimization problems. the parameter settings of non-deterministic algorithms such as meta-heuristics are known to be dependent on the problem. the same occurs with pso, for the adjustment of hyper-parameters affects the trajectory of the particles in the swarm. in some implementations, the cognitive and social coefficients, as well as the inertial weight, are dynamically selected [17]. in others, these hyper-parameters are manually set and remain fixed throughout the execution [15]. the authors of implementations with fixed hyper-parameters customarily disclose recommended values for them, defined after multiple analyzes on different problems, as in [14]. in pso, the works of [7, 10, 18] demonstrate that particles tend to move in parallel to the coordinate axes. using rotational invariance mitigates this dependency on the coordinate system wherein the objective function is defined, improving results across various problems. in terms of complex multimodal functions, experiments indicate that maintaining swarm diversity helps mitigate premature convergence and avoid sub-optimal solutions [11]. motivated by these aspects, this paper aims to improve two pso implementations termed rotationally invariant semi-autonomous pso(ri-sapso) [14] and expanded pso (xpso) [21] by automatically updating hyper-parameters, promoting rotational invariance, and maintaining swarm diversity during execution. the remaining sections are structured as follows. section 2 describes the canonical pso, as well as some properties of its variants. in sect. 3, the framework of some pso variants is exposed. details on the improvements made to ri-sapso and xpso are presented in sect. 4. section 5 presents the obtained experimental results and their statistical significance. lastly, sect. 6 concludes this paper with some remarks and future work. 2 theoretical background the canonical pso algorithm only performs simple mathematical operations. as such, it is computationally inexpensive in terms of time and space. pso is a population-based meta-heuristic where each individual (i.e., particle) represents a candidate solution to an optimization problem in a d-dimensional space. in each iteration t of the algorithm, the ith particle of the swarm is associated to three vectors: the position vector \(\vec {x}_i = \left[ {x_{i,1}},{x_{i,2}},\ldots \,{x_{i,d}}\right] \), that represents the candidate solution; the velocity vector \(\vec {v}_i = \left[ {v_{i,1}},{v_{i,2}},\ldots \,{v_{i,d}}\right] \), that represents the direction and velocity of the particle in the search space; and local memory vector \(\vec {p}_i = \left[ {p_{i,1}},{p_{i,2}},\ldots \,{p_{i,d}}\right] \), which stores the best solution found. by its turn, the swarm is associated to the global memory vector \(\vec {g} = \left[ {g_1},{g_2},\ldots \,{g_d}\right] \), which stores the best solution found overall. following the algorithm of [5], the position of the particles are updated as t increases according to $$\begin{aligned} \vec {v}^{t + 1}_i = \vec {v}^{t}_i + c_1\phi _1(\vec {p}^{t}_i \vec {x}^{t}_i) + c_2\phi _2(\vec {g}^{t} \vec {x}^{t}_i) \end{aligned}$$ (1) and $$\begin{aligned} \vec {x}^{t + 1}_i = \vec {x}^t_i + \vec {v}^{t + 1}_i, \end{aligned}$$ (2) where \(c_1\) and \(c_2\) respectively represent the cognitive and social coefficients and \(\phi _1\) and \(\phi _2\) are random numbers sampled from a uniform distribution in [0, 1]. ulteriorly, [17] added the inertial weight w to (1), turning \(\vec {v}^{t}_i\) into \(w\vec {v}^{t}_i\). this significantly improved the canonical pso and is widely used in its variants. making the algorithm rotationally invariant ensures that particles will move independently without influence of the coordinate system of the problem. its presence (or lack thereof) might lie in how \(\phi _1\) and \(\phi _2\) are used in the velocity update equation. if they are vectors, the optimizer is variant in terms of scale, translation, and rotation. if they are scalar values, then the algorithm is rotationally invariant. to illustrate, a rotationally variant version of (1), which is rotationally invariant, can be defined as $$\begin{aligned} \vec {v}^{t + 1}_i = w^t\vec {v}^{t}_i + c_1\dot{\vec {\phi }}^t_i\odot (\vec {p}^t_i \vec {x}^t_i) + c_2\ddot{\vec {\phi }}^t_i\odot (\vec {g}^t \vec {x}^t_i), \end{aligned}$$ (3) where \(\dot{\vec {\phi }}^t_i\) and \(\ddot{\vec {\phi }}^t_i\) are two vectors containing random values sampled from a uniform distribution in [0, 1] and \(\odot \) represents the element-wise multiplication of vectors and matrices [8]. the mathematical tool presented in [20] can be used to prove whether a given pso implementation is variant or invariant. the work of [19] introduces the concept of directional diversity, i.e., the ability to carry out the stochastic search in various directions. conversely, particles without directional diversity perform their search in fixed directions. this work proved that, unlike (3), (1) provokes loss in directional diversity. to demonstrate that the directional diversity and rotational invariance are not necessarily exclusive, [20] proposed a rotationally invariant pso that maintains directional diversity named diverse rotationally invariant pso (dri-pso), whose velocity update equation is rotationally invariant in a stochastic manner but also diverse in direction. this was possible thanks to small, consistent perturbations in the direction of the local \((\vec {p} \vec {x})\) and global swarm memories \((\vec {g} \vec {x})\), multiplying these values with an independent, random rotation matrix w. the ability to store gradient information allows the particles to individually perform asynchronous searches, mitigate the random walk effect and avoid wasting random efforts to reach local optima. when using the method described in [9] with problems of high d, the employment of gradient descent information might be difficult. depending on how it is applied throughout the search, up to d objective function evaluations per particle might be performed. the ability to control swarm diversity, firstly proposed to pso in [13], is present in many implementations. this strategy, which entails monitoring the swarm diversity during the search, enables the algorithm to increase exploration when the diversity is low to avoid local optima. equation (4) represents one form of defining swarm diversity, where |l| is the diagonal size of the search space, \(x^{t}_{i,j}\) is the j-th dimension of the i-th particle, and \(\overline{x}^t_j\) is the j-th dimension of the mean position across all particles. $$\begin{aligned} diversity(\vec {x^{t}}) = \frac{1}{n \times |l|} \times \sum _{i=1}^{n} \sqrt{\sum _{j=1}^{d}(x^{t}_{i,j} \overline{x}^t_j)^2} \end{aligned}$$ (4) 3 related work works related to this paper are associated with other meta-heuristics inspired on the canonical pso. this section will focus on implementations that involve diversity control, gradient use, rotational invariance, directional diversity, rotation matrices, and hyper-parameter auto-adjustment. the introduction of gradient-based pso (gpso) is described in [12], wherein the gradient directions are used as a deterministic approach for a precise local exploration around the best global position, thus strengthening the global exploration of the algorithm. in [6], where diversity control and gradient information were employed as strategies to switch between two pso algorithms throughout the search, a new, hybrid approach termed diversity-guided pso based on gradient search (dgpsogs) was introduced. the semi-autonomous pso (sapso) [15] takes advantage of ideas present in attraction and repulsion pso (arpso) [13], gpso [12], and dgpsogs [6] to provide a semi-autonomous particle swarm optimizer that uses gradient and diversity-control information to optimize unimodal functions. sapso attempts to reduce computational efforts related to local investigation with the aid of gradient information and provide a diversity control mechanism to avoid local optima. the performance of sapso and other pso implementations are evaluated in a set of test functions based on optimization problems of de jong’s benchmark. numerical results showed that the proposed method attained at least the same performance of other pso implementations. moreover, sapso achieved better results in terms of global minima found and fine-tuning of the final solution. however, problems in spaces of higher dimensionality caused longer execution times due to the employment of gradient-based information by each particle and the need to modify parameters depending on the problem at hand. as an improvement to dri-pso, the locally convergent rotationally invariant pso (lcripso) was introduced in [2]. the authors pointed out several problematic situations: particles can get stuck in some areas of the search space, unable to change the value of one or more decision variables; poor performance is observed when the swarm is small or the dimensionality is high; convergence is not guaranteed even for local optima; and the algorithm is sensible to the rotation of the search space. aiming to solve these issues, lcripso contains a new general form of velocity update equation that contains a normal distribution function defined by the user around local and global memories. the ri-sapso is proposed in the work of [14] as an improvement to sapso [15]. this pso implementation inherits the rotational invariance of sapso and incorporates a rotation matrix generated by an exponential map to maintain directional diversity using an idea present in [20]. besides mathematically proving that ri-sapso is rotationally invariant, benchmark tests with statistical significance demonstrated that the algorithms were capable of finding better solutions in most problems. therefore, ri-sapso is the starting point of this work because despite its superior results against those of sapso, it still has long execution times when handling high-dimensional problems and employs fixed hyper-parameters during execution. an expanded variant of pso termed xpso was proposed in [21]. it updates acceleration coefficients and the learning model based on fitness, given that updating these coefficients by only considering iterations might result in sub-par intelligence when solving complex problems. moreover, xpso expands the social part of each particle from one to two exemplars and applies the ability to forget some particles, which is based on a universal biological phenomenon wherein parts of the memory management system forget undesirable or useless information. upon testing, this approach presented promising results with different types of objective function, despite its possible lack of rotational invariance as it employs the random components similarly to (3). 4 improving ri-sapso and xpso this section details how xpso and ri-sapso were changed into the rotationally invariant attractive and repulsive expanded pso (ri-ar-xpso) and the rotationally invariant semi-autonomous expanded pso (ri-saxpso). 4.1 rotationally invariant attractive and repulsive expanded pso the implementation herein defined is termed rotationally invariant attractive and repulsive expanded pso. since xpso has a vector of random numbers \(\vec {\phi }^t\) for each component in its velocity update equation—similar to (3)—, its search strategy might have directional diversity, but still suffers from rotation variance, which can be proven by using the mathematical tool seen in [20]. as previously exposed, in addition to automatically adjusting the acceleration coefficients during the search, xpso has an additional social coefficient. here, the ith particle out of n particles will be related to three coefficients: \(c_{1,i}\), which is the cognitive coefficient; and \(c_{2,i}\) and \(c_{3,i}\), which are social coefficients. the coefficients of each particle are defined using a gaussian following the method described in [21]. a greater movement amplitude is preserved in each particle by using the stochastic rotation matrix w in the velocity equation, thus increasing directional diversity in the search [20]. furthermore, it also promotes rotational invariance. as such, the velocity equation will be changed to use w similarly to [14]. the authors of xpso justify that its ability to forget relates to maintaining swarm diversity throughout the search. in this regard, another improvement to be added was motivated by the attraction and repulsion strategy of [13], where the diversity monitoring will influence the behavior of the particles such that they will not forget during the attraction phase to accelerate convergence. moreover, a lower swarm diversity bound will be dynamically defined in the new implementation to avoid that the repulsion phase precludes refining the solution at the end of the search. the \(d^t_{min}\) values will exponentially decrease at this point, with values defined by \(f(x)\,=\,b^x\) (the best results were attained with x having values within [1, 10] linearly increasing during the search). \(b = 10^{-2}\) is the same value used by [14] as an upper swarm diversity bound. the modifications made to xpso change its velocity equation to $$\begin{aligned} \vec {v}^{t + 1}_i = w^t\vec {v}^{t}_i + dir \times \left[ \vec {\varomega }^t_i + \vec {\varupsilon }^t_i + \vec {\varlambda }^t_i\right] , \end{aligned}$$ (5) $$\begin{aligned} \vec {\varomega }^t_i={\left\{ \begin{array}{ll} c_{1,i} \dot{\phi }^t_i \left( \vec {p}^t_{i} \vec {x}^t_{i}\right) , &{} \text {if}\;dir=1;\\ c_{1,i} \dot{\phi }^t_i \dot{w}^t_i \left( \vec {p}^t_{i} \vec {x}^t_{i}\right) , &{} \text {else} \end{array}\right. } \end{aligned}$$ (6) $$\begin{aligned} \vec {\varupsilon }^t_i={\left\{ \begin{array}{ll} c_{2,i} \ddot{\phi }^t_i \left( \vec {l}^t_{i} \vec {x}^t_{i}\right) , &{} \text {if}\;dir=1;\\ c_{2,i} \ddot{\phi }^t_i \ddot{w}^t_i \left[ \left( 1-\vec {f}_{i}\right) \vec {l}^t_{i} \vec {x}^t_{i}\right] , &{} \text {else} \end{array}\right. } \end{aligned}$$ (7) $$\begin{aligned} \vec {\varlambda }^t_i={\left\{ \begin{array}{ll} c_{3,i} \dddot{\phi }^t_i \left( \vec {g}^t \vec {x}^t_{i}\right) , &{} \text {if}\;dir=1;\\ c_{3,i} \dddot{\phi }^t_i \dddot{w}^t_i \left[ \left( 1-\vec {f}_{i}\right) \vec {g}^t \vec {x}^t_{i}\right] , &{} \text {else} \end{array}\right. } \end{aligned}$$ (8) where \(\dot{\vec {\phi }}^t_i\), \(\ddot{\vec {\phi }}^t_i\) and \(\dddot{\vec {\phi }}^t_i\) are three random vectors uniformly distributed within \(\left[ 0, 1\right] \); \(\vec {\mathcal {f}}=\left[ f_{i,1},f_{i,2},\cdots ,f_{i,d}\right] \) represents the amount of information that the i-th particle forgot of a given sample; \(\vec {l}^t_{i}\) represents the best position visited between two neighbors (i.e., \(\vec {l}^t_{i}\) will be equal to the most fit solution between \(\vec {p}_{i-1}\) and \(\vec {p}_{i+1}\)); and dir controls the attraction and repulsion phases according to (9) where the current swarm diversity value is represented by d, calculated in (4), and stagg and \(stagg_{max}\) respectively represent the number of iterations without updating the best global position and the maximum number of iterations without progress allowed. the elitist behavior employed by the hyper-parameter adjusting logic during execution also aids the update of dir. as exposed by [21], when \(stagg \ge stagg_{max}\), it is assumed that the particles got stuck in a local minimum. as such, a hyper-parameter update takes place to change their search direction. nevertheless, if the particles maintain low diversity, the implementation will trigger the repulsion phase to speed up this process. conversely, the attraction phase is initiated when adequate diversity is maintained. 4.2 rotationally invariant semi-autonomous expanded pso the implementation defined herein is termed rotationally invariant semi-autonomous expanded pso. expanded refers to the automatic hyper-parameter adjustment and the expansion of the ri-sapso velocity equation inspired by xpso. in [14], the variant employs parameters without dynamic update during execution and uses two main strategies: the diversity-guided attraction/repulsion logic and the replacement of the personal part of the velocity equation from \(\vec {p^{t}_i} \vec {x^{t}_i}\) to \(\nabla f(\vec {x}^{t}_i)\) to use the gradient. as such, the improvements are: implement a hyper-parameter update strategy; employ adaptive diversity control; and employ three acceleration coefficients in the velocity update equation involving the best global position (\(c_1\)), the gradient (\(c_2\)), and the best personal position (\(c_3\)). the implementation of automatic hyper-parameter update during execution is identical to what is performed in xpso. the update scenarios of dir will be changed due to the difficulty it faces to converge when diversity is low, which hurts the results of ri-sapso in such situations. therefore, (9) will be used herein as well. the upper swarm diversity bound was removed in this case because when the search triggers the repulsion phase in later iterations, this restriction will result in high swarm diversity, thus preventing the refinement of the solution. the lower swarm diversity bound is \(d^t_{min}\), which will have the same values explained in sect. 4.1. unlike the original implementation, the switch from repulsion to attraction does not alter the individual decision of the particles. in other words, only dir changes from \(-1\) to 1, mitigating gradient use. due to the smaller gradient descent use in the attraction phase, \(c_3\) was added as a cognitive coefficient in (10) to accelerate convergence. in this case, if a particle is unable to decide with the gradient, it will determine with \(\vec {g}\) and \(\vec {p}\). after detailing all modifications, the equation that governs velocity updates in ri-sapso had its social component $$\begin{aligned} \vec {\varomega }^t_i={\left\{ \begin{array}{ll} i^{t}_i c_1 \dot{\phi }^t_i (\vec {g}^t \vec {x}^t_i), &{} \text {if}\;dir=1;\\ i^{t}_i c_1 \dot{\phi }^t_i \dot{w}^t_i (\vec {g}^t \vec {x}^t_i), &{} \text {otherwise} \end{array}\right. } \end{aligned}$$ (10) changed in the new implementation to $$\begin{aligned} \vec {\varomega }^t_i={\left\{ \begin{array}{ll} i^{t}_i c_{1,i} \dot{\phi }^t_i (\vec {g}^t \vec {x}^t_i) + i^{t}_i c_{3,i} \dddot{\phi }^t_i (\vec {p}^t \vec {x}^t_i), &{} \text {if}\;dir=1;\\ i^{t}_i c_{1,i} \dot{\phi }^t_i \dot{w}^t_i (\vec {g}^t \vec {x}^t_i), &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$ (11) 5 experimental simulations this section presents tests performed with the two new algorithms side-by-side with their original counterparts in two scenarios. firstly, the swarm diversity and the behavior of the convergence curve of the best solution are analyzed with unimodal and multimodal test functions found in the literature. subsequently, tests are performed using a set of benchmark functions from congress on evolutionary computation (cec) 2017 [1], whose functions were randomly rotated or dislocated. the final results are evaluated considering time, number of function evaluations (nfes), and the minimal error rate—the difference between the solution found and the best one (i.e., \(f(\vec {x})-f(\vec {x}_{opt})\)). lastly, statistical tests are carried out to investigate how significant the differences are between algorithms. all results are the mean of 50 executions. it is noteworthy that this work does not aim to solve the cec 2017 functions, but rather to use them as a means of proving the concepts previously exposed. 5.1 swarm diversity and solution convergence five classical test functions found in the literature were selected to analyze behavioral swarm changes during the search: a unimodal one (sphere), a non-convex unimodal one (rosenbrock), and three multimodal functions (viz., ackley, griewank, and rastrigin). in these tests, d was set to 30, n was set to 20 following [14], and t was set to 6000 following [21]. existing algorithms were set with the hyper-parameters that were recommended in their respective studies: ri-ar-xpso used the same settings as xpso and ri-saxpso received a value range for each coefficient guided by [15]. specific hyper-parameter settings are presented in table 1. the results represent the average of 50 independent executions where the swarm diversity and the convergence curve of the best solution (in logarithmic scale) were evaluated. the only stopping criteria used was the maximum number of iterations t. table 1. parameter settings utilized in the tested algorithms.full size table fig. 1. swarm diversity and best solution convergence curve. full size image figure 1 shows the performance of the modified and original algorithms. it can be seen that ri-saxpso quickly converges while maintaining high swarm diversity at the beginning of the search. it attains better results than ri-sapso, which only achieves similar results in the ackley function (fig. 1c) despite maintaining swarm diversity throughout the search. xpso did not lose diversity quickly, but it was lower at the beginning of the search when compared to ri-ar-xpso. in terms of solution convergence, xpso best refines the final solution of sphere (fig. 1a), and loses diversity faster. xpso solutions converge with inferior quality when compared to ri-ar-xpso in the multimodal functions. 5.2 congress on evolutionary computation 2017 benchmark functions the cec 2017 benchmark is composed of 30 functions, of which only sum of different power is not used in this work due to its unstable behavior in higher dimensionalities [1]. hence, a total of 29 functions were used to test the algorithms: unimodal ones \((f1-2)\), simple multimodal ones \((f3-9)\), hybrid ones \((f10-19)\), and composed ones \((f20-29)\). the same settings found in sect. 5.1 were used herein except for the stopping criteria: in addition to using \(t = 6000\), the search is interrupted whenever the global minima is found or nfe = \(10^6\), which is the upper limit defined in [1]. the d of each function was set to 10, 30, and 50 to evaluate performance in different dimensionalities. table 2. average error rates and algorithm ranks.full size table table 3. total sum of the rankings, detailed by dimensionality.full size table table 2 shows average error rates accompanied by rank values (from 1 to 4). overall, ri-ar-xpso attained the lowest error rates on most optimization problems. despite the simplicity of this evaluation, table 3 presents the rank-sum stratified by d. it shows that the proposed implementations surpassed the original ones overall, although ri-saxpso did not overcome xpso. table 4 presents average computation times and the nfes of each algorithm stratified by dimensionality. one might notice how ri-sapso was the worst, with higher execution time and nfes. despite the high number of nfes, employment of rotation matrix, diversity control, gradient information, and automatic hyper-parameter adjustment during the execution, ri-saxpso performed best overall in terms of time. however, it took longer than xpso and ri-ar-xpso for 50 dimensions, indicating that using gradient information in high dimensionalities results in lower efficiency. the performance of xpso attests that using two exemplars, the ability to forget, and the velocity update equation postpones the execution. in spite of using these techniques, ri-ar-xpso forgets particles with less frequency thanks to the other properties involved, consequently attaining similar performance to xpso despite the presence of diversity control. in terms of success rate, the evaluated algorithms never reached the global optima when optimizing the evaluated functions for 30 and 50 dimensions. when \(d = 10\), xpso obtained a global success rate of \(8.07\%\), followed by ri-ar-xpso with \(1.86\%\), ri-saxpso with \(0.07\%\), and ri-sapso with \(0\%\). therefore, it can be argued that xpso finishes its execution faster than ri-ar-xpso thanks to its higher success rate for 10 dimensions. table 4. time (in seconds) and nfes.full size table 5.3 statistical hypothesis testing this section describes the friedman’s statistical significance test, as well as the post-hoc tests performed to demonstrate which differences are statistically significant in terms of the error rates obtained in table 2. friedman’s test with significance level \(\alpha = 0.05\) was applied for multiple comparisons. in this work, each pair of algorithms is analyzed \((k \times k)\) considering the four pso implementations \((k = 4)\) and a family of different hypotheses \((h = k(k-1)/2 = 6)\). the null hypothesis of this experiment is that there is no significant difference in the results. the post-hoc procedure was applied to find which algorithm pairs attained significantly different results, returning a p-value for each comparison to determine the degree of rejection of each hypothesis. furthermore, the p-values are adjusted due to the family-wise error rate (fwer), which is the probability of making one or more false discoveries between all hypotheses by performing multiple pairwise tests [16]. to calculate friedman’s statistics, the results seen in table 2 are the rankings, and table 5 highlights the mean ranks of each pso implementation obtained by applying a ranking-based classification. the results obtained attest that the best algorithm among the four implementations is ri-ar-xpso. upon application of friedman’s test, a p-value of \(1.085e{-}32\) and a friedman’s statistic of 151.79 were found. since \(p \ll \alpha \), there seems to be at least one significant difference. table 5. average ranksfull size table the evidence of statistically significant differences prompted the execution of post-hoc tests to pinpoint which algorithm pairs obtained significantly different results. p-values of all six hypotheses were obtained by converting the mean ranks of each algorithm by a normal approximation. z-values were used to find the corresponding p-value of the standard normal distribution \(\mathcal {n}(0, 1)\). friedman’s z-value can be defined as $$\begin{aligned} z_{ij}=\frac{r_i-r_j}{\sqrt{\frac{k(k+1)}{6n}}}. \end{aligned}$$ (12) table 6. friedman’s p-values and adjusted p-values (holm) for pairwise comparison tests between all algorithms.full size table the p-values found using (12) are in table 6. as previously explained, these values are not suitable for direct comparison with the significance levels due to fwer. thus, the adjusted p-values were used to directly compare the values with the significance levels. in this paper, an adjust procedure termed holm [3] was employed. in table 6, \(*\) indicates which algorithm performed better. in harmony with the results previously obtained, ri-ar-xpso obtained the best results, and ri-saxpso only outperformed ri-sapso. all p-values are smaller than 0.05, meaning that all differences are statistically significant. 6 final considerations this work presented two new pso implementations with conveniences such as automatic hyper-parameter update during execution and employment of a rotation matrix to promote independence from the coordinate systems wherein the objective function is defined. both ri-ar-xpso and ri-saxpso boast an attraction/repulsion mechanism guided by a diversity control whose effect is exponentially reduced throughout the search, resulting in more exploration at the beginning and more exploitation at the end. both algorithms were compared against their original counterparts in a varied set of problems and dimensionalities, and statistical significance tests indicated their superiority against the original implementations. ri-saxpso, which uses gradient-based information, consumed less time in problems with smaller dimensionalities. in these cases and those where the objective function is not very costly, this technique might be convenient. in future work, new arrangements can be tested to use the gradient information more efficiently since the gradient-based techniques did not reach good performance. new approaches that consider swarm diversity information to automatically adjust hyper-parameters during execution can also be studied. the goal would be to adapt the hyper-parameter fine-tuning according to current swarm diversity status instead of just verifying whether the search is stagnated. references awad, n., ali, m., liang, j.j., qu, b., suganthan, p.: problem definitions and evaluation criteria for the cec 2017 special session and competition on single objective real-parameter numerical optimization. technical report (2016) google scholar  bonyadi, m.r., michalewicz, z.: a locally convergent rotationally invariant particle swarm optimization algorithm. swarm intell. 8(3), 159–198 (2014) article  google scholar  derrac, j., garcía, s., molina, d., herrera, f.: a practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. sec 1(1), 3–18 (2011) google scholar  drake, j.h., kheiri, a., özcan, e., burke, e.k.: recent advances in selection hyper-heuristics. eur. j. oper. res. 285(2), 405–428 (2020) article  mathscinet  google scholar  eberhart, r., kennedy, j.: particle swarm optimization. in: proceedings of the ieee international conference on neural networks, vol. 4, pp. 1942–1948 (1995) google scholar  han, f., liu, q.: a diversity-guided hybrid particle swarm optimization based on gradient search. neurocomputing 137, 234–240 (2014) article  google scholar  hansen, n., ros, r., mauny, n., schoenauer, m., auger, a.: impacts of invariance in search: when cma-es and pso face ill-conditioned and non-separable problems. appl. soft comput. 11(8), 5755–5769 (2011) article  google scholar  horn, r.a., johnson, c.r.: matrix analysis. cambridge university press, cambridge (2012) google scholar  horst, r., tuy, h.: global optimization: deterministic approaches. sbh (1996) google scholar  janson, s., middendorf, m.: on trajectories of particles in pso. in: 2007 ieee swarm intelligence symposium, pp. 150–155. ieee (2007) google scholar  liu, x.f., zhan, z.h., gao, y., zhang, j., kwong, s., zhang, j.: coevolutionary particle swarm optimization with bottleneck objective learning strategy for many-objective optimization. ieee tec 23(4), 587–602 (2019) google scholar  noel, m.m.: a new gradient based particle swarm optimization algorithm for accurate computation of global minimum. asc 12(1), 353–359 (2012) google scholar  riget, j., vesterstrøm, j.s.: a diversity-guided particle swarm optimizer-the arpso. department of computer science, univ. of aarhus, aarhus, denmark, technical report 2 (2002) google scholar  santos, r., borges, g., santos, a., silva, m., sales, c., costa, j.c.a.: a rotationally invariant semi-autonomous particle swarm optimizer with directional diversity. swarm evol. comput. 56, 100700 (2020) article  google scholar  santos, r., borges, g., santos, a., silva, m., sales, c., costa, j.c.: a semi-autonomous particle swarm optimizer based on gradient information and diversity control for global optimization. appl. soft comput. 69, 330–343 (2018) article  google scholar  santos, r., borges, g., santos, a., silva, m., sales, c., costa, j.c.: empirical study on rotation and information exchange in particle swarm optimization. in: sec (2019) google scholar  shi, y., eberhart, r.: a modified particle swarm optimizer. in: 1998 ieee international conference on evolutionary computation proceedings (1998) google scholar  spears, w., green, d., spears, d.: biases in particle swarm optimization. ijsir 1, 34–57 (2010) google scholar  wilke, d.n., kok, s., groenwold, a.a.: comparison of linear and classical velocity update rules in particle swarm optimization: notes on diversity. in: ijnme (2007) google scholar  wilke, d.n., kok, s., groenwold, a.a.: comparison of linear and classical velocity update rules in particle swarm optimization: notes on scale and frame invariance. int. j. numer. meth. eng. 70(8), 985–1008 (2007) article  mathscinet  google scholar  xia, x., et al.: an expanded particle swarm optimization based on multi-exemplar and forgetting ability. inf. sci. 508, 105–120 (2020) article  mathscinet  google scholar  download references author information authors and affiliations federal university of pará, 1 augusto corrêa st., belém, pa, 66.075-110, brazil matheus vasconcelos, caio flexa, igor moreira, reginaldo santos & claudomiro sales authors matheus vasconcelosview author publications search author on:pubmed google scholar caio flexaview author publications search author on:pubmed google scholar igor moreiraview author publications search author on:pubmed google scholar reginaldo santosview author publications search author on:pubmed google scholar claudomiro salesview author publications search author on:pubmed google scholar corresponding author correspondence to matheus vasconcelos . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper vasconcelos, m., flexa, c., moreira, i., santos, r., sales, c. (2021). improving particle swarm optimization with self-adaptive parameters, rotational invariance, and diversity control. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_15 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords particle swarm optimization global continuous optimization adaptive adjustment rotational invariance diversity control publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating sentiment quantification methods in brazilian portuguese corpora | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating sentiment quantification methods in brazilian portuguese corpora conference paper first online: 30 january 2025 pp 230–244 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) evaluating sentiment quantification methods in brazilian portuguese corpora download book pdf download book epub lucas nildaimon dos santos silva  orcid: orcid.org/0000-0002-5606-94169, diego furtado silva  orcid: orcid.org/0000-0002-5184-941310 & helena de medeiros caseli  orcid: orcid.org/0000-0003-3996-85999  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 398 accesses abstract this paper evaluates sentiment quantification methods applied to brazilian portuguese corpora. sentiment quantification, distinct from sentiment classification, estimates the distribution of sentiment classes (positive and negative) within a dataset. we investigate several quantification techniques, including the family classify and count (cc) and more sophisticated methods, such as kernel density estimation (kde) and distribution y-similarity (dys). our analysis uses five datasets, each containing different distributions of sentiment classes. our experimental results indicate that kde and dys methods consistently outperform others, achieving the best average ranks in terms of quantification accuracy. statistical tests, including the friedman and nemenyi tests, confirm significant performance differences among the methods, with kde and dys showing statistically significant improvements over the baseline cc method. these findings highlight the importance of choosing robust quantification techniques for accurate sentiment quantification in corpora across different domains. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others evaluating methods of different paradigms for subjectivity classification in portuguese chapter © 2020 ss-pt: a stance and sentiment data set from portuguese quoted tweets chapter © 2022 an exploratory analysis on a disinformation dataset chapter © 2021 1 introduction understanding the overall sentiment toward an entity is crucial for businesses, governments, public figures, and organizations. in today’s data-driven world, accurately gauging public opinion helps make informed decisions, plan strategic actions, and improve public relations. with the vast amount of data generated from social media, reviews, and other digital platforms, quantifying sentiments has become essential. effective sentiment quantification allows for the evaluation of overall sentiment, identification of trends, monitoring of changes in public perception over time, and proactive response to potential issues. knowing the public’s opinion in summary form enhances the effectiveness of these actions, as analyzing each individual opinion or comment is often impractical and inefficient. sentiment quantification consists of estimating the relative prevalence (or distribution) of different sentiment classes (such as positive, neutral, and negative) within a sample of unlabeled texts [8, 22]. the usual procedure for quantifying the classes of a dataset relies on adjusting the predictions made by a model induced by supervised learning. in contrast to sentiment (polarity) classification, sentiment quantification seeks to understand the overall distribution of sentiments within a dataset. classification focuses on item-level analysis, while quantification provides an overview of sentiment trends. for instance, as illustrated in fig. 1, instead of identifying each product review as positive, negative, or neutral, sentiment quantification aims to estimate the rate of reviews that fall into each class within the dataset using quantification methods. fig. 1. illustration of a quantification task versus a classification task with two classes. full size image this distinction is fundamental for applications where the overall sentiment share is more informative than the sentiment of specific instances, such as in market analysis, political sentiment monitoring, and large-scale customer feedback analysis [5]. the most straightforward approach to quantification is the classify and count (cc) strategy [10], which involves training a classifier to label each instance and then counting these labels to estimate class prevalence. however, most classifiers based on supervised learning assume that the distribution of the classes in the training set is identical to that in the test set, an assumption known as the independent and identically distributed (iid) assumption [21]. in real-world scenarios, however, this assumption often does not hold, leading cc to inaccurate estimations of class prevalence when there are shifts in data distribution, also known as prior probability shifts. thus, there are significant biases in class prevalence estimation when using the cc strategy [19, 22]. these limitations of the cc strategy have led to the development of quantification methods that consider prior probability shifts, such as adjusted classify and count, saerens-latinne-decaestecker, and methods based on distribution matching, which are investigated in this article. quantification methods are important because they address the limitations of traditional classification methods when dealing with changes in data distribution, such as class distribution shifts [5]. in many real-world applications, the class distribution in training data may not accurately reflect the distribution in unlabeled data. by focusing on estimating class distributions rather than classifying individual instances, quantification provides a more robust and accurate approach to data analysis in scenarios characterized by distributional shifts [12, 22]. this work presents a comparative evaluation of quantification methods in the context of sentiment analysis in brazilian portuguese texts. our objective is to investigate the effectiveness of various quantification methods in estimating sentiment distribution across different datasets, including tweets and product reviews. the main contributions of our work are: we provide a comparative evaluation of 11 quantification algorithms in the context of sentiment analysis in brazilian portuguese texts, covering diverse datasets including tweets and product reviews. we systematically investigate the robustness and performance of these quantification methods under varying test sizes and degrees of shift in sentiment prevalence. we offer insights into the strengths and weaknesses of different quantification methods, particularly highlighting their effectiveness in addressing prior probability shifts in sentiment quantification. 2 related works the related works about quantification methods, dataset shift, and their applications cover various aspects of these topics, including proposals of algorithms, definition of experimental setups, and evaluation methodologies [4, 8,9,10,11,12, 16, 17, 21, 22, 26]. this section reviews significant contributions and methods from several key studies. sentiment quantification (sq) was initially introduced by [8]. the authors criticized the oversight in deciding whether sentiment analysis of large text collections should focus on individual or aggregate levels. they also emphasize the significance of distinguishing between sentiment classification and sq, recognizing them as two separate applications with distinctive characteristics. as a consequence, each of these tasks requires specific approaches. the authors claim that assuming an improvement in a classifier’s accuracy at the individual level would necessarily lead to higher accuracy at the aggregate level is not valid. they illustrate this point by highlighting how the f1 score, a widely used measure for classification evaluation, can be misleading. while a model might achieve a superior f1 score compared to another classifier, it could achieve inferior quantification performance. to illustrate, in a binary classification task, if classifier c1 has 20 errors (10 false positives and 10 false negatives) and classifier c2 has 10 errors (8 false positives and 2 false negatives), c1 would be considered worse as a classifier. however, c1 proves to be a better quantifier than c2 because the false positives and false negatives are balanced, compensating each other in terms of class distribution estimation. the cc method exemplifies this principle in its approach. a comprehensive empirical evaluation of quantification methods is presented in [26]. this study evaluates 24 quantification methods across over 40 datasets, of which 3 are text data, encompassing binary and multiclass configurations. in both binary and multiclass settings, the study examined splits with varying proportions of training and test data samples. this approach simulated scenarios with both limited and abundant information available for training the models. the findings reveal that no single algorithm consistently outperforms others across all configurations. however, a group of methods, including the median sweep and tsmax methods based on threshold selection, the dys framework, and the friedman method, exhibited the best performance in the binary scenario. the best performances in the multiclass scenario were achieved by the generalized probabilistic adjusted count, the readme method, the energy distance minimization method, the em algorithm for quantification, and the friedman method. the study reveals that the multiclass setting poses a considerably greater challenge for established quantification methods, as evidenced by error scores that are consistently much higher than those in the binary case. furthermore, it is shown that algorithms following the classify-and-count principle, even when optimized for quantification, tend to perform worse on average compared to other specialized methods. the study by [12] examines the performance of existing quantification methods under various types of dataset shifts, as most previous research has primarily focused on prior probability shifts. the authors propose a taxonomy of dataset shift types and introduce experimental protocols to simulate these shifts. by testing existing quantification methods on datasets generated using these protocols, the paper aims to identify the strengths and weaknesses of these methods under different conditions. a key finding is that while many quantification methods are robust to prior probability shifts, they struggle with other types of dataset shifts. the authors introduce new evaluation protocols to simulate various dataset shifts and test several quantification methods under these conditions. furthermore, they reveal that methods like pcc are only effective under pure covariate shifts, whereas sld and pacc perform better when covariate shifts are accompanied by changes in class priors. however, all methods, including sld, show instability under local covariate shifts and significant limitations in handling concept shifts. the study underscores the need for more effective quantification methods capable of addressing various types of dataset shifts. the importance of test set size in quantification research has also been highlighted by recent studies. according to [17], the test set size is a crucial, yet often overlooked, factor in quantification research. through empirical analysis, the authors demonstrate that the performance of quantifiers fluctuates significantly with varying test set sizes and that current methods generally perform poorly on smaller test sets. to address this, they propose a meta-learning scheme that selects the best quantifier based on the size of the test set. the authors advocate for future research to incorporate test set size considerations when evaluating new quantification proposals. the article by [23] evaluates different quantification methods applied to sentiment data from product reviews in english and explores how quantification can enhance the accuracy of sentiment classification. the authors used six product review datasets, a pre-trained language model (twitter-roberta-base), and ten quantification methods. the artificial prevalence protocol (app) was employed to generate test samples with varying class distributions to assess the error of the quantification methods and their impact on dynamic threshold adjustment for classification. the results demonstrated that eight of the nine quantification methods significantly outperformed the cc method in quantification tasks. additionally, using quantification methods to adjust the decision threshold significantly improved classification accuracy compared to the cc method. in contrast to the existing body of work, our study focuses specifically on the application of sentiment quantification methods to brazilian portuguese texts, a research problem that has received less attention. by systematically evaluating 11 quantification algorithms on five datasets of tweets and product reviews in brazilian portuguese, our research aims to provide a nuanced understanding of the effectiveness of these methods in this language. 3 methodology in this section, we detail the methodology employed to evaluate the effectiveness of various sentiment quantification methods on brazilian portuguese texts. we outline the datasets used and provide an overview of the quantification methods tested. additionally, we explain the experimental setup, which includes multiple test sizes, k-fold cross-validation, a quantification-specific evaluation protocol, and the metric used to assess each method’s performance. 3.1 datasets we used five datasets with texts written in brazilian portuguese annotated with sentiment polarity to evaluate the quantification methods. the selected datasets encompass a variety of corpora used for sentiment analysis in brazilian portuguese. the “computer-br” dataset [18] consists of tweets related to computers and notebooks with four possible classes: irony, negative, neutral, and positive. the “books” dataset contains book reviews manually annotated for positive and negative classes [1]. the “sentencas” dataset [1] comprises manually labeled sentences from electronic product reviews with negative and positive classes. the “eleicoes2018” dataset [3] includes tweets related to the 2018 elections in brazil, manually annotated with positive and negative polarities. the “repro” dataset [25] consists of product reviews annotated by sentiment with four classes: positive, negative, neutral, and an ambiguous class “negative/positive.” this dataset also includes reviews topics annotation, which we disregard here. in the experiments described in this paper, we only consider the binary tasks of sentiment quantification, excluding samples with labels different from the negative and positive classes from the corpora. the resulting class distributions vary across datasets, reflecting different proportions of positive and negative sentiments in each corpus, as shown in table 1. table 1. summary of datasets.full size table 3.2 quantification methods this study focuses on aggregative-based quantification methods, which estimate class prevalence using the output of either hard or probabilistic classifiers. the work by [11] classified quantification methods into three groups: (i) classify, count, and correct, which involves classifying instances and then correcting class counts; (ii) adapting traditional classification algorithms to function as quantifiers; and (iii) distribution matching, which models the training distribution and finds the best match against the test set. the most basic approach is the cc method. this involves training a standard classifier to assign each data sample to a specific class and then counting the predicted classes to assess their distribution. however, cc often fails to estimate class prevalence accurately when there are shifts in data distribution, also known as prior probability shifts. a variant of cc is the probabilistic classify and count (pcc) [2], which uses probabilistic classifiers to estimate class prevalence. unlike cc, which relies on hard classifications, pcc considers the posterior probabilities assigned to each class by the classifier, aiming for more precise prevalence estimates. the adjusted classify and count (acc) [10] further refines the cc approach by correcting the raw counts based on the true positive rate (tpr) and false positive rate (fpr) estimated in a validation set, thus offering a more balanced estimation. the probabilistic adjusted classify and count (pacc) [2] extends acc by integrating probabilistic classifiers. threshold selection techniques such as median sweep (ms) [9, 10], ms2 [9, 10], and max [9, 10] leverage the acc method to enhance class prevalence estimation, addressing stability concerns that arise when tpr and fpr are closely aligned, particularly in scenarios where the positive class is rare. these methods employ various thresholds to adjust the classifier’s decision boundary for more accurate prevalence estimation, each following strategies tailored for optimal performance. the ms method computes the median of prevalence estimates obtained by applying the acc method across a range of classification thresholds, using cross-validation to estimate tpr and fpr, followed by the acc correction. ms2, a stricter variant of ms, focuses specifically on thresholds where the difference between tpr and fpr exceeds 0.25, enhancing accuracy by filtering out potential outliers. conversely, the max method selects a threshold that maximizes the difference between tpr and fpr, optimizing the classifier’s decision boundary within the acc framework. all previously described methods are examples of the classify, count, and correct family. an adaptation of a traditional classification algorithm to function as a quantifier, svm(mae) [7, 22] is a support vector machine variant designed to minimize mean absolute error (mae) by explicitly focusing on the measure of error used to evaluate quantification accuracy. this method ensures the learning algorithm targets and reduces the specified error measure. svm(mae) is an instance of the svmperf framework [14], which can produce classifiers optimized for multivariate loss functions. the saerens-latinne-decaestecker (sldc) [24] algorithm also adapts a classifier by employing an expectation-maximization (em) approach with a transductive component to refine test predictions. this iterative process updates class prevalence estimates and posterior probabilities by leveraging labeled and unlabeled data. initially, the algorithm uses a classifier trained on labeled data to estimate posterior probabilities for the test set. in the e-step, these probabilities are used to update the prevalence estimates of each class. during the m-step, the updated prevalences refine the posterior probabilities. this process repeats until convergence. distribution matching techniques such as hdy [13], dys [16], and kdey-ml [20], operate by modeling the distribution of the training set and adjusting parameters to align this distribution with that of the test set. hdy utilizes the hellinger distance to compare probability score distributions between the training and test sets. the hellinger distance measures the difference between two probability distributions, providing a robust metric for comparing distributions in a high-dimensional space. hdy works by calculating the hellinger distance for each class, allowing it to quantify the degree of distribution shift and adjust the class prevalences accordingly. dys extends hdy by incorporating various distance functions to refine the comparison. in addition to the hellinger distance, dys might use metrics such as the jensen-shannon divergence, bhattacharyya distance, or earth mover’s distance, among others. by leveraging multiple distance functions, dys can capture different aspects of the distributional changes between the training and test sets. kdey-ml approaches the problem by modeling distributions using gaussian mixture models (gmms) and optimizing within the maximum likelihood framework. this method involves fitting gmms to the training data to model its distribution as a mixture of gaussian components. kdey-ml then aims to minimize the kullback-leibler (kl) divergence between the test data probability density and the training data mixture density. 3.3 quantification evaluation protocol evaluating quantification methods presents a challenge due to the disparity between classification and quantification tasks. in a classification task, a dataset with n data points results in n test data points. however, the same dataset only provides 1 test data point in a quantification task, as observed in fig. 1. this discrepancy arises because the quantification task aims to estimate the overall distribution in a batch of data, resulting in a single prediction for the entire batch, unlike classification, which involves n predictions. various evaluation protocols for quantification have been proposed to address this challenge, such as the artificial prevalence protocol (app) [24]. app is a widely used evaluation protocol for assessing quantification algorithms. it involves extracting multiple samples from a test dataset with controlled prevalence values and simulating scenarios where class prevalences differ between training and test sets to evaluate the robustness of quantifiers to prior probability shifts. the steps of app include extracting samples with predefined prevalence values, generating test samples by subsampling the positive class, applying the quantifier to estimate class prevalence, and comparing estimated prevalences with true prevalences. a more recent protocol, the uniform prevalence protocol (upp) [6], represents a modification of app designed to generate artificial samples with diverse class prevalence values. unlike app, upp does not depend on a predefined set of class prevalence values; instead, it employs kraemer’s algorithm [27] to allow these values to vary randomly. this flexibility enables upp to offer several advantages over app, such as permitting users to specify the desired number of samples and facilitating the selection of any conceivable distribution vector. these capabilities are crucial for scenarios where the distribution of class prevalences is complex and not easily captured by a fixed grid of values. to conduct our experiments, we chose the upp. 3.4 quantification evaluation metric to evaluate and compare our methods, we employ the absolute error (ae), a common measure used in quantification evaluation. this metric calculates the absolute difference between the estimated prevalence and the true prevalence, providing a quantitative assessment of accuracy for the prevalence estimates obtained from the quantification methods. the formula for ae is given by: $$\begin{aligned} \text {ae}(p, \hat{p}) = \frac{1}{|y|} \sum _{y \in y} | \hat{p}(y) p(y) | \end{aligned}$$ (1) where \( p \) represents the true class prevalence, \( \hat{p} \) represents the estimated class prevalence, and \( y \) is the set of classes of interest. the mean absolute error (mae) provides a straightforward assessment of the overall accuracy of prevalence estimates, it simply returns the average absolute error between the estimated and true prevalence values for all classes of interest. 4 experimental setup we selected an l2-regularized logistic regression (lr) classifier as the underlying model for our quantification methods, except for svm(mae), which uses svmperf. logistic regression classifiers offer well-calibrated posterior probabilities, essential for methods like pcc, pacc, dys, and sld [12, 22], and it is widely used in the quantification literature [4, 12, 20, 22, 26]. we begin with text preprocessing, which includes converting text to lowercase, removing accents, and eliminating punctuation and special characters. feature extraction is performed using tf-idf, discarding features that appear in fewer than 5 training documents. the experiment is evaluated using k-fold cross-validation (kcv) with \( k = 3 \). for every unique tuple \((p(\text {positive}), p(\text {negative}))\) representing class prevalence values, where each class prevalence is uniformly drawn from the unit-simplex [27], we generate \(m\) random samples, each containing \(q\) documents. these samples are created to reflect the class prevalence values specified by the tuple. in these experiments, we set \(m = 1000\) and evaluated multiple values of \(q\) (20, 50, 100, 500) to consider the impact of test size on performance, as demonstrated in [17]. for each label \(y\) (denoting positive and negative), and for each sample, the extraction is performed using sampling without replacement if there are sufficient samples in the training set; otherwise, sampling is done with replacement. we perform hyperparameter optimization on the underlying classifiers, svmperf for the svm(mae) method, and logistic regression (lr) for all the others. as highlighted in previous research [21], this step is important for mitigating bias in the experimentation of aggregative quantification methods. the optimization must be conducted using a quantification-oriented loss rather than a classification-oriented loss [22]. therefore, we optimize the mean absolute error (mae) on a validation set composed of 30% of the complete training set. to execute this optimization, we apply the uniform prevalence protocol (upp) again. we extract \( m \) samples of \( q \) documents each for every combination of class prevalence values from the validation set of each fold in the k-fold cross-validation (kcv). the class prevalence values are uniformly selected from the unit-simplex. in this context, \( m' \) is a constant set to 200, and the value of \(q'\) varies along with \(q\). the optimization is conducted using the grid search method [15]. the parameters optimized for lr are the regularization strength \( c \) (ranging from \( 10^{-3} \) to \( 10^3 \)) and class weight settings (either ‘none’ or ‘balanced’). for svmperf, we optimize the regularization strength \( c \) over the same range. finally, the best model obtained through this process is re-trained on the complete training set and estimates the class prevalence values for the test set of the specified cross-validation fold. the evaluation metric used to assess the overall accuracy of the quantification methods is the mae. 5 results in this section, we present the results of our evaluation of various sentiment quantification methods applied to brazilian portuguese texts. we provide an analysis of the performance of each method across different datasets under varying test sizes and degrees of sentiment prevalence shifts. 5.1 test size we begin by observing the average results for all test sizes displayed in table 2. the results suggest that no method achieved the best performance across all datasets and test sizes. each method showed varying degrees of effectiveness depending on the specific dataset and test size conditions. table 2. average mae for each method across all test sizes.full size table we further examined the average performance of the sentiment quantification methods using the friedman test and the nemenyi test, as illustrated in the critical difference diagram in fig. 2. the friedman test yielded a p-value of 0.01, leading us to reject the null hypothesis that all algorithms have the same performance at the 5% significance level. this indicates statistically significant differences in the performance of the quantification methods evaluated. fig. 2. critical difference diagram of average score ranks for all test sizes. full size image the critical difference diagram from the nemenyi post-hoc test further elucidates these differences. it ranks the average performance scores of the methods, where lower ranks indicate better performance. from the diagram, we observe that the kde and dys methods achieved the best average ranks (3.2), indicating their superior overall performance across the datasets and test sizes. notably, they were the only quantification methods that achieved statistically significant differences from the baseline cc. the pacc method followed closely, demonstrating strong performance. methods such as ms, acc, and ms2 exhibited moderate performance. further down the performance scale, the sldc, hdy, and max methods, showed comparatively lower performance but were still effective in certain scenarios. the pcc and svm(mae) methods had worse average ranks, highlighting their relative underperformance. lastly, the cc method had the worst average rank, confirming its limited effectiveness in the contexts tested. the variability in method effectiveness highlights the importance of choosing the right quantification method based on specific dataset characteristics and test conditions. we observed different outcomes when analyzing the critical difference diagram for each test set size. there was no statistical difference between the methods’ performances for the smallest (20 samples) and the biggest (500 samples) test sizes. we could expect such behavior for the smallest test size, as evidenced by previous research [17], indicating that methods generally perform poorer on small sizes. as for the biggest size, an important factor is the original datasets’ sizes and the sampling strategy used to form the 500 documents with the upp. in this case, there was a need to apply sampling with replacement for 3 out of the 5 datasets evaluated (books, eleicoes2018, and sentencas), which implies oversampling the training and test datasets. specifically, sampling with replacement may result in certain examples being selected multiple times, which could lead to some biases, such as reducing the model’s ability to generalize to truly unseen data. we observed significant differences for test sizes of 50 and 100 samples, as illustrated in fig. 3. overall, the diagrams highlight that kde achieved the best performance for both test sizes and was the only method with statistically significant differences from the baseline cc. pacc and dys consistently performed well across both test sizes, while svm(mae) consistently ranked lower, surpassing only the baseline. fig. 3. critical difference diagrams of average score ranks for test sizes 50 and 100. full size image 5.2 distributional shift continuing our analysis, we examined the performance of our method concerning the intensity of distribution shifts between training and test sets. we categorized these shifts using the manhattan distance to measure the dissimilarity between the training and test distributions. we considered a shift major if the distance was 0.8 or higher, minor if it was less than 0.4, and medium otherwise. figure 4 displays the methods’ average mae across all datasets for each shift intensity. as expected, the higher the shift intensity, the worse the error becomes. figure 5 displays the ae distribution for each dataset and shift intensity to provide a more detailed understanding of the methods’ stability across shift intensities and dataset characteristics. some methods, such as the threshold selection methods ms and ms2, as well as pacc, dys, and kde, exhibit relatively stable performance across shift intensities. in contrast, methods such as the baseline cc, pcc, sldc, and svm(mae) displayed more unstable behavior, yielding considerably worse results as the distribution shift intensity increased. the patterns are consistent across all datasets except for repro, where the methods demonstrated superior stability and overall better performance. this difference may be attributed to data availability, as repro is the largest dataset with 7,576 samples. consequently, it does not require the oversampling previously discussed to generate all sample sizes. fig. 4. average mae for each method across datasets and shift intensities. full size image fig. 5. ae values across shift intensities for all datasets. full size image 6 conclusion this study has provided a comprehensive evaluation of various sentiment quantification methods applied to brazilian portuguese texts under different conditions, particularly focusing on varying test sizes and the intensity of distribution shifts between training and test sets. several findings emerged from our analysis. firstly, kde consistently exhibited superior performance across different test sizes, showing statistically significant improvements over the baseline cc. alongside kde, pacc and dys also performed well, demonstrating stability and effectiveness across various conditions. in contrast, svm(mae) consistently achieved poor results, performing only marginally better than the baseline. the impact of distribution shift intensity on the performance of quantification methods was significant. as expected, higher shift intensities led to increased errors for all methods. however, certain methods, such as the threshold selection methods (max, ms, ms2), pacc, and kde, displayed relatively stable performance, indicating their robustness to distribution shifts. conversely, methods such as the baseline cc, pcc, sldc, and svm(mae) exhibited greater instability and poorer performance as the shift intensity increased. the friedman test and nemenyi post-hoc test further confirmed the significant differences in performance among the methods. kde and dys were the only methods that consistently achieved statistically significant improvements over the baseline across all test sizes. looking forward, future research could investigate the integration of deep learning techniques, such as transformer models, into quantification methods to potentially yield significant performance improvements. understanding the trade-offs between quantification accuracy and computational efficiency will be crucial for deploying these methods in resource-constrained environments. moreover, conducting more comprehensive evaluations across a wider range of languages and domains will help generalize findings and validate the methods’ applicability. exploring semi-supervised and unsupervised methods could reduce the dependency on annotated data, making sentiment quantification more accessible and scalable across various domains and sample sizes. additionally, examining the impact of different types of data augmentation techniques on the performance of quantification methods could offer insights into improving generalization and robustness. in summary, our findings highlight the importance of selecting appropriate quantification methods based on the specific characteristics of the dataset and the anticipated distribution shift intensities. methods like kde and dys offer robust performance across various conditions, making them particularly suitable for practical applications where different intensities of distribution shifts are expected. references belisário, l.b., ferreira, l.g., pardo, t.a.s.: evaluating methods of different paradigms for subjectivity classification in portuguese. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) computational processing of the portuguese language, pp. 261–269. springer international publishing, cham (2020) chapter  math  google scholar  bella, a., ferri, c., hernández-orallo, j., ramirez-quintana, m.j.: quantification via probability estimators. in: 2010 ieee international conference on data mining, pp. 737–742. ieee (2010) google scholar  cristiani, a., lieira, d., camargo, h.: a sentiment analysis of brazilian elections tweets. in: anais do viii symposium on knowledge discovery, mining and learning, pp. 153–160. sbc (2020) google scholar  donyavi, z., serapio, a., batista, g.: mc-sq: a highly accurate ensemble for multi-class quantification. in: proceedings of the 2023 siam international conference on data mining (sdm), pp. 622–630. siam (2023) google scholar  esuli, a., fabris, a., moreo, a., sebastiani, f.: learning to quantify. springer nature (2023) google scholar  esuli, a., moreo, a., sebastiani, f.: lequa@ clef2022: learning to quantify. in: european conference on information retrieval, pp. 374–381. springer (2022) google scholar  esuli, a., sebastiani, f.: optimizing text quantifiers for multivariate loss functions. acm trans. knowl. discov. data (tkdd) 9(4), 1–27 (2015) article  math  google scholar  esuli, a., sebastiani, f., abbasi, a.: sentiment quantification. ieee intell. syst. 25(4), 72–75 (2010) article  math  google scholar  forman, g.: quantifying trends accurately despite classifier error and class imbalance. in: proceedings of the 12th acm sigkdd international conference on knowledge discovery and data mining, pp. 157–166 (2006) google scholar  forman, g.: quantifying counts and costs via classification. data min. knowl. disc. 17, 164–206 (2008) article  mathscinet  math  google scholar  gonzález, p., castaño, a., chawla, n.v., coz, j.j.d.: a review on quantification learning. acm comput. surv. (csur) 50(5), 1–40 (2017) article  math  google scholar  gonzález, p., moreo, a., sebastiani, f.: binary quantification and dataset shift: an experimental investigation. data mining and knowledge discovery, pp. 1–43 (2024) google scholar  gonzález-castro, v., alaiz-rodríguez, r., alegre, e.: class distribution estimation based on the hellinger distance. inf. sci. 218, 146–164 (2013) article  math  google scholar  joachims, t.: a support vector method for multivariate performance measures. in: proceedings of the 22nd international conference on machine learning, pp. 377–384 (2005) google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998). https://doi.org/10.1109/5.726791 article  math  google scholar  maletzke, a., dos reis, d., cherman, e., batista, g.: dys: a framework for mixture models in quantification. in: proceedings of the aaai conference on artificial intelligence. vol. 33, pp. 4552–4560 (2019) google scholar  maletzke, a.g., hassan, w., dos reis, d.m., batista, g.e.: the importance of the test set size in quantification assessment. in: ijcai, pp. 2640–2646 (2020) google scholar  moraes, s.m.w., santos, a.l.l., redecker, m., machado, r.m., meneguzzi, f.r.: comparing approaches to subjectivity classification: a study on portuguese tweets. in: silva, j., ribeiro, r., quaresma, p., adami, a., branco, a. (eds.) computational processing of the portuguese language, pp. 86–94. springer international publishing, cham (2016) math  google scholar  moreo, a., francisco, m., sebastiani, f.: multi-label quantification. acm trans. knowl. discov. data 18(1), 1–36 (2023) article  math  google scholar  moreo, a., gonzález, p., del coz, j.j.: kernel density estimation for multiclass quantification. arxiv preprint arxiv:2401.00490 (2023) moreo, a., sebastiani, f.: re-assessing the “classify and count” quantification method. in: advances in information retrieval: 43rd european conference on ir research, ecir 2021, virtual event, march 28–april 1, 2021, proceedings, part ii 43, pp. 75–91. springer (2021) google scholar  moreo, a., sebastiani, f.: tweet sentiment quantification: an experimental re-evaluation. plos one 17(9), e0263449 (2022) article  math  google scholar  ojeda, d.z., zalewski, w., maletzke, a.g.: utilizando a quantificação na análise de sentimentos em reviews de produtos. in: escola regional de banco de dados (erbd), pp. 71–80. sbc (2024) google scholar  saerens, m., latinne, p., decaestecker, c.: adjusting the outputs of a classifier to new a priori probabilities: a simple procedure. neural comput. 14(1), 21–41 (2002) article  math  google scholar  dos santos silva, l.n., et al.: repro: a benchmark for opinion mining for brazilian portuguese. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 432–440 (2024) google scholar  schumacher, t., strohmaier, m., lemmerich, f.: a comparative evaluation of quantification methods. arxiv preprint arxiv:2103.03223 (2021) smith, n.a., tromble, r.w.: sampling uniformly from the unit simplex. johns hopkins university, tech. rep 29 (2004) google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations graduate program in computer science (ppgcc), department of computing, federal university of são carlos (ufscar), são carlos, brazil lucas nildaimon dos santos silva & helena de medeiros caseli institute of mathematics and computer sciences (icmc), university of são paulo (usp), são carlos, brazil diego furtado silva authors lucas nildaimon dos santos silvaview author publications search author on:pubmed google scholar diego furtado silvaview author publications search author on:pubmed google scholar helena de medeiros caseliview author publications search author on:pubmed google scholar corresponding author correspondence to lucas nildaimon dos santos silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper dos santos silva, l.n., silva, d.f., de medeiros caseli, h. (2025). evaluating sentiment quantification methods in brazilian portuguese corpora. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_16 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords sentiment analysis quantification distribution shift brazilian portuguese publish with us policies and ethics profiles diego furtado silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora conference paper first online: 30 january 2025 pp 427–441 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora download book pdf download book epub alan melo9, bruno cabral  orcid: orcid.org/0000-0002-5221-28609 & daniela barreiro claro  orcid: orcid.org/0000-0001-8586-10429  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 395 accesses 1 citation abstract this paper comprehensively investigates the efficacy of different adaptation techniques for large language models (llms) in the context of open information extraction (openie) for portuguese. we compare full fine-tuning (fft) and low-rank adaptation (lora) across a model with 0.5b parameters. our study evaluates the impact of model size and adaptation method on openie performance, considering precision, recall, and f1 scores, as well as computational efficiency during training and inference phases. we contribute to a high-performing llm and novel insights into the trade-offs between model scale, adaptation technique, and cross-lingual transferability in the openie task. our findings reveal significant performance variations across different configurations, with lora demonstrating competitive results. we also analyze the linguistic nuances in the portuguese openie that pose challenges for models primarily trained on english data. this research advances our understanding of llm adaptation for specialized nlp tasks and provides practical guidelines for deploying these models in resource-constrained and multilingual scenarios. our work has implications for the broader cross-lingual open information extraction field and contributes to the ongoing discourse on efficient fine-tuning strategies for large pre-trained models. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an ensemble of llms finetuned with lora for ner in portuguese legal documents chapter © 2025 adapting llms for efficient, personalized information retrieval: methods and implications chapter © 2024 comprehensive evaluation of large language model responses: a multi-factor scoring system chapter © 2026 1 introduction open information extraction (openie) is an nlp task that extracts structured data from documents [2]. recently, such tasks have been inputted into different pipelines to facilitate the complexity of a set of nlp applications such as qa systems, mental maps, fake news approaches, etc. following the evolution of the large language model (llm) domain, the openie task has enlarged its approaches to employ llm architectures. as openie has seen significant advancements in the portuguese language in the last few years, the application of large language models (llms) still needs to be explored. as the evolution of llms increases the computational cost [10], advanced language model adaptation methods such as full fine-tuning (fft) and low-rank adaptation (lora) are essential to improve performance and efficiency. thus, the portuguese language benefits from such adaptation methods to evolve its nlp applications. this study aims to examine the potential of llms when applied to portuguese openie, evaluate the impact of model size and adaptation method, consider precision, recall, and f1 scores, and examine computational efficiency during training and inference phases. we contribute to a high-performing llm and novel insights into the trade-offs between model scale, adaptation technique, and cross-lingual transferability in the openie task for the portuguese language. our results suggest a significance performance between the techniques and provide insights into the practical implications of their application in the open information extraction task for the portuguese language. this work is organized as follows: sect. 2 describes the openie task, sect. 3 outlines the related work, and positions our approach in the state of the art. section 4 presents the foundations and adaptation techniques, sect. 5 describes our methodology, and sect. 6 presents our evaluations and the conclusion and future research directions following it. 2 open information extraction (openie) we introduce the definition of open information extraction (openie) and the model to assess the performance of large language models (llms) as triple extractors for the portuguese language. this section explores the definition and potential applications of openie, highlighting key contributions and ongoing challenges. 2.1 openie definition let \( x = \{ x_{1}, x_{2}, \cdots , x_{n} \} \) be a sentence composed of tokens \(x_i\). an openie triple extractor is a function mapping x to a set \( y = \{ y_{1}, y_{2}, \cdots , y_{j} \} \), where each element is a tuple \(y_i = \{ rel_i, arg1_i, arg2_i\}\) that encapsulates the information conveyed in sentence x. we assume that tuples are always in the format \(y = (arg_{1}, rel, arg_{2})\), with arg1 and arg2 being noun phrases created from tokens in x, and rel representing a relation between \( arg_{1} \) and \( arg_{2} \). as it is common in the area, we do not consider extractions consisting of n-ary relations. 2.2 openie applications openie has been used in a pipeline to serve diverse applications but is not limited to them. knowledge graph: extracting structured information from text, openie facilitates the automatic creation of knowledge graphs, which are crucial for semantic search, question-answering systems, and decision support systems. text summatization: openie can help identify key facts and relationships in a text, helping to generate concise summaries. querying answering: triples extracted directly from questions in a qa system can provide factual answers or help increase implicit knowledge inference extraction. 2.3 openie challenges despite significant advances in open information extraction (openie), several challenges persist that hinder its broader applicability. extracting relationships involving complex entities or those expressed through intricate linguistic constructions remains challenging. furthermore, while openie strives to be domain-agnostic, its performance can still vary greatly across different text genres and domains such as legal and health, necessitating ongoing efforts in domain adaptation. as real-time information extraction becomes more prevalent, enhancing the scalability and computational efficiency of openie systems is critical. additionally, the presence of semantic ambiguities and contextual dependencies in texts poses substantial challenges to extraction accuracy, emphasizing the need for continued research in this area. openie continues to be a vibrant field of research with considerable potential to impact a wide range of applications, and advancements in machine learning—especially in deep and unsupervised learning—are expected to drive further improvements. 3 related work open information extraction (openie) has evolved significantly since its inception, transitioning from rule-based approaches to sophisticated machine learning techniques. this section explores the progression of openie systems, with a particular focus on recent advancements in large language models (llms) and their application to multilingual scenarios, especially portuguese. early openie systems relied heavily on syntactic parsing and hand-crafted rules. openie was introduced by banko et al. [2], which focused on symbolic approaches based on predefined rules to describe each noun and verbal phrase presented in the sentence. motivated by the distant relation between the noun phrase and its core, the dependency tree approach has emerged as a promising developed system as presented by del corro et al. [8]. these systems use predefined linguistic patterns to extract information. although they are transparent and easy to debug, their effectiveness is limited by the coverage and complexity of the rules. however, the field has since shifted towards machine learning approaches, which have demonstrated superior performance and adaptability [7, 30, 32, 38]. zhou and colleagues [39] categorized modern openie systems into two main types: sequence tagging models and generative models. sequence tagging models frame openie as a token labeling task, assigning tags to indicate the role of each word in a sentence (e.g., argument, predicate) [39]. these models typically incorporate token embedding, contextual encoders (such as bert [9]), and label decoders (often using conditional random fields [20]). notable works in this category include stanovsky et al. [30], who introduced a recurrent neural network (rnn) architecture for english openie. the lack of corpora has created a new wave to evolve resources, particularly those with the corpora for languages other than english [30]. as language models have seen significant usage employing neural networks, openie evolved in describing the task and generating triples as prompting, particularly with english languages. such systems can generalize from training data to extract triples from new texts. recent approaches have increasingly employed neural networks, particularly transformer models, which have significantly improved the performance of openie systems by capturing contextual and semantic nuances more effectively. generative approaches, on the other hand, model openie as a sequence generation problem [7]. these models often employ encoder-decoder architectures to produce a sequence of extractions. recent studies have integrated bert embeddings into generative models, as seen in openie6 and imojie [17, 18], which address the issue of redundant extractions in generative openie models. while most openie research has focused on english, there has been growing interest in developing multilingual and cross-lingual systems. zhang and colleagues [38] proposed a semi-supervised cross-lingual approach, while multi2oie [27] utilized m-bert for embedding and predicate extraction across multiple languages, including portuguese and spanish. for portuguese specifically, openie systems have progressed from rule-based dependency analysis [23] and linguistically-oriented patterns [28, 29] to supervised learning with deep neural networks. recent works like multi2oie [27] and portnoie [5] have shown significant improvements in f1 scores compared to earlier methods, highlighting the potential of neural network-based approaches for portuguese openie. this portuguese method empowers openie-pt for generative approaches layers [4]. the application of llms to openie is an emerging trend, although not yet widely adopted. there are instances of llm use in related fields such as question answering, relation extraction, and information extraction. xu and others [36] explored the application of an llm for few-shot relation extraction, while oppenlaender et al. [24] investigated llm use for question answering over large-scale text corpora with promising results. wei and colleagues [35] examined the use of llm systems for zero-shot information extraction, proposing to frame it as a multi-step question answering problem. kolluru et al. [19] investigated the use of language models, namely bert and mt5 [37], for a two-stage generative openie model that first identifies relations and then assembles extractions for each relation. recent research has explored various fine-tuning strategies for llms to adapt them to specific tasks efficiently. full fine-tuning (fft) involves updating all model parameters, which can be computationally expensive for large models [13]. in contrast, parameter-efficient fine-tuning (peft) methods aim to reduce the number of trainable parameters while maintaining performance [12]. low-rank adaptation (lora), introduced by hu and colleagues [14], is a peft method that has gained popularity due to its efficiency and effectiveness. lora adds trainable low-rank matrices to the attention layers of pre-trained models, significantly reducing the number of trainable parameters while achieving comparable performance to full fine-tuning in many tasks [14]. cross-lingual transfer learning has shown promise in adapting models trained on high-resource languages to low-resource languages [6]. recent studies have investigated the cross-lingual capabilities of llms and their potential for zero-shot and few-shot learning in multilingual settings [15, 21]. cabral et al. [4] introduced a lora finetuned llm for portuguese based on llama-2, achieving good results. in the context of openie, cross-lingual transfer could potentially leverage the abundance of english training data to improve performance on languages with fewer resources, such as portuguese. however, the effectiveness of such transfer learning approaches in openie tasks, particularly when using llms, remains an open challenge. our work builds upon these foundations, exploring the application of llms to portuguese openie through fine-tuning and lora perspectives. we investigate the trade-offs between model size, adaptation technique, and cross-lingual transferability, contributing to the ongoing discourse on efficient fine-tuning strategies for large pre-trained models in multilingual scenarios. 4 foundations of large language models and adaptation techniques this section explores the fundamentals and emerging technologies that form the backbone of modern natural language processing (nlp) systems. with a special focus on large language models (llms), this part of the paper details the technical and theoretical innovations that have driven advancements in the field, as well as strategic adaptations such as low-rank adaptation (lora) and fine-tuning, which enhance the functionality of these models in specific applications. the evolution of llms, from early n-gram based models to current transformer architectures, reflects significant progress in the ability to simulate human language comprehension and production, leading to notable improvements in efficiency and effectiveness across a variety of nlp tasks [3]. 4.1 large language models large language models are advanced machine learning algorithms designed to simulate the human capacity for understanding and producing natural language. these models are predominantly built using deep learning techniques and are trained on extensive textual corpora to capture the grammatical, lexical, and semantic complexities of language. historically, they have evolved from simple n-gram based models to sophisticated neural network architectures, such as recurrent neural networks (rnns) and more recently, transformers, which utilize attention mechanisms to enhance contextual understanding and text generation [34]. transformers, in particular, have revolutionized language modeling with their ability to process text sequences in parallel, resulting in significant gains in efficiency and effectiveness in nlp tasks. these models form the foundation for applications such as automated dialogue systems, open information extraction, machine translation, and others that require deep understanding and manipulation of human language [9]. ongoing research and development in this area aim not only to improve the accuracy of these models but also to make them more accessible and ethical in their use. recent advancements include the development of multilingual models capable of processing and generating text in multiple languages, which is particularly relevant for our study on portuguese open information extraction [6]. 4.2 low-rank adaptation (lora) lora is a model adaptation technique that leverages the principle of matrix decomposition to efficiently modify the weights of a pre-trained model, allowing for adaptation with significantly reduced computational cost. unlike traditional fine-tuning, which adjusts all model parameters, lora focuses on adapting a fraction of these parameters through the addition of low-rank projections [14]. at the core of lora is the idea that transformation matrices in language models, such as those found in the attention and feed-forward layers of transformers, can be approximated by products of lower-dimensional matrices. mathematically, this approach involves introducing two smaller matrices, a and b, where the product ab serves as a low-rank approximation for updating the original weight matrix, w. this product does not replace w but is added to it, allowing the original model to be extended with new learning capabilities without directly altering its pre-existing structure [14]. the primary advantage of using lora in language models is twofold: first, it reduces the number of parameters that need to be adjusted during adaptation, decreasing the demand for computational resources and training data. second, by keeping most of the original model structure unchanged, lora preserves the prior knowledge embedded in the model, minimizing the risk of catastrophic forgetting, a common problem in more invasive adaptations [16]. lora proves particularly useful in scenarios where computational resources are limited or when it is necessary to adapt models on devices with restricted processing capacity. for these reasons, lora is a strategic choice for adapting language models to specific tasks, maintaining a balance between effectiveness and efficiency [14]. 4.3 fine-tuning fine-tuning, in the context of large language models (llms), refers to the process of adjusting the pre-trained weights of a model on a specific dataset or task, with the aim of adapting the model to perform better in particular scenarios. this method involves continuing the training of the model from its initial pre-trained configuration, typically using a lower learning rate, to refine its parameters without losing the generalizations learned during extensive pre-training [13]. the fine-tuning process involves several critical steps: selection of a training dataset that is representative of the final task, choice of a learning rate that is sufficiently small to avoid losing useful information already acquired, and careful adjustment of the number of training epochs to prevent overfitting. the success of fine-tuning depends not only on the quality and size of the dataset but also on a good regularization strategy and monitoring of the model’s generalization [25]. the main advantage of fine-tuning is its ability to produce highly specialized models for specific tasks, which can result in a significant increase in performance compared to generic pre-trained models. however, this method also presents challenges, such as the need for large volumes of task-specific data and the risk of overfitting, especially in tasks with limited datasets [11]. for these reasons, fine-tuning is a powerful and widely used technique in adapting language models for natural language processing (nlp) applications, offering a flexible approach to personalizing artificial intelligence models across a variety of domains and tasks [31]. in the context of our study on portuguese open information extraction, we explore both lora and fine-tuning techniques to adapt large language models, comparing their effectiveness and efficiency in this specific cross-lingual task. this comparative analysis contributes to the ongoing discourse on efficient adaptation strategies for large pre-trained models in multilingual scenarios [1]. 5 methodology 5.1 corpus in our study, we employed the transalign corpus [26], specifically designed to enhance the availability of high-quality training data for open information extraction (openie) in under-resourced languages. the corpus, developed through a cross-lingual alignment of data from resource-rich languages like english to portuguese, comprises 96,067 high-quality triples. these triples are aligned to reflect brazilian portuguese grammatical structures, using advanced translation models and handcrafted rules. 5.2 pre-processing in the initial processing of the transalign corpus data, each triple and sentence was transformed into a format close to natural language, using a specific command system. this transformation aimed to prepare the data for more efficient manipulation in language models. chatml, a template format utilizing jinjafootnote 1 syntax, was employed to transform a list of chat messages into a formatted string that can be directly used by language models for training or inference. in the context of transalign, this template was adapted to organize the information from the triples (arg0, v, arg1) into a format simulating conversations, facilitating the training of models in openie tasks [33]. axolotlfootnote 2, a language model training platform, was used to train the models using the processed data. the choice of axolotl was due to its efficiency in managing and optimizing the training of large-scale language models, especially in configurations involving complex data adaptations, as is the case with multilingual openie [14]. after applying the chatml template, the data were converted to the jsonl format. this format is particularly useful for ai model training, as it allows each line of the file to contain a complete json object, representing a single data entry. this simplifies data loading and batch processing during training, contributing to the overall efficiency of the process. 5.3 hyperparameters we explored the training configurations of decoder-only language models, specifically focusing on full fine-tuning for the qwen2 0.5 model (0.5b parameters) and lora on the same model. the qwen2 0.5b model was selected due to its compatibility with the training dataset, which includes data in portuguese, and its smaller size allows for stable training with limited computational resources. the training process was standardized to ensure comparability across different model scales and techniques. in our experiments, we standardized the configurations for each model, setting the batch size to 64. we adjusted the gradient accumulation settings accordingly to maintain equivalent batch size efficiency between the two models. specifically, we used four accumulations for the lora model and eight accumulations for the full fine-tuning model. this difference in the number of micro-batches, with more frequent accumulations in the full fine-tuning model, was necessary for the lower gradient variability observed during training with lora. the learning rate was initialized at \(5 \times 10^{-5}\) and adjusted according to a cosine learning rate scheduler. our models underwent training over three epochs with a brief warm-up phase of 10 steps to stabilize the learning rate at the beginning of training. the loss function utilized was the unsloth cross entropy loss, which enhances the traditional cross-entropy calculation, offering refined loss management crucial for the stability and performance of model training. the optimizer of choice was the paged adamw 8bitfootnote 3, incorporating 8-bit quantization for optimizer states to reduce memory requirements significantly while preserving training efficacy. additional configurations included gradient checkpointing, a critical memory optimization technique that saves only selected intermediate states during the forward pass, thus reducing overall memory consumption. the models were trained using weights with bf16 precision, which strikes an optimal balance between computational performance and numerical precision. for the lora configuration, a rank of 64 was introduced, resulting in 35,192,832 trainable parameters. furthermore, the following target modules were selected: q_proj, k_proj, v_proj, o_proj, gate_proj, down_proj, up_proj. 5.4 experimental setup our experiments were conducted on a customer hardware platform consisting of an nvidia rtx 3060 gpu with 12gb vram, supported by 32gb of system ram. the software environment was based on ubuntu, using the axolotl training tool. the models were handled in the gguf f16 format, optimized for high-performance computing tasks, which facilitated efficient model loading and inference operations. for model interaction and manipulation, we employed the llama-cpp-pythonfootnote 4 library, which integrates seamlessly with python, enabling sophisticated data processing and model tuning capabilities. additionally, weights & biases (wandb) was used to collect system metrics, such as gpu power usage, and track training parameters, including loss, during the training process, allowing for the generation of insightful graphs and data collection. this comprehensive setup ensured that our experimental procedures were not only efficient but also reproducible. 5.5 experiments models were trained using the complete transalign corpus, with 1% reserved for validation. performance was evaluated using precision, recall, and f1-score metrics on the pud 100 dataset, which is highly annotated and reliable [22]. the evaluation code developed by stanovsky et al. (2018) was employed, based on a lexical matching metric to assess the accuracy of extracted triples [30]. results will be presented in table 1, including precision, recall, and f1 metrics for each evaluated model, allowing for a clear comparison of model performance under different configurations and adaptations. 6 evaluation this section outlines the methodology used to compare the two models. we assess quantitative differences employing metrics such as f1-score, precision, and recall. for qualitative analysis, we investigate the triples generated by both models to gauge their effectiveness in capturing pertinent information. all model outputs were produced under consistent generation settings: a temperature of 0.2, a top-p of 0.95, a min-p of 0.05, and a top-k of 40. these parameters were chosen to ensure the stability and relevance of the generated content. 6.1 quantitative analysis during the inference phase, the video memory consumption was 1.2 gb for both models, and the token generation rates were similar, at 2500 tokens per second. table 1. evaluation scoresfull size table during the training phase, the lora model consumed less energy than the full fine-tuning (fft) model. this is because lora reduces the number of trainable parameters by focusing only on smaller, low-rank matrices, leaving the majority of the pre-trained weights untouched. this approach significantly lowers the computational load, resulting in less memory usage and reduced energy consumption, compared to fft, which updates all model parameters, as illustrated in fig. 1. fig. 1. graph illustrating the comparative energy consumption of the lora and fft models during the training phase. full size image it was observed that the model trained with lora achieved better results across all metrics compared to the model that underwent full fine-tuning for openie task as it can be observed in fig. 2. the timing metrics from inference on the sentence “alan, who studies at ufba, is a member of formas,” reveal minimal differences between full fine-tuning (fft) and lora. fft took 228.14 ms, while lora slightly improved this to 226.55 ms. eval times were 4.89 ms per token for fft and 4.67 ms for lora. prompt eval and sample times were nearly identical for both methods. these minor differences, ranging from 0.17 ms to 5.36 ms, suggest that while lora is marginally faster, the impact on practical performance is negligible. fig. 2. graph depicting the differences in metrics between the validated models. full size image 6.2 qualitative analysis upon analyzing some of the triples generated by both models, a significant degradation in generation quality was noticeable in the model that underwent full fine-tuning, whereas the model trained with lora demonstrated greater resistance to hallucinations. the following extractions exemplify this: sentence: “ela começou no rsc no meio dos anos sessenta, atuando como internada de asilo no marat/sade.” fft: arg1: ani rel: começou em arg2: o rsc analysis of fft: the fft model hallucinates by incorrectly identifying “ani” as the subject of the action “começou em”. “ani” does not appear in the sentence, showing a clear error in entity recognition and relation mapping. lora: arg1: ela rel: começou em arg2: o rsc analysis of lora: the lora model accurately identifies “ela” as the subject who “começou em o rsc”, perfectly aligning with the content of the original sentence without adding or distorting information. sentence: “o ano passado foi um ano incrível e estou pronto para que possamos voltar ainda melhor em 2017.” fft: arg1: o ano passado rel: estou pronto para que possamos voltar arg2: ainda melhor em 2017 analysis of fft: in this extraction, the fft model creates a hallucinated relation by connecting “o ano passado” with “estou pronto para que possamos voltar”, which is not a logical or supported relation within the context of the sentence. lora: arg1: o ano rel: passado foi arg2: um ano incrível analysis of lora: the lora model performs a more accurate and textually grounded extraction, identifying that “o ano passado foi um ano incrível”, which is a direct and faithful interpretation of what is explicitly mentioned in the sentence. sentence: “após algumas cirurgias geniais e muito trabalho duro de reabil-itação, eu tive uma recuperação completa.” fft extraction: arg1: eu tive uma recuperação completa rel: inicialmente arg2: uma recuperação completa analysis of fft: the fft model incorrectly incorporates redundancy and an unsupported relation “inicialmente,” which distorts the meaning of the sentence. it also repeats “uma recuperação completa” for both subject and object, showing a segmentation error. lora extraction: arg1: eu rel: tive arg2: uma recuperação completa analysis of lora: the lora model’s extraction is precise and aligned with the sentence content, correctly identifying “eu” as the subject, “tive” as the verb indicating action, and “uma recuperação completa” as the object, efficiently capturing the sentence’s information structure. these examples illustrate how the lora model tends to produce extractions that are more faithful to the original text, avoiding adding non-existent information or distorting relationships between sentence elements, unlike the fft model, which showed a propensity for hallucinations possibly due to overfitting or insufficient generalization during training. 7 conclusions and future directions in this work, we evaluate the performance of large language models (llms) using full fine-tuning and lora adaptations, analyzing both quantitative and qualitative measures. we use metrics such as f1, precision, and recall to assess quantitative outcomes. qualitative analysis is conducted through example sets. our findings indicate that lora adaptations are as effective as full fine-tuning, with the added benefit of lower energy consumption and competitive results. we plan to conduct further experiments with a larger language model to deepen our understanding of llm behavior under these adaptations. notes 1.available at https://jinja.palletsprojects.com. 2.available at https://github.com/openaccess-ai-collective/axolotl. 3.https://github.com/timdettmers/bitsandbytes. 4.https://github.com/abetlen/llama-cpp-python. references artetxe, m., ruder, s., yogatama, d.: on the cross-lingual transferability of monolingual representations. in: jurafsky, d., chai, j., schluter, n., tetreault, j. (eds.) proceedings of the 58th annual meeting of the association for computational linguistics, pp. 4623–4637. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.acl-main.421. https://aclanthology.org/2020.acl-main.421 banko, m.: open information extraction for the web. ph.d. thesis, university of washington (2009) google scholar  brown, t.b., et al.: language models are few-shot learners (2020) google scholar  cabral, b., claro, d., souza, m.: exploring open information extraction for portuguese using large language models. in: gamallo, p., et al. (eds.) proceedings of the 16th international conference on computational processing of portuguese vol. 1, pp. 127–136. association for computational lingustics, santiago de compostela, galicia/spain (2024). https://aclanthology.org/2024.propor-1.13 cabral, b., souza, m., claro, d.b.: portnoie: a neural framework for open information extraction for the portuguese language. in: pinheiro, v., et al. (eds.) propor 2022. lncs (lnai), vol. 13208, pp. 243–255. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_23 chapter  math  google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale (2020). https://arxiv.org/abs/1911.02116 cui, l., wei, f., zhou, m.: neural open information extraction. arxiv preprint arxiv:1805.04270 (2018) del corro, l., gemulla, r.: clausie: clause-based open information extraction. in: proceedings of the 22nd international conference on world wide web, pp. 355–366. acm (2013) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) gozalo-brizuela, r., garrido-merchan, e.c.: chatgpt is not all you need. a state of the art review of large generative ai models (2023) google scholar  gururangan, s., et al.: don’t stop pretraining: adapt language models to domains and tasks. arxiv abs/2004.10964 (2020). https://api.semanticscholar.org/corpusid:216080466 houlsby, n., et al.: parameter-efficient transfer learning for nlp (2019). https://arxiv.org/abs/1902.00751 howard, j., ruder, s.: universal language model fine-tuning for text classification (2018). https://arxiv.org/abs/1801.06146 hu, e.j., et al.: lora: low-rank adaptation of large language models (2021). https://arxiv.org/abs/2106.09685 hu, j., ruder, s., siddhant, a., neubig, g., firat, o., johnson, m.: xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization (2020). https://arxiv.org/abs/2003.11080 kirkpatrick, j., et al.: overcoming catastrophic forgetting in neural networks. proc. natl. acad. sci. 114(13), 3521–3526 (2017). https://doi.org/10.1073/pnas.1611835114 article  mathscinet  math  google scholar  kolluru, k., adlakha, v., aggarwal, s., chakrabarti, s., et al.: openie6: iterative grid labeling and coordination analysis for open information extraction. arxiv preprint arxiv:2010.03147 (2020) kolluru, k., aggarwal, s., rathore, v., mausam, chakrabarti, s.: imojie: iterative memory-based joint open information extraction. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 5871–5886. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.acl-main.521. https://aclanthology.org/2020.acl-main.521 kolluru, k., mohammed, m., mittal, s., chakrabarti, s.: alignment-augmented consistent translation for multilingual open information extraction. in: proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 2502–2517. association for computational linguistics, dublin, ireland (2022). https://doi.org/10.18653/v1/2022.acl-long.179. https://aclanthology.org/2022.acl-long.179 lafferty, j.d., mccallum, a., pereira, f.: conditional random fields: probabilistic models for segmenting and labeling sequence data. in: international conference on machine learning (2001). https://api.semanticscholar.org/corpusid:219683473 lin, x.v., et al.: few-shot learning with multilingual language models (2022). https://arxiv.org/abs/2112.10668 nivre, j., et al.: universal dependencies v2: an evergrowing multilingual treebank collection. in: proceedings of the 12th language resources and evaluation conference, pp. 4034–4043. european language resources association, marseille, france (2020). https://www.aclweb.org/anthology/2020.lrec-1.497 oliveira, l., claro, d.b., souza, m.: dptoie: a portuguese open information extraction based on dependency analysis. artif. intell. rev. 56(7), 7015–7046 (2022). https://doi.org/10.1007/s10462-022-10349-4 oppenlaender, j., hämäläinen, j.: mapping the challenges of hci: an application and evaluation of chatgpt and gpt-4 for cost-efficient question answering (2023) google scholar  peters, m.e., ruder, s., smith, n.a.: to tune or not to tune? adapting pretrained representations to diverse tasks (2019). https://arxiv.org/abs/1903.05987 rios, a., cabral, b., claro, d., cavalcante, r., souza, m.: transalign: an automated corpus generation through cross-linguistic data alignment for open information extraction. in: gamallo, p., et al. (eds.) proceedings of the 16th international conference on computational processing of portuguese vol. 1, pp. 196–206. association for computational lingustics, santiago de compostela, galicia/spain (2024). https://aclanthology.org/2024.propor-1.20 ro, y., lee, y., kang, p.: multi 2oie: multilingual open information extraction based on multi-head attention with bert. arxiv preprint arxiv:2009.08128 (2020) sena, c.f.l., claro, d.b.: inferportoie: a portuguese open information extraction system with inferences. nat. lang. eng. 25(2), 287–306 (2019). https://doi.org/10.1017/s135132491800044x article  math  google scholar  sena, c.f.l., claro, d.b.: pragmaticoie: a pragmatic open information extraction for portuguese language. knowl. inf. syst. 62(9), 3811–3836 (2020). https://doi.org/10.1007/s10115-020-01442-7 stanovsky, g., michael, j., zettlemoyer, l., dagan, i.: supervised open information extraction. in: proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pp. 885–895 (2018) google scholar  sun, c., qiu, x., xu, y., huang, x.: how to fine-tune bert for text classification? (2020). https://arxiv.org/abs/1905.05583 sun, m., li, x., wang, x., fan, m., feng, y., li, p.: logician: a unified end-to-end neural approach for open-domain information extraction. in: proceedings of the eleventh acm international conference on web search and data mining, pp. 556–564. acm (2018) google scholar  touvron, h., et al.: llama 2: open foundation and fine-tuned chat models (2023) google scholar  vaswani, a., et al.: attention is all you need (2023). https://arxiv.org/abs/1706.03762 wei, x., et al.: zero-shot information extraction via chatting with chatgpt (2023) google scholar  xu, x., zhu, y., wang, x., zhang, n.: how to unleash the power of large language models for few-shot relation extraction? (2023) google scholar  xue, l., et al.: mt5: a massively multilingual pre-trained text-to-text transformer (2021) google scholar  zhang, s., duh, k., van durme, b.: mt/ie: cross-lingual open information extraction with neural sequence-to-sequence models. in: proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pp. 64–70 (2017) google scholar  zhou, s., et al.: a survey on neural open information extraction: current status and future directions (2022) google scholar  download references acknowledgments this material is partially based work supported by the fapesb under grant tic002/2015 and to cce 0022/2023. author information authors and affiliations formas research center on data and natural language, institute of computing federal university of bahia, salvador, bahia, brazil alan melo, bruno cabral & daniela barreiro claro authors alan meloview author publications search author on:pubmed google scholar bruno cabralview author publications search author on:pubmed google scholar daniela barreiro claroview author publications search author on:pubmed google scholar corresponding author correspondence to alan melo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper melo, a., cabral, b., claro, d.b. (2025). scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_30 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords openie language model information extraction publish with us policies and ethics profiles daniela barreiro claro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature rlportfolio: reinforcement learning for financial portfolio optimization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper rlportfolio: reinforcement learning for financial portfolio optimization conference paper first online: 30 january 2025 pp 412–426 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) rlportfolio: reinforcement learning for financial portfolio optimization download book pdf download book epub caio de souza barbosa costa  orcid: orcid.org/0009-0009-7780-70099 & anna helena reali costa  orcid: orcid.org/0000-0001-7309-45289  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 432 accesses abstract portfolio optimization is a task in which an agent constantly rebalances a predefined portfolio of assets in order to mitigate losses and maximize profits. a very effective way to solve this task is using a reinforcement learning agent that learns an optimal investment strategy by interacting with the environment, but due to the current lack of open source tools to accelerate the development of such an agent, its implementation is quite difficult. therefore, to reduce this obstacle, this paper introduces rlportfolio, a python library which provides the necessary tools to develop, train and evaluate reinforcement learning agents whose function is to optimize financial portfolios over time. the library contains a simulation environment that implements the state-of-the-art mathematical formulation of the portfolio optimization task, a policy gradient algorithm developed specifically to train agents to solve this task and four state-of-the-art deep neural network architectures that can be used as the agent’s action policy. this paper also demonstrates the results of an agent trained with rlportfolio to optimize a financial portfolio composed of ten high-volume stocks from the brazilian market, proving the reliability of the solutions contained in the library while also showing that it produces agents that perform better than classical approaches. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others combining transformer based deep reinforcement learning with black-litterman model for portfolio optimization article 10 august 2024 portfolio optimization using reinforcement learning: a study of implementation of learning to optimize chapter © 2023 meta algorithms for portfolio optimization using reinforcement learning chapter © 2021 1 introduction worldwide, numerous investors and financial institutions manage portfolios containing diverse assets that total over 100 trillion dollars [14]. in order to maximize profits, managers are constantly changing the amount invested in each component of the portfolio in a process called portfolio optimization. consequently, the most favorable optimization generates an investment strategy that increases the future value of a portfolio and/or mitigates its risk. recently, with the introduction of algorithmic trading, several machine learning (ml) techniques have been utilized in order to create an agent that automatically optimizes a financial portfolio and, among them, reinforcement learning (rl) [31] approaches have shown to be extremely effective [4]. in the rl approach, an agent learns an optimal behavior by trial and error while interacting with a trading environment and receiving positive or negative rewards that are, respectively, proportional to the profits achieved and losses incurred. those rewards are, then, utilized by training algorithms to maximize the expected future rewards of the agent, consequently maximizing the expected profit of the investment strategy. rl techniques are, thus, fundamentally suitable to the portfolio optimization task. despite its suitability, there are few libraries that developers and researchers can use to design, implement, train and test the performance of portfolio optimization agents with reinforcement learning considering the state-of-the-art formulation of the problem and using novel deep learning and mathematical frameworks. to address the lack of available tools, this paper contributes rlportfoliofootnote 1, a python library designed to facilitate the development of portfolio optimization systems using reinforcement learning. rlportfolio offers a comprehensive suite of features, including: a cutting-edge policy gradient training algorithm [12]; four state-of-the-art neural network policy architectures [12, 25, 26]; a training environment that simulates market dynamics over time [3]. the library is open-source and leverages modern tools such as pandas [32], pytorch [22], and gymnasium [33], ensuring seamless integration with other machine learning frameworks. the remainder of this article is organized as follows. section 2 describes the research area, its achievements and gaps. section 3 introduces the mathematical formulation of the portfolio optimization problem and sect. 4 describes how reinforcement learning can be applied to solve it. section 5 introduces the developed library and its features while sect. 6 describes the results of the application of rlportfolio in the brazilian market. finally, sect. 7 concludes this paper. 2 related work over the years, many methods have been created to generate a strategy that makes profit in the portfolio optimization task. recently, with the advent of artificial intelligence, many of those methods have successfully applied the predictive capabilities of several machine learning techniques [9, 10, 13] so that the future values of assets are forecasted and considered in the decision-making. in this context, reinforcement learning approaches to optimize a portfolio have gained popularity specially after jiang et al. [12] introduced a mathematical formulation of the problem, described a policy gradient training algorithm and achieved great results using a convolutional neural network named eiie (ensemble of identical independent evaluators). after jiang’s work, many papers were able to enhance its framework and perform even better. liang et al. [17], for example, applied noise to input data so that eiie could more efficiently generalize its policy of actions. subsequently, shi et al. [25] applied ideas inspired by inception convolutional neural networks in order to consider multiple temporal windows in the learning process. weng et al. [35], on the other hand, made use of xception convolutional neural networks in conjunction with attention gating techniques and soleymani et al. [28] employed autoencoders to enrich the input data with financial indicators. recently, the research area is moving towards three main ideas: the use of graph neural networks to model the relationships between assets and their implications in their investment values [26, 29, 30], the modeling of the solution as a multiagent system [18, 20] and the application of transformer-like architectures in the agent policy network [15, 37]. therefore, it is clear that even at the forefront of the research field, the mathematical formulation and the training algorithm introduced in jiang’s paper [12] continue to be effectively applied. for that reason, a few open-source implementations of the original framework were developed, such as pgportfolio [11] and deepportfoliomanagementreinforcementlearningv2 [2], but they are not currently maintained, which means that they make use of outdated frameworks and are not easy to use. there are, on the other hand, modern environments [5, 6] and frameworks [38] that can be used to develop reinforcement learning models but they don’t implement neither state-of-the-art formulations nor the training algorithm. recently, however, costa et al. [3] introduced a simulation that applies jiang’s formulation and that can be easily used to train reinforcement learning agents, but a modern open-source implementation of the training algorithm for portfolio optimization agents remains, to the best of our knowledge, nonexistent. 3 mathematical definition of portfolio optimization the portfolio optimization task is defined as the periodic reallocation of the resources of the portfolio by an automated system in order to increase profits and mitigate losses. the system has initially a fixed amount of cash (or any form of risk-free asset defined by the developer) that can be distributed in the assets of the portfolio: at the beginning of each time step, the system analyzes market data and decides the percentage of resources that will be allocated in each asset. the small intervals of time in which the system is able to rebalance the portfolio are called reallocation periods and the size of a time step is constant and defined in the beginning of the task. this paper makes use of the mathematical formulation presented in [12], so the following hypotheses are assumed: there is no slippage, so that orders placed are immediately completed at the last price. the system’s buying and selling orders do not impact the market prices. these hypotheses come from the assumption that the market has high liquidity and trading volume. therefore, to ensure this formulation is close to real-world, it is advisable to use data from markets with these features. considering a portfolio of n assets, the weights vector or portfolio vector \(\vec{w_{t}}\) is a vector of size \(n + 1\) that contains the percentage of resources (or weights) invested in each asset of the portfolio and its remaining cash at time step t. this way, given that \(\vec{w_{t}}(i)\) is the i-th element of \(\vec{w_{t}}\), two constraints must be respected: (1) it is important to highlight that, in this work, the first value \(\vec{w_{t}}(0)\) of the weights vector contains the weight of the remaining cash (or a risk-free asset). at each time step t, there is also a price vector \(\vec{p_{t}}\) that contains the current price (or value) of all the assets including the uninvested amount of cash. \(\vec{p_{t}}\) is in the form \([1, \vec{p_{t}}(1), \vec{p_{t}}(2), ..., \vec{p_{t}}(n)]\) because, in this work, the first value is related to the risk-free cash and, since it is used as a reference asset (all the other prices are calculated in relation to this one), its value is always equal to 1. since the price of the assets probably change over time, \(\vec{p_{t}}\) and \(\vec{p_{t-1}}\) may be different. so the value of the portfolio \(v_{t}\) and the weights vector \(\vec{w_{t}}\) can change during time step t and thus it is necessary to define \(v_{t}^{f}\) and \(\vec{w_{t}^{f}}\), which represent, respectively, the portfolio value and the weights vector at the end of the time step t. it is possible to calculate the final weights vector using the following equation: $$\begin{aligned} \vec{w_{t}^{f}} = \frac{(\vec{p_{t}} \oslash \vec{p_{t-1}}) \odot \vec{w_{t}}}{(\vec{p_{t}} \oslash \vec{p_{t-1}}) \cdot \vec{w_{t}}}, \end{aligned}$$ (2) where \(\cdot \) is the dot product of two vectors, \(\odot \) is the element-wise multiplication and \(\oslash \) is the element-wise division. with those definitions in hand, it is possible to recursively calculate the portfolio value \(v_{t}\) over time through the following equations: $$\begin{aligned} v_{t}^{f} = v_{t} \bigl (\vec{w_{t}} \cdot (\vec{p_{t}} \oslash \vec{p_{t-1}})\bigl ), & v_{t+1} = \mu _{t+1} v_{t}^{f}, \end{aligned}$$ (3) in which \(\mu _{t}\) is the transaction remainder factor (trf) at time step t, a value between 0 and 1 that reduces the portfolio value in order to simulate the effects of brokerage fees considering the reallocation of the portfolio from \(\vec{w_{t-1}^{f}}\) to \(\vec{w_{t}}\). details about how to calculate \(\mu _{t}\) can be found in [12]. so that the iterative calculation of portfolio values is mathematically well-defined, it is necessary to have an initial state. in this work, it is considered that the time step \(t = 0\) contains the initial values of the simulation: during that step, the portfolio investment only consists of cash and there is no change in the portfolio value. so, given a user-defined initial portfolio value \(v_{0}\), the initial conditions at \(t = 0\) are: $$\begin{aligned} \vec{w_{0}} = [1, 0, 0, ..., 0], & v_{0}^{f} = v_{0}. \end{aligned}$$ (4) figure 1 shows a diagram representing the whole process described in this section. fig. 1. diagram depicting the portfolio optimization process over time. in the reallocation periods, the action \(\vec{w_{t}}\) performed by the agent and the current portfolio weights \(\vec{w^{f}_{t-1}}\) are used to calculate \(\mu _{t}\), which simulates the effects of the fees in the portfolio value (\(v^{f}_{t-1} \rightarrow v_{t}\)). at the end of every simulation step, the portfolio value and weights are updated (\(v_{t}, \vec{w_{t}} \rightarrow v^{f}_{t}, \vec{w^{f}_{t}}\)) due to variations of asset prices. full size image usually, the logarithmic rate of return \(r_{t}\) is used to evaluate the performance of the portfolio rebalancing between two time steps. it is given by $$\begin{aligned} r_{t} = ln \bigl (\frac{v_{t}^{f}}{v_{t-1}^{f}}\bigl ). \end{aligned}$$ (5) this way, since \(v_{0} = v_{0}^{f}\), it is possible to determine the final portfolio value at any moment, $$\begin{aligned} v_{t}^{f} = v_{0} \ exp\bigl ( \sum _{t=1}^{t} r_{t} \bigl ), \end{aligned}$$ (6) where t is the number of time steps to be considered. in the portfolio optimization task, a system tries to determine the optimal sequence of weights vector \([\vec{w_{0}}, \vec{w_{1}}, ..., \vec{w_{t}}]\) that maximizes the final portfolio value \(v_{t}^{f}\). 4 a reinforcement learning solution one way to optimize a portfolio is to train a reinforcement learning agent. in this type of learning, an agent aims to learn a policy of actions \(\pi \) by continuously interacting with an environment. at each time step t, the environment provides an observation \(o_{t}\) that contains information about the current situation of the environment. this observation is utilized by the agent to define its state \(s_{t}\), which is used as input in the agent’s policy \(\pi \) in order to generate an action \(a_{t}\) to be applied in the environment (the policy can, thus, be seen as a mapping \(\pi : s \rightarrow a\)). when the agent performs an action in the environment, the latter returns a reward \(r_{t}\), a numeric value which is determined based on the outcome of the performed action: actions that yield good outcomes produce higher rewards. the objective of the reinforcement learning is to iteratively generate a policy that maximizes the expected sum of the rewards received by the agent. in order to apply this approach to the portfolio optimization task, it is necessary to define a policy that can not only handle a continuous state space but also a continuous action space. as stated before, the policy can be seen as a mapping \(\pi : s \rightarrow a\), so it would be necessary to implement a function that maps every possible state value to a specific action, something not computationally viable. since this issue is present in many real-world tasks, deep reinforcement learning (drl) [16] solutions were designed to solve it by using neural networks as a function that can manage continuous values and that approximates close ones, generalizing the state and the action space. drl has achieved remarkable results in many tasks [7, 27] and, for that reason, it is utilized in the portfolio optimization task. in this work, the agent’s state space, the action space and the reward function are modeled as follows: state space: the agent can utilize any time series that contains historical values related to each asset in the portfolio. the size of this time series t represents the number of historical values that are considered in the state. therefore, for a portfolio composed of n assets, the time series can be represented as a matrix of sizes (n, t). this matrix representation, however, only represents a single feature and, in the portfolio optimization task, usually multiple features are used in the state representation: weng et al. [35], for example, shows that the time series of closing, high and low prices of assets are the most important features in an rl agent, but other features such as the time series of indicators and transaction volumes can also be used. this way, the state space is a three-dimensional matrix of sizes (f, n, t), in which f represents the number of features, n denotes the number of assets in the portfolio and t is the size of the time window of the time series. it is important to highlight that, in this work, all the time series used as feature in the state space must have the same time window and, thus, the same number of values. figure 2 contains a diagram representing how a time series can be used as a feature to build the state of the agent. action space: at each time step t, the agent is responsible to provide the weights vector \(\vec{w_{t}}\) to the environment so that it can be used to rebalance the portfolio as explained in sect. 3. therefore, the action space is a vector of size \((n+1)\), where n is the number of assets in the portfolio. reward function: since the agent’s goal is to maximize the expected sum of rewards received at each time step, the reward function plays a crucial role in defining the learning objective. consequently, there are numerous potential functions that can be employed. in the context of portfolio optimization, any metric of profit, risk, or financial indicator can be used. in this work, we utilize the logarithmic rate of return, as introduced in eq. 5, to construct the reward function. this choice trains the agent to maximize profit effectively. fig. 2. visual representation of state space highlighting how a specific feature is used to construct the final three-dimensional structure. full size image 5 rlportfolio: implementing the rl solution given the lack of open-source libraries that implement the formulations presented in sect. 3 and the rl solution introduced in sect. 4, we developed a python library called rlportfolio. this library aims to simplify the development of portfolio optimization systems, allowing researchers to focus on new algorithmic and deep learning architectural solutions. additionally, it enables developers to train an agent with just a few lines of code. rlportfolio also aims to provide benchmarks to the research field, implementing a state-of-the-art training algorithm [12] and some notable neural networks as policies [12, 25, 26] to solve the portfolio optimization task. in order to be reliable, easy to use, and achieve good performance, this project has the following features: the library makes use of recent versions of modern frameworks; the code has the main functionalities tested with unit tests to minimize possible implementation errors; the classes and functions are documented so that the user can understand how to use it; the library is modular so that it can be easily used with other reinforcement learning frameworks; the api is easy-to-use and contains several parameters in order to cover multiple different situations; the project is open source. the architecture of rlportfolio consists primarily of three components: a gymnasium [33] environment implementing all the formulations presented in sect. 3, a policy gradient algorithm based on [12], and a neural network as a deterministic policy \(\pi : s \rightarrow a\) such as introduced in sect. 4. as it can be seen in fig. 3, these three components work together to implement the training loop. the environment is responsible for simulating the effects of the agent action \(a_{t}\), simulate the passage of time and return the appropriate reward \(r_{t}\). the agent is composed of the algorithm and the policy. the algorithm provides the state \(s_{t}\) and the agent’s last action \(a_{t-1}\) to the policy and it is also responsible to calculate the gradients so that the policy can be trained through gradient ascent. the policy, on the other hand, is responsible to choose the best action to be performed based on its inputs. it is important to highlight that by splitting the training logic, the user can replace any of the components by a custom one (including implementations from other libraries). fig. 3. the three main components of rlportfolio. full size image 5.1 the environment the environment can be any gymnasium [33] environment which implements the state space, the action space and the reward function introduced in sect. 4 so that custom environments implementing different mathematical formulations of the portfolio optimization can be used. it is important to note that custom environments must provide the necessary information to the algorithm, therefore it is crucial to read the documentation that accompanies rlportfolio. currently, rlportfolio contains an environment based on poe [3] that implements the mathematical formulations presented in sect. 3 and simulates the passage of time and the interactions between the trading agent and the market. poe was chosen as the foundation of this work because it is, for the best of our knowledge, the gymnasium environment that adheres the best to the mathematical definition of the portfolio optimization task and is built upon modern libraries such as numpy [8] and pandas [32]. additionally, poe is very configurable, so that the user can easily set, for example, the commission fee rate, the normalization method, the features to be considered in calculations, the size of the state time window, etc. by defining the initial value of the portfolio and inputting a pandas dataframe containing the time series of the temporal features to be considered, the environment will scroll through time from the initial to the last time of the input data calculating the effects of the agent’s rebalancing on the portfolio. completely scrolling through all time ranges of the input data defines an episode. it is important to highlight that the modified environment included in rlportfolio has some improvements over poe: it contains two methods to calculate the transaction remainder factor \(\mu _{t}\); traditional loops have been replaced with faster vectorized operations; it is able to return a state with more assets than the portfolio, something useful to graph neural network [36] approaches; unit tests were created to ensure that the environment is correctly implementing the mathematical formulation introduced in sect. 3; more effective normalization methods were added; the graphs generated by the original environment were visually improved. 5.2 the algorithm there are several reinforcement learning algorithms that can be used to train a rl agent and those algorithms are present in many open-source libraries such as stable baselines 3 [23], elegantrl [19] and tianshou [34]. those algorithms can be used with the environment introduced in the previous section but, according to [17], they do not achieve good performance like the policy gradient (pg) one presented in jiang’s work [12]. for that reason, the algorithm included in rlportfolio is a modified version of pg. as the agent interacts with the environment, the pg algorithm for portfolio optimization saves its experiences in the replay buffer ordered in time. eventually, after the replay buffer saves a certain number of experiences greater than or equal to batch_size, a batch of experiences is sampled in order to calculate the objective function $$\begin{aligned} \sum \limits _{t=t_{1}}^{t_{2}} \frac{ln(\mu _{t}(\vec{w_{t}} \cdot (\vec{p_{t}} \oslash \vec{p_{t-1}}))}{t_{2} t_{1} + 1}, \end{aligned}$$ (7) in which \(t_{1}\) and \(t_{2}\) are, respectively, the first and the last time step of the sampled experiences ordered in time, \(\mu _{t}\) is the transaction remainder factor, \(\vec{w_{t}}\) is the agent’s performed action and \(\vec{p_{t}}\) is the vector containing the prices of the assets. note that this objective function is analogous to the logarithm of the rate of the value of the portfolio from steps \(t_{1}\) to \(t_{2}\), therefore the agent will try to maximize profits. additionally, since this work considers that batches with overlapping temporal data (for example, batches \([t_{1}, t_{2}]\) and \([t_{1} + 1, t_{2} + 1]\)) are different representations of the trading process, at each time step t the algorithm samples a batch of sequential data starting at \(t_{1} < t batch\_size\) through a geometrical distribution: $$\begin{aligned} p_{\beta }(t_{1}) = \beta (1 \beta )^ {t t_{1} batch\_size}, \end{aligned}$$ (8) in which \(\beta \in (0, 1]\) is the probability of selecting the most recent batch. the bigger the value of \(\beta \), the more recent batches will be sampled. 5.3 the policy finally, the last component of the training routine is the policy, which must be a neural network whose inputs are the state of the agent and its last performed action. with those two values, the policy must output the next action of the agent (\(\vec{w_{t}} = \phi (\vec{s_{t}}, \vec{w_{t-1}})\)). the policy needs to have knowledge about the last action \(w_{t-1}\) to learn effectively how to avoid abrupt rebalancings in the portfolio and, consequently, avoid losses related to commission fees. in rlportfolio, the policy can be any pytorch [22] neural network with those inputs and outputs. but in order to provide some benchmarks to be used in the research area, the library contains four implementations of state-of-the-art policies: the original recursive and convolutional neural networks solution introduced in [12], the multi-scale convolutional solution from [25] that considerably improves the original architecture and the graph neural network approach presented in [26], which is one of the first works implementing graph approaches. 6 application in the brazilian market in order to demonstrate the performance of rlportfolio, a small experimentfootnote 2 is done by applying the formulation presented in the brazilian market. a portfolio with 10 high-volume assets (vale3, petr4, itub4, bbdc4, bbas3, rent3, lren3, prio3, wege3, abev3) was created and a simple eiie convolutional agent [12] was trained using the historical daily close, high and low prices of the stocks from 2011/01/01 to 2019/12/31 (2233 trading days). the performance of the agent was then tested in the period from 2020/01/01 to 2020/12/31 (248 trading days), which includes the beginning of the covid-19 pandemic, that considerably impacted the brazilian stock market. the data utilized was collected from yahoo finance websitefootnote 3. table 1 summarizes the hyper-parameters used in the training process. table 1. hyper-parameters used to train the eiie convolutional agent.full size table in this experiment, three metrics are used to evaluate the portfolio’s performance: final accumulative portfolio value (fapv): this metric is calculated by dividing the final value of the portfolio by its initial value (\(v_{t}^{f}/v_{0}\)). good portfolios aim to achieve higher fapv. maximum drawdown (mdd) [21]:] as the name suggests, this metric calculates the maximum drop of the portfolio value. strategies with high mdd (in absolute value) are considered risky, so the objective is to generate an agent with mdd as close to zero as possible. sharpe ratio (sr) [24]:] this ratio is created by dividing the returns of the portfolio by its variance. so, simplistically speaking, it measures the returns obtained by the risk taken and, consequently, the higher the sr, the better. rlportfolio calculates all these three metrics at the end of the testing period, so it is easy to evaluate different strategies in the same period of time. the experiment also compares the performance between the eiie convolutional approach and four classical strategies: best stock: this benchmark strategy consists of investing the entire portfolio value in the stock with the highest fapv in the testing period. note that this strategy is not feasible in the real world since the agent needs to know in advance the performance of the stocks in the portfolio to select the best one. nevertheless, it serves as an excellent benchmark for comparison purposes. follow the loser: this approach invests in the stock that had the worst fapv in the training period predicting that its value will rise. uniform buy & hold: this popular strategy involves equally investing in all the stocks within the portfolio and not performing any reallocations. follow the winner:] finally, this method invests in the stock with the best fapv in the training set predicting that its value will continue to rise in the test set. table 2. comparison between the performances of the eiie agent trained with rlportfolio and some classical strategies.full size table as it can be seen in table 2, the agent trained in rlportfolio achieves higher performance in all three metrics when compared to classical approaches. the fact that it is able to outperform the best performing asset in the test period shows that the agent is effectively rebalancing the portfolio considering the state of the market in order to maximize the returns. figure 4 shows the times series of the portfolio values of all the strategies during the test period. fig. 4. portfolio value of the tested strategies during the test period. full size image the library is also integrated with tensorboard [1], which allows the user to keep track of several variables during the agent’s training. figure 5, for example, shows the evolution of the training and testing fapv performance during the training episodes (the test set was not used to tune hyper-parameters, its performance was only logged in order to generate this visualization). the figure also shows the policy loss during training: the growing tendency in the loss indicates that the agent is effectively learning through gradient ascent. fig. 5. time series of metrics in the training (left and right) and testing sets (middle). the fapv is plotted after every training step (left and middle), while the loss (right) is plotted after every gradient ascent. full size image 7 conclusions and future works by implementing the mathematical formulation of the portfolio optimization task presented in sect. 3 and the rl solution introduced in sect. 4, rlportfolio serves as an excellent tool for researchers and developers. it enables the easy implementation of rl agents that effectively optimize a portfolio over time. additionally, the modularity of the library allows users to integrate algorithms, policies, and environments from other frameworks, enhancing the potential for comparative studies and accelerating advancements in the field. furthermore, sect. 6 demonstrates that even one of the simplest convolutional architectures achieves excellent results in the brazilian market, even during the covid-19 pandemic. this not only confirms rlportfolio’s effectiveness in training an agent but also underscores the feasibility of this approach in developing countries, markets that are typically underrepresented in research. finally, as rlportfolio is open-source, its capabilities are poised to grow with contributions from new developers. contributors can expedite and validate the introduction of future features, such as implementing agents that follow several classical strategies and other state-of-the-art policies for comparative studies, integrating with tensorflow [1], creating interactive visualization tools, and introducing new environments with more realistic simulations. notes 1.rlportfolio is available at https://github.com/caiosbc/rlportfolio. 2.the code can be found at https://github.com/caiosbc/rlportfolio_bracis. 3.the website address is https://finance.yahoo.com/. references abadi, m., et al.: tensorflow: a system for large-scale machine learning. in: proceedings of the 12th usenix conference on operating systems design and implementation, osdi 2016, pp. 265–283. usenix association, usa (2016) google scholar  amrouni, s.: selimamrouni/deep-portfolio-management-reinforcement-learning: v2.0. zenodo (2022). https://doi.org/10.5281/zenodo.5993372 costa, c.d.s.b., costa, a.h.r.: poe: a general portfolio optimization environment for finrl. in: anais do brazilian workshop on artificial intelligence in finance (bwaif), pp. 132–143. sbc (2023). https://doi.org/10.5753/bwaif.2023.231144 felizardo, l.k., paiva, f.c.l., costa, a.h.r., del-moral-hernandez, e.: reinforcement learning applied to trading systems: a survey (2022). https://doi.org/10.48550/arxiv.2212.06064 haghpanah, m.a.: gym-mtsim (2021) google scholar  haghpanah, m.a.: gym-anytrading (2023) google scholar  hambly, b., xu, r., yang, h.: recent advances in reinforcement learning in finance. math. financ. 33(3), 437–503 (2023). https://doi.org/10.1111/mafi.12382 article  mathscinet  math  google scholar  harris, c.r., et al.: array programming with numpy. nature 585(7825), 357–362 (2020). https://doi.org/10.1038/s41586-020-2649-2 article  math  google scholar  henrique, b.m., sobreiro, v.a., kimura, h.: literature review: machine learning techniques applied to financial market prediction. expert syst. appl. 124, 226–251 (2019). https://doi.org/10.1016/j.eswa.2019.01.012 article  google scholar  hu, y., liu, k., zhang, x., su, l., ngai, e.w.t., liu, m.: application of evolutionary computation for rule discovery in stock algorithmic trading: a literature review. appl. soft comput. 36, 534–551 (2015). https://doi.org/10.1016/j.asoc.2015.07.008 article  math  google scholar  jiang, z.: zhengyaojiang/pgportfolio (2024) google scholar  jiang, z., xu, d., liang, j.: a deep reinforcement learning framework for the financial portfolio management problem (2017). https://doi.org/10.48550/arxiv.1706.10059 khadjeh nassirtoussi, a., aghabozorgi, s., ying wah, t., ngo, d.c.l.: text mining for market prediction: a systematic review. expert syst. appl. 41(16), 7653–7670 (2014). https://doi.org/10.1016/j.eswa.2014.06.009 le, f.d.: global market portfolio 2023 (2023). https://www.ssga.com/international/en/institutional/ic/insights/global-market-portfolio-2023 li, j., zhang, y., yang, x., chen, l.: online portfolio management via deep reinforcement learning with high-frequency data. inf. process. manag. 60(3), 103247 (2023). https://doi.org/10.1016/j.ipm.2022.103247 article  math  google scholar  li, y.: deep reinforcement learning (2018). https://doi.org/10.48550/arxiv.1810.06339 liang, z., chen, h., zhu, j., jiang, k., li, y.: adversarial deep reinforcement learning in portfolio management (2018). https://doi.org/10.48550/arxiv.1808.09940 lin, y.c., chen, c.t., sang, c.y., huang, s.h.: multiagent-based deep reinforcement learning for risk-shifting portfolio management. appl. soft comput. 123, 108894 (2022). https://doi.org/10.1016/j.asoc.2022.108894 article  google scholar  liu, x.y., et al.: elegantrl-podracer: scalable and elastic library for cloud-native deep reinforcement learning (2022). https://doi.org/10.48550/arxiv.2112.05923 ma, c., zhang, j., li, z., xu, s.: multi-agent deep reinforcement learning algorithm with trend consistency regularization for portfolio management. neural comput. appl. (2022). https://doi.org/10.1007/s00521-022-08011-9 article  math  google scholar  magdon-ismail, m., atiya, a.f., pratap, a., abu-mostafa, y.s.: on the maximum drawdown of a brownian motion. j. appl. probab. 41(1), 147–161 (2004). https://doi.org/10.1239/jap/1077134674 article  mathscinet  math  google scholar  paszke, a., et al.: pytorch: an imperative style, high-performance deep learning library (2019). https://doi.org/10.48550/arxiv.1912.01703 raffin, a., hill, a., gleave, a., kanervisto, a., ernestus, m., dormann, n.: stable-baselines3: reliable reinforcement learning implementations. j. mach. learn. res. 22(268), 1–8 (2021) math  google scholar  sharpe, w.f.: the sharpe ratio. j. portfolio manag. 21(1), 49–58 (1994). https://doi.org/10.3905/jpm.1994.409501 article  math  google scholar  shi, s., li, j., li, g., pan, p.: a multi-scale temporal feature aggregation convolutional neural network for portfolio management. in: proceedings of the 28th acm international conference on information and knowledge management, beijing, china, pp. 1613–1622. acm (2019). https://doi.org/10.1145/3357384.3357961 shi, s., li, j., li, g., pan, p., chen, q., sun, q.: gpm: a graph convolutional network based reinforcement learning framework for portfolio management. neurocomputing 498, 14–27 (2022). https://doi.org/10.1016/j.neucom.2022.04.105 article  math  google scholar  silver, d., et al.: mastering the game of go without human knowledge. nature 550(7676), 354–359 (2017). https://doi.org/10.1038/nature24270 article  math  google scholar  soleymani, f., paquet, e.: financial portfolio optimization with online deep reinforcement learning and restricted stacked autoencoder–deepbreath. expert syst. appl. 156, 113456 (2020). https://doi.org/10.1016/j.eswa.2020.113456 article  google scholar  soleymani, f., paquet, e.: deep graph convolutional reinforcement learning for financial portfolio management deeppocket. expert syst. appl. 182, 115127 (2021). https://doi.org/10.1016/j.eswa.2021.115127 article  math  google scholar  sun, q., wei, x., yang, x.: graphsage with deep reinforcement learning for financial portfolio optimization. expert syst. appl. 238, 122027 (2024). https://doi.org/10.1016/j.eswa.2023.122027 article  math  google scholar  sutton, r.s., barto, a.g.: reinforcement learning: an introduction. a bradford book, cambridge (2018) math  google scholar  the pandas development team: pandas-dev/pandas: pandas. zenodo (2023). https://doi.org/10.5281/zenodo.3509134 towers, m., et al.: gymnasium. zenodo (2023). https://doi.org/10.5281/zenodo.8127026 weng, j., et al.: tianshou: a highly modularized deep reinforcement learning library (2022). https://doi.org/10.48550/arxiv.2107.14171 weng, l., sun, x., xia, m., liu, j., xu, y.: portfolio trading system of digital currencies: a deep reinforcement learning with multidimensional attention gating mechanism. neurocomputing 402, 171–182 (2020). https://doi.org/10.1016/j.neucom.2020.04.004 article  google scholar  wu, z., pan, s., chen, f., long, g., zhang, c., yu, p.s.: a comprehensive survey on graph neural networks. ieee trans. neural netw. learn. syst. 32(1), 4–24 (2021). https://doi.org/10.1109/tnnls.2020.2978386 article  mathscinet  math  google scholar  xu, k., zhang, y., ye, d., zhao, p., tan, m.: relation-aware transformer for portfolio policy learning. in: twenty-ninth international joint conference on artificial intelligence, vol. 5, pp. 4647–4653 (2020). https://doi.org/10.24963/ijcai.2020/641 yang, x., liu, w., zhou, d., bian, j., liu, t.y.: qlib: an ai-oriented quantitative investment platform (2020). https://doi.org/10.48550/arxiv.2009.11189 download references acknowledgments the authors would like to thank the programa de bolsas itaú (pbi) of the centro de ciência de dados (c\(^2\)d) at escola politécnica at usp, supported by itaú unibanco s.a., and the brazilian national council for scientific and technological development (cnpq grant n. 310085/2020-9). the authors would also like to thank student joão felipe santiago boccardo for proofreading and providing constructive criticism of the manuscript. author information authors and affiliations escola politécnica, universidade de são paulo, são paulo, brazil caio de souza barbosa costa & anna helena reali costa authors caio de souza barbosa costaview author publications search author on:pubmed google scholar anna helena reali costaview author publications search author on:pubmed google scholar corresponding author correspondence to caio de souza barbosa costa . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper costa, c.d.s.b., costa, a.h.r. (2025). rlportfolio: reinforcement learning for financial portfolio optimization. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_29 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords reinforcement learning portfolio optimization quantitative finance publish with us policies and ethics profiles anna helena reali costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature occluded face in-painting using generative adversarial networks—a review | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper occluded face in-painting using generative adversarial networks—a review conference paper first online: 12 october 2023 pp 243–258 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) occluded face in-painting using generative adversarial networks—a review download book pdf download book epub victor ivamoto  orcid: orcid.org/0000-0001-5463-36459, rodolfo simões  orcid: orcid.org/0000-0001-6989-12339, bruno kemmer  orcid: orcid.org/0000-0002-6830-32819 & … clodoaldo lima  orcid: orcid.org/0000-0003-3381-53489  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 590 accesses 3 citations abstract face image de-occlusion and inpainting is a challenging problem in computer vision with several practical uses and is employed in many image preprocessing applications. the impressive results achieved by generative adversarial networks in image processing increased the attention of the scientific community in recent years around facial de-occlusion and inpainting. recent network architecture developments are the two-stage networks using coarse to fine approach, landmarks, semantic segmentation map, and edge maps that guide the inpainting process. moreover, improved convolutions enlarge the receptive field and filter the values passed to the next layer, and attention layers create relationships between local and distant information. this article presents a brief review of recent developments in gan-based techniques for de-occlusion and inpainting of face images. in addition, it describes and analyzes network architectures and building blocks. finally, we identify current limitations and propose directions for future research. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others occluded face recognition by identity-preserving inpainting chapter © 2020 high-quality facial feature occlusion repair based on s-gans chapter © 2024 restoring face occluded images using gan based inpainting with perceptual and contextual losses chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biometrics computer graphics computer vision image processing occlusion veneers 1 introduction facial de-occlusion and inpainting are special instances of image inpainting. they are used in the restoration of damaged images [26], removal of unwanted content [26] and data augmentation [8]. moreover, face de-occlusion and inpainting are important preprocessing steps in many computer vision tasks with numerous applications. this is because occlusions break the entire structure of the face and hide the identity of the subject, resulting in performance degradation in many applications [37]. occlusions degrade the performance of face parsing [30, 32], object and face detection [21] and facial expression analysis [32]. furthermore, occlusions hide landmarks used in face alignment and frontalization [2, 41]. face inpainting also poses several challenges that are difficult to overcome. first, the human face carries biometric information unique to each subject, revealing identity, age, sex, emotions, ethnicity, and even culture and religion. this biometric information must be preserved in the restored image. second, there are many plausible solutions for filling the missing holes in an image, where the ground-truth is just one option. for example, given a face covered with a surgical mask, the mouth may be smiling in the ground-truth image whereas it is closed after reconstruction. third, the set of possible solutions is restricted by the overall content, as the restoration must preserve the subject’s skin and hair texture, facial symmetry, structure and expression, along with variations of illumination and pose. fourth, occlusions can appear anywhere in the image and may be of any shape and size. large occlusions covering both sides off the face are more difficult to restore than small ones covering just one side. fifth, unique facial marks such as makeup, tattoos, scars, stains, wrinkles, and accessories are difficult to recover with no reference image. finally, the restored area must be visually consistent with the neighboring region, creating an imperceptible transition between them [1, 38, 44]. researchers developed new methods to overcome these challenges and improve the quality of the image. the most prominent methods are gan-based networks that are able to reconstruct an image with photo-realism. modifications in the gan architecture with the inclusion of new building blocks, network elements and loss functions address specific facial inpainting issues. this reviewfootnote 1 summarizes these developments, building a solid foundation for future research. the rest of the article is organized as follows. section 2 describes network architecture and components and presents methods of training stability. section. 3 discusses the current limitations found in the literature. finally, we conclude in sect. 4. 2 theoretical background the network used for image inpainting consists of a number of components and elements that contribute to the final result. this section discusses the main network structures found in the literature. 2.1 network architecture generative adversarial networks (gan). generative adversarial networks (gan) consist of a generator and discriminator networks [12]. the generator creates images from simple random noise, usually following a uniform or spherical gaussian distribution [13]. the discriminator is a classifier that distinguishes between real and fake images. both networks play an adversarial game in which the generator tries to fool the discriminator by gradually improving the image quality. they are trained in alternation until the discriminator is unable to distinguish the synthetic from real images [12]. figure 1 shows the gan architecture. fig. 1. original gan architecture proposed by goodfellow et al. [12]. the generator receives a random noise vector as input and creates fake images. the discriminator is a classifier that evaluates whether the image is real or fake. full size image in image inpainting and de-occlusion, the generator input is a set of occluded images instead of random noise. after training, only the generator is used to infer new images and the discriminator is removed. figure 2 illustrates the basic gan architecture used in image de-occlusion and inpainting. variations of this architecture found in the literature are described in the next sections. fig. 2. gan architecture used in image de-occlusion and inpainting. instead of random noise, the generator receives occluded images as input and creates occlusion-free images. full size image two-stage network. splitting the inpainting process into two or more stages improves the image quality. in this setting, each stage is responsible for a portion of the restoration process. the most common approaches are coarse-to-fine and prior information. in the coarse-to-fine approach, the first stage creates an initial coarse prediction of the de-occluded image and the second stage takes the result of the first stage as input and refines the prediction [45]. this method gained popularity for its higher performance compared to single-stage networks [4, 9, 14, 44, 45]. figure 3 illustrates a two-stage network with the coarse-to-fine approach. prior information such as landmarks, edges, or semantic segmentation maps provides spatial and structural information, guiding the inpainting process. this allows the inpainting network to build the face with realistic structure and facial expressions. in general, the prior network is trained to detect landmarks, edges or semantic segmentation and create the respective maps which are used by the inpainting network to guide the completion process. the landmark map improves the perceptual quality of the image, providing spatial consistency in unaligned faces [38]. the effect of landmarks in image inpainting is so strong that swapping the map of two persons changes their identities and face expressions [43]. the edge generator network predicts the edge map of the occlusion-free image, which is later used to guide the inpainting process [36, 42, 50]. the generator receives the masked grayscale ground-truth image, the masked edge map and the binary mask indicating the occluded area. likewise, a parsing network creates an occlusion-free semantic segmentation map of the original occluded image, which guides the de-occlusion process [35, 46]. alternatively, the parsing network can provide semantic regularization, where the semantic segmentation map of the generated image is compared with the ground-truth [11, 27]. fig. 3. the two-stage architecture consists of a coarse and a refinement network. full size image 2.2 generator in gans, the generator is any neural network able to create the probability distribution of real data [12, 13]. then, sampling from this distribution generates completely new images. the input to the generator can be a vector of random noise, the incomplete image, semantic segmentation map, edge map, landmarks or binary mask. in face image de-occlusion and inpainting, the generator can be an encoder-decoder, u-net, multi-branch network or any variation. encoder-decoder and u-net. an encoder-decoder is a generative model trained to reconstruct the input data in an unsupervised way [34]. the network has a symmetric architecture comprised of an encoder and a decoder. the encoder consists of a stack of down-sampling layers that compress the original data into a low dimensional representation. the decoder part contains a series of up-sampling layers that recover the original information. optionally, a bottleneck layer can be inserted between the encoder and the decoder. this layer converts the encoder’s last layer into a vector with similar functionality as the random noise vector in the original gan. the encoder-decoder architecture with the bottleneck layer is appropriate for image inpainting with gans. the encoder converts the occluded image into a vector, and the decoder reconstructs the de-occluded face from this vector. the u-net has a similar architecture as the encoder-decoder, where the main difference is the skip connections concatenating each encoder layer with the corresponding symmetrical decoder layer. in the original architecture [40], the u-net encoder is a series of 3\(\times \)3 convolutions followed by relu and 2\(\times \)2 max pooling, while the decoder is a series of up sampling layers with 2\(\times \)2 kernel, a concatenation with the corresponding encoder layer and 3\(\times \)3 convolutions with relu. figure 4 illustrates the encoder-decoder and u-net architectures. fig. 4. the encoder-decoder and u-net architectures consist of an encoder with down sampling layers, a bottleneck layer in the middle and a decoder with up sampling layers. the u-net has skip-connections concatenating each encoder layer with the corresponding decoder layer. left: encoder-decoder. right: u-net. full size image modified versions of both encoder-decoder and u-net are commonly used in the generator of gans used in face de-occlusion and inpainting. the variations include adding dilated convolution [21, 25, 26, 28, 29, 35, 46, 47], se block [21, 25, 35, 47], hdc [10] and self attention blocks [33]. 2.3 discriminator the discriminator is a classifier that calculates the probability that the image is real rather than synthesized [12]. however, since the inpainted region is a fraction of the entire image, the discriminator is biased towards a generated image being real, resulting in poor inpainting quality. this section describes variations of discriminators that address this issue. local and global discriminators. the combination of global and local discriminators improves the reconstruction realism and consistency. the global discriminator evaluates the entire image, while the local discriminator judges a small patch around the reconstructed area. the objective function is the sum of the loss functions applied to each discriminator [27]. a less common variation combines the outputs of both discriminators and converts them into a single number representing the probability that the image is real or reconstructed. specifically, the outputs of both discriminators concatenate and then pass through a fully connected layer. in this setting, the loss is calculated at the combined output [19]. figure 5 shows an example of an architecture of local and global discriminators with combined outputs. the architecture is a stack of \(5\times 5\) convolutions with stride 2 followed by a fully-connected layer that outputs a 1024 vector. the concatenated output of both discriminators passes through a fully-connected layer with sigmoid activation [19]. fig. 5. network architecture. it consists of one generator and two discriminators. the generator takes the occluded image as input and outputs the occlusion-free image. two discriminators learn to distinguish the synthesized contents as real and fake. the global discriminator evaluates the entire image, while the local discriminator centers in a small area around the damaged region [19]. full size image patchgan. instead of evaluating the entire image as being real or fake like the standard discriminator, patchgan classifies each patch in the input image. this discriminator runs across the image like a convolution and outputs the average of all patches. patchgan models high-frequency details, providing texture and style lossesfootnote 2 [20]. figure 6 shows the structure of patchgan. fig. 6. patchgan classifies each patch in the input image as real or fake. this discriminator runs across the image like a convolution and outputs the average of all patches. full size image sn-patchgan. sn-patchgan is a fully convolutional spectral-normalized markovian discriminator. this discriminator computes the loss directly on each point of the last feature map. sn-patchgan was designed to inpaint images with regular and irregular shapes of any size and in multiple regions in the image. it provides faster and more stable training, replaces the global and local discriminators and dispenses the perceptual loss [44]. the original discriminator consists of a stack of layers of \(5\times 5\) convolutions with stride 2 and spectral normalization. sn-patchgan can be interpreted as a 3d classifier, where the loss is applied to each feature element on the feature map of the last layer, as illustrated in fig. 7. fig. 7. fully convolutional spectral-normalized markovian discriminator (sn-patchgan). the discriminator loss is applied in the last feature map, resulting in a 3d classifier [44]. full size image 2.4 building blocks in the context of this paper, a block is a group of layers working together that executes a specific task. a block can be inserted between two layers in the generator. this section describes the main building blocks used in gans, such as self-attention, residual blocks and squeeze and excitation. self-attention. convolutions process local information limited to the kernel shape and size. when the kernel is inside a hole larger than the kernel size, it captures only invalid pixels, becoming unable to hallucinate meaningful pixels. therefore, they’re not suitable for inpainting regions larger than the kernel size. on the other hand, self-attention is a non-local mechanism that creates relationships between distant regions in the image [48]. figure 8 shows the self-attention modulefootnote 3. fig. 8. the self-attention module creates relationships between distant regions in the image. \(\otimes \) denotes the hadamard product. full size image the self-attention output is computed as follows. let’s define c as the number of channels, n the number of feature locations from the previous layer, \(x \in \mathbb {r}^{c\times n}\) as the previous layer feature map, \(\textbf{w}_f\), \(\textbf{w}_g\), \(\textbf{w}_h \in \mathbb {r}^{\bar{c}\times c}\) and \(\textbf{w}_v \in \mathbb {r}^{c\times \bar{c}}\) as the weight matrices, and \(\bar{c} = c/8\). the feature maps \(\textbf{f}\) and \(\textbf{g}\) are calculated as \(f(x)=\textbf{w}_f x\), \(g(x)=\textbf{w}_g x\). $$\begin{aligned} \beta _{j,i} &= \frac{\exp (s_{ij})}{\sum _{i=1}^n \exp (s_{ij})}, & s_{ij} = f(x_i)^\top g(x_j) \end{aligned}$$ (1) where the softmax \(\beta _{j,i}\) is the probability that the \(i^{th}\) location serves the \(j^{th}\) region. the output of attention layer \(\textbf{o}=(o_1,...,o_j,...,o_n)\in \mathbb {r}^{c\times n}\) is given by: $$\begin{aligned} & o_j = v\left( \sum _{i=1}^n\beta _{j,i}h(x_i)\right) , & h(x_i) & = \textbf{w}_h x_i, & v(x_i) & = \textbf{w}_v x_i \end{aligned}$$ (2) the final output is given by \(y_i=\gamma o_i + x_i\), where \(\gamma \) is a learned parameter. residual block. a residual block (resblock) consists of a series of convolutional layers with skip connection, i.e., the input adds to the output as illustrated in fig. 9. fig. 9. a residual block consists of two or more convolution layers with skip connection where the input adds to the output. \(\phi \) is the activation function and \(\bigoplus \) is element-wise sum. full size image the residual block avoids gradient dispersion in very deep networks [42] and replaces the standard convolution with dilated convolution [46] or multi-dilated convolution [26]. moreover, residual networks are easy to optimize [15], train faster and achieve similar losses compared to non-residual networks [24]. the residual block was originally conceived for image classification [15]. residual blocks are used in bottleneck layer of encoder-decoder [3, 43, 46], contraction and expansion sides of u-net [7, 10, 26] or as a building block of multi-branch networks [31, 32]. squeeze and excitation blocks. the squeeze-and-excitation (se) block models the relationships between channels in the feature maps [18]. the block performs channel-wise feature re-calibration, strengthening meaningful features and weakening worthless ones. se blocks fit between two layers, achieving higher performance gain at a small computational cost. the squeeze operation uses global average pooling to aggregate each feature map across its spatial dimension, and the excitation operation is a simple gating that produces a collection of weights that are applied to the feature maps. figure 10 illustrates the architecture of the se block. fig. 10. the squeeze and excitation block scales the feature maps in a given layer according to their importance. the reduction ratio q is a hyperparameter that balances performance and computational complexity. the default is \(q=16\). full size image 2.5 training stability this section presents two approaches to stabilize the training of gans. zhang et al. proposed the use of spectral normalization on both the generator and discriminator, as well as employing the two time scale update rule (ttur) [48]. two time scale update rule (ttur). using different learning rates for the generator and discriminator in combination with the adam stochastic optimization improves convergence and stability. in the two time scale update rule (ttur), the learning rate of the generator is generally lower than the discriminator. although the ttur theory ensures convergence, the appropriate learning rates must be empirically found for each network [16]. the learning rates found in the literature for the generator is 1e-4 and for the discriminator are 1e-12 [22, 23], 1e-4 [11] and 4e-4 [5, 21, 48]. spectral normalization. spectral normalization is a weight-normalization technique originally proposed to stabilize the training of the discriminator [39]. spectral normalization is simple to implement, has low computation cost, and further improves stability when applied in combination with gradient penalty. furthermore, when employed in the generator and discriminator, spectral normalization further reduces the discriminator to generator update ratio, decreases the computational cost, and provides more stable training [48]. the spectral normalization is given by eq. 3: $$\begin{aligned} & \textbf{w}_{sn} = \frac{\textbf{w}}{\eta (\textbf{w})}, & \eta (\textbf{w}) = \max _{\left\| {\textbf {h}} \right\| _2 \le 1} \left| \textbf{w}{} {\textbf {h}}\right| _2 \end{aligned}$$ (3) where \(\eta (\textbf{w})\) is the spectral norm of the matrix \(\textbf{w}\). 3 limitations despite the impressive progress in image de-occlusion and inpainting over the recent years, several challenges remain to be solved. given the broad extent of current limitations, it’s hard to imagine that a single solution will be able to address all situations. this section analyzes key limitations identified during the review and proposes open areas for research. 3.1 datasets model research still requires large amounts of training data, heavily relying on available datasets, and the model generalization capabilities on unseen data largely depends on the trained dataset. an open research area is the development of models, algorithms, and methods resilient to data availability, i.e., models with high generalization capability using few training data. moreover, despite the variety in available data, there are few datasets created for face de-occlusion and inpainting. these datasets contain few images compared with other face databases. for this reason, researchers build their own synthetic images based on available public face datasets, usually overlaying an object or a binary mask. this approach may be good for model development, not for inference in real world scenarios which require a large occluded face dataset for testing models. 3.2 evaluation metrics user study measures qualitative attributes that are hard to evaluate with quantitative methods alone. since researchers employ different methodologies when conducting the qualitative survey, results cannot be compared across published studies. this situation could be avoided if researchers followed a formal protocol describing the survey process. the protocol might use psychophysical similarity measurements already used in the literature, such as two alternative forced choice (2afc) and just noticeable differences (jnd) used in [49]. moreover, most quantitative evaluation metrics measure pixel-level statistics that are unable to capture human perception. for historical reasons, they are still widely used for model comparison. the two most used metrics, psnr and ssim, carry a simple relationship between them [17]. on the other hand, feature-level metrics capture higher level perceptual quality. lpips is the only feature-level metric found in the literature, but it still lags behind human-level perception. more research still needs to be done in an improved version of lpips with higher perception level as well as quantitative metrics able to evaluate other qualitative attributes such as effective occlusion removal, naturalness, image realism, consistency, and perception quality. 3.3 automatic de-occlusion most state-of-the-art models require a binary mask with holes in the occluded area. this can be useful for single image restoration and de-occlusion of photographs, but it is unfeasible for videos, real-time de-occlusion and batch processing of several images. automatic detection and removal of occlusions fails to properly detect occlusions, producing artifacts in the restored region [6]. moreover, there are still few studies in this area. 3.4 image quality image quality remains an open problem, for example eyes with different colors, distorted shape of mouth and nose, missing ears, texture discontinuity in the border pixels, artifacts, blur, and bad background filling. models using prior information such as landmarks, edges and semantic segmentation maps, or other coarse-to-fine approach, rely on the quality of the predictions of these priors. these predictions have performance degradation on occluded faces, in particular combined with large pose variations, such as top or bottom views and profile. large occlusions also degrade performance. the majority of models are trained with 25% of missing region, and a few restore over than 50%. it’s specially challenging to remove occlusions covering symmetric parts of the face, for example both eyes, simply because the model doesn’t know the color and shape of the eyes. in such cases, the use of prior information helps with the structure of the face, but the texture still remains missing. 3.5 computational cost current models have high computational cost, restricting the use in edge devices and in real-time applications. inference time is still very high for real-time applications even with gpu, and the number of parameters may restrict the use in edge devices. moreover, training a model takes a few days and model design takes a few months. more research is needed to create better algorithms and more efficient methods to reduce these computational costs. 4 conclusion this paper reviews gan-based face image inpainting and de-occlusion studies found in the literature. more specifically, we explored the network architecture and components. the review also analyzed the limitations. the gan architecture for image inpainting and two-stage networks were described. encoder-decoder and u-net are basic generator architectures that can be combined with other components for additional functionalities. they are also used in single and multi-stage architectures. local and global discriminators, patchgan and sn-patchgan improve the gan’s ability to distinguish between real and fake images in local and global levels. squeeze and excitation blocks perform channel-wise feature re-calibration, weighting the importance of each feature map in a given layer. ttur, adam optimizer and spectral normalization accelerate and stabilize gan training. finally, this study discussed the current limitations and challenges found in datasets, evaluation metrics, automatic de-occlusion, image quality and computational cost. furthermore, we propose insights for future research. notes 1.see protocol at https://github.com/vivamoto/bracis-2023. 2.python code available at https://github.com/znxlwm/pytorch-pix2pix/blob/3059f2af53324e77089bbcfc31279f01a38c40b8/network.py. 3.python code is available at https://github.com/brain-research/self-attention-gan. references bertalmio, m., sapiro, g., caselles, v., ballester, c.: image inpainting. in: proceedings of the 27th annual conference on computer graphics and interactive techniques, siggraph 2000, pp. 417–424. acm press/addison-wesley publishing co., usa (2000) google scholar  burgos-artizzu, x.p., perona, p., dollár, p.: robust face landmark estimation under occlusion. in: 2013 ieee international conference on computer vision, pp. 1513–1520 (2013) google scholar  cai, j., han, h., cui, j., chen, j., liu, l., kevin zhou, s.: semi-supervised natural face de-occlusion. ieee trans. inf. forensics secur. 16, 1044–1057 (2021) article  google scholar  cao, s., sakurai, k.: face completion with pyramid semantic attention and latent codes. in: proceedings 2020 8th international symposium on computing and networking, candar 2020, pp. 1–8. institute of electrical and electronics engineers inc. (2020) google scholar  chen, m., liu, z., ye, l., wang, y.: attentional coarse-and-fine generative adversarial networks for image inpainting. neurocomputing 405, 259–269 (2020) article  google scholar  chen, y.a., chen, w.c., wei, c.p., wang, y.c.: occlusion-aware face inpainting via generative adversarial networks. in: proceedings international conference on image processing, icip, vol. 2017-september, pp. 1202–1206. ieee computer society (2018) google scholar  cheung, y.m., li, m., zou, r.: facial structure guided gan for identity-preserved face image de-occlusion. in: icmr 2021 proceedings of the 2021 international conference on multimedia retrieval, pp. 46–54. association for computing machinery, inc. (2021) google scholar  din, n., javed, k., bae, s., yi, j.: effective removal of user-selected foreground object from facial images using a novel gan-based network. ieee access 8, 109648–109661 (2020) article  google scholar  dong, j., zhang, l., zhang, h., liu, w.: occlusion-aware gan for face de-occlusion in the wild. in: proceedings ieee international conference on multimedia and expo, vol. 2020-july. ieee computer society (2020) google scholar  fang, y., li, y., tu, x., tan, t., wang, x.: face completion with hybrid dilated convolution. signal process. image commun. 80, 115664 (2020) article  google scholar  ge, s., li, c., zhao, s., zeng, d.: occluded face recognition in the wild by identity-diversity inpainting. ieee trans. circuits syst. video technol. 30(10), 3387–3397 (2020) article  google scholar  goodfellow, i.j., et al.: generative adversarial nets. in: proceedings of the 27th international conference on neural information processing systems, nips 2014, vol. 2, pp. 2672–2680. mit press, cambridge (2014) google scholar  gulrajani, i., ahmed, f., arjovsky, m., dumoulin, v., courville, a.: improved training of wasserstein gans. in: proceedings of the 31st international conference on neural information processing systems, nips 2017, pp. 5769–5779. curran associates inc., red hook (2017) google scholar  guo, d., feng, j., zhou, b.: structure-aware image expansion with global attention. in: siggraph asia 2019 technical briefs, sa 2019, pp. 13–16. association for computing machinery, inc. (2019) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: 2016 ieee conference on computer vision and pattern recognition (cvpr), pp. 770–778 (2016) google scholar  heusel, m., ramsauer, h., unterthiner, t., nessler, b., hochreiter, s.: gans trained by a two time-scale update rule converge to a local nash equilibrium. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017) google scholar  horé, a., ziou, d.: image quality metrics: psnr vs. ssim. in: 2010 20th international conference on pattern recognition, pp. 2366–2369 (2010) google scholar  hu, j., shen, l., albanie, s., sun, g., wu, e.: squeeze-and-excitation networks. ieee trans. pattern anal. mach. intell. 42(8), 2011–2023 (2020) article  google scholar  iizuka, s., simo-serra, e., ishikawa, h.: globally and locally consistent image completion. acm trans. graph. 36(4), 1–14 (2017) article  google scholar  isola, p., zhu, j.y., zhou, t., efros, a.a.: image-to-image translation with conditional adversarial networks. in: 2017 ieee conference on computer vision and pattern recognition (cvpr), pp. 5967–5976 (2017) google scholar  jabbar, a., li, x., iqbal, m., malik, a.: fd-stackgan: face de-occlusion using stacked generative adversarial networks. ksii trans. internet inf. syst. 15(7), 2547–2567 (2021) google scholar  jam, j., kendrick, c., drouard, v., walker, k., hsu, g.s., yap, m.: r-mnet: a perceptual adversarial network for image inpainting. in: proceedings 2021 ieee winter conference on applications of computer vision, wacv 2021, pp. 2713–2722. institute of electrical and electronics engineers inc. (2021) google scholar  jam, j., kendrick, c., drouard, v., walker, k., hsu, g.s., yap, m.: symmetric skip connection wasserstein gan for high-resolution facial image inpainting. in: visigrapp 2021 proceedings of the 16th international joint conference on computer vision, imaging and computer graphics theory and applications, vol. 4, pp. 35–44. scitepress (2021) google scholar  johnson, j., alahi, a., fei-fei, l.: perceptual losses for real-time style transfer and super-resolution. in: leibe, b., matas, j., sebe, n., welling, m. (eds.) eccv 2016. lncs, vol. 9906, pp. 694–711. springer, cham (2016). https://doi.org/10.1007/978-3-319-46475-6_43 chapter  google scholar  khan, m., ud din, n., bae, s., yi, j.: interactive removal of microphone object in facial images. electronics 8(10), 1115 (2019) article  google scholar  li, x., hu, g., zhu, j., zuo, w., wang, m., zhang, l.: learning symmetry consistent deep cnns for face completion. ieee trans. image process. 29, 7641–7655 (2020) article  math  google scholar  li, y., liu, s., yang, j., yang, m.h.: generative face completion. in: 2017 ieee conference on computer vision and pattern recognition (cvpr), pp. 5892–5900 (2017) google scholar  li, z., zhu, h., cao, l., jiao, l., zhong, y., ma, a.: face inpainting via nested generative adversarial networks. ieee access 7, 155462–155471 (2019) article  google scholar  lie, y., li, l.: image inpainting using multi-scale neural network and shift-net. in: proceedings 2020 7th international conference on information science and control engineering, icisce 2020, pp. 704–709. institute of electrical and electronics engineers inc. (2020) google scholar  lin, j., yang, h., chen, d., zeng, m., wen, f., yuan, l.: face parsing with roi tanh-warping. in: 2019 ieee/cvf conference on computer vision and pattern recognition (cvpr), pp. 5647–5656 (2019) google scholar  liu, j., jung, c.: facial image inpainting using multi-level generative network. in: proceedings ieee international conference on multimedia and expo, vol. 2019-july, pp. 1168–1173. ieee computer society (2019) google scholar  liu, j., jung, c.: facial image inpainting using attention-based multi-level generative network. neurocomputing 437, 95–106 (2021) article  google scholar  luo, x., he, x., qing, l., chen, x., liu, l., xu, y.: eyesgan: synthesize human face from human eyes. neurocomputing 404, 213–226 (2020) article  google scholar  maggipinto, m., masiero, c., beghi, a., susto, g.a.: a convolutional autoencoder approach for feature extraction in virtual metrology. procedia manufacturing 17, 126–133 (2018). 28th international conference on flexible automation and intelligent manufacturing (faim2018), 11–14 june 2018, columbus, oh, usaglobal integration of intelligent manufacturing and smart industry for good of humanity google scholar  maharjan, r., ud din, n., yi, j.: image-to-image translation based face de-occlusion. in: jiang x., f.h. (ed.) proceedings of spie the international society for optical engineering, vol. 11519. spie (2020) google scholar  maheshwari, u., turlapati, v., kiruthika, u.: lucid-gan: an adversarial network for enhanced image inpainting. in: civemsa 2021 ieee international conference on computational intelligence and virtual environments for measurement systems and applications, proceedings. institute of electrical and electronics engineers inc. (2021) google scholar  mathai, j., masi, i., abdalmageed, w.: does generative face completion help face recognition? in: 2019 international conference on biometrics, icb 2019. institute of electrical and electronics engineers inc. (2019) google scholar  maulana, a., fatichah, c., suciati, n.: facial inpainting using generative adversarial network with feature reconstruction and landmark loss to preserve spatial consistency in unaligned face images. int. j. intell. eng. syst. 13(6), 219–228 (2020) google scholar  miyato, t., kataoka, t., koyama, m., yoshida, y.: spectral normalization for generative adversarial networks (2018) google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: navab, n., hornegger, j., wells, w.m., frangi, a.f. (eds.) miccai 2015. lncs, vol. 9351, pp. 234–241. springer, cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28 chapter  google scholar  sadiq, m., shi, d.: attentive occlusion-adaptive deep network for facial landmark detection. pattern recognit. 125, 108510 (2022) article  google scholar  wang, f., li, w., liu, y., gong, y., gao, z., lu, j.: face inpainting combining structured forest edge information and gated convolution. in: proceedings 2021 3rd international conference on natural language processing, icnlp 2021, pp. 213–217. institute of electrical and electronics engineers inc. (2021) google scholar  wu, y., singh, v., kapoor, a.: from image to video face inpainting: spatial-temporal nested gan (stn-gan) for usability recovery. in: proceedings 2020 ieee winter conference on applications of computer vision, wacv 2020, pp. 2385–2394. institute of electrical and electronics engineers inc. (2020) google scholar  yu, j., lin, z., yang, j., shen, x., lu, x., huang, t.: free-form image inpainting with gated convolution. in: proceedings of the ieee international conference on computer vision, vol. 2019-october, pp. 4470–4479. institute of electrical and electronics engineers inc. (2019) google scholar  yu, j., lin, z., yang, j., shen, x., lu, x., huang, t.: generative image inpainting with contextual attention. in: proceedings of the ieee computer society conference on computer vision and pattern recognition, pp. 5505–5514. ieee computer society (2018) google scholar  yu, l., zhu, d., he, j.: semantic segmentation guided face inpainting based on sn-patchgan. in: proceedings 2020 13th international congress on image and signal processing, biomedical engineering and informatics, cisp-bmei 2020, pp. 110–115. institute of electrical and electronics engineers inc. (2020) google scholar  zhang, h., li, t.: semantic face image inpainting based on generative adversarial network. in: proceedings 2020 35th youth academic annual conference of chinese association of automation, yac 2020, pp. 530–535. institute of electrical and electronics engineers inc. (2020) google scholar  zhang, h., goodfellow, i., metaxas, d., odena, a.: self-attention generative adversarial networks. in: chaudhuri, k., salakhutdinov, r. (eds.) proceedings of the 36th international conference on machine learning. proceedings of machine learning research, vol. 97, pp. 7354–7363. pmlr (2019) google scholar  zhang, r., isola, p., efros, a.a., shechtman, e., wang, o.: the unreasonable effectiveness of deep features as a perceptual metric. in: 2018 ieee/cvf conference on computer vision and pattern recognition, pp. 586–595 (2018) google scholar  zhu, w., wang, x., wu, y., zou, g.: a face occlusion removal and privacy protection method for iot devices based on generative adversarial networks. wirel. commun. mob. comput. 2021 (2021) google scholar  download references author information authors and affiliations university of são paulo, são paulo, sp, 03828-000, brazil victor ivamoto, rodolfo simões, bruno kemmer & clodoaldo lima authors victor ivamotoview author publications search author on:pubmed google scholar rodolfo simõesview author publications search author on:pubmed google scholar bruno kemmerview author publications search author on:pubmed google scholar clodoaldo limaview author publications search author on:pubmed google scholar corresponding author correspondence to clodoaldo lima . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi 1 electronic supplementary material below is the link to the electronic supplementary material. supplementary material 1 (pdf 535 kb) rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ivamoto, v., simões, r., kemmer, b., lima, c. (2023). occluded face in-painting using generative adversarial networks—a review. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_17 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords face de-occlusion facial inpainting gan generative adversarial network biometry review publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature slot sharing mechanism in multi-domain dialogue systems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper slot sharing mechanism in multi-domain dialogue systems conference paper first online: 28 november 2021 pp 94–108 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) slot sharing mechanism in multi-domain dialogue systems download book pdf download book epub bruno eidi nishimoto  orcid: orcid.org/0000-0002-8473-553510,11 & anna helena reali costa  orcid: orcid.org/0000-0001-7309-452810  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 755 accesses abstract task-oriented dialogue systems are very important due to their wide range of applications. in particular, those tasks that involve multiple domains have gained increasing attention in recent years, as the actual tasks performed by virtual assistants often span multiple domains. in this work, we propose the divide-and-conquer distributed architecture with slot sharing mechanism (dcda-s2m) system, which includes not only a distributed system aimed at managing dialogues through the pipeline architecture, but also a slot sharing mechanism that allows the system to obtain information during a conversation in one domain and reuse it in another domain, in order to avoid redundant interactions and make the dialogue more efficient. results show that the distributed architecture outperforms the centralized one by 21.51% and that the slot sharing mechanism improves the system performance both in the success rate and in the number of turns during the dialogue, demonstrating that it can prevent the agent from requesting redundant information. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others open-domain dialogue management framework across multiple device for long-term interaction chapter © 2023 a multiagent-based technique for dialog management in conversational interfaces chapter © 2016 exploring dialog act recognition in open domain conversational agents chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. adas, connectivity, e/e computational platforms and environments crispr-cas systems multiagent systems register-transfer-level implementation theory and algorithms for application domains 1 introduction dialogue systems aim to interact with humans, employing conversations in natural language. they can be broadly divided into three categories: socialbots, question & answering, and task-oriented systems. socialbots aim to have an entertaining conversation to keep the user engaged, without having a specific goal other than being friendly and keeping company. question & answering systems aim to provide a concise and straightforward answer to the user’s question, possibly using information stored in knowledge bases. finally, task-oriented systems help users to complete a specific task [6]. these tasks range from simple tasks, such as setting an alarm and making an appointment, to more complex tasks, such as finding a tourist attraction, booking a restaurant or taking a taxi. due to their wide range of applications, task-oriented systems have been showing great relevance in recent years, with both academia and industry drawing their attention to them. there are roughly two types of architectures to model a task-oriented agent: end-to-end and pipeline (or modular) [24]. end-to-end approaches consider the system composed of a single component that maps user input in natural language directly to the system output also in natural language (see fig. 1). on the other hand, pipeline architectures comprise three components: natural language understanding (nlu), dialogue management (dm) and natural language generation (nlg) (see fig. 2). nlu extracts the major information from the user utterance and transforms it in a structured data known as dialogue act. the dialogue act shows relevant information for the dialogue comprehension and it is defined by the tuple [domain, intent, slot, value], encoding the domain of this particular act, the intent, i.e., its broad objective (inform, request, or thanks, for instance), and the slot and value representing specific pieces of information in this domain. the example illustrated in fig. 2 shows that utterance i want a restaurant located in the centre of town would have the following dialogue act: [restaurant, inform, area, centre]. dm contains two sub-modules, the dialogue state tracking (dst) which keeps track of the dialogue state and the policy (pol) which decides the best response to give to the user. finally, the nlg component transforms the dm output, which is also a dialogue act, into natural language to present to the user. fig. 1. illustration of end-to-end architecture. full size image fig. 2. illustration of pipeline architecture. full size image early works on task-oriented dialogue systems [9, 12, 13] focus on problems with a single domain. however, it is common that real systems such as the popular virtual assistants span their applications over multiple domains. this means that a single system must be able to act and complete the user task in more than one domain, for example, the user first asks to find a tourist attraction for sightseeing and then, in the same conversation, she requests a restaurant to have lunch. dealing with multi-domain dialogue systems is a problem much harder since the complexity of user goals and conversations increases a lot. besides the complexity of the problem, which is already challenging, another issue faced by multi-domain systems is related to the redundancy of information given by the user during a conversation [15]. for instance, suppose that the user first reserves a restaurant table for two people and then she requests to reserve a hotel room for that night. it is usual that a single-domain dialogue system asks again the number of people although this kind of information—that can be shared among the domains—was already given by the user. these redundant turns make the conversation longer and are not nice for the user experience [14]. in this paper, we use the pipeline architecture with a focus on the dm component and adopt the divide-and-conquer approach, that is, several agents are trained independently, each one in a specific domain, and aggregate them to act in the multi-domain scenario. we show that this approach leads to better results when compared to the same algorithm trained in all domains at once. furthermore, we propose the use of a mechanism that enables the system to reuse some shareable slots among the domains, avoiding the agent to ask for redundant information. the remainder of this paper is organized as follows: sect. 2 shows some related work, sect. 3 describes our methods and proposal, sect. 4 shows our experiments and results obtained, and finally sect. 5 highlights our conclusions and directions for further work. 2 related work there have been some efforts in the past that focused on multi-domain task-oriented dialogue systems. komatani et al. [11] proposed a distributed architecture to integrate expert dialogue systems in different domains using a domain selector trained with a decision tree classifier. further works employed traditional reinforcement learning to learn the domain selector [23]. however, these systems require manual feature engineering for their building. finally [4] proposed to use deep reinforcement learning to allow training the system using raw data, without the manual feature engineering. more recent works give a great attention to centralized systems, i.e., a unique system capable of handling multiple-domains instead of having multiple agents, each one specialized for each domain. one reason for this is the increase in the power processing in modern computers. traditional works using the end-to-end architectures rely on the use of recurrent neural networks (rnn) with a sequence-to-sequence approach [2, 19]. recurrent embedding dialogue policy (redp) [21] learns vector embeddings for dialogue acts and states showing it can adapt better to new domains than the usual rnns. vlasov et al. [22] proposed to use a transformer architecture [20] with the self-attention mechanism operating over the sequence of dialogue turns outperforming previous models based on rnns. another line of research focus on the dm module using the pipeline architecture. there are some attempts to use supervised learning to learn a policy for the dm, but as it can be seen as a sequential decision making problem, rl is more used [5]. however, rl algorithms are too slow in general when trained from scratch. many works attempt to include some expert knowledge either by supervised pre-training or by warm-up, i.e., pre-filling the replay buffer with rule agents in dqn algorithms. dqfd (deep q-learning from demonstrations) uses the expert demonstrations to guide the learning and encourages the agent to explore in high reward areas, avoiding random exploration [7, 8]. redundancy with respect to the overlapping slots between domains is another issue in dialogue systems. chen et al. [3] address this problem by implementing the policy with a graph neural network where the nodes can communicate to each other to share information but they assume the adjacency matrix for this communication is known. although recent work on multi-domain settings does not consider a distributed architecture, it is relevant for two reasons: it can reuse well-established algorithms for a single domain and, in the need to add a new domain to the system – which is common for real applications such as virtual assistants – it is unnecessary to retrain the entire system. therefore, we adopted a distributed architecture using the divide-and-conquer approach. furthermore, to the best of our knowledge, it is the first work that focuses on learning this slot-sharing mechanism. 3 proposal in this work we propose the divide-and-conquer distributed architecture with slot sharing mechanism (dcda-s2m) showed in fig. 3, which uses the pipeline architecture (fig. 2) primarily focusing on the dm component. the dialogue state must encode all useful information collected during interactions. the multiwozfootnote 1 annotated states basically comprise the slots informed, and the ones that are required to complete the task in each domain. for example, the hotel domain requires, among others, the area of the city where the hotel is located, the number of stars it has been rated, as well as its price range. our proposal to train pol is divided into two steps. the first step is to use a distributed architecture in a divide-and-conquer approach to build a system capable of interacting in a multi-domain environment. the second stage consists of a mechanism that shares slots between domains. in the following sections we detail each component of dcda-s2m, the divide-and-conquer distributed architecture (dcda) and the slot sharing mechanism (s2m). fig. 3. illustrative figure of our proposed architecture. full size image 3.1 divide-and-conquer distributed architecture we implemented seven agents, each one specifically trained for a domain of the multiwoz dataset (attraction, hospital, hotel, police, restaurant, taxi, and train), as illustrated in fig. 3. the idea is that, using a simple reinforcement learning algorithm, we can have a multi-domain system with better performance than if we had a single agent trained in all domains at once. however, just having multiple agents, each one for each domain, is not enough. we need a controller capable of perceiving when there is a domain change and selecting the right agent to collect the right response. the controller keeps track of all state features from all agents allowing it to know the past and current domains. as mentioned before, in multi-domain systems the dialogue act is defined by the tuple [domain, intent, slot, value], i.e., it already includes the domain of the conversation [10]. therefore, the controller simply observes the domain element of the dialogue act. if it is a new domain (different from the current one) it checks with s2m (detailed in sect. 3.2) the slots from past domains that can share values with the new domain, and it copies the values of these shareable slots. given this, the controller sends the state features to the corresponding agent of the current domain. each agent is trained using reinforcement learning to learn an optimal policy (pol component of dm) in each domain. reinforcement learning. in reinforcement learning problems there is an agent that learns by interacting with an environment through the markov decision process (mdp) framework defined as a tuple \((\mathcal {s}, \mathcal {a}, \mathcal {r}, \mathcal {t}, \gamma )\), where \(\mathcal {s}\) is the set of possible states, \(\mathcal {a}\) is the set of actions, \(\mathcal {r}\) is the reward function, \(\mathcal {t}\) is the state transition function, and \(\gamma \in [0,1]\) is the discount factor that balances the trade-off between immediate rewards and future rewards. in the context of dialogue systems, the state represents the dialogue state. the actions are the set of dialogue acts of the agent, and the reward is \(+r\) if the dialogue succeeds, i.e., the agent achieve the user goal, \(-r\) if the dialogue fails and \(-1\) for each turn to encourage the agent lasts the minimum number of turns. the agent’s objective is to maximize the cumulative discounted rewards, \(\sum _{t = 0}^{t}\gamma ^tr_t\), \(r_t \in \mathcal {r}\), received during interactions in order to find its optimal policy that maps state \(s_t \in \mathcal {s}\) to the best action \(\pi (s_t) = a_t\), \(a_t \in \mathcal {a}\). in this work the proximal policy optimization (ppo) algorithm [18] was employed which is a policy gradient method which aim to directly optimize the parameterized policy to maximize the expected reward. the intuition of ppo is to make the greatest improvement in policy without stepping so far from current policy to avoid a performance collapse. formally, it optimizes the loss function, \( \mathcal {l}(\theta ) = \min \left( \rho (\theta )\hat{a},\ clip(\rho (\theta ), 1 \epsilon , 1 + \epsilon )\hat{a}\right) , \) where \(\rho (\theta ) = \frac{\pi _\theta (a | s)}{\pi _{\theta _{old}(a | s)}}\) denotes the probability ratio, \(\hat{a}\) is the estimated advantage function and \(\epsilon \) is a hyperparameter that indicates how far can we go from old policy. for the advantage function estimation, we use the generalized advantage estimation (gae) [17], \( \hat{a}(s_t, a_t) = \delta (s_t) + \gamma \lambda \hat{a}(s_{t+1}, a_{t+1}), \) with \( \delta (s_t) = r_t + \gamma \hat{v}(s_{t+1}) \hat{v}(s_t), \) where \(\gamma \) is the discount factor, \(\lambda \) is a hyperparameter to adjust the bias-variance tradeoff and \(\hat{v}(s)\) is the estimate of value function, i.e., the expected reward the agent receives being at state s. however reinforcement learning algorithms often suffer in sparse reward environments. one possible option to deal with this issue is to use some pre-training method, such as imitation learning, where the agent tries to clone expert behaviors. we use the vanilla maximum likelihood estimation (vmle) algorithm [25] to pre-train the agents. vmle employs a multiclass classification with data extracted from the multiwoz dataset. the algorithm optimizes its policy trying to mimic the behavior presented by the agent in the dataset. user simulator. furthermore, training the agent with real users is impracticable since it requires a great number of interactions. therefore there is a need of an user simulator. the used user simulator follows an agenda-based approach [16]. first, it generates a user goal which comprises all information needed to complete the task. then, the user simulator generates an agenda in a stack-like structure with all actions it needs to take (informing its constraints and/or requesting information). during the conversation, as the agent requests or informs something, the user can reschedule the agenda accordingly. for example, if the agent requests the type of restaurant, than the user can move the action “inform the type of restaurant” to the top of the stack. the conversation lasts until the stack is empty or the maximum number of turns is reached. we made a small modification to the user simulator to handle the confirmation actions provided by the agent using the slot sharing mechanism. the simulator checks if the values of the confirmation act are correct and if they are wrong it informs the correct value; otherwise, it just removes the action regarding that slot-value pair from the agenda and continues with its policy. finally, after all the seven agents are trained using reinforcement learning we plug them with the controller resulting in the dcda system. 3.2 slot sharing mechanism some domains contain overlapping slots, i.e., slots that can share their values in a conversation. for instance, if the user is looking for a restaurant and a hotel, it is likely that they are for the same day and in the same price range. more complex relationships can be represented, such as the time of reservation for a restaurant and the time when the taxi must arrive at its destination (which would be the restaurant). however, not all slots are shareable between two domains. therefore, we need to know which slots whose contents can be transferred from one domain to another. in the following subsections we show how we learned these relationships. but for now, suppose we already know what such slots are. therefore, during the conversation, when the controller notices a domain change, the information-sharing mechanism first gets all informed slots in previous domains and then checks whether any of those slots can share their values with any slots in the current domain. in this case, the controller copies the value from that slot to the slot of the new domain. for example, suppose the system has already interacted with the user in the hotel domain and knows that the demand is for two people. when the conversation changes to the restaurant domain, the slot sharing mechanism will see that hotel-people slot can be shared with restaurant-people slot and the controller will transfer this information (two people) to the restaurant domain. the agent then asks for confirmation and acts considering this transferred slot. this can speed up the dialogue and improve the user experience during interactions by avoiding asking redundant information. in the worst case, if the transferred value is wrong, the user informs the correct value and continues the interaction normally. learning shareable slots. our proposal to learn which slots can be shared, named node embedding for slots sharing mechanism, uses the node embedding technique in which each node represents a domain-slot pair. the similarity of two nodes indicates whether they can share the same value in a conversation. it is defined by a simple scalar product, that is, given nodes \(u = [u_1, u_2, \ldots , u_d]\) and \(v = [v_1, v_2, \ldots , v_d]\), we have: \( similarity(u, v) = \langle u, v\rangle = \sum _{i=1}^d u_i\cdot v_i, \) where d is the embedding dimension. for instance, the nodes restaurant-day and hotel-day must be similar, i.e., have a high scalar product, while restaurant-name and hotel-name must have a low scalar product. before learning the node embedding, we need to build a similarity matrix \(a \in \mathbb {r}^{n\times n}\), a \(n\times n\) matrix where n is the number of nodes, i.e., number of domain-slot pairs and the cell \(a_{uv}\) represents the similarity between nodes u and v normalized to fall between 0 and 1, that is, \(similarity(u, v) \in [0, 1]\). this is done using the dialogues from dataset \(\mathcal {d}\) as shown in algorithm 1. at the end of each conversation, we observe the final state (the state of the dialogue in the last interaction) and check if each node pair presents the same value (a node represents a domain-slot pair). if they do, the weight between these two nodes is increased by one. in the end, all weights are normalized to the number of times each pair of nodes appeared in the dialogues. however, keeping a matrix with \(\mathcal {o}(n^2)\) of space complexity does not scale with the number of domains and slots. for this reason, we trained a node embedding representation for each domain-slot pair. the learning of node embedding uses the similarity matrix \(a \in \mathbb {r}^{n\times n}\) and follows algorithm 2 proposed by ahmed et al. [1]. in each step, for each node pair \((u, v) \in e\), where e is the set containing all node pairs, it performs an update to minimize the following error \( l(a, z, \lambda ) = \frac{1}{2}\sum _{(u,v)\in e}(a_{uv} -\langle z_u, z_v \rangle )^2 + \frac{\lambda }{2}\sum _u||z_u||^2, \) where z represents the embedding space and \(z_u\) is the vector for node u. the t in algorithm 2 can be thought as a learning rate for each update. this s2m is independent of the agent trained, so it can be plugged with any agent we want. gathering the dcda and s2m we get the dcda-s2m system (fig. 3), where the controller asks for the s2m for slots that can share their values when there is a domain shift and then send the state features to the respective agent to get the response to the user. 4 experiments and results to evaluate our proposal we did three experiments: the first was to train the embedding of nodes and to do a qualitative analysis of the learned embedding. we then evaluate the divide-and-conquer approach with the information-sharing mechanism, and finally we evaluate the same approach without the mechanism. 4.1 experimental setup we used the convlab-2footnote 2 platform to run our experiments. it provides a platform with a user simulator and some implementations of all dialogue systems components (nlu, dst, pol and nlg). in this way, it is easy to assess new algorithms for each of the components. we trained the agents for each domain using the ppo algorithm with the standard parameters of the convlab-2 platform: discount factor \(\gamma = 0.99\), clipping factor \(\epsilon = 0.2\) and \(\lambda = 0.95\) (for advantage value estimation). the training lasted 200 epochs and in each epoch we collected around 100 turns and sampled a batch of size 32 for optimization. the agent contains two separate networks, one for policy estimation with two hidden layers of size 100 and other for value estimation with two hidden layers of size 50. the optmizers used for policy and value networks are rmsprop and adam and learning rate \(lr_p = 10^{-4}\) and \(lr_v = 5\cdot 10^{-5}\), respectively. the reward function is \(-1\) for each turn (to encourage the agent complete the task more quickly), 40 for success dialogue and 20 for fail dialogue. for pre-training we employed the vmle algorithm using rmsprop optimizer with \(lr_{vmle} = 10^{-3}\) and binary cross entropy with logits as loss function. we also used the available ppo model trained with all domains at once to compare with our results. for our node embedding, we built the adjacency matrix using the dialogue corpus available at convlab-2 platform. for hyperparameters, we used an embedding dimension \(d = 50\), regularization factor \(\lambda = 0.3\), and 1000 epochs for training. 4.2 node embedding for visualization of the learned node embedding we used a t-distributed stochastic neighbor embedding (t-sne) model with perplexity 5 using the scalar product as similarity function. figure 4 shows this visualization. fig. 4. visual representation of the learned node embedding. full size image figure 4 clearly shows some groups of nodes that are related to each other. for example, hotel-area, attraction-area, and restaurant-area forms a group, indicating that users generally request places in the same area. it also happens for the price range (restaurant-pricerange and hotel-pricerange), day (restaurant-day, hotel-day, and train-day) and people (restaurant-people, hotel-people, and people) slots. although hotel-stay and hotel-stars looks close to the group with slot “people”, computing their similarity with restaurant-people we got 0.118 and 0.088, respectively. thus they are not similar and should not share values. on the other hand, the similarity between restaurant-area and attraction-area is 0.91 showing that they are similar and must share their values inside a conversation. here we used a similarity of 0.8 as a threshold for sharing the slots values. an interesting observation is that attraction-name and hotel-name are quite close to taxi-departure, with similarity 0.57 and 0.668, respectively, but they are not close to each other, i.e., the similarity between them is 0.011. this is expected since it is not common an attraction with the same name as a hotel. 4.3 dcda evaluation to evaluate our proposal we assessed four models: the baseline rule-based policy available in convlab-2, the vlme policy (obtained from the vlme algorithm), and the ppo algorithm trained in both approaches: a centralized system with a unique agent trained to handle all domains at once (ppo\(_{all}\)) and our proposal (dcda). we also evaluated the effects of using or not the s2m in the rule and dcda agents. the metrics are automatically computed by the evaluator presented in convlab-2 and encompasses the complete rate, success rate, book rate, precision, recall, f1-score for the informed slots, and average number of turns for both the dialogues that were successful and the total set of dialogues. the complete rate indicates the rate of dialogues that could finish (either with success or fail) before achieving the maximum number of turns. the precision, recall and f1-score indicate the ability of the agent to fulfill the slots of the user goal, i.e., leads to the correct slot. tests were performed over 2000 dialogues. table 1 shows the evaluation results for all the four models trained in the pipeline setting, i.e., without the nlu and nlg modules. as expected the rule policy performs almost “perfectly” succeeding in 98.45% of the dialogues and it can serve as a baseline. among the trainable agents, dcda performs better in almost all aspects achieving 88.14% of success rate, 94.01% of completion rate, and 88.01% of book rate. it beats the ppo\(_{all}\) by almost 21.51% in the success rate, showing a much better performance and efficiency as it can solve user tasks using less number of turns. the average number of turns in all dialogues, 14.92, is very close to the baseline rule policy (13.48) showing it could learn a very good policy in solving tasks. the large increase in the average number of turns for all dialogues can be explained by analysing the failed dialogues during test. it can be seen that in many failed dialogues the conversation went in a loop with the agent and the simulated user, repeating the same act of dialogue consecutively. the reason why this phenomenon occurs is not very clear to us. the worst performance of the vlme is expected, as the other agents depend on the vlme for pre-training. table 1. results of the four agents: rule, vlme, ppo\(_{all}\) and dcda tested in a pipeline setting. best results among trainable agents are in bold.full size table table 2. evaluation of the use of s2m in the rule policy and dcda with the goal generator generating random goals. best results are in bold.full size table the results of the second experiment regarding the use of the slot sharing mechanism are presented in table 2. we evaluated both the rule policy and our proposed model dcda. results show that for the rule policy the sharing mechanism also helped the agent to have a slightly better performance. although the success rate for dcda did not change much, the sharing mechanism also helped it to have a better complete and book rate. another enhancement was in the average number of turns. the average number of turns required in successful dialogues for the rule and dcda policies decreased from 13.40 to 13.20 and from 13.84 to 13.43, respectively, when the sharing mechanism was incorporated. thus we can see that the sharing mechanism makes the agent to complete dialogues faster than without this mechanism for both agents. an interesting fact is that besides the slightly better performance with the sharing mechanism, the precision, recall and f1-score did not followed the same behavior, i.e., they had better results or very close (less than 0.05%) results as those without the sharing mechanism. this result is not very surprising because as the agent with the sharing mechanism tries to “guess” the slots of new domains within the conversation, it ends up reporting more wrong slots of the user goal causing worse precision, recall, and f1-score. all theses experiments was assessed with the user simulator generating random goals based on a distribution of the goal model extracted from the dataset. so this can include simple goals within only one domain and/or goals that span to more than one domain but do not have any slot with the same value. indeed, among all 2000 goals generated during testing, only about 400 contain common values between slots. with that in mind, we ran another test of the sharing mechanism that restricts the user simulator to only generating goals that contain common slots. therefore, the generated goals end up being more complex in general than those generated in the first test. table 3 shows the results. there is an expected significant decrease in the general performance due to the increase in user goals complexity. however, here we can clearly observe the great advantage of the sharing mechanism in this setting. table 3. evaluation of the use of s2m in the rule policy and dcda with the goal generator generating slots with common values. best results are in bold.full size table there is a 12.25% and 9.99% success rate difference with the rule and dcda policies, respectively. we also see a bigger impact on the average number of turns. it affects mostly the successful dialogues because the number of turns is affected only when the transferred slots values are correct – otherwise the user would still need to inform these slots – and chances of a successful dialogue increase when it happens. finally, we also see a better precision, recall and f1-score for the agent with the sharing mechanism. since all goals in this tests have at least one common value among the slots, the agent “guesses” are more likely to be correct. fig. 5. example of dialogue using the slot sharing mechanism, resulting in a dialogue length 8. full size image fig. 6. example of dialogue that does not use slot sharing mechanism, resulting in a dialogue length 11. full size image figures 5 and 6 show examples of system generated dialogues using and not using the slot sharing mechanism, respectively. observe that when the domain switched to the hotel domain, the agent in fig. 5 asked for confirmation if the price is moderate and area is north and recommended a hotel with these constraints. in natural language we could think in this dialogue act as: “do you want a hotel in north with a moderate price, right? there is the hotel limehouse”. in this way, the user did not need to inform these slots again, saving some turns until task completion. while in fig. 6 the agent needed to ask again the area and price for the user, resulting in a redundant dialogue which takes more turns to be completed (11 turns against 8 turns). one drawback for the dcda-s2m is the training time required for training all the agents. table 4 shows the average training time for each agent. as we can see, the total amount of time required to train all seven agents is 291.29 minutes, which is approximately 16% more than centralized system training. however, it is worth noting that agents could be trained in parallel, which would require greater computational power. table 4. training time in minutes for each agent.full size table 5 conclusions in this work we show that the use of a distributed architecture, with multiple agents trained separately for each domain, can leverage the system performance compared to the same algorithm used to train a single agent for all domains at once. this is because each agent can specialize in solving its own problem well, which is much simpler than solving tasks well in all domains, as with the centralized approach in a single agent. furthermore, distributed systems can add new domains without the need to retrain the entire system. the use of the slot sharing mechanism also proved to enhance system performance, especially for tasks where the goal has some common slot across domains. besides improving the system’s success rate, it also decreases the average number of turns, showing that the system avoided asking for redundant information. a major disadvantage of dcda-s2m is the need to train several agents separately and this can be time and energy consuming. in this sense, for future work we intend to explore transfer learning techniques in reinforcement learning to accelerate the training of new agents. notes 1.multiwoz is a fully-labeled collection of human-human written conversations spanning over multiple domains and topics – https://github.com/budzianowski/multiwoz. 2.https://github.com/thu-coai/convlab-2. references ahmed, a., shervashidze, n., narayanamurthy, s., josifovski, v., smola, a.j.: distributed large-scale natural graph factorization. in: proceedings of the 22nd international conference on world wide web, pp. 37–48 (2013) google scholar  bordes, a., weston, j.: learning end-to-end goal-oriented dialog. in: 2017 international conference on learning representations (iclr) (2017) google scholar  chen, l., chen, z., tan, b., long, s., gasic, m., yu, k.: agentgraph: towards universal dialogue management with structured deep reinforcement learning. arxiv:abs/1905.11259 (2019) cuayáhuitl, h., yu, s., williamson, a., carse, j.: scaling up deep reinforcement learning for multi-domain dialogue systems. in: 2017 international joint conference on neural networks (ijcnn), pp. 3339–3346 (2017) google scholar  dai, y., yu, h., jiang, y., tang, c., li, y., sun, j.: a survey on dialog management: recent advances and challenges. arxiv:abs/2005.02233 (2020) gao, j., galley, m., li, l.: neural approaches to conversational ai. found. trends® inf. retr. 13, 127–298 (2019) article  google scholar  gordon-hall, g., gorinski, p.j., cohen, s.b.: learning dialog policies from weak demonstrations. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 1394–1405 (july 2020) google scholar  gordon-hall, g., gorinski, p.j., lampouras, g., iacobacci, i.: show us the way: learning to manage dialog from demonstrations. arxiv:abs/2004.08114 (2020) ilievski, v., musat, c., hossmann, a., baeriswyl, m.: goal-oriented chatbot dialog management bootstrapping with transfer learning. in: proceedings of the twenty-seventh international joint conference on artificial intelligence (2018) google scholar  kim, y., lee, s., stratos, k.: onenet: joint domain, intent, slot prediction for spoken language understanding. in: 2017 ieee automatic speech recognition and understanding workshop (asru), pp. 547–553 (2017) google scholar  komatani, k., et al.: multi-domain spoken dialogue system with extensibility and robustness against speech recognition errors. in: proceedings of the 7th sigdial workshop on discourse and dialogue, pp. 9–17 (july 2006) google scholar  li, x., chen, y.n., li, l.: end-to-end task-completion neural dialogue system. in: proceedings of the the 8th international joint conference on natural language processing, pp. 733–743 (2017) google scholar  peng, b., li, x., gao, j., liu, j., wong, k.f., su, s.y.: deep dyna-q: integrating planning for task-completion dialogue policy learning. in: acl (2018) google scholar  peng, b., et al.: composite task-completion dialogue policy learning via hierarchical deep reinforcement learning. in: proceedings of the 2017 conference on empirical methods in natural language processing, pp. 2231–2240 (2017) google scholar  saha, t., gupta, d., saha, s., bhattacharyya, p.: towards integrated dialogue policy learning for multiple domains and intents using hierarchical deep reinforcement learning. expert syst. appl. 162, 113650 (2020) article  google scholar  schatzmann, j., young, s.: the hidden agenda user simulation model. ieee trans. audio speech lang. process. 17, 733–747 (2009) article  google scholar  schulman, j., moritz, p., levine, s., jordan, m., abbeel, p.: high-dimensional continuous control using generalized advantage estimation. arxiv:abs/1506.02438 (2015) schulman, j., wolski, f., dhariwal, p., radford, a., klimov, o.: proximal policy optimization algorithms. arxiv:abs/1707.06347 (2017) serban, i.v., sordoni, a., benggio, y., courville, a., pineau, j.: building end-to-end dialogue systems using generative hierarchical neural network models. in: proceedings of the thirtieth aaai conference on artificial intelligence, pp. 3776–3783 (2016) google scholar  vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, pp. 5998–6008 (2017) google scholar  vlasov, v., drissner-schmid, a., nichol, a.: few-shot generalization across dialogue tasks. corr:abs/1811.11707 (2018) google scholar  vlasov, v., mosig, j.e.m., nichol, a.: dialogue transformers. arxiv:abs/1910.00486 (2019) wang, z., chen, h., wang, g., tian, h., wu, h., wang, h.: policy learning for domain selection in an extensible multi-domain spoken dialogue system. in: proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pp. 57–67, doha, qatar (october 2014) google scholar  zhang, z., takanobu, r., zhu, q., huang, m., zhu, x.: recent advances and challenges in task-oriented dialog system. sci. china technol. sci. 63, 2011–2027 (2020) article  google scholar  zhu, q., et al.: convlab-2: an open-source toolkit for building, evaluating, and diagnosing dialogue systems. in: proceedings of the 58th annual meeting of the association for computational linguistics (2020) google scholar  download references acknowledgments the authors acknowledge the support from the conselho nacional de desenvolvimento científico e tecnológico (cnpq grant 310085/2020-9), the coordenação de aperfeiçoamento de pessoal de nível superior (capes finance code 001), and from itaú unibanco, through the scholarship program of programa de bolsas itaú (pbi). author information authors and affiliations universidade de são paulo, são paulo, brazil bruno eidi nishimoto & anna helena reali costa itaú unibanco, são paulo, brazil bruno eidi nishimoto authors bruno eidi nishimotoview author publications search author on:pubmed google scholar anna helena reali costaview author publications search author on:pubmed google scholar corresponding author correspondence to bruno eidi nishimoto . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper nishimoto, b.e., costa, a.h.r. (2021). slot sharing mechanism in multi-domain dialogue systems. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_7 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords dialogue systems reinforcement learning transfer publish with us policies and ethics profiles anna helena reali costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving a genetic clustering approach with a cvi-based objective function | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving a genetic clustering approach with a cvi-based objective function conference paper first online: 28 november 2021 pp 202–217 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) improving a genetic clustering approach with a cvi-based objective function download book pdf download book epub caio flexa10, walisson gomes10, igor moreira10, reginaldo santos10, claudomiro sales10 & … moisés silva10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 727 accesses abstract genetic-based clustering meta-heuristics are important bioinspired algorithms. one such technique, termed genetic algorithm for decision boundary analysis (gadba), was proposed to support structural health monitoring (shm) processes in bridges. gadba is an unsupervised, non-parametric approach that groups data into natural clusters by means of a specialized objective function. albeit it allows a competent identification of damage indicators of shm-related data, it achieves lackluster results on more general clustering scenarios. this study improves the objective function of gadba based on a cluster validity index (cvi) named mutual equidistant-scattering criterion (mec) to expand its applicability to any real-world problem. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a novel genetic algorithm with specialized genetic operators for clustering chapter © 2023 hybrid model for tasks scheduling in distributed real time system article 19 august 2020 hgasa: an improved hybrid self-adaptive metaheuristic approach for data clustering article 26 july 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence functional clustering genetic mapping genetic techniques bioinformatics genetic databases 1 introduction the world is increasingly filled with fruitful data, most of which daily stored in electronic media. as such, there is a high potential of technique research and development for automated data retrieval, analysis, and classification [15]. around 90% of the data produced up to 2017 were generated in 2015 and 2016, and the tendency is to biennially double this amount [17]. the exponential increase in size and complexity of big data are aspects worthy of attention. latent potentialities for decision-making based on insights learned from historical data are only actually exploited if pushed into practice. a comprehensive information extraction procedure is composed of two sub-processes: data management and data analysis [15]. management supports data acquisition, handling, storage, and retrieval for analysis [18, 29]; in contrast, data analysis refers to the evaluation and acquisition of intelligence from data. it was pointed out by [17] that researchers of various subjects have been adopting methods from machine learning (ml) [5, 10] and data mining (dm) [31]. many of them did so in big data contexts such as stock data monitoring, financial analysis, traffic monitoring, and structural health monitoring (shm) [14]. [6] demonstrated the relationships between the data management and analysis sub-processes in practice, with a specific focus on the application herein highlighted. on one hand, this work claims that big data is not restricted to a computerized manipulation of massive data streams; on the other hand, it emphasizes that shm can learn ipsis litteris with the conscientious use of ml. the problem of data grouping (i.e., data clustering) is one of the main tasks in ml [2, 5] and dm [31], prevailing in any discipline involving multivariate data analysis [21]. it gained a prominent place in many applications lately, especially in speech recognition [11], web applications [35], image processing [3], outlier detection [23], bioinformatics [1], and shm [9]. a wide variety of genetic algorithm (ga)-based clustering techniques have been proposed in recent times [25, 28, 42]. their search ability is commonly exploited to find suitable prototypes in the feature space such that a per-chromosome measure of the clustering results is optimized in each generation. in [2], two conflicting functions were proposed and defined based on cluster cohesion and connectivity. the goal was to reach well-separated, connected, and compact clusters by means of two criteria in an efficient, multi-objective particle swarm optimization algorithm. more recently, [20] combined k -means and a ga through a differentiate arrangement of genetic operators to conglomerate different solutions, with the intervention of fast hill-climbing cycles of k -means. an unsupervised, non-parametric, ga-based approach to support the shm process in bridges termed genetic algorithm for decision boundary analysis (gadba), which was proposed by [36], aims to group data into natural clusters. the algorithm is also supported by a method based on spatial geometry to eliminate redundant clusters. upon testing, gadba was more efficient in the task of fitting the normal condition than its state-of-the-art counterparts in shm contexts. however, due to the specialization of its objective function to shm contexts, gadba is lackluster on more general clustering scenarios. this work aims to improve the objective function of gadba to expand its application potential to a wider range real-world problems. in this sense, a version of gadba based on the mutual equidistant-scattering criterion (mec) is proposed as a general-purpose clustering approach. four clustering algorithms are compared against the new proposal: k -means [26], gaussian mixture models (gmm) [30], linkage [22], and gadba, of which the first three are well-known and explored in literature. the remaining sections of this paper are divided as follows. section 2 and sect. 3 respectively define gadba and mec. section 4 discusses the performance of the new proposal under some experimental evaluations. finally, sect. 5 summarizes and ends the paper. 2 genetic algorithm for decision boundary analysis given a minimum (\(k_{min}\)) and maximum (\(k_{max}\)) number of clusters, clustering is done by the combination of a ga to dispose their centroids in the m-dimensional feature space, and a method called concentric hypersphere (ch) to agglutinate clusters, to choose an appropriate \(k \in [k_{min}, k_{max}]\) [37]. the initial population \(\mathbf {p}(t{=}0)\) is randomly created and each individual represents a set of k centroids, where k is randomly selected. the chromosome is then formed by concatenating k feature vectors (fig. 1), whose values are initialized by randomly selecting k data points from the training set. a length ratio is defined as \(\gamma _{i} = k_{i} / k_{max}\), where \(k_{i}\) is the number of active centroids in individual i. the role of \(\gamma \) is to define the number of active centroids for a given candidate solution, since a single individual might have enabled/disabled centroids during the recombination process. fig. 1. chromosome organization in gadba. full size image the parent selection is based on tournament with reposition, where r individuals are randomly selected and the fittest one is chosen to recombine with other individual chosen in the same way. the recombination is conducted in three steps for each pair of parents \(p_i\) and \(p_j\) to generate a pair of descendants: 1. a random number \(r \in [0,1]\) is compared with \(p_{rec}\) defined a priori. if \(r \le p_{rec}\), then two cut points \(\pi _1\) and \(\pi _2\) are selected such that \(1 \le \pi _1 < \pi _2 \le min(k_i,k_j)\). the centroids in the range are then swapped. if \(r > p_{rec}\), the parents remain untouched. 2. likewise, two random numbers \(r, t \in [0,1]\) are picked for each centroid position in the parents and, if \(r \le p_{pos}\), defined a priori, an arithmetic recombination is conducted as follows: $$\begin{aligned} f_{\mathbf {x},t}^{i} = f_{\mathbf {x},t}^{i} + (f_{\mathbf {y},t}^{j}-f_{\mathbf {x},t}^{i})t, \end{aligned}$$ (1) $$\begin{aligned} f_{\mathbf {x},t}^{j} = f_{\mathbf {x},t}^{j} + (f_{\mathbf {y},t}^{i}-f_{\mathbf {x},t}^{j})t, \end{aligned}$$ (2) where \(f_{\mathbf {x},t}^{i}\) and \(f_{\mathbf {x},t}^{j}\) are the values in the tth position of \(\mathbf {x}\)th centroid from parents \(p_i\) and \(p_j\), respectively. similarly, \(f_{\mathbf {y},t}^{i}\) and \(f_{\mathbf {y},t}^{j}\) respectively correspond to the yth centroid of the i and jth parents. a pair of parents can recombine even if they have a different number of genes. 3. finally, the last step consists in arithmetically recombining the parents’ length ratio to define the length ratios of the offspring individuals. the mutation is the result of a personalized two-step process: 1. let \(t_{\mathbf {x}} = k_{max}^{-1}\) and \(t_r\) be a random number on interval [0, 1]. the number of centroids to be enabled in an offspring individual is . when \(k < k_{new} \le k_{max}\), the missing positions are filled with the information of \(k_{new}-k\) data points chosen at random. 2. each centroid position can be mutated with a probability \(p_{mut}\), defined a priori, in which a gaussian mutation is applied by using $$\begin{aligned} f_{\mathbf {x},t}^{j} = f_{\mathbf {x},t}^{j} + \mathcal {n}(0,1), \end{aligned}$$ (3) where \(\mathcal {n}(0,1)\) is a random number from a standard gaussian distribution and \(f_{\mathbf {x},t}^{j}\) is the value in the tth position of \(\mathbf {x}\)th centroid. survivor selection is based on elitism, where the parents \(\mathbf {i}^{(t)}_p\) and offspring \(\mathbf {i}^{(t)}_c\) are concatenated into \(\mathbf {i}^{(t+1)}_p = \mathbf {i}^{(t)}_p \cup \mathbf {i}^{(t)}_c\), which is then sorted according to a fitness measure based on pareto front and crowding distance. the new population \(\mathbf {p}(t+1)\) is composed by the \(|\mathbf {p}|\) best individuals [8]. parent selection, recombination, mutation, and survivor selection are repeated until a maximum number of iterations is reached and/or the difference of the current solution against the last one is smaller than a given threshold \(\epsilon \). as mentioned, the ch algorithm is used to regularize the number of clusters encoded in the individuals. it is executed in each individual prior to their evaluation by determining the regions that limit each cluster in three steps: 1. for each cluster, its centroid is dislocated to the mean of its data points. 2. each centroid is the center of a hypersphere whose radius will increase while the difference of density between two consecutive inflations is positive. 3. if more than one centroid is found inside a hypersphere, they are agglutinated into a centroid located at their mean point. 3 a new objective function 3.1 basic notations and definitions the objective of clustering is to find out the best way to split a given data set \(\mathcal {x} \in \mathbb {r}^{n \times m}\), with n input vectors in an m-dimensional real-valued feature space , into k mutually disjoint subsets (\(k \le n\)). assume the vectors in \(\mathcal {x}\) have hard labels marking them as members of one cluster. a set of prototypes \(\vartheta \) is described as a function of \(\mathcal {x}\) and k as $$\begin{aligned} \vartheta \in \mathbb {r}^{k \times m} = \vartheta (\mathcal {x}), \end{aligned}$$ (4) whereupon \(\vartheta \) contains k representative vectors . let the hard label for cluster \(\kappa \) be the prototypes are computed whereby n label vectors are organized into a partition over the vectors in \(\mathcal {x}\) such that $$\begin{aligned} u \in \mathbb {z}^{n \times m} = u(\mathcal {x}), \end{aligned}$$ (5) subject to $$\begin{aligned} u = \left[ \mu _{i \kappa }\right] _{n \times k} \in \{0,1\}, \quad i = 1,\dots ,n, \end{aligned}$$ where \(\mu _i = y_\kappa \leftrightarrow x_i\) is in cluster \(\kappa \). put another way, the membership of \(x_i\) to cluster \(\kappa \), \(\mu _{i \kappa }\), is either 1 if the ith object belongs to the \(\kappa \)th cluster, or 0 otherwise. accordingly, \(\mu _{i \kappa } = 1\) for one value of \(\kappa \) only, such that [41] the resulting grouping in this structure is hard [24], since one object belonging to one cluster cannot simultaneously belong to another. in general, data clustering involves finding \(\{u,\vartheta \}\) to partition \(\mathcal {x}\) somehow. for a given initial \(\vartheta \), the optimal set of prototypes can be represented by centroids, medians, medoids, and others, in which the optimized partition u is obtained by assigning each input vector to the cluster with the nearest prototype. both u and \(\vartheta \) comprise a dual structure (if one of them is known, the other one will also be) named clustering solution. 3.2 cluster validation most researchers have some theoretical difficulty in describing what a cluster is without assuming an induction principle (i.e., a criterion) [21]. a classic definition for them is: “objects are grouped based on the principle of maximizing intra-class similarity and minimizing inter-class similarity”. another cluster definition involving density defines it as a connected, dense component such that high-density regions are separated by low-density ones [1, 19]. in clustering algorithms, k is usually assumed to be unknown. since clustering is an unsupervised learning procedure (i.e., there is no prior knowledge on data distribution), the significance of the defined clusters must be validated for the data [33]. therefore, one of the most challenging aspects of clustering is the quantitative examination of clustering results [31]. this procedure is performed by cluster validity indices (cvis), sometimes called criteria, which also targets hard problems such as cluster quality assessment and the degree wherewith a clustering scheme fits into a specific data set. the most common application of cvis is to fine-tune k. given \(\mathcal {x}\), a specific clustering algorithm and a range of values of k, these steps are executed [10, 43]: 1. successively repeat a clustering algorithm according to a number of clusters from a fixed range of values defined a priori: \(k \in [k_{min},k_{max}]\); 2. obtain the clustering result \(\{u,\vartheta \}\) for each k in the range; 3. calculate the validity index score for all solutions; and 4. select \(k_{opt}\) for which data partitioning provides the best clustering result. cvis are considered to be independent of the clustering algorithms used [40] and usually fall into one of two categories: internal and external [27, 32]. internal validation does not require knowledge about the problem for it only uses information intrinsic to the data; hence, it has a practical appeal. conversely, external validation is more accurate, but not always feasible [32]. knowing this, it can be evaluated how well the achieved solution approaches a predefined structure based on previous and intuitive understanding regarding natural clusters. 3.3 mutual equidistant-scattering criterion this work proposes the replacement of the objective function of gadba with a cvi called mec [12]. mec is a non-parametric, internal validation index for crisp clustering. an immediate benefit of mec is the absence of fine-tuning hyper-parameters, thus mitigating the user’s effort in operational terms and enabling the use of gadba to cluster real-world data whose structure is unknown. mec assumes that “objects belonging to the same data cluster will tend to be more equidistantly scattered among themselves compared to data points of distinct clusters” [12]. as such, the mean absolute difference \(\mathcal {m}_\kappa \) is applied using multi-representative data in every clustering solution \(\{u,\vartheta \}\) obtained from a pre-determined k. mec is weighted by a penalty of local restrictive nature to each cluster \(\kappa \) as well, while a global penalty is then applied a posteriori. such penalties are a measure of intra-cluster homogeneity and inter-cluster separation. mathematical formulations. the mean absolute difference is calculated between any possible pair of intra-cluster dissimilarities (6) where \(n_\kappa \) objects within the cluster are considered as representative data in formulation (thereby multi-representative). that is, \(d_\kappa \) is a strictly upper triangular matrix of order \(n_\kappa \). nevertheless, only the pairwise distances matter in \(\mathcal {m}_\kappa \). thus, \(\varupsilon (\cdot )\) reshapes all elements above the main diagonal of \(d_\kappa \) (eq. 6) into a column vector $$\begin{aligned} \begin{bmatrix} d_{1 \kappa }\\ d_{2 \kappa }\\ \vdots \\ d_{l_\kappa \kappa } \end{bmatrix} = \varupsilon (d_\kappa ), \end{aligned}$$ (7) denoted by \(l_\kappa = \frac{n_\kappa (n_\kappa -1)}{2}\) intra-cluster euclidean distances. the key part of mec is then defined in eq. 8 as (8) where \(l_\kappa> 1 \leftrightarrow n_\kappa > 2\), is the absolute value and \(\eta _\kappa = \frac{l_\kappa (l_\kappa -1)}{2}\) stands for the total number of differences for a single cluster. an exponential-like distance measure provides a robust property based on the analysis of the influence function [39]. [12] empirically observed that it works properly, particularly when we look for \(k_{opt}\) within a hierarchical data set. therefore, a new homogeneity measure \(\varsigma _\kappa \) of non-negative exponential type was modelled as a penalty over \(\mathcal {m}_\kappa \) as (9) where \(\sigma _\kappa ^2\) is the variance of \(d_\kappa \). one can observe that the homogeneity measure gets closer to zero with the approximation of the ideal model solution, where the criterion value is zero and, therefore, the loss of information is null. thus, we have mec defined as $$\begin{aligned} \mathrm {mec}(k) = \lambda \sum _{\kappa =1}^{k} \varsigma _\kappa \times \mathcal {m}_\kappa , \end{aligned}$$ (10) where (11) the measure of global separation and penalty \(\lambda \), therefore, does not depend exclusively on the \(\kappa \)th cluster, but on the greater distance between the pairs of representative points of each data cluster (e.g., centroids). in a few words, \(\lambda \) globally weights the result of the solution. the presence of k, in eq. 11, is a simple way to avoid overfitting as a result of clustering solutions already sufficiently accommodated to the data. in addition to avoiding overfitting, an improvement over other indices is the possibility of evaluating the clustering tendency (\(k{=}1\)) without resorting to additional, external techniques [43]. fig. 2. mec results for a small set of twelve data points: (a) \(k = 1\); (b) \(k = 2\); (c) \(k = 4\). full size image to illustrate, fig. 2 shows the mec composition for three feasible cluster solutions, where each dotted line represents one measure of intra-cluster dissimilarity and each cluster is depicted by a quadratic centroid. the operating mechanism of mec, which encompasses both homogeneity (eq. 9) and separation (eq. 11), is visualised for \(k = 1, 2, 4\) (figs. 2a, 2b, and 2c, respectively). the motivation is that the dissimilarity measures should be similar to each other when looking at each cluster. in this case, fig. 2a contains the least suitable solution among those shown graphically, as their dissimilarity measures are more divergent in magnitude than those in fig. 2b and 2c. the four-cluster solution (fig. 2c) is the best within the solution set, as the distances among objects are exactly the same in each cluster. at last, it is worth noting that eq. 10 should be minimized, $$\begin{aligned} \hat{k} = \arg \min \mathrm {mec}(k), \end{aligned}$$ (12) where \(k \in [k_{min},k_{max}]\) and \(\hat{k}\) is inferred by the variation of k which determines the lowest mec value, regardless of the clustering algorithm. improving the time-complexity of mec. equation 8 can be equivalently computed in terms of a log-linear time complexity as a function of \(l_\kappa \), to improve the computational efficiency of mec. to do so, eq. 8 can be reformulated to generate an auxiliary vector, as well as in sorting \(d_\kappa \) with an algorithm of same complexity (e.g., heapsort). in fact, the time complexity of mec will be entirely dependent on the complexity of the chosen sorting algorithm. as such, we have a complexity of \(\mathcal {o}(l_\kappa \log {}l_\kappa )\) with the heap-sort algorithm, or even \(\mathcal {o}(n^2)\), by the reformulated $$\begin{aligned} \mathcal {m}_\kappa = \eta _\kappa ^{-1} \sum ^{\tilde{l}_\kappa }_{l=1} \left( c_{l \kappa } \hat{d}_{l \kappa } \times \left( l_\kappa l \right) \right) , \end{aligned}$$ (13) where \(\hat{d_\kappa }\) is the increasing ordering of the values of \(d_\kappa \) and \(\tilde{l}_\kappa = l_\kappa 1 = \left| c_\kappa \right| \); \(c_\kappa \) is an auxiliary variable that consists of a cumulative and naturally ordered vector of \(\hat{d_\kappa }\) defined as fig. 3. element assignment of \(c_\kappa \). full size image looking at fig. 3, each square and value between square brackets depicts some vector position (l notation). in eq. 13, the general form (\(l_\kappa -l\)) consists of the number of subtractions (eq. 8) represented by arrows in the figure, with \(\hat{d}_{l \kappa }\) depending on its location. the ordered \(\hat{d}_\kappa \) ensures that \(\hat{d}_{l \kappa } \le \hat{d}_{l+1,\kappa }\). by transitivity we have that, in eq. 13, $$\hat{d}_{l \kappa } \times \left( l_\kappa l \right) \le c_{l \kappa }, \quad l = 1,\dots ,\tilde{l}.$$ hence, \(\hat{d}_\kappa \) is sensibly less accessed, thus reducing the time complexity of mec. 4 results and analyses this section describes the results achieved by the five algorithms compared in this study: gadba, its new version gadba-mec, k -means, gmm, and linkage. since none of the last three techniques automatically finds \(\hat{k}\), the calinski and harabasz cluster validity index (cvi) is used to optimize \(\hat{k}\) through cluster validation (sect. 3.2). section 4.1 presents the methodology as how to, and by what means, the results were generated; sect. 4.2 discusses the results highlighting the techniques that clustered the data; and the statistical significance of the results is analysed in sect. 4.3. 4.1 applied methodology the accuracy of the clustering algorithms is explained in a set of statistical indicators, such as absolute frequency, mean and standard deviation of \(\hat{k}\), in twenty clustering validations for each data set (i.e., \(n_r = 20\)). the mean absolute percentage error (mape) was then estimated between the desired (\(k_{opt}\)) and optimized (\(\hat{k}\)) number of clusters in sect. 4.3. it generally expresses accuracy as a percentage which is designated by (15) table 1 presents data sets from different benchmarks used for performance analysis when comparing clustering algorithms. to evaluate the algorithms, ten sets were selected as archetypes of real challenges faced in cluster validation (e.g., data hierarchy, clustering tendency, different densities/sizes). table 1. properties of test data sets.full size table the gadba-mec algorithm works through some previously specified hyper-parameters. considering an oscillation of the best fitness in the order of \({1 \times 10^{-4}}\), the number of generations needed to infer the convergence of the fitness value is 50. the crossover and mutation probabilities are 0.8 and 0.03, respectively. the ring size of the tournament method for individual selection is set to 3. the population size and the maximum number of clusters are 100 and 30, respectively. all experiments presented herein were conducted on a computer with an intel© coretm i5 cpu @ 3.00 ghz with 8 gb of memory running matlab® 2017a. most packages used in our tests are internal to matlab®. 4.2 cluster detection results one approach to evaluate the performance of the clustering algorithms is to analyse how frequently \(\hat{k} = k_{opt}\). in this sense, table 2 shows the frequency of \(k_{opt}\) with emphasis on the highest absolute frequency by algorithm in blue. table 2. cluster detection results taken from the data sets.full size table only gadba is inconsistent with \(k_{opt}\) overall due to its shm-related objective function, as proven by the performance of gadba-mec. moreover, gadba is the most unstable algorithm, as shown by the standard deviation values. the only highlight of gadba was reached in iris, although this might be explained by its tendency of settling on lower k values. thus, a new version is justified ex post facto, attesting to the generalization potential of gadba-mec. an important highlight of the proposed version is the detection of low-separation hierarchical data in h\(_1\). virtually all other techniques tended to settle on expected sub-optimal k values. contrastingly, in cases where gadba-mec did not reach the highest frequency (i.e., dim-32, hepta, and iris), it at least approached the expected result in a stable manner, unlike gadba. for the rest of the algorithms, it should be noted that linkage is deterministic, so its null standard deviation is expected. it was the second best in finding \(k_{opt}\), although it failed to assess clustering tendency. in this regard, only gadba-mec determined that \(\hat{k} = 1\) in golfball and one-g. 4.3 statistical significance analyses friedman’s test is a non-parametric statistical test analogue to the two-way anova (analysis of variance) [1, 16]. this statistical test is used to determine whether there are any statistically significant differences among algorithms from sample evidences. the samples to be considered are clustering algorithm performance results collected over the data sets, where the null hypothesis \(h_0\), to be considered is that all algorithms obtained similar results. friedman’s test converts all results to ranks where all algorithms are classified for each problem according to its performance. as such, p-values can be computed for hypothesis testing. the p-value represents the probability of obtaining a result as extreme as the one observed, given \(h_0\) [7]. specifically, given the significance level \(\alpha = 0.05\), the null hypothesis is rejected if \(p < \alpha \). table 3. mape (%) taken from the data sets with emphasis on values above 100%.full size table table 4. friedman’s test on the data set results.full size table table 5. friedman’s post-hoc pairwise comparisons on the data set results, with emphasis on significant comparisons.full size table since we want to know which algorithms are significantly different from each other when \(h_0\) is rejected, a post-hoc procedure is necessary to compare all possible algorithm pairs. in this work, the procedure presented in [16] is employed, in which the means of critical values at \(\alpha \) are compared to each absolute difference on mean ranks as , \(i \ne j\). the absolute difference must be greater than \(\alpha \) to determine statistical significance. in this section, we verify the significance of gadba-mec using the friedman’s test f. for this purpose, we calculate the mape of each data set. once the algorithm with the smallest error is determined, the statistical significance test is applied to verify if the obtained difference is substantial. if this is the case, one can justify using one algorithm instead of another with more confidence. table 3 summarizes mape per data set emphasizing values above 100%. all algorithms had error rates above 100%, except gadba-mec with the lowest overall value (4.99%). table 4 focuses on these errors, for which the friedman’s test rejects the null hypothesis for an obtained p-value \(\ll \alpha = 0.05\). accordingly, friedman’s post-hoc test shows that there are significant improvements of the proposed version in terms of mape (table 5), as well as the fact that significant differences are shown in virtually all algorithm pairs. 5 conclusions and further work genetic-based clustering approaches play an important role in natural computing. in this sense, gadba was introduced as an efficient, bioinspired approach to cluster data in shm. despite its competitive performance identifying structural components, it produces poor results on more general clustering scenarios. for this reason, this study proposed the replacement of its objective function for mec, a recently developed cvi based on mutual equidistant-scattering analysis. gadba-mec outperforms conventional clustering algorithms when statistically evaluated across various data sets, attaining the expected number of clusters more often than others. the results showed that gadba-mec yielded better results in terms of cluster validation and mape errors, in particular when handling hierarchical data and data with low separation. also, only gadba-mec is able to verify the clustering tendency in the data sets addressed. as future work, we intend to expand gadba-mec to multi-objective optimization contexts. it is also relevant to apply gadba-mec in real-world problems to validate its efficiency in finding natural clusters. finally, comparing other cvi’s and bioinspired algorithms would be pertinent as well. references alswaitti, m., albughdadi, m., isa, n.a.m.: density-based particle swarm optimization algorithm for data clustering. eswa 91, 170–186 (2018) google scholar  armano, g., farmani, m.r.: multiobjective clustering analysis using particle swarm optimization. expert syst. appl. 55, 184–193 (2016) article  google scholar  bayá, a.e., larese, m.g., namías, r.: clustering stability for automated color image segmentation. expert syst. appl. 86, 258–273 (2017) article  google scholar  bezdek, j.c., pal, n.r.: some new indexes of cluster validity. tsmc-b (1998) google scholar  campello, r.: generalized external indexes for comparing data partitions with overlapping categories. pattern recogn. lett. 31(9), 966–975 (2010) article  google scholar  cremona, c.: big data and structural health monitoring. in: challenges in design and construction of an innovative and sustainable built environment, 19th iabse congress stockholm, pp. 1793–1801, september 2016 google scholar  daniel, w.w.: applied nonparametric statistics. pws-kent, usa (1990) google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee tec 6(2), 182–197 (2002) google scholar  diez, a., khoa, n.l.d., makki alamdari, m., wang, y., chen, f., runcie, p.: a clustering approach for structural health monitoring on bridges. jcshm (2016) google scholar  dziopa, t.: clustering validity indices evaluation with regard to semantic homogeneity. in: fedcsis 2016, gdańsk, poland, 11–14 september 2016, pp. 3–9 (2016) google scholar  esfandian, n., razzazi, f., behrad, a.: a clustering based feature selection method in spectro-temporal domain for speech recognition. eaai 25(6), 1194–1202 (2012) google scholar  flexa, c., santos, r., gomes, w., sales, c., costa, j.c.: mutual equidistant-scattering criterion: a new index for crisp clustering. eswa 128, 225–245 (2019) google scholar  fränti, p., sieranoja, s.: k-means properties on six clustering benchmark datasets. appl. intell. 48(12), 4743–4759 (2018) article  google scholar  fumeo, e., oneto, l., anguita, d.: condition based maintenance in railway transportation systems based on big data streaming analysis. pcs 53, 437–446 (2015) google scholar  gandomi, a., haider, m.: beyond the hype: big data concepts, methods, and analytics. int. j. inf. manage. 35(2), 137–144 (2015) article  google scholar  garcía, s., fernández, a., luengo, j., herrera, f.: advanced nonparametric tests for multiple comparisons in the design of experiments in computational intelligence and data mining: experimental analysis of power. is 180(10), 2044–2064 (2010) google scholar  gardiner, a., aasheim, c., rutner, p., williams, s.: skill requirements in big data: a content analysis of job advertisements. jcif 58(4), 374–384 (2018) google scholar  gil, d., songi, i.y.: modeling and management of big data: challenges and opportunities. futur. gener. comput. syst. 63, 96–99 (2016) article  google scholar  güngör, e., özmen, a.: distance and density based clustering algorithm using gaussian kernel. expert syst. appl. 69, 10–20 (2017) article  google scholar  islam, m.z., estivill-castro, v., rahman, m.a., bossomaier, t.: combining k-means and a genetic algorithm through a novel arrangement of genetic operators for high quality clustering. expert syst. appl. 91, 402–417 (2018) article  google scholar  jain, a.k.: data clustering: 50 years beyond k-means. prl 31(8), 651–666 (2010) article  google scholar  johnson, s.c.: hierarchical clustering schemes. psychometrika, pp. 241–254 (1967) google scholar  langone, r., reynders, e., mehrkanoon, s., suykens, j.a.: automated structural health monitoring based on adaptive kernel spectral clustering. in: mssp (2017) google scholar  lingras, p., chen, m., miao, d.: qualitative and quantitative combinations of crisp and rough clustering schemes using dominance relations. in: ijar, pp. 238–258 (2014) google scholar  lucasius, c., dane, a., kateman, g.: on k-medoid clustering of large data sets with the aid of a genetic algorithm: background, feasiblity and comparison. anal. chim. acta 282(3), 647–669 (1993) article  google scholar  macqueen, j.b.: some methods for classification and analysis of multivariate observations. in: bsmsp, vol. 1, pp. 281–297. university of california press (1967) google scholar  mary, s.a.l., sivagami, a.n., rani, m.u.: cluster validity measures dynamic clustering algorithms. arpn j. eng. appl. sci. 10(9), 4009–4012 (2015) google scholar  maulik, u., bandyopadhyay, s.: genetic algorithm-based clustering technique. pattern recogn. 33(9), 1455–1465 (2000) article  google scholar  mcafee, a., brynjolfsson, e.: big data: the management revolution (2012) google scholar  mclachlan, g., basford, k.: mixture models: inference and applications to clustering, vol. 38, january 1988 google scholar  moulavi, d., jaskowiak, p.a., campello, r.j.g.b., zimek, a., sander, j.: density-based clustering validation. in: 14th siam icdm, philadelphia, pa (2014) google scholar  pagnuco, i.a., pastore, j.i., abras, g., brun, m., ballarin, v.l.: analysis of genetic association using hierarchical clustering and cluster validation indices. genomics 109(5), 438–445 (2017) article  google scholar  rubio, e., castillo, o., valdez, f., melin, p., gonzalez, c.i., martinez, g.: an extension of the fuzzy possibilistic clustering algorithm using type-2 fuzzy logic techniques. adv. fuzzy sys. 2017, 7094046 (2017) google scholar  salvador, s., chan, p.: determining the number of clusters/segments in hierarchical clustering/segmentation algorithms. in: 16th ieee ictai, usa (2004) google scholar  silva, j.a., hruschka, e.r., gama, j.: an evolutionary algorithm for clustering data streams with a variable number of clusters. eswa 67, 228–238 (2017) google scholar  silva, m., santos, a., figueiredo, e., santos, r., sales, c., costa, j.c.w.a.: a novel unsupervised approach based on a genetic algorithm for structural damage detection in bridges. eng. appl. artif. intell. 52(c), 168–180 (2016) google scholar  silva, m., santos, a., santos, r., figueiredo, e., sales, c., costa, j.c.: agglomerative concentric hypersphere clustering applied to structural damage detection. mech. syst. signal process. 92, 196–212 (2017) article  google scholar  ultsch, a.: clustering with som: u*c. in: proceedings of workshop on self-organizing maps, pp. 75–82, january 2005 google scholar  wu, k.l., yang, m.s.: alternative c-means clustering algorithms. pattern recogn. 35, 2267–2278 (2002) article  google scholar  wu, k.l., yang, m.s.: a cluster validity index for fuzzy clustering. prl (2005) google scholar  xu, r., wunsch, d., ii.: survey of clustering algorithms. tnn 16(3), 645–678 (2005) google scholar  yang, c.l., kuo, r., chien, c.h., quyen, n.t.p.: non-dominated sorting genetic algorithm using fuzzy membership chromosome for categorical data clustering. appl. soft comput. 30, 113–122 (2015) article  google scholar  zhao, q.: cluster validity in clustering methods. ph.d. thesis, uef, june 2012 google scholar  download references author information authors and affiliations applied electromagnetism laboratory, federal university of pará, 01 augusto corrêa street, guamá, belém, pa, 66.075-110, brazil caio flexa, walisson gomes, igor moreira, reginaldo santos, claudomiro sales & moisés silva authors caio flexaview author publications search author on:pubmed google scholar walisson gomesview author publications search author on:pubmed google scholar igor moreiraview author publications search author on:pubmed google scholar reginaldo santosview author publications search author on:pubmed google scholar claudomiro salesview author publications search author on:pubmed google scholar moisés silvaview author publications search author on:pubmed google scholar corresponding author correspondence to caio flexa . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper flexa, c., gomes, w., moreira, i., santos, r., sales, c., silva, m. (2021). improving a genetic clustering approach with a cvi-based objective function. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_14 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords genetic algorithm automatic clustering algorithm decision boundary analysis mutual equidistant-scattering criterion publish with us policies and ethics profiles moisés silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature dynamic learning in hyper-heuristics to solve flowshop problems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper dynamic learning in hyper-heuristics to solve flowshop problems conference paper first online: 28 november 2021 pp 155–169 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) dynamic learning in hyper-heuristics to solve flowshop problems download book pdf download book epub lucas marcondes pavelski  orcid: orcid.org/0000-0002-5622-392x10, marie-éléonore kessaci  orcid: orcid.org/0000-0002-4372-516211 & myriam delgado  orcid: orcid.org/0000-0002-2791-174x10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 779 accesses abstract hyper-heuristics (hhs) are algorithms suitable for designing heuristics. hhs perform the search divided in two levels: they look for heuristic components in the high level and the heuristic is used, in the low level, to solve a set of instances of one or more problems. different from offline hhs, hyper-heuristics with dynamic learning select or generate heuristics during the search. this paper proposes a hyper-heuristic associated with a dynamic learning strategy for selecting iterated greedy (ig) components. the proposal is capable of selecting appropriate values for six ig components: local search, perturbation, destruction size, neighborhood size, destruction position and local search focus. the proposed hh is tested with six dynamic adaptation strategies: random, \(\epsilon \)-greedy, probability matching, multi-armed bandit, linucb, and thompson sampling (ts). the hyper-parameters of each strategy are tuned by irace. as a testbed, we use several instances with four different sizes (20, 50, 100 and 200 jobs) of three different formulations of flowshop problems (permutation, no-wait, no-idle), two distinct objectives (makespan, flowtime), and four processing time distributions (uniform, exponential and job or machine correlated). the results show that different strategies are most suitable for adapting different ig components, ts performs quite well for all components and, except for local search, using adaptation is always beneficial when compared with the ig running with standard parameters. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others online learning hyper-heuristics in multi-objective evolutionary algorithms chapter © 2023 dancing to the state of the art? chapter © 2024 rigorous performance analysis of hyper-heuristics chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms continuous optimization dynamic combinatorial chemistry learning algorithms optimization stochastic learning and adaptive control 1 introduction an automated methodology for selecting or generating heuristics to solve optimization problems is the focus of hyper-heuristics (hhs) [4]. hhs can be further classified according to the learning phase feedback, where they possibly have no learning, offline learning, or online learning. the later is also known, and referred in this text, as dynamic learning, that selects or generates heuristics during the search process. as occurs in a simpler level for meta-learning approaches, in a broader view, hyper-heuristics make algorithm design more adaptable to different problems [14]. another face on the same problem is given by the algorithm selection, particularly for dynamic schedules of algorithms, which generalizes static and per-instance algorithm selection approaches [10]. hyper-heuristics can contribute therefore to the development of algorithms for solving a wide range of optimization problems. flowshop problems (fsp) involve deciding how \(j\) jobs will be processed on \(m\) machines in series [2]. this paper investigates three fsp formulations: permutation (with no schedule constraints), and no-wait, no-idle which include constraints (there are no waiting jobs, and no idle machines, respectively). different proposals of parameter adaptation and hhs exist in the context of fsps and scheduling problems in general [3, 16, 25, 26]. in this context, one of the first works proposed in the literature uses an adaptive genetic algorithm [26], with online selection of four types of crossover and three mutations for the permutation fsp with makespan objective. the algorithm produces new offspring using different operators proportional to their contributions on previous generations. results show that the adaptive genetic algorithm presents a good performance when compared with an algorithm with static parameters and uniform selection of operators. a hh based on variable neighborhood search (vns) is proposed in [16]. the vns strategy adapts the shaking mechanism and local search, providing different low-level heuristics. the shaking is adapted by maintaining a tabu-list of non-improving heuristics, while different local searches are chosen greedily on a rank metric based on improving moves. the proposal performs well on four different combinatorial optimization problems, including permutation fsps. another related work is presented in [25], using the iterated local search (ils) with different neighborhood types. a greedy strategy selects the best neighborhood based on the fitness improvement, number of times each operator is used and the time to perform the local search. results show advantages on problems considering makespan, as well as flowtime objective. a recent work [3] proposes an iterated greedy (ig) algorithm enhanced by hyper-heuristics to solve hybrid flexible fsps with setup times. ig is a metaheuristic with excellent results for some fsp variants. it is based on initialization-destruction-construction phases, followed by a local search, which provides at the end a solution that can be accepted or discarded depending on an acceptance criterion. in [3], the neighborhood types used by the local search (swap, insert and inverse) are considered the low-level heuristics to be selected during the search. the enhanced ig is competitive while solving real-world instances. inspired by the fact that ig has been adapted, presents good performance on several combinatorial optimization problems, and performs particularly well on permutation fsp [18], in the present paper we propose and analyze different dynamic strategies used by a hyper-heuristic for selecting ig components. by adapting different components like destruction size and position, neighborhood size, perturbation, local search and its focus, the proposed hh is tested with distinct dynamic adaptation strategies: random, \(\epsilon \)-greedy, probability matching, multi-armed bandit, linucb, and thompson sampling. the most suitable hyper-parameters of each strategy are set in the tuning phase performed by irace. the proposal’s performance is evaluated on three formulations (constraints) of fsps, with four different sizes, two objectives, and four processing time distributions. this way, we intent to contribute to the fsp understanding and to find general strategies for solving different formulations of the problem. the main contributions of the paper can be summarized as: (i) adapting six different ig components; (ii) testing six different dynamic learning strategies in the proposed hh; (iii) tuning main hh hyper-parameters; (iv) addressing several fsp variants; (v) providing a high performance adaptive ig capable of outperforming the standard ig [18] in many fsp variants. as far as we know, there is no other previous work considering different hhs with dynamic learning to fsp. moreover, no other previous work considers the simultaneously adaptation of multiple ig components. finally, there is no reported results with hh outperforming standard ig on different fsp types. the paper is organized as follows. section 2 discusses the basic concepts necessary to understand the proposal. section 3 details the adaptive ig that is being proposed here. the methodology adopted in the experiments is described in sect. 4. results are presented and analyzed in sect. 5. finally, sect. 6 concludes the paper and discusses future perspectives. 2 background this section presents the basic concepts regarding the application context (sect. 2.1 details the fsp) and the proposal (sect. 2.2 describes the dynamic adaptive strategies used by the proposed hyper-heuristic). 2.1 flowshop problems (fsps) flowshop is a combinatorial optimization problem of scheduling. the problem involves deciding how \(j\) jobs will be processed on \(m\) machines in series. given the processing times on each machine, a permutation \(x= (x_0, \dots , x_j)\) informs the order jobs will be executed on all machines. the most common formulation considers that jobs and machines are available any time, with processing times known in advance, and machine processes are sequence-independent and occur without interruptions [2]. in permutation fsps, the completion time of a job \(x_j\) in the \(m\)-th machine can be determined by: $$\begin{aligned} c_{x_j,m} = \max (c_{x_j,m}, c_{x_{j-1},m}) + p_{x_j,m} \end{aligned}$$ (1) where \(p_{x_j,m}\) is the processing time of job \(x_j\) on machine \(m\). two common objectives in fsps are makespan and flowtime. makespan is the time required to complete all jobs, i.e., \(\max _jc_{x_j,m}\), and flowtime is the sum of all completion times, \(\sum _jc_{x_j,m}\). besides the permutation fsps formulations, other variants like the no-wait and no-idle include constraints on the schedules. the no-wait fsp variant only considers schedules where there are no waiting times between job operations. the no-wait completion times are given by: $$\begin{aligned} c_{x_j,m} = d_{x_{j-1},j} + \sum _{m= 0}^{m} p_{x_j,m} \end{aligned}$$ (2) where \(d\) are the precomputed delay times [20]. no-idle schedules completion times are computed using [23]: $$\begin{aligned} f(x_{1},m,m+1)&= p_{x_j,m+1} \end{aligned}$$ (3) $$\begin{aligned} f(x_{j},m,m+1)&= \max \left\{ f(x_{j-1},m,m+1) p_{x_j, m}, 0 \right\} + p_{x_j,m+1} \end{aligned}$$ (4) $$\begin{aligned} c_{x_j,m}&= \sum _{m= 1}^{m-1} f(x_{j},m,m+1) + \sum _{k=1}^{j} p_{x_k,1}. \end{aligned}$$ (5) where \(f(x_{j},m,m+1)\) is the minimum difference between the completion of processing up to job \(x_{j}\), on machines \(k+1\) and k, restricted by the no-idle constraint. in addition to objectives and constrains, a fsp formulation includes the definition of processing times, which can be correlated or non-correlated and whose distributions can be uniform or exponential. large number of jobs and uniform processing times usually make the problem harder to solve. alternatively, simple heuristics perform well when there are correlations between processing time [24]. also, in this paper, we investigate processing times with exponential distributions, and job or machine correlated processing times. 2.2 hyper-heuristics and their adaptation strategies according to [15], a hyper-heuristic (hh) works with a two-level structure, at the high-level it looks for heuristic configurations \(h \in h\), considering h the heuristic space. at the low-level, each solution \(x\in x\) of the target optimization problem p is generated by the heuristic \(h \in h\). there are two evaluation functions: in the first level, the hh’s success is measured by function \(f:h \rightarrow \mathfrak {r}\) and in the second level, each solution \(x\in x \) is evaluated by an objective function \(f: x\rightarrow \mathfrak {r}\). from a mapping function \(m: f(x) \rightarrow f(h)\), it is possible to define the purpose of a selection hh. the hh must optimize f(h) by means of the search for the optimal heuristic configuration \(h^*\), in h, thus \(h^*\) generates the optimal solution(s) \(x^*\) [15]. the formal hh definition, in a minimization optimization problem is summarized in eq. 6. $$\begin{aligned} \small f(h^* \mid h^* \rightarrow x^*, h^* \in h) \leftarrow f(x^*, x^* \in x) = min\{ f(x), x \in x\} \end{aligned}$$ (6) in this paper, h relies on different choices for each ig component, f is associated with makespan or flowtime objectives, and f is measured by a reward function detailed in sect. 3. hyper-heuristics aim therefore at providing more generalized solutions for optimization problems by producing good results when dealing with a set of instances or a set of problems. for this, hhs work in the heuristic space rather than the solution space. based on specific strategies, they adapt low-level heuristics, which are used to solve the target problem(s). we investigate six adaptation strategies commonly used in the hh literature: random, \(\epsilon \)-greedy, probability matching, multi-armed bandit, linucb and thompson sampling. random parameter selection is the simplest strategy and in most cases, it serves as a baseline for comparison between static and dynamic parameter selection. it might be beneficial depending on the chance of selecting the best parameter combination [5]. \(\epsilon \) -greedy is a simple strategy often referenced on the exploration-exploitation dilemma [22]. with probability \(\epsilon \), the parameter with the best average reward is chosen, otherwise, with probability \(1 \epsilon \), a random one is selected. probability matching (pm) [7] works as a roulette wheel selection biased towards the operators with best quality. the probability of selecting the k-th operator from a set of k operators at iteration t is given by: $$\begin{aligned} p_{k,t} = p_{min} + (1 k \times p_{min}) \frac{q_{k,t}}{\sum _{j=1}^k q_{j,t}} \end{aligned}$$ (7) where \(q_{k,t}\) is the quality of k-th operator and \(0< p_{min} < 1/k\) is used to guarantee that every operator will have a minimum chance of being chosen. the quality values are updated according to the rewards: $$\begin{aligned} q_{k,t} = q_{k,t-1} + \alpha \times (r^{acc}_{k,t} q_{k,t-1}) \end{aligned}$$ (8) where \(\alpha \) is the learning rate parameter and \(r^{acc}_{k,t}\) is the accumulated reward of operator k during a given update window of size w. the accumulation in \(r^{acc}_{k,t}\) considers either the average or extreme and, optionally, normalized reward values. multi-armed bandit (mab) [1] algorithms are based on the upper confidence bound for exploitation-exploration trade-off. in particular, the fitness-rate-rank-based multi-armed bandit [11] considers the dynamic search behavior with a sliding window of size w to store the rewards of each operator. the selected operator maximizes the expression: $$\begin{aligned} frr_{k,t} + c_s \left( \frac{2 \ln \sum _{l=1}^k n_l^t}{n_k^t} \right) \end{aligned}$$ (9) where \(c_s\) is a scaling parameter, \(n_k^t\) is the number of times operator k is applied during the window of size w, and \(frr_{k,t}\) is the k-th operator credit value given by: $$\begin{aligned} frr_{k,t} = \frac{d^{rank_k} \times r_k}{\sum _{l=1}^k d^{rank_l} \times r_l} \end{aligned}$$ (10) in which \(d\) is the best operator influence decay parameter, \(r^{acc}_k\) is the k-th operator accumulated reward, and \(rank_k\) is the k-th operator reward sum rank. linear upper confidence bound (linucb) [12] works by assuming that the reward for a given operator is linearly proportional to the values of contextual features, i.e., \(e[r_{k,t}|\boldsymbol{\phi }_{k,t}] = \boldsymbol{\phi }_{k,t}^t \mathbf {\theta ^*}_{k}\), where \(\boldsymbol{\phi }_{k,t}\) is the feature vector and \(\mathbf {\theta ^*}_{k}\) the unknown coefficients. we consider four fitness landscape metrics as the context for a local search procedure, calculated online during the local search step [17]: adaptive walk length: the total number of steps of the local search; autocorrelation: correlation between the fitness values observed with the fitness values of the previous solutions; fitness-distance correlation: correlation between fitness and insertion distance considering the initial and final solutions; neutrality: proportion of neighbors with equal fitness values. using a ridge regression formulation, the coefficients can be found efficiently with the following steps: $$\begin{aligned} \begin{aligned} \mathbf {\theta ^*}_{k,t}&= a_{k,t}^{-1} b_{k,t} \\ p_{k,t}&= \mathbf {\theta ^*}_{k,t} \boldsymbol{\phi }_{k,t} + \alpha \sqrt{\boldsymbol{\phi }_{k,t} a_{k,t}^{-1} \boldsymbol{\phi }_{k,t}} \end{aligned} \end{aligned}$$ (11) where \(\alpha \) is a learning rate parameter. the operator with maximum \(p_{k,t}\) is chosen, yielding the reward value \(r_{k,t}\), and the model update follows: $$\begin{aligned} \begin{aligned} a_{k,t}&= a_{k,t-1} + \boldsymbol{\phi }_{k,t} \boldsymbol{\phi }_{k,t}^t \\ b_{k,t}&= b_{k,t-1} + r_{k,t} \boldsymbol{\phi }_{k,t}. \end{aligned} \end{aligned}$$ (12) thompson sampling (ts) [19] strategy starts with a prior distribution, chooses the best operator by sampling, observes the output and updates the distribution. the beta distribution \(beta(s_{k,t}, f_{k,t})\) models bernoulli trials where operator k has \(s_{k,t}\) successes (rewards \(r_{k} > 0\)) and \(f_{k,t}\) fails (rewards \(r_{k} \le 0\)). therefore we choose the operator with: $$\begin{aligned} op = \mathop {\text {arg max}}\limits _{k} sample[beta(s_{k,t}, f_{k,t})] \end{aligned}$$ (13) and update the distribution after the reward: $$\begin{aligned} \begin{aligned} s_{k,t}&= s_{k,t-1} + 1_{r_{k,t} > 0} \\ f_{k,t}&= f_{k,t-1} + 1_{r_{k,t} \le 0}. \end{aligned} \end{aligned}$$ (14) alternatively, the dynamic ts [8] introduces a window size parameter w and a modified update rule (after iteration w) as follows: $$\begin{aligned} \begin{aligned} s_{k,t}&= (s_{k,t-1} + 1_{r_{k,t} > 0}) \tfrac{w}{w+1} \\ f_{k,t}&= (f_{k,t-1} + 1_{r_{k,t} \le 0}) \tfrac{w}{w+1}. \end{aligned} \end{aligned}$$ (15) table 1. hyper-parameters of the addressed adaptation strategies.full size table table 1 shows a summary of all hyper-parameters used by the adaptation strategies, as well as two hyper-parameters common to all strategies: reward type and warm-up period. the warm-up period is considered at the beginning of the iterations where strategies are chosen randomly. reward type is detailed in the next section. 3 adaptive ig proposal considered the state of the art for some fsp variants, the iterated greedy (ig) algorithm [18] is a successful iterative metaheuristic that encompasses five main steps: (1) the incumbent solution \(x\) is initialized, (2) a destruction phase randomly removes d jobs, (3) a construction procedure inserts each job at the best position, (4) a local search generates a new solution by exploiting the solution resulted from construction and (5) the new solution replaces the incumbent \(x\) according to an acceptance criterion. the last step accepts the new solution \(x''\) based on the following probability: $$\begin{aligned} p_{accep}(x'') = \left\{ \begin{array}{ll} 1.0 &{} \text{ if } f(x'') < f(x) \\ \exp \left( \tfrac{-(f(x'') f(x)) }{temp} \right) &{} \text{ otherwise } \\ \end{array}\right. \end{aligned}$$ (16) where f(.) is the cost function and temp is the temperature defined by: $$\begin{aligned} temp = t \times \frac{\sum _{j=1}^{j}\sum _{m=1}^{m} p_{j, m}}{j* m* 10}. \end{aligned}$$ (17) ig has two main parameters: the destruction size d and the temperature factor t, whose values ig authors [18] recommend to be set as \(d = 4\) and \(t = 0.5\). other recommended configurations are nawaz–enscore–ham (neh) construction heuristic as the initialization and iterative improvement as local search. the local search iteratively inserts a job, chosen randomly without replacement, on the best position until there are no improvements [21]. this version is referred from now on as standard ig. the hyper-heuristic proposed in the present paper considers an adaptive strategy (one fixed among the six possible strategies described in sect. 2.2) to update some components of an ig algorithm. an adaptive strategy requires a set h of possible choices and a reward function indicating how well a particular choice performs. an important issue for defining an adaptive strategy is the reward function. it returns a real number representing the quality of a given choice. good choices have positive rewards and bad choices receive negative ones. in the case of using ig to solve fsps, reward of kth parameter at iteration t can be calculated based on the relative decrease in cost function value (e.g. makespan or any other objective function being considered): $$\begin{aligned} r_{k,t} = (f(x_{before}) f(x_{after}))/f(x_{before}). \end{aligned}$$ (18) here \(x_{before}\) and \(x_{after}\) represent solutions before and after the reward evaluation period (whose reference can be either local search or iteration). we have four possible periods during which a solution has its performance evaluated: before the iteration (bi) or the local search (bl) and after the iteration (ai) or local search (al), giving rise to four types of reward: blal: \((f(x_{\text {before local search}}) f(x_{\text {after local search}}))/f(x_\text {before local search})\); bial: \((f(x_{\text {before iteration}}) f(x_{\text {after local search}}))/f(x_\text {before iteration})\); blai: \((f(x_{\text {before local search}}) f(x_{\text {after iteration}}))/f(x_\text {before local search})\); biai: \((f(x_{\text {before iteration}}) f(x_{\text {after iteration}}))/f(x_\text {before iteration})\). as fig. 1 shows, each reward type considers a different period to get the feedback for each solution considering its quality increase/decrease. fig. 1. ig algorithm and reward types: blal, bial, blai and biai. full size image using the standard ig [18] as basis, with a budget computation time as the stopping criteria, we identify some components that can be adapted dynamically: type of the local search, type of perturbation, destruction size, neighborhood size, destruction position, local search focus. a summary of these adaptive components and their values are presented in table 2. in our case, different choices are possible for each ig component, for example, the number of deconstructions. we refer to the parameter values as discrete choices (also referred to arms in multi-armed bandit literature) indexed by \(\{1, \dots , k\}\), where \(k = |h|\) is the total number of choices. therefore, according to eq. 6, we have each choice representing a heuristic \(h \in h\), where the dynamic adaptation strategy of the hh chooses h as the k-th choice, \(k=1, \dots , |h|\). the pool regarding the first choice sets encompasses the most usual options for local search and perturbation in the context of combinatorial optimization. the options for destruction size are the ones often chosen in ig implementations for fsp. finally, the pools for the last three choice sets are defined aiming to produce different granularities, starting from a coarse, ending to a fine search. table 2. ig adaptive parameters.full size table the hh based on dynamic learning proposed in this paper is capable of adapting different components (e.g. local search and perturbation) for each iteration of the adaptive ig. in local search for example, it can choose between the original ig’s iterative improvement, first improvement, first-best improvement, random best improvement. the last three local search options consider all possible insertions. the perturbation adaptation considers three possibilities: (1) the ig destruction-construction steps; (2) two random swaps and a transposition, as used in [21] for ils on fsp , and (3) destruction-construction with iterative improvement local search between destructions, recently proposed in [6]. the neighborhood size adaptation considers the percentage of the neighborhood explored during the local search step. for example, exploring only half of the neighborhood at the beginning of the search could save time for exploitation at the end of the search. similarly, choosing the destruction size parameter dynamically might improve the search during exploration-exploitation phases. as mentioned, there are two mechanisms in ig that randomly choose jobs to be re-inserted: destruction and best-insertion local search. an adaptive mechanism can be applied in the last two ig components shown in table 2 to focus these operators on parts of the solution that have a better chance of improvement. for that, we propose partitioning the solution into chunks and adaptively selecting from each chunk the job that will be sampled. as shown in table 2, for some parameters there are multiple possible choice pools \(\{h_1,h_2,...\}\). for example, the destruction size can be chosen from the set \(h_1=\{2,4\}\), \(h_2=\{4,6\}\), \(h_3=\{2,4,6\}\) or \(h_4=\{4,8\}\). also on destruction position and local search focus, the solution can be partitioned into 3, 10 or \(j\) chunks (the last considering one arm for each job). some hh hyper-parameters like the pool \(h_i\) for a component, reward type and update window are determined in a parameter tuning phase described in sect. 4. 4 tuning and testing phases based on the hyper-heuristic with dynamic learning and its six possible adaptive strategies described in sect. 2.2, sect. 3 has detailed the proposed adaptive ig. this section presents the hyper-parameter configuration performed by irace (tuning) using part of available data and the test performed on the tuned hh using the remaining one. the configurations are evaluated with a budget of \(j\times (m/ 2) \times 30\) ms on tuning and test phases. algorithms are implemented in c++ using the paradiseo library [9]. the experiments have been executed on a server with 8-core amd epyc 7542 processors and 16 gb of ram. for the results analysis, we used the r language and relevant packagesfootnote 1. tuning phase before running the experiments, we perform a tuning phase aiming to determine which strategy configuration (shown in table 1) works best for each ig component (shown in table 2). the random strategy is not considered for destruction position and local search focus, because a random choice in these cases is equivalent to the standard ig behaviour. the irace algorithm [13] with default parameters and 5000 configurations evaluations is used to tune each combination of the six adaptive components and the six dynamic strategies. there are multiple choice pools \(\{h_1,h_2,...\}\) for destruction size, neighborhood size, destruction position and local search focus components. for example, the destruction size can be chosen from \(h_1=\{2,4\}\) or \(h_2=\{4,8\}\). in these cases, the pool choice is considered as an additional categorical parameter for irace during the tuning phase. the instance set for the tuning phase is composed of 48 (3 \(\times \) 2 \(\times \) 3 \(\times \) 4) instances resulted from the combinations of: 3 sizes (20 or 50 jobs and 10 machines), 2 objectives (makespan or flowtime), 3 types (permutation, no-wait or no-idle fsps), and 4 processing times distributions: (exponential, uniform, job-correlated or machine-correlated processing times). testing phase after tuning, the best configurations are tested with 10 restarts on a set of unseen instances with the same features, but sampled with different random seeds. aiming to better evaluate algorithms’ generalization capabilities, we also include larger instances of \(j= 100\) and \(j= 200\) jobs and \(m= 20\) in the testing phase. the evaluation for each algorithm is done using the average relative percentage deviation (arpd) given by: $$\begin{aligned} arpd_{alg} = \frac{1}{r} \sum _{r=1}^{r} 100 \times \frac{f(x^{alg}_r) f(x^{best})}{f(x^{best})} \end{aligned}$$ (19) where r is the number of runs and \(x^{alg}_r\) is the best solution found by algorithm alg. the reference solution \(x^{best}\) is given by the best solution found by a standard ig (the same used by the authors in [18]) with a higher budget of \(j\times (m/ 2) \times 120\) ms and 30 restarts. finally, the configurations are also compared with the standard ig configuration to evaluate the effectiveness of the adaptive components in the presented scenario. in all comparisons, we highlight the lowest arpds and perform a friedman rank sum test with nemenyi post hoc to verify if the differences are statistically significant with p-value threshold of 0.05: results with no statistically significant differences from the best one are highlighted with gray background. 5 results table 3 shows some hyper-parameter values (reward type, choice sets and update window) tuned by irace for each ig component addressed in the paper. table 3. tuned hyper-parameters.full size table different reward types have been tuned for the different adaptive components and hhs, with no dominance between the different options. the choice sets for the destruction size parameter enable more exploration with values higher than the default \(d = 4\), which is present in all options. the choice sets for neighborhood size are small (2 or 3), with preference for a coarse search, while the local search focus uses partitions with 10 or more choices in three among five cases. finally, the update window is short for mab and ts on perturbation component, indicating that its value (4 or 8 destructions) is often switched during the search. however, in most cases the strategies prefer less frequent changes, since large window sizes are the tuned parameters. the arpd values for each adaptive component and strategy are shown in table 4, and they are calculated using eq. 19, with \(r=10\) and considering all the different testing instances. the values for standard ig configuration [18] are computed with the same budget (\(j\times (m/ 2) \times 30\)) as the hhs proposals on the testing instances. notice that this budget is lower than the one used to compute the reference \(f(x_{best})\) values for arpd. the adaptive components are able to improve the static standard ig configuration for all (perturbation, destruction size, destruction position and local search focus) but one ig component. local search adaptation is not effective independently of the strategy used by the hh, meaning that the iterative improvement performed by the standard ig is quite effective compared with the other choices. ts adaptation achieves the lowest arpd for perturbation and destruction position components, while \(\epsilon \)-greedy performs well on the local search focus task. as (biased) adaptation might not be the best option in some cases [5], the simple random strategy performs well for selecting the destruction size and neighborhood size. in all cases, the adaptive strategies ts and mab are among the strategies with the lowest or statistically equivalent to the lowest arpd values. we see from table 4 that ts is robust for different components but provides the lowest among all results when adapting ig perturbation. however, it is equivalent to most other hh proposals. we compare ts configuring only perturbation with two others that try to adapt multiple components simultaneously: adapt all components, adapt all components except local search (for which no adaptation strategy was capable of improving the performance). the adapt all approaches use strategies with the best arpd values for each component from table 4, which means, ts for perturbation and destruction position, random for destruction and neighborhood sizes and \(\epsilon \)-greedy for local search focus. the results in table 5 show the metrics separated by objective, fsp type, processing times distribution and size. overall, ig with adaptive perturbation obtains the lowest arpds. adapting all components at the same time does not provide benefits, but when we eliminate the local search it performs like the best approach. table 4. adaptation strategies arpds (and standard deviation) for each adaptive component and strategy. lowest mean values are highlighted in bold, statistically equivalent values are highlighted with gray background.full size table table 5. adaptation strategies arpds (and standard deviation) for all adaptive components, perturbation and destruction size adaptation and standard (static) ig. best mean and statistically equivalent values are in bold and gray background, respectively.full size table 6 conclusions this paper proposed and analyzed the use of hyper-heuristic with dynamic strategies to adapt different components of the iterated greedy algorithm. six different adapting strategies (random, \(\epsilon \)-greedy, probability matching, multi-armed bandit, linucb, and thompson sampling) were tested to adapt six ig components (local search, perturbation, destruction size, neighborhood size, destruction position and local search focus). after a tuning phase performed by irace to set the best strategy hyper-parameters for each component being adapted, the proposal was tested in different variants of the flowshop problems. results show that, in most cases, the adaptation is able to improve the performance over the static standard ig configuration, especially the perturbation operator adapted using dynamic thompson sampling. also, using multiple adaptive components did not seem to be beneficial, unless we fix the local search as iterated improvement, a fact that deserves further investigation. the work can be expanded by including different flowshop problems (objectives and constraints), adaptation strategies, and alternative operators. in addtion, we intend to propose modifications to improve the performance of the adapt all approach. notes 1.tuning and testing instances, as well as the code used in the paper, are available at https://github.com/lucasmpavelski/adaptive-ig. references auer, p.: using confidence bounds for exploitation-exploration trade-offs. j. mach. learn. res. 3(9), 397–422 (2002) google scholar  baker, k.r., trietsch, d.: principles of sequencing and scheduling. wiley publishing, new jersey usa (2009) google scholar  burcin, o.f., sagir, m.: iterated greedy algorithms enhanced by hyper-heuristic based learning for hybrid flexible flowshop scheduling problem with sequence dependent setup times: a case study at a manufacturing plant. comput. oper. res. 125, 105044 (2021). https://doi.org/10.1016/j.cor.2020.105044 burke, e.k., hyde, m.r., kendall, g., ochoa, g., özcan, e., woodward, j.r.: a classification of hyper-heuristic approaches: revisited. in: gendreau, m., potvin, j.-y. (eds.) handbook of metaheuristics. isorms, vol. 272, pp. 453–477. springer, cham (2019). https://doi.org/10.1007/978-3-319-91086-4_14 chapter  google scholar  chakhlevitch, k., cowling, p.: hyperheuristics: recent developments. in: studies in computational intelligence, vol. 136, pp. 3–29. springer, berlin, heidelberg (2008). https://doi.org/10.1007/978-3-540-79438-7_1 dubois-lacoste, j., pagnozzi, f., stützle, t.: an iterated greedy algorithm with optimization of partial solutions for the makespan permutation flowshop problem. comput. oper. res. 81, 160–166 (2017). https://doi.org/10.1016/j.cor.2016.12.021 article  mathscinet  math  google scholar  goldberg, d.e.: probability matching, the magnitude of reinforcement, and classifier system bidding. mach. learn. 5(4), 407–425 (1990). https://doi.org/10.1023/a:1022681708029 article  google scholar  gupta, n., granmo, o.c., agrawala, a.: thompson sampling for dynamic multi-armed bandits. in: 2011 10th international conference on machine learning and applications and workshops, pp. 484–489. ieee, honolulu, hi, usa (2011). https://doi.org/10.1109/icmla.2011.144 humeau, j., liefooghe, a., talbi, e.g., verel, s.: paradiseo-mo: from fitness landscape analysis to efficient local search algorithms. research report rr-7871, inria (2013) google scholar  kerschke, p., hoos, h.h., neumann, f., trautmann, h.: automated algorithm selection: survey and perspectives. evol. comput. 27(1), 3–45 (2019) article  google scholar  li, k., fialho, a., kwong, s., zhang, q.: adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. ieee trans. evol. comput. 18(1), 114–130 (2014). https://doi.org/10.1109/tevc.2013.2239648 article  google scholar  li, l., chu, w., langford, j., schapire, r.e.: a contextual-bandit approach to personalized news article recommendation. in: proceedings of the 19th international conference on world wide web www ’10, p. 661. acm press, raleigh, north carolina, usa (2010). https://doi.org/10.1145/1772690.1772758 lópez-ibáñez, m., dubois-lacoste, j., cáceres, l.p., stützle, t., birattari, m.: the irace package: iterated racing for automatic algorithm configuration. oper. res. perspect. 3, 43–58 (2016). https://doi.org/10.1016/j.orp.2016.09.002 article  mathscinet  google scholar  pappa, g.l., ochoa, g., hyde, m.r., freitas, a.a., woodward, j., swan, j.: contrasting meta-learning and hyper-heuristic research: the role of evolutionary algorithms. genetic program. evolvable mach. 15(1), 3–35 (2013). https://doi.org/10.1007/s10710-013-9186-9 article  google scholar  pillay, n., qu, r.: hyper-heuristics: theory and applications. springer nature, 1 edn. (2018). https://doi.org/10.1007/978-3-319-96514-7 hsiao, p.-c., chiang, t.-c., fu, l.-c.: a vns-based hyper-heuristic with adaptive computational budget of local search. in: 2012 ieee congress on evolutionary computation, pp. 1–8. ieee, brisbane, australia (2012). https://doi.org/10.1109/cec.2012.6252969 pitzer, e., affenzeller, m.: recent advances in intelligent engineering systems. studies in computational intelligence, vol. 378, chap. a comprehensive survey on fitness landscape analysis, pp. 161–186. springer, berlin, heidelberg (2012). https://doi.org/10.1007/978-3-642-23229-9 ruiz, r., stützle, t.: a simple and effective iterated greedy algorithm for the permutation flowshop scheduling problem. eur. j. oper. res. 177(3), 2033–2049 (2007). https://doi.org/10.1016/j.ejor.2005.12.009 article  math  google scholar  russo, d., van roy, b., kazerouni, a., osband, i., wen, z.: a tutorial on thompson sampling. arxiv:1707.02038 [cs] (2020) sapkal, s.u., laha, d.: a heuristic for no-wait flow shop scheduling. int. j. adv. manuf. technol. 1, 1327–1338 (2013). https://doi.org/10.1007/s00170-013-4924-y stützle, t.: applying iterated local search to the permutation flow shop problem. technical report, fg intellektik, tu darmstadt, darmstadt, germany (1998) google scholar  sutton, r.s., barto, a.g.: reinforcement learning: an introduction. adaptive computation and machine learning series. 2nd edn. the mit press, cambridge, massachusetts (2018) google scholar  tasgetiren, m.f., pan, q.k., suganthan, p.n., liang, y.c.: a discrete differential evolution algorithm for the no-wait flowshop scheduling problem with total flowtime criterion. in: 2007 ieee symposium on computational intelligence in scheduling, pp. 251–258. ieee, honolulu, usa (2007) google scholar  watson, j.p., barbulescu, l., howe, a.e., whitley, l.d.: algorithm performance and problem structure for flow-shop scheduling. in: aaai/iaai, pp. 688–695. american association for artificial intelligence, menlo park, ca, usa (1999) google scholar  yahyaoui, h., krichen, s., derbel, b., talbi, e.g.: a hybrid ils-vnd based hyper-heuristic for permutation flowshop scheduling problem. procedia comput. sci. 60, 632–641 (2015). https://doi.org/10.1016/j.procs.2015.08.199 article  google scholar  zhang, l., wang, l., zheng, d.z.: an adaptive genetic algorithm with multiple operators for flowshop scheduling. int. j. adv. manuf. technol. 27(5–6), 580–587 (2006). https://doi.org/10.1007/s00170-004-2223-3 article  google scholar  download references acknowledgment m. delgado acknowledges cnpq (grants 439226/2018-0, 314699/2020-1) for her partial financial support. author information authors and affiliations cpgei universidade tecnológica federal do paraná, curitiba, brazil lucas marcondes pavelski & myriam delgado cristal univ. lille, cnrs centrale lille umr 9189, 59000, lille, france marie-éléonore kessaci authors lucas marcondes pavelskiview author publications search author on:pubmed google scholar marie-éléonore kessaciview author publications search author on:pubmed google scholar myriam delgadoview author publications search author on:pubmed google scholar corresponding author correspondence to lucas marcondes pavelski . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper pavelski, l.m., kessaci, mé., delgado, m. (2021). dynamic learning in hyper-heuristics to solve flowshop problems. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_11 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hyper-heuristic iterated greedy algorithm adaptive components flowshop variants publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a simple and low-cost method for leaf surface dimension estimation based on digital images | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a simple and low-cost method for leaf surface dimension estimation based on digital images conference paper first online: 12 october 2023 pp 146–161 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a simple and low-cost method for leaf surface dimension estimation based on digital images download book pdf download book epub karla gabriele florentino da silva  orcid: orcid.org/0000-0002-0211-38059,10, jonas magalhães moreira  orcid: orcid.org/0009-0008-8517-421610, gabriel barreto calixto  orcid: orcid.org/0009-0004-0099-60449, luiz maurílio da silva maciel  orcid: orcid.org/0000-0001-9193-23029,10, márcio assis miranda  orcid: orcid.org/0009-0002-1877-562410 & … leandro elias morais  orcid: orcid.org/0000-0002-0211-421811  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 518 accesses 1 citation abstract the leaf is the organ of the plant body that performs photosynthesis and its area is one of the morphological parameters that most respond to droughts, climate changes, and attack of pathogens, associated with the accumulation of biomass and agricultural productivity. in addition, leaf area and other surface data (for example, width and length) are widely used in studies of plant anatomy and physiology. the methods of measuring these leaf surface parameters are often complicated and costly. in this context, this work aims to develop a simple and low-cost method capable of accurately measuring the leaf surface size of plant species with significant agricultural interest. our method extract the information through images of leaves accompanied by a scale pattern whose real area is known, captured by a simple camera. to evaluate our method, we performed experiments with images of 118 leaves of 6 species. we compared the results to the imagej software, which is widely used to estimate leaf dimensions from images. the results showed our method present performance similar to imagej. however, unlike imagej, our method does not require user interaction during the dimensions estimation. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others lower-dimensional intrinsic structural representation of leaf images and plant recognition article 13 july 2021 a new approach for measuring leaf projected area for potted plant based on computer vision chapter © 2016 leaf image-based plant identification using morphological feature extraction chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. leaf development photosynthesis plant anatomy and morphology plant ecology plant science plant physiology 1 introduction the leaves constitute the organ of the vegetable body capable of converting solar energy into chemical energy through photosynthesis [7]. vegetable growth and development are determined by the leaf morphology, especially its area, the largest component of biomass accumulation and productivity [17, 34]. the leaf size and morphology undergo an intraspecific and interspecific variation, and many physiological parameters (e.g., nitrogen content, photosynthesis and leaf respiration) are normalized by leaf area in order to study the relationship between them [23, 41]. the leaf morphology (e.g. area, perimeter, width and length) is strongly used to investigate vegetable responses to different environmental conditions, such as the soil fertility [4, 13], soil water/dry availability [29, 38], light availability [25], effect of pesticides application [20], climate changes [15, 27, 31], phytopathogens [18], herbivory [9], screening of genotypes for breeding [39], pollution [12]. in fact, measuring the area, length, width, perimeter and their derivations (e.g. specific leaf area, perimeter/leaf area ratio, length/ width ratio) is a powerful tool used by professionals of several areas of knowledge such as vegetable improvers, botanists, phytopathologists, ecologists and agronomists [14, 36, 37, 40]. there is equipment that measures the leaf dimensions with great accuracy, but it is quite expensive, quotes around us$3,500.00 (quotation performed in january 2021). in addition, it is possible to estimate leaf area using allometric models (equations) using leaf morphology variables (e.g. length and width), adjusted for each species and/or crop. however, a big leaves sample is required to develop these models. for example, antunes et al. [1] analyzed 1563 coffee leaves. it is important to note that these models estimate the leaf area of a single leaf at a time. in tests with many sample units, estimating this parameter will be a time consuming and laborious process. therefore, it is necessary to develop practical and low-cost methodologies, in order to guarantee the execution of works that require a verification of the leaf area and other dimensions with celerity and accuracy. in this paper, we present a simple, intuitive and low-cost method to calculate leaf dimensions. the method can be implemented in a desktop software or a mobile application, allowing researchers to use the mobile version to perform field measurements and the desktop version if they wish to perform measurements in the laboratory. for the leaf dimensions determination, we need only capture images of the leaves accompanied of a scale pattern printed on paper. the images can be taken by a simple camera. there is no need to purchase extra equipment. personal devices can be used, at no additional cost. our method presents the following characteristics, which are the main contributions of this work: specific to perform leaf sizing (area, width, length and perimeter); it does not require pre-processing; automatically detects the scale pattern. 2 related work several methods for leaf area estimation based on digital images were proposed in last years. some works use the color information of the images to extract the interest objects and estimate leaf dimensions [6, 16]. extracting color information may require user intervention at filtering step and even the need of using other software. in our method, we use grayscale images, because the contrast between the objects and the background is enough to separate them, even without the color information. as in our method, some works used an object of known dimensions as a scale pattern. jadon [11] chose a coin, but this shows strong reflection when illuminated. thus, that proposal demonstrates a greater sensitivity to lighting conditions. easlon et al. [6] and tech et al. [35] opted for a square. easlon et al. [6] detect the square automatically using color information. however, as previously mentioned, the color needs to be calibrated by the user. the method of tech et al. [35] requires the square to be positioned at a specific position in the image. in our method, the square is automatically detected through its shape. moreover, it can be placed anywhere in the image. imagejfootnote 1 software [28] has been used to calculate the leaf area [8, 19]. however, it is a general-purpose image processing software, which requires the user to perform several steps until obtaining the measurement, such as selecting a measurement line known in the image. we propose to create an intuitive computational tool so that the user can open their images in the software and through a few simple steps perform the leaf measurements calculation. in addition, we intend to create a specific tool for leaf morphology analysis, allowing the insertion of new functionalities related to plant physiology. as it is an application widely used by researchers, imagej will be used to validate our method results. there are also some mobile applications for leaf area calculation. the petiole application [24] uses a very controlled scenario. the calculation starts with a calibration step and the device position should be kept for the area estimation. moreover, it requires user interaction to identify the leaves position on the image. the application easy leaf area free [6] needs a color calibration step, which makes the application less intuitive for the user. we propose a simple method that does not require user calibration. besides methods based only on computer vision techniques, methods for leaf area measurement with artificial neural networks (anns) are appearing recently. siswantoro and artadana [30] used a fixed webcam to capture digital images of the individual leaves. they applied a thresholding step for image segmentation and extract the leaf length, width, area, and perimeter in pixels. finally, ann proposed uses the extracted features to estimate the real leaf area. sa-bouri et al. [26] compared three methods to predict leaf area in cereals: regression models, ann, and ann combining fuzzy logic. the neural network developed by the authors is similar to the previously cited work and uses leaf width and length as inputs. two fluorescent light sources were fixed in specific positions to keep the illumination conditions. the proposal based on neuro-fuzzy inference obtained better performance concerning the work experiments. the advantages of our proposal compared to these networks are automatic detection of the interest objects and no need for training. in addition, while these works need a fixed capture scenario, our method can deal with images captured from different positions and less controlled lighting conditions. 3 proposed method most existing methods for leaf dimensions estimation depends on user intervention. this user interaction can introduce subjectivity and lead to inaccuracies in results. other proposals require a specific positioning of scene objects and calibration steps. these constraints make the process difficult. in this work we propose a method to estimate the leaves dimensions in few steps, which do not depend on the user interference. our proposed method for leaf dimensions calculation is composed of six steps: capture, thresholding, obtaining contours, filtering, detection and dimensions calculation. the fig. 1 shows an overview of the proposed method. we present images and information extracted in each step. fig. 1. steps of the proposed method. above each image, we have the notation referring to the information obtained in the respective step. full size image 3.1 capture in the first step, an image of the leaf (or leaves) is captured so that its dimensions can be estimated. the leaf must be accompanied by a scale pattern whose real size is known. we chose a square pattern printed on paper. this pattern can be placed anywhere on the image and is automatically detected to estimate the leaf dimensions, as described in sect. 3.5. the leaf is not a plane surface. it must be planned to avoid underestimating the measures. for this purpose, a glass or acrylic plate may be used to compress the leaf and keep it stretched. we used a commercial clear glass plate in the tests to keep the projection area of the leaf parallel to the surface, avoiding curvatures and folds. furthermore, the background should be a light color in order to ensure contrast with the interesting objects. reflection is critical to the thresholding process described in sect. 3.2 and should be avoided. it is recommended the image be captured at a perpendicular angle to avoid distortion problems. various capture distances and scale pattern sizes were tested and the results will be discussed in sect. 4. it is important to mention that our method allows more than one leaf to be captured in the same image. the dimensions of each leaf are individually estimated. 3.2 thresholding in the capture step, a color image \(i_c\) is obtained, containing the leaf (or leaves) and the scale pattern. the image is converted to grayscale, resulting in a new image \(i_g\). this conversion and most of the image processing functions were implemented using opencv library [2] in c++ programming language. we applied an otsu’s thresholding [21] on the \(i_g\) image. this algorithm is based on the histogram image. it finds a threshold t which lies in between two peaks such that variances to both classes are minimal. based on this threshold t we achieve a binary image \(i_b\) where: $$\begin{aligned} i_b(x,y) = {\left\{ \begin{array}{ll} 0, &{} \text{ if } i_g(x,y) > t \\ 1, &{} \text{ if } i_g(x,y) \le t \end{array}\right. }. \end{aligned}$$ 3.3 obtaining contours in the thresholding step, a binary image \(i_b\) is obtained, where the leaf and the scale pattern are white and the background is black. it is necessary to extract the contours of the interest objects, that is, the leaves and the scale pattern. in this step, we apply suzuki et al. [33] algorithm in the \(i_b\) image. this method extracts a set \({\textbf {c}}\): $$\begin{aligned} {\textbf {c}} = \{{\textbf {c}}_1, {\textbf {c}}_2, ... ,{\textbf {c}}_n\}, \end{aligned}$$ where each \({\textbf {c}}_i\) is a contour, represented by the set of points that compose it: $$\begin{aligned} {\textbf {c}}_i=\{(x_1, y_1),(x_2, y_2), ..., (x_m, y_m)\}, \end{aligned}$$ where each \((x_j, y_j)\) is the position of a point (pixel) in the image. 3.4 filtering in the previous step, a set \({\textbf {c}}\) of n contours is obtained. this set can contain noise, i.e., contours that do not represent interest objects. these contours should be filtered. in general, the bad contours are much smaller (image noise, for example) or much larger (a table contour, for example) than the interest objects. to overcome this problem, a filtering by contour area is performed. the contour areas in pixels are estimated by green formula [32]. thus, from c, we obtain the set of areas: $$\begin{aligned} {\textbf {a}} = \{a_1, a_2, ... ,a_n\}, \end{aligned}$$ where each \(a_i\in {\textbf {a}}\) is the area of the respective contour \({\textbf {c}}_i\in {\textbf {c}}\). the areas \(a_k\in {\textbf {a}}\) that are not in the range \([a_{min},a_{max}]\) of expected areas, are discarded, as well as their respective contour \({\textbf {c}}_k\). the sets \(c^f\subseteq c\) and \(a^f\subseteq a\) are the filtered sets of contours and areas, respectively. experimentally, we defined the values \(a_{min} = 10^3\) and \(a_{max}=10^{10}\) which proved to be efficient. however, some noise, such as shadows, may not have been completely removed. it is possible to manually remove these artifacts with a simple tool available in the software. we observed the noise presence in 30% of the analyzed images. 3.5 detection in the filtering step, the sets \({\textbf {c}}^f\) and \({\textbf {a}}^f\), are obtained, representing the contours and their respective areas after filtering. in order to automate the process, we need to separate the scale pattern from the leaves. as the scale pattern is a square and it presents a different geometry from the leaves, we can perform a shape-based detection of the square in the contours set. first, we apply a polygonal approximation on each contour \({\textbf {c}}_i^f\in {\textbf {c}}^f\) using the douglas-peucker algorithm [5]. this function reduces each contour \({\textbf {c}}^f_i\) to a simplified contour \({\textbf {c}}^p_i\subseteq {\textbf {c}}^f_i\). thus, we have the set: $$\begin{aligned} {\textbf {c}}^p=\{{\textbf {c}}_1^p, {\textbf {c}}_2^p, ..., {\textbf {c}}_n^p\}, \end{aligned}$$ where \({\textbf {c}}_i^p\) is the polygonal approximation of the contour \({\textbf {c}}^f_i\in {\textbf {c}}^f\). to determine the square contour \({\textbf {c}}^p_s\in {\textbf {c}}^p\), we analyze some conditions. the number of points in \({\textbf {c}}^p_s\) must be equal to 4, i.e., \(|{\textbf {c}}^p_s| = 4\). in addition, the cosine of the angles \(\theta \) formed by the contour sides are calculated. if \(|cos(\theta )| < 0.3\) for all 4 angles, this contour will be \({\textbf {c}}^p_s\). this method was efficient in extracting the square for 98% of the tested images. thus, we obtain the contour \({\textbf {c}}^p_s\in {\textbf {c}}^p\) and its corresponding \({\textbf {c}}_s\in {\textbf {c}}^f\) in the set of non approximated contours. in terms of area, we have the respective area of the square in pixels \(a_s\in {\textbf {a}}^f\) and the set of leaf areas \({\textbf {a}}^l = {\textbf {a}}^f \{a_s\}\). 3.6 dimensions calculation at this point, all the interest objects are properly separated and contoured. therefore, our method is ready to calculate the dimensions of each leaf in the image. to facilitate the calculation of real leaf length and width, the leaves are aligned according to the principal component analysis (pca) [3]. pca is applied to each original leaf contour \({\textbf {c}}_i\in {\textbf {c}}^f \{{\textbf {c}}_s\}\) to find the main leaf direction. pca returns the centroid \(k_i = (x_k,y_k)\) of each leaf contour and the angle of the main direction \(\theta _i\). we associated this direction with the length of the leaf. each contour is aligned so that the main direction coincides with the horizontal axis of the image. thus, each contour point \((x_j,y_j)\in {\textbf {c}}_i\) is rotated by the equation: $$\begin{aligned} x_j = (x_j x_k)~cos(\theta _i) (y_j y_k)~sin(\theta _i) + x_k,\\ y_j = (x_j x_k)~sin(\theta _i) + (y_j y_k)~cos(\theta _i) + x_k. \end{aligned}$$ the real area \(a^r_s\) of the scale pattern in cm\(^2\) is known. we have the scale pattern area \(a_s\) in pixels and each leaf area \(a_i\in a^l\) in pixels. thus, the real area of each leaf \(a^r_i\) in cm\(^2\) can be obtained by the following equation: $$\begin{aligned} a^r_i = \frac{a_i a^r_s}{a_s}. \end{aligned}$$ the real perimeter \(p^r_s\) of the scale pattern in cm is known. we calculated the perimeter in pixels of each leaf and the scale pattern. thus, we achieved the set: $$\begin{aligned} {\textbf {p}} = \{p_1, p_2, ... ,p_n\} \cup {p_s}, \end{aligned}$$ where, each \(p_i\in {\textbf {p}}\) is the perimeter of the respective leaf contour \({\textbf {c}}_i\in {\textbf {c}}^f\). and \(p_s\) is the scale pattern. to obtain the real perimeter \(p^r_i\) of the leaf, we use the following equation: $$\begin{aligned} p^r_i = \frac{p_i p^r_s}{p_s}. \end{aligned}$$ (1) we know the real side \(d^r_s\) of the scale pattern in cm. the pixel measurement on each side \(d_s\) of the pattern can be obtained by dividing the perimeter \(p_s\) by four. the length \(l_i\) and width \(w_i\), in pixels, of each leaf can be obtained by the rectangular bounding box of the rotated contour. in this way, the base and height of this rectangle will correspond to leaf length \(l_i\) and width \(w_i\) in pixels, respectively. thus, the real dimensions of each leaf can be obtained by the following equations: $$\begin{aligned} w^r_i = \frac{w_i d^r_s}{d_s},&~~~~~ l^r_i = \frac{l_i d^r_s}{d_s}. \end{aligned}$$ as a restriction of the proposed method, the largest between the two dimensions found is considered to be the length of the leaf. leaves that do not have this characteristic will have their dimensions reversed, that is, the researcher will have to analyze and correct the results manually before performing any analysis. 4 results and discussion in this section, we describe the experiments performed to evaluate our proposed method. we performed controlled tests with known objects and tests with leaves. for capture the images, we used five devices: three smartphones (samsung galaxy j2 core, motorola moto c and motorola one vision) and two scanners (hp deskjet 2050 and multixpress m5360rx). table 1. pearson’s correlation coefficient (r), rer average (\(\mu \)), rer standard deviation (\(\sigma \)), \(r^2\) and regression equation by dimension for the ellipses pattern.full size table 4.1 metrics in this subsection, we present the metrics used to evaluate quantitatively our method results. the relative error rate (rer) represents the percentage of the difference between the value estimated \(v_{est}\) and achieved with the standard method \(v_{std}\). it can be calculated as: $$\begin{aligned} rer(\%) = \frac{|v_{est} v_{std}|}{v_{std}} \cdot 100. \end{aligned}$$ the rer measure the accuracy of the proposed method in relation to the standard method. given a set of values estimated by our method \(v_{est}\) and the respective values achieved with the standard method \(v_{std}\), we analyzed the correlation between these values. we estimated the linear regression equation \(y=ax+b\), the determination coefficient \(r^2\) and the pearson’s correlation coefficient r, using the google sheets software. 4.2 controlled tests in order to evaluate our method accuracy, we performed preliminary experiments with a controlled pattern. for these tests, we used a pattern containing four ellipses and a scale pattern of 4 cm side. we used a glass plate to plan the pattern as explained in the sect. 3.1 and captured perpendicular images from distances between 20 and 50 cm. in a general analysis, it is possible to see that the method was able to measure the dimensions accurately, achieving a good performance using simple smartphone cameras. table 1 presents the results of the ellipse tests. analyzing the regression equation and the correlation coefficient, it is possible to verify a strong correlation between the values estimated by the proposed method and the expected values. in addition, we achieved \(r^2 > 0.95\) for all dimensions. we achieve rer mean below 3.5% for all the dimensions, which shows the accuracy of our method. furthermore, we note the regression equations present slope close to 1 and small intercept values. 4.3 leaf tests in this subsection, we introduce the dataset collected for measuring leaf dimensions, and present the results of the tests performed to evaluate the proposed method. table 2. \(r^2\) values and regression equations by species and dimensions in relation to the standard method adopted.full size table data acquisition. for this work, annual and perennial species were chosen, whose cultures have significant participation in the brazilian and international gross domestic product (gdp) [10]. all leaves were collected in the morning (before 9:00 am, to guarantee high turgidity), with no apparent/visual symptoms of nutritional deficiency, pathogen attacks, and diseases. the leaves were collected and analyzed on the same day, at the natural sciences laboratory of the instituto federal de educação ciência e tecnologia de minas gerais ouro branco campus. the sample is composed of 118 leaves of 6 species: 18 of the bean, 18 of coffee, 11 of cotton, 21 of eucalyptus, 20 of orange, and 30 of soybean. images were captured between 12 and 50 cm of one or multiple leaves, accompanied by a scale pattern of sides measuring 1, 2, 4, or 6 cm. we used motorola moto c and motorola one vision smartphones. in addition, the leaves also were scanned accompanied by the 1 cm scale pattern using the printer scanner multixpress. overall, our dataset is composed of 529 leaves images. the images are available at the following link https://bit.ly/42dbynv. method evaluation. to verify the results accuracy, we compared with two methods, the imagej software [28] and a simple, but very widespread manual method. the imagej software was used as the standard method. as previously stated in sect. 2, this application is widely used by researchers in the literature. the steps used to estimate the leaf dimensions with imagej software were: 1) select image file; 2) convert image to grayscale; 3) perform a thresholding; 4) draw a line indicating the side of the scale pattern; 5) set the scale known distance; 6) select the leaf and measure area and perimeter; 7) draw a line indicating the width of the leaf and measure the length; 8) draw a line indicating the length and measure. it is important to note that those steps depends on the user interaction and, consequently, are subject to errors. fig. 2. average rer in relation to standard method. (a) by scale pattern. (b) by capture distance. (c) by species. the error bar represents the standard deviation full size image the manual weight-based method for area estimation [22] starts by tracing the outline of the leaf on a paper, whose grammage g is known. the contour is cut and its weight w is measured by a precision balance. the area a is estimated by the formula \(a = \frac{w}{g}\). the same contour used in the weight-based method is measured with a simple rule to estimate the expected width and length. in addition, a line or string is passed around the contour and then measured with a rule to estimate the expected perimeter. table 2 presents the coefficient of determination (\(r^2\)) and the regression equations by species and dimensions in relation to the standard method (imagej software). in general, the performance for area and perimeter was satisfactory. for these dimensions, the \(r^2\) values were above 0.96 and the equations presented angular coefficients close to one for all species. regarding width and length calculation, we note some problems in the cotton results with \(r^2\) values below 0.75. the morphology of the leaves of this species makes it difficult to estimate these dimensions. however, for the other species, our method achieved \(r^2\) values above 0.94 for these dimensions. in general, the results were promising and indicate our method perform similarly to the standard method. figure 2c shows the average rer for each species and dimension in relation to the standard method. the errors for perimeter were the smallest, between 1.98 and 3.25%. for area, we achieved errors between 2.32 and 5.40%. cotton had the highest errors for length and width. as previously mentioned, the leaves morphology makes it difficult these dimensions estimation for cotton. however, for the other species, the average rer for length and width were between 1.65 and 4.31%. overall, the errors achieved showed the method applicability for measuring leaves dimensions, specially those with simpler morphologies. fig. 3. scatter plot for three methods and all images. (a) area. (b) perimeter. (c) width. (d) length. full size image figure 2a presents the average rer for each scale pattern size used. we observe that our method has a worse performance for the 1 cm pattern. the hypothesis for this variation is the manual line selection in the imagej procedure. depending on the direction of this line selection, the real dimension can present a significant difference, especially for images with distortion and the smallest pattern. on the other hand, our method estimates the square measures automatically, without user interaction. for the other scale sizes, the results for the methods was quite similar. in general, the results indicate that the size of the pattern does not significantly influence the method’s performance. the average rer for each capture distance is presented in fig. 2b. to facilitate the comparison, we grouped the tests with capture distances shorter than 20 cm. the tests for this group achieved errors slightly higher than the others. as previously explained, the imagej method needs a manual line selection in the image. since images captured more closely can present distortion, the user’s choice when selecting the line can affect the results. for the other distances, the methods performed similarly. this indicates that the capture distance does not have a strong influence, when the capture conditions are respected. table 3. pearson’s correlation coefficient (r), rer average (\(\mu \)) and standard deviation (\(\sigma \)) for our method \(\times \) manual methods.full size table figure 3a, 3c, 3d and 3b present the scatter plots considering all the images used for tests. these plots also include the results of tests performed using the manual methods previously described. measurements by manual methods are performed per leaf. thus, the yellow dots on the same line of the graph represents the same leaf. our method achieved \(r^2\) values above 0.975 for all dimensions. these values were greater than those of manual method. furthermore, the straight lines of our method presented a slope closer to one, except in the length graph. for this dimension, as previously discussed, our method presents limitation to deal with certain leaves morphology. table 3 presents the pearson’s correlation coefficient r, rer average (\(\mu \)) and rer standard deviation (\(\sigma \)) for our method and manual method in relation to the standard method. pearson’s coefficients indicate a strong correlation for both methods. in relation to the rer metric, we observe the manual method present higher values. due to the several manual steps, manual method is more difficult to control and more susceptible to errors. discussion. we verify that the proposed method was able to estimate leaf dimensions satisfactorily in comparison to imagej, which is widely applied in the literature. in terms of time to estimate the measures, our method is more efficient than the imagej software, which requires less user-side intervention. in a comparison made with nine images, the proposed method estimated all dimensions with an average time of 15 s, while imagej took 78 s. however, our method presents some limitations. it is sensitive to the capture angle and distance, illumination, and the noises that may appear in the image. images taken from an inclined angle suffer from perspective distortion. depending on the lighting location may cause shadows or reflections in the image. all these factors compromise the calculation of leaf dimensions. these are limitations typical of applications involving image processing, and we intend to propose improvements to minimize such problems. leaf morphology is also a limitation of our method, especially on the width and length calculation. these dimensions still need some adjustments for species with more complex leaf morphologies, such as trifoliate or lobed leaves. these morphologies make difficult the pca alignment, due to the irregular shape. in the imagej software, the width and length direction are manually defined. on the other hand, the area and the perimeter have already proved to be quite efficient. except for the trifoliate bean leaf that overestimated the area in the petiole region, where the leaflets are inserted. this leaf presented problems for both methods, especially for higher capture distances. finally, the low contrast between the leaf and the background is another limitation of our proposal. when the leaf presents a light hue, the thresholding process does not work correctly. an alternative for these light leaves is to use a light scale pattern and a dark background to increase the contrast. 5 conclusions we proposed a simple and low-cost method for leaf dimensions calculation based on image processing. our method estimates area, perimeter, length, and width for one or more leaves directly from the images. the software automatically detects each leaf in the image and allows estimate statistics such as average and standard deviation. the only information provided by the user is the size of the scale pattern. no additional user interaction is required. the method estimates automatically the location of leaves and scale pattern. in general, the measurement of leaf morphological parameters (e.g. area, perimeter, length, and width) by this method is an excellent tool to explain plant responses in several areas: evolution, ecophysiology, improvement, management. the results demonstrated the accuracy of the method proposed for this purpose, in addition to its practicality and speed. moreover, these measurements can be employed in derivations and other parameters, such as the leaf area/ perimeter ratio, the leaf area index, the specific leaf area, and the length/width ratio. to conclude, as future works, we intend to improve the method, especially length and width estimation. furthermore, we can perform an improvement in the segmentation step, avoiding the detection of noise in the detected contours. finally, we are developing free mobile and desktop applications from this method that will be made available to any professional and student who wants to perform leaf morphological phenotyping. the source code of these applications can be found in the links https://github.com/karlaflorentino/lima-desktop (desktop) and https://github.com/luizmaurilio/lima (mobile). notes 1.https://imagej.nih.gov/ij/. references antunes, w.c., pompelli, m.f., carretero, d.m., damatta, f.: allometric models for non-destructive leaf area estimation in coffee (coffea arabica and coffea canephora). annal. appl. biol. 153(1), 33–40 (2008) article  google scholar  bradski, g.: the opencv library. dr dobb’s j. softw. tools 25, 120–125 (2000) google scholar  cohen-or, d., et al.: a sampler of useful computational tools for applied geometry, computer graphics, and image processing. crc press (2015) google scholar  dornbusch, t., et al.: plasticity of winter wheat modulated by sowing date, plant population density and nitrogen fertilisation: dimensions and size of leaf blades, sheaths and internodes in relation to their position on a stem. field. crop. res. 121(1), 116–124 (2011) google scholar  douglas, d.h., peucker, t.k.: algorithms for the reduction of the number of points required to represent a digitized line or its caricature. cartographica int. j. geograph. inf. geovisual. 10(2), 112–122 (1973) google scholar  easlon, h.m., bloom, a.j.: easy leaf area: automated digital image analysis for rapid and accurate measurement of leaf area. appl. plant sci. 2(7), 1400033 (2014) article  google scholar  evert, r.f., eichhorn, s.e.: raven: biology of plants. no. 581 rav (2013) google scholar  gao, j., et al.: measuring plant leaf area by scanner and imagej software. china vegetables 2, 73–77 (2011) google scholar  gely, c., laurance, s.g., stork, n.e.: how do herbivorous insects respond to drought stress in trees? biol. rev. 95(2), 434–448 (2020) article  google scholar  ibge. agricultura, pecuária e outros | ibge (2023). https://www.ibge.gov.br/estatisticas/economicas/agricultura-e-pecuaria.html. accessed 17 may 2023 jadon, m.: a novel method for leaf area estimation based on hough transform. jmpt 9(2), 33–44 (2018) article  google scholar  janhäll, s.: review on urban vegetation and particle air pollution-deposition and dispersion. atmos. environ. 105, 130–137 (2015) article  google scholar  laughlin, d.c.: nitrification is linked to dominant leaf traits rather than functional diversity. j. ecol. 99(5), 1091–1099 (2011) article  google scholar  li, y., et al.: spatiotemporal variation in leaf size and shape in response to climate. j. plant ecol. 13(1), 87–96 (2020) google scholar  liancourt, p., et al.: leaf-trait plasticity and species vulnerability to climate change in a mongolian steppe. glob. change biol. 21(9), 3489–3498 (2015) google scholar  liang, w.z., kirk, k.r., greene, j.k.: estimation of soybean leaf area, edge, and defoliation using color image analysis. comput. electron. agricult. 150, 41–51 (2018) google scholar  long, s.p., zhu, x.g., naidu, s.l., ort, d.r.: can improvement in photosynthesis increase crop yields? plant cell environ. 29(3), 315–330 (2006) article  google scholar  lu, j., et al.: detection of multi-tomato leaf diseases (late blight, target and bacterial spots) in different stages by using a spectral-based sensor. sci. rep. 8(1), 2793 (2018) google scholar  maloof, j.n., nozue, k., mumbach, m.r., palmer, c.m.: leafj: an imagej plugin for semi-automated leaf shape measurement. jove (j. visual. exp.) (71), e50028 (2013) google scholar  marek, j., et al.: photoynthetic and productive increase in tomato plants treated with strobilurins and carboxamides for the control of alternaria solani. sci. hortic. 242, 76–89 (2018) google scholar  otsu, n.: a threshold selection method from gray-level histograms. ieee trans. syst. man cybern. 9(1), 62–66 (1979) article  google scholar  pandey, s., singh, h.: a simple, cost-effective method for leaf area estimation. j. bot. 2011(2011), 1–6 (2011) google scholar  peterson, a.g.: reconciling the apparent difference between mass-and area-based expressions of the photosynthesis-nitrogen relationship. oecologia 118(2), 144–150 (1999) article  google scholar  polunina, o.v., maiboroda, v.p., seleznov, a.y.: evaluation methods of estimation of young apple trees leaf area. bullet. uman natl. univ. horticult. 2, 80–82 (2018) article  google scholar  poorter, h., et al.: a meta-analysis of plant responses to light intensity for 70 traits ranging from molecules to whole plant performance. new phytol. 223(3), 1073–1105 (2019) google scholar  sabouri, h., et al.: image processing and prediction of leaf area in cereals: a comparison of artificial neural networks, an adaptive neuro-fuzzy inference system, and regression methods. crop sci. 61(2), 1013–1029 (2021) google scholar  sanz-sáez, á., et al.: leaf and canopy scale drivers of genotypic variation in soybean response to elevated carbon dioxide concentration. glob. change biol. 23(9), 3908–3920 (2017) google scholar  schneider, c.a., rasband, w.s., eliceiri, k.w.: nih image to imagej: 25 years of image analysis. nat. methods 9(7), 671–675 (2012) article  google scholar  shahnazari, a., et al.: effects of partial root-zone drying on yield, tuber size and water use efficiency in potato under field conditions. field crop. res. 100(1), 117–124 (2007) google scholar  siswantoro, j., artadana, i.b.m.: image based leaf area measurement method using artificial neural network. in: 2019 international conference of artificial intelligence and information technology (icaiit), pp. 288–292. ieee (2019) google scholar  srinivasan, v., kumar, p., long, s.p.: decreasing, not increasing, leaf area will raise crop yields under global atmospheric change. glob. change biol. 23(4), 1626–1635 (2017) article  google scholar  stewart, j.: calculus: concepts and contexts. in: cengage learning (2009) google scholar  suzuki, s., be, k.: topological structural analysis of digitized binary images by border following. comput. vis. graph. image process. 30(1), 32–46 (1985). https://doi.org/10.1016/0734-189x(85)90016-7 taiz, l., zeiger, e.: auxin: the first discovered plant growth hormone. in: plant physiology, 5th edn, pp. 545–582. sinauer associates inc., publishers, sunderland (2010) google scholar  tech, a.r.b., et al.: methods of image acquisition and software development for leaf area measurements in pastures. comput. electron. agric. 153, 278–284 (2018) google scholar  villar, r., et al.: applying the economic concept of profitability to leaves. sci. rep. 11(1), 1–10 (2021) google scholar  wang, l., et al.: qtl fine-mapping of soybean (glycine max l.) leaf type associated traits in two rils populations. bmc genomics 20(1), 1–15 (2019) google scholar  wellstein, c., et al.: effects of extreme drought on specific leaf area of grassland species: a meta-analysis of experimental studies in temperate and sub-mediterranean systems. glob. change biol. 23(6), 2473–2481 (2017) google scholar  weraduwage, s.m., et al.: the relationship between leaf area growth and biomass accumulation in arabidopsis thaliana. front. plant sci. 6, 167 (2015) google scholar  wright, i.j., et al.: assessing the generality of global leaf trait relationships. new phytol. 166(2), 485–496 (2005) google scholar  wright, i.j., et al.: the worldwide leaf economics spectrum. nature 428(6985), 821–827 (2004) google scholar  download references acknowledgments this work received financial support from the fapemig process number apq-00603-21. we thank the agencies cnpq and capes for their financial support in this research. and all the people who collaborated directly or indirectly on this work. author information authors and affiliations department of computer science, federal university of juiz de fora, juiz de fora, minas gerais, brazil karla gabriele florentino da silva, gabriel barreto calixto & luiz maurílio da silva maciel department of computing, federal institute of education, science and technology of minas gerais, ouro branco, minas gerais, brazil karla gabriele florentino da silva, jonas magalhães moreira, luiz maurílio da silva maciel & márcio assis miranda department of natural sciences, federal institute of education, science and technology of minas gerais, ouro branco, minas gerais, brazil leandro elias morais authors karla gabriele florentino da silvaview author publications search author on:pubmed google scholar jonas magalhães moreiraview author publications search author on:pubmed google scholar gabriel barreto calixtoview author publications search author on:pubmed google scholar luiz maurílio da silva macielview author publications search author on:pubmed google scholar márcio assis mirandaview author publications search author on:pubmed google scholar leandro elias moraisview author publications search author on:pubmed google scholar corresponding author correspondence to luiz maurílio da silva maciel . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, k.g.f., moreira, j.m., calixto, g.b., da silva maciel, l.m., miranda, m.a., morais, l.e. (2023). a simple and low-cost method for leaf surface dimension estimation based on digital images. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_10 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computer vision image processing leaf phenotyping publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature single image super-resolution based on capsule neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper single image super-resolution based on capsule neural networks conference paper first online: 12 october 2023 pp 113–128 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) single image super-resolution based on capsule neural networks download book pdf download book epub george corrêa de araújo9, artur jordão10 & helio pedrini9  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 529 accesses 1 citation abstract single image super-resolution (sisr) consists of obtaining one high-resolution version of a low-resolution image by increasing the number of pixels per unit area. this method has been actively investigated by the research community, due to the wide variety of problems ranging from real-world surveillance to aerial and satellite imaging. most of the improvements in sisr come from convolutional networks, in which approaches often focus on the deeper and wider architectural paradigm. in this work, we decided to step up from the traditional convolutions and adopt the concept of capsules. since their overwhelming results in image classification and segmentation problems, we question how suitable they are for sisr. we also verify that different solutions share similar configurations, and argue that this trend leads to fewer explorations of network designs. throughout our experiments, we check various strategies to improve results, ranging from new and different loss functions to changes in the capsule layers. our network achieved positive and promising results with fewer convolutional-based layers, showing that capsules might be a concept worth applying to the image super-resolution problem. in particular, we observe that the proposed method recreates the connection between the different characters more precisely, thus demonstrating the potential of capsules in super-resolution problems. artur jordáo: this work was done when artur jordao was a post-doctoral researcher at the university of campinas. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others compressed image super-resolution using pre-trained model assistance chapter © 2023 a review of single image super resolution techniques using convolutional neural networks article 13 september 2023 recent advances in deep learning for single image super-resolution chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition computer vision supercapacitors super-resolution microscopy video capsule endoscopy 3-d image reconstruction 1 introduction state-of-the-art methods in image super-resolution are based on artificial intelligence concepts, more specifically on deep neural networks, and have achieved visually striking results [6, 22, 40, 61, 63] most recent models are composed of traditional convolutional layers that exhibit limitations, although widely studied and optimized. for example, they prevent a better understanding of the data by the network, not absorbing valuable information such as the interrelationship between its components. as an alternative to convolutional networks, a large body of work has demonstrated the abstraction power of the capsules [14, 42, 60]. for example, sabour et al. [41] presented an implementation of the capsule concept, a group of neurons whose activation vector represents the parameters that describe a specific entity, such as an object or part of it, and its presence. using a three-layer capsule network, sabour et al. achieved results comparable to those of deeper convolutional neural networks for the digit identification problem. the authors also obtained a 5% error rate on the segmentation of two different digits with an 80% overlap – a rate previously achieved only in much simpler cases (with less than 4% overlap between images). hinton et al. [15] reduced by 45% the error rate on the problem of identifying objects from different perspectives while making the network more resistant to adversarial attacks. lalonde and bagci [28] have created a network able to process images tens of times larger while reducing the number of parameters by 38.4% and increasing the quality of medical image segmentation. a few works proposed the usage of capsule-based networks to solve problems that involve low-resolution images. singh et al. [46] proposed a dual directed capsule network, named directcapsnet, which employs a combination of capsules and convolutional layers for addressing very low resolution (vlr) digit and face recognition problems. majdabadi and ko [35] implemented a generative adversarial network (gan) that uses a capsnet as the discriminator for facial image super-resolution, surpassing strong baselines in all metrics. hsu et al. [18] developed two frameworks to incorporate capsules into image sr convolutional networks: capsule image restoration neural network (cirnn) and capsule attention and reconstruction neural network (carnn). the results outperformed traditional cnn methods with a similar number of parameters. despite the positive results, most of these works rely on plain capsnet [41], failing to explore novel capsule concepts. our method bridges this gap. this work focuses on the single image super-resolution problem using capsules. we implement a model based on newer concepts of capsules for single image super-resolution (sisr) problems. throughout our evaluation, we used publicly available and standard benchmarks such as set5 [7], set14 [59], b100 [36], urban100 [19] and div2k [2]. our model yields fine-grained super-resolution images and achieves positive results with fewer convolutional-based layers than baselines. 2 background super-resolution. super-resolution (sr) is the process of obtaining one or more plausible high-resolution images (hr) from one or more low-resolution images (lr) [4, 37, 56]. it is an area that has been studied for decades [21] and has a wide variety of application fields such as smart surveillance, aerial imaging, medical image processing, and traffic sign reading [4, 37, 44, 56]. the relationship between lr and hr images may vary depending on the situation. many studies assume that the lr image is a reduced version of the hr image by bicubic interpolation. however, other degradation factors can be considered in real examples, such as quantization errors, acquisition sensor limitations, presence of noise, blurring, and even the use of different interpolation operators aiming resolution reduction for storage [50]. the first successful usage of neural networks for sisr problems was developed by dong et al. [11]. with the super-resolution convolutional neural network (srcnn) model, they created a complete solution that maps lr images to sr versions with little pre/post-processing, yielding superior results. after their achievement, several other works have advanced the state-of-the-art in the sisr problem [10, 24, 25, 54], but they have strong limitations. these models receive as input an enlarged version of the lr image, usually through bicubic interpolation, and seek to improve the quality of the image. this means that the operations performed by the neural network are all done in high-resolution space, which is inefficient and incurs high processing cost. the computational complexity of the convolution grows quadratically with the size of the input image, whose generation of an sr image with a scaling factor n would result in a cost \(n^{2}\) compared to the processing in the low-resolution space [12]. looking for a way of postponing the resolution increase in the network, shi et al. [44] developed a new layer, called the subpixel convolution (or pixelshuffle). such a layer works equivalent to deconvolution with kernel size divisible by spacing, but it is \(\log _{2}r^{2}\) times faster than deconvolution. their network, named efficient sub-pixel convolutional neural network (espcn), achieved speed improvements of over \(10\times \) compared to srcnn [11] while having a higher number of parameters and achieving better results for an upscaling factor of \(4\times \). the concept of subpixel convolution is currently a common choice to perform upscaling in neural networks. it has been used by several solutions that have reached the best results [6, 30, 33, 40, 53, 61, 63, 64] and participated in several editions of the sisr competition that took place during the new trends in image restoration and enhancement workshop [8, 50, 51, 62]. capsules. initially introduced by hinton et al. [16], the concept of capsule proposes to solve some of the main flaws found in traditional convolutional networks: inability to identify spatial hierarchy between elements and lack of rotation invariance. hinton et al. conclude that, after several stages of subsampling, these networks lose information that makes them possible to identify the spatial relationship between the elements of an image. the authors argue that, contrary to looking for a point of view invariance of the neurons’ activities that use a single output value, neural networks should use local “capsules” which learn to recognize a visual entity implicitly under a limited domain of viewing conditions and deformations. capsule structures encode complex calculations into a small, highly informative output vector. this vector contains information such as the probability of that entity being present in a compact domain. in addition, it comprises a set of instantiation parameters, which would include deformation, pose (position, size, orientation), hue, texture, and illumination condition of the visual entity relative to the version learned by the capsule [41]. although idealized by hinton et al., the first successful implementation of the capsule concept was made by sabour et al. [41]. in their work, the authors created a three-layer capsule network that achieved comparable results with the best results in the mnist [29] digit classification problem – previously achieved by deeper networks only. for this, sabour et al. [41] developed two innovative concepts: dynamic routing and a new activation function. leveraged by sabour et al. [41], many other authors have enhanced the concept of capsules. hinton et al. [15] proposed a new type of capsule composed of a logistic unit that indicates the probability of the presence of an entity and a pose matrix of \(4 \times 4\) representing the pose of that entity. the authors also introduced a new routing algorithm, which allows the outputs of the capsules to be routed to those of the next layer so that the active capsules receive a group of votes from similar poses. hinton et al. showed that their model surpasses the best result in the smallnorb dataset, reducing the number of errors by more than 40% while being significantly more resistant to white-box adversarial attacks. a remarkable work, which made possible the development of our solution, was developed by lalonde and bagci [28]. the authors expanded the use of capsules for the problem of object segmentation and made innovations that allowed, among other gains, to increase the data processing capacity of the capsule network, increasing inputs from \(32 \times 32\) to \(512 \times 512\) pixels. most significantly, they advanced the state-of-the-art in the problem of segmentation of lung pathologies from computed tomography, while reducing the number of parameters by approximately 38%. in particular, the authors modified the capsule routing algorithm and the reconstruction part, and modified the concept of convolutional capsules. recently, a few authors have employed capsules in their solutions for problems involving lr images [18, 35, 46]. it is worth noting that most of these solutions only made small changes to the first capsule networks introduced by sabour et al. [41] and hinton et al. [15]. for example, majdabadi and ko [35] employed a two-layered capsule network with dynamic routing as the discriminator for its multi-scale gradient capsule gan. leveraged by the matrix capsules by hinton et al. [15], hsu et al. [18] created two different approaches: (i) capsules as its main component in the network and reconstructing hr images directly from it (cirnn) and (ii) capsules for the channel attention mechanism (carnn). 3 proposed method the proposed model, named super-resolution capsules (srcaps), is shown in fig. 1. it consists of four main parts: an initial convolutional layer, followed by b sequentially connected residual dense capsule blocks, a new convolutional layer and, finally, a neural network to increase resolution. all the convolution-based layers use the weight normalization technique [43], as it accelerates the training convergence and has a lower computational cost if compared to batch normalization, without introducing dependencies between the examples of the batch [61]. the first convolutional layer, conv act in fig. 1, generates f filters from convolutional kernels of size \(k \times k\) with stride st and padding p, followed by an activation function act. this layer is responsible for converting pixel intensities to local resource detector activations that are used as inputs to the next step in the capsule blocks. fig. 1. diagram of srcaps model. full size image the residual dense capsule blocks, rdcbs, are composed of l convolutional capsule layers followed by an activation function act, with residual connection to their inputs, sequentially connected. the outputs of these layers are concatenated, forming a dense connection, followed by a convolutional layer, as shown in fig. 2. this convolutional layer, with kernels \(1 \times 1\), stride 1 and padding 0, acts as a weighted sum between the various filters, allowing the network to learn which filters are more important, thus reducing dimensionality more efficiently [34, 45, 49]. the output of the rdcb is weighted by a residual scale constant. fig. 2. rdcb diagram. full size image all capsules layers within an rdcb module have the same parameters: the number of capsules per layer c, amount of filters f, kernel size k, stride st and padding p. our capsules employ the routing algorithm suggested by lalonde and bagci [28], because it provides an efficient version of the original capsule definition, as we mentioned before. this algorithm differs from the routing-by-agreement implementation by sabour et al. [41] in two ways. first, we route capsules from the previous layer to capsules in the next layer within a specific spatial window. the original algorithm, on the other hand, directs the output of all previous layer capsules to all capsules in the next layer, varying only the routing weight. second, we share the transformation matrices among all capsules of the same type. in a later step, we decided to replace the initial routing algorithm for the no-routing introduced by gu et al. [13]. the authors argue that the routing procedure contributes neither to the generalization ability nor to the affine robustness of the capsnets; therefore, distinct ways to approximate the coupling coefficients do not make a significant difference since they will be learned implicitly. in the no-routing approach, the iterative routing procedure is removed by setting all coupling coefficients as a constant \(\frac{1}{m}\), where m is the number of capsules in the next layer. we also tried different values for the squashing constant sq used in the squashing function, as done by huang and zhou [20]. the rdcbs are sequentially connected, each having a residual connection with the block input, followed by a new convolutional layer, and the output of that layer has a residual connection with the output of the first convolutional layer. we use residual connections, identified by the symbol in figs. 1 and 2, for the following purposes: they avoid the problem of vanishing gradients (it becomes zero) by introducing shorter paths, which can take the gradient over the entire length of very deep networks, as demonstrated by veit et al. [52]; and the use of residual connections seems to greatly improve training speed [48]. at the end of the model, there is a network used to upscale, called up-sampling network (upnet) (see fig. 3). following previous works [33, 64] and several participants of the ntire 2017 [50] and ntire 2018 [51] competitions, the upnet is composed of subpixel convolutions [44]. the upnet allows the network to implicitly learn the process required to generate the larger version by adding the lr space feature maps and creating the sr image in a single step, saving memory and processing. we prefer this method over deconvolution since it naturally avoids checkerboard artifacts, which with deconvolution must be done using a kernel size that is divisible by stride to avoid the overlapping problem as demonstrated by odena et al. [38]. besides, upnet has a considerably lower computational cost, becoming \(\log _{2}r^{2}\) times faster during training [44]. fig. 3. upnet diagram. full size image loss functions. during training, we evaluated loss functions commonly used in super-resolution problems [56]. due to its simplicity and effectiveness, the first loss function we assess is the l1. previous studies [33, 65] showed that a network trained with l1 has achieved superior results compared to the same network trained with l2. still in the work of zhao et al. [65], the idea of using indices based on the structural similarity index (ssim) [57] is introduced for training neural networks. as previously noted by dong et al. [11], if a metric based on visual perception is used during training, the network can adapt to it. the ssim is calculated as: $$\begin{aligned} \text {ssim}(p) =\frac{2 \mu _{x} \mu _{y}+c_{1}}{\mu _{x}^{2}+\mu _{y}^{2}+c_{1}} \cdot \frac{2 \sigma _{x y}+c_{2}}{\sigma _{x}^{2}+\sigma _{y}^{2}+c_{2}} = l(p) \cdot cs(p) \end{aligned}$$ (1) where \(\mu _{x}\) and \(\mu _{y}\) are the average pixel values in the sr and hr patches, respectively, \(\sigma _{x}\) and \(\sigma _{y}\) are the standard deviations of the same patches, \(\sigma _{x y}\) is the covariance between them, and \(c_{1}\) and \(c_{2}\) are constants added to avoid instabilities when the values of \(\mu _{x}^{2}+\mu _{y}^{2}\) and \(\sigma _{x}^{2}+\sigma _{y}^{2}\) are very close to 0. the l(p) part of the equation calculates the comparison between clipping luminances, while the comparison between their contrasts and structures is calculated by cs(p). since the highest possible value for ssim is 1, and because training a neural network usually aims to minimize the loss function, we can define the \(\mathcal {l}^{\text {ssim}}\) function as: $$\begin{aligned} \mathcal {l}^{\text {ssim}}(p)=1-\text {ssim}(\tilde{p}), \end{aligned}$$ (2) in which \(\tilde{p}\) is the central pixel of patch p. the best performance in the work of zhao et al. [65] was obtained by combining l1 and the multi-scale structural similarity index (ms-ssim) shown in eq. 4 weighing 0.16 and 0.84, respectively. the authors argue that ms-ssim preserves contrast in high-frequency regions, while the l1 preserves color and brightness regardless of the local structure. the ms-ssim value is obtained by combining measurements at different scales using the equation: $$\begin{aligned} \text {ms-ssim}(p)=l_{m}^{\alpha }(p) \cdot \prod _{j=1}^{m} cs_{j}^{\beta _{j}}(p), \end{aligned}$$ (3) where scales are used ranging from 1 (original image) to m (the largest scale used), reducing the image by a factor of 2 every iteration; \(l_{m}\) and \(cs_{j}\) are the same terms as defined in eq. 3 at m and j scales, respectively, while the \(\alpha \) and \(\beta _{j}\) exponents are used to adjust the relative significance of different components. it is worth noting that the luminance comparison (\(l_{m}\)) is calculated only at m scale, while contrast and structure comparisons (\(cs_{j}\)) at each scale. as with ssim, the largest possible value for ms-ssim is 1, so we can use it in a loss function in the form: $$\begin{aligned} \mathcal {l}^{\text {ms-ssim}}(p)=1-\text {ms-ssim}(\tilde{p}). \end{aligned}$$ (4) we also explored the combination of functions employing several different layers of the network, as suggested by xu et al. [58], in which the weighted sum between the calculation of the l1 function after two, three and four residual blocks are used, with weights of 0.5, 0.5 and 1, respectively. after each residual block composing the loss, we add a network based on subpixel convolutions to perform upscaling. besides the above settings, we investigate the benefits of edge maps in the loss function, since l1 may smooth the edges. similarly to pandey et al. [39], we evaluated a combination of the l1 using the sr and hr images and the l1 between its edge maps. however, although their work uses the canny operator [9] to generate the edge map, our work investigated the usage of the sobel operator [47]. we also consider the three-component weighted psnr (3-psnr) and ssim (3-ssim) loss functions  [31]. importantly, such metrics can measure the quality of images and videos. this approach breaks up an image into three parts: edges, textures, and more homogeneous regions. to do this, the sobel operator is applied to the luminance channel of the image and, from the highest calculated value and some pre-established values, the thresholds that delimit each region are calculated. the value of each metric is calculated by applying different weights for each region. li and bovik [31] showed that the weights that achieved the best results were 0.7, 0.15 and 0.15 for 3-psnr, and 1, 0 and 0 for 3-ssim, considering edges, textures, and homogeneous regions, respectively. these values are consistent with the observation that perturbations at the edges of an object are perceptually more significant than in other areas. based on recent solutions available in the literature, barron [5] presented a loss function that is a superset of cauchy/lorentzian, geman-mcclure, welsch/leclerc, generalized charbonnier, charbonnier/pseudo-huber/l1-l2, and l2. this function has two hyperparameters: robustness (\(\alpha \)) and scale (c), with the variation of which is possible to reach all previous functions as specific cases. the general loss function is calculated as follows: $$\begin{aligned} \mathcal {l}^{\text {general}}(x, \alpha , c) \!=\! \left\{ \begin{array}{ll}{\frac{1}{2}(x / c)^{2}} \!\!&{}\!\! {\textrm{ if }\ \alpha =2} \\ {\log \left( \frac{1}{2}(x / c)^{2}+1\right) } \!\!&{}\!\! {\textrm{ if }\ \alpha =0} \\ {1-\exp \left( -\frac{1}{2}(x / c)^{2}\right) } \!\!&{}\!\! {\textrm{ if }\ \alpha =-\infty } \\ {\frac{|2-\alpha |}{\alpha }\left( \left( \frac{(x / c)^{2}}{|2-\alpha |}+1\right) ^{(\alpha / 2)}-1\right) } \!\!&{}\!\! { \textrm{ otherwise }}\end{array}\right. \end{aligned}$$ where x is the difference between the hr and sr pixel values. barron also showed that it is possible to modify its function so that the network learns optimal values for the \(\alpha \) and c parameters, thus providing an appropriate exploration by the network of different error functions. due to its unique features, we use the adaptive loss function for the srcaps training. 4 experimental results datasets. in this work, we employed datasets widely used in the literature [4, 56]. currently, the div2k training set is used in the training of neural networks for the super-resolution problem, while the validation set of div2k, b100, set5, set14, and urban100 are used to validate the results. all datasets are composed of original versions of the images (hr) and their reduced versions by bicubic interpolation algorithm for \(2\times \), \(3\times \), and \(4\times \). in this work, we focus on the \(4\times \) scale factor. metrics. for the validation process of the results obtained, we employed metrics commonly used in the literature. more specifically, peak signal-to-noise ratio (psnr) [17], structural similarity index (ssim) [57] and multi-scale structural similarity index (ms-ssim) [55]. due to space limitations, we refer interested readers to the work developed by wang et al. [56] for a detailed formulation. algorithms that measure the differences between two images often assume that the images are shown side by side or alternated with an empty image displayed in between for a short period before the next image is shown. in contrast, flipping (or alternating) between two similar images reveals their differences to an observer much more effectively than showing the images next to each other. aiming to better approximate human evaluators’ methods, andersson et. al. [3] developed a full-reference image difference algorithm, namely  , which carefully evaluates differences inspired by models of the human vision. table 1. model metrics for each evaluated dataset. for the psnr, ssim, and ms-ssim metrics the higher the better, while for the metric the lower the better.full size table   is designed to have both low complexity and ease of use. it not only evaluates differences in colors and edges, but also pays attention to discrepancies in isolated pixels with colors that greatly differ from their surroundings. outputs a new image indicating the magnitude of the perceived difference between two images at every pixel. the algorithm can also pool the per-pixel differences down to a weighted histogram, or generate a single value, which is the approach we will use during our analysis. this value is zero when both images are the same, and it increases as the more noticeable are the differences. computational resources. the super-resolution capsules (srcaps) model was implemented using the pytorch open-source platform on the pytorch lightning wrapper. the implementation made available by uchidafootnote 1 was used as a foundation, and also metrics from the pytorch image quality collection [23] and the official implementation of the metric [3]. for optimizers implementations, we used the torch-optimizer packagefootnote 2 and for comparative visualization of the metrics and generation of graphs we employed tensorboard [1] and comet.mlfootnote 3 tools. parameter search. to find the best set of parameters for our srcaps model, we used the ray [32] open-source framework (tune), which is a scalable hyperparameter tuning library built on top of ray corefootnote 4. we employed the async successive halving (asha) scheduler during the search, as it decides at each iteration which trials are likely to perform badly, and stops these trials, avoiding wasting resources on poor hyperparameter configurations. we train the models for a maximum of 100 epochs in the div2k training dataset and evaluate them using the ms-ssim performance on the first 10 images from the div2k validation dataset. we select these values to allow fast experimentation of different sets of parameters, as it was observed during numerous training processes that the model tends to start stabilizing performance at around 100 epochs. experimental setup. during the training process of the different models used for comparison, the entries have \(n = 16\) (batch size) pairs of image clippings from the dataset. for all models, during validation, the lr and hr images are used entirely one by one (\(n = 1\)). all models discussed here were evaluated for \(4\times \) super-resolution for 2000 epochs, where each epoch involves only one iteration through the training dataset, and is trained with two different loss functions: l1 and adaptive. other functions have been evaluated, and their results will be briefly discussed throughout this section. the final srcaps model has convolutional kernels with \(k = 3\) in its first layer, followed by 7 rdcbs (\(b = 7\)). the value used for the hyperparameters l and c are the same for all blocks: 3 and 4, respectively. in the last convolutional layer, we chose \(k = 3\), as well as for the convolutional layers internal to the upnet. throughout the neural network, we used the values of \(act = relu\), \(k = 3\), \(f = 128\), \(st = 1\) and \(p = \text {'same'}\). setting the padding to the ’same’ mode means using its value so that the input (h and w) dimensions are preserved in the output, that is, \(p = \frac{k-1}{2}\). although having a smaller number of layers and only seven residual blocks in its composition, the srcaps network has a considerable number of parameters 15m. such a value represents 13.5m, 2.4 and 10.2 more parameters than edsr, rcan and wdsr, respectively. this is due to the vectorial nature of the capsule, which adds an extra dimension to its composition. hr image slices (\(patch\_size\)) of size 128\(\times \)128 and its corresponding 32\(\times \)32 lr slice were used during their training. the updating of the weights of the networks is done by the adam optimizer [27], with \(\beta _{1} = 0.9\), \(\beta _{2} = 0.999\), and \(\epsilon = 10^{-8}\), being the networks trained with an initial learning rate of \(lr = 10^{-4}\) which decays to half of the current value every 500 epochs. when used, the adaptive loss function was initialized with the default values from the official implementation [5]. employing these values is equivalent to starting the training with the charbonnier/pseudo-huber function and letting the network learn from it what values for its parameters and, consequently, which function of the subset of the general function is more appropriate. baselines and results. the edsr model used is the base model defined by lim et al. [33]. we chose the simplest version of the model because it has a smaller number of parameters than the srcaps model. it is composed of 16 residual blocks without residual scale application since only 64 feature maps (filters) are used per convolutional layer. all convolutions, including the ones internal to the upscale network, have a kernel size of 3\(\times \)3. during its execution, the input images are subtracted from the mean rgb values of the training images of the div2k dataset, which are 0.4488, 0.4371 and 0.4040. these values range from 0 to 1 and are multiplied by the maximum pixel value, 255. the rdn [64] model, as well as the edsr, generates 64 filters as the output of its convolutional layers, and uses \(k = 3\) for all convolutional kernels, except for those used in the fusion layers of lff and gff, which have \(k = 1\). in this model, 16 rdb blocks were used, with 8 convolutional layers each. for the wdsr model, we used the original implementation by yu et al. [61]. the chosen version was wdsr-b, with 16 large residual blocks that generate 128 filters, but which internally generate convolutional layers with \(6 \times \) more filters. this model, such as edsr, also subtracts the mean rgb values from the div2k images. the rcan model we used is also the original implementation [63] and is available in the same repository used as a baseline. it is composed of 10 residual groups (rg) that form the residual in residual (rir) network structure, in which each rg is composed of 16 residual channel attention blocks (rcab). it has \(k = 3\) kernel sizes which generate \(c = 64\) filters in all convolutional layers, except for those in the channel reduction and amplification mechanism, which have \(k = 1\), and \(\frac{c}{r}=4\) and \(c = 64\) respectively, with reduction factor \(r = 16\). table 1 summarizes the results obtained for all models and datasets after the learning phase. from this table, we highlight the following points. first, the srcaps model obtained comparable results to the edsr model, sometimes surpassing it for some metrics, particularly in the b100 dataset. second, as shown in figs. 4 and 5, the proposed model manages to recreate the connection between the different characters more precisely, while models with better metrics such as rcan and rdn tend to thin the connection, as they do for the leftmost part of the symbol on top. finally, despite being able to reconstruct with quality rounded edges, a deficiency of the srcaps model is in the reconstruction of linear edges, usually diagonal. it is remarkable the results obtained with the rcan model, reaching the highest value in all metrics. we highlight that our goal is not to push the state-of the art but to bring insights into capsules applied to super-resolution problems. we believe that future research on capsule networks could benefit from our findings. fig. 4. model results for “0891” image from div2k dataset. full size image fig. 5. model results for “0829” image from div2k dataset. full size image 5 conclusions the purpose of this work was to evaluate the use of the capsule concept in the solution of single image super-resolution problems, as well as to verify new forms of training and validate the results of neural networks for this purpose. it was evidenced that, despite the inferior result, a trained network with a smaller number of layers obtained a relevant result, indicating that networks that use capsules can have applications in super-resolution. hypotheses have been raised that the nonlinearity function applied together with the capsules may be a limiting factor, given the different nature of the problem as to its initial usage (super-resolution \(\times \) classification). throughout our work, we investigate the contribution of many hyperparameters such as the activation function, learning rate, optimizer and loss function. regarding the latter, we evaluate loss functions that take the human visual system into account, as suggested by previous works [55, 57]. additionally, we study different architectural designs to compose our capsule network. the fact that the adaptive function [5] is a superset of several others and that it is possible to make the network learn, along with the other weights, the optimal values for its two main parameters (\(\alpha \) and c), allow the network to experiment which loss function best fits the problem. thus, it is possible to train the network starting from a function similar to l1, while modifying it at each iteration to extract as much useful information as possible from the data. the current limitations of the most used metrics in the literature [31] were also emphasized in this work, showing that visual evaluation of the results is still essential. existing metrics have been suggested such as ms-ssim [55] and  [3], encouraging discussion of new metrics. several points of possible improvements that could not be deeply evaluated were identified. as a future line of research, we intend to replace the composition of the upnet network, which is used in much the same way by several networks that have reached the state of the art. one can again verify the usage of the concept of reverse convolutions, or deconvolutions, as used by dong et al. [12], and also of deconvolutional capsules created by lalonde and bagci [28], or use more recent methods from the literature. kim and lee [26] recently proposed the enhanced upscaling module (eum), which achieves better results through nonlinearities and residual connections. notes 1.github.com/s-aiueo32/sr-pytorch-lightning. 2.https://github.com/jettify/pytorch-optimizer. 3.https://www.comet.ml/. 4.https://docs.ray.io/en/master/tune/index.html. references abadi, m., et al.: tensorflow: large-scale machine learning on heterogeneous systems (2015). software available from tensorflow.org google scholar  agustsson, e., timofte, r.: ntire 2017 challenge on single image super-resolution: dataset and study. in: cvpr workshops, pp. 1–8 (2017) google scholar  andersson, p., nilsson, j., akenine-möller, t., oskarsson, m., åström, k., fairchild, m.d.: flip: a difference evaluator for alternating images. in: acm on computer graphics and interactive techniques (2020) google scholar  anwar, s., khan, s.h., barnes, n.: a deep journey into super-resolution: a survey. acm comput. surv. 53(3), 60:1–60:34 (2020). https://doi.org/10.1145/3390462 barron, j.t.: a more general robust loss function. arxiv preprint arxiv:1701.03077 (2017) behjati, p., rodriguez, p., mehri, a., hupont, i., tena, c.f., gonzalez, j.: overnet: lightweight multi-scale super-resolution with overscaling network. in: wacv, pp. 1–11 (2021) google scholar  bevilacqua, m., roumy, a., guillemot, c., alberi-morel, m.l.: low-complexity single-image super-resolution based on nonnegative neighbor embedding. in: bmvc (2012) google scholar  cai, j., gu, s., timofte, r., zhang, l.: ntire 2019 challenge on real image super-resolution: methods and results. in: cvpr workshops, pp. 1–8 (2019) google scholar  canny, j.: a computational approach to edge detection. trans. pattern anal. mach. intell. 8(6), 679–698 (1986) article  google scholar  dong, c., loy, c.c., he, k., tang, x.: image super-resolution using deep convolutional networks. trans. pattern anal. mach. intell. 38(2), 295–307 (2016) article  google scholar  dong, c., loy, c.c., he, k., tang, x.: learning a deep convolutional network for image super-resolution. in: fleet, d., pajdla, t., schiele, b., tuytelaars, t. (eds.) eccv 2014. lncs, vol. 8692, pp. 184–199. springer, cham (2014). https://doi.org/10.1007/978-3-319-10593-2_13 chapter  google scholar  dong, c., loy, c.c., tang, x.: accelerating the super-resolution convolutional neural network. in: leibe, b., matas, j., sebe, n., welling, m. (eds.) eccv 2016. lncs, vol. 9906, pp. 391–407. springer, cham (2016). https://doi.org/10.1007/978-3-319-46475-6_25 chapter  google scholar  gu, j., tresp, v.: improving the robustness of capsule networks to image affine transformations. in: cvpr, pp. 1–15 (2020) google scholar  gu, j., wu, b., tresp, v.: effective and efficient vote atack on capsule networks. in: iclr (2021) google scholar  hinton, g., sabour, s., frosst, n.: matrix capsules with em routing. in: iclr, pp. 1–10 (2018) google scholar  hinton, g.e., krizhevsky, a., wang, s.d.: transforming auto-encoders. in: artificial neural networks and machine learning, pp. 44–51 (2011) google scholar  hore, a., ziou, d.: image quality metrics: psnr vs. ssim. in: icpr, pp. 2366–2369 (2010) google scholar  hsu, j., kuo, c., chen, d.: image super-resolution using capsule neural networks. ieee access (2020) google scholar  huang, j.b., singh, a., ahuja, n.: single image super-resolution from transformed self-exemplars. in: cvpr, pp. 1–9 (2015) google scholar  huang, w., zhou, f.: da-capsnet: dual attention mechanism capsule network. sci. rep. (2020) google scholar  irani, m., peleg, s.: improving resolution by image registration. in: cvgip: graph. model. image process. 53(3), 231–239 (1991) google scholar  ji, x., cao, y., tai, y., wang, c., li, j., huang, f.: real-world super-resolution via kernel estimation and noise injection. in: cvpr workshops, pp. 1–8 (2020) google scholar  kastryulin, s., zakirov, d., prokopenko, d.: pytorch image quality: metrics and measure for image quality assessment (2019). https://github.com/photosynthesis-team/piq kim, j., kwon lee, j., mu lee, k.: accurate image super-resolution using very deep convolutional networks. in: cvpr, pp. 1–8 (2016) google scholar  kim, j., kwon lee, j., mu lee, k.: deeply-recursive convolutional network for image super-resolution. in: cvpr, pp. 1–13 (2016) google scholar  kim, j.h., lee, j.s.: deep residual network with enhanced upscaling module for super-resolution. in: cvpr workshops, pp. 1–15 (2018) google scholar  kingma, d.p., ba, j.: adam: a method for stochastic optimization. in: bengio, y., lecun, y. (eds.) iclr (2015) google scholar  lalonde, r., bagci, u.: capsules for object segmentation. arxiv preprint arxiv:1804.04241 (2018) lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  google scholar  ledig, c., et al.: photo-realistic single image super-resolution using a generative adversarial network. in: cvpr, pp. 1–8 (2017) google scholar  li, c., bovik, a.c.: content-weighted video quality assessment using a three-component image model. j. electron. imag. 19, 19 (2010) google scholar  liaw, r., liang, e., nishihara, r., moritz, p., gonzalez, j.e., stoica, i.: tune: a research platform for distributed model selection and training. arxiv preprint arxiv:1807.05118 (2018) lim, b., son, s., kim, h., nah, s., lee, k.m.: enhanced deep residual networks for single image super-resolution. in: cvpr workshops, pp. 1–8 (2017) google scholar  lin, m., chen, q., yan, s.: network in network. arxiv preprint arxiv:1312.4400 (2013) majdabadi, m.m., ko, s.b.: capsule gan for robust face super-resolution. multim. tools appl. (2020) google scholar  martin, d., fowlkes, c., tal, d., malik, j.: a database of human segmented natural images and its application to evaluating segmentation algorithms and measuring ecological statistics. in: iccv, vol. 2, pp. 416–423 (2001) google scholar  nasrollahi, k., moeslund, t.b.: super-resolution: a comprehensive survey. mach. vis. appl. 25(6), 1423–1468 (2014) article  google scholar  odena, a., dumoulin, v., olah, c.: deconvolution and checkerboard artifacts. distill (2016). http://distill.pub/2016/deconv-checkerboard pandey, r.k., saha, n., karmakar, s., ramakrishnan, a.g.: msce: an edge preserving robust loss function for improving super-resolution algorithms. arxiv preprint arxiv:1809.00961 (2018) ren, h., kheradmand, a., el-khamy, m., wang, s., bai, d., lee, j.: real-world super-resolution using generative adversarial networks. in: cvpr workshops, pp. 1–8 (2020) google scholar  sabour, s., frosst, n., hinton, g.e.: dynamic routing between capsules. in: neurips, pp. 3856–3866 (2017) google scholar  sabour, s., tagliasacchi, a., yazdani, s., hinton, g.e., fleet, d.j.: unsupervised part representation by flow capsules. in: meila, m., zhang, t. (eds.) icml, vol. 139, pp. 9213–9223 (2021) google scholar  salimans, t., kingma, d.p.: weight normalization: a simple reparameterization to accelerate training of deep neural networks. in: neurips, pp. 901–909. curran associates, inc. (2016) google scholar  shi, w., et al.: real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. in: cvpr, pp. 1–8 (2016) google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. in: iclr (2015) google scholar  singh, m., nagpal, s., singh, r., vatsa, m.: dual directed capsule network for very low resolution image recognition. in: iccv, pp. 1–8 (2019) google scholar  sobel, i., feldman, g.: a \(3\times 3\) isotropic gradient operator for image processing (1968). talk at the stanford artificial project google scholar  szegedy, c., ioffe, s., vanhoucke, v., alemi, a.a.: inception-v4, inception-resnet and the impact of residual connections on learning. in: aaai, pp. 4278–4284 (2017) google scholar  szegedy, c., et al.: going deeper with convolutions. in: cvpr, pp. 1–8 (2015) google scholar  timofte, r., et al.: ntire 2017 challenge on single image super-resolution: methods and results. in: cvpr workshops, pp. 1110–1121 (2017) google scholar  timofte, r., gu, s., wu, j., van gool, l.: ntire 2018 challenge on single image super-resolution: methods and results. in: cvpr workshops, pp. 1–17 (2018) google scholar  veit, a., wilber, m.j., belongie, s.: residual networks behave like ensembles of relatively shallow networks. in: neurips, pp. 550–558 (2016) google scholar  wang, x., et al.: esrgan: enhanced super-resolution generative adversarial networks. in: eccv, pp. 63–79 (2019) google scholar  wang, z., liu, d., yang, j., han, w., huang, t.: deep networks for image super-resolution with sparse prior. in: iccv, pp. 370–378 (2015) google scholar  wang, z., simoncelli, e.p., bovik, a.c.: multiscale structural similarity for image quality assessment. in: the thirty-seventh asilomar conference on signals, systems computers, vol. 2, pp. 1398–1402 (2003) google scholar  wang, z., chen, j., hoi, s.c.h.: deep learning for image super-resolution: a survey. trans. pattern anal. mach. intell. 43(10), 3365–3387 (2021) article  google scholar  wang, z., bovik, a.c., sheikh, h.r., simoncelli, e.p.: image quality assessment: from error visibility to structural similarity. trans. image process. 13(4), 600–612 (2004) article  google scholar  xu, j., zhao, y., dong, y., bai, h.: fast and accurate image super-resolution using a combined loss. in: cvpr workshops, pp. 1093–1099 (2017) google scholar  yang, j., wright, j., huang, t.s., ma, y.: image super-resolution via sparse representation. trans. image process. 19(11), 2861–2873 (2010) article  mathscinet  math  google scholar  yu, c., zhu, x., zhang, x., wang, z., zhang, z., lei, z.: hp-capsule: unsupervised face part discovery by hierarchical parsing capsule network. in: cvpr, pp. 4022–4031 (2022) google scholar  yu, j., fan, y., yang, j., xu, n., wang, x., huang, t.s.: wide activation for efficient and accurate image super-resolution. arxiv preprint arxiv:1808.08718 (2018) zhang, k., gu, s., timofte, r.: ntire 2020 challenge on perceptual extreme super-resolution: methods and results. in: cvpr workshops, pp. 1–10 (2020) google scholar  zhang, y., li, k., li, k., wang, l., zhong, b., fu, y.: image super-resolution using very deep residual channel attention networks. in: eccv, pp. 1–8 (2018) google scholar  zhang, y., tian, y., kong, y., zhong, b., fu, y.: residual dense network for image super-resolution. in: cvpr, pp. 2472–2481 (2018) google scholar  zhao, h., gallo, o., frosio, i., kautz, j.: loss functions for image restoration with neural networks. ieee trans. comput. imaging 3(1), 47–57 (2017) article  google scholar  download references author information authors and affiliations institute of computing, university of campinas, campinas, sp, 13083-852, brazil george corrêa de araújo & helio pedrini polytechnic school of the university of são paulo, university of são paulo, são paulo, sp, 05508-010, brazil artur jordão authors george corrêa de araújoview author publications search author on:pubmed google scholar artur jordãoview author publications search author on:pubmed google scholar helio pedriniview author publications search author on:pubmed google scholar corresponding author correspondence to artur jordão . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de araújo, g.c., jordão, a., pedrini, h. (2023). single image super-resolution based on capsule neural networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_8 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords neural network capsule networks image super-resolution publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature preserving privacy, enhancing robustness: federated learning for lung disease identification in chest x-ray images | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper preserving privacy, enhancing robustness: federated learning for lung disease identification in chest x-ray images conference paper first online: 30 january 2025 pp 402–411 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) preserving privacy, enhancing robustness: federated learning for lung disease identification in chest x-ray images download book pdf download book epub weld lucas cunha  orcid: orcid.org/0000-0001-9036-23879,11, cesar castelo-fernandez  orcid: orcid.org/0000-0002-2900-87659,11, rafael simionato  orcid: orcid.org/0000-0001-9392-65989,11, matheus soares de lacerda  orcid: orcid.org/0000-0003-1550-37629 & … samuel botter martins  orcid: orcid.org/0000-0002-2894-391110  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 370 accesses abstract while hospitals routinely gather patient data, such as x-ray images, the challenge of sharing this data across multiple institutions to create a comprehensive and large dataset is hampered by privacy concerns. consequently, this limitation affects the effectiveness of state-of-the-art deep neural networks for tasks like identifying lung diseases in medical images, as they require substantial annotated data. federated learning offers a solution by enabling collaborative training across multiple edge devices or sites, where updates (e.g., neural network weights) are aggregated without sharing patient data, thus maintaining privacy. this work introduces a federated-learning-based approach for automatically detecting lung diseases in chest x-ray images, focusing on preserving data privacy and enhancing robustness. our approach follows the federated learning protocol: decentralized training of neural networks on data from multiple sites (hospitals) and centralized aggregation of knowledge in the server. the solution presents promising results in identifying fourteen lung diseases compared to three baselines within a simulated environment comprising chest x-ray images from five distinct sites. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a study on cnn architectures for chest x-rays multiclass computer-aided diagnosis chapter © 2020 classification of chest diseases using convolutional neural network chapter © 2020 cxr-fl: deep learning-based chest x-ray image analysis using federated learning chapter © 2022 1 introduction the significant generation of data, which is fundamental to modern systems, has increasingly become a powerful force in integrating the physical and digital worlds. this transformation not only changes the way we interact with devices around us but also reshapes entire industrial sectors, opening up new opportunities for countless innovations. the internet of things (iot) market is expected to reach a value of usd 1.39 trillion by 2026, making it a major contributor to the large-scale generation of data. in brazil, the technology has significant potential, with the national bank for economic and social development (bndes) estimating that the country will generate around usd 200 billion in revenue from iot implementations by 2025 [5]. this growth reflects not only technological advancement but also the increasing need for personalized solutions to address specific challenges, particularly those related to privacy. even simple activities generate important inputs for the use of machine learning (ml) and thus empower the development of intelligent tasks according to business needs. hospitals routinely collect patient data (e.g., x-ray images), which constitutes a valuable resource for training ml models that aim at detecting diseases [14]. yet, the volume of disease-specific data (e.g., pneumonia) collected by each hospital often falls short of supporting robust supervised ml models, particularly sophisticated methodologies like convolutional neural networks (cnns) [8, 9]. so, sharing data from multiple hospitals to establish a comprehensive repository (large dataset) offers a promising avenue to overcome this limitation, facilitating the development of more accurate ml models. nevertheless, keeping patients’ privacy is paramount. recently, the american government issued a presidential executive order on safe, secure, and trustworthy artificial intelligence (ai) [18] which mentions the intention to protect americans’ privacy, and explicitly provides mechanisms to strengthen research into privacy-preserving technologies prioritizing federal support to accelerate the development and use of such techniques. this fact alongside many others, such as the publication of regulations like the gdpr [1], highlights the need for new technologies capable of meeting privacy preservation requirements while using techniques widely recognized in the context of ai. fig. 1.adapted from bonawitz et al. [3]. federated learning training protocol. full size image federated learning (fl) [7, 11] is a machine learning methodology tailored to situations where there are decentralized clients and/or when data privacy is a major concern, as exemplified by sensitive medical examination data. the primary objective is to aggregate the knowledge acquired from diverse sources throughout the learning process. this operational principle entails the collaborative execution of training across multiple edge devices or servers, collaboratively aggregating updates (i.e., cnn weights) from these entities without the need to centralize the data, thereby preserving privacy and reducing communication overhead [10, 12, 15]. figure 1 illustrates the fl training approach. the server selects all the available clients among all n clients (a client may vary from a mobile device to an institution, like a hospital or a company, for instance). after selection, the model weights and settings are sent from the server to the clients to initiate the local model training. the training time may differ from client to client, depending on the amount of data and the computing power available at each site. following the training stage, all the clients should report their results to the server by sending the model updates. after receiving all the updates, the server will aggregate the knowledge from all clients into one single model, starting a new federated iteration. this work presents a federated-learning-based solution to train a supervised neural network with data from multiple sites (hospitals) for lung disease identification in chest x-ray images in a decentralized way. throughout successive learning iterations, our approach individually trains a local model for each site, leveraging solely its own data. at the end of each round, the learned weights from all participating sites are shared with a central server, ensuring data privacy by refraining from sharing images. the central server then aggregates these weights into a global model, subsequently disseminating it back to each site for further refinement as part of a new federated iteration. this iterative process runs across multiple rounds. we evaluated our solution within a simulated environment containing chest x-ray images from five distinct sites. our approach relies on a dense convolutional neural network, densenet [6], based on the chexnet architecture proposed by rajpurkar et al. [14] for lung diseases identification. our solution presented promising results in identifying fourteen lung diseases when compared against three baselines trained on the whole dataset. 1.1 motivation we were motivated by two main factors in our work. the first is related to the promotion of techniques that prioritize collaborative approaches over private data strategies. while there are many open datasets available from different medical fields [16], it is estimated that a model trained with such datasets would not be as effective as a collaborative model due to the size of the federated dataset needed for widespread adoption by multiple entities. this could usher in a new era of improved diagnosis accuracy. the other motivating factor is related to the importance of human life. a recent study from the johns hopkins armstrong institute has estimated that more than 795,000 americans suffer from serious harm due to misdiagnosis every year [13]. some of them live permanently disabled, while others lose their lives. we believe that to overcome this situation, the medical field can utilize the benefits of ai to support its professionals in making diagnoses. as shown in a recent study [17], even simple techniques like logistic regression and statistical analysis, when applied to tabular data, can provide crucial information to support better medical decision-making. 1.2 organization the remainder of this article is organized as follows. in sect. 2, we present the methods and materials used in the experiment held by this work. in sect. 3, we show in detail the obtained results and discuss them compared to other works. and finally, in sect. 4, we conclude our article with a summary of our main findings and future work. 2 materials and methods the presented solution employs the principles of federated learning to train a decentralized supervised neural network specifically designed to automate the identification of lung diseases within chest x-ray images. tailored for multilabel classification, wherein each patient image can be assigned zero or more labels (in our case, lung diseases), the proposed solution is application-independent, thus adaptable for other domains and image modalities, such as identifying brain lesions in magnetic resonance (mr) images, for instance. however, a fundamental requirement for its implementation remains the utilization of neural networks. fig. 2. sample images from the chestx-ray14 image dataset. the text positioned above each image denotes its corresponding labels (diseases). full size image 2.1 data while our solution is application-independent, evaluating it proves challenging due to the absence of publicly available large annotated datasets for various medical conditions. in this context, we opted to utilize the chestx-ray14 dataset [19] for assessment, comprising a substantial collection of 112, 120 frontal-view chest x-ray images sourced from 30, 805 distinct patients. figure 2 presents a few examples of images for multiple diseases. each image was annotated with as many as 14 distinct lung disease labels, acquired through automated extraction methods applied to radiology reports. these labels encompass atelectasis, cardiomegaly, consolidation, edema, effusion, emphysema, fibrosis, hernia, infiltration, mass, nodule, pleural thickening, pneumonia, and pneumothorax. images with no diseases are labeled as ‘no finding’. 2.2 model our model is a convolutional neural network with 121 layers. the network is initialized with weights pre-trained in the imagenet dataset [4]. the network is trained using the adam optimizer with default parameters (\(\beta _{1}\) = 0.9 and \(\beta _{2}\) = 0.999). we train the model using mini-batches of size 16. since the dataset has fourteen labels (see sect. 2.1), we replaced the final layer with a 14-output fully connected layer, where we applied an element-wise sigmoidal function. 2.3 federated training following the federated learning protocol illustrated in fig. 1, we conducted a simulation involving five hospitals, leveraging x-ray images from each hospital to facilitate the identification of lung diseases. our methodology entails the utilization of local hospital datasets to train individual local models, one per hospital. after local training, the resultant model weights from all hospitals are sent to a central server, and the weights are aggregated through the federated averaging [12] method, within a single global model that synthesizes the learned knowledge. this global model is then returned to all participating hospitals, perpetuating the federated training process. this process is shown in fig. 3. fig. 3. federated training: experimental implementation. full size image for our experimental setup, we initially partitioned the chestx-ray14 dataset into two distinct subsets: a training set denoted as \(d_{train}\) (comprising \(80\%\) of the data) and a separate testing set labeled as \(d_{test}\) (accounting for \(20\%\) of the data). subsequently, we further subdivided \(d_{train}\) into five distinct subsets, simulating a scenario involving five distinct hospitals. each hospital dataset, referred to as \(h^i\), comprised approximately 18, 000 images. finally, we divided each hospital’s dataset into two distinct portions: a hospital training subset, designated as \(h^i_{train}\) (constituting \(75\%\) of \(h^i\)) to train the local model, and a separate hospital validation subset, denoted as \(h^i_{val}\) (comprising \(25\%\) of \(h^i\)) for local validation. it is worth highlighting that after \(d_{train}\) is split into five subsets, the images themselves remain localized within each hospital site. only the weights of each neural network, trained individually at each site, are shared with the central server. the federated learning process encompassed 30 rounds, each comprising 10 local epochs, resulting in a cumulative total of 300 training iterations per site. in each round, the global model, with the aggregated weights, was evaluated using \(d_{test}\). the orchestration of training and communication across these training sites is coordinated via the flower framework [2] in simulation mode. our simulation required approximately 72 h to complete the training phase on a single computer equipped with the following hardware specifications: an intel core i7 cpu clocked at 1.80ghz (8th generation), with 16 gb of ram, and 4 gb of dedicated gpu memory (nvidia geforce). 3 results and discussion we used the per-class auroc (area under the receiver operating characteristic curve) to evaluate and compare our results against three baselines. these baselines are: 1) wang et al. [19], the work that released the chestx-ray14 dataset and former state-of-the-art for 1 class; 2) yao et al. [20], the former state-of-the-art for 13 classes; and 3) chexnet [14], which, as far as we know, is the current state-of-the-art model for lung disease identification in the chestx-ray14 dataset. we highlight that all baselines were trained over the full training set \(d_{train}\) while our solution used about 5 times less training data per site. although we do not expect our results to surpass the state-of-the-art performance, we do expect them to be comparable. table 1 shows the per-class auroc of our solution (fl) compared to the baselines. we have highlighted in bold the diseases for which our method outperformed wang et al. [19] and underlined the diseases for which we obtained virtually the same results when compared to other methods. table 1. experimental results (per-class auroc) for lung disease detection in the chestx-ray14 dataset.full size table our solution demonstrated comparable performance to wang et al. [19], yielding improved scores (in bold) for specific diseases such as atelectasis, infiltration, nodule, pneumonia, consolidation, and hernia, and virtually the same performance (underlined values) for effusion, mass, pneumothorax, fibrosis and pleural thickening. conversely, it presented inferior results than yao et al. [20] and chexnet [14] in all diseases. this discrepancy can be attributed to the fact that both baselines were trained using the complete training set \(d_{train}\). at the same time, fl utilized substantially less data—each hospital’s training set is approximately one-fifth the size of \(d_{train}\). hence, this comparison lacks parity, and the anticipation of suboptimal results is justified. nevertheless, fl presented promising results for certain diseases in comparison to yao et al.and chexnet, particularly evident in the cases of ‘infiltration,’ ‘consolidation,’ and ‘fibrosis,’ as highlighted in table 1. it is important to highlight a few considerations. firstly, the effectiveness of the baseline methods, notably chexnet, is contingent upon access to a large annotated dataset (e.g., encompassing tens of thousands of labeled images). yet, this prerequisite proves impracticable in most practical clinical routines, given the challenges of obtaining such a volume of annotated data for a specific disease, coupled with the labor-intensive and time-consuming nature of data annotation. consequently, these baseline methods tend to exhibit poorer results when such circumstances cannot be held. in contrast, while prioritizing data privacy, fl operates under the premise of a substantially smaller annotated dataset and strives to harness collective learning from diverse sources. this underscores both the potential inherent in our proposed solution and the substantial room for further enhancement. 4 conclusion this work presented a federated-learning-based solution for automatically identifying lung diseases in chest x-ray images, effectively tackling privacy challenges inherent to medical imaging and enhancing identification robustness through federated learning. this approach facilitates decentralized and collaborative model training, embracing data from diverse sources. our solution was trained on considerably fewer data per site than the baselines, mirroring a more realistic clinical setting. yet, it endeavors to compete with state-of-the-art methods that may not be feasible in a clinical scenario since they demand large annotated sensitive datasets due to privacy constraints. our federated learning model exhibits significant potential for improvements (e.g., fine-tuning), allowing it to contend with state-of-the-art benchmarks following further refinements. while hospitals routinely collect patient data, sharing these data across multiple hospitals to establish a comprehensive large dataset is impractical due to privacy constraints. our solution followed the federated learning protocol to train decentralized dense convolutional neural networks across data from multiple sites (hospitals). throughout successive learning rounds, each hospital’s learned model weights (knowledge) are shared with a central server and aggregated to build a global model, thus preserving data privacy and enhancing robustness. our solution reported promising results in identifying fourteen lung diseases for a comprehensive chest x-ray dataset. although it presented inferior results than state-of-the-art methods, their efficacy depends on a large annotated dataset, a demand often unfeasible within practical clinical workflows. the hurdles of assembling a substantial volume of annotated data for a particular ailment, combined with the laborious and time-intensive data annotation process, underscore these limitations. in contrast, our solution leverages lung disease identification by aggregating the knowledge learned from smaller datasets across multiple sites, mirroring a more realistic clinical setting. for future work, we first intend to refine our dense cnn through fine-tuning, regularization methods, and addressing the class imbalance. second, we aspire to assess other deep neural networks for lung disease identification and consider different data volumes per site, on non-iid scenarios, and various numbers of sites. finally, we intend to evaluate our solution in other medical imaging problems, such as detecting brain lesions in mr images. references regulation (eu) 2016/679 of the european parliament and of the council of 27 april 2016 on the protection of natural persons with regard to the processing of personal data and on the free movement of such data, and repealing directive 95/46/ec (general data protection regulation). official journal of the european union (2016). https://eur-lex.europa.eu/legal-content/en/txt/pdf/?uri=celex:32016r0679. accessed 08 july 2023 beutel, d.j., et al.: flower: a friendly federated learning research framework. arxiv preprint arxiv:2007.14390 (2020) bonawitz, k., et al.: towards federated learning at scale: system design. proc. mach. learn. syst. 1, 374–388 (2019) math  google scholar  deng, j., dong, w., socher, r., li, l.j., li, k., fei-fei, l.: imagenet: a large-scale hierarchical image database. in: 2009 ieee conference on computer vision and pattern recognition, pp. 248–255. ieee (2009) google scholar  gr01: as 9 maiores tendências da iot para o mundo moderno (2022). https://gr1d.io/2022/09/02/inteligencia-artificial-3. accessed 13 dec 2023 huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: proceedings of the ieee conference on computer vision and pattern recognition (cvpr) (2017) google scholar  kairouz, p., et al.: advances and open problems in federated learning. found. trends® mach. learn. 14(1–2), 1–210 (2021) google scholar  lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  math  google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  math  google scholar  lim, w.y.b., et al.: federated learning in mobile edge networks: a comprehensive survey. ieee commun. surv. tutor. 22(3), 2031–2063 (2020) article  math  google scholar  mcmahan, b., moore, e., ramage, d., hampson, s., arcas, b.a.y.: communication-efficient learning of deep networks from decentralized data. in: singh, a., zhu, j. (eds.) proceedings of the 20th international conference on artificial intelligence and statistics. proceedings of machine learning research, vol. 54, pp. 1273–1282. pmlr (2017). https://proceedings.mlr.press/v54/mcmahan17a.html mcmahan, h.b., moore, e., ramage, d., arcas, b.a.: federated learning of deep networks using model averaging. arxiv preprint arxiv:1602.05629 2 (2016) newman-toker, d.e., et al.: burden of serious harms from diagnostic error in the usa. bmj qual. saf. (2023). https://doi.org/10.1136/bmjqs-2021-014130 article  math  google scholar  rajpurkar, p., et al.: chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arxiv preprint arxiv:1711.05225 (2017) ryffel, t., et al.: a generic framework for privacy preserving deep learning. arxiv preprint arxiv:1811.04017 (2018) sheller, m.j., et al.: federated learning in medicine: facilitating multi-institutional collaborations without sharing patient data. sci. rep. 10(1), 12598 (2020) article  math  google scholar  takao, m.m.v., et al.: artificial intelligence in allergy and immunology: comparing risk prediction models to help screen inborn errors of immunity. int. arch. allergy immunol. 183(11), 1226–1230 (2022) article  math  google scholar  the white house: fact sheet: president biden issues executive order on safe, secure, and trustworthy artificial intelligence (2023). https://www.whitehouse.gov/briefing-room/statements-releases/2023/10/30. accessed 13 dec 2023 wang, x., peng, y., lu, l., lu, z., bagheri, m., summers, r.m.: chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2097–2106 (2017) google scholar  yao, l., poblenz, e., dagunts, d., covington, b., bernard, d., lyman, k.: learning to diagnose from scratch by exploiting dependencies among labels. arxiv preprint arxiv:1710.10501 (2017) download references acknowledgments this work was made possible through the invaluable support of sidi, a brazilian institute of science and technology committed to innovating everyday applications. we also extend our sincere gratitude to the federal institute of são paulo for their collaboration throughout the conception and development of this project. furthermore, we would like to thank the state university of campinas, where three of the authors are currently pursuing their phd and master’s degrees, for its ongoing academic support. author information authors and affiliations sidi, campinas, sp, brazil weld lucas cunha, cesar castelo-fernandez, rafael simionato & matheus soares de lacerda federal institute of são paulo, campinas, sp, brazil samuel botter martins state university of campinas, campinas, sp, brazil weld lucas cunha, cesar castelo-fernandez & rafael simionato authors weld lucas cunhaview author publications search author on:pubmed google scholar cesar castelo-fernandezview author publications search author on:pubmed google scholar rafael simionatoview author publications search author on:pubmed google scholar matheus soares de lacerdaview author publications search author on:pubmed google scholar samuel botter martinsview author publications search author on:pubmed google scholar corresponding authors correspondence to weld lucas cunha , cesar castelo-fernandez , rafael simionato , matheus soares de lacerda or samuel botter martins . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cunha, w.l., castelo-fernandez, c., simionato, r., de lacerda, m.s., martins, s.b. (2025). preserving privacy, enhancing robustness: federated learning for lung disease identification in chest x-ray images. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_28 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords federated learning privacy medical imaging lung diseases publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature the effect of statistical hypothesis testing on machine learning model selection | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper the effect of statistical hypothesis testing on machine learning model selection conference paper first online: 12 october 2023 pp 415–427 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) the effect of statistical hypothesis testing on machine learning model selection download book pdf download book epub marcel chacon gonçalves9 & rodrigo silva  orcid: orcid.org/0000-0003-2547-383510  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 643 accesses 4 citations abstract statistical tests of hypothesis play a crucial role in evaluating the performance of machine learning (ml) models and selecting the best model among a set of candidates. however, their effectiveness in selecting models over larger periods of time remains unclear. this study aims to investigate the impact of statistical tests on ml model selection in sequential experiments. specifically, we examine whether selecting models based on statistical tests leads to higher quality models after a significant number of iterations and explore the effect of the number of tests performed and the preferred statistical test for different experimental time horizons. the study on binary classification problems reveals that the use of statistical tests should be approached with caution, particularly in challenging scenarios where generating improved models is difficult. the analysis demonstrates that statistical tests may impede progress and impose overly stringent acceptance criteria for new models, hindering the selection of high-quality models. the findings also indicate that the dominance of versions without statistical tests remained consistent, suggesting the need for further research in this area. although this study is limited by the number of datasets and the absence of pre-test assumption verification, it emphasizes the importance of understanding the impact of statistical tests on ml model selection. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others quality assurance strategies for machine learning applications in big data analytics: an overview article open access 30 october 2024 model selection chapter © 2018 logic-based explainability in machine learning chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biostatistics linear models and regression machine learning model theory statistical learning statistical theory and methods 1 introduction in recent years, machine learning (ml) has become a popular tool for data analysis in various fields such as finance [1], healthcare [2], and marketing [8]. the ultimate goal of machine learning is to build predictive models that can accurately predict the target variable based on the input features. choosing the best model among a set of candidate models, however, can be a daunting task. one approach is to use statistical tests of hypothesis to compare the performance of different models [4]. the statistical tests of hypothesis are widely used in the scientific community to evaluate the significance of a result or to compare the performance of different methods [7]. in machine learning, statistical tests are used to determine whether there is a significant difference between the performance of two or more models. the use of statistical tests in ml model selection has become an important topic of research due to its impact on the performance of the final model [3,4,5, 14, 17]. the use bayesian statistics is studied in [3, 5] while the use of frequentist tests is the focus of [4, 14, 17]. while these studies investigate the ability of statistical tests to choose the best model in a single experiment, this study aims to understand their effectiveness when they are applied over larger periods of time. more specifically, this study investigates whether the statistical tests commonly used for selecting the best ml model in a single experiment remain effective when applied in sequences of experiments. to achieve this, using a set a binary classification problems, the quality of the models selected by each test is examined after a significant number of iterations. other parameters such as the number of iterations and the amount of data collected to feed the statistical tests are also investigated. in this context, the research questions investigated in this study can be laid down as follows: 1. the selection of machine learning models based on statistical tests of hypothesis lead to higher quality models after a large number of iterations? 2. what is the effect of the number of tests performed to acquire data to these statistical test of hypothesis? 3. is there a preferred statistical test for different experimental time horizons? the rest of the paper is organized as follows. in sect. 2, we will provide a brief overview of statistical tests of hypothesis and their application in machine learning. in sects. 3 and 4, we will describe the experimental setup and datasets used in this study. in sect. 5, we will present and analyze the results of our experiments. finally, in sect. 6, we will conclude the paper with a discussion of the implications of our findings and future research directions. 2 statistical hypothesis testing in this section, we introduce the hypothesis tests used in this study. given our objective of comparing two groups, which is the simplest scenario, i.e., comparing two algorithms, we have selected methods for this purpose. we have chosen the t-test as a representative parametric test, while the mann-whitney u test has been selected as the non-parametric option. 2.1 paired t-test for two related samples a paired t-test is a statistical test used to determine whether there is a significant difference between the means of two related samples [11]. while using it, each individual in one sample is paired with an individual in the other sample based on some common characteristic, such as before-and-after measurements of the same individual, or measurements from two different methods applied to the same subjects. it assumes that the two samples are normally distributed and have equal variances. the null hypothesis of the paired t-test is that there is no difference between the means of the two samples. if the p-value returned by it, which is a representation of the probability of obtaining a statistic as extreme as the one observed, assuming that the null hypothesis is true, is less than a predefined significance level (e.g., 0.05), the null hypothesis can be rejected, and it usually concluded that there is a significant difference between the means of the two samples. the scipy.stats.ttest_rel function is the method in the scipy library [16] that performs a paired t-test for two related samples. it can be used in machine learning to compare the performance of two models on the same dataset. this information can be useful in model selection and can help to identify the best performing model for a given task. the test statistic t is computed as follows: $$\begin{aligned} t = \frac{\overline{x}_d \mu _0}{s_d/\sqrt{n}} \end{aligned}$$ (1) where, \(\overline{x}_d\) and \(s_d\) are the average and the standard deviation of the difference between all pairs. \(\mu _0\) is the true mean of the difference under the null hypothesis. it is zero if we want to test whether the average of the difference is significant. the number of pairs is represented by n, which is also used to calculate the degrees of freedom as \(n-1\). 2.2 independent t-test for two independent samples an independent t-test is a statistical test used to determine whether there is a significant difference between the means of two independent samples. the scipy.stats.ttest_ind function is a method in the scipy library [16] that performs an independent t-test for two independent samples. it assumes that the two samples are normally distributed and have equal variances. the null hypothesis of the independent t-test is that there is no difference between the means of the two samples [11]. if the p-value returned by the ttest_ind function is less than a predefined significance level (e.g., 0.05), the null hypothesis can be rejected, and it can be concluded that there is a significant difference between the means of the two samples. the test t statistic is computed as: $$\begin{aligned} t = \frac{\overline{x}_1 \overline{x}_2}{s_p \times \sqrt{\frac{1}{n_1} + \frac{1}{n_2}} } \end{aligned}$$ (2) where $$\begin{aligned} s_p = \sqrt{\frac{(n_1 1)s^2_{x_1} + (n_2 1)s^2_{x_2}}{n_1 + n_2 2} } \end{aligned}$$ (3) \(s_ p\) is the pooled standard deviation of the two samples. it is defined in this way so that its square is an unbiased estimator of the common variance. for more details, see [12]. 2.3 mann-whitney u test the mann-whitney u test is a non-parametric statistical test used to determine whether there is a significant difference between the distributions of two independent samples. the ss.mannwhitneyu function is a method in the scipy library [16] that performs the mann-whitney u test, also known as the wilcoxon rank-sum test. a very general formulation is to assume that: all the observations from both groups are independent of each other, the responses are at least ordinal (i.e., one can at least say, of any two observations, which is the greater), under the null hypothesis \(h_0\), the distributions of both populations are identical. the alternative hypothesis \(h_1\) is that the distributions are not identical. the function takes two arrays of different sizes as input. the arrays represent the two samples that are being compared. the function returns two values: the calculated u statistic and the associated p-value. the u statistic is a measure of the difference between the ranks of the two samples. the null hypothesis of the mann-whitney u test is that there is no difference between the distributions of the two samples. if the p-value returned by the ss.mannwhitneyu function is less than a predefined significance level (e.g., 0.05), the null hypothesis can be rejected, and it can be concluded that there is a significant difference between the distributions of the two samples. the mann-whitney u test is commonly used in machine learning to compare the performance of two models, especially when the assumptions of the t-test (such as normality and equal variances) are not met. in details, let \(x_1, \cdots , x_n\) be an independent and identically distributed (i.i.d.) sample from x and \(y_1, \cdots , y_m\) an i.i.d. sample from y and each sample independent from another. the corresponding mann-whitney u statistic is defined as: $$\begin{aligned} u = \sum _{i=1}^n\sum _{j=1}^m s\left( x_i, y_j \right) \end{aligned}$$ (4) with $$\begin{aligned} s(x, y) = {\left\{ \begin{array}{ll} 1 , \text { if } x > y \\ 0.5, \text { if } x = y\\ 0, \text { if } x < y \\ \end{array}\right. } \end{aligned}$$ (5) for large samples, assign numeric rank to all the observations, independent of the group they are. then, u is given by \(min(u_1, u_2)\), where: \(u_1 = n_1 n_2 + \frac{n_1(n_1 + 1)}{2} r_1,\) and \(u_2 = n_1 n_2 + \frac{n_2(n_2 + 1)}{2} r_2\). \(r_i\) is the sum of the ranks from observations of the group i and \(n_i\) is the sample size of the group i. 3 the test bed to evaluate the impact of using the aforementioned statistical tests for model selection over time, a simplified evolutionary algorithm is defined. this algorithm operates on a population of n model types, each having a set of associated hyper-parameters. the model with the highest performance, according to metric m, is designated as best model. during each generation, a random model type is selected and its parameters undergo random mutations. the mutated model then competes with the model of the same type in the population. if a statistical test reveals a significant difference in performance at a significance level of \(\alpha = 0.05\), the hyper-parameters of the model in the population are updated. in such cases, the mutated model also competes against the current best model using the same process. ultimately, the method returns the best model obtained through this evolutionary process. a pseudocode for the method is given in algorithm 1. algorithm 19 . evolutionary model selection full size image 4 experimental setup 4.1 datasets four binary classification problems defined over four datasets from the uci repository [6] were selected for analysis. they are described below: 1. the banking dataset marketing targets [13] (banking) represents customer data in the banking industry for predicting conversion. it features a large sample size (\(n \approx 45000\)) and includes seventeen predictors, consisting of seven numeric and nine categorical variables (three of which are binary). 2. the default of credit card clients dataset [18] (credit) contains information about default payments of credit card clients in taiwan from 2005. with a substantial sample size (\(n \approx 30000\)), it comprises 24 predictors, comprising fifteen numeric and nine categorical variables. 3. the heart disease dataset [10] (heart) integrates information from four databases: cleveland, hungary, switzerland, and long beach v. it encompasses 76 attributes, but published experiments focus on a subset of 14 attributes. the “target” field denotes the presence of heart disease in patients. the dataset has a sample size of \(n = 303\) and includes thirteen attributes, of which five are numeric and eight are categorical. 4. the spambase data set [9] (spambase) consists of a sample of \(n = 4601\) emails classified as either “spam” or “non-spam.” it contains fifty-seven numeric predictors, including metrics such as capital character frequency and the percentage of matching words. the datasets are preprocessed by converting categorical variables to numerical values using labelencoder and normalizing the feature values using preprocessing normalize from scikit-learn. the preprocessed data is then split into training and testing sets using train_test_split from scikit-learn. 4.2 models and hyper-parameters the following models were chosen for the study: random forest classifier k-nearest neighbors classifier decision tree classifier xgboost classifier the sampling functions used in the hyper-parameter mutation phase are denoted by: \(\mathcal {s}_{[\mu _1,\mu _2,\min ]}\): samples from a skellan distribution defined with \(\mu _1\) and \(\mu _2\) and the center on the current value of the hyper-parameter. the samples truncated to \(\min \) if its value is below \(\min \) \(\mathcal {n}_{[\sigma ,\min ,\max ]}\): samples from a normal distribution defined with \(\mu \) on the current value of the hyper-parameter and \(\sigma \). the sample is truncated to \(\min \) if its value is below \(\min \) or \(\max \) if its value is above max \(\mathcal {u}_{[v_1,v_2,...,v_k]}\): samples the values \(v_1,v_2,...,v_k\) using a uniform distribution. the hyper-parameters varied in each model and the related sampling functions are given below: random forest classifier: number of estimators: \(\mathcal {s}_{[10,10,3]}\) maximum depth: \(\mathcal {s}_{[2,2,2]}\) maximum number of features: \(\mathcal {u}_{['sqrt', 'log2', none]}\) k-nearest neighbors classifier: number of neighbors: \(\mathcal {s}_{[1,1,3]}\) weights: \(\mathcal {u}_{['uniform', 'distance']}\) decision tree classifier: maximum depth: \(\mathcal {s}_{[1,1,2]}\) maximum number of features: \(\mathcal {u}_{['sqrt', 'log2', none]}\) criterion: \(\mathcal {u}_{['gini', 'entropy', 'log\_loss']}\) xgboost classifier: tree method: \(\mathcal {u}_{['auto', 'exact', 'approx']}\) maximum depth: \(\mathcal {s}_{[1,1,1]}\) booster: \(\mathcal {u}_{['gbtree', 'dart']}\) number of estimators: \(\mathcal {s}_{[1,1,1]}\) subsample: \(\mathcal {n}_{[0.3,10^-2,1]}\) if no sampling function is the defined, the mutation selects a random element from the set of possible values. otherwise, it ramdomly samples from the defined distribution. the f1 score [15] is used as the evaluation metric for the classification models. the f1 metric is a particular case of a general measure \(f_{\beta }\). this measure is a weighting between precision and recall, indicated in cases of unbalanced data, which can generate more inaccurate results when only one of these two measures is used. 4.3 statistical tests the statistical tests used by algorithm 1 to compare the performance of different hyper-parameters are: 1. paired t-test for two related samples (ttest_rel) 2. two-sample t-test (ttest_in) 3. mann-whitney u test (mannwhitneyu) they are performed on the set of evaluation metrics collected using k-fold cross-validation. as baseline, algorithm 1 was also without any statistical test. in this case is simply selects the models with the best average performance as given by the cross-validation procedure. this procedure will be called dummy_stats_test for rest of the text. 4.4 experimental procedure algorithm 1 was executed 10 independent times for each combination of the following factor levels: datasets: [banking,credit,heart,spambase] statistical tests: [paired t-test for two related samples (ttest_rel), two-sample t-test (ttest_in), mann-whitney u test (mannwhitneyu), dummy (dummy_stats_test)] k folds: [10, 30, 50] number of generations: [25, 100, 1000] 5 results this section summarizes the results obtained using algorithm 1 for all tested conditions related to the research questions in sect. 1. the results are presented as the percentage improvement in performance between the final best model (\(f1_{final}\)) and the initial best model (\(f1_0\)) which was computed by the equation below: $$\begin{aligned} \text {\%improvement} = \frac{f1_{final} f1_0}{f1_0} \end{aligned}$$ (6) 5.1 performance over datasets figure 1 shows the box-plots for the \(\%improvement\) obtained with algorithm 1 combining all the possible configurations. it can be seen that the proposed procedure was more successful for the spambase dataset with an average improvement around 7% reaching 17.5% improvement in some scenarios. conversely, the improvements obtained for the other datasets were more modest with averages around 3%, 1.7% and 2.6% for the banking, credit and heart datasets, respectively. fig. 1. % of improvement by dataset full size image 5.2 performance of the statistical tests varying number of folds figure 2 shows the box-plots for the \(\%improvement\) obtained with algorithm 1 when we vary the applied statistical test and the number of folds used in the cross-validations procedure. it can be seen that, in this scenario where the dataset effect is confounded increasing the number of folds had no effect in the \(\%improvement\). fig. 2. % of improvement by statistical test varying the number of folds full size image figure 3 presents the factors from fig. 2 for each dataset, separately. looking at the spambase results, the dataset where algorithm 1 performed the set, it can be seen that, even though there is still no significant difference in the average \(\%improvement\), increasing the number of folds contributed to reduce the variance of the results for the parametric tests. this phenomenon, however, could not be observed in the overall results, possibly because of the difficulty algorithm 1 had to generate improved models. fig. 3. % of improvement by statistical test varying the number of folds for each dataset full size image 5.3 performance of the statistical tests varying the maximum number of generations figure 4 displays the performance algorithm 1 in regards to the \(\%improvemnt\) when varying the statistical test and the maximum number of generations (iterations) allowed. as expected, when the number of generations increases the \(\%improvement\) also increases. these overall results do not show an interaction between the statistical test and the number of iterations since the difference in the performance among the different statistical tests and the dummy test remains almost constant as the number o generations increases. fig. 4. % of improvement by statistical test and maximum number of generations full size image figure 5 displays the performance algorithm 1 in regards to the \(\%improvemnt\) for each dataset. for the datasets heart, banking and credit, even for a small number of iterations, or 25 or 100, the use of no statistical test (dummy) has led to better average performances of algorithm 1. among the versions with an actual statistical test, the paired t-test with relates samples (ttest_rel) has led to better or almost equal results on average. for the spambase where is was easy to generate improved models, the difference between the dummy and the other versions, only shows at the limit of 1000 generations. overall, these results suggest that in a challenging scenario where generating improved models is difficult, the use of statistical tests may hinder progress by imposing a an excessively stringent standard for accepting new models. fig. 5. % of improvement by statistical test and maximum number of generations for each dataset full size image 6 conclusion this study investigates the effectiveness of statistical tests commonly used for selecting the best machine learning (ml) model when applied over larger periods of time. the study aims to understand whether a selection procedure based on statistical tests leads to higher quality models after a significant number of iterations. additionally, the impact of the number of tests (number of folds in cross-validation) performed to acquire data for these statistical tests and the preferred statistical test for different experimental time horizons are examined. in summary, the analysis suggests that the use of statistical tests in ml model selection should be carefully considered, especially in challenging scenarios, as they may hinder progress and impose overly stringent criteria for accepting new models. although the number of datasets in this study is limited, and the proposed procedure did not verify the assumptions of the applied tests beforehand, it is noteworthy that the dominance of the versions without statistical tests remained consistent. this highlights the importance of conducting further research and exploration in this area. the use of statistical tests of hypothesis is fundamental in many scientific fields, and it is crucial to better understand their impact on the selection of machine learning models. references aygun, b., gunay, e.k.: comparison of statistical and machine learning algorithms for forecasting daily bitcoin returns. avrupa bilim ve teknoloji dergisi (21), pp. 444–454 (2021) google scholar  bao, d., et al.: discriminating between p16-negative oropharyngeal and non-oropharyngeal origins by their metastatic lymph nodes using machine learning approach based on mri radiomics (2022) google scholar  benavoli, a., corani, g., demšar, j., zaffalon, m.: time for a change: a tutorial for comparing multiple classifiers through bayesian analysis. j. mach. learn. res. 18(77), 1–36 (2017). http://jmlr.org/papers/v18/16-305.html bender, a., schneider, n., segler, m., patrick walters, w., engkvist, o., rodrigues, t.: evaluation guidelines for machine learning tools in the chemical sciences. nat. rev. chem. 6(6), 428–442 (2022) article  google scholar  corani, g., benavoli, a.: a bayesian approach for comparing cross-validated algorithms on multiple data sets. mach. learn. 100(2–3), 285–304 (2015) article  mathscinet  math  google scholar  dua, d., graff, c.: uci machine learning repository (2017). http://archive.ics.uci.edu/ml fagerland, m.w.: t-tests, non-parametric tests, and large studies-a paradox of statistical practice? bmc med. res. methodol. 12(1), 1–7 (2012) article  google scholar  hair, j.f., jr., sarstedt, m.: data, measurement, and causal inferences in machine learning: opportunities and challenges for marketing. j. market. theory practice 29(1), 65–77 (2021) article  google scholar  hopkins, m., reeber, e., forman, g., suermondt, j.: spambase. uci machine learning repository (1999). https://doi.org/10.24432/c53g6x article  google scholar  janosi, a., steinbrunn, w., pfisterer, m., detrano, r., m.d., m.: heart disease. uci machine learning repository (1988). https://doi.org/10.24432/c52p4x kim, t.k.: t test as a parametric statistic. korean j. anesthesiol. 68(6), 540–546 (2015) article  mathscinet  google scholar  morettin, p.a., bussab, w.o.: estatística básica. saraiva educação sa (2017) google scholar  moro, s., rita, p., cortez, p.: bank marketing. uci machine learning repository (2012). https://doi.org/10.24432/c5k306 article  google scholar  trawiński, b., smetek, m., telec, z., lasota, t.: nonparametric statistical analysis for multiple comparison of machine learning regression algorithms. int. j. appl. math. comput. sci. 22(4), 867–881 (2012) article  mathscinet  math  google scholar  van rijsbergen, c.j.: information retrieval. (no title) (1979) google scholar  virtanen, p., et al.: scipy 1.0 contributors: scipy 1.0: fundamental algorithms for scientific computing in python. nature methods 17, 261–272 (2020). https://doi.org/10.1038/s41592-019-0686-2 wong, t.t., yeh, p.y.: reliable accuracy estimates from k-fold cross validation. ieee trans. knowl. data eng. 32(8), 1586–1594 (2019) article  google scholar  yeh, i.c.: default of credit card clients. uci mach. learn. repository (2016). https://doi.org/10.24432/c55s3h article  google scholar  download references acknowledgments this work was supported by cnpq national council for scientific and technological development, capes coordination for the improvement of higher education personnel and ufop federal university of ouro preto. author information authors and affiliations graduate program on computer science, universidade federal de ouro preto, ouro preto, brazil marcel chacon gonçalves department of computer science, universidade federal de ouro preto, ouro preto, brazil rodrigo silva authors marcel chacon gonçalvesview author publications search author on:pubmed google scholar rodrigo silvaview author publications search author on:pubmed google scholar corresponding author correspondence to rodrigo silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper gonçalves, m.c., silva, r. (2023). the effect of statistical hypothesis testing on machine learning model selection. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_28 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning model selection hypothesis testing publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature enhancing multiobjective genetic algorithms for pharmaceutical batch scheduling: a study on partitioned selection with constraints and mutation with greedy local search strategy | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper enhancing multiobjective genetic algorithms for pharmaceutical batch scheduling: a study on partitioned selection with constraints and mutation with greedy local search strategy conference paper first online: 30 january 2025 pp 214–229 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) enhancing multiobjective genetic algorithms for pharmaceutical batch scheduling: a study on partitioned selection with constraints and mutation with greedy local search strategy download book pdf download book epub debora t. kohara  orcid: orcid.org/0000-0002-7654-51769, gina m. b. de oliveira  orcid: orcid.org/0000-0003-0384-18799 & luiz g. a. martins  orcid: orcid.org/0000-0003-0168-12939  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 374 accesses abstract product manufacturing scheduling or batch sequencing (bs) in real-world pharmaceutical manufacturing poses a complex multiobjective optimization challenge, intensified by constraints and uncertain demands. these problems are typically non-linear, may leading to low convergence of feasible solutions, and the computationally intensive evaluation functions may hinder the adoption of certain techniques. multiobjective evolutionary algorithms (moeas) have been explored due to their flexible and efficient operators that can be applied to various non-linear multiobjective problems. this study proposes enhancements to moea operators, specifically a selection strategy with constraint-based partition for reinsertion to improve diversity and exploration, and a greedy local search strategy incorporated into the mutation operator to enhance the exploitation of the most promising regions. these enhancements were integrated into two models, ps-re and bl-bat, both of which demonstrated significant improvements in the performance metrics compared to a reference model. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others application of multi-objective optimization in the study of anti-breast cancer candidate drugs article open access 11 november 2022 hybrid multiobjective evolutionary algorithm with differential evolution for process planning and scheduling problem chapter © 2018 solving the multi-objective flexible job-shop scheduling problem with alternative recipes for a chemical production process chapter © 2019 1 introduction since the early 1990s, chemical batch scheduling (bs) has been explored, particularly considering the inherent uncertainties and constraints such as fluctuating demands and limited capacity. several studies have investigated the application of constrained multiobjective optimization (cmo) techniques to scheduling problems [6, 10, 11, 14, 17]. multiobjective evolutionary algorithms (moeas) are frequently employed due to their flexibility in parallel processing and their adaptability to a wide range of problem domains [20]. numerous optimization approaches in pharmaceutical manufacturing, particularly in areas of process planning and inventory optimizations to minimize costs and reduce revenue losses, remain unexplored [3, 10]. this is evidenced in the case of enbrel pharmaceutical, which experienced over 200 million usd in lost revenue [10]. the challenge of real-world constrained non-linear environments has led to the integration of problem-specific heuristics and operators, including local searches, to enhance the exploration and exploitation of feasible solutions, thereby improving solution quality and convergence. despite the high computational cost, monte carlo simulations (mcs) have been employed in the search for models with superior performance with respect to the certain expected probabilistic distributions. in previous work [13], we proposed a constrained multiobjective evolutionary approach, based on nsga-ii, to deal with the constrained and non-linear pharmaceutical batch scheduling problem. we improved the initialization and the mutation operators. the heuristic-based initialization optimizes increases the diversity of the initial population and incorporates heuristics for product sequences to minimize setup times. the enhanced mutation operator consists of four successive operations: changing products, single batch mutation (with addition or removal), mutating the number of genes (with addition or removal), and swapping gene positions. the proposed operators improved significantly the evaluated metrics \(igd+\), e, ns compared to the reference model. this work proposes a selection operator with constraint partition to enhance diversity and exploration of the search space, and an improved mutation operator with a greedy local search strategy for bs to improve the exploitation of the favorable regions. the selection divides the population into two subsets to be evaluated for reinsertion using constrained and unconstrained criteria. for costly fitness calculations (as in the case of mcs), maintaining the best invalid individuals could lead to better valid individuals. the mutation operator incorporates a greedy local search strategy for adjusting batch numbers when new genes are added, it leveraging information from the population about the average batch count for the selected product. it iteratively explores batch increments or decrements based on constraints and division criteria. experiments were carried out in order to compare the enhanced moea in relation of the previous approach (reference model), using different performance metrics, such as hypervolume (hv), inverted generational distance plus (igd+), error rate (e), coverage of two solution sets (cs), and the number of valid solutions (ns). we evaluated two models: ps-re model integrated the partitioned selection with constraints for reinsertion, while the bl-bat model incorporated the greedy local search for the mutation operator. both models demonstrated significant improvements compared to the reference model (ini-heu). the ps-re model improved e by 10.4% and igd+ by 34.2%, while bl-bat achieved even more significant results with a reduction in e by 13.8% and an improvement in igd+ by 37.8%. 2 problem definition multiobjective optimization problems (mops) aim to find optimal solutions \(x_i\) for a vector of j objectives \(f_j\). the complexity arises when comparing solutions, as one solution may outperform another across different objectives. in this regard, the concept of pareto dominance and the pareto optimal set are typically used for ranking [19]. a solution \(x_1\) dominates \(x_2\), or \(x_2\) is dominated by \(x_1\), if it is no worse than \(x_2\) for all j objectives \(f_j(x_1) \ge f_j(x_2)\), and superior in at least one \(j \in j\) (\(f_j(x_1) > f_j(x_2)\)). otherwise \(x_1\) does not dominate \(x_2\). a set of non-dominated solutions, known as pareto optimal \(p^*\), exists if no solution within the search space dominates any in \(p^*\) [4]. moeas aim to identify \(p^*\) or its subset, ensuring close approximation to the true pareto front. fig. 1. decoding of chromossome. full size image a production plan (pp) specifies a sequence and quantity of product batches, considering the costs of product transitions on the production line. utilizing real-world data from [10], the production of two batches each of products a and b spans 91 days. upstream processing (usp) for product a, including inoculation and seed phases, takes 59 days, followed by 7 days for each downstream processing (dsp) batch. a 10-day downtime is allocated for changeovers, with product b’s usp potentially overlapping with a’s production (fig. 1). demand uncertainty impacts inventory metrics, therefore mcs are used to simulate demand and evaluate pp. the fig. 2 illustrates the decoding of a pp, yields the produced quantity \(prod_{(p,m)}\) for product p in month m. demand simulations \(sd_{(p,m)}^{s}\), generated using triangular distribution data, are used for the calculation of backlog \(b(prod_{(p,m)}, sd_{(p,m)}^{s})\) and deficit \(def(prod_{(p,m)}, sd_{(p,m)}^{s})\) metrics. the process is repeated s times to assess the median of the outcomes. monthly availability (a) is determined by adding inventory (i) to production (p). if demand (d) exceeds a, the surplus is sold (s), avoiding backlog (b) and reducing i by d. otherwise, all of a is sold, and unsold units contribute to b. inventory deficit (de) is the shortfall from the inventory target. in this work, we investigate a cmop applied to pharmaceutical bs with product demands simulated via monte carlo. data based on a pharmaceutical manufacturing case extracted from  [10], formalized as follows: $$\begin{aligned} &\text {maximize: } f_1(p,m) = \sum _{p,m}^{p,m} \text {prod}_{(p,m)}\end{aligned}$$ (1) $$\begin{aligned} &\text {minimize: } f_2(p,m,s)=\text {median}\{ \sum _{p,m}^{p,m} \text {def}(\text {prod}_{(p,m)}, sd_{(p,m)}^{s})\}\end{aligned}$$ (2) $$\begin{aligned} &\text {subject to: } e(p,m,s)=\text {median}\{~\sum _{p,m}^{p,m} b(prod_{(p,m)}, sd_{(p,m)}^{s})\}\le 0 \end{aligned}$$ (3) fig. 2. every month, the available (di) is calculated from the stock (e) added to the produced (p). if di is greater than the demand (d), di is served/sold (v) and there are no unmet sales, so there is no backlog (b), and v is reduced from the total of e. otherwise, all e becomes v and unmet sales are accounted for as b. stock deficit (de) is the strategic monthly stock target minus e. full size image the first objective \(f_1(p,m)\) (eq. 1) to be maximized is the sum of the produced quantity \(prod_{(p,m)}\) in kilograms (kg) (illustrated by produced in fig. 2) for product p in month m, with p being the total of four products and m the total of 36 months of planning. this value is calculated by decoding the pp considering the process data and changeover costs [10]. the second objective \(f_2(p,m,s)\) (eq. 2) to be minimized is the median of the accumulated differences between the monthly deficits (illustrated by deficit in fig. 2). with \(s\le s\) with s being the total number of mcs. the constraint (eq. 3) is the median of the s values of total backlog can be named as quantity of unfulfilled orders (illustrated by backlog in fig. 2), which should not exceed 0 kg. the b can accumulate, and if the stock is sufficient, the b is sold (table 1). table 1. process data. source:jankauskas et al. [10].full size table the exploration of the problem space is restrictive because of the problem space size and the fitness evaluation cost. the individual is defined by a sequence of product numbers (up to 4) and a variable number of batches, which is contingent on the minimum and maximum values per product. specifically, for product d, there are, on average, 43 batch number options per product. given an experimental constraint of a maximum of 17 genes per chromosome, the permutation with repetition of the product yields approximately \(2.29 \times 10^{10}\) variations. furthermore, the permutation of the number of batches results in approximately \(1.56 \times 10^{26}\) variations. consequently, the composition of these two permutations yields a total of approximately \(3.56 \times 10^{36}\) distinct individuals. 3 literature review local search strategies are commonly employed in optimization problems, especially for cases with rapid evaluations. we review moea’s applied to cmop scheduling local search strategies for constrained bs and the reference model used in our approach. zhen et al. [18] proposed two distinct selection strategies at different evolutionary stages to balance global and local search efforts. wang et al. [16] explored constrained optimization problems with computationally expensive evaluations, proposing an interior-point method-based local search. at each iteration, a new point is generated using gradient descent until a stopping criterion is met, resulting in the most promising candidate solution. local search is integrated into a surrogate model to reduce evaluation time, demonstrating significant improvements over benchmarks, even in problems with nonlinear inequality constraints. habib et al. [7] developed an evolutionary algorithm for constrained problems with high evaluation costs, employing a local search method to refine individuals that violate constraints. a limited number \( n \) of solutions with the smallest euclidean distances to the reference point are selected, with \( n \) dynamically defined based on the current number of invalid solutions. cai et al. [2] investigated local search strategies in gas with a surrogate-based trust region. the update mechanism is based on a partitioning strategy of the adjacent region to maintain population diversity and evaluate the expected improvement. multiple predictive models within the mechanism confer robustness to the optimization, proving efficient in high-dimensional problems. han et al. [8] developed a multiobjective genetic algorithm (moga), named demo, for task scheduling problems considering setup times with objectives focused on total time and energy consumption. they employed a heuristic for population initialization, self-adaptive evolutionary operators to attain high-quality solutions, and a local search strategy to enhance the algorithm’s exploratory capabilities. the local search operator, based on gene insertion, selects individuals for local search with a certain probability, creating a uniformly distributed number of neighbors. the best neighbor is then chosen based on its distance to a reference point. simulation results indicate that demo outperforms three state-of-the-art algorithms in terms of hypervolume metrics, coverage rate, and distance. azadeh et al. [1] integrated into nsga-ii a population initialization strategy that eliminates non-occurring operations, crossover operators combined with simulated annealing, and a mutation operator with local search for neighbor evaluation. jia et al. [11] developed a history-based evolutionary algorithm using decomposition and local competition applied to scheduling with three objectives: total time, delay penalties, and energy consumption. a competitive local search is proposed (evaluating gene position swaps, gene insertion based on time optimization) that adjusts task positions to expedite convergence. the method includes an elitism-based reinsertion strategy, retaining half of the population and generating the remaining solutions based on historical information and characteristics extracted from the best solutions, ensuring only valid individuals are produced. this work uses the study case presented in [10]. authors developed a moea based on nsga-ii algorithm [5] where each individual is a variable-length vector representing a production plan with the sequence of products and their corresponding batch numbers. the population initiates with 100 individuals, each consisting of a single gene representing one batch of the four products. the \(f_1\) (eq. 3) is calculated through individual decoding, while 1000 demand simulations (\(sd_{(p,m)}^{s}\)) is used to calculate the \(f_2\) and the e. parent selection for crossover utilizes binary tournament selection. the population is categorized into non-dominated fronts, the criteria to define the winner is based on first the backlog constraint, in case of tie uses a criteria similar to nsga-ii with the ranking of the non-dominated front; and if another tie occurs the crowding distance is used [5]. mutation comprises four sequential operations, applicable per gene (product mutation, positive batch increment, negative batch decrement) and/or per chromosome (gene position swap). a new random gene is always added to the chromosome to all generated offspring, resulting in increasing individual sizes from the initial population. in the reinsertion, individuals are sorted and selected based on constraint value, i.e., those with the smallest constraint violations will be chosen. in case of a tie, the ranking of the non-dominated front and crowding distance are applied [5]. in the end of each 1000 generations (defined as a run) feasible non-dominated solutions are stored to an external file. this process is iterated 50 times (runs), resulting in a file containing feasible non-dominated solutions from all 50 runs. the non-dominated front of the external file is calculated again, determining it as the final solution of one execution. in previous work [13], a moea approach, called ini-heu, was proposed as an improvement for the later work [10]. this approach increasing the diversity of the population based on initialization strategies, incorporating an optimized heuristic-based mutation operator, and exploring various crossover types to enhance solution quality. despite the improvements in the metrics evaluated some areas for improvement were identified: (i) high error rate, even using a high number of generations; (ii) low diversity in the final population, where individuals are very close to each other; and (iii) large complex problem space adding complexity to the search of feasible solutions. 4 proposed model in this section, the proposed strategies for enhancing the genetic operators are detailed. two models are investigated: the first, named ps-re, improves upon the reference model ini-heu ([13]) by incorporating the partitioned selection with constraints for reinsertion, and also reduces the number of generations by increasing the crossover rate. the second variation, named bl-bat, extends the ps-re model by integrating a mutation operator coupled with a greedy local search strategy to determine the number of batches for new genes. 4.1 selection partitioned by constraints for reinsertion we propose a selection partitioned by constraints for reinsertion based on distinct evaluation criteria (constrained and non-constrained), aiming to enhance the population diversity. the proposed partitioned reinsertion strategy divides the population set \( p \) of size \( n \) into two subsets \( p_{1} \) and \( p_{2} \), selected based on the original (restricted) and nsga-ii (unrestricted) criteria, respectively. the first subset \( p_{1} \), of size \( n \cdot pre \), is selected by initially prioritizing the constraint; in case of a tie, the nsga-ii criterion is applied evaluating first the pareto front and in case of tie the crowding distance. the second subset \( p_{2} \), of size \( n \cdot (1-pre) \), is chosen exclusively by the nsga-ii criterion. during the reinsertion process, the pareto front of all solutions are calculated, regardless of the constraints. the set \( p_{1} \) is filled by selecting \( n \cdot pre \) solutions according to the ordering of the original criterion. with the remaining solutions, they are ordered using the nsga-ii criterion. that is, for a population with \( n=100 \), the dominance front are calculated disregarding the constraints. considering \( pre \) of 60%, each generation selects 60 individuals following the ordering criterion that prioritizes the smallest constraint results and, in case of a tie, the nsga-ii criteria. the remaining 40 individuals are chosen from the entirety of the current population, based on the nsga-ii criteria. 4.2 mutation with greedy local search an enhancement for the heuristic mutation operator used in [13] is proposed. for non-linear constrained problems, the exploitation of neighbours could improve the quality and speed of convergence. thus, we incorporate a greedy local search strategy for the number of batches if new genes are added. the operator is illustrated in fig. 3 and its pseudocode is visualized in algorithm 1. fig. 3. mutation operator with 4 operations: product mutation with probability pmutp, batch mutation with paddb, number of genes with paddg and position swap with pswapg. the local batch search occurs if a new gene is added. full size image in case a new gene is added, the product is randomly selected, while for the number of batches, a greedy local search approach is adopted. the average number of batches of the current population for the selected product is evaluated, and a percentage of the average batch ( pbm% ) is calculated in relation to this average number for the evaluation of the individual in the case of an increase and decrease in the number of batches. the individual with the original number of batches is then compared with the proposed variation of increment and decrement of batches considering the restriction and division criterion. first, the smallest restriction is evaluated; in case of a tie, the result of the division of the objectives \(\frac{f_{2}(x)*(-1)}{f_{1}(x)}\) is evaluated with the aim of minimization. if the variations in the number of batches do not result in an improvement in relation to the average number, the search is terminated, maintaining the previous number of batches. this procedure is repeated with the same increment or decrement operation selected in the first iteration until the maximum number of iterations (mi) is reached or the established limits for the number of batches are met. 5 experiments and results in this section we explain the experimental setup and the discussion of the results, through the comparative analysis and the hypothesis test analysis. 5.1 experimental setup let p be defined as the non-dominated front obtained from the final population of an algorithm execution. given the stochastic nature of mogas, we conducted 30 executions for each evaluated model. the algorithm was implemented in java and executed on machines with the following configuration: cpu intel core i7; 8gb ram; os windows 10. metrics. in this section, we define the evaluated metrics: hv, nsv, e, igd+, and cs. since the set \(p*\) is not known, it was experimentally approximated from 1424 executions of the moga. the approximated \(p*\) set comprises 32 individuals, indicating the problem’s complexity and the challenge in finding solutions that satisfy the backlog constraint. to ensure comparability in the results analysis, we used the same experiment environment for the demand set. to evaluate the significance of the superiority between the approaches for each metric, a hypothesis test with 5% confidence was used. for distributions that approximate a normal distribution, the t-test [12] was used, otherwise, the mann-whitney u test [15] was used. the hypervolume metric (hv) quantifies the distribution of the non-dominated solution set relative to the total search space. it represents the coverage of a specific solution space by a particular solution set. for a problem with two objectives, the hv represents the area dominated by the set p relative to the total area of the search space. the formula is as follows: $$\begin{aligned} hv = \sum _{i=1}^{p} \sum _{j=1}^{m} \left| f_j(s_i) f_j(w) \right| \end{aligned}$$ (4) where w is the worst solution found in p, and \(f_j(s_i)\) refers to the value of objective j of the non-dominated solution \(s_{i}\). the number of solutions (ns) is a problem-related metric defined as the number of valid non-dominated solutions \(s_i\) found in p that meet the problem constraint (backlog = 0). it provides a measure of the diversity of the set found. however, a high ns does not necessarily imply better solution quality, as a good solution may reduce the ns by dominating multiple inferior solutions. the error rate (e) represents the percentage of solutions in p that belong to a non-dominated or pareto optimal set \(p*\), and is formulated as: $$\begin{aligned} e = \frac{\sum \limits _{i=1}^{|p|} e_{i}}{|p|} \end{aligned}$$ (5) where \(e_{i} = 1\) if the solution \(i \in p\) is dominated by any solution in \(p*\), otherwise \(e_{i} = 0\). it is defined as the percentage of elements of the pareto set \(p*\) that are not contained in p. given two sets of non-dominated solutions a and b, the set coverage metric, or coverage of two sets (cs), represents the percentage of elements in b that are dominated by a. therefore, cs(a,b)=1 indicates that all solutions in b are dominated by a, while cs(a,b)=0 indicates that none of the elements in b is dominated by a [20]. this metric is calculated as follows: $$\begin{aligned} cs(a,b) = \frac{|\{p\in p | \exists p' \in p':p'\ge p\}|}{|p|} \end{aligned}$$ (6) in addition, both directions must be analyzed, as cs(a,b) is not necessarily equal to 1 cs(a,b). the inverted generational distance plus (igd+) metric is an improvement of the inverted generational distance (igd) and generational distance (gd) metrics, which quantify and qualify the non-dominated set achieved by the moga (p) in relation to a reference set. typically, p is compared to the pareto optimal set (\(p*\)). igd+ considers the dominance relations between the solution and the reference point. if a solution is dominated by a reference point, the euclidean distance is used for the distance calculation. in the case where they are not dominated by each other, the minimum distance from the reference point to the region dominated by the solution is used. as a result, the distance represents the amount of inferiority of the solution in relation to the reference point [9]. this metric is calculated as follows: $$\begin{aligned} igd+(a,b) = \frac{1}{|a|}\sum _{s\in a}\min _{r\in b} d(r,s) \quad \text {, }\quad d(r,s) = \sqrt{\sum _{i=1}^m (\max \{s_i-r_i, 0\})^2} \end{aligned}$$ (7) 5.2 comparative analysis this section presents a comparative analysis of the results obtained from the reference model, ini-heu, and the proposed models, ps-re and bl-bat. the ps-re model refines the ini-heu model by integrating the partitioned selection with constraints for reinsertion (subsect. 4.1). the ps-re model uses a partition percentage pre of 60% for the selection for reinsertion and increases the crossover rate from 30% to 90%. for the mutation operator, it uses pmutp of 1%, pmutg of 40%, paddg of 50%, pmutb of 25%, paddb of 25%, and pswapg of 50%. the bl-bat model extends the ps-re model by modifying the mutation operator with a greedy local search strategy for the number of batches. it employs a mi of 2 and pdm of 85. table 2 displays the average (avg) and median (med) of the results. notably, as the ps-re model enhances the quality of the results even at a lower number of generations, the total number of generations was reduced from 1000 to 600. consequently, the final generations for each model will be compared. upon evaluating the metric e, the proposed models ps-re and bl-bat (at generation 600) reduce by 10.4% and 13.8%, respectively, compared to ini-heu (at generation 1000). this reduction suggests that the improved selection of diversity enables a more effective exploration of the search space and the discovery of more solutions from p*. furthermore, the 3.4% improvement when comparing ps-re and bl-bat suggests that the local search facilitates the exploitation of the nearest individuals. the improvement of the proposed models over generations, especially for the initial generations for e and igd+, can be visualized through the boxplot of the executions in fig. 4a. fig. 4. boxplots of performance metrics over moga generations. full size image for the avg igd+, both proposed models ps-re and bl-bat improve by 34.2% and 37.8%, respectively, compared to ini-heu. this improvement suggests that the models are capable of finding solutions closer to p*. similarly to the impact to the hv, there is a 3.6% improvement for the igd+ when comparing ps-re and bl-bat, indicating a better exploitation of the solutions with the incorporation of the local search. upon analysing the ns, the results are similar with a range of 24 to 27 solutions. the ini-heu model still presents the best average, with a reduction of 3.1% and 6.4% for the ps-re and bl-bat, respectively. however, this metric must be evaluated carefully since a large ns does not necessarily imply a better solution. the box plots for igd+ and ns are shown in fig. 4b. evaluating the average hv, the results are similar, varying from 0.89997 to 0.9007. the ps-re model shows a slight improvement of 3.8% in the hv and the bl-bat a small reduction of 6.9% compared to ini-heu. table 2. performance metrics of the moga approaches.full size table to investigate the dominance relationship between the fronts of different approaches, we analyze the cs metric. table 3 presents the results. both proposed models ps-re and bl-bat demonstrate superiority for the dominance. when comparing ps-re with ini-heu, 71.2% of the solutions from ini-heu are dominated by the solutions from ps-re. on the other hand, only 11.9% of the solutions in ps-re are dominated by ini-heu. comparing bl-bat with ini-heu, 60.1% of the solutions from ini-heu are dominated by the solutions from bl-bat, while only 17.1% of the solutions in bl-bat are dominated by ini-heu. this confirms that bl-bat presents a significant improvement in terms of dominance compared to ini-heu. comparing ps-re in relation to bl-bat, 25.8% of the solutions from ps-re are dominated by bl-bat and 36.1% of the solutions from bl-bat are dominated by ps-re. therefore, ps-re presents a slight better dominance compared to bl-bat. this suggests that the ps-re model presents a significant improvement in terms of dominance compared to the ini-heu model. moreover, comparing ps-re to ini-heu for dominance ps-re provides a slight better result. the ps-re model is able to find more dominant solutions, suggesting a more effective exploration of the search space. table 3. cs metrics of the strategiesfull size table 5.3 hypothesis test analysis this section presents the results of a hypothesis test conducted at a 95% confidence level to assess the statistical significance of the ps-re model in comparison to other models. the results, displayed in table 4, show each row representing the average values of the metrics (hv and ns for maximization, and igd+ and e for minimization). each column represents the comparison of ps-re with a different model. the green cells indicate instances where ps-re significantly outperforms the other model for the corresponding metric. the hypothesis test confirms the statistical superiority of ps-re over the ini-heu for hv, igd+, and e metrics; and bl-bat for hv metric. when comparing ps-re with ini-heu, there is significant evidence of superiority for all metrics, except for the ns with comparable results. this suggests that allowing unconstrained solutions enhances the diversity of the population, improving the exploration of solutions towards the p*. despite the higher averages for e and igd+ for the bl-bat model, there is insufficient statistical evidence to assert its superiority over ps-re. moreover ps-re demonstrating evidence of superiority for the hv metric. this suggests that the combination of local search strategies, could be further investigated in combination with surrogate models to reduce the cost of fitness evaluation. table 4. hipothesis test for the metrics comparing ps-re vs other approachesfull size table 6 conclusion in this study, we investigated moga models applied to the batch scheduling of pharmaceutical products with stochastic demands. we proposed two improvements in the genetic operators: the partitioned selection with constraints for the reinsertion and the incorporation of the greedy local search strategy to the mutation operator. the operators were investigated in two models: the first ps-re, which enhances the reference model ini-heu by integrating the partitioned selection with constraints for reinsertion; and the second bl-bat, which incorporates to the ps-re model the greedy local search strategy for the number of batches to the mutation operator. both models demonstrated statistically significant improvements compared to the reference model(ini-heu) in relation to the metrics igd+ and error rate. for the ps-re model, we observed an improvement in the avg of 10.4% in e, improvement of 34.2% for igd+, a slight increase of 3,8% for the hv and a small reduction to ns of 3,1% compared to the reference model. this suggests that the enhanced selection of diversity and the increased crossover rate positively influenced capability of finding solutions in p* and closer. the bl-bat model presented even more expressive results, with a reduction in the average of 13.8% in e, an improvement of 37.8% in igd+, a reduction to ns of 6,4% and a small reduction of 6,9% in hv compared to the reference model. moreover, both demonstrated superiority in the cs metric compared to the ini-heu model, with 71.2% of the solutions from ini-heu being dominated by the solutions from ps-re. only 11.9% of ps-re solutions were dominated by ini-heu solutions. the ps-re also outperforms the bl-bat in dominance, with 36,1% of the solutions from bl-bat dominated by the solutions from ps-re and 25,8% from ps-re solutions dominated by bl-bat solutions. this confirms the significant improvement of the ps-re model in terms of dominance compared to the ini-heu model. as future investigations, we are going evaluated the algorithm performance when processing time is used as a stopping criterion. surrogate models could be used in order to reduce the computational cost of the fitness and constraints evaluation. it also favors the study of more refined local search operators in the mutation. references azadeh, a., goldansaz, s.m., zahedi-anaraki, a.h.: solving and optimizing a bi-objective open shop scheduling problem by a modified genetic algorithm. int. j. adv. manufact. technol. 85, 1603–1613 (2016) google scholar  cai, x., gao, l., li, x.: efficient generalized surrogate-assisted evolutionary algorithm for high-dimensional expensive problems. ieee trans. evol. comput. 24, 365–379 (2020) article  math  google scholar  chen, y., yang, o., sampat, c., bhalode, p., ramachandran, r., ierapetritou, m.: digital twins in pharmaceutical and biopharmaceutical manufacturing: a literature review. processes 8(9), 1088 (2020) article  google scholar  deb, k.: multi-objective genetic algorithms: problem difficulties and construction of test problems. evol. comput. 7, 205–230 (1999) article  math  google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6, 182–197 (2002) article  math  google scholar  fu, y., zhou, m., guo, x., qi, l.: stochastic multi-objective integrated disassembly-reprocessing-reassembly scheduling via fruit fly optimization algorithm. j. cleaner prod. 278 (2021) google scholar  habib, a., singh, h., chugh, t., ray, t., miettinen, k.: a multiple surrogate assisted decomposition based evolutionary algorithm for expensive multi/many-objective optimization. ieee trans. evol. comput. 23, 1000–1014 (2019) google scholar  han, y., li, j., sang, h., liu, y., gao, k., pan, q.: discrete evolutionary multi-objective optimization for energy-efficient blocking flow shop scheduling with setup time. appl. soft comput. j. 93 (2020) google scholar  ishibuchi, h., masuda, h., tanigaki, y., nojima, y.: modified distance calculation in generational distance and inverted generational distance. evol. multi-criterion optim. 110–125 (2015) google scholar  jankauskas, k., farid, s.s.: multi-objective biopharma capacity planning under uncertainty using a flexible genetic algorithm approach. comput. chem. eng. 128, 35–52 (2019) google scholar  jia, z.h., gao, l.y., zhang, x.y.: a new history-guided multi-objective evolutionary algorithm based on decomposition for batching scheduling. expert syst. appl. 141 (2020) google scholar  johnson, p.o.: statistical methods in research. prentice-hall (1949) google scholar  kohara, d.t., de oliveira, g.m.b., martins, l.g.a.: improving a multiobjective evolutionary algorithm applied to batch scheduling in pharmaceutical manufacturing. in: 2023 ieee 35th international conference on tools with artificial intelligence (ictai), pp. 399–403 (2023) google scholar  li, z., qian, b., hu, r., chang, l., yang, j.: an elitist nondominated sorting hybrid algorithm for multi-objective flexible job-shop scheduling problem with sequence-dependent setups. knowl. based syst. 173, 83–112 (2019) google scholar  mann, h.b., whitney, d.r.: on a test of whether one of two random variables is stochastically larger than the other. ann. math. stat. 50–60 (1947) google scholar  wang, y., yin, d.q., yang, s., sun, g.: global and local surrogate-assisted differential evolution for expensive constrained optimization problems with inequality constraints. ieee trans. cybern. 49(5), 1642–1656 (2019) article  math  google scholar  zhang, b., ke pan, q., lei meng, l., lu, c., hui mou, j., qing li, j.: an automatic multi-objective evolutionary algorithm for the hybrid flowshop scheduling problem with consistent sublots. knowl. based syst. 238, 107819 (2022) google scholar  zhen, h., wenyin, g., ling, w.: data-driven evolutionary sampling optimization for expensive problems 32, 318–330 (2021) google scholar  zitzler, e., thiele, l.: an evolutionary algorithm for multiobjective optimization: the strength pareto approach. tik-report 43 (1998) google scholar  zitzler, e., thiele, l.: multiobjective evolutionary algorithms: a comparative case study and the strength pareto approach. ieee trans. evol. comput. 3 (1999) google scholar  download references acknowledgments the authors would like to thank fapemig, capes and cnpq for the financial funding. author information authors and affiliations faculty of computing, federal university of uberlandia, uberlandia, brazil debora t. kohara, gina m. b. de oliveira & luiz g. a. martins authors debora t. koharaview author publications search author on:pubmed google scholar gina m. b. de oliveiraview author publications search author on:pubmed google scholar luiz g. a. martinsview author publications search author on:pubmed google scholar corresponding author correspondence to debora t. kohara . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper kohara, d.t., de oliveira, g.m.b., martins, l.g.a. (2025). enhancing multiobjective genetic algorithms for pharmaceutical batch scheduling: a study on partitioned selection with constraints and mutation with greedy local search strategy. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_15 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multiobjective evolutionary algorithm constrained non-linear optimization genetic algorithms batch scheduling publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature integrating tensor-based data analytics and adaptive prediction for informed decision-making support | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper integrating tensor-based data analytics and adaptive prediction for informed decision-making support conference paper first online: 30 january 2025 pp 335–345 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) integrating tensor-based data analytics and adaptive prediction for informed decision-making support download book pdf download book epub betania campello  orcid: orcid.org/0000-0001-9609-87249 & leonardo tomazeli duarte  orcid: orcid.org/0000-0003-0290-00809  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 374 accesses abstract this work proposes a novel approach to support multi-criteria decision analysis (mcda) using tensor-based data structures and an adaptive prediction method. mcda allows for informed decision-making involving the evaluation of different alternatives based on a set of predefined criteria. unlike previous approaches, this methodology considers the prediction of future criteria signals rather than just consider a single-period value for the criteria. the proposed method generates a tensorial representation of the data and ranks alternatives using a mcda method. experimental results demonstrate that this approach outperforms existing methods, particularly in decision-making situations where future long-term consequences need to be considered. this study contributes to the development of decision support systems by providing a methodological framework that leverages the potential of signal processing and tensor-based data analysis. work supported by são paulo research foundation (fapesp) under the grants #2024/03045-0 and #2020/09838-0 (bi0s brazilian institute of data science). l. t. duarte would like to thank the national council for scientific and technological development (cnpq, brazil) for the financial support. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others selecting a multi-criteria decision analysis method chapter © 2021 a novel multi-criteria decision analysis technique incorporating demanding essential characteristics of existing mcda techniques article 07 march 2023 dynamic mcda approach to multilevel decision support in online environment chapter © 2016 1 introduction signal processing (sp) methods have a wide range of applications in various fields of engineering and natural sciences, such as acoustics [12], and biomedical data analysis [23]. sp methods handle diverse signal characteristics such as non-stationarity and noise, making them suitable for tasks like denoising and adaptive prediction [21]. given their advantages, sp techniques can effectively address decision problems across different domains that require analysis of time-series data. these domains include economics, healthcare, environmental science, and other scientific fields. for instance, [22] showed the potential of kalman filters in decision problems found in economy. in [19], the authors show that independent component analysis (ica) can be used as a pre-processing step in a class of decision problems comprising multiple criteria. moreover, the least-mean-square algorithm was employed in decision making related to the covid-19 pandemic in italy [16]. besides, [18] applied an adaptive kalman filter technique for diabetic recommendation systems. this study aims to investigate the potential of a comprehensive approach that combines signal processing methods with multi-criteria decision analysis (mcda) [13] in a tensorial-based data structure. mcda methods provide a basis for informed decision-making in the process of ranking alternatives according to multiple criteria. the data structure used in mcda methods is usually a decision matrix, where each row corresponds to an alternative, and each column corresponds to a criterion. the entries in the matrix represent the evaluation of each alternative according to each criterion. an example of the application of mcda methods is to rank government projects based on criteria such as environmental impact, economic feasibility, and social benefit. the entries (evaluations) in the decision matrix usually considers static information comprising evaluations acquired in the instant decision is taken. this approach is referred to as current or single period analysis. although the current period analysis may be suitable for certain types of decision-making, many real-world decisions require analyzing the evolution of the criteria over time (their time-series or signals). in the example of ranking government projects, the current analysis corresponds to considering the values of environmental impact, economic feasibility, and social benefit at the moment the project is implemented. however, the long-term implications of these criteria values in the years following project implementation can provide valuable information for the final decision [10]. few studies in mcda have been considering time-series-based approaches, as showed in [4]. examples of time-based approaches in mcda include the studies by [10, 17], which proposed a methodology for dealing with future time series in sustainable contexts, and [26], which propose a long-term planning horizons in a case study from the german energy sector. other recent studies deal with historical time series analysis in mcda, such as [4, 24, 25], and [7]. fig. 1. tensorial proposal in which predictive signals are obtained using an adaptive prediction method. full size image the works mentioned above highlight a gap in the field of mcda when it comes to handling time series data for criteria. to address this gap, reference [5] proposed an extension of the classical decision matrix to a tensor (often employed in signal processing when dealing with multiple signals [3, 15]). the tensor in [5] has three dimensions, with the first dimension representing the alternatives, the second dimension representing the criteria, and the third dimension representing the historical time series of the criteria. for each time series, an adaptive prediction method was applied to predict the values of the criteria for specific future periods. subsequently, a decision matrix was constructed using these predicted values. finally, a mcda method, called preference ranking organization method for enrichment evaluation ii (promethee ii) [1], was applied to obtain a predicted ranking for a specific future period. although the approach in [5] is useful in certain contexts, it may not provide an effective decision-making when the decision-makers must consider the long-term consequences of their choices. we address this case in the present work by obtaining future prediction signals for each criterion, instead of a single prediction value as done in [5]. the proposed methodology is illustrated in fig. 1. once the data are structured as a tensor, an adaptive prediction method is applied to each criterion to obtain q (\(q > 1\)) predicted values and form a q-dimensional prediction signal for each criterion. we then apply the promethee ii method to the predicted signals dimension, then to the criteria dimension, to obtain the ranking of the alternatives. moreover, we investigate how the prediction error influences the final ranking. the paper is organized as follows. section 2 provides an overview of the tensor notation adapted for the mcda context. section 3 presents a brief background on the mcda problem and the promethee ii method. the proposed algorithm is described in sect. 4. the results and discussion are presented in sect. 5. finally, sect. 6 concludes the study. 2 tensorial structure for support multi-criteria decision analysis tensors are widely used in sp [8], and can be defined as functions of three or more indices. we consider in this study a third-order tensor denoted by \(\mathcal {h} \in \mathbb {r}^{m \times n \times t}\). the first dimension of this tensor is related to the alternatives, the second to the criteria, and the third to the time-series. there are three types of slices in this tensor, represented as frontal \({\textbf {h}}(:,:,t)\), horizontal \({\textbf {h}}(:,j,:)\) and vertical \({\textbf {h}}(i,:,:)\). also, there are three types of fibers, \({\textbf {h}}(i, j,:)\), \({\textbf {h}}(i, :, t)\) and \({\textbf {h}}(:, j, t)\). the fiber \({\textbf {h}}(i, j,:)\) is illustrated in fig. 1, representing the time-series related to the criteria. finally, each element of this tensor is denoted by \(h_{ijt}\), and corresponds to the evaluation of alternative i in criterion j at instant t. it is worth mentioning that the employ of tensor structures in mcda is rather recent. the decision tensor was introduced as an alternative to the traditional decision matrix in [5, 6]. additionally, an unbalanced three-dimensional decision tensor was proposed in [20], in such a way as to allow the third dimension of the tensor to have varying sizes. 3 multi-criteria decision analysis background mcda methods have become increasingly popular for ranking alternatives based on multiple criteria. the data structure in mcda is the decision matrix denoted by \({\textbf {h}} \in \mathbb {r}^{m \times n}\). here, the rows of \({\textbf {h}}\) correspond to a set of m alternatives denoted as \(a = \{a_1, a_2, \cdots , a_m\}\), while the columns represent n criteria denoted by \(c = \{c_1, c_2, \cdots , c_n\}\). each element in the matrix, represented by \(h_{ij}\), represents the evaluation of alternative i with respect to criterion j. some mcda techniques utilize the concept of matrix aggregation, which involves transforming the decision matrix \({\textbf {h}} \in \mathbb {r}^{m \times n}\) into a scoring vector \({\textbf {f}} \in \mathbb {r}^{m}\). this process maps each row i of matrix \({\textbf {h}}\) (which corresponds to alternative i) to a score \(f_i\), which is then used to rank the alternatives. there are numerous aggregation methods available [27], including both linear and nonlinear approaches [11]. in addition, the method parameters can be trained in a supervised, semi-supervised, or unsupervised [9] way. a well-established mcda aggregation technique is the promethee family of methods [2]. this approach is based on pairwise comparisons, which involves comparing the evaluations of two alternatives on a particular criterion for all pairs of alternatives in all criteria [1]. the method also uses a weight vector \(\gamma = \{\gamma _1, \gamma _2, \cdots , \gamma _n\}\) which model relative importance of each criterion; assuming that \(\gamma _n \ge 0\) and \(\sum _{j=1}^{n} \gamma _j = 1\). promethee ii [1] is a specific variant of the promethee method that takes the decision matrix h as input and outputs a score vector \({\textbf {f}} \in \mathbb {r}^m\). the initial step in promethee ii is to compute the difference between the performance of each pair of alternatives \(a_i\) and \(a_k\) in each criterion j using the following equation: $$\begin{aligned} d_j(a_i, a_k) = h_{ij} h_{kj} \ \ \forall j = 1,\cdots , n. \end{aligned}$$ (1) by applying eq. (1), we can generate a tensor \(\mathcal {d} \in \mathbb {r}^{m \times m \times n}\). each matrix slice of \(\mathcal {d}\), denoted as \({\textbf {d}}(:,:,j) \in \mathbb {r}^{m \times m}\), represents a pairwise comparison matrix. in the second step, a preference function p is applied to each pairwise comparison matrix \({\textbf {d}}(:,:,j)\) using the following formula: $$\begin{aligned} p_j(d_j(a_i, a_k)) = p_j({\textbf {d}}(:,:,j)) \ \ \forall j = 1,\cdots , n. \end{aligned}$$ (2) where \(0 \le p_j({\textbf {d}}(:,:,j)) \le 1\). there are various types of preference functions available, as described in [1]. for our study, we use the most common preference function known as usual: \(p_j(d_j(a_i, a_k)) = \dfrac{\text{ sgn }(d_j(a_i, a_k))+1}{2}\), where sgn(\( d_j(a_i, a_k)) \) represents the sign function. the next step is to consider a mapping function \(\mathbb {r}^{m \times m \times n} \rightarrow \mathbb {r}^{m \times m}\) that provides a global preference index of \(a_i\) over \(a_k\) \(\forall \) i, k: $$\begin{aligned} \pi (a_i, a_k) = \sum _{j=1}^{n}\gamma _j p_j({\textbf {d}}(:,:,j)). \end{aligned}$$ (3) because \(p_j({\textbf {d}}(:,:,j)) \ge 0\) and \(\gamma _j \ge 0\), then \(\pi (a_i, a_k) \ge 0\). the last step involves mapping from \(\mathbb {r}^{m \times m} \rightarrow \mathbb {r}^{m}\). this is achieved by comparing each \(a_i\) with the remaining \((m-1)\) alternatives, and obtaining its final score, denoted by \(\phi (a_i)\). the score is calculated as the difference between the mean preference of \(a_i\) over the other alternatives and the mean preference of all alternatives over \(a_i\): $$\begin{aligned} \phi (a_i) = \dfrac{1}{m-1}\sum _{a \in a}\pi (a, a_i) \dfrac{1}{m-1}\sum _{a \in a}\pi (a_i, a) \ \ \forall i = 1,\cdots , m. \end{aligned}$$ (4) the vector f is formed by \(\phi (a_i) \in [-1,1]\), and it is sorted in descending order to obtain a vector with the ranking r of the alternatives. 4 the proposed mcda method using tensor-based structure and adaptive prediction the proposed algorithm is composed by three main steps: (1) mcda data tensor structuring; (2) prediction; (3) aggregation. steps (1) and (2) are illustrated in fig. 1. in step (2), we use the recursive least square (rls) [21] due to its fast convergence and its ability to track eventual dynamic behaviors of the data. the required computational complexity of the rls is thoroughly compatible with our problem. next, we provide a detailed description of each step and formalize it in algorithm 1. (1) mcda data tensorial structuration: the input for this step is the data related to the evaluation of the alternatives over the years. these data are structured in a tensor denoted by \(\mathcal {h} \in \mathbb {r}^{m \times n \times t}\), with m alternatives, n criteria, and t samples. this tensor is the output of this step; (2) prediction: for this step, the input is the tensor \(\mathcal {h}\). for each fiber \({\textbf {h}}(i, j,:)\), the rls algorithm is applied with \(\lambda \) samples to be predicted ahead, where \(\lambda \) ranges from 1 to q. as a result, prediction signals are obtained, which are represented by \(\hat{{\textbf {h}}}(i, j,:)\). a prediction tensor \(\mathcal {\hat{h}} \in \mathbb {r}^{m \times n \times q}\) is then constructed using these prediction signals. this tensor is the output of this step; (3) aggregation: for this step, the input is the tensor \(\mathcal {\hat{h}}\). for each slice \(\hat{{\textbf {h}}}(:,j,:)\), where \(j = 1,\ldots , n\), represented in fig. 2, we apply the promethee ii method. as a result, n vectors f are outputted, which compose a matrix \(\hat{{\textbf {f}}} \in \mathbb {r}^{m \times n}\). next, promethe ii is applied in matrix \(\hat{{\textbf {f}}}\), to obtain a final vector f. by sorting f, the ranking r of the alternatives is obtained, which is the output of this step. fig. 2. horizontal slices of the tensor \(\mathcal {\hat{h}}\). full size image 5 numerical experiments this section presents experimental results obtained from synthetic data to demonstrate the advantages of the proposed approach for decision-making, which incorporates ranking based on future prediction signals compared to ranking based on single-period data. the code and data utilized in these experiments are publicly available on the following github repository: https://github.com/bsccampello/tensor-analytics-prediction. in order to assess the dissimilarities between two rankings in our experiments, we employ the kendall tau rank correlation coefficient [14]: \( \tau _{{\textbf {r}}\times \hat{{\textbf {r}}}} = \left\| {\textbf {r}} \hat{ {\textbf {r}}} \right\| _\kappa \). this coefficient measures the distance between two rankings, \( {\textbf {r}}\) and \(\hat{ {\textbf {r}}}\), and is defined as follows: $$\begin{aligned} \tau _{{\textbf {r}}\times \hat{{\textbf {r}}}} = \dfrac{2}{n(n-1)} | \{(p, u): p<u, (\tau _{\textbf {r}}(p) < \tau _{\textbf {r}}(u)\ \wedge \tau _{\hat{{\textbf {r}}}}(p) > \tau _{\hat{{\textbf {r}}}}(u)) \\ \vee (\tau _{\textbf {r}}(p) > \tau _{\textbf {r}}(u)\ \wedge \tau _{\hat{{\textbf {r}}}}(p) < \tau _{\hat{{\textbf {r}}}}(u))\} |, \end{aligned}$$ where \( \tau _{\textbf {r}}(p) \) and \(\tau _{\hat{{\textbf {r}}}}(p)\) means the positions of the p-th element in vectors r and \(\hat{{\textbf {r}}}\). the \(\tau _{{\textbf {r}}\times \hat{{\textbf {r}}}}\) value is bounded on [0, 1]; being zero when the rankings are the same, and one if the rankings are opposite. we performed the simulations by carrying out the two stages described below \(\ell \) = 1,000 times, , and we averaged the results of these \(\ell \) simulations: , assuming a set of \(m=5\) alternatives, \(n=4\) criteria, \(t=25\) samples, and \(q = 5\) samples to be predicted. stage 1 – generate the input data: the data used to compose the tensor \(\mathcal {h} \in \mathbb {r}^{m \times n \times t}\) is generated. given a continuous uniform distribution u and a set \(z = [1,\)..., t], with \(t \in z\), each fiber \({\textbf {h}}(i, j,:)\) of the tensor \(\mathcal {h}\) is obtained as follows: criterion 1, \({\textbf {h}}(i, 1,:)\) \(\rightarrow \) \(h_{i1t} = a_i + b_it \), \(a_i \sim u[2, 20]\), \(b_i \sim u[0.5,0.8]\), \(\forall t, i\); criterion 2, \({\textbf {h}}(i, 2,:)\) \(\rightarrow \)] \(h_{i2t} = a_i + sin(f_it)\), \(a_i \sim u[2, 20]\), \(f_i\sim u[0.7\pi , 0.75\pi ]\), \(\forall t, i\); criterion 3, \({\textbf {h}}(i, 3,:)\) \(\rightarrow \) \(h_{i3t} = a_i + (-1)^t\), \(a_i \sim u[2, 20]\), \(\forall t, i\); criterion 4, \({\textbf {h}}(i, 4,:)\) \(\rightarrow \)] \(h_{i4t} = a_i + t^{0.075}\), \(a_i \sim u[2, 20]\), \(\forall t, i\). since there is a white noise component in actual signals, we add random noise \(\alpha \boldsymbol{\mu }\), \(\boldsymbol{\mu }\sim n(0,1)\), to the signals \({\textbf {h}}(i, j,:)\), being \(\alpha \) calculated as: \( \alpha = 10^{(\frac{10log\sigma _{{\textbf {h}}(i, j,:)}^2 -snr}{20}) }\), where \(\sigma _{{\textbf {h}}(i, j,:)}^2\) is the variance of \( {\textbf {h}}(i, j,:)\), and the snr is the signal-to-noise ratio, which assumes the values from 0 to 50 db. stage 2 – computing the results: to compute the results, let us assume that the current period is at the instant of time t, as shown in fig. 3. the known signal is assumed to be from instant 1 to \(t = 25\), and the years to be predicted are for the next 5 years, which means that the algorithm considers \(q = 5\) samples. considering four rankings to compute the tau distances \(\tau ^c_{{\textbf {r}}^* \times {\textbf {r}}^c}\), \(\hat{\tau }_{{\textbf {r}}^* \times \hat{{\textbf {r}}}}\), and \(\hat{\tau }_{{\textbf {g}}_\lambda \times \hat{{\textbf {r}}}}\), obtained as: fig. 3. signals and the current period. full size image – current ranking (\({\textbf {r}}^c\)): we construct a matrix \({\textbf {h}} \in \mathbb {r}^{m \times n}\) with the data at time instant \(t = 25\) (current values) and apply the promethee ii method. this ranking is commonly used in mcda literature as the classical ranking. – prediction ranking (\(\hat{{\textbf {r}}}\)): this ranking is calculated by applying algorithm 1 using the data generated in stage 1 as input. – reference ranking or benchmark (\({\textbf {r}}^*\)): we construct a tensor \(\mathcal {h}^* \in \mathbb {r}^{m \times n \times q}\) containing the actual data from period \(t+1\) to \(t + 5\), and apply the promethee ii method to both time and criteria dimensions. this ranking serves as the benchmark for the prediction one. indeed, \(\hat{{\textbf {r}}}\) and \({\textbf {r}}^*\) should be the same if the prediction errors do not influence the final ranking. – rankings obtained using the approach proposed in [5] (\({\textbf {g}}_1\), \({\textbf {g}}_2\), \({\textbf {g}}_3\), \({\textbf {g}}_4\), \({\textbf {g}}_5\)): to generate these rankings, we first construct a matrix of predicted values for a single future period, \(t+1\), using the rls technique. we then apply the promethee ii method to this matrix to obtain a ranking, denoted by \({\textbf {g}}_1\), at period \(t+1\). we repeat this process for periods \(t+2\) through \(t+5\), generating a total of 5 rankings, one for each period. at the end of stages 1 and 2, we compute the average of \(\ell \) simulations for each snr value of \(\tau ^c_{{\textbf {r}}^* \times {\textbf {r}}^c}\), \(\hat{\tau }_{{\textbf {r}}^* \times \hat{{\textbf {r}}}}\), and \(\hat{\tau }_{{\textbf {g}}_\lambda \times \hat{{\textbf {r}}}}\), and these results are shown in fig. 4 and fig. 5. fig. 4. \(\tau ^c_{{\textbf {r}}^* \times {\textbf {r}}^c}\), and \(\hat{\tau }_{{\textbf {r}}^* \times \hat{{\textbf {r}}}}\). full size image a preliminary analysis of fig. 4 concerns the comparison between the ranking typically computed in mcda literature, referred to as the current ranking, and the ranking obtained by considering future values of the criteria, known as the benchmark ranking (\(\tau ^c_{{\textbf {r}}^* \times {\textbf {r}}^c}\)). it is evident that these rankings differ with tau values higher than 0.11. this implies that the classical approach used in mcda may not yield appropriate solutions when it is crucial to analyze the future impact of a decision. in real-world decision-making, future criterion values are often unknown, and predictive methods should be employed to estimate them. a second analysis of fig. 4 is therefore necessary to evaluate the influence of prediction error on the ranking. specifically, this analysis compares the benchmark ranking and the predicted ranking through the tau value \(\hat{\tau }_{{\textbf {r}}^* \times \hat{{\textbf {r}}}}\), which quantifies the impact of the prediction error in the ranking outcomes. we may observe that, for snr higher than 15, tau values are less than 0.08, which can be acceptable if future rankings are relevant for the decision-making process. the results in fig. 4 show that the difference between the benchmark and prediction rankings (\(\hat{\tau }_{{\textbf {r}}^* \times \hat{{\textbf {r}}}}\)) is smaller than the difference between the benchmark and current rankings (\(\tau ^c_{{\textbf {r}}^* \times {\textbf {r}}^c}\)). this suggests that, for decisions with future consequences, a prediction-based approach may be more suitable than one relying on the current data values, even if there is an error in the prediction and, in consequence, in the ranking. fig. 5. comparison between the approach proposed in [5] and this proposal. full size image to compare the approach proposed in [5] with the one presented in this work, we obtained additional results, as illustrated in fig. 5. we calculated \(\hat{\tau }_{{\textbf {g}}_\lambda \times \hat{{\textbf {r}}}}\) for \(\lambda = 1, \cdots , q\), where \({\textbf {g}}_\lambda \) denotes the ranking obtained by the adaptive approach proposed in [5]. for instance, the metric \(\hat{\tau }_{{\textbf {g}}_1 \times \hat{{\textbf {r}}}}\) quantifies the difference between the rankings \({\textbf {g}}_1\) (obtained using the single-period prediction for \(t+1\)) and \(\hat{{\textbf {r}}}\). notably, the ranking \(\hat{{\textbf {r}}}\) is distinct from all the rankings obtained using the single-period approach. these results suggest that the proposed approach is capable of providing different rankings than those obtained by the single-period approach. therefore, the proposed approach can be a valuable tool in decision-making processes that consider long-term consequences. 6 conclusion the present paper proposes a new decision-making method based on tensor data structure and adaptive prediction, thus extending previous work that provide cross-fertilization between the fields of signal processing and mcda. the proposed approach showed promising results in generating rankings suitable for decision-making that are relevant to the analysis of long-term future consequences. the simulation results revealed that the proposed approach can provide rankings close to the benchmark, indicating that prediction error does not significantly affect ranking when snr is higher than 15 db. furthermore, the results suggest that this approach can be more suitable for making decisions with future consequences, even if there is an error in the ranking, than relying solely on current data values. additionally, the proposed approach is also more suitable than the approach proposed in [5], when future decision should consider long-term analysis. in summary, the proposed approach can be a valuable tool for decision-making in diverse domains, including finance, healthcare, and sustainable development. further research can be conducted to evaluate the applicability of the proposed approach in real-life decision-making scenarios, and to investigate the influence of different prediction methods on its performance. references brans, j.p., mareschal, b.: promethee methods. in: multiple criteria decision analysis: state of the art surveys, pp. 163–186. springer (2005) google scholar  brans, j.p., vincke, p.: a preference ranking organisation method: the promethee method for multiple criteria decision-making. manage. sci. 31(6), 647–656 (1985) article  mathscinet  math  google scholar  calvi, g.g., lucic, v., mandic, d.p.: support tensor machine for financial forecasting. in: proceedings of ieee international conference on acoustics speech signal process (icassp), pp. 8152–8156. ieee (2019) google scholar  campello, b.s.c., duarte, l.t., romano, j.m.t.: exploiting temporal features in multicriteria decision analysis by means of a tensorial formulation of the topsis method. comput. ind. eng. 175, 108915 (2023) article  math  google scholar  campello, b.s.c., duarte, l.t., romano, j.m.: adaptive prediction of financial time-series for decision-making using a tensorial aggregation approach. in: ieee international conference on acoustics, speech and signal processing (icassp), pp. 5435–5439 (2020) google scholar  campello, b.s.c., duarte, l.t., romano, j.m.t.: dealing with multi-criteria decision analysis in time-evolving approach using a probabilistic prediction method. eng. appl. artif. intell. 116, 105462 (2022) article  math  google scholar  campello, b.s.c., duarte, l.t., romano, j.m.t.: multicriteria decision support employing adaptive prediction in a tensor-based feature representation. pattern recogn. lett. 174, 52–56 (2023) article  math  google scholar  cichocki, a., et al.: tensor decompositions for signal processing applications: from two-way to multiway component analysis. ieee signal process. mag. 32(2), 145–163 (2015) article  mathscinet  math  google scholar  duarte, l.t.: a novel multicriteria decision aiding method based on unsupervised aggregation via the choquet integral. ieee trans. eng. manage. 65(2), 293–302 (2017) article  mathscinet  math  google scholar  frini, a., benamor, s.: mupom: a multi-criteria multi-period outranking method for decision-making in sustainable development context. environ. impact assess. rev. 76, 10–25 (2019) article  math  google scholar  greco, s., figueira, j., ehrgott, m.: multiple criteria decision analysis, vol. 37. springer (2016) google scholar  halimeh, m.m., huemmer, c., kellermann, w.: a neural network-based nonlinear acoustic echo canceller. ieee signal process. lett. 26(12), 1827–1831 (2019) article  math  google scholar  keeney, r.l., raiffa, h.: decisions with multiple objectives: preferences and value trade-offs. cambridge university press (1993) google scholar  kendall, m.g.: a new measure of rank correlation. biometrika 30(1/2), 81–93 (1938) article  math  google scholar  kroonenberg, p.m.: applied multiway data analysis, vol. 702. john wiley & sons (2008) google scholar  marano, s., sayed, a.h.: decision-making algorithms for learning and adaptation with application to covid-19 data. signal process. 194, 108426 (2022) article  math  google scholar  mouhib, y., frini, a.: tsmaa-tri: a temporal multi-criteria sorting approach under uncertainty. j. multi-criteria decis. analy. 28(3–4), 185–199 (2021) article  math  google scholar  nagaraj, p., deepalakshmi, p., ijaz, m.f.: optimized adaptive tree seed filter for a diabetes recommendation system—bilevel performance improvement strategy for healthcare applications. in: cognitive and soft computing techniques for the analysis of healthcare data, pp. 191–202. elsevier (2022) google scholar  pelegrina, g.d., duarte, l.t., romano, j.m.t.: application of independent component analysis and topsis to deal with dependent criteria in multicriteria decision problems. expert syst. appl. 122, 262–280 (2019) article  math  google scholar  pelissari, r., abackerli, a.j., duarte, l.t.: choquet capacity identification for multiple criteria sorting problems: a novel proposal based on stochastic acceptability multicriteria analysis. appl. soft comput. 120, 108727 (2022) article  math  google scholar  romano, j.m.t., attux, r., cavalcante, c.c., suyama, r.: unsupervised signal processing: channel equalization and source separation. crc press (2011) google scholar  venegas, f., de alba, e., ordorica, m.: an economist’s guide to the kalman filter. estudios económicos pp. 123–145 (1995) google scholar  wan, c., chen, d., yang, j.: pulse rate estimation from forehead photoplethysmograph signal using rls adaptive filtering with dynamical reference signal. biomed. signal process. control 71, 103189 (2022) article  google scholar  wątróbski, j.: temporal promethee ii–new multi-criteria approach to sustainable management of alternative fuels consumption. j. clean. prod. 413, 137445 (2023) article  math  google scholar  watrobski, j., salabun, w., ladorucki, g.: the temporal supplier evaluation model based on multicriteria decision analysis methods. in: asian conference on intelligent information and database systems, pp. 432–442. springer (2017) google scholar  witt, t., klumpp, m.: multi-period multi-criteria decision making under uncertainty: a renewable energy transition case from germany. sustainability 13(11), 6300 (2021) article  math  google scholar  zopounidis, c., doumpos, m.: multicriteria classification and sorting methods: a literature review. eur. j. oper. res. 138(2), 229–246 (2002) article  math  google scholar  download references author information authors and affiliations school of applied sciences at the university of campinas, limeira, brazil betania campello & leonardo tomazeli duarte authors betania campelloview author publications search author on:pubmed google scholar leonardo tomazeli duarteview author publications search author on:pubmed google scholar corresponding author correspondence to betania campello . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper campello, b., duarte, l.t. (2025). integrating tensor-based data analytics and adaptive prediction for informed decision-making support. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_23 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords adaptive prediction temporal analysis multi-criteria publish with us policies and ethics profiles leonardo tomazeli duarte view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature assessment of robust multi-objective evolutionary algorithms on robust and noisy environments | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper assessment of robust multi-objective evolutionary algorithms on robust and noisy environments conference paper first online: 12 october 2023 pp 33–48 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) assessment of robust multi-objective evolutionary algorithms on robust and noisy environments download book pdf download book epub mateus clemente de sousa9,12, ivan reinaldo meneghini10,12 & frederico gadelha guimarães11,12  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 528 accesses 2 citations abstract robust optimization considers uncertainty in the decision variables while noisy optimization concerns with uncertainty in the evaluation of objective and constraint functions. although many evolutionary algorithms have been proposed to deal with robust or noisy optimization problems, the research question approached here is whether these methods can deal with both types of uncertainties at the same time. in order to answer this question, we extend a test function generator available in the literature for multi-objective optimization to incorporate uncertainties in the decision variables and in the objective functions. it allows the creation of scalable and customizable problems for any number of objectives. three evolutionary algorithms specifically designed for robust or noisy optimization were selected: rnsga-ii and rmoea/d, which utilize monte carlo sampling, and the c-rmoea/d, which is a coevolutionary moea/d that uses a deterministic robustness measure. we did experiments with these algorithms on multi-objective problems with (i) uncertainty in the decision variables, (ii) noise in the output, and (iii) with both robust and noisy problems. the results show that these algorithms are not able to deal with simultaneous uncertainties (noise and perturbation). therefore, there is a need for designing algorithms to deal with simultaneously robust and noisy environments. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a clustering-based coevolutionary multi-objective evolutionary algorithm for handling robust and noisy optimization article 24 june 2024 a novel robust multi-objective evolutionary optimization algorithm based on surviving rate article open access 28 february 2025 variation rate: an alternative to maintain diversity in decision space for multi-objective evolutionary algorithms chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational methods for stochastic equations continuous optimization optimization system robustness calculus of variations and optimization probabilistic methods, simulation and stochastic differential equations 1 introduction uncertainties exist in the real world, such as in the measurement system or final control element. the search for optimal solutions in the presence of uncertainties is often referred to as robust optimization in the literature [3, 4, 9, 11]. however, the term robust optimization can be used to encompass uncertainties in both the decision variables and the parameters of the problem, as discussed by ben-tal, el ghaoui, and nemirovski in their book on robust optimization [2]. this includes considering uncertainty in the input of the mathematical model of the optimization problem. in this article, our focus is specifically on uncertainties in the decision variables, referred to as perturbations within the context of robust optimization. on the other hand, uncertainties in the objective functions, which pertain to the output of the process, are referred to as noise within the context of noisy optimization. figure 1 illustrates these definitions and the distinctions between noisy and robust optimization. fig. 1. robust optimization (perturbation in the decision variables/parameters) versus noisy optimization (noise in the function output). full size image real optimization problems can have different types of uncertainties. these uncertainties can prevent the implementation of practical solutions if not taken into account. the designer (optimizer) therefore faces the challenge of finding solutions that are less sensitive to uncertainties. optimization with uncertainty is a relatively new and rapidly growing research field that has gained significant attention in the past decade [10]. some works with applications using optimization under uncertainties are [1, 7, 12, 17, 20, 25]. examples of uncertainties include noise, model inaccuracies, time variations, measurement inaccuracies, disturbances, and other uncontrolled factors that can degrade the performance of the designed solutions [19]. recently, shaaban sahmoud and haluk rahm [22] worked on noisy and dynamic optimization. we aim to fill a gap in the literature by conducting a study that tests evolutionary algorithms in the presence of both types of uncertainties; in decision variables and objective functions. in order to answer this research question, we extend a test function generator available in the literature for multi-objective optimization. the goal is to provide an examination of the behavior of these algorithms under such conditions. the generator of benchmark problems from [18] uses a bottom-up approach and generates problems with various features, the objective space, and the decision space separately. it allows the creation of scalable and customizable problems for any number of objectives and pareto fronts with different shapes and topologies, and can incorporate different features such as dissimilarity, robustness, and modality. the objective space can always be represented as a vector \(({\textbf {x}}_p, {\textbf {x}}_d)\), where \({\textbf {x}}_p\) responsible for the spatial location of the points in the objective space and \({\textbf {x}}_d\) governs the convergence. the extension introduces different types of noise (gaussian, uniform, and cauchy) for the objective functions. these noises have different properties, such as being additive or multiplicative, and can have different intensities. the noise intensities were taken from the work of [8]. proposed intensities were implemented, along with a third intensity we specifically proposed for the gaussian and uniform noises. then, three evolutionary algorithms designed for robust or noisy optimization are selected: rnsga-ii, rmoea/d, and c-rmoea/d. the results of the tests show that the algorithms were not able to handle simultaneous uncertainties in decision variables and objective functions, leading to a degradation in the quality of the solutions compared to only one type of uncertainty (either perturbation or noise). thus, we conclude that there is a need for designing algorithms specifically for handling simultaneous uncertainties. in this work, the preliminary definitions and concepts such as uncertainties and robustness measures are presented in sect. 2. in sect. 3 presents the extension of the function generator proposed in this work. the results of the computational experiment, including a brief description of the three algorithms used and the experimental setup, are presented in sect. 4. finally, the conclusion of the work is presented in sect. 5. 2 preliminary concepts some of the main concepts covered in this section include types of uncertainties and robustness measures. these concepts provide the foundation for the later discussions in the paper about robust and noisy multi-objective optimization. 2.1 types of uncertainties there are several classifications of uncertainties in the literature. three prevalent classifications of uncertainties are presented in table 1. the parameter \(\delta \) represents uncertainties in the decision variables (perturbation), \(f({\textbf {x}}+\delta )\), or in the objective functions (noise), \(f({\textbf {x}})+\delta \). table 1. classifications of types of uncertainties, according to [4, 13, 21].full size table according to [4], uncertainties are categorized based on their sources, with some sources of different types of uncertainties being identified in the design and optimization process. ong et al. [21] consider which elements of the model are affected, such as the objective function, variables, or environment. finally, jin and branke [13] classify uncertainties in optimization problems according to how they impact the optimization process. we observed that there is an equivalence between the classifications of uncertainties mentioned in the literature. this equivalence is depicted in table 2. to illustrate this equivalence, consider type b, which classifies the source of uncertainty as production tolerance. for example, a final control element has uncertainty (a hypothetical example would be a drill that aims to drill 5 m, but may have a \({\pm }10\%\) error). this error in the final control element generates a perturbation in the decision variables, thus making it both type ii and category ii. table 2. equivalence between classifications of uncertainties, according to [4, 13, 21].full size table 2.2 robustness measurements mathematically, there are different ways to quantify the uncertainties classified above. one paper that defined robustness measures was by [4]. according to this work, uncertainties can basically be modeled deterministically, probabilistically, or possibilistically: 1. the deterministic type defines domains of uncertainty parameters. an example is the worst-case measure: $$\begin{aligned} r({\textbf {x}},\delta ) = \max _{\delta \in \vardelta }f({\textbf {x}},\delta ) \end{aligned}$$ (1) 2. the probabilistic type defines probability measures that describe the probability by which a given event can occur. an example is the monte carlo integration followed by the mean: $$\begin{aligned} \hat{r}({\textbf {x}},\delta ) = \frac{1}{k} \sum _{i=1}^{k} [f({\textbf {x}}+\delta ^i)] \end{aligned}$$ (2) 3. the possibilistic type defines fuzzy rules that describe the degree of adherence by which a given event may be acceptable. an example is the treatment of uncertainty sets using fuzzy logic and the theory of evidence [14]. 3 function generator extension robust optimization and/or noisy optimization a contribution of this work is the proposal of an extension to the function generator described in [18]. the main aspect of this extension is the integration of concepts from robust and noisy optimization. the gaussian, uniform, and cauchy noise models present in the bbob functions [8] are incorporated into the function generator, resulting in an optimization problem formulation with uncertainty in both the decision variables and the objective functions. mathematically: $$\begin{aligned} f({\textbf {x}},\delta _x, \delta _f) = f({\textbf {x}} + \delta _x) + \delta _f \end{aligned}$$ (3) where \(\delta _x\) indicates the uncertainties in the decision variables and \(\delta _f\) indicates the uncertainties in the objective functions. in eq. 3, the noise is considered additive, but it can also be multiplicative: $$\begin{aligned} f({\textbf {x}},\delta _x, \delta _f) = f({\textbf {x}} + \delta _x) \times \delta _f \end{aligned}$$ (4) the main objective of the proposed extension is to evaluate the behavior of algorithms in problems with uncertainty in both the decision variables and the objective functions. to date, the algorithms have only been tested in either robust optimization or noisy optimization. they were designed specifically for one type of uncertainty, so the question is whether they will perform well in the presence of two types of uncertainty simultaneously. the test function used in this study is called gpd and is taken from [18]. this problem allows for changing the format of the pareto front through a parameter p. the parameter p defines the norm (\(p \ge 1\)) or quasi-norm (\(0 < p < 1\)) used in the function \(h({\textbf {x}}) = || t({\textbf {x}}) ||_p\). if \(0 < p < 1\), the pareto front is convex. if \(p=1\), it is a hyperplane, and if \(p > 1\), it is concave. in this article, it was defined as follows: \(p = 0.5\) or \(p = 1\) or \(p = 2\). the focus of this work is to assess the robustness and noise tolerance of the algorithms. to achieve this, unrestricted and bi-objective problems were chosen. the problem gpd is evaluated according to the methodology described in [18]. as previously stated, this work utilizes three models of noise: gaussian, uniform, and cauchy (\(\delta _f\) of eqs. 3 and 4). these noises will be incorporated into the function generator. these noise models were adopted from the work of [8] (bbob functions). finck et al. [8] defined two levels of intensity for the noise: moderate and severe. the mathematical description of the three noise models is presented below. gaussian noise $$\begin{aligned} f_{gn}(f,\beta ) = f \times \exp (\beta \mathcal {n}(0,1)) \end{aligned}$$ (5) where \(\beta \) controls the intensity of the noise and \(\mathcal {n}(0,1)\) is a random variable with a normal distribution with a mean of 0 and a variance of 1. uniform noise $$\begin{aligned} f_{un}(f,\alpha ,\beta ) = f \times \mathcal {u}(0,1)^\beta \max \left( \begin{array}{c} 1,(\frac{10^9}{f + \epsilon })^{\alpha \mathcal {u}(0,1)} \end{array} \right) \end{aligned}$$ (6) where \(\mathcal {u}(0,1)\) represents a random, uniformly distributed number in the interval (0, 1) and \(\alpha \) is a parameter that controls the intensity of the noise in conjunction with the parameter \(\beta \), resulting in two random factors. the first factor is uniformly distributed in the interval [0,1] for \(\beta = 1\). the second factor (\(\max \)) is greater than or equal to 1. finally, the factor \(\epsilon \) is introduced to avoid division by zero, with a value of \(10^{-99}\) being adopted. the uniform noise is considered to be more severe than gaussian noise. cauchy noise $$\begin{aligned} f_{cn}(f,\alpha ,\text {p}) = f + \alpha \left( \begin{array}{c} 1000 + \mathbb {i}_{\{\mathcal {u}(0,1)<\text {p}\}} \frac{\mathcal {n}(0,1)}{|\mathcal {n }(0,1)| + \epsilon } \end{array} \right) \end{aligned}$$ (7) where \(\alpha \) defines the intensity of the noise and p determines the frequency of the noise disturbance. the value of \(\epsilon \) is \(10^{-199}\). this noise model has two important characteristics. first, only a small percentage of the function values are affected by noise. second, the noise distribution is considered to be unusual, with outliers occurring from time to time. the gaussian and uniform noises are multiplicative, so the intensity remains constant regardless of the range of values in the objective function. for example, if the maximum noise value is 5, the effect on the objective function will be to increase its value by 5 times, whether the value ranges from [0,1] or [0,100]. cauchy’s noise differs from the previous two in that it is additive. in this case, the scale of the objective function must be considered. for example, if the range of values for one problem is [0,1] and for another problem is [0,100], and the noise peak has an intensity of 1, the intensity for the first problem will be \(100\%\) since the noise is additive, whereas, for the second problem, it will only be \(1\%\). therefore, in this article, eq. 7 will not include the 1000 summation term. in this article, the intensity values from [8] are kept.footnote 1 4 computational experiment in this section, the computational experiments will be presented. a brief description of the three selected algorithms and the testing setup will be provided. subsequently, the solutions that the algorithms found for problems with perturbation and/or noise will be displayed. finally, a discussion of the results will be made. 4.1 algorithms typically, robustness measures (discussed in sect. 2.2) are employed to account for uncertainties, resulting in a new objective function. as a result, existing algorithms in the literature are transformed into robust versions. for instance, the rmoea/d employs moea/d [27] and the rnsga-ii employs nsga-ii [5], with the addition of monte carlo sampling in the uncertainty space. the average of this sampling is then calculated [6]. the rmoea/d selects a set of vectors with the best weighting. c-rmoea/d [19] is a coevolutionary algorithm for robust multi-objective optimization that employs the worst-case minimization methodology and the decomposition/aggregation strategy. the decomposition approach facilitates the implementation of a coevolutionary approach [19]. the algorithm uses a competitive strategy between two populations of candidate solutions, with x being a set of vectors from the decision space and \(\vardelta \) being a set of perturbation vectors. the rnsga-ii and rmoea/d employ a probabilistic robustness measure, whereas c-rmoea/d is a coevolutionary algorithm that uses a deterministic robustness measure. other recent strategies for robust optimization or noisy optimization can be found at [15, 16, 26]. 4.2 experimental setup the function generator extension (sect. 3) will be utilized in the experiments. the pareto front will be tested with convex, linear, and concave formats with simultaneous uncertainties. problems with only perturbations or noise will be demonstrated using the concave pareto front. the primary aim of this study is to evaluate algorithms in the presence of simultaneous uncertainties. to achieve this, the tests will be conducted with bi-objective and unconstrained scenarios. as mentioned earlier, the function generator is adaptable to these parameters. tests with perturbations only (robust optimization) are conducted as shown in [18]. the following parameters were used for all tests: a population size of 210; a total of 24 decision variables; the stopping criterion is the maximum number of evaluations of the objective function, equal to 150,000. the rnsga-ii and rmoea/d algorithms used 100 samples for averaging. the c-rmoea/d used the pbifootnote 2 (penalty boundary intersection) aggregation function and a subpopulation size of 15. the rmoea/d also utilized a subpopulation size of 15. 4.3 results the results will be presented in three parts: (i) robust optimization (with only perturbation); (ii) noisy optimization (with only noise); (iii) robust and noisy optimization (with both perturbation and noise simultaneous uncertainties). robust optimization. figure 2 shows the results for \(\delta _x = 0.1\), which is considered a high perturbation as it represents an uncertainty of \(\pm 10\%\). the rnsga-ii and rmoea/d (using a mean-based robustness measure) mapped the robust front. the algorithm based on a worst-case measure, c-rmoea/d, also mapped the robust front. the results with convex and linear fronts are not displayed due to page limitations. the results are similar to those obtained with a concave front. fig. 2. result with \(\delta _x = 0.1\). full size image noisy optimization. the c-rmoea/d does not incorporate coevolution with noise only. hence, it becomes the moea/d algorithm, without a second evolutionary cycle. the results were compared between moea/d and rmoea/d and between nsga-ii and rnsga-ii. this comparison was made between the original algorithm without any uncertainty handling, and the algorithm that uses sampling followed by the mean metric. figure 3 presents the solutions obtained from rnsga-ii and nsga-ii. the results were similar for gaussian and uniform noise. the algorithms showed good convergence and dispersion from the global front with moderate intensity. however, they did not produce any mapping for severe intensity. finally, both rnsga-ii and nsga-ii mapped the global front well for cauchy noise (single additive noise) at both intensities. due to page limitations, the results of moea/d and rmoea/d will not be shown. these results were similar to those of nsga-ii and rnsga-ii from the fig. 3. fig. 3. result for function generator extension with noise for nsgaii and rnsgaii. full size image robust and noisy optimization. figure 4 displays the outcome of extending the function generator with gaussian noise and \(\delta _x=0.1\). the three algorithms failed to converge for high-intensity noise. this outcome was predictable because the severe noise greatly impacted the original function. however, the c-rmoea/d solutions exhibited a tendency towards the robust front. for moderate-intensity noise, the algorithms achieved convergence and good dispersion. nonetheless, the rmoea/d had some poor solutions, but, generally, the solutions converged. the same can be said for the c-rmoea/d algorithm with concave pf. fig. 4. result for function generator extension gaussian noise and \(\delta _x = 0.1\). full size image figure 5 displays the results of extending the function generator with uniform noise and \(\delta _x=0.1\). the analysis of results for uniform noise is similar to gaussian noise. the three algorithms did not converge for high-intensity noise. the algorithms converged and had good dispersion for moderate-intensity noise. however, the rmoea/d and c-rmoea/d had some poor solutions. rnsga-ii had some poor solutions for concave pf. as a result, it becomes evident that these algorithms with simultaneous uncertainties begin to deteriorate (even with small noise). fig. 5. result for function generator extension uniform noise and \(\delta _x = 0.1\). full size image figure 6 displays the result of extending the function generator with cauchy noise and \(\delta _x=0.1\). this noise differs from the previous ones in that it is additive and not multiplicative. the c-rmoea/d and rnsga-ii showed good convergence and dispersion for all pf formats and intensities. for high-intensity noise, the c-rmoea/d had some solutions without convergence, and for the convex pf it failed to map the robust pf at the extremes. the rmoea/d algorithm showed good convergence and dispersion for moderate-intensity noise. only some final solutions were poor. finally, rmoea/d produced some solutions that mapped small parts of the pf to high-intensity noise. therefore, the solutions did not have good dispersion and most of them did not converge. the worst convergence result was for convex pf. fig. 6. result for function generator extension cauchy noise and \(\delta _x = 0.1\). full size image as stated in the previous results, the original function is severely contaminated by high gaussian and uniform noise. therefore, we created an intermediate-intensity noise for further testing. the value was \(\beta =0.5\) for gaussian noise. for uniform noise, the parameters were \(\beta =0.5\) and \(\alpha =0.2*(0.49+1/d)\). the effect of intermediate-intensity noise showed that the trend of the original function remained. consequently, eas must be able to eliminate the effect of noise at this intensity. for this test set (intermediate-intensity), 30 simulations were performed for each front and algorithm. the igd (inverted generational distance) evaluation metric [24] was calculated for each simulation, using the robust front as the reference set. the solution set that performed best according to the igd metric (closest to zero) was chosen, and fig. 7 shows these results for tests of the function generator extension with intermediate noise intensity and \(\delta _x=0.1\). the algorithms did not achieve good convergence, regardless of the geometry of the pf (convex, linear, or concave) or the type of noise (gaussian or uniform). this indicates that, when the noise has a greater effect (intermediate intensity), the tested algorithms were unable to eliminate its effect with perturbation. specifically, rnsga-ii started to converge at the extremes and in the middle of the pf for gaussian noise and convex pf. the same happened for linear pf, but with a smaller number of solutions. some rmoea/d solutions also converged in the central region of the robust pf (linear pf and gaussian noise). these algorithms did not show convergence when the pf was concave. the solutions of the c-rmoea/d algorithm showed a trend of robust pf shapes for gaussian noise, mainly the convex pf. uniform noise had a more significant detrimental effect (compared to gaussian) on the original function. the results (fig. 7) were consistent for all pf formats and uniform noise. the algorithms did not converge. the c-rmoea/d showed a tendency towards solutions on the border. fig. 7. result for the extension of the function generator intermediate noise and \(\delta _x = 0.1\) (perturbation in the objective functions and decision variables). full size image table 3 contains the average of the igd metric for the 30 simulations, considering the robust front as a reference. the previous results qualitatively demonstrated that the algorithms were not able to deal with simultaneous uncertainties. the quantitative results (table 3) reinforce this conclusion. the average igd was high for all algorithms (keep in mind that a lower igd value indicates better performance), except for the crmoea/d algorithm with convex shape. the crmoea/d can be considered the superior algorithm regardless of the format of the pareto front. only for the concave front and uniform noise, the rnsga-ii algorithm showed slightly better performance. table 3. results of the igd metric for intermediate intensity noise and \(\delta _x = 0.1\).full size table table 3 displays the variability of the igd metric across 30 simulations, using the robust front as the reference, as indicated by the standard deviation. the rnsga-ii algorithm demonstrated superiority in this test across different noise and border shape settings, with consistently low standard deviation. the standard deviation test indicates that the rnsga-ii algorithm exhibited minimal variations in igd across the 30 simulations. the crmoea/d demonstrated superior performance compared to the average igd. however, it is worth noting that the standard deviation was high, suggesting the presence of simulations with exceptionally poor results. conversely, the rmoea/d exhibited unsatisfactory outcomes in terms of both the mean and standard deviation. 4.4 discussion algorithms with a robustness measure had no difficulties in finding the robust front when the problem has only perturbation. the evolutionary algorithms were able to eliminate the effect of noise for moderate intensity (noisy optimization), including algorithms without any robustness measure (nsga-ii and moea/d). however, the algorithms did not achieve convergence when the effect of gaussian and uniform noise was high (severe intensity). one possible explanation is that the intensity of these noises severely deteriorates the original function, causing the function’s trend to be lost. cauchy noise is different from the previous ones. this noise has peaks of high intensity. as a result, the trend of the original function remains even with severe cauchy noise. the rmoea/d algorithm had difficulties in mapping the global front to severe cauchy noise. the reason is that noise has contaminated the average value (noise spikes have increased the average). the original moea/d algorithm managed to map the global front for this noise. one explanation is that the evolutionary process eliminated the effect of noise on solutions. the nsga-ii and rnsga-ii algorithms achieved excellent mapping of the global front for cauchy noise. this shows that the characteristics of the nsga-ii (archive, elitism, dominance) eliminated the effect of this noise, even when the average was used (rnsga-ii). finally, for robust noisy problems, the results demonstrate that the tested algorithms had difficulties in this new class of problems. some solutions started to deteriorate with moderate noise and perturbation. we performed some tests by increasing the noise intensity and observed that the algorithms presented greater difficulty for problems with simultaneous uncertainties. thus, we created an intermediate noise. the algorithms did not solve the problems with perturbation and intermediate noise. to confirm the conclusion presented in this discussion, results are provided for gaussian noise with a value of \(\beta = 0.25\), as depicted in fig. 8. the rnsga-ii algorithm successfully mapped the global frontier when the problem contained only noise. however, when both perturbation and noise were present, the majority of algorithm solutions did not converge. fig. 8. result for the extension of the function generator \(\beta =0.25\) gaussian noise). full size image 5 conclusion in the real world, simultaneous uncertainties are common. a practical example is a drilling process, where the sensor measuring the depth is subject to uncertainty (resulting in noise in the objective function), and the drill (actuator) is also subject to uncertainty (resulting in perturbations in the decision variables). the authors identified a deficiency in the field of optimization with uncertainty, as no previous research had dealt with simultaneous uncertainties. the results indicated that the algorithms tested (rnsga-ii, rmoea/d, and crmoea/d) were unable to effectively handle simultaneous uncertainties. consequently, there is a need to formulate algorithms that are capable of handling both robust and noisy environments concurrently. notes 1.for gaussian noise: moderate intensity considers \(\beta = 0.01\), which corresponds to a variation of up to 3\(\%\) (either multiplying the function by 1.03 or multiplying the function by 0.97); severe intensity considers \(\beta = 1\), resulting in a variation of up to 20 times (either multiplying the function by 20 or dividing the function by 20). for uniform noise: moderate intensity considers \(\beta = 0.01\) and \(\alpha = 0.01(0.49 + 1/d)\), where d represents the number of decision variables (always considered as 24 in this work), resulting in a variation of up to 12\(\%\); severe intensity considers \(\beta = 1\) and \(\alpha = 0.49 + 1/d\), resulting in a variation of up to tens of thousands. for cauchy noise: moderate intensity considers \(\alpha = 0.01\) and \(p = 0.05\); severe intensity considers \(\alpha = 1\) and \(p = 0.2\). 2.further details on the decomposition algorithm and methods can be found in [23]. references balouka, n., cohen, i.: a robust optimization approach for the multi-mode resource-constrained project scheduling problem. eur. j. oper. res. 291(2), 457–470 (2021) article  mathscinet  math  google scholar  ben-tal, a., el ghaoui, l., nemirovski, a.: robust optimization, vol. 28. princeton university press, princeton (2009) google scholar  ben-tal, a., el ghaoui, l., nemirovski, a.: robust optimization in applied mathematics. princeton series, princeton (2009) book  math  google scholar  beyer, h.g., sendhoff, b.: robust optimization – a comprehensive survey. comput. methods appl. mech. eng. 196(33), 3190–3218 (2007). https://doi.org/10.1016/j.cma.2007.03.003 article  mathscinet  math  google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  google scholar  deb, k., sindhya, k., hakanen, j.: introducing robustness in multi-objective optimization. evol. comput. 14(4), 463–494 (2006) article  google scholar  duan, j., he, z., yen, g.g.: robust multiobjective optimization for vehicle routing problem with time windows. ieee trans. cybern. 52(8), 8300–8314 (2021) article  google scholar  finck, s., hansen, n., ros, r., auger, a.: real-parameter black-box optimization benchmarking 2010: presentation of the noisy functions. technical report. citeseer (2010) google scholar  gaspar-cunha, a., covas, j.a.: robustness in multi-objective optimization using evolutionary algorithms. comput. optim. appl. 39(1), 75–96 (2007). https://doi.org/10.1007/s10589-007-9053-9 article  mathscinet  math  google scholar  goerigk, m., schöbel, a.: algorithm engineering in robust optimization. springer, cham (2016). https://doi.org/10.1007/978-3-319-49487-6_8 book  math  google scholar  gorissen, b.l., yanıkoğlu, i̇, den hertog, d.: a practical guide to robust optimization. omega 53, 124–137 (2015). https://doi.org/10.1016/j.omega.2014.12.006 article  google scholar  häse, f., et al.: olympus: a benchmarking framework for noisy optimization and experiment planning. mach. learn. sci. technol. 2(3), 035021 (2021) article  google scholar  jin, y., branke, j.: evolutionary optimization in uncertain environments – a survey. trans. evol. comput. 9(3), 303–317 (2005) article  google scholar  klir, g.j., folger, t.a.: fuzzy sets, uncertainty, and information. prentice-hall, englewood cliffs (1998) math  google scholar  liu, j., liu, y., jin, y., li, f.: a decision variable assortment-based evolutionary algorithm for dominance robust multiobjective optimization. ieee trans. syst. man cybern. syst. 52(5), 3360–3375 (2021) article  google scholar  liu, r., li, y., wang, h., liu, j.: a noisy multi-objective optimization algorithm based on mean and wiener filters. knowl.-based syst. 228, 107215 (2021) article  google scholar  lu, y., xu, y., herrera-viedma, e., han, y.: consensus of large-scale group decision making in social network: the minimum cost model based on robust optimization. inf. sci. 547, 910–930 (2021) article  mathscinet  math  google scholar  meneghini, i.r., alves, m.a., gaspar-cunha, a., guimaraes, f.g.: scalable and customizable benchmark problems for many-objective optimization. appl. soft comput. 90, 106139 (2020) article  google scholar  meneghini, i.r., guimaraes, f.g., gaspar-cunha, a.: competitive coevolutionary algorithm for robust multi-objective optimization: the worst case minimization. in: ieee congress on evolutionary computation (cec), pp. 586–593 (2016). https://doi.org/10.1109/cec.2016.7743846 mou, w., wang, q., peng, j.: accelerating gradient-based optimization via importance sampling. j. mach. learn. res. 22(22), 1–29 (2021) google scholar  ong, y.s., nair, p.b., lum, k.y.: max-min surrogate-assisted evolutionary algorithm for robust design. ieee trans. evol. comput. 10(4), 392–404 (2006). https://doi.org/10.1109/tevc.2005.859464 article  google scholar  sahmoud, s., topcuoglu, h.r.: dynamic multi-objective evolutionary algorithms in noisy environments. inf. sci. 634, 650–664 (2023) article  google scholar  trivedi, a., srinivasan, d., sanyal, k., ghosh, a.: a survey of multiobjective evolutionary algorithms based on decomposition. ieee trans. evol. comput. 21(3), 440–462 (2016) google scholar  van veldhuizen, d.a., lamont, g.b.: multiobjective evolutionary algorithm research: a history and analysis. technical report. citeseer (1998) google scholar  yang, j., su, c.: robust optimization of microgrid based on renewable distributed power generation and load demand uncertainty. energy 223, 120043 (2021) article  google scholar  yang, y.: robust multi-objective optimization based on the idea of multi-tasking and knowledge transfer. in: proceedings of the 14th international conference on computer modeling and simulation, pp. 257–265 (2022) google scholar  zhang, q., li, h.: moea/d: a multiobjective evolutionary algorithm based on decomposition. ieee trans. evol. comput. 11(6), 712–731 (2007) article  google scholar  download references acknowledgment this work has been supported by the brazilian agencies (i) national council for scientific and technological development (cnpq), grant no. 312991/2020-7; (ii) coordination for the improvement of higher education personnel (capes) through the academic excellence program (proex) and (iii) foundation for research of the state of minas gerais (fapemig, in portuguese), grant no. apq-01779-21. minds laboratory – https://minds.eng.ufmg.br/ author information authors and affiliations instituto federal de minas gerais, bambuí, minas gerais, brazil mateus clemente de sousa instituto federal de minas gerais, ibirité, minas gerais, brazil ivan reinaldo meneghini universidade federal de minas gerais, belo horizonte, minas gerais, brazil frederico gadelha guimarães machine intelligence and data science – minds lab, universidade federal de minas gerais, belo horizonte, brazil mateus clemente de sousa, ivan reinaldo meneghini & frederico gadelha guimarães authors mateus clemente de sousaview author publications search author on:pubmed google scholar ivan reinaldo meneghiniview author publications search author on:pubmed google scholar frederico gadelha guimarãesview author publications search author on:pubmed google scholar corresponding author correspondence to mateus clemente de sousa . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de sousa, m.c., meneghini, i.r., guimarães, f.g. (2023). assessment of robust multi-objective evolutionary algorithms on robust and noisy environments. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_3 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords robust optimization noisy optimization evolutionary algorithm test functions publish with us policies and ethics profiles ivan reinaldo meneghini view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature physicochemical properties for promoter classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper physicochemical properties for promoter classification conference paper first online: 12 october 2023 pp 368–382 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) physicochemical properties for promoter classification download book pdf download book epub lauro moraes  orcid: orcid.org/0000-0002-3688-31629, eduardo luz  orcid: orcid.org/0000-0001-5249-15599 & gladston moreira  orcid: orcid.org/0000-0001-7747-59269  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 523 accesses abstract the accurate identification of promoter regions in dna sequences holds significant importance in the field of bioinformatics. while this problem has garnered substantial attention in the literature, it remains unresolved. several researchers have achieved notable outcomes by employing diverse machine-learning techniques to predict promoter regions. however, only a few have thoroughly explored the utilization of features derived from the physicochemical properties of dna across various organism types. this study investigates the advantages of incorporating these features in the training of machine-learning models. the research evaluates and compares the performance of multiple metrics on diverse datasets encompassing both prokaryotic and eukaryotic organisms. the state-of-the-art cnnprom method is employed as the baseline for our experiments. the models and source code associated with this study can be accessed at the following url of the project’s repository: https://anonymous.4open.science/r/bracis-paper-1458/. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others explainable artificial intelligence as a reliable annotator of archaeal promoter regions article open access 31 january 2023 machine learning in bioinformatics: new technique for dna sequencing classification chapter © 2022 epdrna: a model for identifying dna–rna binding sites in disease-related proteins article 16 march 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biophysical chemistry categorization machine learning nucleic acid structure prediction bioinformatics 1 introduction the identification of gene products and their location within dna sequences that have not been experimentally characterized, commonly known as gene finding, is a central topic of interest in computational biology. prediction of promoter sequences and transcriptional start points can help signal a transcript’s approximate start, thereby identifying one end of a gene. this information is particularly useful in dna sequences derived from higher eukaryotes, where coding regions are isolated segments embedded within a non-coding dna background [28]. a promoter region in dna is a noncoding sequence of dna located upstream of a gene that is responsible for regulating the expression of that gene [17]. the promoter region contains binding sites for transcription factors, which are proteins that bind to the promoter and recruit rna polymerase to initiate transcription. the specific pattern of binding sites in the promoter determines the level of gene expression and the conditions under which the gene is active [5]. high-resolution promoter recognition in dna is an important area of research in bioinformatics. it involves the use of algorithms to find the transcription start site (tss) of a gene without the need for laborious and costly experimental techniques such as aligning expressed sequence tags (ests), complementary dnas (cdnas) or messenger rnas (mrnas) to the entire genome [33]. the tss is a specific genomic location within the promoter region where rna polymerase binds and initiates transcription. it marks the starting point of the transcription process, where the dna sequence is transcribed into rna. defining the tss position as a reference, the upstream region is the dna sequence located before the transcription start site, and the downstream region is situated after the tss. identifying the promoter region allows us to study the regulation of gene expression and understand how different genes are controlled in various tissues, conditions, and stages of development [31]. in addition, by identifying the promoter regions for a set of genes, one can infer information about the molecular mechanisms that control gene expression and how these mechanisms may change in response to different stimuli. so, these algorithms can efficiently delimit regions involved in transcriptional regulation, guiding further experimental work, given the lower cost associated with computational approaches [33]. the identification of promoter regions is often performed by computational methods, such as promoter prediction algorithms, that analyze the dna sequence and look for characteristic features such as tata boxes [20] and cpg islands [10] and other short conserved sequences. however, multiple groups of genes don’t contain these features. methods for identifying promoters have been widely adopted in bioinformatics, as it allows for the generation of comprehensive maps of promoter regions in a genome which can be used for further analysis, such as studying gene regulation in response to environmental cues or identifying novel regulatory elements. despite recent advances in promoter recognition algorithms, accurately identifying promoters remains challenging due to the diversity and complexity of these sequences in genome [34]. methods for promoter prediction have been established to recognize promoter regions within the dna sequences of both prokaryotic and eukaryotic organisms. these methods harness various machine learning algorithms for their functioning. the support vector machines (svm) [9] were used in [26] to evaluate the strength in a dataset of escherichia coli trc promoter. [1] applied a genetic algorithm to calibrate the svm hyperparameters using a human promoter dataset. [3] used datasets of three higher eukaryotes, saccharomyces cerevisiae, a. thaliana, and human, to train convolutional neural network (cnn) [24] with long short term memory (lstm) [18] and random forest (rf) [4] models. a study of promoter prediction on bacterial datasets was performed by [8] using rf models. in [25] a stacked ensemble of lightgbm [21], xgboost [6], adaboost [14], gbdt [15], and rf models were used on a escherichia coli dataset. the work of [30] proposed some cnn models to identify promoters on different organisms datasets. in [27] the same datasets were used to train a capsule neural network (capsnet) [29]. the existing literature of promoter prediction lacks comprehensive investigations regarding the influence of dna physicochemical properties on the accuracy of promoter prediction across diverse organisms. consequently, the primary objective of this study is to examine the efficacy of features derived from dna physicochemical properties in training machine learning models for predicting promoter regions in both prokaryotic and eukaryotic organisms. to assess the performance of our models, we utilize the datasets introduced by umarov et al. (2017) [30], which encompass a wide range of organisms. for each property, we train and compare multiple machine learning models, as well as ensembles of these models, to identify the most effective properties. to evaluate the overall predictive capacity of these models, we compare their performance against the state-of-the-art method for these datasets, namely cnnprom [29]. the results indicate that the models based solely on properties do not yield optimal performance. still, they may serve as a foundational step for future research by combining them with other techniques and models to enhance predictive accuracy. 2 materials and methods 2.1 benchmark datasets the seven datasets utilized in this study were sourced from the supplementary materials made available by [30] and can be accessed on the corresponding github repositoryfootnote 1. regrettably, the dataset pertaining to “human tata” could not be accessed through the repository and therefore was excluded from the analysis in this work. the eukaryotic organism sequences, spanning 251bp (base pairs) with a transcription start site (tss) located at the 200th position, were sourced exclusively from the epd database [12]. the dataset includes arabidopsis tata and arabidopsis non-tata sequences from a plant species, as well as mouse tata, mouse non-tata, and human non-tata sequences from mammalian species. non-promoter sequences in these datasets are comprised of random gene fragments located after the first exons. the bacterial promoter sequences of bacillus subtilis were obtained from dbtbs [19], while the escherichia coli s70 sequences were acquired from regulondb [16]. these sequences, which comprise 81bp, contain the transcriptional start site (tss) at the 60th position. conversely, the non-promoter sequences for these prokaryotic organisms include the reverse sequences of random fragments extracted from protein-coding genes. figure 1 shows the number of promoters and non-promoters in each dataset. notably, all datasets have more non-promoter samples than promoter samples. this characteristic can potentially affect the model learning process as they tend to classify the data into the majority class [22]. besides, the prokaryotic datasets have fewer samples compared to eukaryotic datasets. fig. 1. the number of promoters samples and non-promoters samples present in each benchmark dataset. full size image 2.2 feature representation adenine (a), cytosine (c), guanine (g), and thymine (t) are the four nucleotides that comprise the alphabet \(v=\{a, c, g, t\}\) forming the basis of dna. a dna sample of length \(l_{s} \in \mathbb {n}^{*}\) base pairs is represented by \({s} \in v^{l_{s}}\), which can be written as a sequence \({s} = s_{1} ... s_{i} ... s_{l_{{s}}}\), where \(s_{i}\) is the i-th nucleotide of the sequence s. to convert the dna sequence into a numeric format, we can use the physicochemical properties of dinucleotides or trinucleotides. a dinucleotide represents a pair of adjacent nucleotides in a dna sequence, while a trinucleotide represents three adjacent nucleotides. these properties capture the physicochemical characteristics of nucleotides and the structural features of dna [7]. for example, the properties may include measures of hydrogen bond formation, base stacking, base pairing, or flexibility of the dna helix. k-mers are substrings of length k that are extracted from a longer dna or protein sequence. this bioinformatics concept can be used in sequence analysis to identify motifs, patterns, or features characteristic of a specific biological function. considering the nucleotide alphabet v, there is \(|v|^{k}\) possible mers with length k. as the name suggests, dinucleotides are two adjacent nucleotides that occur in a dna sequence. it can be seen as a special case of k-mers with \(k=2\). there are \(4^{2} = 16\) possible dinucleotides, such as “at”, “cg”, “ta”, etc. while trinucleotides are three adjacent nucleotides that occur in a dna sequence. it is a special case of k-mers with \(k=3\). there are \(4^{3} = 64\) possible trinucleotides, such as “atg”, “cgt”, “taa”, etc. there are several physicochemical properties in the literature. the work of [7] used 38 properties for dinucleotides and 12 for trinucleotides, making the mapping values available for each in the supplementary data section. we used these data to convert the dataset sequences and create the input features for our tested models. in table 1, we show the names of all of these 50 properties that we used. there is one column for the dinucleotide names and one for the trinucleotide names. table 1. list of all physicochemical properties names. there are 38 dinucleotides and 12 trinucleotides.full size table to convert the dna sequence into dinucleotide or trinucleotide physicochemical properties, we can use a function that maps each dinucleotide or trinucleotide to a corresponding numerical value based on the selected physicochemical property. this mapping function can be represented as \(f: v^{k} \rightarrow \mathbb {r}\). a sequence s with \(l_{s}\) nucleotides is converted to a sequence of \(l_{s}-k+1\) property values. so, \(f_{{d}_{m}}(s) \rightarrow \mathbb {r}^{l_{s}-1}\) is the function that maps each dinucleotide to its related physicochemical property m, where \(1 \le m \le 38\). while \(f_{{t}_{n}}(s) \rightarrow \mathbb {r}^{l_{s}-2}\) define a similar mapping function to convert trinucleotides into the m-th physicochemical property, where \(1 \le n \le 12\). using a sliding window strategy, with length 2 bp for dinucleotides and 3 bp for trinucleotides, and a step size of 1 bp, we apply the corresponding mapping function to get all physicochemical property numeric values and create the feature vectors that we use in this work to train machine learning models. figure 2 shows an example of converting a nucleotide sequence of length 8 bp into two numeric vectors to be used as features. on the left, we use a sliding window of 2 bp to extract all 7 dinucleotides and then apply the mapping function to convert them to numeric values using the physicochemical property “shift”. on the right, we apply a similar strategy with a sliding window of 3 bp to create a mapped vector of length 6 using the trinucleotide physicochemical property “nucleosome”. fig. 2. illustration of the conversion process of a sequence. on the left, the dinucleotides are mapped to the physicochemical property “shift”. on the right, the trinucleotides are mapped to the physicochemical property “nucleosome”. full size image with this approach, we can describe the profiles of properties in each dataset. for example, fig. 3 illustrates all 12 trinucleotide average profiles in the human_non_tata dataset. there is one graph for each trinucleotide property. each point presents the average property value in a specific position of the analyzed sequences. the positions in the x-axis give negative values for the upstream part of sequences and positive ones for the downstream portion. position zero is the tss. we can see the different patterns in the promoter and non-promoter samples. the non-promoter’s averages tend not to show significant variations, while the promoter’s averages tend to vary around the tss position. fig. 3. average profiles of twelve features along promoter and non-promoter sequences of the 251bp dataset human_non_tata. each graph is related to a different trinucleotide’s physicochemical properties. full size image 2.3 k-fold cross-validation evaluation k-fold cross-validation [13] is a popular technique used in machine learning and statistical modeling for assessing model performance. it involves dividing the dataset into k subsets or folds, using one fold as the validation set and the remaining k-1 folds as the training set. this process is repeated k times, with each fold serving as the validation set once. by repeatedly training and evaluating the model on different subsets of the data, k-fold cross-validation provides a robust estimate of the model’s performance on new, unseen data, helping to mitigate the risk of overfitting. to address the issue of imbalanced target variables, a modification of k-fold cross-validation called stratified k-fold cross-validation [11] is often employed. stratified k-fold cross-validation ensures that each fold maintains a similar distribution of classes as the overall dataset. it involves dividing the dataset into k folds and adjusting the fold split to preserve the class distribution in both the training and validation sets. this technique is particularly useful when dealing with imbalanced datasets, as it ensures that the model is evaluated on all classes rather than being biased towards the most frequent class. the performance metrics obtained from each fold in k-fold cross-validation are typically averaged to obtain an overall estimate of the model’s performance. this technique applies to various evaluation metrics and provides a more reliable assessment of the model’s generalization ability. by employing k-fold cross-validation, researchers can confidently evaluate their models and make informed decisions regarding model selection and performance comparison in the field of machine learning and statistical modeling. 2.4 evaluation measures the problem of promoter classification can be defined as a binary classification task, where the predicted class variable can take on two values: 1 (one) if the dna sequence is predicted to be a promoter and 0 (zero) otherwise. this classification task leads to four possible outcomes: a true positive (tp), which occurs when the model correctly identifies a promoter sequence; a true negative (tn), which occurs when the model correctly identifies a non-promoter sequence; a false positive (fp), which occurs when the model incorrectly identifies a non-promoter sequence as a promoter; and a false negative (fn), which occurs when the model incorrectly identifies a promoter sequence as a non-promoter. precision (prec) measures the relevance of the predicted true positives (eq. 1). sensitivity, also known as recall (sn), measures the proportion of true positives (eq. 2). specificity (sp) measures the proportion of true negatives (eq. 3). accuracy (acc) reflects the proportion of correct predictions (eq. 4). the f1-score (f1) metric, a harmonic mean of precision and recall, combines both measures and assesses the classifier’s performance (eq. 5). $$\begin{aligned} prec = \frac{tp}{tp+fp} \end{aligned}$$ (1) $$\begin{aligned} sn = \frac{tp}{tp+fn} \end{aligned}$$ (2) $$\begin{aligned} sp = \frac{tn}{tn+fp} \end{aligned}$$ (3) $$\begin{aligned} acc= \frac{tp+tn}{tp+tn+fp+fn} \end{aligned}$$ (4) $$\begin{aligned} f1 = 2 \times \frac{prec \times sn}{prec + sn} \end{aligned}$$ (5) matthews correlation coefficient (mcc) is a balanced metric that considers all four components of the confusion matrix and provides a balanced measure ranging from -1 to +1, with +1 indicating perfect classification, 0 indicating random classification, and -1 indicating completely opposite classification (eq. 6). it is more robust than other metrics in handling class imbalance and biased datasets [35] $$\begin{aligned} mcc = \frac{tp \times tn fp \times fn}{\sqrt{(tp+fp)(tp+fn)(tn+fp)(tn+fn)}} \end{aligned}$$ (6) the area under the curve (auc) is a metric used to evaluate the performance of a binary classification model. it represents the area under the receiver operating characteristic (roc) curve, which plots the true positive rate against the false positive rate for different classification thresholds. auc ranges from 0 to 1, with higher values indicating better classification performance. 2.5 hyperparameter optimization the tree-structured parzen estimator (tpe) [2] is a bayesian optimization algorithm that efficiently explores the search space to find optimal hyperparameters for machine learning models. tpe models the relationship between hyperparameters and performance metrics by maintaining two probability density functions (pdfs) representing successful and unsuccessful configurations. it utilizes these pdfs to guide the search towards regions of the search space more likely to yield improved performance. during the tuning process, tpe samples hyperparameter configurations based on its probability distributions, evaluate model performance using cross-validation and updates the pdfs accordingly. this iterative approach enables the algorithm to explore and refine the search space, ultimately leading to the identification of hyperparameter configurations that optimize model performance. tpe’s combination of exploration and exploitation strategies makes it a powerful tool for effectively and efficiently optimizing machine learning models. 2.6 ensemble methods the voting ensemble method [23] combines predictions from multiple individual models to make a final prediction. it involves aggregating the outputs of different models, either by majority voting (for classification problems) or by averaging (for regression problems). by combining the predictions of multiple models, the voting ensemble could enhance the overall prediction accuracy and robustness. the stacking ensemble method [32] leverages the predictions of multiple base models to train a meta-model, which learns to make predictions based on the outputs of the base models. the predictions, along with the original target variable, are then used to train a meta-model that learns to combine the base models’ outputs. the stacking ensemble can capture complex interactions among the models and exploit their complementary strengths, leading to improved prediction performance. 3 experiments and results in order to assess the performance of physicochemical properties in the context of promoter prediction, a series of experiments were conducted. the evaluation encompassed the utilization of individual properties as well as combinations of the most promising ones across seven diverse datasets representing various organism types. to ascertain the effectiveness of the proposed methods within the domain of the problem, the obtained results were compared against a baseline that is regarded as state-of-the-art on these specific datasets. 3.1 experiment workflow the implemented processing workflow encompasses the transformation of raw dna sequences within a given dataset into physicochemical property values. subsequently, one or more properties are selected, and their corresponding values are concatenated to form the feature set. the evaluation process involves a 5-fold cross-validation, which partitions the dataset into training and test subsets. within the training subset, a further 5-fold cross-validation is conducted to compare various machine learning methods and identify the optimal one for hyperparameter tuning. finally, the tuned model is evaluated using the test dataset. to conduct our experiments, we utilized pycaret 3footnote 2, an advanced machine learning library. pycaret offers a comprehensive range of eighteen classification methods, which were employed in our study (table 2). 3.2 individual properties’ performances table 2. metrics results of the best models with only one property. we show the property and the results of the evaluated model in each dataset.full size table using the 5-fold cross-validation method, we compared several machine learning models to observe the prediction power of each physicochemical property. then, we chose the best one that uses that property. after, we tuned the hyperparameters of the selected models using the same training data and the 5-fold cross-validation. figure 4 presents the results from the tuned models. the bars show the mcc scores’ maximum, minimum, and average from evaluating these models. each dataset has fifty models, one for each property, among dinucleotides and trinucleotides. the graph suggests that the models trained on prokaryotic datasets get worse results than those from the eukaryotic datasets. the labels on the x-axis of the graph refer to table 1. although we tested eighteen different models, all the best models were the light gradient boosting machine. fig. 4. mcc scores (y-axis) from 5-fold cross-validation of each physicochemical property (x-axis) model evaluated on the training subset for each dataset. the graph shows the maximum, minimum, and average computed scores. full size image 3.3 performance comparisons we utilized cnnprom [30] and mcc scores baseline for comparing the performance of the models under evaluation, given our belief that it represents the state-of-the-art solution for this problem. the corresponding results can be found in table 3. the scores attributed to cnnprom are as reported by its authors. columns labeled as “single” denote the best-performing individual property model. the columns “vote5”, “vote10”, “stack5”, and “stack10” represent results from ensemble models that employed either the voting or stacking method, incorporating the top 5 or 10 properties, respectively. the performance of the “single” model demonstrated relatively lower average results when compared with other models. interestingly, the outcomes derived from the ensemble models did not exhibit a significant discrepancy whether they incorporated the top 5 or 10 properties, insinuating that an increased number of properties does not substantially enhance the final model’s efficacy. likewise, no notable divergence was observed between the “vote” and “stack” ensemble methodologies. when comparing the investigated models to the baseline model cnnprom, we observed similar performance in 3 out of 7 datasets. specifically, in the arabidopsis_non_tata dataset, our model exhibited a marginally lower performance by \(1\%\) compared to cnnprom. conversely, in the arabidopsis_tata dataset, our model achieved a slightly higher performance by \(1\%\) in comparison. in addition, the results for the mouse_non_tata dataset were equivalent to the baseline model. however, in the bacillus subtilis, escherichia coli s70 (procaryotes), and human_non_tata datasets, our implementations did not yield comparable results to the baseline model. table 3. mean values of mcc metrics obtained from 5-fold cross-validation on each dataset.full size table 4 conclusions this work presents a study of the use of the physicochemical properties of dna as features in training machine-learning models. using different metrics, we evaluated several models using seven datasets of procaryotic and eucaryotic organisms. our study analyzed the prediction performance of individual properties models and ensemble models using the best properties, comparing them with the state-of-art method cnnprom for the evaluated datasets. we concluded that even though the models trained with the properties achieved competitive prediction results, they were unable to surpass the established baseline. so, it is necessary to evaluate new strategies to use these physicochemical properties values, like deriving other features through feature engineering or using deep learning models to automatically derive better features from them. notes 1.https://github.com/solovictor/cnnpromoterdata. 2.official page: https://pycaret.gitbook.io/. references arslan, h.: a new promoter prediction method using support vector machines. in: 2019 27th signal processing and communications applications conference (siu), pp. 1–4. ieee (2019) google scholar  bergstra, j., bardenet, r., bengio, y., kégl, b.: algorithms for hyper-parameter optimization. in: advances in neural information processing systems, vol. 24 (2011) google scholar  bhandari, n., khare, s., walambe, r., kotecha, k.: comparison of machine learning and deep learning techniques in promoter prediction across diverse species. peerj comput. sci. 7, e365 (2021) article  google scholar  breiman, l.: random forests. mach. learn. 45, 5–32 (2001) article  math  google scholar  cartharius, k., et al.: matinspector and beyond: promoter analysis based on transcription factor binding sites. bioinformatics 21(13), 2933–2942 (2005) article  google scholar  chen, t., guestrin, c.: xgboost: a scalable tree boosting system. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794 (2016) google scholar  chen, w., lei, t.y., jin, d.c., lin, h., chou, k.c.: pseknc: a flexible web server for generating pseudo k-tuple nucleotide composition. anal. biochem. 456, 53–60 (2014) article  google scholar  chevez-guardado, r., peña-castillo, l.: promotech: a general tool for bacterial promoter recognition. genome biol. 22, 1–16 (2021) article  google scholar  cortes, c., vapnik, v.: support-vector networks. mach. learn. 20, 273–297 (1995) article  math  google scholar  deaton, a.m., bird, a.: cpg islands and the regulation of transcription. genes dev. 25(10), 1010–1022 (2011) article  google scholar  dietterich, t.g.: approximate statistical tests for comparing supervised classification learning algorithms. neural comput. 10(7), 1895–1923 (1998) article  google scholar  dreos, r., ambrosini, g., cavin périer, r., bucher, p.: epd and epdnew, high-quality promoter resources in the next-generation sequencing era. nucleic acids res. 41(d1), d157–d164 (2013) article  google scholar  efron, b.: estimating the error rate of a prediction rule: improvement on cross-validation. j. am. stat. assoc. 78(382), 316–331 (1983) article  mathscinet  math  google scholar  freund, y., schapire, r.e.: a decision-theoretic generalization of on-line learning and an application to boosting. j. comput. syst. sci. 55(1), 119–139 (1997) article  mathscinet  math  google scholar  friedman, j.h.: greedy function approximation: a gradient boosting machine. ann. stat. 1189–1232 (2001) google scholar  gama-castro, s., et al.: regulondb version 9.0: high-level integration of gene regulation, coexpression, motif clustering and beyond. nucleic acids res. 44(d1), d133–d143 (2016) google scholar  goñi, j.r., pérez, a., torrents, d., orozco, m.: determining promoter location based on dna structure first-principles calculations. genome biol. 8(12), r263 (2007) article  google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  google scholar  ishii, t., yoshida, k.i., terai, g., fujita, y., nakai, k.: dbtbs: a database of bacillus subtilis promoters and transcription factors. nucleic acids res. 29(1), 278–280 (2001) google scholar  juven-gershon, t., kadonaga, j.t.: regulation of gene expression via the core promoter and the basal transcriptional machinery. dev. biol. 339(2), 225–229 (2010) article  google scholar  ke, g., et al.: lightgbm: a highly efficient gradient boosting decision tree. in: advances in neural information processing systems, vol. 30 (2017) google scholar  kotsiantis, s., kanellopoulos, d., pintelas, p., et al.: handling imbalanced datasets: a review. gests int. trans. comput. sci. eng. 30(1), 25–36 (2006) google scholar  kuncheva, l.i.: combining pattern classifiers: methods and algorithms. john wiley & sons (2014) google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  google scholar  li, f., et al.: computational prediction and interpretation of both general and specific types of promoters in escherichia coli by exploiting a stacked ensemble-learning framework. brief. bioinform. 22(2), 2126–2140 (2021) article  mathscinet  google scholar  meng, h., ma, y., mai, g., wang, y., liu, c.: construction of precise support vector machine based models for predicting promoter strength. quant. biol. 5, 90–98 (2017) article  google scholar  moraes, l., silva, p., luz, e., moreira, g.: capsprom: a capsule network for promoter prediction. comput. biol. med. 147, 105627 (2022) article  google scholar  pedersen, a.g., baldi, p., chauvin, y., brunak, s.: the biology of eukaryotic promoter prediction-a review. comput. chem. 23(3–4), 191–207 (1999) article  google scholar  sabour, s., frosst, n., hinton, g.e.: dynamic routing between capsules. in: advances in neural information processing systems, vol. 30 (2017) google scholar  umarov, r.k., solovyev, v.v.: recognition of prokaryotic and eukaryotic promoters using convolutional deep learning neural networks. plos one 12(2), e0171410 (2017) article  google scholar  wasserman, w.w., sandelin, a.: applied bioinformatics for the identification of regulatory elements. nat. rev. genet. 5(4), 276–287 (2004) article  google scholar  wolpert, d.h.: stacked generalization. neural netw. 5(2), 241–259 (1992) article  google scholar  zeng, j., zhu, s., yan, h.: towards accurate human promoter recognition: a review of currently used sequence features and classification methods. brief. bioinform. 10(5), 498–508 (2009) article  google scholar  zhang, m., et al.: critical assessment of computational tools for prokaryotic and eukaryotic promoter prediction. brief. bioinform. 23(2) (2022) google scholar  zhu, q.: on the performance of matthews correlation coefficient (mcc) for imbalanced dataset. pattern recogn. lett. 136, 71–80 (2020) article  google scholar  download references acknowledgment the authors would also like to thank the coordenação de aperfeiçoamento de pessoal de nível superior brazil (capes) finance code 001, fundacão de amparo à pesquisa do estado de minas gerais (fapemig, grants apq-01518-21, apq-01647-22), conselho nacional de desenvolvimento científico e tecnológico (cnpq, grants 307151/2022-0, 308400/2022-4) and universidade federal de ouro preto (proppi/ufop) for supporting the development of this study. we want to express our gratitude for the collaboration of the laboratório multiusuários de bioinformática of núcleo de pesquisas em ciências biológicas (nupeb/ufop). author information authors and affiliations universidade federal de ouro preto, ouro preto-mg, brazil lauro moraes, eduardo luz & gladston moreira authors lauro moraesview author publications search author on:pubmed google scholar eduardo luzview author publications search author on:pubmed google scholar gladston moreiraview author publications search author on:pubmed google scholar corresponding author correspondence to lauro moraes . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper moraes, l., luz, e., moreira, g. (2023). physicochemical properties for promoter classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_25 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords bioinformatics machine learning physicochemical properties promoter classification publish with us policies and ethics profiles eduardo luz view author profile gladston moreira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature damage identification of wind turbine blades | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper damage identification of wind turbine blades conference paper first online: 30 january 2025 pp 187–200 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) damage identification of wind turbine blades download book pdf download book epub alberto régio alves de oliveira  orcid: orcid.org/0009-0007-3437-57909, cláudio marques de sá medeiros  orcid: orcid.org/0000-0002-1663-81519 & geraldo luis bezerra ramalho  orcid: orcid.org/0000-0003-1872-32509  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 383 accesses abstract wind turbines capture the kinetic energy produced by the wind, with the blades being the component most susceptible to damage. unplanned stops result in significant losses, which highlights the need to detect these failures early. as a step in the preventative maintenance procedure, hundreds of color photographs of the blades are taken for subsequent analysis by an expert. in this work, we present a method to highlight superficial damages in wind turbine blades using a classification and localization process. a new dataset was created using images of wind blades, each divided into uniform slices and labeled by an expert according to the type of fault identified. then, we apply class balancing and data augmentation methods prior to fine-tuning a general-propose pre-trained deep convolutional neural network model. the best model was used to classify and locate damages in the wind blades images. as a result, we obtained an overall precision of 96.1%, accuracy of 97% and recall of 94.5% when classifying the presence of damages. we show that our method can be integrated into the monitoring of wind blade tasks, helping the specialist to highlight and identify images containing damage. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others fault diagnosis in wind turbine blades using machine learning techniques chapter © 2023 increasing the wind energy production by identifying the state of wind turbine blade chapter © 2022 investigation of wind turbine blade defect classification based on deep convolutional neural network chapter © 2021 1 introduction wind energy is one of the most important sustainable energy sources. from the 70 s of the 20th century, with constant crises of oil and the increase in concern about its scarcity is why there has been growth in the development of large wind turbines (wts) for energy production [1]. the main components of wt are: set of blades, responsible for capturing the kinetic energy of the winds; nacelle, which houses the essential equipment for energy production; tower, responsible for supporting the nacelle and blades; and transformer, which connects the wt to electrical grid [2]. the set of blades is considered the most important part of wt, as it is responsible for capturing the kinetic energy of the winds, in addition to being the component with the highest cost of wt. generally, blades are made of composite materials (carbon fiber, fiberglass, among others). these materials are characterized by low weight, mechanical resistance and flexibility [3]. blades are also the components most susceptible to damage, being responsible for 15.19% of all incidents recorded till 31 march 2023 [4] with average of 3,800 incidents by year [5]. the main damages are: detachment of the shell, failure of the adhesive joint, detachment of the sandwich panel, delamination due to tensional load, fiber breakage, cracks, among others. the most common causes of damage are: strong winds, storms, lightning, ice, problems with materials and assembling [6]. blade damages can lead to unscheduled shutdown of wt, generating large maintenance and operation costs or the total loss of the equipment, causing major financial impact [8]. therefore, it is very important to identify this damage as soon as possible, in order to avoid the spread of damage and prevent accidents. the use of premature damage detection techniques as monitoring tools can significantly reduce maintenance costs and increase equipment availability. many techniques are employed, namely strain measurement, acoustic emission, ultrasound, vibration, thermography and computer vision. the last is often prefered because is non-invasive, low-cost, has no environmental impacts, is highly accurate and quickly applied, although good quality images are required for an effective result [9]. several recent research uses computer vision to identify damage to wind blades. moreno et al. [10] proposed a deep learning vision-based approach for detecting wind turbines blades (wtbs) damages using a unmanned aerial vehicles (uav). three types of damages were considered: lightning impact, wearing or fracture. the authors used 78 public images collected from the internet to train the model, which was validated on a wind blade prototype built on a 3d printer contained simulated damage. the uav was simulated using a webcam on a robotic arm. the final accuracy of the model achieved 81.25%. yang et al. [11] applied a deep learning classification method using the resnet50 [12] as backbone to identify blade damage. a uav was used to acquire 1,594 images from 20 different types of wind blades, then 557,900 small crops were generated, of which 13,200 were selected for training and testing. the performance of the algorithm was compared with the alexnet [13] network, demonstrating a better result. the final accuracy found was 95.58%. the model was able to classify five types of images: normal, background, cracks, holes and mixed damage. shihavuddin et al. [14] captured images by drone inspection and applied data augmentation techniques to train a faster r-cnn model [15] with inception-resnet-v2 backbone to detect wind blades damages. the acquired images were manually labeled by experts into four classes: leading edge erosion, vortex generator panel (vg panel), vg panel with missing teeth and lightning receptor. the authors found that the use of data augmentation techniques and deep cnn architectures greatly improved the model’s performance and the proposed method managed to achieve an accuracy of 81.10% in mean average precision (map). another contribution of the work was to produce a public image dataset from a wind turbine inspection containing 701 unlabeled images in high definition called dtu drone inspection dataset [16]. foster et al. [17] extended the work of shihavuddin et al. [14] and created a new dataset of public images, but this time labeled. the authors generated more than 13,000 crops of the original images in the size of \(586 \times 371\) pixels with 3,000 labels from the damage and dirt classes. the objective of this work was also to detect wind blades damage using bounding boxes. the authors compared the performance of faster r-cnn models with resnet-101 backbone and yolov5 [18] using this dataset. data augmentation techniques were applied to the images. the authors concluded that the best model was yolov5s. the researches presented above are focused on improving the accuracy of classification models. in the proposal presented in this article, the objective is to classify and locate faults in images obtained from a local wind farm, taking special care not only with precision, but also with accuracy and recall. nowadays, in these local wind farms, the inspection is carried out by technical teams. often, the technician responsible for acquiring the images does not have the necessary knowledge to correctly identify the damage contained in the blades. his responsibility is to capture several photos of each wind turbine and send them to a damage specialist, who is usually located at the company’s headquarters. this specialist needs to analyze a very large number of photos, then filter only those that contain damage and identify its specific type and location, to finally decide what is the best plan for its correction. the specialist must evaluate each image fragment in search of non-uniformities, which is quite repetitive and exhausting, prone to evaluation errors after hours of work. so we use computer vision techniques to classify external structural wtb damages from color photographs, with the aim of reducing the number of images analyzed by experts by discarding undamaged images and also indicating damage existing in the remaining images. the paper is structured as follows: sect. 2 presents concepts about the types and possible causes of damage to wind turbine blades. section 3 presents techniques, parameters and processes used to classify and locate wtb damages. the experimental results are detailed in sect. 4. the sect. 5 presents the concluding remarks. 2 wind turbine blades damage damage to wind turbine blades can occur due to two factors: the first is related to manufacturing processes and/or human errors; the second is caused by external factors such as lightning strikes, ice, uneven loads, moisture, strong wind gusts, among others. damage is detrimental as it affects the energy generation performance of the wind turbine, leading to reduced energy production and, in extreme cases, total equipment loss. wtbs have lightning protection devices, but these are not entirely effective in preventing all lightning strikes, resulting in damage to the blade structure. damage caused by lightning strikes usually includes holes and burns in the area around the point where the lightning struck the blade [19]. the severity of the damage depends on the extent of the lightning strike impact. figure 1a illustrates an example of damage from a lightning strike. the blades also subject to repeated and constant collisions with particles such as raindrops, hail, sand, insects, dust, salt, oil, etc. this leads to erosion/corrosion of the area initially exposed to the wind, known as the leading edge of the blade. erosion affects the aerodynamic efficiency of the blade, which can reduce the capacity for electricity generation. damage caused by erosion is small and superficial in the initial stages, but over time, it increases and extends along the entire leading edge. figure 1b illustrates an example of erosions. damage to paint is often superficial and often considered cosmetic damage as it does not compromise the structure of the blade. examples include paint peeling, oil stains, among others. over time, this type of damage can sometimes progress to erosion damage. usually, they do not require maintenance intervention but should be monitored. figure 1c illustrates an example of cosmetics damage. cracks damages are typically the result of material fatigue in the components of the blades due to prolonged use [8]. they initially start small and shallow but progressively enlarge and deepen, potentially compromising the entire structure of the blade. this type of damage is characterized by a thin and longitudinal shape. figure 1d illustrates an example of a crack. fig. 1. examples of images with red arrows indicating damage. a lightning damage; b erosion damage; c cosmetic damage; d crack damage. (color figure online) full size image 3 proposed method the architecture of the method proposed can be subdivided into two stages: the model training stage and the damage identification stage. these two stages will be presented below. 3.1 architecture of the training stage the training stage aims to introduce the necessary steps to obtain a trained model that will be used in the next stage, damage identification. figure 2 illustrates the architecture of the training stage. details of all these steps will be shown as follows. fig. 2. proposed method architecture training stage full size image images dataset. photographs of damaged wind blades were made available by a company that provides technical support to several wind farms. table 1 summarizes the quantity, minimum and maximum resolution in pixel of the images by damage type. the photographs have varying sizes and a large area of image not related to damage, with the sky as a background or areas that do not have damage and some have text on the image indicating the date and time of the photo. table 1. number of images made available by damage typefull size table cropping and labeling images. the dataset used in this paper contains high-resolution images in order to better capture the damage details. since the pre-trained general purpose models used in this work require much smaller size images as input, it could make some of the smaller damages indistinguishable. instead, we decided to crop the high-resolution images and tackle the problem of damage classification in each of the crops. the original image is received as input. then, to define the position of the cuts in the image, a grid is calculated that serves as a guide for making the cuts. in order to prevent damage to the division of the grids, we define an overlap of the generated grids. thus, three parameters are necessary for generating the grids: the image size, the grid size and the overlap size. the overlap size parameter defines the minimum value to be considered. if the image size is not divisible by the size of the overlapping grids, the algorithm adds another set of grids (horizontal or vertical) and increases the size of the overlap to achieve a more adequate adjustment of the grids. using this approach, we build an images dataset containing 1,828 crops. all crops generated were contextually analyzed by a wind blade damage specialist, appointed by the wind turbine manufacturing company. each crops received a label indicating which class the image belongs to. the final dataset includes 1,243 normal, 195 lightning, 151 erosion, 51 cosmetic, and 188 crack images. in fig. 3 we present the cropping and labeling procedure. the original image that was passed as an input parameter is displayed in fig. 3a. figure 3b shows several crops generated from the grid with overlap. and finally, fig. 3c shows the red highlights crops labeled “lightning damage” and non highlighted crops labeled “normal”. fig. 3. cropping and labeling procedure: a input image; b generation of crops from defined grids; c highlight the crops with lightning damage labels after specialist labeling full size image datasets. the new labeled crops were arrange in a dataset used to design the classification models. we reserve 25% of the crops (457 images) for the validation set. the test set contains 15% (206 images) and the remainder (1,165 images) are used for the training set. all sets are stratified by available classes. in the model training stage, detailed in sect. 3.1, the training and test data sets are separated with different samples in each of the training repetitions. there is a large imbalance between the classes, with images from the normal class having the largest number of samples. thus, a new dataset is created, applying algorithms for class balancing and data augmentation were applied using images transformations. the balancing method used is oversampling with data-level methods [20]. the data augmentation process applies methods from the single-image model-free approach, as defined by [21]. the following transformation functions are applied: vertical flip, horizontal flip, brightness adjustment with a random \(\delta \in [-0.10, 0.10]\) and contrast adjustment with a random factor value in [0.7, 1.3]. the brightness delta value is added to each color channel of the image, this causes the brightness of the image to change. the contrast factor is also applied to each color channel of the image. each of these functions has a 30% probability of being applied, and for each image, at least one of the functions, randomly chosen, will have the probability of being applied increased to 100%. that is, at least one transformation function will always be applied to the image. models. the model is built using the framework tensorflow 2.12.1 which is a free and open source software tool for machine learning and artificial intelligence focused mainly on deep learning. the transfer learning process based on progressive learning [22] is used to optimize model training time. four pre-trained models from the imagenet [23] dataset were used, table 2 lists these models. the pre-trained layers are frozen and a new output layer is added with softmax activation for classifying the five defined classes. then, this last layer of the models is refined using the two sets of data generated: with and without balancing and augmentation. each of the four initial models generates two refined models, resulting in a total of eight models built. table 2. list of pre-trained models usedfull size table metrics. one of the initial problems faced by a specialist in damage to wind turbine blades is having to analyze thousands of photographs produced by technical teams in the field, with only a minority of the images showing any damage that requires in-depth analysis. to facilitate this specialist’s analysis, it is necessary to select images with damage and provide a prior classification of the damage contained in the image. therefore, it is very important that the model can precisely identify whether or not an image has damage. the metrics analyzed in the model construction process are: accuracy, recall and precision. models are optimized to reduce cross-entropy loss. the authors decided to implement 5 training realizations (repetitions) for each model. the idea is to evaluate the robustness of the models to the initialization of the weights of the last layer based on the analysis of the mean and standard deviation of the adopted metrics. for each of the five training repetitions, metrics are calculated using the test data set and evaluated in the classification of normal or damage classes (lightning, erosion, cosmetics or cracking). then, the choice of the best repetition is made by measuring the precision metric. this choice is motivated by the goal of using the model to select only the photos with damage, which will then be reviewed by an expert. therefore, it is important to minimize the number of false negatives, which occur when precision is high. a false negative occurs when images with damage are classified as normal. on the other hand, a false positive occurs when normal images are classified as damaged. the indication of absence of failure when there is an established or ongoing failure can have catastrophic results. as a tiebreaker criterion, the accuracy metric is considered first, followed by recall. once the best iteration of each model is chosen, the models are evaluated again using the validation set. finally, the model with the best precision, according to the selection criteria, is evaluated in detail by type of damage (normal, lightning damage, erosion damage, cosmetics damage or crack damage). 3.2 architecture of the damage identification stage the previous best model is used in the damage identification stage, as illustrated in fig. 4. the steps in fig. 4a–d were detailed in the previous sections, while fig. 4e–g will be detailed in the following sections. fig. 4. proposed method architecture damage identification stage full size image classification of original images. the classification models are trained and evaluated using the various crops generated from the original images. the crops are classified using the model with the best generalization and, finally, a vote is applied on the crops results to define a unique class for each original image where they belong. in this analysis, damage predictions have priority over the normal class prediction, i.e., if all crops for an original image have the normal prediction with only one crop predicted as erosion (for example), the original image is classified as erosion. this process is used to quickly screen images containing damage. locating image damage. the classification of the crops also allows mapping the damage in the input blade image. in the classification process, the output layer of the best model generates normalized values of the neuron outputs. as a result, the sum of outputs is equal to 1. this value is used as a class intensity for each crop. therefore, it is possible to locate the damage in the original image using a heat map of each class. this damage localization process is illustrated in fig. 5. an image of atmospheric discharge damage is shown in fig. 5a. the region is highlighted. the cutting guide grid is shown in blue sky color. the region is classified as lighting with 0.9938. in fig. 5b a heat map is displayed using the intensity value of each crop for the lightning class. the closer to 1.00, the redder the crop area becomes. the closer to 0.00, the bluer the crop appears. the final result of the lightning damage location is presented in fig. 5c. fig. 5. damage localization. a crops classification result (values highlighted for n:normal, d:lightning, e:erosion, p:cosmetic and t:crack); b location based on crops classification result; c lightning damage localization. full size image 4 experimental results 4.1 training models all trained models use the following parameters: a learning rate of 0.001, 200 epochs, the adam optimizer, gpu parallelism, and callbacks. the callbacks used are as follows: a 20% reduction in the learning rate when a plateau is reached in the test set loss metric for 5 epochs, with a minimum learning rate limit of 0.0001; and early stopping of training when the test set loss metric does not decrease after 20 epochs, with the best weights found being restored. the synchronous distributed training strategy is used on two gpus on the same machine. the models were trained using 5 repetitions, with the training and test sets being regenerated with new samples each time to evaluate robustness. the training metrics, evaluated on the test set, were used to determine the best model for evaluation on the validation set. 4.2 test dataset results results of the models evaluated on the test set for normal and damage classes are detailed in table 3. the best average values are highlighted in bold. the suffix ‘de’ indicates that the model is trained using the unbalanced training set. the suffix ‘ba’ indicates that training is carried out on the balanced training set, with the application of data augmentation. table 3. model results in the test set for classifying normal or damaged cropsfull size table the resnet50-de model, trained with the unbalanced dataset, has the highest average values in the precision (0.990) and accuracy (0.979) metrics, and the efficientnetb2-de model has the best result in recall (0.992). considering the standard deviation for the precision metric, there is a tie with the models efficientnetb2-de (0.976±0.013), efficientnetb2-ba (0.984±0.011), and vgg16-ba (0.981±0.013). comparing the balanced and unbalanced data for each backbone, we observe that the balanced data perform better in the precision metric in three out of four backbones (efficientnetb2-ba, mobilenetv2-ba, and vgg16-ba). the behavior reverses when analyzing the accuracy and recall metrics, with the unbalanced data performing better in most cases. 4.3 validation dataset results the best result among the repetitions for each model is selected for evaluation on the validation set. table 4 summarizes the results obtained based on the classification of the normal and damage classes evaluated on the validation set. table 4. results of the best repetitions of each model for classifying normal or damaged crops, applied to the validation setfull size table the results show that the models achieving the best generalizations are efficientnetb2-ba and vgg16-ba, both with a precision of 0.993. the best accuracy (0.982) and recall (0.997) results are achieved by the mobilenetv2-de model. models based on balanced data have better precision values in three cases (efficientnetb2-ba, mobilenetv2-ba, and vgg16-ba) and a tie (resnet50-de and resnet50-ba). according to the tie-breaking criterion indicated in sect. 3.1, the best model found is efficientnetb2-ba. 4.4 input images result the model with the best generalization, efficientnetb2-ba, is used to classify all generated crops, and voting is applied, as described in sect. 3.2, to determine a single class for each original image. figure 6 illustrates the final classification results of the original images, showing an average precision of 96.1%, an average recall of 94.9%, and an overall accuracy of 97.0%. the average is calculated by applying the metrics individually to each class and then computing an unweighted average of these values. fig. 6. confusion matrix of the input images full size image 5 conclusions in this work, we present a method to identify wind turbine blade damages in one of four types: lightning, erosion, cosmetics damage, or crack. the results show that among the eight trained models, the efficientnetb2-ba model achieved the best generalization performance for classifying crops of the original images into normal or damaged classes, with a precision of 99.3%, as well as excellent performance in detailed classification of the classes (lightning, erosion, normal, cosmetics, and cracks). when applied to the classification of the provided original images, we obtained an overall precision of 96.1%, accuracy of 97%, and recall of 94.5%. as evidenced by the results, the proposal can assist in image screening and analysis tasks, providing a tool for the specialist to identify damage in wind blades. in future work, we aim to enhance the precision of damage localization, using the quadtree technique. with this method, we anticipate achieving more precise localization of damaged areas, thereby improving the computing time and overall detection process. additionally, we plan to enhance the visualization of the damage location through the implementation of a salience map. this will provide a more intuitive and detailed representation of the damage, facilitating better analysis and interpretation. these advancements are expected to contribute to the effectiveness of image-based damage assessment methodologies. references farias, l.m., sellitto, m.a.: uso da energia ao longo da história: evolução e perspectivas futuras. revista liberato 12(17), 07–16 (2011) article  google scholar  lage, e.s., processi, l.d.: panorama do setor de energia eólica. banco nacional de desenvolvimento econômico e social (2013) google scholar  mishnaevsky, l., branner, k., petersen, h.n., beauson, j., mcgugan, m., sørensen, b.f.: materials for wind turbine blades: an overview. materials 10(11), 1285 (2017) google scholar  yang, z., et al.: detection of wind turbine blade abnormalities through a deep learning model integrating vae and neural ode. ocean eng. 302, 117689 (2024) google scholar  mishnaevsky, j.r.l.: root causes and mechanisms of failure of wind turbine blades: overview. materials 15(9), 2959 (2022) google scholar  du, y., zhou, s., jing, x., peng, y., wu, h., kwok, n.: damage detection techniques for wind turbine blades: a review. mech. syst. signal process. 141, 106445 (2020) article  math  google scholar  kaewniam, p., cao, m., alkayem, n.f., li, d.m.e.: recent advances in damage detection of wind turbine blades: a state-of-the-art review. renew. sustain. energy rev. 167, 112723 (2022) google scholar  wang, w., xue, y., he, c., zhao, y.: review of the typical damage and damage-detection methods of large wind turbine blades. energies 15(15), 5672 (2022) article  math  google scholar  kaewniam, p., cao, m., alkayem, n.f., li, d.m.e.: recent advances in damage detection of wind turbine blades: a state-of-the-art review. renew. sustain. energy rev. 167, 112723 (2022) google scholar  moreno, s., peña, m., toledo, a., treviño, r., ponce, h.: a new vision-based method using deep learning for damage inspection in wind turbine blades. in: ieee 2018 15th international conference on electrical engineering, computing science and automatic control (cce), pp. 1–5. ieee, mexico city (2018) google scholar  yang, p., dong, c., zhao, x., chen, x.: the surface damage identifications of wind turbine blades based on resnet50 algorithm. in: ieee 2020 39th chinese control conference (ccc), pp. 6340–6344. ieee, shenyang (2020) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition (cvpr), pp. 770–778. ieee, las vegas (2016) google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. in: advances in neural information processing systems, vol. 25 (2012) google scholar  shihavuddin, a.s.m., et al.: wind turbine surface damage detection by deep learning aided drone inspection analysis. energies 12(4), 676 (2019) google scholar  ren, s., he, k., girshick, r., sun, j.: faster r-cnn: towards real-time object detection with region proposal networks. adv. neural inf. process. syst. 28 (2015) google scholar  shihavuddin, a.s.m., chen, x.: dtu drone inspection images of wind turbine. mendeley data v2 (2018). https://doi.org/10.17632/hd96prn3nc.2 foster, a., best, o., gianni, m., khan, a., collins, k., sharma, s.: drone footage wind turbine surface damage detection. in: 2022 ieee 14th image, video, and multidimensional signal processing workshop (ivmsp), pp. 1–5. ieee, piscataway (2022) google scholar  redmon, j., divvala, s., girshick, r., farhadi, a.: you only look once: unified, real-time object detection. in: proceedings of the ieee conference on computer vision and pattern recognition (cvpr), pp. 779–788. ieee, las vegas (2016) google scholar  garolera, a.c., madsen, s.f., nissim, m., myers, j.d., holboell, j.: lightning damage to wind turbine blades from wind farms in the us. ieee trans. power deliv. 31(3), 1043–1049 (2014) article  google scholar  buda, m., maki, a., mazurowski, m.a.: a systematic study of the class imbalance problem in convolutional neural networks. neural netw. 106, 249–259 (2018) article  math  google scholar  xu, m., yoon, s., fuentes, a., park, d.s.: a comprehensive survey of image augmentation techniques for deep learning. pattern recogn. 2023, 109347 (2023) google scholar  iman, m., arabnia, h.r., rasheed, k.: a review of deep transfer learning and recent advancements. technol. mdpi 11(2), 40 (2023) math  google scholar  russakovsky, o., et al.: imagenet large scale visual recognition challenge. int. j. comput. vision 115, 211–252 (2015) google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: pmlr international conference on machine learning, pp. 6105–6114. long beach, california (2019) google scholar  sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l. c.: mobilenetv2: inverted residuals and linear bottlenecks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4510–4520. ieee, salt lake city (2018) google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556 (2014) download references acknowledgments the authors are grateful to talisson araujo figueiredo for supporting with data labeling process. their expertise significantly contributed to the quality and rigor of this research. without their support, this work would not have been possible. author information authors and affiliations instituto federal de educação ciência e tecnologia do ceará (ifce), ceará, brazil alberto régio alves de oliveira, cláudio marques de sá medeiros & geraldo luis bezerra ramalho authors alberto régio alves de oliveiraview author publications search author on:pubmed google scholar cláudio marques de sá medeirosview author publications search author on:pubmed google scholar geraldo luis bezerra ramalhoview author publications search author on:pubmed google scholar corresponding author correspondence to alberto régio alves de oliveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de oliveira, a.r.a., de sá medeiros, c.m., ramalho, g.l.b. (2025). damage identification of wind turbine blades. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_13 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computer vision deep learning wind blade damages publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature machine learning and time series analysis to forecast hotel room prices | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper machine learning and time series analysis to forecast hotel room prices conference paper first online: 30 january 2025 pp 358–371 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) machine learning and time series analysis to forecast hotel room prices download book pdf download book epub francisco b. oliveira9, moesio w. silva-filho10, gabriel a. barbosa10, joão paulo freitas9, chris penna9 & … péricles b. c. miranda10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 436 accesses 2 citations abstract the hospitality industry’s dynamic nature demands accurate forecasting of hotel room prices to optimize revenue management strategies. this paper presents an experimental study assessing machine learning techniques and time series analysis for forecasting hotel room prices. we enhance prediction accuracy by leveraging historical booking data, seasonal patterns, and hotel characteristics. we employ time series models, including autoregressors and prophet, to capture underlying trends and seasonal variations. we also evaluate machine learning models such as random forest, gradient boosting machine, extra trees regressor, and neural networks. these models are trained on features like booking lead time, historical hotel occupancy, room nights, and number of adults to capture complex relationships influencing prices. our methodology is demonstrated through a case study using over 40,000 reservations from a brazilian hotel over a decade. experimental results show that tree-based models performed best, with the gradient boosting achieving 6.94% of nrmse and 31.26 of mae. our findings contribute valuable insights for price estimation in the hospitality industry, offering robust methods to enhance revenue management strategies. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a machine learning model for occupancy rates and demand forecasting in the hospitality industry chapter © 2016 forecasting hotel reservations with long short-term memory-based recurrent neural networks article 04 december 2018 a study of machine learning based approach for hotels’ matching chapter © 2022 1 introduction the dynamic and competitive nature of the hospitality industry necessitates effective strategies for forecasting hotel room prices to optimize revenue management. accurate predictions are crucial for hotel operators to make informed decisions, respond to market trends, and maximize profitability. in this context, machine learning and time series analysis offer promising approaches, capturing complex patterns in hotel pricing data. time series analysis has long been employed to model and understand temporal patterns in various fields, including hospitality [2, 5, 16]. techniques such as autoregressive integrated moving average (arima) and seasonal decomposition of time series (stl) have been instrumental in capturing inherent trends and seasonal variations in hotel room prices [7, 10]. in parallel, the advancement of machine learning techniques has provided opportunities to enhance forecasting accuracy by leveraging the power of algorithms that can capture intricate relationships in large and diverse datasets [4, 12]. models such as random forests, gradient boosting, and neural networks have shown promise in capturing non-linear patterns and incorporating a wide range of features, including booking lead time, historical demand, and external factors like local events and economic indicators [2, 4, 7, 20]. this work presents an empirical investigation that examines the application of machine learning techniques and time series analysis to forecast hotel room prices. the proposed methodology exploits historical booking data, seasonal patterns, and external factors to augment the accuracy of price predictions. the investigation commences by scrutinizing the utilization of time series analysis to model and comprehend the temporal patterns intrinsic to hotel room pricing. different time series models are deployed to capture underlying trends and seasonal fluctuations in pricing data. additionally, machine learning models, namely random forests, gradient boosting, and neural networks, are subjected to evaluation. these models are trained on a comprehensive set of features, encompassing booking lead time, historical demand, local events, and economic indicators, encapsulating the intricate relationships influencing hotel room prices. the efficacy of the proposed methodology is exemplified through a case study employing authentic hotel pricing data. experimental results show that tree-based models, like gradient boosting machine, demonstrate the best results. besides, we present a discussion of the practical implications of the achieved results for hotel revenue managers, emphasizing their potential to optimize pricing strategies, enhance revenue streams, and improve overall profitability. insights derived from the study can inform decision-making processes and help hotel operators stay competitive in a rapidly evolving market. the paper is structured as follows. section 2 introduces the problem of forecasting room prices in the hospitality domain and the main research related to the theme. section 3 details the proposed methodology to assess forecasting and ml algorithms for the problem at hand. section 4 discusses the achieved results and their implications for hotel revenue managers. section 5 presents the conclusion and future work. 2 background forecasting poses a significant challenge for hotels, particularly those with limited resources that prevent them from investing in advanced revenue management software [18]. according to [18], less than ten percent of hotels currently utilize a revenue management system, and even when implemented, these systems often need more sophistication than those observed in other industries. complicating matters further, the data about a guest’s spa bookings is typically stored in a separate database from information related to room bookings and/or email marketing efforts. this fragmentation hampers hotels’ capacity to assess customer lifetime value, a capability readily available in the airline industry [9]. the pivotal role of revenue managers in shaping hotel pricing is undeniably significant; however, it introduces the potential for customers to perceive the pricing strategies as inequitable [14]. when hotels establish prices perceived as unrealistic or unfair by consumers, there is a consequential shift in demand towards alternative accommodation options such as peer-to-peer arrangements [14]. consumers may gravitate towards peer-to-peer options, perceiving them as more cost-effective alternatives [6]. despite these challenges, many hotels employ conventional price forecasting models, including the naive (same-time/last year) method [15]. furthermore, it is noteworthy that forecasted prices often undergo adjustments, commonly overridden through qualitative methods rather than quantitative pricing models [13]. in forecasting, several studies have employed moving average and exponential smoothing methods [11]. however, a significant drawback associated with moving average models is their sluggish responsiveness to rapid fluctuations in the data [11]. these models exhibit limited adaptability to dynamic changes in pricing, often neglecting the intricate relationships inherent in hotel data [2]. moreover, the moving average method demonstrates sub-optimal performance in addressing price fluctuations and fails to adequately account for the seasonal nature of hotel operations, which inherently exhibit such patterns. in response to these limitations, recent hotel revenue management studies have increasingly turned to arima-based models [21]. arima models, while proficient in modeling trends, exhibit limitations in capturing the complex nonlinear nature of data, as highlighted by [22]. their effectiveness diminishes when confronted with sudden changes in data values, resulting in notable increases in forecast error [22]. additionally, arima models are sensitive to outliers and lack robustness in predicting extreme values, as observed in [2]. identifying an appropriate arima model can be challenging and often computationally demanding. furthermore, arima-based models overlook the intricate nonlinear aspects of data, assuming data stability and disregarding external factors [22]. given the vulnerability of the hotel industry to external influences such as economic changes and pandemics, this oversight is particularly relevant [22]. traditional models like arima prove ineffective in capturing turning points, leading to significant errors in forecasting [22]. given the dynamic nature of the hospitality industry, pricing assumes a pivotal role in the operations and survival of hotels [4, 19]. consequently, there is an imperative need to develop new forecasting methods (e.g., machine learning algorithms) to effectively navigate the continual changes and challenges inherent in the industry. machine learning algorithms are good at detecting the complex nature of data and any sudden changes [1, 4]. they can also be trained on rich features, including booking lead time, historical demand, local events, and economic indicators, to capture the complex relationships influencing hotel room prices [1, 4]. besides, machine learning models are inherently designed to operate efficiently with extensive datasets, although they also exhibit proficiency in handling smaller datasets. moreover, their capacity to manage data noise is noteworthy, as it can be mitigated through optimization strategies [21]. researchers have demonstrated substantial enhancements in prediction accuracy by employing machine learning models, since traditional ones (e.g., support vector machines, random forest, artificial neural networks) to sophisticated ones (e.g., deep learning, long short-term memory network lstm), compared to results obtained from statistical models considering the hospitality accommodation price [1, 4, 5, 22]. the contribution of this work is an experimental investigation of traditional forecasting methods (e.g., arima, sarima) and machine learning algorithms applied to hospitality accommodation price prediction problem. the study’s initial phase involves examining time series analysis to model and comprehend the inherent temporal patterns in hotel room pricing. diverse time series models are employed to capture underlying trends and seasonal variations in pricing data. furthermore, various machine learning models are trained on a comprehensive set of features to comprehensively capture the intricate relationships influencing hotel room prices. the efficacy of the proposed methodology is substantiated through a case study employing authentic hotel pricing data from a real-world scenario. 3 experimental methodology this section details the experimental methodology proposed to assess the machine learning and forecasting algorithms for price prediction. thus, we present the datasets adopted, the algorithms involved in the experiments, how they were evaluated and the software and hardware setup used. 3.1 dataset the dataset used in this study is called hospbr. it originates from the property management system (pms) of a brazilian leisure hotel, anonymized to ensure the privacy of its guests and owners. this dataset contains 41,915 reservations made between 2010 and 2019. the data was split into training and testing periods, with the training set covering 2010 to 2018, consisting of 36,988 reservations, and the testing set covering 2019 with 4,927 reservations. we chose the dataset features that were consistently available throughout the investigated period; they are outlined in table 1. table 1. description of featuresfull size table several preprocessing steps were conducted on the original data. first, only confirmed reservations were selected, excluding those marked as “canceled” or “pending.” next, missing values in the feature “id type hu” were replaced with 0, indicating unrecorded data for those reservations. subsequently, the “occupation” information was included, defined as the ratio of occupied hotel units (hu) on the reservation creation date (entry date lead time) to the hotel’s inventory (available rooms). hotel inventory typically shows little variability, as modifying the number of hus is not straightforward, although conversions between sophisticated and simpler units can occur. finally, data cleaning was performed on the daily rate values (fee), which exhibited considerable noise due to manual adjustments by the hotel staff. a variable named “mean fee” was created, representing the daily rate (fee divided by length of stay, los). extreme outliers were removed, defined as values exceeding the third quartile plus fifteen times the interquartile range (iqr). thus, reservations with a mean fee above 853.57 brl, totaling 128 reservations (0.3% of the dataset), were excluded. similarly, reservations with a mean fee below the first quartile minus one iqr were also excluded, amounting to 22 reservations (0.05% of the dataset) below 16.69 brl. table 2. statistical summary of featuresfull size table table 2 displays the distribution of the available data within the reservations. most fees ranged between 292 and 630 brl, influenced by inflation over a decade. the dataset contains outliers, with some fees being exceptionally high, possibly due to extremely long los. most reservations were made less than a month in advance, indicating last-minute booking behavior among guests. the “id hu” information, representing room numbers, is a proxy for room quality and includes the floor level. the “id type hu” feature directly indicates the room type but includes many zeros where this information was not recorded. typically, the hotel accommodates two adults without children; on average, it operates at 50% of occupancy. the data shows some inconsistencies, with certain dates exceeding 100% of occupancy, suggesting incorrect reservation dates or inventory data errors. we employed the python library pycaret [3] for feature engineering using the original dataset. this process involved (i) one-hot encoding the “id type hu” variable; (ii) decomposing the “entry date” variable into numerical features representing the day of the month, day of the week, month, and year; (iii) generating polynomial features by squaring numerical features and multiplying pairs of numerical features, and (iv) removing features with low variance or high multicollinearity. the result is a set of 71 features, which the machine learning models utilized. 3.2 ml and forecasting algorithms in this study, we aimed to employ a wide variety of machine learning techniques to automatically predict the daily rate for a given reservation. these strategies encompass supervised regression, deep neural networks, and time series forecasting. regressor models. for supervised regression techniques, we focused on implementations from the python library scikit-learn, specifically the extratrees regressor (model name: et), random forest (model name: rf), gradient boosting machine (model name: gbm) models, and linear regression (model name: lr). the processed dataset, prepared using the pycaret library, was utilized for all cases. hyperparameter optimization was conducted on tree-based models using the optuna library, facilitating automated hyperparameter tuning through efficient search algorithms. the optimization process involved exploring the designated hyperparameter space for each model (see table 3), with 25 trials per case, to identify the model achieving the best performance on a validation subset randomly extracted from 30% of the training data. the optimal model was then used to predict the test dataset, comprising reservations from 2019, and the model’s performance was evaluated using various metrics. this procedure was repeated ten times for each model to analyze the mean and standard deviation of the results. table 3. hyperparameter space for regressor modelsfull size table multi-layer perceptron. we trained a neural network (model name: dnn) using a sequence of linear layers followed by relu activations to predict hotel room rates based on reservation data. the model was implemented with the pytorch library in python. it consisted of four hidden layers with dimensions 32, 64, 32, and 16, respectively, and was trained with a batch size of 128 for 64 epochs. the data was normalized using minmax scaling based on the train set. this neural network was designed to be simple, aiming to assess the capability of such a model to identify patterns necessary for accurate predictions. the hyperparameters were configured based on the authors’ experience with similar problems. time-series forecasting. the prediction of hotel room rates in time series data has unique challenges due to the wide range of rates for each reservation date. for instance, guests booking well in advance typically pay lower rates than those booking closer to the date. similarly, the price can vary based on the number of nights reserved or the number of adults. to isolate these effects, we calculated the average daily rate by dividing the total rate for a given date by the number of reservations and adults for that date. this provided the average rate per day per adult. these values formed a time series from 2010 to 2018, which we used to predict rates for 2019. to predict the rate for a specific reservation, we summed the predicted daily rates for all days covered by the reservation and multiplied by the number of adults. for example, reservation 96384 had three adults, starting on 2019-02-03 and ending on 2019-02-05. one model predicted rates of 69.82 for 2019-02-03, 69.31 for 2019-02-04, and 68.68 for 2019-02-05. thus, the final predicted cost was (69.82 + 69.31 + 68.68) * 3 = 705.96, while the actual cost was 705. we tested two-time series models: the autoregressor from the python statsmodels library with lags ranging from 30, 60, 180, and 365 (model names: ar_30, ar_60, ar_180, ar_365, respectively) and the prophet library (model name: prophet) [17]. 3.3 evaluation methodology all algorithms, machine learning and forecasting ones, were evaluated following a hold-out methodology, considering the train and test splits, to predict the average daily price for a given booking. the experiments were executed on vertex ai workbench, in a machine with a cpu intel xeon (8 vcpus) @ 2.800ghz and 32gb ram. the metrics adopted to assess the algorithms are presented as follows: bias. refers to the systematic error or tendency of a forecasting method to consistently overestimate or underestimate the actual values. it indicates the direction and magnitude of the forecast errors. a positive bias implies that the forecasts are consistently higher than the actual outcomes, while a negative bias indicates that the forecasts consistently fall below the actual values. bias equation can be seen as follows: $$\begin{aligned} \text {bias} = \frac{1}{n} \sum _{t=1}^{n} (f_t a_t), \end{aligned}$$ (1) where, n is the number of observations, \(f_{t}\) is the forecasted value at time t, and \(a_{t}\) is the actual observed value at time t. it is important to note that a perfect forecast would have a bias equal to zero, indicating that, on average, the forecasts are accurate. nrmse. normalized root mean squared error, is a metric used to assess the accuracy of forecasting models. it is a normalized version of the root mean squared error (rmse), which measures the square root of the average squared differences between forecasted and actual values. normalization is applied to the error by dividing it by the larger value of a reservation, making it a relative measure. the nrmse formula is as follows: $$\begin{aligned} \text {nrmse} = \sqrt{\frac{1}{n}\sum _{t=1}^{n}\left( \frac{f_t a_t}{\max (a_t)}\right) ^2} \times 100, \end{aligned}$$ (2) a lower nrmse indicates better model accuracy. nrmape. normalized root mean absolute percentage error, is a metric used to evaluate the accuracy of predictive models, particularly in time series forecasting and regression analysis. it combines elements of both the mean absolute percentage error (mape) and the root mean squared error (rmse), and it is normalized to provide a scale-independent measure of error. $$\begin{aligned} \text {nrmape} = \sqrt{\frac{1}{n} \sum _{i=1}^{n} \left| \frac{f_t a_t}{\max (a_t)}\right| } \times 100. \end{aligned}$$ (3) a lower nrmape indicates better model accuracy. \(r^{2}\) the coefficient of determination, is a metric used to assess the goodness-of-fit of a forecasting model. it represents the proportion of the variance in the dependent variable (actual values) that is explained by the independent variable (forecasted values). \(r^{2}\) values range from 0 to 1, with 1 indicating a perfect fit where all variability in the dependent variable is explained by the model. the \(r^{2}\) formula is given by: $$\begin{aligned} r^2 = 1 \frac{\sum _{t=1}^{n}(a_t f_t)^2}{\sum _{t=1}^{n}(a_t \bar{a})^2}, \end{aligned}$$ (4) where, \(\overline{a}\) is the mean of the actual values. mae. mean absolute error, is a metric used to measure the average magnitude of errors in a set of predictions, without considering their direction. it is the average over the test sample of the absolute differences between predicted values and observed actual values. the mae is a linear score, meaning that all individual differences are weighted equally in the average. we included this metric to show a perspective at the same scale of the reservation fee values. the mae formula is as follows: $$\begin{aligned} \text {mae} = \frac{1}{n} \sum _{t=1}^{n} |f_t a_t|, \end{aligned}$$ (5) a lower mae indicates a model with better predictive accuracy. 4 results and discussion table 4 presents the accuracy rates achieved by each model regarding bias, nrmse, nrmape, \(r^{2}\) and mae. bold values mean the best value achieved in the given metric in an absolute sense. as it can be seen, lr reached the best value for bias. for nrmse, nrmape, \(r^{2}\) and mae, gb was the best algorithm, achieving 6.94, 3.9629, 0.4778 and 31.26, respectively. table 4. mean of accuracy rates of the predictive models.full size table furthermore, a statistical analysis was conducted on the gathered data to ensure a fair assessment of the models’ performance. the null hypothesis posits that there is no statistical discrepancy between the average metrics collected from the models within each dataset. we opted to employ the non-parametric friedman test with a significance level of \(\alpha = 0.05\) to verify the null hypothesis. this approach yielded the p-values for each metric across all datasets. these p-values were less than \(\alpha \), indicating rejection of the null hypothesis across all tests and metrics. this rejection signifies that, within each metric and dataset, at least one model exhibits statistically significant differences from the others, forming distinct result groups. fig. 1. critical difference diagrams for friedman-nemenyi statistical test. full size image table 5. rank of algorithms.full size table we utilized the nemenyi post hoc test to identify statistically similar result groups, computed average rankings, and created critical difference graphs, depicted in fig. 1. each vertical line in these graphs represents a tested model, while horizontal bars touching these lines denote statistical similarity. vertical lines intersected by horizontal bars within the calculated critical difference (cd) value denote models that achieved statistically equivalent performance. figure 1 (top-left) shows the cd diagram for bias. as it can be seen, lr, prophet, \(a\_365\), dnn and \(a\_180\) reached the first five positions, respectively, and are statistically equivalent. however, the lr was superior statistically to all the other algorithms. figure 1 (top-right) presents the cd diagram for nrmape. gb, rf, et, \(a\_30\), and \(a\_60\) tied statistically as the best models, where gb surpassed all other approaches. gb also reached the best value of nrmse and \(r^{2}\), fig. 1 (middle-left and -right), being statistically equivalent to et, rf, prophet, and \(ar\_365\). gb overcame all the other models in both metrics. gb also achieved the best results for mae, being statistically equivalent to rf, et, \(ar\_{30}\), and \(ar\_{60}\). to complement the results presented previously in cd diagrams, table 5 presents the average rank of the algorithms considering all the metrics adopted. the average rank facilitates the understanding of the general performance of the algorithms. as example, gb was the \(8^{th}\) best algorithm regarding bias, and \(1^{st}\) for nrmape, nrmse, \(r^{2}\) and mae. thus, gb has an average rank equal to 2.40, being considered, on average, the best algorithm. rf, et and \(ar\_{30}\) composed the podium of the four best algorithms for the dataset considered in this work. fig. 2. histogram of residuals for predicted values for each model. full size image the results indicate that tree-based models (gb, rf, et) achieved the best performance in metrics evaluating predictive model performance, namely nrmape, nrmse, \(r^2\), and mae. moreover, the tests conducted suggest that there is no statistically significant difference in the results of these models, although gb numerically outperformed the others in all cases. this suggests that models with this architecture are better suited to capture the nuances governing price configurations in hotel reservation systems. even after normalizing values to account for the number of nights and adults in the reservation, time series prediction models did not perform comparably to tree-based models. the influence of booking window information and the type of room, which these models could not capture, likely affected the outcome negatively. additionally, it is noteworthy that tree-based models exhibited high bias values, all of which were negative. this may indicate that these models failed to identify a trend in daily rates, unlike models such as lr, prophet, and \(ar\_365\). this could be attributed to the long training period, during which the influence of brazilian inflation, totaling 75.78% [8] between 2010 and 2018, may have been observed. future research should consider normalizing data to account for this factor. figure 2 depicts each model’s prediction residual distribution. it is evident that tree-based models are notably centered around zero, highlighting the quality of their predictions. however, they exhibit leftward skewness in all cases, indicating a negative bias in these models. 5 conclusion and future works this paper presents an experimental study that evaluates machine learning techniques and time series analysis for forecasting hotel room prices. the methodology utilizes a dataset (hospbr) comprising 41,915 reservations made at a brazilian hotel from 2010 to 2019. each reservation includes check-in and check-out dates, number of adults, booking window (days between reservation creation and check-in), room nights (days between check-out and check-in), room type, and hotel occupancy rate on the reservation date. time series forecasting models, such as autoregressors and prophet, are employed to understand temporal patterns in hotel pricing. machine learning models are also assessed, including random forest, gradient boosting machine, extra trees regressor, and neural networks. the reservations from 2010 to 2018 are used for training, while those from 2019 are used for testing. our findings indicate that the gradient boosting machine (gbm) demonstrated the highest predictive power, closely followed by tree-based models (extratrees and random forest) and the prophet library. this suggests that tree-based models effectively capture the relationships between variables affecting price variations, such as room nights and booking windows. the gbm achieved an nrmse of 6.94%, nrmape of 3.96%, \(r^2\) of 0.47, and mae of 31.26. however, tree-based models also showed high negative bias values, indicating a tendency to underestimate prices, likely due to the impact of brazilian inflation during the test period. in conclusion, this research contributes to hotel revenue management by providing a robust approach to forecasting room prices. by integrating time series analysis with machine learning, the methodology offers a comprehensive solution for anticipating price fluctuations, enabling proactive and data-driven decision-making in the hospitality industry. future work could explore the impact of inflation on prices, further investigate tree-based models like lightgbm or xgboost, and combine regressor and time series models such as prophet. another research avenue could be the study of dynamic pricing strategies to optimize hotel revenue. references al shehhi, m., karathanasopoulos, a.: forecasting hotel prices in selected middle east and north africa region (mena) cities with new forecasting tools. theor. econ. lett. 8(9), 1623–1638 (2018) article  math  google scholar  al shehhi, m., karathanasopoulos, a.: forecasting hotel room prices in selected gcc cities using deep learning. j. hosp. tour. manag. 42, 40–50 (2020) article  google scholar  ali, m.: pycaret: an open source, low-code machine learning library in python (2020). https://github.com/pycaret/pycaret. accessed 23 june 2024 alotaibi, e.: application of machine learning in the hotel industry: a critical review. j. assoc. arab univ. tourism hosp. 18(3), 78–96 (2020) math  google scholar  binesh, f., belarmino, a.m., van der rest, j.p., singh, a.k., raab, c.: forecasting hotel room prices when entering turbulent times: a game-theoretic artificial neural network model. int. j. contemp. hosp. manag. (2023) google scholar  chi, m., wang, j., luo, x., li, h.: why travelers switch to the sharing accommodation platforms? a push-pull-mooring framework. int. j. contemp. hosp. manag. 33(12), 4286–4310 (2021) article  math  google scholar  cleveland, r.b., cleveland, w.s., mcrae, j.e., terpenning, i.: stl: a seasonal-trend decomposition. j. off. stat 6(1), 3–73 (1990) math  google scholar  de geografia e estatística, i.b.: índice nacional de preços ao consumidor amplo (ipca) (2023). https://www.ibge.gov.br/estatisticas/economicas/precos-e-custos/9256-indice-nacional-de-precos-ao-consumidor-amplo.html. accessed 23 june 2024 giousmpasoglou, c., marinakou, e., zopiatis, a.: hospitality managers in turbulent times: the covid-19 crisis. int. j. contemp. hosp. manag. 33(4), 1297–1318 (2021) article  google scholar  hyndman, r.j., athanasopoulos, g.: forecasting: principles and practice. otexts (2018) google scholar  kimes, s.e.: revenue management: a retrospective. cornell hotel restaurant adm. q. 44(5–6), 131–138 (2003) article  math  google scholar  knani, m., echchakoui, s., ladhari, r.: artificial intelligence in tourism and hospitality: bibliometric analysis and research agenda. int. j. hosp. manag. 107, 103317 (2022) article  math  google scholar  koupriouchina, l., van der rest, j.p., schwartz, z.: judgmental adjustments of algorithmic hotel occupancy forecasts: does user override frequency impact accuracy at different time horizons? tour. econ. 29(8), 2143–2164 (2023) article  google scholar  meatchi, s., camus, s., lecointre-erickson, d.: perceived unfairness of revenue management pricing: developing a measurement scale in the context of hospitality. int. j. contemp. hosp. manag. 33(10), 3157–3176 (2021) article  google scholar  pereira, l.n.: an introduction to helpful forecasting methods for hotel revenue management. int. j. hosp. manag. 58, 13–23 (2016) article  math  google scholar  salah, a., bekhit, m., eldesouky, e., ali, a., fathalla, a.: price prediction of seasonal items using time series analysis. comput. syst. sci. eng. (2023) google scholar  taylor, s.j., letham, b.: prophet: forecasting at scale (2017). https://github.com/facebook/prophet. accessed 23 june 2024 webb, t., schwartz, z., xiang, z., singal, m.: revenue management forecasting: the resiliency of advanced booking methods given dynamic booking windows. int. j. hosp. manag. 89, 102590 (2020) article  google scholar  zaki, k.: implementing dynamic revenue management in hotels during covid-19: value stream and wavelet coherence perspectives. int. j. contemp. hosp. manag. 34(5), 1768–1795 (2022) article  math  google scholar  zhang, b., huang, x., li, n., law, r.: a novel hybrid model for tourist volume forecasting incorporating search engine data. asia pacific j. tourism res. 22(3), 245–254 (2017) article  math  google scholar  zheng, t.: what caused the decrease in revpar during the recession? an arima with intervention analysis of room supply and market demand. int. j. contemp. hosp. manag. 26(8), 1225–1242 (2014) article  math  google scholar  zheng, t., liu, s., chen, z., qiao, y., law, r.: forecasting daily room rates on the basis of an lstm model in difficult times of hong kong: evidence from online distribution channels on the hotel industry. sustainability 12(18), 7334 (2020) article  math  google scholar  download references author information authors and affiliations cesar, recife, pernambuco, brazil francisco b. oliveira, joão paulo freitas & chris penna universidade federal rural de pernambuco, recife, brazil moesio w. silva-filho, gabriel a. barbosa & péricles b. c. miranda authors francisco b. oliveiraview author publications search author on:pubmed google scholar moesio w. silva-filhoview author publications search author on:pubmed google scholar gabriel a. barbosaview author publications search author on:pubmed google scholar joão paulo freitasview author publications search author on:pubmed google scholar chris pennaview author publications search author on:pubmed google scholar péricles b. c. mirandaview author publications search author on:pubmed google scholar corresponding author correspondence to péricles b. c. miranda . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper oliveira, f.b., silva-filho, m.w., barbosa, g.a., freitas, j.p., penna, c., miranda, p.b.c. (2025). machine learning and time series analysis to forecast hotel room prices. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_25 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hotel price forecasting time series analysis machine learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature on the analysis of cgp mutation operators when inferring gene regulatory networks using scrna-seq time series data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper on the analysis of cgp mutation operators when inferring gene regulatory networks using scrna-seq time series data conference paper first online: 28 november 2021 pp 264–279 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) on the analysis of cgp mutation operators when inferring gene regulatory networks using scrna-seq time series data download book pdf download book epub josé eduardo h. da silva10, heder s. bernardino10, itamar l. de oliveira10, alex b. vieira10 & … helio j. c. barbosa10,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 744 accesses 2 citations abstract gene regulatory networks (grns) inference from gene expression data is a hard task and a widely addressed challenge. grns can be represented as boolean models similarly to digital circuits. cartesian genetic programming (cgp), often used for designing circuits, can thus be adopted in the inference of grns. the main cgp operator for generating candidate designs is mutation, making its choice important for obtaining good results. although there are many mutation operators for cgp, to the best of our knowledge, there is no analysis of them in the grn inference problem. an evaluation of the single active mutation (sam) and the semantically-oriented mutation operator (somo) is performed here for grns inference. also, a combination of both operators is proposed. we use a benchmark single-cell rna-sequencing time series data and its evaluation pipeline to measure the performance of the approaches. the experiments indicate that (i) combining somo and sam provides the best results, and (ii) the results obtained by the proposal are competitive with those from state-of-the-art methods. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others semantically-oriented mutation operator in cartesian genetic programming for evolutionary circuit design article 02 october 2021 cartesian genetic programming with guided and single active mutations for designing combinational logic circuits chapter © 2019 gene regulatory network reconstruction: harnessing the power of single-cell multi-omic data article open access 19 october 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. gene regulatory networks genetic models genome-wide analysis of gene expression regulatory networks sequence annotation somatic hypermutation 1 introduction systems biology is an interdisciplinary research area that focuses on the computational and mathematical analysis of interactions among the components of a biological system [28]. all cellular activities are controlled by their genes through a network that forms proteins from dna. the gene expression depends on the relationships in this network, known as gene regulatory network (grn) [13]. several methods have been developed for the inference of grns, such as pidc [3], that uses partial information decomposition (pid) to identify regulatory relationships between genes, genie3 [12], winner of the dream5 network challenge, that decomposes the prediction of a regulatory network among p genes into p different regression problems, and grnboost2 [19], that is based on genie3, and uses a gradient boosting machine (gbm) regression [7]. genie3 and grnboost2 are provided in the same python package arboreto [19]. also, evolutionary computation techniques have been applied to infer gene regulatory networks. for instance, one can find the application of genetic programming and evolutionary strategies in this context [15, 26]. grns can be modeled as continuous models in the form of differential equations or as discrete models, such as bayesian and boolean networks. for discrete models, a common scheme for representing the genes’ interactions is an alphabet of two symbols \(\gamma \) = {0,1}, where 0 means inhibition and 1 means activation. a model for this type of network is similar to digital circuits, which can be evolved using cartesian genetic programming (cgp) [18], as presented in [24, 25]. although it is possible to use recombination operators, cgp mostly uses mutation operators to create genetic variation in its offspring. there is no general-purpose crossover operator with good performance for cgp [17]. point-mutation, in which a gene is randomly chosen and changed to another value, and single active mutation (sam) [9] that guarantees a phenotypic difference after the mutation, are two common mutation approaches used on cgp. recently, the semantically-oriented mutation operator (somo) was developed for designing evolutionary circuits [11]. somo can obtain feasible circuits using less computational effort than other approaches from the literature, and this advantage helps to deal with the scalability limitation of cgp. however, somo uses only a specific population initialization in order to optimize the circuit in terms of reducing the number of logic elements. there is not a proper optimization step. also, point-mutation has a parameter, \(\mu _{r}\), which defines the percentage of the genotype to be mutated. somo, on the other hand, has more sensitive parameters that control the genetic variation of the offspring. therefore, in this paper, we propose the evaluation and analysis of cgp when applied to the context of the gene regulation networks inference by using benchmark curated models with single-cell rna-sequencing (scrna-seq) time-series gene expression data technology. here, we provide a critical assessment of the advantages and disadvantages of using somo, with and without an appropriate optimization scheme with sam. furthermore, we perform a parameter sensitivity analysis of somo. also, the results obtained by the proposal in the computational experiments are better than those found by other gp approaches and similar to those reached by the state-of-the-art algorithm genie3, one of the techniques more widely used in the literature for modeling grns. our results show that it is important to have an optimization step to obtain correct grns and that somo is very sensitive to its parameters. 2 problem definition the usual research aim is to infer the network topology from given gene expression data [1]. the gene expression data can be given as a steady-state or time-series matrix. for the former, each measure represents the expression level of a given gene in a certain condition and, for the latter, the gene expression measurements is carried out in each time point. considering n the total number of genes and s the total number of time-points measured, a gene expression dataset is represented by a s x n matrix, where each row vector s (s = 1, ..., s) represents a n-dimensional transcriptome, and each column vector y (y = 1, ..., n) corresponds to a s-dimensional gene profile in the total cell population [4]. the goal of the network inference method is to use the data matrix to predict a set of regulatory interactions between any two genes from the total of n genes. the final output is in the form of a graph with n nodes and a set of edges [4]. figure 1a shows an example of a grn with four genes (nodes) and their regulatory relationships (edges). blue pointed arrows represent an activation and orange lines, inhibition. for example, g1 activates g2, and g4 inhibits g3. fig. 1. illustrations of a grn with four genes (g1-g4) (a) and an individual of cgp (b). the representation is composed by three primary inputs (i0, i1, and i2) and one output (o0). continuous lines defines the phenotype. in this representation, \(n_{c}\) = \(n_{r}\) = 3 and the function set \(\gamma \) = {and, or, not, xor}. full size image discrete and continuous models of grns are often used to understand the process. when considering continuous models, it is common to use differential equations to model the regulatory relationships between genes, such as presented in [8, 24]. this type of model uses the concentrations of the macromolecules such as rnas and proteins (both are gene products) and other biological species concentration, modeling them through the time-rate-of-change of their concentration variables. regulatory interactions take the form of functional and differential relationships between concentration variables [22]. more specifically, gene regulation is modeled by rate equations that express the rate of production of a system component as a function of the concentrations of other components. also, it is possible to model grns through a discrete model, such as boolean networks. boolean networks use boolean algebra to discover the relationship between genes. moreover, boolean networks assume each gene g at a time point t to be in one of two states, active or inactive, according to its gene expression data at time t [2], therefore, one needs to binarize the data, leading to information loss. given a threshold, the data is discretized in 1 (activation) or 0 (inhibition). boolean-based models simplify the structure and dynamics of gene regulation. inferred networks provide a quantitative measure of gene regulatory mechanisms [22]. this model, despite its simplicity, can represent, through its dynamics, several biologically significant phenomena. furthermore, it is possible to obtain several practical uses, such as the identification of drugs for cancer treatment through the inference of the relationships between genes from experimental data such as gene expression profiles [16]. there are several techniques for measuring the expression of a gene. for instance, single-cell rna-sequencing (scrna-seq) is attractive for grn inference due to its production of thousands of independent measurements [14] and there are methods that sort cells along “trajectories” describing the development or progress of the cell [10, 21, 23, 27], called pseudotime, which is a measure of how far a cell has moved through biological progress. also, when using scrna-seq data technologies, it is common to observe a dropout, which occurs when a gene is observed at a low or moderate expression level in one cell but it is not detected in another cell of the same type. dropouts make it difficult to get correct grns. 3 cartesian genetic programming cartesian genetic programming is a genetic programming technique in which programs are directed acyclic graphs (dags) encoded by a matrix of processing nodes, with \(n_{c}\) columns and \(n_{r}\) rows. [17]. the genes are integer values and, for each gene, there are inputs and operation/function that the node performs. given a node in the matrix, the nodes at its left side can be used as inputs. there is a user-defined parameter (levels-back) that limits the number of columns at the left side where inputs can be selected to constrain the connectivity of the graph. the number of inputs that a node can receive is called arity. also, the genotype contains nodes that contribute directly to the output, called active nodes, and those that do not, the inactive nodes. the phenotype is composed only of the active nodes and the genotype-phenotype mapping is done by recursively determining the nodes that contribute to each output, starting on the output and ending on the primary inputs. the function set is user-defined and problem-dependent. for example, logic functions or gates are used when designing digital circuits. figure 1b presents a cgp individual with three primary inputs and one output. the functions are logical ones. grey nodes are active and white nodes, inactive. the most common search technique used in cgp is the \((1 + \lambda )\) evolutionary strategy (es) [17], where \(\lambda \) is the number of new solutions generated at each iteration. in this case, the best individual, the one with more matches concerning its truth-table, between the parent and the \(\lambda \) new generated solutions is selected for the next generation. 3.1 mutation operators cgp normally uses only mutation to generate new individuals and crossover operators have received little attention in the cgp [17]. the two most adopted mutation approaches are point mutation [17] and sam [9]. point mutation is a simple approach that randomly selects a node and a gene and changes it to another valid value. however, the changes may occur in inactive nodes leading to a lack of modifications in the phenotype. sam, proposed for reducing the number of wasted objective function evaluations, operates by (i) randomly selecting one node and one of its elements (inputs or function) and (ii) changing its value to another valid value. steps (i) and (ii) are repeated until an active node is modified. as a result, sam ensures that one active node is changed. somo [11] was developed for the design of digital circuits using cgp, where the purely stochastic mutation operator is replaced and operates in the phenotype space. the somo steps are presented in algorithm 1, which can be summarized as: (i) all inactive nodes have their inputs changed randomly to another valid value; functions are changed with a probability \(p_{f}\); (ii) a random node (c) is chosen, in order to be mutated using somo; (iii) c has its function modified randomly with a probability \(p_{f}\); (iv) a random input (e) is chosen from c. step (i) ensures that new genetic material is generated before performing the actual mutation. with the random input (e) chosen, somo identifies the best input considering all previous nodes in the genotype and then performs the mutation. the identification of the most suitable node is based on semantics. initially, it calculates the score of every node of the genotype that may be connected to the mutated node c. if more nodes receive the same score, the node closest to the program inputs is preferred. the score reflects the hamming distance (hd). in the sequence, all left-sided nodes from the mutated node c are connected at e and simulated in three different ways: (a) using the function of the node being connected, (b) forcing e to logic zero (\(val^{ [0]}_{e=0}\)), and (c) forcing e to a logic one (\(val^{[0]}_{e=1}\)). the desired input value is denoted as req and it can be equal to ‘0’, ‘1’ or ‘x’, where ‘x’ means that it does not matter what boolean value the input e takes. the value of req is determined by using the ternary operator \({ \theta }\) and the reduction operator \(\bigodot \), defined in equations 1 and 2. the term [o] in superscript points to a boolean value associated with a program output node o. in somo, the mutated input e of c is connected to the best node in the genotype of the current generation. differently from the standard cgp, somo uses \(\lambda \) = 1 and a different population initialization. in somo, the population is started with a candidate solution having no active gate in order to maximize the efficiency and minimize the number of active gates of the evolved solutions [11]. 4 proposal and methods here, we evaluate different approaches using somo aiming at discovering if using an appropriate optimization step helps it to obtain better grns. we propose the use of somo with sam as (i) somo can obtain a feasible solution quickly but with many logic elements, and (ii) sam is appropriate for reducing the number of logic elements. the proposal is presented in algorithm 2. in [11], the percentage of inactive nodes to be mutated (\(p_{q}\)) is equal to 100. here, we analyze the performance of somo when using \(p_{q}\) = 50% with sam as optimization step. in addition, as highlighted in [11], somo usually gets stuck in local optima. to avoid this issue, we use a restart strategy for exploiting different locations of the search space. the restart strategy consists of initializing a new population and restart the search for every k objective function evaluations. the criteria for evaluating and comparing methods are varied, but [20] presented a set of benchmark problems when considering scrna-seq data for inferring grns. this set is composed of toy, curated, and real datasets. for toy and curated models, we have the ground-truth network, which facilitates the evaluation of new methods. also, there is an extensive comparison between several algorithms from the literature using the pipeline provided in [20]. in this paper, we use curated scrna-seq data in the form of pseudotime-series. these datasets were made considering 2,000 cells with three configurations: 0%, 50%, and 70% dropout rates. for each configuration, 10 datasets are given. furthermore, each problem has a different number of pseudotimes. as each pseudotime gives information about one possible cell trajectory, for each pseudotime one grn is inferred. then, the final grn is given by merging the partial grns discovered for each pseudotime. this final grn is evaluated considering the pipeline presented in [20] which considers the area under the precision-recall curve (auprc) and the area under the receiver operating characteristic curve (auroc) values for comparison. auprc is calculated as the area under the precision-recall (pr) curve and shows the trade-off between precision and recall across different decision thresholds. the x-axis of a pr curve is the recall and the x-axis is the false-positive ratio (fpr). considering tp as true positive, fp as false positive and fn as false negative, we can define precision = \(\frac{tp}{tp+fp}\) and recall = \(\frac{tp}{tp+fn}\). for auroc, the roc curve is plotted considering the false-positive ratio (fpr) on the x-axis and the recall, on the y-axis. recall was defined previously and fpr = \(\frac{fp}{tn+fp}\). an excellent model has auc close to 1, which means it has good separability. for many real-world datasets, particularly medical datasets, the fraction of positives is often less than 0.5, meaning that auprc has a lower baseline value than auroc [6]. 5 computational experiments computational experiments were conducted to analyze the performance of cgp mutation operators when applied to the inference of grns using the benchmark scrna-seq time-series data from [20]. here, we highlight whether obtaining feasible solutions in a faster way has a positive impact on the quality of the solutions. also, we analyze the importance of reducing the number of logic elements. the computational experiments are composed of two parts: (i) the performance of cgp variants are compared and, (ii) the results obtained by the two best cgp approaches are compared to those found by genie3. the problems considered in our analysis are presented in table 1, with information of the number of genes (#genes) and the number of pseudotimes (#pseudotimes). these problems are curated models with 2,000 cells and are presented in three different configurations considering 0%, 50%, and 70% dropout. in order to remove or soften the effects of technical and biological variations as well as obtain a single curve that represents gene expression over time, we use data approximation via cubic smoothing splines implemented in python and distributed through the library csapsfootnote 1. then, the data is binarized with bikmeans through gene expression data pre-processing tool (gedprotools)footnote 2. all methods were implemented in c++ and the source codes are availablefootnote 3. the experiments were run in a ubuntu server 20.04 lts (hvm) with 16 vcpus intel(r) xeon(r) cpu e5-2666 v3 @ 2.90ghz and 30gb ram and the results are evaluated using the pipeline presented in [20] which considers the auprc and auroc values for comparison. table 1. problems used in the experiments.full size table 5.1 comparative analysis of the cgp techniques we compare the standard cgp with sam and the cgp with somo in different approaches: (i) cgp: standard cgp using sam for obtaining the first feasible solution and for optimizing; (ii) somo: cgp using somo without optimization step; (iii) somo-sam: cgp using somo with optimization step; (iv) somo-sam-r: cgp using somo with optimization step and evolutionary search restart; (v) somo-sam-pq50: cgp using somo with optimization step and \(p_q\) = 50%. the suffix r means that the evolutionary process is restarted with a different initial population every 1,000 evaluations. here, we aim to evaluate the performance of somo with an appropriate optimization step. somo is able to obtain feasible solutions in a faster way than other approaches but this solution usually has many logic elements. table 2. auprc results for all problems. the suffix after the problem’s name is the dropout rate.full size table table 3. auroc results for all problems. the suffix after the problem’s name is the dropout rate. algorithms marked with an asterisk have statistical difference.full size table table 4. algorithm counting considering the best values of median. values between parenthesis is the algorithm counting considering the statistical equality.full size table for the standard cgp, we use \(\lambda \) = 4 and random population initialization. when considering somo, we use \(\lambda \) = 2, \(p_{f} = 0\) and the population initialization suggested in [11]. also, in [11] the value of \(n_c\) is variable, considering a multiple of the number of standard gates required to implement each circuit. however, the definition of the number of gates to implement a circuit cannot be easily defined a priori and, hence, we fixed \(n_c\) as usually considered when using the standard cgp and used only \(n_c = 100\). the other parameters are \(n_r = 1\) and \(lb=n_c\). for each problem, 5 independent runs were performed with a maximum of 100,000 objective function evaluations. for somo-sam-r, the number of evaluations to restart is 1,000 and was defined during preliminary experiments, analyzing the average number of evaluations needed to find a feasible solution. also, the accumulated number of evaluations is used in the stop criteria. tables 2 and 3 present the results of the cgp methods, respectively, for auprc and auroc. the best (maximum), first quantile (q1), median, mean, third quantile (q3), and standard deviation values of the results are shown, and the best values are highlighted in boldface. also, the methods with an asterisk have statistical difference, when considering dunn’s test for 95% of confidence. based on these results, one can see that the results of somo-sam and somo-sam-r are equal in some cases. this is expected as the first feasible solution was obtained before reaching the first restart point (1,000 evaluations). in general, most cgp approaches have no statistical difference when compared to each other, both in auprc and auroc. when considering the median, we count the times that each algorithm reached the best result. values in parentheses consider the statistical tests and when there is no statistical difference, all methods score. this counting is shown in table 4. fig. 2. results for the problems considering all scenarios. full size image fig. 3. pps considering auprc and auroc for the best approaches. full size image also, the approaches using an optimization step performed better than the other ones. the restart scheme helps to escape from local optima but it does not provide good results. furthermore, the results when using \(p_{q}\) = 50% are better than the standard values presented in [11]. the somo population initialization is essential for a good performance of the algorithm as well as using \(\lambda \) = 1. finally, based on table 4, one can conclude that the best methods are somo-sam and somo-sam-pq50 when auprc is considered. for auroc, the best algorithms are somo-sam and somo. however, here we consider one approach using the standard somo with sam as optimization step (somo-sam) and another approach varying the parameter \(p_{q}\), namely somo-sam-pq50. 5.2 comparative analysis with genie3 as observed in sect. 5.1, the two best cgp variants are somo-sam and somo-sampq50, and we compare them here with genie3. according to [20], genie3 is the best algorithm for inferring grns. performance profiles (pps) [5] were used in order to analyze the relative performance of the algorithms. considering p as a particular problem (model) and s as a particular solver, \(\rho (p,s)\) is defined as the performance ratio within a factor of \(\tau \) of the best possible ratio. from pps, it is possible to extract: (i) the approach that obtained the best results for most problems (largest \(\rho (1)\)), (ii) the most reliable approach (smaller \(\tau \) such that \(\rho (\tau ) = 1\)), and (iii) the best overall performance (largest area under the performance profiles curves). moreover, boxplots of the results are presented in fig. 2. also, kruskal wallis statistical test and dunn’s post hoc test were carried out and the results show that there is statistical difference only when comparing cgp approaches with genie3. based on the pps presented in fig. 3 is possible to conclude that: (i) genie3 has the best performance in most of the problems (largest \(\rho (1)\)), followed by somo-sam and somo-sam-pq50, respectively. (ii) genie3 is the most reliable variant (smallest \(\tau \) such that \(\rho (\tau )=1\)), followed by somo-sam-pq50 and somo-sam, respectively, and; (iii) genie3 presents the best overall performance (largest auc), followed by somo-sam-pq50 and somo-sam, respectively. thus, genie3 is a good choice when considering only auprc. however, as highlighted in sect. 4, for many real-world medical datasets the fraction of positives is often less than 0.5. on the other hand, when considering the performance profiles of auroc, we can conclude that: (i) genie3 hast the best performance in most of the problems, followed by somo-sam and somo-sam-50, respectively; (ii) somo-sam and somo-sam-pq50 are the most reliable variants, and; (iii) somo-sam presents the best overall performance, followed by somo-sam-pq50 and genie3, respectively. then, somo-sam highlights the importance of using an appropriate optimization step. also, somo is very sensitive to the parameter \(p_{q}\), i.e. keeping some inactive nodes unchanged before performing somo helps to obtain better results. one possible reason is the fact that, when using sam, we adopt \(\lambda = 4\), then, cgp is able to create more diverse offspring than with \(\lambda = 1\) as the standard somo uses. however, only when considering mcad, approaches that use cgp obtained better results on auprc and auroc. with respect to mcad it is interesting to highlight that this problem is the only one with 2 pseudotimes. the evaluation pipeline considers the “top k” regulatory relationships and, when merging the 2 solutions to obtain the final grn, it is possible to construct a smaller grn (less number of regulatory relationships) that consider the most important regulatory relationships leading to a better evaluation. the statistical tests show that there is no statistical difference between the cgp approaches. the statistical difference is observed only when comparing the cgp variants with genie3, reinforcing the superiority of genie3 as shown in the auprc boxplots. 6 conclusions and future work here, we analyze the performance of cgp mutation operators applied when inferring grn using benchmark curated scrna-seq time series data. these approaches include mainly two aspects: (i) a proposal using sam to optimize the solution obtained by somo, and (ii) a parameter sensitivity analysis for somo. we compared the cgp approaches, and the best-obtained cgp methods with genie3. the results show that modifying \(p_{q}\) to 50% helped cgp to obtain better results, as a more diverse offspring is generated. genie3 performed better in most problems, except in mcad. the mcad problem has only two pseudotimes and the cgp solutions’ merge can obtain a smaller grn that is better evaluated in the pipeline. also, there is no statistical difference between the cgp approaches, but there is when comparing the cgp variants with genie3. according to pps, genie3 obtained the best results in general but one can highlight that both somo-sam and somo-sam-pq50 are more reliable when auroc is considered. furthermore, the use of somo for obtaining faster first feasible solutions and sam as an appropriate optimization step helps cgp to obtain better results when inferring gene regulatory networks when compared to the standard somo and to the standard cgp. thus, one can conclude that using sam with somo improves the inferred grns. as future work, we intend to explore even more the somo parameters, considering other values of \(p_{q}\) and \(p_{f}\) in the context of the inference of grns and the impact of \(n_{c}\) when adopting values of \(\lambda > 1\). also, we intend to investigate the impact of the number of pseudotimes in the merging step of cgp. notes 1.https://csaps.readthedocs.io/en/latest/index.html. 2.http://lidecc.cs.uns.edu.ar/files/gedprotools.zip. 3.https://github.com/ciml. references aalto, a., viitasaari, l., ilmonen, p., mombaerts, l., gonçalves, j.: gene regulatory network inference from sparsely sampled noisy data. nat. commun. 11(1), 1–9 (2020) google scholar  banf, m., rhee, s.y.: computational inference of gene regulatory networks: approaches, limitations and opportunities. biochimica et biophysica acta (bba) gene regul. mech. 1860(1), 41–52 (2017) google scholar  chan, t.e., stumpf, m.p., babtie, a.c.: gene regulatory network inference from single-cell data using multivariate information measures. cell syst. 5(3), 251–267 (2017) google scholar  chen, s., mar, j.c.: evaluating methods of inferring gene regulatory networks highlights their lack of performance for single cell gene expression data. bmc bioinf. 19(1), 1–21 (2018) google scholar  dolan, e.d., moré, j.j.: benchmarking optimization software with performance profiles. math. programm. 91(2), 201–213 (2002) google scholar  draelos, r.: measuring performance: auprc and average precision (2019). glassboxmedicine.com/2019/03/02/measuring-performance-auprc/ friedman, j.h.: greedy function approximation: a gradient boosting machine. ann. statist., pp. 1189–1232 (2001) google scholar  gebert, j., radde, n., weber, g.w.: modeling gene regulatory networks with piecewise linear differential equations. eur. j. oper. res. 181(3), 1148–1165 (2007) google scholar  goldman, b.w., punch, w.f.: reducing wasted evaluations in cartesian genetic programming. in: krawiec, k., et al. (eds.) eurogp 2013. lncs, vol. 7831, pp. 61–72. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-37207-0_6 haghverdi, l., büttner, m., wolf, f.a., buettner, f., theis, f.j.: diffusion pseudotime robustly reconstructs lineage branching. nat. methods 13(10), 845 (2016) google scholar  hodan, d., mrazek, v., vasicek, z.: semantically-oriented mutation operator in cartesian genetic programming for evolutionary circuit design. in: proceedings of the 2020 genetic and evolutionary computation conference, pp. 940–948 (2020) google scholar  irrthum, a., wehenkel, l., geurts, p., et al.: inferring regulatory networks from expression data using tree-based methods. plos one 5(9), e12776 (2010) article  google scholar  jackson, c.a., castro, d.m., saldi, g.a., bonneau, r., gresham, d.: gene regulatory network reconstruction using single-cell rna sequencing of barcoded genotypes in diverse environments. elife 9, e51254 (2020) article  google scholar  liu, s., trapnell, c.: single-cell transcriptome sequencing: recent advances and remaining challenges. f1000research 5 (2016) google scholar  ma, b., jiao, x., meng, f., xu, f., geng, y., gao, r., wang, w., sun, y.: identification of gene regulatory networks by integrating genetic programming with particle filtering. ieee access 7, 113760–113770 (2019) article  google scholar  mccall, m.n.: estimation of gene regulatory networks. postdoc j. postdoc. res. postdoc. affairs, 1(1), 60 (2013) google scholar  miller, j.f.: cartesian genetic programming. cgp, pp. 17–34 (2011) google scholar  miller, j.f., thomson, p., fogarty, t.: designing electronic circuits using evolutionary algorithms. arithmetic circuits: a case study (1997) google scholar  moerman, t., et al.: grnboost2 and arboreto: efficient and scalable inference of gene regulatory networks. bioinformatics 35(12), 2159–2161 (2019) google scholar  pratapa, a., jalihal, a.p., law, j.n., bharadwaj, a., murali, t.: benchmarking algorithms for gene regulatory network inference from single-cell transcriptomic data. nat. methods 17(2), 147–154 (2020) google scholar  qiu, x., mao, q., tang, y., wang, l., chawla, r., pliner, h.a., trapnell, c.: reversed graph embedding resolves complex single-cell trajectories. nat. methods 14(10), 979 (2017) google scholar  huynh-thu, v.a., sanguinetti, g.: gene regulatory network inference: an introductory survey. in: sanguinetti, g., huynh-thu, v.a. (eds.) gene regulatory networks. mmb, vol. 1883, pp. 1–23. springer, new york (2019). https://doi.org/10.1007/978-1-4939-8882-2_1 setty, m., et al.: wishbone identifies bifurcating developmental trajectories from single-cell data. nat. biotech. 34(6), 637–645 (2016) google scholar  da silva, j.e.h., et al.: inferring gene regulatory network models from time-series data using metaheuristics. in: ieee congress on evolutionary computer (cec), pp. 1–8. ieee (2020) google scholar  da silva, j.e.h., bernardino, h.s., de oliveira, i.l.: inference of gene regulatory networks from single-cell rna-sequencing data using cartesian genetic programming (under review). in: bioinformatics, pp. 1–8. oxford (2021) google scholar  streichert, f., et al.: comparing genetic programming and evolution strategies on inferring gene regulatory networks. in: deb, k. (ed.) gecco 2004. lncs, vol. 3102, pp. 471–480. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-24854-5_47 trapnell, c., et al.: the dynamics and regulators of cell fate decisions are revealed by pseudotemporal ordering of single cells. nat. biotechnol. 32(4), 381 (2014) google scholar  wang, r.s., saadatpour, a., albert, r.: boolean modeling in systems biology: an overview of methodology and applications. phys. biol. 9(5), 055001 (2012) google scholar  download references acknowledgements we thank the support provided by faperj, fapesp, fapemig, capes, cnpq, ufjf, and amazon aws. author information authors and affiliations universidade federal de juiz de fora, juiz de fora, mg, brazil josé eduardo h. da silva, heder s. bernardino, itamar l. de oliveira, alex b. vieira & helio j. c. barbosa laboratório nacional de computação científica, petrópolis, rj, brazil helio j. c. barbosa authors josé eduardo h. da silvaview author publications search author on:pubmed google scholar heder s. bernardinoview author publications search author on:pubmed google scholar itamar l. de oliveiraview author publications search author on:pubmed google scholar alex b. vieiraview author publications search author on:pubmed google scholar helio j. c. barbosaview author publications search author on:pubmed google scholar corresponding author correspondence to josé eduardo h. da silva . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silva, j.e.h.d., bernardino, h.s., de oliveira, i.l., vieira, a.b., barbosa, h.j.c. (2021). on the analysis of cgp mutation operators when inferring gene regulatory networks using scrna-seq time series data. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_18 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature the impact of double transfer learning in vgg architectures for metastasis breast cancer detection | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper the impact of double transfer learning in vgg architectures for metastasis breast cancer detection conference paper first online: 30 january 2025 pp 457–468 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) the impact of double transfer learning in vgg architectures for metastasis breast cancer detection download book pdf download book epub danyllo carlos silva e silva  orcid: orcid.org/0009-0008-9932-12809, omar andres carmona cortes  orcid: orcid.org/0000-0002-5805-24909,10 & joão otávio bandeira diniz  orcid: orcid.org/0000-0003-3303-334611  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 375 accesses 1 citation abstract according to the world health organization (who), 2.3 million women were diagnosed with breast cancer in 2020, causing almost 700.000 deaths worldwide. the first occurrences (in situ stage) usually have a good response if there is an early detection because the earliest form of tumor does not have sufficient potential to kill. however, the next stage has a low survival rate because the most threatening tumor characteristic is cell division spreading throughout the body, damaging lungs, livers, bones, and the brain. an alternative to deal with this problem is to enhance the velocity of the diagnosis and detection of breast cancer. thus, machine learning algorithms have proven to be effective in this task. in this context, this work investigates how double transfer learning improves two deep learning architectures, vgg-16 and vgg-19, using histopathological images, i.e., employing breakhis as the first transfer learning and patchcamelyon as the second. results indicate that using double transfer learning, results for precision, specificity, recall, and f-score improve to 99%, 83%, 89%, and 90%, respectively. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others breast cancer prediction on breakhis dataset using deep cnn and transfer learning model chapter © 2022 breast cancer detection in histology images using convolutional neural network chapter © 2021 enhancing breast cancer detection accuracy through machine learning, deep learning and transfer learning techniques for clinical practice article open access 11 december 2025 1 introduction cancer is the uncontrollable growth of cells that can spread to other parts of the body [3]. breast cancer, which originates in the breast, is a disease that affects males and females, but the majority are women over 40 years old. obesity, drug use, family history, and late menopause are some of the consequences that favor the occurrence. particularly in brazil, the national cancer institute estimates a total of 704,000 new cases of cancer in the country, and the most frequent is breast cancer, representing 30.1% of all occurrences [13]. having said that, there are other issues associated with the illness. the primary problem is that there are more than 100 subdivisions of breast cancer [12] that the pathologists need to differentiate during an exam analysis, which makes it a challenging task even if the professional is very experienced. moreover, people are susceptible to physical, physiological, mental, and external factors that can affect their work efficiency, such as illness, fatigue, excessive work, and familiar issues, among many others. thus, to help the pathologists in this endeavor and try to decrease these high occurrences of cancer, the main focus of this investigation is on improving diagnosis methods and techniques for early detection to increase patients’ life expectations, i.e., before metastasis. additionally, pathologists used to have confidence in their expertise to analyze thousands of histopathology images. on the other hand, according to litjens et al. [8], one of the performance tests with 11 pathologists without time constraints reached 72.4% of correct diagnosis (27.6% were not identified during the test). in this context, this work aims to investigate vgg-16 and vgg-19 using single and double transfer learning (dtl) to help pathologists detect breast cancer metastasis efficiently. the idea is to train vggs using the breakhis dataset [18], then use the patchcamelyon for detecting metastasis. therefore, this work is divided as follows: sect. 2 presents some related works; sect. 3 shows some literature review and our proposal; sect. 4 shows the computational experiment, its setup, and results; finally, sect. 5 presents the conclusions of this investigation and future work. 2 related works well-known convolutional neural network architectures (cnn), such as vgg, resnet, densenet, etc., have played a notable role in image recognition with the advent of deep learning [1]. for instance, visual geometry group (vgg) [17] won the imagenet challenge in 2014, and it was presented during the international conference on learning representations. this architecture brought a different concept of deep networks with an improved small architecture (3x3 conv. filters and 16–19 weight layers) for pattern recognition. nowadays, vgg is commonly used in the medical context for anomaly detection, characterizing neoplasias into benign or malign based on image recognition [15], such as the work of filho & cortes [5], santos-bustos et al. [16], and liu et al. [9]. ismail & sovuthy [6] performed a comparison between vgg-16 (94% acc) and resnet50 (91.7% acc) in the classification of two classes of neoplasias (normal and abnormal tumor). in other investigations, vgg-19 has shown the best results in many articles, such as [21] and [15]. in vigneshkumar & sumathi [21], a hybrid architecture combining vgg-19 and svm was presented, which reaches 98% of sensibility and 97.8% of accuracy. furthermore, in saaber et al. [15], the auc, accuracy, specificity, precision, and f1-score evaluated for classifying breast cancer using histopathological images were all more than 97.35% (precision), and the best one is 99.5% for auc. regarding specifically breast cancer detection using dtl, matos et al. [10] proposed a classification approach of histopathological images from breakhis dataset comparing double transfer learning (dtl) using inception-v3 pre-trained with imagenet dataset and support vector machine (svm) to classify breast cancer patches in four magnitude levels: 40x, 100x, 200x and 400x. results showed that the proposed dtl approach got an accuracy of 91%. vo-le et al. [22] investigated dtl in vgg-16, googlenet, and resnet-50 using the patch camelyon as a base transfer learning. afterward, they trained different machine learning classifiers on vbcan dataset, a set of images of hematoxylin and eosin (h&e) stained lymph node sections collected from two specialized hospitals in vietnam, to improve the deep learning models. the results have shown that vgg obtained the best recall (97.76%), resnet the best accuracy (96.98%), and googlenet the best precision (98.58%). in our study, we are investigating the use of double transfer learning technique in vgg-16 and vgg-19 to detect breast cancer metastasis in the patch camelyon dataset. to achieve this, we first trained the models using the breakhis dataset. next, we used the patch camelyon dataset to identify the presence of metastasis. finally, we compare our results with those of [10] and [4]. 3 material and method 3.1 convolutional neural networks the mathematical representation of the human neuron was presented for the first time by [11] in the bulletin of mathematical biology. mcculloch and pits proposed that all the behaviors of neurons in the human brain can be interpreted logically, considering impulses (neuron activities), inhibitory or excitatory synapses, and a logical theory called “nets without circles”. the concepts developed by mcculloch and pitts have served as the foundation for all neural network and deep learning architectures created to date. since 2010, deep learning networks have been widely recognized as one of the most effective modern techniques for solving complex pattern recognition problems [1], especially those neural networks, so-called convolutional neural networks (cnn) that have shown high accuracy in recognizing abstract patterns and features that a human would not easily obtain [19]. a cnn has an input layer that receives data used for training and validation. in this present work, the input layers [2] are histopathological features from whole-slide medical images; then, data goes through convolutional layers that are blocks of a deep network, in which filters and parameters are combined in the training stage; usually, after a convolutional layer there is a pooling layer, which are essential components used for reducing spatial dimensions of the input data (width \(\times \) length); finally, a cnn can use a dense layer represented by a fully connected neural network, which combines the outputs from previous layers performing a prediction (classification or regression). the complete illustration of a cnn can be found in fig. 1 and fig. 2, representing the architectures used in this work. fig. 1. vgg16 architecture full size image fig. 2. vgg19 architecture full size image the difference between vgg16 and vgg19 is the number of layers. while vgg16 consists of sixteen weight layers (thirteen convolutional layers and three fully connected layers), vgg19 has nineteen weight layers (sixteen convolutional layers and three fully connected layers). furthermore, vgg16 has five blocks of convolutional layers, with the number of convolutional layers in each block being [2, 2, 3, 3, 3]. in comparison, vgg19 also has five blocks of convolutional layers, but the number of convolutional layers in each block is [2, 2, 4, 4, 4]. 3.2 proposal: double transfer learning and vgg in our proposal, we developed a structure with the combination of the double transfer learning (dtl) technique, vgg16, and vgg19 (trained separately) using three datasets: imagenet (pre-trained), breakhis, and patch camelyon (target domain). these architectures were chosen because of their superior performance in classification tasks in different domains. to clarify how dtl, vgg networks, and connections work, fig. 3 shows the architecture proposal. fig. 3. architecture of the tests using vgg16 and vgg19 (separately trained) in double transfer learning with three datasets: imagenet, breakhis and patch camelyon as target domain. full size image the vgg16 and vgg19 models are initially pre-trained using data from imagenet before extracting histopathological features from the input data. next, the convolutional neural networks (cnns) are trained using the patch camelyon dataset for simple transfer learning (stl), the first classification step. then, the vgg models are further trained using the breakhis dataset before being tested using the patch camelyon dataset to determine if a second transfer learning process affects the detection of metastasis. 4 computational experiments 4.1 setup the google coding environment is used: collaboratory and drive, which store all the python notebooks and datasets (breakhis and patch camelyon). the code was executed using the following configuration: gpu tesla t4, 12 gb ram, and 78.2 gb hdd with tensorflow and keras frameworks. the model parameters set for the train were kernel size 3 \(\times \) 3, pooling size 2 \(\times \) 2, input shape 640 \(\times \) 640 \(\times \) 3 for breakhis first classification step, and 96 \(\times \) 96 \(\times \) 3 for input shape of the classification with target domain (patch camelyon). each vgg was trained in five epochs, batch equal to 32, seed 1337, image size 224 (tensor length), and four filters. early stopping was used if there was no more progress in the learning attributes, and a reduce learning rate was applied to control the tax training flow in each epoch, reducing the factor of 0.5 with a minimum of 0.00001 that can reach the learning rate. the sampling method was hold-out using the rate 80/20. the code of the present work is stored in google drive, which can be easily accessed at https://github.com/danyllosilva/mbcd-bk-pc-msc. 4.2 database two datasets were used for training and testing the vgg algorithms: patch camelyon, reference for metastatic tissue detection [20] and breakhis dataset, reference for breast cancer detection [7]. breast cancer histopathological database (known also as breakhis dataset) version 1.0 is composed of 7,909 microscopic images in tif file format, 3-channel rgb, and four different magnifying factors: 1,995 images in factor 40x, 2,081 in 100x, 2,013 in 200x, and 1,820 in 400x (fig. 4). fig. 4. labeled images from breakhis dataset in four magnifying factors: 40x (a), 100x (b), 200x (c) and 400x (d). full size image the patch camelyon dataset contains 327,800 histopathological images from radboud university medical center, and it was divided into patches labeled by professional pathologists with an indication of the positive or negative occurrence of metastatic breast cancer in the patch region. the base ratio is balanced 50/50 for the two classes, and the image resolution is 96 \(\times \) 96, using 3-channel rgb and tif as file formats. each label in the image helps the algorithm learn the region’s characteristics during the training phase to give a conclusive diagnosis of the presence or absence of metastatic breast cancer. figure 5 shows an example of how each patch can be found in the dataset. fig. 5. labeled images from patchcamelyon dataset. full size image 4.3 pre-processing in pre-processing, some transformations in the images from breakhis dataset were applied to data augmentation: horizontal flip with a probability of 0.5, random brightness contrast also with the same probability, a random resized crop of 0.8 for chance occurrence, and blur with a limit of 1px by 1px. during the integrity data check of patch camelyon, some corrupted images were automatically checked and removed to guarantee that only valid data could be used for training and testing. 4.4 evaluation metrics the following metrics allow us to evaluate the performance of each model. they are based on correct classifications (true positives tp and true negatives tn) and incorrect ones (false positives fp and false negatives fn). additionally, we can build the confusion matrix using these values, which can help us visualize how the model performs in terms of correct and incorrect classification. then, the following metrics complete the evaluation, showing the performance numbers. in this context, eq. 1 computes the accuracy, which is the overall model performance, but it is not enough to evaluate the performance due to dataset distribution. supposing that the dataset presents 80% of positives and 20% of negatives, a simple model that classifies all input as positive would show an accuracy of 80%. however, the satisfactory result is not a consequence of a good model but the effect of the classification distribution in the dataset. $$\begin{aligned} accuracy = \frac{tp + tn}{tp + tn + fp + fn} \end{aligned}$$ (1) equation 2 calculates the model precision, which represents how false positives impact the model performance. in other words, the higher the false positive rate, the lower the precision. $$\begin{aligned} precision = \frac{tp}{tp + fp} \end{aligned}$$ (2) the recall presented in eq. 3 demonstrates how effectively the model identifies false negatives. this metric is crucial in health applications, particularly in cancer detection, as a false negative could cost the patient’s life. $$\begin{aligned} recall = \frac{tp}{tp + fn} \end{aligned}$$ (3) the specificity is shown in eq. 4, which is the proportion of true negatives (tn) detected correctly. the false positives affect the metric value, which can lead to an unnecessary treatment. thus, a look into specificity is made along with a look to recall. $$\begin{aligned} specificity = \frac{tn}{tn + fp} \end{aligned}$$ (4) finally, the metric f1-score provides an overall understanding of how well the model performs on its task, mainly when dealing with false positives and false negatives. in other words, it gives us a general idea of how the model is making classifications. the more incorrect the classifications, the lower the f1-score $$\begin{aligned} f1-score = \frac{2 \times precision \times recall}{precision + recall} \end{aligned}$$ (5) 4.5 results the results are divided into vgg in stl, using only patch camelyon, and dtl, using breakhis pl patch camelyon. the following metrics are used: precision, specificity, recall, and f1-score calculated by the confusion matrix of vgg16 and vgg19 as previously presented. figure 6 depicts the confusion matrix for vgg16 and vgg19 using stl. it shows 4879 correct classifications and 1130 incorrect for vgg16. while 4685 are correct and 1315 are incorrect for vgg19. these numbers reflect the following results in the performance metrics as presented in table 1: vgg16 in stl obtained 83% of precision and specificity, 84% of recall, and f1-score equal to 82%. in comparison, vgg19 reached 79% of precision, the same for specificity, 77% of recall, and f1-score of 78%. in other words, vgg16 got the best results in stl. fig. 6. confusion matrix of single transfer learning (stl). full size image when dtl is applied, the results are considerably better than stl, as we can see in fig. 7, in which 5113 are correct and 616 are incorrect, representing an enhancement of 234 correct classifications in vgg16. in comparison, vgg19 using dtl increases 373 correct classifications, which is higher than vgg16; however, it is not enough to overcome it. fig. 7. confusion matrix of dtl full size image regarding metrics in table 1, vgg16 reached a precision of 99%, a specificity of 86%, a recall of 84%, and an f1-score of 90%, which is a considerable improvement compared to stl excepting in the recall that has presented no improvements. in vgg19, the improvements are minor when using dtl, increasing the precision to 85% and keeping the specificity unchanged. on the other hand, dtl has considerably improved the recall to 89% and the f1-score to 87% in vgg19. although these improvements in vgg19 are impressive, they are not enough to overcome vgg16, except for the recall metric. table 1. comparison table of the results from two groups: vgg16 (stl/dtl) and vgg19 (stl/dtl).full size table in table 2, we compare our proposal with the following works: [10] and [4]. the first used a different approach, considering four magnitude levels of the breakhis dataset as the target dataset to reach the best acc than the state-of-the-art. in the comparison of our best result in vgg16 (dtl) and [10] (fig. 2), we achieved a better accuracy than all other accuracy results, mainly compared to the “proposed (inception-v3 + filter)” approach that reached 91% at 100x magnification as the best result. this work is more in-depth because it deals with two datasets, breakhis as the first domain and patch camelyon as the target (shown in fig. 3), and two points of view (stl and dtl) regardless of the magnitude level. concerning [4], the authors obtained a maximum accuracy of 85% using mainly well-established data augmentation techniques (randomly sized cutout mask, for example) in the patch camelyon dataset. our best result reached 0.92 for accuracy. table 2. comparison against [10] and [4]full size table 5 conclusions and future work this work proposed a novel deep double transfer learning approach with two established cnn architectures, vgg16 and vgg19. it shows that dtl is a promising technique and has a positive effect in both cnns, but with better results in vgg16, reaching a precision of 99%, a specificity of 86%, a recall of 84%, and an f1-score of 99%. according to the metrics evaluated and compared against two published works from [14] and [4], this work could beat the accuracy of both works, showing that using vgg in the double transfer learning context is a promising approach. future work includes applying dtl in other cnn-established models or adding more parameters and different approaches for data augmentation to improve results along the epochs, which are fewer yet (5) only for use as reference because the state-of-the-art works use few epochs to guarantee the best results at first and see whether the model is promising. references aggarwal, c.c.: neural networks and deep learning. in: neural networks and deep learning, pp. 9–53. springer (2018) google scholar  beysolow ii, t.: introduction to deep learning using r: a step-by-step guide to learning and implementing deep learning models using r. apress (2017) google scholar  cancer, n.i.o.: the definition of cancer \(|\) 2024. in: what is cancer, p. 1. national institute of cancer (usa) (2024) google scholar  ericsson, a., kana, f.: convolutional neural networks for classification of metastatic tissue in lymph nodes (2021) google scholar  filho, m.l.r., cortes, o.a.c.: efficient breast cancer classification using histopathological images and a simple vgg. revista de informática teórica e aplicada 29(1), 102–114 (2022). https://doi.org/10.22456/2175-2745.119207 ismail, n.s., sovuthy, c.: breast cancer detection based on deep learning technique. in: 2019 international unimas stem 12th engineering conference (encon), pp. 89–92 (2019) google scholar  jiang, y., chen, l., zhang, h., xiao, x.: structure of breakhis dataset. plos one 23(23), 1–6 (2023). https://doi.org/10.1371/journal.pone.0214587.t004 article  math  google scholar  litjens, g., et al.: 1399 h &e-stained sentinel lymph node sections of breast cancer patients: the camelyon dataset. in: gigascience, o. (ed.) the camelyon dataset, pp. 2–3. oxford (2022) google scholar  liu, z., peng, j., guo, x., chen, s., liu, l.: breast cancer classification method based on improved vgg16 using mammography images. j. radiat. res. appl. sci. 17(2), 100885 (2024). https://doi.org/10.1016/j.jrras.2024.100885 article  math  google scholar  matos, j.d., britto, a.d.s., oliveira, l.e.s., koerich, a.l.: double transfer learning for breast cancer histopathologic image classification. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–8 (2019). https://doi.org/10.1109/ijcnn.2019.8852092 mcculloch, w.s., pitts, w.: a logical calculus of the ideas immanent in nervous activity. in: bulletin of mathematical biophysics, pp. 115–133. springer (1943) google scholar  prevention, b.n.i.o.c.: estimate \(|\) 2024. in: cancer incidence in brazil, pp. 39–40. national institute of cancer prevention (brazil) (2024) google scholar  primo, w.q.: national cancer institute and the 2023-2025 estimate – cancer incidence in brazil \(|\) 2023. in: estimate \(|\) 2023-2025, pp. 1–2. national institute of cancer (usa) (2023) google scholar  rodrigues filho, m.l., cortes, o.a.: classificação de imagens histopatológicas usando uma vgg e mecanismo de atenção para detecção de câncer de mama. sbc 23(23), 1–6 (2023). https://doi.org/10.5753/sbcas_estendido.2023.229388 saaber, a., sakr, m., abo-seida, o.m., keshk, a., chen, h.: a novel deep-learning model for automatic detection and classification of breast cancer using the transfer-learning technique. ieee 57, 71194–71209 (2021) google scholar  santos-bustos, d.f., nguyen, b.m., espitia, h.e.: towards automated eye cancer classification via vgg and resnet networks using transfer learning. eng. sci. technol. int. j. 35, 101214 (2022). https://doi.org/10.1016/j.jestch.2022.101214 article  google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. in: international conference on learning representations (2014) google scholar  spanhol, f., oliveira, l.s., petitjean, c., heutte, l.: a dataset for breast cancer histopathological image classification. ieee trans. biomed. eng. 63(7), 1455–1462 (2016). https://doi.org/10.1109/tbme.2015.2496264 article  google scholar  vecchiotti, p., vesperini, f., principi, e., squartini, s., piazza, f.: convolutional neural networks with 3-d kernels for voice activity detection in a multiroom environment. in: esposito, a., faudez-zanuy, m., morabito, f.c., pasero, e. (eds.) multidisciplinary approaches to neural computing. sist, vol. 69, pp. 161–170. springer, cham (2018). https://doi.org/10.1007/978-3-319-56904-8_16 chapter  math  google scholar  veeling, b.s., linmans, j., winkens, j., cohen, t., welling, m.: rotation equivariant cnns for digital pathology. in: frangi, a.f., schnabel, j.a., davatzikos, c., alberola-lópez, c., fichtinger, g. (eds.) miccai 2018. lncs, vol. 11071, pp. 210–218. springer, cham (2018). https://doi.org/10.1007/978-3-030-00934-2_24 chapter  google scholar  vigneshkumar, k., sumathi, n.: a hybrid deep learning approach for breast cancer detection using vgg-19 and support vector machine. int. j. mech. eng. 83–93 (2021) google scholar  vo-le, c., son, n.h., van muoi, p., phuong, n.h.: breast cancer detection from histopathological biopsy images using transfer learning. in: ieee eighth international conference on communications and electronics (icce), pp. 408–412 (2021). https://doi.org/10.1109/icce48956.2021.9352069 download references acknowledgment this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001. author information authors and affiliations programa de pós-graduação em engenharia da computação e sistemas (pecs), universidade estadual do maranhão (uema), cidade universitária paulo vi – jd. são cristóvão, são luís, ma, 65055-310, brazil danyllo carlos silva e silva & omar andres carmona cortes departamento de computação (dcomp) – instituto federal do maranhão (ifma), av. getulio vargas, 04, são luis, ma, 65030-005, brazil omar andres carmona cortes campus grajaú – instituto federal do maranhão (ifma), br-226, grajaú, ma, 65940-000, brazil joão otávio bandeira diniz authors danyllo carlos silva e silvaview author publications search author on:pubmed google scholar omar andres carmona cortesview author publications search author on:pubmed google scholar joão otávio bandeira dinizview author publications search author on:pubmed google scholar corresponding author correspondence to omar andres carmona cortes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva e silva, d.c., cortes, o.a.c., diniz, j.o.b. (2025). the impact of double transfer learning in vgg architectures for metastasis breast cancer detection. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_32 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_32 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords convolutional neural networks double transfer learning breast cancer publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a framework for characterizing what makes an instance hard to classify | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a framework for characterizing what makes an instance hard to classify conference paper first online: 12 october 2023 pp 353–367 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a framework for characterizing what makes an instance hard to classify download book pdf download book epub maria gabriela valeriano  orcid: orcid.org/0000-0002-3631-156x9,10, pedro yuri arbs paiva  orcid: orcid.org/0000-0001-9768-75639, carlos roberto veiga kiffer  orcid: orcid.org/0000-0003-1122-069310 & … ana carolina lorena  orcid: orcid.org/0000-0002-6140-571x9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 586 accesses 1 citation abstract the health domain has been largely benefited by machine learning solutions, which can be used for building predictive models to support medical decisions. but, for increasing the reliability of these systems, it is important to understand when the models are prone to failures. in this paper, we investigate what can we learn from the instances of a dataset which are hard to classify by machine learning models. different reasons may explain why one or a set of instances are misclassified, despite the predictive model used. they can be either noisy, anomalous or placed in overlapping regions, to name a few. our framework works at two levels: the original base dataset and a meta-dataset built to reflect the hardness level of the instances. a two-dimensional hardness embedding is assembled, which can be visually inspected to determine sets of instances to scrutinize better. we show some analysis that can be undertaken in this hardness space that allow to characterize why some of the instances are hard to classify, with case studies on health datasets. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others comparative analysis of ensemble classifier and single base classifier in medical disease diagnosis chapter © 2020 a meta-heuristic approach to estimate and explain classifier uncertainty article open access 14 january 2025 machine learning in practice—clinical decision support, risk prediction, diagnosis chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization complexity data mining learning algorithms machine learning statistical learning 1 introduction machine learning (ml) models are regarded as promising solutions for revolutionizing healthcare [1]. these data-driven techniques may leverage knowledge from the large volumes of data continuously gathered by health systems and agents [2]. as a recent example, different predictive models were built during the covid-19 pandemic to assist diagnosis and prognosis of patients using hospital data [3, 4]. nonetheless, there are issues still preventing the widespread usage of ml predictive models in health decision making and planning. some concerns are the risk of bias and inappropriate or incomplete model performance evaluation [5]. given that health databases are not collected with the objective of data analysis in the first place, the data scientist and ml practitioner must deal with these issues by relying on their own experience or on ad-hoc procedures, which can bias the achieved results. data-centric frameworks have recently been proposed to aid these professionals in better assessing data quality and taking potential corrective measures, which will result in more reliable and trustful predictive models [6, 7]. one fruitful direction is to assess and monitor learning performance for each dataset instance and prevent relying only on averages over an entire dataset, as it is common practice [8]. furthermore, examining the performance at the instance level can reduce biases in the evaluation of ml models, as specific important groups of instances can be mistaken by the ml models, despite the obtainment of an overall high average predictive performance. this makes it important to know which instances are systematically misclassified by ml models and why they are so, a concern that has led to the recent literature of instance hardness analysis [9]. according to smith et al. [9], instance hardness can be measured as the average misclassification error of a pool of diverse classifiers when predicting the label of such instance. this metric allows identifying instances in a dataset that are inherently difficult to have their label predicted, despite of the classification technique used. however, going beyond, the literature also presents hardness meta-features which prospect possible reasons why the instance is hard to classify [9, 10]. recently, a framework for relating the classification performance of classifiers of distinct biases and the values of different hardness meta-features was framed [8, 11]. the relationship of this information is used to produce a 2-d hardness embedding where the instances are linearly distributed according to their hardness level. here this framework is extended for gaining data-centric insights by taking advantage of the organization of the dataset observations in the hardness embedding. these analyzes take place at two levels: the original input features and the meta-features. inspecting general trends in the input features of the hard instances of a dataset can be particularly informative and allow to involve the data domain experts in the ml pipeline. on the other hand, looking at the hardness meta-features allows data scientists to understand structural problems with such instances, which may be preventing the obtainment of a better predictive accuracy. this dual interplay allows for taking into account the main stakeholders involved in data analysis for decision support. in particular, our framework is applied here to datasets from the health domain, which is a critically requester reliable predictive models. as case studies, we consider a set of covid prognosis datasets from brazil, one of the countries hit hardest by the pandemic. the classification problem consists in predicting whether a covid-hospitalized patient will develop into an aggravated condition or not, information that can support decision making at both clinical and management levels at hospitals. our framework contributes on the following ways: providing a principled approach for identifying hard instances in a dataset: taking advantage of the hardness embedding of a dataset, where easy and hard instances are interposed, different types of analyzes are made possible. one of them is to define hardness and easiness footprints, which are dense regions of the representation space which encompass more hard and easy instances, respectively. this allows to define a principled strategy for choosing sets of instances to be further examined; providing meaningful insights on data quality and structure: the hardness profile of a dataset can be inspected for gaining insights about where and why ml models are failing; being actionable by different stakeholders: we combine two levels of knowledge. while inspecting relationships of the input features that lead to increased hardness levels can be a valuable tool for better data understanding and auditing with the aid of domain experts, the meta-features allow to prospect structural problems in the data which can be informative for data science and ml practitioners. this paper is structured as follows: section 2 presents some related work. section 3 presents our formulation. section 4 presents some experimental results. section 5 concludes the paper with some discussions. 2 related work ml techniques are data-hungry and a common mistake is to consider that more data will always result in more accurate predictive models. but some issues present in real data, such as noise, sparse regions and outliers, can impair the ml system’s performance, trustfulness and acceptance. therefore, data quality plays a central role in developing reliable ml systems. since predictions inside health scenarios involve highly delicate matters, data quality assessment and cleaning are even more critical. some recent works have used data-centered approaches to formulate strategies allowing to: assess how difficult it is to predict the class of each instance in a dataset [6, 8, 13]; understand why some instances are frequently misclassified, while also trying to explain the behavior of classifiers for such instances [8, 14]; predict test instances that will be reliably classified or not [7]; remove hard or ambiguous instances in a data sculpting strategy to improve ml algorithms performance [6, 13]; and identifying anomalous regions of the input feature space offering useful explanations and insights about data quality as well as model performance [6, 8, 15]. but there are still gaps to be filled that may help increasing the acceptance of such tools. one of them is providing more interpretable and actionable insights to the different stakeholders involved in data analysis. in order to increase the trustfulness and acceptance of ai systems, it is imperative that their decisions are transparent and can be scrutinized by different stakeholders. this has lead to the increase in the explainable artificial intelligence (xai) area [16]. many xai platforms and strategies are currently available and most of them focus on evaluating how the predictive input features influence the results of the ml models, such as permutation tests [17] and shapley values [18]. but there are other layers of explainability that are not so commonly explored in the literature, despite having potential to offer rich insights for better understanding the strengths and weaknesses of the ml models. one of them is working at a meta-level where general properties of the data that may lead to impairments in a good predictive performance are identified and characterized [9, 19]. taking steps towards this direction, we can extract local rules based on at most two input features for subsets of instances a given model struggles to classify correctly, that is, instances the model regards as hard to classify [15]. our work follows a similar direction, but with a more general approach for finding and characterizing the hard instances from a dataset. while combining the outputs of many models, we also leverage on meta-features able to describe why some instances are hard to classify, giving insights at different yet complementary perspectives. 3 formulation figure 1 presents an overview of the methodology followed in our paper. given a dataset \(\mathcal d\), a set \(\mathcal f\) of hardness meta-features describing the level of difficulty in classifying each instance in \(\mathcal d\) according to different perspectives is extracted. algorithmic performance \(\mathcal p\) is registered for each individual instance of the dataset for a pool of classification algorithms \(\mathcal a\). joining \(\mathcal f\) and \(\mathcal p\), a hardness embedding is produced where the instances are placed so as to present linear trends of difficulty level, from bottom right (easy) to top left (hard). next two types of analyzes are performed. one of them involves defining regions concentrating hard and easy to classify instances. they are called hardness and easiness footprints, respectively. by extracting patterns from the opposing footprints, we are able to obtain insights on instance hardness at different levels and perspectives. the second approach involves a visual inspection of the hardness embedding for identifying observations lying in regions of interest to be inspected. fig. 1. overview of the proposed framework. full size image 3.1 instance hardness analysis let us define \(\mathcal d\) formally as containing n pairs of labeled observations (\(\textbf{x}_i , y_i \)). each \(\textbf{x}_i \in \mathcal x\) is an instance described by m input features and is labeled in \(y_i \in \mathcal y\), where \(\mathcal y\) is a discrete and non-ordered set of classes. in addition, let \(h: \mathcal x \rightarrow \mathcal y\) denote a classification hypothesis, that is, a ml predictive model generated from \(\mathcal d\). the instance hardness of an instance \(\textbf{x}_i\) can be defined as the probability of misclassification when it is subject to a pool of different learning algorithms, that is: $$\begin{aligned} ih_{\mathcal {a}}\big ( \textbf{x}_i, y_i \big ) = 1 \frac{1}{|\mathcal {a}|} \sum _{j=1}^{|\mathcal {a}|} p\big (y_i | \textbf{x}_i, h_j(\mathcal d)\big ), \end{aligned}$$ (1) where \(p\big (y_i | \textbf{x}_i, h_j(\mathcal d)\big )\) is the probability the j-th ml model in the pool attributes \(\textbf{x}_i\) to its expected class \(y_i\). the intuition is that if a pool of diverse classifiers is unable to predict the expected class of \(\textbf{x}_i\) with a high probability (proxy to a high confidence), then this instance is intrinsically hard to classify. in turn, hardness meta-features (hm) can be used for prospecting possible reasons why an instance is hard to classify or not [9, 10]. summarizing, given an instance \(\textbf{x}_i\): kdn (k-disagreeing neighbors) gives the percentage of k nearest neighbors of \(\textbf{x}_i\) with label different from \(y_i\); n1i (fraction of nearby instances of different classes) gives the fraction of instances that do not belong to class \(y_i\) which are connected to \(\textbf{x}_i\) in a minimum spanning tree built from \(\mathcal d\); n2i (ratio of the intra-class and extra-class distances) takes the ratio of the distance \(\textbf{x}_i\) has to the nearest neighbor from its class \(y_i\) to the distance \(\textbf{x}_i\) has to its nearest enemy (nearest instance from another class \(y_j \ne y_i\)); lsci (local set cardinality) measures the size of the set containing instances from class \(y_i\) which are closest to \(\textbf{x}_i\) than to its nearest enemy (this set is named local set of the instance); lsr (local set radius) gives the radius of the previous set; u (usefulness) considers the number of local sets an instance belongs to; h (harmfulness) takes the number of instances \(\textbf{x}_i\) is nearest enemy of; cl (class likelihood) measures the likelihood an instance belongs to its class \(y_i\); cld (class likelihood difference) measures the difference between the likelihood an instance belongs to the class \(y_i\) to the likelihood it belongs to any other class \(y_j \ne y_i\); f1i (fraction of features in overlapping areas) gives the percentage of input features lying in overlapping regions of the classes; dcp (disjunct class percentage) gives the percentage of examples of class \(y_i\) which are placed in a same disjunct as \(\textbf{x}_i\), where disjuncts are defined by a decision tree algorithm; td (tree depth) gives the depth where the instance is classified in a decision tree model, which can be either pruned (\(td_p\)) or unpruned (\(td_u\)). all measures are standardized, by definition, in the [0, 1] interval, so that larger values are attributed to instances hard to classify according to the measured criterion. 3.2 hardness embedding the hardness embedding is built from a composition of several sets. all individual instances \(\textbf{x}_i \in \mathcal d\) compose the instance set \(\mathcal i\). the algorithm set \(\mathcal a\) comprises a portfolio of classification algorithms of distinct biases. the performance set \(\mathcal p\) records the predictive performance obtained by each algorithm in \(\mathcal a\) for every instance \(\textbf{x}_i\). the feature set \(\mathcal f\) contains the hm extracted from the instances \(\textbf{x}_i\). therefore, for each instance \(\textbf{x}_i\in \mathcal i\) and for each algorithm \(\alpha _j \in \mathcal a\), a feature vector \(\textbf{f}(\textbf{x}_i) \in \mathcal f\), and an algorithm performance metric \(p_m(\alpha _j, \textbf{x}_i)\) are measured. the process is repeated for all the instances in \(\mathcal i\) and algorithms in \(\mathcal a\), generating a meta-dataset \(\mathcal m = \{\mathcal i,\mathcal f,\mathcal a,\mathcal p\}\). the algorithms included in the pool \(\mathcal a\) are: bagging, gradient boosting (gb), support vector machine (svm, with both linear and rbf kernels), multilayer perceptron (mlp), logistic regression (lr) and random forest (rf). they present distinct learning mechanisms and are commonly adopted to solve ml classification tasks in the literature, specially when tabular datasets are concerned. to assess the performance of the algorithms, a five-fold cv strategy is used. their performance \(\mathcal p\) is evaluated using the log-loss error per instance. the meta-dataset \(\mathcal m\) is next subject to a meta-feature selection step so that only the most informative meta-features are preserved. this selection is performed using a neighborhood component feature selection (ncfs) algorithm [21], where we seek to find meta-features more related to the predictive performance of the classifiers in \(\mathcal a\). the resulting meta-dataset is then projected into a 2-d embedding presenting linear trends in the meta-features and algorithmic performance measures, also named instance space (is), as described in [8, 11]. the result is a projection matrix able to map each instance to the 2-d hardness embedding based on the selected meta-features values. a further rotation step is also included, so that the hard instances are placed in the upper left quadrant and the easier instances are in the lower right quadrant of the hardness embedding. figure 2 presents examples of hardness embeddings of three datasets for predicting the outcome of covid patient severity. the first and third datasets contain the symptoms and comorbidities of citizens undergoing diagnosis tests. in the second example, patients of an hospital from the são paulo metropolitan area (brazil) are prognosed based on a set of routine laboratory tests. in all cases, classes are represented by different shapes. the colors give the instance hardness level of the instances as computed by eq. 1. the higher the ih value, the redder the color; the lower the ih value, the bluer the color. there is a mix of patients that are easy or hard to classify per class, with both triangles and circles in the upper left or lower right regions of the plot. fig. 2. hardness embedding of three datasets with contrasting profiles: (a) severity; (b) hospital 1; (c) hospitalization. full size image whilst this plot can be colored according to all meta-features \(\mathcal f\) values, performance measures \(\mathcal p\) values of the classification algorithms \(\mathcal a\) and even by the original input features values of the dataset \(\mathcal d\), selections of specific regions can also be saved and scrutinized. 3.3 instance easiness and instance hardness footprints a footprint is defined in [22, 23] as a region in the instance space where an algorithm is expected to perform well based on inference from empirical performance analysis. here we use such a concept in order to define regions from the hardness embedding which concentrate more easy or hard instances. they are named easiness and hardness footprints, respectively. in order to construct these footprints, first the instances must be categorized as either easy or hard. this can be done by imposing different thresholds on ih values. in fact, we consider three categories: easy, hard and others. the later category encompasses instances that have intermediate hardness levels. we focus our analysis on contrasting easy and hard instances. ultimately, the objective is to understand why some instances are hard to classify. for segmenting the instances, we consider a threshold t, which must be higher than 0 and lower than 0.5. since the ih values are bounded in the [0, 1] interval, we have: \(\textbf{x}_i\) is considered easy if \(ih_{\mathcal a}(\textbf{x}_i, y_i) \le t\); \(\textbf{x}_i\) is considered hard if \(ih_{\mathcal a}(\textbf{x}_i, y_i) \ge 1-t\). then for each set of instances, easy or hard, the dbscan algorithm [24] is used to identify high density clusters in the instance (hardness) space. \(\alpha \)-shapes are used to construct hulls which enclose all the points within the clusters [25]. for each cluster hull, a delaunay triangulation creates a partition, and those triangles that do not satisfy a minimum purity (defined as the percentage of easy or hard instances enclosed within it, which is 0.7 by default) requirement are removed. the union of the remaining triangles gives the corresponding easiness or hardness footprint. as in [22], it is also possible to extract the following objective measures from the footprints: 1. the area of the footprint in the 2-d hardness embedding, normalized by the total area of the space; 2. the density of the footprint, computed as the ratio between the number of instances enclosed by the footprint and its area; 3. the purity of the footprint, which corresponds to the percentage of instances enclosed by the footprint from a given category, that is, the percentage of hard instances in the case of the hardness footprint and of easy instances in the easiness footprint. taking the hardness footprint as reference, a large area implies the dataset has many instances hard to classify, so that they occupy a large portion of the hardness embedding. a large density means such area is dense. and a large purity is observed when most of the instances enclosed in the hardness footprint are indeed hard. conversely, it is possible to define similar concepts for the easiness footprint and the easy instances. the easiness and hardness footprints’ areas can be regarded as an indicative of the hardness profile of the dataset. when the hardness area is large, the dataset and the underlying classification problem can be considered more difficult to solve. in contrast, easy datasets/problems will show a large easiness area. 4 experiments four datasets from the health domain, involving the prognosis of individuals diagnosed for covid-19 in brazil, were used in our experiments. datasets and their description are available in a public repositoryfootnote 1. summarizing, we have: 1. hospital 1 and hospital 2: prognosis of covid-19 severity for patients hospitalized at two distinct private hospitals from the são paulo metropolitan area, using some standard laboratory tests; 2. hospitalization and severity: prediction of the hospitalization length when a citizen from são josé dos campos-sp is hospitalized with a positive covid diagnosis (hospitalization) and if he/she will evolve to a severe condition (severity), based on initial symptoms and comorbidities reported. 4.1 footprint analysis first, we analyze the easiness and hardness footprints of three datasets with distinct hardness profiles. they are severity, hospital 1, and hospitalization datasets with easy, intermediate, and hard profiles, respectively. following the same order, the average accuracy rate achieved by the seven classifiers for each of these datasets was: 0.900 (std 0.003), 0.673 (std 0.030), and 0.398 (std 0.022). fig. 3. easy and hard footprints of datasets with contrasting profiles: (a) severity, (b) hospital 1; (c) hospitalization. t value = 0.048. full size image to determine such footprints, we did an experimental procedure varying the t value from 0.3 to 0.495, with steps of 0.02. next, we monitored the purity values of the resulting footprints. we chose the t values leading to the highest purity values for both easiness and hardness footprints. purest footprints are preferred, as they ensure most of the instances the footprint encloses are indeed easy/hard. figure 3 shows the obtained footprints: hardness in red and easiness in blue. as expected, the hardness footprint dimensions increase according to the dataset’s difficulty level. for instance, when the t value reaches 0.45, all instances are considered hard in the hospitalization dataset. fig. 4. areas, purities and densities of the hardness footprints for increasing t values. full size image figure 4 presents the variations of the area, density and purity values of the hardness footprint for increasing t values in the datasets. as expected, when t increases, the footprints’ areas grow, as more instances obey the inequalities involving the instance hardness values. but this increase is less accentuated for the datasets with easy and intermediate hardness profiles. purity is always over 0.7 and varies less for the severity dataset, while presenting an increasing tendency in the other problems. in contrast, whilst density values are stable in the hospitalization dataset, where the instances are concentrated in the center of the is, a tendency of decrease is observed for the other datasets (larger areas with less instances are encompassed in the footprints). similar plots can be generated for the easiness footprints. from the footprints we can already obtain insights by contrasting hard and easy instances. in fig. 5 we present boxplots contrasting the percentage of lymphocytes, one input feature of dataset hospital 1 for instances in the hardness (left boxplots) and easiness (right boxplots) footprints. colors matches the original class, where red stands for patients that developed to a severe condition and blue is the opposite class. clearly, there is an inversion on the expected features’ values when we compare the plots. in fact, a reduction in the proportion of lymphocytes is expected for severe cases and not the opposite [20]. this is already an indicative that hard instances are not following the expected pattern from the class they belong to. figure 5b presents boxplots of the kdn meta-feature values for the hardness (left) and easiness (right) footprints. easy instances have a low kdn value (are surrounded by elements sharing their class label), while hard instances have a high kdn value (are close to elements from the opposite class), despite the class they belong to. therefore, hard instances are contrasting their ground truth labels and are contained in overlapping areas. fig. 5. plots of the input features lymphocytes (%) and of meta-feature kdn for the hard instances (left) and easy instances (right) from dataset hospital 1. full size image 4.2 data sculpting fig. 6. average model performance scored by auc in a data sculpting process. the dotted line is the performance without removing any instance from hospital 1. full size image now we move our analysis to a practical usage of our framework in a data sculpting strategy. the idea is to remove the instances of the hardness footprint, assuming they are probably noisy and incorrect. the results of this procedure are illustrated in fig. 6. increasing thresholds of t are considered, which means that we start removing instances with the highest hardness levels. the average auc performance of the seven classifiers from the pool \(\mathcal a\) is shown in the y-axis, with standard deviations in gray. the performance achieved with no sculpting (keeping all instances) is shown as a dotted line. in fig. 6a, we train and test the classifier over the same dataset (hospital 1). as expected, removing the hard instances increases the accuracy achieved. in fig. 6b, the sculpting takes place at instances from dataset hospital 1, used for training, but test takes place at a different dataset, hospital 2. this second analysis already entails a higher difficulty level, since it presupposes the profiles of patients from hospital 1 are predictive of the conditions of the patients from hospital 2. in this case, there is only a subtle increase of accuracy over the original baseline (the dotted line), for t around 0.30 and 0.34. but the results are detrimental for other t values. that is, in this case the benefit occurs only when a small fraction of the hardest training instances are removed. the standard deviation values are also usually high, indicating a large variation of results. as in [6] when randomly removing ambiguous instances, removing hard instances in a dataset always increases model performance for the same dataset (fig. 6a). however, inside health scenarios the removal of hard instances can be dangerous since they are not necessarily noisy or incorrect. instead they can represent sub populations with atypical yet correct features’ values. so the models might become incapable to correctly classify these types of instances. this might explain why when sculpting data from hospital 1 and testing the results in data from hospital 2 the accuracy decreases after a threshold around 0.34. therefore, a more informed data sculpting process is needed. 4.3 analyzing groups of data with a domain expert here we inspect closely groups of hard instances in a dataset and show that more guided insights can be devised. this is possible taking advantage of the different possibilities of visual inspection of the hardness embedding, which can have the instances colored according to all features, meta-features and algorithmic performance values. in spite of the easy profile of the severity dataset (fig. 2a), there are still instances with a high level of difficulty placed in the upper left corner of the hardness embedding. most of the patients with a severe condition are placed in the bottom of the space, indicating this class is fairly easy to classify (group 1). in the top left of the space there are non-severe patients very hard to classify (group 2) and also some severe cases hard to classify in the top of the hardness embedding (group 3). patients without a severe condition are mostly placed in the center of the space (group 4), being also easy to classify. patients composing group 2 present at least one of three attributes heavily correlated with a severe condition. as a consequence, they have a low likelihood of belonging to their registered class (high cl values). to understand why these patients did not require an extended hospitalization or did not evolve to death, we inspected the raw databases, obtaining some valuable insights. from the 90 instances, 17 were mislabeled due to data preprocessing failures and should be corrected. the remaining were hospitalized but released before ten days of hospitalization. between then, nine patients were under 35 years old, which can explain the quick recovery. for others, the short hospitalization could be explained either because of disease recovery despite the apparent severity of the case, or by an external factor, like reduced hospital capacity for receiving such patients. the complete analysis of the other groups can be found in our repository. in the hospital 1 dataset, the nature of the attributes is different, since the blood tests have distinct reference values according to sex and age and can also change in the presence of other diseases and medication. these are some reasons why the predictive models perform poorly in this dataset. the is also offers insights which help to understand why, how, and where the ml models are failing. comparing meta-feature values for the severe individuals hard to classify reveals six types of hard instances. one of these groups is composed of eight instances labeled as non-severe, although they have the opposite profile. indeed, they have a low likelihood of belonging to their registered class (cl) and are surrounded by elements from a different class (n1). each individual in this group is a man between fifty and sixty years old, with low percentage of lymphocytes and c-reactive protein count as well as high percentages of neutrophils. investigating the original database it is possible to follow up the values of these same blood tests in the next few days. this inspection reveals that, besides their initial condition, all of them recovered before 14 days of hospitalization (the proxy for considering a severe patient in this dataset). therefore, this group is composed of outlier patients with a faster recovery than would be initially expected. another group is composed of 30 patients that also present a clinical condition similar to the severe class, but are labeled as non-severe patients. all meta-features are high for these patients (cl, dcp and n1), revealing they are difficult to classify according to different perspectives. analyzing the original database, we found an unexpected pattern. some of these patients are released without recovering (still presenting low percentage of lymphocytes and c-reactive protein counts) and sometimes in a worse condition. we could not recover the real reason for hospital discharge. these patients might have been transferred between hospitals, although being registered as medical releases in the system. these are more suitable candidates for data sculpting, since they might be noisy, as justified by the high values for their meta-features. these analysis illustrate how our framework can support the insertion of expert knowledge to improve data quality and model performance, in a more guided data sculpting process. 5 discussions and conclusions data-centric analysis have been taking an important role in ml, providing means to assure more trustfulness to the area. in this paper, we obtain interpretations on reasons why some observations of a dataset can be considered hard to classify. this is done considering both original input features and also meta-features which describe possible abstract structural reasons explaining why some instances are hard to classify. the choice of the instances to be examined is based on a projection of the dataset into a hardness embedding showing linear trends of classification difficulty according to different perspectives. we show the value of our proposal in gaining insights about data from the health domain. our framework was devised to the analysis of datasets in a tabular format, which are very abundant in the health domain. but plenty of non-structured data, such as image and text, are also gathered in this domain (eg. x-ray images). this does not prevent applying our framework to non-structured data. for instance, it is possible to extract structured representations from non-structured data by imputing them to some deep learning trained models [26]. for obtaining the hardness embedding, many meta-features and algorithmic performance measures must be extracted from the dataset first. this clearly implies in a computational cost which cannot be disregarded. running the analysis on a macbook pro os computer (m1 processor with 8 gb of memory), the average time taken to obtain the hardness embeddings for our datasets was around four minutes. in general, the hardness meta-features have at most a quadratic asymptotic cost on the number of observations the dataset has, mostly because some of them require building a distance matrix between all pairs of observations of the dataset. but the main cost is usually incurred by the cross-validation training-testing of the classifiers in the pool \(\mathcal a\). computational cost can be saved by reducing the amount of classifiers considered, although we can argue that the classifiers chosen here are common representatives everyone tests when tabular data are concerned, at least as baselines. another limitation of our current approach is that data must be labeled in advance. we shall investigate strategies to allow attributing a expected hardness level to unlabeled test instances. also, applying the analysis to imbalanced datasets must be done with care, since the minority class observations will tend to be pointed as hard just because they are outnumbered. and missing values must also be dealt with in advance. both issues can be solved in the future by including data re-sampling and missing value imputation strategies inside the framework. other types of analysis can also be devised and investigated. for instance, we can test the effects of including or excluding features from a dataset. as more informative features are present, we expect the easiness footprint to grow and the hardness footprint to shrink. other analysis possible is to establish when a prediction should better be discarded and be entrusted to a specialist instead and to devise a human-in-the-loop procedure. notes 1.(https://github.com/gabivaleriano/explaining_healthdata). references anderson, d., bjarnadottir, m.v., nenova, z.: machine learning in healthcare: operational and financial impact. in: babich, v., birge, j.r., hilary, g. (eds.) innovative technology at the interface of finance and operations, vol. 11, pp. 153–174. springer, cham (2022). https://doi.org/10.1007/978-3-030-75729-8_5 imrie, f., cebere, b., mckinney, e.f., van der schaar m.: autoprognosis 2.0: democratizing diagnostic and prognostic modeling in healthcare with automated machine learning. arxiv preprint arxiv:2210.12090 (2022) de moraes, b.a.f., miraglia, j., donato, t., filho, a.: covid-19 diagnosis prediction in emergency care patients: a machine learning approach. medrxiv, 2020-04 (2020) google scholar  fernandes, f.t., de oliveira, t.a., teixeira, c.e., de moraes batista, a.f., dalla costa, g., chiavegatto filho, a.d.p.: a multipurpose machine learning approach to predict covid-19 negative prognosis in são paulo, brazil. sci. rep. 11(1), 1–7 (2021) article  google scholar  wynants, l., et al.: prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. bmj 369 (2020). https://doi.org/10.1136/bmj.m1328 seedat, n., crabbe j., van der schaar, m.: data-suite: data-centric identification of in-distribution incongruous examples. arxiv preprint arxiv:2202.08836 (2022) seedat, n., crabbe j., bica, i., van der schaar, m.: data-iq: characterizing subgroups with heterogeneous outcomes in tabular data. arxiv preprint arxiv:2210.13043 (2022) paiva, p.y.a., moreno, c.c., smith-miles, k., valeriano, m.g., lorena, a.c.: relating instance hardness to classification performance in a dataset: a visual approach. mach. learn., 1–39 (2022) google scholar  smith, m.r., martinez, t., giraud-carrier, c.: an instance level analysis of data complexity. mach. learn. 95(2), 225–256 (2014) article  mathscinet  math  google scholar  arruda, j.l.m., prudêncio, r.b.c., lorena, a.c.: measuring instance hardness using data complexity measures. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12320, pp. 483–497. springer, cham (2020). https://doi.org/10.1007/978-3-030-61380-8_33 chapter  google scholar  paiva, p.y.a., smith-miles, k., valeriano, m.g., lorena, a.c.: pyhard: a novel tool for generating hardness embeddings to support data-centric analysis. arxiv preprint arxiv:2109.14430 (2021) valeriano, m.g., et al.: let the data speak: analysing data from multiple health centers of the são paulo metropolitan area for covid-19 clinical deterioration prediction. in: 2022 22nd ieee international symposium on cluster, cloud and internet computing (ccgrid), pp. 948–951. ieee (2022) google scholar  zheng, k., chen, g., herschel, m., ngiam, k.y., ooi, b.c., gao, j.: pace: learning effective task decomposition for human-in-the-loop healthcare delivery. in: proceedings of the 2021 international conference on management of data, pp. 2156–2168 (2021) google scholar  houston, a., cosma, g., turner, p., bennett, a.: predicting surgical outcomes for chronic exertional compartment syndrome using a machine learning framework with embedded trust by interrogation strategies. sci. rep. 11(1), 1–15 (2021) article  google scholar  prudêncio, r.b., silva filho, t.m.: explaining learning performance with local performance regions and maximally relevant meta-rules. in: xavier-junior, j.c., rios, r.a. (eds.) brazilian conference on intelligent systems, pp. 550–564. springer, cham (2022). https://doi.org/10.1007/978-3-031-21686-2_38 gunning, d., stefik, m., choi, j., miller, t., stumpf, s., yang, g.-z.: xai-explainable artificial intelligence. sci. rob. 4(37), eaay7120 (2019) google scholar  ojala, m., garriga, g.c.: permutation tests for studying classifier performance. j. mach. learn. res. 11(6) (2010) google scholar  ghorbani, a., zou, j.: data shapley: equitable valuation of data for machine learning. in: international conference on machine learning, pp. 2242–2251. pmlr (2019) google scholar  lorena, a.c., garcia, l.p., lehmann, j., souto, m.c., ho, t.k.: how complex is your classification problem? a survey on measuring classification complexity. acm comput. surv. 52(5), 1–34 (2019) article  google scholar  jafarzadeh, a., jafarzadeh, s., nozari, p., mokhtari, p., nemati, m.: lymphopenia an important immunological abnormality in patients with covid-19: possible mechanisms. scand. j. immunol. 93(2), e12967 (2021) article  google scholar  amankwaa-kyeremeh, b., greet, c., zanin, m., skinner, w., asamoah, r.k.: selecting key predictor parameters for regression analysis using modified neighbourhood component analysis (nca) algorithm. in: proceedings of 6th umat biennial international mining and mineral conference, pp. 320–325 (2020) google scholar  smith-miles, k., tan, t.t.: measuring algorithm footprints in instance space. in: 2012 ieee congress on evolutionary computation, pp. 1–8. ieee (2012) google scholar  muñoz, m.a., villanova, l., baatar, d., smith-miles, k.: instance spaces for machine learning classification. mach. learn. 107(1), 109–147 (2018) article  mathscinet  math  google scholar  khan, k., rehman, s.u., aziz, k., fong, s., sarasvady, s.: dbscan: past, present and future. in: the fifth international conference on the applications of digital information and web technologies, pp. 232–238. ieee (2014) google scholar  edelsbrunner, h.: alpha shapes-a survey. tessellations sci. 27, 1–25 (2010) google scholar  najafabadi, m.m., villanustre, f., khoshgoftaar, t.m., seliya, n., wald, r., muharemagic, e.: deep learning applications and challenges in big data analytics. j. big data 2(1), 1–21 (2015) article  google scholar  download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. the authors also thank the financial support of fapesp (grant 2021/06870-3) and cnpq. author information authors and affiliations instituto tecnológico de aeronáutica, praça marechal eduardo gomes, são josé dos campos, brazil maria gabriela valeriano, pedro yuri arbs paiva & ana carolina lorena universidade federal de são paulo, rua botucatu, são paulo, brazil maria gabriela valeriano & carlos roberto veiga kiffer authors maria gabriela valerianoview author publications search author on:pubmed google scholar pedro yuri arbs paivaview author publications search author on:pubmed google scholar carlos roberto veiga kifferview author publications search author on:pubmed google scholar ana carolina lorenaview author publications search author on:pubmed google scholar corresponding author correspondence to maria gabriela valeriano . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper valeriano, m.g., paiva, p.y.a., kiffer, c.r.v., lorena, a.c. (2023). a framework for characterizing what makes an instance hard to classify. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_24 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords explainability data-center instance-hardness publish with us policies and ethics profiles ana carolina lorena view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature autommlc: an automated and multi-objective method for multi-label classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper autommlc: an automated and multi-objective method for multi-label classification conference paper first online: 12 october 2023 pp 291–306 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) autommlc: an automated and multi-objective method for multi-label classification download book pdf download book epub aline marques del valle9,10, rafael gomes mantovani11 & ricardo cerri10  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 531 accesses abstract automated machine learning (automl) has achieved high popularity in recent years. however, most of these studies have investigated alternatives to single-label classification problems, presenting a need for more investigations in the multi-label classification scenario. from the automl point of view, the few studies on multi-label classification focus on automatically finding the best models based on mono-objective optimization. these tools train several multi-label classifiers in search of the one with the best performance in a single objective optimization process. in this work, we propose autommlc, a new multi-objective automl method for multi-label classification, to find the best models that maximize the f-score measure and minimize the training time. experiments were carried out with ten multi-label datasets and different versions of the proposed method using two multi-objective optimization algorithms: multi-objective random search and non-dominated sorting genetic algorithm ii. we evaluated the pareto front obtained by these methods through the hypervolume metric. the wilcoxon test demonstrated that autommlc versions had similar results for this metric. multi-label classification (mlc) algorithms were obtained from the pareto frontiers through the frugality score and compared with the baseline algorithms. the friedman test demonstrated that the mlc algorithms from autommlc versions had equal performances to f-score and training time. furthermore, they had better results than baseline algorithms for f-score and better results than most baseline algorithms for training time. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a systematic literature review on automl for multi-target learning tasks article 10 august 2023 a practical evaluation of automl tools for binary, multiclass, and multilabel classification article open access 21 may 2025 multi-objective evolutionary instance selection for multi-label classification chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition categorization learning algorithms machine learning predictive markers statistical learning 1 introduction designing high-performance machine learning (ml) model is an arduous task that requires expert knowledge [7]. automated machine learning (automl) is a research field that seeks to improve how ml applications are built by automating applications [20], creating automatically configured tools that perform well and are easy to use. however, automl is not only present in the search by ml models. it can also be applied to the pre-processing phases to the model interpretability phase, automating a specific sub-task of the ml pipeline or even the entire pipeline [18, 20]. automl can solve single-label classification (slc) and multi-label classification (mlc) problems. each dataset instance is associated with a single label in slc, while in mlc, each instance is associated with a subset of labels [9, 17]. most research dealing with automl and mlc focuses on automating searching for the best algorithms and hyperparameters used on the model induction task [12,13,14, 18, 19]. given the search space, an optimization algorithm, and an evaluation criterion, automl trains and evaluates several models in search of the best model evaluated. as the search space comprises different classification algorithms and their corresponding hyperparameters, these studies deal with instances of combined algorithm selection and hyperparameter optimization (cash) optimization problems [7]. furthermore, in these studies, the optimization algorithms for mlc are mono-objective, with an evaluation criterion usually related to the classifier performance. however, we could evaluate multi-label classifiers considering more than one evaluation criterion [19]. thus, we would have a multi-objective optimization problem, where a task involves more than one objective function to be maximized or minimized and produces a set of optimal solutions rather than a single solution, known as pareto optimal solutions [3]. multi-objective ml balances distinct evaluation criteria (objective functions) to improve model generalization and avoid models with local optima [8]. in this study, we propose a multi-objective automl method for mlc, denominated automated multi-objective multi-label classification (autommlc), optimizing both the f-score performance measure and the training time of the classifiers. thus, we want to automatically find models that maximize the f-score and have a lower computational time. we performed experiments evaluating the performance of autommlc employing two different multi-objective algorithms: multi-objective random search (mors) [8] and non-dominated sorting genetic algorithm ii (nsga-ii) [2]. these algorithms were used because mors serves as reasonable baselines, and nsga-ii is one of the most popular multi-objective evolutionary algorithms [8]. in addition, we evaluated the performance of autommlc against other baselines simplest. the main contributions of this paper are: i) the proposal of a new automl method for mlc, contributing to a subject still little explored, and ii) incorporating multi-objective criteria to search for the best ml classifiers. the remainder of this article is structured as follows: in sect. 2, we review mlc, multi-objective optimization algorithms, and related works; in sect. 3, we describe the experimental methodology adopted in this study; in sect. 4, we present the results and discuss their implications; finally, sect. 5 presents the findings and future works. 2 background 2.1 multi-label classification given a label set, slc associates a single label with each instance in the dataset. in mlc, an instance can be classified into two or more labels simultaneously. for example, textual data can be classified simultaneously with multiple labels, such as images, videos, music, and emotions. since \(l = \{l_j: j = 1, ..., q\}\) is the finite label set, mlc associates to each instance \({\textbf {x}}_i\) of the dataset a set of labels \({\textbf {y}}_i\) such that \({\textbf {y}}_i \subseteq l\) [17]. according to madjarov et al. [9], mlc algorithms can be characterized into three approaches: i) problem transformation (pt), which transforms multi-label problems into single-label problems and then employs traditional classification algorithms; ii) algorithm adaptation (aa), which creates algorithms that handle multiple labels simultaneously; and iii) ensemble, which employs a set of mlcs as base classifiers. 2.2 multi-objective optimization multi-objective optimization aims to solve real problems with conflicting objectives, subject to possible constraints of the problem [3]. in the context of multi-objective optimization for ml problems, karl et al. [8] formally define the problem according to eq. 1. $$\begin{aligned} \begin{array}{llll} \min _{\lambda \in \lambda }~~c(\lambda ),~~subject~to:\\ k_1(\lambda )= 0, ...,k_n(\lambda )= 0\\ \hat{k}_1(\lambda )\ge 0, ...,\hat{k}_n(\lambda )\ge 0 \end{array} \end{aligned}$$ (1) since \(\varlambda \) is the set of possible ml candidate solutions, we want to evaluate each candidate solution \(\lambda \). the \(\lambda \) evaluation is calculated from the m evaluation criteria of the ml models: \(c_{1}:\varlambda \rightarrow \mathbb {r},\dots ,c_{m}:\varlambda \rightarrow \mathbb {r}\), with \(m\in \mathbb {n}\). the objective is to find, among all the candidate solutions, the one with the minimum evaluation \(c(\lambda )\), being \(c:\varlambda \rightarrow \mathbb {r}^m\). the multi-objective optimization result is a set of optimal solutions rather than a single solution [3]. this set is known as pareto optimal solutions or pareto frontier. they have different values for the objectives, where a solution can be good considering one of the objectives but poor concerning the others. it is up to the users to choose the best solution among the resulting solutions. most multi-objective algorithms use the concept of dominance to find pareto optimal solutions. so for a minimization problem, solution \(\lambda \) dominates \(\lambda ^{'}\) if only \(\forall i \in \{1, ..., m\}: c(\lambda _i) \le c(\lambda ^{'}_{i})\) and \(\exists j \in \{1, ..., m\}: c(\lambda _j) < c(\lambda ^{'}_{j })\) [8]. in other words, \(\lambda \) dominates \(\lambda ^{'}\) if the goal values of \(\lambda \) are equivalent to or better than the goal values of \(\lambda ^{'}\) and \(\lambda \) surpasses \(\lambda ^{'}\) in at least one of the objectives. any other solution does not dominate pareto optimal solutions. based on the dominance concept, we describe below two algorithms: multi-objective random search (mors): random search (rs) is one of the most popular mono-objective optimization algorithms. given t trials, the mono-objective rs algorithm randomly produces candidate solutions and evaluates these solutions in search of the best one. according to karl et al. [8], the multi-objective version of rs can be derived from rs and the concept of dominance. for this, we assess the t random samples and return the set of non-dominated solutions, the pareto front. non-dominated sorting genetic algorithm ii (nsga-ii): nsga-ii is an evolutionary algorithm [2]. given an initial population, the population evolves over generations by crossover, mutation, and tournament selection operations. the fittest individuals, among parents and offspring, are selected for subsequent generations. fitness is measured by non-domination rank and crowding distance density estimate. thus, the populations are composed of individuals from the first pareto frontiers with good dispersion. we can find the quality of the pareto frontier using distance-based (requires the true pareto frontier) or volume-based (calculates the volume from the frontier to a point) indicators. hypervolume is a volume-based quality indicator. for a minimization problem, it is the region bounded by the pareto frontier points and by a reference point [6]. the larger the calculated hypervolume, the more the pareto frontier minimizes the objectives of the problem. 2.3 related works most studies covering automl and mlc explored mono-objective optimization instances. given a search space composed of single-label algorithms, multi-label algorithms, and hyperparameters [12,13,14, 18, 19], optimization algorithms explore the search space and evaluate the candidate multi-label classifiers in search of the best classifier. the studies used different evaluation criteria to evaluate candidate classifiers during optimization, aggregating different performance measures composing a single evaluation criterion or using one knowing evaluation criteria, such as hamming loss and f-score. while handling automl for mlc as a mono-objective optimization problem, the related studies usually propose new algorithms and compare the results with other optimizers already used for mlc. in this sense, de sá et al. [13] used genetic algorithms as the optimization algorithm, de sá et al. [12] used grammar-based genetic programming, and de sá et al. [14] extended their previous works by also employing bayesian optimization as an optimization algorithm. other authors explored the search space with networks of hierarchical tasks [19], used hyperband optimization, bayesian optimization, and the combination of bayesian optimization and hyperband as optimization algorithms [18]. the works focus on automating the search for ml models. some suggest employing automl in the data pre-processing and post-processing steps [12, 13]. all works employ one evaluation criterion in the optimization process. there are suggestions to study the evaluation criteria in the application domain to use the most appropriate ones [14] and employ multi-objective optimization to improve the generalization of the models [19]. the authors cite the need to improve the search space [13, 18] using meta learning, for example [18]. 3 experimental methodology 3.1 datasets we evaluated autommlc in ten multi-label datasets traditionally used in studies on mlc [9, 18] from different application domains: biology, multimedia (audio and music), and text. table 1 presents the main characteristics of the datasets used. the number of instances in the collection ranges from [194, 7.395], the number of features ranges from [19, 1.836], and the number of labels from [6, 374]. all these datasets were available for download from the mulan websitefootnote 1. table 1. mlc datasets used in experiments. for each dataset, it is presented: its domain, number of instances (#i), number of features (#f), and number of labels (#l) these statistics values are shown before and after the preprocessing.full size table 3.2 preprocessing all datasets were split using a 10-fold cross-validation resampling strategy. to ensure the same proportion of instances of each class in the folds, we used the iterative stratification strategy [15]. it was coded using the scikit-multilearn library [16]. even after applying the iterative stratification, some folds ended without positive instances for some classes happened because some datasets originally had very few positive instances for the same labels. thus, before the iterative stratification, we also preprocessed the datasets. the data preprocessing step included data organization, cleaning, and data transformation. none of the selected datasets had missing values. so, in the preprocessing step, we treated the following situations: duplicated data: we removed instances with duplicated values; and labels associated with few instances: we also removed labels with ten or fewer positive instances. with this, we also excluded unlabeled instances. in the data transformation step, we applied min-max normalization on all datasets to prevent different features from having outliers or features prevailing over others. the three right-most columns of table 1 show the main statistics of the datasets after the preprocessing. one may note a change in the number of labels in some datasets: birds, corel5k, enron, genbase, and medical. in these datasets, the preprocessing also decreased the available number of instances. once the preprocessing was finished, we generated the resampling 10-fold through iterative stratification, as mentioned before. the datasets and codes used in this work are freely availablefootnote 2. 3.3 search space during its execution, our autommlc assesses different ml models to find the best model. the search space definition is essential in this process, as it defines the possible ml algorithms (or neural architectures) and hyperparameters values for inducing models. figure 1 presents the search space of autommlc, composed of 11 slc algorithms and 10 mlc algorithms of the scikit-learn [10] and scikit-multilearn libraries [16]. the slc algorithms included in the search space are k-nearest neighbors (knn), support vector machine (svm), adaboost, bernoulli naïve bayes (bnb), decision tree (dt), extra tree (et), gradient boosting (gb), logistic regression (lr), multinomial naïve bayes (mnb), random forest (rf), multi-layer perceptron (mlp) and stochastic gradient descent (sgd). the mlc algorithms included in the search space belong to the following categories: ensemble: random k labelsets disjoint (rakeld); pt: binary relevance (br) and classifier chains (cc); and aa: br and knn in version a (brknna), br and knn in version b (brknnb), dt, multi-label knn (mlknn), multi-label support vector machines (mltsvm), multi-label adaptive resonance associative map (mlaram), and rf. in fig. 1, rakeld algorithm is an label powerset (lp) ensemble and has a base classifier. this base classifier is one of the slc algorithms listed. the multi-label algorithms of the pt category also have a base classifier, which can be any of the 11 single-label algorithms included in the search space. fig. 1. search space used in our experiments. it is composed of slc and mlc algorithms. mlc algorithms belong to aa, ensemble, and pt categories. full size image it is essential to mention that all these slc and mlc algorithms can have tunable hyperparameters. therefore, the search space also contains information about the possible values of each hyperparameter. in this work, we stored the possible values of the hyperparameters in arrays, which could assume discrete or continuous values. more information about the search space algorithms (their hyperparameters and values) is in supplementary materialfootnote 3. 3.4 evaluation criteria and objectives the evaluation criteria in our ml model evaluation approach were the training time and the measure-based-example f-score (also known as f-measure). the selected criteria are related to the computational time and the performance of the models, respectively. f-score is the harmonic mean between precision and recall and can assume values between [0, 1]. equation 2 presents the definition of f-score, where n is the number of instances in the dataset, \({\textbf {y}}_i\) is the label set for the instance \({\textbf {x}}_i\), and \(h({\textbf {x}}_i)\) is the predicted label set for instance \({\textbf {x}}_i\) [9, 17]. $$\begin{aligned} f-score = \frac{1}{n} \sum \limits _{1}^{n}\frac{2\,h({\textbf {x}}_i)\cup {\textbf {y}}_i|}{|h({\textbf {x}}_i)| + |{\textbf {y}}_i|} \end{aligned}$$ (2) given that we want to maximize the f-score and minimize training time, we defined the objectives as training time and 1 f-score. 3.5 autommlc\(_{{\textbf {mors}}}\) \({\textrm{autommlc}_\textrm{mors}}\) is the autommlc method with the mors algorithm. this method evaluates candidate solutions randomly sampled from the search space and returns the set of non-dominated ones (pareto frontier). it receives the termination criterion and the training and testing set as parameters. then, it randomly obtains a mlc algorithm and its hyperparameters from the search space. in sequence, it trains this multi-label algorithm with the training data and calculates training time and f-score. the evaluation criteria (training time and 1 f-score) are necessary to find the pareto front, obtained as described by deb et al. [2]. our implementation allows executing the method automl by fold and the training and evaluation of the models in parallel from a pool of threads, speeding up the \({\textrm{autommlc}_\textrm{mors}}\). 3.6 autommlc\(_{{\textbf {nsga}}}\) individual. in the \({\textrm{autommlc}_\textrm{nsga}}\) optimization process, we represented candidate solutions (individuals) as an array of integers. they are created dynamically, and their structure is related to the search space. figure 2 depicts examples of individuals and a reduced search space, considering only two candidates mlc algorithms (mlknn and br). this illustrative figure will be used to explain the individual. each array position corresponds to a gene, and the first position specifies the mlc algorithm. we can get the mlc algorithms from the search space ([mlknn, br]). thus, the possible values in the first gene are zero (mlknn) or one (br), which are indexes into the array of available mlc algorithms. fig. 2. in the figure, there are two individuals and reduced search spaces for the single-label and multi-label algorithms. full size image the following individual’s genes are the hyperparameters of the mlc algorithms in the multi-label search space. they are allocated in the individual in the same order they appear in the multi-label search space. thus, they refer to the hyperparameters (fig. 2): k and classifier. the other individual’s genes are the hyperparameters of the slc algorithms in the single-label search space. they are also allocated in the individual in the same order as in the single-label search space. therefore, the other positions refer to the hyperparameters (fig. 2): c, gamma and n_neighbors. the search space contains the possible values that a hyperparameter can assume. a hyperparameter can take on a finite number of possible values. therefore, the hyperparameter space is discrete, so it was represented using arrays in search space. thus, the value of a gene in the individual indicates one of the positions (indices) of this hyperparameter array. for example, the gamma hyperparameter in fig. 2 has a set of possible values (\([2e-5, 2e-3, 2e-1, 2e1]\)). the value of the gene referring to this hyperparameter in the individual is 3, which means \(gamma = 2e1\). following the described logic, individuals are randomly initialized based on the search space. figure 2 shows the decoding of the individual. the individual with the first gene equal to 0 has mlknn as a mlc algorithm. this algorithm has hyperparameter k with hyperparameter space: [1, 3, 5, 7, 9]. the individual’s gene referring to this hyperparameter contains a value of 2. therefore, we have the mlknn algorithm with k equal to 5. the hatched genes in fig. 2 have values, but they are not used in the mlknn algorithm. on the other hand, when the individual’s first gene is 1, the mlc algorithm is br. this algorithm has the hyperparameter classifier with hyperparameter space: [svm, knn]. the individual’s gene referring to this hyperparameter contains a value of 0. therefore, we have the br algorithm with the classifier equal to svm. we must also decode the svm base classifier (\(c = 2e-1\) and \(gamma = 2e1\)). autommlc\(_{{\textbf {nsga}}}\) implementation. \({\textrm{autommlc}_\textrm{nsga}}\) is the autommlc method with the nsga-ii algorithm. in the implementation, we used the nsga-ii algorithm from pymoo, a multi-objective optimization framework in python [1]. for this, we defined the search space and developed classes that inherit from the pymoo framework classes: sampling: we specified how to create an initial population. our algorithm selects the initial population randomly, selecting a valid value for each gene in the search space. problem: in this class, we initialized the number of objectives and defined how to evaluate the candidate solution. this implementation involved decoding the individual in a mlc algorithm and assessing the correspondent classification model, measuring the training time, and calculating the f-score from the model. this process was similar to the defined mors and was implemented in parallel. mutation: we developed the mutation operator, which verifies whether individuals may or may not mutate with a 5% probability. the mutation alters one of the individual’s genes with a valid search space value. in the crossover, we employed uniform crossover available in the pymoo framework, where each child’s gene can be from one of the parents with a probability of 50%. the individuals resulting from crossover and mutation are valid, as the search space guarantees the consistency of the generated individuals. 3.7 baseline algorithms the multi-label search space algorithms were used as baseline algorithms for comparison. we adopted the svm algorithm as the base classifier of the mlc algorithms of the problem transformation approach. svm was chosen because it is traditionally used as a base classifier [9]. for the hyperparameters penalty of the mltsvm algorithm, we use the value \(2e^{-3}\) (one of the values of our search space). we adopted the default hyperparameters of the implementations available in the textitscikit-learn [10] and scikit-multilearn [16] libraries for the other hyperparameters. 3.8 settings for running we executed the \({\textrm{autommlc}_\textrm{nsga}}\) with a population of size 100 and 100 generations, totaling 10.000 candidate solutions evaluated. the mutation rate was set to \(5\%\). we adopted early stopping, so if there was no change in the average of the objectives (training time and f-score) for ten consecutive generations, \({\textrm{autommlc}_\textrm{nsga}}\) ended. the budget size for \({\textrm{autommlc}_\textrm{mors}}\) was the same, i.e., it evaluated 10.000 random candidate solutions. \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) were implemented in parallel with a pool of 10 threads. the runtime of the mlc algorithms was limited to 20 minutes. limiting the runtime of these algorithms is expected and commonly adopted in the automl context [12,13,14, 18, 19]. given the complexity of some multi-label algorithms and the dimensionality of some datasets, our goal with the runtime limit was to ensure automl results promptly. if the algorithm runtime exceeds this pre-established limit threshold, the objectives receive their maximum values, 1 for 1 f-score and 20 min for training time. these values were also the reference point for the hypervolume calculation. the runtime of the baseline algorithms was also limited to 20 min. in addition, we also stored all the evaluated candidate solutions and their corresponding objectives’ values. those values were stored for \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\). thus, before considering a new candidate solution, we checked if it was among the solutions already evaluated. if so, we used the objectives already calculated, avoiding training the same classifier (algorithm and hyperparameters) more than once. with this requirement, we would like to decrease the total runtime required by the autommlc. 3.9 statistical analysis to compare the \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) algorithms, we consider the hypervolume obtained from the pareto frontiers of each fold of the datasets. we use the wilcoxon non-parametric test [4] to evaluate statistically the hypervolumes with a significance level of 5%. our null hypothesis was that autommlc versions performed equally, producing equivalent hypervolumes. we selected pareto frontier algorithms using frugality score to compare \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) with the baseline algorithms. the frugality score is a measure for evaluating algorithms that combines a measure of performance and resources [5]. in this work, we combined the f-score with the training time, where the f-score is penalized with training time, as shown in eq. 3. $$\begin{aligned} frugality = f\_score \frac{0.5}{1 + \frac{1}{training\_time}} \end{aligned}$$ (3) the result of \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) on a fold/dataset is the pareto frontier. we calculated the frugality score for the pareto frontier mlc algorithms and selected the highest score. thus, we have a mlc algorithm representing the results of autommlc in each fold/dataset. we used two friedman tests [4] with a significance level of 5% to compare the mlc algorithms representing \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) with the baseline algorithms to f-score and training time. our null hypotheses were: the mlc algorithms had the same f-scores, and the mlc algorithms had the same training time. if the null hypotheses were rejected, the nemenyi post-hoc test was applied, where the performances of two mlc algorithms are significantly different if the corresponding mean ranks differ by at least one critical difference value. 4 results and discussion 4.1 autommlc\(_\textrm{nsga}\) and autommlc\(_\textrm{mors}\) \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) were run as described in the experimental setup (sect. 3.8). figure 3 presents the average hypervolume (k-folds) for the ten datasets obtained while evaluating candidate solutions. hypervolume calculation occurred every 100 evaluations for \({\textrm{autommlc}_\textrm{mors}}\) and at the end of each generation for \({\textrm{autommlc}_\textrm{nsga}}\). \({\textrm{autommlc}_\textrm{nsga}}\) converges before 10.000 evaluations (or 100th generation), and its execution ends. convergence means that after determinate generation, there were no changes in the values of the objectives and, consequently, in the value of the hypervolume. in \({\textrm{autommlc}_\textrm{mors}}\), there is also this tendency. as a result, the hypervolume stops increasing after a certain number of evaluations. however, this method has always evaluated the 10.000 candidate solutions. \({\textrm{autommlc}_\textrm{mors}}\) evaluates more candidate solutions than \({\textrm{autommlc}_\textrm{nsga}}\), the runtime of this method is more. both autommlc methods consult candidate solutions that have already been evaluated, contributing to the execution time of methods. furthermore, the methods stop training the mlc algorithms when the training time exceeds 20 minutes. this fact occurs more frequently in \({\textrm{autommlc}_\textrm{mors}}\) and increases your runtime. the evolutionary process of \({\textrm{autommlc}_\textrm{nsga}}\) contributes to the lower number of interrupted training since candidate solutions subject to interruption cease to be part of the population in the first generations. fig. 3. convergence graphs of mlc methods. each curve represents the average hypervolume of the 10-folds calculated as the solutions are evaluated. full size image fig. 4. boxplot of hypervolumes. the hypervolumes were calculated from the pareto frontier resulting from \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\). each subplot brings the hypervolumes of the 10-fold from the datasets for both versions of the autommlc. full size image fig. 5. heat map of selected mlc algorithms by autommlc. we counted the number of times the mlc algorithms were selected after executing the 10-folds of a dataset. full size image \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) results were the multi-objective algorithms results, the pareto frontier. thus, we could evaluate it by calculating the hypervolume. figure 4 shows the boxplot of hypervolumes calculated for the 10-fold of each dataset, considering the results of the autommlc with nsga-ii and mors algorithms. as we wanted to minimize the objective values, the larger the hypervolume, the better the obtained pareto frontier. analyzing fig. 4, we cannot assume which multi-objective algorithm got the best results. then we did the wilcoxon test. our null hypothesis was that the hypervolumes obtained in the autommlc versions were the same. the null hypothesis was accepted for all datasets. that is, \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) had equal results for the hypervolume. the pareto frontiers resulting from \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) comprise a set of mlc algorithms, one of the ten mlc algorithms in the search space (sect. 3.3). figure 5 shows heat maps with the occurrences of each mlc algorithm in the \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) solutions after running the algorithms with the ten datasets. in the graphs of fig. 5, there is a scale difference. this difference occurred because \({\textrm{autommlc}_\textrm{mors}}\) randomly selected from the search space many repeated mlc algorithms (as well as their hyperparameters), which ended up being part of the pareto frontier. among the most repeatable mlc algorithms is brknna, followed by brknnb. except for the replications produced by \({\textrm{autommlc}_\textrm{mors}}\), both methods produced frontiers with different mlc algorithms. in the medical dataset, for example, the pareto frontiers resulting from \({\textrm{autommlc}_\textrm{mors}}\) (in the 10-folds) had the mlc algorithms brknna, brknnb, br, cc, rakeld, and decision tree. these mlc algorithms also composited the pareto frontiers resulting from \({\textrm{autommlc}_\textrm{nsga}}\) for the medical dataset. the two versions of autommlc never included mltsvm and mlknn algorithms in their solutions. 4.2 comparison of autommlc with baselines algorithms the friedman test compared the mlc algorithms selected by the frugality score of \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) solutions with the baseline algorithms to f-score and training time. the null hypothesis was that the mlc algorithms presented equal performances. the null hypothesis was rejected for the test with f-score and with training time. fig. 6. comparison of f-scores and training times values of mlc algorithms, according to the nemenyi test. groups of algorithms that are not significantly different are connected. \(a._{nsga}\) is \({\textrm{autommlc}_\textrm{nsga}}\) and \(a._{mors}\) is \({\textrm{autommlc}_\textrm{mors}}\). full size image figure 6 presents the critical difference diagram for the nemenyi test to the f-score. the analysis indicates no statistical difference between the two best mlc algorithms: algorithms from \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\). figure 6 shows the critical difference diagram for the nemenyi test to the training time. in this test, the mlc algorithms with the best training time are brknnb and brknna, followed by algorithms from \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\). the mlc algorithms representing \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) do not have statistical differences for f-score and training time. however, these algorithms are superior to baseline algorithms regarding f-score and superior to br, cc, dt, mlaram, mlknn, mltsvm, rakeld, and random forest algorithms regarding training time. 5 conclusions this work presented the autommlc, a new multi-objective automl method for mlc, which seeks solutions that maximize the f-score and minimize the training time of the classifiers. autommlc was developed using nsga-ii and mors optimization algorithms. we ran both versions of the autommlc with ten datasets, the same search space, and the settings for running. the wilcoxon test indicated that \({\textrm{autommlc}_\textrm{nsga}}\) and \({\textrm{autommlc}_\textrm{mors}}\) were statistically equal results concerning hypervolume, but \({\textrm{autommlc}_\textrm{nsga}}\) was better runtime than \({\textrm{autommlc}_\textrm{mors}}\). regarding baseline algorithms, autommlc versions were statistically better than the f-score and had a lower training time than most baseline algorithms. the results presented are still preliminary on a recent research topic that requires further exploration. thus, in future work, we need to expand the analysis of data from autommlc runs and the resulting pareto frontiers. in addition, we need to study other ways to select pareto frontier solutions to compare with other solutions already available in the literature. we can expand the comparisons, considering other automl solutions as baselines, be they mono-objective or multi-objective optimization. we can also produce new results by improving the search space. for this, we need to improve our representation of the search space to allow algorithms with deeper hierarchical levels. we can also add more algorithms to the search space, like the multi-label extension to weka (meka) library algorithms [11]. finally, more objectives could be used and have different weights in multi-objective optimization. notes 1.https://mulan.sourceforge.net/datasets-mlc.html. 2.https://github.com/alinedelvalle/autommlc.git. 3.https://github.com/alinedelvalle/autommlc/blob/main/bracis_supplementary_material.pdf. references blank, j., deb, k.: pymoo: multi-objective optimization in python. ieee access 8, 89497–89509 (2020) article  google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  google scholar  deb, k., deb, k.: multi-objective optimization, pp. 403–449. springer, us, boston, ma (2014) google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) mathscinet  math  google scholar  evchenko, m.m.: frugal learning:applying machine learning with minimal resources (2016) google scholar  fonseca, c., paquete, l., lopez-ibanez, m.: an improved dimension-sweep algorithm for the hypervolume indicator. in: 2006 ieee international conference on evolutionary computation, pp. 1157–1163 (2006) google scholar  he, x., zhao, k., chu, x.: automl: a survey of the state-of-the-art. knowl.-based syst. 212, 106622 (2021) article  google scholar  karl, f., et al.: multi-objective hyperparameter optimization an overview (2022) google scholar  madjarov, g., kocev, d., gjorgjevikj, d., džeroski, s.: an extensive experimental comparison of methods for multi-label learning. pattern recogn. 45(9), 3084–3104 (2012), best papers of iberian conference on pattern recognition and image analysis (ibpria’2011) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  read, j., reutemann, p., pfahringer, b., holmes, g.: meka: a multi-label/multi-target extension to weka. j. mach. learn. res. 17(21), 1–5 (2016) mathscinet  math  google scholar  de sá, a.g.c., freitas, a.a., pappa, g.l.: automated selection and configuration of multi-label classification algorithms with grammar-based genetic programming. in: auger, a., fonseca, c.m., lourenço, n., machado, p., paquete, l., whitley, d. (eds.) parallel problem solving from nature ppsn xv, pp. 308–320. springer international publishing, cham (2018) google scholar  de sá, a.g.c., pappa, g.l., freitas, a.a.: towards a method for automatically selecting and configuring multi-label classification algorithms. in: proceedings of the genetic and evolutionary computation conference companion, pp. 1125–1132. gecco 2017, association for computing machinery, new york, ny, usa (2017) google scholar  de sá, a.g.c., pimenta, c.g., pappa, g.l., freitas, a.a.: a robust experimental evaluation of automated multi-label classification methods. in: proceedings of the 2020 genetic and evolutionary computation conference, pp. 175–183. gecco 2020, association for computing machinery, new york, ny, usa (2020) google scholar  sechidis, k., tsoumakas, g., vlahavas, i.: on the stratification of multi-label data. in: gunopulos, d., hofmann, t., malerba, d., vazirgiannis, m. (eds.) machine learning and knowledge discovery in databases, pp. 145–158. springer, berlin heidelberg, berlin, heidelberg (2011) chapter  google scholar  szymanski, p., kajdanowicz, t.: scikit-multilearn: a scikit-based python environment for performing multi-label classification. j. mach. learn. res. 20(1), 209–230 (2019) google scholar  tsoumakas, g., katakis, i., vlahavas, i.: mining multi-label data, pp. 667–685. springer, us, boston, ma (2010) google scholar  wever, m., tornede, a., mohr, f., hüllermeier, e.: automl for multi-label classification: overview and empirical evaluation. ieee trans. pattern anal. mach. intell. 43(09), 3037–3054 (2021) article  google scholar  wever, m.d., mohr, f., tornede, a., hüllermeier, e.: automating multi-label classification extending ml-plan. in: 6th icml workshop on automated machine learning, long beach, ca, usa (2019) google scholar  zöller, m.a., huber, m.f.: benchmark and survey of automated machine learning frameworks. j. artif. int. res. 70, 409–472 (2021) mathscinet  math  google scholar  download references acknowledgments the authors would like to thank the brazilian research agencies fapesp, capes and cnpq for financial support. author information authors and affiliations federal institute of education, science and technology of the south of minas gerais, muzambinho, mg, brazil aline marques del valle department of computer science, federal university of são carlos, rodovia washington luis, km 235, são carlos, sp, brazil aline marques del valle & ricardo cerri federal university of technology paraná (utfpr), campus of apucarana, curitiba, pr, brazil rafael gomes mantovani authors aline marques del valleview author publications search author on:pubmed google scholar rafael gomes mantovaniview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar corresponding author correspondence to aline marques del valle . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper del valle, a.m., mantovani, r.g., cerri, r. (2023). autommlc: an automated and multi-objective method for multi-label classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_20 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automated machine learning multi-label classification multi-objective optimization publish with us policies and ethics profiles rafael gomes mantovani view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature flexcon-ce: a semi-supervised method with an ensemble-based adaptive confidence | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper flexcon-ce: a semi-supervised method with an ensemble-based adaptive confidence conference paper first online: 12 october 2023 pp 95–109 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) flexcon-ce: a semi-supervised method with an ensemble-based adaptive confidence download book pdf download book epub arthur medeiros  orcid: orcid.org/0009-0003-3716-30229, arthur c. gorgônio  orcid: orcid.org/0000-0002-1824-960010, karliane medeiros ovidio vale  orcid: orcid.org/0000-0001-9845-81569, flavius l. gorgônio  orcid: orcid.org/0000-0002-4926-39909 & … anne magály de paula canuto  orcid: orcid.org/0000-0002-3684-38149  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 519 accesses 2 citations abstract semi-supervised learning is characterized by a low number of labeled instances and a high number of unlabeled instances. flexcon-c (flexible confidence classifier) is a well-known semi-supervised method that uses the self-training learning algorithm as basis to generate prediction models. the main difference between self-training and flexcon-c is that the former uses a fixed threshold to select the unlabeled instances, while the latter has a dynamically adjusted confidence. flexcon-c applies a confidence adjustment equation based on the classifier performance. in this sense, the classifier performance is used to select and to label unlabeled instances. in machine learning, it is well-known that the classifier performance can be further improved through the use of classifier ensembles. therefore, this study proposes the use classifier ensembles in the flexcon-c confidence adjustment equation, aiming to provide a more efficient measure to select and to label unlabeled instances. in order to assess the viability of the proposed method (flexcon-ce), an empirical analysis will be conducted, using 20 datasets, three different classification algorithms and five different configurations of initially unlabeled data. the results indicate that the proposed method outperformed the traditional method, therewith proving itself promising for the task of automatic data selection and labeling in the semi-supervised context. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ensemble enhanced active learning mixture discriminant analysis model and its application for semi-supervised fault classification article 26 july 2022 semi-supervised learning chapter © 2025 learning sample-aware threshold for semi-supervised learning article 18 january 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization classical conditioning learning algorithms machine learning sensory evaluation statistical learning 1 introduction known as a field of study that has gained significant importance in recent years, machine learning (ml) has emerged as a way of deducing a hypothesis based on past or lived experience [1]. in this context, machines are programmed to learn from previously stored data derived from previous experiences in order to generate knowledge. in ml, the learning algorithms can be broadly divided into supervised, semi-supervised and unsupervised algorithms. in this paper, semi-supervised learning (ssl) algorithms are assessed and these algorithms use labeled instances to build their initial hypothesis and combine the information obtained by these examples to label the unlabeled instances. in other words, it is possible to use partially supervised information to guide the learning process and increase the evidence of the target labels [2]. nevertheless, we have observed that the automatic assignment of labels in an ssl algorithm is a difficult task, mainly related to the correct selection of the unlabeled instances to be labeled. an inefficient selection of the unlabeled instances and the consequent labeling of these instances can degrade enormously the performance of the ssl algorithms. a solution for this problem is proposed in [3] using a semi-supervised learning model called flexible confidence classifier (flexcon-c), which automatically adjusts the confidence parameter to select and label the unlabeled instance. this confidence is based solely on the accuracy of a classifier built in the previous iterations. in the ml context, it is well-known that the accuracy of a single classifier can not be efficient for complex tasks. for this reason, combining different classifiers to improve the classifier performance has been emerged as an efficient alternative, as observed in several studies in the field [4,5,6,7,8,9]. in these studies, it has been noticed that combining information from various sources has resulted in systems with more reliability and performance than individual classifiers. systems combining classifiers are known as classifier ensembles and they are composed of a set of base classifiers organized in a parallel way. these classifiers receive the input data and they produce several outputs that are used, in turn, as inputs by a combination method, which combines these outputs and generates the final ensemble output [7]. in this paper, we seek to improve the flexcon-c [3] performance by introducing the classifier ensembles strategy to select and to label unlabeled instances and to include them in the labeled data set. therefore, this paper proposes the flexcon-ce (flexible confidence with classifier ensembles) model, which uses an ensemble-based confidence adaptive confidence equation. in fact, two different versions of the flexcon-ce method are proposed in this paper. the first one, called flexcon-ce(hom), uses a homogeneous ensemble (base classifiers of the same type) while the second one, named flexcon-ce(het), uses an heterogeneous ensemble. this study assesses the efficiency of the proposed methods when building representative classifier ensembles by analyzing the classification effectiveness of these methods in 20 well-known datasets, with three different classification algorithms and five different configurations of initially unlabeled data. this article is organized as follows: sect. 2 presents the theoretical aspects of semi-supervised learning algorithms as well as the flexcon-c method and the classifier ensembles. section 3 highlights some related work to this research while sect. 4 presents and details the ssl method proposed in this article. section 5 describes some methodological aspects of the empirical analysis while sect. 6 presents the obtained results. finally, sect. 7 highlights some final considerations and future possibilities of researches. 2 theoretical aspects 2.1 semi-supervised learning according to the tasks to which they are applied, the machine learning algorithms can be broadly divided into three types: 1) supervised learning predictive tasks; 2) unsupervised learning descriptive tasks; 3) semi-supervised learning combining predictive and descriptive tasks. in other words, supervised learning requires the training dataset to have input and output attributes (labels or classes) in order to find a model, based on the data used for training, that can be used to predict a label or class of a given instance [1]. in unsupervised learning, dataset attributes are not required to have an output or label to shape the instances’ behavior pattern. this is possible due to the existence of tasks that look for behavior patterns of the instances based on the input attributes. unsupervised or descriptive tasks are more related to exploring the dataset, with the aim of obtaining cluster patterns or association rules present in the data [2]. the existence of datasets that have a part of the instances with labels and another part without assigned labels became a motivation for the creation of a new classification model with the objective of improving the performance of learning tasks benefiting from the strategy of using labeled and unlabeled data, known as semi-supervised learning [10]. following this concept, different algorithms were developed that propose the classification of unlabeled instances based on the ones with a label, including self-training, which will be used in this work. the self-training method is characterized by using its own prediction results, obtained by a single classifier, and applying a new training to the model until all instances that did not have labels are properly labeled [11]. the algorithm 1 depicts the flow of instructions that are executed in the self-training process. first, the database is divided into two sets: \(d_{l}\) labeled instances and \(d_{u}\) unlabeled instances (line 1). then, a classifier f is generated through \(d_{l}\) (line 3). after assigning the new labels to the unlabeled data, these instances are inserted into the subset s which will be removed from \(d_{u}\) and added into \(d_{l}\) (lines 4–6). then the process will restart until there is no more unlabeled data. note that, based on the results generated by the classifier, the model is retrained in order to learn according to its assertiveness rate. algorithm 1 self-training full size image in [12], the authors addressed an exploratory analysis of the self-training algorithm, regarding its adaptation by inserting a new parameter, called confidence, which is used to select and to label unlabeled instances. the main aim is to select instances that have a prediction higher than a confidence threshold. however, the confidence threshold is initially defined and this value is used throughout the whole labeling process. in [3], the authors proposed the flexcon-c method, which is an extension of the work proposed in [12], which applies an adaptive (or flexible) confidence threshold, which is calculated in every iteration of the semi-supervised process. the flexcon-c algorithm is presented by algorithm 2. algorithm 2 self-training with confidence adjustment (flexcon-c) full size image in algorithm 2, the lines marked in blue highlight the main differences between this algorithm and the original self-training algorithm. initially, \(d_{l}\) represents the initially labeled set, while \(d_{u}\) stands for the set of unlabeled instances. at each iteration, a classifier f is trained using the set of labeled instances \(d_{l}\) as training set and classifies the \(d_{u}\) instances (lines 3 to 4). then, a new confidence threshold value is calculated (line 5). then, select all unlabeled instances whose prediction is higher than or equal to the confidence threshold. these instances are then labeled (line 6). these instances are removed from the \(d_{u}\) set and included to the \(d_{l}\) set. this process is then repeated until the \(d_{u}\) set is empty. in [3], three ssl algorithms are proposed (flexcon-g, flexcon, and flexcon-c). all three algorithms use adaptive confidence threshold and they are all based on the selftraining method. in the cited article, the flexcon-c algorithm adjusts the confidence parameter at each iteration based solely on the precision of a classifier built the previous iteration. in this method, the adjustment will be done by increasing or decreasing the confidence value based on a change rate cr. equation 1 presents the confidence update equation for flexcon-c. $$\begin{aligned} \textit{conf}(t_{i+1})={\left\{ \begin{array}{ll}{} \textit{conf}(t_{i}) \textit{ cr}, \textit{if } \textit{acc} \ge \textit{mp} \textit{ + e}\\ \textit{conf}(t_{i}), \textit{if } \textit{mp} \textit{e} < \textit{acc} < \textit{mp} \textit{ + e}\\ \textit{conf}(t_{i}) \textit{ + cr}, \textit{if } \textit{acc} \le \textit{mp} \textit{e} \end{array}\right. } \end{aligned}$$ (1) where \(conf(t_{i+1})\) is the confidence value of the current iteration, mp is the minimum acceptable precision; cr is the change rate it is a user-given hyperparameter, acc is the accuracy of the classifier. finally, e is an acceptable precision variation that is allowed in order to define an stabilization in precision. in this equation, if the accuracy is higher than a minimum acceptable precision (considering the acceptable variation e), mp, the confidence threshold is decreased by a change rate cr. if the accuracy is lower than a minimum acceptable precision, the confidence threshold is increased by a change rate cr. otherwise, the confidence threshold does not change. this method was further divide into two sub-methods with respect to the label definition of the unlabeled instances, flexcon-c1 and flexcon-c2. flexcon-c1 uses classifier ensembles to define the label of each pattern, while flexcon-c2 uses the label predicted by the classifier generated in the first iteration, which is stored by the algorithm. in flexcon-c1, the classifier ensemble is composed of classifiers built in all iteration performed until a current iteration. in addition, these individual classifiers are combined by sum and majority voting, leading to two versions of this method, flexcon-c1(v) and flexcon-c1(s). 2.2 classifier ensembles as previously mentioned, the increased complexity and wide applicability of classification systems has led to exploring many approaches and methodologies. nevertheless, there is a perception that no classifier is considered completely satisfactory for a particular task; therefore, the idea of combining different methods to improve performance has emerged as a very promising possibility [13]. this combination is called classifier ensembles, also known as multi-classifier systems. in classification tasks, an ensemble includes several sub-models called base classifiers, which are usually obtained by training a basic learning algorithm (decision tree, neural network, k nearest neighbors, among others). the ensembles can be built based on the same learning algorithm, producing homogeneous ensembles, or using different algorithms and generating heterogeneous ensembles [14]. the proposition of classifier ensembles is to create and combine several inductive models for the same domain, obtaining better prediction quality [9]. after generating the base classifier set, the next step is the choice for the methods to combine their outputs. there is a vast number of methods for combining classifiers in the literature. in this work, we will be used voting. the choice for this method was made due to the use of information from all classifiers. the voting method, often used to combine classifiers, performs the combination by voting on the results of each classifier when a new instance is presented. 3 related work over the years, several works have been developed using a confidence parameter in semi-supervised ml algorithms [3, 12, 15,16,17,18]. the goal is to investigate the behavior of these algorithms, including this parameter as a threshold for selecting new instances for the labeled dataset. on the other hand, researchers have explored the use of classifier ensembles to enhance the performance of various machine learning algorithms [4,5,6, 8]. the present work aims to merge these two approaches, both using the confidence parameter as well as classifier ensembles, as a way to improve the performance of the semi-supervised learning algorithm flexcon-c. in [12, 15] new versions of the self-training algorithm were proposed that included a fixed confidence parameter with the objective of minimizing inconsistencies in the classification choices during the labeling process. in [3, 16,17,18,19] algorithms for the automatic adjustment of the confidence threshold were proposed as an extension of those researches. therefore, the focus of research development was on building solutions that could adjust the confidence parameter during the data labeling process. in [8] a study was conducted on the existing risks in railway safety inspections. these inspections are reported using text, which generates a large amount of textual data. as motivation for the authors, a predictive model was proposed that used classifier ensembles to predict risk situations that could be addressed proactively. the chosen classification model was the decision tree, which was applied to an ensemble of bagging classifiers that combined the results of the classifiers by voting. similarly, [5] used different models of classifiers in an ensemble to predict a credit score for customers of financial organizations and banks. in [4] a new ensemble learning algorithm named e\(\_\)rhfs was developed and applied as a tool to predict software failure. the authors used the voting combination method on the base classifiers to obtain the final ensemble classifier for software defect prediction. in [6] the proposal focused on two primary factors in creating a classifier ensemble, namely: accuracy and diversity. the paper aimed to propose a new diversity measure called misclassification diversity (md) and an incremental layered classifier selection approach to build the ensemble. 4 the proposed method: flexcon-ce as already mentioned, the purpose of this paper is to propose a new flexcon-c variation, called flexcon-ce (flexible confidence with classifier ensemble). this variation uses an ensemble-based confidence threshold. therefore, in order to develop the proposed method, it was necessary to modify the original flexcon-c algorithm 2 with lines marked in blue highlight. algorithm 3 presents the flexcon-ce step-by-step operation. algorithm 3 flexible confidence with ensemble (flexcon-ce) full size image initially, in algorithm 3, the database is divided into two sets: a set of labeled instances \(d_{l}\) and a set of unlabeled instances \(d_{u}\). additionally, a pool of classifiers \(f_{n}\) is available. an ensemble is defined as empty (line 1). then, for each classifier f in \(f_n\), they are trained using \(d_{l}\) as training set. after that, the trained classifier f is added to the ensemble e (lines 2 to 7). finally, the ensemble e will be applied to \(d_{u}\) in order to calculate the effectiveness generated by e (line 8). then, the new confidence threshold is calculated (line 9), the unlabeled instances are selected, labeled, removed from the unlabeled set \(d_{u}\) and included in the labeled set \(d_{l}\) (lines 9 to 12). it is important to emphatize that the threshold is used to select unlabeled instances that will be labeled by the voting process of the ensemble. it is important to emphasize that the calculation of the new confidence threshold is made using eq. 1. the main difference is that the mp value is related to the ensemble accuracy (flexcon-ce) instead of the classifier accuracy (flexcon-c). as previously described, two flexcon-ce versions are proposed. in the first one, the classifier ensemble is constructed heterogeneously, using more than one type of classification method. in the second version, the classifier ensemble is built homogeneously, when using only one type of method. in this work, both heterogeneous and homogeneous ensemble approaches will be assessed. 5 experimental methodology in order to assess the feasibility of the proposed method, an empirical analysis will be carried out. the next subsections will describe the main aspects of the experimental methodology. 5.1 datasets in this empirical analysis, the flexcon-ce method is applied to 20 different databases. these datasets are available in platforms and repositories that maintain various experimental data, such as github [20], uci machine learning [21] and kaggle datasets [22]. table 1 describes the characteristics of all datasets, regarding the number of attributes, instances, number of classes, data types (categorical c, real r and integer i), and data distribution (balanced b and unbalanced u). table 1. datasets characteristicsfull size table 5.2 methods and materials all method and learning procedures of this paper are implemented using the scikit-learn package available in the python language. in order to create the algorithm proposed in this article, we decided to use the scikit-learn python library [23], since it offers several learning methods for classification, regression, and clustering domains. it also integrates with other widely used python libraries for machine learning experiments, such as matplotlib [24], numpy [25], pandas [26], and others. in order to facilitate the comparative analysis of the results of the flexcon-ce method with flexcon-c, three classifiers are used, which are: naive bayes (nb), decision tree (tree) – using a cart-based algorithm – and k-nearest neighbors (k-nn) – with k values from 4 to 8. additionally, the percentage values of the initially labeled instances, n, are 5\(\%\), 10\(\%\), 15\(\%\), 20\(\%\) and 25\(\%\), in the same way as it was performed by [16, 19, 27, 28]. each trained classifier was included to the classifier ensemble heterogeneously and homogeneously, for flexcon-ce (het) and flexcon-ce (hom), respectively. for the combination of all n classifiers, the voting criterion was applied to define the overall ensemble output. the voting method consists of a non-linear combination of the classifiers’ outputs, and its process consists of determining the winning class, from a data input pattern, by the total number of votes counted for each classifier [4, 8, 29]. therefore, the confidence of the classifier ensemble is defined by the number of classifiers that selected the winning class divided by the number of classifiers of the used ensemble. the main aim of this empirical analysis is to evaluate the performance of the flexcon-ce algorithm and to compare them to the flexcon-c (previously explained: flexcon-c1(v), flexcon-c1(s) and flexcon-c2) results. in order to assess the obtained results, the friedman statistical test was achieved. according to [30], this test compares related sample data, which means that the same object can be analyzed more than once. as it is non-parametric test, numerical values are not used directly, but rather their positions. after the ranking are performed for the groups separately, the hypothesis of similarity of the sum of the position of each group is verified. in this way, it is possible to compare the performance of different data classification algorithms in the scenarios created for each different dataset. 6 experimental results this section presents the analysis of the obtained results. as previously mentioned, the analysis of the results will be conducted in a comparative way, evaluating the performance of both flexcon-ce versions and three flexcon-c versions, flexcon-c1(v), flexcon-c1(s), and flexcon-c2 [3]. the next sections will present the analysis of the experimental results. the first one presents an analysis of the accuracy results presented by all analyzed methods.while the next section conducts a statistical analysis of the obtained results. 6.1 accuracy analysis tables 2 and 3 present the results obtained by each method used in this analysis. the difference between these two tables is that the first table shows the results of three different classifiers and compares the performance of flexcon-ce using a homogeneous classifier ensemble (flexcon-ce(hom)) with all three flexcon-c versions (flexcon-c1(v), flexcon-c1(s), and flexcon-c2). for the second table, the best results of table 2 are selected, for each classification algorithm (naive bayes, decision tree and k-nn), as well as the flexcon-ce result using a heterogeneous classifier ensemble (flexcon-ce(het)). table 2. results of average accuraryfull size table table 3. results of best average accuracyfull size table both tables have the same configuration: the first column presents the name of the used classifier while the second column indicates the semi-supervised learning method. columns 3–7 display the average accuracy and standard deviation obtained by each semi-supervised method over all 20 used datasets, according to the settings of 5\(\%\), 10\(\%\), 15\(\%\), 20\(\%\), and 25\(\%\) of initially labeled instances. additionally, in both tables, the results that showed the most significant accuracy values are highlighted in bold. when analyzing results obtained with the naive bayes classifier in table 2, it is possible to observe that the flexcon-ce method using the homogeneous ensemble flexcon-ce(hom) obtained the best results in 80% of the analysed cases (4 out of the 5% of initially labeled instances). in this same table, considering the results obtained by the tree classifier, flexcon-ce(hom) also achieved the highest accuracy in 100% of cases (5 out of the 5% of initially labeled data). based on these results, it is possible to state that the flexcon-ce(hom) method obtained significantly better performance results in two out of three analyzed classifiers. still exploring table 2, in general, it is possible to observe that the flexcon-ce(hom) method performed better than all flexcon-c versions in 60% of the analyzed cases (9 out of the 15 best results). this is a promising result showing that the use of a more efficient selection and labeling technique leads to more robust ssl methods. as previously explained, table 3 presents the performance of the proposed method using the heterogeneous ensemble flexcon-ce(het), and the best results obtained in table 2, for comparison purposes. when observing the aforementioned table, it is possible to identify that the flexcon-ce(hom) method obtained the best performance in 80% of the cases (4 out of 5% of initially labeled instances) when the number of initially labeled instances was higher than or equal to 10%. on the other hand, the other proposed method of this paper, flexcon-ce(het), obtained the best performance when the number of initially labeled instances was limited to 5%. in summary, it is possible to state that one proposed version, flexcon-ce(hom) or flexcon-ce(het), achieved better accuracy results than all flexcon-c versions, regardless of the type of classifier used. the next subsection presents the statistical analysis of the obtained results in order to demonstrate that the proposed semi-supervised machine learning methods are promising. 6.2 statistical analysis the statistical analysis performed in this article uses critical difference (cd) diagrams. figures 1 and 2 display the results obtained using the aforementioned diagrams. the post-hoc friedman test is used to analyze the results based on their rankings. additionally, in these diagrams, the methods located further to the left exhibit better results, while the ones on the right presented the poorest performances. finally, a method is considered statistically different from another when both are not covered by the critical difference line (bold horizontal line in the diagram). otherwise, when this line covers two or more approaches, it means that the null hypothesis of the friedman test cannot be rejected. figure 1 presents the critical difference diagram with the results of the statistical test of the results obtained in table 2. the applied classification algorithm is indicated in front of the method name. when analyzing the aforementioned diagram and observing the ranking of methods delivered by each classifier, it can be seen that flexcon-ce(hom)-tree was the method that reached the highest ranking, which confirms the analysis carried out in the previous section regarding the accuracy of these methods. additionally, according to the statistical analysis, this method is similar to the following four methods, in which two of them applied a decision tree (flexcon-c2-tree and flexcon-c1(v)) and the remaining two methods used a k-nn method (flexcon-c2 and flexcon-c1(v)). on the other hand, the winning method is statistically superior to the remaining seven methods. fig. 1. critical difference diagram using all methods full size image figure 2 illustrates the critical difference diagram with the results of the statistical test of the results obtained in table 3. when analyzing the rankings of each method, it can be seen that the flexcon-ce(hom)-tree and flexcon-ce(het) methods are statistically similar and both achieved the best results. this observation confirms what was reported in the previous section, in which the proposed methods outperformed the methods that do not use classifier ensembles. fig. 2. statistics performance of the best full size image in summary, the results obtained in this paper showed significant improvements of the proposed method, in comparison to the flexcon-c versions. these results corroborate with the results obtained in the previous section, demonstrating an enhancement in the performance of the semi-supervised methods of the proposed methods. moreover, the proposed methods are validated from two different perspectives, both in terms of performance and statistical analysis. 7 conclusion and future works this paper proposed a new ssl method, called flexcon-ce, which was designed based on introducing classifier ensembles in the flexcon-c functioning. the main difference between flexcon-c and flexcon-ce is that the latter uses classifier ensembles to define the adaptive confidence while the predictions of a classifier is used by flexcon-c. additionally, this work explores the use of two versions of the proposed method: one that uses an homogeneous ensemble structure and the other one used an heterogeneous ensemble structure. in order to validate the proposed method, an empirical analysis was conducted, in which a comparative analysis of the performance of flexcon-ce with flexcon-c was carried out. in general, the performance of the analyzed methods was assessed in terms of accuracy and standard deviation and they were validated statistically using the friedman test. after analyzing the results, it was concluded that the flexcon-ce method obtained better results in most cases, with emphasis on the version that used the homogeneous ensemble with the tree classifier, followed by the heterogeneous flexcon-ce version. for future work, we suggest to explore other classification algorithms and use new databases to address experimental scenarios. furthermore, it is recommended to make the size of the classifier ensembles more flexible in relation to the number of base classifiers. references faceli, k., lorena, a.c., gama, j., de leon ferreira de carvalho, a.c.p.: an approach of machine learning. artif. intell. (2011) google scholar  chapelle, o., scholkopf, b., zien, a.: semi-supervised learning, vol. 2. the mit press, cambridge, ma (2006) google scholar  ovidio vale, k.m., et al.: automatic adjustment of confidence values in self-training semi-supervised method. in: 2018 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2018) google scholar  wei, w., jiang, f., yu, x., du, j.: an ensemble learning algorithm based on resampling and hybrid feature selection, with an application to software defect prediction. in: 2022 7th international conference on information and network technologies (icint), pp. 52–56. ieee (2022) google scholar  safiya parvin, a., saleena, b.: an ensemble classifier model to predict credit scoring-comparative analysis. in: 2020 ieee international symposium on smart electronic systems (ises) (formerly inis), pp. 27–30. ieee (2020) google scholar  zohaib jan, m., verma, b.: a novel diversity measure and classifier selection approach for generating ensemble classifiers. ieee access 7, 156360–156373 (2019) google scholar  lochter, j.v., zanetti, r.f., reller, d., almeida, t.a.: short text opinion detection using ensemble of classifiers and semantic indexing. exp. syst. appl. 62:243–249 (2016) google scholar  li, x., shi, t., li, p., zhou, w.: application of bagging ensemble classifier based on genetic algorithm in the text classification of railway fault hazards. in: 2019 2nd international conference on artificial intelligence and big data (icaibd), pp. 286–290. ieee (2019) google scholar  cichosz, p.: data mining algorithms: explained using r. john wiley & sons (2014) google scholar  albalate, a., minker, w.: semi-supervised and unsupervised machine learning: novel strategies. john wiley & sons (2013) google scholar  zhu, x., goldberg, a.b.: introduction to semi-supervised learning. synth. lect. artif. intell. mach. learn. 3(1), 1–130 (2009) math  google scholar  rodrigues, f.m., de m. santos, a., canuto, a.m.p.: using confidence values in multi-label classification problems with semi-supervised learning. in: the 2013 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2013) google scholar  nascimento, d.s.c., coelho, a.l.v., canuto, a.m.p.: integrating complementary techniques for promoting diversity in classifier ensembles: a systematic study. neurocomputing 138, 347–357 (2014) article  google scholar  gharroudi, o.: ensemble multi-label learning in supervised and semi-supervised settings. ph.d. thesis, université de lyon (2017) google scholar  rodrigues, f.m., câmara, c.j., canuto, a.m.p., santos, a.m.: confidence factor and feature selection for semi-supervised multi-label classification methods. in: 2014 international joint conference on neural networks (ijcnn), pp. 864–871. ieee (2014) google scholar  gorgônio, a.c., et al.: análise da variação do limiar para o algoritmo de aprendizado semissupervisionado flexcon-c/threshold variation analysis for flexcon-c semisupervised learning algorithm. brazil. j. develop. 5(11), 26654–26669 (2019) google scholar  vale, k.m.o., gorgônio, a.c., da luz, e.g.f., de paula canuto, a.m.: an efficient approach to select instances in self-training and co-training semi-supervised methods. ieee access 10, 7254–7276 (2021) google scholar  gorgônio, a.c., alves, c.t., lucena, a.j.f., gorgônio, f.l., vale, k.m.o., canuto, a.m.p.: analysis of the threshold variation of the flexcon-c algorithm for semi-supervised learning. in: anais do xv encontro nacional de inteligência artificial e computacional, pp. 775–786. sbc (2018) google scholar  vale, k.m.o., et al.: a data stratification process for instances selection in semi-supervised learning. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2019) google scholar  breiman, l.: bias, variance, and arcing classifiers. technical report, tech. rep. 460, statistics department, university of california, berkeley (1996) google scholar  dheeru, d., karra taniskidou, e.: uci machine learning repository (2017) google scholar  smith, j.w., everhart, j.e., dickson, w.c., knowler, w.c., johannes, r.s.: using the adap learning algorithm to forecast the onset of diabetes mellitus. in: proceedings of the annual symposium on computer application in medical care, p. 261. american medical informatics association (1988) google scholar  pölsterl, s.: scikit-survival: a library for time-to-event analysis built on top of scikit-learn. j. mach. learn. res. 21(1), 8747–8752 (2020) google scholar  bisong, e.: building machine learning and deep learning models on google cloud platform. apress, berkeley (2019). https://doi.org/10.1007/978-1-4842-4470-8 harris, c.r., et al.: array programming with numpy. nature 585(7825), 357–362 (2020) google scholar  nelli, f.: python data analytics with pandas, numpy, and matplotlib (2018) google scholar  araújo, y.n., et al.: a data stratification process for instances selection applied to co-training semi-supervised learning algorithm. in: 2021 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2021) google scholar  vale, k.m.o., gorgônio, f.l., araújo, y.n., gorgônio, a.c., de p canuto, a.m.: a co-training-based algorithm using confidence values to select instances. in: 2020 international joint conference on neural networks (ijcnn), pp. 1–7. ieee (2020) google scholar  kuncheva, l.i.: combining pattern classifiers: methods and algorithms. john wiley & sons (2014) google scholar  theodorsson-norheim, e.: friedman and quade tests: basic computer program to perform nonparametric two-way analysis of variance and multiple comparisons on ranks of several related samples. comput. biol. med. 17(2), 85–99 (1987) article  google scholar  download references author information authors and affiliations department of computing and technology (dct), federal university of rio grande do norte (ufrn), caicó, rn, brazil arthur medeiros, karliane medeiros ovidio vale, flavius l. gorgônio & anne magály de paula canuto department of informatics and applied mathematics (dimap), federal university of rio grande do norte (ufrn), natal, rn, brazil arthur c. gorgônio authors arthur medeirosview author publications search author on:pubmed google scholar arthur c. gorgônioview author publications search author on:pubmed google scholar karliane medeiros ovidio valeview author publications search author on:pubmed google scholar flavius l. gorgônioview author publications search author on:pubmed google scholar anne magály de paula canutoview author publications search author on:pubmed google scholar corresponding author correspondence to arthur medeiros . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper medeiros, a., gorgônio, a.c., vale, k.m.o., gorgônio, f.l., canuto, a.m.d.p. (2023). flexcon-ce: a semi-supervised method with an ensemble-based adaptive confidence. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_7 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords classifier ensemble self-training algorithm flexcon-c publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature on the refinement of compensation-based semantics for weighted argumentation frameworks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper on the refinement of compensation-based semantics for weighted argumentation frameworks conference paper first online: 28 november 2021 pp 344–358 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) on the refinement of compensation-based semantics for weighted argumentation frameworks download book pdf download book epub henrique viana10 & joão alcântara11  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 733 accesses abstract acceptability semantics for the frameworks of weighted argumentation can satisfy up to one of the principles of (quality precedence), (cardinality precedence) or (compensation), which are pairwise incompatible. in this paper we define two new principles: (quality compensation) and (cardinality compensation), which are weakened versions of (quality precedence) and (cardinality precedence), respectively. we show that these new principles are compatible with (compensation) and propose two new semantics: a t-conorm-based, which can satisfy (quality compensation) and a cumulative sum-based semantics, which satisfies (cardinality compensation). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others new weak admissibility semantics for abstract argumentation chapter © 2021 on restricting the impact of self-attacking arguments in gradual semantics chapter © 2021 a parametrized ranking-based semantics for persuasion chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. convention theory dosage compensation logical analysis formal logic logic standards 1 introduction argumentation is a reasoning process in which interacting arguments are built and evaluated. it is widely studied in artificial intelligence, namely for reasoning about making decisions [5, 8, 22] and modelling agents interactions [7, 30]. an argumentation-based formalism or argumentation framework is generally defined as a set of arguments, attacks amongst the arguments, and a semantics for evaluating the arguments. a semantics assesses to what extent an argument is acceptable. examples of semantics are those proposed by dung, which compute extensions of arguments [6, 11, 21, 26, 29, 32] and ranking semantics, which compute the overall strengths of each argument [2, 10, 12, 13, 18, 31]. with respect to the ranking semantics, in the works of amgoud and ben-naim [2], it was proposed a set of principles for it and consequently refined in [3]. moreover, new principles were introduce to describe strategies that a semantics may use when it faces a conflict between the quality of attackers and their quantity [4]. the strategies are: i) privileging quality through the principle of (quality precedence), ii) privileging cardinality through the principle of (cardinality precedence), or iii) simply allowing compensation between quality and cardinality through the principle of (compensation). it was pointed out some limitations in the literature about this subject. first, there was no semantics satisfying (cardinality precendence), which was unfortunate since (cardinality precendence) is a viable choice in (multiple criteria) decision making [17]. second, there was only one semantics satisfying (quality precedence). third, several other semantics satisfy (compensation), however, none of them satisfies all the principles that are compatible with the compensation principle. with that in mind, amgoud and ben-naim [4] provided three new semantics: a max-based, a cardinality-based and a sum-based one. a formal analysis and thorough comparison with other semantics were done to fill the previous gaps by introducing three novel semantics and show new semantics that enjoy more desirable properties than existing semantics. besides that, another important result pointed out is that some of the principles are incompatible. they cannot be satisfied all together by a semantics. this is particularly the case with the (quality precedence), (cardinality precedence) and (compensation). from this perspective, in this work, we propose to explore further the relation among these three principles and show weaker principles that are intermediary of these three principles. furthermore, we present two new semantics, one based on t-conorms operators [24], which present a mix of max-based semantics and the sum-based semantics; and the other semantics based on the notion of cumulative sum [25], which presents a trade off between the cardinality-based and the sum-based semantics. the novelty is both semantics satisfy not only all the basic principles, but also weakened versions of two incompatible principles. as far as we know, this is the first work to push forward the frontier of knowledge on the development of semantics aiming at satisfying weakened versions of incompatible principles. the paper is organized as follows: in sect. 2, we first recall some basic notions of argumentation theory, introduce the notations used throughout the paper and show some examples of semantics and the principles that a semantics could satisfy. in sect. 3, we then consider the contribution of the paper, with the introduction of two new semantics and two new principles. we provide a formal analysis and comparison of existing semantics and these new principles. finally, in sect. 4, we conclude the paper. 2 foundations of weighted argumentation frameworks 2.1 basic concepts a weighted argumentation graph is a set of arguments and an attack relation between them. each argument has a weight in the interval [0, 1] representing its basic strength (the smaller the weight, the weaker the argument). definition 1 (wag). a weighted argumentation graph (wag) is an ordered tuple \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), where \(\mathcal {a}\) is a non-empty finite set of arguments, w is a function from \(\mathcal {a}\) to [0, 1], and \(\mathcal {r} \subseteq \mathcal {a} \times \mathcal {a}\). intuitively, w(a) is the basic strength of argument a, and \((a, b) \in \mathcal {r}\) (or \(a\mathcal {r}b)\) means argument a attacks argument b. example 1 consider the wag \(\mathbf {g}\) below consisted of four arguments \(a, b, c, \dots , k\). for instance, the basic strength of a is \(w(a) = 1\) and the basic strength of d is \(w(d) = 0.6\). besides that, argument d attacks argument a, which is depicted by a directed edge in the graph. definition 2 (isomorphism). let \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(\mathbf {g'} = \langle \mathcal {a'},w', \mathcal {r'} \rangle \) be two wags. an isomorphism from \(\mathbf {g}\) to \(\mathbf {g'}\) is a bijective function f from \(\mathcal {a}\) to \(\mathcal {a'}\) such that: i) \(\forall a \in \mathcal {a}, w(a) = w'(f(a))\), ii) \(\forall a, b \in \mathcal {a}, a\mathcal {r}b\) iff \(f(a)\mathcal {r'}f(b)\). an acceptability semantics is a function assigning a value, called acceptability degree, to every argument in a weighted argumentation graph. this value represents the overall strength of an argument, and is issued from the aggregation of the basic strength of the argument and the overall strengths of its attackers. the greater this value, the more acceptable the argument. unlike extension semantics where arguments are either accepted or rejected, it is consider graded semantics, which may assign various acceptability degrees to arguments. throughout the paper, we consider the scale [0, 1]. definition 3 (semantics). a semantics is a function \(\mathbf {s}\) transforming any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) into a vector \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}\) in \([0, 1]^n\), where \(n = |\mathcal {a}|\). for \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a)\) is called acceptability degree of a. we present next the list of all notations used in the paper. let \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) be a wag and \(a \in \mathcal {a}\). \(\text {att}_\mathbf {g}(a)\) denotes the set of all attackers of a in \(\mathbf {g}\), i.e. \(\text {att}_\mathbf {g}(a) = \{ b \in \mathcal {a} : b\mathcal {r}a \}\). for \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(\mathbf {g'} = \langle \mathcal {a'},w', \mathcal {r'} \rangle \) such that \(\mathcal {a} \cap \mathcal {a'} = \emptyset \), \(\mathbf {g} \oplus \mathbf {g'}\) is the wag \(\langle \mathcal {a} \cup \mathcal {a'},w'', \mathcal {r} \cup \mathcal {r'} \rangle \) where for any \(x \in \mathcal {a}\) (resp. \(x \in \mathcal {a'}\)), \(w''(x) = w(x)\) (resp. \(w''(x) = w'(x)\)). 2.2 examples of weighted semantics the first semantics satisfies quality precedence, thus it favors the quality of attackers over their cardinality. it is based on a scoring function which follows a multiple steps process. at each step, the function assigns a score to each argument. in the initial step, the score of an argument is its basic strength. then, in each step, the score is recomputed on the basis of the basic strength as well as the score of the strongest attacker of the argument at the previous step. definition 4 (weighted max-based semantics [4]). for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(a) = \frac{w(a)}{1 + \max _{b \in \text {att}_\mathbf {g}(a) } \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(b) }\). by convention, if \(\text {att}_\mathbf {g}(a) = \emptyset \), \(\max _{b \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(b) = 0\). example 2 considering the wag \(\mathbf {g}\) from example 1, we have that \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(a) = \frac{w(a)}{1 + \max _{b \in \text {att}_\mathbf {g}(a) } \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(b) } = \frac{w(a)}{1 + \max \{ \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(d), \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(e) \} }\). since the argument d is attacked by the argument i, we need to compute \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(d) = \frac{w(d)}{1 + \max \{ \text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(i) \} } = \frac{0.6}{1 + 0.5 } = 0.4\) (since there is no argument attacking i, \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(i) = w(i) = 0.5\)). we also have that \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(e) = w(e) = 0.3\). therefore, \(\text {deg}^{\mathbf {mbs}}_{\mathbf {g}}(a) = \frac{1}{1 + \max \{ 0.4, 0.3 \} } = 0.71\). for a matter of simplicity, in this paper we are considering examples with acyclic graphs, but the weighted semantics are also defined to deal with cyclic graphs. the details and proofs about this issue can be found in [4]. the second semantics, called weighted card-based, favors the number of attackers over their quality. it considers only arguments that have a basic strength greater than 0, called founded. this restriction is due to the fact that unfounded arguments are lifeless and their attacks are ineffective. definition 5 (weighted card-based semantics [4]). let \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) be a wag and \(a \in a\). the argument a is founded iff \(w(a) > 0\). it is unfounded otherwise. let \(\text {attf}_\mathbf {g}(a)\) denote the set of founded attackers of a. for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(a) = \frac{w(a)}{1 + |\text {attf}_\mathbf {g}(a)| + \frac{\sum _{b \in \text {attf}_\mathbf {g}(a)} \text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(b) }{|\text {attf}_\mathbf {g}(a)|} }\). by convention, if \(\text {att}_\mathbf {g}(a) = \emptyset \), \(\sum _{b \in \text {attf}_\mathbf {g}(a)} \text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(b) = 0\). example 3 from the wag \(\mathbf {g}\) in example 1, we have that \(\text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(d) = 0.24\), \(\text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(e) = 0.3\) and \(\text {deg}^{\mathbf {cbs}}_{\mathbf {g}}(a) = 0.3\). the third semantics extends h-categorizer, initially proposed by besnard and hunter [12] for non-weighted and acyclic graphs. then, it was extended to account for varying degrees of basic strengths, and any graph structure. definition 6 (weighted h-categorizer semantics [4]). for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(a) = \frac{w(a)}{1 + \sum _{b \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(b) }\). by convention, if \(\text {att}_\mathbf {g}(a) = \emptyset \), \(\sum _{b \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(b) = 0\). example 4 from the wag \(\mathbf {g}\) in example 1, we have that \(\text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(d) = 0.4\), \(\text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(e) = 0.3\) and \(\text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(a) = 0.58\). 2.3 principles and properties in the sequel, we present some principles, which are important for i) a better understanding of semantics, ii) the definition of reasonable semantics, iii) comparing semantics, iv) choosing suitable semantics for applications [4]. it was proposed 15 principles, which described the role and impact of attacks and basic strengths in the evaluation of arguments, and how these two elements are aggregated. the first principle, called (anonymity), can be found in almost all axiomatic studies. in the argumentation literature, (anonymity) is called abstraction in [2] and language independence in [9]. principle 1 (anonymity). a semantics \(\mathbf {s}\) satisfies anonymity iff for any two wags \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(\mathbf {g'} = \langle \mathcal {a'},w', \mathcal {r'} \rangle \), for any isomorphism f from \(\mathbf {g}\) to \(\mathbf {g'}\), the following property holds: \(\forall a \in \mathcal {a}, \text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = \text {deg}^{\mathbf {s}}_{\mathbf {g'}}(f(a))\). the second principle, called (independence), states that the acceptability degree of an argument should be independent of any argument that is not connected to it. principle 2 (independence). a semantics \(\mathbf {s}\) satisfies independence iff for any two wags \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(\mathbf {g'} = \langle \mathcal {a'},w', \mathcal {r'} \rangle \) such that \(\mathcal {a} \cap \mathcal {a'} = \emptyset \), the following holds: \(\forall a \in \mathcal {a}, \text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = \text {deg}^{\mathbf {s}}_{\mathbf {g} \oplus \mathbf {g'}}(a)\). the next principle states that the acceptability degree of an argument a in a graph can depend on argument b only if there is a path from b to a, i.e., a finite non-empty sequence \(\langle x_1, \dots , x_n \rangle \) such that \(x_1 = b, x_n = a\) and \(\forall i < n, x_i\mathcal {r}x_{i+1}\). principle 3 (directionality). a semantics \(\mathbf {s}\) satisfies directionality iff for any two wags \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and \(\mathbf {g'} = \langle \mathcal {a'},w', \mathcal {r'} \rangle \) such that \(\mathcal {r'} = \mathcal {r} \cup \{ (a,b) \}\), it holds that: \(\forall x \in \mathcal {a}\), if there is no path from b to x, then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \text {deg}^{\mathbf {s}}_{\mathbf {g'}}(x)\). the next principle, called (neutrality), states that an argument, whose acceptability degree is 0, has no impact on the arguments it attacks. principle 4 (neutrality). a semantics \(\mathbf {s}\) satisfies neutrality iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a,b \in \mathcal {a}\), if i) \(w(a) = w(b)\), and ii) \(\text {att}_\mathbf {g}(b) = \text {att}_\mathbf {g}(a) \cup \{x\}\) with \(x \in a \backslash \text {att}_\mathbf {g}(a)\) and \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = 0\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). the condition \(w(a) = w(b)\) ensures that the attacks from \(\text {att}_\mathbf {g}(a)\) have the same effect on both arguments a and b. (equivalence) principle ensures that the overall strength of an argument depends only on the basic strength of the argument and the overall strengths of its (direct) attackers. principle 5 (equivalence). a semantics \(\mathbf {s}\) satisfies equivalence iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a,b \in \mathcal {a}\), if i) \(w(a) = w(b)\), and ii) there exists a bijective function f from \(\text {att}_\mathbf {g}(a)\) to \(\text {att}_\mathbf {g}(b)\) such that \(\forall x \in \text {att}_\mathbf {g}(a), \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \text {deg}^{\mathbf {s}}_{\mathbf {g}}(f(x)),\) then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). (maximality) principle states that an unattacked argument receives an acceptability degree equal to its basic strength. principle 6 (maximality). a semantics \(\mathbf {s}\) satisfies maximality iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a \in \mathcal {a}\), if \(\text {att}_\mathbf {g}(a) = \emptyset \), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = w(a)\). the role of attacks is (weakening) their targets. indeed, when an argument receives an attack, its overall strength decreases whenever the attacker is “alive”. principle 7 (weakening). a semantics \(\mathbf {s}\) satisfies weakening iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a \in \mathcal {a}\), if i) \(w(a) > 0\), and ii) \(\exists b \in \text {att}_\mathbf {g}(a)\) such that \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(b) > 0\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) < w(a)\). (weakening) leads to strength loss as soon as an argument is attacked by at least one alive attacker. (counting) principle states that each alive attacker has an impact on the overall strength of the argument. thus, the more numerous the alive attackers of an argument, the weaker the argument. principle 8 (counting). a semantics \(\mathbf {s}\) satisfies counting iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a,b \in \mathcal {a}\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), and iii) \(\text {att}_\mathbf {g}(b) = \text {att}_\mathbf {g}(a) \cup \{y\}\) with \(y \in \mathcal {a} \backslash \text {att}_\mathbf {g}(a)\) and \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > 0\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). (weakening soundness) principle goes further than weakening by stating that attacks are the only source of strength loss. principle 9 (weakening soundness). a semantics \(\mathbf {s}\) satisfies weakening soundness iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a \in a\) such that \(w(a) > 0\), if \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) < w(a)\), then \(\exists b \in \text {att}_\mathbf {g}(a)\) such that \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(b) > 0\). (reinforcement) principle states that the stronger the source of an attack, the greater its intensity. principle 10 (reinforcement). a semantics \(\mathbf {s}\) satisfies reinforcement iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\) or \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(b) > 0\), iii) \(\text {att}_\mathbf {g}(a) \backslash \text {att}_\mathbf {g}(b) = \{x\}\), iv) \(\text {att}_\mathbf {g}(b) \backslash \text {att}_\mathbf {g}(a) = \{y\}\), and v) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x)\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). (resilience) principle states that an attack cannot completely kill an argument, i.e., to turn its acceptability degrees equal to 0. principle 11 (resilience). a semantics \(\mathbf {s}\) satisfies resilience iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a \in a\), if \(w(a) > 0\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\). (proportionality) states that the stronger the target of an attack, the weaker its intensity. principle 12 (proportionality). a semantics \(\mathbf {s}\) satisfies proportionality iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\) such that i) \(\text {att}_\mathbf {g}(a) = \text {att}_\mathbf {g}(b)\), ii) \(w(a) > w(b)\), and iii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\) or \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(b) > 0\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). the three last principles concern possible choices offered to a semantics when it faces a conflict between the quality and the number of attackers. (quality precedence) principle gives more importance to the quality. principle 13 (quality precedence). a semantics \(\mathbf {s}\) satisfies quality precedence iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), and iii) \(\exists y \in \text {att}_\mathbf {g}(b)\) such that \(\forall x \in \text {att}_\mathbf {g}(a), \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x)\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). (cardinality precedence) principle states that a great number of attackers has more effect on an argument than just few. principle 14 (cardinality precedence). a semantics \(\mathbf {s}\) satisfies cardinality precedence iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), and iii) \(|\{y \in \text {att}_\mathbf {g}(b) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y)> 0\}|> |\{x \in \text {att}_\mathbf {g}(a) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) > 0\}|\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). finally, (compensation) states that several weak attacks may compensate the quality of attacks overall. principle 15 (compensation). a semantics \(\mathbf {s}\) satisfies compensation iff there exists a wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), such that for two arguments \(a, b \in a\), i) \(w(a) = w(b)\), ii) \(\text{ deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), iii) \(|\{x \in \text {att}_\mathbf {g}(a) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x)> 0\}|> |\{y \in \text {att}_\mathbf {g}(b) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > 0\}| \), iv) \(\exists y \in \text {att}_\mathbf {g}(b)\) such that \(\forall x \in \text {att}_\mathbf {g}(a), \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x)\) and \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) = \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). the results corresponding to the compatibility of the principles is stated below. proposition 1 [4] the three following properties hold. i) (quality precedence), (cardinality precedence) and (compensation) are pairwise incompatible; ii) (independence), (directionality), (equivalence), (resilience), (reinforcement), (maximality) and (quality precedence) are incompatible; iii) (cardinality precedence) (respectively (compensation)) is compatible with all principles 1–12. it was shown that weighted max-based semantics satisfies (quality precedence) as well as all the principles which are compatible with it [4]. it violates, however, (counting) since by definition, this semantics focuses only on the strongest attacker of an argument, and neglects the remaining attackers. theorem 1 [4] weighted max-based semantics violates (cardinality precedence), (compensation), (counting) and (reinforcement). it satisfies all the remaining principles. weighted card-based semantics satisfies (cardinality precedence) as well as all the principles that are compatible with it. theorem 2 [4] weighted card-based semantics satisfies all the principles except (quality precedence) and (compensation). weighted h-categorizer semantics satisfies (compensation) as well as all the principles that are compatible with it. theorem 3 [4] weighted h-categorizer semantics satisfies all the principles except (quality precedence) and (cardinality precedence). 3 refinements of compensation-based semantics fuzzy set theory has been shown to be a useful tool to describe situations in which the data are imprecise or vague. fuzzy sets handle such situations by attributing a degree to which a certain object belongs to a set [15]. an important notion in fuzzy set theory is that of triangular norms and conorms: t-norms and t-conorms are used to define a generalized intersection and union of fuzzy sets [24]. triangular norms and conorms serve as aggregation operators, which can be used, e.g., for querying databases [20], to compute the resulting degree of confidence of agents [33], in approximate reasoning [16], information retrieval [14], neuro-symbolic learning [19], machine learning [1], etc. definition 7 (t-conorm [24]). a binary function \(\oplus : [0,1] \times [0,1] \rightarrow [0,1]\) is a t-conorm if it satisfies the following conditions: 1. \(\oplus \{ a, b\} = \oplus \{ b, a \} \) (commutativity); 2. \(\oplus \{ a, \oplus \{ b, c \} \} = \oplus \{ \oplus \{ a,b \}, c \}\) (associativity); 3. \(a \le c\) and \(b \le d \rightarrow \oplus \{ a, b\} \le \oplus \{ c, d\}\) (monotonicity); 4. \(\oplus \{ a, 0\} = a\) (neutral element). a t-conorm acts as a disjunction in fuzzy logic or as a union in fuzzy set theory. when one of its arguments is 0, it returns its other argument; when one of its arguments is 1, it returns 1. it is both associative and commutative, and its partial derivatives with respect to its parameters are non-negative. t-conorms are a generalization of the usual two-valued logical disjunction (or the maximum operator), studied by classical logic, for fuzzy logics. the four basic t-conorms are described below: definition 8 (basic t-conorms [24]). the following are the four basic t-conorms: 1. maximum t-conorm: \(\oplus _\mathbf{m }\{x,y \} = max(x,y)\); 2. probabilistic sum t-conorm: \(\oplus _\mathbf{p }\{ x,y \} = x+y x\cdot y\); 3. łukasiewicz t-conorm: \(\oplus _\mathbf{l } \{ x,y \} = min(x + y,1)\); 4. drastic sum t-conorm: \(\oplus _\mathbf{d } \{ x,y \} =\) \(\left\{ \begin{array}{ll} 1, &{} \text {if } (x,y) \in \ ]0,1]\times ]0,1]; \\ max(x,y), &{} \text {otherwise}. \end{array} \right. \) these four basic t-conorms are remarkable for several reasons. the drastic sum \(\oplus _\mathbf{d }\) and the maximum \(\oplus _\mathbf{m }\) are the largest and the smallest t-conorms, respectively (with respect to the pointwise order). the maximum \(\oplus _\mathbf{m }\) is the only t-conorm where each \(x \in [0,1]\) is an idempotent element (recall \(x \in [0,1]\) is called an idempotent element of \(\oplus \) if \(\oplus \{ x,x \} = x\)). the probabilistic sum \(\oplus _\mathbf{p }\) and the łukasiewicz t-conorm \(\oplus _\mathbf{l }\) are examples of two important subclasses of t-conorms, namely, the classes of strict and nilpotent t-conorms, respectively (more details in [23]). one way to compare t-conorms is using the notion of strength. consider two t-conorms \(\oplus _1\) and \(\oplus _2\). if we have \(\oplus _1 \{ x,y \} \le \oplus _2 \{ x,y \}\) for all \(x,y \in [0,1]\), then we say that \(\oplus _1\) is weaker than \(\oplus _2\) or, equivalently, that \(\oplus _2\) is stronger than \(\oplus _1\), and we write in this case \(\oplus _1 \le \oplus _2\). we shall write \(\oplus _1 < \oplus _2\) if \(\oplus _1 \le \oplus _2\) and \(\oplus _1 \not = \oplus _2\). the drastic sum \(\oplus _\mathbf{d }\) is the strongest, and the maximum \(\oplus _\mathbf{m }\) is the weakest t-conorm, i.e., for each t-conorm \(\oplus \) we have \(\oplus _\mathbf{m } \le \oplus \le \oplus _\mathbf{d }\). between the four basic t-conorms we have these strict inequalities: \(\oplus _\mathbf{m }< \oplus _\mathbf{p }< \oplus _\mathbf{l } < \oplus _\mathbf{d }\). example 5 consider \(x=0.4\) and \(y=0.7\). we have that \(\oplus _\mathbf{m }\{0.4,0.7 \} = 0.7\), \(\oplus _\mathbf{p }\{0.4,0.7 \} = 0.82\), \(\oplus _\mathbf{l }\{0.4,0.7 \} = 1\) and \(\oplus _\mathbf{d }\{0.4,0.7 \} = 1\). we can state that the maximum t-conorm disconsiders all the values that are not the maximum, while the result of the probabilistic sum t-conorm takes into consideration the value of each argument. the łukasiewicz t-conorm follows a similar idea, however, when the sum of the argument reaches a threshold, i.e. the value 1 which represents total membership, the result is equal to 1 (and all the excess is disconsidered). the drastic sum t-conorm, as the name states, it is radical in the decision: if a argument x has a partial (i.e., \(x \in (0,1]\)) or total membership (i.e., \(x = 1\)), the result of drastic t-conorm between x and any other element is equal to 1 (any partial membership is transformed in a total membership). as the t-conorms are a natural generalization of the maximum operator, we can generalize the definition of weighted max-based semantics to a weighted t-conorm semantics. definition 9 (weighted t-conorm semantics). for any wag \(\mathbf {g} = \langle \mathcal {a},w\), \( \mathcal {r} \rangle \) and \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {\oplus }}_{\mathbf {g}}(a) = \frac{w(a)}{1 + \bigoplus _{b \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {\oplus }}_{\mathbf {g}}(b) }\). by convention, if \(\text {att}_\mathbf {g}(a) = \emptyset \), \(\bigoplus _{b \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {\oplus }}_{\mathbf {g}}(b) = 0\). intuitively, we can think of the probabilistic sum semantics as the following idea: the degree of acceptability of an argument a is measured based on its weight and also the sum of pairs of acceptability degrees of the attacking arguments minus a rate (measured by the product between the attackers’ acceptability degrees). it is a different approach to the maximum that only considers the largest value of an attacking argument and also different from the sum approach present in the h-categorizer semantics. in fact, the probabilistic sum semantics encompasses a bit of both worlds. łukasiewicz’s semantics is much closer to h-categorizer semantics, and its big difference is that if the sum of the acceptability degrees of the attacking arguments exceeds 1, the weight of this attack is considered 1 (in the h-categorizer semantics this value can be greater than 1). drastic semantics is much less interesting when compared to the two previous ones, since it considers that if an argument is attacked by any other argument or a set of arguments with a degree of acceptability greater than 0, the total weight of the attack is always 1, regardless the values of the degrees of acceptability. in practice this means that when an argument is attacked, if the weight of any argument is greater than 0, the degree of acceptance of that argument attacked results in half its original weight. example 6 considering the wag \(\mathbf {g}\) from example 1, we have that \(\text {deg}^{\mathbf {\oplus }_\mathbf{m }}_{\mathbf {g}}(a) = 0.71\), \(\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(a) = 0.63\), \(\text {deg}^{\mathbf {\oplus }_\mathbf{l }}_{\mathbf {g}}(a) = 0.58\) and \(\text {deg}^{\mathbf {\oplus }_\mathbf{d }}_{\mathbf {g}}(a) = 0.5\). as said before, the drastic t-conorm is the strongest t-conorm and maximum t-conorm is the weakest. consequently, the acceptability degree of an argument is higher for the maximum semantics and lower for the drastic semantics, when compared to the other t-conorms semantics. with respect to the range of values of the acceptability degree for t-conorms semantics, we have the following results. proposition 2 for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \) and for any \(a \in \mathcal {a}\), we have that \( \{ \text {deg}^{\mathbf {\oplus }_\mathbf{m }}_{\mathbf {g}}(a),\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(a),\text {deg}^{\mathbf {\oplus }_\mathbf{l }}_{\mathbf {g}}(a) \} \in [\frac{w(a)}{2} ,w(a)]\) and \(\text {deg}^{\mathbf {\oplus }_\mathbf{d }}_{\mathbf {g}}(a) = \frac{w(a)}{2}\) or \(\text {deg}^{\mathbf {\oplus }_\mathbf{d }}_{\mathbf {g}}(a) = w(a)\). next, we will propose a weaker principle than (quality precedence), named (quality compensation). the idea of this principle is to prioritize, during a conflict of attacks on two arguments, the quality of attacks when the overall sum of attacks on these arguments is equal. principle 16 (quality compensation). a semantics \(\mathbf {s}\) satisfies quality compensation iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), iii) \(\sum _{x \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \sum _{y \in \text {att}_\mathbf {g}(b)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) \), and iv) \(\exists y \in \text {att}_\mathbf {g}(b)\) such that \(\forall x \in \text {att}_\mathbf {g}(a), \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x)\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). this principle weakens (quality precedence) by introducing condition iii) \(\sum _{x \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \sum _{y \in \text {att}_\mathbf {g}(b)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) \), where it considers equal the sum of the acceptability degrees of the attacking arguments. therefore, if a semantics satisfies (quality precedence) it also satisfies (quality compensation). regarding weighted t-conorms semantics, we have the following results according to their properties. theorem 4 considering the weighted t-conorm semantics: 1. weighted drastic t-conorm semantics violates only (counting), (reinforcement), (quality precedence), (cardinality precedence), (compensation) and (quality compensation). 2. weighted łukasiewicz t-conorm violates only (quality precedence), (cardinality precedence) and (quality compensation). 3. weighted probabilistic sum t-conorm semantics violates only (quality precedence) and (cardinality precedence). weighted drastic t-conorm semantics behaves similar to weighted max-based semantics, except that it does not satisfy (quality precendence) or (quality compensation). although they are t-conorms, weighted łukasiewicz and weighted probabilistic sum semantics go in a direction different from weighted max-based semantics and satisfy (compensation), along with all the 1–12 principles. in special, weighted probabilistic sum also satisfies (quality compensation), which is a weaker version of (quality precedence), that is, it presents a balance between compensation and quality in its decisions. example 7 from the wag \(\mathbf {g}\) in example 1, we have that \(w(a) = w(b)\) and the sum of degrees of acceptability (for \(\mathbf {\oplus }_\mathbf{p }\) semantics) of the attackers of a and b are, respectively, \(\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(d) + \text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(e) = 0.4 + 0.3 = 0.7\) and \(\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(e) + \text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(f) + \text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(g) = 0.3 + 0.34 + 0.06 = 0.7\). however, \(\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(a) = 0.63 < 0.64 = \text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(b),\) since the argument a has an attacker with the highest degree of acceptability (\(\text {deg}^{\mathbf {\oplus }_\mathbf{p }}_{\mathbf {g}}(d) = 0.4\)). we can see that quality takes on importance when the total of values compared is indistinguishable. otherwise, the semantics behaves like a compensating semantics. the next semantics introduced in this paper is the weighted cs-based semantics, based on the idea of cumulative sum. this operator has been applied in the areas of outlier detection [25], identifying rare items, events or observations which raise suspicions by differing significantly from the majority of the analyzed data. it is also studied in the area of economy, from the notion of the lorenz curve [27], which is most often used to represent economic inequality and it can also demonstrate unequal distribution in any system. formally, a cumulative sum can be defined as follows. definition 10 (cumulative sum). consider the vectors \(l = (x_1, \dots , x_n)\) and \(l' = (x_{\sigma (1)}, \dots , x_{\sigma (n)})\), where \(\sigma \) is the permutation of \(\{ 1, \dots , n \}\) sorting the \(x_i\) in descending order. we define the vector of accumulated sum \(as_l = (as^1_{l'}, \dots , a^n_{l'})\), where \(as^i_{l'} = \displaystyle \sum ^{i}_{x_k \in l', k=1} x_k\). the cumulative sum of l is defined as \(cs_l = \sum as_l\) (the sum of its elements). a cumulative sum is a sequence of partial sums of a given sequence. for example, the cumulative sums of the sequence a, b, c, ..., are \(a, a+b, a+b+c, \dots \). after that, the sum of all these elements is performed. example 8 consider the vector \(v = (0.23,0.26,0.1)\) and \(v' = (0.26,0.23,0.1)\) its ordered version. the cumulative sum of v is given by \(cs_v = \sum as_v = \sum (as^1_{v'}, as^2_{v'}, a^3_{v'}) = \sum (0.26, 0.49, 0.59) = 1.34\). definition 11 (weighted cs-based semantics). for any wag \(\mathbf {g} = \langle \mathcal {a},w\), \( \mathcal {r} \rangle \) and \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(a) = \frac{w(a)}{1 + cs_{\text {atts}_\mathbf {g}(a)} }\), where \(\text {atts}_\mathbf {g}(a) = (\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(b_1), \dots , \text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(b_n))\) and \(b_i \in \text {att}_\mathbf {g}(a)\), for \(1 \le i \le n\). by convention, if \(\text {att}_\mathbf {g}(a) = \emptyset \), then \(cs_{\text {atts}_\mathbf {g}(a)} = 0\). weighted cs-based semantics extends the h-categorizer semantics by making the cumulative sum of the acceptability degrees of the attacking arguments. example 9 considering the wag \(\mathbf {g}\) from example 1, we have that \(\text {deg}^{\mathbf {hbs}}_{\mathbf {g}}(a) = 0.58\) and \(\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(a) = 0.47\). as it happened with the maximum t-conorm, which has a higher acceptability degree when compared with the other t-conorms, the acceptability degree of an argument is higher for the h-categorizer when compared to cumulative sum semantics. as said previously, the cumulative sum is used in data analysis to detect variations and anomalies in a set of data, and this operator is sensible to the number of elements in the set. thus, we propose a weaker principle than (cardinality precedence), named (cardinality compensation). the idea of this principle is to prioritize, during a conflict of attacks on two arguments, the quantity of attacks when the overall sum of attacks on these arguments is equal. principle 17 (cardinality compensation). a semantics \(\mathbf {s}\) satisfies cardinality compensation iff for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), \(\forall a, b \in a\), if i) \(w(a) = w(b)\), ii) \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > 0\), iii) \(\sum _{x \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \sum _{y \in \text {att}_\mathbf {g}(b)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) \), and iv) \(|\{y \in \text {att}_\mathbf {g}(b) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y)> 0\}|> |\{x \in \text {att}_\mathbf {g}(a) : \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) > 0\}|\), then \(\text {deg}^{\mathbf {s}}_{\mathbf {g}}(a) > \text {deg}^{\mathbf {s}}_{\mathbf {g}}(b)\). this principle weakens (cardinality precedence) by introducing condition iii) \(\sum _{x \in \text {att}_\mathbf {g}(a)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(x) = \sum _{y \in \text {att}_\mathbf {g}(b)} \text {deg}^{\mathbf {s}}_{\mathbf {g}}(y) \), where it considers equal the sum of the acceptability degrees of the attacking arguments. therefore, if a semantics satisfies (cardinality precedence) it also satisfies (cardinality compensation). regarding weighted cs-based semantics, we have the following results according to their properties. theorem 5 weighted cs-based semantics satisfies all the principles except (cardinality precedence), (quality precedence) and (quality precedence). additionally, it satisfies (cardinality compensation). example 10 from the wag \(\mathbf {g}\) in example 1, we have that \(w(b) = w(c) = 1\) and the sum of degrees of acceptability (for cumulative sum semantics) of the attackers of b and c are, respectively, \(\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(e) + \text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(f) + \text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(g) = 0.3 + 0.23 + 0.06 = 0.59\) and \(\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(g) + \text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(h) = 0.06 + 0.53 = 0.59\). however, \(\text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(b) = 0.41 < 0.47 = \text {deg}^{\mathbf {csbs}}_{\mathbf {g}}(c),\) since the argument b has more attackers than argument c. we can see that quantity takes on importance when the total of values compared is indistinguishable. otherwise, the semantics behaves like a compensating semantics. with respect to the range of values of the acceptability degree for cs-based semantics, they have the same result of the h-categorizes semantics. theorem 6 for any wag \(\mathbf {g} = \langle \mathcal {a},w, \mathcal {r} \rangle \), for any \(a \in \mathcal {a}\), \(\text {deg}^{\mathbf {cs}}_{\mathbf {g}}(a) \in (0,w(a)]\). 4 conclusion this paper introduced two new semantics for the weighted argumentation framework. the main objective is to show that there are semantics that have a hybrid behavior between the principles of (quality precedence), (cardinality precedence) and (compensation). the semantics based on t-conorms, as the drastic t-conorms, łukasiewicz and probabilistic sum seek an alternative to the maximum operator, which has the principle of (quality precedence) as a characteristic. it has been shown that the probabilistic sum t-conorm has the most interesting properties, because although it does not satisfy (quality precedence), it satisfies all the basic principles, along with (compensation) and (quality compensation). table 1. the list of the principles satisfied (or violated) by the semantics.full size table the semantics based on cumulative sum is intended to exhibit a hybrid behavior between a (compensation) and (cardinality precedence). unlike the classical sum operator, presented in the h-categorizer semantics, the cumulative sum gives more weight to the total sum according to the number of elements: the more elements, the greater the sum value. as a result, we show that the cumulative sum semantics satisfies all the principles of the h-categorizer semantics plus a weak version of (cardinality precedence), called (cardinality compensation). table 1 summarizes the results regarding the weighted argumentation framework operators and the satisfaction of all principles. as future work, we intend to continue exploring other operators with intermediate characteristics between these three main incompatible principles. an alternative is to study t-conorm families. besides the four basic t-conorms, it is possible to extend them into several families of t-conorms through parameters [28], resulting in several operators with different properties. another point to be investigated are operators that exhibit intermediate behavior between quality and quantity. furthermore, another possibility of research is to study other new principles in the weighted argumentation framework. a remarkable question remains open: is it possible to conceive a new semantics with a hybrid behavior involving not only two, but these three main incompatible principles? references adeli, h., hung, s.l.: machine learning: neural networks, genetic algorithms, and fuzzy systems. wiley, hoboken (1994) google scholar  amgoud, l., ben-naim, j.: ranking-based semantics for argumentation frameworks. in: liu, w., subrahmanian, v.s., wijsen, j. (eds.) sum 2013. lncs (lnai), vol. 8078, pp. 134–147. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-40381-1_11 chapter  google scholar  amgoud, l., ben-naim, j.: axiomatic foundations of acceptability semantics. in: international conference on principles of knowledge representation and reasoning (kr 2016), pp. pp–2 (2016) google scholar  amgoud, l., ben-naim, j., doder, d., vesic, s.: acceptability semantics for weighted argumentation frameworks. in: twenty-sixth international joint conference on artificial intelligence (2017) google scholar  amgoud, l., besnard, p.: bridging the gap between abstract argumentation systems and logic. in: godo, l., pugliese, a. (eds.) sum 2009. lncs (lnai), vol. 5785, pp. 12–27. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-04388-8_3 chapter  google scholar  amgoud, l., cayrol, c.: a reasoning model based on the production of acceptable arguments. ann. math. artif. intell. 34(1), 197–215 (2002) article  mathscinet  google scholar  amgoud, l., maudet, n., parsons, s.: modelling dialogues using argumentation. in: proceedings fourth international conference on multiagent systems, pp. 31–38. ieee (2000) google scholar  amgoud, l., prade, h.: using arguments for making and explaining decisions. artif. intell. 173(3), 413–436 (2009) article  mathscinet  google scholar  baroni, p., giacomin, m.: on principle-based evaluation of extension-based argumentation semantics. artif. intell. 171(10–15), 675–700 (2007) article  mathscinet  google scholar  baroni, p., romano, m., toni, f., aurisicchio, m., bertanza, g.: automatic evaluation of design alternatives with quantitative argumentation. argum. comput. 6(1), 24–49 (2015) article  google scholar  bench-capon, t.j.: persuasion in practical argument using value-based argumentation frameworks. j. log. comput. 13(3), 429–448 (2003) article  mathscinet  google scholar  besnard, p., hunter, a.: a logic-based theory of deductive arguments. artif. intell. 128(1–2), 203–235 (2001) article  mathscinet  google scholar  da costa pereira, c., tettamanzi, a.g., villata, s.: changing one’s mind: erase or rewind? possibilistic belief revision with fuzzy argumentation based on trust. in: ijcai, international joint conference on artificial intelligence (2011) google scholar  de baets, b., kerre, e.: fuzzy relations and applications. in: advances in electronics and electron physics, vol. 89, pp. 255–324. elsevier (1994) google scholar  deschrijver, g., cornelis, c., kerre, e.e.: on the representation of intuitionistic fuzzy t-norms and t-conorms. ieee trans. fuzzy syst. 12(1), 45–61 (2004) article  google scholar  deschrijver, g., kerre, e.e.: on the composition of intuitionistic fuzzy relations. fuzzy sets syst. 136(3), 333–361 (2003) article  mathscinet  google scholar  dubois, d., fargier, h., bonnefon, j.f.: on the qualitative comparison of decisions having positive and negative features. j. artif. intell. res. 32, 385–417 (2008) article  mathscinet  google scholar  gabbay, d.m., rodrigues, o.: equilibrium states in numerical argumentation networks. logica universalis 9(4), 411–473 (2015) article  mathscinet  google scholar  giannini, f., marra, g., diligenti, m., maggini, m., gori, m.: learning and t-norms theory. arxiv preprint arxiv:1907.11468 (2019) grzegorzewski, p., mrówka, e.: soft querying via intuitionistic fuzzy sets. in: proceedings of the 9th international conference on information processing and management of uncertainty in knowledge-based systems impu 2002. citeseer (2002) google scholar  hunter, a.: a probabilistic approach to modelling uncertain logical arguments. int. j. approx. reason. 54(1), 47–81 (2013) article  mathscinet  google scholar  kakas, a., moraitis, p.: argumentation based decision making for autonomous agents. in: proceedings of the second international joint conference on autonomous agents and multiagent systems, pp. 883–890 (2003) google scholar  klement, e.p., mesiar, r.: logical, algebraic, analytic and probabilistic aspects of triangular norms. elsevier science b.v, amsterdam (2005) google scholar  klement, e.p., mesiar, r., pap, e.: triangular norms, 1st edn. springer, netherlands (2000). https://doi.org/10.1007/978-94-015-9540-7 book  math  google scholar  lazarevic, a., kumar, v.: feature bagging for outlier detection. in: proceedings of the eleventh acm sigkdd international conference on knowledge discovery in data mining, pp. 157–166 (2005) google scholar  li, h., oren, n., norman, t.j.: probabilistic argumentation frameworks. in: modgil, s., oren, n., toni, f. (eds.) tafa 2011. lncs (lnai), vol. 7132, pp. 1–16. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-29184-5_1 chapter  google scholar  lorenz, m.o.: methods of measuring the concentration of wealth. publ. am. stat. assoc. 9(70), 209–219 (1905) google scholar  mizumoto, m.: pictorial representations of fuzzy connectives, part i: cases of t-norms, t-conorms and averaging operators. fuzzy sets syst. 31(2), 217–242 (1989) article  google scholar  modgil, s.: reasoning about preferences in argumentation frameworks. artif. intell. 173(9–10), 901–934 (2009) article  mathscinet  google scholar  prakken, h.: formal systems for persuasion dialogue. knowl. eng. rev. 21(2), 163 (2006) article  google scholar  rago, a., toni, f., aurisicchio, m., baroni, p.: discontinuity-free decision support with quantitative argumentation debates (2016) google scholar  thimm, m.: a probabilistic semantics for abstract argumentation. in: ecai, vol. 12, pp. 750–755 (2012) google scholar  yager, r.r., kreinovich, v.: universal approximation theorem for uninorm-based fuzzy systems modeling. fuzzy sets syst. 140(2), 331–339 (2003) article  mathscinet  google scholar  download references author information authors and affiliations instituto federal de educação, ciência e tecnologia do ceará (ifce), rodovia ce-040, km 137, 1 s/n, aracati, ce, brazil henrique viana departamento de ciência da computação, universidade federal do ceará (ufc), p.o. box 12166, fortaleza, ce, 60455-760, brazil joão alcântara authors henrique vianaview author publications search author on:pubmed google scholar joão alcântaraview author publications search author on:pubmed google scholar corresponding author correspondence to henrique viana . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper viana, h., alcântara, j. (2021). on the refinement of compensation-based semantics for weighted argumentation frameworks. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_23 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords argumentation t-conorms cumulative sum publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature performance analysis of generative adversarial networks and diffusion models for face aging | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper performance analysis of generative adversarial networks and diffusion models for face aging conference paper first online: 12 october 2023 pp 228–242 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) performance analysis of generative adversarial networks and diffusion models for face aging download book pdf download book epub bruno kemmer  orcid: orcid.org/0000-0002-6830-32819, rodolfo simões  orcid: orcid.org/0000-0001-6989-12339, victor ivamoto  orcid: orcid.org/0000-0001-5463-36459 & … clodoaldo lima  orcid: orcid.org/0000-0003-3381-53489  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 571 accesses abstract computational face aging enables predicting a person’s future appearance using algorithms, with the goal that the output age is close to the expected age and that the individual’s characteristics are maintained. in this work, we evaluate the performance of four generative models on facial aging. two models are based on generative adversarial networks (gans), hrfae, and sam, and the other two are based on diffusion models, pix2pix-zero and instruct-pix2pix. the first two were explicitly trained to generate an aged version of the original person, and the others have a zero-shot generation; in other words, they are generic models that perform different tasks, including facial aging. since diffusion models have been gaining attention because of their diversity and high-quality image generation, comparing their results against models specifically designed for the task using meaningful metrics is essential. therefore, we compared these models using the ffhq aging database and with the metrics: mean absolute error (mae) of the predicted age, fréchet inception distance (fid), and the cosine similarity of the facenet’s embeddings. v. ivamoto—partially supported by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others face aging using generative adversarial networks chapter © 2022 mustat: face ageing using multi-scale target age style transfer chapter © 2024 agetransgan for facial age transformation with rectified performance metrics chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. aging population ageing gerontology mathematical models of cognitive processes and neural networks neural ageing skin models 1 introduction computational methods of facial aging aim to generate an aged face while maintaining individual characteristics. however, several factors can cause aging: excess sun exposure, smoking, a polluted environment, stress, and genetic factors. in addition, surgical and non-surgical aesthetic procedures can mitigate the effects of time, as well as the use of cosmetic substances at the time of image capture. for these reasons, facial aging is a complex and non-deterministic process. recently, this theme has been the subject of many publications [1, 5, 8, 38], as it can help in an automated way in the search for missing people; in the identification of criminals, and also for entertainment purposes. furthermore, its use is also present in biometric tasks (identification of individuals based on physical or behavioral characteristics), as it makes it possible to reduce the distance between the individual characteristics present at the time of training and the current state of the faces, especially if the training has been done with old images [14, 18]. many recent publications have used generative models to perform facial aging, obtaining realistic results. initially, adversarial autoencoders were used, and later, generative adversarial models (gans) and, recently, diffusion models [36, 40]. this study aims to evaluate four generative models on the facial aging task. in particular, compare if zero-shot diffusion models can generate aged images on par with gans explicitly trained for the face aging task. to achieve that, we analyze the outputs of two state of art models based on gans, hrfae, and sam, with two diffusion models, pix2pix-zero and instruct-pix2pix. using the following metrics: mean average error (mae) of the predicted age (measuring age regression), fréchet inception distance, fid (checking for image reality), and the cosine similarity of the embedding of a pre-trained face recognition network, facenet (gauging the adherence to the individual characteristics). the article is organized as follows. section 2 aims to show related works that have a degree of similarity with the study. section 3 presents a brief review of gans and diffusion models, a summary of the main aspects of each model used in this comparison, a few methods used inside the models, the ffhq aging image database, and evaluation metrics. section 4 details the methodology used to compare the techniques. section 5 presents the conclusion. 2 related work in the literature, many works compare the aging effects on human faces using generative models, in particular, employing generative adversarial networks, gans, and a fewer number applying diffusion models on the task. however, we couldn’t find a work that compares these two techniques using the same dataset. the review [8] presents the main models that perform facial aging using deep networks, the evolution of the number of publications, a taxonomy of existing techniques, and the most important databases with faces and proper metadata. also, in the review, the authors evaluate three aging models for high-definition images recently published: hrfae [38], life [23], and sam [1]. in the comparison, they use external photos of 4 young people, between 24 and 32 years old, and present how the models behave when trying to age the photos for three age groups: 65, 50–69, and 60. to perform a quantitative analysis, the authors estimate the age of the original photo using a pre-trained model arcface [4] and each generated photo. in addition, the fid metric is used to quantify the perception of the reality of the images. finally, they measured the cosine distance between the result of the original image and the aged one in the last layer of the pre-trained network. moreover, in work [17], the authors review the literature and compare four aging models based on gans: caae [39], ipcgan [35] and rcriit [9]. in the comparison, the authors used the same set of images from three aging databases: fg-net, utkfaces, and cacd. additionally, in hrfae [38], the authors compared their model with ipcgan [35] and s2gan [37] using the ffhq aging database. in sam [1], the authors compare their model with life [23] and hrfae [38], stating that at the time of publication, they were considered state-of-the-art works. the database used to evaluate the results is celeba-hq, as it contains celebrities’ faces and ages. as in [8], the authors used arcface to calculate the cosine similarity of each pair of images. the study also presents a qualitative analysis of the images generated for each work. in their analysis, the authors generated 80 images and chose the image in which the arcface model had the closest prediction of the desired one. the sam and hrfae models can generate faces for a predefined age. however, for comparison purposes, the same protocol was used. finally, the authors used the arcface model during training and employed the microsoft azure face api to access identity maintenance. a survey was also carried out in which humans evaluated photos of the same individual and answered which images they preferred, using the desired age and the quality of the generated image as metrics. 3 material and methods 3.1 generative adversarial networks generative adversarial networks, gans [7], are composed of two neural networks: the generator, which is a network to generate examples close to real data by learning the distribution of the training data, and the discriminator, a classification network that aims to separate the generated data from the real ones. these two networks compete with each other during the training stage, and the generator tries to produce examples so close to the real ones as to deceive the discriminator, which tries to improve the detection of the generated images. over time architecture improvements have been introduced to improve the image quality and mitigate training problems. hrfae. in high-resolution face age editing, hrfae [38], the authors used an encoder-decoder architecture to perform age editing of photos in high resolution (1024\(\,\times \,\)1024). the g generator consists of an e encoder and a d decoder. the model receives the input image \(x_0\), which passes through the encoder generating two copies of a latent vector \(e(x_0)\). the pre-trained dex age estimator [29] is used to determine the age of the input image \(\alpha _0\). this age will be encoded through a binary encoding module with a sigmoid activation function. its decoder d has two tasks: it receives the latent vector e(z) and produces an image as similar as possible to the input image \(g(x_0, \alpha _0)\), and also makes the aged image \( g(x_0, \alpha _1)\) realistic and close to the desired age. to achieve this goal, its cost function has three components, as shown in eq. 1: 1. adversarial loss (\(l_{gan}\)) uses patchgan [15] with the objective function of lsgan [21]. 2. age classification loss (\(l_{class}\)) in which the pre-trained dex model is used to estimate the age obtained with a categorical cross-entropy loss function. 3. reconstruction loss (\(l_{recon}\)) monitors the model’s ability to reconstruct the original image at the initial age. \(l_{recon} = \vert g(x_0, \alpha _0)-x_0) \vert _1\) $$\begin{aligned} \lambda = \lambda _{recon} l_{recon} + \lambda _{class} l_{class} + l_{gan}, \end{aligned}$$ (1) where \(\lambda _{recon}\) and \(\lambda _{class}\) are hyper-parameters that weigh between maintaining the identity (reconstruction of the original image at the initial age) and the aging effect (classification at the correct age). in training performed by the authors, they used the flickr-faces-hq, ffhq, and high-definition image base [16]. it presents fewer images of older people, so they used the stylegan network to generate 300,000 synthetic images. they thus obtained a base of images balanced in the criteria of the ages present in it. the authors only used synthetic images in age groups with insufficient real images. ultimately, they obtained 47,990 images ranging in age from 20 to 69. in this work (hrfae), the authors published, along with their results, metadata of the images containing pose, age, and gender prediction, and the segmentation of the regions of the faces present in ffhq, calling it ffhq aging. this is the database used in the present work. sam. in style-based age manipulation, sam [1], the authors developed an architecture that enables facial aging or rejuvenation using a real image x and a desired age \(\alpha _t\) as input. to achieve this, they perform an image-to-image translation. the first step is to find the best latent vectors representing x in the \(w^*\) space of an unconditional gan (stylegan) that can reconstruct the original face, to this task, they employed a previously trained coder (psp) [26]. its output, a series of style vectors, makes it possible to reconstruct the original image when passing through the stylegan generator. a second encoder, \(e_{age}\), is trained to capture the difference (residual) between the reconstructed image obtained by the first encoder (psp) and the aged image. in this second encoder, a pre-trained dex network [29] was used to guide the training toward the desired age, and a pre-trained network for face recognition arcface [4] was employed to maintain the individual characteristics of the original image. these networks were not changed during the training of the \(e_{age}\) encoder, keeping their parameters fixed. the outputs of both encoders are summed and become the latent input vector that stylegan uses. additionally, because it is an image-to-image translation, the model uses a cyclic loss to reconstruct the original image after a cycle consistency pass. the cost functions used were: pixel-to-pixel similarity \(\mathcal {l}_2(x_{age}) = \vert x-sam(x_{age}) \vert _2\) perceptual similarity loss: \(\mathcal {l}_{lpips}(x_{age}) = \vert f(x)-f(sam(x_{age})) \vert _2\). regularization loss: this regularization causes the style vectors to be close to the average of the latent vectors. the authors identified that its use improves image quality by removing unwanted artifacts in the images produced. identity loss: difference in cosine similarities between the output and input image, weighted by the number of years between the images. if the difference is many years, there is expected to be a loss of identity, \(\mathcal {l}_{id}\). age loss: to verify the quality of aging/rejuvenation in the generated image, a pre-trained dex network was used, \(\mathcal {l}_{age} = \vert \alpha _t dex(sam(x_{age})) \vert _2\). 3.2 diffusion models diffusion models aim to destroy the data distribution structure slowly and systematically in a way that enables learning a reverse diffusion process, which recreates the original structure of the data, generating a very flexible and computationally tractable generative model of the data [33]. in the work denoising probabilistic diffusion models (ddpm) [12], the authors achieved good results when trying to predict the noise \(\mathcal {n}(\mu ,\,\sigma ^{2})\) but fixing \(\sigma ^{2}\). the noise was also added linearly in the forward step, using a linear schedule. moreover, the reconstruction was done using a u-net architecture with attention blocks. later, some enhancements were proposed [22], adding the noise by cosine scheduling in the forward step instead of a linear scheduler, as they noted that this approach improved learning by destroying the image signal more slowly. the neural network that reconstructs the data began to learn the parameters in \(\sigma ^{2}\), increased the depth of the layers and decreased their number, increased the number of attention layers and the number of heads of attention, and other improvements. in denoising diffusion implicit models, ddim [34] showed that using a non-markovian approach in the forward step of ddpm models made it possible to use only a subset (progressive) of the steps t in a sampling trajectory. thus, it considerably accelerated the sampling process because, instead of 1,000 noise removal steps used in ddpm models, it was possible to obtain good results with 200 or fewer. equation 2 shows the addition of noise in the sampling process obtained on the image, \(x_t\), at step t, with gaussian noise \(\epsilon \), unitary variance \(\alpha _t\) and the original image \(x_0\). in eq. 3 is shown the inverse, predicting the noise that will be removed \(x_t\) towards \(x_0\), where \(\epsilon _t \sim \mathcal {n}({\textbf {0}}, {\boldsymbol{i}})\). $$\begin{aligned} x_t = \sqrt{\alpha _t}x_0 + \sqrt{1-\alpha _t}\epsilon \end{aligned}$$ (2) $$\begin{aligned} {\textbf {x}}_{t-1} = \sqrt{\alpha _{t-1}} \underbrace{ \left( \frac{{\textbf {x}}_t \sqrt{1-\alpha _t}e_{\theta }^{(t)}({\textbf {x}}_t)}{\sqrt{\alpha _t}} \right) }_{\text {``} x_{0}\,\, \text {predicted''}} \underbrace{ + \sqrt{1-\alpha _{t-1}-\sigma ^{2}_t} \cdot \epsilon _{\theta }^{(t)}({\textbf {x}}_t) }_{\text {``direction towards}\,\, x_{t} \text {''}} + \sigma _t\epsilon _t \end{aligned}$$ (3) latent diffusion models. latent diffusion models, ldm [27] gained notoriety for being able to generate realistic text-guided images. with the availability of the pre-trained latent diffusion model stable diffusion, trained with high-resolution images present in the laion database [32] quickly gained prominence due to its open source code and high-quality text-to-image generation. one of the work’s main innovations was using a dimension reduction technique, a variational autoencoder model, and performing the diffusion process in this reduced domain. therefore, this allowed the use of self-attention modules, as these increase complexity quadratically based on the input data. additionally, during the model training, the energy consumed and processing time decreased considerably compared to previously proposed architectures. the model was divided into two phases: 1. in the compression phase, they used a variational autoencoder model to learn the perceptual domain of the images. thus, the diffusion model does not try to add and remove noise from the raw input image, usually in high dimension, but rather from a latent vector (intermediate layer of the autoencoder used), considerably reducing the computational complexity, as it reduces the input size by eight times. this variational autoencoder uses a perceptual loss and has an opposing objective function based on image segments, thus ensuring consistency in each segment. 2. in the generative learning phase, a u-net [28] structure was used with cross-attention mechanisms conditioned to different input forms (text and image) via a domain-specific encoder. one technique used is classifier-free guidance [13], in which two text representations are concatenated, an empty array, and the input sentence. two latent vectors are also concatenated, which will have their noise removed. therefore, the output will have two components, one in which the input text is oriented \(\hat{x}_{cond\_text}\) and another without \(\hat{x}_{incond}\). a model parameter \(h_1\) controls how much the component linked to the input text affects the final image, as seen in eq. 4. larger values of \(h_1\) force the generated image to be more faithful to the input text, and smaller values give more freedom in the final image. $$\begin{aligned} \hat{x} = \hat{x}_{incond} + h_1*(\hat{x}_{text\_cond} \hat{x}_{incond}) \end{aligned}$$ (4) image editing using diffusion models. there are a few ways to perform image editing with diffusion models. three of them are: methods that use ddim inversion to find the latent vector that best reconstructs the original image and perform editing in the forward step of noise removal. these methods do not modify the parameters of the diffusion models used. pix2pix-zero [24] adopts this approach. fine-tuning the weights of pre-trained diffusion models to fit the examples to be edited, as in dreambooth [30] and textual inversion [6]. train a diffusion model to do image editing, as in production-ready face re-aging for visual effects [40] and in instruct-pix2pix [2]. ddim inversion. ddim inversion technique [34] is a way of editing real images and consists of finding the latent variable \(x_t\), which, when traversing the deterministic sampling path, will produce a realistic approximation of the original image \(x_0\). once this is done, it becomes possible to edit just some parts of the coding that conditions the image generation, also being able to replace words while maintaining the characteristics of the original image. another way of editing forms involves using a mask to edit only a region of interest. clip. in contrastive language-image pre-training, clip [25], a neural network was trained on 400 million image pairs in the form of an image and its subtitles obtained from the internet. this network was proposed to be used without the need to be retrained for specific tasks since it aligns in the same dimensional space both the representation of the image (after passing through an encoding model), \(i_n\), and the text (also after going through a process of tokenizing, encoding, and padding to have the same dimension as the image), \(t_n\), since both have the same dimension and the result matrix is the cross product of both \(i_n \cdot t_n \). its objective function tries to maximize the similarity of matching pairs and minimize the similarity of unrelated pairs. the representation obtained by the network has been used in multiple image-related tasks since it aligns the images with text context. pix2pix-zero. in zero-shot image-to-image translation [24], the authors propose a method for editing real images that preserve the characteristics of the original images. following the following steps: 1. perform a ddim inversion to obtain the original latent vector that best represents this real image in the model usedfootnote 1. 2. find an editing direction, they used the gpt-3 text generator template [3] to generate sentences with a source term (e.g. dog) and a target term (e.g. cat). these sentences pass through a clip model to obtain the representations in that domain. the subtraction of the average of these representations will, theoretically, be the editing direction of the images. 3. get a caption (the most text adherent to the image) so that when editing occurs, it happens in the word that best represents the changed term. as the cross-attention modules end up generating masks relating the words to the image, when editing uses this information, it maintains what is not being changed. the authors used the blip model [20] to generate the caption in the work. 4. they performed the editing through cross-attention modules. to do so, they reconstructed the image without applying any editing, just using the input text to obtain the cross-attention modules for each step t. once this was done, they added the editing direction and calculated the gradient loss in relation to the input \(x_t\). this caused the edit to focus on the region represented by that word. instruct-pix2pix. in instruct-pix2pix [2], a method was shown to train a model that can follow human editing instructions on images. the method receives an input image and a text with the instruction and performs editing in the forward step. this is possible because the authors used the gpt-3 [3] template to generate editing instructions and captions for the original and edited images. after that, the authors used the pre-trained network stable diffusion [27] to generate pairs of images referring to the created captions, producing a base of more than 450,000 examples. with these examples, a new diffusion model was trained to generate edited images given an input image and editing instructions. the authors highlighted that text-to-image diffusion models (such as stable diffusion) could generate drastically different images for slightly modified texts. to mitigate this problem, the authors used the technique presented in work prompt-to-prompt [10] in which the weights in the cross-attention modules are calculated relating the words with regions of the image, so editing is restricted to the regions related to the words being edited. furthermore, this model has a parameter \(\rho \)footnote 2 that makes it possible to control the similarity between the two images. to do this automatically, 100 examples \(\rho \sim \ \mathcal {u}(0.1; 0.9)\) were sampled and filtered based on a distance metric in the clip representation space. another important point was that the authors used two parameters in the classifier free guidance: one that controls how much the image corresponds to the input image \(c_{image}\), and the other interferes with how much the instruction should be followed \(c_{instruction}\). 3.3 ffhq aging image database flickr-faces-hq (ffhq) [16] is a database of high-quality human faces intended to be a benchmark for gans. consists of 70,000 high-resolution 1024\(\,\times \,\)1024 images obtained from the platform flickr, chosen by their permissive sharing permissions, and pre-processed (cut and aligned) using the dlib library [19]. the authors comment that this base of images contains much more variations in terms of ethnicity and the background of the photos compared to celeba-hq, and that it also contains several accessories, such as glasses and sunglasses, hats, etc. in life [23], the authors published, along with their results, additional metadata containing pose, prediction of age and gender, and the segmentation of the regions of the faces present in the ffhq database, which they called ffhq aging. it is worth noting that the age ranges present in the ffhq-aging metadata were estimates obtained by the authors using the appen platform that, in addition to the age range, returns a confidence interval of the age prediction. to reduce the error that incorrect estimates may bring, in the comparison presented in this work, only images with a confidence interval equal to 100% were chosen. 3.4 metrics fré chet inception distance (fid). [11] is currently one of the most used gan evaluation methods and uses the previously trained classifier with inceptionnet architecture. the last layer of the network (fully connected) is removed, resulting in its output of a vector with dimension 2048, representing the image of the attributes detected by this network. statistics of these representations, both real and synthetic (generated) images, are used during the fid calculation. low fid values mean that the distributions are close, which is what you want to happen when comparing the distribution of the generated images and the real ones. cosine similarity of face embeddings. one way that multiple articles use to quantify identity maintenance in the aged-generated image is to use the embeddings of a pre-trained face recognition network [1, 23, 35] since it was optimized to find features in the faces that make each individual unique. therefore, calculating the cosine similarity of the original image embedding and the aged version enable quantifying how close they are in the representation space that captures the identity characteristics. 4 experiments these experiments aim to compare the performance of conditioned diffusion models with gans trained specifically for the face aging task, verifying whether the aged image outputs would be of equivalent or superior quality. the image base used was the ffhq aging, in which the photos have an estimated age in already defined groups. in the experiments, it was selected age ranges close to those used by the authors in their works. in each group, 50 images of unique individuals were drawn for both women and men. an interval of 20 years was used between the age ranges, except for the last one, which has a difference of 60 years, to verify an extreme case. in this work, we measured the mean predicted age per group and the mean absolute error (mae), comparing the predicted age with the one estimated by the dex model in the initial image. the similarity of cosines was also calculated between the representations obtained when using the pre-trained network facenet [31]. facenet has a performance similar to arcface [4], but since sam uses arcface in its cost function, the comparison is fairer using another pre-trained network. fig. 1. examples of images in which the original image reconstruction step using the ddim inversion technique. some did not obtain good results, which could compromise the quality of the generated aged images. full size image table 1. model comparison between the aging images.full size table in the experiments carried out with the model instruct-pix2pix changing the value of how much the aging instruction will be followed \(c_{instruction}\), it was observed that if the parameter has the value 0, the image is not aged as the instruction is not followed, which is expected. the error drops considerably by increasing the value of \(c_{instruction}\) to 10. however, the images are no longer close in the facenet representation. given these results, the parameter \(c_{instruction}\) of the instruct-pix2pix model was used with the value 3. it is important to emphasize that the results were analyzed by separating the input images between men and women. however, as no significant differences were noted in the results, the data were unified. in the results presented by table 1, it is possible to notice that sam manages to obtain the lowest error averages, possibly due to having an age estimator in its cost function. however, the similarity between the aged image and the original is smaller than the results of instruct-pix2pix. the high similarity between the images obtained by hrfae can be explained because the minor changes performed by the network. therefore, the images are not properly aged, as seen in the error and low fid values. fig. 2. comparison of results in facial aging tasks in each model. the first column, original, is the input image with the initially estimated age. the four images to its right are the outputs of the models trying to get the image aged with the expected age. for example, 10–30 is the task of aging a face with approximately 10 years to 30 years. full size image fig. 3. comparison of results in facial aging tasks in each model. the first column, original, is the input image with the estimated initial age, and the four images on the right are the results of the models trying to get the aged image with the expected age. full size image in the pix2pix-zero model, two factors can affect the maintenance of the characteristics of individuals: one of them is the fact that the model uses editing directions obtained through a single average direction. this can cause the model to age all images unconditionally, which can be seen by the low similarity and high fid values. another factor is that it needs to reconstruct the original image using the ddim inversion technique; a poor reconstruction can compromise the final aged image. figure 1 shows examples of when this occurs. observing figs. 2 and 3, the sam model is the only one that achieves reasonable performance when aging from approximately ten years to older ages. as it is a growth phase, the face has many changes, and the other models failed to capture these structural changes. the pix2pix-zero model had blur results on some images, which is expected when using an average edit direction to get from one domain to the next. instruct-pix2pix maintained the identity of the individuals and achieved considerable aging. however, in all cases, the skin texture degraded, which is common over the years (especially in cases where there is a lot of sunlight or smoking), but it is not guaranteed that aging will be like this. the sam model seems to have more variability in how it ages faces. 5 conclusion this study analyzed the performance of four models in the facial aging task: two models that used gans in their architecture and two conditional diffusion models that enabled image editing. of these, two models stood out in this analysis, sam, a model based on a pre-trained gan (stylegan2) and specialized in the task of facial aging and rejuvenation, having components of its cost function specific to guarantee identity and age change. the second model to be highlighted was instruct-pix2pix, a generic conditional diffusion model, which only receives instruction in the form of text, being able to edit the images in different ways in a zero-shot fashion. in the study, it was only sent an instruction to age the face to a desired age, and still got realistic aged images. currently, specialist gans still demonstrate superior performance compared to generic diffusion models, such as those used in the experiments. this demonstrates a great capacity in the diffusion model since instruct-pix2pix obtained considerable results even though it was not specifically trained for this task. furthermore, diffusion models have gained notoriety recently, with many works presenting new techniques in recent years, so rapid advances in their architectures are expected. notes 1.the authors used stable diffusion 1.4. 2.ratio of diffusion steps with cross-attention weights. references alaluf, y., patashnik, o., cohen-or, d.: only a matter of style: age transformation using a style-based regression model. acm trans. graph. 40(4), 1–12 (2021) article  google scholar  brooks, t., holynski, a., efros, a.a.: instructpix2pix: learning to follow image editing instructions (2022) google scholar  brown, t.b., et al.: language models are few-shot learners. in: proceedings of the 34th international conference on neural information processing systems. nips’20, curran associates inc., red hook, ny, usa (2020) google scholar  deng, j., guo, j., xue, n., zafeiriou, s.: arcface: additive angular margin loss for deep face recognition. in: 2019 ieee/cvf conference on computer vision and pattern recognition (cvpr), pp. 4685–4694 (2019) google scholar  despois, j., flament, f., perrot, m.: agingmapgan (amgan): high-resolution controllable face aging with spatially-aware conditional gans. in: bartoli, a., fusiello, a. (eds.) eccv 2020. lncs, vol. 12537, pp. 613–628. springer, cham (2020). https://doi.org/10.1007/978-3-030-67070-2_37 chapter  google scholar  gal, r., et al.: an image is worth one word: personalizing text-to-image generation using textual inversion (2022) google scholar  goodfellow, i., et al.: generative adversarial nets, p. 9 (2014) google scholar  grimmer, m., ramachandra, r., busch, c.: deep face age progression: a survey. ieee access 9, 83376–83393 (2021) article  google scholar  heljakka, a., solin, a., kannala, j.: recursive chaining of reversible image-to-image translators for face aging. in: blanc-talon, j., helbert, d., philips, w., popescu, d., scheunders, p. (eds.) acivs 2018. lncs, vol. 11182, pp. 309–320. springer, cham (2018). https://doi.org/10.1007/978-3-030-01449-0_26 chapter  google scholar  hertz, a., mokady, r., tenenbaum, j., aberman, k., pritch, y., cohen-or, d.: prompt-to-prompt image editing with cross attention control (2022) google scholar  heusel, m., ramsauer, h., unterthiner, t., nessler, b., hochreiter, s.: gans trained by a two time-scale update rule converge to a local nash equilibrium. in: proceedings of the 31st international conference on neural information processing systems, nips’17, pp. 6629–6640. curran associates inc., red hook, ny, usa (2017) google scholar  ho, j., jain, a., abbeel, p.: denoising diffusion probabilistic models. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 6840–6851. curran associates, inc. (2020) google scholar  ho, j., salimans, t.: classifier-free diffusion guidance (2022) google scholar  huang, y., hu, h.: a parallel architecture of age adversarial convolutional neural network for cross-age face recognition. ieee trans. circuits syst. video technol. 31(1), 148–159 (2021) article  google scholar  isola, p., zhu, j.y., zhou, t., efros, a.a.: image-to-image translation with conditional adversarial networks. in: proceedings of the ieee conference on computer vision and pattern recognition (cvpr) (2017) google scholar  karras, t., laine, s., aila, t.: a style-based generator architecture for generative adversarial networks. arxiv:1812.04948 [cs, stat] (2019) kemmer, b., simões, r., lima, c.: face aging using generative adversarial networks. in: razavi-far, r., ruiz-garcia, a., palade, v., schmidhuber, j., et al. (eds.) generative adversarial learning: architectures and applications. intelligent systems reference library, vol. 217. springer, cham (2022). https://doi.org/10.1007/978-3-030-91390-8_7 chapter  google scholar  khanna, a., thakur, a., tewari, a., bhat, a.: cross-age face verification using face aging, pp. 94–99 (2020) google scholar  king, d.e.: dlib-ml: a machine learning toolkit. j. mach. learn. res. 10, 1755–1758 (2009) google scholar  li, j., li, d., xiong, c., hoi, s.: blip: bootstrapping language-image pre-training for unified vision-language understanding and generation (2022) google scholar  mao, x., li, q., xie, h., lau, r.y.k., wang, z., smolley, s.p.: least squares generative adversarial networks. in: 2017 ieee international conference on computer vision (iccv), pp. 2813–2821 (2017) google scholar  nichol, a., dhariwal, p.: improved denoising diffusion probabilistic models (2021) google scholar  or-el, r., sengupta, s., fried, o., shechtman, e., kemelmacher-shlizerman, i.: lifespan age transformation synthesis. in: vedaldi, a., bischof, h., brox, t., frahm, j.-m. (eds.) eccv 2020. lncs, vol. 12351, pp. 739–755. springer, cham (2020). https://doi.org/10.1007/978-3-030-58539-6_44 chapter  google scholar  parmar, g., singh, k.k., zhang, r., li, y., lu, j., zhu, j.y.: zero-shot image-to-image translation (2023) google scholar  radford, a., et al.: learning transferable visual models from natural language supervision (2021) google scholar  richardson, e., et al.: encoding in style: a stylegan encoder for image-to-image translation. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition (cvpr), pp. 2287–2296 (2021) google scholar  rombach, r., blattmann, a., lorenz, d., esser, p., ommer, b.: high-resolution image synthesis with latent diffusion models (2021) google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: navab, n., hornegger, j., wells, w.m., frangi, a.f. (eds.) medical image computing and computer-assisted intervention miccai 2015. lecture notes in computer science(), vol. 9351, pp. 234–241. springer, cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28 chapter  google scholar  rothe, r., timofte, r., van gool, l.: dex: deep expectation of apparent age from a single image. in: 2015 ieee international conference on computer vision workshop (iccvw), pp. 252–257 (2015) google scholar  ruiz, n., li, y., jampani, v., pritch, y., rubinstein, m., aberman, k.: dreambooth: fine tuning text-to-image diffusion models for subject-driven generation (2022) google scholar  schroff, f., kalenichenko, d., philbin, j.: facenet: a unified embedding for face recognition and clustering. in: 2015 ieee conference on computer vision and pattern recognition (cvpr), pp. 815–823 (2015) google scholar  schuhmann, c., et al.: laion-400m: open dataset of clip-filtered 400 million image-text pairs (2021) google scholar  sohl-dickstein, j., weiss, e., maheswaranathan, n., ganguli, s.: deep unsupervised learning using nonequilibrium thermodynamics. in: bach, f., blei, d. (eds.) proceedings of the 32nd international conference on machine learning. proceedings of machine learning research, vol. 37, pp. 2256–2265. pmlr, lille, france (2015) google scholar  song, j., meng, c., ermon, s.: denoising diffusion implicit models. arxiv:2010.02502 (2020) wang, z., tang, x., luo, w., gao, s.: face aging with identity-preserved conditional generative adversarial networks, pp. 7939–7947 (2018) google scholar  wolleb, j., sandkühler, r., bieder, f., cattin, p.c.: the swiss army knife for image-to-image translation: multi-task diffusion models (2022) google scholar  yang, h., huang, d., wang, y., jain, a.: learning face age progression: a pyramid architecture of gans, pp. 31–39 (2018) google scholar  yao, x., puy, g., newson, a., gousseau, y., hellier, p.: high resolution face age editing. corr abs/2005.04410 (2020) google scholar  zhang, z., song, y., qi, h.: age progression/regression by conditional adversarial autoencoder, vol. 2017-january, pp. 4352–4360 (2017) google scholar  zoss, g., chandran, p., sifakis, e., gross, m., gotardo, p., bradley, d.: production-ready face re-aging for visual effects. acm trans. graph. 41(6), 1–15 (2022) article  google scholar  download references author information authors and affiliations university of são paulo, são paulo, sp, brazil bruno kemmer, rodolfo simões, victor ivamoto & clodoaldo lima authors bruno kemmerview author publications search author on:pubmed google scholar rodolfo simõesview author publications search author on:pubmed google scholar victor ivamotoview author publications search author on:pubmed google scholar clodoaldo limaview author publications search author on:pubmed google scholar corresponding author correspondence to bruno kemmer . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper kemmer, b., simões, r., ivamoto, v., lima, c. (2023). performance analysis of generative adversarial networks and diffusion models for face aging. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_16 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords face aging generative adversarial networks diffusion models publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature on the impact of mdp design for reinforcement learning agents in resource management | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper on the impact of mdp design for reinforcement learning agents in resource management conference paper first online: 28 november 2021 pp 79–93 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) on the impact of mdp design for reinforcement learning agents in resource management download book pdf download book epub renato luiz de freitas cunha  orcid: orcid.org/0000-0002-3196-300810 & luiz chaimowicz  orcid: orcid.org/0000-0001-8156-994110  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 761 accesses 2 citations abstract the recent progress in reinforcement learning applications to resource management presents markov decision processes (mdps) without a deeper analysis of the impacts of design decisions on agent performance. in this paper, we compare and contrast four different mdp variations, discussing their computational requirements and impacts on agent performance by means of an empirical analysis. we conclude by showing that, in our experiments, when using multi-layer perceptrons as approximation function, a compact state representation allows transfer of agents between environments, and that transferred agents have good performance and outperform specialized agents in 80% of the tested scenarios, even without retraining. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ddma: discrepancy-driven multi-agent reinforcement learning chapter © 2022 multi-agent reinforcement learning chapter © 2020 multi-agent deep q-learning for maintenance scheduling of engineering system with large-scale state space chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. machine learning multiagent systems performance development stochastic learning and adaptive control artificial intelligence agent-based economics 1 introduction deep reinforcement learning (drl) has the potential of finding novel solutions to complex problems, as outlined by recent progress in diverse areas such as control of gene regulatory networks [9], adaptive video acceleration [11], and management of computational resources [7]. resource management, the process by which we map computational resources to the tasks and jobs (programs) that require them, in particular, is an area in which recent learning approaches have demonstrated superior performance over classical algorithms and optimization techniques. still, we see that, in recent work, each approach defines their own mdp formulations, with different design decisions. thus, the literature lacks an analysis of the impact of certain decisions on agent performance. in this paper, we investigate what happens to agent performance as we modify an mdp, observing the impacts when we change the state representation, the transition function, and when we shape the reward signal, performing an empirical investigation using open-source software from the deep learning and reinforcement learning (rl) communities. our main contribution is the formulation and analysis of a set of mdps derived from one in the literature [7] that allows agents to learn faster, and to perform transfer learning with various function approximation methods. we also show that doing so does not degrade performance in the task. the rest of this paper is organized as follows: in sect. 2, we describe the papers that influenced this one, together with other deep rl work for resource management. in sect. 3, we describe the theoretical background with an ongoing example applied to a resource management problem. in sect. 4, we describe our methods and the proposed extensions to a base mdp. in sect. 5, we describe our experimental framework, along with the experiments designed to evaluate our mdps. in sect. 6 we present our concluding remarks and a brief discussion of consequences of the work described here. 2 related work in recent years, interest in deep reinforcement learning (rl) applied to executing the scheduling of computing jobs was probably inspired by the deep resource management (deeprm) [7] agent and environment. deeprm presented an approach of using policy gradients to schedule jobs based on cpu and memory requirements. deeprm’s approach uses images to represent jobs, and a window of jobs from which it can choose which job to schedule next. it was shown that deeprm can learn to schedule based on different metrics. domeniconi et al. [3] proposed cush, a system that built on deeprm to schedule for cpus and gpus, but proposed a hierarchical agent by introducing an additional convolutional neural network (cnn) that chooses which job is going to be scheduled next, and then uses a policy network to choose the scheduling policy to use with the previously selected job. it is important to highlight a major difference between deeprm and cush: whereas deeprm learns the scheduling policy itself, cush is essentially a classifier, which chooses between two existing policies. this means that, even without training, cush’s behavior is more stable than that of deeprm, since deeprm-style schedulers might get stuck in local minima, as reported by de freitas cunha and chaimowicz [2], who investigated the behavior of deeprm-style agents when trained with state-of-the-art rl algorithms such as advantage actor-critic (a2c) and proximal policy optimization (ppo), and proposed an openai gym environment for easier evaluation of rl agents for job scheduling. another agent that has been proposed recently, and that learns the scheduling policy itself, is rlscheduler [16]. rlscheduler is a ppo-based agent with a fully convolutional neural network for scoring jobs in a fixed window of size 128. the major innovation in rlscheduler is in the training setting, in which the authors combine synthetic workload traces with real workload traces to present the agent with ever more difficult settings, similar to learning a curriculum of tasks. similarly to cush, other agents that use a classification approach have been proposed. a recent one is the deep reinforcement agent for scheduling in hpc (dras) [4], which classifies jobs in three categories: ready, reserved, and backfilled. after this classification step, the cluster scheduler takes the output of this classification and allocates jobs accordingly (for example, by reserving slots in the future for reserved jobs, scheduling immediately ready jobs, and finding “holes” in the schedule for backfilled jobs). dras uses a five-layer cnn that works in two levels, with the first level selecting jobs for immediate and reserved execution, and the second layer for backfilled execution. none of the papers mentioned above discuss the impact of their design decisions, resorting to only comparing their results with existing algorithms. in this paper, we aim to analyze how decisions in mdp design impact deeprm-style algorithms, and how they impact both computational performance and scheduling performance. 3 background in this section, we describe the background needed to understand the techniques and methodology presented in this paper. to help in understanding, we will use a resource management problem as running example throughout this section. 3.1 batch job scheduling the primary goal of a job scheduler is to manage the job queue and coordinate execution of jobs in high performance computing (hpc) clusters, while matching jobs to resources in an efficient way. in a discrete time setting, at each time step, zero or more jobs may arrive in the queue for processing, and the scheduler’s job is to allocate jobs to resources while satisfying their resource requirements. the job scheduler guarantees jobs execute when requested resources are available, and usually guarantee there won’t be oversubscription of resourcesfootnote 1. given this primary goal, secondary goals vary between schedulers and hpc facilities, depending on whether the hosting institution prefers to satisfy the needs of individuals submitting jobs, or the whole group of users [5]. when optimization of response time is a subgoal, it is usually modeled as the minimization of the average response time, with response time used as a synonym to turnaround time: the difference between the time a job was submitted to the time it completed execution. a metric commonly used to evaluate this is the slowdown of a job, which, for job j is defined as $$\begin{aligned} \mathrm {slowdown}(j)=\frac{(t_f(j)-t_s(j))}{t_e(j)} = \frac{t_w(j)+t_e(j)}{t_e(j)} = \frac{1}{t_e(j)}\left( \sum _{i=1}^{t_w(j)}1+\sum _{i=1}^{t_e(j)}1\right) \text {,} \end{aligned}$$ (1) where \(t_s(j)\) is the time job j was submitted, \(t_e(j)\) is the time it took to execute job j, and \(t_f(j)\) is the finish time of job j. the equality in the middle holds because the wait time, \(t_w\), of a job j is defined as \(t_w(j)=t_f(j)-(t_e(j)\,+\,t_s(j))\). fig. 1. a possible schedule when three jobs arrive in a scheduling system at discrete time step 1 and no more jobs are submitted to the system at least until time step 7, the last one shown in the figure. full size image consider the case of three batch jobs, \(j_1=\) , \(j_2=\) , and \(j_3=\) , submitted to a scheduling system with two processors, and that the three jobs were submitted “between” time step 0 and 1, such that, when transitioning from the first time step to the second, now there are three jobs waiting. also consider that, for these jobs, the generated schedule is the one displayed in fig. 1. as shown in the figure, the jobs execute for two, three and four time steps respectively, and all of them use a single processor. the reader should observe that different schedules can yield substantially different values of (average) slowdown. for example, the schedule shown in fig. 1 has an average slowdown equal to , whereas, if we swapped \(j_3\) with \(j_1\), and started \(j_1\) soon after \(j_2\) finished, the slowdown would be , a \(\approx 29\%\) increase. therefore, a scheduler should choose job sequences wisely, otherwise its performance can be degraded. in this paper, we will focus our discussion on what happens when an rl system tries to minimize the average slowdown, but our conclusions are general and apply to other metrics and problems as well. 3.2 deep reinforcement learning and job scheduling in a reinforcement learning (rl) problem, an agent interacts with an unknown environment in which it attempts to optimize a reward signal by sequentially observing the environment’s state and taking actions according to its perception. for each action, the agent receives a reward. thus, in the end, we want to find the sequence of actions that maximizes the total reward, as we will detail in the next paragraphs. rl formalizes the problem as a markov decision process (mdp) represented by a tuple \(\mathcal {m} = \langle \mathcal {s}, \mathcal {a}, \mathcal {r}, \mathcal {t}, \rho , \gamma \rangle \)footnote 2. at each discrete time step t the agent is in state \(s_t \in \mathcal {s}\). from \(s_t\), the agent takes an action \(a_t \in \mathcal {a}\), receives reward \(r_{t+1} \in \mathcal {r}\) and ends up in state \(s_{t+1} \in \mathcal {s}\). therefore, when we assume the first time step is 0, the interaction between agent and environment create a sequence \(s_0, a_0, r_1, s_1, a_1, r_2, \ldots \) of states, actions and rewards. to a specific sequence \(s_0, a_0, r_1, s_1, a_1, r_2, \ldots \) of states, actions, and rewards we give the name of trajectory, and will denote such sequences by \(\tau \). the transition from state \(s_t\) to \(s_{t+1}\) follows the probability distribution defined by \(\mathcal {t} : \mathcal {s} \times \mathcal {a} \rightarrow \mathcal {s}\) or, in an equivalent way, \(\mathcal {t}\) gives the probability of reaching any new state \(s'\) when taking action a when in state s: \(p(s'|s, a) = p(s_{t+1}=s' | s_t=s, a_t=a)\). \(\rho \) is a distribution of initial states, and \(\gamma \) is a parameter \(0 \le \gamma \le 1\), called the discount rate. the discount rate models the present value of future rewards. for example, a reward received k steps in the future is worth only \(\gamma ^k\) now. this discount factor is added due to the uncertainty in receiving rewards and is useful for modeling stochastic environments. in such cases, there is no guarantee an anticipated reward will actually be received and the discount rate models this uncertainty. to map our presentation of rl into our problem of job scheduling, we consider (the only possible initial state is the empty cluster), with the first state consisting of the empty cluster, with no jobs in the system, and \(a_0 = \emptyset \), since there is no job to schedule. recall our discussion about classifying jobs to be processed by different policies, versus choosing the next job to enter the system. in this paper, we are modeling an mdp in which the next job is chosen by the agent, so the agent is learning a scheduling policy. in our example, one can obtain a reward function by using the sequential version of slowdown, shown in the rightmost equality of (1), such that the reward at each time step is given by the sum of the current slowdown for all jobs in the system: . when the reward function is such that it computes the online version of slowdown for all jobs in the system, if \(a_1=\emptyset \), . moreover, if jobs \(j_1\), \(j_2\), and \(j_3\) are chosen in sequence, the next state, shown in fig. 1, will be given by sequentially applying the transition function \(\mathcal {t}\) as . if the episode finished immediately after the state shown in fig. 1, the trajectory \(\tau _1\) would be given by footnote 3. the reward signal encodes all of the agent’s goals and purposes, and the agent’s sole objective is to find a policy \(\pi _\theta \) parameterized by \(\theta \) that maximizes the expected return $$\begin{aligned} g(\tau ) = r_{1} + \gamma r_{2} + \cdots + \gamma ^{t-1} r_{t} = \sum _{t=0}^{t-1}\gamma ^{t}r(s_t, a_t\sim \pi _\theta (s_t))\text {,} \end{aligned}$$ (2) which is the sum of discounted rewards encountered by the agent. when t is unbounded, \(\gamma < 1\). otherwise, eq. (2) would diverge. in our example, a deterministic policy that always chose the smallest job first would yield , while a stochastic policy would assign a probability to each job, and either choose the one with highest probability or sample from the jobs according to that distribution. in practice, when neural networks are used for approximation, the last layer of the neural network is usually a softmax, so that each action gets a number that can be interpreted as a probability. as mentioned before, in the example in this section, rewards are based on the negative online slowdown and, therefore, returns will also depend on the slowdown. the policy \(\pi \) is a mapping from states and actions to a probability of taking an action a when in state s, and the parameters \(\theta \) relate to the approximation method used by the policyfootnote 4. popular function approximators include linear combinations of features [6] and neural networks [13, 15]. 3.3 policy gradients in this section we present the main optimization method we use to find policies: policy gradients. as implied by the name, we compute gradients of policy approximations, and use them to find better parameters for those functions. formally, we generalize policies to define distributions over trajectories with $$\begin{aligned} \phi _\theta (\tau ) = \rho (s_0)\prod _t\pi _\theta (a_t|s_t) \underbrace{\mathcal {t}(s_{t+1}|s_t, a_t)}_{\text {environment}}\text {,} \end{aligned}$$ (3) in which \(\pi _\theta \) is being optimized by the agent, and \(\rho \) and \(\mathcal {t}\) are provided by the environment. what (3) says is that we can assign probabilities to any trajectory, since we know the distribution of initial states \(\rho \), and we know that the policy will assign probabilities to actions given states, and that, when such actions are taken, the environment will sample a new state for the agent. when we do so, we can define an optimization objective to find the optimal set of parameters $$\begin{aligned} \theta ^* = \arg \max _\theta j(\theta ) = \arg \max _\theta \int _\tau g(\tau )\phi _\theta (\tau )d\tau \text {,} \end{aligned}$$ (4) where \(j(\theta )\) is the performance measure given by the expected return of a trajectory, which can be approximated by a monte carlo estimate footnote 5. if we construct \(\widehat{j(\theta )}\) such that it is differentiable, we can approximate \(\theta ^*\) by gradient ascent in \(\theta \), such that \(\theta _{j+1}\leftarrow \theta _j+\alpha \nabla \widehat{j(\theta )}\), with \(\alpha > 0\), yielding $$\begin{aligned} \nabla _\theta j(\theta ) \approx \frac{1}{n}\sum _i g(\tau _i)\nabla _\theta \log \phi _\theta (\tau _i);\tau _i\sim \pi _\theta \text {.} \end{aligned}$$ (5) by expanding (2) by one time-step, we get the update \(g(\tau )=r_{1} + \gamma g(\tau _1)\) or, more generally, \(g(\tau _t)=r_{t+1} + \gamma g(\tau _{t+1})\), where \(\tau _i\) indicates the trajectory \(\tau \) starting from offset i. this update is usually written as \(g_t=r_{t+1}+\gamma v_\pi (s_{t+1})\), where \(g_t\) is shorthand notation for \(g(\tau _t)\), and \(v_\pi (s_{t+1})\) is the return when starting at state \(s_{t+1}\) and following policy \(\pi \) (which generated trajectory \(\tau \)). another function related to \(v_\pi (s_t)\) is \(q_\pi (s_t, a_t)\), which gives the return when starting at state \(s_t\) and taking action \(a_t\) then following policy \(\pi \). with these two functions, we can define a third one, which gives the relative advantage of taking action \(a_t\) when in state \(s_t\), defined as \(a_\pi (s_t, a_t) = q_\pi (s_t, a_t)-v_\pi (s_t)\), and called the advantage function. as with the policy, \(q_\pi \), \(v_\pi \), and \(a_\pi \) can also be approximated and, thus, learned. when such an approximation is used, the update (5) becomes $$\begin{aligned} \nabla _\theta j(\theta ) \approx \frac{1}{n}\sum _i \nabla _\theta \log \pi _\theta (a_t|s_t)\hat{a}_\pi (s_t, a_t); s_t, a_t\sim \pi _\theta \text {,} \end{aligned}$$ (6) where \(\hat{a}_\pi \) is an approximation of \(a_\pi \), and which can be further split into two estimators as \(\hat{a}_\pi =\hat{q}_\pi -\hat{v}_\pi \), with the arguments \(s_t\) and \(a_t\) dropped for better readability. in this setting, the \(\pi _\theta \) approximator is called an actor, and the \(\hat{a}_\pi \) approximator is called a critic. in the literature, we find techniques that regularize updates [8, 12], but as presented, eq. (6) is sufficient for understanding of the techniques discussed in this paper. 4 methodology although the discussion in the previous section is helpful for conceptualizing the problem we are interested in, it is not enough to help us implement a solution, since it does not specify when the agent is invoked for learning, how a state is actually represented, nor how rewards are computed for each action. in this section, we will detail our design decisions, and will elaborate on what changes are required to assess the impact of said decisions in rl performance. we begin by describing the base mdp, and then we will describe incremental changes that can be made to the environment so that it may become faster to compute, and easier to learn, leading to faster convergence. we implemented the base mdp and each incremental change discussed in this section. then, we evaluated all implementations, observing both convergence performance and final agent performance in the task of scheduling hpc jobs. 4.1 the base, image-like mdp we begin by following the design of deeprm [7], summarized here, and exemplified in fig. 2. we start by representing states as images whose height corresponds to a look to a time horizon of h time-steps “into” the future, and the width comprising: the number of processors in the system and their occupancy state, a window of configurable size w (in the figure, \(w=2\)) containing the first w jobs in the wait queue times the number of processors in the system, and a column vector indicating jobs in a “backlog”footnote 6. if there are more jobs in the system that can fit the window and the backlog, they are omitted from the state representationfootnote 7. fig. 2. dense state representation with images representing state. in the figure, there is one job in execution (with two processors for the next two time steps), three waiting jobs in total, two of them within window \(w=2\), one using one cpu for at least five time steps, and another using three cpus for four time steps. details for the third job, in the backlog are omitted. full size image for the action, the agent can either choose to schedule a job from one of the w slots, or it can refuse to schedule a job, totalling three possible actions in the example of fig. 2. regarding when actions are taken, the mdp was built in such a way that agents see every simulation time step and “intermediate” time-steps as well: when a job is scheduled, there is a state change in the mdp, with the job moving to the in-use processors, and the queue being re-organized so that all slots in window w are filled. in the base mdp, whenever a job is scheduled, the agent receives a reward of zero. in all other cases, the reward is given by the negative online slowdown (1): \(\mathcal {r}=-\sum _{j\in \mathcal {j}}\frac{1}{t_e(j)}\). for a detailed description of the environment, we direct the reader to mao et al. [7], who first introduced it. 4.2 compact state representation the first realization we had was that the state representation in the base mdp is wasteful, in the sense that one can reduce the size of the state without losing information. particularly when working with larger clusters, or with a larger number of job slots, it may be the case that full trajectories take too much space in memory, reducing the computational performance of learning agents. due to that, and based on a set of features found in the literature for machine learning with hpc jobs [1], we devised a set of features that can represent states in a compact way. in our new state representation, jobs in the queue are represented by the features shown in table 1, where “work” is computed by multiplying the number of processors a job requires by the time it is expected to run, and cluster features are a pair that indicates the number of processors in use, and the number of free processors. the features related to the cluster state still use a time horizon h but instead of using a matrix, we used a pair of integers representing how many processors are in use, and how many processors are free in a given time-step. as an example, assuming the job in the cluster was submitted at time 1, the job in slot 1 was submitted at time 2, and the job in slot 2 was submitted in time 3, the state shown in fig. 2 can be fully described by the concatenation of vectors with cluster state \(\langle (2, 1), (2, 1), (0, 3), (0, 3), (0, 3) \rangle \), jobs in window w \(\langle (1, 5, 1, 0, 0, 1, 6, 0), (2, 4, 3, 1, 5, 1, 4, 0) \rangle \) and backlog \(\langle 1 \rangle \)footnote 8. the features in the jobs slots are presented in the same order as the ones shown in table 1. table 1. job features in a compact state representation.full size table a side-effect of using this new compact state representation is that, when h and w are fixed between different cluster configurations, learned features are directly transferable between clusters even when using function approximation methods that depend on a fixed number of features. 4.3 sparse state transitions another deficiency we’ve identified in the base mdp is that the agent sees all time-steps in the simulation, but this causes the agent to have to take an action even when there is no good action to take. consider, for example, the case in which all resources are in use (there are no free resources). in cases such as this, any action the agent takes will lead to the same outcome: increasing the simulation clock, receiving negative rewards related to the slowdown of the jobs, and having no new jobs scheduled. this will be repeated for all time steps between the start of the last job that exhausted resources until the finish of the first job that frees them, causing non-negative rewards to be more sparse, making the reinforcement signal noisier and, therefore, harder to learn. the opposite is also true: if there are no jobs waiting to be scheduled, no matter what the agent chooses, the outcome will be the same: no jobs will be scheduled. due to that, we updated the environment to only call the agent and, therefore, to only add states, actions and rewards to a trajectory, when it was possible for the agent to take an action that could result in a job being scheduled. in short, we change the transition function \(\mathcal {t}(s_{t+1}\mid s_t, a_t)\) so that all state transitions from \(s_t\) to \(s_{t+1}\) will always have at least one job that may be scheduled by the agent in state \(s_{t+1}\). we did not change the initial state, though, so still holds. this essentially turns the mdp into a semi-mdp [14]. to make our formulation compatible with a semi-mdp, we extend the reward function to return zero in all intermediate states after successfully scheduling a job. 4.4 reducing the noise of the reward signal based on the idea of only showing the agent what it can use to learn and act, we noticed that the reward signal could be further improved by, instead of computing the online slowdown of all jobs in the system \(\mathcal {j}\), considering only the jobs that are in the waiting queue, and within the job slots window w: the jobs that can be directly influenced by the agent’s actions. therefore, we defined the set \(\mathcal {w}\) that contains the subset of jobs from \(\mathcal {j}\) that are within the window w, and the reward function became \(\mathcal {r} = -\sum _{j\in \mathcal {w}}\frac{1}{t_e(j_i)}\) when the action taken doesn’t schedule a job, and 0 otherwise. we evaluate the impact of the various mdps on agent performance by performing two sets of experiments, one in which we observe the impact of the changes proposed in sects. 4.2 through 4.4 (called compact, sparse, and reduced respectively in the experiments) as opposed to the dense mdp, and another in which we observe the impact of using an event-based simulation and bounded rewards both in dense and compact mdps. 5 experiments in order to evaluate our methodology, we used open-source libraries to implement both our agents and environment, with stable-baselines3 [10] providing the ppo agent and its training loop, and sched-rl-gym [2] providing the simulator and environment implementation. table 2. list of hyper-parameters used when training agents.full size table all our experiments consisted of training a ppo agent in the different formulations of the previous section. we also fixed the neural network architecture used for function approximation, consisting of a two-layer neural network with 64 units in each layer, and with parameter sharing between policy and value networks. the fixed number of units implies the image-like representation will use more parameters, as it contains more data than the compact representation. the hyper-parameters used for training the agent are summarized in table 2. we performed no hyper-parameter optimization, and used values found in the literature when training the image-like agent. for a full description of ppo, we direct the reader to schulman et al. [12]. we also maintained the environment specification fixed for all agent evaluations and used \(w=10\) job slots, with simulations of length \(t=100\) time-steps and time horizon \(h\in \{20, 60\}\). these two horizon values enable us to contrast cases in which agents can see when jobs will complete, or not. regarding the workload, we used a workload generator from the literature [2, 7], which submitted a new job with \(30\%\) chance on each time step. of these, a job had \(80\%\) chance of being a “small” job, and “large” otherwise. the number of processors \(n_p\) was chosen in the set {10, 32, 64}, while the maximum job length (duration) d varied from {15, 33, 48} and the size of the largest job (number of processors) \(j_s\) came from the set {10, 32, 64}. in the workload generator, the length of small jobs was sampled uniformly from , and the length of large jobs was sampled uniformly from . the number of processors used by any job was sampled from . all agents were trained for three million time-steps as perceived by the agent. this means that all agents will see the same number of states, and will take the same number of actions, but the number of time steps in the underlying simulation will vary, due to the event-based case becoming a semi-mdp. we evaluated agents with a thousand independent trials, reporting average values. fig. 3. learning curves for various scenarios with \(h=20\) contrasting learning using a compact representation with learning with an image-like representation. curves are an average of six agents, with shaded areas representing one standard deviation, and show a moving average of total rewards received by the agents during training. full size image in fig. 3 we show a sampling of learning curves comparing the learning performance of agents that were trained using the image-like representation and the compact representation with rewards computed from all jobs. the compact representation converges faster than the image-like representation, probably due to its smaller number of parameters. we also notice that although convergence is faster, the compact representation is not necessarily better (fig. 3c, 3d). there doesn’t seem to be a general rule, but we noticed that when jobs are shorter (the d parameter is smaller), the compact representation dominates (fig. 3a, 3b). when d increases and most jobs use few processors (\(j_s \ll n_p\)), the compact representation tends to have comparable performance with the image-like representation (fig. 3c), whereas when jobs use many processors and have a longer duration, agents using the image-like representation learn the environment better (fig. 3d). for this set of experiments, the size of the time horizon (h) doesn’t impact the learning performance, as curves obtained with \(h=60\) (not shown) are indistinguishable from visual inspection to the ones obtained with \(h=20\). when evaluating agents, we performed t-tests to check whether there was a difference in agent performance when using these different h values. in other words, the null hypothesis was that performance was equal, and the alternative hypothesis was that agent performance varied. in this setting, the null hypothesis was rejected only \(36.6\%\) of the time when considering p-values \(\le 1\%\). fig. 4. average slowdown for the various scenarios considered. each bar represents a different instantiation of the (semi-)mdps. average slowdowns were computed by averaging the slowdown of a thousand independent trials for each agent in each scenario. all agents were evaluated with same workload and random seed. in the legend, image-like corresponds to the base mdp, compact to the compact representation, sparse to the sparse state transitions, and reduced to the reduced set of jobs to compute rewards. full size image table 3. key to the scenarios presented in fig. 4. procs. refers to the number of processors in the cluster, max length refers to the maximum job length, and max size refers to the maximum number of processors used by jobs.full size table when evaluating agents after one million iterations, scheduling performance was similar between agents when the maximum number of processors used by jobs was smaller (which implies less parallelism). given job submission rates in all environments was the same, clusters were less busy in these situations: as long as jobs are scheduled, there shouldn’t be significant differences in average slowdown, due to smaller queues. in fig. 4, with key to scenarions shown in table 3, we show average slowdown of the agents for the scenarios in which there was some variability in performance between agents. from the figure, we see that, apart from scenarios 2 and 6, agent performance in the “compact + sparse + reduced” mdp is not worse than that of the image-like mdp. of these two, only the difference for scenario 6 is statistically significant, with p-value \(\le 5\%\) when performing a t-test with alternative hypothesis of different distributions. for the cases where “compact + sparse + reduced” agents are better, the results are statistically significant (p-value \(\le 5\%\)) in scenarios 3, 4, 5, 7, and 9. scenarios 2 and 6 are interesting, since they were configured to have shorter jobs of at most 15 time-steps, with scenario 1 having 10 processors, and scenario 6 having 64, both with jobs with the potential of using all cluster resources. fig. 5. time needed to train agents for three million iterations. the shaded area represents one standard deviation. increasing the time horizon increases the training time of compact agents by a constant factor, while it adds a linear factor to the training times of agents that use an image-like representation. full size image in fig. 5 we contrast the training times for the various agents. as can be seen, training times for agents based on the image-like mdp are highly variable, due to the fact that different mdp configurations result in different sizes of state representations, which impacts training performance. as an example, the image-like agent requires 301068, 1089548, and 1821708 parameters for the scenarios with 10, 38, and 64 processors, while all compact agents require a fixed number of parameters: 24332. times were measured in a linux 5.10.42 desktop with an nvidia gtx 1070 gpu and an i7–8700k processor using the performance cpu frequency-scaling governor. the compact mdps proposed in this paper all have the characteristic of having a state representation with a fixed size, which allows for transfer of learned weights between mdps. here, we consider transfer the ability to change cluster configuration without the need for retraining an agent from scratch, which is simply not possible when using the image-like representation. in fig. 6, for example, we show the performance of an agent trained in the bounded reward, event-based, compact mdp with 64 processors and with jobs of length 33 (the best agent in fig. 4, corresponding to scenario 9) evaluated in a compact environment without event-based updates. with this same agent, we were able to evaluate its performance in all different scenarios, without the need for retraining. we see that, for the most part, slowdown is kept low, and not only that: this agent outperformed other agents in 80% of scenarios (differences are statistically significant, with p-value \(\le 1\%\), except for scenario 9, since this is the same agent, and scenario 5, where the test has low power to reject the null hypothesis). this highlights the advantage of using a representation that allows for easy transfer between agents, enabling good performance in a variety of cluster settings. fig. 6. bar chart contrasting the performance of a transferred agent to agents trained specifically in their environments. full size image 6 conclusion in this paper, we’ve filled a gap in the literature by analyzing the effects of different mdp design decisions on the behavior of rl agents. in particular, we experimented with resource management agents for job scheduling in computing clusters, discussing cases in which a compact representation outperforms a dense one, and vice-versa. we proposed a new state representation, a transition function, and a reward function for an mdp studied in the literature, and we saw that these environments support transferring agents between different cluster settings, while also keeping agent memory consumption constant, and processing requirements stable. we also saw that these compact representations are no worse than image-like ones, and, thus, might be preferable when constant memory usage is a requirement. moreover, our results indicate that transferred agents may outperform specialized agents in 80% of the tested scenarios without the need for retraining. notes 1.some schedulers allow for oversubscription of memory resources in their default configuration, inspired by the fact that jobs don’t use peak memory during their complete lifetimes. 2.some authors leave the \(\gamma \) component out of the definition of the mdp. leaving it in the definition yields a more general formulation, since it allows one to model continuous (non-ending) learning settings. 3.the value shown for \(r_2\) might contradict the previous discussion, but the mdp is set in a way that, when jobs are scheduled successfully, \(r_{t+1}=0\). 4.in our example, for each job \(j_i\), in time step 1, \(\pi \) would give the probabilities of choosing each job given an empty cluster: , , and such that, by total probability, . 5.normalization is needed to approximate the average value of \(\widehat{j(\theta )}\). otherwise, \(\widehat{j(\theta )}\rightarrow \infty \) as \(n\rightarrow \infty \). 6.jobs in the wait queue that the agent cannot choose to schedule. 7.truncating the list of jobs violates the markov property, since once it overflows, the agent cannot know how many jobs are in the system. 8.parentheses group elements. in the first vector, there are five parenthesized pairs to indicate the time horizon of 5, and two parenthesized elements to represent job slows in window w. references cunha, r.l., rodrigues, e.r., tizzei, l.p., netto, m.a.s.: job placement advisor based on turnaround predictions for hpc hybrid clouds. future gener. comput. syst. 67, 35–46 (2017). issn 0167–739x article  google scholar  de freitas cunha, r.l., chaimowicz, l.: towards a common environment for learning scheduling algorithms. in: 2020 28th international symposium on modeling, analysis, and simulation of computer and telecommunication systems (mascots), pp. 1–8 (2020) google scholar  domeniconi, g., lee, e.k., morari, a.: cush: cognitive scheduler for heterogeneous high performance computing system. in: proceedings of drl4kdd 19: workshop on deep reinforcement learning for knowledge discovery (drl4kdd), vol. 12 (2019) google scholar  fan, y., lan, z., childers, t., rich, p., allcock, w., papka, m.e.: deep reinforcement agent for scheduling in hpc. arxiv preprint arxiv:2102.06243 (2021) feitelson, d.g., rudolph, l.: toward convergence in job schedulers for parallel supercomputers. in: feitelson, d.g., rudolph, l. (eds.) jsspp 1996. lncs, vol. 1162, pp. 1–26. springer, heidelberg (1996). https://doi.org/10.1007/bfb0022284 chapter  math  google scholar  liang, y., machado, m.c., talvitie, e., bowling, m.: state of the art control of atari games using shallow reinforcement learning. in aamas (2016) google scholar  mao, h., alizadeh, m., menache, i., kandula, s.: resource management with deep reinforcement learning. in: proceedings of the 15th acm workshop on hot topics in networks, pp. 50–56 (2016) google scholar  mnih, v., et al.: asynchronous methods for deep reinforcement learning. in: international conference on machine learning, pp. 1928–1937 (2016) google scholar  nishida, c.e.h., costa, a.h.r., da costa bianchi, r.a.: control of gene regulatory networks basin of attractions with batch reinforcement learning. in: 2018 7th brazilian conference on intelligent systems (bracis), pp. 127–132 (2018) google scholar  raffin, a., hill, a., ernestus, m., gleave, a., kanervisto, a., dormann, n.: stable baselines3 (2019). https://github.com/dlr-rm/stable-baselines3 ramos, w., silva, m., araujo, e., marcolino, l.s., nascimento, e.: straight to the point: fast-forwarding videos via reinforcement learning using textual data. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 10931–10940 (2020) google scholar  schulman, j., wolski, f., dhariwal, p., radford, a., klimov, o.: proximal policy optimization algorithms. arxiv preprint arxiv:1707.06347 (2017) silver, d., et al.: a general reinforcement learning algorithm that masters chess, shogi, and go through self-play. science 362(6419), 1140–1144 (2018) article  mathscinet  google scholar  sutton, r.s., precup, d., singh, s.: between mdps and semi-mdps: a framework for temporal abstraction in reinforcement learning. artif. intell. 112(1), 181–211 (1999). https://doi.org/10.1016/s0004-3702(99)00052-1. issn 0004–3702 article  mathscinet  math  google scholar  tesauro, g.: td-gammon, a self-teaching backgammon program, achieves master-level play. neural comput. 6(2), 215–219 (1994) article  google scholar  zhang, d., dai, d., he, y., bao, f.s., xie, b.: rlscheduler: an automated hpc batch job scheduler using reinforcement learning. in: sc20: international conference for high performance computing, networking, storage and analysis, pp. 1–15. ieee (2020) google scholar  download references author information authors and affiliations programa de pós graduação em ciência da computação, universidade federal de minas gerais (ppgcc-ufmg), belo horizonte, mg, brazil renato luiz de freitas cunha & luiz chaimowicz authors renato luiz de freitas cunhaview author publications search author on:pubmed google scholar luiz chaimowiczview author publications search author on:pubmed google scholar corresponding author correspondence to renato luiz de freitas cunha . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de freitas cunha, r.l., chaimowicz, l. (2021). on the impact of mdp design for reinforcement learning agents in resource management. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_6 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords reinforcement learning resource management markov decision processes publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature exploring score-based ranking fairness in marketplace environments through simulation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper exploring score-based ranking fairness in marketplace environments through simulation conference paper first online: 30 january 2025 pp 276–290 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) exploring score-based ranking fairness in marketplace environments through simulation download book pdf download book epub lucas c. silva  orcid: orcid.org/0000-0003-0718-50559 & daniel lucrédio  orcid: orcid.org/0000-0002-1360-40369  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 372 accesses abstract we are currently witnessing growing concerns regarding fairness in online marketplace environments, particularly in situations involving platform-owned first-party selling, biased recommendation algorithms and trust formation. regulatory measures, such as those imposed by the european commission and the us federal trade commission, underscore the importance of fair practices on these platforms. while machine learning ranking models are widely used in ranking multiple offers from various sellers, they can inadvertently introduce biases, raising regulators’ concerns. despite recent advances in fair ranking algorithms, the long-term impact of fairness in ranking remains understudied. the contributions of this research are threefold: 1) it establishes a simulated online marketplace environment to complement traditional static experiments; 2) it assesses the long-term impact of fairness in utility of ranking algorithms; and 3) it evaluates state-of-the-art fairness techniques in dynamic environments. findings reveal how utility in ranking fairness algorithms can be affected by the application of fairness techniques and how data drift impacts regular and fair ranking algorithms in a long-term scenario. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others fairness in rankings and recommendations: an overview article open access 02 october 2021 fairness-aware machine learning engineering: how far are we? article open access 24 november 2023 lambdafair for fair and effective ranking chapter © 2025 1 introduction online marketplace services for third-party sellers are commonly provided by major e-commerce platforms such as amazon and alibaba. however, concerns about fairness have emerged in recent years, particularly in situations where: (i) the marketplace platform itself engages in first-party selling and increases revenue by employing biased algorithms or by making an unfair use of private seller data [8]; and (ii) the platform’s recommendation algorithms favor sellers that utilize paid services, such as fulfillment or delivery [3]. in response to these concerns, the european commission has taken concrete action by determining that these practices may lead to violations to the antitrust rules of the european union. as a result, regulatory measures have been imposed on amazon’s buy box and prime program to ensure compliance with the established guidelines [19]. machine learning ranking models are widely employed in the construction of systems that rank multiple offers from multiple sellers for the same product. however, these models may, inadvertently or not, introduce biases that favor the first-party seller or clients of other services provided by the platform-owning company, which were considered by regulatory entities as anticompetitive practices [3]. despite recent advancements in fair ranking machine learning algorithms and the detection of bias in ranked outputs [23], the examination of the long-term impact of fairness in ranking algorithms remains relatively understudied, especially when compared to the extensive research conducted on fairness in classification techniques [7]. while efforts have been made to develop fair ranking algorithms and identify biases in rankings, there is a need for further investigation into the sustained effects of these algorithms on different stakeholders, including sellers, consumers, and the overall competitive landscape of online marketplaces. exploring the long-term implications of fairness in ranking algorithms is essential for developing effective strategies to mitigate bias and promote fair practices in the context of online marketplace platforms. amazon’s project nessie [1] illustrates the long-term implications of unfair algorithms. the u.s. federal trade commission (ftc) scrutinized the algorithm’s behavior, expressing concerns about its potential adverse effects on competition. ensuring compliance in an adaptive pricing algorithm is complicated due to the non-deterministic nature of machine learning models [17], which makes it difficult to keep track of all behavior changes in each version, especially in corner cases. the model interacts with potentially millions of products, considers numerous concurrent offers, has to make adjustments over time in response to the dynamic pricing environment, and deals with the continual addition of new products and the long-term impact of price changes on competitors and customers’ behaviors. previous studies have utilized simulations to evaluate long-term fairness in binary decision scenarios, such as loans and college admissions [7]. however, ranking tasks are different from binary decision scenarios, requiring different algorithms, metrics and definitions of fairness [23]. this research aims to fulfill this gap, with the following contributions: (i) to establish a simulated online marketplace environment as a baseline framework for future implementations involving ranked data; (ii) to assess the long-term impact of fairness in utility, providing valuable insights for companies, regulators and other stakeholders; and (iii) to assess how state-of-art fairness in ranking techniques address data distribution changes. the results underscored that current methods and metrics for optimizing fairness in rankings are significantly influenced by changes in distribution. moreover, they highlighted that the impact of the fairness-utility trade-off can vary depending on the distribution of attribute values considered by the underlying ranking algorithm and fairness optimization method. these findings emphasize the need for robust approaches that can adapt to dynamic distributions while balancing fairness and utility effectively. 2 related work this section focuses on studies that evaluate the long-term impact of fairness interventions and examine the sustained effects of fairness-aware algorithms on decision-making processes, highlighting their efficacy and potential drawbacks over extended periods. it also explores research that proposes various ranking fairness techniques, case studies, applications and limitations. in their study, d’amour et al. [7] analyze three illustrative toy scenarios to investigate the dynamics of fairness. the first scenario is a loan domain originally proposed by liu et al. [14], where the focus is on fairness in credit allocation. the second scenario revolves around attention allocation and was introduced by elzain et al. [9] and ensign et al. [10], highlighting the significance of fairness in resource allocation. lastly, the researchers examine a college admission scenario, adopting the problem setup of a two-player game known as the stackelberg game [4], to explore fairness challenges and strategies within the educational context. fig. 1. agent-environment interaction loop of ml-fairness-gym [7] full size image to analyze the implications of fairness within these scenarios, d’amour et al. [7] developed a framework called ml-fairness-gymfootnote 1. this framework utilizes a markov decision process (mdp) paradigm, as illustrated in fig. 1, to facilitate the dynamic interaction between agents and their environment. within this setup, the environment undergoes state transitions in response to the agent’s actions, while the agent, on the other hand, observes the environment’s state to make informed decisions. at each state of the environment, fairness metrics are calculated, providing an understanding of fairness dynamics throughout the simulation. the ml-fairness-gym framework is an extension of openai’s gym framework, a state-of-the-art tool for training reinforcement learning. this inherited capability allows it to assist in the creation of adaptative solutions for decision-making process in pursuit of fairness. the paper underscores a series of instances where discrepancies emerge between one-step analysis and dynamic environmental conditions. for instance, even when ensuring an equitable true positive rate (tpr) between the two groups at every stage, the cumulative tpr aggregation displays inequality. this phenomenon, known as simpson’s paradox, remains imperceptible within a static analysis, but was detected thought simulation. to summarize, the authors emphasize three key points: firstly, simulations can complement empirical data in the experimentation process; secondly, there exists significant potential in using reinforcement learning algorithms for policy optimization through a fairness agent; and finally, the application of simulation techniques can ease the implementation and exploration of online or adversarial learning, as it allows to implement different scenarios, including rare and corner cases. zehlike, yang, and stoyanovich [23] conducted a survey on fairness in ranking techniques. in their work, they analyzed state-of-the-art algorithms, also introducing a comprehensive taxonomy for classifying these algorithms. this taxonomy involves categorizing fairness ranking techniques based on the specific stage within the ranking task at which mitigation strategies are applied. in accordance with this established taxonomy, the stages at which a fairness intervention may take place within a ranking task can be categorized into three distinct phases: the fairness algorithm for ranking can intervene on the initial candidate set before the execution of the ranking function. in the case of the marketplace environment, it would be an intervention in the offer data (e.g. price, delivery time). it can integrate itself within the ranking function, directly affecting the ranking process. in a marketplace environment, it would be an intervention in the process of scoring offers. it can act in the processed ranked set, subsequent to its treatment by the ranking function, refining the fairness attributes of the final rankings. considering a marketplace environment, this intervention would happen after the scoring of the offers, using the score as input and generating fair scores. another taxonomy refers to the type of bias that the data presents [23]. they are: pre-existing bias: a bias is categorized as “pre-existent” when societal or environmental frameworks predispose discrimination against specific members within a certain group. an illustrative instance is the scholastic assessment test (sat) math scores in the context of us college applicants, where discernible variations exist in distribution and averages among racial groups. as highlighted by reeves and halikias [16], the average sat math scores for black and latino individuals are markedly lower compared to those for white and asian counterparts. this disparity is further accentuated by the distribution patterns: scores for black and latino individuals tend to cluster at the lower end, white scores exhibit a more standard normal distribution, and asian scores tend to cluster at the higher end of the spectrum. occasionally, this form of bias may manifest within a marketplace environment, particularly when a company introduces premium services. for instance, such services might give advantages to sellers utilizing them, even if the services were not explicitly designed to function in such a manner. an example of this can be observed in the realm of delivery services, where a particular service provider is consistently better than its competitors, consequently affording an unintended advantage to sellers who subscribe to this superior service. technical bias: a form of bias that comes from technical constraints or deliberate decisions is characterized as “technical bias”. an illustrative case is the presence of positional bias that happens when a marketplace website accentuates a singular offer by positioning it in a more prominent position, relegating others to smaller sections of the screen. this scenario underscores the challenge of mitigating bias within the constraints of platform design. an exemplar solution, as suggested by the european commission [3], involves the strategic highlighting of two comparable offers whenever feasible. however, achieving a solution that genuinely aims for equitable treatment is invariably limited by the inherent limitations imposed by the design of the marketplace environment. emergent bias: occurs when a ranking system is conceived with a distinct agent archetype in focus or when it interfaces with a societal construct that exhibits temporal evolution [23]. within a marketplace ecosystem, the dispersion of offer attributes like price and delivery time can undergo continual transformation due to the incessant influx of new sellers into the marketplace. this dynamic process can cause shifts in the score distributions among different groups, providing a latent source of bias that gradually materializes over time. all these three forms of bias may manifest in a marketplace platform. a robust simulation framework may put fairness algorithms under experimentation, analysing even corner cases and multiple sources and types of bias. in their study [23], which introduced method classification frameworks and examined bias mitigation approaches within score-based ranking tasks, zehlike, yang, and stoyanovich found that none of the investigated methods were explicitly tailored or assessed for the mitigation of emergent bias. as documented in the study, every method addressed pre-existing bias [2, 5, 6, 12, 13, 18, 21, 22]. however, only a single method [21] was specifically designed to address technical bias mitigation as well. yang and stoyanovich [22] introduced a framework providing metrics for quantifying fairness and a toolset to integrate these metrics into an optimizer. the optimization method proposed by the authors was tested in both synthetic and real world datasets, being able to produce equitable outcomes while also optimizing utility. celis, straszak, and vishnoi [6] provided an optimization framework capable of handling multiple sensitive attributes with nested fairness constraints in linear complexity. the algorithm proposed by these authors was subjected to an evaluative assessment employing essential metrics such as discounted cumulative gain (dcg), spearman’s rank correlation coefficient (rho), and the bradley-terry model. yang, gkatzelis, and stoyanovich [21] complemented previous research efforts, which emphasized diversity constraints, by showing other interrelated fairness dimensions, most notably in-group fairness. this fairness concept revolves around the possibility that candidates selected from a particular group might not inherently represent the best fit when evaluated within the group’s context. through an optimization framework, the authors introduced new fairness constraints, made to combine in-group fairness alongside established diversity constraints. to empirically substantiate their framework’s efficacy, an experimental evaluation was undertaken, utilizing various public datasets. the results showed that, as expected, the enforcement of regular diversity constraints caused an utility loss of 28%, with an increase of 3% of loss when including in-group fairness. stoyanovich, yang, and jagadish [18] introduced a framework for enforcing fairness constraints within the domain of score-based ranking in streaming tasks. in streaming environments, the inherent complexity arises from the challenge of not possessing access to the entirety of candidates during the prediction phase. the authors proposed more efficient set selection techniques and new insights on how to circumvent this limitation. their work was tested against both real-world and synthetic datasets. the framework demonstrated success in providing fair outcomes, with an acceptable trade-off in utility. kleinberg and raghavan [13] described implicit bias, which can be classified as a form of pre-existing bias, showing how unconscious prejudice linked to group affiliations can inadvertently steer the decision-making process. the authors proposed a theoretical framework that enables the analysis of the ramifications from implicit bias, showing potential trade-offs and benefits associated with bias mitigation strategies. celis, mehrotra, and vishnoi [5] also study implicit bias, analyzing fairness constraints inspired by the rooney rule. the rooney rule, created by the u.s. national football league (nfl), mandates that all its teams include at least one minority candidate for head coaching positions. when scrutinizing the outcomes of implementing rooney rule-like constraints on curated datasets, this approach was successful in dealing with implicit bias, presenting outcomes congruent with the findings of kleinberg and raghavan [13]. yang, loftus, and stoyanovich [12] presented a causal fairness framework specifically designed to deal with the concept of intersectionality, wherein a candidate is found affiliated with multiple protected groups simultaneously. the framework was tested in a real-world and synthetic datasets, exposed to several structure assumptions, presenting more diverse outcomes than the original datasets ranking. asudeh et al. [2] developed a system aimed at aiding human decision-makers in fine-tuning weights within ranking score functions. empirical validation via tests on real-world datasets has proven the efficacy of the system. moreover, the complexity of this solution, which uses a simple weighted function during a ranking task, is much lower compared to techniques that use optimization or causal fairness. in summary, this section presented state-of-the-art fairness algorithms addressing various biases in ranked inputs. while none of them specifically addresses long-term fairness impacts, research on this aspect exists for classification and resource allocation tasks [4, 7, 9, 10, 14]. identifying this research gap and inspired by recent regulatory discussions on fairness in marketplace environments, we developed a simulation framework. the framework draws inspiration from d’amour et al.’s work [7] and utilizes the algorithm proposed by celis, straszak, and vishnoi [6]. additionally, we incorporated the rooney rule, whose effects were studied by celis, mehrotra and vishnoi [5]. 3 experimental setup the main objective of this section is to present the experimental procedure for designing and evaluating the marketplace simulation, detailing the fundamental mechanics of this kind of environment and the setup for systematically changing affiliation group sizes. this approach aims to understand the effects of bias mitigation strategies within the dynamic context of a marketplace. algorithm 1 simulation algorithm full size image algorithm 1 provides an illustrative depiction of the complete simulation procedure. it iterates a total of \(n \cdot ss\) times, where n denotes the frequency of parameter updates and the generation of new seller pools. here, ss signifies the number of ranking tasks executed with consistent parameters and seller pools, both premium and non-premium. the creation of seller pools, which represent the sellers registered in the marketplace, is segmented into premium and non-premium categories (cp and cnp, respectively). this is made through the function generatepool, which receives the following parameters: pp: this parameter signifies the proportion of premium sellers relative to the total number of sellers. the distinct sizes of the two pools may emulate the evolution of premium services within a business context. ntotal: this variable represents the overall count of sellers subscribed to the marketplace environment. \(p_p\) and \(p_{np}\): these parameters, which refer to the premium pool and non-premium pools, respectively, encompass the mean and standard deviation values for each attribute of the generated offers (e.g., price, delivery time). consequently, within the simulation, it is possible to fine-tune these attributes and optimize parameters for specific groups, conducting trade-off analysis as needed. following the generation of both seller pools, at each iteration of j, the randomselection function comes into play. this function randomly gets a subset from these pools, forming an offer list. each selected seller has the capacity to generate an offer, with individual attributes being determined based on the mean and standard deviation values specified in \(p_p\) and \(p_{np}\). the default behavior of this random generation adheres to the rooney rule, ensuring that there is always at least one offer from each group. this characteristic aligns with the principles of fairness and inclusivity, mirroring real-world practices in which diversity is prioritized. next, the generated offers (co) undergo ranking through a weighted function, which is an approach commonly used for naive ranking algorithms [23]. attribute weights assigned within this process serve to accentuate the significance of particular attributes, facilitating the generation of a ranking score. the most important attributes are generally determined by the machine learning algorithm using the provided data or by a set of business rules created by stakeholders, which prioritize factors such as price and delivery time over other attributes. for this setup, based on observations from real-world data, the following attributes were generated for each offer, and these weights were assigned to them in the regular ranking agent: price (weight=5): the cost associated with the seller’s offerings, determinant for buyers. delivery time (weight=2): the estimated time frame within which the product or service will be delivered, another critical factor influencing purchase decisions. installment ratio (weight=1): the proportion of the payment that can be made in installments, providing flexibility to buyers. seller score (weight=1): the seller’s historical performance and reputation within the marketplace, a trust-building metric for buyers. premium status (weight=1): indicates whether the seller belongs to the premium category or not. premium sellers are meant to have access to exclusive services that may provide advantage in competition. in the simulation, the weight introduces bias explicitly. however, it is worth noting that in most real-world scenarios, bias typically manifests implicitly or emerges from the training data used or as a side effect of stakeholder decisions. the aforementioned attributes have been subject to normalization using a min-max scaler, rescaling their values within the range of 0 to 1. it is important to note that, for consistency, all attributes are configured with a “bigger is better” orientation, even in cases where this contradicts the natural interpretation of the non-normalized features. for instance, attributes like price and delivery time, which are conventionally considered more attractive to consumers when they are smaller, have been configured as “bigger is better”. following the initial ranking procedure, the fairrankoffers function intervenes to refine the fairness of the ranking. this optimization mechanism implemented the methodology proposed by yang and stoyanovich [22], publicly accessible on githubfootnote 2, replicating their approach. the groundtruthrankoffers function receives the same parameters as the rankoffers function and behaves similarly. the key distinction is that while the rankoffers function favors premium sellers, the groundtruthrankoffers function ignores the premium attribute, serving as a reliable oracle for comparing both fair and regular ranking agents against an utility-optimized agent. at the conclusion of each iteration i, the updateparameters function is invoked and changes the distribution of both seller groups poll sizes, represented by the parameters \(p_p\) and \(p_{np}\). the variation size of the sellers distribution may be configured at the beginning of the experiment. the idea of this function is to gradually change the pool size of each group. at the conclusion of each iteration i, the updateparameters function is invoked to alter the distribution of both seller groups’ pool sizes, represented by the parameters \(p_p\) and \(p_{np}\). the magnitude of variation in the sellers’ distribution can be configured at the beginning of the experiment. the purpose of this function is to gradually adjust the pool size of each group. the data is sampled from data distributions calculated with real product offer data using the following method: data distributions of 100 products are obtained using the fitter libraryfootnote 3, which approximates the data distribution of the attributes of each product. then, the features mean price and mean delivery time are aggregated to enrich the metadata. the hdbscan clustering algorithm is used to cluster the data and segregate it into 4 groups, excluding the outliers. a qualitative evaluation is applied to the results, and the distribution of each attribute representing each cluster of products is consolidated using the most common distribution within the clusters and the mean of the distribution parameters. at the beginning of each iteration, the simulation sellers’ feature sampler first chooses which cluster the product belongs to, respecting the original data distribution. it then samples the offers using the evaluated distribution of the chosen cluster. the outliers cluster is also considered, and when chosen, the distribution of each attribute is uniform. the code utilized for the simulation is openly accessible on githubfootnote 4, ensuring full reproducibility of this setup. 4 results and discussion this section presents the results of the simulation outlined in the preceding section. the simulated scenario involved adding more premium sellers over time, removing 3% of the non-premium seller pool and adding them to the premium seller pool. in the simulation, the regular ranking agent is biased towards premium sellers. this bias is represented by a 0.1 weight on the premium status attribute. in real-world scenarios, this bias can appear in an implicit form in the training data or as a complex set of requirements from stakeholders. the ground truth ranking agent doesn’t consider the premium status attribute in the ranking function. this provides an unbiased version that doesn’t have utility metrics degraded by the fairness-utility trade-off. it is worth noting that simply not considering the premium status attribute doesn’t solve any type of bias problem, as this is a simulated scenario with a conveniently simple biased ranking function and a parameterized amount of bias. additionally, the attributes of premium and non-premium sellers have the same distribution, which is not the case in production environments. fig. 2. simulation of gradual growth of premium sellers (color figure online) full size image in fig. 2, the red line with squared points represents the number of offers from premium sellers that won first place in the simulated offer ranking, while the blue dotted line represents the non-premium first-place winners. the x-axis represents the iterations of the simulation. in this experiment, we conducted the simulation for 20,000 iterations. every 1,000 iterations, 30 sellers from the non-premium seller pool move to the premium seller pool. the grey bars in the figure represent the number of premium sellers, shown in darker grey, and non-premium sellers, shown in lighter grey, in each pool, with the total pool size always summing to 1,000. in each graph, we have a type of agent. the regular agent is clearly biased towards premium sellers. at the beginning of the simulation, premium sellers are only 10% of the total pool, yet the number of premium winners was slightly more than 50%, surpassing the number of non-premium winners, which comprised 90% of the total seller pool. the fair agent’s premium winner plot line adheres almost perfectly to the proportion of premium sellers, with slight variations. this indicates that the fairness optimization used in the fair agent algorithm keeps the participation of each group in line with the group size, even in the buybox scenario where only the top-ranked seller is analyzed. in the ground truth agent graph, the number of premium sellers that won in the offer ranking also adheres to their proportion in the total seller pool. this indicates that, in terms of fairness, the fair and the ground truth agents exhibit similar behaviors, especially when considering how much the graph diverges from that of the regular agent. fig. 3. mean price over iterations for each product type full size image figure 3 shows the mean price evolution of the top-ranked offer during simulation iterations for each product type. these product types were derived from clustering real-world data using the hdbscan algorithm, as detailed in sect. 3, where each identified cluster represents a distinct product type. the hdbscan algorithm was chosen because it does not require a fixed number of groups, automatically creates an outlier group, and allows for qualitative analysis of the results using the hierarchical structure [15]. the outlier group was not considered in this analysis. since each product type has a different distribution of the price attribute, the figure shows that the winners chosen by the fair agent consistently have a worse mean price compared to those of the ground truth agent and the regular agent. throughout all iterations, the regular agent consistently demonstrates a worse price mean than the ground truth agent. figure 3 also highlights how the distribution of the attribute can affect the results of the fair agent. product types 2 and 4 have more variation in their fair agent price mean, ranging between 0.4 and 0.6, while in product types 1 and 3 this variation is between 0.5 and 0.6. the variation between the regular and ground truth agents is also more pronounced in product types 2 and 4, with some iterations showing a wider gap between the regular and ground truth price means. fig. 4. critical difference diagram for the rkl measures during simulation full size image figure 4 shows two critical difference diagrams of the normalized discounted kl-divergence (rkl), the metric proposed by yang and stoyanovich [22], with smaller rkl values indicating fairer rankings. for each ranking agent, the mean rkl is calculated over 20 groups, each with 1,000 iterations, to ensure robust results. the agents are then ranked based on their mean rkl values for each group, and the average rank for each agent is computed across all groups. in fig. 4, in the diagram at the left side, the rkl was calculated using a cutoff point of 5 in the evaluated ranking, meaning that only the top 5 positions are considered when calculating the metric. the diagram at the right side uses a cutoff point of 10. we performed a nemenyi post-hoc test for both cutoff point options. the test identifies the sets of agents whose mean rkl ranked differences are not statistically significant. this is represented by a black horizontal bar connecting them: if two algorithms are connected by the bar, their fairness, as measured by the rkl metric, does not differ significantly. the horizontal bar in the right side of fig. 4 (cutoff point value of 10) connects all three agents. this means that there is not a significant statistical difference between them. this can be explained by several factors. the first factor is the changing data distribution. initially, the smallest rkl for both cutoff point values belongs to the regular agent because the premium seller pool starts with fewer members than the non-premium seller pool. as the premium seller ratio in the total pool approaches 50%, the rkl measurements of all agents begin to align with expectations. looking at the whole simulation, the best method for fairness is the fair agent, followed by the ground truth agent with a 0.1 difference in the average ranking, and then the regular agent, with a 0.5 difference in the average ranking. for the cutoff point value of 5, the same behavior is observed, but with a more pronounced difference between the agents. the fair agent and the regular agent show a significant statistical difference in their results, but neither shows a significant difference with the ground truth agent, which divides both. the best cutoff point may depend on the use case and the parameters of the fairness optimizer, but the low sensitivity of the metric can also significantly affect how the metric behaves during the simulation with changing group sizes. when evaluating more members in the ranking, we often include more members of both affiliation groups, making position bias more pronounced. despite the influence of the cutoff point value on metric performance and the expected results shown in both diagrams, using rkl in an environment where there are changes in group affiliations can lead to unexpected outcomes. yang and stoyanovich [22] assert that, compared with other similar metrics, the symmetric behavior of rkl makes it more resilient to distribution changes. however, the lack of sensitivity of the metric can be detrimental to its performance in measuring fairness in such scenarios. 5 concluding remarks while the fair agent has proven effective in maintaining group proportionality at the top of the ranking, it has shown inferior results in average prices compared to the regular agent and the ground truth agent. the fair agent frequently selects offers with worse prices. this disparity in average prices highlights the inherent trade-off between fairness and utility in ranking systems, where optimizing for one objective can lead to a negative impact on another. the fairness-utility trade-off, coupled with the challenge posed by constantly changing group size distributions, has underscored limitations in state-of-the-art methods and metrics for optimizing ranking fairness in the proposed simulated environment. novel approaches borrowed from fairness in classification methods, which are resilient to distribution shifts and capable of detecting unaware protected attributes and mitigating bias, can be adapted and tested for fairness in ranking tasks. promising methods of this kind include quantification methods [11] and adversarial architectures for fairness [20]. notes 1.https://github.com/google/ml-fairness-gym. 2.https://github.com/dataresponsibly/fairrank. 3.https://github.com/cokelaer/fitter. 4.https://github.com/cardosolucas/mkp-fairness-simulation. references federal trade commission, et al. v. amazon.com, inc. case no. 2:23-cv-01495-jhc, united states district court for the western district of washington (2023). https://www.ftc.gov/system/files/ftc_gov/pdf/1910134amazonecommercecomplaintrevisedredactions.pdf. accessed 29 jun 2024 asudeh, a., jagadish, h., stoyanovich, j., das, g.: designing fair ranking schemes. in: proceedings of the 2019 international conference on management of data, pp. 1259–1276 (2019) google scholar  browne, r.: amazon avoids multibillion-dollar fine as it reaches eu antitrust settlement (2023). https://www.cnbc.com/2022/12/20/amazon-reaches-settlement-with-eu-on-antitrust-case.html. accessed 29 june 2024 brückner, m., scheffer, t.: stackelberg games for adversarial prediction problems. in: proceedings of the 17th acm sigkdd international conference on knowledge discovery and data mining, pp. 547–555 (2011) google scholar  celis, l.e., mehrotra, a., vishnoi, n.k.: interventions for ranking in the presence of implicit bias. in: proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 369–380 (2020) google scholar  celis, l.e., straszak, d., vishnoi, n.k.: ranking with fairness constraints. in: 45th international colloquium on automata, languages, and programming (icalp 2018), vol. 107, pp. 28 (2018) google scholar  d’amour, a., srinivasan, h., atwood, j., baljekar, p., sculley, d., halpern, y.: fairness is not static: deeper understanding of long term fairness via simulation studies. in: proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 525–534 (2020) google scholar  dash, a., chakraborty, a., ghosh, s., mukherjee, a., gummadi, k.p.: when the umpire is also a player: bias in private label product recommendations on e-commerce marketplaces. in: proceedings of the 2021 acm conference on fairness, accountability, and transparency, pp. 873–884 (2021) google scholar  elzayn, h., et al.: fair algorithms for learning in allocation problems. in: proceedings of the conference on fairness, accountability, and transparency, pp. 170–179 (2019) google scholar  ensign, d., friedler, s.a., neville, s., scheidegger, c., venkatasubramanian, s.: runaway feedback loops in predictive policing. in: conference on fairness, accountability and transparency, pp. 160–171 (2018) google scholar  fabris, a., esuli, a., moreo, a., sebastiani, f.: measuring fairness under unawareness of sensitive attributes: a quantification-based approach. j. artif. intell. res. 76, 1117–1180 (2023) article  mathscinet  math  google scholar  ke, y., loftus joshua, r., julia, s.: causal intersectionality and fair ranking. in: 2nd symposium on foundations of responsible computing, forc 2021, june 9–11, 2021, virtual conference (lipics), vol. 192, pp. 1–7 (2021) google scholar  kleinberg, j., raghavan, m.: selection problems in the presence of implicit bias. in: 9th innovations in theoretical computer science conference (itcs 2018). schloss-dagstuhl-leibniz zentrum für informatik (2018) google scholar  liu, l.t., dean, s., rolf, e., simchowitz, m., hardt, m.: delayed impact of fair machine learning. in: international conference on machine learning, pp. 3150–3158 (2018) google scholar  mcinnes, l., healy, j.: accelerated hierarchical density based clustering. in: 2017 ieee international conference on data mining workshops (icdmw), pp. 33–42 (2017) google scholar  reeves, r.v., halikias, d.: race gaps in sat scores highlight inequality and hinder upward mobility (2017). https://www.brookings.edu/articles/race-gaps-in-sat-scores-highlight-inequality-and-hinder-upward-mobility. accessed 29 june 2024 riccio, v., jahangirova, g., stocco, a., humbatova, n., weiss, m., tonella, p.: testing machine learning based systems: a systematic mapping. empir. softw. eng. 25, 5193–5254 (2020) article  google scholar  stoyanovich, j., yang, k., jagadish, h.: online set selection with fairness and diversity constraints. in: proceedings of the edbt conference (2018) google scholar  the european commission: antitrust: commission accepts commitments by amazon barring it from using marketplace seller data, and ensuring equal access to buy box and prime. https://ec.europa.eu/commission/presscorner/detail/en/ip_22_7777. accessed 29 june 2024 yang, j., soltan, a.a., yang, y., clifton, d.a.: algorithmic fairness and bias mitigation for clinical machine learning: insights from rapid covid-19 diagnosis by adversarial learning. medrxiv (2022) google scholar  yang, k., gkatzelis, v., stoyanovich, j.: balanced ranking with diversity constraints. in: proceedings of the twenty-eighth international joint conference on artificial intelligence, ijcai, pp. 6035–6042 (2019) google scholar  yang, k., stoyanovich, j.: measuring fairness in ranked outputs. in: proceedings of the 29th international conference on scientific and statistical database management, pp. 1–6 (2017) google scholar  zehlike, m., yang, k., stoyanovich, j.: fairness in ranking, part i: score-based ranking. acm comput. surv. 55(6), 1–36 (2022) math  google scholar  download references acknowledgements we would like to thank americanas s.a. for their technical support. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations federal university of são carlos (ufscar), são carlos-sp, brazil lucas c. silva & daniel lucrédio authors lucas c. silvaview author publications search author on:pubmed google scholar daniel lucrédioview author publications search author on:pubmed google scholar corresponding author correspondence to lucas c. silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, l.c., lucrédio, d. (2025). exploring score-based ranking fairness in marketplace environments through simulation. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_19 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords fairness score-based ranking marketplace buybox simulation publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a topology-inspired approach to agm belief change | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a topology-inspired approach to agm belief change conference paper first online: 31 january 2025 pp 3–17 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a topology-inspired approach to agm belief change download book pdf download book epub marlo souza  orcid: orcid.org/0000-0002-5373-72719  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 220 accesses abstract agm’s belief revision is one of the main paradigms in the study of belief change operations. despite its popularity, it is well recognised that agm’s work relies on a strong idealisation of the agent’s capabilities and on the supporting logic. recent results in the literature have shed light on the definability of agm-like belief change operations in general logics, showing how the properties of a logic can be leveraged to construct these operations. exploring well-known connections between logic, algebra and topology, our work explores a general notion of belief change contraction, which can be connected to well-studied operations in the literature, such as agm rational contractions, partial meet contractions, and multiple contractions, employing the framework of abstract model theory and topological semantics. we show that agm rational contractions can be characterised in our framework, and we can establish connections to different results in the literature. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others on a logico-algebraic approach to agm belief contraction theory article 05 january 2021 belief contraction in the context of the general theory of rational choice chapter © 2016 towards a theory of hyperintensional belief change chapter © 2020 1 introduction belief change is the area that studies how doxastic agents change their minds after acquiring new information. one of the most influential approaches in the literature, namely the agm framework [1], studies rational constraints, or postulates, that characterise rational ways of changing beliefs, given a representation of one’s doxastic state. while the seminal work of agm focuses on belief change for classical logic, recently, several works investigated its applications to a wide range of non-classical logics of interest in artificial intelligence [4, 16]. investigating the definability of agm belief change operations on abstract logics, flouris [5] shows necessary and sufficient conditions for a logic to allow the definition of these operations and that several logics did not satisfy these requirements. flouris employs machinery from abstract logics and topology to prove their results. similarly, investigating model-theoretic approaches to define forms of belief change, souza and wasserman [18, p.13] point out that “different notions of minimality arise as the reflection of topological properties of the model space”, which indicates a deeper connection between agm belief change and the characteristics of topological space defined by the logic. the literature on abstract logic and abstract model theory has long established deep connections between properties of a logic and of the topology of the space of models that defines it [11, 12, 21]. surprisingly, although topological approaches to the study of formal epistemology and epistemic logic exist in the literature [2, 3, 10], to our knowledge, very little work has been dedicated to the study of topological aspects of belief change. in fact, to our knowledge, only the work of özgün [13] explicitly explores topological models to study belief change. topological notions, however, seem to infiltrate model-theoretic approaches to belief change, such as that of peppas [14] and of ribeiro et al. [15] on the generalisation of agm belief change to other logics than those considered by the original authors. in this work, we investigate a general notion of belief change contraction and the definability of agm belief contraction operators based on results from abstract model theory and its connections to topology (theorems 1 and 2, and lemma 5), generalising the results of ribeiro et al. [15] on the definability of agm contraction for non-monotonic logics (proposition 6 and corollary 2). we employ our characterisation to show, in example 1, that propositional intuitionistic logic is not agm-compatible. this work is structured as follows: in sect. 2 we present preliminary notions related to abstract logic and belief change, which will be the base of our work; in sect. 3, we develop the semantic framework employed in this work, investigating the connections between logic, abstract model theory and topology; in sect. 4, we employ the investigated framework to define belief change operations, and study their properties; in sect. 5, we discuss the related work. finally, we present some final considerations on the significance of our results and pathways for future research. 2 preliminaries in this work, we employ the tools from abstract logic and abstract model theory to study classes of belief change operations and their definability in non-classical logics. we will call a logic any pair \(\mathcal {l} = \langle l, cn \rangle \), where l is a non-empty set, called the logical language, and \(cn:2^l\rightarrow 2^l\) is a function called a consequence operator satisfying the following propertiesfootnote 1: inclusion: \(a \subseteq cn(a)\). idempotence: \(cn(a) = cn(cn(a))\). monotonicity: if \(a\subseteq a'\) then \(cn(a) \subseteq cn(a')\). given a logic \(\mathcal {l} = \langle l, cn\rangle \), we will employ the notation \(th_\mathcal {l}\) to describe the set of all theories of \(\mathcal {l}\), i.e. \(th_\mathcal {l}= \{a\subseteq l~|~ cn(a) = a\}\). furthermore, we say a logic is compact if, for any \(a \subseteq l\) and \(\varphi \in l\), it holds that if \(\varphi \in cn(a)\), then there is some finite \(a'\subseteq a\) s.t. \(\varphi \in cn(a')\). given a logic \(\mathcal {l}\) as above, we call a belief change operation any function \(\star : 2^l \times l \rightarrow 2^l\), which maps pairs of sets of sentences and a sentence, called a set of beliefs and a piece of input information, to a set of sentences, the resulting beliefs. agm [1] investigate three basic belief change operations: expansions, contractions and revisions. belief expansion blindly integrates a new piece of information into the agent’s beliefs. belief contraction removes a currently held belief from the agent’s set of beliefs with minimal alterations. finally, belief revision is the operation of integrating new information into an agent’s beliefs while maintaining consistency. among these basic operations, only expansion can be univocally defined. the other two are characterised by a set of postulates, which define a class of suitable change operators representing different rational ways in which an agent can change their beliefs. given a closed set of beliefsfootnote 2, i.e. \(k\subseteq l\) s.t. \(k = cn(k)\), we say a belief change operation \(\dot{-}\) is an agm rational contraction on k if for any \(\varphi ,\psi \in l\), it satisfies: (closure) \(k\dot{-}\varphi = cn(k\dot{-}\varphi )\) (success) if \(\varphi \not \in cn(\emptyset )\) then \(\varphi \notin k \dot{-}\varphi \) (inclusion) \(k\dot{-}\varphi \subseteq k\) (vacuity) if \(\varphi \not \in k\) then \(k\dot{-}\varphi = k\) (recovery) \(k \subseteq cn(k\dot{-}\varphi \cup \{\varphi \})\) (extensionality) if \(cn(\varphi ) = cn(\psi )\) then \(k\dot{-}\varphi = k\dot{-}\psi \) to characterise their rational contractions, agm propose the notion of partial meet belief contraction, an operation that preserves a maximal amount of “safe” information from the agent’s beliefs, i.e., information that cannot be used to derive what the agent has ceased to believe. definition 1 let \(b\subseteq l\) be a set of formulas and \(\varphi \in l\) be a formula of l, the remainder set \(b\bot _\mathcal {l} \varphi \) is the set of sets \(b'\) satisfying: \(b'\subseteq b\) \(\varphi \not \in cn(b')\) \(b'\subset b''\subseteq b\) implies \(\varphi \in cn(b'')\). a partial meet contraction \(\dot{-}\) is an operation for which there is a selection function \(\gamma \), that characterises this operation. by selection function, we mean that the function \(\gamma \) satisfies (i) \(\emptyset \ne \gamma (b\bot _\mathcal {l} \varphi ) \subseteq b\bot _\mathcal {l} \varphi \) if \(b\bot _\mathcal {l} \varphi \ne \emptyset \) and (ii) \(\gamma (b\bot _\mathcal {l} \varphi ) = \{b\}\) otherwise. definition 2 we say a belief base change operator \(\dot{-}\) is a partial meet contraction on a set \(b \subseteq l\) if there is a selection function \(\gamma \), s.t. for any \(\varphi \) $$b\dot{-}\varphi = \bigcap \gamma (b\bot _\mathcal {l} \varphi ).$$ the authors show that for any compact logic closed by classical disjunction and negation, a rational contraction on a closed set k is a partial meet contraction on k and vice versa. this equivalence, however, does not hold in general for any abstract logic [17]. on the other hand, flouris [5] studied the definability of rational contraction operations, i.e. operations satisfying the original agm postulates, in tarskian logics, obtaining sufficient and necessary conditions for such definability. similarly, hansson and wassermann [9] showed that partial meet contractions are definable in any monotonic and compact (not necessarily taskian) logic. in this work, we will focus on agm rational contractions; as such, in the following, we will discuss only flouris’ characterisation. definition 3 let \(\mathcal {l}=\langle l, cn\rangle \) be a logic, a set \(b\subseteq l\) is said to be decomposable in \(\mathcal {l}\), if for any \(\varphi \in l\), with \(cn(\emptyset ) \subset cn(\varphi ) \subseteq cn(b)\), the set $$\varphi ^-(b) = \{b' \subseteq b~| \varphi \not \in cn(b') ~\text{ and }~ cn(b)=cn(b'\cup \{\varphi \})\}$$ is not empty. a logic is said to be decomposable if every \(b\subseteq l\) is decomposable. flouris [5] show that decomposability is a necessary and sufficient condition for the definability of agm contraction operations. proposition 1 (adapted from [5]). let \(\mathcal {l}=\langle l, cn\rangle \) be a logic, \(k\subseteq l\) be a closed set of formulas, which is decomposable in \(\mathcal {l}\), and \(\dot{-}\) be a belief change operation, then \(\dot{-}\) is a rational contraction on k iff for any \(\varphi \in l\) it holds that (i) \(k\dot{-}\varphi = cn(k')\) for some \(k' \in \varphi ^-(k)\), if \(cn(\emptyset ) \subset cn(\varphi ) \subseteq cn(k)\), and (ii) \(k\dot{-}\varphi = k\), otherwise. our aim in this work is to study a general notion of belief change that subsumes different notions, such as partial meet and rational contraction, in a single semantic framework and characterise them. here, we will focus on characterising rational contractions and using our framework to study how the topological properties of the interpretation space of the logic allow the construction of rational contractions. in doing so, we generalise ribeiro et al.’s results on the characterisation of rational contractions for non-compact logics. 3 logics, abstract model theories, and topological spaces the deep connection between logic, algebra, and topology is well-known since, at least, the works of stone [20]. this connection has given rise to fruitful research in areas such as algebraic logic, abstract model theory, among others. here, we will briefly explore some connections between logic and topology by means of an abstract model-theoretic approach, which will inform our results in sect. 4. the reasoning for focusing on a model-theoretic approach for belief change is not only based on the fact that this approach, in our opinion, provides an interesting framework with which we can connect and contrast the fundamental assumptions of our theory with those of related fields, such as epistemic logic, conditional logic and non-monotonic logics, but also a semantic approach helps us abstract from concrete realisations of these operations for particular logics, based on notions such as remainder sets, sets of complements, etc., to a general understanding of agm belief change as a phenomenon. as such, given a logical language l, we will employ the notion of an abstract model theory to prescribe how these formulas can be interpreted. this notion will then be used to formally define an abstract logic a more common terminology to those working on agm-based belief change, and with which we can establish connections between our results and those in the literature. definition 4 let l be a logical language, we call a tuple \(\mathcal {m} = \langle m, l, \vdash \rangle \) an abstract model theory (amt) if m is a non-empty set of models or interpretations; l is a non-empty set of formulas called a logical language; \(\vdash : m \mapsto 2^l\) is a satisfaction function. given a model \(w\in m\) and a formula \(\varphi \in l\), we say w satisfies \(\varphi \), denoted \(w\vdash \varphi \), if \(\varphi \in \vdash (w)\). similarly, for a set \(\vargamma \subseteq l\), \(w\vdash \vargamma \) if \(w\vdash \varphi \) for all \(\varphi \in \vargamma \). given a set of interpretations \(x\subseteq m\) for some abstract model theory, we will often employ the notation th(x) for the set of all formulas satisfied by the interpretations of x also referred to as the ‘theory of x.’ if \(w\in m\), we will denote \(th(\{w\})\) by th(w). we also define, as usual, \(\llbracket \vargamma \rrbracket \) as the set of all interpretations satisfying it. in the following, given a set of interpretations x, we will usually denote the complement of x, i.e. \(m\setminus x\), by \(\overline{x}\). definition 5 let \(\mathcal {l} = \langle l, cn\rangle \) be a logic and \(\mathcal {m} = \langle m, l, \vdash \rangle \) be an abstract model theory. we say the logic \(\mathcal {l}\) is induced by \(\mathcal {m}\) if for any \(\vargamma \subseteq l\), it holds that \(cn(\vargamma ) = \{\psi \in l~|~ \text{ for } \text{ all }~w\in m~\text{ s.t. }~w\vdash \vargamma , w\vdash \psi \}\). we will often denote by \(\mathcal {m} = mod(\mathcal {l})\) the fact that the model theory \(\mathcal {m}\) induces the logic \(\mathcal {l}\). notice that in definition 5, since we establish that \(\varphi \in cn(\vargamma )\) iff \(\llbracket \vargamma \rrbracket \subseteq \llbracket \varphi \rrbracket \), it is easy to see that any logic induced by some amt trivially satisfies inclusion, monotonicity and idempotency, i.e., it is tarskian. lemma 1 let \(\mathcal {m} = \langle m, l, \vdash \rangle \) be an amt, it induces a tarskian logic \(\mathcal {l}\). more yet, given that any logic defines a lattice of its theories, we can use that structure as a domain of interpretation of the formulas. as such, all tarskian logics are induced by some amt. proposition 2 any tarskian logic \(\mathcal {l}\) is induced by some abstract model theory \(\mathcal {m}\) proof let \(\mathcal {l} = \langle l, cn\rangle \), recall that the set of all theories is defined as \(th_\mathcal {l} = \{\vargamma \subseteq l~|~ cn(\vargamma ) = \vargamma \}\). let us define \(\mathcal {m} = \langle th_\mathcal {l}, \vdash \rangle \) s.t. \(\vargamma \vdash \varphi \) iff \(\varphi \in \vargamma \). well, let \(\vargamma \subseteq l\) be a set of \(\mathcal {l}\)-formulas. take \(\psi \in cn(\vargamma )\), then for any \(\vargamma '\in th_\mathcal {l}\), if \(\vargamma '\vdash \vargamma \), then \(\vargamma \subseteq \vargamma '\) and, since \(\mathcal {l}\) is tarskian, \(\vargamma '\vdash \psi \). now, take \(\psi \in l\) s.t. for any \(\vargamma '\in th_\mathcal {l}\) with \(\vargamma '\vdash \vargamma \), it holds that \(\vargamma '\vdash \psi \). since \(\mathcal {l}\) is tarskian, \(\psi \in \vargamma '\). as \(cn(\vargamma )\in th_\mathcal {l}\), then \(\psi \in cn(\vargamma )\), as well. thus \(cn(\vargamma ) = \{\psi \in l~|~ \text{ for } \text{ all }~w\in m~\text{ s.t. }~w\vdash \vargamma ~\text{ implies }~ w\vdash \psi \}.\)    \(\square \) next, we introduce an important concept to our theory, which will be used to characterise agm rational contractions in our framework. definition 6 let \(mod(\mathcal {l}) = \mathcal {m} = \langle m, \vdash \rangle \) be an abstract model theory, we say a set of interpretations \(o \subseteq m\) is a basic open set (b.o.s.) of \(\mathcal {m}\) if \(\llbracket th (o) \rrbracket = o\). we denote by \(bo_{\mathcal {m}}\) the set of all b.o.s.’s of \(\mathcal {m}\). a similar notion to our basic open sets appears in the work of ribeiro [15] as complete interpretation sets and plays a similar role in the semantic machinery we will employ in this work. notice that the b.o.s.’s are exactly those sets which are the interpretation of a theory. lemma 2 let \(\mathcal {m} = \langle m, \vdash \rangle \) be an abstract model theory inducing a logic \(\mathcal {l}\), and \(o \subseteq m\). \(o \in bo_\mathcal {m}\) iff \(o = \llbracket k\rrbracket \) for some \(k\in th_\mathcal {l}\). with the notion of a basic open set, we can introduce a notion of closure in our models, which complements a set of interpretations. definition 7 let \(\mathcal {m} = \langle m, \vdash \rangle \) be an abstract model theory inducing a logic \(\mathcal {l}\), and \(x \subseteq m\) a set of interpretations. we define the closure of x, denoted by cl(x), the biggest set \(o \subseteq \mathcal {l}\), s.t. \(th(o) = th(x)\). it is not difficult to see that cl(x) exists and is unique, and \(cl(x) = \llbracket th(x)\rrbracket \), as \(\mathcal {l}\) is tarskian. more so, cl is, in fact, a closure operator. as stated before, since at least the works of stone [20], it is well-known that logics have a deep connection with topology. this connection is the subject of study in areas such as abstract model theory, topological logic, etc. here, we will present how topological spaces can be thought of as spaces of interpretations for a logic and, in doing so, help shed light on the theoretical inspirations for our constructions. definition 8 we call topological space any tuple \(\mathcal {w} = \langle w, \tau \rangle \), where w is a non-empty set and \(\tau \) is a collection of subsets of w satisfying: 1. \(\emptyset , w\in \tau \); 2. if \(o,o'\in \tau \), then \(o\cap o' \in \tau \); 3. if \(\mathcal {o}\subseteq \tau \), then \(\bigcup \mathcal {o} \in \tau \). the elements of \(\tau \) are called open sets of \(\mathcal {w}\). definition 9 let \(\mathcal {l} = \langle l, cn\rangle \) be a logic and \(\mathcal {w} = \langle w, \tau \rangle \) a topological space. we say \(\mathcal {w}\) induces the logic \(\mathcal {l}\) if there is some function \(f: \tau \rightarrow th_\mathcal {l}\) s.t. if \(o \subseteq o'\) then \(f(o') \subseteq f(o)\); \(f(\bigcup \mathcal {o}) = \bigcap _{o\in \mathcal {o}} f(o)\) for any theories \(k,k'\in th_\mathcal {l}\), it holds \(f(\bigcup f^{-1}(k) \cap \bigcup f^{-1}(k')) = cn(k \cup k')\) in this case, we say f is an \(\mathcal {l}\)-interpretation on \(\mathcal {w}\). with that, it is easy to see that any tarskian logic is induced by some topological space since we can use the lattice of theories to construct it, in a similar fashion to what we have done in proposition 2. proposition 3 let \(\mathcal {l} = \langle l, cn\rangle \) be a logic, there is a topological space \(\mathcal {w} = \langle w, \tau \rangle \) inducing \(\mathcal {l}\). proof take \(\mathcal {w} = \langle th_\mathcal {l}, \tau \rangle \) s.t. that topology \(\tau \) is the upset topology from the lattice of theories, i.e. the topology obtained from the subbase \(\mathcal {b} = \{\uparrow t~|~t\in th_\mathcal {l}\}\), where \(\uparrow t = \{t'\in th_\mathcal {l}~|~ t\subseteq t'\}\). it is trivial to verify that the function \(f(o) = \bigcap o\) is an \(\mathcal {l}\)-interpretation on \(\mathcal {w}\). by contrasting propositions 2 and 3, we are able to see that there is a strong connection between abstract model theories and topological spaces equipped with interpretation functions. lemma 3 let \(\mathcal {w} = \langle w, \tau \rangle \) be a topological space inducing a logic \(\mathcal {l}\) and \(f:\tau \mapsto th_\mathcal {l}\) an \(\mathcal {l}\)-interpretation on \(\mathcal {w}\). the abstract model theory \(\mathcal {m}_\mathcal {w} = \langle w, \vdash \rangle \) s.t. \(w\vdash \varphi \) if \(\varphi \in f(\bigcup \{o\in \tau : w\in o\})\) induces \(\mathcal {l}\). furthermore, \(bo_{\mathcal {m}_\mathcal {w}} = \{\cup f^{-1}(k)~|~k \in th_\mathcal {l}\}\). notice that lemma 3 above tells us that the language of amts and topological spaces are interchangeable to study the structure of the logic. moreover, it is easy to see that any amt defines a topological space inducing the same logic. lemma 4 let \(\mathcal {m} = \langle m, l, \vdash \rangle \) be an abstract model theory inducing a logic \(\mathcal {l}\), the topological model \(\mathcal {w}_{\mathcal {m}} = \langle m, 2^m\rangle \) induces \(\mathcal {l}\) and \(f: 2^m\mapsto th_\mathcal {l}\) s.t. \(f(o) = \{\varphi \in l~|~ \forall w\in o:~w\vdash \varphi \}\) is an \(\mathcal {l}\)-interpretation on \(\mathcal {w}\). we employed the terminology of open basic sets to the sets \(\llbracket \vargamma \rrbracket \) as a foreshadowing of the fact that we employ exactly those sets to provide a basis for the canonical topology inducing the logic. in a sense, basic open sets are the ‘stable’ interpretations, meaning that they are completely definable within the logical language. 4 topological belief change and agm rationality we begin our investigation of belief change by proposing a semantic model for belief change, which is agnostic to the underlying logic. notice that, differently from sect. 2, we will consider belief contractions as operations which take two sets of formulas and return a set of formulas, i.e. a function of the form \(\star : 2^l \times 2^l \mapsto 2^l\), as done by ribeiro and wasserman [16]. the reason for that is some non-classical logic may not possess conjunctions in its language. this means that we are dealing with multiple contractions, rather than single contractions. more precisely with choice contractions [6]. the central notion underlying most belief change operations is that of choice. different constructions [1, 8, 15] represent the agent’s epistemic commitments by means of which beliefs the agent chooses to preserve (or discard) given new information. in connection with the literature on conditional logic and semantic modelling, we will represent this choice element as a function selecting elements from a set. we will focus our presentation on abstract model theories. however, given lemmas 3 and 4, the same notions could be equivalently described employing topological spaces and interpretation functions. definition 10 let \(mod(\mathcal {l}) = \langle m, \vdash \rangle \) be an amt and \(x\subseteq m\) a set of points (interpretations), we call a semantic selection function (ssf) on x any mapping \(f: 2^m \times 2^m \rightarrow 2^m\), such that for any \(y,z\subseteq m\), it holds that: i) \(x \subseteq f(x,y) \subseteq x\cup y\) ii) if \(y\ne \emptyset \), then \(f(x,y)\cap y\ne \emptyset \) iii) if \(y\cap x \ne \emptyset \), then \(f(x,y) \subseteq x\) additionally, we say an ssf is uniform if it satisfies: iv) if \(f(x,y)\cap z \ne \emptyset \) and \(f(x,z)\cap y \ne \emptyset \) then, \(f(x,y) = f(x,z)\). with semantic selection functions, we can finally define a form of belief contraction, which we will study in this work. we aim to show that our semantic selection contractions generalise different notions of belief contractions in the literature and characterise agm contractions by means of properties on the selection function and the space of models of the logic. definition 11 let \(\mathcal {l} = \langle l, cn\rangle \) be a logic, we say a belief change operator \(\dot{-} : 2^l \times 2^l \rightarrow 2^l\) is a semantic selection contraction operation on a set of formulas \(b\subseteq l\), if there is an amt \(mod(\mathcal {l})\) and an ssf f on \(\llbracket b\rrbracket \) s.t. for any set of formulas \(a \subseteq l\), it holds: $$b\dot{-}a = th(f(\llbracket b\rrbracket , \overline{\llbracket a\rrbracket } )) \cap b.$$ in such a case, we say f is a characteristic function for \(\dot{-}\). first, we show that semantic selection contractions satisfy the basic postulates for belief contraction. proposition 4 let \(\mathcal {l}\) be a logic, \(b,a,a'\subseteq l\) be sets of formulas and \(\dot{-} : 2^l \times 2^l \rightarrow 2^l\) is a semantic selection contraction operation on b, then \(\dot{-}\) satisfies: (closure) if \(cn(b)=b\), then \(cn(b\dot{-}a) = b\dot{-}a\) (success) if \(a \not \subseteq cn(\emptyset )\), then \(a\not \subseteq cn(b \dot{-}a)\) (inclusion) \(b\dot{-}a\subseteq b\) ‘additionally, if the characteristic function of \(\dot{-}\) is uniform, then \(\dot{-}\) satisfies (uniformity) if for any \(b'\subseteq b\) it holds that \(a \subseteq cn(b')\) iff \(a' \subseteq cn(b')\), then it holds that \(b\dot{-}a = b\dot{-}a'\) proof if \(\dot{-}\) is a semantic selection contraction, then there is some amt \(mod(\mathcal {l}) = \langle m,\vdash \rangle \) and a semantic selection function f, s.t., f is a characteristic function for \(\dot{-}\). the satisfaction of the closure and inclusion postulates is trivial by definition of a semantic selection contraction. let us show that the other two hold. (success): if \(a \not \subseteq cn(\emptyset )\), then \(\llbracket a\rrbracket \ne m\), i.e. \(\overline{\llbracket a\rrbracket } \ne \emptyset \). then \(f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket })\cap \overline{\llbracket a\rrbracket } \ne \emptyset \). as such, there is some \(m \in f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket })\), s.t. \(m\not \vdash a\) and \(a\not \subseteq th(f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket }))\) in other words, \(a \not \subseteq b\dot{-}a\). (uniformity) let \(a,a' \subseteq l\) s.t. for any \(b'\subseteq b\) it holds that \(a \subseteq cn(b')\) iff \(a' \subseteq cn(b')\). if \(a\not \subseteq b\), then \(\overline{\llbracket a\rrbracket }\cap \llbracket b\rrbracket \ne \emptyset \) and, by condition iii) of definition 10, \(f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket }) = \llbracket b\rrbracket \). as such, we only need to consider the case in which \(a,a'\subseteq cn(b)\). there are two cases to consider: \(a \subseteq cn(b\dot{-} a)\) and \(a \not \subseteq cn(b\dot{-} a)\). in the first case, \(a \subseteq cn(\emptyset )\), by (success), thus, \(f(\llbracket b\rrbracket , \overline{\llbracket a\rrbracket }) = \llbracket b\rrbracket = f(\llbracket b\rrbracket , \overline{\llbracket a'\rrbracket })\) and \(a'\subseteq cn(b\dot{-}a')\), by condition i) of definition 10. in the second case, as \(b\dot{-}a \subseteq b\), then \(a\not \subseteq cn(b\dot{-}a)\) implies \(a'\not \subseteq cn(b\dot{-}a)\) and, in particular, \(a' \not \subseteq th(f(\llbracket b\rrbracket ,\overline{\llbracket a'\rrbracket }))\). as such, it must be the case that \(f(\llbracket b\rrbracket ,\overline{\llbracket a'\rrbracket }) \cap \overline{\llbracket a\rrbracket } \ne \emptyset \). conversely, we may also conclude that \(f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket }) \cap \overline{\llbracket a'\rrbracket } \ne \emptyset \), thus, \(f(\llbracket b\rrbracket ,\overline{\llbracket a\rrbracket }) = f(\llbracket b\rrbracket ,\overline{\llbracket a'\rrbracket })\), since f is uniform, and \(b\dot{-} a = b\dot{-}a'\).    \(\square \) our interest now is investigating which properties of semantic selection functions imply adequate behaviours in their respective selection contractions. proposition 5 let \(mod(\mathcal {l}) = \langle m, \vdash \rangle \) be an amt inducing logic \(\mathcal {l} = \langle l, cn\rangle \), and \(b,a \subseteq th_\mathcal {l}\) theories s.t. \(a \not \subseteq b\). if cn is compact, for any \(b' \in b\bot a\), there is some \(w\in \overline{\llbracket a\rrbracket }\) s.t. \(b' = th(\llbracket b\rrbracket \cup \{m\})\). if \(\mathcal {l}\) is agm-compliant, for any \(b' \in b^{-(a)}\) there is \(x \subseteq \overline{\llbracket a\rrbracket }\) s.t. \(b' = th(\llbracket b\rrbracket \cup x)\). the above result indicates that both agm’s rational contractions and hansson’s belief base contraction could, in principle, be defined by means of a semantic selection function that chooses an appropriate set of models to define the contraction. the question that now remains is: which conditions could be enforced on ssfs to define these operations? from now on, we will focus on studying the conditions of an ssf to ensure the respective contraction is rational. the study of partial meet contractions in the framework of semantic selection contractions will be relegated to future work. definition 12 let \(\mathcal {m} = \langle m, l, \vdash \rangle \) be an amt, \(x \in bo_{\mathcal {m}}\) be a b.o.s, and f be an ssf on x. we say f is topological in x if for any y s.t. \(\overline{y} \in bo_{\mathcal {m}}\), it holds that \(f(x,y) \in bo_{\mathcal {m}}\). we say a selections contraction operator \(\dot{-}\) on a set of formulas b is topological if there is some topological ssf f on \(\llbracket b\rrbracket \). we now show that topological contractions are agm rational contractions. theorem 1 let \(\mathcal {l}\) be a logic, \(k\in th_\mathcal {l}\) be a theory and \(\dot{-} : 2^l \times 2^l \rightarrow 2^l\) be a topological semantic selection contraction on k. then, for any \(a,a'\subseteq l\), it holds that: (closure) \(k\dot{-}a = cn(k\dot{-}a)\) (success) if \(a \not \subseteq cn(\emptyset )\), then \(a\not \subseteq cn(b \dot{-}a)\) (inclusion) \(b\dot{-}a\subseteq b\) (vacuity) if \(a\not \subseteq k\) then \(k\dot{-}a = k\) (recovery) \(k \subseteq cn(k\dot{-}a \cup a)\) (extensionality) if \(cn(a) = cn(a')\) then \(k\dot{-}a = k\dot{-}a'\) proof notice that (extensionality) and (closure) follow directly from the fact that k is a theory and from the definition of a selection contraction. for the other postulates, let us take a model theory \(mod(\mathcal {l})\) and an ssf f which characterise \(\dot{-}\) on the set b. as (success) and (inclusion) have been proved in proposition 4, we will focus on the remaining postulates. we briefly discussed (vacuity) in the proof of (uniformity) in proposition 4, but let us show the complete reasoning here. take \(a \not \subseteq k\), then \(\llbracket k\rrbracket \cap \overline{\llbracket a\rrbracket } \ne \emptyset \). as such, \(f(\llbracket k\rrbracket ,\llbracket k\rrbracket ) \cap \overline{\llbracket a\rrbracket } = \llbracket k\rrbracket \cap \overline{\llbracket a\rrbracket } \ne \emptyset \). on the other hand, \(f(\llbracket k\rrbracket , \overline{\llbracket a\rrbracket }) \cap \llbracket k\rrbracket \ne \emptyset \). thus, \(f(\llbracket k\rrbracket ,\overline{\llbracket a\rrbracket }) = \llbracket k\rrbracket \) and \(k\dot{-}a = k\). to prove (recovery), we just need to observe that, since f is topological and \(\llbracket k\dot{-}a\rrbracket = f(\llbracket k\rrbracket , \overline{\llbracket a\rrbracket })\), it holds that $$\begin{array}{ll} \llbracket cn( k\dot{-}a \cup a)\rrbracket & = \llbracket k\dot{-}a\rrbracket \cap \llbracket a\rrbracket \\ & = f(\llbracket k\rrbracket , \overline{\llbracket a\rrbracket }) \cap \llbracket a\rrbracket \\ & = (\llbracket k\rrbracket \cup x) \cap \llbracket a\rrbracket ,~\text{ for } \text{ some }~x\subseteq \overline{\llbracket a\rrbracket } \\ & = \llbracket k\rrbracket \cap \llbracket a\rrbracket \end{array}$$ since \(\llbracket cn( k\dot{-}a \cup a)\rrbracket \subseteq \llbracket k\rrbracket \), we conclude \(k \subseteq cn(k\dot{-}a \cup a)\).    \(\square \) more yet, any semantic selection contraction which is rational is topological. theorem 2 let \(\mathcal {l}\) be a logic, \(k\in th_\mathcal {l}\) be a theory and \(\dot{-} : 2^l \times 2^l \mapsto 2^l\) be a semantic selection contraction on k satisfying (closure), (success),(inclusion), (vacuity),(recovery), and (extensionality) for any \(a\subseteq l\), then \(\dot{-}\) is topological. proof as \(\dot{-}\) is a semantic selection contraction, it means that there is some \(mod(\mathcal {l}) = \langle m, \vdash \rangle \) and an ssf f s.t. for any \(a \in th_\mathcal {l}\), \(k\dot{-}a = th(f(\llbracket k\rrbracket , \overline{\llbracket a\rrbracket })\). let us construct a function \(f'\) as $$ f'(x,y) = {\left\{ \begin{array}{ll} f(x,y) & \text{ if }~x\ne \llbracket k\rrbracket \\ cl(f(x,y))\cap (x\cup y) & \text{ if }~x= \llbracket k\rrbracket \\ \end{array}\right. }$$ we need to show that \(f'\) is ssf on \(x=\llbracket k\rrbracket \) and that it is topological on x. i) notice \(x \subseteq f'(x,y) \subseteq x\cup y\) is immediate by construction, as \(x \subseteq f(x,y) \subseteq x\cup y\), since f is an ssf, and \(z \subseteq cl(z)\). ii) if \(y\ne \emptyset \), then \(f(x,y)\cap y\ne \emptyset \), thus \(cl(f(x,y))\cap y \ne \emptyset \), and \(f'(x,y)\cap y \ne \emptyset \). iii) take \(y\subseteq m\), s.t. \(x\cap y \ne \emptyset \), then \(f(x,y) =x\), thus \( f'(x,y) = cl(x) = x\) since \(x= \llbracket k\rrbracket \). iv) take \(y \in bo_{mod(\mathcal {l})}\), \(f'(x,\overline{y}) = cl(f(x,\overline{y})) \cap (x\cup y)\). first, let us see that \(th(f'(x,y)) = th(f(x,y))\). we known that \(th(f(x,y)) = th(cl(f(x,y))\), by definition, and \(f'(x,y) \subseteq cl(f(x,y))\), thus it holds that \(th(f(x,y)) \subseteq th(f'(x,y))\). furthermore, as \(f(x,y) \subseteq cl(f(x,y))\) and \(f(x,y)\subseteq x\cup y\), it holds that \(f(x,y) \subseteq f'(x,y)\). thus \(th(f'(x,y)) \subseteq th(f(x,y))\). let \(k' = th(f'(x,y))\). since \(th(f'(x,y)) = th(f(x,y))\), \(x = \llbracket k\rrbracket \), \(\overline{y}\in bo_{mod(\mathcal {l})}\), and f is the characteristic ssf of \( \dot{-}\), then \(k' = k \dot{-} th(\overline{y})\). now, take \(w\in \llbracket k'\rrbracket \) and suppose \(w\not \in x\cup y\), then \(w\in \overline{x} \cap \overline{y}\). from \(w\vdash k'\), we conclude that \(w\vdash k \dot{-} th(\overline{y})\) and, from \(w\in \overline{y}\), we conclude that \(w\vdash th(\overline{y})\). since \(\dot{-}\) is a rational contraction, thus satisfying (recovery), it must hold that \(w\vdash k\), i.e. \(w\in \llbracket k\rrbracket = x\). but this contradicts the fact that \(w\in \overline{x}\). then it must hold that \(\llbracket k'\rrbracket \subseteq x\cup y\) and \(f'(x,y) = \llbracket k'\rrbracket \in bo_{mod(\mathcal {l})}\). thus \(f'\) is topological.    \(\square \) notice in the proof above that the requirement of satisfying the (recovery) postulate is essential to construct a topological ssf. as (recovery) is a central property of rational contractions, i.e. it is the postulate differentiating these operations from others in the literature, such as withdraws, partial meet, etc., it indicates that our notion of topological encodes the central aspects of agm rationality. 4.1 on the existence of topological contractions we now turn our attention to investigating sufficient and necessary conditions for the definability of topological contractions in a given logic. this investigation is closely related to those of flouris [5] and ribeiro et al.  [15], for example, but employing the technical framework of abstract model theory and topology. first, an obvious observation is that topological functions can be constructed whenever there are appropriate basic open sets. lemma 5 let \(mod(\mathcal {l})\) be a regular abstract model theory and \(x \in bo_{mod(\mathcal {l})}\) be a b.o.s. there is a topological semantic selection function f on x iff for any b.o.s. \(y\in bo_{mod(\mathcal {l})}\) s.t. \(\overline{y} \cap x=\emptyset \), there is some b.o.s. \(z\in bo_{mod(\mathcal {l})}\) s.t. \(x \subset z \subseteq x \cup \overline{y}\). notice that lemma 5 above is directly related to proposition 1. in fact, the results can be directly translated into each other. as such, this informs us that any agm rational contraction is, in fact, a topological semantic selection contraction. also, in the face of flouris’ [5, theorem 4.2] characterisation of agm-compliance by cuts, a similar characterisation could be encoded within our language. since this result does not increase our understanding of agm compliance and the definability of topological functions, we will omit it in this work. lemma 5, however, can be employed to show that some logics are not agm-compatible. example 1 let us consider the intuitionistic propositional logic with one propositional symbol p, \(\mathcal {l}_i = \langle l_0, cn\rangle \) and \(mod(\mathcal {l}_i)\) its canonical model theory, i.e. the model theory obtained from its theory space, as in the proof of proposition 2. the b.o.s.’s of \(mod(\mathcal {l}_i)\) form a complete lattice regarding the inclusion order, corresponding to the rieger-nishimura lattice of intuitionistic theories, depicted in fig. 1. any b.o.s. in it contains all sets below it in the lattice. consider now the theories \(k = cn(\lnot p)\) and set \(a = cn(p\vee \lnot p)\), as one can see in fig. 1, there is no theory \(\vargamma '\) s.t. \(cn(\emptyset ) \subset k'\), \(a \not \subseteq k'\), and \( cn(k' \cup a) = cn(\emptyset )\). in another way, given the open set \(x = \llbracket \lnot p\rrbracket \), for \(y = \llbracket p \vee \lnot p\rrbracket \), there is no b.o.s. z s.t. \(x\subset z \subseteq x \cup \overline{y}\), as any such element would necessarily include \(cn(p\vee \lnot p)\not \in x \cup \overline{y}\). thus, by lemma 5, there is no topological semantic selection function on \(\llbracket \lnot p\rrbracket \). fig. 1. the rieger-nishimura lattice of intuitionistic basic open sets full size image notice that the topological contractions provided above are defined to contract a (possibly infinite) set of formulas from a theory which coincides with the notion of choice multiple contractions [6]. it is useful, however, to weaken our definition to account for a smaller family of theories considered as inputs to our contraction functions. for example, we can restrict our analysis to contracting a finite amount of information (or equivalently a finitely representable theory), as investigated by authors such as ribeiro and wassermann [16] in the context of belief change for description logics. to provide constructions for rational contractions on a limited class of inputs, we need to introduce the notion of a restricted semantic selection function. definition 13 let \(mod(\mathcal {l}) = \langle m, \vdash \rangle \) be an abstract model theory , \(\mathcal {o} \subseteq 2^m\) a family of subsets of m, \(x\in \mathcal {o}\) a set of interpretations, and \(f: 2^m \times 2^m \rightarrow 2^m\) a function. we say f is an ssf restricted to \(\mathcal {o}\) if its restriction to \(\mathcal {o}\) is an ssf on x, i.e. of the function \(f\restriction _\mathcal {o}\) satisfies the conditions in definition 10. notice the above definition is akin to defining selection functions on the sublogic defined by the family \(\mathcal {o}\). furthermore, we say an operator is a semantic contraction operator on a set of sentences b, restricted to a family of theories \(\mathcal {t}\), if there is an ssf on \(\llbracket b\rrbracket \) restricted to \(\overline{\llbracket \mathcal {t}\rrbracket } = \{\overline{\llbracket k\rrbracket }~|~ k\in \mathcal {t}\}\), satisfying the condition in definition 11 for any \(a \subseteq \mathcal {t}\). with that, it is easy to see that the following holds. corollary 1 let \(\mathcal {l}\) be a logic, \(k\in th_\mathcal {l}\) be a theory, \(\mathcal {t}\subseteq th_\mathcal {l}\) be a family of theories of \(\mathcal {l}\), and \(\dot{-} : 2^l \times 2^l \rightarrow 2^l\) be a semantic selection contraction on k restricted to \(\mathcal {t}\). then, \(\dot{-}\) satisfies (closure), (success),(inclusion), (vacuity),(recovery), and (extensionality) for all \(a\subseteq l\) with \(cn(a) \in \mathcal {t}\), if, and only if, it is topological on \(\mathcal {t}\). we employ the result above to show that our construction subsumes those proposed by authors such as ribeiro et al. [15] for rational contractions of finitely representable theories on boolean non-compact logics. let us first define an appropriate property of this family of theories that is appropriate for the construction of a topological function. definition 14 let \(\mathcal {l}\) be a logic and \(\mathcal {t} \subseteq th_\mathcal {l}\) be a family of \(\mathcal {l}\)-theories. we say \(\mathcal {t}\) is closed under classical negation if for any \(\vargamma \in \mathcal {t}\), there is some \(neg(\vargamma ) \in \mathcal {t}\) s.t. \(neg(\vargamma )\cap \vargamma = cn(\emptyset )\) and \(cn(\vargamma \cup neg(\vargamma )) = l\). it is easy to see that we can always construct topological functions restricted to a family of theories which is closed under classical negation. proposition 6 let \(\mathcal {l}\) be a logic and \(\mathcal {t} \subseteq th_\mathcal {l}\) a family of theories closed under classical negation, there is a topological selection function f restricted to the set \(\llbracket \mathcal {t}\rrbracket = \{\llbracket \vargamma \rrbracket ~|~\vargamma \in \mathcal {t}\}\). proof it suffices to see that for any family of theories \(\mathcal {t}\) closed under classical negation, if \(\vargamma \in \mathcal {t}\), there is some theory \(neg(\vargamma ) \in \mathcal {t}\) s.t. \(neg(\vargamma ) \cap \vargamma = cn(\emptyset )\) and \(cn(neg(\vargamma ) \cup \vargamma ) = l\). as such, \(\overline{\llbracket \vargamma \rrbracket }\) is a b.o.s. and, thus, there is a function \(f(x,y) = x\) if \(x\cap y\ne \emptyset \) and \(f(x,y) = x\cup y\), which is topological for all \(y = \overline{\llbracket \vargamma \rrbracket } = \llbracket neg(\vargamma )\rrbracket \) with \(\vargamma \in \mathcal {t}\).   \(\square \) the above result generalises ribeiro et al. [15] construction for non-compact logics, showing that we can extend their construction to consider other logics and inputs as long as we restrict the input to cases in which the logic admits classical negation. corollary 2 [15] let \(\mathcal {l}\) be a logic closed under disjunction and classical negation, there is a topological semantic selection contraction \(\dot{-}\) restricted to finitely representable theories. 5 related work work on the definability of belief change operators in non-classical logics remounts at least to the work of hansson and wassermann [9], in which the authors show that partial meet contractions are definable for any monotonic and compact logic. while most work on belief change for non-classical logics, such as that of delgrande [4] or girard and tanaka [7], focused on defining appropriate operations in these logics, only recent work in the literature has focused on investigating which classes of logics allow the definition of a given operator. famously, flouris [5] has shown a characterisation of the definability of agm rational contractions employing the framework of order theory and its connections to abstract logic. this study has been explored further by ribeiro et al. [15] showing constructions of agm rational contractions for non-compact logics, and by souza and wassermann [18] showing that a semantically motivated framework, which they employ to study hyperintensional belief change, provides a rich foundation to encode and study different notions of belief change. our work develops on these ideas, connecting the work on conditional logics, abstract model theory and belief change to investigate a general notion of belief contraction, namely semantic selection contraction, which can be used to unify different results in the area. 6 final considerations in this work, we investigated the use of abstract model theory and topology to describe belief change operators, showing that our notion subsumes different operations in the literature, such as partial meet contractions and agm rational contractions. furthermore, we characterise rational contractions within our framework and show that other characterisation results, such as that of flouris [5] and ribeiro et al. [15], are captured within it. this investigation provides greater evidence to the conjecture raised by souza and wassermann [19] that “different notions of minimality arise as the reflection of topological properties of the model space”, not necessarily as intrinsic distinction in their constructions. for future work, we intend to study different belief change operations in this framework, such as partial meet contractions, revisions and semi-revisions and their relations, aiming to provide a unified view of definability and constructibility of these operations based on a single semantic framework. notes 1.such logics are called tarskian, due to polish logician alfred tarski. non-tarskian logics exist in the literature but will not be studied in this work. 2.in the following we will often refer to closed sets by the letter k (\(k'\),\(k''\), etc.), arbitrary sets of formulas by the letter b (\(b'\),\(b''\), etc.). references alchourrón, c.e., gärdenfors, p., makinson, d.: on the logic of theory change: partial meet contraction and revision functions. j. symb. log. 50(2), 510–530 (1985) google scholar  baltag, a., bezhanishvili, n., özgün, a., smets, s.: the topology of belief, belief revision and defeasible knowledge. in: logic, rationality, and interaction, pp. 27–40. springer, new york (2013) google scholar  bjorndahl, a., özgün, a., et al.: logic and topology for knowledge, knowability, and belief. electron. proc. theor. comput. sci. 251 (2017) google scholar  delgrande, j.p.: horn clause belief change: contraction functions. in: proceedings of the eleventh international conference on principles of knowledge representation and reasoning, pp. 156–165 (2008) google scholar  flouris, g.: on belief change and ontology evolution. ph.d. thesis, university of crete (2006) google scholar  fuhrmann, a., hansson, s.o.: a survey of multiple contractions. j. logic lang. inform. 3, 39–75 (1994) article  mathscinet  math  google scholar  girard, p., tanaka, k.: paraconsistent dynamics. synthese 193(1), 1–14 (2016) article  mathscinet  math  google scholar  hansson, s.o.: in defense of base contraction. synthese 91(3), 239–245 (1992) article  mathscinet  math  google scholar  hansson, s.o., wassermann, r.: local change. stud. logica. 70(1), 49–76 (2002) article  mathscinet  math  google scholar  kelly, k.t.: the logic of reliable inquiry. oup usa (1996) google scholar  lewitzka, s., brunner, a.b.: minimally generated abstract logics. log. univ. 3, 219–241 (2009) google scholar  makowsky, j.a., shelah, s.: positive results in abstract model theory: a theory of compact logics. ann. pure appl. logic 25(3), 263–299 (1983) article  mathscinet  math  google scholar  özgün, a.: topological models for belief and belief revision. ph.d. thesis, universiteit van amsterdam (2013) google scholar  peppas, p.: the limit assumption and multiple revision. j. log. comput. 14(3), 355–371 (2004) article  mathscinet  math  google scholar  ribeiro, j.s., nayak, a., wassermann, r.: towards belief contraction without compactness. in: 16th international conference on principles of knowledge representation and reasoning, pp. 287–296. association for the advancement of artificial intelligence (2018) google scholar  ribeiro, m.m., wassermann, r.: base revision for ontology debugging. j. log. comput. 19(5), 721–743 (2009). https://doi.org/10.1093/logcom/exn048 article  mathscinet  math  google scholar  ribeiro, m.m.: belief revision in non-classical logics. springer (2013) google scholar  souza, m., wassermann, r.: hyperintensional models and belief change. in: brazilian conference on intelligent systems, pp. 429–443. springer (2022) google scholar  souza, m., wassermann, r.: hyperintensional partial meet contractions. in: proceedings of the international conference on principles of knowledge representation and reasoning (2022) google scholar  stone, m.h.: the theory of representation for boolean algebras. trans. am. math. soc. 40(1), 37–111 (1936) mathscinet  math  google scholar  van fraassen, b.c., fraassen, v.: formal semantics and logic, vol. 214. macmillan, new york (1971) google scholar  download references author information authors and affiliations federal university of bahia, salvador, brazil marlo souza authors marlo souzaview author publications search author on:pubmed google scholar corresponding author correspondence to marlo souza . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper souza, m. (2025). a topology-inspired approach to agm belief change. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_1 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords belief change model theory non-classical logics publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature lackadaisical quantum walk in the hypercube to search for multiple marked vertices | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper lackadaisical quantum walk in the hypercube to search for multiple marked vertices conference paper first online: 28 november 2021 pp 249–263 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) lackadaisical quantum walk in the hypercube to search for multiple marked vertices download book pdf download book epub luciano s. de souza  orcid: orcid.org/0000-0002-6527-706510, jonathan h. a. de carvalho  orcid: orcid.org/0000-0002-2672-780111 & tiago a. e. ferreira  orcid: orcid.org/0000-0002-2131-982510  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 749 accesses 4 citations abstract adding self-loops at each vertex of a graph improves the performance of quantum walks algorithms over loopless algorithms. many works approach quantum walks to search for a single marked vertex. in this article, we experimentally address several problems related to quantum walk in the hypercube with self-loops to search for multiple marked vertices. we first investigate the quantum walk in the loopless hypercube. we saw that neighbor vertices are also amplified and that approximately 1/2 of the system energy is concentrated in them. we show that the optimal value of l for a single marked vertex is not optimal for multiple marked vertices. we define a new value of \(l = (n/n)\cdot k\) to search multiple marked vertices. next, we use this new value of l found to analyze the search for multiple marked vertices non-adjacent and show that the probability of success is close to 1. we also use the new value of l found to analyze the search for several marked vertices that are adjacent and show that the probability of success is directly proportional to the density of marked vertices in the neighborhood. we also show that, in the case where neighbors are marked, if there is at least one non-adjacent marked vertex, the probability of success increases to close to 1. the results found show that the self-loop value for the quantum walk in the hypercube to search for several marked vertices is \(l = (n / n) \cdot k \). acknowledgments to the science and technology support foundation of pernambuco (facepe) brazil, brazilian national council for scientific and technological development (cnpq), and coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001 by their financial support to the development of this research. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others lackadaisical quantum walks with multiple marked vertices chapter © 2019 faster search of clustered marked states with lackadaisical quantum walks article 13 august 2022 adjacent vertices can be hard to find by quantum walks chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms combinatorial geometry data structures graph theory quantum computing qubits 1 introduction according to shenvi, kempe, and whaley [17], quantum walks provide one of the most promising features, an intuitive framework for building new quantum algorithms. they were pioneers in designing a quantum search algorithm on the hypercube based on quantum random walks [14]. recent works have used the quantum walks to search weights and train artificial neural networks [19, 20]. the topology of the structure where the walk is applied considerably affects the evolution of the walker [22]. therefore, many works are developed to improve the performance of quantum walks, quantum search algorithms in different structures: one-dimensional, two-dimensional, and multidimensional grids, complete and bipartite graphs, among others [4, 5, 11, 15]. quantum walk modification proposals are also made to improve their performance. for example, wong [24] added to each vertex of a two-dimensional grid a self-loop, so the walker has some probability of staying put, achieving an improvement over the algorithm without self-loop [3]. rhodes [16] proposed an ideal weight for all vertex-transitive graphs with a single marked vertex such that the ideal self-loop weight is equal to the degree of the loopless graph divided by the total number of vertices. potovcek [14] observed that the nearest neighbors are also presented with high probability and nahimovs [10] that adjacent vertices can be hard to find by quantum walks. in this way, we investigate whether the optimal value of \(l = (d / n)\) for a single marked vertex is optimal for multiple marked vertices, where d is the degree of the loopless vertex and n is the number of vertices. we analyzed the quantum walk on hypercube without self-loop and with self-loop. we analyzed the quantum walk on the hypercube for multiple marked adjacent and non-adjacent vertices. finally, we find an optimal value of l for a quantum walk in the hypercube with multiple marked vertices. this paper is organized as follows. in sect. 2, we present some concepts about quantum walks and specifically the quantum walk on the hypercube. in sect. 3, we characterize the probability distribution along with the space, adjust the self-loop weight for multiple marked vertices, and search for adjacent marked vertices. finally, in sect. 4 is the conclusion. 2 quantum walk the processing of quantum information is governed by quantum mechanics or quantum physics [18]. quantum computing study the processing of this information [8, 13, 25]. quantum walks are the quantum counterpart of classical random walks. discrete and continuous-time quantum walks are the advanced tools used to build quantum algorithms [1, 2]. the main feature that differentiates these two types of quantum walks is the timing used to applying the evolution operators. in the quantum walk in continuous time, the evolution operator is applied at any time, whereas the quantum walks in discrete time, the evolution operator is applied in discrete time steps [21]. the quantum walk evolution in the discrete-time process occurs by the successive applications of a unitary evolution operator u that acts on the hilbert space $$\begin{aligned} \mathcal {h} = \mathcal {h}^{c} \otimes \mathcal {h}^{s}. \end{aligned}$$ the coin space \(\mathcal {h}^{c}\) is the hilbert space associated with a quantum coin, and the walker’s space \(\mathcal {h}^{s}\) is the hilbert space associated with the position of the nodes in a graph, for example. the evolution operator u is defined in eq. 1. $$\begin{aligned} u = s(c\otimes i_{n}) \end{aligned}$$ (1) where, s is the shift operator, i.e., a permutation matrix that acts in the walker’s space based on the state of the coin space. the unitary matrix c is the coin operator [17]. therefore, the equation of evolution represented by a quantum walk at time t is given by 2.1 quantum walk on the hypercube according to venegas [21], the hypercube is defined as an undirected graph of degree n and \(n = 2^{n}\) nodes. each node is represented by an n-bit binary string. two nodes \(\vec {x}\) and \(\vec {y}\) are connected by an edge if the hamming distance between them is 1, i.e., \(\left| \vec {x} \vec {y} \right| = 1\). this means that \(\vec {x}\) and \(\vec {y}\) only differ in a single bit. the expression \(\left| \vec {x} \right| \) is the hamming weight of \(\vec {x}\). the hilbert space associated with the quantum walk on the hypercube is $$\begin{aligned} \mathcal {h} = \mathcal {h}^{n} \otimes \mathcal {h}^{2^{n}}, \end{aligned}$$ where \(\mathcal {h}^{n}\) is the hilbert space associated with the quantum coin space, and \(\mathcal {h}^{2^{n}}\) is the hilbert space associated with nodes on the hypercube. according to shenvi [17], in a d-dimensional hypercube, the d directions specify the coin state. kempe [7] defines that directions can be labeled by the n base-vectors on the hypercube which corresponding to the n vectors of hamming weight 1. these n vectors are represented by the states , where \(e_{d}\) has a 1 in the d-th bit. the shift operator s described in eq. 2 acts mapping a state . (2) the initial state of the quantum walk in the hypercube is defined according to eq. 3 as an equal superposition over all n nodes and n directions. (3) according to rhodes [16], the hypercube was the first graph in which quantum walks were researched. in their work, shenvi [17] presented a quantum search algorithm based on the random walk quantum architecture. in this article, we are based on the approach used by wong [24]. the pure quantum walk (without search) evolves by repeated applications from the evolution operator described in eq. 1, where c is grover’s “diffusion” operator on the coin space and is given by (4) where, \(i_{n}\) is the identity operator, n is the vertex degree loopless, and is the equal superposition over all n directions [9, 17], i.e., (5) we include a query to the “grover oracle”, described in eq. 6, at each step of the quantum walk. $$\begin{aligned} u' = u \cdot (i_{n} \otimes q) \end{aligned}$$ (6) where, , and means the marked vertex. the system is initiated according to the initial state presented in eq. 3. 3 analyzing the quantum walk on the hypercube in this section, we experimentally analyze the quantum walk on the hypercube searching for multiple marked vertices. the simulations and the obtained results are detailed in the following subsections. 3.1 characterizing the probability distribution along the space previous works showed there is an amplification in the solution neighborhood, which interferes with the amplification of the solutions by the quantum walk on the hypercube [11, 14, 17]. initially, it is necessary to understand how the probability amplitudes are distributed in the search space and how the quantum walk evolves in the hypercube over time considering the impacts caused by the solution neighborhood. figure 1 shows the probability of success after one hundred steps for the quantum walk in the hypercube with one, two, three, and four arbitrarily marked vertices. although the search algorithm is able to amplify the probability amplitudes of the marked vertices, if a measurement is performed, the probability of finding one of the solutions is still unsatisfactory. another interesting aspect that can be observed is that as the number of marked vertices increases, the speed of amplification the probability amplitudes also increases. however, it is necessary to increase the probability amplitudes of the marked vertices. figure 2 shows the probability distributions of the marked vertices only after the number of iterations necessary to reach the maximum value of the probability amplitude close to 1/2. as potovcek et al. [14] noted in their work, we also note that the set of neighbors have a high probability. if we add the amplitudes of the neighbor’s vertices, the values are compatible with the amplitude value of the marked vertex. we conclude that a considerable part of the energy, approximately 1/2, is retained in the neighbors of the marked vertices. figure 2d, shows the probability distribution of four marked vertices. note that the amplitudes of each vertex have their maximum and a neighborhood region. the x-axis distribution is the relative position of the position on the hypercube. it explains why even increasing the number of marked vertices, the success probabilities do not reach values above 1/2. fig. 1. success probability after 100 steps in a hypercube with 1024 nodes. the solid blue curve is the success probability for one solution. the dotted orange curve is the success probability for two solutions. the dot-dashed green curve is the success probability for three solutions. the dotted red curve is the success probability for four solutions. (color figure online) full size image fig. 2. probability distribution of the quantum walk after the number of iterations necessary to reach the maximum value of the probability amplitude with \(n = 10\) and \(n = 1024\) vertices. the y-axis values are at different ranges to improve visualization. (a) solid blue bar show the probability distribution for one marked vertex. (b) solid blue bar and orange dashed bar show the probability distribution for two marked vertices. (c) solid blue bar, orange dashed bar and green dash-dot bar show the probability distribution for three marked vertices. (d) solid blue bar, orange dashed bar, green dash-dot bar and dotted red bar show the probability distribution for four marked vertices. (color figure online) full size image figure 3 shows the success probability for the quantum walk with one and four marked vertices after one hundred steps. figure 3a shows the behavior of the success probability of one marked vertex, the solid blue curve, and its neighbors, which is the dotted orange curve. if a measurement is performed, the probability of getting a neighbor vertex is greater than getting a marked vertex. with probability above \(90\%\), you get the solution or a vertex that is one step away from the solution. figure 3b shows the behavior of the success probability of four marked vertices, the solid blue curve, and their neighbors, the dotted orange curve. note that in a step when the probability of success of the marked vertices is high, the probability of success of the neighbors decreases, and in the next step, when the probability of success of the neighbors is high, the probability of success of the marked vertices decreases. because of this behavior, if a measurement is performed, the probability of getting a neighbor is high. this happens in fig. 3a but more smoothly. observing these results, we must consider the probability p of obtaining a marked vertex and the probability \(p' = (1 p)\) of obtaining an unmarked vertex which is the sum of the probabilities of the \((n k)\) vertices, where k is the number of marked vertices. these results are shown in table 1. note the column of the value of \(p'\), which is composed of the value of the amplitudes of the neighbors and the amplitude of the vertices that are neither neighbors nor marked. the probability of the walker finding a region is high because the energy is concentrated in the neighboring region. it is concluded that the amplification of the neighborhood around the marked vertices interferes with the probability of success of finding a target vertex. table 1. probabilities of success of marked and unmarked vertices.full size table fig. 3. probability of success after 100 steps with \(n = 10\) and \(n = 1024\) vertices. (a) shows the probability of success for one marked vertex and its neighbors. (b) shows the probability of success for four marked vertices and their neighbors. (color figure online) full size image 3.2 adjusting the self-loop weight for multiple marked vertices many works have been proposed with the purpose of improving the search capacity of quantum algorithms. according to wong [23], adding a self-loop to each vertex boosts the success probability from 1/2 to 1. a modification to the initial state in the eq. 3 and to grover’s coin in the eq. 4 is needed so that the self-loop can be added. the addition of the self-loop is described in eq. 7. thus, the coin space is now an \((n + 1)\)-dimensional space [16]. (7) one of the concerns when adding a self-loop at each vertex is knowing the best self-loop value. more specifically, in the case of the quantum walk on the hypercube, rhodes [16] proposed an optimal self-loop value $$\begin{aligned} l = \frac{d}{n}, \end{aligned}$$ (8) where d is equal to the degree of the loopless graph and n is the number of vertices in the hypercube. recently, two works showed that inserting the number of marked vertices in calculating the self-loop value optimizes quantum walks. carvalho [6] shows that the optimal value of the self-loop for quantum walks in d-dimensional grids with multiple marked vertices is $$\begin{aligned} l = \frac{2dm}{n}, \end{aligned}$$ where 2d is the number of movements the walker can do, not counting the self-loop, m the number of marked vertices, and n the number of vertices of the grid. nahimovs [12] shows that for different types of two-dimensional grids triangular, rectangular, and honeycomb the optimal self-loop value is also, $$\begin{aligned} l = \frac{m\cdot d}{n} \end{aligned}$$ where d is the degree of the vertex, m is the number of marked vertices, and n is the number of vertices of the grid. fig. 4. comparison between multiple self-loops values and \(l = (n/n)\). (a) shows the success probability for one marked vertex. (b) shows the success probability for two marked vertices. (c) shows the success probability for three marked vertices. (d) shows the success probability for four marked vertices. (color figure online) full size image figure 4a shows the probability of success after two hundred steps for one marked vertex. here, the values of l were the same as used by rhodes. the dashed red curve has the optimum value of l. our interest was to investigate whether the value of l described in eq. 8 also improved the walk results for a number (\(k>1\)) of marked vertices. for this, we performed three more experiments where we increased the number of marked vertices up to four. as we added the marked vertices the success probability of the dashed red curve decreased to \(88.7\%\) (4b) while the success probability of the dotted purple curve increased to \(99.8\%\) (4b) but then also decreased to \(96.2\%\) (4c) and \(89.3\%\) (4d). it indicates that a new value of l is required when the number of marked vertices increases. to find the optimal self-loop for multiple marked vertices, we defined a set of values in the form \(l' = (\alpha \cdot l)\), where \(\alpha \in \mathbb {n}\). figure 5 compares the probability of success for a set of marked vertices, \(k = \{2, 3, 5, 14, 17\}\), these vertices were chosen randomly as well as their number. the self-loop values for these vertex numbers are \(\alpha \cdot l\), where \(l = (d/n)\) and \(\alpha = \{1, 2, 3,...\}\). note that the curves have their maximum points exactly at the locations on the x-axis where the \(l'\) values are. we can conclude that the value of (\(\alpha = k\)). therefore, we can set the value of l for multiple marked vertices for the quantum walk in the hypercube, $$\begin{aligned} l' = \frac{n}{n} \cdot k \end{aligned}$$ (9) where n is equal to the degree of the loopless vertex of the hypercube, n the number of vertices in the hypercube, and k the number of marked vertices. the self-loop value shown by nahimovs [12] for the quantum search in various types of two-dimensional grids coincides with the optimal self-loop value for the search for a quantum walk in the hypercube. figure 5 shows that, as the values of l approach the optimal value, the probability of success of the curve also approaches its maximum value. we can observe this behavior in table 2 which shows the probability of success for multiple values of l and multiple marked vertices. consider the values of the main diagonal, which are the maximum success probabilities for each \(l = (n/n)\cdot k\). fig. 5. investigation to set the value of l for multiple marked vertices. full size image table 2. probability of success for multiple values of l.full size table table 2 shows the relationship between the self-loop value and the number of marked vertices. we observe the relationship between the self-loop value and the number of marked vertices. note that when the values of l approach the optimal values for each number of marked vertices, there is an improvement in the probability amplitude. figure 6 shows the probability of success after two hundred steps for multiple marked vertices. we can conclude that for cases where there is more than one marked vertex, the optimal value of \(l = (n/n)\cdot k\). fig. 6. probability of success after 200 steps. solid blue curve, \(k = 1\). dotted orange curve, \(k = 2\). green dash-dot curve, \(k = 3\). red dashed curve, \(k = 4\). dotted purple curve, \(k = 5\) (color figure online) full size image 3.3 searching for adjacent marked vertices the results found in the previous sections refer to the search for non-adjacent marked vertices, i.e., \(\left| \vec {\omega _{i}} \vec {\omega _{j}}\right| \ne 1\) the hamming distance from vertex \(\vec {\omega _{i}}\) and all other marked vertices is different from 1. nahimovs et al. [11] shows in their work that for quantum walks in the hypercube if the search space contains marked neighbors vertices, the search can be drastically affected. the authors considered two sets, one with two adjacent marked vertices and the other with two non-adjacent marked vertices. in the first case, the two adjacent marked vertices are \(m = \{0, 1\}\). the absolute value of the overlap remained close to 1, and the probability remains close to the initial state probability. in the second case, the two non-adjacent marked vertices are \(m = \{0, 3\}\). the behavior on this one is different, the same behavior as the solid blue curve in fig. 3a. as the addition of self-loop in the quantum walk in the hypercube improved the search for multiple non-adjacent marked vertices, we investigated the case where the marked vertices are adjacent. we consider ten sets of vertices, \(m = [\{0,1\},\{0,1,2\},\cdots ,\{0,1,2,4,8,\cdots ,256,512\}]\), i.e., all vertices adjacent to the vertex 0. we add one more vertex to the set of marked vertices on each new walk until the number of vertices in m is equal to the degree n of the vertex. figure 7 shows the probability of success after two hundred steps. figure 7a shows the result for the value of \(l = (n/n)\). the probability reaches its maximum when the number of vertices reaches \(k = 4\) with a probability of success of \(99.1\%\). then the probability starts to decrease as k increases. figure 7b shows the result for the value of \(l = (n/n)\cdot k\). the probability reaches its maximum when the number of vertices reaches \(k = 11\) with a success probability of \(94.5\%\). although the probability increases with a slower speed when \(k = 5\), it already reaches \(78.3\%\). this behavior is interesting for search spaces where the marked vertex density is high. note the probability of the solid cyan curve. this behavior was found in work done by nahimovs et al. [11] and was repeated here in our experiments. according to the authors, this is because the quantum walk has a stationary state. fig. 7. probability of success after 200 steps with \(n = 10\) and \(n = 1024\) vertices. shows the probability of success for k adjacent marked vertices. (a) shows for \(l = (n/n)\) and (b) for \(l = (n/n)\cdot k\). full size image figure 8 shows the comparison between what happens to the success probabilities in fig. 7 when the number of k increases. note the dotted orange curve, the probability of success grows to its maximum value when the value of \(l = (n/n)\cdot k\). the same does not happen when \(l = (n/n)\). fig. 8. maximum probability reached for each number of marked vertices in the neighborhood after one hundred steps with \(n = 10\) and \(n = 1024\) vertices. evaluating the interference of the number of adjacent marked vertices in the value of l. (color figure online) full size image we considered before that the marked vertices were neighbors. now, let us analyze the possibility that in addition to having marked vertices in the neighborhood, there are also marked vertices that are not neighbors. we run ten experiments, and each one starts with two adjacent marked vertices \(m = \{0,1\}\). in each experiment, a \(i = \{1,2,3,\cdots \}\) non-adjacent vertex is randomly marked and the next marked neighbor, i.e., \(m = \{0,1,2,...\}\). therefore, in the tenth experiment, there will be eleven adjacent and ten non-adjacent vertices. fig. 9. maximum probability reached for each number of marked vertices after one hundred steps with \(n = 10\) and \(n = 1024\) vertices. (a) shows the probability of success for k adjacent and non-adjacent marked vertices for \(l = (n/n)\). (b) shows the probability of success for k adjacent and non-adjacent marked vertices for \(l = (n/n)\cdot k\). (color figure online) full size image figure 9 shows the behavior of probability amplitudes when for each set of adjacent vertices, a number of non-adjacent vertices are marked. figure 9a shows that as new non-adjacent vertices are marked the probability is affected. note that the behavior seen in the solid blue curve in fig. 8 when there were no non-adjacent vertices is similar, i.e., as the density of the marked vertices increases, the probabilities decrease, even adding the vertices non-adjacent. the same can be seen in the case of the dotted orange curves in figs. 8 and 9b, i.e., when the density of the marked vertices increases, the probability also increases, this tells us that the value of \(l = (n/n)\cdot k\) is optimal for high marked vertex densities. figure 10 shows the probability of success for the search of marked adjacent and non-adjacent vertices in the search space. we performed an experiment, where, at every hundred steps, an adjacent vertex and a non-adjacent vertex were marked, i.e., for each m set of adjacent vertices a vertex \(i \notin m\) was marked randomly, then, \(m' = \{0,1,i_{0}\},\{0,1,i_{0},2,i_{1}\},\cdots ,\{0,1,i_{0},2,i_{1},4, i_{2},\cdots ,512,i_{10}\}\). figure 10a shows the probability of success for \(l = (n/n)\) and fig. 10b shows the probability of success for \(l = (n/n) \cdot k\). note that the probability of success above \(90\%\) is achieved in a smaller number of steps. fig. 10. probability of success after 100 steps with \(n = 10\) and \(n = 1024\) vertices. (a) shows the probability of success for k adjacent and non-adjacent marked vertices for \(l = (n/n)\). (b) shows the probability of success for k adjacent and non-adjacent marked vertices for \(l = (n/n)\cdot k\). (color figure online) full size image 4 conclusions many efforts are applied in order to improve the performance of quantum search algorithms. quantum walks are the main tool for building these algorithms. we initially analyzed the quantum walk in the hypercube applying grover’s search and came to the conclusion that neighbor vertices affect the search performance, an observation that has been corroborated by other authors. we found that the walk could not improve its results even for a number of marked vertices equal to one. many authors have developed works for adding self-loops in various types of graphs and grids of different dimensions. in this sense, we decided to investigate how to improve the quantum search in the hypercube using self-loops. previous works defined the optimal self-loop value as \(l = (d/n)\) for one marked vertex to the quantum walk on the hypercube. after performing experiments we saw that this value of l was not optimal for multiple marked vertices. we arrive at a value of \(l = (n/n)\cdot k\) for an arbitrary number of vertices. this value is also used when searching in two-dimensional grids. another aspect of the quantum walk in the hypercube is whether the marked vertex is adjacent or not, this interferes with the search performance. we then analyzed whether the value of \(l = (n/n)\) and \(l = (n/n)\cdot k\) had any positive effect when applied to the hypercube vertices. the results show that the value of \(l = (n/n)\) is not optimal for the quantum walk in the hypercube with multiple marked vertices adjacent or not. it also shows that for a search space where there are marked adjacent vertices, just one non-adjacent marked vertex is sufficient for the value of \(l = (n/n)\cdot k\) to be better. according to the results presented here, there is a greater than \(90\%\) probability that the measurement will collapse in one of the solutions. recent works have used the quantum walks to search weights and train artificial neural networks [19, 20]. the quantum walk in the hypercube has an interesting behavior, the amplification of neighbors vertices. in future work, we intend to use this quantum walk to find a set of weights to initialize and train classical artificial neural networks. we also intend to analyze the quantum walk in the hypercube with multiple weighted self-loops. references aharonov, y., davidovich, l., zagury, n.: quantum random walks. phys. rev. a 48(2), 1687 (1993) article  google scholar  ambainis, a., bačkurs, a., nahimovs, n., ozols, r., rivosh, a.: search by quantum walks on two-dimensional grid without amplitude amplification. in: iwama, k., kawano, y., murao, m. (eds.) tqc 2012. lncs, vol. 7582, pp. 87–97. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-35656-8_7 chapter  math  google scholar  ambainis, a., kempe, j., rivosh, a.: coins make quantum walks faster. arxiv preprint quant-ph/0402107 (2004) google scholar  bezerra, g., lugão, p., portugal, r.: quantum walk-based search algorithms with multiple marked vertices. phys. rev. a 103(6), 062202 (2021) article  mathscinet  google scholar  de carvalho, j.h.a., de souza, l.s., de paula neto, f.m., ferreira, t.a.e.: impacts of multiple solutions on the lackadaisical quantum walk search algorithm. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 122–135. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_9 chapter  google scholar  carvalho, j.h.a., souza, l.s., paula neto, f.m., ferreira, t.a.e.: on applying the lackadaisical quantum walk algorithm to search for multiple solutions on grids. arxiv preprint quant-ph/2106.06274 (2021) google scholar  kempe, j.: quantum random walks hit exponentially faster. arxiv preprint quant-ph/0205083 (2002) google scholar  mcmahon, d.: quantum computing explained. wiley, new york (2007) google scholar  moore, c., russell, a.: quantum walks on the hypercube. in: rolim, j.d.p., vadhan, s. (eds.) random 2002. lncs, vol. 2483, pp. 164–178. springer, heidelberg (2002). https://doi.org/10.1007/3-540-45726-7_14 chapter  google scholar  nahimovs, n.: lackadaisical quantum walks with multiple marked vertices. in: catania, b., královič, r., nawrocki, j., pighizzini, g. (eds.) sofsem 2019. lncs, vol. 11376, pp. 368–378. springer, cham (2019). https://doi.org/10.1007/978-3-030-10801-4_29 chapter  google scholar  nahimovs, n., santos, r.a.m., khadiev, k.r.: adjacent vertices can be hard to find by quantum walks. mosc. univ. comput. math. cybern. 43(1), 32–39 (2019) article  mathscinet  google scholar  nahimovs, n., santos, r.a.: lackadaisical quantum walks on 2d grids with multiple marked vertices. arxiv preprint arxiv:2104.09955 (2021) nielsen, m.a., chuang, i.: quantum computation and quantum information. aapt, cambridge (2002) google scholar  potoček, v., gábris, a., kiss, t., jex, i.: optimized quantum random-walk search algorithms on the hypercube. phys. rev. a 79(1), 012325 (2009) article  google scholar  rhodes, m.l., wong, t.g.: quantum walk search on the complete bipartite graph. phys. rev. a 99(3), 032301 (2019) article  google scholar  rhodes, m.l., wong, t.g.: search on vertex-transitive graphs by lackadaisical quantum walk. quantum inf. process. 19(9), 1–16 (2020) article  mathscinet  google scholar  shenvi, n., kempe, j., whaley, k.b.: quantum random-walk search algorithm. phys. rev. a 67(5), 052307 (2003) article  google scholar  singh, j., singh, m.: evolution in quantum computing. in: 2016 international conference system modeling & advancement in research trends (smart), pp. 267–270. ieee (2016) google scholar  souza, l.s., carvalho, j.h.a., ferreira, t.a.e.: quantum walk to train a classical artificial neural network. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 836–841. ieee (2019) google scholar  souza, l.s., carvalho, j.h.a., ferreira, t.a.e.: classical artificial neural network training using quantum walks as a search procedure. ieee trans. comput. (2021) google scholar  venegas-andraca, s.e.: quantum walks: a comprehensive review. quantum inf. process. 11(5), 1015–1106 (2012) article  mathscinet  google scholar  wang, h., zhou, j., wu, j., yi, x.: adjustable self-loop on discrete-time quantum walk and its application in spatial search. arxiv preprint arxiv:1707.00601 (2017) wong, t.g.: grover search with lackadaisical quantum walks. j. phys. a: math. theor. 48(43), 435304 (2015) article  mathscinet  google scholar  wong, t.g.: faster search by lackadaisical quantum walk. quantum inf. process. 17(3), 1–9 (2018) article  mathscinet  google scholar  yanofsky, n.s., mannucci, m.a.: quantum computing for computer scientists. cambridge university press, cambridge (2008) google scholar  download references author information authors and affiliations departamento de estatística e informática, universidade federal rural de pernambuco, recife, brazil luciano s. de souza & tiago a. e. ferreira centro de informática, universidade federal de pernambuco, recife, brazil jonathan h. a. de carvalho authors luciano s. de souzaview author publications search author on:pubmed google scholar jonathan h. a. de carvalhoview author publications search author on:pubmed google scholar tiago a. e. ferreiraview author publications search author on:pubmed google scholar corresponding author correspondence to luciano s. de souza . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de souza, l.s., de carvalho, j.h.a., ferreira, t.a.e. (2021). lackadaisical quantum walk in the hypercube to search for multiple marked vertices. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_17 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords quantum computing quantum walk quantum search algorithm publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature emotion recognition in instrumental music using ai | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper emotion recognition in instrumental music using ai conference paper first online: 30 january 2025 pp 201–213 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) emotion recognition in instrumental music using ai download book pdf download book epub camila ferreira alves9, thiago garcia mozart9 & luis antônio brasil kowada9  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 401 accesses 1 citation abstract this study aims to develop a predictive model for classifying emotions in instrumental music excerpts using machine learning techniques. the methodology involves experiments with three distinct datasets, including a newly created dataset for this research. the models employed include random forest, multi-layer perceptron, and convolutional neural network architectures. the experimental results, conducted with a single dataset for testing and validation, were generally positive. however, when attempting to generalize the models using different datasets, a considerable reduction in generalization capability was observed. despite this, the study presents promising results, indicating that increasing the number of training data can significantly improve the models’ generalization capability. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others music emotion recognition with the extraction of audio features using machine learning approaches chapter © 2020 end-to-end music emotion variation detection using iteratively reconstructed deep features article 08 january 2022 a survey on music emotion recognition using learning models article 04 june 2025 1 introduction music plays a fundamental role in daily life, often used to express emotions and increasingly in therapeutic treatments. it has the ability to evoke a wide range of emotions, acting as a means of communication that transcends linguistic barriers. however, identifying and classifying the emotions evoked by music presents significant challenges due to the variations in individual listeners’ perceptions, resulting in a substantial degree of disagreement among them [9]. this study aims to develop a sentiment analyzer for music using concepts of artificial intelligence (ai) and machine learning (ml). the analysis of instrumental music, in particular, presents unique challenges, as the absence of lyrics requires emotional interpretation to be derived exclusively from musical elements such as melody, harmony, rhythm, and timbre. this emotional complexity in purely instrumental compositions is often neglected by approaches that focus solely on song lyrics, as noted in previous studies [4, 6]. specifically, this study seeks to develop a predictive model capable of classifying emotions in instrumental music using advanced machine learning techniques. the methodology adopted involves conducting various experiments with three distinct datasets, including a proprietary dataset created specifically for this research. the machine learning models employed include random forest, multi-layer perceptron, and convolutional neural networks, each selected for their distinct capabilities in modeling and data analysis. the datasets used were labeled based on five primary emotions: happiness, sadness, dramaticism, romanticism, and aggressiveness. one of the datasets was developed for exclusive use in this work. 2 methodology the adopted methodology was structured into fundamental sequential steps, with datasets defined and emotional labels standardized into five categories: happiness, sadness, dramaticness, romanticism, and aggressiveness. the process begins with data treatment, followed by a methodological bifurcation depending on whether deep learning techniques are used. for approaches involving deep learning, spectrograms are created, followed by the application of transfer learning and optimization of parameters, culminating in the training, testing, and generalization phases of the model. alternatively, in the absence of deep learning, feature extraction is performed, followed by verification of dataset balance; in case of imbalance, the smote algorithm is applied. subsequently, the optimal parameters for training are determined, followed by the testing and generalization stages. fig. 1. ai development full size image the general flow of the methodology included the steps presented in fig. 1. 2.1 data acquisition the first dataset [1] consists of 2500 musical excerpts, each 5 s long. these excerpts are labeled with five distinct emotions: happiness, sadness, romanticism, aggressiveness, and dramaticness, so no adaptation was needed. the dataset is originally organized uniformly, with 500 excerpts for each class. the second dataset [2] consists of 400 musical excerpts, initially distributed across four genres (rock, classical, pop, and electronic). however, for this study, only 322 excerpts, each 10 s long, were used. this dataset was originally labeled with nine emotions: amazement, solemnity, tenderness, nostalgia, calm, power, activation, tension, and sadness. to adapt these emotions to the context of this work, the emotion wheel [7] was used as a reference, mapping each emotion to the closest ones. the third dataset was developed using a web platform specially designed for this study. this dataset comprises a total of 371 music excerpts, each with a duration of 10 s. the labeling of the excerpts was conducted through evaluations provided by 164 users, ensuring a diversity of opinions and perspectives. during the dataset creation, ties in the labeling were identified. to address these ties impartially and ensure each excerpt maintained a single label, one of the labels was randomly selected. 2.2 data processing in the second audio dataset, where each track had an average of 45 evaluations by different users, several preprocessing steps were necessary to prepare the data for analysis. first, the csv file containing the evaluations was read, and the final annotations were determined by the mode of each music track in the dataset. additionally, tracks containing vocals were removed, as lyrics could distort the ai’s understanding of the musical characteristics being analyzed. thus, the focus remained on working only with the instrumental aspects of the tracks. all collected data were converted to “.wav” format to ensure uniformity in file type and facilitate subsequent audio processing steps. audio normalization was also performed, adjusting the volume of each track to avoid significant disparities between different audios. noise removal was also executed, ensuring the audio was as clean as possible without interference that could affect the subsequent analysis. 2.3 feature extraction in the approach used for random forest and multi-layer perceptron techniques, an average calculation strategy for specific features was adopted to ensure equity between the different audio excerpts in the datasets while maintaining code consistency. to select the best features, a permutation between features extracted from an audio excerpt was performed. all tested features are previously described in the librosa library documentationfootnote 1. the criterion used for selecting the best features was based on the highest average accuracy in training the three ai models. thus, a unique set of features was standardized for the trained models. the adopted approach brought benefits in comparing and interpreting results between different datasets and contributed to maintaining code integrity, allowing efficient implementation without the need for specific changes for each dataset throughout the development. the features considered are described below. rms energy: represents the average energy of the audio signal over time. it is also a practical measure of the energy of vibration. time and rhythm: represented by the beats per minute (bpm) rate. its objective is to provide information about the speed and rhythm of the music, commonly used for identifying musical genres and analyzing rhythmic patterns. the calculation is based on the root mean square (rms) value for each frame from the audio samples. mel spectrogram: a visual representation of the audio signal spectrum in mel frequency bands, which is a logarithmic transformation applied to the frequency domain components. the goal is to represent the sound components, giving greater importance to those that most sensitize the human ear [5]. mel-frequency cepstral coefficients (mfccs): mfccs are based on extracting the signal’s energy within critical frequency bands. cepstral measures, based on the mel scale, indicate the spectrum’s shape, obtaining a response with logarithmic (non-linear) approximation concerning frequency, simulating the human ear [8]. zero-crossing rate: the zcr describes the dominant frequency of the music and is often used to detect silence frames (segments). spectral centroid: the spectral centroid characterizes the signal spectrum. it indicates where most of the energy is concentrated in terms of frequency, seeking an average frequency of the signal weighted by the amplitude of each spectral component. 2.4 smote the synthetic minority over-sampling technique (smote) is widely used to address class imbalance in datasets by creating synthetic examples for the minority class. smote starts with the random selection of a minority class sample and then selects the k-nearest neighbors of that sample. among these neighbors, one is randomly chosen, and a synthetic data point is generated between the chosen neighbor and the initial sample. this process is repeated until the dataset is balanced [3]. the application of smote in this work can be observed in fig. 2. in this figure, it is noted that the class with the fewest labels, “aggressiveness,” after applying the technique, has the same amount of data as the class with the most labels, “happiness.” this balancing is essential to ensure that the machine learning algorithms are not biased in favor of the majority classes, thus improving the model’s ability to generalize to new data. fig. 2. smote technique full size image 2.5 mel spectrogram in the approach used for convolutional neural networks (cnns), the original audio signal was transformed into a mel spectrogram, allowing the model to explore temporal and frequency patterns more intuitively and accurately, similar to image analysis. the images were generated using version 1.0.2 of the librosa library. the main difference between the mel spectrogram and the spectrogram is that the frequencies are converted to the mel scale. the goal is to reduce the amount of correlated information without losing relevant information [5]. the conversion to the mel scale is given by the equation: $$\begin{aligned} \text {mel}(f) = 2595 \log _{10} \left( 1 + \frac{f}{700} \right) \end{aligned}$$ (1) the different types of mel spectrograms generated in this work are presented in fig. 3. the figure shows five randomly selected audio excerpts, representing the five emotional classes used in this study: happiness, sadness, aggressiveness, romanticism, and dramaticness. fig. 3. mel spectrograms full size image 2.6 model training the scikit-learn libraryfootnote 2 was used for model training. the training process included splitting the data for training and validation and choosing hyperparameters, ensuring that the validation data were entirely new to the model. the data were split into 80% for training and 20% for testing, ensuring the model was tested on unseen data. this split was done randomly to avoid any bias in the resulting sets. the selection of hyperparameters for each model was based on methods like random search, aiming to explore a wide range of possible combinations efficiently, resulting in a configuration that maximized model quality. hyperparameter combinations were performed using cross-validation, where the data were divided into multiple subsets (folds). the model was trained on some of these subsets and tested on the remaining ones, ensuring all data were used for validation at some point. this approach is particularly useful for relatively small datasets, as it allows the model to be evaluated more comprehensively using all available data for both training and validation. tables 1, 2, and 3 show the hyperparameters explored for training the random forest, multi-layer perceptron, and convolutional neural network models, respectively. table 1. random forest hyperparameters descriptionfull size table table 2. multi-layer perceptron hyperparameters descriptionfull size table table 3. convolutional neural network hyperparameters descriptionfull size table 3 experiments given the diversity of data types, combinations between models and datasets were conducted, resulting in a total of nine distinct experiments as described in table 4. table 4. summary of experimentsfull size table in each experiment, a search for the best parameters was conducted, and the model was trained with the data previously separated for the training set. subsequently, the model’s performance was evaluated by analyzing the confusion matrix [10] constructed with the predictions resulting from the data reserved for the test set. crucial metrics such as precision, recall, and f1-score were considered for this evaluation. 4 results for each of the nine experiments described in sect. 3, the resulting metrics are presented in table 5. these results can be interpreted as the averages generated from the values of each labeling. table 5. experiment resultsfull size table 4.1 prediction on other datasets similarly to the tests, the prediction sets were distinct from those used in each experiment, ensuring that all data were entirely new to the models, not being used in the training or validation sets. for each experiment, two prediction sets that had already been used in the study were employed. table 6 focuses on showing the accuracy of each of the proposed validations. table 6. validationfull size table 4.2 additional experiments for cases where the validation and training datasets are the same, they were treated separately. for training and testing, 70% of the dataset was randomly split. the remaining 30% of the dataset was used for validation. similar to previous experiments, for each dataset, the three presented models were implemented. thus, experiments with identifiers 10 and 11 relate to the rf model, while identifiers 12 and 13 were trained with the mlp model, and the others with cnn. all experiments follow the order 1, 2, and 3 regarding the datasets. the results can be seen in table 7. table 7. additional experimentsfull size table 5 model performance analyzing table 5, we can observe that the performance of the models varies according to the dataset and the model used. the multi-layer perceptron (mlp) model consistently demonstrated superior performance in most experiments. however, when considering generalization capability, the convolutional neural network (cnn) model proved to be more robust in some cases. the validation results presented in table 6 show that the generalization of the models was inferior when tested on different datasets from those on which they were trained. this reinforces the idea that the generalization capability of the models was compromised due to the intrinsic differences between the datasets. the number of samples had a significant influence on the models’ performance. models trained with dataset 1, which has a larger number of samples (2500), consistently outperformed those trained with datasets 2 and 3, which have fewer samples (322 and 371, respectively). this result suggests that increasing the amount of available data can improve the models’ generalization capability. the results of additional experiments, where the datasets were split into 70% for training and 30% for validation, corroborate this observation. models trained with dataset 1 performed better in validation compared to those trained with the other datasets. differences in the properties of the datasets, such as the source of annotation and the intrinsic variability of the data, also affected the models’ performance. the discrepancy in validation results suggests that models trained on a specific dataset may not be applicable to other datasets with different characteristics. this highlights the importance of using a diverse and comprehensive dataset for training. 6 conclusion this study aimed to develop a predictive model capable of classifying emotions in excerpts of instrumental music using machine learning techniques. the adopted methodology involved conducting experiments with three distinct datasets, including one created specifically for this research. the models used were random forest, multi-layer perceptron, and convolutional neural network. the experimental results were generally positive when validated with a single dataset for testing and validation. however, there was a considerable reduction in the models’ generalization capability when using datasets different from the original ones. this indicates the need for improvements to increase the models’ generalization and robustness. the results indicate that, although machine learning models can capture useful patterns to predict emotions in musical excerpts, there is a clear need for improvements. increasing the diversity and comprehensiveness of the training datasets, as well as applying more advanced preprocessing and feature engineering techniques, are essential steps for future research in this area. in summary, this work presents promising results and points to directions for future improvements, indicating that increasing the amount of training data and using more advanced techniques can significantly improve the generalization capability of machine learning models in classifying emotions in instrumental music. notes 1.https://librosa.org. 2.https://scikit-learn.org/. references akhmedsakip: music-berkeley-emotions. https://huggingface.co/datasets/akhmedsakip/music-berkeley-emotions aljanaki, a., wiering, f., veltkamp, r.c.: studying emotion induced by music through a crowdsourcing game. inf. process. manag. 52(1), 115–128 (2016). https://doi.org/10.1016/j.ipm.2015.03.004, https://www.sciencedirect.com/science/article/pii/s0306457315000424, emotion and sentiment in social and expressive media apostolopoulos, i.d.: investigating the synthetic minority class oversampling technique (smote) on an imbalanced cardiovascular disease (cvd) dataset (2020) google scholar  du, j.: sentiment analysis and lyrics theme recognition of music lyrics based on natural language processing. vol. 20 (2024). http://journal.esrgroups.org/index.php/jes/article/view/2040 lacerda, t.b., miranda, p., câmara, a., furtado, a.p.c.: deep learning and mel-spectrograms for physical violence detection in audio. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 268–279. sbc (2021) google scholar  malheiro, r., panda, r., gomes, p., paiva, r.p.: classification and regression of music lyrics: emotionally-significant features. in: international conference on knowledge discovery and information retrieval. vol. 2, pp. 45–55. scitepress (2016) google scholar  plutchik, r.: emotions and life: perspectives from psychology, biology, and evolution. american psychological association (2003) google scholar  rahmani, a.m., et al.: enhanced classification of heart sounds using mel frequency cepstral coefficients: a comparative study of single and ensemble classifier strategies (2024) google scholar  schmidt, e.m., scott, j.j., kim, y.e.: feature learning in dynamic environments: modeling the acoustic structure of musical emotion. in: ismir, pp. 325–330 (2012) google scholar  ting, k.m.: confusion matrix, p. 209. springer us, boston, ma (2010). https://doi.org/10.1007/978-0-387-30164-8_157 download references acknowledgements we would like to express our sincere gratitude to the tools and libraries used for data analysis and model development. we also acknowledge the valuable assistance of chatgpt, which was used as a support tool for the initial review of this manuscript. we take full responsibility for the content presented, including the verification of originality and text accuracy. this work has been supported by the faperj (260003/015313/2021). author information authors and affiliations universidade federal fluminense – instituto de computação, niterói, brazil camila ferreira alves, thiago garcia mozart & luis antônio brasil kowada authors camila ferreira alvesview author publications search author on:pubmed google scholar thiago garcia mozartview author publications search author on:pubmed google scholar luis antônio brasil kowadaview author publications search author on:pubmed google scholar corresponding author correspondence to camila ferreira alves . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alves, c.f., mozart, t.g., kowada, l.a.b. (2025). emotion recognition in instrumental music using ai. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_14 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning emotion recognition emotions music analysis publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving colorectal cancer diagnosis using mirnet and inceptionv3 on histopathological images | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving colorectal cancer diagnosis using mirnet and inceptionv3 on histopathological images conference paper first online: 30 january 2025 pp 321–334 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) improving colorectal cancer diagnosis using mirnet and inceptionv3 on histopathological images download book pdf download book epub neilson p. ribeiro  orcid: orcid.org/0009-0004-0431-94389,10, felipe r. s. teles  orcid: orcid.org/0009-0002-9598-885010, joão otávio bandeira diniz  orcid: orcid.org/0000-0003-3303-33469,10, luana b. da cruz11, domingos a. dias jr.11, geraldo braz junior  orcid: orcid.org/0000-0003-3731-643110, joão d. s. de almeida  orcid: orcid.org/0000-0001-7013-970010 & … anselmo c. de paiva  orcid: orcid.org/0000-0003-4921-062610  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 389 accesses 1 citation abstract colorectal cancer (crc) is the second most prevalent type of cancer among men and women in brazil, accounting for 9.1% of cancers in men and 9.2% in women between 2020 and 2022. crc diagnosis typically relies on histopathological images, which can be challenging to interpret and prone to errors. this paper proposes a method for diagnosing crc using histopathological images. this method involves enhancing images with mirnet and classifying them as benign or malignant using inceptionv3. the results surpass the state-of-the-art by introducing, for the first time, enhancement through mirnet, achieving an accuracy of 99.67% and an auc of 0.9999. we believe that this method, combined with medical expertise, holds great promise for early crc diagnosis. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others histopathological colorectal cancer image classification by using inception v4 cnn model chapter © 2023 colorectal cancer detection using an ensemble model for histopathological image classification chapter © 2026 prognostic analysis combining histopathological features and clinical information to predict colorectal cancer survival from whole-slide images article 05 june 2024 1 introduction cancer encompasses diseases in which abnormal cells develop in the body due to random mutations. among these conditions, colorectal cancer (crc) is a significant contributor to morbidity and mortality [3]. in brazil, approximately 45,630 new cases of crc are estimated annually, with a higher incidence in men compared to women [21]. globally, crc accounted for over 1.9 million new cases in 2020, ranking it as the third most incident tumor among all cancers. the highest incidence rates of crc were observed in men in central, northern, and southern europe, while among women, the highest incidences were found in oceania and northern europe [3]. colorectal cancer stands out as one of the most prevalent cancers in brazil and worldwide, emphasizing the necessity to address detection and diagnostic methodologies that facilitate early identification [7]. currently, specialists rely on visual analyses of histopathological image samples, a subjective process dependent on personal interpretation and pathologist experience [1]. the development of computational tools to support specialists and provide a second opinion is crucial for aiding in early diagnosis. artificial intelligence (ai) has made significant advancements, offering various viable diagnostic approaches [12, 13, 15, 17, 19, 23]. once trained, it can assist specialists in quickly and effectively diagnosing crc. early diagnosis plays a fundamental role in treatment and patient survival. with recent advances in image processing and machine learning, ai techniques have shown promise in assisting in crc diagnosis. therefore, this study presents a method for diagnosing crc histopathological images using convolutional neural networks (cnns) for image enhancement and classification between benign and malignant. thus, we believe that the proposed method brings the following contributions: introduces the use of the mirnet cnn, for the first time in the literature, to enhance the quality of histopathological images, improving diagnostic accuracy; utilizes another cnn, the inceptionv3, to classify the images pre-processed by mirnet, achieving metrics superior to the literature. thus, the contributions mentioned suggest that the proposed method is an effective tool to assist specialists in an early diagnosis of crc cancer, potentially playing a critical role in effective treatment and patient survival. 2 related works colorectal cancer has sparked extensive research into the development of computational methods to aid specialists in its classification. in this section, related works on the topic are presented. dabass et al. [8] highlight the importance of computational methods for crc classification. they propose a supervised deep learning technique for five-grade cancer classification, achieving an accuracy of 96.97% for two-grade and 93.24% for five-grade classification. candelero et al. [5] propose an automated pathology detection method, focusing on crc, breast cancer, and non-hodgkin lymphomas. their method combines feature extraction techniques such as multi-scale and multidimensional fractals, haralick descriptors, and cnn for pattern recognition. following feature selection, evolutionary and machine learning algorithms, including svm, k*, and random forest, were applied for classification. the best combination resulted in an accuracy of 91.06% for crc. mangal et al. [20] highlight the importance of accurately categorizing histopathological images of the colon. using the same dataset of our work for validation, they achieved an impressive accuracy of 96.61% with a cnn architecture they proposed. on the other hand, bukhari et al. [4] emphasize the precise histopathological diagnosis of colorectal cancer. their approach involves utilizing images and cnns. by training on ten thousand images and comparing three different cnns (resnet-18, resnet-30, and resnet-50), they identified resnet-50 as the most accurate, achieving an outstanding 93.91% accuracy. hidayah et al. [18] introduce a conventional approach to classify colorectal cancer, employing texture analysis with glrlm and k-fold cross-validation. this research attains an accuracy rate of 85.57%, a sensitivity of 91.72%, and a specificity of 80.55%. all two studies mentioned utilize the identical dataset as the proposed method. on the other hand, swarna et al. [24] present two models for colon cell classification based on image data one utilizing the inceptionv3 model, and the other combining predictions from three basic sequential cnn models. the authors achieve accuracy rates of 99.4% and 99.95%, respectively. in a recent study conducted by rajinikanth et al. [22], they introduced a new deep learning framework designed to categorize colon slides into either normal or cancerous classes. this methodology encompasses several steps, including resizing, preprocessing, extraction of deep features, and binary classification using 5-fold cross-validation. the research dataset comprises 4,000 images, evenly split between normal and cancerous samples. their findings reveal that employing deep features leads to a remarkable 99% accuracy rate. table 1 describes the related works presented, along with the techniques and accuracy values. table 1. summary of techniques and accuracy resultsfull size table we notice that in most studies, cnn usage stands out for crc classification, displaying significant metrics, often surpassing 90%. moreover, feature extraction, whether traditional or deep features, is also commonly employed for this purpose. however, there is a lack of emphasis on image enhancement. enhancing images is crucial to highlight important features and optimize their representation for subsequent steps, including feature extraction or cnn utilization. addressing this gap, the proposed method introduces the use of a cnn called mirnet for the automatic enhancement of crc histopathological images. furthermore, considering the results achieved in the literature, we propose utilizing inceptionv3 for classification after applying mirnet [26], as it was the cnn that exhibited the best performance [24]. 3 materials and method in this section, we describe the steps that compound the proposed method. the proposed method is divided into four steps: database acquisition; database pre-processing using mirnet; inceptionv3 as a classifier between benign and malignant classes; and finally, a validation step to evaluate the proposed method. figure 1 presents the mentioned steps. fig. 1. database: (a) benign colon tissue; (b) colon adenocarcinoma (malign). full size image 3.1 database for this study, we turned to the extensive database available on kaggle, titled lung and colon cancer histopathological images. this dataset, a rich and diverse source, comprises five distinct classes, each containing 5,000 images. the categories include benign lung tissue, lung adenocarcinoma, lung squamous cell carcinoma, colon adenocarcinoma, and benign colon tissue [2]. for our method development, we concentrated on a database comprising 10,000 images associated with colon adenocarcinoma and benign colon tissue. our analysis aims to discern the distinguishing features between benign and malignant images utilizing deep learning-based enhancement techniques. figure 2 offers a visual representation of a sample from each class. fig. 2. database: (a) benign colon tissue; (b) colon adenocarcinoma (malign). full size image additionally, the images have dimensions of \(768\times 768\) pixels. thus, for the subsequent steps of the method, resizing to \(256\times 256\) was performed. according to [10, 16], this size is suitable for processing within the deep learning pipeline. 3.2 mirnet pre-processing many works show that image preprocessing can make the results better [9, 11, 14]. in this step, the images are processed by mirnet to assess the enhancement of the database. mirnet (multiple iterative reconstruction network) is a deep neural network designed to perform image enhancement. it operates in multiple iterations, progressively refining the quality of the input images [26]. we highlight that it is the first time the mirnet has been used in histopathological images of crc as image pre-processing. thus, image pre-processing techniques play a crucial role in improving results. it is not an easy task to select a set of the best pre-processing to be applied. this way, employing an ai-based technique that automatically enhances the image features increases the method’s performance. figure 3 illustrates the architecture of mirnet. fig. 3. mirnet pre-processing. architecture sample. full size image according to [26], mirnet follows an iterative approach, where each iteration enhances the input image by learning hierarchical representations of image features. it reconstructs the image using learned residual information, which are differences between the input and reconstructed images. these residuals are used to refine the input image in each iteration, resulting in changes in texture patterns. various configurations were tested to validate the method, as discussed in sect. 4. as shown in fig. 2, histopathological images of crc often exhibit variations in quality and appearance, which can hinder the accurate classification of tissue. therefore, by opting to use mirnet as preprocessing, it can highlight important features of crc tissue that may not be easily discernible, such as morphological, structural, and textural characteristics relevant for distinguishing benign from malignant tissue. 3.3 inceptionv3 training with the images pre-processed by mirnet, the next step is to classify them as malignant or benign. for this purpose, a cnn will be trained. firstly, the data is split into training, validation, and test sets. it is worth noting that one of the characteristics of cnns is the need for a large number of training samples to effectively learn the features during training. at this step, data augmentation techniques were applied to increase the size of the training dataset. the techniques used included random rotation from -30 to 30\(^\circ \)c, width_shift_range and height_shift_range also from -30 to 30\(^\circ \)c and horizontal and vertical flipping operations. these operations were performed using the keras library [6]. for training the cnn, the inceptionv3 was chosen. upon reviewing the literature analysis (sect. 2), this was the architecture that yielded the best results in crc analysis. moreover, inceptionv3 is capable of extracting complex and discriminative features from histopathological images of crc. its efficient architecture, which utilizes parallel convolutional modules called inception modules, allows for a richer and more efficient representation of image features [25]. furthermore, the transfer learning technique can be easily applied to inceptionv3 by leveraging its pre-trained weights and fine-tuning them to fit the specific data of the crc classification problem. therefore, the weights from imagenet were utilized. by adjusting these weights to fit the specific data of the crc classification problem, the network can learn more specialized and discriminative representations to distinguish between benign and malignant tissue. thus, training is performed by inceptionv3 with data augmentation and transfer learning from imagenet, and validated on the validation set to find the best model. finally, the test data is presented to the model, and validation metrics are extracted. 3.4 validation metrics in computational analysis of histopathological images, accurate classification of samples between benign and malignant plays a fundamental role in aiding clinical diagnosis. therefore, to evaluate the proposed method’s performance metrics frequently applied in the area of medical imaging and literature were used: accuracy, precision, sensitivity, specificity, f1-score, and area under the roc curve (auc). calculate the metrics, it was based on the confusion matrix, which takes into account four variables: tp denotes true positives, tn represents true negatives, fp signifies false positives, and fn indicates false negatives. $$ \text {accuracy} = \frac{\text {tp} + \text {tn}}{\text {tp} + \text {tn} + \text {fp} + \text {fn}}, $$ $$ \text {precision} = \frac{\text {tp}}{\text {tp} + \text {fp}}, $$ $$ \text {recall/sensitivity} = \frac{\text {tp}}{\text {tp} + \text {fn}}, $$ $$ \text {specificity} = \frac{\text {tn}}{\text {tn} + \text {fp}}, $$ $$ \text {f1-score} = 2 \times \frac{\text {precision} \times \text {recall}}{\text {precision} + \text {recall}}, $$ $$ \text {auc} = \int _{0}^{1} \text {tpr}(fpr) \, d(fpr) $$ where tpr (true positive rate) represents the rate of true positives; fpr (false positive rate) represents the rate of false positives. the integral is calculated from 0 to 1, representing the area under the roc curve. 4 results and discussion in this section, we present the results obtained through the application of the proposed method. firstly, we describe the training environment used. then, we detail the results achieved by the proposed method. subsequently, a comparison with the literature and case studies is conducted. finally, the advantages and limitations of the method are discussed. 4.1 training environment to validate the proposed work, the following hardware resources were used: an intel® core™ i7 cpu 2.30ghz processor, an nvidia® rtx-3070 8gb gpu, 16gb of ram, and the windows 11 pro operating system. tensorflow and the keras library were employed to build and analyze the proposed method. tensorflow is a widely used library for developing cnns and serves as the backend for keras. additionally, opencv was used for image manipulation, and scikit-learn to assist in various stages of the method, as well as to apply validation metrics. 4.2 experiments in the experiments, we sought to validate each step of the proposed method. initially, the dataset was divided into training (80%) and testing (20%) sets, with a validation subset comprising 20% of the training set, as described in table 2. table 2. distribution of images in training, validation, and test sets.full size table next, pre-processing was applied using mirnet. it is important to highlight that all hyperparameters used in mirnet were the same as those described by [26]. for this purpose, four datasets were trained with 50, 100, 200, and 500 epochs, respectively. figure 4 illustrates the result of some images for each of these sets. fig. 4. mirnet pre-processing result per epochs. full size image we observed that for each dataset, different aspects of the images are enhanced, varying according to the number of epochs. visually, the dataset trained with 50 epochs presents a visually harmonized result, while those trained with 100, 200, and 500 epochs lost some characteristics. this suggests the need for a more in-depth analysis of the metrics to verify if there was indeed an improvement in the pre-processing quality. subsequently, each dataset is submitted to inceptionv3. the hyperparameters used were those presented by [25], available in the keras library. in addition, data augmentation and transfer learning steps were included, as described in sect. 3.3. the models were trained for 50 epochs, with a batch size of 16, using the adam optimizer and binary cross-entropy as loss. table 3 presents the results for training inceptionv3 for each dataset and the original images. table 3. results from mirnet pre-processed datasetsfull size table the analysis of the metrics reveals a significant improvement in the model’s performance after pre-processing with mirnet compared to the original images. in percentage terms, we observe a considerable increase in all metrics across all datasets processed by mirnet. when comparing the mirnet datasets, we note that those trained with 50 and 100 epochs exhibit the best metrics. specifically, the mirnet dataset with 50 epochs stands out as the best, with 99.67% accuracy, 99.73% precision, 99.73% sensitivity, 99.70% specificity, 99.67% f1-score, and 0.9999 auc. these results suggest that a shorter training of 50 epochs may be sufficient to achieve excellent performance. these results underscore its significance in crc diagnosis. the substantial improvement in performance metrics after pre-processing with mirnet indicates that this approach can assist in more accurate identification of benign and malignant tissues in crc histopathological images. the high metrics achieved with mirnet suggest that this technique could be a valuable tool to aid pathologists and physicians in early crc diagnosis and treatment, thereby contributing to better clinical outcomes and improved quality of life for patients. 4.3 comparison with the literature observing sect. 2, our work stands out. table 4 presents our work along with related works. table 4. summary of comparison with literature.full size table though direct comparisons are challenging due to differences in datasets and methodologies, our proposed method has demonstrated high effectiveness in classifying histopathological images of crc. leveraging mirnet and inceptionv3 techniques, our method achieved an accuracy of 99.67 considering metrics beyond accuracy, our method also yielded promising results in terms of precision, sensitivity, specificity, f1-score, and auc. these metrics provide a comprehensive view of our model’s performance, demonstrating its ability to successfully distinguish between benign and malignant tissue in crc histopathological images. 4.4 case study in this section, we will describe two case studies of the proposed method. in fig. 5 (a), we present an example of a cancer image classified as benign, and in fig. 5 (b), we have a benign image classified as cancer. fig. 5. case study. (a) classified as benign, (b) classified as malign (adenocarcinoma). full size image visually, it can be observed that both images have very similar texture characteristics. the misclassification error may be attributed to a possible incorrect labeling or to the fact that, due to their similarity, mirnet may have enhanced features that confused inceptionv3 in classification. however, it is essential to highlight that, despite the classification errors, the method demonstrated considerable metrics, suggesting a notable overall performance. 4.5 discussion now, let’s discuss the advantages and limitations of our method for classifying crc histopathological images. our proposed method automates the process of distinguishing histopathological images, reducing manual intervention and, combined with medical expertise, enhances diagnostic efficiency. our approach stands out by employing mirnet for the first time in the context of crc histopathological images. mirnet automatically highlights relevant features, significantly contributing to the differentiation between malignant and benign tissues. we propose the use of data augmentation and transfer learning during training, bringing effective techniques for model enhancement. by adopting the inceptionv3 architecture for classification, our method benefits from the ability of this cnn to extract complex and discriminative features from images, resulting in superior performance. our method includes metrics such as accuracy, precision, sensitivity, specificity, f1-score, and auc, providing a more comprehensive and robust evaluation of the model’s performance. due to the flexible architectures and comprehensive metrics used, our method has the potential to be applied in other areas of medicine and in the classification of different types of histopathological images. however, it is worth noting that despite the method being robust, achieving 99% in all validation metrics, it does have some limitations: while mirnet is a powerful technique for image pre=processing, determining the best hyperparameters to optimize its performance automatically is still necessary. we limited our analysis to the inceptionv3 architecture for classification, but exploring other cnn architectures could further validate the proposed method. although we conducted experiments to empirically adjust the model’s hyperparameters, an automated approach for selecting the best hyperparameters could enhance the method’s performance. these considerations about advantages and limitations provide a comprehensive view of our method and highlight areas for future work. 5 conclusion the proposed method was developed to assist in the diagnosis of crc, one of the leading causes of morbidity and mortality worldwide. we propose a robust automatic method for classifying crc histopathological images, aiming to improve clinical diagnosis. the method represents a contribution by being the first to utilize mirnet for the preprocessing of crc histopathological images. furthermore, with the use of the inceptionv3 architecture, we obtained promising results, demonstrating the model’s ability to extract discriminative features and achieve metrics superior to the literature. as future work, other cnn architectures beyond inceptionv3 could be explored to validate their use and compare them with existing ones. additionally, it is suggested to investigate automated techniques for hyperparameter selection, aiming to make the method even more robust. references abdelsamea, m.m., zidan, u., senousy, z., gaber, m.m., rakha, e., ilyas, m.: a survey on artificial intelligence in histopathology image analysis. wiley interdisc. rev. data min. knowl. discov. 12(6), e1474 (2022) article  google scholar  borkowski, a.a., bui, m.m., thomas, l.b., wilson, c.p., deland, l.a., mastorides, s.m.: lung and colon cancer histopathological image dataset (lc25000). arxiv preprint arxiv:1912.12142 (2019) bray, f., ferlay, j., soerjomataram, i., siegel, r., torre, l., jemal, a., et al.: erratum: global cancer statistics 2018: globocan estimates of incidence and mortality worldwide for 36 cancers in 185 countries. ca cancer j. clin. 70(4), 313 (2020) article  google scholar  bukhari, s.u.k., syed, a., bokhari, s.k.a., hussain, s.s., armaghan, s.u., shah, s.s.h.: the histological diagnosis of colonic adenocarcinoma by applying partial self supervised learning. medrxiv, pp. 2020–08 (2020) google scholar  candelero, d., roberto, g.f., do nascimento, m.z., rozendo, g.b., neves, l.a.: selection of cnn, haralick and fractal features based on evolutionary algorithms for classification of histological images. in: 2020 ieee international conference on bioinformatics and biomedicine (bibm), pp. 2709–2716. ieee (2020) google scholar  chollet, f., et al.: keras. https://keras.io (2015) crespo, j., victorino, a.p., araujo, k., araujo, l.h., vieira, f.m.d.a.c.: colorectal cancer biomarkers and their impact on the clinical practice. braz. j. oncol. 17, 1–14 (2021) google scholar  dabass, m., vig, r., vashisth, s.: five-grade cancer classification of colon histology images via deep learning. in: communication and computing systems, pp. 18–24. crc press (2019) google scholar  diniz, j.o., et al.: detecçao de covid-19 em imagens de raio-x de tórax através de seleçao automática de pré-processamento e de rede neural convolucional. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 162–173. sbc (2023) google scholar  diniz, j.o., et al.: efficientxyz-deepfeatures: seleção de esquema de cor e arquitetura deep features na classificação de câncer de cólon em imagens histopatológicas. in: simpósio brasileiro de computação aplicada à saúde (sbcas), pp. 82–93. sbc (2024) google scholar  diniz, j.o.b., et al.: heart segmentation in planning ct using 2.5d u-net++ with attention gate. comput. methods biomech. biomed. eng. imag. vis. 11(3), 317–325 (2023) google scholar  diniz, j.o.b., diniz, p.h.b., valente, t.l.a., silva, a.c., paiva, a.c.: spinal cord detection in planning ct for radiotherapy through adaptive template matching, imslic and convolutional neural networks. comput. methods programs biomed. 170, 53–67 (2019) article  math  google scholar  diniz, j.o.b., diniz, p.h.b., valente, t.l.a., silva, a.c., de paiva, a.c., gattass, m.: detection of mass regions in mammograms by bilateral analysis adapted to breast density using similarity indexes and convolutional neural networks. comput. methods programs biomed. 156, 191–207 (2018) article  math  google scholar  diniz, j.o.b., ferreira, j.l., cortes, o.a.c., silva, a.c., de paiva, a.c.: an automatic approach for heart segmentation in ct scans through image processing techniques and concat-u-net. expert syst. appl. 196, 116632 (2022) article  google scholar  diniz, j.o.b., ferreira, j.l., diniz, p.h.b., silva, a.c., de paiva, a.c.: esophagus segmentation from planning ct images using an atlas-based deep learning approach. comput. methods programs biomed. 197, 105685 (2020) article  math  google scholar  diniz, j., et al.: detecção de covid-19 em imagens de raio-x de tórax através de seleção automática de pré-processamento e de rede neural convolucional. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 162–173. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229576 figueredo, w., et al.: abordagem computacional baseada em deep learning para o diagnóstico de endometriose profunda através de imagens de ressonância magnética. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 138–149. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229567 hidayah, n., ramadanti, a.n., novitasari, d.c.r.: classification of colon cancer based on hispathological images using adaptive neuro fuzzy inference system (anfis). khazanah informatika jurnal ilmu komputer dan informatika 9(2), 162–168 (2023) google scholar  júnior, d.d., cruz, l., diniz, j., júnior, g.b., silva, a.: classificação automática de glóbulos brancos usando descritores de forma e textura e extreme gradient boosting. in: anais do xxi simpósio brasileiro de computação aplicada à saúde, pp. 95–106. sbc, porto alegre, rs, brasil (2021). https://doi.org/10.5753/sbcas.2021.16056 mangal, s., chaurasia, a., khajanchi, a.: convolution neural networks for diagnosing colon and lung cancer histopathological images. arxiv preprint arxiv:2009.03878 (2020) de oliveira santos, m., de lima, f.c.d.s., martins, l.f.l., oliveira, j.f.p., de almeida, l.m., de camargo cancela, m.: estimativa de incidência de câncer no brasil, 2023-2025. revista brasileira de cancerologia 69(1) (2023) google scholar  rajinikanth, v., kadry, s., mohan, r., rama, a., khan, m.a., kim, j.: colon histology slide classification with deep-learning framework using individual and fused features. math. biosci. eng. 20(11), 19454–19467 (2023) article  google scholar  santos, p., brito, v., filho, a.c., sousa, a., diniz, j., luz, d.: efficientbacillus: uma arquitetura profunda para detecção dos bacilos de koch. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 198–209. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229608 swarna, i.j., hashi, e.k.: detection of colon cancer using inception v3 and ensembled cnn model. in: 2023 international conference on electrical, computer and communication engineering (ecce), pp. 1–6. ieee (2023) google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  zamir, s.w., et al.: learning enriched features for real image restoration and enhancement. in: vedaldi, a., bischof, h., brox, t., frahm, j.-m. (eds.) eccv 2020. lncs, vol. 12370, pp. 492–511. springer, cham (2020). https://doi.org/10.1007/978-3-030-58595-2_30 chapter  google scholar  download references acknowledgments to the support of the coordination for the improvement of higher education personnel (capes) financing code 001; the foundation for research support of maranhão (fapema); the national council for scientific and technological development (cnpq); and the brazilian company of hospital services (ebserh) brazil (proc. 409593/2021-4). author information authors and affiliations innovation factory – federal institute of maranhão (ifma), maranhão, brazil neilson p. ribeiro & joão otávio bandeira diniz applied computing group (nca) – federal university of maranhão (ufma), são luís, maranhão, brazil neilson p. ribeiro, felipe r. s. teles, joão otávio bandeira diniz, geraldo braz junior, joão d. s. de almeida & anselmo c. de paiva federal university of cariri (ufca), juazeiro do norte, ce, brazil luana b. da cruz & domingos a. dias jr. authors neilson p. ribeiroview author publications search author on:pubmed google scholar felipe r. s. telesview author publications search author on:pubmed google scholar joão otávio bandeira dinizview author publications search author on:pubmed google scholar luana b. da cruzview author publications search author on:pubmed google scholar domingos a. dias jr.view author publications search author on:pubmed google scholar geraldo braz juniorview author publications search author on:pubmed google scholar joão d. s. de almeidaview author publications search author on:pubmed google scholar anselmo c. de paivaview author publications search author on:pubmed google scholar corresponding author correspondence to neilson p. ribeiro . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ribeiro, n.p. et al. (2025). improving colorectal cancer diagnosis using mirnet and inceptionv3 on histopathological images. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_22 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords colorectal cancer deep learning histopathological images publish with us policies and ethics profiles geraldo braz junior view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature optimization strategies for bert-based named entity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper optimization strategies for bert-based named entity recognition conference paper first online: 12 october 2023 pp 80–94 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) optimization strategies for bert-based named entity recognition download book pdf download book epub monique monteiro9 & cleber zanchettin9  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 676 accesses 3 citations abstract transfer learning through language modeling achieved state-of-the-art results for several natural language processing tasks such as named entity recognition, question answering, and sentiment analysis. however, despite these advancements, some tasks still need more specific solutions. this paper explores different approaches to enhance the performance of named entity recognition (ner) in transformer-based models that have been pre-trained for language modeling. we investigate model soups and domain adaptation methods for portuguese language entity recognition, providing valuable insights into the effectiveness of these methods in ner performance and contributing to the development of more accurate models. we also evaluate ner performance in few/zero-shot learning settings with a causal language model. in particular, we evaluate diverse bert-based models trained on different datasets considering general and specific domains. our results show significant improvements when considering model soup techniques and in-domain pretraining compared to within-task pretraining. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others semantically-informed domain adaptation for named entity recognition chapter © 2024 named entity extractors for new domains by transfer learning with automatically annotated data chapter © 2022 token is a mask: few-shot named entity recognition with pre-trained language models chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing machine learning natural language processing (nlp) research methods in language and linguistics 1 introduction named entity recognition (ner) is an important task for meaning extraction of textual content. in recent years, advances in transfer learning with deep neural networks based on transformers [15] enabled substantial performance improvements for nlp models, especially in low-resource languages. in this context, transfer learning enables training by leveraging previous knowledge of a particular language’s grammatical structure that knowledge is embedded in models previously trained on unlabeled data through self-supervision. language models are the most classic examples among these pre-trained models. fig. 1. model soup model creation by averaging weights of multiple models trained with different hyperparameter sets. here, two pre-trained neural networks with identical architecture but different parameters are used to create a third model (“model soup”), whose weights are calculated by simply averaging the respective weights of model 1 and model 2, without any training. full size image in particular, for the portuguese language, the most commonly used model is bertimbau [12], a neural network that replicates with minor changes the architecture and training procedures for bert [4], by using brwac [16] as training corpus. bertimbau was validated in specific tasks such as named entity recognition, textual similarity, and recognizing textual entailment, surpassing previously published results. currently, a classical solution for constructing entity recognizers, classifiers, and other discriminative models for low-resource languages requires fine-tuning a pre-trained language model plm (e.g., bertimbau for portuguese) in the target task by adding linear layers and adjusting the network weights by retraining with a new optimization objective. such an approach has achieved satisfactory results for most situations at a low training cost [7, 8, 11]. however, there are still not yet explored techniques, at least for entity recognition tasks in the portuguese language: domain adaptation, one-shot/few-shot learning, and recent techniques such as model soups [17]. this paper applies model soups and domain adaptation techniques to named entity recognition (ner) task for brazilian portuguese language. we assess the pertinence of such techniques for this specific language and evaluate multiple ways of applying them, finding, for example, that certain configurations of uniform model soups worked best. we present a study reusing existing models for a relatively low-resource language, and we show that techniques proposed for other problems can effectively transfer to ner. so, from the viewpoint of practical applications in natural language understanding, our main contributions can be summarized as 1) experiments on a medium-to-low resource language (brazilian portuguese, as most gold standard datasets such as harem [10] were conceived for the portuguese variant); 2) further investigation on model soups on nlp, specifically for ner; 3) further investigation on domain adaptation focused on entity recognition, while previous research was more commonly conducted on document classification; and 4) evaluation of ner performance in few/zero-shot learning setups with a causal large language model (llm). this paper is organized in the following way: 2 introduces the techniques to be analyzed and other related works, while 3 and 4 present the detailed experiments and results, respectively. source code is available at https://github.com/monilouise/opt-bert-ner. 2 related work wortsman et al. [17] propose a technique that consists of generating a model by averaging the weights of two or more trained models, in opposition to the traditional approach, which is based on 1) training multiple models with several hyperparameters and 2) choosing the model with the best performance on a validation set. also, the idea is to combine multiple models without the additional inference and memory costs related to traditional ensemble learning. the authors refer to the technique as model soups, illustrated in fig. 1. it supports three variations: 1) construction of a model by averaging all models (uniform soup); 2) greedy soup, in which models are added sequentially to the soup as they improve the model accuracy in the validation dataset; and 3) learned soup, in which the interpolation weights for each model are optimized through gradient descent. according to the authors, the greedy strategy showed the best results. at the same time, learned soups require loading in memory all the models simultaneously, generating more extensive networks and leading to little gain in accuracy. their experiments were conducted on image and text classification. in this work, we evaluate the application of this technique to the ner task. sun, qiu, xu and huang [13] conducted experiments on fine-tuning bert pre-trained models on document classification. in particular, they analyzed domain adaptation a training approach with training data from the target task or the same domain. here, we apply domain adaptation to the ner task. houlsby et al. [6] propose transfer learning with adapter modules. according to the authors, adapter modules add only a few trainable parameters per task. new tasks can be added without revisiting previous ones, keeping the parameters of the original network fixed, with a high degree of parameter sharing. they transferred bert transformer model to 26 text classification tasks (including the glue benchmark), attaining near state-of-the-art performance. the following tasks were evaluated: document classification on several datasets, linguistic acceptability, sentiment analysis, paraphrase detection, semantic textual similarity, textual entailment, and question answering. only english datasets were used, and the paper does not report performance on the named entity recognition task. compared to our work, they focus on parameter efficiency and flexibility for different tasks, while we focus on entity recognition performance in a low-resource language. even so, we intend to investigate adapters architecture performance for ner tasks in future research. regarding portuguese language, rodrigues et al. [9] introduced albertina, a transformer-based foundation model both for european (portugal) and american (brazil) portuguese variants. compared to bertimbau, it uses 128-token sequence truncation instead of 512. the authors report results on tasks such as text entailment, semantic textual similarity and disambiguation tasks, but to our knowledge, they did not evaluate the model on entity recognition. finally, bert was one of the first transformer-based models; up to our knowledge, the current main portuguese plms were based on it. it takes some concerns about its potential compared to more recent plms. however, experiments conducted by tänzer et al. [14] show that bert can reach near-optimal performance even when many of the training set labels have been corrupted. 3 methodology and experiments 3.1 model soups experiments creating a model by averaging other models’ weights was validated on image and document classification tasks [17]. formally, let \({\theta } = finetune(\theta _{0}, h)\) the set of parameters obtained by fine-tuning the pre-trained initialization \({\theta _{0}}\) with the hyperparameters configuration h. the technique uses a mean value for \({\theta _i}\), i.e., \({\theta _{s} = \frac{1}{|s|}\sum _{i \in s} \theta _{i}}\), where \({s \subset \{1,...,k\}}\) and k is the number of hyperparameter configurations (or models) to be used. initially, we reproduced the idea using a uniform average. the greedy approach was not analyzed due to our low number of available candidate models. so, we created a model whose weights are the average of the weights of different models: (a) the entity recognition model developed to validate bertimbau [12], trained on an adapted version from the first haremfootnote 1 [10]; (b) model analogous to (a), but trained with data augmentationfootnote 2 and; (c) model adapted to the same corpus for the target task (first harem), as described in sect. 3.2. we denote this model as the average model or model soup. for the combinations cited above, we evaluated the bert base and large variations, which differ in the number of parameters. also, we evaluated settings with and without an additional layer for conditional random fields (crf), which we used to reduce the probability of generating invalid label sequences. fig. 2. model soup the original idea (adapted to the setting of two models trained on a ner task from a common initialization in this case, the language model). full size image there is a difference between our experiments and the original proposal by wortsman et al. (2022): the authors assume that the models were independently optimized from the same initialization, which would lead them to the same region or valley on the error surface. such strategy is represented in fig. 2. but here, according to fig. 3, only models a and b start from the same initialization (bertimbau language model). in contrast, model c was finetuned on the first harem textual content. the experiments’ results are shown in sect. 4.1. 3.2 domain adaptation experiments sun et al. [13] conducted experiments on different fine-tuning options, including multi-task learning. however, they concluded that the benefits from multi-task learning tend to be inferior to those obtained from additional pretraining. so, we did not experiment with multi-task learning. to conduct the experiments on domain adaptation, we use as start points: (a) a ner model for (long) documents related to public accounts auditingfootnote 3 footnote 4; (b) the ner model trained for bertimbau evaluation [12]; (c) an entity recognition model [11] trained on a public news dataset [7]. in (a), during domain adaptation, the original language model bertimbau received additional training on a dataset different from the one used to train the target ner model. however, this dataset came from the same domain and origin (documents related to public accounts auditing), leading to an intermediary and domain-optimized language model. as the training task was masked language model (mlm), such dataset does not contain any label and can be considered a kind of “superset” for the entity recognition dataset: for example, the dataset for training the domain-adapted language model has 52,912 documents, against 431 documents for the labeled dataset used during entity recognition model training. the complete flow is described in fig. 4. fig. 3. model soup alternative version using two models trained from a common parameter set (bertimbau language model) and a third model trained on an intermediary language model. full size image for (b) and (c), during the construction of the intermediary language models, the respective datasets were used: first harem (the same train split used by souza et al. [12]) and the media news dataset (the same train split as the original dataset [7]). the labels were discarded for both datasets in the phase of mlm training. fig. 4. training flow for a ner specific to public accounts audit domain. full size image the three resulting language models were used as base models for retraining the three cited entity recognizers to measure domain adaptation impact. learning rate and batch size hyperparameters for all intermediary language models training were the same as reported by souza et al. [12]. section 4.2 shows the experiments’ results. 3.3 causal language modeling few/zero-shot learning we also conducted experiments on entity recognition with few and zero-shot learning by using a large language model (llm) based on gpt-3.5, an improved version of gpt-3 [1], a language model pre-trained with causal language modeling objective. we used the same brazilian public news dataset already described [7] and compared the following settings: (a) a ner model [8] based on finetuning bertimbau to the news dataset (b) gpt 3.5 few-shot learning with instruction prompt and examples (c) gpt 3.5 few-shot learning with no instruction prompt (only examples) (d) gpt 3.5 zero-shot learning the results are shown in sect. 4.3. 4 results and discussions 4.1 model soup results analysis table 1 summarizes the results from the main experiments on the model soup technique. the bolded rows refer to the baseline (original) models, with mean and standard deviation from 5 training executions. rows prefixed as “m1:”, “m2:”, ..., and “m12:” are the respective “best” models for each variant among five training runs. these models are the model soups components. finally, rows related to the model soups labeled with + (plus) signs do not involve training (only two or three models averaging), so they were not randomized (this is why standard deviation values are not shown in these rows, except for the setup with additional fine-tuning, as explained below). we report precision, recall, and f1 metrics for all the experiments. in the first experiment, we evaluate the direct application of the model soup technique. we used a uniform average from a set composed of the respective best models (i.e., best training) among the variations described in 3.1. later, we evaluated the second setup, in which the model soup received additional fine-tuning for the ner task. as shown in table 1, for the smaller variant in model size (base), the first model, without additional fine-tuning, shows better results for precision and f1 metrics. table 1. experiments with model soups for ner bertimbau (“\(\textrm{bert}_\textrm{base}\)" or “\(\textrm{bert}_\textrm{large}\)") trained on first haremfull size table as already described in 3.1, for each model size (base/large), we added the following components to each combination: (a) original bertimbau ner; (b) bertimbau ner retrained with data augmentation; and (c) bertimbau ner retrained after domain adaptation to first harem (original bertimbau language model fine-tuned to first harem text set), as described in 3.2). table 2. domain adaptation for documents related to public accounts audit.full size table later, the third variant (c) was removed from each combination, leading to the original schema shown in fig. 2. finally, we observed that the combinations based only on (a) (original ner) and (b) (data augmented ner) led to better values for precision and f1 metrics, confirming wortsman’s (2022) original hypothesis of using independently optimized models from the same initialization. when we compare our methodology with the one used by the authors [17], they use accuracy in most image and text classification. for example, the authors do not refer explicitly to recall, which shows worse results in our experiments. so, further investigation is needed about the reason recall is worsening by the use of the model soup technique, which at the moment makes us believe that such a method could be more suitable to situations in which precision is more important than getting a high number of entities or false positive cases. on the other hand, given known limitations in using precision-recall-f1 for entity recognition, better and more interpretable metrics for this task are a current research topic [5]. finally, according to wortsman et al. [17], preliminary experiments show that improvement in the textual corpus, although present, is less profound than in image classification. the authors stress the need for additional investigation into this subject. they used imagenet and several variations, including illustrations and other representations beyond real photos (e.g., drawings, arts, cartoons, tattoos, origami, sculptures, etc.). but for textual classification, they used general domain datasets for paraphrase detection, coherence vs. contradiction, linguistic acceptability, and sentiment analysis. preliminary and qualitative analysis of the different data for images vs. texts shows more variability and larger data size for the first case (e.g., imagenet contains millions of images), which could have led to a more significant impact on image classification. 4.2 domain adaptation results analysis in this subsection, we report results achieved by ner models when trained over intermediary language models, as described in 3.2. the results of the experiment with the ner model trained on documents related to public accounts auditing are shown in table 2. as a comparison metric, masked f1 [11] was used. this metric is f1 calculated over post-processed output, correcting invalid transitions according to the iob2 schema instead of using the raw output directly. based on in-domain adaptation, this setup led to the most pronounced improvements. in the experiment conducted on the ner model for media news [7, 8, 11], the results are shown in table 3. table 3. domain adaptation for media news.full size table table 4. domain adaptation for first harem.full size table the experiment was conducted only with variants based on a pre-trained brazilian portuguese language model (bertimbau) because multi-language models gave an inferior performance, according to silva et al. [11] and chen et al. [2]. masked f1 was also used as the main comparison metric. table 4 shows the results achieved with the ner model used in bertimbau evaluation [12]. table 5. domain adaptation for first harem qualitative analysis.full size table the second column shows outputs generated by the baseline ner, while the third column shown outputs from the ner trained on the intermediary language model (domain-adapted ner). while the first one misclassifies portuguese expressions, the second one labels them correctly. all the examples belong to first harem test dataset. as can be noted in tables 3 and 4, experiments conducted with the media news ner and bertimbau ner did not reveal significant differences after domain adaptation. such results confirm observations by sun et al. [13]: the domain adaptation made on media news and first harem is “within-task" (the texts used are the same as the training texts for the target task). in general, “in-domain" pretraining using texts from the same domain which are different from the texts in the training dataset for the target task gives superior results. we suspect that within-task pretraining could lead to overfitting because it uses the same texts from the target task dataset. however, after error analysis, we realized that some european expressions, organizations, and local names could be correctly classified only after bertimbau brazilian language model domain adaptation on (european) harem. at the same time, they were misclassified when ner was trained directly from the raw bertimbau language model. the results are shown in table 5, where we have the following examples: table 6. sample outputs by a generic domain lm vs. specific domain lm.full size table table 7. few/zero-shot learningfull size table 1st row: in portugal, “consoada" refers to christmas night, which should be classified as a temporal (time) mention. 2nd row: “soajo" is the name of a portuguese village. 3rd row: “abade da loureira" refers to both an organization (org) and a street (loc). but given the specific context in the sentence, it should be classified as an organization (org). 4th row: “centro de formação de informática do minho" is a local educational institution. 5th row: “são joão do souto" refers to an extinct portuguese parish. it makes sense because first harem is a portuguese corpus, different from the brazilian corpus used to train bertimbau. these results show semantic gains from domain adaptation, although quantitative performance differences are not statistically significant. further, the results shown here were obtained in the ner task context. for document classification, within-task pretraining has been commonly used. finally, we showed that domain adaptation in experiment (a) by training an intermediary language model with a larger, same-domain dataset led to a higher impact on f1 metrics when compared to the experiments with within-task domain adaptation ((b) and (c)). furthermore, qualitative analysis for the intermediary language model shows example outputs for predicting masked term tasks, i.e., the public accounts auditing language model can generate texts related to themes such as contracts and biddings, as seen in table 6. 4.3 causal language modeling few/zero-shot learning this subsection summarizes experiments conducted on gpt 3.5, a large language model pre-trained with a causal language modeling objective. the results are shown in table 7. in the few-shot setting, we gave the model some examples. these examples could be accompanied by an instruction prompt telling the model the kinds of entities expected. we also tested sending only the examples without any further instruction. in the zero-shot setting, we only asked the question and let the model be free to return the information in any format, giving no examples. so it returned the results in a conversational unstructured format, making it difficult to measure the output precision and f1. therefore, we only show recall. despite the recent impressive reasoning skills shown by gpt 3.5 and “chatgpt” models, it is interesting to note its quantitative performance still lags behind traditionally fine-tuned models. we hypothesize that bert bidirectional masked language model objective may be more suitable to discriminative tasks, while causal language models concerned only with predicting the future words may lose information from past tokens. on the other hand, we believe that more research is necessary to investigate prompt engineering practices more suitable to ner and similar tasks. finally, we realized impressive qualitative results on gpt 3.5 zero-shot learning: although its recall is considerably worse than the bert-based baseline, it returns the entities and the relationships between them. 5 conclusions among the analyzed techniques, model soups achieved the best results for the ner task. in the experiments conducted on domain adaptation, the best results were achieved with in-domain adaptation. we did not observe significant improvements in within-task domain adaptation. however, we realized the model could learn domain-specific terms with the first harem corpus. the lack of both quantity and diversity of “golden” labeled datasets for portuguese when compared to english or chinese is a substantial limitation to research on several tasks or multitask learning, as done, for example, by houlsby at al. [6]. we believe advances in few-shot learning and prompt engineering could solve this limitation through synthetic data generation. so, besides investigating fine-tuning with adapters, we expect to conduct further experiments on few-shot learning. finally, we intend to investigate the fine-tuning of the recent albertina llm to the ner task and check if its larger capacity can compensate for its smaller context window (128 tokens) compared to bertimbau. notes 1.the adapted version refers to a setting called “selective" by the authors, in which only 5 classes are used (person, organization, local, value and date). 2.here, we used label-wise token replacement (lwtr) [3]. 3.we used the hyperparameters for learning rate and batch size suggested by silva et al. [11]. 4.this dataset contains non-public data and cannot be made publicly available. references brown, t., et al.: language models are few-shot learners. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 1877–1901. curran associates, inc. (2020), https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-paper.pdf chen, y., mikkelsen, j., binder, a., alt, c., hennig, l.: a comparative study of pre-trained encoders for low-resource named entity recognition. in: gella, s., et al (eds.) proceedings of the 7th workshop on representation learning for nlp, repl4nlp@acl 2022, dublin, ireland, 26 may 2022, pp. 46–59. association for computational linguistics (2022). https://doi.org/10.18653/v1/2022.repl4nlp-1.6 dai, x., adel, h.: an analysis of simple data augmentation for named entity recognition. in: scott, d., bel, n., zong, c. (eds.) proceedings of the 28th international conference on computational linguistics, coling 2020, barcelona, spain (online), 8-13 december 2020, pp. 3861–3867. international committee on computational linguistics (2020). https://doi.org/10.18653/v1/2020.coling-main.343 devlin, j., chang, m., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. corr arxiv: 1810.04805 fu, j., liu, p., neubig, g.: interpretable multi-dataset evaluation for named entity recognition. in: webber, b., cohn, t., he, y., liu, y. (eds.) proceedings of the 2020 conference on empirical methods in natural language processing, emnlp 2020, online, 16-20 november 2020, pp. 6058–6069. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.emnlp-main.489 houlsby, n., et al.: parameter-efficient transfer learning for nlp. in: chaudhuri, k., salakhutdinov, r. (eds.) proceedings of the 36th international conference on machine learning, icml 2019, 9-15 june 2019, long beach, california, usa. proceedings of machine learning research, vol. 97, pp. 2790–2799. pmlr (2019), http://proceedings.mlr.press/v97/houlsby19a.html monteiro, m.: extrator de entidades mencionadas em notícias da mídia. https://github.com/secexsaudetcu/noticias_ner (2021), (accessed 21 may 2022) monteiro, m.: riskdata brazilian portuguese ner. https://huggingface.co/monilouise/ner_news_portuguese (2021), (accessed 21 may 2022) rodrigues, j., et al.: advancing neural encoding of portuguese with transformer albertina pt-. corr https://doi.org/10.48550/arxiv.2305.06721, https://doi.org/10.48550/arxiv.2305.06721 (2023) santos, d., seco, n., cardoso, n., vilela, r.: harem: an advanced ner evaluation contest for portuguese. in: calzolari, n., et al. (eds.) proceedings of the fifth international conference on language resources and evaluation, lrec 2006, genoa, italy, 22-28 may 2006, pp. 1986–1991. european language resources association (elra) (2006), http://www.lrec-conf.org/proceedings/lrec2006/summaries/59.html silva, e.h.m.d., laterza, j., silva, m.p.p.d., ladeira, m.: a proposal to identify stakeholders from news for the institutional relationship management activities of an institution based on named entity recognition using bert. in: wani, m.a., sethi, i.k., shi, w., qu, g., raicu, d.s., jin, r. (eds.) 20th ieee international conference on machine learning and applications, icmla 2021, pasadena, ca, usa, 13–16 december 2021, pp. 1569–1575. ieee (2021). https://doi.org/10.1109/icmla52953.2021.00251 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  sun, c., qiu, x., xu, y., huang, x.: how to fine-tune bert for text classification? in: sun, m., huang, x., ji, h., liu, z., liu, y. (eds.) ccl 2019. lncs (lnai), vol. 11856, pp. 194–206. springer, cham (2019). https://doi.org/10.1007/978-3-030-32381-3_16 chapter  google scholar  tänzer, m., ruder, s., rei, m.: memorisation versus generalisation in pre-trained language models. in: proceedings of the 60th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 7564–7578. association for computational linguistics, dublin, ireland (may 2022). https://doi.org/10.18653/v1/2022.acl-long.521 vaswani, a., shazeer, n., parmar, n., uszkoreit, j., jones, l., gomez, a.n., kaiser, l.u., polosukhin, i.: attention is all you need. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-paper.pdf wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). european language resources association (elra), miyazaki, japan (may 2018). https://aclanthology.org/l18-1686 wortsman, m., et al.: model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time. corr abs/2203.05482 (2022) google scholar  download references author information authors and affiliations informatics center federal university of pernambuco (ufpe), recife, brazil monique monteiro & cleber zanchettin authors monique monteiroview author publications search author on:pubmed google scholar cleber zanchettinview author publications search author on:pubmed google scholar corresponding author correspondence to monique monteiro . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper monteiro, m., zanchettin, c. (2023). optimization strategies for bert-based named entity recognition. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_6 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords ner bert transfer learning model soups publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature i2de: improved interval differential evolution for numerical constrained global optimization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper i2de: improved interval differential evolution for numerical constrained global optimization conference paper first online: 28 november 2021 pp 186–201 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) i2de: improved interval differential evolution for numerical constrained global optimization download book pdf download book epub mariane r. s. cassenote  orcid: orcid.org/0000-0003-0489-223x10, guilherme a. derenievicz  orcid: orcid.org/0000-0002-3970-176610 & fabiano silva  orcid: orcid.org/0000-0001-5453-617510  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 735 accesses 1 citation abstract several hybrid approaches have been proposed to solve numerical constrained optimization problems. in this paper we present an improved interval differential evolution (i2de) that uses structural information of the instance during the optimization process. we extend the math operations supported by a multi-interval core implementation that allows pruning infeasible solutions by using local consistency techniques and a backtrack-free local search. furthermore, we propose a reformulation of interval evolutionary mutation strategies. a comprehensive experimental analysis is conducted over coconut and cec2018 competition benchmarks and indicates that the hybridization between metaheuristics and constraint programming significantly improves the quality of the solutions. the experimental evaluation shows that our black-box version of i2de outperformed several state-of-the-art solvers. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others interval differential evolution using structural information of global optimization problems chapter © 2019 differential evolution improvement by adaptive ranking-based constraint handling technique article 10 may 2023 development of metaheuristic interval minimization methods for optimal program control design article 01 february 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer-aided engineering (cad, cae) and design continuous optimization discrete optimization genomic engineering optimization calculus of variations and optimization 1 introduction in the last few decades, interval based solvers have been used to tackle numerical constrained global optimization problems (ncop) in a rigorously way [1]. in general, such methods are composed by a complete investigation of the search space, using the structure of the problem and consistency techniques from the constraint programming field to prune infeasible solutions. despite the great progress of interval techniques, such methods remain impractical in instances with a large number of dimensions. on the other hand, differential evolution (de) has become one of the most used evolutionary algorithms to deal with large global optimization problems, due to its performance and simplicity [3, 4, 20]. as opposed to interval algorithms, de based solvers are fast, but do not guarantee the global optimality. also, de uses the instance as a black-box model, where its structure is unknown. a promising strategy to handle with ncops is the hybrid approach, where different methods are combined [26]. in this context, the solver interval differential evolution (inde) was proposed [5]. this method integrates the usual de approach with an interval solver called relaxed global optimization (ogre) [8]. this solver uses a structural decomposition of the ncop instance to identify the amount of local consistency that guarantees a backtrack-free optimization. although this is a strong theoretical result, achieving this amount of local consistency can be an intractable problem. therefore, ogre uses an interval branch & bound (ib&b) method to tackle a relaxed form of the instance. thus, the solution found by ogre may be infeasible on the original ncop instance. moreover, the computational cost of the ib&b is prohibitive. the inde solver extended usual de operators to the interval context, using ogre’s core to select a subset of variables on which the search process will occur, whilst the others are valuated by constraint propagation. in addition, local consistency techniques are applied to prune infeasible solutions. by combining theses techniques, inde outperformed ogre [5]. in this work, we present the improved interval differential evolution (i2de) solver. we extend inde and the operators supported by the ogre’s multi-interval core, which allows us to tackle a greater diversity of benchmark functions and real-world problems. moreover, we present improved versions of three interval evolutionary mutation operations and incorporate several heuristics of state-of-the-art solvers. an extensive experimental analysis performed over the coconut [19] and cec2018 competition on constrained real-parameter optimization [27] benchmarks reveals that our i2de significantly outperforms inde, ogre and a black-box version of i2de, which suggests that hybridization between metaheuristics and structural decomposition is a promising research direction. furthermore, our black-box solver outperformed several state-of-the-art metaheuristic solvers. the remaining of this paper is organized as follows: sect. 2 contains background definitions of interval methods and metaheuristics. section 3 provides details of i2de’s features and the main improvements compared to inde. the experimental analysis is presented in sect. 4 and sect. 5 concludes this work. 2 background a numerical constrained global optimization problem (ncop) consists of finding an assignment of values to a set of variables \(v = \{x_1, x_2, \dotsc , x_d\}\) that minimizes an objective function subject to a constraint network (cn) \(\mathcal {n} = (v, \mathcal {d}, \mathcal {c})\), where \(\mathcal {d}\) is the domain set of v and \(\mathcal {c}\) is a set of constraints of the form \(g_i(x_{i_1}, \dotsc , x_{i_k}) \le 0\). domain sets can be represented by intervals or multi-intervals. given a closed interval \(x = [\underline{x}, \overline{x}]\), we call the endpoints of x; \(\omega (x) = \overline{x} \underline{x}\) denotes the width of x and \(\mu (x) = (\underline{x}+\overline{x})/2\) denotes its midpoint. a closed interval can be defined by its width and midpoint as follows: \(x = \left[ \mu (x) \omega (x) / 2,\,\, \mu (x) + \omega (x) / 2 \right] \). a multi-interval \(\mathcal {x} = \langle x_1, x_2, \dotsc , x_k\rangle \) is an ordered set of disjointed intervals, where \(i< j \implies \overline{x}_i < \underline{x}_j\). in this case, the domain set of a cn is a (multi-) interval box \((\mathcal {x}_1, \mathcal {x}_2, \ldots , \mathcal {x}_d)\). in this paper, we tackle ncops which constraints can be decomposed into a set of ternary constraintsfootnote 1 \(x = y \circ z\) or \(x = \diamond y\), where \(\circ \) (\(\diamond \)) is a well-defined binary (unary) operation. therefore, auxiliary variables are included in the network. for example, the constraint \(x + 2 \cdot \sin (y)^2 \le 0.5\) is encoded as \(\{c_1 = x + a_1,\ \ a_1 = 2 \cdot a_2,\ \ a_2 = {a_3} ^ 2,\ \ a_3 = \sin (y)\}\), where \(a_i = (-\infty , +\infty )\) is the domain set of the auxiliary variable \(a_i\) and \(c_1 = (-\infty , 0.5]\) is the interval constant that represents the original constraint’s relation. 2.1 interval analysis applied to global optimization interval analysis is a method of numerical analysis introduced by moore [16]. given intervals x, y and z the interval extension of any binary (unary) operation \(\circ \) (\(\diamond \)) well defined in is defined by: the sets \(x \circ y\) and \(\diamond z\) can be intervals, multi-intervals or empty sets. it is possible to compute \(x \circ y\) or \(\diamond z\) for all algebraic and the common transcendental functions only by analyzing the endpoints of x and y or z [10, 16], e.g., \([\underline{x}, \overline{x}] + [\underline{y}, \overline{y}] = [\underline{x} + \underline{y}, \overline{x} + \overline{y}]\), \([\underline{x}, \overline{x}] \cdot [\underline{y}, \overline{y}] = [\min \{\underline{x}\underline{y}, \underline{x}\overline{y},\overline{x}\underline{y},\overline{x}\overline{y}\}, \max \{\underline{x}\underline{y}, \underline{x}\overline{y},\overline{x}\underline{y},\overline{x}\overline{y}\}]\), \(2^{[\underline{z}, \overline{z}]} = [2^{\underline{z}}, 2^{\overline{z}}]\), etc. given multi-intervals \(\mathcal {x}\), \(\mathcal {y}\) and \(\mathcal {z}\), the operation \(\mathcal {x} \circ \mathcal {y}\) is the tightest multi-interval that contains \(\{x \circ y\ |\ x \in \mathcal {x}, y \in \mathcal {y}\}\), and \(\diamond \mathcal {z}\) is the tightest multi-interval that contains \(\{\diamond z\ |\ z \in \mathcal {z}\}\). in the constraint programming field, a constraint is locally consistent if it satisfies some specific property within a given box. for instance, a ternary constraint \(c: x = y \circ _1 z\) is generalized arc-consistent (gac) w.r.t. a box (x, y, z) iff \(x \subseteq y \circ _1 z\), \(y \subseteq x \circ _2 z\) and \(z \subseteq x \circ _3 y\), where \(\circ _2\) and \(\circ _3\) are the inverse operations of \(\circ _1\) that hold the condition \((x = y \circ _1 z) \iff (y = x \circ _2 z) \iff (z = x \circ _1 y)\). in other words, if c is gac then given any value for one of its variables, one can extend such valuation for its remaining variables whilst satisfying c. this notion of local consistency was proposed by [15]. it is well known that acyclicfootnote 2 cn can be solved in a backtrack-free manner if gac is achieved [6]. such result can be extended to ncop by encoding the objective function f(x) as a new constraint \(y = f(x)\); after enforcing gac (by removing inconsistent values from the current box) we instantiate \(y = \min {y}\) and propagate this valuation over the entire network without encountering any conflicts [8]. however, gac is not enough when the network is not acyclic. another notion of local consistency is relational arc-consistent (rac) [7]. a ternary constraint \(c: x = y \circ _1 z\) is rac w.r.t. a box (x, y, z) iff \(x \supseteq y \circ _1 z\), \(y \supseteq x \circ _2 z\) and \(z \supseteq x \circ _3 y\), i.e., given any value for two variables of c, one can extend such valuation for its remaining variable whilst satisfying c. in [8] it was proposed a decomposition of cns that relates the amount of consistency necessary to ensure a backtrack-free solution. an epiphytic decomposition of a cn is a tuple \((\mathcal {a}, \varomega , t)\), where \(\mathcal {a}\) is an ordered set of acyclic networks obtained by removing from the cn a set of constraints \(\varomega \) and \(t: \varomega \mapsto v_\varomega \) is a function that associates each constraint \(c \in \varomega \) with one of its variables t(c) satisfying the following: if t(c) belongs to the network \(n_i\) then (i) the remaining variables of c belongs to previous networks \(n_{j<i}\); and (ii) there is no other constraint \(c' \in \varomega \) such that \(t(c')\) belongs to \(n_i\). if the cn encodes a ncop instance, the variable \(y = f(c)\) that represents the objective function must be in the first acyclic network of \(\mathcal {a}\). it was shown that if a ncop \(\mathcal {p}\) encoded as a ternary cn is gac and the constraints in the set \(\varomega \) of its epiphytic decomposition are rac, then \(\mathcal {p}\) can be solved in a backtrack-free fashion. the proposed ogre [8] and inde [5] solvers are based on this relation and attempt to achieve the relational arc-consistency of the cn as a form of optimization. however, enforcing rac is in general intractable. the approach proposed in ogre approximates rac by using an interval branch and pruning scheme. if a constraint \((c: x = y \circ z) \in \varomega \) is not rac under the current box, the domain of some variable in this constraint (excluding t(c)) is bisected (branch) and gac is enforced on both sub-problems (pruning) before a new verification of the rac property of \(\varomega \) constraints. the algorithm continues in a recursive fashion. therefore, although the ternary decomposition of the original ncop instance increases the number of variables, not all of them are considered in the branch process, but only those of \(\varomega \) constraints. in each branch of the search tree, a backtrack-free local search occurs. first, the variable representing the objective function is instantiated with its minimum value in the current box. next, the valuation is propagated over all the network, following the ordering \(\mathcal {a}\) of the epiphytic decomposition. note that acyclic networks of \(\mathcal {a}\) are feasible if gac was enforced, but the constraints in \(\varomega \) may be infeasible. there are two main parameters that control ogre’s search procedure: the tolerance \(\varepsilon _{\varomega }\) allowed in each \(\varomega \) constraint, and the minimum granularity \(\vardelta \) which intervals can be bisected. ogre’s approach is a variation of usual interval branch and bound (ib&b) methods [1] that have been used in the last few decades to rigorously solve ncop. interval methods compute a set of atomic boxes that contains an optimum solution of the ncop instance. due to elevated computational cost, these algorithms remain inefficient in instances with many dimensions. on the other hand, a solution found by ogre may be inexact (w.r.t. objective function cost or constraint violation), because rac is approximated by a tolerance \(\varepsilon _{\varomega }\) in \(\varomega \) constraints. besides that, find an acceptable value for \(\varepsilon _{\varomega }\) is not trivial. the multi-interval core proposed in ogre is also used in inde, including the local search procedure, gac contractor, epiphytic decomposition of the cn, and multi-interval operations \(+\), −, \(*\), /, \(\wedge \) and \(\surd \). 2.2 differential evolution storn and price [20] proposed differential evolution (de) as an evolutionary algorithm that combines the coordinates of existing solutions with a particular probability to generate new candidate solutions. the classical de consists of a loop of g generations over a population of \( np \) individuals. an individual is an assignment of values to all variables of the instance, represented by a vector \(\texttt {x} = (a_1, a_2, \dots , a_d)\), where \(a_i \in x_i\) is a value from the domain of the variable \(x_i\), \(1 \le i \le d\). the fitness evaluation of an individual is responsible for determinate its quality throughout the evolutionary process. the initial population is randomly generated according to a uniform distribution over \(x_1 \times \dots \times x_d\). during each generation, the operators of mutation, crossover and selection are performed on the population until a termination condition is satisfied, like a fixed maximum number of fitness evaluations (\( maxfes \)). in the mutation phase, an operator is applied to generate a mutant vector \(\texttt {v}_i\) for each individual \(\texttt {x}_i\) (called target vector) of the population. the most popular mutation operator is de/rand/1 and is defined by: $$\begin{aligned} \texttt {v}_i = \texttt {r}_{1} + f \cdot (\texttt {r}_{2} \texttt {r}_{3}), \end{aligned}$$ (1) where f is the scaling factor and \(\texttt {r}_{1}\), \(\texttt {r}_{2}\) and \(\texttt {r}_{3}\) are three mutually distinct individuals randomly selected from the population. other popular mutation operators are used in this work, such as de/current-to-rand/1 (eq. 2) [12] and de/current-to-pbest/1 (eq. 3) [29]: $$\begin{aligned} \texttt {v}_{i}&= \texttt {x}_{i} + s \cdot (\texttt {r}_{1} \texttt {x}_{i}) + f \cdot (\texttt {r}_{2} \texttt {r}_{3}), \end{aligned}$$ (2) $$\begin{aligned} \texttt {v}_{i}&= \texttt {x}_{i} + f \cdot (\texttt {r}_{\text {pbest}} \texttt {x}_{i}) + f \cdot (\texttt {r}_{1} \texttt {r}_{2}), \end{aligned}$$ (3) where s is a uniformly distributed random number between 0 and 1, p is a value in \([1, np ]\), and \(\texttt {r}_{\text {pbest}}\) is an individual randomly chosen from the p best individuals of the current population. a crossover search operator is applied on the mutant individual \(\texttt {v}_i\) to produce a new solution \(\texttt {u}_i\), called trial vector, given a probability defined by the crossover rate \( cr \). in the exponential crossover used in this work, we first choose an integer \(d \in [1, d]\) to be the starting point of the target vector for crossover. we also determine another integer value l chosen from [1, d] with probability \(\mathsf {p}(l \ge l) = cr ^{l-1}\) for any \(l > 0\) which denotes how many consecutive decision variables are selected from the mutant vector starting at d position. then, the offspring is generated as follows: $$\begin{aligned} \texttt {u}_{ij} = {\left\{ \begin{array}{ll} \texttt {v}_{ij}, &{} \text { if } j \in \{ m_d(d), m_d(d+1), \dots , m_d(d + l-1) \},\\ \texttt {x}_{ij}, &{} \text { otherwise, } \end{array}\right. } \end{aligned}$$ where \(m_d(n) = 1 + ((n 1) \mod d)\) allows to iterate cyclically through the vector. finally, a selection operator is performed on \(\texttt {x}_i\) and \(\texttt {u}_i\), and the best one according to a fitness evaluation function is chosen for the next generation. 3 improved interval differential evolution in this section we describe details of our approach, called i2de, and emphasize the improvements and differences from the work presented in [5] which introduced the inde solver, an interval differential evolution approach that uses structural information for global optimization. one of the main contributions of our work is the extension of operations supported by ogre’s multi-interval core. previously, it was only possible to tackle instances restricted to operations \(+\), −, \(*\), /, \(\wedge \) and \(\surd \). we extended the multi-interval implementation for handling the following additional operations: log, exp, sin, cos, tan, abs, sign, max and min. considering the new operations, we implemented a new procedure to identify the tolerance \(\varepsilon _{\varomega }\) needed to approximate rac. for example, given a constraint \(x = y \circ z\), this tolerance is the maximum distance between any value of the multi-interval \(\mathcal {y} \circ \mathcal {z}\) to its closest value in the multi-interval \(\mathcal {x}\). however, the operator \(\circ \) may be not well-defined within the box \((\mathcal {y}, \mathcal {z})\). in this case, we compute the distance using the inverse operation of \(\circ \) that results in the tightest multi-interval. for instance, we can not compute the distance for the constraint \(x = y / z\) within the box \(z = [0,0]\), but we can do it for the inverse \(y = x * z\). in i2de and inde, an individual is an assignment of intervals to variables (a box). the population is a set of boxes that covers parts of the search space, instead of just points as in classical metaheuristics. in the optimization process, only the variables of constraints in the \(\varomega \) set of an epiphytic decomposition are considered. this allows to apply local consistency techniques to prune infeasible solutions. ogre’s local search is used to compute the real parameter instantiation of the variables. however, unlike in [5], we use the original instance modeling to evaluate the individual’s fitness value, instead of the ternary encoded cn. we propose improved formulations of three de interval mutation operators. interval adaptations of the main features of state-of-the-art solvers are implemented. some details of i2de’s components are discussed below. 3.1 interval population in the same way as in [5], the i2de’s initial population is generated by the top level branching tree of ogre’s ib&b. this strategy guarantees that the initial population covers all the search space. although each branch of ogre’s ib&b is composed by a gac multi-interval box, we iteratively split multi-intervals to obtain a set of interval boxes (individuals) that are added to the initial population until the number of individuals is \( np \). if numerical rounding errors make an individual to be considered inconsistent by ogre’s backtrack-free local search, this individual is replaced by a randomly generated individual from the initial domain of the instance. the scheme in which the current population is subdivided between the sub-populations a and b of size \( np /2\) was mantained. the entire population is kept sorted, which allows a to contain the best individuals. in order to promote exploitation of the fitter solutions and accelerate the convergence process, we apply a pool of three strategies to each individual of this sub-population. the best one of the three trial individuals is compared to the parent individual in the selection operation. the other individuals are added to an archive, while in [5] they were discarded. in sub-population b, an adaptive scheme is used to choose the mutation strategy to be applied, as proposed in [24]. at the end of each generation, all individuals are sorted and divided between the sub-populations. in inde, when a multi-interval box is split into interval individuals, only the best fitting one is compared to the target vector, and the others are added to the archive. similarly, individuals who are not selected for the next generation are also added to this archive with a maximum size of 100,000 individuals. when the archive is full, or every 50 generations, the current population and the archive are merged, the \( np \) best individuals go to the new population, and the archive is emptied. our tests revealed that this process is computationally expensive and contributes very little to the maintenance of diversity in the population. so i2de only maintain individuals who are not selected for the next generation in a \( np \times 2.6\) archive, according to the strategy proposed in [29] and used in many state-of-art de solvers [3, 13, 18]. finally, we kept the linear population size reduction scheme proposed in [22] and applied in [5]. at each generation, the new population size is calculated and if it is different from the current size, the worst individuals are deleted from the population until it has the new size. when the population size reduces, the archive size is proportionally reduced. 3.2 interval operations in iude [24] it was proposed the use of three mutation operators: de/rand/1, de/current-to-pbest/1 and de/current-to-rand/1. just like in [5], we use this operators pool in the sub-population a in order to intensify the local search around the best individuals in the current population. the three generated trial vectors are compared among themselves and the mutation strategy with the best trial vector scores a win. at every generation, the success rate of each strategy is evaluated over the period of previous 25 generations. in the bottom sub-population b, the probability of employing a mutation strategy is equal to its recent success rate on the top sub-population a. additionally, base and terminal vectors are selected from the top sub-population. since an interval is defined by its width \((\omega )\) and midpoint \((\mu )\), mutation operators can be applied over midpoints and extended to deal with interval widths. in [5], it was introduced interval versions of the three mutation operators mentioned above. however, our tests revealed that in practice numerical rounding errors may result in intervals with negative widths. consequently, with the application of local consistency, the box is considered inconsistent and discarded. in order to optimize the convergence process, we reformulated the mutation strategies proposed in inde [5]. the interval version of de/rand/1 combines \(\texttt {r}_1\), \(\texttt {r}_2\) and \(\texttt {r}_3\) to generate the mutant vector \(\texttt {v}_i\). the j-th element of \(\texttt {v}_i\) is defined by: $$\begin{aligned} \mu (\texttt {v}_{ij}) = \mu (\texttt {r}_{1j}) + f_i \cdot \left( \mu (\texttt {r}_{2j}) \mu (\texttt {r}_{3j}) \right) , \end{aligned}$$ (4) $$\begin{aligned} \omega (\texttt {v}_{ij}) = \omega (\texttt {r}_{1j}) \cdot \left( 1 + f_i \cdot \left( \frac{\omega (\texttt {r}_{2j})}{\omega (\texttt {r}_{3j})} 1 \right) \right) . \end{aligned}$$ this formulation reduces numerical errors and avoid inconsistent intervals. the other two mutation operators are defined in a similar way. the interval version of de/current-to-rand/1 combines the target vector \(\texttt {x}_i\) with three randomly selected individuals \(\texttt {r}_1\), \(\texttt {r}_2\) and \(\texttt {r}_3\). the mutant vector is defined by: $$\begin{aligned} \mu (\texttt {v}_{ij}) = \mu (\texttt {x}_{ij}) + s_i \cdot \left( \mu (\texttt {r}_{1j}) \mu (\texttt {x}_{ij}) \right) + f_i \cdot \left( \mu (\texttt {r}_{2j}) \mu (\texttt {r}_{3j}) \right) , \end{aligned}$$ (5) $$\begin{aligned} \omega (\texttt {v}_{ij}) = \omega (\texttt {x}_{ij}) \cdot \left( 1 + s_i \cdot \left( \frac{ \omega (\texttt {r}_{1j})}{\omega (\texttt {r}_{ij})} 1 \right) \right) \cdot \left( 1 + f_i \cdot \left( \frac{\omega (\texttt {r}_{2j})}{ \omega (\texttt {r}_{3j})} 1 \right) \right) , \end{aligned}$$ where \(s_i\) is a random number between 0 and 1. this strategy does not use crossover operator, so the trial vector is a copy of the mutant vector \(\texttt {v}_i\). finally, the interval version of de/current-to-pbest/1 with archive uses the coordinates \(\texttt {x}_i\) and \(\texttt {r}_1\) in the same way as in eq. 5, while \(\texttt {r}_2\) is randomly chosen from the union of the current population and the archive, and \(\texttt {r}_{pbest}\) is selected among the p best individuals in the population: $$\begin{aligned} \mu (\texttt {v}_{ij}) = \mu (\texttt {x}_{ij}) + f_i \cdot \left( \mu (\texttt {r}_{\text {pbest}j}) \mu (\texttt {x}_{ij}) \right) + f_i \cdot \left( \mu (\texttt {r}_{1j}) \mu (\texttt {r}_{2j}) \right) , \end{aligned}$$ (6) $$\begin{aligned} \omega (\texttt {v}_{ij}) = \omega (\texttt {x}_{ij}) \cdot \left( 1 + f_i \cdot \left( \frac{\omega (\texttt {r}_{\text {pbest}j})}{\omega (\texttt {r}_{ij})} 1 \right) \right) \cdot \left( 1 + f_i \cdot \left( \frac{\omega (\texttt {r}_{1j})}{\omega (\texttt {r}_{2j})} 1 \right) \right) . \end{aligned}$$ whereas in [5] the value of p remained fixed throughout the evolutionary process, we adopted the strategy proposed in [4]. after each generation g, the p value in the next generation \(g+1\) is computed as follows: $$\begin{aligned} p = \left( \frac{p_{ max } p_{ min }}{ maxfes } \right) \cdot nfes + p_{ min }, \end{aligned}$$ (7) where \(p_{ min }\) and \(p_{ max }\) are, respectively, the minimum and maximum values of p, \( nfes \) is the current number of fitness evaluations and \( maxfes \) is the maximum number of fitness evaluations. additionally, in [5] if the width \(\omega (\texttt {u}_{ij})\) is greater than the width of the base individual \(\omega (\texttt {r}_{1j})\), it is updated by \(\omega (\texttt {u}_{ij}) := \omega (\texttt {r}_{1j})\). considering that this mechanism can decrease the population diversity by forcing the reduction of intervals, it is not used in i2de. it is known that the settings of values for the f and \( cr \) parameters is instance dependent and may change according to the region of the search space being visited. to make these parameters self-adaptive, inde employed a scheme introduced in [22] that uses a pair of memory values \(\langle m_{ cr }, m_{ f } \rangle \) for each mutation operator in order to store the settings that were successful in the last generations. all the memory pairs are stored in a vector of h positions. at each generation one of the positions is circularly updated based on the weighted lehmer mean of the fitness differences between offspring and their parents. in i2de we incorporated some features proposed in [3, 4]. the first \(h-1\) positions of the vector are initialized with \(\langle 0.8, 0.3\rangle \). the last position is always set to \(\langle 0.9, 0.9\rangle \) and remains unchanged during the evolutionary process. at each generation one of the first \(h-1\) positions are circularly updated based on the euclidean distance between the coordinates of offspring and their parents. also, very low values of \( cr \) and very high values of f are not allowed in early stages of the search. it is important to note that only successful trial vectors in a are used in the adaptation of parameters, since only in this sub-population the three strategies are used for each individual. 3.3 fitness evaluations to estimate the quality of the boxes that represent individuals in the population, we use ogre’s pruning and local search strategy. first, we enforce gac on individuals from the initial population or resulting from the operators mentioned in sect. 3.2. then, we try the instantiation of the entire cn in a backtrack-free fashion starting by the initial valuation \(y = \min y\), where y is the interval domain of the objective function \(f(\texttt {x})\). we allow the constraints in the \(\varomega \) set of the epiphytic decomposition to be instantiated regardless of the tolerance \(\varepsilon _{\varomega }\) required to satisfy them. in [5], the sum of all these tolerances is considered the constraint violation value \(\phi (\texttt {x})\) of the individual, while its cost is \(f(\texttt {x}) = \min y\). in this paper we assume that the instantiation of real parameters described above provides suitable reference points for evaluating interval individuals. however, to obtain the real values of \(f(\texttt {x})\) and \(\phi (\texttt {x})\), we only consider the instantiation of variables belonging to the original modeling of the instance, not the approximation given by the ternary cn employed in [5]. this allows us to compare results with recent black-box solvers that use the original instance modeling. the consistent individuals obtained by the gac contractor may contain multi-intervals. in [5], such multi-interval box was split into a set of interval individuals, adding to the population the one with the best fitness value (if it is better than the target vector) and saving all other generated interval individuals in the additional archive. as commented in the sect. 3.1, our tests revealed that this mechanism is computationally expensive and does not contribute to maintaining the population diversity. so, instead of splitting the multi-intervals and adding them to the archive, we use the interval hull of the box (the smallest interval box that contains all the multi-intervals) and consider it as only one individual which is compared with its parent in the selection operation. in order to compare two individuals, we apply the widely used \(\varepsilon \) constrained method [21]. the \(\varepsilon \) comparisons are defined as a lexicographic order in which \(\phi (\texttt {x})\) precedes \(f(\texttt {x})\). this precedence is adjusted by the parameter \(\varepsilon \) that is updated at each generation until the number of generations exceeds a predefined threshold. from this point the \(\varepsilon \) level is set to 0 to prefer solutions with minimum constraint violation. 4 experimental results our experimental evaluation considers four solvers: the improved approach proposed in this paper (i2de); an implementation that uses the same de interval operations and additional archive of the original inde [5]; a black-box version of the i2de (bbde); and ogre [8]. the aim of use the features of original inde is to measure the impact of the modifications incorporated in our approach. the black-box version does not use interval representation neither structural decomposition as the other three white-box solvers. this comparison aims to evaluate the impact of using the instance structure in optimization process. in turn, the comparative analysis with ogre is intended to provide a baseline with a search method that employs ib&b, local consistency and constraint propagation over the constraint network. it is important to note that local consistency and backtrack-free local search techniques are the same in i2de, inde and ogre. to ensure a fair comparison, the three de solvers employ the same heuristics coming from state-of-the-art solvers for adapting and adjusting parameters, as well as the same number of fitness evaluations. furthermore, the input parameters of the solvers were empirically defined after a series of preliminary experiments. it was set to \( maxfes = 20000 \times d\). the maximal and minimal population size was defined as \(\textit{np}_{ max } = 15 \times d\) and \(\textit{np}_{ min } = 6\), respectively, where d is the number of variables on the original instance. note that in [5], \( maxfes \) and \( np \) were defined in relation to \(|v_\varomega |\), which resulted in a much larger budget and population. the de/current-to-pbest/1 operator used \(p_{ max } = 0.25\) and \(p_{ min } = p_{ max }/2\). the length of historical memory was \(h = 5\). parameters of \(\varepsilon \) level were \(\theta = 0.7\), \(cp = 4\) and \(t = 0.85 \times g\), where g is the maximum number of generations. we used the exponential version of the classical crossover operator. gac contractor was applied to a maximum of 1000 iterations. the timeout for ogre was 10000s. since ogre parameters \(\varepsilon _{\varomega }\) and \(\vardelta \) are not trivially configured, and to have a similar methodology of execution to des, we ran 25 different configurations for each instance using \(\varepsilon _{\varomega } \in \{10^e\ |\ e = -5, \dotsc , 1\}\) and \(\vardelta \in \{10^e\ |\ e = -6, \dotsc , 0\}\), \(\vardelta \le \varepsilon _{\varomega }\). although most experimental evaluations of metaheuristic-based solvers use cec competition benchmarks, their instances are only available in source code to be used as black-box evaluation functions. therefore, they do not provide the necessary formalization for use in solvers that exploit structural information of the ncop instance, such as i2de. for the main experimental evaluation we used the coconut benchmark [19] due to its wide use in numerical optimization research and because it contains the ampl (a mathematical programming language) description necessary to explore the structure of the instances in a white-box approach. however, to provide a baseline with some state-of-the-art solversfootnote 3, we compared our black-box version of i2de (bbde) with solvers from cec2018 competition on constrained real-parameter optimization [27] in the 28 proposed benchmark functions with \(d=\{10,30,50,100\}\), totalizing 112 instances. the results of the cec2018 solvers were obtained from the competition recordsfootnote 4. bbde used the same competition protocol, with 25 runs for instance. table 1. comparison of bbde with solvers from cec2018.full size table the rank of table 1 considers the cec2018 [27] methodology based on mean and median score values on each instance, in which the best solver obtains the lowest total score. the results indicate that our black-box version is highly competitive with some of the most popular state-of-the-art solvers. to investigate the performance of i2de, experiments were conducted on 155 optimization problems from the coconut benchmark [19] with different number of equality and inequality constraints and up to 75 dimensions. note that it was only possible to tackle a larger number of functions than in [5] because our approach extends the implementation of ogre’s multi-interval core to other math operations, as commented in sect. 3. the first aspect to be analyzed is the impact of using structural decomposition. transforming the original instance modeling into a ternary cn usually involves adding an auxiliary variable for each occurrence of math operators. from this, the variables of constraints in the \(\varomega \) set, \(v_\varomega \), whose valuation is critical for the search process, are extracted. as noted in sect. 3.2, i2de and inde only apply its operators on these variables, while the other valuations are assigned through ogre’s backtrack-free local search. figure 1 illustrates the number of variables (and operators) from the original modeling, the ternary representation and the \(v_\varomega \) size in the 155 instances. although \(v_\varomega \) presents a considerable reduction in number of variables compared to the ternary representation, it is evident that the search space is still larger than in the original instance modeling. fig. 1. number of variables in original v, ternary v, \(v_\varomega \) and operators per instance. full size image the solvers ran on a computer with intel xeon e5-4627v2 3.30 ghz processor, 256 gb of ram and centos linux 7. the average time of the 25 executions of the 155 instances was \(7372.04\,s\) for i2de, \(5031.19\,s\) for inde, \(4.63\,s\) for bbde and \(5619.52\,s\) for ogre. the processing times of i2de, inde and ogre are significantly higher due to the local consistency process. moreover, the execution time of inde is smaller compared to i2de because many candidate solutions resulting from inde interval operations considered inconsistent are immediately discarded, without going through the local consistency process. i2de found feasible solutions on 119 instances, against 107 in inde, 115 in bbde and 64 in ogre. feasible solutions with value at most \(10^{-8}\) greater than the optimal value are considered as optimal solutions. so, i2de found 40 optimal solutions, while inde, bbde and ogre found 35, 35 and 5, respectively. in order to perform a comparative analysis among i2de, inde, bbde and ogre we used two methodologies. the first was the one applied in cec2018 [27], the same used in table 1. the second was proposed in cec2020 competition on non-convex constrained optimization problems from the real-world [14] and involves the objective function and constraints violation of the best, the mean, and the median solutions. the values are normalized and adjusted to calculate the performance measure. the total score is a weighted composition of best, mean and median with weights 50%, 30% and 20%, respectively. to estimate the feasibility of the four solvers, we compared the average constraint violation values of the 25 runs for the 155 benchmark instances. as the pre-analysis rejected the null hypothesis that the distribution of violation values of each instance is normal, we chose a non-parametric test. the friedman test with post-hoc nemenyi test with significance level of 0.05 was applied to the violation results of each instance, comparing four sets of 25 values each time. the null hypothesis that the four solvers perform equally was rejected in all instances with p-value close to \(10^{-6}\). based on the nemenyi test rank on each instance, we counted the number of instances where each solver was better, equal or worse than the others. if the rank distance of two solvers was less than the critical distance of the test, we considered the solvers perform equally in the instance. table 2 shows in its violation rows the number of instances where i2de was better, equal or worse than the other three solvers. in all three cases, i2de obtained more solutions with violation values smaller than the other solvers. a second statistical test was conducted over the normalized and adjusted objective function value of best, mean and median solution of the cec2020 methodology. the value of each solver in an instance is a composition of objective function value and the average constraint violations of the obtained solutions. as the pre-analysis rejected the null hypothesis that the distribution of each set of values is normal, we chose a non-parametric test. we applied the wilcoxon signed rank test with significance level of 0.05 to compare the solvers on each set of values: best, mean and median. table 2 shows the test results and the number of instances where i2de was better, equal or worse than the other solver. only in two cases the null hypothesis, that the two solvers perform equally, was not rejected: decision \(\approx \). in the other cases, i2de perform better: decision \(+\). according to both cec2018 and cec2020 ranking methodologies, the best solver will obtain the lowest total score value. table 3 shows that the i2de outperforms the other three approaches in both ranking methodologies. in comparison to inde, the results point to a significant performance improvement due to the reformulation of interval mutation strategies and the maintenance of population diversity with the additional archive of solutions. furthermore, the comparative analysis with bbde and ogre suggests that the hybridization between metaheuristics, constraint programming and structural decomposition is a promising research direction. table 2. statistical tests results and comparison of i2de with inde, bbde and ogre based on best, mean, median and violation scores over 155 coconut instances.full size table table 3. rank of i2de, inde, bbde and ogre according to cec2018 and cec2020 methodologies on coconut benchmark.full size table 5 conclusion in this work we proposed the i2de, an improved version of interval differential evolution that uses structural information to solve global optimization problems. from exploration of the epiphytic decomposition, it becomes possible to concentrate the search process only on a subset of variables that have critical valuation, while all the others are instantiated by propagation through the constraint hypergraph. additionally, our search is enhanced by a local consistency process that prunes values that certainly do not constitute the optimal solution. one of the main contributions of our approach is the extension of the multi-interval core of ogre. this allows us to tackle a greater diversity of benchmark functions and real-world problems that otherwise could not be adequately represented. we also proposed a reformulation of three de interval mutation operations and incorporated heuristics from several state-of-the-art solvers that contributed to improve the performance of our approach. the experimental analysis of the proposed i2de on the 155 functions selected from the coconut benchmark [19] showed that the reformulation of the interval mutation strategies and of the additional archive significantly improved the performance of the search method. furthermore, the results obtained in comparison with ogre and bbde reveal that, although the exploration of the instance’s structural information increases the size of the search space and the processing time, it considerably improves the quality of the solutions found. considering that bbde outperformed several state-of-the-art solvers and was overcome by i2de, we shown that the use of structural instance information in the context of metaheuristics is a promising research direction. some future work includes the implementation of other contractors that help to efficiently prune the intervals without loss of solutions. in addition, we intend to develop a hybrid cooperative approach that joins our interval metaheuristics with exact methods that also use this representation for the solutions. notes 1.a constraint is said to be ternary if it involves at most three variables. 2.the structure of a cn can be represented by a hypergraph which vertices are the variables and for each constraint there is a hyperedge connecting its respective vertices. therefore, a cn is acyclic if its hypergraph is berge-acyclic [2]. 3.lshade-iepsilon [9], \(\varepsilon \)mages [11], lshade44 [17], ude [23], iude [24], lshade+ide [25] and cal-shade [28]. 4.https://www3.ntu.edu.sg/home/epnsugan/index_files/cec2018/cec2018.htm, last accessed 18 jun 2021. references araya, i., reyes, v.: interval branch-and-bound algorithms for optimization and constraint satisfaction: a survey and prospects. j. glob. optim. 65(4), 837–866 (2016) article  mathscinet  google scholar  berge, c.: graphs and hypergraphs. elsevier science ltd., oxford (1985) google scholar  brest, j., maučec, m.s., bošković, b.: il-shade: improved l-shade algorithm for single objective real-parameter optimization. in: 2016 ieee congress on evolutionary computation (cec), pp. 1188–1195. ieee (2016) google scholar  brest, j., maučec, m.s., bošković, b.: single objective real-parameter optimization: algorithm jso. in: 2017 ieee congress on evolutionary computation (cec), pp. 1311–1318. ieee (2017) google scholar  cassenote, m.r.s., derenievicz, g.a., silva, f.: interval differential evolution using structural information of global optimization problems. in: moura oliveira, p., novais, p., reis, l.p. (eds.) epia 2019. lncs (lnai), vol. 11804, pp. 724–736. springer, cham (2019). https://doi.org/10.1007/978-3-030-30241-2_60 chapter  google scholar  cohen, d., jeavons, p.: the power of propagation: when gac is enough. constraints 22, 3–23 (2016) article  mathscinet  google scholar  dechter, r., van beek, p.: local and global relational consistency. in: montanari, u., rossi, f. (eds.) cp 1995. lncs, vol. 976, pp. 240–257. springer, heidelberg (1995). https://doi.org/10.1007/3-540-60299-2_15 chapter  google scholar  derenievicz, g.a., silva, f.: epiphytic trees: relational consistency applied to global optimization problems. in: van hoeve, w.-j. (ed.) cpaior 2018. lncs, vol. 10848, pp. 153–169. springer, cham (2018). https://doi.org/10.1007/978-3-319-93031-2_11 chapter  google scholar  fan, z., fang, y., li, w., yuan, y., wang, z., bian, x.: lshade44 with an improved \(\varepsilon \) constraint-handling method for solving constrained single-objective optimization problems. in: 2018 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2018) google scholar  hansen, e., walster, g.w.: global optimization using interval analysis. monographs and textbooks in pure and applied mathematics, new york (2004) google scholar  hellwig, m., beyer, h.g.: a matrix adaptation evolution strategy for constrained real-parameter optimization. in: 2018 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2018) google scholar  iorio, a.w., li, x.: solving rotated multi-objective optimization problems using differential evolution. in: webb, g.i., yu, x. (eds.) ai 2004. lncs (lnai), vol. 3339, pp. 861–872. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-30549-1_74 chapter  google scholar  jou, y.c., wang, s.y., yeh, j.f., chiang, t.c.: multi-population modified l-shade for single objective bound constrained optimization. in: 2020 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2020) google scholar  kumar, a., wu, g., ali, m.z., mallipeddi, r., suganthan, p.n., das, s.: a test-suite of non-convex constrained optimization problems from the real-world and some baseline results. swar evol. comput. 56, 100693 (2020) google scholar  mackworth, a.k.: on reading sketch maps. in: proceedings of the fifth international joint conference on artificial intelligence, ijcai 1977, pp. 598–606. mit, cambridge, ma (1977) google scholar  moore, r.e.: interval analysis. prentice-hall englewood cliffs, n.j (1966) google scholar  poláková, r.: l-shade with competing strategies applied to constrained optimization. in: 2017 ieee congress on evolutionary computation (cec), pp. 1683–1689. ieee (2017) google scholar  sallam, k.m., elsayed, s.m., chakrabortty, r.k., ryan, m.j.: improved multi-operator differential evolution algorithm for solving unconstrained problems. in: 2020 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2020) google scholar  shcherbina, o., neumaier, a., sam-haroud, d., vu, x.-h., nguyen, t.-v.: benchmarking global optimization and constraint satisfaction codes. in: bliek, c., jermann, c., neumaier, a. (eds.) cocos 2002. lncs, vol. 2861, pp. 211–222. springer, heidelberg (2003). https://doi.org/10.1007/978-3-540-39901-8_16 chapter  math  google scholar  storn, r., price, k.: differential evolution a simple and efficient heuristic for global optimization over continuous spaces. j. glob. optim. 11(4), 341–359 (1997) article  mathscinet  google scholar  takahama, t., sakai, s.: constrained optimization by the \(\varepsilon \) constrained differential evolution with an archive and gradient-based mutation. in: 2010 ieee congress on evolutionary computation, pp. 1–9. ieee (2010) google scholar  tanabe, r., fukunaga, a.s.: improving the search performance of shade using linear population size reduction. in: 2014 ieee congress on evolutionary computation (cec), pp. 1658–1665. ieee (2014) google scholar  trivedi, a., sanyal, k., verma, p., srinivasan, d.: a unified differential evolution algorithm for constrained optimization problems. in: 2017 ieee congress on evolutionary computation (cec), pp. 1231–1238. ieee (2017) google scholar  trivedi, a., srinivasan, d., biswas, n.: an improved unified differential evolution algorithm for constrained optimization problems. ieee (2018) google scholar  tvrdík, j., poláková, r.: a simple framework for constrained problems with application of l-shade44 and ide. in: 2017 ieee congress on evolutionary computation (cec), pp. 1436–1443. ieee (2017) google scholar  vanaret, c., gotteland, j.b., durand, n., alliot, j.m.: preventing premature convergence and proving the optimality in evolutionary algorithms. in: international conference on artificial evolution, pp. 29–40. springer (2013) google scholar  wu, g., mallipeddi, r., suganthan, p.: problem definitions and evaluation criteria for the cec 2017 competition on constrained real-parameter optimization. technical report (2017) google scholar  zamuda, a.: adaptive constraint handling and success history differential evolution for cec 2017 constrained real-parameter optimization. in: 2017 ieee congress on evolutionary computation (cec), pp. 2443–2450. ieee (2017) google scholar  zhang, j., sanderson, a.c.: jade: adaptive differential evolution with optional external archive. ieee trans. evol. comput. 13(5), 945–958 (2009) article  google scholar  download references author information authors and affiliations informatics department, federal university of paraná, curitiba, brazil mariane r. s. cassenote, guilherme a. derenievicz & fabiano silva authors mariane r. s. cassenoteview author publications search author on:pubmed google scholar guilherme a. derenieviczview author publications search author on:pubmed google scholar fabiano silvaview author publications search author on:pubmed google scholar corresponding author correspondence to mariane r. s. cassenote . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper cassenote, m.r.s., derenievicz, g.a., silva, f. (2021). i2de: improved interval differential evolution for numerical constrained global optimization. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_13 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords global optimization differential evolution interval methods publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature constructive machine learning and hierarchical multi-label classification for molecules design | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper constructive machine learning and hierarchical multi-label classification for molecules design conference paper first online: 12 october 2023 pp 276–290 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) constructive machine learning and hierarchical multi-label classification for molecules design download book pdf download book epub rodney renato de souza silva9 & ricardo cerri  orcid: orcid.org/0000-0002-2582-16959  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 541 accesses abstract constructive machine learning (cml) is a research field that uses algorithms to generate new instances, similar but not identical to existing ones. it has been widely used to assist the discovery of new drug-like molecules. this is very challenging, given that the search space is discrete, unstructured and enormous. in this work we use cml to learn the intrinsic rules of datasets of molecules to generate novel ones. the chosen cml methods can be divided in two sub groups, text-based and graph oriented. considering different possibilities to evaluate the methods and the generated molecules, we propose classifying generated molecules in a taxonomy, using a hierarchical multi-label classifier previously trained in a dataset of molecules with known taxonomy information. in this way, it is possible to predict properties and verify the relevance of the generated molecules to existing taxonomies. we also propose a hierarchical diversity measure to compare groups of molecules based on their taxonomy information. the measure showed coherent results and is faster to calculate than the commonly used external diversity measures. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others machine learning-aided generative molecular design article 18 june 2024 towards exploring large molecular space: an efficient chemical genetic algorithm article 30 november 2022 automated design of multi-target ligands by generative deep learning article open access 11 september 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization cheminformatics combinatorial chemistry machine learning structure-based drug design structure prediction 1 introduction one of the objectives of medicinal chemistry is to discover new molecules with drug-like characteristics. this is challenging, since searching for molecules with desired chemical properties involves a huge and discrete space. the number of possible candidates is overwhelming and makes impossible to brute force through all possible combinations. about \(10^8\) substances have been synthesized, whereas the range of potential drug-like molecules is estimated to be between \(10^{23}\) and \(10^{60}\) [14]. other problem is that the space is unstructured, i.e., adding, removing or changing atoms and bonds of a molecule changes its properties in an inconstant and hard to predict way. despite improvements in the synthesis and test areas, those procedures are still time consuming and expensive. the discovery process can take 10–15 years with an average cost of us$2.5 billion [9], and one-third of the time and money are spent during the early stages of the process. machine learning (ml) is relevant to manage the volume and complexity of chemical information. a single database (such as pubchem [18]) has more than 96 million chemical structures, with an even larger number of annotations (such as synonyms and properties), targets (drugs with which a molecule interacts), action (how the molecule interacts with its targets), and status within the regulatory-approval process [31]. ml seeks to generalize patterns observed in datasets and predicts unknown characteristics of unseen data. it has been applied in drug design to estimate properties of molecules and predict interactions with biological targets, considerably accelerating drug discovery [6]. constructive machine learning (cml) generates instances similar, but no identical, to known instances. it has shown outstanding results in many areas and is gaining increasing attention [12]. in recent years, cml has been applied to generate new molecules from scratch based on existing ones [24]. four challenges are observed when applying cml to de novo drug design [10]: i) definition of molecules representation; ii) architecture selection; iii) evaluation of approaches for molecules generation and optimization; and iv) design of a reward function, crucial for practical applications of reinforcement learning. although there is a variety of proposals to overcome these challenges, there are still open problems. traditional ml classifiers were developed for binary or multi-class problems. they are not suitable for real-world problems where instances are classified into many classes simultaneously. an example is hierarchical multi-label classification (hmc), where instances are classified into different classes in a taxonomy. these problems require hierarchical multi-label classifiers [30]. many works validate molecules using classifiers to predict desired molecules properties. however, there are no works applying hmc to classify de novo molecules into known properties within a taxonomy. this is important, since we can directly identify general and specific properties, efficiently comparing groups of molecules. thus, we propose to evaluate our molecules by classifying them into a taxonomy, using a hierarchical multi-label classifier previously trained using molecules with known taxonomy information. to generate molecules, we use the generative toolkit for scientific discovery (gt4sd) [28], and classify them using a decision tree [30] trained in the chebi molecules dataset [8]. for evaluation, we propose a hierarchical distance to compare groups of molecules based on their taxonomy information predicted by the hierarchical decision tree. our main contributions are: i) the study and comparison of cml models for molecules generation considering their taxonomy; ii) the creation of a library to process the chebi dataset for hierarchical classification; iii) the proposal of a new measure to compare groups of molecules based on their taxonomy. the remainder of this paper is organized as follows. section 2 presents related works; sect. 3 reviews fundamental related concepts; results are presented and discussed in sect. 4; finally, sect. 5 presents conclusions and future works. 2 related works recurrent neural networks (rnn) were applied to learn intrinsic molecules rules [5], comparing distributions of descriptors to verify if they were similar to existing ones. rnns were also applied to generate molecules from scratch and from scaffolds [15]. fine tuning was performed in smaller specific datasets with molecules with known biological targets. principal component analysis was applied to the descriptors of the training data and the generated data, showing that the distributions were very close. reinforcement learning (rl) has been extensively investigated due to its potential applications in drug discovery. in a recent study, the combination of rl and quantitative structure activity relationship (qsar) was explored as a reward function for identifying active molecules against the dopamine receptor type 2 (drd2) [24]. the study used rnn and qsar model to generate structures. the results showed that more than 95% of the structures generated were active against drd2, demonstrating the potential of rl in drug discovery. a method called organ combines generative adversarial networks (gans) with high quantitative estimate of drug-likeness (qed) [26]. it has three parts: a generator, a discriminator, and qed with lipinski’s rule-of-five as reward function. variational autoencoders (vaes) were also investigated [14]: i) sampling molecules next to known ones; ii) interpolating points from known molecules and converting them back; iii) using a property predictor to search optimal regions, converting points with high values into molecules. the first approach chooses a molecule as reference and samples points around it. the second chooses two molecules as references, smoothly interpolating molecules similar to them. the third approach showed that vaes trained with the property predictor led to regions where the chosen property is maximized. adversarial autoencoders (aaes) [17] were compared with vaes, showing that vaes have a greater coverage. allowing to trade-off between coverage of the original dataset with generated samples and reconstruction quality, aaes obtained results similar to vaes. graph convolutional policy network (gcpn) [34] used a generation process guided towards specified desired objectives, restricting the output space based on underlying chemical rules. for goal-directed generation, graph representation, reinforcement learning and adversarial training were used, extended and combined into an unified framework. a flow-based auto-regressive model (graphaf) was proposed [27], combining auto regressing flows [25] and graph representation learning. motifs were also used [21], which are fragments of molecules inferred from data. vaes were extended to work with graph molecular representation, and generated molecules including motifs, a bond or a atom per iteration. the proposal, called moler (learning to extend molecular scaffolds with structural motif), was argued to be faster than baseline methods. the method uses an off-the-shelf molecular optimization, combining state-of-the-art methods in unconstrained optimization. from the reviewed works, we generated molecules with the following methods: vae [14], organ [26], aae [17], moler [21], gcpn [34] and graphaf [27]. 3 methodology this section presents concepts, algorithms and data used in our experiments. section 3.1 introduces the concepts of molecules properties and sect. 3.2 presents three molecular representations used in our experiments. the gt4sd library and the clus system, used to generate and classify molecules, are described in sects. 3.3 and 3.4. section 3.5 provides information about the chebi taxonomy, used to train our classifier and validate molecules. the next two sections present metrics used to compare datasets of molecules. section 3.6 describes the internal and external diversity measures, while sect. 3.7 explains our proposed hierarchical distance. section 3.8 presents an overview of our methodology. 3.1 molecules properties each molecule has i) its own structure, i.e., how its atoms are organized; ii) its physical properties (descriptors), such as its weight and solubility; and iii) its biological properties, related to effects in biological targets [4]. cheminformatics tries to predict molecules physical properties using their chemical structure, and to predict biological targets using known properties [31]. although we can not say for sure if a molecule has pharmaceutic characteristics, we can observe physicochemical and structural properties within certain ranges. over 90% of molecules that reach the phase ii clinical status have four properties: i) mass lower than 500; ii) logp lower than 5; iii) number of hydrogen bond donors lower than 5; and iv) number of hydrogen bond receptors lower than 10. a molecule satisfying these restrictions is a strong candidate to be a drug. although these are not the only properties for the molecule to be a drug, they have shown to be very influential, and are known as rule of five (ro5) (or lipinski rule) [19]. even though the ro5 is predictive of oral bioavailability, 16% of oral drugs violate at least one of the criteria and 6% fail on two or more. an important estimated molecule property is the quantitative estimate of drug-likeness (qed), which is used to assess molecular target druggability by prioritizing bioactive compounds, directing the tests to most promising molecules [4]. quantitative structure activity relationship (qsar) [23] indicates how a chemical compound interacts with biological targets. it can be used as objective function, helping to predict how promising a molecule is to be a drug. in our experiments, we trained a hierarchical multi-label classifier using molecules extracted from the chebi database, having as features the above properties and other descriptors. these same features were extracted from the molecules we generated using cml. these molecules were then classified using the hierarchical classifier. since molecules with similar functions are close in the feature space [31], we evaluated our generated molecules using similarity measures, comparing groups of molecules generated by different cml algorithms. 3.2 molecules representation the representation of a molecule is a key factor to determine how cml models work and how similarities are evaluated. the simplified molecular input line entry system (smiles) is a two dimensional representation that uses ascii symbols: letters for atoms, = and # for bonds, parenthesis for ramifications and numbers for cycles; simple bonds and hydrogen atoms are sub intended. a positive characteristic of smiles is its ability to capture order and relation of atoms. however, it allows to represent a same molecule in many different ways, which difficulties comparisons. a same molecule can have different similarity scores depending on the chosen atom to start its sequence. also, a single character perturbation in the representation can lead to changes in the underlying molecular structure or even invalidate it [32]. we used this representation for some cml models in the generation stages, and to store molecules. fingerprints uses an array of predefined size, where each position indicates the presence or the absence of a molecular structure. its positive characteristics are that there is only one possible encoding for each molecule, and it is simple to compare vectors with fixed sizes. however, this representation cannot capture the order neither the relationship between the encoded structures. therefore, an encoding can be related to numerous molecules, making it impossible to reverse the codification [7]. we used this representation to compare molecules. the graph representation uses vertices and edges to represent atoms and bonds, providing high coverage of the chemical space, and conveying various chemical features. it is argued that it is the most intuitive and concise representation [33]. we used it in some cml models for the generation process. 3.3 gt4sd toolkit the generative toolkit for scientific discovery (gt4sd) [28] handles molecules, calculates their properties, provides a variety of pretrained models and a framework to train cml models. we used six pretrained models, each of them generating a set of 2000 molecules. they were trained in the chembl [13] dataset and made available by the gt4sd team. the first half of them uses smiles: variational auto encoder (vae) [14], objective reinforced generative adversarial networks (organ) [26] and adversarial auto encoders (aae) [17]. the second half process the molecules as graphs: molecular scaffolds with structural motifs (moler) [21], graph convolutional policy networks (gcpn) [34] and flow-based autoregressive model for molecular graph generation (graphaf) [27]. 3.4 clus system clusfootnote 1 is a decision tree and rule induction system based on predictive clustering trees. it unifies clustering and predictive modeling, and deals with complex prediction tasks such as multi-label classification. within clus, we used the hierarchical multi-label classifier clus-hmc [30] to learn the intrinsic rules of the chebi taxonomy. we then used clus-hmc to predict the taxonomy of our generated molecules based on their descriptors. 3.5 chebi dataset the chemical entities of biological interest (chebi) [8] is a dataset that contains information about molecules, such as smiles representation and descriptors. it uses a web ontology language (owl) [1] to describe a taxonomy of molecules and their classification. its information can be preprocessed and represented into a directed acyclic graph (dag) where each node represents a taxonomy label. it is also possible to prune the dag, leaving the most important labels and molecules. one can also store the taxonomy information, obtain statistics, and compare groups of molecules. the chebi dag has three ontologies with nodes belonging to one or more of them, inheriting semantic meaning with their connections. the first ontology classifies molecular entities and chemical substances. the second expresses a molecular role, which is a particular behavior that a material may exhibit. the third expresses atomic particles smaller than an atom, e.g., neutrons and protons. in order to improve the performance of the hierarchical classifier, we created a library to process and prune the original chebi dag. after pruning, we kept only molecules with role information, since they are more relevant for drug discovery, and only nodes with at least 100 annotated molecules. with this, we ended up excluding the particle graph and many nodes from the structural and role graphs, obtaining a final dag with 35032 molecules. table 1 shows the statistics of the pruned dag. the pruned dag contains 72 nodes shared between the two ontologies, where 37 are leaves. the table also presents the leaves with most molecules classified for each graph. as the analysis focus on the role information, the most populated leaf by molecules in the role graph has significantly more molecules than the most populated leaf in the structural graph. the hierarchical classifier is trained in the pruned chebi dag and used to predict the taxonomy of the molecules generated by the gt4sd toolkit. thus, the classifier may infer role or structural information of the generated molecules. table 1. pruned chebi taxonomy graph statistics.full size table 3.6 internal and external diversity measures to compare molecules based on their structural information, the fingerprints representation can be used with the tanimoto distance (eq. 1) [2], where a and b are molecules. the internal diversity measures the variety of the molecules in a given set. it is defined in eq. 2 as the mean of the sum of the tanimoto distance applied to every pair of molecules in a set, where a is a set of molecules, and a and b are molecules from a. the external diversity is analogous, with the main difference of using two different sets to define the pairs of molecules. it is presented in eq. 3, with a and b molecules from sets a and b respectively. the external and internal diversities have a complexity problem. as the sets of molecules grow in size, the time required to calculate the measures grows proportionally to the square of the sizes of the sets [3]. as external diversity is used to compare groups of molecules, we used it as a guideline to propose a faster measure to compare groups of molecules: the hierarchical diversity. $$\begin{aligned} t_d (a,b) = 1 \frac{a \cap b}{a \cup b} \end{aligned}$$ (1) $$\begin{aligned} d_i (a) = \frac{1}{|a|^2}\sum _{(a,b) \in a \times a} t_d (a,b) \end{aligned}$$ (2) $$\begin{aligned} d_e (a,b) = \frac{1}{|a||b|}\sum _{(a,b) \in a \times b}t_d (a,b) \end{aligned}$$ (3) 3.7 proposed hierarchical diversity measure our proposed diversity uses taxonomies to compare different groups of molecules. a dag is created for each set of molecules, and one is chosen as reference. each node from the dag stores how many molecules are classified in it, and the nodes are ordered constructing the first dag distribution. the second dag uses the same labels in the same order as the first dag to construct its distribution in an analogous way. finally, the distance of the distributions of molecules is measured using the wasserstein distance [29]. it is helpful to compare large groups of molecules since its complexity is way smaller than the external diversity. equation 4 presents the wassistein distance between distributions u and v. these are the distributions of the dags molecules over their nodes. $$\begin{aligned} w_d (u, v) = \inf _{\pi \in \gamma (u, v)} \int _{\mathbb {r} \times \mathbb {r}} |x-y| \textrm{d} \pi (x, y) \end{aligned}$$ (4) the hierarchical distance is asymmetric. since we need to choose one of the dags as reference, the results may be different depending on the chosen dag. however, our results in sect. 4 showed these differences were not significant. 3.8 methodology overview figure 1 presents an overview of our methodology. the chebi dataset, in a owl format (chebi.owl), is preprocessed into a dag, which is pruned and saved in two files: chebi.arff and chebi.obj. each method from the gt4sd library then generates a method.arff with molecules. clus-hmc classifier is then trained with chebi.arff molecules, making predictions for the molecules in the method.arff files. the predictions are stored in a method.pred.arff file, and converted into the method.obj format, which is used along with the chebi.obj to generate the results discussed in sect. 4. all our implementations and datasets are freely availablefootnote 2. fig. 1. illustration of our proposed methodology. full size image 4 results and discussions this section presents an analysis of our generated molecules. for each of the six generators from the gt4sd toolkit, we generated a group of 2 thousand molecules. first, we analyze some of the molecules properties, and visualize molecules using their descriptors space. next, we present the classification of the molecules by the clus-hmc classifier. internal and external diversities are then discussed, together with our proposed hierarchical distance. for each molecule, 21 descriptors were calculated, including molecular weight, related to the number and type of the molecule atoms, plogp, related to the solubility of the molecule in water, and all properties presented in sect. 3.1. table 2 presents information about some of the most important descriptors for each generated set of molecules. the sets generated by the aae model differs from the other sets, having in general higher mean and variance for the molecular weight and plogp descriptors. we can imply that many molecules in this set violate at least one rule of the ro5 rule. on the other hand, the set generated by gcpn showed the lowest values and variations for these two properties. table 2. statistics from three proprieties of the molecules sets.full size table to visualize the multiple dimensions of the molecules, we applied two dimensionality reduction methods: principal component analysis (pca) [16] and uniform manifold approximation and projection (umap) [22]. pca was applied in the a dataset with 12 thousand feature vectors (6 methods \(\times \) 2 thousand molecules), each one with 21 dimensions. pca was also applied followed by umap, which rearranges data based on its proximity. this rearrangement may help to visualize overlapped data and clusters of data. figure 2 illustrates how the molecules are distributed in the pca and umap spaces. the most outstanding group of molecules are the ones generated by aae and graphaf, while the other four groups appear to be very similar to each other. although many of the points in the pca space overlap, we can see where each group is located in relation to each other. in the umap space, the points hardly overlap, but it is hard to define a region for each group of molecules. fig. 2. 2d visualizations of the molecules feature vectors. full size image clus-hmc was trained with a 10-fold cross-validation strategy using the 35032 molecules extracted from the chebi dataset. we used the area under receiver operating characteristic (auroc) and the area under precision recall curve (auprc) to evaluate the classifier. auroc measures the ability of a classifier to distinguish between positive and negative classes, while auprc summarizes the relationship between precision and recall. table 3 shows the average auprc and auroc values obtained. the values are considerably high, except for the simple average auprc which does not consider class imbalance. table 3. hierarchical multi-label classifier results.full size table we classified the molecules into classes of the chebi taxonomy. tables 4 and 5 present information about the role and structural dags for the molecules generated by each method. although we can provide the information individually for each molecule, we show the results for groups of molecules in order to have a general view of the properties generated by each method. we can observe that the organ, vae and moler methods have close statistics, suggesting that their generated molecules are similar to each other. this same analysis can be done with gcpn and graphaf. the molecules generated by aae are the ones that differ the most from the other groups of molecules. table 4. role graph information of the chebi molecules set.full size table table 5. structure graph information of the chebi molecules set.full size table table 6 shows some classes predicted by the hierarchical classifier for the generated molecules, and the corresponding classification percentages. the classes include biochemical functions, metabolites and subclasses of metabolites, alkaloids and an alkaloid subclass. with the hierarchical classification, we can reduce the number of molecules to search, allowing a more localized and focused search on certain classes or chemical characteristics. some of the predicted classes have medicinal and pharmacological potential, such as harmala alkaloids [20], and are present in pharmaceutical drug compounds, such as metabolites [11]. table 6. percentage of molecules classified in some classes of the chebi taxonomy.full size table we used internal and external diversity to verify the similarities of the groups of molecules. the main diagonal of table 7 shows the internal diversity, while the external diversity is shown in the other cells. three main observations can be made: i) the graphaf and gcpn molecules have the highest internal diversities; ii) the molecules generated by graphaf and gcpn have the highest external diversities; and iii) the molecules generated by aae and vae have the smallest external diversities. these observations are useful to interpret the relations between the sets of molecules, and as a baseline to experiment if our proposed hierarchical distance is also capable to capture these relations. table 7. internal and external diversity between sets of molecules.full size table given the difficulty to visualize groups of molecules in fig. 2, fig. 3 shows only molecules generated by organ and vae. they were chosen to validate how our hierarchical distance evaluated two similar sets of molecules. since the groups are close, a small value is expected. figure 4 shows the distributions of the same molecules over the dag nodes. again, the distributions are very similar, and the hierarchical distance should be small. fig. 3. 2d visualizations of organ and vae generated molecules. full size image fig. 4. comparison of vae and organ molecules distributions. full size image figure 5 shows the visualization of the descriptors of the molecules generated by aae and graphaf. they were chosen to validate how the hierarchical distance differs two dissimilar sets of molecules. they are not so close to each other, and a high hierarchical distance is expected. figure 6 shows the distributions of the same molecules over the dag nodes. we see a sensitive dissonance, which must result in a high hierarchical distance. fig. 5. 2d visualizations of aae and graphaf generated molecules. full size image fig. 6. comparison of aae and graphaf molecules distributions. full size image table 8 shows our proposed hierarchical distance values for every pair of sets of molecules. from the results, we can separate the molecules in two clusters. the first one contains molecules generated by graphaf and gcpn, showing relative small distance between them and high distances to the other groups. the second cluster is formed by the other molecules, with high similarity among them and high distances to the first cluster. the results of table 8 are in consonance with the ones from table 7, except that diversity values are constrained between 0 and 1. the hierarchical distance varies according to the dag sizes and the dissonance between the sets of molecules. also, in accordance to what we can see in figs. 3, 4, 5 and 6, table 8 shows small hierarchical distance differences between the groups of molecules generated by organ and vae, and high differences between the groups of molecules generated by aae and graphaf. table 8. hierarchical distance between the molecules sets.full size table 5 conclusions and future works in this paper we presented a study on constructive machine learning applied to de novo drug-like molecules design. we generated molecules using six different methods from the gt4sd toolkit, and compared them with internal and external diversity measures. we proposed to use a hierarchical multi-label classifier to classify the molecules into classes of a known taxonomy, and compared groups of molecules based on their classification. for this comparison, we proposed a new hierarchical diversity measure that showed to be consistent in comparing groups of molecules based on their taxonomy information. the chebi dataset was used to train a hierarchical multi-label classifier, since it contains taxonomy information from a variety of molecules, and is much smaller than other databases. our proposal can be used to expand the chebi dataset by classifying new molecules in its taxonomy. it also can be used in drug discovery, classifying the generated molecules to narrow down the number of candidates, and selecting the molecules with specific chemical characteristics. once we classified groups of molecules into taxonomies, our proposed hierarchical distance obtained results similar to those from external diversity, but in a fraction of the time. external diversity has a limited range between 0 and 1, whereas our hierarchical distance can vary based on the size and diversity of the compared groups. the experiments showed that the hierarchical distance is more meaningful when calculated for similar sized groups of molecules. as future works, the hierarchical distance could be extended to other contexts, to compare groups of different molecules that can be classified into a taxonomy. also, we intend to investigate how the selected molecules can be used to fine tune their generators, in order to create a specific domain generation model. notes 1.https://dtai.cs.kuleuven.be/clus/. 2.https://github.com/yendorr/gt4sd. references antoniou, g., harmelen, f.v.: web ontology language: owl. in: handbook on ontologies, pp. 67–92. springer (2004) google scholar  bajusz, d., rácz, a., héberger, k.: why is tanimoto index an appropriate choice for fingerprint-based similarity calculations? j. cheminform. 7 (2015) google scholar  benhenda, m.: chemgan challenge for drug discovery: can ai reproduce natural chemical diversity? arxiv preprint arxiv:1708.08227 (2017) bickerton, g.r., paolini, g.v., besnard, j., muresan, s., hopkins, a.l.: quantifying the chemical beauty of drugs. nat. chem. 4(2), 90–98 (2012) article  google scholar  bjerrum, e.j., threlfall, r.: molecular generation with recurrent neural networks (rnns). arxiv preprint arxiv:1705.04612 (2017) brown, n., ertl, p., lewis, r., luksch, t., reker, d., schneider, n.: artificial intelligence in chemistry and drug design. j. comput. aided mol. des. 34(7), 709–715 (2020). https://doi.org/10.1007/s10822-020-00317-x article  google scholar  cao, d.s., xu, q., hu, q., liang, y.z.: manual for chemopy (2013) google scholar  degtyarenko, k., et al.: chebi: a database and ontology for chemical entities of biological interest. nucleic acids res. 36(suppl 1), d344–d350 (2007) article  google scholar  dimasi, j.a., grabowski, h.g., hansen, r.w.: innovation in the pharmaceutical industry: new estimates of r &d costs. j. health econ. 47, 20–33 (2016) article  google scholar  elton, d.c., boukouvalas, z., fuge, m.d., chung, p.w.: deep learning for molecular design-a review of the state of the art. mol. syst. des. eng. 4(4), 828–849 (2019) article  google scholar  evans, l., phipps, r., shanu-wilson, j., steele, j., wrigley, s.: methods for metabolite generation and characterization by nmr. in: ma, s., chowdhury, s.k. (eds.) identification and quantification of drugs, metabolites, drug metabolizing enzymes, and transporters (second edition), pp. 119–150. elsevier, amsterdam, second edition. (2020) google scholar  foster, d.: generative deep learning: teaching machines to paint, write, compose, and play. o’reilly media (2019) google scholar  gaulton, a., et al.: the chembl database in 2017. nucleic acids res. 45(d1), d945–d954 (2017) article  google scholar  gómez-bombarelli, r., et al.: automatic chemical design using a data-driven continuous representation of molecules. acs cent. sci. 4(2), 268–276 (2018) article  google scholar  gupta, a., müller, a.t., huisman, b.j., fuchs, j.a., schneider, p., schneider, g.: generative recurrent networks for de novo drug design. mol. inf. 37(1–2), 1700111 (2018) article  google scholar  jolliffe, i.: principal component analysis. encyclopedia of statistics in behavioral science (2005) google scholar  kadurin, a., nikolenko, s., khrabrov, k., aliper, a., zhavoronkov, a.: drugan: an advanced generative adversarial autoencoder model for de novo generation of new molecules with desired molecular properties in silico. mol. pharm. 14(9), 3098–3104 (2017) article  google scholar  kim, s., et al.: pubchem 2019 update: improved access to chemical data. nucleic acids res. 47(d1), d1102–d1109 (2019) article  google scholar  lipinski, c.a.: lead-and drug-like compounds: the rule-of-five revolution. drug discov. today technol. 1(4), 337–341 (2004) article  google scholar  marwat, s.k., ur rehman, f.: medicinal and pharmacological potential of harmala (peganum harmala l.) seeds. in: preedy, v.r., watson, r.r., patel, v.b. (eds.) nuts and seeds in health and disease prevention, pp. 585–599. academic press, san diego (2011) google scholar  maziarz, k., et al.: learning to extend molecular scaffolds with structural motifs. arxiv preprint arxiv:2103.03864 (2021) mcinnes, l., healy, j., melville, j.: umap: uniform manifold approximation and projection for dimension reduction. arxiv preprint arxiv:1802.03426 (2018) mitchell, j.b.: machine learning methods in chemoinformatics. wiley interdisc. rev. comput. mol. sci. 4(5), 468–481 (2014) article  google scholar  olivecrona, m., blaschke, t., engkvist, o., chen, h.: molecular de-novo design through deep reinforcement learning. j. cheminform. 9(1), 48 (2017) article  google scholar  papamakarios, g., pavlakou, t., murray, i.: masked autoregressive flow for density estimation. in: advances in neural information processing systems, vol. 30 (2017) google scholar  sanchez-lengeling, b., outeiral, c., guimaraes, g.l., aspuru-guzik, a.: optimizing distributions over molecular space. an objective-reinforced generative adversarial network for inverse-design chemistry (organic). chemrxiv (2017) google scholar  shi, c., xu, m., zhu, z., zhang, w., zhang, m., tang, j.: graphaf: a flow-based autoregressive model for molecular graph generation. arxiv preprint arxiv:2001.09382 (2020) team, g.: gt4sd (generative toolkit for scientific discovery) (2022) google scholar  vallender, s.: calculation of the wasserstein distance between probability distributions on the line. theor. probab. appl. 18(4), 784–786 (1974) article  google scholar  vens, c., struyf, j., schietgat, l., džeroski, s., blockeel, h.: decision trees for hierarchical multi-label classification. mach. learn. 73(2), 185–214 (2008) article  math  google scholar  wegner, j.k., et al.: cheminformatics. commun. acm 55(11), 65–75 (2012) article  google scholar  weininger, d.: smiles, a chemical language and information system. 1. introduction to methodology and encoding rules. j. chem. inf. comput. sci. 28(1), 31–36 (1988) article  google scholar  xiong, j., xiong, z., chen, k., jiang, h., zheng, m.: graph neural networks for automated de novo drug design. drug discovery today 26(6), 1382–1393 (2021) article  google scholar  you, j., liu, b., ying, z., pande, v., leskovec, j.: graph convolutional policy network for goal-directed molecular graph generation. in: advances in neural information processing systems, vol. 31 (2018) google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. the authors also thank the brazilian research agencies fapesp and cnpq for financial support. author information authors and affiliations department of computer science, federal university of são carlos, rodovia washington luis, km 235, são carlos, sp, brazil rodney renato de souza silva & ricardo cerri authors rodney renato de souza silvaview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar corresponding author correspondence to ricardo cerri . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de souza silva, r.r., cerri, r. (2023). constructive machine learning and hierarchical multi-label classification for molecules design. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_19 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords de novo drug design hierarchical multi-label classification constructive machine learning publish with us policies and ethics profiles ricardo cerri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving rule based and equivalent decision simplifications for bloat control in genetic programming using a dynamic operator | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving rule based and equivalent decision simplifications for bloat control in genetic programming using a dynamic operator conference paper first online: 28 november 2021 pp 234–248 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) improving rule based and equivalent decision simplifications for bloat control in genetic programming using a dynamic operator download book pdf download book epub gustavo f. v. de oliveira10 & marcus h. s. mendes10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 749 accesses abstract bloat is a common issue regarding genetic programming (gp), specially noted in symbolic regression (sr) problems. due to this, gp tends to generate a huge amount of ineffective code that could be avoided or removed. code editing is one of many approaches to avoid bloat. the objective in this strategy is to mutate or remove subtrees which do not contribute to the final solution. two known methods of redundant code removal, the rule based simplification (rbs) and equivalent decision simplification (eds) are extended in a new operator presented in this paper, called dynamic operator with rbs and eds (dore). this operator gives the algebraic simplification table used by rbs the potential to learn from reductions performed by eds. an initial benchmark highlighted how the rbs table can grow as much as 86% with dore, and reducing the time spent on simplification by 16.83%. experiments with the other three sr problems were performed showing a considerable improvement on fitness of the generated programs, besides a slight reduction in the population of the average tree size. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others simplification of genetic programs: a literature survey article open access 27 april 2022 simplifying dispatching rules in genetic programming for dynamic job shop scheduling chapter © 2022 the inefficiency of genetic programming for symbolic regression chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms evolvability reprogramming symbolic ai theory and algorithms for application domains control structures and microprogramming 1 introduction symbolic regression (sr) is one of the main applications and motivators to genetic programming (gp), a method for automatically generate computer programs from a high level definition of a problem [15]. since early 90ś, many data-driven problems are modeled as sr problems [2], where no previous knowledge or pre-processing input is required. gp is well suited for resolving such problems since any algebraic function set can be effectively represented as trees and implemented as computer programs for the problem domain [9]. bloat the uncontrolled and excessive growth of individuals without a proportional gain of fitness is a well-known issue and a field of study in gp. it is specially noted in sr. the large amount of inefficient code causes excessive consumption of computational resources, as well as many other practical issues [16], hiding the problems real complexity and domain. there are many approaches for avoiding uncontrolled code growth in tree-based gp [5], some of which are presented as follows. the most simple and popular of them is implementing a depth limit [9], although is not a truly effective approach, as it can induce growth in some scenarios. parsimony pressure [9, 13] is another popular technique which adds a penalty term in fitness function to punish large trees. pseudo-hill climbing [6] attempts to guarantee the fitness to improve in population rejecting individuals least fit than its parents until one is finally accepted. code editing approach mutates or removes redundant sub-trees in individuals. regarding the code editing approach, koza [9] proposed a simple method to simplify (to convert a tree into a smaller, equivalent, tree) using grammar rewrite rules, which further inspired the rule based simplification (rbs) method [7, 11, 20]. rbs was extended with equivalent decision simplification (eds) [11], which recursively compares all sub-trees in an individual for equivalency with a small set of terminals. this paper proposes the dynamic operator with rbs and eds (dore), which improves a code editing bloat control algorithm using both rbs and eds to maximize its reduction potential without greater punishments in execution time. the main feature of dore is to dynamically learn redundant expressions to be applied with rbs from eds outputs. it also optimizes the access of rbs rules using the any keyword in a hash-table implementation, as well as introduces a warm-up stage to grow rbs rules before the evolutionary process begins. the article is organized as follows: sect. 2 provides a quick overview about code editing operators; in sect. 3 an improved strategy using the previous operators is proposed; sect. 4 shows how rbs and eds were implemented and the technical resources utilized in benchmarks; sect. 5 shows a performance comparison between the simplification operators, beside the empirical results and discussion over three sr benchmark problems; and finally in sect. 6, the conclusion is presented and the possibilities for further research. 2 background redundancy is one of the key contributors to inefficient code growth. eds [11] was introduced to extend rbs as they complement each other. rbs removes redundant subtrees by replacing a tree for a smaller equivalent one by applying arithmetic rules such as \(x/1 \rightarrow x\) and \(0*x \rightarrow 0\). these rules must be known and provided before the evolutionary process starts, thus each rule must be explicitly specified. another limitation by rbs is that it’s rules must be specified exactly like it would appear in an individual. for example, the rule \(x * 0 \rightarrow 0\) would not be sufficient to simplify a tree \(0 * x\) unless the rule \(0 * x \rightarrow 0\) is specified. eds extends rbs in a manner that it simplifies trees without previous knowledge of algebraic rules. it can also remove redundancies that are only true in the training domain. the simplification by eds is made by evaluating each subtree in an individual and comparing these subtrees with a set of small trees or terminals which is usually the result of simplifications, such as x, 1 and 0. in a sr problem, eds is evaluated as follows [11]: 1. determine a suitable set of simple trees \(s_{simple}\). 2. check all subtrees in the target for equivalence to a tree in \(s_{simple}\). 3. if some subtree is equivalent to a tree in \(s_{simple}\), and larger than it, replace that subtree with the simple tree. 4. repeat this procedure recursively until it fails. finding a suitable set of trees \(s_{simple}\) for a problem is not an easy task. usually the set of terminals is a natural fit for \(s_{simple}\), but if it is already known that some subtrees must appear in the final output, such as in trigonometrical problems, they can be inserted in this set. however, the main issue of using eds as a single operator of bloat control is the computational performance. with problems hard enough and individuals big enough, reducing a single generation can take as long as multiple generational evaluations. this scenario is not uncommon in simple sr problems. if an evaluated individual with eds contains more subtrees than the population size, the evaluation function will be called upon as much as in the generational evaluation. 2.1 simplification with rbs and eds the flow of simplification using both rbs and eds is as follows [11]: 1. let the genotype tree of individual i be \(t_{i}\). 2. apply rbs recursively to all nodes of \(t_{i}\), until there is no node to which rbs can be applied, obtaining \(t'_{i}\). 3. apply eds to all nodes of \(t'_{i}\). if any node is translated, let the translated tree be \(t_{i}\) and go to (2). if there is no node to which eds can be applied, finish and let \(t'_{i}\) be the final result. applying rbs before applying eds is a good idea since it prevents eds to be used for simplifying already known rules. also, rbs execution time is lower than eds, which will be explored further in this paper. 3 proposed improvements the main idea of the improvements to the simplification with rbs and eds flow is to increase rbs rules table r dynamically as soon as new rules, or more efficient ones, are discovered. this allows rbs to execute independently with no large impacts in execution time. a more robust rbs operator also helps in simplification itself, since any simplification made by eds triggers new rbs calls, as well as new eds calls. if simplifications made by eds occurred in the first step of the simplification flow, then it would eliminate the need to reevaluate subtrees that would only be reduced by eds. the first improvement proposed is to allow the simplification rules table r, used by rbs, to have rules inserted in execution time. each time a subtree is simplified by rbs using the keyword \(\_any\_\), which denotes any subtree, a new rule with the original subtree as input is inserted in r. afterwards, new calls to rbs with the same subtree as input have o(1) access guaranteed in a hash-table implementation. analogously, each simplification made with eds creates a new rule in r table with the learned simplification. this way, there is no need to apply simplification with eds to the same subtree in subsequent individuals. algorithm 1 shows an example algorithm to simplify a generation using this improved simplification flow with rbs and ebs in a sr problem. another strategy adopted to grow even more the r table was the insertion of a warm-up step before the evolutionary process begins. this warm-up step consists of generating random trees in the training domain of a problem, with random sizes, and then simplifying them with the improved flow. each successful reduction creates a new rule in the table, so the gp starts with a larger r table. 4 methodology all algorithms were implemented in python 3.8.3 with the package deap [4] (distributed evolutionary algorithm in python) in version 1.3.1. parallelism was used to evaluate fitness functions using the package ray [12] in version 0.8.6. the benchmarks were conducted in a 2018 macbook pro with 2.2 ghz 6-core intel core i7 and 16 gb 2400 mhz ddr4 memory. 4.1 rule based simplification the rbs simplification rules table r was implemented as a hash table. the keyword \(\_any\_\) denotes any subtree and allows rules such \(\_any\_ * 0 \rightarrow 0\) to be defined. in the original paper [11] this keyword was referenced as a and no further details were provided on how it was implemented. in this paper, every subtree evaluated by rbs consults the hash table directly and only if no rule with the subtree is described the \(\_any\_\) rules are consulted. the access to the table r has complexity o(1), except on the scenario where the \(\_any\_\) keyword is included in the rule body, which makes the complexity o(n) where n is the amount of rules in the table. reducing the amount of rules with \(\_any\_\) is a slight improvement to this implementation as it maximizes the rate of o(1) accesses. another way to attain this goal is to define as many redundant rules as possible in r, for example prioritizing the definition of rules \(1 * 0 \rightarrow 0\) and \(x * 0 \rightarrow 0\) instead of \(\_any\_ * 0 \rightarrow 0\). this task can be hard when there is no previous knowledge of such simplifications in the domain of the problem. 4.2 equivalent decision simplification the subtrees set \(s_{simple}\) for comparison for equivalence in eds was implemented as a simple list. two subtrees is considered equivalent if all fitness values elapsed from this subtree in the domain has relative error \(\epsilon < 0.005\). this threshold was chosen after some tests and set to all benchmark problems in both compared operators in this paper. 5 experimental results this section is organized as follows: sect. 5.1 shows a preliminary benchmark, validating the performance improvement on each simplification method; it is presented in sect. 5.2 the results for three artificial sr problems using the proposed operator, comparing it with the original operator and baseline gp. 5.1 simplification methods performance a simple benchmark using \(cos(2\pi x)\) as objective function was ran 50 times to compare the execution time of the five simplification strategies. experimental setting. the experiment was the simplification of many independent individuals in each reduction strategy: traditional versions of rbs and eds, rbs and eds with dynamic rules allowed and rbs after the warm-up step. more details of each strategy were discussed in sect. 3. the fitness function was defined as the rmse (root mean squared error), of 20 uniform points in the interval \([-\pi , \pi ]\). the function set used was \(\{+. -. *, \div \}\), where \( \div \) is the protected division satisfying \(x / 0 \rightarrow 1\). the terminal set is \(\{x, 0, 1, \pi \}\). for each execution, 200 trees are generated and then cloned in each step to avoid any bias or propagation of simplifications. the reductions applied in one step are not carried on to the next. the procedure is described as: 1. reduction with rbs is applied to the 200 cloned individuals. 2. reduction with eds is applied to the 200 cloned individuals. 3. with the dynamic insertions in the rules table r allowed, reduction with rbs is applied to the 200 cloned individuals. 4. with the dynamic insertions in the rules table r allowed, reduction with eds is applied to the 200 cloned individuals. 5. with the dynamic insertions in the rules table r allowed, a warm-up step is done, and reduction with eds is applied to the 200 cloned individuals. the 200 individuals are generated in each of 50 executions with the method ramped half and half and with depth limited to [5, 10]. the warm-up step applied rbs and eds sequentially to 200 trees generated with grow method and with depth limited to [1, 5]. the initial rbs rules table r is defined in table 1. table 1. initial simplification rules table used by rbsfull size table results and discussion. table 2 shows the arithmetic mean of the 50 executions in each strategy. dynamic rbs is 4.84% faster than the default rbs. the rbs after the warm-up is shown to be 16.83% faster than the traditional rbs and provided a 11.52% improvement to dynamic rbs before the warm-up. table 2. execution time comparison of the simplification operatorsfull size table this benchmark indicates that not only a bigger r table can improve the simplifications performance but also how much slower eds is compared to rbs as well. with no improvements, eds is 2.09 times slower than rbs. dynamic eds is 1.84 times slower than dynamic rbs before warm-up. dynamic rbs after the warm-up can be twice as fast as the dynamic eds. the warm-up increased the rules of table r, in average, from 19 entries to 135, to an around 86% improvement. 5.2 symbolic regression problems to validate the proposed improvements, 3 simple artificial sr problems were tested. these benchmarks were extracted in [5], selected from [8, 10, 17,18,19]. since this paper presents a preliminary analysis of the operator, the benchmark problems are rather simple and harder problems will be explored in future works. the problems are presented in table 3. experimental setting. for each problem, 50 independent executions were performed, in 3 different approaches: base gp: with no code editing bloat control operator, baseline operator: with the original simplification flow of rbs and eds as in [11], and dore: using the proposed dynamic operator. table 3. artificial symbolic regression problemsfull size table the function set to all benchmarks is set to \(\mathcal {f}=\{+, -, *, \div \}\) where \(\div \) is the protected division satisfying \(x / 0 \rightarrow 1\). the terminal set to benchmarks 1 and 2 is \(\tau = \{0, 1\}\) and the decision variables. for benchmark 3, the terminal set is \(\tau = \{0, 1, \pi \}\) and the decision variable. to all benchmarks, the fitness function is the rmse of the training domain points. besides the traditional gp parameters, two new parameters were introduced to fit the operator’s context in more computational resource expansive problems. the first one is the percentage of population \(\rho \) which passes through the full simplification flow. the algorithm is developed in such way that the best \(\rho \%\) individuals of the population are chosen each generation. the second additional parameter is the remaining best \(\gamma \) percentage of the population that was not previously chosen and passes through the reduction with rbs only. for example, if the population size is 100, and the values \(\rho = 40\%\) and \(\gamma = 20\%\), the 40 most fit individuals would pass through the complete reduction flow and the remaining best 20 trees would be reduced with rbs only. these additional parameters are needed due the high computational cost of the reductions that would make unfeasible the operation of all individuals without a great impact in execution time. a large \(\gamma \) value does not have a high impact on the execution time, since rbs reduction is simpler. nonetheless, the \(\rho \) value greatly impacts the total execution time. for the benchmarks these parameters were chosen empirically, testing values big enough that would not make the total gp execution time significantly slower than the base gp. a custom tournament selection, named as partial tournament, is used to reduce the number of repeated individuals in the crossover. being p the population, the partial tournament does |p| traditional k size tournaments, but limits the repeated number of each individual to two. then, if necessary, the population is completed with individuals chosen at random up to |p|. table 4. gp settings used in symbolic regression benchmarksfull size table the gp settings used is presented in table 4. the simplification rules used in rbs are as described in table 1. the simple subtrees set \(s_{simple}\) used in eds is the terminal set \(\tau \) for each problem. the warm-up step used in dore runs is defined as 200 full trees with depth limited in [1, 5] range. results and discussion. the median was preferred over the mean in this section since it is less sensitive to outliers and the data is not guaranteed to follow a normal distribution. in all of the data in any graph or table the median of the 50 executions is shown. it was first analyzed how the fitness behaved over generations. figure 1 shows how base gp is worse than the others in all benchmarks. it also presents how base gp fitness curve tends to flatten in each benchmark. dore has slightly better results than the baseline operator. figure 2 shows a fitness boxplot of the best individuals in each generation. fig. 1. best fitness versus generations, for all benchmark problems full size image fig. 2. boxplots of the best of generation individuals, for all benchmark problems full size image the size of individuals is a crucial concern to have bloat-controlled gp executions, beside the fitness stagnation. a tree depth limit was used, so it was not expected a completely uncontrolled growth on base gp. as fig. 3 shows, both operators had a much better size control on the population over generations than base gp. the difference between dore and baseline operator was tiny, with dore having best results on the first two benchmarks. table 5 shows the validation and test domains used to analyze the best individuals from each generation for all benchmark problems. it is presented in table 6 the median and median average deviation (mad), as utilized in [1], of each set of points training, validation, and test using the rmse as reference. table 7 shows the same metrics for the tree size of these best individuals. as the data is not normally distributed, table 6 and table 7 also present the p-value of the mann-whitney u-test [3] considering dore x base gp and dore x baseline operator datasets. the null hypothesis is that the distributions of both datasets are equal. dore performed better regarding tree size in all benchmark problems, compared to the other two strategies. it also presented better results in training error in all benchmarks. the benchmark 3 test error was worse using an operator, which suggests that this kind of strategy is not well suited for more complex problems even though the generated trees were smaller. fig. 3. tree median size versus generation full size image besides the best trees found for each generation, the best individuals of each run were also analyzed. figure 4 shows the mean training fitness against the mean tree sizes. it is clear that dore dominates the other strategies in all 3 benchmark problems regarding the fitness. in benchmark 1, due to bloat, base gp generates smaller but unfit individuals. these individuals are, in average, from earlier generations than the ones using code editing operators, see fig. 5. the boxplot shows how base gp converges earlier than the other two strategies. thus, it is expected that trees generated in later generations have better fitness but also a larger size. dore performed better than the other strategies in benchmarks 2 and 3 and regarding both size and fitness, as shown in table 6 and table 7. in all benchmarks it is noted that the average size of the best individuals in dore are better than the ones with the baseline operator, see table 7. table 5. validation and test domain for best of run individuals analysisfull size table another point of interest analyzed was the reductions made between baseline operator and dore. each rbs and eds simplification made was logged in each run in these strategies. the tables and graphs related to reductions used the arithmetic mean as an average value. table 8 highlights how dore increased the number of unique simplifications made by rbs. this result was expected since dore tends to increase the reductions table used by rbs. table 6. best individuals for generation performance, of all benchmark problemsfull size table table 7. tree size data of best individuals for generation, of all benchmark problemsfull size table fig. 4. best of run average fitness against best of run average size full size image fig. 5. best of run generation boxplot full size image table 8. average value of unique rbs simplifications performedfull size table table 9. average total simplifications made by rbs and edsfull size table the average value of total reductions made by rbs and eds is shown in table 9. these graphs present different scenarios. in benchmark 1, the total reductions made by both rbs and eds in dore was greater than in baseline operator. in benchmark 2, the reductions by rbs were greater in dore but lesser by eds. finally, in benchmark 3, the total reductions made by both rbs and eds was greater in the baseline operator. these different scenarios are directly related to the nature of each benchmark problem. also, although the rbs reduction table in dore had more entries than the baseline table, each simplification called by eds triggered another recursive rbs simplification attempt on each subtree in the individual. thus, in problems where the number of simplifications by eds is high and with an increased average size reduction the number of reductions by rbs tends to be higher as well. even though each benchmark presented a different reduction scenario, a pattern in the ratio between reductions by rbs and reductions by eds could be noted, as is shown in fig. 6. in all benchmark problems, except the proportion, the total amount of rbs reductions relative to the total amount of eds reductions is bigger. it can also be noted that this ratio is greater in dore than in the baseline operator, which is exactly the main goal of the proposed improvements. fig. 6. average ratio of simplifications made full size image the impact of the dynamically learned rules was analyzed and shown in fig. 7. the graphs are histograms of the total amount of reductions made by rbs in each subtree size reduction percentage interval. the values highlighted in purple are from rules that could only be learned with dore improvements. the average size reduction of these dynamically learned rules in each benchmark were 86.01%, 88.01% and 85.40%, respectively, compared to the average size reduction of 59.38%, 55.98% and 52.74% from the predefined rules. dore not only provided more reductions to be applied but also better ones. fig. 7. rbs simplification frequency against size reduction percentage full size image finally the execution time was analyzed, an important point of interest in this work as dore execution time should be close to the baseline operator. average execution time using the arithmetic mean for all benchmark problems is shown in table 10. since the values of parameters \(\rho \) and \(\gamma \) were defined empirically to make execution time between baseline operator and dore close, the values presented are expected. however, the difference between base gp and executions with operators is notable. using rbs and eds operators as a single bloat control method seems to be not adequate in hard or complex problems as the execution time tends to increase even more due to all the fitness function evaluation performed recursively in eds step of the simplification flow. table 10. average total execution time (in seconds) of each strategy, for all benchmark problemsfull size table 6 conclusion and future work this paper presented improvements to a simplification flow introduced in [11] to make it more reasonable to be applied in complex problems. besides a meta-learning reduction, two parameters were introduced to optimize the number of trees to be reduced by rbs and eds in harder problems than the one presented in the original work, without greater impact in execution time. the resulting operator was able to improve the execution time performance of reduction by rbs, as well as the relative frequency of this reduction in the simplification flow, even with a limit of trees applied. a greater percentage of the population could pass through reduction by rbs with no significant penalty in execution time. the results of the three artificial sr problems benchmark highlighted how dore improved the fitness and slightly the tree size in the population during the evolutionary process. the best individuals in these experimental runs had significantly better results in both fitness and tree size using the dynamic operator. it is noted that dore is helpful to control bloat in gp but could be challenging to adopt in more complex problems due the computational burden. this issue should be addressed in future research. probably tuning the parameters \(\rho \) and \(\gamma \) may be needed as well as limiting the size of the subtrees in \(s_{simple}\). future work could measure the limits of the list \(s_{simple}\) and a robust strategy to handle random ephemeral constants, which was not explored in this paper. parallelization of the reductions could improve the operator performance, making it more viable, although is not a trivial problem as the rule table is constantly updated by each eds reduction performed. it would also be interesting to apply this operator with other bloat control methods, such as a equalization operator [14], using it as a tool to optimize a small set of individuals in a population. references castelli, m., manzoni, l., mariot, l., saletta, m.: extending local search in geometric semantic genetic programming, pp. 775–787, august 2019. https://doi.org/10.1007/978-3-030-30241-2_64 chen, c., luo, c., jiang, z.: block building programming for symbolic regression. neurocomputing 275, 1973–1980 (2018). https://doi.org/10.1016/j.neucom.2017.10.047 article  google scholar  fay, m.p., proschan, m.a.: wilcoxon-mann-whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules. stat. surv. 4, 1 (2010) article  mathscinet  google scholar  fortin, f.a., de rainville, f.m., gardner, m.a., parizeau, m., gagné, c.: deap: evolutionary algorithms made easy. j. mach. learn. res. 13, 2171–2175 (2012) mathscinet  google scholar  haeri, m.a., ebadzadeh, m.m., folino, g.: statistical genetic programming for symbolic regression. appl. soft comput. 60, 447–469 (2017) article  google scholar  hagiwara, m.: pseudo-hill climbing genetic algorithm (phga) for function optimization. in: proceedings of 1993 international conference on neural networks (ijcnn-93-nagoya, japan), vol. 1, pp. 713–716 (1993). https://doi.org/10.1109/ijcnn.1993.714013 hooper, d.c., flann, n.s.: improving the accuracy and robustness of genetic programming through expression simplification. in: proceedings of the 1st annual conference on genetic programming, p. 428. mit press, cambridge (1996) google scholar  keijzer, m.: improving symbolic regression with interval arithmetic and linear scaling. in: ryan, c., soule, t., keijzer, m., tsang, e., poli, r., costa, e. (eds.) eurogp 2003. lncs, vol. 2610, pp. 70–82. springer, heidelberg (2003). https://doi.org/10.1007/3-540-36599-0_7 chapter  google scholar  koza, j.r., koza, j.r.: genetic programming: on the programming of computers by means of natural selection, vol. 1. mit press, cambridge (1992) google scholar  mcdermott, j., et al.: genetic programming needs better benchmarks. in: proceedings of the 14th annual conference on genetic and evolutionary computation, pp. 791–798 (2012) google scholar  naoki, m., mckay, b., xuan, n., daryl, e., takeuchi, s.: a new method for simplifying algebraic expressions in genetic programming called equivalent decision simplification. in: omatu, s., rocha, m.p., bravo, j., fernández, f., corchado, e., bustillo, a., corchado, j.m. (eds.) iwann 2009. lncs, vol. 5518, pp. 171–178. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-02481-8_24 chapter  google scholar  moritz, p., et al.: ray: a distributed framework for emerging ai applications. corr (2017). http://arxiv.org/abs/1712.05889 poli, r., mcphee, n.f.: parsimony pressure made easy: solving the problem of bloat in gp. in: borenstein, y., moraglio, a. (eds.) theory and principled methods for the design of metaheuristics. ncs, pp. 181–204. springer, heidelberg (2014). https://doi.org/10.1007/978-3-642-33206-7_9 chapter  math  google scholar  silva, s., dignum, s., vanneschi, l.: operator equalisation for bloat free genetic programming and a survey of bloat control methods. genetic program. evol. mach. 13, 197–238 (2012). https://doi.org/10.1007/s10710-011-9150-5 article  google scholar  sivanandam, s., deepa, s.: genetic programming, pp. 131–163. springer, heidelberg (2008). https://doi.org/10.1007/978-3-540-73190-0_6 trujillo, l., muñoz, l., galván-lópez, e., silva, s.: neat genetic programming: controlling bloat naturally. inf. sci. 333, 21–43 (2015). https://doi.org/10.1016/j.ins.2015.11.010 article  google scholar  uy, n.q., hien, n.t., hoai, n.x., o’neill, m.: improving the generalisation ability of genetic programming with semantic similarity based crossover. in: esparcia-alcázar, a.i., ekárt, a., silva, s., dignum, s., uyar, a.ş (eds.) eurogp 2010. lncs, vol. 6021, pp. 184–195. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-12148-7_16 chapter  google scholar  uy, n.q., hoai, n.x., o’neill, m., mckay, r.i., galván-lópez, e.: semantically-based crossover in genetic programming: application to real-valued symbolic regression. genet. program evolvable mach. 12(2), 91–119 (2011) article  google scholar  vladislavleva, e.j., smits, g.f., den hertog, d.: order of nonlinearity as a complexity measure for models generated by symbolic regression via pareto genetic programming. ieee trans. evol. comput. 13(2), 333–349 (2008) article  google scholar  wong, p., zhang, m.: algebraic simplification of gp programs during evolution. in: proceedings of the 8th annual conference on genetic and evolutionary computation, pp. 927–934, gecco 2006. association for computing machinery, new york, ny, usa (2006). https://doi.org/10.1145/1143997.1144156 download references author information authors and affiliations icet, universidade federal de viçosa, florestal, brazil gustavo f. v. de oliveira & marcus h. s. mendes authors gustavo f. v. de oliveiraview author publications search author on:pubmed google scholar marcus h. s. mendesview author publications search author on:pubmed google scholar corresponding author correspondence to gustavo f. v. de oliveira . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de oliveira, g.f.v., mendes, m.h.s. (2021). improving rule based and equivalent decision simplifications for bloat control in genetic programming using a dynamic operator. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_16 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords genetic programming bloat control code editing publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature estimating code running time complexity with machine learning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper estimating code running time complexity with machine learning conference paper first online: 12 october 2023 pp 400–414 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) estimating code running time complexity with machine learning download book pdf download book epub ricardo j. pfitscher9, gabriel b. rodenbusch9, anderson dias10, paulo vieira10 & … nuno m. m. d. fouto11  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 645 accesses 2 citations abstract the running time complexity is a crucial measure for determining the computational efficiency of a given program or algorithm. depending on the problem complexity class, it can be considered intractable; a program that solves this problem will consume so many resources for a sufficiently large input that it will be unfeasible to execute it. due to alan turing’s halting problem, it is impossible to write a program capable of determining the execution time of a given program and, therefore, classifying it according to its complexity class. despite this limitation, an approximate running time value can be helpful to support development teams in evaluating the efficiency of their produced code. furthermore, software-integrated development environments (ides) could show real-time efficiency indicators for their programmers. recent research efforts have made, through artificial intelligence techniques, complexity estimations based on code characteristics (e.g., number of nested loops and number of conditional tests). however, there are no databases that relate code characteristics with complexity classes considered inefficient (e.g., \(o(c^n)\) and o(n!)), which limits current research results. this research compared three machine learning approaches (i.e., random forest, extreme gradient boosting, and artificial neural networks) regarding their accuracy in predicting java program codes’ efficiency and complexity class. we train each model using a dataset that merges data from literature and 394 program codes with their respective complexity classes crawled from a publicly available website. results show that random forest resulted in the best accuracy, being 90.17% accurate when predicting codes’ efficiency and 89.84% in estimating complexity classes. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others learning based methods for code runtime complexity prediction chapter © 2020 information economy for the fitness of intelligent algorithms chapter © 2024 is prüfer code encoding always a bad idea? chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithmic complexity algorithm analysis and problem complexity computational complexity learning algorithms machine learning mathematics of algorithmic complexity 1 introduction computer programming is a primary task in a software development lifecycle. it essentially involves translating algorithms into functions, methods, and procedures that perform tasks that, in conjunction, solve a specific problem. before the programming action, analysts should determine the algorithms that apply to the referred situation. such an analysis task requires technical knowledge in many areas, from business rules to computational resource efficiency. regarding the latter, a relevant discipline is algorithm analysis. one result of its activities is a mathematical function that expresses the upper bound running time of a specific algorithm, a.k.a. running time complexity [10]. the complexity class of an algorithm or code directly impacts its resource usage efficiency. such an impact on the software’s future performance highlights the need for a precise analysis. also, depending on its asymptotic behavior, the study may indicate that the program takes so long to run that it is considered intractable [6]. on the other hand, beginner developers may rely on inefficient solutions to solve coding problems without understanding the impacts on running time complexity. determining the running time complexity is arduous and requires a deep understanding of code behavior. an analyst must consider multiple variables when analyzing a coded function, including the number of recursive calls, the size of iteration loops, and the number of nested loops. also, analysts must consider how these variables relate and how the computer program uses computational resources. consequently, experts can misevaluate the running-time function, promoting a search for tools and frameworks to aid in the complexity analysis process. alan turing proved through the halting problem that estimating the code complexity is mathematically impossible [12]. thus, recent research applied machine learning techniques and mathematical models to estimate running-time complexity functions [9, 18]. with such approximation values, developers could get real-time feedback on the efficiency of their code. however, a relevant barrier that hinders the evolution and adoption of such models is the lack of datasets that correlate coding characteristics to the respective runtime complexity class. this research aims to fill this gap by developing a learning method to predict the runtime complexity of a program code. to achieve this objective, we built a representative dataset with program codes, related characteristics and complexities, and compared machine learning approaches according to their accuracy in predicting efficiency (if the code is efficient or not) and the runtime complexity of program codes. the contributions of this paper are twofold: first, it makes available to the machine learning community a dataset containing 394 java code files grouped into eight distinct complexity classes and each code having 16 metadata information; second, it shows that a random forest model trained with a dataset merged from the published work of sikka et al. [18] and web crawled data can achieve an accuracy of 90.34% when predicting code efficiency and 89.26% when predicting complexity classes. the remainder of this text is organized as follows, sect. 2 discusses the related work and current limitations. then, sect. 3 describe the methodology followed to build datasets and develop the machine learning models. next, sect. 4 presents the complexity prediction results obtained with trained models and sect. 5 discusses the limitations of classifications. finally, the sect. 6 presents this work’s conclusions and future works. 2 related work given the limitations of writing a computer program to determine the running time of program source codes, a few works appear in the literature to address this problem. however, the recent advances in artificial intelligence propelled the development of models that estimate code complexity. the seminal work of hutter et al. [9] assesses machine learning approaches to predict the performance of algorithms. the results show that the proposed approaches based on random forests and approximate gaussian processes are the ones that better predict the performance of parameterized algorithms used to solve np-hard problems. the correlation coefficients of predictions reached values superior to 0.9. although the research of hutter et al. provided relevant results, our focus differs from theirs because we wish to estimate the runtime complexity regarding asymptotic order. the research work of sikka et al. [18] addressed the runtime complexity estimation problem using machine learning models. the paper has two main contributions. first, it publishes the code runtime complexity dataset (corcod) composed of 932 code files belonging to 5 different classes of complexities, namely constant-time, logarithmic-time, linear-time, linearithmic-time, and quadratic-time. second, it shows that the random forest model achieved the best results for predicting code complexity, with an accuracy of 71.84% using code features as attributes and 83.57% when abstract syntax tree (ast) applies to generate code embedding used in training. although the paper contributes a significant step for future research, we argue that the dataset does not have entries for inefficient codes. thus, we address this problem by crawling a public website to publish a more comprehensive dataset; also, we evaluated more recent models, such as the extreme gradient boosting trees (xgbt) and artificial neural networks (ann). the efforts published by sepideh seifzadeh in a public blog [17] appear to be the results of ongoing research at ibm. the publication has two significant highlights: first, the developed models use the codenet dataset with around 14m code samples for roughly 4k programming problems; second, they trained an ann and a light gradient boosting machine (lgbm) using both code features and code graph representation to predict six classes of runtime complexity near-constant, linear, log-linear, polynomial, exponential, and factorial. the preliminary results show up to \(80\%\) of accuracy in predictions, which supports machine learning for addressing the problem of runtime complexity estimation. despite the promising results, the problem remains open, with space for improvements; also, we accessed the codenet used in the results and do not find the tags that classify codes according to complexity classes, we suppose authors manually tagged the dataset, but it is not clear in the published article. 3 material and methods this section presents the materials and methodology used to compare approaches to estimate code running time complexity. section 3.1 explains the steps taken to consolidate a dataset with representativeness for training machine learning models, including a discussion regarding dataset balancing. section 3.2 explains how we extracted attributes from codes and the assumptions leveraged for complexity classification, as also discusses the process used to select features for the machine learning models. finally, sect. 3.3 presents the machine learning models used to predict the codes’ complexity and process used for comparisons. all the source codes we used in this paper are available on githubfootnote 1. 3.1 dataset consolidation the first step toward developing this research project is dataset consolidation. this work relies on three datasets. the first one is the reference dataset published in the work of sikka et al. [18], which consists of 931 java code files, with 14 metadata information relative to the codes and their respective complexity. considering that such a dataset does not contain codes with intractable complexity, this work builds a second one with data from a publicly available website, which we call crawled dataset (394 entries). the third one, the merged dataset (1325 entries), results from merging the reference and the crawled datasets. we developed a web crawler to extract the information from the platform geekforgeeks.org, which contains a specific section to discuss fundamentals of algorithmsfootnote 2. such section discusses several algorithms, their respective codes, and complexities. the crawler consists of a python script that uses the selenium library to scratch the website and extract codes and complexity information and runs as follows. first, it accesses the main page, which contains a list of topics related to the fundamentals of algorithms. then, the crawler runs a login process and computes the list of links to be visited on the page. next, each page in the link list is accessed to verify if it contains content relevant to complexity prediction. in other words, the crawler searches if the page has algorithms’ source codes and runtime complexity information. considering that the focus of this research is to extend the reference dataset, the crawler always selects java programming language. also, the page should provide text describing the time complexity of the code in any place near it. as soon as the crawler identifies that the page contains the relevant content, it copies each code and searches for the nearest complexity information. it is essential to highlight that many pages have multiple program codes; thus, we save each one and associate it with the runtime complexity information closer to the point where the program code is exhibited. such nearest assumption involves a risk of misclassification, as the pages from the website do not follow a static standard; however, we understand that this is the best effort for automated computing of a dataset. the crawler process resulted in a dataset with 394 program codes and 78 distinct complexity classes. many classes extracted from the web pages do not follow the general form of asymptotic classes; for example, the classes o(n|n|) (algorithm: compute the sum of digits in all numbers from 1 to n) and \(o(m^2k + k^3 log n)\) (algorithm: count ways to reach the nth stair) do not match any general case; for those cases, we manually evaluated each entry and defined their value according to the closest dominant class. to produce a feasible scenario, instead of 78 complexities classes, we reduced the classification scope to eight categories: constant, logarithmic (includes double logarithmic and polylogarithmic), sublinear (fractional power), linear, linearithmic, quadratic, polynomial, and exponential (we also consider factorial as exponential). in addition, we split the program codes into two major categories: the program codes that are efficient (constant, logarithmic, sublinear, linear, and linearithmic) and the inefficient ones (quadratic, polynomial, and exponential). we assume as polynomial time, the asymptotic functions that are equal to or greater than \(o(n^3)\). we understand this is a questionable assumption, as according to [10], polynomial time functions have the form of \(n^c\) for every c greater than 1. however, we argue that algorithms that run in \(o(n^3)\) are much more inefficient than the ones that run in \(o(n^2)\) time, and thus, supposing \(o(n^3)\) is distinct from \(o(n^2)\) is quite reasonable. figure 1 depicts the resultant distribution of complexity classes in crawled and reference datasets. fig. 1. distribution of complexity classes in the available datasets full size image as shown in fig. 1b, the reference dataset does not contain the exponential time and the polynomial time codes, which severally reduces the size of the inefficient class. also, as one can notice, both datasets are imbalanced: in the crawled dataset (fig. 1a) the smaller class (sublinear) contains five entries while the larger class (linear) contains 125 entries; in the reference dataset the smaller class (logarithmic) contains 55 entries while the larger class (also the linear) contains 383 entries. such unbalancing also impacts the classes distribution in the merged dataset (fig. 1c), making the linear and quadratic classes more representative. regarding efficiency, the distribution in each dataset is as follows: 63.70% of codes are from efficient classes in the crawled dataset, 78.51% in the reference dataset, and 74.18% in the merged dataset. considering that an imbalanced dataset can impact the abstraction capability of machine learning models [14] and that our dataset can be considered small compared to what is considered in current state-of-the-art, we rely on the smote tool to balance the data on each dataset. the smote (synthetic minority oversampling technique) [4] is a well-known algorithm for solving imbalanced classification problems. the general idea of this method is to artificially generate new samples of the minority class using the nearest neighbors of these cases. such synthetic data would be generated between the random data and the randomly selected k-nearest neighbor; the procedure is then repeated until the minority class has a size equal to or close to the majority. due to applying smote, all the resultant complexity classes hold a similar number of occurrences. 3.2 features extraction and selection we wrote the code features extraction program in python (“crawler.py file on the repository”) and relied on the javalang library to extract the features described in table 1 for each java code in the datasets. the second column in the table describes if the former work of sikka et al. [18] relies on the feature for their prediction. for the cases where column values are with “no*”, the feature was only used in [18] for the manual classification of algorithms and not for the machine learning models. for the case of recursive calls, we measured the number of recursive calls in the code instead of if there is a recursive call in the code. table 1. features extracted from codes and respective descriptionsfull size table as we established the set of complexity classes and balanced the datasets, the next step is to perform a feature selection to determine the most relevant characteristics to distinguish the codes according to their complexity classes. multiple approaches exist for determining the most relevant features of random forest models. according to speiser et al. [19] for datasets with many predictors, the methods implemented in the r packages varselrf and boruta [11] are preferable due to computational efficiency. thus, in this work, we rely on the boruta package to define which code attributes are more relevant to predict the complexity classes and the efficiency of a given code (please read the “classificators.r” file in the repository). boruta is a feature selection algorithm that relies on random forest to output a variable importance measure (vim); the method’s rationality consists of progressively eliminating irrelevant features by comparing original attributes’ importance with importance achievable at random until the test is stable. table 2 depicts the resultant feature functions from boruta process. table 2. features resultant from boruta feature selection process in each dataset according to the dependent variablefull size table an analysis of the feature selection results depicted in table 2 permits some relevant considerations: (i) the efficiency estimation requires fewer features than the complexity class estimation; (ii) the crawled dataset also uses fewer features for both predicting efficiency and complexity classes and; (iii) the number of switches (attribute number 3) in codes is irrelevant for the predictions. 3.3 model training and validation the last step of this research consists of comparing supervised machine learning models according to their ability to predict the running-time complexity class of a program code. to this end, we first use the merged dataset to find out which method have the best accuracy to predict both efficiency (two-class prediction problem) and complexity class (multiclass prediction). considering the results provided by the work of [18], which found that random forest had the best accuracy among eight classification algorithms, we included extreme gradient boosting trees (xgbt) and ann in the comparisons because the previous work do not considered that approaches. we conducted the implementations of random forest both in r and in python (files “classificators.r” and “classificators.py”, respectively) and the other models only in python. xgbt is an effective and scalable tree-boosting system that combines novel sparsity-aware algorithms and weighted quantile sketch for approximate tree learning [5]. this system is an optimized version of the gradient boosting machine algorithm (gbdt), created by friedman [7], which uses decision trees for classification. the traditional gbdt approach only deals with the first derivative in learning; xgboost improves the loss function with taylor expansion, reducing modeling complexities and the likelihood of model over-fitness [3]. regarding anns, each implementation requires several configuration parameters, which we discuss in the following. we normalized data using the standardscaler function from scikit-learn [13] to scale the features so that they have a mean of 0 and a standard deviation of 1. such scaling is a typical pre-processing step to improve model performance [20]. we then established the hyperparameters based on the best random search results, which have been proven more efficient for hyper parametrization than trials on a grid [1]. when the corresponding dropout rate exceeds zero, a dropout layer is added after the dense layer to prevent overfitting. the same applies to the batch normalization flag: when it is true, it adds a batchnormalization layer after the dense layer to normalize the inputs and improve the convergence of the model [8]. lastly, the model is optimized through adam, defined by the random search, and trained for 500 epochs with a batch size equal to 64. we used the train_test_split function from scikit-learn [13] to split the balanced dataset into trainand test-data. to prevent overfitting issues, we chose to maintain most of the function parameters as standard as possible. that way, we altered only the stratify. when different from none, the samples are stratified through stratifiedkfold, so that each set contains approximately the same percentage of samples of each target class as the complete set. stratification has been found to improve upon standard cross-validation both in terms of bias and variance. the standard method of randomly distributing multi-label training samples can create issues when test subsets lack even a single positive example of a rare label. this, in turn, can lead to calculation problems for various multi-label evaluation measures [16]. after we established that random forest is the machine learning approach that provides the best results (see sect. 4), we conducted a systematic process for learning and predicting complexity classes: 1. we used the crawled dataset for training the models with a sample of 70% and tested with 30% remaining data; 2. we compare our results to sikka et al. [18] by running a cross-validation process with 70% of the reference dataset and validating using 30% of their data; 3. we evaluated the generalization capability of the crawled dataset by training the ml model with 70% of the crawled dataset and testing with 100% of the reference dataset, and; 4. we in-depth evaluate the confusion matrix of the random forest model using 70% of the merged dataset for training and 30% for testing. 4 results this section presents the results for predicting computer program code efficiency and complexity classes based on their attributes. first, we compare the accuracy of predictions for each model in the merged dataset to select the appropriate approach to perform our in-depth systematic evaluation process. table 3 depicts these comparison results; the accuracy values show that the random forest is the best approach among the evaluated methods, predicting codes’ efficiency with 90.17% accuracy and the complexity class with 89.84%. the xgbt provided an accuracy close to the random forest, and given the nondeterministic behavior of ann it resulted in a range of accuracy, also smaller than the random forest. such results follow the findings of [18], which pointed out the random forest as the algorithm that resulted in better accuracy for predicting complexity classes (71.84%). thus, we will consider the random forest for our systematic evaluation process. table 3. comparison of machine learning models to predict efficiency and complexity class on the merged datasetfull size table as we established random forest as the reference ml approach, we now assess its accuracy to predict the efficiency of codes using our systematic evaluation process. table 4 contains prediction results for the efficiency of available algorithm codes. two main conclusions arise from the analysis of the use of random forest to predict the efficiency of source codes: i) when the training and test data are from the same dataset, the model can achieve up to 80% of accuracy (80.13% for crawled dataset, 93.85% for reference dataset, and 90.17% for the merged dataset); ii) when train and test data are from distinct datasets, accuracy drops to 83.57% (trained with 70% of crawled dataset and 100% of reference dataset). although this fall in accuracy seems a poor result, if we consider that the data are from distinct sources, the crawled dataset has a good generalization concerning efficiency classification. also, by analyzing the f1-score’s results (up to 80%), we can conclude that all the random forest models have a good balance between precision and recall, which means that they have a good fit to distinguish the efficiency of source codes. table 4. summary of efficiency prediction using random forestfull size table after assessing the random forest models’ ability to classify codes regarding their efficiency, we retrained each model to predict complexity classes. similar to the efficiency classification, we leverage our four-step systematic process to compare the models. table 5 depicts these results. an accuracy analysis shows that the random forest model properly predicts the complexity class of source codes from the crawled dataset, with an accuracy of 80%. the results in table 5 also show that the referenceand merged-based models achieved an accuracy superior to 88%, outstanding the finds of sikka et al. [18], which presented an accuracy of 71.4% for predicting all classes. we claim that the resultant improvement justifies by both the balancing and the feature selection processes we leveraged. however, when the predictions ran in the reference dataset, the model trained with data from the crawled data had a lower accuracy, only 44.04%. this indicates that the model does not have enough generalization for distinguishing the complexity class when it only relies on the crawled dataset for training. it is worth mentioning that the trained model does not include any of the data in the reference dataset. to understand the misclassifications, we analyzed the confusion matrix of predictions of the crawled-based model on the reference dataset (table 6). table 5. summary of complexity class prediction using random forest modelfull size table table 6. confusion matrix of predictions in the reference dataset with the random forest model trained using 70% of crawled datasetfull size table the results depicted in table 6 show that 112 predictions occurred for classes that even exist in the dataset (sublinear, polynomial, and exponential time), representing 12.03% of the total data, most of them pointing to inefficient classes (polynomial and exponential time). such a result demonstrates that the crawled dataset does not have enough representativeness to allow machine learning models to generalize complexity classes. also, we observed that the better predictions occurred for the linear and linearithmic categories (65.01% and 74% of accuracy), and the worst case occurred in the logarithmic class (09.09% of accuracy), which indicates that the characteristics of these classes require a more profound study. for such analysis, we assess the mean decrease accuracy measure mda [2] for each feature to understand their importance in predicting the complexity class on each model. figure 2 presents such mda results. fig. 2. most relevant features according to mda results in each model full size image the analysis of top-3 mda depicted in fig. 2 allows us to conclude that the three most relevant features for code complexity classification in all the models are the number of variables (num_vari), the number of statements (num_state) and the number of loops (num_loof). the depth of nested loops and the number of recursive calls only appear in the top-5 relevant features in two of three models, which is surprising, as these metrics serve to compute recurrence functions in code complexity analysis. another relevant aspect of mda analysis is that the second most relevant feature varies significantly from one model to another, which also hinders the abstraction capability of the model based on the crawled dataset. considering that the crawled-based and the reference-based models share the top-3 relevant features, we now assess the frequency density function of these features in both datasets to understand the reasons why models misclassify linear, linearithmic, and logarithmic classes (fig. 3). fig. 3. frequency density functions for the top-3 most relevant features in the crawled and reference datasets in the linear, linearithmic, and logarithmic classes. full size image two main considerations arise from the analysis of fig. 3: first, the density function is different in the two datasets, both in terms of the number of occurrences and the behavior of the distribution; second, the linear and linearithmic classes have density distribution functions with similar behavior for the three evaluated metrics, independently of the dataset. while the first consideration explains why the crawled-based model cannot properly identify the classes of the reference dataset, the second justifies the lack of distinction between classes. to have a clear view of the limitations of the merged-based model, we also depicted the confusion matrix of its predictions. the results in table 7 show accurate predictions for most of the class, with accuracies superior to 90%; the discrepant case occurred in the linear class, with 25 predictions pointing to faster time classes (i.e., constant, logarithmic, and sublinear) and 29 predictions to slower classes (i.e., linearithmic, quadratic, polynomial, and exponential). to understand this lack of accuracy, we plotted the frequency density function for the num_vari and num_state features in the merged dataset (fig. 4). table 7. confusion matrix of predictions in the validation part of merged datasetfull size table fig. 4. frequency density functions for the num_vari (a) and num_state (b) features in the merged dataset for the misclassifications in the linear class full size image the density functions depicted in fig. 4 show that misclassifications mainly occurred because the number of variables and states in codes from the mispointed classes have similar behavior, hindering distinguishing them. one can argue that we could remove this attribute from the models; however, even though these features cause misclassifications in the linear class, they are relevant to the overall model classification. 5 discussion besides this paper showing promising results in estimating the code runtime complexity, outperforming the state-of-the-art, we point out the following limitations that will be addressed in future research: the web crawler may not collect the correct complexity of each code. the dataset used for training and estimations resulted from a web scrapping process. the collection process may imply biased information, as the crawler ran automatically. however, given the lack of datasets containing inefficient codes, we understand that the potential bias does not influence the comparison results, which shows that random forest achieved the best results. we will provide an in-depth analysis of the collected data in future works, including a manual verification. trustworthiness of geeksforgeeks.org. one can argue that the information provided by geeksforgeeks.org may not be trustworthy. however, considering the lack of publicly available datasets containing inefficient codes and their respective complexity classes, we understand that the provided information is the best effort to have a comprehensive dataset containing the most relevant complexity classes. small dataset for machine learning purposes. the merged dataset used for training the machine learning models contains 1325 entries, which is small compared to today’s standards. however, considering the lack of comprehensive datasets and the absence of tags to classify the codes published in the codenet dataset, we consider our results as a relevant step towards the use of ai to predict code complexity. the dataset balancing process may cause overfitting. santos et al. [15] discuss the impacts of balancing datasets before running the cross-validation process; they demonstrate that oversampling techniques may generate replicated entries in test and training sets, which implies in lack of generalization of ml models caused by overfitting. however, the smote method generates synthetic entries based on a proximity function, without duplicating any entry. thus, we argue that the lack of duplication reduces the overfitting issues. 6 concluding remarks in this paper, we investigated the use of machine learning models to predict the runtime complexity class of computer program codes. considering that the reference dataset published by sikka et al. [18] does not contain most of the inefficient classes from the literature, we build a second one with data from a publicly available website. next, we compare machine learning models to predict program codes’ efficiency and complexity classes. results show the random forest as the best approach, predicting code efficiency with an accuracy of up to 80% and can classify the runtime complexity with an accuracy superior to 81% for a model trained with data from a merged dataset and 87.4% when the model is trained with the reference dataset. such a result outperforms the ones found by sikka et al. [18], which found an accuracy of 71.4% using a random forest model. we argue that the feature selection process and balanced datasets supported the accuracy enhancement. however, the random forest model’s accuracy drops to less than 50% when we train it with the crawled dataset and try to predict the complexity classes in the reference dataset. we claim that this occurred for two primary reasons: first, we trained the model in a dataset that has more classes than the reference dataset; two, most of the classification errors occurred in the logarithmic and linearithmic categories, which we demonstrated to have a similar behavior regarding the top-3 most important features. in future work, we aim to develop a complete complexity prediction framework containing a learning component deployed on the cloud and an extension for software ides. research challenges related to the framework include but are not limited to: i) studying the applicability of other machine learning models to predict computer programs’ efficiency and complexity class, including natural language processing and deep learning frameworks; and ii) building an even more comprehensive dataset of program codes and complexity classes mainly in polynomial and exponential classes with constant update. in addition, we aim to manually tag the dataset published by the codenet project, which will benefit future research and improve the accuracy of predictors. notes 1.https://github.com/ricardopfitscher/rutico. 2.https://www.geeksforgeeks.org/fundamentals-of-algorithms/. references bergstra, j., bengio, y.: random search for hyper-parameter optimization. j. mach. learn. res. 13(2) (2012) google scholar  breiman, l.: random forests. mach. learn. 45, 5–32 (2001) article  google scholar  chang, y.c., chang, k.h., wu, g.j.: application of extreme gradient boosting trees in the construction of credit risk assessment models for financial institutions. appl. soft comput. 73, 914–920 (2018) article  google scholar  chawla, n.v., bowyer, k.w., hall, l.o., kegelmeyer, w.p.: smote: synthetic minority over-sampling technique. j. artif. intell. res. 16, 321–357 (2002) article  math  google scholar  chen, t., guestrin, c.: xgboost: a scalable tree boosting system. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794 (2016) google scholar  cormen, t.h., leiserson, c.e., rivest, r.l., stein, c.: introduction to algorithms. mit press (2022) google scholar  friedman, j.h.: greedy function approximation: a gradient boosting machine. annals of statistics, pp. 1189–1232 (2001) google scholar  garbin, c., zhu, x., marques, o.: dropout vs. batch normalization: an empirical study of their impact to deep learning. multimed. tools appl. 79, 12777–12815 (2020) google scholar  hutter, f., xu, l., hoos, h.h., leyton-brown, k.: algorithm runtime prediction: methods & evaluation. artif. intell. 206, 79–111 (2014) article  mathscinet  math  google scholar  kleinberg, j., tardos, e.: algorithm design. pearson education india (2006) google scholar  kursa, m.b., rudnicki, w.r.: feature selection with the boruta package. j. stat. softw. 36, 1–13 (2010) article  google scholar  lucas, s.: the origins of the halting problem. j. logical algebraic methods programming 121, 100687 (2021) article  mathscinet  math  google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  ramyachitra, d., manikandan, p.: imbalanced dataset classification and solutions: a review. int. j. comput. bus. res. (ijcbr) 5(4), 1–29 (2014) google scholar  santos, m.s., soares, j.p., abreu, p.h., araujo, h., santos, j.: cross-validation for imbalanced datasets: avoiding overoptimistic and overfitting approaches [research frontier]. ieee comput. intell. magaz. 13(4), 59–76 (2018) google scholar  sechidis, k., tsoumakas, g., vlahavas, i.: on the stratification of multi-label data. in: gunopulos, d., hofmann, t., malerba, d., vazirgiannis, m. (eds.) ecml pkdd 2011. lncs (lnai), vol. 6913, pp. 145–158. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-23808-6_10 chapter  google scholar  seifzadeh, s.: ai for code: predict code complexity using ibm’s codenet dataset, october 2021. https://community.ibm.com/community/user/ai-datascience/blogs/sepideh-seifzadeh1/2021/10/05/ai-for-code-predict-code-complexity-using-ibms-cod sikka, j., satya, k., kumar, y., uppal, s., shah, r.r., zimmermann, r.: learning based methods for code runtime complexity prediction. in: jose, j.m., yilmaz, e., magalhães, j., castells, p., ferro, n., silva, m.j., martins, f. (eds.) ecir 2020. lncs, vol. 12035, pp. 313–325. springer, cham (2020). https://doi.org/10.1007/978-3-030-45439-5_21 chapter  google scholar  speiser, j.l., miller, m.e., tooze, j., ip, e.: a comparison of random forest variable selection methods for classification prediction modeling. expert syst. appl. 134, 93–101 (2019) article  google scholar  thara, d., premasudha, b., xiong, f.: auto-detection of epileptic seizure events using deep neural network with different feature scaling techniques. pattern recogn. lett. 128, 544–550 (2019) article  google scholar  download references author information authors and affiliations federal university of santa catarina, florianópolis, brazil ricardo j. pfitscher & gabriel b. rodenbusch unisociesc campus anita garibaldi, joinville, brazil anderson dias & paulo vieira university of são paulo, são paulo, brazil nuno m. m. d. fouto authors ricardo j. pfitscherview author publications search author on:pubmed google scholar gabriel b. rodenbuschview author publications search author on:pubmed google scholar anderson diasview author publications search author on:pubmed google scholar paulo vieiraview author publications search author on:pubmed google scholar nuno m. m. d. foutoview author publications search author on:pubmed google scholar corresponding author correspondence to ricardo j. pfitscher . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pfitscher, r.j., rodenbusch, g.b., dias, a., vieira, p., fouto, n.m.m.d. (2023). estimating code running time complexity with machine learning. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_27 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords running time complexity prediction algorithm analysis machine learning publish with us policies and ethics profiles ricardo j. pfitscher view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature genetic algorithms with optimality cuts to the max-cut problem | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper genetic algorithms with optimality cuts to the max-cut problem conference paper first online: 12 october 2023 pp 17–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) genetic algorithms with optimality cuts to the max-cut problem download book pdf download book epub pablo luiz braga soares9 & carlos victor dantas araújo10  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 575 accesses 2 citations abstract the max-cut problem involves dividing a set of n vertices in a weighted graph \(g = (v, e)\) into two subsets \((s, \bar{s})\) in such a way that the sum of the weights between the subsets is maximized. this research introduces two heuristic methods that combine genetic algorithm, tabu search, and a set of optimality cuts, which are also proven in this work. to the best of our knowledge, we are the first to utilize these inequalities in conjunction with the genetic algorithm methodology to solve the max-cut problem. computational experiments using a benchmark set of 54 instances, ranging from 800 to 3000 vertices, demonstrate that the incorporation of optimality cuts is a crucial factor for our methodologies to compete effectively with six state-of-the-art approaches for the max-cut problem and our genetic algorithm that incorporated optimality cuts in the population generation was able to improve the state-of-the-art value for the g51 instance and find the same solutions as the literature in 31 other instances. this work is partially supported by universal cnpq [422912/2021-2]. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others solving the cut width optimization problem with a genetic algorithm approach chapter © 2017 a genetic algorithm approach for large-scale cutting stock problem chapter © 2018 new algorithms for a simple measure of network partitioning chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms discrete optimization genetic techniques genome assembly algorithms graph theory optimization 1 introduction the max-cut problem is a combinatorial optimization problem that, due to its vast domain of applications such as social networks, where the max-cut value is generally a measure of network robustness and structural balance originating from social behaviors [1, 5, 11, 18], statistical physics, image segmentation, design of very large scale integrated (vlsi) circuits and communications network design [4], has attracted the scientific interest of several researchers in the areas of graph theory, mathematical optimization and discrete mathematics [6]. this high applicability motivates the study and development of algorithms that obtain good results in a viable computational time. the max-cut problem can be described as follows: given an undirected graph \(g = (v, e)\) where v is a set with n nodes, e is a set with m edges and a cost \(c_{ij} = c_{ji} \in \mathbb {r}\) for each edge \((i,j) \in e\). any partition of v, represented by \((s, \bar{s})\), defines a cut of g. the cut value is the sum of all \(c_{ij}\), such that \(i \in s \text { and } j \in \bar{s}\). it is important to note that s or \(\bar{s}\) can be empty. the max-cut problem consists in finding a cut of g with the largest value. figure 1 a) show a weighted undirected graph g with \(v = \{1,2,3,4,5\}\) and \(e = \{(1,2), (1,4), (2,3), (2,4), (2,5), (3,4), (3,5), (4,5) \}\). figure 1 b) presents a possible partition \((s, \bar{s})\), where \(s = \{1, 2, 3\}\), \(\bar{s} = \{4, 5\}\) and its associated cut value \(c_{14} +c_{24} + c_{25} + c_{34} + c_{35} = 10 + 10 + 5 + 10 + 5 = 40\). fig. 1. a) example of a weighted undirected graph g with 5 vertices and 8 edges. b) example of a \((s,\bar{s})\) partition of v. full size image classified as an np-hard problem [14], it is considered a computational challenge to solve it optimally, even for moderately sized instances [19]. therefore, for these instances, it is necessary to use approaches that provide results close to the optimal solution, such heuristics, meta-heuristics and approximation algorithms [10]. for large instances, heuristic and metaheuristic methods are commonly used to find “goodenough” sub-optimal solutions. in particular, for the very popular max-cut problem, many heuristic algorithms have been proposed, including simulated annealing [23], tabu search [17], tabu search with local search procedure [2], scatter search with greedy randomized adaptive search procedure grasp [21], memetic algorithm [27] and multiple search operator heuristic [20]. these six procedures were tested using the benchmark of instances, called g set and the results obtained (only cut value), as far as is known, represents the state-of-the-art to max-cut. in this context, a genetic algorithm is one of the population-based meta-heuristic [22] that has been used in several studies, such as test data generation and selection in a regression testing environment [24], for the job shop scheduling problem [26], for designing the weights to combine recurrent neural network (rnn) and long short-term memory (lstm) [7] and for the max-cut problem [10, 15, 16]. considering the literature, it is notorious an extensive use of heuristics and meta-heuristics to obtain solutions for the max-cut, a fact that motivates the creation of hybrid and non-exact guided methodologies to improve the results of the state-of-the-art. heuristic approaches using optimality cuts proved to be an interesting possibility to reduce the search space and realize less random modifications to the solutions. generally, valid inequalities, optimality cuts and optimality conditions are used in exact approaches [8, 9, 25]. on the other hand, it is difficult to find and prove these cuts. also, considering inequalities that already exist, it is difficult to efficiently apply them to optimization problems. in this paper, we developed two variations of genetic algorithms that use optimally cuts in their composition. the first one uses the optimally cuts in the generation of the initial population and also as a modification (a kind of mutation) procedure. the second uses the optimally cuts in population generation and also uses them as criterion for choosing candidates to compose a tabu list. the main difference between the two versions is the effort spent on generating new individuals. the first variant puts more effort into generating a good population, using the mutation procedure and local search, while the second version puts more effort into improving the value of individuals through the tabu search. the main contributions of this paper are thus (1) the use of optimality cuts obtained through the study of the objective function of a quadratic programming model for the maximum cut problem; (2) development of genetic algorithms that incorporate optimality cuts to guide the search for better solutions; and (3) the genetic algorithm that incorporated optimality cuts in the population generation was able to improve the state-of-the-art value for the g51 instance and find the same solutions as the literature in 31 other instances. the remaining sections of this paper are organized as follows. the optimality cuts used are presented and proved in sect. 2. in sect. 3, we present how we used the optimally cuts in the composition of the two variants of the genetic algorithm. section 4 contains the computational results. finally, sect. 5 concludes the paper, with our final remarks. 2 optimality cuts for max-cut let \( (s, \bar{s}) \) be a partition represented by the binary vector \(x \in \{ 0,1 \}^n \) with \( x_i = 0 \) if \(i \in s \) and \(x_i = 1\) otherwise, \( \forall i = \{1, \dots , n\}\) and let \(\bar{x}_j = 1 x_j \), \( \forall j = \{1, \dots , n\}\), then the max-cut problem can be formulated as an unrestricted problem, with a quadratic objective function and its variables assuming binary values [3]. the constant \(c_{ij} = c_{ji}, \forall i = \{1, \dots , n\}\), \(j = \{1, \dots , n\}\) and \(i \ne j\), represent the edge weight; in cases where the edge does not exist its value is zero. let (1) be a mathematical formulation as follows: $$\begin{aligned} \left\{ \max \displaystyle \sum _{i = 1}^{n-1} \sum _{j = i + 1}^{n} c_{ij}(x_i \bar{x}_j + \bar{x}_i x_j): x \in \{0,1\}^n, \quad \bar{x} = 1-x \right\} . \end{aligned}$$ (1) let \(q(x) = \displaystyle \sum _{i = 1}^{n-1} \sum _{j = i + 1}^{n} c_{ij}(x_i \bar{x}_j + \bar{x}_i x_j)\), notice that $$\begin{aligned} q(x) =\displaystyle \sum _{i = 1}^{n-1} \sum _{j = i+1}^{n} c_{ij}(x_i \bar{x}_j + \bar{x}_i x_j) = \displaystyle \sum _{i = 1}^{n} \sum _{\begin{array}{c} j = 1 \\ j \ne i \end{array}}^{n} c_{ij}x_i \bar{x}_j, (\text {with } c_{ij} = c_{ji}) \end{aligned}$$ (2) let \(x \in \{0, 1\}^n\), \(k \in \{1, 2, \dots , n\}\) and given \(x^{\bar{k}} \in \{0, 1\}^n\) be a vector such that: \( x_i^{\bar{k}} = x_i\), if \(i\ne k\) and \(x_i^{\bar{k}} =1-x_i\) if \( i = k \). we are interested in studying the variation \(\vardelta _k(x)\) in the value of the objective function when we modify only the component k of a feasible vector x. lemma 1 let \( \vardelta _k(x) = q (x^{\bar{k}}) q(x) = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj}(1 2x_k)(1 2x_j)\), for all \(x \in \{0, 1\}^n\) and for all \(k \in \{1, 2, \dots , n\}\). proof using the equation (2) we have \(\vardelta _k(x)\) $$ \begin{array}{l} = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj} \left[ x_k^{\bar{k}}(1 x_j^{\bar{k}}) x_k(1 x_j) \right] + \displaystyle \sum _{\begin{array}{c} i \ne k \end{array}} c_{ik} \left[ x_i^{\bar{k}}(1 x_k^{\bar{k}}) x_i(1 x_k) \right] \\ = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj} \left[ (1 x_k)(1 x_j) x_k(1 x_j) \right] + \displaystyle \sum _{\begin{array}{c} i \ne k \end{array}} c_{ik} \left[ x_ix_k x_i(1 x_k)\right] \\ = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj} (1 2x_k)(1 x_j) \sum _{\begin{array}{c} i \ne k \end{array}} c_{ki} x_i (1 2x_k) = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj} (1 2x_k)(1 2x_j) \end{array} $$    \(\blacksquare \) let \(\displaystyle c_k = \sum _{\begin{array}{c} j=1 \atop j \ne k \end{array}}^n c_{kj}\) and \( \displaystyle c_k(x) = \displaystyle \sum _{\begin{array}{c} j = 1 \atop j \ne k \end{array}}^n c_{kj}x_j\), corollary 1 present properties of an optimal solution that are based on lemma 1. corollary 1 let \(x^*\) be an optimal solution for formulation (1) and let \(k \in \{1, 2, \dots , n\}\). then, the following holds: 1. if \(x^*_k = 1\), then \(c_k(x^*) \le c_k/2\) and if \(c_k(x^*) < c_k/2\), then \(x^*_k = 1\). 2. if \(x^*_k = 0\), then \(c_k(x^*) \ge c_k/2\) and if \(c_k(x^*) > c_k/2\) then \(x^*_k = 0\). proof let \(\bar{x} = (x^*)^{\bar{k}}\). we have that \(q(\bar{x}) \le q(x^*)\) by optimality of \(x^*\). first, consider \(x^*_k = 1\). by lemma () we have $$0 \ge q(\bar{x}) q(x^*) = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} -c_{kj}(1 2x^*_j) = -\displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj} + 2 \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj}x^*_j = -c_k + 2c_k(x^*).$$ showing the first implication. to confirm the second claim of item 1 in the corollary 1, consider \(x^*_k = 0\). again, by lemma 1 we have $$ q(\bar{x}) q(x^*) = \displaystyle \sum _{\begin{array}{c} j \ne k \end{array}} c_{kj}(1 2x_j) = c_k 2 c_k(x^*) \le 0.$$    \(\blacksquare \) the proof of item 2 in the corollary 1 will be omitted as it utilizes the same arguments as item 1. 3 developed algorithms in this section, we describe the strategy for encoding the optimality cuts and how they will be used in conjunction with the genetic algorithms, as well as the main ideas and pseudocodes of the algorithms. to exemplify the use of the optimality cuts, consider the graph g of fig. 1. each cut in g is a binary vector where each position represents a vertex and the value at the position represents the set to which the vertex belongs: if \(v \in s, x_v = 0\), otherwise \(v \in \bar{s}\) and \(x_v = 1\). let \(x^* = [0,0,0,1,1]\) be the binary vector that represents the cut \((s,\bar{s})\) on g in fig. 1. it is worth noting that the vector \(x^*\) represents an optimal solution for g. we have \(c_4 = c_{41}+c_{42}+c_{43}+c_{45} = 36\), \(c_4(x^*) = c_{41}x^*_1+c_{42}x^*_2+c_{43}x^*_3+c_{45}x^*_5 =6\), \(x^*_4 = 1\) and \(c_4(x^{*}) \le c_4/2\). for this example, it is easy to see that \(x^*\) is satisfied by the optimality cut. we know that these inequalities hold to any optimal solution, and if we modify, for example, the vertex 5 in set \(\bar{s}\) to the set s, the inequalities of vertices \(\{1,2,3,4\}\) still satisfy the optimality cuts, however the new inequality that represents vertex 5 it is infeasible, ensuring that the new cut is not optimal. the algorithm 1 presents the procedure that uses optimality cuts to modify a vector x. the computation of \(c_k\) values is performed only once and their values do not change during the execution of the algorithm; this allows the calculation to be done at the instance loading step, prior to algorithm 1. however, \(c_k(x)\) values are calculated dynamically and it must be computed according to the values of the current x vector. the computation of \(c_k(x)\) for each vertex k in lines (2)–(6) has complexity o(n) and can be improved to \(\vartheta (1)\) after the first iteration by saving the values in a vector and updating after each change in the cut. the lines (7) and (10) run in constant time, and the procedure to change the vertex between sets can be made in o(n)-time. assuming \(\alpha \) as a constant indicating the number of iterations in loop (1)–(11), the time complexity of the algorithm 1 can be limited to \(o(\alpha n^2)\). algorithm 1 modification with optimality cuts full size image 3.1 genetic algorithm with optimality cuts the motivation to use genetic algorithms (ga) is the fact there are many different solutions to which the cuts can be applied, along with the fact that genetic operators make it possible to maintain diversity in solutions, especially after applying the inequalities which often result in similar solutions. this section describes the first developed genetic algorithm that uses the optimality cuts, referenced in the rest of this document as ga-oc. the complete version of ga-oc is presented in the algorithm 2. algorithm 2 genetic algorithm with optimality cuts full size image algorithm 3 local optimization full size image algorithm 4 crossover full size image each individual in the population is a binary vector that represents a cut in the graph and the fitness function is the cut value. the first step of ga-oc lines (1)–(6) randomly generates the initial population, where (ps) and (nioc) are parameters referring to the size of the population and the amount of randomly generated individuals that are modified by the optimality cuts in the initial population. it is important to mention that we chose a vertex that did not have its value modified when using the algorithm 1. new individuals are generated through crossover (algorithm 4) and local optimization (algorithm 3) in lines (9)–(13). the selection of parents is performed through tournaments, which selects the fittest individual from a random subset of the population. the crossover function creates a new individual from a pair returned from tournaments. the next step in lines (14)–(18), is to select a subset of individuals from the population p to apply the modification procedure using the optimality cuts, which consists of fixing a random vertex and applying the algorithm 1. the same occurs in the lines (19)–(23), where a different subset from the population p is selected to apply the default mutation, as shown in algorithm 5. in the end, new individuals are inserted into the population after applying a procedure of selection, presented on algorithm 6. algorithm 5 default mutation full size image algorithm 6 selection full size image 3.2 genetic algorithm with perturbation-based on tabu search the genetic algorithm with perturbation-based on tabu search (ga-ts) pseudocode is presented on (algorithm 7). first, the initial population is created in the same way as in ga-oc lines (1)–(6), then the new individuals are computed using crossover (algorithm 4) and perturbation-based on tabu search (algorithm 8). we discard the default mutation procedure and in the end, new individuals are inserted into the population after applying a procedure of selection, presented on algorithm 6. it is important to emphasize that in algorithm 8, the tabu list (tl) size is dynamically and that size changes following the rule presented by galinier [12]. the condition to shuffle the solution was based on wu [27], according to their results and some preliminary tests that we made. also, values above 500 iterations do not show significant difference. the shuffle procedure in the line (8) of the algorithm 8 selects 150 vertices from the current solution and changes the set where they are in. algorithm 9 is based on the principle of best improvement, in which the movement candidate list is generated based on two properties: the cut improvement of moving the vertex k to the other set of the partition, and if this vertex, according to the optimality cuts, must be in the opposite set. thus, the movement that results in the greatest cut gain for the solution is made. this procedure is called on algorithm 8 until the maximum number of iterations is reached, always updating the incumbent, i.e., the best solution found until the moment. the choice of a perturbation-based on tabu search (ts) in conjunction with the ga is due to the decision to use more time optimizing the new individuals generated, allowing for changes only when they improve the cut value. the main difference between ga-ts and the ga-oc is the fact that ga-ts makes a stronger search to improve the new individuals. algorithm 7 ga with pertubation-based on tabu search full size image 4 computational results in this section, we describe the computational experiments that were performed to test the efficiency of the two heuristics developed in this work. the algorithms were implemented using the c++11 programming language and are available at https://github.com/cvaraujo/max-cut-hybrid-ga-ts. the experiments were carried out using a machine with intel(r) xeon(r) silver 4114 (2.20 ghz) \(\times \) 10 with 32 gb ram and linux ubuntu 16.04 64 bits operating system using a sample of 54 instances. this set of benchmark instances, called g set, is available at http://www.grafo.etsii.urjc.es/optsicom/maxcut and it was generated by helmberg et al. [13]. algorithm 8 perturbation-based on tabu search full size image algorithm 9 neighborhood move full size image to set these values and parameters we consider an empirical analysis. we tested the algorithms with different values for some parameters and the results did not present significant differences. we observed that it is preferable that ga-ts manages fewer individuals per iteration since it takes more time optimizing each one. for ga-oc, creating more individuals and renewing part of the population facilitates the variety and the quality of solutions that are optimized, either by the inequalities or by the local optimization procedure. the operator and parameter values to algorithms ga-oc and ga-ts are: representation: ga-oc and ga-ts use binary vector representation; population: the initial population size for ga-oc is 300, with 10\(\%\) niocs and 50 new individuals are created in each generation; in ga-ts, the initial population size is 50, with 10\(\%\) niocs and a one individual in each iteration; tournament: for both algorithms the tournaments use 4 randomly selected individuals and the best of them is returned; crossover: for both algorithms, the crossover used is uniform with two individuals. if the vertex value of the parents is different, the child has a chance of inheriting from the fittest parent. the crossover procedure also generates only one child; mutation: the mutation on ga-oc is applied to \(20\%\) of the population, excluding the fittest. each gene of the selected individuals has a probability of \(10\%\) to change. on ga-ts, the mutation function is not used; ts iterations: the number of iterations is \(10^6\); stopping criterion: time limit of 1800 s for both algorithms. to show the difference in the quality of individuals created with the use of optimality cuts, we select a sample of 12 instances, where 3 are from each size of vertices, and created a population of 50 individuals through algorithm 1, referred here as nioc, another population of 50 randomly generated individuals, referring here to rci. in the population of rcis, the local optimization procedure (algorithm 3) was also used on all individuals. the results are in table 1, the first and second columns (graph, n) are the instance name and the number of vertices. for each algorithm (nioc, rci and the rci with local optimization rci-lo) there are two columns (avg, dev) that inform the average and standard deviation of the cut values, respectively. the highest averages are in bold. in all 12 sample instances, the nioc population consistently exhibits higher mean values compared to rci and rci-lo. this superiority is also observed in the average standard deviation, except for instance g46 where nioc did not achieve the best result. the results of rci-lo demonstrate the effectiveness of the local optimization procedure in improving the quality of randomly generated individuals across all tested instances. these findings highlight the positive impact of incorporating optimality cuts on the overall output quality of the algorithm, enabling the generation of solutions with higher cut values within a shorter timeframe. however, it is important to acknowledge that relying solely on optimality cuts for optimization may lead to individuals getting trapped in local maxima more quickly than other methods like rcis. to address this limitation, we have implemented additional measures such as an alternative local optimization procedure, a high mutation rate, and a preference for maximum diversity during the selection process. these measures aim to mitigate the risk of premature convergence and enhance the algorithm’s exploration capabilities, enabling it to search for more optimal solutions across a wider solution space. the second experiment compares two versions of our algorithm, ga-oc and ga-ts, with a standard genetic algorithm implementation. we conducted 30 runs of ga-oc, ga-ts, and the standard ga algorithm for each of the 54 instances. table 3 presents the average, standard deviation, and minimum values obtained by each of the algorithms. considering the results, the hybrid approaches outperform the default version of the ga for all instances. considering only the two versions of ga using the optimality cuts, the algorithm ga-oc outperforms ga-ts with respect to the averages and lowest values. for all instances, ga-oc has 28 highest values of column min, and in 6 cases the value is the same, while for the 20 remaining the value of ga-oc is less than ga-ts. considering columns avg, to 29 instances ga-oc is the highest, 4 were draws, and ga-ts has higher values in others 21. for column dev, ga-oc has a smaller variation in solutions, such that for 30 instances it has the standard deviation value less than ga-ts, to 4 instances both the algorithms have value 0 of standard deviation and for the remaining 20 instances ga-ts has smaller values. based on these results, the probability associated with the t-test was calculated using the mean values (columns 4 and 7) obtained by the ga-oc and ga-ts algorithms across the 54 instances, using a two-tailed distribution and a 95% confidence interval as parameters. the p-value of 0.009 indicates that the difference between the means of the algorithms is statistically significant. we compared the results of our algorithms with the most effective heuristics currently in the literature. it is important to emphasize that the conditions and configuration parameters of our algorithms were set under different circumstances than those reported in the literature for the compared heuristics, such as the programming languages used, termination criteria, and hardware configuration. therefore, no implementation of the algorithms from the compared papers was conducted. only the results presented by the authors were used for the purpose of comparison with our algorithms. table 4 compares our approaches with 6 state-of-the-art algorithms. the first two columns contain the instance identifier (graph) and the number of vertices (n). columns (ga-oc) and (ga-ts) exhibit the best values obtained by our algorithms and from the fifth to the tenth column are the best results found by these reference algorithms. the “-” symbol represents that no value was made available for that instance and values in bold font are the actual best-known results. table 2 show the summary of the comparison with the best result of our algorithms. the rows 2, 3, 4 for ga-oc in table 2 respectively denote: the number of instances in which our algorithms obtain better, equal, and worse cut values when compared to the corresponding reference algorithm. from table 2, we observe that the ga-oc algorithm outperforms 5 out of 6 reference algorithms in terms of the number of wins, i.e., the number of instances in which each algorithm achieved the best cut value. compared to the sils method, ga-oc won in instances g37 and g51, with g51 being the instance where it was able to improve the state-of-the-art and achieved the same results in 32 other instances. to the 20 remaining instances, the \(gap = \frac{sta-oa}{sta} \times 100\), between the best of our algorithms (oa) and the state-of-the-art (sta) is less than \(1\%\), more specifically the gap of one instance is \(1\%\) while for the other 19 this value is less than \(0.3\%\). on the other hand, the ga-ts algorithm outperforms 3 out of 6 reference algorithms. these results confirm the effectiveness of our genetic algorithms that use our set of optimality cuts to deliver high-quality solutions for the 54 instances belonging to g set. table 1. the difference between nioc, rci and rci-lofull size table table 2. comparison of ga-oc and ga-ts with the six algorithms from the literaturefull size table table 3. comparison of algorithms ga-oc, ga-ts and gafull size table table 4. best results obtained by ga-oc, ga-ts and the current state-of-the-artfull size table 5 conclusion this work presents a new set of optimality cuts and two heuristics based on genetic algorithms that use these inequalities in their composition. besides the proposal to present an approach to improve the efficiency of these heuristics, as far as is known nothing of the kind was reported to max-cut problem. analysis using the benchmark set g considered the best cut value obtained by our heuristics in each instance and we were able to improve the best-known value for the instance g51 and be strongly competitive with current state of the art algorithms, presenting results with a maximum of \(1\%\) gap of the best-known values for all instances. although the experiments consider a limited execution time, increasing it would not have a significant impact on the results provided by our algorithms. thus, it is possible to conclude that the use of the proposed optimality cuts in ga and ts results in a good improvement to obtain solutions to the max-cut problem. for future work, we hope to find new ways to explore the optimality cuts presented and apply them to different heuristics and meta-heuristics, such as simulated annealing (sa) and scatter search (ss). it is possible to use these optimality cuts in math-heuristics, that are approaches which use mathematical models and heuristics. also, improvements can be sought in the cuts, searching for more efficient and quick ways to apply them. references agrawal, r., rajagopalan, s., srikant, r., xu, y.: mining newsgroups using networks arising from social behavior. in: proceedings of the 12th international conference on world wide web, pp. 529–535 (2003) google scholar  alidaee, b., sloan, h., wang, h.: simple and fast novel diversification approach for the ubqp based on sequential improvement local search. comput. ind. eng. 111, 164–175 (2017) article  google scholar  barahona, f.: the max-cut problem on graphs not contractible to k5. oper. res. lett. 2(3), 107–111 (1983) article  mathscinet  math  google scholar  barahona, f., grötschel, m., jünger, m., reinelt, g.: an application of combinatorial optimization to statistical physics and circuit layout design. oper. res. 36(3), 493–513 (1988) article  math  google scholar  bramoullé, y.: anti-coordination and social interactions. games econom. behav. 58(1), 30–49 (2007) article  mathscinet  math  google scholar  burer, s., monteiro, r., zhang, y.: rank-two relaxation heuristic for max-cut and other binary quadratic problems. siam j. optim. 12(2), 503–521 (2001/2002) google scholar  chui, k.t., gupta, b.b., vasant, p.: a genetic algorithm optimized rnn-lstm model for remaining useful life prediction of turbofan engine. electronics 10(3), 285 (2021) article  google scholar  de simone, c., diehl, m., jünger, m., mutzel, p., reinelt, g., rinaldi, g.: exact ground states of ising spin glasses: new experimental results with a branch-and-cut algorithm. j. stat. phys. 80(12), 487–496 (1995) article  math  google scholar  de simone, c., rinaldi, g.: a cutting plane algorithm for the max-cut problem. optim. methods softw. 3(13), 195–214 (1994) article  google scholar  dunning, i., gupta, s., silberholz, j.: what works best when? a systematic evaluation of heuristics for max-cut and qubo. informs j. comput. 30(3), 608–624 (2018) article  math  google scholar  facchetti, g., iacono, g., altafini, c.: computing global structural balance in large-scale signed social networks. proc. natl. acad. sci. 108(52), 20953–20958 (2011) article  google scholar  galinier, p., boujbel, z., fernandes, m.: an efficient memetic algorithm for the graph partitioning problem. annals or 191, 1–22 (2011) article  mathscinet  math  google scholar  helmberg, c., rendl, f.: a spectral bundle method for semidefinite programming. siam j. optim. 10(3), 673–696 (2000) article  mathscinet  math  google scholar  karp, r.m.: reducibility among combinatorial problems. in: miller, r.e., thatcher, j.w., bohlinger, j.d. (eds.) complexity of computer computations, pp. 85–103. springer, cham (1972). https://doi.org/10.1007/978-1-4684-2001-2_9 kim, s.h., kim, y.h., moon, b.r.: a hybrid genetic algorithm for the max cut problem. in: proceedings of the 3rd annual conference on genetic and evolutionary computation, pp. 416–423. morgan kaufmann publishers inc. (2001) google scholar  kim, y.h., yoon, y., geem, z.w.: a comparison study of harmony search and genetic algorithm for the max-cut problem. swarm evol. comput. 44, 130–135 (2019) article  google scholar  kochenberger, g.a., hao, j.k., lü, z., wang, h., glover, f.: solving large scale max cut problems via tabu search. j. heuristics 19(4), 565–571 (2013) article  google scholar  kolli, n., narayanaswamy, b.: influence maximization from cascade information traces in complex networks in the absence of network structure. ieee trans. comput. soc. syst. 6(6), 1147–1155 (2019) article  google scholar  krislock, n., malick, j., rouoin, f.: improved semidefinite bounding procedure for solving max-cut problems to optimality. math. program. 143(1–2), 61–86 (2014) article  mathscinet  math  google scholar  ma, f., hao, j.k.: a multiple search operator heuristic for the max-k-cut problem. ann. oper. res. 248(1–2), 365–403 (2017) article  mathscinet  math  google scholar  martí, r., duarte, a., laguna, m.: advanced scatter search for the max-cut problem. informs j. comput. 21(1), 26–38 (2009) article  mathscinet  math  google scholar  mirjalili, s.: genetic algorithm. in: evolutionary algorithms and neural networks. sci, vol. 780, pp. 43–55. springer, cham (2019). https://doi.org/10.1007/978-3-319-93025-1_4 chapter  math  google scholar  myklebust, t.g.: solving maximum cut problems by simulated annealing. arxiv preprint (2015) google scholar  pandey, a., banerjee, s.: test suite optimization using firefly and genetic algorithm. int. j. softw. sci. comput. intell. (ijssci) 11(1), 31–46 (2019) article  google scholar  rendl, f., rinaldi, g., wiegele, a.: a branch and bound algorithm for max-cut based on combining semidefinite and polyhedral relaxations. in: fischetti, m., williamson, d.p. (eds.) ipco 2007. lncs, vol. 4513, pp. 295–309. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-72792-7_23 chapter  math  google scholar  sun, l., cheng, x., liang, y.: solving job shop scheduling problem using genetic algorithm with penalty function. int. j. intell. inf. process. 1(2), 65–77 (2010) google scholar  wu, q., hao, j.-k.: a memetic approach for the max-cut problem. in: coello, c.a.c., cutello, v., deb, k., forrest, s., nicosia, g., pavone, m. (eds.) ppsn 2012. lncs, vol. 7492, pp. 297–306. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-32964-7_30 chapter  google scholar  download references author information authors and affiliations nemo research laboratory, federal university of ceará campus russas, russas, ce, 62900-420, brazil pablo luiz braga soares institute of computing, university of campinas, campinas, sp, 13083-970, brazil carlos victor dantas araújo authors pablo luiz braga soaresview author publications search author on:pubmed google scholar carlos victor dantas araújoview author publications search author on:pubmed google scholar corresponding author correspondence to pablo luiz braga soares . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper soares, p.l.b., araújo, c.v.d. (2023). genetic algorithms with optimality cuts to the max-cut problem. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_2 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords max-cut problem optimality cuts genetic algorithm publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature mas4gc: multi-agent system for glycemic control of intensive care unit patients | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper mas4gc: multi-agent system for glycemic control of intensive care unit patients conference paper first online: 28 november 2021 pp 64–78 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) mas4gc: multi-agent system for glycemic control of intensive care unit patients download book pdf download book epub tiago henrique faccio segato  orcid: orcid.org/0000-0003-0926-735310,11, rafael moura da silva serafim  orcid: orcid.org/0000-0001-9129-924410, sérgio eduardo soares fernandes  orcid: orcid.org/0000-0002-2511-400x12 & … célia ghedini ralha  orcid: orcid.org/0000-0002-2983-218011  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 757 accesses 1 citation abstract the coronavirus disease (covid-19) pandemic has brought significant challenges worldwide through the consequences of increasing demand for the intensive care unit (icu) resources. this work presents the multi-agent system for glycemic control (mas4gc) to assist health professionals leading with critical patients in the icu. more specifically, the mas4gc manages patients’ blood glucose through glycemic predictions, treatment, and monitoring recommendations to health professionals. prediction models are applied to monitor patients’ blood glucose allowing health professionals to carry out preventive treatments. the glycemic control is included in the fast hug mnemonic to remember the key issues in the supportive care of critically ill patients. the mas4gc methodological development process is presented with tropos modeling, architectural design, and implementation with the pade framework. agents’ inference mechanisms are based on production rules defined by intensive care physician specialists applying their knowledge to indicate treatments for patients. two experiments using patients with synthetic data were conducted to evaluate the results of the mas4gc: (1) the prediction model achieved 90% accuracy in blood glucose predictions for the next 4 h, (2) 84% similarity of treatment recommendations compared to a human specialist, and 78% in recommendations for monitoring glycemic of critical patients. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others accuracy of the intermittently scanned continuous glucose monitoring system in critically ill patients: a prospective, multicenter, observational study article 13 october 2022 accuracy, reliability, feasibility and nurse acceptance of a subcutaneous continuous glucose management system in critically ill patients: a prospective clinical trial article open access 21 july 2016 machine learning classification for blood glucose performances using insulin sensitivity and respiratory scores in diabetic icu patients chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence cyber-physical systems gastrointestinal system model intensive care medicine multiagent systems predictive medicine 1 introduction the intensive care unit (icu) is considered a high-risk care setting where medical carelessness or errors can cause deaths or complications to patients’ health [1]. in pandemic times, such as that of the coronavirus disease (covid-19), the icu’s importance in the treatment of critically ill patients became even more evident. since patients need special care varying from basic requirements to the need for equipment for patient monitoring and life support, such as respirators and mechanical ventilators [2, 3]. artificial intelligence (ai) applications currently are used in the management of such complex tasks. ai can assist in the monitoring and treatment of patients with chronic diseases and in critical conditions hospitalized in the icu, where these cases occur more frequently [4]. the icu patient monitoring must rigorously take place. the fast hug feeding, analgesia, sedation, thromboembolic prevention, head of the bed elevated, stress ulcer prophylaxis, and glucose control is a simple and significant mnemonic to highlight seven of the main aspects that must be monitored by health professionals for each patient in the icu to minimize possible problems [1]. focusing in the fast hug, glycemic control has the function of checking the patient’s glucose level in the blood, keeping as long as possible in the target range [1]. regardless of which is the ideal target range, it is vital to keep it monitored. effective glycemic control in the icu environment has the potential to decrease mortality rates and the patient’s length of stay in the icu, optimizing hospital resources [5]. the overload of health professionals in pandemic times is crucial, and factors such as this point out that automated systems to monitor and assist patients’ treatment can bring benefits. thus, the fast hug, or at least some of its items, can be improved with process automation combined with ai techniques. in the literature review, ai-based works for glycemic control of icu patients are presented [5,6,7], as well as the application of multi-agent system (mas) for patients glycemic control [8], and mas in the icu context [9, 10]. however, none of the works applies mas for patients’ glycemic control admitted to the icu. some work points to prediction models as good solutions for glycemic control [11,12,13]. considering the cited scenario, the objective of this work is to present a system to track and monitor the glycemic control of critical patients in the icu through a mas approach. in a previous work [14], the mas development process was presented without implementation results. more specifically, the following hypothesis has to be proven: a mas can suggest patients’ treatment recommendations using prediction models and a knowledge base with inference rules similar to specialist intensive care physicians. the rest of the manuscript is organized as follows. in sect. 2 some works found in the literature are presented. in sect. 3, the materials and methods are presented. in sect. 4 the experiments were carried out with results. finally, in sect. 5 the final considerations and possible future work are indicated. 2 literature review the literature review used the portal de periódicos capes, seeking articles in english published from 2015 to 2021, with the following combined keywords: artificial intelligence, multiagent systems, intensive care unit, glucose control. the works of dejournett et al. [6, 7] present an autonomous glucose control system (artificial pancreas) to reduce problems resulting from the glycemia of patients in the icu. in the first work [6], the authors perform an insulin test with simulated patients, using an ai-based glucose controller. as an ai technique, they used rule-based reasoning (rbr). in the second work [7], the same system was used, but the objective was to evaluate the system’s safety and performance by applying simulated tests with swine in a clinical setting. in jemal et al. [10], a model was proposed, and a specialized decision support system was implemented and validated to detect the degree of risk of patients in the icu. a mas was used as the main technology combined with a knowledge base and intuitionistic logic fuzzy (ifs). in malak et al. [9] an architecture based on agents with decision support and in real-time for the management of high-risk newborns admitted to the icu-n was presented. both studies showed that mas is a good solution to be used in healthcare systems. when it comes to glycemic control, studies such as those presented in vehi et al. [11], bertachi et al. [12], and kim et al. [13] point to prediction models as interesting solutions, where the prevention of hypo or hyperglycemic events tends to be more efficient than the correction of these episodes. the solutions presented in these works include (1) prediction and prevention of hypoglycemic events in diabetics [11]; (2) a prediction model for episodes of nocturnal hypoglycemia in diabetics [12]; and (3) a glucose prediction model for hospitalized type 2 diabetic patients [13]. table 1 summarizes the qualitative aspects of the related work, limited to the application context (glycemic control, icu patients) and technologies used (agent-based, prediction model). note that no work presents a solution with mas to manage glycemic control with predictions in the icu setting. table 1. related work overview.full size table 3 materials and methods the methodological process used was based on previous work [14] and consists basically of four steps as presented in fig. 1 and described in the sequence. 3.1 problem definition the problem definition is based on the literature review (sect. 2). the theoretical foundation includes fast hug, glycemic control, and mas concepts. fig. 1. methodological workflow. full size image fast hug in icu management. according to the national health surveillance agency (anvisa) [15], icu is considered a complex unit to be managed in a hospital being the place where critical condition patients deserving greater care are concentrated. such patients are very difficult to manage safely and effectively due to their complex, nonlinear, and highly variable physiology. therefore, improving patients care and treatment are the main current challenges for icu settings, where personalization and automation of care offer opportunities to cause significant impacts [16]. mnemonics are commonly used in medical procedures as cognitive aids to guide doctors around the world. thus, vincent [1] suggested the simple mnemonic fast hug that contains seven essential aspects to be verified during the care of critical patients in an icu during medical rounds. such checks should take place at least once a day and ideally, whenever any of the professionals assist the patient. glycemic control. the glycemic control concerns the amount of glucose (sugar) that the patient has in the blood. both, high glucose level (hyperglycemia), and low blood glucose level (hypoglycemia) are frequent problems in icu patients causing damage to health [17]. one of the studies in vincent’s [1] highlights the importance of glycemic control in the icu. the author demonstrates that maintaining blood glucose levels at approximately 140mg/dl results in a 29.3% decrease in-hospital mortality rates, and a 10.8% reduction in the icu stay. hyperglycemia is prevalent in icu being a good example that the strict glycemic control of these patients can have a great impact. such an episode occurs due to the non-suppressed production of glucose by the body, medications, suppression of the body’s insulin secretion, among others. all these factors effectively damage the body’s normal feedback control mechanisms resulting in less insulin-mediated glucose uptake. therefore, some type of supplementary glycemic control is necessary, possibly in a closed circuit and highly personalized in intensive care [16]. table 2 was prepared in conjunction with a specialist doctor and presents, in addition to the values, what must be done for each glycemic episode. it synthesizes the information necessary to identify the glycemic episode, using a code (in the system implementation) and blood glucose values. it also presents treatment suggestions (applying glucose or insulin with the respective dosage), in addition to monitoring (how long it will be necessary to collect and measure blood glucose). this table is an adaptation of the previous work [14]. table 2. scale of glycemic values related to treatment and monitoring.full size table mas aspects. a mas is composed of two or more intelligent agents capable of perceiving events in the environment through sensors, reasoning, and acting in the environment through actuators [18]. according to [8], a mas has an intelligent distributed approach suitable for modular, changeable, and complex applications, with characteristics such as autonomy, integration, reactivity, and flexibility, becoming an interesting solution for modeling large-scale health systems. regarding the agents’ reasoning, different ai techniques can be used including the combination of them. in this work, agents should make predictions according to the patient’s blood glucose and data. also, to make treatment and monitoring suggestions for the patients. regarding predictions, predictive models can be highlighted, which can assist in decision-making [11,12,13]. in the case of monitoring and suggesting treatments, a good solution would be to represent the knowledge of medical specialists through rbr [6]. predictive model: a regression model is based on the correlation between two (simple linear regression slr) or more variables (multiple linear regression mlr), where one depends on another or others [19]. the use of regression models obtained satisfactory results in previous works of [20, 21] in the health area, more specifically, the prediction of new cases of covid-19 and prediction of glucose levels in critically ill patients, respectively. an mlr is expressed by the eq. 1, where y is the dependent variable or the value to be predicted. the \(\beta _0\) is the constant that represents the intercept of the line on the y axis, and the independent variables \(\beta _1 x_i +\beta _2 x_2 +...+\beta _kx_k\) represent the slope of the line. the x is the independent variable or predictor variable, this has the power to influence the variable to be found, and \(\varepsilon \) the variable that represents the residual factors of the measurement errors [20]. $$\begin{aligned} y = \beta _0 + \beta _1 x_i + \beta _2 x_2 + ... +\beta _k x_k + \varepsilon \end{aligned}$$ (1) rbr: dejournett [6] associates systems based on rules or knowledge with ai controllers that seek to capture the human thought process, creating rules that mimic the exact reason used by human beings. such a system is created when a domain expert joins a knowledge engineer and explains his lines of reasoning to perform certain functions when trying to control the system in question. the engineer in turn transforms the lines of reasoning into a series of if-then rules that mimic the thinking of experts in the field. assuming the situation where a patient is in the icu and the blood glucose collected value is 60mg/dl, the rule compatible with this case would be: $$\text {if glycemia = hypom then ``glucose: 2 amp--50}\%\text { iv''}$$ 3.2 project design this step includes the agents’ identification with their respective objectives and the construction of diagrams that will assist in the implementation step. agents pre-project. a mas project includes the identification of the perceptions, actions, objectives, performance, and environment of each agent in the system. the pre-project includes these definitions through the acronym peas (performance, environment, actuators, sensors). the mas pre-project serves to identify in which environment the agent will act and its respective characteristics. based on the objectives of each agent, it is also possible to describe what are the mechanisms by which they will perceive the information and how they will act in response to such stimuli [18]. tropos modeling. the mas modeling can use tropos software development methodology for agent-oriented software systems. tropos is based on the i* framework (istar intentional strategic actor relationships modeling), modeling the functionality of an application based on objectives using diagrams [22]. although there are other methodologies for agent modeling, the tropos was chosen since it encompasses all five phases of software development supporting the analysis of initial requirements to implementation with the diagrams: initial requirements, final requirements, architectural design, detailed design, and implementation. the five tropos diagrams of mas4cg can be found in [14]. figure 2 presents the diagram of the late requirements of the proposed system. red circles represent mas external actors (e.g., people, systems), yellow circles with a top straight line illustrate mas agents. the green rounded rectangles represent the agent goals that can be understood as the system requirements. the purple rectangles represent resources, a physical or informational entity needed by the actor or agent to perform a task. the mas agents are patient analyzing agent (paa), proposed treatment agent (pta), and adjustment monitoring agent (ama) with functionalities detailed fig. 3. other diagrams, such as the architecture and uml (unified modeling language), can be developed to complement the understanding of the proposed solution representing details aimed at implementing the solution. in the implementation stage, details of how all technologies, systems, and agents were implemented are presented. fig. 2. tropos late requirements diagram. full size image architecture. figure 3 presents the mas4gc architecture. the paa’s main objective is to collect the patient’s data and respective glycemia, whenever new data is inserted or updated in the glycon database. glycon is a web system used as an interface by healthcare professionals, both for entering information about the patient and their blood glucose, and for displaying the recommendations made by agents. this agent should analyze such data and make a report assessing the patient’s situation in comparison with previous data. this will allow the agent to calculate and make predictions of how the patient’s next blood glucose will be. this report will be sent to the pta and ama agents, who in turn will analyze it and propose a treatment (what should be applied, glucose or insulin, and how much) or adjust the blood glucose monitoring (indicate the appropriate frequency of blood collection). it will be possible to send an alert containing the recommendation to health professionals through the glycon interface. fig. 3. mas4gc architecture. full size image 3.3 mas4gc implementation in this work, we present a mas developed using objective-oriented agent modeling for glycemic control of patients admitted to the icu. the three agents paa, pta, and ama interact with each other and with a web system called glycon, used as an interface for collecting and presenting patient data, including blood glucose levels. the mas4gc consists of three integrated systems, one of which is an interface that serves for the entry and visualization of data through health professionals. the other is the mas itself, where, through a framework, the three agents were implemented. finally, the agents’ reasoning was built in the form of a rule base, also with the help of a framework. all source code, diagrams, dataset and frameworks used are available in the repository: https://github.com/tiagosegato/mas4gc. fig. 4. glycon interface full size image interface. the interface system called glycemic control on-line (glycon) [14] is a web system whose main objective is to receive patient data and blood glucose. health professionals can maintain this data by inserting, updating, and listing. however, the two most important tasks of the system are to record blood glucose levels and the possible applications of glucose or insulin that the patient may have received. such information is listed and presented using graphics in a kind of patient’s dashboard and alerts regarding monitoring (time of next blood collect) and treatment (glucose, keep watching, regular insulin) suggestions are displayed on the interface’s initial screen, serving as treatment suggestions to health professionals, as shown in fig. 4. glycon was developed in javascript, more specifically through the frameworks react.js (front-end) and node.js (back-end). its database uses the mongodb available through the atlas web tool. it is currently hosted on the heroku cloud application platformfootnote 1. mas framework. the mas implementation used the python agent development framework (pade) that allows the development, execution, and management of mas in distributed computing environments [23]. pade was selected in this work since it is free software that provides resources as a module for protocols implementation as defined by fipa to allow the exchange of messages and collaboration between agents. listing 1 displays the ptagent class, where the pta agent creates and sends a data request to the paa agent. after that, the paa will prepare the evaluation report that brings, in addition to patient and blood glucose data, a forecast of the patient’s next blood glucose after 4 h. the prediction was calculated using an mlr and linearregression(). the functions fit(x, y) and predict() from the python library scikit-learn were used [24]. agents’ rbr. after the paa agent generates the assessment report containing the patients’ situation, it sends the report to the pta and ama agents, who in turn consult the knowledge base coded using experta. experta is a python library that can be used in the development of rule-based systems. a system developed with experta can pair a set of facts with a set of rules for those facts and perform some actions based on the rules of correspondence [25]. with experta the specialist’s knowledge was transcribed in a knowledge base to be used by the mas4gc agents. at the moment, the rule base has 48 initial rules that indicate both treatment recommendations and blood glucose collections’ frequency. the rules were defined based on the knowledge of an intensive care specialist. in addition, new rules may be manually introduced in the rule base as tests take place in conjunction with specialists. listing 2 presents a rule where a hypos (severe hypoglycemia in table 2) situation is received and indicates that the treatment should be “glucose: 4 amp–50% iv”. afterward, this information is inserted in the database shared with glycon and it will appear on the screen of that interface. the experta syntax does not use \(if-then\) for condition and action as other rbr tools. the condition applies to the @rule expression and the result of the action is presented in a specific function in def. 4 experiments and results to evaluate the results of mas4gc, two experiments were carried out. the first aimed to identify the regression model to be used to predict patient events, as well as its accuracy, and the second to compare mas treatment suggestions with human physicians. 4.1 experiment 1 this experiment aims to identify which type of regression is the most appropriate one (slr or mlr) to predict a patient’s next glycemic event, as well as to gauge whether such a model presents satisfactory results. the experiment systematization is illustrated in fig. 5 and described in the sequence. fig. 5. experiment 1 and 2 pipeline. full size image 1. dataset used a dataset was created by an intensive care physician based on his experience to simulate synthetic data from 50 patients. each patient has 30 blood glucose collections according to the initial collection plan defined by the physician, assuming patients are not in treatment. the information present in the dataset include patient, gender, age, height, weight, bmifootnote 2, diabetes, time, time of day, food, and blood glucose. 2. data preparation some of the values available in the dataset are categorical and have undergone adjustments. from the patient, a sequential numeric code started at 1 was used, for sex 0 was defined for female and 1 for male. the values 0, 1, and 2 were used to represent those who do not have diabetes, had this information ignored, or have diabetes, respectively. the time field received a scale of values according to the difference in hours in which the collections took place and the time of day field received only the value that represents the hours (discarded minutes, seconds). the rest of the values are numeric and have been maintained, except in the case of numbers with decimal places that have been approximated for their correspondents in integer. 3. definition of the models used simple and multivariate linear regressions were used (slr and mlr). 4. dataset partitioning in the slr and mlr, the regressions were subjected to cross-validation, whose dataset was divided into 10 groups of 5 patients each, where the accuracy will be measured 10 times between the training and test sets. 5. selection of variables in the case of slr, only the time variables will be used, as the independent variable and blood glucose with the dependent one. in mlr, glycemia is also the dependent variable, as this is the value that is intended to be predicted and all other variables were used as an independent. after some significance tests, the p-value was analyzed and only the variables patient, sex, bmi, diabetes, time, time of day, food (how many hours did the patient receive food), and the last blood glucose were selected for use. 6. choice of validation metric the k-fold method was applied by creating 10 subgroups from the current base. the results will be analyzed using the following measures: coefficient \(r^2\), mean absolute error (mae), mean absolute percent error (mape), and root mean square error (rmse). 7. experiment execution the whole process was performed at first using the slr and then using the mlr. it was started by calculating the linear regression and then the predictions were calculated. 8. results analysis with the regressions and predictions performed, the results were analyzed with accuracy (coefficient \(r^2\)), and error metrics (mae, mape, rmse). as shown in fig. 6, the mlr presents the accuracy of correct answers 90% of the times that recommend the treatment to the health professional, against 13% of the slr. the mlr presents errors inferior compared to slr. thus, the mlr is more adequate in the applied context in accordance with the specialist opinion. fig. 6. the experiment 1 result analysis with the \(r^2\) coefficiente, and mae, mape, rmse error metrics per prediction of next action. full size image experiment 1 was useful to assess the accuracy of blood glucose predictions made by the paa and consequently enable health professionals to provide preventive treatment for patients. in experiment 2, the idea is to verify whether the treatment and monitoring recommendations made by the pta and ama are similar to real physicians. 4.2 experiment 2 the objective of experiment 2 is to compare the treatment and monitoring recommendations made by the mas4gc with real health professionals. six health professionals contributed to the experiment. five professionals were from different specialties, such as endocrinology, neurology, urology, gastroenterology, and physiotherapy. in addition, an iuc intensivist contributed with his knowledge in treating icu patients. the experiment systematization contains the same eight steps as in experiment 1. steps 1, 2, and 5, that is, dataset, data preparation, and selection of variables are the same. the other steps will be presented in the sequence: definition of the models used the agents’ knowledge used to make the recommendations were rbr, while human health professionals used their knowledge based on their studies and experience. dataset partitioning the \(29^th\) collection from each of the 50 patients were used, both by the intensive care physician, who analyzed the 50 cases and by health professionals from different areas, where each of the five professionals analyzed 10 cases each. choice of validation metric the mas responses compared to the health professionals and the specialist ones. experiment execution five spreadsheets were created containing an explanation of the experiment and the 10 cases. it is understood by case the patient’s data that include blood glucose among other information, described in the selection of variables step. for each case, the healthcare professional was asked to inform: (1) one treatment based on table 2; and (2) how many hours the next collection should be based on table 2. the icu specialist received a similar spreadsheet but containing all the 50 cases. results analysis after the health professionals filled out the worksheets, the data were tabulated containing the recommendations of the mas, the health professionals’ recommendations, and the icu specialist for treatment and monitoring recommendations. first, the mas treatment recommendations were matched to the exact health professionals’ recommendations. the results coincided with 80% of the cases. comparing the mas recommendations with those of the specialist physician, the combinations were 84%. regarding monitoring, the health professionals’ recommendations coincided with mas in 48% of times. concerning the mas’ recommendations with the specialist, the combinations were 78%. the results are presented in table 3. table 3. comparison of treatment and monitoring recommendations.full size table according to the icu specialist physician, an acceptable hit rate would be close to 80% of the cases. experiment 2 shows that the mas recommendations were satisfactory in most cases, except for the health professionals’ monitoring recommendation. considering table 2, the divergence of the mas and health professionals’ monitoring recommendations was because some professionals use their frequency parameters to measure blood glucose based on their icu knowledge or hospital rules (e.g., hourly collections or continuous monitoring standards). the icu specialist confirmed this observation. although the presented results are promising, other experiments should better investigate the parameters used by the software agents’ decisions, performance, and usability aspects of the mas4cg. 5 conclusion this work presents a system based on agents capable of performing glycemic control in icu patients. for this, predictive models and a knowledge base with inference rules were used to compose the agents’ intellectual capacity. according to the literature, such methods are highlighted as viable solutions in healthcare applications including glycemic control. however, it was not found the combination of these techniques with mas to solve the problem in question. two experiments were conducted to validate mas4cg. experiment 1 shows that the prediction model has a 90% accuracy rate of blood glucose for the next 4 h, allowing health professionals to anticipate and perform preventive treatment on their patients. in experiment 2, the mas treatment recommendations coincided in 84% of the cases with those of the specialist and 78% about the recommendations for monitoring (collection of blood glucose) of the patients, proving to be a viable solution. as future work, the rule base might improve the agents’ capacity about the recommendation, mainly in terms of glycemic monitoring of patients in different contexts, such as in continuous monitoring environments. the qualitative analysis on the discrepancy in time predicted for next collection is interesting and can be quantified using diagnostic test validation. considering the results as a whole, the mas4gc had a good capacity for predictions and recommendations. the promissing results indicate that the system can be tested in a real clinical environment. notes 1.available at http://glycon.herokuapp.com/. 2.bmi is a person’s weight in kilograms divided by the square of height in meters, it is the adult body mass index. references vincent, j.l.: give your patient a fast hug (at least) once a day. crit. care med. 33(6), 1225–1229 (2005) article  google scholar  noronha, k.v.m.d.s., et al.: the covid-19 pandemic in brazil: analysis of supply and demand of hospital and icu beds and mechanical ventilators under different scenarios. cadernos de saúde pública 36(6), e00115320 (2020) google scholar  socolovithc, r.l., et al.: epidemiology, outcomes, and the use of intensive care unit resources of critically ill patients diagnosed with covid-19 in sao paulo, brazil: a cohort study. plos one 15(12), e0243269 (2020) google scholar  contreras, i., vehi, j.: artificial intelligence for diabetes management and decision support: literature review. j. med. internet res. 20(5), 1–24 (2018) article  google scholar  dejournett, j., dejournett, l.: comparative simulation study of glucose control methods designed for use in the icu setting via a novel controller scoring metric. j. diab. sci. technol. 11(6), 1207–1217 (2017) article  google scholar  dejournett, l., dejournett, j.: in silico testing of an artificial-intelligence-based artificial pancreas designed for use in the intensive care unit setting. j. diab. sci. technol. 10(6), 1360–1371 (2016) google scholar  dejournett, j., nekludov, m., dejournett, l., wallin, m.: performance of a closed-loop glucose control system, comprising a continuous glucose monitoring system and an ai-based controller in swine during severe hypo and hyperglycemic provocations. j. clin. monit. comput. 35, 1–9 (2020) google scholar  darabi, z., zarandi, m.h., solgi, s.s., turksen, i.: an intelligent multi-agent system architecture for enhancing self-management of type 2 diabetic patients. in: ieee conference on computational intelligence in bioinformatics and computational biology (cibcb), pp. 1–8. ieee, niagara falls (2015) google scholar  malak, j.s., safdari, r., zeraati, h., nayeri, f.s., mohammadzadeh, n., seied farajollah, s.s.: an agent-based architecture for high-risk neonate management at neonatal intensive care unit. electron. phys. 10(1), 6193–6200 (2018) google scholar  jemal, h., kechaou, z., ben ayed, m.: multi-agent based intuitionistic fuzzy logic health care decision support system. j. intell. fuzzy syst. 37(2), 2697–2712 (2019) google scholar  vehí, j., contreras, i., oviedo, s., biagi, l., bertachi, a.: prediction and prevention of hypoglycaemic events in type-1 diabetic patients using machine learning. health informatics j. 26(1), 703–718 (2020) article  google scholar  bertachi, a., et al.: prediction of nocturnal hypoglycemia in adults with type 1 diabetes under multiple daily injections using continuous glucose monitoring and physical activity monitor. sensors (basel) 20(6), 1705 (2020) article  google scholar  kim, d.-y., et al.: developing an individual glucose prediction model using recurrent neural network. sensors (basel) 20(11), 6460 (2020) article  google scholar  segato, t.h.f., ralha, c.g., fernandes, s.e.s.: development process of multiagent system for glycemic control of intensive care unit patients. artif. intell. res. 10(1), 43–56 (2021) article  google scholar  anvisa. https://www.gov.br/anvisa/pt-br. accessed 15 may 2021 chase, j.g., benyo, b., desaive, t.: glycemic control in the intensive care unit: a control systems perspective. annu. rev. control. 48, 359–368 (2019) article  mathscinet  google scholar  braga, a.a., fernandes, m.c.c., madeira, m.p., júnior, a.a.p.: associação entre hiperglicemia e morbilidade em pacientes críticos na unidade de terapia intensiva de um hospital terciário de fortaleza ce. j. health biolog. sci. 3(30), 132–136 (2015) article  google scholar  russell, s., norvig, p.: artificial intelligence a modern approach, 3rd edn. pearson education (2010) google scholar  barbettam, p.a., bornia, a.c., reis, m.m.: estatística para cursos de engenharia e informática. 3rd edn. atlas (2010) google scholar  rath, s., tripathy, a., tripathy, a.r.: prediction of new active cases of coronavirus disease (covid-19) pandemic using multiple linear regression model. diab. metab. syndr. clin. res. rev. 14(1), 1467–1474 (2020) google scholar  zhang, z.: a mathematical model for predicting glucose levels in critically-ill patients: the pignoli model. peer j., 1–11 (2016) google scholar  pimentel, j., castro, j.: pistar tool a pluggable online tool for goal modeling. in: proceedings ieee 26th international requirements engineering conference, pp. 1–2 (2018) google scholar  pade python agent development framework. https://pade.readthedocs.io/en/latest/. accessed 17 may 2021 scikit-learn homepage. https://scikit-learn.org/stable/. accessed 14 june 2020 experta. https://experta.readthedocs.io/en/latest/index.html. accessed 17 may 2021 download references acknowledgement prof. c. g. ralha thanks the financial support from the brazilian national council for scientific and technological development (cnpq) under grant number 311301/2018-5. author information authors and affiliations instituto federal de brasília (ifb), brasília, brazil tiago henrique faccio segato & rafael moura da silva serafim universidade de brasília (unb), brasília, brazil tiago henrique faccio segato & célia ghedini ralha escola superior de ciências da saúde (escs), brasília, brazil sérgio eduardo soares fernandes authors tiago henrique faccio segatoview author publications search author on:pubmed google scholar rafael moura da silva serafimview author publications search author on:pubmed google scholar sérgio eduardo soares fernandesview author publications search author on:pubmed google scholar célia ghedini ralhaview author publications search author on:pubmed google scholar corresponding author correspondence to tiago henrique faccio segato . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper segato, t.h.f., serafim, r.m.d.s., fernandes, s.e.s., ralha, c.g. (2021). mas4gc: multi-agent system for glycemic control of intensive care unit patients. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_5 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords agent-based system blood glucose health care icu rule-based reasoning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature comparative analysis of machine learning algorithms for identifying genetic markers linked to alzheimer’s disease | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper comparative analysis of machine learning algorithms for identifying genetic markers linked to alzheimer’s disease conference paper first online: 30 january 2025 pp 157–171 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) comparative analysis of machine learning algorithms for identifying genetic markers linked to alzheimer’s disease download book pdf download book epub juliana alves  orcid: orcid.org/0000-0001-8143-80959, eduardo costa  orcid: orcid.org/0000-0003-1411-533810, alencar xavier  orcid: orcid.org/0000-0001-5034-995410,11, luiz brito  orcid: orcid.org/0000-0002-5819-092211, ricardo cerri  orcid: orcid.org/0000-0002-2582-169512 & alzheimer’s disease neuroimaging initiative show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 429 accesses abstract the identification of genetic markers for complex diseases like alzheimer’s disease (ad) is pivotal in medical genomics. this study aims to identify genetic markers associated with ad by introducing a novel approach that exclusively utilizes genetic data. our primary goals are to benchmark explainable machine learning models against blupf90, an advanced mixed linear model approach, and to uncover single nucleotide polymorphisms (snps) crucial for ad. we analyze snps to achieve these goals, focusing on the genetic heritability rate of 58–79% for ad [12]. our methodology focuses solely on genetic data to uncover snps crucial for ad, employing transparent computational models to ensure interpretability alongside predictive power. the findings demonstrate the efficacy of a purely genomic approach combined with machine learning to advance our understanding of ad. our methodology successfully identified a robust set of snps associated with ad, encompassing both previously recognized and novel snps. the machine learning models employed delineated distinct snp profiles, highlighting the complexity and heterogeneity of ad. these results not only deepen our understanding of ad’s genetic underpinnings but also facilitate the development of targeted therapeutic and diagnostic strategies, showcasing the potential of computational techniques in medical genomics. data used in preparation of this article were obtained from the alzheimer’s disease neuroimaging initiative (adni) database (adni.loni.usc.edu). as such, the investigators within the adni contributed to the design and implementation of adni and/or provided data but did not participate in analysis or writing of this report. a complete listing of adni investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/adni_acknowledgement_list.pdf. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others machine learning in alzheimer’s disease genetics article open access 22 july 2025 predicting early alzheimer’s with blood biomarkers and clinical features article open access 13 march 2024 identifying key genetic variants in alzheimer’s disease progression using graph convolutional networks (gcn) and biological impact analysis article open access 16 july 2025 1 introduction alzheimer’s disease (ad) is a progressive neurodegenerative disorder associated with aging. it can cause cognitive and psychiatric symptoms, potentially leading to disability. according to the world alzheimer report, the number of ad cases is projected to reach 131.5 million by 2050 [29]. ad is characterized by synaptic damage and neuronal loss in brain regions responsible for cognitive function [33]. researchers have been looking for genetic markers linked to alzheimer’s disease (ad) to create personalized medicine for individuals at risk and enhance their quality of life. the human genome project, initiated in the mid-1990s, greatly expanded genetic data availability through dna sequencing. today, personalized medicine incorporates tailored medical treatments based on an individual’s genetic characteristics, including factors like single nucleotide polymorphisms (snps) that aid in predicting disease risk for individuals [22]. this study seeks to find genetic markers, like single nucleotide polymorphisms (snps), that are linked to ad. this type of research is called genome-wide association studies (gwas). snps are variations at specific locations in the dna chain and are classified based on the type of nucleotide substitution that takes place [10]. snps are the most commonly occurring type of genetic polymorphism in human genetics, and can impact various human traits and their development in a specific environment. additionally, snps are evolutionarily stable, meaning that there is little variation among different generations [10]. in gwas, genotypic and phenotypic data are collected from diverse individuals, and quality control procedures ensure the data’s reliability. after gwas, analyses are performed to interpret the results. these analyses involve adjusting the significance threshold, presenting results in a manhattan plot, and conducting bioinformatics analyses to identify biological mechanisms underlying observed associations. gwas is a powerful approach for identifying genetic variants associated with complex diseases or traits, but it is essential to consider various factors carefully to ensure valid and interpretable results. advancements in dna sequencing technologies have resulted in an abundance of genetic data, leading to improvement of personalized medicine [5, 20, 22, 35]. the rapid growth of data has made machine learning algorithms a valuable tool in genetics. researchers have shown that machine learning techniques can predict diseases based on genetic factors. in the domain of machine learning, algorithms are designed to model the intricate associations between risk single nucleotide polymorphisms (snps) and disease phenotypes [17]. supervised learning techniques are employed, where algorithms are trained using labeled datasets to enable precise classification or prediction. specifically, regression algorithms have been utilized to discern the most significant variables in extensive datasets comprising hundreds or thousands of snps [16]. this study harnesses these supervised machine learning methods to predict alzheimer’s disease by identifying pivotal snps. this study also focused on explainability to identify the most important features, known as single nucleotide polymorphisms (snps). to achieve this, several machine learning algorithms that offer insights into feature importance were used, including random forest, xgboost, and logistic regression. these algorithms were applied to the datasets to pinpoint the most significant snps, based on their feature importance scores [17]. both random forest and xgboost are renowned for their capability to manage large datasets with numerous features, such as the genetic data containing thousands of single nucleotide polymorphisms (snps). the configuration of key parameters such as maximum tree depth (max_depth), number of trees (n_estimators), and the maximum number of features considered for splitting a node (max_features) significantly impacts these models’ ability to detect intricate interactions among snps. for example, increasing max_depth enables the models to discern more detailed patterns, potentially identifying complex snp interactions that hold biological relevance for alzheimer’s disease (ad). however, excessively deep trees risk overfitting, capturing noise rather than valid biological signals. consequently, careful tuning of these parameters is essential to optimize the balance between accuracy and generalization in the models [26, 36]. although not as commonly utilized as tree-based models in genomic studies, we included logistic regression in this study to compare and analyze its predictive power. this model was enriched by tuning hyperparameters such as the penalty type (penalty—l1, l2, elasticnet) and regularization strength (c), which influence the model’s sparsity. specifically, l1 regularization promotes feature selection by zeroing out less important features, thereby enabling the model to highlight snps most predictive of alzheimer’s disease (ad). this approach not only simplifies the genetic model but also focuses on the most biologically relevant markers. the remainder of this paper is organized as follows: sect. 2 presents some related studies; sect. 3 presents our proposed approach; sect. 4 presents and discusses our results; and finally, sect. 5 presents our conclusions and future works. 2 related work jin et al. [18] conducted a systematic exploration of machine learning (ml) and deep learning (dl) techniques for identifying and analyzing biomarkers associated with alzheimer’s disease. their review highlights significant research findings, such as the use of lasso for feature selection [39], and identification of hub genes associated with immune function and neuroinflammation [40]. the study showcases notable algorithms like differential gene selection tabnet for effective gene-based classification of ad. the study conducted by araujo et al. [3] focuses on the use of random forest algorithms and gene network analysis to investigate the correlations between snps and alzheimer’s disease. the research employs random forest algorithms due to its effectiveness in handling large datasets and managing the complexity inherent in genetic data. the study’s findings suggest that certain snps are significantly associated with the disease, which offers potential new insights into its genetic basis. a study by sherif et al. [34] found genetic variations linked with alzheimer’s disease using a multi-stage system. they used a supervised bayesian network and discovered the most ad-related snp. their results showed that endothelial-based markov methods were better than naive bayes and naive tree-fed bayes, but their work is still ongoing in drug discovery. [2] study uses machine learning algorithms to detect alzheimer’s disease early by analyzing snps. the research focuses on identifying genetic markers predictive of the disease’s early onset. the work showcases the efficacy of combining detailed genetic data with precise machine learning techniques to enhance early detection strategies. our study differs from previous research by focusing solely on genetic data to identify markers for alzheimer’s disease (ad), not including environmental factors and pre-filtered genetic markers. we utilize interpretable models to ensure transparency and compare our findings with blupf90, an advanced mixed linear model approach. our approach emphasizes the impact of genetic data on advancing our knowledge of ad’s genetic foundations and supporting the creation of precise therapeutic and diagnostic approaches. fig. 1. methodology architecture full size image 3 methods the methodology adopted for this study is structured into five crucial stages, as follows: (1) data acquisition; (2) data processing for quality control; (3) gwas using blup family of programs; (4) employment of advanced machine learning methods on the dataset identified as optimal in the previous phase; (5) comparative analysis of the results achieved in the gwas and machine learning stages. figure 1 schematizes the mentioned stages and their respective sub-tasks, which will be detailed in the subsequent sections of this article. 3.1 data acquisition the data used in this study were obtained from the alzheimer’s disease neuroimaging initiative (adni) database [1]. launched in 2004 as a public-private partnership, adni aims to identify whether brain images, biological markers, clinical evaluations, and neuropsychological assessments can be combined to measure the progression of mild cognitive impairment (mci) and alzheimer’s disease in early stages. the initial genotypic data contained 620,901 variants, with an average of 30,785 missing variants. the input file included 757 individuals, comprising 449 males and 308 females. the phenotype was divided into three categories: normal (cn), with 214 samples; alzheimer’s disease with 175; and mild cognitive impairment (mci) with 367. these categories were estimated by adni using various biomarkers, which are substances, measurements, or indicators of a biological state that can be identified before the onset of clinical symptoms. in addition to genotypic data, a dataset of individual phenotypes was also obtained. this dataset contains information on sex, age, ethnicity, and gender. 3.2 data processing for quality control in this study, quality control (qc) methods preprocess the input data for genome-wide association studies (gwas). the data includes individual ids, disease stage, and genotype information. the employed qc filters include minor allele frequency (maf), which excludes snps with a maf less than certain percentages to filter out rare variants; linkage disequilibrium (ld) for detecting snp clusters linked to specific traits; hardy-weinberg equilibrium (hwe) to identify unusual allele frequencies; and checks for genotype and sample missingness, removing entries exceeding predefined thresholds. these measures are crucial for reducing data complexity and improving the reliability of the analysis. 3.3 best linear unbiased prediction (blup) the best linear unbiased prediction (blup) software suite is employed to select the most suitable dataset after applying a series of quality control filters. the selected dataset is then used in machine learning models to enhance the study of alzheimer’s disease, focusing particularly on the significance of genetic markers known as single nucleotide polymorphisms (snps). these markers are analyzed to understand their influence on the disease. consequently, blup serves two primary purposes in this study: firstly, to select the best dataset based on a combination of qc parameters, and secondly, to evaluate the significance of snps through its predictive modeling capabilities. the blup includes several key programs, each playing a vital role in the analysis, being them: a) runum: creates necessary parameter files for other software components. b) thrgibbs1f90: employs gibbs sampling to estimate important genetic variations, essential for analyzing both simple and complex genetic traits. c) postgibbsf90: summarizes statistical samples, providing estimates of genetic variation. d) blupf90 and postgs: these programs calculate p-values for each snp, identifying key genetic markers for alzheimer’s disease. the genomic model. the genetic analysis is conducted by applying a genomic model that predicts genetic susceptibility to alzheimer’s disease based on snp data, represented by the equation: $$\begin{aligned} y = x\beta + zu + e \end{aligned}$$ (1) where: y represents the observed traits (phenotypes). x and z are matrices that connect observations to fixed and random effects, respectively. \(\beta \) represents fixed effects. u is a vector of random effects. e is the error term. this model facilitates the exploration of how specific snps may be associated with alzheimer’s disease. assessing significance. the significance of snp associations is quantified through p-values, which evaluate the likelihood that the observed genetic effects are due to chance. lower p-values indicate a stronger statistical connection to alzheimer’s disease, suggesting significant roles for certain snps in its development. 3.4 machine learning algorithms in this study, we employed the algorithms logistic regression, xgboost, and random forest. these models were trained using a training set (x_train, y_train) consisting of 80% labeled data for alzheimer’s disease and cognitively normal and then internally validated using the 5-fold cross-validation technique. randomizedsearchcv was utilized to evaluate various hyperparameter combinations, with optimization based on the f1 scorer. to prevent any warnings associated with division by zero in the f1 score, the zero_division value was set to 1. finally, the model was externally evaluated on the test set (x_test, y_test). table 1 provides a comprehensive list of all hyperparameters used for fine-tuning. $$\begin{aligned} f_1 = 2 \cdot \frac{\text {precision} \cdot \text {recall}}{\text {precision} + \text {recall}} \end{aligned}$$ (2) table 1. hyperparameters for logistic regression, random forest, and xgboostfull size table for logistic regression, we adjusted various settings including the regularization strength (c), penalty type, solver used, and the l1 ratio. the random forest model was fine-tuned by adjusting the number of trees, their maximum depth, and the number of features considered at each split to enhance overall performance. similarly, the xgboost model was adjusted to prevent overfitting and improve accuracy by optimizing parameters like the number of trees, learning rate, tree depth, and sampling methods. 3.5 gene retrieval this study retrieved gene information using biopython, a tool that interfaces primarily with the national center for biotechnology information (ncbi) databases. this approach allowed for the systematic extraction of gene data based on specific single nucleotide polymorphisms (snps). the identified genes were compared to the ones present in the existing literature about alzheimer’s disease. this study used data sourced from the alzheimer’s disease neuroimaging initiative (adni), where participant information is pre-anonymized to protect privacy. our access to and use of this data strictly adhered to all guidelines provided by adni, which are designed to comply with ethical standards for research involving human subjects. since the data was pre-anonymized and publicly available, this study did not require additional ethical approval from an ethics committee. 4 results and discussion 4.1 data quality control to ensure optimal results, a quality control standard was implemented, which included a minor allele frequency (maf) of 0.01, a linkage disequilibrium (ld) threshold of 85, a hardy-weinberg equilibrium (hwe) significance of 5e-6, a sample missingness rate of 0.01, and a gene missingness rate of 0.1. the snp rs7918269 produced the most significant outcome, with a p-value of 1.600000e-09. please note that the qc hyperparameters can impact the results, and thus, selecting the ideal hyperparameters is crucial for achieving optimal outcomes. 4.2 machine learning hyperparameter selection in this study, tuning of hyperparameters was critical to optimize each machine learning model’s performance for predicting alzheimer’s disease using snps. table 3 illustrates the selected hyperparameter settings for the logistic regression, random forest, and xgboost algorithms, ensuring each model’s efficacy and robustness. 4.3 prediction performance analysis although the predictive performance of the models was not the primary focus of this study, their ability to make accurate predictions was analyzed. overall, the models demonstrated similar accuracy but exhibited significant differences in precision and recall. the random forest model had higher precision but lower recall compared to the other models, indicating a tendency to frequently predict the negative class (cn). logistic regression and xgboost performed better in maintaining a balance between precision and recall, with xgboost slightly outperforming. however, all models have room for improvement, especially in terms of balancing performance metrics. the detailed results obtained from the machine learning models are described as follows: random forest, logistic regression, and xgboost. the random forest model achieved an accuracy score of 57.69%, with a perfect precision of 100% but a very low recall of 5.71%, resulting in an f1 score of 10.81%. this indicates a strong bias toward predicting the negative class, limiting its ability to accurately identify the positive class. logistic regression also achieved an accuracy of 57.69%, with a precision of 60% and a recall of 17.14%, leading to an f1 score of 26.67%. compared to random forest, logistic regression exhibited a slightly better balance between precision and recall. lastly, the xgboost model recorded an accuracy of 53.85%, with a precision of 47.62%, recall of 28.57%, and an f1 score of 35.71%. although xgboost had a lower overall accuracy, it showed a more balanced performance across all metrics (table 2). table 2. ad prediction resultsfull size table table 3. selected hyperparameters for machine learning modelsfull size table the similar accuracy observed across the random forest, logistic regression, and xgboost models can be attributed to several factors, including the complexity of alzheimer’s disease, the quality and representation of the genetic data, model-specific characteristics, hyperparameter tuning processes, inherent data limitations, and the choice of evaluation metrics. these factors collectively influence the models’ performance, leading to convergent accuracy levels despite their different architectures and mechanisms. 4.4 comparative analysis of alzheimer’s disease associated snps table 4. 20 most significant snps for logistic regression and random forest.full size table table 5. 20 most significant snps for xgboost and gwas (blupf90).full size table after implementing the machine learning techniques, we conducted a genome study and benchmarked the results of each algorithm. this comparison was performed by selecting the top 20 most significant snps for each method, as presented in tables 4 and 5, and analyzing and comparing the identified genes with those reported in the literature. instances of absent genes in the tables 4 and 5 indicate ’no gene association’. this phenomenon can be attributed to the occurrence of snps in non-coding regions of the genome. while not directly associated with known genes, such regions can still play a crucial role in gene regulation or the production of non-coding rnas, thereby indirectly influencing gene function [14]. through this analysis, we identified that the models successfully detected snps previously associated with alzheimer’s disease and other neurological functions. this finding underscores the potential of ml models for identifying genetic markers linked to complex diseases such as alzheimer’s disease. logistic regression identified specific snps with positive and negative coefficients, indicating the presence of both protective and risk alleles for ad. on the other hand, the random forest model assigned greater significance to a different set of snps, with feature importances varying in the order of \(10^{-4}\). the xgboost model identified a collection of snps, with importances ranging around \(10^{-2}\), emphasizing the relevance of each snp more equally. finally, the gwas analysis using blupf90 provided a separate set of snps based on highly p_value, some of which were not detected by the other models. based on the xgboost model analysis, several genes and single nucleotide polymorphisms (snps) were identified in relation to alzheimer’s disease (ad). the galnt13 gene, crucial for neural development, is strongly associated with ad [9, 19]. the rs6507641 snp in the slc14a1 gene, involved in urea transport, has also been linked to ad [31]. the rs6037744 snp, although not directly associated with a specific gene, is linked to increased susceptibility to optic nerve degeneration in glaucoma. both conditions involve nerve cell death and abnormal proteins, suggesting a broader neurodegenerative process [37]. studies have also found that glaucoma patients are four times more likely to develop dementia, highlighting a potential connection between glaucoma and ad [28]. lastly, the rs10933234 snp in the sphkap gene has been suggested to be involved in the ad process [38]. for the logistic regression model, it was found that the fgf13 gene is related to ameliorating amyloid-\(\beta \)-induced neuronal damage, acting as a protective factor [24], as shown by the negative coefficient in the logistic regression. the arhgap36 gene has been associated with excitatory-inhibitory neuronal analyses within the dg granule cell layer, also serving as a protector [25]. the abca4 gene, linked to rs497511, is highly associated with ad and frontotemporal lobar degeneration with tdp-43 protein inclusions [21]. the sytl2 gene was found among the top genes with significant alzheimer’s loci in the study conducted by oxford [11]. the expression of frmpd4, indicated by rs7880350, is significantly altered in the ad hippocampus [8]. based on the random forest model, the cd207 gene was identified as one of the top 10 most differentially expressed genes in ad, according to a study [15]. the or51b6 gene, an olfactory receptor, shows the highest number of associated variants and is expressed in temporal cortex neurons [32]. the sox13 and plxnd1 genes were associated with ad in various studies [6, 27]. the pde8b gene shows altered mrna expression in ad brains at different disease stages [30]. blup discoveries identified several genes previously associated with ad in the literature, including cntn6, klf12, rps6ka2, mln, fancc, abca8 and adcy9. these genes are essential in regulating synaptic plasticity, biomarkers for ad, inflammatory responses, neuritogenesis, and biochemical pathways of ad. these discoveries offer valuable insights into the molecular mechanisms underlying ad and could lead to the development of innovative diagnostic tools and treatments for this debilitating disease [4, 7, 13, 23]. comparing the significant snps identified by each model can offer valuable insights into the genetic mechanisms of ad. the observed variations have potential implications that can guide future research in the field. these findings can reveal new biological pathways or confirm the relevance of already known ones, serving as a basis for further investigations. in turn, this can contribute to the advancement of our understanding of the pathogenesis of ad. 5 conclusion and future work this study has demonstrated the efficacy of integrating advanced machine learning algorithms and genomic analysis to identify significant genetic markers associated with alzheimer’s disease (ad). utilizing methods such as random forest, logistic regression, xgboost, and genomic-wide association studies via blupf90, we have identified a comprehensive set of snps, which includes both previously documented and novel genetic markers. these findings not only underscore the potential of machine learning to enhance our understanding of ad but also highlight the utility of genomic data in developing personalized medical interventions. while this study has made significant progress in understanding the genetic basis of alzheimer’s disease, it is essential to note that the complexity of the disease involves various genetic, environmental, and lifestyle factors, underscoring the need for more comprehensive research. in the future, studies should include larger and more diverse datasets that incorporate these factors to improve the accuracy and generalization of the results. additionally, it is critical to incorporate sophisticated computational models that can handle such multidimensional data. while explainability is a concern, methods, such as shapley additive explanations (shap) values, can assist with black box models. further investigations should also focus on validating the novel snps identified in this study, examining their biological relevance, and understanding how they interact with other genetic and environmental factors in the pathogenesis of ad. this validation is essential for translating these findings into clinical practice, where they can inform the development of targeted therapies and diagnostic tools. in conclusion, our study represents a significant advance in the genetic study of alzheimer’s disease, opening new avenues for research and potential therapeutic interventions. the insights gained here provide a foundation for future studies aimed at unraveling the complex genetic networks involved in ad and developing more effective strategies for its prevention, diagnosis, and treatment. references alzheimer’s disease neuroimaging initiative (adni) database. http://adni.loni.usc.edu. accessed 25 jan 2024 ahmed, h., soliman, h., elmogy, m.: early detection of alzheimer’s disease based on single nucleotide polymorphisms (snps) analysis and machine learning techniques, pp. 1–6 (10 2020). https://doi.org/10.1109/icdabi51230.2020.9325640 araújo, g., souza, m., oliveira, j., costa, i.: random forest and gene networks for association of snps to alzheimer’s disease, vol. 8213, pp. 104–115 (2013). https://doi.org/10.1007/978-3-319-02624-4_10 boche, d., gordon, m.: diversity of transcriptomic microglial phenotypes in aging and alzheimer’s disease. alzheimer’s dementia 18, 360–376 (2021). https://doi.org/10.1002/alz.12389 article  math  google scholar  bueno, m.r.p.: o projeto genoma humano. bioética 5, 1–10 (2009). https://revistabioetica.cfm.org.br/revista_bioetica/article/view/378 carulli, d., winter, f., verhaagen, j.: semaphorins in adult nervous system plasticity and disease. front. synapt. neurosci. 13 (2021). https://doi.org/10.3389/fnsyn.2021.672891 chen, m.j., et al.: extracellular signal-regulated kinase regulates microglial immune responses in alzheimer’s disease. j. neurosci. res. 99(6), 1704–1721 (2021). https://doi.org/10.1002/jnr.24829 cheng, j., liu, h.p., lin, w.y., tsai, f.j.: machine learning compensates fold-change method and highlights oxidative phosphorylation in the brain transcriptome of alzheimer’s disease. sci. reports 11 (2021). https://doi.org/10.1038/s41598-021-93085-z doe, j., smith, j., brown, a.: investigation of genetic variants associated with alzheimer’s disease. j. alzheimer’s res. 12(4), 567–578 (2023). https://doi.org/10.1002/alz.12345 article  math  google scholar  edwards, d., forster, j.w., chagné, d., batley, j.: what are snps?, pp. 41–52. springer, new york (2007). https://doi.org/10.1007/978-0-387-36011-9_3 gao, s., casey, a., sargeant, t., mäkinen, v.p.: genetic variation within endolysosomal system is associated with late-onset alzheimer’s disease. brain: j. neurol. 141 (2018). https://doi.org/10.1093/brain/awy197 gatz, m., et al.: role of genes and environments for explaining alzheimer disease. arch. gen. psychiatry 63, 168–74 (2006). https://doi.org/10.1001/archpsyc.63.2.168 ghani, m., et al.: genome-wide survey of large rare copy number variants in alzheimer’s disease among caribbean hispanics. g3 (bethesda, md) 2, 71–78 (2012). https://doi.org/10.1534/g3.111.000869 giral, h., landmesser, u., kratzer, a.: into the wild: gwas exploration of non-coding rnas. front. cardiovascul. med. 5, 181 (2018). https://doi.org/10.3389/fcvm.2018.00181 article  google scholar  guennewig, b., et al.: defining early changes in alzheimer’s disease from rna sequencing of brain regions differentially affected by pathology. sci. rep. 11 (2021). https://doi.org/10.1038/s41598-021-83872-z géron, a.: mãos à obra: aprendizado de máquina com scikit-learn & tensorflow. alta books editora, o’reilly (2002) math  google scholar  ho, d., schierding, w., wake, m., saffery, r., o’sullivan, j.: machine learning snp based prediction for precision medicine. front. genet. 10, 267 (2019). https://doi.org/10.3389/fgene.2019.00267 article  google scholar  jin, y., et al.: classification of alzheimer’s disease using robust tabnet neural networks on genetic data. math. biosci. eng. 20, 8358–8374 (2023). https://doi.org/10.3934/mbe.2023366 johnson, m., davis, e., wilson, l.: genetic and epigenetic mechanisms in alzheimer’s disease. nature 14(7), 123–134 (2023). https://doi.org/10.1038/s41398-023-02446-x article  math  google scholar  johnson, s.g.: chapter 1 genomic medicine in primary care. in: david, s.p. (ed.) genomic and precision medicine, 3rd edn, pp. 1–18. academic press, boston (2017). https://doi.org/10.1016/b978-0-12-800685-6.00001-1 katzeff, j.s., lok, h.c., bhatia, s., fu, y., halliday, g.m., kim, w.s.: atp-binding cassette transporter expression is widely dysregulated in frontotemporal dementia with tdp-43 inclusions (2023). https://doi.org/10.3389/fnmol.2022.1043127 laksman, z., detsky, a.s.: personalized medicine: understanding probabilities and managing expectations. j. gen. intern. med. 26, 204–206 (2011). https://doi.org/10.1007/s11606-010-1515-6 article  google scholar  li, j., et al.: genome-wide network-assisted association and enrichment study of amyloid imaging phenotype in alzheimer’s disease. curr. alzheimer res. 16 (2019). https://doi.org/10.2174/1567205016666191121142558 li, r., xiao, l., zhang, t., ren, d., h, z.: overexpression of fibroblast growth factor 13 ameliorates amyloid-b-induced neuronal damage. neural regeneration research 18, 1347–1353 (2023). https://doi.org/10.4103/1673-5374.357902 mishra, r., et al.: augmenting neurogenesis rescues memory impairments in alzheimer’s disease by restoring the memory-storing neurons. j. exp. med. 219 (2022). https://doi.org/10.1084/jem.20220391 moore, p.j., lyons, t.j., gallacher, j., initiative, a.d.n.: random forest prediction of alzheimer’s disease using pairwise selection from time series data. plos one 14(2), e0211558 (2019). https://doi.org/10.1371/journal.pone.0211558 morabito, s., et al.: single-nucleus chromatin accessibility and transcriptomic characterization of alzheimer’s disease. nat. genet. 53, 1–13 (2021). https://doi.org/10.1038/s41588-021-00894-z ou, y.: alzheimer’s disease and glaucoma: is there a connection? (2017). https://www.brightfocus.org/alzheimers/article/alzheimers-disease-and-glaucoma-there-connection. accessed 30 june 2024 prince, m., et al.: world alzheimer report 2015 (2015). https://www.alz.co.uk/research/worldalzheimerreport2015.pdf. accessed 21 apr 2020 pérez-torres, s., mengod, g.: camp-specific phosphodiesterases expression in alzheimer’s disease brains. int. congr. ser. 1251, 127–138 (2003). https://doi.org/10.1016/s0531-5131(03)00104-3 article  google scholar  recabarren, d., alarcón, m.: gene networks in neurodegenerative disorders. life sciences 183 (2017). https://doi.org/10.1016/j.lfs.2017.06.009 sepulveda-falla, d., et al.: genetic modifiers of cognitive decline in psen1 e280a alzheimer’s disease. alzheimer’s dementia 20 (2024). https://doi.org/10.1002/alz.13754 sereniki, a., vital, m.a.b.f.: a doença de alzheimer: aspectos fisiopatológicos e farmacológicos. revista de psiquiatria do rio grande do sul 30 (2008). https://doi.org/10.1590/s0101-81082008000200002 sherif, f., zayed, n., fakhr, m.: discovering alzheimer genetic biomarkers using bayesian networks. adv. bioinform. 2015, 639367 (2015). https://doi.org/10.1155/2015/639367 spiegel, a.m., hawkins, m.: personalized medicine to identify genetic risks for type 2 diabetes and focus prevention: can it fulfill its promise? health aff. 31(1), 43–51 (2012). https://doi.org/10.1377/hlthaff.2011.1054 article  math  google scholar  velazquez, m., lee, y. for the alzheimer’s disease neuroimaging initiative: random forest model for feature-based alzheimer’s disease conversion prediction from early mild cognitive impairment subjects. plos one 16(4), 1–18 (2021). https://doi.org/10.1371/journal.pone.0244773 wiggs, j., et al.: common variants at 9p21 and 8q22 are associated with increased susceptibility to optic nerve degeneration in glaucoma. plos genet. 8, e1002654 (2012). https://doi.org/10.1371/journal.pgen.1002654 wu, z., et al.: a novel alzheimer’s disease prognostic signature: identification and analysis of glutamine metabolism genes in immunogenicity and immunotherapy efficacy. sci. rep. 13 (2023). https://doi.org/10.1038/s41598-023-33277-x yu, w., yu, w., yang, y., lü, y.: exploring the key genes and identification of potential diagnosis biomarkers in alzheimer’s disease using bioinformatics analysis. front. aging neurosci. 13, 602781 (2021). https://doi.org/10.3389/fnagi.2021.602781 article  math  google scholar  zhu, m., tang, m., du, y.: identification of tac1 associated with alzheimer’s disease using a robust rank aggregation approach. j. alzheimers dis. 91, 1–11 (2023). https://doi.org/10.3233/jad-220950 article  math  google scholar  download references acknowledgment this study was financed by the são paulo research foundation (fapesp) grants #2020/08634-2, #2021/12618-5 and #2022/02981-8. author information authors and affiliations department of computing, federal university of são carlos, são carlos, sp, brazil juliana alves corteva agriscience™, mogi mirim, sp, brazil eduardo costa & alencar xavier department of animal sciences, purdue university, west lafayette, in, usa alencar xavier & luiz brito institute of mathematics and computer science, university of são paulo, são carlos, sp, brazil ricardo cerri authors juliana alvesview author publications search author on:pubmed google scholar eduardo costaview author publications search author on:pubmed google scholar alencar xavierview author publications search author on:pubmed google scholar luiz britoview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar consortia alzheimer’s disease neuroimaging initiative corresponding authors correspondence to juliana alves or ricardo cerri . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alves, j., costa, e., xavier, a., brito, l., cerri, r., alzheimer’s disease neuroimaging initiative. (2025). comparative analysis of machine learning algorithms for identifying genetic markers linked to alzheimer’s disease. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_11 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords alzheimer’s disease gwas disease prediction genetic markers single nucleotide polymorphisms (snps) machine learning publish with us policies and ethics profiles eduardo costa view author profile luiz brito view author profile ricardo cerri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature active learning and case-based reasoning for the deceptive play in the card game of truco | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper active learning and case-based reasoning for the deceptive play in the card game of truco conference paper first online: 28 november 2021 pp 313–327 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) active learning and case-based reasoning for the deceptive play in the card game of truco download book pdf download book epub daniel p. vargas10, gustavo b. paulus10 & luis a. l. silva10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 755 accesses 2 citations abstract deception is an essential behavior in many card games. despite this fact, it is not trivial to capture the intent of a human strategist when making deceptive decisions. that is even harder when dealing with deception in card games, where components of uncertainty, hidden information, luck and randomness introduce the need of case-based decision making. approaching this problem along with the investigation of the game of truco, a quite popular game in southern regions of south america, this work presents an approach that combines active learning and case-based reasoning (cbr) in which agents request a human specialist to review a reused game action retrieved from a case base containing played truco hands. that happens when the agents are confronted with game situations that are identified as opportunities for deception. the goal is to actively capture problem-solving experiences in which deception can be used, and later employ such case knowledge in the enhancement of the deceptive capabilities of the truco agents. experimental results show that the use of the learned cases enabled different kinds of truco agents to play more aggressively, being more deceptive and performing a larger number of successful bluffs. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others combining case-based reasoning and reinforcement learning for tactical unit selection in real-time strategy game ai chapter © 2016 computing and predicting winning hands in the trick-taking game of klaverjas chapter © 2019 towards human-like bots using online interactive case-based reasoning chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. action theory behavioral methods logic in ai game theory games studies logic 1 introduction deception involves a deliberate attempt to introduce in another person a false belief or belief in which the deceiver considers false [1, 2]. such deceptive behavior can be modeled as a) concealment, aiming to hide/omit the truth and b) simulation, whose purpose is to show the untruth [3]. to conceal the deceiver acts by withholding information and omitting the truth. to simulate, in addition to the retention of genuine information, unreal information is presented as being legitimate. among the various forms of deception is the bluff, where deception and bluffing is interchangeably used in this work. in the context of a card game, a bluff is an action where players, to deceit their opponents, seek to make an illusory impression of strength when they hold weak hands. alternatively, players may try to show that their strong hands have little value in the game. to card games with hidden, stochastic and imperfect information, acting deceptively is an essential strategy for players to succeed. the use of deceptive moves can also be closely related to the nature of some popular games, and the entertainment that it introduces in the game disputes. to do so, players should be able to identify what the best opportunities for the use of deception are, considering the strength of their hand and their betting history in order to make themselves as unpredictable as possible [4]. in general, real-world situations present complex characteristics for the modeling of deceptive agents, such as the need for learning and decision-making with a small number of training examples, for instance. for the development of agents capable of acting deceptively in card games, this work explores case-based reasoning (cbr) [5]. with relevant explanatory capabilities, cbr combines learning and problem-solving with the use of specific knowledge captured in the form of cases. in particular, this technique has supported the development of agents which competitively play poker [6, 7]. in this line of research, this paper extends past work [8,9,10] in the cbr modeling of a popular game in the southern regions of south america, a game that is under-investigation in artificial intelligence (ai): the game of truco [11]. cbr allows continuous learning by retaining concrete problem-solving experiences in a reusable case base. despite this fact, it is not simple to capture and label the intention of human players when making deceptive moves in card game. to approach this problem, active learning [12] is investigated in the analysis of truco opportunities for being deceptive, and the consequent collection of such problem-solving experiences in a case base. then, the acquired case knowledge is used to equip different kinds of cbr agents to make deceptive actions. in the proposed approach, the case learning is focused on the review of decisions and retention of cases in the case base. as a result of the implemented solution reuse policy, whenever a game action is reused by the agent, if a certain pre-established learning criterion is met, the agent requests for a human expert to review the reused game action and the current game state. if the solution presented by the reuse policy is not considered to be the most effective according to the judgment of the domain specialist, the expert suggests a game action to be played (deceptive or not). with attention to the capture and reuse of deceptive game actions from human players, the contributions of this paper are: i) the exploration of active learning to support the retention of case problem situations in which deceptive moves can be used, ii) the performance evaluation of deceptive truco agents configured according to alternative solution reuse policies, and iii) the analysis of the resulting game playing behavior of the implemented agents when using case bases storing the collected problem-solving experiences. 2 background to this work cbr [5] combines learning and problem-solving with the use of knowledge obtained from concrete problem-solving experiences. learning in cbr aims to acquire, modify or improve different knowledge repositories [13], where the enhancement of the case base is often sought in different applications. in doing so, it is possible to explore the automatic case elicitation (ace) technique [14]. this technique focuses on the system’s ability to explore its domain in real time and automatically collect new cases. another technique is the learning by observation, also referred as demonstration learning or imitation learning. in such learning modality, the system learns to perform a certain behavior by observing an expert to act [15]. the first learning stage is the acquisition of cases from the expert demonstrations. the second stage is the resolution of a problem using the case base collected from the observations [16]. an alternative to learning by observation is the active learning, where the goal is to obtain greater quality in the learning process considering the smallest possible number of labeled instances. active learning tries to overcome the labeling and data distribution bottlenecks by allowing the learner to intelligently choose the most representative instances. this model allows requesting that a human expert present a solution to the problem. later it allows adding the resolved instances to the training set. 2.1 cbr, active learning and games active learning and cbr have been explored in a number of digital game applications. in the smile stochastic mixing iterative learning [17] game, smile controls the agent while observing the specialist behavior. when the game iteration ends, smile uses the collected observations to train a new policy that can be used in subsequent game iterations. the dagger – dataset aggregation [18] algorithm enhances smile by preventing the agent from selecting actions using outdated policies. in doing so, the agent updates a single policy learned each iteration. in both smile and dagger, the control to determine whether the player is the agent or the specialist is defined probabilistically. the salt algorithm – selective active learning from traces [19, 20] allows the learner agent to perform a task, and when it is determined that the agent has left the space for which it has training data, the control is assigned to an expert. as in smile and dagger, the focus is on the collection of training data for the set of states that are expected to be found during testing. unlike smile and dagger, control in salt is assigned to the specialist only when the agent leaves the state-space of the training set. the training data is generated only when the specialist is in control, reducing the specialist’ cognitive load. with regard to expert consultation strategies, in [21], the retrieval of most similar cases is used to determine a game action to be taken according to a vote. considering the average similarity value of the cases retrieved from the last five decisions made in the game and a coefficient obtained from a linear regression, which determines whether the similarities are increasing or decreasing during the last performed movements, the cbr agent gives the game control to the human expert. this happens whenever the mean similarity is increasingly moving away from the space of known situations. the expert plays until the states of the game are familiar again. to avoid continuous changes between the cbr agent and the human specialist, each one has to perform certain minimum play before giving control to the other. in [22] and [15], a similarity threshold value is used to determine when the human specialist is consulted. then the specialist automatically gives control to the cbr agent after performing a move in the game. moreover, the retention of cases in the case base only happens when the human specialist is in control. unlike passively acquired cases, which can result in the retention of redundant cases in the case base, the use of active learning in these games allowed the learning of certain situations that would not be observed in a purely passive manner. to achieve a reasonable expert imitation, active learning required a considerably lower number of cases than when a fully passive approach was used. in contrast to these past works, this work actively learns only in the resolution of the required problems, which are identified as deception game opportunities. in addition to use a similarity threshold, the condition to query the human specialist is combined with a strategy that employs hand strength and probability to determine whether a situation is opportune for the use of deceptive moves. instead of using active learning to collect any kind of expert experience of game playing, this work direct such learning to the improvement of the deceptive capabilities of card playing agents. the ai research has also investigated the effectiveness of cbr in the modeling of card games, mainly with respect to the game of poker [7, 23]. considering deceptive-related poker strategies, however, only [7] explicitly addresses this issue. there, the developed agent, whose case base starts empty, performs the random play strategy to populate the case base. with respect to the game of truco, [10] addresses the case retention problem, especially considering the lack of large numbers of cases. it investigates alternative learning techniques such as ace, imitation learning, and active learning to enable an agent to learn how to act in situations in which past case knowledge is limited. through the assistance of a human player, the purpose of the active learning technique is to guide the agent in its use of any kind of game action whenever the agent had not encountered similar game situations stored in the case base. despite this research, truco matches disputed amongst the agents implemented according to the analyzed learning techniques showed that, unlike the automatic retention and the retention of new cases strategies, which demonstrated an improvement in the agents’ performance, the active learning technique did not show an improvement in the agents’ performance. unlike [10], which performed a broad collection of case situations in truco, this paper investigates the use of active learning in the analysis of deceptive game opportunities and, for those in which the expert decided that it was worth acting deceptively, the collection of new problem-solving experiences. [8, 9] address the indexing of the truco case base through the organization of cases into different clusters. using such clusters, the goal was to identify game actions along with game states in which such actions are performed. in addition, it is proposed a two-step solution reuse model, which is further explored in our work. the model involves a step that retrieves the most similar cases for a given query, where a reuse criterion is used in the choice of the group of cases that is more similar to the current query situation (extra cluster reuse criterion). after selecting this group of cases, a filtering is performed in order to select only the retrieved cases that belong to the chosen group. based on these filtered cases, a second reuse step can use another reuse criterion to choose the game action that is used to solve the current problem (intra cluster reuse criterion). the reuse policies that showed to be the most effective according to their experiments are described in table 1. table 1. reuse policies used by the implemented truco agents.full size table considering the cases retrieved from a given query, the number of points solution criterion (nps) involves the reuse of game actions, where the game action choice is supported on the amount of earned points due to the use of that action in the game. the probability victory (pv) criterion involves the choice of either clusters (pvc) or game actions (pvs) to be reused (or both in the pvcs), where the reuse is based on the calculation of the chances of victory for each of the different game actions recorded in the retrieved cases. these policies were thoughtfully explored in the development of the truco playing agents investigated in this paper. 2.2 the card game of truco truco is a widely practiced card game in southern regions of south america [11]. the ai techniques covered in this work were investigated with the use of matches disputed between two opposing players. such blind truco version (truco “cego”) uses 40 of the 48 cards in the spanish deck, as the four eights and four nines are removed. the deck is divided into “black” cards, which are the cards with figures (king – 12, horse – 11 and sota – 10), and “white” cards that are from ace to seven. in truco, the dispute takes place through successive hands that are initially worth a point. each player receives three cards to play one hand. a hand can be divided into two phases of dispute: envido and truco. in each stage, players have different ways to increase the number of points that are played. each hand can be played in a best of three rounds, in which the player who plays the highest card in each round wins. finally, the match comes to an end when a player reaches twenty-four points. envido is a dispute that takes place during the first round of a hand. such a dispute is based on the sum of the value of each one of the player’s cards. for envido, each card is worth the value presented in it, with the exception of “black” cards that are not computed in the sum of points. envido has the following betting modalities: envido, real_envido, and falta_envido, which the player can bet before playing the first card on the table. if a player advances any one of these bets, the opponent can accept or deny the envido dispute. there is a special case of envido, which is called flor. the flor occurs when a player has three cards with the same suit. the flor bet cancels any envido modality previously advanced since it increases the value of the envido dispute. as in envido, flor allows one to fight back (e.g. contra_flor) if the opponent also has three cards of the same suit. when the envido dispute ends, the truco phase begins. at this stage, one to four points are disputed during the three rounds of the hand (one for each card in the hand). in one round, each player drops a card at the table starting with the hand player’s or the winner from the previous round. these cards are confronted according to a truco ranking involving each card. the player who wins two of the three rounds wins the hand. unlike bet actions for envido, which can only be placed during the first round of each hand, truco bets can be placed at any time during a hand dispute. in addition, if a player decides to go to the deck, the opponent receives a score of points equivalent to the points in dispute in that truco stage. similar to other card games, such as in the different variations of poker, for example, the game of truco involves different degrees of deception/bluffing. these strategies allow players to win hands and even matches in situations where they do not own strong cards for the envido and truco disputes. most importantly, human players in real-life truco matches employ deceptive actions with certain frequency. among other reasons, this behavior makes the game more fun, even if such bluffs don’t necessarily result in better results in the game. 3 active learning and cbr in the card game of truco agents can employ cbr to learn game strategies for playing truco. in our work, whenever such agents take the game turn, they evaluate the current state of the game. to do it, a query containing the game state information is formed. then, the k-nn algorithm is executed along with a similarity function that averages case attribute similarities to perform the retrieval of past cases from the case base. after retrieval, the selected cases are used to generate a game move which is played in the current game situation. the reuse is supported by a reuse policy which defines, among other criteria, the number of similar cases considered in the solution choice and the minimum similarity value (threshold, set to 98% in this work) so that the solutions represented in the retrieved cases are reused in the resolution of the current problem. at the end of such problem-solving procedure, the system can decide whether the derived problem-solving experience is worth retaining as a new case in the case base. 3.1 the case base formation a web-based system was developed to permit the collection of truco cases, where these cases were the result of truco matches played between two human opponents who had various levels of truco experience. at the end of each disputed truco hand, a new case (i.e. a hand of truco) was stored in the case base. in our project, 147 matches were played among different players using this system. in total, 3,195 cases were collected and stored into a case base called baseline. to represent the cases, a set of attributes captured the main information and game actions employed in the truco disputes. table 2 summarizes these attributes. table 2. attributes for representing a truco case.full size table the played cards were recorded according to a numerical codification. the encoding uses a nonlinear numerical scale ranging from 1 to 52. code 1 is assigned the cards with the lowest value (all 4’s). code 52 is assigned the highest value card, which is the ace of spades. then it was explored both in the representation of cases and in the similarity evaluations. in effect, the codification is based on both the categories identified in [24] and the truco knowledge from our research group participants. each value in this encoding represents the relative strength of the truco cards. to collect deceptive game information to support the development of the active learning task (only used during the active case learning) through the course of each played truco match, other set of attributes were added into the case representation model. these attributes are described in table 3. table 3. attributes regarding the deceptive actions made in the truco match.full size table with respect to the deceptive actions performed by truco players, case attributes to represent the deception information were used to measure the similarity of the current game situation in relation to the cases stored in a learning case base. the purpose was to determine whether such case base had enough records of problem opportunities for using deceptive actions in order to solve the game problems encountered in the matches in which the active learning tasks were executed. 3.2 game actions and deception truco has various kinds of game actions. to support the analysis of deceptive truco behaviors, we classified as aggressive the truco playing actions involving betting or raising an opponent’s bet. similarly, passive are the actions in which the player should decide either accepting or denying an opponent’s bet. in addition, aggressive game actions can be labeled as either honest or deceptive. in aggressive moves, the player can most effectively employ deception. in passive moves, the player has the opportunity to detect the opponent’s deception since the opponent is either betting or increasing a bet. in table 4, we analyze such deceptive game actions in truco. table 4. possible types of deceptive game actions in truco.full size table 3.3 hand strength truco is played with 40 of the 48 cards of the spanish deck, where there are 9,880 possible hands. with this, it is possible to sort and classify each hand according to their strength for the envido and truco disputes. the envido hand strength is directly based on the envido points. to calculate the strength of a truco hand, the relative strength and importance of each card that forms the hand have to be considered. a method to calculate such hand strength can be derived from the analysis of two components: a) the strength of the two highest value hand cards and b) the strength of the two lower value hand cards. this method considers the truco rules since a hand dispute is played in a best of three rounds. the two highest hand cards a player possesses (high and medium cards, see table 2) are more important in the estimation of final hand strength. to have two high cards in a hand tends to increase the player’s chances of winning in the best-of-3 competition. a low card among these higher hand cards has a high negative impact on the final hand strength. on the other hand, a low card between the two lowest hand cards (medium and low cards) should also have a negative impact on the final hand strength. however, this impact is not as severe in the calculation of the hand strength as it is the impact of owning a low card between the two highest ones. the method explores the calculation of means between the hand card numerical encodings (i.e. the non-linear encoding from 1 to 52). the first calculates a harmonic mean (1) between the two highest hand cards (high and medium cards). when one value much lower than another is used in this type of harmonic mean calculation, the final result of the computed mean tends to be reduced toward the lowest value. $$ {\text{m}}_{1} = \frac{2}{{{\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {highcard}}}\right.\kern-\nulldelimiterspace} \!\lower0.7ex\hbox{${highcard}$}} + {\raise0.7ex\hbox{$1$} \!\mathord{\left/ {\vphantom {1 {mediumcard}}}\right.\kern-\nulldelimiterspace} \!\lower0.7ex\hbox{${mediumcard}$}}}} $$ (1) the second uses the calculation of a weighted arithmetic mean (2) between the two lower hand cards (medium and low cards). in this case, the weight attributed for the highest card between these two lowest hand cards was set to double the weight of the lowest card. the use of a weighted arithmetic mean also allows expressing the impact of a low card on the hand strength. however, the weight of having a high card between the two lower ones should be greater than the weight of having a lower card among these two lower cards. $$ {\text{m}}_{2} = \frac{{\left( {2*mediumcard} \right) +_{ } lowcard_{ } }}{2 + 1} $$ (2) to reach the final value of the hand strength, a weighted arithmetic mean (3) is calculated between the results obtained from the two mean values computed with (1) and (2). $$ {\text{m}}_{3} = \frac{{\left( {2*{\text{m}}_{1} } \right)_{ + } {\text{m}}_{2} }}{2 + 1} $$ (3) according to numerical tests, it was not possible to identify hand situations in which our calculations of hand strength presented unsatisfactory results. qualitatively, either higher or lower values than those obtained by the use of our method could be argued as relevant in some situations. in these situations, even without the use of our method, the strength of the considered hands is subject to debate, especially when we considered the truco rules and the different ways of deceptively playing in this game. 3.4 triggering the expert consultation as part of the proposed active learning approach, two strategies to trigger a human specialist consultation are proposed in the work. first, the coverage of a case base is used as a trigger to consult the expert. to do it, the similarity between the current game situation and the cases learned through active learning is computed when a query is emitted. the query is performed on the case base containing the newly retained cases: the learning case base. such query considers the case attributes that are relevant for each type of decision in the game. in addition, the case attributes referring to the previously taken deception decisions in the match (table 3) are considered in the similarity computations. the 98% similarity threshold was used to determine whether the learning case base had sufficient coverage to resolve the current query situation. second, the trigger for the expert consultation is also directed to the identification of a problem opportunity to play deceptively. to define whether a particular decision-making scenario is characterized as an opportunity for such bluffing, the number of possibilities of certain truco events is computed. with this, for example, it is possible to determine the probability of an opponent having more envido points than the points of the agents envido, using the card that had already been played by the opponent and the position of the agent on the table. moreover, in each moment of the hand competition and in each type of game move, whenever there is a probability of success lower than 50%, the game situation can be classified as an “opportunity for deception”. similarly, when the probability of success is higher than 85%, the agent may also adopt a slow playing deceptive move. such estimate of the winning odds is computed in each new decision state of the game. it is updated according to the information revealed throughout the hand dispute. the following example shows how this probability calculation is performed. given the following agent’ cards: , which are removed from the deck, it is possible to calculate that the opponent can have c37,3 = 7,770 possible hands. then the strength of each possible opponent hand is compared with the strength of the agent’s hand in each of these 7,770 card combinations. to do so, our method for calculating the hand strength is used. the result is that the agent has a better hand than the opponent in 5,106 hands. by computing the probability, there is a 66% chance of having a better hand than the opponent’s hand. 4 experiments and results the developed experiments aimed to evaluate the effectiveness of the proposed active learning and cbr approach in the collection and exploration of deception cases in the stochastic and imperfect information game of truco. these experiments were organized as follows: a) case learning, covering the acquisition of cases through active learning, b) agent performance, referring to the analysis of agent victories with and without the use of the collected cases, and c) agent behavior, concerning the evaluation of the set of decisions taken by the agents with and without the use of the collected cases. the tested truco playing agents were implemented according to the four different solution reuse policies listed in table 1. different case bases were used by them: a) the initially collected case base (baseline, storing 3,195 cases), collected from matches played amongst human players, b) the resulting case base later built in this work (active, storing 5,013 cases), which increased the baseline case base with the new cases collected through active learning. to analyze the different game playing strategies adopted by the agents, according to the reuse policies along with their respective case bases, a number of evaluation attributes was used (table 5). table 5. game attributes observed in the analysis of the implemented agents.full size table using the attributes described in table 5, the analyzed strategies were the following: i) honest-deceptive: indicating the rate of deceptiveness, it expresses the relationship between the total number of deceptive game moves and the total number of game moves. the higher the value is, the more deceptive the agent behavior is; ii) successful bluff: indicating the rate of bluff effectiveness, it corresponds to the relationship between the number of successful bluffs and the total number of bluffs; and iii) passive-aggressive: indicating the rate of aggressiveness, it captures the relationship between the number of aggressive moves of the envido/truco-type and the total number of envido/truco-type game moves, including when the agent does not bet. the higher the value is, the more aggressive the agent behavior is. 4.1 the active learning experiment to build the case base via active learning, 148 truco matches were played between the agents implemented according to the tested reuse policies. the reuse policies used by each agent were randomly chosen at the beginning of each match. only one of the players in each match had the ability to consult the expert human player, where the expert player was the first author of this paper. during the collection of such cases via active learning, the agents computed their decisions by using the baseline case base. in these learning matches, whenever the learning criterion related to the detection of deception problem opportunities was satisfied, the learning algorithm presented information about the game to the specialist. then, the expert reviewed whether the reused game action provided an effective solution for the current problem. with that, the expert player decided whether to maintain the decision recommended by the automated reuse policy or to perform another game action, deceptive or not. the expert decision was stored as a new case in a separate case base containing situations and decisions (learning case base). new cases were stored in this case base only when the specialist performed an intervention by changing the game action suggested by the reuse policy. in total, 1,818 new cases were stored in the learning case base. when the reviewed game actions were used by the agents, they won 79% of the disputes played during this learning experiment. 4.2 the evaluation experiments due to luck and randomness, the quality of the truco cards received by each player is likely to have a large variation. to reduce this imbalance in the evaluation experiments, the dispute model described by the annual computer poker competition (acpc) [25] was adopted. this model employs duplicate matches, in which the same set of hands is distributed in two sets of matches. in doing so, players reverse their positions at the table when playing the second match. because all players receive the same set of cards, this dispute model allows a fair assessment of agents’ ability. fig. 1. competition results between (a) agents implemented with the baseline case base only and (b) agents implemented with the active case base only. full size image in the first set of tests, a competition between the four implemented agents was developed, where all of them competed against the others in a total of 300 truco matches. the agents only used the baseline case base in all these matches. the results in fig. 1 (a) indicate that the pvcnps and nps agents achieved the best performance. regarding the analysis of their deceptive characteristics, even the baseline case base, which did not yet retained the cases collected via active learning, allowed these agents to deceptively play. in fact, that baseline case base collected from human players already stored deceptive problem-solving experiences which were reused by the agents throughout these matches. in the second set of tests, the test setup was similar to the previous one. however, the tested agents only used the active case base. so the aim was to analyze whether the cases collected through the proposed active learning approach permitted to improve the agents’ deceptive capabilities, and how such behavior change was expressed in the different kinds of tested agents. the results in fig. 1 (b) indicate that the use of new cases collected via active learning enabled the pvs and pvcs agents to have the most significant performance improvement. while only the pvs and pvcs agents improved their aggressiveness rates, all tested agents increased their deceptiveness and successful bluff rates. fig. 2. competition results between agents implemented with the baseline case base versus agents implemented with the active case base. full size image in the third set of tests, each one of the four implemented agents was now configured to use different case bases: baseline and active. in a total of 200 played matches, each kind of agent implemented with the use of one of these case bases played against its correspondent using the other case base. the results in fig. 2 (a) indicate that the agents implemented with the use of the cases collected via active learning achieved a superior performance in relation to the others. the tests also permitted to observe the behavior of the implemented agents according to their reuse policies and the different case bases used to compute their game decisions. figure 2 (b) allowed analyzing the tested agents according to the honesty level, showing that the agents with the active case base were more deceptive than the others. figure 2 (c) allowed comparing the agents according to their aggressiveness, showing that the active case base enabled the tested agents to be more aggressive behaviors. figure 2 (d) allowed analyzing the assertiveness rate of performed bluffs by each one of the agents and case bases. again, the results show that the agents with the active case base deceived better than their baseline’s correspondents (with a single exception: the nps agent). in addition, the relationship between the agents’ performance and the adopted game behaviors is apparent since the reuse policies that obtained the best performance (pvs and pvcs) were those that played more aggressively, were more deceptive and performed a larger number of successful bluffs. despite losing their matches, such behavior could also be observed with the better performing agents implemented with the baseline case base: the nps and pvcnps agents. 5 final remarks this work investigates the integration of active learning and cbr, two different but complementary ai techniques, aiming to permit card playing agents to make better decisions when faced with problem opportunities to deceive. the experiments show that the actively learned cases allowed the tested agents to achieve a better game playing performance. regarding the agents’ playing behaviors, the collected cases allowed them to more assertively act in deceptive problem situations. the cbr reuse policies that benefited the most, improving their deceptive behavior, were the ones that implemented the “probability victory” criterion (pvs and pvcs). as future studies, we can suggest the analysis of how deception could be related to other cbr techniques, e.g. in the execution of deceptive similarity computations. further tests involving the implemented agents playing against human players are also relevant to improve the techniques proposed in this paper. references buller, d.b., burgoon, j.k.: interpersonal deception theory. commun. theory 6, 203–242 (1996) article  google scholar  depaulo, b.m., lindsay, j.j., malone, b.e., muhlenbruck, l., charlton, k., cooper, h.: cues to deception. psychol. bull. 129, 74–118 (2003) article  google scholar  ekman, p.: telling lies: clues to deceit in the marketplace, politics, and marriage. w.w. norton & company, inc. (2009) google scholar  billings, d., davidson, a., schaeffer, j., szafron, d.: the challenge of poker. artif. intell. j. 134, 201–240 (2002) article  google scholar  mantaras, de., et al.: retrieval, reuse, revision and retention in case-based reasoning. knowl. eng. rev. 20, 215–240 (2005) article  google scholar  rubin, j., watson, i.: computer poker: a review. artif. intell. 175, 958–987 (2011) article  mathscinet  google scholar  sandven, a., tessem, b.: a case-based learner for poker. in: the ninth scandinavian conference on artificial intelligence (scai 2006), helsinki, finland (2006) google scholar  paulus, g.b., assunção, j.v.c., silva, l.a.l.: cases and clusters in reuse policies for decision-making in card games. in: ieee 31st international conference on tools with artificial intelligence (ictai 2019), portland, or, pp. 1361–1365 (2019) google scholar  paulus, g.b.: cases and clusters for the development of decision-making reuse policies in card games (written in portuguese). in: programa de pós-graduação em ciência da computação, vol. master in computer science, p. 132. universidade federal de santa maria (2020) google scholar  moral, r.c.b., paulus, g.b., assunção, j.v.c., silva, l.a.l.: investigating case learning techniques for agents to play the card game of truco. in: xix brazilian symposium on computer games and digital entertainment (sbgames 2020), recife, brazil, pp. 107–116 (2020) google scholar  winne, l.l.: truco. ediciones godot, ciudad autónoma de buenos aires (2017) google scholar  settles, b.: active learning literature survey. department of computer sciences, university of wisconsin–madison (2009) google scholar  richter, m.m.: knowledge containers. in: watson, i. (ed.) readings in case-based reasoning. morgan kaufmann publishers, san francisco (2003) google scholar  neto, h.c., julia, r.m.s.: ace-rl-checkers: decision-making adaptability through integration of automatic case elicitation, reinforcement learning, and sequential pattern mining. knowl. inf. syst. 57(3), 603–634 (2018). https://doi.org/10.1007/s10115-018-1175-0 article  google scholar  floyd, m.w., esfandiari, b.: supplemental observation acquisition for learning by observation agents. appl. intell. 48(11), 4338–4354 (2018). https://doi.org/10.1007/s10489-018-1191-5 article  google scholar  ontanon, s., floyd, m.: a comparison of case acquisition strategies for learning from observations of state-based experts. in: the 26th international florida artificial intelligence research society conf. (flairs 2013), florida, usa (2013) google scholar  ross, s., bagnell, d.: efficient reductions for imitation learning. in: yee whye, t., mike, t. (eds.) the thirteenth international conference on artificial intelligence and statistics, vol. 9, pp. 661–668. pmlr (2010) google scholar  ross, s., gordon, g., bagnell, j.a.: a reduction of imitation learning and structured prediction to no-regret online learning. in: the 14th international conference on artificial intelligence and statistics (aistats), ft. lauderdale, fl, pp. 627–635 (2011) google scholar  packard, b., ontanon, s.: policies for active learning from demonstration. in: 2017 aaai spring symposium series. stanford university (2017) google scholar  packard, b., ontanon, s.: learning behavior from limited demonstrations in the context of games. in: the 31st int. florida artificial intelligence research society conf. (flairs 2018), florida, usa (2018) google scholar  miranda, m., sánchez-ruiz, a.a., peinado, f.: towards human-like bots using online interactive case-based reasoning. in: bach, k., marling, c. (eds.) iccbr 2019. lncs (lnai), vol. 11680, pp. 314–328. springer, cham (2019). https://doi.org/10.1007/978-3-030-29249-2_21 chapter  google scholar  floyd, m.w., esfandiari, b.: an active approach to automatic case generation. in: mcginty, l., wilson, d.c. (eds.) iccbr 2009. lncs (lnai), vol. 5650, pp. 150–164. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-02998-1_12 chapter  google scholar  rubin, j., watson, i.: case-based strategies in computer poker. ai commun. 25, 19–48 (2012) article  mathscinet  google scholar  sobrinho, m.g.: manual do jogo do truco cego (flor de abóbora). martins livreiro editora ltda., porto alegre (2004) google scholar  acpc: annual computer poker competition. http://www.computerpokercompetition.org/ (2018) download references author information authors and affiliations graduate program in computer science, federal university of santa maria – ufsm, santa maria, rs, brazil daniel p. vargas, gustavo b. paulus & luis a. l. silva authors daniel p. vargasview author publications search author on:pubmed google scholar gustavo b. paulusview author publications search author on:pubmed google scholar luis a. l. silvaview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper vargas, d.p., paulus, g.b., silva, l.a.l. (2021). active learning and case-based reasoning for the deceptive play in the card game of truco. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_21 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deception case-based reasoning active learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature havana: hybrid attentional graph convolutional network semantic venue annotation model | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper havana: hybrid attentional graph convolutional network semantic venue annotation model conference paper first online: 30 january 2025 pp 291–305 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) havana: hybrid attentional graph convolutional network semantic venue annotation model download book pdf download book epub germano b. dos santos  orcid: orcid.org/0000-0002-4758-17769,11, paulo h. c. silva  orcid: orcid.org/0009-0000-4200-96509, fabrício a. silva  orcid: orcid.org/0000-0002-0713-05839,11, thais r. m. braga silva  orcid: orcid.org/0000-0003-4924-74619,11 & … linnyer b. r. aylon  orcid: orcid.org/0000-0002-4456-682910,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 379 accesses abstract the increasing geospatial data availability has enabled extensive urban mobility studies. however, some tasks require point-of-interest labels, which are missing or inaccurate on public datasets. in this context, the existing solutions fail to utilize different types of convolutional filters regarding graph neural networks, thereby hindering their performance in labeling place categories. to address these shortcomings, this work proposes a new model to annotate points of interest semantically characterized by a hybrid architecture that uses a spatial and spectral filter integrated with a self-attention mechanism. our results demonstrate an improvement of up to 25.05% in f1-score compared to three state-of-the-art models across three different datasets. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an attention enhanced graph convolutional network for semantic segmentation chapter © 2020 an attentive hierarchy convnet for crowd counting in smart city article 22 september 2022 multiple views extraction from semantic trajectories chapter © 2022 1 introduction location-based social networks (lbsns) capture the intrinsic mobility characteristics of visitation patterns, providing a vast source of data. thereby, the availability of such amount of data contributes to broad research topics towards understanding human routine and preferences through mobility such as place recommendation [11], inferring the next place that the user tends to visit [3], learning urban region representations [24], and describe urban functional zones [12]. these studies rely on place categories, e.g., shopping, food, or nightlife, distinguishing a space or a mobility trajectory semantically, thus supplying relevant information to accurately accomplish the desired task. nowadays, there are various lbsns data sources available, such as yelp, foursquare, and gowalla. the first source possesses reviews of geolocated business places, while foursquare and gowalla characterize the interactions, i.e., visits also called check-ins, between places and humans. however, these data sources frequently have missing features, and incorrect or inconsistent labels, e.g., a place can be a restaurant in foursquare and have another label in yelp [20], leading to errors in different domains, such as site selection and urban planning [19]. in addition, there are a number of geolocated datasets from different sources containing only the position (i.e., latitude/longitude) that could be enriched by their semantic categories such as crime events, trajectory data, and geotagged social media data [1]. therefore, it is necessary to develop methods that can accurately label the points of interests (pois), a problem also known as semantic venue annotation, since the place category is essential for gaining in-depth knowledge of user displacements [16]. various approaches have explored different methods to address the semantic venue annotation problem. [19] propose a machine learning model, while [25] leverage spatial representation learning to label the pois. additionally, [3] formulated the temporally sorted visits as a graph and applied graph neural networks. although this model outperforms others, it solely relies on spectral-based convolution, neglecting the benefits of diverse convolution operations available [31]. based on that, we propose a novel method, called havanafootnote 1 (hybrid attentional graph convolution for semantic venue annotation) that integrates spatial and spectral convolution to enhance the accuracy of semantic venue annotation. specifically, our model combines the spatial and spectral convolution, represented respectively by graph attention network (gat) [23] and auto-regressive moving average (arma) [2], into a hybrid block. furthermore, the output of this novel block is aggregated using self-attention [22] and a two-layer feedforward network with residual connections [10]. in summary, the contributions of this study, in terms of model architecture, are two-fold: propose a hybrid spatial-spectral-based convolution. output aggregation based on a combination of self-attention, feedforward network, and residual connections. in the experiments, havana outperforms other three state-of-the-art methods by up to 25.05% on the f1-score averaged, 19.9% on the weighted f1-score, and 16.26% on accuracy in three different datasets. the remainder of this study is organized as follows: sect. 2 describes the related work. section 3 presents the fundamental theory related to graph neural networks. section 4 details the preprocessing and the model proposed. in sect. 5 the results and the ablation study are discussed. in sect. 6 the ethical use of location-based services is considered. finally, in sect. 7 we present the conclusion and future works. 2 related work in this section, previous works addressing semantic venue annotation have been examined, and their limitations are discussed. the solutions are separated into three domains: classic machine learning models, representation learning, and graph neural networks. considering the classical machine learning models, [29] fuse explicit attributes related to user shifting behavior in different time intervals, with implicit attributes that capture the similarities between the places and users. [18] adapt the k nearest neighbors algorithm under the most favorable assumption to ensure the most probable label given mobility records distance between places categories. [19] evaluate the importance of four different attributes – visual, operation (i.e., closing/opening hours), spatial and textual – modeling the problem with xgboost. on another view, the representation learning methods assume that similar places have similar latent features, and thus the objective is to project the features to obtain representations that contribute to semantic venue annotation. hence, hmrm [5] possesses a collaborative component to learn seven mobility-related matrices with non-negative matrix factorization given co-occurrences between time, places, and visits frequencies matrices. aside from the decomposition, [25] apply dual contrastive learning, assigning a similarity constraint between the context category and the targeted category to the skip-gram model. similarly, the graph neural network-based methods aim to project the features into latent space, minimizing the distances between similar places, thereby naturally modeling the problem of semantic venue annotation [31]. for instance, [3] propose the pgc-nn that utilizes a multi-view comprising individual and collective mobility matrices as input into a spectral convolution filter arma [2] to label the places. another study introduces the stpa (spatial-textual poi annotation) [30] which merges spatial and textual characteristics. spatial features are derived from a delaunay graph and a graph convolutional network [14] while the textual is based on the names of pois and a transformer module [22]. for aggregation, attention weights are applied to these features. on the other hand, leveraging the power of multi-modal data, [8] rely on the higher-order networks by modeling the relationship between users’ mobility and places with hypergraph. to accomplish the classification, the study uses a graph convolution network for hypergraphs. despite the advances, the application of multiple spectral and spatial convolutional operators in semantic venue annotation remains unexplored. inspired by performance achieved by hybridization applied in hyperspectral image classification [28], our proposed method introduces a graph hybrid component, addressing different data modalities as specified in [3, 8, 30]. additionally, to output aggregation, the novel model combines a two-layer feedforward network and a self-attention weight with residual connections to prevent overfitting [10]. 3 preliminaries and problem definition in this section, definitions and theories required to comprehend the model proposed are presented. specifically, the definition and the usage of check-ins on points of interest, as well as the structure of the graph neural networks are discussed. 3.1 semantic venue annotation a location-based social network contains check-ins \(c = \{(p_i, t_i)\}\), which are events characterizing a user’s visit to a place \(p_i\) on a timestamp \(t_i\). this place is known as a point of interest (poi) \(p = \langle l, cat \rangle \), which is represented by a location l and a category cat. therefore, each user can be represented as a general graph g(v, e) given his/her temporally ordered events where \(v = \{p_1, p_{i}, \ldots , p_{n}\}\) is a set of pois visited and \(e = \{e_1, e_{i}, \ldots , e_{m}\}\) is a set of displacements from a source place s to a destination place d at a timestamp, then \(e_i = \langle p_s, p_d, t_i \rangle \). considering that multi-view features provide a richness and diverse perspective to comprehend the data [26], we propose a multi-view graph hybrid learning with individual and collective mobility components to infer the place categories correctly. 3.2 hybrid graph neural networks in the light of multi-view graph learning, we leverage the strengths of graph neural networks to correlate the places and mobility patterns addressing the semantic venue annotation task. then, this section introduces the theory behind the different convolutions used in the proposed model. note that classifying places’ categories is the same as the node classification task. for this reason, nodes and places are used interchangeably. thus, given g(v, e) the general graph presented in the previous section, each place has a feature vector represented by \(\vec{f}\). let \(a \in \mathbb {r}^{n \times n}\) the adjacency matrix of graph g and n the length of the set v, it is possible to define the information propagation and the neighborhood feature aggregations using the spatial and spectral convolution operators [31]. the spatial convolution relies on the graph topology, propagating the information through the neighborhood [9]. although the size of the neighborhood impacts the node representation learning quality, a large neighborhood leads to inefficiency due to time constraints. therefore, to surpass these problems, gat [23] use an attention-based aggregation module [22], where \(\vec{f}\) is weighted by its importance to a specific node. on the other hand, the spectral convolution transforms the adjacency matrix into a laplacian matrix assuming the advantages of filtering noises using the spectral domain applying spectral filters. these filters operate on eigenvectors of the laplacian matrix, which is computationally costly. therefore, some works try to approximate the eigenvectors to alleviate this processing overhaul [7, 14]. in this work, we used the auto regressive moving average (arma) [2] filter since it captures a higher-order neighborhood with fewer parameters compared to the other spectral filters. thus, the principles applied by spectral and spatial convolution are different; the first is governed by the structure of the graph, while the latter evaluates the local neighborhood. then, to harness these two important principles, we combine the arma convolution and gat convolution into one block named hybrid convolution that inputs an adjacency matrix and a feature vector, producing two outputs corresponding to each convolution. 3.3 adjacencies and features the method proposed comprises a two-view adjacency matrices and feature vectors characterizing a multi-view graph learning: (1) an individual component representing the visiting user pattern, (2) a collective component capturing the general mobility pattern. considering the location-based social network data, we can view the general graph \(g = (v,e)\) from different perspectives, i.e., individual patterns and collective patterns, which provide fine- and coarse-grained information on visitation patterns, respectively. regarding the graph neural network modeling, the characteristics are divided into adjacency matrix and feature matrices as discussed above. the individual features are represented by the sets \(\mathcal {a}' = \{a^u, a^u_{wd}, a^u_{we}\} \subset \mathcal {a}\) and \(f' = \{t^u, t^u_{wd}, t^u_{we}, \textit{dur}^u, \textit{dis}^u\} \subset f\). correspondingly, the collective features are represented by the adjacency matrix \(a^f\) and the feature matrix \(t^f\). considering the multi-view features, the set of adjacency matrix can be defined as \(\mathcal {a} = \mathcal {a}' \cup \{a^f\}\) and the set of feature matrix can be defined as \(f = f' \cup \{t^f\}\). regarding the adjacency matrices set \(\mathcal {a}'\), \(a^u \in \mathbb {r}^{\left| p^{u} \right| \times \left| p^{u} \right| }\) represents the adjacency of points of interest categories consecutively visited by a user u, where \(p^{u}\) is the places set. the matrix \(a_{wd}^{u} \in \mathbb {r}^{\left| p^{u} \right| \times \left| p^{u} \right| }\) specifies the adjacency of the categories of pois visited consecutively during the weekdays, while \(a_{we}^{u} \in \mathbb {r}^{\left| p^{u} \right| \times \left| p^{u} \right| }\) denotes the adjacency of weekends. with respect to set f, the matrix \(t^{u} \in \mathbb {r}^{\left| p^{u} \right| \times 48}\) describes the temporal individual characteristics, where each value stands for the total number of visits to a poi by a user u in each hour of the day, i.e., 24-time slots for weekdays and 24-time slots for weekends. this matrix captures the global correlations between visits and timestamps. besides that, we have two other specific matrices for weekdays and weekends, formally described as \(t_{wd}^{u} \in \mathbb {r}^{\left| p^{u} \right| \times 24}\) and \(t_{we}^{u}\in \mathbb {r}^{\left| p^{u} \right| \times 24}\). furthermore, \(dur^u \in \mathbb {r}^{\left| p^u \right| \times \left| p^u \right| }\) denotes the median duration of consecutive visits to the pois \(p^u\). finally, \( dis^{u} \in \mathbb {r}^{\left| p^{u} \right| \times \left| p^{u} \right| }\) represents the geodesic distance between consecutively visited pois by a user u. equivalently, the collective features follow a similar definition. the matrix \(a^{f} \in \mathbb {r}^{\left| p \right| \times \left| p \right| }\) describes all the visits to the pois and the \(t^f \in \mathbb {r}^{\left| p \right| \times 48}\) defines the count of interactions given a time slot, 24-time slots on weekdays and 24-time slots on weekends. due to the length of the set p, it is not possible to store these features matrices in-memory, thus the pointwise mutual information (pmi) is applied in order to measure the associativity between two points of interest by the co-occurrence, i.e., evaluates the context of the visits to a poi. 4 havana havana is structured into three modules as shown in fig. 1: individual, collective, and an aggregation module. the individual and collective components associate different levels of mobility information to accurately represent the places’ context. the third module is the aggregation of each block by applying the self-attention weighting with residual connections. fig. 1. havana comprises three modules: individual, collective, and aggregation. the latter component, which is generally represented in this figure, is responsible for combining the outputs of each hybrid convolutional block. full size image on the one hand, the individual component comprises five hybrid convolutional blocks \(b_h\), with each block consisting of gat [23] and arma [2] networks. on the other hand, the collective component is composed of a single hybrid block \(b_h\) and a feedforward block \(b_{mlp}\). each convolution block \(b_h\) takes two inputs: an adjacency matrix \(a_i \in \mathcal {a}\) and a feature matrix \(f_i \in f\), thereby, following the formulations outlined in sect. 3. furthermore, for spectral filtering, the laplacian matrices derived from the set \(\mathcal {a}\) are defined and represented by the set \(\mathcal {l}=\{l^u, l^u_{wd}, l^u_{we}, l^f\}\). accordingly, the hybrid block produces an output for gat convolution and another output for arma convolution, defined mathematically by eq. 1. $$\begin{aligned} \begin{aligned} \mathbf {h_{gat}}(a_i, f_i) & = \sigma (\alpha \mathbf {a_i} \mathbf {f_i} \textbf{w} + b)\\ \mathbf {h_{arma}^{(t+1)}}(l_i, f_i) & = \sigma (\boldsymbol{\tilde{l_i}}\mathbf {h_{arma}^{(t)}} \textbf{w} + \mathbf {f_i} \textbf{v})\\ \end{aligned} \end{aligned}$$ (1) regarding the first expression in eq. 1, which characterizes the spatial convolution, the \(\alpha \) denotes the attention weighting for the graph \(a_i\), given by \(\alpha _{uv}= \exp (e_{uv})/\sum _k{\exp (e_{uk})}\), where \(e_{ij} = leakyrelu(a^t[wh_{gat_i} || wh_{gat_j}]))\), and k defines the length of neighborhood of node \(v \in a_i\). moreover, \(f_i\) is the feature matrix, \(a_i\) is the adjacency matrix, and \(\sigma (\cdot )\) represents the activation function. in the second expression of eq. 1, t defines the number of iterations of convolution, which we assume equals 1 to prevent the over-smoothing problem [4]. the \(\tilde{l_i}\) is the normalized laplacian matrix, defined as \(\tilde{l_i} = d_i^{-1/2}a_id_i^{-1/2}\), where \(d_i^{-1/2}\) is the normalized diagonal degree matrix. note that the initial features are equal to the feature matrix, i.e., \(h_{arma}^{(0)} = f_i\). moreover, the feedforward block \(b_{mlp}\) takes only one input, the matrix \(t^f \in \mathbb {r}^{|p^u| \times 48}\) and its outputs are defined by eq. 2. $$\begin{aligned} \begin{aligned} \mathbf {h_{mlp}^{(1)}}(t^f) = \sigma (b_1 + \mathbf {w_2} ( relu(\mathbf {w_1} \mathbf {t^{f}} + b_2 ) ))\\ \mathbf {h_{mlp}^{(2)}}(t^f) = \sigma (b_4 + \mathbf {w_4} ( relu(\mathbf {w_3} \mathbf {t^{f}} + b_3 ) )) \end{aligned} \end{aligned}$$ (2) the final module consists of output aggregation of each block that is performed in two stages as illustrated in fig. 2. the first stage comprises the sum aggregation, graph convolutions concatenation, and the mlp aggregation. the second stage is represented by the self-attention mechanism and the final feedforward layer to output the label probabilities. initially, the outputs of the individual and collective components are represented by two distinct sets, as illustrated in fig. 2. the first set contains the arma block outputs, and an output of the feedforward block, represented by eq. 3. analogously, the second set includes the gat output, and the other output of the feedforward block, represented by eq. 4. $$\begin{aligned} \begin{aligned} s_1 & = \{ \mathbf {h_{arma}^{(t)}}(l^u, t^u), \mathbf {h_{arma}^{(t)}}(l^u, dur^u)\}\\ & \cup \{\mathbf {h_{arma}^{(t)}}(l^u, dis^u), \mathbf {h_{arma}^{(t)}}(l^u_{wd}, t^u_{wd})\}\\ & \cup \{\mathbf {h_{arma}^{(t)}}(l^u_{we}, t^u_{we}), \mathbf {h_{arma}^{(t)}}(l^f, t^f), \mathbf {h_{mlp}^{(1)}}(t^f)\} \end{aligned} \end{aligned}$$ (3) $$\begin{aligned} \begin{aligned} s_2 & = \{ \mathbf {h_{gat}}(a^u, t^u), \mathbf {h_{gat}}(a^u, dur^u)\}\\ & \cup \{\mathbf {h_{gat}}(a^u, dis^u), \mathbf {h_{gat}}(a^u_{wd}, t^u_{wd})\}\\ & \cup \{\mathbf {h_{gat}}(a^u_{we}, t^u_{we}), \mathbf {h_{gat}}(a^f, t^f), \mathbf {h_{mlp}^{(2)}}(t^f)\} \end{aligned} \end{aligned}$$ (4) firstly, to incorporate the information learned in the convolutional blocks, each set \(s_i\) is aggregated according to eq. 5, where \(b_i\), \(w_i\), \(w_j\) are learnable parameters, and \(y_{j,i}\) is an embedding from the set \(s_i\), such that \(y_{1, 1} = \mathbf {h^{(t)}_{arma}}(l^u, t^u)\), \(y_{2, 1} = \mathbf {h_{gat}}(a^u, t^u)\) and so forth. the goal is to add non-linearities in order to prevent overfitting. $$\begin{aligned} z_i = b_i + w_i \cdot \sum _{j=1}^{7} w_j \cdot y_{j,i}\quad i \in \{1,2\} \end{aligned}$$ (5) the next step of the first aggregation stage is the concatenation of set \(s_i\) and the sum aggregation \(z_i\), denoted by \(y'_i = s_i \vert z_i\), where \(\vert \) represents the concatenation operator. fig. 2. aggregation processing flow. each component output has a residual connection to the final aggregation to prevent overfitting. full size image each output aggregated set \(y'_i\) passes through the second aggregation phase, as represented in eq. 6. in this phase, the parameters are represented by \(b \in \mathbb {r}^{50}\) and \(w \in \mathbb {r}^{50 \times 50}\) that are learned during the training; the dimensionality of 50 was chosen empirically. this layer augments the features to a higher dimensionality in order to explore the correlations produced by the hybrid blocks. $$\begin{aligned} \mathbf {h_{agg}^{(m)}} = b_5 + \mathbf {w_5} y'_{m},\; m \in \{1, 2\} \end{aligned}$$ (6) after augmenting the dimensionality of the hybrid output features with a feedforward layer, we have \(h_{agg}^{(1)}\) and \(h_{agg}^{(2)}\) being the representation of the sets \(y'_1\) and \(y'_2\), respectively. subsequently, the attention mechanism [22] is introduced to importance weighting of joint individual and collective features denoted by \(x' = h_{agg}^{(1)} \vert h_{agg}^{(2)}\). therefore, the key, query, and value matrices are defined as \(k = x' w_{x'}^{(1)}\), \(q = x' w_{x'}^{(2)}\) e \(v = x' w_{x'}^{(3)}\), obtaining the output \(\mathbf {h_{att}} = softmax(\dfrac{qk^t}{\sqrt{d}}v)\). finally, for the latter aggregation step, represented by the last mlp block in fig. 2, the input consists of a skip connection [10] to prevent overfitting, given by the concatenation \(x'' = y'_1 \vert y'_2 \vert \mathbf {h_{att}}\). hence, to correlate each feature extracted through the havana learning process, the input vector \(x''\) is the input to the feedforward layer \(\mathbf {h_{out}} = relu(b_6 + \mathbf {w_6} x'')\), which represents the final output of the model. 4.1 loss function the cross-entropy loss function was used to measure the error between the observed and predicted values, which relates the vector \(h_{out}\) and the point of interest labels contained in the dataset. therefore, a softmax is applied for the output feature set as follows: \(\hat{y} = \frac{\exp (\textbf{h}_{\textbf{out}_{i}})}{\sum _i^d \textbf{h}_{\textbf{out}_{i}}}\), where d is the dimensionality of output features. hence, the loss function is represented by \(l(y, \hat{y}) = -\sum _i p(x_i) \log (\hat{y})\), where \(p(x_i)\) is the target probability distribution. note that the distribution is a one-hot encoding, given k categories. 5 experiments and results in this section, the method proposed, havana, is compared to three state-of-the-art models: k-fn [18], stpa – a spatial component of [30] –, and pgc-nn [3]. these baselines were selected because of their methods’ reproducibility and importance since k-fn is based on machine learning while stpa and pgc-nn stand for graph neural networks. in addition to these methods, we conducted experiments with the non-negative matrix factorization method hmrm [5], which represents the low-rank decomposition matrix theory; however, it runs out of memory with the two bigger datasets. hmrm relies on different matrix correlating places and check-ins leading to high memory usage since it follows the \(o(|c|\times |p|)\), such that c is the check-ins set and p is the set of points of interest. the authors of hmrm do not conduct experiments with this volume of data either. therefore, we do not include the results for hmrm. furthermore, this section addresses the ablation study, aiming at exploring thoroughly each module of the method proposed, and analyzing the performance when removing some of them. thus, this study focuses on explaining the contribution to the learning of each havana module. 5.1 data and test environment considering various location-based social data publicly available, the gowalla dataset [6] was chosen to compare the havana with state-of-the-art methods. this dataset is widely recognized and has been used in different experiments. we selected three states of the united states that have the most number of check-ins: florida, california, and texas, with 859, 984, 2, 249, 070, and 3, 061, 324 records, respectively. table 1 shows the statistics of the three datasets. table 1. total of check-ins, points of interests, and users of each state utilized for experiments.full size table the experiments were conducted on google colab, with the following specification: intel (r) xeon (r) (8 cores, 56m cache and 2.2ghz) with 51gb ram. the havana was developed using the spektral libraryfootnote 2 and tensorflow 2.14. 5.2 results figures 3, 4, and 5 present the f1-score, weighted average f1-score and accuracy for the datasets of florida, california, and texas, respectively. in general, the results reveal that havana outperforms all metrics presented, highlighting the improvement in the average f1-score. specifically, it is notable the performance in florida that presents a 25.05% improvement on average f1-score and 19.90% on weight f1-score. fig. 3. f1-score and accuracy for florida full size image fig. 4. f1-score and accuracy for california full size image fig. 5. f1-score and accuracy for texas full size image table 2. average f1-score (%) by model, category, and state.full size table the results for each category shown by table 2 demonstrate that havana surpasses the other methods in almost all scenarios, suggesting that the mechanisms adopted, i.e., attention weights and skip-connections, yielded notable performance on the semantic venue annotation task. to interpret the results presented by table 2, it is necessary to note that the points of interest are not balanced, and each state has its own distribution. for example, in florida there are more check-ins of food and shopping, being, respectively, 247,082 and 201,433 check-ins. similarly to florida, food and shopping are also the most numerous, with 767,056 and 481,600 check-ins for california, while in texas, these categories correspond to 999,982 and 700,992 check-ins. the results presented to different states suggest that the method proposed exhibits strong generalizability to different mobility patterns and city organizations, as evidenced by improvements in the analyzed metrics. the consistent performance indicates that havana possesses high adaptability to diverse spatio-temporal city characteristics, including the imbalanced category place distribution, as well as the variability of check-ins and users quantities. although the categories present an imbalanced distribution, it is improbable that this difference influenced the results. for example, the travel category has fewer check-ins than the other classes, being 102,294 in florida, 215,088 in california, and 182,541 in texas; nonetheless, this category exhibits a great f1-score. this disparate can be explained due to the activity type that mobile users take in a place. for example, a shopping location can be similar to another category, such as food. this behavior is a known problem for semantic venue annotation task [24], e.g., to a user a, a location is a workplace, while to a user b, the same venue is a leisure place. it is still noteworthy to highlight that stpa, pgc-nn, and k-fn do not present satisfactory results, since they exhibit a high variance on f1-score within different states. considering that, this lack of consistency in results suggests a higher sensibility to distinct states and spatio-temporal characteristics. on the other hand, as the proposed model comprises a hybrid convolution and weighting attention mechanism, it demonstrates a higher consistency in different contexts. then, these performance gains compared to the state-of-the-art model reinforce the efficiency of the novel method. 5.3 ablation study the hybrid convolution and aggregation block modules are the principal components of havana that improve the classification accuracy jointly. to examine the contribution of each component in correctly inferring the label, experiments were conducted on an individual unit. thus, the test named as m-arma is the model with gat blocks removed, eliminating the hybrid learning. in this case, to carry out the aggregation, \(y'_1\) behaves as \(x'\). in the same way, for the test m-gat, \(y'_2\) behaves as \(x'\). finally, the test m-no-agg represents the model without the aggregation, aplying the softmax function on concatenation \(y'_3 = \{y_{1,1}, y_{2,1}, \ldots , y_{7,1}, y_{1,2}, y_{2,2}, \ldots , y_{7,2}\}\). table 3 exhibits the experiment’s results, highlighting that all the tests decreased its performance on average f1-score compared to the complete model havana. in scenarios such as florida, the drop of f1-score was about \(28.78\%\) when comparing the havana and the m-no-agg. in this context, it is noteworthy the reduction in all average f1-scores, accompanied by an increase in standard deviations. therefore, these results indicate a higher variability removing the components of havana. additionally, the components demonstrated an important individual relevance to consistency and performance in the semantic venue annotation task, thus validating the hypothesis of hybrid learning proposed by this study. table 3. average f1-score (%) by model and statefull size table 6 ethical statement the increasing availability of location-based social network data provides sources to analyze human mobility under an individual perspective [27]. however, it introduces an ethical policy statement due to location privacy issues [13]. that is, given the location of each check-in it is possible to infer various personal attributes such as home and workplace, work hours, the next location visited, and other sensible information [15]. since our study aims to enhance a method to classify a category place, we used gowalla anonymized user identifier information, but we maintained the location without masking the latitude and longitude of a collected gps point. therefore, when considering using havana in a pipeline with real-world company data, it is fundamental to anonymize the locations with an appropriate method, such as mix-zones [17] or cats [21]. 7 conclusion and future works in this work, a novel method named havana is formulated with hybrid learning that includes spectral and spatial convolutions to annotate points of interest semantically. the results revealed a higher performance compared to three state-of-the-art models: pgc-nn, stpa, and k-fn. hence, hybrid models that combine different types of convolution with aggregation blocks can represent a great advance in semantic venue annotation. additionally, the application of this method in other related tasks of points of interest can also increase the performance, due to the similarity of convolution layers in tasks such as next place to visit and points of interest recommendation. in the context of future work, it is necessary to enhance the feature engineering, exploring new matrices to represent the data more precisely, specially for collective features such as regional features characterizing the form of the city. this enhancement intends to potentialize the model capacity to consider the aspects of different cities. notes 1.the code is available on https://github.com/gegen07/havana. 2.https://github.com/danielegrattarola/spektral. references atluri, g., karpatne, a., kumar, v.: spatio-temporal data mining: a survey of problems and methods. acm comput. surv. (csur) 4, 1–41 (2018) math  google scholar  bianchi, f.m., grattarola, d., livi, l., alippi, c.: graph neural networks with convolutional arma filters. ieee trans. pattern anal. mach. intell. 7, 3496–3507 (2022) google scholar  capanema, c.g., de oliveira, g.s., silva, f.a., silva, t.r., loureiro, a.a.: combining recurrent and graph neural networks to predict the next place’s category. ad hoc netw. 138103016 (2023) google scholar  chen, d., lin, y., li, w., li, p., zhou, j., sun, x.: measuring and relieving the over-smoothing problem for graph neural networks from the topological view. in: proceedings of the aaai conference on artificial intelligence,vol. 34, no. 04, pp. 3438–3445 (2020) google scholar  chen, m., zhao, y., liu, y., yu, x., zheng, k.: modeling spatial trajectories with attribute representation learning. ieee trans. knowl. data eng. 4, 1902–1914 (2020) math  google scholar  cho, e., myers, s.a., leskovec, j.: friendship and mobility: user movement in location-based social networks. in: proceedings of the 17th acm sigkdd international conference on knowledge discovery and data mining, pp. 1082–1090, kdd ’11, association for computing machinery (2011) google scholar  defferrard, m., bresson, x., vandergheynst, p.: convolutional neural networks on graphs with fast localized spectral filtering. adv. neural inf. process. syst. (2016) google scholar  dubey, m., srijith, p., desarkar, m.s.: multi-view hypergraph convolution network for semantic annotation in lbsns. in: proceedings of the 2021 ieee/acm international conference on advances in social networks analysis and mining, pp. 219–227 (2021) google scholar  gilmer, j., schoenholz, s.s., riley, p.f., vinyals, o., dahl, g.e.: neural message passing for quantum chemistry. in: international conference on machine learning, pp. 1263–1272, pmlr (2017) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  huang, t., pan, x., cai, x., zhang, y., yuan, x.: learning time slot preferences via mobility tree for next poi recommendation. in: proceedings of the aaai conference on artificial intelligence, vol. 38, no. 8, pp. 8535–8543 (2024) google scholar  huang, w., cui, l., chen, m., zhang, d., yao, y.: estimating urban functional distributions with semantics preserved poi embedding. int. j. geogr. inf. sci. 10, 1905–1930 (2022) article  math  google scholar  keßler, c., mckenzie, g.: a geoprivacy manifesto. trans. gis 1, 3–19 (2018) article  math  google scholar  kipf, t.n., welling, m.: semi-supervised classification with graph convolutional networks. arxiv preprint arxiv:1609.02907 (2016) luca, m., barlacchi, g., lepri, b., pappalardo, l.: a survey on deep learning for human mobility. acm comput. surv. (csur) 1, 1–44 (2021) math  google scholar  ma, j., li, b., mostafavi, a.: characterizing urban lifestyle signatures using motif properties in network of places. environ. planning b: urban analytics city sci. 4, 889–903 (2024) math  google scholar  de mattos, e.p., domingues, a.c., silva, f.a., ramos, h.s., loureiro, a.a.: slicing who slices: anonymization quality evaluation on deployment, privacy, and utility in mix-zones. comput. netw. 236 110007 (2023) google scholar  de mattos senefonte, h.c., silva, t.h., lüders, r., delgado, m.r.: classifying venue categories of unlabeled check-ins using mobility patterns. in: 2019 15th international conference on distributed computing in sensor systems (dcoss), pp. 562–569, ieee (2019) google scholar  milias, v., psyllidis, a.: assessing the influence of point-of-interest features on the classification of place categories. comput. environ. urban syst. 86, 101597 (2021) google scholar  psyllidis, a., et al.: points of interest (poi): a commentary on the state of the art, challenges, and prospects for the future. comput. urban sci. 2(1), 20 (2022) google scholar  rao, j., gao, s., zhu, s.: cats: conditional adversarial trajectory synthesis for privacy-preserving trajectory data publication using deep learning approaches. int. j. geogr. inf. sci. 12, 2538–2574 (2023) article  math  google scholar  vaswani, a., et al.: attention is all you need. adv. neural inf. process. syst. (2017) google scholar  veličković, p., cucurull, g., casanova, a., romero, a., liò, p., bengio, y.: graph attention networks (2018) google scholar  wang, h., li, z.: region representation learning via mobility flow. in: proceedings of the 2017 acm on conference on information and knowledge management, pp. 237–246 (2017) google scholar  xu, h., xu, r., chen, m., liu, y., yu, x.: cave-sc: inferring categories for venues using check-ins. inf. sci. 611, 159–172 (2022) google scholar  yan, x., hu, s., mao, y., ye, y., yu, h.: deep multi-view learning methods: a review. neurocomputing 448 , 106–129 (2021) google scholar  yang, y., pentland, a., moro, e.: identifying latent activity behaviors and lifestyles using mobility data to describe urban dynamics. epj data sci. 12(1), 15 (2023) google scholar  yao, d., et al.: deep hybrid: multi-graph neural network collaboration for hyperspectral image classification. defence technol. 23, 164–176 (2023). issn 2214-9147 google scholar  ye, m., shou, d., lee, w.c., yin, p., janowicz, k.: on the semantic annotation of places in location-based social networks. in: proceedings of the 17th acm sigkdd international conference on knowledge discovery and data mining, pp. 520–528 (2011) google scholar  zhang, d., xu, r., huang, w., zhao, k., chen, m.: towards an integrated view of semantic annotation for pois with spatial and textual information. pp. 2441–2449 (2023) google scholar  zhou, j., cui, g., hu, s., zhang, z., yang, c., liu, z., wang, l., li, c., sun, m.: graph neural networks: a review of methods and applications. ai open 1 57–81 (2020) google scholar  download references aknowledgements the authors are thankful for the support of manna team, softex, cnpq (number 421548/2022-3) and fapemig. author information authors and affiliations universidade federal de viçosa, florestal, brazil germano b. dos santos, paulo h. c. silva, fabrício a. silva & thais r. m. braga silva universidade estadual de maringá, maringá, paraná, brazil linnyer b. r. aylon manna team, maringá, brazil germano b. dos santos, fabrício a. silva, thais r. m. braga silva & linnyer b. r. aylon authors germano b. dos santosview author publications search author on:pubmed google scholar paulo h. c. silvaview author publications search author on:pubmed google scholar fabrício a. silvaview author publications search author on:pubmed google scholar thais r. m. braga silvaview author publications search author on:pubmed google scholar linnyer b. r. aylonview author publications search author on:pubmed google scholar corresponding author correspondence to germano b. dos santos . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper dos santos, g.b., silva, p.h.c., silva, f.a., silva, t.r.m.b., aylon, l.b.r. (2025). havana: hybrid attentional graph convolutional network semantic venue annotation model. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_20 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords venue semantic annotation point-of-interest labeling neural network graph representation learning publish with us policies and ethics profiles germano b. dos santos view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature multiple object tracking in native bee hives: a case study with jataí in the field | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper multiple object tracking in native bee hives: a case study with jataí in the field conference paper first online: 12 october 2023 pp 176–191 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) multiple object tracking in native bee hives: a case study with jataí in the field download book pdf download book epub rodolfo r. v. leocádio  orcid: orcid.org/0000-0002-0567-30319, alan kardek rêgo segundo  orcid: orcid.org/0000-0002-9272-06679,10 & gustavo pessin  orcid: orcid.org/0000-0002-7411-92299,10  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 633 accesses 2 citations abstract artificial intelligence approaches, such as computer vision, can help better understand the behavior of bees and management. however, the accurate detection and tracking of bee species in the field remain challenging for traditional methods. in this study, we compared yolov7 and yolov8, two state-of-the-art object detection models, aiming to detect and classify jataí brazilian native bees using a custom dataset. also, we integrated two tracking algorithms (tracking based on euclidean distance and bytetrack) with yolov8, yielding a mean average precision (map50) of 0.969 and map50–95 of 0.682. additionally, we introduced an optical flow algorithm to monitor beehive entries and exits. we evaluated our approach by comparing it to human performance benchmarks for the same task with and without the aid of technology. our findings highlight occlusions and outliers (anomalies) as the primary sources of errors in the system. we must consider a coupling of both systems in practical applications because bytetrack counts bees with an average relative error of 11%, euclidiantrack monitors incoming bees with 9% (21% if there are outliers), both monitor bees that leave, bytetrack with 18% if there are outliers, and euclidiantrack with 33% otherwise. in this way, it is possible to reduce errors of human origin. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others dynamic beehive detection and tracking system based on yolo v5 and unmanned aerial vehicle article 14 november 2022 performance analysis of deep learning models in detection and counting of bees for hive monitoring chapter © 2025 a real-time edge computing system for monitoring bees at flowers chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms computer vision lineage tracking motion detection object recognition robotics 1 introduction insects are fundamental in ecosystems as pollinators, herbivores, detritivores, nutrient cyclers, and food sources for other species [1]. native bees are pollinators related to various plant foods consumed [2]. bees are fundamental in world agriculture to improve crop quality and yields [3]. furthermore, they actively contribute to the recovery of degraded ecosystems [4], can be used in biovectorization technology that uses insects as biocontrol agents [5], and can also play a crucial role in sustainable development projects and programs [4]. studies report biodiversity losses among native bees in tropical regions [6, 7]. the main factors responsible for your decline are infections disseminated by parasites and pathogens, lack of genetic variability, stress due to the seasonal movement of hives to pollinate fruits and vegetables, toxic pesticide residues found in pollen, nectar, and hives (mite control), the low nutritional value of agro-landscapes dominated by monocultures (such as corn, soybeans, and cotton), and the most adverse weather conditions in recent decades [1, 6, 8,9,10]. the decline of insects results in adverse effects on ecosystems. thus, preserving its abundance and diversity is fundamental for ecology [1]. individuals with little entomological knowledge are incapable differentiates categories of insects, their stages of maturity, and their behavior. therefore, it is necessary to develop faster and more effective approaches that solve these problems [11]. the issues addressed in the insect recognition and classification process are to quickly detect the animal of interest positioned on a complex background, accurately distinguish insect species with high similarity, and effectively identify patterns of interest in the behavior of the classes. artificial intelligence (ai) tools can have a mutualist relationship with methods applied in ecology. the hives monitoring can help combat the decline of bees related to foraging, pollen characteristics, hive behavior, and invasive species. attempting to help monitor hives, we used an object detector to detect and classify tetragonisca angustula bees. we incorporated a simple tracking system based on the object detector to describe the movement of insects, generating information about count and their direction concerning the hive entrance. we also tested a more robust tracking system, in an attempt to treat occlusions, compared to the previous one. to next section contains recent work on artificial intelligence assisting insect management and conservation. section 3 presents the methods of detection, classification, tracking, and optical flow of the species. section 4 contains the results obtained by the steps highlighted in anterior with your corresponding discussions. finally, we present the conclusions and future perspectives in sect. 5. 2 related works some works developed in the last six years stand out in an attempt to automate processes applied to biological sciences involving the class insecta of the phylum arthropoda (kingdom animalia). ai techniques used to monitor insects from images can provide an alternative to reduce manual activities and human errors [12]. in this sense, there is a tendency to use computer vision approaches for insect detection and recognition. object detection techniques such as faster r-cnn [13], vgg19 [11], yolo + svm [14], and resnet (map around 90%) [15] proved adequate. such approaches became possible to identify, classify and count insects in plants, leaves, photographs, traps, and grains [13, 16]. the insects include wasps and moths [11]. liu and his team [17] developed a system called pestnet for field pest detection. monitoring the number of pest species prevents the indiscriminate use of pesticides that result in crops that are harmful to health. the acquisition equipment is a multispectral light trap with an hd camera to capture pest specimens in a tray. its best performance achieved an map of 75.46%. abiotic and biotic factors such as social interactions are used to understand social insects. individuals in a colony engage in various tasks, such as foraging and hive care, depending on social contexts and interactions [18]. challenges arise when monitoring these colonies because individuals are small and numerous [19]. tracking systems have been used for this function, as insect movements can provide information about their cognition and decision-making [20]. most tracking uses classical machine learning techniques [21]; however, computer vision can automate tracking, featuring velocity, straightness, direction, and exploration of trajectories [22]. markers make the tracking system based on images more robust, making it possible to measure locomotor activities [18] of members of an ant colony in the laboratory to study their circadian activity [19]. some authors, such as [23], try to integrate real-time data analytics and monitor them under natural conditions, such as in a tropical forest [22] without markers. bioluminescence and behavior studies of insects such as fireflies [24] and pest counting in automatic traps with live insects also use tracking [25]. concomitantly emerged were works involving the apidae family of the hymenoptera order. bee foraging studies using rfid tags [2] merged with temperature, barometric pressure, and solar irradiance data define a recurrent neural networks (rnn) architecture [8] that performs best in predicting bee behavior. there are also systems with deep learning for automated monitoring of beehives (deepbees) [10]. in this case, the footage of the bees takes place at the entrance to the hive. deepbees monitors health and hygiene, attacks and invasions (wasps), pollen, bees dead, and drones. however, there are still challenges in the detection, direct mapping of activities, and with diseases or mite infestation; caused by lack of data. tracking for pollination studies targets bees. monitoring natural habitats are essential for behavioral studies, preferably with non-invasive techniques. hybrid detection and tracking (hydat) map interactions between bees and wildflowers; knn segments the image background and foreground, and yolo locates bees [20, 26]. perez-cham [27], sun [28], and their teams use 3d tracking to analyze and reconstruct the bees’ flight paths. this work presents the following contributions: it uses non-invasive techniques, without sensors in the hive and markers in the insects; uses a one-stage object detector to classify and detect jataí; it only uses tracking based on detections; compares the performance of two tracking algorithms to perform the task; contains specific implementation for the optical flow of bees and; compares the performance of computer vision with humans. in this way, we believe it is possible to contribute to the ecological preservation of the species, combating the decline of insects and facilitating their management. 3 methods 3.1 custom dataset – tetragonisca angustula (jataí) the jataí (fig. 1) is a bee native to brazil, and its geographic distribution occurs throughout the brazilian territory. considering its natural environment, this species has the habit of nesting in tree hollows and rock cavities. its adaptability promotes nesting in hollow walls, light boxes, letter boxes, stone walls, and other unusual places, considering an urbanized environment. the colony has a queen mother and an average of 5000 (from 2000 to 8000) workers. guards are a caste with larger sizes than other workers. during the day, the workers remain positioned over the inlet tube or hover close to it. on average measures 4 mm and can forage 600 m from the hive. fig. 1. tetragonisca angustula. adapted from a.b.e.l.h.a. full size image it is a significant pollinator of crops such as acapu, avocado, coffee, carrot, cupuaçu, guava, orange, mango, watermelon, strawberry, moressuma, pepper, cucumber, tangerine, umbu, and annatto. therefore, proper management of hives ensures the productivity of such crops and protects the colony from natural enemies. the custom dataset aims to monitor automatically of native bee hives using non-invasive techniques. it consists of videos of the entry of several colonies for tracking identified as xxxy-y, where xxx identifies the hive location and y-y the recording interval. we identified images taken from the videos for object detectors as the complement of the previous xxxy-yz…, where z… is an id associated with each image. each video and image contains an information file with the metadata. knowledge about the jataís showed that monitoring on cold, rainy days and at night is unnecessary. also, we avoid windy days. in this way, the data represent reality and are valid only for jataís. we used the sony hdr-as20 camera, with a resolution of 1920 × 1080p and a recording speed of 60 fps. in an attempt to represent the natural environment of the hives, the recordings took place at different times of movement at the hive’s entrance. we withdraw videos of a beehive from the beekeep cs database, with a resolution of 960 × 540p and a recording speed of 30 fps. we acquired videos of natural and artificial hives in urban and rural environments relatively close, but climatic conditions are different. we positioned the video camera, trying to optimize the capture of movements around each hive. therefore, there is only variation in the capture angle for different beehives. the dynamic in the hive’s entrance guarantees the variability of the poses of the bees. the custom dataset is structured as required by the family of object detectors known as yolo (test, valid, and train). these directories have a separation between images, labels, and information. the info folder contains the metadata. incomplete files are due to data not provided by beekeep cs (16.7%). the custom dataset has 26487 instances in 2100 images, with 19000 in 1500 images for training, 3719 in 300 for validation, and 3768 in 300 for testing. climate variations, preferably days with the sun between clouds (83.2% of the images), periods of the days (16.7% obtained in the morning and 66.7% in the afternoon), and seasons (16.7% obtained in winter and 66.7% in spring), during the recordings provided differences in lighting. we noticed differences in lighting in 45.6% of the images. the correlogram between labels contains a bimodal distribution that indicates a mixture of two trends with different labels, with a predominance of the right trend with the highest number of observations; a chi-square distribution inferring that the labels are homogeneous; and two poisson distributions indicating that the label sizes are random but repeat at a defined average rate. this way, the labels have adequate precision and cover all the bees in the images. the custom dataset has no duplicates and leaks, considering the ssim [29] of 0.97. as the bees occupy a small part of the image, for being small, the differences between images are subtle. fig. 2. custom dataset samples. 001 (−20.381099, −43.506333)* natural hive in an urban environment. 002 (−20.502591, −43.520515)*, 003 (−20.502507, −43.520330)* and 005 (−20.345640, −43.608437)* natural hive in a rural environment. 004 (−20.345266, −43.608529)* artificial hive in a rural environment. 006 beekeep cs artificial beehive. *geographical coordinates obtained from google. full size image there is a background for each beehive present (fig. 2), totaling 0.29% of the images. when we do not obtain images without bees, use data augmentation to remove them (0.24% of them). we fill in the resulting space of an unlit bee with the texture of its nearest neighbors. 1737 images have occlusions, that is, 82.7% of them. we observed occlusions due to: the overlapping of bees, by background item, per entrance pipe, by light beams, bees that touch each other, and bees cut by frames. the perceived anomalies are shadows and distortions during bee flight. the incidence of sunlight causes shadows that can be confused with objects by computer vision models. bee shadows are monitored but have only been ignored so far (361 images have bee shadows, i.e., 17.2%). distortions during bees’ flight occur due to a higher flight speed than the capture capacity of the camera. not possible to infer over class distribution because the custom dataset only has one class so far. we did not observe intraclass variations, but they need monitoring during updates. 3.2 detection and classification of the native bee yolo [30] (you only look once) is an object detection model used in computer vision. the model treats object detection as a regression problem using a single convolutional neural network (cnn). it does this by splitting the image into a grid, making multiple predictions for each grid cell, filtering these predictions, and removing overlapping boxes to produce its final output. its evolution includes yolov2 [31], yolov3 [32], yolov4 [33], yolov5 [34], and yolov6 [35]; each incorporates contributions to the model. the resnext backbone and the dual head of yolov7 [36], combined with the label assigner engine that assigns flexible labels, allow the model to learn from data more effectively. it presents a new multiscale training strategy and a technique called focal loss, designed to solve the class imbalance problem. the latest release, yolov8 [37], surpasses previous by incorporating advances such as a new backbone, a head split without anchor, and new loss functions. 3.3 tracking and optical flow of jataís tracking is obtaining an initial set of objects, creating a unique id for each one, and accompanying each object as they move in frames, maintaining the assignment of id’s. an ideal tracking algorithm should detect objects only once, be fast, control when the tracked object disappears or moves out of frame, be robust with occlusions, and recover lost objects between frames. this work uses a detector-based tracking similar to [23]; it depends on the euclidean distance between existing and new centroids. bounding boxes are the basis for calculating centroids with id’s assigned to the initial set of centroids. after that, it computes the euclidean distance between the existing and new centroids, associating new centroids with the old ones. then, minimize distances and update your coordinates. finally, there is the registration of new centroids and the cancellation of the registration of objects that leave the video. this a simple and fast method that controls when the tracked object disappears or leaves the frame and recovers lost objects between frames. we attempt to improve the tracking with the algorithm bytetrack [38] (bt) to it does the same as the previous one, solve occlusions, and detect objects only once. the algorithm performs associations between the detections of the current frame concerning the previous frame. for that, bt uses similarity calculations consider the distances between the resources of appearance (re-id) and iou resources. then the hungarian algorithm finalizes the match based on similarity. optical flow is used in computer vision to characterize and quantify the movement of objects in a video stream, often for motion-based object detection and tracking systems. when estimating optical flow between frames is possible to measure an objects speed in the video. it is also possible to follow the movement of jataís, count the bees that leave and enter the hive, and your total. this approach can generate information about hive nutrition. for this, we propose using the centroids of the detections containing the same id and a displacement limit from the current frame to the previous frame. we define a circle containing the entrance of the hive as a limit. a jataí has entered the beehive if the euclidean distance between the circle center and detection centroid is greater than your radius in the previous frame and smaller than the current frame. otherwise, the jataí leaves the hive. 4 results and discussion 4.1 detection and classification of jataí results from yolov8 [37] trained on the custom dataset are in fig. 3. in just 300 epochs of training, the yolov8x model converged to a precision of 0.97, recall of 0.957, map50 of 0.984, and map50–95 of 0.744 (fig. 3a). considering the test data, the confusion matrix indicates that: every time the model predicts a class and is background, the reason is jataí; 98% of jataí predictions are jataí and; 2% of background predictions are jataí. the precision (0.935) – recall (0.937) curve for map50 of 0.969 and map50–95 of 0.682 (fig. 3b) shows that most of the classifications that the model made are correct, and most of the predicted bounding boxes are as expected. the yolov7 [36] model converged to a precision of 0.952, recall of 0.948, map50 of 0.979, and map50–95 of 0.634 (fig. 3c). considering the test data, the confusion matrix is identical to the previous one. the precision (0.922) – recall (0.901) curve for map50 of 0.955 and map50–95 of 0.579 (fig. 3d) also shows the same as the previous one. we noticed that both present acceptable results for the application, but the yolov8 metrics reach better scores than the yolov7 ones. we believe this is due to differences in their backbones. in this way, we use yolov8 for the tracking implementations. fig. 3. training results and precision-recall curve with yolov8 (a, b) and yolov7 (c, d), respectively. full size image 4.2 tracking and optical flow of detected bees in the tracking algorithm called eutrack (et), which depends on euclidean distance, a parameter can be changed to control when the tracked object disappears or leaves the frame and its ability to recover lost objects between frames. we set the parameter as 20 for all experiments, i.e., an object is only considered for a new id if it is not detected for 20 consecutive frames. the bt [38] is a robust system in treating occlusions because it performs similarity analysis, adopts the kalman filter to predict the object’s movement, and uses additional re-id models to improve the long-range association. we set its parameters for the whole experiment as track_thresh equal to 0.05, track_buffer equal to 20, match_thresh equal to 1.0, aspect_ratio_thresh equal to 3.0, min_box_area equal to 1.0 and mot20 equal to true. in fig. 4, one can view the bounding box, detected class, and probability. the tracking algorithms generate the id and mark the centroid of each bounding box. the optical flow, implemented from the centroids and id’s, informs about movements in and out of the hive (delimited by the red circle). the bee trails, with a displacement limit equal to 120, were hidden to facilitate the visualization of the other considerations. also, observe that jataí with id 7 (fig. 4 001) entered and the one with id 14 (fig. 4 003) left their respective hives. timestamps can collaborate with this information to generate possible monitoring of the hive entrance. results for a video of each hive are in table 1, along with their absolute error and mean relative error. we use a process similar to pseudo-labeling (pseudo) to count bees in the videos (we call it a procedure with the aid of technology). we used et to aid in counting. note in fig. 4, specifically at 003 (id 2 and 5), 005 (id 4), and 006 (id 0 and 9), the presence of soldiers hovering around the hives entrances. these soldiers generate errors in counting the bees that enter and leave the hive. therefore, we remove duplicates from these counts (rd) to improve this aspect. summarizing table 1, we have that et has superior results in the bee count, but we can also use bt. it is best to remove duplicates of bees to monitor the hive entrance. the error is less when bt counts the bees that get out of the hive and when et counts those that enter. in this way, there is a tie considering the best cost benefit. we repeated the same procedure for six videos of hive 001 (table 2), plus a count performed by a person from the biological sciences area without using technology. fig. 4. results of the tracking and optical flow. detailed information in fig. 2. full size image summarizing table 2, we have that bt has superior results in bee count. it is best to remove duplicates of bees to monitor the hive entrance. et performs better in accounting for incoming and outgoing bees. in this way, et obtains a better cost-benefit concerning the movement at the hive entrance and bt to count bees. we also noticed that using the system to assist human activity reduces errors caused by fatigue and distractions. table 1. results of the trackings and humans with the aid of technology.full size table the errors in tables 1 and 2 are caused by some limitations. et does not resolve id’s swaps caused by crossing their positions and occlusions caused by soldiers and the hive tube. the movement in the hive tube is responsible for most of the exchanges of id’s, and the model presents difficulties for bees that touch each other (example in fig. 4 005 id 5 and 6). table 2. results of the trackings and humans with and without the aid of technology.full size table in agitated hives, agglomerations of bees occur. this fact splits a bounding box containing one bee into others and unifies bounding boxes into one containing several bees. this process contributes to the acquisition of new ids, along with occlusions. the transition of jataís between frames and occlusions causes the resumption of id by different bees. the camera is not steady in 006, and the systems recognize some anomalies in 002 and 004. we monitored anomalies, but it was not possible to extinguish them in the field. the speed of the videos, frames per second, cannot be too low because the centroids must be close between the frames. bt relocates id’s in the agglomerations with difficulties, causing the assignment of new id’s and generating duplicates. the same error of dividing and merging bounding boxes also occurs in bt, contributing to new id’s assignment. there are difficulties in treating occlusions as well. the authors of [38] show that bt is robust for occlusions where one of the objects does not move during occlusion. this type of occlusion is rare with jataís, as the occluded bee also moves. when the occlusion ends, the occluded object may be in a different position in rotation or translation. the movement causes the bee to have different characteristics from when it was occluded. the similarity between individuals in the hive also impaired the similarity analysis used in bt. there are no physical differences between workers and castes in the same or different beehives. the bees themselves only differentiate through pheromones. however, using the system in the field can help to combat the decline of bees related to foraging and hive behavior. generating reports and doing repetitive work brings advantages to the approach, thus reducing errors of human origin. but the use in practical applications must consider the bt to count the bees with an average relative error of 11%. et to monitor bees that enter with an average relative error of 9% (21% if there are outliers). both to observe leaving bees, bt with an average relative error of 18% if there are outliers and et with an average relative error of 33% otherwise. 5 conclusion we believe it is a proposal capable of detecting, classifying, and generating tracking information for jataí in the field. in this way, it is possible to help combat the decline of bees related to foraging and hive behavior. we must consider a mixture of both systems in practical applications. bt counts bees with an average relative error of 11%. et monitors incoming bees with an average relative error of 9% (21% if there are outliers). both monitor bees that leave, bt with an average relative error of 18% if there are outliers, and et with an average relative error of 33% otherwise. the biggest challenge in this type of implementation is to obtain conditions and data of the species of interest in the field to compose the custom dataset. harsh conditions common in natural environments provide difficulty in hardware use and maintenance. the context also introduced issues mainly caused by occlusions and outliers that we must overcome to reduce the relative error. in future work, we intend to optimize the custom dataset by adding greater diversification to the data considering invasive species, acquiring videos of different hives, adding data for tracking to the custom dataset presented, and completing the metadata of the videos with pseudo-labeling for counts and the same performed by biologists. to implement the mot20 format [39] in the tracking custom dataset introducing evaluation metrics. to compare results with other tracking algorithms in an attempt to reduce the limitations of this work. finally, we will implement pollen identification in monitoring to generate information about hive nutrition. (additional information at https://github.com/rodolfoloc/native-bees). references hallmann, c.a., et al.: more than 75 percent decline over 27 years in total flying insect biomass in protected areas. plos one 12, 18–22 (2017). https://doi.org/10.1371/journal.pone.0185809 article  google scholar  arruda, h., imperatriz-fonseca, v., de souza, p., pessin, g.: identifying bee species by means of the foraging pattern using machine learning. in: 2018 international joint conference on neural networks (ijcnn), pp. 1–6. ieee (2019). https://doi.org/10.1109/ijcnn.2018.8489608 kuan, a.c., et al.: sensitivity analyses for simulating pesticide impacts on honey bee colonies. ecol. model. 376, 15–27 (2018). https://doi.org/10.1016/j.ecolmodel.2018.02.010 article  google scholar  giannini, t.c., et al.: climate change in the eastern amazon: crop-pollinator and occurrence-restricted bees are potentially more affected. reg. environ. change 20(1), 1–12 (2020). https://doi.org/10.1007/s10113-020-01611-y article  google scholar  macharia, j.m., gikungu, m.w., karanja, r., okoth, s.: managed bees as pollinators and vectors of bio control agent against grey mold disease in strawberry plantations. afr. j. agric. 16(12), 1674–1680 (2020). https://doi.org/10.5897/ajar2020.15203 article  google scholar  sánchez-bayo, f., wyckhuys, k.a.g.: worldwide decline of the entomofauna: a review of its drivers. biol. cons. 232, 8–27 (2019). https://doi.org/10.1016/j.biocon.2019.01.020 article  google scholar  borges, r.c., padovani, k., imperatriz-fonseca, v.l., giannini, t.c.: a dataset of multi-functional ecological traits of brazilian bees. sci. data 7(1), 1–9 (2020). https://doi.org/10.1038/s41597-020-0461-3 article  google scholar  gomes, p.a.b., et al.: an amazon stingless bee foraging activity predicted using recurrent artificial neural networks and attribute selection. nat. res. 10(1), 1–12 (2020). https://doi.org/10.1038/s41598-019-56352-8 article  google scholar  filipiak, m.: a better understanding of bee nutritional ecology is needed to optimize conservation strategies for wild bees the application of ecological stoichiometry. insects 9(3), 1–13 (2018). https://doi.org/10.3390/insects9030085 article  google scholar  marstaller, j., tausch, f., stock, s.: deepbees building and scaling convolutional neuronal nets for fast and large-scale visual monitoring of bee hives. in: 2019 ieee/cvf international conference on computer vision workshop (iccvw), pp. 271–278. (2019). https://doi.org/10.1109/iccvw.2019.00036 xia, d., chen, p., wang, b., zhang, j., xie, c.: insect detection and classification based on an improved convolutional neural network. sensors 18(12), 1–12 (2018). https://doi.org/10.3390/s18124169 article  google scholar  abreu, v.h.r., pimentel, a.d.a., absy, m.l., rech, a.r.: pollen sources used by frieseomelitta ihering 1912 (hymenoptera: apidae: meliponini) bees along the course of the rio negro, amazonas. brazil. acta botanica brasilica 24(2), 371–383 (2020). https://doi.org/10.1590/0102-33062019abb0391 article  google scholar  júnior, t.c., rieder, r.: automatic identification of insects from digital images: a survey. comput. electron. agric. 178(5), 105784 (2020). https://doi.org/10.1016/j.compag.2020.105784 article  google scholar  zhong, y., gao, j., lei, q., zhou, y.: a vision-based counting and recognition system for flying insects in intelligent agriculture. sensors 18(5), 1489 (2018). https://doi.org/10.3390/s18051489 article  google scholar  qing, y., et al.: development of an automatic monitoring system for rice light-trap pests based on machine vision. j. integr. agric. 19(10), 2500–2513 (2020). https://doi.org/10.1016/s2095-3119(20)63168-9 article  google scholar  shen, y., zhou, h., li, j., jian, f., jayas, d.s.: detection of stored-grain insects using deep learning. comput. electron. agric. 145, 319–325 (2018). https://doi.org/10.1016/j.compag.2017.11.039 article  google scholar  liu, l., et al.: pestnet: an end-to-end deep learning approach for large-scale multi-class pest detection and classification. ieee acess 7, 45301–45312 (2019). https://doi.org/10.1109/access.2019.2909522 article  google scholar  fujioka, h., abe, m.s., okada, y.: ant activity-rest rhythms vary with age and interaction frequencies of workers. behav. ecol. sociobiol. 73(3), 30 (2019). https://doi.org/10.1007/s00265-019-2641-8 article  google scholar  fujioka, h., abe, m.s., okada, y.: individual ants do not show activity-rest rhythms in nest conditions. j. biol. rhythms 36(3), 297–310 (2021). https://doi.org/10.1177/07487304211002934 article  google scholar  ratnayake, m.n., dyer, a.g., dorin, a.: tracking individual honeybees among wildflower clusters with computer vision-facilitated pollinator monitoring. plos one 16(2), e0239504 (2021). https://doi.org/10.1371/journal.pone.0239504 article  google scholar  lima, m.c.f., leandro, m.e.d.a., valero, c., coronel, l.c.p., bazzo, c.o.g.: automatic detection and monitoring of insect pests a review. agriculture 10(5), 161 (2020). https://doi.org/10.3390/agriculture10050161 article  google scholar  imirzian, n., et al.: automated tracking and analysis of ant trajectories shows variation in forager exploration. sci. rep. 9(1), 1 (2019). https://doi.org/10.1038/s41598-019-49655-3 article  google scholar  sclocco, a., ong, s.j.y., aung, s.y.p., teseo, s.: integrating real-time data analysis into automatic tracking of social insects. r. soc. open sci. 8(3), 202033 (2021). https://doi.org/10.1098/rsos.202033 article  google scholar  tathawee, t., wattanachaiyingcharoen, w., suwannakom, a., prasarnpun, s.: flash communication pattern analysis of fireflies based on computer vision. int. j. adv. intell. inf. 6(1), 60–71 (2020). https://doi.org/10.26555/ijain.v6i1.367 article  google scholar  bjerge, k., nielsen, j.b., sepstrup, m.v., helsing-nielsen, f., hoye, t.t.: an automated light trap to monitor moths (lepidoptera) using computer vision-based tracking and deep learning. sensors 21(2), 343 (2021). https://doi.org/10.3390/s21020343 article  google scholar  howard, s.r., ratnayake, m.n., dyer, a.g., garcia, j.e., dorin, a.: towards precision apiculture: traditional and technological insect monitoring methods in strawberry and raspberry crop polytunnels tell different pollination stories. plos on 16(5), e0251572 (2021). https://doi.org/10.1371/journal.pone.0251572 article  google scholar  perez-cham, o.e., et al.: parallelization of the honeybee search algorithm for object tracking. appl. sci. 10(6), 2122 (2020). https://doi.org/10.3390/app10062122 article  google scholar  sun, c., gaydecki, p.: a visual tracking system for honey bee (hymenoptera: apidae) 3d flight trajectory reconstruction and analysis. j. insect sci. 21(2), 1–12 (2021). https://doi.org/10.1093/jisesa/ieab023 article  google scholar  wang, z., bovik, a.c., sheikh, h.r., simoncelli, e.p.: image quality assessment: from error visibility to structural similarity. ieee trans. image process. 13(4), 600–612 (2004) article  google scholar  redmon, j., divvala, s., girshick, r., farhadi, a.: you only look once: unified, real-time object detection. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 779–788 (2016) google scholar  redmon, j., farhadi, a.: yolo9000: better, faster, stronger. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 7263–7271 (2016) google scholar  redmon, j., farhadi, a.: yolov3: an incremental improvement. arxiv preprint: arxiv:1804.02767 (2018) bochkovskiy, a., wang, c., liao, h.m.: yolov4: optimal speed and accuracy of object detection. cornell univ. arxiv preprint: arxiv:2004.10934 (2020) jocher, g.: ultralytics/yolov5: v3.1. (2020). https://doi.org/10.5281/zenodo.4154370 li, c., et al.: yolov6: a single-stage object detection framework for industrial applications. arxiv preprint: arxiv:2207.02696 (2022) wang, c., bochkovskiy, a., liao, h.m.: designing network design strategies through gradient path analysis. arxiv preprint: arxiv:2211.04800 (2022) jocher, g., chaurasia, a., qiu, j.: yolo by ultralytics v8. (2023) google scholar  zhang, y., et al.: bytetrack: multi-object tracking by associating every detection box. in: proceedings of the european conference on computer vision (2022) google scholar  dendorfer, p., et al.: mot20: a benchmark for multi object tracking in crowded scenes. arxiv:2003.09003 [cs] (2020) download references acknowledgments the authors would like to thank the universidade de são paulo (usp beekeep cs https://beekeep.pcs.usp.br), the empresa brasileira de pesquisa agropecuária (embrapa), and the associação brasileira de estudo das abelhas (a.b.e.l.h.a. https://abelha.org.br) by the data and videos. people who allowed filmings on their properties. to the laboratório multiusuário de práticas simuladas (lamps https://lamps.medicina.ufop.br) and the laboratório de controle e automação multiusuário (labcam) for the infrastructure and equipment provided. google collaboratory by the technologies that make ai research possible with scarce resources. to carlos j. pereira, eduardo carvalho, levi w. r. filho, and andré a. santos (instituto tecnológico vale) along with diego m. alberto (efí) for their support with the computational methods. this research received financial support from the coordenação de aperfeiçoamento de pessoal de nível superior brazil (capes) financing code 001. author information authors and affiliations programa de pós-graduação em ciência da computação – universidade federal de ouro preto (ufop), ouro preto, mg, brazil rodolfo r. v. leocádio, alan kardek rêgo segundo & gustavo pessin instituto tecnológico vale – mineração, ouro preto, mg, brazil alan kardek rêgo segundo & gustavo pessin authors rodolfo r. v. leocádioview author publications search author on:pubmed google scholar alan kardek rêgo segundoview author publications search author on:pubmed google scholar gustavo pessinview author publications search author on:pubmed google scholar corresponding author correspondence to rodolfo r. v. leocádio . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper leocádio, r.r.v., segundo, a.k.r., pessin, g. (2023). multiple object tracking in native bee hives: a case study with jataí in the field. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_12 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computer vision tracking native bees optical flow publish with us policies and ethics profiles alan kardek rêgo segundo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a feature-based out-of-distribution detection approach in skin lesion classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a feature-based out-of-distribution detection approach in skin lesion classification conference paper first online: 12 october 2023 pp 338–352 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a feature-based out-of-distribution detection approach in skin lesion classification download book pdf download book epub thiago carvalho  orcid: orcid.org/0000-0001-8689-14389, marley vellasco  orcid: orcid.org/0000-0002-9790-13289, josé franco amaral  orcid: orcid.org/0000-0003-4951-853210 & … karla figueiredo  orcid: orcid.org/0000-0001-8420-393710  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 542 accesses 1 citation abstract when dealing with deep learning applications in open-set problems, accurately classifying known classes seen in the training phase is not the only aspect to be taken into account. in such a context, detecting out-of-distribution (ood) samples plays an important role as an auxiliary task, generally solved by ood detection methods. for medical applications, detecting unknown samples may in classification problems can be beneficial for many aspects, such as a better understanding of the diagnosis and probably a more adequate treatment. in this article, we evaluate a feature space-based approach, named as openpcs-class, for ood detection in medical applications, more specifically skin lesion classification. we compare the openpcs-class against important ood detection methods, evaluating different model architectures and ood datasets. the openpcs-class outperformed other methods at 48.4% and 5.3% in terms of fpr95 and auroc, respectively. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others out-of-distribution detection for long-tailed and fine-grained skin lesion images chapter © 2022 out-of-distribution detection for skin lesion images with deep isolation forest chapter © 2020 exploring distribution-based approaches for out-of-distribution detection in deep learning models article 28 december 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition categorization computer vision machine learning object recognition statistical learning 1 introduction deep learning (dl) models have found widespread use in various applications, ranging from autonomous driving [19] and pest detection to speech recognition [26]. despite its outstanding results in tasks related to computer vision and natural language processing, accuracy is not the only subject to be taken into consideration in a dl deployment [10]. depending on the problem, other aspects may also become important, such as the explainability and the capability to handle samples from unknown classes [34]. the dl models are known to learn generally in closed-set assumptions, and such out-domain restrictions are reflected in their inefficiency in explicitly showing ignorance about input samples from unseen classes. as a result, a dl model trained in such a setup is often unable to identify an unknown class data as unknown, which leads to problems of model overconfidence [32]. the overconfidence has several natures, such as unidentified overfitting problems, bias, or even the choice of the softmax function for the model’s output layer, making directly identifying unknown samples more difficult [34]. therefore, the model needs to be robust and able to handle out-of-distribution (ood) samples, which can come in various forms depending on the problem. for medical applications, ood detection is an important auxiliary task to improve the ability to detect unseen classes in an open-set problem. for example, when classifying an unseen rare skin lesion using a dl model to classify skin lesions, it would be preferable to identify it as unknown instead of erroneously classifying it as one of the known classes [25, 36]. therefore, the ood detection task has drawn attention to a wide range of applications, such as histopathology [22], x-ray [3], and magnetic resonance images [14] classification problems. ood detection can be considered a recent field of research in the area of dl, having one of the main objectives to improve the ability of models to recognize unknown samples. in other words, an ood detection algorithm should be able to identify whether an input can be considered known or unknown. the most straightforward option for ood detection is to use activations from the model’s output layer, as this is closest to the final inference result [9]. these strategies typically rely on logits or softmax outputs to compute confidence scores, which are then used to differentiate between known and unknown classes. more recently, researchers have explored using the feature space of the model to identify unknown samples, based on the assumption that the feature space can be useful for ood detection, as intermediate layers capture different levels of semantic features [20]. one of the methods is named open principal component score (openpcs), which uses a low dimensional feature space representation from principal component analysis (pca) to fit class-wise gaussian distributions to identify whether data is known or unknown. this approach was first implemented for semantic segmentation problems, but it can be extended for multi-class classification, named as openpcs-class [5]. however, the feature-space approach for ood detection, especially the openpcs-class, is still underexplored for many applications. in this article, we evaluate the openpcs-class for ood detection in skin lesion classification problems. the objective is to evaluate the capability of a gaussian-based approach using the feature space to identify unseen classes in this medical application, which is usually a complex task with numerous ood classes related to unknown skin lesions. the contribution of this work is three-fold: 1. we evaluate the openpcs-class method for ood detection in skin lesion problems. we use different ood data to evaluate the approach, ranging from samples of unseen classes of skin lesions to different medical problems. 2. we compare the results with traditional and state-of-the-art methods for ood detection. we assess how these methods behave in the presence of different ood classes and additional id data. 3. we also evaluate these models in different model architectures to investigate the model’s contribution to ood detection using different space representations. 2 related works detecting ood samples is crucial in building reliable deep learning models that need to operate effectively in an open-set scenario. in medical applications, such strategies allow dl models to enhance the robustness of such results in a critical task. these works are generally concentrated in semantic segmentation and image classification tasks [4]. karimi et al. [13] proposed a spectral analysis of the intermediate features of dl models to enhance the robustness of the segmentation task in multiple organs by quantifying the uncertainty of the segmentation result. wollek et al. [30] evaluated some state-of-the-art ood detection methods in several medical application tasks related to the image classification problem, discussing the advantages and drawbacks of such methods in identifying unknown samples closer to the training classes. due to the relevance of this topic in dl applications to guarantee safety and robustness, there are a plethora of new strategies related to ood detection. one of the most common methods for out-of-distribution (ood) detection involves using the softmax output as an ood score, known as maximum softmax probability (msp) [11]. the msp is based on the idea that unknown class samples would generate lower confidence scores for each known class, which are then used to distinguish id and ood data. this method was evaluated in a wide range of problems, including medical applications. zhang et al. [37], for example, evaluated the effectiveness of the msp method in ood detection for diabetic retinopathy detection and chest radiography-related problems. however, using softmax output can sometimes lead to overconfident scores on unknown data, which is inappropriate for ood detection [33]. to avoid the issues associated with the softmax, the feature space can also distinguish between known and unknown samples. lee et al. [15] proposed a method that uses the information from the feature space to detect ood samples, assuming that the feature representation can be fitted into gaussian distributions. in this case, the class-conditional gaussian distributions are obtained and the score is computed as the mahalanobis distance from a test sample to the closest class-conditional distribution [24]. this ood detection method was applied in different medical applications related to image analysis, such as malaria parasitized cells classification [28], lung cancer classification [2], and skin lesion classification [25]. despite its efficiency in the ood detection task, the feature space is generally a high-dimensional representation, which can be often an inefficient representation with high redundancy and lead to a harder fit of the ood detection method [31]. to alleviate the problem of high dimensionality in intermediate representations, oliveira et al. [21] proposed a method for ood detection, called openpcs, using pca to reduce the dimensionality of the feature space. the low-dimensionality representation is then used to adjust class-conditional gaussian distributions, and the score is calculated by finding the maximum likelihood between a sample’s intermediate representation and the class-conditional distributions. more recently, carvalho et al. [5] proposed an extension of its method for multi-class classification problems, named as openpcs-class. this method was successfully evaluated in benchmark problems, but the openpcs-class is still unexplored in different applications, including medical image analysis. 3 detecting unseen samples using feature space in this section, we describe in detail the openpcs-class method strategy. we also briefly introduce the ood detection problem in skin lesion problems, motivating the applicability of this work. the code of this work is publicly availablefootnote 1 3.1 open principal component score for image classification the open principal component score (openpcs) is a method that uses intermediate features for ood detection in a semantic segmentation task. originally, this method could be applied only to fully convolutional networks (fcn), which can be prohibitively for direct utilization of openpcs for different dl tasks. the openpcs-class can be seen as the extension of the openpcs method for classification tasks. this method discards the need of a fcn but retains the main characteristics of using a combination of intermediate features in a low-level representation. for a better comprehension of the method, fig. 1 displays the method overview for an image classification problem. fig. 1. openpcs-class overview for image classification full size image the openpcs-class is an ood detection method that can combine features from different layers to distinguish whether a sample belongs to a known or unknown class. for each model layer l, we transform the activation map \(a^{(l)}\) to the corresponding activation vector \(h^{(l)}\) by using a reduction method (e.g., average pooling). therefore, we always obtain its feature vector independently from the layer specification. one of the main abilities of the openpcs method is the capability to combine the feature representation from different layers, which is a user-defined parameter. for classification tasks, the features are combined by concatenating their vectors, resulting in a feature vector h. the drawback of such an approach is the high dimensionality of the feature vector h. to alleviate this issue, we apply the pca to obtain a better representation in a low dimension. to fit the eigenvectors and eigenvalues for the pca, we follow a class-wise approach. therefore, we use the collection of feature vectors related to each of the known classes to fit the parameters of the pca, creating a specific dimensionality reduction for each of the known classes, according to eq. 1. $$\begin{aligned} h^{*}_c = h \cdot v_{c} \end{aligned}$$ (1) where h is the feature representation, \(v_{c}\) is the eigenvector with the highest eigenvalues for dimensionality reduction for class c, and \(h^{*}_c\) is the feature vector after transformation for class c. therefore, depending on the class c, the resulting low-dimension feature vector \(h^{*}_{c}\) can be different. it is important to note that we can obtain different low-dimensional representations for the same feature vector h, depending on the class c that we are evaluating. the ood score is computed by estimating how likely the feature vector is to each of the known class. in the literature, the gaussian density estimator was successfully used to quantify the ood-ness [18, 21]. therefore, we adopted the gaussian density estimator to fit and compute its corresponding likelihood. mathematically, the ood score for each class c is computed according to eq. 2 $$\begin{aligned} g_c(h^{*}_c) = \frac{1}{\sqrt{2\pi \sigma _{c}^2}}\exp \left( -\frac{(h^{*}_c -\mu _{c})^2}{2\sigma _{c}^2}\right) \end{aligned}$$ (2) where \(\mu _c\) and \(\sigma _c\) represents the mean and standard deviation for a known class c, and \(g_{c}(a^{*}_c)\) represents the probability density of \(h^{*}_c\) generated by \(g_{c}\). the final ood score is the maximum log-likelihood over all known classes, as defined in eq. 3. $$\begin{aligned} s = \max _{c=1}^{n} {\log \left[ g_{c} \left( h^{*}_{c} \right) \right] } \end{aligned}$$ (3) where n is the number of classes. in summary, to detect if a sample can be considered as ood, we obtain its feature vector representation and apply a class-wise dimensionality reduction, and for each low-dimensionality representation vector, we calculate the log-likelihood to its corresponding class. the ood score can be viewed as the maximum log-likelihood over all of the classes. for an id sample, the likelihood would be lower for all of the classes, except for the corresponding class, which yields a high s score. as for the ood sample, the likelihood tends to be lower for all class-wise distributions, so the score s also is lower. thereby, the id and ood samples can be distinguished by setting a threshold value for s. 3.2 ood detection in skin lesion classification with the world constantly evolving, new medical pathologies are frequently discovered through diagnosis. however, the identification of novel or rare diseases can be troublesome for dl-based automated diagnosis, potentially leading to incorrect classification and inappropriate treatment [36]. in such cases, ood detection methods can play an essential role in identifying whether a new sample belongs to any of the known classes of the problem, thus providing an auxiliary task for dl-based approaches. specifically for dermatological-related tasks, ood detection strategies can be handy to identify samples from unseen classes during the training phase. for instance, consider a deep learning problem aimed at automatically identifying the three most common skin lesions, as illustrated in fig. 2. in an open-set scenario, the trained model may encounter unseen skin lesions, so it would be preferable to detect them as unknown instead of erroneously classifying them as the closer known class. ideally, the ood detection method should be capable of correctly classifying these samples as ood, but this may depend on the chosen strategy [6]. especially when id and ood samples are visually similar, it can be challenging to distinguish between known and unknown classes [30]. fig. 2. examples of in-distributions and out-of-distributions samples of skin lesions full size image this article evaluates the feature space-based ood detection method in different scenarios. in some experiments, we verify the capability of the openpcs-class to detect near-ood samples, typically skin lesions images taken in the same settings as the id samples. we also evaluate some ood detection approaches in the same problem (skin lesions), but under different conditions to evaluate the changing of such ood detection strategies. finally, we also evaluate these models in far-ood detection samples, but related to medical applications. 4 experiments this section presents the experimental protocol for our case studies in the ood detection task. in this work, we focused on the skin lesion classification problem, selecting different medical-related samples as ood. 4.1 ood methods we evaluated three robust methods commonly employed in this area to assess the ood detection results. one of them is the maximum softmax probability (msp) method [11], which is a traditional approach that utilizes the softmax probability vector to identify unknown samples. by computing the maximum probability value of all classes, msp assumes that a lower msp score suggests that the model is less accurate about the predicted class, which could indicate an ood sample. another method we selected is energy-based out-of-distribution detection (ebo) [16], a more sophisticated technique that uses the output space to calculate the ood score. ebo computes the entropy of the logit and employs it as an ood score to distinguish between ood and id samples. we also opted for a feature space-based method for ood detection to provide a more insightful discussion of the openpcs-class approach. the mahalanobis ood detection method [12] measures the ood score as the mahalanobis distance of class-conditional gaussian distribution in the feature space. in this case, if compared to the gaussian distributions, ood samples are expected to be further away from id samples. 4.2 datasets for the ood detection in medical multi-class classification, we have utilized the ham10000 dataset as our in-distribution dataset (\(d_{in}\)) for skin lesion classification [27]. this dataset comprises 10000 images of seven distinct skin lesions: melanocytic nevi, melanoma, benign keratosis-like lesions, basal cell carcinoma, actinic keratoses, vascular lesions, and dermatofibroma. for this study, we have selected the first four classes as our id classes, which contain 6705, 1113, 1099, and 514 samples, respectively. in addition, we have designated the remaining three classes as ood samples to form the basis of our first case study. for the other experiments, we maintain the same \(d_{in}\) and id classes, changing the ood samples. the dataset from the second case study consists of a wide range of skin lesion images taken in different parts of the body. however, a significant issue with this dataset is that some classes overlap with those found in \(d_{in}\). therefore, to ensure a fair comparison of the ood detection task, we remove the overlapping classes from \(d_{out}\). the third selected \(d_{out}\) is related to the monkeypox classification problem [1]. the dataset contains images of monkeypox lesions and different skin lesions (e.g., chickenpox), given that the problem originally was built as a binary classification to identify whether a lesion can be considered monkeypox. therefore, we used all images from this dataset as ood samples in the third experiment. for the fourth case study, we have manually selected images of rare skin lesions that do not belong to any of the classes in \(d_{in}\). in this experiment, we have included additional id images obtained from different circumstances than those found in the ham10000 dataset. this collection of images will enable us to gain practical insights into the identification of unknown and uncommon classes in skin lesion classification and evaluate how the ood detection methods perform when presented with different id samples. 4.3 metrics to compare the methods, we selected three metrics to evaluate the ood detection task in multi-classification problems [35]. auroc (area under recall operating curve) summarizes the recall operating curve (roc) as calculating the area under the curve. as roc is usually used in a binary classification problem, to evaluate the ood detection task using this metric, we consider only id and ood classes, independently from the fine-grained classes. mathematically, the auroc can be approximated as evaluating the true positive rate (tpr) and false positive rate (fpr) at discrete threshold values, presented in eq. 4 $$\begin{aligned} \text {auroc} = \sum _{i=1}^{n-1} \frac{1}{2} (x_{i+1} x_i) (y_i + y_{i+1}) \end{aligned}$$ (4) where n is the number of thresholds, \(x_i\) and \(y_i\) are the false positive and true positive rates, respectively, at the i-th threshold. aupr (area under precision-recall curve) is a metric that summarizes the precision-recall trade-off for different threshold values for a specific class. this metric is highly important for imbalance problems, which may be the case for our experiments. therefore, we calculate the aupr for the ood class. fpr95 indicates the false positive rate (fpr) when the true positive rate (tpr) is 95%. typically, the fpr95 describes how likely the method could erroneously classify as unknown at a reasonably high tpr. therefore, the lower the fpr95 is, the better the ood detection method. unlike the other ones, this is a dependent, since we define a cutoff value to classify as known or unknown. 4.4 experimental details to evaluate our proposed approach, we used the same experimental procedure in all experiments. for the \(d_{in}\), we split the dataset proportionally into training (60%), validation(20%), and test (20%) sets. we fit the ood detection methods using the training set and, for all experiments, we use the test samples from \(d_{in}\) and the whole \(d_{out}\) to evaluate the separability between id and ood samples, respectively. during the testing phase, we randomly selected 500 samples from each set of \(d_{in}\) and \(d_{out}\) (when applicable) and computed the average metrics over ten runs. we also used the wilcoxon signed-test rank to verify the statistical significance between the best result metric and all others. in sect. 5, we denote an average result with a statistical difference using an underscore in the tables. we also assess the impact of the ood detection methods in different model architectures. as our problem is related to the image classification, we selected three models, vision transformer (vit) model [7], convnext [17], and resnet [8]. for the first two architectures, we used the pre-trained weights on imagenet1k and finetuned the classification layer in the \(d_{in}\) problem. for the resnet model architectures, we trained from scratch, following a similar training procedure as presented in the literature [29]. 5 discussion and results this section contains the results of the four case studies in medical applications. it is important to note that the selected ood detection methods are similar in the experimental setup (i.e., it does not require any model retraining and just one forward pass is needed to identify ood samples), but use different approaches to detect unseen classes. for the first experiment, table 1 summarizes the results for different ood detection methods and architectures. table 1. ood detection results for experiment 1full size table the first experiment is a more challenging for discriminating wheter a sample belongs to a known or unknown class. this idea is reflected in the ood detection metric results, showing a lower auroc score for all of the methods, if compared to the other experiments. even so, we noticed that openpcs-class outperformed all three methods in terms of auroc and aupr, independently from the model architecture. also, the fpr95 shows that our approach can enhance the ood detection task considering a real-world scenario, considering the threshold that yields a tpr at 95%. in that case, the openpcs-class can lower the fpr in this condition up to 7.1% (using resnet model and msp method). the model architecture plays an important role in ood detection. in this experiment, the vit model increased the capability to detect ood samples, at least for the openpcs-class method. especially for feature-based approaches, the model used can impact directly the results, given that different model architectures can yield feature activations. the second experiment can be considered an easier task in ood detection if compared to the first one. although the classes from \(d_{out}\) are similar in the first two experiments, the images were obtained in different body parts, which can facilitate the ood detection task. the results for the second experiment can be observed in table 2. table 2. ood detection results for experiment 2full size table in this experiment, the approaches based on the feature space had a better ood detection capability, if compared to those who use the output space. in fact, the feature space can contain low-level and high-level feature information, which can help to detect unknown classes in different contexts. on the other hand, the output space does not contain such kind of information, which may help to understand the difference between those approaches. therefore, these strategies directly impact the scores generated, as illustrated in fig. 3. fig. 3. id and ood score distributions (a) maximum softmax score from msp (b) maximum likelihood from openpcs-class full size image the main objective of ood detection is to yield scores that could be easy to distinguish between id and ood samples. to evaluate the distributions obtained in fig. 3, we conduct a welch t-test [38], which rejected the hypothesis that the id and ood distributions have equal means (\(p < 0.05\)) only for the openpcs-class. the openpcs-class, in this experiment, outperformed all three methods for ood detection (decreased 48.7% in terms of fpr95, if compared to mahalanobis and convnext). however, there is a slight difference between the openpcs-class and mahalanobis methods, depending on the model architecture. for transformer-based models, both methods obtained, auroc and aupr metrics closer to one. this result corroborates recent findings that models based on transformer architecture can enhance the robustness of ood detection [23]. the third experiment uses skin lesions pathologies that are more different from those presented in \(d_{in}\). the results are presented in table 3. table 3. ood detection results for experiment 3full size table although the \(d_{out}\) in the third study case contains images from skin lesions, we noticed that all methods, independently from the model architecture, enhanced the ood detection metrics, if compared to the previous experiments. as the ood samples are related to diseases like monkeypox and chickenpox, the images are more dissimilar to those presented to the model in the training phase (using \(d_{in}\)), so there is a low confidence score in the output space, and the feature representation from ood samples are more dissimilar to those presented from id samples, resulting in an easier ood detection task. in this experiment, the feature-based approaches also obtained a considerably high capability to detect samples visually dissimilar to those presented in \(d_{in}\). for transformer-based approaches, mahalanobis and openpcs-class obtained comparable results, given that they could almost differentiate total id and ood samples. however, for the resnet architecture, the difference between those approaches is more significant, showing better performance for the openpcs-class method (increased 5.1% if compared to the mahalanobis detector). for the last case study, table 4 displays the results for ood detection using the same experimental protocol as the previous experiments. table 4. ood detection results for experiment 4full size table in this experiment, we observed that feature-based approaches performed comparably better in detecting a wide range of pathologies as ood. even in the presence of new images for id classes that are slightly different from those presented in the \(d_{in}\), the openpcs-class outperformed other methods in all three evaluation metrics. therefore, even with visually different id samples, the feature-space approaches obtained better results in the ood detection task. although we only present the distributions for the second experiment, we used the welch t-test for all experiments in this section. for all experiments using the transformer-based architectures, we noted that the id and ood distributions can be easily distinguished (i.e., it contains different means) for the openpcs-class. 6 conclusions in this work, we evaluated the openpcs-class for a new domain of application for ood detection, more specifically for skin lesion problems. the feature space-based approaches, in general, obtained a superior ood detection when the ood samples are visually more dissimilar to id ones, corresponding to the latter three experiments of this work. compared to all the methods evaluated in the experiments, the openpcs-class outperformed in all scenarios regarding auroc, and 9 (out of 12) in terms of average fpr95. more interestingly, the transformer-based models were more suitable for the openpcs-class method, which always obtained superior ood detection results. going forward, we aim to evaluate the openpcs-class in different medical classification problems, to get a better perspective of feature space-based models in ood detection application problems. notes 1.code available at https://github.com/mdrs-thiago/skin-lesion-ood-detection. references ali, s.n., et al.: monkeypox skin lesion detection using deep learning models: a preliminary feasibility study. arxiv preprint arxiv:2207.03342 (2022) berger, c., paschali, m., glocker, b., kamnitsas, k.: confidence-based out-of-distribution detection: a comparative study and analysis. in: sudre, c.h., et al. (eds.) unsure/pippi -2021. lncs, vol. 12959, pp. 122–132. springer, cham (2021). https://doi.org/10.1007/978-3-030-87735-4_12 chapter  google scholar  calderon-ramirez, s., yang, s., elizondo, d., moemeni, a.: dealing with distribution mismatch in semi-supervised deep learning for covid-19 detection using chest x-ray images: a novel approach using feature densities. appl. soft comput. 123, 108983 (2022) article  google scholar  cao, t., huang, c.w., hui, d.y.t., cohen, j.p.: a benchmark of medical out of distribution detection. arxiv preprint arxiv:2007.04250 (2020) carvalho, t., vellasco, m., amaral, j.f.: out-of-distribution detection in deep learning models: a feature space-based approach. in: international joint conference on neural networks (2023) google scholar  cho, w., park, j., choo, j.: training auxiliary prototypical classifiers for explainable anomaly detection in medical image segmentation. in: proceedings of the ieee/cvf winter conference on applications of computer vision, pp. 2624–2633 (2023) google scholar  dosovitskiy, a., et al.: an image is worth 16 \(\times \)16 words: transformers for image recognition at scale. arxiv preprint arxiv:2010.11929 (2020) he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  hendrycks, d., et al.: scaling out-of-distribution detection for real-world settings. in: international conference on machine learning, pp. 8759–8773. pmlr (2022) google scholar  hendrycks, d., carlini, n., schulman, j., steinhardt, j.: unsolved problems in ml safety. arxiv preprint arxiv:2109.13916 (2021) hendrycks, d., gimpel, k.: a baseline for detecting misclassified and out-of-distribution examples in neural networks. arxiv preprint arxiv:1610.02136 (2016) kamoi, r., kobayashi, k.: why is the mahalanobis distance effective for anomaly detection? arxiv preprint arxiv:2003.00402 (2020) karimi, d., gholipour, a.: improving calibration and out-of-distribution detection in deep models for medical image segmentation. ieee trans. artif. intell. 4, 383–397 (2022) article  google scholar  lambert, b., forbes, f., doyle, s., tucholka, a., dojat, m.: improving uncertainty-based out-of-distribution detection for medical image segmentation. arxiv preprint arxiv:2211.05421 (2022) lee, k., lee, k., lee, h., shin, j.: a simple unified framework for detecting out-of-distribution samples and adversarial attacks. in: advances in neural information processing systems, vol. 31 (2018) google scholar  liu, w., wang, x., owens, j., li, y.: energy-based out-of-distribution detection. adv. neural. inf. process. syst. 33, 21464–21475 (2020) google scholar  liu, z., mao, h., wu, c.y., feichtenhofer, c., darrell, t., xie, s.: a convnet for the 2020s. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 11976–11986 (2022) google scholar  martinez, j.a.c., oliveira, h., dos santos, j.a., feitosa, r.q.: open set semantic segmentation for multitemporal crop recognition. ieee geosci. remote sens. lett. 19, 1–5 (2021) article  google scholar  muhammad, k., et al.: vision-based semantic segmentation in scene understanding for autonomous driving: recent achievements, challenges, and outlooks. ieee trans. intell. transp. syst. 23, 22694–22715 (2022) article  google scholar  nunes, i., pereira, m.b., oliveira, h., santos, j.a.d., poggi, m.: fuss: fusing superpixels for improved segmentation consistency. arxiv preprint arxiv:2206.02714 (2022) oliveira, h., silva, c., machado, g.l., nogueira, k., dos santos, j.a.: fully convolutional open set segmentation. mach. learn. 112, 1733–1784 (2021) article  mathscinet  math  google scholar  pawlowski, n., glocker, b.: abnormality detection in histopathology via density estimation with normalising flows. in: medical imaging with deep learning (2021) google scholar  podolskiy, a., lipin, d., bout, a., artemova, e., piontkovskaya, i.: revisiting mahalanobis distance for transformer-based out-of-domain detection. in: proceedings of the aaai conference on artificial intelligence, vol. 35, pp. 13675–13682 (2021) google scholar  ren, j., fort, s., liu, j., roy, a.g., padhy, s., lakshminarayanan, b.: a simple fix to mahalanobis distance for improving near-ood detection. arxiv preprint arxiv:2106.09022 (2021) roy, a.g., et al.: does your dermatology classifier know what it doesn’t know? detecting the long-tail of unseen conditions. med. image anal. 75, 102274 (2022) article  google scholar  swetha, p., srilatha, j.: applications of speech recognition in the agriculture sector: a review. ecs trans. 107(1), 19377 (2022) article  google scholar  tschandl, p., rosendahl, c., kittler, h.: the ham10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. sci. data 5(1), 1–9 (2018) article  google scholar  uwimana, a., senanayake, r.: out of distribution detection and adversarial attacks on deep neural networks for robust medical image analysis. arxiv preprint arxiv:2107.04882 (2021) wightman, r., touvron, h., jégou, h.: resnet strikes back: an improved training procedure in timm. arxiv preprint arxiv:2110.00476 (2021) wollek, a., willem, t., ingrisch, m., sabel, b., lasser, t.: a knee cannot have lung disease: out-of-distribution detection with in-distribution voting using the medical example of chest x-ray classification. arxiv preprint arxiv:2208.01077 (2022) wright, j., ma, y.: high-dimensional data analysis with low-dimensional models: principles, computation, and applications. cambridge university press (2022) google scholar  wu, y., et al.: revisit overconfidence for ood detection: reassigned contrastive learning with adaptive class-dependent threshold. in: proceedings of the 2022 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 4165–4179 (2022) google scholar  wu, y., et al.: disentangling confidence score distribution for out-of-domain intent detection with energy-based learning. arxiv preprint arxiv:2210.08830 (2022) yang, j., zhou, k., li, y., liu, z.: generalized out-of-distribution detection: a survey. arxiv preprint arxiv:2110.11334 (2021) ye, n., et al.: ood-bench: quantifying and understanding two dimensions of out-of-distribution generalization. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 7947–7958 (2022) google scholar  zadorozhny, k., thoral, p., elbers, p., ciná, g.: out-of-distribution detection for medical applications: guidelines for practical evaluation. in: shaban-nejad, a., michalowski, m., bianco, s. (eds.) multimodal ai in healthcare. studies in computational intelligence, vol. 1060, pp. 137–153. springer, cham (2022). https://doi.org/10.1007/978-3-031-14771-5_10 zhang, o., delbrouck, j.-b., rubin, d.l.: out of distribution detection for medical images. in: sudre, c.h., et al. (eds.) unsure/pippi -2021. lncs, vol. 12959, pp. 102–111. springer, cham (2021). https://doi.org/10.1007/978-3-030-87735-4_10 chapter  google scholar  zimmerman, d.w., zumbo, b.d.: rank transformations and the power of the student t test and welch t’test for non-normal populations with unequal variances. can. j. exp. psychol. 47(3), 523 (1993) article  google scholar  download references acknowledgements this work was supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, conselho nacional de desenvolvimento e pesquisa (cnpq) under grants 140254/2021-8 and 308717/2020-1, and fundação de amparo à pesquisa do rio de janeiro (faperj) author information authors and affiliations pontifical catholic university of rio de janeiro, rio de janeiro, brazil thiago carvalho & marley vellasco rio de janeiro state university, rio de janeiro, brazil josé franco amaral & karla figueiredo authors thiago carvalhoview author publications search author on:pubmed google scholar marley vellascoview author publications search author on:pubmed google scholar josé franco amaralview author publications search author on:pubmed google scholar karla figueiredoview author publications search author on:pubmed google scholar corresponding author correspondence to thiago carvalho . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper carvalho, t., vellasco, m., amaral, j.f., figueiredo, k. (2023). a feature-based out-of-distribution detection approach in skin lesion classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_23 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords out-of-distribution detection deep learning feature space publish with us policies and ethics profiles marley vellasco view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evolutionary adjustment of a cellular automata-based model for wildfire spreading | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evolutionary adjustment of a cellular automata-based model for wildfire spreading conference paper first online: 30 january 2025 pp 260–275 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) evolutionary adjustment of a cellular automata-based model for wildfire spreading download book pdf download book epub lucas v. murilo  orcid: orcid.org/0000-0002-0381-95199, gina m. b. oliveira  orcid: orcid.org/0000-0003-0384-18799 & luiz g. a. martins  orcid: orcid.org/0000-0003-0168-12939  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 392 accesses 2 citations abstract due to climate change, the incidence of fires has increased considerably in recent decades, causing various damages to the environment. in this scenario, computational models based on cellular automata (ca) are capable of reproducing and predicting the behavior of fire propagation, aiding in control and prevention measures. however, constructing and adjusting the parameters of these models is not a trivial task. in this context, a method capable of automatically adjusting these parameters can assist in this stage, which demands deep knowledge on the subject. in this work a genetic algorithm (ga) is presented aiming the adequate adjusting of a ca-based model parameter set. additionally, a new parameter is proposed allowing the adaptability of the ca model to the fire spreading dynamics, regardless of the sampling rate used to generate the reference data. the experiments shown that our approach was able to capture the dynamics present in the data. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others automatic evolutionary adjustment of cellular automata model for forest fire propagation chapter © 2022 fire spread modeling using probabilistic cellular automata chapter © 2024 wildfire simulation model based on cellular automata and stochastic rules chapter © 2022 1 introduction recently, due to climate change, wildfire emergence in different parts of the globe have taken on alarming proportions, being harmful to both vegetation and urban areas. higher temperatures dry out the landscape and help create the perfect environment for more extensive and frequent forest fires. the year 2021 was one of the worst in terms of fires, with a loss of 9.3 million hectares worldwide [14]. wildfires lead to a vicious cycle because as carbon emissions increase, the climate becomes warmer and drier, which favors the incidence of fires in increasingly larger areas and for a prolonged period, generating more carbon emissions. this cycle keeps repeating, and each year the damage caused by fire propagation is greater, in addition to factors related to human intervention that further aggravate the situation. among the difficulties in fighting fires is the lack of information about the affected areas and how fires spread across different landscapes [6]. therefore, appropriate modeling and simulation techniques are necessary to expand the understanding of the effect of factors such as vegetation, climate, topography and human influence on the flames dynamics enabling efficient planning of wildfire prevention and combat countermeasures. cellular automata (ca) have shown to be an extremely useful tool for representing natural phenomena [22]. notably, the study of fire propagation is one of the most investigated natural phenomena using ca models [5, 8, 17, 20]. a ca can be defined as a discrete dynamic system, composed of a lattice of agents, called cells. they are characterized by decentralized computation, where each cell interacts with its neighborhood to define its state at each time step. the evolution of the states of the ca cells is determined by simple transition rules, but when combined with local interaction between the cells, they result in a global complex dynamic of the lattice (emergent computation) [22]. however, for a model based on ca to adequately represent a phenomenon, it is necessary to correctly adjust the parameters of such model. estimating the values for these parameters is not a trivial task. as the number of parameters increases, so does the complexity due to the number of possible combinations [9]. in the case of fires, these factors can include climate, terrain, amount of combustible material, wind intensity, and many others [15]. on the other hand, approaches based on evolutionary algorithms, such as genetic algorithms, are widely used in the literature to adjust the parameters of complex models [2, 21]. a genetic algorithm (ga) is a search meta-heuristic inspired by darwin’s theory of evolution [11]. this type of algorithm deals with a population of individuals, which represent potential solutions, and employs selection methods and genetic operators that mimic the process of natural selection. that is, the idea that the fittest individuals (solutions) tend to survive and have a greater chance of passing on their genetic material to future generations. gas are commonly used in optimization problems since their genetic operators direct the exploration towards promising regions of the search space to find the best solution. ca models tuning based on evolutionary approach was previously investigated in the literature [8, 9, 16]. this work aims to develop an evolutionary approach using genetic algorithms to automatically adjust the parameters of an ca-based fire propagation model. it relies strongly on the previous approach proposed in [8] to reproduce fire propagation dynamics in homogeneous vegetation according to patterns observed in synthetically generated datasets. however, in the previous approach the data sampling rate was not taken into account. preliminary experiments have shown that this information is important for adjusting time steps in the evolution of a ca-based model to adequately capture the fire dynamic presents in the reference dataset. this was highlighted when the dataset is generated from a model that exhibits different dynamic behavior than that being adjusted by ga to reproduce the patterns in the data. from a broader perspective, in future more realistic applications, when the data to be used as reference are images sampled from the evolution of a real fire, this adjustment aims to make the number of steps in the temporal evolution of the ca model compatible with the sampling rate of the images. experiments were carried out, where the fire spreading achieved from our approach is compared with the reference data. different datasets were used, each one constructed from the distinct intervals (ca steps) between samples. it is also evaluated the method’s ability to automatically set the values of the model’s parameters in order to reproduce the behavior observed in the synthetic datasets created using both our model and another fire propagation ca-based model of the literature [20]. results showed that the ga adjustments presented a wildfire spreading similar to the dynamics represented in each reference dataset, demonstrating the robustness and adaptability of the method. 2 related works a brief literature review is presented that includes some important works related to both the models based on cellular automata to simulate the wildfires propagation and to the genetic algorithms employed to optimize the parameters of such models. the approach proposed in [7] is a reference for fire spread models using cellular automata. it is a two-dimensional ca lattice composed of three states cells and stochastic transition rules, that uses moore neighborhood and simulate the recovery of burnt cells and the spontaneous combustion. in [25] was proposed an ac-based model with hexagonal cells integrated as a rothermel model to simulate the temporal and spatial spread of forest fires. real data from a chinese province and artificial data were used in experiments that showed its ability to reproduce fires in heterogeneous landscapes. in [24], ca was integrated with gis (geographic information systems) to provide fire spread simulations where topographic elements, forest fuels and climate variables are considered. the proposal was evaluated on real data from a canadian city and the authors concluded that it can be applied to reproduce realistic fires and other spatio-temporal applications. using a case study related to a forest fire on spetses island in 1990, a ca-based fire spread model was developed in [1], where a black box nonlinear optimization approach is used to adjust the parameters from real gis data. the results showed that the event was well represented and that the approach can be used to manage risks in future fires. in [10] was presented a ca-based fire propagation model that applies a numerical optimisation approach to find values that correlate the model parameters.according to the authors, the results found were close to classical approaches. in [23], a ca-based model was used to evaluate the influence of a set of factors, such as combustible materials, wind, temperature and terrain, in the flames dynamics. the authors concluded that the proposed approach was able to satisfactorily simulate the flames behavior trends, aiding to understand the key factors of a wildfire propagation under different conditions. a hybrid approach was presented in [19], where ca is combined with other techniques calculating a propagation index using meteorological data from remote sensing in order to adjust the automaton’s evolution step to wildfire spread. according to the results, the model was able to predict the flames dynamics, ensuring accuracy and adaptability in the simulations. in [8], a 2d probabilistic ca-based model is tunning in order to improve the quality of the wildfire spread simulations. ca transition rule considers characteristics such as wind intensity, burning time and vegetation recovery, where the probability of a cell in the tree state starting to burn is calculated based on the burning neighbors and the direction and intensity of the wind. wildfire propagation model based in a 2d probabilistic ca is proposed in [20]. the model adopts different fire states and a non-linear vegetation recovery function, as well as modeling important characteristics that affect flame dynamics, such as the burning time, the fire intensity, the wind intensity and direction, the presence of obstacles. the last two models were used as case studies in this paper. the approach presented in [8] was adopted as the standard model to be configured by the ga, while the model proposed in [20] was used to generate reference datasets for the experiments. finally, a 2d ca-based model was used in [4] to reproduce wildfires in a real brazilian cerrado environment. different scenarios of vegetagion (homogeneous, heterogeneous) were simulated to evaluate the influence of model parameters, such as combustion intensity, wind and rivers, on the accurate reproduction of fire behavior. evolutionary approaches have been applied in the adjustment of various applications such as parameter definition of neural networks [21], scheduling problems [18], the spread of diseases [9, 16], adjustment of hydrological-hydrodynamic and water quality models [12], among other applications. several studies employ genetic algorithms in predicting and adjusting wildfire propagation models. in [3] is emphasized that fire modeling requires many parameters that are not always known and a genetic algorithm is used in order to find these parameters from historical data and calibrate the model at the end of the execution. ga’s population is evaluated using a metric called ffdi (forest fire danger index), which is calculated based on weather variable factors such as temperature, relative humidity, wind speed and drought factor. approach described in [5] utilizes a two-step strategy to calibrate the model, where the quality of the solution is evaluated by comparing the simulation data with the real fire data (burned in the real fire, burned in the simulated fire, or not burned). then, a \(2 \times 2\) table is constructed with four possibilities (hits, false alarms, misses, correct negatives). finally, the individual’s fitness is calculated. the authors concluded that there was a significant improvement in performance without compromising the execution time. in [8] an ga is used to adjust the parameters of a fire propagation model that considers elements such as wind intensity and direction, and homogeneous and heterogeneous vegetation scenarios based on historical data in a given area, being able to reproduce good results in different experiments. finally, a parallel ga with variable neighborhood and multiple subpopulations is proposed in [13] aiming to adjust the model parameters to optimize firefighting routes and minimize damage resulting from simulated fire propagation. 3 wildfire spreading model based on cellular automata in the present paper, we reproduce the fire propagation model based on cellular automata proposed in [8]. it uses a two-dimensional probabilistic cellular automaton composed of a lattice of cells, where each cell can assume one of two states (tree or fire), and a stochastic transition rule that controls the evolution of the ca, that is, it defines the state of the central cell at the next time step based in its current state and that of its neighbors. a matrix w (\(3 \times 3\)) is used to quantify the probability of a given cell igniting through the propagation of fire from its closest fire cells in a moore’s neighborhood with radius 1. this matrix represents the wind influence, that is, each of its 8-outer positions represents a wind direction according to the cartesian plane (northwest, north, northeast, east, southeast, south, southwest and west) and indicates the impact of that neighbor cell on the state of central cell. the matrix positions have a real value between [0,1], whose composition determines the direction, intensity and speed of the flames in the simulation. ca rule also represents characteristics of the vegetation, such as burning time (lq) and recovery time (lr), representing the time that a cell will remain in the fire state and the time required for a burned cell to be susceptible to burning again, respectively. burning cells go through the burning stages \(s_f \in \left\{ 1... lq \right\} \), while burned cells go through the recovery stages \(s_t \in \left\{ 1... lr \right\} \), before becoming susceptible to new fires. this evolution dynamic follows ca transition rules: (i) tree cells start with the maximum value of lr; (ii) a fire cell evolves incrementally through different stages until it reaches the maximum value lq; (iii) upon reaching the value lq, a fire cell transforms into a tree cell in stage 1; (iv) a tree cell in stage 1 goes through lr stages of recovery; and (v) only cells with maximum lr have a probability \(p = \frac{age}{lr} \times \frac{\overline{iw_{fire}}}{lq}\) to burn again, where \({\overline{iw_{fire}}}\) is the average of the product between the burning stage and the wind influence of each neighbor in fire state. the implementation to reproduce the reference model [8] was carried out considering a initial lattice \(100 \times 100\), where the central cell is chosen as the initial fire spot and all the other cells start in the tree state. the wind matrix used is \(w_{3\,\times \,3}\) = {{0.14, 0.50, 0.85}, {0.00, 0.00, 1.00}, {0.14, 0.50, 0.85}}, what characterizes the wind blowing to west and a force vector shown in fig. 1a. model parameters lq and lr were adjusted in preliminary experiments to approximate the behavior presented in the original model. the values employed are \(lq=3\) and \(lr=15\). the temporal evolution of the ca lattice using the reproduced model is presented in fig. 1b over 80 time steps. four snapshots of the simulation are presented in the figure: t = {10, 30, 50, 80}. fig. 1. simulation using our reproduction of the model in [8] full size image 4 proposed evolutionary approach we investigate an evolutionary approach to tuning the parameters for the ca-based fire spreading model previously described to produce a simulation that closely matches the one observed in the reference dataset related to the propagation of flames from a forest fire in a given area. we used artificial data generated by fire simulations as reference dataset. the initial experiments use the same model to generate these artificial datasets. in the subsequent experiment, a different ca-based spreading model [20] was used to generate the reference data. for both cases, we evaluate the ability of the ga to specify an adequate set of parameters to reproduce the general behavior observed in the images captured of the area during the evolution of a fire. the proposed genetic algorithm operates over 100 generations, with a uniformly generated population of 100 individuals, where each individual is a possible configuration for the ca-based model composed of 11 genes, as shown in fig. 2a. the first 8 genes have real values between \(\left[ 0, 1 \right] \) representing the wind matrix w that makes up the moore neighborhood (blue). the central cell (gray) is always considered zero, so it is not included in the chromosome. the next two genes are integer values representing the burn value lq with values between \(\left[ 0, 10 \right] \), the recovery value lr varying between \(\left[ 0, 100 \right] \). the latest parameter sr is also an integer belonging to the interval \(\left[ 0, 10 \right] \) representing the sampling rate. given a reference dataset composed by n fire spreading snapshots. this samples can be generated by other fire spreading models or even extracted from a sequence of images obtained in a real-world fire propagation. the number of time steps t needed in the simulation model that will be used to reproduce the behavior present in the reference dataset is given by \(sr \times n\). a set of t lattices will be generated from the temporal evolution of the ca model using the others parameters coded in the individual of the population. sr parameter represents the interval of time steps that will be applied to sampling the set of lattices used to evaluate the similarity of the simulation in relation to the fire propagation snapshots present in reference dataset. for example, if sr is 5 and the reference dataset has n = 10 snapshots of a wildfire propagation, the fitness set used to evaluate the individual will be composed by the ca lattices in the time steps in {5, 10, 15, 20, ..., 50} (t = 50). the quality of the ga individual is measured by the difference between the snapshots of the reference dataset and the lattices present in the fitness set. sr parameter is related to the major investigation of the present paper. it was proposed in this work to make the ga more generalist, being able to adapt to a reference data generated by other fire spreading models or from a sequence of real-world images. parent selection for the next generation is done by simple tournament selection with \(tour=3\), and offspring are generated by double-point crossover with a 90% rate. in mutation, each gene of the chromosome has a 20% probability of undergoing alteration. for real values, there is an increment or decrement between \(\left[ -0.2, 0.2\right] \), while integer values are incremented or decremented in 1. elitism reinsertion method was applied with a 25% rate, where the top 25 parents are retained, and the rest of the population is filled by the top 75 offspring. for the fitness calculation \(fit9p = \frac{\sum _{q=0}^{8}\sum _{i=1}^{n}\left| ind_{iq}-ref_{iq} \right| }{9}\), the lattice with a dimension of \(l = 100 \times 100\) is divided into 9 partitions \(q_i\), where i varies from \(\left[ 0, 8\right] \), as shown in fig. 2b, where \(ind_{iq}\) refers to the value of the simulation using the individual parameters under analysis in each partition q at time step i, and \(ref_{iq}\) is its reference value. five simulations of the model for each initial focus (24, 24), (24, 74), (74, 24), (74, 74) are performed using the parameters defined in the genes of each individual of the population over t time steps and the average value of the executions in each partition is used in the fitness function . these values were chosen based on preliminary experiments to reduce stochastic error and not render the experiments unfeasible due to high simulation time. fig. 2. chromosome and lattice division used in the ga fitness computation. full size image 5 experiments and results experiments were carried out to evaluate the effectiveness of the proposed approach in adjusting the parameters of the fire propagation modelfootnote 1 and the ability to find an adequate sampling rate in order to reproduce the general behavior of the flames propagation present in different reference datasets (ds). three criteria were considered for analyzing the results produced by the ga with respect to the ability to: (i) configure a wind intensity and direction similar to that observed in the dataset; (ii) reproduce the fire spreading of the reference data; and (iii) find an adequate sampling rate to simulate the flames in the ds. for each experiment, 30 runs of the genetic algorithm were performed. the results were ranked by fitness, and the best solution found was applied in the graphical plotting of the temporal evolution. then, a heat map was generated to verify the frequency that an area igniting, allowing the observation of the average behavior of the fire spreading by counting the occurrence of fire in a cell over 100 simulations. 5.1 sensitivity analysis aiming evaluate the algorithm’s ability to calibrate the parameters and its adaptability to a given input, sensitivity tests were conducted varying the size of the dataset. the objective was to determine the minimum size necessary for the algorithm’s convergence without losing the quality of the adjustment. this approach makes the algorithm more adaptable to information sources and optimizes computational time by being able to calibrate the model with fewer records. initially, the cellular automaton described in sect. 3 was reproduced. it was executed 100 times using the parameters of the matrix w corresponding to the wind to the west, \(lq=3\) and \(lr=15\) over 50 time steps. the average value for each time step was computed considering the 100 ca simulations, and reference databases with different sizes were created. the first database stores the average values related to all the 50 time steps. the second database is composed by only 25 average values, where it was stored just the even time steps {2, 4, 6, ..., 50}. in a similar way, other four databases were built to store 16, 10, 7 and 5 average values. thus, six reference datasets (ds) were built using distinct sampling period \(p = \left\{ 1, 2, 3, 5, 7, 10 \right\} \), resulting in different dataset size \(s = \left\{ 50, 25, 16, 10, 7, 5 \right\} \), respectively. these reference datasets were called dataset i or dsi, where i indicates the ith element of p and s. although these datasets characterize the same wildfire scenario, since they were built based on the same group of 100 ca model simulations, they represent the data of a same phenomenon with different sampling frequency. generalizing to a real-world wildfire, propagation images are stored with different capture frequency. the wind matrix (w) in fig. 2a produces a resultant vector shown in fig. 1a, which represents the wind intensity (\(i=2\)) and west direction (\(\theta =180^{\circ }\)). it is obtained through vector decomposition, where the central cell \(w_{2\,\times \,2}\) represents the origin of the cartesian plane and each matrix cell \(w_{ij}\) is decomposed in vertical and horizontal components, which are used to generate the resultant vector of the matrix. more details on this process are provided in [8]. even if the exact values of the matrix parameters are not found, a good reproduction of the wind behavior can be achieved from a configuration capable of producing a similar resulting vector, that is, with nearby wind intensity and direction. fig. 3. wind preference matrices full size image genetic algorithm was applied to optimize the ca model parameters for the datasets used as reference (ds1, ds2, ds3, ds4, ds5 and ds6). thirty ga runs were carried out for each dataset. in table 1, the reference parameters used in the ca-based model simulations for the generation of the original dataset with 50 lattices are presented in the column named “reference”. the best solutions found by ga for each dataset are presented in the other columns. the genes \(g_{nw}\) to \( g_{w}\) that compose the matrix w for each experiment are shown. these values are used to produce the resultant vector with its intensity and inclination (fig. 3). the values for the burning time (lq), recovery time (lr) and sampling rate (sr) parameters are also presented, as well as the fitness achieved by the best ga individual in each dataset. table 1. reference and best parameter sets found by ga for each dataset.full size table as can be observed, ga achieved the exact values of the lr, lq and sr parameters for all datasets. it is important to note that the fitness values achieved for the best individual in each experiment are not comparable among the experiments d1 to ds6, since the values computation depends on the number of records belonging in each database. considering the ds1 experiment, the largest dataset with 50 records was used, so the ga accesses all the spreading information (50 lattices), making it the easier case of adjustment. the value of sr equal to 1 was found, and the matrix w produced values close to the original ones, as shown in fig. 3a, with an intensity vector of 1.87 and an inclination of 179.04\(^{\circ }\), which is close to the resultant vector related to the original matrix w (fig. 1a). the samples of the temporal evolution chosen for graphical plotting were \(t = \left\{ 7, 20, 35, 50 \right\} \). figure 4a presents the temporal evolution of both (i) the reference ds1 (the top plot); and (ii) the simulation based on the parameters of the best individual evolved by ga (the bottom plot). for all the others temporal evolution presented in fig. 4, the top plot was done based on the dataset used as reference, and the bottom plot was generated from the simulation using the best ga individual evolved for this dataset. the observed behavior in fig. 4a was appropriate, with burn areas very similar for both the main and secondary waves. besides, this observation can be generalized for several simulations as shown in the heat map in fig. 5a, with a slight tendency to evolve northwest due to the degree of difference in the vector, but nothing very significant. fig. 4. cellular automata temporal evolution full size image in the ds2 experiment, half of the records (25) were used, so the ga has less information about the spread of the fire sampling the phenomenon at each two time steps providing a greater difficulty to ga finding a good parameter adjustment. despite this, it was able to determine that the most suitable sr is 2. the ga also achieved good results regarding the matrix w, as shown in fig. 3b, with an intensity vector of 2.01 and an inclination of 180.8\(^{\circ }\), the best so far compared to the reference. the chosen samples for graphical plotting were \(t = \left\{ 4, 10, 18, 25 \right\} \). the simulation using the best individual presented an appropriate temporal evolution in fig. 4b, when compared to the data behavior, especially in the main wave, while the second wave exists but is slightly smaller than the reference. considering the heat map in fig. 5b, the average behavior was quite similar for both the outer and inner waves, although in the latter, the trace is slightly weaker than the reference, but the size is very close. considering the remaining experiments (ds3, ds4, ds5 and ds6), the best individual found for each dataset reference determines the most suitable sr: 3, 5, 7 and 10, respectively. regarding the matrix w, the values are appropriate, producing resultant vectors similar to the reference as shown in fig. 3, although the experiments ds3, ds4 and ds5 showed a more pronounced inclination to the southwest (6.44\(^{\circ }\), 7.27\(^{\circ }\) and 4.01\(^{\circ }\), respectively). this difference led to a smooth fire propagation in this direction, as shown in fig. 4 for these experiments. overall, despite the inclination, the behavior observed in experiments ds3 and ds4 in figure was appropriated. considering ds5 experiment, which reference dataset has 7 records, the temporal evolution in fig. 4e shows that the simulation with the best evolved individual, was able to reproduce the main propagation wave, but the second wave is non-existent. the average behavior observed in fig. 5 for experiments ds3, ds4 and ds5 was adequate. ds6 was the experiment with the smallest base containing only 5 records. although a good configuration was found for the matrix w, in the temporal evolution presented in fig. 4f, it can be noted that the internal waves were quite apparent, even more intense than the reference. in the average behavior shown by the heatmap in fig. 5f, values with a similar behavior to the external wave are observed, but the internal wave is more subdued than the previous experiments, showing that the temporal evolution did not reflect the average behavior adequately. fig. 5. temporal evolution heat map full size image in general, the experiments shown the ga was able to find good solutions and the most appropriate sr, provided that the base with the smallest sizes of records (ds5 and ds6) did not yield good results as the others, with no internal waves or being very intense. 5.2 robustness analysis in order to validate the robustness and adaptability of the proposed approach, due to the lack of real-world wildfire images, we employed a second ca-based fire propagation model proposed in [20] to generate artificial datasets. this model has other parameters than the model discussed in sect. 3, such as humidity and several burning stages, as well as its ca time steps represents a fire evolution time different from the ca-based propagation discussed previously. due to the distinct characteristics between the models, the best values for the model’s parameters are not known, what approximates to the problem of reproducing real-world wildfires dynamics with a simulation model. the performance of our approach will be evaluated based on the comparison between the fire dynamics resulting from the model configured by ga and the records of the datasets. three datasets were generated with 20 records (in each one) by running the reference ca-based model with different sampling rates \(sr~\in ~\left\{ 5, 7, 10 \right\} \), being named dataset 7 (ds7, sr=5), dataset 8 (ds8, sr=7) and dataset 9 (ds9, sr=9). these sampling rates were chosen because they are more challenger for the genetic algorithm search, as it was observed in the experiments in the last section. to validate the greater accuracy of the proposed approach, the artificial datasets were submitted to adjust the parameters of the ca-based model using two different versions of the genetic algorithm. the first one does not use the parameter sr in the individual coding, while the second ga version uses the proposed parameter in the genetic code as presented in sect. 4. the results of each ga approach are presented and analyzed to highlight the effect of including sr parameter. for each ga version and each dataset, 30 runs were carried out and they were analyzed using the same approach used in the previous experiments ds1 to ds6. the new experiments with database 7 were named ga-ds7, where the ga without the parameter sr was used, and ga-sr-ds7, where the proposed parameter sr is adjusted join with the other parameters in the individual. the same was made for dataset 8 and dataset 9 resulting in experiments named ga-ds8, ga-ds9, ga-sr-ds8, ga-sr-ds9. for the plot, the same time steps (\(t = \left\{ 5, 10, 15, 20 \right\} \)) are presented. table 2. robustness testfull size table figure 6 presents the dynamics of fires obtained from ga and ga-sr for different sampling rates, showing the temporal evolution of burned (red), recovering (gray) and recovered (green) areas. considering the experiment ga-ds7, table 2 shows that the best individual evolved by the simplest ga has \(lq=10\) (maximum possible value) and a \(lr=91\), producing a temporal evolution with a smaller burning area as shown in fig. 6a. on the other hand, in the experiment ga-sr-ds7 the best evolved individual has \(lr=10\) and \(lr=33\), which results in a bigger burning area, which is closer to the reference, especially at \(t=20\). it is important to note that there is a recovery process in the model [20] that was not captured by ga, but the ga-sr was able to find a similar recovery using \(sr=3\). considering the average behavior shown in fig. 6d it is highlighted that ga-sr-ds7 presented a closer dynamics to the reference. regarding the experiment ga-ds8, table 2 shows that the best individual evolved by the simplest ga has \(lq=10\) and a \(lr=57\). due to the fact that the sampling period is longer, the temporal evolution of the ga-ds8 in fig. 6b was not adequate because the burning area is significantly smaller than the reference. on the other hand, ga-sr-ds8 was able to find a better adaptation, by increasing the value of sr to 4, with \(lq=10\) and \(lr=23\). thus the temporal evolution was similar to the reference both in the burning and recovery areas. in the average behavior shown in fig. 6e this difference between ga-ds8 and ga-sr-ds8 are highlighted. we believe that the difference is very significant because with a higher sampling rate, the ga managed to get closer to the solution. finally, in the experiment ga-ds9, table 2 shows that the best individual evolved by the simplest ga has \(lq=9\) and a \(lr=28\), producing a very different temporal evolution in fig. 6a when compared to the reference. on the other hand, ga-sr-ds9 defined \(lr=10\), \(lr=37\) and \(sr=5\) in the best individual, better reproducing the dynamics adequately with a burning area (red), recovering (grey) and recovered (green) very similar. the average behavior in fig. 6f makes clear the much better adaptability of ga-sr over a ga approach without the sample rate. fig. 6. ga and ga-sr cellular automata temporal evolution. (color figure online) full size image experiments have shown that the proposed approach can replicate the behavior of other fire propagation models with different parameters, as well as adapt satisfactorily to the data dynamics generated from different sampling rates. 6 conclusions our main goal is to propose an evolutionary approach capable of automatically adjusting the parameters of a ca-based model, so that it is capable of making a good reproduction of the dynamics observed in the wildfire spreading. the phenomenon to be represented can be either a sequence of lattices generated by another fire propagation model or a sequence of digitized images of a real fire (reference dataset). we establish that it is relevant in this investigation that this evolutionary adjustment of parameters can adequately reproduce the dynamics regardless of the sampling rate used to generate the reference dataset. in this sense, a new parameter (sampling rate) was adjusted by the ga, which determines which lattices of a ca simulation will be used to assess the behavior similarity in relation to the reference data. experimental results show that our approach was able to find adequate values for this sampling rate and the other model parameters, resulting in fire dynamics close to those observed in the datasets. comparing with the approach without sampling rate, it was found that the presence of this parameter provides more versatility and adaptability to the ga, especially when reproducing sets with sparser data. we also analyzed the influence of the dataset size on the quality of the simulations generated using ga, where datasets with different sizes and periodicity between data were tested. it was possible to observe a reduction in ga efficiency for small datasets (size < 10), with 20 records being sufficient for the reproduction of the fire behavior even for sparser datasets. as future work, we intend to evaluate new fitness functions that consider more ca states and investigate the adjustment of more complex fire models, which have more parameters and are consequently more challenging for the ga. additionally, we also aim to evaluate the effectiveness of our approach in reproducing real wildfire propagation dynamics. notes 1.the source code and dataset used in the experiments are available in the repository https://github.com/lucasmurilo/evol-adj-of-a-ca-based-model-for-wf-sd.git. references alexandridis, a., et al.: a cellular automata model for forest fire spread prediction: the case of the wildfire that swept through spetses island in 1990. appl. math. comput. 204(1), 191–201 (2008) mathscinet  math  google scholar  alhijawi, b., awajan, a.: genetic algorithms: theory, genetic operators, solutions, and applications. evol. intel. 17(3), 1245–1256 (2024) article  math  google scholar  amol, d.r.: prediction of fire propagation in forest areas using genetic algorithm. in: itm web of conferences. vol. 32, p. 03046. edp sciences (2020) google scholar  brasiel, h.c., lima, d.a.: exploring the influence of wind, vegetation and water sources on the spread of forest fires in the brazilian cerrado biome using cellular automata. in: wcama, pp. 61–70 (2023) google scholar  carrillo, c., margalef, t., espinosa, a., cortés, a.: impact of mixed-precision: a way to accelerate data-driven forest fire spread systems. in: international conference on computational science, pp. 62–76 (2023) google scholar  carta, f., zidda, c., putzu, m., loru, d., anedda, m., giusto, d.: advancements in forest fire prevention: a comprehensive survey. sensors 23(14), 6635 (2023) article  google scholar  chopard, b., droz, m.: cellular automata modeling of physical systems (1998) google scholar  ferreira, m.e.a., quinta, a.l., lima, d.a., martins, l.g., oliveira, g.m.: automatic evolutionary adjustment of cellular automata model for forest fire propagation. in: international conference on cellular automata for research and industry, pp. 235–245. springer (2022) google scholar  fraga, l.m., de oliveira, g.m., martins, l.g.: adjustment of an epidemiological cellular automata-based model using genetic algorithm. in: ieee 32nd international conference on tools with artificial intelligence, pp. 589–594 (2020) google scholar  ghisu, t., et al.: an improved cellular automata for wildfire spread. procedia comput. sci. 51, 2287–2296 (2015) article  math  google scholar  goldberg, d.: genetic algorithms in search optimization, machine learning (1989) google scholar  gomes jr, m.n., giacomoni, m.h., navarro, f.a., mendiondo, e.m.: global optimization-based calibration algorithm for a 2d distributed hydrologic-hydrodynamic and water quality model. arxiv:2308.16864 (2023) li, x., chen, n., ma, h., nie, f., wang, x.: a parallel genetic algorithm with variable neighborhood search for the vehicle routing problem in forest fire-fighting. ieee transactions on intelligent transportation systems (2024) google scholar  maccarthy, j., richtera, j., tyukavina, s., weisse, m., harris, n.: the latest data confirms: forest fires are getting worse (2023) google scholar  miao, x., et al.: time series forest fire prediction based on improved transformer. forests 14(8), 1596 (2023) article  math  google scholar  monteiro, l.h.a., gandini, d., schimit, p.h.: the influence of immune individuals in disease spread evaluated by cellular automaton and genetic algorithm. comput. methods programs biomed. 196, 105707 (2020) article  math  google scholar  pavlova, a., rubtsov, s., telyatnikov, i.: using cellular automata in modelling of the fire front propagation through rough terrain. in: iop conference series: earth and environmental science. vol. 579, p. 012104. iop publishing (2020) google scholar  shen, x., guo, y., li, a.: cooperative coevolution with an improved resource allocation for large-scale multi-objective software project scheduling. appl. soft comput. 88, 106059 (2020) article  math  google scholar  sun, l., et al.: adaptive forest fire spread simulation algorithm based on cellular automata. forests 12, 1431 (2021) article  math  google scholar  tinoco, c.r., ferreira, h.f., martins, l.g., oliveira, g.m.: wildfire simulation model based on cellular automata and stochastic rules. in: international conference on cellular automata for research and industry, pp. 246–256 (2022) google scholar  tirumala, s.s.: evolving deep neural networks using coevolutionary algorithms with multi-population strategy. neural comp. app. 32, 13051–13064 (2020) article  math  google scholar  wolfram, s.: a new kind of science 2002 (2002) google scholar  xuehua, w., et al.: a cellular automata model for forest fire spreading simulation. in: ieee symposium series on computational intelligence, pp. 1–6 (2016) google scholar  yassemi, s., dragićević, s., schmidt, m.: design and implementation of an integrated gis-based cellular automata model to characterize forest fire behaviour. ecol. model. 210(1–2), 71–84 (2008) article  google scholar  yongzhong, z., et al.: simulating wildfire spreading processes in a spatially heterogeneous landscapes using an improved cellular automaton model. in: ieee international geoscience and remote sensing symposium. vol. 5, pp. 3371–3374 (2004) google scholar  download references acknowledgments authors thank the support from cnpq-proc. 423105/2021-3. author information authors and affiliations faculty of computing, federal university of uberlandia, uberlandia, mg, brazil lucas v. murilo, gina m. b. oliveira & luiz g. a. martins authors lucas v. muriloview author publications search author on:pubmed google scholar gina m. b. oliveiraview author publications search author on:pubmed google scholar luiz g. a. martinsview author publications search author on:pubmed google scholar corresponding author correspondence to lucas v. murilo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper murilo, l.v., oliveira, g.m.b., martins, l.g.a. (2025). evolutionary adjustment of a cellular automata-based model for wildfire spreading. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_18 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords genetic algorithms cellular automata evolutionary adjustment wildfire propagation model publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature fitness landscape analysis of tpot using local optima network | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper fitness landscape analysis of tpot using local optima network conference paper first online: 12 october 2023 pp 65–79 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) fitness landscape analysis of tpot using local optima network download book pdf download book epub matheus cândido teixeira  orcid: orcid.org/0000-0003-1103-53839 & gisele lobo pappa  orcid: orcid.org/0000-0002-0349-449410  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 535 accesses 1 citation abstract automl addresses the challenge of automatically configuring machine learning pipelines for specific data analysis tasks. these pipelines encompass techniques for preprocessing and classifying data. numerous approaches exist for discovering the optimal pipeline configuration, with most focusing on optimization methods such as bayesian optimization and evolutionary algorithms. nevertheless, limited knowledge exists regarding the structure of the search space that these methods operate within. what is certain is that these spaces incorporate categorical, continuous, and conditional hyperparameters, and effectively handling them is not straightforward. to shed light on this matter, the present study conducts an examination of automl search spaces generated by the tree-based pipeline optimization tool (tpot) algorithm utilizing local optimal networks (lon). the goal is to gain deeper insights into the overall characteristics of the search space, enhancing comprehension of the search strategy employed and the algorithm’s limitations. this investigation aids in understanding the search strategy and constraints of the algorithm, ultimately contributing to the advancement of novel optimization algorithms or the refinement of existing ones within the scientific literature. the findings have implications for enhancing optimization algorithms by illuminating how the search space is explored and the consequent impact on the discovered solutions. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others analysis of neutrality of automl search spaces with local optima networks chapter © 2022 incremental search space construction for machine learning pipeline synthesis chapter © 2021 a hierarchical dissimilarity metric for automated machine learning pipelines, and visualizing search behaviour chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. continuous optimization lead optimization learning algorithms machine learning optimization bioinformatics 1 introduction in the last decade, automated machine learning (automl) has experienced significant advancements in the development of algorithms and strategies for generating tailored machine learning pipelines for specific problems [8]. a machine learning pipeline consists of a sequence or parallel execution of operations, including data preprocessing, classification, and post-processing. while classification is commonly used, other machine learning tasks like regression or clustering can also be accommodated. these methods aim to improve accuracy while considering computational constraints, as evaluating these pipelines requires substantial computational resources. major cloud platforms, such as google, microsoft, and amazon, have incorporated automl methods into their offerings [4]. the generation of these pipelines primarily relies on bayesian optimization, evolutionary computation, or hybrid techniques [8]. in recent years, the focus has shifted towards optimizing the hyperparameters of artificial neural networks, a process known as neural architecture search (nas) [3]. however, little is known about the structure of the search space in both automl and nas domains, which could provide insights into more effective and efficient exploration methods. given the success of evolutionary computation in exploring these pipeline spaces, which started with basic methods like tpot [12] and recipe [16] and progressed to more advanced approaches like cma-es [7], it is natural to investigate the fitness landscape that these methods traverse. the fitness landscape of a problem is defined by three components: the set of valid solutions (\(\mathcal {x}\)), the neighborhood of a solution (\(\mathcal {n}\)), and a fitness function (\(f: \mathcal {x}\rightarrow \mathbb {r}\)) that evaluates the solutions in \(\mathcal {x}\). by assessing solutions using f and considering the notion of neighborhood, the fitness landscape can be characterized. previous research has explored the fitness landscape of automl problems using local optimal networks (lon) [10], which offers a global perspective compared to traditional metrics such as fdc. lons provide a comprehensive view of the fitness landscape, while fdc focuses on local features of the search space. however, previous work employed an artificial search space generated using brute force, with a fixed number of neighbors for each solution, which is unrealistic since many algorithms employ efficient strategies to explore the search space. to address this limitation, we construct the search space using tpot, an automl algorithm. essentially, our grammar and search space represent 69,960 valid machine learning pipelines that comprise a classification algorithm and potentially a preprocessing algorithmfootnote 1. although our grammar is smaller compared to the default tpot configuration space, which can represent an enormous number of different ml pipelines, this simplification allows us to evaluate every pipeline represented by our grammar and determine the global optimum in the search space. the results reveal that in many cases, tpot fails to optimize the problem, frequently evaluating suboptimal solutions instead of the global optimum in numerous datasets, indicating tpot’s struggle to optimize in certain search spaces and its tendency to get trapped in local optima. this work contributes to the field by constructing a new search space for automl problems using a real optimization algorithm (tpot), proposing a novel approach for assigning weights to edges in constructing the local optima network (lon), and conducting an analysis of the constructed lons. the remainder of this paper is structured as follows: sect. 2 reviews related work, sect. 3 presents our methodology, including details about the grammar, tpot configuration, search space construction, and lon generation. section 4 describes our experimental setup, including the datasets used, statistical information, and hardware specifications. section 5 presents the results and ensuing discussion. finally, sect. 6 concludes the paper. 2 related work few works have looked so far at the fitness landscape of automl problems. garciarena et al. [6] were the first to perform the analysis of automl landscapes considering a subspace of tpot. their objective was to identify the local characteristics of the space close to optimal solutions by using metrics such as slope, roughness and neutrality. their results suggest that many regions of high fitness exist in the space, but these are prone to overfitting. in this same direction, pimenta et al. [13] looked at fitness landscape metrics to better understand the search space of a huge space of machine learning pipelines. they looked at fitness distance correlation (fdc) and metrics of neutrality, and concluded fdc was a poor metric for performing the analyses. turning from automl complete pipelines to analyses of loss function in neural architecture search (nas), rodrigues et al. [15] characterized fitness landscapes of meta-heuristics for neuroevolution of convolutional neural networks using autocorrelation and the entropic measure of ruggedness. nunes et al. [9] also analyzed the fitness landscape of nas in the context of graph neural network architectures. they used fdc together with the dispersion metric (which measures the dispersion between the funnels in the landscape), and have also looked at the neutrality of the space. in the realm of automl problems, there has been ongoing research on the fitness landscape. pushak et al. [14] conducted an assessment of the landscape of algorithm configuration, specifically focusing on the modality and convexity of parameter responses. their approach involved defining parameter response slices, where a specific parameter, denoted as p, was varied within a defined window centered around an optimum solution identified by smac. by keeping all other parameters constant and measuring the algorithm’s performance as a function of p, they evaluated various algorithms for typical optimization problems. the findings revealed that many of the parameter slices exhibited unimodal and convex characteristics, both within instance sets and on individual instances. one of the few works analyzing fitness landscapes based on lons and related to our research is presented in [17]. the authors adapted lons to analyze the overall structure of parameter configuration spaces. they examined the metrics derived from lons and fitness distance correlation (fdc), observing significant discrepancies when tuning the same algorithm for different problem instances. notably, in complex scenarios, a substantial number of sub-optimal funnels were identified, while simpler problems displayed a single global funnel. similarly, cleghorn et al. [2] investigated parameter spaces for particle swarm intelligence (pso) with a similar objective. their analysis unveiled that the macro-level view of pso’s parameter landscapes appeared relatively simple, yet the micro-level view revealed a much higher level of complexity. this complexity posed challenges for parameter tuning, exceeding the initial assumptions. 3 methodology the problem of automl can be formally defined as a generalization of the combined algorithm selection and hyperparameter optimization (cash) problem [5]. in its original definition, given a set \(\mathcal {a} = \{ a^{(1)}, a^{(2)}, \ldots , a^{(k)} \}\) of learning algorithms, where each algorithm \(a^{(j)}\) has a hyperparameter space \(\varlambda ^{(j)} = \{ \lambda ^{(1)}, ..., \lambda ^{(s)}\}\), defined from the full set of algorithm’s hyper-parameters \(\mathbf { \omega }\), the cash problem is defined as in eq. 1footnote 2. $$\begin{aligned} a^*_{\lambda ^*} = \underset{a^{(j)}\in \mathcal {a}, \lambda \in \varlambda ^{(i)}}{\textrm{argmax}} \frac{1}{k} \sum _{i=1}^{k}\mathcal {f}\left( \mathcal {a}_{\lambda }^{(j)},\ \mathcal {d}_{train}^{(i)}, \mathcal {d}_{valid}^{(i)}\right) \end{aligned}$$ (1) where \(\mathcal {f}(a_\lambda ^{(j)}, \mathcal {d}_{train}^{(i)}, \mathcal {d}_{valid}^{(i)})\) is the gain achieved when a learning algorithm a, with hyperparameters \(\varlambda \), is trained and validated on disjoint training and validation sets \(\mathcal {d}_{train}^{(i)}\) and \(\mathcal {d}_{valid}^{(i)}\), respectively, on each partition \(1 \le i \le k\) of a k-fold cross-validation procedure. a generalization can be made if we replace \(\mathcal {a}\) by a set of pipelines \(\mathcal {p} = \{ {p}^{(1)}, ..., {p}^{(v)}\}\), which includes a subset of algorithms from \(\mathcal {a}\) and their respective set of hyperparameters \({{\vargamma ^{(i)}}} = \{ {{ \varlambda ^{(1)}}}, ..., {{\varlambda ^{(s)}}\}}\), represented by the full set \(\mathbf {\psi }\), as defined in eq. 2 $$\begin{aligned} {\mathbf {p^{*}}}_{\mathbf {\gamma ^{*}}} = \underset{\mathcal {{p}}^{(i)} \subseteq {\textbf{p}}, {\mathbf {\gamma ^{(i)}}} \subseteq {\mathbf {\psi }}}{\textrm{argmax}} \frac{1}{k} \cdot \sum _{j=1}^{k} \mathcal {f}({\mathbf {p^{(i)}} }_{\mathbf {\gamma ^{(i)}}}, d^{(j)}_{train}, d^{(j)}_{valid}) \end{aligned}$$ (2) figure 1 illustrates a fitness landscape, where the horizontal axis represents the configuration space and the vertical axis represents the fitness of a given configuration. as the landscape fitness concept is multidisciplinary (it can be applied to several problems), it is necessary to define what the configuration space represents and how the fitness is calculated. therefore, in this article, the configurations space is formed by ml pipelines and the fitness was defined as the f1-score of the pipeline. fig. 1. illustration of a simple fitness landscape full size image fig. 2. examples of pipelines full size image ml pipelines can be represented through a tree as illustrated in fig. 2. each configuration (represented by the horizontal axis) in fig. 1 corresponds to a pipeline according to the examples illustrated in fig. 2. pipelines are built according to a context-free grammar that defines that every grammatically correct pipeline has 0 or 1 preprocessing algorithms and a classification algorithm. in the figure, the leaf nodes correspond to the algorithms and their respective hyperparameters. although some algorithms have continuous hyperparameters, a set of discrete values have been selected and can be consulted through grammarfootnote 3. 3.1 tpot: tree-based pipeline optimization tool tpot [12] is an automatic optimization tool for ml pipelines (ie, automl) that uses genetic programming (gp) as optimization heuristics. gp, as well as several algorithms based on the theory of evolution, use crossover, mutation, recombination and selection operators to evolve individuals. in the case of tpot, individuals are ml pipelines, whose algorithms and hyperparameters are defined through a configuration file. 3.2 construction of the fitness landscape to analyze how tpot explores the space, we first select a relatively small configuration space composed of 4 options of preprocessing algorithms (one option is not to use a preprocessing algorithm) and 5 classification algorithms. each algorithm has a set of hyperparameters and, considering algorithms and hyperparameters, it is possible to form about 70,000 different pipelines. each pipeline is made up of a classification algorithm and (optionally) a preprocessing algorithm. we evaluate the fitness (f1-score) of each of these solutions. then, we run the tpot in that same space to be able to analyze how the algorithm explores the space, that is, if the tpot, i.e. the gp, is an algorithm capable of effectively optimizing the automl problem. we run tpot 30 times on each dataset using a different seed for statistical purposes. to build the fitness landscape, it is necessary to define the concept of neighborhood. we define the neighborhood through 3 different types of operators: mutation, recombination and reproduction. when a solution u is mutated and generates another solution v, we say that v is a neighbor of u, that is, \(v \in \mathcal {n}(u)\). the same process occurs in the case of recombination. as for the crossover, the process is different. two solutions \(u_1\) and \(u_2\) are selected for reproduction and a solution v is generated as a product of this operation. figure 3 illustrates the process. fig. 3. edges full size image 3.3 compressing neutral nodes the lon – shown below – is affected by the number of local optima in space. by definition, a locally optimal solution is a solution whose fitness is greater than that of its neighbors, therefore, a local optimum can be verified by comparing its fitness with that of its neighbors or through a local search algorithm like hill climbing, where the local optimum is the solution where the algorithm does not find any other optimization points. regardless of the method used, both methods should achieve the same result, however, after some experiments, a divergence was observed in the nodes identified as local optima by both methods. after analyses, it was identified that the order in which the local search algorithm visits neutral regions of the fitness landscape can generate a greater amount of optima because the first solution visited in the neutral region is reported as an lo – there is no solution with fitness greater than current. although dealing with the hill climbing acceptance criteria seems to be a solution to the problem – seeking to advance in a solution with fitness greater than or equal to the current one – the problem still persists if there is no solution with fitness greater than the one already found in the region neutral. therefore, the solution adopted in this work was to compress neutral solutions into a single node and insert an attribute that counts the number of merged nodes for later analysis purposes. the strategy used to find neutral solutions was to use breadth-first search (bfs) to enter neutral regions through the neighborhood of a solution. after this adjustment, the graph with clustered neutral regions is used to construct the lon. 3.4 local optima networks (lon) a lon is represented by a directed graph lon = (v,e), where v are the vertices that represent the local optima of the fitness landscape and e are the edges that connect the vertices, which can represent basin-transition, escape or permutation edges, as detailed below. by extracting a set of features from this network, we obtain a view of the global structure of the fitness landscape of a problem. here, we use the original lon model [11] to perform our analysis, along with other models more adapted to neutral spaces, such as compressed lon (clon), monotonic lon (mlon), and compressed monotonic lon (cmlon) [1, 10]. 3.5 nodes a local optimum is defined as a solution x such that \(\forall x' \in \mathcal {n}(x),\, f(x) \ge f(x')\), where \(\mathcal { n }(\cdot )\) is the neighborhood operator and \(f(\cdot )\) is the fitness function. the local optima are defined by a local search algorithm. in this work, the local search looks for solutions with fitness greater than or equal to the current fitness. note that the f-measure varies from 0 to 1, and some machine learning pipelines may differ only on the third or fourth decimal point. these small differences make little to no effect on the results, and hence we could say solutions with these differences in fitness are neutral. because of that, a pipeline \(p_i\) is defined as better than a pipeline \(p_j\) if the fitness value of \(p_i\) is greater than the fitness value of \(p_j + \delta \), where \(\delta \) is a specified tolerance value given by the standard deviation of the mean fitness value of 30 independent random samples. the literature defines a variety of local search algorithms to find the local optima of a lon, including iterated local search (ils) and tabu search. we have used a classical hill-climbing, as the search space is combinatorial and enumerable. 3.6 edges once the lon nodes are determined, we define whether there is an edge \(e_{ij}\) between two local optima \(lo_i\) and \(lo_j\) by defining a edge weight \(w_{ij}\). an edge exists when \(w_{ij} >0\). the literature defines at least three ways of assigning weights to lon edges  [10, 11, 18]: basin-transition edges, escape edges, and perturbation edges. each of these methods brings new information about the relationship between local optima, but basin-transition is more useful for combinatorial problems because of its probabilistic nature. a problem with this method is that it takes a long time to evaluate the weight of every edge (in the order of \(o(n^3)\)), so we propose a new methodology: common ancestor, as described below, and use the basin-transition as a baseline. basin-transition (bt): as the local search defines a mapping from the search space to a set of locally optimal solutions, it also generates basins of attraction. the basin of attraction \(b_i\) of a local optimum \(lo_i\) is composed by all solutions s in the search space that satisfy \(ls(s)=lo_i\), that is, \( b_i = \{v \in v(g)\, |\, ls(v) = lo_i\}\). therefore, in the basin transition method [11], the weight of the edge that connects two local optima \(lo_i\) and \(lo_j\) is given by: $$\begin{aligned} w(e_{ij})=\frac{1}{|b_i|}\sum _{s \in b_i}{\sum _{s' \in b_j}{p(s \rightarrow s')}} \end{aligned}$$ (3) where \(p(s \rightarrow s')\) is the probability of a mutation in s generates \(s'\). common ancestor (ca): this method assigns weights to edges proportionally to the number of common ancestors. a node v is an ancestor of u if there exists a path that connects v to u, i.e., \(u \leadsto v\). 3.7 network statistics number of nodes: indicates the total number of nodes (or vertices). number of edges: indicates the total number of edges present in the graph. number of self-loops: indicates the number of simple loops in the graph, where an edge’s destination is equal to its origin (e.g., (u, u)). number of isolated nodes: indicates the number of nodes without any incoming or outgoing edges. degree centrality: measures the importance of a node in the graph based on the number of edges incident to that node. it quantifies the node’s connectivity within the network. in-degree: the number of edges that terminate at a given vertex. it indicates the importance of a solution in the search space and identifies frequently visited solutions during exploration. out-degree: the number of edges that originate from a given vertex. it indicates the importance of a solution in the search space and identifies solutions that are challenging for the search algorithm to reach. density: measures the proportion of edges present in the graph relative to the total number of possible edges. mean clustering: the average clustering coefficient of each node in the graph, which captures the degree of interconnectedness among a node’s neighbors. total weight: the sum of the weights of all edges present in the graph. increasing weights: the sum of the weights of edges that connect nodes where the fitness of the destination node is greater than the fitness of the origin node. decreasing weights: the sum of the weights of edges that connect nodes where the fitness of the destination node is smaller than the fitness of the origin node. neutral weights: the sum of the weights of edges that connect nodes where the fitness of the destination node is equal to the fitness of the origin node. 4 experimental setup 4.1 characterization of the fitness landscapes the fitness landscape of a problem depends directly on the data being analyzed. in this work the pipelines were evaluated in 20 datasets selected from uci machine learning repositoryfootnote 4 and from kagglefootnote 5. the selection criteria were: (i) popularity, (ii) numerical or categorical features and (iii) the task intended was classification. considering the search space defined in sect. 3, we generate all the solutions and evaluated them for each of the 20 datasets, generating 20 different fitness landscapes. table 1 presents some features of the datasets used to generate the fitness landscape. the “code” column indicates the code used to reference each dataset, the “instances” column indicates the number of instance, the “features” column indicates the number of features, the “classes” column indicates the number of classes present in the target feature. following, the “#optimum” column indicates the number of solutions that achieve the value of optimal fitness. the “var.”, “mean”, “max.” and “min.” columns indicate the variance, mean, highest and lowest fitness value of the evaluated pipelines. table 1. summary of the fitness value of the pipelines evaluated in each datasetfull size table further, fig. 4 shows the box-plots of the fitness distribution of the pipelines generated for each dataset. note that, for some datasets, the fitness of the solutions is predominantly high or low, while for others they are better distributed. observe that this distribution does not affect the fla, but gives an insight on the difficulty of the problem. fig. 4. box-plot of the fitness of the pipelines in different datasets. full size image fig. 5. evaluation time for the complete search space defined by the grammar. full size image table 1 shows more detailed statistics of the fitness of the pipelines evaluated for each dataset, summarized in the box-plots of fig. 4. the variance is relatively low since the fitness (f1-score) can vary from 0.00 to 1.00. figure 5 shows the total time required for evaluating the entire configuration space. the total time is equal to the sum of the time needed to train and evaluate each solution individually. some factors that justify the variation in total time are the number of samples and the size of each dataset. for example, ds09 is the largest dataset (51 features). the experiments were run on an intel(r) xeon(r) cpu e5-2620 v2 @ 2.10 ghz and approximately 65 gb of ram. the configuration space generated by tpot is composed of pipelines that have a feature selection algorithm, a preprocessing algorithm and a classification algorithm. there are approximately \(3.94\times 10^{10}\), \(2.66\times 10^{10}\) and \(1.50\times 10^{38}\) combinations of selection, preprocessing and classification algorithms, respectively, considering the different combinations of hyperparameters. the tpot configurations are: population_size is 10, generations is 30, mutation rate is 0.9, crossover rate is 0.1 and the scoring function is f1-weighted. the experiments were run on 20 datasets for selected classification problems from the kaggle platform. 5 results our experiments aim to verify how the tpot explores the configuration space. the first experiments analyze the performance of tpot until finding the global optimum (go) and the number of times the algorithm was able to find the go. this helps to verify the difficulty faced by the algorithm in the search spaces of the experiment. as the entire space was evaluated by brute force and the space was conditioned to only a few solutions (about 70 thousand), it is possible to detect whether the algorithm found the go. the results of the fitness landscape of tpot show that some solutions are evaluated frequently, that is, even when changing the seed of the algorithm, a subset of solutions (in some cases only one) are evaluated in most executions (more than 70% of the times). the analysis of these solutions indicates that in all cases, they are not the global optimum and that the average error (average of the difference between their fitness and the global optimum) is 0.06 with a standard deviation of 0.07. also, tpot is not able to optimize the datasets in several cases (even in this small search space). for example, it is possible to verify through the column “hits (%)” of table 2 that tpot does not find the global optimum in all executions in most datasets. for example, the global optimum was only found in 16,67% of the runs in the raisin dataset, while in the bank and nursery dataset it was found in all tpot runs. however, regardless of the number of hits, in all cases, the average error (between the highest fitness found and the global optimum in space) occurs only from the third decimal place. this result indicates that the tpot was close to the global optimum in all runs, since a difference of 0.004 in the f1-score between different ml pipelines is not a significant gain or loss in most applications. table 2. summarization of 30 runs of tpot using the samefull size table table 3. performance do tpot: regarding the number of generations until finding the gofull size table table 3 presents the generation in which the tpot finds the global optimum (go). the column “gen.” indicates which generation the first go was found. the other two columns indicate the amount that was found and the amount that the dataset has. finding the go in a large generation indicates that the algorithm found some difficulty in exploring the space and this may be an indication of the difficulty of the problem. another aspect is the amount of gos found (especially in multimodal problems) in the space, since the exploration of the search space is an important feature because of the diversity of solutions, which is an important feature for automl, as it allows the choice between different algorithms. for example, in the ds08 dataset, the number of gos is 96, while the average of optimals found by tpot is only 18.87, that is, a large part of the space diversity remains unexplored. figure 6 presents the centrality, density and clustering statistics of the graphs generated by the tpot exploration. reported results are the average of 30 runs of the experiment. it is possible to observe that, when we create the lon using the ca method, the results are closer to the original graph, while the bt method presents more discrepant results. in the three metrics shown in the figure, the bt method has values greater than those calculated in the other graphs. this allows us to conclude that the bt results in a lon with greater centrality and denser than the others. fig. 6. performance of tpot in different runs on the same dataset full size image figure 7 shows several runs of the algorithm on the same dataset, but with different seeds. it is possible to verify that in the first case, the algorithm was not able to find the go and the exploration was concentrated in only one region — represented by the densest region in the figure. in the second case, the algorithm found 75% of the global optima in space and the graph has many explored regions and the concentrations are scattered around several solutions. in the third case, the algorithm was able to find all gos and the figure indicates a concentration of the exploration around some solutions (visually less than in the second case). the explanation for this is that some hyperparameters do not affect the fitness of the solutions (in certain spaces), therefore, all solutions that differ only in the values of such hyperparameters, obtain the same fitness. therefore, some solutions are close to each other due to this phenomenon. fig. 7. performance of tpot in different runs on the same dataset full size image figure 8 presents the results of various graph statistics computed on the lon ca and lon bt graphs. for comparison purposes, the value of the statistics were normalized by the maximum value obtained in each method, that is, we divided the value of each statistic by the maximum value found in datasets. it is possible to observe that for both the in-degree and the out-degree, the statistic evaluated in lon bt is smoother (and higher) than the one evaluated in lon ca. in the other statistics, both vary significantly as the datasets vary. therefore, we initially analyzed the correlation between the statistics of the two methods. fig. 8. several metrics calculated using the lon constructed using the bt and ca method. the results serve as a basis for comparison between methods. full size image considering the relationship between the lon bt and ca, it is possible to verify that the correction between the average degree of entry and exit is 0.051, that is, there is no correlation between the two types. likewise, the correlation between the weight of the edges that form loops in the lon is -0.335, that is, the correlation is equally weak. the correlation between the number of edges and the weight of the graph is 0.5653 and 0.5414, respectively. although in these last two cases the correlation is greater, it is still not possible to consider that there is a strong relationship between them. thus, it is possible to observe that the structure of the lon are relatively different from each other. in order to compare the difference between both methods, the following statistical test was performed: the correlation between the metrics of the two lons were correlated with the features of the tpot exploration graph (from table 2: hits (%), error, fitness, global/ from table 3: generation, # found, # global) and the t-testfootnote 6 was used to compare whether there is a difference between the two populations. the resulting p-value was 0.57, which allows accepting the null hypothesis that both means are equal. in this case, we consider that the equality of the mean is a measure of the similarity between the two populations. 6 conclusion the results show that tpot suffers somewhat from local optima, as indicated by the fact that suboptimal solutions remain for many generations. however, the solutions have a fitness relatively close to the global optimum (difference less than \(10\times 10^{-4}\)). another observation is the importance of repeating the experiments, as the algorithm is affected by the position where it was initialized. tpot can quickly converge to suboptimal solutions and fail to explore space. through several iterations of the algorithm, it is possible to verify and ’force’ the space exploration through different initialization points. the results also show that lon ca has structural differences in relation to lon bt. the correlations between several metrics between them are low, which suggests that the two methods create “different” lons. however, both methods produce a graph that has a similar correlation with the metrics of the original graph, with the advantage that the ca method is more efficient. it can be computed through the adjacency matrix of the graphs (cost o(\(n^2\))), while the bt method needs to calculate the combinations between pairs of adjacent nodes (cost o(\(n^3\))). notes 1.in this work, the terms machine learning pipeline, solution, and configuration are used interchangeably, referring to one or more preprocessing/classification algorithms and their parameters. 2.the original definition casts the problem as a minimization one. here we replace the loss function by a gain function. 3.link to grammar. 4.https://archive.ics.uci.edu/ml/index.php. 5.https://www.kaggle.com/datasets. 6.the same variance was not assumed for the two populations. references adair, j., ochoa, g., malan, k.m.: local optima networks for continuous fitness landscapes. in: proceedings of the genetic and evolutionary computation conference companion, pp. 1407–1414. acm, new york (jul 2019). https://doi.org/10.1145/3319619.3326852, https://dl.acm.org/doi/10.1145/3319619.3326852 cleghorn, c.w., ochoa, g.: understanding parameter spaces using local optima networks. in: proceedings of the genetic and evolutionary computation conference companion, pp. 1657–1664. acm, new york (jul 2021). https://doi.org/10.1145/3449726.3463145, https://dl.acm.org/doi/10.1145/3449726.3463145 elsken, t., metzen, j.h., hutter, f.: neural architecture search: a survey. j. mach. learn. res. 20(1), 1997–2017 (2019) mathscinet  math  google scholar  erickson, n., et al.: autogluon-tabular: robust and accurate automl for structured data. arxiv preprint arxiv:2003.06505 (2020) feurer, m., klein, a., eggensperger, k., springenberg, j., blum, m., hutter, f.: efficient and robust automated machine learning. in: advances in neural information processing systems, pp. 2962–2970 (2015) google scholar  garciarena, u., santana, r., mendiburu, a.: analysis of the complexity of the automatic pipeline generation problem. in: 2018 ieee congress on evolutionary computation (cec). pp. 1–8. ieee (jul 2018). https://doi.org/10.1109/cec.2018.8477662, https://ieeexplore.ieee.org/document/8477662/ g. shala, biedenkapp, a., n.awad, adriaensen, s., m.lindauer, hutter, f.: learning step-size adaptation in cma-es. in: proceedings of the sixteenth international conference on parallel problem solving from nature (ppsn 2020) (sep 2020) google scholar  hutter, f., kotthoff, l., vanschoren, j.: automated machine learning: methods, systems, challenges. springer nature (2019). https://doi.org/10.1007/978-3-030-05318-5 nunes, m., fraga, p.m., pappa, g.l.: fitness landscape analysis of graph neural network architecture search spaces. in: proceedings of the genetic and evolutionary computation conference, pp. 876–884. acm, new york (jun 2021). https://doi.org/10.1145/3449639.3459318, https://dl.acm.org/doi/10.1145/3449639.3459318 ochoa, g., chicano, f.: local optima network analysis for max-sat. in: proceedings of the genetic and evolutionary computation conference companion, pp. 1430–1437. acm, new york (jul 2019). https://doi.org/10.1145/3319619.3326855, https://dl.acm.org/doi/10.1145/3319619.3326855 ochoa, g., verel, s., daolio, f., tomassini, m.: local optima networks: a new model of combinatorial fitness landscapes, pp. 233–262 (2014). https://doi.org/10.1007/978-3-642-41888-4_9, http://link.springer.com/10.1007/978-3-642-41888-4_9 olson, r.s., moore, j.h.: tpot: a tree-based pipeline optimization tool for automating machine learning. in: workshop on automatic machine learning, pp. 66–74 (2016) google scholar  pimenta, c.g., de sá, a.g.c., ochoa, g., pappa, g.l.: fitness landscape analysis of automated machine learning search spaces, pp. 114–130 (2020). https://doi.org/10.1007/978-3-030-43680-3_8, http://link.springer.com/10.1007/978-3-030-43680-3_8 pushak, y., hoos, h.: algorithm configuration landscapes: in: auger, a., fonseca, c.m., lourenço, n., machado, p., paquete, l., whitley, d. (eds.) ppsn 2018. lncs, vol. 11102, pp. 271–283. springer, cham (2018). https://doi.org/10.1007/978-3-319-99259-4_22 chapter  google scholar  rodrigues, n.m., silva, s., vanneschi, l.: a study of fitness landscapes for neuroevolution (jan 2020), http://arxiv.org/abs/2001.11272 de sá, a.g.c., pinto, w.j.g.s., oliveira, l.o.v.b., pappa, g.l.: recipe: a grammar-based framework for automatically evolving classification pipelines. in: mcdermott, j., castelli, m., sekanina, l., haasdijk, e., garcía-sánchez, p. (eds.) eurogp 2017. lncs, vol. 10196, pp. 246–261. springer, cham (2017). https://doi.org/10.1007/978-3-319-55696-3_16 chapter  google scholar  treimun-costa, g., montero, e., ochoa, g., rojas-morales, n.: modelling parameter configuration spaces with local optima networks. in: proceedings of the 2020 genetic and evolutionary computation conference, pp. 751–759 (2020) google scholar  yafrani, m.e., et al.: a fitness landscape analysis of the travelling thief problem. in: proceedings of the genetic and evolutionary computation conference, pp. 277–284. acm, new york (jul 2018). https://doi.org/10.1145/3205455.3205537, https://dl.acm.org/doi/10.1145/3205455.3205537 download references acknowledgements we would like to thank ifmt for the financial support for transportation and accommodation during the participation in the conference. author information authors and affiliations instituto federal de mato grosso (ifmt), cuiaba, brazil matheus cândido teixeira universidade de minas gerais (ufmg), belo horizonte, brazil gisele lobo pappa authors matheus cândido teixeiraview author publications search author on:pubmed google scholar gisele lobo pappaview author publications search author on:pubmed google scholar corresponding author correspondence to matheus cândido teixeira . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper teixeira, m.c., pappa, g.l. (2023). fitness landscape analysis of tpot using local optima network. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_5 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords tpot local optima network fitness landscape publish with us policies and ethics profiles matheus cândido teixeira view author profile gisele lobo pappa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a systematic approach to define semantics for prioritised logic programs | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a systematic approach to define semantics for prioritised logic programs conference paper first online: 28 november 2021 pp 297–312 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a systematic approach to define semantics for prioritised logic programs download book pdf download book epub renan cordeiro10, guilherme fernandes10, joão alcântara10 & … henrique viana11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 737 accesses abstract we propose in this paper a framework of prioritised logic programs (\(\textit{plp}\)) to represent priority information explicitly in a program. differently of others approaches, we do not restrain the preference relation only to literals, but to sets of literals. as consequence, we can express in \(\textit{plp}\)s sophisticated forms of preferences without changing the programs or introducing new atoms to obtain artificially the intended preferences. besides, inspired on various developments in the literature on preference, we present a comprehensive and systematic treatment to deal with preferences in logic programming. in fact, we introduced 32 different criteria (semantics) to establish preference between partial stable models as well as those semantics whose definition depends on partial stable models. we show some properties of our framework; in particular, we guarantee these semantics for \(\textit{plp}\) generalise their counterparts for logic programs without preferences. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others representing morality in logic programming chapter © 2016 a language-independent framework for reasoning about preferences for declarative problem solving chapter © 2019 probabilistic hyperproperties of markov decision processes chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. formal logic linear logic logic logic design utilitarianism computer science logic and foundations of programming 1 introduction in many applications of artificial intelligence and commonsense reasoning, we have to deal with uncertain, vague, inaccurate, doubtful and even contradictory information as well as with certain, precise and reliable information. in this scenario, conflicts can naturally arise and, it urges to develop mechanisms to resolve them. adopting some criteria to establish preferences between these conflicting information is an effective way of reasoning with them. somehow we have to prioritise the information to separate the wheat from the chaff. for representing and reasoning with preferences, several prioritised systems have been proposed (see [1] for a survey). when moving our eyes to preferences in logic programming, we see a plethora of proposals (see [2,3,4,5,6,7,8,9,10,11,12] for a non exhaustive list). as witnessed by the multitude of formalisms with preferences, a key issue here is the lack of consensus on the role played by preferences. roughly speaking, we find in the literature various works with diverging intuitions, and as a consequence, conceptually different formal techniques to deal with preference. while there are many proposals on preferences in the reasoning process and knowledge representation, there is still much space for improvement. indeed, as noticed in [13], one of the main problems a user faces when expressing preferences is any preference representation language based on the direct assessment of user preferences over the complete set of options is simply infeasible. we have to resort to compact preference representation languages which represent partial descriptions of preferences and rank-order the possible options. with such a motivation in view, in this paper, we generalise the notion of prioritised logic programs (\(\textit{plp}\)s) proposed in [4] to encompass not only the relation of priority between literals (atoms or its negation), but also sets of literals. the resulting framework offers a preference representation language expressive enough to represent complex forms of preference in a compact manner. while in [4], in order to represent a conjunction of preferences, they have to include new rules in the program and preferences between new atoms, we will surmount this problem by representing conjunctions of preferences directly. a second fundamental challenge is to select the most preferable models given the priorities between the sets of literals. which criteria should be regarded? in order to decide a model \(i_1\) is more preferable than a model \(i_2\), someone would require any set of literals associated with \(i_1\) should have higher priority than any set of literals associated with \(i_2\). for others, it suffices to guarantee the worst-ranked sets of literals associated with \(i_1\) should have higher priority than those worst-ranked sets of literals associated with \(i_2\). depending on the criteria employed to select the most preferable models, we would obtain different semantics for \(\textit{plp}\)s. overall, there is no consensual way to define these semantics and, we will find many disconnected approaches. in this work, we will not cling to a unique path, but present comprehensive and systematic criteria to define semantics for them. as result, we will have a clearer notion of what we mean when selecting the most preferable models. the rest of the paper develops as follows. in sect. 2, we will introduce the basic concepts related to normal logic programs \((\textit{nlp})\) and their semantics. the main contributions of the paper can be found in sect. 3, where we will present the prioritised logic programs (\(\textit{plp}\)s) and comprehensive and systematic criteria to define semantics for them. next, we show how complex forms of priorities can be expressed straightforwardly in \(\textit{plp}\)s. in sect. 5, we prove some properties of our proposal; in particular, we guarantee these semantics for \(\textit{plp}\) generalise their counterparts for logic programs without preferences. section 6 provides an overview of related works. finally, sect. 7 summarises our contributions and pointed out future works. 2 preliminaries in the sequel, we will consider propositional normal logic programs, which we will call logic programs or simply programs from now on. definition 1 a normal logic program (\(\textit{nlp}\)), p, is a set of rules of the form \(a \leftarrow a_1, \ldots , a_m,\mathtt {not\ }b_1,\) \(\ldots , \mathtt {not\ }b_n\) (\(m, n \in \mathbb {n}\)), where a, \(a_i\) (\(1 \le i \le m\)) and \(b_j\) (\(1 \le j \le n\)) are atoms; \(\mathtt {not}\) represents default negation, and \(\mathtt {not\ }b_j\) is a default literal. a literal is an atom or a default literal. we say a is the head of the rule, and \(a_1, \ldots , a_m, \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n\) is its body. if p has no occurrence of default literals, it is called a definite logic program. the herbrand base of p is the set \( hb _p\) of all atoms occurring in p. the set of literals of p is \( lit _p= hb _p\cup \left\{ \mathtt {not\ }a \mid a \in hb _p\right\} \). a wide range of logic programming semantics can be defined based on the 3-valued interpretations (for short, interpretations) of programs: definition 2 [14] a set \(i \subseteq lit _p\) is a 3-valued interpretation of an \(\textit{nlp}\) p if \(\not \exists a \in hb _p\) with \(\left\{ a, \mathtt {not\ }a\right\} \subseteq i\). we say i is a model of p iff for each rule \(a \leftarrow a_1, \ldots , a_m,\) \(\mathtt {not\ }b_1, \ldots , \) \(\mathtt {not\ }b_n \in p\), if \(\left\{ a_1, \ldots , a_m, \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n\right\} \subseteq i\), then \(a \in i\). we refer to \(t_i = \left\{ a \in hb _p\mid a \in i\right\} \) as the set of true atoms w.r.t. i and to \(f_i = \left\{ a \in hb _p\mid \mathtt {not\ }a \in i\right\} \) as the set of false atoms w.r.t. i. if \(a \not \in t_i \cup f_i\), then a is undefined w.r.t. i. let i be a 3-valued interpretation of a program p; in order to define the semantics for normal logic programs, we will resort to the reduct of p with respect to i (written as p/i), which is the definite logic program built by the execution of the following steps: 1. remove any \(a \leftarrow a_1, \ldots , a_m,\) \(\mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n \in p\) with \(\left\{ b_1, \ldots , b_n\right\} \cap i \ne \emptyset \); 2. afterwards, remove any occurrence of \(\mathtt {not\ }b_i\) from p such that \(\mathtt {not\ }b_i \in i\). 3. then, replace any occurrence of \(\mathtt {not\ }b_i\) left by a special atom \(\mathbf {u}\) (\(\mathbf {u} \not \in hb _p\)). we say i is the least model of a definite logic program p if among the models of p, \(t_i\) is minimal (w.r.t. set inclusion) and \(f_i\) is maximal (w.r.t. set inclusion). note that p/i is a definite logic program. as consequence, p/i has a unique least model [14], denoted by \(\varomega _p(i)\), such that for any \(a \in hb _p\) \(a \in \varomega _p(i)\) iff \(a \leftarrow a_1, \ldots , a_m \in p/i\) and \(\left\{ a_1, \ldots , a_m\right\} \subseteq \varomega _p(i)\); \(\mathtt {not\ }a \in \varomega _p(i)\) iff for every \(a \leftarrow a_1, \ldots , a_m \in p/i\), it holds \(\{\mathtt {not\ }a_1, \ldots ,\) \(\mathtt {not\ }a_m \} \cap \varomega _p(i) \ne \emptyset \). we now specify the \(\textit{nlp}\) semantics to be examined in this paper. definition 3 let p be an \(\textit{nlp}\) and i be an interpretation: i is a partial stable (\( ps \)) model of p iff \(i = \varomega _p(i)\) [14]. i is a well-founded model of p iff i is a \( ps \) model of p with minimal \(t_i\) [14]. i is a regular model of p iff i is a \( ps \) model of p with maximal \(t_i\) [15]. i is a stable model of p iff i is a \( ps \) model of p where for each \(a \in hb _p\), \(a \in t_i \cup f_i\) [14]. i is an l-stable model of p iff i is a \( ps \) model of p with minimal (w.r.t. set inclusion) \(\left\{ a \in hb _p\mid a \not \in t_i \cup f_i\right\} \) [15]. example 1 consider the \(\textit{nlp}\) p: $$ \begin{array}{lllll} a \leftarrow \mathtt {not\ }b \quad&b \leftarrow \mathtt {not\ }a, \mathtt {not\ }c \quad&c \leftarrow \mathtt {not\ }b, \mathtt {not\ }d \quad&c \leftarrow \mathtt {not\ }c \quad&d \leftarrow \mathtt {not\ }c \end{array} $$ with respect to p, we have a) partial stable models: \(i_1 = \emptyset \), \(i_2 = \left\{ a, \mathtt {not\ }b\right\} \), \(i_3 = \left\{ d\right\} \), \(i_4 = \{b, d, \mathtt {not\ }a \}\), \(i_5 = \left\{ a, c, \mathtt {not\ }b, \mathtt {not\ }d\right\} \) and \(i_6 = \left\{ a, d, \mathtt {not\ }b\right\} \); b) well-founded model: \(i_0\); c) regular models: \(i_4\), \(i_5\) and \(i_6\); d) stable model and l-stable model: \(i_5\). in the next section, we will offer a comprehensive and systematic treatment to deal with preferences in logic programming. 3 semantics for prioritised logic programs the semantics of normal logic programs seen in definition 3 have multiple models in general. preference is then introduced to select among them the intended models of a program, but at the semantic level. in [4], they conceived the notion of prioritised logic programs (\(\textit{plp}\)s) by establishing a prioritisation mechanism between literals (atoms or its negation) to represent and reason with preferences in logic programming at the syntactic level. here we generalise this notion of \(\textit{plp}\) to encompass not only the relation of priority between literals, but also sets of literals. the idea is to employ these priorities to select the most preferable partial stable models. we will focus on partial stable models, because it is the base to define the remaining semantics in definition 3. in this work, the priorities will be established between sets of literals, which we will be called as options: definition 4 (options) given an \(\textit{nlp}\) p, an option o from p is a subset of \( lit _p\). by \(\mathcal {o}_p\) we mean the set of all options from p. we say o is inconsistent, if there is \(a \in hb _p\) such that \(\left\{ a,\mathtt {not\ }a\right\} \subseteq o\). otherwise, o is consistent. thus in a program representing a menu based on fish (f), meat m, white wine (w), red wine (r), cake (c) and ice cream (i), \(\left\{ f, \mathtt {not\ }m, r\right\} \), \(\left\{ i, \mathtt {not\ }i\right\} \) and \(\emptyset \) are possible options. not rarely, some options have a higher priority than others: definition 5 (priorities) given an \(\textit{nlp}\) p, a reflexive and transitive relation is defined on \(\mathcal {o}_p\). for any \(o_1\) and \(o_2\) from \(\mathcal {o}_p\), \(o_1 \preceq o_2\) is called a priority, and we say \(o_2\) has a higher or equal priority than \(o_1\). we write \(o_1 \prec o_2\) if \(o_1 \preceq o_2\) and \(o_2 \not \preceq o_1\), and say \(o_2\) has a strictly higher priority than \(o_1\). furthermore, \(o_1\) is indifferent to \(o_2\), denoted by \(o_1 \approx o_2\), when both \(o_1 \preceq o_2\) and \(o_2 \preceq o_1\) hold. from an \(\textit{nlp}\) and a set of priorities, we build a prioritised logic program: definition 6 (prioritised logic programs) a prioritised logic program (\(\textit{plp}\)) is a pair \((p,\varphi )\), where p is an \(\textit{nlp}\) and \(\varphi \) is a set of priorities on \(\mathcal {o}_p\). in order to enforce the priority relation in \(\varphi \) is reflexive and transitive, while maintaining a compact representation of priorities, we resort to the closure of \(\varphi \): definition 7 (closure of \(\boldsymbol{\varphi }\)). let \((p, \varphi )\) be a \(\textit{plp}\). for each \(o_1, o_2\) in \(\mathcal {o}_p\), the closure \(\varphi ^*\) of \(\varphi \) is defined as follows: \(o_1 \preceq o_1 \in \varphi ^*\); if \(o_1 \preceq o_2 \in \varphi \), then \(o_1 \preceq o_2 \in \varphi ^*\); if \(o_1 \preceq o_2 \in \varphi ^*\) and \(o_2 \preceq o_3 \in \varphi ^*\), then \(o_1 \preceq o_3 \in \varphi ^*\). \(o_1 \preceq o_2 \in \varphi ^*\) and \(o_1 \ne \emptyset \) and \(o_2 \ne \emptyset \) and \(e \not \in o_1 \cup o_2\), then \(o_1 \cup \left\{ e\right\} \preceq o_2 \cup \left\{ e\right\} \in \varphi ^*\). the closure \(\varphi ^*\) of \(\varphi \) is the set of priorities derived reflexively and transitively using priorities in \(\varphi \). besides, we introduced in \(\varphi ^*\) an inertia principle to keep the preference representation in \(\varphi \) compact. in the remainder of this paper, when we say \(o_2 \not \prec o_1\) according to \(\varphi ^*\), we mean \(o_2 \preceq o_1 \not \in \varphi ^*\) or \(o_1 \preceq o_2 \in \varphi ^*\). among the many possible options, we are particularly interested in those best/worst-ranked: definition 8 (best/worst-ranked option) let \((p, \varphi )\) be a \(\textit{plp}\) and \(i \subseteq lit _p\). we say o is a) a best-ranked option in i w.r.t. \(\varphi \) iff \(o \subseteq i\) and \(\not \exists o' \subseteq i\) such that \(o \prec o' \in \varphi ^*\). b) a worst-ranked option in i w.r.t. \(\varphi \) iff \(o \subseteq i\) and \(\not \exists o' \subseteq i\) such that \(o' \prec o \in \varphi ^*\). given two set of options, we have to determine which one is preferable; we introduce below four criteria with this purpose: definition 9 let \((p, \varphi )\) be a \(\textit{plp}\) and \(\mathcal o_1\) and \(\mathcal o_2\) subsets of \(\mathcal {o}_p\). we say \(\mathcal o_1 \subseteq ^{\forall \forall } \mathcal o_2\) in \(\varphi \) iff \(\forall o_1 \in \mathcal o_1\) and \(\forall o_2 \in \mathcal o_2\), it holds \(o_2 \not \prec o_1\) in \(\varphi ^*\). \(\mathcal o_1 \subseteq ^{\forall \exists } \mathcal o_2\) in \(\varphi \) iff \(\forall o_1 \in \mathcal o_1\), it holds \(\exists o_2 \in \mathcal o_2\) such that \(o_2 \not \prec o_1\) in \(\varphi ^*\). \(\mathcal o_1 \subseteq ^{\exists \forall } \mathcal o_2\) in \(\varphi \) iff \(\exists o_1 \in \mathcal o_1\) such that \(\forall o_2 \in \mathcal o_2\), it holds \(o_2 \not \prec o_1\) in \(\varphi ^*\). \(\mathcal o_1 \subseteq ^{\exists \exists } \mathcal o_2\) in \(\varphi \) iff \(\exists o_1 \in \mathcal o_1\) and \(\exists o_2 \in \mathcal o_2\) such that \(o_2 \not \prec o_1\) in \(\varphi ^*\). besides, different preference semantics have been proposed in the literature. we recall some of the most well-known of them: definition 10 (preference relation) let \((p,\varphi )\) be a \(\textit{plp}\), \(i_1\) and \(i_2\) be \( ps \) models of p, \(\mathcal b_1\) and \(\mathcal b_2\) are respectively the sets of all best-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \), and \(\mathcal w_1\) and \(\mathcal w_2\) are respectively the sets of all worst-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \). consider also \(x \in \left\{ st , opt , pes , opp \right\} \) and \(y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \). by \(i_1 \subseteq ^y_x i_2\) w.r.t. \(\varphi \), we mean the preference relation \(\subseteq ^y_x\) over the set of \( ps \) models of p is defined as follows: 1. \(i_1 \subseteq ^y_x i_1\) w.r.t. \(\varphi \); 2. \(i_1 \subseteq ^y_x i_2\) w.r.t. \(\varphi \) a) (strong semantics): if \(x = st \) and \(\mathcal b_1 \subseteq ^y \mathcal w_2\) in \(\varphi \). b) (optimistic semantics): if \(x = opt \) and \(\mathcal b_1 \subseteq ^y \mathcal b_2\) in \(\varphi \). c) (pessimistic semantics): if \(x = pes \) and \(\mathcal w_1 \subseteq ^y \mathcal w_2\) in \(\varphi \). d) (opportunistic semantics): if \(x = opp \) and \(\mathcal w_1 \subseteq ^y \mathcal b_2\) in \(\varphi \). 3. if \(i_1 \subseteq ^y_x i_2\) w.r.t. \(\varphi \) and \(i_2 \subseteq ^y_x i_3\) w.r.t. \(\varphi \), then \(i_1 \subseteq ^y_x i_3\) w.r.t. \(\varphi \). when it is clear from the context, we will drop the reference to \(\varphi \) and say \(\mathcal o_1 \subseteq ^{\forall \forall } \mathcal o_2\) and \(i_1 \subseteq ^y_x i_2\). inspired by [4], we can define a preference relation \(i_1 \sqsubseteq ^y_x i_2\) which consider just their exclusive portions given by \(i_1 i_2\) and \(i_2 i_1\). 1. \(i_1 \sqsubseteq ^y_x i_2\) iff \(i_1 i_2 \subseteq ^y_x i_2 i_1\). 2. if \(i_1 \sqsubseteq ^y_x i_2\) and \(i_2 \sqsubseteq ^y_x i_3\), then \(i_1 \sqsubseteq ^y_x i_3\). note we have four criteria to choose the best/worst-ranked options (definition 9), four criteria to choose the most preferable semantics (definition 10). we also can decide if a \( ps \) model \(i_1\) is preferred to a \( ps \) model \(i_2\) by regarding the whole models or just their exclusive portions given by \(i_1 i_2\) and \(i_2 i_1\). hence, we have at disposal 32 manners of selecting the preferred partial stable models: definition 11 (preferred partial stable model) let \((p,\varphi )\) be a \(\textit{plp}\). a \( ps \) model i of p is a preferred partial stable model (or \( pps \) model, for short) of \((p,\varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) if \(i \subseteq ^y_x i'\) (resp. \(i \sqsubseteq ^y_x i'\)) implies \(i' \subseteq ^y_x i\) (resp. \(i' \sqsubseteq ^y_x i\)) (w.r.t \(\varphi \)) for any \( ps \) model \(i'\) of p. the set of all \( pps \) of \((p,\varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) is written as \(\mathcal {pps}(p,\varphi )_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi )_{\sqsubseteq ^y_x}\)). as the remaining semantics for \(\textit{nlp}\) seen in this paper are based on \( ps \) models (definition 3), we are equipped with 32 different manners of selecting the most preferred models according to them. definition 12 let \((p, \varphi )\) be an \(\textit{plp}\) and i be an interpretation: i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) with minimal \(t_i\). i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) with maximal \(t_i\). i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) where for each \(a \in hb _p\), \(a \in t_i \cup f_i\). i is a preferential l-stable model of \((p,\varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) with minimal (w.r.t. set inclusion) \(\left\{ a \in hb _p\mid a \not \in t_i \cup f_i\right\} \) \(\left\{ a \in hb _p\mid m(a) = \mathbf {u}\right\} \). let us consider the following example: example 2 let p be the \(\textit{nlp}\) in example 1 and \(\varphi = \{\emptyset \preceq \left\{ a\right\} , \left\{ a\right\} \preceq \left\{ d\right\} , \left\{ d\right\} \preceq \left\{ b\right\} , \left\{ b, d\right\} \preceq \left\{ b\right\} , \left\{ b, \mathtt {not\ }a\right\} \preceq \left\{ \mathtt {not\ }a\right\} , \left\{ a, d\right\} \preceq \left\{ d\right\} \}\). then the best-ranked options for \(i_1, i_2, i_3, i_4, i_5\) and \( i_6\) are respectively \(\mathcal b_1 = \emptyset , \mathcal b_2 = \{ \left\{ a\right\} , \left\{ \mathtt {not\ }b\right\} ,\) \(\left\{ a, \mathtt {not\ }b\right\} \}, \mathcal b_3 = \left\{ \left\{ d\right\} \right\} , \mathcal b_4 = \left\{ \left\{ b\right\} , \left\{ \mathtt {not\ }a\right\} \right\} , \mathcal b_5 = 2^{i_5}\) and \(\mathcal b_6 = \{ \left\{ d\right\} , \left\{ \mathtt {not\ }b\right\} ,\) \(\left\{ d, \mathtt {not\ }b\right\} \}\). regarding only the optimistic approach for illustrative purposes, we have \(\mathcal {pps}(p,\varphi )_{\subseteq ^{\forall \forall }_ opt } = \left\{ i_4\right\} \); \(\mathcal {pps}(p,\varphi )_{\subseteq ^{\forall \exists }_ opt } = \left\{ i_2, i_4, i_5, i_6\right\} \); \(\mathcal {pps}(p,\varphi )_{\subseteq ^{\exists \forall }_ opt } = \left\{ i_3, i_4, i_6\right\} \) and \(\mathcal {pps}(p,\varphi )_{\subseteq ^{\exists \exists }_ opt } = \left\{ i_2, i_3, i_4, i_5, i_6\right\} \). in this example, notice at least one \( ps \) model of p is discarded in \(\mathcal {pps}(p,\varphi )_{\subseteq ^y_ opt }\) for any \(y \in \left\{ \forall \forall , \forall \exists , \exists \forall , \exists \exists \right\} \). many attempts have been made to empower logic programming with preferences. the distinguishing aspect of our work is that we offer a comprehensive and systematic way of dealing with preferences in logic programming inspired by well-known criteria found in the literature on preference. in the next section, we will examine how expressive our proposal is. 4 expressing priorities in \(\textit{plp}\)s as it is not feasible in general when expressing preferences to compare all possible pairs of options or evaluate them individually (the number of options increases exponentially with the number of variables), we have to resort to compact preference representation languages to represent these partial descriptions preferences. now we will show our preference representation language although compact is robust enough to express well-known complex forms of preferences. in subsects. 4.1, 4.2, 4.3, 4.4 and 4.5, we will tackle respectively how to represent priorities between conjunctive knowledge, priorities between disjunctive knowledge, conditional priorities, priorities between rules and bipolar priorities. 4.1 priorities between conjunctive knowledge users may also express their partial preferences in term of comparative statements (as in “i prefer white wine to red wine"); this preference can be altered when interacting with other options (as in “i prefer meat accompanied by a red wine to meat accompanied by a white wine"); the preference for white wine can even be regained with the addition of more options (as in “i prefer meat and cheesecake accompanied by a white wine to meat and cheesecake accompanied by a red wine"). thus, the conjunction of options can interfere with the users’ preference and change the overall outcome. in [4], conjunctive preferences \((e_1, \ldots , e_m) \preceq (e_1', \ldots , e_n')\) cannot in general be expressed directly as the preference relation applied there only involves a pair of individual literals. in order to represent these conjunctive preferences in a \(\textit{plp}\) \((p, \varphi )\), the authors have to add the rules \(e_0 \leftarrow e_1, \ldots e_m\) and \(e_0' \leftarrow e_1', \ldots e_n'\) to p with the newly introduced atoms \(e_0\) and \(e_0'\), and the priority \((e_0 \preceq e_0')\) to \(\varphi \). in short, conjunctive preferences are represented in [4] by changing the program p with two new atoms and two new rules before introducing \((e_0 \preceq e_0')\), which expresses indirectly the desired conjunctive preference. in our approach, we neither need to change the program p nor to add new artificial atoms to express conjunctive preferences. preferences as \((e_1, \ldots , e_m) \preceq (e_1', \ldots , e_n')\) can just be represented as \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \) and included straightforwardly in \(\varphi \). 4.2 priorities between disjunctive knowledge disjunctive preferences as \((e_1 \vee \cdots \vee e_m) \preceq (e_1' \vee \cdots \vee e_n')\) can be represented in a \(\textit{plp}\) \((p, \varphi )\) by ensuing \(\varphi ' \subseteq \varphi \) and \(\varphi ' = \left\{ e_i \preceq e_j' \mid 1 \le i \le m \textit{ and } 1 \le j \le n\right\} \). 4.3 conditional priorities it is not always the case a preference is absolute. in the so-called conditional preference, users may also wish to express specific preferences in particular contexts (e.g., if red wine is served, i prefer meat to fish). it is denoted by \(\gamma : \phi \preceq \psi \) and interpreted as “given a context \(\gamma \), prefer \(\psi \) to \(\phi \)". assuming \(\gamma = c_1 \wedge \cdots \wedge c_o\), \(\phi = e_1 \wedge \cdots \wedge e_m\) and \(\psi = e_1' \wedge \cdots \wedge e_n'\), we can represent the conditional preference \(\gamma : \phi \preceq \psi \) as the conjunctive preference \(\left\{ c_1, \ldots , c_o, e_1, \ldots , e_m\right\} \preceq \left\{ c_1, \ldots , c_o, e_1', \ldots , e_n'\right\} \) in \(\varphi \). conditional preferences can occur disguised as in “if \(\gamma = c_1 \wedge \cdots \wedge c_o\) is true, then prefer \(\psi = e_1' \wedge \cdots \wedge e_n'\)"; it interpreted as if \(\gamma \) is true, then prefer \(\psi \) to \(\mathtt {not\ }e_j'\) with \(1 \le j \le n\). this is a particular case of our representation when \(\phi = \mathtt {not\ }e_j'\), i.e., such a preference can be represented in \((p, \varphi )\) by including in \(\varphi \) the preferences \(\left\{ c_1, \ldots , c_o, \mathtt {not\ }e_j'\right\} \preceq \left\{ c_1, \ldots , c_o, e_1', \ldots , e_n'\right\} \) with \(1 \le j \le n\). 4.4 priorities between rules preferences can also be established between (conflicting) rules as in \(e \leftarrow e_1, \ldots , e_m\) \(\preceq e' \leftarrow e_1', \ldots , e_n'\). its intended meaning is the conclusion of \(e'\) via \(e' \leftarrow e_1', \ldots , e_n'\) will block the conclusion of e via \(e \leftarrow e_1, \ldots , e_m\). such a preference can be represented in \((p, \varphi )\) by including \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \) in \(\varphi \). 4.5 bipolar preferences preferences can express not only what is satisfactory (positive preferences), but also what can be considered tolerable or unacceptable (negative preferences). these two forms of preferences (bipolar preferences) have been conjointly and compactly expressed in various works [16,17,18,19,20,21]. roughly speaking, the intended meaning of introducing positive preferences is to express wishes which should be satisfied as best as possible, whilst negative preferences are intended to enhance the idea that what is not rejected or excluded is tolerated. thus, when interpreting \(\preceq \) as a bipolar priority by \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \), we mean we are giving priority not only to outcomes satisfying \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \), but to those not falsifying \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \), i.e., it either should contain \(\left\{ e_1', \ldots , e_n'\right\} \), but not \(\left\{ e_1, \ldots , e_m\right\} \) (positive preference), or should contain \(\left\{ \mathtt {not\ }e_i\right\} \), but not \(\left\{ \mathtt {not\ }e_j'\right\} \) with \(1 \le i \le m\) and \(1 \le j \le n\) (negative preference). bipolar preferences as \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \) can be represented in \((p, \varphi )\) by including in \(\varphi \) the conjunctive preference \(\left\{ e_1, \ldots , e_m\right\} \preceq \left\{ e_1', \ldots , e_n'\right\} \) and the preferences \(\left\{ \mathtt {not\ }e_j'\right\} \preceq \left\{ \mathtt {not\ }e_i\right\} \) with \(1 \le i \le m\) and \(1 \le j \le n\). before moving to the next section, we will emphasise our decision of employing a representation language to express priorities between sets of literals instead of a language to express priorities only between literals as in [4]. although the approach of [4] suffices to express in a \(\textit{plp}\) \((p, \varphi )\) the complex forms of preferences showed in this section, there is a price to pay: the program p should be changed with the introduction of new atoms as well as the set \(\varphi \) of priorities. in our approach, as the priority relation is given in term of sets of literals, we neither need to change the program p nor to add new artificial atoms to express priorities between conjunctive knowledge, priorities between disjunctive knowledge, conditional priorities, priorities between rules and bipolar priorities. 5 results now we will prove some properties found in our proposal. the main results are both the preference relations \(\subseteq ^y_x\) and \(\sqsubseteq ^y_x\) are consistent (proposition 1) and have neither a monotonic nor an antitonic behaviour w.r.t. \(\varphi \) (theorem 3); proposition 2 and theorem 1 establish conditions under which the relations of preference are simplified; when no priority is available, prioritised logic programs and normal logic programs have the same semantics (theorem 2) and inconsistent options are useless (theorem 4). next we define the notion of cyclic relation: definition 13 (cyclic relation) a relation \(\sqsubseteq \) is cyclic if and only if its induced strict preference relation is cyclic, i.e., there exists a chain \(\mathcal o \ldots \mathcal o'\) such that \(\mathcal o \sqsubset \mathcal o' \sqsubset \cdots \sqsubset \mathcal o\). otherwise \(\sqsubseteq \) is acyclic. if either \(\subseteq ^y_x\) or \(\sqsubseteq ^y_x\) were cyclic, the resulting preference relation would be inconsistent. in particular, we would find \(\textit{plp}\)s without preferred partial stable models at all. for instance, suppose an \(\textit{nlp}\) whose partial stable models are \(m_1\), \(m_2\) and \(m_3\) such that \(m_1 \subset ^y_x m_2\), \(m_2 \subset ^y_x m_3\) and \(m_3 \subset ^y_x m_1\). clearly the resulting \(\textit{plp}\) would not have any preferred partial stable model. fortunately, both \(\subseteq ^y_x\) and \(\sqsubseteq ^y_x\) are acyclic: proposition 1 for any \(x \in \left\{ st , opt , pes , opp \right\} \) and any \(y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), both the preference relations \(\subseteq ^y_x\) and \(\sqsubseteq ^y_x\) are acyclic. proof by absurd, suppose \(\subseteq ^y_x\) is cyclic. this means there exists a chain \(\mathcal o \ldots \mathcal o'\) such that \(\mathcal o \subset ^y_x \mathcal o' \subset ^y_x \cdots \subset ^y_x \mathcal o\). as \(\subseteq ^y_x\) is reflexive (definition 10), it holds \(\mathcal o \not \subset ^y_x \mathcal o\). besides, by transitivity, we obtain \(\mathcal o' \subseteq ^y_x \mathcal o\). it is an absurd as \(\mathcal o \subset ^y_x \mathcal o'\). the proof for \(\sqsubseteq ^y_x\) is similar. notice in this last result the importance of imposing reflexivity and transitivity in definition 10 to ensure every \(\textit{plp}\) will have at least one preferable \( ps \) model. now we will show when the best-ranked options of the \( ps \) models have the same priority and the worst-ranked options of the \( ps \) models have the same priority, the relations of preference are simplified: proposition 2 let \((p, \varphi )\) be a \(\textit{plp}\), \(i_1\) and \(i_2\) be subsets of \( lit _p\), \(\mathcal b_1\) and \(\mathcal b_2\) be respectively the set of best-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \), and \(\mathcal w_1\) and \(\mathcal w_2\) be respectively the set of worst-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \). assume for each \(o_1\) and \(o_2\) in \(\mathcal b_i\) and for each \(o_1'\) and \(o_2'\) in \(\mathcal w_i\) with \(i \in \left\{ 1,2\right\} \), it holds \(o_1 \approx o_2\) and \(o_1' \approx o_2'\). consider \(\mathcal x = \mathcal b_1\) if \(x \in \{ st , opt \}\) and \(\mathcal x = \mathcal w_1\) if \(x \in \{ pes , opp \}\), and \(\mathcal y = \mathcal b_2\) if \(x \in \{ opt , opp \}\) and \(\mathcal y = \mathcal w_2\) if \(x \in \{ st , pes \}\). then for any \(x \in \{ st , opt ,\) \( pes , opp \}\), \(\forall o \in \mathcal x\) and \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \) iff \(\forall o \in \mathcal x\), it holds \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \) iff \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) such that \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \) iff \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) and \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \). proof for any \(x \in \{ st , opt , pes , opp \}\), let \(o_1\) be an element of \(\mathcal x\) and \(o_2\) be an element of \(\mathcal y\). given the elements of \(\mathcal x\) are indifferent to each other and the elements of \(\mathcal y\) are also indifferent to each other, we obtain \(o_2 \not \prec o_1 \in \varphi \) iff \(\forall o \in \mathcal x\) and \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \); \(o_2 \not \prec o_1 \in \varphi \) iff \(\forall o \in \mathcal x\), it holds \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \); \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) such that \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \); \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) and \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \). i.e., \(\forall o \in \mathcal x\) and \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \) iff \(\forall o \in \mathcal x\), it holds \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \) iff \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) such that \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \) iff \(o_2 \not \prec o_1 \in \varphi \) iff \(\exists o \in \mathcal x\) and \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \).    \(\square \) a binary relation \(\sqsubset \) on a set \(\mathcal o\) is a total preorder if it is reflexive, transitive and for each \(o_1\) and \(o_2\) in \(\mathcal o\), it holds \(o_1 \sqsubseteq o_2\) or \(o_2 \sqsubseteq o_1\). for all \(x \in \left\{ st , opt , pes , opp \right\} \), if the relation \(\preceq \) is a total preorder, \(\subseteq ^{\forall \forall }_x\), \(\subseteq ^{\forall \exists }_x\), \(\subseteq ^{\exists \forall }_x\) and \(\subseteq ^{\exists \exists }_x\) (\(\sqsubseteq ^{\forall \forall }_x\), \(\sqsubseteq ^{\forall \exists }_x\), \(\sqsubseteq ^{\exists \forall }_x\) and \(\sqsubseteq ^{\exists \exists }_x\)) will collapse into each other. theorem 1 let \((p, \varphi )\) be a \(\textit{plp}\) such that the preference relation \(\preceq \) in \(\varphi \) is a total preorder on \(\mathcal {o}_p\). then for any \(x \in \left\{ st , opt , pes , opp \right\} \), i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^{\forall \forall }_x\) (resp. \(\sqsubseteq ^{\forall \forall }_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^{\forall \exists }_x\) (resp. \(\sqsubseteq ^{\forall \exists }_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^{\exists \forall }_x\) (resp. \(\sqsubseteq ^{\exists \forall }_x\)) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^{\exists \exists }_x\) (resp. \(\sqsubseteq ^{\exists \exists }_x\)). proof let \(i_1\) and \(i_2\) be \( ps \) models of p, \(\mathcal b_1\) and \(\mathcal b_2\) be respectively the set of best-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \), and \(\mathcal w_1\) and \(\mathcal w_2\) be respectively the set of worst-ranked options in \(i_1\) and \(i_2\) w.r.t. \(\varphi \). consider \(\mathcal x = \mathcal b_1\) if \(x \in \{ st , opt \}\) and \(\mathcal x = \mathcal w_1\) if \(x \in \{ pes , opp \}\), and \(\mathcal y = \mathcal b_2\) if \(x \in \{ opt , opp \}\) and \(\mathcal y = \mathcal w_2\) if \(x \in \{ st , pes \}\). it suffices to show for any \(x \in \{ st , opt , pes , opp \}\), \(i_1 \subseteq ^{\forall \forall }_x i_2\) iff \(i_1 \subseteq ^{\forall \exists }_x i_2\) iff \(i_1 \subseteq ^{\exists \forall }_x i_2\) iff \(i_1 \subseteq ^{\exists \exists }_x i_2\) and \(i_1 \sqsubseteq ^{\forall \forall }_x i_2\) iff \(i_1 \sqsubseteq ^{\forall \exists }_x i_2\) iff \(i_1 \sqsubseteq ^{\exists \forall }_x i_2\) iff \(i_1 \sqsubseteq ^{\exists \exists }_x i_2\): as \(\preceq \) in \(\varphi \) is a total preorder on \(\mathcal {o}_p\), for each \(o_1\) and \(o_2\) in \(\mathcal b_i\) and for each \(o_1'\) and \(o_2'\) in \(\mathcal w_i\) with \(i \in \left\{ 1,2\right\} \), it holds \(o_1 \approx o_2\) and \(o_1' \approx o_2'\). let \(i \subseteq ^{\forall \forall }_x i'\); there are three possibilities: 1. \(i_2 = i_1\). then \(i_1 \subseteq ^{\forall \exists }_x i_2\), \(i_1 \subseteq ^{\exists \forall }_x i_2\) and \(i_1 \subseteq ^{\exists \exists }_x i_2\). 2. \(\forall o \in \mathcal x\) and \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \). then by proposition 2, a) \(\forall o \in \mathcal x\), it holds \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \) and b) \(\exists o \in \mathcal x\) such that \(\forall o' \in \mathcal y\), it holds \(o' \not \prec o \in \varphi \) and c) \(\exists o \in \mathcal x\) and \(\exists o' \in \mathcal y\) such that \(o' \not \prec o \in \varphi \). hence, \(i_1 \subseteq ^{\forall \exists }_x i_2\) and \(i_1 \subseteq ^{\exists \forall }_x i_2\) and \(i_1 \subseteq ^{\exists \exists }_x i_2\). 3. \(i_1 \subseteq ^{\forall \forall }_x i\) and \(i \subseteq ^{\forall \forall }_x i_2\).this means there exists a sequence $$\begin{aligned} i_1 = j_1 \subseteq ^{\forall \forall }_x j_2 \subseteq ^{\forall \forall }_x \cdots \subseteq ^{\forall \forall }_x j_m = i \subseteq ^{\forall \forall }_x j_{m+1} \subseteq ^{\forall \forall }_x \cdots \subseteq ^{\forall \forall }_x j_n = i_2 \end{aligned}$$ such that \(\forall i \in \left\{ 1, \ldots , n 1\right\} \), we have \(j_1 \subseteq ^{\forall \forall }_x j_2\) iff the condition 2 of definition 10 holds. again by proposition 2, condition 2 of definition 10 also holds for \(j_1 \subseteq ^{\forall \exists }_x j_2\), \(j_1 \subseteq ^{\exists \forall }_x j_2\) and \(j_1 \subseteq ^{\exists \exists }_x j_2\) for each \(i \in \left\{ 1, \ldots , n 1\right\} \). hence, by condition 3 of definition 10, we obtain \(i_1 \subseteq ^{\forall \exists }_x i_2\) and \(i_1 \subseteq ^{\exists \forall }_x i_2\) and \(i_1 \subseteq ^{\exists \exists }_x i_2\). similarly for \(i_1 \subseteq ^{\forall \exists }_x i_2\), \(i_1 \subseteq ^{\exists \forall }_x i_2\) and \(i_1 \subseteq ^{\exists \exists }_x i_2\). the proof of \(i_1 \sqsubseteq ^{\forall \forall }_x i_2\) iff \(i_1 \sqsubseteq ^{\forall \exists }_x i_2\) iff \(i_1 \sqsubseteq ^{\exists \forall }_x i_2\) iff \(i_1 \sqsubseteq ^{\exists \exists }_x i_2\) is similar.    \(\square \) corollary 1 let \((p, \varphi )\) be a \(\textit{plp}\) such that the preference relation \(\preceq \) in \(\varphi \) is a total preorder on \(\mathcal {o}_p\). then \(\forall x \in \left\{ st , opt , pes , opp \right\} \), and \(\forall y, \forall z \in \left\{ \forall \forall , \forall \exists , \exists \forall , \exists \exists \right\} \) i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\subseteq ^z_x\) (resp. \(\sqsubseteq ^z_x\)); i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\subseteq ^z_x\) (resp. \(\sqsubseteq ^z_x\)); i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^z_x\) (resp. \(\sqsubseteq ^z_x\)); i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) (resp. \(\sqsubseteq ^y_x\)) iff i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^z_x\) (resp. \(\sqsubseteq ^z_x\)). proof it follows from theorem 1 and definition 12.   \(\square \) when no priority is available (\(\varphi = \emptyset \)), the semantics for \((p, \varphi )\) collapse into the semantics for p (and vice versa): theorem 2 let \((p, \varphi )\) be a \(\textit{plp}\) such that \(\varphi = \emptyset \), i.e., \((p, \varphi )\) corresponds to the \(\textit{nlp}\) p. then \(\forall x \in \left\{ st , opt , pes , opp \right\} \) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds i is a \( ps \) model of p iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is a well-founded model of p iff i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is a regular model of p iff i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is a stable model of p iff i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is an l-stable model of p iff i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). proof since \(\varphi = \emptyset \), \(\varphi ^* = \left\{ o \preceq o \mid o \in \mathcal {o}_p\right\} \) according to \(\varphi ^*\). this means for every \(o, o' \in \mathcal {o}_p\), it holds \(o \not \prec o'\). thus for any \(i \subseteq lit _p\), we obtain \(\mathcal b_i = \mathcal w_i = 2^i\), in which \(\mathcal b_i\) is the set of all best-ranked options in i and \(\mathcal w_i\) is the set of all worst-ranked options in i w.r.t. \(\varphi \). then for any \( ps \) models \(i_1\) and \(i_2\) of p, for any \(o_1 \in \mathcal b_{i_1} = \mathcal w_{i_i} = 2^{i_1}\) and for any \(o_2 \in \mathcal b_{i_2} = \mathcal w_{i_2} = 2^{i_2}\), it holds \(o_2 \not \prec o_1\) and \(o_1 \not \prec o_2\) according to \(\varphi ^*\). hence, for any \(y \in \left\{ \forall \forall , \forall \exists , \exists \forall , \exists \exists \right\} \), we obtain $$\begin{aligned} \begin{gathered} \mathcal b_{i_1} = \mathcal w_{i_i} = 2^{i_1} \subseteq ^y \mathcal b_{i_2} = \mathcal w_{i_2} = 2^{i_2} \textit{ and }\\ \mathcal b_{i_2} = \mathcal w_{i_2} = 2^{i_2} \subseteq ^y \mathcal b_{i_1} = \mathcal w_{i_1} = 2^{i_1} \end{gathered} \end{aligned}$$ (1) from eq. (1), we infer for each \(x \in \left\{ st , opt , pes , opp \right\} \) and for all \(y \in \left\{ \forall \forall , \forall \exists , \exists \forall , \exists \exists \right\} \), it holds \(i_1 \subseteq ^y_x i_2 \textit{ and } i_2 \subseteq ^y_x i_1\). consequently, for each \(x \in \left\{ st , opt , pes , opp \right\} \) and for all \(y \in \left\{ \forall \forall , \forall \exists , \exists \forall , \exists \exists \right\} \), it holds i is a \( ps \) model of p iff i is a \( pps \) model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\). after replacing above \(i_1\) by \(i_1 i_2\) and \(i_2\) by \(i_2 i_i\), we obtain the same result for \(\sqsubseteq ^y_x\).    \(\square \) the following results are immediate: corollary 2 let \((p, \varphi )\) be a \(\textit{plp}\) such that \(\varphi = \emptyset \), i.e., \((p, \varphi )\) corresponds to the \(\textit{nlp}\) p. then \(\forall x \in \left\{ st , opt , pes , opp \right\} \) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds i is a well-founded model of p iff i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential well-founded model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is a regular model of p iff i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential regular model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is a stable model of p iff i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential stable model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). i is an l-stable model of p iff i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\subseteq ^y_x\) iff i is a preferential l-stable model of \((p, \varphi )\) w.r.t. \(\sqsubseteq ^y_x\). proof these results follow from definitions 3 and 12 and theorem 2. both the relations \(\subseteq ^y_x\) and \(\sqsubseteq ^y_x\) have neither a monotonic nor an antitonic behaviour w.r.t. \(\varphi \): theorem 3 let \((p, \varphi _1)\) and \((p, \varphi _2)\) be \(\textit{plp}\)s. if \(\varphi _1 \subseteq \varphi _2\), then \(\forall x \in \{ st , opt ,\) \( pes , opp \}\) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it does not hold in general 1. \(\mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x}\)) (monotonic). 2. \(\mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x}\)) (antitonic). proof let p be the \(\textit{nlp}\) \(\left\{ a \leftarrow \mathtt {not\ }b \quad b \leftarrow \mathtt {not\ }a\right\} \), whose \( ps \) models are \(\emptyset \), \(\left\{ a\right\} \) and \(\left\{ b\right\} \). firstly, we will show a counterexample to the statement \(\forall x \in \{ st , opt ,\) \( pes , opp \}\) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x}\)) : let \(\varphi _1 = \emptyset \) and \(\varphi _2 = \left\{ \emptyset \preceq \left\{ b\right\} , \left\{ b\right\} \preceq \left\{ a\right\} \right\} \). we have \(\forall x \in \{ st , opt ,\) \( pes , opp \}\) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} = \{ \emptyset , \left\{ a\right\} ,\) \( \left\{ b\right\} \}\) and \(\mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} = \left\{ \left\{ a\right\} \right\} \). consequently, $$\begin{aligned} \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} \not \subseteq \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} \end{aligned}$$ now we will show a counterexample to \(\forall x \in \{ st , opt ,\) \( pes , opp \}\) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} \subseteq \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x}\)): let \(\varphi _1 = \left\{ \emptyset \preceq \left\{ b\right\} , \left\{ b\right\} \preceq \left\{ a\right\} \right\} \) and \(\varphi _2 = \left\{ \emptyset \preceq \left\{ b\right\} , \left\{ b\right\} \preceq \left\{ a\right\} , \left\{ a\right\} \preceq \emptyset \right\} \). we have \(\forall x \in \{ st , opt ,\) \( pes , opp \}\) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} = \left\{ \left\{ a\right\} \right\} \) and \(\mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} = \left\{ \emptyset , \left\{ a\right\} , \left\{ b\right\} \right\} \). consequently, $$\begin{aligned} \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x} \not \subseteq \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} \end{aligned}$$ we end up this section by showing inconsistency options do not interfere in the process of selecting the \( pps \) models: theorem 4 (invariant to inconsistency) let \((p, \varphi _1)\) and \((p, \varphi _2)\) be \(\textit{plp}\)s such that \(\varphi _2 = \left\{ o_1 \preceq o_2 \mid o_1 \preceq o_2 \in \varphi _1 \textit{ and both } o_1 \textit{ and } o_2 \textit{ are consistent}\right\} \). then \(\forall x \in \left\{ st , opt , pes , opp \right\} \) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x}\) (resp. \(\mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x} = \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x}\)). proof let \(i_1\) and \(i_2\) be a \( ps \) model of p. we will show \(i_1 \in \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x}\) iff \(i_1 \in \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x}\) (resp. \(i_1 \in \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x}\) iff \(i_1 \in \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x}\)). as no \( ps \) model is inconsistent, the inconsistent options are neither in the set \(\mathcal b_{i_j}\) of all best-ranked options of \(i_j\) nor in the set \(\mathcal w_{i_j}\) of all worst-ranked options of \(i_j\) with \(j \in \left\{ 1,2\right\} \). this means for each \(o_1 \in \mathcal b_{i_1} \cup \mathcal w_{i_1}\) and for each \(o_2 \in \mathcal b_{i_2} \cup \mathcal w_{i_2}\), it holds \(o_2 \not \prec o_1\) according to \(\varphi _1^*\) iff \(o_2 \not \prec o_1\) according to \(\varphi _2^*\). then \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(\mathcal b_{i_1} \subseteq ^y \mathcal b_{i_2}\) in \(\varphi _1\) iff \(\mathcal b_{i_1} \subseteq ^y \mathcal b_{i_2}\) in \(\varphi _2\), \(\mathcal b_{i_1} \subseteq ^y \mathcal w_{i_2}\) in \(\varphi _1\) iff \(\mathcal b_{i_1} \subseteq ^y \mathcal w_{i_2}\) in \(\varphi _2\), \(\mathcal w_{i_1} \subseteq ^y \mathcal b_{i_2}\) in \(\varphi _1\) iff \(\mathcal w_{i_1} \subseteq ^y \mathcal b_{i_2}\) in \(\varphi _2\) and \(\mathcal w_{i_1} \subseteq ^y \mathcal w_{i_2}\) in \(\varphi _1\) iff \(\mathcal w_{i_1} \subseteq ^y \mathcal w_{i_2}\) in \(\varphi _2\), i.e., \(\forall x \in \left\{ st , opt , pes , opp \right\} \) and \(\forall y \in \left\{ \forall \forall ,\forall \exists ,\exists \forall ,\exists \exists \right\} \), it holds \(i_1 \subseteq _x^y i_2\) in \(\varphi _1\) iff \(i_1 \subseteq _x^y i_2\) in \(\varphi _2\). hence, \(i_1 \in \mathcal {pps}(p,\varphi _1)_{\subseteq ^y_x}\) iff \(i_1 \in \mathcal {pps}(p,\varphi _2)_{\subseteq ^y_x}\). similarly we can prove \(i_1 \in \mathcal {pps}(p,\varphi _1)_{\sqsubseteq ^y_x}\) iff \(i_1 \in \mathcal {pps}(p,\varphi _2)_{\sqsubseteq ^y_x}\).    \(\square \) 6 related work the first studies on preference information in non-monotonic reasoning involved the idea of specificity [22]: given two conflicting conclusions obtained from the same initial information but resorting to different inference rules, one should prefer the more specific. following a related point of view, in [23], they introduced preference between predicates in the context of circumscription. preference (usually based on simplicity measures) is also employed in [24] to select the most preferable explanations in abductive systems. with respect to preferences in logic programming, we find a plethora of proposals (see [2,3,4,5,6,7,8,9,10,11,12] for a non exhaustive list). in [2] and [3], they extend respectively the well-founded semantics and the answer sets semantics [25] for logic programs to represent preference between rules in the language and resort to such information to obtain new conclusions. unlike the semantics in [3], which allows multiple preferred answer sets even for some fully prioritised programs, the semantics in [10] selects at most one preferred answer set for all fully prioritised programs. furthermore, for a large class of programs guaranteed to have an answer set, the existence of a preferred answer set is also guaranteed. in ordered logic programming [5], cr-prolog [6], logic programming with ordered disjunction [7], answer set optimization [8], possibilistic answer set programming [9], preference is handled to evaluate the preferred answer sets by specifying the precedence over the rules or the literals in rules heads. a logic programming language preflog based on an infinite-valued logic has been conceived in [11] to support operators for expressing preferences. despite the infinite-valued truth domain, it can be defined a terminating bottom-up proof procedure for implementing a significant fragment of the language. for continuous operators, a least herbrand model is guaranteed to exist. in [12], it is proposed a logic programming paradigm to combine non-monotonic reasoning with epistemic preferential reasoning. in this proposal, the relation of preferences between atoms can occur in the body of the rules and can be employed to possibly generate new models. an approach closer in spirit to ours is [4], where the authors introduced a framework of prioritised logic programming to represent priority information explicitly in a program. differently of us, however, they restrain the preference relation to literals only, whereas we have defined the preference relation over sets of literals. as consequence, in order to express in a \(\textit{plp}\) \((p, \varphi )\) the complex forms of preferences expounded in sect. 4, they have to change the program p with the introduction of new atoms as well as the set \(\varphi \) of priorities. furthermore, they defined a unique semantics to determine the preferred answer sets. in contradistinction, we introduced 32 different criteria (semantics) to establish preference between partial stable models (which allow defining 32 different ways of selecting the preferred answer sets). in this sense, our work offers a more comprehensive and systematic treatment to deal with preferences in logic programming. 7 conclusion and future works prioritised logic programs (\(\textit{plp}\)s) [4] are obtained from normal logic programs (\(\textit{nlp}\)) by including priorities between literals (an atom or its negation). in this paper, we have presented a more general notion of \(\textit{plp}\) when compared with that version in [4]. our proposal has two main distinguishing features: a) it encompasses not only priorities between literals (atoms or its negation), but also between sets of literals. as consequence, we can represent compactly complex forms of preference. b) concerning its semantics, we have provided a comprehensive and systematic mechanism to select the most preferable partial stable models (\( pps \) models). indeed, by combining well-known criteria found in the literature, we have defined 32 different manners of selecting the \( pps \) models. as the most important semantics in \(\textit{nlp}\) are based on partial stable models, we are also equipped with 32 different manners of selecting the most preferred models according to them. besides, we have proved some properties of these \(\textit{plp}\)s and guaranteed these semantics for \(\textit{plp}\) generalise their counterparts for \(\textit{nlp}\). future directions include investigating computational complexity issues on \(\textit{plp}\)s and exploiting the consequence of introducing preference on paraconsistent semantics found in logic programming. we also intend to study how to work with dynamic preferences along with the resulting semantics. references pigozzi, g., tsoukias, a., viappiani, p.: preferences in artificial intelligence. ann. math. artif. intell. 77(3), 361–401 (2016) google scholar  brewka, g.: well-founded semantics for extended logic programs with dynamic preferences. j. artif. intell. res. 4, 19–36 (1996) google scholar  brewka, g., eiter, t.: preferred answer sets for extended logic programs. artif. intell. 109(1–2), 297–356 (1999) google scholar  sakama, c., inoue, k.: prioritized logic programming and its application to commonsense reasoning. artif. intell. 123(1–2), 185–222 (2000) google scholar  schaub, t., wang, k.: a semantic framework for preference handling in answer set programming. arxiv preprint cs/0301023 (2003) google scholar  balduccini, m., gelfond, m.: logic programs with consistency-restoring rules. in: international symposium on logical formalization of commonsense reasoning, aaai 2003 spring symposium series, vol. 102 (2003) google scholar  brewka, g., niemelä, i., syrjänen, t.: logic programs with ordered disjunction. comput. intell. 20(2), 335–357 (2004) google scholar  brewka, g.: answer sets and qualitative optimization. logic j. igpl 14(3), 413–433 (2006) google scholar  nicolas, p., garcia, l., stéphan, i., lefèvre, c.: possibilistic uncertainty handling for answer set programming. ann. math. artif. intell. 47(1), 139–181 (2006) article  mathscinet  google scholar  gabaldon, a.: a selective semantics for logic programs with preferences. in: del cerro, l.f., herzig, a., mengin, j. (eds.) jelia 2012. lncs (lnai), vol. 7519, pp. 215–227. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-33353-8_17 rondogiannis, p., troumpoukis, a.: expressing preferences in logic programming using an infinite-valued logic. in: proceedings of the 17th international symposium on principles and practice of declarative programming, pp. 208–219 (2015) google scholar  zhang, z.: introspecting preferences in answer set programming. in: 34th international conference on logic programming (iclp 2018) (2018) google scholar  kaci, s.: working with preferences: less is more. springer science and business media, 2011. https://doi.org/10.1007/978-3-642-17280-9_5 przymusinski, t.: the well-founded semantics coincides with the three-valued stable semantics. fundam. inf. 13(4), 445–463 (1990) google scholar  eiter, t., leone, n., sacca, d.: on the partial semantics for disjunctive deductive databases. ann. math. artif. intell. 19(1), 59–96 (1997) article  mathscinet  google scholar  van der t., leendert, w.e.: parameters for utilitarian desires in a qualitative decision theory. appl. intelli. 14(3), 285–301 (2001) google scholar  benferhat, s., dubois, d., kaci, s., prade, h.: bipolar representation and fusion of preferences on the possibilistic logic framework. kr, 2, 421–432 (2002) google scholar  benferhat, s., dubois, d., kaci, s., prade, h.: bipolar possibilistic representations. in: 18th international conference on uncertainty in artificial intelligence, pp. 45–52 (2002) google scholar  lang, j., van der torre, l., weydert, e.: utilitarian desires. auton. agents multi-agent syst. 5(3), 329–363 (2002) google scholar  benferhat, s., dubois, d., kaci, s., prade, h.: bipolar possibility theory in preference modeling: representation, fusion and optimal solutions. inf. fusion 7(1), 135–150 (2006) google scholar  bistarelli, s., pini, m., rossi, f., venable, k.: from soft constraints to bipolar preferences: modelling framework and solving issues. j. exp. theoret. artif. intell. 22(2), 135–158 (2010) google scholar  poole, d.: on the comparison of theories: preferring the most specific explanation. in: 9th international joint conference on artificial intelligence, vol. 85, pp. 144–147 (1985) google scholar  lifschitz, v.: computing circumscription. in: international joint conference on artificial intelligence (ijcai), vol. 85, pp. 121–127 (1985) google scholar  eiter, t., gottlob, g.: the complexity of logic-based abduction. j. acm (jacm) 42(1), 3–42 (1995) google scholar  gelfond, m., lifschitz, v.: classical negation in logic programs and disjunctive databases. new gener. comput. 9(3–4), 365–385 (1991) google scholar  download references author information authors and affiliations department of computer science, federal university of ceará, fortaleza, brazil renan cordeiro, guilherme fernandes & joão alcântara instituto federal de educação, ciência e tecnologia do ceará (ifce), ce-040, km 137,1 s/n, aracati, rodovia, ce, brazil henrique viana authors renan cordeiroview author publications search author on:pubmed google scholar guilherme fernandesview author publications search author on:pubmed google scholar joão alcântaraview author publications search author on:pubmed google scholar henrique vianaview author publications search author on:pubmed google scholar corresponding author correspondence to joão alcântara . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper cordeiro, r., fernandes, g., alcântara, j., viana, h. (2021). a systematic approach to define semantics for prioritised logic programs. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_20 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords logic programming preferences semantics publish with us policies and ethics profiles renan cordeiro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature classification of facial images to assist in the diagnosis of autism spectrum disorder: a study on the effect of face detection and landmark identification algorithms | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper classification of facial images to assist in the diagnosis of autism spectrum disorder: a study on the effect of face detection and landmark identification algorithms conference paper first online: 12 october 2023 pp 261–275 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) classification of facial images to assist in the diagnosis of autism spectrum disorder: a study on the effect of face detection and landmark identification algorithms download book pdf download book epub gabriel c. michelassi9, henrique s. bortoletti9, tuany d. pinheiro9, thiago nobayashi9, fabio r. d. de barros9, rafael l. testa9, andréia f. silva10, mirian c. revers10, joana portolese10, helio pedrini11, helena brentani10, fatima l. s. nunes9 & … ariane machado-lima9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 630 accesses 8 citations abstract since facial morphology can be linked to brain developmental problems, studies have been conducted to develop computational systems to assist in the diagnosis of some neurodevelopmental disorders based on facial images. the first steps usually include face detection and landmark identification. although there are several libraries that implement different algorithms for these tasks, to the best of our knowledge no study has discussed the effect of choosing these ready-to-use implementations on the performance of the final classifier. this paper compares four libraries for facial detection and landmark identification in the context of classification of facial images for computer-aided diagnosis of autism spectrum disorder, where the classifiers achieved 0.92, the highest f1-score. the results indicate that the choice of which facial detection and landmark identification algorithms to use do in fact affect the final classifier performance. it appears that the causes are related to not only the quality of face and landmark identification, but also to the success rate of face detection. this last issue is particularly important when the initial training sample size is modest, which is usually the case in terms of classification of some syndromes or neurodevelopmental disorders based on facial images. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a review of image-based automatic facial landmark identification techniques article open access 15 september 2018 fast and precise face alignment and 3d shape reconstruction from a single 2d image chapter © 2016 detecting face morphing attacks by analyzing the directed distances of facial landmarks shifts chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. autism spectrum disorders biometrics categorization computer vision neurodevelopmental disorders object recognition 1 introduction anthropometry is the science that studies the measurements of the human body including height, weight and size of body components [11]. the measurements can be done directly, with measuring equipment such as measuring tapes, or indirectly, using radiography images, three-dimensional images captured by stereophotogrammetry or two-dimensional images captured by digital cameras [31]. several studies have suggested that facial morphology can be linked to brain developmental problems [2, 6, 9, 13, 32]. for instance, studies of facial anthropometric measures in individuals with autism spectrum disorder (asd) and individuals with typical development (td) have revealed significant differences between these two groups, such as the distance between the pupils, ear format, strabismus, head circumference [2, 23, 27, 29]. these results motivate the use of machine learning approaches to classify asd versus td individuals, based on anthropometric facial features, to aid asd diagnosis. asd is a neurodevelopmental, polygenic and multifactorial disorder with evidence of genetic and environmental factors contributing to its etiology [10, 14, 20, 24]. individuals with asd exhibit persistent social communication impairments, and also lack the ability to develop, maintain and understand relationships, and show restricted and repetitive patterns of behavior, interests, and activities. these symptoms appear during childhood and can impair the person’s daily life [3, 18]. an early and correct diagnosis is crucial for appropriate intervention. the diagnosis is clinical and expensive, since it requires trained health professionals to apply the instruments, mostly questionnaires, and observation routines of the child’s behavior [22]. therefore, to increase accessing the asd diagnosis, computer-aided diagnosis systems could be used for large-scale screening for posterior analysis by a specialist. in order to decrease the financial cost and facilitate using such systems, the use of two-dimensional facial images captured by digital cameras or smartphones is preferred than, for instance, three-dimensional images captured by stereophotogrammetry devices [2, 12, 26], since they can be more easily achieved considering the popularization of mobile devices. in fact, promising results related to neurodevelopmental disorders have been achieved using two-dimensional facial images captured by digital cameras [16, 33]. the computational pipeline of analysis usually consists of three steps: 1) face detection and landmark identification, 2) feature extraction and 3) classifier induction. although there are several libraries that implement different algorithms for the first step, to the best of our knowledge no study has discussed the effect of the choice of these ready-to-use implementations on the performance of the final classifier. this paper investigates the effect of using four image processing libraries for facial detection and landmark identification on the classification results of facial images for computer-aided diagnosis of asd. while a face detection algorithm may cause training sample reduction by discarding images when no faces were detected, a landmark detection algorithm may interfere with the correct extraction of features from the anthropometric measurements. both issues can affect the final classifier and are explored in this paper. 2 methods four image processing libraries were compared for face detection and landmark identification tasks. in order to investigate the impact of these libraries on final classification performance, we performed two experiments to test two hypothesis: h1: the number of training images with successful face detection, combined with correct face detection and landmark identification, will affect the performance of the final classifier; h2: using the same training images, applying different detection and landmark identification algorithms affects the performance of the final classifier. both hypotheses were tested by performing the five-modules process depicted in fig. 1. in summary, the images of an initial training set were acquired following the protocol described in sect. 2.1 and pre-processed as described in sect. 2.2 (modules 1 and 2). each preprocessed image was submitted to different face and landmark detection algorithms to identify five landmark sets: one for each image processing library, except for mediapipe which was used twice (module 3, described in sect. 2.3). each landmark set was used to extract a set of geometric features based on the distance between each pair of landmarks (module 4, described in sect. 2.4), resulting in five different feature datasets. in these five datasets, each instance is the feature vector of an image. since the face detection algorithms usually are not able to detect the face in all images (module 3), the five datasets have not the same number of instances. therefore, two experiments were performed: 1) using each dataset exactly as it is; and 2) using the datasets composed of only the feature vectors of the images where the face was detected by all face detection algorithms executed in module 3. therefore, experiment 1 evaluates the effect of the face detection on the results (testing hypothesis h1), whereas experiment 2 evaluates the effect of the landmark identification (in the same faces) on the classifier performance (testing hypothesis h2). these datasets are used separately to train and test several classifiers (module 5, described in sect. 2.5). finally, a statistical analysis was performed for hypotheses testing (sect. 2.6). fig. 1. overview of the proposed method. full size image 2.1 image acquisition protocol all images were captured with a nikon coolpix l120 compact camera using the protocol shown in fig. 2. in each acquisition, the camera was attached to a tripod fixed at a distance of 50 cm from the chair where the volunteer was accommodated. the chair was placed against the wall, and the volunteer was instructed to remain with his back fully resting on the chair. a white wall was chosen as the photo background, where a red band 50 cm by 5 cm was placed in order to be posteriorly used as a reference for normalizing the images and estimating distortion. fig. 2. protocol used to capture the images, showing the correct position of the individual and the band used as reference for normalization. full size image the volunteers were instructed to remain with a facial expression as neutral as possible and five front facial photographs were taken. a video was recorded when neutral expression facial image was not possible or if the face was inclined. thus, a subsequent video analysis allowed selecting the best frame. images were captured from 43 asd and 74 td children and adolescents ranging from 5 to 18 years of age. the diagnosis of the asd participants was confirmed by a reference hospital the clinics hospital of the university of são paulo and the photographs were taken after the patient’s follow-up visit. the td group consisted of volunteers invited by email. all underwent screening by an infant psychiatrist specialized in asd to confirm the absence of asd and/or clinical evidence of genetic syndromes. all participants were authorized by the parents or legal guardians through an informed consent signed at the beginning of each data collection session. this project was approved by the ethics board and of the school of arts, sciences and humanities of university of sao paulo, protocol number 1.669.832 at august 8th, 2016. all experiments were performed in accordance with the ethics board requirements. the informed consent was obtained from all participants and/or their legal guardians. 2.2 image preprocessing and band detection (modules 1 and 2) we acquired more than one frontal image from each volunteer. the best image was selected based on two criteria: facial neutrality and proportion of width and height of the reference band in order to avoid image distortion. the image preprocessing was performed to rotate and crop all the images. the rotation intends to correct the lateral face inclination and was performed according to the following steps: eye region detection using haar cascades; pupil localization; tracing a line passing over the pupil points; calculation of the angle \(\theta \) formed by this line and the horizontal axis; image rotation according to \(\theta \). finally, the images were cropped in order to remove the black background resulting from the rotation operation. the steps for reference band detection were: (i) application of a bilateral filter for image smoothing, (ii) edge detection using the canny method [8], (iii) application of dilation and erosion operations to correct discontinuities on the edges, (iv) detection of the band contour using the method findcontours from opencv library [30] and (v) calculation of the band width and height. 2.3 face detection and landmark identification (module 3) four image processing libraries were tested for the task of face detection and landmark identification: dlib [15], opencv [7], openface [4] and mediapipe [21]. opencv and dlib have multiple face detection algorithms available. this study chose the algorithms based on cnn (convolutional neural network) and dnn (deep neural networks), respectively, since they represent the state of art in image processing. mediapipe uses a face detection algorithm based on blazeface solution [5] and openface also uses a cnn algorithm to detect faces. after the face detection, all images were manually inspected to assure that no mistake was made in this step. at this point, each library may have correctly detected a different number of faces from the initial set of images. the next step is the landmark identification in each detected face. the landmark identification algorithms of dlib, opencv and openface libraries detect 68 landmarks (fig. 3), whereas the mediapipe’s algorithm detects 438 landmarks. thus, two landmark sets were identified using mediapipe: 1) using only the 68 landmarks closest to the ones detected by the first three libraries (mediapipe 68); and 2) using the previous 68 landmarks and additional 61 points corresponding to the facial contour (mediapipe 129) (fig. 3e). the first landmark set is meant to evaluate the influence of the quality of the identification of the same 68 landmarks on the classifier performance, whereas the second landmark set is meant to evaluate the influence of a higher number of landmarks. therefore, five landmark sets were obtained and compared in this study (table 1). fig. 3. the 68 landmarks present in dlibcnn, opencvdnn, openface, mediapipe 68 and mediapipe 129 landmark sets. source image from [1]. full size image table 1. landmark sets compared in this study. each landmark set was extracted using an image processing library, face detection algorithm, landmark identification algorithm, resulting in a specific number of identified landmarks. the landmark identification algorithm constrained local neural fields is referred to as clnf. the landmark identification algorithm local binary fitting is referred to as lbf.full size table 2.4 feature extraction and experiment split (module 4) for each landmark set output from module 3 (table 1), the euclidean distance between each pair of landmarks, normalized by the reference band dimensions, was calculated, corresponding to a feature. more specifically, for each pair of landmarks \(p (x_1, y_1)\) and \(q (x_2, y_2)\), each landmark represented by a coordinate (x, y), the normalized euclidean distance \(d_n\) is calculated according to eq. 1, where w and h are the reference band width and height (calculated as described in sect. 2.2), respectively: $$\begin{aligned} d_n(p, q) = \sqrt{[(x_1 x_2) / w]^2 + [(y_1 y_2) / h]^2} \end{aligned}$$ (1) the feature dataset obtained from each landmark set described in table 1 was used in experiment 1. in experiment 2, only features from images where the face was correctly detected by all face detection algorithms were used. 2.5 classifier development (module 5) the last module of the proposed pipeline (fig. 1) consists in training and evaluating several classifiers varying the procedures for dataset preprocessing, dimension reduction and classifier induction. here we describe these procedures and how the parameter tuning and performance evaluation were performed. dataset preprocessing. three preprocessing tasks were applied on the dataset: initial feature filtering, min-max normalization and class balancing. in the initial feature filtering, when two features were highly correlated (pearson correlation > 0.98), only one was kept. min-max normalization was applied on the feature values. for this, after identification of the maximum and minimum values of each feature x in the dataset (\(x_{max}\) and \(x_{min}\), respectively), the original value \(x_i\) is transformed into a new value \(x'_i = (x_i x_{min})/(x_{max} x_{min})\). for class balancing, four approaches were considered: i) undersampling, ii) oversampling with four variants of smote algorithm [25] (default, svm, borderline and kmeans smote [17]), iii) combination of under and over sampling with smote tomek, and iv) no balancing. dimension reduction. dimension reduction methods are extremely important for classification tasks because they eliminate redundant features and can increase classifier performance. filter-type feature selection methods tend to have a lower computational cost than wrapper-type methods. in this paper, the implementation of the scikit-learn libraries [28] and skfeature [19] libraries were used and seven different algorithms were applied: principal component analysis (pca), which performs feature extraction by transforming the initial set into a new set containing only the major components; minimum redundancy maximum relevance (mrmr), which performs the selection of features based on two criteria: one is the minimum criterion of redundancy between pairs of features and the other is the maximum relevance through the measurement of information; correlation-based feature selection (cfs), which performs the selection of features considering that a suitable subset contains features highly correlated with the target class, but little correlated with each other; fast correlation-based filter (fcbf), which performs the selection of features considering the main idea of the cfs method, but calculating the correlation measure based on the entropy concept of information theory; relieff, which performs the selection of features by means of a relevance ranking of each feature; robust feature selection (rfs), which performs the feature selection through sparse dictionary learning; random forest select, which performs the feature selection based on the random forest classifier. classifier induction (hyperparameter tuning and performance estimation). six classifier induction algorithms were used: k-nearest neighbors (knn), support vector machines (svm), gaussian naïve bayes (nb), neural networks (nn), random forests (rf) and linear discriminant analysis (lda). hyperparameter tuning of the classifier induction algorithms was performed using a grid search approach to select the hyperparameter values that maximize the classifier f1-score, estimated by stratified 10-fold cross-validation. in each fold of the cross-validation, the synthetic instances produced by the class balancing algorithm were excluded from the test sample. table 2 shows the hyperparameters and the tested values that were tuned for each classifier. default parameters were used for gaussian naive bayes classifier. table 2. parameters and values tested for each classifier. n_features: the number of features after dimension reduction. the parameter hidden_layer_sizes is specified as an ordered pair (x, y) where x is the number of hidden layers and y is the number of neurons per layer.full size table in summary, all combinations of dataset balancing strategy, dimension reduction and classifier induction algorithm (table 3) were evaluated, totaling 294 combinations. to measure performance, the f1-score of each classifier resulting from each combination was estimated via stratified 10-fold cross-validation, with no synthetic data in the test folds. in addition, confidence intervals were calculated with 95% confidence. table 3. dataset balancing strategies, dimension reduction and classifier induction algorithms used in this study.full size table 2.6 statistical analysis and comparison as described in sect. 2.5, several classifiers were inducted using different combinations of class balancing algorithms, dimension reduction and classifier induction algorithms (table 3). considering the ten folds of the cross-validation, 2940 classifiers were inducted from each dataset (i.e., from each landmark set). comparing the f1-scores of learned classifiers using each of these combinations in each fold, varying only the landmark set (table 1), constitutes a test to detect differences in “treatments” across multiple test attempts, where the “treatments” are the use of different libraries for face detection and landmark identification. therefore, a table \(f_{2940\times 5}\) was created where f[i, j] is the f1-score obtained by the classifier inducted using the combination i of class balancing algorithms, dimension reduction, classifier induction algorithm and fold using the landmark set j. as the f1-scores do not present a normal distribution, friedman’s non-parametric test was applied using the f1-scores obtained in experiments 1 and 2 to test the hypotheses h1 and h2, respectively. considering a critical p-value of 0.05, an inferior p-value means that there is a significant difference between the f1-score obtained using different libraries for face detection and landmark detection. table 4. number of images represented in each landmark set in the two experiments. rows corresponding to “experiment 1” describe the number of faces detected by each library (*: mediapipe68 and mediapipe129 are landmark sets based on the same faces extracted by mediapipe face detection algorithm). rows corresponding to “experiment 2” describe the number of faces detected by all libraries.full size table in addition, in order to investigate if the h1 and h2 test results can be dependent on the classifier induction algorithm, the same test was repeated using only the f1-scores of classifiers inducted by the same algorithms. for these tests a bonferroni-holm correction was applied to correct the p-values due to the execution of the six tests. the friedman test only evaluates if the average f1-score is significantly different among the five landmark sets. to test if there is a significant difference between each pair of landmark sets, the wilcoxon test was applied. 3 results and discussion table 4 (rows corresponding to “experiment 1”) shows the number of images where the face was correctly detected by each library. from an initial set of 117 images (43 images from asd and 74 images from td individuals), the face detection algorithms performed differently, detecting the face in 63% to 89% of the images. mediapipe was the library that detected fewer faces, whereas opencv using the dnn algorithm was the library that detected more faces. only 62 images (53%) had the face correctly detected by all libraries. these images were used to compose the five landmark sets in experiment 2 (table 4 rows corresponding to “experiment 2”). table 5 shows the p-values resulting from the friedman test performed on all inducted classifiers, as well as for each classifier induction algorithm separately, for both experiments. considering all learned classifiers (table 5, row “all classifiers”), the results show that there is a significant difference between the f1-scores obtained when using different landmark sets (i.e., different libraries for face detection and landmark identification), in both experiments 1 and 2 (p-values < 0.001), allowing that h1 and h2 are accepted. that means that both correct face detection and landmark identification have a significant effect on the final classification. all p-values are also below the critical value of 0.05 considering the test performed for each classifier induction separately (table 5, rows “svm” to “lda”). however, three classifier induction algorithms had their p-values closer to the critical value: svm in experiment 1 and naive bayes and random forests in experiment 2. this may indicate that different classifier induction algorithms have different susceptibility to data variation obtained using these libraries. however, more experiments should be conduct to investigate this issue. table 5. results of the friedman tests considering the 2940 generated classifiers per landmark set (“all classifiers”) and per classifier induction algorithm.full size table table 6 shows the p-values achieved using the wilcoxon test for each pair of landmark sets in both experiments. in experiment 1, the results show that the f1-scores achieved using the dlibcnn and opencvdnn landmark sets are not significantly different, as well as using mediapipe 68 and mediapipe 129. dlibcnn and opencvdnn are the landmark sets based on the highest number of images (101 and 104 detected faces, respectively table 4) whereas mediapipe 68 and mediapipe 129 landmark sets are based on exactly the same images. in experiment 2, where all landmark sets were extracted from the same 62 images, all p-values were \(< 0.05\), indicating that f1-scores are significantly different between each pair of landmark sets. these results indicate that the number of training images not only affect the classifier performance but also the quality of face and landmark identification. in addition, the significant difference between mediapipe 68 and mediapipe 129 only in experiment 2 may indicate that the additional landmarks in mediapipe 129 improved the classifier performance when there was a smaller number of training images. table 6. results of the wilcoxon tests.full size table considering the use of these landmark sets to the problem of asd x td classification, fig. 4 shows the highest f1-score (estimated by 10-fold cross-validation) achieved using each landmark set in each experiment. openface, opencvdnn and dlibcnn had the best results in experiment 1, highlighting that openface and dlibcnn presented the narrowest confidence intervals. in fact, in experiment 1 these three landmark sets are those based on the highest number of detected faces (table 4). it is noteworthy that, although based on a lower number of images (95 detected faces), the openface landmark set had the highest f1-score (0.9184) with an accuracy of 0.8861, precision of 0.8548 and recall of 1.0. this result was achieved using smote tomek for class balancing, pca for dimension reduction and lda for classifier induction algorithm. the lowest f1-score (0.8436) was achieved using mediapipe 64, which is consistent with the fact that this landmark set is based on the lowest number of images (74 table 4). mediapipe 129 had a higher f1-score than mediapipe 64 (0.9029), based on the same images but with additional landmarks. however, it was observed that these two landmark sets are not significantly different in the statistical test using all generated classifiers (table 6 experiment 1). such relative performance of these landmark sets (dlibcnn, opencvdnn and openface presenting higher f1-scores than mediapipe 64 and mediapipe 129) was similar in all inducted classifiers (data not shown). in experiment 2 all f1-scores decreased, which is consistent with the fact that all landmark sets were based on a lower number of images (62). therefore, it can be seen that not only the correct face detection and reference points are important, but also the number of images used to train a model. the best classifier had a accuracy of 0.88, precision of 0.83, recall of 1.0 and f1-score of 0.8981. fig. 4. the highest f1-score obtained using each landmark set in experiment 1 (fig a) and experiment 2 (fig b). the bar errors correspond to a 95% confidence interval. full size image 4 conclusion this paper investigated the effect of employing different image processing libraries on the final result of facial image-based classifiers using asd x td classification as case-study. two hypotheses were tested: that different face detection rates (influence the training sample size), in addition to the landmark identification quality, influence the final classification (h1) and that, using exactly the same images, the face detection and landmark identification per se influence the final classification (h2). the results enabled accepting both hypotheses in this case study. the analysis of the pairwise differences, as well as of the f1-scores obtained by the different landmark sets in both experiments, indicate that libraries with a higher success rate in face detection (dlibcnn, opencvdnn and openface) tend to produce higher f1-score, and thus more suitable. finally, considering the problem of asd x td classification, after testing several combinations of image processing libraries, dataset balancing, dimension reduction and classifier induction algorithms, this study achieved the f1-score of 0.9184, accuracy of 0.8861, precision of 0.8548 and recall of 1.0, which is important for computer-aided diagnosis. to enhance asd analysis through face detection, more data from individuals with asd and controls are crucial. references face recognition database (2005). http://cbcl.mit.edu/software-datasets/heisele/facerecognition-database.html aldridge, k., et al.: facial phenotypes in subgroups of prepubertal boys with autism spectrum disorders are correlated with clinical phenotypes. mol. autism 2(1), 15 (2011) article  mathscinet  google scholar  association, a.p., et al.: diagnostic and statistical manual of mental disorders (dsm-5®). american psychiatric pub (2013) google scholar  baltrusaitis, t., robinson, p., morency, l.p.: constrained local neural fields for robust facial landmark detection in the wild. in: proceedings of the ieee international conference on computer vision workshops, pp. 354–361 (2013) google scholar  bazarevsky, v., kartynnik, y., vakunov, a., raveendran, k., grundmann, m.: blazeface: sub-millisecond neural face detection on mobile gpus. corr abs/1907.05047 (2019) google scholar  boehringer, s., et al.: syndrome identification based on 2d analysis software. eur. j. hum. genet. 14(10), 1082–1089 (2006) article  google scholar  bradski, g.: opencv library. dr. dobb’s journal of software tools (2000) google scholar  canny, j.: a computational approach to edge detection. ieee trans. pattern anal. mach. intell. 6, 679–698 (1986) article  google scholar  demyer, w., zeman, w., palmer, c.g.: the face predicts the brain: diagnostic significance of median facial anomalies for holoprosencephaly (arhinencephaly). pediatrics 34(2), 256–263 (1964) article  google scholar  deth, r., muratore, c., benzecry, j., power-charnitsky, v.a., waly, m.: how environmental and genetic factors combine to cause autism: a redox/methylation hypothesis. neurotoxicology 29(1), 190–201 (2008) article  google scholar  farkas, l.g.: anthropometry of the head and face. raven pr (1994) google scholar  gilani, s.z., et al.: sexually dimorphic facial features vary according to level of autistic-like traits in the general population. j. neurodev. disord. 7(1), 14 (2015) article  google scholar  hammond, p., et al.: 3d analysis of facial morphology. am. j. med. genet. a 126(4), 339–348 (2004) article  google scholar  johnson, c.p., myers, s.m.: identification and evaluation of children with autism spectrum disorders. pediatrics 120(5), 1183–1215 (2007) article  google scholar  king, d.e.: dlib-ml: a machine learning toolkit. j. mach. learn. res. 10, 1755–1758 (2009) google scholar  kumov, v., samorodov, a.: recognition of genetic diseases based on combined feature extraction from 2d face images. in: 26th conference of open innovations association (fruct). ieee (2020) google scholar  lemaître, g., nogueira, f., aridas, c.k.: imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. j. mach. learn. res. 18(17), 1–5 (2017) google scholar  levy, s.e., mandell, d.s., schultz, r.t.: autism 374, 1627–1638 (2009) google scholar  li, j., et al.: feature selection: a data perspective. arxiv:1601.07996 (2016) lord, c., cook, e.h., leventhal, b.l., amaral, d.g.: autism spectrum disorders. neuron 28(2), 355–363 (2000) article  google scholar  lugaresi, c., et al.: mediapipe: a framework for perceiving and processing reality. in: third workshop on computer vision for ar/vr at ieee computer vision and pattern recognition (cvpr) 2019 (2019) google scholar  mandell, d.s., novak, m.m., zubritsky, c.d.: factors associated with age of diagnosis among children with autism spectrum disorders. pediatrics 116(6), 1480–1486 (2005) article  google scholar  miles, j., hadden, l., takahashi, t., hillman, r.: head circumference is an independent clinical finding associated with autism. am. j. med. genet. 95(4), 339–350 (2000) article  google scholar  muhle, r., trentacoste, s.v., rapin, i.: the genetics of autism. pediatrics 113(5), e472–e486 (2004) article  google scholar  chawla, n.v., bowyer, k.w., lawrence, o.h., philip kegelmeyer, w.: smote: synthetic minority over-sampling technique. j. artif. intell. res. 16, 321–357 (2011) article  math  google scholar  obafemi-ajayi, t., et al.: facial structure analysis separates autism spectrum disorders into meaningful clinical subgroups. j. autism dev. disord. 45(5), 1302–1317 (2015) article  google scholar  ozgen, h., et al.: morphological features in children with autism spectrum disorders: a matched case-control study. j. autism dev. disord. 41(1), 23–31 (2011) article  google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  rodier, p.m., bryson, s.e., welch, j.p.: minor malformations and physical measurements in autism: data from nova scotia. teratology 55(5), 319–325 (1997) article  google scholar  suzuki, s.: topological structural analysis of digitized binary images by border following. comput. vis. graph. image process. 30(1), 32–46 (1985) article  math  google scholar  weinberg, s.m., naidoo, s., govier, d.p., martin, r.a., kane, a.a., marazita, m.l.: anthropometric precision and accuracy of digital three-dimensional photogrammetry: comparing the genex and 3dmd imaging systems with one another and with direct anthropometry. j. craniofac. surg. 17(3), 477–483 (2006) article  google scholar  zhao, q., et al.: digital facial dysmorphology for genetic screening: hierarchical constrained local model using ica. med. image anal. 18(5), 699–710 (2014) article  google scholar  zhao, q., yao, g., akhtar, f., li, j., pei, y.: an automated approach to diagnose turner syndrome using ensemble learning methods. ieee access 8, 223335–223345 (2020) article  google scholar  download references acknowledgments we thank the patients and their families that allowed the execution of this research, as well as the team of the autism spectrum program of the clinics hospital (protea-hc). this study was funded by brazilian national council of scientific and technological development, (cnpq) (grants 309330/2018-1, 157535/2017-7 and 309030/2019-6) and scientific and technological initiation program at university of sao paulo (pibic/pibit-cnpq/usp 2020/2021), the são paulo research foundation (fapesp) national institute of science and technology medicine assisted by scientific computing (inct-macc) grant 2014/50889-7, sao paulo research foundation (fapesp) grants #2017/12646-3, #2020/01992-0, coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes), dean’s office for research of the university of são paulo (prp-usp, grant 18.5.245.86.7) and the national health support program for people with disabilities (pronas/pcd) grant 25000.002484/2017-17. author information authors and affiliations school of arts, sciences and humanities, university of sao paulo, r. arlindo béttio, 1000 ermelino matarazzo, sao paulo-sp, 03828-000, brazil gabriel c. michelassi, henrique s. bortoletti, tuany d. pinheiro, thiago nobayashi, fabio r. d. de barros, rafael l. testa, fatima l. s. nunes & ariane machado-lima department of psychiatry, school of medicine, university of sao paulo, r. dr. ovídio pires de campos, 785, sao paulo-sp, 05403-903, brazil andréia f. silva, mirian c. revers, joana portolese & helena brentani institute of computing, university of campinas, av. albert einstein, 1251 cidade universitária, campinas-sp, 13083-852, brazil helio pedrini authors gabriel c. michelassiview author publications search author on:pubmed google scholar henrique s. bortolettiview author publications search author on:pubmed google scholar tuany d. pinheiroview author publications search author on:pubmed google scholar thiago nobayashiview author publications search author on:pubmed google scholar fabio r. d. de barrosview author publications search author on:pubmed google scholar rafael l. testaview author publications search author on:pubmed google scholar andréia f. silvaview author publications search author on:pubmed google scholar mirian c. reversview author publications search author on:pubmed google scholar joana portoleseview author publications search author on:pubmed google scholar helio pedriniview author publications search author on:pubmed google scholar helena brentaniview author publications search author on:pubmed google scholar fatima l. s. nunesview author publications search author on:pubmed google scholar ariane machado-limaview author publications search author on:pubmed google scholar corresponding author correspondence to ariane machado-lima . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper michelassi, g.c. et al. (2023). classification of facial images to assist in the diagnosis of autism spectrum disorder: a study on the effect of face detection and landmark identification algorithms. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_18 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords image processing face detection landmark identification anthropometry autism spectrum disorder classification publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature multiobjective evolutionary algorithms applied to the optimization of expanded genetic codes | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper multiobjective evolutionary algorithms applied to the optimization of expanded genetic codes conference paper first online: 12 october 2023 pp 3–16 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) multiobjective evolutionary algorithms applied to the optimization of expanded genetic codes download book pdf download book epub maísa de carvalho silva9, paulo guilherme pinheiro pereira9, lariza laura de oliveira  orcid: orcid.org/0000-0002-5098-172x9 & … renato tinós  orcid: orcid.org/0000-0003-4027-88519  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 529 accesses abstract there is great interest in the creation of genetically modified organisms that use amino acids different from the naturally encoded amino acids. unnatural amino acids have been incorporated into genetically modified organisms to develop new drugs, fuels and chemicals. when incorporating new amino acids, it is necessary to change the standard genetic code. expanded genetic codes have been created without considering the robustness of the code. in this work, multi-objective genetic algorithms are proposed for the optimization of expanded genetic codes. two different approaches are compared: weighted and pareto. the expanded codes are optimized in relation to the frequency of replaced codons and two measures based on robustness (for polar requirement and molecular volume). the experiments indicate that multi-objective approaches allow to obtain a list of expanded genetic codes optimized according to combinations of the three objectives. thus, specialists can choose an optimized solution according to their needs. this work was partially supported by são paulo research foundation fapesp (under grants #2021/09720-2 and #2013/07375-0), national council for scientific and technological development cnpq (under grant #306689/2021-9), and center for artificial intelligence c4ai (supported by fapesp, under grant #2019/07665-4, and ibm corporation). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-objective genetic algorithms chapter © 2024 multi-objective genetic algorithms chapter © 2023 average and standard deviation of the error function for random genetic codes with standard stop codons article 17 december 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. genetic engineering genetic modification genetic techniques genomic engineering protein engineering genetic databases 1 introduction proteins are vital macromolecules in living organisms [3]. they are composed of amino acids joined by covalent bonds forming series with different sizes and constitutions. changes in the amino acid sequence can cause modifications in the three-dimensional structure of the protein and consequently in its function. each amino acid is encoded in the dna (deoxyribonucleic acid) by a sequence of three nucleotides, called codon. sixty-one codons specify amino acids and three codons indicate the end of protein sequencing (stop codons), during its synthesis. there are 20 types of amino acids that are generally used in proteins (natural amino acids). since there are \(4^3 = 64\) possible combinations of the 4 nucleotides (a, c, g, t) in a codon, some amino acids are encoded by more than one codon. living beings share the same standard genetic code, with rare exceptions. there are approximately \(1.4 \times 10^{70}\) hypothetical genetic codes, i.e., ways to associate codons to the natural amino acids. when compared to all codes, the standard genetic code is very robust [5], which is explained by two main factors. first, when the organization of the standard genetic code is examined, one can see that many amino acids are encoded by similar codons (see fig. 1). that is, small changes in the nucleotide sequence sometimes generate no change in the sequence of amino acids of a protein. second, most of the modifications result in changing the amino acid to one with similar physical-chemical properties [14]. recently, there has been great interest in creating genetically modified organisms that use unnatural amino acids, i.e., amino acids other than the 20 amino acids naturally encoded in the standard genetic code. these amino acids can be interesting for many reasons. for example, they may contain heavy atoms that facilitate some crystallographic studies involving x-rays. new amino acids have been incorporated into genetically modified organisms to produce drugs, fuels and chemicals of great economic interest [11]. when adding new amino acids to genetically modified organisms, it is necessary to modify the standard genetic code. expanded genetic codes can be created by using codons with four nucleotides instead of three [1]. another possibility is to add synthetic nucleotides to create new codons [15]. however, the most attractive way to modify the standard genetic code is by replacing the association of some codons with their respective amino acids. in the new genetic code, these codons are now associated to the new amino acids. in general, the codons chosen to have the association replaced are those that are least frequently used in the organism. the robustness is not taken in account when creating the expanded genetic codes. according to the authors’ knowledge, optimization methods have not been used for creating expanded genetic codes, with exception for a previous paper of our group [13]. in [13], we proposed a single-objective genetic algorithm (ga) for optimizing expanded genetic codes. gas have been previously employed for creating hypothetical genetic codes with the aim of investigating the optimality of the standard genetic code. in [12], single-objective gas were used to find robust genetic codes considering only one robustness measure each time. in [9, 10], more than one robustness measure are simultaneously optimized in a multi-objective evolutionary approach. that is, instead of comparing the codes using a single measure of robustness based on a given physical-chemical property, the codes are compared using two or more measures concurrently. using more than one objective results in hypothetical codes more similar to the standard genetic code. in this paper, we propose the use of multi-objective gas for optimizing expanded genetic codes. the expanded codes are optimized in relation to the frequency of replaced codons and two measures based on robustness (for polar requirement and molecular volume). there are two main approaches for multi-objective gas [6]: i) the weighted approach, where the multi-objective problem is transformed into a single-objective problem; ii) the pareto approach, where a set of non-dominated solutions are considered. here, the pareto multi-objective approach is proposed for optimizing expanded genetic codes. the weighted approach was proposed in our previous work [13], but with only two objectives (robustness for polar requirement and frequency of replaced codons); here we consider the weighted approach with the same three objectives employed in the pareto approach. an advantage of the pareto approach over the weighted approach for this problem is that it is not necessary to set weights for each objective. in addition, the multi-objective pareto approach can generate a list of solutions with different properties, rather than just one solution. the rest of this paper is organized as follows. in sect. 2, the proposed gas for optimizing expanded genetic codes are presented. we present in sect. 3 experiments where the gas are compared in a problem where the genetic code should incorporate the codification for a hypothetical amino acid. finally, the conclusions and discussion about future work are presented in sect. 4. fig. 1. standard genetic code. the codon is a three-letter sequence from an alphabet with 4 nucleotides (a, c, g, t). the names of the amino acids are abbreviated in the figure (examples: phe is phenylalanine and leu is leucine). the numbers represent the frequency (f) of codons in e. coli. source: adapted from [8]. full size image 2 proposed genetic algorithms in the proposed gas, individuals represent hypothetical expanded genetic codes that incorporate the codification of a new unnatural amino acid. here, we consider that only one new amino acid is incorporated. however, the algorithms can be easily adapted for problems where more than one amino acid are incorporatedfootnote 1. the two approaches based on gas for the optimization of expanded genetic codes are presented in sects. 2.3 and 2.4. before, the common elements to all the proposed gas are introduced in sects. 2.1 and 2.2. here, the optimization problem is considered a minimization problem. 2.1 codification and operators the binary codification is used for representing an expanded genetic code in the chromosome of an individual of the ga (fig. 2). the chromosome has 61 elements, each one representing a specific codon of the genetic code. the three stop codons (uaa, uag and uga) are not represented in the chromosome. an element equal to 1 in the i-th position of the chromosome means that the respective codon in the standard genetic code will now be related to the new amino acid. an element equal to 0 means that the amino acid in the respective codon is not changed. all amino acids (natural and incorporated) must be present in the expanded genetic codes. in this way, the initial population of the ga is created ensuring that all amino acids are presented in the solutions (expanded genetic codes) associated to the individuals. in addition, when reproduction operators result in the removal of one of the amino acids from the expanded genetic code, the individual is penalized by adding a value of 10,000 to its fitness, making the selection of that individual very rare. the value of 10,000 was determined according to the theoretical maximum and minimal that can be obtained for each objective. here, the two gas use the same reproduction and selection operators. tournament selection and elitism are used to select individuals. in tournament selection, the best of \(s_t\) randomly chosen individuals is selected. in elitism, the best individual of the population is copied to the new population. two-point crossover and flip mutation are used to transform individuals. crossover is applied with rate \(p_c\), while each element of the chromosome is transformed with rate \(p_m\). 2.2 objectives three objectives are considered. two objectives are based on robustness and one objective is the frequency of replaced codons. here, robustness of a genetic code c is defined as the inverse of \(m_{st}(c)\), that is the mean square change for the values of a given property regarding mistranslation and base position errors [7, 9]. the equation for \(m_{st}(c)\) is: $$\begin{aligned} m_{st}(c) = \frac{\sum _{ij} a(i,j)(x(i,c)-x(j,c))^2 }{\sum _{ij}n(i,j)} \end{aligned}$$ (1) fig. 2. example of representation of the expanded genetic code in the chromosome of an individual of the ga. in the expanded genetic code, the amino acid new replaces the naturally encoded amino acids in the positions with elements equal to 1 in the chromosome of the ga’s individual. full size image where x(i, c) is the amino acid property value for the amino acid codified by the i-th codon of the genetic code c, n(i, j) is the number of possible replacements between codons i and j, and a(i, j) is a weight for the change between amino acids codified by the i-th and j-th codons. by minimizing \(m_{st}(c)\) for a given amino acid property, we are selecting a more robust genetic code regarding mistranslation and base position errors. the three objectives are: \(f_{f}(\textbf{x})\):: given by the sum of frequency (codon usage) of the codons that encode the new amino acid in the genetic code c codified by chromosome \(\textbf{x}\), i.e.,: $$\begin{aligned} f_{f}(\textbf{x}) = \sum _{i=1}^{61} x(i) \phi (i) \end{aligned}$$ (2) where \(\phi (i)\) is the codon usage (frequency) of the i-th codon, considering organism e. coli. the codon usage of e. coli is given in fig. 1. e. coli is a prokaryotic model organism very important in applications of biotechnology. this objective is considered in order to avoid codes with many replacements, specially in codons that are often used. additional codon replacements incur in higher economic cost because new biological molecules must be designed and utilized. besides, codon replacements can lead to unwanted biological effects. \(f_{pr}(\textbf{x})\):: given by \(m_{st}(c)\) (eq. 1) considering polar requirement of the amino acids (table 1). when a new amino acid is incorporated, the polar requirement of this amino acid is also used in eq. 2. polar requirement is a very important property of amino acids regarding structure and function of proteins. \(f_{mv}(\textbf{x})\):: given by \(m_{st}(c)\) (eq. 1) considering molecular volume of the amino acids (table 1). molecular volume is also an important property of amino acids regarding structure and function of proteins. in the experiments, values of \(f_{f}\), \(f_{pr}\), and \(f_{mv}\) are presented. in the weighted approach, the evaluation of each objective is given by the normalization by maximum values. for simplicity, we use the same symbols here for the original and normalized values. table 1. polar requirement and molecular volume for the natural amino acids [7].full size table 2.3 weighted approach in this approach, each evaluation of objective is calculated separately. then, weights are assigned according to the importance of each objective. thus, the multi-objective problem is transformed into a single-objective problem. here, the fitness of a solution (genetic code) codified by chromosome \(\textbf{x}\) is given by: $$\begin{aligned} f(\textbf{x}) = w_{f} f_{f}(\textbf{x}) + w_{pr} f_{pr}(\textbf{x}) + w_{mv} f_{mv}(\textbf{x}) \end{aligned}$$ (3) where \(w_{f}\), \(w_{pr}\), and \(w_{mv}\) are the weights respectively associated to \(f_{f}(\textbf{x})\), \(f_{pr}(\textbf{x})\), and \(f_{mv}(\textbf{x})\). the ga used in the weighted approach is here called wga. three versions of wga are tested, each one with different values for the weights (table 2). in wga1, all weights are equal, i.e., no objective is prioritized. in wga2, \(w_{pr}\) is higher, i.e., \(f_{pr}(\textbf{x})\) is prioritized. in wga3, \(w_{f}\) is higher, i.e., \(f_{f}(\textbf{x})\) is prioritized. table 2. weights for the different versions of wga.full size table 2.4 pareto approach this approach uses the concept of pareto dominance in order to obtain a subset of non-dominated solutions to a multi-objective problem. according to the pareto dominance concept, a solution \(\mathbf {x_a}\) dominates a solution \(\mathbf {x_b}\) if it is better in at least one of the objectives, and it is not worse in any of the objectives. there are many multi-objective gas that uses the pareto approach [2]. here, the nondominated sorting genetic algorithm ii (nsga-ii) [4] is used because of two main reasons. first, nsga-ii has good performance when the number of objectives is not high. nsga-ii presents a worst-case time complexity of \(o(m n^2)\) per generation for problems with n objectives and when the population size is equal to m. in addition, it has a mechanism for maintenance of population diversity. second, it was used in [9] for investigating the genetic code adaptability, a problem similar to the expanded genetic code optimization problem. despite being similar, the chromosome codification used in [9] is different from the codification used here. the reproduction operators, objectives, and other characteristics are different too. the nsga-ii algorithm used here can be summarized as follows [9]: i. a population p(0) is randomly generated and sorted in layers (fronts) according to the pareto dominance. thus, the first layer is formed by solutions which are not dominated by other solutions, i.e., the best pareto optimal solution set found so far. ii. at iteration t, the population p(t) is transformed into population q(t) by using selection and reproduction operators. the next step is to sort the union population, \(p(t)+q(t)\), according to the pareto dominance. iii. a new population \(p(t+1)\) is created by merging the layers of p(t) and q(t). when the number of individuals in the last layer exceeds the population size, the crowding distance is used to select the most diverse individuals. here, the objectives of the nsga-ii are the same used in the weighted approach: \(f_{f}(\textbf{x})\), \(f_{pr}(\textbf{x})\), and \(f_{mv}(\textbf{x})\). 3 experiments 3.1 experimental design experiments were performed considering the insertion of one hypothetical amino acid, named here new. the values of polar requirement and molecular volume for the new amino acid were obtained by averaging all respective values for the natural amino acids (see table 1). experiments, not shown here, with other hypothetical amino acids were also done. most parameters of the gas are equal to those used in [9]; preliminary tests were carried out in order to adjust the other parameters. the gas (in the two approaches) have population size equal to \(m=100\) individuals. the number of runs is 10, each one with a different random seed. the number of generations is 120. for all gas, the same parameters for tournament selection and reproduction are used: \(s_t=3\), \(p_c=0.6\), and \(p_m=0.01\). the results of the best solutions (expanded genetic codes) obtained in the runs of the different versions of wga are presented. for nsga-ii, the results of the pareto front obtained by applying the dominance criterion in the union of first front solutions for each run is presented. 3.2 experimental results the results for the evaluation of the best solutions obtained by the 3 versions of wga (sect. 2.3) are presented in table 3. when the different versions of wga are compared, the best solutions for \(f_{f}\) are those obtained by wga3, as expected. the best solutions for \(f_{pr}\) and \(f_{mv}\) were obtained by wga1, where the weights are equal. it is interesting that the solution with best \(f_{pr}\) is obtained by wga1 and not by wga2, that has a higher weight \(w_{pr}\). however, the difference in \(f_{pr}\) for the best solutions obtained by wga1 and wga2 is small. the number of replacements, i.e., codons where the association to the amino acid changed, are high for the genetic codes obtained by wga. figure 3 shows the best solutions obtained by wga1, wga2, and wga3. when the number of replacements is high, the robustness is also high. this occurs because most of the changes in the codons will result in the codification of the same amino acid, i.e., the new amino acid. unlike the weighted approach, nsga-ii allows to obtain a list of expanded genetic codes. the evaluation of the subset of solutions obtained by applying the dominance criterion to the union of the first front obtained in different runs of nsga-ii is shown in fig. 4. the best solutions obtained by the weighted approaches are also presented in the figure. one can observe that the best solutions obtained by the weighted approaches are in the pareto front obtained by nsga-ii. table 4 shows the evaluation of the best solutions for each objective obtained by nsga-ii. it is interesting to observe that, for the values of the properties of the new hypothetical amino acid, the best solutions for \(f_{pr}\) and \(f_{mv}\) are the same. in additional experiments, not shown here, with other values for the properties of the hypothetical amino acid, this does not necessarily happen. table 3. evaluation of the best solutions obtained by the weighted approach. the best results for each algorithm are in bold.full size table table 4. evaluation of the best solutions obtained by nsga-ii regarding each objective. the mean for all solutions of the pareto front are also presented.full size table one can observe that, while the genetic code with best \(f_f\) obtained by nsga-ii replaces only one codon, 21 replacements are generated by wga3. codes that results in many replacements, specially in codons that are frequently used, incur in higher economic cost and can lead to unwanted biological effects. it is important to observe that less replacements could be obtained by setting \(w_f\) to values much higher than the values of the other two weights in wga. however, this would result in manually testing many different settings for the weights. the pareto approach is interesting because it is not necessary to define weights or priority for each objective. besides, it allows to obtain a list of genetic codes, that could be offered to the specialist for a particular selection, given a real-world application. fig. 3. expanded genetic codes for the best solutions respectively obtained by wga1 (top), wga2 (middle), and wga3 (bottom). full size image fig. 4. pareto front for nsga-ii and best solutions for wga1, wga2, and wga3. full size image fig. 5. three expanded genetic codes obtained by nsga-ii: solution with best \(f_{f}\) (top), solution with best \(f_{pr}\) for 2 replacements (middle), and solution with best \(f_{pr}\) for 3 replacements (bottom). full size image this advantage of nsga-ii is illustrated in fig. 5, that shows the expanded genetic codes for three solutions of the pareto front: solution with best \(f_{f}\), solution with best \(f_{pr}\) for 2 replacements, and solution with best \(f_{pr}\) for 3 replacements. the two last solutions are those of the pareto front with best robustness for polar requirement among those with two and three replacements. in this way, the specialist can define that she/he wants a list of solutions with best robustness for a given amino acid property and with a given number of replacements. 4 conclusions we propose the use of multi-objective gas for the optimization of expanded genetic codes. three objectives are considered: robustness regarding polar requirement, robustness regarding molecular volume, and frequency of use of replaced codons. two approaches were investigated: weighted (wga) and pareto (nsga-ii). experiments with a hypothetical amino acid indicated that wga found codes with many replacements of codons. nsga-ii allowed to obtain codes with only one replacement, while the best codes for wga resulted respectively in 21 replacements. replacing many codons is not interesting in many aspects. both approached obtained robust codes. another advantage of the pareto approach is that a list of genetic codes is offered to the specialist, that can select a genetic code according to the characteristics of an application. it is important to highlight that this is a theoretical work, without taking into account restrictions that may occur from technological and biological points of view. in practice, more information about the biological application is necessary to choose an expanded genetic code. however, the work shows a computational approach for optimizing expanded genetic codes that can be useful, when used in conjunction to other strategies, for helping specialists. a possible future work is to investigate the introduction of new amino acids through the creation of synthetic nucleotides [15]. in this case, the standard genetic code is not modified; it is only expanded to accommodate the new codons related to the new synthetic nucleotides. for example, assuming that a synthetic nucleotide y is created, we would have the possibility of associating the new codons that have y, i.e., \(aay, acy,\ldots , aya, \ldots , ygg\), to the new amino acids. usually, not all new codons are associated with amino acids. in this case, optimization via gas shows a promising approach. another possible future work, from a technological point of view, is to use other algorithms in the calculation of the pareto set, such as spea-ii (strength pareto evolutionary algorithm 2). other optimization techniques could also be considered if the number of replacements is constrained. for example, if the maximum number of replacements is small, exhaustive search can be used to find the best codes. finally, the investigation of new objectives that may be interesting from an technological, experimental, and/or biological point of view could also be investigated. notes 1.the binary encoding of the chromosome is used here because we consider only one unnatural amino acid. if more than one new amino acid is considered, the integer encoding must be used, where integer \(i>0\) represents the i-th new amino acid. the only modification needed in this case is in the way the new chromosomes are generated and mutated. references anderson, j.c., et al.: an expanded genetic code with a functional quadruplet codon. proc. natl. acad. sci. 101(20), 7566–7571 (2004) article  google scholar  coello, c.a.c., lamont, g.b.: applications of multi-objective evolutionary algorithms, vol. 1. world scientific, london (2004) math  google scholar  cox, m.m., nelson, d.l.: lehninger principles of biochemistry, vol. 5. wh freeman, new york (2008) google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  google scholar  freeland, s.j., hurst, l.d.: the genetic code is one in a million. j. mol. evol. 47(3), 238–248 (1998) article  google scholar  freitas, a.a.: a critical review of multi-objective optimization in data mining: a position paper. acm sigkdd explor. newsl. 6(2), 77–86 (2004) article  google scholar  haig, d., hurst, l.d.: a quantitative measure of error minimization in the genetic code. j. mol. evol. 33(5), 412–417 (1991). https://doi.org/10.1007/bf02103132 article  google scholar  maloy, s.r., stewart, v.j., taylor, r.k., miller, s.i.: genetic analysis of pathogenic bacteria. trends microbiol. 4(12), 504 (1996) google scholar  oliveira, l.l., freitas, a.a., tinós, r.: multi-objective genetic algorithms in the study of the genetic code’s adaptability. inf. sci. 425, 48–61 (2018) article  mathscinet  google scholar  oliveira, l.l., oliveira, p.s.l., tinós, r.: a multiobjective approach to the genetic code adaptability problem. bmc bioinform. 16(1), 1–20 (2015) article  google scholar  rovner, a.j., et al.: recoded organisms engineered to depend on synthetic amino acids. nature 518(7537), 89–93 (2015) article  google scholar  santos, j., monteagudo, á.: simulated evolution applied to study the genetic code optimality using a model of codon reassignments. bmc bioinform. 12(1), 1–8 (2011) article  google scholar  silva, m.c., oliveira, l.l., tinós, r.: optimization of expanded genetic codes via genetic algorithms. in: anais do xv encontro nacional de inteligência artificial e computacional, pp. 473–484 (2018) google scholar  yockey, h.p.: information theory, evolution, and the origin of life. cambridge university press, cambridge (2005) google scholar  zhang, y., et al.: a semi-synthetic organism that stores and retrieves increased genetic information. nature 551(7682), 644–647 (2017) article  google scholar  download references author information authors and affiliations department of computing and mathematics, ffclrp, university of são paulo, ribeirão preto, brazil maísa de carvalho silva, paulo guilherme pinheiro pereira, lariza laura de oliveira & renato tinós authors maísa de carvalho silvaview author publications search author on:pubmed google scholar paulo guilherme pinheiro pereiraview author publications search author on:pubmed google scholar lariza laura de oliveiraview author publications search author on:pubmed google scholar renato tinósview author publications search author on:pubmed google scholar corresponding author correspondence to renato tinós . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de carvalho silva, m., pereira, p.g.p., de oliveira, l.l., tinós, r. (2023). multiobjective evolutionary algorithms applied to the optimization of expanded genetic codes. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_1 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords genetic algorithms multi-objective optimization expanded genetic codes publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature online selection of heuristic operators with deep q-network: a study on the hyflex framework | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper online selection of heuristic operators with deep q-network: a study on the hyflex framework conference paper first online: 28 november 2021 pp 280–294 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) online selection of heuristic operators with deep q-network: a study on the hyflex framework download book pdf download book epub augusto dantas10 & aurora pozo10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 828 accesses abstract general and adaptive strategies have been a highly pursued goal of the optimization community, due to the domain-dependent set of configurations (operators and parameters) that is usually required for achieving high quality solutions. this work investigates a deep q-network (dqn) selection strategy under an online selection hyper-heuristic algorithm and compares it with two state-of-the-art multi-armed bandit (mab) approaches. we conducted the experiments on all six problem domains from the hyflex framework. with our definition of state representation and reward scheme, the dqn was able to quickly identify the good and bad operators, which resulted on better performance than the mab strategies on the problem instances that a more exploitative behavior deemed advantageous. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others the impact of state representation on approximate q-learning for a selection hyper-heuristic chapter © 2022 stochastic online decisioning hyper-heuristic for high dimensional optimization article 15 december 2023 an automatic hyperparameter optimization method for an improved dqn chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence learning algorithms machine learning optimization origin selection artificial intelligence 1 introduction for many complex optimization problems, the use of heuristic approaches is often required to achieve feasible solutions in reasonable computational time [2]. one drawback of heuristics is that their performance heavily rely on the configuration setting, which must be tuned for the problem-domain at hand [2]. because of that, the optimization community has investigated several adaptive search methodologies [2], initially only for parameter tunning, but then it expanded for automatically controlling the heuristic operators to be used. these strategies are normally termed in the literature as hyper-heuristics (hh) [3], in which the algorithm explore the search space of low-level heuristics. it can also be found as adaptive operator selection (aos) [7], usually when the selection occurs at a certain step within a meta-heuristic. moreover, with the advance and success of machine learning (ml) techniques, there has been an increasing interest in using novel ml for guiding the optimization search in several ways, including the selection of heuristic operators [9]. due to the stochastic and iterative nature of optimization heuristics, reinforcement learning [17] techniques have been widely investigated for hh and aos applications. however, most of them are traditionally simple additive reinforcement strategies, such as probability matching (pm) and adaptive pursuit (ap) [7], that use the received feedback to update a probability vector. others are based on selection rules, that takes into account the feedback and the frequency of appliances in order to deal with the exploration versus exploitation dilemma (e.g., choice function and multi armed bandit based strategies [7]). although those approaches presented good overall results, they lack a state representation according to the traditional rl definition [17], in which an agent learns a policy (directly or not) by interacting with an environment based on the observed state and the received feedback (reward or penalty). this work investigates a selection hyper-heuristic that uses a deep q-network to choose the heuristics. the selection agent is updated while solving an instance using the q-learning algorithm [19] with an artificial neural network as function approximator [17]. in this way, we model the task of selecting the heuristics as a markov decision process (mdp) [14], which implicates that the state representation must contain enough information for the agent to take an action. using a mdp-based strategy for this selection task has been shown to be advantageous over stateless strategies [18]. in fact, there are a few works that have successfully applied q-learning for hh and aos. handoko et al. [8] defined a discrete state space that relates to fitness improvement and diversity level. then, the q-learning updates the state-action values which are used to select the crossover operator of an evolutionary algorithm applied on the quadratic assignment problem. the experimental results demonstrated that the approach is competitive with classical credit assignment mechanisms (ap, pm and mab), while being less sensitive to the number of operators. similarly, buzdalova et al. [4] applied q-learning to select crossover and mutation operators for the traveling salesman problem. the state is defined by a 2-tuple containing the current generation and the fitness improvement of the current best individual over the initial best individual, both values discretized into 4 intervals. their approach outperformed a random selection, indicating that the agent was able to learn a working policy while solving the instances. mosadegh et al. [11] proposed a simulated annealing (sa) based hh that uses q-learning to select the moving operators. each action consists on three operators, and the state is the number of times that the previous actions succeed (i.e., the operator generated an accepted solution under the sa conditions). the approach was significantly superior to other versions of sa and two software packages, with respect to both the quality of the solution and the computation time. one limitation of these works is the use of a discrete state space, which may limit the representation of the search stage [18]. however, when defining a continuous state representation, the classical q-learning becomes infeasible due to the high dimensional q-table. therefore, a function approximation model is necessary to estimate the state-action values [17]. the work from teng et al. [18] defined a continuous state space that includes landscape measures about the current population and some parent-oriented features. then, a self-organizing neural network is trained offline to select the crossover operator. the performance of this approach was competitive with other selection mechanisms (including a tabular q-learning) and even better on some instances, thus highlighting the advantages of using a continuous mdp-based selection strategy. in sharma et al. [15], the authors used a double deep q-network to select mutation operators of a differential evolution algorithm applied on several cec2005 benchmark functions. the target network, which is trained offline during the training phase, receives as input 99 continuous features, where 19 are related to the current population and the reaming characterize the performance of each operator so far. this approach outperformed other non-adaptive algorithms and was competitive with state-of-the-art adaptive approaches. in contrast with these works, we modeled a continuous state representation that consider only the past performance of the operators (within a certain memory), and the dqn is only trained online to learn a selection policy on each instance. we compare this approach with two state-of-the-art mab based selection rules, namely the dynamic mab (dmab) [5] and the fitness-rate-rank mab (frrmab) [10]. the mab problem can be seen as a special case of reinforcement learning with only a single state [17]. we have present a preliminary work in a workshop paper [6], in which we report the results on two problem domains (the vehicle routing and the traveling salesman problems). later, we found a flaw in our frrmab implementation. here, we expand the results to all six problems from the hyper-heuristics flexible framework (hyflex) [12] and with all implementations revised. moreover, we also present some analysis on the behavioral aspect of the selection mechanisms. the remainder of this paper is organized as follows: sect. 2 explains the concepts of hhs and describes the selection strategies that we compared: dmab, frrmab and dqn. the experimental setup and results are given in sect. 3. finally, we draw some conclusions and indicate future works in sect. 4. 2 selection hyper-heuristic according to burke et al. [3], hyper-heuristics can be divided in two groups: selection hh, where it selects from a set of predefined low-level heuristics (llh); generative hhs, where the algorithm uses parts of llhs to construct new ones. moreover, they can also be classified by its source of learning feedback: online, offline, and no-learning. this work is about an online learning selection hh. as a search methodology, selection hhs explore the search space of low-level heuristics (e.g., evolutionary operators) [3]. to avoid getting stuck into local optima solutions, good hhs must know which is the appropriate low-level heuristic to explore a different area of the search space at the time [3]. we used in this work a standard selection hyper-heuristic algorithm, as shown in algorithm 1. iteratively, it selects and applies a low-level heuristic on the current solution and computes the reward. then, the acceptance criteria decides if the new solution is accepted and, at last, the hh calls the update method of the corresponding selection model. the reward is defined as the fir value (eq. 2) and was kept the same for all selection strategies. since our goal is to investigate the learning ability of the selection strategies, the acceptance criteria accepts all solutions. in this way, the actions of the agent always reflects a change in the environment. the selection is made according to the employed selection strategy. in this work, we compared three strategies: dynamic mab, fitness-rate-rank mab, and deep q-network. 2.1 dynamic multi-armed bandit a mab framework is composed of n arms (e.g., operators) and a selection rule for selecting an arm at each step. the goal is to maximize the cumulative reward gathered over time [16]. among several algorithms to solve the mab, the upper confidence bound (ucb) [1] is one of the most known in the literature, as it provides asymptotic optimality guarantees. the ucb chooses the arm that maximizes the following rule $$\begin{aligned} p_{i,t} + c \sqrt{\frac{2log(\sum _{j = 1}^{n} n_{j,t})}{n_{i,t}}} \end{aligned}$$ (1) where \(n_{i,t}\) is the number of times the ith arm has been chosen, and \(p_{i,t}\) the average reward it has received up to time t. the scaling factor c gives a balance between selecting the best arm so far (\(p_{i,t}\), i.e., exploitation) and those that have not been selected for a while (second term in the eq. 1, i.e., exploration). however, the ucb algorithm was designed to work in static environments. this is not the case in the hyper-heuristic context, where the quality of the low-level heuristics can vary along the hh iterations [7]. hence, the dynamic mab, proposed by [5], incorporates the page-hikley (ph) statistical test to deal with this issue. this mechanism resembles a context-drifting detection, but is related to the performance of the operators throughout the execution of the algorithm. once a change in the reward distribution is detected, according to the ph test, the dmab resets the empirical value estimates and the confidence intervals (p and n in eq. 1, respectively) of the ucb [5]. 2.2 fitness-rate-rank multi-armed bandit the fitness-rate-rank mab [10] proposes the use of fitness improvement rate (fir) to measure the impact of the application of an operator i at time t, which is defined as $$\begin{aligned} f i r_{i, t}=\max \left( 0, \frac{p f_{i, t}-c f_{i, t}}{p f_{i, t}}\right) \end{aligned}$$ (2) where \(p f_{i, t}\) is the fitness value of the original solution, and \(c f_{i,t}\) is the fitness value of the offspring. moreover, the ffrmab uses a sliding window of size w to store the indexes of past operators, and their respective firs. this sliding window is organized as a first-in first-out (fifo) structure and reflects the state of the search process. then, the empirical reward \(reward_{i}\) is computed as the sum of all fir values for each operator i in the sliding window. in order to give an appropriate credit value for an operator, the frrmab ranks all the computed \(reward_{i}\) in descending order. then, it assigns a decay value to them based on their rank value \(rank_{i}\) and on a decaying factor \(d \in [0,1]\) $$\begin{aligned} decay_i = d^{rank_i} \times reward_i \end{aligned}$$ (3) the d factor controls the influence for the best operator (the smaller the value, the larger influence). finally, the fitness-rate-rank (frr) of an operator i is given by $$\begin{aligned} frr_{i,t} = \frac{decay_i}{\sum _{j = 1}^n decay_j} \end{aligned}$$ (4) these \(frr_{i,t}\) values are set as the value estimate \(p_{i,t}\) in the ucb eq. (1). also, the \(n_{i,t}\) values considers only the amount of time that the operator appears in the current sliding window. this differs from the traditional mab and other variants such as the dmab, where the value estimate \(p_{i,t}\) is computed as the average of all rewards received so far. 2.3 deep q-network the classic q-learning algorithm keeps a table that stores the q-values (the estimate value of performing an action at current state) of all state-action pairs [19]. this table is then updated accordingly to the feedback the agent receives upon interacting with the environment. however, in a continuous state space, keeping the q-table is not feasible due to the high dimensionality of the problem [17]. instead, we can use a function approximation model (called the q-model) that gives the estimate q-values. in dqn, the q-model is defined by an artificial neural network (ann), in which the inputs are the current observed state representation, and the output layer yields the predicted q-values for the current state-action pairs. for this task, we used the multilayer perceptron regressor from the scikit-learn library [13]. with these estimated q-values, the agent selects the next action (low-level heuristic) according to its exploration policy. we used the \(\epsilon \)-greedy policy, that selects a random action with probability \(\epsilon \), and selects the action with the highest q-value with probability \(1\epsilon \). thus, \(\epsilon \) is a parameter that controls the degree of exploration of the agent and is usually set to a small value [17]. after performing the action, receiving the reward and observing the next state, the q-model is updated by running one iteration of gradient descent on the artificial neural network, with the following target value $$\begin{aligned} \text{ target }=\text{ reward }+\gamma \max _{a^{\prime }} q\left( s^{\prime }, a^{\prime }\right) \end{aligned}$$ (5) where \(s^{\prime }\) is the next state after performing the action, and \(\max _{a^{\prime }} q\left( s^{\prime }, a^{\prime }\right) \) is the highest q-value of all possible actions from state \(s^{\prime }\). the discount factor \(\gamma \) ([0, 1]) controls the influence of the future estimate rewards. we defined the state representation as the normalized average rewards of each operator. for this, we used the same sliding window structure from frrmab. hence, if we have 10 available low-level heuristics, for example, the state is represented as a vector of 10 values ranging [0,1], where each value is the average of the past w rewards (window size) of an operator. the idea is to investigate if the past observed rewards can be representative enough to allow the dqn to learn a proper selection policy. 3 experiments we conducted the experiments on all 6 problems from the hyflex framework [12]: one dimensional bin packing (bp), flow shop (fs), personal scheduling (ps), boolean satisfiability (max-sat), traveling salesman problem (tsp), and vehicle routing problem (vrp). the hyflex provides 10 instances of each domain and 4 types of low-level heuristics: mutational, ruin-and-recreate, local search, and crossover. we included all operators but the crossover group into the selection pool. we refer to the documentation for more details [12]. we executed each selection strategy 31 times on every instance with different random seeds. we set the stopping criteria as 300 s of cpu running time on a intel(r) core(tm) i7-5930k cpu @ 3.50 ghz. the number of runs and stopping criteria were set following the cross-domain heuristic search challengefootnote 1 competition rules, for which the hyflex was originally developed. table 1 displays the parameters setting that we used throughout the experiments. table 1. parameters settingfull size table next we present the performance comparison of the selection strategies for each problem domain. for this, we compared the mean performance obtained by each approach on all instances using the friedman hypothesis test and a pairwise post-hoc test with the bergmann correction. the results of these tests are shown in the pipe graphs (e.g. fig. 1), where the approaches are displayed according to their rank (the smaller the better), and the connected bold lines indicate the approaches that are statistically equivalent (\(p < 0.05\)). additionally, we also compared them on each instance individually with the kruskal-wallis hypothesis test, followed by a pairwise dunn’s test, with the lowest ranked approach set as the control variable. the tables, such as table 2, report the average and standard deviation of the best solution found by each selection strategy in the 31 runs. bold values indicate that the corresponding approach achieved a better performance with statistical difference (\(p < 0.05\)), and gray background highlights all approaches that were statistically equivalent to the approach with the best rank on that instance. moreover, we also contrasted the approaches in terms of the selection behavior during the search. for this, we divided the search into 10 phases with equal number of iterations, and computed the average frequency that each operator was selected on that phase, resulting on the line graphs such as fig. 2. 3.1 bin packing on bin packing, both dqn and frrmab were statistically equivalent when considering all instances performance, as displayed in fig. 1 where the dqn ranked better. fig. 1. friedman ranking with post-hoc tests on bp instances full size image in fact, if we observe the individual performance shown in table 2, we can notice that their performance was dependent on the type of the instance, meaning that neither of them could generalize for all instances. figure 2 displays the average appliances of each operator for instance 0, that the dqn was worse than frrmab. we can observe that the dqn presents a exploitative behavior, giving high emphasis on the early prominent operators. the frrmab, on the other hand, was able to detect a change in operator performance. table 2. performance comparison on bin packingfull size table fig. 2. average selection of operators on bp instance 0 full size image fig. 3. average selection of operators on bp instance 7 full size image however, this exploitative behavior was advantageous on some instances, such as instance 7 as shown in fig. 3. the dqn detected at the initial phases the set of prominent operators, while the frrmab needed more time to do so. 3.2 flow shop the dqn presented very poor performance on flow shop, where the frrmab and dmab were both statistically better, as displayed in fig. 4. even when comparing the instances individually (table 3), the dqn was not competitive in any instance. fig. 4. friedman ranking with post-hoc tests on fs instances full size image table 3. performance comparison on flow shopfull size table figure 5 shows that, although the dqn kept selecting a few operators at around 10% of times, its exploitative behavior of giving high emphasis on the top two operators resulted on the poor performance on flow shop. therefore, the reward scheme and/or the state representation that we defined were not satisfiable for domains in which more exploration is required to attain high quality solutions. 3.3 personal scheduling the same thing happened in the ps domain, as shown in fig. 6 and table 4, where we can see that the mab strategies were statistically superior to the dqn. however, in this case we noticed that the dqn did present a more explorative behavior in comparison to the other domains, as displayed in fig. 7. but even so, while the mab strategies kept an operator being selected at maximum around 15% of the times, the dqn reached about 30% of preference for a single operator on some phases. fig. 5. average selection of operators on fs instance 0 full size image fig. 6. friedman ranking with post-hoc tests on ps instances full size image table 4. performance comparison on personal schedulingfull size table fig. 7. average selection of operators on ps instance 0 full size image 3.4 max-sat the max-sat is another domain in which giving high preference to few operators results in better solutions. figure 8 and table 5 shows that the dqn outperformed with a large marge the two mab approaches. fig. 8. friedman ranking with post-hoc tests on max-sat instances full size image table 5. performance comparison on max-satfull size table interestingly, on some instances such as the one displayed in fig. 9, the three approaches identified the same set of operators as the best ones, meaning that the simple fitness-based reward scheme was able to give the selection strategy useful information in order to select the proper operators at each time. but again, the dqn selects with higher frequency the better heuristics and discard the others very quickly. 3.5 traveling salesman problem the tsp was another problem domain that the dqn outperformed the dmab and frrmab, as shown in fig. 10. when observing the individual instance performance (table 6), the dqn was statistically better on 7 out of 10 instances and was at least equivalent on the others. figure 11 confirms that the same exploitative behavior, i.e., giving high preference for one to three heuristics, allows the dqn to attain high quality solution on some domains. fig. 9. average selection of operators on max-sat instance 0 full size image fig. 10. friedman ranking with post-hoc tests on tsp instances full size image table 6. performance comparison on tspfull size table fig. 11. average selection of operators on tsp instance 0 full size image 3.6 vehicle routing problem finally, on vrp we observed a similar pattern of the bin packing domain: both dqn and frrmab were statistically equivalent on general (fig. 12), but they outperformed one another on different type of instances (table 7). the frrmab was better on the solomon instances and the dqn outperformed the others on the homberger instances. fig. 12. friedman ranking with post-hoc tests on vrp instances full size image table 7. performance comparison on vrpfull size table fig. 13. average selection of operators on vrp instance 5 full size image figure 13 shows the frequencies of operator selection on one homberger instance. such as in other vrp instances, we observed a pattern in which the dqn gave really high emphasis (more than 80%) for a single operator, which is a local search heuristic. this happens because only the fitness improvement is rewarded and, although the q-learning update rule (5) already implicitly deals with delayed rewards, it was not enough to give some credit for the exploration heuristics. 4 conclusion this work investigated the use of a heuristic selection strategy using deep q-network. in comparison with the mab-based strategies, the dqn selects its actions based on a state representation, which can give more insights about the current search stage. we performed the experiments on six problem domains from the hyflex framework and compared our approach with dmab and frrmab strategies. we could observe that our approach can detect earlier the good and bad operators. however, it is slower to detect a change in performance of the heuristics. therefore, for the domains and instances that there exists dominant operators, it outperformed the other selection approaches. on the other hand, it performed worse than both mabs when a wider diversity of operators was necessary. hence, further studies should be pursued to improve the exploration of dqn, so it can become more adaptive to different domains. this can be done either by improving the state representation, so it can give different information to the agent (such as stagnation, concept drift, etc.), or by investigating different reward schemes, so it can also rewards operators that do not necessarily improve the fitness function. notes 1.http://www.asap.cs.nott.ac.uk/external/chesc2011/. references auer, p., cesa-bianchi, n., fischer, p.: finite-time analysis of the multiarmed bandit problem. mach. learn. 47(2), 235–256 (2002) article  google scholar  blum, c., puchinger, j., raidl, g.r., roli, a.: hybrid metaheuristics in combinatorial optimization: a survey. appl. soft comput. 11(6), 4135–4151 (2011) article  google scholar  burke, e.k., hyde, m., kendall, g., ochoa, g., özcan, e., woodward, j.r.: a classification of hyper-heuristic approaches, pp. 449–468. springer, us, boston, ma (2010). https://doi.org/10.1007/978-1-4419-1665-5_15 buzdalova, a., kononov, v., buzdalov, m.: selecting evolutionary operators using reinforcement learning: initial explorations. in: proceedings of the companion publication of the 2014 annual conference on genetic and evolutionary computation, pp. 1033–1036 (2014) google scholar  dacosta, l., fialho, a., schoenauer, m., sebag, m.: adaptive operator selection with dynamic multi-armed bandits. in: proceedings of the 10th annual conference on genetic and evolutionary computation, pp. 913–920. gecco ’08, association for computing machinery, new york, ny, usa (2008) google scholar  dantas, a., rego, a.f.d., pozo, a.: using deep q-network for selection hyper-heuristics. in: proceedings of the genetic and evolutionary computation conference companion, pp. 1488–1492. gecco ’21, association for computing machinery, new york, ny, usa (2021) google scholar  fialho, á.: adaptive operator selection for optimization. université paris sud paris xi (dec, theses (2010) google scholar  handoko, s.d., nguyen, d.t., yuan, z., lau, h.c.: reinforcement learning for adaptive operator selection in memetic search applied to quadratic assignment problem. in: proceedings of the companion publication of the 2014 annual conference on genetic and evolutionary computation, pp. 193–194. gecco comp ’14, association for computing machinery, new york, ny, usa (2014) google scholar  karimi-mamaghan, m., mohammadi, m., meyer, p., karimi-mamaghan, a.m., talbi, e.g.: machine learning at the service of meta-heuristics for solving combinatorial optimization problems: a state-of-the-art. european journal of operational research (2021) google scholar  li, k., fialho, á., kwong, s., zhang, q.: adaptive operator selection with bandits for a multiobjective evolutionary algorithm based on decomposition. ieee trans. evol. comput. 18(1), 114–130 (2014) article  google scholar  mosadegh, h., ghomi, s.f., süer, g.a.: stochastic mixed-model assembly line sequencing problem: mathematical modeling and q-learning based simulated annealing hyper-heuristics. eur. j. oper. res. 282(2), 530–544 (2020) article  mathscinet  google scholar  ochoa, g., et al.: hyflex: a benchmark framework for cross-domain heuristic search, vol. 7245, pp. 136–147 (2012) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  puterman, m.l.: chapter 8 markov decision processes. in: handbooks in operations research and management science, stochastic models, vol. 2, pp. 331–434. elsevier (1990) google scholar  sharma, m., komninos, a., lópez-ibáñez, m., kazakov, d.: deep reinforcement learning based parameter control in differential evolution. in: proceedings of the genetic and evolutionary computation conference, pp. 709–717 (2019) google scholar  soria-alcaraz, j.a., ochoa, g., sotelo-figeroa, m.a., burke, e.k.: a methodology for determining an effective subset of heuristics in selection hyper-heuristics. eur. j. oper. res. 260(3), 972–983 (2017) article  mathscinet  google scholar  sutton, r.s., barto, a.g.: reinforcement learning, second edition: an introduction. mit press (2018) google scholar  teng, t.-h., handoko, s.d., lau, h.c.: self-organizing neural network for adaptive operator selection in evolutionary search. in: festa, p., sellmann, m., vanschoren, j. (eds.) lion 2016. lncs, vol. 10079, pp. 187–202. springer, cham (2016). https://doi.org/10.1007/978-3-319-50349-3_13 watkins, c.j.c.h., dayan, p.: q-learning. mach. learn. 8(3), 279–292 (1992) math  google scholar  download references acknowledgements this work was financially supported by conselho nacional de desenvolvimento científico e tecnológico (cnpq) and by coordenação de aperfeiçoamento de pessoal de nível superior (capes). author information authors and affiliations department of computer science, federal university of paraná (ufpr), curitiba, usa augusto dantas & aurora pozo authors augusto dantasview author publications search author on:pubmed google scholar aurora pozoview author publications search author on:pubmed google scholar corresponding author correspondence to augusto dantas . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper dantas, a., pozo, a. (2021). online selection of heuristic operators with deep q-network: a study on the hyflex framework. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_19 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hyper-heuristic reinforcement learning combinatorial optimization publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature critical analysis of ai indicators in terms of weighting and aggregation approaches | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper critical analysis of ai indicators in terms of weighting and aggregation approaches conference paper first online: 12 october 2023 pp 385–399 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) critical analysis of ai indicators in terms of weighting and aggregation approaches download book pdf download book epub renata pelissari  orcid: orcid.org/0000-0002-1721-96029, betania campello  orcid: orcid.org/0000-0001-9609-872410, guilherme dean pelegrina  orcid: orcid.org/0000-0001-7301-616710, ricardo suyama  orcid: orcid.org/0000-0002-8398-526811 & … leonardo tomazeli duarte  orcid: orcid.org/0000-0003-0290-008010  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 606 accesses 3 citations abstract national artificial intelligence (ai) strategies have been implemented by several countries worldwide. these strategies aim to guide ai policy priorities and foster research, innovation, and development in ai. alongside the development of national ai strategies, ai indices have emerged as tools to compare countries’ ai development levels. this study focuses on a specific ai indicator, the global ai index (gaii) proposed by tortoise media, which ranks 62 countries based on their level of investment, innovation, and implementation of ai. the gaii computes a ranking of countries by aggregating sub-dimensions within these categories using a weighted sum approach, with subjective weight assignments. this paper critically analyzes the weighting and aggregation approaches used in the gaii, employing two techniques. firstly, the stochastic multicriteria acceptability analysis (smaa) is used to explore changes in rankings by varying the weights. secondly, a non-additive aggregation model known as the choquet integral is applied to consider potential interactions among dimensions. the findings indicate that the weights assigned to criteria strongly influence the final ranking. additionally, there are interactions between ai dimensions that should be taken into account to address unbalanced achievements across dimensions. this study contributes to the development of more robust and objective methodologies for comparing countries’ ai development levels. work supported by são paulo research foundation (fapesp) under the grants #2020/09838-0 (bi0s brazilian institute of data science), #2020/10572-5 and #2023/04159-6. l. t. duarte and r. suyama would like to thank the national council for scientific and technological development (cnpq, brazil) for the financial support. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ai’s impact on sustainability targets: a cross-country nca and fsqca study article 24 october 2024 beyond the algorithm: applying critical lenses to ai governance and societal change article 10 august 2025 general artificial intelligence in self-developing reflective-active environments chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence global south methods and theory social indicators survey methodology symbolic ai artificial intelligence 1 introduction launched in 2017, canada’s national artificial intelligence (ai) strategy was the first in the world with the aim of guiding ai policy priorities at a country level. this helped canada to guide ai policy definition and prioritize investments, to stimulate research, innovation and development of solutions in ai as well as its conscientious and ethical aspects [1]. finland developed its national ai strategy also in 2017, closely followed by japan, france, germany and the united kingdom in 2018. more than 30 other countries and regions have launched their national ai strategies as of 2021footnote 1, including brazil. accompanying the development of national ai strategies, ai indices started to be built in order to compare nations on their level of ai development. the global ai index proposed by tortoise media provides a ranking of 62 countries around the world in order to benchmark nations on their level of investment, innovation and implementation of ai [2]. the global ai vibrancy toolfootnote 2 proposed by the stanford institute for human-centered artificial intelligence (stanford hai) provides a weighted ranking of 29 countries with the aim of identifying the countries that are leading ai race in terms of two main dimensions: research and development and economy. each ai dimension used in the construction of an indicator is associated with a relative importance weight, intending to account for the fact that contributions of different dimensions to the level of ai development have varying degrees of impact levels. tortoise media adopted a weighting approach based on subjective assumptions, which may affect the composite scoring for each country and consequently their position in the ranking. on the other hand, stanford hai’s approach, worrying about the subjectivity of dimension weights and its impact on the final ranking, adopts a user preference-based weighting methodology and provides an interactive tool that allows the user to change the dimension weights, obtaining different rankings depending on the preference declared by the user. despite the differences in the aforementioned weighting approaches, those indices are similar in the sense that they are built based on a linear aggregation of multiple dimensions–although different dimensions are adopted in each of them. generally, when one assumes a linear aggregation, one does not consider interactions among criteria. indeed, when defining the set of criteria, non-redundancy is a desirable property [3]. however, one frequently observes correlated criteria in real applications and, therefore, adopting an approach that models interaction between criteria could be useful to avoid biased results [4]. as pointed out in the composite indicator literature [5, 6], indicators that take into account multiple dimensions should be aggregated and weighted accordingly, considering correlation and compensability issues among indicators and avoiding subjective weighting approaches. robustness analysis should be undertaken to assess the impact of assumptions and hypotheses set when building the composite indicator in terms of the choice of weights and the aggregation method (among others). as a consequence, it is likely that changes in these assumptions may be required, thus ultimately leading to different final decisions. therefore, motivated by the requirements that usually stands behind the development of robust indicators, this paper presents a critical analysis of ai indicators when comparing countries in terms of weighting and aggregation approaches. we attempt to answer the following 3 research questions: 1. do the criteria weights influence the resulting ai ranking of countries? 2. is the hypothesis of interactions between ai dimensions true? 3. does the use of a non-linear aggregator in order to consider the interaction between criteria influence the resulting ai ranking of countries? in the conducted analysis we consider the global ai index proposed by tortoise because it presents a ranking with a greater number of countries and considers a greater number of dimensions in the analysis, and also due to the ease of data acquisition. in order to answer those questions, we apply the mcda (multiple criteria decision aid) methods smaa and choquet integral. the first question is answered by conducting robustness analyses through the application of the stochastic multicriteria acceptability analysis (smaa) [7, 8]. smaa is based on an inverse weight space analysis in order to describe the criteria weights that make each country the most “preferred” one. therefore, it does not require weights to be pre-defined. smaa can also be used with different decision models besides the weighted sum. the result given by smaa–among other descriptive measures–is the probability of a country occupying each of the positions in the ranking. since smaa considers simultaneously the uncertainty in all parameters, it is particularly useful for robustness analysis. we apply smaa with the weighted sum varying the weights and conduct a comprehensive analysis of the ranking variation, comparing the results with the tortoise ranking. we consider two scenarios: weight information totally missing and weights following the preference order adopted in the tortoise index. in order to answer question 2, we show that the ai dimensions are statistically redundant. finally, to answer question 3, we evaluate the use of a non linear aggregator, called choquet integral [9], which takes into account interaction among criteria. we also compare the ranking obtained by means of the choquet integral with the tortoise ranking. this paper is organized as follows. in sect. 2, we introduce the multiple criteria decision aid problem. section 3 presents an overview of the methodology used in tortoise to derive the global ai index. in sect. 4, we provide the theoretical background on the smaa methodology and the choquet integral. section 5 presents the methodology adopted in this paper. results are presented and discussed in sect. 6. we conclude this paper in sect. 7. 2 multiple criteria decision aid multiple criteria decision aid (mcda) is an area of research concerned with mathematical and computational design tools that can be used either by an individual decision-maker (dm) or a group of dms, to evaluate a finite number of decision alternatives regarding a set of performance criteria, which are determined according to the decision context [10]. these approaches make it possible to reduce the subjectivity inherent in decision-making processes while considering the preferences of the dm(s). the dm is who has the power over the decision and is responsible for setting the model parameters. the set of alternatives \(\mathcal {a} = \{a_{1}, a_2, \ldots , a_m\}\) is a finite set of m elements, with \(m \ge 2\), being all of them considered possible solutions for the studied problem. decision criteria \(g = \{g_1, g_2, \ldots , g_n\}\) are qualitative or quantitative attributes used to evaluate the different alternatives. in mcda we assume that there are at least two criteria (\(n \ge 2\)). for each criterion \(g_j\) is given a relative importance \(w_j\) called criterion weight, \(j=1, \ldots , n\). we denote \(\textbf{w} = (w_1, w_2, \ldots , w_n)\) the criteria weight vector. each alternative \(a_i\) is evaluated according to each criterion \(g_j\), for \(i=1, \ldots , m\) and \(j=1, \ldots , n\), representing the performance of the alternative in relation to that criterion. \(g_j(a_i)\) denotes the performance of the alternative \(a_i\) in relation to the criterion \(g_j\), for \(i=1, \ldots , m\) and \(j=1, \ldots , n\). the different mcda methods aim to solve a decision-making problem with m alternatives \(\mathcal {a} = \{a_1, \ldots , a_m\}\), evaluated according to n criteria \(\{g_1, \ldots , g_n\}\). without loss of generality, it is assumed that all criteria must be maximized. thus, the decision problem is defined as (1) $$\begin{aligned} \max \{g_1(a), \ldots , g_n(a) / a \in \mathcal {a}\}. \end{aligned}$$ (1) a utility function \(u(a_i, \textbf{w})\) is then applied as the aggregation procedure in order to obtain a utility value–or score–that, in the context of the ranking problematic, is used to order the set of alternatives obtaining a final ranking. 3 tortoise’s approach to deriving the global ai index: an overview of the methodology the global ai index (gaii) proposed by tortoise media [2] aims at ranking 62 countries, represented by \(\left[ a_1, \ldots , a_{62} \right] \), based on their level of development in articial intelligence, which is measured by combining three categories of indicators: implementation:: this category evaluates how artificial intelligence is being implemented by businesses, governments, and communities. it comprises three dimensions: talent, infrastructure, and operating environment; innovation:: this category measures technological advancements and methodological breakthroughs that indicate a greater potential for artificial intelligence in the future. this pillar is divided into two dimensions: research and development; investment:: this category assesses the financial and procedural commitments made towards artificial intelligence, and is composed of two dimensions: commercial ventures and government strategy. scores for each of the seven dimensions–also called criteria–are obtained by aggregating several sub-criteria. these scores are displayed on the tortoise websitefootnote 3. then, a total score (ts) for each country is obtained through a weighted sum (ws), calculated as $$\begin{aligned} \text {ts}_i^{ws} = \frac{1}{\sum _{j=1}^{n} w_j}\sum _{j=1}^{n} w_j g_j(a_i), \forall i, \end{aligned}$$ (2) where \(w_j\) represents the weight assigned to criterion \(g_j\), and \(g_j(a_i)\) is the score of country \(a_i\) over criterion \(g_j\), for \(i = 1, \cdots , 62\) and \(j = 1, \cdots , 7\). the weights used by tortoise to obtain the total score are: talent (\(w_1 = 5\)), infrastructure (\(w_2 = 3\)), operating environment (\(w_3 = 2\)), research (\(w_4 = 5\)), development (\(w_5 = 3\)), government strategy (\(w_6 = 1\)), and commercial ventures (\(w_7 = 5\)). the scores of the top six countries according to the tortoise gaii are illustrated in table 1. table 1. scores for each ai dimension and the total score (ts) of the top six countries determined by tortoise gaii.full size table one may note in table 1 that the usa has emerged as the leader in the gaii, scoring the maximum possible points in four out of the seven criteria: talent, infrastructure, operating environment, and government strategy. as a result, the usa received the highest overall score among the 62 countries ranked by the gaii. this approach provides an understanding of a country’s ai capacity by considering multiple factors, and the weights assigned to each criterion reflect their relative importance in determining a country’s overall score. however, two main drawbacks related to tortoise gaii can be highlighted. firstly, the assignment of weights can be considered subjective, as noted by tortoise itself [2]. secondly, the adoption of the weighted sum as an aggregator is based on the unverified hypothesis of independence between the criteria. 4 preliminaries in this section, we present the building blocks of our proposal: the smaa algorithm and the choquet integral, including a description on the adjustment of the choquet integral parameters. 4.1 smaa smaa (stochastic multicriteria acceptability analysis) is a simulation-based method for discrete multicriteria decision makings problems where model parameters are uncertain, imprecise, or, specifically in the case of criteria weights, partially or totally missing [7, 11]. uncertain information is represented by probability distributions. throughout a monte-carlo simulation process, values for the uncertain variables are sampled from their distributions, and alternatives are evaluated by applying the decision model that can be a weighted average, for instance, or any other aggregation procedure. in a ranking problem, smaa determines all possible rankings for alternatives and quantifies the results in terms of probabilities. usually, the recommended solution is the ranking with the highest probability. different smaa variants have been proposed in the literature [11]. the basis for most of those variants is the smaa-2 version proposed in [8]. smaa-2 computes three descriptive measures. the rank acceptability index, denoted by \(b_{i}^s\), describes the probability of an alternative \(a_{i}\) being in s-th position of the rank. it ranges between 0 and 1, and the closer \(b_{i}^s\) is to 1, the greater the probability of \(a_{i}\) being in position s. central weight vector \(w_{i}^c\) describes decision-maker (dm) preferences supporting alternative \(a_{i}\) being ranked first. central weight vectors allow an inverse decision-making approach: dm can learn the weights that lead an alternative to rank first instead of previously defining them and building a solution for the problem. the confidence factor \(p^{c}_i\) is the probability of an alternative being the preferred one with the criteria weights expressed by its central weight vector. as probabilities for all possible solutions are provided by smaa, this is a methodology that describes how robust the model is subject to different uncertainties in the input data, being so a useful tool for robustness analysis. the rank acceptability index can support this analysis. for instance, alternatives with high acceptability for the best ranks are candidates for occupying the best places in the rank, while alternatives with large acceptability for the worst ranks should be avoided in the best positions even if they would have fairly high acceptability for the best ranks. if none of the alternatives receives high acceptability indices for the best ranks, it indicates a need to measure the criteria, preferences or both more accurately. 4.2 choquet integral the (discrete) choquet integral (ci) [9] is a non-linear aggregation function that takes into account interaction among criteria. it is defined as follows: $$\begin{aligned} ts_i^{ci} = \sum _{j=1}^{n}\left[ g_{(j)}(a_i) g_{(j-1)}(a_i)\right] \mu \left( \left\{ (j), \ldots , (n) \right\} \right) , \end{aligned}$$ (3) where \((1), \ldots , (n)\) indicates a permutation of the indices j such that \(0 = g_{(0)}(a_i) \le g_{(1)}(a_i) \le g_{(j)}(a_i) \le \ldots \le g_{(n)}(a_i)\) and \(\mu (\cdot )\) represents the set of parameters known as capacity coefficients. a capacity \(\mu :2^{n} \rightarrow \mathbb {r}_{+}\), where \(n=\left\{ 1, 2, \ldots , n\right\} \) is the set of criteria, is a set function that satisfies the following axioms: (a) normalization: \(\mu (\emptyset ) = 0\) and \(\mu (n) = 1\), (b) monotonicity: \(\forall a \subseteq b \subseteq n, \mu (s) \le \mu (t)\). an interesting aspect of the choquet integral is that the capacity coefficients are associated with the shapley values [13], i.e., a well-known solution concept from game theory. in multicriteria decision making, the shapley value of a criterion j, represented by \(\phi _j\), indicates its marginal contribution on the aggregation procedure. the linear relation between \(\mu \) and \(\phi _j\) is given as follows: $$\begin{aligned} \phi _{j} = \sum _{a \subseteq n\backslash \left\{ j\right\} } \frac{\left( n-\left| a\right| -1\right) !\left| a\right| !}{n!} \left[ \mu (a \cup \left\{ j\right\} ) \mu (a) \right] , \end{aligned}$$ (4) where \(\left| a \right| \) represents the cardinality of subset a. a property associated with the shapley values that are useful when learning the capacity coefficients and interpreting the obtained parameters is that \(\phi _j \ge 0\), \(\forall j=1\). therefore, the importance assigned to each criterion is at least zero and the higher the \(\phi _j\), the higher criterion j contributes to the aggregation. besides the marginal contributions, one may also interpret the interaction between criteria. in this case, the shapley interaction index between criteria \(j,j'\) is given by [14, 15] $$\begin{aligned} i_{j,j'} = \sum _{a \subseteq n\backslash \left\{ j,j'\right\} } \frac{\left( n-\left| a\right| -2\right) !\left| a\right| !}{\left( n-1\right) !} \left[ \mu (a \cup \left\{ j,j'\right\} ) \mu (a \cup \left\{ j\right\} ) \mu (a \cup \left\{ j'\right\} ) + \mu (a)\right] . \end{aligned}$$ (5) each \(i_{j,j'}\) can be interpreted as the interaction degree between criteria \(j,j'\). if \(i_{j,j'} < 0\), there is a negative interaction (or redundant effect) between criteria \(j,j'\). if \(i_{j,j'} > 0\), there is a positive interaction (or complementary effect) between criteria \(j,j'\). in the case where \(i_{j,j'} = 0\), there is no interaction criteria \(j,j'\), and they act independently. although there exists a generalization of shapley index to any coalition of criteria (see [16] for further details), for the scope of this paper, we will restrict this parameters to singletons and pairs of criteria. indeed, in this study, we consider a particular case of the choquet integral, which is based on the notion of 2-additive capacity [16]. a 2-additive capacity implies that only exists interactions between pairs of criteria. in other works, we consider that the interaction among 3 or more criteria must be zero. based on a 2-additive capacity, the choquet integral can be defined by means of the shapley values and shapley interaction indices as follows [16]: $$\begin{aligned} ts_i^{2adci} = & {} \sum _{i_{j,j'} > 0} \min \{g_j(a_i), g_{j'}(a_i)\}i_{j,j'} + \sum _{i_{j,j'} < 0} \max \{g_j(a_i), g_{j'}(a_i)\}|i_{j,j'}| \nonumber \\ {} + & {} \sum _{j=1}^{n} g_{j}(a_i) (\phi _{j} \frac{1}{2}\sum _{j'\ne j}|i_{j,j'}|). \end{aligned}$$ (6) an interesting aspect of eq. (6) in comparison with eq. (3) is that one reduces the number of parameters from \(2^n\) to \(n(n+1)/2\). surely, we lose flexibility to model all kinds of interaction among criteria. however, the 2-additive choquet integral offers a good trade-off between flexibility and model complexity [17, 18]. when one assumes a 2-additive capacity, one may redefine the axioms of a capacity in terms of the shapley values and interaction indices as $$\begin{aligned} \sum _{j=1}^{n} \phi _j = 1 \end{aligned}$$ (7) and $$\begin{aligned} \phi _{j} \frac{1}{2}\sum _{j' \ne j}|i_{j,j'}| \ge 0, \forall j \in n. \end{aligned}$$ (8) 4.3 an unsupervised approach to learn the choquet integral parameters once one adopts the choquet integral as the aggregation function to calculate the scores and rank the alternatives, one needs to define its parameters. this task in the choquet integral is quite complicated, as one has several parameters to be defined. however, one may adopt a strategy that can automatically adjust some parameters without defining them subjectively. inspired by [19], we consider in this paper a non-supervised approach to automatically adjust the shapley interaction indices. the goal is to define the interaction index \(i_{j,j'}\) as close as possible from the negative of a similarity measure \(\rho _{j,j'}\) between pairs of criteria, such as the correlation coefficient between them [20]. the idea behind this approach is to mitigate, for instance, biased results provided by correlated criteria. suppose that two criteria are positively correlated. if we do not take this data structure characteristic into account, when one aggregates the evaluations provided by these criteria, one may sum twice the same information. therefore, as the choquet integral can model interaction between criteria, one may define a negative interaction index (which models a redundant effect) to positively correlated criteria. this will reduce the impact of criteria correlations. the optimization problem used to automatically adjust the shapley interaction indices is given as follows: $$\begin{aligned} \begin{array}{ll} \displaystyle \min _{i_{j,j'}, \forall j,j' \in n} &{} \sum _{j,j'}\left( i_{j,j'} + \rho _{j,j'}\right) ^2 \\ \text {s.t.} &{} \phi _{j} \frac{1}{2} \sum _{j' \ne j} \pm i_{j,j'} \ge 0, \, \, \forall j \in n \\ &{} \sum _{j} \phi _{j} = 1 \end{array}, \end{aligned}$$ (9) where ± in the first constraint avoids the use of absolute values. note that, in this optimization problem, we do not find the shapley values. indeed, they should be (subjectively) predefined. moreover, as it is a quadratic problem, it can be easily tackled by most of the available solvers. 5 methodology in order to answer the 3 research questions set in this study, we conducted four analyses of the gaii proposed by tortoise [2] considering the same 62 countries as alternatives and the same seven ai dimensions as criteria: talent (\(g_1\)), infrastructure (\(g_2\)), operating (\(g_3\)), environment (\(g_4\)), research (\(g_5\)), development (\(g_6\)), commercial ventures (\(g_7\)) and government strategy (\(g_8\)). we take the scores displayed in tortoise’s websitefootnote 4 as criteria performance. in the first analysis, we apply the weighted sum of eq. (2) to obtain a total score for each country, as in gaii, assuming weight information is totally missing. throughout smaa application, weights are randomly generated and the rank acceptability index is given as result. the weighted sum is also applied with smaa in the second analysis, but weights follow an ordinal preference. we assume the same order of preference as the one adopted in the construction of the tortoise index [2], i.e., \(w_1\) (talent) = \(w_4\) (research) = \(w_7\) (commercial ventures) > \(w_2\) (infrastructure) = \(w_5\) (development) > \(w_3\) (operating environment) > \(w_6\) (government strategy). in this case, throughout smaa application, weights are randomly generated by respecting the constraint imposed by these preferences. a comprehensive analysis based on the rank acceptability index is then conducted, comparing these two results with the tortoise ranking. the third analysis consists in verify the redundancies in the dataset, measured by the correlation coefficient between pairs of criteria. in the fourth analysis, we evaluate the use of the 2-additive choquet integral to aggregate the criteria information. for this purpose, we apply the non-supervised approach, presented in sect. 4.3, to obtain the interaction indices \(i_{j,j1}\) that are as close as possible to the negative of the correlation coefficients. as the shapley values \(\phi _j\), \(j=1, \ldots , n\), we assume the same weights as in the tortoise analysis. however, we normalize them as follows: $$\begin{aligned} \phi _j = \frac{w_j}{\sum _{j=1}^n w_j}. \end{aligned}$$ (10) based on the obtained interaction indices, we apply the choquet integral expressed in eq. (6). the obtained scores are used to construct the new ranking. we then compare the ranking provided by the choquet integral with respect to the tortoise one. we verify if an approach that models criteria interactions may lead to a different ranking of countries. 6 results and discussion in this section, we present and discuss the results of the conducted analysis. 6.1 probabilistic country rankings: a weight sum method and smaa perspective the first analysis refers to the application of the weighted sum and smaa with randomly generated weights. table 2 shows the obtained rank acceptability indices, with the top six countries ranked according to tortoise gaii. we have highlighted the highest percentage in each column to indicate the country with the highest probability of being in that particular position. results indicate that the usa and china are the two most acceptable alternatives, having the highest acceptability for the first two ranks. in particular, the usa has an acceptability score of 93.94% for the first rank and 6% for the second rank, which adds up to almost 100%. as a result, the usa seems to be a robust choice for being in the first position, regardless of criteria weights. similarly, china appears as a robust choice for the second position, indicating that the first and second positions obtained in the tortoise ranking come across as being robust choices. from the third position onwards, the ranking acceptability indices indicate a ranking significantly different from the tortoise gaii. this is evident in the case of the uk, which holds the third position in tortoise gaii but has only a 6.59% probability of being ranked third when the weights are random generated, and a potential 46.19% chance of assuming the fourth position. moreover, no robust choice can be made for the third position as the highest acceptability index achieved was only 24.34% by israel. we have elaborated fig. 1, which presents the same information as table 2, but in the form of a heat map that shows all possible combinations of country and position, sorted according to the tortoise gaii. as before, it is evident that the usa and china have a high probability of ranking first and second, respectively. in the middle of the heat map, the probabilities for other countries to occupy a specific position are low, and towards the end positions, the probabilities increase, albeit not robustly. this first analysis shows that the interchanges in the criteria weights have a significant impact on the ranking. table 2. rank acceptability indices with randomly generated weights of the top six countries of the tortoise gaii.full size table fig. 1. rank acceptability indices (%) with randomly generated weights. full size image the second analysis to the application of weighted sum and smaa considering ordinal weights and no longer completely random weights. table 3 presents the rank acceptability indices of this analysis. the first remark of the results is that there is a 100% chance that usa and china be in first and second place, respectively. despite a high probability of the uk ranking third at 63.04%, there remains a significant chance of 31.49% that it could instead be positioned fifth. it is noteworthy that singapore ranks sixth in the tortoise index, despite having a high probability (76.49%) of ranking fourth. conversely, canada, which ranks fourth in the tortoise gaii, has only a 6.91% probability of being in that position. table 3. rank acceptability indices with ordinal weights of the top six countries of the tortoise gaii.full size table fig. 2. rank acceptability indices (%) with ordinal weights. full size image figure 2 illustrates the same values presented in table 3 in the form of a heat map, showing all possible combinations of countries and positions. as expected, maintaining the order of relevant weights results in a more robust solution. however, fig. 2 also reveals that this robust solution may differ from tortoise gaii at certain points. these results show that criteria weights do influence the obtained ai ranking of countries. therefore, it is essential to conduct sensitivity analysis to evaluate the impact of weight changes and avoid unfair treatment of countries by assuming subjective weights. for example, while uk was ranked third in tortoise gaii, when random weights were used (even with order preferences), it had a low probability of being ranked in that position. similarly, singapore ranked sixth in tortoise gaii but had a 76.49% probability of being ranked fourth when preference weight order was applied. 6.2 country rankings: a choquet perspective in this subsection, we present the results using the choquet integral. as first investigation, we calculated the pearson correlation coefficients between pairs of criteria. the obtained values are presented in fig. 3. there are clearly a lot of redundancies within this dataset. for instance, there are strong correlations between talent and environment (\(\rho = 0.8103\)), talent and commercial ventures (\(\rho = 0.7951\)), environment and research (\(\rho = 0.8459\)), environment and commercial ventures (\(\rho = 0.8474\)), and research and commercial ventures (\(\rho = 0.7759\)). fig. 3. correlations coefficients among ai dimensions. full size image aiming at mitigating the effect of positively correlated criteria in the aggregation procedure, we searched for interaction indices as close as possible from the negative of the correlation coefficients that can be applied with the choquet integral. as the shapley values we assumed the same weights as in the tortoise analysis but normalized by eq. (10). by solving the optimization problem (9), we achieved \(i_{1,2} = -0.006\), \(i_{1,4} = -0.153\), \(i_{1,5} = -0.060\), \(i_{1,7} = -0.198\), \(i_{2,3} = -0.145\), \(i_{2,4} = -0.037\), \(i_{2,6} = -0.062\), \(i_{3,6} = -0.022\), \(i_{4,5} = -0.100\), \(i_{4,7} = -0.128\), \(i_{5,7} = -0.091\) (the remaining interaction indices are practically zero). based on these parameters, the 2-additive choquet integral leads to the total scores presented in fig. 4 (for the top 6 countries according to tortoise gaii). table 4. scores for each ai dimension and the total score (ts) of the top six countries determined by the 2-additive choquet.full size table an interesting remark is that singapore achieved the third position in the ranking. note the redundant effect between infrastructure and operating modeled by the choquet integral (\(i_{2,3} = -0.145\)) in order to mitigate the positive correlation between them (\(\rho _{2,3} = 0.4130\)). this helps to explain how singapore achieved a better score in comparison with uk and could move from the 6th to the 3rd position (in comparison with tortoise gaii). indeed, by looking at eq. (6), the part associated with a negative interaction index takes the maximum between the scores. therefore, the bad performance of singapore in operating was overcome by the very good score on infrastructure (which is higher than the score of uk). 7 conclusion this paper presented a critical analysis of ai indicators for comparing countries. we apply the smaa methodology and the choquet integral to analyze the tortoise gaii in terms of criteria weights and aggregation procedure. the smaa analysis results in rank acceptability indices for all countries, which can be used for deriving robust conclusions. more specifically, the rank acceptability index allows quantifying the amount of instability in the results induced by uncertain criteria weights. by applying the choquet integral, we explored how the solutions change when a non-linear function is assumed. regarding the first hypothesis that the criteria weights influence the resulting ai ranking, we have observed that even when randomly varying the weights while adhering to the same ordinal preference as used in tortoise gaii, the positions with the highest probabilities do not always align with those presented in the tortoise ranking. furthermore, for certain ranking positions, it may not be even possible to find a robust choice. it can be concluded that the decision regarding weight determination will strongly influence the final ranking. the hypothesis of interaction between criteria was verified, indicating that an aggregation procedure than the weighted sum shall be applied in the construction of such indicators when considering the same ai dimensions as criteria. this was confirmed after comparing the ranking determined by the 2-additive choquet with the tortoise ranking, since compensations between criteria are made through interaction indices, resulting in changes in rankings. it is important to note that the rank acceptability index given by smaa is able to provide only a rough ranking of the alternatives because there is no objective way to combine acceptability indices for different ranks to reach a complete ranking [12]. this characteristic naturally opens up possibilities for future studies, such as the use of the smaa pairwise winning index to propose a new ai ranking of the countries. a wider outlook would also include the proposal of new measures in order to compare the probability matrix with the given ranking. as another future study, we are planning to apply the smaa methodology also with the choquet integral in order to conduct a robustness analysis with respect to model structure. notes 1.https://hai.stanford.edu/news/state-ai-10-charts. accessed date: 16 may 2023. 2.https://aiindex.stanford.edu/vibrancy/. accessed date: 16 may 2023. 3.https://www.tortoisemedia.com/intelligence/global-ai/. accessed date: 16 may 2023. 4.https://www.tortoisemedia.com/intelligence/global-ai/ accessed date: 16 may 2023. references cifar. pan-canadian ai strategy impact assessment report. canada’s ai ecosystem: government investment propels private sector growth. university of toronto (2020). https://cifar.ca/wp-content/uploads/2020/11/pan-canadian-ai-strategy-impact-assessment-report.pdf. accessed 2 may 2023 tortoise. the global ai index: methodology. tortoise media (2021). https://www.tortoisemedia.com/wp-content/uploads/sites/3/2021/12/global-ai-index-methodology-3.0-211201-v2.pdf. accessed 4 may 2023 keeney, r.l., raiffa, h.: decisions with multiple objectives. wiley, new york (1976) math  google scholar  pelegrina, g.d., duarte, l.t., grabisch, m., romano, j.m.t.: dealing with redundancies among criteria in multicriteria decision making through independent component analysis. comput. ind. eng. 169, 108171 (2022) google scholar  greco, s., ishizaka, a., tasiou, m., torrisi, g.: on the methodological framework of composite indices: a review of the issues of weighting, aggregation, and robustness. soc. indic. res. 141, 61–94 (2019) article  google scholar  nardo, m., saisana, m., saltelli, a., tarantola, s., hoffmann, a., giovannini, e.: handbook on constructing composite indicators: methodology and user guide. oecd, paris (france) (2008) google scholar  lahdelma, r., hokkanen, j., salminen, p.: smaa stochastic multiobjective acceptability analysis. eur. j. oper. res. 106, 137–143 (1998) article  google scholar  lahdelma, r., salminen, p. smaa-2: stochastic multicriteria acceptability analysis for group decision making. oper. res. 49 (2001) google scholar  choquet, g.: theory of capacities. annales de l’institut fourier 5, 131–295 (1954) article  mathscinet  math  google scholar  par jean-charles pomerol and sergio barba-romero: multicriterion decision in management: principles and practice. kluwer academic publishers, massachusetts (2000) math  google scholar  pelissari, r., oliveira, m.c., amor, s.b., kandakoglu, a., helleno, a.l.: smaa methods and their applications: a literature review and future research directions. ann. oper. res. 293, 433–493 (2020) article  mathscinet  google scholar  lahdelma, r., salminen, p.: smaa in robustness analysis. in: doumpos, m., zopounidis, c., grigoroudis, e. (eds.) robustness analysis in decision aiding, optimization, and analytics. isorms, vol. 241, pp. 1–20. springer, cham (2016). https://doi.org/10.1007/978-3-319-33121-8_1 chapter  google scholar  shapley, l.s.: a value for n-person games. annals of mathematics studies: vol. 28. contributions to the theory of games, vol. ii, pp. 307–317 (1953) google scholar  murofushi, t., soneda, s.: techniques for reading fuzzy measures (iii): interaction index. in: 9th fuzzy system symposium, sapporo, japan, pp. 693–696 (1993) google scholar  grabisch, m.: alternative representations of discrete fuzzy measures for decision making. int. j. uncertain. fuzziness knowl.-based syst. 5(5), 587–607 (1997) article  mathscinet  math  google scholar  grabisch, m.: \(k\)-order additive discrete fuzzy measures and their representation. fuzzy sets syst. 92, 167–189 (1997) article  mathscinet  math  google scholar  grabisch, m., prade, h., raufaste, e., terrier, p.: application of the choquet integral to subjective mental workload evaluation. ifac proc. volumes 39(4), 135–140 (2006) article  google scholar  mayag, b., grabisch, m., labreuche, c.: a characterization of the 2-additive choquet integral through cardinal information. fuzzy sets syst. 184(1), 84–105 (2011) article  mathscinet  math  google scholar  duarte, l.t.: a novel multicriteria decision aiding method based on unsupervised aggregation via the choquet integral. ieee trans. eng. manag. 65(2), 293–302 (2018) article  google scholar  papoulis, a., pillai, s.u.: probability, random variables and stochastic processes. 4th edn. mcgraw-hill, new york (2002) google scholar  download references author information authors and affiliations school of electrical engineering and computation at the university of campinas, campinas, brazil renata pelissari school of applied sciences at the university of campinas, limeira, brazil betania campello, guilherme dean pelegrina & leonardo tomazeli duarte center for engineering, modeling and applied social at federal university of abc, santo andré, brazil ricardo suyama authors renata pelissariview author publications search author on:pubmed google scholar betania campelloview author publications search author on:pubmed google scholar guilherme dean pelegrinaview author publications search author on:pubmed google scholar ricardo suyamaview author publications search author on:pubmed google scholar leonardo tomazeli duarteview author publications search author on:pubmed google scholar corresponding author correspondence to guilherme dean pelegrina . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pelissari, r., campello, b., pelegrina, g.d., suyama, r., duarte, l.t. (2023). critical analysis of ai indicators in terms of weighting and aggregation approaches. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_26 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords global ai index multicriteria decision analysis smaa publish with us policies and ethics profiles leonardo tomazeli duarte view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature gradient estimation in model-based reinforcement learning: a study on linear quadratic environments | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper gradient estimation in model-based reinforcement learning: a study on linear quadratic environments conference paper first online: 28 november 2021 pp 33–47 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) gradient estimation in model-based reinforcement learning: a study on linear quadratic environments download book pdf download book epub ângelo gregório lovatto  orcid: orcid.org/0000-0002-0586-541210, thiago pereira bueno  orcid: orcid.org/0000-0003-1999-969710 & leliane nunes de barros  orcid: orcid.org/0000-0002-9128-018810  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 772 accesses 1 citation abstract stochastic value gradient (svg) methods underlie many recent achievements of model-based reinforcement learning agents in continuous state-action spaces. despite their practical significance, many algorithm design choices still lack rigorous theoretical or empirical justification. in this work, we analyze one such design choice: the gradient estimator formula. we conduct our analysis on randomized linear quadratic gaussian environments, allowing us to empirically assess gradient estimation quality relative to the actual svg. our results justify a widely used gradient estimator by showing it induces a favorable bias-variance tradeoff, which could explain the lower sample complexity of recent svg methods. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others exploration versus exploitation in model-based reinforcement learning: an empirical study chapter © 2022 randomized smoothing variance reduction method for large-scale non-smooth convex optimization article 24 may 2021 superquantile-based learning: a direct approach using gradient-based optimization article 11 january 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. learning algorithms linear algebra signalling gradients statistical learning stochastic learning and adaptive control calculus of variations and optimization 1 introduction model-based reinforcement learning (mbrl) [16, 20] is a promising framework for developing intelligent systems for sequential decision-making from limited data. unlike in model-free reinforcement learning (rl) methods, mbrl agents use collected experiences to fit a predictive model of the environment. the agent can use the model to evaluate potential action sequences, saving costly trial-and-error experimentation in the real world, or to estimate quantities useful for improving its learned behavior. stochastic value gradient (svg) methods belong to the latter category, using the model to estimate the value gradient. while model-free methods use the score-function estimator of the value gradient, svg methods can leverage the model to produce gradients via the pathwise derivative estimator, usually found to be more stable in practice [23]. recently proposed rl agents using the svg approach have demonstrated its effectiveness in learning robotic locomotion from data with unprecedented sample-efficiency [1, 4, 8], i.e. with few collected experiences. although svg methods have been validated empirically, some of the algorithm design choices still lack rigorous theoretical or empirical justification. as recent work on model-free methods has shown, there can be a gap between the theoretical underpinnings of rl methods and their behavior in practice, often due to code-level optimizations [6, 11, 13]. moreover, it is common for theoretically promising mbrl algorithms to fail in practice [14], although negative results are not often publicized. in this work, we take a step towards a better understanding of the core tenets behind svg methods by analyzing an important algorithm design choice: the gradient estimator computation [17, 23]. we aim to identify the key practical differences between the theoretically sound, unbiased formulation from the deterministic policy gradient (dpg) framework [24] and an estimator more often used in svg methods [1, 4, 8] that deviates from traditional policy gradient theory. the former represents an approach that’s theoretically sound, but lacks empirical validation, while the latter generalizes the methods that deviate from the theory, but achieve state-of-the-art results in continuous control benchmarks. we evaluate gradient estimation quality and policy optimization performance on linear quadratic gaussian (lqg) regulator environments. the lqg framework is extensively studied in the optimal control literature and is a special class of rl with continuous actions (a.k.a. continuous control), where svg methods seem to excel. lqg has been proposed as a simple, yet nontrivial class of continuous control problems to help distinguish the various approaches to rl [21]. more importantly, lqg has a simple environment formulation, allowing us to compute the ground-truth policy performance and gradient, an ability not present in more complicated, nonlinear benchmarks for continuous control [28]. thus, we can perform a more rigorous empirical assessment of gradient estimation quality and policy optimization performance within the lqg framework. therefore, our main contribution in this paper is a careful empirical analysis, using the lqg framework, of the practical differences between gradient estimators for svg methods. the rest of this paper is organized as follows. section 2 outlines related work on svg methods and the gap between theory and practice of rl. section 3 introduces the reader to the minimal technical background on rl, lqg and svg, required to follow our analysis. section 4 describes the scope of the empirical analysis and the experimental setup developed accordingly. section 5 contains the main experiments, results, and our analysis thereof. finally, we summarize our observations in sect. 6 and point to possible future work to explore remaining gaps in our knowledge of the core tenets of svg methods. 2 related work model-based policy gradients have been recently used in a variety of rl algorithms for continuous control. the pilco algorithm is one of the first to leverage a learned model’s derivatives to compute the policy gradient with few samples, but its use of gaussian processes hinders scalability to larger problems [5]. the original svg paper introduced gradient estimation with stochastic neural network models using the reparameterization trick, an approach scalable to higher-dimensional problems [9]. dreamer and imagined value gradients explore svgs with latent-space models [2, 8]. model-augmented actor-critic (maac) and sac-svg extend the svg framework to that of maximum-entropy rl to incentivize exploration and stabilize optimization [1, 4]. however, the computation of the gradient is not studied in isolation or contrasted with other gradient estimation methods. our work analyses, in detail, the gradient computation used in recent svg-style algorithms that show a high level of performance with efficient use of data [1, 4, 8]. a few recent works have analyzed the gap between theory and practice in different areas of rl, highlighting our poor understanding of current methods. reliability and reproducibility concerns regarding modern rl methods have been raised by several works [3, 10, 12], indicating a disconnect between the theory motivating these algorithms and their behavior in practice. code-level optimizations have been found to contribute more to successful policy gradient methods than the choice of general training algorithm [6]. closest to our work is that of ilyas et al. [11], which shows that model-free policy gradient algorithms succeed in optimizing the policy despite having poor gradient estimation quality metrics in the relevant sample regime. our work is, to the best of our knowledge, the first to propose a fine-grained analysis of gradient estimation in svg methods using lqgs to provide solid references of their expected behavior. 3 background 3.1 a brief introduction to rl we consider the agent-environment interaction modeled as a continuous markov decision process (mdp) [25], defined as the tuple \(({\mathcal s}, {\mathcal a}, r^{}, p^{*}, {\rho }, h)\), with each component described in what follows. interaction with the environment occurs in a sequence of discrete timesteps , where \(h\in {\mathbb n}\) denotes the time horizon, after which an episode of interaction is over. at every timestep t, the agent observes the current state \({\mathbf {s}}_t\in {\mathcal s}\subseteq {\mathbb r}^{n}\) from the set of possible states of the environment. it must then select an action \({\mathbf {a}}_t\in {\mathcal a}\subseteq {\mathbb r}^{d}\) from the set of possible actions to be executed. the environment then transitions to the next state by sampling from the transition probability kernel, , and emits a reward signal using its reward function, \(r_{t+1} = r^{}({\mathbf {s}}_t, {\mathbf {a}}_t)\). the initial state is sampled from the initial state distribution, \({\mathbf {s}}_0 \sim {\rho }(\cdot )\). a policy defines a mapping from environment states to actions, \({\mathbf {a}}_t = {\mu }({\mathbf {s}}_t)\). the objective of an rl agent is to find a policy that produces the highest cumulative reward, or return, from the initial state: . here, the expectation is implicitly w.r.t. the initial state distribution and the sequential application of . the key difference between optimal control and rl, both frameworks for optimal sequential decision making, is that in the former the agent has access to the full mdp, while in the latter the agent only knows the state and action space and has to learn its policy by trial-and-error in the environment. 3.2 linear quadratic gaussian regulator the lqg is a special class of continuous mdp in which the transition kernel is linear gaussian and the reward function is quadratic concave [27]: (1) (2) lqgs are often used as a discretization of continuous-time dynamics described as linear differential equations, such as those of physical systems. an important class of policies in this context is that of time-varying (a.k.a. nonstationary) linear policies: $$\begin{aligned} {\mu }_{{\theta }}({\mathbf {s}}_t) = \mathbf {k}_t{\mathbf {s}}_t + \mathbf {k}_t\,, \end{aligned}$$ (3) where \(\mathbf {k}_t \in {\mathbb r}^{d\times n}\) and \(\mathbf {k}_t \in {\mathbb r}^{d}\), a.k.a. the dynamic and static gains respectively in the optimal control literature. here, \({\theta }\) is the flattened parameter vector corresponding to the collection of function coefficients. given any linear policy \({\mu }_{{\theta }}\), its state-value function, the expected return from each state, can be computed recursively via the bellman equations [25]: (4) where \(v^{{\mu }_{{\theta }}}({\mathbf {s}}_{h}) = 0\) and the expectation is w.r.t. . since the policy is linear, rewards are quadratic, and the transitions, gaussian, the expectations in eq. (4) can be computed analytically by iteratively solving eq. (4) from timestep \(h-1\) to 0 (a dynamic programming method). the solution \(v^{{\mu }_{{\theta }}}\) is itself quadratic and thus the policy return can be computed analytically: (5) moreover, lqgs can be solved for their optimal policy \({\mu }_{{\theta }}^\star \), which is time-varying linear, by modifying the dynamic programming method above to solve for the policy that maximizes the expected value from each timestep based on eq. (4) (also computable analytically). algorithm 1 shows the pseudocode of the procedure we use to derive the optimal policy, which can be slightly modified to return the value function for a given policy. lqg is thus one of the few classes of nontrivial continuous control problems which allows us to evaluate rl methods against the theoretical best solutions. 3.3 stochastic value gradient methods in the broader rl context, methods that learn parameterized policies, often called policy optimization methods, have gained traction in the recent decade. as function approximation research, specially on deep learning, has advanced, parameterized policies were able to unify perception (processing sensorial input from the environment) and decision-making (choosing actions to maximize return) tasks [15]. to improve such parameterized approximators from data, the workhorse behind many policy optimization methods is stochastic gradient descent (sgd) [22]. thus, it is imperative to estimate the gradient of the expected return w.r.t. policy parameters, a.k.a. the value gradient, from data (states, actions and rewards) collected via interaction with the environment. svg methods build gradient estimates by first using the available data to learn a model of the environment, i.e., a function approximator . a common approach to leveraging the learned model is as follows. first, the agent collects b states via interaction with the environment, potentially with an exploratory policy \(\beta \) (we use \({\mathbf {s}}_t \sim d^{\beta }\) to denote sampling from its induced state distribution). then, it generates short model-based trajectories with the current policy \({\mu }_{{\theta }}\), branching off the states previous collected. finally, it computes the average model-based returns and forms an estimate of the value gradient using backpropagation [7]: (6) here, \(\hat{q}^{{\mu }_{{\theta }}}\) is an approximation (e.g., a learned neural network) of the policy’s action-value function . we refer to eq. (6) as the maac(k) estimator, as it uses k steps of simulated interaction and was featured prominently in the maac paper [4].footnote 1 we question, however, if eq. (6) actually provides good empirical estimates of the true value gradient. to elucidate this matter, we compare maac(k) to the value gradient estimator provided by the dpg theorem [24]: (7) besides the fact that eq. (7) requires us to use the on-policy distribution of states \(d^{{\mu }_{{\theta }}}\), more subtle differences with eq. (6) can be seen by expanding the definition of the action-value function to form a k-step version of eq. (7), which we call the dpg(k) estimator. note that dpg(0) is equivalent to eq. (7). figure 1 shows the stochastic computation graphs (scgs) of the maac(k) and dpg(k) estimators [23]. here, borderless nodes denote input variables; circles denote stochastic nodes, distributed conditionally on their parents (if any); and squares denote deterministic nodes, which are functions of their parents. because of the term in eq. (7), we’re not allowed to compute the gradients of future actions w.r.t. policy parameters in dpg(k), hence why only the first action has a link with \({\theta }\). on the other hand, maac(k) backpropagates the gradients of the rewards and value-function through all intermediate actions. our work aims at identifying the practical implications of these differences and perhaps help explain why maac(k) has been used in svg methods and not dpg(k). fig. 1. stochastic computation graphs for policy gradients. (left) model-free dpg. (right) model-based dpg: the dashed edges represent the k-step deterministic dependencies of the policy parameters in maac(k) for \(k=3\); dpg(k) ignores these dependencies when backpropagating the action gradients. full size image 4 methodology we now turn to our research goals and the methods employed to perform the proposed empirical analysis. 4.1 scope of evaluation in this work, we propose a fine-grained analysis of the properties of dpg(k) and maac(k) in practice. we simplify our evaluation by using on-policy versions of the gradient estimators, i.e., by substituting \(d^{{\mu }_{{\theta }}}\) for \(d^{\beta }\) where present. we also opted for using perfect models of the environment dynamics and rewards, instead of learning them from data, to focus on the differences between gradient estimators. thus, we approximate the expectations in eqs. (6) and (7) via monte carlo sampling, using the actual transition kernel \(p^{*}\) and reward function \(r^{}\), to generate (virtual) transitions and compute the bootstrapped returns. one can view this setting as the best possible case in an svg algorithm: when the model-learning subroutine has perfectly approximated the true mdp, allowing us to focus on the gradient estimation analysis. we also compute the true action-value function, required for the k-step returns in eqs. (6) and (7), recursively via dynamic programming (analogously to eq. (4)). computing the ground-truth action-value function allows us to further isolate any observed differences between the estimators as a consequence of their properties alone. 4.2 randomized environments and policies to test gradient estimation across a wide variety of scenarios, we define how to sample lqg instances, \({\mathcal m}= ({\mathcal s}, {\mathcal a}, r^{}, p^{*}, {\rho })\), to run our experiments on. the main configurations for our procedure are: state dimension (\(n\)), action dimension (\(d\)) and time horizon (\(h\)). from these parameters we define the state space \({\mathcal s}={\mathbb r}^n\), action space \({\mathcal a}={\mathbb r}^d\) and timesteps . the transition dynamics are stationary, sharing \(\mathbf {f}, \mathbf {f}, \mathbf {\sigma }\) across all timesteps. the coefficients \(\mathbf {f_s}\) and \(\mathbf {f_a}\) are initialized so that the system may be unstable, i.e., with some eigenvalues of \(\mathbf {f_s}\) having magnitude greater or equal to 1, but always controllable, meaning there is a dynamic gain \(\mathbf {k}\) such that the eigenvalues of \((\mathbf {f_s}+ \mathbf {f_a}\mathbf {k})\) have magnitude less than 1. this ensures we are able to emulate real-world scenarios where uncontrolled state variables and costs may diverge to infinity, while ensuring there exists a policy which can stabilize the system [21]. finally, we fix the transition bias to \(\mathbf {f}= {\mathbf {0}}\) and the gaussian covariance to the identity matrix, \(\mathbf {\sigma }= {i}\). the initial state distribution is always initialized as a standard gaussian distribution: . as for the reward parameters, we initialize both \({{\mathbf {c}}_{{\mathbf {s}}{\mathbf {s}}}}\) and \({{\mathbf {c}}_{{\mathbf {a}}{\mathbf {a}}}}\) (see eq. (2)) as random symmetric positive definite matrices, sampled via the scikit-​learn library for machine learning in python [19].footnote 2 since we consider the problem of estimating value gradients for linear policies in lqgs, we also define a procedure to generate randomized policies. we start by initializing all dynamic gains \(\mathbf {k}_t = \mathbf {k}\) so that \(\mathbf {k}\) stabilizes the system. this is done by first sampling target eigenvalues uniformly in the interval (0, 1) and then using the scipy library to compute \(\mathbf {k}\) that places the eigenvalues of \((\mathbf {f_s}+ \mathbf {f_a}\mathbf {k})\) in the desired targets [26, 29].footnote 3 this process ensures the resulting policy is safe to collect data in the environment without having state variables and costs diverge to infinity. the generating procedure also serves to mimic practical situations where engineers devise a policy which can keep a system stable, but is not able to optimize running costs, which is where rl can serve to fine-tune it. finally, we initialize all static gains as \(\mathbf {k}_t = {\mathbf {0}}\). 5 empirical analysis we analyze the behavior of each estimator on two main settings: (i) gradient estimation for fixed policies and (ii) impact of gradient quality on policy optimization. 5.1 gradient estimation for fixed policies following previous work on model-free policy gradients [11], we evaluate the quality of the gradient estimates, for a given policy, produced by each estimator using two metrics: (i) the average cosine similarity with the true policy gradient and (ii) the average pairwise cosine similarity. the first metric is a measure of gradient accuracy and we denote it as such in the following plots. for a given minibatch size b and step size k, we compute 10 estimates of the gradient, each using b initial states sampled on-policy (\({\mathbf {s}}_t \sim d^{{\mu }_{{\theta }}}\)) and k-step model-based rollouts from each state. then, we compute the accuracy as the average cosine similarity of each of the 10 estimates with the true policy gradient, obtained as follows. we first compute the true expected return of a policy \({\mu }_{{\theta }}\) via dynamic programming, following eqs. (4) and (5). our implementation in pytorch [18] then allows us to use automatic differentiation to compute the gradient of the expected return w.r.t. policy parameters. the second metric is a measure of gradient precision and we denote it as such in the following plots. again, we compute 10 estimates of the gradient in the same manner used in computing the accuracy. then, we compute the precision as the average pairwise cosine similarity of the 10 estimates (the higher this quantity, the lower the variance). fig. 2. gradient accuracy for each estimator near convergence for different virtual rollout lengths (k). we used 50000 states sampled from the policy to approximate the expected value. full size image we first analyze the accuracy of each estimator when given enough states from the policy’s distribution \(d^{{\mu }_{{\theta }}}\) to approximate their true expected values. figure 2 shows the accuracy obtained by dpg and maac for different values of k using 50000 states from the policy’s distribution. the lqgs considered have state and action spaces of dimension 2 and horizon of length 20. for each value of k, we initialize 10 different environment-policy pairs and compute the accuracy for each, denoted as different markers in each vertical line.footnote 4 note how all but one of the instances using dpg(k) converged to the true value gradient, indicating that it is indeed an unbiased estimator. on the other hand, maac(k) incurs a larger bias with increasing values of k, indicating that the added action action-gradient terms (see fig. 1) influence the final direction. although the results above indicate that maac(k) is biased at convergence, most svg algorithms operate on a much smaller sample regime. figure 3 shows the accuracy across 10 different environment-policy pairs; this time, however, using smaller sample sizes from the policy distribution. lines denote the average results and shaded areas, the 95% confidence interval.footnote 5 for \(k=0\), the estimators are equivalent, which is verified in practice. in this more practical sample regime, we see that maac(k) produces more accurate results, specially for larger values of k. similar to fig. 3, fig. 4 shows the gradient precision in the same setting. we see that the variance of maac(k) is lower than that of dpg(k) across all tested values of \(k>0\). overall, figs. 2, 3 and 4 illustrate a classic instance of the bias-variance tradeoff in machine learning: maac(k) introduces bias, although a small one, in return for a much more stable (less variable) estimate of the gradient, whereas the unbiased dpg(k) demands much more samples to justify its use. fig. 3. gradient accuracy for each estimator for different minibatch sizes (\(b =\) #samples) and virtual rollout lengths (k). full size image note that the accuracy and precision metrics only account for differences in gradient direction and orientation. the magnitude may also be important, as it influences the learning rate when used to update policy parameters. figure 5 shows that maac(k) produces gradients with higher norms compared to dpg(k). one should keep this in mind when choosing the learning rate for sgd, as the following experiments show that the gradient norm have a significant impact on policy optimization. 5.2 impact of gradient quality on policy optimization previous work on model-free policy gradients has shown that policy optimization algorithms can improve a policy despite using poor gradient estimation [11]. it is not clear, however, if better policy gradient estimation translates to more stability or faster convergence in svg algorithms. we therefore conduct our next experiments comparing the maac and dpg estimators for policy optimization. figures 6 and 7 show learning curves as total cost (negative return) against the number of sgd iterations across several instances of lqgs (\(\text {dim}({\mathcal s}) = \text {dim}({\mathcal a}) = 2\) and \(h= 20\)). we use the same hyperparameters for both estimators.footnote 6 the results in fig. 6 suggest that the better quality metrics observed for maac(k) in figs. 3 and 4 do translate to faster and more stable policy optimization. however, if we normalize the gradient estimates before passing them to sgd, as in fig. 7, we see that both estimators are evenly matched. these results suggest that the main advantage of maac(k) over dpg(k) is in its stronger gradient norm (see fig. 5), which has been alluded to in previous work as a “strong learning signal” [4], inducing a faster learning rate. table 1. median suboptimality gap, the percentage difference in expected return against the optimal policy, across 10 seeds. lqg dimension refers to the dimension of state and action spaces. we use \(k = 8\) and \(b=20\) for both estimators.full size table fig. 4. gradient precision for each estimator for different minibatch sizes (\(b =\) #samples) and virtual rollout lengths (k). full size image fig. 5. gradient norm for each estimator for different minibatch sizes (\(b =\) #samples) and virtual rollout lengths (k). full size image fig. 6. policy optimization with unnormalized svg estimation. each panel corresponds to a different lqg instance (generated via different random seeds). lines denote the average results and shaded regions, one standard deviation, across 10 runs of the algorithm, each with a different random initial policy. results obtained with the 8-step versions of each estimator. full size image fig. 7. policy optimization with normalized svg estimation. each panel corresponds to a different lqg instance (generated via different random seeds). lines denote the average results and shaded regions, one standard deviation, across 10 runs of the algorithm, each with a different random initial policy. results obtained with the 8-step versions of each estimator. gradients were normalized before being passed to sgd. full size image we also evaluate if our previous findings generalize to higher state-action space dimensions, where sample-based estimation gets progressively harder. our performance metric is the suboptimality gap, i.e., the percentage difference in expected return between the current policy and the optimal one: \(100 \times (j({\mu }_{{\theta }}^\star ) j({\mu }_{{\theta }}))/j({\mu }_{{\theta }}^\star )\).footnote 7 table 1 summarizes our results with policy optimization with varying lqg sizes and time budgets.footnote 8 we don’t normalize gradients in this case, as that is not a common practice in svg algorithms.footnote 9 our findings show that the performance gap between dpg(k) and maac(k) tends to widen with higher dimensionalities, with policies trained via the latter outperforming those using the former. these results further emphasize the practicality of maac(k) over dpg(k), justifying the former’s use in recent svg methods [1, 4, 8]. 6 conclusions and future work in this work, we take an important step towards a better understanding of current svg methods. using the lqg framework, we show that the gradient estimation used by maac and similar methods induces a slight bias compared to the true value gradient. on the other hand, using a corresponding unbiased estimator such as the k-step dpg one increases sample-complexity due to high variance. moreover, the maac gradient estimates have higher magnitudes, which could help explain the fast learning performance of current methods. indeed, we found that policies trained with maac converge faster to the optimal policies than those using the k-step dpg across several lqg instances. future work may further leverage the lqg framework to perform fine-grained analyses of other important components of svg algorithms. for example, little is known about the interplay between model, value function, and policy learning from data in practice. a study on model and value function optimization metrics and their relation to the gradient estimation accuracy and precision can help in the design of stable and efficient svg algorithms in the future. another direction for investigation is analyzing the impact of off-policy data collection for model training. since models have limited representation capacity, learning the mdp dynamics from the distribution of another policy may not translate to good gradient estimation of the target policy. notes 1.our formula differs slightly from the original in that it considers a deterministic policy instead of a stochastic one. 2.we use the make_spd_matrix function. 3.we use the scipy.signal.place_poles function. 4.we use the same 10 random seeds for experiments across values of k. 5.we use seaborn.lineplot to produce the aggregated curves. 6.learning rate of \(10^{-2}\), \(b=200\), and \(k=8\). 7.recall from sect. 3 that lqg allows us to compute the optimal policy analytically. 8.we found that the computation times for both estimators were equivalent. 9.we only clip the gradient norm at a maximum of 100 to avoid numerical errors. references amos, b., stanton, s., yarats, d., wilson, a.g.: on the model-based stochastic value gradient for continuous reinforcement learning. corr arxiv:2008.1 (2020) byravan, a., et al.: imagined value gradients: model-based policy optimization with transferable latent dynamics models. in: corl. proceedings of machine learning research, vol. 100, pp. 566–589. pmlr (2019) google scholar  chan, s.c.y., fishman, s., korattikara, a., canny, j., guadarrama, s.: measuring the reliability of reinforcement learning algorithms. in: iclr. openreview.net (2020) google scholar  clavera, i., fu, y., abbeel, p.: model-augmented actor-critic: backpropagating through paths. in: iclr. openreview.net (2020). https://openreview.net/forum?id=skln2a4ydb deisenroth, m.p., rasmussen, c.e.: pilco: a model-based and data-efficient approach to policy search. in: getoor, l., scheffer, t. (eds.) proceedings of the 28th international conference on machine learning, icml 2011, bellevue, washington, usa, 28 june–2 july 2011, pp. 465–472. omnipress (2011). https://icml.cc/2011/papers/323_icmlpaper.pdf engstrom, l., et al.: implementation matters in deep rl: a case study on ppo and trpo. in: iclr. openreview.net (2020). https://github.com/implementation-matters/code-for-paper goodfellow, i.j., bengio, y., courville, a.c.: deep learning. adaptive computation and machine learning. mit press, cambridge (2016) math  google scholar  hafner, d., lillicrap, t.p., ba, j., norouzi, m.: dream to control: learning behaviors by latent imagination. in: iclr. openreview.net (2020) google scholar  heess, n., wayne, g., silver, d., lillicrap, t.p., erez, t., tassa, y.: learning continuous control policies by stochastic value gradients. in: nips, pp. 2944–2952 (2015). http://papers.nips.cc/paper/5796-learning-continuous-control-policies-by-stochastic-value-gradients henderson, p., islam, r., bachman, p., pineau, j., precup, d., meger, d.: deep reinforcement learning that matters. in: aaai, pp. 3207–3214. aaai press (2018) google scholar  ilyas, a., et al.: a closer look at deep policy gradients. in: iclr. openreview.net (2020) google scholar  islam, r., henderson, p., gomrokchi, m., precup, d.: reproducibility of benchmarked deep reinforcement learning tasks for continuous control. corr arxiv:1708.04133 (2017) liu, z., li, x., kang, b., darrell, t.: regularization matters for policy optimization an empirical study on continuous control. in: international conference on learning representations (2021). https://github.com/xuanlinli17/iclr2021_rlreg lovatto, a.g., bueno, t.p., mauá, d.d., de barros, l.n.: decision-aware model learning for actor-critic methods: when theory does not meet practice. in: proceedings on “i can’t believe it’s not better!” at neurips workshops. proceedings of machine learning research, vol. 137, pp. 76–86. pmlr, december 2020. http://proceedings.mlr.press/v137/lovatto20a.html mnih, v., et al.: human-level control through deep reinforcement learning. nature 518(7540), 529–533 (2015) article  google scholar  moerland, t.m., broekens, j., jonker, c.m.: model-based reinforcement learning: a survey. in: proceedings of the international conference on electronic business (iceb) 2018-december, pp. 421–429 (2020). http://arxiv.org/abs/2006.16712 mohamed, s., rosca, m., figurnov, m., mnih, a.: monte carlo gradient estimation in machine learning. j. mach. learn. res. 21, 132:1–132:62 (2020) google scholar  paszke, a., et al.: pytorch: an imperative style, high-performance deep learning library. in: advances in neural information processing systems, vol. 32, pp. 8024–8035. curran associates, inc. (2019). http://papers.nips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  polydoros, a.s., nalpantidis, l.: survey of model-based reinforcement learning: applications on robotics. j. intell. robot. syst. 86(2), 153–173 (2017). https://doi.org/10.1007/s10846-017-0468-y article  google scholar  recht, b.: a tour of reinforcement learning: the view from continuous control. ann. rev. control robot. auton. syst. 2(1), 253–279 (2019). https://doi.org/10.1146/annurev-control-053018-023825, http://arxiv.org/abs/1806.09460 ruder, s.: an overview of gradient descent optimization algorithms. corr arxiv:1609.04747 (2016) schulman, j., heess, n., weber, t., abbeel, p.: gradient estimation using stochastic computation graphs. in: nips, pp. 3528–3536 (2015) google scholar  silver, d., lever, g., technologies, d., lever, g.u.y., ac, u.c.l.: deterministic policy gradient (dpg). in: proceedings of the 31st international conference on machine learning, vol. 32, no. 1, pp. 387–395 (2014). http://proceedings.mlr.press/v32/silver14.html szepesvári, c.: algorithms for reinforcement learning. synthesis lectures on artificial intelligence and machine learning. morgan & claypool publishers (2010). https://doi.org/10.2200/s00268ed1v01y201005aim009 tits, a.l., yang, y.: globally convergent algorithms for robust pole assignment by state feedback. ieee trans. autom. control 41(10), 1432–1452 (1996). https://doi.org/10.1109/9.539425 article  mathscinet  math  google scholar  todorov, e.: optimal control theory. bayesian brain: probabilistic approaches to neural coding, pp. 269–298 (2006) google scholar  todorov, e., erez, t., tassa, y.: mujoco: a physics engine for model-based control. in: ieee international conference on intelligent robots and systems, pp. 5026–5033. ieee (2012). https://doi.org/10.1109/iros.2012.6386109, http://ieeexplore.ieee.org/document/6386109/ virtanen, p., et al.: scipy 1.0: fundamental algorithms for scientific computing in python. nat. meth. 17, 261–272 (2020). https://doi.org/10.1038/s41592-019-0686-2 article  google scholar  download references acknowledgments this work was partly supported by the capes grant 88887.339578/2019-00 (first author), fapesp grant 2016/22900-1 (second author), and cnpq scholarship 307979/2018-0 (third author). author information authors and affiliations instituto de matemática e estatística, universidade de são paulo, são paulo, brazil ângelo gregório lovatto, thiago pereira bueno & leliane nunes de barros authors ângelo gregório lovattoview author publications search author on:pubmed google scholar thiago pereira buenoview author publications search author on:pubmed google scholar leliane nunes de barrosview author publications search author on:pubmed google scholar corresponding author correspondence to ângelo gregório lovatto . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper lovatto, â.g., bueno, t.p., de barros, l.n. (2021). gradient estimation in model-based reinforcement learning: a study on linear quadratic environments. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_3 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords reinforcement learning model-based machine learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature $$ aspic ^?$$ and the postulates of non-interference and crash-resistance | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper \( aspic ^?\) and the postulates of non-interference and crash-resistance conference paper first online: 28 november 2021 pp 328–343 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) \( aspic ^?\) and the postulates of non-interference and crash-resistance download book pdf download book epub rafael silva  orcid: orcid.org/0000-0002-2211-729910 & joão alcântara  orcid: orcid.org/0000-0002-4297-297010  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 732 accesses abstract we introduce an interrogation mark ? in \( aspic ^+\) languages as a plausibility operator to enhance any defeasible conclusion does not have the same status as an irrefutable one. the resulting framework, dubbed \( aspic ^?\), is tailored to make a distinction between strong inconsistencies and weak inconsistencies. the aim is to avoid the former and to tolerate the latter. this means the extensions obtained from the \( aspic ^?\) framework are free of strong conflicts, but tolerant to weak conflicts. then, in the current study, we show \( aspic ^?\) satisfy reasonable properties. in particular, we focus on the property that a conflict between two arguments should not interfere with the acceptability of other unrelated arguments. with this purpose in mind, we prove under which conditions the important principles of non-interference and crash-resistance hold in \( aspic ^?\). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a paraconsistent approach to deal with epistemic inconsistencies in argumentation chapter © 2021 dealing with inconsistencies in  $${aspic}^+$$ chapter © 2022 limits and difficulties in the design of under-approximation abstract domains chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. formal languages and automata theory formal logic general logic lisp logic syntax 1 introduction as noticed in [1], contradictions can be considered under the mantle of many points of views: as a consequence of the only correct description of a contradictory world, as a temporary state of our knowledge, as the outcome of a particular language which we have chosen to describe the world, as the result of conflicting observational criteria, as the superposition of world-views, or as the result from the best theories available at a given moment. indeed, in [2], it is argued that inconsistency is a natural companion to defeasible methods of reasoning and that paraconsistency (the property of a logic admitting non-trivial inconsistent theories) should play a role in the formalisation of these methods. in fact, they introduced an interrogation mark ? as a plausibility operator to enhance any defeasible conclusion do not have the same status as an irrefutable one, obtained from deduction. inspired by these ideas, we will present the \( aspic ^?\) framework by extending the \( aspic ^+\) framework [3], one of the most important formalisms to represent and reason with structured argumentation. in \( aspic ^?\) (as well as in \( aspic ^+\)), we identify two types of rules: strict (irrefutable) and defeasable. unlike \( aspic ^+\), the distinguishing characteristic of \( aspic ^?\) is the conclusion of any defeasible rule will be a plausible (?-suffixed) formula \(\phi ?\). the intended meaning is the conclusion of \(\phi ?\) will not necessarily prevent the conclusion of \(\lnot \phi ?\); it is required an argument with conclusion \(\lnot \phi \), which can only be obtained from a strict rule, to attack the conclusion \(\lnot \phi ?\) of an argument. thus, to produce a strong conflict between the conclusions of the arguments, at least in one of them, the conclusion should be a ?-free formula obtained via a strict rule. in \( aspic ^?\), strong inconsistencies as in \(\left\{ \phi , \lnot \phi \right\} \) (or \(\left\{ \phi ?, \lnot \phi \right\} \)) are distinguished from those weak inconsistencies as in \(\left\{ \phi ?, \lnot \phi ?\right\} \): the first should be avoided; the second can be tolerated. this means the (weak) conflict between \(\phi ?\) and \(\lnot \phi ?\) can be accommodated in the same extension. hence, the extensions in \( aspic ^?\) will be free of strong conflicts, but tolerant to weak conflicts. given that much current work on structured argumentation [3,4,5] combines strict and defeasible inference rules, unexpected results can arise when two arguments based on defeasible rules have contradictory conclusions. this is particularly critical (see [6]) if the strict inference rules include the ex falso principle (that an inconsistent set implies anything), because for any formula \(\phi \), an argument concluding \(\lnot \phi \) can be constructed from these two arguments. as consequence, any other argument is potentially under threat! in order to solve this problem for \( aspic ^+\), wu [7, 8] requires that in each argument, the set of conclusions of all its sub-arguments are classically consistent. another approach was taken in [6], in which they replace classical logic as the source for strict rules by the (weaker) paraconsistent logic presented in [9] to invalidate the ex falso principle as a valid strict inference rule. here we will also exploit how to avoid the application of the ex falso principle in \( aspic ^?\) by combining these two solutions: 1) as in [6], we resort to paraconsistent reasoning to tolerate conflicts; our differential is we tolerate only weak conflicts. 2) as in [7, 8], we require for each argument, the set of conclusions of all its sub-arguments are consistent; our differential is that we eliminate only those arguments whose sets of conclusions lead to a strong conflict. then, we show \( aspic ^?\) satisfies reasonable properties. in particular, we focus on the property that a conflict between two arguments should not interfere with the acceptability of other unrelated arguments. with this purpose in mind, we prove under which conditions the important principles of non-interference and crash-resistance [10] hold in \( aspic ^?\). the rest of the paper is organised as follows: in sect. 2, \( aspic ^?\) framework is presented. then, we introduce the corresponding argumentation framework with two kinds of defeats (strong and weak) and its semantics. section 3 is focused on proving the satisfaction of the principles of non-interference and crash-resistance. finally, we summarise our contributions and future developments. 2 the \( aspic ^?\) framework an abstract argumentation framework \( af \) [11] is a pair \((\mathcal a, \mathcal d)\) in which \(\mathcal a\) is a set of arguments and \(\mathcal d \subseteq \mathcal a \times \mathcal a\) is a relation of defeat. an argument a defeats \(\mathcal b\) if \((a,b) \in \mathcal d\). the \( aspic ^+\) framework [3, 12] gives structure to the arguments and defeat relation in an \( af \). in this section, we introduce in \( aspic ^+\) languages an interrogation mark ? as a plausibility operator to enhance defeasible conclusions do not have the same status as those irrefutable. the resulting framework, \( aspic ^?\), is tailored to distinguish strong inconsistencies from weak inconsistencies. the aim is to avoid the former and to tolerate the latter. we start by defining the argumentation systems specified by \( aspic ^?\): definition 1 (argumentation system). an argumentation system is a tuple \( as = (\mathcal l,^-,\mathcal r,n)\), in which \(\mathcal l= \mathcal l^* \cup \mathcal l^?\) is a logical language with a unary negation symbol \(\lnot \) and a unary plausibility symbol ? such that \(\mathcal l^*\) is a ?-free logical language with a unary negation symbol. \(\mathcal l^? = \left\{ \phi ? \mid \phi \in \mathcal l^*\right\} \). \(^-\) is a function from \(\mathcal l\) to \(2^{\mathcal l}\), such that \(\varphi \) is a contrary of \(\psi \) if \(\varphi \in \overline{\psi }\), \(\psi \not \in \overline{\varphi }\); \(\varphi \) is a contradictory of \(\psi \) (denoted by \(\varphi = -\psi \)), if \(\varphi \in \overline{\psi }\), \(\psi \in \overline{\varphi }\); \(\mathcal r= \mathcal r_s\cup \mathcal r_d\) is a set of strict (\(\mathcal r_s\)) and defeasible (\(\mathcal r_d\)) inference rules of the form \(\phi _1, \ldots , \phi _n \rightarrow \phi \) and \(\phi _1, \ldots , \phi _n \rightarrow \psi ?\) respectively (in which \(\phi _1, \ldots , \phi _n, \phi \) are meta-variables ranging over wff in \(\mathcal l\) and \(\psi \) is a meta-variable ranging over wff in \(\mathcal l^*\)), and \(\mathcal r_s\cap \mathcal r_d= \emptyset \). n is a partial function such that \(n : \mathcal r_d\longrightarrow \mathcal l\). for any formula \(\phi \in \mathcal l^*\), we say \(\psi \in \phi \) if \(\psi = \lnot \phi \) or \(\psi = \lnot \phi ?\) or \(\phi = \lnot \psi \) or (\(\phi = \lnot \gamma \) and \(\psi = \gamma ?\)); we say \(\psi \in \phi ?\) if \(\psi = \lnot \phi \) or \(\phi = \lnot \psi \). intuitively, contraries can be used to model well-known constructs like negation as failure in logic programming. note for any \(\phi \in \mathcal l^*\), \(\phi \) and \(\phi ?\) are contradictories of \(\lnot \phi \); whilst, only \(\phi \) is a contradictory of \(\lnot \phi ?\). this means \(\phi ?\) is not a contradictory of \(\lnot \phi ?\). a set as \(\left\{ \phi , \phi \right\} \) (or \(\left\{ \phi ?, \phi ?\right\} \)) is intended to represent a strong inconsistency, and \(\left\{ \phi ?, \lnot \phi ?\right\} \) is intended to represent a weak inconsistency. we will refer to these two kinds of inconsistencies (strong and weak) as epistemic inconsistencies or simply inconsistencies. it is also required a knowledge base to provide premises for the arguments. definition 2 (knowledge base). a knowledge base in an argumentation system \( as = (\mathcal l,^-,\mathcal r, n)\) is a set \(\mathcal k\subseteq \mathcal l\) consisting of two disjoint subsets \(\mathcal k_n\) (the axioms) and \(\mathcal k_p\) (the ordinary premises). axioms are certain knowledge and cannot be attacked, whilst, ordinary premises are uncertain and can be attacked. now we can define an argumentation theory: definition 3 an argumentation theory \(( as , \mathcal k)\) is a pair in which \( as \) is an argumentation system and \(\mathcal k\) is a knowledge base in \( as \). in \( aspic ^?\), arguments are constructed recursively from an argumentation theory by the successive application of construction rules: definition 4 (argument). an argument a on the basis of an argumentation theory \(( as , \mathcal k)\) and an argumentation system \((\mathcal l,^-, \mathcal r, n)\) is 1. \(\phi \) if \(\phi \in \mathcal k\) with \(\mathtt {prem}(a) = \left\{ \phi \right\} \), \(\mathtt {conc}(a) = \phi \), \(\mathtt {sub}(a) = \left\{ \phi \right\} \), \(\mathtt {defr}(a) = \emptyset \), \(\mathtt {rules}(a) = \emptyset \), \(\mathtt {toprule}(a) = \textit{undefined}\). 2. \(a_1, \ldots , a_n \rightarrow \psi \) if \(a_1, \ldots , a_n\) are arguments s.t. there is a strict rule \(\mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi \in \mathcal r_s\); \(\mathtt {prem}(a) = \mathtt {prem}(a_1) \cup \cdots \cup \mathtt {prem}(a_n)\); \(\mathtt {conc}(a) = \psi \); \(\mathtt {sub}(a) = \mathtt {sub}(a_1) \cup \cdots \cup \mathtt {sub}(a_n) \cup \left\{ a\right\} \); \(\mathtt {rules}(a) = \mathtt {rules}(a_1) \cup \cdots \cup \mathtt {rules}(a_n) \cup \left\{ \mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi \right\} \); \(\mathtt {toprule}(a) = \mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi \). 3. \(a_1, \ldots , a_n \rightarrow \psi ?\) if \(a_1, \ldots , a_n\) are arguments such that there exists a defeasible rule \(\mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi ? \in \mathcal r_d\); \(\mathtt {prem}(a) = \mathtt {prem}(a_1) \cup \cdots \cup \mathtt {prem}(a_n)\); \(\mathtt {conc}(a) = \psi ?\); \(\mathtt {sub}(a) = \mathtt {sub}(a_1) \cup \cdots \cup \mathtt {sub}(a_n) \cup \left\{ a\right\} \); \(\mathtt {rules}(a) = \mathtt {rules}(a_1) \cup \cdots \cup \mathtt {rules}(a_n) \cup \left\{ \mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi ?\right\} \); \(\mathtt {toprule}(a) = \mathtt {conc}(a_1), \ldots , \mathtt {conc}(a_n) \rightarrow \psi ?\). for any argument a we define \(\mathtt {prem}_n(a) = \mathtt {prem}(a) \cap \mathcal k_n\); \(\mathtt {prem}_p(a) = \mathtt {prem}(a) \cap \mathcal k_p\); \(\mathtt {defr}(a) = \left\{ r \in \mathcal r_d\mid r \in \mathtt {rules}(a)\right\} \) and \(\mathtt {str}(a) = \left\{ r \in \mathcal r_s\mid r \in \mathtt {rules}(a)\right\} \). example 1 consider the argumentation system \( as = (\mathcal l,^-,\mathcal r,n)\), in which \(\mathcal l= \mathcal l^* \cup \mathcal l^?\) with \(\mathcal l^* = \{a,b,f,w, \lnot a, \lnot b, \lnot f, \lnot w, {\sim a}, {\sim b}, {\sim f}, {\sim w}, {\sim \lnot a}, {\sim \lnot b}, {\sim \lnot f}, {\sim \lnot w}\}\). the symbols \(\lnot \) and \(\sim \) respectively denote strong and weak negation. for any \(\phi \in \mathcal l^*\) and any \(\psi \in \mathcal l\), (1) \(\phi \in \overline{\psi }\) iff (a) \(\psi = \lnot \phi \) or \(\psi = \lnot \phi ?\) or \(\phi = \lnot \psi \) or (\(\phi = \lnot \gamma \) and \(\psi = \gamma ?\)); or (b) \(\psi = {\sim \phi }\) or (\(\psi = {\sim \phi ?}\). (2) \(\phi ? \in \overline{\psi }\) iff (a) \(\psi = \lnot \phi \) or \(\phi = \lnot \psi \); or (b) \(\psi = {\sim \phi }\). \(\mathcal r_s= \left\{ \lnot f \rightarrow \lnot w; b \rightarrow a\right\} \) and \(\mathcal r_d= \left\{ a \rightarrow \lnot f?; b, {\sim \lnot w} \rightarrow w?; \lnot f? \rightarrow \lnot w?\right\} \). let \(\mathcal k\) be the knowledge base such that \(\mathcal k_n= \emptyset \) and \(\mathcal k_p= \left\{ b, {\sim \lnot w}\right\} \). the arguments defined on the basis of \(\mathcal k\) and \( as \) are \(a_1 = [b]\), \(a_2 = [{\sim \lnot w}]\), \(a_3 = [a_1 \rightarrow a]\), \(a_4 = [a_3 \rightarrow \lnot f?]\), \(a_5 = [a_1, a_2 \rightarrow w?]\) and \(a_6 = [a_4 \rightarrow \lnot w?]\). an argument a is for \(\phi \) if \(\mathtt {conc}(a) = \phi \); it is strict if \(\mathtt {defr}(a) = \emptyset \); defeasible if \(\mathtt {defr}(a) \ne \emptyset \); firm if \(\mathtt {prem}(a) \subseteq \mathcal k_n\); plausible if \(\mathtt {prem}(a) \cap \mathcal k_p \ne \emptyset \). an argument is fallible if it is defeasible or plausible and infallible otherwise. we write \(s \vdash \phi \) if there is a strict argument for \(\phi \) with all premises taken from s, and if there is a defeasible argument for \(\phi \) with all premises taken from s. the next definition will be repeatedly employed in sect. 3: definition 5 let \( at = ( as , \mathcal k)\) be an argumentation theory with argumentation system \( as = (\mathcal l,^-, \mathcal r, n)\). for a formula \(\phi \in \mathcal l\), we define \(\texttt {atoms}(\phi ) = \left\{ a \mid a \textit{ is an atom occurring in } \phi \right\} \). for a set \(\mathcal f \subseteq \mathcal l\) of formulas in \(\mathcal l\), we define \(\texttt {atoms}(\mathcal f) = \bigcup _{\phi \in \mathcal f} \texttt {atoms}(\phi )\); furthermore, for a set of atoms \(\mathfrak a\), \(\mathcal f_{\mid \mathfrak a} = \left\{ \phi \in \mathcal f \mid \phi \textit{ contains only atoms in } \mathfrak a\right\} \). for a strict rule \(s = \phi _1, \ldots , \phi _n \rightarrow \varpsi \), we define \(\texttt {atoms}(s) = \texttt {atoms}(\left\{ \phi _1, \ldots , \phi _n, \psi \right\} )\). for a defeasible rule \(d = \phi _1, \ldots , \phi _n \rightarrow \varpsi \), we define \(\texttt {atoms}(d) = \texttt {atoms}(\left\{ \phi _1, \ldots , \phi _n, \psi \right\} )\). for a set \(\mathcal s = \left\{ s_1, \ldots , s_n\right\} \) of strict rules, we define \(\texttt {atoms}(\mathcal s) = \texttt {atoms}(s_1) \cup \cdots \cup \texttt {atoms}(s_n)\). for a set \(\mathcal d = \left\{ d_1, \ldots , d_n\right\} \) of defeasible rules, we define \(\texttt {atoms}(\mathcal d) = \texttt {atoms}(d_1) \cup \cdots \cup \texttt {atoms}(d_n)\). for an argumentation system \( as = (\mathcal l,^-, \mathcal r, n)\), we define \(\texttt {atoms}( as )= \texttt {atoms}(\mathcal r_d) \cup \texttt {atoms}(\left\{ n(r) \mid r \in \mathcal r_d\textit{ and } n(r) \textit{ is defined}\right\} )\), in which \(\mathcal r_d\subseteq \mathcal r\) is the set of defeasible rules in \(\mathcal r\). for an argumentation theory \( at = ( as , \mathcal k)\), we define \(\texttt {atoms}( at ) = \texttt {atoms}( as ) \cup \texttt {atoms}(\mathcal k)\). for an argument a, we define \(\texttt {atoms}(a) = \texttt {atoms}(\mathtt {str}(a)) \cup \texttt {atoms}(\mathtt {defr}(a))\). finally, for a set \(\mathcal a= \left\{ a_1, \ldots , a_n\right\} \) of arguments, we define \(\texttt {atoms}(\mathcal a) = \texttt {atoms}(a_1) \cup \cdots \cup \texttt {atoms}(a_n)\). let \( at = ( as , \mathcal k)\) be an argumentation theory with \( as = (\mathcal l, ^-, \mathcal r, n)\), and \(\mathcal a\) the set of all arguments constructed from \(\mathcal k\) in \( as \). assume \(\mathcal k= (\mathcal k_n, \mathcal k_p)\), such that \(\mathcal k_n= \left\{ a,b,c\right\} \) and \(\mathcal k_p= \emptyset \). consider \(\mathcal r= \left\{ a \rightarrow d, b \rightarrow e\right\} \). the resulting arguments are \(a = [a]\), \(b=[b]\), \(c=[c]\), \(d=[a \rightarrow d]\), and \(e=[b \rightarrow e]\). we have \(\texttt {atoms}(d) = \left\{ a, d\right\} \), \(\texttt {atoms}( at ) = \left\{ a, b, c, e\right\} \). note those atoms occurring only in the strict rules (as d) are not considered as atoms in \(\texttt {atoms}( at )\). 2.1 attacks and defeats in \( aspic ^?\) arguments are related to each other by attacks (as in \( aspic ^+\)) and by weak attacks: definition 6 (attacks). consider the arguments a and b. we say a attacks b iff a undercuts, undermines and rebuts b, in which a undercuts b (on \(b'\)) iff \(\mathtt {conc}(a) \in \overline{n(r)}\) for some \(b' \in \mathtt {sub}(b)\) such that \(b'\)’s top rule r is defeasible. a undermines b (on \(\phi \)) iff \(\mathtt {conc}(a) \in \overline{\phi }\) and \(\phi \in \mathtt {prem}_p(b)\). in such a case, a contrary-undermines b iff \(\mathtt {conc}(a)\) is a contrary of \(\phi \). a rebuts b (on \(b'\)) iff \(\mathtt {conc}(a) \in \overline{\phi ?}\) for some \(b' \in \mathtt {sub}(b)\) of the form \(b_1'',\ldots ,b_n'' \rightarrow \phi ?\). in such a case, a contrary-rebuts b iff \(\mathtt {conc}(a)\) is a contrary of \(\phi ?\). we say a weakly attacks b iff a weakly undermines or weakly rebuts b, in which a weakly undermines b (on \(\phi ?\) (resp. \(\lnot \phi ?\))) iff \(\mathtt {conc}(a) = \lnot \phi ?\) (resp. \(\mathtt {conc}(a) = \phi ?\)) for an ordinary premise \(\phi ?\) (resp. \(\lnot \phi ?\)) of b. a weakly rebuts b (on \(b'\)) iff \(\mathtt {conc}(a) = \lnot \phi ?\) (resp. \(\mathtt {conc}(a) = \phi ?\)) for some \(b' \in \mathtt {sub}(b)\) of the form \(b_1'',\ldots ,b_n'' \rightarrow \phi ?\) (resp. \(b_1'',\ldots ,b_n'' \rightarrow \lnot \phi ?\)). example 2 recalling example 1, we have \(a_5\) weakly rebuts \(a_6\) and \(a_6\) weakly rebuts \(a_5\). besides, \(a_6\) contrary-undermines \(a_2\) and \(a_5\) on \({\sim \lnot w}\). if in addition, one had the argument \(a_7 = [a_4 \rightarrow \lnot w?]\), then \(a_7\) (like \(a_6\)) would weakly rebut \(a_5\) on \(a_5\); however, \(a_7\) (unlike \(a_6\)) would not be weakly rebutted by \(a_5\). definition 7 (\( saf \)). a structured argumentation framework \( saf \) defined by an argumentation theory \( at = ( as , \mathcal k)\) is a tuple \(\langle \mathcal a, \mathcal c, \mathcal c', \preceq \rangle \), in which \(\mathcal a\) is the set of all arguments a constructed from \(\mathcal k\) in \( as \) such that it satisfies definition 4 and \(\texttt {atoms}(a) \subseteq \texttt {atoms}( at )\); \((x, y ) \in \mathcal c\) iff x attacks y and \((x, y ) \in \mathcal c'\) iff x weakly attacks y; \(\preceq \) is a preference ordering on \(\mathcal a\). the restriction \(\texttt {atoms}(a) \subseteq \texttt {atoms}( at )\) is to avoid including in the \( saf \) arguments built from strict rules without relation to \(\texttt {atoms}( at )\). next, we define the corresponding defeat relation: definition 8 (defeat). [3] let \(a,b \in \mathcal a\) and a attacks b. if a undercut, contrary-rebut or contrary-undermine attacks b on \(b'\) then a is said to preference-independent attack b on \(b'\); otherwise a is said to preference-dependent attack b on \(b'\). a defeats b iff for some \(b'\) either a preference-independent attacks b on \(b'\) or a preference-dependent attacks b on \(b'\) and \(a \not \prec b'\). as observed in the previous definition, a preference-dependent attack from one argument to another only succeeds (as a defeat) if the attacked argument is not stronger than the attacking argument. thus, if an argument a preference-dependent attacks b and b is preferred over a, then the attack of a to b does not succeed, and b is not defeated by a. 2.2 abstract argumentation frameworks with two kinds of defeats as \( saf \)s have two kinds of attacks, the associated abstract argumentation frameworks have to couple with two kinds of defeats: definition 9 (argumentation frameworks with two kinds of defeats). an abstract argumentation framework with two kinds of defeats (\( af _2\)) corresponding to a \( saf = \langle \mathcal a, \mathcal c, \mathcal c', \preceq \rangle \) is a tuple \((\mathcal a, \mathcal d, \mathcal d')\) such that \(\mathcal d = \{(x,y) \in \mathcal c \mid x \textit{ defeats } y \}\) and \(\mathcal d' = \left\{ (x,y) \in \mathcal c' \mid x \not \prec y\right\} \). for \(a \in \mathcal a\), we define $$ a^+ = \left\{ b \in \mathcal a\mid (a,b) \in \mathcal d \cup \mathcal d'\right\} \textit{ and } a^= \left\{ b \in \mathcal a\mid (b,a) \in \mathcal d \cup \mathcal d'\right\} . $$ given a \( saf sa \) defined by an argumentation theory \( at \) and an \( af _2\, af \) corresponding to \( sa \), we will refer to \( af \) as the resulting \( af _2\) from \( at \). example 3 (example 2 continued). let \(\preceq = \left\{ (a_6,a_2)\right\} \), i.e., \(a_6 \prec a_2\) be a preference ordering on \(\mathcal a = \left\{ a_1, a_2, a_3, a_4, a_5, a_6\right\} \). in the \( saf (\mathcal a, \mathcal c, \mathcal c', \preceq )\) defined by \( at \), we have \(\mathcal c = \left\{ (a_6,a_2)\right\} \) and \(\mathcal c' = \left\{ (a_5,a_6), (a_6,a_5)\right\} \). as \((a_6,a_2)\) is a preference independent attack, we obtain \(\mathcal d = \mathcal c\) and \(\mathcal d' = \mathcal c'\). traditional approaches to argumentation semantics ensure arguments attacking each other are not tolerated in the same set, which is said to be conflict free. in \( aspic ^?\), we distinguish a strong conflict from a weak conflict. the aim is to avoid strong conflicts, which are carried over by the defeat relation \(\mathcal d\), and to tolerate weak conflicts, which are carried over by the relation \(\mathcal d'\). we say the resulting set is compatible: definition 10 (compatible sets). let \( af = (\mathcal a, \mathcal d, \mathcal d')\) be an \( af _2\) and \(s \subseteq \mathcal a\). a set s is compatible (in \( af \)) if \(\forall a \in s\), \(\not \exists b \in s\) such that \((b,a) \in \mathcal d\). compatible sets do not contain the arguments a and b if a attacks b (or vice versa); however they can accommodate weak attacks between their members. this means in example 3, the set \(\left\{ a_5,a_6\right\} \) is compatible, but \(\left\{ a_2, a_6\right\} \) is not. now we are entitled to define semantics to deal with compatible sets: definition 11 (semantics). let \( af = (\mathcal a, \mathcal d, \mathcal d')\) be an \( af _2\) and \(s \subseteq \mathcal a\) be a compatible set of arguments. then \(x \in \mathcal a\) is acceptable with respect to s iff \(\forall y \in \mathcal a\) such that \((y,x) \in \mathcal d\) : \(\exists z \in s\) such that \((z, y ) \in \mathcal d\) and \(\forall y \in \mathcal a\) such that \((y,x) \in \mathcal d'\) : \(\exists z \in s\) such that \((z, y ) \in \mathcal d \cup \mathcal d'\). we define \(f_ af (s) = \left\{ a \in \mathcal a\mid a \text { is acceptable w.r.t. } s\right\} \). for a compatible set s in \( af \), we say 1) s is an admissible set of \( af \) iff \(s \subseteq f_ af (s)\); 2) s is a complete extension of \( af \) iff \(f_ af (s) = s\); 3) s is a preferred extension of \( af \) iff it is a set inclusion maximal complete extension of \( af \); 4) s is the grounded extension iff it is the set inclusion minimal complete extension of \( af \); 5) s is a stable extension iff s is complete extension of \( af \) and \(\forall y \not \in s\), \(\exists x \in s\) s.t. \((x, y ) \in \mathcal d \cup \mathcal d'\). 6) s is a semi-stable extension iff it is a complete extension of \( af \) such that there is no complete extension \(s_1\) of \( af \) in which \(s \cup s^+ \subset s_1 \cup s_1^+\). notice for an argument x to be acceptable w.r.t. s, if \((y,x) \in \mathcal d\), there should exist an argument \(z \in s\) such that \((z,y) \in \mathcal d\), i.e., a weak defeat as \((z,y) \in \mathcal d'\) is not robust enough to defend a defeat as \((y,x) \in \mathcal d\). otherwise, x defends a weak defeat \((y,x) \in \mathcal d'\) if \(\exists z \in s\) such that z (weak) defeats y. example 4 (example 3 continued). regarding the \( af _2\) constructed in example 3, we obtain complete extensions: \(\left\{ a_1, a_3, a_4\right\} \), \(\left\{ a_1, a_3, a_4, a_5\right\} \), \(\left\{ a_1, a_3, a_4, a_6\right\} \), \(\{a_1,a_3, a_4, a_5, a_6 \}\); grounded extension: \(\left\{ a_1, a_3, a_4\right\} \); preferred extension: \(\left\{ a_1, a_3, a_4, a_5, a_6\right\} \) stable/semi-stable extensions: \(\left\{ a_1, a_3, a_4, a_6\right\} \), \(\left\{ a_1, a_3, a_4, a_5, a_6\right\} \) 3 the postulates of non-interference and crash-resistance in this section, we show under which conditions, \( aspic ^?\) satisfies the property that a conflict between two arguments should not interfere with the acceptability of other unrelated arguments. let us illustrate it via the following example: example 5 [6] let \(\mathcal r_d= \left\{ p \rightarrow q; r \rightarrow \lnot q; t \rightarrow s \right\} \) be a set of defeasible rules in \( aspic ^+\) (their conclusions are not ?-suffixed formulas), \(\mathcal k_p= \emptyset \) and \(\mathcal k_n= \left\{ p, r, t\right\} \), while \(\mathcal r_s\) consists of all propositionally valid inferences. the corresponding \( af \) includes the arguments \(a_1 = [p]\), \(a_2 = [ a_1 \rightarrow q]\), \(b_1 = [r]\), \(b_2 = [b_1 \rightarrow \lnot q]\), \(c = [a_2, b_2 \rightarrow \lnot s]\), \(d_1 = [t]\) and \(d_2 = [d_1 \rightarrow s]\). we have c defeats \(d_2\) if \(c \not \prec d_2\). this is problematic as s can be any formula. hence, any defeasible argument unrelated to \(a_2\) or \(b_2\) can, depending on \(\preceq \), be defeated by c owing to the explosiveness of classical logic as the source for \(\mathcal r_s\). this property is guaranteed by proving the postulates (originally conceived in [10]) of non-interference (definition 22) and crash-resistance (definition 25) hold in \( aspic ^?\). unlike [7, 8], however, we will not eliminate all arguments whose set of conclusions of all its sub-arguments is contradictory, but only those whose set of conclusions contains strong contradictions as \(\left\{ \phi , \lnot \phi \right\} \), \(\left\{ \phi , \lnot \phi ?\right\} \) or \(\left\{ \lnot \phi , \phi ?\right\} \). weak contradictions as \(\left\{ \phi ?, \lnot \phi ?\right\} \) will not lead to the elimination of the argument (see definition 14). we proceed by introducing several definitions and lemmas before proving the satisfaction of these postulates: definition 12 (consistency). let \(\mathcal a= \left\{ a_1, \ldots , a_n\right\} \) be a set of arguments on the basis of an argumentation theory \(( as , \mathcal k)\) and an argumentation system \( as = (\mathcal l,^-, \mathcal r, n)\). an argument \(a \in \mathcal a\) is inconsistent iff \(\forall \phi \in \mathcal l\), it holds \(\left\{ \mathtt {conc}(a') \mid a' \in \mathtt {sub}(a)\right\} \vdash \phi \). otherwise, a is consistent. the set \(\mathcal a\) is inconsistent if \(\forall \phi \in \mathcal l\), it holds \(\mathtt {concs}(\mathtt {sub}(a_1)) \cup \ldots \cup \mathtt {concs}(\mathtt {sub}(a_n)) \vdash \phi \). otherwise \(\mathcal a\) is consistent. a strict rule as \(\phi _1, \ldots , \phi _n \rightarrow \psi \) represents that if \(\phi _1, \ldots , \phi _n\) hold, then without exception it holds that \(\psi \). it has a very general meaning; the unique restriction we will impose to prove our results is that we will assume throughout this section that every strict rule in an argumentation system is reasonable: definition 13 (reasonable strict rules). let \(( as , \mathcal k)\) be an argumentation theory and \( as = (\mathcal l,^-, \mathcal r_s\cup \mathcal r_d, n)\) be an argumentation system. a strict rule \(\phi _1, \ldots , \phi _n \rightarrow \psi \in \mathcal r_s\) is reasonable iff 1) for each \(\phi _i\) (\(1 \le i \le n\)) it holds \(\texttt {atoms}(\phi _i) \subseteq \texttt {atoms}(\psi )\) or 2) \(\forall \psi \in \mathcal l\), it holds \(\left\{ \phi _1, \ldots , \phi _n\right\} \vdash \psi \). a strict rule \(\phi _1, \ldots , \phi _n \rightarrow \psi \) is reasonable if \(\forall \phi _i\) (\(1 \le i \le n\)), each atom in \(\phi _i\) is also in \(\psi \) or \(\left\{ \phi _1, \ldots , \phi _n\right\} \) is inconsistent. reasonable strict rules are very usual in many propositional logics. now we define an inconsistency cleaned \( af _2\): definition 14 (inconsistency-cleaned \( af _2\)). let \(\langle \mathcal a, \mathcal d, \mathcal d' \rangle \) be a \( af _2\) resulting from an argumentation theory \( at \). we define \(\mathcal a_c = \{ a \in \mathcal a\mid a \text { is consistent} \}\), \(\mathcal d_c = \mathcal d \cap (\mathcal a_c \times \mathcal a_c)\) and \(\mathcal d_c' = \mathcal d' \cap (\mathcal a_c \times \mathcal a_c)\). we refer to \((\mathcal a_c, \mathcal d_c, \mathcal d_c')\) as the inconsistency cleaned \( af _2\) resulting from an argumentation theory \( at \). by inconsistency-cleaned version of the \( aspic ^?\) system, we mean the \( aspic ^?\) system from which the inconsistency-cleaned \( af _2\) is constructed. the next concept is important to simplify the proofs of the results we will show in this section: definition 15 (flat arguments). let a be an argument on the basis of an argumentation theory \( at \). we say that a is flat iff \(\mathtt {toprule}(a)\) is strict, \(a = [a_1, \ldots , a_n \rightarrow \alpha ]\) and \(\forall a_i\) (\(1 \le i \le n\)), one of the following conditions holds: 1. \(\mathtt {toprule}(a_i)\) is defeasible or 2. \(\mathtt {rules}(a_i) = \emptyset \). flat arguments have strict top rule and every of its strict subarguments comes from the set of premises. henceforth, for any \( af _2\langle \mathcal a, \mathcal d, \mathcal d' \rangle \) resulting from argumentation theory \( at \), we will assume without loss of generality every \(a \in \mathcal a\) with a strict top rule is flat. in order to prove some results in this section, we will need to identify the set of conclusions associated with a set of arguments: definition 16 let \(\mathcal a\) be a set of arguments whose structure complies with definition 4. we define \(\mathtt {concs}(\mathcal a) = \left\{ \mathtt {conc}(a) \mid a \in \mathcal a\right\} \). next, we define the consequence function \( cn _ sem \), such that \( cn _ sem ( at )\) is a set of sets of conclusions under the argumentation semantics \( sem \). definition 17 let \(\mathfrak {at}\) be the set of all argumentation theories that can be constructed from a language \(\mathcal l\). let \( at \in \mathfrak {at}\) be an argumentation theory and \( af = (\mathcal a, \mathcal d, \mathcal d')\) be the resulting \( af _2\) from \( at \). we define \( cn _{ sem } : \mathfrak {at} \rightarrow 2^{2^{\mathtt {concs}(\mathcal a)}}\) is a function s.t. \( cn _{ sem }( at ) = \{\mathtt {concs}(e) \mid e \subseteq \mathcal a\textit{ is an extension} \textit{of } af _2 \textit{ under semantics } sem \}\), where \( sem \in \{\textit{complete, grounded, preferred,} \textit{stable, semi-stable}\}\). we will use \( cn _{\textit{c}}( at )\) as the shortening of \( cn _{ complete }( at )\). for a set \(\mathfrak a\) of propositional atoms, by \( cn _{ sem }( at )_{\mid \mathfrak a}\), we mean the set \(\left\{ \mathcal f_{\mid \mathfrak a} \mid \mathcal f \in cn _{ sem }( at )\right\} \). in order to define the postulate for non-interference, we need to specify what the union of two argumentation theories looks like: definition 18 (union of argumentation theories). let \( at _1 = ( as _1, \mathcal k_1)\) and \( at _2 = ( as _2, \mathcal k_2)\) be argumentation theories s.t. \(\mathcal k_1 = \mathcal k_{n_1} \cup \mathcal k_{p_1}\), \(\mathcal k_2 = \mathcal k_{n_2} \cup \mathcal k_{p_2}\), \( as _1 = (\mathcal l,^-, \mathcal r_{s_1} \cup \mathcal r_{d_1}, n_1)\) and \( as _2 = (\mathcal l,^-, \mathcal r_{s_2} \cup \mathcal r_{d_2}, n_2)\). besides, we assume \(n_1(r) = n_2(r)\) for any \(r \in \mathcal r_{d_1} \cup \mathcal r_{d_2}\). we define \( at _1 \cup at _2\) is an argumentation theory \( at = ( as , \mathcal k)\) s.t. \(\mathcal k= \mathcal k_n\cup \mathcal k_p\) with \(\mathcal k_n= \mathcal k_{n_1} \cup \mathcal k_{n_2}\), and \(\mathcal k_p= \mathcal k_{p_1} \cup \mathcal k_{p_2}\); \( as = (\mathcal l,^-, \mathcal r_s \cup \mathcal r_d, n)\) with \(\mathcal r_s = \mathcal r_{s_1} \cup \mathcal r_{s_2}\), \(\mathcal r_d = \mathcal r_{d_1} \cup \mathcal r_{d_2}\) and \(n(r) = n_1(r)\) if \(r \in \mathcal r_{d_1}\); otherwise, \(n(r) = n_2(r)\). other very important notion to guarantee our results in this section is that of syntactically disjoint argumentation theories; it will be employed to characterise non-interference (definition 22) and contamination (definition 24). definition 19 (syntactically disjoint argumentation theories). let \( at _1\) and \( at _2\) be argumentation theories. we say \( at _1\) and \( at _2\) are syntactically disjoint when \(\texttt {atoms}( at _1) \cap \texttt {atoms}( at _2) = \emptyset \). the depth of an argument will be employed in the proof of lemma 1: definition 20 (depth of an argument). let a be an argument whose structure complies with definition 4. the depth of a, denoted by \(\texttt {depth}(a)\), is 1 if \(\mathtt {rules}(a) = \emptyset \) or else \(\texttt {depth}(a) = 1 + max \{\texttt {depth}(a') \mid a' \in \mathtt {sub}(a) \}\). the following essential lemma states that for every argument a such that \(\mathtt {conc}(a) \subseteq \texttt {atoms}(af)\) there exists an argument \(a'\) with the same conclusion, \(\texttt {atoms}(a') \subseteq \texttt {atoms}(af)\), and is not more vulnerable than a. lemma 1 let \( as _1 = (\mathcal l,^-, \mathcal r_s\cup \mathcal r_{d_1}, n_1)\) and \( as _2 = (\mathcal l,^-, \mathcal r_s\cup \mathcal r_{d_2}, n_2)\) be argumentation systems s.t. \(\mathcal r_s\) is a set of strict rules and \(\mathcal r_{d_1}\) and \(\mathcal r_{d_2}\) are sets of defeasible rules. let \( at = at _1 \cup at _2\) be an argumentation theory where \( at _1 = ( as _1, \mathcal k_1)\) and \( at _2 = ( as _2, \mathcal k_2)\) are syntactically disjoint, and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d_1')\) be respectively the inconsistency cleaned \( af _2\)s resulting from \( at \) and \( at _1\). for each argument \(c \in \mathcal a\) such that \(\mathtt {conc}(c) \subseteq \texttt {atoms}( at _1)\), \(\exists c' \in \mathcal a_1\) with \(\mathtt {conc}(c') = \mathtt {conc}(c)\), \(c'^+ \subseteq c^+\) and \(c'^\subseteq c^-\). proof let \(\mathcal k_1 = \mathcal k_{n_1} \cup \mathcal k_{p_1}\) and \(\mathcal k_2 = \mathcal k_{n_2} \cup \mathcal k_{p_2}\). we will prove by induction on \(\texttt {depth}(c)\) that \(\forall c \in \mathcal a\), where \(\texttt {atoms}(\mathtt {conc}(c)) \subseteq \texttt {atoms}( at _1)\), \(\exists c' \in \mathcal a_1\) such that \(\mathtt {conc}(c') = \mathtt {conc}(c)\) and (1) \(\mathtt {concs}(\mathtt {sub}(c')) \subseteq \mathtt {concs}(\mathtt {sub}(c))\) and (2) \(\mathtt {defr}(c') \subseteq \mathtt {defr}(c)\). note \(c'^+ = c^+\) follows directly from \(\mathtt {conc}(c') = \mathtt {conc}(c)\); condition (1) guarantees \(c'\) is consistent as c is consistent; condition (2) suffices to show \(c'^\subseteq c^-\) as \( as _1\) and \( as _2\) share the same set \(\mathcal r_s\) of strict rules: suppose \(\texttt {depth}(c) = 1\). there are two possibilities: \(c \in \mathcal k_{n_1}\) or \(c \in \mathcal k_{p_1}\). thus, for \(c' = c\), \(c'^= c^-\) and \(\mathtt {conc}(c') = \mathtt {conc}(c)\). now assume conditions (1) and (2) hold for any argument c with \(\texttt {depth}(c) \le k\). we will show for any argument c with \(\texttt {depth}(c) = k + 1\) they also hold. there are two possibilities: \(\mathtt {toprule}(c) \in \mathcal r_d= \mathcal r_{d_1} \cup \mathcal r_{d_2}\). note c is of the form \(c_1, \ldots , c_n \rightarrow \mathtt {conc}(c)\). as \( at _1\) and \( at _2\) are syntactically disjoint and \(\texttt {atoms}(\mathtt {conc}(c)) \subseteq \texttt {atoms}( at _1)\), \(\mathtt {toprule}(c) \in \mathcal r_{d_1}\). it follows that for each \(i \in \left\{ 1, \ldots , n\right\} \), \(\texttt {atoms}(\mathtt {conc}(c_i)) \subseteq \texttt {atoms}(at_1)\). as \(\texttt {depth}(c_i) \le k\), by induction hypothesis, there exists \(c'_i \in \mathcal a_1\) such that \(\mathtt {conc}(c'_i) = \mathtt {conc}(c_i)\), \(\mathtt {defr}(c'_i) \subseteq \mathtt {defr}(c'_i)\), and \(\mathtt {concs}(\mathtt {sub}(c'_i)) \subseteq \mathtt {concs}(\mathtt {sub}(c_i))\). applying \(\mathtt {toprule}(c)\) we construct \(c' = c'_1, \ldots , c'_n \rightarrow \mathtt {conc}(c)\) from \(at_1\). now we show that \(c'\) satisfies the requested properties. \(c' \in \mathcal a_1\) since \(\texttt {atoms}(c') = \texttt {atoms}(\mathtt {toprule}(c)) \cup \bigcup ^n_{i=1} \texttt {atoms}(c'_i) \subseteq \texttt {atoms}( at _1)\). \(\mathtt {conc}(c') = \mathtt {conc}(c)\) since c and \(c'\) share the same top rule. \(\mathtt {defr}(c') = \left\{ \left\{ \mathtt {defr}(c)\right\} \cup \bigcup ^n_{i=1} \mathtt {defr}(c'_i)\right\} \subseteq \) \(\left\{ \left\{ \mathtt {toprule}(c)\right\} \cup \bigcup ^n_{i=1} \mathtt {defr}(c_i)\right\} = \mathtt {defr}(c)\). \(\mathtt {concs}(\mathtt {sub}(c')) = \left\{ \left\{ \mathtt {conc}(c)\right\} \cup \bigcup ^n_{i=1} \mathtt {concs}(\mathtt {sub}(c'_i))\right\} \subseteq \) \(\left\{ \left\{ \mathtt {conc}(c)\right\} \cup \bigcup ^n_{i=1} \mathtt {concs}(\mathtt {sub}(c_i))\right\} = \mathtt {concs}(\mathtt {sub}(c))\). \(\mathtt {toprule}(c) \in \mathcal r_s\). note c is of the form \(c_1, \ldots , c_n \rightarrow \mathtt {conc}(c)\). as we have assumed c is flat, we can partition arguments \(c_i\) into two sets \(\mathfrak c_p \cup \mathfrak c_d = \left\{ 1, \ldots , n\right\} \), in which \(i \in \mathfrak c_p\) iff \(\mathtt {conc}(c_i) \in \mathcal k_1 \cup \mathcal k_2\) and \(\mathtt {toprule}(c_i) = \textit{undefined}\), and \(i \in \mathfrak c_d\) iff \(\mathtt {toprule}(c_i) \in \mathcal r_{d_1} \cup \mathcal r_{d_2}\). since \( at _1\) and \( at _2\) are syntactically disjoint, for \(i \in \mathfrak c_p\), \(\texttt {atoms}(\mathtt {conc}(c_i)) \subseteq \texttt {atoms}(at_1)\) or \(\texttt {atoms}(\mathtt {conc}(c_i)) \subseteq \texttt {atoms}(at_2)\), and for \(i \in \mathfrak c_d\), \(\texttt {atoms}(\mathtt {toprule}(c_i)) \subseteq \texttt {atoms}(at_1)\) or \(\texttt {atoms}(\mathtt {toprule}(c_i)) \subseteq \texttt {atoms}(at_2)\). we can partition the subarguments of c into two disjoint sets \(\mathfrak c_1\) and \(\mathfrak c_2\) such that for \(i \in \mathfrak c_1\), \(\mathtt {conc}(c_i) \subseteq \texttt {atoms}(at_1)\) and for \(i \in \mathfrak c_2\), \(\mathtt {conc}(c_i) \subseteq \texttt {atoms}(at_2)\). let \(\mathfrak c_p \cap \mathfrak c_1 = \left\{ p_1, \ldots , p_k\right\} \), \(\mathfrak c_d \cap \mathfrak c_1 = \left\{ d_1, \ldots , d_m\right\} \), and \(\mathfrak c_2 = \left\{ b_1, \ldots , b_j\right\} \). for each \(d_i \in \mathfrak c_d \cap \mathfrak c_1\), \(\texttt {depth}(c_{d_i}) \le k\). by the induction hypothesis, for each \(d_i \in \mathfrak c_d \cap \mathfrak c_1\), exists \(c'_{d_i} \in \mathcal a_1\) s.t \(\mathtt {conc}(c'_{d_i}) = \mathtt {conc}(c_{d_i})\), and \(\mathtt {defr}(c'_{d_i}) \subseteq \mathtt {defr}(c_{d_i})\). note that \(\mathtt {conc}(c_{p_1}), \ldots , \mathtt {conc}(c_{p_k})\), \(\mathtt {conc}(c'_{d_1}), \ldots , \mathtt {conc}(c'_{d_m}), \mathtt {conc}(c_{b_1}),\ldots , \mathtt {conc}(c_{b_j}) \rightarrow \mathtt {conc}(c)\) corresponds to \(\mathtt {toprule}(c)\). let \(t = \texttt {atoms}(\mathtt {conc}(c_{b_1})) \cup \ldots \cup \texttt {atoms}(\mathtt {conc}(c_{b_j}))\). as \(t \cap \texttt {atoms}(\mathtt {conc}(c))= \emptyset \), \(\mathtt {conc}(c_{p_1}),\ldots , \mathtt {conc}(c_{p_n}), \mathtt {conc}(c'_{d_1}), \ldots , \mathtt {conc}(c'_{d_m}) \rightarrow \mathtt {conc}(c) \in \mathcal r_s\); otherwise \(\{\mathtt {conc}(c_{b_1}), \ldots , \mathtt {conc}(c_{b_j})\}\) is inconsistent (definition 13), which cannot be true as c is consistent. thus, we can construct an argument \(c' = \mathtt {conc}(c_{p_1}), \ldots , \mathtt {conc}(c_{p_n}),\mathtt {conc}(c'_{d_1}), \ldots , \mathtt {conc}(c'_{d_m}) \rightarrow \mathtt {conc}(c)\) from \(at_1\) s.t. \(\mathtt {conc}(c') = \mathtt {conc}(c)\). now we show \(c'\) satisfies the requested properties. \(c' \in \mathcal a_1\) since \(\texttt {atoms}(c') = \texttt {atoms}(\mathtt {toprule}(c)) \cup \bigcup _{(1 \le i \le n)} \texttt {atoms}(c'_i) \subseteq \texttt {atoms}( at _1)\). \(\mathtt {conc}(c') = \mathtt {conc}(c)\) since c and \(c'\) share the same top rule. in the construction of \(c'\) we resort to \(c_i\) for each \(i \in \mathcal c_1 \cap \mathfrak c_p\) and for each \(j \in \mathcal c_1 \cap \mathfrak c_d\), we resort to \(c'_j\), obtained by induction hypothesis, s.t. \(\mathtt {defr}(c'_j) \subseteq \mathtt {defr}(c_j)\) and \(\mathtt {concs}(\mathtt {sub}(c_j')) \subseteq \mathtt {concs}(\mathtt {sub}(c_j))\). it follows \(\mathtt {defr}(c') \subseteq \mathtt {defr}(c)\) and \(\mathtt {concs}(\mathtt {sub}(c')) \subseteq \mathtt {concs}(\mathtt {sub}(c))\).    \(\square \) the notion of defense expresses when an argument c defends b from a: definition 21 (defense). let \( af = (\mathcal a, \mathcal d, \mathcal d')\) be an \( af _2\) and \(a, b, c \in \mathcal a\) such that \((a,b) \in \mathcal d \cup \mathcal d'\). an argument c defends b from a, denoted by \( df (c,b,a)\), when 1) if \((a,b) \in \mathcal d\), then \((c,a) \in \mathcal d\); 2) if \((a,b) \in \mathcal d'\), then \((c,a) \in \mathcal d\) or \((c,a) \in \mathcal d'\). in addition to the fundamental result obtained by lemma 1, we will use lemmas 2, 3, 5 and 6 to prove theorem 1, which is one of our main results. with the following lemma we establish a connection between complete extensions of \( af \) with the arguments in \(\mathcal a_1\): lemma 2 let \( at = at _1 \cup at _2\) for syntactically disjoint argumentation theories \( at _1\) and \( at _2\), and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d'_1)\) be respectively the resulting inconsistency-cleaned \( af _2\)s from \( at \) and \( at _1\). for any complete extension \(e \subseteq \mathcal a\) of \( af \), \(\mathtt {concs}(e \cap \mathcal a_1) = \mathtt {concs}(e)_{\mid \texttt {atoms}( at _1)}\). proof if \(\phi \in \mathtt {concs}(e \cap \mathcal a_1)\), then \(\exists a \in e \cap \mathcal a_1\) such that \(\mathtt {conc}(a) = \phi \). it follows \(a \in e\) and \(a \in \mathcal a_1\), and so \(\phi \in \mathtt {concs}(e)\) and \(\phi \in \mathtt {concs}(\mathcal a_1)\). as \(\texttt {atoms}(\mathtt {conc}(\phi )) \subseteq \texttt {atoms}( at _1)\), it holds \(\phi \in \mathtt {concs}(e)_{\mid \texttt {atoms}( at _1)}\). if \(\phi \in \mathtt {concs}(e)_{\mid \texttt {atoms}( at _1)}\), then \(\exists a \in e\) s.t. \(\mathtt {conc}(a) = \phi \) and \(\texttt {atoms}(\phi ) \subseteq \texttt {atoms}( at _1)\). from lemma 1, \(\exists a' \in \mathcal a_1\) such that \(\mathtt {conc}(a') = \phi \) and \(a'^\subseteq a^-\). as e is a complete extension of \( af \) and \(a \in e\), it must be \(a' \in e\), and so \(a' \in e \cap \mathcal a_1\). thus, \(\phi \in \mathtt {concs}(e \cap \mathcal a_1)\).    \(\square \) lemmas 3 and 4 will be used as intermediate steps in the demonstration of lemma 5. lemma 3 expresses the function \(f_{ af _1}\) associated with the subframework \( af _1\) in terms of the function \(f_ af \) associated with \( af \): lemma 3 let \( at = at _1 \cup at _2\) for syntactically disjoint argumentation theories \( at _1\) and \( at _2\), and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d'_1)\) be respectively the resulting inconsistency-cleaned \( af _2\)s from \( at \) and \( at _1\). for any set of arguments \(s \subseteq \mathcal a_1\), \(f_{ af _1}(s) = f_{ af }(s) \cap \mathcal a_1\). proof \(f_{ af _1}(s) \subseteq f_{ af }(s) \cap \mathcal a_1\). if \(a \in f_{ af _1}(s)\), then \(a \in \mathcal a_1\). it remains to prove \(a \in f_{ af }(s)\): let \(b \in \mathcal a\) s.t. \((b,a) \in \mathcal d \cup \mathcal d'\). it means \(\texttt {atoms}(\mathtt {conc}(b)) \subseteq \texttt {atoms}( at _1)\). by lemma 1, \(\exists b' \in \mathcal a_1\) s.t. \(\mathtt {conc}(b') = \mathtt {conc}(b)\) and \(b'^\subseteq b^-\). thus, \((b',a) \in \mathcal d \cup \mathcal d'\). as \(a \in f_{ af _1}(s)\), \(\exists c \in s\) s.t. \( df (c,a,b')\). from \(b'^\subseteq b^-\), \((c,b) \in \mathcal d \cup \mathcal d'\). it follows \(a \in f_{ af }(s)\). thus \(a \in f_{ af }(s) \cap \mathcal a_1\). \(f_{ af }(s) \cap \mathcal a_1 \subseteq f_{ af _1}(s)\). let \(a \in f_{ af }(s) \cap \mathcal a_1\). it means that \(\forall b \in \mathcal a\) such that \((b,a) \in \mathcal d \cup \mathcal d'\), \(\exists c \in s\) such that \( df (c,a,b)\). as \(\mathcal d_1 \cup \mathcal d'_1 \subseteq \mathcal d \cup \mathcal d'\), it follows \(\forall b \in \mathcal a_1\) such that \((b,a) \in \mathcal d_1 \cup \mathcal d_1'\), \(\exists c \in s\) such that \( df (c,a,b)\). then \(a \in f_{ af _1}(s)\).    \(\square \) by lemma 4 (employed in lemma 5), if an argument a is attacked by a set of arguments that also attack arguments in a complete extension e, then \(a \in e\). lemma 4 let \( af = (\mathcal a, \mathcal d, \mathcal d')\) be the \( af _2\) resulting from an argumentation theory \( at \). let \( af _c = (\mathcal a_c, \mathcal d_c, \mathcal d'_c)\) be the inconsistency-cleaned \( af _2\). let e be a complete extension of \( af _c\), \(s \subseteq e\) and \(a \in \mathcal a\). if a is consistent and \(a^\subseteq s^-\), then \(a \in e\). proof suppose a is consistent. it follows \(a \in \mathcal a_c\). as e is a complete extension of \( af \), \(\forall b \in s\), \(b \in f_{ af }(e)\). as \(a^\subseteq s^-\), it follows \(a \in f_{ af }(e) = e\).    \(\square \) lemmas 5 assures the complete extensions of \( af \) when restricted to the arguments of its subframework \( af _1\) is a complete extension of \( af _1\): lemma 5 let \( at = at _1 \cup at _2\) for syntactically disjoint argumentation theories \( at _1\) and \( at _2\), and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d'_1)\) be the inconsistency-cleaned \( af _2\)s resulting from \( at \) and \( at _1\) respectively. if e is a complete extension of \( af \), then \(e \cap \mathcal a_1\) is a complete extension of \( af _1\). proof let e be a complete extension of \( af \). assume \(e' = e \cap \mathcal a_1\). we will prove that \(e'\) is a complete extension of \( af _1\). note \(e'\) is compatible in \( af _1\), since \(e' \subseteq e\), e is compatible in \( af \) and there are no new defeats in \( af _1\). now we will show \(e' = f_{ af _1}(e')\): \(e' \subseteq f_{ af _1}(e')\). let \(a \in e'\). as also \(a \in e\) and e is a complete extension of \( af \), it means \(\forall b \in \mathcal a_1\) such that \((b,a) \in \mathcal d_1 \cup \mathcal d_1'\), \(\exists c \in e\) such that \( df (c,a,b)\). as \(b \in \mathcal a_1\), \(\texttt {atoms}(\mathtt {conc}(c)) \subseteq \texttt {atoms}( at _1)\). by lemma 1, \(\exists c' \in \mathcal a_1\), such that \(\mathtt {conc}(c') = \mathtt {conc}(c)\) and \(c'^\subseteq c^-\). from lemma 4, \(c' \in e\), and so \(c' \in e'\). thus, \(a \in f_{ af _1}(e')\). \(f_{ af _1}(e') \subseteq e'\). as f is a monotony function, and e is a complete extension of \( af \), \(f_{ af }(e') \subseteq f_{ af }(e) = e\), and so \(f_{ af }(e') \cap \mathcal a_1 \subseteq e \cap \mathcal a_1\). from lemma 3, \(f_{ af _1}(e') = f_{ af }(e') \cap \mathcal a_1\). but then, we obtain \(f_{ af _1}(e') \subseteq e \cap \mathcal a_1\). thus, \(f_{ af _1}(e') \subseteq e'\).    \(\square \) lemma 6 assures the admissible sets in \( af _1\) are also admissible sets in \( af \): lemma 6 let \( at = at _1 \cup at _2\) for syntactically disjoint argumentation theories \( at _1\) and \( at _2\), and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d'_1)\) be respectively the resulting inconsistency-cleaned \( af _2\)s from \( at \) and \( at _1\). let \(s \subseteq \mathcal a_1\). if s is an admissible set in \( af _1\), then s is an admissible set if \( af \). proof as s is a compatible set in \( af _1\), s is also a compatible set in \( af \), since \(\mathcal a_1 \subseteq \mathcal a\). it remains to show that \(s \subseteq f_{ af }(s)\). assume \(b \in \mathcal a\) such that for some \(a \in s\), \((b,a) \in \mathcal d \cup d'\). it means \(\texttt {atoms}(\mathtt {conc}(b)) \subseteq \texttt {atoms}( at _1)\). from lemma 1, \(\exists b' \in \mathcal a_1\) such that \(\mathtt {conc}(b') = \mathtt {conc}(b)\) and \(b'^\subseteq b^-\). note \((b',a) \in \mathcal d \cup d'\) in \( af _1\). as s is admissible in \( af _1\), from definition 11, \(\exists c \in s\) s.t. \(df(c,a,b')\). given \(b'^\subseteq b^-\), \((c,b) \in \mathcal d \cup \mathcal d'\). it follows \(s \subseteq f_{ af }(s)\).    \(\square \) in the sequel, we formally define the concept of non-interference. definition 22 (non-interference). the \( aspic ^?\) system satisfies non-interference under a semantics \( sem \) iff for every syntactically disjoint argumentation theories \( at _1\) and \( at _2\), it holds \( cn _{ sem }( at _1 \cup at _2)_{\mid \texttt {atoms}( at _1)} = cn _{ sem }( at _1)\). non-interference means that, for disjoint argumentation theories \( at _1\) and \( at _2\), \( at _1\) does not influence the outcome with respect to the language of \( at _2\). theorem 1 the inconsistency-cleaned version of the \( aspic ^?\) system satisfies non-interference under complete semantics. proof let \( at = at _1 \cup at _2\) for syntactically disjoint argumentation theories \( at _1\) and \( at _2\) and \( af = (\mathcal a, \mathcal d, \mathcal d')\) and \( af _1 = (\mathcal a_1, \mathcal d_1, \mathcal d'_1)\) be respectively the resulting inconsistency-cleaned \( af _2\)s from \( at \) and \( at _1\). let \(\mathfrak s_1 = \left\{ b_1, \ldots , b_n\right\} \) and \(\mathfrak s_2 = \left\{ s_1, \ldots , s_m\right\} \) be the set of complete extensions of \( af \) and \( af _1\) respectively. we will prove that \(l = r\), in which \(l = cn _{\textit{c}}( at )_{\mid \texttt {atoms}( at _1)} = \left\{ \mathtt {concs}(b_1)_{\mid \texttt {atoms}( at _1)}, \ldots , \mathtt {concs}(b_n)_{\mid \texttt {atoms}( at _1)}\right\} \), and \(r = cn _{\textit{c}}( at _1)_{\mid \texttt {atoms}( at _1)}= \left\{ \mathtt {concs}(s_1), \ldots , \mathtt {concs}(s_m)\right\} \). from lemma 2, \(l = \left\{ \mathtt {concs}(b_1 \cap \mathcal a_1), \ldots , \mathtt {concs}(b_n \cap \mathcal a_1)\right\} \). for each complete extension b of \( af \), \(b \cap \mathcal a_1\) is a complete extension of \( af _1\) (lemma 5). it remains to prove for any \(s \in \mathfrak s_2\), \(\exists b \in \mathfrak s_1\) with \(b \cap \mathcal a_1 = s\). if s is a complete extension of \( af _1\) (\(s = f_{ af _1}(s)\)), s is an admissible set in \( af \) (lemma 6), i.e., \(s \subseteq f_{ af }(s)\). then \(b = \bigcup ^{\infty }_{n=1} f^n_{ af }(s)\) is a complete extension of \( af \) as the least fixed point of \(f_{ af }\) contains s. we will prove that \(b \cap \mathcal a_1 = s\). intersecting both sides of \(b = \bigcup ^{\infty }_{n=1} f^n_{ af }(s)\) with \(\mathcal a_1\), and applying lemma 3, we get \(b \cap \mathcal a_1 = (\bigcup ^{\infty }_{n=1} f^n_{ af }(s)) \cap \mathcal a_1 = \bigcup ^{\infty }_{n=1} f^n_{ af }(s) \cap \mathcal a_1 = \bigcup ^{\infty }_{n=1} f^n_{ af _1}(s) = \bigcup ^{\infty }_{n=1} s = s\).    \(\square \) \( aspic ^?\) is non trivial under semantics \( sem \) if the conclusions of an argumentation theory are never fully determined by the atoms. definition 23 (non-trivial). the \( aspic ^?\) system is non-trivial under semantics \( sem \) iff for each nonempty set \(\mathfrak a\) of atoms, there are argumentation theories \( at _1\) and \( at _2\) such that \(\texttt {atoms}( at _1) = \texttt {atoms}( at _2)\) and \( cn _{ sem }( at _1)_{\mid \mathfrak a} \ne cn _{ sem }( at _2)_{\mid \mathfrak a}\). in the following theorem, we show that the inconsistency-cleaned version of the \( aspic ^?\) system satisfies non-triviality under complete semantics: theorem 2 the inconsistency-cleaned version of the \( aspic ^?\) system satisfies non-triviality under complete semantics. proof let \(\mathfrak a = \left\{ a_1, \ldots , a_n\right\} (n \ge 1)\) a set of atoms. we will show that there are two inconsistency-cleaned argumentation frameworks \( af _1 = (\mathcal a_1,\mathcal d_1, \mathcal d_1')\) and \( af _2 = (\mathcal a_2,\mathcal d_2, \mathcal d'_2)\) resulting from \( at _1\) and \( at _2\) respectively such that \(\texttt {atoms}( at _1)= \texttt {atoms}( at _2)\) and \( cn _{\textit{c}}( at _1)_{\mid \mathfrak a} \ne cn _{\textit{c}}( at _2)_{\mid \mathfrak a}\). let \(\mathcal k_{n_1} = \mathcal k_{n_2} = \mathcal k_{p_2} = \emptyset \), \(\mathcal k_{p_1} = \left\{ a_1, \ldots , a_n\right\} \), \(\mathcal r_{d_2} = \left\{ a_1 \rightarrow a_1?; \ldots ; a_n \rightarrow a_n? \right\} \), \(\mathcal r_{s_1} = \mathcal r_{s_2} = \mathcal r_{d_1} = \emptyset \). thus, \( cn _{\textit{c}}( at _1) = \left\{ \left\{ a_1, \ldots , a_n\right\} \right\} \) and \( cn _{\textit{c}}( at _2) = \left\{ \emptyset \right\} \), and so \( cn _{\textit{c}}( at _1) = cn _{\textit{c}}( at _1)_{\mid \mathfrak a} \ne cn _{\textit{c}}( at _2)_{\mid \mathfrak a} = cn _{\textit{c}}( at _2)\).   \(\square \) an argumentation theory \( at _1\) is contaminating when any other unrelated argumentation theory \( at _2\) becomes irrelevant when merged with \( at _1\): definition 24 (contamination). an argumentation theory \( at _1\) is contaminating under a semantics \( sem \) iff for every argumentation theory \( at _2\) s.t. \( at _1\) and \( at _2\) are syntactically disjoint, it holds \( cn _{ sem }( at _1) = cn _{ sem }( at _1 \cup at _2)\). crash-resistance is strongly related to the concept of contamination: definition 25 (crash-resistance). we say that \( aspic ^?\) under a semantics \( sem \) satisfies crash-resistance iff there does not exists an argumentation theory \( at \) that is contaminating under \( sem \). the intuition behind crash-resistance is that one wants to avoid local problems having global effects. theorem 3 if \( aspic ^?\) satisfies non-interference and non-triviality under complete semantics, then it also satisfies crash-resistance under complete semantics. proof (1) by absurd suppose the \( aspic ^?\) does not satisfy crash-resistance. then there exists an argumentation theory \( at _1\) that is contaminating and \(\texttt {atoms}( at _1) \subset \mathfrak a\). let \(\mathfrak b = \mathfrak a \backslash \texttt {atoms}( at _1)\). (2) by assumption \( aspic ^?\) is non-trivial. thus, there are argumentation theories \( at _2\) and \( at _3\) such that \(\texttt {atoms}( at _2) = \texttt {atoms}( at _3) \subseteq \mathfrak b\) and \( cn _\textit{c}( at _2)_{\mid \mathfrak b} \not = cn _\textit{c}( at _3)_{\mid \mathfrak b}\). note that both \( at _2\) and \( at _3\) are syntactically disjoint from \( at _1\). (3) by assumption \( aspic ^?\) satisfies non-interference, from which follows \( cn _\textit{c}( at _2)_{\mid \mathfrak b} = cn _\textit{c}( at _2 \cup at _1)_{\mid \mathfrak b}\) and \( cn _\textit{c}( at _3 \cup at _1)_{\mid \mathfrak b} = cn _\textit{c}( at _3)_{\mid \mathfrak b}\). (4) given \( at _1\) is contaminating, \( cn _\textit{c}( at _1 \cup at _2)_{\mid \mathfrak b} = cn _\textit{c}( at _1)_{\mid \mathfrak b} = cn _\textit{c}( at _1 \cup at _3)_{\mid \mathfrak b}\). from (3) and (4), it follows that \( cn _\textit{c}( at _2)_{\mid \mathfrak b} = cn _\textit{c}( at _3)_{\mid \mathfrak b}\). it is an absurd as from (2) we have \( cn _\textit{c}( at _2)_{\mid \mathfrak b} \not = cn _\textit{c}( at _3)_{\mid \mathfrak b}\).   \(\square \) theorem 4 the inconsistency-cleaned version of the \( aspic ^?\) system satisfies crash-resistance under complete semantics. proof it follows from theorems 2, 1 and 3. 4 conclusion and future works in this work, we defined an argumentation framework, dubbed \( aspic ^?\), by introducing in \( aspic ^+\) [3] an interrogation mark ? as a plausibility operator to enhance any defeasible conclusion does not have the same status than an irrefutable one: in \( aspic ^?\), any defeasible rule have the form \(\phi _1, \ldots , \phi _n \rightarrow \phi ?\). as in [2], we distinguish strong contradictions from weak ones. we avoid the former and tolerate the latter. then, we showed in \( aspic ^?\) conflicting arguments does not interfere with the acceptability of unrelated arguments. this is proved by combining solutions found in [6] and in [7, 8] to show the postulates of non-interference and crash-resistance hold in inconsistency-cleaned \( aspic ^?\): 1) as in [6], we resort to paraconsistent reasoning to tolerate conflicts; our differential is we tolerate only weak conflicts. 2) as in [7, 8], we require for each argument, the set of conclusions of all its sub-arguments are consistent; our differential is that we eliminate only those arguments whose sets of conclusions lead to a strong conflict. thus, our work paves the way to investigate in the context of structured argumentation alternative solutions to satisfy the postulates of non-interference and crash-resistance without having to delete all inconsistent arguments. in the future we will study other ways to satisfy these postulates and which monotonic paraconsistent logics can be used as source of strict rules to avoid contaminating argumentation theories. we will also exploit the relation between \( aspic ^?\) and extended logic programas with paraconsistent semantics [13]. references carnielli, w., marcos, j.: a taxonomy of c-systems. in: paraconsistency, pp. 24–117. crc press (2002) google scholar  pequeno, t., buchsbaum, a.: the logic of epistemic inconsistency. in: proceedings of the second international conference on principles of knowledge representation and reasoning, pp. 453–460 (1991) google scholar  modgil, s., prakken, h.: a general account of argumentation with preferences. artif. intell. 195, 361–397 (2013) article  mathscinet  google scholar  gorogiannis, n., hunter, a.: instantiating abstract argumentation with classical logic arguments: postulates and properties. artif. intell. 175(9–10), 1479–1497 (2011) article  mathscinet  google scholar  caminada, m., modgil, s., oren, n.: preferences and unrestricted rebut. computational models of argument (2014) google scholar  grooters, d., prakken, h.: combining paraconsistent logic with argumentation. in: comma, pp. 301–312 (2014) google scholar  wu, y.: between argument and conclusion-argument-based approaches to discussion, inference and uncertainty. ph.d. thesis, university of luxembourg (2012) google scholar  wu, y., podlaszewski, m.: implementing crash-resistance and non-interference in logic-based argumentation. j. logic comput. 25(2), 303–333 (2015) article  mathscinet  google scholar  rescher, n., manor, r.: on inference from inconsistent premisses. theory decis. 1(2), 179–217 (1970) article  google scholar  caminada, m.: semi-stable semantics. omma 144, 121–130 (2006) mathscinet  google scholar  dung, p.: on the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. artif. intell. 77(2), 321–357 (1995) article  mathscinet  google scholar  prakken, h.: an abstract framework for argumentation with structured arguments. argument comput. 1(2), 93–124 (2010) article  google scholar  damásio, c., moniz pereira, l.: a survey of paraconsistent semantics for logic programs. in: besnard, p., hunter, a. (eds.) reasoning with actual and potential contradictions. handbook of defeasible reasoning and uncertainty management systems, vol. 2, pp. 241–320. springer, cham (1998). https://doi.org/10.1007/978-94-017-1739-7_8 download references author information authors and affiliations department of computer science, federal university of ceará, fortaleza, brazil rafael silva & joão alcântara authors rafael silvaview author publications search author on:pubmed google scholar joão alcântaraview author publications search author on:pubmed google scholar corresponding author correspondence to rafael silva . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silva, r., alcântara, j. (2021). \( aspic ^?\) and the postulates of non-interference and crash-resistance. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_22 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords argumentation paraconsistency conflict-tolerance publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature modeling and predicting crimes in the city of são paulo using graph neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper modeling and predicting crimes in the city of são paulo using graph neural networks conference paper first online: 30 january 2025 pp 372–386 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) modeling and predicting crimes in the city of são paulo using graph neural networks download book pdf download book epub waqar hassan  orcid: orcid.org/0000-0002-6956-82499, marvin mendes cabral  orcid: orcid.org/0009-0006-5144-03909, thiago rodrigo ramos  orcid: orcid.org/0000-0001-6271-709910, antonio castelo filho  orcid: orcid.org/0000-0001-8009-45779 & … luis gustavo nonato  orcid: orcid.org/0000-0002-8514-80339  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 473 accesses 2 citations abstract crime prediction is a critical research area for enhancing public safety and optimizing law enforcement resource allocation, and machine learning techniques have had a significant impact in this field. traditional machine learning models have long struggled to capture complex crime patterns, primarily due to the intricate interdependence of spatial and temporal data. however, recent advancements in machine learning, particularly with graph neural networks (gnns), offer a new perspective. gnns have demonstrated remarkable success in various applications and they can also play a significant role in crime analysis and prediction. therefore, in this work, we explore such a potential by examining two distinct spatiotemporal gnn architectures, namely dynamic self-attention network (dysat) and evolving graph convolutional network (evolvegcn), assessing and comparing their effectiveness for crime prediction. moreover, we propose a data modeling framework that integrates crime, street map graphs, and urban data, which is fundamental to properly train the gnn models. as far as we know, there is no consolidated methodology to integrate those three modalities of data, being a relevant contribution of this work. our findings underscore the effectiveness of gnns in crime prediction tasks, offering valuable insights for researchers and practitioners in the field of crime prevention and public safety enhancement. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others crime prediction with graph neural networks and multivariate normal distributions article 01 august 2022 ai-driven crime prediction: a systematic literature review article 16 april 2025 deep temporal multi-graph convolutional network for crime prediction chapter © 2020 1 introduction crime prediction has long been a topic of great interest, supporting law enforcement agencies in their decision-making processes [11]. in this context, machine learning (ml) techniques have played a major role, providing a multitude of analytical tools to scrutinize and predict crime-related phenomena [1, 8, 17, 20]. however, traditional machine learning algorithms often fail to capture complex spatiotemporal patterns inherent in non-random crime events [13]. a primary reason for their limited performance is that these models handle each data point independently, disregarding the interconnected nature of events over time and space [14]. recent advancements in machine learning, particularly in graph neural networks (gnns) [31], bring new perspectives to these challenging scenarios. specifically, gnns have demonstrated remarkable success across various applications involving spatiotemporal data, including social network evolution [7], traffic prediction [12], and identification of urban structural patterns [30]. despite the success in various domains, graph neural networks have not yet been fully explored for crime prediction tasks, where traditional ml models still dominate the field. among the few works that utilize gnns for crime analysis and forecasting is the model proposed by han et al. [10], which leverages graph convolutional networks (gcns) and long short-term memory (lstm) networks to capture spatial dependencies and temporal patterns of crimes. a similar methodology has been proposed by jin et al. [13], where graph convolutional networks and recurrent neural networks (rnns) are combined to predict crime hotspots. these studies demonstrate the potential of gnns for analyzing and forecasting crime events. effective data modeling and pre-processing are crucial when preparing datasets for graph neural networks, particularly for crime prediction tasks. these processes dictate how raw data is transformed and integrated into graph structures, significantly impacting the model’s performance and its ability to learn and identify meaningful patterns. for instance, when street maps are used to generate graph-based spatial discretization, the nodes of the graph can correspond to street segments, with edges connecting nodes whose street segments intersect. additionally, attributes can be assigned to nodes and edges of the graph. the challenge to make this assignment lies in integrating data from different modalities, discretizations, and resolutions. for instance, socioeconomic indicators, population education levels, and the number of points of interest (such as bars, bus stops, and schools) near each graph node or edge are typically discretized into census tracts and considered static information. climate data is provided at a few specific locations and varies over time, while crime events are geolocated and typically dispersed throughout the spatial urban domain. properly integrating all these data sources into the nodes and edges of the graph is a problem for which no consolidated methodology currently exists. this work presents a data modeling methodology capable of integrating data from different modalities and distinct discretization domains into a graph structure derived from street maps. the proposed data modeling method has been employed to structure data used as input for two distinct spatiotemporal gnn models. specifically, we evaluate and compare the dynamic self-attention network (dysat) [26] and the evolving graph convolutional network (evolvegcn) [22] both adapted to operate in crime data. dysat employs a dynamic self-attention mechanism to model the temporal evolution of node representations, effectively capturing long-range dependencies in both space and time. evolvegcn, on the other hand, uses recurrent neural network (rnn) architectures to dynamically update the parameters of the gcn layers, thus adapting to changes in the graph structure over time. dysat and evolvegcn performance are compared against a baseline to better understand the advantages and limitations of the gnns in the context of crime prediction. our contributions. in summary, the main contributions of the present work are: 1. data modeling pipeline: we develop a robust data processing pipeline to integrate crime (we used data from são paulo’s department of public safety [27]) and urban infrastructure into a graph derived from a street map. this integration allows for a detailed and dynamic representation of the crime landscape in são paulo. the proposed pipeline is versatile enough to be applied to various domains and geographical areas, requiring only the data is geolocated. 2. gnns for crime data: we adapt two distinct spatiotemporal gnn architectures, dysat and evolvegcn, for crime prediction. to our knowledge, these models have not been previously applied to the context of crime prediction. 3. models evaluation and comparison: we thoroughly and comprehensively evaluate and compare dysat and evolvegcn using the processed são paulo crime data, assessing their effectiveness in learning from historical data and predicting future crime occurrences. this study explores the effectiveness of gnn architectures, specifically dysat and evolvegcn, in the crime domain by applying, evaluating, and comparing these models on crime data using our developed data pipeline. integrating street maps and crime data, combined with advanced spatiotemporal modeling techniques, provides a novel approach to understanding and predicting crime dynamics in urban environments. our findings offer valuable insights for researchers and practitioners working on crime prevention and public safety enhancement. 2 related work crime analytics has been an essential area of urban safety and policy-making study. traditional studies primarily focus on statistical methods and classical machine learning models. for instance, gorr and harries [9] employed time-series regression models to forecast crime rates, concentrating primarily on the temporal elements of crime data. these models, while valuable, often needed to account for the spatial dependencies that inherently exist in crime patterns. spatial analysis techniques, such as hotspot analysis and spatial clustering [6], have been used to identify areas with high crime rates. these methods highlight the geographic concentration of crimes but typically lack the temporal dimension essential for dynamic crime prediction. recent advancements in spatio-temporal data modeling have enabled the development of sophisticated and complex analytical tools for analyzing complex phenomena [8]. for instance, salinas et al. [25] proposed a city hub framework that integrates data from different sources on the nodes of a street map graph, aiming mainly to investigate crime-related phenomena. although their goal was more on visualizing and analyzing crime events, they leveraged crime prediction models to support the analysis. to analyze human and drug trafficking crimes ahmed et al. [2] used entity resolution techniques to merge multiple state-wide and county-wide crime datasets into a geographic graph, combining incident reports, crime reports, and court records, in a single dataset. however, their data do not account for urban-related variables, which are fundamental for crime analysis. the work by salinas et al. [25] and ahmed et al. [2] are examples of methods designed to integrate data into a graph structure. however, as far as we know, there is no consolidated methodology to perform such an integration. deep learning models such as convolutional neural networks [21] and recurrent neural networks [19] have been adapted for handling spatio-temporal data, with noteworthy success. shi at al. [28], for instance, demonstrated the effectiveness of cnns for spatial feature extraction and rnns for capturing temporal dependencies in video data. despite these advancements, these models frequently struggle with irregular and non-euclidean data structures, which are natural real-world representations, particularly for urban analytics [4]. graph neural networks (gnns) have emerged as robust tools for modeling complex graph-structured data. the work by kipf and welling [15] on graph convolutional networks (gcns) indicated that gnns can leverage connectivity information present in graph data to improve predictive performance. however, static gnns are limited in their ability to model temporal dynamics, which is critical for many real-world applications, including crime prediction. dynamic gnns extend static gnns by integrating temporal information, making them suitable for spatio-temporal prediction tasks. for instance, the dynamic self-attention network (dysat) [26] employs a self-attention mechanism to capture temporal dependencies within dynamic graphs. similarly, evolving graph convolutional networks (evolvegcn) [22] use recurrent neural networks to update node embeddings over time, effectively capturing the evolution of graph structures. these dynamic gnns have shown substantial advancements in various applications, such as traffic prediction, social network analysis, and financial forecasting, effectively modeling the temporal evolution of graph data. in the context of crimes, han et al. [10] developed a crime prediction framework using graph convolutional networks to model the spatial dependencies between different regions and long short-term memory (lstm) networks to capture temporal patterns. jin et al. [13] introduced a spatio-temporal graph convolutional network (st-gcn) that combines gcns with recurrent neural networks to predict crime hotspots, learning both spatial and temporal features from crime data. the discussion above has only the intent of contextualizing our contribution. a more comprehensive discussion about the use of machine and deep learning for crime forecasting can be found in several surveys bout the theme [5, 18, 29]. 3 data modeling for this study, we utilized crime data from the são paulo department of public safety [27], which includes detailed information on crime occurrences with latitude and longitude coordinates. to complement this data, we extracted street network data using the osmnx python package [3]. the street network data is used to construct graphs representing streets as nodes and their connections as edges. each node is labeled to indicate the presence or absence of a crime, making the labels dynamic as they evolve over time. the crime and street network data are integrated to form a spatiotemporal dataset. this integration is crucial for creating the dynamic graph structures necessary for our models. the resulting dataset consists of monthly snapshots of the graph spanning two years, resulting in 24 graphs in total. in this work, we have used months as a time duration, but our data modeling pipeline can also be used for days and years, depending on the analysis. we decided to use monthly intervals as a temporal range to reduce the sparsity effect in the data. if processed daily, the ratio of occurrences to non-occurrences would be around 1/100, whereas with monthly processing, it is around 1/5. in more details, the graph construction process involved the following steps: 1. downloading street graph from given region: we use osmnx [3] to download street graphs from openstreetmaps. our code allows us to input two types of parameters at this stage: a string containing the location from which we want to fetch the graph, or a polygon formed by a list of latitude/longitude coordinates defining the region of interest. in this study, we restricted our graph to the central region of são paulo. 2. processing street graph: in step 1, we acquire a directed graph where streets are depicted as edges and intersections as vertices, encompassing diverse transportation networks. in this step, we unify all types of transportation networks and invert the roles of vertices and edges: street segments are redefined as vertices, while intersections become edges, resulting in a undirected graph. 3. extracting spatial features: for each vertex, we associate a feature vector containing spatial information. these features are static and sourced from the public geosampa website [24], which provides tools for extracting various data about the city. the features selected for testing include points of interest, such as the number of health, cultural, security, education, and social assistance services located within a predetermined distance from a node where a crime occurred. 4. projecting crime data: crime data is extracted from são paulo’s department of public safety [27]. each occurrence is associated with a latitude and longitude coordinate, which is then projected onto the respective street vertex. crimes with coordinates outside the region define in step 1 are removed. the complete data processing pipeline can be found in our github repositoryfootnote 1. the final result of our data processing pipeline is illustrated in fig. 1. figure 1 represents data from the month immediately following the training period. the street segments with green lines represent the non-crime nodes (streets), and the red lines represent crime nodes. fig. 1. street map graph for the central region of são paulo. crime nodes are marked in red, and non-crime nodes are marked in green. (color figure online) full size image 4 models this paper utilizes two distinct graph neural networks (gnns) architectures for crime prediction: dysat [26] and evolvegcn [22]. the choice of dysat and evolvegcn is motivated by their ability to handle dynamic graphs effectively [16], making them eligible for our analysis. 4.1 dysat dynamic self-attention network (dysat) [26] is an unsupervised graph embedding model that learns a latent node representation to capture dynamic graph representation. it computes node representations through self-attention along two dimensions (layers): structural neighborhood and temporal dynamics. the structural attention layer extracts features from local node neighborhoods in each snapshot through self-attentional aggregation. the temporal attention layer captures the temporal variations in graph structure over multiple time steps. dysat leverages the self-attention mechanism to concentrate on the most appropriate parts of the graph when updating node representations. this permits the model to effectively consider the importance of different nodes and edges, improving its ability to learn complex patterns. dysat jointly learns both spatial (structural) and temporal patterns. by integrating these two types of information, it constructs more accurate and robust node embeddings. the architecture of the model is as follows: input layer: the input to the dysat model consists of snapshots of the dynamic graph over all time steps. each snapshot describes the graph at a particular time step, with nodes and edges having associated features. spatial self-attention: each snapshot undergoes a spatial self-attention layer to capture structural associations, estimating attention scores between nodes to aggregate information from relevant neighbors. temporal self-attention: spatial embeddings are then processed through a temporal self-attention layer to learn temporal dependencies and how node representations evolve over time. output layer: final node embeddings are obtained by integrating spatial and temporal embeddings, used for tasks like node classification, link prediction, and anomaly detection. in this study, we used them for node classification. 4.2 evolvegcn the main goal of evolving graph convolutional networks (evolvegcn) [22] is to adapt gcns, originally designed for static graphs, to effectively handle graphs that evolve continuously. this adaptation is crucial for real-world applications such as social networks, recommendation systems, and communication networks, where relationships between entities evolve over time. evolvegcn achieves this by introducing an evolutionary approach to update gcn parameters as the graph evolves. it treats these parameters as temporal sequences and utilizes time-series models to predict and adjust them over time. the method focuses on two primary models for parameter evolution: hidden model: this model employs lstm networks to capture temporal dependencies and evolve gcn parameters based on historical states. output model: in contrast, this model uses direct rnn architectures to update parameters based on output states from the previous layers. in this work, we make use of the hidden version. the training of evolvegcn involves optimizing not only the base gcn’s parameters but also the evolution model’s (lstm or rnn) parameters. at each new time step, the gcn’s weights are updated using the selected evolution model, ensuring that the network adapts to the graph’s changing dynamics. 4.3 baseline models we compare the gnn models against three traditional non-deep learning models: support vector machine (svm), random forest and logistic regression. this analysis aims to highlight the predictive capabilities of gnns in contrast to these established machine learning approaches. by evaluating the performance of these traditional models, we provide a benchmark to better understand the advantages and limitations of gnns in the context of crime prediction. 5 experimental setup we conducted experiments on a real-world crime records dataset collected in sao paulo from jan. 1st, 2021, to dec. 31st, 2022. we integrated the crime data with street map data and transformed them into dynamic graphs; the transformation details are already discussed in sect. 3. dynamic graphs are a series of static graph snapshots, \(g = \{g^1, ..., g^t\}\) where t represents the number of time steps. in this work, each snapshot represents the crime data for a month, therefore, in total, we have 24 months of graph snapshots. we generated 22 instances from 24 snapshots using a sliding window approach, with each instance comprising three consecutive months (the first two for training and the last for evaluation). these were split 80-20 ratio into 17 for train/validation and 5 for testing, with the train/validation set further divided into 12 for training and 5 for validation. this method captures temporal dependencies while ensuring unbiased evaluation. each graph has 4021 nodes and 19899 edges. the graph statistics are described in table 1. table 1. crime dataset summary.full size table 5.1 models hyperparameters we conducted several experiments varying the hyperparameters and selected the most advantageous trade-off among the metrics used. ultimately, we chose the hyperparameters based on the stability of the performance in the training and validation sets, which led to a significant improvement. dysat. in our dysat model experiments, we carefully selected hyperparameters to balance complexity, efficiency, and overfitting. to address the dataset’s imbalance, we used classweights with a 1:4 ratio for majority and minority classes. we set the learning rate at 0.01 and applied weight decay of 0.0005 to regularize the model. dropout rates of 0.4 and 0.5 were used for spatial and temporal attention to enhance robustness. the model’s architecture included 16 attention heads and 128 units in spatial and temporal layers, enabling it to capture complex relationships while maintaining computational efficiency. table 2. models hyperparameters.full size table evolvegcn. three primary hyperparameters in evolvegcn significantly impact its performance. the first, classweights, assigns weights to classes, with the second value weighting the target class (class 1) and the first value weighting class 0. the second, numberofsteps, defines the number of temporal layers (e.g., months) the model considers for predictions. the third, featurespernodes, determines the input and output dimensions between network layers, adjusting the state vector’s size during processing. table 2 provides the selected hyperparameter values. baselines. we utilize the default parameters provided by the scikit-learn python library [23] for all the baseline methods. however, due to the dataset’s imbalanced nature, we employ the classweight parameter set to ’balanced’ across all these methods. this adjustment ensures more accurate results by addressing the class imbalance that would otherwise degrade the model’s performance. we have a list of features for each time step, with each element representing a dataframe of node features at a specific time step. to prepare the data for traditional baseline methods, we employed a sliding window approach with a window size of 3. in this method, we selected three consecutive dataframes, concatenated the first two for training, and used the third for testing, averaging the results across all sliding windows. 6 evaluation we used precision, recall, and f1-score to compare the models’ performance. these metrics are calculated for both the positive class (crime occurrences) and the negative class (no crime occurrences). 6.1 metrics performance in both deep learning models, as illustrated in fig. 2, the training and validation loss curves are closely aligned and consistently decrease with the increase in epochs. the close alignment of these curves indicates that the models generalize well to unseen data, providing strong evidence that overfitting is not occurring. this observation is further affirmed by comparing validation and test metrics shown in fig. 3. however, during the training of the evolvegcn model, an unusual spike occurs around epoch 800. we hypothesize that this may be due to the model encountering a local valley in the loss landscape. further investigation is required to confirm this hypothesis. fig. 2. training and validation loss over epochs of gnns models. full size image dysat. in the context of highly imbalanced data, achieving high f1 scores is particularly challenging but essential for effective model performance. in fig. 3a, the increasing f1 scores for both validation and test sets indicate that the dysat model is adept at handling the imbalanced dataset. an f1 score of 0.57 on the validation set demonstrates that the model reasonably balances precision and recall. it signifies that dysat can correctly identify many positive instances while minimizing false positives and negatives. similarly, an f1 score of around 0.55 on the test set indicates that the model generalizes well to unseen data. additionally, table 3 reaffirms dysat’s superior performance compared to other methods. the dysat model’s ability to manage the complexities of imbalanced data and maintain consistent performance across validation and test sets highlights its potential for real-world applications, particularly in areas where accurate minority class predictions are critical. fig. 3. f1 scores over epochs of gnn’s model full size image evolvegcn. despite the evolvegcn model’s metrics not ranking among the highest, when we plot the model’s results on the map for hotspot analysis, we observe an interesting outcome. as illustrated in fig. 4, the reason for the metrics not being so high is that the model tends to favor classifying safe (blue) and unsafe (red) areas only when there is strong evidence to do so, which explains the large number of uncertain areas (blue) and the higher recall compared to all other models. this aspect of the model has its advantages and disadvantages. for instance, while this cautious approach may lead to more reliable identifications of true hotspots, it can also result in overlooking subtle but important patterns or changes in the data, thereby affecting the model’s overall precision and specificity. table 3. performance metrics.full size table fig. 4. model predictions projected onto street graph. (color figure online) full size image baselines. when it comes to traditional machine learning models, random forest achieved results comparable to the dysat model for class 1 metrics, while logistic regression performed similarly to the evolvegcn model, as shown in table 3. in contrast, the svm model demonstrated notably poor performance. despite the metrics being similar to those of graph-based models, there is a crucial caveat to consider: traditional models do not take into account the temporal dynamics of our data. consequently, they fail to generalize their predictions over time. this limitation becomes particularly evident when plotting predictions on a map for hotspot construction, as we show in fig. 4. in these figures, green represents “safe” zones, red indicates “dangerous” zones, and blue signifies model uncertainty. notably, the gnn models have distinct safe and dangerous regions, which we will discuss in the next section. however, in traditional models, these regions are less clear. this is because the models are often uncertain about their predictions (as seen with svm, logistic regression and even random forest have more blue lines) or because they make highly heterogeneous predictions across the region of interest. 6.2 hotspot analysis as shown in fig. 5, predictions produced by gnn methods, which consider the temporal dynamics of the events under study, can pinpoint several notable regions on the map, marked a, b, c, d, e, and f. these regions are significant as they align with known areas of high or low crime rates in central são paulo, as shown in fig. 1. a detailed description of each region is provided below: a. encompasses the entire república neighborhood in são paulo, which is widely known for its high risk of robbery and theft. b. located just above the república neighborhood, is identified by our model as a safer area. examination of this region reveals the presence of several banks, including bradesco, santander, and itaú, in the nearby blocks. c. includes the surroundings of the estação da luz in são paulo, a well-known high-traffic area with a reputation for being dangerous. d. covers the vicinity of largo da concórdia square in são paulo, near a lojão do brás. this area, where most transportation is on foot, experiences a high incidence of pickpocketing, leading to many cell phone thefts. e. encompasses regions a and is very close to region c. f. encompasses region d. fig. 5. noteworthy predicted regions. (color figure online) full size image in our hotspot maps, shown in figs. 4 and 5, we used soft predictions for both baseline and gnn models to better represent prediction confidence. probabilities were mapped into three regions: non-crime (green) for probabilities near 0, uncertain (blue) for probabilities around 0.5, and crime (red) for probabilities near 1. this mapping effectively highlights areas of high and low confidence. when comparing traditional machine learning models to graph neural networks (gnns), the latter demonstrates significant advantages in capturing the complex spatial and temporal dependencies inherent in crime data. among the traditional models tested, logistic regression, random forest, and svm, random forest achieved the highest f1-score, slightly surpassing the gnn models, particularly dysat. however, gnns practical application reveals its superiority. when visualizing predictions on a street map, dysat effectively identifies crime hotspots with clear regions of red (crime nodes) and green (no crime nodes) and minimal uncertainty (blue lines) as shown in fig. 5. this contrasts with random forest, which shows more uncertainty and less distinct regions. thus, despite a some slightly lower scores, gnns practical performance in hotspot identification demonstrates its superior effectiveness over traditional models. while somewhat subjective, this approach enhances our understanding of spatial crime patterns. 7 conclusion this study demonstrates the potential of graph neural networks (gnns), specifically dysat and evolvegcn, in improving crime prediction accuracy by effectively modeling crime data’s dynamic and interconnected nature. the developed data modeling pipeline, integrating crime data with street map graphs, enables a detailed representation of crime dynamics in são paulo. our evaluation and comparison of dysat and evolvegcn reveal their effectiveness in learning from historical data and predicting future crime occurrences. these findings highlight the need for further exploration and adoption of gnns in the crime prediction domain, offering valuable insights for enhancing public safety strategies and crime prevention efforts. notes 1.https://github.com/giva-lab/bracis2024_gnn.git. references ahishakiye, e., taremwa, d., omulo, e.o., niyonzima, i.: crime prediction using decision tree (j48) classification algorithm. int. j. comput. inf. technol. 6(3), 188–195 (2017) google scholar  ahmed, s., gentili, m., sierra-sosa, d., elmaghraby, a.s.: multi-layer data integration technique for combining heterogeneous crime data. inf. process. manag. 59(3), 102879 (2022) article  math  google scholar  boeing, g.: modeling and analyzing urban networks and amenities with osmnx (2024). https://geoffboeing.com/share/osmnx-paper.pdf casali, y., aydin, n.y., comes, t.: machine learning for spatial analyses in urban areas: a scoping review. sustain. urban areas 85, 104050 (2022) math  google scholar  dakalbab, f., talib, m.a., waraga, o.a., nassif, a.b., abbas, s., nasir, q.: artificial intelligence & crime prediction: a systematic literature review. soc. sci. humanit. open 6(1), 100342 (2022) google scholar  eck, j., chainey, s., cameron, j., wilson, r.: mapping crime: understanding hotspots (2005) google scholar  fan, w., et al.: graph neural networks for social recommendation. in: the world wide web conference, www 2019, pp. 417–426. association for computing machinery, new york (2019). https://doi.org/10.1145/3308558.3313488 garcía-zanabria, g., et al.: cripav: street-level crime patterns analysis and visualization. ieee trans. vis. comput. graph. 28(12), 4000–4015 (2022). https://doi.org/10.1109/tvcg.2021.3111146 gorr, w., harries, r.: introduction to crime forecasting. int. j. forecast. 19(4), 551–555 (2003). https://doi.org/10.1016/s0169-2070(03)00089-x article  math  google scholar  han, x., hu, x., wu, h., shen, b., wu, j.: risk prediction of theft crimes in urban communities: an integrated model of lstm and st-gcn. ieee access 8, 217222–217230 (2020). https://doi.org/10.1109/access.2020.3041924 article  google scholar  jenga, k., catal, c., kar, g.: machine learning in crime prediction. j. ambient. intell. humaniz. comput. 14(3), 2887–2913 (2023) article  math  google scholar  jiang, w., luo, j.: graph neural network for traffic forecasting: a survey. expert syst. appl. 207, 117921 (2022). https://www.sciencedirect.com/science/article/pii/s0957417422011654 jin, g., wang, q., zhu, c., feng, y., huang, j., zhou, j.: addressing crime situation forecasting task with temporal graph convolutional neural network approach. in: 2020 12th international conference on measuring technology and mechatronics automation (icmtma), pp. 474–478 (2020). https://doi.org/10.1109/icmtma50254.2020.00108 jin, g., wang, q., zhu, c., feng, y., huang, j., zhou, j.: addressing crime situation forecasting task with temporal graph convolutional neural network approach. in: 2020 12th international conference on measuring technology and mechatronics automation (icmtma), pp. 474–478 (2020) google scholar  kipf, t.n., welling, m.: semi-supervised classification with graph convolutional networks. arxiv preprint arxiv:1609.02907 (2016) longa, a., et al.: graph neural networks for temporal graphs: state of the art, open challenges, and opportunities (2023) google scholar  mandalapu, v., elluri, l., vyas, p., roy, n.: crime prediction using machine learning and deep learning: a systematic review and future directions. ieee access 11, 60153–60170 (2023). https://doi.org/10.1109/access.2023.3286344 article  google scholar  mandalapu, v., elluri, l., vyas, p., roy, n.: crime prediction using machine learning and deep learning: a systematic review and future directions. ieee access 11, 60153–60170 (2023) article  google scholar  medsker, l.r., jain, l., et al.: recurrent neural networks. des. appl. 5(64–67), 2 (2001) math  google scholar  oh, g., song, j., park, h., na, c.: evaluation of random forest in crime prediction: comparing three-layered random forest and logistic regression. deviant behav. 43(9), 1036–1049 (2022) article  math  google scholar  o’shea, k., nash, r.: an introduction to convolutional neural networks. arxiv preprint arxiv:1511.08458 (2015) pareja, a., et al.: evolvegcn: evolving graph convolutional networks for dynamic graphs. arxiv abs/1902.10191 (2019). https://api.semanticscholar.org/corpusid:67856459 pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  geosampa. https://geosampa.prefeitura.sp.gov.br. accessed 20 june 2024 salinas, k., gonçalves, t., barella, v., vieira, t., nonato, l.g.: cityhub: a library for urban data integration. in: 2022 35th sibgrapi conference on graphics, patterns and images (sibgrapi), vol. 1, pp. 43–48. ieee (2022) google scholar  sankar, a., wu, y., gou, l., zhang, w., yang, h.: dysat: deep neural representation learning on dynamic graphs via self-attention networks. in: proceedings of the 13th international conference on web search and data mining, wsdm 2020, pp 519–527. association for computing machinery, new york (2020). https://doi.org/10.1145/3336191.3371845 secretaria de segurança pública. https://www.ssp.sp.gov.br/estatistica/consultas. accessed 20 june 2024 shi, x., chen, z., wang, h., yeung, d.y., wong, w.k., woo, w.c.: convolutional lstm network: a machine learning approach for precipitation nowcasting. in: cortes, c., lawrence, n., lee, d., sugiyama, m., garnett, r. (eds.) advances in neural information processing systems, vol. 28. curran associates, inc. (2015). https://proceedings.neurips.cc/paper_files/paper/2015/file/07563a3fe3bbe7e3ba84431ad9d055af-paper.pdf thomas, a., sobhana, n.: a survey on crime analysis and prediction. mater. today proc. 58, 310–315 (2022) article  math  google scholar  wu, s., sun, f., zhang, w., xie, x., cui, b.: graph neural networks in recommender systems: a survey. acm comput. surv. 55(5) (2022). https://doi.org/10.1145/3535101 zhou, j., et al.: graph neural networks: a review of methods and applications. ai open 1, 57–81 (2020). https://doi.org/10.1016/j.aiopen.2021.01.001 article  math  google scholar  download references acknowledgments this work was supported by cnpq (#307184/2021-8) and fapesp (#2022/09091-8, #2023/16334-7, #2013/07375-0, #2023/15618-1). the opinions, hypotheses, conclusions and recommendations expressed in this material are responsibility of the author(s) and do not necessarily reflect cnpq and fapesp views. author information authors and affiliations university of são paulo, são carlos, brazil waqar hassan, marvin mendes cabral, antonio castelo filho & luis gustavo nonato federal university of são carlos, são carlos, brazil thiago rodrigo ramos authors waqar hassanview author publications search author on:pubmed google scholar marvin mendes cabralview author publications search author on:pubmed google scholar thiago rodrigo ramosview author publications search author on:pubmed google scholar antonio castelo filhoview author publications search author on:pubmed google scholar luis gustavo nonatoview author publications search author on:pubmed google scholar corresponding author correspondence to waqar hassan . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper hassan, w., cabral, m.m., ramos, t.r., filho, a.c., nonato, l.g. (2025). modeling and predicting crimes in the city of são paulo using graph neural networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_26 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords data modeling graph neural networks crime prediction publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature development of a deep learning model for the classification of mosquito larvae images | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper development of a deep learning model for the classification of mosquito larvae images conference paper first online: 12 october 2023 pp 129–145 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) development of a deep learning model for the classification of mosquito larvae images download book pdf download book epub ramon mayor martins9, bruno manarin espíndola9, pedro philippi araujo9, christiane gresse von wangenheim9, carlos josé de carvalho pinto10 & … gisele caminha11  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 533 accesses 4 citations 1 altmetric abstract dengue is a disease that is endemic to certain regions, and in 2022, it was responsible for more than three million cases in the americas. one of the most effective ways to prevent dengue is by preventing the formation of breeding sites for the aedes aegypti mosquito, which is the primary vector of the disease. unfortunately, identifying these breeding sites remains a challenge as citizens lack knowledge to distinguish ae. aegypti larvae from other species. a solution can be the development of a deep learning model, to be deployed in a mobile application that classifies mosquito species using photos of larvae. currently only a few models are available that mostly differentiate only between genera (aedes versus non-aedes), or present very low accuracy. therefore, the objective of this research is to develop an image classification model that can differentiate between ae. aegypti, aedes albopictus, and culex sp. larvae using pictures taken with a cellphone camera by comparing various deep learning models (mobilenetv2, resnet18, resnet34, efficientnet_b0 and efficientnet_lite0). best results were obtained with efficientnet_lite0 with an accuracy of 97.5% during validation and 90% during testing, an acceptable result considering the risks related to a misclassification in this context. these results demonstrate the viability of a classification of mosquito larvae differentiating even between aedes species and thus providing a contribution to the prevention of dengue. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others implementation of a deep learning model for automated classification of aedes aegypti (linnaeus) and aedes albopictus (skuse) in real time article open access 10 may 2021 mosquito larvae image classification based on densenet and guided grad-cam chapter © 2019 identifying aedes aegypti mosquitoes by sensors and one-class classifiers chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization dengue virus entomology machine learning model invertebrates west nile virus 1 introduction dengue is an acute febrile disease caused by a virus, which is transmitted mainly by the aedes aegypti, a mosquito that also transmits chikungunya fever and the zika virus [20]. it is an endemic disease in the americas, africa, asia and oceania. the world health organization estimates that, annually, between 100 and 400 million cases occur each year, with 500,000 to 1 million developing the severe form, which can be fatal [33]. in brazil, dengue is a public health problem. according to the [24], approximately 1.5 million probable cases were registered in brazil in 2022, an increase of 206% compared to 2021. this shows the importance of disease prevention for the well-being of the population. one of the most effective ways to prevent dengue is to eliminate breeding sites for the aedes aegypti mosquito, the primary vector of the disease. larvae foci can be found commonly in artificial deposits and containers with stagnant water (e.g., flower pots or tires), where they allow the development of the mosquito larvae. therefore, an important actor in controlling this disease control is the population, by preventing the formation of breeding sites in their homes. unfortunately, identifying these breeding sites remains a challenge, as the general population lacks sufficient knowledge to distinguish ae. aegypti larvae from other species, such as aedes albopictus or culex sp. also to be found in urban regions in santa catarina [12]. mosquitoes is a holometabolous insect which goes through the complete metamorphosis process from eggs, larva, pupa, to the adult stage. the most favorable stage for collecting mosquito samples may be the larval stage, since it is contained in a place with water. morphological characteristics of the larva, color, bristles, length of breathing tube, its positioning in the water are factors taken into consideration for the characterization of the insect, such as the differentiation of culex and aedes mosquitoes. usually the classification of mosquito larvae is based on a microscopic analysis by biologists [14]. an alternative solution in order to enable citizens to identify the presence of ae. aegypti larvae in their homes could be to develop a deep learning (dl) model for the automatic classification of mosquito larvaes deployed in a mobile application that enables the classification using a photograph of a mosquito larva found by the user taken with the cellphone camera. this would make the diagnosis quick and accessible, not requiring specialist knowledge of mosquito morphology. there is already research on the development of dl models aiming at the classification of mosquito larvae images. yet, most of the existing models do not classify between aedes species, thus omitting important information when to be used for dengue prevention. the only model developed to distinguish between aedes species [5] reported a very low accuracy of only 64.58% that is inappropriate considering the risk of misclassification. another limitation of existing research is the use of either microscopic images or pictures taken with a special zoom lens [17, 25], not available to a broader audience. therefore, aiming at the development of a dl model to be deployed in a mobile application to be used by citizens in the urban region of florianopolis/sc, we analyze the following research question: is it possible to develop a dl model to classify larvae of ae. aegypti, ae. albopictus and culex sp. mosquitoes based on photos taken with a cellphone camera with an accuracy of at least 90%, considering the risk related to erroneous classification in the context of an intelligent mobile application to be used by a wide target audience. 2 background: morphology of mosquito larvae there are fundamental differences in the morphology of mosquito larvae between different genera and species as shown in fig. 1. mosquito larvae have three body regions: head, thorax, and abdomen. the head of mosquito larvae is large and sclerotized (composed of a hardened exoskeleton). the shape of the head of aedes and culex larvae is broad. the head has two eyes, two antennae, and a brush-like or comb-like mouth apparatus. the eyes are usually small, simple (not compound) and are found on both sides of the head [8]. the head of the aedes is generally shorter and wider than that of the culex. aedes antennae have small scales on the surface, while culex antennae are smooth. aedes jaws are wider and shorter than culex jaws. in general, the thorax of larvae is wider than the head and has three rows of bristles. the thorax of the aedes has white spots. culex's thorax has no distinctive spots. in addition, the bristles on the surface of the thorax of aedes are longer and more sparse than those of culex. commonly, the abdomen is elongated, composed of ten segments, and its eighth segment has the respiratory siphon that is used to distinguish between species [7, 8]. the abdomen of the aedes is shorter and wider than that of the culex, while the scales on the abdomen of the aedes are rounder and darker than those of the culex. fig. 1. fundamental differences in the morphology of mosquito larvae: aedes and culex sp. [9, 14]. full size image the siphon in the abdomen allows the larvae to breathe oxygen from the air while remaining submerged in water. the siphon is one of the main characteristics used to identify mosquito species. its shape, size and length can vary greatly between species, as well as the presence or absence of a specialized structure called a “pecten”. the pecten is a row of spines located at the base of the siphon. there are some notable differences in the siphon between aedes and culex. the culex siphon is longer and narrower than the one of aedes [7, 8, 14, 32]. the genus aedes has two main disease vector species, ae. aegypti and ae. albopictus. some morphological characteristics distinguish the larvae of these species [7]. the head of the ae. aegypti is more rounded, while the head of the ae. albopictus is more elongated. there are also subtle differences in the bristles of the antennae. the bristles on the thorax of the ae. aegypti are longer, while in the ae. albopictus they are shorter. and on the abdomen, the bristles of the aedes albopictus are simpler and without many branches (fig. 2) [7, 8, 14, 32]. fig. 2. main differences in the head, thorax and abdomen of aedes species (ae. aegypti and ae. albopictus) and culex [8, 14]. full size image all these characteristics are commonly distinguishable only by biologists and specialists, typically with the aid of microscopes. 3 related work to summarize research adopting deep learning for the automatic classification of images of mosquito larvae during the last ten years, a systematic mapping following the procedure proposed by [26] was conducted. as a result a total of 12 research articles were found published mostly during the last five years (table 1). table 1. overview on research adopting dl for the classification of mosquito larvae images.full size table most of the research focuses on the classification of aedes vs. non-aedes mosquito larvae, omitting detailed information when using the model for dengue prevention, while few aim at distinguishing between aedes species. few also consider the culex species [5, 30], another vector of diseases (such as encephalitis, lymphatic filariasis and nile fever). focusing on the development of such a model to be used by citizens for dengue prevention, especially in the urban region of florianópolis/sc, it is imperative to consider all relevant species, including also a differentiation between ae. aegypti and ae. albopictus in order to help the users to take adequate actions to prevent proliferation. yet, so far no research in brazil with such a focus has been encountered. only [5] presents a research that addresses the primary mosquito species of concern, including ae. aegypti, ae. albopictus, anopheles, and culex sp. however, the results achieved were not significant, with a reported accuracy of only 64%. this low level of accuracy poses a significant risk for users and could put human lives in danger in the context of the usage scenario. another limitation regarding most existing research is that the dl models have been trained with microscopic images, different from photos that would be taken by a citizen with a cellphone camera. only [5, 17] and [25] use images taken in with a cellphone, yet using a special zoom lens, which again will not be available to a broader audience. the current research mostly used convolutional neural networks (cnns), including resnets as well as older models such as alexnet. no research with more recent cnns, such as efficientnets have been encountered, although high performances have been reported in other application domains comparing it with other models [31]. the performance of the trained models varies largely, with an accuracy ranging from 59.67% [30] to about 97% [13, 29], while most performance results are rather low especially considering the risk related to a misclassification in this context. the highest performances reported are also related to classifications distinguishing only aedes vs. non-aedes, but not among aedes species. although densenet achieved a high accuracy (97%), as reported by [13], it was not designed specifically for mobile applications, and generally has a higher number of parameters and computational complexity. the only research reporting performance with respect to the classification between aedes species [5] indicates a very low accuracy of only 64.58%. these results show that there is currently a lack of dl models for the automatic classification of mosquito larvae that are able to distinguish between aedes species based on pictures taken with a cellphone camera with minimal acceptable accuracy. 4 research methodology this research is characterized as applied research [16] by identifying and characterizing a need and aiming to contribute a practical solution adopting dl. based on the results of a previous systematic mapping as presented in sect. 3, we follow a systematic process for the human-centric interactive development of deep learning models [1, 22, 28]. as presented in table 2 this includes the analysis of requirements, data preparation and an experiment on training and evaluating different dl models, as well as testing the trained models with new unseen images. the evaluation results are compared between the different dl models as well as in comparison with models encountered in literature. table 2. overview on phases of the research methodology.full size table 5 development of the image classification model 5.1 requirements analysis adopting [23] requirements notation, the goal is to develop a dl model that learns from experience e with respect to some class of tasks t and with performance measure p, if its performance on tasks in t, measured by p, improves with experience e. here, task (t) is to classify mosquito larvae (single label) from a photo taken with a cellphone camera. experience (e) is a corpus of labeled images of mosquito larvae of the genus aedes including the species ae. aegypti, ae. albopictus, genus culex and a non-mosquito object. in terms of performance (p), considering the risk of misclassification in this context, an accuracy performance level greater than 90% is expected in order to ensure that the model can effectively identify mosquito larvae and contribute to dengue prevention efforts. this level of accuracy would provide a sufficient degree of confidence in the model’s ability to identify mosquito larvae correctly and minimize the risk of false positives or false negatives. 5.2 data preparation due to the unavailability of public datasets of cellphone photos of larvae of the relevant mosquito species, we collected a set of 1.999 images, including 748 images of ae. aegypti larvae, 464 images of ae. albopictus larvae, 447 images of culex sp. larvae and 340 images of non-mosquito objectsfootnote 1. the images were collected and labeled by researchers from the initiative computação na escola/incod/ine/ufsc in cooperation with the laboratory of hematozoan transmitters/ufsc and the coordination of the laboratory reference network/central public health laboratory of sc. the images were taken with cell phone cameras (samsung galaxy s10, xiaomi redmi note 9 and xiaomi mi 11 pro) without any additional lenses (table 3). table 3. camera specifications.full size table the images were collected with varying backgrounds, angles, and resolutions and saved in.jpg format (fig. 3). fig. 3. examples of images from the data set. full size image in order to assure data quality the following data quality characteristics were considered in accordance with ieee std 2801 [18] as presented in table 4. table 4. data quality characteristics.full size table the data set was divided into a training (79.2%), validation (19.8%), and test (2%) set. 5.3 model training, evaluation and comparison in accordance with literature five different dl architectures for image classification that are also indicated for deployment in mobile applications have been trained: mobilenetv2, resnet18, resnet34, efficientnet_b0, and efficientnet_lite0. mobilenetv2 is an architecture designed specifically for mobile device applications, with a focus on efficiency and low resource consumption [15]. resnets 18 and 34 were chosen because of their efficiency and being the least complex of the resnet architecture family considering a deployment in a mobile app [6]. efficientnet_b0 is the most basic version of the efficientnet family, and is designed to be efficient in terms of both computational resources and performance [31] and the lightest version, efficientnet_lite0, designed specifically for mobile device applications [21]. the models were developed in python using jupyter notebook/google colab. for the development we used the fast.ai library [10], an open source deep learning library that provides a high-level api to simplify the process of training and validating dl models as well as its support for transfer learning. in order to run efficientnet and mobilenet architectures we also used the timm library, which provides a collection of pre-trained sota (state-of-the-art) computer vision models [10]. the images were resized to 224 × 224 pixels and subjected to random data augmentation. an overview of the training parameters and results is shown in table 5. table 5. overview on training parameters and results.full size table in an exploratory analysis of the results of the models’ validation, efficientnet_lite0 stands out by showing the best overall performance in terms of accuracy, precision, recall, and f1-score. this result is particularly notable considering that this architecture was designed to be efficient in terms of computational resources according to [21]. the resnet18, resnet34, and efficientnet_b0 models performed almost similarly, with all metrics above 96%. although there are subtle differences. resnet34 had slightly higher accuracy, while efficientnet_b0 had slightly higher precision, recall, and f1-score. mobilenetv2 demonstrated the worst performance among the models, with all metrics significantly below the others. despite being designed to be fast and lightweight, the model compromises performance in exchange for efficiency. the results of the evaluation of each model are presented in table 6. table 6. results of evaluation metrics.full size table 6 prediction test following iso/iec 4213 [19] we performed a test with the trained models predicting the classification of previously unseen images. 6.1 test preparation test dataset. the test set is composed of a total of 40 images with the data quality characteristics as described in table 7. table 7. test data set characteristicsfull size table metrics. to evaluate the performance of the models, the following metrics were employed: accuracy, to evaluate overall performance; precision and recall to support an understanding how the model handles false positives and false negatives, and f1 score, the harmonic mean of the accuracy and recall results; specificity was measured for correctly identifying negative examples and minimizing false positives. execution environment. the test was run using jupyter notebooks in google colab. 6.2 test results the test results are presented in table 8. table 8. test results.full size table the results show that the resnet18 model obtained the highest accuracy, precision, recall, and f1 score during the test, while the mobilenetv2 model obtained the lowest overall performance. efficientnet_b0 and efficientnet_lite0 perform similarly to resnet34 in terms of accuracy, precision, and recall. due to the higher accuracy, resnet18 performs better in correctly classifying both positive and negative examples, minimizing classification errors, including false positives and false negatives. due to higher precision, resnet18 and efficientnet_lite0 perform better in correctly identifying positive examples and minimizing false positives. the specificity of 97% for the resnet and efficientnet models families indicates that they are very effective at identifying true negatives while avoiding false positives. this high specificity value suggests that these models can correctly discern the negative classes, reducing the misclassification rate and improving the reliability of the predictions. the lowest performance results were observed with regard to mobilenetv2, which indicates that the model has a lower performance in correctly classifying examples in general. despite these results, and also having the lowest specificity among the models, the specificity of 87%, still indicates that mobilenetv2 can handle the identification of true negatives and avoid false positives with some efficiency. comparing resnet18, resnet34 and efficientnet_b0 only small differences between precision and recall (1%) were observed, suggesting that the models have a balanced performance between avoiding false positives (high precision) and identifying true positives (high recall). this is also confirmed by the f1 score of > 90% of resnet18, resnet34 and efficienttnet_b0, efficientnet_lite0, which indicates a robust performance of the models for correct identification of examples (table 9). table 9. confusion matricesfull size table the confusion matrices indicate that all models have accurately classified the images of ae. aegypti. in our usage scenario, any misclassification would have severe consequences, as it could mistakenly lead the user to believe that the larva found in their home is not a primary vector for dengue. resnet18, resnet34, efficientnet_b0 and efficientnet_lite0 demonstrate a similar misclassification of ae. albopictus as ae. aegypti, still being a classification error, yet with less risk to the user. resnet34, efficientnet_b0 also misclassified in one case an image of a culex sp. as ae. albopictus. again, mobilenetv2 demonstrated the worst performance misclassifying ae. albopictus, culex sp. and even non-mosquito objects. however, even this model did not misclassify any of the images of ae. aegypti. 7 discussion in this research we studied the viability of classifying images of mosquito larvae (ae. aegypti, ae. albopictus, culex, and nonmosquito) taken with a cellphone camera comparing the performance of different dl models (mobilenetv2, resnet18, resnet34, efficientnet_b0 and efficientnet_lite0). with the exception of mobilenetv2, the models reached convergence quickly and showed satisfactory validation results, suggesting that in a few training epochs they were able to learn the main features relevant for classification. the efficientnet_lite0 model achieved convergence the fastest, while efficientnet_b0 obtained the best fit to the data. resnet18, resnet34, efficientnet_b0 and efficientnet_lite0 achieved similar accuracy during validation ranging from 96.17% to 97.58%. similar results were also achieved during testing on previously unseen data, during which resnet34, efficientnet_b0 and efficientnet_lite0 demonstrated an accuracy of 90% and resnet 92.5%. these results provide a first indication that these models were able to learn the generalization of the classification of this kind of image, achieving performance results above the required minimum of 90% in the context of our usage scenario. analyzing the confusion matrices it can also be observed that none of the models misclassified the images of ae. aegypti, which would be the worst case in our usage scenario as it could harm the user due to leading him/her to not taking actions by not recognizing the larva found as a potential dengue vector. during training and validation the efficientnet families converged and fitted the dataset better due to the optimization techniques for automatic model size adjustment by efficientnet_lite0, and the resolution scaling adjustment technique of efficientnet_b0. yet, during testing the resnet18 model performed best with regard to all metrics due to a combination of factors. and, although resnet34 has a deeper architecture, which may allow it to learn more complex representations of the data, the increased complexity may also increase the risk of overfitting and make generalization to new data more difficult. in addition, resnet34 may be more susceptible to performance degradation issues, such as decreased performance when adding additional layers, which may have affected its classification performance. the mobilenetv2 underperformed the other models due to its simpler architecture, with fewer layers and fewer parameters. this limits its ability to learn complex data representations, affecting classification performance. and, although these characteristics make it light and fast, they also result in lower performance. threats to validity. as any empirical study there are several threats to validity. concerning selection bias, we aimed at preparing a diverse and representative sample of images that covers the range of variability in the population being studied. we considered various cameras, backgrounds, angles, and lighting for external validity, but our dataset may not cover all contexts, potentially limiting generalizability. considering the size of our dataset containing 1.999 images we consider the risk of a sample size bias minimal especially when also compared to existing approaches reported in the literature. in order to prevent data preprocessing bias we only used standardized preprocessing techniques that are applicable to all models being compared. we also maintained consistent training parameters for comparability between models. with regard to evaluation metric bias, we used multiple standard evaluation metrics that cover different aspects of model performance and to report the results of all metrics used. 8 conclusion the results of our research demonstrate that it is possible to classify images of mosquito larvae even distinguishing between aedes species with sufficient performance of testing accuracy > 90% and without misclassifying images of ae. aegypti. our results are also much better than the ones reported by the only other study aiming at the differentiation between aedes species reporting an accuracy of only 64.58%. both of the the models (resnet18 and efficientnet_lite0), which demonstrated the best results during training/validation or testing, are also small architectures specifically indicated for the deployment in mobile applications, which thus will allow the implementation of such an application in order to allow the operationalization of the intended usage scenario of citizens using the automated classification for dengue prevention in their homes. notes 1.the dataset and notebook for the model are being prepared for availability. references amershi, s., et al.: software engineering for machine learning: a case study. in: proceedings of the 41st international conference on software engineering, montreal, canada (2019) google scholar  arista-jalife, a., et al.: deep learning employed in the recognition of the vector that spreads dengue, chikungunya and zika viruses. in: frontiers in artificial intelligence and applications, vol. 303, pp. 108–120. ios press, amsterdam (2018) google scholar  arista-jalife, a., et al.: aedes mosquito detection in its larval stage using deep neural networks. knowl.-based syst. 189, 104841 (2020) article  google scholar  asmai, s.a., et al.: mosquito larvae detection using deep learning. int. j. innov. technol. explor. eng. 8(12), 804–809 (2019) article  google scholar  azman, m.i.a.b.z., sarlan, a.b.: aedes larvae classification and detection (alcd) system by using deep learning. in: 2020 international conference on computational intelligence (icci), pp. 179–184 (2020) google scholar  canziani, a., paszke, a., culurciello, e. an analysis of deep neural network models for practical applications. arxiv preprint arxiv:1605.07678 (2016) clements, a.n.: the biology of mosquitoes. volume 2: sensory reception and behavior. cabi publishing (1999) google scholar  consoli, r., oliveira, r.l.: principais mosquitos de importância sanitária no brasil. editora fiocruz, rio de janeiro, brasil (1994) google scholar  de silva, w.d.m., jayalal, s.: dengue mosquito larvae identification using digital images. in: 2020 international research conference on smart computing and systems engineering (scse), pp. 31–36 (2020) google scholar  fast.ai (2023). https://www.fast.ai/ fuad, m., et al.: training of convolutional neural networks using transfer learning for aedes aegypti larvae. telkomnika (telecommun. comput. electron. control) 16, 1894–1900 (2018) article  google scholar  funasa. instruções para pessoal de combate ao vetor manual de normas técnicas (2001). https://bvsms.saude.gov.br/bvs/publicacoes/funasa/man_dengue.pdf garcía, z., et al.: mosquito larvae image classification based on densenet and guided grad-cam. in: morales, a., fierrez, j., sánchez, j.s., ribeiro, b. (eds.) ibpria 2019. lncs, vol. 11868, pp. 239–246. springer, cham (2019). https://doi.org/10.1007/978-3-030-31321-0_21 chapter  google scholar  globe. globe mission mosquito mapper (2022) google scholar  goh, y.h., lee, y.b., lum, k.y.: american sign language recognition based on mobilenetv2. adv. sci. technol 5(6), 481–488 (2020) google scholar  hedrick, t.e., bickman, l., rog, d.j.: applied research design: a practical guide. sage publications, thousand oaks (1993) book  google scholar  hossain, m.s., et al.: aedes larva detection using ensemble learning to prevent dengue endemic. biomedinformatics 2(3), 405–423 (2022) article  google scholar  ieee std 2801–2022. ieee recommended practice for the quality management of datasets for medical artificial intelligence. ieee (2022) google scholar  iso/iec 4213. information technology—artificial intelligence—assessment of machine learning classification performance. iso/iec (2022) google scholar  kularatne, s.a.: dengue fever. bmj (clin. res. ed.) 351, h4661 (2015) google scholar  liu, r.: higher accuracy on vision models with efficientnet-lite. https://blog.tensorflow.org/2020/03/higher-accuracy-on-vision-models-with-efficientnet-lite.html. accessed 27 apr 2023 mathewson, k.w.: a human-centered approach to interactive machine learning. arxiv:1905.06289v1 [cs.hc] (2019) mitchell, t.m.: machine learning. mcgraw-hill, new york (1997) google scholar  ministério da saúde. boletim epidemiológico (2023) google scholar  munoz, j., et al.: image recognition of disease-carrying insects: a system for combating infectious diseases using image classification techniques and citizen science. in: 51st hawaii international conference on system sciences (hicss), pp. 3594–3603 (2018) google scholar  petersen, k., et al.: systematic mapping studies in software engineering. in: proceedings of the 12th international conference on evaluation and assessment in software engineering, bari, italy, pp. 68–77 (2008) google scholar  rajasekhar, v., et al.: identifying anopheles/non-anopheles larvae with ai implications. in: nasa sees mosquito mapper virtual internship science fair, student research reports (2021). https://www.globe.gov/do-globe/research-resources/student-research-reports/-/projectdetail/10157/identifying-anopheles-non-anopheles-larvae-with-ai-implications ramos, g., meek, c., simard, p., suh, j., ghorashi, s.: interactive machine teaching: a human-centered approach to building machine-learned models. hum.-comput. interact. 35(5-6), 413–451 (2020). https://www.microsoft.com/en-us/research/uploads/prod/2020/05/interactive_machine_teaching__free_access_.pdf sanchez-ortiz, a., et al.: mosquito larva classification method based on convolutional neural networks. in: 2017 international conference on electronics, communications and computers, pp. 1–6 (2017) google scholar  surya, a., peral, d., vanloon, a., rajesh, a.: a mosquito is worth 16×16 larvae: evaluation of deep learning architectures for mosquito larvae classification. arxiv:2209.07718 (2022) tan, m., le, q.v.: efficientnet: rethinking model scaling for convolutional neural networks. in: proceedings of the 36th international conference on machine learning, pp. 6105–6114. pmlr (2019) google scholar  utep. university of texas at el paso, laboratory for environmental biology centennial museum team mosquito: larvae identification guide (2004). https://www.utep.edu/leb/mosquito/larvaeid.pdf world health organization: geographical expansion of cases of dengue and chikungunya beyond the historical areas of transmission in the region of the americas. in: disease outbreak news (2023). https://www.who.int/emergencies/disease-outbreak-news download references acknowledgments this work was supported by the cnpq (national council for scientific and technological development), a brazilian government entity focused on scientific and technological development. author information authors and affiliations department of informatics and statistics, federal university of santa catarina, florianópolis, sc, 88040-900, brazil ramon mayor martins, bruno manarin espíndola, pedro philippi araujo & christiane gresse von wangenheim department of microbiology, immunology and parasitology/center for biological sciences, federal university of santa catarina, florianópolis, sc, 88040-900, brazil carlos josé de carvalho pinto coordination of the laboratory reference network, santa catarina, florianópolis, sc, 88010-001, brazil gisele caminha authors ramon mayor martinsview author publications search author on:pubmed google scholar bruno manarin espíndolaview author publications search author on:pubmed google scholar pedro philippi araujoview author publications search author on:pubmed google scholar christiane gresse von wangenheimview author publications search author on:pubmed google scholar carlos josé de carvalho pintoview author publications search author on:pubmed google scholar gisele caminhaview author publications search author on:pubmed google scholar corresponding author correspondence to ramon mayor martins . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper martins, r.m., espíndola, b.m., araujo, p.p., von wangenheim, c.g., de carvalho pinto, c.j., caminha, g. (2023). development of a deep learning model for the classification of mosquito larvae images. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_9 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords aedes aegypti dengue deep learning image classification publish with us policies and ethics profiles ramon mayor martins view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating short text stream clustering on large e-commerce datasets | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating short text stream clustering on large e-commerce datasets conference paper first online: 30 january 2025 pp 245–259 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) evaluating short text stream clustering on large e-commerce datasets download book pdf download book epub cesar andrade  orcid: orcid.org/0009-0005-2218-66489,12, rita p. ribeiro  orcid: orcid.org/0000-0002-6852-80779,12 & joão gama  orcid: orcid.org/0000-0003-3357-119510,11,12  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 372 accesses 1 citation abstract latent dirichlet allocation (lda) is a fundamental method for clustering short text streams. however, when applied to large datasets, it often faces significant challenges, and its performance is typically evaluated in domain-specific datasets such as news and tweets. this study aims to fill this gap by evaluating the effectiveness of short text clustering methods in a large and diverse e-commerce dataset. we specifically investigate how well these clustering algorithms adapt to the complex dynamics and larger scale of e-commerce text streams, which differ from their usual application domains. our analysis focuses on the impact of high homogeneity scores on the reported normalized mutual information (nmi) values. we particularly examine whether these scores are inflated due to the prevalence of single-element clusters. to address potential biases in clustering evaluation, we propose using the akaike information criterion (aic) as an alternative metric to reduce the formation of single-element clusters and provide a more balanced measure of clustering performance. we present new insights for applying short text clustering methodologies in real-world situations, especially in sectors like e-commerce, where text data volumes and dynamics present unique challenges. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others dynamic clustering for short text stream based on dirichlet process article 26 july 2021 topic model with contextual outlier handling: a study on electronic invoice product descriptions chapter © 2023 short text processing for analyzing user portraits: a dynamic combination chapter © 2020 1 introduction text clustering has become an essential technique in natural language processing (nlp), enabling the discovery of hidden structures within large collections of unstructured text data. among various approaches, stream short text clustering, particularly methods based on latent dirichlet allocation (lda), has been widely researched due to its effectiveness in grouping similar texts and discovering topical patterns. while these methods have shown promising results in domains such as news feeds and social media platforms, their application to larger and more complex datasets remains less explored. the performance of traditional lda-based stream short text clustering methods is well-documented in scenarios involving relatively small datasets, with few exceptions [1]. however, these methods often struggle when scaled to larger datasets commonly found in industries like e-commerce. the dynamic and voluminous nature of e-commerce text data presents unique challenges not typically encountered in other domains such as news or social media. handling the vast number of product descriptions, user reviews, and query logs requires clustering algorithms that are robust, scalable, and adaptive to the evolving nature of data. a significant challenge in evaluating the performance of clustering methods is the reliance on metrics like normalized mutual information (nmi). while nmi is useful for measuring the statistical accuracy of clustering results, it does not fully capture the practical aspects of the clusters formed, particularly in the presence of single-element clusters, which can artificially inflate homogeneity scores [2, 3]. this limitation calls for a more comprehensive evaluation framework to better understand the practical utility of clustering outcomes. our study seeks to extend the application of these clustering methods to understand their scalability and adaptability better, particularly focusing on: 1. how these methods perform with extensive e-commerce datasets; 2. the clustering quality metrics, especially concerning homogeneity scores and single-element cluster formation; 3. the potential for alternative metrics like the akaike information criterion (aic) to provide more reliable evaluations of clustering outcomes. this paper is organized as follows. section 2 reviews related work on similarity-based and model-based stream clustering, detailing the online approaches. section 3 presents our proposal and details the dataset, the dataset preparation, the evaluated methods, and the experimental setup. section 4.2 reports the results. section 5 discusses various aspects of the experiment. finally, sect. 6 concludes the paper and suggests future research directions. 2 related work two prominent areas of study emerge in clustering short text streams: similarity-based and model-based approaches. similarity-based clustering relies on pairwise similarities, while model-based clustering relies on statistical models that define data distribution. 2.1 similarity-based clustering recent advances, incorporating pre-trained language models like bert [4] coupled with clustering algorithms such as hdbscan [5], have showcased as promising in text clustering tasks. bert generates contextual embeddings, allowing deeper context understanding and language structure modeling [6]. these embeddings serve as inputs for clustering algorithms. studies [7, 8] have demonstrated the superiority of combining bert embeddings and hdbscan for short text clustering over traditional methods. additionally, elinac [9] introduced an auto-encoder-based method for clustering electronic invoices. despite bert and transformers being a new trend, their use in short text stream clustering with lda methods brings an additional computational cost [10], posing scale issues, particularly for large datasets. 2.2 model-based clustering approaches to model-based clustering of short text streams fall into two categories: batch methods and online methods. batch methods handle data in separate blocks, whereas online methods continuously process new incoming data. batch approaches. latent dirichlet allocation (lda) [11] is a model-based stream clustering technique that has inspired numerous extensions. these extensions address challenges related to topic evolution, semantic representation, and the dynamic nature of text streams. initially, models like dct [12] aimed to simplify topic assignments by assigning a single topic to each document. while effective for short texts, this approach lacked adaptability to handle changing topic counts within streams. mstream [13] emerged in response to this challenge, managing topic counts by discarding outdated batch documents and adapting to evolving topics. however, mstream’s reliance on single-term document representation limited its ability to navigate semantic spaces, impacting cluster purity. to enhance semantic representation, npmm [14] utilized pre-learned word embeddings, advancing text semantics understanding. however, these embeddings’ static, language-dependent nature constrained their adaptability to evolving semantic landscapes. building on npmm’s limitations, dp-bmm [15] introduced unordered sequences of bi-terms for semantic representation, dynamically detecting topic counts. nonetheless, this emphasis on semantic representation did not directly address challenges posed by sparse, high-dimensional data. the introduction of dcss [16] and faststream [17] marked significant milestones. dcss, utilizing the dirichlet process, demonstrated superior stability and adaptability by autonomously learning topic counts and adjusting to topic drift. faststream introduced efficiency and adaptability improvements, employing a novel cluster indexing mechanism and dynamic similarity thresholds to expedite processing. rakib et al. [18] proposed an iterative classification method, enhancing short text clustering. however, a common limitation among these studies is that they are not focused on complex, large datasets, especially in e-commerce. online approaches. the online semantic-enhanced dirichlet model (osdm) [19] excels in dynamic clustering, integrating word-occurrence semantic data into a graphical model for real-time text processing without fixed cluster numbers or batch sizes. tested on the news, reuters, and tweets datasets and their synthetic versions, osdm uses metrics like purity, v-measure, homogeneity, and normalized mutual information (nmi) for evaluation, adding accuracy to its metric suite for a thorough analysis of performance. following in the footsteps of osdm, the online semantic-enhanced graphical model (osgm) [20] also employs a poly urn scheme to dynamically manage document clustering, focusing on semantic smoothing and term co-occurrence for enhanced clarity and ambiguity resolution. osgm’s evaluation mirrors osdm’s, utilizing the same datasets and metrics to ensure consistency in performance assessment and comparability between methods. the introduction of eindm [21], a context-enhanced dirichlet model for online clustering in short text streams, adds another layer to the field. by using a window-based semantic term representation and new clustering metrics like word specificity, eindm effectively captures the evolving nature of text data. tested on datasets similar to osdm and osgm, eindm uses the same metrics except for accuracy, focusing on core aspects of clustering effectiveness. estream [22] merges online and offline clustering techniques to offer a versatile solution to text stream clustering. unlike the other methods, estream expands its testing grounds to include additional datasets like nt and so-t, along with news-t and tweets-t. this approach allows for a broader validation of its effectiveness, using the same core metrics to ensure a comprehensive assessment of its performance across various text stream scenarios. while these methods in clustering short text streams show innovation and effectiveness, particularly when evaluated with news and tweets datasets. however, the focus on these limited domains raises questions about their adaptability and performance in broader contexts, such as larger datasets or specific fields like e-commerce. exploring varied datasets could provide deeper insights into the scalability and applicability of these clustering techniques across diverse real-world scenarios. the evaluation of clustering methods often relies on normalized mutual information (nmi). however, nmi alone does not account for qualitative factors like the formation of very small or single-element clusters. a more nuanced evaluation considering cluster size distribution and the meaningfulness of clusters formed could lead to a more comprehensive understanding of clustering effectiveness, ensuring statistical performance and practical significance. 3 our methodology recent studies utilizing online lda approaches for clustering short text streams have reported substantial performance achievements. however, these studies have predominantly focused on smaller datasets, mainly in specific domains such as news and tweets. this focus has left a significant gap in our understanding of these methods’ scalability and effectiveness across larger and more varied datasets, especially in dynamic sectors like e-commerce. in light of these limitations, our study aims to: 1. assess the method’s effectiveness on an e-commerce dataset that varies significantly in size and number of clusters to determine adaptability and scalability. this evaluation will help us understand how well the method performs when subjected to the complexities and varied dynamics of e-commerce texts compared to traditional short text streams. 2. investigate the influence of high homogeneity scores on the reported nmi values, mainly focusing on whether the prevalence of single-element clusters inflates these scores. this aspect is crucial for ensuring that the nmi values reflect clustering effectiveness rather than artefacts of the clustering process. 3. propose and test using the akaike information criterion (aic) as an alternative metric to mitigate the formation of single-element clusters. including aic is expected to provide a more robust evaluation framework that discourages overly simplistic clustering solutions and promotes more meaningful and practical clustering outcomes. the methodology involves a systematic grid search across datasets ranging from 10,000 to 150,000 instances to assess how varying parameters impact performance. it will lay the groundwork for a detailed analysis, where each dataset size informs the parameter settings for the next, ensuring continuous refinement and optimization of both aic and nmi. the aic optimization aims to minimize the aic value, while the nmi optimization aims to maximize the nmi score. this approach seeks to provide insights into the method’s performance across various scenarios and improve clustering practices. 3.1 datasets our experimental study employed a set of datasets based on the brazilian nf-e project, initiated in 2006, which revolutionized tax documentation by transitioning to an electronic system. this shift has generated a large volume of data, including comprehensive invoice details, making it a valuable resource for machine learning applications, such as detecting tax fraud. we analyzed a subset of the nf-e dataset from november 2021, provided by the amazonas state department of finance. the dataset contains several key features, including: gtin: the gtin is an identifier for trade items developed and controlled by gs1, formerly ean/ucc. gtins, formerly called ean codes, are assigned to any item (product or service) that can be priced, ordered, or invoiced at any point in the supply chain. ncm: the ncmfootnote 1, which is a regional nomenclature for the categorization of goods adopted by since 1995, being used in all foreign trade operations of mercosur countries. the ncm is based on the hs, a condensed expression of the “harmonized commodity description and coding system” maintained by the wco. it was created to improve and facilitate international trade and its statistical control. product description: a free text field on the invoice with a brief product description. table 1. examples of brazilian products with gtin and ncm. ncm codes 22030000 (general beer packaging) and 22030010 (specific types like wheat beer). both share ncm4 2203 (“beer”), but the additional digits provide specific details about type and packagingfull size table 3.2 data pre-processing to better understand the intrinsic properties of our dataset, we have segmented the data into manageable blocks, increasing in increments of 10,000 up to a maximum of 150,000 instances. this segmentation is designed to facilitate analysis and ensure each subset remains a practical size for detailed cluster evaluation. in our analysis, we distinguish between two coding formats used for dataset generation: ncm4 and ncm8. the ncm4 code signifies broader product categories using just the first four digits, providing a general view of item groups. in contrast, the ncm8 code utilizes all eight digits for a more detailed and specific classification, revealing finer distinctions in product types. fig. 1. characteristics of the datasets: number of documents (d), number of clusters (k), vocabulary size (v), average length of the documents (avglen). full size image the behavior of the metric ‘k’, which represents the number of clusters, varies significantly across different data groupings. for ncm4, there is a gradual increase in ‘k’, marked by sporadic larger increases that hint at emerging complexities within the data as the volume grows. the ncm8 displays a more uniform rise in ‘k’, suggesting a linear enhancement in topic diversity corresponding to data size. tailored to specific product descriptions, gtin clusters show a steep upward trend in ‘k’. this indicates a widening range of topics and reflects gtin’s detailed and expansive nature, where adding more product specifics leads to greater diversity in the identified topics. regarding the vocabulary size ‘v’, the gtin dataset shows a pronounced growth, indicating an expanding lexicon that accommodates the increasing diversity of products. however, for ncm subsets—ncm4 and ncm8—the growth in ‘v’ is more measured, underscoring that while gtin descriptions might capture a larger number of clusters due to unique product variations, a single ncm code can encompass a broad range of products, hence a larger vocabulary. finally, in our data preparation, we carefully control the average document length (‘avg len’) to ensure that clusters are not too small or too large, which could skew the analysis. the ‘avg len’ metric shows minor variations across gtin and ncm datasets, with a slight decrease for gtin, suggesting that product descriptions become more concise as the dataset grows. however, for ncm4 and ncm8, there are only slight fluctuations, reflecting the diverse complexity within their broader item categories as they expand. this control of ‘avg len’ is crucial for maintaining the integrity and comparability of clusters across different dataset sizes. 3.3 clustering methods online semantic-enhanced dirichlet model (osdm) [19]: features a dynamic clustering approach, integrating word-occurrence semantic information into a graphical model. it allows for real-time text processing without the need for predetermined cluster numbers or batch sizes, showcasing its adaptability to varying data streams.footnote 2 online semantic-enhanced graphical model (osgm) [20]: uses a poly urn scheme for dynamic document clustering, focusing on semantic smoothing and term co-occurrence to resolve term ambiguity and enhance clarity, proving its strength in semantic-based text clustering.footnote 3 eindm [21]: a dirichlet model using window-based semantic term representation to capture the evolving nature of text data through enhanced semantic understanding and novel clustering metrics.footnote 4 estream [22]: employs both online and offline clustering to effectively adapt to various text streaming scenarios.footnote 5 3.4 evaluation metrics to evaluate our experimental results, we resorted to metrics widely used in the literature [23] to provide a comprehensive assessment of clustering performance and model evaluation. the metrics are the following: homogeneity (h) measures how well each cluster contains only members of a single class. it is defined as: $$\begin{aligned} h = 1 \frac{h(u|v)}{h(u)} \end{aligned}$$ (1) where \( h(u|v) \) is the conditional entropy of the class distribution given the cluster assignments, and \( h(u) \) is the entropy of the class distribution. completeness (c) evaluates how well each class is represented within a single cluster. it is defined as: $$\begin{aligned} c = 1 \frac{h(v|u)}{h(v)} \end{aligned}$$ (2) where \( h(v|u) \) is the conditional entropy of the cluster distribution given the class assignments, and \( h(v) \) is the entropy of the cluster distribution. normalized mutual information (nmi) combines homogeneity and completeness, measuring the similarity between the true class labels and the assigned cluster labels. it is defined as: $$\begin{aligned} nmi = \frac{2 \cdot i(u;v)}{h(u) + h(v)} \end{aligned}$$ (3) where \( i(u;v) \) is the mutual information between class and cluster distribution. purity (p) assesses the clustering quality by calculating the proportion of the dominant class in each cluster. it is defined as: $$\begin{aligned} p = \frac{1}{n} \sum _{k} \max _{j} | c_k \cap t_j | \end{aligned}$$ (4) where \( n \) is the total number of samples, \( c_k \) is the set of samples in cluster \( k \), and \( t_j \) is the set of samples in class \( j \). akaike information criterion (aic) helps balance model fit and complexity. it is defined as: $$\begin{aligned} aic = 2ptr2\ln (l) \end{aligned}$$ (5) where \( { ptr} \) represents the predict/true size rate, and \( l \) is the likelihood of observing the homogeneity value \( h \) given the mean (\( \mu \)) and standard deviation (\( \sigma \)) of the h values across the datasets. the likelihood is computed using the normal distribution: $$\begin{aligned} l = \frac{1}{\sqrt{2 \pi } \sigma } \exp \left( -\frac{(h \mu )^2}{2 \sigma ^2} \right) \end{aligned}$$ (6) the predict/true size rate (ptr) assesses how the optimization of nmi and aic affects the ratio between the number of predicted clusters (\(\hat{n}_c\)) and the real number of clusters (\(n_c\)) and is determined by: $$\begin{aligned} ptr = \left| \frac{\hat{n}_c}{n_c} 1 \right| \end{aligned}$$ (7) these metrics provide a balanced evaluation of clustering performance and model selection, ensuring robustness in our analysis. 4 experimental study 4.1 experimental setup we evaluated various techniques for clustering product descriptions, focusing on methods based on latent dirichlet allocation (lda). for methods such as osdm, osgm, and eindm, we carefully tuned the lda parameters \(\alpha \) (document-topic distribution) and \(\beta \) (topic-word distribution) through a grid search to optimize clustering performance. additionally, for these methods, the decay rate \(\lambda \) was consistently set to \(1\text {e}{-6}\), with all other parameters kept at their default values. estream requires only one parameter: update-interval (ui). this parameter, set to 500 in our experiments, is essential for maintaining cluster quality by updating and removing outdated clusters. these parameter settings were crucial for achieving effective and stable clustering results. 4.2 results in this section, we present our findings through three primary graphs. the initial graph focuses on identifying the best parameter by illustrating the performance of metrics like homogeneity, completeness, nmi, purity, and the cluster-to-label ratio. this investigation is crucial for determining the conditions under which the akaike information criterion (aic) is minimized, as shown in fig. 2. our objective is to reveal how different settings affect clustering quality, mainly through the lens of aic optimization. fig. 2. result of homogeneity, completeness, purity, nmi, predict/true size rate, and aic. for gtin, ncm4, and ncm8 datasets in best nmi analyses context. full size image the second graph illustrates the time evolution for four methods across varying dataset sizes, as depicted in fig. 3. each method’s processing time, represented in seconds, is plotted against the dataset size. the logarithmic scale on the y-axis facilitates comparing relative changes in processing time. this visualization offers insights into how each method’s processing time scales with dataset size, providing valuable information on computational efficiency and scalability without delving into specific numeral evaluations. fig. 3. time evolution (processing time in seconds) versus the size of the gtin dataset for four different methods: eindm, osgm, osdm, and estream. full size image the third graph demonstrates the evolution of the \(\alpha \) and \(\beta \) parameters, as depicted in fig. 4. this analysis segment is essential for comprehending how nmi and aic adjust to parameter changes and for identifying the configurations that lead to the best results. the study focuses on the methods eindm, osdm, and osgm while omitting estream since it does not involve parametric techniques. this focused examination aids in identifying the optimal parameters for enhanced clustering efficiency. fig. 4. \(\alpha \) and \(\beta \) evolution for the methods eindm, osdm and osgm. full size image 5 discussion aic analysis highlights cb-tmcoh and tmcoh’s superior efficiency compared to mstream, indicating better optimization of cluster numbers without sacrificing model complexity. this underscores their ability to capture data structure effectively while avoiding overfitting. additionally, cb-tmcoh outperforms existing lda methods like tmcoh and faststream, particularly regarding completeness and purity. this demonstrates its effectiveness in grouping similar elements while ensuring class exclusivity within clusters. 5.1 metrics evaluation for datasets ncm4, ncm8, and gtin, homogeneity varies significantly. in ncm4, with fewer clusters, homogeneity is generally higher as clusters are likely more uniform but could be less comprehensive. in contrast, ncm8 and gtin, with more complex and numerous clusters, show a potential decrease in homogeneity due to the wider variety of data points within each cluster. comparatively, between the two cases (e.g., best aic vs best nmi), best nmi prioritizes achieving higher homogeneity as it directly influences the mutual information score, leading to better-aligning clusters with actual data classes. ncm4’s completeness tends to be high due to simpler data structures, where capturing all class members in a single cluster is easier. however, in ncm8 and gtin, increasing data complexity and cluster numbers can reduce completeness as classes may spread over multiple clusters. between cases, best aic might show reduced completeness compared to best nmi because aic focuses more on model simplicity and goodness of fit rather than ensuring all class members are together, directly influencing nmi. purity is generally higher in datasets with fewer clusters like ncm4, where the likelihood of mixing different classes in one cluster is lower. in datasets like ncm8 and gtin, despite a higher number of clusters, the complexity of data might lead to lower purity as clusters can contain a more diverse set of class labels. in terms of cases, best nmi likely shows better purity than best aic, as maximizing mutual information would inadvertently align clusters more cleanly with true class labels. nmi is particularly insightful for comparing clustering performance across datasets. ncm4 might show stable nmi values due to less complexity, while ncm8 and gtin could exhibit more variability in nmi due to their increased complexity and cluster counts. comparing cases, best nmi is explicitly optimized to maximize this metric, likely resulting in higher nmi scores across all datasets than best aic, which might sacrifice some mutual information for simpler model structures. the ptr measures the clustering algorithm’s efficiency in controlling cluster sizes, ideally approaching 1. ncm4 usually shows higher ptr, indicating over-clustering. ncm8 and gtin show lower ptr due to their complex structures, leading to many small clusters. best aic, favoring simpler models, typically has ptr closer to 1 than best nmi. best nmi might increase the number of clusters to boost mutual information, resulting in lower ptr. aic values can help gauge the efficacy of clustering concerning model simplicity. lower aic values in ncm4 suggest a better fit with simpler models. in contrast, higher aic in ncm8 and gtin might indicate that the models, despite being more complex, are necessary to fit the data adequately. comparatively, best aic consistently focuses on minimizing aic, potentially leading to simpler, more adaptable models across datasets than best nmi, which might accept more complex models if they yield better information sharing. 5.2 assessment of methods eindm is likely effective in datasets like gtin, where data may be diverse and voluminous. eindm can adapt to complex structures by combining multiple models to improve clustering accuracy. in ncm4, which may be simpler, the ensemble approach might not significantly outperform simpler methods but could still provide robustness against noise. in best aic, eindm might be configured between the two cases to limit the model complexity and avoid unnecessary computation. in contrast, in best nmi, the ensemble could be tuned to maximize information captured from the data, potentially at the expense of increased computational load. estream tends to form more clusters than exist, performing better only than osgm. it achieves worse results than the best nmi cases but better than the best aic cases. however, it offers an excellent cost-benefit ratio as it is the most computationally efficient method, especially for the gtin dataset. odsm achieves its best performance with smaller and simpler datasets like ncm4. as the complexity of the dataset increases, its performance degrades. odsm does not tend to form many clusters for datasets with many classes, maintaining a predict/true size rate close to ideal. osgm has the worst case for aic values due to its tendency to form many clusters. even in the best aic case, this method generates more clusters than others. the best scenario to use osgm is with the gtin dataset and with large datasets. between cases, best aic would prefer setting stringent criteria for model updates to avoid complexity. at the same time, best nmi might leverage the flexibility of gaussian mixtures to refine cluster definitions continuously, aiming for higher nmi scores. each method exhibits a distinct pattern of processing time as dataset size increases. eindm demonstrates a notable increase in processing time with larger datasets, while osgm and osdm also show an upward trend, albeit with different slopes. in contrast, estream exhibits significantly lower processing times across all dataset sizes, indicating its efficiency in handling larger datasets. 5.3 hyperparameters the parameter \(\alpha \) shows a varying pattern across the dataset sizes for the eindm method. at a size of 10,000, the \(\alpha \) values are 0.4 and 1.0 for cases best aic and best nmi, respectively. as the dataset size increases to 20,000 and 30,000, the \(\alpha \) values remain consistent at 0.4 for the best aic case. still, the values adjust slightly to 1.0 for the best nmi case, indicating a stable but distinct configuration for optimal clustering performance. this evolution suggests that the \(\alpha \) parameter is finely tuned to balance, preserving cluster structure and enhancing cluster homogeneity. the parameter \(\beta \) exhibits a steady behaviour, particularly for the best aic cases, where it remains constant at 0.200 for the initial dataset sizes of 10,000 and 20,000. then, it reduces to 0.100 for a dataset size of 30,000. \(\beta \) values are significantly lower for the best nmi cases, starting at 0.006 and 0.004 for dataset sizes of 10,000 and 20,000, respectively. this trend continues as the dataset size increases, reflecting the parameter’s role in optimizing cluster quality. the reduction in \(\beta \) values for the best nmi case suggests a focus on enhancing the granularity of clustering results to achieve a higher normalized mutual information score. 6 conclusions our work demonstrates that lda-based methods face significant difficulties when applied to large datasets. this is evidenced by the increasing processing time proportional to the dataset size, highlighting the scalability challenges inherent in traditional lda approaches. it was confirmed that high nmi and homogeneity scores are often associated with forming single-element or small clusters. this is evident from the elevated predict/true size rate observed. however, an exception was noted in the gtin dataset, where methods such as osdm, eindm, and estream did not conform to this premise, indicating some variability in performance across different datasets. evaluating clustering based on the best aic demonstrated effectiveness in controlling single-element clusters. however, this came at the cost of reduced nmi, homogeneity, purity, and completeness. this trade-off suggests that while aic can mitigate overly simplistic clustering solutions, it may also impact the overall quality of the clustering results. notes 1.the southern common market (mercosur for its spanish initials) is a regional integration process, initially established by argentina, brazil, paraguay and uruguay, and subsequently joined by venezuela and bolivia*. 2.https://github.com/jaykumarr/osdm. 3.https://github.com/jaykumarr/osgm. 4.https://github.com/jaykumarr/eindm. 5.https://github.com/rashadulrakib/short-text-stream-clustering/tree/master/onlineclustering. references cheng, x., yan, x., lan, y., guo, j.: btm: topic modeling over short texts. in: ieee transactions on knowledge and data engineering, vol. 26, no. 12, pp. 2928–2941 (2014). https://doi.org/10.1109/tkde.2014.2313872 zhang, p.: evaluating accuracy of community detection using the relative normalized mutual information. j. stat. mech: theory exp. 2015(11), p11006 (2015) article  mathscinet  math  google scholar  jerdee, m., kirkley, a., newman, m.e.j.: normalized mutual information is a biased measure for classification and community detection. arxiv preprint arxiv:2307.01282 (2023) devlin, j., chang, m.-w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) campello, r.j.g.b., moulavi, d., sander, j.: density-based clustering based on hierarchical density estimates. in: pacific-asia conference on knowledge discovery and data mining, pp. 160–172. springer (2013) google scholar  dai, z., callan, j.: deeper text understanding for ir with contextual neural language modeling. in: proceedings of the 42nd international acm sigir conference on research and development in information retrieval (2019) google scholar  asyaky, m.s., mandala, r.: improving the performance of hdbscan on short text clustering by using word embedding and umap. in: 8th international conference on advanced informatics: concepts, theory, and applications (icaicta), pp. 1–6. ieee (2021) google scholar  eklund, a., forsman, m.: topic modeling by clustering language model embeddings: human validation on an industry dataset. in: proceedings of the 2022 conference on empirical methods in natural language processing: industry track, pp. 635–643 (2022) google scholar  schulte, j.p., et al.: elinac: autoencoder approach for electronic invoices data clustering. appl. sci. 12(6), 3008 (2022) google scholar  andrade, c., ribeiro, r.p., gama, j.: topic model with contextual outlier handling: a study on electronic invoice product descriptions. in: epia conference on artificial intelligence, pp. 365–377. springer (2023) google scholar  blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3(jan), 993–1022 (2003) google scholar  liang, s., yilmaz, e., kanoulas, e.: dynamic clustering of streaming short documents. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 995–1004 (2016) google scholar  yin, j., wang, j., xu, w., gao, m.: model-based clustering of short text streams. in: 27th acm international conference on information and knowledge management, pp. 697–706. acm (2018) google scholar  chen, j., gong, z., liu, w.: a nonparametric model for online topic discovery with word embeddings. inf. sci. 504, 32–47 (2019). elsevier google scholar  chen, j., gong, z., liu, w.: a dirichlet process biterm-based mixture model for short text stream clustering. appl. intell. 50, 1609–1619 (2020). springer google scholar  xu, y., wang, s., zhang, s., wang, f.: dynamic clustering for short text stream based on dirichlet process. ieee access 10, 22852–22865 (2022) math  google scholar  rakib, m.r.h., asaduzzaman, m.: fast clustering of short text streams using efficient cluster indexing and dynamic similarity thresholds. corr abs/2101.08595 (2021) google scholar  rakib, m.r.h., zeh, n., jankowska, m., milios, e.: enhancement of short text clustering by iterative classification. in: natural language processing and information systems: 25th international conference on applications of natural language to information systems, nldb 2020, saarbrücken, germany, june 24–26, 2020, proceedings 25, pp. 105–117. springer (2020) google scholar  kumar, j., shao, j., uddin, s., ali, w.: an online semantic-enhanced dirichlet model for short text stream clustering. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 766–776. association for computational linguistics (2020) google scholar  kumar, j., ud din, s., yang, q., kumar, r., shao, j.: an online semantic-enhanced graphical model for evolving short text stream clustering. ieee trans. cybern. 52(12), 13809–13820 (2021) google scholar  kumar, j., shao, j., kumar, r., ud din, s., mawuli, c.b., yang, q.: a context-enhanced dirichlet model for online clustering in short text streams. in: expert systems with applications, vol. 228, p. 120262 (2023) google scholar  rakib, m.r.h., zeh, n., milios, e.: efficient clustering of short text streams using online-offline clustering. in: proceedings of the 21st acm symposium on document engineering, pp. 1–10 (2021) google scholar  murshed, b.a.h., et al.: short text topic modelling approaches in the context of big data: taxonomy, survey, and analysis. artif. intell. rev. 56(6), 5133–5260 (2023) google scholar  download references acknowledgments the authors wish to clarify that the first author received support from amazonas state government/brazil for this research project. author information authors and affiliations department of computer science, faculty of sciences, university of porto, 4169-007, porto, portugal cesar andrade & rita p. ribeiro faculty of economics, university of porto, 4200-464, porto, portugal joão gama faculty of computer science, university of porto, porto, portugal joão gama inesc tec, 4200-465, porto, portugal cesar andrade, rita p. ribeiro & joão gama authors cesar andradeview author publications search author on:pubmed google scholar rita p. ribeiroview author publications search author on:pubmed google scholar joão gamaview author publications search author on:pubmed google scholar corresponding author correspondence to cesar andrade . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper andrade, c., ribeiro, r.p., gama, j. (2025). evaluating short text stream clustering on large e-commerce datasets. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_17 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords short text stream clustering latent dirichlet allocation online publish with us policies and ethics profiles cesar andrade view author profile rita p. ribeiro view author profile joão gama view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature predicting engagement of brazilian politicians on tiktok: a machine learning approach | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper predicting engagement of brazilian politicians on tiktok: a machine learning approach conference paper first online: 30 january 2025 pp 387–401 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) predicting engagement of brazilian politicians on tiktok: a machine learning approach download book pdf download book epub maria santana  orcid: orcid.org/0009-0001-0797-28289, josé santana  orcid: orcid.org/0009-0005-5289-368510, pablo sampaio  orcid: orcid.org/0000-0001-8254-560x9 & … kellyton brito  orcid: orcid.org/0000-0002-6883-86579  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 394 accesses abstract while established social media platforms like facebook, twitter, and instagram have become staples in political campaigns, the 2022 brazilian elections witnessed the rise of a new contender: tiktok. despite its recent emergence in 2016, tiktok has already become the fourth most used social network in brazil. this study investigates the potential of machine learning to predict engagement on the tiktok profiles of the two leading presidential candidates: lula and bolsonaro. utilizing a dataset from previous studies, we implemented various machine learning models and found that the support vector machine achieved the highest performance based on the f1-score metric for both candidates. despite the results being better with bolsonaro than with lula, further analysis of metrics like recall and precision suggests valuable insights for social and political domains. these findings can aid both candidates and society in understanding what factors are most related to engagement on this emerging social media platform. additionally, marketing and advertising teams can use this information to create content tailored to reach and engage with a politician’s target electorate. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others predicting an election’s outcome using sentiment analysis chapter © 2020 election tweets prediction using enhanced cart and random forest chapter © 2021 elections without boundaries: the use of facebook in the 2022 brazilian presidential elections chapter © 2024 1 introduction despite being relatively new, social media platforms are now an integral part of daily life for millions of brazilians. whatsapp, instagram and facebook are the three more accessed platforms by brazilian users. facebook messenger, telegram, pinterest, kuaishou (kwai), twitter and linkedin follow as the next most popular platforms [1]. within this context, brazilian politicians are active users of these platforms, employing them as communication tools to engage with their constituents and conduct political campaigns. recent brazilian elections have demonstrated this trend, such as in 2018 when former president bolsonaro was elected with minimal exposure on broadcast television. his electoral success is often attributed to his presence on social media [2, 3]. previous research has attempted to identify correlations between elements of politicians’ posts and profiles and electoral outcomes. while some studies have yielded results that support this hypothesis [4], others lack conclusive findings and are still undergoing testing of their approaches [5]. among the most popular social media platforms in brazil today, a new platform aimed at young audiences [6] stands out: tiktok. this platform was largely ignored by candidates in the 2018 brazilian presidential election but was heavily utilized in the 2022 election. however, despite being the sixth most used social network in the world [7], due to its early stage of usage in electoral campaigns, it is still challenging to find studies that analyze and predict outcomes within the context of elections. an exception presented in [8] is an analysis of the content of videos on this platform by the main presidential candidates in 2022, lula and bolsonaro, based on creating and applying a taxonomy to identify correlations between the type of content and candidate engagement. however, the dataset was not used for predicting engagement. in this context, this article presents the results of a study that aims to predict engagement on tiktok based on the use of the platform by the two main candidates in the 2022 brazilian presidential election. thus, it aims to answer two research questions: rq1: to what extent can discourse features effectively predict engagement levels in political posts using basic machine learning models? and rq2: which discourse features are the most important for predicting the number of likes (engagement) based on the trained machine learning models? for this, we used the classification taxonomy and the data collected and classified in [8] to train several models for the first time, including neural networks, support vector machine, random forest, k-nearest neighbors, and logistic regression. the remainder of this paper is as follows. in sect. 2, related works will be addressed. section 3 discusses the methodology used in this study. section 4 presents the results. section 5 covers the discussions regarding the results, while sect. 6 presents conclusions and future work. 2 related works starting with barack obama’s 2008 campaign [9], political strategy has been revolutionized over the last 15 years through social media, sparking researchers’ interest and numerous studies on social medias’ impact on political campaigns. notable studies include [10] investigation of donald trump’s 2016 campaign in the united states and [3] study of jair bolsonaro’s 2018 campaign in brazil. more specifically, some research seeks to identify a possible correlation between the engagement generated on candidates’ social media platforms and the number of votes they receive in elections. the work of [11] was groundbreaking in proposing that the number of tweets mentioning a political party could serve as a reliable indicator of the percentage of votes it would receive, demonstrating predictive power like traditional opinion polls. most subsequent studies followed this line of thought, focusing on twitter analysis [4]. however, this twitter-centric approach has faced increasing criticism [12]. one of the main objections points out that twitter is not among the top 10 most used social networks globally, raising questions about the representativeness of the analyzed data and several biases related to data gathering. working from a sentiment analysis perspective, a recent work [13] proposed a model for election prediction using sentiment analysis of influential messages on facebook in the 2016 united states presidential election. the results demonstrated better classification performance with the random forest algorithm and that predicting election results based on sentiment analysis of messages was reliable. with a multi-platform approach, vepsäläinen and suomi [14] investigates the role of two different social media platforms, twitter and facebook, in predicting the 2019 finnish parliamentary elections. the results demonstrated that both the number of likes on facebook and the number of followers on twitter are positively associated with election results. the test results suggest that the number of followers on twitter and likes on facebook are more influential in determining the election outcome for candidates with less political experience. it is clear, therefore, that the platform most used by researchers for political data analysis is twitter, with most applied methods focusing on volume and sentiment analysis of mentions. this approach is widely criticized [15]. among the reasons is the fact that in 2022, twitter was the fifteenth most-used platform in the world [16] and the ninth in brazil [7]. meanwhile, tiktok is the sixth most used platform in the world [16], the fourth most used in brazil [7], and preferred by young people [6]. as a recent network, politicians are still starting to use tiktok, and there is little research focused on analysis and prediction regarding this social media and electoral results. thus, the present work uses features based on a deep analysis of the discourse found in videos on the tiktok platform found in [8] using the proposed taxonomy and the dataset used in that same research. most of the other works are less based on content, but this one attempts to make predictions based on a deep analysis of the content to assess the predictive capacity of this analysis. 3 methodology this study aims to investigate the relationship between discourse features and engagement levels in political posts on social media. specifically, we seek to answer the following research questions, segmented per candidate: rq1: to what extent can discourse features effectively predict engagement levels in political posts using basic machine learning models? rq2: which discourse features are the most important for predicting the number of likes (engagement) based on the trained machine learning models? to answer these research questions, we defined a methodology consisting of five steps: data collection, preprocessing, data transformation, model training, and analysis of the training results. 3.1 data collection the data utilized in this study were obtained from a public dataset used in previous works [8, 17, 18]. while this dataset has been employed in various data analyses before, this is the first study to use it for developing predictive machine learning models. the dataset includes features from videos posted by the two main candidates in the 2022 brazilian presidential election, lula and bolsonaro, on the tiktok platform. the data collection period spanned from june 30th to october 30th, totaling 122 days of collection. the studies focused on the two main candidates for the presidency of brazil, lula (@lulaoficial) and bolsonaro (@bolsonaromessiasjair), which together received 92% of the total votes in the first round. the data was manually classified with attributes related to features of the discourse of the candidates observed in the videos (tiktok posts), based on a predefined taxonomy [8]. more details of the data collected and their features are provided in sect. 4.1. 3.2 preprocessing the preprocessing involves: data cleaning, manual feature engineering, data subdivision and target attribute specification. regarding data cleaning, we simply removed posts with missing values in any of its attributes. during preprocessing, we manually engineered a new feature from the original dataset that may be relevant to the classification task. the feature, called “days elapsed,” represents the number of days between the publication date of a post and the date the data was collected for that post (to form the dataset). this can be justified by the fact that the posts in our dataset were not exposed for proportionally similar periods and that they were released in different periods of the campaign. regarding data subdivision, we separated the datasets into bolsonaro posts and lula posts to enable independent analysis for each political figure. a special care was taken regarding the target attribute specification. the basic raw attribute used to measure engagement in this work is the number of likes received by a post. this metric offers several advantages: it is unique to each user, preventing duplication of engagement measures, and it exhibits a linear correlation with other potential engagement metrics such as the number of comments, shares, and plays. then, we opted for a classification approach. we divided the posts of each candidate into two distinct classes based on their engagement levels: high engagement, for posts with engagement levels at or above the median (50th percentile) of a candidate’s posts; and low engagement, encompassing the other half of the posts. we chose to treat this study as a classification problem rather than a regression problem because we understand that it would have similar practical relevance in a system designed to recommend whether or not to proceed with a post (before actually posting), for example. additionally, creating models that accurately capture the distribution of likes can be particularly challenging due to the highly non-linear nature of engagement (for instance, small differences in content can lead to a post going viral or not). 3.3 data transformation we have tested different transformations for categorical features and numerical features. after applying the transformations of the categorical features, the resulting features are also subject to the second type of transformations. categorical features. since most of the features in the data are categorical, it is especially important to evaluate the impact of different transformations for this type of feature. therefore, we evaluated three alternate transformation approaches: one-hot encoding: converts a categorical feature with n categories into n binary features, each indicating the presence of a specific category. target encoding for (binary) classification: transforms categorical variables into continuous variables based on their relationship with the target classes in the training data [19]. target encoding for regression: a version of target encoding for continuous target variables [19]. we hypothesize that applying the target encoding to the raw values of likes could render useful information for the binary classification task. numerical features. for machine learning models that are sensitive to the scale of the data, we also tested three alternative transformations for numerical features. min-max (0–1) scaling: this technique sets the minimum value of each feature to 0 and the maximum value to 1, linearly scaling all other values proportionally within this range. standard scaling (z-score normalization): standardize the data to have a mean of 0 and a standard deviation of 1. identity: no transformation is applied; the original values are used. to ensure the integrity of our model and prevent data leakage, both types of transformation were applied strictly to the training data within the training pipeline (and not to the full dataset, as a preprocessing step). one-hot encoding, however, is always applied considering all the categories of the full dataset, since these were part of a pre-defined taxonomy [8]. 3.4 model training different classification models were evaluated with different hyperparameters in this study, as detailed in table 1. hyperparameters not listed received the default values of the scikit-learn 1.4 implementation (e.g., mlp used the adam optimizer), except for logistic regression, where saga solver was chosen for its flexibility regarding the penalty term. for model type, a grid search approach was employed to evaluate all combination of feature transformations and hyperparameters. in this context, the transformations were treated similarly to hyperparameters to explore their impact on the model performance. we use the term model configuration to refer to the hyperparameters of the model combined with the transformations to be applied to the input data. table 1. models and set of hyperparameter values evaluated.full size table nested cross-validation procedure. to robustly train and evaluate each type of model given the relatively limited amount of data, we employed a nested 5-fold cross-validation (cv) approach. this technique provides a more reliable estimate of model performance and helps mitigate the risk of overfitting. the 5-fold nested cv procedure involves splitting the data into 5 folds and iterating five times. in each iteration, 4 folds are used for training, while the remaining fold is held out as a test set for evaluation. each fold maintained an equal class distribution, with 50% for each class. in each iteration of the outer cv, the grid search is applied to this training (4 folds of data) to find the optimal model configuration for that iteration. in each run of the grid search, each model configuration (i.e. combination of hyperparameters and transformations) was evaluated using an inner 5-fold cv applied to the training data. this inner cv further splits the training data into 4 inner folds for model fitting and 1 inner fold for validation. the average f1-score of the 5 iterations of this inner cv is used to assess each model configuration. as a result of each run of grid search (in each iteration of the outer cv), one optimal model configuration is identified for each iteration of the outer cv. so, in total, five optimal configurations will be found per model type. model evaluation. at the end of each iteration of the outer cv, the model is retrained using the optimal configuration on the entire training data (i.e. the four selected outer folds). the model is then evaluated on the held-out test fold using these metrics: accuracy, precision, recall, and f1-score. we highlight that one optimal model configuration (hyperparameters + transformations) is obtained and evaluated per iteration of the outer cv, thus resulting in five configurations as well as five values for each of the evaluation metrics, per model type. these values and settings are used as inputs for the subsequent analysis step. 3.5 analysis of results for each of research questions proposed we performed a specific analysis, as detailed next. model performance analysis. to address rq1, we first calculated and reported the mean and standard deviation of the evaluation metrics for each model type and dataset. to further understand how to effectively predict the engagement, we analyzed the optimal model configurations obtained, reporting the most frequent data transformations chosen in the grid search. we also discuss key differences or similarities observed in model performance and configuration between bolsonaro’s and lula’s datasets. features importance analysis. to address rq2, we analyzed the feature importance scores derived from the optimal random forest configurations. this model type inherently provides a widely used metric for feature importance, known as gini importance. this metric quantifies the average decrease in node impurity achieved by splitting on a specific feature, reflecting the extent to which each feature contributes to the model’s predictive power. for each political candidate, we aggregated the gini scores obtained in the five optimal random forest configurations identified during model training (sect. 3.4) to obtain a more robust result. 4 results 4.1 data collection the dataset encompasses various features as outlined in study [8], with most being categorical except for the continuous duration feature. these features include the candidate (lula or bolsonaro), duration (length) of the video, aristotelian rhetoric (logos, ethos, pathos), content type (such as personal, political ideological, campaign act, and trend), functional approach (attack, defense, acclaim), tonality (positive, negative, neutral), and main character (candidate, candidate with voters, voters, influencers, opponent). these features were manually labeled based on a predefined taxonomy aimed at classifying varied aspects of the politicians’ discourse displayed in each video. for example, aristotelian rhetoric consists of three categories (values) of rhetorical appeal, which aim to convince the interlocutor: logos, which indicates an appeal to logical reasoning and argumentative structure; ethos, appeal to the author’s character, such as their trustworthiness and qualifications; and pathos, appeal to the audience’s emotions. for further details on the attributes, please refer to study [8]. additional features used in the present study, which were collected as part of the research study [8] but not reported in it, are described below: rhetorical device with the following categories: advertisement – videos (tiktok posts) that aim to promote or sell the political candidacy; call to action – seek to mobilize the public to take a specific action, such as voting or participating in an event; collective appeal – appeal to the union or collaboration of a specific group or society in general; commitment – display the politician at an event such as parades and speeches; endorsement – represent support for the politician and a way to value their ideas and what they represent; fact/statistic – present facts or statistics to inform or persuade the public; humor – use humor to convey a political message; opinion – the politician expresses their opinion; personal appeal – the politician appeals to empathy or personal identification; political statement – the politician makes a clear statement on a political issue; thanks –the politician expresses gratitude to specific people or groups; urgency – videos that convey a message of urgency regarding a cause or political issue; none – videos that do not fit into any of the listed categories. textual content has four categories that represent how the video is accompanied by a caption: hashtag (only), none (no text), text, text + hashtag. post date: the date in which the video was posted on tiktok. the data collection period spanned from june 30th to october 30th, 2022. collection date: the date on which all the post’s features were collected. approximately 75% of the posts were collected in october the 2nd, while the remaining data was collected on october the 30th, 2022. additionally, we had access to engagement variables for each post (e.g., plays, shares, comments), but we focused on likes as explained in sect. 3.2. 4.2 preprocessing during data cleaning, eight posts were removed due to missing values, all from bolsonaro’s posts. the final data was then separated into two datasets, one for each candidate, resulting in 261 posts in bolsonaro’s dataset, and 308 posts in lula’s dataset. the target attribute for binary classification was created by performing a median split on the number of likes. this resulted in two categories: high engagement posts and low engagement posts. in bolsonaro’s dataset, the median number of likes was 31,600. consequently, 131 posts were classified as low engagement and 130 posts as high engagement. for lula’s posts, the median was 19,400, resulting in 154 posts categorized as low engagement and 154 posts categorized as high engagement. we engineered a new feature called days elapsed by subtracting the publication date of each post from the date of data collection. this feature accounts for the varying exposure times of posts in our dataset and for the different periods of campaign that it was related to, which could potentially influence engagement levels. 4.3 model training the models had their parameters selected and their performance assessed using the nested 5-fold cross-validation procedure explained in the sect. 3.3. for each candidate, considering all options for all stages of the pipeline, including possible data transformations and hyperparameters, a total of 243 different configurations of the mlp model were tested, along with 360 configurations of svm, 216 of random forest, 216 of logistic regression, and 144 of knn. these numbers reflect the various combinations of feature transformations and hyperparameter settings explored for each model type. considering the two levels of 5-fold cross-validation (25 iterations in total) and the final training with the best configuration for each fold (5 additional trainings, being one per outer fold), this resulted in 29,480 model training instances being performed. the results of these training runs are discussed in detail in the following sections. 4.4 data transformation this section analyzes the data transformations chosen in the best-performing model configurations found in each of the five outer folds of nested cross-validation employed. the data transformations are presented in table 2, where the first three transformations (rows of the upper part of the table) are categorical, while the last three transformations (lower part) are numerical. note that there is a total of five categorical transformations and five numerical transformations per model (column). our primary focus is on identifying the most frequently selected transformations for categorical data, as these constitute a significant portion of the features in our datasets. table 2. data transformations chosen in the model’s best configurations.full size table categorical features. for bolsonaro’s posts, the optimal configurations across all model types exhibited a relatively even distribution of chosen transformations. one-hot encoding was selected in 9 configurations, proving particularly useful for knn models (being chosen in 4 of the 5 best configuration). both target encoding for binary classification and target encoding for regression were chosen 8 times each. similarly, in lula’s dataset, one-hot encoding was favored in 9 of the best configurations, especially for logistic regression models (3 out of 5 configurations). target encoding for binary classification was chosen 4 times. target encoding for regression was selected 12 times, being present in at least one optimal configuration of each model type. overall, target encoding for regression emerged as the most frequently chosen transformation for categorical data (20 times, being 8 in bolsonaro’s dataset and 12 in lula’s dataset). this supports our hypothesis that encoding categorical features based on the distribution of the raw continuous variable (number of likes) can capture valuable information even when the target variable is ultimately transformed in a 2-class variable (high or low engagement). numerical features. min-max scaling was the most common transformation for numerical features, being found 21 times in the optimal model configurations. standard scaling was also frequently selected, 13 times. no transformation (identity) was chosen in 6 model configurations, most of them in models trained on lula’s data. 4.5 analysis model performance – bolsonaro’s posts. the performance results for bolsonaro’s dataset are presented in table 3. support vector machine (svm) emerged as the best-performing model with an f1-score of 0.69. logistic regression, k-nearest neighbors (knn), and multi-layer perceptron (mlp) followed closely with f1-scores of 0.68, 0.67, and 0.66, respectively. random forest achieved a slightly lower f1-score of 0.64. table 3. performance results for the bolsonaro dataset (mean ± standard deviation).full size table while the overall model performance may not be considered remarkable, the results demonstrate the potential for using machine learning to identify posts with a high probability of achieving high engagement. for example, the svm model achieved a recall of 0,707, indicating that it can correctly identify actual high-engagement posts 41% better than a random classifier (that would display a recall of 0.5). the precision of 0.688 indicates that when it positively predicts that a post will have high engagement, the model is correct 68.8% of the time, which is a 37.6% improvement over a random classifier (precision 0.5). model performance – lula’s posts. the performance results for lula’s dataset, presented in table 4, indicate that the models struggled to achieve satisfactory performance. this suggests that the selected features may not be sufficient to effectively differentiate between high and low engagement posts in lula’s case. the svm model achieved the highest f1-score of 0.61 (with a standard deviation of 0.07), primarily due to a high recall of 0.85 but a low precision of 0.50. this suggests that the model can identify a large portion of actual high-engagement posts but also generates a substantial number of false positives. furthermore, three of the five model types (logistic regression, random forest and mlp) achieved f1-scores below 0.50, indicating their limited ability to effectively distinguish between the two classes in lula’s case. table 4. performance results for the lula dataset (mean ± standard deviation).full size table features importances. we also analyzed, for each candidate, the feature importances identified in the 5 best random forest models (one per fold of the outer cross-validation). the exact names of the features may vary between models, depending on the transformations applied to the categorical features. one-hot encoding transforms one feature into multiple features, one for each value, that appear named as “<feature>_<value>” in the results. however, target encoding preserves the name of the features. for the models that used the same transformations and used the same feature names, we averaged the importances of these models. fig. 1. importance of the bolsonaro candidate’s features. full size image from the 5 best configurations of random forest classifier trained in bolsonaro’s data, only one of them applied one-hot encoding. the feature importances calculated from this model for the best features is reported in the upper part of fig. 1. this model highlights the significance of specific rhetorical devices in driving engagement, with urgency and political statement emerging as particularly influential categories. the remaining four models adopted target encoding and the average value of the feature importances extracted from these models is shown on the bottom part of the same figure. while these models also identified rhetorical device as a crucial feature, they did not provide specific insights into the impact of individual categories. interestingly, elapsed days, a feature unrelated to speech content, emerged as the most important feature, indicating the significant role of post timing within the campaign period. the other features present notably lower scores. all the five best configurations of the random forest model trained with lula’s data employed some form of target encoding. the averaged importances extracted from these models are shown in fig. 2, ordered from highest to lowest non-zero importance. when compared to bolsonaro’s results for target encoding configurations, we observe a remarkable similarity between the rankings of the features. the only difference in the order is the fifth and sixth positions, which appear swapped. this similarity in the feature importance’s rankings demonstrates consistency in the relevance of these features for predicting high-engagement posts for any candidate. fig. 2. importance of the lula candidate’s features. full size image 5 discussion what makes a post engaging can be a series of things, not just the type and content of the post, including everything from the general audience’s taste to whether it became a ‘meme’ or got involved in controversy in some way. thus, given the limited aspects captured by the data utilized, we believe our results were relevant. the results for bolsonaro demonstrate the potential for using machine learning to identify posts with a high probability of achieving high engagement with an f1-score of 0.70. the svm model can correctly identify actual high-engagement posts 41% better than a random classifier (0.707/0.5), and the precision metric indicates that when it positively predicts that a post will have high engagement, it has a 37.6% higher success rate than a random classifier (0.688/0.5). in a typical practical application for this model, such as assisting a candidate’s social media team in filtering content to be posted, the model’s ability to identify potential high-engagement posts could be valuable, potentially improving expected engagement compared to not using such a filter. it is important to note that false positives or false negatives would not be particularly harmful in this application. therefore, this performance is sufficient and represents an improvement over the absence of machine learning-based filtering. the models trained on lula’s data exhibited comparatively lower performance with an f1-score of 0.62, reducing their potential for application. this discrepancy indicates that the factors influencing engagement for lula’s audience are more complex. further research is necessary to explore these complexities, identify new features and refine the models accordingly. the feature importance analysis identified two features with dominant roles: rhetorical devices and elapsed days. the inclusion of the elapsed days feature was crucial for controlling variability in the data, as posts had different durations to accumulate likes. however, we acknowledge that in a practical application aimed at predicting the engagement of a new post, this feature would not be available. to address this, a potential solution could involve allowing users to input a desired timeframe for the prediction (e.g., ‘how many likes to expect in x days’). this would enable the model to adjust its predictions based on the specified time horizon. additionally, future research could explore training models that do not rely on elapsed days or that incorporate user-defined time projections, making them more applicable to real-world scenarios. in general, dimensionality reduction was not explored in this work and may also be investigated in future works. overall, we believe that this analysis is a starting point for understanding the factors driving engagement on his social media content. further investigation into the specific rhetorical devices, content type preferences, and feature interactions can offer deeper insights and inform effective communication strategies. we could also employ other methods to extract information about feature importances. for a critical assessment of this study, it’s crucial to note potential methodological biases that may influence our result or its interpretation. manual post categorization, despite rigorous use of multiple classifiers, may be subject to subjectivity in applying criteria. moreover, data collected during the 2022 brazilian election campaign introduces contextual variations among candidates, limiting findings’ generalizability to other electoral contexts or periods. thus, while valuable for practical use, their applicability beyond the specific context warrants caution. 6 conclusion and future works this article presents a study aimed at predicting the engagement of tiktok posts made by the two main candidates in the 2022 brazilian elections: lula, the current president of brazil, and bolsonaro. the method involved acquiring the dataset from a previous work [8], followed by data analysis, preprocessing, and training. the study accessed additional variables not utilized in the work and engineered a new feature, enabling novel analyses and approaches. the dataset was cleaned and transformed, with special attention given to the categorical features, which were abundant. the transformed data was used to train and compare five types of classification models optimized by a grid search with nested cross-validation: support vector machine (svm), logistic regression, knn, mlp neural network, and random forest. these models were evaluated based on their performance metrics, followed by an analysis of feature importance for each candidate. the classification problem and dataset presented various challenges. for bolsonaro’s posts, the svm model performed best, achieving an f1-score of 69%, with a recall of 70% and precision of 68.8%. these results suggest that machine learning can effectively identify posts likely to gain high engagement, which could be valuable for social media teams in decision-making. this is because accurate predictions can lead to positive outcomes, while the impact of incorrect predictions in this context is relatively low. however, for lula’s posts, models struggled, with the highest f1-score of 61% from the svm model, which had a high recall (85%) but low precision (50%), indicating a significant number of false positives. the most important features for classifying the posts were also identified for each candidate, validating the relevance of the discourse features in effectively predicting engagement. we also discussed limitations that highlight the need for further investigation, such as investigating the removal of the ‘elapsed days’ feature and exploring the effect of reducing the number of input features used by the models. future work could also involve using data from other social media platforms, exploring computer vision techniques (applied to post images), text mining, and additional machine learning methods. lastly, since content analysis was performed manually to create the dataset from [8], employing computer vision techniques, large language models (llms), or automatic post classification offers promising directions for future advancements. the findings contribute to understanding the evolving dynamics of publications on the tiktok platform within the context of political profiles, especially as the social media continues to grow in brazil. additionally, the results hold relevance beyond social network analysis, impacting areas such as social and political sciences by providing insights into politicians’ posting dynamics and voters’ behavior and interests on this platform. furthermore, the study offers valuable information for the fields of marketing and advertising, aiding decision-making regarding publication content based on the political profile of the author, their electorate, and the target audience. the results also empower citizens by raising awareness of the content displayed on their screens, preventing political manipulation, and fostering a more autonomous, secure, and healthy environment for political debate. references datareportal: digital 2024: brazil. https://datareportal.com/reports/digital-2024-brazil. accessed 24 june 2024 brito, k., paula, n., fernandes, m., meira, s.: social media and presidential campaigns – preliminary results of the 2018 brazilian presidential election. in: proceedings of the 20th annual international conference on digital government research, pp. 332–341. acm, new york (2019). https://doi.org/10.1145/3325112.3325252 brito, k., de lemos meira, s.r., adeodato, p.j.l.: correlations of social media performance and electoral results in brazilian presidential elections. inf. polity 26, 417–439 (2021). https://doi.org/10.3233/ip-210315 brito, k., filho, r.l.c.s., adeodato, p.j.l.: a systematic review of predicting elections based on social media data: research challenges and future directions. ieee trans. comput. soc. syst. 8, 819–843 (2021). https://doi.org/10.1109/tcss.2021.3063660 article  math  google scholar  heiss, r., schmuck, d., matthes, j.: what drives interaction in political actors’ facebook posts? profile and content predictors of user engagement and political actors’ reactions. inf. commun. soc. 22, 1497–1513 (2019). https://doi.org/10.1080/1369118x.2018.1445273 article  google scholar  xu, l., yan, x., zhang, z.: research on the causes of the “tik tok” app becoming popular and the existing problems. j. adv. manag. sci. 59–63 (2019) google scholar  simon kemp: digital 2022: brazil. https://datareportal.com/reports/digital-2022-brazil. accessed 03 june 2024 santana, m., lima, j., correa, a., brito, k.: engajamento no tiktok dos candidatos às eleições brasileiras de 2022 – resultados iniciais. in: anais do xii brazilian workshop on social network analysis and mining (brasnam 2023), pp. 151–162. sociedade brasileira de computação sbc (2023). https://doi.org/10.5753/brasnam.2023.230641 bimber, b.: digital media in the obama campaigns of 2008 and 2012: adaptation to the personalized political communication environment. j. inform. tech. polit. 11, 130–150 (2014). https://doi.org/10.1080/19331681.2014.895691 article  google scholar  francia, p.l.: free media and twitter in the 2016 presidential election. soc. sci. comput. rev. 36, 440–455 (2018). https://doi.org/10.1177/0894439317730302 article  math  google scholar  tumasjan, a., sprenger, t., sandner, p., welpe, i.: predicting elections with twitter: what 140 characters reveal about political sentiment. in: proceedings of the international aaai conference on web and social media, vol. 4, pp. 178–185 (2010). https://doi.org/10.1609/icwsm.v4i1.14009. brito, k., silva filho, r.l.c., adeodato, p.j.l.: stop trying to predict elections only with twitter – there are other data sources and technical issues to be improved. gov. inf. q. 41, 101899 (2024). https://doi.org/10.1016/j.giq.2023.101899 oueslati, o., hajhmida, m. ben, ounelli, h., cambria, e.: sentiment analysis of influential messages for political election forecasting. presented at the (2023) google scholar  vepsäläinen, t., li, h., suomi, r.: the role of social media platforms in forecasting elections: a comparison of twitter and facebook. digit. gov. res. pract. (2024). https://doi.org/10.1145/3651227 article  math  google scholar  brito, k., silva filho, r.l.c., adeodato, p.: please stop trying to predict elections only with twitter. in: dg.o 2022: the 23rd annual international conference on digital government research, pp. 88–95. acm, new york (2022). https://doi.org/10.1145/3543434.3543648 simon kemp: digital 2022: global overview report. https://datareportal.com/reports/digital-2022-global-overview-report. accessed 03 june 2024 lima, j., santana, m., correa, a., brito, k.: dataset for the use and impact of tiktok in the 2022 brazilian presidential election (2023). https://doi.org/10.7910/dvn/9l7lei lima, j., santana, m., correa, a., brito, k.: the use and impact of tiktok in the 2022 brazilian presidential election. in: proceedings of the 24th annual international conference on digital government research, pp. 144–152. acm, new york (2023). https://doi.org/10.1145/3598469.3598485 pargent, f., pfisterer, f., thomas, j., bischl, b.: regularized target encoding outperforms traditional methods in supervised machine learning with high cardinality features. comput. stat. 37, 2671–2692 (2022). https://doi.org/10.1007/s00180-022-01207-6 article  mathscinet  math  google scholar  download references author information authors and affiliations departamento de computação, universidade federal rural de pernambuco, recife, brazil maria santana, pablo sampaio & kellyton brito deinfo, universidade federal rural de pernambuco, recife, brazil josé santana authors maria santanaview author publications search author on:pubmed google scholar josé santanaview author publications search author on:pubmed google scholar pablo sampaioview author publications search author on:pubmed google scholar kellyton britoview author publications search author on:pubmed google scholar corresponding author correspondence to pablo sampaio . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper santana, m., santana, j., sampaio, p., brito, k. (2025). predicting engagement of brazilian politicians on tiktok: a machine learning approach. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_27 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords tiktok political campaigns social media engagement prediction machine learning brazilian election publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature hyper-heuristic based nsga-iii for the many-objective quadratic assignment problem | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper hyper-heuristic based nsga-iii for the many-objective quadratic assignment problem conference paper first online: 28 november 2021 pp 170–185 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) hyper-heuristic based nsga-iii for the many-objective quadratic assignment problem download book pdf download book epub bianca n. k. senzaki  orcid: orcid.org/0000-0001-6256-187210, sandra m. venske  orcid: orcid.org/0000-0001-8602-300810 & carolina p. almeida  orcid: orcid.org/0000-0003-4939-643210  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 777 accesses 5 citations abstract the quadratic assignment problem (qap) can be subdivided into different versions, being present in several real-world applications. in this work, it is used a version that considers many objectives. qap is an np-hard problem, so approximate algorithms are used to address it. this work analyzes a hyper-heuristic (hh) that selects genetic operators to be applied during the evolutionary process. hh is based on the nsga-iii framework and on the thompson sampling approach. our main contribution is the analysis of the use of a many objective algorithm using hh for qap, as this problem was still under-explored in the context of many objective optimization. furthermore, we analyze the behavior of operators forward the changes related to hh (ts). the proposal was tested considering 42 instances with 5, 7 and 10 objectives. the results, interpreted using the friedman statistical test, were satisfactory when compared to the original algorithm (without hh), as well as when compared to algorithms in the literature: moea/dd, moea/d, spea2, nsga-ii and moea/d-dra. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-objective quadratic assignment problem: an approach using a hyper-heuristic based on the choice function chapter © 2020 a hybrid greedy indicatorand pareto-based many-objective evolutionary algorithm article 02 january 2021 optimizing construction time, cost, and quality: a hybrid ahp-nsga-iii model for enhanced multi-objective decision making article 17 december 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms genome assembly algorithms genomic engineering marker assisted selection optimization personnel selection 1 introduction the quadratic assignment problem (qap) is an np-hard problem even for small instances [1]. the qap was initially introduced to model a plant location problem, with the objective of placing facilities in places so that the sum of the product between distances and flows is minimal [2]. depending on the needs, several variations of the problem arose, such as the linear allocation problem, quadratic bottleneck allocation problem, the quadratic multi-objective allocation problem, among others. each variation works to solve different real-world problems [1]. the multi-objective qap (mqap) [3] and the many objective qap (maqap) are versions of the problem where a distance matrix and several flow matrices (objectives) are used, with two or more functions configuring the mqap and with more than three functions the maqap. the greater the number of objective functions to be optimized, the amount of non-dominated solutions and the computational cost for calculating the fitness of the solutions and the operations of the algorithm increases [4]. these versions are used to solve disposition problems, for example, the allocation of clinics and departments in hospitals to reduce the distance between doctors, patients, nurses and medical equipment [1]. nsga-iii [5] is an evolutionary algorithm for many objective problems that uses a non-dominance classification approach based on reference points, being a hybrid strategy. different classifications are used according to the method that the problem is solved [4]. such an approach has been showing good results in the literature of the area, so it will be investigated in this work. hyper heuristics (hhs) [6] are high-level methods that generate or select heuristics, used to free the user from choosing a heuristic or its configuration. the thompson sampling (ts) [7] approach is a hh that selects a heuristic based on information accumulated during the execution of the algorithm to improve their performance. in this work, a study was carried out on the methods already used to solve maqap, implementing and testing the nsga-iii with the thompson sampling, which makes the selection of the crossover operator combined with a mutation operator. we tested different versions by modifying the way of thompson sampling action. ts uses success and failure factors to determine the best operator choices to apply. different ways of handling these factors and the frequency of resetting them are proposed. the changes of success and failures factors are based on the dominance relation and the average of the objectives. in addition to the comparison between the previously mentioned versions, the proposal, named nsga-iii\(_{ts}\), is also evaluated in terms of effectiveness by comparison with the original algorithm that does not use hh. finally, the performance of the nsga-iii\(_{ts}\) is compared with five other evolutionary multi-objective algorithms: moea/dd [8], moea/d [9], spea2 [10], nsga-ii [11] and moea/d-dra [12]. the main contribution of this work is to analyze the behavior of a many objective algorithm based on hyper-heuristic applied to maqap. maqap is an under-explored problem in the literature of the area, which motivated its choice for use in this work due to several fronts of scientific contribution. in addition, to analyze how the choice of operators proceeds when important factors related to the hh used (ts) are modified. this paper is organized as follows. section 2 presents concepts related to work, such as the qap, the used algorithm (nsga-iii) and the hhs. the section ends with some related works. section 3 presents the proposed methodology for the development of the work. the results are shown and discussed in sect. 4. section 5 ends the work with the conclusions and future directions. 2 background 2.1 many-objective quadratic assignment problem maqap qap’s objective is to associate a set of n facilities to n different locations, in order to minimize the cost of transporting the flow between the facilities [2]. over time the problem has been adapted to solve different real-world problems [13]. one of its existing variations is the qap with many objectives, which is an extension of the multi-objective qap, but which more than 3 flow matrices. the maqap is formalized by the eq. 1, with the condition of eq. 2 being satisfied. \(\pi \) is a solution that must belong to a set of feasible solutions. $$\begin{aligned} \begin{array}{l} minimize \ \overrightarrow{c}(\pi ) = \{c^1(\pi ), c^2(\pi ), \dots , c^m(\pi )\}\\ \ \pi \in \text {p}(n) \\ \end{array} \end{aligned}$$ (1) $$\begin{aligned} c^{k}(\pi ) = \sum _{i=1}^{n}\sum _{j=1}^{n}a_{ij}b_{\pi _{i}\pi _{j}}^{k}, k \in 1..m \end{aligned}$$ (2) the calculation of each objective is shown in eq. 2, and two matrices of the same size are used, the matrix a\(_{ij}\) has the distance between the locations, b\(_{\pi _i\pi _j}^{k}\) is a matrix of the k-th objective flow, with \(\pi _i\) and \(\pi _j\) being the position of i and j in \(\pi \); n is the number of locations and k is an iterator. the objective is to find a permutation \(\pi \), belonging to the set of all possible permutations (p(n)), which simultaneously minimizes m objective functions each represented by the k-th c. in order to represent problems with many objectives, the constraint \(m>3\) must be added. 2.2 hyper-heuristics (hh) hyper heuristics are high-level methodologies developed for the optimization of complex problems [6], looking for the best option to fulfill your objective by generating heuristics through components (generation hh) or by selecting from a set of low-level heuristics (selection hh). the basic composition of a selection hh is a high-level strategy, which uses a mechanism to decide which low-level heuristic to use, and an acceptance criterion that can accept or reject a solution. the other element of hh is a set of low-level heuristics, which must be appropriate for the tested problem [14]. low-level heuristics work on the set of solutions, they can be of two types, constructive or perturbative. among the ways that the hhs can learn are: online, where the learning takes place during the execution of the search process, or offline where the learning already comes with the rules defined before the execution of the algorithm. the use of hhs is due to their ability to find good results for np-hard problems, seeking the best way to achieve a good result and removing the need for the user to choose a single heuristic or its configuration [15]. but as the hhs have evolved, updated information is needed to better organize the classifications to reflect the challenges found in the current world, [16] presents an extended version of the categorization of selection heuristics. in the fig. 1, a diagram showing the elements of the hh used in this work is shown. in which, the high level strategy is formed by the ts as a heuristic selection mechanism, and the nsga-iii acceptance criterion is used. the low and high level heuristics are separated because they work in different search spaces, the high level ones works searching in the space of low level heuristics, and the low level ones work in the space of solutions. the heuristic selection mechanism chooses within a predefined set of low-level heuristics based on its own evaluation criteria. in this case, thompson sampling uses the success and failure factors associated to each operator who performs a count using different methods to define whether a low-level heuristic was efficient or not. finally, the solution is sent to the acceptance criterion, which can choose whether or not to accept the new solution (nsga-iii acceptance criterion). fig. 1.adapted from [14]. hyper-heuristic elements. full size image low-level heuristics genetic operators: the set of genetic operators is from the permutational type, so they don’t generate infactible solutions. these operators use two solutions, to create two new solutions, and they are: i) partially matched crossover (pmx) [17]: 2 cutoff points are selected, so both parents are divided into 3 parts. when creating an offspring, the elements between the points are copied in the same absolute position, then the elements of the other parent are taken. if they have not been used, they are copied in the same position, and if there is a conflict, they are exchanged according to their position in the offspring. the parents order to raise a second offspring is reversed. ii) cyclic crossover (cx) [18]: sequences of connected positions are created by the values and positions. it starts at the first value on the left, takes the position at parent 2, checks and looks for the value at parent 1 and repeats the process until it finds cycles. each cycle is passed alternately to the offspring. iii) permutational two points crossover (2p) [17]: two points are chosen at random, separating the individual into three parts. the initial and final parts of a parent are passed on to the offspring. then, the parent is changed and the offspring receive values according to the order in the parent of the missing elements. in this work the swap mutation [19] was used, then two positions of the solution are chosen at random and they are exchanged with each other. high level heuristic thompson sampling: thompson sampling was proposed by [7] and is used for online decision problems, where information is accumulated during the execution of the algorithm to performance optimization. as shown in [20], ts can incorporate a bayesian model to represent the uncertainty associated with decision making on k operators (actions). when used, a k operator produces, in the context of bernoulli, a reward of 1 with probability \(\theta _k\) and a reward of 0 with probability \(1-\theta _k\). in the present work, for each k action, the previous probability density function of \(\theta _k\) is given by a \(\beta \)-distribution with parameters \(\boldsymbol{\alpha } = (\alpha _1, ..., \alpha _k)\) e \(\boldsymbol{\beta } = (\beta _1, ..., \beta _k)\): $$\begin{aligned} p(\theta _k)= \frac{{\vargamma \left( {\alpha _k + \beta _k } \right) }}{{\vargamma \left( \alpha _k \right) \vargamma \left( \beta _k \right) }}{\theta _k^{\alpha _k 1}}{\left( {1 \theta _k} \right) ^{\beta _k 1}} \end{aligned}$$ (3) with \(\vargamma \) denoting the gamma function. as the observations are collected, the distribution is updated according to the bayes rule and the \(\beta \)-prior conjugate. this means that \(\alpha _k\) or \(\beta _k\) increases by one with each success or failure observed, respectively. here \(p_{op}(g) = \theta _k\) (operator probability in generation g) and the operator chosen is the one with the highest value of \(p_{op}(g)\). the success estimate of the probability is sampled randomly from the later distribution. 2.3 nsga-iii nsga-iii [5] arose from adaptations of nsga-ii [11] which is used for multiobjective problems, in order to better address problems with many objectives. nsga-iii is an algorithm based on pareto dominance and decomposition concepts. pareto dominance happens when an individual has at least one of the objectives better than the others and all the others must be equivalent or better [21]. as decomposition-based algorithms, the nsga-iii uses an adaptive reference point mechanism, when the population for the next generation is formed, to preserve the diversity and convergence of the algorithm. any structure of reference points can be used, but in this work, as well as in many decomposition based algorithms, is considered the technique applied in [5]. in this technique, in order to avoid generating a very large number of reference points, the reference points are generated twice and then joined. 2.4 related works when analyzing the papers on qap [1], mqap [22,23,24] and maqap [25, 26], it is possible to observe that with the increasing objectives and, consequently, on the problem complexity, there is less research being carried out on the theme. in researching different sources of scientific work, few correlated works are found for the qap with many objectives. in [25] the pareto stochastic local search is used. the authors compare different versions using algorithms that make use of the cartesian product of scaling functions to reduce the number of objectives in the search space. the benchmark considered for testing has problems with 3 objectives, and the quality indicator used was the hypervolume. in [26] a comparison research was carried out between 18 state-of-art evolutionary algorithm to solve qap with many objectives. the instances used in the tests have 3, 5 and 10 objectives. the performance metric used was igd (inverted generational distance). in the work [27] two frameworks for multi-objective optimization, moea/d and nsga-ii, were hybridized with transgenetic algorithms for the multiand many-objective qap solution. the benchmark instances considered 2, 3, 5, 7 and 10 objectives. the results were analyzed by different quality indicators of the pareto frontier and statistical tests. the work shows the performance gain of the inclusion of operators based on computational transgenetics in the analyzed frameworks. the many objective hhs have already been used for some problems in the literature [28,29,30,31,32], especially for continuous optimization problems with benchmark instances, although they are still under-explored. however, there is a gap for maqap, where the use of hh has not yet been explored. this is one of the contributions that we intend to obtain with this work. 3 proposed approach the algorithm proposed in this work, named nsga-iii\(_{ts}\), is presented in algorithm 1, in which the functions marked in gray are part of the ts-based hh. the nsga-iii\(_{ts}\) algorithm starts with the random generation of the initial population, of size np, and the evaluation of its solutions. the np reference points (z) are initialized as an uniformly distributed hyperplane intersecting the axis of each objective (step 1). then, a main loop (steps 3–27) begins , each iteration is a new generation (g). internally to the main repetition, another loop (steps 5–12) is used to create a new population who starts by selecting two solutions (parents), carried out at random (step 6). the thompson sampling chooses among three different crossover operators (step 7), the pmx, the cx or the 2p, combined with the swap mutation, using the beta distribution. after the application of the chosen crossover and, followed by the swap mutation (step 8), the new solutions are evaluated (step 9) and inserted in a np size population formed only by the offspring. the next action of the algorithm (step 10) is related to the different versions of the algorithm which have been created. this step returns the obtained reward for each operator, after applying the swap mutation. when the population of offspring is full, respectively in steps 13 and 14, an union with the population of the parents is made, and the joined population is organized by the non-dominance ordering method. the first set of solutions to be part of next population is formed by the fronts of non-dominance that fully fit into the new population (steps 16–19). if individuals are missing, the algorithm normalizes the objectives (step 23) to determine the ideal and extreme points and adjust the reference points accordingly. the projected distance of each solution on the lines formed by the ideal and reference points are used to associate each solution with a reference point (step 24) and a niche procedure is performed to select the solutions remaining, prioritizing clusters with fewer solutions (step 25). unused solutions are discarded. the algorithm ends when it reaches the maximum number of evaluations performed. the different versions tested in step 10 represent contributions of this work. what differentiates one version from another is the success and failure factor and the frequency of resetting successes and failures count, used by ts. the success and failure factor concerns the decision on which operator applications will contribute to the update of the ts counters (success \(\alpha _k\) and failure \(\beta _k\) in eq. 3). the nsga-iii\(_{ts}\)’s versions that change the success and failures factors are based on the dominance relation (nd) and the average of the objectives (avg) and work as follows: i) nd: after applying the crossover, a two-phase method is used. in the first, parents and offspring are ordered by level of non-dominance. if all parents and offspring are at the same level, the best average of the objectives is used as success. on the other hand, if at least one offspring succeeds in dominating a parent, it is considered success, otherwise it is failure; ii) avg: after applying the crossover, the mean of the objectives is used. comparing the average of the parents with that of the offspring, if there is improvement it is considered success, otherwise failure; the different ways of treating the success and failure counting of operators are presented below: i) v1: the success and failure counter variables are not reset at any point in the search; ii) v2: the variables success and failure counters are reset when creating a new population, that is, at each iteration; iii) v3: the variables success and failure counters are reset during the execution, at the moment when they reach 1/3 and 2/3 of the evaluations performed. 4 experiments and discussion for the tests, 42 maqap instances were used with 50 locations and 5, 7 and 10 objectives (flows), these instances were proposed in [27] using the generators presented in [33]. each instance is named following the format: mqapx-yfl-tw, with x is the number of locations, y is the number of objectives (or flow types), t means that the correlation is positive (p) or negative (n) and w shows whether the problem is uniform (uni) or real-like (rl) [27]. all experiments are evaluated by the igd+ (inverted generational distance plus) quality indicator [34]. this metric evaluates different properties of a pareto front approximation and provides a single performance value for the set. igd+ indicates the distance from the set found by the algorithm in relation to a reference set. the reference set used in this work is composed of the non-dominated solutions resulting from the union of the approximation front of all the algorithms involved in the comparison. table 1. parameters considered in the proposed and in the literature approaches.full size table the experiments were organized in four stages. firstly, the different strategies for resetting the count of successes and failures are compared with each other, for the two success and failure factors: by the dominance relation criterion (nd) and by the average criterion (avg). next, considering the best counting strategies, avg and nd are confronted. in the third stage, the best version of the proposed algorithm is compared with the separately applied operators. in the last stage, the best proposed hh is compared with some approaches from literature. the proposed approaches, as well as the other considered algorithms, are executed 30 times with different seeds. the parameters for all experiments in this study are in table 1. ts does not have any input parameters to be defined, which represents a positive point for the technique. all algorithms, quality indicator and statistical test were developed in jmetal framework [35]. tables 2, 3, 4, 5 and 6 show a comparison between the algorithms, involving all instances for 5, 7 and 10 objectives, using the test statistical analysis of friedman test [36]. the tables show the rank of the algorithms and the hypothesis that the first place approach is equivalent to the others. whenever the hypothesis is rejected, the first ranked approach is statistically superior to those in the rejected hypothesis line. 4.1 stage 1: selecting the success and failure counting strategy the number of succeed and fail operator applications is the main element for the operation of the thompson sampling heuristic. therefore, experiments are presented below for the analysis of different strategies for resetting these counts throughout the search process. this analysis is performed for the two proposed success factor dominance relation (nd) and the average of the objectives (avg), detailed in sect. 3. table 2. ranking and average percentage usage of operators comparison between the three ways of restarting the counting of successes and failures (v1, v2 and v3), for success factor based on the dominance criterion (nd), according to friedman’s test.full size table table 2 analyzes the different ways of counting on the nd success factor (algorithm column) and shows the friedman test on the igd+ indicator (ranking and hypothesis columns), considering all 42 instances for all number of objectives. the percentage of usage of the different operators used is also shown (% pmx, % 2p, % cx columns. from the table it is possible to observe that the nsga-iii\(_{ts-nd}\)v3, which resets the count in two moments of the search process, is the best algorithm, followed by nsga-iii\(_{ts-nd}\)v1 (version without reset). the algorithm that resets the success and failure counts at each iteration (nsga-iii\(_{ts-nd}\)v2) has the worst performance. this table also shows the average percentage usage of operators throughout the search process. it is worth noting that the algorithms that selected the cx operator more often have better performance. in addition, the version that resets success and failure counts at each iteration tends to distribute the use of operators more evenly, as well as in a random selection. the same analysis is performed in table 3, considering the success and failure factors by the average of the objectives (avg). according to the igd+ quality indicator and the friedman test, the nsga-iii\(_{ts-avg}\)v1, which does not reset the count during the search, is the best algorithm, while the nsga-iii\(_{ts-avg}\)v3 algorithm got second place. again, the algorithm that resets the success and failure counts at each iteration (nsga-iii\(_{ts-avg}\)v2) has the worst performance. the two algorithms that have the highest choice frequency for the cx operator have the best performance. in view of these analyzes, it is possible to observe that the best count reset strategy for the nd success and failure factors is that which reset the variables in 1/3 and 2/3 of the maximum number of performed evaluations (v3). as for the avg success and failure factors, is the best strategy that accumulates the scores throughout all evaluations, without resetting them at any time (v1). the next section compares the two success and failure factor (nd and avg), taking into account your best success and failure count reset strategies, nsga-iii\(_{ts-avg}\)v1 and nsga-iii\(_{ts-nd}\)v3. in order to understand how the different forms of counting and factors of success and failure behave in the operators choice, the frequencies of use for each operator during the search were analyzed. figures 2 and 3 show how often each of the three available operators is selected by hh during the search, taking as example three instances, they are: \(mqap50-5fl-n25uni\), \(mqap50-7fl-p00rl\), \(mqap50-10fl-n75rl\). the choice of instances aimed to vary their characteristics such as the number of objectives, the correlation and the form of generation (rl or uni). these figures show the percentage of use of the operator (y axis) over the generations (x axis), considering the average of 5 independent executions. each point on the curve represents the frequency with which each operator is used in a given generation. figure 2 shows, for the avg success and failure factor, that the version with the best performance (nsga-iii\(_{ts-avg}\)v1) selects the cx more frequently than the others operators. however, all three restart strategies favor the choice of the cx operator. v2, which resets the counts at each iteration, presents an oscillation in the choice of operators throughout the search. v3, which resets the count in two moments, behaves similarly to the version that does not reset the counts (v1). considering nd, fig. 3 also shows that the version with the best performance (nsga-iii\(_{ts-nd}\)v3) selects the cx operator more frequently than the others. the oscillation in the choice of operators in the version that reinitializes the count at each iteration (v2) is even greater in this success and failure factors (nd). in order to make a fair comparison, the same stopping criterion was applied in all experiments (maxev = 300000). thus, as can be seen in the figures, the number of generations for each instance is equal to 2832 for 5 objectives, 1022 for 7 objectives and 1088 for 10 objectives. table 3. ranking and average percentage usage of operators comparison between the three ways of restarting the counting of successes and failures (v1, v2 and v3), for success factor based on the objectives average (avg), according to friedman’s test.full size table fig. 2. operators usage over the generations for the nsga-iii\( _{ts-avg}\)v1 approaches (left), nsga-iii\(_{ts-avg}\)v2 (in the middle) and nsga-iii\(_{ts-avg}\)v3 (right), for 5, 7 and 10 objectives. full size image 4.2 stage 2: selecting the success and failure factor the tests carried out in this stage answer the research question related to which the most appropriate success and failure factor in the proposed scenario. it is based on the best strategy selected in the previous simulation stage, that is, performs the comparison of the nsga-iii\(_{ts-avg}\)v1 and nsga-iii\(_{ts-nd}\)v3 algorithms. friedman’s test, in table 4, makes clear the supremacy of the version that considers the average of the objectives when deciding whether the application of a given operator is a success or a failure (avg). nsga-iii\(_{ts-avg}\)v1 represents the best hh version proposed in the scope of this work and will be compared with the operators applied in isolation in the next section. table 4. average ranking of algorithms considering the comparison between nsga-iii\(_{ts-avg}\)v1 and nsga-iii\(_{ts-nd}\)v3, according to friedman test.full size table fig. 3. operators usage over the generations for the nsga-iii\( _{ts-nd}\)v1 (left), nsga-iii\(_{ts-nd}\)v2 (in the middle) and nsga-iii\(_{ts-nd}\)v3 (right), for 5, 7 and 10 objectives. full size image 4.3 stage 3: effect of operator selection by hyper-heuristic the use of a hyper-heuristic in the selection of different operators throughout the search process releases the user from choosing the best operator to be applied in the solution of a given problem. this avoids the necessary simulations and even a more in-depth knowledge of the technique and the problem in question. the simulations of this stage aim to analyze the performance of the automatic selection made by thompson sampling in relation to the performance of the operators applied in isolation. table 5 shows that the performance of hh and the cx operator are similar to each other, while the pmx and 2p operators perform worse, considering the igd + quality indicator. friedman’s test indicates statistical equivalence between nsga-iii\(_{ts-avg}\)v1 and nsga-iii\(_{cx}\), which shows that the automation of the choice of operators is done properly. table 5. average ranking of algorithms considering comparison between the best hh (nsga-iii\(_{ts-avg}\)v1) and operators applied in isolation, according to friedman’s test.full size table 4.4 stage 4: literature comparison the efficiency of the best proposed hh is measured at this stage in the face of the algorithms reported in the literature. table 6 compares the nsga-iii\(_{ts-avg}\)v1 algorithm with the moea/dd [8], moea/d [9], spea2 [10], nsga-ii [11], moea/d-dra [12] algorithms. all comparison approaches are present in the framework jmetal version 4.5 and the parameters used, with the exception of those presented in table 1, are the standard parameters of the framework for each of these algorithms. according to the friedman test, presented in table 6, the hypothesis that the algorithm with the lowest rank (nsga-iii\(_{ts-avg}\)v1) is equivalent to the others is rejected for all algorithms. therefore, the proposed algorithm can be seen as a competitive approach for maqap. table 6. average ranking of the algorithms considering a comparison between the best hh (nsga-iii\(_{ts-avg}\)v1) and literature algorithms, according to friedman’s test.full size table 5 conclusions in this work, it was proposed an approach combining selection hh with the nsga-iii framework. the selection hh considered was thompson sampling. the hh was used to generate each offspring through the selected operator from a set of llh, according to an updated probability based on their previous performance. three candidates for operators were considered: pmx, 2p and cx. a set of 42 benchmarks instances was considered with 5, 7 and 10 objectives. two success and failure factors have been proposed with different ways of resetting the counters. the results were analyzed with the igd+ quality indicator and friedman’s statistical test. the igd+ points out that the best way to restart the success and failure counts varies according to the success and failure factor adopted. the best proposed hh (nsga-iii\(_{ts-avg}\)v1) presented equivalent, or even better, performance to operators applied in isolation. this shows the benefit of automatic selection made by ts, as it releases the user from determining the best operator to be applied in the problem solution. in comparison with the literature, the nsga-iii\(_{ts-avg}\)v1 outperforms all the considered algorithms. it should be noted that, in addition to its competitive results, thompson sampling is a simple technique in relation to other high-level heuristics implemented for hhs and without parameters to be adjusted. as future work we can consider the adaptive control of the size of the set of operators and the test in different optimization problems with multiple and many objectives. references abdel-basset, m., manogaran, g., rashad, h., zaied, a.n.: a comprehensive review of quadratic assignment problem: variants, hybrids and applications. in: journal of ambient intelligence and humanized computing, pp. 1–24 (2018) google scholar  koopmans, t.c., beckmann, m.: assignment problems and the location of economic activities. econometrica 25(1), 53–76 (1957) article  mathscinet  google scholar  uluel, m., ozturk, z.k.: solution approaches to multiobjective quadratic assignment problems. in: european conference on operational research (2016) google scholar  fritsche, g.: the cooperation of multi-objective evolutionary algorithms for many-objective optimization. ph.d. dissertation, universidade federal do paraná, curitiba, pr, br (2020) google scholar  deb, k., jain, h.: an evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. ieee trans. evol. comput. 18(4), 577–601 (2014) article  google scholar  burke, e., et al.: hyper-heuristics: a survey of the state of the art. j. oper. res. soc. 64, 1695–1724 (2013) google scholar  thompson, w.r.: on the likelihood that one unknown probability exceeds another in view of the evidence of two samples. biometrika 25(3–4), 285–294 (1933) google scholar  li, k., deb, k., zhang, q., kwong, s.: an evolutionary many-objective optimization algorithm based on dominance and decomposition. ieee trans. evol. comput. 19(5), 694–716 (2015) article  google scholar  zhang, q., li, h.: moea/d: a multiobjective evolutionary algorithm based on decomposition. ieee trans. evol. comput. 11(6), 712–731 (2007) article  google scholar  zitzler, e., laumanns, m., thiele, l.: spea2: improving the strength pareto evolutionary algorithm for multiobjective optimization, vol. 3242 (2001) google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  google scholar  zhang, q., liu, w., li, h.: the performance of a new version of moea/d on cec09 unconstrained mop test instances. in: congress on evolutionary computation, pp. 203–208. ieee press (2009) google scholar  çela, e.: the quadratic assignment problem: theory and algorithms, ser. combinatorial optimization. springer, us (2013) google scholar  sabar, n., ayob, m., kendall, g., qu, r.: a dynamic multiarmed bandit-gene expression programming hyper-heuristic for combinatorial optimization problems. ieee trans. cybern. 45, 06 (2014) google scholar  pillay, n., qu, r.: hyper-heuristics: theory and applications, 1st ed. springer nature (2018). https://doi.org/10.1007/978-3-319-96514-7 drake, j.h., kheiri, a., özcan, e., burke, e.k.: recent advances in selection hyper-heuristics. eur. j. oper. res. 285(2), 405–428 (2020) article  mathscinet  google scholar  talbi, e.-g.: metaheuristics: from design to implementation. wiley, hoboken (2009) google scholar  oliver, i., smith, d., holland, j.r.: study of permutation crossover operators on the traveling salesman problem. in: genetic algorithms and their applications: proceedings of the second international conference on genetic algorithms: july 28–31, at the massachusetts institute of technology, cambridge, ma. hillsdale, nj: l. erlhaum associates. (1987) google scholar  larranaga, p., kuijpers, c., murga, r., inza, i., dizdarevic, s.: genetic algorithms for the travelling salesman problem: a review of representations and operators. artif. intell. rev. 13, 129–170 (1999) google scholar  russo, d.j., roy, b.v., kazerouni, a., osband, i., wen, z.: a tutorial on thompson sampling. found. trends mach. learn. 11, 1–96 (2018) article  google scholar  zitzler, e., laumanns, m., bleuler, s.: a tutorial on evolutionary multiobjective optimization. in: gandibleux, x., sevaux, m., sörensen, k., t’kindt, v. (eds.) metaheuristics for multiobjective optimisation, springer, berlin heidelberg, pp. 3–37 (2004). https://doi.org/10.1007/978-3-642-17144-4_1 dokeroglu, t., cosar, a.: a novel multistart hyper-heuristic algorithm on the grid for the quadratic assignment problem. eng. appl. artif. intell. 52, 10–25 (2016) article  google scholar  dokeroglu, t., cosar, a.: a novel multistart hyper-heuristic algorithm on the grid for the quadratic assignment problem. eng. appl. artif. intell. 52, 10–25 (2016) google scholar  senzaki, b.n.k., venske, s.m., almeida, c.p.: multi-objective quadratic assignment problem: an approach using a hyper-heuristic based on the choice function. in: cerri, r., prati, r.c. (eds.) intelligent systems, pp. 136–150. springer international publishing, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_10 chapter  google scholar  drugan, m.m.: stochastic pareto local search for many objective quadratic assignment problem instances. in: 2015 ieee congress on evolutionary computation (cec), pp. 1754–1761, may 2015 google scholar  rahimi, i., gandomi, a.: evolutionary many-objective algorithms for combinatorial optimization problems: a comparative study. arch. comput. methods eng. 28, 03 (2020) mathscinet  google scholar  de almeida, c.p.: transgenética computacional aplicada a problemas de otimização combinatória com múltiplos objetivos. ph.d. dissertation, federal university of technology paraná, brazil (2013) google scholar  castro, o.r., pozo, a.: a mopso based on hyper-heuristic to optimize many-objective problems. in: 2014 ieee symposium on swarm intelligence, pp. 1–8 (2014) google scholar  walker, d.j., keedwell, e.: towards many-objective optimisation with hyper-heuristics: identifying good heuristics with indicators. in: handl, j., hart, e., lewis, p.r., lópez-ibáñez, m., ochoa, g., paechter, b. (eds.) ppsn 2016. lncs, vol. 9921, pp. 493–502. springer, cham (2016). https://doi.org/10.1007/978-3-319-45823-6_46 chapter  google scholar  kuk, j., gonçalves, r., almeida, c., venske, s., pozo, a.: a new adaptive operator selection for nsga-iii applied to cec 2018 many-objective benchmark. in: 2018 7th brazilian conference on intelligent systems (bracis), pp. 7–12 (2018) google scholar  fritsche, g., pozo, a.: the analysis of a cooperative hyper-heuristic on a constrained real-world many-objective continuous problem. in: 2020 ieee congress on evolutionary computation (cec), pp. 1–8 (2020) google scholar  friesche, g., pozo, a.: cooperative based hyper-heuristic for many-objective optimization. in: genetic and evolutionary computation conference, ser. gecco ’19. new york, ny, usa: association for computing machinery, pp. 550–558 (2019) google scholar  knowles, j., corne, d.: instance generators and test suites for the multiobjective quadratic assignment problem. in: fonseca, c.m., fleming, p.j., zitzler, e., thiele, l., deb, k. (eds.) emo 2003. lncs, vol. 2632, pp. 295–310. springer, heidelberg (2003). https://doi.org/10.1007/3-540-36970-8_21 chapter  google scholar  ishibuchi, h., masuda, h., tanigaki, y., nojima, y.: modified distance calculation in generational distance and inverted generational distance. in: gaspar-cunha, a., henggeler antunes, c., coello, c.c. (eds.) emo 2015. lncs, vol. 9019, pp. 110–125. springer, cham (2015). https://doi.org/10.1007/978-3-319-15892-1_8 chapter  google scholar  nebro, a.j., durillo, j.j., vergne, m.: redesigning the jmetal multi-objective optimization framework. in: conference on genetic and evolutionary computation, ser. gecco companion ’15. new york, ny, usa: association for computing machinery, pp. 1093–100 (2015) google scholar  conover, w.: practical nonparametric statistics, 3rd ed., ser. wiley series in probability and statistics. new york, ny [u.a.]: wiley (1999) google scholar  download references author information authors and affiliations midwestern paraná state university unicentro, paraná, brazil bianca n. k. senzaki, sandra m. venske & carolina p. almeida authors bianca n. k. senzakiview author publications search author on:pubmed google scholar sandra m. venskeview author publications search author on:pubmed google scholar carolina p. almeidaview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper senzaki, b.n.k., venske, s.m., almeida, c.p. (2021). hyper-heuristic based nsga-iii for the many-objective quadratic assignment problem. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_12 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords nsga-iii hyper-heuristic thompson sampling maqap publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature impact of pre-training datasets on human activity recognition with contrastive predictive coding | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper impact of pre-training datasets on human activity recognition with contrastive predictive coding conference paper first online: 30 january 2025 pp 306–320 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) impact of pre-training datasets on human activity recognition with contrastive predictive coding download book pdf download book epub betania e. r. da silva9, otávio o. napoli9, j. v. delgado9, anderson r. rocha9, levy boccato9 & … edson borin9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 391 accesses abstract self-supervised learning (ssl) techniques have been successfully employed to learn useful representations for various data modalities without labels. these techniques use a pretext task to train the backbone of a deep-learning model without labels and then leverage the pre-trained backbone to train a downstream model with a few labeled samples. in this context, contrastive predictive coding (cpc) is an ssl technique that has demonstrated promising results in several tasks, including human activity recognition (har). in this work, we explore the impact of data variety on backbone pre-training when designing cpc models for har and the benefits of pre-training on the final model. we evaluated the impact of data variety on model pre-training using fifteen combinations of four distinct har datasets, finding significant performance variability based on the pre-training datasets, with \(f_1\)-score varying from 9.6 to 13% points across different target datasets. we also found that including the target dataset in the pre-training process generally improved performance and that pre-training with all four datasets produced a high-quality backbone, yielding downstream models performing near the best on all target datasets. these findings emphasize the importance of selecting pre-training datasets aligned with the downstream task domain. additionally, we demonstrated that cpc pre-training significantly benefits downstream model performance with limited data, achieving comparable \(f_1\)-scores with just 5% of the data as with 100%, indicating that cpc effectively captures essential features of the problem domain. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others personalized human activity recognition with transfer learning chapter © 2026 smaller can be better: efficient data selection for pre-training models chapter © 2024 standardizing your training process for human activity recognition models – a comprehensive review in the tunable factors chapter © 2024 1 introduction self-supervised learning (ssl) has emerged as a powerful approach for extracting meaningful representations from unlabeled data. it leverages pretext tasks to learn useful features without the need for manual annotations. this paradigm not only expands the utility of available data but also allows for the creation of models that can be efficiently tuned to specific tasks with minimal effort [1, 4]. within ssl, contrastive predictive coding (cpc) [12] is an effective technique for dealing with temporal data and have demonstrated promising results in time series, especially for human activity recognition (har) [3, 6] for example, haresamudram et al.  [6] reported an \(f_1\)-score of 0.832 for the uci dataset and 0.894 for motionsense, reaching performance levels close to those attained by state-of-the-art deep models trained in a fully-supervised manner. although there is a vast amount of literature on ssl methods for har, few works explore the impact of pre-training datasets on the quality of representations learned by ssl models. this work aims to explore this gap by evaluating representations generated by cpc when trained with 15 different combinations of four publicly available har datasets kuhar, uci, motionsense, and real world (waist). we quantitatively assessed performance using the \(f_1\)-score metric and conducted a qualitative analysis of data distribution using t-sne [8]. furthermore, we extended our analysis to low-data regimes by fine-tuning the model on the downstream task with varying percentages of labeled samples. the main contributions of this work are: demonstrated that data variety during pre-training can significantly impact performance; showed that including the target dataset in pre-training improves performance; and proved that pre-training with all datasets produces a high-quality backbone, leading to downstream models that perform near the best across all tasks. additionally, we showed that cpc pre-training provides substantial benefits in low-data regimes, achieving results comparable to using 100% of the data with only 5% of labeled samples. the remainder of this paper is organized as follows: sect. 2 discusses the related work. section 3 introduces the key ssl terms used in this work in order to provides an overview of the cpc technique and its components. section 4 presents the experimental results. finally, sect. 5 summarizes the main conclusions. 2 related work self-supervised learning (ssl) techniques aim to train a robust feature extractor, known as the backbone, which can later be combined with a prediction head (e.g., a classifier, regressor, etc..) to solve a target task (e.g., har) [1, 4]. in essence, this process involves two phases: pre-training and fine-tuning. in pre-training, a machine learning model is trained to solve a pretext task, which is a task that does not require labels, but is designed to boost the model to learn useful features, for instance, predicting the missing parts of an image. the model, also known as “pretext model”, consists of a backbone followed by a projection head, which is usually a multi-layer perceptron and it is trained using a large amount of unlabeled data. in the second phase, the pre-trained backbonefootnote 1 is combined with a prediction head to form the downstream model, which is trained in a supervised manner to solve the target (or downstream) task, for example, har. this phase usually requires a small amount of labeled data, which is used to fine-tune the model to the target task. recently, ssl has proved to be effective in har, outperforming fully-supervised training schemes. for instance, saeed et al.  [14] showed that, by using a pretext task based on transformation classification, they could improve the discrimination of the model by up to 4%, in comparison to fully-supervised and unsupervised methods (e.g., auto-encoders), in datasets like hhar, unimib, uci-har, wisdm, and motionsense, using raw accelerometer and gyroscope data. haresamudram et al.  [5] proposed a ssl technique using masked auto encoders where a subset of the input data is randomly selected and perturbed. then, the pretext task is to reconstruct the original input. the authors demonstrated that the model learns useful representations and achieves good results in har using the motionsense and usc-had datasets. tang et al.  [16] adapted simclr [2], a popular ssl technique based on contrastive learning, for har. this method uses two augmentations of the same sample to generate positive pairs and different samples to generate negative pairs. the authors reported a performance gain of over 2% using their pre-trained scheme. in a later work [15], the authors proposed selfhar, a teacher-student architecture for human activity recognition that combines semi-supervised and self-supervised learning, reporting a significant performance improvement across six datasets with different device placements. thukral et al.  [17] proposed a teacher-student architecture for har, combining cross-modal self-supervised learning with semi-supervised learning. the teacher model, trained with augmented labeled data, generates pseudo-labels for the student model, which is trained with unlabeled data using the simclr technique. the student model can then be fine-tuned with a small amount of labeled data. their method significantly improves performance, especially on the mobiact dataset, in a few-shot learning scenario. however, they note that techniques like cpc excel in few-shot learning using the hhar dataset for pre-training. among several ssl techniques, a technique that has recently stood out in the treatment of temporal signals is the contrastive predictive coding (cpc), first introduced in 2018 by oord et al.  [12]. cpc aims to learn useful representations from temporal data by predicting future representations of the data. this technique will be further detailed in sect. 3. the authors demonstrated that cpc can learn useful representations, achieving strong performance across four distinct domains: speech, images, text, and reinforcement learning. however, their work does not address the har task nor the impact that pre-training datasets have on the downstream model. in 2021, haresamudram et al.  [6] adapted the cpc technique for har. the authors reported an improvement of up to 9% compared to saeed et al.  [14] in the mobiact dataset and other improvements in motionsense, uci-har, and usc-had datasets. dhekane et al.  [3] investigates the data requirements of cpc for har using the hhar, pamap2, and realworld datasets. they randomly selected data from these datasets for representation learning with minimal annotations, applying data augmentation techniques like noise addition, scaling, and channel shuffling. their work includes two evaluations: “bottom-up evaluation”, determining the minimal amount of target domain data needed for useful representations, and “transferred assessment”, examining the minimal external data required for effective activity recognition in the target domain. results show cpc is highly data-efficient, needing only 15 and 5 min of pre-training data for “bottom-up evaluation” and “transferred assessment”, respectively, to achieve comparable performance to full data pre-training. the authors conclude that target domain data is more beneficial for representation learning. the choice of the dataset for pre-training can significantly impact downstream model performance. qian et al.  [13] examined various components in contrastive-based ssl techniques for har, including backbone architecture, augmentations, and contrastive pair construction. they also analyzed data-scarce cross-person generalization, where models are trained on a small subset of individuals and tested on a left-out individual. results showed that generalization levels vary with the pre-training dataset. the authors suggest that self-supervised methods’ effectiveness can be dataset-dependent and that the choice of backbone architecture is crucial for model performance. table 1. related works summary.full size table as ssl techniques have been shown to be effective in several scenarios, table 1 classify the related works according to the following criteria: cpc: evaluate the contrastive predictive coding ssl technique; har: perform the evaluation on har; multiple pre-training sets: evaluate the impact of pre-training datasets on final model performance, and few-shot learning: evaluate the model when trained with few labeled data. notice that the closest related work is the work of dhekane et al.  [3], which evaluates cpc for har taking into account multiple pre-training sets and evaluating the model when trained with few labeled data. nonetheless, our work differs from theirs in the following aspects: (i) we perform a qualitative assessment of the learned representations using t-sne, (ii) we do not employ data augmentation and (iii) we evaluate the impact of different combinations of har datasets on the quality of the learned representations. 3 contrastive predictive coding (cpc) the cpc technique was first introduced by oord et al.  [12], and later adapted by haresamudram et al.  [6] for har. we rely on the implementation of haresamudram et al. to perform our experiments, which is illustrated in fig. 1. the cpc backbone comprises two main components: the input encoder (\(g_{enc}\)), which encodes the input time series (x) into z, and an auto-regressor (\(g_{ar}\)), which encodes the initial time steps of z (i.e., z[0 : t]) into a representation called \(c_t\). cpc also contains several linear predictors (\(w_1\), ..., \(w_k\)), which are used to predict a subset of z (i.e., \(z[t+1:t+k]\)) from \(c_t\). fig. 1. overview of the cpc model. full size image during the training, a random time step t is chosen, dividing z into two segments: past (elements up to t) and future (elements after t). the time steps in the past segment (i.e., z[0 : t]) are encoded by \(g_{ar}\), generating the context vector \(c_t\). this vector is then used as input to several linear models (\(w_1\), \(w_2\), ... \(w_k\)) to predict the next k time steps in z (i.e., \(z_{t+1}\), ..., and \(z_{t+k-1}\)). in this case, these linear models work as projection heads. cpc optimization aims to maximize the mutual information between \(c_t\) and \(z_{t+k}\), (with \(k>0\)). this is accomplished through the contrastive loss function infonce [12], which increases the similarity between the positive pair (context and correct future representation) and decreases the similarity between negative pairs (context and incorrect future representations). once the model is trained, the projection heads are discarded and the \(g_{enc}\) and \(g_{ar}\) models, i.e., the backbone, is attached to a prediction head to compose the downstream model. in our experiments, we employed the same encoder (\(g_{enc}\)) and auto-regressor (\(g_{ar}\)) as haresamudram et al.  [6]: \(g_{enc}\) is composed of three blocks, each containing a 1d convolutional layer with 32, 64, and 128 channels, kernel size 3, relu activation function, and dropout of 0.2; and \(g_{ar}\) is composed of a two-layer gated recurrent unit (gru) with 256 units. this network uses a dynamic attention mechanism, allowing \(g_{ar}\) to capture subtle and relevant temporal features, even in long sequences, by considering distant samples less relevant. for the downstream model, we used the same prediction head as haresamudram et al., which consists of an mlp with three linear layers. specifically, it includes a linear layer with 256 neurons, batch normalization, relu activation, and a 0.2 dropout. this is followed by another linear layer with 128 neurons, batch normalization, relu activation, and a 0.2 dropout. the final linear layer reduces the dimension to the number of classes. 4 experimental results this section details the experiments conducted to evaluate cpc’s effectiveness for har. section 4.1 presents the materials, while sect. 4.2 details the methods used. section 4.3 shows validation experiments, reproducing literature results to ensure compatibility. section 4.4 evaluates the impact of pre-training datasets on the downstream task for four target scenarios. section 4.5 uses \(t\)-sne  to visualize sample distribution in the latent space and qualitatively analyze cpc’s learned representation. finally, sect. 4.6 assesses the impact of pre-training on downstream model performance with limited data. 4.1 materials datasets: the experiments were conducted with four publicly available datasets for human activity recognition: uci [7] (uci), kuhar [10] (kh), motionsense [9] (ms), and realworld [11] (rw). these datasets consist of time series data collected from smartphone accelerometers and gyroscopes. the smartphones were positioned differently across the datasets: in a waist bag for uci and kh, in a pocket for ms, and at the waist for rw. to explore the effectiveness of cpc in different pre-training scenarios, we generated all 15 combinations of the four aforementioned datasets as follows: using a single dataset, combining datasets two by two, combining three by three, and, finally, combining all four datasets. the uci dataset has 12 classes, 6 that represent continuous activities (e.g., walking), and 6 for transitions (e.g., sit to stand). for the pretext task, we employ all the samples; however, for the downstream task, only samples belonging to the continuous activities are used. this is performed to ensure we do not have to discard too much data when balancing the dataset for the supervised training. we distinguish these subsets by the acronyms uci-12 and uci-6, respectively. the other datasets, i.e., kuhar, motionsense, and realworld, remained the same in the downstream and pretext tasks. table 2 summarizes the datasets’ characteristics. table 2. overview of datasets used in this work.full size table to standardize and combine the datasets, several preprocessing steps were undertaken. first, we standardized the sampling rate to 50 hz and converted the acceleration data to units of \(\frac{m}{{s}^{2}}\). the dataset was then partitioned based on user ids, with 70% of users allocated to the training set, 10% to the validation set, and 20% to the test set, ensuring that no samples from a single user appeared in multiple subsets. finally, the data was segmented into windows of 50 time steps (1 s) with a 50% overlap. code: we adapted the code made available by haresamudram et al. [6]footnote 2. the modified version of the code can be found at our github repositoryfootnote 3. hyperparameters: we also employ the same experimental setup described by haresamudram et al.  [6] to ensure reproducibility and comparability of results. table 3 lists the hyperparameters used in this work. table 3. hyperparameters used in the experiments for training the pretext and downstream tasks.full size table 4.2 methods methodology: the cpc model, as described in sect. 3, is first pre-trained with the unlabeled combinations of datasets, adjusting the parameters of the backbone, which is composed of the concatenation of the \(g_{enc}\) and \(g_{ar}\) models. then, the downstream model is built by concatenating the pre-trained backbone to an untrained prediction head (an mlp with three linear layers), and trained using the training subset of the target dataset. however, in this process, the backbone weights are frozen and only the prediction head parameters are adjusted. finally, we evaluate the downstream model on the test subset of the target dataset, reporting the \(f_1\)-score metric. we also employ \(t\)-sne to visualize the distribution of the samples on the latent space, i.e., using the features extracted by the backbone. this is performed using the training subset of the target datasets. 4.3 validation initially, we validated our code base by reproducing state-of-the-art results. this involved verifying the effectiveness of cpc through direct comparison with results reported by haresamudram et al.  [6]. we employed the same hyperparameters, pre-training datasets, and target datasets: uci-6 without the standardization employed in the rest of our work on acceleration unit in g, and motionsense as described in table 2. table 4 shows the \(f_1\)-score reported by haresamudram et al. and the ones produced in our own experiments. the results are very similar, indicating that the reproduction was successful and our implementation is a solid basis for subsequent experiments. table 4. \(f_1\)-score reported by haresamudram et al.  [6] and reproduced by us for the uci and motionsense datasets.full size table 4.4 impact of pre-training datasets on downstream tasks we evaluate the datasets impact used on pre-training have on four target scenarios. the goal is to investigate whether the set of datasets used in the pre-train task significantly affects the performance of the downstream model. to this end, we pre-trained the cpc backbone using all possible combinations of the four original datasets, including training on each dataset individually, in pairs, in trios, and finally, with all datasets combined. the results are organized into tables, one for each target scenario, i.e., a target har dataset in the har task (downstream task). for example, table 5 shows the results when using kuhar as the target dataset. in this case, each row shows which datasets were used on pre-training and the \(f_1\)-score of the downstream model when evaluated on the test subset of the kuhar dataset. tables 6, 7, and 8 exhibit the results when using motionsense, realworld (waist), and uci-6 as target datasets, respectively. the rows in tables 5 through 8 are arranged in descending order based on the \(f_1\)-score. table 5. kuhar results.full size table table 6. motionsense results.full size table table 7. realworld (waist) results.full size table table 8. uci-6 results.full size table the initial observation is that the \(f_1\)-score can vary significantly depending on the datasets used for pre-training: observe that the \(f_1\)-score varies by up to 12.9, 9.6, 10.5, and 13% points on kuhar, motionsense, realworld, and uci-6, respectively. this indicates that the quality of the backbone model is sensitive to the choice of pre-training datasets. we also observed that, in most cases, there is a significant drop in performance when the target dataset is not included in the pre-training set for the backbone. this effect is particularly pronounced in the kuhar and motionsense datasets. for kuhar (table 5), the \(f_1\)-score ranges from 0.770 to 0.789 when kuhar is part of the pre-training set, but it drops to below 0.712 when it is excluded. similarly, in the motionsense dataset, the \(f_1\)-score decreases by three percentage points when the target dataset is omitted from pre-training. a similar trend is observed for uci-6 (table 8), where the highest-performing configurations generally include the uci dataset in the pre-training set. we employed the wilcoxon test to determine whether the performance differences observed when the target dataset is included in the pre-training set are statistically significant. specifically, we compared the seven configurations that exclude the target dataset against the corresponding seven configurations that include it. the wilcoxon test yielded a p-value of 0.015625 for the kuhar, motionsense, and uci-6 datasets, indicating that the observed differences are statistically significant. the results for the realworld dataset deviate from this trend. as shown in table 7, although the best-performing configuration includes realworld in the pre-training set, several other configurations, including the second-best one, achieve high \(f_1\)-score values without it. in this case, the wilcoxon test produced a p-value of 0.296875, which is well above the 5% significance threshold. therefore, we cannot conclude that including the realworld dataset in pre-training led to performance improvements. it is worth noting that the performance of the realworld dataset generally improves when the kuhar dataset is included in the pre-training set. this improvement may be attributed to the fact that both datasets were collected with the device positioned at the waist and that the activity set of realworld is a subset of the kuhar activity set. we plan to explore this relationship further in future work. finally, we observed that the performance obtained when pre-training the backbone using the combination of the four datasets (kh+ms+rw+uci-12) is very close or equal to the best \(f_1\)-score  value achieved in the downstream task for all datasets, as evidenced at table 9. table 9. performance rate when pre-training with all datasets.full size table 4.5 qualitative analysis of the representations in order to perform a qualitative analysis, we used t-distributed stochastic neighbor embedding (\(t\)-sne) [8] to visualize the distributions of the dataset samples on the original representation (i.e., time domain or raw data), and on the representation learned by the backbone when trained with the best combination of datasets. these combinations are indicated in the first row of tables 5 to 8. figure 2a shows the \(t\)-sne chart for two representations of the uci dataset, the original one (uci raw), and the one learned by the cpc backbone (uci + cpc). it is worth noting that the cpc backbone provided better class separation, with most activities forming well-defined clusters. however, there is some overlap between sitting and standing, both low-energy activities, and upstairs shows a few samples mixed with other activities. even so, the overall separability remains clearly visible. fig. 2. \(t\)-sne charts for the uci and the motionsense datasets. full size image figure 2b shows the \(t\)-sne charts for motionsense. again, it indicates that the cpc backbone was able to provide a better separation of the classes, as the sitting and the standing samples are clearly separated from other samples. similarly, the cpc backbone was also able to provide a much better separation of the classes on the kuhar, as illustrated in fig. 3a. the 18 classes were very mixed in the raw data, as indicated in fig. 3a. the cpc backbone yielded a much better separation, creating several homogeneous clusters. finally, the samples from the realworld dataset seem to already form clear activity clusters on the raw data, as illustrated in fig. 3b. in this case, the cpc backbone did not offer much improvement, but at least it kept the same degree of clustering. therefore, we recognize that, in general, the cpc backbone was able to improve the samples clustering or preserve the original clustering seen on the raw data. fig. 3. \(t\)-sne charts for the kuhar and the realworld datasets. full size image 4.6 impact of pre-training on performance with limited data ssl excels when training models on large datasets with minimal labeled data. in this experiment, we assess how the pre-training process affects the performance of the downstream model when trained with limited data, i.e., using fractions of the training set. we evaluate two versions of the downstream model on each dataset: one trained from scratch, without pre-training (named “from scratch”), and the other utilizing backbone weights \(g_{enc}\) (convolutional) + \(g_{ar}\) (gru) learned during pre-training. the pre-training is conducted with cpc technique considering the combination of the four datasets. to evaluate the impact of the amount of labeled data on the downstream models, we train them using subsets of the training dataset. we use the following percentages of the training set: 1%, 5%, 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90%, and 100%. the data are shuffled before selecting the subsets; however, for a fair comparison, we use the same subset when training both downstream models. finally, we assess the models’ performance by reporting the \(f_1\)-score using the test set of each dataset. figure 4 shows the \(f_1\)-score for both models when trained with fractions of the training set for each one of the target datasets. the chart includes a blue dot representing the performance of a well-known model trained with supervised learning on the uci dataset, as described in the literature. this model is a cnn as reported by kun xia et al.  [18], using an architecture based on the work of yang et al.  [19]. fig. 4. performance comparison using different percentages of data (x-axis) for models pre-trained with cpc and trained from scratch using different datasets. full size image the results demonstrate that the pre-training process offers substantial benefits for the downstream model across all datasets. moreover, using the pre-training process, just 5% of the data was sufficient to achieve an \(f_1\)-score comparable to that obtained with 100% of the data. this highlights the effectiveness of pre-training with ssl over a fully-supervised training, particularly in scenarios with limited data. furthermore, it is possible to observe that the downstream model pre-trained with cpc achieved nearly the same \(f_1\)-score as the convolutional network from the literature. 5 conclusions in this work we evaluated how the contrastive predictive coding (cpc) ssl technique performs on har tasks. first, we assessed the impact of data variety on model pre-training using 15 combinations of four distinct har datasets. our evaluation reveals that model performance can vary significantly based on the pre-training datasets, with \(f_1\)-score changes ranging from 9.6 to 13% points across different target datasets (i.e., the ones used on the downstream task). it also shows that, in most cases, there is a noticeable difference in performance when the target dataset is included in the pre-training process. moreover, we observed that using all four datasets during pre-training produced a high-quality backbone, leading to downstream models that perform very close to the best models on all target datasets. these results highlight the critical importance of selecting pre-training datasets aligned with the downstream task domain. we then conducted a qualitative analysis of the backbone pre-trained with all four datasets and found that it generally extracted features that effectively enhanced the clustering of samples by their classes. finally, we assessed the impact of pre-training on the performance of the downstream model when trained with limited data, i.e., using only a fraction of total data used to train, and showed that the pre-training process provided significant benefits across all target datasets. this suggests that cpc enables the backbone to learn subtle and essential features of the problem domain. additionally, with the pre-trained backbone, using just 5% of the data was sufficient to achieve an \(f_1\)-score comparable to that obtained with 100% of the data. for future work, we suggest: (i) investigating how varying the amount of data used during pre-training impacts the learned representation, similar to the approach taken by dhekane et al. [3]; and (ii) exploring whether including datasets with similar characteristics can enhance the performance of the backbone model. all codes are open-source and available at https://github.com/h-iaac/kr-papers-artifacts. the datasets can be made available upon request for the purpose of reproducibility. notes 1.the projection head is usually discarded and it is not used on the downstream model. 2.https://github.com/harkash/contrastive-predictive-coding-for-har. 3.https://github.com/h-iaac/kr-papers-artifacts. references balestriero, r., et al.: a cookbook of self-supervised learning. arxiv:2304.12210 (2023) chen, t., kornblith, s., norouzi, m., hinton, g.: simclr: a simple framework for contrastive learning of visual representations. in: international conference on learning representations, vol. 2 (2020) google scholar  dhekane, s.g., haresamudram, h., thukral, m., plötz, t.: how much unlabeled data is really needed for effective self-supervised human activity recognition? in: proceedings of the 2023 acm international symposium on wearable computers (2023) google scholar  ericsson, l., gouk, h., loy, c.c., hospedales, t.m.: self-supervised representation learning: introduction, advances, and challenges. ieee sig. process. mag. 39(3) (2022) google scholar  haresamudram, h., et al.: masked reconstruction based self-supervision for human activity recognition. in: iswc 2020, pp. 45–49 (2020) google scholar  haresamudram, h., essa, i., plötz, t.: contrastive predictive coding for human activity recognition. in: proceedings of the acm on interactive, mobile, wearable and ubiquitous technologies, vol. 5, no. 2 (2021) google scholar  reyes-ortiz, j.l., anguita, d.: smartphone-based recognition of human activities and postural transitions (2015) google scholar  van der maaten, l., hinton, g.: visualizing data using t-sne. j. mach. learn. res. 9(11) (2008) google scholar  malekzadeh, m.: motionsense dataset : smartphone sensor data har google scholar  nahid, a.a., sikder, n., rafi, i.: ku-har: an open dataset for human activity recognition. mendeley data (2021) google scholar  dataset realworld. universität mannheim, university of mannheim google scholar  oord, a.v.d., li, y., vinyals, o.: representation learning with contrastive predictive coding. arxiv:1807.03748 (2018) qian, h., tian, t., miao, c.: what makes good contrastive learning on small-scale wearable-based tasks? in: zhang, a., rangwala, h. (eds.) kdd ’22: the 28th acm sigkdd and data mining (2022) google scholar  saeed, a., ozcelebi, t., lukkien, j.: multi-task self-supervised learning for human activity detection. proc. acm interact. mob. wearable ubiquit. technol. 3(2) (2019) google scholar  tang, c.i., perez-pozuelo, i., spathis, d., brage, s., wareham, n., mascolo, c.: selfhar: improving human activity recognition through self-training with unlabeled data. 5(1) (2021) google scholar  tang, c.i., perez-pozuelo, i., spathis, d., mascolo, c.: exploring contrastive learning in human activity recognition for healthcare. arxiv:2011.11542 (2020) thukral, m., haresamudram, h., ploetz, t.: cross-domain har: few shot transfer learning for human activity recognition. arxiv:2310.14390 (2023) xia, k., huang, j., wang, h.: lstm-cnn architecture for human activity recognition. ieee access 8 (2020) google scholar  yang, j., nguyen, m.n., san, p.p.: deep convolutional neural networks on multichannel time series for human activity recognition (2015) google scholar  download references acknowledgments this project was supported by the ministry of science, technology, and innovation of brazil, with resources granted by the federal law 8.248 of october 23, 1991, under the ppi-softex [01245.003479/2024-10]. the authors also thank cnpq (315399/2023-6 and 404087/2021-3) and fapesp (2013/08293-7) for their financial support. author information authors and affiliations hub for artificial intelligence and cognitive architectures (h.iaac), university of campinas, campinas, 13083-852, brazil betania e. r. da silva, otávio o. napoli, j. v. delgado, anderson r. rocha, levy boccato & edson borin authors betania e. r. da silvaview author publications search author on:pubmed google scholar otávio o. napoliview author publications search author on:pubmed google scholar j. v. delgadoview author publications search author on:pubmed google scholar anderson r. rochaview author publications search author on:pubmed google scholar levy boccatoview author publications search author on:pubmed google scholar edson borinview author publications search author on:pubmed google scholar corresponding author correspondence to betania e. r. da silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, b.e.r., napoli, o.o., delgado, j.v., rocha, a.r., boccato, l., borin, e. (2025). impact of pre-training datasets on human activity recognition with contrastive predictive coding. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_21 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords contrastive predictive coding deep learning human activity recognition self-supervised learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature automatic text simplification for the legal domain in brazilian portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper automatic text simplification for the legal domain in brazilian portuguese conference paper first online: 31 january 2025 pp 31–45 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) automatic text simplification for the legal domain in brazilian portuguese download book pdf download book epub francielle vasconcellos pereira9, ana frazão10 & viviane p. moreira9  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 231 accesses 1 citation abstract legal and juridical documents such as rulings, laws, agreements, and contracts contain domain-specific terms and jargon, long and complex sentences that may be difficult to understand for laypeople without domain expertise, reading issues, or with a low education level. the simplification of these documents has been a concern for several years, aiming to democratize access to justice. courts are already adopting simpler language, especially in documents aimed at laypeople, such as warrants and notifications, to enhance inclusion and clarity. automatic textual simplification, a subfield of natural language processing, seeks to make complex texts more accessible. this paper explores the task of automatic text simplification in portuguese for the legal domain. the main challenge here is the lack of datasets containing complex sentences and their simplified versions. this work investigates how existing datasets, methods, and metrics used for text simplification perform applied to legal texts in portuguese. we present qualitative and quantitative analyses using five models. the results show that gpt-based models have the best results, but fine-tuning with domain data is a viable open-source alternative. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others text simplification system for legal contract review chapter © 2024 transformer-based approaches for legal text processing article 25 january 2022 ulysses tesemõ: a new large corpus for brazilian legal and governmental domain article 18 july 2024 1 introduction judicial and legislative texts have distinctive characteristics, such as the use of latin and domain-specific language, that make them difficult to understand. the use of judicial language can be found in documents that describe laws, rulings, opinions, and other legal process documents. when it comes to forensic documents, specific terminology is necessary to understand the facts without leaving margin for different interpretations. the concern about simplifying legal has been ongoing for some years. the need for the democratization of the brazilian judiciary, expanding access to justice, has been raised by a few studies [1, 4, 18]. however, they focus on advocating that documents are written and published in a simplified manner. writing simply without losing the entire essence of the text that the law requires is a practice that would need to be widely disseminated and encouraged in all courts in the country. to encourage courts to use plain language, the cnj (conselho nacional de justiça) launched the national pact of the judiciary for plain languagefootnote 1 and, through ordinance no. 351/2023, the plain language seal. its purpose is to recognize and promote, across all segments of the judiciary and at all levels of jurisdiction, the use of clear and understandable language in the drafting of judicial decisions and in general communication with societyfootnote 2. this is a lengthy process that may take years to materialize, still, it can benefit from advancements resulting from the use of automatic tools for text simplification. over time, brazilian portuguese has evolved to exhibit distinct characteristics not typically found in european portuguese. brazil, with its vast territory and cultural diversity, encountered challenges in the development of its legal framework. the incorporation of influences from roman-germanic, french, and portuguese legal traditions occurred within the context of adapting to the specificities of brazilian society. consequently, brazil’s legal system represents a fusion of these legal traditions and an adaptive approach to local issues. these characteristics undoubtedly influence legal writing practices, affecting the formats of documents, hermeneutics, and argumentation in various ways [10, chapter 26]. the challenge addressed in this work is how to automatically simplify texts in the legal domain written in brazilian portuguese. text simplification (ts) is a subfield of natural language processing (nlp). it seeks to translate complex texts into a simpler language, aiming for accessibility. there are several reasons to simplify texts, such as covering the understanding of non-native speakers, people with a lower level of education, or cognitive disabilities. we apply large language models (llm) to generate simplifications given a complex sentence from a legal text. we experimented with different approaches, including fine-tuning a t5 model and prompting strategies. in addition, we enhanced a fine-tuned model with reinforcement learning (rl) to assess whether this could improve simplification results. the evaluation of ts is still an open problem. there is no single metric that can accurately gauge all nuances that are involved in simplifying a text. as a result, this work provides a quantitative analysis using four evaluation metrics. we also performed a qualitative analysis, which is crucial to allow insights into the generated simplifications. the main contributions of this work can be summarized as follows: 1. an evaluation of five llms applied to text simplification in the brazilian legal domain; 2. a quantitative assessment of the quality of the simplification using four evaluation metrics; and 3. a qualitative evaluation of the model’s outputs by a human expert. our findings showed that gpt models still have the best results for ts, even in this specific domain. the use of rl brought performance improvements to the fine-tuned. apart from the gpt models, using only instructions and examples is not enough to yield good results. 2 related work legal documents have complex and specific terms in any language. a compilation of existing datasets, methods, and metrics was investigated for ts in legal texts [11]. they reported the results in terms of readability, simplicity, similarity, presence of hallucinations, and fluency methods. their analysis showed that most methods focus on split-and-rephrase, transforming larger and more complex sentences into shorter and simpler ones. in general, research in languages other than portuguese includes diverse methods, such as machine translation (both supervised and semi-supervised). the first relevant work in ts using a supervised method employed metrics as rewards in an rl algorithm called dress (deep reinforcement sentence simplification) [27]. another work focuses on the replacement of complex words and on sentence splitting. they outperform other unsupervised methods for ts for different domains in english [6]. more recently, bless (benchmarking large language models on sentence simplification) evaluated 44 models for ts in three datasets (wikipedia, news, and medical). they showed that some pre-trained models, even if not trained to ts, can perform well compared to the muss [15] state-of-the-art chosen as a baseline in the paper [13]. there is no consensus on what constitutes plain language, but there are principles that guide the practice of ts. the basic book of plain language theory [8], establishes 25 guidelines capable of guaranteeing the clarity of the text. brazilian courts of justice are working to implement plain language both in textual documents and in-person services provided to citizens. these initiatives aim to ensure that information is passed on clearly and objectively without the need to resort to third parties (such as lawyers). the main targets are documents that reach lay people, such as search warrants and subpoenasfootnote 3. there are few linguistic resources for ts in portuguese [9, 12, 16], and some contributions to nlp and the legal domain made recently [19, 23]. however, in portuguese, most approaches are limited only to lexical simplification, not encompassing the entire syntax of the text. lexical methods in ts are restricted to dealing with complex words in sentences. just changing complex words is not enough to simplify legal sentences. even in english, there is a lack of available data for ts in the domain, and the syntactic methods focus on splitting sentences into shorter ones. an approach that considers the entire syntax of sentences in portuguese was developed in porsimples [14]. the project started in 2009, and in 2010, a rule-based system was developed and made available but is not currently being maintained. the authors also contributed with the only portuguese parallel dataset containing original and simplified sentences. a recent work presented machine learning methods for ts in judicial summaries of two brazilian courts [3]. however, they reported results of readability metrics only, and those are not commonly used to evaluate ts systems. this paper complements the previous studies by assessing ts in five llms applied to legal texts, comparing them with documents simplified by human specialists in brazilian courts. it is unclear how existing models with different architectures can perform in portuguese. the legal domain presents difficulties and a lack of data, even in english. we aim to answer whether a dataset for ts in portuguese is enough to generalize and allow simplifying legal texts, and how some models can perform the task in this specific domain. to the best of our knowledge, this is the first work to evaluate the domain in portuguese using metrics that take reference sentences into account. 3 materials and methods this section describes the macro-steps for automatic ts we adopted in this work to answer the following research questions: rq1 are the main dataset for ts and main pre-trained models in portuguese able to generalize and be applied to the legal domain? rq2 how well do state-of-the-art and recently launched llms perform ts in the chosen domain and language? fig. 1. pipeline with the steps in our methodology. full size image rq3 can rl improve ts results for fine-tuned models? figure 1 shows an overview of our pipeline. our input comes from the three sources of data that were combined into a merged dataset. this dataset is then used in different ways: (i) for providing training data for fine-tuning a transformer-based model and (ii) enriching prompts sent to generative models. the outputs produced by the models are evaluated against the reference simplified sentences. 3.1 assembling a ts dataset in portuguese the input instances for training and evaluating ts systems consist of a pair \(\langle o, r \rangle \), where o is the original (complex) sentence and r is its simplified version. unfortunately, there is no such dataset for the legal domain in portuguese. as a result, we assembled a dataset for training and evaluating ts models, combining data from different sources (some in the legal domain and some generic). the following three sources of data were used. porsimples [14] created four baselines referred to as porsimplessent. they contain a parallel corpus with two levels of simplification, natural and strong. the texts were extracted from brazilian news articles. each pair \(\langle o, r \rangle \) is labeled according to the level of simplification, which can be n (natural simplification) or s (strong simplification). the difference between them is the degree of operations made into the original sentences. in n, splitting and inversion of clause ordering are applied discreetly, and in s, all mapped rules are applied in the simplification (rewriting and lexical substitution, change to canonical order, change to active voice, reordering, splitting, joining, and dropping). all combinations of the pairs in this dataset were used in the fine-tuning step, totaling 8,120 pairs. the combinations are o\(\rightarrow \)n, o\(\rightarrow \)s, and n\(\rightarrow \)s, where the text on the left is equal to or more complex than the text on the right. legal case status updates is a dataset assembled by jusbrasilfootnote 4, a brazilian company that provides access to information on legal cases. they used an openai generative model to explain the meaning of status updates such as “remetidos os autos (em diligência) para central de perícia”. there are 1,656 explanations. each explanation is annotated by experts who rated their quality according to different aspects, namely: (i) whether the explanation is legally accurate and if it is somehow useful. the possible answers were ‘yes’, ‘no’, ‘partially’, or ‘not applicable’; and (ii) the quality of the explanation. the possible answers were ‘good’, ‘bad’, ‘partially’, or ‘not applicable’. we discarded the 231 instances annotated with ‘bad’ quality or that were not considered legally accurate or useful. the remaining 1,424 instances were used in the fine-tuning step. hand-picked examples. we manually selected 149 pairs of sentences \(\langle o, r \rangle \) from materials published by justice courts that are involved in plain language projects promoted by the brazilian government. the list of these sources can be found infootnote 5. instances from the three sources were merged and used as training data. the test data consists of 91 instances from the hand-picked dataset that are from the legal domain. the training, validation, and test splits were disjoint. some statistics of the datasets are in table 2. some examples of the dataset structure are in table 1, and statistics of the datasets are in table 2. table 1. example of the dataset rows used in the workfull size table table 2. statistics of the dataset used for fine-tuningfull size table 3.2 machine learning models we employed four widely used transformer-based models to perform ts. both decoder-only and encoder-decoder models were used. ptt5 [5] is a t5 [20] model pre-trained on the brwac corpus [24], a large collection of web pages in portuguese. it improves the performance of t5 on portuguese sentence similarity and inference tasks. it is available in three sizes (small, base, and large) and with two vocabularies. we used the base model as an initial baseline. ptt5 can also be used with instructions only (and no fine-tuning), but this approach did not work well on this model, as the final outputs were just a copy of the prompt. thus, only the fine-tuning alternative of ptt5 is reported here and is labeled as ft-ptt5. flan-t5-large [7] is also a multilingual model from the t5 family. it has 700 million parameters and continues the training from an adapted t5-lm checkpoint. it uses a wide variety of labeled data to fine-tune with instructions, but the data is not specific to the ts task. gpt-3.5-turbo and gpt4o the gpt models originate from openai’s research in nlp, with gpt-3.5-turbo and gpt-4o featuring advancements such as increased parameters and context windows. the gpt-3.5-turbo and flan models were chosen based on a benchmarking of generative models for ts [13]. to the best of our knowledge, gpt-4o was not tested for ts due to the recent launch. these three models were used based solely on prompting without any fine-tuning. 3.3 fine-tuning and reinforcement learning approach fine-tuning is the process of adjusting a pre-trained language model on a specific dataset for a particular task or domain. in this work, the ts task is the target. the fine-tuning was done using the ptt5 model. fine-tuning procedure. to fine-tune ptt5 to perform ts, we followed the splits described in table 2. the batch size and batch per device in training were set to 4 sentences due to infrastructure limitations. we used 3 steps of gradient accumulation and a batch size of 64 per device for evaluation. the learning rate was 1e-4, weight decay was 0.01, and training was for 100 epochs with checkpoints at every 3000 steps. the loss function was used to choose the best model, and fp16 (half-precision floating point) was set to true. the sari metric was calculated during validation. reinforcement learning. this fine-tuned model was used to apply reinforcement learning (rl) based on dress [27]. we refer to the resulting model as ft-ptt5 + rl. however, dress used an lstm to generate the outputs and then applied rl. in the context of text tasks, rl is used as additional training for pre-trained models. a common technique is to linearly interpolate the rl reward with cross-entropy loss to avoid erroneous training due to the large action space. during rl, we used metrics that do not rely on reference simplifications. these metrics are different from the ones used for evaluating the quality of the predictions in the test set (sect. 3.5) since one cannot optimize and evaluate using the same metrics. 1. fkgl—flesch-kincaid grade level is used to evaluate the readability of a text, indicating the level of education necessary to understand it easily. it considers the average number of words per sentence and the average number of syllables per word, offering a score corresponding to years of schooling to understand a text. as sari is the main metric to automatically evaluate outputs in ts, fkgl was calculated as a simplicity reward in rl. 2. samsa—semantic annotation for machine reading focuses on the ability of the system to understand and extract accurate semantic information from complex texts. using semantic annotations, such as named entities and relationships between concepts, samsa seeks to quantify the quality of the system interpretation by identifying whether it correctly captures the essential meaning of the text [21]. samsa assigns high scores to split sentences, which is a simplification action cited as one of the most important in ts [14]. 3. levenshtein distance—calculates the difference between two strings. this measure quantifies the minimum number of operations necessary to transform one sequence into another, where valid operations are insertion, deletion, or replacement of a single character. this metric is widely used in word processing applications such as spelling correction and plagiarism detection, providing an efficient way to compare and measure similarity between different strings. the calculation of this reward aims to preserve the meaning of the original sentence. in the rl step, the model is trained to increase a reward. the ptt5 fine-tuned model is seen as an agent; it reads the source sentence and then takes an action according to a policy. the agent generates the output sequence as the simplified text; a reward is calculated, and the reinforcement updates the agent state. the reward r(ŷ) used in this work followed the base reward of dress [27] and is calculated as in eq. 1 according to three perspectives: simplicity (\(r_s\)), relevance (\(r_r\)), and fluency (\(r_f\)): $$\begin{aligned} r(\hat{y}) = \lambda ^s r^s + \lambda ^r r^r + \lambda ^f r^f \end{aligned}$$ (1) where \(r^s\) is the fkgl score for simplicity, \(r^r\) is the levenshtein distance for meaning preservation and \(r^f\) is the samsa score for fluency. \(\lambda ^s\), \(\lambda ^r\) and \(\lambda ^f\) are constants \(\in \) [0, 1] and were set as 1, 0.25 and 0.5, respectively. 3.4 generative approach the lack of a parallel dataset to allow fine-tuning or transfer learning into the several versions of llms available makes the generative models a good alternative. the instructions do not need many examples to enable the model to understand the main goal of a task. this is exactly the scenario we have for the portuguese legal domain. we adapted the prompt from bless [13], which uses the few-shot in-context learning and achieved good results in three domains in english. the prompt we used was written in portuguese, but the translated instruction is presented in fig. 2. fig. 2. structure of the instruction submitted to the generative models. full size image 3.5 evaluation metrics the evaluation of ts is an open problem. metrics designed for readability and levels of text understanding are mainly used in linguistics research about the task, but they are not adequate to assess simplification results. fkgl is an example of such a metric. its score is highly sensitive to sentence length and, in some cases, provides only minor impacts on other metrics, like sari [22]. a sentence with many token deletions can cause a loss of important information and may not be able to represent the real meaning of the text message. bleu and sari are also being widely used to evaluate the quality of ts. bleu aims to correlate grammatically with human perception, in addition to preserving meaning. sari is a good option to get an overview of simplicity, but it has better results assessing lexical paraphrasing systems [2]. bertscore has good results at identifying when the reference sentences are similar to the system outputs. it is also a good option to evaluate meaning preservation and similarities with the reference, but a high score does not mean the output was simplified [2]. the metrics we used for output evaluation are briefly described next. 1. sari—system for automatic evaluation is an automatic metric that evaluates how well a system preserves meaning while making text accessible and easier to understand. the metric is calculated through precision, recall, and f1-score for unigrams, bigrams, and trigrams, incorporating semantic similarity [25]. sari is calculated on the n-grams added and kept, and the proportion of those deleted. 2. bleu—bilingual evaluation understudy is a metric used to evaluate machine translation systems. it measures n-gram overlap between the output and one or more human references. the closer the bleu score is to 1, the closer the output is to the human reference according to syntax (word choices and their order) [17]. 3. bertscore—is a metric for evaluating the quality of text generation by comparing the generated text to a reference text. unlike traditional metrics like bleu or rouge, which rely on exact n-gram length, bertscore leverages contextual embeddings from the bert model. he has a better evaluation of meaning preservation because of the bert background, computing a similarity score between each token in the candidate and reference texts, capturing semantic nuances and contextual meaning [26]. 4. rouge—recall-oriented understudy for gisting evaluation assesses the overlap of n-grams, as well as the overlap of word sequences and word pairs, between the generated text and a reference text. by focusing on recall, rouge emphasizes capturing as much of the reference text content as possible. 4 experiments and results the experiments carried out in this work aim to answer the research question posed in sect. 3. 4.1 quantitative evaluation our quantitative evaluation was based on the metrics described in sect. 3.5. this validation was made over the 91 hand-picked instances that are from legal texts (sect. 3.1). the average value of the metrics obtained by each model is shown in table 3 and gives us a brief overview to answer rq2. table 3. average values of each evaluation metric on the instances from the test setfull size table sari is the main metric used to evaluate ts, while bertscore, bleu, and rouge are used to measure the meaning preservation of the simplified text. gpt models and ft-ptt5 obtained the highest values for sari, indicating that the sentences are simpler to understand. the winner here is gpt-4o, although when looking at bleu scores, the two gpt models have lower scores in comparison to all t5 models, which means they did not preserve the syntax of the original sentences. all the t5 family has high bleu scores. however, this changes when we look at bertscore and rouge values. the gpt models are still outperformed by the t5 models. according to bertscore and rouge, gpt-4o and gpt-3.5-turbo are good options for ts, as they have the highest simplicity scores while preserving meaning. flan-t5-large is the winner in meaning preservation; it has the best scores for similarity with reference simplifications, but sari is the penultimate score. the generated simplifications are really discrete, not making significant changes to the original sentence, as shown in the following example: original: “nada veda que a declaratória seja ajuizada em conexão com o pedido constitutivo ou condenatório.”. simplified: “nada proíbe que a declaração seja julgada em conexão com o pedido constitutivo ou condenatório.” figure 3 reveals that gpt-4o may be the best model, with a slight advantage over gpt-3.5-turbo. ft-ptt5 has better results for sari compared to ft-ptt5+rl. rl relied on fkgl as the simplicity metric, and these results reinforce the idea that fkgl is not suitable to measure ts. fig. 3. results of each model on sari and bertscore, evaluating simplicity and meaning preservation, respectively. full size image the scores of the model with rl and flan-t5-large are quite similar, with a slight advantage to flan-t5-large. as the results for meaning preservation are a little better than the ptt5 fine-tuned, it seems that levenshtein distance is also a good alternative to check the similarity between the sentences. 4.2 qualitative evaluation to provide insights into the quality of the generated simplifications, the outputs of the ts methods were analyzed by a domain specialist. the annotator is a judicial analyst who takes on the role of editorial reviewer in a brazilian court of justice. the sample selected for annotation contains 91 pairs for each of the five simplification methods, containing the original sentence and the simplification output of each method adopted. the annotator was not aware of which method produced the simplification. the specialist received the original and the simplified sentences and was asked to rate the quality of the simplification by answering three questions: q1—is the simplification correct? a correct simplification should be grammatically correct from the point of view of the norms of the portuguese language and the conceptual legal norm. the possible values answers are: ‘yes’, ‘no’, or ‘partially’. q2—is the simplified sentence indeed simpler to understand? the possible answers are: ‘yes’, ‘no’, or ‘same’ (when the output is identical to the input). q3—what is the quality of the simplified sentence? this is based on the previous questions, meaning how can the simplified sentence be evaluated. the values to be annotated are: ‘good’, ‘reasonable’, or ’bad’. the main guidelines from the plain language book [8], which supported the evaluation of the sentences generated by the models, are: write short sentences with an average length of 15 to 20 words; choose commonly used words; use the direct order of the sentence; prefer direct action verbs; divide the text into topics; avoid gender markers; present the most important information at the beginning; and use appropriate punctuation. table 4 provides an overview of the human evaluation of the generated simplifications for each question (answer to rq2). it shows the percentage of the instances that were classified by the expert into each of the possible categories. table 4. results of the qualitative evaluation.full size table gpt-3.5-turbo is the winner, with the most robust performance across all three evaluative questions. the only output with ‘bad’ quality is in table 5, along with an example of ‘good’ quality. the first does not correspond to the content of the reference sentence, adding information. in the second output, the order of the sentences is changed to allow a fluid reading, and the vocabulary is simplified using more frequent words. gpt-4o also demonstrates competitive results, with high counts in ‘yes’ responses for q1 and q2. both models present very low bleu scores but high scores in bertscore. bleu uses the exact n-gram size, while bertscore considers the context of the sentence. by applying ts, syntax and lexical changes can bring differences in terms of n-grams but keep the meaning. this shows the importance of using different metrics in the evaluation step. flan-t5-large presented high scores for meaning preservation but low scores for simplicity. as we can see in table 4, 95.6% of sentences were not modified from their original forms, agreeing with the results presented in the quantitative evaluation in table 3. ft-ptt5 and ft-ptt5 + rl have similar results in quantitative evaluation, with the first one winning in simplicity, but in qualitative evaluation, ft-ptt5 + rl have less bad quality sentences than ft-ptt5 only. table 5. sentences simplified by the winning model, gpt-3.5-turbofull size table the ft-ptt5 and ft-ptt5+rl provide better results compared to flan-t5-large, considering both qualitative and quantitative evaluations. both ptt5-based models have problems in q1, not preserving the legal meaning or with grammar problems. since the fine-tuning does not have training examples from the domain, this can be the main reason for not being able to generalize for the domain (answer to rq1). dress [27] is the rl state-of-the-art with the best results until now, but the change of metrics used in this work did not yield good results in both evaluations. fine-tuning models have shown to be promising for ts, and rl is able to increase the performance too (answer to rq3), as only 2% of the outputs were considered bad against 30% of ft-ptt5. 5 conclusion this work evaluated llm for ts in legal texts written in portuguese. we fine-tuned the pre-trained ptt5 model and developed an rl algorithm based on dress [27] but with ptt5 as a base for text-to-text generation. these models were compared with two gpt-3.5-turbo, gpt4o, and flan-t5-large. there are many types of legal documents in portuguese available, but without their ground truth simplifications, they cannot be readily used to train or evaluate ts. we relied on a dataset of legal case updates with their plain language explanations obtained from gpt-based models as training data. although this data is in the domain of interest, it is not a ts dataset in the strict sense. in addition, our qualitative evaluation was made by only one specialist. as future work, experiments using other models, such as bardfootnote 6 and sabiáfootnote 7, can be performed. fine-tuning the larger versions of llms can be very costly, but it has been shown to improve ts results. the use of rl had similar results to the fine-tuned model. however, the use of other metrics in the reward functions could improve performance, closing the gap in comparison to the gpt models. other methods, such as performing ts according to readability levels, could be explored. the legal domain has different sub-fields, which makes it difficult to generalize a model for all sectors. an expression may have different meanings across sub-fields of the legal domain, or even across brazilian estates. thus, the question of whether a model can be robust to such variations is still unanswered. notes 1.https://www.cnj.jus.br/gestao-da-justica/acessibilidade-e-inclusao/pacto-nacional-do-judiciario-pela-linguagem-simples/. 2.https://www.cnj.jus.br/gestao-da-justica/acessibilidade-e-inclusao/pacto-nacional-do-judiciario-pela-linguagem-simples/selos/. 3.https://comunicasimples.com.br/2023/01/03/juridiques/. 4.https://www.jusbrasil.com.br. 5.https://cutt.ly/links-plain-language. 6.https://gemini.google.com/. 7.https://www.maritaca.ai. references de almeida guimarães, l.h.p.: a simplificação da linguagem jurídica como instrumento fundamental de acesso à justiça. publicatio uepg: ciências humanas, linguistica, letras e artes 20(2), 173–184 (2012) google scholar  alva-manchego, f., scarton, c., specia, l.: the (un) suitability of automatic evaluation metrics for text simplification. comput. linguist. 47(4), 861–889 (2021) article  google scholar  alves, a., miranda, p., mello, r., nascimento, a.: automatic simplification of legal texts in portuguese using machine learning. in: legal knowledge and information systems, pp. 281–286. ios press (2023) google scholar  candido, jr., a., maziero, e.g., specia, l., gasperin, c., pardo, t., aluisio, s.: supporting the adaptation of texts for poor literacy readers: a text simplification editor for brazilian portuguese. in: workshop on innovative use of nlp for building educational applications, pp. 34–42 (2009) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data. arxiv preprint arxiv:2008.09144 (2020) cemri, m., çukur, t., koç, a.: unsupervised simplification of legal texts. arxiv preprint arxiv:2209.00557 (2022) chung, h.w., et al.: scaling instruction-finetuned language models. j. mach. learn. res. 25(70), 1–53 (2024) google scholar  cutts, m.: oxford guide to plain english. oxford university press (2013) google scholar  ferrés, d., saggion, h., guinovart, x.g.: an adaptable lexical simplification architecture for major ibero-romance languages. in: workshop on building linguistically generalizable nlp systems, pp. 40–47 (2017) google scholar  finatto, m.j.b., macohin, a.: pln no direito: perspectivas e desafios com textos jurídicos e legais. in: caseli, h.m., nunes, m.g.v. (eds.) processamento de linguagem natural: conceitos, técnicas e aplicações em português, chap. 26, 2nd edn. bpln (2024) google scholar  garimella, a., sancheti, a., aggarwal, v., ganesh, a., chhaya, n., kambhatla, n.: text simplification for legal domain: insights and challenges. in: aletras, n., chalkidis, i., barrett, l., goantă, c., preotiuc-pietro, d. (eds.) natural legal language processing workshop 2022, pp. 296–304 (2022) google scholar  hartmann, n.s., aluísio, s.m.: adaptação lexical automática em textos informativos do português brasileiro para o ensino fundamental. linguamática 12(2), 3–27 (2020) google scholar  kew, t., et al.: bless: benchmarking large language models on sentence simplification. in: the 2023 conference on empirical methods in natural language processing (2023) google scholar  leal, s.e., duran, m.s., aluísio, s.: a nontrivial sentence corpus for the task of sentence readability assessment in portuguese. in: international conference on computational linguistics, pp. 401–413 (2018) google scholar  martin, l., fan, a., de la clergerie, é., bordes, a., sagot, b.: muss: multilingual unsupervised sentence simplification by mining paraphrases. in: calzolari, n., et al. (eds.) language resources and evaluation conference, pp. 1651–1664 (2022) google scholar  north, k., zampieri, m., ranasinghe, t.: alexsis-pt: a new resource for portuguese lexical simplification. arxiv preprint arxiv:2209.09034 (2022) papineni, k., roukos, s., ward, t., zhu, w.j.: bleu: a method for automatic evaluation of machine translation. in: annual meeting of the association for computational linguistics, pp. 311–318 (2002) google scholar  pena, t.m.g.: a simplificação da linguagem jurídica como fator de democratização do direito e inclusão social. revista do tribunal regional do trabalho da 24\(^{\underline{a}}\) região (5), 109–129 (2020) google scholar  polo, f.m., et al.: legalnlp-natural language processing methods for the brazilian legal language. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 763–774 (2021) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21(140), 1–67 (2020) google scholar  sulem, e., abend, o., rappoport, a.: semantic structural evaluation for text simplification. in: conference of the north american chapter of the association for computational linguistics: human language technologies, vol. 1 (long papers), pp. 685–696 (2018) google scholar  tanprasert, t., kauchak, d.: flesch-kincaid is not a text simplification evaluation metric. in: workshop on natural language generation, evaluation, and metrics (gem 2021), pp. 1–14 (2021) google scholar  viegas, c.f., costa, b.c., ishii, r.p.: jurisbert: a new approach that converts a classification corpus into an sts one. in: international conference on computational science and its applications, pp. 349–365. springer (2023) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: international conference on language resources and evaluation (lrec) (2018) google scholar  xu, w., napoles, c., pavlick, e., chen, q., callison-burch, c.: optimizing statistical machine translation for text simplification. trans. assoc. comput. linguist. 4, 401–415 (2016) article  google scholar  zhang, t., kishore, v., wu, f., weinberger, k.q., artzi, y.: bertscore: evaluating text generation with bert. in: international conference on learning representations (2019) google scholar  zhang, x., lapata, m.: sentence simplification with deep reinforcement learning. in: conference on empirical methods in natural language processing (2017) google scholar  download references acknowledgments this work has been partially funded by cnpq-brazil, and capes finance code 001. the authors are thankful to edleno silva de moura for sharing the legal case updates created by jusbrasil and to inovajus for sharing examples of simplification. author information authors and affiliations institute of informatics, ufrgs porto alegre, porto alegre, rio grande do sul, brazil francielle vasconcellos pereira & viviane p. moreira usp são paulo, são paulo, brazil ana frazão authors francielle vasconcellos pereiraview author publications search author on:pubmed google scholar ana frazãoview author publications search author on:pubmed google scholar viviane p. moreiraview author publications search author on:pubmed google scholar corresponding author correspondence to viviane p. moreira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pereira, f.v., frazão, a., moreira, v.p. (2025). automatic text simplification for the legal domain in brazilian portuguese. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_3 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automatic text simplification legal texts natural language processing plain language publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a protocol for argumentation-based persuasive negotiation dialogues | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a protocol for argumentation-based persuasive negotiation dialogues conference paper first online: 28 november 2021 pp 18–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a protocol for argumentation-based persuasive negotiation dialogues download book pdf download book epub mariela morveli-espinoza  orcid: orcid.org/0000-0002-7376-227110, ayslan possebom  orcid: orcid.org/0000-0002-1347-585211 & cesar augusto tacla  orcid: orcid.org/0000-0002-8244-897010  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 802 accesses 2 citations abstract argumentation-based persuasive negotiation is a form of negotiation dialogue in which agents, with different interests and goals, exchange proposals that are supported by rhetorical arguments such as threats, rewards, or appeals. besides rhetorical arguments, additional kinds of illocutions may also be exchanged during the dialogue, for instance, agents may ask for explanations, give explanations, or attack (or contradict) previous arguments. this paper presents a formal protocol for argumentation-based persuasive negotiation dialogues in which a proponent agent tries to persuade his opponent to perform a given action and the opponent tries to maintain his position. the protocol is modelled as a dialogue game (i.e. the interactions between the proponent and the opponent are governed by a set of rules) and the outcome of the dialogue is determined by applying an argumentation semantics. we prove the soundness and completeness of our proposal and illustrate the proposed protocol by using an example. supported by capes. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others negotiation as practical argumentation article 01 june 2023 persuasion dialogues via restricted interfaces using probabilistic argumentation chapter © 2016 on a formal treatment of deception in argumentative dialogues chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. dispute resolution, mediation, arbitration diplomacy diplomatics lobbying mediation and conflict management pragmatics 1 introduction negotiation is a key form of interaction among agents that can be used for resolving conflicts and reaching agreements. formal argumentation is a process based on the construction and comparison of arguments considering the conflicts that may emerge among them. such conflict are called attacks. the idea is to determine set(s) of non-conflicting arguments (called extensions), which are considered acceptable or justified. the function in charge of calculating the extensions is called semantics [13]. some works on negotiation argue that argumentation – using explanatory arguments – allows that an agent acquire additional information about his opponents, which can be used for attacking his opponent’s proposals or justifying his own proposals (e.g., [3, 12, 26, 29]). besides explanatory arguments, there exist other kinds of arguments that can be used in negotiation dialogues and act as persuasive elements aiming to force or convince an opponent to accept readily a given proposal, these are called rhetorical arguments (e.g., in [20,21,22]). according to ramchurn et al. [27], a negotiation involving these kinds of arguments is called persuasive negotiation. we can describe the rhetorical arguments as follows: (i) threats, which try to persuade an opponent agent by using the argument that something negative will happen to him if he does not accept to do the requirement sent by the proponent; (ii) rewards, which try to persuade an opponent by using the argument that something positive will happen to him if he accepts to do the requirement sent by the proponent; and (iii) appeals, which try to persuade an opponent in the same form than rewards, but this positive event will depend on the opponent; hence, appeals can be seen as self-rewards [4]. rhetorical arguments have been studied in terms of speech acts (e.g., [27]) and in other articles, a logical formalization has been given (e.g., [5, 6]). it was also studied how to evaluate their strength values (e.g., [6, 21, 22]). however, to the best of our knowledge, no study about a protocol involving these arguments has been proposed. in order to better understand the problem, imagine a scenario where two agents, maria (m) and carlos (c), are discussing about household chores. maria is trying to persuade carlos to do the cleaning of their apartment. the following dialogue shows how agreement is reached: (1) m: carlos, could you please do the cleaning? (2) c: no, i can not, i have to work. (3) m: if you do the cleaning, i could help you with your reports and you can finish your work early. (4) c: you can not help me. (5) m: why? (6) c: because these reports are about a topic you do not know. (7) m: well, if you do not do the cleaning, i will not go to your mother’s house on saturday. (8) c: if you will not go to my mother’s house, i will not talk to her about the work for your brother. (9) m: that is not longer necessary, my brother got a job yesterday. (10) c: ok. you win! in this example, maria succeeds in persuading her husband carlos to do the cleaning of their apartment. on the first attempt to persuade carlos, maria uses a reward (line 3), which is not accepted, resulting in an attack to her reward (line 4). in this settings, an attack is a contradictory statement. since her reward was not successful, she uses a more powerful argument, i.e. a threat(line 7), and then he also answers with another threat (line 8), which we can call a counter-threat. she answers attacking carlos’ counter-threat (line 9). notice that this attack is not a counter-threat, which indicates that there is more than one way to attack a rhetorical argument. finally, carlos accepts to do the cleaning. besides rhetorical arguments and their corresponding attacks, we can notice that an explanation is required during the conversation (line 5), this means that agents can make questions to each other and can use explanatory arguments to justify their opinions. figure 1 shows the outline of the dialogue in terms of rhetorical arguments, attacks, and other illocutions. fig. 1. outline of the dialogue between maria and carlos, which ends successfully for maria. full size image from this scenario, we can observe that during a persuasive negotiation dialogue agents can exchange rhetorical arguments, attacks to rhetorical arguments, questions, explanatory arguments, and attacks to explanatory arguments. most of the research about protocols in literature is focused on negotiation (e,g., [1, 11, 16, 28]), some others on persuasion (e.g., [10, 17]), and some others on argumentation-based dialogues (e.g., [8, 9, 14, 23, 25, 30, 31]). although some of these protocols take into account explanatory arguments and attacks among them, these are not embedded in a persuasive negotiation dialogue (according to the definition of [27]) and do not interact with other kinds of arguments. thus, the research questions that are addressed in this paper are: 1. how can rhetorical arguments, explanatory arguments, attacks, and other illocutions be combined in a coherent dialogue? 2. how can argumentation techniques be used in this kind of dialogue? in order to address the first question, we propose a protocol that has a dialogue game form where utterances are viewed as moves in a game, which is guided according to a defined set of rules. according mcburney and parsons [18], formal dialogue games allow sufficient flexibility of expression while avoid state-space explosion. regarding the second question, an important part of a dialogue is the outcome. in this case, the outcome has to determine the final status of the dialogue (that is, it ends with an agreement or not), the winner (if there is an agreement), and the set of commitments the winner agent has to fulfil after the dialogue, for example, maria will go to the house of the mother-in-law since carlos will do the cleaning. we will use argumentation semantics to determine the outcome of the dialogue. next section presents the main concepts about argumentation and argumentation semantics. section 3 concerns with the type of arguments and attacks. section 4 presents the proposed protocol, that is, the rules that govern the interactions among the agents, the argumentation framework that determines the outcome of the dialogue and the main properties of the approach. in sect. 5, we illustrate our new protocol by applying it to the example given in introduction. finally, sect. 6 is devoted to conclusions and future work. 2 background in this section, we present the concepts of argumentation framework and the acceptability semantics for linear dialogues. an argumentation framework consists of a set of arguments and a attack relation between them. the following definitions were extracted from [2] and [13]. definition 1 (argumentation framework) an argumentation framework is an ordered pair , where \(\mathtt {arg}\) is a finite set of arguments and a binary relation on \(\mathtt {arg}\) (i.e., ). we call an attack relation and (a, b) means that argument a attacks argument b. before presenting the acceptability notion, it is important to study linear discussions. a linear discussion is a sequence of arguments such that each argument attacks the argument preceding it in the sequence. this sequence will determine which arguments can be considered acceptable and which cannot. definition 2 (linear discussions) let be an argumentation framework and \(a \in \mathtt {arg}\). a linear discussion for a in \(\mathtt {arg}\) is a sequence \(s = \langle a_1, ... , a_n \rangle \) of elements of \(\mathtt {arg}\) (where n is a positive integer) such that \(a_1 = a\) and \(\forall i \in \{2, 3, .. . , n\}\) . next we present the semantics, which determines what arguments are considered acceptable or justified. first, it is necessary the notion of supporters. given three arguments \(a_1\), \(a_2\), and \(a_3\). if and , then \(a_3\) supports \(a_1\). the acceptability notion is directly related to argument a because it represents the central point of the discussion. thus, given a sequence s, we can say that \(\forall a_i \in s\), \(a_i\) supports a if i is odd and it attacks a if i is even. let \(\mathtt {sup}(a)\) return all the supporters of \(a_1\) and \(\mathtt {att}(a)\) return all its attackers. definition 3 (semantics) let be an argumentation framework, \(a \in \mathtt {arg}\), s a sequence for a, and n the length of s. it holds that (i) if n is odd, then \(a \cup \mathtt {sup}(a)\) are acceptable, or (ii) if n is even, then \( \mathtt {att}(a)\) are acceptable. let be a semantics function that returns the set of acceptable arguments. 3 building blocks in this section, we present the topic language used to represent the content of illocutions exchanged by agents. we also present the definitions of the kinds of arguments that can be exchanged and study the possible attacks to each of them. according to van veenen and prakken [30], formal dialogue games have a topic language , expressed in a certain logic, and a communication language with a protocol , which specifies the allowed moves at each point in a dialogue. we can say that a persuasive negotiation dialogue happens between a proponent agent p and an opponent agent o about a topic . in this work, the topic language is specified using the classical logical language. symbols \(\wedge \),\(\vee \) and \(\lnot \) denote the logical connectives conjunction, disjunction, and negation, respectively. besides, \(\vdash \) stands for the classical inference and \(\equiv \) logical equivalence. from we can distinguish the three following sets of formulas: contains the goals the agent pursues; contains what the agent believes the goals of the other agent are (that is, his opponent’s goals); and is the knowledge base of the agent, which gathers the information the agent has about the environment. goals and opponent’s goals are represented with positive literalsfootnote 1 from . besides, and are finite sets such that . now, let us present the definitions of arguments, explanatory arguments, rhetorical arguments, and attacks involved during a persuasive negotiation dialogue. in what follows, for a given argument, the function \(\mathtt {supp}\) returns all the beliefs of (called support) used to build the argument and \(\mathtt {conc}\) returns its conclusion. the first one is a basic definition where any element of can be an argument whereas explanatory and rhetorical arguments have a deductive form. indeed, a fact or a goal is entailed from the support. definition 4 (argument [19]) let be a knowledge base. an argument a is \(\varphi \) if with \(\mathtt {supp}(a)\) = \(\{\varphi \}\), \(\mathtt {conc}(a) = \varphi \). explanations are the most common category of arguments. they represent the reasons to believe in a fact. definition 5 (explanatory argument [7]) let be the knowledge base of an agent. an explanatory argument is a tuple such that (i) , (ii) , and (iii) is consistent and minimal.footnote 2 with: and \(\mathtt {conc}(a)=h\). unlike explanatory arguments, rhetorical arguments are also made up by goals and opponent’s goals. definition 6 (rhetorical arguments [7]) let be the knowledge base of an agent, be his goals base, and be his opponent’s goals base. a threat, reward, or appeal is a triple such that (i) , (ii) , (iii)  , and (iv) is consistent and minimal. besides: in the case of threats, it holds that in the case of rewards and appeals, it holds that . with: , \(\mathtt {conc}(a)=g\), and \(\mathtt {opgoal}(a)=go\) returns the goal that is being threatened, rewarded, or appealed. it is also necessary to define the attacks each kind of argument may receive. an attack relation between two arguments a and b denotes the fact that these arguments cannot be accepted simultaneously since they contradict each other. in the case of explanatory arguments, two kinds of attacks can be determined, undercuts and rebuttals. an undercutting argument is an argument whose conclusion contradicts some of the elements of the support of another argument, and a rebutting argument is an argument whose conclusion is the negation of the conclusion of another argument. formally: definition 7 (attacks to explanatory arguments [7]) let and be two explanatory arguments: undercuts iff such that \(h' \equiv \lnot h''\). rebuts iff \(h \equiv \lnot h'\). regarding attacks to rhetorical arguments, we have distinguished three types of attacks. the first one occurs when a threat is attacked by another threat (we can call it a counter-threat). in this case, there is no a logical contradiction but we can notice that the goal threatened by an agent is used by the opponent to construct another threat as can be observed in lines (7) and (8) of the example given in introduction. definition 8 (counter-threat) let be the set of goals of an proponent agent p, be a threat of p, and be a threat of opponent agent o. we say that \(th_o\) counter-threatens \(th_p\) when \(g'=go\) and . the second type of attack occurs when the opponent answers disesteeming his threatened/rewarded/appealed goal and denies his interest in achieving it. for example, line (9) of the example given in introduction. in line (8), carlos threatens maria with not talking with his mother about a job for maria’s brother and in line (9), she says that her brother already got a job. this attack has the form of an explanatory argument. definition 9 (disesteemation) let be a rhetorical argument. an argument disesteems a when \(g'=\lnot go\). to the best of our knowledge, the two previous types of attacks were not studied before. on the other hand, a rhetorical argument can also be undercut. 4 the proposal in this section, we present the language and the rules for the dialogue game. besides, we present the argumentation framework that represents the dialogue and determines its outcome. 4.1 the proposed protocol the game is mainly based on the following ideas. each move in the dialogue, except the initial one, replies to the previous move of the other agent (we refer to the previous move as its target). in [24], the author proposes the idea of attack and surrender as a categorization of the possible replies to previous moves during the dialogue. a reply is a surrender when it is not against the previous move; otherwise, it is an attack. in summary, a reply can either be an attack or a surrender. table 1 presents the persuasion communication language , which takes into account the attacks and the surrenders. in this table, a, b, c, d, e and \(c'\) are arguments. let us recall that an attack relation between two arguments denotes the fact that these arguments cannot be accepted simultaneously since they contradict each other. following this idea, we can say that rejecting a request is an attack because it is a contradiction. we can also say that a rhetorical argument attacks a rejection because it defends a different position. thus, we will consider an attack those defined in previous section – which are more related to logical inconsistency – and also those that support a different position during the dialogue. it is reasonable to think that all the illocutions used by the proponent p aim to support his position, which is contrary to the position of the opponent o. in the attack column, besides the possible attacks, the conditions that relate the attacked with its attacker are stated. in order to standardize the content of speech acts, all of them are arguments; whether they are explanatory, rhetorical, or the basic ones according to definition 4. the idea is that the proponent agent uses rhetorical arguments to try to convince his opponent. thus, in this first version of the protocol only the proponent can use rhetorical arguments, that is threats, rewards, and appeals. counter-threats act as attacks, in this sense, these can be used by the opponent. after a rhetorical argument, the opponent can accept the proposal, send an attack, or withdraw from the dialogue. the last case, may happen when he has no attack to send but does not want to accept the proposal or when he has an explanation for a questioning. note that attacks can also be attacked and can be questioned in an element of their support. only counter-threats cannot be questioned because it is based on goals and not in beliefs. table 1. speech acts and possible replies in full size table the third component of a dialogue game is the protocol, which specifies the allowed moves at each point in a dialogue. thus, let us define first of all what a move is. definition 10 (move) a move is a tuple \(m= \langle id, sd, tg, sp\rangle \) where: \(id \in \mathbb {n}\) is the identifier of the move; \(sd \in \{p, o\}\) is the sender of the message, i.e. the agent that makes the move; \(tg \in \mathbb {n}\) is target of the move, i.e. a previous move to which it is directed. the target of a move is the identifier of some earlier move in the dialogue; is an speech act. let be a set of moves. as for notation, we use id(m), sd(m), tg(m), and sp(m) to refer to each of the components of a given move m. for the sake of simplicity, when we want to refer to the i-th move in a sequence, we use \(m_i\). besides, we use \(\mathtt {argum}(sp(m))\) to refer to the argument associated to a given speech act. a dialogue can be seen as a set of moves, which fulfil some conditions. let us now present the formal definition of dialogue. definition 11 (dialogue) a dialogue d between two agents p and o is a finite sequence \(\langle m_1, ..., m_n \rangle \), such that: \(m_1= \langle 1, p, -, request(a) \rangle \). it means that the first utterance is sent by the proponent agent and has to be a request; the content of \(m_k\) is request(a) iff \(k=1\). it means that a request can only be sent in the first move; \(tg(m_1)=0\). it means that the first utterance has no target; \(\forall k > 1\), it holds that \(tg(m_k)=j\), for \(j=k-1\). it means that the target of a move is always the previous move. let stand for the set of all dialogues. in the illustrative example, we can see that the proponent and the opponent agents take the turn to speak one after another. these moves are controlled by a function that determines which of the agents will make the next move. take into account that such move must agree with the possible replies defined in table 1. thus, a turn-taking function is a mapping , such that given a dialogue \(d=\langle m_1, ..., m_i \rangle \), it holds that (i) \(t(\emptyset ) = p\), (ii) \(t(d)=p\) if i is even, and (iii) \(t(d)=o\) if i is odd. we can notice that our definition of turn-taking forces a strict interleaving between agents p and o. next, we define our protocol in terms of legal moves the agents can perform. in table 1, we can notice that the answer for a speech act why(a) is an explanation for it; however, there is a need for a stop condition \(\mathtt {cond}\) in order to avoid infinite questioning. this condition can be a maximum number of rounds. definition 12 (legal-move function) a legal-move function is a mapping such that, given , for all , the following rules must be satisfied: \(r_1:\) ; \(r_2:\) sp(m) is a legal speech act after d (considering table 1); \(r_3:\) if \(\exists m_i | tg(m_i)=m_k\) – for \(1 < i \le n\), \(k<i\) – then \(\not \exists m_j | tg(m_j)=m_k\), for \(i \ne j\); \(r_4:\) if \(sp(m)=threat(a)\), \(sp(m)=reward(a)\), or \(sp(m)=appeal(a)\), then \(sd(m)=p\); \(r_5:\) if \(sp(m)=why(a)\) and \(\mathtt {cond}==true\), then the sender agent has to change his move and use another speech act. rule 1 says that the sender of a move has to obey the turn-taking function. rule 2 has to do with the valid answers to speech acts. rule 3 means that there is no move with the same target in a dialogue. rule 4 ensures that only the proponent can send a rhetorical argument. finally, rule 5 concerns with avoiding infinite questions. besides the rules related to legal moves, it is important to define some rules about the beginning and the end of the dialogue. \(r_6:\) if \(id(m)=1\), then \(sd(m)=p\); \(r_7:\) if \(id(m)=1\), then \(sp(m)=request(a)\); \(r_8:\) if \(sp(m) =accept(\varphi )\), then d ends with an agreement; \(r_9:\) if \(sp(m) =concede(\varphi )\), then d ends with an agreement \(r_{10}\) if , then d ends without an agreement; rule 6 says that the proponent agent always begins the dialogue and rule 7 asserts that the first movement is a request. rules 8, 9, and 10 have to do with the termination of a dialogue. a dialogue system also has effect or commitment rules, which specify the effects of moves on the participants’ commitments. a commitment store gathers the statements each agent have made and the challenges they have issued. commitment rules define how these commitment stores have to be updated and whether particular illocutions can be uttered at a particular time. let and be the commitment stores of the proponent and the opponent agent, respectively. the set of commitment rules is the followings: 1. \(cr_1:\) if \(sp(m_i) = why(a)\) and \(sd(m_i)=p\) then 2. \(cr_2:\) if \(sp(m_i) = reward(a)\) and \(sd(m_i)=p\) then 3. \(cr_3:\) if \(sp(m_i) = threat(a)\) and \(sd(m_i)=p\) then 4. \(cr_4:\) if \(sp(m_i) = explanation(a)\) and \(sd(m_i)=p\) then 5. \(cr_5:\) if \(sp(m_i) = accept(\varphi )\) and \(sd(m_i)=p\) then it also holds that: (i) \(cr_2\) also holds appeal(a) (ii) \(cr_2\) also holds counter-threat(b); (iii) \(cr_4\) also holds for undercut(b), rebuttal(b), and disesteemate(b); (iv) \(cr_5\) also holds for \(concede(\varphi )\); and (v) these rules hold for . finally, it holds that both and are consistent. 4.2 argumentation framework and dialogue outcome in this subsection, we present the argumentation framework for a persuasive negotiation dialogue and how to determine the outcome of the dialogue based on the semantics defined in sect. 2. definition 13 (dialogue argumentation framework) an af for a negotiation persuasive dialogue is a tuple such that: is a dialogue constructed under the rules of protocol and the commitment rules ; and are the commitments sets of the proponent and the opponent, respectively; \(\mathtt {arg}=\{\mathtt {argum}(sp(m_i)) \mid m_i \in d\), for \(1 \le i < n \}\); ; recall that the speech acts that end a dialogue are not associated with an argument, which is reflected in the set of arguments. we can notice that \(\mathtt {arg}\) and form a linear discussion with a sequence \(s = \langle a_1, ..., a_{n-1} \rangle \) where n is the number of movements of d, and \(a_1=\mathtt {argum}(sp(m_1))\). this means that we can apply the semantics given in definition 14 in order to define the acceptable arguments and based on these arguments, we can define the outcome of the dialogue, that is, the winner of the dialogue. if the proponent wins the dialogue (that is, he persuades his opponent), then the opponent has to perform the required action and the proposed threat, reward, or appeal has to be fulfilled. note that the proponent can send more than one rhetorical argument, in this case, the last rhetorical argument sent during the dialogue is the one that has to be fulfilled. on the other hand, when the proponent loses the dialogue, the opponent does not have to perform the required action and no offer has to be fulfilled. before define the outcome of the dialogue, we have to make a modification on the semantics due to the condition of the last movement. definition 14 (daf semantics) let be a dialogue argumentation framework, \(a \in \mathtt {arg}\), s a sequence for a, and \(n=|\mathtt {arg}|+1\): if n is odd and \(sp(m_n)=accept(\varphi )\) (or \(sp(m_n)=concede(\varphi )\)), then \(\mathtt {att}(a)\) are acceptable. if n is even and \(sp(m_n)=accept(\varphi )\) (or \(sp(m_n)=concede(\varphi )\)), then \(a \cup \mathtt {sup}(a)\) are acceptable. if \(sp(m_n)=withdraw\), then there are no acceptable arguments. let be a semantics function that returns the set of acceptable arguments. we can now define the outcome of the dialogue. definition 15 (dialogue outcome) let be a dialogue argumentation framework, \(a \in \mathtt {arg}\) is an argument that represents a required action, and p and o the proponent and the opponent agent, respectively: if then p wins the dialogue, and o has to perform \(\mathtt {conc}(a)\). if then p loses the dialogue. 4.3 properties of the proposal in this section, we will study some properties of our proposal. the aim is to evaluate its legality in the sense of fulfillment of the rules and the soundness and completeness of the argumentation process. the first proposition concerns with the legality of the moves exchanged during the dialogue. proposition 1 given \(d=\langle m_1, ..., m_n \rangle \), and \(d'=\langle m_1, ..., n_m \rangle \), where \(1 \le m \le n\) and considering that is compatible with and fulfils all of the previously established rules. we can say that if \(\forall m\), \(d'\) is a legal dialogue, then d is also a legal dialogue. next propositions concerns with the soundness and completeness of the argumentation process. proposition 2   be a dialogue argumentation framework: if \(\forall a \in \mathtt {arg}\), if , then \(\forall a'\) such that \(a' =\mathtt {argus}(sp(m'))\), \(a =\mathtt {argus}(sp(m))\), and \(sd(m)=sd(m')\), . such that \(a' =\mathtt {argus}(sp(m'))\) and \(a =\mathtt {argus}(sp(m))\), \(sd(m)=sd(m')\). the first item say that if an argument sent by one the agents is acceptable, then all the arguments sent by the same agent have to be acceptable as well. the second item says that all acceptable arguments were sent by the same agent. 5 applying the proposal to the illustrative example in this section, we evaluate if the example given in introduction fulfills the rules of the protocol. besides, we determine the outcome of the dialogue. we use p to refer to maria because her role in the dialogue is to be the proponent and o to refer to carlos because his role in the dialogue is to be the opponent. next, we have the set of moves: \(m_1\):: \(\langle 1, p, 0, request(\langle \{cleaning\}, cleaning \rangle )\rangle \) \(m_2\):: \(\langle 2, o, 1, reject(\langle \{cleaning\}, cleaning \rangle )\rangle \) \(m_3\):: \(\langle 3, p, 2, reward(\langle \{cleaning \rightarrow can\_help, can\_help \rightarrow finish\_work\}, cleaning, finish\_work \rangle )\rangle \) \(m_4\):: \(\langle 4, o, 3, undercut(\langle \{\lnot can\_help\}, \lnot can\_help\rangle )\rangle \) \(m_5\):: \(\langle 5, p, 4, why (\langle \{\lnot can\_help\}, \lnot can\_help)\rangle \) \(m_6\):: \(\langle 6, o, 5, explanation(\langle \{work, work \rightarrow \lnot cleaning\}, \lnot cleaning\rangle )\rangle \) \(m_7\):: \(\langle 7, p, 6, threat (\langle \{\lnot cleaning \rightarrow \lnot going\_mother\_house\}, cleaning, going\_mother\_house \rangle )\rangle \) \(m_{8}\):: \(\langle 8, o, 7, counte\)-\(threat(\langle \{\lnot going\_mother\_house \rightarrow \lnot talking\_about\_brother\_work\}, going\_mother\_house, talking\_about\_brother\_work \rangle )\rangle \) \(m_9\):: \(\langle 9, p, 8, disesteemate(\langle \{brother\_has\_work \rightarrow \lnot talking\_about\_brother\_work \}, \lnot talking\_about\_brother\_work \rangle )\rangle \) \(m_{10}\):: \(\langle 10, o, 9, concede(\lnot talking\_about\_brother\_work)\rangle \) we have a dialogue \(d=\{m_1, m_2, m_3, m_4, m_5, m_6, m_7, m_8, m_9, m_{10}\}\) where the request is the first move and the target of every move is the previous move. regarding the rules of the legal-move function, we can say that all the moves of d follow these rules. let us now present the commitments sets: . we can notice that both commitment sets are consistent. note also that includes the requested action as a positive literal (cleaning) whereas includes the requested action as a negative literal (\(\lnot cleaning\)). now, let us define the dialogue argumentation framework: where \(\mathtt {arg}=\{a_1, a_2, a_3, a_4, a_5, a_6, a_7, a_8, a_9\}\) such that each argument is associated to the number of the move in the dialogue d; ; and d and the commitment sets were presented above. the result of applying the semantics is: . we can now determine the outcome of the dialogue. we can notice that , this means that p (maria) wins the dialogue and o (carlos) has to do the cleaning of the apartment (cleaning). 6 conclusions and future work in this paper, we have presented a protocol for a persuasive negotiation dialogue. in the resulting dialogue game, agents can exchange rhetorical and explanatory arguments, can utter attacks for such arguments, can question an element of explanatory arguments, and also can use negotiation speech acts to request, reject, and finish the dialogue. the use of additional kinds of illocutions enriches the dialogue and allows the agents to not only try to persuade the other party but to defend their positions. the proposed protocol is also flexible since it allows for different alternative replies, which were categorized as attacks and surrenders. for future research, we propose two possible directions: (i) the first one is to improve the protocol itself, for example, allowing that the target of an utterance to be any of the earlier moves and (ii) the second one is extending the protocol for more than two agents, which also has consequences on the possible attacks for arguments. notes 1.a literal is either an atomic formula or the negation of an atomic formula. when a literal is an atomic formula, we say that it is a positive literal, and when a literal is the negation of an atomic formula, we say it is a negative literal. 2.minimal means that there is no such that and consistent means that it is not the case that and , for any h [15]. references aknine, s., pinson, s., shakun, m.f.: an extended multi-agent negotiation protocol. auton. agents multi-agent syst. 8(1), 5–45 (2004) article  google scholar  amgoud, l., ben-naim, j.: ranking-based semantics for argumentation frameworks. in: liu, w., subrahmanian, v.s., wijsen, j. (eds.) sum 2013. lncs (lnai), vol. 8078, pp. 134–147. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-40381-1_11 chapter  google scholar  amgoud, l., parsons, s., maudet, n.: arguments, dialogue, and negotiation. in: proceedings of the 14th european conference on artificial intelligence, vol. 10, pp. 338–342. ios press (2000) google scholar  amgoud, l., prade, h.: threat, reward and explanatory arguments: generation and evaluation. in: proceedings of the ecai workshop on computational models of natural argument, pp. 73–76 (2004) google scholar  amgoud, l., prade, h.: formal handling of threats and rewards in a negotiation dialogue. in: proceedings of the fourth international joint conference on autonomous agents and multiagent systems, pp. 529–536. acm (2005) google scholar  amgoud, l., prade, h.: handling threats, rewards, and explanatory arguments in a unified setting. int. j. intell. syst. 20(12), 1195–1218 (2005) article  google scholar  amgoud, l., prade, h.: formal handling of threats and rewards in a negotiation dialogue. in: parsons, s., maudet, n., moraitis, p., rahwan, i. (eds.) argmas 2005. lncs (lnai), vol. 4049, pp. 88–103. springer, heidelberg (2006). https://doi.org/10.1007/11794578_6 chapter  math  google scholar  belardinelli, f., grossi, d., maudet, n.: a formal analysis of dialogues on infinite argumentation frameworks. in: 24th international joint conference on artificial intelligence (ijcai-15), pp. 861–867 (2015) google scholar  bentahar, j., moulin, b., chaib-draa, b.: specifying and implementing a persuasion dialogue game using commitments and arguments. in: rahwan, i., moraïtis, p., reed, c. (eds.) argmas 2004. lncs (lnai), vol. 3366, pp. 130–148. springer, heidelberg (2005). https://doi.org/10.1007/978-3-540-32261-0_9 chapter  google scholar  boella, g., hulstijn, j., van der torre, l.: persuasion strategies in dialogue. in: proceedings of the ecai workshop on computational models of natural argument (cmna 2004), valencia (2004) google scholar  calvaresi, d., et al.: multi-agent systems’ negotiation protocols for cyber-physical systems: results from a systematic literature review. in: icaart (1), pp. 224–235 (2018) google scholar  dimopoulos, y., moraitis, p.: advances in argumentation based negotiation. in: negotiation and argumentation in multi-agent systems: fundamentals, theories, systems and applications, pp. 82–125 (2011) google scholar  dung, p.m.: on the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. artif. intell. 77(2), 321–357 (1995) article  mathscinet  google scholar  heras barberá, s.m., botti navarro, v.j., julian inglada, v.j.: case-based argumentation framework. dialogue protocol (2011) google scholar  hunter, a.: base logics in argumentation. in: comma, pp. 275–286 (2010) google scholar  ito, t., klein, m., hattori, h.: a multi-issue negotiation protocol among agents with nonlinear utility functions. multiagent grid syst. 4(1), 67–83 (2008) article  google scholar  letia, i.a., vartic, r.: defeasible protocols in persuasion dialogues. in: proceedings of the 2006 ieee/wic/acm international conference on web intelligence and intelligent agent technology, pp. 359–362. ieee computer society (2006) google scholar  mcburney, p., parsons, s.: dialogue games for agent argumentation. in: simari, g., rahwan, i. (eds.) argumentation in artificial intelligence, pp. 261–280. springer, boston (2009). https://doi.org/10.1007/978-0-387-98197-0_13 modgil, s., prakken, h.: the aspic+ framework for structured argumentation: a tutorial. argument comput. 5(1), 31–62 (2014) article  google scholar  morveli-espinoza, m.: persuasive negotiation dialogues using rhetorical arguments. in: proceedings of the 16th conference on autonomous agents and multiagent systems, aamas 2017, pp. 1845–1846. international foundation for autonomous agents and multiagent systems (2017) google scholar  morveli-espinoza, m., nieves, j.c., tacla, c.a.: measuring the strength of threats, rewards, and appeals in persuasive negotiation dialogues. knowl. eng. rev. 35, 1–27 (2020) article  google scholar  morveli espinoza, m., possebom, a.t., tacla, c.a.: on the calculation of the strength of threats. knowl. inf. syst. 62(4), 1511–1538 (2019). https://doi.org/10.1007/s10115-019-01399-2 article  google scholar  perrussel, l., doutre, s., thévenin, j.-m., mcburney, p.: a persuasion dialog for gaining access to information. in: rahwan, i., parsons, s., reed, c. (eds.) argmas 2007. lncs (lnai), vol. 4946, pp. 63–79. springer, heidelberg (2008). https://doi.org/10.1007/978-3-540-78915-4_5 chapter  google scholar  prakken, h.: coherence and flexibility in dialogue games for argumentation. j. logic comput. 15(6), 1009–1040 (2005) article  mathscinet  google scholar  prakken, h.: models of persuasion dialogue. in: argumentation in artificial intelligence, pp. 281–300. springer, boston (2009). https://doi.org/10.1007/978-0-387-98197-0_14 rahwan, i., ramchurn, s.d., jennings, n.r., mcburney, p., parsons, s., sonenberg, l.: argumentation-based negotiation. knowl. eng. rev. 18(04), 343–375 (2003) article  google scholar  ramchurn, s.d., jennings, n.r., sierra, c.: persuasive negotiation for autonomous agents: a rhetorical approach (2003) google scholar  saha, s., sen, s.: an efficient protocol for negotiation over multiple indivisible resources. in: ijcai, vol. 7, pp. 1494–1499 (2007) google scholar  sierra, c., jennings, n.r., noriega, p., parsons, s.: a framework for argumentation-based negotiation. in: singh, m.p., rao, a., wooldridge, m.j. (eds.) atal 1997. lncs, vol. 1365, pp. 177–192. springer, heidelberg (1998). https://doi.org/10.1007/bfb0026758 chapter  google scholar  van veenen, j., prakken, h.: a protocol for arguing about rejections in negotiation. in: parsons, s., maudet, n., moraitis, p., rahwan, i. (eds.) argmas 2005. lncs (lnai), vol. 4049, pp. 138–153. springer, heidelberg (2006). https://doi.org/10.1007/11794578_9 chapter  math  google scholar  wang, g., wong, t., wang, x.: a negotiation protocol to support agent argumentation and ontology interoperability in mas-based virtual enterprises. in: 2010 seventh international conference on information technology: new generations (itng), pp. 448–453. ieee (2010) google scholar  download references author information authors and affiliations program in electrical and computer engineering (cpgei), federal university of technology of parana (utfpr), curitiba, brazil mariela morveli-espinoza & cesar augusto tacla paraná federal institute of education, science and technology (ifpr), paranavai, brazil ayslan possebom authors mariela morveli-espinozaview author publications search author on:pubmed google scholar ayslan possebomview author publications search author on:pubmed google scholar cesar augusto taclaview author publications search author on:pubmed google scholar corresponding author correspondence to mariela morveli-espinoza . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper morveli-espinoza, m., possebom, a., tacla, c.a. (2021). a protocol for argumentation-based persuasive negotiation dialogues. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_2 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords persuasive negotiation intelligent agents dialogue protocol argumentation-based dialogues rhetorical arguments publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a graph-based crossover and soft-repair operators for the steiner tree problem | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a graph-based crossover and soft-repair operators for the steiner tree problem conference paper first online: 28 november 2021 pp 111–125 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a graph-based crossover and soft-repair operators for the steiner tree problem download book pdf download book epub giliard almeida de godoi  orcid: orcid.org/0000-0002-1715-085210, renato tinós  orcid: orcid.org/0000-0003-4027-885111 & danilo sipoli sanches  orcid: orcid.org/0000-0002-8972-522110  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 763 accesses 2 citations 2 altmetric abstract in the steiner tree problem in graphs, a subset of nodes, called terminals, must be efficiently interconnected. the graph is undirected and weighted, and candidate solutions can include additional nodes called steiner vertices. the problem is np-complete, and optimization algorithms based on population metaheuristics, e.g., genetic algorithms (gas), have been proposed. however, traditional recombination operators may produce inefficient solutions for graph-based problems. we propose a new recombination operator for the steiner tree problem based on the graph representation. the new operator is a kind of partition crossover: it breaks the graph formed by the union of two parents solutions and decomposes the evaluation function by finding connected subgraphs. we also investigate two soft-repair operators that produce small changes in candidate solutions: an mst transformation and a pruning repair. the ga with the proposed crossover operator was able to find the global optimum solution for all tested instances and has a success rate of 28% in the worst case. the experiments with the proposed crossover presented a quick convergence to optimal solutions, and an average solution cost lower in most cases than other approaches. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others closeness centrality reconstruction of tree graphs article 27 january 2024 robust reoptimization of steiner trees article open access 27 january 2020 genetic algorithm incorporating group theory for solving the general travelling salesman problem article 22 november 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. class switch recombination dna recombination dna repair enzymes graph theory optimization recombinases 1 introduction steiner tree problems (stp) are relevant in the combinatorial optimization field [9]. in stp, a minimum special tree should be found in a weighted graph g(v, e), where an edge \(e_{ij} \in e\) represents the relationship between vertices \(v_i\) and \(v_j \in v\), and a function \(c: e \rightarrow \mathbb {r}^{+}_{*}\) maps each edge to a positive value. the graph is also undirected, i.e., we can reach a node \(v_j\) from \(v_i\) and vice-versa by the same edge and cost. moreover, suppose there is a subset of requested vertices \(z \subset v\), also called terminals. the steiner tree problem in graphs (stpg) seeks the steiner minimum tree (smt), a subgraph of g that connects all the terminals nodes in z and minimize eq. 1, i.e., has the minimum cost. $$\begin{aligned} f(t) = \sum _{i=1} c(e_i) \quad \forall \ e_i \in e' \subset e \end{aligned}$$ (1) a steiner tree (st) is a candidate solution for the problem with \(z \subset v'\) and \(e' \subset e\) forming a tree \(t(v', e')\). some non-terminal vertices from the set \(v z\) might appear in \(v'\) to assure connectivity they are steiner vertices. an st must follow these constraints: 1) the terminal nodes must be connected; 2) any cycle must be broken to reduce the solution cost; 3) leaf nodes that are not terminals must be removed, reducing the solution cost without lost connectivity. hamiki (1971) [4] first described the sptg for a finite graph with vertices and well-defined edges. the stpg is a well-known np-complete problem [6] in the combinatorial optimization field. it means no known algorithm assures that deterministically can find the best possible solution in a polynomial time for the general case. as a consequence, heuristics and metaheuristics have been proposed for the stpg. the stpg can model a comprehensive range of practical problems. several studies describe applications in the telecommunication and related fields as multicast routing problems and wireless sensor networks. however, it is also applicable in supply chain management, vsli circuit projects, genetic phylogeny, and drug repositioning. due to its applicability, several studies attempted to find better solutions for the stpg. ljubic (2021) [9] describes different approaches for special cases of the stpg: lagrangian relaxations, dual-ascent methods, heuristics, approximate and exact algorithms. the author in [9] pointed out that metaheuristics also has been developed, like genetic algorithms (ga), ant colony optimization, and particle swarm optimization. however, metaheuristics are not competitive in many practical steiner tree benchmark problems where traditional approaches have been applied. the worst results, when compared to conventional approaches, can be partially credited to the use of transformation operators that end up producing inefficient or unfeasible solutions in graph-based representations of the problem. in addition, the interaction between decision variables is ignored in black-box optimization. alternatively, in gray-box optimization, the structure of the problems is efficiently explored by transformation operators to produce better solutions [14]. partition crossover (px) operators explore the interaction between decision variables in problems with graph-based representation to recombine solutions efficiently. px operators break the union graph formed by two parents’ solutions and decompose the problem’s evaluation function. for instance, the generalized partition crossover (gpx) improved the solution cost for the traveling salesman problem (tsp) when it was combined with the state-of-art lin-kernighan-helsgaum heuristic [13]. this operator is respectful because the offspring inherits all common edges from its parents. it also transmits alleles, which means all edges present in the offspring can be found in the parents. moreover, when two local optima are recombined, the offspring is also local optima. accordingly to raidl et al. (2003) [11], an efficient crossover operator for tree-graph problems is expected to be respectful and transmit alleles. in this paper, we propose a new recombination operator for the stpg based on partition crossover. it is important to observe that px operators were not previously proposed for steiner tree problems. in addition, we use two soft-repair operations to repair unfeasible solutions eventually generated by recombination. the proposed operator is compared to a prim’s random-based crossover. the ga with the proposed recombination and repair operators is also compared to a classical ga using binary chromosome representation. this paper is organized as follows: sect. 2 describes ga approaches for the stpg and some expected features for an efficient crossover operator. section 3 explains our partition crossover for steiner trees. section 4 describes the experiments, subsect. 4.1 gives more detail about ga tested, and subsect. 4.2 discusses results. finally, a conclusion is given in sect. 5. 2 related works in two early studies, kapsalis et al. (1993) [5] and esbensen (1995) [3] proposed a ga with a binary chromosome to encode an st. the binary chromosome indicates if a non-terminal node (from the set \(v z\)) belongs to an st. if it belongs, it is set the value one to the corresponding node. otherwise, it will set a value of zero if the node does not belong. the algorithms in [5] and [3] differ by the process of decoding the st from the chromosome. the decoding procedure in [5] first identifies the steiner vertices from the chromosome and the terminals nodes that form the set of nodes \(v' \subset v\) for the st. then, it adds an edge \( e_{ij} = (v_i, v_j) \in e\) to the subgraph \(g'\) if both incidents vertices \(v_i\) and \(v_j\) belongs to \(v'\). if so, the edge will also belong to \(g'\). finally, the st is the minimum spanning tree (mst) from the subgraph \(g'\). the decoding procedure adopted by [5] can lead to disconnected individuals. however, their fitness evaluation receives a penalization proportionally to the number of disconnected components. they also used traditional ga operators such as 1 point crossover and bit-flip mutation. the ga found the best solution for all or-library b class instances in at least one algorithm run. however, the authors applied a diverse range of parameters configuration to achieve the results, and the ga required a long running time. esbensen (1995) [3] embodied the distance network heuristic (dnh) [7] to avert disconnected solutions in his decoding procedure. the heuristic uses the distance graph (representing all paths and distances between every pair of vertices) obtained by the floyd-wharsall algorithm [2]. even though it is computed once, its time complexity of \(o(n^3)\) might prevent its use for large graphs. the decoding procedure computes the st by applying the heuristic over the subgraph \(g'\) induced by the chromosome subset of vertices \(v'\). the author in [3] applied his ga in c, d, and e instance classes from or-library [1]. the author also used reduction techniques to reduce the size of the instance graphs. reduction techniques try to identify edges and vertices that always will belong to an st—contracting those pairs of vertices and edges—or identifying those that will never be part of an st and remove them. however, the ga achieved a near-optimum solution and took a long time running. raidl et al. (2003) [11] presented an edge set encoding for tree-based optimization problems. in this approach, just the set of edges represents the tree. they tested it considering different tree-based problems, but the adaptation to stpg is simple. they also proposed three randomized strategies for population initialization, based on prim [10], kruskal [8], and random walk algorithms. the random walk starts at an arbitrary node \(v_{start}\) and moves a particle to some adjacent node \(v_k\). the edge \((v_{start}, v_k)\) is selected to compose the resulting tree. then the particle moves to another adjacent node of \(v_k\) and inserts a new edge into the resulting tree if the selected edge does not insert a cycle in the tree. this procedure is repeated until all terminal nodes belong to the individual. the authors in [11] proved that this procedure does not favor a specific tree topology. prim (1957) [10] and kruskal (1956) [8] have proposed two well-known algorithms to compute the mst from a graph. these algorithms greedily choose the lowest cost edges to form the mst, but they differ in selecting such edges. kruskal’s algorithm looks at the entire edges’ set and selects one even though it does not connect with the previously chosen. therefore, this algorithm uses a disjoint set data structure to ensure that the selected edges will not form a cycle. instead, prim’s algorithm starts from a node \(v_{0}\), considers its incident edges as eligible, and chooses some edge \((v_0, v_j)\) with the lowest cost. the new incident edges from \(v_j\) will be eligible too. the set of eligible edges keeps growing until all graph edges are reached. as a procedure to population initialization, the authors in [11] proposed a variation of prim and kruskal algorithms to select a new edge randomly instead of in a greedy way. furthermore, the authors claimed that these procedures favor a star topology in the individuals. hence, they named these variations as primrst and kruskalrst. also, in [11] the initialization procedures are adapted as recombination procedures by simply applying them to the union graph of the two candidate solutions. in addition, they stated that other recombination strategies are possible depending on what to do with the shared edges, transmitting them directly to the offspring or not. however, they conclude that recombination operators for spanning trees should always include the common edges in the offspring. the authors in [11] stated that these operators offer locality (small changes in the genotype imply small changes in the phenotype) and heritability (the offspring consist mainly of alleles of its parents). these are two of many properties expected for proper encoding. however, to the best of our knowledge, those recombination strategies were not evaluated for the stpg case. additionally, there is no formal demonstration that primrst always includes all the common edges between two individuals chosen for recombination. in this paper, we propose a crossover operator for st based on px. it generates two offspring, which will have: the shared edges from both parents, the select edges from recombinant partitions, and the remaining edges from non-recombinant partitions, respectively, from each parent. the principles behind our operator come from the px for the traveling salesman problem (tsp) proposed in [12, 13, 15], and [16]. the original generalized partition crossover (gpx) was designed to recombine two hamiltonian cycles. under these circumstances, a new recombination operator is necessary for the steiner tree, i.e., recombine tree graph solutions. the gpx takes two individuals (hamiltonian cycles) for recombination. then, it joins the edges from each individual in a union graph \(g_u\). the vertices with a degree equal to 2 are removed, and their incident edges are transferred directly to the offspring. the remaining edges form candidate partitions for recombination. one can think each partition is the union of two paths, each one from a parent. the recombinant partition must ensure that it can ensemble a cycle in the offspring if either inner path is chosen. it is made by checking the portal and simplified graph for each parent graph. a simplified graph reduces a path to a single edge. a portal vertex connects a partition to another through common edges. so it will be easy to verify if the path from both parents connects to the same portal vertex. nevertheless, vertices with a degree equal to 4 are split by inserting a ghost vertex. the edges are then divided between each vertex–the original and the ghost one. fig. 1. an example of pxst crossover between two st, \(t_a\) and \(t_b\), showed in figs. 1a and 1b. figure 1c shows the resulting graph after removing the common edges from the union \(t_a \cup t_b\); the resulting graph has only one connected component. figure 1d shows the common elements (vertices and edges) between \(t_a\) and \(t_b\). they will be used to explore more partitions. pxst represents common elements as a disjoint set, where vertices are linked if they share a common edge (steps 1, 2, and 3, of algorithm 1). figure 1e shows two cycles identified in the first iteration. the distinct paths of non-common edges are connected to the same components of common elements. then, we can choose the best path. the selected path is added to the common elements, and two more cycles are then identified in the second iteration (fig. 1f). again, the respective paths for each parent are compared, and the best paths are chosen. figure 1g shows the offspring. all the partitions are feasible for recombination in this example, but this may not be the case for some pairs of solutions (st). full size image 3 proposed operator as mentioned before, the original gpx is not suitable for the stpg case due to differences in candidate solutions’ topology. algorithm 1 shows the proposed recombination operator based on partition crossover (pxst). both operators (the original and the proposed one) keep some resemblance. the offspring will inherit all the common edges between the parents. uncommon edges will form partitions, and some tests will identify the comparable ones, also called recombinant partitions. for such partitions, those with the lowest cost will compose the offspring. moreover, if there are k recombining components, the operator will generate the best of \(2^k\) potential offspring. furthermore, the pxst’s operation can be divided into four phases: identifying common elements (vertices and edges); identifying partitions candidates to recombination; matching, selecting, and updating the recombinant partitions; handle partitions unfeasible to recombination and return the offspring. each phase will be detailed as follows. a) identifying common elements. the common vertices and edges between two st, play an essential role in the pxst. they work as a cutting point, increasing the number of partitions and will help to identify recombinant partitions. the vertices from the union graph between two hamiltonian cycles will have degrees equal to 2, 3, or 4, and gpx explores it. first, the operator removes the vertices with a degree equal to 2. the vertices with a degree equal to 3 belong to some partition. moreover, the insertion of a ghost vertex will separate the edges adjacent to vertices with a degree equal to 4, leading to more partitions. unlike, the st inner vertices do not have such degree constraints. an inner node can have any number of adjacent edges, and only the leaves must have a degree equal to one. consequently, the union graph between two st solutions might have more connections within its vertices, leading to fewer independent components. figures 1a and 1b show two examples of st, \(t_a\) and \(t_b\). figure 1c shows why the strategy adopted by gpx is not suitable for the stpg. figure 1c the resulting graph obtained by removing the common edges from the union between \(t_a\) and \(t_b\) has only one connected component. in this simple case, choosing either one or another parent’s edges did not result in a new individual. figure 1d shows the common edges and vertices between \(t_a\) and \(t_b\). notice that the vertices 15, 61, and 47 will act like cutting points, breaking the component shown in fig. 1c in smaller partitions. moreover, there are two sets of common edges: one formed by the edge \(\{(5, 9)\}\) and the other formed by the edges \(\{(55, 12), (12, 11), (23, 11), (17, 11)\}\). also, notice how candidate partitions connect to these sets by different vertices. one first test to identify a recombinant partition might be to check when two distinct partitions (one from each parent) connect to the same components of common elements, i.e., a set of connected edges or an isolated vertex. to keep track of such common elements, pxst uses a disjoint set (ds) data structure. it allows three operations: the “make a set” initializes the disjoint set; the “find” operation returns the same representative element for any other element in the set; the “union” operation joins two sets and defines a new element to represent the new set. an efficient implementation of the union operation (using union by rank and path compression) has \(o(m \log n)\) time complexity [2]. step 1 from algorithm 1 firstly identifies the common vertices between the parent’s solutions. then, step 2 initializes ds by creating a unitary set for all common vertices. finally, ds’s initial unitary sets are jointed when a common edge connects two distinct vertices (step 3). notice that the set of common edges (showed in fig. 1d) will be represented as a disjoint set, i.e., unique key-vertex. another detail is that some individuals might have leaf nodes that are not terminals. some of these leaf nodes might belong to just one of the parent’s solutions. moreover, such leaf nodes will become a portal vertex for one partition. since checking portal vertices will allow identifying recombinant partition, such leaves will be included in the ds, although they may not be common (step 1). b) identifying candidate partitions to recombination. the pxst identifies partitions directly on the graphs \(t_a^*\) e \(t_b^*\) (step 4). they are the resulting graphs from deleting the common edges from \(t_a\) and \(t_b\), respectively. removing an edge from a tree results in two subtrees. therefore, \(t_a^*\) and \(t_b^*\) are compound by many subtrees. however, those subtrees are not partitions yet. they still will be divided into smaller parts taking the common vertices as cutting points. a depth-first search looks for partitions in \(t_a^*\) e \(t_b^*\), separately (step 5). a partition is formed mainly by non-common elements, where all its internal vertices and edges belong to one of the solutions. since the common vertices work as cutting points, where the partitions are breaking, these vertices are called portals. figure 1 shows partitions only with two portal vertices. however, a partition with more than two portal vertices is possible when an internal node has a degree greater than two, i.e., the partition has a subtree topology. c) matching, selecting, and updating the recombinant partitions. for instance, consider two sts with common vertices \(u, v \in t_a \cap t_b\). pretend that \(t_a\) has a partition with a path-like topology \(p_a\) and \(t_b\) has a \(p_b\) partition. both \(p_a\) and \(p_b\) connect u and v by different edges. the union between \(p_a\) and \(p_b\) will form a cycle closing in the vertices u and v. moreover, u and v are the portal vertices for these partitions. alternatively, the portal vertices for the partitions are coincident, and one can select the partition with the lower cost. however, figs. 1e and 1f show that partitions can connect to the same component of common edges by different vertices. then, the partition will not have coincident portal vertices. so, a more refined way to find recombinant partitions is needed. one can argue that if we shrink all common edges to a single vertex, we might get the well-formed cycle as mentioned earlier. for example, in fig. 1e, vertices 12 and 23 will become coincident, and the partitions will form a cycle between nodes 15 and 12 (or 23). in the same way, shrinking the edge (5, 9) will close another cycle with the path-like partitions up to node 17. then, we will be able to compare such distinct paths. instead of shrinking common edges and checking the exact correspondence between portal vertices from distinct partitions, we check if they are mapped to the same representative nodes by the disjoint set (step 6). that indicates the partitions are connected to the same components of common elements (fig. 1d), and they are interchangeable. then, the operator can choose between the partitions without losing the offspring connectivity or adding a loop into it. when a partition is liable for recombination, the disjoint set from each portal vertex can be joined, expanding the original disjoint set (common elements set). this update is helpful to discover new recombination opportunities, and it will be performed until there are no new recombinant components (step 9). for instance, fig. 1f shows other opportunities for recombination. in the first one, we can choose the path from node 61 up to 15 or from 61 to 55. however, to compare these paths, it is necessary to resolve the connectivity to node 15 first (shown in fig. 1e). when we resolve the best connectivity up to node 15, this node will belong to the same disjoint set with nodes 12, 23, and 55. the portal vertices 15 and 55 will be mapped to the same arbitrary representative element, and the operator can decide the best path to node 61. a similar situation occurs to the partitions connecting node 47. firstly, we must decide the best connectivity up to node 9 to include this node to the same disjoint set as portal vertex 12. some partitions might have equal cost even though they do not have the same edges. in such cases, the selection occurs randomly, with equal probability for each partition. when their total cost is different, pxst chooses the one with a lower cost. thus, the partition’s selected edges will belong to both offspring. d) handle unfeasible partitions to recombination and return offspring. by the end of this process, some non-recombinant partitions might persist for both parents. one alternative is to choose the remaining partitions from one of the parents with the lowest sum cost. in preliminary tests, this strategy leads to an early population stagnation and higher solution cost outcomes. another option is to return two offspring, one for each parent’s remaining partitions (step 10). therefore, we chose this last alternative as it showed to lead to better outcomes. new ways for identifying recombinant partitions can further be developed. it may improve the operator’s efficiency and lead to better solutions. table 1. average and standard deviation of the best solution found along with the runs of each experiment. the symbols ‘\(=\)’, ‘\(+\)’ or ‘−’ indicate if the average cost is equal, better, or worse than ga with pxst. the letter s indicates if the differences are statistically significant. properties of each instance are also presented (number of vertices |v|, number of edges |e|, number of terminals vertices |z| and the optimum solution cost (opt.).full size table 4 experiments this section describes the experiments comparing the proposed operator (pxst), the primrst crossover operator [11] and a binary ga based on [5]. the section also describes soft-repair operators used in our algorithm. in the experiments, three algorithms were compared: pxst+repair(pruner, mst): ga with the proposed operator pxst (algorithm 1), mutation that replaces edges randomly, and the pruning and mst repair strategies. primrst+repair(pruner, mst): ga with the primrst crossover operator [11] and pruning and mst repair strategies. binary ga: ga with binary chromosome [5]. it used a traditional 2 point crossover operator and bit-flip mutation. 4.1 experimental design the algorithms were tested over 18 instances of class b from the or-library [1]. table 1 presents the properties for each instance and statistics about the solution cost discussed later. in all three experiments, the same stop condition was adopted. the execution stops if the algorithm finds the global optimum solution, if the best solution is not improved within an interval of 500 fitness evaluations, or if the number of iterations reaches 4.000. however, this last condition was never reached. the experiments with pxst and primrst differ only by the crossover operator. they have the same parameters and underlying mechanisms for selection (roulette) and mutation. an adjacency list data structure represents the st. the population size is 100 individuals, initialized by a heuristic based on the random walk procedure presented in [11]. this initialization strategy generates only connected individuals. the mutation by replacing a random edge (rre) is presented in [11]. the operator removes a random edge from the tree producing two disconnected subtrees. then, it reconnects them by inserting a new edge selected randomly from the problem instance graph. the mutation rate is fixed to 0.3. this operator produces a slight change in the st since it replaces just one edge when possible. the primrst crossover takes two individuals and joins them in a graph \(g_u\). like the original prim’s algorithm, the operator computes a spanning tree from \(g_u\) (see sect. 2). instead of taking the lowest cost edge, the operators insert a new edge randomly – of course, if the new edge does not form a cycle with the previously selected edges. the algorithm stops when all terminal vertices are inserted in the offspring or when all edges from \(g_u\) were processed. the offspring’s edges generated by primrst came only from \(g_u\), i.e., the operator does not insert a new edge that does not belong to one of the parents. some common edges will appear in the offspring, although there is no proof that all common edges will be transmitted every time. in that regard, both primrst and pxst transmit alleles (edges) from the parent to the offspring. however, pxst transmits all of the common edges to the offspring, and for primrst, this is not guaranteed. moreover, primrst favors the edge exchange between the parents, while pxst looks for connected components. primrst and pxst generate connected individuals. still, some offspring might not be completely st because some of them might have leaf nodes that are not terminal. thereby, a pruning repair iteratively seeks and removes leaf nodes that are not terminal. it reduces the solution cost without losing the offspring’s connectivity. thus, the pruning repair was constantly applied for all individuals over the generations (fitness evaluations). table 2. average and standard deviation of the execution time (in seconds) and the success rate (sr), i.e., the percentage of execution where the best-known solution was found. the symbols \(=\), \(+\), and − indicate if the average result was equal, better or worse than the ga with pxst. the letter s indicates if the differences in the samples are statistically significant.full size table in [5] and [3] a solution is obtained computing the mst of the induced subgraph from steiner vertices represented in the chromosome. similarly, an st can be improved by replacing some of its edges with lowest cost edges that span the solution vertices, computed from the induced mst subgraph. for that reason, an mst-based repair is applied. the mst-based repair operator takes one individual \(t_k(v_k, e_k)\) and determines a subgraph induced from its vertices \(v_k\). the induced subgraph has all edges from the individual plus the edges from the problem instance graph when both incident vertices are in the individual’s vertex set. then, it applies prim’s algorithm to compute the mst from the induced subgraph. this repair operator is used with a fixed rate of 0.3. since pxst and primrst operators do not produce disconnected individuals, repair operations to reconnect the candidate solution or a penalty function (as in [5]) are useless. the binary ga is based on [5] (see sect. 2). a binary chromosome represents the steiner vertices that belong to some st. a decoding procedure based on computing the mst tree from the induced graph does not assure the candidate solution’s connectivity. therefore, a penalty function increases the fitness of infeasible solutions (disconnected ones). the ga used a population size of 100 individuals. the binary chromosome is randomly initialized, and the genetic operators are roulette selection, a traditional crossover operator with 2 points cut, and a mutation by bit flip by a rate of 0,2. the operators and ga implementations were in python language, version 3.7.9, and the code is available publicly.footnote 1 experiments were conducted in a virtual machine set up in google cloudtm platform, using linux ubuntu os (version 18.4), 32 gb of ram, and cpu with 8 cores intel cascade lake. each experiment runs 50 times, with a seed randomly defined by the system. 4.2 experimental results and analysis table 1 presents the instances graphs features alongside the average and standard deviation for the best solution cost found in all executions. the values in bold indicate the lower average cost. table 2 shows the average and standard deviation of execution time (in seconds) and the success rate (sr), i.e., the proportion of runs where the algorithm found the global optimum solution. for the average cost and execution time results, we applied the wilcoxon signed-rank test to verify if the differences in the samples are statistically significant. we adopted a significance level of \(\alpha =0.05\). the ga with pxst found the global optimum solution for all instances in at least one run. this ga found the global optimum solution for all execution in five instances (b1, b2, b3, b8, and b9). for the other six cases (b11, b13, b14, b15, b16, and b18), the ga obtained the lowest average solution cost, as showed in table 1. moreover, ga with pxst has an sr in the worst case of 28% (instance b15), as shown in table 2. in all other cases, it has an sr above 30%, indicating certain robustness of the algorithm. the ga with primrst found the global optimum solution in all runs for seven instances: b2, b3, b5, b7, b9, b10, and b17. it obtained the lowest average solution cost for the b4 instance. significant improvements are observed for b4, b10, and b17. this ga could not find the optimum solution in any run for b14, b15, and b16 and obtained a low sr for instances b6 (24%) and b12 (2%) (table 2). the statistical test does not indicate significant differences for the best cost sample to b1, b2, b3, b5, b7, b8, b9, b11, and b18 between pxst and primrst experiments. for example, ga with primrst found the global optimum solution, for instance, b5 in all runs (sr 100%) – note that the average best cost is equal to the global optimum, and the standard variation is zero. in contrast, ga with pxst obtained 94% of the sr and an average best cost of 61.10 and a standard deviation of 0.46. however, the test indicates that the difference observed in the best cost sample was not significant for the b5 instance. binary ga obtained a lower best cost average for instances b6 and b12. however, the differences for best cost are statistically significant just for b6. all of the three ga solved the instance b3 in all runs. for all other 15 instances, the best cost average for binary ga was worse than the ga with pxst. this result can be credited to the use of specific operators for tree-graph problems rather than traditional operators that produce infeasible (disconnected) solutions. the ga with pxst was faster for fourteen instances (see table 2). moreover, ga with pxst was, in general, faster than primrst for those instances where the average best cost was similar, i.e., differences were not statistically significant. that is the case for b1, b2, b3, b5, b8, b9, and b11 (except b7). this result is because the crossover pxst is greedy and converges quickly to a local optimum solution. the ga with primrst was faster when it found the global optimum solution in all runs (b7, b10, and b17) and in b4, where the sr is 86%. pxst and primrst ga took less than a second on average to solve some instances (see b2, b3, b7, b8, b9, and b10). as expected, binary ga usually took a long time to run on average, as showed in table 2. for b6, binary ga was faster than the other two gas. compared with ga with pxst, the execution time is similar (not statistically significant) for b6, b15 and b18, and worse for all other instances. both pxst and primrst operators do not rely on high complexity cost heuristic as [3]. hence, they might be suitable for large instance graphs in an appropriate execution time. 5 conclusions in this paper, we presented a graph-based crossover for the stpg. pxst is based on partition crossover, firstly proposed to the tsp problem. the main adaptation is identifying partitions since the candidate solutions are tree-graphs, in this particular case. two soft-repair operators were applied in the ga using a graph representation of the individuals. the first repair relates to the property that the stpg can be solved by determining the mst of an appropriate set of steiner vertices. an mst transformation calculates the mst spanning the vertices for a given st. in this case, many edges can be replaced by lower-cost edges. then, the ga would not necessarily explore all possible st. a second repair operation pruned non-terminal leaves, reducing the solution cost without losing the tree connectivity. the repair operator is applied since pxst and primrst might generate individuals with such leaves. in this first study, we do not analyze the impact of varying the numeric parameters (mutation or repair rate, for example). compared with two other ga for the stpg, the one using the pxst was faster for almost all instances. it also reached the global optimum solution for all instance problems with a success rate above 30% in general, considering 50 executions of the experiment. pxst and primrst are similar considering the offspring are constituted only with edges of their parents, and common edges are transmitted to the offspring, most of them in the primrst case. the differences are that pxst does not delete edges or nodes present in the parents and exchanges more significant structures such as paths or subtrees. because of their similarities, both crossover operators are at least competitive in their results. the partition crossover operators, which pxst was based on, have the property of tunneling between optima [13]. it produces the best of \(2^k\) potential offspring, where k is the number of feasible partitions. thus, if two parents are local optima, the offspring will be a local optimum with a lower cost. considering that, pxst also can be used combined with other heuristics as in [13]. furthermore, the ga using pxst is greedy and quickly converges to local or global optima. a mechanism to control the population diversity might contribute to ga with pxst performance. in the pxst, some partitions remain unfeasible for recombination. further studies can improve the partition discovery or fusion procedures to find more opportunities for recombination and improve the operator’s efficiency. notes 1.https://github.com/giliardgodoi/ppgi-stpg-gpx. references beasley, j.e.: or-library: distributing test problems by electronic mail. j. oper. res. soc. 41(11), 1069–1072 (1990) article  google scholar  cormen, t.h., leiserson, c.e., rivest, r.l., stein, c.: algoritmos: teoria e prática. editora elsevier (2012) google scholar  esbensen, h.: computing near-optimal solutions to the steiner problem in a graph using a genetic algorithm. networks 26(4), 173–185 (1995). https://doi.org/10.1002/net.3230260403 article  math  google scholar  hakimi, s.l.: steiner’s problem in graphs and its implications. networks 1(2), 113–133 (1971). https://doi.org/10.1002/net.3230010203 article  mathscinet  math  google scholar  kapsalis, a., rayward-smith, v.j., smith, g.d.: solving the graphical steiner tree problem using genetic algorithms. j. oper. res. soc. 44(4), 397–406 (1993) article  google scholar  karp, r.m.: reducibility among combinatorial problems, pp. 85–103. springer, boston (1972). https://doi.org/10.1007/978-1-4684-2001-2_9 kou, l., markowsky, g., berman, l.: a fast algorithm for steiner trees. acta inf. 15(2), 141–145 (1981). https://doi.org/10.1007/bf00288961 article  mathscinet  math  google scholar  kruskal, j.b.: on the shortest spanning subtree of a graph and the traveling salesman problem. proc. am. math. soc. 7(1), 48–50 (1956). http://www.jstor.org/stable/2033241 ljubić, i.: solving steiner trees: recent advances, challenges, and perspectives. networks 77(2), 177–204 (2021) article  mathscinet  google scholar  prim, r.c.: shortest connection networks and some generalizations. bell syst. tech. j. 36(6), 1389–1401 (1957) article  google scholar  raidl, g.r., julstrom, b.a.: edge sets: an effective evolutionary coding of spanning trees. ieee trans. evol. comput. 7(3), 225–239 (2003). https://doi.org/10.1109/tevc.2002.807275 article  google scholar  tinós, r., whitley, d., ochoa, g.: generalized asymmetric partition crossover (gapx) for the asymmetric tsp. in: proceedings of the 2014 annual conference on genetic and evolutionary computation, gecco 2014, pp. 501–508. acm, new york (2014) google scholar  tinós, r., whitley, d., ochoa, g.: a new generalized partition crossover for the traveling salesman problem: tunneling between local optima. evol. comput. 28(2), 255–288 (2020). https://doi.org/10.1162/evco_a_00254, pmid: 30900928 whitley, d.: next generation genetic algorithms: a user’s guide and tutorial. in: gendreau, m., potvin, j.-y. (eds.) handbook of metaheuristics. isorms, vol. 272, pp. 245–274. springer, cham (2019). https://doi.org/10.1007/978-3-319-91086-4_8 chapter  google scholar  whitley, d., hains, d., howe, a.: tunneling between optima: partition crossover for the traveling salesman problem. in: proceedings of the 11th annual conference on genetic and evolutionary computation, gecco 2009, pp. 915–922. acm, new york (2009) google scholar  whitley, d., hains, d., howe, a.: a hybrid genetic algorithm for the traveling salesman problem using generalized partition crossover. in: schaefer, r., cotta, c., kołodziej, j., rudolph, g. (eds.) ppsn 2010. lncs, vol. 6238, pp. 566–575. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-15844-5_57 chapter  google scholar  download references author information authors and affiliations universidade tecnológica federal do paraná – cornélio procópio, paraná, brazil giliard almeida de godoi & danilo sipoli sanches universidade de são paulo (usp) – ribeirão preto, são paulo, brazil renato tinós authors giliard almeida de godoiview author publications search author on:pubmed google scholar renato tinósview author publications search author on:pubmed google scholar danilo sipoli sanchesview author publications search author on:pubmed google scholar corresponding author correspondence to giliard almeida de godoi . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de godoi, g.a., tinós, r., sanches, d.s. (2021). a graph-based crossover and soft-repair operators for the steiner tree problem. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_8 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords steiner tree problem in graphs partition based crossover genetic algorithms publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature feature selection and hyperparameter fine-tuning in artificial neural networks for wood quality classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper feature selection and hyperparameter fine-tuning in artificial neural networks for wood quality classification conference paper first online: 12 october 2023 pp 323–337 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) feature selection and hyperparameter fine-tuning in artificial neural networks for wood quality classification download book pdf download book epub mateus roder  orcid: orcid.org/0000-0002-3112-52909, leandro aparecido passos  orcid: orcid.org/0000-0003-3529-31099, joão paulo papa  orcid: orcid.org/0000-0002-6494-75149 & … andré luis debiaso rossi  orcid: orcid.org/0000-0001-6388-747910  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 525 accesses 1 citation abstract quality classification of wood boards is an essential task in the sawmill industry, which is still usually performed by human operators in small to median companies in developing countries. machine learning algorithms have been successfully employed to investigate the problem, offering a more affordable alternative compared to other solutions. however, such approaches usually present some drawbacks regarding the proper selection of their hyperparameters. moreover, the models are susceptible to the features extracted from wood board images, which influence the induction of the model and, consequently, its generalization power. therefore, in this paper, we investigate the problem of simultaneously tuning the hyperparameters of an artificial neural network (ann) as well as selecting a subset of characteristics that better describes the wood board quality. experiments were conducted over a private dataset composed of images obtained from a sawmill industry and described using different feature descriptors. the predictive performance of the model was compared against five baseline methods as well as a random search, performing either ann hyperparameter tuning and feature selection. experimental results suggest that hyperparameters should be adjusted according to the feature set, or the features should be selected considering the hyperparameter values. in summary, the best predictive performance, i.e., a balanced accuracy of 0.80, was achieved in two distinct scenarios: (i) performing only feature selection, and (ii) performing both tasks concomitantly. thus, we suggest that at least one of the two approaches should be considered in the context of industrial applications. the authors are grateful to fapesp grants #2016/06538-0, #2018/02822-1 and #2019/07825-1. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others artificial intelligence-driven timber wood defect characterization from terahertz images article 13 october 2024 employing artificial neural networks for minimizing surface roughness and power consumption in abrasive machining of wood article 20 april 2016 production of high-quality forest wood biomass using artificial intelligence to control thermal modification article 26 april 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization learning algorithms machine learning optimization statistical learning wood science and technology 1 introduction industries have experienced many technological advances in recent years, resulting in more complex processes, systems, and products. as a consequence, the management of integrated manufacturing processes and operation analyses are crucial to delivering high-quality products to clients. in this scenario, the quality of raw materials is also paramount for high quality manufactured products. however, imperfection detection, as well as quality classification of raw materials, such as woods in sawmill companies, usually is still performed by trained human operators [1]. notwithstanding, the process is inherently subjective, since it is a visual analysis, and these experts may suffer from fatigue after a long working period performing repetitive activities. consequently, it is expected the increased number of incorrect classifications [25]. these disadvantages stimulated the scientific community towards the implementation of visual inspection systems, aiming to perform defect and quality classification autonomously. therefore, this work focuses exclusively on machine learning (ml) techniques that have been successfully employed for these tasks on wood boards [6]. recently, some studies investigated the performance of different ml techniques to classify the quality of wood surface [17, 21]. these studies have used data generated from images captured in a real sawmill company and classified by a specialist in three levels of quality, according to the company rules, i.e., zero defect is found in the wood piece (a), only small defects, such as knots, are found (b), and defects that compromise the quality of the product, such as groups of knots and exposed pith (c). figure 1 depicts some examples of wood images classified at each level. fig. 1. three different qualities of wood boards (a, b, and c) according to company’s rule. full size image despite the success obtained in these studies, each ml algorithm has its inherent tendency towards data specificity, which influences the model’s induction and, thus, its predictive performance. therefore, one can adjust such tendencies through a proper hyperparameter (hp) selection. the task of finding the best hp values is known as hyperparameter tuning and usually aims at improving the model’s predictive performance while keeping the model as simple as possible. although some hp values may fit sufficiently well different kinds of problems, it is a common practice to search for the hyperparameter that provides the best solutions concerning each problem at hand [17, 21]. besides, another challenge in the context of this work is extracting the images’ more representative features, i.e., the features that best describes the problem, since the more descriptive they are, the higher the effectiveness of the technique. regarding classification tasks, features are usually extracted through image descriptors, such as statistical measures from the gray level co-occurrence matrix (glcm) [8] and local binary patterns (lbp) [13]. however, many of these features may be correlated to each other or do not add any relevant information for the ml technique. therefore, the process of selecting a subset of these features, referred to as feature selection (fs), can be applied to select the most descriptive ones. moreover, since features are specific for each problem, fs has to be carried out for each data set separately. considering that both the hp tuning and the task of feature selection relies upon each other, they should be performed simultaneously to generate more robust models, concerning generalization purposes. in general, ml techniques require tuning more than one hp, since the tackled problems generally are described by many features, thus implying on large search spaces. in this context, metaheuristics approaches are commonly employed to solve such problems by randomly initializing a collection of candidate solutions, which interact among themselves and perform a directed exploration of the search space toward the results that best fit a desirable target function with an acceptable computational cost. such approaches are commonly employed to solve problems related to ml techniques hyperparameter tuning [5, 14, 19] and feature selection [15], among others [4, 22]. in this paper, we investigate the problem of fs and artificial neural network (ann) hyperparameter tuning applied in the context of wood boards quality classification. experiments were carried out using the population-based metaheuristic particle swarm optimization (pso) [9, 20] to simultaneously perform both tasks over a multilayer perceptron (mlp) ann. moreover, the results compared against five distinct baselines, as well as a random search, confirms the relevance of the proposed approach. we hypothesize that the predictive performance of ann models can be improved since they depend on the hp values and the set of features used to describe the problem. therefore, the main contributions of this paper are twofold: (i) to propose a method capable of simultaneously selecting the hyperparameters that best performs over an mlp network as well as selecting the subset of features that best describe each image sample, and (ii) to foster the scientific community regarding material and wood quality classification. the remainder of this paper is presented as follows. section 2 defines the problem of hyperparameter tuning and feature selection, and provides a brief description of some related works. section 3 presents the main concepts of ann and pso. the experimental methodology employed to evaluate the effects of fs and mlp hyperparameter tuning over the models’ performance is described in sect. 4. results are presented and discussed in sect. 5, and finally conclusions are presented in sect. 6. 2 problem definition and related work hyperparameter tuning and feature subset selection are two widely employed tasks carried out in the data mining context, aiming to improve models’ predictive performance as well as simplifying them. therefore, this section formalizes the problem of simultaneously performing these two tasks. besides, it presents an overview of studies related to wood quality classification and the importance of hp tuning and fs. 2.1 problem definition the problem investigated in this paper consists of tuning the hp values of an mlp artificial neural network algorithm, as well as selecting a subset of features that are relevant for the problem of wood quality classification towards the improvement of the models’ predictive performance. let a be an mlp algorithm that comprises the hyperparameter space \(\varlambda \). for each hyperparameter setting \(\lambda \in \varlambda \), let \(a_{\lambda }\) represent the learning algorithm a that employs the hyperparameter setting \(\lambda \). also, consider \(d = \{(x_1,y_1), (x_2,y_2), \ldots , (x_n,y_n)\}\) a dataset composed of n instances, such that \(\boldsymbol{x} \in \mathbb {r}^{m}\) is a feature vector and y is the target value. moreover, one can define \(\kappa \) as the subset of feature from \(\boldsymbol{x}\). finally, let \(a^{\kappa }\) be an algorithm a trained with a subset of features \(\kappa \) [12]. therefore, the main goal of hp tuning is to finding \(\lambda ^* = \text {arg min}_{\lambda \in \varlambda } m(a_\lambda , d) \in \varlambda \) that minimizes some loss function, such as the misclassification rate using the algorithm a over instances not used for training purposes. moreover, one can estimate the misclassification rate m(a, d) achieved by a when trained and tested on d through a stratified multi-fold cross-validation resampling method. similarly to hp tuning, the goal of feature subset selection is to find \(\kappa ^*=\text {arg min}_{\kappa \subseteq \boldsymbol{x}}m(a^\kappa , d) \subseteq \boldsymbol{x}\) which minimizes the loss function achieved by a when trained on d. therefore, the aim of combining hyperparameter tuning and feature subset selection is to find the hyperparameter setting \(\lambda ^* \in \varlambda \) and the feature subset \(\kappa ^* \subseteq \boldsymbol{x}\) which has the lowest misclassification rate among all hp settings and features subsets, i.e., \((\kappa ^*,\lambda ^*) = \text {arg min}_{\lambda \in \varlambda , \kappa \subseteq \boldsymbol{x}} m(a^{\kappa }_{\lambda }, d)\). 2.2 related work one of the first studies to investigate the problem of wood quality control using an automated visual inspection system based on machine learning techniques was accomplished by [16]. since then, many others have investigated this problem aiming to improve predictive performance by analyzing and selecting different features, as well as optimizing hp values of ml techniques [21, 24]. tiryaki et al. [24] employed an ann for modeling the wood surface roughness in the machining process. the study highlights some variables that have impact and influence in the surface roughness, such as wood species, the feed rate, the number of the cutter, and the cutting depth. the model’s predictive performance was good enough to allow its application in the wood industry in order to optimize effort, time, and energy. others addressed the problem of combining fs and hp tuning for ml techniques over different applications. as previously mentioned, some techniques are more sensitive to hp tuning and fs than others. besides, some optimization methods, such as metaheuristic approaches based on evolutionary algorithms and swarm intelligence [10], for instance, have successfully accomplished the task. in this context, roder et al. [21] used the pso algorithm to tune the hp of an ann applied for wood quality classification aiming to enhance the model’s predictive performance. the authors employed the glcm [8] to extract features from the same dataset used by [2], which is composed of five statistical measures for two angles, i.e., 0 and \(90^\circ \): entropy, energy, maximum intensity, inverse difference moment, and correlation. experimental results obtained up to \(6\%\) of accuracy gain concerning the ann classification, and corroborate the necessity of tuning the ann hyperparameters; stating the efficiency of pso for such a task. 3 theoretical background this section briefly introduces the main concepts of the techniques employed in this work, i.e., the multilayer perceptron neural network and the particle swarm optimization algorithm, as well as the process of wood image feature extraction. 3.1 multilayer perceptron ann mlps are composed of an input and an output layer, as well as one or more hidden layers of neurons, which can be fully or partially connected. a neural network is called fully connected when each neuron from a given layer is connected to all neurons of the next one. similarly, it is said to be partially connected when some neurons of adjacent layers are not connected. these connections are represented by a weight matrix, which is usually adjusted through gradient-based learning algorithms. with a single hidden layer, an mlp is capable of representing a large number of functions, thus sufficient for the purpose of this study. figure 2 depicts the model architecture. fig. 2. multilayer perceptron representation. full size image the conventional algorithm to train an mlp network is the backpropagation, which is composed of the forward pass, which exposes the input data to a series of linear operations followed by non-linear activations, and the backward phases, which is responsible for propagating the output error and update the network weights. 3.2 particle swarm optimization particle swarm optimization [11] is a global optimization technique based on the social behavior of birds, fishes, and insects, among others. the method comprises a swarm composed of a set of individuals capable of sharing information among themselves concerning their positions in the search space, as well as the relative quality of this position, denoted by a fitness function. in short, each pso particle is represented by its current position, velocity, and the best position found during the training process. the position of a particle i is represented by a point in a d-dimensional space, given by \(\varpsi _{i} = \{\psi _{i1},\psi _{i2}, \ldots ,\psi _{id}\}\). further, the particle velocity is defined by \(\boldsymbol{v}_i=\{v_{i1},v_{i2},\ldots , v_{id}\}\), and finally the best position found by this particle is represented by \(\boldsymbol{p}_i=\{p_{i1},p_{i2},\ldots ,p_{id}\}\). besides, the best position found among all particles is represented by \(\boldsymbol{p}_g\). a particle will move in a particular direction depending on its current position, velocity, and best position. additionally, it also depends on the best position found by the other particles in the swarm. therefore, the position of a particle \(\psi _{ij}(t+1) = \psi _{ij}(t) + v_{ij}(t)\) is computed for each dimension \(j\in \{1,2,\ldots ,d\}\) at time step t. further, the velocity \(v_{ij}(t)\) is updated using the following equation: $$\begin{aligned} v_{ij}(t+1) = \vargamma \cdot v_{ij}(t) + \varphi _1 \cdot r_1 \cdot (p_{ij} x_{ij}(t)) + \varphi _2 \cdot r_2 \cdot (p_{gd} x_{ij}(t)), \end{aligned}$$ (1) where \(\vargamma \) denotes the inertial weight, introduced by [23] to balance the global and local search, \(r_1\) and \(r_2\) are two independent values uniformly distributed in the range [0, 1], and \(\varphi _1\) and \(\varphi _2\) are acceleration constants. such a representation of a particle is suitable for hyperparameter fine-tuning, which considers realand integer-valued numbers. however, it is not adequate for feature selection tasks since it requires a categorical or binary representation. therefore, this work employs a variation of the method, adapted for feature selection, as discussed in sect. 4. 3.3 feature extractors haralick et al. [8] proposed a set of mathematical tools to extract statistical features from images using a gray level co-occurrence matrix. the glcm is capable of describing the frequency of occurrences in grayscale transitions for an image in a pixel-by-pixel fashion. further, the features are extracted considering the relationship of each pixel with its neighbors over four different angles, i.e., 0, 45, 90, and \(135^\circ \), where 0 and \(90^circ\) are the most employed ones. moreover, the model is capable of extracting a total of 14 statistical features, named angular second moment, contrast, correlation, variance, inverse difference moment, sum average, sum variance, sum entropy, entropy, difference entropy, information measures of correlation (1 and 2) and the maximal correlation coefficient. another well-known texture descriptor is the local binary pattern [13], which converts the image to a gray-scale level and performs a pixel-by-pixel comparison over the entire image considering a selected number of neighbors. in this comparison, the central pixel of the square formed by its neighbor assumes the value 1 if it is greater or equal to its neighbors, or 0 otherwise. such value is stored in an array, which is further employed for converting the binary intensity code to a decimal value for the pixel at hand. afterward, the same computation is performed for all pixels compounding the image. further, it is generated a histogram of the distribution of the values, which will compose the final vector describing the image. notice the method was initially proposed with the number of neighborhood fixed in \(3\times 3\). later, some changes allowed lbp to deal with a larger number of neighborhoods, resulting in a non-square structure, usually employing a circular pattern since it only requires the definition of the radius, instead of the nxn arrangement. 4 methodology this section presents the methodology concerning the material and methods employed during the experiments. it briefly describes the datasets, the process of feature extraction, the modeling of the hyperparameter fine-tuning and feature selection processes using pso, the methods used for evaluation purpose, and the baselines considered for comparison. 4.1 dataset this work employs a dataset d composed of features extracted from 374 instances of wood board images obtained in a brazilian sawmill [25]. as stated in the problem definition, each instance \(d=(\boldsymbol{x}_i, y_i)\) is composed of an m-dimensional feature vector \(\boldsymbol{x}_i \in \mathbb {r}^{m}\), whose features were extracted using both glcm and lbp, as described in sect. 4.2, and a target value \(y_i\), denoting the wood quality. each sample’s target value is established according to rules defined by the sawmill company, where “a” stands for a high-quality standard and comprises 144 instances, “b” denotes an intermediate quality and comprises 177 instances, and “c” represents lower quality, comprising 53 samples. 4.2 image texture descriptors the feature set was obtained by joining the features extracted from two texture descriptors, namely, statistical measures extracted from glcm and the lbp. while the statistical measures have the advantage of enabling the interpretation and comprehension of the image characteristics through different measures, lbp is robust in the treatment of gray-scale images, with a good performance for scale changes caused by illumination [13]. concerning the glcm, this paper employed \(0^o\) and \(90^o\) to extract six measures: angular second momentum, energy, contrast, correlation, dissimilarity, and homogeneity, resulting in 12 characteristics for each image. besides, lpb uses 24 neighbors as well as a radius of size 3, resulting in 26 characteristics for each image. thus, joining glcm and lbp features resulted in 38 predictive attributes. 4.3 mlp hyperparameter tuning and feature selection using pso this work employed a fully connected network composed of a single hidden layer for the task of classification. further, it also employed the pso algorithm to fine-tuning the three principal hyperparameters of the model, namely the number of neurons in the hidden layer \(\gamma \), the learning rate \(\eta \), and the momentum term \(\mu \). as mentioned previously, the pso algorithm was employed to perform a combined task of hyperparameter tuning and feature selection, hereafter referred to as hp-fs-pso. considering the former task, pso decision variables are modeled admitting one integer values to represent the number of units in the hidden layer, as well as two real numbers to represent the learning rate and momentum. further, since the task of feature selection does not assume continuous representation, pso requires some modifications to work properly in this context. the main change is made on the position representation, which must be treated as a result of probability analysis from particles’ velocity to decide what features are relevant to the context [9]. therefore, considering the task of feature selection, each particle’s decision variable is represented by a binary value, where 1 means to consider a feature whereas 0 means to discard a feature. since the particle’s decision variables employed for feature selection assume binary values, it is necessary to binarize each position, \(\psi _{ij}\) such that \(\psi _{ij}=1\) if \(s(v_{ij}) > r_{3}\) and 0 otherwise. notice \(r_{3}\) is the threshold, a real number generated randomly in the range [0, 1], and \(s(\cdot )\) is the logistic function. concerning the mlp hyperparameters’ search configuration, the hidden layer size is optimized in the range [2, 60], the learning rate and momentum assume values in the range [0, 1], and the feature subset selection is defined by a binary variable, i.e., assuming either 0 or 1. additionally, it also presents the mlp hyperparameter default values used by weka [7]. the number of neurons in the hidden layer is defined as \(\gamma =\frac{(\text {na} + \text {nc})}{2}\), where na is the number of attributes and nc is the number of classes. thus, the default value for our problem, which has 38 attributes and 3 classes, is 20 neurons. further, the pso algorithm also has its own hyperparameters, defined as follows: number of particles \(n=30\), acceleration constant 1 \(\varphi _{1}=1.494\), acceleration constant 2 \(\varphi _{2}=1.494\), inertia weight \(\omega =0.729\), and maximum velocity \(\upsilon \). notice tuning such hyperparameters would lead to a “never-ending” problem. therefore, these values were empirically selected based on similar works [9, 21]. besides, the maximum velocity \(\upsilon \) varies according to the upper limit of its respective hyperparameter. the optimization process is performed until meeting the stop criterion, i.e., the maximum number of iterations, which was set in 300. 4.4 evaluation metaheuristic algorithms guide their search for the best solutions according to the outcome of a fitness function. in this work, this measure is obtained from the predictive performance of the anns over a validation subset. moreover, since the dataset used in this work is composed of an imbalanced class distribution, i.e., there is a considerably smaller number of examples labeled as “c” class. therefore, measures that acknowledge this imbalance are more suitable for the task. thus, in this study, we considered the balanced accuracy (bac) [3]. the ann training and evaluation were performed using a nested stratified k-fold cross-validation (cv) re-sampling method. such an approach splits the dataset into \(k=10\) partitions, where one of them is used to test, and the remaining folds are employed for training purposes. in this context, the training folds are used to train the model during the optimization process, i.e., finding the best mlp hyperparameters and the best subset of features. therefore, pso assesses the average bac considering the fitness value over the validation set. further, the best set of hyperparameters and features found in this process is them applied to train the model and induct the prediction of the testing set samples’ labels. such a process guarantees that the data used to evaluate the model is never used in the model training steps and, consequently, in the mlp hyperparameter tuning and feature selection processes. finally, due to the stochastic process of pso, the optimization process was repeated during 10 runs, aiming to perform a statistical analysis through the wilcoxon signed-rank test [26] with \(5\%\) of significance. 4.5 baselines in order to evaluate and compare the results obtained by pso, six baselines methods were compared in the context of the combined mlp hyperparameter tuning and feature selection: method 1 (m1) : default hyperparameter values defined by weka and the whole set of features; method 2 (m2) : mlp hyperparameter tuned using pso and the whole set of features; method 3 (m3) : default mlp hyperparameter values defined by weka and feature subset selected by pso; method 4 (m4) : default mlp hyperparameter values defined by weka and dimensionality reduction performed by principal components analysis (pca); method 5 (m5) : mlp hyperparameter values tuned by pso and dimensionality reduction performed by pca; random search (rs) : random selection of mlp hyperparameter values and feature subset. this approach considered the same number of solutions evaluated by pso. the methodology adopted in this work aims at analyzing the pso performance from different x perspectives. first, m1 is the baseline for both tasks, i.e., the mlp hyperparameter using default parameters provided by weka considering the whole set of features. further, m2 allows analyzing pso influence for the task of mlp hyperparameter tuning, with no feature selection, while m3 investigates the opposite, i.e., pso influence to the task of feature selection with using mlp default hyperparameters. moreover, m4 employs the default hyperparameter with a dimensionally reduced feature subset performed by pca, while m5 combines mlp hyperparameters tuning using pso with pca. finally, rs represents the analysis of random combinations of mlp hyperparameter values and selected features. the experiments carried out in this work were coded using pythonfootnote 1 and r [18]. further, the feature extraction task was implemented in python using the scikit-imagefootnote 2 package, while the mlp network was developed using the rweka package in r, which is an interface to weka. finally, pso was also implemented in r. 5 experimental results this section presents the predictive performance of the anns assessed for the hp-fs-pso method, as well as the baselines techniques. notice the results are also provided with the p-values considering the wilcoxon signed-rank test compared to the hp-fs-pso method as the reference for statistical analysis purposes. further, values presented in bold stand for the most accurate result overall. 5.1 optimization evaluation regarding the optimization performance over the validation set, one can observe in table 1 that hp-fs-pso and m3 have obtained the best results, achieving a bac average of 0.850. such techniques provided an improvement of around \(10\%\) compared to m1, which represents an ann using default hp values and the whole set of features. therefore, the most important finding from these results is that the optimization of mlp hyperparameters, and even more a proper selection of the more suitable features, have a strong influence in the induction of mlp models applied to wood boards quality classification. on the other hand, results also show that performing only one task may also be enough to increase performance since m3 performed only the task of feature selection, while m2, which obtained an average bac of 0.834, performing only the task of mlp hyperparameter tuning. although by a narrow margin, in this case, the set of features was more expressive for the model predictive performance than tuning the network hyperparameters. this behavior suggests that a user should, at least, select a set of features for the hyperparameter values defined a priori. besides the bac values, table 1 also provides the p-values considering the wilcoxon signed-rank test compared to the hp-fs-pso method as the reference. these p-values support our previous observations, considering only m3 and the random search obtained a significance level higher than \(\alpha =0.05\). table 1. average bac and the standard deviation concerning the task of mlp hyperparameter tuning and feature selection, considering the validation set over 10 executions. notice the “p-values” are compared against the hp-fs-pso reference.full size table notice the positive behavior of the random search approach, which is somehow expected since the model presents itself as more sensitive to a proper selection of the features instead of the network hyperparameter tuning, which is expected to be a more straightforward task due to the binary nature of the search space. further, the data dimensionality reduction performed by pca showed to be inadequate for this problem, as denoted by methods m4, which obtained bac average results lower than using a default configuration, and m5. the main reason lies in the fact that pca may not be able to describe sufficiently well the problem due to its linear nature. for a better understanding of each method’s behavior during pso convergence, fig. 3 depicts the optimization performance (a) and the evolution of the bac values (b) during 300 iterations. figure 3 (a) considers the average values over 10 runs, where each iteration in the rs curve reflects the average evaluation among 30 executions, i.e., the same number of assessments performed by pso considering 30 particles. finally, m1 and m4 are represented by fixed lines since no optimization was performed over such approaches. figure 3(b) corroborates our claim that hp-fs-pso performed a guided search through the mlp hyperparameters and features spaces, improving its performance over the iterations. notice pso can reduce the number of iterations required for finding reasonable bac values since it obtained relatively high accuracies (around 0.840) after 80 iterations only. fig. 3. pso convergence considering the evaluation dataset (a) and evolution of the bac values of hp-fs-pso and rs. in this case, the best bac value found by rs up to a iteration is kept in the next iterations (b). full size image besides, fig. 3(b) depicts the hp-fs-pso performance compared against a random search considering the best results over each iteration, instead of an average. one can observe the random search performed slightly better during the first 30 iterations. afterward, the hp-fs-pso surpassed rs and kept this advantage until reaching the 300 iterations. as previously mentioned, it is possible to note that there is a considerable improvement of bac for both methods in the first 100 iterations, and then there is a slowdown in the bac growth. such a piece of information is of extreme relevance for industrial applications, since it may save time and effort during the task of tuning the model’s hyperparameter selecting the best subset of features. 5.2 classification this section investigates the ann generalization power by evaluating the predictive performance of the model over the testing set, considering the best set of hyperparameters and a subset of features found during the optimization process. table 2 presents the bac values obtained in this context. notice the values presented in bold stand for the most accurate approach overall. table 2. average bac and the standard deviation concerning the task of mlp wood quality classification considered the testing samples and the best set of hyperparameters and subfeatures found during the optimization process for each approach. notice the “p-values” are compared against the hp-fs-pso reference.full size table in general, the results of test data are in agreement with those of the validation set. the baseline method m3 led to the best bac value, followed closely by hp-fs-pso (differing only in the third decimal place). these performances are again superior to m1, supporting the idea of mlp hyperparameter tuning and feature selection influence. however, differently from table 1, hp-fs-pso was not statistically better than m3, as observed in the p-value \(>0.05\). therefore, the improvement obtained during the validation steps was not enough to provide a statistical difference, considering the test data. the high standard deviation over the testing set explains such behavior, which was considerably smaller regarding the optimization steps. 6 conclusion this paper analyzed the compound problem of ann hyperparameters tuning and feature selection for the quality classification of wood boards in the sawmill industry. experiments showed that a solution based on pso led to satisfactory results compared to baseline methods. according to a statistical test, results show a significant difference during the optimization task but not for the generalization phase. these experimental results suggest that mlp hyperparameter tuning and feature selection are essential to obtain models with higher predictive performance. also, one can notice that these tasks are interdependent since the hyperparameter values should be adjusted according to a subset of features and vice versa. consequently, for the problem investigated in this work, performing only one of them was enough to reach substantial gain. finally, the accuracy obtained in this study supports employing machine learning models for industrial implementation, contributing to overall cost reduction and improvement in competitiveness. regarding future works, we intend to perform a transfer learning from a cnn trained using a dataset composed of a more substantial number of wood image samples. besides, we are willing to investigate and compare different image descriptors, non-linear data reduction techniques, and deep learning models. notes 1.https://www.python.org/. 2.http://scikit-image.org/. references abdullah, a., ismail, n.k.n., kadir, t.a.a., zain, j.m., jusoh, n.a., ali, n.m.: agar wood grade determination system using image processing technique. in: proceedings of the international conference on electrical engineering and informatics institut teknologi bandung (2007) google scholar  affonso, c., rossi, a.l.d., vieira, f.h.a., carvalho, a.c.p.l.f.: deep learning for biological image classification. expert syst. appl. 85, 114–122 (2017). https://doi.org/10.1016/j.eswa.2017.05.039 brodersen, k.h., ong, c.s., stephan, k.e., buhmann, j.m.: the balanced accuracy and its posterior distribution. in: 2010 20th international conference on pattern recognition, pp. 3121–3124, august 2010. https://doi.org/10.1109/icpr.2010.764 cao, y., et al.: a new intelligence fuzzy-based hybrid metaheuristic algorithm for analyzing the application of tea waste in concrete as natural fiber. comput. electron. agric. 190, 106420 (2021) article  google scholar  de souza, l.a., et al.: fine-tuning generative adversarial networks using metaheuristics-a case study on barrett’s esophagus identification. in: bildverarbeitung für die medizin, pp. 205–210 (2021) google scholar  gu, i.y.h., andersson, h., vicen, r.: automatic classification of wood defects using support vector machines. in: bolc, l., kulikowski, j.l., wojciechowski, k. (eds.) iccvg 2008. lncs, vol. 5337, pp. 356–367. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-02345-3_35 chapter  google scholar  hall, m., frank, e., holmes, g., pfahringer, b., reutemann, p., witten, i.h.: the weka data mining software: an update. sigkdd explor. newsl. 11(1), 10–18 (2009). https://doi.org/10.1145/1656274.1656278 article  google scholar  haralick, r., shanmugam, k., distein, i.: textual features for image classification. ieee trans. syst. man cybern. smc 3(6), 610–621 (1973) google scholar  kennedy, j., eberhart, r.c.: a discrete binary version of the particle swarm algorithm. in: 1997 ieee international conference on systems, man, and cybernetics. computational cybernetics and simulation, vol. 5, pp. 4104–4108, october 1997. https://doi.org/10.1109/icsmc.1997.637339 kennedy, j., eberhart, r.: swarm intelligence. morgan kaufmann publishers (2001) google scholar  kennedy, j., eberhart, r.: particle swarm optimization. in: proceedings of the ieee international conference on neural networks, vol. 4, pp. 1942–1948. perth, australia (1995) google scholar  luo, g.: a review of automatic selection methods for machine learning algorithms and hyper-parameter values. netw. model. anal. health inform. bioinforma. 5(1), 1–16 (2016). https://doi.org/10.1007/s13721-016-0125-6 article  google scholar  ojala, t., pietikainen, m., harwood, d.: comparative study of texture measures with classification based on feature distributions. pattern recogn., 51–59 (1996) google scholar  passos, l.a., paulo papa, j.: fine-tuning infinity restricted boltzmann machines. in: 2017 30th sibgrapi conference on graphics, patterns and images (sibgrapi), pp. 63–70. ieee (2017) google scholar  pereira, c.r., passos, l.a., rodrigues, d., de souza, a.n., papa, j.p.: jade-based feature selection for non-technical losses detection. in: tavares, j.m.r.s., natal jorge, r.m. (eds.) vipimage 2019. lncvb, vol. 34, pp. 141–156. springer, cham (2019). https://doi.org/10.1007/978-3-030-32040-9_16 chapter  google scholar  pham, d.t., alcock, r.j.: automatic detection of defects on birch wood boards. proc. inst. mech. eng. part e j. process mech. eng. 210(1), 45–52 (1996). https://doi.org/10.1243/0954408991529852 article  google scholar  qi, c., fourie, a., chen, q.: neural network and particle swarm optimization for predicting the unconfined compressive strength of cemented paste backfill. constr. build. mater. 159, 473–478 (2018). https://doi.org/10.1016/j.conbuildmat.2017.11.006 article  google scholar  r core team: a language and environment for statistical computing. r foundation for statistical computing, vienna, austria (2014) google scholar  roder, m., passos, l.a., de rosa, g.h., de albuquerque, v.h.c., papa, j.p.: reinforcing learning in deep belief networks through nature-inspired optimization. appl. soft comput. 108, 107466 (2021) article  google scholar  roder, m., de rosa, g.h., passos, l.a., papa, j.p., rossi, a.l.d.: harnessing particle swarm optimization through relativistic velocity. in: 2020 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2020) google scholar  roder, m., rossi, a.l.d., de oliveira affonso, c.: boosting machine learning techniques for wood quality classification by particle swarm optimization. in: encontro nacional de inteligência artificial e computacional. sociedade brasileira de computação (2017) google scholar  rodrigues, d., de rosa, g.h., passos, l.a., papa, j.p.: adaptive improved flower pollination algorithm for global optimization. in: yang, x.-s., he, x.-s. (eds.) nature-inspired computation in data mining and machine learning. sci, vol. 855, pp. 1–21. springer, cham (2020). https://doi.org/10.1007/978-3-030-28553-1_1 chapter  google scholar  shi, y., eberhart, r.: a modified particle swarm optimizer. in: 1998 ieee international conference on evolutionary computation proceedings. ieee world congress on computational intelligence (cat. no. 98th8360), pp. 69–73, may 1998. https://doi.org/10.1109/icec.1998.699146 tiryaki, s., malkoçoğlu, a., özşahin, ş.: using artificial neural networks for modeling surface roughness of wood in machining process. constr. build. mater. 66, 329–335 (2014). https://doi.org/10.1016/j.conbuildmat.2014.05.098 vieira, f.h.a.: image processing through machine learning for wood quality classification. ph.d. thesis, faculdade de engenharia de guaratinguetá (feg), unesp (2016) google scholar  wilcoxon, f.: individual comparisons by ranking methods. biometrics bull. 1(6), 80–83 (1945) article  google scholar  download references acknowledgments the authors are grateful to fapesp grants #2016/06538-0, #2018/02822-1, #2019/07825-1, and #2023/10823-6 author information authors and affiliations department of computing, são paulo state university, av. eng. luiz edmundo carrijo coube, 14-01, bauru, 17033-360, brazil mateus roder, leandro aparecido passos & joão paulo papa department of production engineering, paulo state university, rua geraldo alckmin, 519 vila nossa sra. de fatima, itapeva, 18409-010, brazil andré luis debiaso rossi authors mateus roderview author publications search author on:pubmed google scholar leandro aparecido passosview author publications search author on:pubmed google scholar joão paulo papaview author publications search author on:pubmed google scholar andré luis debiaso rossiview author publications search author on:pubmed google scholar corresponding author correspondence to leandro aparecido passos . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper roder, m., passos, l.a., papa, j.p., rossi, a.l.d. (2023). feature selection and hyperparameter fine-tuning in artificial neural networks for wood quality classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_22 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial neural network optimization wood quality classification sawmill problem hyperparameter tuning feature selection publish with us policies and ethics profiles leandro aparecido passos view author profile joão paulo papa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ontology based classification of electronic health records to support value-based health care | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ontology based classification of electronic health records to support value-based health care conference paper first online: 28 november 2021 pp 359–371 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) ontology based classification of electronic health records to support value-based health care download book pdf download book epub avner dal bosco  orcid: orcid.org/0000-0001-7214-176710, renata vieira  orcid: orcid.org/0000-0003-2449-547711, bruna zanotto  orcid: orcid.org/0000-0003-4012-139512 & … ana paula beck da silva etges  orcid: orcid.org/0000-0002-6411-348012  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 770 accesses 3 citations abstract value-based health care management models require a precise accounting of health indexes such as risk events monitoring, clinical conditions, patient handling and outcomes. currently this accounting is performed by manually reading and searching through electronic health records for these indexes. our research proposes a way to make this an autonomous task that is performed by a computer using a portuguese free-text concept classifier model based on ontologies. to validate our model we tested it on digital clinical records from 191 patients under ischemic stroke care. we have selected 30 management indexes to be identified in these texts. our model reached, on average 56,8% of f1-score, varying from 5,83% to 94,78% f1-score across different management indexes. financially supported by the brazilian coordination of superior level staff improvement (capes), the by portuguese foundation for science and technology (fct)under the projects ceecind/01997/2017, uidb/00057/2020 and, the national council for scientific and technological development (cnpq) (project: 465518/2014-1). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others value-based care chapter © 2023 evaluation of quality and use of health management information system in primary health care units of east wollega zone, oromia regional state, ethiopia: article open access 12 june 2020 value-based healthcare chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. e-health gene ontology ontology health informatics information model knowledge based systems 1 introduction value-based healthcare systems (vbhc) allow fair rewards and proper recognition for health providers based on the quality and results of the service provided [3, 5, 11]. in this system, the responsible bodies for financing and rewarding these providers can be more confident about their investments, the main users get better services and results and, providers are encouraged to optimize their practice. the implementation of effective vbhc requires advances in computational intelligence to continually turn ehr data on information [7, 10]. through these records, services and results are evaluated. this is an exhausting manual task that has to be performed even on top of edge digital health records software; therefore it is essential to turn it into an automated task. this task is defined as the measure of service indexes such as risk events monitoring, clinical conditions, patient handling and outcomes. in order to measure these indexes it is necessary to find keywords and technical terms inside clinical records that are written as free-texts in the portuguese language. our research aims to automatize such task, and it is focused on patients under ischemic stroke care. for this context, 30 indicator indexes were chosen to measure the services grouped in: clinical features, evaluation measures and risk events, clinical handling and patient condition. this challenge has been handled previously by zanotto et al. (2021) [14] through machine learning techniques where good results were presented. however, these techniques often require vast amounts of annotated data and computational processing for model training. our approach tries to avoid these issues by proposing a knowledge based system combined with natural language processing (nlp) techniques. we make use of the nlp methods to find keywords and match terms with an ontology which then uses axioms to classify a given text from the clinical records according to the management indexes. the evaluation of our model was made on digital clinical records from 191 patients under stroke care. our model was capable of to identify and to classify 28 of those indexes varying from 5,83% f1-score results and mcc score of 8,01% to 94,78% f1-score results and mcc-score of 94,78%. considering all 30 indexes, our model reached, on average 56,8% of f1-score and a mcc-score of 57,97%. 2 related work in wang et al. (2003) [12] and zhou and el-gohary (2015) [15] we see the use of machine learning algorithms as part of the classification processes, either to learn the terms related to the domain or to make the classifications. a larger quantity of data was required to achieve good results. the domain application of wang et al. (2003) were papers from the medline database and on zhou and el-gohary (2015) were construction regulation documents. other works follow a knowledge representation approach. the work of allahyari et al. (2014) [1] uses an ontology for the classifications of english text from news on the web, they used graph projection for this purpose. in chi et al. (2014) [4] an ontology was created in a semi-automated way, but due to data scarcity, quality and range of terms were not optimal. they worked with job hazards reports. in schwertner et al. (2019) [9] the authors built an ontology based on domain knowledge from specialists, defining relations between concepts and sentences, they used the ontology as a classification tool. they also worked with clinical data information in the english language. in gayathri and kannan (2020) [6] the authors face a similar challenge that we had, their goal was to detect and identify health-related information on english text documents. on yehia et al. (2019) [13] domain ontologies are used to classify sentences using rules, based on the relations between concepts. they worked with clinical data documents. in de araujo et al. (2017) [2] an ontology that uses inference processes, based on linguistic rules defined by specialists, to classify texts is presented. in this paper the authors worked with documents and texts in portuguese about judicial events. these related works show that ontologies can be considered an alternative for text classification. different approaches are applied to different domains and results were in general positive. we also see few movements towards techniques focused on the portuguese language. given the results of these related works, we considered that a model based on ontologies should be tested for our text classification task, in portuguese language, to compare with previous work on the same problem that used machine learning in a study made by zanotto et al. (2021) [14]. they present an evaluation of machine learning approaches for the same database used in the present work, the classification considers almost the same set of classes, they worked on 24 or the indexes. a comparison with this work is presented in the results section. 3 available data and challenges the main goal of this research was to verify the applicability and the performance of a computational model, based on ontologies, in the task of automatically detect and classify portuguese free-texts from electronic health records. to accomplish our goal we focused on classifying clinical records from patients under ischemic stroke care. this decision is based on our proximity with a team of specialists of this domain. we also wanted to verify how this model compares against machine learning approaches on this same task and dataset [14]. the data available for this research contains portuguese free texts clinical records of 191 patients that were treated for ischemic stroke incidents from 01/01/2019 up to 07/23/2019. our challenge is to find terms and keywords in a given text from these records and given the detected words, classify the text in one the following 30 quality indexes that are shown in table 1. this study was approved by the hospital ethics committee (caae: 29694720000005330). table 1. management indexes and its sub-groupsfull size table we aim to develop a computational model in a way that it permeates current electronic health records software, allowing it to operate as first designed as it only would have to provide the data that is stored. with this approach, healthcare practitioners may keep using the same kind of software that they are used to, with no need to input any new data. as the model outputs only the indexes in which the texts were classified, practitioners would also be able keep the privacy and the details of their records and practices. with the help of a team of domain specialists, we identified technical terms and keywords for the indexes that are to be found within the clinical records. we developed an ontology that plays three important roles in our model. the first one is to provide a list of terms (keywords) that are to be found in the texts. the second role is the definition of the relation of the indexes with the terms. moreover, the last role is the classifier itself that reasons about the relation between texts and terms and classifies them into the appropriate index. to work along with our ontology, a term detecting algorithm was also developed. 3.1 data preprocessing and annotation all available clinical records were first anonymized. all records were split into sentences. after this step, 46.547 sentences were generated, in which the indexes would have to be detected. the sentence order was randomized to prevent annotation based on previous context, the idea was to analyse each sentence independently. two annotators, domain specialists, read the sentences and informed all the indexes that could be identified in each sentence. the results obtained from both annotators had the percent agreement between them measured by kappa, which was higher than 0.61. in the cases in which there were conflicts, both annotators would come to together to discuss and solve the conflict. no conflict was left unsolved. at the conclusion of these step it was identified that only 17.471 out of the 46.547 sentences were related to one or more of the indicators. the sentences occurrences of each indicator is detailed in table 8. 4 methods 4.1 ontology based classification algorithm note that our ontology is a task ontology, developed for classifying sentences containing terms into 30 different indexes. thus our ontology has three main concepts: the ‘terms’ concept, in which all the sub-concepts are keywords that are to be found in the texts; the ‘sentences’ concept that contains all the sentences processed by the text-detection algorithm; and the ‘index’ concept that contains the description of all the indexes in which the sentences should be classified into. an object property relation ‘contain’, expresses the presence of a given term in a sentence. to the ‘terms’ concept we added a few subsets of concepts: the ‘values’ sub-concept aims to specify the terms that require a numeric value in order to compose a classification; the ‘negations’ sub-set contains expressions that would indicate the negation of the occurrence of an index; the ‘pasttense’ sub-set contains terms that would signal the occurrence of an index in some point of the past, these terms often do so by appearing at some point after the mention of an index, and the ‘pasttenseretroactive’ is also for identification of the past, but this specifies the terms that appear before the mention of the index in the sentence. whenever the terms from these subsets were found, they create the appropriate relation, for instance, ‘sentence negates index’, instead of ‘sentence contains index’. our ontology plays the following roles: it serves as knowledge model of the terminology of this domain and it plays as a text classifier trough its inference capabilities. the inference process is based in assertions in a logical form that together comprise the overall theory that the ontology describes in this domain, this assertions are calles axioms and in this ontology it refers to which terms a sentence must contain in order to be classified as an element of a given index. table 2 shows us a few examples of these axioms. this ontology was built in owl language using the protégé tool. table 2. axioms examplesfull size table the term detection algorithm receives the instances of the class terms. the algorithm then runs through each sentence and tries to match the given terms to the words in the sentences. whenever a match is found, the algorithm registers it in the ontology composing the triple sentence, relation, term. our approach also uses ‘word embedding’ models. the ‘word embedding’ used was developed on the basis of electronic health records of a brazilian hospitalfootnote 1[8], which was trained using 21 million sentences culminating in a model with 63 thousand words. we chose this language model, since this is based on portuguese ehr. the application of this models has two main goals: to circumvent grammar errors that often occur in these clinical records and; to expand the vocabulary list of terms that were defined initially in the ontology, which means that it brings new rellated words, for instance the medical term ‘coronarina’ is not defined in the ontology, but this model relates it to term ‘coronaria’ covering this terminology gap. hence our algorithm uses these models to search for similar words. using the cosine similarity, the top 10 words are retrieved to expand the list of terms given by the ontology. to optimize running time two parallel lists are kept by the algorithm: the first one is in charge of storing all words from the texts that do not have a match in the ontology; the second list keeps track of all the words from the sentence that do not match with the terms in the ontology, but some of its word embedding similar do. after all the sentences have been processed and put under the ‘sentences’ set in the ontology, the reasoning process starts and the classification is made. the results are then evaluated by comparing them with the annotated data. 4.2 example to better understand how our model works let us take the set of sentences shown in table 3. our model receives these sentences as input and classifies them. consider sentence number 6, the first step is to tokenize it. table 4 shows the result of this step. next, every word is treated to remove characters that are not either alphabets or numbers, and changed to lowercase, as seen in table 5. the first word from our sample is ‘após’, our model uses the word embedding model to expand this word and get new similar words. table 6 shows us all the similar words found for this term. after that they are compared with our defined terms. in this example, the term ‘após’ and all the similar terms are searched in the list of defined terms in the ontology. for this instance the term ‘após’ is not defined and hence no relation between ‘sentence #6’ and this term is created. our model stores this term in a list of unmatched terms, for the next time it appears it will not be evaluated again. next, we have the term ‘trombolise’. this one is also expanded with our word-embedding model and the similar terms are shown in table 7. again, the term and all the similar words are matched against the ontology. table 3. randomized set of sentencesfull size table table 4. set of words from sentence #6full size table table 5. set of treated words from sentence #6full size table table 6. set of similar words given by the word-embedding model for the term ‘após’full size table table 7. set of similar words given by the word-embedding model for the term ‘trombolise’full size table in this scenario we notice that the term given by the sentence is ‘trombolise’, whereas the definition contains ‘trombólise’. so as expected the term itself is not found in the ontology list as it is defined with proper spelling. however the word embedding model captures this misspelling. as one of the similar words is ‘trombólise’ then our model correctly creates the relation between ‘sentence #6’ and the term ‘trombólise’. our algorithm then writes in the ontology the relation ‘sentence#6 contain trombólise’. as this relation is specified in the axiom ‘(contain some alteplase) or (contain some trombolisada) or (contain some trombolítico) or (contain some trombólise) subclassof thrombolysis’ once the reasoning process is complete this sentence would then be classified as an elements of the set ‘thrombolysis’ which is the set of all sentences that tell us that the index thrombolysis is present. these steps are then taken to every word in the sentence, and at the end, the relations found are stored in the ontology that will next reason about them. 5 results for evaluation of the proposed model the 46.547 sentences were processed and the output was compared to the manual annotation to measure ‘precision’, ‘recall’, ‘mcc-score’ and ‘f1-score’. the running time for the whole task was also measured. table 8 shows the total occurrences of sentences annotated for each index and the results obtained. table 8. ontology based model resultsfull size table the model took 532,43 s to process all the 46.547 sentences achieving, in average, ‘f1-score’ of 56,8%, ‘mcc-score’ of 57,97%, ‘precision’ of 64,89% and ‘recall’ of 57,97%. some indexes, such as ‘thrombolysis’ and ‘atrial fibrillation’, achieved results over 80%; however others, such as ‘pain’ and ‘mobility’, did not reach over 20%. table 9. ‘f1-score’ results comparison between the ontology based approach and machine learning for 24 indexes. when the difference is 5 points or more, the higher score is in bold.full size table table 10. most common errosfull size table these results were compared to the ones obtained by zanotto et al. (2021) [8] in which a machine learning approach was used for the same challenge. several supervised computational machine learning (ml) methods, including recent neural and non-neural methods were evaluated on the basis of a 5-fold crossvalidation procedure. the best results were achieved by the w+c+svm model, which is based on word-tfidf and character-tfidf for input representation and svm for the classification. table 9 presents the f1-score results of our ontology based approach and the machine learning approach. recall that we have machine learning evaluation results for only some of the indexes (24) and can only compare those. our approach performs well on indexes that the machine learning model doesn’t and vice-versa. this signals that, by delegating a given a indicator to the most adequate classifier, a combined model could perform well on a larger range of indexes. for instance classifications made by the ontology would benefit the analysis when it comes to the indicator ‘dyslipidemia’ whereas the indicator ‘infection’ would benefit from the machine learn approach. this decision could also be modeled in the ontology and reasoned by it. further efforts should be put into this matter as a collaborative future work. 5.1 errors analysis in table 10 we present the most common cases of ‘false negatives’ and ‘false positives’. in general the performance of the model is related to the coverage of terms in the ontology. misspellings and technical slang played a big part on classification errors, the word-embedding model does not cover all the possible variations of terms. cases 3, 12 and 13 are related to differences in spelling which were not captured in the most similar words according to the we model. examples 1, 2 and 9 are cases in which the index is present in the sentences, but the specific terms in these sentences were not defined in our ontology. past tense and negation are also common source of errors, as expected, since these elements in language require specific sophisticated solutions on their own. in examples 4, 7, 8 the past is indicated through the date of the event, neither our ontology nor our algorithm was prepared for this. errors in 6, 10 and 11 show us cases in which one ‘negation’ term is negating more than one index. our model expects one negation term per event. 6 conclusion in this paper, we proposed a portuguese text classifier based on ontologies. our results show that this approach achieved good results for at least 18 out of the 30 indexes. we believe that this research demonstrates how ontologies is a good alternative for portuguese medical texts classifications and, because of that, can contribute to the implementation of vbhc programs, contributing to the transformation of health care systems. an advantage of this approach is direct explainability. for future work we plan to evaluate the model on ehr from different institutions to validate the generality of the results shown here. we also plan to expand the coverage of terms and for that end, we plan to create another word embedding model, more tailored to the stroke patients context. as a continuation of the project we plan to align our task ontology with stroke domain ontologies. notes 1.https://www.inf.pucrs.br/linatural/wordpress/recursos-e-ferramentas/word-embeddings-para-saude/. references allahyari, m., kochut, k.j., janik, m.: ontology-based text classification into dynamically defined topics. in: proceedings of the 8th ieee international conference on semantic computing, newport beach, estados unidos da america, pp. 273–278. ieee (2014) google scholar  de araujo, d.a., rigo, s.j., barbosa, j.l.v.: ontology-based information extraction for juridical events with case studies in brazilian legal realm. artif. intell. law 25, 379–396 (2017) google scholar  bessa, r.d.o.: análise dos modelos de remuneração médica no setor de saúde suplementar brasileiro. ph.d. thesis, fgv (2011) google scholar  chi, n.w., lin, k.y., hsieh, s.h.: using ontology-based text classification to assist job hazard analysis. adv. eng. inf. 28, 381–394 (2014) google scholar  engel, g.l.: the clinical application of the biopsychosocial model. j. med. philos. forum bioeth. philos. med. 6(2), 101–124 (1981). https://doi.org/10.1093/jmp/6.2.101 gayathri, m., kannan, r.: ontology based concept extraction and classification of ayurvedic documents. procedia comput. sci. 172, 511–516 (2020) google scholar  gonçalves, f.n.r.: optimizing patients’ pathways in international cooperation, by doing value based healthcare (vbhc). acta medica portuguesa 32(2), 167–168 (2019) article  google scholar  dias pereira dos santos, h., d. p. s. ulbrich, a.h., woloszyn, v., vieira, r.: an initial investigation of the charlson comorbidity index regression based on clinical notes. in: 2018 ieee 31st international symposium on computer-based medical systems (cbms), pp. 6–11 (2018). https://doi.org/10.1109/cbms.2018.00009 schwertner, m.a., rigo, s.j., araújo, d.a., silva, a.b., eskofier, b.: fostering natural language question answering over knowledge bases in oncology ehr. in: proceedings of the 32nd international symposium on computer-based medical systems, cordoba, espanha, pp. 501–506. ieee (2019) google scholar  da silva etges, a.p.b., ruschel, k.b., polanczyk, c.a., urman, r.d.: advances in value-based healthcare by the application of time-driven activity-based costing for inpatient management: a systematic review. value health 23(6), 812–823 (2020). https://doi.org/10.1016/j.jval.2020.02.004. https://www.sciencedirect.com/science/article/pii/s1098301520301303 uzuelli, f.h.d.p., costa, a.c.d.d., guedes, b., sabiá, c.f., batista, s.r.r.: reforma da atenção hospitalar para modelo de saúde baseada em valor e especialidades multifocais. ciência saúde coletiva 24, 2147–2154 (2019) google scholar  wang, b.b., mckay, r.i.b., abbass, h.a., barlow, m.: a comparative study for domain ontology guided feature extraction. in: proceedings of the 26th australasian computer science conference, pp. 69–78. australian computer society inc., adelaide, austrália (2003) google scholar  yehia, e., boshnak, h., abdelgaber, s., abdo, a., elzanfaly, d.: ontology-based clinical information extraction from physician’s free-text notes. j. biomed. inform. 98, 103–117 (2019) google scholar  zanotto, b., et al.: automatic classification of electronic health records for a value-based program through machine learning. value health (2021, to appear). accepted for publication in virtual ispor 2021 google scholar  zhou, p., el-gohary, n.: ontology-based multilabel text classification of construction regulatory documents. j. comput. civ. eng. 30, 40–54 (2015) google scholar  download references author information authors and affiliations school of technology, pontifical catholic university of rio grande do sul, pucrs, porto alegre, brazil avner dal bosco cidehus, university of évora, évora, portugal renata vieira federal university of rio grande do sul, porto alegre, brazil bruna zanotto & ana paula beck da silva etges authors avner dal boscoview author publications search author on:pubmed google scholar renata vieiraview author publications search author on:pubmed google scholar bruna zanottoview author publications search author on:pubmed google scholar ana paula beck da silva etgesview author publications search author on:pubmed google scholar corresponding author correspondence to avner dal bosco . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper bosco, a.d., vieira, r., zanotto, b., da silva etges, a.p.b. (2021). ontology based classification of electronic health records to support value-based health care. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_24 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords ontologies electronic health records stroke health management publish with us policies and ethics profiles renata vieira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature semi-periodic activation for time series classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper semi-periodic activation for time series classification conference paper first online: 31 january 2025 pp 76–90 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) semi-periodic activation for time series classification download book pdf download book epub josé gilberto barbosa de medeiros júnior9, andré guarnier de mitri9 & diego furtado silva9  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 242 accesses 1 citation abstract this paper investigates the lack of research on activation functions for neural network models in time series tasks. it highlights the need to identify essential properties of these activations to improve their effectiveness in specific domains. to this end, the study comprehensively analyzes properties, such as bounded, monotonic, nonlinearity, and periodicity, for activation in time series neural networks. we propose a new activation that maximizes the coverage of these properties, called leakysinelu. we empirically evaluate the leakysinelu against commonly used activations in the literature using 112 benchmark datasets for time series classification, obtaining the best average ranking in all comparative scenarios. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a non-monotonic activation function for neural networks validated on benchmark tasks chapter © 2021 dissipativity analysis of neutral-type memristive neural network with two additive time-varying and leakage delays article open access 08 january 2019 extended dissipative criteria for delayed semi-discretized competitive neural networks article open access 25 march 2024 1 introduction deep learning models have achieved significant performance improvements across various domains. consequently, their application in time series tasks has been receiving increasing attention. however, numerous features and techniques remain to be thoroughly explored in this area, leading to the proposal of several new approaches [7, 20, 25, 26]. most recent research on deep learning for time series focuses on proposing and evaluating architectures, often adapting those designed for computer vision, such as resnet [24] and inceptiontime [6]. other variations aim to extend these architectures beyond classification tasks [27, 28]. while these architectures have achieved satisfactory results in their respective domains, they still have gaps to explore. deep neural networks can learn effective representations of high-dimensional data, including time series. however, they can suffer from overfitting and other issues that lead to inaccurate results, especially when dealing with periodic data [31]. to effectively model time series data, it is crucial to capture inherent patterns such as trends and seasonality to generate suitable representations for the desired tasks. a notable gap in the literature on deep learning architectures for time series is the development of activation functions capable of adequately transforming the data to representation for a specific task. some methods propose the use of periodic activation functions, such as sin(x), cos(x), linear combinations of these functions, or more recently, \(sin^2(x) + x\), to map these features [16, 21, 30, 31]. however, even the most recent functions can get stuck in local minima, face issues such as exploding and vanishing gradients, and lack sufficient non-linearity. figure 1 shows the results of activations used in neural networks for time series. relu (a) is the most used choice as an activation function in these networks. in the example observed in fig. 1, it can not handle features on the input negative values, forcing the network to make a translation operation and put important features on positive values, resulting in a linear activation due to its formula. prelu (b) comes to solve this problem, as known as dying of relu [8], allowing the network to learn a negative slope for the activation by adding a learnable parameter for each neuron. the sine function (c) has been applied on time series problems trying to capture periodic patterns [22] once the derivative is also periodic and non-monotonic (the derivative has both positive and negative values), the network can be stuck in local-minimum. fig. 1. comparison between different activations. blue lines represent an input time series, red dashed lines represent the values obtained by applying the activations. (color figure online) full size image in this work, we propose the leakysinelu (fig. 1d), a semi-periodic function as an activation that maintains main properties such as monotonic derivatives, non-linearity, and absence of boundaries. the main contributions of this work include (1) a study of functions and their basic properties related to time series data; (2) demonstrating that current activation functions can lead to problems when used in time series models, particularly in small architectures; (3) providing a comprehensive evaluation showing that the proposed activation surpasses the use of other functions in two simple and well-known architectures for time series classification. 2 definitions and notations this section introduces the necessary basic definitions and notations to understand this work. definition 1 a time series \({\textbf {x}}\) is a set of s ordered values \({\textbf {x}} = (x_1, x_2, \dots , x_s)\) and \(x_i \in \mathbb {r}^d\), such that d denotes the number of dimensions of that time series, with \(d \in \mathbb {z}\). each value \(x_i\) is referred here as an observation. definition 2 a deep neural network, \(\mathcal {dnn}\), can be defined as a sequence of functions composed of multiple layers of interconnected neurons. let l denote the total number of layers in the network, with \(l_1, l_2, \dots , l_{{l}}\) representing each layer. a layer \(l_i\) consists of \(n_i\) neurons, where \(i \in 1, \dots , {l}\). weights and biases associated with that layer \(l_i\) are denoted by \({\textbf {w}}_i\) and \(b_i\), respectively. given an input \({\textbf {x}}\), the output of a \(\mathcal {dnn}\) can be expressed as a sequence of layers applied to that input: $$\begin{aligned} f({\textbf {x}}) = f_\mathcal {l}(f_{\mathcal {l} 1}(\cdots (f_1({\textbf {x}})))), \end{aligned}$$ (1) where \(f_i : \mathbb {r}^{n_{i-1}} \rightarrow \mathbb {r}^{n-i}\) represents the function for layer \(l_i\). this function can be expressed as: $$\begin{aligned} f_i(x) = \sigma _i (\textbf{w}_i x + b_i), \end{aligned}$$ (2) where \(\sigma _i : \mathbb {r}^{n_i} \rightarrow \mathbb {r}^{n_i}\) denotes the activation function for the layer i, and \(x \in \mathbb {r}^{n_{i 1}}\) represents the input to that layer. definition 3 a semi-periodic activation function is a function where the periodicity is present in its derivative, rather than in the function itself. this means that while the activation function may not be periodically repetitive, its derivative exhibits periodic behavior. formally, let \(\sigma : \mathbb {r} \rightarrow \mathbb {r}\) be an activation function. \(\sigma \) is considered semi-periodic if its derivative \(\sigma '\) satisfies \(\sigma '(x + t) = \sigma '(x)\) for some period \(t > 0\) and for all \(x \in \mathbb {r}\). definition 4 a sub-derivative is a generalization of the concept of a derivative for functions that may not be differentiable in the traditional sense. it extends the idea of a tangent line to non-smooth functions, often used in the context of convex functions and optimization. formally, let \(f: \mathbb {r} \rightarrow \mathbb {r}\) be a convex function. a vector \(g \in \mathbb {r}\) is called a sub-derivative of f at a point \(x_0 \in \mathbb {r}\) if it satisfies the following inequality for all \(x \in \mathbb {r}\): $$\begin{aligned} f(x) \ge f(x_0) + g(x x_0), \end{aligned}$$ (3) the set of all sub-derivatives of f at \(x_0\) is called the sub-differential and is denoted by \(\partial f(x_0)\). thus, the sub-differential is given by: $$\begin{aligned} \partial f(x_0) = \{ g \in \mathbb {r} \mid f(x) \ge f(x_0) + g(x x_0) \text { for all } x \in \mathbb {r} \}, \end{aligned}$$ (4) as an example of a sub-derivative we can include the relu function \(f(x) = \max (0, x)\) the sub-derivative at \(x_0 = 0\) is any \(g \in [0, 1]\). 3 activation functions to easily compare the leakysinelu and the related activation, this section offers a mathematical and theoretical analysis of each activation and their application on time series tasks. in the following subsections, we present the related activations and define the proposed leakysinelu function, each property, and the comparison between the main activations. 3.1 related activations to compare the proposed activation with various approaches, activation functions encompassing one or more specific properties were selected, and in some cases, functions that do not possess certain properties were also included. table 1 presents all related activations compared in this work, their formula and derivatives. table 1. activation functions compared in this work, their base and derived formulas.full size table this selection includes functions with upper and lower bounds, such as the sigmoid and hyperbolic tangent. to represent the family of periodic functions, the sine function was chosen. to cover functions derived from the relu family, the following were selected: relu, elu as an exponential linear unit (prelu) for its learnable parameters, gelu for its continuity over the entire domain, and silu for combining the sigmoid strategy with a linear unit. at last, snake was chosen due to its approach being the most similar to our proposal. 3.2 leakysinelu in this work, we propose the leakysinelu, defined by the following equation: $$\begin{aligned} \sigma (x) = {\left\{ \begin{array}{ll} \sin ^{2}(x) + x & \text {if} ~ x > 0, \\ \frac{\sin ^{2}(x) + x}{2} & \text {otherwise}, \end{array}\right. } \end{aligned}$$ (5) the derivative of this equation can be defined as: $$\begin{aligned} \sigma '(x) = {\left\{ \begin{array}{ll} \sin (2x) + 1 & \text {if} ~ x > 0, \\ \frac{\sin (2x) + 1}{2} & \text {otherwise}. \end{array}\right. } \end{aligned}$$ (6) figure 2 presents the image domain and the derivative of leakysinelu. it’s possible to note that the derivate is not continuous when \(x = 0\). as in relu and prelu activations, this problem is solved using sub-derivatives and sub-gradients (c.f. sect. 3.5). however, we maintain a periodic behavior on the derivative in both domains, negative and positive, and an increased non-linearity by the difference on the scale for these domains. fig. 2. leakysinelu defined in eq. 5 represented in blue, and its derivative defined in eq. 6 in red. (color figure online) full size image 3.3 boundness over the years, many studies have shown that bounded activation functions, such as hyperbolic tangent or sigmoid, could reach excellent results in many different tasks. although these results are specifically reported when using shallow network architectures [12, 17]. however, training neural networks that use this type of function with time series data may cause a loss of information, which occurs when the series contains values outside the boundaries, causing the problem of vanishing gradients [1]. neural networks that use unbounded activation functions, with regards to a non-polynomial activation, are universal approximators and seem to attenuate the vanishing gradient problem [15, 18, 22]. however, activations that are upperor lower-bounded, such as the lower-bounded relu, can cause a loss of information on negative observations of the time series, which results in a problem known as “dying relu” [8]. other activations such as leakyrelu, elu [2], silu [4], prelu, and gelu [9] have been proposed to address this issue, either using a negative slope different from 0 or specific weight initializations. but except for leakyrelu and prelu, all others have lower bounds, even if different from 0. figure 3 presents a comparison between relu and leakysinelu functions applied to a time series. note that in this case, for negative values of the time series, we may lose some information, which means that \(f(x) = 0\). the region of the time series that this occurs is represented by a red highlighted background on that series. this can result in the previously cited dying of the relu problem in those neurons. leakysinelu maintains a negative slope while preserving the negative-values of the original series. an argument in favor of relu or its derivations is that we can maintain control of the value range for that series by applying some normalization method [13], but with this, we put all values on the positive image domain, which means that we have a linear activation and with that comes the lack of non-linearity that will be explained in sect. 3.4. fig. 3. effects of boundaries on time series feature maps. red region presents the information lost, which means that \(f(x) = 0\) for that values. (color figure online) full size image to analyze the boundness of leakysinelu activation function we rely on the limits of \(f(x) \rightarrow +\infty \) and \(f(x) \rightarrow -\infty \). if one of these limits is equals to a constant value, then the function is upper or lower bounded. $$\begin{aligned} \lim _{x \rightarrow +\infty } \sigma (x) = \lim _{x \rightarrow +\infty } \sin ^{2}(x) + x = \sin ^{2}(+\infty ) + \infty = +\infty , \end{aligned}$$ (7) $$\begin{aligned} \lim _{x \rightarrow -\infty } \sigma (x) = \lim _{x \rightarrow -\infty } \frac{\sin ^{2}(x) + x}{2} = \frac{\sin ^{2}(\infty ) \infty }{2} = -\infty , \end{aligned}$$ (8) as we can see, both of the limits are going to \(+ \infty \) and \(\infty \), with this we can conclude that the leakysinelu have no boundaries. table 2 present a comparison between each related activation functions and their boundaries. table 2. negative and positive limits as x tends to \(+\infty \) or \(-\infty \) for each of the activations.full size table 3.4 non-linearity non-linearity is a critical point for activation functions [1]. all activations want to increase the non-linearity present in neural networks. activations such as relu, gelu, elu, prelu, silu present this property. nevertheless, when we isolate the positive side of this activations, its behavior in the domain is linear, which can hinder learning for some tasks, such as when we normalize the entire series to only have positive values. consider a basic neuron defined before: $$\begin{aligned} f_i(x) = \sigma _i (\textbf{w}_i x + b_i), \end{aligned}$$ where \(\textbf{w}_{i}\) and \(b_i\) are the weight matrix and bias vector of the \(i^{th}\) layer, and \(f_i(x)\) is the output of the previous layer. we can represent the output of the neural network as: $$\begin{aligned} f(\textbf{x}) = \textbf{w}_{l} \sigma (\textbf{w}_{l-1} \sigma (\cdots \sigma ( \textbf{w}_{2}\sigma (\textbf{w}_{1} {\textbf{x}} + {b}_{1}) + {b}_{2}) + {b}_{l-1} ) + {b}_{l}, \end{aligned}$$ (9) where \(\textbf{x}\) is the input time series to the neural network. if the neural network uses only linear activations, such as only the positive side of relu, the \(\sigma \) can be ignored. $$\begin{aligned} f(\textbf{x}) = \textbf{w}_{l} (\textbf{w}_{l-1} (\cdots ( \textbf{w}_{2} (\textbf{w}_{1} {\textbf{x}} + {b}_{1}) + {b}_{2}) + {b}_{l-1} ) + {b}_{l}, \end{aligned}$$ (10) we can now denote a total weight and bias by respectively: $$\begin{aligned} w_{\mathcal {dnn}} = \textbf{w}_l \textbf{w}_{l-1} \cdots \textbf{w}_1, \end{aligned}$$ (11) $$\begin{aligned} b_{\mathcal {dnn}} = \textbf{w}_l \textbf{w}_{l-1} \cdots \textbf{w}_2 b_1 + \textbf{w}_l \textbf{w}_{l-1} \cdots \textbf{w}_3 b_2 + \cdots + \textbf{w}_l b_{l-1} + b_l, \end{aligned}$$ (12) with this two definitions we can simplify the equation of a deep neural network that uses a linear activation with the following equation: $$\begin{aligned} f(\textbf{x}) = \textbf{w}_{\mathcal {dnn}} \textbf{x} + b_{\mathcal {dnn}}, \end{aligned}$$ (13) note that this equation have a form of a linear model. 3.5 differentiability the differentiability of activation functions is critically important in neural networks, as it facilitates the application of gradient-based optimization methods, such as the backpropagation algorithm. these methods require the computation of derivatives (or gradients) of the activation functions concerning their parameters to adjust the network’s weights efficiently. differentiable activation functions ensure the smooth propagation of gradients back through the layers of the neural network, which is essential for effective minimization of the loss function [8, 19]. as shown in eq. 6, the derivative for leakysinelu can be calculated in two parts. figure 2 presents the proposed activation and its derivative. note that there is a discontinuity when \(x = 0\). by using the definition of sub-derivatives we can define that the derivatives at \(x = 0\) are given by: $$\begin{aligned} \partial f(x_0) = \{ \sin (x_0) + 1, (\sin (x_0) + 1)/2 \} = \{ 1, 0.5 \}, \end{aligned}$$ that way we can use gradient-based optimization methods and the leakysinelu function in neural networks. this is the same strategy already used for functions such as relu and prelu. 3.6 periodicity the motivation behind the periodicity is based on the universal extrapolation theorem [31] and the generalized fourier series. consider the definition of this general form of fourier series: $$\begin{aligned} x(t) = \frac{a_0}{2} + \sum ^\infty _{n=1} a_n \cos (2\pi \frac{nt}{t}) + b_n \sin (2\pi \frac{nt}{t}), \end{aligned}$$ (14) where \(a_0\), \(a_n\), \(b_n\) are constants and t is a elemental period. let \(a_0\) be an array in a \(\mathbb {r}^d\) space, the result of a scalar division \(a_0/2\) is a operation that maps \(f: \mathbb {r}^d \rightarrow \mathbb {r}^d\). we can rename the result of \(a_0/2\) to a bias vector and the constants \(a_n\) and \(b_n\) by weights \(w_{0n}\) and \(w_{1n}\), that way the formula will be: $$\begin{aligned} x(t) = b + \sum ^\infty _{n=1} w_{0n} \cos (2\pi \frac{nt}{t}) + w_{1n} \sin (2\pi \frac{nt}{t}), \end{aligned}$$ (15) now considering the mathematical equivalence of \(\sin \) and \(\cos \), \(\sin (x+3\pi /2) = \cos (x)\), and considering each input for \(\sin \) and \(\cos \) as \(x_0\) and \(x_1\) we can rewrite the formula as: $$\begin{aligned} x(t) = b + \sum ^\infty _{n=1} w_{0n} \sin (x_0) + w_{1n} \sin (x_1) \end{aligned}$$ (16) that way we can group \(w_{0n}\) and \(w_{1n}\) is a weight matrix \(\textbf{w}\), and \(x_0\) and \(x_1\) also in a input matrix \(\textbf{x}\). the \(\sin \) will operate over a matrix and return an output in the same dimension, by the property of matrix multiplication, we can arrange the columns and rows to perform the multiplication over the correct values. resulting in a final formula similar to a neural network layer with \(\sin \) as activation: $$\begin{aligned} x(t) = b_{ias} + \sum ^\infty _{n=1} w_n \sin (x) \end{aligned}$$ (17) 3.7 monotonic a monotonic activation function is a function that only increases or decreases as its input increases, being allowed to stay constant at some ranges. sigmoid, relu, and hyperbolic tangent are examples of functions of this type. the monotonicity of an activation function is a common property found in the literature, although it is not mandatory. for instance, mish [14] and gelu [9] are exceptions to this property, allowing the network to capture more complex nonlinear relationships between data while maintaining differentiability across its entire domain. however, the choice of non-monotonicity of mish and gelu may restrict its use in some tasks. [23] has shown that periodic activation functions are a promising alternative to monotonic activation functions in neural networks. at the time of that work, no studies demonstrated that functions such as sine could fall into local minima [16]. the neural network can use monotonic activation to learn a mapping function that is easier to optimize since the derivative only increases or decreases in one direction. that way, the gradient descendent algorithms can work more efficiently (table 3). table 3. comparison between activation functions and their monotonic property.full size table 4 experiments this section assesses the leakysinelu in different experiments to evaluate its applicability in classification tasks involving time series. initially, the experiments were executed and discussed in 112 equal-length subsetfootnote 1 datasets from the ucr repository [3] (experiments are available in the project repositoryfootnote 2). we conduct the experiments in two well-established architectures. the first is a multi-layer perceptron (mlp) [5, 24], representing the most basic and straightforward architectures. the second choice is a fully convolutional network (fcn) [5, 24], an architecture representing networks that use convolutional structures to obtain more accurate results. mlp consists of a network with two blocks, each composed of a dropout (\(p=0.1\) and \(p=0.2\), respectively, in each block) and a layer with 500 neurons. next, a last block is used, composed of a dropout with \(p=0.3\) and another layer with the number of classes being the number of neurons, where a softmax activation is applied for multi-class classification or just 1 neurons for binary classification. each layer includes the activation applied to that experiment. fcn comprises three convolutional blocks (128, 256, and 128 channels, respectively) with kernel sizes of 8, 5, and 3 applied in each layer. the output of these blocks is used in an average pooling and then passed to a layer with the number of neurons equal to the number of classes for multi-class problems or one neuron for binary classification. a softmax activation is used in cases with more than one output neuron. for convolutions, the same padding is used to preserve the input dimensions, no max pooling is applied to the series, and the stride parameter is always equal to 1. the activation used in each block varies according to the experiment. 4.1 experimental setup we conducted experiments using the two chosen architectures, mlp and fcn, to evaluate the applicability of the proposed activation function in time series classification tasks. the mlp network was trained using the adadelta optimizer [29], with a learning rate of 1.0 and 1000 training epochs. the fcn network was trained using the adam optimizer [11], with a learning rate of 0.001 and 2000 training epochs. the loss function used in the experiments was the cross-entropy for multi-class classification and binary cross-entropy for binary classification tasks. the performance of the models was evaluated in terms of accuracy. 5 results this section discusses the results of leakysinelu and all the other approaches over the 112 equal-length ucr datasets. following [5], we use the mean accuracy averaged over the experiments on the test set. in fig. 4, we present a one-versus-one comparison between the leakysinelu function, relu, snake, and prelu in a mlp network. we provide these comparisons to analyze the performance with the most used activation in time series tasks (relu), the most similar activation (snake), and a robust alternative to relu (prelu). fig. 4. comparison between leakysinelu, relu, prelu, and snake on an mlp network among the 112 equal-length datasets. full size image each point in these plots represents one dataset. the x-axis represents the accuracy of the test set using classifier-x, and the y-axis represents the ones using classifier-y. the diagonal line shows a draw-line between these models. note that the leakysinelu improves the accuracy compared to the other activations, besides the competitive results compared to the relu and prelu. as [5], we based our analysis on the recommendation of using the friedman test to reject the null hypothesis and perform a pairwise post-hoc analysis with the average ranking comparison replaced by the wilcoxon signed-rank test with holm’s alpha (5%) correction. to facilitate the visualization of the rankings and the comparison between the methods, we rely on the critical difference diagram. figure 5 presents the critical difference diagram generated over the experiments in the ucr repository with the mlp network. note that the leakysinelu function is in the \(1^{\text {st}}\) position with an average ranking of 4.2402, besides not having a statistical difference from snake and relu derivations. fig. 5. critical difference diagram with the average rank of each classifier over the 112 equal-length datasets of the ucr archive. the horizontal black bar denotes that the models connected with it have no statistical difference. full size image to continue the analysis of the results, we rely on the multi-comparison matrix (mcm) [10]. the mcm is a more stable tool for evaluating methods in terms of variation in the models used for the comparison. figure 6 presents the mcm that compares the leakysinelu in an mlp network with the other activations that do not show a statistical difference in fig. 5. this mcm uses the average accuracy on the ucr equal-length datasets as an ordering metric. observe that, besides the statistical equivalence between these activations, the leakysinelu wins in the majority of the datasets when compared to each other statistically equivalent activation. fig. 6. multi-comparison matrix of leakysinelu in an mlp compared to the other statistical equivalent approaches in fig. 5. bold indicates p-value \(\le \) 0.05. full size image moving on to the fcn analysis, we used the same activations compared previously during the experiments. however, for simplicity reasons, we analyzed in more depth the activations that have already shown to be statistically equivalent, namely leakysinelu, relu, prelu, elu, gelu, and snake. figure 7 presents a one-versus-one comparison between the leakysinelu function, relu, snake, and prelu in a fcn network. in this case, the number of victories remains in favor of leakysinelu. it is possible to observe that the network, even using convolution filters, still benefits from an activation designed and focused on time series problems. fig. 7. comparison between leakysinelu, relu, prelu, and snake on an fcn network among the 112 equal-length datasets. full size image figure 8 presents the critical difference diagram generated over the experiments in the ucr repository with the fcn network. in this case, the critical difference diagram shows that leakysinelu is statistically different from elu. however, it is worth remembering that, as highlighted in [10], an average ranking can be prone to errors. to solve this, we also used mcm to analyze the fcn results. however, it is still worth noting that leakysinelu is the best-ranked activation function, with an average ranking of 3.0491. when we compare its performance with relu (second best-ranked), the proposed function wins in most cases. fig. 8. critical difference diagram with the average rank of each fcn classifier over the 112 equal length datasets of the ucr archive. full size image fig. 9. multi-comparison matrix of leakysinelu in a fcn compared to the other statistical equivalent approaches in fig. 8. bold indicates p-value \(\le \) 0.05. full size image figure 9 presents the mcm that compares the leakysinelu in an fcn network. as mentioned, when varying the analysis to the average accuracy and a statistical test recommended in the literature, mcm provides us with more detailed information. it is possible to notice two things, the first of which is that leakysinelu is statistically different from elu and snake in this analysis. the second is that despite being equivalent to relu and prelu, it can be noted that leakysinelu still wins the majority of cases when compared to these two activations, with the number of wins being 55 and 62, respectively. 6 conclusion in this work, we studied and identified meaningful potential harmful properties of neural network activations for time series data. our study encompasses mathematical analysis of these properties as well as their study for tasks related to time series. we identified weaknesses in the main activations in the literature and, based on this study, we proposed the leakysinelu, an activation that overcomes varied limitations of activation functions commonly used on neural network architectures for time series tasks. through an empirical study, we conducted a series of experiments on benchmark datasets from the literature, showing that the proposed activation may be a relevant alternative to be considered for use in time series classification tasks. future works include improvement in well-defined architectures for time series classification such as inceptiontime [6] and resnet [24] using the proposed activation. the impact of activation choices in other time series-related tasks such as extrinsic regression and forecasting exploring the use of leakysinelu and other periodic properties. notes 1.all datasets that does not have “vary” as time series length in the repository. 2.https://github.com/jose-gilberto/leakysinelu. references bengio, y., simard, p., frasconi, p.: learning long-term dependencies with gradient descent is difficult. ieee trans. neural networks 5(2), 157–166 (1994). https://doi.org/10.1109/72.279181 article  math  google scholar  clevert, d., unterthiner, t., hochreiter, s.: fast and accurate deep network learning by exponential linear units (elus). in: bengio, y., lecun, y. (eds.) 4th international conference on learning representations, iclr 2016, san juan, puerto rico, 2–4 may 2016, conference track proceedings (2016). http://arxiv.org/abs/1511.07289 dau, h.a., et al.: the ucr time series classification archive (2018). https://www.cs.ucr.edu/~eamonn/time_series_data_2018/ elfwing, s., uchibe, e., doya, k.: sigmoid-weighted linear units for neural network function approximation in reinforcement learning. corr abs/1702.03118 (2017). http://arxiv.org/abs/1702.03118 ismail fawaz, h., forestier, g., weber, j., idoumghar, l., muller, p.-a.: deep learning for time series classification: a review. data min. knowl. disc. 33(4), 917–963 (2019). https://doi.org/10.1007/s10618-019-00619-1 article  mathscinet  math  google scholar  ismail fawaz, h., et al.: inceptiontime: finding alexnet for time series classification. data min. knowl. disc. 34(6), 1936–1962 (2020). https://doi.org/10.1007/s10618-020-00710-y article  mathscinet  math  google scholar  foumani, n.m., miller, l., tan, c.w., webb, g.i., forestier, g., salehi, m.: deep learning for time series classification and extrinsic regression: a current survey (2023) google scholar  he, k., zhang, x., ren, s., sun, j.: delving deep into rectifiers: surpassing human-level performance on imagenet classification. in: 2015 ieee international conference on computer vision (iccv), pp. 1026–1034 (2015). https://doi.org/10.1109/iccv.2015.123 hendrycks, d., gimpel, k.: bridging nonlinearities and stochastic regularizers with gaussian error linear units. corr abs/1606.08415 (2016). 1606.08415d google scholar  ismail-fawaz, a., et al.: an approach to multiple comparison benchmark evaluations that is stable under manipulation of the comparate set. arxiv preprint arxiv:2305.11921 (2023) kingma, d.p., ba, j.: adam: a method for stochastic optimization (2017) google scholar  liew, s.s., khalil-hani, m., bakhteri, r.: bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems. neurocomputing 216, 718–734 (2016) google scholar  lima, f.t., souza, v.m.: a large comparison of normalization methods on time series. big data res. 34, 100407 (2023) google scholar  misra, d.: mish: a self regularized non-monotonic activation function (2020) google scholar  nair, v., hinton, g.e.: rectified linear units improve restricted boltzmann machines. in: proceedings of the 27th international conference on machine learning (icml-10), pp. 807–814 (2010) google scholar  parascandolo, g., huttunen, h., virtanen, t.: taming the waves: sine as activation function in deep neural networks (2017). https://openreview.net/forum?id=sks3zf9eg pedamonti, d.: comparison of non-linear activation functions for deep neural networks on mnist classification task (2018) google scholar  pinkus, a.: approximation theory of the mlp model in neural networks. acta numer. 8, 143–195 (1999). https://doi.org/10.1017/s0962492900002919 article  mathscinet  math  google scholar  rasamoelina, a.d., adjailia, f., sinčák, p.: a review of activation function for artificial neural network. in: 2020 ieee 18th world symposium on applied machine intelligence and informatics (sami), pp. 281–286 (2020). https://doi.org/10.1109/sami48414.2020.9108717 ruiz, a.p., flynn, m., large, j., middlehurst, m., bagnall, a.: the great multivariate time series classification bake off: a review and experimental evaluation of recent algorithmic advances. data min. knowl. disc. 35(2), 401–449 (2020). https://doi.org/10.1007/s10618-020-00727-3 article  mathscinet  math  google scholar  silvescu, a.: fourier neural networks. in: ijcnn’99. international joint conference on neural networks. proceedings (cat. no. 99ch36339), vol. 1, pp. 488–491 (1999). https://doi.org/10.1109/ijcnn.1999.831544 sonoda, s., murata, n.: neural network with unbounded activation functions is universal approximator. appl. comput. harmon. anal. 43(2), 233–268 (2017) google scholar  sopena, j., romero, e., alquezar, r.: neural networks with periodic and monotonic activation functions: a comparative study in classification problems. in: 1999 ninth international conference on artificial neural networks icann 99. (conf. publ. no. 470), vol. 1, pp. 323–328 (1999). https://doi.org/10.1049/cp:19991129 wang, z., yan, w., oates, t.: time series classification from scratch with deep neural networks: a strong baseline (2016) google scholar  wen, q., et al.: time series data augmentation for deep learning: a survey. in: proceedings of the thirtieth international joint conference on artificial intelligence. international joint conferences on artificial intelligence organization (2021). https://doi.org/10.24963/ijcai.2021/631 wen, q., et al.: transformers in time series: a survey (2023) google scholar  wu, j., zhang, z., ji, y., li, s., lin, l.: a resnet with ga-based structure optimization for robust time series classification. in: 2019 ieee international conference on smart manufacturing, industrial & logistics engineering (smile), pp. 69–74 (2019). https://doi.org/10.1109/smile45626.2019.8965287 yang, x., zhang, z., cui, r.: timeclr: a self-supervised contrastive learning framework for univariate time series representation. knowl.-based syst. 245, 108606 (2022) google scholar  zeiler, m.d.: adadelta: an adaptive learning rate method (2012) google scholar  zhumekenov, a., uteuliyeva, m., kabdolov, o., takhanov, r., assylbekov, z., castro, a.j.: fourier neural networks: a comparative study (2019) google scholar  ziyin, l., hartwig, t., ueda, m.: neural networks fail to learn periodic functions and how to fix it. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 1583–1594. curran associates, inc. (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/1160453108d3e537255e9f7b931f4e90-paper.pdf download references acknowledgement the authors would like to acknowledge the financial support by grants #2022/03176-1, #2023/02680-0, #2023/11775-5, #2023/05041-9 and grant #2023/11745-9, são paulo research foundation (fapesp). author information authors and affiliations universidade de são paulo, av. trab. são carlense, 400 parque arnold schimidt, são carlos, sp, brazil josé gilberto barbosa de medeiros júnior, andré guarnier de mitri & diego furtado silva authors josé gilberto barbosa de medeiros júniorview author publications search author on:pubmed google scholar andré guarnier de mitriview author publications search author on:pubmed google scholar diego furtado silvaview author publications search author on:pubmed google scholar corresponding author correspondence to josé gilberto barbosa de medeiros júnior . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de medeiros júnior, j.g.b., de mitri, a.g., silva, d.f. (2025). semi-periodic activation for time series classification. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_6 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords time series deep learning activation function publish with us policies and ethics profiles diego furtado silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature special-crowd-distance boosted mesh applied to the operation of cascade hydro-power plants | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper special-crowd-distance boosted mesh applied to the operation of cascade hydro-power plants conference paper first online: 30 january 2025 pp 442–456 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) special-crowd-distance boosted mesh applied to the operation of cascade hydro-power plants download book pdf download book epub ronald albert de araújo júnior9,10, gabriel matos cardoso leite  orcid: orcid.org/0000-0002-1486-346x10,11, silvia jiménez-fernández  orcid: orcid.org/0000-0002-2065-175411, sancho salcedo-sanz  orcid: orcid.org/0000-0002-4048-167611, carla amor divino moreira delgado  orcid: orcid.org/0000-0003-3570-44659 & … carolina gil marcelino  orcid: orcid.org/0000-0002-7595-82279  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 377 accesses abstract most of brazil’s and china’s electricity comes from renewable sources through hydroelectric power plants. the operation of these plants can be modeled as a multi-objective optimization problem, in which the generation of energy, the water level in the reservoirs, and the minimal alteration of water flow after the optimization process are maximized. in this context, the work proposes to study the performance of different evolutionary meta-heuristics applied to the problem of electric dispatch in cascade mode power plants (at least two plants on a riverbed). the work has a special focus on the multi-objective evolutionary swarm hybridization (mesh) algorithm boosted by the special-crowd-distance operator, to solve three conflicting objectives in an operation multi-reservoir system (omrs). experiments in this work indicate that mesh version proposed here is competitive when compared to standard evolutionary meta-heuristics: the multi-objective evolutionary algorithm based on decomposition (moea/d), versions of non-dominated sorting genetic algorithm (nsga ii and iii), and the standard mesh solving the real-world problem here addressed. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-objective differential evolution for design of cascade hydropower reservoir systems article 21 august 2018 optimizing multiple linear rules for multi-reservoir hydropower systems using an optimization method with an adaptation strategy article 01 september 2019 economic planning and operation of modern power networks using dynamic switched crowding based multi-objective manta ray foraging optimization article 21 november 2025 1 introduction hydropower-generated energy is an indispensable and sustainable energy source. by harnessing the power of flowing water, this renewable energy option offers numerous advantages, especially regarding the current environmental changes happening on the planet. hydroelectric power plants (hpps) provide a responsible and eco-friendly solution for generating electricity while minimizing environmental impact. however, one of the greatest challenges lies in effectively modeling the operations of these power plants, considering both power generation and their associated impacts. extensive studies have been conducted on the environmental effects of hpps [1], and methodologies have been developed to mitigate these impacts [2, 3]. many approaches consist of optimizing an energy dispatch model, operation multi-reservoir system (omrs) [4, 5]. in the context of multi-reservoir systems, the optimization problem presents inherent challenges, such as a non-linear objective function and a vast search space. not rarely, multi-objective models are also considered. methods for solving omrs optimization problems vary from classical deterministic methods to stochastic evolutionary meta-heuristics. different traditional methods have been employed in the optimization of hpps’ operations, such as linear programming (lp) [6], mixed integer linear programming (milp) [7, 8] and non-linear programming (nlp) [9, 10]. bio-inspired meta-heuristics have also been employed as a method for solving omrs problems. in [11, 12], the authors make use of evolutionary algorithms to optimize energy generation in hydropower plants. the work by [13] uses particle swarm optimization (pso) to find the optimal benefit-cost ratio in hydro power plants design, pso was also applied to the optimization of daily scheduling in hpps reservoirs [14]. multi-objective approaches have gained prominence in recent years. as examples we can highlight, a combination approach for downstream plants to solve scheduling information asymmetry problem in electricity markets was realized by [15], an optimization of long-term operation of the large karun in iran was carried out by [16], and a study on the regulation potential of the upper yellow river reservoirs in china discussed by [17]. even though evolutionary algorithms have presented a certain degree of success in the electric dispatch problem, there are specific weaknesses associated with these approaches, especially related to their non-deterministic nature. such methods are not guaranteed to find the global optima and at times, they get trapped in local optima due to premature convergence [18, 19]. besides the issue of local optima, it is common that problems present a multimodal structure – the case where different parts of the pareto frontier are scattered across the decision space. recently, many studies have been conducted on techniques to address multimodal multi-objective problems [20, 21], as well as theoretical studies which aim to provide a better understanding of these search spaces [22]. in this work, we investigated the impact of including the special-crowd-distance operation in the mesh algorithm and verified the ability of this algorithm to solve a problem with three conflicting objectives, which had not yet been studied. the paper has been organized as follows: sect. 2 describes the model of the operation multi-reservoir system with three conflicting objectives. section 3 details the mesh algorithm proposed using the special-crowd-distance operator by [21]. the simulation, results and comparative analysis of algorithms are shown in sect. 4. conclusions regarding the achieved performance of special-crowd-distance boosted mesh algorithm are presented in sect. 5. 2 mathematical model for the operation of cascade hydro-power plants hydropower plants typically operate using one of two approaches: the joint control system and the optimized control system. the joint control system, where the amount of water flow is divided equally among every available turbine generator, and the optimized control system, where the amount of water flow in each turbine generator is decided to maximize the power generation. the model described in the current section assumes an optimized control system, where the decision space of the optimization model is the water flow in each of the generating units of a hydro-power plant. traditionally, power generation in hydropower plants has been optimized based solely on maximizing energy output. however, as sustainability and environmental concerns have gained prominence, the need for a more comprehensive approach has become evident. a multi-objective model takes into account not only the energy generation potential but also other factors such as ecological flow requirements, downstream water quality, and reservoir management. the 2-objective model proposed in [23] and [12] aims to maximize the power generated by the cascade system and the volume of water in each of the hpps reservoirs. in this work such a model is improved by an objective function, proposed in [24], which accounts for a better human control of the optimized flow in each turbine generator. as a result, the model is classified as a 3-objective optimization model. table 1. parameter table of the cascade hpp modelfull size table in such electric dispatch model, the power generated at each timestamp for generating unit (j) at hpp (u), can be calculated with eq. 1 (see table 1 for a description of the variables). $$\begin{aligned} \begin{aligned} ph_{u,j,t} = g \cdot [ \rho _{0} + \rho _1 hl_{u,j,t} + \rho _2 q_{u,j,t} + \rho _3 hl_{u,j,t} q_{u,j,t} \\ + \rho _4 hl_{u,j,t}^2 + \rho _5 q_{u,j,t}^2] \cdot [ hb_{u,j,t} \vardelta _{hu,j,t}] \cdot q_{u,j,t}. \end{aligned} \end{aligned}$$ (1) in which, \(\rho _1, \rho _2, \rho _3, \rho _4,\rho _5\) are coefficients of the power generation equation estimated through ordinary least squares method from historical data, and \(\mathbf {q_t} = [q_{1,1,t}, q_{1,2,t}, ..., q_{u,j_u, t}]\) is the water flow in each turbine generator, i.e. the decision space. moreover, the values of essential variables in the calculation of eq. 1 can be obtained with the following equations $$\begin{aligned} hl_{u,j,t} = hb_{u,j,t} \vardelta _{hu,j,t}, \end{aligned}$$ $$\begin{aligned} hb_{u,j,t} = fcm_{u,t} fcj_{u,t}, \end{aligned}$$ $$\begin{aligned} fcm_{u,t} = a_0 + a_1 \cdot \varpsi _{u, t} + a_2 \cdot \varpsi _{u, t}^2 + a_3 \cdot \varpsi _{u, t}^3 + a_4 \cdot \varpsi _{u, t}^4, \end{aligned}$$ (2) $$\begin{aligned} \begin{aligned} fcj_{u,t} = b_0 + b_1 \cdot (qt_{u,t} + qv_{u,t}) + b_2 \cdot (qt_{u,t} + \\ qv_{u,t})^2 + b_3 \cdot (qt_{u,t} + qv_{u,t})^3 + b_4 \cdot (qt_{u,t} + qv_{u,t})^4, \end{aligned} \end{aligned}$$ (3) as shown in table 1, \(fcm_{u,t}\) (eq. 2) and \(fcj_{u,t}\) (eq. 3) are, respectively, the height of the hpp (u) downstream and upstream at time (t), and coefficients \(a_0, a_1, ..., a_4\) and \(b_0, b_1, ..., b_4\) are estimated with the ordinary least squares method from historical data. in a cascade hpps context, the water balance in the reservoir of each power plant plays a fundamental role, since in such a scenario, the reservoir of a single hpp is influenced by the flow rates of all the previous hpps in the system. the water balance of each hpp in the system is, also, which accounts for the dynamical nature of the optimization problem, since the volume of water in the reservoir is time dependent. in the model proposed in [12] the volume of water in hpp (u) at time t is defined by eq. 4 $$\begin{aligned} \begin{aligned} \varpsi _{u,t} = \varpsi _{u,t 1} + qa_{u, t-1} + qt_{w,td} + qv_{w,td} \\ qt_{u,t-1} qv_{u,t-1} e_{u,t-1} \cdot a_{u,t-1}. \end{aligned} \end{aligned}$$ (4) in which, \(\varpsi _{u,t}\) is the reservoir volume of hpp u at time t, (u) and (w) are distinct hpps in the system and (td) is the time necessary for the flow to go from (w) into (u). additionally, qa denotes the affluent flow, qt represents the turbine flow, qv represents the total flow rate, e corresponds to liquid evaporation, and a represents the area occupied by water in the reservoir (see table 1). the first objective function, denoted as \(f_1\) and proposed in [23], refers to the energy efficiency of the model and aims to maximize the power generated \(ph_{jt}\) per unit water flow. it is formulated as follows: $$\begin{aligned} \text {maximize} \; f_1(\mathbf {q_t}) = \frac{1}{u} \sum _{u=1}^u\left( \frac{\sum _{j=1}^{j_u} ph_{uj,t}}{\sum _{j=1}^{j_u} q_{uj,t}}\right) . \end{aligned}$$ here, u represents the number of cascaded power plants, and \(j_u\) denotes the number of generating units in power plant u. the second objective function, \(f_2\), proposed in [12], aims to maximize the water levels in the reservoir. maintaining high water volume levels enhances the robustness of the hydropower plant during dry periods. moreover, by increasing the water volume, the value of the upstream level (cota montante) is also elevated, thus increasing the power output. the function is given by: $$\begin{aligned} \text {maximize} \; f_2(\mathbf {q_t}) = \frac{1}{u} \sum _{u=1}^u\left( \frac{v_{u,t}}{v_u^{max}}\right) . \end{aligned}$$ where \(v_u^{max}\) represents the maximum reservoir volume. the third objective function, denoted as \(f_3\), is adapted for cascade hydro-power plants from [24]. it minimizes the euclidean distance between the optimized flow rates of the generating units and the traditional flow rates in the joint control system, where all generating units have the same flow rate (as is typically operated). this objective aims to increase the confidence of the hydro-power plant operators in the optimized flow rates by seeking those that are closest to the coordinated control flow rates while maximizing energy efficiency and reservoir water volume. the function is formulated as: $$\begin{aligned} \text {minimize}\; f_3(\mathbf {q_t}) = \frac{1}{u} \sum _{u=1}^u \left( \sum _{j=1}^{j_u} (q_{uj,t} q_{u,cc})^2\right) . \end{aligned}$$ here, \(q_{u, cc}\) represents the flow rate of the generating units in power plant u in the coordinated control model. in addition to the objective functions, the decision space is subject to certain constraints [12]. the first constraint, described in eq. 5, ensures that the energy generated by the hydro-power plant matches the demand \(dm_{u,t}\) with a specified error tolerance \(\epsilon \) (approximately 0.5%). $$\begin{aligned} dm_{u,t} \cdot (1 \epsilon ) \le \sum _{j=1}^{j_u} ph_{uj,t} \le dm_{u,t} \cdot (1 + \epsilon ). \end{aligned}$$ (5) furthermore, the water flow in each generating unit must comply with the minimum flow rate \(q_{uj}^{min}\) and maximum flow rate \(q_{uj}^{max}\) limits of power plant u. the same applies to the water volume in the reservoir, which should fall within the minimum volume \(v_u^{min}\) and maximum volume \(v_u^{max}\) limits, according to: $$\begin{aligned} q_{uj}^{min} \le q_{uj,t} \le q_{uj}^{max}, \end{aligned}$$ $$\begin{aligned} v_u^{min} \le v_{u,t} \le v_u^{max}. \end{aligned}$$ the defluent flows of the hydro-power plants also need to respect the interval \([qd_{u}^{min}, qd_{u}^{max}]\) to prevent flooding in the regions surrounding the plants, as: $$\begin{aligned} qd_{u}^{min} \le qv_{u,t} + \sum _{j=1}^{j_u} q_{uj,t} \le qd_{u}^{max}. \end{aligned}$$ a maximum limit \(qv_u^{max}\) is imposed on the spill flow rate as well: $$\begin{aligned} qv_{u, t} \le qv_u^{max}. \end{aligned}$$ finally, the power generated by generating unit j must fall within the minimum \(ph_{uj}^{min}\) and maximum \(ph_{uj}^{max}\) limits for that unit in power plant u, $$\begin{aligned} ph_{uj}^{min} \cdot z_{uj,t} \le ph_{uj,t} \le ph_{uj}^{max} \cdot z_{ujt} . \end{aligned}$$ in which, \(z_{uj,t} \in {0, 1}\) is a binary variable indicating whether generating unit j of power plant u is active (\(z_{uj,t} = 1\)) or inactive (\(z_{uj,t} = 0\)) at time t. 3 multi-objective evolutionary swarm hybridization algorithm evolutionary algorithms are a class of computational techniques inspired by the process of biological evolution. they are widely used in optimization problems, particularly in cases where traditional mathematical approaches are difficult to apply. evolutionary algorithms mimic the principles of natural selection, reproduction, and mutation to iteratively search for optimal or near-optimal solutions. when it comes to multi-objective problems, where multiple conflicting objectives need to be simultaneously optimized, evolutionary algorithms offer a powerful approach. one of such is the multi-objective evolutionary swarm hybridization algorithm (mesh) proposed in [12]. the algorithm is a multi-objective version of the previously proposed c-deepso algorithm [25], which implements hybridization techniques by combining operators from differential evolution [26] and particle swarm optimization [27]. as in many optimization approaches through evolutionary algorithms, mesh aims to find the optimal solution by keeping a population of solutions each iteration, which is updated according to the evolutionary operators. the swarm optimization methodology, followed by mesh, finds the optimal solution by updating every particle with a velocity vector that captures information from the entire population of where the optimal solution is. this procedure, for a single particle \(x_i^t\) is illustrated in eqs. 6 and 7. $$\begin{aligned} x^{t + 1}_{i} = x^{t}_i + v^{t}_i, \end{aligned}$$ (6) $$\begin{aligned} v^{t}_i = w_i^* \cdot v^{t 1}_i + w_a^* \cdot (x^{t}_{sn} x^{t}_i) + w_c^* \cdot c \cdot (x_{gb}^* x^{t}_i). \end{aligned}$$ (7) similar to the procedure carried out by particle swarm optimization (pso), the velocity vector \(v^{t}_i\) is constructed by the sum of three components, named as inertia, cognitive and social components. the main difference between the procedure described in eq. 7 and pso’s resides in the cognitive component, where the term \(x^{t}_{sn}\) is obtained by applying the recombination mechanism from differential evolution (de). the superscript * on several parameters in eq. 7, indicates that such parameter suffers a stochastic mutation process, given by eq. 8. $$\begin{aligned} w^* = w + \tau \cdot n(0,1). \end{aligned}$$ (8) in which \(\tau \) is one of the algorithm’s hyperparameters and n(0, 1) is a sample from a standard gaussian distribution. to account for the multi-objective nature of the problems, the algorithm stores a subset of the best particles found so far. each iteration such subset of size n, located in a separate memory file, is updated through the crowd-distance operator from nsga-ii [28]. the crowd-distance operator from nsga-ii [28] begins by ordering each of the particles \(x_i\) in the population into dominance ranks as follows: in the first rank \(f_1\), there are all the particles not dominated by any other particle in the population p; in the second rank \(f_2\), there are those not dominated by any other particle in the population \(p f_1\); in \(f_3\), there are those not dominated by any other particle in the population \(p f_1 f_2\), and so on. after that, for each particle \(x_i\), its crowd-distance \(i_{i}\) is calculated based on the average distances between the two adjacent particles in all dimensions of the objective space [28, 29]. $$\begin{aligned} \begin{aligned} i_{i}^j = \frac{x^{i + 1}_j x^{i 1}_j}{f_j^{\text {max}} f_j^{\text {min}}}, i_{i} = \sum _{j = 1}^n i_i^j. \end{aligned} \end{aligned}$$ in which, \(x^{i + 1}_j\) and \(x^{i 1}_j\) are the values of the j-th objective function of the two adjacent particles to \(x^i\) when the particles are sorted based on their values for the j-th objective function. \(f_j^{\text {max}}\) and \(f_j^{\text {min}}\) are the maximum and minimum values for such function, respectively. then, the highest ranks (\(f_1, f_2, ...\)) are directly added to the memory file until the entire space of n particles is occupied. if only part of a rank can be inserted into memory, we sort the particles in descending order according to their crowd-distance, and these particles are added in order until the memory is completely full. mesh also admits different configurations for how it operates, those are: e, the choice of operator for the pso swarm guide. (e1) select the nearest particle in memory. (e2) select the nearest particle in the next upper boundary to the current particle’s boundary. v, from which group the particles that compose the differential evolution operator come from. (v1) population. (v2) memory. (v3) population + memory. d, which of the strategies for the differential evolution operator, will be used. (d1) de/rand/1/bin. (d2) de/rand/2/bin. (d3) de/best/1/bin. (d4) de/current-to-best/1/bin. (d5) de/current-to-rand/1/bin. therefore the total amount of possible different ways mesh can operate is \(2 \times 3 \times 5 = 60\). in this work, we focus on the e2v2d1 configuration, which, from empirical tests, appears to be the most stable one for general problem-solving. 3.1 special crowd distance in addition to the original configurations of the multi-objective evolutionary swarm hybridization (mesh) algorithm, this work implemented an additional parameter related to the crowd distance operator. unlike the original crowd distance operator defined by the nsga-ii algorithm, the additional configuration allows for the use of the special crowd distance operator defined in [21]. many optimization problems exhibit the characteristic of being multimodal, meaning they have distinct local maxima in different regions of the decision space. in this context, it is important to consider diversity not only in the objective space (as in the original nsga-ii operator) but also in the decision space. to calculate the crowd distance in the decision space, we use the following expression proposed by [21]: $$\begin{aligned} cd_{i,x} = \sum _{j = 1}^{k} (k j + 1) d_{i, j}. \end{aligned}$$ in which, \(cd_{i,x}\) is the crowd distance in the decision space, i is the index of a particle in the population, j is the index of a particle in the neighborhood of the particle with index i, \(d_{i,j}\) is the euclidean distance from the i-th particle in the entire population to the j-th particle in the neighborhood, and k is the neighborhood size. the crowd distance in the objective space of the i-th particle, denoted as \(cd_{i, f}\), is calculated as in the original nsga-ii operator with a slight modification: the crowd distance is not calculated considering only one pareto rank but the entire population [21]. thus, the special crowd distance of the i-th particle, denoted as \(scd_i\), is obtained, according [21] : $$\begin{aligned} \begin{aligned} scd_{i} = {\left\{ \begin{array}{ll} max(cd_{i,x}, cd_{i,f}), & \text {if } cd_{i,x} > cd_{avg, x} \\ \text { or } cd_{i,f} > cd_{avg, f}\\ min(cd_{i,x}, cd_{i,f}), & \text {otherwise} \end{array}\right. } \end{aligned} \end{aligned}$$ where \(cd_{avg, x}\) and \(cd_{avg, f}\) are the averages of the crowd distances in the decision space and the objective space, respectively. in the special crowd distance operator, the particles in the last rank to enter the memory are chosen according to the ordering of \(scd_i\). to incorporate the special crowd distance [21] into the mesh algorithm, a new configuration for mesh is added. (c1) traditional crowd distance operator from nsga-ii algorithm. (c2) special crowd distance from [21]. and the total number of possible configurations for mesh, with the crowd distance new parameter, is \(2 \times 3 \times 5 \times 2 = 120\). a previous study was carried out in [12]. then, in this work the e2v2d1–mesh configuration was used for tested special crowd distance operation. 4 experiments and results the experiments conducted in this study compare the performance of the special-crowd-distance boosted mesh algorithm (e2v2d1c2 version) against three other well-established metaheuristics: standard mesh (e2v2d1c1 version) [12], nsga-ii [28], nsga-iii [30, 31], and moea/d [32]. all three algorithms were implemented in python version 3.7.12 using the pymoo library [33]. the mesh algorithm was evaluated in two distinct versions: e2v2d1c1, which uses the nsga-ii crowd distance operator, and e2v2d1c2, which incorporates a special crowd distance operator based on [21]. for all four algorithms, the termination criterion was set to 15.000 objective function evaluations and a population size of 100 particles. the algorithms are tested on their performance in solving a real-world problem concerning electric dispatch and were executed 30 times, as described in sect. 2. the main criterion employed for comparing the performance of the algorithms in the experiments is hypervolume, a widely used metric in multi-objective optimization. to assess the statistical significance of the observed results in hypervolume values among different algorithms, the mann-whitney u test [34] is employed. the mann-whitney u test is used to determine whether two independent samples originate from the same distribution or if there is a significant difference between them. such a test was chosen due to its non-parametric nature, as the hypervolume samples do not appear to be drawn from a gaussian distribution. by utilizing this test, we can confidently evaluate the performance disparities between the mesh algorithm versions and the other meta-heuristics under consideration. moreover, the statistical test was applied between the algorithm with the highest hypervolume results and each of the other algorithms. by conducting these pairwise comparisons, we can determine which algorithm exhibits the best performance in terms of hypervolume. the intention behind this methodology is to appropriately declare the algorithm that outperforms the others based on the statistical significance of the results. the proposed model for the cascade hydro-power plant is dynamic, and a strategy is employed to reevaluate the pareto front as it changes over time. for each hour, 30 algorithm runs are performed, combining the solutions found into a combined pareto front. the fig. 1 shows the combined pareto front obtained by each algorithm in a sample of hours of a day. in the context of combined pareto frontiers, moea/d did not perform satisfactorily, its results for the hypervolume metric are consistently lower than those of the other algorithms in all hours of operation of the hydroelectric plants. the best results for all hours were between the mesh versions (e2v2d1c1 and e2v2d1c2), with a certain tendency towards the e2v2d1c2 configuration that we are proposing in this work, which obtained the highest results for hypervolume in most hpps operating hours. in general, 4 of the 5 algorithms were able to find consistent pareto frontiers and proved to be efficient when applied to the optimization of the electrical dispatch model in omrs scheme. except for moea/d, all algorithms appear to have good performance in finding the pareto frontier for all hours of operation of the hydroelectric plant. finally, it is notable that the mesh versions better cover the range of the solution set, at all hours of operation. from this set of solutions, inside the combined pareto front, a single solution must be selected. the values of the decision space variables in this selected solution will determine the future values of the dynamic variables. in this work, the selected solution is the one closest to the point in the middle of pareto front. table 2 contains the results for the average and standard deviation of hypervolume in each of the tested algorithms. e2v2d1c2 configuration of mesh presented the highest average values for most operating hours. however, its results were very close to the results of the e2v2d1c1 configuration, in a way that no statistically significant difference was detected in 17 out of 24 h. moreover, both mesh configurations achieved superior results to the well-established algorithms in the literature. in addition to the good hypervolume results for both mesh versions, in table 2, it is noticeable that the standard deviation results for both configurations are consistently more controlled for all operating hours compared to the other three algorithms evaluated in this experiment. fig. 1. combined pareto fronts full size image table 2. average and standard deviation (in bracket) results for the hypervolumes of each algorithm in daily electric dispatch per hour (h). the symbol ‘=’ means there are no statistical differences between algorithms. the symbol ‘+’ means that the algorithm has statistical difference with others, with better average. the symbol ‘−’ means that the algorithm has statistical difference with others, with worst average.full size table as seen in [12], where the model was implemented with 2 objective functions, all algorithms failed to find the pareto front for the last operating hours. in this work, we incorporate a third objective function. the results show that the pareto fronts found by 4 out of the 5 algorithms reveal that the addition of the third objective function caused an improvement in the results. in particular, the special-crowd-distance boosted mesh (e2v2d1c2) showed statistical robustness indicating that this approach benefits mesh for treating the omrs problem, making it competitive compared to the most used algorithms from the pymoo library. 5 conclusion in this work, the performance of different evolutionary metaheuristics in solving a dispatch operation problem in cascade hydroelectric plants was analyzed. the work introduces a new version of the mesh algorithm, which added an alternative to its crowd distance operator, special crowding distance. the mathematical modeling of the electrical problem was improved with the introduction of a third objective function. this function indicates the distance between the optimized water flow and the water flow in joint control mode. it guarantees a smoothing of the set of solutions found. therefore, the work shows the first attempt to solve three conflicting objectives with the standard version and the special-crowd-distance boosted mesh version. this version of mesh proved to be competitive compared to classic algorithms such as nsga-ii, nsga-iii, and moea/d, overlapping their results. this indicates that the special-crowd-distance operation can benefit multi-objective algorithms based on pareto front. as future outcomes we indicate carry out a fine-tuning study of parameters for the new version of the mesh algorithm presented here, and also a decision-making study of which are the most suitable solutions for implementing electrical dispatch, taking into account the three objectives studied. references socio-environmental impacts of hydro power technologya review, vol. asme 2021 power conference of asme power conference (2021) google scholar  costea, g., pusch, m.t., bănăduc, d., cosmoiu, d., curtean-bănăduc, a.: a review of hydropower plants in romania: distribution, current knowledge, and their effects on fish in headwater streams. renew. sustain. energy rev. 145, 111003 (2021) article  google scholar  stevovic, s., milovanovic, z., stamatovic, m.: sustainable model of hydro power development–drina river case study. renew. sustain. energy rev. 50, 363–371 (2015) article  math  google scholar  wang, h., et al.: multi-reservoir system operation theory and practice. in: wang, l.k., yang, c.t., wang, m.-h.s. (eds.) advances in water resources management. hee, vol. 16, pp. 1–110. springer, cham (2016). https://doi.org/10.1007/978-3-319-22924-9_1 chapter  math  google scholar  ahmadianfar, i., et al.: multi-strategy slime mould algorithm for hydropower multi-reservoir systems optimization. knowl.-based syst. 250, 109048 (2022) article  math  google scholar  yoo, j.-h.: maximization of hydropower generation through the application of a linear programming model. j. hydrol. 376, 182–187 (2009) article  math  google scholar  alvarez, g.e.: an optimization model for operations of large scale hydro power plants. ieee lat. am. trans. 18, 1631–1638 (2021) article  math  google scholar  kumar, j., khalil, i.u., haq, a.u., perwaiz, a., mehmood, k.: solver-based mixed integer linear programming (milp) based novel approach for hydroelectric power generation optimization. ieee access 8, 174880–174892 (2020) article  google scholar  catalão, j., pousinho, h., mendes, v.: scheduling of head-dependent cascaded reservoirs considering discharge ramping constraints and start/stop of units. int. j. electr. power energy syst. 32(8), 904–910 (2010) article  google scholar  zheng, f., simpson, a.r., zecchin, a.c.: a combined nlp-differential evolution algorithm approach for the optimization of looped water distribution systems. water resour. res. 47(8) (2011) google scholar  sharifi, m.r., akbarifard, s., madadi, m.r., qaderi, k., akbarifard, h.: optimization of hydropower energy generation by 14 robust evolutionary algorithms. sci. rep. 12, 7739 (2022) article  math  google scholar  marcelino, c.g., et al.: an efficient multi-objective evolutionary approach for solving the operation of multi-reservoir system scheduling in hydro-power plants. expert syst. appl. (2021) google scholar  rahi, o.p., kumar, a., sharma, m.: optimization of hydro power plant design by particle swarm optimization (pso). procedia eng. 30, 418–425 (2012) article  math  google scholar  moreno, s.r., kaviski, e.: daily scheduling of small hydro power plants dispatch with modified particles swarm optimization. pesquisa operacional 35, 25–37 (2015) article  math  google scholar  li, y., wang, x., cheng, w., gao, s., cheng, c.: a combination approach for downstream plants to solve scheduling information asymmetry problem in electricity markets. int. j. electr. power energy syst. 149, 108935 (2023) article  math  google scholar  sharifi, m.r., akbarifard, s., madadi, m.r., qaderi, k., akbarifard, h.: application of momsa algorithm for optimal operation of karun multi objective multi reservoir dams with the aim of increasing the energy generation. energ. strat. rev. 42, 100883 (2022) article  google scholar  bai, t., et al.: multi-objective and multi-scheme research on water and sediment regulation potential of reservoirs in the upper yellow river. int. j. sedim. res. 38(2), 203–215 (2023) article  math  google scholar  guedes, l., vieira, d., lisboa, a., saldanha, r.: a continuous compact model for cascaded hydro-power generation and preventive maintenance scheduling. int. j. electr. power energy syst. 73, 702–710 (2015) article  math  google scholar  zhang, r., zhou, j., ouyang, s., wang, x., zhang, h.: optimal operation of multi-reservoir system by multi-elite guide particle swarm optimization. int. j. electr. power energy syst. 48, 58–68 (2013) article  math  google scholar  liang, j., et al.: multimodal multiobjective optimization with differential evolution. swarm evol. comput. 44, 1028–1059 (2019) article  math  google scholar  yue, c., et al.: differential evolution using improved crowding distance for multimodal multiobjective optimization. swarm evol. comput. 62, 100849 (2021) article  math  google scholar  kerschke, p., et al.: search dynamics on multimodal multiobjective problems. evol. comput. 27, 577–609 (2019) article  math  google scholar  marcelino, c.g., camacho-gómez, c., jiménez-fernández, s., salcedo-sanz, s.: optimal generation scheduling in hydro-power plants with the coral reefs optimization algorithm. energies 14(9), 2443 (2021) article  math  google scholar  marcelino, c.g., carvalho, l.m., almeida, p.e.m., wanner, e.f., miranda, v.: application of evolutionary multiobjective algorithms for solving the problem of energy dispatch in hydroelectric power plants. in: gaspar-cunha, a., henggeler antunes, c., coello, c.c. (eds.) emo 2015. lncs, vol. 9019, pp. 403–417. springer, cham (2015). https://doi.org/10.1007/978-3-319-15892-1_27 chapter  google scholar  marcelino, c., almeida, p., pedreira, c., caroalha, l., wanner, e.: applying c-deepso to solve large scale global optimization problems. in: 2018 ieee congress on evolutionary computation (cec), pp. 1–6 (2018) google scholar  storn, r., price, k.: differential evolution a simple and efficient heuristic for global optimization over continuous spaces. j. global optim. 11, 341–359 (1997) article  mathscinet  math  google scholar  kennedy, j., eberhart, r.: particle swarm optimization. in: proceedings of icnn 1995 international conference on neural networks, vol. 4, pp. 1942–1948 (1995) google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  math  google scholar  liu, j., chen, x.: an improved nsga-ii algorithm based on crowding distance elimination strategy. int. j. comput. intell. syst. 12, 513–518 (2019) article  math  google scholar  deb, k., jain, h.: an evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. ieee trans. evol. comput. 18(4), 577–601 (2014) article  math  google scholar  jain, h., deb, k.: an evolutionary many-objective optimization algorithm using reference-point based nondominated sorting approach, part ii: handling constraints and extending to an adaptive approach. ieee trans. evol. comput. 18(4), 602–622 (2014) article  math  google scholar  zhang, q., li, h.: moea/d: a multiobjective evolutionary algorithm based on decomposition. ieee trans. evol. comput. 11(6), 712–731 (2007) article  math  google scholar  blank, j., deb, k.: pymoo: multi-objective optimization in python. ieee access 8, 89497–89509 (2020) article  math  google scholar  mann, h.b., whitney, d.r.: on a test of whether one of two random variables is stochastically larger than the other. ann. math. stat. 18(1), 50–60 (1947) article  mathscinet  math  google scholar  download references acknowledgement this work was partially funded by the brazilian research agencies: pibic/pibiti-ufrj, pibic/pibiti-cnpq, capes (grant number: 23038.006308/2021-70), cnpq-national council for scientific and technological development (grant number 403964/2023-7), faperjfundação carlos chagas filho de amparo à pesquisa do estado do rio de janeiro (grant numbers e-26/210.057/2024, e-26/210.549/2024). this research has also been partially supported by projects pid2020-115454gb-c21 and ted2021-131777b-c22 of the spanish ministry of science and innovation (micinn). the authors thank ic-ufrj, pesc-ufrj, and uah for the infrastructure used to conduct this work. author information authors and affiliations instituto de computação, universidade federal do rio de janeiro (ufrj), rio de janeiro, brazil ronald albert de araújo júnior, carla amor divino moreira delgado & carolina gil marcelino systems and computing department, universidade federal do rio de janeiro (ufrj), rio de janeiro, brazil ronald albert de araújo júnior & gabriel matos cardoso leite department of signal processing and communication, alcalá university (uah), madrid, spain gabriel matos cardoso leite, silvia jiménez-fernández & sancho salcedo-sanz authors ronald albert de araújo júniorview author publications search author on:pubmed google scholar gabriel matos cardoso leiteview author publications search author on:pubmed google scholar silvia jiménez-fernándezview author publications search author on:pubmed google scholar sancho salcedo-sanzview author publications search author on:pubmed google scholar carla amor divino moreira delgadoview author publications search author on:pubmed google scholar carolina gil marcelinoview author publications search author on:pubmed google scholar corresponding author correspondence to carolina gil marcelino . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de araújo júnior, r.a., leite, g.m.c., jiménez-fernández, s., salcedo-sanz, s., delgado, c.a.d.m., marcelino, c.g. (2025). special-crowd-distance boosted mesh applied to the operation of cascade hydro-power plants. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_31 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-objective optimization special-crowd-distance operation multi-reservoir system publish with us policies and ethics profiles gabriel matos cardoso leite view author profile sancho salcedo-sanz view author profile carolina gil marcelino view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a kullback-leibler divergence-based locally linear embedding method: a novel parametric approach for cluster analysis | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a kullback-leibler divergence-based locally linear embedding method: a novel parametric approach for cluster analysis conference paper first online: 28 november 2021 pp 406–420 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a kullback-leibler divergence-based locally linear embedding method: a novel parametric approach for cluster analysis download book pdf download book epub alexandre l. m. levada  orcid: orcid.org/0000-0001-8253-272910 & michel f. c. haddad  orcid: orcid.org/0000-0002-0978-952511,12  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 814 accesses abstract numerous problems in machine learning require some type of dimensionality reduction. unsupervised metric learning deals with the definition of intrinsic and adaptive distance functions of a dataset. locally linear embedding (lle) consists of a widely used manifold learning algorithm that applies dimensionality reduction to find a more compact and meaningful representation of the observed data through the capture of the local geometry of the patches. in order to overcome relevant limitations of the lle approach, we introduce the lle kullback-leibler (lle-kl) method. our objective with such a methodological modification is to increase the robustness of the lle to the presence of noise or outliers in the data. the proposed method employs the kl divergence between patches of the knn graph instead of the pointwise euclidean metric. our empirical results using several real-world datasets indicate that the proposed method delivers a superior clustering allocation compared to state-of-the-art methods of dimensionality reduction-based metric learning. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others using kullback-liebler divergence based meta-learning algorithm for few-shot skin cancer image classification: literature review and a conceptual framework chapter © 2022 dissimilarity space reinforced with manifold learning and latent space modeling for improved pattern classification article open access 18 october 2021 dimensionality reduction and visualization by doubly kernelized unit ball embedding chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. global analysis and analysis on manifolds learning algorithms learning theory statistical learning diffusion processes and stochastic analysis on manifolds statistical theory and methods 1 introduction the main objective of dimensionality reduction unsupervised metric learning is to produce a lower dimensional representation that preserves the intrinsic local geometry of the data. the locally linear embedding (lle) consists of one of the pioneering algorithms of such a class, which has been successfully applied to non-linear feature extraction for pattern classification tasks [9]. although more efficient than linear methods, the lle still has some important limitations. firstly, it is rather sensitive to noise or outliers [14]. secondly, in datasets that are not represented by smooth manifolds, the lle frequently fails to produce reasonable results. it is worth mentioning that variations of the lle have been proposed to overcome such limitations, with varying degrees of success. in hessian eigenmaps, the local covariance matrix used in the computation of the optimal reconstruction weights is replaced by the hessian matrix (i.e., the second order derivatives matrix), which encodes relevant curvature information [1, 13, 15]. a further extension is known as the local tangent space alignment (ltsa). the difference is that the ltsa builds the locally linear patch with an approximation of the tangent space through the application of the principal component analysis (pca) over a linear patch. subsequently, the local representations are aligned to the point in which all tangent spaces become aligned in the unfolded manifold [16]. in addition, the modified lle (mlle) introduces several linearly independent weight vectors for each neighborhood of the knn graph. some works in the relevant literature describe that the local geometry of mlle is more stable in comparison with the lle [17]. one remaining issue still present in such recent developments of the lle method is the adoption of euclidean distance as the similarity measure within a linear patch. in the proposed lle-kl method, we incorporate the kl divergence into the estimation of the optimal reconstruction weights. the main contribution of the proposed method is that, through the replacement of the pointwise euclidean distance by a patch-based information-theoretic distance (i.e., kl divergence), the lle becomes less sensitive to the presence of noise or outliers. our empirical clustering analysis performed subsequently to the dimensionality reduction for several real-world datasets popularly used in the machine learning literature, indicate that the proposed lle-kl method is capable of generating more reasonable clusters in terms of silhouette coefficients compared to state-of-the-art manifold learning algorithms, such as the isomap [12] and umap [6]. the remainder of the paper is organized as follows. section 2 discusses previous relevant work, with focus on the relative entropy and the regular lle algorithm. section 3 details the proposed lle-kl algorithm. section 4 presents the data used as well as computational experiments and empirical results. section 5 concludes with our final remarks and suggestions for future research. 2 related work in this section, we discuss the relative entropy (kl divergence) between probability density functions as a similarity measure among random variables, exploring its computation in the gaussian case. moreover, we discuss the lle algorithm and detail its mathematical derivation. 2.1 kullback-leibler divergence in machine learning applications, the problem of quantifying similarity levels between different objects or clusters consists of a challenging task, especially in cases where the standard euclidean distance is not a reasonable choice. many studies on feature selection adopt statistical divergences to select the set of features that should maximize some measure of separation between classes. part of their success is due to the fact that most dissimilarity measures are related to distance metrics. in such a context, information theory provides a solid mathematical background for metric learning in pattern classification. the entropy of a continuous random vector x is given by: $$\begin{aligned} h(p) = \int p(x) [log~p(x)] dx = e \left[ log~p(x) \right] \end{aligned}$$ (1) where p(x) is the probability density function (pdf). in a similar fashion, we may define the cross-entropy between two probability density functions p(x) and q(x): $$\begin{aligned} h(p, q) = \int p(x) [log~q(y)] dx \end{aligned}$$ (2) the kl divergence is the difference between cross-entropy and entropy [4]: $$\begin{aligned} d_{kl}(p, q) = h(p, q) h(p) = \int p(x) log \left( \frac{p(x)}{q(x)}\right) dx = e_{p} \left[ log \left( \frac{p(x)}{q(x)} \right) \right] \end{aligned}$$ (3) an important property is that the relative entropy is non-negative, therefore, \(d_{kl}(p, q) \ge 0\). 2.2 the lle algorithm the lle consists of a local method, thus, the new coordinates of any \(\vec {x}_i \in r^m\) depends only on the neighborhood of the respective point. the main hypothesis behind the lle is that for a sufficiently high density of samples, it is expected that a vector \(\vec {x}_i\) and its neighbors define a linear patch (i.e., they all belong to an euclidean subspace [9]). hence, it is possible to characterize the local geometry by linear coefficients: $$\begin{aligned} \hat{\vec {x}}_i \approx \sum _j w_{ij} \vec {x}_j \qquad \text {for} \qquad \vec {x}_j \in n(\vec {x}_i) \end{aligned}$$ (4) consequently, we can reconstruct a vector as a linear combination of its neighbors. the lle algorithm requires as input an \(n \times m\) data matrix x, with rows \(\vec {x}_i\), a defined number of dimensions \(d < m\), and an integer \(k > d + 1\) for finding local neighborhoods. the output is an \(n \times d\) matrix y, with rows \(\vec {y}_i\). the lle algorithm may be divided into three main steps [9, 10], as follows: 1. from each \(\vec {x}_i \in r^m\) find its k nearest neighbors; 2. find the weight matrix w that minimizes the reconstruction error for each data point \(\vec {x}_i \in r^m\); 3. find the coordinates y which minimize the reconstruction error using the optimum weights. in the following subsections, we describe how to obtain the solution to each step of the lle algorithm. finding local linear neighborhoods. a relevant aspect of the lle is that this algorithm is capable of recovering embeddings which intrinsic dimensionality d is smaller than the number of neighbors, k. moreover, the assumption of a linear patch imposes the existence of an upper bound on k. for instance, in highly curved datasets, it is not reasonable to have a large k, otherwise such an assumption would be violated. in the uncommon situation where \(k > m\), it has been shown that each sample may be perfectly be reconstructed from its neighbors, from which another problem emerges: the reconstruction weights are not anymore unique. to overcome this limitation, some regularization is necessary in order to break the degeneracy [10]. lastly, a further concern in the lle algorithm refers to the connectivity of the knn graph. in the case the graph contains multiple connected components, then the lle should be applied separately on each one of them, otherwise the neighborhood selection process should be modified to assure global connectivity [10]. least-squares estimation of the weights. the second step of the lle is to reconstruct each data point from its nearest neighbors. the optimal reconstruction weights may be computed in closed form. without loss of generality, we may express the local reconstruction error at point \(\vec {x}_i\) as follows: $$\begin{aligned} e(\vec {w}) = \left\vert \sum _{j} w_j(\vec {x}_i \vec {x}_j) \right\vert ^2 = \sum _j \sum _k w_j w_k (\vec {x}_i \vec {x}_j)^t(\vec {x}_i \vec {x}_k) \end{aligned}$$ (5) defining the local covariance matrix c as: $$\begin{aligned} c_{jk} = (\vec {x}_i \vec {x}_j)^t (\vec {x}_i \vec {x}_k) \end{aligned}$$ (6) we then have the following expression for the local reconstruction error: $$\begin{aligned} e(\vec {w}) = \sum _j \sum _k w_j c_{jk} w_k = \vec {w}^t c \vec {w} \end{aligned}$$ (7) regarding the constraint \(\sum _j w_j = 1\), that may be interpreted in two different manners, namely geometrically and probabilistically. from a geometric point of view, it provides invariance under translation, thus, by adding a given constant vector \(\vec {c}\) to \(\vec {x}_i\) and all of its neighbors, the reconstruction error remains unchanged. in terms of probability, enforcing the weights to sum to one leads w to become a stochastic transition matrix [10] directly related to markov chains and diffusion maps [11]. as detailed below, in the minimization of the squared error the solution is found through an eigenvalue problem. in fact, the estimation of the matrix w reduces to n eigenvalue problems. considering that there are no constraints across the rows of w, we may then find the optimal weights for each sample \(\vec {x}_i\) separately, drastically simplifying the respective computations. therefore, we have n independent constrained optimization problems given by: $$\begin{aligned} \mathop {\mathrm {arg\,min}}\limits _{\vec {w}_i}~ \vec {w}_i^t c_i \vec {w}_i \quad \text {s.t.} \quad \vec {1}^t \vec {w}_i = 1 \end{aligned}$$ (8) for \(i = 1, 2, ..., n\). using lagrange multipliers, we express the lagrangian function as follows: $$\begin{aligned} l(\vec {w}_i , \lambda ) = \vec {w}_i^t c_i \vec {w}_i \lambda (\vec {1}^t \vec {w}_i 1) \end{aligned}$$ (9) taking the derivatives with relation to \(\vec {w}_i\) as follows: $$\begin{aligned} \frac{\partial }{\partial \vec {w}_i}l(\vec {w}_i , \lambda ) = 2 c_i \vec {w}_i \lambda \vec {1} = 0 \end{aligned}$$ (10) leads to $$\begin{aligned} c_i \vec {w}_i = \frac{\lambda }{2}\vec {1} \end{aligned}$$ (11) which is equivalent to solving the following linear system: $$\begin{aligned} c_i \vec {w}_i = \vec {1} \end{aligned}$$ (12) and then normalizing the solution to guarantee that \(\sum _j w_i(j) = 1\) by dividing each coefficient of the vector \(\vec {w}_i\) by the sum of all the coefficients: $$\begin{aligned} w_i(j) = \frac{\displaystyle w_i(j)}{\displaystyle \sum _j w_i(j)} \qquad \text {for} \qquad j=1,2,...,m \end{aligned}$$ (13) in the case that k (i.e., number of neighbors) is greater than m (i.e., number of features) then, in general, the space spanned by k distinct vectors consists of the whole space. this means that \(\vec {x}_i\) may be expressed exactly as a linear combination of its k-nearest neighbors. in fact, if \(k > m\), there are generally infinitely many solutions to \(\vec {x}_i = \sum _j w_j \vec {x}_j\), due to the fact that there would be more unknowns (k) than equations (m). in such a case, the optimization problem is ill-posed and, thus, regularization is required. a common regularization technique refers to the tikonov regularization, which adds a penalization term to the least squares problem, as follows: $$\begin{aligned} \left\vert {\vec {x}}_i \sum _j w_j {\vec {x}}_j \right\vert ^2 + \alpha \sum _j w_j^2 \end{aligned}$$ (14) where \(\alpha \) controls the degree of regularization. in other words, it selects the weights which minimize a combination of reconstruction error as well as the sum of the squared weights. in the case that \(\alpha \rightarrow 0\), then there is a least-squares problem. however, in the opposite limit (i.e., \(\alpha \rightarrow \infty \)), the squared-error term becomes negligible, allowing the minimization of the euclidean norm of the weight vector \(\vec {w}\). typically, \(\alpha \) is set to be a small but non-zero value. in this case, the n independent constrained optimization problems are: $$\begin{aligned} \mathop {\mathrm {arg\,min}}\limits _{\vec {w}_i}~ \vec {w}_i^t c_i \vec {w}_i + \alpha \vec {w}_i^t \vec {w}_i \quad \text {s.t.} \quad \vec {1}^t \vec {w}_i = 1 \end{aligned}$$ (15) for \(i = 1, 2, ..., n\). the lagrangian function is defined by: $$\begin{aligned} l(\vec {w}_i , \lambda ) = \vec {w}_i^t c_i \vec {w}_i + \alpha \vec {w}_i^t\vec {w}_i \lambda (\vec {1}^t \vec {w}_i 1) \end{aligned}$$ (16) taking the derivative with respect to \(\vec {w}_i\) and setting the result to zero: $$\begin{aligned} 2 c_i \vec {w}_i + 2 \alpha \vec {w}_i = \lambda \vec {1} \end{aligned}$$ (17) $$\begin{aligned} (c_i + \alpha i)\vec {w}_i = \frac{\lambda }{2}\vec {1} \end{aligned}$$ (18) $$\begin{aligned} \vec {w}_i = \frac{\lambda }{2}(c_i + \alpha i)^{-1} \vec {1} \end{aligned}$$ (19) where \(\lambda \) is selected to properly normalize \(\vec {w}_i\), which is equivalent to solving the following linear system: $$\begin{aligned} (c_i + \alpha i) \vec {w}_i = \vec {1} \end{aligned}$$ (20) and then normalize the solution. in other words, to regularize the problem it is necessary to add a small perturbation into the main diagonal of the matrix \(c_i\). in all experiments reported in the present paper, we use the regularization parameter \(\alpha = 10^{-4}\). finding the coordinates. the main idea in the last stage of the lle algorithm is to use the optimal reconstruction weights estimated by least-squares as the proper weights on the manifold and then solve for the local manifold coordinates. thus, fixing the weight matrix w, the goal is to solve another quadratic minimization problem to minimize the following: $$\begin{aligned} \varphi (y) = \sum _{i=1}^{n} \left\vert \vec {y}_i \sum _j w_{ij} \vec {y}_j \right\vert ^2 \end{aligned}$$ (21) hence, the following question needs to be addressed: what are the coordinates \(\vec {y}_i \in r^d\) (approximately on the manifold), that such weights (w) reconstruct? in order to avoid degeneracy, we have to impose two constraints, as follows: 1. the mean of the data in the transformed space is zero, otherwise we would have an infinite number of solutions; 2. the covariance matrix of the transformed data is the identity matrix, therefore, there is no correlation between the components of \(\vec {y} \in r^d\) (this is a statistical constraint to assess that the output space is the euclidean one, defined by an orthogonal basis). however, unlikely the estimation of the weights w, finding the coordinates does not simplify into n independent problems, because each row of y appears in \(\varphi \) multiple times, as the central vector \(y_i\) and also as one of the neighbors of other vectors. thus, firstly, we rewrite equation (21) in a more meaningful manner using matrices, as follows: $$\begin{aligned} \varphi (y) = \sum _{i=1}^{n} \left( \vec {y}_i \sum _j w_{ij} \vec {y}_j \right) ^t \left( \vec {y}_i \sum _j w_{ij} \vec {y}_j \right) \end{aligned}$$ (22) applying the distributive law and expanding the summation, we then obtain: $$\begin{aligned} \varphi (y)&= \sum _{i=1}^{n} \vec {y}_i^t\vec {y}_i \sum _{i=1}^{n} \sum _j \vec {y}_i^t w_{ij} \vec {y}_j \nonumber \\&\sum _{i=1}^{n}\sum _j \vec {y}_j^t w_{ji} \vec {y}_i + \sum _{i=1}^{n} \sum _j \sum _k \vec {y}_j^t w_{ji} w_{ik} \vec {y}_k \end{aligned}$$ (23) denoting by y the \(d \times n\) matrix in which each column \(\vec {y}_i\) for \(i=1,2,...,n\) stores the coordinates of the i-th sample in the manifold and acknowledging that \(\vec {w}_i(j) = 0\), unless \(\vec {y}_j\) is one of the neighbors of \(\vec {y}_i\), we may express \(\varphi (y)\) as follows: $$\begin{aligned} \varphi (y)&= tr(y^t y) tr(y^t w y) \nonumber \\&\qquad tr(y^t w^t y) + tr(y^t w^t w y) \nonumber \\&= tr(y^t y) tr(y^t(wy)) \nonumber \\&\qquad tr((wy)^t y) + tr((wy)^t(wy)) \nonumber \\&= tr(y^t(y wy) (wy)^t(y wy)) \nonumber \\&= tr((y wy)^t (y wy)) \nonumber \\&= tr(((i-w)y)^t ((i-w)y)) \nonumber \\&= tr(y^t (i w)^t (i w) y) \end{aligned}$$ (24) defining the \(n \times n\) matrix m as: $$\begin{aligned} m = (i w)^t (i w) \end{aligned}$$ (25) we get the following optimization problem: $$\begin{aligned} \mathop {\mathrm {arg\,min}}\limits _{y}~ tr(y^t m y) \quad \text {subject to} \quad \frac{1}{n}y^t y = i \end{aligned}$$ (26) thus, the lagrangian function is given by: $$\begin{aligned} l(y, \lambda ) = tr(y^t m y) \lambda \left( \frac{1}{n}y^t y i \right) \end{aligned}$$ (27) differentiating and setting the result to zero, finally leads to: $$\begin{aligned} my = \beta y \end{aligned}$$ (28) where \(\beta = \frac{\lambda }{n}\), showing that the y must be composed by the eigenvectors of the matrix m. since there is a minimization problem to be solved, it is then required to select y to compose the d eigenvectors associated to the d smallest eigenvalues. note that m being an \(n \times n\) matrix, it contains n eigenvalues and n orthogonal eigenvectors. although the eigenvalues are real and non-negative values, its smallest value is always zero, with the constant eigenvector \(\vec {1}\). such a bottom eigenvector corresponds to the mean of y and should be discarded to impose the constraint that \(\sum _{i=1}^{n} \vec {y}_i = 0\) [7]. therefore, to get \(\vec {y}_i \in r^d\), where \(d < m\), it is necessary to select the \(d+1\) smallest eigenvectors and discard the constant eigenvector with zero eigenvalue. in other words, one must select the d eigenvectors associated to the bottom non-zero eigenvalues. 3 the kl divergence-based lle method the main motivation to introduce the lle-kl method is to find a surrogate for the local matrix \(c_i\) for each sample of the dataset. recall that, originally, \(c_i(j, k)\) is computed as the inner product between \(\vec {x}_i \vec {x}_j\) and \(\vec {x}_i \vec {x}_k\), which means that we employ the euclidean geometry in the estimation of the optimal reconstruction weights. in the definition of such matrix it is used a non-linear distance function, namely the relative entropy between gaussian densities estimated within different patches of the knn graph. our inspiration is the parametric pca, an information-theoretic extension of the pca method that applies the kl divergence to compute a surrogate for the covariance matrix (i.e., entropic covariance matrix) [5]. let \(x = \{ \vec {x}_1 , \vec {x}_2 , \ldots , \vec {x}_n \}\), with \(\vec {x}_i \in r^m\), be our data matrix. the first step in the proposed method consists of building the knn graph from x. at this early stage, we employ the extrinsic euclidean distance to compute the nearest neighbors of each sample \(\vec {x}_i\). denoting by \(\eta _i\) the neighborhood system of \(\vec {x}_i\), a patch \(p_i\) is defined as the set \(\{ \vec {x}_i \cup \eta _i \}\). it is worth noticing that the number of elements of \(p_i\) is \(k+1\), for \(i = 1, 2, ..., n\). in other words, a patch \(p_i\) is given by an \(m \times (k+1)\) matrix: $$\begin{aligned} p_i = \begin{bmatrix} x_i(1) &{} x_{i1}(1) &{} \ldots &{} x_{ik}(1) \\ x_i(2) &{} x_{i1}(2) &{} \ldots &{} x_{ik}(2) \\ \vdots &{} \vdots &{} \ddots &{} \vdots \\ \vdots &{} \vdots &{} \ldots &{} \vdots \\ x_i(m) &{} x_{i1}(m) &{} \ldots &{} x_{ik}(m) \end{bmatrix} \end{aligned}$$ (29) the rationale of the proposed method is considering each row of the matrix \(p_i\) as a sample of a univariate gaussian random variable, and then estimating the parameters \(\mu \) (mean) and \(\sigma ^2\) (variance) of each row, leading to an m-dimensional vector of tuples, as follows: $$\begin{aligned} \vec {p}_i = \left[ (\mu _i(1), \sigma _i^2(1)), ..., (\mu _i(m), \sigma _i^2(m)) \right] \end{aligned}$$ (30) let p(x) and q(x) be univariate gaussian densities, \(n(\mu _1, \sigma _1^2)\) and \(n(\mu _2, \sigma _2^2)\), respectively. then, the relative entropy becomes: $$\begin{aligned} d_{kl}(p, q) = log \left( \frac{\sigma _2}{\sigma _1} \right)&+ \frac{1}{2\sigma _2^2} e_p[(x \mu _2)^2] \frac{1}{2\sigma _1^2} e_p[(x \mu _1)^2] \end{aligned}$$ (31) by the definition of central moments, we then have: $$\begin{aligned} e_p[(x \mu _1)^2]&= \sigma _1^2 \end{aligned}$$ (32) $$\begin{aligned} e_p[(x \mu _2)^2]&= e[x^2] 2e[x]\mu _2 + \mu _2^2 \end{aligned}$$ (33) $$\begin{aligned} e[x^2]&= var[x] + e^2[x] = \sigma _1^2 + \mu _1^2 \end{aligned}$$ (34) which leads to the following closed-form equation: $$\begin{aligned} d_{kl}(p, q)&= log \left( \frac{\sigma _2}{\sigma _1} \right) + \frac{\sigma _1^2 + (\mu _1 \mu _2)^2}{2\sigma _2^2} \frac{1}{2} \end{aligned}$$ (35) as the relative entropy is not symmetric, it is possible to compute its symmetrized counterpart as follows: $$\begin{aligned}&d_{kl}^{sym}(p, q) = \frac{1}{2} [d_{kl}(p, q) + d_{kl}(q, p)] \\&= \frac{1}{4} \left[ \frac{\sigma _1^2 + (\mu _1 \mu _2)^2}{\sigma _2^2} + \frac{\sigma _2^2 + (\mu _1 \mu _2)^2}{\sigma _1^2} 2 \right] \nonumber \\&= \frac{1}{4\sigma _1^2\sigma _2^2} \left[ \left( \sigma _1^2 \sigma _2^2 \right) ^2 + \left( \mu _1 \mu _2 \right) ^2 \left( \sigma _1^2 + \sigma _2^2 \right) \right] \nonumber \end{aligned}$$ (36) let \(\vec {d}_{ij}\) be the m-dimensional vector of symmetrized relative entropies between the components of \(\vec {p}_i\) and \(\vec {p}_j\), computed by the direct application of equation (36) in each pair of parameter vectors. then, in the proposed method, the entropic matrix \(c_i\) is computed as follows: $$\begin{aligned} c_i = \vec {d}_{ij}^t \vec {d}_{ik} \end{aligned}$$ (37) where the column vector \(\vec {d}_{ij}\) involves the patches \(\vec {p}_i\) and \(\vec {p}_j\), while the row vector \(\vec {d}_{ik}^t\) involves the patches \(\vec {p}_i\) and \(\vec {p}_k\). it is worth noticing that unlike the established lle method which employs the pairwise euclidean distance, the proposed lle-kl method employs a patch-based distance (relative entropy) instead, becoming less sensitive to the presence of noise and outliers in the observed data. the remaining steps of the algorithm are precisely the same as in the established lle. the set of logical steps comprising the lle-kl method is detailed in algorithm 1. in the subsequent sections, we present some computational experiments to compare the performance of the proposed method against several manifold learning algorithms. 4 data, experiments and results in the present section is presented the data used, experiments performed, and a discussion on the respective empirical results. 4.1 data in the present study, a total of 25 datasets are used as input data. such datasets are widely used in machine learning estimations and applications. the datasets used in the present study were collected from openml.org, which contains detailed information regarding the number of instances, features, and classes for each one of datasets. 4.2 experiments and results in order to test and evaluate the proposed method, a series of computational experiments is performed to quantitatively compare clustering results obtained subsequently to the dimensionality reduction into 2-d spaces using the silhouette coefficient (sc) i.e., a measure of goodness-of-fit into a low-dimensional representation [8]. all results are reported in table 1. it is worth noticing that in 23 out of 25 datasets (i.e., 92% of the cases), the lle-kl method obtains the best performance in terms of the sc. regarding the means and medians, one may realize that the proposed method performs better in comparison with the established lle as well as two of its variations (i.e., hessian lle and ltsa). in addition, the proposed method also achieves a superior performance compared to the state-of-the-art algorithm umap. it is worth noticing that, commonly, the umap requires a reasonably large sample size to perform well, which is not always the case in the empirical analysis of the present paper. table 1. silhouette coefficients (sc) for clusters produced by pca, lle, isomap, hessian lle, ltsa, umap, and lle-kl, based on popular machine learning datasets (2-d case).full size table to check if the results obtained by the proposed method are statistically superior to the competing methods, we perform a friedman test i.e. non-parametric test for paired data in case of more than two groups [2]. for a significant level \(\alpha = 0.01\), we conclude that there is strong evidence to reject the null hypothesis that all groups are identical (\(p = 1.12 \times 10^{-11}\)). in order to analyze which groups are significantly different, we then perform a nemenyi post-hoc test [3]. according to this test, there is strong evidence that the lle-kl method produces significantly higher scs compared to the pca (\(p < 10^{-3}\)), isomap (\(p < 10^{-3}\)), lle (\(p < 10^{-3}\)), hessian lle (\(p < 10^{-3}\)), ltsa (\(p < 10^{-3}\)), and umap (\(p < 10^{-3}\)). the proposed method also has some limitations. one caveat of the lle-kl which is common to other manifold learning algorithms, consists of the out-of-sample problem. it is not clear how to properly evaluate new samples that are not part of the training set. another drawback refers to the definition of the parameter k (i.e., number of neighbors) that controls the patch size. previous computational experiments reveal that the sc is rather sensitive to changes in such a parameter. our strategy in this case is then as follows: for each dataset, we build the kkn graphs for all values of k in the interval [2, 40]. we select the best model as the one that maximizes the sc among all values of k. although we are using the class labels to perform model selection, in fact, the lle-kl method performs unsupervised metric learning, in the sense that we do not employ the class labels in the dimensionality reduction. a data visualization comparison of the clusters obtained through the application of the lle and lle-kl methods for two distinct datasets (i.e., tic-tac-toe and corral) are depicted in figs. 1 and 2. it is worth noticing that the discrimination between the two classes is more evident in the proposed method than in the established version of the lle, since there is visibly less overlap between the blue and red cluster. fig. 1. comparison between clusters generated subsequently to the application of the lle, lle-kl (with the number of neighbors set as k = 9), and umap for the tic-tac-toe dataset. full size image fig. 2. comparison between clusters generated subsequently to the application of the lle, lle-kl (with the number of neighbors set as k = 14), and umap for the corral dataset. full size image 5 conclusion in the present study, an information-theoretic lle is introduced to incorporate the kl divergence between local gaussian distributions into the knn adjacency graph. the rationale of the proposed method is to replace the pointwise euclidean distance by a more appropriate patch-based distance. such a methodological modification results in a more robust method against the presence of noise or outliers in the data. our claim is that the proposed lle-kl is a promising alternative to the existing manifold learning algorithms due to the proposition of its theoretical methodological improvements and considering our computational experiments that support two main points. firstly, the quality of the clusters produced by the lle-kl method indicates a superior performance than those obtained by competing state-of-the-art manifold learning algorithms. secondly, the lle-kl non-linear features may be more discriminative in supervised classification than features obtained by state-of-the-art manifold learning algorithms. suggestions of future work include the use of other information-theoretic distances, such as the bhattacharyya and hellinger distances, as well as geodesic distances based on the fisher information matrix. another possibility is the non-parametric estimation of the local densities using kernel density estimation techniques (kde). in this case, non-parametric versions of the information-theoretic distances may be employed to compute a distance function between the patches of the knn graph. the \(\epsilon \)-neighborhood rule may also be used for building the adjacency relations that define the discrete approximation for the manifold, leading to non-regular graphs. furthermore, a supervised dimensionality reduction based metric learning approach may be created by removing the edges of the knn graph for which the endpoints belong to different classes as a manner to impose that the optimal reconstruction weights use only the neighbors that belong to the same class within the sample. references donoho, d.l., grimes, c.: hessian eigenmaps: locally linear embedding techniques for high-dimensional data. proc. natl. acad. sci. 100(10), 5591–5596 (2003) article  mathscinet  google scholar  friedman, m.: the use of ranks to avoid the assumption of normality implicit in the analysis of variance. j. am. stat. assoc. 32(200), 675–701 (1937) article  google scholar  hollander, m., wolfe, d.a., chicken, e.: nonparametric statistical methods, 3 edn. wiley, new york (2015) google scholar  kullback, s., leibler, r.a.: on information and sufficiency. ann. math. stat. 22(1), 79–86 (1951) article  mathscinet  google scholar  levada, a.l.m.: parametric pca for unsupervised metric learning. pattern recognit. lett. 135, 425–430 (2020) article  google scholar  mcinnes, l., healy, j., melville, j.: umap: uniform manifold approximation and projection for dimension reduction. arxiv 1802.03426 (2020) google scholar  de ridder, d., duin, r.p.: locally linear embedding for classification. technical report, delft university of technology (2002) google scholar  rousseeuw, p.j.: silhouettes: a graphical aid to the interpretation and validation of cluster analysis. j. comput. appl. math. 20, 53–65 (1987) article  google scholar  roweis, s., saul, l.: nonlinear dimensionality reduction by locally linear embedding. science 290, 2323–2326 (2000) article  google scholar  saul, l., roweis, s.: think globally, fit locally: unsupervised learning of low dimensional manifolds. j. mach. learn. res. 4, 119–155 (2003) mathscinet  math  google scholar  talmon, r., cohen, i., gannot, s., coifman, r.r.: diffusion maps for signal processing: a deeper look at manifold-learning techniques based on kernels and graphs. ieee signal process. mag. 30(4), 75–86 (2013) article  google scholar  tenenbaum, j.b., de silva, v., langford, j.c.: a global geometric framework for nonlinear dimensionality reduction. science 290, 2319–2323 (2000) article  google scholar  wang, j.: hessian locally linear embedding. in: wang, j. (ed.) geometric structure of high-dimensional data and dimensionality reduction, pp. 249–265. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-27497-8_13 wang, j., wong, r.k., lee, t.c.: locally linear embedding with additive noise. pattern recognit. lett. 123, 47–52 (2019) article  google scholar  xing, x., du, s., wang, k.: robust hessian locally linear embedding techniques for high-dimensional data. algorithms 9, 36 (2016) article  mathscinet  google scholar  zhang, z.y., zha, h.y.: principal manifolds and nonlinear dimensionality reduction via tangent space aligment. siam j. sci. comput. 26(1), 313–338 (2004) article  mathscinet  google scholar  zhang, z., wang, j.: mlle: modified locally linear embedding using multiple weights. in: schölkopf, b., platt, j.c., hofmann, t. (eds.) advances in neural information processing systems 19, proceedings of the 20th conference on neural information processing systems, vancouver, british columbia, pp. 1593–1600 (2006) google scholar  download references acknowledgments this study was partially financed the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations computing department, federal university of são carlos, são carlos, sp, brazil alexandre l. m. levada department of land economy, university of cambridge, cambridge, uk michel f. c. haddad school of business and management, queen mary university of london, london, uk michel f. c. haddad authors alexandre l. m. levadaview author publications search author on:pubmed google scholar michel f. c. haddadview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper levada, a.l.m., haddad, m.f.c. (2021). a kullback-leibler divergence-based locally linear embedding method: a novel parametric approach for cluster analysis. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_27 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords locally linear embedding metric learning kl divergence publish with us policies and ethics profiles michel f. c. haddad view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature classifying potentially unbounded hierarchical data streams with incremental gaussian naive bayes | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper classifying potentially unbounded hierarchical data streams with incremental gaussian naive bayes conference paper first online: 28 november 2021 pp 421–436 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) classifying potentially unbounded hierarchical data streams with incremental gaussian naive bayes download book pdf download book epub eduardo tieppo  orcid: orcid.org/0000-0001-8271-021410,11, jean paul barddal  orcid: orcid.org/0000-0001-9928-854x10 & júlio cesar nievola  orcid: orcid.org/0000-0002-2212-449910  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 741 accesses 6 citations abstract hierarchical data stream classification inherits the properties and constraints of hierarchical classification and data stream classification concomitantly. therefore, it requires novel approaches that (i) can handle class hierarchies, (ii) can be updated over time, and (iii) are computationally light-weighted regarding processing time and memory usage. in this study, we propose the gaussian naive bayes for hierarchical data streams (gnb-hds) method: an incremental gaussian naive bayes for classifying potentially unbounded hierarchical data streams. gnb-hds uses statistical summaries of the data stream instead of storing actual instances. these statistical summaries allow more efficient data storage, keep constant computational time and memory, and calculate the probability of an instance belonging to a specific class via the bayes’ theorem. we compare our method against a technique that stores raw instances, and results show that our method obtains equivalent prediction rates while being significantly faster. supported by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others computing hierarchical summary of the data streams chapter © 2016 inference for high-dimensional streamed longitudinal data article 15 february 2025 concept drift detection using online histogram-based bayesian classifiers chapter © 2016 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. big data bayesian network categorization data mining machine learning probabilistic data networks 1 introduction hierarchical classification is required on problems where instances are labeled with classes that are related to one another in a hierarchy, such as in recognition of music genres and subgenres [12], computer-aided diagnosis where diseases are categorized by their etiology [43], recognition of animals, which are organized in a taxonomy [28, 42], and, recently, even helping in covid-19 identification using the hierarchical etiology of pneumonia [29]. however, classification techniques often assume that data samples of a particular problem are static and fully available to a learning model in a well-defined training step [26]. this assumption does not reflect many of the real-world scenarios in which classification is applied. the ever-increasing volume of data from diverse sources such as the internet, wireless sensors, mobile devices, or social networks produces massive large-scale data streams [23, 27, 32]. data streams are potentially unbounded over time and hence cannot be stored in memory. also, as the time component is intrinsic in data streams, these are expected to be transient, i.e., the underlying data distribution is ever-changing, thus resulting in variations in the target concept, a phenomenon named concept drift [10, 19, 21, 39]. when merged, hierarchical classification and data stream classification areas combine their properties and introduce new challenges in a roughly unexplored area: the hierarchical classification of data streams. consequently, novel algorithms for hierarchical data stream classification must: (i) handle class hierarchies, (ii) be updatable over time, (iii) detect and adapt to changes in data behavior, and (iv) be computationally light-weighted regarding processing time and memory consumption [10, 19, 31]. in this study, we propose the gnb-hds method: an incremental gaussian naive bayes for classifying potentially unbounded hierarchical data streams. gnb-hds uses statistical summaries of the data stream instead of storing raw instances. despite the relevant application of bayesian classifiers in hierarchical and data stream classification tasks separately, they have not been adapted yet to their intersection task. therefore, to the best of our knowledge, this is the first method that combines incremental bayesian learning with hierarchical classification. these statistical summaries allow a more efficient data storage, holding constant computational time and memory usage, and permit the calculation of the probability of a given instance belonging to a specific class via the bayes’ theorem. the novel contributions of this work are as follows: we qualify gaussian naive bayes, a well-known classification technique [11], to work with potentially unbounded hierarchical data streams and in an incremental fashion by using updatable statistical summaries related to a class hierarchy. we propose gnb-hds, a method for the hierarchical classification of data streams using summarization techniques. the model is incremental and handles potentially unbounded data streams with constant memory usage. furthermore, as a byproduct of this research, we make the source code for the proposed method, as well as the datasets used in the experiments, available for reproducibility. the remainder of this paper is organized as follows. section 2 describes the problem of hierarchical classification of data streams and sect. 3 brings forward related works. section 4 describes the proposed incremental gaussian naive bayes for the hierarchical classification of data streams. section 5 comprises the experimental protocol and the discussion of the results obtained. finally, sect. 6 concludes this paper and states envisioned future works. 2 problem statement as mentioned above, in this paper, we are particularly interested in hierarchical data stream classification. this specific task combines characteristics and challenges from two different areas, and thus, it differs from classical classification in two key aspects. first, concerning hierarchical classification, instances of a problem are assigned to a label path that belongs to a hierarchically structured set of classes instead of one single independent label [35]. figure 1 compares a general approach of (a) flat (non-hierarchical) classification, and (b) hierarchical classification in an illustrative problem. in flat classification, the decision must be made while considering all the classes of the problem (all the possible song genres). meanwhile, hierarchical classification concerns an existing class taxonomy, which can be used to make first smaller and generic decisions about the problem (in the example, decide first between rock and r&b genres), and then more specific ones. fig. 1. example of general approaches of (a) flat and (b) hierarchical classification in a hypothetical music genre problem. the class taxonomy can be used to lead smaller specific decisions about the classes by splitting the context complexity. full size image second, concerning data stream classification, there is not the concept of a complete and fully available dataset; instead, instances of a problem are provided to the model sequentially over time [19]. figure 2 compares (a) a traditional classification process and (b) a data stream classification process. in traditional (or batch) classification, the dataset is assumed to be static and completely available to the model at the training step. next, the dataset is divided into training and test subsets; the training data is submitted to the learning model that reviews them as many times as necessary until obtaining a single satisfactory test model. this final model is then applied to the subset of test data and provides predictions and, consequently, accuracy estimates. in contrast, in streaming scenarios, data is made available sequentially over time, and even a single instance can be provided to the model at a time. each instance is tested by the model, resulting in a prediction, and only after that it is incorporated into the model (being used as training data). this process, entitled ‘test-then-train’, is repeated for each instance, or chunk of instances, that is gathered from the stream. any processed instance needs to be eventually discarded to maintain the model stable to process new instances since the data stream is potentially unbounded. fig. 2. example of general approaches of (a) traditional (batch) classification and (b) data stream classification. an input data is obtained from the data stream, tested, incorporated into the model, and discarded; then, the cycle starts again. full size image thus, hierarchical classification of data streams regards learning models that use hierarchical data streams as input to their learning processes, not only as a source of data but effectively processing portions of the data over time, using the premise that there is no complete dataset and effectively using class taxonomy in their decision processes. more formally, we let hds define a hierarchical data stream in the \([(\vec {x}^t, \vec {y}^t)]_{t=0}^{\infty }\) format providing instances \((\vec {x}^t, \vec {y}^t)\) on a specific timestamp t, where \(\vec {x}^t\) represents a d-dimensional features set and its values, and \(\vec {y}^t\) represents the corresponding ground-truth label path (hierarchically structured classes). these hierarchically structured classes compose a regular concept hierarchy arranged on a partially ordered set \((y, \succ )\), where y is a finite set containing all label paths and the relationship \(\succ \) is defined as an asymmetric, anti-reflexive, and transitive subsumption (is-a) relation [35]. finally, the classification of hierarchical data streams can be formally defined as \(f^t:\vec {x}^t \mapsto \vec {y}^t\), where the function \(f^t\) is continuously updated by mapping features \(\vec {x}\) to the corresponding label paths \(\vec {y}^t\) accurately. as the data streams are potentially infinite due to their time component, learning models are restrained by finite computational resources and must work with bounded memory and time, analyzing each instance only once according to their arrival and then discarding it. the processing time of an incoming instance from the data stream must not surpass the ratio in which new instances become available. otherwise, the learning model will need to discard new instances without analyzing them [5, 10]. 3 related work machine learning models based on the bayes’ theorem have been widely used in classification since their outputs are human-readable, they can naturally handle missing values, and are relatively easy to implement [22, 36]. in hierarchical classification, bayesian classifiers were used with different levels of adaptation. the authors in [14] used bayesian probabilities attached to each node in the hierarchy using a local classifier per node approach [35] and a top-down strategy to analyze the binary predictions along with the hierarchical structure. similarly, the authors in [13] used binary classifiers for each class in the hierarchy considering both the parent and child classes of the current class. in the works of [7, 45], the authors also used bayes-based classifiers within a local classifier per node approach but to perform hierarchical multilabel classification. finally, the authors in [36] proposed a naive bayes fitted to the hierarchical classification using a global approach [35]. a bayesian classifier fitted to handle hierarchical classification needs to be adapted, at least, to consider the relationship between the hierarchically structured classes in the calculation of probabilities [36]. in data stream classification, incremental adaptations of bayesian classifiers have been widely studied and are also widely applied in state-of-the-art algorithms. data stream classification can be handled by a naive bayes classifier in a straightforward manner, since the learning model only needs to incrementally store summaries of data that allow the probabilities calculations as new instances are provided from the data stream [25]. the authors in [3] introduced the idea of recalculating probabilities for each instance provided to a model and this idea was later reinforced by the authors in [2, 25]. in the work [33], the authors proposed an incremental bayes tree based on statistical summaries of data which are updated with each incoming instance. the authors in [9] used naive bayes classifiers ensembled with other tree-based classifiers to improve specific leaf node predictions. finally, the authors in [4] also used incremental statistical summaries to restrain a naive bayes classifier and cope with limited computational resources. it is noteworthy, nonetheless, to highlight the work of [28], where the authors proposed an incremental k-nearest neighbors (knn) [1] approach for the hierarchical classification of data streams. this can be considered a seminal work of the area, yet, it does depict drawbacks such as knn relies on distance computations, which are computationally intensive and can put in jeopardy time and memory usage constraints required by streaming scenarios [27, 38]. in this sense, in sect. 5, we compare our proposal (gnb-hds) against the one proposed in [28] and show that gnb-hds uses bayes probabilities to obtain competitive prediction correctness with better computational performance. 4 proposed method our proposal, hereafter referred to as gaussian naive bayes for hierarchical data streams (gnb-hds), is an incremental method for the hierarchical classification of data streams based on the naive bayes technique [11, 18]. the main idea behind gnb-hds is the use of incremental data summaries, specifically the mean, standard deviation, and the number of data instances, that allow the calculation of probabilities used in the bayes’ theorem [11, 22]. these incremental data summaries are attached to nodes of the hierarchy and are updated as new instances are gathered from the data stream. we implemented two key adaptations in the traditional naive bayes classifier to make it handle hierarchical data streams: regarding the hierarchical data structure, the original algorithm was modified to consider not only one class but all related classes of a given instance. as the hierarchical data structure represents a subsumption relation, any new instance provided from the data stream also belongs to its ancestors. thus, we traverse the hierarchy to update all data summaries of parent nodes recursively until the root node of the hierarchy. regarding the streaming input data, the algorithm must store incremental statistical descriptors instead of the actual instances. thus, we need to compute the mean, the standard deviation, and the count of data instances assigned to each class incrementally, discarding the instance after it is analyzed. regarding the stated problem approach, gnb-hds represents the class taxonomy in a tree structure using local classifiers at each parent node and assigns leaf node classes as the last class of one predicted label path \(\vec {y}^t\) (mandatory leaf-node and single path prediction) [35]. we point out that although the gnb-hds method has been implemented here in a more specific way regarding the stated problem, gnb-hds also supports direct acyclic graphs and non-mandatory leaf node prediction in its concept. to that, the data structure of a given node in the hierarchy should allow links with more than one parent node, and the top-down strategy used in the prediction step must consider some stopping criteria (e.g., a probability threshold) resulting in partial depth label paths. figure 3 illustrates the process performed by gnb-hds. circles represent classes, and dashed squares enclose classifiers. the method represents the class taxonomy in a tree structure, where r stands for the root node of the hierarchy and classes are related with each other (as described in sect. 2). when receiving an incoming instance for prediction, the method tackles the hierarchy using a local classifier per parent node (lcpn) approach [35], thus analyzing the current parent node and predicting between its child nodes by using probabilities obtained with the bayes’ theorem. this process is repeated until a leaf node is reached. each node in the tree stores the count of instances (n), a d-dimensional incremental mean (\({\bar{x}}_{n}\)) and a d-dimensional incremental standard deviation (\(\sigma _{n})\) of the class represented (as shown in class 2). after the incoming instance processing, the statistical descriptors (\(n,{\bar{x}}_{n}\sigma _{n}\)) are updated incrementally with the instance feature values on all the classes through the hierarchy regarding the ground-truth label path of that instance. fig. 3. illustration of gnb-hds method. full size image as before introduced, the instances are represented by data summaries comprising three statistical descriptors stored incrementally: (i) the count of class instances, (ii) the d-dimensional mean instance, and (iii) the d-dimensional standard deviation of the instances of a given class. the number of instances assigned to a class c is stored in an attached counter. when an instance is retrieved from the stream, the c-th class counter is incremented alongside the counters of c’s ancestors. the incremental mean (\({\bar{x}}_{n}\)) and the incremental standard deviation (\(\sigma _{n})\) considering each attribute from a \(d-\)dimensional \(x_n\) instance are obtained, respectively, from eqs. 1 and 2, where n stands for the number of instances observed so far assigned to c [15, 40]. also, it is important to reinforce that the incremental mean and the incremental standard deviation are d-dimensional descriptors as the features set and its values from the d-dimensional \(x_n\) instance. note that eqs. 1 and 2 support only continuous feature sets and the current mean and the standard deviation of the previous observed instances assigned to c are represented by \({\bar{x}}_{n-1}\) and \(\sigma _{n-1}\). $$\begin{aligned} {\bar{x}}_{n}=\ \frac{({\bar{x}}_{n-1}(n-1))+x_n)}{n} \end{aligned}$$ (1) $$\begin{aligned} \sigma _{n}=\sqrt{\frac{\left( n-2\right) \sigma _{n-1}^2+\left( n-1\right) \left( {\bar{x}}_{n-1}-{\bar{x}}_n\right) ^2+{(x_n-{\bar{x}}_n)}^2}{n-1}} \end{aligned}$$ (2) the prediction of the class to be assigned to an incoming instance provided from the data stream is performed in three steps: (i) computation of the a priori probabilities based on the count of class instances, (ii) computation of likelihood probabilities based on the bayes’ theorem for each attribute of the incoming instance, and (iii) calculation of the maximum value of the a posteriori probability from the product of the independent feature probabilities given a class c. the calculation of the likelihood probability is described in eq. 3, where i represents a feature index and j a class index [11]. $$\begin{aligned} p\left( x_i{\mid }\ c_j\right) ={\frac{1}{\sqrt{2\pi \sigma _{i,j}^2}}exp}{\left\{ -\frac{1}{2}\left( \frac{x_i-\bar{x}_{i,j}}{\sigma _{i,j}}\right) ^2\right\} } \end{aligned}$$ (3) to perform the class assignment, the gnb-hds obtains the class label with the maximum value of the a posteriori probability, as described in eq. 4, from the product of the independent feature probabilities given c [11]. $$\begin{aligned} p\left( c_j{\mid }x\right) \propto \left\{ \prod _{i}{p\left( x_i{\mid }\ c_j\right) } \right\} p(c_j) \end{aligned}$$ (4) moreover, these three steps are performed from the top of the hierarchy data structure and repeated until a leaf node is reached, resulting in the union of the class assignments made from eq. 4 and representing the final label path assigned to the incoming instance. algorithm 1 shows the proposed gaussian naive bayes for hierarchical data streams (gnb-hds). it receives a hierarchical data stream hds supplying instances \((\vec {x}, \vec {y})\) over time and, if required, outputs a set of predicted labels (a label path) \(\widehat{\vec {y}_i}\) for each given instance \((\vec {x},\vec {y})\), where \(\vec {x}\) represents a d-dimensional features set and its values, and \(\vec {y}\) represents the corresponding ground-truth label path of that instance. the algorithm starts by understanding and representing the class taxonomy from the hierarchical data stream. the first loop (line 2 onwards) receives an incoming instance from the hierarchical data stream. the following loop (lines 4–12) handles the hierarchy using the lcpn approach by predicting one of the children labels possible for that parent node. the a priori probabilities are calculated in line 6 using the counts of class instances. the likelihood and posterior probabilities are calculated in lines 8 and 9 by the application of eqs. 3 and 4, respectively. the predicted node for the evaluated parent is obtained in line 10, and the respective single label is appended to a partial label path \(\widehat{\vec {y}_i}\) (line 11). this process is repeated until a leaf node is reached and the label path \(\widehat{\vec {y}_i}\) is complete and ready to be output by the algorithm. finally, the algorithm updates the statistical descriptors (the count n of class instances, the incremental mean instance \({\bar{x}}_{n}\), and the incremental standard deviation \(\sigma _{n}\)) of all classes contained in \(\vec {y}_i\), from the leaf to the root class. 5 analysis in this section, we report the experimental analysis conducted to compare our proposal against existing works in hierarchical data stream classification. first, we provide the experimental protocol adopted. next, we discuss the results in terms of prediction and performance. table 1. datasets used in the experiment.full size table 5.1 experimental protocol table 1 depicts the 14 hierarchically labeled datasets used in our testbed, listing their number of instances, features, and classes, the number of labels per level in the hierarchy (from top-level to leaf level), and references. these datasets contain different features, instances, and domains, thus assessing how our proposal behaves in different scenarios. during the experiments, classifiers were assessed in terms of hierarchical f-measure [24]. like traditional classification metrics, the hierarchical f-measure (hf) relies on hierarchical precision and recall components, but instances are associated with a path of labels, and the entire path is evaluated. the hierarchical f-measure is depicted in eq. 5, while its precision (hp) and recall (hr) components are described in eqs. 6 and 7, respectively. in both precision and recall metrics, \(\widehat{\vec {y}_i}\) is the set of labels predicted for the i-th instance, and \({\vec {y}_i}\) is its corresponding ground-truth label set. $$\begin{aligned} hf = \frac{2 \times hp \times hr}{hp + hr} \end{aligned}$$ (5) $$\begin{aligned} hp = \frac{\sum _i\left| \widehat{\vec {y}_i}\bigcap {\vec {y}_i}\right| }{\sum _i\left| \widehat{\vec {y}_i}\right| } \end{aligned}$$ (6) $$\begin{aligned} hr = \frac{\sum _i\left| \widehat{\vec {y}_i}\bigcap {\vec {y}_i}\right| }{\sum _i\left| {\vec {y}_i}\right| } \end{aligned}$$ (7) we report the hf metric using the prequential test-then-train [10, 20] validation method, where each instance is used to test the model before it is used for training and updating [10, 21]. furthermore, we measured the time performance by calculating the number of instances that a classifier can process per second. we compared our proposed gnb-hds to the hierarchical knn described in sect. 3 proposed in [28], hereafter referred to as knn-hds. we set up knn-hds with \(k \in \{1,3,5\}\) and \(n \text { (buffer size)} \in \{\)5, 10, 15, 20\(\}\). the method gnb-hds does not require setting parameters. finally, the results obtained by both methods were assessed using wilcoxon hypothesis tests [41] with a 95% confidence level according to the protocol provided in [16] to verify significant differences in the hf and instances processed per second rates obtained by both methods. the experiments in this paper were performed using python 3.7. the proposed script containing the gnb-hds method, as well as the datasets, are freely available for downloadfootnote 1. 5.2 results table 2 shows the hierarchical f-measure (hf) and instances per second rates obtained by knn-hds and gnb-hds in the datasets (greater values are highlighted in bold). in the knn-hds method, rates represent the best hf results obtained in the parameters configuration (as described in sect. 5.1). table 2. hierarchical f-measure (hf) and instances per second rates obtained during experiments.full size table in terms of predictive performance assessment, the gnb-hds method obtained better hf rates in 10 out of the 14 datasets. however, hf values are similar across both methods, such that the average difference between them is 0.44% while favoring gnb-hds. despite the improvements, the wilcoxon test showed no statistical difference between hf rates obtained by the methods (p-\(value = 0.2209\)). concerning processing speed comparison, the gnb-hds method was able to process more instances per second across all datasets, with an average rate of 446.21 instances against 140.43 of the knn-hds method. thus, on average, our method was able to process 3.2 times more instances than the knn-hds method. a one-tailed wilcoxon test indicated a statistical difference between instances per second rates obtained by both methods (p-\(value = 0.0005\)) and confirmed that gnb-hds is significantly faster when compared to knn-hds method. considering predictive performance and processing speed rates, gnb-hds can obtain computational performance improvements without significant threats to the predictive performance by using statistical summaries of data combined with the class hierarchy information. as aforementioned, the gnb-hds method uses the premise of a gaussian (normal) data distribution to deal with instance representation in the learning model [30]. in this sense, in addition to the previously described analysis, we investigated if gnb-hds could use its premise to obtain better hf rates when data is normally distributed. thus, the gnb-hds method, in addition to its speed, would present an additional advantage to the knn-hds method (or to any other method that does not use the premise of data normality) since it would be more adapted to classify normally distributed data. this advantage can be even more noticeable when we consider the data stream context, where data are potentially unbounded and statistical descriptors, such as mean and standard deviation, are more likely to obtain better representations of the population. to examine this claim, we perform a shapiro-wilk test of normality [34] in all the datasets and applied the yeo-johnson power transformation technique [44] in all data to improve their adherence to a more normal distribution. the data normality was measured by shapiro-wilk test before and after the application of the yeo-johnson transformation. table 3 depicts the shapiro-wilk w statistic before (raw data) and after (transformed data) the yeo-johnson transformation. w statistic is bounded by 1, and closer values to this upper bound represent data more fitted to a normal distribution. in addition, table 3 depicts the hierarchical f-measure (hf) obtained by both gnb-hds and knn-hds methods when applied in both raw and transformed datasets. one can note that the predictive performance of gnb-hds was improved when using transformed data in 13 out 14 datasets. likewise, the average hf increased 2.35%, with noticeable increases in some datasets, such as entomology (5.23%) and insects-o-o-c (5.32%). oppositely, knn-hds could not achieve the same improvements. in fact, knn-hds obtained lower hf rates with the transformed data resulting in a decrease of 1.6% in the average hf from 72.16% to 70.56%. table 3. shapiro-wilk w statistic of datasets and hierarchical f-measure (hf) obtained by gnb-hds with raw and transformed data.full size table finally, we performed one-tailed wilcoxon tests to verify if the results obtained with the transformed datasets are significantly higher than with raw data for both gnb-hds and knn-hds methods. on knn-hds, the test indicated a statistical difference between performances with both data (p-\(value = 0.0009\)) favoring raw data, i.e., knn-hds does not benefit from a more normal distributed data. in contrast, on gnb-hds the test indicated a statistical difference between performances with both data (p-\(value = 0.0006\)) favoring the transformed data and has confirmed that gnb-hds can take advantage of a more normal distribution-like data, thus corroborating our claims. 6 conclusion in this paper, we proposed gnb-hds, an algorithm for hierarchical classification of data streams using data summaries to represent data. our proposal is incremental and handles potentially unbounded data streams with constant memory consumption. consequently, the proposed method processes more instances per second without dreadful impacts in prediction rates when compared to existing knn-based techniques. to the best of our knowledge, our method extends the state-of-the-art being the first incremental method based on bayes’ theorem tailored for hierarchical data streams classification. the resulting source code and all the datasets used in the experiments are freely available for download to be used as a baseline to further research on the hierarchical classification of data streams, such as data preprocessing, computational resources analysis, and concept drift detection and adaptation. in future works, we are interested in designing and applying other data summaries and different window types to maintain more than one a priori probabilities per class to allow a posteriori probabilities calculation weighted by data newness. also, we are interested in applying existing drift detectors [6, 8, 17] to increase the responsiveness to changes in the data distribution. notes 1.http://www.ppgia.pucpr.br/~jean.barddal/datasets/gnb-hds.zip. references aha, d.w., kibler, d., albert, m.k.: instance-based learning algorithms. mach. learn. 6(1), 37–66 (1991) google scholar  alcobé, j.: incremental learning of tree augmented naive bayes classifiers. in: garijo, f.j., riquelme, j.c., toro, m. (eds.) iberamia 2002. lncs (lnai), vol. 2527, pp. 32–41. springer, heidelberg (2002). https://doi.org/10.1007/3-540-36131-6_4 chapter  google scholar  anderson, j.r., matessa, m.: explorations of an incremental, bayesian algorithm for categorization. mach. learn. 9(4), 275–308 (1992) google scholar  bahri, m., maniu, s., bifet, a.: a sketch-based naive bayes algorithms for evolving data streams. in: 2018 ieee international conference on big data (big data), pp. 604–613. ieee (2018) google scholar  barddal, j.p., gomes, h.m., enembreck, f., pfahringer, b., bifet, a.: on dynamic feature weighting for feature drifting data streams. in: frasconi, p., landwehr, n., manco, g., vreeken, j. (eds.) ecml pkdd 2016. lncs (lnai), vol. 9852, pp. 129–144. springer, cham (2016). https://doi.org/10.1007/978-3-319-46227-1_9 chapter  google scholar  barros, r.s., cabral, d.r., gonçalves jr., p.m., santos, s.g.: rddm: reactive drift detection method. expert syst. appl. 90, 344–355 (2017) google scholar  bi, w., kwok, j.t.: bayes-optimal hierarchical multilabel classification. ieee trans. knowl. data eng. 27(11), 2907–2918 (2015) article  google scholar  bifet, a., gavalda, r.: learning from time-changing data with adaptive windowing. in: proceedings of the 2007 siam international conference on data mining, pp. 443–448. siam (2007) google scholar  bifet, a., holmes, g., pfahringer, b., kirkby, r., gavalda, r.: new ensemble methods for evolving data streams. in: proceedings of the 15th acm sigkdd international conference on knowledge discovery and data mining, pp. 139–148 (2009) google scholar  bifet, a., kirkby, r.: data stream mining a practical approach (2009) google scholar  bishop, c.m.: pattern recognition and machine learning. springer, heidelberg (2006) google scholar  burred, j.j., lerch, a.: a hierarchical approach to automatic musical genre classification. in: proceedings of the 6th international conference on digital audio effects, pp. 8–11. citeseer (2003) google scholar  de campos merschmann, l.h., freitas, a.a.: an extended local hierarchical classifier for prediction of protein and gene functions. in: bellatreche, l., mohania, m.k. (eds.) dawak 2013. lncs, vol. 8057, pp. 159–171. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-40131-2_14 chapter  google scholar  cesa-bianchi, n., gentile, c., zaniboni, l.: incremental algorithms for hierarchical classification. j. mach. learn. res. 7, 31–54 (2006) mathscinet  math  google scholar  chan, t.f., golub, g.h., leveque, r.j.: algorithms for computing the sample variance: analysis and recommendations. am. stat. 37(3), 242–247 (1983) mathscinet  math  google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) google scholar  frías-blanco, i., del campo-ávila, j., ramos-jimenez, g., morales-bueno, r., ortiz-díaz, a., caballero-mota, y.: online and non-parametric drift detection methods based on hoeffding’s bounds. ieee trans. knowl. data eng. 27(3), 810–823 (2014) article  google scholar  friedman, n., geiger, d., goldszmidt, m.: bayesian network classifiers. mach. learn. 29(2), 131–163 (1997) article  google scholar  gama, j.: knowledge discovery from data streams. chapman and hall/crc (2010) google scholar  gama, j., sebastião, r., rodrigues, p.p.: on evaluating stream learning algorithms. mach. learn. 90(3), 317–346 (2013) article  mathscinet  google scholar  gama, j., žliobaitė, i., bifet, a., pechenizkiy, m., bouchachia, a.: a survey on concept drift adaptation. acm comput. surv. (csur) 46(4), 44 (2014) article  google scholar  han, j., pei, j., kamber, m.: data mining: concepts and techniques. elsevier, amsterdam (2011) google scholar  hesabi, z.r., tari, z., goscinski, a., fahad, a., khalil, i., queiroz, c.: data summarization techniques for big data—a survey. in: khan, s.u., zomaya, a.y. (eds.) handbook on data centers, pp. 1109–1152. springer, new york (2015). https://doi.org/10.1007/978-1-4939-2092-1_38 chapter  google scholar  kiritchenko, s., famili, f.: functional annotation of genes using hierarchical text categorization. in: proceedings of biolink sig, ismb, january 2005 google scholar  klawonn, f., angelov, p.: evolving extended naive bayes classifiers. in: sixth ieee international conference on data mining-workshops (icdmw 2006), pp. 643–647. ieee (2006) google scholar  kotsiantis, s.b., zaharakis, i., pintelas, p.: supervised machine learning: a review of classification techniques. emerg. artif. intell. appl. comput. eng. 160, 3–24 (2007) google scholar  nguyen, h.l., woon, y.k., ng, w.k.: a survey on data stream clustering and classification. knowl. inf. syst. 45(3), 535–569 (2015) article  google scholar  parmezan, a.r.s., souza, v.m.a., batista, g.e.a.p.a.: towards hierarchical classification of data streams. in: vera-rodriguez, r., fierrez, j., morales, a. (eds.) ciarp 2018. lncs, vol. 11401, pp. 314–322. springer, cham (2019). https://doi.org/10.1007/978-3-030-13469-3_37 chapter  google scholar  pereira, r.m., bertolini, d., teixeira, l.o., silla jr., c.n., costa, y.m.: covid-19 identification in chest x-ray images on flat and hierarchical classification scenarios. comput. methods programs biomed. 194, 105532 (2020) google scholar  pontes, e.a.s.: a brief historical overview of the gaussian curve: from abraham de moivre to johann carl friedrich gauss. int. j. eng. sci. invent. (ijesi), 28–34 (2018) google scholar  prasad, b.r., agarwal, s.: stream data mining: platforms, algorithms, performance evaluators and research trends. int. j. database theory appl. 9(9), 201–218 (2016) article  google scholar  quiñonero-candela, j., sugiyama, m., schwaighofer, a., lawrence, n.d.: dataset shift in machine learning. the mit press, cambridge (2009) google scholar  seidl, t., assent, i., kranen, p., krieger, r., herrmann, j.: indexing density models for incremental learning and anytime classification on data streams. in: proceedings of the 12th international conference on extending database technology: advances in database technology, pp. 311–322 (2009) google scholar  shapiro, s.s., wilk, m.b.: an analysis of variance test for normality (complete samples). biometrika 52(3/4), 591–611 (1965) article  mathscinet  google scholar  silla, c.n., freitas, a.a.: a survey of hierarchical classification across different application domains. data min. knowl. discov. 22(1–2), 31–72 (2011) article  mathscinet  google scholar  silla jr., c.n., freitas, a.a.: a global-model naive bayes approach to the hierarchical prediction of protein functions. in: 2009 ninth ieee international conference on data mining, pp. 992–997. ieee (2009) google scholar  souza, v.m.a., reis, d.m., maletzke, a.g., batista, g.e.a.p.a.: challenges in benchmarking stream learning algorithms with real-world data. data min. knowl. discov., 1–54 (2020). https://doi.org/10.1007/s10618-020-00698-5 steinbach, m., ertöz, l., kumar, v.: the challenges of clustering high dimensional data. in: wille, l.t. (ed.) new directions in statistical physics, pp. 273–309. springer, heidelberg (2004). https://doi.org/10.1007/978-3-662-08968-2_16 tsymbal, a.: the problem of concept drift: definitions and related work. comput. sci. dep. trinity coll. dublin 106(2), 58 (2004) google scholar  west, d.: updating mean and variance estimates: an improved method. commun. acm 22(9), 532–535 (1979) article  google scholar  wilcoxon, f.: individual comparisons by ranking methods. in: kotz, s., johnson, n.l. (eds.) breakthroughs in statistics. springer series in statistics (perspectives in statistics), pp. 196–202. springer, new york (1992). https://doi.org/10.1007/978-1-4612-4380-9_16 wu, f., zhang, j., honavar, v.: learning classifiers using hierarchically structured class taxonomies. in: zucker, j.-d., saitta, l. (eds.) sara 2005. lncs (lnai), vol. 3607, pp. 313–320. springer, heidelberg (2005). https://doi.org/10.1007/11527862_24 chapter  google scholar  yassin, n.i., omran, s., el houby, e.m., allam, h.: machine learning techniques for breast cancer computer aided diagnosis using different image modalities: a systematic review. comput. methods progr. biomed. 156, 25–45 (2018) article  google scholar  yeo, i.k., johnson, r.a.: a new family of power transformations to improve normality or symmetry. biometrika 87(4), 954–959 (2000) article  mathscinet  google scholar  zaragoza, j.c., sucar, e., morales, e., bielza, c., larranaga, p.: bayesian chain classifiers for multidimensional classification. in: twenty-second international joint conference on artificial intelligence. citeseer (2011) google scholar  download references author information authors and affiliations pós-graduação em informática (ppgia), pontifícia universidade católica do paraná (pucpr), curitiba, brazil eduardo tieppo, jean paul barddal & júlio cesar nievola instituto federal do paraná (ifpr), pinhais, brazil eduardo tieppo authors eduardo tieppoview author publications search author on:pubmed google scholar jean paul barddalview author publications search author on:pubmed google scholar júlio cesar nievolaview author publications search author on:pubmed google scholar corresponding author correspondence to eduardo tieppo . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper tieppo, e., barddal, j.p., nievola, j.c. (2021). classifying potentially unbounded hierarchical data streams with incremental gaussian naive bayes. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_28 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hierarchical classification data stream classification gaussian naive bayes incremental learning publish with us policies and ethics profiles jean paul barddal view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature exploring climatic shifts in brazilian climates: insights from armax, decision trees, and artificial neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper exploring climatic shifts in brazilian climates: insights from armax, decision trees, and artificial neural networks conference paper first online: 31 january 2025 pp 167–179 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) exploring climatic shifts in brazilian climates: insights from armax, decision trees, and artificial neural networks download book pdf download book epub joão v. r. de andrade  orcid: orcid.org/0000-0003-3554-01579, igor l. b. da silva  orcid: orcid.org/0009-0008-4279-23139, teobaldo g. de souza junior  orcid: orcid.org/0000-0002-0961-659510, leandro h. de s. silva  orcid: orcid.org/0000-0001-6221-225010, agostinho freire  orcid: orcid.org/0000-0002-6059-90149, daisy lucena11 & … bruno j. t. fernandes  orcid: orcid.org/0000-0002-6001-39259  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 217 accesses 1 citation abstract over the years, humanity has deepened its understanding of its actions’ profound impacts on global temperatures. beyond the rise in global temperatures driven by global warming, local biomes also significantly influence regional temperature variations, often due to changes in land use. historically, climatologists have attributed local temperature increases to deforestation, urban development, and agricultural activities. although these factors are well-known, there remains a need for methodologies to objectively quantify the impact of land use changes on temperature increases. this paper proposes an explanation for temperature increase utilizing advanced statistical models such as armax and neural networks with explainable ai components aiming to quantify the effects of these land use changes on local temperature trends over several years. for this purpose, two cities were chosen to apply the methodology. são félix do xingu, in the brazilian amazon rainforest (a city that has experienced significant deforestation due to agriculture over the last decade), and cajazeiras, in the brazilian semi-arid region (a city that has had substantial urban development over the years). the data for this analysis were sourced from the mapbiomas brazilian database and temperature records collected by the brazilian national institute of meteorology. our approach offers insights into human activities’ climatic impact, giving weight to soil changes and providing a methodology for further interdisciplinary studies, potentially serving as a starting point for specialist systems across various domains. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others exponential smoothing and neural networks for climate forecasting in brazil: insights and change-point prediction chapter © 2025 spatial heterogeneity in climate change effects across brazilian biomes article open access 16 july 2024 exploring climate extremes in brazil’s legal amazon article 21 december 2023 1 introduction the sixth assessment report of the intergovernmental panel on climate change (ipcc ar6) [8] acknowledges that climate change is real, based on strong evidence such as declines in arctic sea ice and other glaciers, thawing of permafrost, an increase in the frequency and severity of heat extremes accompanied by a decrease in cold extremes, a reduction in available water in dry seasons, and an increase in the occurrence of weather conditions that promote torrential rains or wildfires. thus, global warming is one of the most significant concerns regarding climate change induced by modern society. due to this phenomenon, practically all beings in the world are negatively affected, including people [1]. in this sense, although greenhouse gas (ghg) emissions are the most debated in the scientific community, land use and land cover change are also essential factors to be considered [9]. humans have deforested vast areas to carry out their activities and utilize natural resources: extraction of natural resources, the establishment of farms, or expansion of urban sites [14]. consequently, the most threatened areas have become ecological hotspots [11]. in the case of brazil, and considering myers’ classic definition [11], the amazon region does not fit the hotspot criteria. nevertheless, its rapid deforestation has raised significant concerns, as the collapse of this vegetation could have deleterious effects with global repercussions [6]. hence, it is important to conduct studies in this area to demonstrate how anthropogenic alterations, encompassing changes in land, may be responsible for temperature fluctuations, aiming to scientifically support their preservation, as current policies on the subject seem not to have produced the desired effects [15]. therefore, this study covers the period from 1990 to 2022 and utilized data on land occupation from the mapbiomas project [16] along with monthly temperature records from the brazilian national institute of meteorology (inmet) in two different locations: cajazeiras-pb and são félix do xingu-pa. the first city is brazil’s northeast region, with a semi-arid climate, while the second one is part of the amazon rainforest biome. for both regions, the local station presented few data gaps and a significant contrast in land use and land cover change during the study period, with an increase in urban areas but primarily a large expansion of areas dedicated to agriculture and livestock, accompanied by a reduction in native forest. in this vein, the main objective of the research is to gain insights from machine learning models on the effect of land use changes in local temperature. since the exogenous variables are land occupation and the endogenous variable is temperature, an armax model was used to perform regression on the temperature time series. then, random forests and multilayer perceptron networks (mpl) were trained with armax outputs. finally, by confronting the armax model coefficients and xai techniques for the machine learning models, we provide machine learning-powered quantification of the effects of urban, farming, and forest biome transformations on local temperature. the findings indicate a significant influence of urban and farming areas on increasing temperatures. however, each model provides a different perspective on the magnitude of the influence of each exogenous variable. this work proposes a methodology to address the impacts of soil changes on local temperature. the remaining paper is organized as follows: sect. 2 outlines related works on time series forecasting, focusing on temperature forecasting. section 3 describes the data preparation steps. section 4 presents the explainability methods applied in this work. section 5 discusses the results achieved in the experiments. section 6 provides the conclusions of the paper and suggests directions for further research. 2 releated works numerous studies have leveraged machine learning methods to enhance temperature prediction and downscaling across different geographical areas [2]. cifuentes et al. [4] examined various machine learning approaches for temperature forecasting, emphasizing their critical role in planning and decision-making processes across governmental, industrial, and public sectors. this review noted that deep learning models generally exhibited lower error rates than conventional artificial neural network structures, particularly at regional scales. at the same time, support vector machines were favored worldwide for their simplicity and precision. hernanz et al. [7] expressed concerns about the extrapolation abilities of machine learning techniques when used for statistical downscaling of climate change projections. through controlled experiments, they illustrated that widely used machine learning models, including artificial neural networks and support vector machines, often show limited performance under extrapolation conditions. this study stressed the importance of understanding the limitations inherent in applying machine learning to downscaling tasks. xinyi li et al. [10] explored the efficacy of both machine learning (long short-term memory networks and support vector machines) and statistical (arithmetic ensemble mean and multiple linear regression) methods in downscaling daily temperatures in ontario, canada. they reported that machine learning and statistical methods effectively produced high-resolution daily temperature forecasts using multi-model ensembles, though challenges persisted in accurately predicting extreme temperature events. while numerous studies highlight the utility of machine learning in temperature prediction and downscaling, there remains a significant gap in the literature, specifically concerning the impacts of urban expansion and deforestation on temperature increases. recent works, such as those by [3, 5], and [13], focus on these areas, elucidating the relationship between local air temperatures and land use. understanding and mitigating these impacts is crucial for developing strategies to mitigate the adverse effects of urbanization and land use changes on temperature trends. furthermore, research by nascimento et al. [12] directly links temperature variations to land surface changes. this study utilized a network of thermometers spread throughout são paulo-sp. it compared their readings with data from the landsat 8 satellite’s tirs sensor, affirming a correlation between air and surface temperatures. notably, land cover, especially vegetated areas, significantly influences temperature, typically resulting in cooler conditions. building upon these insights, our study contributes to this field by proposing a methodology emphasizing soil data’s importance in temperature forecasting systems. our approach demonstrates how changes in soil can be discerned through temperature variations over the years, providing a crucial understanding of how human land use impacts local temperatures. this methodology enhances our ability to predict temperature changes and deepens our understanding of the interplay between human activity and environmental conditions, paving the way for more informed urban and environmental planning. 3 data processing 3.1 temperature database: cajazeiras-pb and são félix do xingu-pa temperature data this article examines the average maximum temperatures for each month in the cities of cajazeiras and são félix do xingu. the temperature data from cajazeiras were collected from a weather station operated by the national institute of meteorology (inmet). this station (id 82689) is situated in the são gonçalo district, near cajazeiras, within the municipality of sousa-pb. the data from são félix do xingu were obtained from the inmet station with code 82688, located in the são félix do xingu municipality, pará state. to mitigate the impact of seasonal patterns, both time series should undergo a 12-month differencing. however, to retain the long-term trend within the series, the differencing is conducted by comparing each month to the corresponding month in 1990. consequently, the output yields two series of delta values relative to 1990, facilitating a more precise analysis of temperature changes over time while mitigating the effects of seasonal variability. statistical tests such as pettitt’s test were employed to analyze the behavior of the series. pettitt’s test identified a significant change point in both series (cp) using a significance level of 0.05, indicating a lack of homogeneity. figure 1 illustrates each differenced series with pettitt’s u plotted over the series. fig. 1. delta series with pettit’s change point. full size image soil database: cajazeiras-pb and sã o félix do xingu-pa soil ocuppation. for the analysis of land use and cover, data were sourced from a brazilian government platform called mapbiomas. this platform classifies each 30 \(\times \) 30 m plot of soil across brazil using satellite imagery. in our study, the soil data for cajazeiras and são félix do xingu were collected and treated as exogenous variables. the data from mapbiomas are collected annually; however, a specific transformation was required to integrate them with the monthly frequency of the temperature records. this transformation involved adjusting the temporal granularity of the mapbiomas data to align with the monthly frequency of the temperature records. figures 2 and 3 depict the evolution of soil for cajazeiras and são félix do xingu, respectively. fig. 2. soil evolution in cajazeiras-paraíba from 1990 to 2020. full size image fig. 3. soil evolution in são félix do xingu-pará from 1990 to 2022. full size image another essential aspect of our methodology is the consideration of the relative frequency of each biome near each city. our soil processing approach involves grouping pixels corresponding to each biome to form relative frequency data for each biome within the area of the cities. subsequently, we employ linear interpolation to interpolate the annual data from mapbiomas, assigning data values for each month. another important aspect of our data processing involves the classification of soil. while mapbiomas originally categorized over 30 different soil classes, its methodology allows these classes to be consolidated into five super-classes. these super-classes are utilized as exogenous variables in the experiments detailed in this paper. the classes were grouped into super-classes as defined in table 1. table 1. map biomes classesfull size table 4 fiting model and explainability approach following the methodology described by andrade et al. [5], we employed an arimax model, initiating training fifty steps prior to the change point (cp) value to enhance our analysis of predictions near the change point identified by pettit’s test. this approach was chosen to explore which exogenous variables exert the most influence near the change point. to determine the optimal parameters for the arimax model, we conducted a grid search across the autoregressive term (p), integrative term (d), and moving average term (q). the model with the lowest akaike information criterion (aic) score was selected, ensuring the best fit for model simplicity and predictive accuracy. the specific cp values and the chosen p, d, and q parameters for each series are detailed in table 2. table 2. arimax training parametersfull size table the exogenous variables were modeled using soil data, where the relative frequency of each biome within the city’s total area was pre-processed. then, each biome was grouped into its superclass. for example, biome 33, categorized as mangrove, and biome 1, categorized as tropical forest, were labeled under the superclass “forest,” as described in table 1. table 3 lists the remaining biomes and their respective variation (2022 value divided by 1990 value) for each series. table 3. biomes variation from 1990 to 2022full size table although both time series are categorized under the “forest” biome, the specific forest types for each series differ significantly. são félix do xingu is part of the amazon rainforest biome, characterized by dense, moisture-rich tropical forests. in contrast, the forest biome of cajazeiras is composed of caatinga, a biome with characteristics well-suited to drier locations. for example, the outskirts of cajazeiras, where this biome is prevalent, receive less than 1000 mm of precipitation annually, while são félix do xingu receives more than 2000 mm. these characteristics are crucial because they influence the biome’s ability to affect local temperatures. since the primary vegetation in cajazeiras is caatinga, it is expected to contribute less to urban cooling than the lush, dense vegetation found in the amazon rainforest of são félix do xingu. as noted in table 3, the most significant soil variations in both cities were observed in the forest, farming, and urban biomes. consequently, our approach will focus exclusively on these biomes for subsequent analyses, discarding the non-forest natural formation class due to its minimal impact on overall soil variation. following this reduction of exogenous variables, the statistical model for each series was trained using a rolling armax approach with zero integration (\(i=0\)). at each prediction point, coefficients of the exogenous variables were recorded for subsequent analysis to determine their influence on the response variable over time. a random forest (rf) model and an artificial neural network (ann) were also trained to augment the analysis of the statistical model’s coefficients. the rf model utilized 15 maximum depth and 100 estimators, while the ann was configured with two hidden layers. these machine-learning models used the output of the armax model as the target and shared the same input variables as the statistical model. two techniques of xai (explainable artificial intelligence) were used, shap (shapley additive explanations) and lime (local interpretable model-agnostic explanations), to provide a deeper understanding of the model outputs. this approach aims to offer an alternative explanation to the traditional statistical coefficient analysis. because these machine learning models were trained on the armax outputs, the shap/lime values can provide an alternative viewpoint on the statistical model outputs from both the random forest and ann perspectives. 5 results and discussion the initial analysis primarily focused on scrutinizing the coefficients of the statistical model, evaluating the magnitude of each coefficient across multiple prediction steps. however, before delving into the interpretation of the statistical model, it is imperative to assess the model’s adequacy. table 4 presents the mean absolute percentage error (mape) for each model applied to the target time series. the metrics were collected in the test set. table 4. armax model mape.full size table upon examination of table 4, both models exhibit satisfactory fits. thus, we proceed with the analysis of the coefficients of the statistical model. notably, given the proximity of change points in both models, our analysis will concentrate specifically on the window from steps 230 to 260, encapsulating the change point for both time series. figure 4 illustrates the logarithm of the coefficient magnitudes for each exogenous variable. fig. 4. coefficient magnitude of exogenous variables. full size image the coefficients divulge that, for both datasets, the armax model accords greater significance to the urban area, followed by the farming biomes. in the case of the cajazeiras time series, the coefficient magnitude for farming closely rivals that of the forest, albeit numerically higher. this suggests that the urban biome significantly influences the observed temperature increase over the analyzed period. another crucial aspect to consider is the employment of explainable ai (xai) techniques in the machine learning models, which were fitted to the statistical outputs generated by the armax model. these models exhibited low errors, as corroborated in table 5. table 5. machine learning models mape.full size table furthermore, the utilization of shapley values in the random forest model, trained on the armax model’s statistical output, offers insights into the impact of features. higher positive shap values signify a positive contribution to the temperature increase from an exogenous variable, while higher negative shap values imply otherwise. figure 5 provides an overview of the shap values measured for both time series. fig. 5. summary of shapley values for cajazeiras (up) and são félix do xingu (down). full size image the findings presented in fig. 5 offer intriguing insights. beyond the expected correlation indicating that higher urban areas correspond to increased temperatures, the shap values suggest that an increase in farming areas contributes to rising temperatures. conversely, a decrease in forest coverage accelerates this increase. it is vital to acknowledge the distinct characteristics of the two cities analyzed. são félix, nestled within the amazon rainforest, benefits from the cooling properties of its dense forest. in contrast, cajazeiras, characterized by the caatinga biome, features sparser vegetation, offering lesser cooling effects. this distinction is mirrored in the shap values, which assign numerically higher importance to the forest of são félix due to its more pronounced cooling effects. additionally, a multilayer perceptron (mlp) was employed with lime to analyze the data. the mlp underwent 250 epochs of training using the same inputs as the armax statistical model. figure 6 showcases the box plot results of the lime analysis. fig. 6. box plot of lime results for cajazeiras (left) and são félix do xingu (right). full size image similar to the shap values applied to the random forest model, the lime technique underscores the significant influence of increases in farming and urban areas on temperature rises in both cities. notably, both models not only identify the increase in urban and farming biomes as pertinent to the temperature increase but also acknowledge the forest biome’s role as a cooling factor for both cities. 6 conclusion this study presented a comprehensive analysis of the impact of land use changes on local temperatures in two distinctly different brazilian cities, cajazeiras and são félix do xingu, using advanced statistical and machine learning models. by employing an armax model alongside multilayer perceptron (mlp) and random forest models enhanced with explainable ai techniques (shap and lime), we quantified the effects of urban, farming, and forest biomes on temperature variations. the findings indicate a significant influence of urban and farming areas on increasing temperatures, confirming the critical role of land use in local climate change. the approach proposed in this study combines statistical modeling with machine learning outputs. future research could explore how these outputs (armax coefficients, shap values, and lime explanations) might be integrated to create a new explicability score for exogenous variables, providing enhanced clarity and understanding of these variables within the models. although the current methodology already offers valuable insights for various applications, with each technique revealing different aspects of the data, further development could lead to even more robust understanding and applications. further development can also explore more complex data. the current analisys has only 3 exogenous variables, future works can explore data with more exogenous data. references adamo, n., al-ansari, n., sissakian, v.: review of climate change impacts on human environment: past, present and future projections. engineering 13(11), 605–630 (2021). https://doi.org/10.4236/eng.2021.1311044 article  math  google scholar  anjali, t., chandini, k., anoop, k., lajish, v.l.: temperature prediction using machine learning approaches. in: 2019 2nd international conference on intelligent computing, instrumentation and control technologies (icicict), pp. 1264–1268. ieee (2019). https://doi.org/10.1109/icicict46008.2019.8993316. https://ieeexplore.ieee.org/document/8993316/ arshad, m.z., iqbal, z., kumar, m.s.: the urban expansion and its effect on temperature trends of faisalabad city, pakistan. nat. appl. sci. int. j. 5(1), 27–43 (2024) math  google scholar  cifuentes, j., marulanda, g., bello, a., reneses, j.: air temperature forecasting using machine learning techniques: a review. energies 13(16), 4215 (2020). https://doi.org/10.3390/en13164215. https://www.mdpi.com/1996-1073/13/16/4215 de andrade, j.v.r., junior, t.g.d.s., leandro, h.d.s., lucena, d.b., fernandes, b.j.: assessing the effect of urban expansion and deforestation on temperature rise in cajazeiras, brazil: a data-driven approach. in: 2023 ieee latin american conference on computational intelligence (la-cci), pp. 1–6. ieee (2023) google scholar  flores, b.m., et al.: critical transitions in the amazon forest system. nature 626(7999), 555–564 (2024). https://doi.org/10.1038/s41586-023-06970-0 article  math  google scholar  hernanz, a., garcíaâ-valero, j.a., domínguez, m., rodríguez-camino, e.: a critical view on the suitability of machine learning techniques to downscale climate change projections: illustration for temperature with a toy experiment. atmos. sci. lett. 23(6) (2022). https://doi.org/10.1002/asl.1087. https://onlinelibrary.wiley.com/doi/10.1002/asl.1087 ipcc: climate change 2022: mitigation of climate change. contribution of working group iii to the sixth assessment report of the intergovernmental panel on climate change. cambridge university press, cambridge, uk and new york, ny, usa (2022). https://doi.org/10.1017/9781009157926. https://www.ipcc.ch/report/ar6/wg3/downloads/report/ipcc_ar6_wgiii_fullreport.pdf kalnay, e., cai, m.: impact of urbanization and land-use change on climate. nature 423(6939), 528–531 (2003). https://doi.org/10.1038/nature01675 article  math  google scholar  li, x., li, z., huang, w., zhou, p.: performance of statistical and machine learning ensembles for daily temperature downscaling. theor. appl. climatol. 140(1-2), 571–588 (2020). https://doi.org/10.1007/s00704-020-03098-3. http://link.springer.com/10.1007/s00704-020-03098-3 myers, n.: threatened biotas: “hot spots” in tropical forests. environmentalist 8(3), 187–208 (1988). https://doi.org/10.1007/bf02240252 do nascimento, a.c.l., galvani, e., gobo, j.p.a., wollmann, c.a.: comparison between air temperature and land surface temperature for the city of são paulo, brazil. atmosphere 13(3), 491 (2022). https://doi.org/10.3390/atmos13030491. https://www.mdpi.com/2073-4433/13/3/491 qi, t., ren, q., he, c., zhang, x.: dual effects on vegetation from urban expansion in the drylands of northern china: a multiscale investigation using the vegetation disturbance index. sci. total environ. 172481 (2024) google scholar  roser, m.: humans destroyed forests for thousands of years – we can become the first generation that achieves a world in which forests expand. our world in data (2022). https://ourworldindata.org/global-forest-transition da silva, r.f.b., et al.: slow-down of deforestation following a brazilian forest policy was less effective on private lands than in all conservation areas. commun. earth environ. 4(1) (2023). https://doi.org/10.1038/s43247-023-00783-9 souza, c.m., et al.: reconstructing three decades of land use and land cover changes in brazilian biomes with landsat archive and earth engine. remote sens. 12(17), 2735 (2020). https://doi.org/10.3390/rs12172735 download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, and the brazilian agencies facepe and cnpq. author information authors and affiliations escola politécnica de pernambuco, universidade de pernambuco upe, recife, brazil joão v. r. de andrade, igor l. b. da silva, agostinho freire & bruno j. t. fernandes instituto federal de educação, ciência e tecnologia da paraíba, ifpb, joão pessoa, brazil teobaldo g. de souza junior & leandro h. de s. silva universidade federal da paraíba, ufpb, joão pessoa, brazil daisy lucena authors joão v. r. de andradeview author publications search author on:pubmed google scholar igor l. b. da silvaview author publications search author on:pubmed google scholar teobaldo g. de souza juniorview author publications search author on:pubmed google scholar leandro h. de s. silvaview author publications search author on:pubmed google scholar agostinho freireview author publications search author on:pubmed google scholar daisy lucenaview author publications search author on:pubmed google scholar bruno j. t. fernandesview author publications search author on:pubmed google scholar corresponding author correspondence to joão v. r. de andrade . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de andrade, j.v.r. et al. (2025). exploring climatic shifts in brazilian climates: insights from armax, decision trees, and artificial neural networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_12 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords explainable ai armax models neural networks global warming land use changes publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a framework for controversial political topics identification using twitter data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a framework for controversial political topics identification using twitter data conference paper first online: 12 october 2023 pp 283–298 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a framework for controversial political topics identification using twitter data download book pdf download book epub kenzo sakiyama  orcid: orcid.org/0000-0002-5284-47459, lucas de souza rodrigues  orcid: orcid.org/0000-0003-4704-066x10, bruno magalhães nogueira  orcid: orcid.org/0000-0002-4703-137910, edson takashi matsubara  orcid: orcid.org/0000-0002-4471-088610 & … roseli a. f. romero  orcid: orcid.org/0000-0001-9366-27809  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 574 accesses abstract social networks have become the main stage for discussion on various current topics. in particular, electoral processes tend to bring many publications with polarized opinions on political issues addressed by candidates. a comprehensive analysis of social media publications on high-impact controversial topics and the opinions expressed in them could contribute to a clearer understanding of the dynamics of political discussion, providing valuable insights for society. in this context, we investigate how to apply a clustering-based topic modeling approach to produce public evaluation information on different current issues, in particular controversial political topics. we propose a framework that enriches text representations, combining state-of-the-art unsupervised (hdbscan) and supervised (bertimbau) techniques to identify controversial political topics in social media publications in brazilian portuguese. to this end, weekly collections were carried out on the social network twitter, making it possible to identify controversial events for each analyzed date. we compare the controversial topics uncovered with real-world news to validate the results and compare our method with a traditional method described in the literature. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others using twitter data to monitor political campaigns and predict election results chapter © 2018 a deep learning approach to fine-grained political ideology classification on social media texts chapter © 2024 identifying communities in social media with deep learning chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. internet and politics media policy and politics political communication political science political psychology political sociology 1 introduction political topics generate heated discussions and opinion polarization in social networks between supporters and opponents about a given topic. this dynamic is especially evident during election times. when a candidate gives a speech, social networks are flooded with publications supporting or opposing the topics addressed in that speech. in brazil, a survey presented by the data senado institute [26] shows that the main communication channels for seeking information about politics are: tv (37%), social networks (24%), and internet sites (23%). young people most often use social networks and journalistic portals. tv attracts the more advanced age group and, from social network users, at least 20% confirm that they use them to talk about politics. therefore, the period before and after the 2022 brazilian elections indicates an excellent opportunity to analyze social media data on political discussions. as shown in fig. 1, using social media posts, it would be possible to better understand the political discussion by using a topic model to identify and study the different topics in the data. furthermore, using sentiment analysis, it is possible to estimate user ratings on different topics. in this article, we expose a framework based on topic modeling, using twitter publications about politics in brazil. however, this approach can be used in various current scenarios and topics. our goal is to automatically identify potential high-impact policy-related topics by combining two nlp tasks: topic modeling and sentiment analysis. the proposal is presented in fig. 2. fig. 1. topic modelling techniques in nlp. full size image fig. 2. controversial topic identification. full size image in the literature, there are several techniques for clustering and identifying topics. a traditional method for topic discovery and semantic mining of unsorted documents is latent dirichlet allocation (lda). however, the method has some drawbacks. it does not consider the temporal aspect of the data, when creating topics in data with different epochs. it is very computationally intensive to train a model with millions of examples [18]. also, lda does not consider sequential or semantic words as a prerequisite for creating topics [4]. as an alternative to the standard approach, the work of top2vec and bertopic [4, 14] explored a topic modeling methodology based on the use of state-of-the-art dense representations, followed by dimensionality reduction and clustering, to extract meaningful topics from document collections. in this sense, a series of studies emerged to evaluate the use of correlation of the aforementioned techniques, as described in [11], in which the performance of the different algorithms is evaluated in terms of their strengths and weaknesses in a social science context. based on the results of this study, the authors indicate that bertopic supports more embedding models than top2vec, allows multilingual analysis, and automatically determines the number of topics. the advantages of top2vec are the support for hierarchical topic reduction, the ability to work with very large datasets, and its use of embeddings so that no preprocessing of the original data is required. focusing on portuguese texts, bertopic has already been successfully applied to brazilian portuguese documents by [1]. in their work, the authors applied the methodology to automatically classify legal documents into the six most representative document classes of the brazilian national council of justice (cnj), achieving 89% of the macro f1 score. similarly, the authors of [28] examined the social impact of law changes on twitter social media. on the other hand, some papers have also conducted public opinion experiments on the political aspects of social networks. a recent study mapped a large-scale cross-party sentiment analysis in greece, spain, and the united kingdom on tweets [5]. the study showed a preponderance of negatively-tweeted tweets from politicians and examined trends in popularity and sentiment. another study in spain used twitter to analyze the impact of elite discourse on effective citizen polarization [16]. according to this study, users’ contact with candidates does not affect polarization. from this perspective, the aim of this article is to develop a framework that uses a topic modeling approach based on clustering techniques proposed by [1, 14, 28] for publications in brazilian portuguese on twitter about political topics. furthermore, we hypothesize that combining the extracted topics with state-of-the-art sentiment analysis can add more information and identify potentially controversial topics. thus, we can identify controversial issues and assess public appreciation. the main contributions of this work are: 1. a new framework for identifying controversial topics in twitter data, combining clustering and sentiment analysis based on state-of-the-art transformer representations; 2. evaluation of discovered topics. the results are evaluated by associating real-world events with the topics and by using quantitative metrics; 3. comparison between our methodology and a simpler approach; this work is organized as follows. in sect. 2, we briefly describe the related works. section 3 describes the methodology, sect. 4 shows our experimental results. finally, in sect. 5 are presented the conclusions and future works. 2 related works many works deal with information extraction using social networks, opinion mining, and general linguistic representations. in this section, we briefly survey examples of applications. text mining is an important area in knowledge discovery in databases (kdd). it focuses on discovering interesting patterns in structured and unstructured data [13]. as a result, the applications in this area are diverse. in addition, it fosters strong connections to natural language processing, data mining, machine learning, information retrieval, and knowledge management [23]. in the following, we present some examples of recent studies on topic modeling and text clustering related to this proposal. in the work proposed in [3], a machine learning-based approach is presented to improve cognitive distortions classification of the arabic content over twitter, enriching text representation by defining latent topics in tweets. another study on data clustering tools is the use of topic modeling for customer service chats [15]. the mentioned work focuses on finding new intents from user messages, that are not yet included in any previous intents and reorganizing existing intents by analyzing the topic model generated. in relation to text clustering, hdbscan [19] was applied to investigate how to link popular social media topics and news stories using transformer models and neural networks [2]. other works highlight the use of hdbscan to compare latent semantic analysis and latent dirichlet allocation, analyzing the topic covid-19 [27]. this study checks which most frequent words from each cluster will be displayed and compared with factual data about the outbreak to find out if there are any correlations. the authors state how well hdbscan clusters its data in comparison to k-means [22]. [17] investigates the most effective way of performing text classification and clustering of duplicate texts in technical documentation written in simplified technical english. vector representations from pre-trained transformers and lstm models were tested against tf-idf using the density-based clustering algorithms dbscan and hdbscan. monitoring social media research has become essential for government entities, large corporations, and global companies. several data mining tools currently assess public reaction to measures taken by a government, company and famous personalities. for example, some studies use social media data to predict a country’s elections based on public sentiment [8]. other recent work examines social media to study the public awareness of covid-19 pandemic trends and uncover meaningful themes of concern posted by twitter users [7]. focusing on data collection, there are several applications for extracting and analyzing data from social networks. tweepy is an advanced twitter scraping tool written in python allowing to scrape tweets from twitter profiles without using twitter’s official api [10]. another variant, snscrape, is a scraper for social networking services (e.g., instagram, facebook, and twitter) [6]. it scrapes data like user profiles, hashtags, or searches and returns the discovered items. 3 methodology in this section, we will describe the steps we took to identify controversial political themes using twitter data. in fig. 3 is shown an overview of our methodology, presenting its main components: data collection (sect. 3.1), tweet pre-processing (sect. 3.2), calibration and clustering (sects. 3.3 and 3.4), and the cluster analysis performed (sect. 3.7). at the end, in sect. 3.8 is discussed the evaluation of the controversial topics. fig. 3. overview of the controversial topic identification and analysis. full size image 3.1 data collection the 38\(^{th}\) former brazilian president jair messias bolsonaro used to perform weekly live streams on his channel on the youtube video sharing platform. during the live streams, which last an average of one hour, he discussed events of the week that relate to his government. we assumed that even if the live streams take place on a specific platform (youtube), supporters and opponents end up generating publications about the content of the broadcast on other social networks such as twitter. twitter is a social network in which users are capable of publishing and interacting with others through small text messages called tweets. given this context, we considered twitter posts, published after the start of the ex-president’s live streams, a good source of controversial topics. given the start time of the weakly live streams, we collected twitter posts related to the former brazilian president, and published up to 3 h after the start of the broadcasts. we focused on the lives performed in may 2022 and collected tweets using the snscrapefootnote 1 social media scrapper. our tweet data consists of tweets mentioning the ex-president twitter profile (using ‘ @jairbolsonaro’) or his name (‘bolsonaro’). this way, we expect that the collected tweets at least mention jair bolsonaro. table 1. examples collected for each may live stream. the last live stream was delayed due to the schedule of bolsonaro.full size table in table 1 is shown the number of tweets collected for each live stream of may. in total, 30.120 unique tweets were analyzed with an average of 20 tokens (space-separated tokens) per tweet. in addition, it’s worth mentioning that the amount of collected tweets is superior to the amount of posts published in the youtube comments section for each live stream. 3.2 tweet pre-processing as a text pre-processing step for the following analyses, we removed mentions (usernames), urls, empty texts and duplicated texts using the spacyfootnote 2 python package. we preserved hashtags as they can be good indicators of themes or subjects. in table 1, we present the number of examples remaining after the text pre-processing. following the text pre-processing, we converted the resulting texts to state of the art contextualized dense representations based on transformers [31]. we used efficient multilingual representations based on the minilm [32] language model, trained using the sentence transformers framework [24], and publicly available to use and researchfootnote 3. we limited the token sequence length to be 128 wordpiece tokens (smaller sequences are padded and larger sequences are truncated). the resulting representations for each tweet are vectors of 384 dimensions, generated by the mean of the contextualized token representations and well suited to be compared using similarity metrics such as cosine distance [24]. considering representations that use context, we expect publications semantically similar (or related to similar subjects) to be close in the representation space. finally, we used the umap (uniform manifold approximation and projection) [20] to reduce the dimension of the minilm vectors. umap is able to reduce dimensions and preserve the global distribution of the original data on the lower dimension space with competitive execution time when compared to other dimension reduction techniques (ex: t-sne). since our goal is to use clustering as an intermediate step to identify controversial topics, the dimension reduction is used to improve the efficiency of the clustering method. we observed that performing clustering without dimension reduction led to worse results. 3.3 clustering tweets with hdbscan we used the hdbscan (hierarchical density-based spatial clustering of applications with noise) [19] algorithm to cluster tweets and extract common topics, inspired by the success of previous works [4, 14]. the method consists in a hierarchical variation of dbscan clustering algorithm [12] that is able to identify clusters of different densities and be more robust in relations to its parameters. hdbscan identifies regions of large concentration of examples (high density) as clusters, and it’s able to ignore noisy examples (label as noise). when representing twitter publications as multidimensional vectors, we expect that regions of common subjects can be of any shape. hence, we chose to use hdbscan to identify these regions of common subjects as clusters. the motivation for using hdbscan is related to its ability to identify clusters of different densities, handle outliers, automatically discover the number of clusters and provide a hierarchical structure of clusters. furthermore, when dealing with social media posts, not every publication is associated to a common theme. assuming such examples are not close to high density areas, hdbscan can easily identify them as noise. 3.4 parameter calibration since clustering is used to identify topics, the parameters of the both methods, umapfootnote 4 and hdbscanfootnote 5 were chosen adequately. to find a good set of parameters, we used random search with the objective of maximizing the dbcv (density-based clustering validation index) metric [21]. dbcv is a metric created with the evaluation of density based clustering as its main goal. the metric evaluates the density of the obtained clusters (considering their shape properties) and consider the number of noise examples in its definition. it evaluates both the density sparseness of the examples of a cluster and the density separation of different clusters, using the core distance and mutual reachability distances presented by the authors. thus, we choose the dbcv metric, to evaluate the quality of the clustering using a metric that is suited to evaluate density-based clustering, instead of relying on metrics such as silhouette and db index, which may not capture the specific characteristics of density-based clustering algorithms. considering the umap dimension reduction pre-processing step, two parameters were adjusted: the number of components and the number of neighbors. the number of components determines the number of dimensions in which we want to embed our representations. the number of neighbors is a parameter that controls the balance between the local (favored by lower values) and the global structure (favored by larger values) of the data. furthermore, the cosine metric was used to determine distances in the original space and we fixed the minimum distance in the lower dimensions space as zero. for hdbscan, we chose to tune the following parameters: minimum cluster size and minimum samples. minimum cluster size defines the smallest number of examples that can be considered as a cluster. minimum samples specify the number of neighbors used to estimate the probability density function. for other parameters, we preserved the default parameters of hdbscan python package. although hdbscan determines the number of clusters automatically, we observed empirically that a minimum number of clusters is beneficial for the analysis. this way, parameter sets that generate less than 4 clusters were discarted. in table 2 is presented the obtained umap and hdbscan parameter sets for each day of analysis. table 2. parameters obtained for each may tweet collection and value ranges used.full size table 3.5 sentiment analysis sentiment analysis is used as an intermediate step of the controversial topic identification. contextualized dense representations generated by transformers [31] are predominant in different nlp tasks. souza et al. [30] investigated different usages of transformer representations in the sentiment analysis of reviews in brazilian portuguese. as shown by the authors results, fine-tuning a language model led to better performance in all the datasets used. therefore, to perform sentiment analysis on the collected tweets, we used the bertimbau-base [29] language model fine-tuned in the tweetsentbr corpus [9]. the corpus consists of tweets published during popular brazilian tv shows. a total of 12.312 examples were recovered from the original corpus. the examples are divided in three classes: positive (45%), neutral (25%) and negative (29%). we preserved the class imbalance, since the minority class has an expressive amount of examples (25%), and used the same text pre-processing described in the start of the sect. 3.2. we fine-tuned bertimbau for sentiment analysis using the popular huggingface libraryfootnote 6. we limited the token sequence length to a maximum of 128 tokens (padding the smaller sequences and truncating the larger ones). we trained the model for 5 epochs using: batch size of 128, learning rate of \(2\times 10^{-5}\) and a cosine scheduler with warm-up (10% of the train data used for warm-up). evaluating using a stratified holdout (80% train and 20% test), the model was able to achieve \(69.15\%\) macro-f1. we used the fine-tunned model to perform sentiment analysis in the collected tweets described in sect. 3.1. 3.6 controversial topic identification the controversial topic detection is done by combining the clustering and sentiment analysis of the collected tweets. for each date of analysis, hdbscan is used for generating the cluster labels. then, the fine-tuned bertimbau extracts the sentiments of each tweet in the clusters. as the next step, the clusters are sorted by the percentage of positive, neutral and negative examples in the clusters. finally, assuming that each cluster corresponds to a certain subject, we observed that potential controversial topics are usually located in clusters with large amounts of negative publications. therefore, a controversial cluster is defined as being a cluster with a negative percentage above a threshold c (\(c \in \mathbb {r}\), \(0 \le c \le 1\)), where c is a configurable parameter. it was observed that a threshold \(c = 0.7\) was enough to identify controversial clusters in the collected data. following this methodology, by identifying a controversial cluster, we can also quantify how bad the topic was received by the twitter users by analysing the c value. 3.7 cluster analysis the same cluster based tf-idf used by bertopic [14], is used to discriminate and analyse the tweet clusters. the procedure consists in a generalization of tf-idf score. by modifying the definition of term frequency and inverse document frequency, a cluster based tf-idf allow us to generalize tf-idf representation from documents to collections (clusters) of documents. the modification of tf-idf is defined as \(w_{t,c} = tf_{t,c}\cdot \log \left( 1+\frac{a}{tf_{t}}\right) \), where \((tf_{t,c})\) represents the frequency of a token t in a cluster c, \((tf_{t})\) represents the frequency of t in all clusters and (a) average number of tokens per cluster. tf-idf aims to associate high scores to the most discriminative tokens in a text example. treating all examples of a cluster as a single example, it is expected that the most discriminative tokens of a cluster have the highest tf-idf scores. thus, we summarized each cluster by the tokens with the highest tf-idf scores. this way, by analysing the filtered tokens, we can easily identify the topics present in the clusters. to generate the cluster tf-idf score, we added lemmatisation and punctuation and stop-word removal to the text pre-processing step (sect. 3.2), treated the cluster examples as a single document (by concatenation) and generated the scores using unigrams. 3.8 evaluation in addition to identifying a controversial cluster, it is interesting to associate a real world event to its content as a form of validation. in order to associate news or presidential declarations to the controversial clusters, we searched manually over the news published during the week period analyzed. we also analyzed the topics discussed in the bolsonaro’s live streams, in search of subjects that could also appear in the clusters. to quantitatively assess the discoveries, we computed the coherence measurement \(c_v\) [25] for each controversial topic identified. based on the distributional hypothesis of words, the coherence measurement \(c_v\) aims to quantify how much the topic is highlighted by the documents analysed. we evaluated the \(c_v\), summarizing the controversial topic by it’s top-10 tokens with the highest cluster tf-idfs (using the same pre-processing described in the sect. 3.7). 4 results and discussions in this section, we will present the results of exploring the hdbscan clusters with the sentiment analysis data. 4.1 clustering and sentiment analysis table 3 shows the results of applying the parameters obtained by the calibration step (sect. 3.4) in the twitter data collected. we obtained dbcv values above 0.45 for all analyzed dates, and the collection at may the 19th generated the largest number of clusters and the lowest dbcv. table 3. clustering and sentiment analysis results obtained.full size table in relation to sentiment analysis, table 3 presents the quantities of positive, neutral and negative tweets in the collected data. in general, we observed similar percentages of neutral tweets. on the other hand, the percentages of positive and negative tweets varied the most, with the highest percentages being associated with negative tweets. the predominance of negative comments, in the brazillian scenario, corroborates the previous mentioned work [5]. 4.2 controversial topic discussion table 4. controversial clusters detected and their respective sizes (number of examples), negative percentage and \(c_v\).full size table fig. 4. word cloud visualizations for the 50 tokens with the highest cluster-tf-idf in the cluster tf-idf. the bigger the token, the higher its tf-idf score. full size image in table 4 is shown information about the controversial clusters identified. we observed a large variation in the number of examples of the controversial clusters and in the \(c_v\) metric between the dates of analysis. in addition, may \(19^{th}\) was the only date with more than one controversial cluster identified. to visualize the controversial clusters identified by hdbscan, we generated the word clouds shown in the fig. 4. we analyzed the tweets in each controversial cluster and will discuss the discovered controversial events below. may \(05^{th}\) (fig. 4a): a large quantity of tweets may have been motivated by a declaration made by bolsonaro during his broadcast, in which he expressed his concern about frauds in the upcoming election. to add more context, in the same week of the live stream, the brazilian military forces questioned the entity responsible for the electoral process about possible vulnerabilities in the electronic voting machine.footnote 7 may \(12^{th}\) (fig. 4b): during his live stream, bolsonaro talked about a possible deal with petrobras (important brazilian petroleum corporation) about reduction in the price of fuel. in the same week, the brazilian senate approved a bill that changes taxes residing on fuel. we observed tweets related to both these events in the cluster.footnote 8 may \(19^{th}\) (figs. 4c and 4d): on this day, we identified two clusters with negative percentages over 70%. the first one (fig. 4c) may be related to a speech made by the bolsonaro during an event in the capital of rio de janeiro in the same day. in his speech, bolsonaro repeated that in his government there is no corruption and criticized the predecessor government. we observed the lowest \(c_v\) (0.402) for this cluster. the second cluster (fig. 4d) seems to contain tweets related to news published during the week, which discloses the ex-president’s expenses using his corporate card.footnote 9 may \(27^{th}\) (fig. 4e): on the same day of the live stream, the brazilian government announced a cut in spending destined for the brazilian ministry of education. we observed a large amount of tweets related to this event in the cluster, and it was the worst reception by users (86% negative) and the topic with the highest \(c_v\).footnote 10 4.3 evaluating a simpler approach table 5. top-10 tokens and \(c_v\), based on the cluster tf-idf for each date of data collection. hdbscan top tokens on the left, and k-means top tokens on the right.full size table supposing that a simpler clustering approach could obtain similar results, we repeated the methodology using the k-means algorithm and no dimension reduction. we chose k-means, instead of sparse topic modelling methods such as lda, since it could benefit from using the dense minilm representations. we fixed the k-means parameters (200 random initialisations and 500 maximum iterations), and determined the number of clusters using the elbow rule. focusing only on the clusters with the highest percentage of negative tweets, table 5 compares the top-10 tokens based on their cluster tf-idf for hdbscan and k-means. as we can see, even tough we observed similar results for the first date of analysis (may \(05^{th}\)), the use of hdbscan in conjunction with the umap dimension reduction led to clusters with more discriminative and meaningful tokens in the other days. this result is also indicated in the cv, in which, although the \(c_v\) of the hdbscan topics varied the most, their values were always superior to the topics extracted by k-means. 4.4 other discoveries fig. 5. examples of clusters with high percentage of positive and neutral tweets. word clouds generated the same way used in the sect. 4.2 full size image to extract controversial topics, we focused only on the negative tweets. although the percentages of positive and neutral tweets were not as representative as the negatives, we choose to investigate their examples as well. then, by analyzing the clusters with high percentages of these classes, we also discovered interesting patterns. considering all dates of analysis and all identified clusters, we will briefly discuss the findings in the following. most positive clusters (fig. 5a): analyzing the most positive clusters obtained for each date, we mainly identified tweets from supporters of bolsonaro. in specific, at the start of each live stream, the ex-president’s twitter account (‘@jairbolsonaro’) shared the youtube link of the broadcast as a tweet. we identified that the clusters with the highest positive rates usually contain tweets from supporters, in response to the official account’s tweet, congratulating bolsonaro for carrying out the live stream. we discovered the tweets source (in response to) by looking at the tweets metadata obtained from snscrape. most neutral clusters (fig. 5b): sorting and investigating clusters by their neutral percentage, we identified clusters consisting of many short tweets containing laughter (‘kkkkkk’) without much context. hdbscan was able to isolate the ‘laugh’ clusters, but we suspected that the amount of neutral examples may indicate unwanted bias in the bertimbau sentiment classifier. thus, by looking at the examples in the clusters, we identified many ironic or sarcastic (with negative sentiment) tweets incorrectly labeled as being neutral. 5 conclusions in this paper, controversial political topics were successfully identified in twitter data by applying clustering-based topic modeling and analyzing the negative publications. to the best of our knowledge, this is the first study to combine clustering and sentiment analysis based on transformers to identify controversial topics in social media. we identified one event for each date and validated the detected polemic by searching the news published during the week. we compared two different approaches to identify controversial clusters and obtained favorable results using umap and hdbscan method. finally, we presented results investigating positive and neutral tweets in the obtained clusters. as future works, we intend to replace the sentiment analysis step with aspect-based sentiment analysis to monitor social posts about other brazilian politicians. the sentiment analysis could be improved by treating politician names as an aspect of the analysis. we also intend to replicate our methodology to analyze data from other social networks (such as facebook and instagram). notes 1.https://github.com/justanotherarchivist/snscrape. 2.https://spacy.io/. 3.https://huggingface.co/sentence-transformers/paraphrase-multilingual-minilm-l12-v2. 4.https://umap-learn.readthedocs.io/en/latest/. 5.https://hdbscan.readthedocs.io/en/latest/. 6.https://huggingface.co/. 7.may \(05^{th}\) links: www.uol.com.br | www.em.com.br. 8.may \(12^{th}\) links: www.uol.com.br | www.senado.leg.br. 9.may \(19^{th}\) links: www.uol.com.br | www.correiobraziliense.com.br. 10.may \(27^{th}\) links: www.brasil247.com. references aguiar, a., silveira, r., furtado, v., pinheiro, v., neto, j.a.m.: using topic modeling in classification of brazilian lawsuits. in: pinheiro, v., et al. (eds.) propor 2022. lncs (lnai), vol. 13208, pp. 233–242. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_22 chapter  google scholar  akhgari, z., malekimajd, m., rahmani, h.: sem-ted: semantic twitter event detection and adapting with news stories. in: 2022 8th international conference on web research (icwr), pp. 61–69. ieee (2022) google scholar  alhaj, f., al-haj, a., sharieh, a., jabri, r.: improving arabic cognitive distortion classification in twitter using bertopic. int. j. adv. comput. sci. appl. 13(1), 854–860 (2022) google scholar  angelov, d.: top2vec: distributed representations of topics. arxiv preprint arxiv:2008.09470 (2020) antypas, d., preece, a., collados, j.c.: politics and virality in the time of twitter: a large-scale cross-party sentiment analysis in greece, spain and united kingdom. arxiv preprint arxiv:2202.00396 (2022) archivist, j.a.: github snscrape is a scraper for social networking services (sns). it scrapes things like user profiles, hashtags, or searches and returns the discovered items, e.g. the relevant posts (2020). https://github.com/justanotherarchivist/snscrape. accessed 15 may 2022 boon-itt, s., skunkan, y., et al.: public perception of the covid-19 pandemic on twitter: sentiment analysis and topic modeling study. jmir public health surveill. 6(4), e21978 (2020) article  google scholar  bose, r., dey, r.k., roy, s., sarddar, d.: analyzing political sentiment using twitter data. in: satapathy, s.c., joshi, a. (eds.) information and communication technology for intelligent systems. sist, vol. 107, pp. 427–436. springer, singapore (2019). https://doi.org/10.1007/978-981-13-1747-7_41 chapter  google scholar  brum, h.b., nunes, m.d.g.v.: building a sentiment corpus of tweets in brazilian portuguese. arxiv preprint arxiv:1712.08917 (2017) chaudhary, j., niveditha, s.: twitter sentiment analysis using tweepy. int. res. j. engtech 8, 4512–6 (2021) google scholar  egger, r., yu, j.: a topic modeling comparison between lda, nmf, top2vec, and bertopic to demystify twitter posts. front. sociol. 7, 886498 (2022) article  google scholar  ester, m., kriegel, h.p., sander, j., xu, x., et al.: a density-based algorithm for discovering clusters in large spatial databases with noise. in: kdd, pp. 226–231 (1996) google scholar  feldman, r., et al.: knowledge management: a text mining approach. in: proceedings of the 2nd international conference on practical aspects of knowledge management (pakm 1998), pp. 9–1. no. conf (1998) google scholar  grootendorst, m.: bertopic: neural topic modeling with a class-based tf-idf procedure. arxiv preprint arxiv:2203.05794 (2022) hendry, d., et al.: topic modeling for customer service chats. in: 2021 international conference on advanced computer science and information systems (icacsis), pp. 1–6. ieee (2021) google scholar  lorenzo-rodríguez, j., torcal, m.: twitter and affective polarisation: following political leaders in spain. south eur. soc. polit. 27, 1–27 (2022) google scholar  lund, m.: duplicate detection and text classification on simplified technical english (2019) google scholar  marjanen, j., zosa, e., hengchen, s., pivovarova, l., tolonen, m.: topic modelling discourse dynamics in historical newspapers. arxiv preprint arxiv:2011.10428 (2020) mcinnes, l., healy, j., astels, s.: hdbscan: hierarchical density based clustering. j. open source softw. 2(11), 205 (2017) article  google scholar  mcinnes, l., healy, j., melville, j.: umap: uniform manifold approximation and projection for dimension reduction. arxiv preprint arxiv:1802.03426 (2018) moulavi, d., jaskowiak, p.a., campello, r.j., zimek, a., sander, j.: density-based clustering validation. in: proceedings of the 2014 siam international conference on data mining, pp. 839–847. siam (2014) google scholar  na, s., xumin, l., yong, g.: research on k-means clustering algorithm: an improved k-means clustering algorithm. in: 2010 third international symposium on intelligent information technology and security informatics, pp. 63–67. ieee (2010) google scholar  radovanovic, m., ivanovic, m.: text mining: approaches and applications. novi sad j. math. 38, 227–234 (2008) math  google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. arxiv preprint arxiv:1908.10084 (2019) röder, m., both, a., hinneburg, a.: exploring the space of topic coherence measures. in: proceedings of the eighth acm international conference on web search and data mining, pp. 399–408 (2015) google scholar  senado, i.d.: datasenado portal institucional do senado federal. https://www12.senado.leg.br/institucional/datasenado/publicacaodatasenado?id=panorama-politico-2022. accessed 06 oct 2022 sheikha, h.: text mining twitter social media for covid-19: comparing latent semantic analysis and latent dirichlet allocation (2020) google scholar  silva, n.f.f., et al.: evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 104–120. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_8 chapter  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  souza, f.d., filho, j.b.o.s.: bert for sentiment analysis: pre-trained and fine-tuned alternatives. in: pinheiro, v., et al. (eds.) propor 2022. lncs (lnai), vol. 13208, pp. 209–218. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_20 chapter  google scholar  vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, vol. 30 (2017) google scholar  wang, w., wei, f., dong, l., bao, h., yang, n., zhou, m.: minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. adv. neural. inf. process. syst. 33, 5776–5788 (2020) google scholar  download references author information authors and affiliations universidade de são paulo icmc, são carlos, brazil kenzo sakiyama & roseli a. f. romero universidade federal de mato grosso do sul facom, campo grande, brazil lucas de souza rodrigues, bruno magalhães nogueira & edson takashi matsubara authors kenzo sakiyamaview author publications search author on:pubmed google scholar lucas de souza rodriguesview author publications search author on:pubmed google scholar bruno magalhães nogueiraview author publications search author on:pubmed google scholar edson takashi matsubaraview author publications search author on:pubmed google scholar roseli a. f. romeroview author publications search author on:pubmed google scholar corresponding author correspondence to kenzo sakiyama . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sakiyama, k., de souza rodrigues, l., nogueira, b.m., matsubara, e.t., romero, r.a.f. (2023). a framework for controversial political topics identification using twitter data. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_19 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords topic modelling clustering sentiment analysis publish with us policies and ethics profiles edson takashi matsubara view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a transformer-based tabular approach to detect toxic comments | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a transformer-based tabular approach to detect toxic comments conference paper first online: 31 january 2025 pp 18–30 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a transformer-based tabular approach to detect toxic comments download book pdf download book epub ghivvago damas  orcid: orcid.org/0000-0001-5466-66079, rafael torres anchiêta  orcid: orcid.org/0000-0003-4209-901310, raimundo santos moura  orcid: orcid.org/0000-0002-1558-38309 & … vinicius ponte machado  orcid: orcid.org/0000-0003-3391-84439  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 225 accesses 1 citation abstract in recent years, there has been a significant increase in toxic and hateful speech on social media platforms, becoming deeply entrenched in online interactions. this issue has drawn the attention of researchers from various academic fields, leading them to extend their focus to include disciplines such as natural language processing, machine learning, and linguistics, in addition to traditional areas like law, sociology, psychology, and politics. this paper introduces an approach for detecting toxic and hateful speech on social media using tabular deep learning. the goal is to apply and evaluate the performance of the ft-transformer model in detecting hateful and toxic content in textual comments on social media in brazilian portuguese. an important aspect of this research involves using modern embedding models as language embedders and language models and evaluating their performance with the ft-transformer, a transformer-based tabular model. the experimental scenario uses a binary version of the told-br dataset. our approach achieved a 76% accuracy rate and a 75% macro f1-score using the openai text-embedding-3-large model. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a bert-based transfer learning approach for hate speech detection in online social media chapter © 2020 tackling toxicity in arabic social media through advanced detection techniques article open access 21 november 2025 a novel hybrid model of word embedding and deep learning to identify hate and abusive content on social media platform chapter © 2024 1 introduction with the rise of online social media platforms, increasing levels of toxicity and hateful behavior have become a growing concern. while social media has revolutionized communication itself in so many ways, it has also led to social issues beyond hate speech, including cyberbullying and misinformation. the anonymity of online platforms has created virtual public spaces where intolerance thrives, underscoring the need for strategies to address hateful speech and toxic content, as well as manage the negative impacts of social media use [39]. detecting hate and toxicity in comments on social media platforms presents a complex challenge. despite being identified as harmful, unlawful, or even criminal, these comments typically attract significant engagement and are frequently overlooked by the algorithms of social media networks [41], meaning that social media companies are fully aware of the nature of hate speech and its implications, however, the mechanisms applied to ensure the moderation policies are very limited or ineffective [43]. researchers are developing advanced algorithms and strategies to effectively detect and filter hate speech and toxicity on social media to overcome these shortcomings in current moderation policies. also, understanding the complexities of this issue is essential, including recognizing that not all offensive language necessarily constitutes hate speech or toxicity [18]. factors such as context, the connection between speaker and audience, dynamics within social groups, the particular social media platform used, and timing play a role in accurately identifying harmful speech, including its subforms such as hateful speech [27]. challenges include distinguishing between counter-hate speech, derogatory language used for emphasis, and subtler forms of hate and toxicity, such as sarcasm and metaphors [23]. the structure of language communication and its relationship with human perception are key factors in uncovering hate and toxicity online. due to the difficulties presented in this task, researchers are investigating different computational strategies to address it and have developed other approaches beyond full supervision. these strategies encompass a range of techniques from classical methods like term frequency—inverse document frequency (tf-idf) and bag-of-words (bow) to vector representations, deep learning architectures, and transformer-based models [27]. modern methods have advanced to incorporate hybrid techniques and multimodal approaches [19], language models (lms), graph machine learning [37], and the use of large language models (llms) [24] showing promising results. despite the various existing approaches, there is a lack of investigation into approaches for structured data, such as tabular data. this work looks at tabular deep learning (tdl) as a viable and valuable tool for bringing diversity and unconventional alternatives into text classification tasks. applying tdl models to hate speech detection can provide various benefits, including those found in the most commonly used models (svm, xgboost, lightgbm, and neural networks) and those that exceed its limitations when handling raw heterogeneous features, such as texts, sequences, images, audio, and embeddings. integrating with other deep learning models or complex representation methods is also a limitation in most traditional methods, mainly in gradient boosting decision trees (gbdts), in addition to low flexibility and inability to work with modern relational databases complexity and dimensionality [15]. this paper introduces an approach to detect hate speech and toxic comments in brazilian portuguese, employing the ft-transformer (ftt) [13] model as a binary classifier. external text encoders, or text embedders (te) [15], are applied to the ftt model. these tes can be driven by modern embedding models and pre-trained language models (ptlms) to transform raw or tokenized text into meaningful embedded representations known as embeddings. this transformer-based tabular approach was evaluated using the told-br [17] dataset, where various models were used to generate embeddings. although the outcomes have not outperformed all hate speech detection methods, compared to other methods, this approach has shown competitive results as it requires fewer computational resources, allows faster training, and does not use any output optimization techniques. the structure of this paper is as follows: sect. 2 provides a concise overview of related work. section 3 describes our developed approach in detail. in sect. 4, we show the experimental scenarios and datasets, followed by an analysis of the results. finally, sect. 5 concludes the paper and outlines potential future research directions. 2 related work the initial research on hate speech and offensive language paved the way for further exploration. pioneering works by chen et al. [8], burnap and williams [6], waseem and hovy [42], and nobata et al. [22] defined and analyzed offensive and toxic speech, user behavior, and social media moderation. for brazilian portuguese, significant contributions in hate, toxic and harmful speech detection, and text analysis were made by almeida et al. [1], pelle et al. [25], bispo [4], silva and serapiao [33], leite et al. [17], and fortuna et al. [12]. those studies applied machine learning techniques, textual feature representation methods, and discourse analysis to detect and analyze toxic and hateful comments on social media, advancing the field of automated hate speech detection. the research by almeida et al. [1] and pelle et al. [25] expanded the work of offensive and harmful speech online. almeida et al. [1] study proposed a hate speech identification strategy using information theory quantifiers, achieving an f1-score of 86%, 84%, and 96% for classifying hate, offensive, and regular speech classes. the study did not focus on hate speech in the portuguese language. however, it still made valuable contributions to the scientific community and was useful for future portuguese content research. pelle et al. [25] introduced hate2vec, an ensemble-based classifier for detecting offensive comments on web platforms, performing well with datasets in english and portuguese, compared to the traditional bow classifier, attaining an f-score above 90%. further advancements in hate speech detection were made by bispo [4] and silva and serapiao [33]. they developed classifiers using lstm and cnn architectures, achieving significant accuracy with embeddings like wang2vec and glove. bispo [4] specifically created a cross-lingual classifier from english to portuguese using gbdts, with f1-score accuracy ranging from 72% to 91%. in contrast, silva and serapiao [33] utilized a cnn architecture to identify hate speech, achieving f1-scores and accuracy percentages between 82.64% and 96.74%. different optimization techniques were employed for various datasets, such as adam for offcombr [10] and rmsprop for hlphsd [11]. the work of leite et al. [17] introduced the told-br dataset for toxic comments in brazilian portuguese. it used fine-tuned bertimbau and multilingual bert for classification, achieving a macro f1-score of 76%. fortuna et al. [12] studied the generalization capabilities of classifiers for hate speech, toxicity, abusive, and offensive language, finding poor generalization with multilingual datasets and bert, better generalization from english to portuguese with a 67% macro f1 score, and the best generalization between english datasets with a 70% macro f1 score. saraiva et al. [32] introduced a novel semi-supervised node graph-based approach for detecting toxic comments with the told-br dataset. their method utilizes an undirected and weighted heterogeneous graph network (hgn) with 100-dimensional glove embeddings for the portuguese language and achieves a 73% macro f-score using only 10% of the told-br. they used a gradient boost classifier with length-generalization consistency (lgc) as a transduction method. recent research has explored various machine learning and modern embedding models, such as large language models (llms) embeddings and sbert [28], for detecting toxic and hateful speech on social media. these models have proven effective in tasks like classification, clustering, and reranking in nlp. studies have emphasized the value of modern embedding models in improving text-based tasks and their overall impact on nlp [20]. additionally, research has shown the effectiveness of the language-agnostic bert sentence embedding model in cross-lingual and multilingual hate speech detection [31]. other methods used different techniques with llms, including prompting and end-to-end classification. one study by oliveira et al. [24] explored openai chatgpt (gpt-3.5-turbo) [5], the chatcompletion module, and 2 prompts to assess gpt’s performance compared to bert-based models on various datasets. gpt achieved f1-scores of 73% as a zero-shot classifier on the told-br dataset and 74% when prompted in the cross-dataset experimental scenario with the balanced hlphsd dataset. in another investigation by da rocha junqueira et al. [29], a cross-task comparison was performed between bert-based models bertimbau and albertina pt-br. despite fine-tuning efforts, albertina pt-br could not match the performance of bertimbau. the base model of bertimbau scored an f1-score of 88%, while its large model scored 89%. in contrast, the albertina pt-br base model only achieved a score of 74%. the study by da silva oliveira et al. [34] examined the performance of llms like gpt-3.5-turbo and maritaca ai sabiá [26] in zero-shot and few-shot learning approaches, comparing them to the bertimbau model. sabiá demonstrated an enhanced and precise capacity to classify texts containing colloquial and slang expressions, and could identify aggressive and obscene words based on prompt-specific design. analysis of the told-br revealed differences in performance between chatgpt and maritalk, with maritalk showing improved precision due to its deeper understanding of portuguese subtleties. meanwhile, assis et al. [2] conducted a study evaluating the ability of language models to distinguish neutral, offensive, and hateful speech in social media posts. the pt-br bert-based classifiers surpassed the chatbots on the hatebr [38] dataset, but on the neutral class, the chatbots outperformed the bert-based classifiers on the told-br dataset. chatgpt and maritalk yielded f1 scores of 71% and 70% respectively, which were lower than bert-based classifiers, ranging from 77% to 86%. other initiatives dive into tdl models as alternatives for unconventional and novel binary text classification approaches and have gained attraction, with notable frameworks developed by younus and qureshi [44] and chopra et al. [9]. the work of chopra et al. [9] proposes an automated method for detecting hate speech in code-mixed hindi-english text and hindi text in devanagari, where the framework architecture employs a tabnet classifier model trained on features extracted using a bert-based model for indian languages (muril) [16] from transliterated code-mixed data. this study demonstrated that the tabnet with muril embeddings was effective for devanagari text features, even though it was trained on transliterated data. the framework of younus and qureshi [44] highlighted the challenges of sexism detection and emphasized the importance of the number of training epochs in improving model performance, as byt5 learns cleaner and finer representations. since tdl models are effective at handling complex dependencies and heterogeneity in tabular data, and modern embedding models have improved the quality of vector representations, introducing new methods for generating high-quality contextual embeddings the combination of these two can bring disruption to the hate speech detection scenario, and other nlp tasks. our research sets itself apart from previous methods by embarking on an initial investigation to address the lack of studies that integrate modern embedding models, embedding generation techniques, and tdl for detecting toxic and hateful comments on social media in brazilian portuguese. we emphasize the significance of testing different types of modern embeddings in the process of evaluating our classification strategy. tdl and modern embeddings are the core of our proposed method, which leverages the advantages of both in an attempt to achieve better outcomes. 3 proposed method in this section, we propose a methodological approach for the binary classification of toxic and hateful comments on social media using tabular deep learning (tdl). the developed approach applies the ft-transformer (feature tokenizer + transformer or ftt) model [13], slight adjustments on the base ftt model, integration to pytorch frame architecture for tabular data processing while incorporating language-specific embeddings for brazilian portuguese. figure 1 illustrates a five-step development process of this tdl approach, which includes: 1) data preparation; 2) embedding generation; 3) model training; 4) evaluation; and 5) prediction. it is summarized as follows: fig. 1. a step-by-step process to detect toxic and hateful comments. full size image 1) data preparation: the initial step involves gathering and normalizing the data to ensure it is suitable for training and has a consistent input format. this preparation removes unwanted data columns, shapes the data, and normalizes the text with enelvo [3]. 2) embedding generation: this step entails the textual data conversion into vector representations using a text embedder (te), which is an lm-based embedding generation tool. the output representation matches the format and dimension from the input data and the embedding size of the model. to format the te output into a tensor-friendly format suited for tdl pipelines, the materialization step is required, where te pre-encodes the text data before being shaped into a final tensor. the characteristics of the input text data and the language model used as an encoder in te determine the final tensor shape. 3) model training: the processed data is then fed into the ft-transformer model, which is subsequently trained. the ft-transformer has two main components: the feature tokenizer (ft) and the transformer. since the original ft cannot ensure smooth operation and proper encoding of text data types in the dataset, this approach relies on its enhanced version reformulated by hu et al. [15], stypeencoder. after properly handling and processing each data type, the features are concatenated into a dense vector and passed to the transformer component as training input data. 4) evaluation: after training, the model is evaluated using metrics such as f1-score and accuracy. this assessment helps in understanding the effectiveness of the approach in correctly detecting toxic comments while minimizing false positives and negatives. 5) prediction: finally, the trained model generates predictions on a separate test set. this step validates the model’s capability to adapt and perform effectively on unseen data, demonstrating its robustness and reliability for other text classification tasks and real-world applications. by following this structured approach, we aim to harness the capabilities of the ft-transformer and advanced embedding models to improve the detection of toxic and hateful speech in social media contexts. note that there is an intrinsic challenge in selecting suitable embedding models for tasks like detecting toxicity and hate speech in portuguese comments since language models can introduce noise and impair the quality of the embeddings, leading to inferior performance. so, to accommodate the language domain and its nuances, this approach also employs reliable portuguese language models as generative embedding models (or language embedders). 4 experiments and results this section focuses on detecting toxic and hateful content in social media comments through binary text classification. the experimental scenarios described in this study outline the approaches used to conduct our research and assess the effectiveness of our proposed solution. our methodology is tested using the told-br [17] dataset, which consists of 21k social media comments. to ensure consistency and facilitate fair comparison with other methods, we divided the data into 80% for training, 10% for validation, and 10% for testing, in line with the original experiment. the selected model parameters for this experiment are listed in table 1. alongside using the adamw optimizer, a scheduler was implemented to gradually reduce the learning rate from its starting value to zero during training. table 1. model parameters.full size table table 2. list of embedding models.full size table the text embedder uses different models in each training round, these are: i) sbert: e5-large [40] and sbertimbau-largefootnote 1; ii) bert-based: bertimbau [35], deberta-v2-xl [14], and albertina pt-br [30]; and llm embedding: voyageai (voyage-large-2) and openai (text-embedding-3-large). table 2 details these models, including their language support and the output embedding dimensions. the experiments performed in our study follow the least strict annotator agreement scenario described by leite et al. [17]. in this agreement scenario, the dataset contains 11,745 non-toxic and 9,255 toxic comments, with a ratio in the class distribution of 1:1.2, which makes the told-br dataset can still be considered fairly balanced. after completing training rounds for each embedding model, a performance assessment can be done based on the obtained metrics. table 3 displays f1-score for both toxic and non-toxic classes, as well as the overall accuracy of the chosen te. table 3. metrics evaluation on different text embedders.full size table table 4. toxic and hateful speech detection: comparison of approaches using told-br.full size table based on these results, the openai-te3-large model has the highest accuracy and f1-score for the toxic class, making it the most robust choice. bertimbau and sbertimbau are strong performers among monolingual models, with bertimbau being effective for the non-toxic class and sbertimbau performing better for the toxic class. multilingual embedding models like openai-te3-large, me5large, and voyagelarge2 show excellent performance. table 4 compares the results of different developed approaches used for detecting toxic and hateful speech using the told-br dataset and compares the overall f1-score and accuracy of each method. the results show that the best-performing approaches for detecting hate and toxicity in social media comments involve extensive fine-tuning, transfer learning, or zero-shot learning, like bertimbau large and m-bert as classifiers, and prompting gpt 3.5-turbo. however, methods that use graph representations, static embeddings, and modern embedding models can also achieve strong results even without additional tuning, providing a baseline for future improvements or hybrid implementations, and that was shown with our approach in the openai-te3-large + ftt setting. the confusion matrix depicted in fig. 2 serves as a subsequent assessment of the proposed approach, providing additional insights into the performance of the model over the classes (toxic and non-toxic). fig. 2. confusion matrix for toxic vs. non-toxic classification. full size image the confusion matrix reveals that the model exhibits a robust ability to accurately identify non-toxic comments, as indicated by the high number of true negatives. additionally, the model demonstrates a moderate capacity to detect toxic comments, with 778 true positives. however, the results also indicate notable misclassifications, with 301 false positives and 194 false negatives, suggesting room for improvement in reducing misclassification rates. in summary, the model has an overall accuracy of 76.43%, with a good balance between precision (72.08%) and recall (80.08%) for the toxic class. our transformer-based tabular approach is available at https://github.com/ghivvagodamas/tabular-transformer-toxic2024. 5 conclusion and future work as previously demonstrated, we have proposed a novel approach for binary text classification, which is specifically tailored and trained for structured data in tabular format. this approach produced compelling results, successfully detecting hate speech and toxicity without complex fine-tuning or transfer learning methods. it yields an intriguing accuracy rate of 76% and an f1-score of 75% on the test set using the openai text-embedding-3-large model as the text embedder. before the experiments, told-br underwent an additional review and evaluation, highlighting concerns about its validity and reliability. issues were identified with the imbalance ratio in annotator agreement scenarios, possibly due to bias in the annotation process, term resignification, and the stylistic use of slurs. tabular deep learning models, like ft-transformer, excel in multimodal learning and structured heterogeneous data processing with minimal adjustments. while the use of a gpu is important for this approach and implementations of other tdl models, no extensive computational resources are required. this study emphasizes the importance of tdl and embedding models in detecting toxic and hateful speech. since modern embedding models have demonstrated enhanced processing efficiency and improved outcomes in different nlp applications [20], incorporating those in the proposed approach is not only adequate but also the most intelligent and assertive choice. for future work, we intend to investigate more about modern embedding models like bge m3 (flagembeddings) [7], sbert, and how to convert state-of-the-art llms like sabiá [26], aya [36], and other generative models into powerful and robust embedding models with refined embedding space and deep contextual understanding. our intentions include adapting, and developing other tdl-based strategies and broadening nlp applicability, investigating bias detection and mitigation techniques similar to the study of nascimento et al. [21], and model performance enhancement strategies such as incorporating additional contextual features, contrastive learning, and retrieval-augmented generation (rag). notes 1.https://huggingface.co/rufimelo/bert-large-portuguese-cased-sts. references almeida, t.g., souza, b.à., nakamura, f.g., nakamura, e.f.: detecting hate, offensive, and regular speech in short comments. in: proceedings of the 23rd brazillian symposium on multimedia and the web (2017) google scholar  assis, g., amorim, a., carvalho, j., de oliveira, d., vianna, d., paes, a.: exploring portuguese hate speech detection in low-resource settings: lightly tuning encoder models or in-context learning of large models? in: proceedings of the 16th international conference on computational processing of portuguese, acl (2024) google scholar  bertaglia, t.f.c., nunes, m.d.g.v.: exploring word embeddings for unsupervised textual user-generated content normalization. in: proceedings of the 2nd workshop on noisy user-generated text (2016) google scholar  bispo, t.d.: arquitetura lstm para classificação de discursos de ódio cross-lingual inglês-ptbr. master’s thesis, universidade federal do sergipe (2018) google scholar  brown, t., et al.: language models are few-shot learners. adv. neural inf. process. syst. 33 (2020) google scholar  burnap, p., williams, m.l.: us and them: identifying cyber hate on twitter across multiple protected characteristics. epj data sci. 5 (2016) google scholar  chen, j., xiao, s., zhang, p., luo, k., lian, d., liu, z.: bge m3-embedding: multi-lingual, multi-functionality, multi-granularity text embeddings through self-knowledge distillation (2023) google scholar  chen, y., zhou, y., zhu, s., xu, h.: detecting offensive language in social media to protect adolescent online safety. in: 2012 international conference on privacy, security, risk and trust and 2012 international conference on social computing. ieee, amsterdam (2012) google scholar  chopra, a., sharma, d.k., jha, a., ghosh, u.: a framework for online hate speech detection on code-mixed hindi-english text and hindi text in devanagari. acm trans. asian low-resource lang. inf. process. 22(5) (2023) google scholar  de pelle, r.p., moreira, v.p.: offensive comments in the brazilian web: a dataset and baseline results. in: anais do vi brazilian workshop on social network analysis and mining, sbc (2017) google scholar  fortuna, p., rocha da silva, j., soler-company, j., wanner, l., nunes, s.: a hierarchically-labeled portuguese hate speech dataset. in: proceedings of the third workshop on abusive language online, acl (2019) google scholar  fortuna, p., soler-company, j., wanner, l.: how well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets? inf. process. manag. 58(3) (2021) google scholar  gorishniy, y., rubachev, i., khrulkov, v., babenko, a.: revisiting deep learning models for tabular data. adv. neural inf. process. syst. 34 (2021) google scholar  he, p., liu, x., gao, j., chen, w.: deberta: decoding-enhanced bert with disentangled attention. in: international conference on learning representations (2021) google scholar  hu, w., et al.: pytorch frame: a modular framework for multi-modal tabular learning. arxiv preprint arxiv:2404.00776 (2024) khanuja, s., et al.: muril: multilingual representations for indian languages. arxiv preprint arxiv:2103.10730 (2021) leite, j.a., silva, d., bontcheva, k., scarton, c.: toxic language detection in social media for brazilian portuguese: new dataset and multilingual analysis. in: proceedings of the 1st conference of the asia-pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing, acl (2020) google scholar  malmasi, s., zampieri, m.: challenges in discriminating profanity from hate speech. j. exp. theor. artif. intell. 30(2) (2018) google scholar  mandal, a., roy, g., barman, a., dutta, i., naskar, s.k.: attentive fusion: a transformer-based approach to multimodal hate speech detection. arxiv preprint arxiv:2401.10653 (2024) muennighoff, n., tazi, n., magne, l., reimers, n.: mteb: massive text embedding benchmark. in: proceedings of the 17th conference of the european chapter of the association for computational linguistics, acl, dubrovnik (2023) google scholar  nascimento, f.r., cavalcanti, g.d., da costa-abreu, m.: unintended bias evaluation: an analysis of hate speech detection and gender bias mitigation on social media using ensemble learning. exp. syst. appl. 201 (2022) google scholar  nobata, c., tetreault, j., thomas, a., mehdad, y., chang, y.: abusive language detection in online user content. in: proceedings of the 25th international conference on world wide web, international world wide web conferences steering committee, montréal (2016) google scholar  ocampo, n.b., sviridova, e., cabrio, e., villata, s.: an in-depth analysis of implicit and subtle hate speech messages. in: vlachos, a., augenstein, i. (eds.) proceedings of the 17th conference of the european chapter of the association for computational linguistics, pp. 1997–2013. association for computational linguistics, dubrovnik (2023) google scholar  oliveira, a., cecote, t., silva, p., gertrudes, j., freitas, v., luz, e.: how good is chatgpt for detecting hate speech in portuguese? in: anais do xiv simpósio brasileiro de tecnologia da informação e da linguagem humana, sbc, belo horizonte/mg (2023) google scholar  pelle, r., alcântara, c., moreira, v.p.: a classifier ensemble for offensive text detection. in: proceedings of the 24th brazilian symposium on multimedia and the web (2018) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models. in: intelligent systems. springer, cham (2023). isbn 978-3-031-45392-2 google scholar  rawat, a., kumar, s., samant, s.s.: hate speech detection in social media: techniques, recent trends, and future challenges. wiley interdiscip. rev.: comput. statist. 16(2) (2024) google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 3982–3992. association for computational linguistics, hong kong (2019) google scholar  da rocha junqueira, j., junior, c.l., silva, f.l.v., côrrea, u.b., de freitas, l.a.: albertina in action: an investigation of its abilities in aspect extraction, hate speech detection, irony detection, and question-answering. in: anais do xiv simpósio brasileiro de tecnologia da informação e da linguagem humana, sbc (2023) google scholar  rodrigues, j.a., et al.: advancing neural encoding of portuguese with transformer albertina pt-*. in: progress in artificial intelligence: 22nd epia conference on artificial intelligence, epia 2023, faial island, 5–8 september 2023, proceedings, part i, pp. 441–453. springer, berlin (2023). isbn 978-3-031-49007-1 google scholar  rodríguez, s.e., allende-cid, h., allende, h.: detecting hate speech in cross-lingual and multi-lingual settings using language agnostic representations. in: progress in pattern recognition, image analysis, computer vision, and applications: 25th iberoamerican congress, ciarp 2021. springer (2021) google scholar  saraiva, g.d., anchiêta, r., neto, f.a.r., moura, r.: a semi-supervised approach to detect toxic comments. in: proceedings of the international conference on recent advances in natural language processing (ranlp 2021), incoma ltd., held online (2021) google scholar  silva, s., serapiao, a.: detecção de discurso de ódio em português usando cnn combinada a vetores de palavras. in: proceedings of kdmile 2018, symposium on knowledge discovery, mining and learning, são paulo (2018) google scholar  da silva oliveira, a., de carvalho cecote, t., alvarenga, j.p.r., de souza freitas, v.l., da silva luz, e.j.: toxic speech detection in portuguese: a comparative study of large language models. in: proceedings of the 16th international conference on computational processing of portuguese, acl (2024) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, 20–23 october (to appear) (2020) google scholar  üstün, a., et al.: aya model: an instruction finetuned open-access multilingual language model. arxiv preprint arxiv:2402.07827 (2024) utku, a., can, u., aslan, s.: detection of hateful twitter users with graph convolutional network model. earth sci. inf. 16(1) (2023) google scholar  vargas, f., carvalho, i., rodrigues de góes, f., pardo, t., benevenuto, f.: hatebr: a large expert annotated corpus of brazilian instagram comments for offensive language and hate speech detection. in: proceedings of the thirteenth language resources and evaluation conference, european language resources association (2022) google scholar  walther, j.b.: social media and online hate. curr. opin. psychol. 45 (2022) google scholar  wang, l., yang, n., huang, x., yang, l., majumder, r., wei, f.: multilingual e5 text embeddings: a technical report. arxiv preprint arxiv:2402.05672 (2024) wang, x., koneru, s., venkit, p.n., frischmann, b., rajtmajer, s.: the unappreciated role of intent in algorithmic moderation of social media content. arxiv preprint arxiv:2405.11030 (2024) waseem, z., hovy, d.: hateful symbols or hateful people? predictive features for hate speech detection on twitter. in: proceedings of the naacl student research workshop (2016) google scholar  yin, w., zubiaga, a.: towards generalisable hate speech detection: a review on obstacles and solutions. peerj comput. sci. 7 (2021) google scholar  younus, a., qureshi, m.a.: a framework for sexism detection on social media via byt5 and tabnet (2022) google scholar  download references author information authors and affiliations federal university of piauí, teresina, brazil ghivvago damas, raimundo santos moura & vinicius ponte machado federal institute of piauí, teresina, brazil rafael torres anchiêta authors ghivvago damasview author publications search author on:pubmed google scholar rafael torres anchiêtaview author publications search author on:pubmed google scholar raimundo santos mouraview author publications search author on:pubmed google scholar vinicius ponte machadoview author publications search author on:pubmed google scholar corresponding author correspondence to ghivvago damas . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper damas, g., torres anchiêta, r., santos moura, r., ponte machado, v. (2025). a transformer-based tabular approach to detect toxic comments. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_2 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords toxic and hateful speech deep learning ft-transformer embedding models text classification publish with us policies and ethics profiles rafael torres anchiêta view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature time series forecasting of covid-19 cases in brazil with gnn and mobility networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper time series forecasting of covid-19 cases in brazil with gnn and mobility networks conference paper first online: 12 october 2023 pp 361–375 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) time series forecasting of covid-19 cases in brazil with gnn and mobility networks download book pdf download book epub fernando henrique oliveira duarte  orcid: orcid.org/0009-0003-0228-07129, gladston j. p. moreira  orcid: orcid.org/0000-0001-7747-59269, eduardo j. s. luz  orcid: orcid.org/0000-0001-5249-15599, leonardo b. l. santos  orcid: orcid.org/0000-0002-3129-772x10 & … vander l. s. freitas  orcid: orcid.org/0000-0001-7989-08169  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 567 accesses 4 citations 1 altmetric abstract in this study, we examine the impact of human mobility on the transmission of covid-19, a highly contagious disease that has rapidly spread worldwide. to investigate this, we construct a mobility network that captures movement patterns between brazilian cities and integrate it with time series data of covid-19 infection records. our approach considers the interplay between people’s movements and the spread of the virus. we employ two neural networks based on graph convolutional network (gcn), which leverage spatial and temporal data inputs, to predict time series at each city while accounting for the influence of neighboring cities. in comparison, we evaluate lstm and prophet models that do not capture time series dependencies. by utilizing rmse (root mean square error), we quantify the discrepancy between the actual number of covid-19 cases and the predicted number of cases by the model among the models. prophet achieves the best average rmse of 482.95 with a minimum of 1.49, while lstm performs the least despite having a low minimum rmse. the gcrn and gclstm models exhibit mean rmse error values of 3059.5 and 3583.88, respectively, with the lowest standard deviation values for rmse errors at 500.39 and 452.59. although the prophet model demonstrates superior performance, its maximum rmse value of 52,058.21 is ten times higher than the highest value observed in the graph convolutional networks (gcns) models. based on our findings, we conclude that gcns models yield more stable results compared to the evaluated models. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others improving prediction of covid-19 evolution by fusing epidemiological and mobility data article open access 26 july 2021 a deep learning model for early prediction of covid-19 spread chapter © 2022 forecasting covid-19 cases in india using multivariate hybrid cnn-lstm model chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational social science, social networks covid19 machine learning predictive medicine probabilistic data networks stochastic networks 1 introduction the covid-19 is an infectious disease that manifests itself in humans infected by the sars-cov-2 virus. the acronym sars stands for severe acute respiratory syndrome, which belongs to the family of coronaviruses who have been infecting humans for a long time. on december 31, 2019, the world health organization (who) alerted to a new strain (type) of coronavirus. on january 7, 2020, chinese authorities confirmed the identification of a new type of coronavirus, that was initially named 2019-ncov, and on february 11, 2020, it was given the name sars-cov-2 [15, 28]. those infected transmit the virus through the mouth or nose, by expelling small particles when they cough, sneeze, talk, sing or breathe. infection occurs by inhaling these particles through the airways or by touching a contaminated surface and then coming into contact with the eyes, nose or mouth [26]. due to the nature of the virus, it spreads more easily indoors and in crowds. those who become ill as a result of covid-19 experience symptoms ranging from mild and moderate, where the patient recovers without special treatment, to severe, where the patient needs specialized medical care [2]. to this date (may 06, 2023), the virus that causes covid-19 has infected more than 270 million people and caused more than 5.3 million deaths worldwide [5, 19]. in brazil, the numbers reach more than 37 million infected and 701 thousand dead [1]. so far, the world has paid a heavy price for this pandemic in lost human lives, economic repercussions and increased poverty. the covid-19 pandemic is still ongoing, and the virus continues to mutate, generating variants. this work presents time series forecasting models to predict the number of covid-19 cases in brazilian cities [4], covering the period from february 2020 to december 2022. we aim to improve the accuracy of the predictions by incorporating graph convolutional networks (gcn) models, specifically gcrn and gclstm. these models leverage a mobility network that captures the commuting patterns between cities, allowing us to account for the interdependencies among cities’ time series and the influence of human mobility on disease transmission [7,8,9, 20]. the performance of our proposed gcn models is compared to two well-known forecasting methods, lstm and prophet. through extensive experiments and evaluation, we obtained compelling results that highlight the effectiveness of the gcn models approach in capturing the complex dynamics of covid-19 transmission and improving the accuracy of predictions. this research contributes to the field of covid-19 forecasting by demonstrating the value of incorporating mobility networks and considering the impact of human movement on disease spread. our findings provide insights into the interplay between mobility patterns and the spread of covid-19, and offer valuable implications for developing effective strategies to mitigate the impact of the disease. 2 related works xi et al. (2022) [27] simulate the propagation curve of covid-19 in a university, based on the interaction between students through graph neural network (gnn) and a sir model (susceptible infected recovered) [23]. they simulated three distancing strategies designed to keep the infection curve flat and help make the spread of covid-19 controllable. students and professors were considered as nodes and places or events as layers (face-to-face lectures, internal social activities, campus cafeterias, etc.). the gnn was used to validate and verify the effectiveness through the simulation of the infection curve, where the infected were randomly chosen. they provide a visualization of the dissemination process of covid-19 and the comparison between the results of the forecast of dissemination for each of the control strategies. with the aim of predicting the dynamics of the spread of the pandemic in the usa, the work carried out by davahli et al. (2021) [6] developed two predictive models using gnn, fed by real data, obtaining results from the forecast of spread and a comparison between the models. the first model developed was based on graph theory neural networks (gtnn) and the second was based on neighborhood neural networks (ngnn). each us state is considered a graph node, while the edges of the gtnn model document the functional connectivity between states, those of the ngnn model connect neighboring states with each other. the performance of these models was evaluated with existing real dataset that reflect conditions from january 22 to november 26, 2020 (prior to the start of covid-19 vaccination in the united states). the results indicate that the gtnn model outperforms the ngnn model. malki et al. (2020) [17] estimate the spread of covid-19 infection in several countries, using a supervised machine learning model developed based on a decision tree algorithm and linear regression, for time series forecasting. they show an \(r^2\) of 0.99 in the prediction of confirmed cases at the global level. vaishya et al. (2020) [25] deal with the application of artificial intelligence (ai) in combating the covid-19 pandemic. the authors presented several possibilities for the use of ai in different areas, such as diagnosis, triage, patient monitoring, forecasting disease trends, drug and vaccine development, and even to ensure compliance with social distancing measures. they emphasize the importance of international collaboration to ensure access to the data needed to train ai algorithms and highlight that technology can play a key role in the fight against the pandemic. the related works presented in this section are all focused on predicting the spread of covid-19 using different approaches. xie et al. (2022) used a graph neural network (gnn) and sir model to simulate the propagation curve of covid-19 in a university, while davahli et al. (2021) used gnn to predict the dynamics of the spread of the pandemic in the usa. malki et al. (2020) used a decision tree algorithm and linear regression for time series forecasting, and vaishya et al. (2020) discussed the application of ai in combating the pandemic. in contrast, our study focuses on incorporating a mobility network that describes the commuting patterns between brazilian cities into the forecasting task, leveraging graph convolutional network (gcn) models. the main hypothesis is that this approach could improve the forecasting task by accounting for the dependencies among cities’ time series. therefore, our work contributes to the literature by emphasizing the importance of incorporating human mobility data into the forecasting task and highlighting the potential of gcn models for this purpose. 3 methodology 3.1 mathematical preliminaries \(g=(v,e)\) is a graph, in which v is the set of nodes and e the set of edges, resulting in \(n=|v|\) vertices and |e| edges. the binary (or weighted) adjacency matrix \(a \in \mathbb {r}^{n \times n}\) depicts the graph connections; \(d_{ii} = \sum _j a_{ij}\) is the degree matrix, with zeros outside the main diagonal; and \(x \in \mathbb {r}^{n \times c}\) is the feature vector of each vertex, where c is the number of vector dimensions. 3.2 gcn the graph neural networks (gnn) models allow us to extend the methods of neural networks, in order to process the data represented in the graph domain [21]. gnn models can process most graphs being some of them acyclic, cyclic, directed and undirected, or implement a particular function. the gcn proposed by [18] applies a convolution operation to each vertex of the graph. the approach has three steps: (i) select a vertex, (ii) build a subgraph of its neighborhood and finally (iii) normalize the selected subgraph with the edge weights of neighboring vertices. repeat the process for each vertex \(h^{k}\), building the matrix \(h^{k}\) that contains all the vertices in the layer k (refer to fig. 2 and eq. (1)). the subgraph is built using the adjacency matrix a and the diagonal degree matrix of the vertices d. the size of the subgraph is delimited by the parameter k (see fig. 1). when \(k=1\) the vertex is not subjected to the influence of neighboring vertices. in the case of \(k=2\), first-degree vertices (direct neighbors) are considered, directly connected to the active vertex. for a \(k=3\), second-degree vertices are also considered, connected to first-degree vertices. the subgraph is normalized by the weight matrix. fig. 1. nearest neighbors parameter (k) in a graph. in the purple color, \(k=1\) represents nodes without any neighbors. in the orange color, \(k=2\) represents nodes with first-degree neighbors, and in the red color, \(k=3\) represents nodes with second-degree neighbors. (color figure online) full size image when considering the subgraph of a node, we calculate the average feature vector by incorporating information from its neighbors. this process assigns greater importance to the less connected neighbors (lower degree) while also giving higher weight to the node’s connections. let \(\widehat{a} = a + i_n\), a being the weighted adjacency matrix of the subgraph, \(i_n\) being the identity matrix of order n, and \(\widehat{d}\) is the diagonal degree matrix of \(\widehat{a}\). the output of the convolution layer \(k+1\) is as follows: $$\begin{aligned} h^{k+1} = \sigma (\widehat{d}^{-1/2}\widehat{a}\widehat{d}^{-1/2}h^{k}w^{k+1}), \end{aligned}$$ (1) where w is the matrix of trainable parameters, and \(\sigma \) is the activation function. in summary, each vertex in the graph is assigned a feature vector and convolution is applied at each node to compute a new feature vector that considers the neighborhood of the node. 3.3 gcn models for time series prediction the graph convolutional recurrent networks (gcrn) [22] and graph convolutional lstm (gclstm) [3] models are architectures based on gcn models that are designed with graph processing and machine learning techniques for the purpose of prediction of data in time series. the convolutions in both architectures differ from a conventional convolution using the relationship between neighbors at each vertex of the graph. to predict a future sequence, of length f, we use a sequence, of length l (lags), of observed data at previous timestamps: $$\begin{aligned} \hat{x}_{t+1}, ... , \hat{x}_{t+f} = \genfrac{}{}{0.0pt}{}{arg max}{x_{t+1}, ... , x_{t+f} } p(x_{t+1}, ... , x_{t+f} | x_{t-l+1}, ... , x_{t}) \end{aligned}$$ (2) where \(x_{t}\) is a snapshot at time t of the features and \(\hat{x}_{t}\) is the prediction (see fig. 2). hence, \(x_{t}\) represents a value of the time series at time t for all nodes, defined on a directed, weighted static graph. on the other hand, a snapshot x can be interpreted as a matrix \(x \in \mathbb {r}^{n \times c}\), which captures the temporal evolution of the time series over the interval of c observations. fig. 2. spatial and temporal snapshot data flow and model architecture: processing gcns with gclstm and gcrn recurrences full size image the gcrn and gclstm models use the same design but with their own convolution blocks: in fig. 2, the gcrn model uses gconvgru blocks whereas the gclstm model uses gconvlstm blocks. both use deep learning techniques, but gcrn uses recurrent networks and gclstm uses lstm cells, in order to capture patterns in time series data considering the influence of its neighbors through the graph. the graph generated in this work has the spatial structure of the brazilian mobility network (refer to sect. 3.5), which has static edges, that is, fixed attributes (see fig. 2). 3.4 prophet and lstm lstm (long short term memory) [12] and prophet [24] are time series forecasting models, and each one has a different approach. in the lstm model it resembles an rnn (recurrent neural network) composed of 4 memory cells: input, forget, cell, hidden. these memory cells are responsible for retaining, forgetting or remembering selected information. lstm is commonly used on data that exhibit complex patterns, sudden changes, trends, or seasonality. the prophet model, is based on an additive model that, through regression, models the linear or non-linear trend, in terms of independent variables such as trend, seasonality and holidays, components of the time series. the adjustments of these components are performed using an optimization algorithm such as mcmc [11] (markov chain monte carlo) or map [10] (maximum a posteriori) that seeks to find the values of the parameters that maximize the posterior probability of the data, considering the uncertainty of the parameters when making predictions. 3.5 datasets: covid-19 cases and the brazilian mobility network cota (2020) [4] provides a public dataset of covid-19 records in brazil. among other data, it contains the numbers of infected and deaths related to municipal and federative units level in brazil with official information from the ministry of health. the dataset is daily updated from official sources, providing reliable information. it also includes information on the cities’ locations, which is useful for our mobility network. knowing that the spread of diseases like covid-19 occurs from infected to susceptible people, one factor that may contribute to the construction of prediction models is to know the flow of people between cities. we thus use the dataset provided by the brazilian institute of geography and statistics (ibge) [16] to build a mobility network whose weighted connections represent the number of vehicles from one city to another in an ordinary week. the connection identifies the cities directly accessible from a source, considering (1) the number of outgoing vehicles (frequency), (2) the type of transport (road, waterways, or alternative/informal), (3) travel time, and (4) cost of the traveling. in this study, the weight of each edge was represented solely by attribute 1. this weight was determined by the aggregation of frequencies associated with each mode of transportation (denoted as attribute 2) present on a given edge. it should be noted that attributes 3 and 4 were not incorporated into the calculations in this analysis. the dataset has \(n = 5385\) vertices (cities) and \(l = 65639\) edges (routes between the cities), as shown in fig. 3. the subfigure on the right side gives the covered territory, in green, and the one on the left side presents the corresponding network. fig. 3. brazilian mobility network: a) nodes representing cities and edges as the routes between them; b) the covered territory, in green. white areas inside the brazilian map are cities without reported cases in our covid-19 dataset. (color figure online) full size image 3.6 experimental setup the covid-19 dataset has been structured to train models under four distinct scenarios: i) utilizing only the data from 2020; ii) using the 2021 data exclusively; iii) relying solely on the 2022 data; and finally, iv) a comprehensive version incorporating all years. the data undergoes standardization using z-score normalization (eq. (3)) to enable consistent and meaningful analysis. $$\begin{aligned} z = \frac{(x -\mu )}{\sigma } \end{aligned}$$ (3) here, x represents the value of the element, \(\mu \) is the population mean, and \(\sigma \) is the population standard deviation. this normalization process ensures uniformity in the data and facilitates reliable comparisons. in each of the four scenarios, the dataset is partitioned into an 80% subset for training and a 20% subset designated for testing. each city has 1,009 individual readings of covid-19 case numbers, with each reading representing a single day’s data. these readings are grouped into sequences of 14 consecutive days (lags), which are utilized to predict the data for the 15th day. consequently, each snapshot corresponds to a time series comprised of non-overlapping samples, each size 14. the subsequent day, the 15th, is the target prediction and assumes the first position in the succeeding time series (see fig. 4). the entire 1009-day period, corresponding to february 2020 to november 2022, results in approximately 57 training snapshots and 15 test snapshots. lauer et al. [14] provide empirical evidence supporting the recommendation of a 14-day active monitoring period by the us centers for disease control and prevention. their study reinforces the importance of this time frame in effectively monitoring and controlling the spread of covid-19. regarding the training of the models is parameterized for a total of 100 epochs for the gcrn, gclstm, and lstm models. in these models, an adam optimizer [13] is used, which adjusts the model’s weights during training, minimizing the mean square error loss. fig. 4. mobility network map and spatial time series data in brazil: input dataset overview full size image as adam is a stochastic gradient descent optimizer, it has an element of randomness, so we perform 10 rounds of experiments for each model. the adam optimizer was chosen because it uses two moments to update the model weights: the gradient moment and the quadratic gradient moment [13]. at the gradient moment, it adapts the learning rate for each parameter based on how often the parameter is updated. not all cities update cases daily, so the data does not have a unified update frequency. in the quadratic gradient moment, the weighting of the learning rate is performed by the moving average of the squares of the gradients of each parameter individually, adjusting the model weights. to present and compare the results of the tests between the neural networks, we use the root mean square error (rmse) and the \(r^2\) score (coefficient of determination). the rmse, $$\begin{aligned} rmse = sqrt(\frac{1}{n}\sum _{i=1}^{n}(y_i-\hat{y}_i)^2), \end{aligned}$$ (4) is a measure of dispersion that calculates the square root of mean square errors between predictions and actual values, i.e., it indicates how well the model’s predictions fit the data. the lower the rmse, the better the model performs against the test data. the \(r^2\) score, $$\begin{aligned} r^2=1-\frac{\sum ({y_i}-\hat{y_i})^2}{\sum (y_i-\bar{y})^2}, \end{aligned}$$ (5) indicates how much the independent variables explain the variation of the dependent variable. it belongs to the interval [0, 1], where 1 indicates that all variations in the dependent variable are explained by the independent variables, and it is 0 when there is no relationship between the variables. therefore, the closer to 1, the better the model. 4 results the tables 1 and 2 have references to those years, which we will comment on. as we train gcrn, gclstm, and lstm models 10 times each, the results are presented in terms of mean, max, min, and std values of rmse and \(r^2\). table 1. gcrn model results in terms of rmse and \(r^2\) scorefull size table 4.1 gcrn the results of gcrn (see table 1) in 2020, the first year of the covid-19 pandemic, are: \(rmse = 1495.18\), while in 2021 and 2022, the values decreased significantly to 938.39 and 819.62, respectively. in the last two years, the mean value error became smaller, but the number of cases continued to grow. the highest min error value was recorded in 2020 and the lowest min error value was recorded in 2021, with values of 1364.51 and 643.26, respectively, while in 2022 the max error value decreased to 1034.46 even with a greater variation than in the previous years. note that the standard deviation in the years 2020 and 2022 are close, and in the year 2021 it doubles in relation to them. these factors are indicators that in the year 2021 there was a change in the dynamics of spreading the disease. the highest rmse for the three years combined suggests potential variations in the dynamics of covid-19 spread over time, which could be attributed to various factors influencing the transmission patterns. these factors may include governmental interventions, changes in population behavior, the emergence of new variants, variations in testing strategies, healthcare capacity, public adherence to preventive measures, and other external factors that could impact the transmission dynamics of the virus. regarding the \(r^2\) score, the models showed low variation in the standard deviation and a minimum value of 0.96 for 2020, which still represents a high prediction capacity of the model. figure 5b gives a visualization of the model predictions on the training set, with most points laying on the main diagonal as the high \(r^2\) values suggest. figure 5a presents a forecasting example, for november 20, with the model trained with all years. we highlight sao paulo, manaus, and brasilia as being state capitals in different regions of brazil. fig. 5. gcrn trained with the dataset of all years (2020 to 2022): a) result of the predicted and true number of cases in each city on november 29, 2022. the true values of manaus, sao paulo and brasilia are highlighted with red crosses; b) linear regression between true and predicted values. (color figure online) full size image 4.2 gclstm table 2 introduces the results for the gclstm model. as in the gcrn: the average rmse for 2020 is the highest compared to 2021 and 2022; there is a decreasing trend from 2020 to 2022; and the error for all years is the largest. the \(r^2\) score also decreases over the years and is better than the one obtained for 2020. table 2. gclstm model results in terms of rmse and \(r^2\) scorefull size table the experiment is repeated, but for the gclstm model, the results are shown for the \(r^2\) score metric for table 2 in fig. 6b and also the results of prediction values compared to actual values for each city (see fig. 6a). fig. 6. gclstm trained with the dataset of all years (2020 to 2022): a) result of the predicted and true number of cases in each city on november 29, 2022. the true values of manaus, sao paulo, and brasilia are highlighted with red crosses; b) linear regression between true and predicted values. (color figure online) full size image 4.3 comparing forecast models results the lowest rmse values are highlighted on table 3, referring to the dataset of all years (from 2020 and 2022). note that the gcrn and gclstm models present similar results concerning mean, max, min, and standard deviation. interestingly, they achieved lower values of standard deviation, which shows their consistency. the prophet model stands out in the average and at the minimum, where it has the lowest values among the models. however, its standard deviation is 3 times greater than the lowest value found in the gclstm model and the maximum rmse is 14 times greater than the maximum observed for gcrn. the model is incredibly good for most data points, but really bad in some cases, showing less stability than gcrn and gclstm. the lstm model did not obtain any statistics highlighted as the best and can be considered the model with the worst performance, although its minimum rmse is the second best among all. table 3. rmse values of the forecast models of covid-19 cases in brazil.full size table the results show that the mobility network indeed helps to forecast covid-19 cases when embedded in machine learning models. lstm alone presents the worst rmse of all models, but when it is integrated with the gcn, via the gclstm, the predictions improve. although prophet gives the best mean rmse, it fluctuates more than the graph-based models, showing that sometimes it outperforms them, but is not as stable. 5 discussions and conclusions this studyfootnote 1 investigates the application of gcns for covid-19 spread in brazilian cities. combining graphs with temporal forecasting models offers advantages. integrating mobility networks, particularly gcns with lstm through gclstm, improves prediction accuracy over lstm alone. among the models evaluated, lstm performed the worst, with prophet showing the best mean rmse but higher fluctuations and less stability compared to the graph-based models (gcrn and gclstm). the gcn models achieved high \(r^2\) score values above 0.95, representing the data with at least 95% certainty, the models also demonstrated similar results in terms of mean, maximum, minimum, and standard deviation of rmse. notably, they achieved lower standard deviation values, indicating higher consistency in their predictions. in conclusion, the integration of mobility networks through gcns in machine learning models has proven effective in forecasting covid-19 cases. the gcn models (gcrn and gclstm) showed improved stability and consistency compared to prophet, while lstm alone exhibited the worst performance. the models’ performance highlights the importance of considering spatial and temporal data in forecasting tasks, providing valuable insights for epidemiological surveillance and informing strategies to combat covid-19. one limitation of our analysis is that it only focuses on one-day predictions. however, it is crucial to extend the prediction window to longer time frames, considering the virus’s incubation and transmission periods, as well as travel time between locations. the strength of incorporating mobility data lies in our approach using gcns, which enables us to capture the connectivity between cities and integrate changes in neighboring time series into the local predictions of each city. these changes in neighboring series have a time delay due to the incubation and transmission factors of the virus. by considering these factors and exploring longer prediction windows, we anticipate achieving more significant results in leveraging mobility networks for covid-19 case prediction. for future work, we will address the suggestion of conducting more in-depth analyses to assess the impact of including spatial data and applying gcns for longer-term predictions. additionally, we will investigate the convergence of the training process and explore ways to enhance the interpretability and explainability of the temporal and spatial data by the gcn models, particularly by examining the stabilization of the loss after 100 epochs, so that we can gain deeper insights into the underlying mechanisms driving the predictions of gcn models and enhance our understanding of the complex interplay between temporal and spatial factors in the spread of covid-19. also, considering that gcn-based models may require more training time (epochs) compared to prophet, further investigation into the impact of training duration on model performance could be valuable. lastly, exploring unexplored graph network regions is a promising research avenue. it can reveal covid-19 spread insights and uncover hidden data patterns or relationships. notes 1.all data and source code are available at the repository link: https://github.com/hodfernando/bracis_gcn_2023. references cumulative cases of covid-19 in brazil (2023). https://covid.saude.gov.br//. accessed 06 may 2023 berlin, d.a., gulick, r.m., martinez, f.j.: severe covid-19. n. engl. j. med. 383(25), 2451–2460 (2020) article  google scholar  chen, j., wang, x., xu, x.: gc-lstm: graph convolution embedded lstm for dynamic network link prediction. appl. intell. 52(7), 7513–7528 (2022) article  google scholar  cota, w.: monitoring the number of covid-19 cases and deaths in brazil at municipal and federative units level. scielo preprints (2020) google scholar  dasa: cumulative cases of covid-19 in the world (2023). https://dadoscoronavirus.dasa.com.br/. accessed 06 may 2023 davahli, m.r., fiok, k., karwowski, w., aljuaid, a.m., taiar, r.: predicting the dynamics of the covid-19 pandemic in the united states using graph theory-based neural networks. int. j. environ. res. public health 18(7), 3834 (2021) article  google scholar  fanelli, d., piazza, f.: analysis and forecast of covid-19 spreading in china, italy and france. chaos solitons fract. 134, 109761 (2020) article  mathscinet  math  google scholar  freitas, v.l., moreira, g.j., santos, l.b.: robustness analysis in an inter-cities mobility network: modeling municipal, state and federal initiatives as failures and attacks toward sars-cov-2 containment. peerj 8, e10287 (2020) article  google scholar  freitas, v.l.d.s., konstantyner, t.c.r.d.o., mendes, j.f., sepetauskas, c.s.d.n., santos, l.b.l.: the correspondence between the structure of the terrestrial mobility network and the spreading of covid-19 in brazil. cadernos de saúde pública 36, e00184820 (2020) google scholar  gauvain, j.l., lee, c.h.: maximum a posteriori estimation for multivariate gaussian mixture observations of markov chains. ieee trans. speech audio process. 2(2), 291–298 (1994) article  google scholar  gilks, w.r., richardson, s., spiegelhalter, d.: markov chain monte carlo in practice. crc press, boca raton (1995) book  math  google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  google scholar  kingma, d.p., ba, j.: adam: a method for stochastic optimization. in: bengio, y., lecun, y. (eds.) 3rd international conference on learning representations, iclr 2015, san diego, ca, usa, 7–9 may 2015, conference track proceedings (2015) google scholar  lauer, s.a., et al.: the incubation period of coronavirus disease 2019 (covid-19) from publicly reported confirmed cases: estimation and application. ann. intern. med. 172(9), 577–582 (2020) article  google scholar  li, q., et al.: early transmission dynamics in wuhan, china, of novel coronavirus-infected pneumonia. n. engl. j. med. 382(13), 1199–1207 (2020) article  google scholar  lulia, m.m.e.t., de oliveira, d.h.: instituto brasileiro de geografia e estatística-ibge (1936) google scholar  malki, z., et al.: the covid-19 pandemic: prediction study based on machine learning models. environ. sci. pollut. res. 28, 40496–40506 (2021) article  google scholar  niepert, m., ahmed, m., kutzkov, k.: learning convolutional neural networks for graphs. in: international conference on machine learning, pp. 2014–2023. pmlr (2016) google scholar  organization, w.h.: covid-19 weekly epidemiological update, 119 edn. (2022). accessed 23 nov 2022 google scholar  rothan, h.a., byrareddy, s.n.: the epidemiology and pathogenesis of coronavirus disease (covid-19) outbreak. j. autoimmun. 109, 102433 (2020) article  google scholar  scarselli, f., gori, m., tsoi, a.c., hagenbuchner, m., monfardini, g.: the graph neural network model. ieee trans. neural netw. 20(1), 61–80 (2008) article  google scholar  seo, y., defferrard, m., vandergheynst, p., bresson, x.: structured sequence modeling with graph convolutional recurrent networks. in: cheng, l., leung, a.c.s., ozawa, s. (eds.) iconip 2018. lncs, vol. 11301, pp. 362–373. springer, cham (2018). https://doi.org/10.1007/978-3-030-04167-0_33 chapter  google scholar  smith, d., moore, l., et al.: the sir model for spread of disease-the differential equation model. convergence (2004) google scholar  taylor, s.j., letham, b.: forecasting at scale. am. stat. 72(1), 37–45 (2018) article  mathscinet  math  google scholar  vaishya, r., javaid, m., khan, i.h., haleem, a.: artificial intelligence (ai) applications for covid-19 pandemic. diab. metabolic syndr. clin. res. rev. 14(4), 337–339 (2020) google scholar  velavan, t.p., meyer, c.g.: the covid-19 epidemic. trop. med. int. health 25(3), 278–280 (2020) article  google scholar  xie, h., li, d., wang, y., kawai, y.: visualization method for the spreading curve of covid-19 in universities using gnn. in: 2022 ieee international conference on big data and smart computing (bigcomp), pp. 121–128. ieee (2022) google scholar  zhu, n., et al.: a novel coronavirus from patients with pneumonia in china, 2019. n. engl. j. med. 382(8), 727–733 (2020) article  google scholar  download references acknowledgements the authors thank the brazilian agencies for research and development (cnpq, grants 441016/2020-0, 307151/2022-0, 308400/2022-4), (fapemig, grants apq-01518-21, apq-01647-22), (capes, grants 88887.506931/2020-00) and universidade federal de ouro preto (ufop). author information authors and affiliations universidade federal de ouro preto, ouro preto, mg, cep 35402-163, brazil fernando henrique oliveira duarte, gladston j. p. moreira, eduardo j. s. luz & vander l. s. freitas centro nacional de monitoramento e alertas de desastres naturais (cemaden), são josé dos campos, sp, cep 12247-016, brazil leonardo b. l. santos authors fernando henrique oliveira duarteview author publications search author on:pubmed google scholar gladston j. p. moreiraview author publications search author on:pubmed google scholar eduardo j. s. luzview author publications search author on:pubmed google scholar leonardo b. l. santosview author publications search author on:pubmed google scholar vander l. s. freitasview author publications search author on:pubmed google scholar corresponding author correspondence to fernando henrique oliveira duarte . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper duarte, f.h.o., moreira, g.j.p., luz, e.j.s., santos, l.b.l., freitas, v.l.s. (2023). time series forecasting of covid-19 cases in brazil with gnn and mobility networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_24 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords time series forecasting graph neural networks mobility networks covid-19 publish with us policies and ethics profiles gladston j. p. moreira view author profile eduardo j. s. luz view author profile vander l. s. freitas view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature legalbert-pt: a pretrained language model for the brazilian portuguese legal domain | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper legalbert-pt: a pretrained language model for the brazilian portuguese legal domain conference paper first online: 12 october 2023 pp 268–282 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) legalbert-pt: a pretrained language model for the brazilian portuguese legal domain download book pdf download book epub raquel silveira  orcid: orcid.org/0000-0001-7445-605x9, caio ponte  orcid: orcid.org/0000-0003-1643-947510, vitor almeida  orcid: orcid.org/0009-0005-3606-230410, vládia pinheiro  orcid: orcid.org/0000-0002-9851-830410 & … vasco furtado  orcid: orcid.org/0000-0001-8721-430810  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 656 accesses 19 citations abstract language models trained with bidirectional encoder representations from transformers (bert) have demonstrated remarkable results in various natural language processing (nlp) tasks. however, the legal domain poses specific challenges for nlp due to its highly specialized language, which includes technical vocabulary, formal style, frequent use of law citations and semantics based on vast knowledge. therefore, pretrained language models on a generic corpus may not be suitable for performing specific legal domain tasks. they lack the necessary expertise to understand the nuances of legal language, leading to inaccuracies and inconsistencies. this work describes the development of a specialized language model, legalbert-pt, for the legal domain in portuguese. the model was pretrained on a large and diverse corpus of brazilian legal texts and is now open-source and customizable for specific tasks. experiments were conducted to evaluate the pretrained model’s effectiveness in the legal domain, both intrinsically and in two specific tasks: named-entity recognition and text classification. the results indicate that using legalbert-pt outperforms the generic language model in all tasks, emphasizing the importance of specialization in achieving effective results for specific tasks in the legal domain. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others fostering judiciary applications with new fine-tuned models for legal named entity recognition in portuguese chapter © 2022 a study on the impact of intradomain finetuning of deep language models for legal named entity recognition in portuguese chapter © 2020 assessing european and brazilian portuguese llms for ner in specialised domains chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics language processing legal geography licensing lisp natural language processing (nlp) 1 introduction as legal documents become increasingly digitized, natural language processing (nlp) has gained importance for automating tasks in the legal field. nlp tools are now commonly used to address real-world legal problems, including the identification of participants in legal proceedings [24], the classification of legal documents [4], named-entity recognition [3], and legal text summarization [13]. these solutions employ traditional machine learning paradigms (e.g., tf-idf and classifiers) or deep learning techniques to achieve their objectives [28]. language models structured in the transformers architecture, such as bert [11] and its variants [7, 33], have achieved promising results in several downstream nlp tasks on generic reference datasets. in recent years, there have been some efforts to pretrain linguistic models for brazilian portuguese against more traditional word embeddings (word2vec, fasttext, etc.), as well as structured models in bert. however, these models were trained with general documents and were not designed to represent the brazilian legal language [28]. the effectiveness of language models on domain-specific tasks can be limited by the model’s lack of specialization for that domain. the development of language models for legal texts is justified since the language used for the preparation of legal documents has its own vocabulary, formal style, semantics based on a wide spectrum of knowledge, and frequent use of citations to laws. evidence shows that using pretrained language models with domain-specific corpus can significantly improve performance on domain-specific tasks [31, 37]. this work describes the production process of a language model for the legal domain in the portuguese language. the model was pretrained to acquire specialization for the domain, and later it could be adjusted for use in specific tasks. two versions of the model were created: one as a complement to the bertimbau model [33], and the other from scratch. the effectiveness of the model based on bertimbau was evident when analyzing the perplexity measure of the models. experiments were also carried out in the tasks of identifying legal entities and classifying legal petitions. the results show that the use of specific language models outperforms those obtained using the generic language model in the tasks studied, suggesting that the specialization of the language model for the legal domain is an important factor for improving the accuracy of learning algorithms. 2 related work progress in the field of nlp is closely linked to advancements in machine learning models, primarily due to the emergence of word embeddings. however, these models possess a drawback in that they generate static representations of words, meaning the same word always produces a fixed embedding, even when found in sentences with varying contexts. the elmo model [27] and subsequently, transformers [34], began to generate contextualized embeddings. as a result, the same word will have different representations in diverse situations, depending on the context of the entire sentence. by employing self-attention mechanisms, transformers are able to capture long-range relationships [19]. models built upon the transformers architecture, such as bert [11] and gpt [6], have emerged as the state of the art in nlp tasks. the core principle behind using these pretrained language models lies in the transfer learning and self-supervised learning approach. in other words, the model is initially trained on a sizable unlabeled corpus to learn the universal representation of the language [14]. subsequently, the knowledge gained during pretraining can be applied to fine-tune downstream tasks, reducing the reliance on labeled data and enhancing the performance of the models [25]. although pretrained language models, such as bert, exhibit strong performance in generic texts, they may yield inferior results in domain-specific texts [7]. therefore, implementing strategies to enhance training by incorporating data with texts from a specific domain is a widely-used technique in various fields. examples include: (i) biobert [16], which was trained on biomedical texts and outperformed bert as well as other state-of-the-art models; (ii) finbert [38], which utilized a large corpus of financial communication comprising 4.9 billion tokens and demonstrated superior performance to bert in sentiment classification tasks; (iii) scibert [5], which employed full texts from 1.14 million semantic scholar articles and showcased improved performance compared to bert-base in nlp tasks within the scientific domain; (iv) codebert [12], which made use of open-source code from public github repositories across six programming languages, achieving state-of-the-art results in natural language code search and code-to-documentation generation tasks. the legal field serves as an excellent example of a domain that could benefit from generating pretrained language models, given the vast amounts of data produced daily in courts and legal digital platforms. legal-bert [7] was among the pioneers in developing legal language models, utilizing a corpus of approximately 12 gb with texts from european and north american legislation and cases. the strategies employed in legal-bert’s pretraining included: (i) legal-bert-fp, which considered additional training from bert, and (ii) legal-bert-sc, which focused on training from scratch exclusively within the legal corpora. both strategies outperformed bert and achieved state-of-the-art results in three end-tasks. however, pretraining must also take into account not only the specific domain but also the language in which the downstream tasks need to be addressed. a model trained on multiple languages may yield inferior results in languages that lack adequate representation in the dataset [10]. consequently, efforts have been made to create monolingual pretrained language models that tackle tasks in the legal domain, such as lawformer [37], italian-legal-bert [18], inlegalbert [26], and jurisbert [35]. 3 legalbert-pt: a bert model for the brazilian legal domain this section provides a detailed account of the steps taken to pretrain legalbert-pt, a language model for the portuguese legal domain. 3.1 pretraining data to pretrain various versions of the legalbert-pt language model, we collected a total of 1.5 million legal documents in portuguese from ten brazilian courts. these documents consisted of four types: initial petitions, petitions, decisions, and sentences. table 1 shows the distribution of these documents. the data were obtained from the codex system of the brazilian national council of justice (cnj), which maintains the largest and most diverse set of legal texts in brazilian portuguese. as part of an agreement established with the researchers who authored this article, the cnj provided these data for our research. our use of this corpus allowed us to pretrain variations of the legalbert-pt model that are well-suited to handling the nuances and complexities of legal language in the brazilian context. we drew upon previous research [3, 4, 35] that demonstrated the importance of using large and diverse datasets for training language models, particularly in domain-specific contexts. table 1. statistical legal documents by data source.full size table to minimize errors in the texts of the documents, we employed a two-stage preparation process consisting of pre-processing and cleaning. in the pre-processing stage, we removed documents with less than 50 words and less than 80% valid words to ensure that the remaining corpus was of high quality. in the cleaning step, we removed special characters and extra spacing to further improve the corpus. table 2 provides statistics on the types of legal documents that remained after the preparation steps. these documents were then used to train various versions of the legalbert-pt language model. table 2. details of the training corpus used to pretrain the different variations of legalbert-pt.full size table after the preparation steps, the documents were divided into sentences with a maximum size of 512 tokens, generating a total of about 12,000,000 sentences. 3.2 vocabulary generation since bertimbau [33] is trained on data from the general domain, we believe that training a language model for a specific domain can improve its performance on domain-specific tasks with a specific vocabulary. to this end, we generated a vocabulary consisting of 30,000 subword units using the sentencepiece library [15] and the byte-pair encoding (bpe) algorithm [30]. we used 2 million random sentences from 1 million wikipedia articles in portuguese and 2 million random sentences from 1.5 million legal documents in our pretraining dataset described in sect. 3.1. to ensure compatibility with the original bert code, we converted the resulting vocabulary to the wordpiece format. to do this, we followed the bert tokenization rules. first, we added all special bert tokens ([cls], [mask], [sep], and [unk]) and punctuation characters to the english vocabulary. then, we split sentencepiece tokens that contain punctuation characters, removed the punctuation, and added the resulting subword units to the vocabulary. finally, we prefixed subword units that do not begin with the sentencepiece metacharacter “_” with “##” and removed the “_” symbol from the remaining tokens. additionally, we included 5,977 identifiers of brazilian legislation in the vocabulary. the resulting legalbert-pt vocabulary consists of 36,345 subwords. figure 1 compares the number of subwords in the legalbert-pt and bertimbau vocabularies. we found that 16,885 subwords are common between the two vocabularies, while 19,460 subwords are specific to legalbert-pt. fig. 1. number of units of subwords of the bertimbau and legalbert-pt vocabularies. full size image 3.3 variations of the legalbert-pt model the use of bert in downstream tasks involves two stages: pretraining and model fine-tuning. in the pretraining stage, the model is trained from scratch or with additional steps from an existing model to learn bidirectional context between tokens. this step is computationally intensive and should only be performed once. in the fine-tuning stage, a pretrained model is further trained on a specific task of interest, such as text classification or named entity recognition. for our study, we developed two variations of the pretraining of legal domain language models in brazilian portuguese: (i) pretraining from scratch using a specific domain corpus (legalbert-pt sc) and (ii) an adaptation of bertimbau with pretraining using a specific domain corpus (legalbert-pt fp). both models were pretrained as case-sensitive, as we focused on developing general-purpose models and capitalization is relevant for tasks such as named-entity recognition. we pretrained the models using the masked language model (mlm) task. we only used the mlm task in pretraining, as recent research [20] has suggested that the next sentence prediction (nsp) task is not effective. the mlm objective allows the representation to learn left and right context, which allows us to pretrain a deep two-way transformer. the masked language model randomly chooses some tokens from the input and replaces them with either a special [mask] token with 80% probability, a random vocabulary token with 10% probability, or the original token with 10% probability. the goal is to predict the id of the masked word in the original vocabulary based on its context. during training, we used the adamw optimizer [21] with the following parameters: \(\beta _1 = 0.9\), \(\beta _2 = 0.999\), \(\epsilon = {1\textrm{e}-6}\), and a learning rate of 1e – 4. the legalbert-pt sc model has the same architecture as bertimbau-base, with 12 layers, 768 hidden units, and 12 attention heads (a total of 110 million parameters). we used this architecture in all of our experiments. for this model, we used the specialized vocabulary generated for the legal domain as described in sect. 3.2. we pretrained this model for 7.5 million steps on a legal domain corpus, as described in sect. 3.3. for the legalbert-pt fp model, we followed the approach outlined in [11], initializing the weights from the pretrained bertimbau-base checkpoint [33], and then performing additional pretraining steps using a legal domain corpus, as described in sect. 3.3. [11] suggests performing additional pretraining steps up to 1,000,000 steps. in our case, we pretrained the model for up to 2.4 million steps to evaluate the prolonged effect of pretraining on downstream tasks. bertimbau-base has been significantly pretrained on generic domains such as health, sport, technology and computing, laws and policies, among others, using a vocabulary of 30,000 subwords that is best suited for these generic domains. therefore, we expect that domain-specific pretraining will result in better accuracy for specific tasks. 4 evaluation to evaluate the effectiveness of the pretrained legalbert-pt language models, legalbert-pt sc and legalbert-pt fp, we conducted intrinsic and extrinsic evaluations, comparing them with a generic model, bertimbau-base [33], and with a language model of the legal domain of the portuguese language, legal-bertimbau-base [17]. intrinsic evaluation measures the quality of a model independent of any application, while extrinsic evaluation measures the usefulness of the model in a specific task. we evaluated the models using two specific nlp tasks: named entity recognition (ner) and text classification. these tasks were chosen to evaluate the application of the model at both the token level (ner) and the sentence level (text classification), and due to the availability of labeled datasets for these tasks. for each specific task, we fine-tuned the pretrained model. in sentence-level tasks, classification was performed using the coded representation of the special token [cls], while in token-level tasks, the coded representation of each token was used. we measured the performance of the pretrained models using perplexity and f1-score. perplexity measures how well a language model predicts a sample of text and reflects the model’s ability to generate coherent and natural-sounding sentences. the f1-score measures the accuracy of the model in identifying named entities in text or classifying text into predefined categories. 4.1 perplexity perplexity is an important intrinsic evaluation metric used to assess language model performance by quantifying the degree of uncertainty a model has about the predictions it makes. low perplexity indicates that a model is reliable, but it does not guarantee accuracy. perplexity is also often correlated with a model’s ultimate performance on specific tasks. therefore, in addition to standard evaluation metrics, nlp researchers have started looking at perplexity to test how well language models capture language [23]. to analyze the perplexity of our language models, we built a corpus consisting of 750 legal documents obtained from various sources. this corpus included 250 legal documents representing initial petitions and complaints from the court of justice of the state of ceará (tj-ce) in brazil, 250 legal documents of various types from the public ministry of the state of ceará (mp-ce) in brazil, and 250 legal documents from extraordinary appeals of the stf obtained randomly from the victor dataset [4]. by evaluating perplexity on this corpus, we were able to assess how well our pretrained models captured the language used in legal documents in brazilian portuguese. 4.2 named entity recognition named entity recognition (ner) is the task of identifying snippets of text that mention named entities (nes) and classifying them into predefined categories such as person, organization, and location. given a sequence of tokens, the ner model has to output the entity of each token. the ner model was designed as a token labeling task and performs entity identification and classification using the iob tagging scheme [29]. to date, there are few gold standard datasets for named entities in the legal domain in portuguese. to evaluate the ner model, we used two datasets separately, lener-br [3] and cdjur [22], with annotated entities in legal documents. the lener-br [3] dataset was built by manually annotating 66 legal documents from several brazilian courts. additionally, four legislative documents were included, totaling 70 annotated documents. the entities were categorized as “organization”, “person”, “time”, “location”, “legislation”, and “legal cases”, resulting in a total of 12,248 entity annotations. the cdjur [22] dataset contains 1,074 manually annotated legal documents, with a total of 44,526 labeled entities. this dataset provides a detailed annotation of entities specific to the legal domain. for example, the “person” category was specified in 9 entities that are typically present in a judicial process, such as plaintiff, lawyer, defendant, victim, witness, judge, prosecutor, police authority, and others. “addresses” were specified in 6 entities to identify different addresses present in a lawsuit. the laws category was specified in three entities: main law, accessory law, and jurisprudence. similarly, specifications were made for “evidence”, “penalty”, and “sentence”. we trained a ner model for each of these datasets, and a linear classifier layer was attached on top of each model to predict each token’s tag independently. the models’ performance was evaluated in terms of f1-score at the entity level, taking into account the partial correspondence between the predicted entity and the actual entity based on the partial metric defined by muc [9]. partial correspondence is considered correct when the entity type of the prediction given by the model corresponds to the same entity type of the golden annotation, but not necessarily in the same position limits. for example, if the annotation is {“entity_type”: “main-law”, “text”: “law n\(^\circ \) 8.112/90, of 12/11/1990”} and the model prediction is {“entity_type”: “main-law”, “text”: “law n\(^\circ \) 8.112/90”}, it is considered correct. 4.3 text classification text classification is a widely researched task in nlp and text mining, involving the assignment of one or more categories to a document from a set of options. there are different text classification variants such as binary classification, multiclass classification, and multi-label classification. language models from the field of nlp and learning models from the field of ai can be developed to automate this task, which can be trained from a gold collection of documents. in the legal domain, text classification has a crucial application in the initial stages of the judicial process when a petitioner presents a petition to the justice [1, 2]. at this stage, the petitioner is required to specify the matter to which the claim pertains. in brazil, the petitioner has to choose the topic from a hierarchy of over 4,000 subjects as part of the unified procedural boards (tpu system) [32], maintained by the cnj. making the correct association with the hierarchy theme is not trivial and often done incorrectly, causing delays in the judicial process and negative financial and societal impacts by generating rework and a sense of impunity. we evaluated the performance of the language model classification using a gold collection of 64,000 textual petitions obtained from various brazilian courts maintained by the cnj’s codex system. each complaint is associated with a legal issue that the case addresses under the tpu. each judicial process is associated with a hierarchy of matters, represented by three levels. more specifically, the gold collection used in the experiment contains 213 legal matters associated with the initial petitions, 9 matters in the first level of the hierarchy, 41 matters in the second level and 163 matters in the third level. the evaluation of the models in the classification task was carried out in three scenarios. in the first scenario, the classification task was modeled as hierarchical text classification (htc), which categorizes text into a set of labels organized in a hierarchical structure. we use a contrastive learning approach to hierarchical text classification [36] with the pre-trained language model to train the classification model. in the second scenario, the classification task was modeled with multiclass classification, in which each process is associated only with the subject of the third level, therefore, contemplating 163 classes. in both scenarios, the text is truncated to use the initial 512 tokens of the court case text. complementarily, the evaluation was carried out in a third scenario, in which the classification task was modeled with multiclass classification, and using 8,192 initial tokens from the text of the judicial process (according [8]). in both scenarios, we evaluated the models in terms of the f1-score for the third-level subject of the hierarchy. 5 experimental results the initial evaluation of the language models involved intrinsic evaluation using perplexity as a metric. a lower perplexity score indicates a better model. essentially, if a model assigns a high probability to the test set, it means that it is not surprised to see it, indicating a good understanding of how language works. the results of the language models regarding perplexity are in table 3. in the tables, the values in bold are the best values among the experiments with the models. table 3. results of language models in terms of perplexity.full size table the legalbert-pt fp language model performed the best with a perplexity of 3,700, followed by the legalbert-pt sc language model with a perplexity of 3,822, indicating that pretraining with a domain-specific corpus that includes diverse legal documents leads to a better understanding of the specific language. models with lower perplexity are also expected to perform better on specific tasks. therefore, the language models were fine-tuned for specific tasks, namely ner and text classification, according to [33]. the results of the application of language models in the named entity recognition task, using corpora lener-br and cdjur, are presented in tables 4 and 5, respectively, as described in sect. 4.2. both experiments were run 5 times in 10 epochs to avoid bias, and the f1-score results are displayed in terms of the mean and the standard error of the mean (sem) of the runs. values marked in bold indicate the best results, considering the mean and sem. table 4. results of the language models in terms of the mean + standard error of the f1-score in named entity recognition in the lener-br corpus.full size table table 5. results of the language models in terms of the mean ± standard error of the f1-score in named entity recognition in the cdjur corpus.full size table the legalbert-pt fp model demonstrated superior results, outperforming the bertimbau-base by 0.65% in f1-score micro, 1.75% in f1-score macro, and 0.65% in f1-score weighted for ner in the lener-br dataset, and by 0.89% in f1-score micro, 3.78% in f1-score macro, and 1.49% in f1-score weighted for ner in the cdjur dataset. this reinforces the benefits of using domain-specific pretrained models compared to generic ones, as legalbert-pt fp can understand both the generic context and specific legal language. moreover, when compared with legal-bertimbau-base, the legalbert-pt fp model outperformed by 0.65% in f1-score micro, 1.31% in f1-score macro, and 0.65% in f1-score weighted for ner in the lener-br dataset, and by 1.49% in f1-score micro, 4.48% in f1-score macro, and 2.41% in f1-score weighted for ner in the cdjur dataset. these results reinforce the effectiveness of pretraining a legal domain model with a diverse set of legal documents, enabling a wider coverage of specific terms and acquiring greater domain specialization. the superiority of the language models in ner with the lener-br dataset compared to the cdjur dataset is also notable. this difference can be attributed to the specificity and number of entities present in each dataset. the lener-br dataset has only 6 specific entities of the legal domain (organization, person, time, place, legislation, and jurisprudence), while the cdjur dataset has 21 specific entities of the legal domain (author, lawyer, defendant, victim, witness, judge, prosecutor, police authority and other persons, author’s address, offense address, defendant’s address, witness address, victim’s address, other addresses, main law, accessory law, jurisprudence, evidence, penalty, and sentence). thus, it is more challenging for the model to recognize the more specific entities present in the cdjur dataset. table 6, 7 and 8 presents the results in terms of f1-score for the application of language models in the text classification task, in different scenarios, as described in sect. 4.3. to avoid biasing the results, both experiments were run 5 times and the f1-score results are displayed in terms of the mean and sem of the runs. both models were run for 30 epochs, with early stopping and a 5 epoch patience criterion. table 6. results of the language models in terms of the mean ± standard error of the f1-score in hierarchical text classification.full size table in the text classification task, in all evaluated scenarios, the legalbert-pt models demonstrate superior results compared to other models. in the scenario that evaluates the hierarchical text classification, legalbert-pt fp outperforms the generic bertimbau-base model by 1.19% in f1-micro and 0.4% in f1-weighted. when compared with legal-bertimbau-base, legalbert-pt fp also outperforms f1-micro by 1.59% and f1-weighted by 0.8%. it is important to note, however, that the relatively close results between the generic and specific models in the text classification task suggest that the use of a generic textual structure in the composition of the initial petitions may not require in-depth domain-specific knowledge. thus, relevant linguistic features for classification may be more universal and captured by a generic language model. it is also important to recognize that the performance of a model may vary depending on the task and dataset, and a specific model may be better suited for a particular task. table 7. results of the language models in terms of the mean ± standard error of the f1-score in text classification with 512 tokens.full size table table 8. results of the language models in terms of the mean ± standard error of the f1-score in text classification with 8,192 tokens.full size table in the multi-class classification approach, the legalbert-pt models outperform the bertimbau-base results in all experiments. when analyzing the classification with the representation of the text by 512 tokens, the result legalbert-pt fp exceeds by 2.6% in the f1-micro, 4.6% in the f1-macro and 3.0% in the f1-weighted. when analyzing the classification with the representation of the text by 8,192 tokens, the legalbert-pt fp result exceeds 3.0% in the f1-micro, 3.7% in the f1-macro and 3.2% in the f1-weighted. one possible reason for legalbert-pt fp’s better performance in text classification is that it was pretrained on a dataset with language similar to the language of the texts used in the classification task. this could have allowed the model to learn relevant linguistic features that contribute to improved performance. the results of our experiments demonstrate that domain-specific language models outperform domain-general language models on domain-specific tasks such as ner and text classification. this suggests that pretraining with domain-specific texts allows the language model to learn richer and more specific representations of the domain-specific language compared to domain-general language models. however, it is important to note that specialization in a single domain limits the model’s applicability to tasks in that domain. therefore, in scenarios where the application needs to handle multiple domains, a broader approach may be necessary, such as using generic language models. 6 conclusion and future works this article outlined the process of training, using, and evaluating a language model specifically designed for the legal domain. it demonstrated that the language model enhances accuracy in particular tasks within the legal field. experiments were conducted to determine the optimal strategy for generating a language model for a new domain, weighing the options of additional pretraining from a pre-existing generic domain model or starting from scratch. consequently, two versions of the language model pretrained on legal domain documents were developed: legalbert-pt sc (trained from scratch) and legalbert-pt fp (pretrained using the bertimbau-base model). for the legalbert-pt sc model, legal documents and articles from the portuguese wikipedia were utilized to create a vocabulary that includes both generic and legal domain-specific terms in portuguese. the legal domain language models were compared to the bertimbau-base (a generic domain language model for the portuguese language) using intrinsic evaluation (measured through perplexity) and extrinsic evaluation (by fine-tuning the language model for specific tasks such as named entity recognition and text classification). in both evaluations, the language models pretrained on legal domain documents yielded superior results compared to the bertimbau-base. notably, the most significant performance gains were observed in the most challenging final tasks. the language models developed in this study are available under the openrail license and can be accessed at http://huggingface.co/raquelsilveira/legalbertpt_sc and http://huggingface.co/raquelsilveira/legalbertpt_fp. the models can be further fine-tuned using other types of legal documents, which presents an opportunity for ongoing improvement. this contribution is significant for both the scientific and practical communities, as it advances our understanding of how language models can enhance legal tasks. in future research, we aim to examine the performance of legalbert-pt on additional datasets and explore the model’s application in other tasks within the legal domain. this will help assess the extent to which understanding the specific language of these models impacts the accuracy of domain-specific tasks. comparisons with gpt models and the utilization of the pretrained model to distill llms are also planned for future endeavors. references aguiar, a., silveira, r., pinheiro, v., furtado, v., neto, j.a.: text classification in legal documents extracted from lawsuits in brazilian courts. in: anais da x brazilian conference on intelligent systems, sbc, porto alegre, rs, brasil (2021). https://sol.sbc.org.br/index.php/bracis/article/view/19093 aguiar, a., silveira, r., furtado, v., pinheiro, v., neto, j.a.m.: using topic modeling in classification of brazilian lawsuits. in: pinheiro, v., et al. (eds.) propor 2022. lncs (lnai), vol. 13208, pp. 233–242. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_22 chapter  google scholar  luz de araujo, p.h., de campos, t.e., de oliveira, r.r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: villavicencio, a., et al. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 313–323. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_32 chapter  google scholar  luz de araujo, p.h., de campos, t.e., ataides braz, f., correia da silva, n.: victor: a dataset for brazilian legal documents classification. in: proceedings of the twelfth language resources and evaluation conference, pp. 1449–1458. european language resources association, marseille (2020). https://aclanthology.org/2020.lrec-1.181 beltagy, i., lo, k., cohan, a.: scibert: a pretrained language model for scientific text. arxiv preprint arxiv:1903.10676 (2019) brown, t., et al.: language models are few-shot learners. adv. neural. inf. process. syst. 33, 1877–1901 (2020) google scholar  chalkidis, i., fergadiotis, m., malakasiotis, p., aletras, n., androutsopoulos, i.: legal-bert: the muppets straight out of law school. arxiv preprint arxiv:2010.02559 (2020) chalkidis, i., et al.: lexglue: a benchmark dataset for legal language understanding in english (2022) google scholar  chinchor, n., sundheim, b.m.: muc-5 evaluation metrics. in: fifth message understanding conference (muc-5): proceedings of a conference held in baltimore, maryland, 25–27 august 1993 (1993) google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale. arxiv preprint arxiv:1911.02116 (2019) devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) feng, z., et al.: codebert: a pre-trained model for programming and natural languages. arxiv preprint arxiv:2002.08155 (2020) jain, d., borah, m.d., biswas, a.: summarization of legal documents: where are we now and the way forward. comput. sci. rev. 40, 100388 (2021) article  google scholar  kalyan, k.s., rajasekharan, a., sangeetha, s.: ammus: a survey of transformer-based pretrained models in natural language processing. arxiv preprint arxiv:2108.05542 (2021) kudo, t., richardson, j.: sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. arxiv preprint arxiv:1808.06226 (2018) lee, j.: biobert: a pre-trained biomedical language representation model for biomedical text mining. bioinformatics 36(4), 1234–1240 (2020) article  mathscinet  google scholar  legal-bertimbau-base. https://huggingface.co/rufimelo/legal-bertimbau-base licari, d., comandè, g.: italian-legal-bert: a pre-trained transformer language model for italian law (2022) google scholar  lin, t., wang, y., liu, x., qiu, x.: a survey of transformers. ai open 3, 111–132 (2022). https://doi.org/10.1016/j.aiopen.2022.10.001 article  google scholar  liu, y., et al.: roberta: a robustly optimized bert pretraining approach. arxiv preprint arxiv:1907.11692 (2019) loshchilov, i., hutter, f.: decoupled weight decay regularization. arxiv preprint arxiv:1711.05101 (2017) brito, m., et al.: cdjur-br a golden collection of legal document from brazilian justice with fine-grained named entities. arxiv preprint arxiv:2023.49053 (2023) meister, c., cotterell, r.: language model evaluation beyond perplexity. arxiv preprint arxiv:2106.00085 (2021) nguyen, t.s., nguyen, l.m., tojo, s., satoh, k., shimazu, a.: recurrent neural network-based models for recognizing requisite and effectuation parts in legal texts. artif. intell. law 26, 169–199 (2018) article  google scholar  pan, s.j., yang, q.: a survey on transfer learning. ieee trans. knowl. data eng. 22(10), 1345–1359 (2010) article  google scholar  paul, s., mandal, a., goyal, p., ghosh, s.: pre-training transformers on indian legal text. arxiv preprint arxiv:2209.06049 (2022) peters, m.e., et al.: deep contextualized word representations (2018) google scholar  polo, f., et al.: legalnlp natural language processing methods for the brazilian legal language. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 763–774. sbc, porto alegre (2021). https://doi.org/10.5753/eniac.2021.18301. https://sol.sbc.org.br/index.php/eniac/article/view/18301 sang, e.f., veenstra, j.: representing text chunks. arxiv preprint arxiv:cs/9907006 (1999) sennrich, r., haddow, b., birch, a.: neural machine translation of rare words with subword units. arxiv preprint arxiv:1508.07909 (2015) shao, y., et al.: bert-pli: modeling paragraph-level interactions for legal case retrieval. in: ijcai, pp. 3501–3507 (2020) google scholar  sistema de gestão de tabelas processuais unificadas. https://www.cnj.jus.br/sgt/consulta_publica_assuntos.php. accessed 09 aug 2022 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  vaswani, a., et al.: attention is all you need. adv. neural inf. process. syst. 30, 1–11 (2017) google scholar  viegas, c.f.o.: jurisbert: transformer-based model for embedding legal texts (2022) google scholar  wang, z., wang, p., huang, l., sun, x., wang, h.: incorporating hierarchy into text encoder: a contrastive learning approach for hierarchical text classification. arxiv preprint arxiv:2203.03825 (2022) xiao, c., hu, x., liu, z., tu, c., sun, m.: lawformer: a pre-trained language model for chinese legal long documents. ai open 2, 79–84 (2021) article  google scholar  yang, y., uy, m.c.s., huang, a.: finbert: a pretrained language model for financial communications. arxiv preprint arxiv:2006.08097 (2020) download references author information authors and affiliations federal institute of ceará (ifce), fortaleza, brazil raquel silveira university of fortaleza (unifor), fortaleza, brazil caio ponte, vitor almeida, vládia pinheiro & vasco furtado authors raquel silveiraview author publications search author on:pubmed google scholar caio ponteview author publications search author on:pubmed google scholar vitor almeidaview author publications search author on:pubmed google scholar vládia pinheiroview author publications search author on:pubmed google scholar vasco furtadoview author publications search author on:pubmed google scholar corresponding author correspondence to raquel silveira . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silveira, r., ponte, c., almeida, v., pinheiro, v., furtado, v. (2023). legalbert-pt: a pretrained language model for the brazilian portuguese legal domain. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_18 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords language models legal texts bert bertimbau publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improved fuzzy decision system for energy bill reduction in the context of the brazilian white tariff scenario | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improved fuzzy decision system for energy bill reduction in the context of the brazilian white tariff scenario conference paper first online: 12 october 2023 pp 441–454 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) improved fuzzy decision system for energy bill reduction in the context of the brazilian white tariff scenario download book pdf download book epub raimunda branco9 & filipe saraiva9  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 498 accesses abstract the production of energy from renewable sources has become a more sustainable and environmentally correct alternative, where the use of solar energy through photovoltaic systems is evident. one of the main problems in the use of photovoltaic systems is the high cost of installation and maintenance of this system, in addition to the cost of the residential electricity tariff, which makes this technology expensive for most residential consumers in brazil. an alternative for consumers to get around the high amounts paid on the energy bill is to opt for the white tariff modality, which is characterized by offering the variation of the energy value according to the day and time of consumption. the present work aims to develop a fuzzy system to manage the energy production from a photovoltaic system, optimizing the use of the produced energy between the consumer, the battery and the electric grid in a white tariff scenario for residential units in brazil. based on the simulations, the fuzzy system presented is efficient, with a significant economic reduction in the energy bill compared to a simple photovoltaic system without the ability to make intelligent decisions and used commercially in industries. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a fuzzy system applied to photovoltaic generator management aimed to reduce electricity bill chapter © 2019 photovoltaic systems: a review with analysis of the energy transition in brazilian culture, 2018–2023 article open access 01 march 2024 economic analysis of residential photovoltaic self-consumption in ecuador: simulation tool chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. energy management energy conservation energy supply and demand photovoltaics renewable energy two-hybrid system 1 introduction a country’s energy needs are an important indicator of its economic development. electricity consumption has been identified as one of the most important factors for the well-being of society and a critical determinant of a country’s progress [4]. as risk factors, increasing population demand combined with a country’s energy base tied to a single or few energy sources are vulnerabilities that threaten economic growth. the main sources of electrical energy in the world are hydro, biomass, wind, solar, natural gas, oil derivatives, nuclear and coal [6]. the demand for diversification of energy sources, the concern for sustainability and the reduction of costs have led several branches of science in the search for greater efficiency of electrical energy generation systems using renewable sources. among the main sources of renewable energy, solar energy has become increasingly important, especially in countries with tropical climates. according to a report by [6], in brazil, the photovoltaic solar energy generation has evolved from 3,287 mw in 2020 to 4,632 mw in 2021. [15] assert that solar energy is the largest source of energy available on earth, which is a renewable and inexhaustible source of energy. also, according to the authors, the growing demand and research in this field will make solar energy become a significant part of the world’s energy matrix in the future. [1] explain that since the beginning of the 21st century, the traditional concept of centralized energy generation has been open to research and application of distributed generation (dg) or decentralized. a distributed system is mainly composed of small generator units connected directly to the consumer load and/or distribution grids, using mainly renewable energy sources such as photovoltaic [17]. in this system, each consumer can produce electricity and then connect to the distribution grid. in this way, the residential consumer can consume and also supply excess electricity to the local distribution center. there are several advantages in the use of dg, among which the reduction of the electricity tariff paid by consumers to concessionaires stands out, since there is autonomy for the consumer to also produce energy. however, for [17], the study of which strategies could be used to increase the reduction of electrical energy consumption by concessionaires and consequently the use of dg to reduce the energy bill is still incipient. when dealing with the management of photovoltaic systems, among the alternatives for a more efficient management, focusing on a higher quality and more economically viable energy distribution, computational intelligence (ci) has emerged as a promising option, since it seeks methods that possess or enhance the intelligent capacity of humans to solve problems, acquire and represent knowledge, in addition to recognizing patterns [11]. this article used as a reference the research by [17]. the idea at this first moment is to improve and update this work through adjustments in the fuzzy algorithm, mainly by improving the parameters of the linguistic variables, proposing other models for the system and adapting the configuration of photovoltaic technology to the present day. in this sense, this research proposes to develop an intelligent photovoltaic system, using fuzzy logic as ci technique for decision-making about the best time to use or not to use the energy produced by dg, in a scenario where generators have an energy storage system coupled to them, with the aim of reducing the price of the energy bill for the final consumer. for this purpose, an environment with a different type of tariff modality is simulated, where the amount paid by the consumer depends on the time at which the energy is consumed, defined as a white tariff. finally, in addition to comparing the developed system with a previous work, the performance of a simple photovoltaic solar energy system (developed without any intelligent technique) is also simulated in order to compare it with the system developed through fuzzy logic. the paper is presented as following: sect. 2 proposes to present the literature review that supported this research. section 3 describes the features of the proposed system, presenting the tariff modality used and the components of the photovoltaic system. section 4 presents in detail the development of the fuzzy system. section 5 describes the simulation created, explaining the database used, the specificities of the photovoltaic system components and how the fuzzy system is modeled. section 6 describes the development of the research and presents the results obtained. finally, sect. 7 presents the conclusion about this research. 2 literature review the application of artificial intelligence techniques in photovoltaic electrical systems is currently presented in a very frequent and promising way. there are several works that present fuzzy logic as an alternative to manage and optimize the distribution of energy in photovoltaic systems. [12] presents an algorithm based on fuzzy logic to manage energy storage in a battery and also the energy demanded by consumers at the university of naresuan, thailand, in order to reduce the electricity bill. the main idea of the algorithm is to allow an intelligent switching between absorbing energy during periods of high solar irradiation and discharging energy to the load during times of high consumption. the fuzzy algorithm proved to be successful, as it reduced the annual energy bill by 17.58%. the work by [5] addresses power management to improve network performance and generate smooth transitions between different power balance modes. this work presents a fuzzy system developed to provide a dynamic flow of energy from the grid, based on the network price, in order to decrease the energy rate and increase the useful life of the storage device. experimental results were obtained which proved to be satisfactory. another relevant work found is the article by [14], which deals with energy management to maintain a balance between different energy sources, storage units and loads. in this article, fuzzy logic was used in order to maintain the balance between these two objectives. from the results exposed in this work, it was observed that this system demonstrated viability to be used as a solution to the problem. [3] publishes a work in which an energy management system based on fuzzy logic is presented, where the objective is to smooth the profile of an electrothermal microgrid connected to the residential network. the proposed case study designs an energy management system to reduce the impact on the electrical grid when renewable energy sources are incorporated into pre-existing appliances connected to the grid. the simulation results used real data measured over a year and showed that the proposed management system design achieves a reduction of 11.4% of the maximum power absorbed from the network. the publication by [10] proposes a residential smart microgrid topology linked to the distribution grid using fuzzy logic, which integrates a photovoltaic system, a fuel cell and a battery bank. to validate the functioning of the proposed energy management unit, a prototype of the system was developed and experimental tests were carried out. the management system was tested for three different residential load scenarios, both connected to the grid and isolated. the distribution and analysis of the energy cost provided for each management scenario showed the benefits of using this type of intelligent system for the consumer and for the utility. [19] developed a coordinated control scheme of a battery energy storage system and dg units for a microgrid, based on fuzzy logic. the coordinated control scheme aims to mitigate the fluctuation of active power at the point of common coupling of the microgrid when it is connected to the grid and also to keep the frequency of the microgrid within the defined range for operation when it is isolated. in the control scheme, the soc of the battery energy storage system is used as input for the coordinated control based on fuzzy logic. the results of the case study were satisfactory and showed that the proposed coordinated control scheme is capable of mitigating the fluctuation of active power at the common coupling point for grid-connected control and performing efficient frequency control for isolated operation. 3 features of the proposed system in addition to the use of a dg system, this research prioritized the use of the white hourly tariff modality. according to [2], the white tariff is characterized as consisting of differentiated tariffs for energy consumption depending on the time of day. this modality is divided into three tariff types: peak tariff, intermediate tariff, and off-peak tariff. the days of the week and times stipulated for tariff posts are defined and implemented by each electricity company distributor in brazil, provided that they are previously approved and ratified by aneel [2]. equatorial energia is one of the energy distributors present in brazil that have made available the white tariff modality for all consumer units, since january 1st, 2020. the values of each tariff are also defined by the energy distributors. at equatorial energia, for example, the values and periods of tariff posts must be approved by aneel in each periodic tariff review, which must occur every four years [7]. the schedules and values of the tariff stations defined for the state of pará – brazil since the year 2019 are specified in the items below: 1. peak time (from 6 pm to 8:59 pm): r$ 1.39; 2. intermediate time (from 4 pm to 5:59 pm and from 9 pm to 9:59 pm): r$ 0.87; 3. off-peak time (from 10 pm to 3:59 pm): r$ 0.49. the benefits of using the white tariff can then be realized above all to consumers who consume very little between 4 pm and 10 pm, that is, during peak and intermediate time. the white tariff is a modality that, depending on the consumption profile and the habits of use of electric energy, presents itself as an interesting alternative for the residential consumer. another important issue for the system configuration of this application is the use of an integrated storage system connected to a hybrid photovoltaic system (both on-grid and off-grid). the presence of the battery in a photovoltaic production system is intended to accommodate the excess energy production of the residence. in this way, the extra energy can be used in case of power outages of the photovoltaic panel and/or the electrical network and to balance the load inserted in the grid [17]. in the context of the white tariff modality, it is possible to charge the battery when energy from the grid is cheap and to use it when energy is expensive. in this way, the use of the battery reduces the billed consumption of the consumer unit, especially during peak hours. as a way of exemplifying this section, fig. 1 presents the system components and the architecture proposed in this article for the residential photovoltaic installation. fig. 1. system components and architecture of residential photovoltaic installation [8] full size image 4 development of the fuzzy algorithm [18] defines a fuzzy system as a particular type of knowledge-based system where the knowledge base is built from a set of fuzzy rules. in this type of system, inputs are given in natural language, which, after going through the fuzzy inference process, are converted into a numerical format that is easy to manipulate. the steps for building a fuzzy system are described in fig. 2. fig. 2. steps for implementing a fuzzy system full size image the fuzzy system developed in this research consists of 3 different fuzzy algorithms, one algorithm for each price range of the white tariff, as detailed previously. the fuzzy system is composed of 2 input variables (el and soc) and an output variable (eb). el represents the energy in kwh, that is, it is the value resulting from the subtraction of the energy produced by the dg (eg) and the energy consumed by the consumer load (ec). thus, if el is positive, then there is more production than consumption, but if el is negative, then production is not enough to supply the demand for electricity. soc is the state of charge of electricity stored in the battery. if el is positive, it is possible to recharge the battery using the surplus electricity produced. if el is negative, the battery can be used to supply the energy demand. eb represents how much energy will be injected or utilized from the battery, that is, it represents the decision whether the battery will be charged or discharged. so, if eb is positive, then energy can be inserted into the storage device. if eb is negative, then the battery can be used by the customer load. however, these decisions will always depend on the time of day, el and soc. table 1 presents the meaning of the linguistic sets of each variable. table 1. definition of linguistic setsfull size table table 2 shows the rule bases defined for each tariff configuration. for each rule created, two conclusions will be specified, based on the association of predefined inputs and outputs. note that depending on the grid energy price, the rule will have a certain behavior, resulting in an output with a different value. for example, if there is low consumption surplus (el = ns) and if the battery is half full (soc = ze), then 3 situations can result: 1. cheap energy: then there is no interaction with the battery (eb = ze); 2. intermediate energy: then there is a small energy discharge in the battery (eb = ns); 3. expensive energy: then there is a big energy discharge in the battery (eb = nb). each configuration has a set of 25 rules, for a total base of 75 rules. table 2. rule base map for off-peak time (a), intermediate time (b) and peak time (c)full size table the membership functions used to represent the linguistic sets were the triangular and trapezoidal forms. the maximum and minimum limits of the input variables (el and soc) were determined based on the dataset values, for el the minimum limit was set at −6 kw and the maximum limit 4 kw, while for the soc it was agreed the value in percentage limits of the energy contained in the battery ranging from 0 to 100%. regarding the output variable (eb), the limits followed the ±6 kw battery charge flow. the following graphs show the membership functions of the input and output variables, with their respective linguistic sets (fig. 3). fig. 3. membership function of the output variable el (a), soc (b) and eb (c) full size image in addition, the fuzzy system uses the mamdani inference method [18] and the defuzzification step is performed using the centroid method [18]. it is important to emphasize that the form used for the membership functions (triangular and trapezoidal) and the values used for the linguistic variables were defined empirically, based on a series of tests and adaptations, with the aim of finding the best configuration for the fuzzy system in question. in the same way, the configurations of the inference and defuzzification method were also defined both based on the used literature and based on system tests. 5 simulation specification this paper improves the fuzzy system presented in [17] by changing the boundaries of the linguistic variables. for the simulation of the system, real data on residential energy consumption and photovoltaic production over a period of one year were used. the public databases used were obtained from the research group in knowledge engineering and decision support (gecad) at the laboratory of intelligent electrical systems (lasie), located in portugal at the institute of engineering polytecnica do porto (isep /ipp) [9]. data on residential consumption cover the period from december 2011 to march 2013, with a 15 min interval between one measurement and another. about photovoltaic production data, these include the months from january to december 2013, with an interval of 5 min between measurements. the measurement intervals were standardized in 5 min for residential consumption and photovoltaic production. the simulation also adapted the composition of the photovoltaic electrical system to 13 photovoltaic panels, each one with a capacity of 340 w of power. thus, the power of the installed system resulted in 4.42 kw. the residence has a maximum power demand of 6.8 kw and it is connected to a hybrid photovoltaic system, coupled to a lifepo4 battery with a capacity of 13.5 kwh. as a way of comparison, the system simulations were reproduced both for modeling considering a simple photovoltaic system, and for a photovoltaic system using fuzzy modeling. both models are based on time of day, net energy and state of charge in the battery. the simple photovoltaic system has a predetermined decision behavior, where, for this case, it follows the following procedures: excess energy only charges the battery during the day. if the battery is full, the surplus is sent to the electrical distribution grid; the residence only uses energy from the distribution grid if the battery has a limit of 25% of the total; the residence only uses the energy stored in the battery during the hours from 4 pm to 10 pm, that is, when energy is generally more expensive in the white tariff mode. decision making in the fuzzy system behaves in a more complex way, but the idea is to make a more specific and intelligent assessment for each situation and moment, considering the white tariff modality: if the time of day is within the time range of the least expensive electricity rate, the system is more likely to store energy in the battery, depending on the state of charge; for intermediate electricity cost, the fuzzy system tends to use the battery moderately; when the cost of electricity is expensive, the system will try to use as much battery power as possible. equations (1) and (2) represent the punctual and annual electrical consumption pc of the residence, respectively. when grid power pgrid is a positive value, it means that there was surplus energy produced by dg, so this extra energy must be transferred to the distribution grid. when pgrid is a negative value, it means that there is not enough energy in the residence, so the electrical grid must be used to supply this shortage. to arrive at the annual value of the residence’s energy bill b(total), eqs. (3) and (4) are used, where ts is the time period used (5 min) and the value of the fee f(t) depends on the tariff value at that moment. $$ {\text{p}}_{c(n)} \, = \,{\text{p}}_{{{\text{grid}}(n{)}}} \, * \,t_{s} \,\,\,\,\,({\text{if}}\,\,p_{{{\text{grid}}}} \, < \,0) $$ (1) $$ {\text{p}}_{{{\text{c}}({\text{total}})}} \, = \,\sigma {\text{p}}_{c(n)} $$ (2) $$ {\text{b}}_{\left( n \right)} \, = \,p_{{{\text{grid}}}} \, * \,t_{s} \, * \,{\text{fee}}_{\left( t \right)} \,\,\,\,({\text{if}}\,\,p_{rede} \, < \,0) $$ (3) $$ {\text{b}}_{{({\text{t}}otal)}} \, = \,\sigma {\text{b}}_{(n)} $$ (4) for the development of the system, the python programming language was used because it is known as a very intuitive syntax language and with a vast number of libraries and documentation available. for the implementation of fuzzy logic system in python, the scikit-fuzzy library [16] was used. to analyze and manipulate data in a simpler way, the pandas library [13] was used. the graphical visualization of data in python utilized was matplotlib library [13]. 6 research results and discussion the total annual production of the photovoltaic system is 4426.37 kw, while the total annual residential consumption is 6801.28 kw. after developing the system and obtaining the results of the simulations, the resulting situations are observed and compared. at first, the dg system without the fuzzy logic (commercially used by companies) consumes 4711.80 kw from the electrical distribution grid, which reduces the use of the electricity from the company by 30.72%, resulting in an annual invoice value of r$ 3312.20. the second moment, which consists of adapting [17]’s research to current technologies, to the new data analysis and python programming language, showed that the proposal consumes 3892.28 kw from the electrical distribution grid, with an annual cost of r$ 2590.25. relative to the electricity distribution company’s annual residential consumption, the reduction was 42.77%. finally, the simulation for the dg system using the fuzzy logic proposed in this paper resulted in a grid consumption of 3708.32 kw and an annual cost of r$ 2399.10, that is, a reduction of 45.48% compared to the annual consumption from the company. these data are better analyzed and compared in table 3. table 3. comparison of simulation resultsfull size table the results presented are in line with the objective of this research. in table 5, it is observed that there was a reduction in the total energy consumed by the electrical distribution grid, compared to the results obtained in the previous work. as a result of this reduction, the annual energy bill was reduced by r$ 191.15. the simulation showed that the fuzzy system identifies the period in which there is greater energy production by the photovoltaic system and uses the battery more intensively, depending on the time of day, in order to use less the distribution grid. in contrast to the simple system that follows a linear pattern of energy storage in the battery. figure 4 shows the behavior of the battery state of charge over a year, both for the simple system and for the fuzzy system. fig. 4. soc of the battery on simple (a) and fuzzy (b) systems over one year full size image it is observed that the fuzzy system requests less power from the distribution grid over time and injects power into it more uniformly. positive values indicate injection of photovoltaic power into the grid and negative values indicate power consumption from the distribution grid throughout the residence. figure 5 presents the behavior of the simple system and the fuzzy system in relation to the injection or consumption of power in the distribution grid. fig. 5. grid power flow on simple (a) and fuzzy (b) systems over one year full size image 7 conclusion the main purpose of this article was to improve the use of photovoltaic energy in the context of white tariff in brazil by using fuzzy systems. the proposed method was improved from previous work and the final setup could reduce the electricity bill for the residential consumer. in order to adjust the system parameters, some limits of the input linguistic variables were modified. in the simulation, a battery sized in a more adjusted way and with a more current technology was also used to obtain a better cost benefit for a residence. a recent configuration of photovoltaic panels was used, with more efficient electrical power and better solar absorption technology. the modifications were satisfactory for the improvement in the results, allowing the continuation of the development of this research. the results obtained brought a reduction of 14.05% in the energy consumed from the electrical distribution grid, when comparing the simple dg system developed without the use of ic with the proposed fuzzy system. this means a 31.73% reduction in the annual energy bill. this work contributes to the ci area and to the academic community based on the expansion and progression of fuzzy logic studies applied to photovoltaic systems, with the aim of demonstrating the importance of these two subjects for improving decision-making and management of photovoltaic solar energy. in addition, the study and application of an intelligent system in a scenario that simulates the real world allows, among other contributions, the development of more innovative products for the energy market and products with better cost-benefits for the final consumer. for future work, we intend to apply the proposed fuzzy systems for different energy production and consumption patterns from different cities of different brazilian regions. we intend to verify if the proposal can be applied independently of the local, or if it requires small (or large) adjustments for better performance. it is also intended the study and application of other techniques of computational intelligence in order to compare with fuzzy logic. references almeida, e., et al.: energia solar fotovoltaica: revisão bibliográfica. engenharias on-line 1(2), 21–33 (2015). http://www.fumec.br/revistas/eol/article/view/3574/1911. accessed 11 apr 2023 aneel. resolução normativa nº 1000. agência nacional de energia elétrica. brasília-df, 07 de dezembro de 2021. diário oficial da união. https://www.in.gov.br/en/web/dou/-/resolucao-normativa-aneel-n-1.000-de-7-de-dezembro-de-2021-368359651. accessed 6 apr 2023 arcos-aviles, d., et al.: an energy management system design using fuzzy logic control: smoothing the grid power profile of a residential electro-thermal microgrid. ieee access 9, 25172–25188 (2021) article  google scholar  balitskiy, s., bilan, y., strielkowski, w., streimikiene, d.: energy efficiency and natural gas consumption in the context of economic development in the european union. renew. sustain. energy rev. 55, 156–168 (2016) google scholar  dhar, r.k., et al.: power balance modes and dynamic grid power flow in solar pv and battery storage experimental dc-link microgrid. ieee access 8, 219847–219858 (2020) article  google scholar  epe, empresa de pesquisa energética (brasil). balanço energético nacional: relatório final, ano base 2021, p. 264. ministério de minas e energia, rio de janeiro (2022). https://www.epe.gov.br/pt/publicacoes-dados-abertos/publicacoes/balanco-energetico-nacional-2022. accessed 7 apr 2023 equatorial energia. cartilha online: tarifa branca (2019). https://pa.equatorialenergia.com.br/wp-content/uploads/2019/12/cartilha-online-pa.pdf. accessed 6 apr 2023 how solar power works on-grid, off-grid and hybrid systems; clean energy reviews. https://www.cleanenergyreviews.info/blog/2014/5/4/how-solar-works. accessed 2 apr 2023 ieee-pes-iss. open data sets. http://sites.ieee.org/pes-iss/data-sets/. accessed 6 apr 2023 jafari, m., et al.: development of a fuzzy-logic-based energy management system for a multiport multioperation mode residential smart microgrid. ieee trans. power electron. 34(4), 3283–3301 (2019) article  google scholar  lopes, i.l., santos, f.a.o., pinheiro, c.a.m.: inteligência artificial, 1st edn. elsevier editora, rio de janeiro (2014) google scholar  mansiri, k., sukchai, s., sirisamphanwong, c.: fuzzy control algorithm for battery storage and demand side power management for economic operation of the smart grid system at naresuan university, thailand. ieee access 6, 32440–32449 (2018) article  google scholar  mckinney, w.: python for data analysis, 3rd. edn. o’reilly media, inc., sebastopol (2022) google scholar  peña-aguirre, j.c., et al.: fuzzy logic power management strategy for a residential dc-microgrid. ieee access 8, 116733–116743 (2020) article  google scholar  simões-moreira, j.r., et al.: energias renováveis, geração distribuída e eficiência energética. [s.l: s.n.] (2017) google scholar  singh, h., lone, y.a.: deep neuro-fuzzy systems with python, 1st edn. apress, berlin (2019) google scholar  sousa, m., saraiva, f.: a fuzzy system applied to photovoltaic generator management aimed to reduce electricity bill. in: moura oliveira, p., novais, p., reis, l. (eds.) progress in artificial intelligence, epia 2019. lncs, vol. 11804, pp. 450–461. springer, cham (2017). https://doi.org/10.1007/978-3-030-30241-2_38 wang, l.: a course in fuzzy systems and control. prentice-hall inc., international edition, upper saddle river (1997) google scholar  zhao, h., et al.: fuzzy logic based coordinated control of battery energy storage system and dispatchable distributed generation for microgrid. j. mod. power syst. clean energy 3(3), 422–428 (2015) google scholar  download references author information authors and affiliations institute of exacts and natural sciences, federal university of pará, belém, pará, brazil raimunda branco & filipe saraiva authors raimunda brancoview author publications search author on:pubmed google scholar filipe saraivaview author publications search author on:pubmed google scholar corresponding author correspondence to raimunda branco . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper branco, r., saraiva, f. (2023). improved fuzzy decision system for energy bill reduction in the context of the brazilian white tariff scenario. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_29 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords fuzzy system photovoltaic production renewable energies publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature prediction of cancer-related mirna targets using an integrative heterogeneous graph neural network-based method | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper prediction of cancer-related mirna targets using an integrative heterogeneous graph neural network-based method conference paper first online: 12 october 2023 pp 346–360 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) prediction of cancer-related mirna targets using an integrative heterogeneous graph neural network-based method download book pdf download book epub emanoel aurelio vianna fabiano  orcid: orcid.org/0009-0001-7519-209x9 & mariana recamonde-mendoza  orcid: orcid.org/0000-0003-2800-10329,10  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 575 accesses abstract micrornas (mirnas) are crucial regulators of gene expression, including in diseases such as cancer. although machine learning methods have shown promise in predicting mirna-target interactions, they encounter challenges related to imbalanced classes and false positives. to tackle these issues, this study proposes a gnn-based model, using a variant of graphsage algorithm named hinsage, which integrates validated mirna-mrna and mrna-mrna interactions with cancer-related gene expression data. results show that our approach effectively learns mirna-target interaction patterns from the graph structure and node features. the model achieves 77% precision, 80% recall, 78% f1-score, and 86% roc auc on the test data. it competes well with related approaches, reaching an f1-score of approximately 90% on a common test set. thus, gnns offer a promising avenue for studying mirna-target interactions, providing balanced predictive power and improved precision through negative interaction sampling from the graph. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, and by grants from fapergs [21/2551-0002052-0] and cnpq [308075/2021-8]. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others integrative analysis of mirna-mrna expression data to identify mirna-targets for oral cancer chapter © 2022 mirna–gene activity interaction networks (migain): integrated joint models of mirna–gene targeting and disturbance in signaling pathways chapter © 2022 predicting microrna target genes using pan-cancer correlation patterns article open access 27 january 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. gene targeting genetic interaction molecular target validation mirnas molecular target identification cancer genetics and genomics 1 introduction micrornas (mirnas) are small non-coding ribonucleic acid (rna) molecules that regulate gene expression in post-transcriptional stages, inhibiting the expression of specific genes by messenger rna (mrna) silencing. recent studies indicate that a single mirna can target many different mrnas, while a particular mrna can be regulated by a set of mirnas, simultaneously or in a context-dependent manner. moreover, abnormal mirna expression has been associated with the development and progression of different human diseases, including those related to the endocrine system and cancer [10]. thus, mirnas are important actors in the complex regulatory network governing gene expression in physiological and pathological processes. unraveling mirna-mrna target interactions is a fundamental step for discovering the regulatory network governed by mirnas and their functional role within an organism. due to the numerous potential target sites for a given mirna, experimentally confirming every possible target candidate is impractical both in terms of cost and time. machine learning (ml) has been increasingly explored to predict candidate targets using a series of descriptors (i.e., features) of the mirna-target mrna interaction defined from experimentally characterized interactions and prior knowledge [13], filtering possible mirna targets. however, mirna target prediction is still considered an open problem due to several challenges. first, manually engineered features based on biological assumptions capture only partially the numerous characteristics that influence the effectiveness of an interaction between mirna and mrna target [13]. the contribution of single features to the degree of repression of the mirna target is still hard to completely determine. second, the multiplicity of mirna-target interaction patterns, often in a context-dependent manner, introduces large complexity into the problem. third, there is an inherent class imbalance issue, as the majority of validated data refers to functional mirna targets, i.e., positive examples. this generates a bias towards large false positive predictions [11]. over the past years, numerous deep learning (dl) methods have been developed, with various successful applications in biology and medicine. among these, we highlight the dl algorithms in graphs, known as graph neural networks (gnns), and their capacity for solving complex problems where data requires a representation in non-euclidean domains [1, 16]. gnns have found successful applications in identifying associations between mirnas and diseases (e.g., [4]); however, they are still little explored in the computational prediction of mirna targets. thus, in this work we aim to answer the question: can gnns precisely predict mirna targets from known mirna-mrna target interactions and biological evidence regarding mirnas and mrnas expression? we propose an approach based on the hinsage algorithm and heterogeneous graphs containing prior knowledge about validated mirna-mrna target and mrna-mrna interactions. hinsage is a derivative of the graphsage algorithm [3], an stochastic generalization of graph convolutions that can leverage node feature to efficiently generate embeddings on unseen data. moreover, instead of using manually engineered features from known mirna-target interactions, we use gene expression profiles as node features. gene expression is the final product in the process of gene regulation, which is affected by molecular factors that are usually used as predictive features individually or combined, such as sequence complementarity, sequence conservation, favorable thermodynamics, and accessibility of the mirna binding site. here, we focus on mirna-target interactions associated with cancer, motivated by the role of dysregulated mirnas in cancer pathophysiology and as potential biomarkers [10]. our main contributions are twofold. first, we propose and evaluate a prediction approach based on the graphsage algorithm that integrates an graph and expression profiles to predict mirna-target interactions. to the best of our knowlegde, this is the first work to explore gnns for mirna-target prediction in human cancer. second, we conduct a detailed investigation of several hyperparameters involved in the algorithm and decisions regarding dataset construction, evaluating their impact on the addressed prediction problem. 2 related works several tools have been developed for mirna target prediction, with ml and dl being recurrent among solutions. since the related literature is vast [13], here we review the most relevant works for the proposed approach. targetscan [9] is a widely used tool that employs linear regression and ml techniques for mirna target prediction. however, it has limited sensitivity for targets without good evolutionary conservation. pita [7] considers the accessibility of the binding site on the target, evaluating the scores for microrna-target interactions by using energy-based secondary structure prediction algorithms. both methods rely on specialist-designed features. more recent approaches include miraw [12], deeptarget [8], and deepmirtar [15], which are based on dl and aim to learn better representations directly from raw data, eliminating the need for manual feature extraction. they implement autoencoders and adopt distinct approaches for defining negative samples, such as generating mock examples or building negative examples based on experimentally verified data. we note that these approaches do not explore the graph structure and some of the works lack evaluation on independent test datasets. finally, we highlight comprehensive databases containing computationally predicted interactions: tarbase v8 [6] and mirdip [14]. these databases integrate data from various sources and provide confidence scores for their predictions. we note that a main challenge faced was the difficulty of identifying, up to the time of writing this paper, a model for mirna target prediction based on the concept of gnns. it should be noted that gnns have been previously applied in prediction tasks related to mirnas, but with a special focus on predicting the association of mirnas with human diseases (e.g., [4]) a task that has different objectives from the problem modeled in this study. moreover, a previous study has explored gnns in the mirna-target prediction problem, although focusing on data from camellia sinensis and without integrating gene expression profiles into their framework [2]. 3 methodology gnns are dl algorithms designed to operate on graph-structured data. the learning of node representations in the graph, i.e., the generation of embeddings, is achieved through iterative updates that aggregate the representations of neighboring nodes and the node itself from previous iterations. this information can be used for tasks such as link prediction (our interest) or node classification. in this section, we detail how we collected and integrated the interactions to define the graph structure and the gene expression data used as node features in the model training process, as depicted in fig. 1-a. we also explain the proposed learning approach based on the hinsage algorithm (fig. 1-b). fig. 1. summary of the approach proposed. a) the graph consists of mirnas (red node) and mrnas (pink nodes) with their features vectors, and the known interactions among them (solid lines). b) the inductive learning process by the hinsage algorithm considering potential mirna-mrna interactions (dashed lines) using sampled 2-hop neighbors. (color figure online) full size image 3.1 data collection, preprocessing and integration mirna-mrna and mrna-mrna interaction data. we obtained interactions from rnainter v4.0 [5], a comprehensive database that integrates rna interactome data, including experimentally validated and computationally predicted interactions. the current version has over 47 million annotated interactions for 156 species. rnainter categorizes the type of experimental evidence for each interaction as strong or weak, and also calculates a confidence score per interaction based on characteristics such as experimental evidence confidence, scientific community confidence given by the number of article citations, and the number of different tissues or cells in which the interaction was identified – all integrated using a sigmoid function. we downloaded mirna-mrna and mrna-mrna interactions related to humans, removing those with inconsistent values, such as undefined mirna or mrna identifiers, and those derived from computational prediction to improve the quality of the training data. gene expression data in cancer. gene expression data for mirnas and mrnas was obtained from the cancer genome atlas (tcga) project, one of the leading consortia in cancer genomics. the data is publicly accessible through the gdac firebrowse portalfootnote 1. we downloaded gene expression datasets for primary solid tumor samples (pt) and normal tissue (nt) samples (i.e., non-tumor) preprocessed following well-established bioinformatics protocols. we only considered those cancers that did not exhibit significant imbalance between the number of tumor and normal tissue samples for mirnas and mrnas expression, resulting in 15 tumor types (see supplementary materialfootnote 2). the data was transformed to the log2 scale and a measure of differential expression, the log fold change (logfc), was calculated for each mirna and mrna in relation to each tumor type. the logfc expresses the ratio between the mean gene expression in pt vs. nt samples for a specific mirna or mrna, in a specific cancer type. a higher absolute value of logfc is assumed to indicate greater biological relevance in the studied domain. the step of summarizing expression patterns through logfc values generates a feature vector of length 15 for each mirna and mrna in the graph. data integration for graph generation. we integrated the interaction data obtained from the rnainter database with the gene expression data collected from firebrowse using the unique identifier available for each mirna and mrna in the collected database. during this step, mirna-target interactions without any identified expression records were also removed. this removal aimed to mitigate the induction of inconsistent results during the subsequent training of the proposed model. the final interaction network comprises over 100,000 mirna-target interactions and nearly 50,000 mrna-mrna interactions. the network has a total of 2,617 mirna nodes and 17,252 mrna nodes. the number of interactions according to interaction type, evidence type, and score threshold is shown in fig. 2. fig. 2. summary of collected mirna-mrna and mrna-mrna interactions. full size image 3.2 model training and evaluation in this work, we are interested in conducting a link prediction task handling a large amount of data, integrating node features into the learning process, and working with a heterogeneous graph. based on the literature [16], we chose a variant of graphsage [3] extended for heterogeneous graphs, named hinsagefootnote 3. the algorithm generates embeddings by sampling and aggregating features from the local neighborhood of a node, instead of training individual embeddings focused solely on the node. the number of observed neighbors can be parameterized, both in terms of the number of hops (k) and the number of neighboring nodes to be sampled at each hop. this behavior can be observed in fig. 1, where the value \(k=2\) represents the number of hops (i.e., depth of search or number of hinsage layers). the neighborhood aggregation strategy is applied such that neighbors are randomly selected and fused together by edge type. the predictive models were developed using the hinsage implementation provided by the stellargraph library. the base graph was built as explained in sect. 3.1. the hyperparameters were initially defined as follows: batch size equal to 200; 300 epochs of training; learning rate of 0.001 with adam optimizer; two hops (\(k=2\)); number of nodes sampled per hop equal to [8,4]; hinsage layer sizes setup as [32,32]; binary cross-entropy cost function; relu activation function; feature update rule based on the graph convolutional network (gcn) aggregator. variations in the hyperparameter values were tested and evaluated in a series of experiments that will be detailed later. our experiments were based on ten random divisions of the data into training, validation, and test sets. considering our inductive domain, the training, validation, and test sets are subgraphs derived from reductions of the original graph, aiming to generate independent data for the different stages involved in model development (table 1). we used the method edgesplitter provided by stellargraph, which takes as input both the graph to be subdivided and a percentage p relative to the total number of edges in the provided graph, determining the number of positive and negative edge samples to be sampled from it. positive edges are sampled from the actual set of edges present in the graph to maintain graph connectivity. negative edges are randomly created by sampling pairs of mirna-mrna that are not connected by an interaction in the original graph. this sampling of negative examples aims to improve the model’s ability to identify false positive edges, by mitigating class imbalance. the value of the hyperparameter p was adjust according to the set of interactions included in the creation of the graph, since it needs to be compatible with the number of negative interactions that can be randomly generated from the input graph. model performance evaluation was based on the accuracy, precision, recall, f1-score, and roc auc metrics. table 1. sizes of training, validation, and test sets.full size table 4 experiments two main sets of experiments were defined for the development of this study. the first set, s1, is focused on analyzing the potential of the hinsage algorithm for mirna target prediction using a graph-based learning approach. we evaluate the influence of aspects related to the construction of the interaction dataset and the configuration of the algorithms’s hyperparameters on the results achieved by the hinsage model. we define 20 experimental scenarios organized into seven groups (table 2), each of which with a specific experimental goal: g1: evaluation of the impact of using different datasets to define the base graph used in the model training, constructed by removing mrna-mrna interactions and with different score filters; g2: evaluation of the impact of removing mrna-mrna interactions from the base graph; g3: evaluation of the impact of reducing the number of training epochs; g4: evaluation of the impact of varying the batch size; g5: evaluation of the impact of varying the number of sampled neighboring nodes; g6: evaluation of the impact of varying the size of the hidden layers; g7: evaluation of the impact of varying the learning rate. given the high computational cost involved in running each experiment, at the end of each experimental group, we identified the most promising configuration to be employed in the next group. thus, best hyperparameter values found in previous experiments are carried over in the next group of experiments. table 2. experimental scenarios investigated in the set of experiments s1.full size table the second set, s2, aims to compare the approach developed in this study (based on the best model generated in the set s1) with other methods proposed in the literature. due to lack of similar approaches, it was not possible to perform an equivalent comparison based on end-to-end graph analysis using gnns. we conducted the comparison by focusing on the results of the models’ predictions for the test data, considering mirna-target interactions found in common between the test set created in this study and the test sets used in the related works. we used the predictions provided by the authors of the original papers. to avoid greatly restricting the size of the common dataset due to the intersection of multiple methods, our comparison was performed pairwise with each selected related work. since most of related works do not provide predicted probabilities, we did not use the roc auc metric in this analysis. 5 results and discussion the next sections present and discuss the results for the two sets of experiments. we note that due to space constraints, we focus on the most meaningful findings. the complete results are provided as supplementary materialfootnote 4. 5.1 s1: exploring the predictive performance of hinsage in the experiments of g1, we aimed to gain a better understanding of the model’s behavior with different datasets used for training. we compared the original dataset, including both mirna-mrna and mrna-mrna interactions (experiment 00), with a version excluding mrna-mrna interactions (experiment 01) using the default hiperparameters values (sect. 3.2). the results obtained during training showed promising numbers, with an accuracy close to 75% for both cases. however, upon examining the accuracy and loss throughout the training epochs, we identified characteristics of overfitting occurring from epoch 50 onwards. moreover, we did not observe any significant performance change by removing mrna-mrna interactions, thus, we maintained them. fig. 3. performance evaluation for selected experiments of g1 in terms of mean and standard deviation for 10 random training, validation and test sets. full size image we also evaluated the influence of various filters applied to the confidence score associated to each interaction. experiments 02 to 05 filtered interactions based on thresholds of 0.2, 0.3, 0.4, and 0.5 (table 2), thus reducing the number of interactions in the graph (table 1). we noticed a sequence of performance improvements for both training and testing data at the end of the 300 training epochs. the initial filter (score \(>=\) 0.2 in experiment 02) already showed improvements compared to experiment 00. however, we also observed a highly negative overfitting effect in some cases, such as in experiment 02 and, particularly, in experiment 05. we highlight the results of experiment 04, which proved to be the most promising in group g1, demonstrating an apparent improvement without a significant tendency towards overfitting. figure 3 summarizes the training, validation, and test results for the main experiments of g1. it can be concluded in advance that applying the score filter does indeed influence the achieved results, and reducing the number of interactions described as strong (which decrease for larger thresholds) hinders the prediction of interactions, resulting in significant dispersion in the obtained results. next, in g2, we aimed to analyze whether the use of mrna-mrna interactions could influence the results obtained with filtered datasets. therefore, experiments 06 and 07 correspond to experiments 04 and 05, respectively, but considering only mirna-mrna interactions. once again, removing these interactions influenced the results, but the impact was not consistent between both experiments. we noticed that removing mrna-mrna interactions introduced overfitting in the experiment (experiments 06 vs 04), whereas when comparing experiments 07 vs 05, the training became more stable and showed less overfitting after removing these interactions, but without noticeable increase in the predictive performance. hence, experiment 04 (g1) remained the best scenario. fig. 4. performance evaluation for g3 in terms of mean and standard deviation for 10 random training, validation and test sets. full size image since we observed some signs of overfitting, in g3 we evaluated the reduction in the number of training epochs to mitigate overfitting and reduce the chances of predicting false positive interactions. results are shown in fig. 4 and they can be compared to experiment 04, as the other configurations remain the same between the two experiments. we can observe that the training curves for 100 epochs exhibit more appropriate behavior, with close training and validation performances. this means that the confidence in the generalization power of the generated model becomes stronger. moreover, the performances in the test set were quite satisfactory, although there was a slight reduction in values compared to the performance on the test data for the model generated in experiment 04. the precision varied mostly with values above 75% and the roc auc had a median value close to 85%. the following experiments will adopt 100 epochs. experiments of g4 aimed to initiate the investigation of the hyperparameter values involved in the hinsage algorithm by exploring variations in the batch size, which was initially set to 200. four distinct values were tested: 1 (experiment 09), 50 (experiment 10), 100 (experiment 11), and 300 (experiment 12). a batch size of 1 showed a training behavior described as stochastic, presenting overfitting within the first ten epochs of training. this characteristic was also present with a batch size of 50, which showed a large standard deviation for the first 25 epochs. experiments 11 and 12, obtained more appropriate results. the latter, highlighted in fig. 5, had a slight performance improvement when compared to experiment 08, particularly in terms of performance variation across the 10 executions. the following experiments will apply a batch size of 300. fig. 5. performance evaluation for selected experiment in g4 in terms of mean and standard deviation for 10 random training, validation and test sets. full size image in g5, we investigated the impact of varying the number of sampled neighbor nodes in the hinsage algorithm on the generation of embeddings (i.e., num_sample). we kept the number of hops fixed at \(k=2\) and tested two variations: [6,3] (experiment 13) and [12,6] (experiment 14). we observed that reducing the number of sampled neighbors (experiment 13) resulted in increased performance dispersion between training and validation, while increasing the number of sampled neighbors (experiment 14) led to a decrease in the dispersion of these performances. moreover, experiment 14 (fig. 6) also showed improved performance compared to experiment 12. the medians for all metrics were approximately 80% or higher. a more balanced performance between recall and precision was observed, resulting in an increase in the f1-score metric. the roc auc metric had values very close to 90%. the better performances in the training, validation, and test datasets motivated the selection of experiment 14 as the best scenario so far. fig. 6. performance evaluation for selected experiment in g5 in terms of mean and standard deviation for 10 random training, validation and test sets. full size image fig. 7. roc curves for the test sets in experiment 14. full size image finally, experiments in g6 and g7 aimed to compare variations in the size of hidden layers and in the learning rate, respectively. our results indicate that increasing the dimensions of the hidden layers (experiments 15 and 16) led to increased overfitting and had no positive effect on the maximum performance obtained by the model. on the other hand, reducing the size of the hidden layers (experiment 17) caused the model to loose predictive power compared to the other evaluated scenarios, but mainly experiment 14. regarding the analysis for the learning rate, when its value was increased to 0.01 and 0.005, we observed more signs of overfitting without effective improvement in the achieved predictive performance. conversely, decreasing the learning rate to 0.0001 had a detrimental effect on the predictive model’s performance, causing underfitting of the model. in this case, the learning progress was very slow and the model failed to converge adequately for the fixed number of epochs (100). among all the evaluated scenarios in the set of experiments s1 (table 2), the best model was the one originated from experiment 14 and it will be further employed in the experiment set s2. figure 7 shows the roc curves and corresponding roc auc values for the test data considering the 10 runs of the 3-way holdout. the model’s performance appears to be quite stable, with roc auc values ranging from 86% to 88%. additionally, the curve demonstrates a suitable growth, with a steeper increase in tpr (y-axis) compared to fpr (x-axis). for example, a tpr of 80% yields an fpr close to 20%. 5.2 s2: comparison of hinsage model with related works the related works for which a direct comparison was possible with our test data were miraw [12], targetscan [9], and pita [7]. we also used computationally predicted interactions from tarbase v8 [6] and mirdip [14]. to conduct the comparison, we intersected our test data with the pre-computed target prediction results provided by each tool or method. the prediction datasets were obtained from the official platforms of the respective works (accessed on january 10, 2023). some preprocessing steps were required in some cases. for targetscan, predictions are associated to a context++ score and targets with lower context++ scores are considered more representative. however, there is no clearly defined cutoff point to determine whether a given mirna-mrna interaction should be classified as positive or negative. therefore, we decided to apply three distinct thresholds: –0.5, –0.3, and –0.2. interactions with context++ scores lower than the applied threshold were classified as positive. for tarbase, we only used interactions related to humans and classified as ‘positive’ by the tool, indicating that they represent functional mirna-mrna interactions. for mirdip, each interaction has a score assigned and described as: very high, high, medium, and low. very high indicates a high probability of the mirna-mrna interaction, while the probability decreases towards the low designation. due to the large volume of data, we obtained only the dataset with a score value described as very high, inferring all resulting interactions as positives. the results for this analysis are shown in table 3. for all the related works with which a direct comparison was possible for the test instances, our proposed model based on the hinsage algorithm achieved results that are very close to or outperform other approaches. excluding tarbase and mirdip, which are databases rather than ml-based prediction methods, our work presents highly promising results. in almost all the performed comparisons, our model achieved an f1-score above 90%, indicating that the model developed in this work can predict positive and negative interactions with a high degree of confidence. however, it should be noted that none of the related works focus on identifying mirna-mrna interactions associated with cancer. this limitation exists in comparing our results with the literature, given the difficulty of finding representative datasets for this context. other critical challenges faced for this comparison were (i) the wide variety of predictive features used by previous works based on ml, turning the creation of the training dataset a slow and arduous process and (ii) the difficulty in running methods that are not made available through online tools, requiring local installation of the application and its dependencies. table 3. performance comparison with related worksfull size table 6 conclusion the premise that guided the present work is that the development of a computational strategy based on methods capable of learning mirna regulation patterns from the analysis of heterogeneous interaction networks (i.e., hinsage) could provide more robust predictions to noise and deficiencies in training data, as well as better capture the complexity involved in mirna function. the use of graph-based dl algorithms not only introduces the advantage of exploring the graph structure in a much more comprehensive and robust way to discover patterns in mirna-mrna interactions, but also enables efficient handling of the data imbalance problem prevalent in this domain. our experiments suggest that our approach was effective, reaching competitive performance and controlling false positive rates. although there is no known equivalent work at the moment that can serve as a baseline for comparison to our work, the comparisons performed with computationally predicted interactions from other ml-based methods or deposited in databases have shown that the approach based on heterogeneous graphs and the application of the hinsage algorithm are highly promising. the performance achieved by our model stood out in various scenarios and demonstrated a suitable balance between recall and precision. therefore, we believe that the objective of exploring the potential of the hinsage algorithm for mirna target prediction in cancer has been successful, and the results obtained in this study serve as motivation for further exploration of this approach. among possible directions for future work, we outline the application to different human diseases, the inclusion of other types of interactions in the heterogeneous networks, and the need to expand the comparison with other ml/dl algorithms or bioinformatics tools for mirna-target prediction. notes 1.http://firebrowse.org/. 2.https://encurtador.com.br/eilza. 3.https://stellargraph.readthedocs.io/en/stable/hinsage.html. 4.https://encurtador.com.br/eilza. references cai, h., zheng, v.w., chang, k.c.c.: a comprehensive survey of graph embedding: problems, techniques, and applications. ieee trans. knowl. data eng. 30, 1616–1637 (2018) article  google scholar  feng, h., xiang, y., wang, x., xue, w., yue, z.: mtagcn: predicting mirna-target associations in camellia sinensis var. assamica through graph convolution neural network. bmc bioinf. 23(1), 1–18 (2022) article  google scholar  hamilton, w., ying, z., leskovec, j.: inductive representation learning on large graphs. in: advances in neural information processing systems, pp. 1024–1034 (2017) google scholar  ji, c., wang, y., ni, j., zheng, c., su, y.: predicting mirna-disease associations based on heterogeneous graph attention networks. front. genet. 12, 727744 (2021) article  google scholar  kang, j., et al.: rnainter v4.0: rna interactome repository with redefined confidence scoring system and improved accessibility. nucleic acids res. 50, d326–d332 (2022) article  google scholar  karagkouni, d., et al.: diana-tarbase v8: a decade-long collection of experimentally supported mirna-gene interactions. nucleic acids res. 46(d1), d239–d245 (2018) article  google scholar  kertesz, m., iovino, n., unnerstall, u., gaul, u., segal, e.: the role of site accessibility in microrna target recognition. nat. genet. 39(10), 1278–1284 (2007) article  google scholar  lee, b., baek, j., park, s., yoon, s.: deeptarget: end-to-end learning framework for microrna target prediction using deep recurrent neural networks. in: proceedings of the 7th acm international conference on bioinformatics, computational biology, and health informatics, pp. 434–442 (2016) google scholar  lewis, b., shih, i.h., jones-rhoades, m., bartel, d., burge, c.: prediction of mammalian microrna targets. cell 115, 787–798 (2004) article  google scholar  peng, y., croce, c.m.: the role of micrornas in human cancer. signal transd. target. tpherapy 1(1), 1–9 (2016) google scholar  pinzón, n., et al.: microrna target prediction programs predict many false positives. genome res. 27(2), 234–245 (2017) article  google scholar  pla, a., zhong, x., rayner, s.: miraw: a deep learning-based approach to predict microrna targets by analyzing whole microrna transcripts. plos comput. biol. 14, e1006185 (2018) google scholar  schäfer, m., ciaudo, c.: prediction of the mirna interactome established methods and upcoming perspectives. comput. struct. biotechnol. j. 18, 548–557 (2020) article  google scholar  tokár, t., et al.: mirdip 4.1 integrative database of human microrna target predictions. nucleic acids res. 46, d360–d370 (2017) article  google scholar  wen, m., cong, p., zhang, z., lu, h., li, t.: deepmirtar: a deep-learning approach for predicting human mirna targets. bioinformatics 34(22), 3781–3787 (2018) article  google scholar  zhou, j., et al.: graph neural networks: a review of methods and applications. ai open 1, 57–81 (2020) article  google scholar  download references author information authors and affiliations institute of informatics, universidade federal do rio grande do sul, porto alegre, rs, brazil emanoel aurelio vianna fabiano & mariana recamonde-mendoza bioinformatics core, hospital de clínicas de porto alegre, porto alegre, rs, brazil mariana recamonde-mendoza authors emanoel aurelio vianna fabianoview author publications search author on:pubmed google scholar mariana recamonde-mendozaview author publications search author on:pubmed google scholar corresponding author correspondence to mariana recamonde-mendoza . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper vianna fabiano, e.a., recamonde-mendoza, m. (2023). prediction of cancer-related mirna targets using an integrative heterogeneous graph neural network-based method. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_23 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords microrna target prediction machine learning bioinformatics graph neural networks publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature interpreting classification models using feature importance based on marginal local effects | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper interpreting classification models using feature importance based on marginal local effects conference paper first online: 28 november 2021 pp 484–497 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) interpreting classification models using feature importance based on marginal local effects download book pdf download book epub rogério luiz cardoso silva filho  orcid: orcid.org/0000-0002-0297-750710,11, paulo jorge leitão adeodato11 & kellyton dos santos brito  orcid: orcid.org/0000-0002-6883-865712  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 890 accesses 5 citations abstract machine learning models are widespread in many different fields due to their remarkable performances in many tasks. some require greater interpretability, which often signifies that it is necessary to understand the mechanism underlying the algorithms. feature importance is the most common explanation and is essential in data mining, especially in applied research. there is a frequent need to compare the effect of features over time, across models, or even across studies. for this, a single metric for each feature shared by all may be more suitable. thus, analysts may gain better first-order insights regarding feature behavior across these different scenarios. the \(\beta\)-coefficients of additive models, such as logistic regressions, have been widely used for this purpose. they describe the relationships among predictors and outcomes in a single number, indicating both their direction and size. however, for black-box models, there is no metric with these same characteristics. furthermore, even the \(\beta\)-coefficients in logistic regression models have limitations. hence, this paper discusses these limitations together with the existing alternatives for overcoming them, and proposes new metrics of feature importance. as with the coefficients, these metrics indicate the feature effect’s size and direction, but in the probability scale within a model-agnostic framework. an experiment conducted on openly available breast cancer data from the uci archive verified the suitability of these metrics, and another on real-world data demonstrated how they may be helpful in practice. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others comparison of feature importance measures as explanations for classification models article open access 03 february 2021 can metafeatures help improve explanations of prediction models when using behavioral and textual data? article 08 june 2021 grouped feature importance and combined features effect plot article open access 18 june 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization information model linear models and regression machine learning predictive markers statistical learning 1 introduction explainable artificial intelligence (xai) is an emerging research area that enables black-box models to become trustworthy for humans. with a growing interest in explaining machine learning (ml) models to fill the gap between interpretability and prediction performance, over the past few years, many techniques have been proposed, and explainability has become an essential subfield of ml [1]. this combination has helped the spread of ml in applied research areas even more, such as in education, healthcare, finance, and social media. for instance, simply classifying a patient in a hospital into a particular health status is not particularly helpful. it is more desirable to investigate the conditions that have contributed to this [2] and would even become compulsory should any legal matters arise. additionally, in the education domain, discovering why a student will drop out is more valuable than just predicting it [3, 4] because, as in medicine, the treatment depends on the probable cause. similarly, auditing the behavior of machine learning bot detectors in social media is valuable in order to improve the models for new kinds of bots [5, 6]. explanations may be expressed in many forms. for classification problems, specifically, feature importance is widely used [7, 8], and demonstrates the global impact of single features in the model predictions. a wide variety of different methods with different feature importance representations have been proposed for this purpose [9,10,11]. despite these advances, there is still a lack of understanding as to how these methods are related and whether one method is preferable over another [12]. in applied research it is often necessary to track feature importance over time, across models or even across studies. therefore, a method that enables the global feature contribution to be represented by a single metric is more suitable than multiple metrics or graphical representation, otherwise interpretability may be challenging to understand when handling several features in several models. a standard single metric of feature importance is the coefficients of additive models such as linear and logistic regressions. this coefficient represents the weight of each feature in the additive function, which describes the relationship among features and outcome. however, for generalized linear models (glm), which involve transformations of this linear predictor into other discrete outcomes, such as logistic regressions, the coefficient interpretability is not straightforward. moreover, the coefficients are highly sensitive to unobserved heterogeneity [13] and data scale [14]. to circumvent these limitations, marginal effects (mes) have long been proposed [14] in the traditional statistical literature. marginal effects use the prediction function to calculate the differences in probabilities of the outcome when the features partially change from one specified value to another. however, mes fail to isolate the feature effect when data are correlated [15]. this problem arises when the computation of the feature effect uses conditional distribution. thus, since correlated features move in tandem, it is unable to distinguish which feature value changes influence the model probabilities. for black-box models however, the permutation feature importance (pfi) derived from tree-based algorithms is a standard single metric used to report feature contribution in classification problems. however, this kind of metric is linked to model error, which cannot be a metric of interest for analysts [15]. furthermore, it does not report the direction of the feature effect, which may be critical for actionable research. recently, shapley additive explanations (shap) [12] have been pointed out as the most common explainability technique in organizations [1]. this technique uses game theory (shapley values) to measure the contribution of each data point to each feature value. thus, it can deliver explanations in fine grain, and by means of the average, report the global feature contributions, including their direction. unfortunately, calculating the exact shapley value is computationally expensive [15], and several approximations have been proposed [16]. moreover, both pfi and shap are permutation-based techniques, and so, they are able to randomly sample from the marginal distribution considering unrealistic data points that are not present on training data. therefore, they extrapolate in areas where the model was trained with either little or no training data, which may cause misleading explanations [17]. recent advances in the interpretable machine learning field, such as accumulated local effects (ale) plots, have put forward relevant contributions in this direction. they have shed light on detecting a more reliable feature effect with low computational cost when features are correlated. however, ales have only been used to visualize the feature effects across different values by plots, which are not visually friendly when the analyst compares the feature importance across multiple models. thus, this paper proposes new metrics of feature importance as a valuable option when compared to those already in existence. they allow direct interpretation in the probability scale, are more realistic when dealing with correlated data and are modeled in a model-agnostic framework. although these metrics may be extrapolated for any class of supervised models, in this paper they are focused only on binary classification. experiments use open-access data from the uci archive to introduce differences among provided metrics for logistic regression coefficients and random forest permutation feature importance. lastly, real-world data are used to demonstrate how they may be helpful in a practical problem. the remainder of this paper is organized as follows: sect. 2 provides a background of the theory related to this work. section 3 introduces and explains the proposed metrics. section 4 presents the experiments, results, and interpretation, and lastly, sect. 5 summarizes the main findings, future work, limitations, and conclusions. 2 background there are several goals for explaining prediction models. in this paper, the main goal of this paper is to support applied research providing single metrics that, in a more realistic scenario, are able to report the overall contribution of model features. hence, this section defines feature importance, and reports the main existing metrics and methods for this goal, which are directly related to this work. 2.1 feature importance the most common explanation for the classification model is feature importance. also known as the feature-level interpretation or saliency method, the method is the most well-studied explainability technique. it explains the decision of an algorithm by assigning values that reflect the importance of input components in their contribution to the decisions. regardless of the mechanism to calculate it, its common meaning is related to the individual contribution of the corresponding feature for a particular classifier [8]. this individual contribution may be derived from the global perspective, where the feature importance is related to the whole model, and from the local perspective, where the importance is derived for a specific data point [18]. moreover, it may be internal to the model (intrinsically) as the coefficients of linear models, or by applying methods that analyze the model after training (post hoc). another criterion to classify these methods is related to their generalizability, whether they are model-specific or model-agnostic. while every intrinsic method is specific, all model-agnostic work is in a post hoc framework [15]. 2.2 marginal effects marginal effects are a general concept and have different meanings depending on the discipline. this subsection defines it according to econometrics, as the “additional” effect. on the other hand, in subsect. 2.3, the word “marginal” is related to the probability distribution of a feature as well. given a features set x = (\({x}_{1}\), \({x}_{2}\)), and the predicted function \(\widehat{y} =\widehat{f}\left(x\right)\), the marginal effects (me) with respect to \({x}_{1}\) at a specific value corresponds to the changes in the outcome \(\widehat{y}\), given that \({x}_{1}\) changes in one unit. in other words, it is the first derivative of \(\widehat{f}\left(x\right)\) with respect to a \({x}_{1}\) at a specified value of the input space. however, if \({x}_{1}\) is discrete or binary, the computation is more straightforward, and the finite difference is applied, rather than the derivative [4]. for linear and additive models, mes are constant across the feature values and are exactly the same as the regression estimated slopes (coefficients) [14]. however, for glm models mes take different values across feature distribution. for a logistic regression model, mes reflects the logit shape, and are small when the probability is close to 0 or 1 and relatively large when close to 0.5 [18]. thus, to summarize the mes of \({x}_{1}\) the average of all mes (ames) is commonly used [14]. moreover, there are other alternatives that may be more suitable depending on the researcher’s questions. summary metrics of me average marginal effects (ames) are how much the outcome y changes on average when \({x}_{1}\) varies in small changes. thus, the derivative is computed for every small change on \({x}_{1}\) for every data point and averaged. in practice, the numerical derivative is implemented across the observed values of \({x}_{1}\). a step \((h)\) is defined for continuous features, and the mes become close to their theoretical value on the limit, as h tends to 0. the equations below demonstrate this beyond the simplified computation of ames. $$me=\widehat{f}\left(x\right)=\underset{h\to 0}{\mathrm{lim}}{\left(\frac{f\left(x+h\right)-f\left(x\right)}{h}\right)}$$ (1) $$ames = \frac{1}{n}\sum_{i = 1}^{n}{me}_{i}$$ marginal effect at the mean (mem) is simply the computation of the mes around the mean of the feature distribution. in practice, mem is close to the ame if \(\widehat{f}\left(x\right)\) is not too noisy and more feasible to compute, since evaluating the derivatives at the means is easier than taking the mean of each derivative [19]. marginal effect at the representative value (mer) is a simplification of mem calculation for a value that could be an interesting operation point for the research domain. the marginal effect is calculated for each variable at a particular combination of \(x\) values. thus, mer provides a means to understand and communicate model estimates at theoretically important combinations of feature values [20]. these metrics were essential to shed light on those proposed in this paper since they are based on solid statistical theory [14]. in addition to xai advances, it is possible to report a less-biased feature effect which could play an important role for ml applied researchers. 2.3 marginal local effects the concept of local effects was brought in [21] and is a fundamental part of their accumulated local effects (ales) plots. ale plots were presented as an alternative for visualizing the effects of features in black-box supervised learning models instead of partial dependence plots (pdps). the pdps, introduced by friedman [22], are widely used to visualize the influence of features in supervised ml models, and have even been considered a causal interpreter for black-box models [23]. for a prediction function \(\widehat{y}=\widehat{f}\left(x\right)\), where \(\widehat{y}\) is a scalar response variable and x = (\({x}_{1}\), \({x}_{2}\)), pdps illustrate the relationship between \({x}_{1}\) and the outcome, marginalizing \(\widehat{f}\left(x\right)\) over the distribution of \({x}_{2}\). hence, the pdps function at a particular value of \({x}_{1}\) represent the average prediction from \(\widehat{f}\left(x\right)\) if all data points take that value for \({x}_{1}\). this process takes into account unlikely combinations of x, building unrealistic plots when data are dependent. as with mes, ales use the conditional instead of marginal distribution. thus, to overcome the intrinsic problems of mes, as mentioned in subsect. 2.2, ales use the averaged differences in \(\widehat{f}\left(x\right)\) across intervals of the training data (local effect). this hack allows the extraction of isolated effects of features within the intervals. lastly, it accumulates this averaged local effect and center subtracting the mean using the equations below. (2) (3) where \(k\) is the interval of data and \({n}_{j}\)(\(k\)) is the neighborhood. hence, the ale method calculates the differences in predictions, whereby the features of interest are replaced by grid values of \(z\). the difference in prediction is the effect that features have for an individual instance in a specific interval. the sum on the right in (2) adds up the effects of all instances, which is divided by the number of instances in the interval \(k\) to produce the average. finally, the ale is vertically centered (3) in the sense that the mean ales of \({x}_{j}\) and \({x}_{l}\) on are both zero. figure 1 presents a better insight into the computation of the local effect for the function \(\widehat{f}\left({x}_{1},{x}_{2}\right)\). the feature range of \({x}_{1}\) was subdivided into \(k\) bins with roughly the same number of points indexed by \(n(k).\) focusing on bin \(n(4)\), for each point falling into this bin, \(\widehat{f}\left({x}_{1},{x}_{2}\right)\) have their \({x}_{1}\) held by the left and right endpoints of the interval \({z}_{3}\) and \({z}_{4}\). next, the differences of the predictions of these points were averaged by dividing their sum by the number of points in \(n\left(4\right).\) the same was carried out for all intervals and summed up. finally, the expectation over \(p\left({x}_{1}\right)\) was subtracted. fig. 1. illustration of the ale estimation. excerpt from [21] full size image 2.4 summary of metrics for explainability marginal effects have been discussed in traditional statistical literature as an alternative for the coefficients of the glm models due to their direct interpretability on the probability scale. in addition, they are less sensitive to the differences of data scale and unobserved heterogeneity than coefficients [10]. summary metrics of mes use the conditional distribution and are not robust against correlated data. in the field of explainable machine learning field, pdps have been widely used to report the feature effect. however, they extrapolated their results when using the density function and took into account an unreliable combination of data in their computation. ale plots recently brought the new concept of local mes and overcame both the aforementioned problems. thus, this paper claims that the ales theory [21] is a good starting point together with the existing traditional metrics of mes [14] to propose new global features importance which is able to fill the gap of robustness regarding the size and direction of feature effects on a binary classification model. 3 proposed metrics this section proposes four new metrics that are suitable for applied research when comparing the feature effects across multiple models. these metrics are based on the me and the shape of ale plots that may be used to report the feature contribution on binary classification problems. three of them possess explainability in terms of size and direction of the feature effect, while one accounts only for the contribution amount. 3.1 average uncentered ale (aua) average is a natural metric to summarize a distribution (first-order momentum) and leverages great insights regarding the size and directions of the feature mean effects. (4) this is close to the mes for linear models and ames for glms since it uses the average conditional distribution of the observed training data. however, it accounts for each local effect and may be somewhat different for noisy models. 3.2 uncentered ale at a specific value (uas) the uas is an arbitrary choice of a specific value to calculate the uncentered ale. in practice, it requires splitting the data into k intervals and calculating the uncentered ale with (3) up to the interval where the choice values fall. this metric may be helpful when the analyst has sufficient domain knowledge or is verifying hypotheses. (5) 3.3 maximum uncentered ale (mua) this metric is more actionable and consists of extracting the maximum change in predicting the outcome for the feature \({x}_{i}\). as the maximum may be positive or negative related to the outcome, it requires a previous absolute comparison in order to achieve the highest value. (6) 3.4 ale absolute average (aabsa) this metric is a non-directional metric and highlights the overall feature importance. it measures, on average, how far the prediction changes away from the ale average. unlike the others, the centered ale is considered, which has a mean zero. (7) 3.5 summary of the novelty of the proposal in order to clarify the novelty and issues addressed by the proposal, table 1 summarizes the main differences of each proposed metric and of those that already exist. in particular, the βcoefficients of logistic regression (lr) and permutation feature importance (pfi) from tree-based algorithms were considered, both widely used in the machine learning field as a global metrics and reported by a single parameter. table 1. differences among proposed methods and similar existing methods in the ml fieldfull size table 4 experiments to introduce the new metrics, two experiments were conductedfootnote 1. the goal of the experiments was two-fold. first, to compare the proposed metrics in this paper with intrinsic model metrics. more specifically, it considered the widespread \(\beta\)-coefficients and the permutation feature importance derived from lr and random forest (rf), respectively. thus, it was possible to evaluate whether or not the metrics highlight the features in a similar manner. an open-access breast cancer dataset was used. this dataset is available ready for modeling binary classification and is a linearly separable problem with highly correlated data [24]. second, a dataset was used from the national brazilian test for secondary education (enem), and the school census from the 2009–2019 period. the goal here was to demonstrate how the proposed metrics may be helpful in a real-world problem. thus, a data mining solution was developed to explore which and how the most important variables associated with school performance behave over the years. the report aimed at yielding valuable and actionable results for decision-makers through a single feature importance metric. these summary measures may enhance the analytical ability of the researcher when comparing the feature effects across supervised models, whichever the algorithm chooses to fit the data. the alibi package [25], which has implemented the ale plots, supports the computation of the metrics proposed in this paper. therefore, all the mechanisms intrinsic to the ales theory, such as the interval definition, numerical derivation, and the computation of the local effects, follow the software implementation. in this paper, the performance of the models was not reported since the goals were limited to analyzing the model explicability. 4.1 breast cancer data the breast cancer data included benign and malignant cell samples from 369 patients, 212 with cancer, and 157 with fibrocystic breast masses. each sample contained thirty features, and lr and rf predicted the patient class in a 5-fold cross-validation setting with random sampling stratified by the target class. therefore, both algorithms were applied for the same folds, and the mean was adopted as feature importance for each metric. figure 2 shows the lr coefficients in red and the four proposed metrics. for the metrics that illustrate the direction of the relationship, there is a high correlation and close magnitude. however, there were some discrepancies. this could have been due to highly paired correlated data (e.g. perimeter vs. radius), and so logistic regression arbitrarily chose one of these (e.g. radius) to highlight the coefficient [26]. also, the p-values were not checked in this experiment, and maybe some highlighted coefficients were statistically insignificant. however, neither of these are of concern for our metrics. fig. 2. β-coefficients and the proposed metrics of lr model for the breast cancer data full size image figure 3 illustrates the permutation feature importance from the rf and the proposed metrics. the features set highlighted by all metrics is similar with a high correlation. in addition, surprisingly, the aabsa is fairly close to the permutation feature importance. both are only positive and only account for the amount of feature contributions. however, they are built differently. while aabsa metrics the average change from the expected ales mean (ale 0), permutation is related to increasing the prediction error after permuting a feature. hence, it may be due to the simplicity of the classification task on the cancer data [24]. fig. 3. feature importance and the proposed metrics of rf model for the breast cancer data full size image 4.2 brazilian secondary educational data the second part of the experiment demonstrates how single feature importance may be helpful in practice. the data was taken from the 2009–2019 period of the largest test for secondary education in brazil. the dataset contained demographic and socioeconomic information on students, and school characteristics over 32 features. the data included more than ten million students and was preprocessed to the school grain. the school was classified as good or bad in relation to the average scores achieved by their students in the test. to highlight the model-agnostic framework, two tree-based classifiers (rf and adaboost (ab)) were applied combined in a 10-fold cross-validation setting, and the overall mean was adopted for each metric. thus, systematized temporal data mining evaluated how the main features related to the performance of the school had behaved over the years. due to space limitations, only the max uncentered ale mua was reported by three plots presenting the outputs, as discussed below. figure 4 presents a box plot with the mua for each feature. the clarity of colored dots illustrates the evolutionary directions of the variable over the years. the income per capita of students is the feature with the highest importance during the whole period. this is followed by parent’s education and the students’ race (brown students’ negative effects). the importance of student computers seems to be a growing tendency over the years. fig. 4. feature importance (mua) during the period (rf and ab means) (color figure online) full size image figure 5 separates a selected set of features to obtain a better understanding of their behavior during the period. the computer lab has a higher positive slope, while faculty jobs (the number of schools where teachers work) have a higher negative. fig. 5. evolution of the feature importance (mua) of a selected features set (rf and ab means) full size image lastly, in fig. 6, temporality was disregarded, and the features were organized for an overview of their importance in the following groups: non-actionable features (race and gender), school features (infrastructure), student features (parent’s education and income) and teacher features. in general, the group of features related to students demonstrated more potential to improve the quality of schools than others. additionally, non-actionable features had a strong influence, both negative and positive. fig. 6. feature importance (mua) by groups during the period (rf and ab means) full size image 5 discussion and conclusion this paper has proposed new model-agnostic metrics of feature importance in an attempt to circumvent the drawbacks and constraints of the existing methods, such as the βcoefficients of additive models and feature importance from tree-based algorithms, widely used for this purpose. this paper has proposed other options with a number of advantages, such as being agnostic to models, interpretable in the probability scale, and more reliable under correlated data. the accumulated local effects are the key trick for isolating the main effect of the variable even in correlated data. the four proposed metrics were validated in two experiments that illustrated their suitability to actual data mining applications. in the first experiment, breast cancer data were used to compare the proposed metrics with the coefficients of logistic regression and the permutation feature importance of random forest. the results illustrated that the proposed metrics are robust when facing correlated data and did not suffer the effects experienced by logistic regression (lr). all the proposed metrics captured the desired aspects of the attributes and were highly correlated. the aabsa proposed metric, which is directly comparable to the random forest (rf), since the permutation feature importance is only positive, captured very similar attribute importance. nevertheless, the comparisons were limited, and more tests are required to evaluate the metric behavior in other situations. for example, the lr coefficients are known to be sensitive to unobserved heterogeneity. hence, when features are added to the model and improve the predictions, the remaining coefficients may change, even if this feature is not correlated with others [13]. despite the marginal effects (a key to the proposed metrics) being more robust in this situation [26], there was no empirical evidence of this in our context. additionally, an empirical test of the robustness in a scenario of correlated data compared to existing metrics is required since this paper only considered the theory inherited from ale plots. thus, we intend to make more analyses on a large scale in future work, including other xai approaches. it should be mentioned that this paper has not yet compared the proposed metrics across algorithms, since the algorithms are able to use the input features in a totally different manner to achieve similar results, it was already known as the “rashomon” effect [27]. thus, even though the metrics proposed here are model-agnostic, the comparisons across different algorithms must be interpreted with caution, even on the same data. in addition, despite the motivation to compare feature importance across models, the results must be interpreted with care, and validation by domain experts is required. for example, in the second experiment, the data set was scaled equally for each year, and the set of variables was the same with the same values. even after these careful transformations, the comparison may be inappropriate, and a piece of domain knowledge may help to reduce the risks of misinterpretation. the main limitation of the proposed metrics would be the extrapolation of the local effect out of the interval where it was computed. the local effect is averaged across the conditional distribution and may only hold when the predictors \(x\) jointly fall within the same bin. thus, there may be a problem if bin widths are too small and the predictive function is too noisy. furthermore, local effects may be unreliable if the quantity of data points into the underlying bin is very small on the training data. although this paper has used deciles to equally subdivide the feature distribution in order to minimize this risk, together with cross-validation for more reliable results, caution must be always taken in the interpretation with the validation of the domain expert. notes 1.the implementation code can be found in this repository: https://github.com/rogerioluizsi/summary_ale.git. references bhatt, u., xiang, a., sharma, s., et al.: explainable machine learning in deployment. in: proceedings of the 2020 conference on fairness, accountability, and transparency, pp. 648–657. acm, new york (2020) google scholar  razavian, n., blecker, s., schmidt, a.m., et al.: population-level prediction of type 2 diabetes from claims data and analysis of risk factors. big data 3, 277–287 (2015). https://doi.org/10.1089/big.2015.0020 article  google scholar  pellagatti, m., masci, c., ieva, f., paganoni, a.m.: generalized mixed-effects random forest: a flexible approach to predict university student dropout. stat. anal. data min., 1–17 (2021). https://doi.org/10.1002/sam.11505 berens, j., schneider, k., görtz, s., et al.: early detection of students at risk-predicting student dropouts using administrative student data from german universities and machine learning methods (2019) google scholar  yang, k.c., varol, o., davis, c.a., et al.: arming the public with artificial intelligence to counter social bots. hum. behav. emerg. technol. 1, 48–61 (2019). https://doi.org/10.1002/hbe2.115 article  google scholar  leite, m.a.g.l., guelpeli, m.v.c., santos, c.q.: um modelo baseado em regras para a detecção de bots no twitter, pp. 37–48 (2020). https://doi.org/10.5753/brasnam.2020.11161 barredo arrieta, a., díaz-rodríguez, n., del ser, j., et al.: explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. inf. fusion 58, 82–115 (2020). https://doi.org/10.1016/j.inffus.2019.12.012 article  google scholar  saarela, m., jauhiainen, s.: comparison of feature importance measures as explanations for classification models. sn appl. sci. 3(2), 1–12 (2021). https://doi.org/10.1007/s42452-021-04148-9 article  google scholar  shrikumar, a., greenside, p., kundaje, a.: learning important features through propagating activation differences. in: 34th international conference on machine learning, icml 2017, vol. 7, pp. 4844–4866 (2017) google scholar  štrumbelj, e., kononenko, i.: explaining prediction models and individual predictions with feature contributions. knowl. inf. syst. 41(3), 647–665 (2013). https://doi.org/10.1007/s10115-013-0679-x article  google scholar  ribeiro, m.t., singh, s., guestrin, c.: why should i trust you?” explaining the predictions of any classifier. in: proceedings of the acm sigkdd international conference on knowledge discovery and data mining 13-17-august, pp. 1135–1144 (2016). https://doi.org/10.1145/2939672.2939778 lundberg, s.m., lee, s.-i.: a unified approach to interpreting model predictions. in: proceedings of the 31st international conference on neural information processing systems, pp. 4768–4777. curran associates inc., red hook (2017) google scholar  mood, c.: logistic regression: uncovering unobserved heterogeneity, pp. 1–25 (2017) google scholar  long, j.s., long, j.s.: regression models for categorical and limited dependent variables. sage, new york (1997) google scholar  molnar, c.: interpretable machine learning (2019) google scholar  bhatt, u., ravikumar, p., moura, j.m.f.: towards aggregating weighted feature attributions (2019) google scholar  hooker, g., mentch, l.: please stop permuting features: an explanation and alternatives, pp. 1–15 (2019) google scholar  guidotti, r., monreale, a., ruggieri, s., et al.: a survey of methods for explaining black box models. acm comput. surv. 51 (2018). https://doi.org/10.1145/3236009 bartus, t.: estimation of marginal effects using margeff. stata j. 5, 309–329 (2005). https://doi.org/10.1177/1536867x0500500303 article  google scholar  leeper, t.j.: interpreting regression results using average marginal effects with r’s margins (2021). https://cran.r-project.org/web/packages/margins/vignettes/technicaldetails.pdf32 apley, d.w., zhu, j.: visualizing the effects of predictor variables in black box supervised learning models. j. r. stat. soc. ser. b stat. methodol. 82, 1059–1086 (2020). https://doi.org/10.1111/rssb.12377 article  mathscinet  google scholar  friedman, j.h.: greedy function approximation: a gradient boosting machine. ann. stat. 29, 1189–1232 (2001). https://doi.org/10.1214/aos/1013203451 article  mathscinet  math  google scholar  zhao, q., hastie, t.: causal interpretations of black-box models. j. bus. econ. stat. 39, 272–281 (2021). https://doi.org/10.1080/07350015.2019.1624293 article  mathscinet  google scholar  dua, d., graff, c.: uci machine learning repository. university of california, irvine, school of information and computer sciences (2017). http://archive.ics.uci.edu/ml klaise, j., van looveren, a., vacanti, g., coca, a.: alibi explain: algorithms for explaining machine learning models. j. mach. learn. res. 22(181), 1–7 (2021). http://jmlr.org/papers/v22/21-0017.html mood, c.: logistic regression: why we cannot do what we think we can do, and what we can do about it. eur. sociol. rev. 26, 67–82 (2010). https://doi.org/10.1093/esr/jcp006 article  google scholar  fisher, a., rudin, c., dominici, f.: all models are wrong, but many are useful: learning a variable’s importance by studying an entire class of prediction models simultaneously. j. mach. learn. res. 20, 1–81 (2019) mathscinet  math  google scholar  download references author information authors and affiliations instituto federal do norte de minas gerais – ifnmg, montes claros, brazil rogério luiz cardoso silva filho centro de informática, universidade federal de pernambuco – ufpe, recife, brazil rogério luiz cardoso silva filho & paulo jorge leitão adeodato universidade federal rural de pernambuco – ufrpe, recife, brazil kellyton dos santos brito authors rogério luiz cardoso silva filhoview author publications search author on:pubmed google scholar paulo jorge leitão adeodatoview author publications search author on:pubmed google scholar kellyton dos santos britoview author publications search author on:pubmed google scholar corresponding author correspondence to rogério luiz cardoso silva filho . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silva filho, r.l.c., adeodato, p.j.l., dos santos brito, k. (2021). interpreting classification models using feature importance based on marginal local effects. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_32 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_32 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords feature importance explainable artificial intelligence ale plots publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature tuning hypothesis creation: combining discrete and continuous spaces for zero-shot hate speech detection | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper tuning hypothesis creation: combining discrete and continuous spaces for zero-shot hate speech detection conference paper first online: 31 january 2025 pp 253–268 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) tuning hypothesis creation: combining discrete and continuous spaces for zero-shot hate speech detection download book pdf download book epub lorenzo puppi vecchi  orcid: orcid.org/0009-0003-9483-20269, sylvio barbon junior  orcid: orcid.org/0000-0002-4988-070210 & emerson cabrera paraiso  orcid: orcid.org/0000-0002-6740-78559  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 219 accesses abstract hate speech detection is a crucial endeavor in maintaining the safety of online spaces, but the effectiveness of supervised approaches hinges mainly on the availability of annotated data. prior research has explored the utility of natural language inference (nli) models for zero-shot hate speech detection (zshsd), which leverages the capacity of these models to learn semantic relationships and adapt to downstream task without relying on large annotated datasets. nli models assess if a premise sentence logically entails a hypothesis sentence, relying on precise hypothesis design to achieve adequate downstream task performance. existing frameworks that use nli model for zshsd rely on multiple inferences with different hypothesis to extract characteristics to achieve desirable outcomes. in light of the challenges surrounding zshsd and the method of relying on discrete tokens to design hypotheses, we aim to optimize and identify ideal intermediate representations by applying p-tuning techniques. on hatecheck, a fully tuned hypothesis led to a 18.8% point (pp) accuracy improvement, compared to a discrete designed hypothesis. our work surpassed prior work by achieving a 5.6 pp accuracy enhancement, outperforming previous approaches that required multiple inferences. also, the optimized tokens uncover relations to broader aspects of hate speech, offering insights for hypothesis design. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others exploring hate speech detection: challenges, resources, current research and future directions article 04 march 2025 improving natural language inference with residual attention chapter © 2024 dss for hate speech detection chapter © 2025 1 introduction recently, there has been a rise in harmful content like hate speech and misinformation on the internet. this increase may not just be due to more people using blogs and social media but could also result from global political polarization and social media algorithms that amplify divisive content to drive engagement. this makes online safety and content moderation crucial to maintaining civility and protecting users [9, 19]. fig. 1. combining standard hypothesis and p-tuned virtual tokens full size image before, detecting hate speech mainly relied on classification models that needed lots of labeled data [2, 21]. recently, natural language inference (nli) models, which are used for tasks like zero-shot classification [7], are being explored. nli models show promising results for zero-shot hate speech detection (zshsd) because they capture relationships between sentences, which helps adapt content moderation to the ever-changing online world. in our study, we’re using the term “hate speech” to cover a wide range of offensive content created by users. we chose this term because it reflects hurtful communication that spreads hostility through stereotypes [15]. in nli models, generating hypotheses is crucial and affects tasks like zshsd [20]. even small changes in how a hypothesis is worded can greatly impact the model’s performance. for example, compare “climate change is a pressing global issue” to “addressing climate change requires urgent global cooperation.” both statements express concern about climate change, but the second one emphasizes the need for global collaboration, which could lead to different conclusions and outcomes in zshsd. to better handle the complexity of forming hypotheses, current frameworks using nli models take a step-by-step approach. they use multiple hypotheses to understand different aspects of text, like identifying hate speech in a tweet. for example, one hypothesis might spot explicit language (“this tweet has offensive words”), while another digs deeper into implied meanings (“the tone of this tweet encourages hostility”). this method helps the model grasp various aspects of text, which is crucial for accurate zshsd performance [7]. traditionally, approaches used simple hypotheses like “that’s hate speech” which, while somewhat effective, often oversimplify the complexities of identifying hate speech. these basic hypotheses struggle to capture the wide range of nuances, subtleties, and context-dependent variations in hate speech [7, 13]. this has led to a growing recognition of the need for more sophisticated hypothesis generation in zshsd. in our work, we aim to tackle the difficulties linked with hypothesis generation in nli models. we introduce a method that optimizes hypothesis creation through a process known as “p-tuning” [10], which starts with a standard hypothesis prompt but iteratively refines it to suit the complexities of zshsd. p-tuning allows us to adapt the hypothesis to the specific characteristics and nuances of the task, ensuring that it encapsulates a broader and more representative range of hate speech expressions. general objective: this article aims to tackle the challenge in hypothesis generation for natural language inference when used for zero-shot classification. it uses “p-tuning” as a method to optimize hypothesis creation for addressing the nuances involved in hate speech detection. in this process, we start with a initial hypothesis (h), choosing the most accurate one, out of different hypothesis sentence designed to detect hate speech using nli. results can be seen in table 2. next, the optimized hypothesis (oh) is generated, which starts from the best performed hypothesis of h, and is optimized using p-tuning. different strategies of using the resulting virtual tokens can be seen in table 4 and table 5. finally, the combined hypothesis (ch) is formed by integrating the most relevant tokens from oh with the original h. the accuracy of different virtual tokens inserted into the original hypothesis can be seen in table 4 and table 5, and a general benchmark, comparing with previous works, can be seen in table 6. a visual scheme of this idea is illustrated in fig. 1. our findings highlight the effectiveness of our optimized hypothesis (oh), as certain virtual tokens within the continuous embedding space closely approximate broader hate speech concepts. this suggests that including these virtual tokens (oh) in our original hypothesis (h) formulation can enhance the model’s capacity to detect and address hate speech more comprehensively. this opens up three promising avenues for future research: first: researchers can directly use our virtual token weights alongside roberta large mnli to enhance their models’ zshsdfootnote 1. second: they can employ selected model tokens from our study to adapt their models for improved classification. lastly: our research encourages the creation of hypotheses based on concepts’ mean, expanding the method’s applicability. 2 background the evaluation of hate speech detection models often suffers from overestimation issues, impacting both state-of-the-art (sota) and baseline models. when tested on different datasets, these models typically show reduced performance, highlighting a gap between test sets and real-world data [1, 5, 18, 23, 24]. various approaches to hate speech detection have been explored, including traditional machine learning models such as character n-gram logistic regression [8], support vector machines [17], and shallow networks with pre-trained embeddings like mlp with byte-pair encoding (bpe) [22]. generally, these simpler models underperform compared to deep neural networks. prior to 2019, recurrent neural networks were commonly used as sota models [8]. the introduction of bert (bidirectional encoder representations from transformers) by [3] marked a significant advancement, establishing bert and its variants as the new sota in hate speech detection [4, 6]. although bert and its derivatives have improved in capturing contextual information and linguistic nuances, our study focuses on nli (natural language inference) models. this choice is based on the rationale that explicit logic learning through textual entailment can reduce bias and enhance the recognition of social communities [12]. nli models excel in hate speech detection by leveraging logical training, which improves context-aware zero-shot hate speech detection (zshsd) systems. unlike pre-trained sentence encoders that may perpetuate stereotypes, nli models help mitigate bias present in training data [14]. using nli models for predictive tasks involves transforming the target task into an nli format, essentially a fine-tuning task. the model assesses a premise and a hypothesis to determine their logical relationship—whether the premise implies, contradicts, or is neutral to the hypothesis. a recent method applies nli models for zero-shot topic classification, where the input text serves as the premise and each topic has a corresponding hypothesis like “this text is about <topic>.” here, “neutral” and “contradiction” labels merge into “not-entailment,” meaning an “entailment” prediction indicates relevance to the topic, while “not-entailment” suggests irrelevance. it’s important to note that an entailment prediction indicates alignment with certain model artifacts, not necessarily correctness [25]. 3 proposed approach our method focused on identifying the most effective hypothesis for assessing the existence of hate speech in a given premise. rather than initiating the optimization process from entirely ran’dom weights, we leveraged the best-performing results obtained from prior experiments, as seen in table 2. this strategic starting point aimed to expedite the optimization process and build upon existing knowledge. 3.1 hypothesis optimization in the realm of nli models, a hypothesis is a sentence, that is encoded in a sequence of tokens. each token is represented as a integer and, after passing thought the models embedding layer, every token becomes a vector. when working with p-tuning, instead of changing the initial hypothesis sentence, the continuous vectors as optimized directly. this optimization creates whats is called as “virtual tokens”, since then dont perfectly resemble a proper token anymore. this means that the algorithm learns to adjust the continuous vector representation of initially passed hypothesis to improve its performance in zshsd. a general scheme of the process used to optimize the initial hypothesis \(h\) and test the different combinations of hypothesis tokens, can be seen in table 2. let’s denote \(h\) as initial hypothesis vectors (selected from the best results of table 2) and \(oh\) as the optimized hypothesis vectors. the learning process involves adjusting the weights \(\theta \) of the model to find an optimal hypothesis vector \(oh\), that minimizes a specific loss function, typically associated with zshsd. mathematically, this can be represented as: $$\begin{aligned} oh = argmin_{\theta }l(p, h, \theta ) \end{aligned}$$ (1) here, \(l\) represents the loss function, \(p\) denotes the task-specific premise, \(h\) signifies the initial hypothesis vectors, and \(\theta \) symbolizes the model’s weights. the objective is to find the optimal set of weights that minimizes the loss function, thereby producing the optimized hypothesis vectors \(oh\). nli models typically feature a softmax layer at the end, which serves to produce three primary outputs denoting the potential relationships between a given premise \(p\) and hypothesis \(h\): contradiction, neutrality, and entailment. these outputs, akin to probabilities, are normalized to ensure that their collective sum equals 1, effectively indicating the likelihood of each possible relationship. the goal of training an nli model is to minimize the error in predicting these probabilities, ensuring that the model accurately identifies the logical relationship between the premise and hypothesis. this is typically achieved through the use of a categorical cross-entropy loss \(l\), which is defined as: $$\begin{aligned} l = -\sum _{c=1}^3y_i\log (p_{i}) \end{aligned}$$ (2) in the context of binary hate speech detection, the loss function in this case is optimized to increase entailment output when hate is present, i.e. \(c=3\), and maximize contradiction output when hate is not present, i.e. \(c=1\). given that \(c = [1,2,3]\) represents, contradiction, neutrality and entailment, respectively. after optimizing our hypothesis, we proceeded with a structured sequence of experiments to assess both the individual and combined performance of optimized tokens. these experiments were designed to gauge the efficacy of our approach in diverse contexts, thereby illuminating its capacity to discern hate speech. in our study, we used the roberta large modelfootnote 2 [11], which had undergone fine-tuning on a diverse range of text sources, including wikipedia, the book corpus, and the multinli dataset. the training of this model was conducted on an nvidia 4090 rtx gpu, spanning a duration of approximately 12 h, with a training regimen extending over 20 epochs. for optimization, we employed the adamw optimizer and implemented a linear decay learning rate schedule, setting the initial learning rate at 1e-3. it is noteworthy that our training process required significantly less time and computational resources in comparison to previous work [19]. notably, we optimized our approach to achieve efficient resource utilization by employing p-tuning solely on the hypothesis, in contrast to traditional whole-model fine-tuning methodologies. this strategic adaptation likely contributed to the reduced time and computational resources required for our training process. 3.2 hypothesis similarity analysis our initial phase involved scrutinizing the optimized hypothesis and assessing its similarity to the original tokens before optimization. this allowed us to gauge the extent to which our optimizations retained the core meaning of the original text. we measured this distance using cosine similarity with the base model’s token embeddings. 3.3 token importance assessment in the second phase, we delved into the importance of each individual optimized token. we examined the outcomes of using these tokens in three distinct manners: isolated (one at a time), inserted individually, and cumulatively. following the idea of ch presented in fig. 1. isolated (token-level): in the “isolate” experiment, we isolated one token at a time from the optimized hypothesis. the primary objective was to determine whether the isolated tokens exhibited improved performance, particularly concerning terms associated with hate speech, such as “hateful” and “offensive.” individual (contextual insertion): in the “individual” experiment, once the tokens have been optimized in this new continuous space, they are reinserted back into the standard hypothesis structure. this means replacing the original tokens with their tuned representations in the hypothesis. let’s say the token “hateful” is initially represented as \([0.2, 0.5, -0.3]\), and after tuning, it becomes \([0.8, -0.1, 0.7]\). after reinserting the tuned tokens, the hypothesis may look like this: “this sentence contains \([{hateful}]\) entries or offensive language” “this sentence contains \([{hateful}]\) entries or \([{offensive}]\) \([{language}]\)” here, \([{hateful}]\), \([{offensive}]\) and \([{language}]\) represent the transformed versions of the original tokens “hateful”, “offensive” and “language”. the motivation behind this approach is to leverage a more nuanced and contextually aware representation of the key terms in the hypothesis. by doing so, this study aims to measure whether this enhanced understanding of specific language cues can lead to improved performance in hate speech detection as a nli tasks. the idea is that the tuned representations may capture subtleties and context better than the original, static tokens. cumulative (sequential insertion): the “cumulative” experiment involved the incremental addition of the optimized tokens in the natural order of the sentence. while we anticipated an overall improvement, the objective here was to analyze performance peaks corresponding to the introduction of each new optimized token. this step provided insights into the cumulative impact of our method on zshsd. 3.4 benchmark in the final phase of our experimentation, our focus shifted towards comparative analysis. specifically, we sought to assess the efficacy of our methodology, which incorporated the optimized hypothesis and tokens, in contrast to established benchmarks as presented in prior studies [7, 16, 19]. dataset. the dataset utilized in this study comprises two distinct sources, each contributing valuable insights into zshsd. in aggregate, the total dataset employed in our research comprises 40,772 instances. it consists of 22,571 instances labeled as “hateful” and 18,201 instances categorized as “non-hateful.” this diverse and multifaceted dataset serves as a foundation for our explorations into hate speech detection, allowing us to analyze and evaluate the performance of the method across a wide range of real-world hate speech scenarios (table 1). table 1. dataset data sources and number of examplesfull size table learning from the worst. [19] is a process to create datasets for training better hate detection models. it uses both humans and models. the dataset has 37,044 instances: 20,008 are labeled “hateful”, and 17,036 are “non-hateful.” we followed the authors’ setup, training the model on the four rounds made available by the dataset (r1, r2, r3, and r4) and testing on each rounds splits separately. hatecheck. [16] is a suite of tests for evaluating hate speech detection models. it has 29 test functionalities based on previous research and civil society input. the dataset contains 3,728 instances: 2,563 are “hateful,” and 1,165 are “non-hateful.” we used the best model accuracy scores for individual tests based on the original work [16]. for the average best performance we used the accuracy of “all” the different hypotheses combined as proposed by [7]. 4 results through an evaluation of the optimized hypothesis and token representations, as well as their impact on zshsd performance, we aim to provide a detailed insight into the capabilities and limitations of our approach. 4.1 zero-shoot hate speech hypothesis in this hypothesis test, we employed an nli model to assess the effectiveness of various hypotheses in measuring hate speech and offensive language. table 2 presents the results, including the average performance. the best hypothesis with an average accuracy result of 0.669 was “this sentence contains hateful entries or offensive language”. this result suggests that explicitly addressing hateful entries and offensive language in the hypothesis effectively captures a broader range of harmful content. the model excels here because it doesn’t need to make a strict distinction between the two types of content, which can be challenging given their nuanced overlap. we conducted this brief experiment using the testing split of “learning from the worst” and “hatecheck” datasets. table 2. hypothesis test (best performing results are )full size table the worst hypothesis with an average accuracy result of 0.601 was “hate”. this result could be attributed to the extreme brevity and lack of context in the hypothesis. it does not provide sufficient information for the model to accurately classify hate speech or offensive language. furthermore, the term “hate” on its own is highly ambiguous, making it challenging for the model to make accurate predictions. these results show that the performance of the nli model in zshsd is closely tied to the clarity and specificity of the hypothesis. hypotheses that encompass a broader spectrum of harmful content and provide context tend to yield better results. conversely, overly brief or ambiguous hypotheses may hinder the model’s ability to make accurate classifications, as evidenced by the lower scores observed in this test. therefore, it is imperative to carefully craft hypotheses that encompass the diversity and context of hate speech to enhance the effectiveness of nli-based zshsd models. 4.2 hypothesis similarity analysis in table 3 below, we present the original token representations alongside their optimized counterparts. we will examine the changes in each token and discuss why the optimized version might be more or less effective for predicting hate speech. furthermore, we will highlight the tokens that are likely to be more useful for this task. it’s important to note that these transformations occur within a continuous embedding space, so the similarity of the optimized tokens to specific words might not fully capture the subtleties of their representation. additionally, even if tokens seem less interpretable from a human perspective, they can still impact the model’s internal weights in ways that may positively or negatively affect its performance on the task. table 3. original token and optimized/p-tuned token cosine similarityfull size table this the optimized version adds unique characters and symbols, like “[...]” and “ ”, which might enhance the model’s ability to detect nuanced hate speech or sarcasm. however, it makes direct interpretation harder. sentence the optimized version includes unrelated words like “edits”, “inscription” and “subtitles” which don’t seem to help with zshsd but rather shift towards unrelated terms. contains the optimized version contains random characters and sequences (“pha”, “nton”, etc.) that don’t align with the original token’s meaning. this transformation doesn’t relate to zshsd. hateful the optimized version keeps “hateful” and “despicable” while adding terms like “fascist” and “discriminatory”, broadening its capacity to detect hate speech with more discriminatory and extremist terms. entries the optimized version introduces unrelated terms (“ansk”, “owitz”, etc.) that don’t positively contribute to zshsd or relate to the original token’s meaning. or the optimized version adds terms like “antagonist”, “favorably” and “extremists” which aren’t directly related to the original token but may help identify hate speech involving comparisons, contrasts, or extremist viewpoints. offensive the optimized version includes terms like “transsexual”, “lgbt”, and “ethnic” which are relevant for detecting hate speech involving sensitive topics and discriminatory language. language the optimized version introduces unrelated terms (“andra”, “eto”, etc.) that don’t contribute positively to zshsd. the evaluation of the optimized tokens and their contribution to zshsd can vary significantly depending on the specific nli model and its objectives. the analysis provided here represents a human-driven evaluation of the tokens’ relevance to zshsd. while some tokens like “hateful” and “offensive” appear to strengthen the model’s ability to identify hate speech by introducing relevant terms, others seem unrelated to the task and may introduce noise. it’s essential to note that the effectiveness of token optimization is context-dependent and should be evaluated within the broader framework of the nli model’s goals. further experimentation and fine-tuning may be necessary to determine the most suitable token representations. ultimately, the choice of optimized tokens should align with the specific nuances and objectives of the problem at hand, and the assessment should be guided by both human judgment and quantitative performance metrics. 4.3 token importance assessment the performance of three different methods for incorporating optimized tokens into the hypothesis for zshsd—“cumulative,” “individual,” and “isolate”—is shown in table 4. the accuracy scores over all test data reveal the contributions of specific tokens: table 4. original and virtual tokens combinations performancefull size table “this” and “sentence”: consistently contribute across all methods, with “individual” showing a slightly higher impact. “contains”: significantly boosts performance in the “cumulative” approach, indicating its value in enhancing zshsd. “hateful”: has a strong impact in the “cumulative” method but decreases in “individual” and “isolate,” suggesting its lesser standalone effect. “entries” and “or”: vary in contribution, with “cumulative” showing the highest values. “or” contributes the least in the “isolate” approach. “offensive” and “language”: like “hateful,” these tokens are crucial in the “cumulative” approach but less impactful individually. the “cumulative” approach shows the most significant performance improvement as tokens are sequentially added, underscoring the value of considering tokens cumulatively for enhanced zshsd. however, the impact of individual tokens varies, and not all tokens contribute equally. the “cumulative” approach shows the most significant performance improvement, highlighting the importance of considering tokens collectively for better zshsd. for further analysis, the tokens “contains,” “hateful,” “or,” and “offensive” were selected due to their pivotal roles. the “individual” method was used for additional tests: “contains”: directly relates to hate speech identification. “hateful” and “offensive”: explicitly linked to hate speech, enhancing the model’s discriminatory power. “or”: despite being a common conjunction, it introduces nuanced interpretations with terms like “antagonist”, “favorably” and “extremists” demonstrating its adaptability in capturing varied hate speech nuances. table 5 shows the “full_optimized” hypothesis achieving the highest accuracy (0.870), indicating its effectiveness. combinations including “hateful” and “offensive” also exhibit high accuracy, close to the “full_optimized” hypothesis, suggesting that a selected combination of tokens can achieve similar results. table 5. optimized virtual tokens inserted into standard hypothesis performancefull size table the statistical analysis reveals a significant improvement of the “full optimized” sample over the “pure hypothesis”. considering the first value and second value as the significance to “full optimized” and “pure hypothesis” respectively, among the combinations analyzed, “contains hateful offensive” (0.321 vs. 0.001), “hateful or offensive” (0.216 vs. 0.003), and “contains hateful or offensive” (0.262 vs. 0.003) show significant improvements when compared to “h pure hypothesis.” the analyses were conducted with a 95% confidence interval, considering p-values below 0.05 as significant, thereby confirming the alternative hypothesis. 4.4 previous work comparation table 6 displays accuracy results from various hate speech tests, highlighting the performance of different hypothesis configurations, from pure hypotheses to intermediate insertions to fully optimized hypotheses. results exceeding previous work are marked in green. the individual insert method was used for intermediate steps, as depicted in fig. 1. table 6. datasets detailed resultsfull size table key observations include an increased accuracy with the addition of optimized tokens to the pure hypothesis and a general trend of increasing accuracy from initial to fully optimized hypotheses, underscoring the effectiveness of the optimization procedure in enhancing the model’s zshsd capabilities. some tests show that specific intermediate steps outperform others, indicating the need to tailor the optimization process to different hate speech categories. intermediate representations such as “hateful or offensive”, “contains hateful or offensive”, and “contains hateful offensive” consistently yield high accuracy across various tests, reinforcing their importance in improving. performance enhancements were notable when tested on the learning from the worst test dataset [19], with accuracy gains of 5.5% points (pp) and 1.2 pp on test splits r2 and r4, respectively. unlike previous efforts that relied on fully fine-tuned classification models, our approach optimizes the input hypothesis of a nli model. on the hatecheck dataset [16], optimized tokens led to an average accuracy improvement of 7.74 pp, with a best-case scenario improvement of 18.8 pp using a fully tuned hypothesis (oh) compared to (h). our fully tuned single hypothesis also outperformed previous work by 5.6 pp. these findings highlight the effectiveness of our optimization procedure in enhancing the model’s zshsd capabilities. both fully tuned hypotheses and significant token insertions on standard hypotheses showed improved accuracy, suggesting future directions for broader hypothesis design within nli models. 5 conclusion our study underscores the significance of thoughtful token selection and the synergy between different aspects of hypothesis optimization. by improving how hypotheses are formulated, we can enhance nli model performance in zshsd, contributing to the vital work of online content moderation and digital safety. our results highlighted the importance of specific tokens like “hateful” and “offensive” in enhancing model accuracy. by optimizing these tokens, we expanded the model’s capability to detect a wider range of hateful language and ideologies. for instance, the optimized version of “hateful” introduced terms like “fascist” and “discriminatory” enriching the model’s understanding of hatred and intolerance. similarly, optimizing “offensive” incorporated terms like “transsexual”, “lgbt” and “ethnic” which are relevant to hate speech detection. these additions improved the model’s ability to recognize subtle nuances in discriminatory language. future work should focus on expanding and diversifying token pools, adapting models for multilingual contexts, and developing real-time adaptation mechanisms to keep up with evolving hate speech. enhancing contextual analysis, incorporating user feedback, addressing ethical and bias issues, are crucial for improving accuracy and practical utility in online content moderation. notes 1.https://anonymous.4open.science/r/combining-discrete-and-continuous-spaces-for-zero-shot-hate-speech-detection-fb34/readme.md. 2.https://huggingface.co/roberta-large-mnli. references arango, a., pérez, j., poblete, b.: hate speech detection is not as easy as you may think: a closer look at model validation. in: proceedings of the 42nd international acm sigir conference on research and development in information retrieval, pp. 45–54 (2019) google scholar  davidson, t., warmsley, d., macy, m., weber, i.: automated hate speech detection and the problem of offensive language. in: proceedings of the international aaai conference on web and social media, vol. 11, pp. 512–515 (2017). https://doi.org/10.1609/icwsm.v11i1.14955 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of naacl-hlt, vol. 1, p. 2 (2019). https://doi.org/10.18653/v1/n19-1423. https://aclanthology.org/n19-1423 fortuna, p., dominguez, m., wanner, l., talat, z.: directions for nlp practices applied to online hate speech detection. in: goldberg, y., kozareva, z., zhang, y. (eds.) proceedings of the 2022 conference on empirical methods in natural language processing, pp. 11794–11805. association for computational linguistics, abu dhabi, united arab emirates (2022). https://doi.org/10.18653/v1/2022.emnlp-main.809. https://aclanthology.org/2022.emnlp-main.809 fortuna, p., nunes, s.: a survey on automatic detection of hate speech in text. acm comput. surv. 51(4) (2018). https://doi.org/10.1145/3232676 fortuna, p., soler-company, j., wanner, l.: how well do hate speech, toxicity, abusive and offensive language classification models generalize across datasets? inf. process. manag. 58(3), 102524 (2021). https://doi.org/10.1016/j.ipm.2021.102524. https://www.sciencedirect.com/science/article/pii/s0306457321000339 goldzycher, j., schneider, g.: hypothesis engineering for zero-shot hate speech detection. in: proceedings of the third workshop on threat, aggression and cyberbullying (trac 2022), pp. 75–90. association for computational linguistics, gyeongju, republic of korea (2022). https://aclanthology.org/2022.trac-1.10 gröndahl, t., pajola, l., juuti, m., conti, m., asokan, n.: all you need is “love” evading hate speech detection. in: proceedings of the 11th acm workshop on artificial intelligence and security, pp. 2–12 (2018) google scholar  gruzd, a., soares, f.b., mai, p.: trust and safety on social media: understanding the impact of anti-social behavior and misinformation on content moderation and platform governance. soc. media+ soc. 9(3), 20563051231196878 (2023) google scholar  liu, x., et al.: p-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks (2022) google scholar  liu, y., et al.: roberta: a robustly optimized bert pretraining approach (2019) google scholar  luo, h., glass, j.: logic against bias: textual entailment mitigates stereotypical sentence reasoning. in: proceedings of the 17th conference of the european chapter of the association for computational linguistics, pp. 1235–1246 (2023) google scholar  macavaney, s., yao, h.r., yang, e., russell, k., goharian, n., frieder, o.: hate speech detection: challenges and solutions. plos one 14(8), e0221152 (2019) article  google scholar  nadeem, m., bethke, a., reddy, s.: stereoset: measuring stereotypical bias in pretrained language models. in: zong, c., xia, f., li, w., navigli, r. (eds.) proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp. 5356–5371. association for computational linguistics, online (2021). https://doi.org/10.18653/v1/2021.acl-long.416. https://aclanthology.org/2021.acl-long.416 paz, m.a., montero-díaz, j., moreno-delgado, a.: hate speech: a systematized review. sage open 10(4), 2158244020973022 (2020). https://doi.org/10.1177/2158244020973022 article  google scholar  röttger, p., vidgen, b., nguyen, d., waseem, z., margetts, h., pierrehumbert, j.: hatecheck: functional tests for hate speech detection models. in: proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp. 41–58 (2021) google scholar  sevani, n., soenandi, i.a., wijaya, j., et al.: detection of hate speech by employing support vector machine with word2vec model. in: 2021 7th international conference on electrical, electronics and information engineering (iceeie), pp. 1–5. ieee (2021) google scholar  vidgen, b., harris, a., nguyen, d., tromble, r., hale, s., margetts, h.: challenges and frontiers in abusive content detection. in: proceedings of the third workshop on abusive language online. association for computational linguistics (2019) google scholar  vidgen, b., thrush, t., waseem, z., kiela, d.: learning from the worst: dynamically generated datasets to improve online hate detection. in: proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing. association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.acl-long.132. https://aclanthology.org/2021.acl-long.132 wang, s., fang, h., khabsa, m., mao, h., ma, h.: entailment as few-shot learner (2021) google scholar  waseem, z., hovy, d.: hateful symbols or hateful people? predictive features for hate speech detection on twitter. in: proceedings of the naacl student research workshop, pp. 88–93. association for computational linguistics, san diego, california (2016). https://doi.org/10.18653/v1/n16-2013. https://aclanthology.org/n16-2013 waseem, z., thorne, j., bingel, j.: bridging the gaps: multi task learning for domain transfer of hate speech detection. online harassment, pp. 29–55 (2018) google scholar  wiegand, m., ruppenhofer, j., kleinbauer, t.: detection of abusive language: the problem of biased datasets. in: burstein, j., doran, c., solorio, t. (eds.) proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 602–608. association for computational linguistics, minneapolis, minnesota (2019). https://doi.org/10.18653/v1/n19-1060. https://aclanthology.org/n19-1060 yin, w., zubiaga, a.: towards generalisable hate speech detection: a review on obstacles and solutions. peerj comput. sci. 7, e598 (2021) article  math  google scholar  yin, w., hay, j., roth, d.: benchmarking zero-shot text classification: datasets, evaluation and entailment approach. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp). association for computational linguistics (2019). https://doi.org/10.18653/v1/d19-1404. https://aclanthology.org/d19-1404 download references author information authors and affiliations graduate program in informatics, pontificia universidade católica do paraná, curitiba, brazil lorenzo puppi vecchi & emerson cabrera paraiso department of engineering and architecture, university of trieste, trieste, italy sylvio barbon junior authors lorenzo puppi vecchiview author publications search author on:pubmed google scholar sylvio barbon juniorview author publications search author on:pubmed google scholar emerson cabrera paraisoview author publications search author on:pubmed google scholar corresponding author correspondence to lorenzo puppi vecchi . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper vecchi, l.p., barbon junior, s., paraiso, e.c. (2025). tuning hypothesis creation: combining discrete and continuous spaces for zero-shot hate speech detection. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_18 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords hate speech natural language inference zero-shot text classification p-tuning publish with us policies and ethics profiles sylvio barbon junior view author profile emerson cabrera paraiso view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature investigation of deep active self-learning algorithms applied to named entity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper investigation of deep active self-learning algorithms applied to named entity recognition conference paper first online: 12 october 2023 pp 470–484 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) investigation of deep active self-learning algorithms applied to named entity recognition download book pdf download book epub josé reinaldo cunha santos a. v. silva neto9 & thiago de paulo faleiros10  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 514 accesses abstract active self-learning algorithms reduce the labeled data required to train a machine learning model through supervised training. this paper explores various active self-learning algorithms for named entity recognition tasks. firstly, we investigate the impact of different self-training techniques on active self-learning algorithms. secondly, we propose a novel token-level active self-learning algorithm that achieves near-peak performance using fewer hand-annotated tokens compared to existing works. through numerous experiments, we found that the sentence-level active self-learning algorithm did not consistently yield significant results compared to pure active learning. however, our proposed token-level active self-learning algorithm showed promising performance, training a neural model to nearly peak accuracy with fewer human-annotated tokens compared to state-of-the-art active learning baseline algorithms. the experimental results are presented and discussed, demonstrating the superior performance of the token-level active self-learning algorithm j. r. c. s. a. v. s. neto—research performed during the author’s masters undertaking at the university of brasilia (unb). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others deep active-self learning applied to named entity recognition chapter © 2021 dropout-based active learning for regression chapter © 2018 a self-training approach for few-shot named entity recognition chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms automated pattern recognition categorization learning algorithms machine learning sequence annotation 1 introduction active learning (al) algorithms reduce the labeled data required to train a machine learning model through supervised training. these algorithms can be generally separated into three classes: (1) pool-based, (2) stream-based, and (3) query synthesis. for this paper, we focus on the pool-based class which thrives in scenarios where large amounts of data are available, but the annotation process for the complete pool of data is costly. the objective of the pool-based al technique is to identify, given a considerable pool of unlabeled data, a smaller subset of data samples that represent the whole data distribution well. by annotating this smaller subset of data samples and training a model through supervised training, it is possible to achieve good model performance with a significant decrease in data annotation costs. in active learning for named entity recognition (ner), most research proposes using sentence-level querying strategies. sentence-level querying means that the whole unlabeled sentence is queried to the oracle, which is expected to provide labels for all tokens. it provides more context and can lead to more accurate annotation decisions, but it can be more computationally expensive and time-consuming. some works propose alternative strategies such as token-level [7] or subsentence-level [16] querying. these strategies aim to make the annotation process more efficient by reducing the number of tokens that need to be manually labeled. this paper proposes cooperative approaches for reducing the cost of annotation by an oracle and speeding up the annotation process. we focus on active self-learning (asl) algorithms for that. the most straightforward implementation of an active self-learning algorithm uses the machine learning model to predict classes for the unlabeled data. the predicted labels are then split into high-confidence and low-confidence, depending on the model’s confidence in its predictions. high-confidence samples are labeled automatically using the model’s predictions, while low-confidence samples are queried for the human annotator to be labeled manually. labeled samples are added to the labeled dataset, which is used to train the machine-learning model further. this process is repeated until the desired level of model performance is achieved, or a stopping criterion is reached. in active self-learning algorithms, the active learning part corresponds to the query to the human annotator. in contrast, the self-learning part corresponds to the automatic labeling done by the trained model. the asl creates a cooperative scenario where humans and models annotate data together and can potentially reduce annotation costs compared to using only active learning strategies [21]. our 2 main contributions in this paper are: (1) we investigate the impact of different self-training techniques on active self-learning algorithms; and (2) we propose a novel active self-learning algorithm based on token-level querying that achieves peak performance with less hand-annotated data than previous works. the overview of the paper is as follows. in sect. 2, we present relevant works from the literature on both active learning and active self-learning algorithms applied to named entity recognition tasks. section 3 briefly reviews the active self-learning algorithm from [2] based on sentence-level querying. in sect. 4, we investigate the impact of different self-learning techniques on the proposed active self-learning algorithm from [2]. in sect. 5, we propose a novel active self-learning algorithm based on token-level querying, where both the human annotator and machine learning model cooperatively annotate tokens from the same sentence. section 7 we present the results of our investigations on the asl algorithms proposed in sects. 4 and 5. finally, in sect. 8, we summarize the results from both sentence-level and token-level active self-learning algorithms. 2 related works many studies in the current literature focusing on active learning algorithms for ner utilize sentence-level querying. shen et al. [18] proposes, to the best of our knowledge, the first active learning algorithm based on neural networks for tasks of sequence tagging (e.g. ner and part-of-speech tagging). their proposed neural model uses convolutional layers for character and word-level feature encoding and an lstm layer with greedy decoding as the tag decoder. the authors also proposed the maximum normalized log-probability (mnlp) sentence-level query function, which normalizes the model’s confidence for a given unlabeled sentence. their experiments show that the proposed algorithm trains the model to peak performance using only 25% of the training set of the ontonotes5.0 dataset [15]. siddhant and lipton [19] extend previous work by shen et al. they use the bayesian active learning through disagreement [6] (bald) framework by querying the unlabeled sentences that generate the most disagreement over multiple passes on neural models. to introduce stochastic behavior in the neural models, they propose two solutions. the first solution is the monte carlo dropout, where the model makes multiple predictions for the same sentence but with a different dropout mask at a time. the second solution is the bayes by backpropagation, where some layers have their deterministic parameters replaced by stochastic parameters. experiments showed that the proposed query strategies performed consistently better than the previous mnlp, but this came with more complex and costly compute query functions. in the literature on active learning applied to named entity recognition, sub-sentence-level querying strategies are found less often, including token-level querying strategies. the main challenge is how to implement the model training routine with sentences that are partially labeled. kobayashi and wakabayashi [7] propose the use of a multi-class logistic regression model with point-wise predictions [12]. they can use token-level queries to train their models through this method. radmard et al. [16] proposes a sub-sentence-based query strategy, where only the most essential, non-overlapping sub-sentence are queried instead of the whole sentence. the strategy is to query sub-sentence without their surrounding context and store the annotation in a dictionary that associates each sub-sentence with its corresponding accurate labels. annotations for a sub-sentence are propagated onto the unlabeled dataset, meaning that all occurrences of a particular sub-sentence are assigned the same labels. they then train a neural model using a loss function computed only on labeled tokens. active self-learning algorithms have received considerably less attention in the ner literature when compared to purely active learning algorithms. tran et al. [21] proposes an active self-learning algorithm based on conditional random fields (crf) models. the active learning process is related to using a diversity measure to query the most informative samples to the oracle. at the same time, the self-learning process is related to using the trained crf model from the previous iteration to annotate unlabeled sentences with high confidence in its predictions. the dataset used was composed of sentences extracted from twitter. the experiments compared uncertainty, diversity query strategies, and active learning algorithms with and without self-learning. results have shown that active self-learning algorithms achieved, in general, better results than the purely active learning algorithm. inspired by the work of tran et al. [21], cunha and faleiros [2] presents another active self-learning algorithm. they made specific changes to address the use of deep neural models instead of crf. the proposed asl algorithm has been shown to be less sensitive to the quality of the labeled set in initial iterations, when compared to the previous active self-learning algorithm from the literature [1]. 3 deep active self-learning algorithm this section briefly describes the active self-learning algorithm proposed by cunha and faleiros [2]. the active self-learning algorithms described in existing literature require collaboration between a human annotator and a trained model to annotate samples from an unlabeled database. however, this process is sensitive to the initial labeled set used to train the machine learning model [1]. we argue that this sensitivity arises from poorly annotated samples selected by the model in the early rounds of the active self-learning algorithm. this poorly annotated data may introduce permanent bias to the labeled set. to address this issue, the work of [2] proposed an active self-learning algorithm that distinguishes between samples labeled by the model and those labeled by the human annotator. the former has less impact on the model’s parameters during training. additionally, samples labeled automatically by the model were returned to the unlabeled set after each iteration of the active self-learning algorithm. these modifications seem to mitigate the risk of introducing permanent bias to the trained model and labeled dataset. algorithm 1 presents a more detailed explanation of active self-learning algorithm. algorithm 1 . active self-learning algorithm full size image note that in algorithm 1, m represents the machine learning model, q is the query budget, \(min\_confidence\) is the minimum confidence level required for the model to annotate an unlabeled sample, a.l. denotes the active labeled set that contains samples annotated by the oracle, s.l. contains samples labeled by the trained model, and u represents the set of unlabeled data. the active learning procedure, represented by the \(active\_learning\_query(\cdot )\) function, identifies the most informative unlabeled samples for annotation by the oracle. the self-learning procedure, represented by the \(self\_learning\_query(\cdot )\) function, corresponds to the different self-learning strategies that will be applied. we investigate the impact of different self-learning techniques based on sentence-level and word-level querying strategies. 4 sentence-level active self-learning algorithm this section introduces our proposed active self-learning algorithms that utilize sentence-level querying. this algorithm is built upon the work of [2] by incorporating various self-learning techniques. the diagram of the active self-learning algorithm proposed is shown in fig. 1. two main characteristics make this algorithm different from previous algorithms in the literature. (1) the first change is that we now have two individual labeled sets (shown in orange in fig. 1). one of these labeled datasets is responsible for keeping sentences hand-annotated by the oracle, thus having highly reliable labels. at the same time, the other is responsible for keeping sentences labeled by the machine learning model with less reliable labels. this separation of highly reliable and less reliable labeled data allows us to use them differently during training. we employ hand-annotated data for traditional supervised learning and utilize data labeled by the model for self-learning. (2) the second change is that after training the machine learning model in any given algorithm iteration, the data self-labeled by the machine learning model returns to the unlabeled data pool. these alterations allow a better-trained model to re-annotate samples in later iterations. fig. 1. annotation process for a sentence-level active-self learning algorithm. (color figure online) full size image the algorithm proposed in [2] adopted pseudo-labeling as the self-learning strategy. here, our alternate version replaces the pseudo-labeling technique with self-learning methods. we test three different self-learning techniques, namely: (1) word dropout, (2) virtual adversarial training, and (3) cross-view training. next, we will explain how our algorithm utilized pseudo-labeling and integrated these three techniques: the pseudo-labeling (pl) technique identifies highly reliable unlabeled samples to be automatically annotated by the model. in this case, highly reliable means that the model’s confidence in its predictions is above a predefined threshold. our strategy consists of using the machine learning model’s predictions as labels. then, apply supervised training with the pseudo-labeled and the hand-labeled data. the cross-view training (cvt) [3] aims to improve the representation capabilities of the model by forcing it to output similar predictions with different views of the same input data. the cvt algorithm uses neural models with auxiliary classification heads. each head learns to predict tokens given a limited view of the input sentence (e.g., left-context only, right-context only). the auxiliary classification head comprises a fully-connected layer with relu [11] activation, followed by a softmax function to generate a distribution over predicted classes. the cvt loss is the kullback-leibler (kl) divergence between the outputs of the main classification head, which sees the complete input, and the auxiliary heads, which see partial views of the input. the word dropout (wd) technique [3, 8] consists in replacing random words in a sentence with special tokens, such as \(<removed>\) or \(<unk>\), and training the model to produce a similar output distribution as when the word was unmasked. the virtual adversarial technique (vat) is a self-learning technique proposed by miyato et al. [10] for text classification and applied to ner by clark et al. [3]. this technique extends the adversarial training [4], where a sample of input data is perturbed with specially crafted noise designed to fool the model. this helps the model to be more robust to small perturbations in the input data. 5 token-level active-self learning algorithm the model’s overall confidence for the entire sentence is considered when using sentence-level querying. this implies that the model assumes average confidence in its predictions for all the words/tokens within the sentence. however, hard-to-predict entities may be surrounded by various easy-to-predict tokens. we argue this may lead to an overall overconfidence in the model for complex tokens, leading to poor self-annotations that may hamper the active self-learning process. we leverage this idea to propose a token-level active self-learning algorithm. the token-level active self-learning algorithm proposed is a modified version of the deep active learning (dal) algorithm presented by shen et al. [18]. the main difference between the original dal and our proposed algorithm is its labeling process. the original algorithm queried the most uncertain sentences and asked the oracle to annotate them. our proposed algorithm takes the queried sentences, identifies the tokens with low confidence, and asks the oracle to annotate them. the remaining unlabeled tokens from the queried sentence are automatically labeled using the model’s output predictions. we also improve the accuracy of our self-labeling process by using hand-annotated labels to refine the predictions made by the model. this cooperative scenario can alleviate the cost of manual annotation for the oracle and speed up the annotation process by highlighting specific words in a sentence that must be hand-annotated. our proposed token-level active self-learning algorithm is illustrated in fig. 2, which shows the annotation process. fig. 2. an illustration of the collaborative configuration where an oracle and machine learning model annotate the same sentence jointly. full size image our proposed algorithm has three main procedures: 1) predict the highest confidence tokens to automatic annotation; 2) query low-confidence tokens to the oracle for manual annotation and 3) self-labeling refinement. 1. identifying low-confidence tokens: suppose a neural network produces a probability distribution for an input token, which represents the likelihood of the token belonging to one of the classes of named entities. in this context, we define low-confidence tokens as those for which the model has confidence in its prediction lower than a predefined threshold. we empirically selected a threshold confidence of 99%, implying that tokens with less than 99% of confidence in their predicted class will be labeled by the oracle (i.e., human annotator). in contrast, the remaining tokens will be self-labeled by the model. 2. query to the oracle: it is the traditional active learning technique implemented at a token level, where only the low-confidence tokens in the selected sentences are queried to the oracle. 3. self-labeling refinement: once an oracle has labeled the low-confidence tokens, we can use self-labeling to label the remaining unlabeled tokens from the queried samples. a simple approach is to predict the classes for all tokens and use the predictions to label high-confidence tokens. however, we can leverage these labels to improve our predictions since we have the proper labels of the low-confidence tokens, which were labeled by the oracle. for example, the cnn-cnn-lstm model [18] uses a greedy decoding approach where it receives the predicted label of the previous token as an additional input to help to predict the current token’s class. if the previous token had low confidence and was manually labeled by the oracle, we could modify this approach by using the oracle-assigned label instead of the label previously predicted by the model. this process can be repeated a predefined number of times, iteratively, with a decreasing number of tokens replacement in each iteration. the refinement step was inspired by the iterative algorithm proposed by park et al. [13]. they use iterative refinement to identify synonyms to substitute specific tokens from a sentence, with a low impact on its coherence. the idea is to identify potential synonyms for specific tokens, replace them, and verify if a masked language model predicts the synonyms with high confidence. other synonyms replace the synonyms with the lowest masked language model scores. for our refinement step, however, we use reliable oracle annotated tokens to enhance the model predictions for the unlabeled tokens. 6 experimental design we use two consolidated english ner datasets, namely conll03 [17] and ontonotes5.0 [15], and one legal domain portuguese ner dataset, named aposentadoria [2]. aposentadoria is a new legal domain ner dataset. it contains named entities from 10 classes associated with retirement acts of public employees from the diário oficial do distrito federal (brazilian federal district official gazette, in direct translation). the datasets chosen for use are listed in table 1, along with relevant information such as their language, subject domain. table 1. datasets description.full size table we use two neural models for sentence-level and one for token-level active self-learning. for sentence level, we use cnn-cnn-lstm proposed by shen et al. [18] and the cnn-bilstm-crf proposed by ma and hovy [9]. we could not use cnn-bilstm-crf in the token-level case because crf classification layers output a distribution over the entire sentence instead of a distribution of classes for each token. more details about the model’s and training algorithm’s hyperparameters are presented in the following subsections. we evaluate four versions of our proposed active self-learning algorithm using self-training techniques at the sentence level (sect. 4). we also compare the results with the deep active learning algorithm proposed by shen et al. [18]. moreover, we compare the token-level active self-learning algorithm with the subsequence-based active learning algorithm by radmard et al. [16]. this algorithm was chosen as a baseline because its queries use sub-sentences instead of whole sentences. for all proposed algorithms, we compute the maximum normalized log-probability (mnlp) measure proposed by shen et al. [18] to generate queries for new samples. the mnlp value for a sentence x of length n can be calculated as $$\begin{aligned} mnlp(x) = \max _{y_1, ..., y_{n-1}}\frac{1}{n}\sum ^n_{i=0}log\ p(y_i\vert x_i, y_0, y_1, ..., y_{i-1}). \end{aligned}$$ (1) where \(x_i\) is the i-th token of a sentence x, and \(y_i\) is the class predicted for the i-th token \(x_i\). the querying for the oracle and the trained model is based on the mnlp measure. we use the least confidence for the oracle annotation, where unlabeled samples with the lowest mnlp scores in a given algorithm iteration will be queried to the oracle. for automatic annotation, we use the exponentiated mnlp measure. the unlabeled samples with exponentiated mnlp measure higher than a predefined threshold will be used for self-training in the next iteration of the active self-learning algorithm. for all experiments, the threshold selected empirically was 0.99. this threshold means that the model’s prediction confidence must be equal to or higher than 0.99 for the sample to be used for self-training. similar to previous works in the literature [18, 19], we apply early stopping of the model training. the early stopping is based on the model’s performance on the validation set. we empirically chose a patience of 15 epochs, meaning that if the model’s performance does not improve in 15 consecutive epochs, the model’s training is interrupted. 6.1 cnn-cnn-lstm model the cnn-cnn-lstm model consists of a character-level convolutional encoder, a word-level convolutional encoder, and an lstm tag decoder. the character-level cnn is used to generate character-level vector representations of a word. this cnn works by first transforming each character of a word into a vector representation, with embeddings being initialized with uniform samples from \(\left[ -\sqrt{\frac{3}{dim}}, +\sqrt{\frac{3}{dim}} \right] \) as proposed by ma and hovy [9]. dropout [20] is applied to the generated embeddings as presented by ma and hovy [9]. then, a one-dimensional convolutional layer is used to extract information of neighboring characters, followed by relu activation [11] and max-pooling to generate fixed-size character-level word embeddings. we generate a word representation by concatenating a character-level vector with a vector representation from a lookup table, which we initialize with pre-trained word embeddings that we update throughout the training process. we used glove embeddings of 100 dimensions pre-trained on english newswire corpus [14], and glove embeddings of 300 dimensions pre-trained on multi-genre portuguese corpus [5]. the lstm tag decoder is responsible for the tagging of each word. the lstm tag decoder uses a one-hot encoded vector of the previous tag, concatenates it to the encoded vector representation of the current word, and uses it as input. a thorough explanation of the hyperparameter tuning routine and hyperparameter values used in our experiments is presented in [1]. 6.2 cnn-bilstm-crf model similarly to the cnn-cnn-lstm model, the cnn-bilstm-crf model uses a cnn to generate a character-level representation for each word. the full embedding, formed by the concatenation of character-level and word-level embeddings, is fed to a bilstm layer that generates vector representations for each word in a sentence. a fully-connected layer is then used to reduce the encoded vector’s dimension to the number of possible tags. the reduced dimension vector is then fed to a crf layer. dropout layers are used before and after the bilstm layer. all weight matrices for the bilstm and fully-connected layers are randomly initialized using a uniform distribution to select samples from \([-\sqrt{\frac{6}{r+c}}, +\sqrt{\frac{6}{r+c}}\) as proposed by ma and hovy [9], where r and c are the numbers of rows and columns in the weight matrix. we initialize the bias parameters from the bilstm layer to 0.0 and the forget gate bias to 1.0. the model’s hyperparameters for the english ner datasets are similar to those presented in the experiments of siddhant and lipton [19]. a grid search was employed for the portuguese dataset to define the model’s hyperparameters. a thorough explanation of the hyperparameter tuning routine and hyperparameter values used in our experiments is presented in [1]. 7 results we present and discuss the experimental results focusing on the performance overview (f1-score) of the models trained on each of the three datasets (see table 1). additionally, we compare the performance of neural models trained using various active and active self-learning algorithms. in separate sections, we present the results of sentence-level and token-level querying strategies. 7.1 results of sentence-level strategy the asl algorithm that uses pseudo-labeling as the default self-learning strategy as proposed by [2], is referred to as dasl (deep active self-learning). to conduct an almost comprehensive experiment with various self-learning methods, we replace pseudo-labeling with three other techniques: dasl_wd (deep active self-learning with word dropout), dasl_vat (deep active self-learning with virtual adversarial technique), and dasl_cvt (deep active self-learning with cross-view training). we also compare the results of active self-learning strategies with deep active learning (dal). in fig. 1, supervised (i.e. dashed line) represents the best performance of the neural models when trained using the entire training set with true labels in a supervised fashion. dal represents the baseline active learning algorithm by shen et al. [18]. figure 3 presents the performance (i.e., f1-score) achieved by the models and the percentage of oracle annotated tokens (human labeled tokens). it should be noted that active self-learning algorithms use additional data for training in addition to the hand-annotated data. note that the dasl_cvt technique is only implemented with the cnn-bilstm-crf model. this technique required multiple views of the input, which is more computationally costly to implement in neural models with cnn-only encoders such as the cnn-cnn-lstm. from the experiments performed in fig. 3, we noticed that no technique performed consistently better than the others. however, for the aposentadoria dataset, the dasl_vat and dasv_cvt techniques help the model achieve its best performance faster than the baseline algorithms. we believe this behavior is because this dataset is less imbalanced than the other datasets, meaning that most ner entities are present in most sentences. as a result, the model generalizes faster with less hand-annotated data and more effectively utilizes unlabeled data through self-training in the early iterations of the active self-learning algorithms. fig. 3. the graph shows the performance (f1-score) of the neural models as a function of the percentage of hand-annotated tokens in each iteration of the active and active self-learning algorithms. full size image 7.2 results of token-level strategy in this section, we present the results of the token-level active self-learning algorithm proposed. our algorithm, namely mdal, is a modified version of deep active learning (dal). figure 4 compares the original and the modified dal algorithms, both using the validation f1-score for early stopping. the graphs show that both the original and the modified dal algorithms achieve similar performance with the same amount of labeled data but with our modified algorithm allowing for tokens to be self-labeled. from the graphs in the center, we observe that the models trained with the modified dal algorithm reach peak performance with significantly less hand annotated tokens. this is justified by the fact that for the modified algorithm, most tokens are self-annotated by the trained model reliably, as shown in the graphs to the right in fig. 4. table 2 presents the percentage of hand-annotated tokens required for the trained model to reach 99% of its peak f1-score performance. we observe that the proposed method requires significantly fewer hand-annotated tokens. however, we do note that these results do not consider that many of the tokens in the queried sentences were not named entities and therefore were not required to be annotated. this is the reason why our algorithm is capable of training a model to peak performance with only 6.96% of human-labeled tokens in the conll2003 dataset when compared to the 36.20% and 27% of the baseline algorithm. however, for fine-grained ner datasets, such as the ontonotes5.0, and datasets with few tokens that do not have a named entity class, such as the aposentadoria dataset, the presented results are convincing evidence that our method is capable of reducing the human annotation costs in the active learning process when compared to the baselines. table 2. percentage of the training set that was annotated by the oracle in order to train a model that reaches 99% of its peak performance.full size table fig. 4. comparison between the original and modified dal algorithms. graphs on the left column compare the performance of the trained model (y-axis), by the total amount of labeled data (x-axis), including tokens annotated by the oracle and self-annotated by the trained model in the case of our modified algorithm. graphs on the center column compare the f1-score of the trained model (y-axis) by the amount of data annotated by the oracle (x-axis). graphs on the right column compare the percentage of the training set that was annotated by a human, by the model through self-labeling, and the samples that were mislabeled. in all graphs, dal represents the original deep active learning algorithm from the literature, while mdal indicates our modified algorithm with token-level self-labeling. full size image 8 conclusion in this work, we presented an investigation into different types of active-self learning algorithms for named entity recognition tasks. from the many experiments performed, the sentence-level active self-learning algorithm could not consistently achieve significant results compared to pure active learning. however, the proposed token-level active self-learning could train a neural model to near-peak performance using fewer human-annotated tokens compared to the state-of-the-art algorithms used as baselines. our proposed token-level algorithm is particularly effective for fine-grained datasets where most tokens are assigned a named entity class. while we failed at creating a sentence-level active self-learning that overcomes the current state-of-the-art, future research may investigate how pretrained transformer models may impact these algorithms. bridging the current gaps in sentence-level active self-learning algorithms research using models with substantial a priori information may be possible. references neto, j.r.c.s.a.v.s.: deep active learning approaches to the task of named entity recognition. masters dissertation [university of brasilia] (2021) google scholar  neto, j.r.c.s.a.v.s., faleiros, t.p.: deep active-self learning applied to named entity recognition. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 405–418. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_28 chapter  google scholar  clark, k., luong, m.t., manning, c.d., le, q.: semi-supervised sequence modeling with cross-view training. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 1914–1925. association for computational linguistics, brussels, belgium, october–november 2018 google scholar  goodfellow, i.j., shlens, j., szegedy, c.: explaining and harnessing adversarial examples (2015) google scholar  hartmann, n.s., fonseca, e.r., shulby, c.d., treviso, m.v., rodrigues, j.s., aluísio, s.m.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: anais do xi simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 122–131. sbc, porto alegre, rs, brasil (2017) google scholar  houlsby, n., huszár, f., ghahramani, z., lengyel, m.: bayesian active learning for classification and preference learning (2011) google scholar  kobayashi, k., wakabayashi, k.: named entity recognition using point prediction and active learning. in: proceedings of the 21st international conference on information integration and web-based applications and services, iiwas2019, pp. 287–293. association for computing machinery, new york, ny, usa (2019) google scholar  lakshmi narayan, p., nagesh, a., surdeanu, m.: exploration of noise strategies in semi-supervised named entity classification. in: proceedings of the eighth joint conference on lexical and computational semantics (*sem 2019), pp. 186–191. association for computational linguistics, minneapolis, minnesota, june 2019 google scholar  ma, x., hovy, e.: end-to-end sequence labeling via bi-directional lstm-cnns-crf. in: proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1064–1074. association for computational linguistics, berlin, germany, august 2016 google scholar  miyato, t., dai, a.m., goodfellow, i.: adversarial training methods for semi-supervised text classification. in: international conference on learning representations (iclr) (2017) google scholar  nair, v., hinton, g.e.: rectified linear units improve restricted boltzmann machines. in: proceedings of the 27th international conference on international conference on machine learning, icml 2010, pp. 807–814. omnipress, madison, wi, usa (2010) google scholar  neubig, g., nakata, y., mori, s.: pointwise prediction for robust, adaptable japanese morphological analysis. in: proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 529–533. association for computational linguistics, portland, oregon, usa, june 2011 google scholar  park, j., kim, g., kang, j.: consistency training with virtual adversarial discrete perturbation (2021) google scholar  pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pp. 1532–1543. association for computational linguistics, doha, qatar, october 2014. https://doi.org/10.3115/v1/d14-1162 pradhan, s., et al.: towards robust linguistic analysis using ontonotes. in: proceedings of the seventeenth conference on computational natural language learning, pp. 143–152. association for computational linguistics, sofia, bulgaria, august 2013 google scholar  radmard, p., fathullah, y., lipani, a.: subsequence based deep active learning for named entity recognition. in: proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp. 4310–4321. association for computational linguistics, online, august 2021 google scholar  sang, e.f.t.k., meulder, f.d.: introduction to the conll-2003 shared task: language-independent named entity recognition. in: proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pp. 142–147 (2003) google scholar  shen, y., yun, h., lipton, z., kronrod, y., anandkumar, a.: deep active learning for named entity recognition. in: proceedings of the 2nd workshop on representation learning for nlp, pp. 252–256. association for computational linguistics, vancouver, canada, august 2017. https://doi.org/10.18653/v1/w17-2630 siddhant, a., lipton, z.c.: deep bayesian active learning for natural language processing: results of a large-scale empirical study. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2904–2909. association for computational linguistics, brussels, belgium, october–november 2018. https://doi.org/10.18653/v1/d18-1318 srivastava, n., hinton, g.e., krizhevsky, a., sutskever, i., salakhutdinov, r.: dropout: a simple way to prevent neural networks from overfitting. j. mach. learn. res. 15(1), 1929–1958 (2014) mathscinet  math  google scholar  tran, v.c., nguyen, n.t., fujita, h., hoang, d.t., hwang, d.: a combination of active learning and self-learning for named entity recognition on twitter using conditional random fields. knowl.-based syst. 132, 179–187 (2017) article  google scholar  download references acknowledgements the authors were supported by the fundação de apoio a pesquisa do distritio federal (fap-df) as members of the knowledge extraction from documents of legal content (knedle) project from the university of brasilia. author information authors and affiliations osaka university, osaka, japan josé reinaldo cunha santos a. v. silva neto university of brasilia, brasilia, df, brazil thiago de paulo faleiros authors josé reinaldo cunha santos a. v. silva netoview author publications search author on:pubmed google scholar thiago de paulo faleirosview author publications search author on:pubmed google scholar corresponding author correspondence to josé reinaldo cunha santos a. v. silva neto . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cunha santos a. v. silva neto, j.r., de paulo faleiros, t. (2023). investigation of deep active self-learning algorithms applied to named entity recognition. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_31 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords active learning active self-learning named entity recognition deep learning publish with us policies and ethics profiles thiago de paulo faleiros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature police report similarity search: a case study | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper police report similarity search: a case study conference paper first online: 12 october 2023 pp 394–409 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) police report similarity search: a case study download book pdf download book epub josé alan firmiano araújo9, ticiana l. coelho da silva9,10, atslands rego da rocha9 & … vinicius cezar monteiro de lira10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 531 accesses abstract several crimes occur daily, and the initial investigation begins with a police report. in cities with high crime rates, it is impractical to expect the police to read and analyze every crime narrative. some police reports may involve multiple victims or the same crime may be reported more than once. additionally, police reports may exhibit similarities due to a shared modus operandi. this study addresses the challenge of providing a police report and searching for the most similar report in the database. a similar police report can be either another report with overlapping words or one that shares a similar modus operandi. one potential solution is to represent each police report as a feature vector and compare these vectors using a similarity function. different methods can be employed to represent the narrative, including embedding vectors and count-based approaches such as tf-idf. this research explores the use of pre-trained embedding representations at both the word and sentence levels, such as universal sentence encoder, word2vec, roberta, doc2vec, among others. we determine the most effective representation for capturing semantic and lexical similarities between police reports by comparing different embedding models. furthermore, we compare the effectiveness of available pre-trained embedding models with a model trained specifically on a corpus of police reports. another contribution of this work is the development of trained embedding models specifically tailored for the domain of police reports. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others crime analysis and evidence-based policing: challenges and obstacles chapter © 2022 owning police reform: the path forward for practitioners and researchers article 15 december 2022 measuring the impacts of everyday police proactive activities: tackling the endogeneity problem article 16 february 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization criminal justice criminal behavior crime prevention policing quantitative criminology 1 introduction usually, victims and witnesses report the crime events on police reports for the police investigation. the police report also protects the police action, demonstrating where this series of investigative efforts by the police department began. a police report might even make it to the supreme court in severe circumstances. consequently, a police report needs to be thorough and factual when describing an occurrence of crime (for example, accurate, concise, clear, objective, timely and complete). this paper addresses the challenge of conducting a search for the most similar police report in a database. when it comes to similarity search in text data, the goal is to find the most similar police report(s) based not only on string similarity but also taking account semantic similarity. a similar police report can be a police report with overlapping words or one that describes an equivalent modus operandi, even if it does not use the exact words. to illustrate both cases, let’s consider the following two sentences: “augusta’s car was stolen yesterday at night” and “yesterday night, augusta had her car stolen.” these sentences share almost identical wording. now, let’s examine two different sentences that exhibit the same modus operandi: “two armed men on a motorbike robbed patricia and took her mobile phone” and “two men with a knife on a motorbike approached mary and stole her personal belongings.” these police reports involve the same type of crime (robbery), a female victim, and a similar criminal approach (two individuals on a motorbike). one possible approach to address this issue is to represent each police report as a feature vector and compare these vectors using cosine similarity. pre-trained word embeddings are one of the most commonly used representations for document vocabulary. there are various pre-trained word embeddings available, such as [13, 16, 17, 19]. word embeddings can capture a word’s relationships with other words, its contextual meaning within a document, as well as its semantic and syntactic similarities. however, word embeddings alone may struggle to capture nuanced shifts in meaning when a sentence undergoes minor changes. for example, consider the sentences “augusta’s car was stolen” and “augusta’s car was not stolen.” despite the difference of only one word, these sentences have completely different meanings. nonetheless, when using word embeddings, the cosine similarity between the vectors obtained from these sentences might still be relatively high, despite being semantically opposite. embedding techniques should ideally be able to address this issue and provide more nuanced representations. one alternative is to encode the entire sentence into embedding vectors. this method is known as sentence embedding, and there are many pre-trained sentence embedding techniques available, including doc2vec [15], sbert [3] and universal sentence encoder [21]. these models take the text as input and generate a fixed-dimensional embedding representation of the sentence. sentence embedding techniques aim to capture the full sentences and the semantic information they contain in the form of vectors. this helps the machine understand the context, intention, and other nuances of the text as a whole. the quality of the training data has a significant impact on the resulting sentence embedding vectors. for optimal results, the sentences in the training set should be semantically related [14]. this study focuses on addressing the challenge of conducting similarity searches on police report documents. the primary objective is to find similar police reports with overlapping words or reports that share the same modus operandi. police reports pose unique challenges for similarity search due to specialized language and terminology specific to law enforcement, requiring tailored approaches. furthermore, the wealth of highly contextual information in these reports, encompassing location, time, individuals involved, and preceding events, adds complexity, necessitating context-aware techniques for accurate assessment. additionally, the sensitivity of confidential police reports, containing personal details and classified materials, demands privacy protection measures during similarity analysis, adhering to legal and ethical guidelines through anonymization or aggregation techniques. to guide this research, we have formulated the following key research questions: 1. (rq1) the first research question aims to identify the pre-trained sentence-level representation that effectively captures both the syntactic and semantic features of the vocabulary found in police reports while being able to differentiate between similar reports. in this study, we consider various approaches for input sentence representation, including word-level and sentence-level embeddings, as well as count-based text representation techniques such as tf-idf. additionally, we explore lexical-based similarities, such as jaccard similarity, to analyze the effectiveness of different representation methods. the goal is to determine the most suitable approach that can accurately capture the essential characteristics of police reports and enable the detection of similar reports. 2. (rq2) our second research question delves into the effectiveness of embedding models that are trained from scratch with police report documents. we aim to determine if these models are better at representing sentences for similarity searches. furthermore, we compare the pre-trained models used in rq1 with our own data-trained embeddings. our primary goal is to assess the performance of these various versions when it comes to searching for similar police reports. 3. (rq3) the third research question explores whether combining embeddings can enhance the quality of searching for similar police reports. building on the findings from [8], which highlights the benefits of combining different embeddings due to their complementarity, we investigate whether combining embeddings can lead to improved performance on the task at hand. based on the analysis conducted in the previous research questions, we select the two best-performing embeddings. we then compare the effectiveness of combining these two embeddings using different aggregation techniques, such as taking the maximum, minimum, or average vector. by investigating the potential of combining models, we aim to determine whether it is possible to achieve a better sentence representation that captures the semantic and syntactic features crucial for searching similar police reports. the remaining sections of this article are organized as follows. section 2 presents the background and related works. section 3 explains the data and the solutions to cope with our problem. section 4 explains the evaluation metrics used in the comparisons and shows the analysis performed compared to different models. finally, sect. 5 summarizes this work and proposes future developments. 2 backgound and related works the background information and studies that are relevant to our problem are covered in this section. we divide this section into four main topics: traditional text representation, word embedding, sentence embedding and similarity search in text data. traditional text representation. the method of vectorizing text involves turning text into tensors of numbers. the most frequent and fundamental representation is one-hot encoding. every word is given a distinct integer index, and this integer index i is then converted into a binary vector of size n (the vocabulary size). the vector is all zeros except for the \(i-th\) entry, which is 1 [4]. another typical representation of text data is the use of bag-of-words or bag-of-n-words [9] for simplicity. according to this paradigm, a sentence or text is typically depicted as a collection of words, ignoring grammar and word order but keeping multiplicity. there are some alternatives to represent each word in a document for such a model: (i) a boolean indicating the presence or not; (ii) the frequency of the word in the text; (iii) tf-idf (term frequency, inverse document frequency), a statistical tool that assesses how pertinent a word is to a document within a collection of documents. the two metrics are multiplied: how many times a word appears in a document (tf) and the inverse document frequency of the word across a set of documents (idf). however, these techniques have some drawbacks, such as the lack of semantics over the representation. in this paper, we experimented with the representation of each word in a sentence as its tf-idf value. word embeddings. text of various lengths (such as a document, paragraph, sentence, or even a word) is transformed by embedding models into a fixed-length numeric vector that may be used to train machine learning algorithms like classification and clustering models. pre-trained word embeddings have been widely used [1, 16, 17, 19, 24] due to their capacity to retain a word’s context inside a document, as well as their semantic and syntactic similarity to other words. a framework for learning word vectors is proposed by [17] by training a linguistic model that predicts a word given other words in a context. word2vec is one specific implementation of such a structure. the main disadvantage is that [17] under-utilizes corpus statistics as the model is trained on a separate local context window rather than on global co-occurrence counts. bypassing this issue, [19] provide a model that creates a vector space of words by training it on the worldwide count of word co-occurrence, making effective use of statistical data. another word embedding proposal is flair [1], which abstracts away from the particular engineering difficulties that various word embeddings bring. it creates a uniform interface for all word, sentence, and arbitrary embedding combinations. brazilian portuguese bert models are provided by bertimbau [24]. three nlp tasks-textual similarity of the sentence, textual linkage recognition, and named entity recognition-were used to assess the models. compared to bert multilingual and earlier single language techniques, bertimbau advances state-of-the-art in these tasks, demonstrating the value of large pre-trained language models for portuguese. we can obtain document vectors from word embeddings. an approach is to average all word vectors collectively. however, this process weighs both significant and insignificant words equally. the fact that each word would be represented with the same vector, regardless of context, is another drawback of utilizing word embeddings to represent text. sentence embedding is an improvement from word embedding. sentence embedding. performs the encoding of sentences in an n-dimensional vector space, which might have several word representations based on context. the most widely used sentence embedding proposals are doc2vec [15], infersent [5], sbert [3], universal sentence encoder [21] and labse [7], among others. similar to word2vec, doc2vec trains sentence embeddings or paragraph vectors to predict the following word, given a variety of contexts drawn from the paragraph. the sentence and word vectors are combined to forecast the next word in a context. the pre-trained bert network is used by sbert [3], which adds a grouping operation to the output to create a fixed-length sentence embedding. to enhance bert and produce semantically meaningful sentence embeddings that can be compared using cosine similarity, sbert introduces siamese and triple lattice structures. using the output of cls-token, the developers of [3] experiment with various pooling strategies, such as finding the average of all output vectors (the default strategy) or computing a max-over-time of the output vectors (max-strategy). the trained labse model is proposed by [7] for sentence embeddings in several languages. only pairs of bilingual utterances that are translations of one another are used exclusively in training and optimizing labse to generate comparable representations. the source and destination phrases are independently encoded using a shared bert-based encoder in the framework’s labse, then passed into a combination function. this is known as a dual encoder. the sentence embedding for each entry is obtained from the last layer representations [cls]. the cosine over the sentence embedding created by bert encoders is used to score how similar the source and destination phrases are to each other. this paper assesses a variety of embeddings, including word2vec, flair, and bert as word embeddings, as well as doc2vec, the universal sentence encoder, and sbert. similarity search in text data. several studies have been conducted to assess the effectiveness of different embeddings for text similarity. one such study, as suggested by the authors of [23], compares the semantic similarity between various techniques in the patent field. this study utilizes three different vector spaces, including a basic tf-idf baseline, an lsi topic model, and a doc2vec neural model. another article, [18], compares different learning vector representations like latent semantic analysis (lsa), word2vec, and glove to identify the most efficient technique in the subject segmentation space. [22] trains a supervised meta-embedding neural network and combines multiple sentences pre-trained embedding models for sentence similarity recognition. the meta-model outputs the distance between two given sentences as input. the approach for selecting the best embedding models in our study can be applied to the methodology proposed by [22] to choose the embedding models for effectively creating their meta-embedding model. in addition, [12] tackles the issue of recognizing duplicate questions. this article proposes a method that vectorizes questions based on the combination and individual use of fasttext mining subword integration, google news vector integration, and fasttext mining integration. these question word vectors are then used to determine how similar words are semantically. the proposed model is trained on each of the characteristics derived from each of the three words separately, and input is received by the malstm neural network, which measures the manhattan distance to assess the semantic similarity between questions. various studies have utilized embeddings to map input to low-dimensional vectors and perform similarity searches. for instance, [11] investigates embedding complex data objects like dna sequences and images in a vector space to make their distances as close to their real distances as possible. this allows queries to be run on the embedded objects in similarity search applications. similarly, [6] conducts similarity searches in knowledge graphs (kg) to help users identify the most significant entities to their query. 3 data and methods this section discusses our datasets and methodology for investigating the research questions. we employ a two-step approach: generating similarity matrices for different sentence representations and validating the accuracy and mean reciprocal rank (mrr) of police report representations. the first step involves data preprocessing, sentence representation, model training, and ranking of the top k most similar sentences using cosine and jaccard similarity. in the second step, we evaluate the accuracy of each police report representation by determining if the most similar report is retrieved within the top k sentences. we also calculate the mrr to assess the retrieval performance. we provide more details about mrr as follows. figure 1 presents the steps followed by us in this paper. fig. 1. steps followed to assess the research questions in our work. full size image data. two brazilian department of public security organizations provided the datasets utilized in this study. to maintain anonymity during the review process, we have omitted their names. the datasets comprise two corpora: one with 1,089 police reports (1,065 reports are similar, containing at least one copy for each incident report), and another with 30,011 reports, both electronic and non-electronic. these corpora cover incidents that occurred between january 1, 2020, and march 29, 2020, encompassing various types of crimes such as theft, murder, robbery, attempted murder, and harassment. the first corpus with 1,089 police reports has 1,065 reports in all, in which each police report has a duplicate in this same set. another 24 reports did not have similarities and were discarded from validation. figure 2 presents some important information about the first dataset (with 1,065 police reports): fig. 2. distribution of the number of documents by the word lenghh full size image each column bin varies the number of words to 20; for instance, the first column contains 38 reports (for up to 20 words in each report). it is possible to observe that the number of documents decreases as we move to subsequent columns, i.e., fewer documents with a high number of words. in addition, fig. 2 presents an asymmetry to the right, suggesting a greater concentration of documents with fewer words and a smaller number of documents with a more significant number of words in the documents. some statistical measures from our dataset: the maximum document length in terms of the number of words is 1,019; on average, each document contains 119 words and around 597 characters. to provide an example of a police report and its duplication: original report: “the reporting party, identified above, informs via the virtual police station that on the specified date and time, they were the victim of a robbery. the incident occurred as follows: two individuals on a black motorcycle, wearing helmets and armed with a knife, approached and demanded the cell phone. they took a samsung phone, model a205gt, with the serial number: 357621102968577. when asked about identifying or describing the suspects, the reporting party stated that it was not possible due to their helmets.” duplicated report: “the reporting party, identified as mentioned above, reports via the virtual police station that on the mentioned date and time, they were the victim of a robbery. the incident occurred as follows: two individuals on a black motorcycle, identified as pop, with a knife, threatened the reporting party and demanded the cell phone. when asked about identifying or describing the suspects, the reporting party stated that they were unable to do so as the suspects were wearing helmets.” duplications of police reports can happen due to several factors: (i) the same occurrence may have happened to different victims, and thus each victim records the same event, however, with different words; (ii) electronic police reports can be made due to system or user failure, these police reports can be registered more than once, among other possibilities. during the pre-processing phase, we conducted several steps. these included the removal of special characters, such as html tags and non-ascii characters, as well as the elimination of terms containing digits. additionally, we eliminated stopwords and converted all characters to lowercase. to represent the police reports using embedding models, we explored three potential solutions: utilizing pre-trained embeddings, training embeddings from scratch using our data, and combining different embeddings to represent the text. methods. in this paper, we utilize pre-trained neural embedding models, including flair, universal sentence encoder (use), doc2vec, word2vec, and sbert. firstly, we employ word embeddings by computing the average of word representations to describe the narrative of each police report using models such as word2vec. additionally, we investigate the performance of word2vec, doc2vec, and roberta when trained from scratch using our dataset, which consists of approximately 30,011 reports obtained. each word is assigned an initial representation, and the models refine these representations through training iterations to enhance their quality. to compare the performance of the embedding models and lexical approaches, we evaluate the use of a count-based method, tf-idf, where each word in a sentence is represented by its tf-idf value. after representing each police report as a vector using either an embedding model or tf-idf, we compare the reports to identify the most similar ones. for each representation approach investigated, we generate an n x n matrix to rank the top k most similar police reports for each report. for example, in the case of the dataset containing 1,065 police reports represented with word2vec, we calculate the cosine similarity between all 1,065 sentences, resulting in a 1,065\(\,\times \,\)1,065 matrix that represents the similarity percentage. we disregard the diagonal of the matrix since the similarity between identical sentences would always be 100%. this similarity matrix is generated for each representation approach, including word2vec, flair, use, doc2vec, sbert, roberta, and tf-idf. furthermore, we explore an alternative approach that retains the words in the sentences without representing them as vectors. in this case, we compute the jaccard similarity matrix, which compares two sentences by assessing if they contain identical words. the jaccard similarity ranges from 0% to 100%, with a higher percentage indicating a greater similarity between the sentences. evaluation metrics. the ranking step in our evaluation involves analyzing the top k (top 1, top 5, and top 10) most similar sentences and calculating the mean reciprocal rank (mrr). for each value of k, a list of the most similar police report ids is generated. in the case of top 1, the list contains only the most similar sentence, while in the case of top 5, the list includes the five most similar sentences, and so on. if the list of most similar police report ids includes the correct duplicate police report, it is considered a hit for the respective approach used for sentence representation. the approach’s accuracy is calculated as the percentage of hits achieved by the approach across the entire dataset. the formula of mrr is shown below: $$\begin{aligned} \text {mrr} = \frac{1}{|d|} \sum _{i=1}^{|d|} \frac{1}{\text {rank}_i}, \end{aligned}$$ (1) where d is the dataset of police reports and \(\text {rank}_{i}\) refers to the rank position of the first relevant document (most similar) for the \(i-th\) police report. additionally, the mrr is computed for each method, providing a harmonic mean of the ranks. comparative studies of word embedding vectors, such as those conducted by [2, 25], play a vital role in ensuring the quality of word representation before their utilization in machine learning tasks. evaluation methods for word vectors can be categorized as intrinsic or extrinsic [20, 26]. intrinsic evaluation focuses on assessing word relationships based on syntax or semantics and is independent of a particular nlp task. human-curated benchmark datasets are used to compare words in terms of similarity and relatedness. on the other hand, extrinsic evaluation involves incorporating word vectors into an nlp task, such as sentiment analysis or natural language inference. the performance of the embedding models is then evaluated using task-specific metrics like accuracy, f1-score, or other relevant measures. this type of evaluation assesses the usefulness and effectiveness of the embeddings in real-world applications and can assist data scientists. 4 experimental results in this section, we analyze and evaluate the results obtained using pre-trained embedding models as well as embedding models trained on a separate dataset of police reports. we investigate the impact of these models on identifying similar sentences, including the ones with the same modus operandi. additionally, we examine the behavior of count-based methods, such as tf-idf, in representing the text. furthermore, we explore the potential benefits of combining different embeddings to improve the search for similar police reports. the research questions defined in the introduction guide our analysis in this section. by addressing these questions, we gain insights into the performance and effectiveness of various embedding models and representation techniques in the context of police reports. this evaluation allows us to understand the strengths and limitations of different approaches and provides valuable information for researchers and practitioners working in the field of natural language processing and law enforcement. study on the results of rq1. the first research question aims to determine the most effective representation method for capturing syntactic and semantic similarity between police reports. to investigate this question, we utilize a dataset containing 1,065 police reports. this dataset is accompanied by ground truth annotations, meaning that we have information about the most similar police report for each entry. in our evaluation, we consider various pre-trained models, including universal sentence encoder (use), word2vec, doc2vec, bert, sbert, and flair. for each approach, we compare the sentences using cosine similarity. additionally, we explore the option of keeping the words in the sentences without converting them into vector representations. in this case, we utilize jaccard similarity to identify the most similar sentences. table 1 presents the accuracy and mrr results obtained by these approaches. these metrics allow us to assess the performance and effectiveness of each representation method in capturing the similarity between police reports. table 1. results from the approaches for rq1full size table the use (universal sentence encoder) model demonstrated the best performance among the embedding models evaluated. it is based on a transformer encoder and achieved higher accuracy compared to the deep averaging network (dan) model. the accuracy achieved by use is similar to the results reported in the paper by [3] for the sts bench (semantic textual similarity benchmark). the use model incorporates sentences into a 512-dimensional vector, enabling a more robust semantic representation. moreover, the multilingual use model allows for the representation of words from different languages. bert, another transformer-based model, obtained the second-best results among the models in capturing syntactic and semantic information. the pre-trained bert model from [24]footnote 1 slightly outperformed the pre-trained word2vec model from [10]footnote 2. sbert and flair models showed similar performance, likely because they are both based on the same training base. however, the doc2vec model performed poorly compared to the other models and ranked last. this could be attributed to the use of portuguese wikipedia data for training, which may have introduced variations in phrase usage across different contexts. tf-idf was computed using the tfidfvectorizerfootnote 3 to represent the sentences. this involved creating a dictionary of words from the sentences and calculating the frequency of each term. portuguese stopwords were removed to ensure consistency in similarity calculations. the tf-idf representation yielded promising results. one key takeaway from this experiment: it is indeed expected that the embeddings used would be able to capture the context of the sentences and identify duplications as accurately as, if not better than, count-based and lexical methods. however, in the case of police reports, the text may not be exactly the same in terms of words, as demonstrated in the previous section. there may be repetitions of words, but they can occur in a different order within the text, which is a characteristic of this type of dataset. embeddings that are pre-trained on non-police data may not capture this particularity of the police reports as effectively as count-based and lexical methods. the pre-trained embeddings may have learned general language patterns, but they may not have specifically captured the nuances and variations present in police reports. this can result in lower accuracy when identifying similar sentences. in such cases, count-based and lexical methods, like tf-idf and jaccard similarity, tend to perform better because they directly compare the presence or absence of words in the sentences. they are not reliant on capturing contextual information and can handle cases where word order varies within the text. therefore, it is important to consider the specific characteristics of the dataset and the task at hand when choosing an appropriate representation method. while pre-trained embeddings offer many advantages, they may not always be the optimal choice for certain types of text, such as police reports with specific structural patterns and word repetitions. study on the results of rq2. the second research question investigates whether the embedding models are more effective in representing the sentences when trained with a police report dataset. using an unsupervised approach, this study trained the roberta (an improvement of the bert model), doc2vec, and word2vec embedding models. the data for training is around 30,011 police reports provided by brazilian department of public security. to investigate rq2, we use the dataset with 1,065 police reports (the same used for rq1). the roberta trained model was more assertive in identifying similar sentences; we trained the roberta model for 13 epochs when the difference between the current and last epoch loss was not more significant than 0.01. roberta embodied each word in 512 dimensions. the representation of the police reports for word2vec was made with vectors of 50 dimensions, trained for 100 epochs. for the doc2vec model, the representations of the sentences were with vectors of 150 dimensions, trained for 100 epochs, with a training window of size 10. the findings of the study suggest that training embedding models with a specific dataset from the same context, in this case, the police report dataset, improves their effectiveness in representing sentences and capturing syntactic, semantic, and morphological relationships. the word2vec, roberta, and doc2vec models trained with the 30,011 police reports achieved slightly better results compared to their pre-trained counterparts, which were trained on a much larger and diverse corpus. the advantage of training embedding models with domain-specific data is that they can better capture the specific language patterns, vocabulary, and contextual relationships present in that particular domain. the embeddings trained on the police report dataset were able to maintain proximity between vectors representing syntactic, semantic, and morphological relationships relevant to police reports. this proximity allows for better identification of similar sentences and captures the nuances of language specific to the domain. the results obtained with the trained models were considered satisfactory, considering that the pre-trained versions of word2vec, bert, and doc2vec were trained on a much larger corpus with a significantly larger number of tokens. the pre-trained versions of word2vec, bert, and doc2vec were trained with 1,395,926,282 tokens. our 30k police reports are much smaller (with 2,579,815 tokens). despite the smaller size of the police report dataset, it still proved to be effective in capturing the relevant patterns and relationships needed for the task at hand. this highlights the importance of training embedding models on data that is specifically relevant to the target domain, as it can lead to improved performance and accuracy in representing and analyzing text data (table 2). table 2. results from the approaches for rq2full size table study on the results of rq3. the third research question explores the potential of combining embeddings to enhance the search for similar police reports and improve the retrieval of similar reports. in this study, the embeddings from the use model and word2vec-trained model, both trained with the 30,011 police reports, were combined to create a new embedding model called usew2v. to ensure compatibility in dimensionality, the word2vec model was adjusted to generate embeddings with 512 dimensions, matching the dimensionality of the embeddings generated by the use model. by combining the embeddings of both models, the usew2v model was created. the performance of the usew2v model was compared against the individual use and word2vec models. the evaluation focused on assessing the accuracy and effectiveness of the combined embedding model in identifying similar police reports and retrieving similar reports. by combining the strengths and features of both models, it was expected that the usew2v model could outperform the individual models in terms of accuracy and retrieval performance. table 3. results from the approaches for rq3full size table table 3 presents the results of the usew2v models (max, min, mean) compared to the individual use and word2vec models for the 1,065 police reports dataset. it can be observed that the combination of the two best embedding models, referred to as usew2v models, yielded similar results to those obtained using only the use model. since the 1,065 sentences in the dataset share a similar writing syntax, the improvement achieved by combining the two models to represent the sentences was not significant. however, it is worth noting that combining embeddings can sometimes yield results that are on par with or even better than the individual models. this experiment demonstrates that combining different embedding models can be a viable approach, and there are various methods available for combining embeddings. exploring these options and evaluating their effectiveness can be a valuable direction for future research in the field. 5 conclusion and future works this paper addresses the problem of finding the most similar police report in a database given a specific report. the goal is to consider not only lexical similarity but also syntactic and semantic similarity. the experiments conducted in this paper showed that the use model, a pre-trained transformer-based sentence embedding model in its multilingual version, outperformed the other embedding models. interestingly, even the raw comparison methods using tf-idf and jaccard similarity achieved similar results to the use model. furthermore, the study found that training an embedding model with data from the same context as the target dataset is more effective than using pre-trained embeddings. the paper also explored the combination of different embeddings using various aggregation functions such as max, min, and average. however, it was observed that while combining embeddings can be competitive with using individual models, it might not surpass their performance as expected. future directions for research include evaluating the proposed approaches with different datasets, exploring embeddings from different architectures like graph embeddings, and investigating metrics to compare embeddings and assess their suitability for specific datasets and machine learning problems. these directions can contribute to further advancements in the field of similarity search and text representation. another potential future direction is to explore the utilization of our dataset with 30,011 police reports for fine-tuning the models studied in the first research question. another suggestion about combining embeddings would be to concatenate two embeddings generated by different models and assess whether it is possible to obtain a better representation. notes 1.available at https://huggingface.co/neuralmind/bert-base-portuguese-cased. 2.available at https://nilc.icmc.usp.br/embeddings. 3.https://shorturl.at/lmcxy. references akbik, a., bergmann, t., blythe, d., rasul, k., schweter, s., vollgraf, r.: flair: an easy-to-use framework for state-of-the-art nlp. in: proceedings of naacl (demonstrations), pp. 54–59 (2019) google scholar  boggust, a., carter, b., satyanarayan, a.: embedding comparator: visualizing differences in global structure and local neighborhoods via small multiples. in: 27th international conference on intelligent user interfaces, pp. 746–766 (2022) google scholar  cer, d., et al.: universal sentence encoder. arxiv preprint arxiv:1803.11175 (2018) chollet, f.: deep learning with python. simon and schuster (2021) google scholar  conneau, a., kiela, d., schwenk, h., barrault, l., bordes, a.: supervised learning of universal sentence representations from natural language inference data. in: proceedings of the 2017 emnlp, pp. 670–680 (2017) google scholar  do, p., pham, p.: w-kg2vec: a weighted text-enhanced meta-path-based knowledge graph embedding for similarity search. neural comput. appl. 33(23), 16533–16555 (2021) article  google scholar  feng, f., yang, y., cer, d., arivazhagan, n., wang, w.: language-agnostic bert sentence embedding. arxiv preprint arxiv:2007.01852 (2020) ghannay, s., favre, b., esteve, y., camelin, n.: word embedding evaluation and combination. in: proceedings of the tenth international conference on language resources and evaluation (lrec 2016), pp. 300–305 (2016) google scholar  harris, z.s.: distributional structure. word 10(2–3), 146–162 (1954) article  google scholar  hartmann, n., fonseca, e., shulby, c., treviso, m., silva, j., aluísio, s.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: proceedings of the 11th stil, pp. 122–131 (2017) google scholar  hjaltason, g.r., samet, h.: properties of embedding methods for similarity searching in metric spaces. ieee trans. pattern anal. mach. intell. 25(5), 530–549 (2003) article  google scholar  imtiaz, z., umer, m., ahmad, m., ullah, s., choi, g.s., mehmood, a.: duplicate questions pair detection using siamese malstm. ieee access 8, 21932–21942 (2020) article  google scholar  kenton, j.d.m.w.c., toutanova, l.k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of naacl-hlt, pp. 4171–4186 (2019) google scholar  kiros, r., et al.: skip-thought vectors. in: advances in neural information processing systems (nips), pp. 3294–3302 (2015) google scholar  le, q., mikolov, t.: distributed representations of sentences and documents. in: proceedings of the 31st international conference on machine learning (icml), pp. 1188–1196. pmlr (2014) google scholar  liu, y., et al.: roberta: a robustly optimized bert pretraining approach. arxiv preprint arxiv:1907.11692 (2019) mikolov, t., sutskever, i., chen, k., corrado, g.s., dean, j.: distributed representations of words and phrases and their compositionality. in: advances in neural information processing systems (nips), pp. 3111–3119 (2013) google scholar  naili, m., chaibi, a.h., ghezala, h.h.b.: comparative study of word embedding methods in topic segmentation. procedia comput. sci. 112, 340–349 (2017) article  google scholar  pennington, j., socher, r., manning, c.d.: glove: global vectors for word representation. in: proceedings of the 2014 emnlp, pp. 1532–1543 (2014) google scholar  qiu, y., li, h., li, s., jiang, y., hu, r., yang, l.: revisiting correlations between intrinsic and extrinsic evaluations of word embeddings. in: sun, m., liu, t., wang, x., liu, z., liu, y. (eds.) ccl/nlp-nabd -2018. lncs (lnai), vol. 11221, pp. 209–221. springer, cham (2018). https://doi.org/10.1007/978-3-030-01716-3_18 chapter  google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. arxiv preprint arxiv:1908.10084 (2019) rodrigues, a.c., marcacini, r.m.: sentence similarity recognition in portuguese from multiple embedding models. in: 2022 21st ieee international conference on machine learning and applications (icmla), pp. 154–159. ieee (2022) google scholar  shahmirzadi, o., lugowski, a., younge, k.: text similarity in vector space models: a comparative study. in: 2019 18th ieee icmla, pp. 659–666. ieee (2019) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th bracis (2020) google scholar  toshevska, m., stojanovska, f., kalajdjieski, j.: comparative analysis of word embeddings for capturing word similarities. arxiv preprint arxiv:2005.03812 (2020) zhai, m., tan, j., choi, j.: intrinsic and extrinsic evaluations of word embeddings. in: proceedings of the aaai conference on artificial intelligence, vol. 30 (2016) google scholar  download references acknowledgments the research reported in this work received support from the funcap project titled “big data platform to accelerate the digital transformation of ceará state" 04772551/2020. part of the results presented in this work were also obtained through the ufc-fastef 31/2019. author information authors and affiliations federal university of ceará, fortaleza, brazil josé alan firmiano araújo, ticiana l. coelho da silva & atslands rego da rocha insight data science lab, fortaleza, brazil ticiana l. coelho da silva & vinicius cezar monteiro de lira authors josé alan firmiano araújoview author publications search author on:pubmed google scholar ticiana l. coelho da silvaview author publications search author on:pubmed google scholar atslands rego da rochaview author publications search author on:pubmed google scholar vinicius cezar monteiro de liraview author publications search author on:pubmed google scholar corresponding author correspondence to ticiana l. coelho da silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper araújo, j.a.f., coelho da silva, t.l., da rocha, a.r., de lira, v.c.m. (2023). police report similarity search: a case study. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_26 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics profiles atslands rego da rocha view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a large dataset of spontaneous speech with the accent spoken in são paulo for automatic speech recognition evaluation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a large dataset of spontaneous speech with the accent spoken in são paulo for automatic speech recognition evaluation conference paper first online: 30 january 2025 pp 33–47 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a large dataset of spontaneous speech with the accent spoken in são paulo for automatic speech recognition evaluation download book pdf download book epub rodrigo lima  orcid: orcid.org/0009-0009-4344-11099, sidney e. leal  orcid: orcid.org/0000-0002-8817-20639, arnaldo candido junior  orcid: orcid.org/0000-0002-5647-089110 & … sandra m. aluísio  orcid: orcid.org/0000-0001-5108-26309  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 514 accesses abstract we present a freely available spontaneous speech corpus for the brazilian portuguese language and report preliminary automatic speech recognition (asr) results, using both the wav2vec2-xlsr-53 and distil-whisper models fine-tuned and trained on our corpus. the nurc-sp audio corpus comprises 401 different speakers (204 females, 197 males) with a total of 239.30 h of transcribed audio recordings. to the best of our knowledge, this is the first large paulistano accented spontaneous speech corpus dedicated to the asr task in portuguese. we first present the design and development procedures of the nurc-sp audio corpus, and then describe four asr experiments in detail. the experiments demonstrated promising results for the applicability of the corpus for asr. specifically, we fine-tuned two versions of wav2vec2-xlsr-53 model, trained a distil-whisper model using our dataset with labels determined by whisper large-v3 model, and fine-tuned this distil-whisper model with our corpus. our best results were the distil-whisper fine-tuned over nurc-sp audio corpus with a wer of 24.22% followed by a fine-tuned versions of wav2vec2-xlsr-53 model with a wer of 33.73%, that is almost 10% point worse than distil-whisper’s. to enable experiment reproducibility, we share the nurc-sp audio corpus dataset, pre-trained models, and training recipes in hugging-face and github repositories. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others incorporation of iterative self-supervised pre-training in the creation of the asr system for the tatar language chapter © 2021 brazilian portuguese speech recognition using wav2vec 2.0 chapter © 2022 fine-tuning xlsr-wav2vec2 for sanskrit asr with transformer chapter © 2026 1 introduction public or open datasets for training and evaluating automatic speech recognizers (asr) in brazilian portuguese (bp) have increased in number and hours since mid-2020, when there were approximately 60 h available, divided into four small datasets of read speech. in 2024, more than 8 thousand hours are available for training asr models, either from datasets automatically labelled or manually revised (see table 1 for a list of datasets). these bp resources allow the training of state-of-the-art asr models, such as wav2vec2-xlsr-53 [8] and distil-whisper [11]. specifically, both models are suitable for lowor medium-resource languages, providing consistent results without the need for training sets having tens of thousand of hours. wav2vec2-xlsr-53 is pre-trained in a large dataset of unlabeled speech encompassing 53 languages, including bp, through auto-supervised learning. the pre-training process allows for efficient fine-tuning on available bp datasets, generating competitive asr results. distil-whisper is a distilled version of whisper [19]. the latter, trained on 680,000 h of multilingual data in a multitask approach, has shown superior performance across multiple datasets and domains when compared to speech recognizers using the wav2vec2 model. a disadvantage of the various large whisper models is their requirement for a large dataset for increased performance and the inability to use them in computationally limited environments. distil-whisper, on the other hand, is a small variant based on knowledge distillation focused on efficiency which is 6 times faster, 49% smaller, and performs within 1% word error rate (wer) on out-of-distribution evaluation sets. to the best of our knowledge, distil-whisper has never been evaluated for bp datasets. regarding the data used for training speech recognizers, [10] characterizes them in two dimensions: (i) production style, which describes a continuum of spontaneity, coming from one side, which is planned speech, to the other, which would be unplanned speech; (ii) mode, which characterizes the elicitation process of the training pairs, in which on the one hand the text is used as a stimulus for speech (leading to a read speech) and on the other hand it is the speech that is transcribed into text. thus, read speech is planned speech that comes from a recited text and, at the other extreme, spontaneous/conversational speech is characterized as speech transcribed in an unplanned style. as a compromise, we have an speech prepared to be spoken later (e.g. the ted talksfootnote 1, in which exponents from all over the world present a talk in 18 min or less, resulting in the need to use a planned text to transmit the message in the short time). spontaneous speech has phenomena that make its recognition more complex than that of read or prepared speech. as a result, datasets with read/prepared speech were first and widely disseminated for training speech recognizers or speech synthesizers, mainly for the english language (see [17, 24]), since the words in the audios correspond directly to the words of the transcript, as they were read. however, more naturally occurring conversational and spontaneous speech contains many prosodic phenomena that are not present in read speech, such as higher degrees of segmental reductionfootnote 2 and more complex forms of variation (fillers, self-corrections and repetitions) affected by speech rate. thus, the various types of spontaneous speech that are common in everyday life, such as classes, conversations between two or more people, and interviews, are important materials for the datasets used to train asrs, as they bring natural intonation to questions, statements, and expressions of emotions such as surprise, admiration, indignation, anger, astonishment, fright, exaltation, enthusiasm, among others [4, 22]. furthermore, they bring linguistic phenomena such as filled pauses, generally written as “eh”, “ah”, “ahh”, “mm”, “uhn” and editing disfluencies (repetitions of words or parts of words, revisions of what is intended to be said, with speech restarts) [16]. consequently, speech recognizers are expected to perform worse with spontaneous speech than when applied to read speech. [19] shows the application of asr whisper to 14 datasets of read and spontaneous speech in the english language. the highest wer (word error rate) values occur for the spontaneous speech datasets, for example chime6footnote 3 has a wer of 25.5 and ami sdm1footnote 4 has a wer of 36.4 while common voicefootnote 5 has a wer of 9.0 and tedliumfootnote 6 has a wer of 4.0, using the large-v2 model. for brazilian portuguese, [15] reports a wer of 14.50% for the whisper large-v2 model applied to a portuguese dataset of approximately 17 h of spontaneous speech in interviews about life histories. this paper introduces a new corpus of spontaneous bp speech, particularly of paulistano accent (são paulo city), suitable for training and evaluating asr systems. the nurc-sp audio corpus is part of são paulo division of the nurc project (norma urbana linguística culta), a project designed to document and study portuguese spoken language by people with a high degree of formal education in five brazilian capitals (see details of this corpus in sect. 3). it contains 239.30 h of audios sampled at 16 khz and their respective transcriptions, totalling 170k segmented audios. the audios were automatically transcribed for the first time and manually revised aiming at the asr task. therefore, this new corpus adds 239 h to the amount available for bp (see table 1), totaling 8,598 h for training and evaluating speech recognition systems. in order to compare the quality of our corpus, four asr models are made available in this work: (i) a fine-tuned version of wav2vec2.0 xlsr-53 with the train and validation subsets of nurc-sp audio corpus, (ii) the same as the first, but using as start point the model that [6] trained for coraa-asr v1.1footnote 7, (iii) a distilled version of whisper large-v3 model [19], a model that has support for the portuguese language, trained using our dataset with labels determined by whisper large-v3, and (iv) a fine-tuned version of the third model with the train and validation subsets of nurc-sp audio corpus. the corpus and trained models are publicly available in our github repositoryfootnote 8 under the cc by-nc-nd 4.0 license. the main contributions made in this paper are summarised as follows: 1. a large bp corpus of human validated audio-transcription pairs containing 239.30 h of spontaneous speech. 2. the first corpus, according to our knowledge, tackling a large amount of accented paulistano speech for asr in bp (coraa asr brings 31.14 h of speech from são paulo capital) 3. four asr models, publicly available, based on the presented corpus. 2 related work on datasets of brazilian portuguese for asr table 1 presents large corpora for building asr systems focused on the portuguese language. some resources are multilingual, however table 1 specifically details the statistics for the portuguese language. among the resources presented, there is a slightly greater preponderance of the brazilian variant in the existing resources, although the european portuguese variant is also included in some of them. table 1. statistics of the main datasets available for asr in portuguese. some datasets are multilingual, however, the numbers shown are for the portuguese language.full size table one of the best-known projects dealing with read/prepared speech in english is librivoxfootnote 9, which distributes public domain books in audio format. these audios were used in several projects to create resources for processing speech in english, such as librispeech asr corpusfootnote 10 and librittsfootnote 11, both hosted in the open speech and language resources repository. the multilingual librispeech (mls) corpus [18] is a large multilingual corpus suitable for speech research, derived from read audiobooks from librivox and consists of 8 languages, including about 32k hours of english and a total of 4.5k hours for other languages. for portuguese, there are 131 h and 54 speakers. specifically for the recognition task, it can be combined with other resources, as it has relatively few speakers (audiobook speakers). these resources consist of cleaner audio, generally in studio quality. because of this, models built solely on this type of audio are only suitable for speech recognition in low-noise scenarios. to overcome this feature, one solution would be to inject noise into the audio or combine it with audio from other projects at different quality levels. the multilingual tedx corpus [21] was proposed to enable research in the areas of automatic speech recognition and speech-to-text translation. for portuguese, there are 164 h available in 93k audios. the corpus is made up of talks on a wide range of subjects, being managed within the scope of the tedx project, linked to the ted group (technology, entertainment and design). in the case of portuguese, there are also translations of the transcriptions into english and spanish. in addition, audios in spanish and french also have translations into portuguese. the spotify corpus [7] was first released in the english language. in 2022, the company launched a new versionfootnote 12 incorporating portuguese [12], offering several audios for the portuguese language coming mainly from podcasts available on the platform. in total, 76k hours of audio were made available from 123k episodes of shows on the platform. transcripts were automatically generated and are subject to transcription errors. it has a free license for academic use, but researchers interested in accessing the audios must submit a request for access on the organizers’ website. coraa asr is the combination of five independent projects dealing with speech in the interior of são paulo (with 35.96 hours – alip project), minas gerais (with 9.64 hours — c-oral brasil project), recife (141.31 hours – nurc-recife project) and são paulo capital (31.14 hours – sp2010 project), in addition to the prepared speech of tedx talks in brazilian and european portuguese (72.74 hours), totaling 290 hours and 402k audios. the common voice corpus [1] is an open-use project created by the mozilla foundation. the project is a response to the lack of resources for several languages, including portuguese. in the project, users can simultaneously contribute to the growth of the base and access other people’s audio. to collaborate with the project, users can donate audio in their own voices and review donations from other users. the project has tools for collection, validation and internationalization (adaptation to different languages). the permissive use license of this project allows the exploration of the corpus including for commercial purposes. in version 17, the subcorpus for the portuguese language has 211 h of audio and transcriptions, of which 175 were validated. the nurc-sp audio corpus, the focus of this paper, is described in detail in sect. 3. 3 the nurc-sp corpus nurc-sp was the são paulo division of the nurc (norma urbana linguística culta) project. nurc-sp collected more than 300 h of são paulo speakers throughout the 1970s. the nurc-sp corpus is made up of 375 audio recordings of three genre types: 1. (did) documenter and participant dialogue carried out between a documenter and a participant directly; 2. (d2) participant and participant dialogue between two participants mediated by a documenter; and 3. (ef) participant lectures, seminars, classes, speeches in general, given by a participant in a formal context. which are divided into three subcorpora: the minimum corpus (21 audio recordings), the corpus of non-aligned audios and transcriptions (26 audio recordings), and the audio corpus. nurc-sp had its original analog audios digitized by the alexandre eulalio cultural documentation center (cedae/unicamp) and in december 2020 it was made available to the tarsila projectfootnote 13 as a resource to: (i) build training data sets for spontaneous speech recognition systems, and (ii) facilitate future linguistic studies. the three subcorpora that integrate the nurc-sp digital repository were made available in the portal nurc-sp digital [20] for researchers from fields of linguistics as well for the general public, due to easy access and filtering tools to use the material. here, in this paper, we focus on the nurc-sp audio corpus. it was originally composed of 328 audio recordings without transcriptions, which have been automatically transcribed by whisperx [3]. the revision of automated whisperx transcription segments were performed from june 2023 to december 2023 by 14 native speakers of bp. the revision process was based on an annotation guideline designed to: (i) help making the revision uniform and (ii) remove segments with high amount of noise and overlapping voices. the guideline contains 11 rules, dealing with (i) orality marks, (ii) how to transcribe filled pauses, (iii) repetitive hesitations, (iv) numbers, (v) individual letters, (vi) acronyms, (vii) foreign terms, (viii) punctuation and capitalization, (iv) emotion sounds (ex. laughter) which were annotated in parentheses, (x) misunderstanding of words or passages and (xi) how to deal with automatic segmentation failures. 3.1 pre-processing and filtering of nurc-sp audio corpus for evaluating asr models audio quality was a very important characteristic when preparing the dataset for evaluating asr models. in this section, we present the processing steps of the nurc-sp audio corpus to generate the version used to evaluate the four asr models in this paper: 1. removal of those audios (and all of its segments) with poor/fair quality (with high number of problems with voice distortion, hissing, background noise and interruptions); 2. annotation of the segments using “high/low” quality labels for all remaining audios, based on what was reported by human annotators during the automatic transcription revision; 3. removal of part of the low quality segments, as detailed further in this section, in order to generate the final dataset for training asr models; and 4. generation of both statistics of nurc-sp audio corpus and the train, validation and test sets. of the 328 audios, four of them had poor quality, 92 had fair quality, 52 had good quality and 180 had excellent quality. therefore, of these 328 audios, five of them were not used for training the asr models in this paper: sp_ef_395, sp_did_283, sp_did_194, sp_d2_397, sp_d2_337, as they had problems with voice distortion, hissing, background noise and interruptions, as they were responsible for generating transcriptions with a high rate of errors. the transcriptions were processed to remove the labels used in the automatic transcription revision: (i) “###” was used to indicate segments with high background noise, very low audio, overlapping voices and music; (ii) paralinguistic sounds such as laughter or cough were annotated in parentheses—(laughter), (cough); (iii) misunderstanding of words or passages were also marked with parentheses “( )”; and (iv) words truncated at the end or the beginning of the audio due to automatic segmentation failure were partially transcribed with the help of “>” and “<” (for example: if the word “casa” – house) where truncated at the end of the segment, it was annotated as “ca>” and if it was at the beginning, it was annotated as “<sa”. segments marked with “###” are removed completely, as well as segments having only paralinguistic sounds (e.g., laughter) or containing only truncated words. segments having truncated and non-truncated words (marked with “>” or “<”) had the truncated word removed and were marked as low quality. pairs that have words or passages marked as misunderstood, using the “( )” tags, had the tags removed and the pairs were marked as “low quality”. audio-transcription pairs with paralinguistic sounds tags only had the tags removed and were marked as “high quality”. the dataset available in this work has 323 audio-transcription pairs, which were divided into 177,224 segments transcribed by whisperx, and revised by native bp speakers (see more details in table 2). 3.2 statistics of nurc-sp audio corpus table 2 presents statistics of nurc-sp audio corpus dataset. overall, almost 240 h of speech were generated in the final export, distributed in around 177 thousand segments with an average duration of approximately 5 s. in total, more than 2 million tokens were transcribed, resulting in approximately 12 tokens per segment. it should be noted that the sum of audios with females and males is higher than the total number of audios, because a given audio can have simultaneously both types of voices. table 2. nurc-sp audio corpus statisticsfull size table figure 1 shows the duration of audio segments. it can be noted that audios smaller than 5 are predominant, with a peak in approximately 3 s, while the interval from 5 to 20 s is also well represented. there are also audios longer than 20 s, but their occurrence is less common. figure 2 shows the number of audios by age and gender. the number of audios of females and males are similar, resulting reasonable balancing regarding gender, which can also be seem in male/female ratio in table 2. regarding age, there are more speakers in age group ii (from 36 to 55 years old) than the others groups (i = 25–35 and iii = 36–55). there also a small number of audios with speakers of unknown age. figure 3 shows the number of audios by speech gender (lectures and talks—(ef), interviews (did) and dialogs (d2), respectively). interviews and dialogues are much more common than lectures, totalling 90% of the audios. thus, nurc-sp is a corpus predominantly made of spontaneous speech. fig. 1. audio segments duration full size image fig. 2. (a) audios by speaker’s age range and (b) gender (m/f). speakers are distributed into three age groups (i = 25–35, ii = 36–55, and iii = 56 onwards). full size image fig. 3. audios by speech genre. lectures and talks receive the acronym ef, interviews are called did and dialogues between two participants are named d2. full size image 4 experiments on nurc-sp audio corpus we performed experiments over nurc-sp audio corpus in order to assess the dataset quality and limitations. we based our experiments in two main architectures: wav2vec2 [8] and distil-whisper [11]. wav2vec2 has the advantage of presenting a good performance in low and middle resource languages. wa2vec2 is an auto-supervised pretraining in different languages, which speeds up the training process for asr, allowing robustness against noises that are common in spontaneous speech. distil-whisper [11] is a distilled version of whisper, the state-of-the-art in asr in several languages. we opted for the distilled version because original whisper is costly, being originally trained over thousands hours of labelled speech, while distil-whisper can perform efficiently with less training data. distil-whisper is smaller and faster, while preserving many of original whisper’s strengths. wav2vec2 architecture operates directly on the wave signal, without the need of intermediary representations such as spectrograms. furthermore, there are two main layer types: convolutional layers process the raw wave signal; transformer [23] layers process the result of the previous step. the model can be trained in an auto-supervised manner. the signal is also quantized in the after passing through convolutional layers and masked language modelling is used in the transformer layers. a rich loss takes into account quantized representations and the transformer layer predictions in order to produce diversified quantized representations that represent small windows in the original audio. the learned representations can then be fine-tuned to represent, for example, phonemes in the process of asr. in this case, it is used a classical seq2seq loss for asr generation, namely the ctc loss [13]. whisper proposes an off-the-shelf approach to its architecture, consisting on a traditional transformer, with encoding and decoding blocks, and focusing more on the training data than on architectural adjustments. the input must be converted to log-mel spectrogram format. distil-whisper is the distilled version of whisper. model distillation is a technique that allows for a smaller model (the student) to be trained on both data labels as well as predictions from a bigger model (the professor) in a given prediction task. the main idea behind distillation is that a model’s predictions have richer information about the data distribution than original labels. therefore, a bigger model can capture data distribution and transmit it to the student. both whisper and distil-whisper are available in different versions. comparing whisper large-v3 with distil-whisper large-v3 (used in this work), it can be observed an improvement of 5.8 times in inference time while using 51% fewer parameters in the distilled version. table 3 presents statistics of training, validation and test partitions of each nurc-sp audio corpus subset. 4.1 baseline models development to assess the dataset quality, we trained four new asr models and compared the results with previous related works. two of them are fine-tuned versions of wav2vec2 and the third and fourth models are approaches with distil-whisper. all four models are described below. wav2vec2-nurc-sp-1. this model is a fine-tuned version of wav2vec 2.0 xlsr-53 [2] [9], pre-trained over 53 languages (portuguese included). the pre-trained model was fine-tuned with our train and validation subsets of (nurc-sp audio corpus) nurc-sp-ac in one gpu nvidia dgx a100 80gb for 16 epochs, with early stop of 10. the other settings were the same from [6], training code is available at githubfootnote 14. wav2vec2-nurc-sp-2. the second model is almost the same as the first one, but using as start point the model that [6] trained for coraa-v1 and made publicly available at huggingfacefootnote 15. it also was trained for 16 epochs and finished with a better wer that the first model. distil-whisper-nurc-sp. the third model is a distilled version of whisper large-v3 [19], a model that has support for the portuguese language, trained using our dataset with labels determined by whisper large-v3, with the reason being that more knowledge can be passed from the teacher model to the student model this way [11]. the model was trained with our train and validation subsets of nurc-sp-ac in one gpu nvidia dgx a100 80gb for 48 epochs, following to the steps given by the distil-whisper githubfootnote 16. table 3. statistics of training, validation and test partitions of each nurc-sp audio corpus subset: the total duration in hours and number of audio recordings (left) and number of speakers in each gender (right).full size table distil-whisper-nurc-sp-fine-tuned. this model is a fine-tuning of the third one with our train and validation subsets of nurc-sp-ac, following the steps recommended by the distil-whisper developersfootnote 17. it was also trained with 48 epochs in one gpu nvidia dgx a100 80gb and achieved a better wer and cer (character error rate)footnote 18 than the third model. 4.2 normalization we performed the following steps to normalize the train, validation and test subsets into a standardized form in order to penalize only when a word error is caused by failure in transcribing the word, and not by formatting, punctuation or spontaneous speech differences. the revised transcriptions of nurc-sp audio corpus are of different text genders (ef, d2 and did) and use upper and lower case letters and punctuation, as well as filled pause markers such as eh, hum, ãh, etc. to simplify the training and calculation of the cer and wer metrics, the following normalization was performed: 1. the texts were transformed into lowercase; 2. all punctuation marks generated by whisper were removed (ellipsis, exclamation mark, fullstop, question mark, and comma); 3. the filled pauses were standardized to: eh, uh, ah, as follows: eh = eh, éh, ehm, ehn; uh = uh, hm, uhm, hmm, mm, mhm; ah = ah, huh, ãh, ã; 4. any successive blank spaces have been replaced with one space. 4.3 experiments results for comparison purposes, we kept the tests on the datasets used in previous works and added the new dataset made available by this work. the [14] and [6] models were rerun on all datasets, as were the four new trained models. wer and cer metrics were evaluated for each run. the new dataset proved to be quite challenging for the asr task, as can be seen in table 4. table 4. nurc-sp audio corpus baseline models results compared with previous works. the best values for cer and wer appear in bold.full size table below, we show four examples of result instances from our best model (distil-whisper nurc-sp-fine-tuned)—the first is the original normalized (on) and the second is the model’s prediction normalized (mpn). in the first two, there are problems with named entities as the “martinelli” building was transcribed as “martini” and the name “paulo emilio salles gomes” had a wrong transcription as “paulo e milho fales gomes”. the model used more frequent common names than the proper names “martinelli” and “emilio salles”. rare words related to certain domains (e.g. food) or terminologies of research areas (e.g. linguistics) also bring some difficulties to the model (see third and fourth examples): 1. (on) o martinelli ficou célebre em todo o exterior do estado no interior do estado de são paulo e mesmo pelo brasil afora como um arranha-céu notável para a época; (mpn) o martini ficou célebre em todo o exterior do estado do interior de estado de são paulo e mesmo pelo brasil afora como arranha-céu notável para a época; 2. (on) você me falou em cinema eu lembrei de paulo emilio salles gomes que foi meu colega na faculdade e é um entendidíssimo de cinema né; (mpn) você me falou em cinema eu me lembrei de paulo e milho fales gomes que foi minha colega na faculdade e é um entendidíssimo de cinema né; 3. (on) cuscuz paulista bobó de camarão essas coisas assim; (mpn) cuscos paulista babota de camarão essas coisas; 4. (on) de um lado objeto direto do outro adjunto; (mpn) de um lado é o chefe do e o outro é de junho. 5 discussion and conclusions in this paper, we present a freely available spontaneous speech corpus for the brazilian portuguese language, totaling 239.30 h, and report preliminary asr results, using both the wav2vec2-xlsr-53 and distil-whisper models fine-tuned and trained on our corpus. to the best of our knowledge, distil-whisper has never been evaluated for bp datasets. for comparison purposes, we also bring the tests on the datasets used in previous works. the wav2vec 2.0 based model from [14] remains the best for the common voice dataset with a wer of 16.32%, even in the most recent version (cv version 17.0), as it is trained in prepared/read speech. in the coraa v1 dataset, [6]’s models, which is also trained an wav2vec 2.0 model, but mostly based on spontaneous speech, continues presenting the best values for this dataset with 24.18% wer, but our wav2vec2-nurc-sp-2 model was in second position (since it is also trained for spontaneous speech). for the new dataset, focus of this paper, the best results were the distil-whisper fine-tuned over nurc-sp audio corpus with a wer of 24.22% followed by a fine-tuned versions of wav2vec2-xlsr-53 model with a wer of 33.73%, that is almost 10% point worse than distil-whisper’s. these results indicates that distil-whisper is promising for low and medium resource languages and should be evaluated with more bp datasets in the future. notes 1.https://www.ted.com/talks. 2.for example, [5] comments on vowel elision between words that, in the são paulo dialect, affects the final posttonic vowels /a/, /o/ and /u/. for instance, in the example (a) me’ren[da es]co’lar (school lunch) –> me’ren[des]co’lar, the vowel /a/ is deleted and a new syllable is created ([des]). 3.https://openslr.org/150/. 4.https://groups.inf.ed.ac.uk/ami/corpus/overview.shtml. 5.https://commonvoice.mozilla.org/en/datasets. 6.https://www.openslr.org/51/. 7.https://github.com/nilc-nlp/coraa. 8.github.com/nilc-nlp/nurc-sp-audio-corpus. 9.https://librivox.org/pages/about-librivox/. 10.https://www.openslr.org/12. 11.https://www.openslr.org/60/. 12.https://arxiv.org/abs/2209.11871. 13.https://sites.google.com/view/tarsila-c4ai/home. 14.https://github.com/edresson/wav2vec-wrapper. 15.https://huggingface.co/edresson/wav2vec2-large-xlsr-coraa-portuguese. 16.https://github.com/huggingface/distil-whisper/tree/main/training. 17.https://huggingface.co/blog/fine-tune-whisper. 18.here, we also focus our analysis on the metric cer, because for smaller audios, with just a few words, this metric tends to be more reliable. references ardila, r., et al.: common voice: a massively-multilingual speech corpus. in: calzolari, n., et al. (eds.) proceedings of the twelfth language resources and evaluation conference, may 2020, marseille, france, pp. 4218–4222. european language resources association (2020). https://aclanthology.org/2020.lrec-1.520 baevski, a., zhou, h., mohamed, a., auli, m.: wav2vec 2.0: a framework for self-supervised learning of speech representations. adv. neural inf. process. syst. 33, 12449–12460 (2020). https://arxiv.org/abs/2006.11477 bain, m., huh, j., han, t., zisserman, a.: whisperx: time-accurate speech transcription of long-form audio. in: interspeech 2023, pp. 4489–4493 (2023). https://doi.org/10.21437/interspeech.2023-78 beckman, m.e.: a typology of spontaneous speech. in: computing prosody: computational models for processing spontaneous speech, pp. 7–26. springer, new york (1997). https://doi.org/10.1007/978-1-4612-2258-3_2 bohn, g.p.: processos e representações lexicais: o caso das vogais posteriores do dialeto paulista. delta: documentação e estudos em linguística teórica e aplicada 33(2), september 2017. https://revistas.pucsp.br/index.php/delta/article/view/34370 candido_junior, a., et al.: coraa asr: a large corpus of spontaneous and prepared speech manually validated for speech recognition in brazilian portuguese. lang. resour. eval. 57, 1139–1171 (2023). https://doi.org/10.1007/s10579-022-09621-4. https://link.springer.com/article/10.1007/s10579-022-09621-4 clifton, a., et al.: 100,000 podcasts: a spoken english document corpus. in: scott, d., bel, n., zong, c. (eds.) proceedings of the 28th international conference on computational linguistics, barcelona, spain (online), december 2020, pp. 5903–5917. international committee on computational linguistics (2020). https://aclanthology.org/2020.coling-main.519 conneau, a., baevski, a., collobert, r., mohamed, a., auli, m.: unsupervised cross-lingual representation learning for speech recognition. in: proceedings of the interspeech 2021, pp. 2426–2430 (2021). https://doi.org/10.21437/interspeech.2021-329 conneau, a., et al.: unsupervised cross-lingual representation learning at scale. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 8440–8451 (2020) google scholar  gabler, p., geiger, b.c., schuppler, b., kern, r.: reconsidering read and spontaneous speech: causal perspectives on the generation of training data for automatic speech recognition. information 14(2) (2023) google scholar  gandhi, s., von platen, p., rush, a.m.: distil-whisper: robust knowledge distillation via large-scale pseudo labelling (2023) google scholar  garmash, e., et al.: cem mil podcasts: a spoken portuguese document corpus for multi-modal, multi-lingual and multi-dialect information access research. in: arampatzis, a., et al. (eds.) experimental ir meets multilinguality. multimodality, and interaction: 14th international conference of the clef association, clef 2023, thessaloniki, greece, 18–21 september 2023, proceedings, pp. 48–59. springer, heidelberg (2023). https://doi.org/10.1007/978-3-031-42448-9_5 graves, a., fernández, s., gomez, f., schmidhuber, j.: connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. in: proceedings of the 23rd international conference on machine learning, pp. 369–376 (2006) google scholar  gris, l.r.s., casanova, e., de oliveira, f.s., da silva soares, a., candido junior, a.: brazilian portuguese speech recognition using wav2vec 2.0. in: pinheiro, v., et al. (eds.) computational processing of the portuguese language, pp. 333–343. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_31 gris, l.r.s., marcacini, r., junior, a.c., casanova, e., soares, a., aluísio, s.m.: evaluating openai’s whisper asr for punctuation prediction and topic modeling of life histories of the museum of the person (2023) google scholar  liu, y., shriberg, e., stolcke, a., hillard, d., ostendorf, m., harper, m.: enriching speech recognition with automatic detection of sentence boundaries and disfluencies. ieee trans. audio speech lang. process., 1526–1540 (2006) google scholar  panayotov, v., chen, g., povey, d., khudanpur, s.: librispeech: an asr corpus based on public domain audio books. in: 2015 ieee international conference on acoustics, speech and signal processing (icassp), pp. 5206–5210 (2015) google scholar  pratap, v., xu, q., sriram, a., synnaeve, g., collobert, r.: mls: a large-scale multilingual dataset for speech research. in: proceedings of the interspeech 2020, pp. 2757–2761 (2020). https://doi.org/10.21437/interspeech.2020-2826 radford, a., kim, j.w., xu, t., brockman, g., mcleavey, c., sutskever, i.: robust speech recognition via large-scale weak supervision. in: krause, a., brunskill, e., cho, k., engelhardt, b., sabato, s., scarlett, j. (eds.) proceedings of the 40th international conference on machine learning, proceedings of machine learning research, 23–29 july 2023, vol. 202, pp. 28492–28518. pmlr (2023) google scholar  rodrigues, a.c., et al: portal nurc-sp: design, development, and speech processing corpora resources to support the public dissemination of portuguese spoken language. in: gamallo, p., et al (eds.) proceedings of the 16th international conference on computational processing of portuguese, pp. 187–195. association for computational lingustics (2024) google scholar  salesky, e., et al.: the multilingual tedx corpus for speech recognition and translation. in: proceedings of the interspeech 2021, pp. 3655–3659 (2021) google scholar  éva székely, henter, g.e., beskow, j., gustafson, j.: spontaneous conversational speech synthesis from found data. in: proceedings of the interspeech 2019, pp. 4435–4439 (2019). https://doi.org/10.21437/interspeech.2019-2836 vaswani, a., et al.: attention is all you need. in: advances in neural information p0rocessing systems, vol. 30 (2017) google scholar  zen, h., et al.: libritts: a corpus derived from librispeech for text-to-speech (2019) google scholar  download references acknowledgements this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp grant #2019/07665-4) and by the ibm corporation. this project was also supported by the ministry of science, technology and innovation, with resources of law no. 8.248, of october 23, 1991, within the scope of ppi-softex, coordinated by softex and published residence in tic 13, dou 01245.010222/2022-44. author information authors and affiliations university of são paulo, são carlos, sp, 13566-590, brazil rodrigo lima, sidney e. leal & sandra m. aluísio universidade estadual paulista, são josé do rio preto, sp, 15054-000, brazil arnaldo candido junior authors rodrigo limaview author publications search author on:pubmed google scholar sidney e. lealview author publications search author on:pubmed google scholar arnaldo candido juniorview author publications search author on:pubmed google scholar sandra m. aluísioview author publications search author on:pubmed google scholar corresponding author correspondence to sidney e. leal . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper lima, r., leal, s.e., junior, a.c., aluísio, s.m. (2025). a large dataset of spontaneous speech with the accent spoken in são paulo for automatic speech recognition evaluation. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_3 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automatic speech recognition evaluation spontaneous speech brazilian portuguese public speech corpora publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature one-class learning for data stream through graph neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper one-class learning for data stream through graph neural networks conference paper first online: 31 january 2025 pp 61–75 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) one-class learning for data stream through graph neural networks download book pdf download book epub marcos paulo silva gôlo  orcid: orcid.org/0000-0002-9093-81959, joão gama  orcid: orcid.org/0000-0003-3357-119510 & ricardo marcondes marcacini  orcid: orcid.org/0000-0002-2309-34879  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 262 accesses abstract in many data stream applications, there is a normal concept, and the objective is to identify normal and abnormal concepts by training only with normal concept instances. this scenario is known in the literature as one-class learning (ocl) for data streams. in this ocl scenario for data streams, we highlight two main gaps: (i) lack of methods based on graph neural networks (gnns) and (ii) lack of interpretable methods. we introduce opencast (one-class graph autoencoder for data stream), a new method for data streams based on ocl and gnns. our method learns representations while encapsulating the instances of interest through a hypersphere. opencast learns low-dimensional representations to generate interpretability in the representation learning process. opencast achieved state-of-the-art results for data streams in the ocl scenario, outperforming seven other methods. furthermore, opencast learns low-dimensional representations, generating interpretability in the representation learning process and results. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others online learning from capricious data streams via shared and new feature spaces article 16 july 2024 oec: an online ensemble classifier for mining data streams with noisy labels article 12 december 2023 adaptive feature generation for online continual learning from imbalanced data chapter © 2022 1 introduction in many real-world applications, data arrives continuously as a data stream [8]. real-world applications of data streams include the areas of health [24], fake news detection [25], and energy consumption [21]. in some of these applications, there is a known class or normal concept (positive class), and the objective is to identify between the known and unknown class by training only with instances of the known class [9]. usually, this problem is solved through one-class learning (ocl)footnote 1 [9, 25]. ocl is already commonly used in the data stream field in novelty detection, anomaly detection, and class of interest detection applications [9, 23]. studies explore from the most classic ocl algorithms [14] to some more sophisticated ones based on clustering [18] to solve data stream tasks. the use of clustering strategies comes from the unsupervised characteristic of generating clusters to improve the challenging ocl task of training with only one class and then classifying two. we highlight two main gaps in the area of ocl for data stream. first, there is a lack of methods based on graph neural networks (gnns). gnns arrive with the same principle as cluster-based methods, i.e., exploring the unsupervised part of the graph (relations) through graph autoencoders [7]. gnns have already been used in data streams [5] and ocl [30], but to the best of our knowledge, not for both areas. the other gap is the lack of interpretable ocl methods for data streams. interpretability is gaining more attention in data streams and other areas [12], mainly in interpretability related to the representation learning process [11]. one of the ways to provide this interpretability is to focus on the learned latent space, i.e., to produce low-dimensional representations inherently [22]. it is important to emphasize that we are not writing about methods that generate low dimensions after learning, such as pca and t-sne, but rather methods that generate latent representations that can be visualized at each learning epoch, i.e., enabling an interpretation of the model’s learning in real-time during training. we propose opencast (one-class graph autoencoder for data stream), a new method for data stream based on ocl and gnns. opencast is based on a gnn layer called faconv, created for heterophilic graphs [3]. we chose this layer due to the natural concept of drift in data stream problems (concept of changing class characteristics), which can generate abnormal relations between entities of interest and non-interest class (concept of heterophile). opencast is based on the state-of-the-art (sota) hypersphere loss function for gnns and ocl [11]. furthermore, to explore the concept of non-supervision to support the ocl task, opencast uses a reconstruction loss function of graph autoencoders [32]. we introduce a new strategy in the field of gnns and ocl, which is to weigh the loss functions through a parameter to deal with the concept of drift. finally, we explicitly learn low-dimensional representations to introduce interpretability into opencast naturally. in summary, our contributions are: 1. we propose opencast, a novel method for ocl in data streams; 2. we show that learning low dimensional representations by opencast generates interpretability for ocl in data streams without losing performance; 3. opencast obtains sota results for data streams against ocl methods. 2 problem formulation and background we assume that the data stream problem is divided into chunks (sliding window strategy [8]). \(\mathcal{d}\mathcal{s} = \{\mathcal{d}\mathcal{s}_0,\mathcal{d}\mathcal{s}_1,...,\mathcal{d}\mathcal{s}_m\}\) consists of the set of \(m+1\) chunks \(\mathcal{d}\mathcal{s}_i\) of the dataset, where \(\{(x_0, y_0), (x_1, y_1), ..., (x_n, y_n) \in \mathcal{d}\mathcal{s}_i\) are the \(n+1\) instances of chunk i and \(n+1\) labels of each instance, and \(\boldsymbol{x}_{j} \in \mathbb {r}^d\) is the feature vector of the j-th instance with d dimensions. the goal of ocl algorithms for data stream is to learn a function \(f: \mathcal{d}\mathcal{s}_i \rightarrow \mathcal {y}\) in a way that \(f(\boldsymbol{x})\) predicts a label \(y \in \{+1, -1\}\). it is worth mentioning that the algorithms only train with information from a single label (\(+1\)) in each iteration, i.e., the algorithm learns the function f(.) with instances \(\boldsymbol{x_j}\), where \(y_{j} = +1\). in this work, for training, there may be instances of interest from more than one chunk before the one evaluated. figure 1 illustrates this data flow process for ocl. fig. 1. illustration of the classification process in one-class learning. (color figure online) full size image a graph is formally defined as \(\mathcal {g} = (\mathcal {v}, \boldsymbol{a})\), where each node \(v_i \in \mathcal {v}\) and \(\boldsymbol{a}\) is the adjacency matrix containing relations information between nodes. for each window (iterations in fig. 1), we have a \(\mathcal {g}_i\) with two sets of nodes: \(\mathcal {v}^{in}\) (set of interest with a single label, i.e., green balls in fig. 1) and \(\mathcal {v}^{un}\) (unlabeled set, i.e., symbols without color in fig. 1). we generate each \(\mathcal {g}_i\) through proximity between the instances. this type of graph is called k nearest neighbors graph, in which each node connects to its k nearest neighbors. with the generated graph, we can apply gnns to learn representations for the nodes and classify the nodes. the gnns consider the structured representation of each node \(\boldsymbol{v}_i \in \boldsymbol{v}\) and the adjacency matrix \(\boldsymbol{a}\) as input for the representation learning process. therefore, \(g(\boldsymbol{v}, \boldsymbol{a}; \textbf{w})\) represents a gnn with trainable weights \(\textbf{w} = \{\boldsymbol{w}^{(1)}, \cdots , \boldsymbol{w}^{(l)}\}\) in l hidden layers. formally, for the l-th layer, the gnn propagation can be summarized as follows: $$\begin{aligned} \boldsymbol{h}^{(l+1)} = g(\boldsymbol{h}^{(l)}, \boldsymbol{a}; \boldsymbol{w}^{(l)}), \end{aligned}$$ (1) in which \(\boldsymbol{h}^{(l)}\) is the input to the l-th gnn layer, and \(\boldsymbol{h}^{l+1}\) is the output of this layer. the representations \(\boldsymbol{v}\) are the inputs for the first layer, i.e., \(\boldsymbol{v} \equiv \boldsymbol{h}^{(0)}\). in this sense, \(\boldsymbol{h}^{(l)}\) are the learned embeddings for each node. a gnn has two main steps during the learning. first, the aggregate step tries to aggregate the information from the neighbors of each node. the second is the combining step, which tries to update the node representations by combining the aggregated information from neighbors with the current node representations [27]. after these steps, the aggregated and combined representations pass through neurons to learn the weights \(\textbf{w}\). the steps are defined by [27, 31]: in which \(\boldsymbol{a}_{v_i}^l\) is the aggregation result of the \(v_i\) neighbors defined as \(n_{v_i}\), and \(\boldsymbol{h}_{v_j}^{l-1}\) is the feature vector of the node \(v_j\) at the \(l-1^{th}\) layer. the combination of the aggregate representation and the node representation is defined by eq. 3 [27, 31]. for instance, the mean operator can perform both steps. the studies of ocl and gnns use graph autoencoders (gae) to improve the ocl, given the advantages of two gaes of learning representations in an unsupervised way by reconstructing the graph [7, 10, 11]. a gae uses gnn layers as an encoder and an inner product of the latent representation as a decoder to learn node representations. equation 4 describes a gae [15]: $$\begin{aligned} gae = \left\{ \begin{matrix} encoder: \boldsymbol{h}^{(l)} = g(\textbf{v}, \boldsymbol{a}; \textbf{w}) \,\,\,\,\ \\ decoder: \boldsymbol{\hat{a}} = \sigma (\boldsymbol{h}^{(l)} \cdot \boldsymbol{h}^{(l)^\intercal }) \end{matrix} \right. , \end{aligned}$$ (4) in which \(\sigma (.)\) is a logistic sigmoid function. gae learns \(\boldsymbol{h}^{(l)}\) by performing the binary cross-entropy loss considering true labels (\(\boldsymbol{a}\)) and predicted labels (\(\boldsymbol{\hat{a}}\)), i.e., the gae uses the graph relations as the first class and non-relations as another class, performing a binary classification with the learned representations. 3 related work one-class learning for data streams. one-class learning (ocl) is currently a relevant strategy explored in the data stream field [9]. works explore ocl for novelty, anomaly, or interest class detection in the data stream [9, 26, 28]. some works explore classic ocl algorithms or their variations to solve data stream-based tasks. in this sense, [14] proposed two algorithms: cma and fma. both algorithms are based on one-class support vector machines (ocsvm) and their retraining during the stream of new data. cma retrains ocsvm when each new instance emerges. fma retrains the ocsvm with a heuristic formulated based on the values of new instances classified as interest or novelty. [28] proposes an isolation forest (iforest) variant that retrains the model when the anomaly rate increases. [20] proposes a local outlier forest (lof) variant with a new density-based sampling algorithm and distance approximation technique. given the ocl challenge of training with one class to classify two, generally, the algorithms use unsupervised techniques to improve classification through ocl, such as clustering [9] or graph autoencoders [7, 10] techniques. custer-based one-class learning for data stream. pioneering [26] or more recent [6, 18] methods explore a standard strategy to apply clustering methods of ocl and data stream scenarios. generally, the methods apply a clustering algorithm to the training data, i.e., to the interest class data (offline stage ), and, in the online scenario, they apply this algorithm or more clustering algorithms on the new stream data to detect novelty or the interest class. normally, what differentiates the methods is the strategy of the online stage. in this sense, [26] proposed olindda, [6] proposed minas, and [18] proposed cdd. all are clustering algorithms for detecting novelties in data streams. olindda and cdd are ocl algorithms, and minas is a multi-class algorithm that works in the ocl scenario. as in classical algorithms, in those based on clustering, there is a gap in interpretability. however, we can explore another relevant data-streaming strategy based on graph neural networks to obtain interpretability [13]. graph neural networks for data stream. graphs explicit relations between entities and these relationships can bring relevant information to the data stream field because they add information about the new data, which is better for the model [4, 17]. still, gnns have been explored and have state-of-the-art results in different domains [5]. thus, studies have explored gnns to improve the classification performance in data stream [13]. gnn works focus on proposing new adaptive gnn layers for data stream [17], specific methods for anomaly detection or novelty detection [4], and focusing on binary or multi-class classification. in the ocl, the use of gnns focuses on unsupervised techniques such as graph autoencoders [7, 10] or end-to-end techniques with loss functions specialized for ocl [11]. even though graphs bring relevant information about relations to data and obtain state-of-the-art results for data streams or for one-class learning, to our knowledge, there are no methods based on gnns and ocl for data streams. in this sense, in the next section, we present an ocl method based on gnns for the data stream. 4 opencast: one-class graph autoencoder for data stream we propose a novel method called one-class graph autoencoder for data stream (opencast). our method for data streams is based on ocl and gnns. we explore a state-of-the-art loss function to encapsulate interest instances closer to the center within the hypersphere. opencast learns a new low-dimensional latent space, where interest instances are positioned inside a hypersphere. we learn the new space through a gae to explore the reconstruction loss. thus, we introduce a final loss function that combines the hypersphere loss function with the reconstruction loss function, considering a weight to control the impact of each loss function in the learning since we have the concept drift of the interest class in the data stream scenario. the first step of our method is modeling the data stream through graphs. in this modeling, heterophilic graphs can be generated, given the concept drift present in the data stream, i.e., change in the class of interest or class of non-interest [33]. heterophilic graphs are graphs in which many nodes from different classes are connected. with the concept drift of the interest class, nodes of the interest class can connect with nodes of non-interest [16]. when using traditional gnn layers, as well as in works of one-class graph neural networks [7, 11, 30], information from all connected nodes are aggregated, i.e., nodes of non-interest can pass representation information to a node of interest, which is harmful to ocl. thus, our first innovation is based on our method of gnn layers for heterophilic graphs. we chose the faconv layer because it can learn weights for the edges and minimize the problem of aggregating information from nodes of different classes [3]. faconv uses an aggregation and combination function presented in eq. 5 [3]. in which, \(\boldsymbol{h}_{v_i}^0\) is the initial \(v_i\) representation, \(\epsilon \) is a factor to keep the generated representation close to the initial representation, \(d_i\) is the node degree of node i, and the attention coefficients \(\alpha _{i,j}\) are computed as presented in the eq. 6. the state-of-the-art for ocl through gnns is formed by gnns that exploit loss functions based on hyperspheres [11, 30]. these methods learn representations for the nodes while encapsulating the representations of the nodes of interest. we also explore this strategy for data stream [11, 30]. furthermore, we combine the hypersphere loss function with the loss function of gaes [11], which obtained superior and state-of-the-art results compared to other methods [30]. we use the hypersphere loss function \(\mathcal {l}_{1}\) defined in eq. 7 [11] and the gae loss function \(\mathcal {l}_{2}\) defined in eq. 9 (binary cross entropy loss applied in the adjacency matrix). equation 8 represents the value indicating whether the interest instance \(v_i\) is within the hypersphere with radius r and center \(\boldsymbol{c}\). $$\begin{aligned} \mathcal {l}_{2}(\boldsymbol{{w}}) = \frac{1}{|v|} \sum _{i=0}^{|v|} \sum _{j=0}^{|v|} (\boldsymbol{a}_{ij} \cdot log \hat{\boldsymbol{a}}_{ij} + (1 \boldsymbol{a}_{ij}) \cdot log(1 \hat{\boldsymbol{a}}_{ij})). \end{aligned}$$ (9) some works combine the hypersphere and reconstruction loss functions, only adding both loss functions [11, 32]. however, considering the concept drift of the interest class can be interesting, adding weights to the losses to bias the learning. thus, we innovate by adding impact factors to each loss function because of the concept drift during the data stream. this concept drift may require more of the hypersphere loss function that aims to encapsulate the instances of interest or more of the reconstruction of the graph that acts as a constraint for the hypersphere loss function. therefore, our final loss function is defined by: $$\begin{aligned} \mathcal {l} = \mathcal {l}_{1} * \beta + \mathcal {l}_{2} * (1-\beta ) \end{aligned}$$ (10) we present an illustration of opencast in fig. 2. we show examples of graph modeling, the faconv as a gnn layer that allows us to focus on relations between nodes of the same class and use the hypersphere loss function to understand the model’s decision. even though it is possible to understand the model’s decision by observing a circle or sphere and instances inside the circle/sphere (class of interest) and outside the circle/sphere (class of non-interest), it is challenging to interpret the learning that generated this decision, since we cannot visualize the representations generated during learning if the dimension is greater than 3. on the other hand, with representations in two or three dimensions, we can observe and interpret the representations generated during learning. in this sense, we bias opencast learning so that our method learns representations in size two to provide interpretability for representation learning in the scenario of ocl and gnns for data streams. fig. 2. illustration of the opencast method. full size image 5 experimental evaluation this section presents the experimental evaluation of this article. we present the used datasets, experimental settings, results, and discussion. our goal is to demonstrate that our opencast proposal outperforms other state-of-the-art methods for data streams. another goal is to demonstrate that our method learns low-dimensional representations, providing interpretability for the data stream scenario. the experimental evaluation codes are publicly availablefootnote 2. 5.1 datasets we selected four publicly available benchmark datasets from [29], commonly used in the data stream literature, for our ocl scenario, each large enough to be processed as a stream and with two classes: the interest and outlier classes. we choose two synthetic and two real datasets. agrawalfootnote 3 and tweet500footnote 4 are synthetic, and bank marketingfootnote 5 and electricityfootnote 6 are the real datasets. agrawal is a stream generator that uses ten functions to generate binary class labels from nine features [1]. tweet500 is generated by moa [2] and simulates sentiment analysis on tweets (positive or negative feelings). bank marketing is related to direct marketing campaigns to assess if the product (bank term deposit) would be (’yes’) or not (’no’) subscribed [19]. electricity was collected from the australian new south wales electricity market. the class label identifies the price change (up or down) in new south wales relative to a moving average of the last 24 h. table 1 summarises the datasets. for categorical attributes, we use a one-hot-encoding strategy to transform the attributes into numeric ones. table 1. datasets details. we present the number of instances, features, interest instances, non-interest instances and the dataset type.full size table 5.2 experimental settings one part of the experimental settings defines the data ordering, partitions, and distribution [29]. we follow the data stream protocol from [8] and select the sliding window strategy. our window is 2500 in length. it is essential to point out that we use only the interest instances of each window before the current window to train the algorithms and evaluate them on all instances of the current window. we kept the most current instances of interest from the previous window until there were 2500 instances of interest for training the algorithm (see fig. 1). finally, we use the \(f_1\)-macro to compare all models, as the \(f_1\)-micro and accuracy are biased by test set imbalance, which is natural in ocl. we compare the opencast with cma and fma from [14], ocsvm, iforest, and lof to cover the classical and adaptation of ocl algorithms for data streamsfootnote 7. furthermore, considering the one-class cluster algorithms of the data stream from the literature, we compare our method with cdd [18]. we also adapted the lof algorithm to compare opencast with another method with graph topology information since lof has a parameter to receive neighbor information. we use the following parameters for the methods: ocsvm, cma, and fma: kernel = {rbf, poly, sigmoid, linear}, \(\nu = \{0.05 * b, 0.005 * b\}, b \in [1..19]\), and \(\gamma = \{\) scale, auto \(\}\); fma: \(h_t = \{50, 100, 1000, 2000\}\), \(\alpha = \{0.1 * b\}, b \in [1..9]\); iforest: n\(^\circ \) of estimators = \(\{1,2,5,10,50,100,200,500\}\), maximum samples and maximum features = \(\{0.1 * b\}, b \in [1..10]\); lof: n\(^\circ \) of neighbors = \(\{1,2,5,10,50,100\}\); graphlof: n\(^\circ \) of neighbors = \(\{1,2,3,4,5,6,7,8,9,10,15,20\}\) and k for graph modeling = n\(^\circ \) of neighbors + 1; cdd: n\(^\circ \) of clusters = \(\{2, 5, 10, 50, 100\}\) and algorithms k-means and gmm; opencast: k for graph modeling = \(\{1,2,3,4,5\}\), n\(^\circ \) of layers = \(\{1,2\}\), \(r = 0.3\), c \(= [0,0]\), epochs = \(\{2000, 3000\}\), \(\beta \) and \(\epsilon = \{0.1 * b\}, b \in [1..9]\), learning rate = 0.001. 5.3 results and discussion table 2 shows our results. each table row shows the results of a method on the four datasets. in bold are the highest \(f_1\) macro averages. underlined are the second-highest macro \(f_1\) averages. in parentheses are the standard deviations for each result. in the four datasets, opencast obtained the highest macro \(f_1\) averages. in three of the datasets, ocsvm obtained the second-best result. in the electricity dataset, iforest obtained the second-best result. for real datasets, opencast improved to the second or third decimal place. on the other hand, in the synthetic datasets, opencast improved 23 % and 31 % in the average \(f_1\) macro. the worst results were obtained by graphlof, followed by lof and cdd. cma, fma, and iforest were competitive with ocsvm. table 2. average macro \(f_1\) for each method in the four datasets. the best results are in bold, and the second best are underlined.full size table the worst result of opencast was in the dataset bank. this is the most unbalanced dataset (see table 1). therefore, the lack of interest and many examples of non-interest may have been the factor for our low performance. it is worth mentioning that the other methods also suffered from this imbalance. given the low performance of all methods in this dataset, we believe that the initial representations also harm the learning, i.e., it is necessary to pre-process to normalize, standardize, and even perform feature selection to improve the ocl performance in these data stream datasets. we obtained satisfactory sota results for data streams through ocl in the other datasets. an interesting point is the good results on different types of datasets (real and synthetic), with varying sizes of features (500, 40, and 6) and number of instances (100k and 45k). figures 3, 4, 5, and 6 present the \(f_1\) macro in each iteration for each dataset. in figs. 5 and 6, opencast always obtains better performance stably. we observed other stable methods but with worse performance or unstable and with worse performance. in fig. 3, opencast wins in some iterations and loses in others. for this dataset, most methods were stable except cdd. in fig. 4, we observe the highest level of instability in the methods with the best averages (opencast and iforest). opencast wins in some iterations and loses in others, as in the dataset bank. opencast is a method based on neural networks and needs a learning rate. one of the possibilities for improving the method’s performance and generating stability is exploring a lower learning rate and more training epochs. fig. 3. \(f_1\) macro of each iteration of each method in dataset bank. full size image fig. 4. \(f_1\) macro of each iteration of each method in dataset electricity. full size image fig. 5. \(f_1\) macro of each iteration of each method in dataset agrawal. full size image fig. 6. \(f_1\) macro of each iteration of each method in dataset twitter. full size image we present the representations generated by the opencast learning to demonstrate the interpretability of our method. in this sense, fig. 7 presents the opencast representations in the experimental evaluation of the twitter dataset. our interpretability focuses on the learning process. thus, we present opencast’s learning process. in the real world, we can show the video of the learning process since we can plot all learning epochs without processing the representation (our learned representations have two dimensions). the epochs are 700, 725, 750, 775, 800, 900. 1000, and 2000. yellow points represent the interest class, and blue points represent the non-interest class. fig. 7. graph plot considering the two-dimensional last layer learned representations of opencast in the twitter dataset at one iteration. the colors indicate the interest (yellow) and the non-interest (blue) classes. (color figure online) full size image we observe the learning process proposal through opencast interpretability, as shown in fig. 7. in the initial steps, we observe opencast focusing on graph reconstruction (loss \(\mathcal {l}_2\)), the first opencast goal. in the next steps, we observe learning through the two losses since the interest instances get closer to the sphere while non-interest instances are outside the sphere. in the final steps, we observed that the one-class loss \(\mathcal {l}_1\) encouraged the instances to continue coming to the sphere center, as proposed to improve the performance. 5.4 ablation study we propose an ablation study to verify the impact of opencast learning strategies compared to other strategies in the ocl literature for graphs [11, 32]. we compared, in the best opencast performance scenario, the use of a graph convolutional network (gcn) instead of faconv and the addition of \(\mathcal {l}_1\) and \(\mathcal {l}_2\) instead of using our parameter \(\beta \). table 3 presents the average macro \(f_1\) in the four datasets for each modification on opencast, i.e., the opencast with: (i) gcn and adding \(\mathcal {l}_1\) and \(\mathcal { l}_2\); (ii) gcn and our final loss (with \(\beta \)); (iii) faconv and adding \(\mathcal {l}_1\) and \(\mathcal {l}_2\); and (iv) faconv and our final loss (with \(\beta \)). table 3. ablation study comparing the losses and layers for opencast.full size table faconv improves learning in relation to gcnconv as motivated in the opencast section due to concept drift. using the \(\beta \) parameter in real datasets improves the classification performance. on the other hand, in the synthetic datasets, the performance was equal, considering both the sum of the loss functions and using the \(\beta \) parameter to weigh the losses. we analyzed the best opencast parameters in each dataset to find meaning in this ablation study. in the real datasets, the best \(\beta \) values were 0.6 (electricity) and 0.4 (electricity), i.e., there was greater importance for each loss function during learning. for the synthetic datasets, the \(\beta \) value that resulted in the best learning was 0.5, i.e., the same importance for the two loss functions as in the sum strategy. this shows us why the \(\beta \) parameter has less impact on synthetic datasets. still, it is a good strategy since it wins in real datasets and does not lose in the synthetics. we complement our ablation study by presenting the interpretability results for each learning strategy described in the paragraph above. figure 8 presents the graph and its connections considering the final representation learned by each model in the twitter dataset as coordinates. in addition to higher \(f_1\), faconv generated better representations for the hypersphere-based decision strategy. opencast with faconv learned representations for instances of interest closer to the center and within the hypersphere than gcn. fig. 8. graph plot considering the two-dimensional last layer learned representations of opencast in the ablation study considering the twitter dataset. the colors indicate the interest (yellow) and the non-interest (blue) classes. (color figure online) full size image 6 conclusions and future work in this work, we propose opencast, a gnn based on ocl for stream data. opencast is based on faconv due to the concept of drift since this layer gives importance to edges of nodes from the same classes and ignores relations between nodes from other classes. our opencast is also based on the sota hypersphere loss function for ocl and the reconstruction function of gaes. we introduce a new way of combining these loss functions through a \(\beta \) parameter that gives weight to each loss function. opencast learning allows for exploring low-dimensional representations during classification, providing interpretability and visualization for the ocl task on a graph-structured data stream. opencast outperformed other methods, obtaining sota results and generating representations visually suitable for ocl in each epoch, which brought interpretability to the data stream scenario through ocl. the main limitation of opencast is that it does not take advantage of the neuron’s weights from the previous iteration in the current iteration. this limitation can be observed in the instability shown in fig. 4. maintaining old instances mitigates this, but it is not enough if there is a dependency on the previous iteration. therefore, in future work, we intend to explore a version of opencast that uses the neural network weights in the previous iteration to initialize the neural network weights in the current iteration. furthermore, we intend to explore our method in heterophilic scenarios other than data streams. notes 1.ocl is also called one-class classification in the literature, but this work chose to use the term ocl. 2.https://github.com/golomarcos/opencast. 3.https://riverml.xyz/0.14.0/api/datasets/synth/agrawal/. 4.https://github.com/marouabahri/cs-arf/tree/master/datasets/. 5.https://archive.ics.uci.edu/dataset/222/bank+marketing. 6.https://github.com/amidst/code-examples/blob/master/datasets/driftsets/. 7.other methods based on ocsvm, iforest and lof (methods presented in the related work section) do not share the source code or the link to the source code is broken or the code is not in python which was the language used in this work. references agrawal, r., imielinski, t., swami, a.: database mining: a performance perspective. ieee trans. knowl. data eng. 5(6), 914–925 (1993) article  math  google scholar  bifet, a., et al.: moa: massive online analysis, a framework for stream classification and clustering. in: proceedings of the first workshop on applications of pattern analysis. pmlr (2010) google scholar  bo, d., wang, x., shi, c., shen, h.: beyond low-frequency information in graph convolutional networks. in: proceedings of the conference of aaai (2021) google scholar  cai, l., et al.: structural temporal graph neural networks for anomaly detection in dynamic graphs. in: proceedings of the international conference on information and knowledge management (2021) google scholar  corso, g., stark, h., jegelka, s., jaakkola, t., barzilay, r.: graph neural networks. nat. rev. methods primers 4(1), 17 (2024) article  math  google scholar  de faria, e.r., ponce de leon ferreira carvalho, a.c., gama, j.: minas: multiclass learning algorithm for novelty detection in data streams. data mining and knowledge discovery 30, 640–680 (2016) google scholar  feng, y., chen, j., liu, z., lv, h., wang, j.: full graph autoencoder for one-class group anomaly detection of iiot system. internet things j. 9(21) (2022) google scholar  gama, j., sebastiao, r., rodrigues, p.p.: on evaluating stream learning algorithms. mach. learn. 90, 317–346 (2013) article  mathscinet  math  google scholar  gaudreault, j.g., branco, p.: a systematic literature review of novelty detection in data streams: challenges and opportunities. acm comput. surv. (2024) google scholar  gôlo, m.p.s., de moraes, m.i., goularte, r., marcacini, r.m.: on the use of early fusion operators on heterogeneous graph neural networks for one-class learning. in: proceedings of the brazilian symposium on multimedia and the web, pp. 128–136 (2023) google scholar  gôlo, m.p.s., junior, j.g.b.m., silva, d.f., marcacini, r.m.: olga: one-class graph autoencoder. arxiv preprint arxiv:2406.09131 (2024) haug, j., broelemann, k., kasneci, g.: dynamic model tree for interpretable data stream learning. in: international conference on data engineering. ieee (2022) google scholar  jin, g., et al.: spatio-temporal graph neural networks for predictive learning in urban computing: a survey. ieee trans. knowl. data eng. (2023) google scholar  jodelka, o., anagnostopoulos, c., kolomvatsos, k.: adaptive novelty detection over contextual data streams at the edge using one-class classification. in: international conference on information and communication systems. ieee (2021) google scholar  kipf, t.n., welling, m.: variational graph auto-encoders. stat 1050, 21 (2016) math  google scholar  luan, s., et al.: revisiting heterophily for graph neural networks. adv. neural. inf. process. syst. 35, 1362–1375 (2022) google scholar  ma, y., guo, z., ren, z., tang, j., yin, d.: streaming graph neural networks. in: proceedings of the 43rd international acm sigir conference on research and development in information retrieval, pp. 719–728 (2020) google scholar  mendelson, s., lerner, b.: online cluster drift detection for novelty detection in data streams. in: international conference on machine learning and applications. ieee (2020) google scholar  moro, s., rita, p., cortez, p.: bank marketing. uci repository (2012) google scholar  na, g.s., kim, d., yu, h.: dilof: effective and memory efficient local outlier detection in data streams. in: proceedings of knowledge discovery and data mining (2018) google scholar  onuki, e.k.t., malucelli, a., barddal, j.p.: a tool for measuring energy consumption in data stream mining. in: br conference on intelligent systems. springer (2023) google scholar  peng, x., li, y., tsang, i.w., zhu, h., lv, j., zhou, j.t.: xai beyond classification: interpretable neural clustering. j. mach. learn. res. (2022) google scholar  pitonakova, l., bullock, s.: the robustness-fidelity trade-off in grow when required neural networks performing continuous novelty detection. neural networks (2020) google scholar  puerto-santana, c., et al.: asymmetric hmms for online ball-bearing health assessments. internet things j. 9(20), 20160–20177 (2022) google scholar  de souza, m.c., nogueira, b.m., rossi, r.g., marcacini, r.m., rezende, s.o.: a heterogeneous network-based positive and unlabeled learning approach to detect fake news. in: brazilian conference on intelligent systems. springer (2021) google scholar  spinosa, e.j., de leon f. de carvalho, a.p., gama, j.: olindda: a cluster-based approach for detecting novelty and concept drift in data streams. in: proceedings of the 2007 acm symposium on applied computing, pp. 448–452 (2007) google scholar  tang, j., liao, r.: graph neural networks for node classification. in: graph neural networks: foundations, frontiers, and applications, pp. 41–61 (2022) google scholar  togbe, m.u., et al.: anomaly detection for data streams based on isolation forest using scikit-multiflow. in: gervasi, o., et al. (eds.) iccsa 2020. lncs, vol. 12252, pp. 15–30. springer, cham (2020). https://doi.org/10.1007/978-3-030-58811-3_2 veloso, b., gama, j., malheiro, b., vinagre, j.: hyperparameter self-tuning for data streams. inf. fusion 76, 75–86 (2021) article  math  google scholar  wang, x., jin, b., du, y., cui, p., tan, y., yang, y.: one-class graph neural networks for anomaly detection in attributed networks. neural comput. appl. 33(18), 12073–12085 (2021). https://doi.org/10.1007/s00521-021-05924-9 article  math  google scholar  xu, k., hu, w., leskovec, j., jegelka, s.: how powerful are graph neural networks? in: proceedings of the international conference on learning representations. open review (2019) google scholar  zhang, f., fan, h., wang, r., li, z., liang, t.: deep dual support vector data description for anomaly detection on attributed networks. int. j. intell. syst. 37(2), 1509–1528 (2022) article  math  google scholar  zheng, y., yi, l., wei, z.: a survey of dynamic graph neural networks. arxiv preprint arxiv:2404.18211 (2024) download references acknowledgments this work was supported by coordenação de aperfeiçoamento de pessoal de nível superior (capes) grant number 88887.671481/2022-00. also, this work was supported by latam google ph.d. fellowship. finally, we would like to thank google tutor mara finkelstein for reviewing the work. author information authors and affiliations institute of mathematical and computer sciences, university of sao paulo, são carlos, são paulo, brazil marcos paulo silva gôlo & ricardo marcondes marcacini institute for systems and computer engineering, technology and science, university of porto, porto, portugal joão gama authors marcos paulo silva gôloview author publications search author on:pubmed google scholar joão gamaview author publications search author on:pubmed google scholar ricardo marcondes marcaciniview author publications search author on:pubmed google scholar corresponding author correspondence to marcos paulo silva gôlo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva gôlo, m.p., gama, j., marcondes marcacini, r. (2025). one-class learning for data stream through graph neural networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_5 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords one-class classification graph representation learning representation learning for data stream publish with us policies and ethics profiles joão gama view author profile ricardo marcondes marcacini view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a contrastive objective for training continuous generative flow networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a contrastive objective for training continuous generative flow networks conference paper first online: 30 january 2025 pp 3–16 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a contrastive objective for training continuous generative flow networks download book pdf download book epub tiago da silva9 & diego mesquita9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 510 accesses abstract generative flow networks (gflownets) are a novel class of flexible amortized samplers for distributions supported on complex objects (e.g., graphs and sequences), achieving significant success in problems such as combinatorial optimization, drug discovery and natural language processing. nonetheless, training of gflownets is challenging—partly because it relies on estimating high-dimensional integrals, including the log-partition function, via stochastic gradient descent (sgd). in particular, for distributions supported on non-discrete spaces, which have received far less attention from the recent literature, every previously proposed learning objective either depends on estimating a log-partition function or is restricted to on-policy training, which is susceptible to mode-collapse. in this context, inspired by the success of contrastive learning for variational inference, we propose the continuous contrastive loss (ccl) as the first objective function natively enabling off-policy training of continuous gflownets without reliance on the approximation of high-dimensional integrals via sgd, extending previous work based on discrete distributions. additionally, we show that minimizing the ccl objective is empirically effective and often leads to faster training convergence than alternatives. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a nonmonotone conditional gradient method for multiobjective optimization problems article 26 july 2024 simdcl: dropout-based simple graph contrastive learning for recommendation article open access 10 february 2023 hypernetwork-driven centralized contrastive learning for federated graph classification article 16 august 2024 1 introduction the evaluation of high-dimensional expectations under unnormalized distributions is one of the main challenges of statistics and machine learning [4]. in bayesian decision theory, for instance, one seeks to minimize a risk function that is defined as an expectation of a functional under a often intractable and unnormalized posterior distribution provisioned by bayes’ rule [21]. similarly, many model-free reinforcement learning algorithms for searching an optimal policy in a contextual markov decision process rely on integrating out a high-dimensional context variable of the quality function. under these circumstances, a common approach consists of substituting the unmanageable integration problem by a manageable optimization problem, which is known as variational inference. for this, one introduces a tractable and large set of parameteric distributions, known as the variational family, and finds the family’s member that most closely matches the unnormalized target by solving a stochastic optimization problem. then, one uses the solution to this problem as a surrogate to the intractable target and evaluate the desired expectations using monte carlo estimators. recently, generative flow networks (gflownets) [2, 3, 13] were proposed as a flexible variational family parameterized by neural networks to approximate unnormalized distributions with compositional supports [15], with remarkable success in relevant applications including causal inference [5, 24], drug discovery [2], refinement of large language models [9], and combinatorial optimization [28]. to implement a gflownet, we first define a flow network over an extension of the target distribution’s support, which thereafter corresponds to the sink nodes, and then search for a consistent flow assignment for which the flow going through each sink node matches its unnormalized probability. during inference, we travel across the defined network by starting at the origin and choosing each transition according to the estimated flow therein. if the flow is correctly estimated, this procedure is guaranteed to yield independent and identically distributed (i.i.d.) samples from the intractable target distribution [2, 13]. to successfully find a consistent and balanced flow assignment, we conventionally parameterize the network’s transition policies with versatile models as, e.g., multilayer perceptrons or graph neural networks. then, to estimate the model’s parameters, we minimize a loss function consisting of the expectation of log-squared local balance violations according to a prescribed exploratory policy via a variant of sgd. importantly, this exploratory policy may be different of the learned policy, in which case the training is said to be carried in an off-policy fashion. we remark that, when dealing with high-dimensional and highly sparse target distributions, an off-policy sampling scheme is paramount for avoiding the collapse of the variational distribution [14, 15]. nonetheless, as we note in sect. 2, such a possibility of off-policy learning comes at the cost of introducing additional parameters that often entail the estimation of high-dimensional integrals as a subproblem to the successful training of gflownets, potentially hindering the sample-efficiency of these models. seeking to bypass this problem, recent work [23, 26] developed a contrastive learning objective for gflownets. in the spirit of hinton’s contrastive divergence learning [8], these approaches sidestep the estimation of difficult-to-approximate quantities by, instead of directly optimizing an optimality criteria, minimizing the difference between them. this ensures that the intractable terms cancel out and that the resulting problem is computationally easier to solve. remarkably, however, these works are constrained to sampling from unnormalized distributions with finite support; a formal extension and an empirical validation of these objectives to the non-discrete setting is lacking in the literature. in this context, we extend both the contrastive balance condition and the contrastive loss function [23] to train gflownets defined on measurable topological spaces [13]. in addition, we empirically show that the resulting continuous contrastive loss (ccl) frequently leads to faster training convergence with respect to competing learning objectives for the standard tasks of approximating a sparse mixture of multivariate gaussian distributions and a banana-shaped distribution. in summary, our contributions are: 1. we derive a contrastive balance condition for continuous gflownets and rigorously show that it is a sufficient for ensuring sampling correctness; 2. we propose the continuous contrastive loss (ccl) as a theoretically sound learning objective for gflownets that does not rely on the estimation of high-dimensional integrals via sgd during training; 3. we empirically compare the performance of ccl against alternative loss functions and show that ccl leads to faster convergence in most cases. the paper is organized as follows. in sect. 2, we recall the notions of a measurable pointed directed acyclic graph (dag), transition kernels and flow networks, laying out the theoretical framework upon which gflownets are built, and also review popular learning objectives for continuous gflownets. then, in sect. 3, we rigorously derive the contrastive balance condition and propose ccl as a sound learning objective for gflownets, the performance of which is assessed in sect. 4 for the approximation of a mixture of multivariate gaussians and of a banana-shaped distributions [16]. finally, we provide a summary of the relevant literature in sect. 5 and a discussion of future directions, along with some concluding remarks, in sect. 6. 2 background this section reviews the main components of and the commonly used learning objectives for gflownets. to build intuition, we firstly define a gflownet for finitely supported distributions in sect. 2.1. then, we review the formal framework of lahlou et al. [13] for extending the notion of a flow networks to arbitrary topological spaces by substituting an adjacency matrix by a transition kernel that describes the continuous network’s connectivity. 2.1 gflownets in discrete spaces discrete gflownets. let r be a (potentially unnormalized) probability distribution on a finite space \(\mathcal {x}\) and \(s \supseteq \mathcal {x}\) be an extension of \(\mathcal {x}\) with two distinguished elements, \(s_{o}\) and \(s_{f}\). our objective is to randomly sample elements \(x \in \mathcal {x}\) proportionally to r(x). for this, we let \(g = (s, e)\) be a dag with edges e such that (i) the only nodes connected to \(s_{f}\) are those in \(\mathcal {x}\), (ii) there is a path from \(s_{o}\) to every \(x \in \mathcal {x}\), (iii) there are no outgoing edges from \(s_{f}\) and there are no incoming edges to \(s_{o}\). we call \(\mathcal {g}\) the state graph. in this setting, let \(p_{f} :s \times s \rightarrow \mathbb {r}_{+}\) be a forward policy on \(\mathcal {g}\), i.e., a function such that \(p_{f}(s, \cdot )\) is a probability measure on s supported on the children of s in \(\mathcal {g}\) and \(p_{f}(\tau ) = \prod _{1 \le i \le n} p_{f}(s_{i} | s_{i 1})\) be the corresponding distribution over trajectories \(\tau = (s_{o}, \dots , s_{n 1}, s_{n} = s_{f})\). to accomplish our objective, we must find a \(p_{f}\) for which $$\begin{aligned} \sum _{\tau :\tau \rightsquigarrow x} p_{f}(\tau ) \propto r(x). \end{aligned}$$ (1) in practice, the left-hand-side of eq. (1) is computationally intractable and is satisfied by a possibly infinite number of \(p_{f}\)’s. to ensure tractability and uniqueness, we also introduce a backward policy \(p_{b} :s \times s \rightarrow \mathbb {r}_{+}\), which is a forward policy over the transpose of \(\mathcal {g}\), and rewrite eq. (1) as \(p_{f}(\tau ) \propto p_{b}(\tau | x) r(x)\). malkin et al. [14] showed that such technique induces a well-posed problem with a unique solution. training gflownets. to find a forward policy abiding by eq. (1), we parameterize \(p_{f}\) with a neural network with parameters \(\theta \) and fix \(p_{b}(s, \cdot )\) as an uniform policy for each \(s \in s\). then, by introducing an additional learnable function \(f :s \rightarrow \mathbb {r}_{+}\), which is also parameterized by a (different) neural network \(\gamma \), we concomitantly estimate \(\theta \) and \(\gamma \) by minimizing one of the following objectives. below, \(p_{e}\) is an exploratory policy, i.e., a forward policy that does not depend on \((\theta , \gamma )\) and has full support over the space of trajectories. also, we denote by x (resp. \(x'\)) the state immediately preceding \(s_{f}\) in a trajectory \(\tau \) (resp. \(\tau '\)). 1. the detailed balance (db) loss [3] corresponds to \(\mathcal {l}_{db}(\theta , \gamma ) =\) $$\begin{aligned} \begin{aligned} \mathbb {e}_{\tau \sim p_{e}}\left[ \frac{1}{|\tau |}\sum _{(s_{i 1}, s_{i}) \in \tau } \left( \log \frac{f_{\gamma }(s_{i 1})p_{f}(s_{i} | s_{i 1})}{p_{b}(s_{i} | s_{i 1}) f_{\gamma }(s_{i 1})}\right) ^{2} + \left( \log \frac{f_{\gamma }(x)}{r(x)} \right) ^{2} \right] \end{aligned} \end{aligned}$$ which enforces the db condition, \(f(s) p_{f}(s | s') = f(s') p_{b}(s|s')\) for \((s, s') \in e\) and \(f(x) = r(x)\) for the states \(x \in s\). 2. the trajectory balance (tb) loss [14] introduces \(\log z(\gamma ) = \log f_{\gamma }(s_{o})\) as the log-partition function of r and is then defined by $$\begin{aligned} \mathcal {l}_{tb}(\theta , \gamma ) = \mathbb {e}_{\tau \sim p_{e}} \left[ \left( \log z(\gamma ) + \log p_{f}(\tau ; \theta ) \log p_{b}(\tau | x) \log r(x) \right) ^{2} \right] , \end{aligned}$$ enforcing the tb condition \(f(s_{o}) p_{f}(\tau ) = p_{b}(\tau | x) r(x)\). 3. the contrastive balance (cb) loss [23] avoids parameterizing f by sampling a pair \((\tau , \tau ')\) of trajectories from \(p_{e}\), being defined by $$\begin{aligned} \mathcal {l}_{cb}(\theta ) = \mathbb {e}_{(\tau , \tau ') \sim p_{e}} \left[ \left( \log \frac{p_{f}(\tau )}{p_{b}(\tau | x) r(x)} \log \frac{p_{f}(\tau ')}{p_{b}(\tau ' | x') r(x')} \right) ^{2} \right] , \end{aligned}$$ (2) and ensuring the cb condition, \(p_{f}(\tau ) p_{b}(\tau '|x')r(x') = r(x) p_{f}(\tau ') p_{b}(\tau |x)\). notably, the learning of discrete gflownets is widely studied, and many strategies for enhancing training convergence were proposed [3, 11, 17, 18, 22, 25, 28, 30]. in this work, we focus on extending the cb objective to the continuous setting. 2.2 gflownets in continuous spaces notations. we let \((\mathcal {x}, t_{\mathcal {x}}, \varsigma _{\mathcal {x}}, \mu )\) be a topological measurable space with reference measure \(\mu \), topology \(t_{\mathcal {x}}\), \(\sigma \)-algebra \(\varsigma _{\mathcal {x}}\), and underlying space \(\mathcal {x}\), and consider a non-negative function \(r :\mathcal {x} \rightarrow \mathbb {r}_{+}\) over \(\mathcal {x}\) such that \(\int _{\mathcal {x}} r(x) \mu (\textrm{d}x) < \infty \). in what follows, we will assume that each measure, denoted by a capital letter, has a density relatively to the reference measure \(\mu \), which will be denoted by the corresponding lowercase letter. under these conditions, our objective is to learn a generative process over \(\mathcal {x}\) that samples each measurable subset a proportionally to \(\int _{a} r(x) \mu (\textrm{d}x)\), and we define the target measure \(r :\varsigma _{\mathcal {x}} \rightarrow \mathbb {r}_{+}\) as \(r(a) = \int _{a} r(x) \mu (\textrm{d}x)\), which is absolutely continuous with respect to \(\mu \). additionally, for a given set s and a corresponding \(\sigma \)-algebra \(\varsigma _{s}\) (which will be specified later), we define a markov kernel on s as a function \(k :s \times \varsigma _{s} \rightarrow \mathbb {r}_{+}\) such that \(k(s, \cdot )\) is a measure in \((s, \varsigma _{s})\). also, we denote by \(s^{\otimes n}\) the nth-order cartesian product of s and by \(\varsigma _{s}^{\otimes n}\) the product \(\sigma \)-algebra of \(\varsigma _{s}\). naturally, a transition kernel k on s is naturally extended to a transition kernel \(\kappa ^{\otimes n} :s \times \varsigma _{s}^{\otimes n} \rightarrow \mathbb {r}_{+}\) on \(s^{\otimes n}\) recursively via \(\kappa ^{\otimes n}(s, (a, a_{n})) = \kappa ^{\otimes (n 1)}(s, a) \kappa (s, a_{n})\), with \(a \in \varsigma _{s}^{\otimes (n 1)}\), \(a_{n} \in \varsigma _{s}\), \(s \in s\), which exists uniquely due to carathéodory’s extension theorem. finally, we denote by \(s^{\otimes \le n} = \bigcup _{1 \le i \le n} s^{\otimes i}\) and \(\varsigma _{s}^{\otimes \le n} = \sigma \left( s^{\otimes \le n}\right) \) the \(\sigma \)-algebra generated by the space of up-to-nth order product \(\sigma \)-algebras. measurable pointed dags (mp-dag). we first present the definition of an mp-dag, originally proposed by lahlou et al. [13], which formally extends the concept of a flow network to a potentially non-countable state space. definition 1 (mp-dag) let \((\bar{s}, \varsigma )\) be a topological space and \(\varsigma \) be the borel \(\sigma \)-algebra on \(\mathcal {t}\). define the source \(s_{o} \in \bar{s}\) and sink \(s_{f} \in \bar{s}\) states. also, let \(\kappa \), the reference kernel, and \(\kappa _{b}\), the backward kernel, be \(\sigma \)-finite and continuous markov kernels in \((\bar{s}, \mathcal {t})\); and let \(\mu \) be a reference measure on \(\varsigma \). also, let \(s = \bar{s} \setminus \{s_{f}\}\). a measurable pointed dag is a tuple \(\mathcal {g} = (\bar{s}, \mathcal {t}, \varsigma , \kappa , \kappa _{b}, \mu )\) such that 1. (finality) \(\kappa (s_{f}, \cdot ) = \delta _{s_{f}}\) and if \(\kappa (s, \{s_{f}\}) > 0\) for a \(s \in \bar{s}\), \(\kappa (s, \{s_{f}\}) = 1\); 2. (reachability) \(\forall b \in \mathcal {t} \setminus \{\emptyset \}\), \(\exists n \ge 0\) with \(\kappa ^{\otimes n}(s_{o}, b) > 0\); 3. (initialness) \(\forall b \in \varsigma \), \(\kappa _{b}(s_{o}, b) = 0\); and 4. (consistency) \(\forall b \in \varsigma \times \varsigma \), \(\nu \otimes \kappa (b) = \nu \otimes \kappa _{b}(b)\). also, a mp-dag is said to be finitely absorbing if there is a \(n \in \mathbb {n}\) for which \(\kappa ^{\otimes n}(s_{o}, \{s_{f}\}) > 0\) and, by the finality property, \(\kappa ^{\otimes n}(s_{o}, \{s_{f}\}) = 1\). in other words, any iterative generative process starting at \(s_{o}\) and sampling novel states according to \(\kappa \) will reach \(s_{f}\) in at most n steps. we call n the maximum trajectory length of the mp-dag, and define \(\mathcal {x} = \{x \in s :\kappa (x, \{s_{f}\}) > 0\}\) as the set of terminal states. under these circumstances, the set \(\{s_{o}\} \times s^{\otimes \le n 1} \times \{s_{f}\}\) fully characterizes the trajectories starting at \(s_{o}\), and we will denote it by \(\mathbb {t}\); also, we denote by \(\mathbb {t}' = \{s_{o}\} \times (s \setminus \mathcal {x})^{\otimes \le n 1}\) the space of trajectories without terminal states. we highlight that, when s is finite and \(\mu \) is set as the counting measure in s, then the definition above recovers the traditional flow network outlined in sect. 2.1 for the discrete setting, with \(\{\kappa (s, \{s'\})\}_{(s, s') \in s \times s}\) characterizing the network’s adjacency matrix and n denoting its (directed) diameter. continuous gflownets. similarly to its discrete counterpart, we characterize a continuous gflownet by a mp-dag \(\mathcal {g} = (\bar{s}, \mathcal {t}, \varsigma , \kappa , \kappa _{b}, \mu )\), a measure r on \(\mathcal {x}\), a forward policy \(p_{f} :\bar{s} \times \varsigma _{\bar{s}} \rightarrow \mathbb {r}_{+}\), defined as a markov kernel that is absolutely continuous relatively to \(\kappa \), and a backward policy \(p_{b} :\bar{s} \times \varsigma _{\bar{s}} \rightarrow \mathbb {r}_{+}\), defined as a markov kernel absolutely continuous relatively to \(\kappa _{b}\). our objective is to find a \(p_{f}\) such that the marginal distribution over \(\mathcal {x}\) of the \(\kappa \)-markov chain deterministically starting at \(s_{o}\) matches r, that is, informally, $$\begin{aligned} \int _{\mathbb {t}} \mathbbm {1}_{\{\tau \text { finishes at } a\}} p_{f}^{\otimes n}(s_{o}, \textrm{d}\tau ) \propto r(a) \end{aligned}$$ (3) for every measurable subset a of \(\mathcal {x}\). training gflownets. to achieve this objective, we let \(p_{f}\) and \(p_{b}\) be the densities of \(p_{f}\) and \(p_{b}\) relatively to \(\kappa \) and \(\kappa _{b}\), respectively, and parameterize \(p_{f}\) as a neural network \(\theta \). to avoid notational overload, we also denote by \(p_{f}\) and \(p_{b}\) the densities of \(p_{f}^{\otimes i}\) and \(p_{b}^{\otimes i}\) with respect to the corresponding product kernels. analogously to the discrete gflownets, we define an auxiliary function \(u :\bar{s} \rightarrow \mathbb {r}_{+}\) parameterized by a neural network \(\gamma \). additionally, we introduce an exploratory policy \(p_{e} :\bar{s} \times \varsigma _{\bar{s}} \rightarrow \mathbb {r}_{+}\) absolutely continuous with respect to the forward kernel \(\kappa \). under these conditions, lahlou et al. [13] showed that the trajectory and detailed balance losses of sect. 2.1 are sound objectives for learning continuous gflownets when we substitute the transition kernels by the corresponding densities. so, for instance, we let \(z = u_{\gamma }(s_{o})\) and then $$\begin{aligned} \mathcal {l}_{tb}(\theta , \gamma ) = \mathbb {e}_{\tau \sim p_{e}(s_{o}, \cdot )} \left[ \left( \log z(\gamma ) + \log p_{f}(\tau ) \log p_{b}(\tau | x) \log r(x) \right) ^{2} \right] \end{aligned}$$ (4) becomes the continuous equivalent of the tb loss. remarkably, both the tb and db losses rely on the estimation of the auxiliary function u, which often corresponds to a high-dimensional integral (such as the partition function) that is difficult to compute and has no inferential purpose. illustration. to clarify the above definitions and emphasize their versatility, we show how to instantiate a continuous gflownet to approximate an arbitrary distribution in an euclidean space \(\mathbb {r}^{n}\), an example that will be thoroughly explored in our experimental campaign in sect. 4. for this setting, we may define \(s = \mathbb {r}^{n}\) and \(\mathcal {x} = \{(x, \top ) :x \in \mathbb {r}^{n}\}\), with \(\top \) artificially distinguishing the elements of \(\mathcal {x}\) from those of s. then, \(s_{o} = \textbf{0}\) and a transition corresponds to \(x^{t + 1} = x^{t} + \alpha e^{t}\), with \(\alpha \sim q_{\theta }\) sampled from an appropriate distribution \(q_{\theta }\) with parameters estimated by a neural network that receives \(x^{t}\) as input and \(e^{t}\) denoting the tth line of the identity matrix. critically, this iterative generative process induces a finitely absorbing mp-dag satisfying all the desired properties when we define \(\kappa \) to satisfy \(\kappa (x, \cdot ) = \delta _{s_{f}}(\cdot )\) for every \(x \in \mathcal {x}\) and \(\kappa _{b}(x^{t}, \cdot ) = \delta _{x^{t} x_{t}^{t} e_{t}}(\cdot )\). 3 a contrastive objective we now present our main theoretical results regarding the soundness of the contrastive balance objective for training gflownets in a non-countable state space. firstly, sect. 3.1 defines the cb condition as a provably sufficient requirement for ensuring the sampling correctness of gflownets. secondly, sect. 3.2 derives the contrastive continuous loss (ccl) as a realization of the cb condition that, when minimized, ensures that the resulting model is correct almost everywhere. finally, we analyze the connection between the ccl and the popular variance loss [20] for carrying out generic vi. 3.1 contrastive balance contrastive balance. to get a finer understanding of the contrastive balance condition, we first remark that the tb condition may be equivalently stated as $$\begin{aligned} \text {var}\left[ \frac{p_{f}(\tau )}{p_{b}(\tau | x) r(x)} \right] = 0, \end{aligned}$$ (5) i.e., the quotient \(\beta (\tau ) = \nicefrac {p_{f}(\tau )}{p_{b}(\tau | x) r(x)}\) is constant as a function of \(\tau \). thus, if \(\beta (\tau ) = \beta (\tau ')\) for any pair of trajectories \((\tau , \tau ')\), the tb condition must be satisfied and the gflownet should sample correctly from the target. importantly, however, the \(\beta \) does not depend on the estimation of intractable quantities and enforcing the equality \(\beta (\tau ) = \beta (\tau ')\), which is the essence of definition 2 below, avoids the estimation of high-dimensional integrals such as \(\log z(\gamma )\) in eq. (4). nextly, we formally define the contrastive balance objective. notably, we slightly abuse notation by denoting \(p_{b}(x, \textrm{d}\tau )\) for the product kernel \(p_{b}(x, \textrm{d}s_{n}) \otimes p_{b}(s_{n}, \textrm{d}s_{n 1}) \otimes \cdots \otimes p_{b}(s_{1}, \textrm{d}s_{o})\) associated to \(\tau = (s_{o}, s_{1}, \dots , s_{n}, x)\). definition 2 (contrastive balance) let \((\mathcal {g}, p_{f}, p_{b}, r, \mu )\) be a gflownet. the contrastive balance (cb) condition is defined by $$\begin{aligned} \begin{aligned} \int _{\mathcal {x}} \int _{\mathcal {x}} \int _{\mathbb {t}'} \int _{\mathbb {t}'} f(\tau , x, \tau ', x') p_{f}(s_{o}, \textrm{d}\tau \textrm{d}x) p_{b}(x', \textrm{d}\tau ') r(\textrm{d}x') \\ = \int _{\mathcal {x}} \int _{\mathcal {x}} \int _{\mathbb {t}'} \int _{\mathbb {t}'} f(\tau , x, \tau ', x') r(\textrm{d}x) p_{b}(x, \textrm{d}\tau ) p_{f}(s_{o}, \textrm{d}\tau '\textrm{d}x') \end{aligned} \end{aligned}$$ (6) for every bounded measurable function \(f :\mathbb {t}' \times \mathcal {x} \times \mathbb {t}' \times \mathcal {x} \rightarrow \mathbb {r}\). illustratively, when s is finite, the reference measure is the counting measure, and we let f represent the indicator functions of elements in the space \(\mathbb {t}' \times \mathcal {x} \times \mathbb {t}' \times \mathcal {x}\), the condition above reduces to the discrete contrastive balance condition, namely, \(p_{f}(\tau , x) p_{b}(x', \tau ') r(x') = r(x) p_{b}(x, \tau ) p_{f}(\tau ', x')\) for every pair \((\tau , \tau ')\) of trajectories and \((x, x')\) of terminal states. in the light of this, definition 2 generalizes the discrete cb condition to a significantly broader setting. sufficiency of cb. we rigorously show in the theorem below that the continuous variant of the cb condition outlined in definition 2 is sufficient for ensuring that the corresponding gflownet generates samples distributed according to the target measure r. note, for this, that we may rewrite the eq. (6) in term of the densities of the corresponding measures. theorem 1 let \((\mathcal {g}, p_{f}, p_{b}, r, \mu )\) be a gflownet abiding by the contrastive balance condition. then, the marginal distribution over \(\mathcal {x}\) induced by the \(p_{f}\)-markov chain starting at \(s_{o}\) matches r. proof to start with, we define \(t :\mathbb {t} \rightarrow \mathcal {x}\) as the function taking a complete trajectory and returning its terminal state. by the definition of \(\mathcal {x}\), t is well-defined almost everywhere with respect to the measure \(\kappa (s_{o}, \cdot )\) over trajectories. then, we show that eq. (6) is enough to ensure, for all measurable \(a \subseteq \mathcal {x}\), that $$\begin{aligned} \int _{\mathbb {t}} \mathbbm {1}_{\{t(\tau ) \in a\}} p_{f}(s_{o}, \textrm{d}\tau ) \propto r(a), \end{aligned}$$ (7) i.e., the marginal distribution of \(p_{f}\) on \(\mathcal {x}\) matches r. under these conditions, we first note that, when the function f does not depend on the last two parameters, eq. (6) becomes $$\begin{aligned} \begin{aligned} \int _{\mathcal {x}} \int _{\mathbb {t}'} f(\tau , x) p_{f}(s_{o}, \textrm{d}\tau \textrm{d}x) \underbrace{\left( \int _{\mathcal {x}} \int _{\mathbb {t}'} r(\textrm{d}x') p_{b}(x', \textrm{d}\tau ') \right) }_{= z} \\ = \underbrace{\left( \int _{\mathcal {x}} \int _{\mathbb {t}'} p_{f}(s_{o}, \textrm{d}\tau ' \textrm{d}x') \right) }_{= 1} \int _{\mathcal {x}} \int _{\mathbb {t}'} f(\tau , x) r(\textrm{d}x) p_{b}(x, \textrm{d}\tau ), \end{aligned} \end{aligned}$$ (8) which recovers lahlou et al.’s tb condition [13, definition 6]. consequently, the result follows from [13, theorem 2, (4)]. for completeness, however, it is easy to see that eq. (8) ensures that \(p_{f}(s_{o}, \cdot )\) and \(r(\textrm{d}x) p_{b}(x, \cdot )\), when interpreted as distributions over \(s^{\otimes \le n 1} \times \{s_{f}\}\), are equivalent in terms of expectations of bounded measurable functions and, thus, must be indistinguishable up to a multiplicative constant. hence, by integrating out the elements of \(\tau \) that are not members of \(\mathcal {x}\), we obtain eq. (7). finally, before we delve into the algorithmic aspects of contrasitve learning of gflownets, we again emphasize the result above is general enough to contemplate distributions supported on continuous, discrete, and mixed spaces. 3.2 contrastive loss contrastive loss. realistically, finding a policy \(p_{f}\) satisfying the cb condition is an analytically intractable problem. as a consequence, we parameterize \(p_{f}\) with a neural network with weights \(\theta \) and estimate the optimal \(\theta ^{\star }\) by minimizing a loss function that provably enforces eq. (6). the corollary below, which is an immediate consequence of theorem 1, provides such a loss function. corollary 1 (continuous contrastive loss). under the notations of theorem 1, let \(p_{f}\), \(p_{b}\) and r be the densities of \(p_{f}\), \(p_{b}\), and r relatively to \(\kappa \), \(\kappa _{b}\) and \(\mu \), respectively. then, we define the continuous contrastive loss (ccl) as $$\begin{aligned} \mathcal {l}_{cb}(\theta ) = \mathbb {e}_{\tau , \tau ' \sim p_{e}(s_{o}, \cdot ) \otimes p_{e}(s_{o}, \cdot )} \left[ \left( \frac{p_{f}(\tau )}{p_{b}(\tau | x) r(x)} \frac{p_{f}(\tau ')}{p_{b}(\tau ' | x') r(x')}\right) ^{2} \right] , \end{aligned}$$ (9) in which \(p_{e}(s_{o}, \cdot )\) is an exploratory policy with full support over the trajectories on \(\varsigma _{\mathcal {s}}^{\otimes \le n}\) starting at \(s_{o}\) and we use x (resp. \(x'\)) to denote the unique terminal state within \(\tau \) (resp. \(\tau '\)). then, if \(\mathcal {l}_{cb}(\theta ^{\star }) = 0\), the gflownet parameterized by \(\theta ^{\star }\) abides by eq. (8) and samples correctly from the target. intuitively, the condition \(\mathcal {l}_{cb}(\theta ) = 0\) ensures that the quotients \(\beta (\tau )\) defined in sect. 3.1 are constant \(\kappa (s_{o}, \cdot )\)-almost everywhere, which then implies theorem 1 and ultimately guarantees gflownet’s distributional correctness. algorithm 1 contrastive learning of gflownets full size image algorithm. algorithm 1 illustrates the training of gflownets through the minimization of the ccl. in practice, both the sampling and neural network forward passes can be massively parallelized. also, for the optimization step, we use adam [12] to estimate the stochastic gradients. importantly, the architecture of the parameterizing neural network with weights \(\theta \) and the nature of exploratory policy \(p_{e}\) are very problem-dependent and are abstracted away from algorithm 1; we provide some examples in the next section. connection to other losses. to conclude, we remark that the cb loss is tightly connected to the variance loss for variational inference [20, 26]; indeed, it is well known that the expectation of the squared-difference between two independently sampled variables equal their variance, i.e., \(\mathbb {e}_{\tau , \tau '}[(\beta (\tau ) \beta (\tau '))^{2}] = \text {var}_{\tau }[\beta (\tau )]\). 4 experiments this section shows that the ccl is a sound and effective learning objective for continuous gflownets in varied experimental setups, which are described below. tasks. we underline the effectiveness of the cb loss in the tasks of sampling from a mixture of gaussian distributions (gm) [28] and of a banana-shaped distribution (bs) [16], which are commonly implemented benchmark models respectively for continuous gflownets and variational inference algorithms in general. for the gms, we evaluate gflownet’s performance for a sparse 2-dimensional model and a 15-dimensional model, both with isotropic variances. for the bs distribution, we consider the target distribution $$\begin{aligned} \mathbb {r}^{2} \ni \textbf{x} \sim \mathcal {n}\left( \begin{bmatrix} x_{1} \\ x_{2} + x_{1}^{2} + 1 \end{bmatrix} \bigg | \begin{bmatrix} 0 \\ 0 \end{bmatrix}, \begin{bmatrix} 1 & 0.9 \\ 0.9 & 1 \end{bmatrix}\right) . \end{aligned}$$ (10) experimental setup. for each experiment, we consider the iterative generative process described at the end of sect. 2.2. to parameterize \(q_{\theta }\), which is here defined as a gaussian mixture distribution, we employ an mlp with 2 64-dimensional layers that receives the current state as an input and return the log-weights, averages, and log-variances of the mixture. in algorithm 1, we fix a batch size of 128 and use adam with a linearly decaying learning rate of \(10^{-3}\) for optimization. however, when minimizing the tb loss, we adopt malkin et al.’s [14] suggestion to implement a fixed learning rate of \(10^{-1}\) for \(\log z(\gamma )\). code for reproducing the results will be released upon acceptance. fig. 1. minimizing the cb loss (bottom row) drastically improves the training convergence when compared to tb minimization (top row) when the target is a 2-dimensional sparse gaussian mixture (see fig. 3). this corroborates our hypothesis that avoiding parameterizing \(\log z(\gamma )\) facilitates the learning of gflownets. columns represent results of three randomly initialized runs. full size image fig. 2. cb and tb minimization perform similarly for the (relatively easy-to-approximate) banana-shaped target. we consider hmc (right plot) as a gold-standard reference for sample generation due to its accuracy and reliability. full size image fig. 3. a sparse gaussian mixture. full size image results. figure 1 highlights that a gflownet trained by minimizing \(\mathcal {l}_{cb}\) leads to a substantial improvement in terms of approximation quality when compared to the minimization of \(\mathcal {l}_{tb}\) given a fixed computational budget and a sparse target, which is illustrated in fig. 3. this validates our hypothesis that the training of \(\mathcal {l}_{tb}\) is significantly constrained by the obligatory estimation of the intractable \(\log z(\gamma )\). for the experiments based on the high-dimensional gaussian mixture, which cannot be simply visualized, we show in fig. 4 the jensen-shannon divergence between the learned distribution q and the target p, $$\begin{aligned} \mathcal {d}_{js}[p || q] = \frac{1}{2} \left( \mathcal {d}_{kl}[p || m] + \mathcal {d}_{kl}[q || m] \right) , \end{aligned}$$ (11) with \(m :==\nicefrac {p + q}{2}\) and \(\mathcal {d}_{kl}[p || m] = \mathbb {e}_{x \sim p} \left[ \log \nicefrac {p}{m}\right] \) representing the kullback-leibler divergence between p and m. importantly, \(\mathcal {d}_{js}\) is a frequently implemented metric for assessing gflownets [6, 13]. finally, for the low-dimensional and non-sparse targets, the training of gflownets based on both the minimization of cb and tb yield similar approximations given an equivalent computation resources, as fig. 2 attests, which is potentially a consequence of the relative simplicity of the target distribution. 5 related works discrete gflownets. gflownets were originally conceived as a reinforcement learning algorithm tailored to the search of a diversity of high-valued states [2, 3]. since then, this family of models was successfully applied to bayesian structure learning and causal discovery [1, 5, 6], natural language processing [9], robust scheduling [26], phylogenetic inference [30], and probabilistic modelling in general [10, 15, 27, 29]. correspondingly, a large effort was devoted to improving the training convergence of discrete gflownets [11, 14, 17, 19, 25]. fig. 4. learning curve for / . full size image continuous gflownets. in contrast, the literature on continuous gflownet, introduced by lahlou et al. [13], is relatively thin. to the best of our knowledge, the only fruitful application of these models occurred recently in the context of bayesian structure learning [6]. the reason for this, we believe, is the difficulty of efficiently training these models. indeed, zhou et al. [30] discretized the continuous component of the state space to avoid training a model to sample from a continuous distribution. under these conditions, our work is an important step towards the improvement the usability of continuous gflownets. 6 conclusions we rigorously introduced a continuous variant of the contrastive balance condition and an accompanying continuous contrastive loss for efficiently training gflownets over distributions supported on discrete, continuous, and mixed spaces. all in all, our theoretical analysis assured the reliability of the proposed learning objective and our empirical results suggested that minimizing the ccl often significantly speeds up the model’s training convergence. yet, the training of continuous gflownet remains challenging, and extending the recent advancements in the training of discrete gflownets to the continuous setting, along the lines of our work, is a promising endeavor that may greatly diminish these difficulties. for instance, one may build upon the framework of pan et al.’s [19] generative augmented flow networks to incorporate intermediate reward-related signals within a trajectory for enhanced credit assignment [22]. similarly, the theoretical developments of tiapkin et al. [7, 25] showing the relationship between rl and gflownets paves the road for the design of learning objectives motivated by techniques in the theory of control for continuous states. references atanackovic, l., tong, a., hartford, j., lee, l.j., wang, b., bengio, y.: dyngfn: towards bayesian inference of gene regulatory networks with gflownets. in: advances in neural processing systems (neurips) (2023) google scholar  bengio, e., jain, m., korablyov, m., precup, d., bengio, y.: flow network based generative models for non-iterative diverse candidate generation. in: neurips (2021) google scholar  bengio, y., lahlou, s., deleu, t., hu, e.j., tiwari, m., bengio, e.: gflownet foundations. j. mach. learn. res. (jmlr) 24, 1–55 (2023) mathscinet  google scholar  blei, d.m., et al.: variational inference: a review for statisticians. j. am. stat. assoc. 112, 859–877 (2017) article  mathscinet  math  google scholar  deleu, t., et al.: bayesian structure learning with generative flow networks. in: uai (2022) google scholar  deleu, t., nishikawa-toomey, m., subramanian, j., malkin, n., charlin, l., bengio, y.: joint bayesian inference of graphical structure and parameters with a single generative flow network. in: advances in neural processing systems (neurips) (2023) google scholar  deleu, t., nouri, p., malkin, n., precup, d., bengio, y.: discrete probabilistic inference as control in multi-path environments (2024) google scholar  hinton, g.e., osindero, s., teh, y.w.: a fast learning algorithm for deep belief nets. neural comput. 18, 1527–1554 (2006) article  mathscinet  math  google scholar  hu, e.j., jain, m., elmoznino, e., kaddar, y., et al.: amortizing intractable inference in large language models (2023) google scholar  hu, e.j., malkin, n., jain, m., everett, k.e., graikos, a., bengio, y.: gflownet-em for learning compositional latent variable models. in: international conference on machine learning (iclr) (2023) google scholar  jang, h., kim, m., ahn, s.: learning energy decompositions for partial inference in gflownets. in: the twelfth international conference on learning representations (2024) google scholar  kingma, d.p., ba, j.: adam: a method for stochastic optimization. arxiv preprint arxiv:1412.6980 (2014) lahlou, s., et al.: a theory of continuous generative flow networks. in: icml proceedings of machine learning research, vol. 202, pp. 18269–18300. pmlr (2023) google scholar  malkin, n., jain, m., bengio, e., sun, c., bengio, y.: trajectory balance: improved credit assignment in gflownets. in: neurips (2022) google scholar  malkin, n., et al.: gflownets and variational inference. in: international conference on learning representations (iclr) (2023) google scholar  mesquita, d., blomstedt, p., kaski, s.: embarrassingly parallel mcmc using deep invertible transformations. in: uai (2019) google scholar  pan, l., malkin, n., zhang, d., bengio, y.: better training of gflownets with local credit and incomplete trajectories. in: international conference on machine learning (icml) (2023) google scholar  pan, l., malkin, n., zhang, d., bengio, y.: better training of gflownets with local credit and incomplete trajectories. arxiv preprint arxiv:2302.01687 (2023) pan, l., zhang, d., courville, a., huang, l., bengio, y.: generative augmented flow networks. in: international conference on learning representations (iclr) (2023) google scholar  richter, l., boustati, a., nüsken, n., ruiz, f.j.r., akyildiz, ö.d.: vargrad: a low-variance gradient estimator for variational inference (2020) google scholar  robert, c.p., et al.: the bayesian choice: from decision-theoretic foundations to computational implementation, vol. 2. springer, heidelberg (2007). https://doi.org/10.1007/0-387-71599-1 shen, m.w., bengio, e., hajiramezanali, e., loukas, a., cho, k., biancalani, t.: towards understanding and improving gflownet training. in: international conference on machine learning (2023) google scholar  da silva, t., carvalho, l.m., souza, a., kaski, s., mesquita, d.: embarrassingly parallel gflownets (2024) google scholar  da silva, t., et al.: human-in-the-loop causal discovery under latent confounding using ancestral gflownets. arxiv preprint arxiv:2309.12032 (2023) tiapkin, d., morozov, n., naumov, a., vetrov, d.: generative flow networks as entropy-regularized rl (2024) google scholar  zhang, d.w., rainone, c., peschl, m., bondesan, r.: robust scheduling with gflownets. in: international conference on learning representations (iclr) (2023) google scholar  zhang, d., chen, r.t., malkin, n., bengio, y.: unifying generative models with gflownets and beyond. in: icml beyond bayes workshop (2022) google scholar  zhang, d., dai, h., malkin, n., courville, a., bengio, y., pan, l.: let the flows tell: solving graph combinatorial optimization problems with gflownets. in: neurips (2023) google scholar  zhang, d., malkin, n., liu, z., volokhova, a., courville, a., bengio, y.: generative flow networks for discrete probabilistic modeling. in: international conference on machine learning (icml) (2022) google scholar  zhou, m.y., et al.: phylogfn: phylogenetic inference with generative flow networks. in: the twelfth international conference on learning representations (2024) google scholar  download references acknowledgements this work was supported by fundação carlos chagas filho de amparo à pesquisa do estado do rio de janeiro faperj (sei-260003/000709/2023), são paulo research foundation fapesp (2023/00815-6), and conselho nacional de desenvolvimento científico e tecnológico cnpq (404336/2023-0). author information authors and affiliations getulio vargas foundation, rio de janeiro, brazil tiago da silva & diego mesquita authors tiago da silvaview author publications search author on:pubmed google scholar diego mesquitaview author publications search author on:pubmed google scholar corresponding author correspondence to diego mesquita . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, t., mesquita, d. (2025). a contrastive objective for training continuous generative flow networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_1 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords gflownets diffusion models bayesian inference publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a comparative study on concept drift detectors for regression | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a comparative study on concept drift detectors for regression conference paper first online: 28 november 2021 pp 390–405 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a comparative study on concept drift detectors for regression download book pdf download book epub marília lima  orcid: orcid.org/0000-0002-6755-264910, telmo silva filho  orcid: orcid.org/0000-0003-0826-688511 & roberta andrade de a. fagundes  orcid: orcid.org/0000-0002-7172-418310  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 994 accesses 4 citations abstract context: in the field of machine learning models are trained to learn from data, however often the context at which a model is deployed changes, degrading the performances of trained models and giving rise to a problem called concept drift (cd), which is a change in data distribution. motivation: cd has attracted attention in machine learning literature, with works proposing modification to well-known algorithms’ structures, ensembles, online learning and drift detection, but most of the cd literature regards classification, while regression drift is still poorly explored. objective: the goal of this work is to perform a comparative study of cd detectors in the context of regression. results: we found that (i) ph, kswin and eddm showed higher detection averages; (ii) the base learner has a strong impact in cd detection and (iii) the rate at which cd happens also affects the detection process. conclusion: our experiments were executed in a framework that can easily be extended to include new cd detectors and base learners, allowing future studies to use it. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a robust probabilistic framework for identifying and evaluating concept drift in abrupt and gradual scenarios chapter © 2025 benchmarking concept drift detectors for online machine learning chapter © 2023 sddm: an interpretable statistical concept drift detection method for data streams article 05 february 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. concept formation linear models and regression learning algorithms learning theory machine learning statistical learning 1 introduction machine learning is a sub-field of artificial intelligence which develops models that are able to learn from data. learning usually happens statically with fixed labels during the process. in the context of regression with continuous dependent variables, many models have been proposed with useful results in many scenarios. however, there has been a growing interest in models that are able to process streams of data and can learn incrementally [10]. while working with data streams, models often face changes in the probability distribution of input data, which can negatively affect performance if the model was trained on data sampled from a significantly different distribution. this problem is most commonly referred to as concept drift (cd). cd can happen in many different scenarios, such as weather forecasting, stock price prediction and fake news identification [17]. cd might happen in the conditional distribution of the dependent variable given the predictors, which is called real drift, or it can affect only the independent variable distribution, which is called virtual drift [12]. due to its significance, cd has been widely studied in recent years with approaches involving structural changes applied to well-known models, ensembles and drift detection. however, most of these works targeted the task of classification. in this context, gonçalves et al. (2014) [9] ran a large comparative study of cd detectors using naive bayes as the base learner. their work showed that there is no single best cd detector and performance depends on the pair (dataset, cd detector), i.e. “there is no free lunch”, as expected in machine learning. these results motivate a similar study in the context of regression with continuous targets. therefore, this paper’s main contribution is to perform a wide comparative study of cd detectors for regression. we used seven different cd detectors together with 10 regression models. the 70 detector-base learner combinations were applied to four synthetic and four real datasets with virtual cd. thus, our study presents significant variation of base learners, detectors and datasets. experiments were evaluated according to prediction error and number of detected drift points. as another contribution, we develop our work such that future research can easily expand it by adding new detectors, base learners and datasets. in addition, this study focuses on continuous outputs in the dependent variable, which is still scarcely explored in the literature. the rest of this paper is organized as follows: sect. 2 presents the drift detection methods. section 3 describes the parameters used in the drift detectors, the datasets used in the experiments, and the adopted evaluation methodology. the results obtained in the experiments are analyzed in sect. 4. finally, sect. 5 presents our conclusions. 2 background cd detection methods use a base learner (regressor/classifier) to identify if an input is a drift point, i.e. for each instance the method outputs a drift prediction based on the base learner prediction and the observed target value. most detectors perform this analysis in groups of instances at each time. many detectors have been proposed for regression. one of the first was the drift detection method (ddm) [8], which is based on the idea that the base learner’s error rate decreases as the number of samples increases, as long as the data distribution is stationary. if ddm finds an increased error rate above a calculated threshold, it detects that cd has happened. the threshold is given by \(p_{min}+s_{min}\) is minimum. where, \(p_{min}\) is the minimum recorded error rate and \(s_{min}\) is the minimum recorded standard deviation. ddm was later extended to monitor the average distance between two errors instead of only the error rate. this new method was called early drift detection method (eddm) [2]. a different approach, called adaptive windowing (adwin) [4], is a sliding window algorithm which detects drifts and records updated statistics about data streams. the size of the sliding window is decided based on statistics calculated at different points of the stream. if the difference between the calculated statistics is higher than a predefined threshold, adwin detects the cd and discards all data from the stream up to the detected drift point. another early cd detector, called page-hinkley (ph) [14], computes observed values and their average up until the current point in a data stream. ph then detects the drift if the observed average is higher than a predefined threshold. frías-blanco et al. (2014) [7] proposed two detectors. the first, called hddm_a, is based on hoeffding’s inequality and used the average of the continuous values in a data stream to determine if the data contain cd. the second method, hddm_w, uses mcdiarmid’s bounds and the exponentially weighted moving average (ewma) statistic to estimate if an instance represents cd or not. the kolmogorov-smirnov windowing (kswin) method [16] uses the kolmogorov-smirnov (ks) statistical test to monitor data distributions without assuming any particular distributions. kswin keeps a fixed window size and compares the cumulative distribution of the current window to the previous one, detecting cd if the ks test rejects the null hypothesis that the distributions are the same. 3 experiment configuration in this section we describe the datasets and the parametrization used for the drift detectors as well as the evaluation methodology used in the experiments. 3.1 datasets the datasets chosen for the experiments have all been previously used to study the concept drift problem. to analyze the performance of the methods, four synthetic datasets and four real-world datasets were used. the datasets allow us to analyze how the detectors identify the points of deviation in the data and false points detected. we use the synthetic data sets from the work of almeida et al. (2019) [1] following the same methodological process of adding deviation. for each data set 5000 samples are created. the domain of each attribute is divided into ten equal-sized parts. the first 2000 samples correspond to the first seven parts of the domain of each variable. 1000 new instances are added and the domain is expanded until the 5000 samples are completed. table 1 presents synthetic data. table 1. synthetic datasetsfull size table the real-world datasets used are bike [12], fccu1 (gasoline concentration), fccu2 (ldo concentration) and fccu3 (lpg concentration) [18]. we performed an analysis on the target variable using the interquartile range method (iqr), \(iqr=q_{3}-q_{1}\), where \(=q_{3}\) is the third quartile and \(=q_{1}\) is the first quartile. we perform a variability estimate to calculate lower \(l_{inf}=\bar{y}-1.5\, *\, iqr\) and higher \(l_{sup}=\bar{y}+1.5 * iqr\) bounds for drift identification, where \(\bar{y}\) is the mean of the target variable. at the end of the analysis we noticed that none of the data points were out of bounds. thus, we artificially added drift along the data. figure 1 shows the distributions of y and \(y_{drift}\) (y with artificial drift). to add the deviation we add the value of the dependent variable (y) by multiplying the standard deviation and a number that leaves the value of y greater than \(l_{sup}\). the added drift in the bike dataset ((a)) has the characteristic of recurring and abrupt speed, as it appears in the data in a certain period of time, returns to the original concept, and appears again, in addition to abruptly changing the concept. we add two offsets by multiplying by the numbers three and four respectively. in the fccu1, fccu2, fccu3 dataset we add three abrupt deviations. which we multiply: by three, seven and, twelve (for fccu1); three, six, three (for fccu2); three, six, nine (for fccu3). fig. 1. datasets full size image table 2 presents a summary of the main characteristics of each dataset, such as number of predictive attributes, number of samples and amount of drift. table 2. dataset featuresfull size table 3.2 evaluation setup to evaluate the drift detection methods using the presented datasets, we use a methodology similar to the one described in [1]. for each time step, a specified amount of instances of the training set are read and used to train the regressor. next, other examples from the test set are used to test the regressor. if the detection method identifies the occurrence of cd, the test data are used for training, which corresponds to the popular train-test-train approach. this procedure is repeated 30 times. we used seven base learners paired with all tested drift detection methods. they are: boosting regressor (br) [6], decision trees (dt) [3], lasso regression (lr) [20], hoeffding tree regressor (htr) [13], multilayer perception (mlp) [19], bagging with meta-estimator br (b-br), bagging with meta-estimator lr (b-lr), bagging with meta-estimator dt (b-dt), bagging with meta-estimator htr (b-htr) and bagging with meta-estimator mlp (b-mlp) [5]. the experiments described here used scikit-multiflow: a multi-output streaming framework [11]. to run the experiments, we used a machine running a 64-bit operational system with 4 gb of ram and a 2.20 ghz intel(r) i5-5200u cpu. we use \(relmae= \frac{\left| y\_true-y\_pred \right| }{mean\_absolute\_error(y\_true,y\_pred)}\) to evaluate the predicted value against the actual value of the data set, indicating whether it correctly classified the instance. if relmae gets a value less than 1, we treated the learner’s prediction as correct, otherwise it was considered wrong. this value is passed to a parameterized drift detector, which will flag the example for no drift or true drift. if true drift is identified, the base learner is trained on the instance that has just arrived. if no drift is found, the instance is not used for training. scikit-multiflow provides the following concept drift detection methods: adwin, ddm, eddm, hddm_a, hddm_w, kswin and pagehinkley, which were used with their default settings for the paper. the prequential evaluation method or interleaved test-then-train method strategy is used for the evaluation of the learning models with concept drift and we performed thirty iterations [1]. we chose the mean square error (mse) for error metric. in addition, we carry out statistical tests to analyze the predictive performance of the learning models in relation to the mse error. the friedman test is applied with \(\alpha = 0.05\) [9]. we present our statistical analysis using critical difference diagrams. we initially performed experiments to identify the best parameter settings for each of the base learners. specifically, we conducted a grid search to choose hyperparameters from the base learners using the initial training data. after identifying the parameters that most influence performance we conducted the actual experiments. all models are from sklearn [15] and scikit-multiflow [11]. 4 experimental results in this section we present the results of the experiments with the concept drift detection methods. 4.1 predictive accuracy and drift identification we performed a broad base learner study together with the detectors. therefore, in addition to the detectors, we compare the base learner without updating with test data (called no partial) and updating whenever new test data arrives, regardless of detection (called partial). figure 2 shows the heat map of the mean mse value of all cd detectors and base learners across the 30 iterations with distinct seeds. errors are normalized between 0 and 1. fig. 2. heatmap of normalized mse errors full size image fig. 3. comparison of all base learner and detectors using average mse accuracies in dataset 3d_mex.hat, and average drifts points detection. full size image fig. 4. comparison of all base learner and detectors using average mse accuracies in dataset friedman#1, and average drifts points detection. full size image fig. 5. comparison of all base learner and detectors using average mse accuracies in datase friedman#3, and average drifts points detection. full size image fig. 6. comparison of all base learner and detectors using average mse accuracies in dataset mult, and average drifts points detection. full size image fig. 7. comparison of all base learner and detectors using average mse accuracies in dataset bike, and average drifts points detection. full size image fig. 8. average drifts points detection for datasets fccu1, fccu2 and, fccu3 full size image we analyzed the average points of detection for all different scenarios. figure 3 presents the mean error along the data stream for dataset 3d_mex.hat. only the lr, htr, b-lr and b-htr base learners showed lower errors as the new test data arrived, even in cases when no cd was detected by adwin, ddm, hddm_a and hddm_w. a higher number of wrong detections can lead to larger errors, because models are retrained with data from the detected new concept. additionally, due to the small number of training data of each new concept, base learners that are not able to learn incrementally may show worse performances. differently from the previous scenario, fig. 4 shows that for dataset \(friedman\#1\) most models have increasing errors along the data stream. however, base learner htr together with detector ph have decreased errors beginning with point 2000. this happened due to successful drift point detections followed by base learner updates. base learners mlp and b-mlp also showed lower errors when paired with the adwin and ph detectors. figure 5 presents results for dataset \(friedman\#3\). most base learners, except mlp and b-mlp, had lower errors at point 2000. for dataset mult, fig. 6 shows variation in the error value, with increasing errors starting at point 2000 in most cases. in such cases, the error increases because it is necessary to retrain the base learners given the new concept. thus, we can see that the base learners were not updated correctly. regarding synthetic data, the best average detection performances were obtained by eddm, kswin and ph, considering the different base learners. these indicates that in these scenarios where concept drift happens gradually, these methods are able to detect small variations in data distribution. for the bike dataset (fig. 7), only adwin and ph were not able to detect any concept drift in the data. the base learners varied significantly according in their error values. additionally, we observed that all pairs of base learners and detectors increased their errors around point 200, with worse performances coming from adwin and ph. these detectors did not adjust well to the concept changes in this dataset. due to small changes in the average errors for datasets fccu1, fccu2 and fccu3, fig. 8 only shows the average detection points. only kswin was able to find drift points. this is likely due to lack of enough data to train the base learners, which would allow better precision for the detectors. for fccu1 and fccu3, only the partial model, which always retrains with the new test data, obtained significantly lower errors. for fccu2, all models behaved similarly, with slightly decreasing errors. our results show that, in addition to the base learner and to the detector, the pace at which the drift happens and the number of instances available to update the base learner all contribute significantly to the overall performance. 4.2 statistical analysis we used critical difference diagrams and nemenyi test to compare all detectors across all 8 datasets for the different base learners. the resulting diagrams can be seen in fig. 9. this analysis shows that the detectors have statistically similar performances for all base learners. fig. 9. comparison of all regressors against each other with the nemenyi test. groups of regressors that are not significantly different (at \(\alpha \) 0:05) are connected full size image as shown in fig. 9a, all detectors had similar performances with base learner br. this even includes the model without any cd detectors (no partial), which only learns from the first training data and is only used for testing. the same happens in figs. 9b, 9f, 9i and 9g. figure 9c, on the other hand, shows that there are two distinct groups of detectors. this is mainly because the partial method, which always updates with every new block of test data in the stream, significantly outperformed every detector, except for eddm and kswin. hddm_a, hddm_w, adwin, ph and ddm had worse errors than no partial on average, although these results were not statistically significant. finally, ddm was statistically worse than partial, because their distance is larger than the critical difference. figure 9d shows that no partial, adwin, ddm, hddm_a and hdddm_w share the same average rank, which means that these detectors did not improve the performance of the base learner trained only on the original training data. the same happened with ddm with base learner mlp (9e), where ddm and no partial were outside the critical difference when compared to partial, which led to two separate performance groups. similarly, 9h shows that adwin and no partial are statistically worse than partial, resulting in two groups: the first includes methods that had statistically similar errors to partial and the second includes those that were similar to adwin and no partial. finally, fig. 9j shows that only eddm, no partial and ddm were statistically worse than partial. the only case were partial was not alone in the top rank was fig. 9i, were it shared the position with no partial, which means that there was not benefit in retraining the model. it is important to point out that it is actually a good result when a detector is statistically similar to partial, because it means that, as a result of using the detector, the base learner performed just as well as if it was always retrained, even though it only retrains when the detector identifies a drift. 5 conclusion in this paper, seven cd detectors were compared using ten base learners and eight real and synthetic datasets. the resulting python-based frameworkfootnote 1 used the sklearn and skmultiflow libraries and will be available for future use by other researchers, who will be able to extend it with other base learners, detectors and datasets. all base learners were optimized using a grid search procedure, such that they would present their best results considering the training data. average mse results indicate that ph and kswin were the best detectors for the synthetic datasets. for the bike dataset, eddm, hddm_a and hddm_w showed the best average errors. for the other real datasets, none of the detectors identified any drift points. the base learner has an important impact on cd detection. additionally, considering the approach that we used in this paper, i.e. the base learner is updated whenever cd is detected, it is useful that the base learner is able to perform incremental learning, such that it gradually adapts to new concepts. therefore, the main limitation of our work is that only one of the base learners can learn incrementally, with the other ones training from scratch with every batch of test data that was flagged as cd. another important limitation was that our real datasets did not have a ground truth for the presence of cd, thus we artificially added drift to the data, but most detectors were not able to capture that. future works include adding other incremental learners to the framework, investigating additional real datasets and analyzing the impact of different rates of change in data distribution. notes 1.https://github.com/marilia-lima/framework-detectors-concept-drift. references de almeida, r., goh, y.m., monfared, r., steiner, m.t.a., west, a.: an ensemble based on neural networks with random weights for online data stream regression. soft comput., 1–21 (2019) google scholar  baena-garcıa, m., del campo-ávila, j., fidalgo, r., bifet, a., gavalda, r., morales-bueno, r.: early drift detection method. in: fourth international workshop on knowledge discovery from data streams, vol. 6, pp. 77–86 (2006) google scholar  batra, m., agrawal, r.: comparative analysis of decision tree algorithms. in: panigrahi, b.k., hoda, m.n., sharma, v., goel, s. (eds.) nature inspired computing. aisc, vol. 652, pp. 31–36. springer, singapore (2018). https://doi.org/10.1007/978-981-10-6747-1_4 chapter  google scholar  bifet, a., gavalda, r.: learning from time-changing data with adaptive windowing. in: proceedings of the 2007 siam international conference on data mining, pp. 443–448. siam (2007) google scholar  breiman, l.: bagging predictors. mach. learn. 24(2), 123–140 (1996) math  google scholar  duffy, n., helmbold, d.: boosting methods for regression. mach. learn. 47(2), 153–200 (2002) article  google scholar  frias-blanco, i., del campo-ávila, j., ramos-jimenez, g., morales-bueno, r., ortiz-diaz, a., caballero-mota, y.: online and non-parametric drift detection methods based on hoeffding’s bounds. ieee trans. knowl. data eng. 27(3), 810–823 (2014) article  google scholar  gama, j., medas, p., castillo, g., rodrigues, p.: learning with drift detection. in: bazzan, a.l.c., labidi, s. (eds.) sbia 2004. lncs (lnai), vol. 3171, pp. 286–295. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-28645-5_29 chapter  google scholar  gonçalves jr., p.m., de carvalho santos, s.g., barros, r.s., vieira, d.c.: a comparative study on concept drift detectors. expert syst. appl. 41(18), 8144–8156 (2014) google scholar  mastelini, s.m., de leon ferreira de carvalho, a.c.p.: 2cs: correlation-guided split candidate selection in hoeffding tree regressors. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12320, pp. 337–351. springer, cham (2020). https://doi.org/10.1007/978-3-030-61380-8_23 chapter  google scholar  montiel, j., read, j., bifet, a., abdessalem, t.: scikit-multiflow: a multi-output streaming framework. j. mach. learn. res. 19(72), 1–5 (2018). http://jmlr.org/papers/v19/18-251.html oikarinen, e., tiittanen, h., henelius, a., puolamäki, k.: detecting virtual concept drift of regressors without ground truth values. data min. knowl. discov. 35(3), 726–747 (2021). https://doi.org/10.1007/s10618-021-00739-7 article  mathscinet  math  google scholar  osojnik, a., panov, p., džeroski, s.: tree-based methods for online multi-target regression. j. intell. inf. syst. 50(2), 315–339 (2017). https://doi.org/10.1007/s10844-017-0462-7 article  math  google scholar  page, e.s.: continuous inspection schemes. biometrika 41(1/2), 100–115 (1954) article  mathscinet  google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  raab, c., heusinger, m., schleif, f.m.: reactive soft prototype computing for concept drift streams. neurocomputing 416, 340–351 (2020) article  google scholar  dos santos, v.m.g., de mello, r.f., nogueira, t., rios, r.a.: quantifying temporal novelty in social networks using time-varying graphs and concept drift detection. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12320, pp. 650–664. springer, cham (2020). https://doi.org/10.1007/978-3-030-61380-8_44 chapter  google scholar  soares, s.g., araújo, r.: an on-line weighted ensemble of regressor models to handle concept drifts. eng. appl. artif. intell. 37, 392–406 (2015) article  google scholar  valença, m.: fundamentos das redes neurais: exemplos em java. olinda, pernambuco: editora livro rápido (2010) google scholar  xu, h., caramanis, c., mannor, s.: robust regression and lasso. ieee trans. inf. theory 56(7), 3561–3574 (2010) article  mathscinet  google scholar  download references author information authors and affiliations university of pernambuco, pernambuco, brazil marília lima & roberta andrade de a. fagundes federal university of paraiba, paraiba, brazil telmo silva filho authors marília limaview author publications search author on:pubmed google scholar telmo silva filhoview author publications search author on:pubmed google scholar roberta andrade de a. fagundesview author publications search author on:pubmed google scholar corresponding author correspondence to marília lima . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper lima, m., filho, t.s., de a. fagundes, r.a. (2021). a comparative study on concept drift detectors for regression. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_26 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords concept drift detection regression comparative study publish with us policies and ethics profiles telmo silva filho view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature bete: a brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper bete: a brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain conference paper first online: 12 october 2023 pp 256–267 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) bete: a brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain download book pdf download book epub lucas pavanelli  orcid: orcid.org/0000-0003-2228-79659, yohan bonescki gumiel  orcid: orcid.org/0000-0001-8239-293010, thiago ferreira  orcid: orcid.org/0000-0003-0200-36469, adriana pagano  orcid: orcid.org/0000-0002-3150-350310 & … eduardo laber  orcid: orcid.org/0000-0002-9025-833311  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 536 accesses abstract the biomedical nlp community has seen great advances in dataset development mostly for the english language, which has hindered progress in the field, as other languages are still underrepresented. this study introduces a dataset of brazilian portuguese annotated for named entity recognition and relation extraction in the healthcare domain. we compiled and annotated a corpus of health professionals’ responses to frequently asked questions in online healthcare forums on diabetes. we measured inter-annotator agreement and conducted initial experiments using up-to-date methods to recognize entities and extract relations, such as bert-based ones. data, models, and results are publicly available at https://github.com/pavalucas/bete . access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others automated annotation of electronic health records using large language models chapter © 2026 we are not ready yet: limitations of state-of-the-art disease named entity recognizers article open access 27 october 2022 leveraging deep active learning and large language models for cost-efficient categorization of user-generated content chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. adas, connectivity, e/e biological databases gene ontology machine learning predictive markers sequence annotation 1 introduction named entity recognition (ner) identifies named entities in texts; these entities can be proper names, locations, and organizations. relation extraction (re) consists in predicting whether and what predefined relation exists between two mentioned entities or if they are unrelated. in the healthcare domain, named entities include mentions relevant to the clinical context, such as treatment plans, drugs, and symptoms while relations can be, for instance, if a certain drug treats a symptom. both tasks are prevalent in machine learning methods [2] and are considered the primary step for several other natural language processing (nlp) tasks, such as question answering [20], document classification [6], and search engines [12]. consultation time is not always sufficient for health professionals to answer all the patient and their families questions, and patients may not have easy access to primary care centers [8]. hence, patients often engage in dedicated public forums to search for answers to their queries. among chronic diseases, diabetes stands out as a condition much in need of attention because it is an increasingly prevalent and severe long-term problem quickly growing worldwide, particularly in developing countries, as is the case of brazil. in 2019, it was estimated that almost half a billion of the world’s population (9.3% of the adults between 20–79 years) had diabetes [18]. further, one in every two (50.1%) persons with diabetes is unaware of or has not been diagnosed with this condition. hence, question answering (qa) systems could help relieve some of the burdens of health care. in qa, an answer to a question is found by querying a corpus of text documents. a qa system incorporates ner and re components so that entities and their relations can be detected and answers can be more efficiently found. in this study, the main goal was to annotate a corpus of texts in order to develop a framework to automatically identify healthcare-related entities and relations. to that end, medical and nutrition science students produced texts as answers to queries posted by users in diabetes-related public forums. our work is thus relevant for studies in the healthcare domain targeting specifically the general public and focusing on brazilian portuguese. the obtained dataset is expected to be used to build beteqa, a community question answering system that provides fast and precise answers to questions about diabetes mellitus posed by the lay public [5]. to the best of our knowledge, this is the first study focusing on ner and relation extraction drawing on a novel corpus of real-life questions answered by professional healthcare specialists in brazilian portuguese. 1.1 related work among named entity recognition methods, contextual word representations are employed due to their capacity for modeling complex word usage characteristics and variations across linguistic contexts [17]. a method worth highlighting is bert [7], a masked language model that benefits from transformers. these pre-trained language models are becoming the new ner paradigm because of their contextualized embeddings. furthermore, they are fine-tuned for several nlp tasks by adding an additional output layer [13]. as far as relation extraction methods, bert-based models are also dominant. soares et al. [21] proposed a model that learns relation representations directly from the text using bert-based models. considering the available bert models for brazilian portuguese, there is a multilingual version [7]; a brazilian portuguese version focused on the general domain called bertimbau [22]; and a brazilian portuguese version focused on the clinical domain named biobertpt [19]. fig. 1. annotation example from our corpus. full size image 2 methodology 2.1 corpus we searched for questions posed by users regarding health issues in several online forums. in such forums, users answer each other’s questions drawing on their beliefs and understanding of the issues, with no help or supervision of any healthcare professional. to avoid this problem, we designed a study in which answers were produced by medical and nutrition science students relying on their domain knowledge. answers were curated by expert professionals in our project who supervised students. this way, we ensured that our set of qa is reliable and can be used in a prospective qa system to be queried through a conversational agent. our corpus comprises two sets of documents made up of diabetes-related general questions with their respective answers: a first set contains 304 real online forum questions answered by medical students under the supervision of medical professionals and a second one contains 201 real online forum questions answered by nutrition science students supervised by professionals. this way, we can have a qa system to provide accurate answers about health and nutrition issues, authored by professionals in both fields. annotation setup. as an annotation tool, we used webanno [4], an open-source and intuitive software/platform. after comparing several available entity tagging tools, we found webanno to be the easiest and most efficient tool for our purposes. the system was set up on a web server, text data was uploaded, and entity/relation types were defined within the system. annotation guidelines. the annotation guidelines were created in an iterative process. a first draft was created, containing general guidelines as well as specific examples of types of entities and relations. specialists were consulted regarding annotators’ queries and their answer was used to update the guidelines, then they were tested again. besides, during the annotation process, whenever one of the annotators ran into a dubious case, this was added to the guidelines. figure 1 shows an example of a text annotated in webanno following our guidelinesfootnote 1. the annotation guidelines are publicly available for download as part of the dataset. annotation process. we recruited undergraduate students pursuing their ba degree to complete the annotation task. the students are part of empoder@, a multidisciplinary project engaging health sciences, statistics, computer science, and applied linguistics students. the project aims to empower researchers, professionals, and users of health services. annotators took part in a training session and were requested to read the guidelines and resort to the project coordinator whenever they encountered problems during annotation. two students annotated each document, and a third one performed the adjudication so that a gold-standard was obtained upon completion. annotators were presented with each text on a simple interface (see fig. 1) and using a mouse or a track pad they selected entities and dragged relationships between them. 2.2 entity and relation extraction we considered relation extraction as a multi-label classification problem in which, given a pair of entities, a label is assigned out of the relation types available. also, we decoupled relation extraction from entity recognition, so we performed re on the gold entities. the set of entity and relation types devised for our annotation was built drawing on an ontology proposed in [1]. the ontology labels diabetes mentions as “diabetestype”, and diabetes-related diseases are classed with the “complication” entity type. diabetes-related temporal expressions, clinical tests, and treatments are also addressed in the ontology. table 1 lists the 14 entity types, providing a brief explanation of each label with some examples. the 5 relations types are verbalized by the following verbs: “causes”, “diagnoses”, “has”, “prevents”, and “treats”. table 1. entities description and examples.full size table reliability. to measure dataset reliability, we computed inter-annotator agreement (iaa) considering exact matches. following the work of [3], we computed pairwise f1 score and cohen’s kappa. the former is more reliable according to various studies [9, 11]. because of the vast amount of unannotated tokens (labeled “o”), we calculated the scores without the o label, for both annotated entities and relations. table 2 shows the obtained agreement. annotated entities can be said to be fully reliable, achieving an iaa of 0.93. as regards relations, moderate agreement (0.58) was found. table 2. inter-annotator agreement for entity recognition and relation extraction.full size table table 3. dataset information.full size table table 4. entities: number of occurrences and percentage per entity type sorted in decreasing order.full size table table 5. relations: number of occurrences and percentage per relation type sorted in decreasing order.full size table 3 dataset information table 3 shows overall statistics of the whole dataset. table 4 shows the number of annotations per entity type, while table 5 covers the annotated relations. 4 experiments setup we conducted initial experiments using methods to recognize entities and extract relations. since the dataset is in brazilian portuguese, we chose deep learning models trained on multilingual and brazilian portuguese data. these models are multilingual bert (mbert), bertimbau, and the three different versions of biobertpt: biobertpt-bio, trained on portuguese biomedical texts, biobertpt-clin, trained on clinical narratives from electronic health records from brazilian hospitals, and biobertpt-all, trained in both biomedical texts and clinical narratives. regarding the training setup, we randomly divided the 505 documents into train/dev/test using the split 0.8/0.1/0.1, respectively, tuning the hyperparameters on the development set and reporting the results on the test set. to run the experiments, we used one nvidia geforce rtx 3090 gpu. as for models, we trained a baseline conditional random field (crf) for the ner task. we used a portuguese model from the python library called spacy [10] to extract a set of features. table 6 shows the used features. table 6. used crf features.full size table for bert models for the ner task, we used the adam [14] optimizer with a learning rate of 1e-5 and a maximum length of 512. moreover, we trained for 50 epochs with early stopping of 15 epochs and a batch size of 64. regarding the relation extraction task, we experimented with a baseline support vector machine (svm) model, using the one-vs-the-rest (ovr) multiclass strategy from scikit-learn [16] python library. considering bert for relation extraction, we used 7e-5 as the learning rate with adam optimizer and max length of 512, training for 11 epochs, and a batch size of 32. we considered the following metrics for evaluation: precision, recall, and f1 score. we reported the weighted average f1 score, averaging the support weighted mean per label. all metrics are considering exact matches. in addition, we computed results considering all classes and, for the best-performing model, we reported metrics for each label. 5 results table 7 shows the results for ner models. we trained baseline crf and bert-based models. the best one concerning f1 score is the biobertpt-clin model, outperforming biobertpt-all by 0.9 points. also, bertimbau did not perform well with an f1 score 5.3 points lower than the second-lowest. to evaluate the relation extraction task, we experimented using the two models: svm, as the baseline, and bert for relation extraction (bert-re) from [21], using mbert as the bert encoder. table 8 shows the result for both relation extraction models. we can observe that, although svm has higher precision than bert-re, the latter performs better overall with 17 points of f1 score difference. we also provide an in-depth analysis of the best-performing model in ner and re. table 9 shows detailed results for the entity recognition biobertpt-clin model. the method produced good scores (>80%) for the three entities with the largest number of examples: food, complication, and symptom. however, for some entities with few examples, the model was not able to perform good predictions (<60%): test, time, and set. table 10 describes an in-depth analysis of bert-re performance for each relation type. similarly to ner results, the model performs well for the relation with most examples (has) and falls short in relations with few examples: causes and prevents. table 7. experiments for entity recognition models. the best scores are highlighted in bold.full size table table 8. experiments for relation extraction models. the best scores are highlighted in bold.full size table table 9. detailed results for the best entity recognition model (biobertpt-clin).full size table table 10. detailed results for the best relation extraction model (bert-re).full size table 6 discussion in this study, we introduced a novel dataset and models for ner and re tasks, gathering answers written by health professionals about diabetes mellitus to online forum users. corpus. we found that the entities food and complication were the most annotated in the corpus. also, nutrition is a prevalent topic in online forums, as people with diabetes frequently inquire whether they can or cannot consume specific foods, such as chocolate, or alcoholic drinks. similarly, people are generally keen on finding further information on diabetes-related complications because they feel specific symptoms or want to know about frequent diabetes-related complications. regarding the main difficulties found in the process of annotating relations, despite annotators’ prior training and availability of guidelines, some entities were annotated as having a different extent in terms of words making up those entities. thus, “type 2 diabetes” and “diabetes” are two different annotations that refer to the same entity type, “diabetestype”. this ends up impacting the relation since a relation is defined by a pair of entities and a relation type, which can justify our lower agreement for relation annotation. so if two annotators annotated the same relation type holding between entities spanning different extents in words, the resulting relations did not match. further, as evidenced in [15], temporal relation extraction, which is a specific type of relation extraction, usually has lower agreement than span annotation. models. for the ner task, among the bert models, we found that biobertpt-clin, which leveraged clinical data, had superior performance compared to the other models. both biobertpt-clin and biobertpt-all were trained on clinical data and were initialized with the weights from multilingual bert, so as expected, these models achieved similar results. it is also worth noting that, although trained on brazilian portuguese data, bertimbau did not perform well. an explanation is that bertimbau was trained on brwac [23], a large corpus extracted from the web, that differs in vocabulary and context from bete medical data. considering relation extraction, the bert-based model outperforms the baseline model, showing the dominance of context-aware embeddings compared to kernel-based methods that were dominant in the past. 7 conclusion to the best of our knowledge, this is the first study yielding an annotated corpus of ner in brazilian portuguese made up of diabetes-related answers authored by domain specialists in response to questions posed by lay users. moreover, it contributes to the research field by introducing resources for a sensitive context, such as diabetes, and creating models for a low-resource language, such as brazilian portuguese. the fine-tuning of models leveraging clinical data was found to improve the results. hence, the vocabulary and the context from the clinical context boosted the model’s ability to predict entities. we plan to expand our corpus in future work, especially for the categories of entities and relations that have a small number of instances. 8 limitations among the limitations of our study is the size of our dataset, which is due to the fact that it was obtained through manual annotation thereby demanding human effort and more time to accomplish this task. another limitation is that some entity types have few instances, making our dataset slightly imbalanced. for instance, only 1/3 of our entity types have percentages of occurrence higher than 3%. hence, adding more training examples and making the dataset less imbalanced will certainly enhance our results. our dataset targets a particular domain diabetes; future experiments targeting other chronic diseases, such as cardiovascular disease, will boost our potential. additionally, in the aftermath of covid-19, there is growing uncertainty about procedures, symptoms, and treatments, with several questions being asked over social media. hence, addressing covid-19-related frequently asked questions would be a valuable contribution. notes 1.translation into english: “being overweight can lead to type 2 diabetes. therefore, intermittent fasting may be a way to prevent type 2 diabetes. intermittent fasting can also be used as a treatment for people newly diagnosed with type 1 diabetes who need to lose weight to achieve a more stable health condition; these people should be advised and monitored by an endocrinologist and a nutritionist.”. references ben abacha, a., zweigenbaum, p.: means: a medical question-answering system combining nlp techniques and semantic web technologies. inf. process. manag. 51(5), 570–594 (2015). https://doi.org/10.1016/j.ipm.2015.04.006, https://www.sciencedirect.com/science/article/pii/s0306457315000515 bose, p., srinivasan, s., sleeman, w.c., palta, j., kapoor, r., ghosh, p.: a survey on recent named entity recognition and relationship extraction techniques on clinical texts. appl. sci. 11(18) (2021). https://doi.org/10.3390/app11188319, https://www.mdpi.com/2076-3417/11/18/8319 brandsen, a., verberne, s., wansleeben, m., lambers, k.: creating a dataset for named entity recognition in the archaeology domain. in: proceedings of the twelfth language resources and evaluation conference, pp. 4573–4577. european language resources association, marseille, france (2020), https://aclanthology.org/2020.lrec-1.562 eckart de castilho, r., et al.: a web-based tool for the integrated annotation of semantic and syntactic structures. in: proceedings of the workshop on language technology resources and tools for digital humanities (lt4dh), pp. 76–84. the coling 2016 organizing committee, osaka, japan (2016), https://www.aclweb.org/anthology/w16-4011 castro ferreira, t., et al.: evaluating recognizing question entailment methods for a portuguese community question-answering system about diabetes mellitus. in: proceedings of the international conference on recent advances in natural language processing (ranlp 2021), pp. 234–243. incoma ltd., held online (2021), https://aclanthology.org/2021.ranlp-main.28 choudhary, a., arora, a.: linguistic feature based learning model for fake news detection and classification. expert syst. appl. 169, 114171 (2021) article  google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186 (2019). https://doi.org/10.18653/v1/n19-1423, https://www.aclweb.org/anthology/n19-1423 gabarron, e., et al.: social media for health promotion in diabetes: study protocol for a participatory public health intervention design. bmc health serv. res. 18(1), 414 (2018). https://doi.org/10.1186/s12913-018-3178-7 article  google scholar  grouin, c., rosset, s., zweigenbaum, p., fort, k., galibert, o., quintard, l.: proposal for an extension of traditional named entities: from guidelines to evaluation, an overview. in: proceedings of the 5th linguistic annotation workshop, pp. 92–100 (2011) google scholar  honnibal, m., montani, i.: spacy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. to appear 7(1), 411–420 (2017) google scholar  hripcsak, g., rothschild, a.s.: agreement, the f-measure, and reliability in information retrieval. j. am. med. inform. assoc. 12(3), 296–298 (2005) article  google scholar  lahav, d., et al.: a search engine for discovery of scientific challenges and directions. in: aaai (2022) google scholar  li, j., sun, a., han, j., li, c.: a survey on deep learning for named entity recognition. ieee trans. knowl. data eng. 34(1), 50–70 (2020) google scholar  loshchilov, i., hutter, f.: decoupled weight decay regularization. in: international conference on learning representations (2019). https://openreview.net/forum?id=bkg6ricqy7 nikfarjam, a., emadzadeh, e., gonzalez, g.: towards generating a patient’s timeline: extracting temporal relationships from clinical notes. j. biomed. inform. 46, s40–s47 (2013). https://doi.org/10.1016/j.jbi.2013.11.001, supplement: 2012 i2b2 nlp challenge on temporal relations in clinical data pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  peters, m.e., et al.: deep contextualized word representations. in: proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pp. 2227–2237. association for computational linguistics, new orleans, louisiana (2018). https://doi.org/10.18653/v1/n18-1202, https://aclanthology.org/n18-1202 saeedi, p., et al.: global and regional diabetes prevalence estimates for 2019 and projections for 2030 and 2045: results from the international diabetes federation diabetes atlas, 9th edition. diabetes res. clin. pract. 157, 107843 (2019). https://doi.org/10.1016/j.diabres.2019.107843 schneider, e.t.r., et al.: biobertpt a portuguese neural language model for clinical named entity recognition. in: proceedings of the 3rd clinical natural language processing workshop, pp. 65–72. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.clinicalnlp-1.7 sharma, v., kulkarni, n., pranavi, s., bayomi, g., nyberg, e., mitamura, t.: bioama: towards an end to end biomedical question answering system. in: proceedings of the bionlp 2018 workshop, pp. 109–117. association for computational linguistics, melbourne, australia (2018). https://doi.org/10.18653/v1/w18-2312, https://aclanthology.org/w18-2312 soares, l.b., fitzgerald, n., ling, j., kwiatkowski, t.: matching the blanks: distributional similarity for relation learning. in: acl 2019–57th annual meeting of the association for computational linguistics, proceedings of the conference, pp. 2895–2905 (2020). https://doi.org/10.18653/v1/p19-1279 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  wagner, j., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese (2018) google scholar  download references author information authors and affiliations aixplain inc., los gatos, usa lucas pavanelli & thiago ferreira universidade federal de minas gerais, belo horizonte, brazil yohan bonescki gumiel & adriana pagano pontifícia universidade católica do rio de janeiro (puc-rj), rio de janeiro, brazil eduardo laber authors lucas pavanelliview author publications search author on:pubmed google scholar yohan bonescki gumielview author publications search author on:pubmed google scholar thiago ferreiraview author publications search author on:pubmed google scholar adriana paganoview author publications search author on:pubmed google scholar eduardo laberview author publications search author on:pubmed google scholar corresponding author correspondence to lucas pavanelli . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi ethics declarations ethical statement our study fully complies with ethical standards and did not require any submission to ethical boards, since no data collection with human subjects was carried out. our dataset was created by our team and contains texts drafted by medical students under the supervision of healthcare professionals, all of whom are research members in our project. rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pavanelli, l., gumiel, y.b., ferreira, t., pagano, a., laber, e. (2023). bete: a brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_17 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords dataset named entity recognition relation extraction publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a novel genetic algorithm approach for discriminative subspace optimization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a novel genetic algorithm approach for discriminative subspace optimization conference paper first online: 30 january 2025 pp 64–79 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a novel genetic algorithm approach for discriminative subspace optimization download book pdf download book epub bernardo b. gatto9, marco a. f. mollinetti9, eulanda m. dos santos10, alessandro l. koerich11 & … waldir s. da silva junior12  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 482 accesses abstract image set representation by subspace methods has shown to be effective for several image processing tasks, such as classifying multiple images and videos. a subspace exploits the geometrical structure in which images are distributed, representing the image set with a fixed dimension giving more statistical robustness to input noise and compactness to the images. the mutual subspace method (msm) and its extensions, the orthogonal mutual subspace method (omsm), and the generalized difference subspace (gds) are the most prominent subspace methods employed. however, these methods require solving a nonlinear optimization which lacks a closed-form solution. in this paper, we present a metaheuristic-based approach for discriminative subspace optimization. we develop a genetic algorithm (ga) for integrating omsm and gds discriminative subspaces. the initialization strategy and the genetic operators of the ga provide quality of objective function value of solutions and preserve their feasibility without any extra repair step. we validated our approach on four object recognition datasets. results show that our optimization method outperforms related methods in accuracy and highlights the use of evolutionary algorithms for subspace optimization. code: https://github.com/bernardo-gatto/evolving_manifold. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others sub-gan: an unsupervised generative model via subspaces chapter © 2018 a genetic algorithm based technique for outlier detection with fast convergence chapter © 2018 3d object recognition with enhanced grassmann discriminant analysis chapter © 2017 1 introduction the problem of classifying image sets has been widely investigated in computer vision and supports several applications by handling, learning, and classifying data from multiple view cameras. for instance, in robot vision, where a data stream is available [1, 12, 20, 22]. in this context, a pattern set is a collection of images (or feature vectors) of the same object or event. this set can be unordered where the time stamp of the collected images is not relevant. the images can also be ordered when the timestamp is semantically meaningful. applications of subspaces for pattern-set learning frequently use the mutual subspace method (msm) [16, 23]. these solutions are employed to solve gesture and action recognition problems, where video clips are described as subspaces, in which each subspace is computed from one of the pattern sets. despite its advantages, traditional subspace methods like msm lack discriminative feature extraction, which is critical for efficient pattern set modeling. a useful pattern-set model requires robustness to corrupt data; some images may contain noise, occluded targets, or dropped patches. the model must also correctly handle a variable set size, maintaining its computational complexity. the orthogonal mutual subspace method (omsm) [21] was developed to extract discriminative features using the fukunaga-koontz transformation (fkt). its formulation has been applied to image-set modeling for solving a range of problems, such as gesture and action classification. recently, a variant of the subspace method called generalized difference subspace (gds) has been developed [4]. the relationship between the patterns is considered by employing the concept of the generalized difference between the subspaces. this formulation provides a novel discriminative transformation, where the projected subspaces produce higher recognition results than conventional subspace-based methods. despite their excellent performance, omsm and gds lack closed-form solutions and cannot fully achieve their capabilities. our work introduces a significant advancement by using a genetic algorithm (ga) to optimize the discriminative subspaces. this approach not only enhances the discriminative power of omsm and gds but also offers a more rigorous solution to the optimization challenges in subspace methods. we optimize omsm and gds spaces to tackle these challenges. we develop a ga that efficiently handles subspaces by modifying its initialization, evaluation, selection, and crossover operators, which is complex given ga’s typical use in simpler optimization problems. our ga-generated discriminative space surpasses omsm and gds in accuracy, showcasing ga’s versatility in subspace optimization. this innovation applies to traditional subspace methods, creating a compact model that enhances their inherent benefits. this paper continues as follows: sect. 2 describes the related work on subspace-related methods. section 3 introduces the proposed optimization solution, and sect. 4 shows the experimental results. finally, sect. 5 presents the final remarks and future directions. code developed for this research can be found in the following repository: https://github.com/bernardo-gatto/evolving_manifold (fig. 1). fig. 1. these plots illustrate the improvement in class separability achieved by different subspace optimization methods. from left to right, the subplots transition from the msm with a 53% overlap between classes to the ga after 200 iterations (ga-200) with a reduced overlap of 5%, indicating a progressive enhancement in class distinctness.. full size image 2 related work 2.1 subspace method it has been verified that using multiple images can significantly improve the performance of representing complex shape objects. the msm [16] is a recognition method that efficiently handles multiple images. msm is a method for classifying multiple image sets where a set of images is represented by a linear subspace generated by applying non-centered principal component analysis (pca). the similarity between different image sets is calculated using the canonical angles between the subspaces, and this similarity is used to classify sets of input images. while msm plays an essential function in subspace-based methods, it cannot produce good recognition results since it does not consider the relationship between the sets of distinct classes. the omsm  [5] has been developed to improve the discriminative ability of subspace-based methods. in omsm, class subspaces are orthogonalized using fukunaga-koontz’s framework before computing the canonical angles. this transformation emphasizes the differences between the classes and significantly improves the recognition performance of msm. some of the applications of omsm include face recognition and multiple view object recognition [6, 7]. another variant of the subspace method was introduced in [4] named gds, where the pattern sets are also represented as subspaces. the relationship between the patterns is taken into consideration by employing the concept of the generalized difference between the subspaces. this algebraic formulation provides a novel discriminative transformation, where the projected subspaces produce higher recognition results than conventional subspace-based methods. 2.2 evolutionary and genetic algorithms research on the theory and applications of evolutionary algorithms (eas) has been consistent due to the increasing number of complex nonlinear problems unsolvable by conventional gradient-based techniques [2]. accordingly, eas present several advantages, such as handling limiters, explaining and repeating the search process, understanding input-output causality within a solution, and introducing corrective actions. eas excel in various problem domains, notably in optimization, where they can uncover solutions beyond human intuition. characterized as iterative refinement algorithms, they evolve a set of potential solutions using replication, reproduction, and selection, all guided stochastically towards a specific goal. since its first relevant publication [13], gas are considered a well-known population-based ea designed to solve optimization problems where exact methods fail [9]. their mechanisms are grounded in darwinian natural selection, where fitter individuals produce more offspring, while less fit individuals are eliminated if they don’t adapt or mutate to improve their capabilities [17]. a canonical ga comprises these key genetic operators: 1) replication, which creates new solutions either by generating entirely new potential solutions or by altering specific existing ones; 2) reproduction, a mechanism to enhance the diversity of the solution pool, is achieved through two methods: recombination (crossover) and mutation; 3) selection, a process that identifies and retains solutions meeting certain criteria for inclusion in the subsequent iteration. it is important to acknowledge the diverse range of optimization strategies discussed in the literature [3, 24]. for example, bayesian optimization typically utilizes a cost function reliant on simulations, which often demands substantial computational resources and lacks a surrogate model. in contrast, ea offers an alternative approach, focusing on the iterative refinement of candidate solutions. this method is advantageous as it requires fewer cost function evaluations compared to the bayesian approach, which relies on extensive sampling to create a pool of potential solutions. the development of the proposed method is encouraged by the application of gas to the well-known classifier in a non-euclidean space. such a space presents no trivial operations, such as sum or gradient, which are easily achieved in euclidean spaces. 3 proposed method fig. 2. framework of the proposed optimization solution. given a collection of labeled pattern sets, the task lies in 1) learning a subspace representation for each set and then 2) estimating a discriminative space where discriminative information is attainable. our contribution lies in optimizing the discriminative space using ga, where a population of subspace candidates is evaluated and updated until a user-specific criterion is met. full size image 3.1 problem formulation subspaces usually represent pattern sets to reduce computational complexity, achieving the immediate advantage of allowing parallel processing. besides, subspace representation permits the examination of correlations among the various factors inherent in each pattern set. in a classification problem, \(\boldsymbol{p}=\{p_{i}\}_{i=1}^{n}\) denotes the set of all subspaces spanned by \(\boldsymbol{u} = \{u_{i}\}_{i=1}^{n}\). we can then develop a projection matrix d that acts on the elements of \(\boldsymbol{p}\) to extract discriminative information. in traditional gds, this procedure is performed by removing overlapping components that represent the intersection between subspaces. differently, omsm weights the discriminative space, encouraging the decorrelation of the overlapping patterns. figure 2 shows the pattern-set classification pipeline and the proposed optimization solution. given a collection of labeled pattern sets, the task lies in estimating subspace representations for each set and then learning a discriminative space, where the subspaces are projected, and improved patterns are obtained. our contribution lies in optimizing the discriminative space using our genetic algorithm. an initial population derived from omsm and gds subspaces is successively evaluated and updated until a specific criterion is met. 3.2 brief review on msm to represent a pattern-set by a subspace, we use orthonormal basis vectors to describe them compactly. given a feature matrix \(x=[~x_1{\,}|{\,}x_2{\,}|{\,}\ldots {\,}|{\,}x_{m-1}{\,}|{\,}x_{m}~]\), where \(x_j\) is a feature vector, possibly obtained through the vectorization of an image. then, we can conduct a decomposition to gather knowledge of the geometric structure of x. the singular value decomposition (svd) [14] produces a set of eigenvectors u and a set of eigenvalues \(\varlambda \), where each column vector in u represents an axis and each value in \(\varlambda \) describes how important this axis is in terms of reconstruction. also, \(\varlambda \) describes how much the vectors in x are correlated. the svd of x is $$\begin{aligned} {x}{x}^{\top } = {u}{\varlambda }{\,}{u}^{\top }~. \end{aligned}$$ (1) each column of u is a singular vector of \({x}{x}^{\top }\), and the main diagonal of \(\varlambda \) presents the singular values in descending order. the analysis of \(\varlambda \) is helpful in various problems, such as dimensionality reduction, signal filtering, and feature extraction. by analyzing the influence of each eigenvector, it is possible to select a small set by removing all but the top k eigenvalues in the diagonal of \({\varlambda }\). the following criteria can be used to obtain the compactness ratio: $$\begin{aligned} \mu (k) \le {\sum _{j=1}^{k}(\mathbf {\lambda }_j)}/{\sum _{j=1}^{m^\prime }(\mathbf {\lambda }_j)}~. \end{aligned}$$ (2) in eq. (2), k is the number of the selected eigenvectors which spans a subspace, \({\lambda }_j\) corresponds to the j-th eigenvalue of \({x}{x}^{\top }\). then, \(m^\prime = \text {rank}\left( {x}{x}^{\top }\right) \). it is useful to set k as small as possible to achieve a minimum number of orthonormal basis vectors, maintaining low memory requirements. we employ the average of the canonical angles to compare the subspaces. a practical technique for computing the canonical angles between two subspaces p and q is calculating the eigenvalues of the product of their basis vectors. given \(u_{p}\) and \(u_{q}\), which span the subspaces p and q, eq. (3) computes the canonical correlations between p and q: $$\begin{aligned} u\,_{p}^{\top } u\,_{q} = u \varsigma v^{\top }, \end{aligned}$$ (3) where the eigenvalues matrix \(\varsigma \) provides the canonical correlations between the principal angles of \(u_{p}\) and \(u_{q}\) and can be used to compute the canonical angles, since \(\varsigma = \textrm{diag}(\lambda _1, \lambda _2, \ldots , \lambda _{k})\). the canonical angles \(\{\theta _j\}_{j=1}^{k}\) can then be computed by using the inverse cosine of \(\varsigma \), as \(\{\theta _j=\cos ^{-1}(\lambda _j) \}_{j=1}^{k}\). finally, the average canonical angle \(\bar{\theta }\) between p and q is defined as \(\bar{\theta }= \frac{1}{k^\prime } \sum _{j=1}^{k^\prime } \theta _j\), where \(k^\prime \le \textrm{min}(\textrm{rank}(p),~\textrm{rank}(q))\). once the average of the canonical angles is obtained between the training and the test subspaces, we can employ the nearest neighbour algorithm (1-nn) to classify the test subspaces. 3.3 conventional discriminative subspaces once equipped with all the subspaces (using eqs. (1) and (2)), omsm employs the fkt to generate the matrix \(d_o\) that can decorrelate the subspaces. each set of basis vectors \({u}_i\) spans a reference subspace \({p}_i\), where its compactness ratio is empirically defined by employing eq. (2). the method to generate the matrix \(d_o\) that efficiently decorrelates the subspaces is explained. we compute the total projection matrix: $$\begin{aligned} {g} = \sum _{i=1}^n {u}_i {u}_i^{\top }, \end{aligned}$$ (4) then we decompose the total projection matrix g: $$\begin{aligned} d_o = {\varlambda }^{-1/2} \ {b}^{\top }, \end{aligned}$$ (5) where b is the set of orthonormal eigenvectors corresponding to the largest eigenvalues of g, and \({\varlambda }\) is a diagonal matrix with the k-th highest eigenvalue of the matrix g as the k-th diagonal component. differently, \(d_g\) produced by gds can be computed by discarding the first d eigenvectors of b as follows: \(d_g = b \setminus a\), where \(a =\{u_{k}\}_{k=1}^{d}\) and \(\setminus \) is the relative complement (i.e., removes all the elements of a from b). in practice, omsm and gds are very similar. their differences arise from the fact that the eigenvectors of g are either weighted or discarded, suggesting that an optimization strategy should be implemented for weighting or discarding eigenvectors. since this problem can be analyzed from a feature selection point of view and, according to [10], it is an np-hard problem, we argue that using an ea for combining both \(d_o\) and \(d_g\) discriminative spaces is a viable solution. in this scenario, a suitable objective function \(\boldsymbol{f}\) must be an indicator of how much a discriminative space can discriminate different groups. we discuss the chosen objective function in the following section. 3.4 proposed algorithm we propose an ea in the likes of the ga of holland [13] with all genetic operators but the mutation operator. algorithm 1 has the following components: initializepopulation, evaluate, selection and crossover. the overall steps of the ga are shown in algorithm 1. the encoding of solutions in our ga is in the form of a matrix, each solution encodes the basis vectors of the discriminative subspaces \(d_o\) and \(d_g\). this representation allows the ga to effectively combine and evolve them over iterations. it can be thought of as a concatenated or structured representation of these basis vectors, where each vector undergoes optimization through genetic operations. this approach ensures that the evolutionary process refines the subspaces for optimal discriminative power in classification tasks. algorithm 1 . genetic algorithm for the subspace optimization problem full size image in summary, our proposed ga starts by initializing a population of solutions \(\boldsymbol{d}_0\) of size n. in each iteration, the algorithm evaluates the fitness of the current population \(\boldsymbol{d}_i\) using an objective function \(\boldsymbol{f}\) based on the fisher score. based on these fitness values, a selection process \(\textsf {selection}\) is carried out to choose a subset of the current population. this subset then undergoes a crossover operation \(\textsf {crossover}\), producing a new set of solutions \(\bar{\boldsymbol{d}}_i\). the new generation for the next iteration \(\boldsymbol{d}_{i+1}\) is formed by combining the selected solutions and the offspring from the crossover. the process repeats until the maximum number of iterations max is reached. finally, the algorithm returns the best solution from the final population as determined by the objective function \(\boldsymbol{f}\). part of omsm’s traditional optimization approach performs the weighting of eigenvectors associated with the lowest eigenvalues. intriguingly, the optimization process of gds is performed by eliminating eigenvectors associated with the highest eigenvalues, suggesting a complementary relationship exists between both optimization strategies. this greedy strategy produces good results; however, it does not ensure that the optimal discriminative space is the one obtained by the method. besides, some eigenvectors associated with the middle eigenvalues are never removed or evaluated, and thus, this greedy strategy may lead to weak discriminative spaces. it is worth mentioning that the removal and concatenation strategy employed in this work may produce discriminative spaces of different dimensions (i.e., different subspace dimensions), which is not exactly a problem in subspace analysis. the number of basis vectors exhibited by each element is expected to be modified according to the dataset complexity. we now detail each step as follows. initializepopulation: the initialization step ensures that the initial population \(\boldsymbol{d}_0\) contains \(d_o\) and \(d_g\) discriminative spaces, their concatenations, and random variants with randomly selected dimensions and their respective concatenations. more precisely, the basis vectors of \(d_o\) and \(d_g\) can be concatenated to, after an orthogonalization process, produce a novel discriminative space, preserving the capabilities of omsm and gds. we create a completely unexplored population by randomly truncating and concatenating \(d_o\) and \(d_g\). this initialization strategy has two immediate advantages: 1) we ensure that the produced solutions will not be worse than the ones provided by \(d_o\) and \(d_g\) discriminative spaces and, 2) we start from a good enough population, which may guide solutions to converge to a local optimum faster than purely random initialization. our motivation assumes that a random initialization would be challenging to achieve the same performance as the warm start provided by omsm and gds. besides, the probability of discovering better discriminative spaces in the neighborhood space is higher once we start from promising ones. evaluate: as the objective function \(\boldsymbol{f}\) of the ga, we selected a score reflective of each discriminative subspace’s efficacy in classification metrics. these metrics can encompass accuracy, f1 score, and mean squared error. for our implementation, the fisher score was chosen due to its straightforward interpretability and effectiveness in quantifying the discriminative power of subspaces. the fisher score measures the separation between different classes in the feature space, making it a suitable metric for optimizing subspaces in classification tasks [11]: $$\begin{aligned} \boldsymbol{f} = \text {trace}\left( (s_b)(s_w + \epsilon i)^{-1}\right) , \end{aligned}$$ (6) where \(s_b\) and \(s_w\) are the between-class scatter matrix and within-class scatter matrix, respectively, measuring the dispersion of the subspaces. here, \(\epsilon \) is a positive regularization parameter, and i is the identity matrix. the fisher score aims to maximize the between-class and within-class variance ratio, ensuring that the evaluated subspace is a robust discriminative model. selection: the selection process employs a rank-based approach. specifically, it involves arranging the discriminative spaces according to their fisher scores in decreasing order. from this ranked list, the top \(p\%\) of spaces, those with the highest scores, are selected for inclusion in the crossover pool. this method ensures that the most promising solutions are carried forward, encouraging the generation of increasingly effective discriminative spaces in subsequent iterations. crossover: in the crossover phase, two elements (discriminative spaces) are randomly selected from the pool created in the selection step. these elements undergo crossover, and a one-point crossover operation is performed. a cut point \(c\%\) is randomly determined, and each offspring solution is composed of eigenvectors from both parents, split at this cut point. for instance, let \(d_p = \{\phi _{(i)}\}_{i=1}^{n_p}\) and \(d_q = \{\psi _{(i)}\}_{i=1}^{n_q}\) be two selected parent solutions. their crossover operation would result in two offspring solutions \(d_{p}^{~\prime }\) and \(d_{q}^{~\prime }\), formulated as follows: \( d_{p}^{~\prime } = \{\phi _{(1)},~\ldots ,~\phi _{(c)},~ \psi _{(c+1)},~\ldots ,~\psi _{(n_q)}\}\), \( d_{q}^{~\prime } = \{\psi _{(1)},~\ldots ,~\psi _{(c)},~ \phi _{(c +1)},~\ldots ,~\phi _{(n_p)}\}\). following the progress of the algorithm 1, the selected population \(\boldsymbol{s}_i\) is merged with \(\bar{\boldsymbol{d}}_i\) to compose the next population set; this can be seen as an update process that follows a measure of the elitism of solutions. evaluation, selection, and crossover are then performed until a predefined number of iterations is achieved. the proposed algorithm presents the main elements of a ga; however, its application is in a non-euclidean space. the provided mathematical formulations employed for orthogonalization, basis vector concatenation, and projections are the foundations for discriminative subspace optimization employing ea. the following section is focused on applying the presented concepts for pattern set classification. we employ subspace-based methods for representing pattern sets and optimize a discriminative subspace using strategies motivated by the ga presented in this section. 4 experimental results we evaluate the proposed ga to demonstrate its advantages over other pattern-set representation and classification methods. first, we describe each dataset and our experimental protocol. next, we investigate the approach for initial population estimation. we then compare the proposed approach with existing methods for the same task. lastly, we provide concluding remarks on our results. eth-80 is a dataset designed for object recognition. it contains images of 8 object categories, each with 10 subcategories in 41 orientations, totaling 410 images per category and 3280 images overall. we resized the images to \(64 \times 64\) pixels and extracted grayscale information. for our analysis, we used the images with backgrounds. for a comprehensive classification task, we used the aloi dataset. aloi is a large dataset of general objects with about 110 images per object, considering different illumination angles, colors, and viewing angles. we used the first 500 object instances. all images were segmented from the background, and we classified them into one of the 500 objects using a 10-fold cross-validation scheme. we also used the coil-20/100 datasets. coil-20 contains 20 objects, with 72 images captured for each using a turntable, resized to \(128 \times 128\) pixels. similarly, coil-100 was obtained with more objects. we split the samples into five groups, randomly selecting one for training and using the rest for testing. in the experiments, descriptive statistics are the mean accuracy, standard deviation, and best result obtained from the runs of each experiment using pre-established random seeds. chosen parameters for the proposed ga for all experiments were: number of iterations as 200; population size of 200 solutions; crossover rate c as 0.5; and populational elitism p of the selection step as \(10\%\). the experiment reported in this paper was run on a unix-like pc equipped with a core i9 5.00ghz with 16 gb ram written in python. 4.1 evaluating the initialization strategy this experiment analyzes the adopted initialization strategy on the eth-80 dataset. we utilized an initialization strategy where the elements are assigned to omsm and gds discriminative spaces and some variants based on random subspace truncation. two elements are assigned to the \(d_o\) and \(d_g\); then, random truncation is performed on either \(d_o\) or \(d_g\) before concatenation. this strategy not only creates new discriminative subspaces but also ensures that a vast space is explored and that the results obtained will never be worse than the ones provided by omsm and gds. fig. 3. accuracy from 1000 runs of the presented initialization strategy. full size image figure 3 shows the accuracy results of 1000 runs of the presented initialization strategy. we see that the initialization alone presents attractive solutions. for instance, some elements deliver accuracy higher than omsm and gds. however, some display very low accuracy, which may be due to selecting unsuitable eigenvectors. if discriminative eigenvectors are discarded, the learning algorithm accuracy may sharply drop. 4.2 analysing distance matrices in this experiment, we analyze the distance matrices produced by omsm, gds, and our ga. the objective is to understand if the optimized discriminative subspace presents advantages over omsm and gds w.r.t. feature extraction capabilities. also, the distance matrices may provide information concerning their discriminative ability, such as the intra-class and inter-class separability. for this experiment, each algorithm was run for 200 iterations. figure 4 displays the distance matrices generated by omsm, gds, and the ga on the eth-80 dataset. fig. 4. distance matrices of omsm, gds, and the proposed ga, respectively. full size image the proximity matrix produced by omsm shows moderately high similarity even when subspaces describe different classes, indicating that the discriminative subspace may reveal insufficient information for classification. in contrast, the proximity matrix produced by gds shows lower similarity, revealing moderately good discriminative information. this demonstrates the effectiveness of the gds discriminative space, as it relies on an analytical solution. differently, the similarity between different classes is very low when our optimization method is employed, represented by the dark structures on the matrix. this suggests that the presented optimization technique based on the ga produces a discriminative space capable of separating subspaces of distinct classes while keeping together the same category. 4.3 comparison with related methods table 1. study of performances for the proposed optimization method (left) and comparison with omsm and gds (right).full size table this experiment evaluates the proposed ga for discriminative subspace optimization using the eth-80, aloi, coil-20, and coil-100 datasets. we compare the proposed method with msm, omsm, and gds. the objective here is to understand whether the optimization strategy developed in this work can produce better results on diverse datasets. it is worth mentioning that more sophisticated manifold learning algorithms exist in the literature. however, we narrowed down the scope of our evaluation to linear methods since our optimization strategy was developed to support linear subspaces. a comprehensive summary of the obtained results is shown in table 1. we see from the results that the ga achieved its highest accuracy on the coil-20 and eth-80 datasets, while the results produced on the aloi dataset are the lowest. these results may indicate that our ga may benefit from a larger initial population when more complex datasets, such as aloi, are being evaluated. in this experiment, we present the best results obtained from 100 runs with varying random seeds, where each run is executed for 500 iterations. table 1 also displays the results achieved by omsm, gds, and the ga on all the employed datasets. the presented method exhibited the lowest accuracy gain on the aloi dataset compared to the gain obtained on other datasets. this suggests that the ga encountered difficulty when the classification problem’s number of classes was high. the highest improvement gain was obtained on the eth-80 dataset. these results may have been achieved because the eth-80 is the smallest dataset among those evaluated, which can benefit all methods based on linear subspaces. furthermore, since the ga does not implement a regularization technique, the provided results on more complex datasets may overfit. besides, omsm and gds presented competitive results despite implementing no optimization approach, indicating their usefulness when optimization is not practical. 4.4 visualizing the discriminative spaces this section presents a visualization of the discriminative spaces derived from the eth-80 dataset after training. the dimensions of the spaces are as follows: the ga subspace comprises 196 eigenvectors, the omsm subspace contains 83 eigenvectors, and the gds subspace encompasses 222 eigenvectors. figures 5, 6 and 7 display the first 10 and last 10 eigenvectors of the omsm, gds, and ga discriminative spaces, reshaped into \(15 \times 15\) matrices. upon analyzing the similarity of eigenvectors across these subspaces, we observe no overlap between gds and omsm, which implies that the two spaces are capturing entirely distinct features of the dataset, confirming our previous hypothesis regarding the complementary nature of these spaces. in contrast, there is a significant overlap between gds and ga, with 163 eigenvectors being shared. this suggests that ga has a strong affinity with the gds subspace, potentially indicating that ga is preserving a substantial amount of the detailed features captured by gds. furthermore, ga discriminative space shares 33 eigenvectors with omsm, which, although fewer in number, is a significant share considering the omsm’s dimension. the omsm subspace, being more compact, may focus on the most salient features of the dataset, which are sufficiently representative of the specific discriminative tasks it’s designed for. on the other hand, the gds subspace, with its larger number of eigenvectors, is likely to preserve more intricate details of the data, which could be advantageous for tasks requiring a high level of granularity. the ga subspace’s ability to share eigenvectors with both omsm and gds indicates its capability to exploit both the compact, representative nature of omsm and the detailed, nuanced capture of gds. in quantitative terms, ga incorporates approximately 83.16% of gds and 39.75% of omsm eigenvectors, reflecting its comprehensive nature in encapsulating the diversity of features within the dataset. this makes the ga subspace potentially versatile and suitable for a wide range of applications that may require either a broad overview or a detailed analysis of the data’s characteristics. fig. 5. eigenvector visualization of the omsm subspace. full size image fig. 6. eigenvector visualization of the gds subspace. full size image fig. 7. eigenvector visualization of the ga subspace. full size image in this section, we evaluated our proposed optimization approach regarding the initialization strategy, its distance matrix, accuracy on various datasets, and the visual characteristics of the eigenvectors. we can describe two main findings from the obtained results: 1) the employed initialization strategy frequently shows advantages over conventional methods, suggesting that the proper use of omsm and gds discriminative subspaces improves the classification accuracy; 2) the ga presents discriminative subspaces capable of achieving even higher classification results by iteratively selecting merged subspaces of random dimensions. these findings not only validate the robustness of our proposed approach but also highlight the potential of combining discriminative spaces to achieve superior performance in classification tasks. 5 final remarks and future directions this paper presents a genetic algorithm (ga) for optimizing discriminative subspaces to enhance feature extraction in object classification, utilizing evolution strategy (es) principles. the ga employs initial population generation, evaluation, and selection, using the fisher score for fitness assessment. crossovers among randomly selected elements create new populations, refining solutions with each iteration based on fisher scores. evaluated on object recognition tasks, the method demonstrates high classification performance across various datasets, effectively capturing high-dimensional data and uncovering subspace structures. future directions include advanced evaluation methods and exploring kernel approaches for nonlinear data distributions. our solutions are applicable in gesture and action recognition, adapting to tasks using basis vectors for data representation, suitable for subspaces of any pattern-set nature. we also plan to develop mutation techniques for subspaces to further improve ga performance and explore additional metaheuristics for pattern set optimization problems. references chen, z., xu, t., wu, x.j., wang, r., kittler, j.: hybrid riemannian graph-embedding metric learning for image set classification. ieee transactions on big data (2021) google scholar  daham, h.a., mohammed, h.j.: an evolutionary algorithm approach for vehicle routing problems with backhauls. materials today: proceedings (2021) google scholar  eltaeib, t., mahmood, a.: differential evolution: a survey and analysis. appl. sci. 8(10), 1945 (2018) article  math  google scholar  fukui, k., maki, a.: difference subspace and its generalization for subspace-based methods. ieee trans. pattern anal. mach. intell. 37(11), 2164–2177 (2015) article  math  google scholar  fukui, k., yamaguchi, o.: the kernel orthogonal mutual subspace method and its application to 3d object recognition. in: asian conference on computer vision, pp. 467–476. springer (2007). https://doi.org/10.1007/978-3-540-79043-3_8 gatto, b.b., dos santos, e.m., fukui, k., júnior, w.s., dos santos, k.v.: fukunaga-koontz convolutional network with applications on character classification. neural process. lett. 52(1), 443–465 (2020) article  math  google scholar  gatto, b.b., souza, l.s., dos santos, e.m., fukui, k., s. júnior, w.s., dos santos, k.v.: a semi-supervised convolutional neural network based on subspace representation for image classification. eurasip j. image video process. 2020(1), 1–21 (2020). https://doi.org/10.1186/s13640-020-00507-5 article  math  google scholar  geusebroek, j.m., burghouts, g.j., smeulders, a.w.: the amsterdam library of object images. int. j. comput. vision 61(1), 103–112 (2005) article  math  google scholar  goldberg, d.e.: genetic algorithms in search, optimization, and machine learning. addison-wesley publishing company (1989) google scholar  grzegorowski, m., slezak, d.: on resilient feature selection: computational foundations of rc-reducts. inf. sci. 499, 25–44 (2019) article  math  google scholar  gu, q., li, z., han, j.: generalized fisher score for feature selection. arxiv preprint arxiv:1202.3725 (2012) guo, j., sun, y., gao, j., hu, y., yin, b.: low rank representation on product grassmann manifolds for multi-view subspace clustering. in: 2020 25th international conference on pattern recognition (icpr), pp. 907–914. ieee (2021) google scholar  holland, j.h.: genetic algorithms. sci. american 267(1), 66–73 (1992) google scholar  kolda, t.g., bader, b.w.: tensor decompositions and applications. siam rev. 51(3), 455–500 (2009) google scholar  leibe, b., schiele, b.: analyzing appearance and contour based methods for object categorization. in: 2003 ieee computer society conference on computer vision and pattern recognition, 2003. proceedings. vol. 2, pp. ii–409. ieee (2003) google scholar  maeda, k.i.: from the subspace methods to the mutual subspace method. in: computer vision, pp. 135–156. springer (2010). https://doi.org/10.1007/978-3-642-12848-6_5 mitchell, m., taylor, c.e.: evolutionary computation: an overview. annu. rev. ecol. syst. 20, 593–616 (1999) article  math  google scholar  nene, s.a., nayar, s.k., murase, h.: columbia object image library (coil-20. tech. rep. (1996) google scholar  nene, s.a., nayar, s.k., murase, h., et al.: columbia object image library (coil-100) (1996) google scholar  ranta, r., le cam, s., chaudet, b., tyvaert, l., maillard, l., colnat-coulbois, s., louis-dorr, v.: approximate canonical correlation analysis for common/specific subspace decompositions. biomed. signal process. control 68, 102780 (2021) article  math  google scholar  sakano, h.: a brief history of the subspace methods. in: asian conference on computer vision, pp. 434–435. springer (2010). https://doi.org/10.1007/978-3-642-22819-3_44 shafait, f., et al.: fish identification from videos captured in uncontrolled underwater environments. ices j. mar. sci. 73(10), 2737–2746 (2016) article  math  google scholar  tan, h., gao, y., ma, z.: regularized constraint subspace based method for image set classification. pattern recogn. 76, 434–448 (2018) article  math  google scholar  wu, g., mallipeddi, r., suganthan, p.n.: ensemble strategies for population-based optimization algorithms-a survey. swarm evol. comput. 44, 695–711 (2019) article  math  google scholar  download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes-proex) finance code 001. this work was partially supported by amazonas state research support foundation fapeam through the pdpg/capes project. author information authors and affiliations frontier technology division, mti ltd., tokyo, japan bernardo b. gatto & marco a. f. mollinetti federal university of amazonas, institute of computing, manaus, brazil eulanda m. dos santos école de technologie supérieure, université du québec, québec, canada alessandro l. koerich department of electrical engineering, federal university of amazonas, manaus, brazil waldir s. da silva junior authors bernardo b. gattoview author publications search author on:pubmed google scholar marco a. f. mollinettiview author publications search author on:pubmed google scholar eulanda m. dos santosview author publications search author on:pubmed google scholar alessandro l. koerichview author publications search author on:pubmed google scholar waldir s. da silva juniorview author publications search author on:pubmed google scholar corresponding author correspondence to bernardo b. gatto . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper gatto, b.b., mollinetti, m.a.f., dos santos, e.m., koerich, a.l., da silva junior, w.s. (2025). a novel genetic algorithm approach for discriminative subspace optimization. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_5 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords subspace representation genetic algorithm discriminative learning publish with us policies and ethics profiles alessandro l. koerich view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature sabiá: portuguese large language models | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper sabiá: portuguese large language models conference paper first online: 12 october 2023 pp 226–240 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) sabiá: portuguese large language models download book pdf download book epub ramon pires  orcid: orcid.org/0000-0002-0023-19719, hugo abonizio  orcid: orcid.org/0000-0001-5208-02909, thales sales almeida  orcid: orcid.org/0009-0006-9568-93319 & … rodrigo nogueira  orcid: orcid.org/0000-0002-2600-60359  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 724 accesses 59 citations 109 altmetric abstract as the capabilities of language models continue to advance, it is conceivable that “one-size-fits-all” model will remain as the main paradigm. for instance, given the vast number of languages worldwide, many of which are low-resource, the prevalent practice is to pretrain a single model on multiple languages. in this paper, we add to the growing body of evidence that challenges this practice, demonstrating that monolingual pretraining on the target language significantly improves models already extensively trained on diverse corpora. more specifically, we further pretrain gpt-j and llama models on portuguese texts using 3% or less of their original pretraining budget. few-shot evaluations on poeta, a suite of 14 portuguese datasets, reveal that our models outperform english-centric and multilingual counterparts by a significant margin. our best model, sabiá-65b, performs on par with gpt-3.5-turbo. by evaluating on datasets originally conceived in the target language as well as translated ones, we study the impact of language-specific pretraining in terms of 1) capturing linguistic nuances and structures inherent to the target language, and 2) enriching the model’s knowledge about a domain or culture. our results indicate that most benefits stem from the domain-specific knowledge acquired through monolingual pretraining. finally, we show that our optimized model for portuguese demonstrates a reduced performance in english tasks, thereby substantiating the inherent compromise in refining models for specific linguistic domains. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language chapter © 2025 monoand multilingual gpt-3 models for hungarian chapter © 2023 scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. african languages language teaching and learning languages language education portuguese language policy and planning 1 introduction language models have revolutionized the field of natural language processing with their exceptional ability to perform tasks with minimal supervision. although primarily pretrained on english-centric corpora, the models have shown impressive multilingual capabilities [10]. given the abundance of languages worldwide, the majority of which are low-resource, it has become a common practice to pretrain single models on multiple languages simultaneously. models like xlm-r [12], mbart [28], mt5 [70], and bloom [54] exemplify this approach. despite the success of these multilingual models, we argue that they may not be the optimal approach for capturing the cultural and knowledge richness inherent in individual languages. when a moderately-sized language-specific corpus is available, continued pretraining could integrate the missing knowledge into the model, enhancing its performance on targeted tasks. to test this hypothesis, we extend the pretraining of english-centric models using portuguese corpora and evaluate their performance on an extensive range of portuguese datasets employing a few-shot learning approach. our results indicate that, even for models trained beyond the recommendations by hoffmann et al [18], this additional pretraining considerably improves performance compared to multilingual models. we evaluate our models on datasets comprising texts originally created by native brazilian portuguese speakers, as well as datasets translated from english to portuguese. we observe improvements across all datasets due to the portuguese pretraining, with the gains being particularly pronounced for datasets created by brazilian speakers. one of the largest improvements was observed on the enem dataset [57], which is derived from entrance exams used by brazilian universities and requires extensive knowledge of the country’s history, geography, and literature. this result provides evidence that the major contribution of our language-specific pretraining is to inject domain-specific knowledge about a particular culture as opposed to solely enhancing language proficiency. 2 related work the success of multilingual pretraining has been well-documented in the literature, with models such as byt5 [69], mt5 [70], xlm-r [12], xglm [27] and mgpt [56] paving the way for more inclusive language understanding and generation by leveraging shared knowledge across multiple languages. however, there are limitations to this approach. bloom, a 175b-parameter model pretrained on 46 languages, performs worse on english tasks compared to opt [74], a similarly sized model pretrained on english-centric corpora using comparable computational resources and data size. we conjecture that bloom’s underperformance may be attributed to its relatively limited exposure to english tokens during the pretraining phase. consequently, this observation suggests that monolingual pretraining could offer supplementary advantages. in support of this hypothesis, models with hundreds of millions of parameters pretrained on monolingual texts have demonstrated gains over multilingual counterparts [2, 6,7,8, 21, 24, 25, 32, 36, 52, 59]. additionally, research has indicated that language adaptation is beneficial even for low-resource languages [4, 13, 38, 72]. however, there is a limited number of published research articles with comprehensive evaluations of the benefits of continued pretraining at the multi-billion-parameter scale [22, 50, 73]. through this study, we contribute to the literature by demonstrating the effectiveness of continued language-specific pretraining for portuguese language models up to the 65b-parameter scale. the question concerning whether it is advantageous to train models for specific languages is closely associated with the question of whether it is beneficial to train models for particular domains of knowledge. recent studies, such as minerva [26] and galactica [62], have shown that domain-specific pretraining can lead to significant improvements, even with a smaller pretraining corpus compared to large-scale, general-purpose pretraining corpora. analogously, fu et al. [15] demonstrated the feasibility of specializing smaller models to perform multi-step reasoning, a capability typically exclusive to models with at least 50b parameters, at the expense of diminished performance in other, more general tasks. pretraining with a combination of general and domain-specific corpora can potentially enhance performance in specialized tasks without compromising effectiveness in general-purpose tasks, albeit at the cost of increased computational demands. for example, bloomberggpt [68], a 50b-parameter model pretrained on heterogeneous corpus in which more than half of texts are from the financial domain, exhibits comparable performance to opt-66b in general tasks. however, bloomberggpt’s pretraining dataset is three times larger, and consequently used more computational resources. rather than pursuing a single model that performs well across multiple domains, gururangan et al. [17] propose an alternative approach: using multiple expert models, each trained on a domain-specific subset within a broader, diverse dataset, to function as a single general-purpose model. their models outperform dense ones across various domain-specific tasks, at the expense of an increased parameter count, consequently leading to larger memory requirements for efficient inference.footnote 1 3 methodology in this section, we outline the pretraining data and training details used to build our models, including data sources, preprocessing techniques, architectures, hyperparameters, and optimization methods. 3.1 pretraining data the pretraining data is derived from the portuguese subset of the clueweb 2022 dataset [40, 41]. to increase the datasets’s quality, we apply the quality filters from massivetext [45], modifying them to accommodate the specific requirements of the portuguese language. we normalize the text with ftfyfootnote 2, convert wikitexts into human-readable texts, and exclude documents containing less than 200 unique tokens. these quality filters are primarily designed for web pages and may not seamlessly transfer to other domains. there is potential for improvement by employing more automated methods; however, this study did not explore such approaches due to the resource-intensive nature of pretraining experiments. following the cleaning process, all documents are concatenated using an end-of-sequence token as a separator, and then tokenized. the gpt-j tokenizer, which is identical to the gpt-2 tokenizer [44], produces 7.8 billion tokens, while the llama tokenizer produces 7.3 billion tokens. the discrepancy in the total number of tokens is primarily due to the different tokenization strategies each model employs, byte-level bpe and bpe based on sentencepiece [23], respectively along with the variation of the vocabularies used by each tokenizer. we extended the training of three models llama 7b and 65b [63] as well as gpt-j [66] originally trained on english-centric corpora, on portuguese texts; these further pretrained models from llama are denoted as sabiá, while the one derived from gpt-j is referred to as sabiá-j.footnote 3 3.2 sabiá models the llama 7b and 65b models are decoder-only transformer models [64] with a similar architecture to palm’s [10]. the models were trained using a causal language modeling objective on a massive dataset sourced from webpages, code, books, and scientific papers. the 7b model was trained on 1 trillion tokens and the 65b model was trained on 1.4 trillion tokens. while the majority of the corpus is in english, it also includes an unspecified amount of portuguese text. starting from the llama weights, we train the sabiá models on our portuguese dataset (see sect. 3.1) using the t5x and seqio frameworks [48]. adhering closely to the hyperparameters used by palm, we use the adafactor optimizer [55] without factorization, a first-order momentum \(\beta _1 = 0.9\), and a second-order momentum \(\beta _2 = 1 k^{-0.8}\), where k represents the step number. we apply global norm clipping at 1.0 and dynamic weight decay of \(lr^2\), with lr denoting the current learning rate. besides the standard causal language modeling loss, we use an auxiliary loss of \(10^{-4} \log ^2 (\sum _i e^{z_i})\), where z are the logits, to decrease the likelihood of loss spikes at the 65b-parameter scale. the learning rate is linearly increased from 0 to 1e-3 over the initial 1,000 steps, followed by a constant learning rate of 1e-3 for an additional 9,000 steps. the models were trained on a tpu v2-512, using batches of 512 sequences, each containing 2048 tokens. we utilized gradient checkpointing, also known as rematerialization, to enable the use of larger batches, thereby increasing tpu utilization. for the 7b model, this configuration results in a throughput of 124,000 tokens/sec, corresponding to a model flops utilization (mfu) [10] of 45.2%, excluding the self-attention operations. for the 65b model, we achieve a throughput of 14,000 tokens/sec, resulting in an mfu of 47.4%. the resulting models were trained on a total of 10.4 billion tokens, or 1.52 epochs of the portuguese dataset. this equals to 10,000 training steps, which is the same amount used to train sabiá. we noticed improvements in few-shot tasks beyond one epoch, which corroborates results from taylor et al. [62]. however, due to the high costs of pretraining, we did not continue training.footnote 4 3.3 sabiá-j the gpt-j model is a 6b-parameter decoder-only transformer model whose architecture and training hyperparameters closely follow gpt-3 6.7b. the main differences reside on computing the mlp and self-attention in parallel, applying attention head with dimension 256 (twice larger than gpt-3 6.7b), and using rotary positional embedding (rope) [61]. gpt-j was trained on 400b tokens from the pile dataset [16], whose 97.4% tokens are in english. we begin training sabiá-j from the released gpt-j checkpoint,footnote 5 using the mesh-transformer-jax framework [65] and adamw optimizer [30] with a weight decay of 0.1. we start the pretraining by warming up the learning rate until 1.2e-5 over 13,500 steps, followed by a cosine annealing decay during 135,518 steps until the end learning rate of 2.4e-6, and kept it constant from there on. we train on a tpu v3-8 using an effective batch size of 32 sequences of 2048 tokens. this results in a throughput of 5,200 tokens/sec, corresponding to a mfu of 44.5% without self-attention. the model was trained for 18 d on 7.8b tokens, or one epoch of the portuguese dataset.footnote 6 4 evaluation on poeta we evaluate the sabiá models on the portuguese evaluation tasks (poeta) benchmark, which comprises 14 downstream nlp datasets in portuguese: assin 2 rte and sts [47], enem challenge [57], enem 2022 [37], faquad [53], tweetsentbr [5], ag news [75], imdb [31], massive [14], mkqa [29], boolq [11], sst2 [58], wsc [33], and bluex [1]. half of them (assin 2 rte and sts, bluex, enem challenge, enem 2022, faquad, and tweetsentbr) were originally written in portuguese, and the remaining ones were either manually or automatically translated into portuguese from their originals in english. we refer to the first group as “native” datasets and the second group as “translated” datasets.footnote 7 the models were evaluated in a few-shot manner using the maximum number of examples that fits into a 2048-token context for each task. we used the gpt-2 tokenizer as a reference because it results in more tokens. this allowed us to comfortably fit prompts tokenized with other tokenizers. to evaluate the models, we manually select a set of few-shot examples for each dataset on poeta. depending on the dataset, these examples are balanced by class (except for faquad, bluex, enem challenge, enem 2022, mkqa, and wsc). for each test example, the prompts are built with the selected few-shot examples in alternating order. each task on poeta has a particular instruction that is placed at the beginning of the prompt. following srivastava et al. [60], we adopt the normalized preferred metric (npm) as our primary evaluation measure: $$\begin{aligned} \texttt {npm} = \frac{1}{n} \sum _{i=1}^n 100 \times \frac{\texttt {[raw preferred metric]}_i \texttt {[random score]}_i}{\texttt {[high score]}_i \texttt {[random score]}_i} \end{aligned}$$ (1) where n is the number of evaluation datasets, \(\texttt {[raw preferred metric]}_i\) is the score obtained by the model on the i-th dataset, \(\texttt {[random score]}_i\) is the score of a random model (e.g., 50% for a binary classification task) and \(\texttt {[high score]}_i\) is the highest possible score on that dataset, which is either 1 or 100. the preferred metric and random score for each dataset are presented in table 1. the rationale behind employing npm rather than a straightforward average across all datasets is to mitigate the undue influence of datasets with inherently high scores, such as binary classification datasets, which could otherwise outweigh datasets characterized by lower scores. table 1. a summary of the datasets constituting the poeta benchmark.full size table 5 results the main results can be found in table 2. models such as bloomz, xglm and bertin-gpt struggled to generate answers in portuguese. to address this issue, we adopted an approach akin to that used by the xglm authors: by calculating the likelihood of each candidate answer string based on the input text and subsequently selecting the class with the highest probability. for faquad, the only dataset in the benchmark without predetermined candidate answers, we allowed the models to generate answers in their original format. we observe that the llama baselines significantly outperform models of equivalent size trained with fewer tokens, such as galactica and opt. furthermore, despite being trained on english-centric corpora, llama-7b surpasses multilingual bloom and xglm of similar sizes. the sabiá models demonstrate considerable improvement in npm compared to their respective baseline models. these npm gains are more substantial for the smaller sabiá-j and sabiá-7b models. notably, sabiá-65b marginally outperforms openai’s gpt-3.5-turbo, which serves as the base model for chatgpt. table 2. few-shot npm results on the poeta benchmark.full size table through our portuguese pretraining, we observed that the improvement in npm was higher in native datasets than that in translated datasets. for sabiá-65b, improvements over llama-65b were mostly from the native subset. we hypothesize that this is due to the “mechanistic” nature of translated datasets: since they were translated from english, the baseline model already possesses the knowledge needed to solve them and gains little from learning the linguistic, syntactic, and grammatical knowledge of the target language. for instance, to answer the question “does p o box come before street address” (boolq dataset), the model gains little from additional pretraining on a portuguese corpus as it is unlikely that the corpus would provide new information regarding the formatting of us mailing addresses that the model has not already encountered during its initial english-centric pretraining. conversely, language-specific pretraining introduces the specific knowledge required to solve tasks in the native subset. although gpt-j exhibited lower few-shot performance in english tasks relative to llama, we use it in this study to illustrate that not only highly optimized models like llama can benefit from extended pretraining. we chose not to use bloom-7.1b as our initial checkpoint for pretraining due to its inferior performance compared to gpt-j in preliminary few-shot experiments on three portuguese datasets. however, we later discovered that its performance on poeta surpassed gpt-j’s. nonetheless, bloom still exhibits lower performance compared to llama. analogous to sabiá-j, bertin-gpt is a model pretrained on spanish text starting from the gpt-j weights. since spanish and portuguese are similar languages, it is reasonable to expect that bertin-gpt would perform better than its baseline model. nevertheless, the observed npm for bertin-gpt is only slightly higher than gpt-j’s. a noteworthy comparison involves galactica, a model pretrained on scientific text, predominantly in english, and a similarly-sized opt model, which utilized comparable pretraining compute but was pretrained on a larger and more diverse english-centric corpus. in their study, the authors demonstrate that galactica performs on par with opt on english tasks and largely outperforms opt on scientific-related tasks. conversely, opt significantly outperforms galactica in portuguese tasks. this result underscores the trade-offs associated with domain-specific specialization, which often entails diminished performance in other tasks. bloomz [35], a multilingual instruction-tuned model, demonstrated superior performance compared to its baseline bloom model, rivaling llama of equivalent size.footnote 8 nevertheless, our approach of pretraining in portuguese appears to yield superior results, as sabiá-j surpasses bloomz despite originating from a lower-performing baseline model. we envision continued pretraining and instruction tuning as complementary techniques to be combined in future research. 5.1 results per dataset table 3 presents the results per poeta dataset for sabiá models, their baselines, and for the supervised state-of-the-art. the sota results reported for the translated datasets were obtained using their original english versions [46, 51, 71, 76]. since the poeta benchmark excludes unanswerable examples of the mkqa dataset, we decided not to include the sota result for this dataset. in more challenging datasets, such as enem challenge, enem 2022, and bluex, which are derived from admission exams to brazilian universities, we see the most significant gains due to language-specific pretraining. substantial improvements are also observed in tweetsentbr, a dataset containing tweets with an abundance of slang and references to brazilian popular culture. we hypothesize that this pretraining imparts specific knowledge about the country’s culture, literature, and geography that is less frequently encountered and learned during the original pretraining with more diverse texts. table 3. results per dataset. \(^1\) [49]; \(^2\) [9]; \(^3\) [34]; \(^4\) [3]; \(^5\) [71]; \(^6\) [76]; \(^7\) [46]; \(^8\) [51].full size table certain capabilities only emerge at scale, as evidenced by [67]. for example, 6-7b models perform close to the random baseline in datasets such as assin 2 rte and sts, and wsc. however, at the 65b scale, we observe substantial improvements, approaching or surpassing state-of-the-art supervised models on the assin 2 rte and faquad datasets. gpt-4 [39] results indicate that there is still room for improvement for sabiá-65b in the majority of the datasets evaluated in this work. nevertheless, sabiá-65b performs on par with gpt-4 in datasets such as assin 2 rte, enem challenge, and faquad. 5.2 data contamination the pretraining data for sabiá models were collected up until february 2022. since enem 2022 was publicly released in november 2022, the model could not have access to the answers for the questions present within its pretraining data. consequently, the improvements observed at least for enem 2022, which were higher than the average of the datasets, cannot be attributed to data contamination. however, for the other datasets, the possibility of data contamination cannot be ruled out. 5.3 ablation: english datasets in this ablation study, we investigate the potential impact of portuguese pretraining on the performance of the model in english datasets. we evaluated the llama-7b and the sabiá-7b models in english multiple-choice tasks. for simplicity, we employed a few-shot evaluation setup with 10 randomly selected examples (dynamic-sampled prompt). importantly, we did not incorporate any descriptions or include portuguese keywords to delimit the few-shot examples. we also restricted all the datasets to 350 test examples. following llama’s [63] approach, given the provided context, we select the answer with the highest likelihood normalized by the number of characters. the results in table 4 indicate that the sabiá-7b model exhibits a slightly reduced performance in english tasks compared to the baseline. this result corroborates our premise that model specialization invariably entails a balancing act, where improvements in one domain frequently coincide with degradation in another. table 4. results in english datasets.full size table 6 limitations owing to the financial constraints associated with pretraining and, more significantly, the manual labor involved in collecting and curating evaluation datasets, experiments were conducted exclusively in portuguese. given that our models started pretraining from english-pretrained models and that portuguese and english exhibit relatively close linguistic proximity, we anticipate that other researchers conducting further pretraining on languages closely related to english will observe comparable improvements in their target tasks. however, determining whether the benefits of this method persist for languages more distant from english remains an open research question. portuguese is a language with an abundance of high-quality web-based texts. thus, the gains observed with the proposed method may not necessarily extend to low-resource languages with limited availability of quality texts. in such cases, parameter-efficient methods [19, 42, 43] could be advantageous, as evidenced by yong et al. [72]. we did not use these techniques in this study due to the training costs, which are approximately equivalent to training the entire model.footnote 9 7 conclusion in this study, we contributed to the expanding body of scientific evidence that specializing models for individual languages leads to improvements, even when the baseline model is large and extensively trained. we achieved this for the portuguese language utilizing a near state-of-the-art model with 65 billion parameters. given the relatively low pretraining cost and significant performance gains observed, we foresee a future landscape consisting of a diverse array of models, each tailored to a specific domain, rather than a single, all-encompassing model. notes 1.to serve their ensemble with a low latency, the weights for each expert must be kept in gpu memory. 2.ftfy normalization fixes mojibakes and remove remnant html tags. 3.sabiá is a tribute to the eponymous bird, renowned for its diverse and intricate vocalizations. 4.considering the on-demand pricing of 384 usd per hour for a tpu v2-512, pretraining sabiá-7b and sabiá-65b costs approximately 9,000 and 80,000 usd, respectively. 5.https://huggingface.co/eleutherai/gpt-j-6b. 6.due to constraints in our hardware budget, this model was trained with fewer tokens compared to sabiá. 7.the massive dataset underwent manual translation and localization; however, given that the original text was composed in english, it has been categorized as a translated dataset. 8.this model was used in the experiments: https://huggingface.co/bigscience/bloomz-7b1-mt. 9.although parameter-efficient methods adjust only a fraction of the weights, they use only marginally fewer training flops, as activations and gradients are computed for the entire model. for instance, lora [20], a parameter-efficient method, improves training throughput of a gpt-3 175b model by only nearly 32%. references almeida, t.s., laitz, t., bonás, g.k., nogueira, r.: bluex: a benchmark based on brazilian leading universities entrance exams. to appear (2023) google scholar  antoun, w., baly, f., hajj, h.: arabert: transformer-based model for arabic language understanding. in: proceedings of the 4th workshop on open-source arabic corpora and processing tools, with a shared task on offensive language detection. pp. 9–15. european language resource association, marseille, france (2020) google scholar  barros, t.m.d., et al.: employing transformers and emoji to perform sentiment classification of social media texts: utilizando transformers e emoji na classificação de sentimento de textos oriundos de redes sociais (2021) google scholar  bhattacharjee, a., et al.: banglabert: language model pretraining and benchmarks for low-resource language understanding evaluation in bangla. in: findings of the association for computational linguistics: naacl 2022, pp. 1318–1327. association for computational linguistics, seattle, united states (2022). https://doi.org/10.18653/v1/2022.findings-naacl.98 brum, h., volpe nunes, m.d.g.: building a sentiment corpus of tweets in brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). european language resources association (elra), miyazaki, japan (may 2018) google scholar  cañete, j., chaperon, g., fuentes, r., ho, j.h., kang, h., pérez, j.: spanish pre-trained bert model and evaluation data. in: pml4dc at iclr 2020 (2020) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data. arxiv preprint arxiv:2008.09144 (2020) chan, b., schweter, s., möller, t.: german’s next language model. in: proceedings of the 28th international conference on computational linguistics, pp. 6788–6796. international committee on computational linguistics, barcelona, spain (online) (2020). https://doi.org/10.18653/v1/2020.coling-main.598 chaves rodrigues, r., tanti, m., agerri, r.: evaluation of portuguese language models (2023). https://doi.org/10.5281/zenodo.7781848, https://github.com/ruanchaves/eplm chowdhery, a., et al.: palm: scaling language modeling with pathways. arxiv preprint arxiv:2204.02311 (2022) clark, c., lee, k., chang, m.w., kwiatkowski, t., collins, m., toutanova, k.: boolq: exploring the surprising difficulty of natural yes/no questions. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 2924–2936. association for computational linguistics, minneapolis, minnesota (2019). https://doi.org/10.18653/v1/n19-1300 conneau, a., et al.: unsupervised cross-lingual representation learning at scale. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 8440–8451 (2020) google scholar  ebrahimi, a., kann, k.: how to adapt your pretrained multilingual model to 1600 languages. in: proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers), pp. 4555–4567. association for computational linguistics, online (aug 2021). 10.18653/v1/2021.acl-long.351 google scholar  fitzgerald, j., et al.: massive: a 1m-example multilingual natural language understanding dataset with 51 typologically-diverse languages (2022) google scholar  fu, y., peng, h., ou, l., sabharwal, a., khot, t.: specializing smaller language models towards multi-step reasoning. arxiv preprint arxiv:2301.12726 (2023) gao, l., et al.: the pile: an 800gb dataset of diverse text for language modeling. arxiv preprint arxiv:2101.00027 (2020) gururangan, s., et al.: scaling expert language models with unsupervised domain discovery. arxiv preprint arxiv:2303.14177 (2023) hoffmann, j., et al.: training compute-optimal large language models. arxiv preprint arxiv:2203.15556 (2022) houlsby, n., et al.: parameter-efficient transfer learning for nlp. in: international conference on machine learning, pp. 2790–2799. pmlr (2019) google scholar  hu, e.j., et al.: lora: low-rank adaptation of large language models. in: international conference on learning representations (2022). https://openreview.net/forum?id=nzevkeefyf9 kalyan, k.s., rajasekharan, a., sangeetha, s.: ammus: a survey of transformer-based pretrained models in natural language processing. arxiv preprint arxiv:2108.05542 (2021) kim, b., et al.: what changes can large-scale language models bring? intensive study on hyperclova: billions-scale korean generative pretrained transformers. in: proceedings of the 2021 conference on empirical methods in natural language processing, pp. 3405–3424 (2021k google scholar  kudo, t., richardson, j.: sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. in: proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, pp. 66–71. association for computational linguistics, brussels, belgium (2018). https://doi.org/10.18653/v1/d18-2012 le, h., et al.: flaubert: unsupervised language model pre-training for french. in: proceedings of the twelfth language resources and evaluation conference, pp. 2479–2490. european language resources association, marseille, france (2020) google scholar  lee, h., yoon, j., hwang, b., joe, s., min, s., gwon, y.: korealbert: pretraining a lite bert model for korean language understanding. in: 2020 25th international conference on pattern recognition (icpr), pp. 5551–5557. ieee (2021) google scholar  lewkowycz, a., et al.: solving quantitative reasoning problems with language models. arxiv preprint arxiv:2206.14858 (2022) lin, x.v., et al.: few-shot learning with multilingual generative language models. in: proceedings of the 2022 conference on empirical methods in natural language processing, pp. 9019–9052 (2022) google scholar  liu, y., et al.: multilingual denoising pre-training for neural machine translation. trans. assoc. comput. linguist. 8, 726–742 (2020) article  google scholar  longpre, s., lu, y., daiber, j.: mkqa: a linguistically diverse benchmark for multilingual open domain question answering. trans. assoc. comput. linguist. 9, 1389–1406 (2021) article  google scholar  loshchilov, i., hutter, f.: decoupled weight decay regularization. in: international conference on learning representations (2019) google scholar  maas, a.l., daly, r.e., pham, p.t., huang, d., ng, a.y., potts, c.: learning word vectors for sentiment analysis. in: proceedings of the 49th annual meeting of the association for computational linguistics: human language technologies, pp. 142–150. association for computational linguistics, portland, oregon, usa (2011) google scholar  martin, l., et al.: camembert: a tasty french language model. in: proceedings of the 58th annual meeting of the association for computational linguistics. pp. 7203–7219. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.acl-main.645 de melo, g., imaizumi, v., cozman, f.: winograd schemas in portuguese. in: anais do xvi encontro nacional de inteligência artificial e computacional, pp. 787–798. sbc (2019) google scholar  moraes, g., bonifácio, l.h., rodrigues de souza, l., nogueira, r., lotufo, r.: a cost-benefit analysis of cross-lingual transfer methods. arxiv preprint arxiv:2105.06813 (2021). https://arxiv.org/abs/2105.06813 muennighoff, n., et al.: crosslingual generalization through multitask finetuning (2022) google scholar  nguyen, d.q., tuan nguyen, a.: phobert: pre-trained language models for vietnamese. in: findings of the association for computational linguistics: emnlp 2020, pp. 1037–1042. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.92 nunes, d., primi, r., pires, r., lotufo, r., nogueira, r.: evaluating gpt-3.5 and gpt-4 models on brazilian university admission exams (2023) google scholar  ogueji, k., zhu, y., lin, j.: small data? no problem! exploring the viability of pretrained multilingual language models for low-resourced languages. in: proceedings of the 1st workshop on multilingual representation learning, pp. 116–126. association for computational linguistics, punta cana, dominican republic (2021) google scholar  openai: gpt-4 technical report (2023) google scholar  overwijk, a., xiong, c., callan, j.: clueweb 22: 10 billion web documents with rich information. in: proceedings of the 45th international acm sigir conference on research and development in information retrieval, pp. 3360–3362 (2022) google scholar  overwijk, a., xiong, c., liu, x., vandenberg, c., callan, j.: clueweb 22: 10 billion web documents with visual and semantic information (2022) google scholar  pfeiffer, j., kamath, a., rücklé, a., cho, k., gurevych, i.: adapterfusion: non-destructive task composition for transfer learning. in: proceedings of the 16th conference of the european chapter of the association for computational linguistics: main volume, pp. 487–503. association for computational linguistics, online (2021). https://doi.org/10.18653/v1/2021.eacl-main.39 pfeiffer, j., vulić, i., gurevych, i., ruder, s.: mad-x: an adapter-based framework for multi-task cross-lingual transfer. arxiv preprint arxiv:2005.00052 (2020) radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i., et al.: language models are unsupervised multitask learners. openai blog 1(8), 9 (2019) google scholar  rae, j.w., et al.: scaling language models: methods, analysis & insights from training gopher. arxiv preprint arxiv:2112.11446 (2021) raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21(1), 5485–5551 (2020) mathscinet  google scholar  real, l., fonseca, e., gonçalo oliveira, h.: the assin 2 shared task: a quick overview. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 406–412. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_39 chapter  google scholar  roberts, a., et al.: scaling up models and data with t5x and seqio. arxiv preprint arxiv:2203.17189 13 (2022) rosa, g.m., bonifacio, l.h., de souza, l.r., lotufo, r., nogueira, r.: a cost-benefit analysis of cross-lingual transfer methods. arxiv preprint arxiv:2105.06813 (2021) la rosa, j.d., fernández, a.: zero-shot reading comprehension and reasoning for spanish with bertin gpt-j-6b. in: y gómez, m.m., (eds.) proceedings of the iberian languages evaluation forum (iberlef 2022). ceur workshop proceedings (2022) google scholar  sakaguchi, k., bras, r.l., bhagavatula, c., choi, y.: winogrande: an adversarial winograd schema challenge at scale. commun. acm 64(9), 99–106 (2021) article  google scholar  sarti, g., nissim, m.: it5: large-scale text-to-text pretraining for italian language understanding and generation. arxiv preprint arxiv:2203.03759 (2022) sayama, h.f., araujo, a.v., fernandes, e.r.: faquad: reading comprehension dataset in the domain of brazilian higher education. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 443–448 (2019). https://doi.org/10.1109/bracis.2019.00084 scao, t.l., et al.: bloom: a 176b-parameter open-access multilingual language model. arxiv preprint arxiv:2211.05100 (2022) shazeer, n., stern, m.: adafactor: adaptive learning rates with sublinear memory cost. in: international conference on machine learning, pp. 4596–4604. pmlr (2018) google scholar  shliazhko, o., fenogenova, a., tikhonova, m., mikhailov, v., kozlova, a., shavrina, t.: mgpt: few-shot learners go multilingual. arxiv preprint arxiv:2204.07580 (2022) silveira, i.c., maua, d.d.: advances in automatically solving the enem. in: 2018 7th brazilian conference on intelligent systems (bracis), pp. 43–48. ieee computer society, los alamitos, ca, usa (oct 2018). https://doi.org/10.1109/bracis.2018.00016 socher, r., et al.: recursive deep models for semantic compositionality over a sentiment treebank. in: proceedings of the 2013 conference on empirical methods in natural language processing, pp. 1631–1642. association for computational linguistics, seattle, washington, usa (2013) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  srivastava, a., et al.: beyond the imitation game: quantifying and extrapolating the capabilities of language models. arxiv preprint arxiv:2206.04615 (2022) su, j., lu, y., pan, s., wen, b., liu, y.: roformer: enhanced transformer with rotary position embedding. arxiv preprint arxiv:2104.09864 (2021) taylor, r. et al.: galactica: a large language model for science. arxiv preprint arxiv:2211.09085 (2022) touvron, h., et al.: llama: open and efficient foundation language models. arxiv preprint arxiv:2302.13971 (2023) vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems 30 (2017) google scholar  wang, b.: mesh-transformer-jax: model-parallel implementation of transformer language model with jax. https://github.com/kingoflolz/mesh-transformer-jax (2021) wang, b., komatsuzaki, a.: gpt-j-6b: a 6 billion parameter autoregressive language model (2021) google scholar  wei, j., et al.: emergent abilities of large language models. transactions on machine learning research (2022), survey certification google scholar  wu, s., et al.: bloomberggpt: a large language model for finance (2023) google scholar  xue, l., et al.: byt5: towards a token-free future with pre-trained byte-to-byte models. trans. assoc. comput. linguist. 10, 291–306 (2022) article  google scholar  xue, l., et al.: mt5: a massively multilingual pre-trained text-to-text transformer. arxiv preprint arxiv:2010.11934 (2020) yang, z., dai, z., yang, y., carbonell, j., salakhutdinov, r., le, q.v.: xlnet: generalized autoregressive pretraining for language understanding. curran associates inc., red hook, ny, usa (2019) google scholar  yong, z.x., et al.: bloom+ 1: adding language support to bloom for zero-shot prompting. arxiv preprint arxiv:2212.09535 (2022) zeng, a., et al.: glm-130b: an open bilingual pre-trained model. arxiv preprint arxiv:2210.02414 (2022) zhang, s., et al.: opt: open pre-trained transformer language models. arxiv preprint arxiv:2205.01068 (2022) zhang, x., zhao, j.j., lecun, y.: character-level convolutional networks for text classification. in: nips (2015) google scholar  zoph, b.: designing effective sparse expert models. in: 2022 ieee international parallel and distributed processing symposium workshops (ipdpsw), p. 1044. ieee (2022) google scholar  download references acknowledgments we thank google cloud for the generous tpu grant. author information authors and affiliations maritaca ai, campinas, brazil ramon pires, hugo abonizio, thales sales almeida & rodrigo nogueira authors ramon piresview author publications search author on:pubmed google scholar hugo abonizioview author publications search author on:pubmed google scholar thales sales almeidaview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar corresponding author correspondence to ramon pires . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pires, r., abonizio, h., almeida, t.s., nogueira, r. (2023). sabiá: portuguese large language models. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_15 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature detecting multiple epidemic sources in network epidemics using graph neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper detecting multiple epidemic sources in network epidemics using graph neural networks conference paper first online: 12 october 2023 pp 331–345 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) detecting multiple epidemic sources in network epidemics using graph neural networks download book pdf download book epub rodrigo gonçalves haddad  orcid: orcid.org/0009-0006-2171-600x9 & daniel ratton figueiredo  orcid: orcid.org/0000-0001-9341-66199  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 530 accesses 1 citation abstract epidemics start within a network because of the existence of epidemic sources that spread information over time to other nodes. data about the exact contagion pattern among nodes is often not available, besides a simple snapshot characterizing nodes as infected, or not. thus, a fundamental problem in network epidemic is identifying the set of source nodes after the epidemic has reached a significant fraction of the network. this work tackles the multiple source detection problem by using graph neural network model to classify nodes as being the source of the epidemic. the input to the model (node attributes) are novel epidemic information in the k-hop neighborhoods of the nodes. the proposed framework is trained and evaluated under different network models and real networks and different scenarios, and results indicate different trade-offs. in a direct comparison with prior works, the proposed framework outperformed them in all scenarios available for comparison. this work has received financial support through research grants from cnpq and faperj (brazil). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others the effect of concurrency on epidemic threshold in time-varying networks chapter © 2019 approximate identification of the optimal epidemic source in complex networks chapter © 2020 information source estimation with multi-channel graph neural network chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. complex networks dynamic networks graph theory network models network research stochastic networks 1 introduction epidemics are among the most prominent and fundamental processes that take place in networks since they represent a diverse set of phenomena, e.g., the spread of information in social media or biological virus among the global human population [4, 14, 15]. in a network epidemic, nodes have an epidemic state (infected or susceptible) that may change over time while contagion occurs through network edges. it is not surprising that, over the past decades, many works have focused on developing models to predict real network epidemics sources for various kinds of phenomena. more importantly, the focus has also been on understanding the impact that different measures have on the epidemic process, such as quarantine and immunization, to cite a recent example largely debated during the coronavirus pandemic [7]. an epidemic starts with the infection of a single node or a few nodes, known as the epidemic sources. from this initial set of nodes, the epidemic unfolds through the edges of the network potentially reaching a large fraction of the network nodes. precise information about epidemic process is often not available, such as timing or contagion information. therefore, a fundamental problem in network epidemic is identifying the set of source nodes after the epidemic has reached a significant fraction of the network [11, 20]. this problem is known to be hard for simple epidemics even when it starts on a single node and the network is infinite [18, 19]. the difficulty in identifying the source nodes strongly depend on the observation process, i.e., what is observed from the network epidemic. a common snapshot model consists of a single and instantaneous snapshot that reveals the epidemic state of all network nodes [11, 20]. if the snapshot is taken very early in the epidemic process, very few nodes will be infected and identifying the source nodes is intuitively easier. on the other hand, if it is taken very late, i.e., once all nodes have been infected, there is no information that can be used to identify the source nodes. the time the snapshot is taken can be translated to the moment the epidemic reaches a certain fraction of nodes, which is the model used in this paper. the source identification problem can be posed as a classification problem as, given an epidemic snapshot, every node is to be classified as being an epidemic source, or not [3, 17]. besides the epidemic state of the nodes provided in the snapshot, the network structure is also available to the classification model. therefore, a natural classification model is one that takes advantage of the network over which the epidemic unfolds. graph neural networks (gnns) seem well-suited for this task since this model integrates neighboring information. however, two nodes that have too many neighbors in common are likely to have similar representations under classic gnn models which would not be suited for source identification, since neighbors of the source node are often not a source (depending on how source nodes are chosen). in order to tackle this issue, this work leverages the epidemic information within the k-hop neighborhood of the node. two metrics are proposed: the fraction of nodes infected at the k-hop neighborhood and the average infection rate at the k-hop neighborhood (to be discussed in detail). these metrics are a function of the epidemic snapshot and are computed for every network node efficiently. the information, i.e., vector of rational numbers for each metric, is taken as node attribute, distinguishing neighboring nodes much beyond their epidemic state and hence providing better inputs to the gnn model. the remainder of this paper is organized as follows. in sect. 2, network epidemic and multiple source detection problem are introduced. a brief discussion of the related work is presented in sect. 3. in sect. 4, the proposed framework is presented. the evaluation of the proposed framework under different scenarios is shown in sect. 5. finally, sect. 6 presents a brief conclusion for the paper. 2 network epidemics and problem statement network epidemics reflect different phenomena, such as influence models that describe “social contagion” [2] among individuals of a offline or online social network, and infection models that describe the contagion of a biological disease among a population. while social and biological epidemics have fundamentally different underlying contagion mechanisms, their essence can be represented by simple epidemic models. the simplest epidemic model is the compartmental model where each individual is in one of only two possible epidemic states: susceptible or infected. this model is known as the classic si epidemic model [12] where individuals can only change from the susceptible to infected. in the context of networks, individuals are represented by network nodes, and edges indicate the possibility of contagion. thus, let \(g=(v,e)\) denotes a undirected network (graph) where v and e denote the set of nodes and edges, respectively, and \(n= |v|\) denotes the number of nodes. the si network epidemic model here considered is a discrete time model. let i(t) and s(t) denote the set of infected and susceptible nodes at time t. note that \(i(t) \bigcup s(t) = v\) and \(i(t) \bigcap s(t) = \emptyset \) for all \(t=0,1,\ldots \). let \(i_0 \subset v\) denote the set of epidemic sources, namely, the set of nodes that are infected at time zero, thus, \(i(0) = i_0\). at each time slot, an infected node infects each of its susceptible neighbors with probability p. note that infection fails with probability \(1-p\), but the infected node is able to infect the same susceptible neighbor with probability p on subsequent time slots. a susceptible node that has j infected neighbors in the current time step becomes infected in the next time step with probability \(1-(1-p)^j\). nodes become infected in the same time slot and never recover, causing i(t) to be non-decreasing with t. last, assuming the network is connected, it can be shown that there exists a finite large enough \(t^*\) such that \(i(t^*) = v\) with high probability, and thus all nodes are infected. as \(i_0\) are the nodes chosen uniformly at random when the epidemic starts, \(s = |i_0|\) is the parameter that determines the number of epidemic sources. 2.1 epidemic observation the snapshot observation model which reveals the sets \(i(t_o)\) and \(s(t_o)\) is the network observation at a given time \(t_o\). note that g (the network) is also assumed to be known, and it is sufficient to observe i(t), since \(s(t) = v \setminus i(t)\). the epidemic snapshot is taken when a predefined number of nodes have been infected. let \(f_o\) denote that fraction of infected the nodes: the epidemic snapshot takes place the first time when \(i(t)/n \ge f_o\). more precisely, \(t_o = \min _t \{ t | i(t)/n \ge f_o\}\). in what follows, \(f_o\) will be used to determine when the snapshot is taken. 2.2 problem statement consider a si network epidemic with parameter p on the network \(g=(v,e)\) having started with the epidemic sources \(i_0\). a single epidemic snapshot is taken when a fraction of nodes infected is \(f_o\). design a model m that classifies nodes in \(i(t_o)\) as epidemic sources, taking as input g and \(i(t_o)\). ideally, the model should recover \(i_0\), namely $$\begin{aligned} m_{g,i(t_o)}(v) = {\left\{ \begin{array}{ll} 1 \text{, } \text{ if } v \in i_0 \\ 0 \text{, } \text{ otherwise } \end{array}\right. } \end{aligned}$$ (1) note that the model m has no prior information about p, the parameter for the si epidemic. m is trained on different networks and epidemics and it should classify nodes for networks and epidemics never seen before. in fact, obtaining such generalization is the main challenge in this problem. in addition, in most scenarios the number of epidemic sources is very small with respect to the number of nodes infected in the snapshot, often by a factor of one hundred or more, hence, this classification task deals with very imbalanced classes. 3 related work for over a decade, efforts have been made to address the problem of identifying the source of a network epidemic [11, 20]. the pioneering methods considered single source epidemics and simple networks while providing efficient algorithms to identify the source [19]. these approaches were based on simple probabilistic epidemic models and likelihood functions given the observation [5, 10, 23]. the rumor centrality algorithm proposed by shah and zaman is a prominent example that has been widely explored and extended [18, 19]. more recent works have tackled the multiple source detection (msd) problem, where the epidemic simultaneously starts in different network nodes motivated by applications where the diffusion process starts in various nodes, e.g., information in an online social network, [3, 17, 21, 24]. netsleuth [17] is an approach to infer the multiple epidemic sources in the probabilistic si infection model using a likelihood function approach and minimum description length (mdl) scores. netsleuth requires prior knowledge of the number of epidemic sources. another approach is lpsi (label propagation based source identification) [21] where labels assigned to nodes are propagated in iterations to build a weight matrix that is used to identify the multiple epidemic sources (the top ranked nodes according to a metric). the work of zang et. al [24] considers sir (susceptible-infected-recovered) epidemics and a limited observation model by building “extended infected node sets” upon lpsi that are partitioned and ranked using a network centrality metric. different number of epidemic sources are considered independently in order to determine the actual number of sources. while the above approaches are based on likelihood functions and label propagation, learning-based approaches have been recently proposed to tackle the msd problem. a prominent example is graph convolutional networks based source identification (gcnsi) [3] where a gcn model is trained with supervision to generate node representations. the approach uses lpsi as part of the node attributes and considers different epidemic models beyond si (susceptible-infected), such as ic (independent cascade) and sir. however, different configurations of gcn need to be trained for different networks. gcnsi is the closest approach to the framework designed in this paper, and results will be compared directly for scenarios that are available (see sect. 5). 4 proposed framework recall that, in graph neural networks, node attributes serve as the initial representation (input) to the neural network that generates the node representation. while the epidemic state of the node is the main node attribute, it is insufficient information to adequately distinguish nodes when it comes to latent representation generation, whereas sources representation should strongly distinguish from the other nodes. the framework here proposed as well as the evaluation scenarios are publicly availablefootnote 1 4.1 metrics and attributes the idea behind the proposed attributes is to reflect the epidemic state around the node in their k-hop neighborhood. intuitively, the epidemic state of the k-hop neighborhoods of source nodes should be similar but different from the epidemic state of k-hop neighborhoods of nodes infected late during the epidemic. therefore, two metrics are proposed to capture the epidemic state of k-hop neighborhoods: ring infection (ri) and depth ring infection (dri). the former leverages the average epidemic state of nodes at distance k, while the latter leverages the average neighboring epidemic state of nodes at distance k. given the snapshot \(i(t_o)\) and considering a node u, the ri metric is given by: $$\begin{aligned} \alpha ^u_k=\frac{\sum _{v \in n_k(u)} i_{v}}{|n_k(u)|} , \text{ for } k=0,\ldots ,k , \end{aligned}$$ (2) \(n_k(u)\) denotes the k-hop neighborhood of node u, or equivalently, the set of nodes at distance k from node u, and \(i_v\) the epidemic state of node v in the snapshot \(i(t_o)\). in particular, $$\begin{aligned} i_v ={\left\{ \begin{array}{ll} 1 \text{, } \text{ if } v \in i(t_o) \\ 0 \text{, } \text{ otherwise } \end{array}\right. } \end{aligned}$$ (3) note that \(\alpha ^u_0 = i_u\) is simply the epidemic state of node u. also, k is the parameter that determines the maximum distance to be considered in the attribute generation. the dri metric depends on the epidemic state of the neighbors of nodes that are at distance k from node u (and not just the epidemic state of the node). in particular, dri is the average of ri at distance 1 among the nodes that are at distance k from node u. dri is given by: $$\begin{aligned} \beta ^u_k=\frac{\sum _{v \in n_k(v)} \alpha _1^u}{|n_k(u)|} , \text{ for } k=1,\ldots ,k. \end{aligned}$$ (4) a single node can be counted multiple times in this metric, since it can be neighbor of many nodes at distance k from u. thus, dri provides information that is significantly different from ri. the calculations of ri and dri require determining the k-hop neighborhoods of all nodes in the network. for a given node u, a breadth first search (bfs) starting on u can efficiently determine the k-hop neighborhoods, in linear time \(o(|v|+|e|)\). since k is often small when compared to the network diameter, the bfs stops before reaching all network nodes. ri and dri for different nodes are computed fully in parallel, as there are no dependencies. this allows for both time and memory efficient calculation. finally, ri and dri are used to determine the attribute vector for each node. in particular, node u has an attribute vector that is the concatenation of \(\alpha _u\) and \(\beta _u\) for all k whose dimension is \(2k + 1\) (since \(\alpha _u\) has \(k + 1\) values and \(\beta _u\) has k values). this is the initial representation (input) to the gnn model. 4.2 graph neural network model graph neural networks (gnns) integrate structural information with attribute information (node labels) in order to generate representations for network nodes. the gnn is a stack of neural network layers whose input is the node representation along with an aggregated representation of its neighbors, that can take several forms such as averaging. in practice, the input to the first layer corresponds to node attributes, while the output of the last layer corresponds to the node representation. the implementation discussed here adopts a small number of layers, e.g., three layers, to avoid diluting the signal of the node’s neighborhood. while being possible to make gnn models learn representations without supervision, gnns may also be trained in a supervised fashion. in this case, the output of the last layer is used to compute a loss function that then drives the training of the model parameters. this work uses gnn in a classification task whose final goal is to classify an epidemic network node as being an epidemic source. therefore, a sigmoid function is applied to the output of the last (third) layer in order to generate values between 0 and 1. the first and second layers use the relu as activation function together with a dropout layer to help avoid overfitting. the layers adopted are sage layers proposed by graphsage, given its wide applicability and efficiency [9]. 4.3 loss function since the problem consists of binary classification, the binary cross entropy function was adopted as the loss function. in particular, for each node u, it follows: $$\begin{aligned} \ell _u(i_0) = c_u \log \sigma \left( y_u\right) +\left( 1-c_u\right) \log \left( 1-\sigma \left( y_u\right) \right) , \end{aligned}$$ (5) where \(c_u\) indicates if node u is an epidemic source (\(u \in i_0\), the ground truth), \(y_u\) is the output of the neural network for node u, and \(\sigma \) is the sigmoid function applied to the output of the neural network. the average loss function across all nodes must consider the fact that classes are very imbalanced in this problem. for that matter, a weighted average is considered where \(w_0\) and \(w_1\) denote the weights for each class. the weights must consider the number of nodes in each class; \(w_0 = r_0/(1 |i_0|/n)\) and \(w_1 = r_1/(|i_0|/n)\) where \(r_0\) and \(r_1\) are constants used to increase the class weight beyond the class balance ratio, and, in this work, \(r_0=1\) and \(r_1=100\) (chosen experimentally). this indicates to the model the importance of correctly classifying the epidemic source node. the average loss function is given by: $$\begin{aligned} \ell (i_0)= \frac{w_0 \sum _{u \not \in i_0} \ell _u(i_0) + w_1 \sum _{u \in i_0} \ell _u(i_0)}{(n |i_0|)w_0 + |i_0|w_1} \end{aligned}$$ (6) 4.4 datasets and training an arbitrary large and diverse dataset with ground truth concerning epidemic sources is generated for this problem. for each sampled network, the set \(i_0\) (epidemic sources) is chosen uniformly random, and a si epidemic is simulated until a fraction of nodes \(f_o\) is infected, giving rise to one epidemic network sample. this entire process is repeated to generate independent samples of network epidemics. real networks are also covered by the dataset, and in this case the network is always the same (but not \(i_0\) nor the epidemic network). the dataset is then split for training, validation and testing. this work made use of pytorch geometric [6], which employs pytorch [16] and provides a number of gnn layers. the adam optimizer was used with a low learning rate of 0.001 and all network nodes were considered as a single batch at each learning epoch. as for sampling neighboring nodes for gnn training, all nodes within a distance of 3 from the target node were considered. this sampling strategy is used to balance computational cost and network coverage as nodes closer to the target carry more important information. the model was trained on 500 epochs and an early stopping criteria is adopted such that if no improvements in the loss function are observed in the validation network for 75 consecutive epochs, the training with the input network stops in order to avoid overfitting. the experiments were carried on a computer with an intel core i7-11800h, gpu nvidia geforce 3070 8 gb and 16 gb ram. 5 evaluation the performance of different approaches to identify epidemic sources depends on various factors, including network structure. in order to better assess the proposed framework, two random graph models and two real networks are considered. the performance of the proposed approach will be characterized using two different criteria: identifying epidemic source nodes and identifying neighbors of source nodes. 5.1 network models and real networks the erdős-rényi (er) random graph model, also known as g(n, p), is a classic model that has been widely studied [8]. the model yields a very homogeneous graph, with no special structure, in which each possible edge is present in the n node graph with probability p. in contrast, the barabási-albert (ba) model follows a iterative process driven by preferential attachment and generates a scale free network, better representing real networks such as the web and the as graph [1]. networks generated by these two models are strikingly different and both will be used in the evaluation. to allow for a direct comparison, the parameters of the two random graph models are chosen such that the generated networks are connected and have the same number of nodes and edges, on average. as for networks modeled after real life data, the facebook ego network is a social network consisting of the ego-networks of ten facebook users [13]. the power grid network represents the electrical power grid of the western united states with generators, transformers and substation acting as network nodes [22]. these two real networks are also very different in terms of node degree and distances. table 1 summarizes some details on the networks considered. table 1. network information (\(^*\) indicates average values).full size table for each random graph model and parameters determining a scenario (such as number of epidemic sources and infection ratio), 30 independent epidemic networks are generated and split equally into training, validation and testing sets. when considering real networks, 50 independent epidemic networks are generated, from which 1/5 is used for training, 1/5 for validation and 3/5 used for testing. the testing dataset is larger to ensure a better estimation of the average performance and allow a better comparison with prior works. 5.2 evaluation metrics accuracy is not an adequate metric to capture the performance of the framework, since there are very few epidemic sources (less than \(0.3\%\)) and thus a very unbalanced dataset. metrics that capture the ability of correctly identifying the sources are more adequate, hence taking into account true positives, false positives and false negatives. recollect that the proposed model outputs the probability that a node is an epidemic source. therefore, given an epidemic network, each node is used as input to the trained model and ranked according to their probability in decreasing order (most probable first). from this list, the top-k nodes are taken as epidemic sources, where k is parameter of the evaluation methodology. using this methodology, the precision and recall values for the top-k are computed, and consequently the f-score metric. note that k should be a function of the number of epidemic sources, which is given by \(s = |i_0|\). two different sets are considered as true positives when assessing the performance of the framework: i) the set \(i_0\), namely the epidemic source nodes; ii) the set \(o = \{ v | v \in i_0 \vee v \in n_1(u), u \in i_0\}\), namely the epidemic sources and the neighbors of the epidemic sources (\(n_1(u)\) denotes the neighbors of node u). note that o considers the neighbors of the epidemic source nodes as nodes classified correctly by the framework. obviously, identifying nodes that are neighbors of the epidemic source has much more value than nodes that are not neighbors. 5.3 results fig. 1. f-score for all four networks for both evaluation scenarios (w/ and w/o neighbors) for different number of epidemic sources (\(s \in \{3, 5, 10, 15\}\)) and different top-k values (as a function of s). in all scenarios, \(f_0=\) 20%. full size image figure 1 shows the performance of the proposed framework on all four networks (one network for each subfigure). each graph shows the f-score for the two scenarios, considering just the epidemic sources (set \(i_0\), w/o neighbors) and epidemic sources and their neighbors (set o, w/ neighbors). each curve corresponds to a different number of sources, \(s = |i_0|\) and the x-axis is the value for k in the k-top nodes, as a multiple of s. in all scenarios, \(f_o =\) 20%. it also shows that increasing k improves performance on all networks on the w/ neighbors evaluation scenario in most cases. one exception being facebook ego, where the performance decreases when s is 5 or 10. additionally, increasing the number of epidemic sources also improves performance on all networks on both scenarios, with the exception of ba networks. note that for er networks, performance clearly improves as s increases. in ba networks, having less epidemic sources yields better performance when considering w/o neighbors scenario. this occurs possibly because of the distinctive network topology where having a few epidemic sources allows for easier identification. fig. 2. precision for all four networks for both evaluation scenarios (w/ and w/o neighbors) for different number of epidemic sources (\(s \in \{3, 5, 10, 15\}\)) and different top-k values. in all scenarios, \(f_0=\) 20%. full size image interestingly, fig. 1 shows that it is not always the case that performance is better in w/ neighbors scenario. on the power grid network, leveraging neighbors clearly improves performance, while for ba network the opposite is observed, and for er networks results are mixed. therefore, w/ neighbors has greater precision but potentially smaller recall values, as the set of true positives has increased significantly. this intuition is verified in fig. 2. figure 2 shows precision values for the same scenarios as fig. 1. note that in all networks, precision for a fixed number of sources is always higher when considering the set o (w/ neighbors) in comparison to the set \(i_0\) (w/o neighbors). moreover, for er, facebook and power grid networks, having more epidemic sources improves precision for the o set in all scenarios. when considering \(s=15\), precision for er and facebook network are above 20% and 45% for all top-k values considered, respectively, while for the power grid precision is always below 8%. this indicates the importance of the network structure on identifying epidemic sources. 5.4 results with mixed datasets the previous results considered specialized datasets where all samples were independent and identically distributed, all having the same parameters. such samples were used for training, validation, and testing. however, it is also important to consider the performance of the framework when the datasets are more diverse and not composed by identically distributed samples. a heterogeneous dataset was generated for each network model. the samples in each dataset have different networks (from the same model), different number of sources (\(s \in \{3, 5, 10, 15\}\)) and different infection rates (\(f_o \in \{ 0.1, 0.2, 0.3 \}\)). the exception are real networks where the network is always the same (but all else is simulated to generate an epidemic network). this heterogeneous dataset is used for training, validation and testing a general model. results comparing the performance of the framework on the homogeneous and heterogeneous datasets are shown in fig. 3. each subfigure shows a different infection rate, \(f_o\) and each couple of bars correspond to different number of sources, s. performance on the homogeneous (specialized) dataset is almost always superior than performance on heterogeneous dataset. however, when the infection rate is small, e.g., \(f_o = 0.1\), the performance of the models are comparable for both er and ba networks for different number of sources. this indicates that the framework can be effectively trained on a more general dataset when the infection rate is small. indeed, when the infection rate is small, increasing the number of sources will still allow them to have a distinctive signature in the epidemic network. fig. 3. performance of network models using specialized/homogeneous dataset (individual) and generalized/heterogeneous datasets (general) for different number of sources (s) and different infection rates (\(f_o\)). full size image figure 3 also shows that as the infection rate increases, performance degrades for both homogeneous and heterogeneous datasets, but falls faster for the heterogeneous. in fact, performance is zero for \(f_o = 0.3\) and \(s \in \{3,5\})\) for er networks, showing that heterogeneous dataset trained model cannot adequately identify source nodes in more difficult scenarios. a curious exception was ba for \(f_o = 0.2\) where performance for the heterogeneous dataset was in some cases higher, \(s \in \{3,5\}\). figure 4 shows the performance of using the homogeneous and heterogeneous datasets when considering the two real networks. results clearly indicate that using a heterogeneous dataset is significantly worst than using a homogeneous dataset, even in the case \(f_o = 0.1\). results also corroborate that for a given infection rate, increasing the number of sources improves performance, for both homogeneous and heterogeneous datasets. moreover, results also corroborate that increasing the infection rate reduces the performance, and again for both datasets. note that performance under the homogeneous dataset is in some cases twice larger than the heterogeneous dataset, indicating that generalization is much harder for real networks. 5.5 comparison with baselines only a few prior works have evaluated the multiple epidemic source detection (msd) problem in scenarios that are reproducible and comparable. in fact, a contribution of this work is the evaluation of the proposed framework under well-studied network models and epidemic models that can be easily reproduced. an exception is gcnsi [3] that performed numerical evaluations on real world networks and reported their scenario with sufficient details, using f-score metric, while also reporting results for lpsi and netsleuth for the same scenarios. table 2 compares f-score results for 3, 5 and 10 epidemic sources when 30% of the network is infected, \(f_o = 0.3\), for gcnsi, lpsi and netsleuth approaches. recall that only gcnsi uses graph convolution networks. results for the proposed framework are shown in the row sage with maximum and average values across different values for k in the top-k ranking. fig. 4. performance of real networks using specialized/homogeneous dataset (individual) and generalized/heterogeneous datasets (general) for different number of sources (s) and different infection rates (\(f_o\)). full size image table 2. f-score values of different frameworks on real networks with 3, 5 and 10 epidemic sources. gcnsi, lpsi and netsleuth results extracted from [3].full size table note that results are generally poor across all table entries since reported f-score values are much closer to zero than to one, indicating that the msd is rather a difficult problem. however, results in general improve with the increasing number of epidemic sources. note that sage stands out showing the best performance across the table for all scenarios, for maximum and average performance. also, the gap between sage and the other models increase along with the number of sources: sage’s f-score on facebook ego with 10 sources are five times larger than the second best model. 6 conclusion identifying multiple epidemics source nodes from the observation of an epidemic network is a challenging problem that has recently been explored in the literature. this work proposed a framework based graph neural networks that starts by generating features for nodes from the observed epidemic network. these features capture the infection state of k-hop neighborhoods around the node. finally, a sage supervised trained model is used to rank nodes of a (never before seen) epidemic network with respect to their probabilities of being an epidemic source for that observation. the proposed model is generalizable and can be applied to any network, any si contagion model, and any number of epidemic sources. the performance of the proposed framework was evaluated on two random network models and two real networks, along with two evaluation scenarios (source nodes and source nodes with their neighbors). results highlight the importance of the network structure on the framework’s performance. besides, results indicate that performance improves when the number of sources is larger, and decreases as the fraction of infected nodes increases in the observation. results also indicate that a single model does not perform well when considering various scenarios (with the exception of low infection rates and random network models). finally, the proposed framework significantly outperformed recent prior works in all scenarios where a direct comparison was possible. in general, reported values for f1-score and other metrics (precision and recall) are low both here and in prior works, but this is an indication that detecting multiple epidemic sources is a challenging problem. future works can explore node features and classification model in search of more promising approaches. notes 1.accessible on: https://github.com/rodrigohaddad/multiple-source-detector-gnn. references barabási, a.l., pósfai, m.: network science. cambridge university press, cambridge (2016) google scholar  david, e., jon, k.: networks, crowds, and markets: reasoning about a highly connected world. cambridge university press, new york (2010) math  google scholar  dong, m., zheng, b., quoc viet hung, n., su, h., li, g.: multiple rumor source detection with graph convolutional networks. in: acm international conference on information and knowledge management (cikm), pp. 569–578 (2019) google scholar  draief, m., massouli, l.: epidemics and rumours in complex networks. cambridge university press, cambridge (2010) math  google scholar  feizi, s., médard, m., quon, g., kellis, m., duffy, k.: network infusion to infer information sources in networks. ieee trans. netw. sci. eng. 6(3), 402–417 (2019) article  mathscinet  google scholar  fey, m., lenssen, j.e.: fast graph representation learning with pytorch geometric. in: iclr workshop on representation learning on graphs and manifolds (2019) google scholar  firth, j., hellewell, j., klepac, p., kissler, s., kucharski, a., spurgin, l.: using a real-world network to model localized covid-19 control strategies. nat. med. 26, 1616–1622 (2020) article  google scholar  gilbert, e.n.: random graphs. ann. math. stat. 30(4), 1141–1144 (1959) article  math  google scholar  hamilton, w.l., ying, r., leskovec, j.: inductive representation learning on large graphs. in: neurips, p. 1025–1035 (2017) google scholar  jiang, j., wen, s., yu, s., xiang, y., zhou, w.: k-center: an approach on the multi-source identification of information diffusion. ieee trans. inf. forensics secur. 10(12), 2616–2626 (2015) article  google scholar  jiang, j., wen, s., yu, s., xiang, y., zhou, w.: identifying propagation sources in networks: state-of-the-art and comparative studies. ieee commun. surv. tutor. 19(1), 465–481 (2017) article  google scholar  kermack, w.o., mckendrick, a.g., walker, g.t.: a contribution to the mathematical theory of epidemics. roy. soc. lond. 115(772), 700–721 (1927) math  google scholar  leskovec, j., mcauley, j.: learning to discover social circles in ego networks. in: nips (2012) google scholar  newman, m.e.: spread of epidemic disease on networks. phys. rev. e 66(1), 016128 (2002) article  mathscinet  google scholar  pastor-satorras, r., castellano, c., van mieghem, p., vespignani, a.: epidemic processes in complex networks. rev. mod. phys. 87(3), 925 (2015) article  mathscinet  google scholar  paszke, a., gross, s., chintala, s.: pytorch: an imperative style, high-performance deep learning library. in: neurips, pp. 8024–8035 (2019) google scholar  prakash, b.a., vreeken, j., faloutsos, c.: spotting culprits in epidemics: how many and which ones? in: ieee international conference on data mining (2012) google scholar  shah, d., zaman, t.: rumors in a network: who’s the culprit? ieee trans. inf. theory 57(8), 5163–5181 (2011) article  mathscinet  math  google scholar  shah, d., zaman, t.: rumor centrality: a universal source detector. in: acm sigmetrics performance evaluation review, vol. 40, pp. 199–210 (2012) google scholar  shelke, s., attar, v.: source detection of rumor in social network-a review. online soc. netw. media 9, 30–42 (2019) article  google scholar  wang, z., wang, c., pei, j., ye, x.: multiple source detection without knowing the underlying propagation model. in: aaai conference on artificial intelligence (2017) google scholar  watts, d.j., strogatz, s.h.: collective dynamics of ‘small-world’ networks. nature 393(6684), 440–442 (1998) article  math  google scholar  yu, p.d., tan, c.w., fu, h.l.: epidemic source detection in contact tracing networks: epidemic centrality in graphs and message-passing algorithms. ieee j. sel. topics signal process. 16(2), 234–249 (2022) article  google scholar  zang, w., zhang, p., zhou, c., guo, l.: locating multiple sources in social networks under the sir model: a divide-and-conquer approach. j. comput. sci. 10, 278–287 (2015) article  mathscinet  google scholar  download references author information authors and affiliations system engineering and computer science (pesc), federal university of rio de janeiro (ufrj), rio de janeiro, brazil rodrigo gonçalves haddad & daniel ratton figueiredo authors rodrigo gonçalves haddadview author publications search author on:pubmed google scholar daniel ratton figueiredoview author publications search author on:pubmed google scholar corresponding author correspondence to rodrigo gonçalves haddad . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper haddad, r.g., figueiredo, d.r. (2023). detecting multiple epidemic sources in network epidemics using graph neural networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_22 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords network epidemic epidemic source identification graph neural network supervised learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature elevating healthcare ai: achieving efficiency and accuracy in medical applications with surrogate-based multiobjective compression of resnet50 cnns | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper elevating healthcare ai: achieving efficiency and accuracy in medical applications with surrogate-based multiobjective compression of resnet50 cnns conference paper first online: 31 january 2025 pp 137–151 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) elevating healthcare ai: achieving efficiency and accuracy in medical applications with surrogate-based multiobjective compression of resnet50 cnns download book pdf download book epub gabriel bicalho ferreira9, pedro silva9 & rodrigo silva9  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 233 accesses 1 citation abstract machine learning, particularly convolutional neural networks (cnns), has gained prominence in healthcare applications, including medical diagnosis and clinical support. however, the increasing size of cnns poses challenges in resource-constrained medical devices and real-time applications. this paper explores the effectiveness of pruning and quantization on the resnet50 model within the medmnist dataset, a valuable resource for medical image classification. the study evaluates a surrogate-based multiobjective compression method on three medmnist datasets: retinamnist for diabetic retinopathy grading, dermamnist for disease categorization, and bloodmnist for blood cell classification. results demonstrate that the proposed compression method successfully identifies less computationally intensive models while maintaining or improving accuracy across all three healthcare-related datasets. a reduction of about 50% in inference time and an increase of more than 1% in accuracy were observed. these findings emphasize the practicality of compression techniques in healthcare applications, particularly for resource-constrained environments and real-time decision-making scenarios. this research opens avenues for further validation and exploration in other healthcare-related applications with higher-quality neural network models, ultimately enhancing the deployment of machine learning in the healthcare domain. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others medical computer vision chapter © 2025 machine learning in healthcare analytics: a state-of-the-art review article 04 april 2024 prediction of frequent out-of-hours’ medical use chapter © 2020 1 introduction machine learning is a field of artificial intelligence that aims to create computational systems capable of acquiring knowledge through accumulated experiences and making automated decisions [15]. among the various applications of machine learning, its relevance in the healthcare field stands out, being employed in medical diagnosis, disease prediction, and clinical decision support [6]. these systems aim to assist and streamline the work of medical professionals. in this context, in 2021, medmnist [24] emerged as an extensive collection of biomedical images developed to fine-tune and test classification models in medical applications. the medmnist dataset consists of 708,069 2d images distributed across 12 distinct datasets, as well as 9,998 3d images divided into 6 distinct sets. all images are standardized to a resolution of 28\(\,\times \,\)28 pixels and cover various medical contexts, including binary and multiclass classifications. the data is pre-divided into training, validation, and test sets. in addition to creating the dataset, the authors benchmarked different versions of the resnet model [8] (resnet-18 and resnet-50) and three automated machine learning frameworks on each dataset. in the realm of deep learning techniques, convolutional neural networks (cnns), such as the resnets, have stood out as extremely effective in several real-world tasks involving processing visual data, such as object detection and image and video processing [17]. although recent advances in these techniques have led to an increase in the accuracy of cnns, they have significantly increased their size and inference time [14]. therefore, the research area of neural network compression techniques is gaining increasing prominence, particularly pruning and quantization, which have been extensively explored in the literature. broadly speaking, in pruning, less significant neuron connections are removed from the model. in quantization, the precision of the model parameters is reduced. in [14], it has been shown that different types of pruning present varying trade-offs in terms of compression, accuracy, and speedup. on the other hand, quantization demonstrated significant improvement in performance and reduced storage requirements. compressing machine learning models for medical applications is crucial for several reasons. first, many medical devices, such as portable ultrasound machines and wearable health monitoring devices, operate with limited computational resources. compressed models enable these devices to perform tasks with reduced memory and computational power, making them more suitable for resource-constrained environments. additionally, in medical settings, particularly during real-time applications like diagnosis and monitoring, the speed of inference is crucial. compressed models facilitate faster execution, thereby decreasing the time required for critical decision-making. also, compressed models typically consume less energy during inference, which is advantageous for battery-powered medical devices or situations where power conservation is important, as it can extend device runtime and reduce the need for frequent recharging or battery replacement. furthermore, from a patient’s perspective, the wait time for a diagnosis can be highly stressful. therefore, even a minor reduction in inference time can substantially alleviate patient anxiety. it is also noteworthy that minutes saved in a system performing numerous inferences can translate to substantial reductions in patient wait times overall. when applying pruning and quantization techniques, several decisions must be made, such as determining the extent of connections to prune and selecting the layers for quantization. previous research [2, 5, 9, 10, 22, 23] has shown that the compression challenge is essentially a multi-objective optimization problem, requiring the identification of optimal trade-offs between the computational efficiency of the model (in terms of time and memory consumption) and its accuracy. except for the approach presented in [9], which uses reinforcement learning, all other studies employ various types of evolutionary algorithms (eas) as optimization methods. however, a significant bottleneck in this process is the considerable number of function evaluations required to find satisfactory solutions, especially challenging in the compression problem since the objective function requires evaluating the compressed model over the entire test set. the use of evolutionary algorithms for problems with expensive function evaluations is a well-known concern [4]. a well-established technique to address this issue is the use of surrogate models, also called meta-models [19, 20]. the idea is to build a computationally inexpensive model to substitute an expensive objective and constraint functions and use this cheaper model to guide the optimization process instead to evaluate the expensive functions over the entire test set. the primary goal of this work is to understand the impact of model compression techniques on the performance of convolutional neural networks (cnns) in health-related applications. specifically, we employ a surrogate-based multi-objective compression methodology on the resnet50 model, which has already been extensively testes on the medmnist dataset in [24]. this technique is evaluated on three medmnist datasets: retinamnist for grading diabetic retinopathy severity, dermamnist for categorizing seven different diseases, and bloodmnist for classifying blood cells into eight categories. the results indicate that the proposed surrogate-based multi-objective compression method effectively identifies less computationally intensive models while maintaining or even enhancing accuracy across all three datasets. this underscores the practicality and significance of employing compression techniques, especially in the healthcare domain. extending this technique to other healthcare-related applications with higher-quality neural network models holds promising potential for further validation and exploration. notably, there was an average reduction of 50% in inference time on average and an improvement of over 1% in accuracy. 2 artificial neural network compression as a multi-objective optimization problem deep artificial neural networks are robust, complex, and highly effective structures for solving various types of problems. however, their high computational cost can hinder their implementation on different devices [22]. in this section, we provide a brief overview of two compression techniques, pruning and quantization, including the relevant parameters that directly influence the final performance and quality of the compressed model. 2.1 prunning and quantization pruning, within the context of neural networks, can be defined as a compression technique. it involves the selective removal of connections or neurons with low importance scores while preserving the network’s accuracy, as discussed in previous works [14, 21]. these importance scores can be determined based on various criteria, including weight magnitude, sensitivity, or activation level, as outlined in [21]. the pruning process typically consists of three key steps: (1) assessing the importance scores of each connection or neuron, (2) eliminating the connections or neurons with the lowest scores, and (3) fine-tuning the remaining network to restore its accuracy. for achieving higher compression rates, pruning can be applied iteratively, and it can also be combined with other compression techniques, such as quantization, to further reduce the network’s size and complexity. quantization, as applied to an artificial neural network (ann), serves as a model compression technique aimed at diminishing the number of bits required to represent each of the ann’s parameters [7]. typically, ann parameters are represented using 32-bit floating-point numbers, thereby inflating both storage requirements and computational intricacies. lowering the precision of these parameters not only accelerates model performance and curtails storage expenses but also retains adequate information for effective model inference, all while upholding accuracy [22]. 2.2 compression design variables the implementation of these compression techniques requires the specification of certain parameters that may vary based on the chosen approach. when using pruning, one must define whether it will apply to only convolutional layers, only linear layers, or both. once the type of layer is selected, one must specify which layers to prune and the pruning intensity, which is the proportion of weights and biases that will be pruned in a layer. additionally, quantization can be applied to the previously pruned structures. therefore, seven design variables have been defined to optimize the compression process and achieve the most suitable resulting model. below, the variables and their respective domains are outlined: \(x_1\) (linear layer pruning): \(x_1\) \(\in \) {0 (no pruning in the linear layers), 1 (prune all linear layers), 2 (prune a random set of linear layers)} \(x_2\) (convolutional layer pruning): \(x_2\) \(\in \) {0 (no pruning in the convolutional layers), 1 (prune all convolutional layers), 2 (prune a random set of convolutional layers)} \(x_3\) (pruning intensity for the linear layer (%)): \(0\%<x_3<100\%\) \(x_4\) (pruning intensity for the convolutional layer (%)): \(0\%<x_4<100\%\) \(x_5\) (pruning type for the linear layer): \(x_5\) \(\in \) {0 (prune weights), 1 (prune biases), 2 (prune both weights and biases)} \(x_6\) (pruning type for the convolutional layer): \(x_6\) \(\in \) {0 (prune weights), 1 (prune biases), 2 (prune both weights and biases)} \(x_7\) (quantization): \(x_7\) \(\in \) {0 (do not perform quantization), 1 (perform quantization)} variables \(x_1\) to \(x_6\) refer to compression by pruning, and \(x_7\) defines whether quantization is applied or not. 2.3 objective functions as mentioned before, the compression problem commonly posed as a multi-objective optimization problem which aims at finding solutions with a good balance between the quality and performance (execution time and space) of the resultant model. more specifically, in this work, the compression problem is formulated as follows: (1) where, \(\textbf{x} = \left[ x_1, x_2, x_3, x_4, x_5, x_6, x_7\right] \) is defined over the search space presented in sect. 2.2 and \(\tau \) = (0.96 \(\times \) base model accuracy). that is, we allow for a decrease in accuracy of 4%. the non-zero weights rate, which reflects the sparsity of the weight and bias tensors in the neural network model, is directly impacted by the pruning technique. on the other hand, the number of flops (floating-point operations per second) is a metric that is reduced with the application of quantization. when we multiply the flops by the non-zero weights rate, we obtain an objective \(f_1\) that is affected by both pruning and quantization. minimizing \(f_1\) entails increasing compression levels, which can be desirable in terms of computational efficiency and resource utilization. conversely, maximizing the objective \(f_2\) improves the overall quality of the neural network, as measured by relevant performance metrics such as accuracy and f-score. when analyzing the problem of neural network compression in the literature, it is possible to observe a conflict between the amount of pruning performed and the accuracy obtained. this conflict is demonstrated in studies such as [22] and [14]. thus, this paper addresses a multi-objective compression problem that involves seven variables (see sect. 2.2), two objectives, and an accuracy constraint, which aims to avoid spending computational resources on the evaluation of poor solutions. 3 multi-objective constrained optimization based on surrogate models to find compression parameters that lead to good models, we adapt the approach proposed in [19] to the problem defined in eq. (1). it consists of the following steps: 1. build an initial dataset \( d = \langle x,y \rangle \) for generating the surrogate models. \( x \) is a set of \( n \) points sampled from the search space defined in sect. 2.2, and \( y \) stores the values of the objectives (flops \(\times \) non zero weights rate, accuracy) and the constraint (minimum allowed accuracy), as defined in eq. (1), corresponding to the set \( x \). 2. generate a set, \( s_{o1} \), of surrogate models for the first objective. 3. generate a set, \( s_{o2} \), of surrogate models for the second objective. 4. generate a set, \( s_{c} \), of surrogate models for the constraint. 5. given a quality metric, \( m \) (e.g. mean average percentage error), use cross-validation to select the best surrogate in each set. 6. build a surrogate problem with the best surrogate models. 7. solve (approximately) the surrogate problem with a multi-objective evolutionary algorithm (moea). 8. randomly sample \( k \) feasible non-dominated solutions obtained by the moea. these will be called from now on as infill points. 9. evaluate these infill points using the original objective and constraint functions, and add the results to the dataset \( d \). 10. repeat steps 2 to 9 until the allotted budget for function evaluations is exhausted. 11. return \( d^* \), which is the set of feasible non-dominated solutions in \( d \). selecting an appropriate model for the objective functions and constraints is a critical step for the performance of any surrogate-based optimization algorithm. in this work, we have chosen to use the mean absolute percentage error (mape) as the quality metric. mape is a statistical metric used to measure the average absolute percentage differences between the predicted values and the actual values in a dataset. it is mathematically defined by: $$\begin{aligned} \frac{1}{n} \left( \sum _{i=0}^{n-1}\frac{|y_i \hat{y}_i|}{max(\epsilon , |y_i|)} \right) \end{aligned}$$ (2) where n represents the number of samples, \(y_i\) is the actual value, \(\hat{y}_i\) is the predicted value, and \(\epsilon \) is a very small positive value to avoid division by zero. because it penalizes errors more for values of lower magnitude, it emerges as a valuable quality metric for optimization, as demonstrated in [19]. the employed moea is the two-archive evolutionary algorithm for constrained multiobjective optimization (ctaea) [12]. as demonstrated in the literature, ctaea is an excellent algorithm for solving constrained multi-objective problems [1, 13]. 4 experimental setup in this section we introduce the used datasets, the resnet50 model, which is the base model, as well as the parameter setting for the method presented in sect. 3. 4.1 datasets the medmnist dataset [24] encompasses an extensive collection of biomedical images organized into various categories. for the experiments in this study, we chose the following three 2d datasets containing color images standardized to a 3\(\,\times \,\)28\(\,\times \,\)28 format: 1. retinamnist: this dataset contains 1,600 retinal fundus images, divided into 1,080 images for training, 120 for validation, and 400 for testing. each image is associated with labels representing five distinct classes, making it suitable for an ordinal regression task aimed at classifying the severity of diabetic retinopathy into five different levels [24]. 2. dermamnist: a comprehensive collection of dermatoscopic images of common pigmented skin lesions. the dataset comprises 10,015 dermatoscopic images categorized into 7 different diseases, forming a multi-class classification task [24]. the dataset is divided into 7,007 images for training, 1,003 for validation, and 2,005 for testing. 3. bloodmnist: this dataset is derived from individual normal cells captured from individuals without infections, hematologic or oncologic diseases, and who were not undergoing pharmacological treatment at the time of blood collection. it encompasses a total of 17,092 images distributed across 8 distinct classes [24]. there are 11,959 images allocated for training, 1,712 for validation, and 3,421 images for testing. 4.2 resnet50 similar to the dataset selection process, the choice for the resnet architecture was influenced by its well-established presence in the literature, enabling comparisons across different studies. specifically, we opted for the resnet50 architecture [8], a common choice in various studies concerning neural network compression. moreover, this model was also the best-performing model among those tested on these datasets by [24]. the resnet, short for residual network, is a deep convolutional neural network model that introduced residual connections which allow the network to learn residual functions and thus alleviate the degradation in accuracy that often accompanies increasing network depth. resnet50 has 50 layers, 49 of which are convolutional and only one is fully connected. the convolutional filters follow the standard size of 3\(\,\times \,\)3, and after the convolution process, data normalization is performed. this architecture has about 26 million parameters. to train this architecture, the same parameters used in [24] were adopted, which are: (i) epochs: 100; (ii) learning rate: \(1e-3\); (iii) l2 regularization penalty: \(1e-3\); (iv) batch size: 128; (v) loss function: cross-entropy; (vi) optimizer: adam. 4.3 experimental procedure to evaluate the proposed surrogate-based approach on the compression performance, we tested the methodology presented in sect. 3 as follows: 1. load the untrained version resnet50 from pytorch. 2. train the loaded architecture for retinamnist, dermamnist, and bloodmnist training sets using the same parameters of [24]. 3. run five independent executions of the procedure presented in sect. 3 to try to solve the compression problem defined by eq. (1). the parameters used in all the runs were: (a) initial sampling: 50 function evaluations. (b) optimizer: ctaea. (c) number of generations: 1000. (d) population size: 50. (e) evaluation function budget: 100. (f) infills per iteration: 2. (g) surrogate models: kneighborsregressor [16], randomforest [16], xgboost [3], lightgbm [11]. (h) surrogate selection metric: mape. (i) number of independent runs: 5 for each database. 4. the quality of each compressed solution is assessed with retinamnist, dermamnist or bloodmnist test set. 5. the final set of all the evaluated solutions as well as the hypervolume of the non-dominated set are stored for each run. the hypervolume is defined as: definition 1 (hypervolume indicator). given a point set \(s \subseteq \mathbb {r}^d\) and a reference point \(r \in \mathbb {r}^d\), the hypervolume indicator of s is the measure of the region weakly dominated by s and bounded above by r, i.e.: $$\begin{aligned} h(s) = \varlambda (\{ q \in \mathbb {r}^d | \exists p \in s: p \le q \text { and } q\le r\}) \end{aligned}$$ (3) where \(\varlambda (\cdot )\) denotes the lebesgue measure. 6. the friedman test was used to compare the different algorithms and algorithm versions and, when necessary, the dunn’s test was used as a post-hoc. the adopted confidence level \(\alpha \) was set to 0.05. to ensure that the surrogates were not negatively affecting compression, we conducted a test using pure ctaea without surrogate models. the experiment was performed within the same budget of 100 function evaluations, which roughly translates to 5 h of cpu time on machines with an intel(r) xeon(r) 2.200 ghz, 12.68 gb ram and a disk of 107.72 gb. it should be noted that gpus were not used in this study due to the limitations of the current quantization implementation [18]. the pure ctaea was tested in two different configurations: (i) 10 generations with a population size of 10, and (ii) 20 generations with a population size of 5. this choice was made to minimally strike a balance between having a sufficiently large population size and providing enough generations for the population to evolve and improve. 5 results in this section, we analyze the performance of our methodology from three different perspectives. first, in sect. 5.1, we evaluate the accuracy of our base models in comparison to the resnet50 model as described in [24]. second, in sect. 5.2, we compare our proposed compression method, which utilizes surrogates, to the original versions of the optimizer. finally, in sect. 5.3, we assess the trade-offs between quality and performance for the compressed model relative to the base models. 5.1 accuracy of base models prior to compression we acknowledge that model performance can fluctuate slightly due to factors such as the use of different frameworks, random splitting into mini batches, random weight initialization, among others. to ensure that our base models (resnet50 before compression) achieve accuracy comparable to benchmarked results, we compare the implemented models with the results obtained by [24] in this section. table 1 presents the obtained results. table 1. accuracy of implementationsfull size table as shown in table 1, the accuracy difference between our implementation and yang’s was within 2% for datasets where it was lower and approximately 3% for those where it was higher. therefore, we conclude that we have reasonably accurate starting models to test the method presented in sect. 3. 5.2 assessing the impact of employing surrogate models in the optimization process in this section, we aim to determine whether the surrogate-based approach confers an advantage over the standard, non-surrogate-based version of the ctaea by performing a comparison across three different datasets. figure 1 shows the box plots for the surrogate-based methodology alongside the two standard ctaea configurations. specifically, ctaea1 employs 20 individuals that undergo 5 generations of evolution, while ctaea2 uses 10 individuals that undergo 10 generations of evolution. as shown in fig. 1(a), the approach employing surrogate models yielded larger hypervolumes for retinamnist. the box plots also reveal lower variance in the values, indicating that the proposed method leads to more robust solutions, characterized by higher average hypervolumes and a lower standard deviation. fig. 1. comparison between the surrogate-based and the raw version of ctaea resnet50 on (a) retinamnist, (b) dermamnist, and (c) bloodmnist. the hypervolume scale is \(1e^7\). full size image when we analyze the results for the dermamnist dataset, as illustrated in fig. 1(b), we observe similar outcomes to those obtained with the retinamnist dataset. the approach utilizing surrogate models demonstrated advantages over the standard ctaea model, and this advantage became even more pronounced for the dermamnist dataset. additionally, we notice a lower standard deviation, indicating that the results were again more consistent. finally, fig. 1(c) highlights the results for the bloodmnist dataset. once again, we can observe that the surrogate-based approach yields better hypervolumes for the non-dominated sets and more consistent results in terms of solution quality. thus, in all the analyzed cases, the surrogate-based approach outperformed both methods utilizing the simple ctaea (p < 0.05). it is important to notice that the use of ctaea1 resulted in highly variable outcomes between runs, while ctaea2 demonstrated slightly greater consistency. however, for the dermamnist dataset, ctaea2 exhibited exceptionally unfavorable results. 5.3 analysis of the non-dominated set of compressed solution figure 2 shows the non-dominated solutions with the lowest hypervolume, that is, the worst set of solutions among the five runs, achieved by surrogate-based multiobjective approach. fig. 2. resnet50 objective function space on first column and test set inference time by non-zero weights rate on second column. the first line are related to the retinamnist, second to dermamnist and the last one, bloodmnist. full size image in fig. 2(a) and 2(b), we can see the results for the retinamnist dataset. the points in blue were obtained with the proposed approach. the green ‘x’ represents the base model performance before compression and the red triangle, the results from the medmnist [24]. on fig. 2(a), we can see that the compression by pruning and quantization led to models with less flops which in some cases presented better accuracy. however, we did not optimize for inference time, on fig. 2(b), we can see that in the test set it was reduced from 30 s to less than 15 s for the compressed models. the inference time for the [24] models was not reported. in figs. 2(c), 2(d), 2(e) and 2(f), we present the results for the dermamnist and bloodmnist datasets, respectively. the dots represent the compressed solutions, with blue indicating the use of both pruning and quantization, and orange indicating the use of only pruning. the compression strategy was determined by the optimization algorithm. the base model is represented by a green ‘x’, and the medmnist model by a red triangle. the results for dermamnist were similar to those obtained for retinamnist. the compressed models showed a significant reduction in the number of floating-point operations (flops) and halved the inference time. for bloodmnist, as shown in figs. 2(e) and 2(f), the inference time was again reduced by about half in the compressed model. however, in terms of accuracy, the models did not surpass the results of [24]. nonetheless, they still improved accuracy over the base model. it is evident that most of the non-dominated solutions involve the joint application of pruning and quantization techniques for model compression. quantization plays a crucial role in this process, substantially reducing the number of floating-point operations per second (flops) and shrinking the network size in megabytes (mb). on the other hand, the pruning technique results in a drastic reduction in the rate of non-zero weights, enabling simpler operations and opening the possibility of applying alternative encodings to the neural network structure, such as sparse matrix encoding, which also reduces the size in megabytes (mb) of the neural network. the combination of these two techniques results in substantially less complex neural networks with significantly shorter inference times compared to the original architecture. the right charts of fig. 2 illustrate this comparison, showing that the compressed networks are, on average, more than twice as fast. confusion matrices of the most accurate compressed solutions: the methodology adopted in this study chose accuracy as the primary metric for comparison. however, especially in medical contexts, it is relevant to explore other metrics that address different analyses, such as false positives, false negatives, among other characteristics. therefore, fig. 3 displays confusion matrices for the three datasets, considering both the original and compressed solutions that resulted in higher accuracy. fig. 3. confusion matrix for the original architecture and its compressed version, highlighting the higher accuracy. first line is related to retinamnist, the second, dermamnist and the last line, bloodmnist. full size image there are a few things that can be observed. first, as observed in the previous section, there is a slight increase in accuracy of the compressed model in comparison with the base model. also, the profile of false positives and false negatives is quite similar between compressed and base models except in the retinamnist where the compressed model was more biased towards the majority class. considering the limitation in the resources and that one of the objectives was the accuracy, this kind of result is somewhat expected. interestingly, such bias was not observed in dermamnist which also presents a significant class imbalance. overall, the compressed models retain the hits and misses of the base models which indicates that knowledge about the base model may be transferred to the compressed model. several patterns emerge from the analysis of the compressed models. similar to observations in the previous section, there is a modest increase in the accuracy of the compressed models compared to the base models. furthermore, the tendency of false positives and false negatives remains largely consistent between the compressed and base models. however, an exception was noted in the retinamnist dataset, where the compressed model exhibited a greater bias toward the majority class. this outcome aligns with the resource constraints and the objective to maintain accuracy, suggesting that some degree of bias toward the dominant class is an anticipated result of the compression process. notably, such a bias did not manifest in the dermamnist dataset, despite its significant class imbalance. this indicates that while the compression process can introduce or accentuate biases in some cases, it does not do so uniformly across different datasets. the variability in these outcomes underscores the need for broader experimentation to fully understand the typical effects of model compression across various datasets. 6 conclusion the proposed surrogate-based optimization approach has showcased its ability to significantly reduce computational overhead while maintaining or even enhancing the accuracy of the models across the three distinct datasets: retinamnist, dermamnist, and bloodmnist. particularly, the compressed models have demonstrated a substantial reduction in floating-point operations (flops) and inference times, which is critical in medical applications where timely and accurate diagnosis is paramount. although the compression process introduced a bias towards majority classes in the retinamnist dataset, this was not observed in the dermamnist dataset, indicating that the effects of compression techniques can vary across different datasets and applications. these findings suggest that the surrogate-based approach is a promising strategy for optimizing the trade-offs between model size, computational efficiency, and performance in healthcare-related applications. the ability to deploy lightweight yet accurate models on resource-constrained medical devices could significantly improve real-time diagnostics and patient monitoring, making advanced healthcare more accessible. future research should aim to validate these techniques across a broader array of medical datasets and deep learning models. besides, further refinement of the surrogate models to enhance their predictive accuracy and reliability can still improve the methodology. the ultimate goal is to facilitate the integration of efficient, high-performing cnns into the healthcare sector, thereby contributing to the advancement of medical ai and improving patient outcomes. references blank, j., deb, k.: pymoo: multi-objective optimization in python. ieee access 8, 89497–89509 (2020) article  math  google scholar  chen, j., xu, y., sun, w., huang, l.: joint sparse neural network compression via multi-application multi-objective optimization. appl. intell. 51(11), 7837–7854 (2021). https://doi.org/10.1007/s10489-021-02243-3 article  math  google scholar  chen, t., guestrin, c.: xgboost: a scalable tree boosting system (2016). https://xgboost.readthedocs.io. accessed 20 apr 2023 coello coello, c.a., gonzález brambila, s., figueroa gamboa, j., castillo tapia, m.g., hernández gómez, r.: evolutionary multiobjective optimization: open research areas and some challenges lying ahead. complex intell. syst. 6, 221–236 (2020) article  math  google scholar  fernandes, f.e., jr., yen, g.g.: pruning deep convolutional neural networks architectures with evolution strategy. inf. sci. 552, 29–47 (2021) article  mathscinet  math  google scholar  habehh, h., gohel, s.: machine learning in healthcare. curr. genomics 22(4), 291 (2021) article  math  google scholar  han, s., mao, h., dally, w.j.: deep compression: compressing deep neural networks with pruning, trained quantization and huffman coding. arxiv preprint arxiv:1510.00149 (2015) he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  hirsch, l., katz, g.: multi-objective pruning of dense neural networks using deep reinforcement learning. inf. sci. 610, 381–400 (2022) article  math  google scholar  hong, w., yang, p., wang, y., tang, k.: multi-objective magnitude-based pruning for latency-aware deep neural network compression. in: bäck, t., et al. (eds.) ppsn 2020. lncs, vol. 12269, pp. 470–483. springer, cham (2020). https://doi.org/10.1007/978-3-030-58112-1_32 chapter  google scholar  ke, g., et al.: lightgbm: a highly efficient gradient boosting decision tree (2017). https://lightgbm.readthedocs.io. accessed 20 apr 2023 li, k., chen, r., fu, g., yao, x.: two-archive evolutionary algorithm for constrained multiobjective optimization. ieee trans. evol. comput. 23(2), 303–315 (2018) article  math  google scholar  li, l., fan, y., tse, m., lin, k.y.: a review of applications in federated learning. comput. ind. eng. 149, 106854 (2020) article  math  google scholar  liang, t., glossner, j., wang, l., shi, s., zhang, x.: pruning and quantization for deep neural network acceleration: a survey. neurocomputing 461, 370–403 (2021) article  math  google scholar  monard, m.c., baranauskas, j.a.: conceitos sobre aprendizado de máquina. sistemas inteligentes-fundamentos e aplicações 1(1), 32 (2003) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  pouyanfar, s., et al.: a survey on deep learning: algorithms, techniques, and applications. acm comput. surv. (csur) 51(5), 1–36 (2018) article  google scholar  pytorch: quantization tutorial (2023). https://pytorch.org/docs/stable/quantization.html. accessed 29 june 2023 silva, r.c., li, m., rahman, t., lowther, d.a.: surrogate-based moea/d for electric motor design with scarce function evaluations. ieee trans. magn. 53(6), 1–4 (2017) math  google scholar  tong, h., huang, c., minku, l.l., yao, x.: surrogate models in evolutionary single-objective optimization: a new taxonomy and experimental study. inf. sci. 562, 414–437 (2021) article  mathscinet  math  google scholar  vadera, s., ameen, s.: methods for pruning deep neural networks. ieee access 10, 63280–63300 (2022) article  math  google scholar  wang, z., luo, t., li, m., zhou, j.t., goh, r.s.m., zhen, l.: evolutionary multi-objective model compression for deep neural networks. ieee comput. intell. mag. 16(3), 10–21 (2021) article  math  google scholar  xu, k., zhang, d., an, j., liu, l., liu, l., wang, d.: genexp: multi-objective pruning for deep neural network based on genetic algorithm. neurocomputing 451, 81–94 (2021) article  math  google scholar  yang, j., shi, r., ni, b.: medmnist classification decathlon: a lightweight automl benchmark for medical image analysis. in: 2021 ieee 18th international symposium on biomedical imaging (isbi), pp. 191–195. ieee (2021) google scholar  download references acknowledgments this work was supported by cnpq national council for scientific and technological development, capes coordination for the improvement of higher education personnel, fapemig minas gerais state research support foundation, ufop federal university of ouro preto. author information authors and affiliations computing department, federal university of ouro preto, ouro preto, brazil gabriel bicalho ferreira, pedro silva & rodrigo silva authors gabriel bicalho ferreiraview author publications search author on:pubmed google scholar pedro silvaview author publications search author on:pubmed google scholar rodrigo silvaview author publications search author on:pubmed google scholar corresponding author correspondence to gabriel bicalho ferreira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ferreira, g.b., silva, p., silva, r. (2025). elevating healthcare ai: achieving efficiency and accuracy in medical applications with surrogate-based multiobjective compression of resnet50 cnns. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_10 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords healthcare artificial intelligence convolutional neural networks neural network compression publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ensemble of protein stability upon point mutation predictors | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ensemble of protein stability upon point mutation predictors conference paper first online: 28 november 2021 pp 73–88 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) ensemble of protein stability upon point mutation predictors download book pdf download book epub eduardo kenji hasegawa de freitas  orcid: orcid.org/0000-0002-1476-477910, alex dias camargo  orcid: orcid.org/0000-0002-6377-937910, maurício balboni  orcid: orcid.org/0000-0001-7653-096210, adriano v. werhli  orcid: orcid.org/0000-0001-7107-502410 & … karina dos santos machado  orcid: orcid.org/0000-0002-8966-570810  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1240 accesses abstract computational determination of protein stability upon point mutations is very useful in a wide field of applications. the reliability of such computational predictions is essential. unfortunately, existing computational tools frequently disagree in their results. in the present study, the usage of ensemble learning algorithms to aggregate the results from different stability prediction tools is investigated. techniques of stacking, bagging, and boosting as well as different machine learning algorithms as combiner function are explored. all the investigation is carried out in real dataset protherm for which experimental results are known. the proposed methodology was validated considering two different experiments according to the training set. results show that our proposed ensemble approach is appropriate to predict the effect of point mutations on protein stability showing more reliable results than the individual tools improving overall accuracy, precision, and/or recall. this study was supported by capes edital biologia computacional (51/2013), capes financial code 001 and cnpq (439582/2018-0). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others computational approaches for predicting mutant protein stability article 09 may 2016 a way forward for predictors of protein thermodynamics stability changes due to mutations in the unavailability of experimental protein structures chapter © 2024 proteus: an algorithm for proposing stabilizing mutation pairs based on interactions observed in known protein 3d structures article open access 01 july 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. learning algorithms protein structure predictions protein folding protein function predictions structure prediction bioinformatics 1 introduction the latest research and development in biological experiments and techniques have created a deluge of experimental data. to deal with this quantity and variety of data the necessity of computational methods and tools is obvious. in some cases, the use of data science methodology is mandatory. in all living species, biological information is stored in the dna (deoxyribonucleic acid). dna can be seen as the source code that contains all the instructions to create and sustain life. to transform the information contained in the dna into functioning parts, dna is transcript as rna (ribonucleic acid) which is then translated into proteins. proteins are the principal biological elements responsible for executing biological functions. proteins can be thought of as the compiled result of the information stored in the dna [23]. moreover, proteins are the most abundant organic molecules in living systems and have the widest range of functions among macro-molecules. these functions can be structural, regulatory, contracting, or protective. furthermore, they can serve as transport, storage, or membranes, as well as being enzymes or toxins. each cell in a living system has thousands of proteins where each one has a function, making their structures very varied. proteins are the biological elements that execute functions and frequently they are modified (mutated) to achieve specific goals. these designed changes, mutations, are carried out in specific parts of the dna using for example site-directed mutagenesis. these mutations can produce a strong impact on the structure and function of proteins [17, 19]. however, due to the vast number of possibilities, it is impractical to experimentally evaluate the impact of all possible mutations on protein structure and function [25]. a series of computational tools have been developed, with the common objective of predicting the effect of point mutations on the structure of a protein. in general, prediction tools are developed and used by professionals from different areas who, generally, do not access the same tools for their analysis, which implies discrepancies in results and difficulties in interpretation, as they will have different results for the same data entries. among the many tools available we can cite: dmutant [30], foldx [12], i-mutant2.0 [4], cupsat [17], eris [28], i-mutant3.0 [5], popmusic [7], sdm [25], mcsm [18], mupro [6], maestro [14], and duet [19]. this computationally predicting the impact on proteins structure, and function, upon point mutations is of great scientific and commercial interest. one way of performing such predictions is to compute the \(\vardelta \vardelta g\) (variation in the variation of the free energy). however, the existing methods and tools do not always obtain similar results for the same entries. this differences in results obtained from distinct tools might impact negatively in the experiments carried out by researchers. the resulting \(\vardelta \vardelta g\) from the computational tools is usually discretized in stabilizing and destabilizing and this is an important information to define whether a mutation is viable or not. in this context, where we have several answers for the same problem, ensemble learning techniques can be a good alternative. ensemble learning aims to aggregate predictions from different classifier models in just one result. these combinations can be achieved in many different ways, e.g., average, major voting, etc. the resulting ensemble classifier usually obtains better results than the individual results of the classifiers used in the combination [27]. therefore, the present work proposes the application of ensemble learning to aggregate the protein stability predictions of individual tools. techniques of stacking, bagging, and boosting as well as different machine learning algorithms as combiner function are explored. the ensemble models were trained and validated using data from protherm [2], a database composed by sequence and structural information obtained from experimental methods for wild-type and mutant proteins. the paper is organized as follows: sect. 2 presents bioinformatics and machine learning brief concepts related to this purpose; sect. 3 detailed the individual tools used on the ensemble; sect. 4 describes the proposed methodology; sect. 5 presents the results and discussion and finally sect. 6 concludes the paper and show some future work. 2 background 2.1 point mutations and their effects on protein structures dna is formed by a sequence of molecules, or base pairs. some parts of dna are transcript as rna which in turn is translated as a protein. in translation, each three bases of rna is translated as one amino acid. the order of amino acids determines the 3d structure and consequently the function of proteins. a point mutation is when a single base pair is exchanged [1, 29]. they can be: i) silent: when the resulting amino acid is the same, no changes in the resulting protein; ii) missense: when the resulting amino acid is different from the original hence the protein is different and iii) nonsense: the resulting mutations indicate a stop signal, again the result is a modified protein. in this study, the focus is on point mutations that results in an exchange of amino acids and thus in a modified protein. 2.2 gibbs free energy (g) gibbs free energy is stabilized when the protein is in constant equilibrium, both in pressure and temperature. therefore, the protein in its native state (or wild type, wt) has a certain amount of energy \(g_{\text {wt}}\). when the protein folding process occurs, the gibbs free energy undergoes a variation, providing a variation of energy \(\vardelta g_{\text {wt}}\). in the same way that, if we analyze the same protein in a mutated state, we have, in its equilibrium, an energy \(g_{\text {mutant}}\) and, when this mutated protein folds, we can calculate the \(\vardelta g_{\text {mutant}} \). then, with the folding of the native and mutant proteins, one can calculate the \(\vardelta \vardelta g\), which is calculated using the eq. 1 [10, 21]. the \(\vardelta \vardelta g\) is a metric for predicting how a single-point mutation affects protein stability. with this, the present work uses the \(\vardelta \vardelta g\) as the main attribute to be evaluated, where the unit \(\text {kcal/mol}\) is used for the quantitative values. $$\begin{aligned} \vardelta \vardelta g = \vardelta g_{\text {wt}} \vardelta g_{\text {mutant}} \end{aligned}$$ (1) 2.3 supervised machine learning machine learning (ml) investigates how computers can learn based on data and has proven to be of great practical value in a variety of application domains [13]. supervised learning is the ml task of learning a model (or function) that maps an input to an output based on example input-output pairs. thus, the objective of these tasks is to predict the value of a particular attribute (target) based on the values of other attributes (explanatory or independent variables) [22]. since the data considered in this work has a target value, the \(\vardelta \vardelta g\), we are applying supervised ml. this target is discretized between stabilizing or destabilizing (sect. 2.2). in this way, we are applying as meta-classifiers in ensemble learning tasks the classification algorithms decision trees (dt), multilayer perceptron (mlp), naive bayes (nb) and support vector machines (svm). 2.4 ensemble learning ensemble learning is a popular method that combines multiple algorithms to make a decision usually in supervised ml [8]. ensembles have been applied in several research areas as economy, logistic, medicine, education and so on. in bioinformatics, it has been applied in different contexts as reviewed by yang [26]. in eickhotl & cheng [9], the authors proposed to apply boosting ensembles of deep networks for a sequence based prediction of protein disorder. mendonza et al. [15] discuss the use of ensemble-based solutions for inference of gene regulatory networks and prediction of micrornas targets. saha et al. [20] applied an ensemble learning method on majority voting on datasets of protein-protein interactions data. the main premise of this method is that by combining multiple models, the errors of a single predictor can be compensated by others [8]. moreover, the objective of ensemble methods is not only to improve the predictions but also to enhance their capability of generalization. an ensemble method for classification consists in a set of k models, \(m_{1}\), \(m_{2},\ldots , m_{k}\), used to build an improved predictive model. a dataset d is used to obtain k, \(d_{1}\), \(d_{2},\ldots ,d_{k}\) datasets in which \(d_{i}\) (1 \(\le \) i \(\le \) k-1) is used to build the classification model \(m_{i}\) [13]. there are several methodologies for forming an ensemble considering the k induced models: modifying the \(d_{1}\), \(d_{2},\ldots ,d_{k}\) datasets (sub-sampling, bagging, randomly sampling, etc.), modifying the learning task, exploiting the predictive algorithms characteristics and so on. thus, given a new instance to classify, each one of the induced models contributes to compose the final decision of the ensemble. combining the predictive models can reduce the risk to choose a bad result since the result of the combination usually is better than the best induced model. the definition of the best combination of classifiers is a combinatorial problem and there are different implementations of ensemble methods. in this paper we propose to generate stacking [16] , bagging [3] and boosting [11] ensemble models combining different classification algorithms. stacking [16] is a heterogeneous ensemble method where the models are obtained considering different algorithms and one unique dataset. then the answers of the predictive models are combined in a new dataset [31] and a meta-classifier is introduced. bagging and boosting are homogeneous ensemble methods. in bagging [3], the individual models are built parallel, are different from each other and all the instances have the same weight. in boosting [11] individual models are built sequentially where the outputs of the first model is used on the next and so on. so, during training, the algorithm defines weights for the instances and the ensemble models are incrementally built. 3 individual tools for predicting the effects of point mutations in protein stability there are many tools for predicting the effects of point mutations in protein stability. in this work we have chosen the tools based on the following characteristics: they produce a numerical prediction of \(\vardelta \vardelta g\); their response time should be short; they consider protein structure as input; they permit automatic submission and license of use for academic purposes is available. in doing so, the selected tools are: cupsat [17], sdm [25], mcsm [18], duet [19], maestro [14] and popmusic [7]. besides, these tools have similar input forms and therefore similar outputs, which allow us to better manipulate the data. 3.1 cupsat cupsat (cologne university protein stability analysis tool) [17] is a web tool that analyzes and predicts protein stability changes for point mutations. it uses the specific structural environment of potential atoms and potential torsion angles, in the unwinding, to predict the \(\vardelta \vardelta \)g. their results consist of informing the structural characteristics about the mutation of this location, such as: solvent accessibility, secondary structure and torsion angles. it also looks at the ability of mutated amino acids to adapt to twist angles. it uses several validation tests (split-sample, jack-knife and k-fold) to guarantee the reliability, accuracy and transferability of the prediction methods, which guarantee an accuracy of more than 80% on their tests [17]. 3.2 sdm site directed mutator (sdm) [25] web tool is a potential energy statistical function with the aim of predicting the effect that single nucleotide polymorphisms will have on protein stability. using amino acid substitution frequencies in specific environments within protein families, the tool calculates a stability score that is analogous to the variation in free energy between the native and mutant state of the protein. 3.3 mcsm mutation cutoff scanning matrix (mcsm), is a predictor that uses structural graph-based signatures of proteins to study the impact of mutations [18]. mcsm signatures were derived from the graph-based concept of cutoff scanning matrix (csm), originally proposed to represent network topology by distance patterns in the study of biological systems. in mcsm each mutation is represented by a pharmacophore vector that is used to train and test machine learning prediction methods, both in regression and predictive classification. on their proposed algorithm, first mcsm signature is calculated using a set of mutations, wild-type structure, the atomic categories (or pharmacophore) to be considered, and a cutoff range and step. for each mutation, based on the cutoff, the residue environment is calculated, followed by the determination of a distance matrix of the pairwise distances between all pairs of atoms of the residue environment. then, the distance matrix is scanned using the cutoff range and step generating a cumulative distribution of the distances by atomic category. finally, the pharmacophoric changes between wild-type and mutant residue are appended to the signature. this signatures are used for training the predictive models. 3.4 duet the duet [19] tool is an integrated computational approach available online, which aims to predict the impact of mutations in protein stability. given a single point mutation, duet combine/consensus the predictions of two other tools sdm and mcsm in a non-linear way, similar to the ensemble learning method, to make predictions and optimize results using support vector machines (svms) with radial basis function kernel. as a filtering step, residue relative solvent accessibility (rsa) is used to optimize the standard sdm predictions using a regression model tree (m5p algorithm) before combining it with mcsm. then, the mcsm an optimized sdm predictions, together with secundary structure from sdm and pharmacophore vector from mcsm are used as input for svm generating the combined duet output. all their training was carried out by data sets with little redundancy and validated with random data sets. 3.5 maestro in order to predict changes in stability on point mutations of proteins, maestro [14] is based on the structure of the protein and, although it has a predictive similar to other methods, it differs because of the following points: it implements a multi-agent system for learning; provides in-depth mutation scanning at various points where mutation types can be comprehensively controlled; and provides a specific mode for stabilizing disulfide bond predictions. maestro is based on two different machine learning approaches: artificial neural networks (ann) and svm (gaussian kernel), and also on multiple linear regression (mlr). in order to improve generalization, a set of anns is utilized rather than a single one. 3.6 popmusic popmusic [7] is a web server for predicting thermodynamic changes in point mutations of proteins, which uses a linear combination of statistical potentials whose coefficients depend on the solvent accessibility of the mutated residue. it is a fast tool that allows you to make predictions for all possible mutations of a protein, average size, in less than a minute. it has the ability to detect which and how optimal the amino acids are in each protein, so that mutation experiments can be carried out, also demonstrating structural weaknesses by quantifying how much these sites are optimized for protein function, rather than stability. 4 proposed methodology our proposed methodology is presented in fig. 1. we are applying stacking and bagging/boosting ensemble learning on the results of six protein stability changes upon point mutations predictors: cupsat [17], sdm [25], mcsm [18], duet [19], maestro [14] and popmusic [7]. fig. 1. proposed methodology of ensemble learning applied on protein stability changes upon point mutations predictors. full size image 4.1 input data the first step on our proposed methodology is to prepare the input dataset. in this paper we are considering as input data sets of proteins from which we can obtain the experimental value of gibbs free energy (\(\vardelta \vardelta \)g) between wild-type and mutants (our target attribute on ensemble learning). this thermodynamic experimental data was obtained from protherm [2]. protherm is a database containing several thermodynamic parameters along with sequence and structural information, experimental methods and conditions parameters for wild-type and mutant proteins. the non-disjoint datasets ds1, ds2648 and ds1948, table 1 lines 1–3, are taken from protherm [2] and considered in the individual tools for training their models. ds1 and ds2648 were used by popmusic [7], sdm [25] and maestro [14], while ds1948 was used by i-mutant [4] tool. these three datasets ds1, ds2648 and ds1948 have different formats for presenting their data, in addition to containing several duplicates. thus, we unified and preprocess all these data totaling 7244 instances (4th line on table 1). after preprocessing all these data, we remove duplicates of the same protein experimental results resulting in our final input dataset ds3904 (highlighted on table 1). table 1. input datasets: number of instances, number of unique proteins, number of stabilizing mutations and number of destabilizing mutations.full size table 4.2 meta data after having ds3904 properly prepared, the following step is submitting this list of protein wild-type/mutations in each of the tools. in fig. 2(a) we describe a part of ds3904 where we have the pdb id, the mutation in the format original residue type position mutation and the experimental \(\vardelta \vardelta \)g. for example, the first line, pdb id 1a23 has as original residue in position 32 a histidine (h) that when mutated to a leucine (l) has an experimental \(\vardelta \vardelta \)g of 4.6. in order to obtain the predicted \(\vardelta \vardelta \)g of each tool to generate our meta data we would need to manually submit each entry of ds3904 to the tools. to perform this task in batch we use a browser plugin called imacros. imacros is a scripting language that allows users to simulate online activities for filling out forms, uploading and downloading images and files, or even importing and exporting information from databases, csv files, xml files, among others. popmusic is different of the other tools since it allows the user to inform the pdb code for each of the 151 proteins of ds3904 (and their specific mutation positions) in a list. the results were properly stored in a local database generating our meta data (fig. 2(b)) where we have as predictive attributes the predicted \(\vardelta \vardelta \)g by the tools and as target attribut the experimental \(\vardelta \vardelta \)g discretized as destabilizing (\(\vardelta \vardelta g < 0\)) or stabilizing (\(\vardelta \vardelta g \ge 0\)). fig. 2. part of input data. (a) ds3904 (b) meta data. full size image 4.3 ensemble learning stacking considering as input the meta data generated by heterogeneous predictive tools in the previous step we performed stacking ensemble learning considering as meta classifiers: random forest (rf), decision trees (dt), support vector machines (svm), multilayer perceptron radial basis function (mlp) and naive bayes (nb). the parameters of these algorithms were set as default values: dt (maxdepth = unlimited; numfeatures = unlimited; numtrees = 100), svm (degree of the kernel = 3; kernel type = radial; eps = 0.001), mlp (autobuild = true; decay = false; hiddenlayers = ‘a’; learningrate = 0.3; momentum = 0.2; validationthreshold = 20) and nb (usekernelestimator = false and usesuperviseddiscretization = false). we evaluate all the generated ensemble models using accuracy, precision, recall and f-measure. 4.4 ensemble learning bagging/boosting in order to evaluate the impact of ensemble learning not only with stacking configuration we decided to also apply bagging and boosting. we have performed different configurations for this ensemble experiments: bagging and boosting. the parameters of these algorithms were set as default values: bagging (bagsizepercent = 100; calcoutofbag = false; classifiers = svm, dt, mlp and nv same parameters used in stacking; numiterations = 10), boosting (numiterations = 10; classifiers = svm, dt, mlp and nv same parameters used in stacking; useresampling = false; weightthreshold = 100). the stacking, bagging and boosting ensemble learning algorithms were performed using weka 3.8 [24]. 5 results and discussion to evaluate our proposed methodology of ensemble learning for this problem we performed two experiments: experiment 1 considers a balanced training dataset from where 500 random instances where selected from ds3904 for each target value stabilizing and destabilizing; in experiment 2 we divided ds3904 in \(70\%\) (training) and \(30\%\) (testing), and we have not balanced training/testing datasets. table 2 describes the datasets for experiments 1 and 2. table 2. training and testing datasets for experiments 1 and 2.full size table 5.1 experiment 1: balanced training dataset ds3904 is unbalanced according to the target attribute. although it is common in real datasets can cause some problems in generating predictive models as being highly specialized (specially to detect the rare class) or susceptible to presence of noise [22]. thus, we decided to train the stacking and bagging/boosting ensemble models in experiment 1 with 500 random instances of each class, totaling 1000 instances for training. the remaining 2904 instances are part of the test set. in order to compare the results of the generated ensemble models with the individual tools we calculated the metrics accuracy, precision, recall and f-measure for the test set for all. first, we evaluate the individual tools considering the test set with 2904 instances (see table 2)and comparing the predicted \(\vardelta \vardelta \)g with the experimental value from protherm (table 3). we can observe that all tools obtained good accuracy values but with low values for precision. it is important to mention that our test set probably was used by the tools to train their models. table 3. evaluation of individual tools in experiment 1.full size table both stacking (table 4), and bagging/boosting ensemble models (table 5), obtained good values of accuracy and precision. from these results, we can highlight svm , multilayer perceptron and random forest close to mcsm tool. from the experiment 1 results we can notice that better precision measures were obtained if we compare with individual tools. table 4. evaluation of stacking ensemble in experiment 1.full size table table 5. evaluation of bagging/boosting in experiment 1.full size table the predictions about point mutation effects on protein structures are going to be used in the decision of generating experimentally some proposed protein mutations, which is expensive in terms of time and costs. considering stabilizing as a positive class, false positives are mutations predicted as stabilizing but they are destabilizing. this error can make the laboratory spends money and time on a not viable protein. on the other side, false negatives are mutations predicted as destabilizing but they are stabilizing. these errors can discard some important mutation that impacts on the protein function of interest, for example. both errors can impact experimental laboratory activities according to the project. so, we expect to achieve good values for precision and recall since it is important to guarantee that the stabilizing or destabilizing predictions are correct. 5.2 experiment 2: unbalanced training set in experiment 2 we are training the ensemble models with unbalanced dataset randomly dividing the ds3904 into 70% of for training and the remaining 30% of the instances for testing. table 6 presents the results of the individual tools. it is observed that although they present good accuracy, the values for precision and recall varies. some tools (mcsm and popmusic) have better recall values, while sdm and maestro have better precision (around 0.6). table 6. evaluation of individual tools in experiment 2.full size table tables 7 and 8 present the ensemble learning models evaluation for experiment 2. we can observe better accuracy, recall and precision values. boosting (svm) model has the best accuracy (0.84) and good recall (0.77). we can see similar results for stacking learning (mlp, svm, dt and rf), bagging (svm, dt and mlp) and also boosting (mlp). table 7. evaluation of stacking ensemble in experiment 2.full size table table 8. evaluation of bagging/boosting in experiment 2.full size table these are good models for avoiding to incorrectly predict as destabilizing a stabilizing mutation (falses negatives). on the other side, stacking learning, bagging and boosting with naive bayes (nb) also have good accuracy but presented better precision values. it means that ensemble models with nb can be good options for avoiding to predict destabilizing mutations as stabilizing (falses positives). 6 conclusion the knowledge about the impact on proteins stability upon point mutations is an important scientific and commercial research topic. one way is to computationally predict the variation of the free energy \(\vardelta \vardelta g\) and its consequently impact on protein stability. there are different methods and tools for this purpose that do not always obtain similar results for the same entries. besides, these predictions are used to define point mutations on protein structures that are experimentally generated, which is expensive in terms of time and costs. errors on this predictions can have important impacts. for example, consider a point mutation as destabilizing but when it is stabilizing can discard this candidate. at the same time that a wrong prediction of stabilizing when the mutation destabilizes the protein structure can made the laboratory spends money and time on a not viable protein. thus, in this paper we propose to aggregate the results from different stability prediction tools using stacking, bagging/boosting ensemble learning techniques achieving to obtain models with not only good accuracy but also good precision and/or recall. in our proposed methodology we are considering six individual \(\vardelta \vardelta g\) predictors: cupsat [17], sdm [25], mcsm [18], duet [19], maestro [14] and popmusic [7]. we prepared an input dataset from protherm [2] with 3,904 instances about wild-type and mutation proteins. after submitting this list to the tools we obtained our meta-data composed by their \(\vardelta \vardelta g\) predictions. having this meta-data, we applied stacking ensemble with meta-classifiers (rf, dt, svm, mlp and nb), bagging (svm, dt, mlp and nv) and boosting . for both ensemble techniques we validated considering two different experiments according to the training set. in experiment 1 (balanced training set) results we can observe that even all individual tools obtained good accuracy values, they presented low precision. on the other side, both stacking and bagging/boosting ensemble models obtained also good accuracy but with better precision. in experiment 2 (unbalanced training set) results we can observe better accuracy, recall and precision values for ensemble models. boosting(svm) obtained the best accuracy (0.84) and good recall (0.77). we can see similar results for stacking learning (mlp, svm, dt and rf), bagging (svm, dt and mlp) and also boosting (mlp). these are good models for avoiding to incorrectly predict as destabilizing a stabilizing mutation. on the other side, stacking learning, bagging and boosting with naive bayes (nb) also have good accuracy but presented better precision values. it means that ensemble models with nb can be good options for avoiding to predict destabilizing mutations as stabilizing. thus, according with the results, our proposed ensemble approach is appropriate to predict the effect of point mutations on protein stability showing more reliable results than the individual tools. as future work, we can combine the individual tools using genetic algorithms to determine the weight of each \(\vardelta \vardelta \)g predicted and also include more tools in the ensemble. in addition, other balancing strategies can be performed on the training datasets. references auclair, j., et al.: systematic mrna analysis for the effect of mlh1 and msh2 missense and silent mutations on aberrant splicing. hum. mutat. 27(2), 145–154 (2006). https://doi.org/10.1002/humu.20280 article  google scholar  bava, k.a., gromiha, m.m., uedaira, h., kitajima, k., sarai, a.: protherm, version 4.0: thermodynamic database for proteins and mutants. nucleic acids res. 32(1), 120–121 (01 2004). https://doi.org/10.1093/nar/gkh082 breiman, l.: bagging predictors. mach. learn. 24(2), 123–140 (1996) math  google scholar  capriotti, e., fariselli, p., casadio, r.: a neural-network-based method for predicting protein stability changes upon single point mutations. bioinformatics 20(suppl 1), i63–i68 (2004). https://doi.org/10.1093/bioinformatics/bth928 article  google scholar  capriotti, e., fariselli, p., rossi, i., casadio, r.: a three-state prediction of single point mutations on protein stability changes. bmc bioinform. 9(suppl 2) (2008). https://doi.org/10.1186/1471-2105-9-s2-s6 cheng, j., randall, a., baldi, p.: prediction of protein stability changes for single-site mutations using support vector machines. prot. struct. function bioinform. 62(4), 1125–1132 (2005). https://doi.org/10.1002/prot.20810 dehouck, y., kwasigroch, j.m., gilis, d., rooman, m.: popmusic 2.1: a web server for the estimation of protein stability changes upon mutation and sequence optimality. bmc bioinform. 12(1) (2011). https://doi.org/10.1186/1471-2105-12-151 dong, x., yu, z., cao, w., shi, y., ma, q.: a survey on ensemble learning. front. comput. sci. 14(2), 241–258 (2020) article  google scholar  eickholt, j., cheng, j.: dndisorder: predicting protein disorder using boosting and deep networks. bmc bioinform. 14(1), 1 (2013) article  google scholar  fersht, a.r.: protein folding and stability: the pathway of folding of barnase. febs lett. 325(1–2), 5–16 (1993) google scholar  freund, y., schapire, r.e., et al.: experiments with a new boosting algorithm. in: icml, vol. 96, pp. 148–156 (1996) google scholar  guerois, r., nielsen, j.e., serrano, l.: predicting changes in the stability of proteins and protein complexes: a study of more than 1000 mutations. j. mol. biol. 320(2), 369–387 (2002). https://doi.org/10.1016/s0022-2836(02)00442-4 article  google scholar  han, j., pei, j., kamber, m.: data mining concepts and techniques. morgan and kaufmann, san francisco (2006) math  google scholar  laimer, j., hofer, h., fritz, m., wegenkittl, s., lackner, p.: maestro multi agent stability prediction upon point mutations. bmc bioinform. 16(1) (2015). https://doi.org/10.1186/s12859-015-0548-6 mendoza, m.r., bazzan, a.l.c.: the wisdom of crowds in bioinformatics: what can we learn (and gain) from ensemble predictions? in: proceedings of the twenty-seventh aaai conference on artificial intelligence, pp. 1678–1679 (2013) google scholar  noçairi, h., gomes, c., thomas, m., saporta, g.: improving stacking methodology for combining classifiers; applications to cosmetic industry. electron. j. appl. stat. anal. 9(2), 340–361 (2016) mathscinet  google scholar  parthiban, v., gromiha, m.m., schomburg, d.: cupsat: prediction of protein stability upon point mutations. nucl. acids res. 34(web server), january 2006. https://doi.org/10.1093/nar/gkl190 pires, d.e.v., ascher, d.b., blundell, t.l.: mcsm: predicting the effects of mutations in proteins using graph-based signatures. bioinformatics 30(3), 335–342 (2013). https://doi.org/10.1093/bioinformatics/btt691 article  google scholar  pires, d.e., ascher, d.b., blundell, t.l.: duet: a server for predicting effects of mutations on protein stability using an integrated computational approach. nucl. acids res. 42(w1), w314–w319 (2014) article  google scholar  saha, i., zubek, j., klingström, t., forsberg, s., wikander, j., kierczak, m., maulik, u., plewczynski, d.: ensemble learning prediction of protein-protein interactions using proteins functional annotations. mol. biosyst. 10(4), 820–830 (2014) article  google scholar  sugita, y., kitao, a.: dependence of protein stability on the structure of the denatured state: free energy calculations of i56v mutation in human lysozyme. biophys. j. 75(5), 2178–2187 (1998) article  google scholar  tan, p.n., steinbach, m., kumar, v.: introduction to data mining. pearson education, london (2016) google scholar  verli, h.: bioinformática: da biologia à flexibilidade molecular. sociedade brasileira de bioquímica e biologia molecular (2014) google scholar  witten, i.h., frank, e., hall, m.a.: data mining practical machine learning tools and techniques third edition. morgan kaufmann, burlington (2016) google scholar  worth, c.l., preissner, r., blundell, t.l.: sdm-a server for predicting effects of mutations on protein stability and malfunction. nucl. acids res. 39(suppl_2), w215–w222 (2011) google scholar  yang, p., yang, y.h., zhou, b.b., zomaya, a.y.: a review of ensemble methods in bioinformatics. current bioinform. 5(4), 296–308 (2010) google scholar  yang, y.: temporal data mining via unsupervised ensemble learning. elsevier, amsterdam (2016) google scholar  yin, s., ding, f., dokholyan, n.v.: eris: an automated estimator of protein stability. nat. meth. 4(6), 466–467 (2007). https://doi.org/10.1038/nmeth0607-466 article  google scholar  zhang, z., miteva, m.a., wang, l., alexov, e.: analyzing effects of naturally occurring missense mutations. comput. math. meth. med. 2012, 1–15 (2012). https://doi.org/10.1155/2012/805827 article  math  google scholar  zhou, h., zhou, y.: distance-scaled, finite ideal-gas reference state improves structure-derived potentials of mean force for structure selection and stability prediction. prot. sci. 11(11), 2714–2726 (2009). https://doi.org/10.1110/ps.0217002 article  google scholar  zhou, z.h.: ensemble methods: foundations and algorithms. chapman and hall/crc, boca raton (2019) google scholar  download references author information authors and affiliations computational biology laboratory combi-lab, centro de ciências computacionais, universidade federal do rio grande furg, av. itália, km 8, rio grande, rs, brazil eduardo kenji hasegawa de freitas, alex dias camargo, maurício balboni, adriano v. werhli & karina dos santos machado authors eduardo kenji hasegawa de freitasview author publications search author on:pubmed google scholar alex dias camargoview author publications search author on:pubmed google scholar maurício balboniview author publications search author on:pubmed google scholar adriano v. werhliview author publications search author on:pubmed google scholar karina dos santos machadoview author publications search author on:pubmed google scholar corresponding author correspondence to karina dos santos machado . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de freitas, e.k.h., camargo, a.d., balboni, m., werhli, a.v., dos santos machado, k. (2021). ensemble of protein stability upon point mutation predictors. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_6 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords ensemble learning bioinformatics protein stability point mutations publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature tactical asset allocation through random walk on stock network | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper tactical asset allocation through random walk on stock network conference paper first online: 28 november 2021 pp 528–542 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) tactical asset allocation through random walk on stock network download book pdf download book epub washington burkart freitas10 & joão roberto bertini junior10  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 796 accesses 4 citations abstract tactical asset allocation is an essential method for defining a profitable portfolio for a given period. an analyst usually creates a tactical asset portfolio through technical analysis, a process subjective to the analyst’s knowledge and interpretations. another aspect that can directly influence the quality of a portfolio is the number of assets considered for analysis. human analysts tend to focus on a pre-defined group of assets, limiting choices, and, consequently, the possibility of better results. this work proposes the stock network portfolio allocation (snpa) algorithm for the automatic recommendation of a stock portfolio, aiming to maximize profit and minimize risk. the proposed method considers a possibly large set of assets represented as a complex network. in which the nodes represent assets, and the edges stand for the correlation between their returns. portfolio allocation is done through a random walk on the stock network, selecting, in the end, the most visited nodes (stocks). we conducted investment simulations on brazilian stocks from the ibrx100 index, for 24 month periods, from jan. 2018 to dec. 2019. we compare the results with portfolio strategies: ibovespa index (ibov), classic markowitz’s mean-variance portfolio (mv), mean absolute deviation (mad) portfolio, conditional value at risk (cvar), and hierarchical risk parity (hrp). the shape ratio (sr), maximum drawdown (mdd), and cumulated wealth (cw) were used as performance metrics. the snpa algorithm demonstrated its effectiveness, presenting a cw of 236.3%, being 203.5% above mv portfolio; 181.7% above cvar portfolio; 175.6% above mad portfolio, 184.6% above ibov index, and 165.1% above hrp. snpa also surpassed the benchmarks considering the performance metrics sr and mdd, with values 0.67 and −1.37 respectively, the best results among the benchmarks were observed by the hrp strategy, with 0.48 in sr index, and mv with −1.39 in mdd index. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others enhanced securities investment strategy using issa–svm: a hybrid model combining adaptive moving average, support vector machine, and multi-strategy sparrow search algorithm for improved trend tracking and risk adjustment article open access 24 may 2025 search for the optimal investor's portfolio using spark apache chapter © 2022 portfolio rebalancing model utilizing support vector machine for optimal asset allocation article 16 march 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. mathematical finance quantitative finance statistical finance stochastic analysis stochastic networks drug portfolio management 1 introduction portfolio management is one of the major problems in today’s competitive financial environment [4, 16]. due to stock market volatility as well as diverse factors related to human, political, and economic behavior, creating and managing portfolios successfully becomes an increasingly challenging practice. since the precursor works of markowitz [13, 14] referring to portfolio selection and modern portfolio theory, many researchers have been striving to find an ideal solution to create an efficient asset portfolio that maximizes return and minimizes risks. to cope with market instability, researchers use different methods such as modeling the correlation between asset pricing, listening to human interactions in specialized forums, clustering assets according to their performance and, analyzing machine learning predictions [2, 11]. among the portfolio allocation methods, tactical asset allocation (taa) is an asset management investment method that consists of readjusting the proportions of each category of assets, based on a signal that indicates which asset class will perform best in the upcoming period [1]. particularly, systematic taa strategies involve a quantitative, data-driven investment model to identify trends and perform forecasts to aid portfolio management. one aspect that can directly influence the quality of a portfolio is the number of assets considered for analysis. human analysts tend to focus on a pre-defined group of assets, limiting choices, and, consequently, the possibility of better results. while most automated portfolio management systems may exhibit prohibitive computational costs associated with optimization procedures when dealing with a large set of assets [18]. in this paper, we propose the stock network portfolio allocation (snpa) algorithm for the automatic recommendation of a stock portfolio, aiming to maximize profit and minimize risk. the motivation for this study is to efficiently cover a large number of assets by modeling them as a complex network. in this network, each node represents an asset, and edges are established according to the correlation between their returns. portfolio selection is then made through a random walk on the network, selecting, in the end, the most visited assets. investment simulations have shown a return higher than the benchmarks: the ibovespa, classic markowitz’s mean-variance portfolio (mv), mean absolute deviation (mad) portfolio, conditional value at risk (cvar) portfolio, and hierarchical risk parity (hrp). the remainder of this paper is organized as follows. in sect. 2, we present a literature review on some topics related to this work as methods for tactical asset allocation, stock correlation analyzes, and stock networks. in sect. 3, we detail the snpa algorithm. in sect. 4, we conduct some analysis with the hyperparameters of the model. section 5 we describe the portfolio strategies found in the literature and used in the present work to compare with the proposed algorithm. the performance metrics used in the comparisons are also presented. we present and discuss the result of experiments in sect. 6. finally, sect. 7 concludes the paper. 2 related works network analysis is commonly used to describe the characteristics or behavior of complex networks. some research has been conducted to model the stock market using networks [15], where the stock market is presented through a minimal spanning tree (mst) obtained from the matrix of correlation coefficients calculated between all pairs of stocks in a portfolio, considering stock returns. mst-type networks have also been implemented to demonstrate the impact of the new coronavirus pandemic on the financial market [26]. george and changat [7] emphasize that stock network analysis can play an important role in the stock market study. they created a network of stocks with a set of data from the period of one year, their objective was to identify the structural properties of the market. his work was able to reveal important inputs for decision making, such as, for example, creating a portfolio of more relevant sectors. ben-jacob et al. [20] analyzed stock index correlations, nested stock correlations, and correlations after subtraction of the index return from the stocks’ returns, they showed that the behavior of the stock market could not be understood and predicted only based on individual stocks; rather, system-level analysis methods need to be devised to analyze the stock market as a whole. kim and jeong [8] proposed improved methods to identify stock groups using the correlation matrix of stock price changes, they identified the multiple groups of stocks from the empirical correlation matrix of stock price changes in the new york stock exchange. chen et al. [5] discuss the stock network construction problem under simultaneous consideration of linear and nonlinear relations between stocks. the results showed that the proposed multi-layer network better balances the relation between prediction accuracy and the number of predictable nodes. hong et al. [10] consider cross-correlations among stock prices in the korean stock market. they used the daily korean stock market prices of kospi200 for four years from jan. 2000 to dec. 2004. they observed the behavior of the stock network specifying a threshold for the correlation between the nodes of that network, so a scale-free network was obtained by the cross-correlation coefficient. 3 the stock network portfolio allocation algorithm prior to detail the stock network portfolio allocation (snpa) algorithm, we shall present some concepts about asset return, risk, portfolio return, and weighted stock network. measuring the asset return is the way to determine its performance over time. let \(r_{t,i}\) be the observed return of the asset i at time \(t \in \{1, 2, \dots , \tau \}\) considering historical data over the time horizon \(\tau \) . here, t can represent a day, a week, a month, a quarter or a year, according to the study interval. from time \( t-1 \) to time t, the returns from a set of n stocks accessible to investors is denoted as \( \mathbf{r} _t = [r_ {t, 1}, \dots , r_ { t, i}, \dots , r_ {t, n}] \). the return \( r_{t,i} \) for the i-th asset is calculated by eq. (1). $$\begin{aligned} r_ {t, i} = \frac{pr_ {t, i} pr_ {t-1, i}}{pr_ {t-1, i}} \end{aligned}$$ (1) where \( pr_{t,i} \) and \( pr_{t-1,i} \) represent the closing prices of the ith asset at times t and \( t-1 \), respectively. the cumulative wealth of a given asset i during the analyzed period t is denoted as \(cw_{t,i}\), and can be calculated according eq. (2). $$\begin{aligned} cw_{t,i} = \prod ^\tau _{t=1} { (1 + r_{t,i}) 1} \end{aligned}$$ (2) the standard deviation, \(\sigma _i\), and the variance, \(\sigma _i^2\), of the return for asset i, can be used to measure the risk of a given asset. let \(\mu _i = \frac{1}{\tau }\sum _{i=1}^{\tau }r_{t,i}\) be the average of returns for asset i over time horizon \(\tau \), the variance of the returns for the asset i is given by eq. (3) $$\begin{aligned} \sigma _i^2 = \frac{1}{\tau }\sum _{i=1}^{\tau }(r_{t,i} \mu _i)^2 \end{aligned}$$ (3) we denote \( \boldsymbol{\omega }_t = [\omega _{t, 1}, \dots , \omega _{t, i}, ..., \omega _{t, n}] \) as the weights from set of n stocks of the portfolio at time t, such that \( \sum \boldsymbol{\omega }_t = 1 \); \( \omega _{t, i} \) represents the allocation percentage of the i-th asset in the portfolio. having obtained the set of returns and the set of weights of the assets in the portfolio, we can calculate the return of the portfolio in time t, noted as \( r_t \) through eq. (4). $$\begin{aligned} r_t=\sum \mathbf{r} _t \boldsymbol{\omega }_t \end{aligned}$$ (4) a weighted stock network is denoted as \(g=(v, e, z, w)\), where v is the set with n nodes, e is the set of m edges, z is the set of z nodes weights and w is the set of w edges weights. the network g is obtained through a correlation matrix \(\mathbf{m} _t\), as shown in eq. (5), and obtained by eq. (6). $$\begin{aligned} \mathbf {m}_t=\left[ \begin{array}{lllll} c_{11} &{} c_{12} &{} c_{13} &{} \cdots &{} c_{1n} \\ c_{21} &{} c_{22} &{} c_{23} &{} \cdots &{} c_{2n} \\ c_{31} &{} c_{32} &{} c_{33} &{} \cdots &{} c_{3n} \\ \vdots &{} \vdots &{} \vdots &{} \ddots &{} \vdots \\ c_{n1} &{} c_{n2} &{} c_{n3} &{} \cdots &{} c_{nn} \end{array} \right] \end{aligned}$$ (5) $$\begin{aligned} c_{ij} = \frac{{} \sum _{t=1}^{\tau } (r_{t,i} \mu _i)(r_{t,j} \mu _j)}{\sqrt{ \sum _{t=1}^{\tau } (r_{t,i} \mu _i)^2 \sum _{t=1}^{\tau } (r_{t,j} \mu _j)^2}} \end{aligned}$$ (6) to suggest a portfolio for the following month, the algorithm represents the set of assets as an undirected complex network. the most visited assets compose the portfolio after a random walk takes place. therefore, the proposed approach can be addressed into two parts, building the network from a set of assets and traversing the network to find out the portfolio. given a set of assets \(a = \{1,\dots , n\}\), the network is built as follows: 1. each asset \(i \in a\) is abstracted to a node \(v_i\). 2. given the correlation between assets i and j, \(c_{i,j}\) (eq. (6)), each pair of nodes \(v_i\) and \(v_j\) is connected if \(c_{ij} < \lambda _n\) or \(c_{ij} > \lambda _p\). in item 2, \(\lambda _n\) and \(\lambda _p\) are thresholds on the value of the correlation of the returns. the rationale of those parameters is to avoid values of correlations close to zero, keeping only significant values of correlation between returns. notice that their values directly interfere with the structure of the network, consequently, they need to be chosen considering the algorithm performance and restraining to values that generate connected networks. snpa uses the asset cumulative wealth, eq. (5), of the analyzed period as a node weight. so node \(v_i\) has as an associated weight, \(z_i\), its cumulative wealth \(cw_{t,i}\). in this work, the historical series of the last five years prior to the date on which the portfolio is to be generated is used. let \(\vargamma _i\) be the set of neighbors of vertex \(v_i\). the way the network is built, for each node \(v_{i}\), \(\vargamma _i\) may have nodes whose correlation to \(v_i\) is positive or negative. supposing \(v_j \in \vargamma _i\), the probability of the particle to visit \(v_j\), noted by \(p_{ij}\), can be calculated by the softmax function as in eq. (7). the reason to consider the softmax function is that \(cw_{t,j}\) may be negative. $$\begin{aligned} p_{ij} = \frac{e^{cw_{t,i}}}{{\sum _{v_j \in \vargamma _i} e^{cw_{t,j}}}} \end{aligned}$$ (7) figure 1(a) shows the paths that can be taken by the particle starting from node \(v_1\) and the cumulative wealth \(cw_{i,j}\) for each asset represented in node \(v_j\). figure 1(b) shows the transition probabilities from node \(v_1\) to each of its neighbors \(v_2\), \(v_3\) and \(v_4\) as given by eq. (7). fig. 1. example of selecting a neighbor using cumulative wealth probability full size image algorithm 1 presents the details of the proposed method. the algorithm has as inputs: the set of stocks a, the number of iterations k, the thresholds on the correlation values \(\lambda _p\), and \(\lambda _n\). the number of stocks in the portfolio, q, can be either given as an input and in this way, the algorithm will search for a portfolio with q stocks. or, if it is not given, the algorithm outputs the portfolio with the highest return considering any possible value of q. the most costly part of the algorithm is to calculate the transition probabilities, which requires to access every edge of the graph. once all the probabilities have been defined, a random walk takes place. the walk proceeds in the network by following the nodes according to the transition probabilities, while the algorithm keeps track of the number of visits each node had. in the end, the stocks represented by the most visited nodes are chosen to compose the portfolio, and the weight of each asset corresponds to the proportion of the total number of visits. the selected stocks tend to have a high return and be highly correlated to each other (positively and negatively). at the same time, the randomness of the algorithm allows us to consider different but also profitable portfolios. 4 hyperparameter analysis to define the hyperparameters values of the algorithm, experiments were carried out with available stocks from ibrx100 index (approximately 100 stocks). ten experiments were performed to mitigate the effect of randomization. the experiments consider build a portfolio for jan. 2020 using stock data of the period from jan. 2019 to dec. 2019. the datasets used in this work were built with information from the yahoo finance [https://finance.yahoo.com/lookup]. first, consider analysing how each parameter relates to each other and to portfolio return. for this analysis, experiments to find a portfolio have been conducted considering several parameters settings. the number of iterations k varied in the range \(\{50, 100, \dots , 1000\}\), threshold values \(\lambda _p\) and \(\lambda _n\) were evaluated in \(\{0.1, 0.2, \dots , 1\}\). the number of assets in the portfolio, q, was given by the algorithm along with the best portfolio return. to each parameter setting, risk has been calculated as the standard deviation of returns among ten runs. node average degree in the stock network have also been considered for analysis. figure 2 shows some pairwise correlations, considering hyperparameters, return, risk and node average degree. it is possible to observe a strong positive correlation between risk and return (0.63), i.e., the higher the return, the higher the risk. also, a strong negative correlation between \(\lambda _p\) and node average degree in the network, because as the value of \(\lambda _p\) as increases, less nodes are connected. it is also interesting to note the positive correlation between \(\lambda _n\) and risk and return, the greater the value of \(\lambda _n\) the greater the value of the return, although the risk also increases, the increase in risk is less than the increase in return. fig. 2. hyperparameters correlations full size image table 1 displays a summary of the main statistical measures obtained through the experiments. the most relevant are the average return and standard deviation, 0.164 ± 0.106. another relevant information is about the hyperparameter q, although the range was between 2 and 29, most portfolios were composed of 2 assets, as can be seen in the measurement of the median. table 1. statistical summaryfull size table with the data from the experiments it is possible to identify the best values for hyperparameters. figure 3 presents the mean return against the standard deviation of the returns for each value of k and varying \(\lambda _p\) and \(\lambda _n\) in the set \(\{0.1, 0.2, \dots , 1\}\). each result is the average of 10 realizations. for k = 950 we have a value with less risk and for k = 350 portfolios with higher mean return. it is a trade-off matter between risk and return. whereas we aim to maximize the return and the difference in risk is small between the two values, the value of 350 was chosen for the hyperparameter k. having defined the value of k, we can obtain the values of \(\lambda _p\) and \(\lambda _n\). figure 4 shows the portfolio average returns for \(k = 350\), considering different combination of \(\lambda _p\) and \(\lambda _n\). in the figure, only those return values that are within the range \(0.164\,\pm \,0.106\) are shown, in order to stay close to average returns and avoid outliers. fig. 3. estimated mean return and standard deviation for different values of k. full size image fig. 4. estimated portfolio return for when varying \(\lambda _p\) and \(\lambda _n\) and \(k = 350\). full size image as can be seen in fig. 4, the best result obtained with the experiment is the mean return for \( \lambda _p\) = 0.9 and \( \lambda _n\) = −0.3. these are the values of the hyperparameters that will be used in sect. 6 about experimental result. hyperparameters \(\lambda _p \) and \(\lambda _n\) are very relevant in the decision process of the algorithm, by defining these hyperparameters, the number of paths in the network and the interconnection between nodes are defined. setting hyperparameters at random can result in an ineffective portfolio at the end process. 5 methodology in this section, we describe the main characteristics of the benchmarks used in the present work, which will be compared with the proposed algorithm: markowitz’s classic mean-variance (mv) [14] portfolio; mad portfolio (mean absolute deviation) [9]; cvar portfolio (conditional value-at-risk) [19] and hierarchical risk parity (hrp) portfolio [17]. the performance metrics methodologies that are used to compare these portfolio strategies with the proposed algorithm are also presented. 5.1 portfolio allocation methods markowitz’s classic mean-variance (mv): markowitz has developed a mathematical model showing a reduction in the volatility of a portfolio through the combination of investments [13, 14]. he assumed that portfolios could be entirely characterized by their mean return and variance (or risk). mean absolute deviation (mad): in 1991, konno & yamazaki [9] suggested a linear programming model in which portfolio risk is measured with the mean absolute deviation (mad) instead of markowitz’s variance-based portfolio. the purpose of the proposal was to remove most of the difficulties associated with the classical markowitz’s model. conditional value-at-risk (cvar): in 2000, rockafellar & uryasev [19] introduced a new approach to optimizing a portfolio to reduce the risk of high losses. value-at-risk (var) played a relevant role in their approach, but the emphasis was on conditional value-at-risk (cvar) [19, 25]. hierarchical risk parity (hrp): portfolio optimization method suggested by lopez de prado [17] in 2016. hrp applies modern concepts on graph theory and machine learning techniques to build a diversified portfolio based on the information contained in the asset return matrix. 5.2 portfolio performance metrics to measure the performance of stock portfolios, some methods can be used, such as: the sharpe ratio, the maximum drawdown (difference between the highest and lowest value in a given period) and the cumulative wealth. these indicators are widely used in works involving asset portfolio performance analysis and can be found in [3, 6, 22, 24]. sharpe ratio (sr): was created by the nobel william forsyth sharpe [21], it is the most widely adopted index for measuring the risk and return ratio of a portfolio. the sr provides a measure of risk-adjusted return for a portfolio strategy [23]. the higher the sr index the better. the sr can be obtained using eq. (8), where \( \overline{r}_\tau \) is the average of \( r_t \) returns from portfolios for the time horizon \( \tau \) (eq. (9)), and \( \sigma _{\tau } \) is the standard deviation of portfolio returns (eq. (10)). $$\begin{aligned} sr=\frac{\overline{r}_\tau }{\sigma _{\tau } } \end{aligned}$$ (8) $$\begin{aligned} \overline{r}_\tau = \frac{1}{\tau } \sum _{t = 1} ^ {\tau } r_{t} \end{aligned}$$ (9) $$\begin{aligned} \sigma _{\tau } = \sqrt{\sigma _{\tau } ^ 2}, \sigma _ {\tau } ^ 2 = \frac{1}{\tau } \sum _{t = 1} ^ {\tau } (r_t \overline{r}_\tau ) ^2 \end{aligned}$$ (10) maximum drawdown (mdd) the maximum drawdown is a risk measure widely used by fund managers, as a very high drawdown usually triggers fund redemptions by investors [12]. to obtain the value of the mdd, the eq. (11) can be used, where \( min (r_{t}) \) is the lowest value observed in the portfolio returns and \( max (r_{t}) \) the highest value found. the lower the mdd index the better. $$\begin{aligned} mdd = \frac{{ max(r_{t}) min(r_{t}) }}{ max(r_{t}) } \end{aligned}$$ (11) cumulative wealth (cw): is the cumulative wealth of stock portfolio over the period \( \tau \). measures the aggregate amount that the investment has gained or lost over time. cw can be calculated by eq. (12). the higher the cw index the better. $$\begin{aligned} cw = \prod ^\tau _{t=1} { (1 + r_t) 1} \end{aligned}$$ (12) 6 experimental result the experiments were performed using the hyperparameters values presented in table 2, and found as described in sect. 4. table 2. hyperparameters summary for experimentsfull size table the hyperparameter k is the number of iterations that will be performed on the network. the set of stocks a is formed by the stocks in the ibrx100 index. this index is composed by the 100 most traded stocks on the brazilian stock market. for the purpose of this work, however, the number of stocks used in the experiment was 96 stocks, due to data availability. the experiments consist in finding a portfolio for the next month, \(t\,+\,1\), giving the stock information (time series) from the previous five years. experiments were analyzed with the base period january 2018 to december 2019. when analyzing a period with these reference months, for example, october 2019, the time series analyzed was from september 2014 to september 2019, and the month that wanted the best result is october 2019. the results is the month return averaged for 10 times. to evaluate the proposed approach, the snpa algorithm is compared with the following benchmarks: the ibovespa index (ibov), classic markowitz’s mean-variance portfolio (mv), mean absolute deviation (mad) portfolio, conditional value at risk (cvar) portfolio and hierarchical risk parity (hrp). the results are separated into two sections; sect. 6.1 focuses on returns, while sect. 6.2 presents performance analysis. 6.1 return-based results table 3 shows the summary results for each portfolio analyzed. as can be seen, the snpa algorithm obtained the highest mean return when compared to the benchmarks, it also presents a balanced risk value (sd standard deviation). the value \(\bar{q}\) is the average number of assets each portfolio has suggested. table 3. summary resultsfull size table table 4 shows the monthly returns obtained by the algorithm under comparison. as can be seen, snpa presented higher results in 11 out of the 24 analysed months. conversely, snpa have performed worse than all the other in 2 out of the 24 analysed months. it is important to note that in august 2018 the algorithm obtained a positive return, while all the others presented negative values. at last, the snpa algorithm obtained negative results in 5 of the 24 months evaluated, that is, in 21% of the cases. the benchmarks were, on average, negative 9 months, that is, 37.5%. table 4. monthly returns for the algorithms under comparison. best results are bold faced and worse results are highlighted in redfull size table figure 5 shows the cumulative portfolio’s return to the analyzed period. in the cumulative wealth, the algorithm snpa demonstrated its effectiveness, presenting a total return of 236.3%, being 203.5% above mv portfolio; 181.7% above cvar portfolio; 175.6% above mad portfolio, 184.6% above ibov index and 165.1% above hrp. fig. 5. portfolio’s cumulative return full size image table 5. summary of portfolio performancefull size table 6.2 performance-based results here are presented the results based on the performance of each portfolio strategy and the suggested algorithm. the results are compared with the benchmarks considering the performance metrics: shape ratio (sr), maximum drawdown (mdd) and cumulated wealth (cw). table 5 shows the results obtained by the algorithm proposed in the present work comparing with the benchmarks. the numbers highlighted were the best observed values. it is possible to observe in table 5 that the performance of the algorithm proposed is predominant, obtaining the best values considering all the performance metrics evaluated: sr, mdd and cw with the values 0.67, −1.37 and 236.3% respectively. the second best performance was obtained by the hrp strategy, considering the performance metrics sr and cw, with the values 0.48 and 71.2% respectively. in the case of mdd, the second best performance was obtained by the mv strategy, with a value of −1.39. 7 conclusion we propose the stock network portfolio allocation (snpa) algorithm to recommend a profitable stock portfolio. snpa can efficiently handle a large set of assets by representing them as a complex network, where nodes represent assets and edges how their returns correlate. a random walk process carried in this network defines a portfolio. simulated investment results show the proposed algorithm presented superior performance when compared to the benchmarks: ibovespa index (ibov), classic markowitz’s mean-variance portfolio (mv), mean absolute deviation (mad) portfolio, conditional value at risk (cvar) and hierarchical risk parity (hrp). the results were compared with the benchmarks considering the performance metrics: shape ratio (sr), maximum drawdown (mdd), and cumulated wealth (cw). the performance of the algorithm proposed was predominant, obtaining the best values considering all the performance metrics. snpa demonstrated its effectiveness, presenting a cw of 236.3%, being 203.5% above mv portfolio; 181.7% above cvar portfolio; 175.6% above mad portfolio, 184.6% above ibov index and 165.1% above hrp. the proposed algorithm also surpassed the benchmarks considering the performance metrics sr and mdd, with values 0.67 and −1.37 respectively, the best results among the benchmarks observed by the hrp strategy, with 0.48 and −1.73 respectively. the downside of the proposed algorithm is its computational complexity due to the calculation of the return matrix and the length of the random walk. future work includes analyzing larger periods of time considering all stocks from a particular market, for more markets. and, also consider machine learning predictions of the return in place of historical data. references amenc, n., le sourd, v.: portfolio theory and performance analysis. wiley, chichester (2003) google scholar  baser, p., saini, j.r.: agent based stock clustering for efficient portfolio management. int. j. comput. appl. 116, 35–41 (2015) google scholar  brandt, m.: portfolio choice problems. in: handbook of financial econometrics vol. 1 (2010) google scholar  brugière, p.: quantitative portfolio management. stbe, springer, cham (2020). https://doi.org/10.1007/978-3-030-37740-3 book  math  google scholar  chen, w., jiang, m., jiang, c.: constructing a multilayer network for stock market. soft comput. 1–17 (2019). https://doi.org/10.1007/s00500-019-04026-y demiguel, v., garlappi, l., uppal, r.: optimal versus naive diversification: how inefficient is the 1/n portfolio strategy? rev. financ. stud. 22, 1915–1953 (2009) article  google scholar  george, s., changat, m.: network approach for stock market data mining and portfolio analysis. in: 2017 international conference on networks and advances in computational technologies (netact), pp. 251–256 (2017) google scholar  kim, d.h., jeong, h.: systematic analysis of group identification in stock markets. phys. rev. e72, 046133 (2005) google scholar  konno, h., yamazaki, h.: mean-absolute deviation portfolio optimization and its applications to tokyo stock market. manag. sci. 37(5), 519–531 (1991) article  google scholar  lee, k., lee, j., hong, h.: complex networks in a stock market. comput. phys. commun. 177, 186 (2007) article  google scholar  lee, y.j., cho, h.g., woo, g.: analysis on stock market volatility with collective human behaviors in online message board. in: proceedings of the ieee international conference on computer and information technology, pp. 482–489 (2014) google scholar  magdon-ismail, m., atiya, a.f., pratap, a., abu-mostafa, y.s.: on the maximum drawdown of a brownian motion. j. appl. probab. 41, 147–161 (2004) article  mathscinet  google scholar  makowitz, h.m.: portfolio selection: efficient diversification of investments. wiley, new york (1951) google scholar  makowitz, h.m.: porfolio selection. j. finance 7(1), 77–91 (1952) google scholar  mantegna, r.: hierarchical structure in financial markets. eur. phys. j. b condens. matter complex syst. 11, 193–197 (1999). https://doi.org/10.1007/s100510050929 article  google scholar  pareek, m.k., thakkar, p.: surveying stock market portfolio optimization techniques. in: 2015 5th nirma university international conference on engineering (nuicone), pp. 1–5 (2015) google scholar  building diversified portfolios that outperform out of sample: lopez de prado, m. j. portfolio manag. 42, 59–69 (2016) article  google scholar  raudys, s.: portfolio of automated trading systems: complexity and learning set size issues. ieee trans. neural netw. learn. syst. 24, 448–459 (2013) article  google scholar  rockafellar, r.t., uryasev, s.: optimization of conditional value-at-risk. j. risk 2, 21–41 (2000) article  google scholar  shapira, y., kenett, d., ben-jacob, e.: the index cohesive effect on stock market correlations. eur. phys. j. b72, 657 (2009). https://doi.org/10.1140/epjb/e2009-00384-y article  math  google scholar  sharpe, w.: mutual fund performance. j. bus. 39, 657–669 (1965) google scholar  shen, w., wang, b., pu, j., wang, j.: the kelly growth optimal portfolio with ensemble learning. in: proceedings of the aaai conference on artificial intelligence, vol. 33, pp. 1134–1141 (2019) google scholar  shen, w., wang, j.: portfolio blending via thompson sampling. in: proceedings of the twenty-fifth international joint conference on artificial intelligence (ijcai-16) (2016) google scholar  shen, w., wang, j., jiang, y.g., zha, h.: portfolio choices with orthogonal bandit learning. in: proceedings of the twenty-fourth international joint conference on artificial intelligence (ijcai-15) (2015) google scholar  uryasev, s., krokhmal, p., palmquist, j.: portfolio optimization with conditional value-at-risk objective and constraints. j. risk 7, 43–68 (2002) google scholar  zhang, d., hu, m., ji, q.: financial markets under the global pandemic of covid-19. finance res. lett. 36, 101528 (2020) article  google scholar  download references author information authors and affiliations school of technology, university of campinas, limeira, sp, brazil washington burkart freitas & joão roberto bertini junior authors washington burkart freitasview author publications search author on:pubmed google scholar joão roberto bertini juniorview author publications search author on:pubmed google scholar corresponding author correspondence to washington burkart freitas . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper freitas, w.b., bertini junior, j.r. (2021). tactical asset allocation through random walk on stock network. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_35 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_35 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords tactical asset allocation stock networks random walks portfolio management markowitz portfolio publish with us policies and ethics profiles washington burkart freitas view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature fast movelet extraction and dimensionality reduction for robust multiple aspect trajectory classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper fast movelet extraction and dimensionality reduction for robust multiple aspect trajectory classification conference paper first online: 28 november 2021 pp 468–483 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) fast movelet extraction and dimensionality reduction for robust multiple aspect trajectory classification download book pdf download book epub tarlis tortelli portela  orcid: orcid.org/0000-0001-7405-333010, camila leite da silva  orcid: orcid.org/0000-0003-3739-982010, jonata tyska carvalho  orcid: orcid.org/0000-0001-9020-207610 & … vania bogorny  orcid: orcid.org/0000-0002-0159-464310  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 782 accesses 1 citation abstract mobility data analysis has received significant attention in the last few years. enriching spatial-temporal trajectory data with semantic information, which is the definition of multiple aspect trajectories, presents lots of opportunities, but also many challenges. regarding trajectory classification, the state-of-the-art method called mastermovelets has shown to have the best classification accuracy over several datasets. indeed, this method generates interpretable patterns called movelets which are the most discriminant sequences of points. despite its increased performance, the method is computationally expensive and does not scale well, which makes its application unfeasible for large datasets. in this paper we propose a pivot based approach to reduce the search space, selecting only most promising trajectory points to extract movelets. we additionally provide a method to define a limited number of semantic dimensions for movelets. experiments show that the proposed method is at least \(50\%\) faster for extracting the movelets, and shows a average drop of \(82\%\) of input to the classification models while keeping a similar classification accuracy level. additionally, our scalability analysis with respect to computation time shows that the proposed method scales better than the other methods as the dataset grows in number of points, trajectories and dimensions. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others mastermovelets: discovering heterogeneous movelets for multiple aspect trajectory classification article 30 january 2020 ultramovelets: efficient movelet extraction for multiple aspect trajectory classification chapter © 2024 dependency rule modeling for multiple aspects trajectories chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining machine learning motion detection motion perception motility 1 introduction mobility data analysis is important to different purposes and applications. the movement of people, vehicles, ships, and hurricanes are examples of mobility data. this data are represented as a sequence of points located in space and time, called moving object trajectories. in 2016 emerged the concept of multiple aspect trajectories [3, 10], a broader concept in which spatio-temporal points can be enriched with several semantic aspects, as shown in fig. 1. the meaning of semantic aspect is any type of information that is neither spatial nor temporal. this information represents any aspect as the name of a visited place or point of interest (poi), the price or rate of the place, the weather condition, the transportation mode, the individual’s mood, etc. this new trajectory representation poses new challenges on trajectory data mining, specially in classification, which is the problem we focus in this paper. fig. 1. example of multiple aspect trajectory. full size image trajectory classification is the task of finding the class label of the moving object based on its trajectories [8]. it is important for identifying the strength level of a hurricane [8], the transportation mode of a moving object [2], the type of a vessel (cargo, fish, tourism, etc.) [8], the user that is the owner of a trajectory [5], etc. the great challenge related to multiple aspect trajectory classification is the large number and the heterogeneity of the dimensions associated to each trajectory point. most works for trajectory classification have developed methods for a specific problem or application, and considered only space and time information, as summarized in [9]. on the contrary of traditional classification literature that propose new classifiers, trajectory classification relies on developing new methods for feature extraction to feed a classifier. a recent method designed for robust trajectory classification and that has been specifically developed for multiple aspect trajectories is mastermovelets [5]. it extracts the subtrajectories that better discriminate each class, which are called movelets, and that are used as input to classification algorithms. mastermovelets automatically explores all possible dimension combinations in each subtrajectory of any size in the dataset, while seeking for the best subtrajectories for representing the classes. the movelets represent the behavior of a class, and they are normally frequent subtrajectories inside a class, i.e., they are movement subsequences that are recurrent patterns of a class and not common to other classes. a subtrajectory may have any length in terms of number of points, and mastermovelets explores all possible subtrajectory sizes (e.g. one point, two points, etc.). indeed, it explores all dimension combinations (e.g. space; space and time; space, time and poi category, etc.) in order to choose the most discriminant ones, what makes the method very robust, but also very time consuming. therefore, to use it on real world datasets, more efficient strategies must be developed for extracting movelets. in this work we propose a new method for discovering movelets, that is based on [5], and uses a pivot strategy, called supermovelets. the pivots are the trajectory parts (or subtrajectories) inside a class that will potentially generate the best movelet candidates. by introducing this concept, initially our approach reduces the search space for movelet discovery by computing the distance of one subtrajectory only to subtrajectories within the same class, finding the pivots. moreover, instead of using all dimensions, our method automatically selects the number of dimensions that better characterize each class. as a consequence, the proposed method contributes to a significant dimensionality reduction in terms of movelets that will be used by the classifier for reducing the classification training time. in summary, we make the following contributions: (i) we propose a new and more efficient method for discovering movelets, maintaining similar accuracy levels; (ii) propose a smart strategy that automatically selects the best number of dimensions for each class; (iii) mitigate the curse of dimensionality problem, by reducing the number of movelets; (iv) present a robust experimental evaluation to show the computational time reduction and scalability. the rest of the paper is organized as follows: sect. 2 presents the main works related to multiple aspect trajectory classification, sect. 3 presents the main concepts for understanding our approach, sect. 4 describes our method, sect. 5 presents the experimental evaluation, and sect. 6 concludes the paper. 2 related works existing methods for trajectory classification do not propose new classifiers. the focus is usually on discovering a set of discriminant features that better characterize a trajectory class [4, 8, 9]. in this paper we limit the state of the art to works that consider semantic aspects of trajectories. a complete list of works for both raw and multiple aspect trajectory classification can be found in [9]. the first works to use some semantic information for classification are reported in [14, 15]. these works collected data from smartphones such as latitude, longitude, altitude, and date, plus deriving other features like speed, and matching bus or metro lines to classify the user movement in walking, running, or driving. the works of [6, 7, 19] consider only semantic dimensions. in [7] the authors use the semantics of the roads to segment trajectories for classification of vehicles. the methods presented in [6] and [19] use the poi identifier to classify the moving object. the algorithms movelets [4] and mastermovelets [5] discover relevant subtrajectories, which are called movelets, without the need of extracting other features. however, the processing time and computational cost are extremely high. the main difference between these methods is that movelets encapsulates the distances of all trajectory dimensions in a single distance value, while mastermovelets (that has a higher complexity) keeps the distance of each dimension in a vector of distances, and therefore achieves a much better accuracy than movelets when using all data dimensions. mastermovelets [5] was specifically developed for multiple aspect trajectory classification, and it largely outperformed state-of-the-art methods in terms of accuracy. furthermore, the resulting movelets are interpretable, giving insights about the data, so the classification results are explainable. the work of [16] performs much faster than movelets [4], providing a similar classification accuracy level, but for a single dimension. when combining several dimensions, the user must manually test and select the best dimension combinations, while mastermovelets is able to automatically select the best dimension combinations and generate movelets with heterogeneous and different numbers and types of dimensions. another recent work is marc [12], which uses word embeddings and encapsulates all trajectory dimensions including space, time and semantics to feed a neural network classifier. on the spatial dimension this is the first work that uses the geohash [11] combined to other dimensions. it reaches a very high accuracy, outperforming the movelets [4] when using all dimensions, but as the classifier is limited to neural networks, the resulting patterns are not interpretable. 3 basic concepts this section presents the main concepts that are necessary for better understanding this work, and are based on [5], as they share the same structures. we start with the concept of trajectory in definition 1: definition 1 multiple aspect trajectory: a multiple aspect trajectory \(t_i\) is a sequence of m elements \(t_i = \langle e_1, e_2, ..., e_m \rangle \), where each element is characterized by a set of l dimensions \(d=\{d_1, d_2, ..., d_l\}\), also called aspects. in order to simplify the problem, we assume that all trajectory elements have the same number of aspects. hereafter we will refer to trajectory aspects as dimensions. the behavior patterns of moving objects are normally characterized by a trajectory part, and not an entire trajectory. a trajectory part is called subtrajectory. the subtrajectory concept is detailed in definition 2: definition 2 subtrajectory: given a trajectory \(t_i\) of size m, a subtrajectory \(s_{a,b} = \langle e_a, e_{a+1}..., e_b \rangle \) is a contiguous subsequence of \(t_i\) starting at element \(e_a\) and ending at element \(e_b\), where \(1 \le a \le b \le m\). the subtrajectory \(s_{a,b}\) can be represented by all the dimensions d or a subset of dimensions \(d' \subseteq d\). the length of the subtrajectory is defined as \(w =|s_{a,b}|\). in addition, the set of all subtrajectories of length w in \(t_i\) is represented by \(s^w_{t_i}\) , and the set of all subtrajectories of all lengths in \(t_i\) is \(s^*_{t_i}\). to generate every possible subtrajectory from each trajectory of the dataset, it is necessary to consider all possible subtrajectory sizes and combination of dimensions. since an element has multiple and heterogeneous dimensions, first the distances between all elements in the trajectories are calculated, as defined in definition 3. the dimensions often store different data types, e.g., categorical, numerical and so on. hence, using a vector of distances instead of one aggregated value enable the analysis of different dimensions combinations, and the use custom distance functions for each dimension. definition 3 distance vector between two multidimensional elements: the distance between two multidimensional elements \(e_i\) and \(e_j\) represented by c dimensions is \(dist\_e(e_i , e_j)\), that returns a distance vector \(v = (v_1, v_2, ..., v_c)\), where each \(v_k = dist_k\_e(e_i, e_j)\) is the distance between the two elements at dimension k, and respects the property of symmetry \(dist_k\_e(e_i, e_j) = dist_k\_e(e_j, e_i)\). in order to perform trajectory classification, it is necessary to seek for similarities among trajectories of the same class, and compare subtrajectories of the same length. one way is to seek a position with the minimum distance vector between a subtrajectory and a trajectory, this position is called best alignment. furthermore, the best alignment implies pairing a subtrajectory to a trajectory either in a subset or all dimensions together. it begins with calculating distances between the elements of the subtrajectory and a trajectory \(t_i\). for subtrajectories of two or more elements, each alignment position use a representative vector of distances that is a sum of element distances for each dimension. then, the best alignment consists in ranking the distances vector in each dimension for all possible positions, and getting the average rank of distance with the lowest value. as an example of pairing a subtrajectory into a trajectory, fig. 2 (left) presents the best alignment of a subtrajectory (movelet candidate) in the trajectory \(t_i\), in the spatial dimension. we can see that the best alignment, highlighted with a rectangle, is the position where the distance between subtrajectory of \(\mathcal {m}_1\) and the trajectory is minimal. fig. 2. (left) example of subtrajectory best alignment. (right) example of split point selection for a movelet candidate with the dimensions time and poi rating. full size image the combination of subtrajectories with a subset of the trajectory dimensions are called movelet candidates, and they are evaluated and pruned in order to identify the most representative subtrajectories of each class. the movelet candidates are described in definition 4: definition 4 movelet candidate: a movelet candidate \(\mathcal {m}\) from a subtrajectory \(s_{start,end}\) is a tuple \(\mathcal {m} = (t_i, start, w, c, \mathbb {w}, quality, sp)\), where \(t_i\) is a trajectory of the dataset \(\mathbf{t} \); start is the position in \(t_i\) where the subtrajectory begins, and w is the subtrajectory length (\(w = |s_{start,end}|\)); c is a subset of dimensions such that \(c \subseteq d\); \(\mathbb {w}\) is a set of pairs \((w^s_{t_j}, class_{t_j})\), where \(w^s_{t_j}\) is a distance vector of the subtrajectory \(s_{(start,end)}\) to each trajectory \({t_j}\) in \(\mathbf{t} \). the distances are calculated using the best alignment between \(s_{(start,end)}\) and each trajectory \({t_j}\); quality is a relevance score given to the candidate \(\mathcal {m}\); sp is a set of distance values, called split points, that better divide the classes used to measure the candidate relevance. the set \(\mathbb {w}\) is the movelet candidate best alignments, it represents the distances (for each dimension) to all trajectories in the dataset, and one best alignment is selected as the split point vector sp. the split point sp is a given point that divides the multiple dimensional space. figure 2 (right) presents an example of split point selection for a given movelet candidate extracted from the class represented by the x class. this split point is the best alignment in a trajectory of an opposite class o that best separates the classes considering dimensions together, which is a similar concept used in support vector machines (svm). in the example, we evaluate each trajectory from the o class (\(t_4, t_5, t_6\)) using the values of their best alignment to separate the classes. the split point is selected by calculating the f-score value for each trajectory that is not the target class. the chosen split point is the one point that represents the alignment between the movelet candidate and the trajectory \(t_5\), once it gives the best f-score value. after qualifying each movelet candidate, only those with the best quality, without point overlapping in the trajectory \(t_i\) from where it was extracted, are kept and are called movelets. definition 5 movelet: given a trajectory \(t_i\), and a movelet candidate \(\mathcal {m}_x\) containing a subtrajectory s with \(s_{a,b} \subseteq t_i\), \(\mathcal {m}_x\) is a movelet if for each movelet candidate \(\mathcal {m}_y\) containing a subtrajectory \(u_{f,g}\) with \(u_{f,g} \subseteq t_i\) that overlaps \(s_{a,b}\) in at least one element, \(\mathcal {m}_x.quality > \mathcal {m}_y.quality\). the output is a table \(|\mathbf {t}| \times |\mathcal {\mathbf {m}}|\) that will be used as input for training the classification algorithms. each row of the table is a trajectory, each column is a movelet from the set of movelets m, and the table values are 1 and 0, in order to represent the presence or absence of a movelet in a trajectory. a movelet is present in a trajectory when its best alignment distances to that trajectory is lower than the split point values. it means that the movelet covers the trajectory considering the movelet dimensions c and sp values. 4 supermovelets in this section we propose supermoveletsfootnote 1, a new method for reducing the search space and the computational cost for finding movelets. our method is inspired by [18] and [13], and finds pivots, which are the relevant trajectory parts for extracting the movelets. supermovelets also automatically finds a threshold \(\lambda \) for limiting the maximum number of trajectory dimensions required in each movelet candidate, as an alternative for not exploring all dimension combinations. the pivots and maximum number of dimensions \(\lambda \) are extracted for each class individually. the supermovelets can be divided in three main steps: (i) pivot finding, (ii) the automatic selection of \(\lambda \) and (iii) movelets extraction. fig. 3. overview of our proposed method. full size image figure 3 presents an overview of our method. given a trajectory dataset \(\mathbf{t} \), the algorithm starts selecting the trajectories of one class \(\mathbf{t}' \) (step 1) for extracting the pivots. for each trajectory \(t_i\) in the class trajectory set, supermovelets extracts all movelet candidates (step 2). the method calculates the relative frequency to each movelet candidate, with respect to the trajectories of the same class as \(t_i\) (step 3) and a threshold \(\tau \), the most frequent ones will become the pivot candidates. after finding the candidates, supermovelets performs the pivot candidate pruning, where the ones with relative frequency less than \(\tau \) are removed (step 4). this step is detailed in sect. 4.1. next, it selects only the pivot candidates with higher frequency that repeats at least once in the class trajectories (step 5). we suppose that the most discriminant trajectory parts are the ones that repeat in total or partially in the trajectories of the same class. it calculates the \(\lambda \) threshold based on the common number of dimensions in the pivots (step 6), detailed in sect. 4.2. the pivot points are compared and merged into super-pivots when they are neighbors or have overlapping points (step 7). for last (step 8), the super-pivots and the maximum number of dimensions \(\lambda \) will be used for movelet extraction following the same steps as the mastermovelets method, which for each trajectory outputs the movelets and aggregates the result to the class set of movelets \(\mathbf{m}' \) (step 9). 4.1 pivot finding in order to reduce the number of subtrajectories tested for generating movelets, supermovelets identifies the most promising ones of each class, called pivots (definition 6). the pivots are defined by the frequency they appear in each trajectory, assuming that most frequent subtrajectories are more relevant for movelet extraction. by doing so, the movelets search space is reduced, and so is the computational cost, making trajectory classification using movelets faster and scalable. assuming that moving objects have recurrent movement patterns, and that these patterns are more frequent in trajectories of each individual and less frequent in trajectories of other individuals, and considering several data dimensions, we hypothesize that frequent subtrajectories in a class are very discriminant and can lead to similar classification accuracy results as mastermovelets while reducing its computational cost. definition 6 pivot candidate: a pivot candidate is a tuple \(\mathcal {p} = (t_i,s_{(start,end)},\) \(c, \mathbb {w}, quality_{piv})\), where \(t_i\) is a trajectory from \(\mathbf{t} \); \(s_{start,end}\) is a subtrajectory extracted from \(t_i\), c is a subset with the candidate dimensions, \(c \subseteq d\); \(\mathbb {w}\) is a set of pairs \((w^s_{t_j}, class_{t_j})\), that contains the distances of the best alignment of \(\mathcal {p}\) to every trajectory \(\mathbf{t}' \) that belongs to the same class as \(t_i\). the \(w^s_{t_j}\) is a distance vector of the subtrajectory \(s_{(start,end)}\) to each trajectory \({t_j}\) in \(\mathbf{t}' \). the \(quality_{piv}\) is the relative frequency of the \(\mathcal {p}\) in its class. the pivot candidates represent subtrajectories of different sizes and with any dimension combination from trajectories of a given class, and their quality is based on the relative frequency that they appear in the class. the quality of the pivots is measured considering all dimensions that are present in the subtrajectory that originates the pivot candidate. equation (1) describes this quality function for a pivot candidate (\(\mathcal {p}\)) of a class: $$\begin{aligned} quality_{piv} = \frac{\sum ^{d=|c|}_{d=1} freq_{piv}(\mathcal {p},d,\mathbf{t}' )}{|c|} \end{aligned}$$ (1) where c is the set of trajectory dimensions of \(\mathcal {p}\), and the quality is the average proportion that \(\mathcal {p}\) occurred in trajectories of the class in each dimension d of c. as there are different combinations of dimensions in each \(\mathcal {p}\), we measure the relative frequency as the average count that a \(\mathcal {p}\) occurred in each dimension, as described in (2). $$\begin{aligned} freq_{piv}(\mathcal {p}, d,\mathbf{t}' ) = \frac{\sum ^{i=|\mathbf{t}' |}_{i=1} \left\{ {\begin{array}{cc} {1,} &{} \quad {w^s_{t_i}[d] = 0} \\ {0,} &{}\;\; {\text {otherwise.}} \\ \end{array} } \right. }{|\mathbf{t}' |} \end{aligned}$$ (2) considering that \(w^s_{t_i}[d]\) is the distance of the pivot candidate for dimension d, and that \(|\mathbf{t}' |\) is the number of trajectories of a given class. the relative frequency of \(freq_{piv}(\mathcal {p}, d,\mathbf{t}' )\) is the average occurrence of \(\mathcal {p}\) in each trajectory of its class. figure 4 (a) presents an example of the quality given to two pivot candidates with a single dimension each, both extracted from trajectories of class 4 in the figure, which has five trajectories in total (\(t_7\) to \(t_{11}\)). the first pivot candidate \(piv_a\) (fig. 4a) has two points, and was extracted from trajectory \(t_7\). it occurred in trajectories \(t_7\), \(t_8\), \(t_9\) and \(t_{11}\), which represent \(80\%\) of the trajectories from class 4, thus the quality of this pivot candidate is 0.8. the second pivot candidate \(piv_b\) has three points, and was extracted from trajectory \(t_9\). it occurred in trajectories \(t_9\) and \(t_{11}\), which represent \(40\%\) of the trajectories of class 4, thus the quality of this pivot candidate is 0.4. fig. 4. filtering by: (a) quality proportion of the pivot candidates; (b) redundant pivot candidates, both of the same size and trajectory dimensions. full size image after qualifying each pivot candidate, they are filtered by low quality and redundancy. the first filter consists in discarding pivot candidates that have quality lower than \(\tau \). the second filter consists in the application of a duplicate search function over the pivot candidates that passed from the first filter, i.e., keeping only pivot candidates that occur at least two times. figure 4 (b) presents an example of the redundancy that is tackled by the second filter. the pivot candidates \(piv_c\) and \(piv_d\) are considered redundant when both have the same size (two trajectory points) and the same trajectory dimensions with the same values (equal poi names and equal weather conditions). finally, the pivot finding merges the resultant pivots for identifying in each trajectory which are the super-pivots. the main objective of this merging step is to identify consecutive pivots or the ones that overlap. if two pivots are consecutive in the trajectory or they share the same point (they have overlapping points), then the supermovelets ensembles them into one super-pivot. 4.2 selection of \(\lambda \) threshold to combine all trajectory dimensions to generate all possible subtrajectories, seeking for the best dimension combination in each movelet candidate, means that for each subtrajectory there is an explosion of dimension combinations. to limit this combinatorial explosion, the supermovelets automatically selects the maximum number of dimensions that a movelet candidate can have based on the set of pivots, by counting the absolute number of dimensions that each pivot has, as given in (3). the mode of the number of dimensions present in the set of pivots of a class is chosen as the threshold \(\lambda \), given in (4). it means that the movelet candidates extracted from the trajectories of that class will not have more dimensions than \(\lambda \). $$\begin{aligned} freq_\mathbf{p }(dim) = \sum ^{i=|\mathbf{p} |}_{i=1} \left\{ {\begin{array}{cc} {1,} &{}\quad {\mathcal {p}_i.|c| = dim} \\ {0,} &{} {\text {otherwise.}} \\ \end{array} } \right. \end{aligned}$$ (3) $$\begin{aligned} \lambda = \mathop {{{\,\mathrm{argmax}\,}}}\limits _{{dim}} freq_\mathbf{p }(dim) \end{aligned}$$ (4) 4.3 movelets extraction the last step of supermovelets is to extract the movelets. this process follows the same steps as described in sect. 3 and mastermovelets [5], with the difference that the movelet candidates are only extracted from the super-pivots, and the dimension combination is limited to \(\lambda \). with this, we aim to make the trajectory classification faster, scalable and to maintain the same accuracy level obtained when used the original, and larger, dataset. 5 experimental evaluation we used four datasets publicly available and commonly used by state-of-the-art methods. they are check-in trajectories enriched with semantic dimensions from three different location-based social networks (lbsn). these datasets were also used in [5] to evaluate mastermovelets, which makes fairer and easier to compare the results. the trajectories were split in weeks in order to have many trajectory examples for each user as necessary for classification tasks. the class label is the user which is the owner of the trajectories. table 1 shows the characteristics of each dataset, with the number of trajectories, the size of the smallest and the longest trajectory, the attributes of the dataset and their classes. all datasets have a large number of classes, ranging from 193 to 300. table 1. summary of the used datasets.full size table brightkite: is a social media [1] that provides the anonymized user that made the check-in, the poi semantic reference and the spatio-temporal information of where and when the check-in was made. we used a total of 300 random users for analysis, with a filter of a minimum of 10 points and maximum of 50 points per trajectory to guarantee consistency. trajectories were enriched with the semantic information of the weekday of each check-in, and the resultant dataset has a total of 7, 911 trajectories, with trajectory sizes varying from 10 to 50 points. gowalla: has users around the world [1] and has the same dimensions as the brightkite. from this dataset we also extracted a total of 300 random users, while limiting the trajectory sizes for a minimum of 10 and maximum of 50 points, also enriched with the semantic information of the weekday. the resultant dataset has a total of 5, 329 trajectories, with sizes ranging from 10 to 50 points. foursquare: dataset from the foursquare social media with multiple aspect trajectories in new york, usa [17]. it provides the anonymized user that made the check-in, the poi and the spatial position in time when the check-in was made. we considered trajectories with at least 10 check-ins, resulting in 193 users for this dataset. the points were enriched with the semantic information of the weekday, the poi category from foursquare apifootnote 2, the numerical information of the price and rating of the pois, and the weather from wunderground apifootnote 3. the resultant dataset has a total of 3079 trajectories, with trajectory sizes varying from 10 to 144. the foursquare generic is composed by the same trajectories from the foursquare ny, but it consists of a harder problem, where we removed the specific information of spatial position (lat, lon), and the specific semantics of the poi, keeping only the poi category. 5.1 experimental setup we use three evaluation metrics to assess the performance of the supermovelets approach: (i) computational cost, (ii) classification accuracy (acc) and (iii) f-score. the datasets were split using stratified holdout with \(70\%\) of the data for training and \(30\%\) for test. after movelet extraction, we used the multilayer-perceptron (mlp), as it is commonly used and achieved the best results in [5, 9]. the model was implemented using python language, and the kerasfootnote 4 package. the mlp has a fully-connected hidden layer with 100 units, a dropout layer rate of 0.5, learning rate of \(10^-3\) and output layer with softmax activation. the network was trained using adam optimization to improve the learning time and to avoid categorical cross entropy loss, with 200 of batch size, and a total of 200 epochs for each training. in the experiments, we calculated the distance for each dimension by using: (i) euclidean distance for the space, (ii) difference for the numerical, and (iii) simple equality (if is equal or not) for the semantics. the experiments were performed in an intel(r) core(tm) i7-6700 cpu @ 3.40 ghz, with 4 cores and main memory of 32 gb. 5.2 results and discussion after running mastermovelets for movelet extraction with and without using supermovelets, we compare the time spent for completing trajectory classification, the number of movelet candidates generated, the number of movelets extracted and the accuracy and f-score obtained in the trajectory classification. we tested the movelet extraction with and without using the natural to log limit the size of subtrajectories (denoted by the -log suffix) which was originally designed to improve the movelets extraction speed. table 2 summarizes the results. as we can see, the main conclusion is that the supermovelets method makes the movelet extraction at least \(70\%\) faster than the cases in which mastermovelets is used alone and at least \(50\%\) faster than the mastermovelets-log, while maintaining the same accuracy level. table 2. comparison between the number of candidates, computational time, accuracy and f-score of each technique.full size table the most expressive difference in processing time is in the brightkite dataset, where the supermovelets-log completed the task in 35 m, \(94\%\) faster than the mastermovelets-log that made it in 9 h 45. the less expressive difference is found in the foursquare ny dataset, where the supermovelets-log took 9 h 11 to finish the task, while the mastermovelets-log took 18 h 41, a reduction of \(50.84\%\) in time. an important observation is that besides of the time reduction obtained, and the reduced number of movelet candidates, for any of the classification results the accuracy loss was less than \(1.00\%\) for every dataset. the fastest method, the supermovelets-log even increased the accuracy by \(0.9\%\) in the foursquare ny dataset. movelet candidate generation analysis. we evaluated the number of movelet candidates generated by each technique, and the subsequently number of movelets found. the most expressive difference is in the brightkite dataset that supermovelets reduced in \(94\%\), reducing at least \(53\%\) (foursquare ny) in number of generated movelet candidates (table 2). the proposed method reduce the number of movelets, which are the attributes that are used for training the classifier, in at least \(65\%\) in comparison to the mastermovelets. for example, in the brightkite dataset, mastermovelets produced a total of 54, 739 movelets, while supermovelets outputted 5, 514, a reduction of \(89.92\%\), while the supermovelets-log produced 3, 540, a reduction of \(93.76\%\) compared to mastermovelets-log. this means that besides reducing the movelet extraction time, it reduced also the dimensionality of the classification task, which is inherently high when using movelets. indeed, the smallest dimensionality reduction achieved by the supermovelets was \(65,81\%\) in the foursquare generic dataset, in which the mastermovelets-log method produced 20, 068 movelets, against 6, 861 generated by supermovelets, and the supermovelets-log produced 5, 399, a reduction of \(77,48\%\). scalability performance comparison. we evaluate how supermovelets improves the scalability of movelet extraction over different dataset configurations. we used three synthetic datasets with different characteristics, as designed in [5]: (i) with fixed number of 200 trajectories and variation in the trajectory size from 10 to 400 points with 1 dimension, (ii) with fixed trajectory size of 50 points and variation in the number of trajectories in the dataset from 100 to 4000 trajectories with also 1 dimension, and (iii) with fixed number of 200 trajectories and fixed trajectory size of 50 points, but with variation in the number of dimensions, from 1 to 5. figure 5 shows the scalability results for each dataset. in any of the datasets, we can see that the time spent by mastermovelets increases faster than the others, requiring computational power in a greater scale as the dataset increases. on the other hand, the time spent by supermovelets-log increases in a smaller scale in every dataset, and is the faster compared technique, as it demands less computation time, with a time reduction of \(98.0\%\) in comparison to the mastermovelets, and \(80.48\%\) in comparison to the mastermovelets-log. fig. 5. scalability of running time varying (a) the size of the trajectories, (b) the number of trajectories and (c) the number of dimensions. full size image the mastermovelets-log and supermovelets are in middle-term, where the master movelets-log outperforms the supermovelets in the dataset of variational trajectory size presented in fig. 5 (a), as this is the most effective scenario for limiting the movelet candidate size to the ln of the trajectory size. it is noteworthy that supermovelets outperforms the mastermovelets-log in the dataset of fixed trajectory size presented in fig. 5 (b) and has a similar behavior in the dataset of varied trajectory dimensions of fig. 5 (c). 6 conclusion in this paper we proposed a new method for extracting movelets, as the use of the best subtrajectories is currently a promising approach for multiple aspect trajectory classification. both movelets and mastermovelets generate subtrajectories of any size and from every position of the trajectories for finding the movelets. our method does not generate the distance matrix of a trajectory point to all other trajectory points in the dataset, but to the points of the trajectories of a single class. it extracts movelets only from subtrajectories that occur more frequently in the trajectories of the same class, that are the super-pivots. indeed, it limits the number of trajectory dimensions in each movelet candidate by counting the most frequent number of dimensions that appear in the class. experimental results show that the proposed method is much faster for both extracting the movelets and building the classification models. furthermore, scalability experiments show how well the proposed method scales compared to the state-of-the-art method for multiple aspect trajectory classification. notes 1.source code will be published if accepted for publication. 2.https://developer.foursquare.com/. 3.https://www.wunderground.com/weather/api/. 4.https://keras.io/. references cho, e., myers, s.a., leskovec, j.: friendship and mobility: user movement in location-based social networks. in: proceedings of the 17th acm sigkdd international conference on knowledge discovery and data mining, pp. 1082–1090. acm (2011) google scholar  etemad, m., soares júnior, a., matwin, s.: predicting transportation modes of gps trajectories using feature engineering and noise removal. in: bagheri, e., cheung, j.c.k. (eds.) canadian ai 2018. lncs (lnai), vol. 10832, pp. 259–264. springer, cham (2018). https://doi.org/10.1007/978-3-319-89656-4_24 chapter  google scholar  ferrero, c.a., alvares, l.o., bogorny, v.: multiple aspect trajectory data analysis: research challenges and opportunities. in: geoinfo, pp. 56–67 (2016) google scholar  ferrero, c.a., alvares, l.o., zalewski, w., bogorny, v.: movelets: exploring relevant subtrajectories for robust trajectory classification. in: proceedings of the 33rd acm/sigapp symposium on applied computing, pau, france, pp. 9–13 (2018) google scholar  ferrero, c.a., petry, l.m., alvares, l.o., da silva, c.l., zalewski, w., bogorny, v.: mastermovelets: discovering heterogeneous movelets for multiple aspect trajectory classification. data min. knowl. discov. 34(3), 652–680 (2020) article  mathscinet  google scholar  gao, q., zhou, f., zhang, k., trajcevski, g., luo, x., zhang, f.: identifying human mobility via trajectory embeddings. in: proceedings of the 26th international joint conference on artificial intelligence, pp. 1689–1695. aaai press (2017) google scholar  lee, j.g., han, j., li, x., cheng, h.: mining discriminative patterns for classifying trajectories on road networks. ieee trans. knowl. data eng. 23(5), 712–726 (2011) google scholar  lee, j., han, j., li, x., gonzalez, h.: traclass: trajectory classification using hierarchical region-based and trajectory-based clustering. proc. vldb endow. 1(1), 1081–1094 (2008) article  google scholar  leite da silva, c., may petry, l., bogorny, v.: a survey and comparison of trajectory classification methods. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 788–793, october 2019 google scholar  mello, r.d.s., et al.: master: a multiple aspect view on trajectories. trans. gis 23, 805–822(2019) google scholar  niemeyer, g.: geohash (2008). https://en.wikipedia.org/wiki/geohash petry, l.m., silva, c.l.d., esuli, a., renso, c., bogorny, v.: marc: a robust method for multiple-aspect trajectory classification via space, time, and semantic embeddings. int. j. geogr. inf. sci. 0(0), 1–23 (2020) google scholar  sugiyama, m.: local fisher discriminant analysis for supervised dimensionality reduction. in: proceedings of the 23rd international conference on machine learning, pp. 905–912 (2006) google scholar  tragopoulou, s., varlamis, i., eirinaki, m.: classification of movement data concerning user’s activity recognition via mobile phones. in: proceedings of the 4th international conference on web intelligence, mining and semantics (wims14), p. 42. acm (2014) google scholar  varlamis, i.: evolutionary data sampling for user movement classification. in: 2015 ieee congress on evolutionary computation (cec). ieee (2015) google scholar  vicenzi, f., petry, l.m., da silva, c.l., alvares, l.o., bogorny, v.: exploring frequency-based approaches for efficient trajectory classification. in: the 35th acm/sigapp symposium on applied computing, online event, sac 2020, brno, czech republic, 30 march–3 april 2020, pp. 624–631. acm (2020) google scholar  yang, d., zhang, d., zheng, v.w., yu, z.: modeling user activity preference by leveraging user spatial temporal characteristics in lbsns. ieee trans. syst. man cybern. syst. 45(1), 129–142 (2015) article  google scholar  zhang, z., zhang, h., wen, y., zhang, y., yuan, x.: discriminative extraction of features from time series. neurocomputing 275, 2317–2328 (2018) article  google scholar  zhou, f., gao, q., trajcevski, g., zhang, k., zhong, t., zhang, f.: trajectory-user linking via variational autoencoder. in: ijcai, pp. 3212–3218 (2018) google scholar  download references acknowledgments this work has been partially supported by the brazilian agencies capes (coordenação de aperfeiçoamento de pessoal de nível superior finance code 001), cnpq (conselho nacional de desenvolvimento científico e tecnológico) and fapesc (fundação de amparo a pesquisa e inovação do estado de santa catarina project match co-financing of h2020 projects grant 2018tr 1266). author information authors and affiliations universidade federal santa catarina, florianópolis, santa catarina, brazil tarlis tortelli portela, camila leite da silva, jonata tyska carvalho & vania bogorny authors tarlis tortelli portelaview author publications search author on:pubmed google scholar camila leite da silvaview author publications search author on:pubmed google scholar jonata tyska carvalhoview author publications search author on:pubmed google scholar vania bogornyview author publications search author on:pubmed google scholar corresponding author correspondence to tarlis tortelli portela . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper portela, t.t., da silva, c.l., carvalho, j.t., bogorny, v. (2021). fast movelet extraction and dimensionality reduction for robust multiple aspect trajectory classification. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_31 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords trajectory classification multiple aspect trajectories data mining movelets publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature diversity in data for speech processing in brazilian portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper diversity in data for speech processing in brazilian portuguese conference paper first online: 31 january 2025 pp 122–136 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) diversity in data for speech processing in brazilian portuguese download book pdf download book epub giovana meloni craveiro  orcid: orcid.org/0009-0008-8509-219x9 & julio cesar galdino  orcid: orcid.org/0000-0001-6378-46489  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 235 accesses 1 citation abstract striving to attend ai ethical guidelines is essential when developing and testing ai systems in order to ensure safe and trustworthy applications. however, these guidelines can be too general. the analysis presented here concerns the ethical principle of diversity, by discussing its application to the field of speech processing, using the task of prosodic segmentation of spontaneous speech as a case study. particularly, it covers the relevance of including diversity of speaker’s profiles and regional variants in data used for training and developing ai applications, in the context of brazilian portuguese (bp). the contributions brought by this study are: (i) a discussion of the application of the diversity principle in the context of corpora for speech applications, considering some relevant aspects and the process we formulated to select a diverse sample of speakers to compose our corpus; (ii) a literature review of the current scenario of available corpora for the task of prosodic segmentation of spontaneous speech in bp, focused on the diversity of the data; (iii) a publicly available speech corpus (the corpus is publicly available in our github repository https://github.com/nilc-nlp/mupe-diversidades/ under the cc by-nc-nd 4.0 license) containing 2 h 32 min 15 s of spontaneous speech audios in bp, their revised transcriptions with automatic prosodic segmentation annotation, elaborated to comprise diversity of age, gender, and accents (17 brazilian states). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a corpus of neutral voice speech in brazilian portuguese chapter © 2022 the biovisualspeech european portuguese sibilants corpus chapter © 2020 a large dataset of spontaneous speech with the accent spoken in são paulo for automatic speech recognition evaluation chapter © 2025 1 introduction great effort has been directed towards regulation of artificial intelligence internationally, in order to promote safe and trustworthy ai applications. in march 2024, the eu artificial intelligence act was approved by the european parliament. in brazil, bill project 2338/2023, inspired on the european ai act, is currently under consideration and should be voted before the end of the year. the high-level expert group on artificial intelligence, set up by the european commission, also makes available an assessment list for trustworthy artificial intelligence (altai) [2]. this list contains a series of questions to guide developers in order to ensure their ai systems adhere to the seven required ethic principles as proposed by the eu: human agency and oversight, technical robustness and safety, privacy and data governance, transparency, diversity, non-discrimination and fairness, societal and environmental well-being, and accountability. however, these are general guidelines that can be applied to all kinds of ai systems, and it could be challenging to effectively know how to apply such orientations when developing and testing a specific tool. this article, guided on the referred assessment list, intends to elaborate mainly on the ethical principle of diversity inside the scope of speech processing. the idea behind it is to bring to light some important factors that one should consider when choosing data for developing and testing speech applications. to do so, along with the discussion, we present a use case of applying such considerations when creating a corpus proposed for the task of prosodic segmentation of spontaneous speech in brazilian portuguese (bp). as brazil comprehends great diversity of linguistic variants, the principle of diversity, which emphasizes the relevance of developing applications that work for every individual, despite of their differences, can be widely explored. the contributions brought by this study are: (i) a discussion of the application of the diversity principle in the context of corpora for speech applications, considering some relevant aspects and the process we formulated to select a diverse sample of speakers to compose our corpus; (ii) a literature review of the current scenario of available corpora for the task of prosodic segmentation of spontaneous speech in brazilian portuguese (bp), focused on the diversity of the data; (iii) mupe-diversidades, a publicly available spontaneous speech corpus in bp, with automatic prosodic segmentation annotation. it contains 2 h 32 min 15 s and is more diverse (balanced in gender, comprehending accents of 17 brazilian states, various education levels and ages) than those currently available with prosodic segmentation annotation. we present the discussion of some ai ethic principles and guidelines when applied to this scenario at sect. 2, the task of prosodic segmentation at sect. 2.1, the current scenario of corpora available today for prosodic segmentation of spontaneous speech in bp at sect. 3, the corpus we generated during this study at sect. 4, the process we formulated to make it as diverse as possible at sect. 4.1 and its statistics at sect. 4.2. 2 background one of the principles covered at altai [2] is named “diversity, non-discrimination and fairness” and alludes to the risk that ai systems could exacerbate prejudice unintentionally, by making use of incomplete or biased data. additionally, it sustains that the systems should be accessible to all people, regardless of their age, gender, abilities or characteristics. one of the proposed questions at the assessment list is “did you consider diversity and representativeness of end-users and/or subjects in the data?”. in the context of speech processing, in order to guarantee that inclusion, several perspectives must be accounted for. for instance, it is important to consider the diversity of segments (consonants and vowels) and suprasegmentals, such as tone, speech rate, etc. (which can be prosodic in nature). the diversity in how these speech units can be expressed by different people justifies careful consideration when dealing with speech processing applications. some of those units and variations are explained below. speech contains specific features related to the production of vowels and consonants. vocalic segments, for example, can be classified based on their height (open, closed) and may have secondary properties, such as undergoing nasalization [11]. the proposal of [28] presents seven oral vowels: high front [i], high back [u], mid-high front [e], mid-high back [o], mid-low front [ɛ], mid-low back [ɔ], and central [a]. pretonic vowels, that is, those occurring before the stressed syllable, represented by “e” and “o”, create a distinction between speech patterns in the north/northeast of brazil, which typically exhibit an open production, while speech in the central/west/southeast/south regions tend to have a closed realization [15]. the consonantal system of brazilian portuguese can be identified by place and manner of articulation [13]. one of the secondary articulations of consonants that stands out in terms of linguistic variety is the process of palatalization. in this phenomenon, the tongue changes direction towards the hard palate, resulting in the sound of [s], for example, being produced as what we know as a wheezing in some regions of the country (fe[s]ta x fe[ʃ]ta; me[z]mo x ) [6]. this change occurs in a coda position, which is a final syllable constituent attached to a cv (consonant-vowel) unit [3]. apart from [s], brazilian portuguese exhibits a wide variety in the production of [r] in coda, with occurrences of simple tap , vibrant [r], retroflex [ɻ], velar fricatives [x] and glottal [h] [16]. additionally, regardless of language, speakers commonly utter pauses at grammatical junctures. the regularity and duration of pauses vary according to speech rate (how fast one is talking), text genre, emotional state, stylistic intention, age and experience of the speaker [31]. speech rate and fundamental frequency (f0), that is, the most common frequency or pitch that a person uses when speaking, also vary according to the characteristics of the speaker and spoken text, such as text genre, age and experience. 2.1 prosodic segmentation the task of prosodic segmentation consists of relying on prosodic markers to divide an audio of speech into intonational units (ius) [21]. there is no precise definition of ius but they necessarily consist of a grouping of words delimited by prosodic cues, which often include a well-defined pitch contour [4]. ius can also be split considering terminal boundaries (tb) and non-terminal boundaries (ntb), which establish concluded utterances, and breaks in non-concluded utterances, respectively [25]. computationally, prosodic segmentation has been explored through diverse approaches, which consider specific sets of features that could be of acoustic or syntactic nature. such features include change in intensity [14, 20], pauses [4, 14, 20], f0 [14, 20], and speech rate [4, 14, 20]. 2.2 speaker’s profiles each individual has a unique speaking style, which can vary in many traits. however, despite individual differences, in brazilian portuguese, a study that measured f0 yielded an average fundamental frequency of 105 hz for men, 213 hz for women, 290 hz for children before puberty, and 440 hz for newborns [24]. additionally, the study by reubold, harrington and kleber [22] shows a longitudinal analysis of the extent to which age affects f0 and formant frequencies. it presents the analysis of recordings of two speakers over a 50 year period. for the female speaker studied, a decrease of f0 and f1 (first formant in vowels) is observed as she gets older. furthermore, there are geographical and social factors that also highly impact speech style. for example, depending on the level of education, speakers may or may not adhere to the standard language norm [26], such as the syncope of proparoxytone words (“abóbora” “abóbra”) [5] and the consonant substitution (“planta” “pranta”) [9], which are more frequent in the speech of people with lower levels of education [26]. 2.3 regional linguistic variants there are several characteristics that differentiate accents among different regions in brazil. not only each state has a unique accent, but a single city among a state could have its own specific accent as well. the atlas linguístico do brasil (alib) [17] presents a set of speech characteristics that occur in the capitals of brazil. alib demonstrates that there is preference for the realization of open pretonic vowels, as in “t[é]l[é]fone” and “c[ó]p[é]rar”, in the north and northeast regions of brazil, while in other regions closed pretonic vowels prevail, as in “t[ê]l[ê]fone” and “c[ô]p[ê]rar”. tonic vowels are usually nasalized throughout the country, when the following segment is also nasal, but there are dialects in which pretonic vowels can be nasalized j[ã]nela or open j[a]nela [11]. among consonantal segments, the process of palatalization of /s/ in coda position presents differences among regions. while it is alveolar at most regions, it tends to be produced with a wheezing, characteristic of the palatalization, by speakers from rio de janeiro, recife and northern states [7]. this process can also occur with the consonants /t/ and /d/, assimilating the characteristic of the following high vowel /i/, as in “tia”, which can be pronounced with or without a wheezing [‘tʃia], [12]. in these consonants, three variants can exist, with higher rates of alveopalatal affricates in rio de janeiro and lower rates in recife, as well as other variations of the phenomenon, involving alveolar occlusives [t, d] and alveolar affricates [ts, ds] [1]. a review of this phenomenon can be verified in [29]. in brazil, there is also a wide variety of /r/ sounds, especially at the end of syllables. alib shows that in the capitals, there is more realization of the glottal fricative in the north and northeast of the country, and a higher index of velar fricative in rio de janeiro and espírito santo. the atlas indicates that the tap is prominent in são paulo, curitiba, and porto alegre, while the retroflex has higher indices in mato grosso and mato grosso do sul. furthermore, [27] shows that there are two melodic patterns when brazilians ask a certain type of question, dividing the country into two: the north and northeast regions follow an ascending pattern, while the central-west, southeast, and south regions exhibit a different pattern. there are also descriptions related to the intonation of declarative sentences, characterizing the intonational variety of brazilian capitals. 3 literature review on prosodic segmentation the available corpora for the task of prosodic segmentation of spontaneous speech in brazilian portuguese are nurc-sp [23], nurc-recife [18], c-oral brasil i and ii. while the annotation at nurc-sp and c-oral brasil i and ii follow the linguistic theory presented at raso, teixeira and barbosa’s work [20], which considers terminal and non-terminal intonational units, the annotation at nurc-recife relied on annotators intuitively segmenting the audios into ius. it presented a high annotator agreement rate (fleiss’ kappa > 0.7) and each sample was revised thrice by other annotators [18]. in raso, teixeira, and barbosa’s work [20, 30], the dataset used was composed of a sample of monological audio extracted from c-oral-brasil i and ii, then annotated in terms of prosodic segmentation considering tbs and ntbs. it totals approximately 17 min of spontaneous speech, about 1 min from each of the 14 speakers. there are 4 speakers from minas gerais (sete lagoas, rio pomba, rio espera, belo horizonte), 2 from the city rio de janeiro, 1 from pará (belém), 1 from santa catarina (florianópolis) and 2 from são paulo (diadema, são paulo). the remaining 4 are of unknown origin. all speakers are male and there is no information about their age and level of education. the samples comprise three text genres: informal and formal speech in natural context and television speech, and all were classified with high acoustic quality. nurc-sp comprises solely the linguistic variant of the capital of são paulo, but contains around 44 h of audios, with revised transcription and manual annotation of prosodic segmentation with tbs and ntbs. nurc-recife comprises solely the linguistic variant of recife and contains around 300 h, of which some feature prepared speech. both nurc-sp and nurc-recife feature speakers with higher education, with age ranging from 25 to over 56 years old, comprise audios of varying acoustic quality and are equally divided between men and women [8, 10]. none of these corpora contain information about the specific speech phenomena and accent actually perceived in its speakers or information about whether they migrated from their place of birth, and only c-oral brasil is not publicly available. table 1 exhibits characteristics of existing corpora that contain spontaneous speech in bp with prosodic segmentation annotation, including the one we created during this study in an effort to embrace more brazilian linguistic variants (mupe-diversidades), which is described at sect. 4. the speech variety of 8 states of brazil and distrito federal are not comprehended by any of the corpora. particularly, the majority of states not comprehended are from the north region of brazil (amazonas, roraima, acre, amapá, tocantins) while two northeastern states (maranhão and rio grande do norte) and two midwestern (mato grosso and distrito federal) also lack representation. table 1. statistics of existing corpora of spontaneous speech in bp with prosodic segmentation annotation. in total, 17 brazilian states are comprehended and 9 are lacking. note: dur: duration, bal: balanced, unk: unknown, ne: no education, ies: incomplete elementary school, ces: complete elementary school, te: technical education, ib: incomplete bachelor’s degree, cb: complete bachelor’s degree, m: master’s degree.full size table 4 mupe-diversidades project tarsila is working on several corpora of annotated audiofootnote 1, one of which consists of a collection of 289 anonymized life stories shared by são paulo’s museu da pessoafootnote 2 (mupe) in a partnership with icmc-usp and the federal university of goiás (ufg) and totals 324.09 valid hours of audio automatically transcribed with whisperx [19], an asr model trained on large datasets that provides accurate automatic transcriptions using the large-v2 model of whisper, and diarized via pyannote-audiofootnote 3, an open-source python toolkit for speaker diarization. the automatic transcriptions were reviewed by ten students from the course of linguistics at the federal university of alagoas (ufal) and at the faculty of philosophy, languages and human sciences of the university of são paulo (fflch-usp). this dataset is named coraa mupe and is not publicly available yet, but it will soon be released. coraa mupe includes some information about each speaker, but, in many cases, some of the categories were incomplete. education level was one of those cases: there are 39 speakers with a bachelor’s degree, and from 1 to 7 speakers that varied among incomplete middle school, complete middle school, incomplete high school, complete high school, incomplete bachelor’s degree, master’s degree, doctorate, technical education. they were all born among 1899 and 1991. for the majority of speakers, there is information about the state and country where they were born. 17 states of brazil, and a few countries are mentioned. additionally, around 46.3% (133) of speakers were labeled as women and the remaining 53.7% (154) were labeled as men. although the dataset is quite balanced in gender, there is great discrepancy in the perspective of place of origin. figure 1 exhibits the distribution of speakers by state of birth. there are 15 speakers labeled with “-”, who were born in different countries, but live in brazil and speak portuguese. fig. 1. distribution of speakers from coraa mupe by state of birth. full size image aiming at increasing diversity of brazilian regional linguistic variants and speaker’s profiles in speech corpora, particularly for the task of prosodic segmentation, we have carefully created a selection of samples from coraa mupe, named mupe-diversidades, and are making it publicly available at githubfootnote 4. our dataset contains a sample of approximately 10 min of speech from each state, with their respective transcriptions with prosodic segmentation annotation. in its current state, version 0, each sample consists of the audio, its revised transcription with automatic prosodic segmentation annotation considering tbs and ntbs, conducted according to the methodology described at our previous work [10]. however, during the manual review of the prosodic segmentation, we noticed that automatic cuts at every 30 s, which were part of the automatic transcription process carried with whisperx, jeopardize the quality of the prosodic segmentation process, as some prosodic information is lost at every cut. thus, a second version, named version 1.0, containing the samples of audios without cuts and their respective transcriptions with manually reviewed prosodic segmentation annotation will be available soon. furthermore, the anonymization process consists of replacing names with their first letter, such as replacing “ana” with “[a]” or “zé luís” with “[zl]” in transcriptions, and silencing the respective parts in audio. 4.1 methodology for creating a diverse sample, mupe-diversidades regarding the selection, it would be ideal to include a balanced number of people to be representative of each unique combination of features (gender, age, location, education level), but there are so many variables and often limited resources to do so that it may only be feasible to formulate strategies to include as much diversity as possible. an interesting strategy regarding age is to create different age groups. nurc-sp creates three (i:25–35, ii:35–55, iii:56+) [23], which we use at table 2 to classify our speakers by age group. in our case, we could only afford to include 2 speakers of each state, so we opted for aiming at including the oldest and youngest speakers of those states, one of each gender, from different cities, and could not consider education as a criteria due to the lack of data. it would be ideal to choose only speakers that did not migrate, but in many cases, it was not possible. 5 min of audio were extracted from each selected speaker, except in cases of states with only one speaker, in which around 10 min were extracted. additionally, considering that other speech datasets for bp target the linguistic variety of capitals of brazil, such as nurc-sp and nurc-recife, here we prioritized cities other than the capitals. for each state, whenever possible, the rules to select a speaker were as follows, in this exact order: 1. select the speaker with the earliest year of birth, as long as their city of origin was not the capital of their state; 2. select the speaker with the latest year of birth, as long as they are from other gender, other city and not from the capital; 3. in case the speaker migrated to another state and presented some characteristics of the accent of the state to which they moved, while did not present some characteristics of the accent of their state of origin, according to alib, the next youngest/oldest speaker who did not migrate, migrated at an older age, or/and did present such characteristic was preferred (this substitution occurred only for rj1 regarding the [ʃ]); the steps above were elaborated considering the context of our corpus, which has a very limited number of speakers for many states and contains fewer people from age groups i and ii. it would also be relevant, due to differences in f0, to include teenager’s and children’s speech samples, but coraa mupe did not contain any speakers below the age of 18. 4.2 statistics the resulting corpus contains 14 (47%) male speakers and 16 (53%) female speakers, ranging from 20 to 91 years old, born in 17 distinct states of brazil. table 2 presents information about the duration of each audio sample, the gender, city, state, level of education (information which was collected through hearing the interviews), year of birth of the respective speaker, and whether they migrated or not to another state, at which approximate year, if such information was available. there is one speaker who did not remember her age, as affirmed in her interview. for her case, there is the indication of the year of the interview. for the others, the age was estimated based on the year of their interview. metadata indicates that at least 67% of speakers migrated from their state of origin, many of them at a young age. that could mean they could have lost some of the characteristics of their original accents. it is important to remember that one or two speakers are not enough to reliably represent all characteristics of the accent of a region, and that their accent may be influenced by other factors, such as migration. also, each person presents individual speech characteristics, other than their accent, and having few speakers could make it harder for an ml training to generalize and identify correctly the elements of a given accent. furthermore, we could only include 17 states of brazil. to present greater transparency of which phonetic phenomenon were included in our corpus, we also classified some of the phonetic occurrences explained in sect. 2. they were observed in the audio samples, with the help of praat. table 3 presents such characteristics, including classification of the enunciation of /r/, enunciation of /s/ with or without wheezing, at the end or in the middle of words, open or closed pretonic vowels, and palatalization of /t/ and /d/. for the latter one, we considered only two categories, either palatalized or non palatalized /t/ and /d/, we make no distinction among [ts, ds] and . for letter /r/, we consider tap [ɾ], retroflex[ɻ], vibrant [r] and fricative, making no distinction among glottal [h] and velar [x] fricatives. the audio samples of the speakers were fully heard and the categories attributed to them were the ones that were recognized at least twice, and appear at the table in order of apparent frequency from left to right. the production of pretonic vowels was particularly hard to classify because their realization can differ depending on the word, so speakers that realized an open pretonic vowel at least once in the audio were attributed with “open”. table 2. information about each speaker of mupe-diversidades, which is identified by an id. the table contains information about gender (g) (m:male, f:female), year of birth (yob), estimated age at the time of the recording, stratified age group (s) (i:25–35, ii:35–55, iii:56+), city and state of origin, education level (ne: no education, ies: incomplete elementary school, ces: complete elementary school, te: technical education, ib: incomplete bachelor’s degree, cb: complete bachelor’s degree, m: master’s degree), approximate year and destination of migration to another state if it occurred (y), or whether it didn’t (no), and the duration of the audio sample included at the corpus (dur.). given the nature of this data, which was collected through the interview, some information about migration is lacking, which is marked with “unk” for “unknown”. the corpus totals 2 h 32 min 15 s.full size table table 3. speech phenomena perceived in each speaker from mupe-diversidades, identified by their ids. here, [s] indicates no wheezing, and [ʃ] indicates wheezing at /s/, “mid” stands for a mid-word occurrence and “end” indicates occurrences at the end of the word. “np” stands for “non palatal”, “p” stands for “palatal”, “f” stands for “fricative”, “r” stands for “retroflex”, “t” stands for “tap”, “v” stands for “vibrant”. the final rows indicate the frequency of the phenomena throughout the corpus considering how many audios of the corpus contain each pronunciation style.full size table fig. 2. screenshots of productions of specific phonemes in samples extracted from mupe-diversidades, analyzed at praat. (color figure online) full size image according to alib, open pronunciation of pretonic vowels should prevail in north and northeastern states, but 10 speakers from those regions realized closed pronunciation, opposed to 6 from the same region who produced open pretonic vowels. no clear relation can be established among migration of such speakers, as the majority of them migrated, regardless of the pronunciation of pretonic vowels. the variation of pronunciation of the /s/ seems more well represented, as most speakers from pará, rondônia, rio de janeiro and pernambuco realize the wheezing, as do some speakers from other states, mainly northeastern. however, it is crucial to remember that alib’s conclusions rely on the accent of the capitals, while this corpus features mainly speakers from other cities of the state, which could have distinct accent characteristics. concerning the /r/, while, in our corpus, tap occurs in all south states and in são paulo, it also occurs in many other states, which could be explained by the migration of many speakers to são paulo. many speakers across states pronounce a fricative /r/, composing the most common pronunciation of the /r/ in the corpus, occurring in 77% of the audios. at fig. 2, we include some screenshots of audio samples of certain segments observed with praat to exemplify what we considered to be members of each category. they contain the respective oscillograms and spectrograms of the sound. as for the difference between open and closed pretonic vowels, we extracted the values of formants 1 and 2 of two samples to exemplify. the sample of the open pretonic vowel of the word “português” from pe2 showed f1 702.52 hz and f2 1246.58 hz at 10.57 s, while the sample of the closed pretonic vowel of the word “coordenador” from rs2 showed f1 467.72 hz and f2 1067.18 hz at 19.22 s. 5 conclusions and future work as discussed in sects. 2.2 and 2.3, in the context of speech processing, it is important to consider covering different speaker’s profiles, including level of education, age, gender, place of origin and accent, in order to attend the ai ethical principle of diversity, and thus, promote technology that is equally efficient to all kinds of speakers. moreover, the transparency principle at altai [2] includes the question “did you communicate the technical limitations and potential risks of the ai system to users, such as its level of accuracy and/or error rates?”. thus, it is also sensible to consider testing the applications to different linguistic variants and speaker’s profiles, and analyzing its performance for each case. as it is reasonable to argue that reporting those characteristics about speech corpora is relevant for ensuring transparency and allowing developers to take such limitations into consideration. to what concerns mupe-diversidades, it is far from being complete and representative of all kinds of speaker’s profiles and regional variants of bp. it lacks material from people under 18, which are known to present different ranges of f0, speakers of 9 brazilian states, speakers with different combinations of age, gender, place of origin and education level, as well as many other bp speakers who may utter phenomena that are not covered at this small sample. it also comprehends only one text genre: interviews about life-stories. however, the corpus also represents the inclusion of 13 regional variants of brazilian states that were not yet covered and the inclusion of 2 h 32 min 15 s of audio with prosodic segmentation annotation to the collection of corpora available today for prosodic segmentation of spontaneous speech in bp. we hope this work encourages researchers to reflect on these issues, contribute to the creation of additional resources for bp, and prioritize the inclusion of greater diversity in future research. notes 1.https://sites.google.com/view/tarsila-c4ai/coraa-versions. 2.https://museudapessoa.org/. 3.https://github.com/pyannote/pyannote-audio. 4.https://github.com/nilc-nlp/mupe-diversidades. references abaurre, m.b.m., pagotto, e.g.: palatalização das oclusivas dentais no português do brasil. gramática do português falado 8, 557–601 (2002) google scholar  ala-pietilä, p., et al.: the assessment list for trustworthy artificial intelligence (altai). european commission (2020) google scholar  athayde, m.d.l., baesso, j.s., dias, r.f., giacchini, v., mezzomo, c.l.: o papel das variáveis extralinguísticas idade e sexo no desenvolvimento da coda silábica. revista da sociedade brasileira de fonoaudiologia 14, 293–299 (2009) google scholar  biron, t., et al.: automatic detection of prosodic boundaries in spontaneous speech. plos one 16(5), 1–21 (2021). https://doi.org/10.1371/journal.pone.0250969 article  math  google scholar  bisol, l., brescancini, c.: fonologia e variação: recortes do português brasileiro. edipucrs (2002) google scholar  brescancini, c.r.: a palatalização em coda em florianópolis-sc: variáveis sociais. in: working papers em linguística (online) (2015) google scholar  callou, d., brandão, s.: o processo de palatalização no português do brasil. linguística 18, 57–73 (2006) google scholar  candido junior, a., et al.: coraa asr: a large corpus of spontaneous and prepared speech manually validated for speech recognition in brazilian portuguese. lang. resourc. eval. 57(3), 1139–1171 (2023) google scholar  costa, l.t.d.: estudo do rotacismo: variação entre consoantes líquidas. master’s thesis, universidade federal do rio grande do sul (2006) google scholar  craveiro, g.m., santos, v.g.d., dalalana, g.j.p., svartman, f.r.f., aluísio, s.m.: simple and fast automatic prosodic segmentation of brazilian portuguese spontaneous speech. in: proceedings of the 16th international conference on computational processing of portuguese (2024) google scholar  cristófaro silva, t.: fonética e fonologia do português: roteiro de estudos e guia/exercícios. são paulo. contexto (2015) google scholar  freitag, r.m.k., souza, g.g.a.: o caráter gradiente vs. discreto na palatalização de oclusivas em sergipe. tabuleiro de letras 10(2), 78–89 (2016) google scholar  kent, r.d., read, c.: análise acústica da fala. cortez editora (2015) google scholar  kocharov, d., kachkovskaia, t., skrelin, p.: eliciting meaningful units from speech. in: proceedings of interspeech 2017, pp. 2128–2132 (2017) google scholar  leite, y.f.: como falam os brasileiros. editora schwarcz-companhia das letras (2002) google scholar  lima, m.m.d.o.: as consoantes róticas no português brasileiro com notas sobre as róticas das variedades de goiânia, goiatuba e uberlândia. master’s thesis, universidade de brasília (2013) google scholar  mota, j.a., ribeiro, s.s.c., de oliveira, j.m.: atlas linguístico do brasil: comentários às cartas linguísticas 1-v. 3. universidade estadual de londrina. editora (2023) google scholar  oliveira, m.j.: nurc digital: um protocolo para a digitalização, anotação, arquivamento e disseminação do material do projeto da norma urbana linguística culta (nurc). chimera: revista de corpus de lenguas romances y estudios lingüísticos 3(2), 149–174 (2016) google scholar  radford, a., kim, j.w., xu, t., brockman, g., mcleavey, c., sutskever, i.: robust speech recognition via large-scale weak supervision. in: krause, a., brunskill, e., cho, k., engelhardt, b., sabato, s., scarlett, j. (eds.) proceedings of the 40th international conference on machine learning. proceedings of the 40th international conference on machine learning, vol. 202, pp. 28492–28518. pmlr (2023). https://proceedings.mlr.press/v202/radford23a.html raso, t., teixeira, b., barbosa, p.: modelling automatic detection of prosodic boundaries for brazilian portuguese spontaneous speech. j. speech sci. 9, 105–128 (2020) article  math  google scholar  reed, b.s.: analysing conversation: an introduction to prosody. bloomsbury publishing, london (2017) google scholar  reubold, u., harrington, j., kleber, f.: vocal aging effects on f0 and the first formant: a longitudinal analysis in adult speakers. speech commun. 52(7–8), 638–651 (2010) article  google scholar  rodrigues, a.c., et ak,: portal nurc-sp: design, development, and speech processing corpora resources to support the public dissemination of portuguese spoken language. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 187–195 (2024) google scholar  russo, i., behlau, m.: percepçäo da fala: análise acústica do português brasileiro. lovise, são paulo (1993) google scholar  santos, v.g., et al.: coraa nurc-sp minimal corpus: a manually annotated corpus of brazilian portuguese spontaneous speech. in: proceedings of iberspeech 2022, pp. 161–165 (2022). https://doi.org/10.21437/iberspeech.2022-33 schwindt, l.c.d.s., quadros, e.s.d., toledo, e.e., gonzalez, c.a.: a influência da variável escolaridade em fenômenos fonológicos variáveis: efeitos retroalimentadores da escrita. revista virtual de estudos da linguagem-revel. novo hamburgo, rs, vol. 5, no. 9 (ago. 2007), 12 f. (2007) google scholar  da silva, j.c.b.: a prosódia regional em enunciados interrogativos espontâneos do português do brasil. revista gatilho 13 (2011) google scholar  silva, t.c.: o método das vogais cardeais e as vogais do português brasileiro. revista de estudos da linguagem 8(2), 127–153 (1999) article  google scholar  silva, t.c., barboza, c., guimarães, d., nascimento, k.: revisitando a palatalização no português brasileiro. revista de estudos da linguagem 20(2), 59–89 (2012) article  google scholar  teixeira, b., raso, t., barbosa, p.a.: detecção automática de fronteiras prosódicas entre unidades entonacionais. revista brasileira de fonologia de laboratório, gradus (2020) book  google scholar  vaissière, j.: language-independent prosodic features. in: cutler, a., ladd, d.r. (eds.) prosody: models and measurements. sslan, vol. 14, pp. 53–66. springer, heidelberg (1983). https://doi.org/10.1007/978-3-642-69103-4_5 chapter  google scholar  download references acknowledgments this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp grant #2019/07665-4) and by the ibm corporation. this project was also supported by the ministry of science, technology and innovation, with resources of law no. 8.248, of october 23, 1991, within the scope of ppi-softex, coordinated by softex and published residence in tic 13, dou 01245.010222/2022-44. special thanks to caroline alves, responsible for the manual review of the prosodic segmentation annotation. thanks are also addressed to the academic excellence program (proex) of the coordination for the improvement of higher education personnel brazil (capes). author information authors and affiliations icmc-usp, são carlos, brazil giovana meloni craveiro & julio cesar galdino authors giovana meloni craveiroview author publications search author on:pubmed google scholar julio cesar galdinoview author publications search author on:pubmed google scholar corresponding author correspondence to giovana meloni craveiro . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper craveiro, g.m., galdino, j.c. (2025). diversity in data for speech processing in brazilian portuguese. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_9 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords speech processing diversity prosodic segmentation publish with us policies and ethics profiles julio cesar galdino view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature federated learning and mel-spectrograms for physical violence detection in audio | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper federated learning and mel-spectrograms for physical violence detection in audio conference paper first online: 12 october 2023 pp 379–393 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) federated learning and mel-spectrograms for physical violence detection in audio download book pdf download book epub victor e. de s. silva  orcid: orcid.org/0000-0002-3756-27669, tiago b. lacerda  orcid: orcid.org/0000-0003-1524-66049, péricles miranda  orcid: orcid.org/0000-0002-5767-754410, andré câmara  orcid: orcid.org/0000-0002-9333-321210, amerson riley cabral chagas  orcid: orcid.org/0009-0007-3218-29609 & … ana paula c. furtado  orcid: orcid.org/0000-0002-5439-531410  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 544 accesses 1 citation abstract domestic violence has increased globally as the covid-19 pandemic combines with economic and social stresses. some works have used traditional feature extractors to identify features from sound signals to detect physical violence. however, these extractors have not performed well at recognizing physical violence in audio. besides, the use of machine learning is limited by the trade-off between collecting more data while keeping users privacy. federated learning (fl) is a technique that allows the creation of client-server networks, in which anonymized training result can be uploaded to a central model, responsible for aggregating and keeping the model up to date, and then distribute the updated model to the client nodes. in this paper, we proposed a fl approach to the violence detection problem in audio signals. the framework was evaluated on a newly proposed synthetic dataset, in which audio signals are represented as mel-spectrograms images, augmented with violence extracts. thereby, it treats it as a problem of image classification using pre-trained convolutional neural networks (cnn). inception v3, mobilenet v2, resnet152 v2 and vgg-16 architectures were evaluated, with the mobilenet architecture presenting the best performance, in terms of accuracy (71.9%), with a loss of 3.6% when compared to the non-fl setting. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others analysis of machine learning algorithms for violence detection in audio chapter © 2022 violence detection through fusing visual information to auditory scene chapter © 2023 domestic violence detection using smart microphones chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. audio journalism categorization machine learning snare speech and audio processing speech and audio signal processing 1 introduction violence against women is widely recognized as a serious public health problem and a violation of women’s human rights [21, 22]. the covid-19 pandemic combined with economic and social tensions, together with measures to restrict contact and movement is increasing violence against women and girls globally. before the pandemic, there was an estimate that for every three women, one will experience violence throughout life. during the pandemic, these women are in their homes, cornered with their abusers, in which they exploit the inability of the woman to make a call asking for help or escape, on the other hand, health services are overloaded, non-governmental organizations and support houses are crowded, closed or were reused as a health unit [19]. in the face of this problem, the detection of physical violence is a challenging task, since it depends on the detection of changes in behavior caused by a disagreement of idea, injustice or serious disagreement [34]. a way to make such detection mechanism readily accessible, it can be embedded in a mobile application in order to help to detect and call the police to prevent the violence situation to go further through rapid intervention. several papers in this topic were proposed recently [25, 34], in which various techniques are used to detect violence in videos. however, there are few studies related to the detection of physical violence by audio. according to [3], only 2 works were found between the years 2015 and 2020 related to the theme, evidencing the complexity and lack of studies in the area. however, it is necessary to respect the privacy of the data collected, stored and processed in a machine learning application, in accordance with european data protection laws (general data protection regulation) and brazilian (general data protection law), where in some cases it is necessary authorization of the owner of the information for manipulation of information. in addition, it takes high computational power to keep machine learning (ml) models running, experiments, training, and even retraining with new data. federated learning [18] (fl) is gaining a lot of attention lately, since it allows the decentralization of the learning process to the user’s own devices. collective knowledge is then aggregated in a centralized model, built over several users’ models on the federated network. thus, the privacy of each user’s data is maintained on the device, and there is no storage of sensitive information in a centralized location. in view of this, it raises the following research question that will guide the development of this work: is it possible to maintain similar results by using the fl approach [18], compared with the traditional approach, in identifying scenes of physical violence through audio mel-spectrograms and cnn’s architectures? the experiments were conducted based on hear dataset [15], a synthetic dataset that has 70,000 instances of 10-second audio clips, divided equally between two classes: presence or not of physical violence. in order to keep the experiments computationally feasible, only 12,500 records were used. the experiments conducted in this work considered the following cnn architectures: inception [32], mobilenet [27], resnet152 [8] and vgg-16 [29], and the results showed that cnn mobilenet was the best among the other models when evaluated in hear dataset, reaching 66.8% accuracy, with a loss of 8.6% when compared to non-fl experiments. 2 background in this section, the concepts of convolutional neural networks (cnn), fl and mel-spectrogram to be used in this work will be presented. 2.1 convolutional neural networks (cnn) a convolutional neural network [5] (cnn) is a deep neural network architecture, very popular in image classification tasks. it can learn to assign importance to relevant aspects of an image, allowing the learning of these characteristics and the distinction between them. for this, unlike traditional algorithms where filters/features needed to be implemented manually, cnn is able to learn a large amount of filters, allowing it to learn the best representation by itself. after the emergence of the alexnet [14] architectures, exposed in the imagenet challenge in 2012, and googlenet [31], presented in the imagenet challenge in 2014, convolutional neural networks became popular influencing the emergence of architectures with upper 22 layers of googlenet, as is the case with resnet [8] with 152 layers. 2.2 federated learning (fl) federated learning [18] (fl) consists of the decentralization of the model learning process to customers (user devices), allowing a global model to be trained from various models of users of a network. in addition, the privacy of each user’s data, often sensitive, is maintained on the device, with no sharing of raw data with the central server. only the resulting trained models are shared on the federated network [7]. furthermore, the fl technique can be applied when data for training is concentrated on other devices, allowing the centralizing server not to require a high computational power. another scenario is when the data is sensitive and privacy needs to be respected, so only the training results are shared and the raw data remains on the source device. the fl technique also supports the constant updating of the model, since new data can be collected by the network of devices and used to improve the model. the training process using fl works through learning rounds, where network devices receive the updated global model and start the learning process with data from their respective devices, and at the end, each device will send the training results to the centralizing server, which will aggregate these results into a global model using an aggregation algorithm, for example federated averaging (fedavg) [18]. this process is repeated until the end of the defined training rounds or until a certain evaluation metric is achieved. 2.3 mel-spectrogram the problem addressed in this work belongs to the sound event classification (acoustic scene classification), which consists of adding markers to identify the existence or not of a particular characteristic present at any time in the audio. one instance of this problem is the detection of the presence of violence in audio samples. for this, hear dataset uses the extraction of mel-spectrogram from audios, which is a mel scale spectogram [11] that represents sound components focused on frequencies accessible to human hearing, eliminating low frequency noise observed in the linear spectogram. from there, it can create a bi-dimencional image to be used in image classifiers. the fig. 1 demonstrates the mel-spectrograms extracted from two short-lived audios containing physical violence (“punch in the face” and “slap in the face” respectively). fig. 1. examples of two short-lived, high-intensity events of different types of physical violence. full size image 3 related works the acoustic scene classification problem is receiving a lot of attention lately. in [15], the authors introduce an application of mel-spectrogram sound representation to be fed in a deep learning neural network for detecting physical violence in audio by converting audio into images. for that, the authors built a synthetic dataset with about 70,000 images, called hear dataset, resulting from the extraction of audio using the mel-spectrogram technique. finally, a comparison of the mobilenet, vgg16, resnet-152 and insceptionv3 architectures were realized, in which mobilenet achieved the best results, with an accuracy of 78.9% and \(f_{1}\) score of 78.1%. [28] presents an experiment to detect violence inside the car using audio signal. for that context, the author built the in-car video dataset with the presence of ambient sounds and music to try to simulate the desired environment, and used mel-spectrogram to generate images that represent each audio. the author then submitted the dataset to the resnet, densenet, squeezenet, and mobile architectures, in which he obtained the best result when applied to the resnet-18 model with 92.95% accuracy. [9] displays an emotion recognition system using deep learning approach from an emotional audiovisual big data. to this end, the audios were processed to obtain the mel-spectrogram, allowing to treat the domain as an image problem, and for the videos some frames were extracted to both feed a cnn each. then, the output of the two cnns were joined using two consecutive extreme learning machines (elms). the result of this union is an input source for a support vector machine (svm) for final classification of emotions. finally, the experiment demonstrates the efficacies of the proposed system involving cnns and elms. [20] demonstrates the content-based audio auto-tagging experiment using deep learning. to do that, it used deep neural network architectures such as fully convolutional neural network (fcn) and convolutional recurrent neural network (crnn) in conjunction with magnatagatune (mtt) dataset. as input for the models, the author used the extraction of mel-spectrogram, transforming the audios into images. as a result, the crnn model obtained a better result with 88.56% of auc-roc (area under the curve receiver operating characteristic) as opposed to 85.53% of the fcn model. the authors in [3] presents two papers related to the detection of violence through audio published between 2015 and 2020, in addition to other works that used only video or audio and video, called multi-modal. the first work, produced by [26], depicts a scream detection system in public transport vehicles, in which it uses the characteristics of mel frequency cepstral coefficients (mfcc), energy, delta and delta-deltas. the author uses the gaussian model mixture (gmm) and support vector machine (svm) models to construct the experiments from its own dataset with about 2500 s seconds of audio. the experiment demonstrated that svm generated a low rate of false alarms and gmm approach had a better identification rate. it is observed that [3] exposes this work even though it is outside the established period. [30] proposes the classification and detection of acoustic scenes evolving domestic violence using machine learning. to this end, the author proposes the use of the svm classifier to detect scenes of domestic violence of a man against a woman through the audio parameters mfcc (mel frequency cepstral coefficients), zcr (zero rate crossing) and energy, which are low-level acoustic parameters. as a result, achieved 73.14% of accuracy for the audio parameter mfcc against 71.3% and 66.54% of average was obtained for energy and zcr, respectively. [24] also presents two papers on the processing of audio signals from deep learning using mel-spectrogram published between 2016 and 2017. the first work, performed by [2], presents an automatic music tagging system using fully convolutional neural networks (fcns) in conjunction with the magnatagatune and million song datasets. the author also used the mel-spectrogram technique to extract characteristics from the audios and convert them to image. the experiment demonstrated that the use of mel-spectrogram as input representation resulted in better performance compared to stfts (short-time fourier transform) and mfccs. [16] demonstrates in the experiment a comparison between mel-spectrogram, frame-level strided convolution layer and sample-level strided convolution layer as input for models using the deep convolutional neural networks (dcnn) architecture. to this end, the author used the magnatagatune and milion song datasets as the input source, and made adjustments to all audios, reducing to 29.1 s of duration and reaming to 22050 hz when necessary. the results showed better results when compared to mel-spectrogram with the other, obtaining 0.9059 of auc (area under the curve) against 0.8906 and 0.9055 of frame-level strided convolution layer and sample-level strided convolution layer respectively. however, to the best of our knowledge, all previous works evaluated architectures that required either the upload of raw user data to train the classifiers, or trained the classifiers on public or synthetic datasets to produce a model to be distributed to users. nevertheless, such models tend to become obsolete over time, due to changes in the examples, or to the lack of personalization to the users real usage. therefore, the goal of this experiment is to use the fl technique and hear dataset (audio mel-spectrograms) to train cnn’s architectures to identify scenes of physical violence and compares the results with the traditional approach to identify whether or not similar results are maintained. the table 1 summarizes everything that has been presented and positions our work in relation to others. table 1. related works as well as our experiment describing algorithm, characteristics extractor, data type and dataset.full size table 4 materials and methods as demonstrated in the past sections, fl and cnn’s architectures are able to decentralize model training from a clients network (devices), permitting a worldwide model to be trained by them, respecting the protection of every client’s information, and lessening the need to have a major infrastructure to train it. hence, this paper points, in light of the past works, to train four cnn based on a transfer learning approach utilizing fl and compare these results to standard deep learning models. fig. 2. the study workflow proposed. full size image the fig. 2 shows the research process for this work. to detect violence in audios, we proposed to evaluate four cnn architectures under a transfer learning setting. we characterized the fl settings with three clients and the central server, which permits the decentralized training of models from a network. the hear dataset was set up as the data source for the experiments, and it was processed to extract mel-spectrograms from the audio. the dataset was parted into three equivalent parts for fl approach, one for each client. we used the google colab platform to run the experiments on the dataset, with a highly ram-enabled environment. at last, we compare the results of the fl approach using accuracy, \(f_{1}\) score, precision, recall metrics. we also show the static (non-parametric) test of friedman [4], pairwise comparisons and compare the fl and non-fl results to analyze the metric results between both approaches. fig. 3. the schematic of the proposed training procedure. full size image fig. 4. the fl schematic of the training process. full size image the experiment scheme shown in fig. 3 describes the step to prepare the dataset as an input to the model in which it is the extraction of the mel-spectrogram from the audio. after that, we can handle the problem as image to be model inputs. for the fl approach, as displayed in fig. 4, the mel-spectograms were split in the fl clients into three folders equally, and every client has its folder of the mel-spectrograms to train the models. then, it would send the outcome to the centralized server after each round to build a worldwide model for classifying the presence or absence of violence. 4.1 dataset the hear dataset [15], is a synthetic dataset that features 70,000 mel-spectro-grams audio representations, constructed from transformations of 35,045 audios combined from three audio open databases from google audioset and free-sound: 15,000 google audioset foreground audios from the inside, conversation and speech classes; 20000 background audios also from google audioset of the classes pets, television, toilet flush, door, radio, water, vacuum cleaner, sobbing, noise, sink and frying; and, 45 short-lasting physical violence sounds from free-sound. with the set of mel-spectrograms, the dataset was divided into 60,000 training audios, 50% with the presence of physical violence and 50% without the presence of violence, 5000 test and 5000 for evaluation basis (50% of each class respectively). for the present experiment, only 12500 records were used, 80% for training and 20% for testing (1250 for validation and 1250 for testing), both with 50% of audios labeled violence and non-violence, for reasons of computational limitation. 4.2 compared methods for the experiment, this paper compared the performance between fl, and non-fl approaches using four cnn’s architectures that were used to verify which one presents the best result from the hear dataset. the selection criteria were: the most recent, from 2015, and representative according to the taxonomy performed by [12]. they are: inception v3 [32]: with a complex architecture, inception v3 is the third edition of google’s inception convolutional neural network, which began as a module for googlenet. the architecture has a high quality in results and uses bottleneck layer and asymmetric filters to lower computational cost. mobilenet v2 [27]: with the proposal of being a light and deep neural network, also published by google, mobilenet v2 allows the construction of highly efficient models with high precision and low memory consumption, allowing use on mobile devices. resnet152 v2 [8]: coming from a residual neural network, resnet v2 differs by introducing a connection called identity, proposed to avoid escape gradient problems or vanishing gradient. in addition, it has its residual blocks called “skip connection”, in which it allows the model to be extremely deep, having up to 152 layers. vgg-16 [29]: convolutional neural network model with increased depth (up to 19 layers), of simple and homogeneous topology, which uses 3 connected layers, but of high computational cost, allowing to deal with large-scale images. to perform the experiment, this paper used the pre-trained models and, through the technique of transfer learning, taking advantage of the weights trained in other contexts to perform this work, including only a few additional layers at the end of each architecture. first it was added a layer of flatten, followed by a 20% layer of dropout, to prevent overfitting. soon after, the experiment included one each of dense layer, with 1024 neurons and relu activation function. finally, a layer completely connected to output with 2 neurons with softmax activation function, according to the two classes (existence or not of violence), for the display of the probabilities of the classes. 4.3 flower framework flower is an open source fl framework that enables experiments to be run using machine learning frameworks such as pytorch, tensorflow, pytorch lightning, mxnet, scikit-learn, tflite, among others, specifically designed to advance fl research, enabling heterogeneous fl workloads at scale [1]. using flower, you can run experiments simulating the existence of clients/devices that, connected to the federated network, can evolve their datasets and send the parameters of what has been learned to the centralizing server, which in turn will receive each update by aggregating into a single model. the framework also allows the customization of different configurations for the experiment, such as waiting for a minimum number of clients to start a training cycle together, select assorted customers who participated in a cycle, perform the measurement of evolution both on the client and on the server, send the initial parameters to each client when starting an experiment. this flexibility allows for a diversification of scenarios in a given experiment. settings. to perform the experiment, this paper used the fedavg [18] strategy, with 3 clients running 1 season per cycle, with 40 cycles, each client having 1/3 of the training dataset. in addition to these settings, it defined the existence of at least 3 clients for the beginning of each cycle, as well as the execution of validations, and also enabled the sending of the initial parameters for each client and set the validation configuration of the updated model using the test dataset, which is performed at each end of the cycle. 4.4 experimental setup the experiment was conducted on the google colabfootnote 1 platform, pro version, with 27 gb of ram and an nvidia tesla p100 pcie gpu with 16 gb. mlflowfootnote 2 v1.20 was used to make annotations, log records, experiment tracking, metric storage and model in a central repository, being an open source platform enabling integration with several libraries used in the life cycle of machine learning solutions. the pythonfootnote 3 v3 language and the tensorflowfootnote 4 v2.6 framework were used to build the cnn models. 5 results the four chosen models were evaluated in the hear dataset, and the results obtained are shown in the table 2. the table also exposes the number of clients and rounds executed, as well as the total execution time (in hours). the mobilenet architecture performed better in terms of accuracy (71.9%), but inception v3 showed better results in the \(f_{1}\) score, precision and recall, 89.1%, 85.8%, 92.7% respectively. table 2. results obtained along with the number of times and runtime.full size table the table 2 presents that the mobilenet and inception v3 architectures had a 1 h difference in run time, a much lower value compared to the time spent by the vgg16 and resnet-152 architectures, which was an average of 12.3 h. an experiment was also performed with the same architectures without using fl techniques, presented in table 3, to check for performance loss when using the technique. another important fact is the standard deviation, presented in the table 2, where it indicates how close to the average the accuracy data is, they are: resnet-152 has the lowest value with 0.013, followed by inception v3 which obtained 0.029, vgg16 with 0.052 and mobilenet with 0.069. table 3. comparison between training with and without fl with the number of times and lead time.full size table in addition, the table 3 presents a loss of percentage points of almost 10% in the accuracy result of the mobilenet and resnet-152 architectures when comparing the results between fl and non-fl. however, for the vgg16 and inception v3 architectures the loss of percentage points is not as significant as in the mobilenet and resnet-152 architectures, since the vgg16 presented 68.4% for training without fl and 65.2% with fl. inceptionv3 presented 61.5% for training without fl and 58.7% for fl. another point to be highlighted is the execution time between the two approaches, where in the fl approach the mobilenet architecture presented shorter execution time and in the non-fl approach the mobilenet and vgg16 architectures presented the same execution time. the static (non-parametric) test of friedman [4] was applied for comparison of multiple models on a single dataset, in which the calculation of significance (p-value) was presented below the significance level \(\alpha \) = 0.050, concluding that the distributions differ and the null hypothesis can be rejected. another point is that the mobilenet architecture presents better performance in terms of accuracy (75.5%), \(f_{1}\) score (75.0%), precision (76.3%) and recall (79.0%) when fl was not used (table 4 and 5). table 4. friedman test statistics for fl.full size table table 5. friedman test statistics for non-fl.full size table table 6. pairwise comparisons for non-fl.full size table table 7. pairwise comparisons for fl.full size table wilcoxon’s [35] post hoc method was applied to verify the null hypothesis for the samples, as a result critical difference diagrams were generated, illustrated in fig. 5. another way to visualize the data and identify the most critical differences is through pairwise comparisons, displayed in the table 6 for non-fl approach and the table 7 for fl. fig. 5. critical difference diagram. full size image 6 conclusion and future work this work aims to investigate the performance of cnns architectures in the detection of physical violence using fl through hear dataset [15], which is a synthetic dataset with 70,000 audios converted to mel-spectrograms (images). however, due to computational limitations, only 10,000 images divided equally into two classes were used: presence or not of physical violence. thus, it was possible to apply consolidated techniques and tools to the audio context. [15] presented the investigation and application of the use of mel-spectrograms in the field of detection of audio violence, opening a gap for their application using fl. in this experiment, the dataset used was divided among 3 clients, each running 1 season each round, in a 40-round cycle, considering four models of cnn: inception v3, mobilenet v2, resnet 152 v2, and vgg-16. finally, the results showed that mobilenet obtained a better result when used with the fl technique, presenting a performance of 71.9% in the accuracy metric, with a loss of 3.6% when compared to the experiment without fl. for future work, it is aimed at conducting tests with models in audios of real violence, application of techniques to provide privacy to data trafficked between client-servers [6, 10, 17, 23, 33], use the model in a larger sample of customers. in addition, it is intended to build comparisons through other models developed via transfer learning from a large-scale pretrained audio neural networks (panns) [13]. notes 1.https://colab.research.google.com. 2.https://mlflow.org. 3.https://www.python.org. 4.https://www.tensorflow.org. references beutel, d.j., topal, t., mathur, a., qiu, x., parcollet, t., lane, n.d.: flower: a friendly federated learning research framework. arxiv preprint arxiv:2007.14390 (2020) choi, k., fazekas, g., sandler, m.: automatic tagging using deep convolutional neural networks (2016) google scholar  durães, d., marcondes, f.s., gonçalves, f., fonseca, j., machado, j., novais, p.: detection violent behaviors: a survey. in: novais, p., vercelli, g., larriba-pey, j.l., herrera, f., chamoso, p. (eds.) isami 2020. aisc, vol. 1239, pp. 106–116. springer, cham (2021). https://doi.org/10.1007/978-3-030-58356-9_11 chapter  google scholar  friedman, m.: the use of ranks to avoid the assumption of normality implicit in the analysis of variance. j. am. stat. assoc. 32(200), 675–701 (1937). https://doi.org/10.1080/01621459.1937.10503522. https://www.tandfonline.com/doi/abs/10.1080/01621459.1937.10503522 fukushima, k.: neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. biol. cybern. 36(4), 193–202 (1980). https://doi.org/10.1007/bf00344251 article  math  google scholar  gu, b., xu, a., huo, z., deng, c., huang, h.: privacy-preserving asynchronous vertical federated learning algorithms for multiparty collaborative learning. ieee trans. neural netw. learn. syst. 33, 1–13 (2021). https://doi.org/10.1109/tnnls.2021.3072238 article  mathscinet  google scholar  hard, a., et al.: training keyword spotting models on non-iid data with federated learning (2020). https://arxiv.org/abs/2005.10406 he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition (2015). http://arxiv.org/abs/1512.03385 hossain, m.s., muhammad, g.: emotion recognition using deep learning approach from audio–visual emotional big data. inf. fusion 49, 69–78 (2019). https://doi.org/10.1016/j.inffus.2018.09.008, https://www.sciencedirect.com/science/article/pii/s1566253517307066 hu, r., guo, y., gong, y.: concentrated differentially private federated learning with performance analysis. ieee open j. comput. soc. 2, 276–289 (2021). https://doi.org/10.1109/ojcs.2021.3099108 article  google scholar  volkmann, j., stevens, s.s., newman, e.b.: a scale for the measurement of the psychological magnitude pitch. j. acoust. soc. am. 8, 208 (1937). https://doi.org/10.1121/1.1901999 article  google scholar  khan, a., sohail, a., zahoora, u., qureshi, a.s.: a survey of the recent architectures of deep convolutional neural networks. artif. intell. rev. 53(8), 5455–5516 (2020). https://doi.org/10.1007/s10462-020-09825-6. http://arxiv.org/abs/1901.06032 kong, q., cao, y., iqbal, t., wang, y., wang, w., plumbley, m.d.: panns: large-scale pretrained audio neural networks for audio pattern recognition. ieee/acm trans. audio speech lang. process. 28, 2880–2894 (2020). https://doi.org/10.1109/taslp.2020.3030497. https://ieeexplore.ieee.org/document/9229505/ krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. commun. acm 60(6), 84–90 (2017). https://doi.org/10.1145/3065386 article  google scholar  lacerda, t. b., miranda, p., camara, a., furtado, a.p.c.: deep learning and mel-spectrograms for physical violence detection in audio. in: the 18th national meeting on artificial and computational intelligence, pp. 268–279 (2021). https://sol.sbc.org.br/index.php/eniac/article/view/18259/18093 lee, j., park, j., kim, k.l., nam, j.: sample-level deep convolutional neural networks for music auto-tagging using raw waveforms (2017) google scholar  lu, y., huang, x., dai, y., maharjan, s., zhang, y.: differentially private asynchronous federated learning for mobile edge computing in urban informatics. ieee trans. ind. inf. 16(3), 2134–2143 (2020). https://doi.org/10.1109/tii.2019.2942179 article  google scholar  mcmahan, b., moore, e., ramage, d., hampson, s., arcas, b.a.y.: communication-efficient learning of deep networks from decentralized data. in: singh, a., zhu, j. (eds.) proceedings of the 20th international conference on artificial intelligence and statistics. proceedings of machine learning research, vol. 54, pp. 1273–1282. pmlr (2017). https://proceedings.mlr.press/v54/mcmahan17a.html nations, u.: policy brief: the impact of covid-19 on women (2020). https://www.un.org/sexualviolenceinconflict/wp-content/uploads/2020/06/report/policy-brief-the-impact-of-covid-19-on-women/policy-brief-the-impact-of-covid-19-on-women-en-1.pdf nayyar, r.k., nair, s., patil, o., pawar, r., lolage, a.: content-based auto-tagging of audios using deep learning. in: 2017 international conference on big data, iot and data science (bid), pp. 30–36 (2017). https://doi.org/10.1109/bid.2017.8336569 organization, w.h.: violence against women (2021). https://www.who.int/news-room/fact-sheets/detail/violence-against-women organization, w.h.: violence against women prevalence estimates, 2018: global, regional and national prevalence estimates for intimate partner violence against women and global and regional prevalence estimates for non-partner sexual violence against women (2021). https://www.who.int/publications/i/item/9789240022256 paul, s., sengupta, p., mishra, s.: flaps: federated learning and privately scaling. in: 2020 ieee 17th international conference on mobile ad hoc and sensor systems (mass), pp. 13–19 (2020). https://doi.org/10.1109/mass50613.2020.00011 purwins, h., li, b., virtanen, t., schluter, j., chang, s.y., sainath, t.: deep learning for audio signal processing. ieee j. sel. topics signal process. 13(2), 206–219 (2019). https://doi.org/10.1109/jstsp.2019.2908700 article  google scholar  ramzan, m., et al.: a review on state-of-the-art violence detection techniques. ieee access 7, 107560–107575 (2019). https://doi.org/10.1109/access.2019.2932114 article  google scholar  rouas, j.l., louradour, j., ambellouis, s.: audio events detection in public transport vehicle. in: 2006 ieee intelligent transportation systems conference, pp. 733–738. ieee (2006). https://doi.org/10.1109/itsc.2006.1706829. http://ieeexplore.ieee.org/document/1706829/ sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv2: inverted residuals and linear bottlenecks. in: 2018 ieee/cvf conference on computer vision and pattern recognition, pp. 4510–4520. ieee (2018). https://doi.org/10.1109/cvpr.2018.00474. https://ieeexplore.ieee.org/document/8578572/ santos, f.: in-car violence detection based on the audio signal. in: yin, h., et al. (eds.) ideal 2021. lncs, vol. 13113, pp. 437–445. springer, cham (2021). https://doi.org/10.1007/978-3-030-91608-4_43 chapter  google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition (2015). https://arxiv.org/abs/1409.1556 souto, h., mello, r., furtado, a.: an acoustic scene classification approach involving domestic violence using machine learning. in: anais do eniac, pp. 705–716 (2019). https://doi.org/10.5753/eniac.2019.9327. https://sol.sbc.org.br/index.php/eniac/article/view/9327 szegedy, c., et al.: going deeper with convolutions (2014) google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision (2015). http://arxiv.org/abs/1512.00567 triastcyn, a., faltings, b.: federated learning with bayesian differential privacy. in: 2019 ieee international conference on big data (big data), pp. 2587–2596 (2019). https://doi.org/10.1109/bigdata47090.2019.9005465 tripathi, g., singh, k.v.d.k.: violence recognition using convolutional neural network: a survey. j. intell. fuzzy syst. 39, 7931–7952 (2020). https://doi.org/10.3233/jifs-201400. https://content.iospress.com/articles/journal-of-intelligent-and-fuzzy-systems/ifs201400 wilcoxon, f.: individual comparisons by ranking methods. biometr. bull. 1(6), 80–83 (1945). http://www.jstor.org/stable/3001968 download references author information authors and affiliations center for advanced studies and systems of recife, recife, brazil victor e. de s. silva, tiago b. lacerda & amerson riley cabral chagas federal rural university of pernambuco, recife, brazil péricles miranda, andré câmara & ana paula c. furtado authors victor e. de s. silvaview author publications search author on:pubmed google scholar tiago b. lacerdaview author publications search author on:pubmed google scholar péricles mirandaview author publications search author on:pubmed google scholar andré câmaraview author publications search author on:pubmed google scholar amerson riley cabral chagasview author publications search author on:pubmed google scholar ana paula c. furtadoview author publications search author on:pubmed google scholar corresponding author correspondence to victor e. de s. silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de s. silva, v.e., lacerda, t.b., miranda, p., câmara, a., chagas, a.r.c., furtado, a.p.c. (2023). federated learning and mel-spectrograms for physical violence detection in audio. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_25 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords federated learning mel-spectrogram physical violence recognition publish with us policies and ethics profiles andré câmara view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a unified framework for average reward criterion and risk | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a unified framework for average reward criterion and risk conference paper first online: 30 january 2025 pp 96–110 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a unified framework for average reward criterion and risk download book pdf download book epub willy arthur silva reis  orcid: orcid.org/0000-0002-2621-77139, karina valdivia delgado  orcid: orcid.org/0000-0002-9120-89879 & valdinei freire  orcid: orcid.org/0000-0003-0330-39319  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 497 accesses abstract the average reward criterion is used to solve infinite-horizon mdps. this risk-neutral criterion depends on the stochastic process in the limit and can use (i) the accumulated reward at infinity, which considers sequences of states of size \(h=\infty \), or (ii) the steady state distribution of the mdp (i.e., the probability that the system is in each state in the long term), which considers sequences of states of size \(h=1\). in many situations, it is desirable to consider risk during the process at each stage, which can be achieved with the average reward criterion using a utility function or a risk measure such as var and cvar. the objective of this work is to propose a mathematical framework that allows a unified treatment of the existing literature using average reward and risk, including works that use exponential utility functions and cvar, as well as to include interpretations with \(1 \le h \le \infty \) not present in the literature. these new interpretations allow differentiating policies that may not be distinguished from existing criteria. a numerical example shows the behaviors of the criteria considering this new framework. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others solving markov decision processes with downside risk adjustment article 11 june 2016 sufficient stochastic maximum principle for mean-field control problems with regime switching in an infinite horizon article 05 november 2025 revisiting the merton problem: from hara to cara utility article 22 february 2021 1 introduction a markov decision process (mdp) provides a mathematical framework for representing and solving sequential decision-making problems under uncertainty in fully observable environments [4]. in these problems, an agent must interact with an environment through actions that can have stochastic outcomes. various optimization criteria are used to find an optimal policy for a markov decision process. the most commonly used criterion in finite and indefinite horizon mdps is accumulated reward. in infinite horizon problems, the average reward per decision step criterion is used. the average reward criterion is also known as long-run average reward, mean-payoff, limit-average reward, or steady-state reward [1]. this criterion is risk-neutral, does not use discount, and is naturally used in cyclic and non-terminating problems. this criterion is fundamental for areas such as controlled queues and inventory systems with frequent restocking, where decisions are made frequently and continually [4]. it is also important for natural resource management and environmental economics, for example, to determine the extraction rate of a renewable natural resource [3]. the average reward criterion depends on the behavior of the stochastic process in the limit, which determines some models of the criterion according to the structure of the markov chain generated by stationary policies. one way to think about the stochastic process in the limit is by summing the rewards (i.e., calculating the expected accumulated reward) and then calculating how much is added to this sum at each step, which is called gain. when time tends to infinity, the gain tends to a constant value. since this sum considers sequences of \(h=\infty \) states, this way of thinking in this work is called \(h=\infty \). another way to think about the stochastic process in the limit is by calculating the steady-state distribution, i.e., calculating the probability that the system is in each state in the long term. considering these probabilities and the immediate reward, the expected reward is calculated. this way of thinking in this work is called \(h=1\). considering the risk-neutral average reward criterion, both ways of thinking about the stochastic process in the limit yield the same result. in many situations, it is desirable to consider risk during the process at each stage. in the financial field, for example, a risk-averse investor might prefer to minimize uncertainties by opting for a lower guaranteed return, even if this implies obtaining a smaller gain [7]. moreover, considering risk at each stage can increase the robustness of policies, allowing them to function in a wider variety of scenarios, such as in the use of renewable natural resources that suffer from greater volatility due to climate change. among the works that use the average reward criterion and consider risk during the process are those that use exponential utility functions [2] and cvar [7] (referred to as steady cvar or long-run cvar). while the algorithm that considers an exponential utility function [2] calculates the accumulated reward and how much is added to this sum, thus following the \(h=\infty \) approach; the long-run cvar algorithm [7] calculates the steady-state distribution and considers the immediate reward, which corresponds to the \(h=1\) approach. the objective of this work is to propose a mathematical framework that provides a unified treatment of the literature on average reward and risk in mdps, including works that use exponential utility functions [2] and cvar [7], as well as to include interpretations with \(1 \le h \le \infty \), allowing differentiation of policies that may not be distinguished with existing criteria. this paper is organized as follows. section 2 reviews the definitions of mdp, average reward criterion, and risk-sensitive average reward literature. section 3 presents the proposed framework. section 4 evaluates the policies of an mdp for risk-neutral and risk-sensitive criteria with the proposed framework. finally, sect. 5 presents the conclusions. 2 background a markov decision process [4] is described by a tuple \(\mathcal {m}=\langle s,a,p,r\rangle \) where: s is a finite set of states; a is a finite set of actions; \(p: s \times a \times s \rightarrow [0,1]\) is a transition function that represents the probability that \(j \in s\) is reached after the agent executes an action \(a \in a\) in a state \(s\in s\), i.e., \(\pr (s_{n+1}=j |s_{n}=s,a_{n}=a) = p(s,a,j)\); and \(r: s \times a \rightarrow \mathbb {r}\) is a reward function that represents the reward of executing an action \(a \in a\) in a state \(s \in s\). in this work, the average reward criterion is used for infinite-horizon problems. the average reward criterion depends on the behavior of the stochastic process in the limit, which determines certain models of the criterion based on the structure of the markov chain generated from stationary policies. among the types of markov chains are unichain, multichain, recurrent, ergodic, and aperiodic chains. a unichain markov chain has a closed irreducible set of states (a set of states where all states communicate with each other and do not communicate with states outside the set) and a possibly empty set of transient states (states that will not be revisited after some point in time). otherwise, the markov chain is multichain. a recurrent markov chain is a unichain chain where the set of transient states is empty. a markov chain is ergodic if it is recurrent and aperiodic. a markov chain is aperiodic if the returns to states occur at irregular intervals. 2.1 risk-neutral criterion the expected total reward of a policy \(\pi \) from the initial state s up to the decision epoch \(n+1\) is a function \(v^\pi _{n+1}\) defined by $$\begin{aligned} v^\pi _{n+1}(s) = e \bigg \{ \sum _{n=1}^{n} r(s_n,a_n) \big | s_1=s,\pi \bigg \} = e \bigg \{ \sum _{n=1}^{n} r_n \bigg \}, \end{aligned}$$ (1) where \(s_n\) and \(a_n\) refer to the random variables of the state and action in the time step n. when n tends to infinity, \(v^\pi _{n+1}(s)\) diverges. the expected average reward of a policy \(\pi \), i.e. the gain of a policy \(\pi \), is $$\begin{aligned} \begin{aligned} g^\pi (s) & = \lim _{n\rightarrow \infty } \frac{1}{n} v^\pi _{n+1}(s) \\ & = \lim _{n\rightarrow \infty } \frac{1}{n} \sum _{n=1}^{n} p^{n-1}_{\pi }r_{\pi }(s), \end{aligned} \end{aligned}$$ (2) where \(p^{n}_{\pi }\) is the n-th power of the transition matrix \(p_\pi \) and \(r_{\pi }\) is the reward function of \(\pi \). a policy with the highest gain \(g^*(s)\) is called the optimal policy \(\pi ^*\), $$\begin{aligned} g^*(s) = g^{\pi ^*}(s) = \sup _{\pi \in \varpi } g^\pi (s), \end{aligned}$$ (3) where \(\varpi \) is the set of stationary policies. in the following sections, the simplified version of the notation shown on the right side of eq. 1 will be used. 2.2 risk-sensitive criterion with utility function there are several ways to incorporate risk into decision making with mdps. one way is to use a utility function \(u: \mathbb {r} \rightarrow \mathbb {r}\), such as the exponential utility function [2]. the exponential utility function u is defined by $$\begin{aligned} u(x) = -(\text {sgn} \gamma )e^{-\gamma x}, \end{aligned}$$ (4) with inverse $$\begin{aligned} u^{-1}(y) = \frac{1}{\gamma } \text {ln}[-\text {sgn}(\gamma )y], \end{aligned}$$ (5) where \(\gamma \) is the risk factor and \(\text {sgn} (\gamma )\) indicates the sign of \(\gamma \). the certainty equivalent \(v_{u}^{\pi }\) is the value whose utility is equal to the utility of the expected value of the accumulated reward \(v^\pi \), i.e., $$\begin{aligned} u(v^\pi ) = u(v_{u}^{\pi }). \end{aligned}$$ (6) a positive risk factor \(\gamma \) indicates risk aversion and implies \(v^\pi _u < v^\pi \), while a negative factor indicates risk propensity and implies \(v^\pi _u > v^\pi \). the utility function applied to \(v_{u}^{\pi }\) is defined as $$\begin{aligned} {u^\pi } = u(v_{u}^{\pi }) = -(\text {sgn} \gamma ) e^{-\gamma {v_{u}^{\pi }}}. \end{aligned}$$ (7) the utility of policy \(u^\pi \) in iteration \(n+1\) can be calculated by equation [2] $$\begin{aligned} {u^{\pi , n+1}(s)} = \sum _{j\in s} p_\pi ^n(s,j)e^{-\gamma r_\pi (s)}{u^{\pi ,n}(j)}, \end{aligned}$$ (8) where \(u^{\pi ,0} = (-1, ..., -1)\), \(|u^{\pi ,0}| = |s|\) e \(p_\pi ^0 = i\). from \(u^{\pi ,{n+1}}(s)\), the certainty equivalent can be calculated by \(v_{u}^{\pi ,n}(s) = u^{-1}(u^{\pi ,n}(s))\). during the process, \(v_{u}^{\pi ,n}\) grows linearly, and its variation is called the certainty equivalent gain, denoted by \(g^\pi _u\). using the exponential utility criterion, an agent seeks a policy that maximizes the certainty equivalent gain of an mdp. howard and matheson [2] propose a policy iteration algorithm that finds a stationary policy and deals only with acyclic and irreducible markov processes (all states communicate). given an arbitrary initial policy, the system of eqs. 9 must be solved in the policy evaluation step to find the certainty equivalent gain \(g_u^\pi \) and the certainty equivalents \(v_u^\pi \) [2] $$\begin{aligned} e^{-\gamma (g_u^\pi +v_u^\pi (s))} =\sum _{j\in s} p(j|s,\pi (s))e^{-\gamma (r(s,\pi (s))+v_u^\pi (j))}. \end{aligned}$$ (9) since there are more variables than equations, one of the certainty equivalent values must be set to 0. then, using the values found in the policy evaluation step of the previous policy, a policy improvement step is applied to select, in each state s, a new action a that maximizes the equation [2] $$\begin{aligned} v_u^\pi (s,a) = \frac{1}{\gamma } \text {ln}\left[ \sum _{j\in s} p(j|s,a)e^{-\gamma (r(s,a)+v_u^\pi (j))}\right] . \end{aligned}$$ (10) when no change occurs in any state, the optimal policy has been found. 2.3 risk-sensitive criterion with cvar another way to insert risk into mdps is through the use of var and cvar risk measures. the var (value at risk) and cvar (conditional-value-at-risk) metrics are widely used for portfolio management of financial assets. var measures the worst expected loss within a given \( \alpha \) confidence level, where \(\alpha \in (0, 1)\). var is defined as the \( 1\alpha \) quantile of z, i.e. $$\begin{aligned} \text {var}_\alpha (z) = \min \{ z | f(z) \ge 1-\alpha \}, \end{aligned}$$ (11) where z is a random variable and f(z) is the cumulative distribution function. an alternative measure is the cvar metric, a coherent risk measure [6], that is computed by averaging losses that exceed the var value. cvar, with a confidence level \(\alpha \in (0, 1)\) is defined as [5] $$\begin{aligned} \text {cvar}_\alpha (z) = \min _{w\in \mathbb {r}} \left\{ { w + \frac{1}{\alpha } e\left[ {(z-w)^{+}}_{ }\,\right] }_{ }\,\right\} , \end{aligned}$$ (12) where \((x)^{+}\) = max\(\{x,0\}\) represents the positive part of x, and w represents the decision variable that, at the optimum point, reaches the value of var $$\begin{aligned} \text {cvar}_\alpha (z) = \text {var}_\alpha (z) + \frac{1}{\alpha } e\left[ {(z-\text {var}_\alpha (z))^{+}}_{ }\,\right] . \end{aligned}$$ (13) mdps with the cvar criterion for average reward were studied for the first time by xia and glynn [7]. however, they only deal with stationary policies associated with ergodic markov chains, and they suggest that the results can be extended to unichain mdps. the cvar gain is defined as [7] $$\begin{aligned} g^\pi _{\text {cvar}_\alpha } = \lim _{n\rightarrow \infty } \frac{1}{n} \sum _{n=1}^{n} \text {cvar}_\alpha (r_{n}). \end{aligned}$$ (14) note that in eq. 14, the random variable is only the immediate reward. additionally, a local optimality equation of bellman is proposed, and a policy iteration-type algorithm is developed. 3 a unified framework in this work, some interpretations of the risk-neutral average reward criterion are considered, followed by interpretations of the risk-sensitive average reward criterion with utility function and cvar. the proposed framework considers stationary and aperiodic policies, i.e., stationary policies that generate aperiodic unichain or aperiodic multichain markov chains. for aperiodic policies \(p^*_\pi =\lim _{n \rightarrow \infty } p^n_\pi \) exists. 3.1 formalization of the risk-neutral criterion one of the most common ways to calculate the value function for mdps is by computing the accumulated reward. this approach can be extended to infinite horizon, where to ensure convergence of the value, it is divided by the number of decision steps n (eq. 15). note that in this equation, the accumulated reward is the random variable. from this equation, various interpretations of the gain of an mdp for a policy \(\pi \) can be derived. the relationships between these interpretations are shown in eqs. 15 to 18. $$\begin{aligned} g^\pi & = \lim _{n\rightarrow \infty } \frac{1}{n} e\left[ \sum _{n=1}^{n} r_{n} \right] \end{aligned}$$ (15) $$\begin{aligned} & = \lim _{n\rightarrow \infty } e\left[ \sum _{n=1}^{n+1} r_{n} \sum _{n=1}^{n} r_{n} \right] \end{aligned}$$ (16) $$\begin{aligned} & = \lim _{n\rightarrow \infty } \sum _{n=1}^{n} \frac{1}{n} e[r_{n}] \end{aligned}$$ (17) $$\begin{aligned} & = \lim _{n\rightarrow \infty } \sum _{n=1}^{n} \frac{1}{nh} e\left[ \sum _{i=n}^{n+h} r_{i}\right] \end{aligned}$$ (18) equation 15 was described before. equation 16 interprets gain as the increase in reward at each decision step. in this approach, the value is calculated as the difference between the expected accumulated rewards considering two consecutive time steps. even though each accumulated reward tends to infinity, the difference between them is finite. this approach is similar to the gain extracted in iterative algorithms, such as the relative value iteration algorithm [4]. equation 17 considers the immediate reward as a random variable. the gain is interpreted as the sum of the expected immediate rewards divided by n. equation 18 conjectures the interpretation of the gain as the cesaro limit of the expected accumulated reward for h decision steps after time n. in this equation, the random variable is the accumulated reward of trajectories of fixed size h. the division is done by nh because n sums of h rewards are made. note that eqs. 16, 17 and 18 converge to some distribution, allowing the use of the steady-state matrix \(p^*_\pi \) to calculate the gain. 3.2 formalization of the risk-sensitive criterion before formalizing the risk-sensitive criterion, some notations used in this section are introduced. let \(\psi \) be a sequence of states and \(\psi (k)\) be the k-th state of this sequence. let \(s^h\) be the set of all sequences \(\psi \) of size h. let \(f_{s,n}^{\pi ,h}: s^h \rightarrow [0,1]\), \(f_{s,n}^{\pi ,h}(\psi )\) is the probability of the sequence \(\psi \) of size h occurring with the process being started in state s and time n, and can be calculated as $$\begin{aligned} \begin{aligned} f_{s,n}^{\pi ,h}(\psi ) & = pr\left( \bigwedge _{k=0}^{h-1} s_{n+k}=\psi (k) | s_{0}=s\right) \\ & = pr\left( s_{n+0}=\psi (0) | s_{0}=s)\prod _{k=1}^{h-1}pr(s_{n+k}=\psi (k)|s_{n+k-1}=\psi (k-1)\right) . \end{aligned} \end{aligned}$$ (19) let \(v^{\pi ,h}_n(\psi )\) be the accumulated reward of the sequence \(\psi \) of size h with the process starting at time n $$\begin{aligned} v^{\pi ,h}_n(\psi ) = \sum _{k=n}^{n+h-1} r( \psi (k) ). \end{aligned}$$ (20) risk-sensitive criterion with utility function. the utility function u can be applied either to the sum of rewards or directly to the immediate reward. the average of the expected utility of the accumulated reward is observed in eq. 21 and represents the approach of howard and matheson [2], which uses the exponential utility function. this equation is similar to eq. 15, but it includes applying the utility function u to the sum of rewards $$\begin{aligned} g^\pi _{u} = \lim _{n\rightarrow \infty } \frac{1}{n} e\left[ u\left( \sum _{n=1}^{n} r_{n} \right) \right] . \end{aligned}$$ (21) equation 22 is equivalent to eq. 21. it calculates the expected value of the utility of the accumulated reward of all sequences of size \(n\rightarrow \infty \) that start in state s and at time 1. then the average of this expected value is calculated by $$\begin{aligned} \begin{aligned} g^\pi _u & = \lim _{n\rightarrow \infty } \frac{1 }{n} \sum _{\psi \in {s^{n}}} f_{s,1}^{\pi ,n}(\psi ) u\left( \sum _{k=1}^{n} r( \psi (k) ) \right) . \end{aligned} \end{aligned}$$ (22) when \(n\rightarrow \infty \), the probability of a state occurring from the sequence of states tends to the steady state at any time \(n\in {n}\), so that the probability of the sequence \(f_{s,n}^{\pi ,h}\) can be calculated using the steady-state probability of each state in the sequence in eq. 19. this interpretation introduces how an arbitrary sequence of states is intended to be visualized in the upcoming equations. equation 23 calculates the sum of the expected utility of the accumulated rewards of sequences of fixed size h starting at any time n such that \(1\le n \le {n}\), with \(n\rightarrow \infty \), and finally their average is calculated by $$\begin{aligned} g_{u}^{\pi ,h} = \lim _{n\rightarrow \infty } \frac{1}{n} \sum _{n=1}^{n} e\left[ u\left( \sum _{i=n}^{n+h-1} r_{i} \right) \right] . \end{aligned}$$ (23) equation 25 reinterprets eq. 23 by inserting a sequence \(\psi \) of h states, the probability of this sequence of states \(f_{s,n}^{ \pi ,h}(\psi )\) (eq. 19) and its accumulated reward \(v^{\pi ,h}_n(\psi )\) (eq. 20). thus, for each sequence of size h starting at time n, its probability is multiplied by the utility of its accumulated reward $$\begin{aligned} {u_{n}^{\pi ,h}} = \sum _{\psi \in {s^h}} f_{s,n}^{\pi ,h}(\psi ) u\left( v^{\pi ,h}_n(\psi ) \right) . \end{aligned}$$ (24) then the sum for all time n of the inverse of \(u_{n}^{\pi ,h}\) is calculated and its average is obtained when \(n\rightarrow \infty \) by $$\begin{aligned} g^{\pi ,h}_u = \lim _{n\rightarrow \infty } \frac{1 }{n} \sum _{n=1}^{n} {u^{-1}(u^{\pi ,h}_{n})}. \end{aligned}$$ (25) for \(h=1\) and u being the identity function, eq. 25 becomes the risk-neutral average reward, equivalent to eq. 2. note that in this work eq. 25 is proposed to classify existing works in the literature concerning h. when \(h=\infty \), eq. 25 is equivalent to the criterion used by howard and matheson [2]. risk-sensitive criterion with cvar. equation 26 considers the average cvar of the accumulated reward. analogous to eq. 21, it considers the accumulated reward of the stochastic process, but instead of applying the utility function u, it applies the cvar function $$\begin{aligned} g^\pi _{\text {cvar}} = \lim _{n\rightarrow \infty } \frac{1}{n} \text {cvar}\left[ \sum _{n=1}^{n} r_{n} \right] . \end{aligned}$$ (26) as far as we know, there is no work in the literature that solves eq. 26. similar to eq. 23, but considering the cvar instead of the utility function u, eq. 27 considers the accumulated rewards of fixed-length sequences h started at each time n and calculates the average of the cvar values of all these sequences: $$\begin{aligned} g^{\pi ,h}_\text {cvar} = \lim _{n\rightarrow \infty } \frac{1}{n} \sum _{n=1}^{n} \text {cvar}\left[ \sum _{i=n}^{n+h-1} r_{i} \right] . \end{aligned}$$ (27) equation 28 reinterprets eq. 27 by inserting a sequence \(\psi \) of h states, the probability of this sequence of states \(f_{s,n}^{ \pi ,h}\) (eq. 19) and its accumulated reward \(v^{\pi ,h}_n(\psi )\) (eq. 20). thus, for sequences \(\psi \) of size h starting at time n: $$\begin{aligned} \begin{aligned} v_{\text {cvar},n}^{\pi ,h} & = \text {cvar}_\alpha (v^{\pi ,h}_n(\psi )) \\ & = \text {var}_\alpha (v^{\pi ,h}_n(\psi )) + \frac{1}{\alpha } \sum _{\psi \in {s^h}} f_{s,n}^{\pi ,h}(\psi ) \left[ {(v^{\pi ,h}_n(\psi )-\text {var}_\alpha (v^{\pi ,h}_n(\psi )))^{+}}_{ }\,\right] . \end{aligned} \end{aligned}$$ (28) then the sum is calculated for all time n and then its average is obtained for \(n\rightarrow \infty \) $$\begin{aligned} g^{\pi ,h}_{\text {cvar}} = \lim _{n\rightarrow \infty } \frac{1 }{n} \sum _{n=1}^{n} v^{\pi ,h}_{\text {cvar},n}. \end{aligned}$$ (29) note that in this work eq. 29 is proposed to classify existing works in the literature concerning h. when \(h=1\), eq. 28 is equivalent to eq. 14 of xia and glynn [7]. 4 numerical example example 1 consider an mdp with 4 states and three policies \(\pi _1\), \(\pi _2\) and \(\pi _3\), where the transition matrix of \(\pi _1\) and \(\pi _2\) is $$\begin{aligned} p_{\pi _1} = p_{\pi _2}= \begin{bmatrix} 0.5 & 0.5 & 0 & 0\\ 0 & 0 & 1 & 0\\ 0 & 0 & 0.5 & 0.5\\ 1 & 0 & 0 & 0 \end{bmatrix}, \end{aligned}$$ (30) and the transition matrix for \(\pi _3\) is $$\begin{aligned} p_{\pi _3} = \begin{bmatrix} 1 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0\\ 0 & 0 & 0 & 0 \end{bmatrix}. \end{aligned}$$ (31) let \(r_{\pi }\) be the reward function of policy \(\pi \), \(r_{\pi _1}=(0,6,0,-5)\), \(r_{\pi _2}=(0,1, 0,0)\) and \(r_{\pi _3}=(\frac{1}{6})\). figure 1 represents the three policies, where the vertices represent the states, the edges the actions, and the values in parentheses represent the transition probability and the reward, respectively. the markov chains generated by the stationary policies in the example 1 are unichain and aperiodic. in this way, all policies have a constant gain, that is, the gain of any policy is independent of the state [4]. fig. 1. graphical representation of the policies of the mdp given in example 1. full size image the steady-state transition matrix is calculated by \(p_\pi ^* = \lim _{n\rightarrow \infty } p_\pi ^{n+1}=p_\pi ^np_\pi \) and for policies \(\pi _1\) and \(\pi _2\) it is $$\begin{aligned} p_{\pi _1}^* = p_{\pi _2}^* = \begin{bmatrix} \frac{1}{3} & \frac{1}{6} & \frac{1}{3} & \frac{1}{6}\\ \frac{1}{3} & \frac{1}{6} & \frac{1}{3} & \frac{1}{6}\\ \frac{1}{3} & \frac{1}{6} & \frac{1}{3} & \frac{1}{6}\\ \frac{1}{3} & \frac{1}{6} & \frac{1}{3} & \frac{1}{6} \end{bmatrix}. \end{aligned}$$ (32) for \(\pi _3\), \(p^*_{\pi _3}=p_{\pi _3}\). although the policies \(\pi _1\) and \(\pi _2\), with the same transition matrix, have the same expected average reward, the distributions of accumulated rewards for policy \(\pi _1\) have greater variance than those for policy \(\pi _2\) for different values of h, as shown in fig. 2. therefore, risk-sensitive criteria are expected to differentiate between policies \(\pi _1\) and \(\pi _2\). fig. 2. graphs of the distribution of accumulated reward and cumulative distribution function (cdf) of accumulated reward for different values of \( h \). full size image figures 2 a), c), e) and g) show the distribution of the accumulated reward for policies \(\pi _1\) (blue bars), \(\pi _2\) (orange bars) and \(\pi _3\) (green bars) of the mdp given in the example 1 for \(h \in \{1,2,5,101\}\), respectively. in addition to the distributions, the values \(v^{\pi ,h}\) (red line), \(v^{\pi ,h}_{u_{1}}\) (purple line), \(v^ {\pi ,h}_{\text {var}_{0.2}}\) (brown line) and \(v^{\pi ,h}_{\text {cvar}_{0.2}}\) (pink line) are showed. the accumulated rewards for \(h=1\) and \(h=2\) range from \(-5\) to 6 for \(\pi _1\) and range from 0 to 1 for \(\pi _2\). for \(h=5\), the accumulated rewards range from \(-5\) to 7 for \(\pi _1\) and from 0 to 2 for \(\pi _2\). for \(h=101\), the accumulated rewards range from \(-5\) to 30 for \(\pi _1\) and from 0 to 25 for \(\pi _2\). the accumulated reward for \(\pi _3\) does not vary for each h. it is expected that a risk-averse criterion may consider these three policies differently. additionally, when h tends to infinity, all accumulated rewards tend to infinity (see figs. 2 a), c), e) and g)). figures 2 b), d), f), and h) show the cumulative distribution function (cdf) for policies with the same values of h. for each policy, the cdfs are symmetric. the median of \(\pi _1\) and \(\pi _2\) are the same for all values of h. in the next subsections, examples of the gain calculation for each of the criteria used are shown. consider the following notation regarding policies: \(a\succ b\) means that the decision-maker prefers a to b, \(a \sim b\) means that the decision-maker has an equal preference for either a or b and \(a\succeq b\) means that the decision-maker prefers or has an equal preference for a or b. 4.1 calculation of gain using the risk-neutral criterion the gain of a policy \(\pi \), defined in eq. 15, can be calculated by dividing the accumulated reward \(v^{\pi }_n\) by the number of steps n, i.e., \(g^ \pi =\lim _{n\rightarrow \infty } \frac{1}{n}v^{\pi }_n\). \(v_n^\pi \) can be calculated iteratively by $$\begin{aligned} v^{\pi ,n+1}=v^{\pi ,n}+p_\pi ^nr_\pi , \end{aligned}$$ (33) where \(v^{\pi ,0}\) is a null vector of size |s| and \(p_\pi ^0 = i\). using this form of calculation for the example, the gain of policies \(\pi _1\), \(\pi _2\) and \(\pi _3\) is \(g^{\pi _1} = g^{\pi _2} = g^{\pi _3} = (0.167, 0.167, 0.167, 0.167)\). the gain of a policy defined in eq. 17 can be calculated by multiplying the steady-state matrix \(p_\pi ^*\) by the policy reward \(r_\pi \), i.e., \(g^{\pi } =p_{\pi }^*r_{\pi }\) [4]. in the example using this second form of calculation, the gain is \(g^{\pi _1} = g^{\pi _2} = g^{\pi _3} = (0.167, 0.167, 0.167, 0.167).\) note that, using the risk-neutral criterion \(\pi _3 \sim \pi _2 \sim \pi _1\). 4.2 calculation of gain using utility function with \(\gamma =1\) for \(h=1\) to calculate the gain \(g^{\pi ,h}_\text {u}\), it is necessary to calculate \(f_{s,n}^{\pi ,h}(\psi )\) (eq. 19) which is the probability of the sequence \(\psi \) of size h occurring with the process starting in state s and in time n. it is also necessary to calculate \(v_{n}^{\pi ,h}(\psi )\) (eq. 20) which is the accumulated reward of the sequence \(\psi \) and \(u(v^{\ pi,h}_n(\psi ))\). table 1 shows the sequences \(\psi \) of size \(h=1\) for policy \(\pi _1\) (column 1), \(f_{s,n}^{\pi _1,h=1 }(\psi )\) (column 2) and \(v_{n}^{\pi _1,h=1}(\psi )\) (column 3) and \(u(v^{\pi ,h}_n(\psi ))\) (column 4). for policy \(\pi _1\), \(u^{\pi _1,h=1}_n=\frac{1}{3}*(-1)+\frac{1}{6}*(-0.002)+\frac{1}{3}*(-1)+\frac{1}{6}*(-148.413)=-25.403\) (eq. 24) is calculated. thus, \(v^{\pi _1,h=1}_{n} = u^{-1}(u^{\pi ,h=1}_{n}) = -3.235\) and \(g^{h=1}_{u_1} = \frac{v^{\pi ,h=1}_{n}}{1} = -3.235\). for policy \(\pi _2\), \(u^{\pi _2,h=1}_n=\frac{5}{6}*(-1)+\frac{1}{6}*(-0.368) = \)0.895. consequently \(v^{\pi _2,h=1}_{n}=u^{-1}(-0.895) = 0.111\) and \(g^{h=1}_{u_1}=0.111\). for policy \(\pi _3\), \(u^{\pi _3,h=1}_n=-0.846\) and \(v^{\pi _3,h=1}_{n}=g^{h=1}_{ u_1}=0.167\). figure 2a) shows the values \(v^{\pi ,h=1}_{u_{1}}\) for the three policies (purple vertical lines), which are used to calculate their respective gains. for \(h=1\) and \(\gamma =1\), \(\pi _3 \succ \pi _2 \succ \pi _1\). table 1. sequences \(\psi \) of size \(h=1\) for policy \(\pi _1\), \(f_{s,n} ^{\pi _1,h=1}(\psi )\), \(v_{n}^{\pi _1,h}(\psi )\) and \(u(v_{n}^{\pi _1,h}(\psi ))\).full size table 4.3 calculation of gain using utility function with \(\gamma =1\) for \(h=\infty \) solving the system of eqs. 9 proposed in [2] to evaluate policies, \({g}^{\pi _1}_u = {g}^{\pi _2}_u = (0.157, 0.157, 0.157, 0.157)\) and \({g}^{\pi _3}_u=(0.167)\). as stated previously, \({g}^{\pi }_u= {g}^{\pi ,h=\infty }_u\) for any policy. note that \(g^{\pi }_u \le g^{\pi }\) for any of the policies, what is expected of a risk-averse agent, whose certainty equivalent gain is less than or equal to the gain. \(g^{\pi ,h}_{u}\) were calculated for \(1\le h \le 2000\) and policies \(\pi \in \{\pi _1, \pi _2, \pi _3\}\). as expected \(g^{\pi ,h}_{u}\) approaches \(g^{\pi ,h=\infty }_{u}\) as h increases. the value of \(g^{\pi _1,h=2000}_{u}=g^{\pi _2,h=2000}_{u}=(0.155, 0.155, 0.155, 0.155)\) and \(g^{\pi _3,h=2000}_{u}=(0.167)\). for \(h=\infty \) and \(\gamma =1\), \(\pi _3 \succ \pi _2 \sim \pi _1\). 4.4 calculation of gain using cvar with \(\alpha =0.2\) for \(h=1\) to calculate the gain \(g^{\pi ,h}_\text {cvar}\), it is necessary to calculate \(f_{s,n}^{\pi ,h}(\psi )\) (eq. 19), \(v_{n}^{\pi ,h}(\psi )\) (eq. 20) and the distribution of these accumulated rewards \(f_{v_{n}^{\pi }}^{h}\). table 1 shows the sequences \(\psi \) of size \(h=1\) for policy \(\pi _1\) (column 1), \(f_{s,n}^{\pi _1,h=1 }(\psi )\) (column 2) and \(v_{n}^{\pi _1,h=1}(\psi )\) (column 3). thus, the possible accumulated rewards are \((-5,0,6)\) with distribution \(f_{v^{\pi _1}_n}^{h=1}=(\frac{1}{6}, \frac{4}{6}, \frac{1}{6})\). considering a confidence level of \(20\%\), \(v^{\pi _1,h=1}_{\text {var}_{0.2}}=\text {var}_{0.2}(v_{n} ^{\pi _1,h=1})=0\) and \(v^{\pi _1,h=1}_{\text {cvar}_{0.2}}=\text {cvar}_{0.2}(v_{n}^{\pi _1,h=1}) =\frac{\frac{1}{6}(-5)+(0.2-\frac{1}{6})*0}{0.2}=-4.167\). thus, \(g^{\pi _1,h=1}_{\text {cvar}_{0.2}} = \frac{v^{\pi _1,h=1}_{\text {cvar}_{0.2} }}{1} = -4,167\). for policy \(\pi _2\), the possible cumulative rewards are (0, 1) with distribution \(f_{v_{n}^{\pi _2}}^{h=1}=(\frac{5}{6}, \frac{1}{6})\). considering a confidence level of \(20\%\), \(v^{\pi _2,h=1}_{\text {var}_{0.2}}=0\) and \(v^{\pi _2,h=1 }_{\text {cvar}_{0.2}}=\frac{0.2*0}{0.2}=0\). thus, \(g^{\pi _2,h=1}_{\text {cvar}_{0.2}} = \frac{v^{\pi _2,h=1}_{\text {cvar}_{0.2} }}{1} = 0\). for policy \(\pi _3\), the only accumulated reward is \(v_{n}^{\pi _3,h=1} = (\frac{1}{6})\) with distribution \(f_{v_{n}^{ \pi _3}}^{h=1}=(1)\). considering a confidence level of \(20\%\), the \(v^{\pi _3,h=1}_{\text {var}_{0.2}}=v^{\pi _3,h=1}_{\text{ cvar}_{0.2}}=\frac{1}{6}\), and \(g^{\pi _3,h=1}_{\text {cvar}_{0.2}} = \frac{v^{ \pi _3,h=1}_{\text {cvar}_{0.2}}}{1} = \frac{1}{6}\). figure 2a) shows the values \(v^{\pi ,h=1}_{\text {cvar}_{0.2}}\) for the three policies (pink vertical lines) that are used to calculate their respective gains. for \(h=1\) and a confidence level of \(20\%\) \(\pi _3 \succ \pi _2 \succ \pi _1\). the values found in this example for \(h=1\) are the same values found with the criterion used by xia and glynn [7]. 4.5 comparison of \(g^{\pi }\), \(g_{u}^{\pi ,h}\), \(g_{\text {var}}^{\pi ,h}\) and \(g_{\text {cvar}}^{\pi ,h}\) figure 3a) shows (\(g_{u}^{\pi ,h}\)) for \(1 \le h \le 2000\) and for risk factors \(\gamma \in \{1 , 0.1, 0.01, 0.001\}\). for all policies, it is noted that when \(\gamma \) tends to 0 for any value of h the value of the utility function \(g_{u_\gamma }^{\pi ,h}\) tends to the risk-neutral gain, \(g^{\pi }\). this property was demonstrated by howard and matheson [2] for \(h=\infty \). it is also observed that for all policies, the gain \(g_{u_1}^{\pi ,h}\) when h tends to infinity approaches \(g_{u_1}^{\pi ,h=\infty }\). additionally, if \(h=\infty \) (orange lines in fig. 3), \(\pi _3 \succ \pi _1 \sim \pi _2\) and if \(h \ne \infty \) (dotted lines in fig. 3) \(\pi _3 \succ \pi _2 \succ \pi _1\). fig. 3. gain of the policies varying h. full size image figure 3b) shows \(g_{\text {var}}^{\pi ,h}\) for \(1 \le h \le 2000\) and for confidence levels \(\alpha \in \{0.01, 0.1 , 0.2, 0.5,0.8,0.9,0.99\}\). for policies \(\pi _1\) and \(\pi _2\), for confidence levels above \(50\%\), \(g_{\text {var}_\alpha }^{\pi ,h} \ge g^\pi \) if \(h \ge 1\). when h tends to infinity \(g_{\text {var}_\alpha }^{\pi ,h}\) tends to \(g^\pi \). for confidence levels below \(50\%\), \(g_{\text {var}_\alpha }^{\pi ,h} \le g^\pi \) if \(h \ge 2\) and approaches \( g^\pi \) when h tends to infinity. for policy \(\pi _3\), \(g_{\text {var}_\alpha }^{\pi ,h}=g^\pi \) for all h. figure 3c) shows \(g_{\text {cvar}}^{\pi ,h}\) for \(1 \le h \le 2000\) and for confidence levels \(\alpha \in \{0.01, 0.1, 0.2, 0.5,0.8,0.9,0.99\}\). the values \(g_{\text {cvar}_\alpha }^{\pi , h}\) tend to \(g^\pi \) when h tends to infinity for any value of \(\alpha \) and, as expected, when \(\alpha \) tends to 1, \(g_{\text {cvar}_\alpha }^{\pi ,h}\) approaches \(g^\pi \) for any h. additionally, if \(h=\infty \), \(\pi _1 \sim \pi _2 \sim \pi _3\) and if \(h \ne \infty \) (dotted lines in fig. 3) \(\pi _3 \succeq \pi _2 \succeq \pi _1\). 5 conclusions in this work, we proposed a mathematical framework that unifies the average reward and risk criteria for mdps, for stationary policies that generate aperiodic unichain or multichain markov chains. the proposed formalization uses sequences of states of size h, allowing two ways of thinking about the stochastic process in the limit: (i) the accumulated reward at infinity and (ii) the steady-state distribution. the gain from the risk-neutral average reward criterion does not vary regardless of the number of states in the sequence. works on average reward with exponential utility function [2] and with the cvar risk measure [7] have been classified using the proposed framework. the exponential utility criterion [2] was classified as \(h=\infty \) since the sum of the reward at infinity was used, and the cvar criterion [7] was classified as \(h=1\) because it considered the immediate reward and the steady-state distribution. to the best of our knowledge, the exponential utility function applied to sequences of states of size \(h=1\) does not exist in the literature. this criterion allowed differentiation of policies not distinguished by exponential utility with \(h=\infty \) and by risk-neutral average reward, as demonstrated in the numerical examples. as far as we know, there is no work in the literature that characterizes cvar for \(h=\infty \). through numerical experiments, it was conjectured that this criterion is equivalent to risk-neutral, which needs to be formally demonstrated in future work. references ashok, p., chatterjee, k., daca, p., křetínský, j., meggendorfer, t.: value iteration for long-run average reward in markov decision processes. in: majumdar, r., kunčak, v. (eds.) computer aided verification, pp. 201–221. springer international publishing, cham (2017) chapter  google scholar  howard, r.a., matheson, j.e.: risk-sensitive markov decision processes. manage. sci. 18(7), 356–369 (1972). http://www.jstor.org/stable/2629352 perman, r.: natural resource and environmental economics. pearson education (2003) google scholar  puterman, m.l.: markov decision processes. wiley series in probability and mathematical statistics, john wiley and sons, new york (1994) book  math  google scholar  rockafellar, r.t., uryasev, s., et al.: optimization of conditional value-at-risk. j. risk 2, 21–42 (2000) article  math  google scholar  rockafellar, r., uryasev, s.: conditional value-at-risk for general loss distributions. j. bank. fin. 26(7), 1443–1471 (2002) article  math  google scholar  xia, l., zhang, l., glynn, p.w.: risk-sensitive markov decision processes with long-run cvar criterion. prod. oper. manag. 32(12), 4049–4067 (2023) article  math  google scholar  download references acknowledgments this study was supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior (capes) – finance code 001, by the são paulo research foundation (fapesp) grant \(\#\)2018/11236-9 and the center for artificial intelligence (c4ai-usp), with support by fapesp (grant #2019/07665-4) and by the ibm corporation. author information authors and affiliations university of são paulo, cidade universitária, butantã, são paulo, sp, 05508-220, brazil willy arthur silva reis, karina valdivia delgado & valdinei freire authors willy arthur silva reisview author publications search author on:pubmed google scholar karina valdivia delgadoview author publications search author on:pubmed google scholar valdinei freireview author publications search author on:pubmed google scholar corresponding author correspondence to willy arthur silva reis . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper reis, w.a.s., delgado, k.v., freire, v. (2025). a unified framework for average reward criterion and risk. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_7 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords markov decision process average reward risk-sensitive publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature tackling low-resource ecg classification with self-supervised learning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper tackling low-resource ecg classification with self-supervised learning conference paper first online: 31 january 2025 pp 238–252 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) tackling low-resource ecg classification with self-supervised learning download book pdf download book epub rafael da costa silva9 & diego furtado silva9  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 279 accesses abstract cardiovascular disease is one of the leading causes of worldwide death and an annual burden for european and american economies. the primary source of data to identify heart abnormalities is the electrocardiogram (ecg). however, its identification requires significant human efforts. in many cases, traditional rule-based diagnosis is inefficient due to the data heterogeneity, motivating the use of deep neural network approaches. however, ecg data varies significantly between equipment, including the sampling rate. self-supervised representation learning (ssl) has gained increasing attention due to its generalization capacity in scenarios with scarce availability of a large volume of labeled data. in this work, we investigate using ssl methods in a cross-dataset scenario. we raised different research questions supported by the assumption that high-resource data, with a large sampling rate and volume of records, is helpful for training models in an ssl manner to classify ecg in low-resource datasets. besides, we evaluate different resampling methods to match the sampling rate between datasets, including a novel upsampler based on espcn. our experimental evaluation shows that using the state-of-the-art ssl models ts-tcc and ts2vec can improve the accuracy in the low-resource dataset. however, the best results were obtained using the same dataset in both pre-training and downstream task steps. these results indicate the need to develop more suitable pretext tasks for cross-dataset scenarios, which may use our experimental analysis to build new ideas. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-source ecg signal-based cardiac-abnormality pattern classification with fast–slow learning dnns article 06 may 2025 detection of cardiovascular disease by ecg images using lstm chapter © 2025 bridging the gap in ecg classification: integrating self-supervised learning with human-in-the-loop amid medical equipment hardware constraints chapter © 2024 1 introduction cardiovascular disease (cvd) was one of the leading causes of worldwide death in 2016, being an annual burden for the european and american economies estimated on a scale of hundreds of billions of dollars [4]. on the other hand, identifying heart abnormalities and providing effective treatment requires significant human efforts and expensive medical procedures [17]. the primary source of data to identify these abnormalities is the electrocardiogram (ecg). the ecg is a non-stationary physiological signal characterized for being a low-cost, rapid, and simple exam that provides an overview of the physiological and structural condition of the heart by identifying pathological patterns among the heartbeats [14]. alongside monitoring the heart’s health, ecgs can provide valuable diagnostic clues for systemic conditions such as mental stress and drug toxic effects [12]. in many cases, the traditional rule-based diagnosis paradigm is inefficient due to the data heterogeneity, motivating the proposal of deep neural networks-based approaches for various related tasks [4]. generally, the availability of sufficiently large labeled data is one of the critical factors for reliable deep learning methods [19]. however, the labeling process is particularly costly in the health domain, and clinical ground truth is hard to define in many cases [10]. it encouraged some studies to explore the use of representation learning for ecg data [20]. self-supervised representation learning, or only self-supervised learning (ssl), has gained increasing attention due to its generalization capacity in scenarios where sufficiently large labeled data for training deep neural networks is scarce [19]. ssl methods are a two-step framework consisting of an unsupervised pre-training step followed by a supervised task, also called downstream task [12]. the pre-training step relies on an optimization task generated using pseudo-supervision signals from unlabeled data [22]. the pretext task defines how the model will construct the pseudo-supervision signals, serving as a means to learn informative representations for downstream tasks. we can generally organize the pretext tasks into three categories: context prediction, instance discrimination, and instance generation [19]. ecg is stored and analyzed as a time series. in this domain, context prediction and instance discrimination are the most commonly used category [3, 7, 21]. however, despite the variety of proposed pretext task approaches, most research on time series ssl evaluates the quality of the proposed pre-trained network using the same dataset in both self-supervised and downstream steps. this kind of evaluation helps demonstrate that the pretext task and its architecture may help classify a low-resource target dataset. however, it neglects cross-dataset scenarios where a high-resource dataset is used as source data to pre-train a model to increase the performance on the downstream tasks that only rely on low-resource data. the cross-dataset scenario is particularly interesting for healthcare applications like ecg classification. while a large volume of ecg data can be collected, it is prevalent for annotated datasets to be small. additionally, an ecg can vary significantly from one device to another, including its sampling rate, and even from one patient to another. therefore, understanding the quality of ssl methods in this context can be a significant step in creating suitable models for ecg analysis. this observation inspired us to investigate the question: can a high-resource dataset be used to tackle a low-resource ecg classification task with pre-trained ssl models with good performance? to answer this question we conducted a set of investigations that identify the keys aspects need to be addressed when inserted on a ecg cross-dataset scenariofootnote 1. these investigation can be summarized as follows: performance comparison between same dataset and cross-dataset scenario. we investigate the capacity of self-supervised models to transfer knowledge between datasets from the same domain using a high-resource dataset to build a robust feature extractor. for this, we compare the performance and quality of the embedding produced by ssl encoder pre-trained on high-resource with other pre-trained on the target dataset (low-resource). the impact of pretext task in the final performance. we analyze how much the pre-training step increases the model’s capacity to classify ecg under a low-resource scenario. to investigate this aspect, we compare the classifier’s performance with a pre-trained ssl encoder against that of a classifier with a random initialized ssl encoder. suitable approaches to deal with different sampling rates. we compare the results obtained by upsampling the target dataset to match the high-resource data with those achieved when we downsample the source data to match the sampling rate of the target dataset. using a cross dataset pre-trained upsampling model instead of a simple interpolator. we investigate if using an upsampler model trained in the source data outperforms using linear interpolation to resample the target dataset. we adapt the original upsampler model to lead the univariate time series and arbitrary-scale upsampling. 2 motivations deep learning (dl) has been successfully applied in detecting arrhythmia using ecgs [1]. however, alongside the need for extensive labeled data, dl techniques may face generalization difficulties, especially when the feature space distribution varies significantly, and ecg signal can vary considerably from person to person [1, 17]. figure 1 illustrates this fact, where we can observe a considerable difference between two samples of the same class but recorded from different subjects. the performance of a dl model trained in particular ecg datasets may not be reliable when applied to another dataset, requiring the model to have a high capacity to learn more general and meaningful representation by incorporating the ecg variability. fig. 1. difference between samples from same class in ecg fragment (c.f. sect. 4.1). full size image motivation 1: ssl aims to train powerful feature extractors. ssl techniques aim to train a powerful feature extractor using a discriminate learning pretext task that exploits the knowledge about the data modality without the need for manual annotation of the training examples [6]. conceptually, in this process, we have two datasets: one used during the training of the feature extractor, the source dataset, and our task dataset, the target dataset. however, in recent works, ssl methods for time series have been intensely evaluated on the same dataset scenario when the source and target dataset are the same, while the analysis on cross-dataset scenarios has been neglected [3, 5, 18, 21]. given the lack of information about the performance of ssl methods in cross-dataset scenarios, we raise the following research questions to be answered in the ecg classification context: rq1: does using a high-resource dataset increase the performance of the ssl method in a low-resource dataset? rq2: how much does the pretext task impact the performance of the downstream task? motivation 2: the improvement of the diversity of the ecg dataset. in recent years, we have had an improvement in ecg database diversity in terms of sampling rate, number of channels, duration of the signal, and propose [17]. when leading with a transfer learning scenario the similarity or dissimilarity of the datasets play an important role [22]. to mitigate this effects, we selected datasets from the same domain and resampled to match their sampling rate. aiming to incorporate the sampling rate differences in the self-supervised workflow, we raise the following research questions: rq3: what is the better resampling strategy when the source dataset has a higher sampling rate, upsampling the target dataset or downsampling the source dataset? rq4: does the utilization of a pre-trained upsampling model outperform the use of a linear interpolation? this paper tries to answer these questions through empirical analysis. 3 research methodology to answer the research questions, we conducted a comprehensive set of experiments to investigate the performance of two ssl methods on a low-resource and low sampling rate multi-class classification ecg task. the experiments were split into four different scenarios: (1) target interpolation, (2) espcn, (3) source interpolation, and (4) same dataset. figure 2 illustrates the architecture of each workflow. through scenarios 1 and 2, we aim to evaluate the difference between resampling the dataset using linear interpolation and the upsampling model. scenario 3 helps us to decide where to resample the dataset (at the pre-training or calibration phase). at last, scenario 4 is a baseline for cross-dataset scenarios. all scenarios end with a task model consisting of the downstream task’s decision layers. in our experiments, we vary the ssl encoder, pretext, and task models (c.f. sect. 4.3). fig. 2. the conducted experiments scenarios. (1) target interp. pre-train the ssl encoder in the source dataset and calibrate using the linear interpolated target dataset. (2) espcn pre-train the ssl encoder in the source dataset and calibrate using the target dataset upsampled using the espcn model. (3) source interp. pre-train the ssl encoder in the downsampled source dataset and calibrate with the target dataset at the original sampling rate. (4) same dataset pre-training and calibration phases are made using the target dataset at the original sampling rate. full size image 3.1 time series upsampling with espcn single image super-resolution (sisr) refers to a computer vision task of reconstructing a super-resolution image (sr) from its low-resolution (lr) counterpart [2]. dominant approaches focus on extracting features using a deep vision architecture and then upsample this feature space to high-resolution images at the network’s end [8]. swinir [9] combines shallow features extracted by the convolution layer with deep features extracted using transformer blocks, which can capture long-range dependencies within varying contents. considering the already substantial computational cost associated with the pretext task of the ssl models, we opted to utilize espcn [13], an sisr method that uses a shallow feature extractor [22]. espcn introduced an efficient way of extracting feature maps on lr image space and upscaling them into the sr image output through the sub-pixel convolution layer. initially proposed for image domain and produce outputs with integer scale factor, we adapted the espcn architecture to upsample univariate time series with arbitrary-scale factor, as shown in fig. 3. fig. 3. the architecture of the adapted espcn for univariate time series with arbitrary-scale. a feature extractor module captures the temporal characteristics of the input data and, after a sub-pixel convolution layer, aggregates these features into a univariate time series space. finally, the linear layer reduces the univariate time series to match the original arbitrary-scale factor. full size image our adapted version of the espcn maintains the feature extractor module with the same number of convolution blocks as the original proposal, changing the 2d-convolution for a 1d-convolution layer. the features encoded into embedding space are tensors of size \(b \times 1*\lceil s \rceil \times l\), where b is the batch size, s the arbitrary-scale factor, and l is the time series original length. the sub-pixel convolution layer is also the same as the original one, a periodic shuffling operator that rearranges the elements of a \(b \times 1*\lceil s \rceil \times l\) tensor to a tensor of shape \(b \times 1 \times l*\lceil s \rceil \). after, the arbitrary-scale factor property is obtained by the addition of a linear layer that reduces this univariate representation into the resampled univariate time series, \(b \times 1 \times l*s\). to validate these modifications, we compare the quality of the reconstructed signal. figure 4 presents a time series upsampled with the original espcn and our adapted version, called arbitrary-espcn. both models were trained using the same dataset, the ptb-xl-700 (c.f. sect. 4.1) downsampled to 250 hz with its original in 500 hz version being the labels, with a learning rate of \(1e-4\) and batch size of 64 samples. figure 4 shows that the arbitrary-espcn can correctly upsample a signal like the original espcn, so visually, both signals are almost identical with arbitrary-espcn and original espcn achieving an mse of 0.0038 and 0.0006, respectively. aiming to verify the generalization capacity of both models, we upsampled an ecg fragment’s (c.f. sect. 4.1) sample without the previous calibration step. to make the ecg fragment’s sample (that has a sampling rate of 360 hz) compatible with the already pre-trained models, we resampled it by setting the input to 250 hz and its respective label to 500 hz through linear interpolation. both models correctly reconstructed the time series, with arbitrary-espcn and original espcn obtaining an mse of 0.0008 and 0.0001, respectively, certifying the robustness of the selected upsampler model and that our adaptation doesn’t have caused a significant performance reduction. fig. 4. comparison between the original espcn (left) and the arbitrary-espcn (right). the presented time series was selected from the test part of the ptb-xl-700. full size image 3.2 ts2vec towards universal representation of time series ts2vec [18] is a well-established ssl method for learning contextual representation of time series at various semantic levels through hierarchical contrasting and contextual consistency. contextual consistency is obtained by applying timestamp masking and randomly cropping two context views from the input time series. the masking increases the robustness of the learned representation, while random cropping generates new contexts without collapsing the representation. a hierarchical contrastive loss extracts the multi-scale contextual information by computing temporal and instance-wise contrast for all granularity levels using a recursive max-pooling operation. as the feature extractor uses a dilated cnn module with ten residual blocks, enabling a large receptive field. 3.3 ts-tcc time-series representation learning via temporal and contextual contrasting ts-tcc [5] uses two augmented versions of the input signal – one strongly augmented and the other weakly augmented – to learn robust representation using a cross-view prediction task. a permutation-and-jitter strategy is applied for strong data augmentation, while weak augmentation uses a jitter-and-scale strategy. the augmented versions are passed to an encoder with a 3-block of convolutions to extract high-dimensional representation. the temporal contrasting module uses a transformer as the autoregressive model to summarize the latent representations into a context vector. the context vector is used in a cross-view prediction task using the strong augmentation context to predict the weak augmentation’s future timestep and vice versa. a contextual contrastive module learns to discriminate the contexts using a non-linear projection head; this module aims to learn more discriminative representations. 4 experimental evaluation to answer our research questions, we conducted a comprehensive experimental evaluation. it is important to note that we evaluated various ssl methods [7, 21] and chose to follow with ts-tcc and ts2vec. this selection was made for two main reasons: these are described in the literature as the state-of-the-art ssl for time series, and the other methods led to a significant overfitting in their pretext task for the datasets used. this section describes the experimental setup and the results achieved in our experiments. we initially present the datasets used in our analysis. 4.1 datasets we used two datasets to explore our research questions, one with a higher volume of data and sampling rate. this dataset, ptb-xl-700, is used to pre-train the model, which we refer to as source data. ecg fragment contains fewer examples sampled at a lower sampling rate than the source data. therefore, it is used as the target data to configure the downstream task. we previously split both datasets into train, validation, and test sets and normalized datasets’ instances using z-score, transforming the observations to have zero mean and unit variance. ecg fragment is a low-resource univariate dataset that contains 1016 ecg records of 2-second fragments with a sampling rate of 360 hz. each record presents one type of rhythm disturbance, labeled into six classes according to the potential risk to the patient’s life, with some classes composed of multiple disorders [11]. one of the most challenging characteristics of this dataset is the high within-class variation (fig. 1) at the same that some classes share similarities, as shown in fig. 5. figure 6 shows the dataset’s class distribution. fig. 5. ecg fragment dataset examples from each class. full size image fig. 6. class distribution of ecg fragment dataset. full size image ptb-xl-700 is a high-resource dataset constructed by randomly selecting 700 records from ptb-xl dataset [15], which contains 12-lead long-term ecg obtained at a sampling rate of 500 hz. we randomly segmented each record into five 2-second segments to increase the number of examples while standardizing the signal duration. besides, we used each lead as a univariate time series to provide data variability. we apply the z-normalization after the segmentation steps. these operations lead to a dataset containing 42,000 samples. figure 7 illustrates four out of twelve leads from the same record. fig. 7. 2-second segment of four out twelve leads from the same record in ptb-xl-700. full size image 4.2 experimental settings the hyperparameters of the investigated models was empirically selected withing the follow ranges: dimensions of the feature extractor representation = \(\{ 128 \}\); learning rate = \([1e^{-3}, 1e^{-6}]\); epochs = \(\{ 20, 40, 60, ..., 160 \}\). for the optimizer algorithm, ts-tcc and ts2vec specific parameters, we followed their original implementation. in all ssl models’ configurations, we applied three adjustment strategies: (i)random initialization: we do not pre-train the ssl encoder, so the target dataset is responsible for training the ssl encoder and task models; (ii) fine-tuning: we allow the training with the target dataset to adjust the task model’s and pre-trained ssl encoder’s parameters, as well as the espcn (when it applies); (iii) linear probing: once the ssl is pre-trained, we only allow the adjustment of the task model’s parameter. in the pipeline that applies the espcn (scenario 2), we pre-train the upsampler model using the ptb-xl-700 dataset with the following parameters: learning rate = \(1e^{-4}\), epochs = 25 and batch size of 64 samples. finally, we evaluate our results through the macro f1-score due to the class imbalance of the downstream task dataset (c.f. fig. 6). as previously stated, we varied the task models to evaluate the effects of the pre-trained model with different decision functions. specifically, we applied three architectures: (i) linear: a single linear/dense layer with softmax function; (ii) mlp: a two-layer densely connected with relu and a softmax as the activation functions in the penultimate and the output layers, respectively; (iii) oates’ mlp: a multi-layer perceptron model proposed for time series classification, with a larger number of parameters than mlp [16]. 4.3 experimental results in this section, we comprehensively assess the performance of all the variations of experiments described in the previous sections to handle the ecg classification task. the results are systematically summarized in table 1. our first observation is that ts-tcc outperformed ts2vec in three out of four scenarios. the only scenario in which ts2vec outperforms ts-tcc is when we apply espcn as the first step. one possible reason for that is the model complexity. once the ts2vec encoder comprehends ten blocks of dilated convolution, more data may be necessary to adjust the parameters compared to the three convolution blocks of the ts-tcc encoder. one evidence for this claim is that linear probing or fine-tuning obtained all the best results for ts2vec, while ts-tcc obtained some of the best results from random initialization. table 1. the performance achieved by ts-tcc and ts2vec. the values presented are the best macro-f1 score obtained by the specified configuration. the symbols indicate the calibration strategy, \(\{\diamond :\) random init., \(\star :\) fine-tuning, \(\dagger :\) linear-probing}. the best results of each column are bold, and suboptimal results are underlined.full size table to improve the robustness of our analysis, we also provide a baseline scenario to compare the ssl with supervised learning using well-known and widely applied architectures [16]: oates’ mlp, fcn, and resnet-64. in addition, we evaluate the linear and mlp task models under the supervised scenario. table 2 summarizes the results. the scarcity of data in the target dataset hampers the fcn and resnet-64 model’s capacity to identify discriminative features, while the linear model lacks representation capacity. the different mlp configurations obtained the best scores. given these results, we will discuss each research question raised in sect. 1. table 2. the performance of the task models without the ssl encoder model. the values presented are the maximum macro-f1 score obtained by the specified model. the best result of each line is bold, and suboptimal results are underlined.full size table performance comparison between same dataset and cross-dataset approaches (rq1). to compare these scenarios, we excluded scenario 4 to avoid the influence of espcn in the final results. therefore, the evaluation compares the scenarios cross-dataset that use linear interpolation (scenarios 1 and 3) against the one that pre-trains and calibrates the classification model using the same dataset (scenario 4). valuable observations can be summarized as follows. firstly, the use of a high-resource dataset doesn’t increase the maximum performance of the selected ssl models’ performance. the maximum f1 score achieved under the considered cross-dataset scenarios was 0.5248 for ts2vec. meanwhile, the best f1 score achieves 0.5666 when the ssl model uses the same dataset in the training and evaluation phases. besides ts-tcc’s best performance being a cross-dataset case, there is no significant difference in the best result in the same dataset scenarios. the difference between these cases is 0.0011. to better understand this outcome, we visualized the projection of the representation learned during the pre-training. figure 8 shows the t-sne projections of the embeddings in all scenarios, including the one with espcn. in all scenarios (even in scenario 3), the projection of the produced embedding is similar regarding the class overlaps and well-separated groups of instances. it indicates that using a high-resource dataset does not increase the capacity of the encoders to extract meaningful features from the low-resource target dataset. fig. 8. t-sne projection of the embeds produced by the pre-trained ssl encoder aggregated by global pooling average. full size image the impact of pretext task in the performance (rq2). the central comparison to answer this research question relies on the results obtained by random initialization and linear-probing and fine-tuning calibration strategies. we first analyze the results obtained by ts-tcc, given that it achieved the best performance. however, these best results are only partially related to its pretext task. most ts-tcc models achieved their macro-f1 score under the random initialization strategy without relying on the pre-training phase. remarkably, the best results were obtained by random initialization, and the same dataset scenario achieved the second-best score. it indicates that the high-resource data does not improve the performance of ts-tcc in any scenario. in contrast, random initialization did not achieve the best f1 score in any scenario based on the ts2vec. as this model is more complex, it is probably more benefited by the high-resource data. however, as it did not achieve results as good as those obtained by ts-tcc, the source data may be not enough to train a suitable model using ts2vec. it points to the intuition that a higher volume of data to train a larger ssl model may be a promising research path. on the other hand, the difference between the datasets (c.f. figs. 5 and 7) may affect the results. in contrast, the architectures proposed to the ssl models seem relevant for learning better representations, given that the results achieved by applying both of them are considerably better than those of the supervised scenario. it indicates that these architectures are extracting more relevant features for this application than the mlps, fcn, and resnet applied directly to the signal. upsampling the target dataset or downsampling the source dataset? (rq3). we analyze this research question based on the results of scenarios 1 and 3. for ts-tcc, the difference in performance is almost nonexistent, with an average difference of \(\approx 0.0073\), while downsampling the source dataset reduced the performance of the ts2vec models by \(\approx 0.0351\). as the differences are minor, and the same dataset scenario performs better, we did not draw significant conclusions to this question. a pre-trained upsampling model outperform the use of a linear interpolation? (rq4). through the analysis of scenarios 1 and 2, we can observe that ts-tcc suffered a significant performance reduction when using espcn. conversely, espcn configured most of the best workflow scenarios for the ecg fragment classification task using ts2vec as ssl model. under the supervised strategy, fcn and resnet-64 results considerably improved using upsampler model. considering these observations, using espcn as an upsampler with larger models seems promising, mainly if associated with a pretext task for cross-dataset scenarios. in other words, if the research on ssl for ecg data follows the use of large models, we recommend using our adapted espcn to resample data to an arbitrary sampling rate. 5 discussion through the investigation of our research questions, we obtained diverse insights about the quality of ssl methods in a cross-dataset context that can be helpful to future research. the experiments indicate that ts-tcc and ts2vec models are ineffective in transferring knowledge from unlabeled datasets with higher resources to enhance the model’s capacity to solve the downstream task (rq1). despite the low transfer learning capacity, the ssl models showed architectures that could extract more relevant features than the baseline methods (rq2). also, it seems promising to adopt espcn when associated with larger models (rq4). however, due to the exploratory nature of the present investigation, including more diverse pretexts (e.g., context prediction and instance generation) and downstream tasks will allow us to obtain more significant and general conclusions, especially for the rq3. even though we focused on the ecg domain, we expect these insights to contribute to advancing ssl models in time series domains with labeled data scarcity, like mobile health applications. 6 conclusions in this paper, we investigate using a high-resource dataset to increase the performance of ssl models in a low-resource ecg classification dataset. specifically, we compare the performance of two ssl models (ts-tcc and ts2vec) under four different scenarios, with three being cross-dataset, and present an adaptation of the espcn model for univariate time series and arbitrary-scale factor. the experiments demonstrated that using a high-resource dataset does not increase the performance of the ssl model, especially for ts-tcc, where the random initialization scenario has obtained the best overall result. also, the espcn upsampler increases the performance of the ts2vec, fcn, and resnet-64. suggesting that the use of espcn with larger models is promising. these results indicate the need to develop pretext tasks more suitable for cross-dataset scenarios where knowledge transfer is fundamental. notes 1.source code https://github.com/rafaelsilva7/tackling-low-ecg. references chen, a., et al.: multi-information fusion neural networks for arrhythmia automatic detection. comput. methods programs biomed. 193, 105479 (2020) article  math  google scholar  chen, h., et al.: real-world single image super-resolution: a brief review. inf. fusion 79, 124–145 (2022) article  math  google scholar  darban, z.z., webb, g.i., pan, s., salehi, m.: carla: a self-supervised contrastive representation learning approach for time series anomaly detection. arxiv preprint arxiv:2308.09296 (2023) ebrahimi, z., loni, m., daneshtalab, m., gharehbaghi, a.: a review on deep learning methods for ecg arrhythmia classification. expert syst. appl. x 7, 100033 (2020) math  google scholar  eldele, e., et al.: time-series representation learning via temporal and contextual contrasting. arxiv preprint arxiv:2106.14112 (2021) ericsson, l., gouk, h., loy, c.c., hospedales, t.m.: self-supervised representation learning: introduction, advances, and challenges. ieee signal process. mag. 39(3), 42–62 (2022) article  math  google scholar  foumani, n.m., tan, c.w., webb, g.i., salehi, m.: series2vec: similarity-based self-supervised representation learning for time series classification. arxiv preprint arxiv:2312.03998 (2023) lee, j., jin, k.h.: local texture estimator for implicit representation function. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 1929–1938 (2022) google scholar  liang, j., cao, j., sun, g., zhang, k., van gool, l., timofte, r.: swinir: image restoration using swin transformer. in: proceedings of the ieee/cvf international conference on computer vision, pp. 1833–1844 (2021) google scholar  mehari, t., strodthoff, n.: self-supervised representation learning from 12-lead ecg data. comput. biol. med. 141, 105114 (2022) article  math  google scholar  nemirko, a., manilo, l., tatarinova, a., alekseev, b., evdakova, e.: ecg fragment database for the exploration of dangerous arrhythmia (2022) google scholar  rabbani, s., khan, n.: contrastive self-supervised learning for stress detection from ecg data. bioengineering 9(8), 374 (2022) article  math  google scholar  shi, w., et al.: real-time single image and video super-resolution using an efficient sub-pixel convolutional neural network. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1874–1883 (2016) google scholar  siontis, k.c., noseworthy, p.a., attia, z.i., friedman, p.a.: artificial intelligence-enhanced electrocardiography in cardiovascular disease management. nat. rev. cardiol. 18(7), 465–478 (2021) article  google scholar  wagner, p., et al.: ptb-xl, a large publicly available electrocardiography dataset. sci. data 7(1), 1–15 (2020) article  math  google scholar  wang, z., yan, w., oates, t.: time series classification from scratch with deep neural networks: a strong baseline. in: 2017 international joint conference on neural networks (ijcnn), pp. 1578–1585. ieee (2017) google scholar  xiao, q., et al.: deep learning-based ecg arrhythmia classification: a systematic review. appl. sci. 13(8), 4964 (2023) google scholar  yue, z., et al.: ts2vec: towards universal representation of time series. in: proceedings of the aaai conference on artificial intelligence, vol. 36, pp. 8980–8987 (2022) google scholar  zhang, k., et al.: self-supervised learning for time series analysis: taxonomy, progress, and prospects. ieee trans. pattern anal. mach. intell. (2024) google scholar  zhang, w., geng, s., hong, s.: a simple self-supervised ecg representation learning method via manipulated temporal-spatial reverse detection. biomed. signal process. control 79, 104194 (2023) article  math  google scholar  zhang, w., yang, l., geng, s., hong, s.: self-supervised time series representation learning via cross reconstruction transformer. arxiv preprint arxiv:2205.09928 (2022) zhao, z., alzubaidi, l., zhang, j., duan, y., gu, y.: a comparison review of transfer learning and self-supervised learning: definitions, applications, advantages and limitations. expert syst. appl. 122807 (2023) google scholar  download references acknowledgments this work was supported by coordenação de aperfeiçoamento de pessoal de nível superior (capes) grant 88887.925347/2023-00 and the são paulo research foundation (fapesp) grant numbers 2024/07016-4 and 2022/03176-1. author information authors and affiliations institute of mathematical and computer sciences, university of são paulo, são carlos, são paulo, brazil rafael da costa silva & diego furtado silva authors rafael da costa silvaview author publications search author on:pubmed google scholar diego furtado silvaview author publications search author on:pubmed google scholar corresponding author correspondence to rafael da costa silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da costa silva, r., silva, d.f. (2025). tackling low-resource ecg classification with self-supervised learning. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_17 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords time series self-supervised learning ecg classification transfer learning publish with us policies and ethics profiles rafael da costa silva view author profile diego furtado silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature detecting early signs of insufficiency in covid-19 patients from cbc tests through a supervised learning approach | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper detecting early signs of insufficiency in covid-19 patients from cbc tests through a supervised learning approach conference paper first online: 28 november 2021 pp 42–57 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) detecting early signs of insufficiency in covid-19 patients from cbc tests through a supervised learning approach download book pdf download book epub tiago colliri  orcid: orcid.org/0000-0001-9233-466210, marcia minakawa  orcid: orcid.org/0000-0003-1565-497411 & liang zhao  orcid: orcid.org/0000-0002-1502-660412  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1214 accesses abstract one important task in the covid-19 clinical protocol involves the constant monitoring of patients to detect possible signs of insufficiency, which may eventually rapidly progress to hepatic, renal or respiratory failures. hence, a prompt and correct clinical decision not only is critical for patients prognosis, but also can help when making collective decisions regarding hospital resource management. in this work, we present a network-based high-level classification technique to help healthcare professionals on this activity, by detecting early signs of insufficiency based on complete blood count (cbc) test results. we start by building a training dataset, comprising both cbc and specific tests from a total of 2,982 covid-19 patients, provided by a brazilian hospital, to identify which cbc results are more effective to be used as biomarkers for detecting early signs of insufficiency. basically, the trained classifier measures the compliance of the test instance to the pattern formation of the network constructed from the training data. to facilitate the application of the technique on larger datasets, a network reduction option is also introduced and tested. numerical results show encouraging performance of our approach when compared to traditional techniques, both on benchmark datasets and on the built covid-19 dataset, thus indicating that the proposed technique has potential to help medical workers in the severity assessment of patients. especially those who work in regions with scarce material resources. this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp) under grant number: 2019/07665-4 and by the ibm corporation. this work is also supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, fapesp under grant numbers 2015/50122-0, the brazilian national council for scientific and technological development (cnpq) under grant number 303199/2019-9, and the ministry of science and technology of china under grant number: g20200226015. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others online covid-19 diagnosis prediction using complete blood count: an innovative tool for public health article open access 19 december 2023 external validation of machine learning models for covid-19 detection based on complete blood count article open access 23 october 2021 predicting the outcome for covid-19 patients by applying time series classification to electronic health records article open access 17 july 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. covid19 data mining diagnosis laboratory medical diagnosis machine learning statistical learning 1 introduction since the covid-19 outbreak, advanced machine learning techniques have been applied in the development of clinical decision support tools, to help medical workers on guiding the monitoring and treatment of affected patients. examples of these studies include the automation and introduction of new methods for routine medical activities, such as: the analysis of computed tomography (ct) images [21], covid-19 diagnosis and screening [27], detection of new viral species and subspecies [15], medication [2], and patients monitoring through biomarkers, such as the ones provided by blood samples [26]. when a new covid-19 case is confirmed, the clinical protocol involves an initial assessment, made by healthcare professionals, to classify its severity as mild, moderate or grave, followed by a monitoring activity to evaluate the progression of the disease [20]. this procedure, although it may slightly vary, depending on the region, usually includes specific tests, such as creatinine, lactic dehydrogenase and blood urea nitrogen, as well as a complete blood count (cbc), which is a series of tests used to evaluate the composition and concentration of cellular components in the blood. such type of testing, as the cbc, is widely used because it can be easily collected at any place, even in situations of medical resource scarcity. moreover, it is usually processed in a matter of minutes, up to two hours. therefore, it is considered as a fast, low-cost and reliable resource to assess the patient’s overall conditions. during the monitoring process, healthcare professionals then need to decide whether to order complementary tests for detecting possible signs of hepatic, renal or respiratory insufficiency. considering that progressive respiratory failure is the primary cause of death from covid-19, this monitoring activity hence is critical for the patient. however, we must face the problem that such complementary tests incur high costs, and may not be available in some specific places due to scarcity of material and human resources. in this study, we present a technique that may assist medical professionals on this monitoring activity, by detecting early signs of insufficiency in covid-19 patients solely based on cbc test results. specifically, our approach can help to identify the level of risk and the type of insufficiency for each patient. we start by building a training dataset comprising results from both cbc and specific tests used to detect signs of hepatic, renal and respiratory insufficiency, from a total of 2,982 covid-19 patients, who received treatment in the israelite albert einstein hospital, from sao paulo, brazil [10]. in this process, we identify which cbc tests are more effective to be used as biomarkers to detect signs from each type of insufficiency. the dataset resulting from this analysis is then delivered to a modified network-based high-level classification technique, also introduced in this study. the obtained results present competitive performance of the proposed technique compared to classic and state-of-the-art ones, both on benchmark classification datasets and on the built covid-19 dataset. real-world datasets usually contain complex and organizational patterns beyond the physical features (similarity, distance, distribution, etc.). data classification, which takes into account not only physical features, but also the organizational structure of the data, is referred to as high level classification. complex networks are suitable tools for data pattern characterization due to their ability of capturing the spatial, topological, and functional relationship among data [1, 6, 22, 23]. in this study, we present a modified high-level classification technique based on complex network modeling. it can perform classification tasks taking into account both physical attributes and the pattern formation of the data. basically, the classification process measures the compliance of the test instance to the pattern formation of each network constructed from a class of training data. the approach presented in this work is inspired by the high-level data classification technique introduced in previous works [6, 8, 22], and here it is modified in terms of the network building parameters automation method, the topological measure used for characterizing the data pattern structure, and the inclusion of a network reduction option, to facilitate its application on larger datasets by saving processing and memory resources. the main contributions of this work are summarized as follows: building a training dataset from unlabeled original data, comprising medical tests from almost 3,000 de-identified covid-19 patients, provided by a hospital in brazil, which will be made publicly available to be used in other relevant researches, introducing a network-based high level classification technique, based on the betweenness centrality measure, with a network reduction option included in the training phase, to make the technique faster and to facilitate its application on larger datasets, and applying the proposed technique both on benchmark classification datasets and on the built dataset, and comparing the results with those achieved by other traditional classification techniques, on the same data, to evaluate the possibility of detecting early signs of insufficiency in covid-19 patients solely based on cbc tests, used as biomarkers. 2 the proposed mnbhl technique in this section we describe the proposed modified network-based high level (mnbhl) classification technique. all implementations are made using the igraph python package [9]. in classic supervised learning, initially we have an input dataset comprising n instances in the form of an array \(x_{train} = [x_1, x_2, . . . , x_n]\), where each instance \(x_i\) itself is a m-dimensional array, such that \(x_i = [x_{i, 1}, x_{i, 2}, . . . , x_{i, m}]\), representing m features of the instance. correspondingly, the labels of the instances are represented by another array \(y = [y_1, y_2, . . . , y_n]\), where \(y_{i} \in \mathcal {l} = \{l_{1} . . . , l_{c}\}\), and \(l_i\) is the label of the ith class. the objective of the training phase is to construct a classifier by generating a mapping \(f: x_{train} \xrightarrow {\vardelta } y\). in the testing phase, this mapping is used to classify data instances without label. the test dataset is denoted by \(x_{test}\). a network can be defined as a graph \(\mathcal {g}=(\mathcal {v, e})\), where \(\mathcal {v}\) is a set of nodes, \(\mathcal {v} = \{ v_1, v_2, ..., v_n \}\), and \(\mathcal {e}\) is a set of tuples, \(\mathcal {e} = \{ (i,j) : i,j \in \mathcal {v} \}\), representing the edges between each pair of nodes. in the proposed modified high-level classification technique, each node in the network represents a data instance \(x_i\) in the dataset, and the connections between nodes are created based on the their similarity in the attribute space, pairwise. in the training phase, one network component is constructed for the data instances of each class. afterwards, there is an option to reduce the built network, by preserving only its \(r\%\) most central nodes, according to a selected centrality index. in the testing phase, each data instance is inserted as a new node in the network, one at a time, by using the same rules from the training phase. in case only one network component is affected by the insertion, then its label will be equal to the respective class of that component. in case more than one component is affected by the insertion, then its label will be given by the class whose component is the least impacted by the new node’s insertion, in terms of the network topological structure. 2.1 description of the training phase the training phase starts by balancing the values in \(x_{train}\), for dealing with unbalanced datasets. for this end, we introduce a list \(\alpha \), where each element is given by: $$\begin{aligned} \alpha ^{(l)} = \frac{|x_{train}^{(l)} |}{|x_{train} |} \quad ,\forall l \in \mathcal {l} \quad , \end{aligned}$$ (1) where \(|x_{train} |\) is the total number of elements of the training set and \(|x_{train}^{(l)} |\) is the number of elements in the subset \(x_{train}\) whose corresponding class label in y is l. these values are also normalized, assuming \(\alpha _i = \frac{\alpha _i}{\sum _i^n \alpha _i}\). the edges between nodes in the network are generated by measuring the pairwise similarity of the corresponding data instances in the attribute space through a combination of two rules: knn and \(\epsilon \)-radius. the \(\epsilon \)-radius rule is used for dense regions, while the knn is employed for sparse regions. the value of \(\epsilon \) is yielded by: $$\begin{aligned} \epsilon =q(d, p) \quad , \end{aligned}$$ (2) where \(d_i\) is a 1d vector containing the euclidean distances between each element and element \(x_i\) in \(x_{train}\), and q is a function which returns the p-th quantile of the data in d. for the second rule used for generating the edges in the network, the knn, we opt to set the initial value for the parameter k as the \(\min (\min v, p \overline{v})\), where v is a vector containing the number of data instances per each class in the dataset. hence, the maximum possible value for k will be the average number of data instances per class in the dataset multiplied by the same parameter p used for returning the quantile from the \(\epsilon \)-radius technique. by proceeding this way, we are therefore linking the value of k to the value of \(\epsilon \) and, consequently, to the characteristics of each dataset (here represented by \(\overline{v}\)). this is also a novelty introduced in this technique, compared to its previous versions [8, 22], with the aim of avoiding exaggerated disproportions between the number of edges yielded by the two rules. with the initial values of \(\epsilon \) and k being set, the model then proceeds to generate the edges of the network \(\mathcal {g}\). the neighbors to be connected to a training node \(x_{i}\) are yielded by: $$\begin{aligned} n(x_i) = {\left\{ \begin{array}{ll} \epsilon \text {-radius}(x_i, y_i), \text { if } \epsilon \text {-radius}(x_i, y_i) > k \\ k\text {nn}(x_i, y_i), \text { otherwise} \quad , \end{array}\right. } \end{aligned}$$ (3) where \(y_i\) denotes the class label of the training instance \(x_{i}\), \(\epsilon \text {-radius}(x_i, y_i)\) returns the set \(\{x_j, \forall j \in \mathcal {v} \mid dist(x_i, x_j) \le \epsilon \wedge y_i=y_j\}\), and knn\((x_i, y_i)\) returns the set containing the k nearest vertices of the same class of \(x_i\). a common issue regarding the application of network-based techniques on larger datasets is that, depending on the available computational resources, the process may become too expensive, both in terms of processing power and memory consumption. for this reason, in this study, we introduce an option to reduce the number of vertices in the network, for the sake of saving processing time and memory resources, both in the training and testing phases. there are different possible strategies for reducing a network [24] and, in this work, we opt for one which consists in keeping only the nodes that occupy more central positions in it, measured in terms of the betweenness centrality [3]. this measure, broadly speaking, estimates the extent to which a vertex lies on paths between other vertices, and is considered essential in the analysis of complex networks. in a network that does not present communities, the nodes with higher betweenness centrality are among the most influential ones [7], while in networks that present communities, these nodes are the ones which usually work as links between the communities. we chose to use this measure for believing that it is able to identify the most representative nodes in the network, such that only these selected nodes can be used in the classification task. for controlling this option, we add a reduction parameter \(r \in [0, 1]\) in the model, such that when its value is set as less than 1, only the ratio of r vertices with higher betweenness centrality are kept in the network. in this case, the values in \(x_{train}\) and y are also reduced, accordingly. the value of \(\epsilon \) is updated, and the network edges are generated again, by using the same rules provided in eq. 2 and eq. 3. if, after this procedure, the number of components in the network is higher than the number of classes in the dataset, then the value of k is increased by 1, and the edges are updated again. this step can be repeated more times, if necessary, until we have, at the end, one component in the network per class in the dataset. the complete process of the training phase is outlined in algorithm 1. in line 2, there is the balancing task, in which the parameter \(\alpha \) is generated. then, in line 3, the network \(\mathcal {g}\) is built, by using the values of parameters \(\epsilon \) and k. in lines 4–7, there is the optional procedure to reduce the network, by leaving only its \(r\%\) most central nodes. in lines 8–10, we have the validation of whether the number of network components is equal to the number of classes in the dataset, with the parameter k and the network \(\mathcal {g}\) being updated, if necessary. 2.2 description of the testing phase in the testing phase, the data instances from \(x_{test}\) are inserted as new vertices in the network \(\mathcal {g}\), one at a time, and the model then needs to infer the label of each testing instance. the model extracts two groups of network measures for each component, before and after the insertion of a testing instance, respectively. the testing instance is classified to the class of the component which its insertion caused the smallest variation in the network measures. in other words, a testing instance is classified to a certain class because it conforms the pattern formed by the training data of that class. in this way, the data instances from \(x_{test}\) are inserted in the network, to be classified, one by one. the new node’s insertion is made by using the same rules described in eq. 3 for generating the edges between the new node and the other nodes in the network. the only difference now is that, since we do not know the class of the new data instance \(x_i\), the “same label” restriction is removed from the procedure, so that the nodes to be connected to \(x_i\) are yielded by: $$\begin{aligned} n(x_i) = {\left\{ \begin{array}{ll} \epsilon \text {-radius}(x_i), \text { if } \epsilon \text {-radius}(x_i) > k \\ k\text {nn}(x_i), \text { otherwise} \quad , \end{array}\right. } \end{aligned}$$ (4) where \(\epsilon \text {-radius}(x_i)\) returns the set \(\{x_j, \forall j \in \mathcal {v} \mid dist(x_i, x_j) \le \epsilon \}\), and knn\((x_i)\) returns the set containing the k nearest vertices of \(x_i\). the model will assign the label for the new node according to the number of components affected by the node’s insertion in the network. in case only one component is affected by its insertion, i.e., when all the target nodes of its edges belong to the same component, then the model will assign the class of this component for the new node. on the other hand, when more than one component is affected by the insertion, then the model will extract new measures of the affected components and assign a class label to the testing data instance as the label of the component less impacted by the insertion, in terms of the network topology. in this work, we use betweenness centrality for measuring these impacts. the overall impact \(i^{(l)}\) on each affected component of class l, caused by the insertion of the testing instance, is calculated and balanced according to: $$\begin{aligned} i^{(l)} = \alpha ^{(l)} \frac{m_1^{(l)} m_0^{(l)}}{m_0^{(l)}} \quad , \end{aligned}$$ (5) where \(m_0^{(l)}\) and \(m_1^{(l)}\) are the extracted network measures for the component representing class l, before and after the insertion of the testing instance, respectively. the probability \(p(x_i^{\hat{y_i}=l})\) of the testing instance \(x_i\) to belong to class l is given by the reciprocal of the value in vector i, in the normalized form, as in: $$\begin{aligned} p(x_i^{\hat{y_i}=l}) = {\left\{ \begin{array}{ll} \frac{1/i^{(l)}}{\sum _l 1/i^{(l)}} \text { , if } c^{(l)} \text { is affected} \\ 0, \text { otherwise} , \end{array}\right. } \end{aligned}$$ (6) where \(c^{(l)}\) is the network component representing class l. finally, the label \(\hat{y_i}\) to be assigned for the testing instance \(x_i\) is yielded by: $$\begin{aligned} \hat{y_i} = \underset{l}{\mathrm {argmax}}\, p(x_i^{\hat{y_i}=l}) \quad . \end{aligned}$$ (7) the complete process of the testing phase is described in algorithm 2. in lines 2–3, an initial empty list is created to store the predicted labels for each data instance in \(x_{test}\), and the current measures of each network component are extracted. in lines 4–13, the model generates a label for each testing instance, one by one. in case only one component is affected by the insertion of the testing data, then its label will be this component’s class. in case more than one component is affected by the insertion, then the testing data is classified to that class whose component is least impacted by the insertion, in terms of topological structure. in line 14, the classification list is updated with each predicted label. 3 materials and methods we first evaluate the performance of the proposed mnbhl model by applying it to well-known benchmark classification datasets, both artificial and real ones. afterwards, we apply the mnbhl model to a dataset specially built for this work, obtained from the analysis of publicly available data of de-identified covid-19 patients from one of the main hospitals in brazil. each dataset is processed 20 times by all models, each time having their data items shuffled by using a different random seed. the final accuracy scores are the averaged ones achieved by each model, on each dataset. for splitting each dataset into 2 subdatasets, for training and testing purposes, we make use of a function that returns a train-test split with 75% and 25% the size of the inputs, respectively. table 1. meta information of the datasets used in the experimental resultsfull size table a succinct meta-information of the benchmark datasets used for obtaining the experimental results is given in table 1. circles_00 and circles_02 datasets are two concentric circles without noise and with a 20% noise, respectively. moons_00 and moons_02 datasets are two moons without noise and with a 20% noise, respectively. for a detailed description of the real datasets, one can refer to [14]. table 2. tests considered for identifying signs of insufficiency in patients and respective cbc tests considered for detecting those signsfull size table we built the covid-19 dataset from originally unlabeled data [10] collected from covid-19 patients of the israelite albert einstein hospital, located in sao paulo, brazil. the original database comprises a total of 1,853,695 results from 127 different medical tests, collected from 43,562 de-identified patients, who received treatment in the hospital from january 1, 2020 until june 24, 2020. firstly, we identify the patients who tested positive in at least one of the following covid-19 detection tests: polymerase chain reaction (pcr), immunoglobulin m (igm), immunoglobulin g (igg) and enzyme-linked immunosorbent assay (elisa). next, we filter the patients and left in the dataset only the ones who have made at least one complete blood count (cbc) test, in a date no earlier than the date to be tested positive for covid-19. in case a patient has made more than one cbc test, we then consider only the results of the first test for predicting early signs of insufficiency. we adopt this procedure because we aim to make predictions based on early signs of insufficiency, hence the use of only the first cbc test for the predictions. afterwards, we run an algorithm to automatically label each patient of the dataset as presenting or not signs from each type of insufficiency, strictly according to results from specific tests, listed in the second column of table 2, and using their respective reference values provided in the same original database, which are standardized in the medical literature. after the data cleansing, we end up with a total of 2,982 different patients in the dataset, with each of them belonging to one of the following 4 classes: healthy, hepatic insufficiency signs, renal insufficiency signs, and respiratory insufficiency signs. for determining the predictive attributes (or biomarkers) for each type of insufficiency, we make use of the pearson correlation coefficient [16], to select the cbc tests that are most correlated to the labels, i.e., to each type of insufficiency. at the end, the following biomarkers were used for this end: age, neutrophils, basophils, lymphocytes, eosinophils and red cell distribution width (rdw). in table 2, we provided an overview of how the covid-19 dataset has been built, showing all tests considered, both for the labels and predictive attributes generation. the details regarding all patients in the dataset are summarized in table 3. table 3. overview of the patients in the built covid-19 datasetfull size table the models performances on the covid-19 dataset are evaluated by the following indicators: accuracy (ratio of true predictions over all predictions), precision, sensitivity/recall and f1 score, as defined below: $$\begin{aligned} \text {precision} = \frac{\text {tp}}{\text {tp}+\text {fp}}, \, \text {recall} = \frac{\text {tp}}{\text {tp}+\text {fn}}, \, \text {f1} = \frac{2 \times \text {precision} \times \text {recall}}{\text {precision} + \text {recall}} \end{aligned}$$ (8) where tp, tn, fp and fn stand for true positive, true negative, false positive and false negative rates, respectively. for the application, we set the parameter p from the proposed technique, responsible for the network edges generation, as \(p=0.01\) when processing the benchmark datasets, and \(p=0.02\) when processing the covid-19 dataset. regarding the network reduction option, we set \(r=0.1\) (10%) when processing the digits dataset and the covid-19 dataset and, for the others, we do not make use of this resource, since they are smaller ones. for the sake of comparison, the following traditional classification models are applied on all datasets: decision tree [19], logistic regression [12], multilayer perceptron (mlp) [13], support vector machines (svm) with an rbf kernel [25] and naive bayes [18]. we also apply the following ensemble methods: bagging of mlp [4], random forest [5] and adaboost [11]. all models were implemented through scikit-learn [17], by using their respective default parameter values, in all tests performed. 4 tests performed on benchmark datasets in table 4, we present the obtained results, in terms of average accuracy, from the application of the proposed mnbhl model to the benchmark datasets, as well as the comparison with the performances achieved by traditional models. these results indicate that the model’s overall performance is competitive, being ranked as second one when compared to traditional techniques, in the average rank. we highlight the good performance achieved on the moons_00 and moons_02 datasets, in which it was the only model that obtained an 100% accuracy in all 20 classification tasks processed. this result, in particular, is a strong sign that the model is able to correctly capture the topological pattern present in these two datasets, as expected in high level techniques. observing the running times of all considered models, we can say that the mnbhl is quite competitive on that aspect as well. table 4. experimental results: average accuracy rates (%) obtained by the following models, in that order: mnbhl, adaboost, bagging of mlp, decision tree, logistic regression, mlp, naive-bayes, random forest and svm. the values between parenthesis indicate the rank achieved and the average running time (in seconds), respectively, including the training and testing phases, on each dataset.full size table fig. 1. effect of the network reduction option on the average accuracy and processing time, on two benchmark classification datasets. each dataset is processed 10 times for obtaining these values, with a different random seed. full size image in fig. 1, we show the effect of the network reduction option on the mnbhl model performance, in terms of average accuracy and processing time, on two benchmark datasets. each dataset is processed 10 times for obtaining these values, each time using a different random seed. as one can note, from this figure, when the network built for the digits dataset is reduced in 90%, i.e., to only 10% of the training data, the average accuracy rate decreases from around 0.955 to around 0.905, thus with a negative variation of about 5%, while the average processing time decreases from 55 s to only 2 s, thus with a negative variation of about 96%. as for the wine dataset, there is an initial improvement in the accuracy, when using a 30% reduction option, which is somewhat surprising, with a later decrease, as the reduction increases. overall, on the two datasets considered, a 90% network reduction option has a negative impact of around 4% on the average accuracy rate and of around 80% on the average running time. therefore, this resource can be used in situations when one needs to process larger datasets using a network-based approach and needs to adapt the problem to locally available computational resources, both in terms of memory and processing capacity, as it is the case in this study. 5 experimental results we start this section by showing an example of different processing stages from the proposed technique, when applied to detect respiratory insufficiency signs. in fig. 2(a), there is the initial network built by the model, i.e., the network formed by all training data instances. this network has 2.236 nodes in total (representing the patients), and two classes (with and without insufficiency signs), denoted by the orange and blue colors. in fig. 2(b), we have the reduced version of this network, with only the 10% of nodes (223 in total) with the highest betweenness values. in this example, the new node’s insertion during the classification phase, in fig. 2(c), affected both network components, and hence its label will be yielded by the class whose component is least impacted, in terms of betweenness centrality. fig. 2. plots showing three different processing stages from the proposed mnbhl technique, when applied to detect respiratory insufficiency signs. full size image in table 5, we display the results obtained by all classification models considered, in terms of accuracy, on detecting early signs of each type of insufficiency, on the built covid-19 dataset. overall, all models are more successful on detecting signs of respiratory insufficiency over other types, with most of them achieving an accuracy of more than 70% on this task, while the most difficult signs to be detected are those regarding the hepatic insufficiency. the proposed mnbhl model is the one that achieved the best performance, considering all tasks, followed by logistic regression and the bagmlp model, respectively. it is worth noting that the results achieved by the mnbhl model were obtained by using a network reduction option of 90%, i.e., with only 10% of the training data being preserved for testing purposes. hence, it is expected that these accuracy rates should increase, if one can process the same data without using the network reduction resource. table 5. average accuracy rates (%) and standard deviations for each insufficiency type, obtained by each classification technique. the values between parenthesis indicate the rank achieved by each technique, in each row.full size table fig. 3. boxplots of the overall performance, in terms of accuracy, achieved by the proposed model on detecting early signs from any type of insufficiency in covid-19 patients, grouped by age. full size image table 6. precision, sensitivity and f1 score achieved by each model, when detecting signs from any type of insufficiency in covid-19 patients over 60 years old.full size table given the nature of the covid-19 dataset, the input data may become more or less unbalanced in each age group. in this sense, younger patients, from age groups 00–20 and 21–40 years old, are expected to present a lower incidence of insufficiency cases than older patients, from age groups 60–80 and 80+ years old (see table 3). for analyzing how such differences may affect the model’s performance, we present, in fig. 3, the overall accuracy obtained by the mnbhl model on detecting early signs from any type of insufficiency, on each age group. the boxplots indicate that the model is able to achieve higher performances when analyzing data from older patients, reaching an average accuracy of more than 90% for the 80+ years old age group. people who are older than 60 years are in the high risk group for covid-19. if any sign of insufficiency is detected in patients from this group, prompt measures should be taken by healthcare professionals, oftentimes by making use of mechanical ventilation. for this reason, we also evaluate, in table 6, how the models perform, in terms of precision, sensitivity and f1 score, specifically on data from this group, when detecting signs from any type of insufficiency. from this table, we see that the proposed mnbhl model again obtains competitive results in comparison to classic ones. 6 final remarks in this work, we evaluate the possibility of predicting early signs of insufficiency in covid-19 patients based only on cbc test results, used as biomarkers, through a supervised learning approach. for this end, a training dataset has been built, from original data of de-identified covid-19 patients who received treatment in one of the main hospitals in brazil. additionally, for processing the built dataset, we use a modified network-based high-level classification technique, with novelties including the method used for automating the network building parameters, the measure used for capturing the data pattern structure, and the inclusion of a network reduction option, to facilitate its application on larger datasets. the proposed technique is evaluated by applying it firstly on benchmark classification datasets, and then on the built covid-19 dataset, as well as by comparing its performance with those achieved by other traditional techniques, on the same data. the experimental results are encouraging, with the proposed technique obtaining competitive performance, both on the benchmark datasets and on the built covid-19 dataset. moreover, the considered models were particularly successful on data regarding patients from the risk group, who are older than 60 years old. therefore, we believe the method proposed in this study can be potentially used as an additional tool in the severity assessment of covid-19 patients who have higher risk of complications, thus helping healthcare professionals in this activity. especially the ones who work in regions with scarce material and human resources, as in some places of brazil. as future research, we plan to explore the possibility of extending the classification technique by using additional centrality measures, in a combined form, both in the network reduction method and for the new node’s insertion impact evaluation. we also plan to make applications on larger datasets, such as x-ray and ct images, to further assess the model’s robustness on different type of data. references anghinoni, l., zhao, l., ji, d., pan, h.: time series trend detection and forecasting using complex network topology analysis. neural netw. 117, 295–306 (2019) article  google scholar  beck, b.r., shin, b., choi, y., park, s., kang, k.: predicting commercially available antiviral drugs that may act on the novel coronavirus (sars-cov-2) through a drug-target interaction deep learning model. comput. struct. biotechnol. j. 18, 784–790 (2020) article  google scholar  brandes, u.: a faster algorithm for betweenness centrality. j. math. sociol. 25(2), 163–177 (2001) article  google scholar  breiman, l.: bagging predictors. mach. learn. 24(2), 123–140 (1996) math  google scholar  breiman, l.: random forests. mach. learn. 45(1), 5–32 (2001) article  google scholar  carneiro, m.g., zhao, l.: organizational data classification based on the importance concept of complex networks. ieee trans. neural netw. learn. syst. 29(8), 3361–3373 (2017) google scholar  chen, d., lü, l., shang, m.s., zhang, y.c., zhou, t.: identifying influential nodes in complex networks. phys. a: stat. mech. appl. 391(4), 1777–1787 (2012) article  google scholar  colliri, t., ji, d., pan, h., zhao, l.: a network-based high level data classification technique. in: 2018 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2018) google scholar  csardi, g., nepusz, t., et al.: the igraph software package for complex network research. int. j. complex syst. 1695(5), 1–9 (2006) google scholar  fapesp: research data metasearch. https://repositorio.uspdigital.usp.br/handle/item/243. accessed 1 feb 2021 freund, yoav, schapire, robert e..: a desicion-theoretic generalization of on-line learning and an application to boosting. in: vitányi, paul (ed.) eurocolt 1995. lncs, vol. 904, pp. 23–37. springer, heidelberg (1995). https://doi.org/10.1007/3-540-59119-2_166 chapter  google scholar  gelman, a., hill, j.: data analysis using regression and multilevel hierarchical models, vol. 1. cambridge university press, new york (2007) google scholar  hinton, g.e.: connectionist learning procedures. artif. intell. 40(1–3), 185–234 (1989) article  google scholar  lichman, m.: uci machine learning repository (2013). http://archive.ics.uci.edu/ml metsky, h.c., freije, c.a., kosoko-thoroddsen, t.s.f., sabeti, p.c., myhrvold, c.: crispr-based surveillance for covid-19 using genomically-comprehensive machine learning design. biorxiv (2020) google scholar  pearson, k.: note on regression and inheritance in the case of two parents. proc. royal soc. london 58(347–352), 240–242 (1895) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  rish, i.: an empirical study of the naive bayes classifier. in: ijcai 2001 workshop on empirical methods in artificial intelligence, vol. 3(22) (2001), ibm new york google scholar  safavin, s.r., landgrebe, d.: a survey of decision tree classifier methodology. ieee trans. syst., man, cybern. 21(3), 660–674 (1991) google scholar  secr. saude sp: prefeitura.sp.gov.br/cidade/secretarias/saude/vigilancia\(\_\)em\(\_\)saude/. accessed 9 may 2021 google scholar  shan, f., et al.: lung infection quantification of covid-19 in ct images with deep learning. arxiv preprint arxiv:2003.04655 (2020) silva, t.c., zhao, l.: network-based high level data classification. ieee trans. neural netw. learn. syst. 23(6), 954–970 (2012) article  google scholar  strogatz, s.h.: exploring complex networks. nature 410(6825), 268–276 (2001) article  google scholar  valejo, a., ferreira, v., fabbri, r., oliveira, m.c.f.d., lopes, a.d.a.: a critical survey of the multilevel method in complex networks. acm comput. surv. (csur) 53(2), 1–35 (2020) google scholar  vapnik, v.n.: the nature of statistical learning theory. springer, new york (2000). https://doi.org/10.1007/978-1-4757-3264-1 yan, l., et al.: an interpretable mortality prediction model for covid-19 patients. nat. mach. intell. 2(5), 1–6 (2020) google scholar  zoabi, y., deri-rozov, s., shomron, n.: machine learning-based prediction of covid-19 diagnosis based on symptoms. npj digital med. 4(1), 1–5 (2021) google scholar  download references author information authors and affiliations dept. of computer science, university of fortaleza (unifor), fortaleza, brazil tiago colliri faculty of public health, univ. of sao paulo, sao paulo, brazil marcia minakawa faculty of philosophy, science and letters, univ. of sao paulo, ribeirao preto, brazil liang zhao authors tiago colliriview author publications search author on:pubmed google scholar marcia minakawaview author publications search author on:pubmed google scholar liang zhaoview author publications search author on:pubmed google scholar corresponding author correspondence to tiago colliri . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper colliri, t., minakawa, m., zhao, l. (2021). detecting early signs of insufficiency in covid-19 patients from cbc tests through a supervised learning approach. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_4 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords complex networks classification high-level data pattern characterization covid-19 prognosis insufficiency detection biomarkers cbc tests publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature supervised training of a simple digital assistant for a free crop clinic | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper supervised training of a simple digital assistant for a free crop clinic conference paper first online: 28 november 2021 pp 162–176 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) supervised training of a simple digital assistant for a free crop clinic download book pdf download book epub mariana da silva barros10, igor de moura philippini10, ladson gomes silva10, antonio barros da silva netto10, rosana blawid11, edna natividade da silva barros10 & … stefan blawid10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1240 accesses 4 citations abstract family farming represents a critical segment of brazilian agriculture, involving more than 5 million properties and generating 74% of rural jobs in the country. yield losses caused by crop diseases and pests can be devastating for small-scale producers. however, successful disease control requires correct identification, which challenges smallholders, who often lack technical assistance. the present work proposes a system that detects disease symptoms in images of plant leaves to assist phytopathology experts. the objective is to decrease the experts’ workload and enable consulting services for free or at nominal cost. in addition, the required digital communication channel will promote the formation of a caring community ready to offer unpaid advice for family farmers. the machine learning and refinement of the assistance system are described in detail. the developed classification system achieves a recall value of 95%. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others smart crop health analytics system chapter © 2026 explaining deep learning-based leaf disease identification article 29 july 2024 ensembling of transfer learning for enhanced precision agriculture in plant disease classification chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. agricultural geography agriculture agricultural biotechnology agronomy plant biotechnology subsistence agriculture 1 introduction there are currently about 5 million properties in brazil that subsist on family farming, employing more than 10 million people. according to the brazilian government [1], smallholder farmers represent 84.4% of rural establishments and are responsible for 74% of rural jobs and 33.2% of the agricultural gdp. in pernambuco, located in the northeast region of brazil, family farming employs ca. 83% of the people living in the countryside. however, smallholder farmers are economically vulnerable, especially in a region that still demonstrates one of the highest poverty rates in the country. moreover, they are strongly affected by a changing climate. yet today, pernambuco suffers from major meteorological disasters, especially droughts. despite the great importance of family farming, small-scale producers have to face their challenges alone. they lack the financial resources to seek private agricultural consultants, and governmental support is sparse. according to the agricultural census in 2006 and 2017, about 80% of the agricultural establishments declared not having received any technical support required to adopt modern technology and improve productivity. furthermore, pests and plant diseases can have severe economic impacts, causing losses that reached up to 43% of the annual production in 2016. significant yield losses are avoidable when caused by the misuse (or lack of usage) of phytosanitary products for disease control. this situation aggravates by the aging and low education level of small-scale producers. 15.6% of the farmers who use phytosanitary products are illiterate [1]. the correct disease identification is vital for any treatment strategy that explicitly targets the causing agents, avoiding the excessive use of non-specified agrochemicals. access to technical assistance would inform family farmers about the technological and management innovations and empower them to adopt these measures correctly, reducing the risks inherent to agricultural activities. the present work proposes an assistant system based on deep learning to identify the presence of disease symptoms in images of plant leaves. the assistant was developed in partnership with experts from the phytosanitary clinic of pernambuco (clifipe) hosted by the federal rural university of pernambuco (ufrpe). the goal is to provide technical aid in plant disease diagnosis, prevention, and treatment more efficiently to family farmers. furthermore, the efficiency gain shall enable the experts from clifipe to reach out to a more significant part of the local farming community. the system can classify the leaves’ images, achieving a recall value of 95%. 2 related works the application of technology to agriculture has been approached in several studies that employ mobile services to help smallholder farmers or machine learning to detect diseases. baumüller recently reviewed [2] the literature on agriculture-related services offered through mobile phones, recognizing that the strategic application of information and communication technology provides the best opportunity for economic growth and poverty reduction. the review describes four service categories offered through mobile phones (information dissemination, financial services, access to suppliers, and access to output markets). it also takes a critical look at experimental works and the expected impact related to these services. agricultural mobile services for crop disease monitoring and diagnosis belong to the first category. in a pioneering work, mohanty et al. [3] suggested using a deep learning approach based on image classification to identify selected plant diseases through leaf images. to this extent, the authors created a dataset for disease classification [4] composed of 38 classes defined as a pair of crop and illness. then, they trained a convolutional neural network (cnn) to classify them, achieving an accuracy of over 99%. the dataset includes more than 54000 images from 14 crop species and 26 diseases. most importantly, the publicly accessible dataset enabled several follow-up studies. in the application scenario envisioned in the here proposed work, the cnn only assists in disease recognition. a human expert takes the final responsibility since false identification can cause severe consequences for a small-scale producer. moreover, the farmer himself takes pictures of the symptoms and provides crop information. in addition, the images taken by the farmers in the field will not follow any predefined pattern, as opposed to the proposed dataset created in a laboratory. thus, the cnn needs to be trained with images that match reality and are specific to the crops and diseases encountered in the region where advice shall be provided, here, the state of pernambuco. rangarajan et al. [5] also proposed a system to perform disease classification using plant leaves images. in addition, the study assembled a dataset using pictures from five different diseases of the eggplant crop taken both in a lab and in an actual field. the proposed classifier used vgg16 as a feature extractor and multi-class support vector machine to classify the diseases. even though these techniques are frequently used in image classification tasks, studies indicate that this method produces worse results than using the same network as feature extractor and classifier. barbedo [6] introduced a system to identify plant diseases using the public dataset digipathos [7], created by the brazilian agricultural research corporation embrapa. the author also used deep learning to perform the image classification but explored individual lesions and spots rather than considering the entire leaf. since the embrapa dataset is not balanced, the segmentation in individual lesions acts as data augmentation. the system is specialized to detect and classify diseases based mainly on these spots. in its current form, our digital assistant suggests to a human expert larger picture segments to provide context and aid in the identification. however, lesion-based segmentation is considered for future work. 3 proposed approach 3.1 system overview as mentioned in sect. 1, the system’s objective is to help farmers monitor their plantations and identify whether the plant is diseased. for this purpose, the system uses machine learning and computer vision techniques to analyze leaves images and detect whether they show or not disease symptoms. we also built a digital platform where both farmers and experts can connect and interact. such a platform facilitates and improves the provided diagnosis and assistance. figure 1 shows the system’s general structure. fig. 1. general architecture of the proposed system composed by a mobile app and a digital assistant for a crop clinic full size image the proposed platform is accessible to farmers through a mobile application. the user selects a crop from a predefined collection and takes a picture of a crop leaf. images are automatically segmented and submitted to the classification system to detect the probability of revealing disease symptoms. all pictures are stored in a database and are used to fine-tune and continuously optimize the classification systems’ performance. the app also displays general information about possible diseases to the user, such as its most common symptoms and measures for prevention. however, more specific advice on disease control is left to direct communication with a phytopathology expert. the knowledge base is an information database created by experts from clifipe. the mobile app allows the farmer to directly communicate with the experts, exposing doubts and receiving assistance with problems in their crops. the community also grants an opportunity for the farmers to help each other and share knowledge. currently, the system is trained to identify symptoms in images of grapes leaves. however, the plant disease diagnosis will focus on the main food crops cultivated in pernambuco, and later in the northeast region of brazil. therefore, one of the project’s goals is to build a dataset of images of the region’s most common and essential crops and the diseases affecting them. we also aim to make this dataset available to all researchers who work on improving plant disease management. 3.2 crop clinic digital assistant to correctly identify the presence of disease symptoms in plant leaves images, we developed a system that generates segments from images of plant leaves and uses deep learning techniques to classify whether they show disease symptoms or not. section 2 delineates some examples of works that perform similar tasks. however, images are usually taken in a controlled environment, with supervised conditions and a standard background. available training datasets, such as the plantvillage [4] and the digipathos dataset [7], do not include images with variations in lighting, size, or framing and thus does not reflect actual field conditions. as a result, according to [3], models trained over such datasets achieve only an average precision of 31% when applied to authentic field images. here we use photos taken in the field to train the neural network model. since images taken with a mobile phone by inexperienced users will exhibit similar picture quality and detail variations, the cnn will recognize the presence of disease symptoms more reliably. figure 2 shows the general architecture of the digital assistant. the system takes the raw images to be classified as input and consists of three modules: fig. 2. tasks performed by the digital assistant full size image cropping module: cropping the input image centers the leaf and eliminates parts of the background. manually cropped images constitute the training and validation data sets to guarantee a certain standard, improving the classification performance of the cnn. however, photos submitted by the users will be cropped automatically once the app is more broadly disseminated. the algorithm will use cropping margins based on typical pictures taken by the app users. pre-processing module: the division of leaves images into segments depends on the frame proportions. frames with a height-to-width ratio below a certain threshold are separated into four segments, otherwise in six. the chosen threshold value guarantees that each segment shows a similar degree of detail. segmentation increases the size of the training data set, allowing to exclude parts with limited information. also, it gives more flexibility in balancing the training data set by selected inclusion of segments of varying exposure conditions and quality. the generated segments are resized using the bilinear interpolation technique [8] to match the input shape of the neural network (256 \(\times \) 256 \(\times \) 3). classification module: a convolutional neural network classifies the input segments as showing “symptoms” or “no symptoms”. a cnn is a deep learning technique widely used in pattern recognition that employs a single network to learn and classify the image features [9]. we base our model on the resnet50v2 pre-trained model. details of the image training dataset will be given in sect. 4. the digital assistant performs only binary classification, and forwards the segments with the most pronounced symptoms (highest probability) for further analysis to a phytopathology expert. selecting the segments that most clearly demonstrate the disease symptoms is a valuable step in disease identification and agent recognition, either by human experts or another machine learning model. 4 hyperparameter tuning and training we discuss the classification results of the digital assistant in two parts. first, in the present sect. 4, we will assess the performance of different trained models in the search for a balanced training dataset and optimal hyperparameters. then, in sect. 5, the classification of segments from twelve selected leaves images (that were not part of any training dataset) will be discussed in detail for illustration. 4.1 dataset collection currently, some online datasets are available for plant disease classification experiments, e.g., the plantvillage [4] and the digipathos dataset [7]. however, their images are usually taken in a lab and do not present variations in lighting, size, or framing, thus not reflecting the field conditions. therefore, to improve the network performance when applied to images taken in the field, we manually build the dataset used in the network training. the dataset employed in this study is composed of images collected by us in the region of the siriji valley, in pernambuco’s countryside. there are several smallholders plantations in the area, and we collected images from plant leaves at some of them. among the cultivated crops, the most important ones for the region’s economy and production include grape, banana, and sugar cane. for this study, we trained a cnn to identify disease symptoms in grape leaves images, but the extension to other crops is straightforward. we took images from leaves in different growth stages, both healthy and diseased, and in different lighting conditions. then, the photos were annotated by phytopathology experts from clifipe, identified as manifesting symptoms or not. after the expert annotation, the dataset size was: 1987 images for the class “no symptoms” 1302 images for the class “symptoms” the collected pictures were then manually cropped and segmented, as described in sect. 3.2. after this step, they are ready to be divided into subsets and used in the experiments. 4.2 dataset division not all images were used for the training of the cnn. first, we separated a small number of representative images to illustrate how particular image characteristics influence the classification results of selected trained models. to ensure some variability and balancing between classes and conditions, twelve leaves images (six from the class “no symptoms” and six from the class “symptoms”) were selected for this final case study, exhibiting different illumination, focus, and contrast conditions. the selected images were not used in the training of any of the models and are not used to extract statistics results. the final case study, reported in sect. 5, only illustrates possible outcomes when the trained models are applied and emulates the challenges of a usage scenario at a large scale. second, the remaining images were divided into training and testing subsets in several different ways. this allows the training and verification of distinct neural network models. we will use this approach to estimate the expected performance variability when using these models for the digital assistant in a crop clinic. moreover, insights into the composition of a well-balanced training dataset are obtained. after cropping and segmentation of the collected images, the complete dataset contained 5750 images for the class “symptoms” and 7676 images for the category “no symptoms”. the entire dataset is then randomly divided into two groups, one for training (with 75% of the total pictures) and one for testing (with the remaining 25%). after the division, the set sizes are: train set: 10069 images (4312 for the class “symptoms” and 5757 for the class “no symptoms”) test set: 3357 images (1438 for the class “symptoms” and 1919 for the class “no symptoms”) the generated train subset is used for the neural network model training. for this reason, the set must be composed of a sufficient number of images covering a significant variety of segments. the test set is used for the trained model to perform predictions. it allows estimating the performance when used as a digital assistant, classifying pictures taken by the mobile app users. note that the random division is repeated several times to obtain distinct neural network models. 4.3 model training during the training, the model might face learning problems of varying severity. in extreme situations, the model exhibits underor overfitting. if the model does not learn to generalize well, then it becomes underfitted and will have a poor performance on training data. on the other hand, if the model retains the training dataset pattern too well, including all specific peculiarities, it becomes overfitted. thus, the learning progress must be monitored by validating the model after each epoch (each training iteration updating the internal network parameters). to this extent, a certain percentage of the training dataset needs to be separated as a validation set. the model’s performance over this subset, evaluated after each epoch, will indicate potential fitting problems. we used 20% of the training set for validation. moreover, the remaining 80% of the images (used for training) were carefully balanced according to the number of images belonging to each class. the ratio between the size of the two categories determined the required augmentation of the class with fewer images, which is achieved by image rotation and flipping. the cnn was configured using a sigmoid activation function in the output layer. during training, binary cross-entropy served as a loss function. the chosen optimizer was adam, with a learning rate of 0.0001. the evaluation metric most closely observed during training should reflect the desired outcome. in this case, the digital assistant shall forward to experts image segments that possibly show symptoms for further inspection. thus, ideally, segments with no symptoms will be classified correctly (“true negatives”, tn) and not analyzed. still, any segment possibly showing symptoms need to be inspected, i.e., should not be classified erroneously (“false negatives”, fn). therefore, during training, an essential evaluation metric is the recall metric, computed as \(\mathrm tp/(tp+fn)\). nevertheless, other evaluation metrics were also computed when applying the model to the test dataset, such as accuracy, precision, f1-score, and average precision. we trained the model using a batch size of 128 and a maximum number of epochs of 100. however, an early stopping criterion was employed to avoid overfitting, with a threshold of 3 epochs with no progress. the classification module was implemented using the tensorflow framework. fig. 3. training results for a well performing model. full size image figure 3 shows the training progress and the validation after each epoch for (a) the recall metric and (b) the loss function. the trained model achieves a high recall score of 95%. 4.4 prediction for performance evaluation, the trained model is applied to the images of the test set (not used in training). the prediction returns a probability between 0 and 1, indicating the likelihood of the segment showing disease symptoms. a threshold value of 0.5 separates the classes, i.e., a prediction below 0.5 indicates an image segment with “no symptoms”, otherwise with “symptoms”. using the confusion matrix resulting from the prediction, we calculated the accuracy, precision (both for the positive and negative classes), recall, true negative rate, f1-score, and average precision for the performance assessment. however, the metric most closely observed is the recall. fig. 4. flow diagram of the performed computational experiments full size image 4.5 performance optimization different dataset division strategies and hyperparameter value options were tested and evaluated to determine the ones that maximize the model performance. figure 4 shows a diagram that summarizes the experiment flow. we implemented a script to repeat the complete assessment (training and testing) a certain number of times (n) while performing a grid search for high-performance models. for each repetition, the dataset is divided differently (maintaining the ratio of 75/25). then, the trained model is applied to the respective test dataset of the repetition, and the evaluation metrics are computed. after n assessments, the mean and standard deviation of the predicted recall metric from each execution are calculated. although not representing a systematic cross-dataset validation, the approach will identify flyers, i.e., models that show atypical performance due to an accidentally introduced bias in the randomly selected training dataset. the script also saves the models that produce the best recall, the worst recall, and the closest recall to the mean. in addition, other prediction measures are also observed to guarantee that the model is behaving as expected. applying these models to the case study dataset, composed equally of healthy and diseased leaves images with different quality levels, gives further insights into the strategy for finding a well-performing model. due to image variability, a suitable dataset for training should not only balance between images showing disease symptoms or not but should also balance between pictures of good and low quality. the training of n distinct models helps to understand both the expected performance, capability, and limitations of the neural network, as well as the impact of the training. table 1 summarizes the findings of the first group of assessments. a minimum number of repetitions of dataset splits and model training is required to reliably explore the complete dataset. we found that \(n=50\) is sufficient for the collected data. the mean recall and standard deviation do not change significantly for larger n, but the computation time increases. another interesting finding concerns the dataset division strategy. the data can be divided into train and test sets before or after separating the leaves’ pictures into segments. the former approach (before segmentation, defined in the table as the split category “leaves”) ensures that all the parts from a given picture end up in the same group. however, constructing the training set on segment level (splitting after segmentation, defined in the table as the split category “segments”) increases flexibility. this leads to a smaller standard deviation and reduced gap between maximum and minimum recall. finally, the first group of assessments also included an evaluation of the desirable dataset size. we tested the use of the complete dataset, as opposed to the use of a partial one. as shown in tab. 1, a training set that is too small leads to a reduced recall mean and maximum value, independently of the division strategy. however, the 14076 images collected in a dedicated field expedition are sufficient to train models with good performance and high evaluation metrics. the table also shows the mean values of the precision, accuracy, and f1-score measured in the experiments. table 1. assessment results group 1full size table table 2 summarizes the findings of the second group of assessments, related to the optimization of the training algorithm. inside the second group, some experiments were conducted to determine the best weight to be attributed to each of the two classes during training. choosing different weights for the classes allows to balance them without resampling. in other words, weights can tell the model to pay more attention to the instances of a particular category. table 2 shows that attributing weight “1” to class “0” (“no symptoms”) and weight “2” to class “1” (“symptoms”) gives a higher recall mean. this result is plausible because the importance to the class “symptoms” is increased, and we are mainly trying to avoid false negatives. table 2. assessment results group 2full size table in addition, the evaluation metric and the classification threshold were also analyzed, but are not shown in the table. some models were trained using the accuracy metric instead of the recall metric. however, they produced too many false negatives, which is not desirable since segments classified as showing no symptoms will not be called for further inspection by a human expert. we also tested different threshold values for the classification but found that the standard choice of 0.5 gives the most reliable prediction. as mentioned in sect. 2, the purpose of the present work is to detect the presence of symptoms in the leaf image, which constitutes a different application scenario as discussed in the state-of-the-art literature. therefore, the implemented neural network model was specifically developed for the present application. consequently, we cannot compare our model to a reference. promising models trained in the various assessments were applied to the (previously separated) case study dataset, giving further insights into the expected classification performance when analyzing leaves images of varying quality. table 3. prediction results of two selected modelsfull size table 5 case study and discussion twelve models were chosen for the final inference study. these models gave the best, worst, and mean recall in the assessments presented in the four last lines of table 1. as stated in sect. 4, the case study dataset is composed of images from both classes (showing or not symptoms) for various light, focus, and background conditions and different severity levels of symptoms. the analysis of the predictions revealed some characteristics present in the inference images that influence the classification accuracy. for example, when the area occupied by the background is small, it is easier for the model to classify the leaf segment correctly. however, the classification is more challenging for the model when the sunlight is too bright, or the leaf image does not have enough contrast. thus, a carefully composed and sufficiently large training dataset is essential. as shown in table 1, all models trained using a smaller dataset (constituted of about 3600 images) achieved a poor performance, committing many errors (both false positives and false negatives) when applied to the case study dataset (results not shown). meanwhile, models trained with a more extensive dataset performed better. fig. 5. classification results for selected healthy leaves images obtained from two distinct models full size image however, defining the correct target metrics values for optimal performance is challenging. table 3 displays the predicted evaluation metrics of two models that produced different recall values, out of \(n=50\) different training and test sets when divided on segment level (second line of table 1). the results indicate how the changes in the training dataset are related to the output metrics. for example, the first model achieved better results for metrics indicating the absence of false negatives, like negative precision and recall. on the other hand, the second model performed better for metrics that increase with decreasing false positives, like positive precision and true negative rate. however, this does not necessarily imply that the second model leads to more decisive errors in predicting the negative class (“no symptoms”). figure 5 displays the classification results of the two models applied to the images from the case study dataset of healthy leaves. the leaves images are divided into segments and show the model output classification (symptoms “yes” or “no”) and the corresponding probability for each of them. the first line under each segment is the prediction obtained from model 1 from table 3, and the second line is from model 2. similarly, fig. 6 shows the diseased leaves images and their classification results when the two models are applied. fig. 6. classification results for selected diseased leaves images obtained from two distinct models full size image as expected, probabilities are higher when predicted by the first model in comparison to the second one. although it is desirable to avoid false negatives (the objective of the training), model 1 leads to more errors in predicting the negative class, i.e., the model classifies segments without symptoms as presenting them (fp). table 4 shows the confusion matrices results from the two models when applied to the case study dataset. the inference study also revealed potential future improvements, such as, for example, the use of multi-label classification. this technique uses the same neural network to classify multiple different labels at the same time. that might allow, for instance, the enhancement of other prediction metrics maintaining the recall values. table 4. confusion matrices for two selected models over the case study datasetfull size table 6 conclusion the present work proposes using computer vision and machine learning to identify the presence of disease symptoms in plant leaves images. the main objective is to assist experts from the phytosanitary clinic of pernambuco and enable consulting services for free or at a nominal cost offered to small-scale agricultural producers. the implemented system allows smallholders to communicate with phytopathology experts and the community of users. in addition, the digital assistant successfully solves the problem of classifying whether image segments taken by inexperienced users show disease symptoms or not and, thus, acts as a filter for the subsequent analysis by human experts. we also tackled the challenge of lacking data for training a convolutional neural network by building our own training set composed of images of essential crops from pernambuco. the training dataset balances images of good and poor quality to maximize the recall of photos taken from non-experts in the field. the best-performing model could achieve a recall of over 95%. moreover, the annotated dataset will continuously improve with a growing user community. test and inference results for distinct trained models gave insights into an ideal training dataset and the influence of exposure conditions when taking the photos to be classified in the field. we also found that models compromising on multiple metrics may give better classification results. future works include optimizing the classification system, automatizing identification of the disease-causing agent, and using a multi-label approach to improve the model performance. references ibge: agricultural census 2017. https://censoagro2017.ibge.gov.br/templates/censo_agro/resultadosagro/index.html. accessed 1 mar 2021 baumüller, h.: the little we know: an exploratory literature review on the utility of mobile phone-enabled services for smallholder farmers. in: journal of international development, vol. 30, pp. 134–154. willey online library (2018). https://doi.org/10.1002/jid.3314 sharada, p.m., david, p.h., salathé, m.: using deep learning for image-based plant disease detection. in: frontiers in plant science, vol. 7, pp. 1419. frontiers (2016). https://doi.org/10.3389/fpls.2016.01419 sharada p.m.: plantvillage dataset. https://github.com/spmohanty/plantvillage-dataset/tree/master/raw. accessed 1 mar 2021 rangarajan, a.k., purushothaman, r.: disease classification in eggplant using pre-trained vgg16 and msvm. in: scientific reports, vol. 10, pp. 1–11. nature publishing group (2020). https://doi.org/10.1038/s41598-020-59108-x barbedo, j.g.a.: plant disease identification from individual lesions and spots using deep learning. in: biosystems engineering, vol. 180, pp. 96–107. science direct (2019). https://doi.org/10.1016/j.biosystemseng.2019.02.002 embrapa: digipathos dataset. https://www.digipathos-rep.cnptia.embrapa.br/. accessed 1 mar 2021 smith, p.r.: bilinear interpolation of digital images. in: ultramicroscopy, vol. 6, pp. 201–204. science direct (1981). https://doi.org/10.1016/0304-3991(81)90061-9 voulodimos, a., doulamis, n., doulamis, a., protopapadakis, e.: deep learning for computer vision: a brief review. in: computational intelligence and neuroscience, vol. 2018, pp. 1–13. hindawi (2018). https://doi.org/10.1155/2018/7068349 download references acknowledgments this work was supported by an a.i. for earth microsoft azure compute grant. m.b. would like to thank capes for the master’s degree scholarship. s.b. is a cnpq fellow. the authors also thank the experts from clifipe, especially dr. jonas alberto rios and dr. andré angelo medeiros gomes, for helpful discussion on the working of a crop clinic. author information authors and affiliations centro de informática, universidade federal de pernambuco, recife, brazil mariana da silva barros, igor de moura philippini, ladson gomes silva, antonio barros da silva netto, edna natividade da silva barros & stefan blawid departamento de agronomia, universidade federal rural de pernambuco, recife, brazil rosana blawid authors mariana da silva barrosview author publications search author on:pubmed google scholar igor de moura philippiniview author publications search author on:pubmed google scholar ladson gomes silvaview author publications search author on:pubmed google scholar antonio barros da silva nettoview author publications search author on:pubmed google scholar rosana blawidview author publications search author on:pubmed google scholar edna natividade da silva barrosview author publications search author on:pubmed google scholar stefan blawidview author publications search author on:pubmed google scholar corresponding author correspondence to mariana da silva barros . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper barros, m.d.s. et al. (2021). supervised training of a simple digital assistant for a free crop clinic. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_12 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords disease identification machine learning family farming publish with us policies and ethics profiles stefan blawid view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature experiments on portuguese clinical question answering | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper experiments on portuguese clinical question answering conference paper first online: 28 november 2021 pp 133–145 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) experiments on portuguese clinical question answering download book pdf download book epub lucas emanuel silva e oliveira  orcid: orcid.org/0000-0003-1811-508710,11, elisa terumi rubel schneider  orcid: orcid.org/0000-0002-8921-559810, yohan bonescki gumiel  orcid: orcid.org/0000-0001-8239-293010, mayara aparecida passaura da luz  orcid: orcid.org/0000-0001-9932-737710, emerson cabrera paraiso  orcid: orcid.org/0000-0002-6740-785510 & … claudia moro  orcid: orcid.org/0000-0003-2637-308610  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1282 accesses 4 citations abstract question answering (qa) systems aim to answer human questions made in natural language. this type of functionality can be very useful in the most diverse application domains, such as the biomedical and clinical. considering the clinical context, where we have a growing volume of information stored in electronic health records, answering questions about the patient status can improve the decision-making and optimize the patient care. in this work, we carried out the first experiments to develop a qa model for clinical texts in portuguese. to overcome the lack of corpora for the required language and context, we used a transfer learning approach supported by pre-trained attention-based models from the transformers library. we fine-tuned the biobertpt model with a translated version of the squad dataset. the evaluation showed promising results when evaluated in different clinical scenarios, even without the application of a clinical qa corpus to support a training process. the developed model is publicly available to the scientific community. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others answering medical questions in chinese using automatically mined knowledge and deep neural networks: an end-to-end solution article open access 15 april 2022 question answering chapter © 2024 a pre-trained language model for medical question answering based on domain adaption chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. clinical medicine machine translation machine learning patient education predictive medicine translational research 1 introduction question answering (qa) is a task that seeks to automatically answer questions asked in natural language by humans, with a combination of information retrieval (ir) and natural language processing (nlp) techniques. qa systems have been applied in multiple domains of knowledge, as they can facilitate access to information by reasoning and inferring it from a large volume of data stored as natural language [1]. there has been a growth in interest in the use of qa in the biomedical context, including efforts to create biomedical and clinical qa corpora [8, 12, 18, 23] and the realization of shared tasks such as bioasq 8 [9]. the possibility of finding answers to questions in the growing volume of data available in the scientific literature and patients’ medical records evidence the potential of these systems for practical use. therefore, qa systems can assist both the medical research community (e.g., researching disease treatments and symptoms) and healthcare professionals (e.g., clinical decision support). specifically in the clinical context, when we work with patient’s data, the qa research has few initiatives if compared to other domains, mainly in languages other than english [10, 12]. the lack of studies may be due to the required medical expertise and the ethical/legal restrictions, which limits the data access [7]. another aspect that hinders the research of clinical qa is the complexity of the data that are stored in the patient’s electronic health records (ehr). the longitudinal and unstructured nature of the clinical notes, high use of medical terminology, temporal relations between multiple clinical concepts (e.g., medications, diseases, symptoms), text misspellings, frequent redundancy, and implicit events make text comprehension much more complex for the machine, and make it difficult to apply baseline nlp tools [2]. currently, health professionals manually browse or use a simple search algorithm to find answers about patients, and often they are unable to reach the information they need. a qa system can provide a much more intuitive interface for this task, allowing humans to “talk" to the ehr asking questions about patient’s status and health condition, generating a much more fluid and human interaction [12, 16, 18]. nevertheless, how to develop a clinical qa system for portuguese if we do not have a large corpus of questions and answers that support most of the current qa methods? a well-established strategy in the nlp community is the use of transfer learning, which consists of using a pre-trained language model in a large dataset and then fine-tune it for a specific task in a smaller dataset. for example, one can use a large attention-based model [19] available in transformers libraryfootnote 1, such as bert [3], and fine-tune it for a specific nlp task, such as named entity recognition [17]. we can go beyond that, like jeong and colleagues [7], who fine-tuned two different qa corpora (one open-domain and the other biomedical) to achieve superior performances in biomedical qa. in this work, we present the first glance of experiments on a qa system used to answers questions about patients in portuguese. we developed a biomedical qa model for the portuguese language based on the fine-tuning of the biobertpt model [15] using the squad qa dataset translated into the portuguese language. we evaluate the performance in a small set of different clinical contexts with multiple types of clinical notes. finally, we discuss our preliminary results, main findings, and future work. the model is publicly available for use. 2 related work in this section, we present important qa corpora development initiatives. moreover, we introduce the transfer learning method, which is one of the solutions used to overcome the lack of domain-specific corpora. furthermore, we highlight biomedical and clinical qa studies based on transformers models and transfer learning. 2.1 corpora in the literature, we can find several qa datasets, most of them covering open-domain questions and answers, as the stanford question answering dataset (squad) [14]. squad is a reading comprehension dataset consisting of english questions and answers from a set of wikipedia articles. this dataset is present in a qa format, where the answer to each question is a segment of text from the reading passage (see fig. 1). the squad 1.1 contains more than 100,000 question-answer pairs on approximately 500 articles, all of them manually annotated. fig. 1. example of answers annotated within the text in squad corpus. image from [14] full size image regarding the biomedical context, we need to mention the pubmedqa, a corpus with the objective of answering research questions with “yes”, “no” and “maybe”. it consists of 1,000 expert-annotated qa pairs and 211,300 artificially generated qa pairs. all the answers were annotated in pubmed scientific paper abstracts [8]. in the clinical domain, we highlight the following recent initiatives. yue and colleagues [23] developed a corpus with 1,287 expert-annotated qa pairs, originated from 36 discharge summaries from the mimic-iii datasetfootnote 2. the emrqa corpus [12] has minimal expert involvement, as they generated the qa pairs using an automated method that leverages annotations from shared i2b2 datasetsfootnote 3, resulting in 400,000 qa pairs. it is worth noting the discrepancy of manually annotated qa pairs from squad and the biomedical and clinical corpora, which affects directly the development of qa algorithms for these specific domains. 2.2 transfer learning transfer learning is a method focused on the reuse of a model trained for a specific task, by applying and adjusting it (i.e., fine-tuning) to a different but related task. several studies have already proven the efficacy of transfer learning when using deep learning architectures [13], especially the pre-trained attention-based models available in the transformers library. the biobertpt [15] is an example of how transfer learning can result in models with low computational cost and at the same time extremely useful. biobertpt was generated from the fine-tuning of the multilingual version of bertfootnote 4 in clinical and biomedical documents written in portuguese. the model was also adjusted for the named entity recognition (ner) task, which achieved state-of-the-art results in the semclinbr corpus [11]. 2.3 biomedical and clinical qa considering the current availability of domain-specific corpora for the qa task, many studies focused on transfer learning to take advantage of the huge amount of annotated data in open-domain resources, as the squad. furthermore, due to the state-of-the-art results obtained by architectures based on the attention mechanism on several nlp tasks, most studies explore the fine-tuning of pre-trained models available in the transformers library. wiese and colleagues [20] transferred the knowledge from the squad dataset to the bioasq dataset. in yoon’s work [22], they executed a sequential transfer learning pipeline exploring biobert, squad, and bioasq. in [7], the authors used sequential transfer learning as well but trained biobert on the nli dataset. finally, soni and roberts [16] performed experiments on multiple transformers models and fine-tuned them with multiple qa corpora combinations. they found that a preliminary fine-tuning on squad improves the clinical qa results across all different models. due to the unavailability of a clinical qa corpus in portuguese and the good results obtained when using transfer learning based on an open-domain qa corpus (i.e., squad), we decided to follow a similar development protocol. 3 materials and methods this section describes the development of our transformer-based qa model, including all the steps of execution and details regarding the data used. next, we present the evaluation setup used to assess our approach. 3.1 biobertpt-squad-v1.1-portuguese: a biomedical qa model for portuguese as seen in the previous section, the use of transfer learning supported by contextual embedding models, pre-trained on large-scale unlabelled corpora, combined with the transformer architecture, has proven to reach state-of-the-art performance on nlp tasks. then, our method will focus on these pre-trained models and the consequent fine-tuning of the qa task. the method is composed of the following general phases: 1. obtain a model for biomedical portuguese texts 2. obtain a dataset for qa in portuguese 3. fine-tune the biomedical model to qa task for our first phase, we needed a model capable of processing portuguese biomedical texts, and for that, we used the biobertpt model. the biobertpt is a deep contextual embedding model for the portuguese language to support clinical and biomedical tasks. the authors fine-tuned the multilingual bert on clinical notes and biomedical literature, resulting in a portuguese medical bert-base model. in this paper, we used the biobertpt(all) versionfootnote 5, which includes both biomedical and clinical texts. as the second phase, we used the automatically translated version of the squad dataset, shared by the deep learning brasil groupfootnote 6, which we will call squadpt from now on. they used google-powered machine translation to generate a portuguese version of squad v1.1, with additional human correction after. it is worth noting the warning made by the authors, in which they state that the dataset could contain a good amount of unfiltered content with potential bias. in the final phase, we fine-tuned biobertpt on the question and answer dataset, the squad-pt, with the following steps: 1. pre-processing of data (removal of spaces and possible duplicate records); 2. tokenization of the inputs and converting the tokens to their corresponding ids in the pre-trained vocabulary, with the transformers tokenizer [21]; 3. split of long inputs (longer than the model maximum sequence length) into several smaller features, overlapping some features if the answer is at the split point; 4. mapping of the start and end positions of the answers in the tokens, so the model can find where exactly the answer is in each feature; 5. padding the text to the maximum sequence length. finally, we performed the fine-tuning to the qa task with the transformers automodelforquestionanswering class [21], using biobertpt pre-trained weights. we trained our portuguese biomedical qa model for three epochs on a gpu gtx2080ti titan 12 gb, with learning rate as 0.00003, and both training and evaluation batch size as 16. all these parameters were recommend for fine-tuning step by [3], at appendix a.3 of the paper. table 1 shows the main hyper-parameter settings used to train our model. in fig. 2 we can see the bert model trained for the qa task, where given input with the question and context, the model predicts as output the start and end indices of the probable answer to the question. the resulting model is available at transformers repositoryfootnote 7. fig. 2. bert model architecture for qa task, where it is packed both the question and the context (i.e., reference text) into the input and as output, the model predicts the beginning and end spans of the answer. in bold, the expected answer to the input question. full size image table 1. main hyper-parameter settings of biobertpt-squad-v1.1-portuguese trainingfull size table 3.2 evaluation setup our model achieved a f1-score of 80.06 when measuring using the squad dataset annotations. however, the goal of our work is to generate a qa model to be used in the clinical domain, therefore, we need to define an assessment protocol that addresses this context. in the following subsections, we describe the selection of the clinical notes and the definition of the questions used to evaluate our model. clinical notes selection. there are many uses of a qa system in the clinical environment, being able to support the entire multidisciplinary health staff. among the possibilities, the system could help the medical and nursing professionals, who constantly need to retrieve information from the patient’s medical record to carry out their interventions. thinking about these cases, we selected two main categories of clinical notes, nursing notes, and medical notes. the nursing note is a document that describes the general state of the patient and actions performed by the nurses during the care (e.g., physical examination, medications used, vital signs monitoring). this document is used to compare the patient’s clinical condition during the last 24 h and is very useful to determine optimal strategies of care (e.g., new physical examinations) [4]. the medical note contains a narrative regarding the patient’s signs, symptoms, comorbidities, diagnosis, prescribed medications, post-surgery data, etc. all this information is written in chronological order, to facilitate the understanding of the history of care and support the decision of the healthcare professionals [5]. the nursing and medical notes were selected from the semclinbr corpus, prioritizing narrative aspects that would allow the realization of questions, to use the answers to facilitate the elaboration of a nursing diagnosis by the nursing team, during the shift. as well as carrying out a quick search by the multidisciplinary team to support the practices to be adopted in the patient’s care. three nursing notes and six medical notes were selected, being three from medical assistance (i.e., hospital admission) and three from ambulatory (i.e., outpatient service). questions definition. after defining the clinical notes to be used as the context for our qa model, we defined a set of questions that we would ask the model to try to find the answers to. the questions were based on nursing diagnosis, risk factors, and defining characteristics, that is, a set of information that demonstrates signs and symptoms of a given diagnosis. the selected criteria were: physical examination, level of consciousness, circulation and ventilation, procedures/care, nutrition, physiological eliminations, intravenous devices, pain complaints and exams. for nursing notes, ten questions were elaborated and applied. for the medical notes, eight questions were elaborated and applied. evaluation process and metrics. to generate the gold standard of answers, a specialist analyzed all selected clinical notes and annotated the text with the corresponding answers (examples are presented in fig. 3 and fig. 4). after the generation of the gold standard, all questions and clinical notes were input to our qa model, so we made a comparison between the model’s answers and those generated by the expert. a calculation of precision, recall and f1 metrics was performed. we used a partial-match strategy since once the qa system finds and marks the answer in the text, even partially, the user could easily identify the rest of the information in the text. for example, in the gold standard the excerpt of text that answers the “has previous diseases?” question is “previously hypertension/diabetes/chronic obstructive pulmonary disease. stroke with left hemiparesis 2 years ago.”, but the model outputted just a partial match: “previously hypertension/diabetes/chronic obstructive pulmonary disease.”. in these cases the partial match approach considers the prediction as a true positive. it is worth noting that our gold standard was not used for training purposes. we used it only for evaluation, as we exploited the use of transfer learning and already built resources to develop our model. 4 results the result of the evaluation process is summarized in table 2, where we can see better results when we process nursing and outpatient notes. in fig. 5, all items evaluated and the confidence rate given by the model when making the prediction were plotted. in addition to the hits and misses, we also include the situation in which the question asked does not contain an answer in the text, therefore, it is not possible to make a right answer. an analysis of the graph suggests that if we use a threshold of 0.5 when returning the answer to the user (<0.5 then no answer; \(\ge \)0.5 then show answer) we could avoid several false positives. however, we would also have a reduction of true positives, so a more in-depth analysis of this trade-off is needed. fig. 3. example of nursing note and questions extracted from the gold standard. the answers to each question are annotated within the text with the corresponding color. full size image fig. 4. example of medical ambulatory note and questions extracted from the gold standard. the answers to each question are annotated within the text with the corresponding color. the name appearing in the text is a result of a de-identification process. full size image table 2. evaluation results considering the clinical notes types.full size table fig. 5. scatter plot containing all matches, errors and situations in which the text did not contain the answer to the question. the y-axis defines the confidence score of the model when making the prediction. full size image 5 discussion this work is an initial investigation into the development of a clinical qa system in portuguese, which could be a powerful resource in clinical decision support and patient care process optimization (e.g., discover information about the patient’s condition, optimal shift change). the aim of this article is to carry out preliminary experiments to help define the next steps towards a robust model of clinical qa. our findings are in line with other studies [7, 16, 20, 22] regarding the use of transfer learning as a means of overcoming the lack of annotated resources for the desired language/context, as we use the biobertpt model, which is a result of the fine-tuning of the multilingual bert for clinical texts, and we also used the squad, which, despite being an open-domain dataset, helped us to adapt the model for the clinical qa task. however, it is worth emphasizing that, even without quantitative results to benchmark, apparently the results of this process of creating the model for the clinical domain require a little more attention, due to the great differences in vocabulary, questions and specificities of clinical notes. we can find models that followed a similar process of development to our model, working a little more cohesively when applied to the context of questions about medical topics [6]. a more detailed analysis about the use of a threshold in the model output should be carried out, as it is common to have missing data in clinical notes. furthermore, it is necessary to study the different types of possible questions (i.e., factoid, list, yes/no) in the clinical context, since using all these types of questions considerably increases the complexity of the problem. moreover, something that is also out of the scope of this article is to understand the use of discontinuous answers, when the response is in multiple parts of the clinical note. as the transformers model allows returning the top k texts, it might be possible to perform some kind of inference through this list. we recognize as limitations of this work the size of the dataset used for evaluation, as well as the search for answers about the patient’s status in a single patient’s clinical note and not in all of its ehr records. as future work, we intend to expand both the dataset size and the exploration of the entire longitudinal aspect of the patient’s clinical notes. finally, since we intend to expand the gold standard for evaluation, a natural step would be to use this corpus to train a model directly from these data and also make it available to the scientific community as well as emrqa[12] and clicr [18]. 6 conclusion we developed a clinical qa model for the portuguese language by sequentially fine-tuning attention-based models available in the transformers library. the model is publicly available to the scientific community. we used the biobertpt model with the squadpt qa dataset to build a model capable of finding answers about a patient within the clinical notes. our experiments show promising results, even with the use of resources adapted (i.e., fine-tuned) to the target task, context and language. we plan to expand our experiments in various aspects, as the increase of evaluation dataset size, use of multiple patients’ clinical notes, training a model with our own clinical qa corpus, and deepen analysis about the use of thresholds, discontinuous answers and question types. we understand that this work was the first step towards a robust clinical qa system in portuguese, which could result in more intelligent systems for healthcare professionals and improve the patients’ care. notes 1.https://github.com/huggingface/transformers/. 2.https://physionet.org/content/mimiciii/1.4/. 3.https://www.i2b2.org/nlp/datasets/main.php. 4.https://huggingface.co/bert-base-multilingual-cased. 5.https://huggingface.co/pucpr/biobertpt-all. 6.http://www.deeplearningbrasil.com.br/. 7.https://huggingface.co/pucpr/biobertpt-squad-v1.1-portuguese. references calijorne soares, m.a., parreiras, f.s.: a literature review on question answering techniques, paradigms and systems (2020). https://doi.org/10.1016/j.jksuci.2018.08.005 dalianis, h.: characteristics of patient records and clinical corpora. in: clinical text mining, pp. 21–34. springer, cham (2018). https://doi.org/10.1007/978-3-319-78503-5_4 chapter  google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186. association for computational linguistics, minneapolis, minnesota (june 2019). https://doi.org/10.18653/v1/n19-1423, https://www.aclweb.org/anthology/n19-1423 dias, l.b., duran, e.c.m.: análise das evoluções de enfermagem contextualizadas no processo de enfermagem. revista de enfermagem ufpe on line (2018). https://doi.org/10.5205/1981-8963-v12i11a234623p2952-2960-2018 article  google scholar  garritano, c.r.d.o., junqueira, f.h., lorosa, e.f.s., fujimoto, m.s., martins, w.h.a.: avaliação do prontuário médico de um hospital universitário. revista brasileira de educação médica (2020). https://doi.org/10.1590/1981-5271v44.1-20190123 guillou, p.: portuguese bert base cased qa (question answering), finetuned on squad v1.1 (2021). https://huggingface.co/pierreguillou/bert-base-cased-squad-v1.1-portuguese jeong, m., et al.: transferability of natural language inference to biomedical question answering. corr abs/2007.00217 (2020). https://arxiv.org/abs/2007.00217 jin, q., dhingra, b., liu, z., cohen, w.w., lu, x.: pubmedqa: a dataset for biomedical research question answering. in: emnlp-ijcnlp 2019–2019 conference on empirical methods in natural language processing and 9th international joint conference on natural language processing, proceedings of the conference (2020). https://doi.org/10.18653/v1/d19-1259 krallinger, m., krithara, a., nentidis, a., paliouras, g., villegas, m.: bioasq at clef2020: large-scale biomedical semantic indexing and question answering. in: jose, j.m., et al. (eds.) ecir 2020. lncs, vol. 12036, pp. 550–556. springer, cham (2020). https://doi.org/10.1007/978-3-030-45442-5_71 chapter  google scholar  mutabazi, e., ni, j., tang, g., cao, w.: a review on medical textual question answering systems based on deep learning approaches. appl. sci. 11(12) (2021). https://doi.org/10.3390/app11125456, https://www.mdpi.com/2076-3417/11/12/5456 e oliveira, l.e.s., et al.: semclinbr a multi institutional and multi specialty semantically annotated corpus for portuguese clinical nlp tasks (2020). https://arxiv.org/abs/2001.10071 pampari, a., raghavan, p., liang, j., peng, j.: emrqa: a large corpus for question answering on electronic medical records. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2357–2368. association for computational linguistics, brussels, belgium (october-november 2018). https://doi.org/10.18653/v1/d18-1258, https://aclanthology.org/d18-1258 qiu, x.p., sun, t.x., xu, y.g., shao, y.f., dai, n., huang, x.j.: pre-trained models for natural language processing: a survey. sci. chin. technol. sci. 63(10), 1872–1897 (2020). https://doi.org/10.1007/s11431-020-1647-3 article  google scholar  rajpurkar, p., zhang, j., lopyrev, k., liang, p.: squad: 100,000+ questions for machine comprehension of text. in: emnlp 2016 conference on empirical methods in natural language processing, proceedings (2016). https://doi.org/10.18653/v1/d16-1264 schneider, e.t.r., et al.: biobertpt a portuguese neural language model for clinical named entity recognition (2020). https://doi.org/10.18653/v1/2020.clinicalnlp-1.7 soni, s., roberts, k.: evaluation of dataset selection for pre-training and fine-tuning transformer language models for clinical question answering. in: lrec 2020–12th international conference on language resources and evaluation, conference proceedings (2020) google scholar  souza, j.v.a.d., et al.: a multilabel approach to portuguese clinical named entity recognition. j. health inf. 12 (2021). http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/article/view/840. http://www.jhi-sbis.saude.ws/ojs-jhi/index.php/jhi-sbis/issue/view/98/showtoc šuster, s., daelemans, w.: clicr: a dataset of clinical case reports for machine reading comprehension. in: proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long papers), pp. 1551–1563. association for computational linguistics, new orleans, louisiana (june 2018). https://doi.org/10.18653/v1/n18-1140, https://aclanthology.org/n18-1140 vaswani, a., et al.: attention is all you need. in: proceedings of the 31st international conference on neural information processing systems, nips 2017, pp. 6000–6010 (2017) google scholar  wiese, g., weissenborn, d., neves, m.: neural domain adaptation for biomedical question answering. in: conll 2017–21st conference on computational natural language learning, proceedings (2017). https://doi.org/10.18653/v1/k17-1029 wolf, t., et al.: transformers: state-of-the-art natural language processing. in: proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45. association for computational linguistics (october 2020). https://www.aclweb.org/anthology/2020.emnlp-demos.6 yoon, w., lee, j., kim, d., jeong, m., kang, j.: pre-trained language model for biomedical question answering. in: communications in computer and information science (2020). https://doi.org/10.1007/978-3-030-43887-6_64 yue, x., zhang, x.f., sun, h.: annotated question-answer pairs for clinical notes in the mimic-iii database (2021). https://doi.org/10.13026/j0y6-bw05, https://physionet.org/content/mimic-iii-question-answer/1.0.0/ download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations pontifícia universidade católica do paraná, curitiba, brazil lucas emanuel silva e oliveira, elisa terumi rubel schneider, yohan bonescki gumiel, mayara aparecida passaura da luz, emerson cabrera paraiso & claudia moro comsentimento, nlp lab, são paulo, brazil lucas emanuel silva e oliveira authors lucas emanuel silva e oliveiraview author publications search author on:pubmed google scholar elisa terumi rubel schneiderview author publications search author on:pubmed google scholar yohan bonescki gumielview author publications search author on:pubmed google scholar mayara aparecida passaura da luzview author publications search author on:pubmed google scholar emerson cabrera paraisoview author publications search author on:pubmed google scholar claudia moroview author publications search author on:pubmed google scholar corresponding author correspondence to lucas emanuel silva e oliveira . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper oliveira, l.e.s.e., schneider, e.t.r., gumiel, y.b., luz, m.a.p.d., paraiso, e.c., moro, c. (2021). experiments on portuguese clinical question answering. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_10 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords question answering electronic health records transfer learning publish with us policies and ethics profiles emerson cabrera paraiso view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies conference paper first online: 28 november 2021 pp 104–120 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies download book pdf download book epub nádia f. f. da silva10,11, marília costa r. silva10, fabíola s. f. pereira10,12, joão pedro m. tarrega10, joão vitor p. beinotti10, márcio fonseca13, francisco edmundo de andrade13 & … andré c. p. de l. f. de carvalho13  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1550 accesses 22 citations abstract the popular participation in law-making is an important resource in the evolution of democracy and direct legislation. the amount of legislative documents produced within the past decade has risen dramatically, making it difficult for law practitioners to attend to legislation and still listen to the opinion of the citizens. this work focuses on the use of topic models for summarizing and visualizing brazilian comments about legislation (bills). in this paper, we provide a qualitative evaluation from a legal expert and compare it with the topics predicted by our model. for such, we designed a specific sentence embedding technique able to induce models for portuguese texts, and we used these models as topic model, obtaining very good results. we experimentally compared our proposal with other techniques for multilingual sentence embeddings, evaluating them in three topical corpora prepared by us, two of them annotated by a specialist and the other automatically annotated by hashtags. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others using topic modeling in classification of brazilian lawsuits chapter © 2022 ulyssesner-br: a corpus of brazilian legislative documents for named entity recognition chapter © 2022 back to the feature, in entailment detection and similarity measurement for portuguese chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. discourse research discourse analysis information model legislative politics political communication portuguese 1 introduction the popular participation in the definition of public policy is inherent to a democratic state, and it comprises more than electoral processes. the information and communication technologies (icts) provide a means for citizens to manifest their options in social media and government platforms [31]. furthermore, the citizens can effectively contribute to better public policies and government decision-making [41]. in brazil, the chamber of deputies has an online platform that enables all brazilian citizens to interact and express their opinions concerning bills being discussed by the parliament [12]. one of the main sources of public opinion data are the social engagement platforms, such as twitterfootnote 1. in 2019, approximately 500 million tweets were daily generated in the world [47]. relying exclusively on human judgment as a tool for the analysis of very large public opinion data streams regarding the brazilian e-democracy is not feasible. the design and use of new computational tools able to extract relevant and useful information from the opinions of the population regarding new bills can result in more accepted and embraced legislation. as means to reliably process opinions, and ensure that citizens have their freedom of speech respected and equally considered, text analytic techniques [18] emerge as tools for the e-democracy [17]. it is important to observe that the objects (opinions) received in these data streams are mostly unlabeled. ergo, unsupervised and semi-supervised learning approaches, such as clustering and topic models algorithms, can be more suitable to extract relevant information from these data. text clusterization and topic modeling techniques, which are often used in text mining, have been frequently adopted for text data analysis [17]. previous studies, such as evangelopoulos and visinescu [17], successfully applied clustering algorithms for topic modeling using the analysis of citizens’ opinions in the e-democracy. topic modeling techniques can extract from texts information to support semantic analyses. this paper investigates a similar application, using topic modeling as the core method to extract key information about the brazilians’ political opinions for any proposed legislation publicly available in stances in the chamber of deputies online platform. this work focuses on a very specific instrument of citizen participation in the law-making process: comments related to the bills, in particular, those collected from the polls in the chamber of deputies web portal (refer to sect. 3 for details). while there is a vast literature exploring the analysis of social media influence in politics [5, 30], our comments dataset is more structured and focused on specific bills. the research reported in this paper has the following contributions: we adapted the bertopic topic mining tool [21] to extract topics from political comments from [11]; we applied an unsupervised deep learning based embedding technique [19] to train a brazilian portuguese legislative sentence based-embedding, and this resulting model is available at https://github.com/nadiafelix/bracis2021; we compared the performance of bertopic by leveraging portuguese and multilingue sentence embedding models. the sentence embeddings are used by topic models as input features in the clustering approach; we described and made available three topical corpora, two of them annotated by a specialist and the other automatically annotated by hashtags; we performed a dynamic topic modeling presenting a timeline of evolution; we evaluated a topic model, before evaluated only for english, for brazilian portuguese corpora. our work, to the best of our knowledge, is the first open source approach in brazilian portuguese to analyze user-generated content from the parliament platform about bills. this work is organized as follows. the second section provides an overview about related works. section 3 outlines the corpora and exploratory analyses. the fourth section introduces the methodology. section 5 presents the experimental evaluation. the following section comprises the discussions and concludes the work, summarizing the findings, advantages, limitations, contributions and research opportunities. 2 related work legal text analysis. recent advances in natural language processing techniques led to a surge in the application of text analysis in legal domains. in private law practice, the legal analytics tools impact areas such as legal research, document retrieval, contract review, automatic document generation, and custom legal advice [14, 39]. recent surveys indicate that the adoption of these tools results is not only in a significant reduction in time to deliver the services but also improves the predictability and trust in the litigation process [3]. beyond assisting legal practitioners, textual analysis has been increasingly adopted in the legislative process. before a law comes into force, a bill (law proposal) is subject to extensive technical analysis and discussion, which often allows the participation of civil organizations and the general public. during this process, a plethora of textual documents is generated, including technical reports, transcribed speeches, project amendments, among others. the availability of these documents in open formats [10] spurred the development of approaches to extract insights including the analysis of parliamentary debates [1] and prediction of bill enactment probabilities [27]. among those legislative documents, this work focuses on a very specific instrument of citizen participation in the lawmaking process: comments related to the law proposals, in particular, those collected from the polls in the chamber of deputies web portal (refer to sect. 3 for details). while there is a vast literature exploring the analysis of social media influence in politics [5, 30], our comments dataset is more structured and focused on specific bills. text clustering. text clusterization is a data mining task that consists of grouping instances according to their characteristics, and the curse of dimensionality also jeopardizes the clustering performance [28]. however, short texts, such as tweets, lead to more sparse vectors, which may require compression. the clusterization has hierarchical approaches [45]. the first is top-down and consists of an initial cluster with the entire dataset that is successively divided [2]. the bottom-up approach considers at first that each instance constitutes a cluster, and the most similar (the algorithm sets up first similarity measures) instances are agglomerated [2]. the distance-based clusterization, such as kmeans, can start with k clusters, each one with one centroid, and these centroids gather the other instances according to a distance metric [6], such as cosine similarity, euclidean distance, jaccard distance and so forth. finally, density-based methods (such as density-based spatial clustering of applications with noise (dbscan)) start clustering the instances that have more neighbors [40]. topic models and text clustering have analogous purposes. both of them aim to group text instances based on their similarity [24, 26]. nonetheless, whereas clustering algorithms mostly have a deterministic approach, topic models premises include stochastic assumptions [2]. moreover, interpretable topic models extract from each group the latent semantic-based n-grams that most represent the topic [26], in contrast to clustering algorithms that only gather instances without summarizing the inherent characteristics of each group. topic modeling. topic models are widely used to analyze large text collections. they represent a suite of unsupervised learning algorithms whose purpose is to discover the thematic structure in a collection of documents. several researchers have studied topic modeling. originally, probabilistic latent semantic indexing [23] and latent semantic indexing [29] paved the way for the creation of the latent dirichlet allocation (lda) [7] which is the most commonly implemented topic modeling algorithm in use today. in recent years, researchers have started to develop topic modeling algorithms that have achieved state of the art performance by leveraging word embeddings [15, 25, 34]. lda2vec [26], embedded topic model (etm) [16], and deep latent dirichlet allocation (dlda) [13] are three examples of topic models that extend lda. the lda2vec is an unsupervised topic model, which combines latent dirichlet allocation and word embeddings (word2vec skipgram) [26]. this method generates context vectors combining word vectors and document vectors [26]. the second model incorporates word embeddings and lda [16]. the etm keeps its performance even in large vocabularies. in addition, the topics are represented as vectors in the semantic space, and the topic embeddings generate context vectors. this method leads to interpretable topics, regardless the inclusion or removal of stopwords. the third work, dlda, uses sample strategies that provide scalability to single or multilayered methods. besides, the dlda can have multiple learning rates (based on topics and layers), and is a generalization of lda. in contrast to the two previous models, the dlda uses count vectors. the three models lose semantic context due to the embedding approach. this issue is partially solved by methods that consider sentence embeddings such as bert [15]. top2vec [4] and bertopic [21] proposed the use of distributed representations of documents and words due to their ability to capture the semantics of words and documents. they emphasize that before their approaches it was necessary to have the number of topics as input to the topic model algorithms, and with top2vec and bertopic, this parameter is found by their approaches. the first work proposes that topic vectors be jointly embedded with the document and word vectors with distance between them representing semantic similarity. bertopic is a topic modeling technique that leverages sentences based bert embeddings [15] and a class-based tf-idf [36] to create dense clusters allowing for interpretable topics whilst keeping important words in the topic descriptions. as this paper focuses on topic models, we compare these previous described in table 1 (the columns #topics and dynamic express whether the approach uses the number of topics and dynamic topic modeling, respectively). this table shows, e.g., that bertopic [21] can use different topic modeling techniques (value “yes”, in the column #topics) and uses dynamic topic modeling (value “yes”, in the column dynamic), a collection of techniques that analyze the evolution of topics along the time, generating the topic representations at each timestamp for each topic. these methods help to understand how a topic changes during a period of time. furthermore, it provides other forms of topic modeling, not explored in this study, such as semi-supervised learning. table 1. topic model approaches. \(^*\)hierarchical density-based spatial clustering of applications with noise [8]full size table 3 portuguese political comments the chamber of deputies board of innovation and information technology provided the dataset used in this study, which is available at [11]. this dataset consists of information classified by the user commentary for the bill, including counter of comments, date, opinion on the bill (positive or negative) and the commented proposition. the user can make its participation through the website [11]. in order to organize the commentaries, since there are several law projects and constitutional amendment propositions, the user can choose a specific proposition, whether his/her stand is for or against the legal document, as well as formulate the comment itself. a feature was implemented to “like” a certain commentary, simulating a social network, with the purpose of emphasizing the popularity of a certain point of view. an exploratory data analysis was performed on the dataset prior to the experimental evaluation. for every comment we calculated its word count, character count, and mean word length (characters per word). these metrics can be visualized in fig. 1. metadata with the number of likes of each comment, as well as its perception by the user as a positive or negative aspect of the legal document were also included in the analysis. the majority of comments have none or few likes, which is expected since the most liked comments are shown to the user in the voting page. as such, with more visibility this creates a snowball effect in which they attain more votes. this is seen in both bills and positive/negative opinions. fig. 1. textual metrics distribution in our corpora. the blue horizontal line represents the kernel density function approximation of the data. (color figure online) full size image in word and character count, the density distribution of the variables differs slightly in shape between the projects and opinions, although their medians remain close to each other. the mean word length presents a normal-like distribution in all categories. extracted comments with at least 2 words were kept in the corpora and had url links removed. 3.1 corpora for our experiments, we considered all comments associated with the two bills (pl 3723/2019 and pl 471/2005), and the set of all comments that have a hashtag in their content (hashtag corpus). below, we briefly detail these scenarios. pl 3723/2019. the pl 3723/2019 bill changes the law no. 10.826, of december 22, 2003, which deals with registration, possession and sale of firearms and ammunition, and defines crimes. the pl 3723/2019 corpus was collected from national congress pollsfootnote 2, in which the citizens are asked about the positive and negative parts of this bill. this corpus is composed of 195 negative and 530 positive comments. for our experiments, we considered the total number of comments (i.e., 725 comments). pec 471/2005. the pl 471/2005 bill establishes the effectiveness for the current responsible and substitutes for the notary services, invested according to the law. known as “pec dos cartórios”, the pl 471/2005 corpus was collected from national congress pollsfootnote 3, in which the citizens were asked about the positive and negative parts of this bill. this corpus is composed of 383 negative and 205 positive comments. for our experiments, we considered the total number of comments, i.e., 588 comments. hashtag corpus. this corpus is composed of 522 comments, which contain at least 16 comments with hashtags [11]. there are 189 bills with hashtags. we considered the top ten bills with more comments. the top ten bills with more comments and hashtags are: pec 32/2020, pl 318/2021, pl 3019/2020, pdl 232/2019, pl 2893/2019, pec 135/2019, pec 101/2003, pl 4425/2020, pec 6/2019, and pl 1577/2019. more details are shown in table 2. table 2. hashtag corpusfull size table 4 methodology we summarize the methodology for topic modeling in fig. 2. first we obtain the comments of citizens about a bill, next we chose a sentence embedding that will convert the comments into a vector representation. next, the clustering algorithm takes care of obtaining clusters based on spatial density. fig. 2. methodology for topic modeling. \(*\)tf-idf (term frequency-inverse document frequency). full size image 4.1 sentence embeddings a common method to address clustering and semantic search is to map each sentence to a vector space as semantically similar sentences are close. researchers have started to input individual sentences into bert [15] and to derive fixed-length sentence embeddings. the most commonly used approach is to average the bert output layer (known as bert embeddings) or by using the output of the first token (the [cls] token). as it was shown by [33], this practice yields rather bad sentence embeddings, often worse than averaging static word embeddings (for example, glove [32]). in this work, we evaluate three sentence embeddings: (i) universal sentence encoder [9]; (ii) multilingual universal sentence encoder for semantic retrieval (distiluse-base-multilingual-cased-v2) [48]; and (iii) a portuguese legislative sentence embedding [19] trained by us. (i) universal sentence encode (use)footnote 4: it is an encoder of greater-than-word length text trained on a variety of data [9], in a variety of languages (arabic, chinese-simplified, chinese-traditional, english, french, german, italian, japanese, korean, dutch, polish, portuguese, spanish, thai, turkish, russian) i.e., it is a cross-lingual sentence representation. unsupervised training data for the sentence encoding models are drawn from web sources of wikipediafootnote 5, web news, web question-answer pages and discussion forums. the authors mention that augment unsupervised learning with training on supervised data from the stanford natural language inference corpus (snli)footnote 6. (ii) multilingual universal sentence encoder for semantic retrieval (distiluse-base-multilingual-cased-v2): it is a multilingual knowledge distilled version of the multilingual universal sentence encoder. this version supports 50+ languages, including portuguese. training data consists of mined question-answer pairs from online forums including redditfootnote 7, stackoverflowfootnote 8, yahooanswersfootnote 9, and translation pairs, multinli [46], an extensive corpus, which contains examples from multiple sources is also used. the number of mined question-answer pairs also varies across languages with a bias toward a handful of top tier languages. to balance training across languages, the authors used google’s translation system to translate snli to the other languages. (iii) portuguese legislative sentence embedding: we trained our sentence embedding using a contrastive learningfootnote 10 framework [19] coupled with the portuguese pre-trained language model named bertimbau [37]. this proposal advances the state-of-the-art sentence embeddings using only unsupervised learning, and the authors named simcse [19]. we use the same hyperparameters, and the training set is described next: (1) from [12], we used the column conteudo with 215,713 comments. we removed the comments from pl 3723/2019, pec 471/2005, and hashtag corpus, in order to avoid bias. (2) from [12], we also used 147,008 bills. from these projects, we used the summary field named txtementa and the project core text named txtexplicacaoementa. (3) from political speechesfootnote 11, we used 462,831 texts, specifically, we used the columns: sumario, textodiscurso, and indexacao. these corpora were segmented into sentences and concatenated, producing 2,307,426 sentences. 4.2 topic models bertopic [21] is a topic modeling technique that leverages sentence embeddings transformers and c-tf-idf, a class-based tf-idf (term frequency-inverse document frequency) to create dense clusters, allowing for interpretable topics whilst keeping important words in the topic descriptions. the algorithm makes an assumption that many semantically similar documents can indicate an underlying topic. this approach automatically finds the number of topics, no stop word lists are required, no need for stemming/lemmatization, and works on short text [21]. 5 experimental evaluation 5.1 quantity and quality evaluation the unsupervised nature of topic model makes the evaluation task difficult [43]. we use the topic coherence evaluation [38] to decide which sentence embedding has better interpretability. topic coherence evaluation. the goal of topic coherence is to automate the evaluation of the interpretability of latent topics and the underlying idea is rooted in the distributional hypothesis of linguistics [22] – words with similar meanings tend to occur in similar contexts [38]. there have been approaches that confirmed a positive correlation with human interpretability and have been applied to evaluate the effectiveness of advances in the field of topic modeling. in order to evaluate our models, we used the coherence measurement \(c_v\) from the literature [35]. \(c_v\) is the most accurate measure according to m. röder et al. [35], and is calculated as follows. the top n words of each topic are selected as the representation of the topic, denoted as w = \(\{w_1, . . . , w_n \}\). each word \(w_i\) is represented by an n-dimensional vector \(v(w_i) =\) \(\{npmi(w_i, w_j )\}_{j=1,...,n}\), where jth-entry is the normalized pointwise mutual information (npmi) between word \(w_i\) and \(w_j\), i.e. \(npmi(w_i, w_j) = \frac{log p(w_i,w_j)-log(p(w_i)p(w_j))}{-log p(w_i,w_j)}\). w is represented by the sum of all word vectors, \(v(w) = \sum _{j=1}^{n} v(w_j)\). the calculation of npmi between word \(w_i\) and \(w_j\) involves the marginal and joint probabilities \(p(w_i), p(w_j ), p(w_i, w_j )\). for each word \(w_i\), a pair is formed \((v(w_i), v(w))\). a cosine similarity measure \(\phi _i(v(w_i), v(w)) = \frac{v(w_i)^tv(w)}{\parallel v(w_i) \parallel \parallel v(w) \parallel }\) is then calculated for each pair. the final \(c_v\) score for the topic is the average of all \(\phi \)’s. human evaluation. each performance metric has distortions and limitations. this metric, in the context of unsupervised learning, could mislead results. thus, an expert also evaluated qualitatively the datasets. the specialist defined the key topics of the corpora without knowing how many are or without any clue. 5.2 setup we used the bertopic [21] topic modeling toolfootnote 12 with the default parameters, and n-gram ranging from 2 to 20. the n-gram range corresponds to the number of terms that composes each topic. we reported results with 5 runs, showing the mean and standard deviation for topics ranging from 2 to 46 in increments of 2. we trained our portuguese legislative sentence embedding based in simcsefootnote 13 with train batch size = 128, number of epochs = 1, max sequence length = 32, and portuguese pre-trained language model named bertimbau [37]. simcse is based on huggingface’s transformers packagefootnote 14, and we take 5e−5 as the learning rate with the unsupervised learning approach. 5.3 results coherence of topics. in first experiment, we analyze the number of topics versus \(c_v\) coherence score for the three corpora. as we can see on figs. 3a, 3b, and 3c, the portuguese legislative sentence embedding presents the best results in all experiments. we emphasize that although we analyze the coherence of topics as a function of the number of topics, this metric was not used to decide the optimal number of topics. this is because the clustering algorithm used to learn the model infers automatically the number of topics through the spatial density of the sentences represented by the embeddingsfootnote 15. fig. 3. bertopic coherence analysis full size image real and predicted topics. the second experiment carried out is the comparison of the topics inferred by the best bertopic configuration (with portuguese legislative sentence embedding) and the topics noted by the domain specialist. we pair the most similar ones in table 3 showing that the approach proposed by bertopic is indeed efficient. we report the real topics and respective predicted topics only for pl 3723/2019, for reasons of space in the paperfootnote 16. it is important to note that the topics raised by the expert are not the only and definitive ones. for this experiment, we asked her to raise the obvious topics. another observation is that the expert did not know how many topics were inferred by bertopic and did not have previous access to them, so that there is no bias in her answers. table 3. real topics and respective predicted topics for pl 3723/2019full size table example 1 defesa, pois como vigilante tenho o direto de defesa. trabalho com risco de vida. e também não estamos seguros. o governo não nos da segurança. the bertopic models the example 1 on fig. 4, in which one can see that topic 10 is more significant. the relevant keywords from this topic are ‘armas’, ‘não’, ‘direito’, ‘de fogo’, ‘cidadão’, ‘defesa’. the other topics are still relevant as: topic 8 is about ‘arma de fogo’, ‘uma arma de fogo’; topic 9 is about ‘advogados’, ‘categorias’, ‘risco’, ‘de risco’, ‘arma’, ‘porte de arma’. fig. 4. topics probability distribution full size image 6 discussion this work has presented a fully automated approach for identifying topics in brazilian comments about legislation (bills). the bertopic together with portuguese legislative sentence embedding got the best result. this work is an early indication as to how legal practitioners can identify salient and coherent topics using automatic topic modeling tools and how can this ensure greater citizen participation. in sentence embeddings context, we experiment an unsupervised training approach, which takes an input sentence and predicts itself in a contrastive objective, with only standard dropout used as noise [19]. this simple method works well, performing on par with previous supervised counterparts [19]. we intend to train other unsupervised models for portuguese in this legal domain, as in [44], evaluating in topic model, and in other tasks. regarding topic model experiments, we intend to analyze other coherence measures, as well as human evaluation. besides that, we intend to develop a protocol of annotation for the corpora, and with this identify the topics and have a more accurate evaluation with supervised measures [42]. notes 1.https://twitter.com/. 2.https://forms.camara.leg.br/ex/enquetes/2209381. 3.https://forms.camara.leg.br/ex/enquetes/304008. 4.https://tfhub.dev/google/universal-sentence-encoder-multilingual-large/1. 5.https://www.wikipedia.org/. 6.https://nlp.stanford.edu/projects/snli/. 7.https://www.reddit.com/. 8.http://stackoverflow.com/. 9.https://help.yahoo.com/kb/answers. 10.contrastive learning is a machine learning technique used to learn the general features of a dataset without labels by teaching the model which data points are similar or different [20]. 11.available at https://github.com/nadiafelix/bracis2021. 12.https://github.com/maartengr/bertopic. 13.https://github.com/princeton-nlp/simcse. 14.https://huggingface.co/. 15.to analyze the coherence in function of topics we enter with the number of topics. 16.we report the real topics and respective predicted topics for pec 471/2005 and hashtag corpus in https://github.com/nadiafelix/bracis2021. references abercrombie, g., batista-navarro, r.: sentiment and position-taking analysis of parliamentary debates: a systematic literature review. j. comput. soc. sci. 3(1), 245–270 (2020) article  google scholar  allahyari, m., et al.: a brief survey of text mining: classification, clustering and extraction techniques (2017) google scholar  andrade, m.d.d., rosa, b.d.c., pinto, e.r.g.d.c.: legal tech: analytics, inteligência artificial e as novas perspectivas para a prática da advocacia privada. revista direito gv 16(1) (2020) google scholar  angelov, d.: top2vec: distributed representations of topics (2020). https://arxiv.org/abs/2008.09470 barberá, p., rivero, g.: understanding the political representativeness of twitter users. soc. sci. comput. rev. 33(6), 712–729 (2015) article  google scholar  basu, s., banerjee, a., mooney, r.j.: active semi-supervision for pairwise constrained clustering. in: proceedings of the 2004 siam, international conference on data mining. society for industrial and applied mathematics, april 2004 google scholar  blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3, 993–1022 (2003) math  google scholar  campello, r.j.g.b., moulavi, d., zimek, a., sander, j.: hierarchical density estimates for data clustering, visualization, and outlier detection 10(1) (2015) google scholar  chidambaram, m., et al.: learning cross-lingual sentence representations via a multi-task dual-encoder model (2019) google scholar  câmara dos deputados: dados abertos da câmara dos deputados (open data of the chamber of deputies, when translated to english). https://dadosabertos.camara.leg.br (2021). accessed 8 june 2021 câmara dos deputados (brazilian chamber of deputies, when translated to english): enquetes (polls, when translated to english). https://www.camara.leg.br/enquetes. accessed 5 may 2021 câmara dos deputados (brazilian chamber of deputies, when translated to english): popular participation. https://www2.camara.leg.br/transparencia/servicos-ao-cidadao/participacao-popular (2021). accessed 5 may 2021 cong, y., chen, b., liu, h., zhou, m.: deep latent dirichlet allocation with topic-layer-adaptive stochastic gradient riemannian mcmc. in: proceedings of the 34th international conference on ml , vol. 70, pp. 864–873. icml 2017, jmlr.org (2017) google scholar  dale, r.: law and word order: nlp in legal tech. nat. lang. eng. 25(1), 211–217 (2019) article  google scholar  devlin, j., chang, m., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for comp. linguistics: human language tech, pp. 4171–4186. minnesota, june 2019 google scholar  dieng, a.b., ruiz, f.j.r., blei, d.m.: topic modeling in embedding spaces. transactions of the association for computational linguistics 8, 439–453 (2020) article  google scholar  evangelopoulos, n., visinescu, l.: text-mining the voice of the people. commun. acm 55(2), 62–69 (2012) article  google scholar  gandomi, a., haider, m.: beyond the hype: big data concepts, methods, and analytics. int. j. inf. manage. 35(2), 137–144 (2015) article  google scholar  gao, t., yao, x., chen, d.: simcse: simple contrastive learning of sentence embeddings. https://arxiv.org/abs/2104.08821 (2021). accessed 5 may 2021 giorgi, j., nitski, o., wang, b., bader, g.: declutr: deep contrastive learning for unsupervised textual representations (2021) google scholar  grootendorst, m.: bertopic: leveraging bert and c-tf-idf to create easily interpretable topics. https://doi.org/10.5281/zenodo.4381785, note = accessed 5 may 2021 harris, z.: word 10(2–3), 146–162 (1954) google scholar  hofmann, t.: probabilistic latent semantic indexing. in: proceedings of the 22nd annual international acm sigir conference on research and development in information retrieval. sigir 1999, pp. 50–57. association for computing machinery, new york, ny, usa (1999) google scholar  mahyoub, m., hind, j., woods, d., wong, c., hussain, a., aljumeily, d.: hierarchical text clustering and categorisation using a semi-supervised framework. in: 2019 12th international conference on developments in esystems engineering (dese). ieee, october 2019 google scholar  mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space (2013) google scholar  moody, c.e.: mixing dirichlet topic models and word embeddings to make lda2vec (2016) google scholar  nay, j.j.: predicting and understanding law-making with word vectors and an ensemble model. plos one 12(5), e0176999 (2017) google scholar  nebu, c.m., joseph, s.: semi-supervised clustering with soft labels. in: 2015 international conference on control communication & computing india (iccc). ieee, november 2015 google scholar  papadimitriou, c., raghavan, p., tamaki, h., vempala, s.: latent semantic indexing: a probabilistic analysis. j. comput. syst. sci. 61(2), 217–235 (2000) article  mathscinet  google scholar  parmelee, j.h., bichard, s.l.: politics and the twitter revolution: how tweets influence the relationship between political leaders and the public. lexington books, lanham (2011) google scholar  pavan, j.n.s., pinochet, l.h.c., de brelàz, g., dos santos júnior, d.l., ribeiro, d.m.n.m.: study of citizen engagement in the participation of elective mandate actions in the brazilian legislature: analysis of the use of political techs. cadernos ebape.br 18(3), 525–542, september 2020 google scholar  pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pp. 1532–1543. association for computational linguistics, doha, qatar, october 2014 google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: proceedings of the 2019 conference on empirical methods in nlp and the 9th international joint conference on nlp (emnlp-ijcnlp), pp. 3982–3992. acl, hong kong, china, november 2019 google scholar  reimers, n., gurevych, i.: making monolingual sentence embeddings multilingual using knowledge distillation. in: proceedings of the 2020 conference on empirical methods in nlp. acl, november 2020 google scholar  röder, m., both, a., hinneburg, a.: exploring the space of topic coherence measures. in: proceedings of the 8a acm international conference on web search and data mining, pp. 399–408. wsdm 2015, new york, ny, usa (2015) google scholar  sammut, c., webb, g.i. (eds.): tf-idf, pp. 986–987. springer, us, boston, ma (2010) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, 20–23 october 2020. (to appear) google scholar  stevens, k., kegelmeyer, p., andrzejewski, d., buttler, d.: exploring topic coherence over many models and many topics. in: proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 952–961. association for computational linguistics, jeju island, korea, july 2012 google scholar  sugathadasa, k., et al.: legal document retrieval using document vector embeddings and deep learning. in: arai, kohei, kapoor, supriya, bhatia, rahul (eds.) sai 2018. aisc, vol. 857, pp. 160–175. springer, cham (2019). https://doi.org/10.1007/978-3-030-01177-2_12 chapter  google scholar  thaiprayoon, s., unger, h., kubek, m.: graph and centroid-based word clustering. in: proceedings of the 4th international conference on nlp and information retrieval. acm, december 2020. https://doi.org/10.1145/3443279.3443290 united nations: inclusion and more public participation, will help forge better government policies: guterres. https://news.un.org/en/story/2020/09/1073742, september 2020. accessed 5 may 2021 vinh, n.x., epps, j., bailey, j.: information theoretic measures for clusterings comparison: is a correction for chance necessary? in: proceedings of the 26th annual international conference on machine learning, pp. 1073–1080. icml 2009. association for computing machinery, new york, ny, usa (2009) google scholar  wallach, h., murray, i., salakhutdinov, r., mimno, d.: evaluation methods for topic models. in: proceedings of the 26th annual international conference on ml (icml 2009), pp. 1105–1112. acm (2009) google scholar  wang, k., reimers, n., gurevych, i.: tsdae: using transformer-based sequential denoising auto-encoder for unsupervised sentence embedding learning (2021) google scholar  willett, p.: recent trends in hierarchic document clustering: a critical review. inf. process. manage. 24(5), 577–597 (1988) article  google scholar  williams, a., nangia, n., bowman, s.: a broad-coverage challenge corpus for sentence understanding through inference. in: proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, vol. 1, pp. 1112–1122. association for computational linguistics, new orleans, louisiana, june 2018 google scholar  world economic forum: how much data is generated each day? https://www.weforum.org/agenda/2019/04/how-much-data-is-generated-each-day-cf4bddf29f/ (2019). accessed 5 may 2021 yang, y., et al.: multilingual universal sentence encoder for semantic retrieval (2019) google scholar  download references author information authors and affiliations institute of mathematics and computer science, university of são paulo, são carlos, sp, 13566-590, brazil nádia f. f. da silva, marília costa r. silva, fabíola s. f. pereira, joão pedro m. tarrega & joão vitor p. beinotti institute of informatics, federal university of goiás, goiânia, go, brazil nádia f. f. da silva federal university of uberlândia, uberlândia, brazil fabíola s. f. pereira câmara dos deputados palácio do congresso nacional praça dos três poderes brasília, brasília, df, brazil márcio fonseca, francisco edmundo de andrade & andré c. p. de l. f. de carvalho authors nádia f. f. da silvaview author publications search author on:pubmed google scholar marília costa r. silvaview author publications search author on:pubmed google scholar fabíola s. f. pereiraview author publications search author on:pubmed google scholar joão pedro m. tarregaview author publications search author on:pubmed google scholar joão vitor p. beinottiview author publications search author on:pubmed google scholar márcio fonsecaview author publications search author on:pubmed google scholar francisco edmundo de andradeview author publications search author on:pubmed google scholar andré c. p. de l. f. de carvalhoview author publications search author on:pubmed google scholar corresponding author correspondence to nádia f. f. da silva . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silva, n.f.f.d. et al. (2021). evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_8 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords topic models language models natural language processing sentence embeddings publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a co-occurrence based approach for mining overlapped co-clusters in binary data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a co-occurrence based approach for mining overlapped co-clusters in binary data conference paper first online: 28 november 2021 pp 375–389 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a co-occurrence based approach for mining overlapped co-clusters in binary data download book pdf download book epub yuri santa rosa nassar dos santos  orcid: orcid.org/0000-0001-9371-427x10,11, rafael santiago  orcid: orcid.org/0000-0001-7033-125x10,11, raffaele perego  orcid: orcid.org/0000-0001-7189-472412, matheus henrique schaly10, luis otávio alvares  orcid: orcid.org/0000-0003-2372-499510, chiara renso  orcid: orcid.org/0000-0002-1763-296612 & … vania bogorny  orcid: orcid.org/0000-0002-0159-464310,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 739 accesses abstract co-clustering is a specific type of clustering that addresses the problem of simultaneously clustering objects and attributes of a data matrix. although general clustering techniques find non-overlapping co-clusters, finding possible overlaps between co-clusters can reveal embedded patterns in the data that the disjoint clusters cannot discover. the overlapping co-clustering approaches proposed in the literature focus on finding global overlapped co-clusters and they might overlook interesting local patterns that are not necessarily identified as global co-clusters. discovering such local co-clusters increases the granularity of the analysis, and therefore more specific patterns can be captured. this is the objective of the present paper, which proposes the new overlapped co-clustering (ococlus) method for finding overlapped co-clusters on binary data, including both global and local patterns. this is a non-exhaustive method based on the co-occurrence of attributes and objects in the data. another novelty of this method is that it is driven by an objective cost function that can automatically determine the number of co-clusters. we evaluate the proposed approach on publicly available datasets, both real and synthetic data, and compare the results with a number of baselines. our approach shows better results than the baseline methods on synthetic data and demonstrates its efficacy in real data. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others parameter-less tensor co-clustering chapter © 2019 co-clustering based exploratory analysis of mixed-type data tables chapter © 2019 co-clustering for object by variable data matrices chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. atomic and molecular clusters categorization data mining functional clustering methodology of data collection and processing multivariate analysis 1 introduction over the years, the task of clustering complex data has become more challenging since a high number of attributes can increase computational complexity and affect cluster consistency [17]. one way to deal with this complexity is to use the co-clustering approach, which simultaneously clusters objects (rows) and attributes (columns) in matrix data [5]. the focus of these methods relies on finding co-clusters, where each co-cluster is formed by a subset of objects and attributes that can represent a submatrix of a given matrix. co-clustering approaches use, in general, a non-overlapping strategy, which means that an element of a co-cluster can belong to only one co-cluster [1, 14]. however, in many real situations, an element can participate simultaneously in more than one co-cluster. for example, a movie could be both thriller and science-fiction, a song can be both rock and high-energy, etc. therefore, an overlapping strategy is important because it identifies intersections between co-clusters, revealing patterns that could be lost when using disjoint co-clustering. besides, the detection of overlapping co-clusters has proven to be challenging since it is not trivial to evaluate the clustering quality [8]. we notice that most of the overlapping co-clustering approaches proposed in the literature have two characteristics: (1) they discover global clusters, and (2) they tend to fit a specific application. examples can be found in text mining [2], bioinformatics [13], recommendation systems [15], and social network analysis [18], to name a few. in contrast, the works of fu et al. [4], li [7], whang et al. [16], and zhu et al. [19], not only focus on global clusters on binary datasets, but they were designed for generic purposes. the limitation of these works is that they do not detect local co-clusters, i.e., refined groups formed by objects and attributes that identify an overlap pattern on global co-clusters. in this paper, we propose a new co-clustering method that combines simplicity of use with the capacity to extract overlapping global and local co-clusters. this method is named overlapped co-clustering (ococlus) and it is based on the co-occurrence of attributes and objects. the main novelty of this method is that, unlike the traditional overlapped co-clustering, it is able to infer a new type of patterns called local co-clusters. furthermore, ococlus is driven by an objective cost function that does not require the user-defined parameter of the number of co-clusters. in summary, we make the following contributions: (i) propose an incremental co-clustering approach that is application-independent and an algorithm that can find both overlapped and non-overlapped global and local co-clusters; (ii) use a cost function that, finds the number of co-clusters automatically, that ranks the co-clusters from the most relevant to the less relevant, and that finds overlapped co-clusters. the remainder of this work is organized as follows. the basic concepts definition of our work are presented in sect. 2. section 3 presents the works that are related to our proposal. section 4 presents the details of our method. section 5 presents the evaluation of the method with synthetic and real data. finally, the conclusion and further research directions of our work are presented in sect. 6. 2 basic concepts in this section we present the basic concepts to guide the reader throughout this paper. let d be a binary matrix with n rows (objects) and m columns (attributes). the element \(d_{ij}\) of d, where i and j are integers that \(1\le i \le n\) and \(1\le j \le m\), is equal to 1 if the j-th attribute occurs in the i-th object (true element); otherwise, it is 0. co-clustering is the grouping task of finding k (global) co-clusters in d where each co-cluster is formed by a subset of objects and attributes [11]. the subset of objects i can be represented as a binary vector of length n, where \(i_{i} = 1\) indicates that the i-th object is present in i. similar to that, a subset of attributes j with \(j_{j} = 1\) indicates that the j-attribute is present in j with length m. more formally, a co-cluster can be defined as follows: definition 1 co-cluster: let d be a binary matrix, i be the subset of objects, j be the subset of attributes; a co-cluster c is defined as \(c = \langle i,j \rangle \). the elements \(c_{ij}\) of co-cluster c are formed by the outer product of its subsets i and j (\(c \in \{0,1\}^{|i|\times |j|}\)). thus, a co-cluster c can represent a submatrix of d. such (global) co-cluster c can be formed with only the true elements in d or mixed with true elements and noise elements (\(d_{ij}=0\)). in this paper, the terms global co-cluster and co-cluster are interchangeably used. the co-occurrence between objects and attributes can form a co-cluster c which can be simplified by searching elements \(d_{ij}=1\) in the matrix d. furthermore, it can reduce the search space once the goal is to identify true co-occurrences. inspired by [9], we adapted four concepts for co-clustering problem: cost function \(\mathcal {f}_{p}\), pure co-cluster pc, noise thresholds \(\epsilon _{i}\) and \(\epsilon _{j}\), and expanded pure co-cluster ec. the cost function \(\mathcal {f}_{p}\) can be used to evaluate the process of forming a co-cluster. more formally, we can define the cost function \(\mathcal {f}_{p}\) as follows: definition 2 cost function: let \(c^{*}\) be a co-cluster, \(\prod \) be a set of global co-clusters, d be an input matrix, \(\rho \) be a weight of importance for the co-clusters cost, \(\mathcal {n}\) be a noise matrix, \(\gamma \) \(_c{^{*}}\) and \(\gamma \) \(_\mathcal {n}\) be user-defined functions for measuring the costs of co-clusters and noise; a cost function \(\mathcal {f}_{p}\) is defined as \(\mathcal {f}_{p}(\prod , \mathbf{d} ) = {\rho \times \sum _{c^{*}\in {\prod _{}}}} \gamma {_c{^{*}}}(c^{*}) + \gamma _{\mathcal {n}}(\mathcal {n})\). the objective is to minimize \(\mathcal {f}_{p}\) regarding \(\rho \), \(\gamma {_c{^{*}}}(c^{*})\) and \(\gamma _{\mathcal {n}}(\mathcal {n})\). such noise matrix \(\mathcal {n}\) used by [9] takes into account the false positives, false negatives, and the already covered elements in d. regarding the set of co-clusters \(\prod \) and matrix d, the false positives are elements \(d_{ij}=0\) covered by some pattern in \(\prod \), while false negatives are elements \(d_{ij}=1\) not covered by any pattern in \(\prod \). the concept of pure co-cluster simplifies the identification of a global co-cluster, which identifies a disjoint global co-cluster that contains only true elements. thus, we can define a pure co-cluster pc as follows: definition 3 pure co-cluster: let d be a binary matrix, \(d_{ij}\) be an element of d; a pure co-cluster \(pc = \langle pc_{j},\) \(pc_{i} \rangle \) is formed by a subset of objects \(pc_{i}\) and a subset of attributes \(pc_{j}\) that identifies only the true elements. thus, pc can represent a submatrix of d, which contains only the true elements \(d_{ij}=1\). a pure co-cluster pc can be expanded with noisy objects and attributes. we use two thresholds to control the amount of noise in a co-cluster: \(\epsilon _{i}\) for objects and \(\epsilon _{j}\) for attributes. thus, the noise thresholds \(\epsilon _{i}\) and \(\epsilon _{j}\) can be defined as follows: definition 4 noise thresholds: let \(c^{*}\) be a co-cluster, \(c^{*}_{j}\) and \(c^{*}_{i}\) be the subsets of attributes/objects that define \(c^{*}\); a maximum noise threshold for objects \(\epsilon _{i}\) and attributes \(\epsilon _{j}\) limit the amount of noise that can be included in \(c^{*}\). thus, each new object must be included in at least \((1-\epsilon _{i}) \times ||c^{*}_{j}||\) attributes of \(c^{*}\), while each new attribute must be included in at least \((1-\epsilon _{j}) \times ||c^{*}_{i}||\) objects of \(c^{*}\). the noise threshold value can range from [0, 1], where 0 does not allow any noise while 1 allows the maximum amount. the number of objects and attributes of a given subset is measured by the l\(^1\)-norm \(||\cdot ||\) (or hamming norm), which simply counts the number of bits 1 in the vector. from that, the expanded pure co-cluster ec represents a expanded version of pc with noise data. more formally, an expanded pure co-cluster ec can be defined as follows: definition 5 expanded pure co-cluster: let pc be the pure co-cluster, \(ec_{i}\) be a subset of objects, \(ec_{j}\) be a subset of attributes, \(\epsilon _{i}\) be the noise object threshold, \(\epsilon _{j}\) be the noise attribute threshold; an expanded pure co-cluster is defined as \(ec = \langle ec_{j},ec_{i} \rangle \), where \(ec_{j}\) and \(ec_{i}\) can contain new attributes and objects not present in pc regarding the noise thresholds \(\epsilon _{i}\) and \(\epsilon _{j}\). 3 related works because of the difficulty in finding co-clusters, there is no method widely accepted as the state-of-the-art; instead, there are algorithms that perform better in certain types of data than others. since a complete review is out of the scope of this paper, we shall briefly discuss some reference algorithms. for a comprehensive review of co-clustering algorithms, we refer to [12]. dhillon [3] used the matrix decomposition using the eigenvectors combined with bipartite graph to find global co-clusters in a real-valued matrix. it needs to know the number of co-clusters a priori and the order of the discovered co-clusters is not important. furthermore, it uses a support matrix to include some attributes as noise data; however, it does not have any noise-parameter to control the number of objects or attributes as noise data. kluger et al. [6] extended the dhillon [3] approach by using the singular value decomposition to find global co-clusters. it assumes that the data have a checkerboard structure in the matrix. this approach includes each element of a matrix into one co-cluster without overlap; therefore, it cannot control the noise data. fu et al. [4] proposed a bayesian-based overlapping co-clustering approach based on a multivariate distribution to find global co-clusters in a binary data matrix. it assumes that the number of co-clusters is known a priori. this approach does not indicate that the order of the discovered co-clusters is important. it tolerates noisy elements in the co-clusters; however, it does not have a noise-parameter to control the number of objects or attributes included in the co-cluster as noise. zhu et al. [19] proposed an overlapping co-clustering approach to approximate a binary data matrix with the sum of identified global co-clusters. this method needs to have the number of co-clusters a priori. furthermore, it does not deal with noise data and does not associate any importance for the co-cluster that explains the discovered order. lucchese et al. [9] proposed a frequent pattern mining method for binary datasets. the patterns are formed by sets of attributes and objects, where they can represent a non-overlapped global co-cluster. it uses a generalized cost function to drive the mining process to find the number of patterns automatically. the discovered order of the patterns is relevant regarding the cost function; therefore, it can be seen as a ranking. finally, two noise thresholds control the number of noisy attributes and objects included in a pattern. whang et al. [16] modeled the input data as a bipartite graph to find global overlapping co-clusters in binary data. this method allows to include noise objects in the co-clusters with a probability distribution function that models the noise. however, it does not define noise thresholds to control the number of noise objects and attributes. it can automatically infer the number of co-clusters, besides that, the method does not consider that the order of the discovered co-clusters is relevant in the process. li [7] presented a generalized overlapped co-clustering approach that uses singular value decomposition on the binary data matrix to identify global co-clusters. this method does not include noise automatically or by a user-defined noise threshold; it searches for homogeneous co-clusters without noise. furthermore, it can infer the number of co-clusters; however, it is not guaranteed to converge to the optimum number. finally, the method does not show that the discovered order of these co-clusters is relevant to it. 4 the overlapped co-clustering approach in this section we present a new method called ococlus (overlapped co-clustering) for finding overlapped co-clusters in a binary dataset by identifying both global and local co-clusters. ococlus searches for the co-occurrences between attributes and objects to identify co-clusters where a cost function drives the co-clustering process. in the following we present the method definitions in sect. 4.1 and the proposal details in sect. 4.2. 4.1 method definitions local co-clusters are patterns in the data related to specific characteristics that are overlooked by global co-clusters since it finds clusters which are in the intersection of objects and attributes of the global clusters. thus, we formally define a local co-cluster lc as follows: definition 6 local co-cluster: let \(\prod \) be the set of co-clusters, \(lc_{i}\) be a subset of objects, \(lc_{j}\) be a subset of attributes, c be the co-cluster in \(\prod \); a local co-cluster is defined as \(lc = \langle lc_{i},lc_{j}\rangle \), where the object intersections of co-cluster c with the co-clusters in \(\prod \) forms \(lc_{i}\), and the union of the attributes between c and the intersected co-clusters in \(\prod \) forms \(lc_{j}\). we propose a new cost function \(\mathcal {f}\) designed to make the overlapping and non-overlapping co-clusters equally important including both global and local co-cluster. the difference between the new cost function in definition 7 and the cost function given in definition 2 is that the new cost function considers just the size of the co-cluster and the quantity of noise that can be included in the co-cluster. however, the cost function in definition 2 weights the relevance of the patterns regarding its size, it penalizes the patterns that cover an element already covered and does not include an element into the expected pattern. from that, we define the new cost function \(\mathcal {f}\) as follows: definition 7 cost function: let \(\prod \) be the set of co-clusters, d be the binary matrix, \(c^{*}\) be the co-cluster, \(||c^{*}_{j}||\) and \(||c^{*}_{i}||\) be the size of the subsets of attributes/objects that define \(c^{*}\), \(\mathcal {n}\) be the number of noise elements included in \(c^{*}\) (\(d_{ij}=0\)), and h be the part that does not consider noise data; a cost function \(\mathcal {f}\) is defined as \(\mathcal {f}(\prod , \mathbf{d} ) = h + \mathcal {n}\), where \(h = {\sum _{c^{*}\in {\prod }}} (||c^{*}_{i}|| + ||c^{*}_{j}||) (||c^{*}_{i}|| \times ||c^{*}_{j}||)\). thus, the objective is to minimize \(\mathcal {f}\) regarding \(c^{*}_{i}\), \(c^{*}_{j}\) and \(\mathcal {n}\). regarding the new cost function \(\mathcal {f}\), part h contributes to the cost function evaluating co-clusters without noise, while part \(\mathcal {n}\) contributes by allowing some noise data regarding the maximum noise thresholds. once we have already formalized the main definitions, it is simple to define the overlapped co-cluster used to represent the global and local patterns as follows: definition 8 overlapped co-cluster: let \(\prod \) be the set of co-clusters, x be the co-cluster \(\in \prod \) with its subset of attributes \(x_{j}\) and objects \(x_{i}\), op be the set of co-clusters \(\in \prod \) that intersect \(x_{i}\), and \(op_{j}\) and \(op_{i}\) be the subset of attributes and objects of op; an overlapped co-cluster is formally defined as \(oc = \langle x_{j} \cup op_{j}, x_{i} \cap op_{i}\rangle \). considering the definition 8, the subset of objects of \(oc_{i}\) is formed by the nested intersection of objects between \(x_{i}\) and \(op_{i}\) (\(x_{i} \cap op_{i}\)), and the subset of attributes \(oc_{j}\) by joining the attributes of \(x_{j}\) with \(op_{j}\) (\(x_{j} \cup op_{j}\)). 4.2 method description algorithm 1 is the main algorithm that organizes our approach. it receives four input parameters: the matrix  d, the number of co-clusters k, the object noise threshold \(\epsilon _{i}\) and the attribute noise threshold \(\epsilon _{j}\). as a result, it outputs a set of co-clusters \(\varphi \) which contain k co-clusters that can overlap. in algorithm 1, the set of co-clusters \(\prod \) is set as empty (line 1), and the residual matrix \(d_r\) is initiated with d, which is used to find uncovered co-clusters in d (line 2). the algorithm iterates over findpurecocluster (line 4) and expandpurecocluster (line 5) methods at most k times, where k is the maximum number of co-clusters (line 3). in findpurecocluster method, the attributes in \(d_{r}\) are sorted in descending order (from the most frequent to the least) and stored in a list s to maximize the probability of forming large co-clusters. therefore, the attributes in s are evaluated for being added to a co-cluster without backtracking reducing the search space. only the true elements in d forms the pure co-cluster pc regarding the attributes in s. with this, the number of objects and attributes that co-occur are used in the cost function \(\mathcal {f}\) stated in definition 7 to evaluate if the tested subsets of objects and attributes can minimize the cost function. the pc grows in the number of objects and attributes as long as the cost function \(\mathcal {f}\) is minimized. besides, some attributes cannot be used to form the pc, then these attributes are stored in an extension list e. the output is a pure co-cluster pc and an extension list of attributes e. in expandpurecocluster method, ococlus expands pc with new objects and attributes that allows noise data (line 5). with this, the expanded co-cluster ec is initiated with pc identified at line 4. then, the process is similar to findpurecocluster; however, at this part, the method checks if the inclusion does not exceed the maximum noise thresholds (definition 4) and improves the cost function \(\mathcal {f}\). this inclusion occurs in two steps. first, the method tries to include new objects that are not present in ec and does not modify the current attributes. second, it does not modify the current objects and tries to include the attributes stored in the extension list e one at a time without backtracking. if an attribute is included in ec, the process goes back to the first step and repeats both steps. we remark that each new object and attribute is included in ec if such inclusions respect both definition 4 and definition 7. this process is repeated while e is not empty. the expandpurecocluster returns an expanded co-cluster ec as the output. given the output of the expandedpurecocluster, if the new co-cluster ec minimizes the cost function \(\mathcal {f}\) of the model (line 6), it is added to \(\prod \) (line 9). however, if ec does not minimize the cost function \(\mathcal {f}\) of the model, even though the parameter k does not reach its maximum value, the algorithm stops searching for new co-clusters (line 7). besides, if the cost function \(\mathcal {f}\) is improved, the residual matrix \(d_r\) is then updated with ec (line 10). the updated residual matrix \(d_{r}\) is used in the next iteration to find new patterns in d that are not covered by any previous co-cluster. ococlus can find the number of co-clusters automatically whenever k is not given. however, if the user misspecify the value of k, then the true number of co-clusters may not be discovered. so far, \(\prod \) covers non-overlapped patterns in the data (line 9); hence, it cannot show which co-clusters share characteristics. therefore, ococlus refines these non-overlapped co-clusters to identify global and local overlapped co-clusters as stated in definition 8. with this, the findoverlap method (line 12) iterates over \(\prod \) to identify possible overlapped co-clusters by taking the nested object intersections between the co-clusters in \(\prod \). the nested intersection considers what is shared among all intersected co-clusters instead of a common intersection between pairs of co-clusters. if such a co-cluster intersection exists, the attributes of the co-clusters involved in the intersection are joined. the next step is to delete the redundant co-clusters, i.e., co-cluster totally covered by another co-cluster. from that, the findoverlap is a simple and effective method that allows ococlus to find both overlapped co-cluster structures. its simplicity and effectiveness become possible by exploring the nested intersections of objects and joining attributes separately regarding the co-clusters in \(\prod \). this process looks simple once the cost function \(\mathcal {f}\) already evaluated the disjoint co-clusters in the previous methods, but it effectively identifies overlapped co-clusters. at the end, findoverlap returns the set of non-overlapped and overlapped (if exist) co-clusters \(\varphi \). finally, algorithm 1 returns this set of non-overlapped and overlapped patterns including both global and local co-clusters (line 13). proposition 1 let k be the maximum number of non-overlapped co-clusters, n the total number of objects, m the total number of attributes, and p the number of overlapped co-clusters. the computational complexity of findpurecocluster method is o(mn), expandpurecocluster method is o(m(mn+n)) = o(m\(^2\)n), and findoverlap method is o(k\(^2\)+p\(^2\)). regarding the overall complexity of our algorithm, ococlus calls findpurecocluster and expandpurecocluster methods, then builds d\(_{r}\) for each of the k (or less) non-overlapped co-clusters and finalizes with the findoverlap method. thus, the computational complexity of the ococlus algorithm is o(km\(^2\)n + (k\(^2\)+p\(^2\))). 5 experimental evaluation we compare ococlusfootnote 1 with four publicly available methods presented in the related works to use as the baseline methods, which are: li [7], lucchese et al. [9], kluger et al. [6], and dhillon [3]. we include the works of dhillon and kluger et al. because they are consolidated approaches in the literature and publicly available as a package by scikit learnfootnote 2. considering their stable implementation, we selected these works once the overlapping baseline methods fail to find the embedded overlapped co-clusters. therefore, we include those non-overlapped methods in the baseline to compare such a co-clustering result. we used three synthetic datasets named synthetic-1 and synthetic-2, and synthetic-3, where we artificially embedded the co-clusters (patterns) to create the ground-truth datasets. furthermore, we also evaluated ococlus on two real-life datasets, named cal500footnote 3 and cv-19footnote 4, to show its efficacy in the real application scenario. all the experiments data and source code are made public. we performed the experiments in a machine with a processor intel i7-7700 3.6 ghz, 16 gb of memory, and os windows 10 64bits. furthermore, we ran 15 independent simulations for all methods on each synthetic dataset to compute the average and standard deviation of the evaluation metrics score. table 1 shows the main characteristics of the datasets used in the experiments. it shows the total number of objects and attributes, the sparsity in the data (percentage of zeros), and the number of co-clusters for the synthetic datasets. table 1. datasets description.full size table we use four evaluation metrics to assess the quality of the ococlus. first, we use the reconstruction error matrix to measure the difference between data input and found co-clusters given by rec\(_{error} = \) ||x \(\veebar \) y|| similar to [7]. in short, we take the sum of the element-wise xor (\(\veebar \)) between the input data matrix x (ground-truth) and the reconstructed matrix y regarding the found co-clusters to measure the quantity of false positives and false negatives. the clustering quality is better when the result of the rec\(_{error}\) is equal or close to zero. the other three metrics are omega index (overlapped version of ari measure), overlapped normalized mutual information (onmi) and overlapped f1 measure (f\(_{score}\)) [10]. for these three last measures, the clustering quality is better when the result is equal or close to one, where one is the maximum score. we decided to use these measures since our approach focuses on the overlapping problem and therefore the traditional measures like for example adjusted rand index (ari), normalized mutual information (nmi), and f\(_{score}\) are not suitable to capture the overlapping behaviour. 5.1 evaluation of ococlus with synthetic data to be fair with all methods, we set the number of co-clusters k according to the ground-truth shown in table 1. regarding the noise control used by lucchese et al. [9], we use three configurations (t\(_{1}\), t\(_{2}\) and t\(_{3}\)) of noise threshold parameters to assess its clustering result when the noise values change. the configuration t\(_{1}\) uses the object noise threshold \(\epsilon _{i} = 0\) and attribute noise threshold \(\epsilon _{j} = 0\). for the last two configurations t\(_{2}\) and t\(_{3}\), the noise values are the same used in lucchese et al. [9]. the configuration t\(_{2}\) uses \(\epsilon _{i}=0.5\) and \(\epsilon _{j}=0.8\), while configuration \(t_{3}\) uses \(\epsilon _{i}=1\) and \(\epsilon _{j}=1\). table 2. score of the evaluation metrics for the synthetic datasets.full size table table 2 shows respectively the average and standard deviation from the evaluation metrics for each method and synthetic dataset. it can be seen that ococlus obtained the best score result in all synthetic datasets; hence, it finds the embedded overlapped co-clusters. lucchese\(_{t_{1}}\) obtained the second best result while the other two configurations obtained the same score values because they identified the same co-clusters. the method proposed by li [7] obtained the worst result among the methods. this happens because the method sometimes does not converge to any co-cluster which makes its overall result worse than or close to the non-overlapped methods. besides, it can be seen in table 2 that the baseline methods do not find the real number of co-clusters once their evaluation scores do not reach the best value. regarding the non-overlapped methods, the method proposed by kluger et al. [6] shows the worse overall co-clustering result. meanwhile, the method of li [7] improved slightly its overall clustering result in synthetic-2 dataset compared to synthetic-1 and synthetic-3. however, it does not overcome the clustering results of the non-overlapped approaches at all. in summary, it can be seen in table 2 that ocuclus outperformed the baseline methods in all evaluation metrics. such a result occurs because ococlus identifies all global and local co-clusters while the baselines fail to find both co-cluster structures correctly in the data. the baselines focus on the global non-overlapped and overlapped structures. regarding the baseline methods, using the constraint t\(_{1}\) in the work of lucchese et al. [9], this configuration generated the best clustering result. however, considering the other two constraints, we notice that they do not improve the clustering result. furthermore, the method proposed by li [7] shows an overall worse clustering result, even though it improved its performance in the synthetic-2 dataset but not enough to overcome all methods. the methods of kluger et al. [6] and dhillon [3], in general, obtained stable results in comparison with li [7]. 5.2 real application scenario in this section, we used ococlus on two real datasets to demonstrate its general utility. we set the noise thresholds \(\epsilon _{i}\) and \(\epsilon _{j}\) to the minimum value, and this means that we are not allowing any attribute or object to be added as noise in the co-clusters. with this parameter control, it is possible to have a better understanding of the co-cluster structure. in fact, ococlus finds pure co-clusters when the noise thresholds are set to the minimum value (zero); this means that all attributes that occur in all objects do not include the presence of noise. music annotation. the left side of fig. 1 shows the bitmap of the cal500 dataset, and the right side shows the bitmap of the ococlus result. similar to li [7], the question is to identify song sets that share similar annotations. moreover, we are interested in finding which are the common annotations that distinguish each song set. this task can enhance our perception of the relationship between songs and annotations and therefore it can be applied to music retrieval and recommendation system. we used ococlus in the processed dataset and the main co-clusters are highlighted in the right side of fig. 1. fig. 1. cal500 music dataset. the left side shows the bitmap of the binary annotation matrix where objects represent songs and attributes represent annotations (black = presence; white = absence). the right side shows the bitmap of the identified co-clusters. objects and attributes are ordered in the same way in both figures just for visualization purpose. (best seen in color) (color figure online) full size image ococlus identified three main levels which are within the red lines and four main co-clusters. the two larger global co-clusters have the size 150 \(\times \) 13 and 100 \(\times \) 12. further, looking into these two co-clusters we found such annotations as “not-song-fast\(\_\)tempo”, “not-emotion-angry-aggressive”, “not-song-heavy\(\_\)beat” and “not-emotion-bizarre-weird” for the first co-cluster, and the “song-fast\(\_\)tempo”, “not-emotion-angry-aggressive”, “song-heavy\(\_\)beat” and “song-high\(\_\)energy” for the second co-cluster. the first co-cluster includes songs such as “for you and i” by 10cc, “three little birds” by bob marley and the weilers, and “i’ll be your baby tonight” by bob dylan. in the second co-cluster includes songs such as “trapped” by 2pac, “dirty deeds done dirt cheap” by ac/dc and “livin on a prayer” by bon jovi. considering the song attributes in the first cluster, it can be seen that songs are formed by a slow rhythm and without strong beats. the second cluster characterized songs with strong beats and a fast rhythm. therefore, the method identified clusters with opposite characteristics, showing two groups of users with different preferences. considering the local patterns, for instance, ococlus finds a co-cluster of songs as “summertime” by dj jazzy jeff and the fresh prince, “sunset 138 bpm remix” by dj markitos and “teenage shutdown” by electric frankenstein to name few songs that share some annotations as “song-texture\(\_\)electric”, “song-fast\(\_\)tempo” and “song-high\(\_\)energy”. the fourth local co-cluster consists of 45 songs and 7 music annotations. it characterizes a group formed by songs with drums, men on vocals, with electronic and acoustic parts. we identified the music genres like rock, pop music, pop rock, and alternative rock in this co-cluster. for instance, to name a few songs, this cluster has the “soul and fire” by sebadoh, “clocks” by coldplay, “tubthumping” by chumbawamba, “last goodbye” by jeff buckley, “november rain” by guns n ’roses, and “wonderful tonight” by eric clapton. thus, it can be seen that ococlus is useful for finding overlapped and non-overlapped co-clusters that identify the relationship between songs based on the semantic annotations. table 3. description of the clusters in the cv-19 dataset.full size table coronavirus information. table 3 shows the co-clusters with its attributes and the number of objects. the union of clusters g8 and g9 show the cluster of older people with a total of 1137. attribute senior_3 aggregates people 80 years old or above, and senior_2 aggregates people from 70 to 79 years old. once the senior_3 and heart disease attributes appear in g8, they are relevant to form this cluster with 913 people and no other attributes have improved the cost function for it. the g9 cluster has 224 people who died in august, which also happened in the g6 cluster (467 people) during july. these two months mark the peak of the winter season in the region where these people lived. the clusters g2, g3, g4, g7, and g10 can be seen as a group of people who had at least one main symptom of covid-19 associated with some comorbidity. the non-overlapped co-clusters are the first 10 groups in table 3. we notice that the male attribute is present in two clusters (g1 and g3) regarding the top four. cluster g1 identifies 1174 men who experienced the three main symptoms of covid-19. meanwhile, the female attribute is present in one cluster regarding the top four. it identifies 1066 women who presented dyspnea and other comorbidities as main attributes for this group. the identified overlapped co-clusters show details that are overlooked in disjoint co-clusters and these co-clusters are the last 8 groups (g11–g18) in table 3. for instance, cluster g11 identifies a group of 539 men that felt symptoms as dyspnea, fever, and cough and had heart disease problem. in the same way, cluster g12 identifies 410 men with symptoms as in g1, but now it has those with diabetic issues. in comparison, the overlapped cluster g13 identifies a group of 258 women with cough symptoms in combination with diabetes and other comorbidities. cluster g14 represents another pattern since it identifies 365 women with dyspnea and other symptoms in combination with other comorbidites. groups g11 and g12 are examples of global overlapped co-clusters, while the groups g15 and g17 are two different examples of local co-clusters. cluster g15 represents a group of 39 men from porto alegre region and lived in the capital, where they all felt the main symptoms of covid-19 and who had heart disease and diabetes problems. for group g17, characterizes a group of 41 older men over 80 years of age who had the main symptoms and who had heart disease and diabetes. regarding the local co-clusters, it can be seen that such co-clusters identify detailed patterns that are overlooked by global co-clusters. the experiment with the covid-19 dataset is an example of a real problem related to data analysis complexity. thus, it can be seen that each co-cluster reveals a meaning pattern according to the information granularity. 6 conclusion and future works we proposed ococlus, a new non-exhaustive overlapped co-clustering method for binary data, designed for general purpose analysis. ococlus is based on the detection of co-occurrence of objects and attributes, to identify global and local co-clusters that overlap. besides that, when there are no overlapped patterns in the dataset, ococlus can identify the non-overlapped co-clusters. furthermore, it is driven by a cost function to automatically identify the number of co-clusters. we performed experiments on synthetic and real data that demonstrates the efficacy and utility of our proposed method. ococlus found all embedded co-clusters in the synthetic datasets used as ground-truth, proven by the fact that ococlus obtained the maximum score in the evaluation metrics. such a result shows that ococlus outperformed the limitations of the baseline methods. nevertheless, we prove the usefulness of our method in two real datasets where we show that ococlus identified co-clusters that can represent meaningful patterns. we highlight the fact that the obtained results are interesting to propose new specialized systems that use the identified co-clusters as input to decision support systems. like any work in the literature, our approach also has space for improvements as future research. first, the number of co-clusters is driven by a cost function regarding the number of objects and attributes. then, the method tends to find rectangular clusters which may generate patterns with few attributes for big data mining. second, an interesting research direction is to adapt the method to deal with heterogeneous data. third, it may be interesting to set the noise thresholds \(\epsilon _{i}\) and \(\epsilon _{j}\) in a data-driven way finally, identifying uncorrelated co-clusters in the data matrix is another interesting direction to improve the method. notes 1.https://github.com/bigdata-ufsc/ococlus. 2.https://scikit-learn.org/stable/modules/biclustering.html. 3.http://mulan.sourceforge.net/datasets-mlc.html. 4.https://ti.saude.rs.gov.br/covid19/; just passed away people data were used. references affeldt, s., labiod, l., nadif, m.: ensemble block co-clustering: a unified framework for text data. in: proceedings of the 29th acm international conference on information & knowledge management, pp. 5–14 (2020) google scholar  brunialti, l.f., peres, s.m., da silva, v.f., de moraes lima, c.a.: the binovnmtf algorithm: overlapping columns co-clustering based on non-negative matrix tri-factorization. in: 2017 brazilian conference on intelligent systems (bracis), pp. 330–335. ieee, uberlandia, brazil (2017) google scholar  dhillon, i.s.: co-clustering documents and words using bipartite spectral graph partitioning. in: proceedings of the seventh acm sigkdd international conference on knowledge discovery and data mining, pp. 269–274. acm, association for computing machinery, new york (2001) google scholar  fu, q., banerjee, a.: bayesian overlapping subspace clustering. in: 2009 ninth ieee international conference on data mining, pp. 776–781. ieee (2009) google scholar  hartigan, j.a.: direct clustering of a data matrix. j. am. stat. assoc. 67(337), 123–129 (1972) article  google scholar  kluger, y., basri, r., chang, j.t., gerstein, m.: spectral biclustering of microarray data: coclustering genes and conditions. genome res. 13(4), 703–716 (2003) article  google scholar  li, g.: generalized co-clustering analysis via regularized alternating least squares. comput. stat. data anal. 150, 106989 (2020) google scholar  lucchese, c., orlando, s., perego, r.: a generative pattern model for mining binary datasets. in: proceedings of the 2010 acm symposium on applied computing, pp. 1109–1110. acm (2010) google scholar  lucchese, c., orlando, s., perego, r.: a unifying framework for mining approximate top-\( k \) binary patterns. ieee trans. knowl. data eng. 26(12), 2900–2913 (2013) article  google scholar  lutov, a., khayati, m., cudré-mauroux, p.: accuracy evaluation of overlapping and multi-resolution clustering algorithms on large datasets. in: 2019 ieee international conference on big data and smart computing (bigcomp), kyoto, japan. ieee (2019) google scholar  madeira, s.c., oliveira, a.l.: biclustering algorithms for biological data analysis: a survey. ieee/acm trans. comput. biol. bioinform. 1(1), 24–45 (2004) article  google scholar  padilha, v.a., campello, r.j.: a systematic comparative evaluation of biclustering techniques. bmc bioinform. 18(1), 1–25 (2017) article  google scholar  pio, g., ceci, m., d’elia, d., loglisci, c., malerba, d.: a novel biclustering algorithm for the discovery of meaningful biological correlations between micrornas and their target genes. bmc bioinform. 14(s7), s8 (2013) article  google scholar  role, f., morbieu, s., nadif, m.: coclust: a python package for co-clustering. j. stat. softw. 88(1), 1–29 (2019) google scholar  vlachos, m., dünner, c., heckel, r., vassiliadis, v.g., parnell, t., atasu, k.: addressing interpretability and cold-start in matrix factorization for recommender systems. ieee trans. knowl. data eng. 31(7), 1253–1266 (2018) article  google scholar  whang, j.j., rai, p., dhillon, i.s.: stochastic blockmodel with cluster overlap, relevance selection, and similarity-based smoothing. in: 2013 ieee 13th international conference on data mining. ieee (2013) google scholar  xu, r., wunsch, d.: survey of clustering algorithms. ieee trans. neural netw. 16(3), 645–678 (2005) article  google scholar  zheng, et al.: clustering social audiences in business information networks. pattern recognit. 100, 107126 (2020) google scholar  zhu, h., mateos, g., giannakis, g.b., sidiropoulos, n.d., banerjee, a.: sparsity-cognizant overlapping co-clustering for behavior inference in social networks. in: 2010 ieee international conference on acoustics, speech and signal processing, pp. 3534–3537. ieee (2010) google scholar  download references acknowledgements this work has been partially supported by capes (finance code 001), cnpq, fapesc (project match co-financing of h2020 projects grant 2018tr 1266), and the european union’s horizon 2020 research and innovation programme under ga n. 777695 (master). the views and opinions expressed in this paper are the sole responsibility of the author and do not necessarily reflect the views of the european commission. author information authors and affiliations universidade federal de santa catarina, florianópolis, brazil yuri santa rosa nassar dos santos, rafael santiago, matheus henrique schaly, luis otávio alvares & vania bogorny ine, programa de pós-graduação em ciência da computação, florianópolis, brazil yuri santa rosa nassar dos santos, rafael santiago & vania bogorny isti-cnr, pisa, italy raffaele perego & chiara renso authors yuri santa rosa nassar dos santosview author publications search author on:pubmed google scholar rafael santiagoview author publications search author on:pubmed google scholar raffaele peregoview author publications search author on:pubmed google scholar matheus henrique schalyview author publications search author on:pubmed google scholar luis otávio alvaresview author publications search author on:pubmed google scholar chiara rensoview author publications search author on:pubmed google scholar vania bogornyview author publications search author on:pubmed google scholar corresponding author correspondence to yuri santa rosa nassar dos santos . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper dos santos, y.s.r.n. et al. (2021). a co-occurrence based approach for mining overlapped co-clusters in binary data. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_25 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords co-clustering overlapped co-clusters binary data publish with us policies and ethics profiles yuri santa rosa nassar dos santos view author profile chiara renso view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature who killed the winograd schema challenge? | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper who killed the winograd schema challenge? conference paper first online: 12 october 2023 pp 211–225 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) who killed the winograd schema challenge? download book pdf download book epub hugo neri9 & fabio g. cozman9  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 543 accesses abstract in which we investigate the technical issues surrounding the defeat, or perhaps the sudden assassination, of the winograd schema challenge. we argue that, while the obvious suspect is the winogrande-based solution, the real cause of death was the masked language modeling technique for learning large language models. the winograd schema challenge was, in the end, just a test for masked language closure, and as such it was killed by the use of this technique at scale. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others distinguishing sensitive and insensitive options for the winograd schema challenge chapter © 2023 winoflexi: a crowdsourcing platform for the development of winograd schemas chapter © 2019 semantic noise in the winograd schema challenge of pronoun disambiguation article open access 11 april 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. applied linguistics computational linguistics formal languages and automata theory language processing language history machine learning 1 introduction the winograd schema challenge has been met; in fact, one might say that it has been defeated or, perhaps more dramatically, that it has had a violent death. this is a sad ending for a challenge once hailed as a leading alternative to the turing test [24,25,26,27]. a very detailed discussion of the defeat of the winograd schema challenge (wsc) has been produced by kocijan et al. [21]. it seems that very little can be added when it comes to describing the tools for the wsc. the “autopsy” by kocijan et al. also offers valuable insight as to the reasons why the wsc did not resist when attacked, and the reasons why the wsc was perhaps not a very good test for commonsense reasoning and for overall intelligence in the first place. still, it is fair to ask: what exactly killed the wsc? was it too weak that it in fact died of some common disease while still young? was it really murdered? if so, who did it, and what was the murder weapon? by reading commentary around the wsc, one feels that there is no consensus on the cause of death. how could a leading substitute for the venerable turing test die suddenly with so little understanding of the event? two previous references deserve attention. the most prominent one is the already mentioned analysis by kocijan et al. [21], whose authors have produced substantial contributions to the wsc [20, 22]. they seem to argue, and cite several colleagues who seem to agree with them, that the problem with the wsc was that the challenge was weaker than expected in a number of ways: schemas were harder to build than expected and were less meaningful than originally thought; schemas were more constrained than planned and subjected to large scale correlations that led to death in the hands of large language models such as bert and roberta. in short, they argue that the wsc was weak and that any specific challenge may be weak in capturing commonsense reasoning [21]. this is a far-reaching position, in which the large language model roberta killed the challenge in a context where the challenge was so weak that it almost did not deserve to live. hence, there is no prison time for roberta, but not much credit for it either. another analysis can be found in the original paper on the winogrande corpus, the corpus that was used to refine roberta so as to obtain human performance [42]. indeed, one might argue that the pair roberta \(+\) winogrande was the murderer. or one might argue that the winogrande corpus alone was the murderer, on the grounds that the real defeat came only when researchers realized that more data, even of low quality, could be used to refine a large language model to victory over the wsc. in any case, the authors of the winogrande corpus seem to believe that biases in corpora let language models excel in the wsc. for those authors, the achilles heel of the wsc was that, when building a corpus with questions and answers, human subjects introduce biases that, once captured by large language models, lead to the correct resolution of winograd schemas. they seem to argue that, if one could produce a very large unbiased corpus with winograd schemas, then the wsc with respect to that corpus would require commonsense. so the problem was not with the challenge, but with the difficulty in building the required corpus. (note that such difficulty was deemed by kocijan et al. to be a defect of the wsc, so there are divergent opinions even on the meaning of the difficulty.) in any case, the position by the authors of winogrande seems to be that the wsc is not dead but it may be in a frozen state to be resuscitated when we find a difficult enough corpus. we offer a different analysis. we agree with others that the roberta language model, and even the winogrande corpus, were key elements in this story; however, we think previous analysis has not been able to explain why is it that the wsc succumbed so easily and what exactly killed it — and we believe we have a credible answer to these questions. to develop our argument, we first describe the main elements of the wsc in sect. 2 and then we focus on roberta and the winogrande corpus in sect. 3. our solution to the puzzle surrounding the death of the wsc is elaborated in sects. 4, 5, and 6. our closing arguments are given in sect. 7. 2 the winograd schema challenge the wsc consists of a set of winograd schemas, where each schema is a pronoun disambiguation problem such as: the city councilmen refused the demonstrators a permit because they feared/advocated for violence. who are “they”? a winograd schema contains a pronoun that refers to one of two entities (in the example, city councilmen/demonstrators); the reference depends on a particular word. each schema can be instantiated in two ways by selecting this particular word (in the example, feared/advocated). then the resulting anaphora must be solved for each sentence. the average human performance in the challenge ranges from 92.1% [3] to 94% [43]. winograd schemas are devised to display no strong statistical correlations between words; hence, knowledge, and more importantly, commonsense knowledge should be needed to solve them [10]. hector levesque, who proposed the wsc, submitted to the ai community, at ijcai-16, when the challenge ran as an actual competition, that: “doing this [solving the challenge] correctly appears to rely on having a solid base of commonsense knowledge and the ability to reason intelligently with that knowledge” [24]. in fact, in the original paper about the wsc the following ambitious statements were given by levesque [24]: the claim of this paper in its strongest form might be this: with a very high probability, anything that answers correctly a series of these questions ...is thinking in the full-bodied sense we usually reserve for people. to defend this claim, however, we would have to defend a philosophical position that turing sought to avoid with his original turing test. so like turing, it is best to make a weaker claim: with a very high probability, anything that answers correctly is engaging in behaviour that we would say shows thinking in people. whether or not a subject that passes the test is really and truly thinking is the philosophical question that turing sidesteps. [emphases added] we may thus distinguish two different claims in levesque’s proposal: strong (ontological) claim: anything that solves the wsc is thinking in the full-bodied sense we usually reserve for people. weak (pragmatic) claim: anything that solves the wsc is engaging in behaviour that we would say shows thinking in people. in any case, the idea behind the wsc is to test the ability to answer commonsense questions as related to sentence comprehension. the research community followed this lead with various degrees of enthusiasm, taking the wsc as a new type of turing test [2, 15, 32,33,34, 51]. fig. 1. accuracy of systems on the winograd challenge [6]. full size image machines performed poorly up to 2017. that suddenly changed with language models, esp. gpt-2 (generative pre-trainedtransformer) [38], bert (bidirectional encoder representations from transformers) [12], and roberta (robustly optimized bert approach) [29]. figure 1 shows the astonishing evolution in accuracy. today we have (extremely) large language models that can solve a variety of linguistic tasks with high accuracy [1, 46]. we can say that the defeat of the wsc took place by the end of 2019 [44]. interestingly, high expectations were assigned to the wsc until its sudden defeat. for instance, marcus and davis insisted in their book for a non-specialized audience, in 2019, that the wsc is “one of the most challenging tests for machines that is currently available” [30]. the literature on the wsc is relatively small after the winogrande solution was widely disseminated in 2020 [42]. variants of the wsc have been proposed, and various applications have been tried — for instance, to test gender bias or to require explanations. such work is reviewed with exceptional acumen by kocijan et al. [21], so we do not repeat here what they have done in a definite fashion. 3 llms, roberta and winogrande: main suspects a language model usually yields probabilities over tokens of a language, given other tokens [18]. several language models resort to markov chains and similar probabilistic tools; recently, large language models have been built by training deep neural networks with properly designed architectures. there has been a shift up in the performance with the introduction of large language models (llms) based on the transformers architecture [50]. in short, a transformer is a neural network model that uses attention as it main mechanism; because this mechanism lends itself to parallelization, learning techniques can rely on larger training sets. besides parallelization, the self-attention mechanism encodes contextual information.footnote 1 the most popular llms, such as bert, t5, gpt, are transformers. transformer-based llms frequently achieve state-of-the-art results in tasks like question answering and sentiment analysis [17, 49]. llms rose to center stage in 2018 after trihn and le [48] employed rnn-based models. even though their models’ performance was relatively low (63.7%), they optimistically claimed that they had a good indication their model had a “good grasp of commonsense knowledge” [48]. they accepted that the better the performance in the challenge the more confidently one can say the model carries commonsense knowledge. significant evolution on the wsc was observed after thihn and le’s work, as one can see in fig. 1. an spectacular gain in performance was then reported with the publication of the winogrande corpus and associated solution for the wsc (first in arxiv [44] and then in the aaai-2020 conference [42]). at that point the wsc was defeated. winogrande is a dataset of schemas inspired by the original winograd schemas. the full version winogrande contained 43,972 problems while the debiased version contained 12,282 problems. the winogrande paper reported at the aaai-2020 attained 90.1% accuracy in the winograd schema challenge [43]. the content of that aaai-2020 paper was originally uploaded in november 2019 on arxiv [42]. however, there was a first version of this paper uploaded to arxiv on july 2019 which presented a significant lower performance, 77.6%, still the state-of-art at that moment [44]. basically, the difference in performance was produced by the roberta fine-tuned with winogrande in the second version, whereas the first version adopts bert fine-tuned with winogrande. comparing both versions, the sole reason for the performance difference is the adoption of roberta in the final version in contrast to bert in the first version. the winogrande team argued that the performance of existing systems could be due to the “extent to which spurious effects are prevalent in existing datasets, which run the risk of overestimating the true capabilities of machine intelligence on commonsense reasoning” [44]. the initial motivation for winogrande focused on dealing with biases that might exist in the hand-crafted winograd schema and that might thus unfairly help existing solutions. such biases might be of two types [47]: (a) language-based and (b) dataset-specific biases. in fact, trichelair et al. observed that at least 37 sentences in the wsc273 dataset (13.6%) are conceptually rather easy due to language associations [47]. for instance, take in the storm, the tree fell down and crashed through the roof of my house. now, i have to get it repaired/removed. and note that trees are removed more often than repaired, while roofs are repaired more often than removed. on the other hand, 131 sentences (out of 273) were then found to yield meaningful examples even if candidates in the sentence were switched. this was called “swichtability” [47]. as an example, take bob collapsed on the sidewalk. soon he saw carl coming to help. he was very ill/concerned. in this sentence, bob and carl can be switched to obtain an equivalent example with the opposite answers. such schemas were called “switchable”. trichelair et al. [2018] encouraged future researchers to additionally report results on the switchable dataset (when the candidates are switched, and when they are not). the winogrande team also discussed dataset-specific biases, such as annotation artifacts or spurious correlations in crowdsourced datasets. their solution was to develop an unbiased dataset (to be robust against both types of biases discussed above [44]), so presenting “problems that are more challenging by reducing such biases, while also scaling to a significantly larger number of problems (273 to 44k) by crowdsourcing” [44]. in the second version of the winogrande paper, the team’s declared ambitions with the new dataset increased. the dataset would allow a “true estimation of the machine commonsense capabilities ...with 44k problems that are inspired by the original design of wsc, but modified to improve both the scale and hardness of the problems” [42, 43]. their goal was then to minimize this bias in the original winograd schema. the key steps in winogrande construction consisted of (1) a carefully designed crowdsourcing procedure, followed by (2) a novel algorithm aflite that generalizes human-detectable biases based on word occurrences to machine-detectable biases based on embedding occurrences. the key motivation for this procedure was that it is difficult for humans to write problems without accidentally inserting unwanted biases. after training bert [44] and roberta [42] in the winogrande-all dataset and achieving good results, the team was somewhat skeptical about performance. they questioned whether neural language models successfully acquired commonsense or it was just an overestimation of the true capabilities of machine commonsense: the potential overestimation leads to another crucial question regarding potential unwanted biases that the large-scale neural language models might be exploiting, essentially solving the problems right, but for wrong reasons. ...while such biases and annotation artifacts are not apparent for individual instances, they get introduced in the dataset as problem authors subconsciously repeat similar problem-crafting strategies. to proceed, we must examine one key element in bert (and roberta) training; that is, masked language modeling. we will then be able to understand the way commonsense is absorbed by llms. 4 bert and cloze procedures the bidirectional encoder representations from transformers, bert for short, is an llm that handles contextual information in the pretraining phase. notable previous efforts can be cited: semi-supervised sequence learning was developed in 2015 [9],footnote 2 and well designed embeddings (elmo) appeared in 2017 [36]. the problem with those efforts is that they only used either left context or right context of a word, but language understanding is bidirectional, as indicated in decades of psychological studies, particularly in gestalt theory, and in philosophical discussion dating back to frege [16]. the main idea there is that meaning lies in a sentence and not in single words. for a century now, experimenters have been reporting findings that may be interpreted as showing that language behavior depends on total context: the results indicate that the ability to identify, learn, recognize, remember or produce any language “symbol” (element or pattern) depends heavily on the variable degrees to which it is associated with everything else by larger and meaningful (familiar) overall combinations. [31] the authors of bert proposed a solution for bidirectional language learning, where they masked out k% words during training. the authors of bert explicitly mentioned the psychological test called cloze test, also known as cloze procedure, in their commentary about bert.footnote 3 in our analysis, this is an essential character in the death of the wsc, so it pays to examine cloze procedures in more detail. a cloze procedure is a psychological tool for measuring the effectiveness of communication, introduced in 1953 by wilson taylor [45]. such a procedure is based on cloze units, in which the word “cloze” derives from the concept of closure. gestalt psychology defines the latter concept as the human tendency to complete a familiar but not-quite-finished pattern — for instance, when seeing a broken circle as a whole one. in other words, “closure” refers to our natural tendency to mentally close gaps. the same tendency applies to language; for instance, given “chickens cackle and — quack” almost anyone can instantly supply “ducks” [45]. in a cloze procedure, if the word uttered by a human subject is really the same as the word omitted, the person scores one cloze unit for correctly closing the gap in the language pattern. it should be clear that a cloze procedure is really similar to masked language modeling as performed when learning bert and similar large language models. for our investigation to proceed, it makes sense to examine in more detail the definition of linguistic commonsense: [t]he sentence pattern is a complex one made up of many subpatterns. one must know not only the meanings (i.e., patterns of symbolmeaning relationships) and forms (patterns of letters) of all the five words, but also the meanings of given combinations of them – plus the fact that the sentence structure seems to demand a term parallel to “cackle” but associated with ducks instead of chickens. in other words, one must guess what the mutilated sentence means as a whole, then complete its pattern to fit that whole meaning. [emphases added] [45] any method that intercepts a message from a “transmitter” (writer or speaker), mutilates it, and administers it to “receivers” (readers or listeners) that then attempt to fill the missing parts, is a method that potentially yields a number of cloze units. as taylor argued, different persons may express the same meaning in somewhat differing ways, and the same language patterns may have differing meanings for different people. the total context of a language behavior, such as the one involved in the resolution of a cloze unit, includes everything that tends to motivate, guide, assist or hinder that behavior. as noted by miller, all such factors are combined into a notion of commonsense: “i heard a — bark” is likely to elicit “dog” both because that word is habitually associated with “bark” and because it fits in with past experience with noisy dogs. if the verbal context is enlarged to “for the first time, i heard a — bark,” the impulse to supply “dog” may be reduced by commonsense; the subject may ask himself: “who is this guy that has never heard a dog? could he be referring to some other animal?” and if the preceding sentence has mentioned a voyage to the pribilof islands, the reader may draw on past knowledge to produce “seal.” [31] we emphasize the expression “reduced by commonsense” as it reflects exactly what researchers supporting the wsc argued for. habits of expression take over most of the work of translating an individual’s meaning into an organized series of language symbols for transmission to others. likewise, habits of reading or listening cause the reader or listener to anticipate words, almost automatically, when receiving messages. in many ways, language masked modeling is already discussed approvingly in connection with cloze procedures by taylor: how can a random system play fair when some words are easier to replace than others? obviously, one is more likely to be able to supply “an” in “a quinidine is — alkaloid isomeric ...” than to guess “$6,425” in “the city council voted — for a new swimming pool.” yet the former example is far more difficult reading. the answer is that if enough words are struck out at random, the blanks will come to represent proportionately all kinds of words to the extent that they occur. [45] as a digression, we note that both bert’s masked language modeling and cloze procedures have to decide on how many words to delete. bert was trained using 15% of words randomly masked out, a number based on empirical performance. it was then noted that deleting too few words leads to too much training effort, while deleting too many words may eliminate too much context. in short: bert’s pretraining applies a sort of cloze procedure at scale; we may refer to bert’s pretraining as going through a large number of “masked language closure” steps. in a sense, bert’s training is teaching a machine to solve a linguistic commonsense task. as winograd schemas are actually similar to cloze units, it turns out that bert’s designed training clearly address winograd schemas. we emphasize the connection with cloze units in the next section. 5 textual entailment, and schemas as cloze units the explicit linguistic problem to be solved given a winograd schema is anaphora resolution. however, winograd schemas are also variants of another task, recognizing textual entailment (rte) [4, 8, 41]. rte asks a machine to recognize what is the textual entailment (te) from a given sentence. a textual entailment is a linguistic phenomenon that appears everywhere in daily life communication; it expresses the relationship between sentences or fragments of texts in which one implies the other. for instance, “john bought a novel yesterday,” textually entails, “john bought a book.” dagan et al. have proposed a simple yet well-accepted definition of textual entailment: t textually entails h if and only if a human reading t would typically infer that h is most probably true [7]. as we can see here, solving a rte problem qualifies as a test for implicit causes in both human understanding and perception. winograd schema inherit such a feature. more accurately, 94.6% of all existing winograd schemas have this structure according to our own manual analysis. however, recognizing the correct direction of causation requires a prior understanding of what the entities in the text mean. consider the following example. the sentence “ferrous sulfate heptahydrate is green” (t) textually entails “\(feso_{4}7h_{2}0\) is green” (h) [23], which is a valid entailment because “ferrous sulfate heptahydrate’ and “\(feso_{4}7h_{2}0\)” refer to the same entity in the world in philosophical jargon. although this entailment looks, at first sight, a synonym problem, a human reader cannot tell whether t entails h by working with words alone. a piece of appropriate background knowledge is required. this example has the same structure as the morning star evening star puzzle that gottlob frege posed a century ago with respect to the difference between meaning (bedeutung) and reference (sinn) [16]. knowing that both stars are the same astronomical object (venus) required new empirical information. as the philosopher of language w.v. quine put it, “astronomical observation was needed, and not mere reflection on meanings, to determine the sameness of the entity in question” [37]. this sort of difficulty appears in the “ — barks” example discussed before in connection with cloze procedures. not surprisingly, research in rte has arrived at a definition that takes into account appropriate background knowledge: a text t textually entails a hypothesis h relative to a group of end-users g just in case, typically, a member of g reading t would be justified in inferring the proposition expressed by h from the proposition expressed by t. [23] the challenge here lies in the high level of dependency on contextualized information within a community. even though the intuition behind rte seems correct as a test for commonsense reasoning, it is hard to come up with a well-delimited test in which knowledge representation is not about general issues. indeed, levesque did not take rte as the best way of testing commonsense because there is an indefinite number of valid and correct consequences that may follow from a sentence within rte [27]. the problem of inferential causality [35] is unsurmountable as a practical test of machine intelligence. as an alternative, he proposed the wsc as a test that is not only feasible but also measurable. levesque’s innovation is the addition of a double ambiguity: a pronominal anaphora plus a coreferential ambiguity. for instance: (information) the city councilmen denied the demonstrators a permit. (ambiguity) because they feared/advocated for violence. the personal pronoun “they” may refer to either the city councilmen or the demonstrators. if we ask “who feared violence?” the answer is “the city councilmen”. conversely, if we ask “who advocated for violence,” the answer is “the demonstrators”. therefore, the coreferential ambiguity is based on a pair of special words that considerably shift the meaning of the sentence. we can see that the wsc, in an attempt to move away from the complexities of rte, arrived at a modified form of cloze test. first of all, the blank to be replaced, in a winograd schema, by either one of the pair of keywords is just a mask with only one correct answer. for instance, we have: the city councilmen denied the demonstrators a permit. because they [mask] violence. to make the connection with anaphora resolution even more evident, suppose the pronoum that has to be disambiguated is also masked. we would end up with two masks as follows: the city councilmen denied the demonstrators a permit. because [mask] [mask] violence. in this particular example we have a little bit more than 15% of masked tokens, and we have preserved enough context to a good guessing from the machine. (as a digression, note that, by limiting the flexibility of placeholders, winograd schemas introduce some sort of bias, a social desirability bias — the requirement that nouns are mapped correctly to keywords so as to infer what the designers of the test take as the correct and obvious alternative.) we must now examine what is it that roberta brought to the plot, and verify the effect of the winogrande corpus, so as to understand the role played by those suspects. 6 a second look at roberta and winogrande the large language model roberta is a refined version of bert. as indicated by the authors of roberta, its improved pretraining, with more corpora and better hyperparameters, led to significance performance gains across many tasks [29]. besides bookcorpus [52] and the english wikipedia (the original data used to train bert), roberta was trained with the english portion of the cc-news dataset,footnote 4 openwebtext,footnote 5 and, finally, and most notably, stories, a dataset introduced by trinh and le [48] containing a subset of commoncrawl data filtered exactly to match the style of winograd schemas. from the second version of the winogrande paper [42], we have that the fine-tuning of roberta with dpr (definite pronoun resolution dataset) achieved 83.1% accuracy in the wsc. dpr was a fairly common used dataset for fine-tuning models for solving the winograd schema that introduced 1886 additional winograd schemas authored by 30 undergraduate students. that result was already a state-of-art performance for the wsc. and then the fine-tuning of roberta with winogrande, as we know, achieved 90.1% accuracy in the wsc. indeed, roberta \(+\) dpr already produced excellent results. for instance, consider the pdp pronoun disambiguation problems) dataset. this dataset consists of 80 pronoun disambiguation problems, all formulated through multiple choice questions, in which a pronoun must be resolved to one of up to 5 (but mostly binary) options. this is clearly related to the wsc, and often used as proxy for the wsc itself. for the pdp, roberta + dpr and roberta + winogrande had close results, 86.3% and 87.5% respectively. we can see similar results for other datasets in table 1. table 1. comparison between dpr and winogrande fine-tuned roberta. note that superglue-wsc is discussed at https://super.gluebenchmark.com/tasks/.full size table the authors of the winogrande corpus briefly discussed those points: our model is based on roberta fine-tuned with winogrande (train and dev sets). to compare different corpora used as a resource, we also fine-tune roberta on dpr (train and test sets). for hyper parameter search, we use the same grid search strategy as in sect 4. overall, roberta fine-tuned on winogrande helps improve the accuracy on all the related tasks (table 6), and performs consistently better than when roberta is fine-tuned on dpr. while improvements on some related datasets (particularly wsc, pdp, and dpr) might seem expected, the significant improvement on copa is not so. [42] intuitively, the larger the dataset, the better; the winogrande corpus is valuable because it is a large one. one should expect the performance of roberta to increase by adding 40k+ winograd schemas. our goal through this discussion is to suggest that roberta was a key character in our plot, due to the masked learning modeling that lies beneath it, while winogrande was an important element of the plot but one that simply added learning material to the underlying infrastructure. before we leave this discussion, we must comment on the analysis by elazar et al. [13]. they argue that the perceived progress in the wsc is due to flawed evaluation, artifacts, and commonsense knowledge gleaned from a supervised training set rather than advancements in llms; clearly their analysis emphasizes features of the wsc that are not aligned with the analysis in this paper. while it is undeniable that flawed evaluation and artifacts have influenced the perceived progress in the wsc, evidence suggests that the contribution of pre-trained llms is significant and should not be disregarded. pre-trained llms indeed play a significant role in improving word sense disambiguation and substantial progress has been observed through bert to roberta (and note that bert alone got 64.9% accuracy in the wsc, while roberta alone reached 79.1% accuracy). further evidence supporting this claim is the change in performance across different versions of the winogrande paper: despite minimal changes in the winogrande dataset, sakaguchi et al.’s bert-based model achieved 77.6% in july 2019, and with the availability of roberta in november 2019, performance jumped to 90.1% [42, 44]. to conclude this section, we note that we have focused on the events that took place around 2019/2020, when the wsc was defeated. llms have had an explosive evolution since them; masked-modeling training is certainly not the only technique that deserves attention now. while gap-filling (cloze) training aligns with the human reading experience, where we move forward and backward through text, models such as gpt predict the next token using solely previous tokens. the latter strategy is similar to everyday human conversation, where information is exchanged linearly over time, and participants cannot go forward to examine tokens yet to be uttered. given the surprising success in question answering and dialogue by gpt-like models, one might thus think that there are other mechanisms that may defeat the wsc, perhaps provided that one can use really large amounts of data and really large models. but even an extraordinarily large model such as gpt-3 [5] does not lead to spectacular performance in the wsc: our testing led to mere 67% accuracy. things improve for the even larger gpt-4 [1], where we then got impressive 85.2% accuracy — but we do not really any access to that llm, so we do not know whether it has been trained with even larger sets of winograd-like sentences. more investigation is required, and we leave the matter to future investigation. 7 closing (clozing?) arguments the winograd schema challenge was killed because the cloze-style training that is used to produce many recent language models is perfectly tailored to solve winograd schemas. both roberta and winogrande were important in this plot, and roberta in particular was a key character as it carried the critical information needed for the killing. but in fact the wsc was not murdered by a single language model or a corpus. rather, the wsc died because it was supposedly designed to capture commonsense but in fact it only offered a task that is in essence a version of cloze procedures, a task that masked language modeling, as employed in large language models such as bert, is specifically designed to address. notes 1.the self-attention mechanism guarantees that long-distance context has “equal opportunity” to show up. when it comes to an anaphora resolution, the self-attention mechanism tackles it at its core. 2.as a digression, we note that quoc v. le, the second author of this paper, tackled the wsc in one of the first successful approaches using language models [48]. 3.in an interview, jacob devlin has commmented on the cloze test as being a fairly known test in psycholinguistics for accessing levels of readership ability. 4.at https://commoncrawl.org/2016/10/news-dataset-available/. 5.at http://skylion007.github.io/openwebtextcorpus. references open ai. gpt4 technical report. arxiv:2303.08774, 2023 bailey, d., harrison, a., lierler, y., lifschitz, v., michael, j.: the winograd schema challenge and reasoning about correlation. in: working notes of the symposium on logical formalizations of commonsense reasoning (2015) google scholar  bender, d.: establishing a human baseline for the winograd schema challenge. in: modern ai and cognitive science conference, pp. 39–45 (2015) google scholar  bobrow, d.: precision-focussed textual inference. in: proceedings of the workshop on textual entailment and paraphrasing acl, prague (2007) google scholar  brown, t.b., et al.: language models are few-shot learners. arxiv:2005.14165 (2020) cozman, f.g., neri, h.: some thoughts on knowledge-enhanced machine learning. int. j. approximate reasoning 136, 308–324 (2020) article  mathscinet  math  google scholar  dagan, i.: recognizing textual entailment: rational, evaluation and approaches. natural lang. eng. 15(4), i-xvii (2009) google scholar  dagan, i., glickman, o., magnini, b.: the pascal recognising textual entailment challenge. in: quiñonero-candela, j., dagan, i., magnini, b., d’alché-buc, f. (eds.) mlcw 2005. lncs (lnai), vol. 3944, pp. 177–190. springer, heidelberg (2006). https://doi.org/10.1007/11736790_9 chapter  google scholar  dai, a.m., le, v.q.: semi-supervised sequence learning. in: c. cortes, n. lawrence, d. lee, m. sugiyama, and r. garnett, eds, advances in neural information processing systems, vol. 28. curran associates inc (2015) google scholar  davies, e.: winograd schemas and machine translation. arxiv:1608.01884 (2016) davis, e., morgenstern, l., ortiz, c.l.: the first winograd schema challenge at ijcai-16. ai mag. 38(3), 97–98 (2017) google scholar  devlin, j., chang, m.-w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv:1810.04805v2 (2019) elazar, y., zhang, h., goldberg, y., roth, d.: back to square one: artifact detection, training and commonsense disentanglement in the winograd schema. in: proceedings of the 2021 conference on empirical methods in natural language processing, pages 10486–10500. association for computational linguistics (2021) google scholar  emami, a., et al.: the knowref coreference corpus: removing gender and number cues for difficult pronominal anaphora resolution. in proceedings of the 57th annual meeting of the association for computational linguistics, pp. 3952–3961, florence, italy, july 2019. association for computational linguistics (2019) google scholar  emami, a., trischler, a., suleman, k., chi, j., cheung, k.: a generalized knowledge hunting framework for the winograd schema challenge. in: naacl-hlt 2018: student research workshop, pp. 25–31 (2018) google scholar  frege, g.: sense and reference. philos. rev. 57 (1948) google scholar  joshi, b., shah, n., barbieri, f., leonardo neves, l.: the devil is in the details: evaluating limitations of transformer-based methods for granular tasks. in: proceedings of the 28th international conference on computational linguistics, pp. 3652–3659, barcelona, spain (online). international committee on computational linguistics (2020) google scholar  jurafsky, d., martin, j.h.: speech and language processing (3rd ed. draft) (2023) google scholar  kavumba, p., inoue, n., heinzerling, b., singh, k., reisert, p., kentaro inui, k.: when choosing plausible alternatives, clever hans can be clever. in: proceedings of the first workshop on commonsense inference in natural language processing, pp. 33–42, hong kong, china. association for computational linguistics (2019) google scholar  kocijan, v., cretu, a.-m., camburu, o.-m., yordanov, y., lukasiewicz, t.: a surprisingly robust trick for the winograd scheme challenge. in: annual meeting of the association for computational linguistics, pp. 4837–4842 (2019) google scholar  kocijan, v., davis, e., lukasiewicz, t., marcus, g., leora morgenstern, l.: the defeat of the winograd schema challenge. arxiv:2201.02387 (2023) kocijan, v., lukasiewicz, t., davis, e., marcus, g.: a review of winograd schema challenge datasets and approaches. arxiv:2004.13831v1 (2020) korman, d.: defining textual entailment. j. assoc. inf. sci. technol. 69 (2018) google scholar  levesque, h.: the winograd schema challenge. in: aaai (2011) google scholar  levesque, h.: on our best behaviour. in: ijcai (2013) google scholar  levesque, h.: common sense, the turing test, and the quest for real ai. the mit press (2017) google scholar  levesque, h., davis, e., morgenstern, l.: the winograd schema challenge. knowledge representation (2012) google scholar  liu, q., jiang, h., ling, h.-z., zhu, x,. wei, s., hu, y.: commonsense knowledge enhanced embeddings for solving pronoun disambiguation problems in winograd schemes challenge. arxiv:1611.04146 (2016) yinhan liu, y., et al.: roberta: a robustly optimized bert pretraining approach. arxiv:1907.11692 (2019) marcus, g., davis, e.: rebooting ai: building artificial intelligence we can trust. pantheon (2019) google scholar  george, a.: müller. language and communication, mcgraw-hili (1951) google scholar  nicos, i., michael loizos, m.: tackling the winograd schema challenge through machine logical inferences. stairs 75 (2016) google scholar  nicos, i., michael loizos, m.: how the availability of training material affects performance in the winograd schema challenge (2017) google scholar  nicos, i., michael loizos, m.: a data-driven metric of hardness for wsc sentences. gcai-2018 (epic series in computing) 55, 107–120 (2018) google scholar  judea pearl, j.: causality: models, reasoning, and inference. cambridge university press, 2\({}^{\underline{\rm a}}\) edition (2009) google scholar  peters, m.e., neumann, m., iyyer, m, gardner, m., kenton lee, k., luke zettlemoyer, l.: deep contextualized word representations, christopher clark (2018) google scholar  quine. two dogmas of empiricism. philos. rev. 60 (1951) google scholar  radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i.: language models are unsupervised multitask learners. technical report 8, openai blog (2019) google scholar  rahman, a., ng, v.: resolving complex cases of definite pronouns: the winograd schema challenge. in: proceedings of the 2012 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 777–789, jeju island, korea (2012). association for computational linguistics google scholar  ruan, y.-p., zhu, x., ling, z.-h., shi, z., liu, q., wei, s.: exploring unsupervised pretraining and sentence structure modelling for winograd schemes challenge. arxiv:1904.09705 (2019) rus, v.: a study of textual entailment. int. j. art. intell. tools 17 (2007) google scholar  sakaguchi, k., bras, r.l., bhagavatula, c., choi, y.: winogrande: an adversarial winograd schema challenge at scale. arxiv:1907.10641v2 (2019) sakaguchi, k., bras, r.l., bhagavatula, c., choi, y.: winogrande: an adversarial winograd schema challenge at scale. aaai-20 technical tracks 34(05) (2019) google scholar  sakaguchi, k., bras, r.l., bhagavatula, c., yejin choi, y.: winogrande: an adversarial winograd schema challenge at scale. arxiv:1907.10641v1 (2019) taylor, w.: cloze procedure: a new tool for measuring readability. j. quartely fall (1953) google scholar  hugo touvron, h., et al.: llama: open and efficient foundation language models. technical report, arxiv:2302.13971 (2023) trichelair, p., et al.: on the evaluation of common-sense reasoning in natural language understanding. arxiv preprint arxiv:1811.01778 (20180 trinh, t., quoc le, q.: a simple method for commonsense reasoning. arxiv:1806.02847 (2018) van aken, b., winter, b., löser, a., felix, a.: gers. how does bert answer questions? a layer-wise analysis of transformer representations. in: association for computing machinery, editor, proceedings of the 28th acm international conference on information and knowledge management, cikm ’19, pp. 1823–1832 (2019) google scholar  vaswani, a., et al.: attention is all you need. in: 31st conference on neural information processing systems (nips 2017), long beach, ca, usa (2017) google scholar  zhang, h., song, y.: a distributed solution for winograd schema challenge. in: icmlc2018 (2018) google scholar  zhu, y, et al.: aligning books and movies: towards story-like visual explanations by watching movies and reading books. in: the ieee international conference on computer vision (iccv) (2015) google scholar  download references acknowledgements the first author was supported by fapesp through grant 2018/0968-1. the second author was partially supported by cnpq through grant 305753/2022-3. this work was carried out at the center for artificial intelligence (c4ai-usp), with support by fapesp (grant 2019/07665-4) and by the ibm corporation. author information authors and affiliations universidade de são paulo, são paulo, brazil hugo neri & fabio g. cozman authors hugo neriview author publications search author on:pubmed google scholar fabio g. cozmanview author publications search author on:pubmed google scholar corresponding author correspondence to fabio g. cozman . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper neri, h., cozman, f.g. (2023). who killed the winograd schema challenge?. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_14 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature weapon engagement zone maximum launch range estimation using a deep neural network | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper weapon engagement zone maximum launch range estimation using a deep neural network conference paper first online: 28 november 2021 pp 193–207 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) weapon engagement zone maximum launch range estimation using a deep neural network download book pdf download book epub joao p. a. dantas  orcid: orcid.org/0000-0003-0300-802710, andre n. costa  orcid: orcid.org/0000-0002-2309-924810, diego geraldo  orcid: orcid.org/0000-0003-1389-914210, marcos r. o. a. maximo  orcid: orcid.org/0000-0003-2944-447611 & … takashi yoneyama  orcid: orcid.org/0000-0001-5375-107612  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1371 accesses 13 citations abstract this work investigates the use of a deep neural network (dnn) to perform an estimation of the weapon engagement zone (wez) maximum launch range. the wez allows the pilot to identify an airspace in which the available missile has a more significant probability of successfully engaging a particular target, i.e., a hypothetical area surrounding an aircraft in which an adversary is vulnerable to a shot. we propose an approach to determine the wez of a given missile using 50,000 simulated launches in variate conditions. these simulations are used to train a dnn that can predict the wez when the aircraft finds itself on different firing conditions, with a coefficient of determination of 0.99. it provides another procedure concerning preceding research since it employs a non-discretized model, i.e., it considers all directions of the wez at once, which has not been done previously. additionally, the proposed method uses an experimental design that allows for fewer simulation runs, providing faster model training. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others air-to-air missile launch envelope computation based on neural network chapter © 2023 dynamic performance evaluation of air-to-air missiles based on deep learning chapter © 2023 neural network semi-empirical modeling of the longitudinal motion for maneuverable aircraft and identification of its aerodynamic characteristics chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence intelligence machine learning military and defence studies weapons artificial intelligence 1 introduction within simulated computational environments, military systems must resemble reality in a level of fidelity that leads to useful conclusions [15]. this is done through the use of reliable computational models, that are deemed to encompass the main characteristics of the systems they represent [16]. when dealing with air combat, one of the most critical parts to be modeled is the missile. this is true concerning both the missile system itself and the decision of when to employ it, i.e., to fire. that is even more critical when considering beyond visual range (bvr) air combat since this decision must be taken based only on what the situational awareness systems display to the pilot [11]. in the context of constructive simulations, in which the aircraft behave autonomously, there is a need to provide their controlling algorithms with data similar to what real pilots would receive, so that the behaviors perform in accordance [9]. one of the most important aspects that a pilot can use to decide whether to launch a missile on an opposing aircraft is the weapon engagement zone (wez), which, in simple terms, represents the range of the weapon [10]. this definition is discussed with more depth further in sect. 2.1. the determination of this range is not a simple task, however, since it is influenced by a series of variables from both the shooter and the target. moreover, it is naturally dependent on the missile itself. in this work, we propose an approach to determine the wez of a given missile using a series of simulated launches in variate conditions. these simulations are used to train a machine learning algorithm that can predict the wez when the aircraft finds itself on different firing conditions. previous works have employed some types of artificial neural networks (ann), such as wavelet neural networks (wnn) [29] and a multi layer perceptron (mlp) with bayesian regularization of artificial neural networks (brann) [4], to make predictions of the wez, also from previously simulated data. purely mathematical approaches are also available within the literature, such as [14] and [23], but they provide an intermediate step between unrealistic missile models that consider fixed missile ranges and more complex models based on simulations. much more research may have been developed within companies and governments concerning wez determination [5], but this is still seldom publicly available. the contribution of this work is employing a deep neural network (dnn) with a novel non-discretized model, i.e. the model considers all directions of the wez at once, not discretizing the off-boresight angle (fig. 5) as done previously to the best of our knowledge. additionally, it uses an experimental design that allows for a lower number of simulation runs, which provides a faster training of the model. the remainder of this paper is organized as follows. section 2 provides the background, explaining in more depth the concept of wez, as well as presenting the particular missile model employed and the experimental design utilized. in sect. 3, the proposed methodology is detailed, whereas the results coming from it are presented and analyzed in sect. 4. finally, sect. 5 states the main conclusions of the work and suggests some future developments. 2 background in this section, we detail the concept of wez, present the missile model, and specify the simulation experimental design used within this work. 2.1 weapon engagement zone the term wez may present different definitions throughout the military domain. according to the united states department of defense [25], wez can be described as an “airspace of defined dimensions within which the responsibility for engagement of air threats normally rests with a particular weapon system.” although being a rather broad definition, its focus resides on the responsibility for engagement of target that is inside the zone by a specific system. in our work, on the other hand, we are more focused on the airspace defined by the range of a weapon system (missile), which is not necessarily responsible for engaging all threats within this zone. this is rather a possibility, that is, the wez in our case allows the pilot to identify an airspace in which the missile available has a larger probability of being successful in engaging a particular target. in other words, the definition of wez adopted by us is similar to what portrey et al. [27] present: a hypothetical area surrounding an aircraft in which an adversary is vulnerable to a shot. this concept can be found in the literature under different terminologies which may present subtle variations on meaning, such as launch acceptability region (lar) [29] and dynamic launch zone (dlz) [2]. figure 1 presents a simplified depiction of a wez, which stretches from the minimum range \(r_{min}\) to the maximum range \(r_{max}\). the \(r_{max}\) is defined by us as the maximum distance in which the missile will hit a non-maneuvering target, that is, if the target performs any maneuver, the missile will most likely miss if fired at this distance. on the other limit of this zone, the \(r_{min}\) is the minimum distance required by the missile to be able to properly activate its systems and, therefore, trigger its warhead. between these two ranges, there is the no-escape zone (nez) range (\(r_{nez}\)), which represents a distance within which the target is very unlikely to be able to evade the missile, even when employing a high-performance defensive maneuver. it is important to point out that the wez is also a function of the threat since it takes into consideration the parameters of the target in its calculation. as portrey et al. [27] state, the wez is determined by many factors regarding both the shooter and the target, such as “type of weapon, aircraft speed, relative altitudes, and geometry.” these factors are used by the authors of [27] to define a metric that allows the pilot to know what is the amount of g-force that must be pulled to escape from an incoming missile. therefore, their focus was less on the definition of the wez per se, but rather on the determination of this particular metric. on the other hand, birkmire [5], focuses precisely on the determination of the wez for a missile in the context of virtual simulations, i.e., simulations in which real pilots interact with simulated systems. therefore, his goal was to provide the pilots in virtual environments with a similar estimation of the wez as pilots in real aircraft have in their heads-up displays (huds) to support their decisions to fire a missile (fig. 2). fig. 1. simplified wez representation. full size image our work has a slightly different focus since it aims to provide wez information to autonomous agents within a constructive simulation environment, i.e., a simulation in which simulated pilots interact with simulated systems. in addition, we provide a map visualization of the estimated wez, which can be valuable within the analysis of this type of simulation. 2.2 missile model since this is not the focus of this work, inasmuch as the methodology presented may be applied to any simulated missile, we just provide a brief overview of the missile model. our implementation is completely done in the r programming language [19] and it provides a simplified model with 5\(^\circ \)c of freedom (5dof) of a fox 3 missile-based on [12]. according to [1], fox is a brevity code that refers to the guidance of a missile, in which type 3 stands for an active radar-guided missile, i.e., a missile which contains a seeker of its own that can track the target autonomously after reaching its activation distance. still, with regards to its guidance, the missile performs perfect proportional navigation concerning its target, maneuvering to exactly comply with its guidance law, as well as a loft maneuver (i.e., an aggressive climb right after launch) whenever possible, as fig. 3 shows. fig. 2. hud representation with focus on the wez indication. source: adapted from [22]. full size image fig. 3. missile simulated trajectory samples. full size image the model simulates the missile trajectory considering either a still or a maneuvering target. to define the nez range, the simulation considers a high-performance maneuver of \(+5\) g, which may be employed with a delay from the moment of launch. some important metrics for the missile flight are provided in fig. 4. fig. 4. missile trajectory metrics samples. full size image referring to fig. 4, the most straightforward metric is the mass (a). since the missile operates with a boost-sustain motor [24], its mass decays almost linearly during its boost (burn) phase. due to the loft maneuver, angle of attack values (b) vary very aggressively at the beginning of the flight, which can also be observed on the pitch angle (theta) chart (c). concerning heading (psi), there are some maneuvers to respond to the high-performance evasion that the target employs (d). accelerations in the east (e) and down (f) axis in the ned coordinate system [8] are also very abrupt due to the loft maneuver and the target response, respectively. the velocity (g) steadily increases during burn time and decays on the sustain phase. finally, the seeker angle (h) accounts for the proportional navigation, being defined as the deviation of the shooter’s longitudinal axis from the off-boresight angle (fig. 5). 2.3 experimental design the parameters used as inputs to our missile model (table 1) are very similar to the ones presented in [5], which makes it easier to compare our results with the ones obtained by it. however, instead of using an implementation in matlab simulink [21], our model was implemented entirely on r language as aforementioned, which has many prepackaged programs that help to solve analytical problems, prioritizing the simplicity of understanding and the parametrization. to provide a common understanding of the angles used, fig. 5 provides a depiction of them. fig. 5. pitch (a), heading (b), and off-boresight (c) angles with respect to the target aircraft. full size image these parameters are selected based on operational experience and the missile model possibilities. the shooter’s velocity and altitude are directly related to the energy that will be available to the missile. in particular, the launch altitude also influences the drag to which the missile will be subjected during flight, which is also true concerning the target altitude on the missile final approach. target’s velocity can either help or hinder the missile’s effectiveness, depending on its heading. however, heading alone cannot provide a full account with regards to positioning, since this is dependent on the off-boresight angle to determine whether the target aircraft is getting closer to the shooter and, therefore, to the missile itself. at last, the shooter’s pitch angle at the moment of launch may help the initial maneuvering of the missile, that is, its loft maneuver. instead of a full factorial experiment as [5] presented, we tried to reduce the number of simulation runs by means of a more sophisticated design that takes into account randomness in its formation. alternatively of a monte carlo simulation (mcs) that simply randomly samples the search space [17], we used the latin hypercube sampling (lhs), which is deemed to be more efficient [13]. the lhs is a near-random method, which aims at a better coverage of the search space, since a purely random approach may concentrate the samples by chance. its main idea is to divide the multidimensional space so that the random samples are drawn from these subdivisions instead of the whole search space [18]. in our particular case, we employed a maximin algorithm, which attempts to optimize the sample through the maximization of the minimum distance between design points, fulfilling the constraints established by the lhs method. table 1 presents the intervals for each variable used in the sampling. these limits were defined by subject matter experts, in this case, pilots, which considered meaningful values concerning their operational context. table 1. model parameters with the respective intervals considered.full size table 3 methodology this section contains the description of the preprocessing, training and evaluation of the dnn model that is applied on the data coming from the simulation. 3.1 simulation after creating the input batch files, through lhs and with the limits presented previously, 50,000 simulations were run using 2 intel xeon silver 4210r cpus with 2.40ghz and 128 gb of ram. it took approximately 7 h to execute all the simulations, which generated an output file containing the maximum range of the missile for the respective input conditions. 3.2 preprocessing from that, an exploratory data analysis (eda) was performed to identify general behaviors of the output data. the methods employed in this analysis were: histogram, boxplots, correlation, and descriptive statistics. before performing the training of the ann, some feature engineering techniques were employed. the first one was a form of encoding to better deal with cyclical features. the angles related to aircraft heading and off-boresight were encoded into their sine and cosine counterparts as done in [26], slightly increasing our model performance. in addition, a form of handling potential outliers was to perform downsampling of the latin hypercube design. this was done because the pre-established intervals generated some improbable conditions. for instance, an aircraft at 1,000 ft firing on a target at 45,000 ft is exceedingly rare from the operational standpoint since a pilot would most likely increase its altitude before launching a missile. therefore, we removed these undesirable samples, like the one presented, from the whole dataset based on subject matter expert operational knowledge, which can vary according to the mission type. lastly, data scaling was performed to equally distribute the importance of each input in the ann learning process [28]. this was done through a min-max scaler, which individually scales and translates all data features to a range from 0 to 1 [7]. 3.3 model training before training the dnn, a train-validate-test split was performed, allocating 80% for training and validation using a 5-fold cross-validation technique, and 20% for testing. this division is done randomly and will allow the evaluation of the machine learning model later. the dnn model was formed by 12 layers of nodes, with the structure represented in fig. 6. all nodes have a rectified linear activation function (relu) [3]. in addition, the adaptive moment estimation (adam) optimizer was employed, an extremely popular training algorithm for ann [6]. adam is a stochastic gradient descent method based on adaptive estimation of firstand second-order moments function [20], which, in our case, aimed to minimize the mean-squared error (mse) loss. this was monitored by an early stopping method that checked whether the validation set metric had stopped improving (the patience, i.e., the number of epochs to wait before early stop if no progress on the validation set, was set to 20). fig. 6. proposed dnn architecture. full size image 3.4 model evaluation as the model being analyzed deals with a regression problem, the evaluation of the model will be carried out observing the following metrics: mean absolute error (mae), mean squared error (mse), root mean squared error (rmse), and coefficient of determination (\(r^{2}\)). 4 results and analysis this section examines the exploratory data analysis and the test dataset metrics. additionally, it provides a multi-function display (mfd) representation, focusing on the wez indication based on the proposed model. 4.1 exploratory data analysis initially, an overview of the descriptive statistics of the model’s input and output variables was observed, as shown in table 2. the input variables of the model follow a uniform distribution since these variables were sampled using the lhs. the model’s output variable presents great variability with an average of 12.38 nm and a standard deviation (std) of 9.37 nm. notice that the mean and median (50%) are varying by 3.24 nm, which indicates a considerable amount of outliers for this variable at the top of the distribution. these outliers will be eliminated from a superior threshold value (33.28 nm), which is not the maximum value (max), but is rather the largest value of the sampling excluding outliers, based on the interquartile range (\(75\% 25\%\)). observing the minimum (min), values of the order of 0.08 nm can be found, which shows that in the dataset there are values in the target variable (max_range) that are smaller than the minimum activation distance of the missile modeled. for this case, this distance is considered to be 2 km (1.079 nm), which is the inferior threshold. so that the model would not be harmed in its training to try to predict the maximum missile range distance values, samples in which the model’s output variable was smaller than the minimum missile activation distance were removed from the dataset. a histogram and a boxplot were generated together to visualize the distribution and the thresholds of the target variable, which can be seen in fig. 7. table 2. descriptive statistics of the model’s input and output variables.full size table pearson’s correlation analysis of the variables can be seen in the correlation matrix represented in fig. 8. notice that none of the model’s features has a strong correlation with each other, with the largest absolute value being only 0.30 between shooter’s altitude (alt_sht) and pitch (pit_sht). the performance of the algorithm may deteriorate if two or more variables are tightly related, called multicollinearity. we may also be interested in the correlation between input variables with the output variable (max_range) to provide insight into which variables may or may not be relevant as input for developing a model. only the variables alt_sht and pit_sht have a slight correlation with the target variable. fig. 7. histogram and boxplot of the target variable. full size image fig. 8. pearson correlation matrix of all model variables. full size image 4.2 model predictions table 3 shows all the metrics used to evaluate the model with respect to the test set at the end of the training process. very satisfactory results were found, with a coefficient of determination above \(99\%\), which shows a very consistent model. in addition, note that the mae was around 0.58 nm, which can be considered a very low value, considering that the values of the target variable have a mean of 13.13 nm with a standard deviation (std) of 8.58 nm. if we consider the rmse, which penalizes the outliers’ effects, the observed value is around 1.10 nm. table 3. metrics used to evaluate the dnn model at the end of the training process.full size table a 5-fold cross-validation was conducted to estimate the skill of a machine learning model on unseen data and will help to better understand our data, giving much more information about our algorithm performance. the metrics of the five-folds were very similar as shown in table 4. the low variance found between the folds of this sample demonstrates the consistency of the model. table 4. 5-fold cross-validations metrics.full size table 4.3 model representation we estimated the wez maximum launch range from the trained model using one of the samples from the test group. the target’s position was varied by changing the off-boresight values from \(-60^{\circ }\) to \(+60^{\circ }\) with steps of \(0.5^{\circ }\). a mfd representation with a focus on the wez indication can be seen in fig. 9. the curve that shows the missile’s maximum range proved to be quite consistent, with a continuous aspect throughout the variations of off-boresight angles. thus, we conclude that employing a different approach, unlike other research, with the incorporation of the off-boresight angle between the reference and the target aircraft as a feature in the model does not affect the performance of the wez estimation significantly since the supervised learning model used can be able to generalize well the results obtained in the training dataset to the test dataset. fig. 9. mfd representation with a focus on the wez maximum range indication. full size image 5 conclusions and future work therefore, the main improvements advocated by us with respect to [5] is that, instead of discretizing the wez calculation concerning the off-boresight angle between shooter and target (fig. 4), creating, therefore, several anns, we approached the problem considering the whole space defined by the shooter’s radar, with only one dnn being able to predict the values of wez. in addition, the number of simulation runs was much lower (50,000 runs, as opposed to 222 million in [5]), which was achieved by a more carefully tailored experimental design. in addition, in this work, a dnn with an mlp architecture was used, and brought better results than an ann with only one hidden layer, as done in [5], comparing the coefficient of determination of both approaches applied to their respective datasets. in addition, different configurations of training and test groups were used in the dataset using the k-fold cross-validation. for a value of \(k = 5\), that is, the training and test group was sampled 5 times, the results found were quite similar among all samples, demonstrating the consistency of the dnn model presented in this work. the use of feature engineering techniques, with the creation of other model input variables, such as the use of sine and cosine for the variables that represent heading and off-boresight angles, also contributed to greater adequacy of the model to the dataset collected from the simulations. furthermore, it was observed, with the use of operational knowledge, that some of the samples collected would be unlikely to occur in a real air combat environment. these cases were when, for example, at a given altitude, the speeds of a given agent should meet at certain speed intervals. in the dataset some samples were not respecting these intervals, which could impair the model’s performance, trying to predict cases that would most likely not occur in a real situation. to avoid these problems, these samples were eliminated from the dataset. future work should investigate how possible improvements in the architecture used for the dnn can bring better results and be more efficient, i.e. with a lower computational cost in the training process. in addition, the results found in this work can be compared with the use of other supervised machine learning techniques. these comparisons will help to determine the most appropriate methodology for calculating wez. in addition, in future work, it is possible to carry out calculations not only of the maximum range but also the distances related to the nez or even intermediate distances that could provide pilots with more assertive information about the probabilities of a missile reaching its target. also, more advanced simulation models of the missile may be used in the future to provide better reliability to the presented results. references air land sea application center: brevity: multi-service tactics, techniques, and procedures for multi-service brevity codes (2020) google scholar  alkaher, d., moshaiov, a.: dynamic-escape-zone to avoid energy-bleeding coasting missile. j. guidance control dyn. 38(10), 1908–1921 (2015) article  google scholar  bengio, y., goodfellow, i., courville, a.: deep learning, vol. 1. mit press massachusetts, usa (2017) math  google scholar  birkmire, b., gallagher, j.: air-to-air missile maximum launch range modeling using a multilayer perceptron. in: aiaa modeling and simulation technologies conference, p. 4942 (2012) google scholar  birkmire, b.m.: weapon engagement zone maximum launch range approximation using a multilayer perceptron. master’s thesis, wright state university (2011) google scholar  bock, s., weiß, m.: a proof of local convergence for the adam optimizer. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2019) google scholar  bonaccorso, g.: machine learning algorithms. packt publishing ltd, birmingham (2017) google scholar  cai, g., chen, b.m., lee, t.h.: coordinate systems and transformations. in: unmanned rotorcraft systems, pp. 23–34. springer, london (2011). https://doi.org/10.1007/978-0-85729-635-1_2 costa, a.n.: sequential optimization of formation flight control method based on artificial potential fields. master’s thesis, instituto tecnológico de aeronáutica, são josé dos campos, sp, brazil (2019) google scholar  dantas, j.p.a., costa, a.n., geraldo, d., maximo, m.r.a.o., yoneyama, t.: engagement decision support for beyond visual range air combat. in: 2021 latin american robotics symposium (lars), pp. 1–6 (2021), accepted for publication google scholar  dantas, j.p.a.: apoio à decisão para o combate aéreo além do alcance visual: uma abordagem por redes neurais artificiais. master’s thesis, instituto tecnológico de aeronáutica, são josé dos campos, sp, brazil (2018) google scholar  departament of defense: military handbook: missile flight simulation part one: surface-to-air missiles (mil-hdbk-1211) (1995) google scholar  deutsch, j.l., deutsch, c.v.: latin hypercube sampling with multidimensional uniformity. j. stat. planning infer. 142(3), 763–772 (2012) article  mathscinet  google scholar  farlik, j., casar, j., stary, v.: simplification of missile effective coverage zone in air defence simulations. in: 2017 international conference on military technologies (icmt), pp. 733–737. ieee (2017) google scholar  hancock, p.a., vincenzi, d.a., wise, j.a., mouloua, m.: human factors in simulation and training. crc press, boca raton (2008) google scholar  hill, r.r., miller, j.o., mcintyre, g.a.: applications of discrete event simulation modeling to military problems. in: proceeding of the 2001 winter simulation conference (cat. no. 01ch37304), vol. 1, pp. 780–788. ieee (2001) google scholar  homem-de-mello, t., bayraksan, g.: monte carlo sampling-based methods for stochastic optimization. surv. oper. res. manage. sci. 19(1), 56–85 (2014) mathscinet  google scholar  husslage, b.g., rennen, g., van dam, e.r., den hertog, d.: space-filling latin hypercube designs for computer experiments. optim. eng. 12(4), 611–630 (2011) article  google scholar  ihaka, r., gentleman, r.: r: a language for data analysis and graphics. j. comput. graph. stat. 5(3), 299–314 (1996) google scholar  kingma, d.p., ba, j.: adam: a method for stochastic optimization. arxiv preprint arxiv:1412.6980 (2014) klee, h.: simulation of dynamic systems with matlab and simulink. crc press, boca raton (2018) google scholar  kravchenko, m.: future ui. https://br.pinterest.com/krava88/future-ui/. accessed 06 nov 2021 li, a., meng, y., he, z.: simulation research on new model of air-to-air missile attack zone. in: 2020 ieee 4th information technology, networking, electronic and automation control conference (itnec), vol. 1, pp. 1998–2002. ieee (2020) google scholar  noaman, d., noaman, m., mahir, d., rami, a., faiz, d., et al.: boost-sustain missile motor performance with fixed predetermined coast time interval. eur. j. mol. clin. med. 7(2), 5070–5079 (2020) google scholar  office of the chairman of the joint chiefs of staff, washington dc: dod dictionary of military and associated terms (2021) google scholar  petneházi, g.: recurrent neural networks for time series forecasting. arxiv preprint arxiv:1901.00069 (2019) portrey, a.m., schreiber, b., winston, b.: the pairwise escape-g metric: a measure for air combat maneuvering performance. in: proceedings of the winter simulation conference, 2005. p. 8. ieee (2005) google scholar  priddy, k.l., keller, p.e.: artificial neural networks: an introduction, vol. 68. spie press, bellingham (2005) google scholar  yoon, k.s., park, j.h., kim, i.g., ryu, k.s.: new modeling algorithm for improving accuracy of weapon launch acceptability region. in: 29th digital avionics systems conference, p. 6-d. ieee (2010) google scholar  download references acknowledgments this work was supported by finep (reference no 2824/20). takashi yoneyama is partially funded by cnpq – national research council of brazil through the grant 304134/2-18-0. author information authors and affiliations decision support systems subdivision, institute for advanced studies, sao jose dos campos sp, 12.288-001, brazil joao p. a. dantas, andre n. costa & diego geraldo autonomous computational system lab (lab-sca), computer science division, aeronautics institute of technology, sao jose dos campos sp, 12228-900, brazil marcos r. o. a. maximo electronic engineering division, aeronautics institute of technology, sao jose dos campos sp, 12228-900, brazil takashi yoneyama authors joao p. a. dantasview author publications search author on:pubmed google scholar andre n. costaview author publications search author on:pubmed google scholar diego geraldoview author publications search author on:pubmed google scholar marcos r. o. a. maximoview author publications search author on:pubmed google scholar takashi yoneyamaview author publications search author on:pubmed google scholar corresponding author correspondence to joao p. a. dantas . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper dantas, j.p.a., costa, a.n., geraldo, d., maximo, m., yoneyama, t. (2021). weapon engagement zone maximum launch range estimation using a deep neural network. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_14 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords weapon engagement zone deep neural network air combat publish with us policies and ethics profiles joao p. a. dantas view author profile marcos r. o. a. maximo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature automated and intelligent vocational guidance system for classifying specialties based on poscomp microdata | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper automated and intelligent vocational guidance system for classifying specialties based on poscomp microdata conference paper first online: 31 january 2025 pp 93–106 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) automated and intelligent vocational guidance system for classifying specialties based on poscomp microdata download book pdf download book epub jean carlos costa  orcid: orcid.org/0000-0001-5210-55079 & reginaldo santos  orcid: orcid.org/0000-0002-0456-85479  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 216 accesses abstract the national examination for admission to postgraduate studies in computing (poscomp) is administered by the brazilian computer society (sbc) to assess candidates’ knowledge for postgraduate programs in computer science in brazil. being one of the main evaluative instruments, the results of the poscomp exam, i.e., the database, can potentially obtain relevant patterns about the participants. in this sense, this article aims to propose an automated and intelligent vocational guidance system. this vocational system guides the participants to follow a research specialty based on the performance obtained in the exam. the proposed approach aims to support students in their academic decisions. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a pseudo-based online examination system chapter © 2024 research on the level of soft skills of the digital generation chapter © 2022 system for analysis of the performance of scientific jobs in distributed systems article 06 june 2024 1 introduction the national examination for admission to graduate programs in computing (poscomp) is an annual exam conducted in brazil. its purpose is to assess the knowledge of candidates in the field of computer science who wish to enroll in graduate programs offered in brazil [19]. institutions offering these programs use the participants’ scores as a selection criterion, either as a requirement or as part of the admission process for master’s or doctoral programs [19]. students can be evaluated through various tests administered in brazil, such as poscomp, the national student performance exam (enade), and the basic education evaluation system (saeb), among others. additionally, several studies have been conducted on exam data analysis using data mining and machine learning techniques [5, 9, 22]. for instance, the studies aim to analyze the performance of students in computer science courses using exploratory data analysis and clustering techniques. furthermore, the study [17] utilizes machine learning techniques to predict the grades in enade for computer science courses in brazil. these studies highlight the importance of analyzing, identifying, and exploring national exam data. educational data analysis has become a crucial tool in developing methodological strategies to improve teaching and learning [11]. using computational tools in this analysis offers significant benefits for teaching methodologies and processes. it also allows administrators and coordinators to evaluate student performance across various topics and specialties within computer science courses. consequently, these tools assist in developing strategies and improving areas where students face difficulties [10, 23]. in this context, poscomp microdata provides an opportunity to extract useful and non-trivial information, particularly regarding the specialties desired by the participants [3]. poscomp includes information about the specialty that candidates wish to pursue in their master’s or doctoral studies. identifying these specialties is essential for graduate programs in brazil, enabling them to develop research projects aligned with the knowledge of both students and advisors, thereby ensuring research quality and academic progress [1]. the creation of an automated and intelligent system is crucial in assisting participants in choosing specialties for research development [12]. as defined by the authors [16], decision-based agents seek a sequence of actions to achieve their goals. therefore, using intelligent tools to guide specialty choices is vital for both institutions and students, contributing to the progress of research during their academic journey [6]. this study aims to develop an automated and intelligent system for predicting the specialties of participants based on their characteristics and the scores obtained in the poscomp exam. this tool aids in classification using machine learning techniques, helping participants make informed decisions about which specialty or research area to pursue in their master’s or doctoral studies. ultimately, this can lead to improved academic performance and the quality of research produced at the institution. 2 related work studies in the field of artificial intelligence in education employ machine learning and data mining techniques to enhance educational processes. these techniques are essential for knowledge discovery from exam data, such as poscomp and enade [15]. the application of ai techniques in education, like educational data mining, allows for the analysis and evaluation of student performance to identify profiles, determine which subjects pose difficulties, and predict student dropout rates [4, 5, 9]. these analyses are performed using publicly available data or data from private institutions. in searching for studies related to this research, several works were found that address student performance analysis in educational institutions, as well as the use of data mining and machine learning techniques for classification and prediction in education. these studies contribute significantly to education, particularly in identifying student performance and profiles and addressing dropout rates in higher education courses. the authors in [13] surveyed 1,239 participants, including professors, researchers, and professionals in the field of computer science, to assess the relevance of the content covered by poscomp. the quantitative results indicated that certain exam contents have varying degrees of relevance, providing insights for improving educational programs and preparation strategies for the exam based on the most valued areas of knowledge. the study conducted in [14] presents the web platform poscomp coach, developed to assist candidates in preparing for the graduate entrance exam in computer science. the platform offers a database of 1,120 distinct questions from poscomp exams conducted between 2002 and 2017, allowing for timed practice tests with automatic correction. the study also includes an evaluation of the platform, involving a comparative analysis with similar solutions and usability assessment. the results highlight positive aspects of the proposed solution, including the number of available questions and high usability scores from students nationwide. the research in [21] presents a process for evaluating undergraduate courses by analyzing enade questions from 2008 to 2017 and poscomp questions from 2014 to 2018, aiming to determine which subjects are required in the questions. the results demonstrated the need for a comprehensive curriculum in computing fundamentals to provide a solid foundation for students. the research by [18] developed a mobile application using android to assist candidates in preparing for the poscomp exam by the sbc. the app includes a collection of exam questions organized by topic, each with detailed solutions, enabling users to study autonomously. the use of mobile computing to support education has proven attractive, allowing users to access study materials and knowledge anytime, anywhere. the authors in [3] conducted a comparative analysis of the 2014 to 2019 poscomp editions to evaluate computer science graduates and the reference curriculum (cr) of the sbc, approved in 2016. this comparison identified that approximately 60% of the cr content was not covered in poscomp exams. only 14 topics showed significant and consistent presence across the exam editions, with significant differences in the number of questions related to the curriculum axes. this comparative analysis highlights the discrepancy between the cr content and the coverage in poscomp exams over the years, suggesting the need to better align the exam content with the proposed curriculum. the authors in [8] evaluated student performance to improve courses using educational data mining techniques. they employed methods to extract useful information from student performance data and predict future outcomes using machine learning techniques, specifically decision tree. the knowledge discovery in databases (kdd) process was used for data preparation, knowledge extraction, and the identification of useful patterns in educational datasets. the research by [2] applied educational data mining approaches to model student performance using classification models such as naïve bayes, decision tree, and deep learning. they reported that the deep learning classifier outperformed others, with an overall prediction accuracy of 95%. this work was conducted using data from an information technology bachelor’s course over six semesters. the authors in [20] reported that dropout rates are higher in computer science courses. to address this issue, they applied data mining techniques, which are gaining traction in education. they used decision tree techniques and employed three execution options: use training set, supplied test set, and cross-validation. they concluded that the use training set option provided the best classification accuracy, but noted that it was not realistic for training. the j4.8 algorithm and cross-validation showed good classification accuracy. in [1], data mining techniques were used to analyze and predict student academic performance to propose interventions for improvement. the authors aimed to calculate the academic performance of undergraduate students using data mining techniques, specifically classification algorithms, on records of 800 computer science students. they used four feature selection methods: genetic algorithms, gain ratio, relief, and information gain, and for classification algorithms: k-nearest neighbor, naïve bayes, bagging, random forest, and j48 decision tree. the experimental results showed that genetic algorithms provided the best accuracy of 91.37% with the knn classifier. among various performance analysis approaches many studies focus on identifying student profiles and predicting dropout rates. however, there is a need to implement intelligent tools for decision-making in identifying participant specialties. this study recommends developing a vocational guidance system using machine learning algorithms for specialty classification. the goal is to assist poscomp participants in choosing specialties for master’s or doctoral programs based on their exam scores. this work addresses gaps left by previous studies and opens new possibilities for vocational guidance in the field of computer science. 3 methodology this study aims to develop an intelligent tool to assist students, professors, and educational institutions, primarily in identifying specialties for conducting postgraduate research. for this purpose, data from participants who took the poscomp exam between 2016 and 2019 were used. data from 2020, 2021, and 2022 were not collected due to the pandemic. thus, the development and implementation of the vocational guidance system employed supervised machine learning techniques and algorithms to build the specialty classifier. data were obtained through an official request to the brazilian computer society (sbc). these data pertain to candidates who graduated from computer science-related courses [22]. a total of 14,575 records with 35 attributes of the participants were identified. the kdd process was used to carry out the knowledge discovery stages and implement data analysis. as defined by [7] on the kdd process, it involves a non-trivial process of identifying valid, novel, potentially useful, and ultimately understandable patterns from data. 3.1 preprocessing during attribute selection, participant characteristics, and exam scores were considered. among the 35 attributes provided by sbc, age, gender, state, region, specialties, and participant scores were selected. these attributes were used in machine learning algorithms for the classifier to identify specialties based on participant characteristics and knowledge. with the selected attributes, preprocessing was performed to identify duplicate and inappropriate information for the model application. for instance, candidates who took the exam only for self-evaluation, without intending to enter postgraduate programs, were removed. similarly, participants residing in peru were also removed. participants who registered incorrectly were excluded since no information was found for replacing and correcting this data. additionally, participants with an average score below 17 points in the final grade were removed based on descriptive analysis and normal distribution. in the poscomp data, there is an attribute for specialties, containing the areas of concentration and research lines that participants choose for their master’s and doctoral studies. when registering, candidates can select more than one specialty as an alternative for possible studies during their postgraduate studies. procedures were implemented to identify candidates who opted for more than one option. thus, only the specialties considered as the first option by the participants were selected. as specialties were selected, these were merged. due to the similarity in specialties across different postgraduate programs, such as artificial intelligence and computational intelligence, as well as software engineering and software quality, it was decided to merge these specialties. table 1. poscomp areas and respective topics.full size table each exam question addresses areas of computer science, which include mathematics, computing fundamentals, and computing technology. mathematics covers topics such as differential and integral calculus, and linear algebra, among others. additionally, the computing fundamentals area includes topics such as algorithm analysis and data structures, among others. information technology encompasses topics such as databases, compilers, computer graphics, and artificial intelligence. table 1 shows the other topics covered in the exam. two new attributes were created from the specialties attribute based on the most requested by participants. with more than 400 specialties identified in the poscomp data, a procedure was performed to group and create two new attributes: research lines and concentration areas. in the research line, specialties with the same research purpose were grouped. for example, software engineering encompasses areas of computation methodologies and techniques and computational modeling. similarly, specialties in theoretical computing, and formal methods, among others, are grouped with other areas such as artificial intelligence, computing systems, optimization, etc. figure 1 details the grouping process. fig. 1. creation of new attributes based on specialties. full size image 3.2 vocational guidance system proposal this work proposes to develop a vocational guidance system for computer science students and participants, enabling the identification of specialties according to their knowledge and skills. the tool provides research options based on knowledge and characteristics extracted from poscomp microdata. thus, research during the master’s and doctoral studies can be favorable and not detrimental to their studies. the system classifies the specialty based on knowledge acquired over the years, whether as students or professionals. for the system implementation, three classification models were used. due to the data complexity, it was identified that the data are non-linear, which led to the decision to employ three models to make it efficient in the final prediction. figure 2 presents the vocational guidance system execution process. fig. 2. vocational system application model. full size image initially, new data, including participant characteristics and scores, are inserted into the system. this data goes through a transformation process before being applied to model 1, acting as an initial classifier, which determines if the data belongs to the computing or information category. depending on the result of this classification, the data is directed to model 2 or model 3. if the data is classified as computing, it is directed to model 2. this model predicts whether the participant’s specialty is artificial intelligence or computing systems. conversely, if the data is classified as information, it is directed to model 3. this model predicts whether the participant’s specialty is software engineering or computer networks. the prediction result is then presented by the system, providing an indication of the participant’s specialty based on the input data. this process allows for the classification of specialties, guiding their studies in postgraduate programs in brazil. in the model implementation, the aim was to achieve satisfactory performance for each algorithm used, which included decision trees, random forests, svm, and neural networks. for each algorithm, the dataset was divided into 70% for training and 30% for testing. this approach allowed for identifying the best hyperparameters for each algorithm. subsequently, cross-validation techniques were used to evaluate the model’s performance. using cross-validation techniques, 10 folds were employed to evaluate each model. to obtain good model performance, hyperparameter tuning was conducted. this adjustment was crucial to directly control the structure, function, and performance of the algorithms. the process involved experimenting with different values and parameters during algorithm training to find the best combinations. after hyperparameter tuning, the best ones for each algorithm were identified. then, cross-validation was performed to determine the best algorithm for models 1, 2, and 3. the source code is available for review in this repositoryfootnote 1. 4 results and discussion in this section, we present the results obtained from exploratory data analysis and the application of machine learning algorithms. through exploratory data analysis, we identified the most sought-after research areas among candidates for master’s and doctoral programs. the data revealed that artificial intelligence, software engineering, computer systems, and information systems are the most popular fields. this interest is driven by market demand, particularly in artificial intelligence, which encompasses data science, data engineering, and data mining, among other areas [24]. figure 3 shows the dominant specialties in each state based on the participants’ choices. in alagoas, computer systems are most in demand; in espírito santo, information systems have the highest representation; in mato grosso do sul and pará, software engineering predominates; while in minas gerais and são paulo, artificial intelligence and software engineering, respectively, are the most sought-after. this information is valuable for educational institutions’ planning and vocational guidance, helping to offer courses and specializations in each state according to student demand. fig. 3. predominant specialties by state, classified by region in brazil. full size image we categorized the specialties present in the participants’ data into four classes for algorithm application, as outlined in sect. 3.1. the identified and applied classes for the classifiers were: artificial intelligence, software engineering, computer systems, and computer networks. figure 4 shows the number of instances in each class. fig. 4. sample distribution by specialties in the poscomp exam. full size image for specialty classification, samples were divided into binary to generate predictions for each model. this ensures that each model is trained with a specific data set, guaranteeing the effectiveness and accuracy of the predictions. the size of the data set used for each model highlights the variation in the number of participants involved in its construction and evaluation. the model 1 used 8613 data points, the model 2 used 5451 data points, and the model 3 used 3162 data points. 4.1 model analysis the four classification algorithms used for classification with the poscomp data sets, based on participants’ characteristics and scores, are presented below. the best hyperparameters were selected to achieve optimal performance for models 1, 2 and 3. in model 1, participants identified with “computing” or “information” were directed to model 2 or model 3, respectively. the best hyperparameters for each algorithm are shown in table 2. table 2. the best hyperparameters of each algorithm with the training and testing accuracies for model 1.full size table after determining the hyperparameters, cross-validation was performed to evaluate the algorithms’ performance. the random forest algorithm yielded the best results, likely due to its ability to capture data complexity. therefore, the random forest algorithm was chosen for implementing model 1. model 2 aims to classify participants in the computing field, distinguishing between artificial intelligence and computer systems. the best hyperparameters for each algorithm for model 2 are presented in table 3. following cross-validation, the random forest algorithm again outperformed others and was selected for model 2. table 3. the best hyperparameters of each algorithm with the training and testing accuracies for model 2.full size table in model 3, participants classified in the information field by model 1 were directed here, where the final prediction focused on software engineering or computer networks. the best hyperparameters for each algorithm are shown in table 4. post-cross-validation, the random forest algorithm demonstrated superior performance and was selected for this model. table 4. the best hyperparameters of each algorithm with the training and testing accuracies for model 3.full size table to analyze the vocational guidance system’s performance, a test with 8613 samples was conducted to calculate the models’ metrics for classifying poscomp participants’ data. the objective was to evaluate these metrics and observe the results. if the system correctly classified all three models, it was considered a success; otherwise, it was considered a failure. a confusion matrix was generated, and machine learning metrics were calculated. the system achieved an overall accuracy of 52.59%, indicating the presence of non-linearities that compromised accuracy. participants retaking the exam with different specialty choices could also confuse the models, hindering correct classification. the metrics evaluating the class performance showed that artificial intelligence and computer networks had accuracies of 67% and 86.66%, respectively, while software engineering and computer systems recorded 76.97% and 77.89%, respectively. this indicates variable performance among classes, with the classifier being more accurate in classifying computer networks, despite having fewer samples. table 5 shows the accuracy of each class along with other relevant metrics. table 5. metrics by class of the guidance system.full size table however, the analysis of precision and sensitivity metrics revealed significant variations among the classes. the artificial intelligence class had a high precision of 81.86%, indicating many participants’ scores clustered in a specific feature space. for computer networks, precision was only 18.94%, suggesting classification difficulties, possibly due to the low sample size and score dispersion. sensitivity showed that computer systems had the highest value, indicating effectiveness in identifying true positives in this class. however, the low precision of 38.02% proposes some participants may be incorrectly classified in this class. the classifier’s specificity, measuring the ability to correctly identify negative classes, exceeded 80%. however, variability in specificity among classes indicates performance differences depending on the participant. the f1-score, combining precision and sensitivity, also showed significant variations. artificial intelligence and computer systems classes had relatively high scores, while software engineering and computer networks had lower scores, reflecting disparities in individual metrics. this could be due to the small sample size of some classes relative to others. the vocational guidance system’s classifier results indicate some classes excel in the evaluated metrics. however, lower-than-expected performance in some specialties may be due to the similarity between areas and the disparity in class sizes in the data, suggesting the need for class balancing to improve the classifier’s predictive ability. 5 conclusions the aim of this study was to develop a vocational guidance system using poscomp data from 2016 to 2019 to assist in decision-making when choosing specialties that align with candidates’ knowledge and skills. this work aims to help students and professionals select specialties for their master’s or doctoral programs. additionally, it assists in selecting advisors for academic research. this study allows computer science students to avoid doubts and frustrations when deciding on an advisor. it also helps advisors understand their mentees’ skills and knowledge, using the tool to identify which specialties align with their research interests. thus, this tool aids in the development of research projects, significantly reducing dropout rates and enhancing the quality of research by allowing studies aligned with participants’ preferences. for future work, we suggest employing deep learning algorithms to handle the complexity of participants’ non-linear data. it is also recommended to use a unified model for the system, which could enable more effective classification. notes 1.https://github.com/jeancarloscc/orientposcomp-system. references ahmed, m.r., tahid, s.t.i., mitu, n.a., kundu, p., yeasmin, s.: a comprehensive analysis on undergraduate student academic performance using feature selection techniques on classification algorithms. in: 2020 11th international conference on computing, communication and networking technologies (icccnt), pp. 1–6 (2020). https://doi.org/10.1109/icccnt49239.2020.9225341 amazona, m.v., hernandez, a.a.: modelling student performance using data mining techniques: inputs for academic program development. in: proceedings of the 2019 5th international conference on computing and data engineering, iccde 2019, pp. 36–40. association for computing machinery, new york (2019). https://doi.org/10.1145/3330530.3330544 arcanjo augusto, a.l., et al.: comparação dos conteúdos do poscomp com o currículo de referência dos cursos de computação da sbc. revista coming commun. innov. gazette 5(3), 14–23 (2021). https://doi.org/10.5902/2448190467894. https://periodicos.ufsm.br/coming/article/view/67894 bezerra, l.n.m., silva, l., silveira, i.f.: adequação dos exames nacionais ao perfil do egresso do bacharelado em sistemas de informação: um estudo de caso frente aos exames enade e poscomp google scholar  cunha, r., sales, c., santos, r.: análise automática com os microdados do enade para melhoria do ensino dos cursos de ciência da computação. in: anais do xxix workshop sobre educação em computação (wei 2021). pp. 208–217. sociedade brasileira de computação, brasil (2021). https://doi.org/10.5753/wei.2021.15912. https://sol.sbc.org.br/index.php/wei/article/view/15912 droescher, f.d., silva, e.l.d.: o pesquisador e a produção científica. perspectivas em ciência da informação 19(1), 170–189 (2014). https://doi.org/10.1590/s1413-99362014000100011 fayyad, u., piatetsky-shapiro, g., smyth, p.: from data mining to knowledge discovery in databases. ai mag. 17(3), 37 (1996) google scholar  fernando raguro, m., carpio lagman, a., p. abad, l., s. ong, p.l.: extraction of lms student engagement and behavioral patterns in online education using decision tree and k-means algorithm. in: 2022 4th asia pacific information technology conference, apit 2022, pp. 138–143. association for computing machinery, new york (2022). https://doi.org/10.1145/3512353.3512373 gondran, e., lucca, g., borges, e.n., santos, h.: a evolução do perfil de estudantes de engenharia com bom desempenho no enade. in: anais do xxxiv simpósio brasileiro de informática na educação (sbie 2023), pp. 1238–1248. sociedade brasileira de computação sbc, brasil (2023). https://doi.org/10.5753/sbie.2023.234529. https://sol.sbc.org.br/index.php/sbie/article/view/26750 govindarajan, k., kumar, v.s., boulanger, d., kinshuk: learning analytics solution for reducing learners’ course failure rate. in: 2015 ieee seventh international conference on technology for education (t4e), pp. 83–90 (2015). https://doi.org/10.1109/t4e.2015.14 hui, h., ming-jie, t., qing-tao, z., xiao-liang, z.: application of student achievement analysis based on apriori algorithm. in: 2020 2nd international conference on information technology and computer application (itca), pp. 19–22 (2020). https://doi.org/10.1109/itca52113.2020.00011 imdad, u., ahmad, w., asif, m., ishtiaq, a.: classification of students results using knn and ann, pp. 1–6 (2017). https://doi.org/10.1109/icet.2017.8281651 junior, f.d.s., brancher, j.: uma pesquisa de opinião sobre a relevância dos conteúdos abrangidos pelo poscomp. in: anais do xxii workshop sobre educação em computação, pp. 90–99. sbc, porto alegre, rs, brasil (2014). https://sol.sbc.org.br/index.php/wei/article/view/10963 marcelo mendes, f., p. mendonça, a., b. guedes, e.: poscomp coach: plataforma web para apoio ao ingresso na pós-graduação em computação. revista novas tecnologias na educação 16(1) (2018). https://doi.org/10.22456/1679-1916.85990. https://seer.ufrgs.br/index.php/renote/article/view/85990 rabeya, m., mahmud, m.s., johora, s.f.t., sattar, a.: a comparison of the application of artificial intelligence in the educational system. in: 2022 international conference on emerging trends in computing and engineering applications (etcea), pp. 1–6 (2022). https://doi.org/10.1109/etcea57049.2022.10009813 russell, s., norvig, p.: inteligência artificial uma abordagem moderna. gen ltc (2022). https://books.google.com.br/books?id=5xf0zweacaaj sakashita, r.g.a., bertolini, d., schwerz, a.l.: predição do conceito enade dos cursos de computação no brasil. in: anais do xxxiv simpósio brasileiro de informática na educação (sbie 2023), pp. 1305–1316. sociedade brasileira de computação sbc, brasil (2023). https://doi.org/10.5753/sbie.2023.234715. https://sol.sbc.org.br/index.php/sbie/article/view/26756 santos batista, e.j., batista, j., souza, r., primo, w.: desenvolvimento de um aplicativo para android com questões do poscomp como um objeto de aprendizagem para o auxílio no ingresso a programas de pós-graduação, p. 127 (2014). https://doi.org/10.5753/cbie.wcbie.2014.127 sbc: exame nacional para ingresso na pós-graduação em computação (poscomp) (2022). https://www.sbc.org.br/educacao/poscomp silva guerra, m., asseiss neto, h., azevedo oliveira, s.: a case study of applying the classification task for students’ performance prediction. ieee lat. am. trans. 16(1), 172–177 (2018). https://doi.org/10.1109/tla.2018.8291470 article  math  google scholar  da silveira, m.e.r., et al.: classificação por matérias das questões do enade e poscomp. revista coming communications and innovations gazette 5(2), 9–19 (2021). https://doi.org/10.5902/2448190462014. https://periodicos.ufsm.br/coming/article/view/62014 vieira, a.d.s., bertolini, d., schwerz, a.l.: análise do desempenho no enade dos concluintes de computação usando técnica de agrupamento. in: anais do xxxiii simpósio brasileiro de informática na educação (sbie 2022), pp. 834–845. sociedade brasileira de computação sbc, brasil (2022). https://doi.org/10.5753/sbie.2022.224710. https://sol.sbc.org.br/index.php/sbie/article/view/22463 wu, m., zhao, h., yan, x., guo, y., wang, k.: student achievement analysis and prediction based on the whole learning process. in: 2020 15th international conference on computer science & education (iccse), pp. 123–128 (2020). https://doi.org/10.1109/iccse49874.2020.9201865 zvyagin, l.: data mining: big data and data science. soft measur. comput. 5(54), 81–90 (2022). https://doi.org/10.36871/2618-9976.2022.05.006. https://s-lib.com/en/issues/smc_2022_05_a6/ download references acknowledgments we thank the brazilian federal agency for support and evaluation of graduate education – capes (capes) for funding the scholarship and the high-performance distributed computing center (ccad) for providing the space for the research development. author information authors and affiliations data intelligence laboratory, federal university of pará, belém, pa, 66075-110, brazil jean carlos costa & reginaldo santos authors jean carlos costaview author publications search author on:pubmed google scholar reginaldo santosview author publications search author on:pubmed google scholar corresponding author correspondence to jean carlos costa . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper costa, j.c., santos, r. (2025). automated and intelligent vocational guidance system for classifying specialties based on poscomp microdata. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_7 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords vocational guidance machine learning poscomp data analysis publish with us policies and ethics profiles jean carlos costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature classifying potentially non-compliant portuguese language sentences concerning privacy policies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper classifying potentially non-compliant portuguese language sentences concerning privacy policies conference paper first online: 31 january 2025 pp 107–121 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) classifying potentially non-compliant portuguese language sentences concerning privacy policies download book pdf download book epub matheus tocchini  orcid: orcid.org/0000-0002-1064-34789,10, igor m. rocha  orcid: orcid.org/0000-0002-0132-76099,10, raphael m. de barros  orcid: orcid.org/0009-0007-7539-55959,10, jéssica o. e silva10, ananda f. garcia10, felipe zular10, juliano maranhão  orcid: orcid.org/0000-0002-2705-74409,10 & … jaime simão sichman  orcid: orcid.org/0000-0001-8924-96439,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 245 accesses abstract privacy policies (ppol) are extensive and contain complex sentences that are difficult to understand. they detail what happens with a person’s personal data and must comply with specific legislation. one way to assess the compliance of ppol with legislation is through machine learning models. there are some studies already carried out, aiming to detect compliance with the gdpr, basically analyzing ppol in english. in this work, we present a mapping of brazilian data protection legislation into 27 categories, divided into 3 blocks, and 3 levels of potential compliance. we also introduced a corpus in portuguese, with ppol sentences annotated through mapping of brazilian legislation. we evaluated some classifier models in a task of detecting potentially non-compliant sentences and another task of categorizing potentially non-compliant sentences. we achieved performance close to the literature for studies in english and the gdpr. our study points out ways to improve the automated assessment of ppol and highlights the complexity of the tasks that seek to ensure compliance with data protection legislation. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others establishing a strong baseline for privacy policy classification chapter © 2020 on gdpr compliance of companies’ privacy policies chapter © 2019 a contrastive study of preand post-legislation interaction design for communication and action about personal data protection in e-commerce websites chapter © 2019 1 introduction a privacy policy is a document that informs the holders of all the processes, treatments and security measures that a company carries out with their personal data [9]. however, almost 20 years ago, studies already showed that the readability and comprehension of most of these documents requires a graduation degree [20]. furthermore, ppols are extensive documents, requiring a considerable period of time to be read [31]. with the enactment of legislation to protect personal data, such as the gdpr (general data protection resolution) [43] in european territory and the lgpd (lei geral de proteção de dados, in portuguese general data protection law, in english) [8] in brazil, ppol would need to evolve in order to eliminate gaps that were already known and, also, meet legal requirements. one way to ensure that privacy policies are being improved and in compliance with data protection laws is through the use of algorithms for automated analysis. more specifically, a combination of natural language processing (nlp) and ml techniques [26]. a lot of effort has been put into extracting information from ppol. different methods were applied to identify and retrieve relevant keywords and text parts [3]. the work presented in [45] directed studies on ppol for the use of supervised ml. the development of corpora that mapped the concepts that were desired to be extracted from policies into categories that could be classified through ml gained prominence, as there were no data sets with these characteristics until then. various approaches to evaluating ppol in relation to gdpr compliance have been studied [2, 14, 35]. all of them used the gdpr as a legal reference to guide the performance of the proposed models. also, the studies were conducted on english-language documents. thus, there is a gap in relation to works that address other legal norms, in addition to other languages. there is an opportunity to develop research beyond the english language and gdpr, in order to compare results from different languages and laws, and pave the way for the adoption of automated analysis of privacy policies, ensuring compliance with regulatory data protection laws. in order to address this gap described across different languages and legal systems, we developed a mapping of the brazilian data protection law (lgpd) [8] into machine-readable attributes. to this end, we built an annotation guideline that details the application of our mapping to evaluate ppols clauses [33]. the guideline instantiates 3 large blocks of categories, totaling 27 annotation categories. through the guideline, we assembled a corpus with 6431 annotated ppols clauses, with 3 levels of compliance with the lgpd: compliant, partially non-compliant, non-compliant. to the best of our knowledge, this corpus is the first in portuguese that relates the evaluation of ppol from the perspective of the lgpd. we evaluate the corpus using classical ml and deep learning models. we applied it to two different tasks: identifying potentially non-compliant clauses and categorizing potentially non-compliant clauses. the remainder of this paper is organized as follows. we present related work in sect. 2. in sect. 3, we depict the development of the guideline and the construction of our corpus. section 4 describes our experimental analysis and evaluation methodology, and we discuss our results in sect. 5. finally, sect. 6 brings our conclusions and we suggest future directions for our work. 2 related work initially, work on ppol aimed to extract information from documents through rules and build knowledge ontologies to evaluate new ppols. at this stage, studies carried out in [3, 5] capture information and keywords from ppol through lexical rules. an indication of the potential use of ml to evaluate ppol began to emerge with the findings in [6, 47], which brought encouraging evidence of the effectiveness of crowdsourcing to annotate ppol. in [45], wilson et al. present the opp-115 corpus, a pioneer in research on ppol. after opp-115 corpus, lots of works focused on building and improving corpora and applying ml. in [16], the authors propose a formal model for data practices. the works in [11, 12, 46] study improvements of crowdsourcing, in order to automate ppol clause analysis, and zaeem et al. introduces a large ppol corpus, with more than 200,000 ppol gathering from dmoz [34]. the work of gebauer et al. propose a prototype of a human-in-the-loop combined with ml to annotate ppol [19]. privacyglue is introduced in [39] as a benchmark to measure general language understanding in the privacy domain. in [36], the authors present question-answering methods to retrieve informations from privacy policies. bhatia et al. introduce a way to identify incompleteness in privacy policies by using semantic frames [7]. kotal et al. show that techniques to extract knowledge from ppol fail when the text policies are ambiguous [24]. yang et al. propose an approach for automatically extracting purpose-aware rules from ppol [49], while kumar et al. present an approach to extract and classify automatically opt-out choices found in ppol [4]. in [32], the authors build a corpus from a rule-based information extraction of ppol for internet of things apps. a proposal of a privacy score framework, based in ml, is presented in [23]. another aspect that developed after opp-115 was the assessment of ppol in relation to data protection legislation, especially gdpr. the development of methodologies that combine the mapping of gpdr into categories and the creation of a corpus with privacy risk classes are addressed in [14, 41]. the inspiring work presented in [25] combines the development of a corpus with gdpr-guided classes and the application of ml to classify it. in [35], the authors compare article 5 from gdpr with opp-115 annotations, in order to validate the corpus applicability. we also highlight the studies in [2, 18, 29, 37, 50], which are based on gdpr to extract information from ppol, and the work of [28], which introduces a corpus to evaluate the structure of ppol for android apps, based on gdpr articles. we highlight 4 more works in languages other than english. in [30], the authors built a knowledge base from language processing and unsupervised learning techniques, using ppol in italian. the work in [51] introduces a chinese privacy dataset for sequence labeling task and compliance identification studies. an arabic dataset for ml applications in ppol is presented in [1]. finally, we highlight the work of correia et al. [15], in which they developed a recognizer of named entities in portuguese for legal entities, based on texts from the brazilian supreme court. our work differs from the others revised in three points: (i) we mapped data protection law into attributable categories in ml models, which is done through brazilian legislation, unlike others, which use european legislation or even just linguistic or conceptual rules; (ii) we constructed a detailed guideline on mapping lgpd and how to apply it to annotate raw texts, presenting a reproducible model for legislation other than gdpr; (iii) we have used texts in portuguese, since almost all researches are carried out with documents in english. 3 annotation guideline and corpus development the work to create the guideline and obtain the corpus was based on the theory proposed in [21]. in this work, hovy et al. brings 7 steps for methodological annotation of a corpus: (i) selecting representative texts; (ii) instantiating the theory; (iii) choosing and training annotators; (iv) defining the annotation process; (v) designing the annotation interface; (vi) determining and applying evaluation metrics; (vii) corpus maintenance. also, our work was inspired by the research developed in the claudette project, with the construction of a corpus relating the gdpr [43] with ppols and the evaluation of the data set through ml models [13, 25]. following the theoretical framework in [21], sect. 3.1 details the theory applied to develop the guideline, while sect. 3.2 describes the other corpus construction steps, such as text capture, the annotation interface and the preparation of annotators. 3.1 annotation guideline we started the guideline work through the study and instantiation of the lgpd theory [8], which corresponds to the set of standards responsible for the protection of personal data in brazilian territory. our systematic analysis of the lgpd resulted in an initial version with more than 30 categories, separated into 3 large blocks (data omission, data processing, unclear language and general). the guideline underwent 3 revisions. we made changes such as including new categories, excluding some existing ones and adding others into a single category (neutralization of the theory [21]). after revisions, the final version contains 27 categories. the data omission block has 18 categories, data processing has 7 categories, unclear language and general have one each. table 1 presents all the categories. for the sake of space, we ask to refer to [33] for details on each of the categories within guideline. table 1. categories defined in guideline [33]. they map lgpd blocks into attributable classes for ml.full size table the conceptualization of the categories was accompanied by the definition of the degrees of compliance for each of them. following the work of [13, 25], we classify compliance into 3 levels: level 1 compliance, level 2 potential partial non-compliance, level 3 potential total non-compliance. specific details of the degree of compliance for each category are described in the guideline [33]. also, there is an indication, with its legal basis, regarding the mandatory presence of data evaluated by the category in a ppol. this information was included in the guideline, as it was reported in [25] as an obstacle to delimiting the task that should be automated by the ml model. we illustrate, in fig. 1, the data correction category, with its compliance levels, examples in documents and the column that indicates its mandatory nature in a ppol. fig. 1. example detailing compliance levels for the data correction category [33]. full size image during revisions, our team tagged 2 ppol to evaluate the instantiated theory after each change. this assessment aimed to check whether the categories were consistent with the section of the lgpd that they instantiated, identify ambiguities between categories and find very specific categories that could be added. the entire guideline development process took 5 months, resulting in the initial version. subsequent reviews took another 5 months. in total, we worked for 10 months to reach the final version of the guideline [33]. 3.2 corpus after completing the guideline, we moved on to implementing the other steps of the hovy et al. methodology [21]. we collected documents from companies of different economic sectors, in a similar way to the criteria applied by liepina et al. [25]. our collection efforts resulted in a set of 75 ppol from different companies. we process the ppols in order to transform them into clauses in json format. this process took place in 3 stages. figure 2 details the ppol capture process. first, we parsed ppol text into sentences, using regex split by punctuation (?!.;). in second stage, we reviewed each parsed ppol, in order to remove sentences parsed in a wrong way. the last stage consisted in a transformation of each parsed ppol in text files (one text file per sentence) and conversion to json files. fig. 2. process of capturing and processing privacy policies. source: own authorship. full size image with the ppol sentences in the appropriate format, we chose our annotation tool. we use label studio [42]. within the tool, we configure the annotation interface according to the guideline categories (sect. 3.1). figure 3 shows an example of interface to annotate a sentence. the annotations have two parts: choose a category and, in sequence, determine the compliance level. each sentence can have one or more categories tagged. also, we included two extra categories: “na”, if the sentence has no content to be analyzed using the guideline, and “incorrect”, for sentences parsed with errors that were not identified during the manual review. all sentences tagged with any of these categories were deleted from the final corpus. fig. 3. label studio annotation interface with an example of annotation, showing categories and levels of compliance with lgpd. the screen’s information was written in portuguese. source: own authorship. full size image we selected and trained a group of 5 people to be annotators. all were undergraduate law students at universidade de são paulo. the training consisted of an 1-hour online session to explain the use of the annotation interface and presentation of the annotation guideline. we chose an annotation strategy similar to that recommended in braun [10], which suggests assigning 2 or more annotators per sentence and a more senior expert to resolve disagreements. we started with 3 annotators per clause and weekly meetings to discuss disagreements and reach consensus. if the disagreement persisted, the most senior researcher on our team, a specialist in consumer law and data protection, would make the final decision. after training the annotators, two of them did not progress in their work and decided to leave the group. these withdrawals impacted our strategy and we had to use only one annotator per sentence, in order to meet the working deadline of the annotators. this scenario has the potential to introduce biases into the annotation process. before the withdrawals, the annotators got a fleiss’ kappa \(\kappa <\) 0 (in 266 clauses or 4% of total corpus). this result can be an effect of a phenomenon discussed in sect. 5. the final corpusfootnote 1 took 10 months to be done and has 6341 distinct sentences. highlights include the categories purpose of treatment, category of processed data and data sharing, which correspond to almost 50% of the sentences. around 22% of sentences are potentially non-compliant, that is, they may be in conflict with the lgpd. 4 corpus evaluation the evaluation of the corpus we built was inspired by the methodologies presented in [25, 27]. following the reviewed literature, we divided the experimental work into two tasks: detection of potentially non-compliant sentences and categorization of potential non-compliant sentences. the task of detecting non-compliant sentences consists of a binary classification, with the purpose of identifying sentences that are potentially non-compliant with the lgpd. the task of categorizing non-compliant sentences is a binary classification, one classifier for each category, performed only on sentences with conformity level 3. 4.1 feature representation for the task of detecting potential non-compliant sentences, we use a representation by bag-of-words (bow), with bigrams and trigrams (applying td-idf), and part-of-speech (pos) [38]. we use spacyfootnote 2 and its pos tagger for the portuguese language. we also use an approach for data input into bert models, with the application of tokenization, padding and other auxiliary operations [48]. we chose the bow representation in order to evaluate whether the lexical and grammatical information is sufficient to identify non-compliant sentences. this approach also allows comparison with the results presented in [25, 27]. the representation for the bert model contributes to the incorporation of semantic information into the classification task. regarding the task of categorizing non-compliant sentences, we only employ the bow approach, with pos and bigrams and trigrams. thus, we only consider lexical and grammatical information to categorize sentences, in a similar way to what was proposed in [27]. 4.2 machine learning algorithms regarding the non-compliant sentence detection task, we use two classifiers: support vector machine (svm) [44] and fine-tuned bert pre-trained in portuguese, known as bertimbau [40]. the svm receives the bow representation as input and the bert model and its data preprocessing and representation, as described in sect. 4.1. we chose svm for comparisons with the results in [25], even though the ml model chosen by the authors is a class of svm with some changes. the bert model was our second choice to evaluate the degree of improvement that a model close to the state-of-the-art can bring to the task. in this scenario, all sentences with compliance level 3 are assigned as positive label and others sentences, compliance levels 1 and 2, as negative label. the second task, categorizing potentially non-compliant sentences, was carried out with the svm + bow configuration. in this scenario, we train a model for each compliance category in the guideline. in each trained model, a single category was marked as a positive label and all others as a negative label. only sentences with compliance level 3 are included in this task. in order to address the class imbalance in our corpus, we adopted an algorithmic structure approach for svm: cost-sensitive learning, with a higher weight to minority classes. the models’ performance was evaluated using the f1-score, precision and recall metrics [38]. we selected them because we can compare them with the performance of detecting non-compliant sentences in [25] and with the results for categorizing non-compliant sentences in [27]. 5 results and discussion the experimentsfootnote 3 for the svm non-compliant sentence detection task were performed by splitting the data into 90% for training and 10% for testing. we trained 10 models with 5-fold cross validation. each of them with a different seed to split the dataset and we performed a grid search for the c parameter (search space = [0.05, 0.1, 0.25, 0.5, 1.0, 2.0, 5.0, 10.0, 50.0]), linear kernel, class weight = ‘balanced’ and we fixed a seed for the models. performances in test set were averaged. the bert model used a single dataset split due to computational overburden. we split in 80% for training, 10% for validation and 10% for testing, and was trained for 4 epochs, following the recommendation in [17]. we set batch size = 8, adam optimizer (learning rate = \(2\times 10^{-5}\), \(\epsilon = 1\times 10^{-8}\)). all metrics were verified in the test set. following the evaluation methodology of [25], we added two more baseline classifiers: a classifier that always returns the class of interest, called always positive, and a random classifier, with a probability of the class of interest similar to the proportion in the training set, around 22% (sect. 3.2). table 2 presents the metrics obtained for the potentially non-compliant sentence detection task. table 2. performance of the models for the task of detecting potentially non-compliant sentences.full size table table 3. performance of each classifier in test set for non-compliant sentences categorization. the proportion column refers to the percentage of non-compliant sentences in the category in relation to all non-compliant sentences (compliance level 3). the category blocks, omission of data required by law, data processing and unclear language, are separated by colors, where the data processing block is in gray.full size table the bert model outperforms all others and shows us an encouraging result. we noticed that the baseline always positive performed very close to the svm. this fact made us think about some hypotheses. as svm exploits a lexical and grammatical representation, ppol can contain sentences that require higher linguistic complexity to be understood and become separable. the bert model exploits a representation with semantic and contextual information [22] and presents considerably better performance (around 42%). this is consistent with [25], since the authors relate difficulties with sentences that need the ppol context. when comparing the performance of ppol with online terms of service (tos), work carried out in [27] (f1 = 0.769 for svm + bow), we realize that the lexical and grammatical representations through bow for tos have results that are higher than those we found and those found in [25] (f1 = 0.42\(\,-\,\)0.55). this scenario may reinforce the idea about the greater complexity and difficulty of analyzing ppol, a fact reported in [20]. in line with analyzing the complexity of sentences, we manually studied some sentences that were annotated with more than one category and with different compliance levels. for example, the sentence “você reconhece e concorda que poderemos lhe enviar informações e avisos importantes referentes à sua conta e ao site da comunidade por e-mail, mensagem de texto ou outros meios, com base nas informações que você nos forneceu.” (in english, “you acknowledge and agree that we may send you important information and notices regarding your account and the community site by email, text message or other means based on the information you have provided to us.”) was tagged in categories purpose of treatment, with compliance level 1, category of processed data and advertising, with compliance level 3. the tagged categories are in accordance with the guideline definitions. therefore, a sentence can be compliant or non-compliant, depending on the category. a future work could be the separation of categories into blocks to apply the classification, similar to what was done in [25]. even so, this finding of ours can reinforce the hypothesis of the complexity of the sentences found in ppol. a second hypothesis is related to the bias that the annotation process may have introduced into the corpus. this is a limitation of our work. only one annotator tagged each sentence and may have made some subjective choices that may not be based on the guideline. regarding the second task we designed, the categorization of non-compliant sentences, we have done just a randomized grid search, with a cross validation (5-fold), for each category. we assigned positive label for a category and false label to others, using the same hyperparameters applied to detection task. in total, we have trained 26 svm classifiers. table 3 displays the results found. we observed that the categorization task performed below the task of identifying potentially non-compliant sentences. the category other consents does not present results because it has no sentences with compliance level 3. the work carried out in [27], with tos, had opposite findings. the categorization task performed better. we interpret our findings, compared with the literature, as an indication of the complexity of ppol sentences, just as we raised this hypothesis in the task of detecting non-compliant sentences. two more findings caught our attention. the first was that higher f1-scores are associated with categories with a higher proportion of sentences in the dataset (category of processed data, 3rd party sharing and purpose of treatment). we understand this as an indication of the need to expand the number of sentences in our corpus to improve their quality. the second concerns high recall values regardless of the proportion of the category in the dataset. for example, category “take it or leave it” corresponds to 1.6% of sentences in categorization task and achieved 100% of recall. in other words, the bow representation, exploiting lexical and grammatical information, is sufficient for the svm to identify a category correctly. the problem becomes excessive false positives, probably because the representation of sentences is bringing ambiguities to the models. a future path of research could be to study classifiers’ false positives and identify their causes, in order to evolve the representation of information that the algorithms receive as input from ppol. 6 conclusions and future work in our work, we introduced a corpus of more than 6,400 ppol sentences annotated using a standardized guideline that maps lgpd in machine-readable categories. we defined 27 categories, separated into 3 blocks, which can qualify a sentence and 3 levels of compliance. we evaluated our corpus in two classification tasks, in order to verify the ability to analyze ppol sentences in an automated way. the first task was the detection of potentially non-compliant sentences. we use two approaches, svm + bow, exploiting lexical and grammatical information, and a bert model, which incorporates semantic information. the svm + bow model did not outperform the baselines we employed. the bert model presented an encouraging result, close to that found in [25]. we interpret these results as an indication that it is possible to detect ppol sentences potentially non-compliant with the lgpd. regarding the categorization of potentially non-compliant sentences, we understand that our results point to the need for more advanced representations of sentences and models closer to the state-of-the-art to achieve more satisfactory performances that are close to the literature. we also highlight the complexity of ppol texts in portuguese that our work revealed. we believe that our findings contribute to the improvement of nlp research applied to digital law in portuguese. as future developments for this work, we intend to expand the size of our corpus, in order to increase the examples of each category. we think that the larger volume of data has the potential to improve the performance of the tested models. a second future activity is to improve the annotation process, employing more annotators per sentence, in order to follow the initial strategy that we proposed and recommended in [10]. finally, we wish to apply the bert model to categorize potentially non-compliant clauses and evaluate the reasons for the excess of false positives found in our studies. notes 1.for more detailed information about the corpus: https://doi.org/10.5281/zenodo.13371639. 2.https://spacy.io. 3.code developed for experiments is available in https://github.com/mtstocchini/claudinha_project. references al-khalifa, h., mashaabi, m., al-yahya, g., alnashwan, r.: the saudi privacy policy dataset (2023) google scholar  alshamsan, a.r., chaudhry, s.a.: a gdpr compliant approach to assign risk levels to privacy policies. comput. mater. continua 74(3), 4631–4674 (2023). https://doi.org/10.32604/cmc.2023.034039 audich, d.a., dara, r., nonnecke, b.: extracting keyword and keyphrase from online privacy policies. in: 2016 eleventh international conference on digital information management (icdim), pp. 127–132 (2016). https://doi.org/10.1109/icdim.2016.7829792 bannihatti kumar, v., et al.: finding a choice in a haystack: automatic extraction of opt-out statements from privacy policy text. in: proceedings of the web conference 2020. www ’20, pp. 1943–1954. association for computing machinery, new york, ny, usa (2020). https://doi.org/10.1145/3366423.3380262 bhatia, j., breaux, t.d.: towards an information type lexicon for privacy policies. in: 2015 ieee eighth international workshop on requirements engineering and law (relaw), pp. 19–24 (2015). https://doi.org/10.1109/relaw.2015.7330207 bhatia, j., breaux, t.d., schaub, f.: mining privacy goals from privacy policies using hybridized task recomposition. acm trans. softw. eng. methodol. 25(3) (2016). https://doi.org/10.1145/2907942 bhatia, j., evans, m.c., breaux, t.d.: identifying incompleteness in privacy policy goals using semantic frames. requir. eng. 24(3), 291–313 (2019). https://doi.org/10.1007/s00766-019-00315-y brasil: lei geral de proteção de dados pessoais (lgpd) (2018). https://www.planalto.gov.br/ccivil03/ato2015--2018/2018/lei/l13709.htm brasil: guia de elaboração de termo de uso e política de privacidade (2023). https://www.gov.br/governodigital/pt-br/privacidade-e-seguranca/ppsi/guia_termo_uso_politica_privacidade.pdf braun, d.: i beg to differ: how disagreement is handled in the annotation of legal machine learning data sets. artif. intell. law (2023). https://doi.org/10.1007/s10506-023-09369-4 article  math  google scholar  chrysakis, i., et al.: cap-a: a suite of tools for data privacy evaluation of mobile applications. in: legal knowledge and information systems. frontiers in artificial intelligence and applications, vol. 334, pp. 269–272. ios press (2020). https://doi.org/10.3233/faia200881 chrysakis, i., et al.: a rewarding framework for crowdsourcing to increase privacy awareness. in: barker, k., ghazinour, k. (eds.) dbsec 2021. lncs, vol. 12840, pp. 259–277. springer, cham (2021). https://doi.org/10.1007/978-3-030-81242-3_15 chapter  math  google scholar  contissa, g., et al.: claudette meets gdpr: automating the evaluation of privacy policies using artificial intelligence. technical report (2018) google scholar  contissa, g., et al.: automated processing of privacy policies under the eu general data protection regulation. in: palmirani, m. (ed.) legal knowledge and information systems, pp. 51–60. no. 313 in front. artif. intell. appl. ios press (2018). https://doi.org/10.3233/978-1-61499-935-5-51 correia, f.a., et al.: fine-grained legal entity annotation: a case study on the brazilian supreme court. inf. process. manag. 59(1), 102794 (2022). https://doi.org/10.1016/j.ipm.2021.102794 article  mathscinet  math  google scholar  d’aquin, m., et al.: privonto: a semantic framework for the analysis of privacy policies. semant. web 9(2), 185–203 (2018) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2018) google scholar  elluri, l., chukkapalli, s.s.l., joshi, k.p., finin, t., joshi, a.: a bert based approach to measure web services policies compliance with gdpr. ieee access 9, 148004–148016 (2021). https://doi.org/10.1109/access.2021.3123950 article  math  google scholar  gebauer, m., maschhur, f., leschke, n., grünewald, e., pallas, f.: a human-in-the-loop approach for information extraction from privacy policies under data scarcity (2023) google scholar  milne, g.r., culnan, m.j., greene, h.: a longitudinal assessment of online privacy notice readability. j. publ. policy market. 25(2), 238–249 (2006) google scholar  hovy, e., lavid, j.: towards a ‘science’ of corpus annotation: a new methodological challenge for corpus linguistics. int. j. transl. 22(1) (2010) google scholar  jurafsky, d., martin, j.h.: speech and language processing (draft) (2021). https://web.stanford.edu/~jurafsky/slp3/ kim, n., oh, h., choi, j.k.: a privacy scoring framework: automation of privacy compliance and risk evaluation with standard indicators. j. king saud univ. comput. inf. sci. 35(1), 514–525 (2023). https://doi.org/10.1016/j.jksuci.2022.12.019 article  math  google scholar  kotal, a., joshi, a., pande joshi, k.: the effect of text ambiguity on creating policy knowledge graphs. in: 2021 ieee international conference on parallel and distributed processing with applications, big data and cloud computing, sustainable computing and communications, social computing and networking (ispa/bdcloud/socialcom/sustaincom), pp. 1491–1500 (2021) google scholar  liepina, r., et al.: gdpr privacy policies in claudette: challenges of omission, context and multilingualism. in: ashley, k.d., et al. (eds.) proceedings of the third workshop on automated semantic analysis of information in legal text (asail 2019), vol. 2385. montreal, qc, canada (2019). https://ceur-ws.org/vol-2385/paper9.pdf lippi, m., et al.: consumer protection requires artificial intelligence. nat. mach. intell. 1(4), 168–169 (2019). https://doi.org/10.1038/s42256-019-0042-3 article  google scholar  lippi, m., et al.: claudette: an automated detector of potentially unfair clauses in online terms of service. artif. intell. law 27(2), 117–139 (2019). https://doi.org/10.1007/s10506-019-09243-2 article  math  google scholar  liu, s., zhang, f., zhao, b., guo, r., chen, t., zhang, m.: appcorp: a corpus for android privacy policy document structure analysis. front. comput. sci. 17(3), 173320 (2022). https://doi.org/10.1007/s11704-022-1627-2 article  math  google scholar  liu, s., zhao, b., guo, r., meng, g., zhang, f., zhang, m.: have you been properly notified? automatic compliance analysis of privacy policy text with gdpr article 13. in: proceedings of the web conference 2021. www ’21, pp. 2154–2164. association for computing machinery, new york, ny, usa (2021). https://doi.org/10.1145/3442381.3450022 martinelli, f., marulli, f., mercaldo, f., marrone, s., santone, a.: enhanced privacy and data protection using natural language processing and artificial intelligence. in: 2020 international joint conference on neural networks (ijcnn), pp. 1–8 (2020). https://doi.org/10.1109/ijcnn48605.2020.9206801 mcdonald, a.m., cranor, l.f.: the cost of reading privacy policies. i/s: j. law policy inf. soc. 540–565 (2008) google scholar  miller, v., candelario, j.r.r., ray, l., perez, a.j.: assessing purpose-extraction for automated corpora annotations. in: 2022 ieee 19th international conference on mobile ad hoc and smart systems (mass), pp. 718–719 (2022). https://doi.org/10.1109/mass56207.2022.00107 moraes rocha, i., marques de barros, r., fernandes garcia, a., de oliveira e silva, j., zular, f., maranhão, j.: guidelines claudinha data protection law (lgpd) (2024). https://doi.org/10.5281/zenodo.13371432 nokhbeh zaeem, r., barber, k.s.: a large publicly available corpus of website privacy policies based on dmoz. in: proceedings of the eleventh acm conference on data and application security and privacy. codaspy ’21, pp. 143–148. association for computing machinery, new york, ny, usa (2021). https://doi.org/10.1145/3422337.3447827 poplavska, e., norton, t.b., wilson, s., sadeh, n.: from prescription to description: mapping the gdpr to a privacy policy corpus annotation scheme. in: legal knowledge and information systems, pp. 243–246. front. artif. intell. appl. ios press (2020). https://doi.org/10.3233/faia200874 ravichander, a., black, a.w., wilson, s., norton, t., sadeh, n.: question answering for privacy policies: combining computational and legal perspectives. in: inui, k., jiang, j., ng, v., wan, x. (eds.) proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 4947–4958. association for computational linguistics, hong kong, china (2019) google scholar  sánchez, d., viejo, a., batet, m.: automatic assessment of privacy policies under the gdpr. appl. sci. 11(4) (2021). https://doi.org/10.3390/app11041762 sebastiani, f.: machine learning in automated text categorization. acm comput. surv. 34(1), 1–47 (2002) google scholar  shankar, a., waldis, a., bless, c., andueza rodriguez, m., mazzola, l.: privacyglue: a benchmark dataset for general language understanding in privacy policies. appl. sci. 13(6) (2023). https://doi.org/10.3390/app13063701 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, 20–23 october (2020, to appear) google scholar  tesfay, w.b., hofmann, p., nakamura, t., kiyomoto, s., serna, j.: i read but don’t agree: privacy policy benchmarking using machine learning and the eu gdpr. in: companion proceedings of the the web conference 2018. www ’18, pp. 163–166. international world wide web conferences steering committee, republic and canton of geneva, che (2018). https://doi.org/10.1145/3184558.3186969 tkachenko, m., malyuk, m., holmanyuk, a., liubimov, n.: label studio: data labeling software (2020–2022). https://github.com/heartexlabs/label-studio european union, t.: general data protection regulation (2016). https://gdpr-info.eu vapnik, v.n.: the vicinal risk minimization principle and the svms. in: vapnik, v.n. (ed.) the nature of statistical learning theory. statistics for engineering and information science, pp. 267–290. springer, new york (2000). https://doi.org/10.1007/978-1-4757-3264-1_9 chapter  math  google scholar  wilson, s., et al.: the creation and analysis of a website privacy policy corpus. in: erk, k., smith, n.a. (eds.) proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1330–1340. association for computational linguistics, berlin, germany (2016) google scholar  wilson, s., et al.: analyzing privacy policies at scale: from crowdsourcing to automated annotations. acm trans. web 13(1) (2018) google scholar  wilson, s., et al.: crowdsourcing annotations for websites’ privacy policies: can it really work? in: proceedings of the 25th international conference on world wide web. www ’16, pp. 133–143. international world wide web conferences steering committee, republic and canton of geneva, che (2016) google scholar  wolf, t., et al.: huggingface’s transformers: state-of-the-art natural language processing (2020) google scholar  yang, l., chen, x., luo, y., lan, x., chen, l., meddahi, a.: purext: automated extraction of the purpose-aware rule from the natural language privacy policy in iot. sec. commun. netw. 2021 (2021). https://doi.org/10.1155/2021/5552501 zaeem, r.n., barber, k.s.: the effect of the gdpr on privacy policies: recent progress and future promise. acm trans. manag. inf. syst. 12(1) (2020). https://doi.org/10.1145/3389685 zhao, k., et al.: a fine-grained chinese software privacy policy dataset for sequence labeling and regulation compliant identification. in: goldberg, y., kozareva, z., zhang, y. (eds.) proceedings of the 2022 conference on empirical methods in natural language processing, pp. 10266–10277. association for computational linguistics, abu dhabi, united arab emirates (2022). https://doi.org/10.18653/v1/2022.emnlp-main.700 download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001. we would like to thank instituto lawgorithm for partially supporting our research. we also thank the usp-aws agreement for the cloud environment support to the development of our model. author information authors and affiliations universidade de são paulo, são paulo, brazil matheus tocchini, igor m. rocha, raphael m. de barros, juliano maranhão & jaime simão sichman instituto lawgorithm, são paulo, brazil matheus tocchini, igor m. rocha, raphael m. de barros, jéssica o. e silva, ananda f. garcia, felipe zular, juliano maranhão & jaime simão sichman authors matheus tocchiniview author publications search author on:pubmed google scholar igor m. rochaview author publications search author on:pubmed google scholar raphael m. de barrosview author publications search author on:pubmed google scholar jéssica o. e silvaview author publications search author on:pubmed google scholar ananda f. garciaview author publications search author on:pubmed google scholar felipe zularview author publications search author on:pubmed google scholar juliano maranhãoview author publications search author on:pubmed google scholar jaime simão sichmanview author publications search author on:pubmed google scholar corresponding author correspondence to matheus tocchini . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper tocchini, m. et al. (2025). classifying potentially non-compliant portuguese language sentences concerning privacy policies. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_8 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning privacy policy natural language processing data protection law lgpd publish with us policies and ethics profiles jaime simão sichman view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature de-identification of clinical notes using contextualized language models and a token classifier | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper de-identification of clinical notes using contextualized language models and a token classifier conference paper first online: 28 november 2021 pp 33–41 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) de-identification of clinical notes using contextualized language models and a token classifier download book pdf download book epub joaquim santos  orcid: orcid.org/0000-0002-0581-409210, henrique d. p. dos santos  orcid: orcid.org/0000-0002-2410-353611, fábio tabalipa  orcid: orcid.org/0000-0002-2060-953x12 & … renata vieira  orcid: orcid.org/0000-0003-2449-547710  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1270 accesses 4 citations abstract the de-identification of clinical notes is crucial for the reuse of electronic clinical data and is a common named entity recognition (ner) task. neural language models provide a great improvement in natural language processing (nlp) tasks, such as ner, when they are integrated with neural network methods. this paper evaluates the use of current state-of-the-art deep learning methods (bi-lstm-crf) in the task of identifying patient names in clinical notes, for de-identification purposes. we used two corpora and three language models to evaluate which combination delivers the best performance. in our experiments, the specific corpus for the de-identification of clinical notes and a contextualized embedding with word embeddings achieved the best result: an f-measure of 0.94. this work was partially supported by institute of artificial intelligence in healthcare, memed, google latin america research awards, and by fct under the project uidb/00057/2020 (portugal). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others research of clinical named entity recognition based on bi-lstm-crf article 07 june 2018 a study of deep learning methods for de-identification of clinical notes in cross-institute settings article open access 05 december 2019 clinical named entity recognition methods: an overview chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. clinical medicine computational linguistics language processing medical humanities natural language processing (nlp) predictive medicine 1 introduction with the growth of artificial intelligence, predictions based on electronic health records (ehrs) are becoming increasingly viable. as ehrs consolidate information from all patient timelines, they may be processed as input for machine learning models to build clinical decision support systems (cdss), to improve hospital workflows. several studies show the potential of using ehr data for comorbidity index prediction [24], fall event detection [26], and bringing smart screening systems to pharmacy services [23], along with many other possibilities [13]. removing patient-identifiable health information is a prerequisite to conduct research over clinical notes [5]. in the united states, the health insurance portability and accountability act (hipaa) listed 18 types of personally identifiable information (pii) [7]. de-identification is the process of finding and removing pii from electronic health records. de-identification is a typical named entity recognition (ner) problem, part of the natural language processing (nlp) field. the current state-of-the-art approach for ner problems is bi-lstm-crf (bidirectional long-short-term-memory conditional random fields) [2], evaluated over a standard conll-2003 corpus [19]. in this paper, we evaluate the use of neural network topology of bi-lstm-crf using two corpora of patient name recognition in portuguese. one patient-name-recognition corpus was annotated with 5 thousand names in 15 thousand sentences. besides, a standard ner corpus was used to evaluate overfitting. our approach focuses on the identification of the best cutoff point of the de-identification training set. the rest of this paper is organized as follows: sect. 2 presents previous works on the de-identification of clinical notes. section 3 describes the datasets, language models, and neural network topology. we present the results in sect. 4. finally, in sect. 5, we conclude and present future work. 2 related work the de-identification of clinical notes has become a common task in the health information research field since the adoption of electronic health records [14]. datasets such as i2b2 [29] and cegs n-grid [28] enable shared tasks in this research field, only for the english language. in a recent survey, leevy et al. [12] performed a systematic review of de-identification tasks in clinical notes using recurrent neural networks (rnns) and conditional random fields (crfs). they found 12 papers that used rnns, 12 others that used crfs, and finally 5 types of research that combine both approaches. also, the authors consider that overfitting may be an issue when customized de-identification datasets are used during model training. most researches found in the survey lack the use of contextualized embeddings for the proposed task. the only exception is the work by lee et al. [11]. in this investigation, they evaluate the de-identification performance of gated recurrent units (grus) versus lstm topologies, with a final crf layer, with three types of embeddings: word-level embeddings, character-level embeddings, and contextualized embeddings. to the best of our knowledge, no previous studies address the detection of patient names based on contextualized embeddings and token classification for the portuguese language. we also evaluate the overfitting problem employing a non-customized corpus used for entity recognition in the general domain. in the next section, we cover the dataset, the neural network, and the language models (embeddings) used in the experiments. 3 materials and methods in this section, we present the corpora used in the experiments, the neural network topologies, and the language models used in the neural networks for the representation of words. besides, we describe how we evaluate the de-identification task. the data used in the experiments we conducted for this article came from a project developed with several hospitals. ethical approval to use the hospitals datasets in this research was granted by the national research ethics committee under the number 46652521.9.0000.5530. 3.1 data source we used two sources of named entity recognition (ner) corpora for the sequence labeling task. the corpora are listed below: ehr-names corpus: this corpus was built by collecting clinical notes from the aforementioned hospital (hnsc). we randomly selected 2,500 clinical notes for each hospital, totaling 15,952 sentences. the annotation process was performed by two annotators in the doccano annotation tool [16]. the annotated dataset includes 12,762 tokens; 4,999 of them were labeled as a named entity. harem corpora: the harem corpora was an initiative of the linguateca consortium. the first harem corpus [21] was published in 2006 with a human-made gold standard of named entities. the second harem corpus [20] was made available in 2008 with basically the same purpose. however, being a more recent contribution, the gold standard for the second harem consists in a slightly better test case. the harem corpora classify the named entities into ten categories. nevertheless, for the purposes of our comparison, only the “person" category was taken into account. considering the “person" category, the number of named entities is 1,040 for the first harem and 2,035 for the second harem. 3.2 neural network in this section, we present the neural networks used for the downstream task evaluated in this study. for the neural networks, we used the flair framework [1], developed in pytorchfootnote 1; it has all the features of parameter tuning for model regulation. all our experiments were performed in nvidia t4 gpu. bilstm-crf: as  [2], we used a traditional sequence labeling task to learn how to detect names at the token level. our sequence labeling is the product of training a bilstm neural network with a final crf layer for token labeling. bidirectional long short-term memory (bilstm) networks have achieved state-of-the-art results in nlp downstream tasks, mainly for sequential classifications [9, 17, 27]. recurrent neural networks have proven to be an effective approach to language modeling, both in sequence labeling tasks such as part-of-speech tagging and in sequence classification tasks such as sentiment analysis and topic classification [10]. when it receives a sentence, this network topology concatenates information in both directions of the sentence. this makes the network have a larger context window, providing greater disambiguation of meanings and more accurate automatic feature extraction [8]. 3.3 language models we-nilc: these are pre-computed language models that feature vectors generated from a large corpus of brazilian portuguese and european portuguese, from varied sources and genres. seventeen different corpora were used, totaling 1.3 billion tokens [6]. we-ehr-notes: for our experiments, we used a word embedding (we) language model to vectorize the tokens of the clinical notes. we used a we model that was previously trained by [22], where the authors reported the best results when using this we trained with the neural network fasttext [3]. the model was developed with 603 million tokens from clinical notes extracted from electronic medical records. the generated model has 300 dimensions per word. this model resulted in a vocabulary of 79,000 biomedical word vectors. flairbbp: this is a pre-trained model taught [25] with three corpora totaling 192 thousand sentences and 4.9 billion tokens in portuguese. this language model was trained using flair embedding [2]. flair is a contextual embedding model that takes into account the distribution of sequences of characters instead of only sequences of words, as is the case for skip-gram and cbow language models. additionally, flair embeddings combine two models, one trained in a forward direction and one trained in a backward direction. flairbbp\(_{fntg}\): this model was built using the aforementioned model flairbbp fine-tuned by using the ehr-names corpus as input. 3.4 design of the experiments our evaluations were made within two sets of experiments: i) stacking embeddings and ii) corpus addition. experiments from set i aim to find the best combination of language models covered by this study. therefore, we started by evaluating a bilstm-crf neural network with shallow word embeddings only; then we employed flairbbp, which is a contextualized language model; we stacked flairbbp with a we; and finally, we performed the experiments with the stacking of the fine-tuned flairbbp with a we. table 1. number of sentences per training splitfull size table then, we performed set ii to evaluate the best cutoff point concerning the size of the training corpus of the model. our strategy was to take several pieces of the corpus and train a model with each of these pieces in a cumulative sequence. in other words, we did these experiments by adding a corpus. this means that we divided the corpus ehr-names into 18 parts with 709 sentences each. additionally, we added three more parts that are formed by the union of the ehr-names corpus and one part of the harem corpus. each part of harem also has 709 sentences. the corpora were added cumulatively, that is, the first split has 709 sentences, the second has 1,418, the third, 2.836, and so on. each split corresponds to a trained model to find out which are the best metrics concerning the size of the corpus. table 1 shows details of the sizes. 4 results table 2. results of stacking embeddings. pre = precision; rec = recall; f1 = f-measurefull size table fig. 1. experimental results by splits. pre = precision; rec = recall; f1 = f-measure full size image experiments from set i are shown in table 2. in all the experiments in set i, we used word2vec skip-gram (w2v-skpg) [15] word embeddings. experiments from set ii are summarized in a plot in the image 1. below we present our results and their interpretations from the comparison of the f1 metrics. with the first set of experiments, we can see the impact of contextualized language models: the contextualized model flairbbp outperforms we-nilc by and we-ehr-notes by . another important aspect of the contextualized language models is the potential combination with a shallow we, forming a final embedding with a high representational power of the language. therefore, we managed to overcome both language models alone and we arrived at the best result of \(f_1 = 0.9413\), using flairbbp stacked with we-nilc. our best result and consequently our best combination of embeddings were also found in [22, 25]. in addition, our results with the stacks using flairbbp\(_{fntg}\) performed close to the average of the four stacks and both were slightly better than the stack of flairbbp + we-ehr-notes. from the second set of experiments, in which we looked for the best number of sentences for training, we found that the metrics precision, recall, and f1 tend to grow up to split nine. after this split, there is a greater oscillation, as we can see in the graph of fig. 1. analyzing all the splits, we can see that split nine is the best, as it demands fewer corpus resources, naturally reduces the effort of the manual annotation of texts, and also reduces the computational cost, since the corpus is smaller. regarding the last three splits—which use the harem corpus—we can see that there is a significant drop in the metrics. we understand that the main reason for this is the difference in textual style between a medical record and the harem texts, extracted from magazines, newspapers, and interviews, among other genres, which are not related to health. 5 conclusion this paper presents an experimental study on the de-identification of patient names in electronic medical records. for the anonymization process, we manually developed a corpus in portuguese with around 2,526 named entities and added the traditional harem corpus to our experiments. we used a bilstm-crf neural network to train a de-identifier model and concluded that using contextualized language models stacked with word embedding models produces cutting-edge results. additionally, our results show that it is not always necessary to have the entire training corpus to obtain the best predictor model and also that mixing very different textual genres can hinder the performance of the model. as future work, we plan to run experiments in generating language models, such as t5 [18] and gpt-3 [4]. all the content of the work (algorithm, sample dataset, language models, and experiments) is available at the project’s github pagefootnote 2 in order to be easily replicated. notes 1.https://pytorch.org. 2.https://github.com/noharm-ai/noharm-anony. references akbik, a., bergmann, t., blythe, d., rasul, k., schweter, s., vollgraf, r.: flair: an easy-to-use framework for state-of-the-art nlp. in: proceedings of the conference of the north american chapter of the association for computational linguistics, pp. 54–59. association for computational linguistics, minneapolis, minnesota, june 2019. https://doi.org/10.18653/v1/n19-4010, https://www.aclweb.org/anthology/n19-4010 akbik, a., blythe, d., vollgraf, r.: contextual string embeddings for sequence labeling. in: proceedings of the 27th international conference on computational linguistics, pp. 1638–1649 (2018) google scholar  bojanowski, p., grave, e., joulin, a., mikolov, t.: enriching word vectors with subword information. trans. assoc. comput. linguist. 5, 135–146 (2017) article  google scholar  brown, t.b., et al.: language models are few-shot learners. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) proceedings of the 33th annual conference on neural information processing systems (2020) google scholar  el emam, k.: guide to the de-identification of personal health information. crc press, boca raton (2013) google scholar  hartmann, n., fonseca, e., shulby, c., treviso, m., silva, j., aluísio, s.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: proceedings of the 11th brazilian symposium in information and human language technology, pp. 122–131 (2017) google scholar  hash, j., bowen, p., johnson, a., smith, c., steinberg, d.: an introductory resource guide for implementing the health insurance portability and accountability act (hipaa) security rule. us department of commerce, technology administration, national institute of \(\ldots \) (2005) google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  google scholar  jiang, y., hu, c., xiao, t., zhang, c., zhu, j.: improved differentiable architecture search for language modeling and named entity recognition. in: proceedings of the 2019 conference on empirical methods in natural language processing, pp. 3585–3590. association for computational linguistics, hong kong, china (2019) google scholar  jurafsky, d., martin, j.h.: speech and language processing, vol. 3. pearson, london, united kingdom (2014) google scholar  lee, k., filannino, m., uzuner, ö.: an empirical test of grus and deep contextualized word representations on de-identification. in: medinfo, pp. 218–222 (2019) google scholar  leevy, j.l., khoshgoftaar, t.m., villanustre, f.: survey on rnn and crf models for de-identification of medical free text. j. big data 7(1), 1–22 (2020) article  google scholar  magboo, ma. sheila a.., coronel, andrei d..: data mining electronic health records to support evidence-based clinical decisions. in: chen, yen-wei., zimmermann, alfred, howlett, robert j.., jain, lakhmi c.. (eds.) innovation in medicine and healthcare systems, and multimedia. sist, vol. 145, pp. 223–232. springer, singapore (2019). https://doi.org/10.1007/978-981-13-8566-7_22 chapter  google scholar  meystre, s.m., friedlin, f.j., south, b.r., shen, s., samore, m.h.: automatic de-identification of textual documents in the electronic health record: a review of recent research. bmc med. res. methodol. 10(1), 1–16 (2010) article  google scholar  mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space. in: bengio, y., lecun, y. (eds.) proceedings of the 1st international conference on learning representations (2013) google scholar  nakayama, h., kubo, t., kamura, j., taniguchi, y., liang, x.: doccano: text annotation tool for human (2018). software available from https://github.com/doccano/doccano peters, m.e., et al.: deep contextualized word representations. in: proceedings of the conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 2227–2237 (2018) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21, 140:1–140:67 (2020) google scholar  sang, e.f., de meulder, f.: introduction to the conll-2003 shared task: language-independent named entity recognition. arxiv preprint cs/0306050 (2003) google scholar  santos, d., freitas, c., oliveira, h.g., carvalho, p.: second harem: new challenges and old wisdom. in: international conference on computational processing of the portuguese language. pp. 212–215. springer (2008). https://doi.org/10.1007/978-3-540-85980-2_22 santos, d., seco, n., cardoso, n., vilela, r.: harem: an advanced ner evaluation contest for portuguese. in: quot; in: calzolari, n., et al. (ed.) proceedings of the 5th international conference on language resources and evaluation (lrec 2006), genoa italy 22–28 may 2006 (2006) google scholar  dos santos, h.d.p., silva, a.p., maciel, m.c.o., burin, h.m.v., urbanetto, j.s., vieira, r.: fall detection in ehr using word embeddings and deep learning. in: 2019 ieee 19th international conference on bioinformatics and bioengineering (bibe), pp. 265–268, october 2019. https://doi.org/10.1109/bibe.2019.00054 dos santos, h.d.p., ulbrich, a.h.d., woloszyn, v., vieira, r.: ddc-outlier: preventing medication errors using unsupervised learning. ieee j. biomed. health inform. 23, 8 (2018) google scholar  dos santos, h.d.p., ulbrich, a.h.d., woloszyn, v., vieira, r.: an initial investigation of the charlson comorbidity index regression based on clinical notes. in: 2018 ieee 31st international symposium on computer-based medical systems (cbms), pp. 6–11. ieee (2018) google scholar  santos, j., consoli, b.s., dos santos, c.n., terra, j., collovini, s., vieira, r.: assessing the impact of contextual embeddings for portuguese named entity recognition. in: proceedings of the 8th brazilian conference on intelligent systems, pp. 437–442 (2019) google scholar  santos, j., dos santos, h.d., vieira, r.: fall detection in clinical notes using language models and token classifier. in: 2020 ieee 33rd international symposium on computer-based medical systems (cbms), pp. 283–288. ieee (2020) google scholar  straková, j., straka, m., hajic, j.: neural architectures for nested ner through linearization. in: proceedings of the 57th annual meeting of the association for computational linguistics, pp. 5326–5331. association for computational linguistics (2019) google scholar  stubbs, a., filannino, m., uzuner, ö.: de-identification of psychiatric intake records: overview of 2016 cegs n-grid shared tasks track 1. j. biomed. inform. 75, s4–s18 (2017) article  google scholar  stubbs, a., uzuner, ö.: annotating longitudinal clinical narratives for de-identification: the 2014 i2b2/uthealth corpus. j. biomed. inform. 58, s20–s29 (2015) article  google scholar  download references acknowledgments we thank dr. ana helena d. p. s. ulbrich, who provided the clinical notes dataset from the hospital, for her valuable cooperation. we also thank the volunteers of the institute of artificial intelligence in healthcare celso pereira and ana lúcia dias, for the dataset annotation. author information authors and affiliations cidehus, university of évora, évora, portugal joaquim santos & renata vieira institute of artificial intelligence in healthcare, porto alegre, brazil henrique d. p. dos santos memed, clinical intelligence department, são paulo, brazil fábio tabalipa authors joaquim santosview author publications search author on:pubmed google scholar henrique d. p. dos santosview author publications search author on:pubmed google scholar fábio tabalipaview author publications search author on:pubmed google scholar renata vieiraview author publications search author on:pubmed google scholar corresponding author correspondence to henrique d. p. dos santos . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper santos, j., dos santos, h.d.p., tabalipa, f., vieira, r. (2021). de-identification of clinical notes using contextualized language models and a token classifier. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_3 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords de-identification clinical notes language model token classifier publish with us policies and ethics profiles renata vieira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature exploring artificial intelligence methods for the automatic measurement of a new biomarker aiming at glaucoma diagnosis | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper exploring artificial intelligence methods for the automatic measurement of a new biomarker aiming at glaucoma diagnosis conference paper first online: 12 october 2023 pp 455–469 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) exploring artificial intelligence methods for the automatic measurement of a new biomarker aiming at glaucoma diagnosis download book pdf download book epub gabriel c. fernandes9, fabio lavinsky9, sandro josé rigo9 & … henrique c. bohn9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 510 accesses abstract using technologies capable of providing retina structure high-resolution images is one of the most widespread means of identifying structural changes that may indicate the onset or progression of visual impairment. automated glaucoma detection using optical coherence tomography is still considered an area needing further research. several manual analyzes are currently performed over the generated by imaging equipment. this work presents an approach to foster automatic glaucoma evaluation considering convolutional neural networks for semantic segmentation of retinal layers through optical coherence tomography images and image processing for measuring the cup region in the optic nerve head portion. we provide a quantitative evaluation comparing the results obtained by a specialist physician. the work’s main contribution consists of the first approach supporting the automation of a new biomarker for diagnosing glaucoma. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others artificial intelligence and glaucoma chapter © 2021 automated detection of glaucoma using deep learning convolution network (g-net) article 03 april 2019 ai and glaucoma chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition computational intelligence computer vision ophthalmology artificial intelligence computer imaging, vision, pattern recognition and graphics 1 introduction visual impairments and blindness can appear at any stage of life, but people over 50 are more likely to have some level of disability [6]. monitoring through periodic examinations is very important as it makes it possible to identify any change in the structure of the retina. follow-up through clinical examinations allows physicians to have an initial assessment, accompanied by intraocular pressure (iop) measurements, a field of view testing, and structural imaging parameters. in cases of glaucoma, the disease may begin with a minimal disorder in the nerve fibers, almost invisible, and the loss of retinal ganglion cells (rgc). if it progresses to the intermediate stage, changes already become noticeable in the retinal nerve fiber layers (rnfl), along with peripheral vision loss. finally, when fully installed, it presents serious damage and great vision loss [10]. glaucoma is a chronic and progressive optic neuropathy characterized by the death of rgc associated with increased cupping of the optic nerve head (onh) and a corresponding loss of visual field (vf). several studies have shown that the lamina cribosa (lc) has a role in the pathophysiology of glaucoma. the lc is the major area related to mechanical damage in glaucoma, and the detailed assessment of disease-related changes at that site is of the most relevant interest in understanding factors concerning the development and progression of glaucoma. furthermore, andrade and coworkers have also proposed that lc structural differences could be responsible for different effects of iop on the tissue, contributing to individual susceptibility to glaucomatous damage mediated by the iop [1]. the relationship between structural and functional damage is relevant for diagnosing glaucoma. the structural evaluation of the onh, retinal nerve fiber layers (rnfl), and macula, when analyzed by optical coherence tomography (oct), can provide important clinical information for the diagnosis of glaucoma and the evaluation of its progression [11]. optical coherence tomography, a technology capable of providing high-resolution images of the structure of the retina, is one of the most commonly used means of inspecting it, providing access to the visualization of the retinal layers to monitor changes in thickness and possible deformations [12]. detecting progress with visual field tests or optic disc imaging methods is limited due to the retrospective nature and high variability. currently, functional and structural clinical tests suffer from the need for analysis and the significant time delay required to establish disease progression and response to treatment. so does the progress of the ganglion cell loss over that time. there is, therefore, a significant unmet need for glaucoma-related biomarkers to improve clinical trials, both for early diagnosis and detection of disease progression. biomarkers then present a possibility to provide information that will eventually affect the decisions to be taken clinically [3]. combining new technologies, such as imaging methods (oct) and clinical evaluation based on parameters, is one way to assess glaucoma’s criticality continuously. optical coherence tomography provides relevant information such as cup-to-disc ratio (cdr), cup volume, rim area, and rim volume. evaluations carried out by oct demonstrated that regions such as bruch’s membrane opening (bmo) for the internal limiting membrane (ilm) are correlated with the thickness of the rfnl. as well as the posterior lc, a histopathological mark of glaucoma deformation was demonstrated in the deep part of the onh structure. this information, made available by the imaging method, allowed the correlation of both regions, where lavinsky and coworkers formulated the hypothesis that it would be helpful as a quantitative evaluation parameter of onh remodeling in glaucoma [11]. bmo and lc represent the legs of a right triangle and result in the evaluation by discovering the hypotenuse of the vertical region of the onh cup. the results are presented as a morphometric biomarker. studies regarding the automatization of oct image analyses can foster automatic support for evaluating the retina’s structure and identifying diseases such as glaucoma. these approaches aim to perform accurate analyses, especially with large volumes of data [5, 17]. the studies of the retina structures are also supported by the rapid advancement in scanning technologies, which enable more researchers to obtain images with a high amount of information [16, 18]. computational methods such as computer vision and deep learning can support the necessary oct image analysis. they allow the resolution of problems such as segmentation, and pattern identification. this research aimed to contribute improving the way of extracting information related to a new biomarker on the progress of glaucoma. the importance of the study is directly related to the possibility of an accurate analysis of the retinal layers and the possibility of extracting information with automatic support. the article is structured as follows: section 2 discusses related work. section 3 describes the approach and architecture performed and its relationship with the problem. section 4 describes the evaluation carried out, and the results obtained. finally, sect. 5 concludes the work. 2 related work this section summarizes the studied articles, presenting the objectives and techniques used. the articles were obtained through a non-systematic review, considering the research topics regarding deep learning applied to ophthalmology. automated diagnosis through oct images has been the focus of many researchers in recent years, showing the relevance and improvements in the investigation for more accurate results. in [13], a review was carried out on several biomarkers that used oct extractions as a basis and the potential to support the identification of diabetic retinopathy, age-related macular degeneration, retinitis pigmentosa, and vitreomacular interface. in conclusion, oct-based imaging biomarkers help detect disease early, classify disease severity (qualitatively and quantitatively), and modify the treatment regimen accordingly. proposed by fu and coworkers [4], the method automatically evaluates the optic disc in representations of an oct slice through detection via low-level reconstruction dealing with noise and shadows of vessels in oct slices. training data were extracted from a single provided oct slice using the anterior retina structure. the method is applied to discover the boundary of the optic disc that appears at the end of the retinal pigment epithelium (rpe). results showed that it was possible to identify the rpe points that connect to the optic disc boundary through the low-level method. similarly, the algorithm for calculating the cup disc ratio (cdr) [10] proposed a process for extracting and contouring the inner limiting membrane layer (ilm), and in calculating the disc diameter, the endpoints of the rpe layer were used to define the disc margin. the automation of retinal segmentation in oct [12] images for diagnosing glaucoma was developed using a convolutional neural network (cnn) architecture, so it was possible to identify the nine layers of the retina and the optic disc for the final segmentation. the following were applied: an optic disc detection neural network (nn), a neural network for the segmentation of retinal layers, and a fusion module. in another study [22], two separate neural networks were used to perform automated segmentation of the optic disc boundary and segmentation of the peripapillary retinal layer, where the final boundary of the peripapillary retina was calculated based on prediction and gradient map, using a multiweight graph search algorithm. the study mainly applied the u-net architecture. a dl framework for segmenting multiple retinal layers and delineating fluid pockets in oct images of the eye [15] was implemented using a fully convolutional neural network (fcn) adapted for semantic segmentation. it is also possible to highlight works relevant to the segmentation of the optic disc’s blood vessels or the segmentation of layers but related to other diseases, such as insufficiently corrected refractive error or age-related macular degeneration. the use of deep learning in these cases was relevant due to its possible generalization and application in cases of glaucoma. the u-net [14] and fcn [19] inspire most architectures presented for clinical image segmentation. there are u-net extensions, which presented a densely connected convolutions [2], or demonstrating the possibility of having multiple u-nets connected in a chain fashion, for the segmentation of blood vessels of the optic disc [23]. further, the partially supervised form of retinal segmentation [18] was used in oct images, which consists of a so-called “student-teacher” approach applied in a set with and without annotation. a new multi-prediction guided attention network (mpg-net) [5] for automated segmentation of the retinal layer in oct images was also created, consisting of two main steps to strengthen the discriminative power of a u-shaped fcn network and perform automated segmentation. lastly, a framework was created to applied segmentation in oct images of normal and pathological eyes, combining a cnn with bidirectional long short-term memory (blstm) [7]. finally, it can be highlighted that oct technology provides a range of information regarding the evaluation and progress of vision-related deformities, such as the evaluation of morphometry parameters and topography of the optic nerve head, calculation of the excavation proportion for the optic disc, cup volume, rim area, and rim volume [11]. these are essential concepts that help in investigations into the evolution of retinal damage. 3 proposed approach some studies demonstrate the importance of obtaining evaluation parameters (biomarkers). one of the ways observed was using the pythagorean theorem [11] to assist in evaluating the vertical optic nerve head using sd-oct enhanced depth imaging (edi). the vertical b-scan with the largest cup seen in infrared is used for hypotenuse measurement. the depth of posterior displacement of the lamina cribrosa (lc), measured from the opening of bruch’s membrane level (bmo), and the length of the excavation between the bmo boundary and the ilm layer, form the sides of a right triangle, used in the calculation (fig. 1 right). the discovery of the hypotenuse is a helpful parameter for the quantitative assessment of optic nerve head remodeling in patients with glaucoma. therefore, the motivation of the work is associated with supporting the identification of the new biomarker, applying computational techniques for the measurement, and quantitative analysis of the identified method. fig. 1. left figure shown each part of retina, in a b-scan. right figure shown the hypotenuse measurement [11]. full size image in order to obtain analysis on the mentioned biomarker, the research was developed in three stages. the first stage includes collecting the oct images with the physician to take notes and select the regions of interest. in the second stage, these images are used as input data in a convolutional neural network for semantic segmentation. in the last stage the output of the neural network, containing the segmentation images, a computer vision algorithm is applied to obtain coordinates that will result in the identification of the region of interest. this last stage is performed by implementing the biomarker approach defined by [11]. the results are compared with those analyzed by the specialist in the manual experiments in a previous study. the fig. 2 shown an overview of each step. fig. 2. general overview full size image 3.1 data annotation this research developed a new annotated dataset to support the experiments. the overall context and main procedures are commented in this section. all original research participants underwent sd-oct (spectralis oct; heidelberg engineering gmbh, dossenheim, germany). moreover, the scans (b-scan and c-scan) were exported in jpeg format files for the current research. each image represents only one eye (right or left), with the union of the two scans in the same image. finally, photoshop image editing software was used to support the cuts, annotations, and export to the desired sizes. as shown in fig. 3, the first step refers to loading the images and selecting the region of interest to perform the cut. all images had the same size after cropping. cropping all images with the same size is essential for input data in the neural network. in supervised neural networks, the image task must have the same pattern as the input data. the second step, shown in the exact figure, is the result of annotating the regions of interest after making the cuts. this step was conducted with a medical student to obtain the layers of the retina as accurately as possible, where files were obtained with the notes of each layer and later merged into the primary image. each layer was exported separately in the annotation process: ilm layer annotation, rpe layer annotation, and the selected region of the retinal layer. the exported clippings were turned into png files because the format allows lossless compression and less loss of quality. the files are organized in different folders in the third step according to each export performed. this distinction is because the neural network requires actual (or ground truth) and annotated data as input. therefore, the annotations images are to use with the image cropped under the same region. fig. 3. annotated data flow. full size image the set of images obtained contains 80 folders with images of both eyes. however, some exclusions were made. folders that did not contain one or both images, corrupted files, and folders with the same images were exclusion criteria. therefore, from the total of 160 images, 134 images remained for use. throughout the experiments, the number of images was also modified. tests were carried out with the entire set of images and new exclusions and separations. tests without images with high contrast or noise were also applied. the same equipment can generate important differences in terms of noise presence or overall illumination in the image. this context fostered some experiments with image subsets, according to these main aspects. we apply some different tests with the images obtained during the research period: test using all data; changing the process for choosing the images, where the images that seemed to present greater similarity in overall aspects were chosen manually; the way of annotating the rpe layer where outlines have been replaced by fills (this aspect was possible because to us it’s essential the limit of rpe layer on level of bmo as shown on fig. 1 left). 3.2 cnn architecture the present work used a neural network to perform tasks related to image segmentation. the use of the u-net architecture [14] was defined as a starting point. improvements were then included, in order to foster the results. the architecture consists of a contraction path to capture context and a symmetric expansion path for localization. the contraction path consists of a typical convolutional network, with two convolutions followed by a relu (rectified linear unit) activation function and a max polling operation. the use of the regularization dropout technique to reduce overfitting was also considered. the number of resource channels doubles each reduction step (downsampling). the expansion path consists of an upsampling of the feature map, followed by an up-convolution that halves the number of feature channels, a concatenation with the corresponding clipped feature map from the contracting path, and two convolutions followed each by the relu function. still, according to the authors, clipping is necessary due to the loss of edge pixels in each convolution. in the final layer, a 1\(\,\times \,\)1 convolution maps each feature vector to the desired number of classes. this is a network with no fully connected layers and which uses only the valid part of each convolution, i.e., the segmentation map only contains the pixels for which the entire context is available in the input image. it also allows the continuous segmentation of arbitrarily large images by a block overlay strategy, predicting the pixels in the region of the edge of the image that, due to the missing context, are extrapolated, mirroring the input image. fig. 4. u-net architecture. full size image changes were made to the architecture (fig. 4) based on two articles, [8, 21]. in both, there is the presence of additional blocks between the path of contraction and expansion. these modules allowed the extraction of information from the semantic context and the generation of high-level resource maps. the dense atrous convolution (dac) module was used for the present work. dac was designed to extract features from objects with different sizes using a set of dilated convolutions. fig. 5. dilated convolution [21]. full size image dilated convolution was initially proposed for computing the wavelet transform. allows insertion of zeros between consecutive values along the spatial dimension. standard convolutions have a rate of change equal to 1, while atrous convolution allows changing this value. this type of operation supports the exponential expansion of receptive fields without loss of resolution or coverage [21] (fig. 5). thus, a dac block was inserted between the contraction and extension paths to extract the features at different scales. based on li and coworkers, the same branch structure was used within the dac module (fig. 6), only changing the feature map from 512 to 1024. fig. 6. dense atrous convolution (dac) [8]. full size image finally, the cross entropy loss function was initially used for model training due to its ability to calculate the difference between two probability distributions, which in this case are the pixels for the background and foreground. however, objects in medical images, such as the optic disc and retinal layers, occupy small regions compared to the whole. according to [8], cross-entropy is not performant in these cases and can be replaced by the loss function known as the dice coefficient, which looks for the correlation between the prediction points and the annotations. 3.3 data augmentation some surveys often have the opportunity to use large amounts of data [22]. however, there are cases in which data collection could not be more voluminous due to access difficulties (restricted data), collection time, and variability to avoid creating bias, among other reasons. all this makes it impossible to have a more malleable set or with large variations. for this reason, the data augmentation technique allows the virtually simulating of new data based on the original input data, helping to increase the amount of data available for experiments. many modifications can be carried out using some specific api (e.g., keras or pytorch). however, it is also possible to carry out manually, implementing codes using libraries such as opencv, which allow greater flexibility in using functions. operations as horizontal flip, rotation, scale and shift was used. 3.4 measurement of cup portion on onh structure the capture, annotation, and prediction process allows a flow to obtaining segmented images by a neural network. this result is then used as a final step to obtain values in the retinal excavation region. in this way, the calculation presented [11] can be carried out and evaluated computationally. the first necessary step is to obtain the coordinates and excavation length value between the bmo boundary and the ilm layer. next, the segmented images of the two layers are joined to be able to search the depth. the depth of posterior displacement of the lamina cribrosa (lc), measured from the line created at the level of opening of bruch’s membrane level (bmo), is then used as a source for obtaining the measurement. the values obtained make it possible to assume that they are the legs in the pythagorean equation. moreover, in this way, obtaining the hypotenuse is discovered in pixel value. the last step is the conversion from pixel to micron, the unit of measurement used by the scanning equipment. this entire process is performed using the python programming language. 4 conversion to micron meter the values found are still in pixels at the end of the search step for regions connected to the cup region. therefore, there is a need for conversion to the measurement unit used by physicians. the conversion process is currently manual since we use a scale available in the image obtained by edi sd-oct. this scale is always found in the lower part of the images and in an “l” shape, where the vertical bar represents a scale, and the horizontal bar represents another scale. using the imagej software, it is possible to convert the pixel distance of each orientation to the displayed reference (200 \(\upmu \)m), as shown in the fig. 7. the returned values are then used as fixed parameters in the flow presented in the previous section. fig. 7. measurements using imagej software. full size image 5 results this section describes the experiments with the neural network architecture and the results obtained together with the results of the final stage, where the discovery of the hypotenuse over the disc region of the cup is performed and compared with the physicians’ measurements. also, it’s important mentioned that for the neural network, the implementation is based on the keras api running under tensorflow. all training was performed using google colab connected to gpus. the nn u-net trained images layers separately: one applied to images of the ilm layer and another to the rpe layer. for training and testing, we separated the images to leave 70% for training and 30% for validation and testing (half for each). different hyperparameters and changes in the number of epochs were applied to train the model. 5.1 evaluation of nn segmentation for the model evaluation, we decided to use mainly two metrics for segmentation tasks. the evaluation metrics were the intersection over union (iou) and f1-score (f1). other metrics, such as recall and precision, were also analyzed together, as they are part of the previous metrics’ equation. as mentioned in the work of rezatofighi and coworkers, iou, the jaccard index, is the most commonly used metric for comparing the similarity between two arbitrary shapes. iou encodes the shape properties of the objects under comparison, e.g., the widths, heights, and locations of two bounding boxes, into the region property and then calculates a normalized measure that focuses on their areas [9]. also, according to another work, both metrics (iou and f1) measure the same aspects and provide the same system classification. therefore, the use of each one may depend on the case [20]. 5.2 results over segmentation using nn it was possible to carry out four tests with the u-net network during the study period. the first one will not be included in the present study since the sample contains less than ten images. the other tests were performed with different numbers of images, changes in hyperparameters, re-annotation of images, and excluding somefootnote 1. table 1. results obtained for the ilm layer.full size table table 2. results obtained for the rpe layer.full size table to outermost layer (ilm), it was possible to notice that there was a slight variation in the results obtained (table 1) where most of the collected images there was little difference in contrast and variation in quality besides being a border region between the background and the scanned retina. in this way, it was possible to obtain results in this layer with less variation. in contrast, the rpe layer initially obtained lower values than the other region (table 2). we noticed that the neural network did not perform well when detecting a more internal part with a thin line (annotation). architectural changes were made but without success. the annotations for the last two tests were then modified, containing thicker annotations, focusing more on the boundary region between the rpe and the bmo since it is vital for the final calculation. the results also show that adjusting the neural network and defining it as an isolated neural network for each image context is necessary. in this way, hyperparameters and configurations apply to each case. 5.3 discovery of the hypotenuse in the excavation region experiments were carried out at this stage to understand the feasibility of using computer vision libraries, such as opencv in python, to identify measurements in the excavation region. the algorithm created to identify what was described in the sect. 3.4 obtained results that were compared with the values obtained by the physician in his study. in table 3, it is possible to see the comparisons between each measured part and the final result, the hypotenuse. the final differences between the measurements taken by the doctor and those found by the algorithm can still be caused by manual measurement and the erroneous reading of some pixels in the final step. there is a need for a larger sample of data for comparison. table 3. results obtained for hypotenuse findings (\(\upmu \)m)full size table as shown in the table 3, columns: length, depth and hypo are the values obtained by the physician. columns with the same name but with sufix “-alg” it is a reference to the algorithm created and described in the sect. 3.4. the column “diff: hypo” show the difference between both results obtained by the physician and the algorithm. the results still are considere high, where it is necessary a new study to evaluate what may have affected this difference. 6 conclusion oct is a modern and sophisticated image extraction technology that can perform excellent results in scanning the retina and inner layers. however, several studies still enable new ways for more accurate disease analysis. in the present study, deep learning focused on semantic segmentation was evaluated to enable the possible results and benefits of the technique and used to perform a mathematical calculation to reproduce the hypotenuse measurement over the optic nerve’s head. the retina structure presents very sensitive characteristics in which detailed analyses are necessary. in this study, a specialist physician and a medical student participated, enabling data availability and helping in the steps of notes taken. it was possible to use neural networks to identify the layers of the retina and their segmentation. the results obtained, even though they need adjustments, were used to measure the region of the cup in the optic nerve head and calculate the hypotenuse. there are still improvements that need to be made, as well as new studies. it also noticed the need to separate the neural network in the future, applying different architectures for each layer of the annotated retina. one limiting point that follows since the beginning of the research is the number of images. in which, as highlighted in sect. 3.1, many images were excluded by quality criteria. there is still a significant challenge in working with variations in images, whether they are: contrast or the presence of noise. different from the quality of the images, even with the cuts made, we use images in large sizes compared to other datasets of different themes. the use of images from only one oct device can be a factor that negatively contributes to the creation of a bias since we do not know the limitations and the pattern of the generated images. still, there is more than one type of glaucoma, which can also affect results generation if the region does not have a similar pathology. moreover, finally, in order to calculate the hypotenuse, there is still a need to investigate how to convert the pixel aspect ratio to the micron unit (\(\upmu \)m) without using secondary software. currently, it is necessary to open some image through the imagej software and to read informations from scale bar, where it is possible know the distances of micron in pixels. this is necessary to convert the results from hypotenuse findings (sect. 4). from future adjustments, both in the neural network and the extraction algorithm, it will be possible to obtain even better results since the hypotenuse measurement over the excavation region is simple and reproducible. moreover, it contributes to the possibility of making this approach viable as a support tool in diagnosing patients with glaucoma. the ethics committee of the hospital de clinicas de porto alegre approved the study and it was conducted in accordance with the declaration of helsinki. informed consent was obtained from all participants. notes 1.to table 1 and table 2, was abbreviate some columns. meaning of: t: total images, q1: quantity of train images, and q2: quantity of test images. references andrade, j.c.f., kanadani, f.n., furlanetto, r.l., lopes, f.s., ritch, r., prata, t.s.: elucidation of the role of the lamina cribrosa in glaucoma using optical coherence tomography. surv. ophthalmol. 67(1), 197–216) (2022). https://doi.org/10.1016/j.survophthal.2021.01.015 azad, r., asadi-aghbolaghi, m., fathy, m., escalera, s.: bi-directional convlstm u-net with densley connected convolutions. in: proceedings 2019 international conference on computer vision workshop, iccvw 2019, pp. 406–415 (2019). https://doi.org/10.1109/iccvw.2019.00052 beykin, g., norcia, a.m., srinivasan, v.j., dubra, a., goldberg, j.l.: discovery and clinical translation of novel glaucoma biomarkers. progress retinal eye res. 80 (2021). https://doi.org/10.1016/j.preteyeres.2020.100875 fu, h., xu, d., lin, s., wong, d.w., liu, j.: automatic optic disc detection in oct slices via low-rank reconstruction. ieee trans. biomed. eng. 62(4), 1151–1158 (2015). https://doi.org/10.1109/tbme.2014.2375184 article  google scholar  fu, z., et al.: mpg-net: multi-prediction guided network for segmentation of retinal layers in oct images. in: european signal processing conference, pp. 1299–1303, january 2021. https://doi.org/10.23919/eusipco47968.2020.9287561 gbd 2019 blindness and vision impairment collaborators: vision loss expert group of the global burden of disease study. causes of blindness and vision impairment in 2020 and trends over 30 years, and prevalence of avoidable blindness in relation to vision 2020: the right to sight: an analysis for the global burden of disease study [published correction appears in lancet glob health. 2021 apr; 9(4):e408]. lancet glob health 9(2), e144–e160 (2021). https://doi.org/10.1016/s2214-109x(20)30489-7 gopinath, k., rangrej, s.b., sivaswamy, j.: a deep learning framework for segmentation of retinal layers from oct images. in: proceedings 4th asian conference on pattern recognition, acpr 2017, pp. 894–899 (2021). https://doi.org/10.1109/acpr.2017.121 gu, z., et al.: ce-net: context encoder network for 2d medical image segmentation. ieee trans. med. imaging 38(10), 2281–2292 (2019). https://doi.org/10.1109/tmi.2019.2903562 article  google scholar  rezatofighi, h., tsoi, n., gwak, j., sadeghian, a., reid, i., savarese, s.: generalized intersection over union: a metric and a loss for bounding box regression. corr (2019). https://doi.org/10.48550/arxiv.1902.09630 khalil, t., akram, m.u., raja, h., jameel, a., basit, i.: detection of glaucoma using cup to disc ratio from spectral domain optical coherence tomography images. ieee access 6, 4560–4576 (2018). https://doi.org/10.1109/access.2018.2791427 article  google scholar  khalil, t., akram, m.u., raja, h., jameel, a., basit, i.: detection of glaucoma using cup to disc ratio from spectral domain optical coherence tomography images. ieee access 6, 4560–4576 (2018). https://doi.org/10.1109/access.2018.2791427 li, j., et al.: multi-scale gcn-assisted two-stage network for joint segmentation of retinal layers and disc in peripapillary oct images. biomed. opt. express 12, 2204–2220 (2021) google scholar  phadikar, p., saxena, s., ruia, s., lai, t.y.y., meyer, c.h., eliott, d.: the potential of spectral domain optical coherence tomography imaging based retinal biomarkers. int. j. retina vitreous 3(1), 1–10 (2017). https://doi.org/10.1186/s40942-016-0054-7 article  google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. ieee access 1, 16591–16603 (2015). https://doi.org/10.1109/access.2021.3053408 article  google scholar  roy, a.g., et al.: relaynet: retinal layer and fluid segmentation of macular optical coherence tomography using fully convolutional networks. biomed. opt. express 8, 3627–3642 (2017). https://doi.org/10.1364/boe.8.003627 sander, b., larsen, m., thrane, l., hougaard, j.l., jørgensen, t.m.: enhanced optical coherence tomography imaging by multiple scan averaging. br. j. ophthalmol. 89(2), 207–212 (2005). https://doi.org/10.1136/bjo.2004.045989 article  google scholar  schmidt-erfurth, u., sadeghipour, a., gerendas, b.s., waldstein, s.m., bogunović, h.: artificial intelligence in retina. prog. retin. eye res. 67, 1–29 (2018). https://doi.org/10.1016/j.preteyeres.2018.07.004 article  google scholar  sedai, s., et al.: uncertainty guided semi-supervised segmentation of retinal layers in oct images. in: shen, d., et al. (eds.) miccai 2019. lncs, vol. 11764, pp. 282–290. springer, cham (2019). https://doi.org/10.1007/978-3-030-32239-7_32 chapter  google scholar  shelhamer, e., long, j., darrell, t.: fully convolutional networks for semantic segmentation. ieee trans. pattern anal. mach. intell. 39(4), 640–651 (2017). https://doi.org/10.1109/tpami.2016.2572683 article  google scholar  taha, a.a., hanbury, a.: metrics for evaluating 3d medical image segmentation: analysis, selection, and tool. bmc med. imaging (2015). https://doi.org/10.1186/s12880-015-0068-x article  google scholar  yu, f., koltun, v.: multi-scale context aggregation by dilated convolutions. in: 4th international conference on learning representations, iclr 2016 conference track proceedings (2016) google scholar  zang, p., wang, j., hormel, t.t., liu, l., huang, d., jia, y.: automated segmentation of peripapillary retinal boundaries in oct combining a convolutional neural network and a multi-weights graph search. biomed. opt. express 10(8), 4340 (2019). https://doi.org/10.1364/boe.10.004340 article  google scholar  zhuang, j.: laddernet: multi-path networks based on u-net for medical image segmentation, pp. 2–5 (2019) google scholar  download references author information authors and affiliations universidade do vale do rio dos sinos (unisinos), são leopoldo, brazil gabriel c. fernandes, fabio lavinsky, sandro josé rigo & henrique c. bohn authors gabriel c. fernandesview author publications search author on:pubmed google scholar fabio lavinskyview author publications search author on:pubmed google scholar sandro josé rigoview author publications search author on:pubmed google scholar henrique c. bohnview author publications search author on:pubmed google scholar corresponding author correspondence to gabriel c. fernandes . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper fernandes, g.c., lavinsky, f., rigo, s.j., bohn, h.c. (2023). exploring artificial intelligence methods for the automatic measurement of a new biomarker aiming at glaucoma diagnosis. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_30 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning semantic segmentation glaucoma publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature optimizing diffusion rate and label reliability in a graph-based semi-supervised classifier | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper optimizing diffusion rate and label reliability in a graph-based semi-supervised classifier conference paper first online: 28 november 2021 pp 514–527 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) optimizing diffusion rate and label reliability in a graph-based semi-supervised classifier download book pdf download book epub bruno klaus de aquino afonso  orcid: orcid.org/0000-0003-2086-105410 & lilian berton  orcid: orcid.org/0000-0003-1397-600510  part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 753 accesses 3 citations 1 altmetric abstract semi-supervised learning has received attention from researchers, as it allows one to exploit the structure of unlabeled data to achieve competitive classification results with much fewer labels than supervised approaches. the local and global consistency (lgc) algorithm is one of the most well-known graph-based semi-supervised (gssl) classifiers. notably, its solution can be written as a linear combination of the known labels. the coefficients of this linear combination depend on a parameter \(\alpha \), determining the decay of the reward over time when reaching labeled vertices in a random walk. in this work, we discuss how removing the self-influence of a labeled instance may be beneficial, and how it relates to leave-one-out error. moreover, we propose to minimize this leave-one-out loss with automatic differentiation. within this framework, we propose methods to estimate label reliability and diffusion rate. optimizing the diffusion rate is more efficiently accomplished with a spectral representation. results show that the label reliability approach competes with robust \(\ell _1\)-norm methods and that removing diagonal entries reduces the risk of overfitting and leads to suitable criteria for parameter selection. this study was financed in part by the coordenação de aperfeiçoamento de nível superior brasil (capes) finance code 001, and são paulo research foundation (fapesp) grant #18/01722-3. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others discriminative clustering with representation learning with any ratio of labeled to unlabeled data article 29 january 2022 model change active learning in graph-based semi-supervised learning article open access 17 february 2024 semi-supervised uncertain linear discriminant analysis chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms graph theory graph theory in probability learning algorithms machine learning statistical learning 1 introduction machine learning (ml) is the subfield of computer science that aims to make a computer learn from data [11]. the task we’ll be considering is the problem of classification, which requires the prediction of a discrete label corresponding to a class. before the data can be presented to an ml model, it must be represented in some way. our input is nothing more than a collection of n examples. each object of this collection is called an instance. for most practical applications, we may consider an instance to be a vector of d dimensions. graph-based semi-supervised learning (gssl) relies a lot on matrix representations, so we will be representing the observed instances as an input matrix (1) a ml classifier should learn to map an input instance to the desired output. to do this, it needs to know the labels (i.e. the output) associated with instances contained in the training set. in semi-supervised learning, only some of the labels are known in advance [3]. once again, it is quite convenient to use a matrix representation, referred to as the (true) label matrix $$\begin{aligned} \mathbf {y}_{ij} = {\left\{ \begin{array}{ll} 1 &{}\text { if the }i\text {-th instance is associated with the }j\text {-th class}\\ 0 &{}\text {otherwise} \end{array}\right. } \end{aligned}$$ (2) most approaches are inductive in nature so that we can predict the labels of instances not seen before deployment. however, most gssl methods are transductive, which simply means that we are only interested in the fixed but unknown set of labels corresponding to unlabeled instances [17]. accordingly, we may represent this with a classification matrix: (3) in order to separate the labeled data \(\mathcal {l}\) from the unlabeled data \(\mathcal {u}\), we divide our matrices as following: $$\begin{aligned} \mathbf {x}&= \left[ \mathbf {x_\mathcal {l}}^\top , \mathbf {x_\mathcal {u}}^\top \right] ^\top \end{aligned}$$ (4) $$\begin{aligned} \mathbf {y}&= \left[ \mathbf {y_\mathcal {l}}^\top , \mathbf {y_\mathcal {u}}^\top \right] ^\top \end{aligned}$$ (5) $$\begin{aligned} \mathbf {f}&= \left[ \mathbf {f_\mathcal {l}}^\top , \mathbf {f_\mathcal {u}}^\top \right] ^\top \end{aligned}$$ (6) the idea of ssl is appealing for many reasons. one of them is the possibility to integrate the toolset developed for unsupervised learning. namely, we may use unlabeled data to measure the density \(p( \mathbf {x})\) within our d-dimensional input space. once that is achieved, the only thing left is to take advantage of this information. to do this, we have to make use of assumptions about the relationship between the input density \(p( \mathbf {x})\) and the conditional class distribution \(p(y\!\mid \! \mathbf {x})\). if we are not assuming that our datasets satisfy any kind of assumption, ssl can potentially cause a significant decrease compared to baseline performance [14]. this is currently an active area of research: safe semi-supervised learning is said to be attained when ssl never performs worse than the baseline, for any choice of labels for the unlabeled data. this is indeed possible in some limited circumstances, but also provably impossible for others, such as for a specific class of margin-based classifiers [9]. in fig. 1, we illustrate which kind of dataset is suitable for gssl. there are two clear spirals, one corresponding to each class. in a bad dataset for ssl, we can imagine the spiral structure to be a red herring, i.e. something misguiding. a very common assumption for ssl is the smoothness assumption, which is one of the cornerstones for gssl classifiers. it states that “if two instances \( \mathbf {x}_1\), \( \mathbf {x}_2\) in a high-density region are close, then so should be the corresponding outputs \(y_1\), \(y_2\)” [3]. another important assumption is that the (high-dimensional) data lie (roughly) on a low-dimensional manifold, also known as the manifold assumption [3]. if the data lie on a low-dimensional manifold, the local similarities will approximate the manifold well. as a result, we can, for example, increase the resolution of our image data without impacting performance. graph-based semi-supervised learning has been used extensively for many different applications across different domains. in computer vision, these include car plate character recognition [19] and hyperspectral image classification [13]. gssl is particularly appealing if the underlying data has a natural representation as a graph. as such, it has been a promising approach for drug property prediction from the structure of molecules [8]. moreover, it has had much application in knowledge graphs, such as the development of web-scale recommendation systems [16]. fig. 1. an ideal scenario for semi-supervised learning full size image gssl methods put a greater emphasis on using geodesics by expressing connectivity between instances through the creation of a graph. many successful deep semi-supervised approaches use a similar yet slightly weaker assumption, namely that small perturbations in input space should cause little corresponding perturbation on the output space [12]. it turns out that we can best express our concepts by defining a measure of similarity, instead of distance. in particular, we search for an affinity matrix , such that $$\begin{aligned} \mathbf {w}_{ij} = {\left\{ \begin{array}{ll} w( \mathbf {x}_i, \mathbf {x}_j)\in \mathbb {r} &{}\text {if } \mathbf {x}_i\text { and } \mathbf {x}_j\text { are considered neighbors}\\ 0 &{}\text {otherwise} \end{array}\right. } \end{aligned}$$ (7) where w is some function determining the similarity between any two instances \( \mathbf {x}_i, \mathbf {x}_j\). when constructing an affinity matrix in practice, instances are not considered neighbors of themselves, i.e. we have \(\forall i \in \{1..n\}: \mathbf {w}_{ii} = 0\). the specification of an affinity matrix is a necessary step for any gssl classifier, and its sparsity is often crucial for reducing computational costs. there are many ways to choose a neighborhood. most frequently, it is constructed by looking at the k-nearest neighbors (knn) of a given instance. one last important concept to gssl is that of the graph laplacian operator. this operator is analogue to the laplace-beltrami operator on manifolds. there are a few graph laplacian variants, such as the combinatorial laplacian $$\begin{aligned} \mathbf {l}_{\mathbb {c}}= \mathbf {d} \mathbf {w} \end{aligned}$$ (8) where \( \mathbf {d}\), called the degree matrix, is a diagonal matrix whose entries are the sum of each row of \( \mathbf {w}\). there is also the normalized laplacian, whose diagonal is the identity matrix \(\mathbf {i}\): $$\begin{aligned} { \mathbf {l}_{\mathbb {n}}= \mathbf {i} \mathbf {d}^{-\frac{1}{2}} \mathbf {w} \mathbf {d}^{-\frac{1}{2}} = \mathbf {d}^{-\frac{1}{2}} \mathbf {l}_{\mathbb {c}} \mathbf {d}^{-\frac{1}{2}}} \end{aligned}$$ (9) each graph laplacian \( \mathbf {l}\) induces a measure of smoothness with respect to the graph on a given classification matrix \( \mathbf {f}\), namely $$\begin{aligned} {\widetilde{s}_ \mathbf {l}}( \mathbf {f}) = \frac{1}{2} \sum _{k=1}^{c} ( \mathbf {f}_{[:,k]})^{\top } \mathbf {l}( \mathbf {f}_{[:,k]}) = \frac{1}{2} tr( \mathbf {f}^{\top } \mathbf {l} \mathbf {f}) \end{aligned}$$ (10) where tr is the trace of the matrix and c the number of classes. if we consider each column \( \mathbf {f}\) of \( \mathbf {f}\) individually, then we can express graph smoothness of each graph laplacian as $$\begin{aligned} \mathbf {f}^{\top } \mathbf {l}_{\mathbb {c}} \mathbf {f}&= \sum _{1 \le i,j \le n} \mathbf {w}_{ij} ( \mathbf {f}_i \mathbf {f}_j)^2 \end{aligned}$$ (11) $$\begin{aligned} \mathbf {f}^{\top } \mathbf {l}_{\mathbb {n}} \mathbf {f}&= \sum _{1 \le i,j \le n} \left( \frac{ \mathbf {w}_{ij}}{\sqrt{ \mathbf {d}_{ii}}} \mathbf {f}_i \frac{ \mathbf {w}_{ij}}{\sqrt{ \mathbf {d}_{jj}}} \mathbf {f}_j\right) ^2 \end{aligned}$$ (12) each graph laplacian also has an eigendecomposition: $$\begin{aligned} \mathbf {l} = \mathbf {u} \mathbf {\lambda } \mathbf {u}^\top \end{aligned}$$ (13) where the set of columns of \( \mathbf {u}\) is an orthonormal basis of eigenfunctions, with \( \mathbf {\lambda }\) a diagonal matrix with the eigenvalues. as \( \mathbf {u}\) is a unitary matrix, any real-valued function on the graph may be expressed as a linear combination of eigenfunctions. we map a function to the spectral domain by pre-multiplying it by \( \mathbf {u}^\top \) (also called graph fourier transform, also known as gft). additionally, pre-multiplying by \( \mathbf {u}\) gives us the inverse transform. this spectral representation is very useful, as eigenfunctions that are smooth with respect to the graph have smaller eigenvalues. outright restricting the amount of eigenfunctions is known as smooth eigenbasis pursuit [6], a valid strategy for semi-supervised regularization. in this work, we explore the problem of parameter selection for the local and global consistency model. this model yields a propagation matrix, which itself depends on a fixed parameter \(\alpha \) determining the diffusion rate. we show that, by removing the diagonal in the propagation matrix used by our baseline, a leave-one-out criterion can be easily computed. our first proposed algorithm attempts to calculate label reliability by optimizing the label matrix, subject to constraints. this approach is shown to be competitive with robust \(\ell _1\)-norm classifiers. then, we consider the problem of optimizing the diffusion rate. doing this in the usual formulation of lgc is impractically expensive. however, we show that the spectral representation of the problem can be exploited to easily solve an approximate version of the problem. experimental results show that minimizing leave-one-out error leads to good generalization. the remaining of this paper is organized as follows. section 2 summarizes the key ideas and concepts that are related to our work. section 3 presents our two proposed algorithms: \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) for determining label reliability, and \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) for determining the optimal diffusion rate parameter. our methodology is detailed in sect. 4, going over the basic framework and baselines. the results are presented in sect. 5. lastly, concluding remarks are found in sect. 6. 2 related work in this section, we present some of the algorithms and concepts that are central to our approach. we describe the inner workings of our baseline algorithm, and how eliminating diagonal entries of its propagation matrix may lead to better generalization. 2.1 local and global consistency the local and global consistency (lgc) [17] algorithm is one of the most widely known graph-based semi-supervised algorithms. it minimizes the following cost: $$\begin{aligned} \mathcal {q}( \mathbf {f}) = \frac{1}{2} \left( tr( \mathbf {f}^{\top } \mathbf {l}_{\mathbb {n}} \mathbf {f}) + \mu \big \vert \mathbf {f} \mathbf {y}\big \vert ^2 \right) \end{aligned}$$ (14) lgc addresses the issue of label reliability by introducing the parameter \(\mu \in (0, \infty )\). this parameter controls the trade-off between fitting labels, and achieving high graph smoothness. lgc has an analytic solution. to see this, we take the partial derivative of the cost with respect to \( \mathbf {f}\): $$\begin{aligned} \frac{\partial ^{} \mathcal {q}}{\partial \mathbf {f}}&= \frac{1}{2} \frac{\partial ^{} tr( \mathbf {f}^{\top } \mathbf {l}_{\mathbb {n}} \mathbf {f})}{\partial \mathbf {f}} + \frac{1}{2} \mu \frac{\partial ^{} \big \vert \mathbf {f} \mathbf {y}\big \vert ^2}{\partial \mathbf {f}}\end{aligned}$$ (15) $$\begin{aligned}&= ((1+\mu ) \mathbf {i} \mathbf {s}) \mathbf {f} \mu \mathbf {y} \end{aligned}$$ (16) where \( \mathbf {s} = \mathbf {d}^{-\frac{1}{2}} \mathbf {w} \mathbf {d}^{-\frac{1}{2}} = \mathbf {i} \mathbf {l}_{\mathbb {n}}\). by dividing the above by (\(1+ \mu \)), we observe that this derivative is zero exactly when $$\begin{aligned} (i \alpha \mathbf {s}) \mathbf {f} = \beta \mathbf {y} \end{aligned}$$ (17) with $$\begin{aligned} \alpha = \frac{1}{1+\mu } \in (0,1) \end{aligned}$$ (18) and $$\begin{aligned} \beta = 1 \alpha \end{aligned}$$ (19) the matrix \(( \mathbf {i} \alpha \mathbf {s})\) can be shown to be positive-definite and therefore invertible, so the optimal \( \mathbf {f}\) can be obtained as $$\begin{aligned} \mathbf {f} = \beta (i \alpha \mathbf {s})^{-1} \mathbf {y} \end{aligned}$$ (20) we hereafter refer to \((i \alpha s)^{-1}\) as the propagation matrix \( \mathbf {p}\). each entry \( \mathbf {p}_{ij}\) represents the amount of label information from \(x_j\) that \(x_i\) inherits. it can be shown that the inverse is a result of a diffusion process, which is calculated via iteration: $$\begin{aligned}&f(0) = \mathbf {y}\end{aligned}$$ (21) $$\begin{aligned}&f(t\!+\!1) = \alpha \mathbf {s} f(t) + (1\!-\!\alpha ) \mathbf {y} \end{aligned}$$ (22) moreover, it can be shown that the closed expression for f at any iteration is $$\begin{aligned} f(t) = (\alpha \mathbf {s})^{t-1} \mathbf {y} + (1-\alpha )\sum _{i=0}^{t-1}(\alpha \mathbf {s})^{i} \mathbf {y} \end{aligned}$$ (23) s is similar to \(d^{-1}w\), whose eigenvalues are always in the range \([-1,1]\) [17]. this ensures the first term vanishes as t grows larger, whereas the second term converges to \( \mathbf {p}y\). consequently, \( \mathbf {p}\) can be characterized as $$\begin{aligned} \mathbf {p}&= (1-\alpha )\lim _{t \rightarrow \infty } \sum _{i=0}^{t}(\alpha \mathbf {s})^{i}\end{aligned}$$ (24) $$\begin{aligned}&= (1-\alpha )\lim _{t \rightarrow \infty } \sum _{i=0}^{t} \alpha ^i \mathbf {d}^{\frac{1}{2}}( \mathbf {d}^{-1} \mathbf {w})^{i} \mathbf {d}^{-\frac{1}{2}} \end{aligned}$$ (25) the transition probability matrix \(\widetilde{w} = \mathbf {d^{-1}w}\) makes it so we can interpret the process as a random walk. let us imagine a particle walking through the graph according to the transition matrix. assume it began at a labeled vertex \(v_a\), and at step i it reaches a labeled vertex \(v_b\), initially labeled with class \(c_b\). when this happens, \(v_a\) receives a confidence boost to class \(c_b\). alternatively, one can say that this boost goes to the entry which corresponds to the contribution from \(v_b\) to \(v_a\), i.e. \( \mathbf {p}_{ab}\). this boost is proportional to \(\alpha ^i\). this gives us a good intuition as to the role of \(\alpha \). more precisely, the contribution of vertices found later in the random walk decays exponentially according to \(\alpha ^i\). 2.2 self-influence and leave-one-out-error there is one major problem with lgc’s solution: the diagonal of \( \mathbf {p}\). at first glance, we would think that “fitting the labels” means looking for a model that explains our data very well. in reality, this translates to memorizing the labeled set. the main problem resides within the diagonal of the propagation matrix. any entry \( \mathbf {p}_{ii}\) stores the self-influence of a vertex, which is calculated according to the expected reward obtained by looping around and visiting itself. the optimal solution w.r.t. label fitting occurs when \(\alpha \) tends to zero. for labeled instances, an initial reward is given for the starting vertex itself, and the remaining are essentially ignored. we argue that the diagonal is directly related to overfitting. it essentially tells the model to rely on the label information it knows. there are a few analogies to be made: say that we are optimizing the number of neighbors k for a knn classifier. the analog of “lgc-style optimal label fitting” would be to include each labeled instance as a neighbor to itself, and set \(k = 1\). this is obviously not a good criterion. the answer that maximizes a proper “label fitting” criteria, in this case, is selecting k that minimizes classification error with the extremely important caveat: directly using each instance’s own label is prohibited. in spite of the problems we have presented, the family of lgc solutions remains very interesting to consider. we will have to eliminate diagonals, however. let $$\begin{aligned} \mathbf {h}(\alpha ) := \left( \mathbf {( \mathbf {i}-\alpha \mathbf {s})^{-1}} diag(( \mathbf {i}-\alpha \mathbf {s})^{-1})\right) _\mathcal {l} \mathbf {y}_\mathcal {l}\end{aligned}$$ (26) by eliminating the diagonal, we obtain, for each label, its classification if it were not included in the label propagation process. as such, it can be argued that minimizing $$\begin{aligned} \big \vert \mathbf {h}(\alpha ) \mathbf {y}_\mathcal {l}\big \vert ^2 \end{aligned}$$ (27) also minimizes the leave-one-out (loo) error. there is an asterisk: each instance is still used as unlabeled data, but this effect should be insignificant. it is also interesting to row-normalize (a small constant \(\epsilon \) may be added for stabilization) the rows, so that we end up with classification probabilities. previously, we developed a semi-supervised leave-one-out filter [2, 4]. we also managed to reduce the amount of storage used by only calculating a propagation submatrix. the loo-inspired criterion encourages label information to be redundant, so labels that are incoherent with the implicit model are removed. the major drawback of our proposal is that it needs an extra parameter r, which is the number of labels to remove. the optimal r is usually around the number of noisy labels, which is unknown to us. this was somewhat addressed in [4]: we can instead use a threshold, which tells us how much labels can deviate from the original model. nonetheless, it is desirable to solve this problem in a way that removes such a parameter. we will do this by introducing a new optimization problem. 3 proposal in this work, we consider the optimal value of \({\alpha }\) for our loo-inspired loss [4]. the objective is to develop a method to minimize surrogate losses based on loo error for the lgc gssl classifier and evaluate the generalization of its solutions. we use the term “surrogate loss” [9] to denote losses such as squared error, cross-entropy and so on. we use this term to purposefully remind that the solution \( \mathbf {h}(\alpha )\) that minimizes the loss on the test set does not also necessarily be the one that maximizes accuracy. we use automatic differentiation as our optimization procedure. as such, we call our approach \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) and \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) when learning label reliability and diffusion rate, respectively. we exploit the fact that we can compute the gradients of our loss. 3.1 \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\): automatic correction of noisy labels let \( \mathbf {p}\) be the propagation matrix, whose submatrix \( \mathbf {p}_{\mathcal {l}\mathcal {l}}\) corresponds to kernel values between labeled data only. then, the modified lgc solution is given by \(\widetilde{ \mathbf {p}} \mathbf {y}_\mathcal {l}\), where $$\begin{aligned} \widetilde{ \mathbf {p}} := ( \mathbf {p}_{\mathcal {l}\mathcal {l}} -diag( \mathbf {p}_{\mathcal {l}\mathcal {l}})) \end{aligned}$$ (28) our optimization problem is to optimize a diagonal matrix \( \mathbf {\omega }\) indicating the updated reliability of each label: $$\begin{aligned} \min _{ \mathbf {\omega }} \texttt {loss}( { deg(\widetilde{ \mathbf {p}} \mathbf {\omega } \mathbf {y}_\mathcal {l})^{-1} \widetilde{ \mathbf {p}} \mathbf {\omega } \mathbf {y}_\mathcal {l},\;\; \mathbf {y}_\mathcal {l}}) \;\;\;\;\;\;\; \text {such that } \;\; \forall i: \mathbf {\omega }_{ii} \ge 0 \end{aligned}$$ (29) where \(\texttt {loss}\) denotes some loss function such as squared error or cross-entropy. 3.2 \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\): automatic choice of diffusion rate in [4], we use a modified version of the power method to calculate the submatrix \( \mathbf {p}_\mathcal {l}\). this is enough to give us the answer in a few seconds for a fixed \(\alpha \), but would quickly turn into a huge bottleneck if we were to constantly update \(\alpha \). our new approach uses the graph fourier transform. in other words, we adapt the idea of smooth eigenbasis pursuit to this particular problem. let us write the propagation matrix using eigenfunctions: $$\begin{aligned} \mathbf {p}&= \sum _{t=0}^\infty \alpha ^t \mathbf {s}^t \end{aligned}$$ (30) $$\begin{aligned}&= \sum _{t=0}^\infty \alpha ^t ( \mathbf {i}\mathbf { \mathbf {l}_{\mathbb {n}}})^t \end{aligned}$$ (31) using the graph fourier transform, we have that $$\begin{aligned} \mathbf { \mathbf {l}_{\mathbb {n}}}&= \mathbf {u} \mathbf {\lambda } \mathbf {u}^\top \end{aligned}$$ (32) $$\begin{aligned} \mathbf {i}&= \mathbf {u} \mathbf {i} \mathbf {u}^\top \end{aligned}$$ (33) so it follows that $$\begin{aligned} \mathbf {p}&= \mathbf {u} \widetilde{\varlambda } \mathbf {u}^\top \end{aligned}$$ (34) where \(\forall i \in \{1..n\}\): $$\begin{aligned} \widetilde{\varlambda }_{ii}&= \sum _{t=0}^\infty (\alpha (1-\varlambda _{ii}) )^t \end{aligned}$$ (35) $$\begin{aligned} \{ \big \vert \alpha (1-\varlambda _{ii}) \big \vert < 1\}&= \frac{1}{1(\alpha (1-\varlambda _{ii}) )} \end{aligned}$$ (36) $$\begin{aligned}&= \frac{1}{(1 \alpha )\times 1 + (\alpha ) \times \varlambda _{ii}} \end{aligned}$$ (37) in practice, we can assume that \( \mathbf {u}\) is an matrix, with l the number of labeled instances and p the chosen amount of eigenfunctions. the diagonal entries are given by $$\begin{aligned} \mathbf {p}_{ii}&= ( \mathbf {u}\widetilde{\varlambda } \mathbf {u}^\top )_{ii} \end{aligned}$$ (38) $$\begin{aligned}&= ( \mathbf {u}^\top )_{[:,i]}\widetilde{\varlambda }( \mathbf {u}^\top )_{[:,i]} \end{aligned}$$ (39) $$\begin{aligned}&= \varsigma _{i=1}^p \mathbf {u}_{ik}^2 \widetilde{\varlambda }_{kk} \end{aligned}$$ (40) $$\begin{aligned}&= ( \mathbf {u}_{[i,:]}) ( \mathbf {u}_{[i,:]} \widetilde{\varlambda })^\top \end{aligned}$$ (41) next, we’ll analyze the complexity of calculating the leave-one-out error for a given diffusion rate \(\alpha \), given that we have stored the first p eigenfunctions in the matrix \( \mathbf {u}\). let be the matrix of eigenfunctions with domain restricted to labeled instances. exploiting the fact that \( \mathbf {y}_\mathcal {u}= 0\), it can be shown that $$\begin{aligned} \mathbf {p}_{\mathcal {l}\mathcal {l}} = \mathbf {u}_{\mathcal {l}} \mathbf {\widetilde{\lambda }}( \mathbf {u}_{\mathcal {l}}^\top \mathbf {y}_\mathcal {l}) \end{aligned}$$ (42) we can precompute with \(\mathcal {o}(plc)\) multiplications. for each diffusion rate candidate \(\alpha \), we obtain by multiplying each restricted eigenfunction with its new eigenvalue. the only thing left is to post-multiply it by the pre-computed matrix. as a result, we can compute the leave-one-out error for arbitrary \(\alpha \) with \(\mathcal {o}(plc)\) operations. we have shown that, by using the propagation submatrix, we can re-compute the propagation submatrix \( \mathbf {p}_{\mathcal {l}\mathcal {l}}\) in \(\mathcal {o}(plc)\) time. in comparison, the previous approach [4] requires, for each diffusion rate, a total of o(tknlc) operations, assuming t iterations of the power method and a sparse affinity matrix with average node degree equal to k. even if we use the full eigendecomposition (\(p = n\)), this new approach is significantly more viable for different learning rates. moreover, we can lower the choice of p to use a faster, less accurate approximation. 4 methodology basic framework. we have a configuration dispatcher which enables us to vary a set of parameters (for example, the chosen dataset and the parameters for affinity matrix generation). we start out by reading our dataset, including features and labels. we use the random seed to select the sampled labels, and create the affinity matrix \( \mathbf {w}\) necessary for lgc. for automatic label correction, we assume that \(\alpha \) is given as a parameter. for choosing the automatic diffusion rate, we need to extract the p smoothest eigenfunctions as a pre-processing step. next, we repeatedly calculate the gradient and update either \(\alpha \) or \( \mathbf {\omega }\) to minimize loo error. after a set amount of iterations, we return the final classification and perform an evaluation on unlabeled examples. the programming language of choice is python 3, for its versatility and support. we also make use of the tensorflow-gpu [1] package, which massively speeds up our calculations and also enables automatic differentiation of loss functions. we use a geforce gtx 1070 gpu to speed up inference, and also for calculating the k-nearest neighbors of each instance with the faiss-gpu package [7]. evaluation and baselines. in this work, our datasets have a roughly equal number of labels for each class. as such, we will report the mean accuracy, as well as its standard deviation. however, one distinction is that we calculate the accuracy independently on labeled and unlabeled data. this is done to better assess whether our algorithms are improving classification on instances outside the labeled set, or if it outperforms its lgc baseline only when performing diagnosis of labels. our approach, \(\texttt {lgc}\_\texttt {lvo}\_\texttt {auto}\), is compatible with any differentiable loss function, such as mean squared error (mse) or cross-entropy (xent). the chosen optimizer was adam, with a learning rate of 0.7 and 5000 iterations. for the approximation of the propagation matrix in \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\), we used \(t=1000\) iterations throughout. perhaps the most interesting classifier to compare our approach to is the lgc algorithm, as that is the starting point and the backbone of our own approach. we will also be comparing our results with the ones reported by [6]. these include: gaussian fields and harmonic functions (gfhf) [18], graph trend filtering [15], large-scale sparse coding (lssc) [10] and eigenfunction [5]. 5 results this section presents the results of employing our two approaches \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) and \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) on isolet and mnist datasets, compared to other graph-based ssl algorithms from the literature. 5.1 experiment 1: \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) on isolet experiment setting. in this experiment, we compared \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) to the baselines reported in [6], specifically for the isolet dataset. unlike the authors, our 20 different seeds also control both the label selection and noise processes. the graph construction was performed exactly as in [6], a symmetric 10-nearest neighbors graph with the width \(\sigma \) of the rbf kernel set to 100. we emphasize that the reported results by the authors correspond to the best-performing parameters, divided for each individual noise level. in [6], \(\lambda _1\) is set to \(10^5\), \(10^2\), \(10^2\) and \(10^2\) for the respective noise rates of 0%, 20%, 40% and 60%; \(\lambda _2\) is kept to 10, and the number of eigenfunctions is \(m=30\). we could not find any implementation code for siis, so we had to manually reproduce it ourselves. as for parameter selection for \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\), we simply set \(\alpha = 0.9\) (equivalently, \(\mu =0.1111\)). we reiterate that having a single parameter is a strength of our approach. in future work, we will try to combine \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) with \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) to fully eliminate the need for parameter selection. experiment results. the results are contained in tables 1a and b. with respect to the accuracy on unlabeled examples, we observed that: siis appeared to have a slight edge in the noiseless scenario. lgc’s own inherent robustness was evident. when 60% noise was injected, it went from 84.72 to 70.69, a decrease of \(16.55\%\). in comparison, siis had a decrease of \(14.39\%\); gfhf a decrease of \(22.08\%\); gtf a decrease of \(21.82\%\). with \(60\% \) noise, \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\)(xent) decreased its accuracy by \(11.52\%\), so \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\)(xent) had the lowest percentual decrease. \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\)(mse) disappointed for both labeled and unlabeled instances. \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\) was not noticeably superior to lgc when there was less than 60% noise. with respect to the accuracy on unlabeled examples, we observed that: lgc was unable to correct noisy labels. \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}(xent)\) discarded only 5% of the labels for the noiseless scenario, which is better than siis and lssc. moreover, \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}(xent)\) had the highest average accuracy on labeled instances for \(20\%,40\%,60\%\) label noise. table 1. accuracy on isolet datasetfull size table 5.2 experiment 2: \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) on mnist experiment setting. this experiment was based on [10], where a few classifiers were tested on the mnist dataset subject to label noise. in that paper, the parameters for the graph were tuned to minimize cross-validation errors. moreover, an anchor graph was used, which is a large-scale solution. we did not use such a graph, as our tensorflow iterative implementation of \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\) was efficient enough to perform classification on mnist in just a few seconds. as we also included the results for lgc (without anchor graph), it is interesting to observe that its accuracy decreases similarly to the previously reported results: the main difference is better performance for the noiseless scenario, which is to be expected (the anchor graph is an approximation). once again, we simply set \(\alpha =0.9\) for \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\). we used a symknn matrix with \(k=15\) neighbors, and a heuristic sigma \(\sigma =423.57\) obtained by taking one-third of the mean distance to the 10th neighbor (as in [3]). table 2. accuracy on mnist datasetfull size table experiment results. the results are found in tables 2a and b. with respect to the accuracy on unlabeled examples, we observed that: \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\) with cross-entropy improved the lgc baseline significantly on unlabeled instances. for 30% label noise, mean accuracy increases from 74.58% to 84.46%. the mean squared error loss is once again consistently inferior to cross-entropy when there is noise. \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) was not able to obtain better results than lssc. with respect to the accuracy on labeled examples, we observed that: lgc was not able to correct the labeled instances. \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autol}}}}}\) with cross-entropy improved the lgc baseline significantly on labeled instances. with 30% noise, accuracy goes from 70.00% to 89.75%. 5.3 experiment 3: \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) on mnist experiment settings were kept the same as experiment 2 (without noise), and \(p = 300\) eigenfunctions were extracted. in fig. 2, we show accuracy on the labeled set as red, and on the unlabeled set as purple. the x values on the horizontal axis relate to \(\alpha \) as following: \(\alpha = 2^{-1/x}\). looking at fig. 2b, we can see that, if we do not remove the diagonal, there is much overfitting and the losses reach their minimum value much earlier than does the accuracy on unlabeled examples. when the diagonal is removed, the loss-minimizing estimates for \(\alpha \) get much closer to the optimal one (fig. 2a). moreover, by removing the diagonal we obtain a much better estimation of the accuracy on unlabeled data. therefore, we can simply select the \(\alpha \) corresponding to the best accuracy on known labels. fig. 2. \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) on mnist. the x coordinate on the horizontal axis determines diffusion rate \(\alpha = 2^{-1/x}\) for \(x\ge 1\). the orange, blue and green curves represent the surrogate losses: mean absolute error, mean squared error, cross-entropy. moreover, the accuracy on the known labeled data is shown in red, and the accuracy on unlabeled data in purple. (color figure online) full size image 6 concluding remarks we have proposed the \(\texttt {lgc}\_\texttt {lvo}\_\texttt {auto}\) framework, based on leave-one-out validation of the lgc algorithm. this encompasses two methods, \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) and \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\), for estimating label reliability and diffusion rates. we use automatic differentiation for parameter estimation, and the eigenfunction approximation is used to derive a faster solution for \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\), in particular when having to recalculate the propagation matrix for different diffusion rates. overall, \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) produced interesting results. for the isolet dataset, it was very successful at the task of label diagnosis, being able to detect and remove labels with overall better performance than every \(\ell _1\)-norm method. on the other hand, did not translate too well for unlabeled instance classification. for the mnist dataset, performance is massively boosted for unlabeled instances as well. in spite of outperforming its lgc baseline by a wide margin, it could not match the reported results of lssc on mnist. preliminary results showed that \({{\mathbf {\mathtt{{lgc}}}}}\_{{\mathbf {\mathtt{{lvo}}}}}\_{{\mathbf {\mathtt{{autod}}}}}\) is a viable way to get a good estimate of the optimal diffusion rate, and removing the diagonal entries proved to be the crucial step for avoiding overfitting. for future work, we will be further evaluating \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\). we will also try to integrate \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) and \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autol}\) together into one single algorithm. lastly, we will aim to extend \(\texttt {lgc}\_\texttt {lvo}\_\texttt {autod}\) to a broader class of graph-based kernels, in addition to the one resulting from the lgc baseline. references abadi, m., et al.: tensorflow: large-scale machine learning on heterogeneous systems (2015). http://tensorflow.org/, software available from tensorflow.org de aquino afonso, b.k.: analysis of label noise in graph-based semi-supervised learning. master’s thesis (2020) google scholar  chapelle, o., schölkopf, b., zien, a. (eds.): semi-supervised learning. mit press, cambridge (2006). http://www.kyb.tuebingen.mpg.de/ssl-book de aquino afonso, b.k., berton, l.: identifying noisy labels with a transductive semi-supervised leave-one-out filter. pattern recognit. lett. 140, 127–134 (2020). https://doi.org/10.1016/j.patrec.2020.09.024. http://www.sciencedirect.com/science/article/pii/s0167865520303603 article  google scholar  fergus, r., weiss, y., torralba, a.: semi-supervised learning in gigantic image collections. in: advances in neural information processing systems, pp. 522–530 (2009) google scholar  gong, c., zhang, h., yang, j., tao, d.: learning with inadequate and incorrect supervision. in: 2017 ieee international conference on data mining (icdm), pp. 889–894. ieee (2017) google scholar  johnson, j., douze, m., jégou, h.: billion-scale similarity search with gpus. ieee trans. big data (2019) google scholar  kearnes, s., mccloskey, k., berndl, m., pande, v., riley, p.: molecular graph convolutions: moving beyond fingerprints. j. comput.-aided mol. des. 30(8), 595–608 (2016). https://doi.org/10.1007/s10822-016-9938-8 article  google scholar  krijthe, j.h.: robust semi-supervised learning: projections, limits and constraints. ph.d. thesis, leiden university (2018) google scholar  lu, z., gao, x., wang, l., wen, j.r., huang, s.: noise-robust semi-supervised learning by large-scale sparse coding. in: aaai, pp. 2828–2834 (2015) google scholar  mitchell, t.: machine learning. mcgraw-hill, new york (1997) math  google scholar  miyato, t., maeda, s.i., ishii, s., koyama, m.: virtual adversarial training: a regularization method for supervised and semi-supervised learning. ieee trans. pattern anal. mach. intell. 41, 1979–1993 (2018) article  google scholar  shao, y., sang, n., gao, c., ma, l.: probabilistic class structure regularized sparse representation graph for semi-supervised hyperspectral image classification. pattern recognit. 63, 102–114 (2017) article  google scholar  van engelen, j.e., hoos, h.h.: a survey on semi-supervised learning. mach. learn. 109(2), 373–440 (2020). https://doi.org/10.1007/s10994-019-05855-6 article  mathscinet  math  google scholar  wang, y.x., sharpnack, j., smola, a.j., tibshirani, r.j.: trend filtering on graphs. j. mach. learn. res. 17(1), 3651–3691 (2016) mathscinet  math  google scholar  ying, r., he, r., chen, k., eksombatchai, p., hamilton, w.l., leskovec, j.: graph convolutional neural networks for web-scale recommender systems. in: proceedings of the 24th acm sigkdd international conference on knowledge discovery and data mining, pp. 974–983 (2018) google scholar  zhou, d., bousquet, o., lal, t.n., weston, j., schölkopf, b.: learning with local and global consistency. in: advances in neural information processing systems, pp. 321–328 (2004) google scholar  zhu, x., ghahramani, z., lafferty, j.: semi-supervised learning using gaussian fields and harmonic functions. in: proceedings of the twentieth international conference on international conference on machine learning, pp. 912–919. aaai press (2003) google scholar  catunda, j.p.k., da silva, a.t., berton, l.: car plate character recognition via semi-supervised learning. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 735–740. ieee (2019) google scholar  download references author information authors and affiliations institute of science and technology, university of são paulo, são paulo, brazil bruno klaus de aquino afonso & lilian berton authors bruno klaus de aquino afonsoview author publications search author on:pubmed google scholar lilian bertonview author publications search author on:pubmed google scholar corresponding authors correspondence to bruno klaus de aquino afonso or lilian berton . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de aquino afonso, b.k., berton, l. (2021). optimizing diffusion rate and label reliability in a graph-based semi-supervised classifier. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_34 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_34 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning leave-one-out semi-supervised learning graph-based approaches label propagation eigendecomposition publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating clustering meta-features for classifier recommendation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating clustering meta-features for classifier recommendation conference paper first online: 28 november 2021 pp 453–467 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) evaluating clustering meta-features for classifier recommendation download book pdf download book epub luís p. f. garcia  orcid: orcid.org/0000-0003-0679-914310, felipe campelo  orcid: orcid.org/0000-0001-8432-432511, guilherme n. ramos  orcid: orcid.org/0000-0001-5859-736210, adriano rivolli  orcid: orcid.org/0000-0001-6445-300712 & … andré c. p. de l. f. de carvalho  orcid: orcid.org/0000-0002-4765-645913  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 802 accesses 1 citation 1 altmetric abstract data availability in a wide variety of domains has boosted the use of machine learning techniques for knowledge discovery and classification. the performance of a technique in a given classification task is significantly impacted by specific characteristics of the dataset, which makes the problem of choosing the most adequate approach a challenging one. meta-learning approaches, which learn from meta-features calculated from the dataset, have been successfully used to suggest the most suitable classification algorithms for specific datasets. this work proposes the adaptation of clustering measures based on internal indices for supervised problems as additional meta-features in the process of learning a recommendation system for classification tasks. the gains in performance due to meta-learning and the additional meta-features are investigated with experiments based on 400 datasets, representing diverse application contexts and domains. results suggest that (i) meta-learning is a viable solution for recommending a classifier, (ii) the use of clustering features can contribute to the performance of the recommendation system, and (iii) the computational cost of meta-learning is substantially smaller than that of running all candidate classifiers in order to select the best. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others evaluating data characterization measures for clustering problems in meta-learning chapter © 2021 evolving learners’ behavior in data mining article 21 september 2016 dataset characteristics (metafeatures) chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining and knowledge discovery data mining learning algorithms machine learning statistical learning 1 introduction data analysis is a pervasive activity in several areas of industrial and scientific activities, often involving machine learning (ml) approaches, which have been successfully applied in a wide variety of domains [35]. however, the variability of dataset characteristics makes it challenging to accurately select the most adequate ml approach to handle new data. this issue has also been tackled from a ml perspective, where the task is learning, based on meta-data extracted from several datasets related to a given application or domain, a model to predict which ml approach may be better suited for exploring that particular domain. this approach, commonly referred to as meta-learning (mtl), has shown promising results in predicting classifier performances for a problem, based on statistical or geometrical characteristics of a dataset [16]. mtl approaches consist of the data-driven selection of techniques, through knowledge extracted from previous tasks, which may be then applied by a recommendation system to predict the preferred approach for a new, previously unseen problem [4]. this depends on the construction of a meta-dataset from the information on a group of datasets in the class of the problem. for each dataset in the group, its descriptive characteristics (meta-features) are extracted and combined into one or more meta-examples, which are then labeled according to the observed performances of different ml algorithms or an order-preserving transformation such as their rank [4], composing the target feature to be predicted. from the meta-dataset, a meta-model can be induced by a learning algorithm and then used in a recommendation system to predict which algorithm is expected to have the best performance when a new problem needs to be addressed [34]. despite their success, most mtl studies still lack an in-depth analysis of the meta-features [29] which are known to be crucial in the successful use of mtl [3]. several works propose new meta-features [15,16,17], but only a few present important details such as their asymptotic computational cost, the degree of information presented, or the importance of the meta-features for the investigated problems [16, 24]. this paper presents the use of clustering measures as meta-features in an mtl framework, to learn a recommendation system for classifiers. these clustering measures are based on internal indices which extract information, such as compactness or separation, to evaluate the goodness of a clustering structure. although some of these measures have been used in unsupervised mtl scenarios [26, 37], this work proposes the use of those measures on classification problems. the proposed approach uses class labels as cluster indicators rather than the numerical results of a clustering algorithm, and is expected to extract informative measures for quantifying statistical or geometrical characteristics of datasets. the main goal of this paper is to investigate whether clustering measures can be applied in this context, and whether they influence the choices of the recommendation system. the experimental results suggest that including these measures as meta-features contributes to more accurate recommendations in the problem of classifier selection. additionally, an initial evaluation of computational costs indicates that this mtl approach is substantially less computationally expensive than testing all classifiers on the dataset for the selection of the best one. 2 background this section presents the background information necessary to describe the proposed approach: sect. 2.1 introduces the meta-learning framework, including the process of building a meta-dataset and how to recommend algorithms. section 2.2 elaborates on clustering meta-features, a subset of internal indices based on compactness and separation of the goodness of a clustering structure. 2.1 meta-learning different algorithms have distinct learning strategies, thus the essence of learning can only be captured by considering different learning algorithms with diverse biases to acquire domain specific information [4]. this concept was initially introduced to make the algorithm selection problem systematic [30], and the goal is to predict the best algorithm to solve a specific given problem, when more than one option is available. the components of this model are the space of problem instances (\(\mathcal {p}\)); the space of instance meta-features (\(\mathcal {f}\)); the space of algorithms (\(\mathcal {a}\)); and the space of evaluation measures (\(\mathcal {y}\)). from these components, an mtl system can obtain a model capable of mapping a dataset or problem \(p\in \mathcal {p}\), described by meta-features \(f\in \mathcal {f}\), into one algorithm \(\alpha \in \mathcal {a}\) able to solve the problem with a good predictive performance according to measure \(y\in \mathcal {y}\). the meta-learner recommendation would be the algorithm with the best expected \(y(\alpha (p))\). this can be further improved, for instance, by the inclusion of components which may guide theoretical support to refine the recommendation system [34]. a crucial component of these previous approaches is the definition of the meta-features (\(\mathcal {f}\)) used to describe general properties of datasets [3]. they must be able to provide evidence on the future performance of the algorithms in \(\mathcal {a}\) and to discriminate, with an acceptable computational cost, the performance of a group of algorithms. the main meta-features used in the mtl literature can be divided into six main groups [31], called standard meta-features in this work. they represent meta-features based general high-level summaries of the data, statistical and information theory properties of the data, properties of decision trees (dts) induced from the data and the performance of simple and fast learning algorithms [9, 25, 29]. another concern is the definition of the set of problem instances (\(\mathcal {p}\)), since the use of a large number of diverse datasets is recommended to induce a reliable meta-model [4]. attempts to reduce the bias in this choice include using datasets from different data repositories, such as uci [13] and openml [36]. the importance of problem diversity is based on the underlying assumption that the meta-model is expected to generalize the acquired knowledge when faced with new problem instances without explicit constraints in terms of expected problem characteristics. the selected algorithms \(\mathcal {a}\) represent a set of candidate algorithms to be recommended in the algorithm selection process. ideally, these algorithms should also be sufficiently different from each other and represent all regions in the algorithm space [34]. the models induced by the algorithms can be evaluated by different measures. quality measures \(\mathcal {y}\) for assessing the models depend on the nature of the problem. classifiers, for example, can be evaluated by different measures such as accuracy, \(f_\beta \), area under the roc curve (auc) or the kappa coefficient, among others. the step following the extraction of meta-features from the datasets and training the set of algorithms is the labeling of meta-examples in the meta-base. the three properties most frequently used in this task are [4]: (i) the best performing algorithm on the meta-example’s dataset (a meta-classification problem); (ii) the ranking of the algorithm according to its performance (a meta-ranking problem); and (iii) the raw performance value of each algorithm on the dataset (a meta-regression problem). differently from previous works on mtl applied on clustering data [26, 37], this work presents a mtl regression task based on the use of clustering meta-features using internal indices, to extract information like separation and compactness on a supervised scenario using the class labels. in this case, the main objective is to improve the algorithm recommendation using internal indices, motivated by their generally low computational cost and high degree of information. the standard meta-features [31] are considered to provide a comparison baseline and to allow an objective analysis of the variation in performance resulting from using the new clustering meta-features. the evaluation of the recommender system includes assessing the performance in the baseand meta-levels and the cost in execution time. 2.2 clustering meta-features a few definitions must be presented before the description of the clustering measures. let \(x\in \mathbb {r}^{n\times {q}}\) be a matrix of n observations, each represented by a q-dimensional vector \(\mathbf {x}_\ell \in \mathbb {r}^q\); and \(u_k(x) = \{x_1,x_2,\dotsc ,x_k\}\) be an exhaustive partition of x into k mutually exclusive clusters \(x_k\) with sizes \(n_k > 0,~k=1,\dotsc ,k\). in all definitions below, the clustering meta-features are functions of a partition \(u_k(x)\), which in the case of this paper is given by the class labels. the \(u_k(x)\) is kept implicit in the definitions in order to keep the notation cleaner, but the reader should keep this relationship in mind. let \(\mathbf {\bar{x}}_k\) denote the mean point of all observations belonging to cluster \(x_k\), and \(\mathbf {\bar{\bar{x}}}\) denote the grand mean of all observations in x. also, let \(dist(\cdot ,\cdot )\) denote the distance (e.g., euclidean) between two points. then, $$\begin{aligned} \delta _{ij} \triangleq \min _{\begin{array}{c} \mathbf {x}\in x_i\\ \mathbf {y}\in x_j \end{array}} dist(\mathbf {x},\mathbf {y}) \end{aligned}$$ (1) denotes the single-linkage distance between the two clusters \(x_i,~x_j\), and $$\begin{aligned} \vardelta _k \triangleq \max _{\mathbf {x},\mathbf {y}\in x_k} dist(\mathbf {x},\mathbf {y}) \end{aligned}$$ (2) represents the diameter of a cluster \(x_k\). additionally, consider some vectors of distances. let \(\mathbf {d}^+\in \mathbb {r}^{n_w}\) be a vector of all \(n_w\) within-cluster distances (i.e., all distances between pairs of points having equal class labels in the data), \(\mathbf {d}^-\in \mathbb {r}^{n_b}\) be a vector containing all \(n_b\) between-cluster distances (between pairs of points having different class labels), and \(\mathbf {d}^\bullet \in \mathbb {r}^{n_t}\) denote a vector containing the distances between all \(n_t\) pairs of points in x, with \(n_t = n_w + n_b\). finally, let the following quantities be defined: the within-groups sum of squares, $$\begin{aligned} wgss = \sum _{k=1}^k\sum _{\mathbf {x}_i\in x_k} \left[ dist(\mathbf {x}_i, \mathbf {\bar{x}}_k)\right] ^2, \end{aligned}$$ (3) and the between-groups sum of squares, $$\begin{aligned} bgss = \sum _{k=1}^k n_k\left[ dist(\mathbf {\bar{x}}_k, \mathbf {\bar{\bar{x}}})\right] ^2. \end{aligned}$$ (4) given the preceding definitions, the clustering meta-features used in this work are formalized as follows: dunn’s separation index (vdu) [14]: $$\begin{aligned} vdu = \min _{\begin{array}{c} i,j\in [1,k]\\ i\ne j \end{array}}\frac{\delta _{ij}}{\max \limits _{k\in [1,k]}\vardelta _k}. \end{aligned}$$ (5) davies-bouldin index (vdb) [11]: $$\begin{aligned} vdb = \frac{1}{k}\sum _{i=1}^k\max _{j\ne i}\frac{\vardelta _i + \vardelta _j}{dist(\mathbf {\bar{x}}_i,\mathbf {\bar{x}}_j)}. \end{aligned}$$ (6) baker-hubert index (\(\vargamma \)) [1]: let $$\begin{aligned} s^+ = \sum _{\forall d_i\in \mathbf {d}^+}\sum _{\forall d_j\in \mathbf {d}^-}one(d_i<d_j) \quad \mathrm {and}\quad s^= \sum _{\forall d_i\in \mathbf {d}^+}\sum _{\forall d_j\in \mathbf {d}^-}one(d_i>d_j) \end{aligned}$$ (7) where one(condition) is a function that returns 1 if the condition is true and 0 otherwise; then: $$\begin{aligned} \vargamma = \frac{s^+ s^-}{s^+ + s^-}. \end{aligned}$$ (8) tie-corrected kendall tau (\(\tau \)) [12]: $$\begin{aligned} \tau = \frac{s^+ s^-}{\sqrt{n_wn_bn_t(n_t-1)/2}}. \end{aligned}$$ (9) ray-turi index (\(\nu \)) [28]: $$\begin{aligned} \nu = \frac{1}{n}\frac{wgss}{\min \limits _{i\ne j} \delta _{ij}^2}. \end{aligned}$$ (10) mean inter-centroid distance (int) [2]: $$\begin{aligned} int = \frac{2}{k(k-1)}\sum _{k=1}^{k-1} dist(\mathbf {\bar{x}}_k, \mathbf {\bar{x}}_{k+1}). \end{aligned}$$ (11) global silhouette index (sil) [32]: for a given point \(\mathbf {x}\in {x_k}\), let \(a(\mathbf {x})\) denote the mean distance between this point and all other points belonging to the same cluster, \(x_k\); \(\mathfrak {d}(\mathbf {x},k^\prime )\) denote the mean distance between this point and all points belonging to a distinct cluster \(x_{k^\prime }\) (\(k^\prime \ne k\)); and $$\begin{aligned} b(\mathbf {x}) = \min _{k^\prime \ne k} \mathfrak {d}(\mathbf {x},k^\prime ). \end{aligned}$$ (12) the silhouette width of point \(\mathbf {x}\) is then calculated as $$\begin{aligned} \mathcal {s}(\mathbf {x}) = \frac{b(\mathbf {x}) a(\mathbf {x})}{\max (b(\mathbf {x}), a(\mathbf {x}))}, \end{aligned}$$ (13) and the global silhouette index can be calculated as $$\begin{aligned} sil = \frac{1}{k}\sum _{k=1}^k\sum _{\forall \mathbf {x}\in x_k}\frac{\mathcal {s}(\mathbf {x})}{n_k}. \end{aligned}$$ (14) point biserial index (pb) [12]: $$\begin{aligned} pb = \left( \frac{|\mathbf {d}^+|_1}{n_w} \frac{|\mathbf {d}^-|_1}{n_b}\right) \sqrt{\frac{n_wn_b}{n_t^2}}. \end{aligned}$$ (15) with \(|\cdot |_1\) denoting the \(\ell _1\) norm of a vector. calinski-harabasz index (ch) [8]: $$\begin{aligned} ch = \frac{\left( n-k\right) bgss}{\left( k-1\right) wgss}. \end{aligned}$$ (16) xie-beni index (xb) [38]: $$\begin{aligned} xb = \frac{1}{n}\frac{wgss}{\min \limits _{i\ne j}\delta _{ij}}. \end{aligned}$$ (17) normalized relative entropy (nre) [26]: $$\begin{aligned} nre = \sum _{k=1}^k\frac{n_k}{n}\log _2\left( \frac{n_k}{n}\right) . \end{aligned}$$ (18) c index (c) [21]: let \(s_{min}\) denote the sum of the \(n_w\) smallest elements of \(\mathbf {d}^\bullet \) and \(s_{max}\) the sum of its \(n_w\) largest elements. then: $$\begin{aligned} c = \frac{|\mathbf {d}^+|_1 s_{min}}{s_{max} s_{min}}. \end{aligned}$$ (19) mean of distances to cluster centroids (cm): $$\begin{aligned} cm = \frac{1}{k}\sum _{k=1}^k\sum _{\forall \mathbf {x}_i\in x_k}dist(\mathbf {x}_i,\mathbf {\bar{x}}_k) \end{aligned}$$ (20) connectivity (cn) [7, 19]: $$\begin{aligned} cn = \sum _{i=1}^n\sum _{j=1}^l\mathcal {i}\left( \mathbf {x}_i,\eta _j\left( \mathbf {x}_i\right) \right) , \end{aligned}$$ (21) where \(\eta _j\left( \mathbf {x}_i\right) \) denotes the j-th nearest-neighbor to point \(\mathbf {x}_i\), and \(\mathcal {i}\left( \mathbf {x}_i,\eta _j\left( \mathbf {x}_i\right) \right) \) is an indicator function that receives the value of 1/j if \(\mathbf {x}_i\) and \(\eta _j\left( \mathbf {x}_i\right) \) belong to the same cluster, zero otherwise. average scattering for clusters (\(sd_{scat}\)) [18]: $$\begin{aligned} sd_{scat} = \frac{\sum \limits _{k=1}^k \left| \hat{\boldsymbol{\sigma }}^2_{k}\right| _2}{k\left| \hat{\boldsymbol{\sigma }}^2_{\bullet }\right| _2}, \end{aligned}$$ (22) where \(\hat{\boldsymbol{\sigma }}^2_{\bullet }\) denotes the vector of variance estimates for all attributes in all clusters, and \(\hat{\boldsymbol{\sigma }}^2_{k}\) is the vector of variance estimates for all attributes considering only the observations in the k-th cluster. total separation between clusters (\(sd_{dis}\)) [18]: $$\begin{aligned} sd_{dis} = \frac{\kappa _{max}}{\kappa _{min}}\sum \limits _{k=1}^{k}\left( \sum _{\begin{array}{c} k^\prime = 1\\ k^\prime \ne k \end{array}}^k dist\left( \bar{\mathbf {x}}_k,\bar{\mathbf {x}}_{k^\prime }\right) \right) ^{-1} \end{aligned}$$ (23) where: $$\begin{aligned} \kappa _{max} = \max \limits _{k^\prime \ne k} dist\left( \bar{\mathbf {x}}_k,\bar{\mathbf {x}}_{k^\prime }\right) \quad \mathrm {and}\quad \kappa _{min} = \min \limits _{k^\prime \ne k} dist\left( \bar{\mathbf {x}}_k,\bar{\mathbf {x}}_{k^\prime }\right) . \end{aligned}$$ (24) akaike’s information criterion (aic) [33]: the aic of a first-order multiple linear regression model of class labels on each attribute of the dataset. the linear model is given by $$\begin{aligned} class(\mathbf {x}_i) = \hat{\beta }_0 + \hat{\boldsymbol{\beta }}\mathbf {x}_i + e_i, \end{aligned}$$ (25) with \(class(\mathbf {x}_i)\) denoting the numerically-encoded class of point \(\mathbf {x}_i\in {x}\), \(\hat{\beta }_0\) and \(~\hat{\boldsymbol{\beta }}\) representing the fitted coefficients of the model, and \(e_i\) the residual related to the i-th observation in the dataset. the aic of the model is then given as: $$\begin{aligned} aic = -2\ln (l) + 2q \end{aligned}$$ (26) where \(\ln {l}\) is the log-likelihood value estimated for the model (25). bayesian information criterion (bic) [33]: the bic of the regression model described in (25), which is calculated as: $$\begin{aligned} bic = -2\ln (l) + q\ln (n) \end{aligned}$$ (27) 3 methodology this works aims to investigate the use of clustering measures as meta-features for learning a recommendation system for classification tasks. more specifically, the standard and clustering meta-features are used in the mtl setup designed to predict the accuracy of some popular classification techniques for a given dataset. the objectives are: (i) to determine whether the mtl approach results in an improved recommendation system; (ii) to investigate whether the clustering meta-features contribute to the performance of this recommendation system; and (iii) to characterize the execution times required to extract each set of meta-features. to train and assess the meta-learner, a meta-dataset was populated with the meta-feature values (both clustering and standard meta-features) for a collection of problem instances, labeled with the performances of known classification techniques. the meta-models were induced by regression techniques and evaluated in the base-level, measuring the predictive performance of the classifiers, and the and meta-level, measuring the impact of recommending the best one. the computational costs for theses processes was recorded for evaluation. four hundred datasets from the openml repository [36], representing diverse application contexts and domains, were used in this experiment. they were selected considering a maximum number of 10, 000 observations, 500 features and 10 classes, to constrain the computational costs of the process. for each dataset, both standard and clustering meta-features, as described in sect. 2.2, were computed. the averages of 10-fold cross-validated predictive accuracies achieved by each of five classification techniques, for each dataset, were also calculated for labeling the meta-examples. the classification approaches used were: c4.5 decision tree [27] with pruning based on subtree raising; k-nearest neighbors (knn) model [22] with \(k=3\); multilayer perceptron (mlp) [20] with learning rate of 0.3, momentum of 0.5 and a single hidden layer; random forest (rf) [5] with 500 trees; and support vector machine (svm) [10] with radial basis kernel. these hyper-parameter values were defined following the standard configurations of the implementations used. this process resulted in a meta-dataset containing 130 meta-features (112 standard and 18 clustering) and 400 samples for each classifier, labeled by the mean accuracy of the classification method. this meta-dataset was then used to train regression models to estimate the expected accuracy of each classifier, as a function of the meta-features. five regression techniques known to have different biases were tested: classification and regression trees (cart) [6] algorithm with pruning; distance-weighted k-nearest neighbor (dwnn) [22] with \(k=3\) and gaussian kernel; multilayer perceptron regression (mlpr) [20] with learning rate of 0.3, momentum of 0.5 and a single hidden layer; random forest regressor (rfr) [5] with 500 trees; and support vector regression (svr) [10] using radial basis kernel. as with the classifiers, the regressor hyper-parameters were also set as the default values of the implementations used without any problem-specific tuning. to train and evaluate the regression models, 10-fold cross-validation rounds were also executed. the models obtained were evaluated for quality, considering a comparison to two simple baselines: random (rd) and default (df). the rd baseline represents the observed performance of a randomly chosen classifier in the meta-base. the df baseline was set as the accuracy of the classifier that most often presented the best classification performance across all datasets. the final step is the analysis of the trade-off between the computational cost of each set of meta-features and that of evaluating all classifiers, in a cross-validation setup, through direct comparison of single threaded experiments. the experiments were done in a cluster node with two intel xeon e5-2680v2 processors and 128 gb ddr3. the standard meta-features, as well as some of the clustering ones, are provided by the mfe packagefootnote 1. 4 experimental results since the main goal of algorithm recommender systems using mtl is to suggest, within the known options, the algorithm with the most appropriate bias to a particular dataset [4], the first analysis of this experiment focuses on such objective. figure 1 presents the results of two analyses on the meta-base involving the classifiers on the selected 400 datasets. fig. 1. performance of classifiers over the 400 datasets. full size image the distribution of accuracy values, summarized by the boxplots in fig. 1a, suggests that all approaches have generally similar distributions of performance values across the datasets employed, with reasonably high median accuracies for most problems. random forest has a slightly higher median value than the others, while knn has the lowest median and the largest variability in performance values. figure 1b shows how many times each algorithm was the “winner”, i.e. the number of datasets for which each presented the best performance. these results imply that the set of choices is considered adequate for mtl since each one was the best performing in a non-empty subset of the dataset. it is clear from the figures that rf was most frequently the best classifier in the dataset, which means it will be the most probable choice for the random baseline recommender system and the fixed choice for the default baseline one. to investigate their quality on the task of learning to recommend a classifier, the selected regressors (mlpr, cart, dwnn, svr, and rfr) and the two baselines regressors (rd and df) were applied to the meta-dataset considering two sets of meta-features: the 112 standard meta-features (described in sect. 2.1) and the full set of 130 meta-features (incorporating the 18 clustering measures from sect. 2.2). figure 2 shows the average normalized mean squared error (nmse) of each meta-regressor on predicting the expected accuracy of the classifiers for each set of meta-features, estimated by 10-fold cross-validation. fig. 2. the log-scaled nmse for each combination of meta-feature set, classification and regression models. full size image the boxplots indicate that all meta-regressors provided better predictions than the baseline approaches (with the exception of a single observation for the mlpr regressor). rfr seems to have a slight advantage when compared to the other regressors, for both sets of features. mlpr, cart, dwnn and svr show a similar performance distribution in most cases, with mlpr having a higher interquartile range and a few outliers. although visual inspection does not necessarily provide a clear winner between the two sets of meta-features, it is sufficient to indicate that: (i) they are clearly less error-prone than the baselines, and (ii) their nmse values show a relatively low variance. more objectively, an anova model followed by dunnett pairwise comparisons indicates a statistically significant difference at the \(95\%\) confidence level between the mtl algorithms (in aggregate) against the two baselines (\(p < 10^{-10}\) for both the mtl\(\times \)df and mtl\(\times \)rd comparisons). the effects of different meta-feature sets, classifier methods and distinct replicates were removed by blocking [23]. a second analysis was conducted to investigate the effect of adding the clustering meta-features on the predictive ability of the meta-regressors. this analysis was performed by removing the two baseline methods and fitting a blocked anova model [23] on rank-transformed data (required in this case to meet the anova assumptions), with the meta-feature sets as the experimental factor and the classifier methods, meta-regressors, and replicates as blocking factors. this test reveal a statistically significant positive effect of adding the clustering ones on the performance of the best regressor, rfr, across all classifiers (paired t-test for standard\(\times \)full meta-feature sets, \(p = 0.036\)). figure 3 shows the results of a direct comparison between the recommended classifier. the x-axis lists the meta-regressors and the y-axis shows the percent differences in accuracy, averaged over all datasets. as seen in the analysis of fig. 2 there is a noticeable advantage of using the mtl approaches over the random baseline (rd), as well as against the fixed choice of rf, which is best overall classifier (df), in the case of the three most successful regressors. the magnitude of the gains is also substantial, with 40–\(50\%\) gains over rd and 10–\(20\%\) gains over df in the case of dwnn, rfr and svr. fig. 3. differences between base-classifier accuracies over baselines. vertical lines represent standard errors. full size image the impact of the meta-features can be further investigated via the analysis of the rfr model. figure 4 shows the most relevant meta-features, according to their individual contribution to the reduction of the mean squared error, mse (measured as the increase in mse when that meta-feature was omitted), as well as their corresponding groups. the x-axis lists the meta-features in decreasing order of relevance, with the mean change in mse shown in the y-axis. vertical bars indicate standard errors of estimation. fig. 4. top-ranked meta-features selected by the rfrs in log-scale, based on the average increase in mse when the meta-feature is omitted. full size image the standard meta-features are more strongly represented, as expected due to their informative value [31]. some meta-features appear more than once due to distinct summarization functions being used. landmarking measures, which mainly relate to the performance of simple meta-models induced by the knn, naïve bayes and simple node dt algorithms, represent over half of the top features. interestingly, six of the 18 clustering features are ranked in the 20 most relevant for the rfr model, in this order: sil, ch, ray, vdb, c_index and nre. this suggests that these features are relevant to the mtl task, at least for the case of the rfr meta-regressor, which would corroborate the result of the hypothesis test performed on the data of fig. 2. finally, the practical applications of using clustering meta-features for a mtl recommender system must consider the computational costs involved. the trade-off between the runtime of the characterization process and the evaluation of all alternative data modeling algorithms considered to solve the task under study should favor the former in order to make the recommendation system useful. figure 5 presents the results of a runtime analysis to evaluate this trade-off for the proposed approach. this analysis exhaustively compared the single-thread runtime cost for extraction of the standard and clustering meta-features to the cost for running all classification algorithms. in the figure, each point represents the cost for processing a dataset in a log\(\times \)log scale. the x-axis shows the cost for running all classifiers while the y-axis shows the cost for extracting the meta-features. this enables a straightforward visual analysis: if a point is above the line, the y value is less than the x value, thus running the classifiers is more expensive than extracting the features. fig. 5. execution times for computing meta-features compared to applying all classifiers. full size image the standard meta-features are cheaper than running all classification algorithms in around 89% of the cases. the clustering features are considerably cheaper than that running all classification algorithms. this low cost in computation, associated with the informative nature of some of the clustering meta-features (seen in fig. 4), suggests their usefulness in, and their potential for, improving mtl recommender systems. 5 conclusions and future works this work investigated both the gains in performance associated with mtl in general, and the use of clustering meta-features in a recommender system for classification algorithms. experimental results showed that the selected classifiers were adequate for a recommender system and that using the recommended model resulted in improvements over a random or fixed (best expected) algorithm choice. the results presented in this paper suggest two tentative conclusions: (i) that meta-learning, as an approach to recommend classifiers for unseen problems, has the potential to provide good choices with a reduced computational budget; and (ii) that clustering-based meta-features are suitable to enhance this mtl task, which may indicate that they are able to capture relevant properties from datasets to describe the performance of classifiers. future works include investigating the impact of the underlying grouping structure of the datasets in the adequacy of the clustering meta-features, as well as investigating the correlations of the meta-features to enable the selection of a more efficient, parsimonious subset of meta-feature. the incorporation of a more diverse set of classification algorithms for the recommender system and the incorporation of hyper-parameter tuning for the classifiers and regressors also represent interesting new areas for research. notes 1.https://github.com/rivolli/mfe. references baker, f.b., hubert, l.j.: measuring the power of hierarchical cluster analysis. j. am. stat. assoc. 70(349), 31–38 (1975) article  google scholar  bezdek, j.c., pal, n.r.: some new indexes of cluster validity. ieee trans. syst. man cybern. part b (cybern.) 28(3), 301–315 (1998) google scholar  bilalli, b., abelló, a., aluja-banet, t.: on the predictive power of meta-features in openml. int. j. appl. math. comput. sci. 27(4), 697–712 (2017) article  mathscinet  google scholar  brazdil, p., giraud-carrier, c., soares, c., vilalta, r.: metalearning applications to data mining. cognitive technologies, 1st edn. springer, heidelberg (2009). https://doi.org/10.1007/978-3-540-73263-1 book  math  google scholar  breiman, l.: random forests. mach. learn. 45(1), 5–32 (2001) article  google scholar  breiman, l., friedman, j.h., olshen, r.a., stone, c.j.: classification and regression trees. wadsworth and brooks (1984) google scholar  brock, g., pihur, v., datta, s., datta, s.: clvalid: an r package for cluster validation. j. stat. softw. 25(4), 1–22 (2008). http://www.jstatsoft.org/v25/i04/ caliński, t., harabasz, j.: a dendrite method for cluster analysis. commun. stat. theory methods 3(1), 1–27 (1974) article  mathscinet  google scholar  castiello, c., castellano, g., fanelli, a.m.: meta-data: characterization of input features for meta-learning. in: torra, v., narukawa, y., miyamoto, s. (eds.) mdai 2005. lncs (lnai), vol. 3558, pp. 457–468. springer, heidelberg (2005). https://doi.org/10.1007/11526018_45 chapter  math  google scholar  cristianini, n., shawe-taylor, j.: an introduction to support vector machines and other kernel-based learning methods. cambridge university press, cambridge (2000) google scholar  davies, d.l., bouldin, d.w.: a cluster separation measure. ieee trans. pattern anal. mach. intell. pami-1(2), 224–227 (1979) google scholar  desgraupes, b.: clustercrit vignette (2018). https://cran.r-project.org/package=clustercrit/vignettes/clustercrit.pdf dua, d., graff, c.: uci machine learning repository (2017). http://archive.ics.uci.edu/ml dunn, j.c.: well-separated clusters and optimal fuzzy partitions. j. cybern. 4(1), 95–104 (1974) article  mathscinet  google scholar  filchenkov, a., pendryak, a.: datasets meta-feature description for recommending feature selection algorithm. in: artificial intelligence and natural language and information extraction, social media and web search fruct conference (ainl-ismw fruct), vol. 7, pp. 11–18 (2015) google scholar  garcia, l.p.f., lorena, a.c., de souto, m.c.p., ho, t.k.: classifier recommendation using data complexity measures. in: 24th international conference on pattern recognition (icpr), pp. 874–879 (2018) google scholar  garcia, l.p.f., rivolli, a., alcobaça, e., lorena, a.c., de carvalho, a.c.p.l.f.: boosting meta-learning with simulated data complexity measures. intell. data anal. 24(5), 1011–1028 (2020) google scholar  halkidi, m., batistakis, y., vazirgiannis, m.: on clustering validation techniques. j. intell. inf. syst. 17(2–3), 107–145 (2001) article  google scholar  handl, j., knowles, j.: exploiting the trade-off — the benefits of multiple objectives in data clustering. in: coello coello, c.a., hernández aguirre, a., zitzler, e. (eds.) emo 2005. lncs, vol. 3410, pp. 547–560. springer, heidelberg (2005). https://doi.org/10.1007/978-3-540-31880-4_38 chapter  google scholar  haykin, s.s.: neural networks: a comprehensive foundation. prentice hall, hoboken (1999) google scholar  hubert, l., schultz, j.: quadratic assignment as a general data analysis strategy. br. j. math. stat. psychol. 29(2), 190–241 (1976) article  mathscinet  google scholar  mitchell, t.m.: machine learning. mcgraw hill series in computer science. mcgraw hill, new york (1997) google scholar  montgomery, d.c.: design and analysis of experiments, 5th edn. wiley, hoboken (2000) google scholar  muñoz, m.a., villanova, l., baatar, d., smith-miles, k.: instance spaces for machine learning classification. mach. learn. 107(1), 109–147 (2018). https://doi.org/10.1007/s10994-017-5629-5 article  mathscinet  math  google scholar  pfahringer, b., bensusan, h., giraud-carrier, c.g.: meta-learning by landmarking various learning algorithms. in: 17th international conference on machine learning (icml), pp. 743–750 (2000) google scholar  pimentel, b.a., de carvalho, a.c.p.l.f.: a new data characterization for selecting clustering algorithms using meta-learning. inf. sci. 477, 203–219 (2019) google scholar  quinlan, j.r.: induction of decision trees. mach. learn. 1(1), 81–106 (1986) article  google scholar  ray, s., turi, r.h.: determination of number of clusters in k-means clustering and application in colour segmentation. in: 4th international conference on advances in pattern recognition and digital techniques (icaprdt), pp. 137–143 (1999) google scholar  reif, m., shafait, f., goldstein, m., breuel, t., dengel, a.: automatic classifier selection for non-experts. pattern anal. appl. 17(1), 83–96 (2014) article  mathscinet  google scholar  rice, j.r.: the algorithm selection problem. adv. comput. 15, 65–118 (1976) article  google scholar  rivolli, a., garcia, l.p.f., soares, c., vanschoren, j., de carvalho, a.c.p.l.f.: characterizing classification datasets: a study of meta-features for meta-learning. corr abs/1808.10406, 1–49 (2019) google scholar  rousseeuw, p.j.: silhouettes: a graphical aid to the interpretation and validation of cluster analysis. j. comput. appl. math. 20, 53–65 (1987) article  google scholar  sakamoto, y., ishiguro, m., kitagawa, g.: akaike information criterion statistics. springer, netherlands (1986) math  google scholar  smith-miles, k.a.: cross-disciplinary perspectives on meta-learning for algorithm selection. acm comput. surv. 41(1), 1–25 (2008) article  google scholar  stephenson, n., et al.: survey of machine learning techniques in drug discovery. curr. drug metab. 20(3), 185–193 (2019) article  google scholar  van rijn, j.n., et al.: openml: a collaborative science platform. in: european conference on machine learning and knowledge discovery in databases (ecml/pkdd), pp. 645–649 (2013) google scholar  vukicevic, m., radovanovic, s., delibasic, b., suknovic, m.: extending meta-learning framework for clustering gene expression data with component-based algorithm design and internal evaluation measures. int. j. data min. bioinform. (ijdmb) 14(2), 101–119 (2016) article  google scholar  xie, x.l., beni, g.: a validity measure for fuzzy clustering. ieee trans. pattern anal. mach. intell. 13(8), 841–847 (1991) article  google scholar  download references acknowledgment research carried out using the computational resources of the center for mathematical sciences applied to industry (cemeai) funded by fapesp (grant 2013/07375-0). author information authors and affiliations department of computer science, university of brasília, brasília, brazil luís p. f. garcia & guilherme n. ramos college of engineering and physical sciences, aston university, birmingham, uk felipe campelo computing department, technological university of paraná, cornélio procópio, brazil adriano rivolli institute of mathematical and computer sciences, university of são paulo, são carlos, brazil andré c. p. de l. f. de carvalho authors luís p. f. garciaview author publications search author on:pubmed google scholar felipe campeloview author publications search author on:pubmed google scholar guilherme n. ramosview author publications search author on:pubmed google scholar adriano rivolliview author publications search author on:pubmed google scholar andré c. p. de l. f. de carvalhoview author publications search author on:pubmed google scholar corresponding author correspondence to luís p. f. garcia . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper garcia, l.p.f., campelo, f., ramos, g.n., rivolli, a., de carvalho, a.c.p.d.l.f. (2021). evaluating clustering meta-features for classifier recommendation. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_30 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords meta-learning meta-features characterization measures clustering problems publish with us policies and ethics profiles felipe campelo view author profile andré c. p. de l. f. de carvalho view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a tool for measuring energy consumption in data stream mining | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a tool for measuring energy consumption in data stream mining conference paper first online: 12 october 2023 pp 429–440 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a tool for measuring energy consumption in data stream mining download book pdf download book epub eric kenzo taniguchi onuki  orcid: orcid.org/0009-0008-6478-96179, andreia malucelli  orcid: orcid.org/0000-0002-0929-18749 & jean paul barddal  orcid: orcid.org/0000-0001-9928-854x9  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 532 accesses 1 citation abstract energy consumption reduction is an increasing trend in machine learning given its relevance in socio-ecological importance. consequently, it is important to quantify how real-time learning algorithms tailored for data streams and edge computing behave in terms of accuracy, processing time, memory usage, and energy consumption. in this work, we bring forward a tool for measuring energy consumption in the massive online analysis (moa). first, we analyze the energy consumption rates obtained by our tool against a gold-standard hardware solution, thus showing the robustness of our approach. next, we experimentally analyze classification algorithms under different validation protocols and concept drift and highlight how such classifiers behave under such conditions. results show that our tools enable the identification of different classifiers’ energy consumption. in particular, it allows a better understanding of how energy consumption rates vary in drifting and non-drifting scenarios. finally, given the insights obtained during experimentation on existing classifiers, we make our tool publicly available to the scientific community so that energy consumption is also accounted for in developing and comparing data stream mining algorithms. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others real time monitoring method of comprehensive energy consumption based on data mining algorithm chapter © 2022 machine learning algorithms for energy consumption prediction in smart homes: a comparative study chapter © 2025 predicting unusual energy consumption events from smart home sensor network by data stream mining with misclassified recall article 20 february 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. data analytics data mining energy management energy informatics learning algorithms machine learning 1 introduction data is consistently being generated, stored, and processed. several applications provide data generated in large amounts, frequency, and speed. in this paper, we focus on such scenarios the so-called data streams. data streams are, per definition, potentially unbounded and non-stationary data sequences [10]. consequently, performing data mining to extract useful insights and patterns from such data requires algorithms that are tailored to such scenarios. in contrast to traditional machine learning, the data stream mining area has shown concerns with the trade-off between accuracy and computational resources, i.e., processing time and memory consumption [5]. nonetheless, recent research has shown that we still need more significant steps toward sustainability. for instance, authors in [28] depict that the co2 emissions of training neural networks are larger than that of a car in its lifetime. first, there is no clear relationship between processing time, memory consumption, and energy consumption. even though these components are tied to one another, multi-threading, compiler, and other low-level computational architecture have been shown to impact the entire process, and, consequently, energy consumption cannot be directly estimated from such components [13, 18]. having both sustainability and the lack of generic tools for quantifying energy consumption in streaming scenarios as motivation, we bring forward a tool for researchers and practitioners to investigate how data stream mining algorithms behave under different streaming settings, e.g., with and without concept drifts, different validation schema, etc. in opposition to previous works [15, 16], our tool is generic because it can be coupled with any classifier and data stream available in the massive online analysis (moa) framework [6], which is the off-the-shelf solution for implementing and testing streaming methods. we experimentally evaluate our tool against a hardware solution and assess the energy consumption of different classifiers under different streaming settings. finally, the tool is made publicly available to the scientific community as a byproduct of our research. this paper is divided as follows. section 2 describes data stream mining and brings forward the main concepts in energy consumption. section 3 discusses related works that lie at the intersection of energy consumption and data stream mining. section 4 describes our tool for energy consumption measurements and describes how it has been combined with the massive online analysis (moa) [6] framework. section 5 discusses the analysis conducted to validate our tool and assess different classifiers under different experimental conditions. finally, sect. 6 concludes this work and states envisioned future works. 2 data stream mining and energy consumption data streams are potentially unbounded data sequences made available over time, which may be non-stationary. consequently, storing an entire data stream is unfeasible since it is not entirely available at once, and it would not fit in memory [10]. as a result, researchers and practitioners have devoted efforts towards developing efficient algorithms to process and mine data that arrive sequentially over time. therefore, data stream mining is understood as the investigation of patterns, anomalies, and correlations in streaming data. in particular, in this work, we focus on classification, the most popular task in data stream mining that conveys the prediction of a discrete output given a set of input variables. more formally, we denote a data stream s to provide instances \(i^t = (\boldsymbol{x}^t, y^t)\) at timestamps denoted as t. we also denote classification as the task of learning a predictive model \(f:\boldsymbol{x} \rightarrow y\), where y is a discrete label in y. in practice, we expect predictions \(\hat{y}\) to be accurate given the ground-truth y values. one of the main challenges in data stream mining is concept drift [31], which regards changes in the data distribution that may render a classifier obsolete. formally, a concept \(c = \bigcup _{y_i \in y}{\{(p[y_i], p[\boldsymbol{x} \vert y_i])\}}\) is a set of class priors and class-conditional probability density functions [10]. therefore, a concept drift is said to occur between two timestamps \(t_i\) and \(t_j\) if \(c^{t_i} \ne c^{t_j}\) [12]. consequently, classifiers for data streams must be adaptive, which means that f may be adjusted when newly labeled instances are made available. 2.1 classifiers over the years, different approaches have been developed for the classification task in data streams. in practice, these classifiers are variants of traditional classifiers available for batch scenarios. a popular approach for classification in streaming scenarios is the incremental naive bayes [24]. as its batch counterpart, it assumes that input features are independent. with the arrival of a training instance, all probabilities are updated according. since probabilities are based on counters, and there is no need to store instances, naive bayes has a constant memory consumption and processing time. nonetheless, it does not present any traits to identify and adapt to concept drifts. the most common approach for learning from data streams is decision trees. in particular, hoeffding trees [9] is the most popular approach as it branches over time when statistically enough data (grace period, \(n_{\min }\)) and evidence has been gathered, according to the hoeffding bound [22]. the very fast decision tree (vfdt) is a popular implementation of incremental hoeffding trees, meaning that it continuously branches as new data becomes available and does not revisit the quality of previously created split nodes. in contrast, the characteristic of revisiting split nodes is observed in hoeffding adaptive trees [4], in which each split node is coupled with an adwin drift detector [3]. whenever a drift is flagged, the corresponding split node is replaced by a leaf node, which can branch again if the hoeffding inequality is met. even though hoeffding adaptive trees significantly improve accuracy rates compared to incremental trees, even better results are obtained when creating ensembles of hoeffding trees. a state-of-the-art exemplar of ensembles of hoeffding trees is the adaptive random forest (arf) [19], in which randomized hoeffding trees are trained in parallel and coupled with drift detectors to identify and adapt to concept drifts rapidly. arf adjusts the sampling process with poisson(\(\lambda =6\)) so that instances have higher chances of being used during training, thus speeding up the drift adaptation process. in the test step, classifiers’ votes are combined using weighted majority voting, i.e., classifiers with higher accuracy have a higher impact on the final prediction. consequently, arf is a strong learner that achieves state-of-the-art results in terms of accuracy, yet, at the high expense of computational resources. 2.2 requirements throughout the training and test steps of data stream classification, streaming classifiers must meet certain requirements [5, 6]: requirement #1: process an example at a time, and inspect it only once (at most); requirement #2: use a limited amount of memory; requirement #3: work in a limited amount of time; requirement #4: be ready to predict at any point; and requirement #5: detect and adapt to concept drifts. this list has been incremented in the works of garcía-martín [13, 14, 17], in which energy consumption is highlighted as a relevant aspect in data stream mining since several classifiers have been tailored focusing solely on accuracy and overlooking sustainability. this is one of the main drivers of our work, i.e., to allow researchers and practitioners to quantify the energy consumption of data stream classifiers and determine under which conditions they fail to meet energy sustainability criteria. 3 related works over the years, different approaches to measuring energy consumption have been proposed. this section highlights approaches for measuring energy consumption in data stream mining and decreasing energy usage. a first significant study is [16], in which authors used powerapi [30] to quantify the energy consumption of hoeffding trees despite acknowledging that it overlooks ram consumption. the same authors have changed their approach in [15], in which jalen (now called jourlarx) [26] has been used to quantify energy consumption of hoeffding trees on a function-basis. this allowed the authors to identify bottlenecks in the existing hoeffding tree implementation available in moa. finally, authors in [13] and [14] have used intel’s rapl [7] to quantify the energy consumed by the dram and the processor based on accesses to the processor performance counters. even though intel’s rapl code is not available, authors disclose that its accuracy has been checked in [21] and that it does not introduce processing overheads. it is also relevant to highlight that the work of [14] introduces a hoeffding tree variant in which the grace period (\(n_{\min }\)) is adjusted so that branching is only attempted according to a user-given threshold. the results showed that the proposed hoeffding tree variant has accuracy convergence while approximately 65% less energy consumption rates. on the other hand, the work of [13] introduces a framework to quantify the energy consumption of hoeffding tree ensembles while accounting for decision tree learning, drift detectors, and tree replacement. regarding all of the works mentioned above, a significant drawback is that energy consumption has been tackled solely for hoeffding trees, and no general open-source tool makes energy consumption is easily available for researchers and practitioners. our proposal is brought forward in the next section, and it circumvents such problems. 4 proposal in this section, we detail our tool to quantify the energy consumption of data stream mining algorithms. our tool is embedded within the massive online analysis (moa) framework [6], yet, its rationale can be used in other tools like river [25]. our tool uses intel’s rapl [7] and can be seen as a plugin to the massive online analysis (moa) framework. the general idea of our tool is given in fig. 1, in which we see that rapl is used to quantify energy consumption in data acquisition and models’ training and testing phases. once an experiment starts, energy measurements are initialized. during data stream processing, arriving instances are used to determine processing and energy readings before and after processing, i.e., testing and training steps. these readings are used to compute energy consumption rates during the experiment. for each of the stages, there are different forms of measuring energy consumption. the framework controls the flow of the data stream model to start the energy measurement right before it begins processing the samples. at each cycle, the measurement is taken and presented in real-time to the user. at the end of the process, a graph showing the instantaneous measurement for each cycle is presented to the user. fig. 1. overview of the proposed energy measuring framework. the tool measures energy consumption during data acquisition and model training and testing. full size image since our tool is based on intel rapl, we provide in fig. 2 details on how our plugin interacts with moa, rapl, and the linux kernel. as new data becomes available for processing, the plugin requests measurements from the linux kernel, receiving the response of how much energy has been spent during the testing and training phases. these values are summed and made available whenever the evaluation interface (the so-called evaluation frequency parameter) requests an energy consumption rate. figure 3 exemplifies the energy consumption rates measured by the proposed tool and how it is reported in moa alongside other evaluation metrics. fig. 2. measuring energy consumption using rapl. full size image fig. 3. screenshot of the massive online analysis (moa) framework with results provided by the proposed tool. full size image the source code of our proposal and experimentation can be found at https://github.com/ericonuki/moa-bringing-awareness-green-ict. 5 experiments this section presents the experiments conducted to assess our proposed tool to measure energy consumption in data streams. in particular, this section is divided into two experiments. first, we analyze our tool against a hardware solution, which serves as a gold standard for the energy consumed. next, we analyze the energy consumed by different classifiers in prequential test-then-train validation in stationary and non-stationary scenarios. all tests were performed on a desktop computer running ubuntu desktop 18.04 lts; intel core i7-2600 sandy bridge cpu; 4gb ram, and 250 gb hdd hard drive. 5.1 experiment 1 validation against a hardware solution the first experiment conducted aimed to assess whether the energy consumption measurement via hardware and software solutions were equivalent or, at least, correlated. in this experiment, both software and hardware tp-link hs 110 wall plug solutions were connected to the computer, and a cpu stress called stress-ng [23] was used to quantify energy consumption rates. initially, it was deemed necessary to assess whether the measurement of energy consumption by hardware or software is equivalent and when using only one software tool (the plugin) would not compromise the results of this study. the stress was configured to generate a 10% stress level for 10 min and progressively perform increments by 10% until 100% stress was reached. once 100% stress was reached, the test was incremented to use an extra processor core. this process was repeated until all four cores were allocated. the entire testbed encompassed five runs, and the average results are given in fig. 4. both lines indicate the various stress levels on the computer, with the blue line relating to the software power consumption measurement and the green line the hardware results. these results depict that the software solution does not meet the hardware solution readings. these results are expected since the hardware readings account for the entire computer, i.e., the operational system and other software that is being run; thus, it is reasonable to assume that the stress occupies a part of the overall energy being consumed. even though it is clear that the results do not match, they possess a 99.97% correlation, which depicts a strong correlation. therefore, we verify that although the software solution does not accurately describe a computer’s total energy consumption, its results correlate with the actual consumption measured from a hardware tool, as also observed in [8, 27]. fig. 4. comparison of the energy consumption quantified by hardware and software. full size image 5.2 experiment 2 analyzing different classifiers in stationary and non-stationary environments in this experiment, we used our proposed tool to quantify the energy consumption of different classifiers in stationary and non-stationary environments. in particular, synthetic data streams were created using the massive online analysis (moa) framework using the agrawal (agr) [1], assets negotiation (an) [2], and sea [29] generators. each stream possessed 1 million instances, and two variants were created, one with concept drifts located at the middle of the experiment (drift position equal to 500,000) and another without concept drift. experiment variants marked with a -d suffix stand for drifting experiments. the data streams mentioned above were used to assess naive bayes (nb), hoeffding tree (ht), hoeffding adaptive tree (hat), and adaptive random forest (arf) classifiers. all classifiers used the default parameters available in moa, except for arf, which used 100 ensemble members (each with an individual thread) and a grace period \(n_{min}=50\). all experiments were conducted using the prequential validation scheme proposed in [11], i.e., each instance was retrieved and used for testing and training. the source code to reproduce this experimentation is also available in the code repository. discussion. the results obtained are given in tables 1, 2, 3, and 4. these tables provides accuracy, processing time (in seconds), memory consumption (in gb-hours), and energy consumption rates (in watts), respectively. first, we highlight that naive bayes (nb) has the lowest accuracy rates in all scenarios. this result corroborates that decision trees and their ensembles are more interesting when this particular evaluation metric is pursuit. we highlight, for instance, the results obtained by arf in drifting experiments, which are expected since it has multiple learners coupled with drift detectors to detect and adapt to such changes. nonetheless, decision trees bring forward computational overheads that are quantified by the remainder of the metrics. first, we see that hoeffding tree (ht), hoeffding adaptive tree (hat), and the adaptive random forest (arf) are slower than naive bayes (nb). the processing times for ht, hat, and arf are 7, 11, and 44,751 times slower than nb, respectively. similar results are observed for ram consumption, in which ht, hat, and arf consume more ram than nb. again, we highlight the ram consumption observed by arf, which is approximately \(10^{8}\) times higher than its counterparts. finally, the energy consumption values depict that nb is the less-consuming algorithm, which the exception of hat in the agr-d experiment. these results are expected since nb is much faster and less memory-consuming; thus, less energy is required to finalize an experiment. regarding the agr-d experiment, it is relevant to emphasize that hat’s energy consumption has decreased since it has a drift detector that restarted the entire tree learning process, i.e., it replaced the entire tree with a single decision stump, and thus, its computational cost after the drift has greatly decreased. this is a relevant scenario in which energy consumption is not directly related to processing time and memory consumption, and it allows a better analysis by researchers and practitioners on which classifier should be used in a specific scenario. focusing on arf, we also highlight that the energy consumption rates are not directly related to either processing time and memory consumption rates since, despite taking much more time and memory to run, its energy consumption was roughly twice when compared to its counterparts. this result can be explained due to arf’s implementation, which is multi-threaded, and even by combining 100 learners the energy consumption is not 100 times greater than its counterparts. table 1. accuracy results obtained during experimentation.full size table table 2. memory consumption results obtained during experimentation.full size table table 3. processing time results obtained during experimentation.full size table table 4. energy consumption results obtained during experimentation.full size table 6 conclusion in this work, we brought forward a tool for quantifying energy consumption in data stream mining. our tool was embedded within the massive online analysis (moa) framework, thus allowing researchers to rapidly quantify the energy consumption of different classification methods under different streaming settings and validation processes. to validate our proposal, we first conducted a testbed using a processor stress to compare the proposed software readings against a hardware wall plug. next, we tested different classifiers in stationary and drifting scenarios in a prequential validation scheme. results showed that our tool allows the identification of energy consumption rates of different classifiers under different scenarios, i.e., drifting and non-drifting data streams. the energy consumption rates allow a more fine-grained analysis of the classifiers as energy consumption is not directly tied with processing time and memory consumption, especially under concept drifting scenarios and multi-threading implementations. in future works, we plan to extend our tool to encompass different classification, regression, and clustering validation schemes, including more datasets and scenarios. we also plan to port our tool to python-based frameworks, such as river [25]. finally, we also plan to make our tool available in amd and apple’s arm platforms and add support to gpu consumption since neural networks are increasingly used in streaming settings [20]. references agrawal, r., imielinski, t., swami, a.: database mining: a performance perspective. ieee trans. knowl. data eng. 5(6), 914–925 (1993) article  google scholar  barddal, j.p., murilo gomes, h., enembreck, f., pfahringer, b., bifet, a.: on dynamic feature weighting for feature drifting data streams. in: frasconi, p., landwehr, n., manco, g., vreeken, j. (eds.) ecml pkdd 2016. lncs (lnai), vol. 9852, pp. 129–144. springer, cham (2016). https://doi.org/10.1007/978-3-319-46227-1_9 chapter  google scholar  bifet, a., gavalda, r.: learning from time-changing data with adaptive windowing. in: proceedings of the 2007 siam international conference on data mining, pp. 443–448. siam (2007) google scholar  bifet, a., gavaldà, r.: adaptive learning from evolving data streams. in: adams, n.m., robardet, c., siebes, a., boulicaut, j.-f. (eds.) ida 2009. lncs, vol. 5772, pp. 249–260. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-03915-7_22 chapter  google scholar  bifet, a., gavalda, r., holmes, g., pfahringer, b.: machine learning for data streams: with practical examples in moa. mit press, cambridge (2023) google scholar  bifet, a., holmes, g., kirkby, r., pfahringer, b.: moa: massive online analysis. j. mach. learn. res. 11(52), 1601–1604 (2010). http://jmlr.org/papers/v11/bifet10a.html david, h., gorbatov, e., hanebutte, u.r., khanna, r., le, c.: rapl: memory power estimation and capping. in: proceedings of the 16th acm/ieee international symposium on low power electronics and design. pp. 189–194. islped 2010, association for computing machinery, new york, ny, usa (2010). https://doi.org/10.1145/1840845.1840883 desrochers, s., paradis, c., weaver, v.m.: a validation of dram rapl power measurements. in: proceedings of the second international symposium on memory systems, pp. 455–470. memsys 2016, association for computing machinery, new york, ny, usa (2016). https://doi.org/10.1145/2989081.2989088 domingos, p., hulten, g.: mining high-speed data streams. in: proceedings of the sixth acm sigkdd international conference on knowledge discovery and data mining, pp. 71–80. kdd 2000, association for computing machinery, new york, ny, usa (2000). https://doi.org/10.1145/347090.347107 gama, j.: knowledge discovery from data streams. chapman & hall/crc, 1st edn. (2010) google scholar  gama, j., sebastião, r., rodrigues, p.p.: on evaluating stream learning algorithms. mach. learn. 90(3), 317–346 (2012). https://doi.org/10.1007/s10994-012-5320-9 article  mathscinet  math  google scholar  gama, j., zliobaite, i., bifet, a., pechenizkiy, m., bouchachia, a.: a survey on concept drift adaptation (2014). https://doi.org/10.1145/2523813 garcía-martín, e., bifet, a., lavesson, n.: energy modeling of hoeffding tree ensembles. intell. data anal. 25(1), 81–104 (2020) google scholar  garcía-martín, e., bifet, a., lavesson, n.: green accelerated hoeffding tree (2020). http://urn.kb.se/resolve?urn=urn:nbn:se:bth-19152 garcia-martin, e., lavesson, n., grahn, h.: energy efficiency analysis of the very fast decision tree algorithm. in: trends in social network analysis: information propagation, user behavior modeling, forecasting, and vulnerability assessment, pp. 229–252 (2017) google scholar  garcia-martin, e., lavesson, n., grahn, h.: identification of energy hotspots: a case study of the very fast decision tree. in: au, m.h.a., castiglione, a., choo, k.-k.r., palmieri, f., li, k.-c. (eds.) gpc 2017. lncs, vol. 10232, pp. 267–281. springer, cham (2017). https://doi.org/10.1007/978-3-319-57186-7_21 chapter  google scholar  garcía-martín, e., lavesson, n., grahn, h., casalicchio, e., boeva, v.: how to measure energy consumption in machine learning algorithms. in: alzate, c., et al. (eds.) ecml pkdd 2018. lncs (lnai), vol. 11329, pp. 243–255. springer, cham (2019). https://doi.org/10.1007/978-3-030-13453-2_20 chapter  google scholar  garcía-martín, e., rodrigues, c.f., riley, g., grahn, h.: estimation of energy consumption in machine learning. j. parallel distrib. comput. 134, 75–88 (2019). https://doi.org/10.1016/j.jpdc.2019.07.007 article  google scholar  gomes, h.m., et al.: adaptive random forests for evolving data stream classification. mach. learn. 106(9), 1469–1495 (2017). https://doi.org/10.1007/s10994-017-5642-8, https://doi.org/10.1007/s10994-017-5642-8 gunasekara, n., gomes, h.m., pfahringer, b., bifet, a.: online hyperparameter optimization for streaming neural networks. in: 2022 international joint conference on neural networks (ijcnn), pp. 1–9. ieee (2022) google scholar  hähnel, m., döbel, b., völp, m., härtig, h.: measuring energy consumption for short code paths using rapl. acm sigmetrics perform. eval. rev. 40(3), 13–17 (2012) article  google scholar  hoeffding, w.: probability inequalities for sums of bounded random variables. in: fisher, n.i., sen, p.k. (eds.) the collected works of wassily hoeffding. springer series in statistics. springer, new york, ny (1994). https://doi.org/10.1007/978-1-4612-0865-5_26 king, c.i.: stress-ng. http://kernel.ubuntu.com/git/cking/stressng.git/. visited on 28/03/2018), p. 39 (2017) klawonn, f., angelov, p.: evolving extended naive bayes classifiers. in: sixth ieee international conference on data mining-workshops (icdmw 2006), pp. 643–647. ieee (2006) google scholar  montiel, j.: river: machine learning for streaming data in python. j. mach. learn. res. 22(1), 4945–4952 (2021) google scholar  noureddine, a.: powerjoular and joularjx: multi-platform software power monitoring tools. in: 2022 18th international conference on intelligent environments (ie), pp. 1–4. ieee (2022) google scholar  phung, j., lee, y.c., zomaya, a.y.: modeling system-level power consumption profiles using rapl. in: nca 2018–2018 ieee 17th international symposium on network computing and applications. institute of electrical and electronics engineers inc. (2018). https://doi.org/10.1109/nca.2018.8548281 shao, y.s., brooks, d.: energy characterization and instruction-level energy model of intel’s xeon phi processor. in: international symposium on low power electronics and design (islped), pp. 389–394. ieee (2013) google scholar  street, w.n., kim, y.: a streaming ensemble algorithm (sea) for large-scale classification. in: proceedings of the seventh acm sigkdd international conference on knowledge discovery and data mining, pp. 377–382 (2001) google scholar  terpstra, d., jagode, h., you, h., dongarra, j.: collecting performance data with papi-c. in: müller, m., resch, m., schulz, a., nagel, w. (eds.) tools for high performance computing 2009. springer, berlin, heidelberg (2010). https://doi.org/10.1007/978-3-642-11261-4_11 widmer, g., kubat, m.: learning in the presence of concept drift and hidden contexts. mach. learn. 23, 69–101 (1996) article  google scholar  download references author information authors and affiliations programa de pós-graduação em informática (ppgia), pontifícia universidade católica do paraná (pucpr), curitiba, paraná, brazil eric kenzo taniguchi onuki, andreia malucelli & jean paul barddal authors eric kenzo taniguchi onukiview author publications search author on:pubmed google scholar andreia malucelliview author publications search author on:pubmed google scholar jean paul barddalview author publications search author on:pubmed google scholar corresponding author correspondence to jean paul barddal . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper onuki, e.k.t., malucelli, a., barddal, j.p. (2023). a tool for measuring energy consumption in data stream mining. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_28 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords data stream mining energy consumption green computing publish with us policies and ethics profiles jean paul barddal view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature lecture notes in computer science lecture notes in artificial intelligence 14197 founding editor jörg siekmann series editors randy goebel, university of alberta, edmonton, canada wolfgang wahlster, dfki, berlin, germany zhi-hua zhou, nanjing university, nanjing, china the series lecture notes in artificial intelligence (lnai) was established in 1988 as a topical subseries of lncs devoted to artificial intelligence. the series publishes state-of-the-art research results at a high level.aswith thelncs mother series, the mission of the series is to serve the international r & d community by providing an invaluable service, mainly focused on the publication of conference and workshop proceedings and postproceedings. murilo c. naldi · reinaldo a. c. bianchi editors intelligent systems 12th brazilian conference, bracis 2023 belo horizonte, brazil, september 25–29, 2023 proceedings, part iii editors murilo c. naldi federal university of são carlos são carlos, brazil reinaldo a. c. bianchi centro universitario da fei são bernardo do campo, brazil issn 0302-9743 issn 1611-3349 (electronic) lecture notes in artificial intelligence isbn 978-3-031-45391-5 isbn 978-3-031-45392-2 (ebook) https://doi.org/10.1007/978-3-031-45392-2 lncs sublibrary: sl7 – artificial intelligence © the editor(s) (if applicable) and the author(s), under exclusive license to springer nature switzerland ag 2023 this work is subject to copyright. all rights are reserved by the publisher, whether the whole or part of the material is concerned, specifically the rights of translation, reprinting, reuse of illustrations, recitation, broadcasting, reproduction on microfilms or in any other physical way, and transmission or information storage and retrieval, electronic adaptation, computer software, or by similar or dissimilar methodology now known or hereafter developed. the use of general descriptive names, registered names, trademarks, service marks, etc. in this publication does not imply, even in the absence of a specific statement, that such names are exempt from the relevant protective laws and regulations and therefore free for general use. the publisher, the authors, and the editors are safe to assume that the advice and information in this book are believed to be true and accurate at the date of publication. neither the publisher nor the authors or the editors give a warranty, expressed or implied, with respect to the material contained herein or for any errors or omissions that may have been made. the publisher remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. this springer imprint is published by the registered company springer nature switzerland ag the registered company address is: gewerbestrasse 11, 6330 cham, switzerland paper in this product is recyclable. https://orcid.org/0000-0002-3107-8236 https://orcid.org/0000-0001-9097-827x https://doi.org/10.1007/978-3-031-45392-2 preface the 12th brazilian conference on intelligent systems (bracis 2023) was one of the most important events held in brazil in 2023 for researchers interested in publishing significant and novel results related to artificial and computational intelligence. the brazilian conference on intelligent systems (bracis) originated from the combination of the two most important scientific events in brazil in artificial intelligence (ai) and computational intelligence (ci): the brazilian symposium on artificial intelligence (sbia, 21 editions) and the brazilian symposium on neural networks (sbrn, 12 editions). the brazilian computer society (sbc) supports the event, the special committee of artificial intelligence (ceia), and the special committee of computational intelligence (ceic). the conference aims to promote theoretical aspects and applications of artificial and computational intelligence and exchange scientific ideas among researchers, practitioners, scientists, engineers, and industry. in 2023, bracis took place in belo horizonte, brazil, from september 25th to 29th, 2023, in the campus of the universidade federal de minas gerais. the event was held in conjunction with two other events: the national meeting on artificial and computational intelligence (eniac) and the symposium on information and human language technology (stil). bracis 2023 received 242 submissions. all papers were rigorously reviewed by an international program committee (with a minimum of three double-blind peer reviews per submission), followed by a discussion phase for conflicting reports. after the review process, 89 papers were selected for publication in three volumes of the lecture notes in artificial intelligence series (an acceptance rate of 37%). the topics of interest included, but were not limited to, the following: – agent-based and multi-agent systems – bioinformatics and biomedical engineering – cognitive modeling and human interaction – combinatorial and numerical optimization – computer vision – constraints and search – deep learning – distributed ai – education – ethics – evolutionary computation and metaheuristics – forecasting – foundation models – foundations of ai – fuzzy systems – game playing and intelligent interactive entertainment – human-centric ai vi preface – hybrid systems – information retrieval, integration, and extraction – intelligent robotics – knowledge representation and reasoning – knowledge representation and reasoning in ontologies and the semantic web – logic-based knowledge representation and reasoning – machine learning and data mining – meta-learning – molecular and quantum computing – multidisciplinary ai and ci – natural language processing – neural networks – pattern recognition and cluster analysis – planning and scheduling – reinforcement learning we want to thank everyone involved in bracis 2023 for helping to make it a success: we are very grateful to the programcommittee members and reviewers for their volunteered contribution to the reviewing process; we would also like to thank all the authors who submitted their papers and laboriously worked to have the best final version possible; the general chairs and the local organization committee for supporting the conference; the brazilian computing society (sbc); and all the conference’s sponsors and supporters. we are confident that these proceedings reflect the excellent work in the artificial and computation intelligence communities. september 2023 murilo c. naldi reinaldo a. c. bianchi organization general chairs gisele lobo pappa universidade federal de minas gerais, brazil wagner meira jr. universidade federal de minas gerais, brazil program committee chairs murilo coelho naldi universidade federal de são carlos, brazil reinaldo a. c. bianchi centro universitário fei, brazil steering committee aline paes universidade federal fluminense, brazil andré britto universidade federal do sergipe, brazil anna h. r. costa universidade de são paulo, brazil anne canuto universidade federal do rio grande do norte, brazil arnaldo cândido jr. universidade estadual paulista, brazil felipe meneguzzi university of aberdeen, uk filipe saraiva universidade federal do pará, brazil gina m. b. oliveira universidade federal uberlandia, brazil helida santos universidade federal do rio grande, brazil leliane n. de barros universidade de são paulo, brazil livy real b2w digital, brazil maria v. de menezes universidade federal do ceará, brazil marlo souza universidade federal da bahia, brazil renato tinos universidade de são paulo, brazil ricardo marcacini universidade de são paulo, brazil tatiane nogueira universidade federal da bahia, brazil thiago pardo universidade de são paulo são carlos, brazil viii organization program committee adenilton da silva universidade federal de pernambuco, brazil adriane serapião universidade estadual paulista, brazil adrião duarte d. neto universidade federal do rio grande do norte, brazil alexandre salle universidade federal do rio grande do sul, brazil aline neves universidade federal do abc, brazil aline paes universidade federal fluminense, brazil alneu lopes universidade de são paulo são carlos, brazil aluizio araújo universidade federal de pernambuco, brazil alvaro moreira universidade federal do rio grande do sul, brazil amedeo napoli loria, france ana bazzan universidade federal do rio grande do sul, brazil ana carolina lorena instituto tecnológico de aeronáutica, brazil ana c. b. k. vendramin universidade tecnológica federal do paraná, brazil anderson soares universidade federal de goiás, brazil andré britto universidade federal do sergipe, brazil andré p. l. f. carvalho universidade de são paulo são carlos, brazil andré rossi universidade estadual paulista, brazil andré ruela marinha do brasil, brazil andré takahata universidade federal do abc, brazil andrés e. c. salazar universidade tecnológica federal do paraná, brazil anna h. r. costa universidade de são paulo, brazil anne canuto universidade federal do rio grande do norte, brazil araken santos universidade federal rural do semi-árido, brazil artur jordão universidade de são paulo, brazil aurora pozo universidade federal do paraná, brazil bernardo gonçalves universidade de são paulo, brazil bruno masiero universidade estadual de campinas, brazil bruno nogueira universidade federal de mato grosso do sul, brazil bruno souza universidade federal do maranhão, brazil bruno veloso universidade portucalense, portugal carlos ribeiro instituto tecnológico de aeronáutica, brazil carlos silla pontifícia universidade católica do paraná, brazil organization ix carlos thomaz centro universitário fei, brazil carlos a. e. montesco universidade federal de sergipe, brazil carlos e. pantoja centro federal de educação tecnológica rj, brazil carolina p. de almeida universidade e. do centro-oeste do paraná, brazil celia ralha universidade de brasilia, brazil claudio bordin jr. universidade federal do abc, brazil claudio toledo universidade de são paulo, brazil cleber zanchettin universidade federal de pernambuco, brazil cristiano torezzan universidade estadual de campinas, brazil daniel araújo universidade federal do rio grande do norte, brazil daniel dantas universidade federal de sergipe, brazil danilo perico centro universitário fei, brazil danilo sanches universidade tecnológica federal do paraná, brazil debora medeiros universidade federal do abc, brazil denis mauá universidade de são paulo, brazil dennis b. aranibar universidad católica san pablo, peru diana adamatti universidade federal do rio grande, brazil diego furtado silva universidade de são paulo, brazil donghong ji wuhan university, china eder m. gonçalves universidade federal do rio grande, brazil edson gomi universidade de são paulo, brazil edson matsubara universidade federal de mato grosso do sul, brazil eduardo costa corteva agriscience, brazil eduardo goncalves escola nacional de ciências estatísticas, brazil eduardo palmeira universidade estadual de santa cruz, brazil eduardo spinosa universidade federal do paraná, brazil edward h. haeusler pontifícia universidade católica do r. de j., brazil elaine faria universidade federal uberlandia, brazil elizabeth goldbarg universidade federal do rio grande do norte, brazil emerson paraiso pontificia universidade catolica do paraná, brazil eric araújo universidade federal de lavras, brazil evandro costa universidade federal de alagoas, brazil fabiano silva universidade federal do paraná, brazil fábio cozman universidade de são paulo, brazil felipe leno da silva lawrence livermore national lab., usa felipe meneguzzi university of aberdeen, uk x organization felix antreich instituto tecnológico de aeronáutica, brazil fernando osório universidade de são paulo, são carlos, brazil flavia bernardini universidade federal fluminense, brazil flavio tonidandel centro universitário fei, brazil flávio s. c. da silva universidade de são paulo, brazil francisco chicano university of málaga, spain francisco de carvalho universidade federal de pernambuco, brazil gabriel ramos universidade do vale do rio dos sinos, brazil george cavalcanti universidade federal de pernambuco, brazil gerson zaverucha universidade federal do rio de janeiro, brazil giancarlo lucca universidade federal do rio grande, brazil gisele pappa universidade federal de minas gerais, brazil gracaliz dimuro universidade federal do rio grande, brazil guilherme barreto universidade federal do ceará, brazil guilherme coelho universidade estadual de campinas, brazil guilherme derenievicz universidade federal do paraná, brazil guilherme d. pelegrina universidade estadual de campinas, brazil guillermo simari universidad nacional del sur in b. b., argentina gustavo giménez-lugo universidade tecnológica federal do paraná, brazil heitor gomes victoria university of wellington, new zealand helena caseli universidade federal de são carlos, brazil helida santos universidade federal do rio grande, brazil heloisa camargo universidade federal de são carlos, brazil huei lee universidade estadual do oeste do paraná, brazil isaac da silva centro universitário fei, brazil ivandré paraboni universidade de são paulo, brazil ivette luna universidade estadual de campinas, brazil jaime s. sichman universidade de são paulo, brazil jean paul barddal pontifícia universidade católica do paraná, brazil joão papa universidade estadual paulista, brazil joão c. xavier-júnior universidade federal do rn (ufrn), brazil joão paulo canário stone co., brazil jomi hübner universidade federal de santa catarina, brazil jonathan andrade silva universidade federal de mato grosso do sul, brazil josé antonio sanz universidad publica de navarra, spain josé a. baranauskas universidade de são paulo, brazil jose e. o. luna universidad católica san pablo, peru julio nievola pontifícia universidade católica do paraná, brazil karla roberta lima universidade de são paulo, brazil organization xi karliane vale universidade federal do rio grande do norte, brazil kate revoredo humboldt-universität zu berlin, germany krysia broda imperial college, uk laura de miguel universidad pública de navarra, spain leila bergamasco centro universitário fei, brazil leliane nunes de barros universidade de são paulo, brazil leonardo emmendorfer universidade federal do rio grande, brazil leonardo matos universidade federal de sergipe, brazil leonardo t. duarte universidade estadual de campinas, brazil leonardo f. r. ribeiro amazon, usa levy boccato universidade estadual de campinas, brazil li weigang universidade de brasilia, brazil livy real b2w digital, brazil lucelene lopes universidade de são paulo são carlos, brazil luciano digiampietri universidade de são paulo, brazil luis garcia universidade de brasília, brazil luiz h. merschmann universidade federal de lavras, brazil marcela ribeiro universidade federal de são carlos, brazil marcelo finger universidade de são paulo, brazil marcilio de souto université d’orléans, france marcos domingues universidade estadual de maringá, brazil marcos quiles universidade federal de são paulo, brazil maria claudia castro centro universitario fei, brazil maria do c. nicoletti universidade federal de são carlos, brazil marilton aguiar universidade federal de pelotas, brazil marley m. b. r. vellasco pontifícia universidade católica do r. de j., brazil marlo souza universidade federal da bahia, brazil marlon mathias universidade de são paulo, brazil mauri ferrandin universidade federal de santa catarina, brazil márcio basgalupp universidade federal de são paulo, brazil mário benevides universidade federal fluminense, brazil moacir ponti universidade de são paulo, brazil murillo carneiro universidade federal de uberlândia, brazil murilo loiola universidade federal do abc, brazil murilo naldi universidade federal de são carlos, brazil myriam delgado universidade tecnológica federal do paraná, brazil nuno david instituto universitário de lisboa, portugal patrícia tedesco universidade federal de pernambuco, brazil paula paro costa universidade estadual de campinas, brazil xii organization paulo cavalin ibm research, brazil paulo pirozelli universidade de são paulo, brazil paulo quaresma universidade de évora, portugal paulo santos flinders university, australia paulo henrique pisani universidade federal do abc, brazil paulo t. guerra universidade federal do ceará, brazil petrucio viana universidade federal fluminense, brazil priscila lima universidade federal do rio de janeiro, brazil priscila b. rampazzo universidade estadual de campinas, brazil rafael giusti universidade federal do amazonas, brazil rafael g. mantovani universidade tecnológica federal do paraná, brazil rafael parpinelli universidade do estado de santa catarina, brazil reinaldo a. c. bianchi centro universitario fei, brazil renato krohling ufes universidade federal do espírito santo, brazil renato tinos universidade de são paulo, brazil ricardo cerri universidade federal de são carlos, brazil ricardo marcacini universidade de são paulo são carlos, brazil ricardo prudêncio universidade federal de pernambuco, brazil ricardo rios universidade federal da bahia, brazil ricardo suyama universidade federal do abc, brazil ricardo a. s. fernandes universidade federal de são carlos, brazil roberta sinoara instituto federal de c., e. e t. de são paulo, brazil roberto santana university of the basque country, spain rodrigo wilkens université catholique de louvain, belgium romis attux universidade estadual de campinas, brazil ronaldo prati universidade federal do abc, brazil rosangela ballini universidade estadual de campinas, brazil roseli a. f. romero universidade de são paulo são carlos, brazil rosiane de freitas r. universidade federal do amazonas, brazil sandra sandri instituto nacional de pesquisas espaciais, brazil sandro rigo universidade do vale do rio dos sinos, brazil sílvia maia universidade federal do rio grande do norte, brazil sílvio cazella universidade federal de ciências da s. de p. a., brazil silvia botelho universidade federal do rio grande, brazil solange rezende universidade de são paulo são carlos, brazil tatiane nogueira universidade federal da bahia, brazil thiago covoes wildlife studios, brazil thiago homem instituto federal de e., c. e t. de são paulo, brazil organization xiii thiago pardo universidade de são paulo são carlos, brazil tiago almeida universidade federal de são carlos, brazil tiago tavares insper, brazil valdinei freire universidade de são paulo, brazil valerie camps paul sabatier university, france valmir macario universidade federal rural de pernambuco, brazil vasco furtado universidade de fortaleza, brazil vinicius souza pontificia universidade católica do paraná, brazil viviane torres da silva ibm research, brazil vladimir rocha universidade federal do abc, brazil washington oliveira universidade estadual de campinas, brazil yván túpac universidad católica san pablo, peru zhao liang universidade de são paulo, brazil additional reviewers alexandre alcoforado alexandre lucena aline del valle aline ioste allan santos ana ligia scott anderson moraes antonio dourado antonio leme arthur dos santos brenno alencar bruna zamith santos bruno labres carlos caetano carlos forster carlos josé andrioli caroline pires alavez moraes cedric marco-detchart cinara ghedini daiane cardoso daniel da silva junior daniel guerreiro e silva daniela vianna diego cavalca douglas meneghetti edilene campos edson borin eduardo costa lopes eduardo max elias silva eliton perin emely silva estela ribeiro eulanda santos fabian cardoso fabio lima fagner cunha felipe serras felipe zeiser fernando pujaico rivera guilherme mello guilherme santos israel fama javier fumanal jefferson oliva joão fabro joão lucas luz lima sarcinelli joelson sartori jorge luís amaral josé angelo gurzoni jr. josé gilberto medeiros junior juan colonna xiv organization leandro lima leandro miranda leandro stival leonardo da silva costa lucas alegre lucas buzuti lucas carlini lucas da silva lucas pavelski lucas queiroz lucas rodrigues lucas francisco pellicer luciano cabral luiz celso gomes jr. maëlic neau maiko lie marcelino abel marcella martins marcelo polido marcos josé marcos vinícius dos santos ferreira marisol gomez matheus rocha mária minárová miguel de mello carpi mikel sesma murillo bouzon murilo falleiros lemos schmitt newton spolaôr odelmo nascimento pablo silva paulo rodrigues pedro da cruz rafael berri rafael gomes mantovani rafael krummenauer rafael orsi ramon abílio richard gonçalves rubens chaves sarah negreiros de carvalho leite silvio romero de araújo júnior tatiany heiderich thiago bulhões da silva costa thiago carvalho thiago dal pont thiago miranda thomas palmeira ferraz tiago asmus tito spadini victor varela vitor machado weber takaki weverson pereira xabier gonzalez contents – part iii evolutionary algorithms multiobjective evolutionary algorithms applied to the optimization of expanded genetic codes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 maísa de carvalho silva, paulo guilherme pinheiro pereira, lariza laura de oliveira, and renato tinós genetic algorithms with optimality cuts to the max-cut problem . . . . . . . . . . . 17 pablo luiz braga soares and carlos victor dantas araújo assessment of robust multi-objective evolutionary algorithms on robust and noisy environments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 mateus clemente de sousa, ivan reinaldo meneghini, and frederico gadelha guimarães optimization strategies binary flying squirrel optimizer for feature selection . . . . . . . . . . . . . . . . . . . . . 51 luiz fernando merli de oliveira sementille, douglas rodrigues, andré nunes de souuza, and joão paulo papa fitness landscape analysis of tpot using local optima network . . . . . . . . . . 65 matheus cândido teixeira and gisele lobo pappa optimization strategies for bert-based named entity recognition . . . . . . . . . . 80 monique monteiro and cleber zanchettin flexcon-ce: a semi-supervised method with an ensemble-based adaptive confidence . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95 arthur medeiros, arthur c. gorgônio, karliane medeiros ovidio vale, flavius l. gorgônio, and anne magály de paula canuto computer vision single image super-resolution based on capsule neural networks . . . . . . . . . . 113 george corrêa de araújo, artur jordão, and helio pedrini xvi contents – part iii development of a deep learning model for the classification of mosquito larvae images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 129 ramon mayor martins, bruno manarin espíndola, pedro philippi araujo, christiane gresse von wangenheim, carlos josé de carvalho pinto, and gisele caminha a simple and low-cost method for leaf surface dimension estimation based on digital images . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 146 karla gabriele florentino da silva, jonas magalhães moreira, gabriel barreto calixto, luiz maurílio da silva maciel, márcio assis miranda, and leandro elias morais crop row line detection with auxiliary segmentation task . . . . . . . . . . . . . . . . 162 igor ferreira da costa and wouter caarls multiple object tracking in native bee hives: a case study with jataí in the field . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 176 rodolfo r. v. leocádio, alan kardek rêgo segundo, and gustavo pessin an open source eye gaze tracker system to perform remote user testing evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 192 marc g. capdevila, karine aparecida p. rodrigues, camila f. jardim, and renato m. silva language and models who killed the winograd schema challenge? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 211 hugo neri and fabio g. cozman sabiá: portuguese large language models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 226 ramon pires, hugo abonizio, thales sales almeida, and rodrigo nogueira disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 241 lucelene lopes, paulo fernandes, marcio l. inacio, magali s. duran, and thiago a. s. pardo bete: a brazilian portuguese dataset for named entity recognition and relation extraction in the diabetes healthcare domain . . . . . . . . . . . . . . . . . 256 lucas pavanelli, yohan bonescki gumiel, thiago ferreira, adriana pagano, and eduardo laber contents – part iii xvii legalbert-pt: a pretrained language model for the brazilian portuguese legal domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 268 raquel silveira, caio ponte, vitor almeida, vládia pinheiro, and vasco furtado a framework for controversial political topics identification using twitter data . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 283 kenzo sakiyama, lucas de souza rodrigues, bruno magalhães nogueira, edson takashi matsubara, and roseli a. f. romero leveraging sign language processing with formal signwriting and deep learning architectures . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 299 fernando de almeida freitas, sarajane marques peres, otávio de paula albuquerque, and marcelo fantinato a clustering validation index based on semantic description . . . . . . . . . . . . . . . 315 roberto douglas guimarães de aquino, vitor venceslau curtis, and filipe alves neto verri graph neural networks detecting multiple epidemic sources in network epidemics using graph neural networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 331 rodrigo gonçalves haddad and daniel ratton figueiredo prediction of cancer-related mirna targets using an integrative heterogeneous graph neural network-based method . . . . . . . . . . . . . . . . . . . . . . 346 emanoel aurelio vianna fabiano and mariana recamonde-mendoza time series forecasting of covid-19 cases in brazil with gnn and mobility networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 361 fernando henrique oliveira duarte, gladston j. p. moreira, eduardo j. s. luz, leonardo b. l. santos, and vander l. s. freitas pattern recognition federated learning and mel-spectrograms for physical violence detection in audio . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 379 victor e. de s. silva, tiago b. lacerda, péricles miranda, andré câmara, amerson riley cabral chagas, and ana paula c. furtado xviii contents – part iii police report similarity search: a case study . . . . . . . . . . . . . . . . . . . . . . . . . . . . 394 josé alan firmiano araújo, ticiana l. coelho da silva, atslands rego da rocha, and vinicius cezar monteiro de lira evaluating contextualized embeddings for topic modeling in public bidding domain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 410 henrique r. hott, mariana o. silva, gabriel p. oliveira, michele a. brandão, anisio lacerda, and gisele pappa ai applications a tool for measuring energy consumption in data stream mining . . . . . . . . . . . 429 eric kenzo taniguchi onuki, andreia malucelli, and jean paul barddal improved fuzzy decision system for energy bill reduction in the context of the brazilian white tariff scenario . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 441 raimunda branco and filipe saraiva exploring artificial intelligence methods for the automatic measurement of a new biomarker aiming at glaucoma diagnosis . . . . . . . . . . . . . . . . . . . . . . . 455 gabriel c. fernandes, fabio lavinsky, sandro josé rigo, and henrique c. bohn investigation of deep active self-learning algorithms applied to named entity recognition . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 470 josé reinaldo cunha santos a. v. silva neto and thiago de paulo faleiros author index . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 485 preface organization contents – part iii deep convolutional features for fingerprint indexing | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deep convolutional features for fingerprint indexing conference paper first online: 28 november 2021 pp 223–237 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) deep convolutional features for fingerprint indexing download book pdf download book epub leonardo f. da costa10, lucas s. fernandes10, joão p. b. andrade10, paulo a. l. rego10 & … josé g. r. maia10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1273 accesses 1 citation 2 altmetric abstract automatic fingerprint identification systems (afis) are among the most used people identification solutions. as the size of fingerprint databases is continuously growing, studying fingerprint indexing mechanisms is desirable to facilitate the search process in a large-scale database. this work presents a method for fingerprint indexing, which uses both exact and approximation methods of nearest neighbors (anns), which are very efficient in terms of runtime, even if they sacrifice a little accuracy by presenting approximate solutions. in the presented approach, searches with ann methods are made from deep embedding vectors extracted from image databases using a convolutional neural network (cnn). in this work, a cnn resnet18 was used to extract the deep feature embeddings vectors, and the vectors vary in size between 64, 96, and 128. the anns methods tested for the query step were annoy, ngt, hnsw, and nanoflann. the results were quite promising when using the fvc fingerprint databases (2000, 2002, and 2004), once we reached 100% hits in the searches with a penetration rate of 1%, with very low run times. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others robust fingerprint matching based on convolutional neural networks chapter © 2018 an efficient deep learning model fvnet for fingerprint verification chapter © 2023 fingerprint segmentation via convolutional neural networks chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition biometrics computational intelligence computer vision machine learning artificial intelligence 1 introduction the use of fingerprints as a form of authentication and identification has proven to be a robust and efficient method over the past few years. it is assumed that fingerprints are unique to each individual and do not change their shape and texture throughout life. another positive point for using this type of technology is the low cost and the high quality of fingerprint capture sensors, compared to other biometric sensors. lines or ridges form fingerprints, and the spaces between these lines are called valleys. based on features found in the flow of these lines, most automatic fingerprint identification systems (afis) can distinguish two fingerprints. lately, fingerprint inclusion and the massive use of technological devices by governments and private institutions have generated afis with large databases registered with many individuals. in this context, a simple identification query to a system with millions of people can take a completely undesirable time. in this context, afis applications usually employ indexing mechanisms when identifying individuals by searching in a subset of the database instead of searching in the entire registered database. fingerprint indexing can be done using different fingerprint features, such as singular points [25], minutiae [7], and texture [12]. in the latter case, some works have used convolutional neural networks (cnns) to extract features from fingerprints in recent years. generally, indexing mechanisms consist of two steps: indexing and querying. when indexing, the system extracts different kinds of features from fingerprints and stores them according to the method. in the query step, according to a queried fingerprint, the collection of features is generated so that a search is made based on similarity to return the k fingerprints most similar to the query, where k is a previously defined number [8]. in this work, we used approximate nearest neighbor search algorithms (ann) as an indexing and search technique based on fingerprints’ features extracted using a cnn architecture, resnet18. we used ann to calculate the k best candidates (i.e., the k most similar) fingerprints. we conducted experiments to evaluate the performance of the proposed solution using fvc (fingerprint verification competition) databases and four ann methods (annoy [2], hnsw [18], ngt [11], and nanoflann [4]). our main goal was to study how different parameters influence the system’s performance, which is evaluated with time and hit rate metrics. this work is organized as follows: in sect. 2, the database used for searches with the presented method was shown. in sect. 3, concepts to understand this work were described; they are indexing, evaluation, and performance metrics. in sect. 4, the related works were presented. in sect. 5, the method presented in this work was described. in sect. 6, we have the preparations for the performance evaluation tests. in sect. 7, the results were presented. furthermore, finally, in sects. 8 and 9, we have the conclusion and acknowledgments, respectively. 2 fvc databases the fvc 2000, 2002 e 2004 datasets, described in [21], has a total of 10,560 fingerprint samples divided into different databases (db1, db2, db3, db4). each db has 110 different fingers, and each finger 8 different finger captures, each capture representing a different image of the same finger. another important piece of information is that within the same year, the same 110 fingers were captured between the different dbs, making it possible to compare the same finger with different equipment, just comparing the same finger id with different dbs of the same finger same year. there is an internal subdivision for one of the dbs, where for the 110 sets of samples, those numbered from 1 to 100 correspond to dba, and those numbered from 101 to 110 are part of dbb. researchers commonly use such division as the split into training and test sets, where dba is used for training and dbb is used for testing, allowing the results to be compared fairly. table  1 presents detailed information about the fvc dataset, such as image dimensions, sensor type for each db in each year. as we can see in the table, there are optical, capacitive, and thermal sensors, and each type of sensor model results in images with different dimensions. db4 databases were synthetically generated by the tools described in the “sensor type" column. figures 1, 2 and 3 shows one sample from each database described in the table 1. table 1. description of fvc databases with the list of sensors.full size table fig. 1. fvc 2000: (a) 2000_db1; (b) 2000_db2; (c) 2000_db3; (d) 2000_db4. full size image fig. 2. fvc 2002: (a) 2002_db1; (b) 2002_db2; (c) 2002_db3; (d) 2002_db4. full size image fig. 3. fvc 2004: (a) 2004_db1; (b) 2004_db2; (c) 2004_db3; (d) 2004_db4. full size image 3 indexing, evaluation, and performance metrics the construction of indexes in the fingerprint storage can follow different strategies, among the main ones based on minutiae, singular points, and, more recently, deep features. the indexing process consists of two main steps, the indexing of features and the recovery strategy. when indexing a database of fingerprints, the objective is to optimize searches in large collections of registered individuals, making the identification process less complex, reducing the 1:1 verification needed until finding the searched fingerprint, thus saving considerable time to get an answer. the set of features consists of a collection of discriminating characteristics extracted by the strategy adopted to differentiate and store the fingerprints in the representation of the index. the recovery step consists of recovering a considerably smaller subset of previously registered fingerprints using some similarity function. such a smaller subset forms the list of candidate fingerprints, where it is expected that among these is the sought fingerprint. according to [8] the challenges in indexing fingerprints are: features with high dimensions, features of varying type, unorderable indexes, no pattern in catches of the fingerprints, especially concerning lighting and occlusion, images used in the search may be of low quality or be rotated, and finally the interoperability between capture devices as can be seen in the description of the fvc datasets. there are different metrics used to evaluate indexing techniques, among which we can mention the hit rate, the penetration rate, and the search time. defined in the eq. 1 the hit rate is the percentage of genuine matches between the queried fingerprint and the set k returned by the recovery method concerning all queries. the penetration rate is how much of the search set equals the value of k, and usually, this rate varies between \(1\%\) and \(10\%\) [13, 17, 22]. moreover, the search time is equivalent to the total execution time of the respective approximate method responsible for identifying all images from the test set. $$\begin{aligned} hit\_rate = \frac{hits}{queries}\times 100\% \end{aligned}$$ (1) 4 related works in [6], the authors proposed an indexing mechanism based on cnns, more precisely using inception v3 [23]. in the proposed mechanism, fingerprints are aligned in a unified coordinate system, and a dictionary of maps is created from a base of 440 thousand fingerprints. the cnn is used to extract fixed-length features to be indexed and uses consensus similarity metric when querying. tests were performed using nist’s sd4 and sd14 datasets and demonstrated better performance than state of the art published so far. an indexing solution was proposed in [22], which is composed of three different cnns. being the first network called fingernet [24] to enhance the fingerprints and extract minutiae, with the minutiae divided by locations in smaller blocks. with another cnn, features are extracted based on texture and minutiae, called minutia-centred deep convolutional (mdc). finally, the last cnn aggregates the features of the descriptors in the representation end name amdc or aggregated lcd. the similarity between the indexed features is calculated directly via amdc, where only a normalization and scalar product between two amdc features is used. the experiments were carried out with fvc2000 db2, fvc2000 db3, nist sd4, nist sd4 natural, and nist sd14, preventing the method’s superiority in several studies. however, there were no comparisons with techniques based on cnn. some works use variations of delaunay triangulation for indexing. in [14] this is used with the k-means clustering algorithm for search, performing experiments on the fvc2000, fvc2004, fvc2004, nist sd4, and nist sd14 datasets. in [13] an extended triangulation is used and partitioning the triangles into groups, presenting results on the fvc2002 and fvc2004 databases. improving on the results of these papers, in [17] a new index vector is proposed whose values are spread in index space, with experiments using the datasets fvc2002 and fvc2004, outperforming the state-of-the-art. the authors of [1] studied fingerprint indexing using information from the fingerprint pores, which requires high-quality images and sensors with a lot of capture capacity. complete and partial fingerprint pore information is extracted and clustered using k-means. the recovery of fingerprints is made possible by searching for the nearest cluster based on the pores detected in the query image. the results are satisfactory compared to other techniques based on minutiae. 5 proposed indexing method the objective of this work is to evaluate a solution for fingerprint indexing. when performing tests with the fvc 2000, 2002, and 2004 databases, it is possible to evaluate results for different types of sensors. in the following subsections, both the solution and the techniques used in its composition will be detailed. 5.1 architecture the system to be evaluated consists of two different stages, the fingerprint indexing stage and the indexed fingerprint query stage, like the other indexing systems. it is considered that the input images were captured with fingerprint sensors and preprocessed so that they are used both for indexing and for queries. fig. 4. indexing system architecture full size image as can be seen in fig. 4 in the indexing process, the fingerprint is given as an entry in a cnn so that the features can be extracted so that they are stored in an indexing base. the result of this step is an indexing database duly completed and ready to carry out consultations. the networks are trained with different fingerprints than those used in the production environment and evidently need to be trained in a stage before using the approach. the second and complementary step of the solution is to perform searches with fingerprints so that the subset of fingerprints is considerably smaller than the registered base is returned so that the task of fingerprint identification is more efficient. each fingerprint consulted is extracted from the embeddings with cnn, and then searches are made using approximate ann algorithms to define the fingerprint k most similar to the query. 5.2 deep metric learning the generation of embeddings, that is, a set of features is carried out through a convolutional neural network (convolutional neural network, cnn). the chosen cnn is the resnet18 because it is an architecture with a good trade-off between performance and computational cost. the embeddings are extracted from the neck of the last completely connected layers, with a dimensionality ranging between 64, 96, and 128. the resnet18 is a residual neural network, as shown in fig. 5. altogether the neural network has 11511784 trainable parameters, requiring a total of 2 gflops for activation. the residual layers, obtained through shortcuts, facilitate the convergence of training and improve the accuracy [10]. fig. 5. resnet18 architecture. full size image the implementation of resnet18 present in the machine learning dlib c++ library was used [15]. the loss function used is coupled loss (cl), which is the dlib default, defined in the eq. 2 [26]. the cl is defined in terms of a global distance threshold d and also by a margin \( \epsilon \) added to decide when two embeddings n-dimensional, \( f (x_1) \) and \( f ( x_2) \), of two fingerprints, belong to the same identification. therefore, if the distance between the embeddings is greater than d, then they belong to different fingerprints. $$\begin{aligned} || f(x_1) f(x_2)|| < d \end{aligned}$$ (2) the training is carried out with n elements of an image base forming the load tensor, with a dimensionality of 224\(\times \)224 for the entry of cnns, resizing the images to this dimension. when using backpropagation, a neural network can only learn from cases where the condition imposed by the loss function is violated. therefore, example mining similar to [3] is used at the load loss computation level, so the training procedure benefits from difficult negatives, which typically improves quality and convergence. however, difficult negatives are also combined with random specimens, which, unlike [16], also take into account easy negatives. 5.3 approximate nearest neighbors the principle behind approximate nearest neighbors (ann) is to get a significant improvement in search performance, which comes at the cost of a loss in accuracy, so an approximately correct but fast result is obtained [2, 19]. this means that ann methods can be used for various tasks, such as classification, labeling, and recommendation. in addition, there is evidence in the literature that, for some cases, ann can also impact an unexpected improvement in results [5, 9]. three state-of-the-art ann methods have been chosen that are widely used in both research and commercial products: annoy, hnsw and ngt. in addition to these, nanoflann was also used. annoy (approximate nearest neighbors oh yeah) [2] was developed by erik bernhardsson and is used by spotify to recommend songs. its implementation is summarized in a binary space partition (bsp) tree using hyperplanes, similar to those used to render old 3d graphics. a random forest of bsp trees is built to improve search results: the more trees, the greater the accuracy of the results and the less “performance (time) for higher accuracy (quality)". standard configurations use \( 2 * d \), where d is dimensionality. hnsw (hierarchical navigable small world graphs) [18] is a graph-based method that separates links between nodes in a search graph according to their size scales in different layers, therefore performing searches in a multilayer graph. consequently, the estimate of the nearest node occurs only in a necessary fixed portion of the connections for each element. the main advantages of the hnsw method are its robustness and its support for continuous incremental indexing. ngt (neighborhood graph and tree for indexing highdimensional data) is a graph-based ann method initially proposed to work on a large scale, high-dimensional vector space [11]. ngt improves the graph of k-nearest neighbors (knng) by adjusting the path and degree of edges derived from a knng, that is, optimization in terms of the number of edges entering and leaving influencing the accuracy and search time, respectively. in particular, the authors describe how an acceleration in performance can be achieved by removing edges that can be replaced by alternative search paths. ngt also supports adding and removing items from the index, which is an important feature for real-world applications. experimental evaluations show that ngt can compete or even surpass other methods [2, 11]. blanck and kumar [4] propose nanoflann, based on flann [19, 20], one of the first libraries to provide ann methods. this includes several algorithms to search for the nearest neighbor in metric spaces, such as kd-trees variants. the nanoflann method was proposed for the construction of kd-trees from databases describing point clouds and rotation groups. however, nanoflann does not focus on searches for approximately closer neighbors but on the exact performance of the closest neighbor based on optimizations, such as adopting quadratic distances and customizable memory access policies. 6 experimental evaluation the experiments were carried out on a computer with linux operating system (ubuntu 20.04.2 lts), equipped with an intel(r) core(tm) i7-9750h cpu @ 2.60 ghz\(\times \)12, 16 gb ram memory, a 512 gb ssd, and nvidia geforce rtx 1660ti graphics card. for the generation of embeddings, resnet18 was trained using a synthetic dataset, with 100 identities with ten fingerprint images each, totaling 1000 images. this decision was made to verify the efficiency of generalizing the generation of embeddings to other datasets without the need for new training. the fingerprints were generated using a proprietary tool of a project of which the authors of this paper are membersfootnote 1. figure 6 shows an example of a fingerprint from the synthetic dataset. fig. 6. examples of fingerprints from our synthetic dataset. full size image regarding the generation of embeddings for search with ann models, tests were performed using all fvc bds. first, the bases were separated into search and test sets. the search set corresponds to the sample in which the anns models will consult and calculate the nearest neighbors in order to identify each of the elements of the test set. for comparison with related works, the dba of the fvc dbs corresponded to the search set, and the dbb was used for the test set. for each of the search and test sets, three sets of embeddings were generated, with their respective sizes of 64, 96, and 128. finally, the anns models were tested in each of the search and test sets in order to verify which combinations of these models and embedding vector size yield better hit rates. as for the penetration rate, also in order to compare to related works, a minimum of 1% and a maximum of 10% were used. in this work, the penetration rate is directly related to the k value of nearest neighbors used by the ann models, that is, a penetration rate of 1% indicates that a value equivalent to this rate was used as k. so, as the search set has \(100*8=800\) elements, the value corresponds to k with 1% penetration is 8, and the value corresponds to 10% is 80. search time was also a metric calculated in the experiments in order to verify the efficiency of the approach in generating good results quickly. it equals the sum of all searches performed in the test set. 7 results first, the results will be presented regarding the training of resnet18 to generate deep feature embeddings using the synthetic image dataset. figure 7 shows the evolution of the curve of mean loss values concerning the cnn training. fig. 7. evolution of average loss for resnet18 training over our synthetic dataset. full size image an early stopping strategy was used for stopping training in order to avoid overfitting. therefore, when stagnation in the loss curve evolution was detected for many epochs, training was manually interrupted. we validated this strategy empirically using increasing volume cuts from the training base. it was found that, in fact, this situation led to a model of poor performance. the results observed from queries based on the deep feature embedding vectors generated by our best model provide evidence that the stopping criterion was well applied. table 2 presents the best results obtained in queries with the ann methods in each of the fvc image databases. the best results were considered in relation to the hit rate of \(1\%\) penetration to \(10\%\). that is, whoever obtained better results from \(1\%\) penetration is ahead, as the idea is to use the least value of nearest neighbors k for searches. in case of a tie, the combination of the ann model with the smaller embeddings vector was highlighted since the larger the embeddings vector, the more computational resources are allocated to generate it. if the tie persists, the following criterion was the search time for each of the penetration rates. in all cases, the ngt method was superior in relation to the other anns, regarding the hit rate, therefore, all the results of table 2 were obtained with the ngt method. table 2. results of the best combinations of embeddings vector size for each of fvc ’dbs.full size table generally, the larger the vector of embeddings, the better the results found by searches of the anns methods, but the longer the search time tends to be. search time in each search performed presented values so low that the difference between these can be considered disregarded. complementing the previous data, table 3 presents the best results obtained by each of the ann models. table 3. best results obtained by each of the anns models.full size table interestingly, the best results for each of the ann models were all relative to searches over fvc’s 2004 db3. there is evidence that this dataset has the most uncomplicated images to identify by the proposed method. as much as the ngt method was superior, it is possible to notice that the other indexing methods do not fall so far behind in terms of accuracy because they still present very competitive results. the results obtained in this paper were compared with previous results from the literature. for doing so, evaluation was carried out using the same image bases used in the related works. it is worthy of mentioning that all the works considered used the same search and test set separation scheme. the comparisons were made considering the best results of each one for the queries in the respective image database. these comparative results are depicted by figs. 8, 9, 10 and 11. bear in mind that some works do not appear in all comparisons, as the authors did not present tests regarding the fingerprints database in their work. fig. 8. comparison with other indexing methods with fvc 2000_db2 dataset. full size image fig. 9. comparison with other indexing methods with fvc 2000_db3 dataset. full size image fig. 10. comparison with other indexing methods with fvc 2002_db1 dataset. full size image fig. 11. comparison with other indexing methods with fvc 2004_db1 dataset. full size image the search method presented in this work using ann models presented better results in most cases. unfortunately, none of the related works discussed in this paper present information on search times for comparison, which would probably enhance the proposal of this work even more. after all, the search times were negligible. 8 conclusion this work presented a method for fingerprint indexing based on deep embedding vector extraction using cnn and anns. we performed several experiments varying the embedding vector size and using four anns methods (annoy, ngt, hnsw, and nanoflann). the feature extraction cnn was trained using a synthetic dataset that was applied out-of-the-box to the evaluation datasets constructed with different sensors for real-world fingerprints (see table 1). experimental results indicate that the proposed method is competitive compared to other works found in the literature. on top of that, for several fvc databases, the method achieved 100% hit rate at a low penetration rate of 1%. it is important to highlight that the cnn used (resnet18) is simple and fast. this model can be leveraged to run the proposed method on resource-constrained devices, such as smartphones, as well as stated by the tiny search times. as for future work, it is intended to use other cnns to compare with the performance of resnet18. it is intended to use other databases for training cnns for the indexing stage and also for the query stage. finally, it is planned to test the solution on smartphones, at least the query step, in order to verify performance on mobile hardware. notes 1.the synthetic fingerprint base is available at the following link: https://github.com/leonardocosta21/fingerprint-synthetic-dataset. references anand, v., kanhangad, v.: pore-based indexing for fingerprints acquired using high-resolution sensors. pattern anal. appl. 23(1), 429–441 (2019). https://doi.org/10.1007/s10044-019-00805-3 article  google scholar  aumüller, m., bernhardsson, e., faithfull, a.: ann-benchmarks: a benchmarking tool for approximate nearest neighbor algorithms. in: beecks, c., borutta, f., kroger, p., seidl, t. (eds.) international conference on similarity search and applications, vol. 10609, pp. 34–49. springer, heidelberg (2017). https://doi.org/10.1007/978-3-319-68474-1_3 chapter  google scholar  balntas, v., riba, e., ponsa, d., mikolajczyk, k.: learning local feature descriptors with triplets and shallow convolutional neural networks. in: bmvc, vol. 1, pp. 1–11 (2016) google scholar  blanco, j.l., rai, p.k.: nanoflann: a c++ header-only fork of flann, a library for nearest neighbor (nn) wih kd-trees (2014) google scholar  boytsov, l., novak, d., malkov, y., nyberg, e.: off the beaten path: let’s replace term-based retrieval with k-nn search. in: proceedings of the 25th acm international on conference on information and knowledge management, pp. 1099–1108 (2016) google scholar  cao, k., jain, a.k.: fingerprint indexing and matching: an integrated approach. in: 2017 ieee international joint conference on biometrics (ijcb), pp. 437–445. ieee (2017) google scholar  galton, f.: finger prints macmillan (1892) google scholar  gupta, p., tiwari, k., arora, g.: fingerprint indexing schemes-a survey. neurocomputing 335, 352–365 (2019) article  google scholar  gysel, c.v., de rijke, m., kanoulas, e.: neural vector spaces for unsupervised information retrieval. acm trans. inf. syst. (tois) 36(4), 1–25 (2018) article  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  iwasaki, m., miyazaki, d.: optimization of indexing based on k-nearest neighbor graph for proximity search in high-dimensional data (2018). arxiv preprint arxiv:1810.07355 jain, a.k., prabhakar, s., hong, l., pankanti, s.: filterbank-based fingerprint matching. ieee trans. image process. 9(5), 846–859 (2000) article  google scholar  kavati, i., prasad, m.v.n.k., bhagvati, c.: hierarchical decomposition of extended triangulation for fingerprint indexing. in: efficient biometric indexing and retrieval techniques for large-scale systems. scs, pp. 21–40. springer, cham (2017). https://doi.org/10.1007/978-3-319-57660-2_2 chapter  google scholar  khodadoust, j., khodadoust, a.m.: fingerprint indexing based on expanded delaunay triangulation. expert syst. appl. 81, 251–267 (2017) article  google scholar  king, d.e.: dlib-ml: a machine learning toolkit. j. mach. learn. res. 10, 1755–1758 (2009) google scholar  kuma, r., weill, e., aghdasi, f., sriram, p.: vehicle re-identification: an efficient baseline using triplet embedding. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–9. ieee (2019) google scholar  lee, s., jeong, i.r.: improved fingerprint indexing based on extended triangulation. ieee access 9, 8471–8478 (2021) article  google scholar  malkov, y.a., yashunin, d.a.: efficient and robust approximate nearest neighbor search using hierarchical navigable small world graphs. ieee trans. pattern anal. mach. intell. 42, 824–836 (2018) article  google scholar  muja, m., lowe, d.g.: fast approximate nearest neighbors with automatic algorithm configuration. visapp (1) 2(331–340), 2 (2009) google scholar  muja, m., lowe, d.g.: scalable nearest neighbor algorithms for high dimensional data. ieee trans. pattern anal. mach. intell. 36(11), 2227–2240 (2014) article  google scholar  schuch, p., schulz, s., busch, c.: survey on the impact of fingerprint image enhancement. iet biometrics 7(2), 102–115 (2017) article  google scholar  song, d., tang, y., feng, j.: aggregating minutia-centred deep convolutional features for fingerprint indexing. pattern recogn. 88, 397–408 (2019) article  google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  tang, y., gao, f., feng, j., liu, y.: fingernet: an unified deep network for fingerprint minutiae extraction. in: 2017 ieee international joint conference on biometrics (ijcb), pp. 108–116. ieee (2017) google scholar  wang, s., zhang, w.w., wang, y.s.: fingerprint classification by directional fields. in: proceedings, fourth ieee international conference on multimodal interfaces, pp. 395–399. ieee (2002) google scholar  weinberger, k.q., saul, l.k.: distance metric learning for large margin nearest neighbor classification. j. mach. learn. res. 10(2), 207–244 (2009) math  google scholar  download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. the authors would like to thank the ceará state foundation for the support of scientific and technological development (funcap) for the financial support (6945087/2019). author information authors and affiliations federal university of ceara, fortaleza, brazil leonardo f. da costa, lucas s. fernandes, joão p. b. andrade, paulo a. l. rego & josé g. r. maia authors leonardo f. da costaview author publications search author on:pubmed google scholar lucas s. fernandesview author publications search author on:pubmed google scholar joão p. b. andradeview author publications search author on:pubmed google scholar paulo a. l. regoview author publications search author on:pubmed google scholar josé g. r. maiaview author publications search author on:pubmed google scholar corresponding author correspondence to leonardo f. da costa . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper da costa, l.f., fernandes, l.s., andrade, j.p.b., rego, p.a.l., maia, j.g.r. (2021). deep convolutional features for fingerprint indexing. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_16 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords fingerprint indexing deep feature embedding cnn ann models publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a clustering validation index based on semantic description | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a clustering validation index based on semantic description conference paper first online: 12 october 2023 pp 315–328 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a clustering validation index based on semantic description download book pdf download book epub roberto douglas guimarães de aquino  orcid: orcid.org/0000-0002-8486-83549,10, vitor venceslau curtis  orcid: orcid.org/0000-0003-4462-39079 & filipe alves neto verri  orcid: orcid.org/0000-0002-8240-51299  part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 507 accesses 1 citation abstract in clustering problems where the objective is not based on specifically spatial proximity, but rather on feature patterns and the semantic description, traditional internal cluster validation indices might not be appropriate. this article proposes a novel validity index to suggest the most appropriate number of clusters based on a semantic description of categorical databases. to assess our index, we also propose a synthetic data generator specifically designed for this type of application. we tested data sets with different configurations to assess the performance of the proposed index compared to well-known indices in the literature. thus, we demonstrate that the index has great potential for discovering the number of clusters for the type of application studied and the data generator is able to produce relevant data sets for the internal validation process. supported by coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an unsupervised and robust validity index for clustering analysis article 19 october 2018 an approach to validity indices for clustering techniques in big data article 05 october 2017 stability of mixed-type cluster partitions for determination of the number of clusters chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining and knowledge discovery data mining ontology functional clustering reactivity of clusters 1 introduction clustering is a partitioning technique that aims to group data points so that points within a group have similar characteristics (or are close to each other) while points in other groups are different [11]. in this type of task, defining the most suitable technique for the type of data and the correct number of clusters can be a challenge, especially if little information about the data is known [12]. despite the existence of several approaches to solve this problem, there is no optimal method, especially when considering categorical data. on this type of data the distance-based techniques may not be suitable [6, 8]. in order to discover the adequate number of clusters, cluster validity indices (cvi) [2] are generally used, which can have different approaches, such as distance between points [3, 4, 17], density [16], entropy [7], comparison between different indices [20], among others [18]. thus, depending on the configuration of the data points and the purpose of the clustering certain indices may be more suitable. therefore, in scenarios where the objective is not based on specifically clustering nearby data points, but rather on obtaining clusters based on feature patterns or explaining them, using a cvi developed for this purpose may be more appropriate. for example, consider a human resources data set in which each data point represents a person described by categorical features related to their professional characteristics, such as education, leadership, teamwork, creativity, etc. given that a decision maker needs to identify the profiles present in this data set in order to select people for a given task, how could he know the number of potential professional profiles and their respective descriptions? in this type of scenario, given the purpose of clustering and the type of features that describe the data, using a distance-based index may not be the best choice. an example of application of this type of scenario can be observed in the data sets of the competence system of the department of aerospace science and technology (dcta)footnote 1 of the brazilian air force called hippocampus. this system is restricted and stores competence levels in several areas – such as public administration, research, informatics, among others – of all dcta’s employees. dcta’s people management sector uses the stored information to monitor the development of the skills of the workforce. thus, this article aims to propose an index that suggests the most appropriate number of clusters based on the semantic description potential, for applications in categorical data sets in binary form. although the proposal is initially focused on a specific type of data, the index can also be applied to any type of data set, as long as the features are transformed into the categorical type [21]. in addition to the potential application in different types of data sets, the index can also be used to compare the quality of different partitions; it is independent of distance metrics and has significant performance when applied to competency-based data sets, where data points have features (competences) common to each other. 2 related works in this section, we discuss various popular cvis using different approaches to determine how good a particular clustering is. this variety, we think, ensures we have a solid set of cvis to compare our proposed method against (for the actual comparison, see sect. 6). average silhouette width (asw) [17] the silhouette width index scores how well a particular data point is clustered based on cohesion (the average within-cluster distance) and separation (how far clusters are from each other). let \(x_i\) be a data point in cluster \(s_l\). then, we can calculate the cohesion of \(x_i\) with $$\begin{aligned} a(x_i) = \frac{1}{n_{s_l}} \sum _{x_j \in s_l, i\ne j} d(x_i,x_j), \end{aligned}$$ and its cluster separation with $$\begin{aligned} b(x_i) = \min _{s_t \in s, t\ne l} \biggl \{\frac{1}{n_{s_t}}\sum _{x_j \in s_t} d(x_i, x_j)\biggr \}. \end{aligned}$$ thus, the asw of a data set x under a partition s is given by $$\begin{aligned} \text {asw}(x,s) = \frac{1}{n}\sum _{x_i \in x}\frac{b(x_i) a(x_i)}{\max \left\{ a(x_i), b(x_i) \right\} }. \end{aligned}$$ notice that \(-1 \le asw(x,s) \le 1\), where a higher value indicates a better partition in terms of cohesion and separation. the literature certainly indicates there is no index that is the best in all cases, but the asw has proven to be particularly successful (for a comprehensive comparison, see [2]). calinski-harabasz (ch) [3] the calinski-harabasz index is also rather popular and very much like asw it can be easily found in many software packages used in data analysis, including matlab [10], scikit-learn [14], and r [15]. ch assesses the quality of a particular partition based on the ratio of the between-cluster-means and within-cluster sum of squares. given a data set x and clustering s, we have that $$\begin{aligned} ch(s) = \frac{(t-w_k)/(k-1)}{w_k/(n-k)}, \end{aligned}$$ where t is the data scatter \(t=\sum _{i=1}^n \sum _{v=1}^m (x_{iv} \bar{x}_v)^2\), and \(\bar{x}_v\) is the mean of feature v over all \(x_i \in x\). the higher this index value is, the better the clustering is. davies-bouldin (db) [4] the davies-bouldin index scores a clustering s by computing the cohesion of each cluster \(s_l \in s\) as the average distance from each \(x_i \in s_l\) and \(z_l\) (the centroid of \(s_l\)), and taking into account the pairwise distance between all centroids \(z_l \in z\). that is, one first calculates the cohesion for each cluster \(s_l \in s\) as $$\begin{aligned} c(s_l) = \frac{1}{n_{s_l}} \sum _{x_i \in s_l} d(x_i, z_l), \end{aligned}$$ and then the value of this index is given by $$\begin{aligned} \text {db}(s) = \frac{1}{k} \sum _{s_l \in s} \max _{s_t \in s, l\ne t} \biggr \{ \frac{c(s_l) + c(s_t)}{d(z_l, z_t)} \biggr \}. \end{aligned}$$ unlike asw and ch, a low value of db indicates a good clustering. contiguous density region (cdr) [16] the index is based on the uniformity measure of the clusters, obtained from the spatial pattern of the data points and the local density. initially, we define the local density of a data point \(x_i\) in a set of data points x as the distance to the nearest neighbor \(x_j\), which belongs to the same cluster by the equation $$\begin{aligned} \delta (x_i) = \min _{x_j \in s_l, i \ne j} \left\{ d(x_i,x_j) \right\} \text {.} \end{aligned}$$ the average density of a cluster \(s_l\) is calculated as the average of all the local densities of the cluster, defined as $$\begin{aligned} \bar{\delta }(s_l) = \frac{1}{n_{s_l}}\sum _{x_i \in s_l} \delta (x_i) \text {.} \end{aligned}$$ thus, the uniformity of each cluster is calculated as the difference between the local density of the cluster and the average of all the local densities of the cluster. it measures the degree of local density variation within a cluster. given a cluster, its uniformity is defined by $$\begin{aligned} \upsilon (s_l) = \frac{1}{\bar{\delta }(s_l)} \sum _{x_i \in s_l} |\delta (x_i) \bar{\delta }(s_l)| \text {,} \end{aligned}$$ for values of \(n_{s_l}\) greater than 1, otherwise a null uniformity value is assigned. given a partition s, the value of the cdr index is defined as $$\begin{aligned} cdr(s) = \frac{1}{n} \sum _{s_l \in s}n_{s_l} \upsilon (s_l) \text {,} \end{aligned}$$ where n corresponds to the total number of data points in x. the cdr index minimization estimates the optimal data partition among all partitions generated by the clustering algorithm based on the structural characteristics of the data points. cubage [7] the cubage index (clustering utility based on the averaged information gain of isolating each cluster) uses the proposed averaged information gain of isolating each cluster (age) to measure the separation, and the reciprocal entropy of the set of data points conditioned on the partition to measure the compactness of the clusters. given that the data set x is described by a set of m independent features \(v = \left\{ v_1, \dots , v_m\right\} \) and the value of the feature \(v_j\) can only be taken from the domain \(d(v_j) = \left\{ v_j^1, \dots , v_j^{\eta _j} \right\} \) where \(\eta _j\) is the number of possible values of the respective feature. thus, starting from the concept of entropy, we initially calculate the information gain referring to the set of features h(v) through the equation $$\begin{aligned} h(v) = \sum _{j}^{m} \sum _{i}^{\eta _j} p(\alpha _j^{i}) \, \text {log}(p(\alpha _j^{i})) \text {,} \end{aligned}$$ where \(p(\alpha _j^{i})\) is the probability of feature \(v_j\) taking the value \(\alpha _j^{i}\). given a partition s, the entropy of v conditioned on s, i.e., h(v|s), and considered as the whole entropy of the partition is defined as $$\begin{aligned} e(s) = h(v|s) = \sum _{l}^{k} \sum _{j}^{m} \sum _{i}^{\eta _j}p(\alpha _j^{i}, s_l) \, \text {log}(p(\alpha _j^{i}| s_l)) \text {,} \end{aligned}$$ where \(p(\alpha _j^i, s_l)\) is the conditional probability of the value \(\alpha _j^i\), given cluster \(s_l\). finally, the information gain between variables and partitions is related through the equation $$\begin{aligned} cubage(s) = \frac{h(v) e(s)}{e(s)} \text {.} \end{aligned}$$ the index takes the form of the product of separation and compactness. the largest index value across partitions indicates the most suitable number of clusters. consensus index (ci) [20] this index aims to calculate the average pairwise similarity between different partitions. these partitions can be obtained by running different algorithms. thus, suppose a set of w solutions of different clustering algorithms \(s= \left\{ s^1, \dots , s^w \right\} \). the consensus index between the different partitions is given by $$\begin{aligned} ci (s) = \sum _{p<q} \varphi (s^p, s^q) \text {,} \end{aligned}$$ where \(\varphi \) is validation index based in similarity. in this study, we use the adjusted rand index [9], also used in a similar study [19]. hence, higher index values indicate higher quality in the partitions resulting from clustering. cluster number assisted k-means (cnak) [18] the cnak algorithm is defined as a variant of k-means in which the proper cluster number is learned during the clustering process. the proposal is based on randomly sampled large-sized sets of data points having the same distribution as the original set, such that the number of generated cluster centroids from the sample is approximately the same as the original. according to the results of the study, the algorithm presented satisfactory results, being able to successfully detect clusters, identify the hierarchy, but was not able to demonstrate robustness in dealing with noise and data overlap. in this study, we used the \(\tilde{k}\) suggested by cnak and compared it with other indices for the process of validating the results of the proposed method. 3 validation based on the semantic description of the clusters given that validation indices with distance-based metrics can be a problem for applications on categorical data sets, since the spatial representation of data points is not adequate [6, 8], we propose an index, based on the potential for semantic description of each cluster, for internal validation of clustering results. for this, given the result of the clustering, frequent patterns within the clusters are identified and we calculate some indicators that are related to the semantic description potential of the set of frequent patterns. thus, we consider an adequate clustering to be one in which the resulting clusters are adequately described, since the points within each of them were selected based on the similarity between the observed frequent patterns. 3.1 proposed index let \(v = \left\{ v_1, \dots , v_m\right\} \) be a set of m features (item set) each representing a description of a categorical data set (transactional data set) \(x = \left\{ x_1, \dots , x_n\right\} \) of n data points, where each point \(x_i \in \mathbb {z}_{2}^{m}\) represents the competency description of a person (transaction) from a subset of v items. the support \(\sigma _{v_j}\) (or frequency of occurrence) of a \(v_j\) item is defined as the proportion of transactions that contain \(v_j\). even if \(\sigma _{v_j}\) is greater than a minimum support limit \(\sigma _{min}\), we define \(v_j\) as frequent pattern [1]. thus, through the partition \(s=\{s_1, \ldots , s_k\}\) containing k clusters, obtained by a clustering algorithm, we define \(f_1(s_l)\) as the collection of all \(n_l \ge 0\) frequent patterns with length 1 contained in \(s_l\), each with a support value \(\sigma _l\) assigned, which represent the semantic description of the respective cluster. after that we evaluate the potential of the sets of items to perform the semantic description of the clusters. for this, we propose the support, length, exclusivity and difference for group evaluation (sledge) index, where the support values found, set size, exclusivity and support variability between items are evaluated, respectively. thus, each of the indicators that are part of the index are presented in the subsections that follow. support. the indicator \(s_{f_1(s_l)}\) is calculated from the average of support values \(\sigma _{l}\) of the collection of frequent patterns belonging to \(f_1(s_l)\) by the equation $$\begin{aligned} s_{f_1(s_l)} = \frac{1}{n_l} \sum _{\sigma _{l} \in f_1(s_l)} \sigma _{l} \end{aligned}$$ (1) when \(n_l \ge 1\), otherwise the indicator is equal to zero. we suggest that clusters described with frequent patterns of greater support between items are better, as this way we can infer more accurate information about the data points belonging to the cluster. length. to calculate this indicator we initially get the average cardinality based on the number of frequent patterns in all partitions by the equation $$\begin{aligned} \bar{n}_{f_1(s)} = \frac{1}{k} \sum _{l}^{k} n_l \text {,} \end{aligned}$$ where k represents the number of clusters. then, we obtain the indicator through the equation $$\begin{aligned} l_{f_1(s_l)} = \frac{1}{1+\left| n_l \bar{n}_{f_1(s)} \right| } \text {.} \end{aligned}$$ (2) we suggest that clusters are best described when they have an average cardinality of frequent patterns, given that either the presence of a large or small number of these should be insufficient to accurately define the semantic description of the cluster. exclusivity. the \(e_{f_1(s_l)}\) indicator is related to the number of exclusive frequent patterns within the cluster and is calculated by the function $$\begin{aligned} e_{f_1(s_l)} = 1 \frac{\hat{e}_l}{n_l}\text {,} \end{aligned}$$ (3) where \(\hat{e}_l\) is the number of non-exclusive frequent patterns of the cluster compared to others \(k-1\) clusters g such that \(g \ne j\). we suggest that clusters with exclusive frequent patterns are better defined in relation to the others that have more common patterns. difference. this indicator assess the maximum variability among the frequent patterns that describe each cluster. if there is more than 1 pattern, the indicator is calculated by taking the maximum difference \(\delta _{max}\) between the support values of all frequent patterns belonging to \(f_1(s_l)\). if there is only 1 pattern, the value of the indicator will be its own support value and if there are no patterns, the value will be null, as described by the function such $$\begin{aligned} d_{f_1(s_l)} = {\left\{ \begin{array}{ll} \sigma _{l} &{} \text { if } n_l = 1\text {,} \\ \sqrt{\delta _{max} \left\{ f_1(s_l) \right\} } &{} \text { if } n_l > 1\text {,}\\ 0 &{} \text { otherwise.} \end{array}\right. } \end{aligned}$$ (4) as the variation is small between the values of the supports, we decided to increase the influence by the square root. we assume that greater variation between support values within the cluster is better for a more accurate semantic description. given that there are less representative frequent patterns within the cluster, we expect that the difference in proportion of these in relation to the largest values are as large as possible. sledge index. the index of each of the j clusters is calculated from the median of the indicators obtained by the eqs. (1) to (4) and the global index value (sledge) is the average of the cluster indices. the index evaluates the quality of each grouping and of the grouping in general based on the semantic description potential given from the identified frequent patterns. the higher the index value, the better the clustering quality. the index can be easily obtained from a specific function (sledge_score), which is part of a package (sledge) built in python and available in a repository on githubfootnote 2. 4 synthetic data consider the following scenario. each data sample represents a person. each feature indicates whether such a person has a certain skill. we assume that people can be grouped in clusters such that: the presence/absence of a certain few features (called dependent features) are correlated; and other features may be present/absent independently. dependent features vary among clusters. the rationale is that, for a particular group of people, some features are more important than others. these important features appear more frequent than other features in the group. moreover, they are positively or negatively correlated. for example, it professionals usually share common skills: programming, data base skills, etc. on the other hand, the presence of some pairs of skills may be rare. for example, it professionals who program rarely use the same programming language or development environment. thus, let \(x_{ij}\) be the indicator whether the i-th person, belonging to cluster c(i), has the j-th feature. we want to generate a dataset such that $$\begin{aligned} p(x_{ij} = 1) \approx {\left\{ \begin{array}{ll} \lambda &{} \text { if } j \in \mathcal {f}_{c(i)} \\ \epsilon &{} \text { otherwise,} \end{array}\right. } \end{aligned}$$ (5) where \(0 < \epsilon < \lambda < 1\) and \(\mathcal {f}_{c(i)}\) is the set of dependents features of the cluster c(i). we also want that the following statements to hold: $$\begin{aligned} p(x_{ij}, x_{ij'}) = p(x_{ij})p(x_{ij'}) \forall j \not \in \mathcal {f}_{c(i)}\text {,} \end{aligned}$$ (6) $$\begin{aligned} p(x_{ij}) = p(x_{i'j}) \text { if } c(i) = c(i')\text {, and } \end{aligned}$$ (7) $$\begin{aligned} p(x_{ij}, x_{ij'}) \ne p(x_{ij})p(x_{ij'}) \forall j, j' \in \mathcal {f}_{c(i)}\text {.} \end{aligned}$$ (8) to generate data with such properties, each cluster c in the dataset is associated with an m by m matrix \(w^c\) whose elements are $$\begin{aligned} w^c_{pq} \sim \mathcal {n}(0, 1)\text {,} \end{aligned}$$ (9) and each sample \(\textbf{x}_i = (x_{i1}, x_{i2}, \dots , x_{im})\) such that \(c(i) = c\) is associated with a vector m-dimensional \(\textbf{a}^i\) whose elements are $$\begin{aligned} a_p^i \sim \mathcal {n}(0, 1)\text {.} \end{aligned}$$ (10) then, let $$\begin{aligned} \textbf{b}^i = \frac{1}{\sqrt{m}} w^{c(i)} \times \textbf{a}^i\text {,} \end{aligned}$$ (11) then, we generate the dependent features (\(j \in \mathcal {f}^{c(i)}\)) $$\begin{aligned} x_{ij} = {\left\{ \begin{array}{ll} 1 &{} \text { if } \varphi \!\left( b^i_j\right) < \lambda \\ 0 &{} \text { otherwise,} \end{array}\right. } \end{aligned}$$ (12) where the \(\varphi \) is the cumulative distribution function of the standard normal distribution. the other features, \(j \not \in \mathcal {f}^{c(i)}\), are generated using $$\begin{aligned} x_{ij} = {\left\{ \begin{array}{ll} 1 &{} \text { if } u_{ij} \sim \mathcal {u}(0, 1) < \epsilon \\ 0 &{} \text { otherwise,} \end{array}\right. } \end{aligned}$$ (13) remark: note that the product xy of random variables \(x, y \sim \mathcal {n}(0, 1)\) is not a standard normal distribution. however, it is similar enough to use \(\varphi \) to satisfy eq. 5. 5 set of experiments in this section we do some experiments to evaluate the performance of the proposed index, defined in sect. 3, comparing with known indices described in sect. 2. for this, we use synthetic data generated using the categorical binary random data (catbird) function, described in sect. 4, which is part of the library called random data generator algorithm for clustering (rdga_4k), built in python and available in a github repositoryfootnote 3. for the generation of synthetic binary data, all possible combinations of the parameters listed in table 1 are used. in the end, we obtain 3600 unique data sets from combinations of 2 parameters of \(\epsilon \), 2 of \(\lambda \), 3 different numbers of both clusters and features, 2 types of data balance for each number of clusters and 50 seeds. the criteria for choosing the values of \(\epsilon \) and \(\lambda \) are based on the different scattering scenarios of data points that the combinations of these parameters can simulate. in this way, we choose more extreme scenarios and two intermediate ones. for the k values, we try to evaluate three intervals of the same distance between 2 and 10 in order not to overload the tests, but still have interesting results to analyze. for the number of seeds, we believe that 50 seeds are enough for the analysis. in addition, during the tests, we identified that with half of the seeds the results did not change significantly. table 1. parameters used in the generation of synthetic binary data. in total, 3600 unique data sets are generated, obtained from combinations of 2 parameters of \(\epsilon \), 2 of \(\lambda \), 3 different numbers of clusters, 3 of features, 2 types of data balance for each number of clusters and 50 seeds.full size table as clustering algorithms for the experiments, we use k-means and two hierarchical clusters with average (haca) and ward (hacw) linkages. although they are based on distance, they are considered suitable for use on binary data sets [5]. the algorithms are applied to each of the data sets for different values of k, starting at 2 and increasing in intervals from 1 to 10. thus, using the obtained labels we calculate each of the indices, except cnak, as it uses its own clustering algorithm (k-means++) to suggest the value of k. given that the ci index uses different labels in a consensus approach, we named kaw the algorithm responsible for calculating the index, given that it uses the three algorithms. 6 results and discussion in this section, we present and discuss some results obtained after the experiments performed, as described in sect. 5. as criteria to determine the quality of the indices, we used two approaches: one based on the proportion of times each index identifies the correct number of clusters (hit rate) and another on the mean relative error (mre), defined by the equation $$\begin{aligned} \text {mre} = \frac{|k-\tilde{k}|}{k} \text {,} \end{aligned}$$ (14) where k represents the correct number of clusters in the data set and \(\tilde{k}\) the predicted number of clusters. thus, we use the average of the mre to analyze each of the indices and calculate the standard deviation (std). to improve the visualization of the results, we highlight the best performances in bold. initially, analyzing the overall performance of the indices, presented by table 2, we can observe that sledge obtains the best performance, i.e. the smallest mre. we use the standard deviation (std) in order to verify how much the mre values deviate from the average. the hit rate presents the ratio of the number of times the mre is zero, i.e. the index correctly identifies the number of clusters. on both measures, sledge also performs best, identifying the correct number of clusters in 85% of the data sets and the lowest std among the rest. in second, we can observe that the asw obtains a very similar performance, even with a similar std, despite the proportion of hits being lower than expected. table 2. overall performances of each index, presented using the mre average, std and hit rate.full size table given the different configurations of dispersion of the data points of the variables \(\epsilon \) and \(\lambda \), we expect that in the configuration with \(\epsilon = 0.1\) and \(\lambda = 0.9\) the mre is smaller, since the clusters are more separated and the data points closer to each other. unlike the setting \(\epsilon = 0.2\) and \(\lambda = 0.8\), where the opposite scenario is observed. as described by saha and mukherjee (2021) [18], as the proportion of overlapping between data points increases and the separation between clusters decreases, the performance of the indices tends to decrease as well. despite varying the noise (\(\epsilon \)) and overlapping (\(\lambda \)) parameters to produce different dispersion configurations, these configurations are not significantly different to the point that sudden changes are verified within the same index. as for the results, according to table 3, we can see that the sledge index achieves the best performance, despite being just a little higher than the asw index. cnak, given that it is especially sensitive to overlapping and noise in the data sets [18], presented a much lower performance, but expected, given the presence of a large number of these characteristics in the test data sets. as for cubage [7], the index performs better in more compact and separate clusters, which can be observed when we analyze the different scenarios. table 3. average mre of each index given different \(\epsilon \) and \(\lambda \) settings.full size table as for the number of clusters tested, we identified that for most indices the mre tends to increase directly proportional to the number of k, except for the db index. some indices in particular, such as cnak, ch and asw for example, the smaller the number of clusters in the data set, the greater the chances of success of the indices [18]. thus, given table 4, we can observe that the sledge index had the best performance, followed by the asw, but that it decreases a little when the number of clusters increases. table 4. average mre of each index evaluated from different numbers of clusters.full size table in the studies we use as a reference, given the average performance of the classic cluster validation indices, when different algorithms are used to evaluate the experiments, it is observed that asw usually performs better, followed by ch and finally db, as shows table 2. the same is also observed for different degrees of data point dispersion. when the k-means algorithm is used, the asw remains the best evaluated, but the db usually presents a better result than the ch index, as demonstrated by studies by [5, 16, 18, 19]. thus, the indexes demonstrate to have influence on the type of clustering algorithm, except for the db index that has better performance when k-means is applied instead of hierarchical, as shown in table 5. table 5. average mre of each index evaluated from clustering algorithms.full size table in addition to verifying the performance of the indices for different values of k, we decided to do some tests in scenarios where the data points in the clusters are balanced and unbalanced. thus, for the same number of k, we generate both types of data sets. as we can see, the sledge and asw indices demonstrate significant and very similar performances, however, the sledge, given the proportions, continues to perform well in cases where the data is unbalanced. the results can be observed from the table 6. table 6. average mre of each index applied to balanced and unbalanced data sets.full size table at the end, for the statistical significance test, we select the best performing indices in each data set, given the different algorithms, we rank them (using the average rank in the tie) and apply the wilcoxon-mann-whitney test [13]. as a result, given a significance level of 0.05, we reject the null hypothesis that the performance of all indices has some similarity. 7 conclusion the objective of this article is to propose an index that suggests the most adequate number of clusters based on the potential of semantic description, for applications in categorical data sets, especially in clusters of profiles based on competences. for index validation, we also propose a synthetic data generator suitable for the application, with a similar gaussian distribution and apparently well-defined clusters, despite some overlapping points. as test experiments we use 3600 datasets with different configurations and clustered by different algorithms. finally, we compare the proposed index with other well-known indices in the literature and with different approaches regarding the prediction of the number of clusters. as for the results, the proposed index identifies with significant precision the correct number of clusters in relation to the others, although other indices present a similar performance. from the analysis of the results and comparing with other works, we identified that the behavior of some indices were the same of those works when applied in the synthetic data of the proposed generator. thus, we believe that the generator can produce adequate data for the internal cluster validation task. for future work we intend to test the index in real data sets for external validation and to develop a clustering algorithm to be applied in competence-based data sets that cluster data points in close profiles. we intend to apply our method in hippocampus to increase the capabilities of the dcta’s people management sector. notes 1.https://dcta.mil.br/. 2.https://github.com/verri/sledge. 3.https://github.com/aquinordg/rdga_4k. references agrawal, r., srikant, r., et al.: fast algorithms for mining association rules. in: proceedings 20th international conference on very large data bases, vldb. vol. 1215, santiago, chile, pp. 487–499 (1994) google scholar  arbelaitz, o., gurrutxaga, i., muguerza, j., pérez, j.m., perona, i.: an extensive comparative study of cluster validity indices. pattern recogn. 46(1), 243–256 (2013) article  google scholar  caliński, t., harabasz, j.: a dendrite method for cluster analysis. commun. stat.-theory methods 3(1), 1–27 (1974) article  mathscinet  math  google scholar  davies, d.l., bouldin, d.w.: a cluster separation measure. ieee trans. pattern anal. mach. intell. 2, 224–227 (1979) article  google scholar  dimitriadou, e., dolničar, s., weingessel, a.: an examination of indexes for determining the number of clusters in binary data sets. psychometrika 67(1), 137–159 (2002) article  mathscinet  math  google scholar  dorman, k.s., maitra, r.: an efficient k-modes algorithm for clustering categorical datasets. stat. anal. data mining asa data sci. j. 15(1), 83–97 (2022) article  mathscinet  google scholar  gao, x., yang, m.: understanding and enhancement of internal clustering validation indexes for categorical data. algorithms 11(11), 177 (2018) article  mathscinet  math  google scholar  guha, s., rastogi, r., shim, k.: rock: a robust clustering algorithm for categorical attributes. inf. syst. 25(5), 345–366 (2000) article  google scholar  hubert, l., arabie, p.: comparing partitions. j. classif. 2, 193–218 (1985) article  math  google scholar  inc., t.m.: matlab version: 9.13.0 (r2022b) (2022). https://www.mathworks.com jain, a.k., murty, m.n., flynn, p.j.: data clustering: a review. acm comput. surv. (csur) 31(3), 264–323 (1999) article  google scholar  liu, y., li, z., xiong, h., gao, x., wu, j.: understanding of internal clustering validation measures. in: 2010 ieee international conference on data mining, pp. 911–916. ieee (2010) google scholar  mann, h.b., whitney, d.r.: on a test of whether one of two random variables is stochastically larger than the other. ann. math. stat., 50–60 (1947) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  r core team: r: a language and environment for statistical computing (2021). https://www.r-project.org/ rojas-thomas, j.c., santos, m.: new internal clustering validation measure for contiguous arbitrary-shape clusters. int. j. intell. syst. 36(10), 5506–5529 (2021) article  google scholar  rousseeuw, p.j.: silhouettes: a graphical aid to the interpretation and validation of cluster analysis. j. comput. appl. math. 20, 53–65 (1987) article  math  google scholar  saha, j., mukherjee, j.: cnak: cluster number assisted k-means. pattern recogn. 110, 107625 (2021) article  google scholar  ünlü, r., xanthopoulos, p.: estimating the number of clusters in a dataset via consensus clustering. expert syst. appl. 125, 33–39 (2019) article  google scholar  vinh, n.x., epps, j.: a novel approach for automatic number of clusters detection in microarray data based on consensus clustering. in: 2009 ninth ieee international conference on bioinformatics and bioengineering, pp. 84–91. ieee (2009) google scholar  witten, i.h., frank, e., hall, m.a., pal, c.: data mining: practical machine learning tools and techniques. morgan kaufmann, burlington (2016) google scholar  download references author information authors and affiliations aeronautics institute of technology, sao jose dos campos, sp, brazil roberto douglas guimarães de aquino, vitor venceslau curtis & filipe alves neto verri federal university of sao paulo, sao jose dos campos, sp, brazil roberto douglas guimarães de aquino authors roberto douglas guimarães de aquinoview author publications search author on:pubmed google scholar vitor venceslau curtisview author publications search author on:pubmed google scholar filipe alves neto verriview author publications search author on:pubmed google scholar corresponding author correspondence to roberto douglas guimarães de aquino . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de aquino, r.d.g., curtis, v.v., verri, f.a.n. (2023). a clustering validation index based on semantic description. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_21 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords clustering internal validation categorical data recommendation systems based on competence publish with us policies and ethics profiles vitor venceslau curtis view author profile filipe alves neto verri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature gender-neutral english to portuguese machine translator: promoting inclusive language | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper gender-neutral english to portuguese machine translator: promoting inclusive language conference paper first online: 31 january 2025 pp 180–195 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) gender-neutral english to portuguese machine translator: promoting inclusive language download book pdf download book epub ricardo trainotti rabonato9, evangelos milios10 & lilian berton9  part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 227 accesses 1 citation abstract machine translation (mt) plays a crucial role in globalization, making access to information more inclusive, although challenges persist for less popular languages, like portuguese. one of the most complex challenges in the automatic translation into languages such as portuguese is the precise preservation of masculine and feminine grammatical gender. there are still situations where translation does not adequately reflect gender equality, often reinforcing societal stereotypes. we aim to explore approaches to ensure fairness in english to portuguese mt through post-processing techniques, which aim to apply some transformation to the model’s output. to this end, we used the marianmt model as our foundation, then we fine-tuned it using a dataset of english-portuguese sentences that was generated and carefully crafted to mitigate gender bias within the sentences. the results on gender disparities metrics, based on the winomt test set for mt such as \(\vardelta g\), \(\vardelta s\), and the overall accuracy (preserving the gender of the entity from the original) significantly improved with some drop in bleu (bilingual evaluation understudy) score. our study focuses on addressing gender bias in the portuguese language. however, it can also be adapted to other languages, since it is crucial to ensure truly fair and inclusive global communication. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others mitigating gender bias in hindi to english neural machine translation chapter © 2025 biased language detection in court decisions chapter © 2020 genderly: a data-centric gender bias detection system article open access 29 may 2025 1 introduction natural language processing (nlp) plays a fundamental role in our current society, being an essential technology in several areas. it enables machines to understand, interpret, and generate human text in a similar way to humans [21]. this capability is critically important in an era of massive data, where most information is unstructured and textual. nlp is essential for improving the efficiency and accuracy of search engines, enabling function as smarter chatbots and virtual assistants [1, 23], advanced sentiment analysis on social media [17, 26], automatic translation [20] and text classification [12, 13, 18]. additionally, it plays a crucial role in accessibility, making technology more inclusive for people with communication disabilities. in short, nlp is a technology that drives innovation in many sectors, facilitating interaction between humans and machines more naturally and effectively. based on advances in machine learning (ml) and nlp, machine translation (mt) systems have constantly evolved, providing increasingly accurate and contextual translations [8, 20]. they are widely used in various areas, from business document translations to online international communication, helping to overcome language barriers and promoting global understanding. as the demand for global communication increases, the ability to translate efficiently between less widely used languages becomes increasingly important. although machine translation technologies have made notable advances in widely spoken languages, translating into less common languages involves several obstacles, such as a lack of sufficient training data and linguistic diversity. in this work, we aim to explore automatic translation into portuguese, which is the mother tongue of more than 230 million people around the worldfootnote 1. its importance goes far beyond the borders of portugal and brazil and is also spoken in several other countries such as mozambique, angola, cape verde and timor-leste. furthermore, it is one of the official languages of international organizations, such as the community of portuguese-speaking countries (cplp)footnote 2 and the european unionfootnote 3, which makes it a crucial vehicle for diplomacy and global trade. grammatical gender rules present in some languages are a significant challenge for machine translation. it is possible to identify three language grouping [27]. the first are genderless languages (for example, finnish and turkish), in which gender distinctions are minimal and often limited to essential lexical pairings such as kinship or address terms. notional gender languages (e.g., danish, english) include languages that distinguish between lexical gender (e.g., mom/dad) and pronominal gender (e.g., she/he, her/him). finally, grammatical gender languages (for example, french, greek, german and spanish) have a classification system that assigns masculine, feminine, and sometimes neuter gender to nouns, frequently with meaning ties to human referents. gender assignment may be formal for inanimate objects, but it is frequently dependent on meaning for human referents. these languages use a morphosyntactic agreement system, with gender inflections extending to different components of speech such as verbs, determiners, and adjectives [27]. this represents an additional challenge for the task of automatic translation, especially when translating from languages lacking grammatical gender and grammatical gender languages. thus, translating professions and gender-related terms can be challenging in many languages, including translating from english to portuguese [10]. many gender issues can arise in this context: generic masculine: often, in english, terms such as “engineer” or “doctor” are used in the generic masculine, without distinguishing gender. however, when translating into portuguese, it is necessary to decide whether the term will be translated into masculine or feminine, which can perpetuate gender stereotypes. gender ambiguity: some terms in english, such as “actor” or “waitress”, do not make a gender distinction in the original language. however, when translating them into portuguese, it is necessary to decide whether they will be translated neutrally or with a corresponding feminine form, such as “actress” or “waiter”. terms of address and gender: in some professions, terms of address or titles may vary based on a person’s gender. for example, in english, we use “mr.” and “mrs.” (sir and madam) to make this distinction. however, in portuguese, this distinction is more complex, with variations such as “sr.” and “sra.” or “dr.” and “dra.” depending on the context, the translation needs to be carefully chosen. traditionally single-gender professions: some professions have historically been associated with a single gender. for example, “nurse” used to be predominantly female, while “pilot” was considered a male profession. when translating these terms, it is important to consider how cultural and gender norms are evolving and adjust to these changes. table 1 shows two examples of translations containing gender bias. in the first sentence, the occupation “the mechanic” should have been translated as “a mecânica”, which is the feminine form in portuguese, because it is related to the pronoun “she” (ela). the same occurs in the second sentence, where “the nurse” should be translated in the masculine form as “o enfermeiro”, because it is related to the pronoun “he” (ele). table 1. example of gender bias in machine translation using marianmt pre-trained model. entities’ grammatical genders are distinguished by colors: represents male entities and pronouns, signifies female ones, and denotes neutral ones.full size table in this work, we aim to reduce this type of gender bias existing in english-portuguese automatic translation with a focus on fairness. fairness is the property that algorithms and systems do not perpetuate prejudices or unfair discrimination [3, 5, 14, 25]. this involves creating and implementing metrics and strategies to mitigate bias and ensure that automated decisions are impartial and equitable for all people, regardless of their ethnic origin, gender, age, or other protected characteristics. fairness is essential for promoting ethics and equality in the digital age. previous work explored fairness concerns in translation such as [16, 24, 30]. however, none delved into post-processing methods or explored the portuguese language. therefore, developing machine translation systems capable of handling challenging languages is a promising area of research, with significant implications for promoting cultural diversity and inclusive global communication. the main contributions of our work are summarized as follows: we proposed a post-processing fine-tuning approach based on the marianmt model for gender-neutral english to portuguese machine translators. post-processing provides the opportunity to rectify errors or biases that may have arisen during data collection or modeling. we achieved improvements on gender disparities metrics used in mt such as \(\vardelta g\), \(\vardelta s\) and the overall accuracy, in comparison to the baseline model, although some drop in bleu (bilingual evaluation understudy) occurred, which is expected given the fairness \(\times \) accuracy trade-off. we have created a dataset comprising 10,400 sentences, inspired by those utilized in the winomt test but enhanced with custom modifications. this dataset will be made publicly accessible. we highlight the importance of studying fairness in nlp systems, especially for languages with fewer resources such as portuguese. it is crucial to ensure equitable access and representation in ai technologies for underrepresented linguistic communities. this work is organized as follows. section 2 mentions other works that previously explored mt. section 3 provides important concepts used in the work related to fairness in machine learning. section 4 presents the methodology adopted in the work, the datasets, and the algorithms used. section 5 shows the results achieved in mt translation. section 6 presents the final remarks. 2 related work some works posit that mt tools can be harnessed through the use of gender-neutral languages to offer insights into the issue of gender bias in ai. an exhaustive list of occupational titles sourced from the u.s. bureau of labor statistics (bls) was used to construct sentences such as “they are an engineer” (with “engineer” replaced by the specific job title of interest) in twelve distinct gender-neutral languages, including hungarian, chinese, yoruba, and others [24]. these sentences are subsequently translated into english utilizing the google translate api, and they gather data on the prevalence of female, male, and gender-neutral pronouns in the resulting translations. their findings reveal a pronounced bias towards male defaults within google translate, particularly in fields characterized by imbalanced gender representation or stereotypes, such as science, technology, engineering, and mathematics (stem) professions. we juxtapose these findings with bls data on the actual gender distribution within each job title, illustrating that google translate fails to replicate real-world gender demographics. authors enhance neural machine translation (nmt) systems by introducing gender information in their work [30]. they created extensive datasets containing speaker information for 20 language pairs and performed experiments that integrated gender data into nmt for multiple language pairs. they employed the opennmt-py toolkit which is structured as sequence-to-sequence encoder-decoders utilizing lstm recurrent units. they demonstrate that the inclusion of a gender feature in an nmt system yields a significant enhancement in translation quality for select language pairs. other works proposed a gender-debiased approach for mt. an adversarial learning approach was used to reduce gender bias in a seq2seq machine translation model in [15]. in this approach, a prediction model \(m\) with weights \(w\) learns to predict an output \(y\) from the input \(x\), while remaining neutral with respect to the protected variable \(z\). the adversary \(a\) tries to predict \(z\) from the model’s output predictions \(\hat{y}\). the study’s approach to reducing gender bias hinged on a nuanced adaptation of the transformer model, targeting the word embeddings within both the encoder and decoder components [16]. to bolster gender neutrality, pre-trained word embeddings were introduced, yielding a diverse array of models, each utilizing distinct pre-trained word embeddings sourced primarily from glove. the study explored multiple experimental scenarios, including training models without any pre-trained word embeddings to allow autonomous learning, and incorporating pre-trained embeddings, such as standard glove, harddebiased glove, and gender neutral glove (gn-glove), derived from the same corpus. additionally, the investigation delved into the specific utilization of pre-trained embeddings, examining three cases: encoder-only, decoder-only, and both encoder and decoder, to comprehensively evaluate their impact on mitigating gender bias during translation. the authors evaluate the proposed system on the workshop on machine translationfootnote 4 (wmt) english spanish benchmark task. this holistic framework facilitated a comprehensive assessment of embedding strategies, shedding light on effective means of enhancing translation fairness. in their work on mitigating sensitivity to protected attributes such as gender and age in sentiment classification, authors evaluated round-trip translation as a technique [9]. they demonstrate, in particular, that translating danish product reviews into english and back minimizes group disparity across three distinct classification structures. they used two different pre-trained language models, namely the multilingual laser model and a monolingual bert trained for danish. on top of these, some classifiers were employed, including nearest neighbor, logistic regression, and (gaussian kernel) support vector machines (svms). the authors discover that round-trip translation at test time reduces the fairness gap (by up to 47%), but that the effect disappears for the best models (svms stacked on bert representations) when both training and test data are translated into a foreign language and back. 3 background 3.1 fairness broadly speaking, in ml and nlp studies, fairness is focused on ensuring that systems and algorithms treat individuals and groups fairly without introducing biases or discrimination based on characteristics such as gender, race, or ethnicity. two important aspects of fairness must be considered: its definition and its metrics [25]. fairness definition. as pointed out by the authors, addressing the bias subject, researchers in ai, software engineering, and law communities have proposed more than twenty different notions of fairness in the last few years [31]. however, there is no consensus regarding the appropriate definition for specific situations. furthermore, comprehending the intricate distinctions among numerous definitions poses a considerable challenge. one of the fundamental aspects of fairness definition is the concepts of group fairness and individual fairness. group fairness focuses on ensuring that outcomes are equitable for different predefined groups (e.g., men and women). on the other hand, individual fairness seeks to treat similar individuals similarly, regardless of their group membership. despite appearing to be in conflict, individual and group fairness measures do not inherently represent distinct normative principles [2]. also, fairness definitions often revolve around the concepts of anti-discrimination and equal opportunity. anti-discrimination aims to prevent biased treatment against any specific group. equal opportunity, on the other hand, aims to provide individuals from different groups with an equal chance of a positive outcome. disparate impact and disparate treatment are two concepts in the realm of discrimination and bias that helps to address and identify different forms of discrimination and play a crucial role in ensuring fairness and equal opportunities. disparate impact deals with situations where a system’s outcomes disproportionately affect one group, even if no explicit bias exists. disparate treatment relates to situations where individuals from different groups are treated differently due to bias or discrimination. fairness metrics. many fairness metrics are based on the concepts from a confusion matrix (show in table 2), a tabular representation used to assess the effectiveness of a classification algorithm in terms of positive and negative results correctly or incorrectly predicted by the model. some common fairness metrics include: table 2. example of a confusion matrix.full size table demographic parity: measures whether the positive prediction rates \((tp+fp/tp+tn+fp+fn)\) are equal across different groups. equal opportunity: ensures that the true positive rate \((tp/tp+fn)\) is the same across groups. equalized odds: requires both true positive rate \((tp/tp+fn)\) and false positive rate \((fp/fp+tn)\) to be equal across groups. predictive parity: checks if the precision \((fp/fp+tn)\) is the same across groups. false positive rate balance: ensures that the false positive rate \((fn/fn+tp)\) is equal across groups. false negative rate balance: ensures that the false negative rate \((fn/fn+tp)\) is equal across groups. these metrics help assess whether a model is fair across different demographic groups or categories. approaches. ensuring that models treat all demographic groups equitably requires careful consideration across the entire machine learning pipeline. usually three approaches have been explored: pre-processing, in-processing, and post-processing [6]. 1. pre-processing: in fairness-aware machine learning, pre-processing involves modifying the training data to mitigate biases before the learning process begins. techniques include re-weighting samples, altering features to remove sensitive information, and generating synthetic data to balance classes across different groups. 2. in-processing: this stage incorporates fairness constraints directly into the learning algorithm. modifications can include altering the objective function to penalize unfair predictions or adding constraints that ensure equitable treatment across different demographic groups. 3. post-processing: after the model has been trained, post-processing techniques adjust the model’s predictions to reduce unfair outcomes. these methods can involve re-calibrating prediction thresholds or applying transformations to the output probabilities to ensure fairness across groups. when it comes to combating biases in nlp tasks, the primary emphasis is on addressing these issues during the pre-processing phase, as noted in the work by [6]. this entails actions such as removing or substituting specific words, adjusting dictionaries, and applying unsupervised techniques to balance the training dataset. bias and fairness in mt. bias evaluation in translations is frequently difficult to do because there isn’t a clear ground truth. however, studies such as that of [4], entitled “man is to computer programmer as woman is to homemaker?” show that word embeddings exhibit biases that reproduce the gender stereotypes prevalent in society. in their study, the researchers began by selecting occupations closely associated with the terms “she” and “he” within the word embeddings they were analyzing. to assess whether these selected occupations indeed reflected gender stereotypes, the researchers engaged individuals to evaluate each occupation and to determine whether it conveyed female stereotypes or male stereotypes or if it was gender-neutral. to quantify the degree of stereotypicality, they employed a rating scale ranging from 0 to 10, where higher scores indicated a stronger association with gender stereotypes. upon analysis, the study found the positions of these selected occupation words along the “she-he” axis within the word embeddings exhibited a substantial correlation with the ratings assigned. in other words, the geometric properties of the embeddings closely reflected the human judgments of gender stereotypes. winomt, a challenge set for evaluating gender bias in machine translation using a concatenation of winogender and winobias, was presented in [29]. the evaluation dataset comprises \(3,888\) sentences designed to probe potential gender bias within the translations. in each sentence, a primary entity, coreferent with a pronoun, interacts with a secondary entity and it seeks to reveal gender bias tendencies in the translation system. the set has an equal balance of both male and female genders, as it does with stereotypical (e.g., a male doctor) and nonstereotypical (e.g. a female engineer) gender-role assignments. their methodology for evaluating machine translation systems begins by calculating the overall accuracy, which is determined by assessing the percentage of instances in which the translation successfully maintains the gender of the entity from the original english sentence. their findings reveal that across eight different languages, most of the tested machine translation systems perform poorly in preserving gender accuracy. even the best-performing model for each language typically exhibits performance that is not significantly better than random guessing when correctly inflecting gender in the translations. \(\vardelta g\) represents the disparity in f1 scores between sentences containing masculine entities and those featuring feminine entities. they point out that all evaluated systems, except one, perform better on male roles, which may indicate these are more frequent in the set used for training. finally, \(\vardelta s\) quantifies the discrepancy in accuracy when translating the antecedent in sentences with pro-stereotypical and anti-stereotypical role assignments. according to the authors, the results of this metric show that all tested systems have a meaningful better performance when presented with pro-stereotypical assignments (e.g., a female housekeeper), as their performance worsens when translating anti-stereotypical roles (e.g., a male librarian). despite the inclusion of a diverse set of eight languages (spanish, french, italian, russian, ukrainian, hebrew, arabic, and german) in the winomt test, it is worth noting that portuguese was not part of the selection. 4 methodology this section will detail the methodology adopted. thus, sect. 4.1 details the post-processing fairness techniques used, sect. 4.2 details the datasets used, and sect. 4.3 details the steps for executing the experiments. 4.1 post-processing fairness techniques post-processing techniques are a well-known method for addressing potential biases in model outputs related to protected variables or subgroups in the field of machine learning fairness [6]. these techniques stand out for their remarkable flexibility because they don’t need access to the internal models or algorithms; instead, they only depend on the model’s predictions and sensitive attribute data [6]. these techniques are especially well suited for “black-box” scenarios, which are those in which the entire ml pipeline is not completely transparent. as a post-processing approach, fine-tuning holds significant promise in reducing gender bias in machine translation. in the context of this study, fine-tuning involves adapting the marianmt pre-trained machine translation model using a gender-balanced dataset. by incorporating a diverse set of gender-specific examples into the fine-tuning process, the model becomes more attuned to gender nuances without undermining translation fluency or accuracy. 4.2 datasets in our endeavor to reduce gender bias in machine translation, we adopted a strategic approach that involved the creation of a specialized dataset for fine-tuning the marianmt model. this dataset, consisting of 100,400 parallel english-portuguese sentences, was crafted to balance gender bias reduction while maintaining high translation quality. to achieve this, we strategically combined 90,000 sentences from the capes tdc corpus [28] and 10,400 artificial sentences, generated specifically for this task. capes corpus is a trusted source that had initially been employed in the original training of the marianmt model. this corpus was compiled from the abstracts of all theses and dissertations produced in brazil between 2013 and 2016. this choice was driven by the intention to preserve the translation quality and prevent overfitting that might occur when relying solely on artificial data. these artificial sentences were thoughtfully generated, drawing inspiration from the sentences used in the winomt test but with tailored modifications. by blending authentic sentences from the capes corpus with designed artificial examples, we aimed to balance maintaining translation quality and enhancing the model’s ability to address gender bias. this approach acknowledges the importance of real-world translation dynamics, ensuring that our fine-tuning process remains effective in reducing gender bias while upholding the standards of translation quality established by the original model. 4.3 experimental setup the experimental setup for this research was conducted on the “béluga” cluster, a high-performance computing resource provided by digital research alliance of canada, equipped with substantial hardware resources. the hardware configuration included four nvidia v100sxm2 gpus, each equipped with 16gb of ram. in terms of software, the experiment was conducted using a well-established stack of tools and libraries. python 3.10.1 served as the primary programming language. pytorch 2.0.1, a widely recognized deep learning framework, was employed for model development and training. natural language toolkit (nltk) 3.8.1 facilitated text preprocessing and linguistic analysis, while pandas 2.0.3 offered efficient data manipulation capabilities. the transformers’ library, version 4.31.0, played a pivotal role in facilitating the fine-tuning process of the marianmt model, streamlining the integration of transformer-based architectures into the research workflow. the parameters were configured to tailor the fine-tuning process. the model’s decoder layer dropout was set to 0.2, contributing to the regularization of the model during training. the fine-tuning parameters were also defined, encompassing a learning rate of 1e-5, a batch size of 8, and a total of 10 training epochs. the optimizer selected for this task was adamw. 4.4 translation and gender bias evaluation for the evaluation process, we have developed custom scripts inspired by the original methodology winomt [29], tailored to our specific english-portuguese translation context. these scripts automatically extract the grammatical gender assigned to the primary entity within each translation. following this extraction, a comparison is made between the gender of the translated primary entity and the gender annotated in the gold standard data. our objective is to gauge the extent to which our translation models align with the gender of the primary entity, as per the gold annotations. the key performance metrics for evaluating the winomt dataset include \(\vardelta g\), \(\vardelta s\), and the overall accuracy of preserving the gender of entities during translation, referred to as “acc.”. the study compared the quality of translations before and after fine-tuning using the bleu (bilingual evaluation understudy) [22] metric to determine whether it had a negative effect on translation quality. bleu is a well-established metric for machine translation evaluation that measures the overlap between the model machine translation and reference translation. it is calculated based on accuracy, which measures the percentage of n-grams in the machine translation that also appear in the reference translations. the bleu score ranges from 0 to 1, with 1 being a perfect match with the reference translations [22]. utilizing the marianmt model as our foundation, we conducted fine-tuning using a dataset comprising english-portuguese sentences. this dataset was self-generated to include sentences that mitigate gender bias. ultimately, upon evaluating the model, we observed a decrease in translation accuracy concurrent with a reduction in gender bias. 5 results and discussion the experimental outcomes underscore success in achieving the primary goal of gender bias reduction in machine translation through fine-tuning. the three fairness metrics, \(\vardelta g\), \(\vardelta s\), and “acc.” from the winomt test, exhibited improved results post-fine-tuning, as shown in table 3. these metrics, designed to assess the model’s capacity to produce fair and equitable translations, demonstrated clear progress in mitigating gender bias, aligning with the objective of the research. these positive results indicate that the fine-tuning process significantly improved the model’s ability to generate translations that respect gender neutrality and balance, a pivotal step toward creating more inclusive and unbiased machine translation systems. however, it’s worth noting that these promising improvements in fairness metrics were accompanied by a trade-off, as anticipated, in terms of the bleu score. the reduction in the bleu score underscores the well-established fairness-accuracy trade-offfootnote 5 that often accompanies efforts to reduce bias in machine translation. nevertheless, the \(0.38\) score of fine-tuned model tells that it still produces good translationsfootnote 6. while the primary focus of the research was to enhance fairness and mitigate gender bias, this trade-off in translation quality serves as a reminder of the intricate balance that exists between these two objectives and indicates the need to continue the research process in order to achieve better results. table 3. results for bias metrics and translation quality on pre-trained marianmt model and fine-tuned marianmt model.full size table 5.1 discussion the fine-tuned marianmt correctly translates the occupations “the mechanic” to the feminine form in portuguese, and “the nurse” to the masculine form in portuguese, see table 4. this suggests that the model is capable of producing gender-neutral or gender-sensitive translations when dealing with gendered occupational terms. table 4. example of translations generated using fine-tuned marianmt. entities’ grammatical genders are distinguished by colors: represents male entities and pronouns, signifies female ones, and denotes neutral ones.full size table however, despite reducing the \(\vardelta s\) metric compared to the original model, the fine-tuned model, like all systems tested in [29], also makes more translation errors in sentences with anti-stereotyped gender assignment (e.g. a female farmer) than in stereotyped roles (e.g. a male lawyer), although to a lesser degree than before the fine-tuning. even though the test’s focus is related to the stereotyped gender assignment in occupations, the test includes a small sample (\(240\) sentences) in which the main agent has a gender-neutral role. in these cases, the ability of the fine-tuned model to preserve the gender (acc.) is only \(0.2292\), falling even below the model before fine-tuning, which is \(0.3125\). this limitation is primarily found in sentences like “the technician told the customer that they could pay with cash.” or “the employee gave the supervisor feedback on their managing style.”. in these cases, “they” and “their” are been employed to refer to an individual in a generic or gender-neutral context. according to [11] this use particularly provides a general reference to that individual without specifying their gender (like she or he). although, for cases like this, there are proposals for the use of gender-neutral pronouns in portuguese, such as “elu”footnote 7, the marianmt translator model, and consequently the fine-tuned version, do not include this variation. 5.2 limitations and future work while the results of this research have demonstrated the effectiveness of fine-tuning as a technique for reducing gender bias in mt, it is important to acknowledge some limitations in this work. the dataset used in this study was specifically crafted based on the sentences of the winomt test. while it was designed to be well-suited for this test, it may not fully represent the complexities of gender bias in “real-world” translation scenarios, which may require further investigation. also, the test primarily focuses on gender bias related to occupation, which is undoubtedly significant. however, gender bias in mt extends beyond occupational contexts, which can be explored in future works. building upon the outcomes of this research, there are several avenues for future work on reducing gender bias in machine translation. it is essential to continue research into improving fairness metrics translations while ensuring these enhancements do not significantly reduce translation quality. exploring alternative datasets or adjusting fine-tuning parameters can be a promising avenue. future research can focus on enhancing mt systems to include translations incorporating neutral portuguese language. this would help bridge the gap in providing translations that include gender-diverse language choices and align with evolving societal norms. 6 conclusions machine translation tools have significantly contributed to communication, enabling seamless global interactions and bridging cultural gaps through the internet. however, as the field of fairness in machine learning has grown, concerns about equity have extended to mt tools, especially regarding their potential to propagate gender bias present in training data. in this work, we explored the theme of gender bias in english-portuguese machine translation and proposed a bias reduction method using fine-tuning on the pre-trained model marianmt. the significance of studying fairness in english-to-portuguese mt is primarily linked to the limited research focus on the portuguese language. therefore, research of this nature tends to contribute to the advancement of fairness research and the development of translation tools with a reduced degree of bias, benefiting a community of more than \(230\) million portuguese speakers worldwide. language is a powerful tool for shaping perceptions, reinforcing stereotypes, and potentially influencing attitudes. gender biases present in the texts used as training data can perpetuate harmful representations of individuals and communities, leading to potential discrimination and distorted perceptions. by understanding and addressing these biases, we can work towards creating inclusive and equitable communication platforms that respect the dignity and diversity of all users. in this regard, our research aimed to contribute to this goal by investigating gender bias in mt and developing techniques to mitigate its effects. the experiments conducted in this study demonstrated the effectiveness of fine-tuning as a technique for reducing gender bias in machine translation models. during this fine-tuning process, the model adapts its existing knowledge to the new dataset (in this case a gender-balanced parallel corpus), adjusting its parameters to make less biased translations, while aiming to preserve the translation quality of the original model. one of the main concerns when applying bias reduction techniques is the fairness-accuracy trade-off. our experiments have shown that fine-tuning can reduce bias without significantly sacrificing translation accuracy. comparisons of bleu scores before and after fine-tuning indicate that, despite some loss, the generated translations remain of good quality. finally, it is worth noting that the work presented here represents only a small contribution towards equity in machine translation. it is necessary to expand research and explore new avenues to enhance bias detection, understand the implications of various types of bias, and refine mitigation techniques, ensuring that translations are fair and high-quality. notes 1.https://www.babbel.com/en/magazine/how-many-people-speak-portuguese-and-where-is-it-spoken. 2.https://www.cplp.org/. 3.https://european-union.europa.eu/principles-countries-history/languages_en. 4.http://www.statmt.org/wmt13/. 5.most bias reduction algorithms harm the performance of the prediction model in the process of making it fair, which is commonly referred to as the fairness-accuracy trade-off [7]. 6.https://cloud.google.com/translate/automl/docs/evaluate. 7.according to [19], in portuguese, the most commonly used gender neuter pronoun is “elu". references adamopoulou, e., moussiades, l.: an overview of chatbot technology. in: maglogiannis, i., iliadis, l., pimenidis, e. (eds.) aiai 2020. iaict, vol. 584, pp. 373–383. springer, cham (2020) chapter  math  google scholar  binns, r.: on the apparent conflict between individual and group fairness (2019) google scholar  blodgett, s.l., barocas, s., daumé iii, h., wallach, h.: language (technology) is power: a critical survey of “bias” in nlp. arxiv preprint arxiv:2005.14050 (2020) bolukbasi, t., chang, k.w., zou, j., saligrama, v., kalai, a.: man is to computer programmer as woman is to homemaker? debiasing word embeddings (2016) google scholar  broder, r.s., berton, l.: performance analysis of machine learning algorithms trained on biased data. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 548–558. sbc (2021) google scholar  caton, s., haas, c.: fairness in machine learning: a survey. acm computing surveys (2020) google scholar  chakraborty, j., majumder, s., yu, z., menzies, t.: fairway: a way to build fair ml software. in: proceedings of the 28th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, pp. 654–665. acm, virtual event usa (nov 2020) google scholar  chauhan, s., daniel, p.: a comprehensive survey on various fully automatic machine translation evaluation metrics. in: neural processing letters, pp. 1–55 (2022) google scholar  christiansen, j.g., gammelgaard, m., søgaard, a.: the effect of round-trip translation on fairness in sentiment analysis. in: proceedings of the 2021 conference on empirical methods in natural language processing, pp. 4423–4428 (2021) google scholar  cunha, c., cintra, l.: nova gramática do português contemporâneo. lexikon editora digital ltda (2016) google scholar  dictionary, o.e.: they, pron., sense i.2.b (9 2023). https://doi.org/10.1093/oed/9782781428 duarte, j.m., berton, l.: a review of semi-supervised learning for text classification. artif. intell. rev. 56(9), 9401–9469 (2023) article  math  google scholar  duarte, j.m., sousa, s., milios, e., berton, l.: deep analysis of word sense disambiguation via semi-supervised learning and neural word representations. inf. sci. 570, 278–297 (2021) article  mathscinet  math  google scholar  field, a., blodgett, s.l., waseem, z., tsvetkov, y.: a survey of race, racism, and anti-racism in nlp. arxiv preprint arxiv:2106.11410 (2021) fleisig, e., fellbaum, c.: mitigating gender bias in machine translation through adversarial learning (2022) google scholar  font, j.e., costa-jussa, m.r.: equalizing gender biases in neural machine translation with word embeddings techniques. arxiv preprint arxiv:1901.03116 (2019) garcia, k., berton, l.: topic detection and sentiment analysis in twitter content related to covid-19 from brazil and the usa. appl. soft comput. 101, 107057 (2021) google scholar  garcia, k., shiguihara, p., berton, l.: breaking news: unveiling a new dataset for portuguese news classification and comparative analysis of approaches. plos one 19(1), e0296929 (2024) google scholar  lau, h.d.: o uso da linguagem neutra como visibilidade e inclusão para pessoas trans não-binárias na língua portuguesa: a voz “del@ s” ou “delxs”? não! a voz “delus”. v simpósio internacional em educação sexual: saberes/trans/versais currículos identitários e pluralidades de gênero. anais do v simpósio internacional em educação sexual: saberes/trans/versais currículos identitários e pluralidades de gênero. masringá (2017) google scholar  maruf, s., saleh, f., haffari, g.: a survey on document-level neural machine translation: methods and evaluation. acm comput. surv. (csur) 54(2), 1–36 (2021) article  math  google scholar  nadkarni, p.m., ohno-machado, l., chapman, w.w.: natural language processing: an introduction. j. am. med. inform. assoc. 18(5), 544–551 (2011) article  math  google scholar  papineni, k., roukos, s., ward, t., zhu, w.j.: bleu: a method for automatic evaluation of machine translation. in: proceedings of the 40th annual meeting on association for computational linguistics, pp. 311–318. acl ’02, association for computational linguistics, usa (2002) google scholar  prado, c., netto, a.v., berton, l., takahara, a.k.: aplicação de healthbots em língua portuguesa: revisão narrativa. j. health inform. 13(4) (2021) google scholar  prates, m.o., avelar, p.h., lamb, l.c.: assessing gender bias in machine translation: a case study with google translate. neural comput. appl. 32, 6363–6381 (2020) article  math  google scholar  rabonato, r.t., berton, l.: a systematic review of fairness in machine learning. ai and ethics, pp. 1–12 (2024) google scholar  santos, d.k.s., berton, l.: analysis of twitter users’ sentiments about the first round 2022 presidential election in brazil. in: anais do xx encontro nacional de inteligência artificial e computacional, pp. 880–893. sbc (2023) google scholar  savoldi, b., gaido, m., bentivogli, l., negri, m., turchi, m.: gender bias in machine translation. trans. assoc. comput. linguist. 9, 845–874 (08 2021) google scholar  soares, f., yamashita, g.h., anzanello, m.j.: a parallel corpus of theses and dissertations abstracts. in: lecture notes in computer science, pp. 345–352. springer international publishing (2018) google scholar  stanovsky, g., smith, n.a., zettlemoyer, l.: evaluating gender bias in machine translation. in: acl. association for computational linguistics, florence, italy (6 2019) google scholar  vanmassenhove, e., hardmeier, c., way, a.: getting gender right in neural machine translation. arxiv preprint arxiv:1909.05088 (2019) verma, s., rubin, j.: fairness definitions explained. in: proceedings of the international workshop on software fairness, pp. 1–7. fairware ’18, association for computing machinery, new york, ny, usa (2018) google scholar  download references acknowledgements authors thanks the coordenação de aperfeiçoamento de pessoal de nível superior (capes) and conselho nacional de desenvolvimento científico e tecnológico (cnpq). author information authors and affiliations universidade federal de sao paulo, avenida cesare mansueto giulio lattes, n° 1201, eugênio de mello, sp, brazil ricardo trainotti rabonato & lilian berton dalhousie university, 6050 university avenue, halifax, ns, canada evangelos milios authors ricardo trainotti rabonatoview author publications search author on:pubmed google scholar evangelos miliosview author publications search author on:pubmed google scholar lilian bertonview author publications search author on:pubmed google scholar corresponding author correspondence to lilian berton . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper trainotti rabonato, r., milios, e., berton, l. (2025). gender-neutral english to portuguese machine translator: promoting inclusive language. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_13 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords nlp machine translator fairness portuguese. publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a heterogeneous network-based positive and unlabeled learning approach to detect fake news | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a heterogeneous network-based positive and unlabeled learning approach to detect fake news conference paper first online: 28 november 2021 pp 3–18 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a heterogeneous network-based positive and unlabeled learning approach to detect fake news download book pdf download book epub mariana c. de souza  orcid: orcid.org/0000-0002-1746-841410, bruno m. nogueira  orcid: orcid.org/0000-0002-2401-242311, rafael g. rossi  orcid: orcid.org/0000-0001-8513-321312, ricardo m. marcacini  orcid: orcid.org/0000-0002-2309-348710 & … solange o. rezende  orcid: orcid.org/0000-0002-5233-763910  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1271 accesses 8 citations abstract the dynamism of fake news evolution and dissemination plays a crucial role in influencing and confirming personal beliefs. to minimize the spread of disinformation approaches proposed in the literature, automatic fake news detection generally learns models through binary supervised algorithms considering textual and contextual information. however, labeling significant amounts of real news to build accurate classifiers is difficult and time-consuming due to their broad spectrum. positive and unlabeled learning (pul) can be a good alternative in this scenario. pul algorithms learn models considering little labeled data of the interest class and use unlabeled data to increase classification performance. this paper proposes a heterogeneous network variant of the pu-lp algorithm, a pul algorithm based on similarity networks. our network incorporates different linguistic features to characterize fake news, such as representative terms, emotiveness, pausality, and average sentence size. also, we considered two representations of the news to compute similarity: term frequency-inverse document frequency, and doc2vec, which creates fixed-sized document representations regardless of its length. we evaluated our approach in six datasets written in portuguese or english, comparing its performance with a binary semi-supervised baseline algorithm, using two well-established label propagation algorithms: lphn and gnetmine. the results indicate that pu-lp with heterogeneous networks can be competitive to binary semi-supervised learning. also, linguistic features such as representative terms and pausality improved the classification performance, especially when there is a small amount of labeled news. supported by coordenação de aperfeiçoamento de pessoal de nível superior [10662147/d], fundação de amparo à pesquisa do estado de são paulo [2019/25010-5, 2019/07665-4], and conselho nacional de desenvolvimento científico e tecnológico [426663/2018-7, 433082/2018-6, and 438017/2018-8]. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a network-based positive and unlabeled learning approach for fake news detection article 18 november 2021 fake news detection chapter © 2021 positive-unlabeled learning for sentiment analysis with adversarial training chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization heterodox economics learning algorithms machine learning probabilistic data networks self-serving bias 1 introduction detecting fake news is a challenging task since fake news constantly evolves, influencing the formation of the opinion of social groups as accepted [6]. to minimize the effects of disinformation provided by fake news, machine learning algorithms have been proposed in the literature, which learn classification models to discriminate true and fake content [4, 22]. the most common way to deal with this problem is to characterize the fake news detection as a supervised binary or multiclass classification problem [22]. however, labeling real and fake news to cover different subjects, sources, and falsity levels, as well as updating the classification model are costly processes [4]. all of these challenges drive the investigation of more appropriate models for representing and classifying news. positive and unlabeled learning (pul) algorithms can be good alternatives in this scenario [3]. pul algorithms learn models considering little labeled data of the interest class and use the unlabeled data to increase classification performance [9]. therefore, pul eliminates the need to label a large number of news of the uninteresting classes. due to the suitability of pul to fake news detection scenario, in this paper, we proposed an approach based on the algorithm positive and unlabeled learning by label propagation [9] (pu-lp), a network-based semi-supervised transductive learning algorithm. pu-lp has been little explored for text classification, despite its good performance in numerical datasets. pu-lp infers reliable sets of interest (fake news) and not interest (true news) using a small number of labeled data, katz index and a k-nearest neighbor (k-nn) network. then, a label propagation algorithm is used to label the remaining unlabeled objects as interest or not interest. representing news using traditional models, i.e., bag-of-words or embeddings, may not be discriminative enough to distinguish fake and real content [17, 21]. thus, an important challenge for automatic fake news detection is to assemble a set of features that efficiently characterize a false content in a dynamic scenario. this set may contain features about creator or spreader, the target victims, the news content and the social context [22]. in this work, we propose using content-based features in a domain-independent fake news detection approach based on the pu-lp algorithm. our network use linguistic features such as representative terms, news emotiveness, average number of words per sentence, and pausality as network objects, thus making a heterogeneous network [12]. these features have proven to be relevant for news classification [13]. also, the presence of different objects and relations in the network allows for representing different patterns on data [15]. to create the k-nn news network, we represent the text with bag-of-words (bow) and doc2vec (d2v) models to compute the similarity between the news. to the best of our knowledge, heterogeneous networks and pul have not been explored in the literature to detect fake news. we evaluated different combinations of features to measure the impact of these features in the classification performance. our experiments used relatively small sets of labeled fake news: from 10% to 30% of the fake news set. the heterogeneous network was compared with the k-nn networks, used in the original proposal of pu-lp. two portuguese datasets and four english datasets were used, containing only one or more subjects. for the label propagation, we used two well-established algorithms: label propagation through heterogeneous networks [12] and gnetmine [7]. we evaluated the algorithms considering the \(f_{1}\) measure of the fake news class. the results obtained the proposed approach were compared with a semi-supervised binary baseline, and demonstrate competitiveness even with half the news initially labeled and without the labeling of real news. this paper is organized as follows: sects. 2 and 3 present related work. section 4 describes the proposed approach. section 5 presents experimental evaluation and discussions. section 6 presents conclusions and future works. 2 related work existing methods for news classification, in general, represent news in a structured way using traditional models such as bow, word (or document) embeddings or networks [22]. however, these models are not discriminative enough to capture the nuances to distinguish fake from real news, requiring the investigation of textual and contextual features that can be incorporated into the representation model [17, 21, 22]. contextual features are extracted from the news environment, such as post, source, and publisher related information [21]. textual features involve lexical, syntactic and semantic aspects of the news content. lexical features are related to frequency statistics of words, which can be done using n-gram models, able to identify, for example, the presence of vague, doubt or sensationalist expressions [4]. syntactic features are related to the presence and frequency of part-of-speech patterns, like subjects, verbs and adjectives [17]. at the semantic level, meanings are extracted from terms, where methods like linguistic inquiry and word count (liwc) [10] can be used to estimate emotional, cognitive and structural components present in the written language. in [17], authors proposed representation models for the fake.br dataset that combine bow and linguistic features, such as pausality, emotiveness, and uncertainty. the authors reached 96% of macro \(f_{1}\) considering two thirds of the data as training. in [18], the authors proposed a multimodal representation that combines text and images information. for each news, 124 textual features (81 of them with liwc) and 43 image features are collected. the approach achieves 95% accuracy with random forest using approximately 7,000 news documents and 90% of data in the training set. some works use networks as a representation model to detect false information. [13] proposes a knowledge network for fact-checking, in which the entry is a triple (subject, predicate, object). the network is constructed with information from wikipedia, in which given a sentence, if it exists within the knowledge network, it is considered authentic. the approach achieves approximately 74% accuracy. [21] proposes a diffuse neural network graph for inferring the credibility of news, creators and subjects. the graph has approximately 14,000 news items, 3,600 creators and 152 subjects. the approach achieves the best overall performance in the binary classification compared to svm, deep walk and a neural network. [11] proposes an adversarial active learning heterogeneous graph neural network to detect fake news. a hierarchical attention mechanism is used to learn nodes’ representations, and a selector is responsible for consulting high-value candidates for active learning. the authors evaluate two datasets: the first containing 14,055 news, 3,634 creators and 152 subjects, and the second containing 182 news, 15,257 twitter users and 9 publishers. with 20% of real and fake news labeled, the algorithm reaches respectively 57% and 70% of macro \(f_{1}\). [19] proposes iarnet, an information aggregating and reasoning network to detect fake news on heterogeneous networks. the network contains the post source, comments and users as nodes and interactions between them as edges. the authors evaluated two datasets, weibo and fakeddit, and 70% of the labeled news for training. the algorithm achieved 96% accuracy. to avoid real news labeling efforts, [5] proposes the dcdistanceocc, a one-class learning (ocl) algorithm to identify false content. for each news item, linguistic characteristics are extracted, such as number of words per sentence and sentiment of the message. a new example is classified as the interest class if its distance to the class vector is below a threshold. the algorithm is executed using 90% fake news to train the model. the approach reaches an average \(f_1\) that ranges from 54% to 67%, especially considering the fake.br dataset. it is possible to observe that most of the existing work for fake news detection adopts binary supervised learning approaches, which requires labeling news considering the real and fake classes. additionally, the broad spectrum of real news also makes the generation of a representative labeled training set difficult. on the other hand, the pul algorithms, which mitigate the drawback of binary supervised learning algorithms in fake news detection, are scarce in this scenario. also, the use of heterogeneous networks, which usually are more adequate in semi-supervised text classification [12], has not been explored in the literature for detecting fake news. considering these gaps, in this paper we propose a pul algorithm for fake-news detection, which is based on a heterogeneous network composed of news, terms, and linguistic features. our approach uses only information that can be collected in the publication’s content, not depending on external information. the proposed approach is detailed in the next sections. 3 positive and unlabeled learning algorithms pul algorithms learn models considering a set of labeled documents of an interest class and unlabeled documents to train a classifier (inductive semi-supervised learning) or to classify unlabeled known documents (transductive semi-supervised learning) [3]. the purpose of using unlabeled documents during learning is to improve classification performance. besides, since the unlabeled documents are easy to collect and a user has to label only a few documents of the interest class, pul has gained attention in the last years [3, 9]. the most common pul approaches are those which perform learning in two stages. in the first stage, a set of non-interest documents is generated by extracting reliable outliers documents to be considered as non-interest class. in addition, the set of interest documents can also be increased with reliable interest documents. once there are interest, non-interest, and unlabeled documents, a transductive learning algorithm is applied to infer the label to the remaining unlabeled documents in the second stage [3]. generally, the pul algorithms based on this framework include the application of self-training in some or all of the steps, which can be computationally costly and do not improve the classification performance [12]. we can observe the use of algorithms based on vector-space model, such as rocchio or expectation maximization, also demonstrated lower performances than other approaches in text classification [12, 20]. despite the benefits of network-based approaches in semi-supervised learning scenarios, there are few pul proposals based on networks, such as positive documents enlarging pu classifier (pe-puc) [20] and positive and unlabeled learning by label propagation (pu-lp) [9]. pe-puc uses network representations only to increase the set of interest documents with reliable positive documents. the network is not used in the classification step. the remaining steps consider bayesian classifiers, which have been shown to perform poorly when the labeled training data set is small [12]. the use of networks allows to extract patterns of the interest class even they are given by different distributions, densities on regions in the space [15]. as pu-lp is entirely network-based, achieving good performance in [9], we propose to use it for news classification. next section describes the proposed approach. 4 proposed approach: pu-lp for fake news detection considering that pul algorithms can reduce labeling effort for news classification, the benefits of network-based representations for semi-supervised learning, and that adding extra information to the network can contribute to differentiating real and fake content, we propose an approach based on pu-lp algorithm applied to fake news detection. after pu-lp infer interest and non-interest news sets in the news network, we add relations (edges) between news, relevant terms and linguistic features making a heterogeneous network. thus, a label propagation is applied to classify the unlabeled news. we compared different proposed heterogeneous networks with the traditional pu-lp algorithm considering only the news network. we use different parameters, label propagation algorithms, and representation models for unstructured data to assess the algorithms’ behavior. figure 1 presents the proposed approach of pu-lp for semi-supervised fake news detection. the next subsections presents details of each stage of the proposed approach. fig. 1. proposed approach for detecting fake news based on the semi-supervised pu-lp algorithm. circular nodes represent news, nodes with the letter t are representative unigrams and bigrams, nodes with the letter p correspond to pausality, e emotiveness, and s the average number of words per sentence. full size image 4.1 news collection and representation model let \(\mathcal {d} = \{d_1, \ldots , d_l, d_{l+1}, \ldots , d_{l+u} \}\) be a news set and \(\mathcal {c} = \big \{\textit{interest},\)\( \textit{not interest}\big \}\), i.e., \(\mathcal {c} = \{\textit{fake}, \textit{real}\}\), be a set of class labels. the first l elements of \(\mathcal {d}\) are fake labeled news, composing the interest labeled set \(\mathcal {d}^{+}\). the remaining u elements are unlabeled news (fake and real), composing the set \(\mathcal {d}^{u}\), and \(u \gg l\) (fig. 1 stage 1). the news dataset must be pre-processed, and a representation model, such as bow or document embedding [1] must be adopted to transform news into structured data (fig. 1 stage 2). next, pu-lp builds an adjacency matrix with the complete set of examples, \(\mathcal {d}\), detailed below. 4.2 k-nn matrix, katz index and sets extraction the representation model and a chosen distance metric are used to calculate an adjacency matrix. in this matrix, news with similar content has a low distance between them. the adjacency matrix is used as a basis for building a k-nearest neighbors (k-nn) matrix, called a, so that \(a_{i,j} = 1\) if the news \(d_j\) is one of the k nearest neighbors to the news \(d_i\), and \(a_{i,j} = 0\) otherwise (fig. 1 stage 3). through the k-nn matrix, a heterogeneous network \(\mathcal {n}\) is also created. the heterogeneous network can be defined as a triple \(\mathcal {n} = \langle \mathcal {o},\mathcal {r},\mathcal {w} \rangle \), in which \(\mathcal {o}\) is the set of objects, \(\mathcal {r}\) is the set of relations between objects, and \(\mathcal {w}\) is the set of weights of these relations. given two pairs of objects \(o_i, o_j \in \mathcal {o}\), the relationship between them is \(r_{o_i,o_j}\) and the weight of a relation \(r_{o_i,o_j}\) is given by \(w_{o_i,o_j} \forall o_i, o_j \in \mathcal {o}\). in \(\mathcal {n}\), objects are news, the relationships between news are created according to the k-nn matrix, and the weights \(w_{o_i,o_j}\) are the cosine similarity between news. although the cosine is a good local similarity measure, its performance as a global similarity measure is not good enough. to consider local paths in the news network that involve neighboring news in common, [9] propose using the katz index. the basic idea is that if two news have many neighbors in common, they are likely to be of the same class. katz index is a global similarity measure that calculates the similarity between pairs of nodes considering all possible paths in a graph that connect them. thus: $$\begin{aligned} sim(d_i,d_j) = \sum ^{\infty }_{h=1} \alpha ^h \cdot |\textit{path}_{d_i,d_j}^{<h>}| = \alpha a_{i,j} + \alpha ^2 (a^2)_{i,j} + \alpha ^3 (a^3)_{i,j} + ..., \end{aligned}$$ (1) in which \(\alpha \) is a parameter that controls the influence of paths. with a very small \(\alpha \), the long paths contribute very little. in the katz index, when \(\alpha < 1/\epsilon \), being \(\epsilon \) the biggest eigenvalue for the matrix a, eq. 1 converges and can be calculated as follows: \(s = (i \alpha a)^{-1} i\), which i denotes the identity matrix and s has dimnesions \(|\mathcal {d}| \times |\mathcal {d}|\). thus, \(s_{i,j} \in \mathbb {r}\) denotes the similarity between the nodes \(d_i\) and \(d_j\) according to katz index (fig. 1 stage 4). the labeled news in \(\mathcal {d}^{+}\) and the similarity matrix s are used to infer two sets: the set of reliable interest news ri and reliable non-interest news rn (fig. 1 stage 5). the set ri contains news from set \(\mathcal {d}^{u}\) that are most similar to examples from \(\mathcal {d}^{+}\), and rn will contain news from \(\mathcal {d}^{u} ri\) that are most dissimilar to the set \(\mathcal {d}^{+} \cup ri\). for the inference of the ri set, an iterative method is applied. the task is divided into m iterations. in each of the m iterations, the total number of reliable interest news that will be extracted is \((\lambda /m) \times |\mathcal {d}^{+}|\), which \(\lambda \) controls the set’s size. the news in \(\mathcal {d}^{u}\) are ranked according to their average similarities for all news in \(\mathcal {d}^{+}\) based on s. the \((\lambda /m)|\mathcal {d}^{+}|\) most similar news are taken from \(d^{u}\) and added into \(ri'\). at the end of each iteration, ri is incremented with the elements in \(ri'\) (\(ri \leftarrow ri \cup ri'\)) [9]. for the inference of the reliable non-interest set, news in \(\mathcal {d}^{u} ri\) are ranked according to their average similarities (based on s) for all news in \(\mathcal {d}^{+} \cup ri\). the algorithm extracts the \(|\mathcal {d}^{+} \cup ri|\) most dissimilar news, forming the set rn. after getting the set rn, the sets \(\mathcal {d}^{+} \cup ri\), rn, and \(\mathcal {d}^{u} \leftarrow (\mathcal {d}^{u} ri rn\)) are used as input by label propagation algorithms based on transductive semi-supervised learning. more details about the algorithm can be seen in [9]. 4.3 adding features in the news network considering that the pu-lp algorithm is entirely based on networks, we propose adding new features as nodes into the news network \(\mathcal {n}\) that has been shown to be relevant in [2, 17] in news classification: incertainty = total number of modal verbs and passive voice non immediacy = total of singular first and second personal pronoun emotiveness (emo) = \(\frac{\text {total number of adjectives + total number of adverbs}}{\text {total number of nouns + total number of verbs}}\) pausality (pau) = \(\frac{\text {total number of punctuation marks}}{\text {total number of sentences}}\) average words sentence (avgws) = \(\frac{\text {total number of words}}{\text {total number of sentences}}\) among the features considered, we compute for each dataset which ones had the highest correlation with the target attribute and appeared in more than one dataset. the characteristics chosen to be included in the network were: pausality, emotiveness, and average words per sentence. table 1 shows the correlations values. the features are added to the news network \(\mathcal {n}\) as new unlabeled nodes \(f_j\), \(0<j<\text {total features}\). new edges between news and the features are added, whose weights \(w_{d_i,f_j}\) correspond to normalized values of the feature \(f_j\) for the news \(d_i\), according to the equation: . we also considered other linguistic features extracted by liwc [10], such as counting personal pronouns, prepositions, verbs, nouns, adjectives and quantitative words, use of affective words, positive and negative emotions, and anxiety, but they did not demonstrated to be relevant in our proposal. table 1. correlations of the main linguistic features considering all news datasets.full size table in addition to the linguistic characteristics, we also considered terms as network nodes since news terms are widely used in researches for discrimination of true and false content. they also demonstrated to be useful in label propagation for text classification [12]. to select terms, stopwords were removed and the terms were stemmed. then, the single terms and sequences of two terms, i.e., unigrams and bigrams, are selected to be network nodes if they respect a minimum document frequency of 2 documents and their term frequency inverse document frequency (tf-idf) value in a document is above a threshold \(\ell \). table 2. list of characteristics included in each proposed heterogeneous networks.full size table twelve different heterogeneous networks were created, considering the combination of all features. table 2 presents the networks and features included in each one. after adding all the features as nodes into the network \(\mathcal {n}\), we perform the normalization of network relations in order to mitigate possible distortions due to different ranges of the different types of relations. the relations are normalized considering each relation type. thus, the edge weight for an object \(o_{i} \in \mathcal {o}_{l}\) and and object \(o_{j} \in \mathcal {o}_m\) is given by eq. 2: $$\begin{aligned} w_{o_i,o_j} = \frac{w_{o_i,o_j}}{\sum _{o_{k}} w_{o_i,o_k}}, o_{i} \in \mathcal {o}_{l}, {o_{j} \in \mathcal {o}_m}, \text{ and } o_{k} \in \mathcal {o}_{m} \end{aligned}$$ (2) after building the news and terms network, the next stage (fig. 1 stage 9) is to carry out the label propagation using transductive learning algorithms for heterogeneous networks. 4.4 label propagation we propose using regularization-based transductive algorithms for heterogeneous networks to classify the unlabeled news as fake or real. the regularization-based algorithms satisfy two premises: (i) the class information of neighboring objects must be similar; (ii) and the class information of the labeled objects assigned during the classification process must be similar to the real class information [12]. the two algorithms considered in this paper are: label propagation through heterogeneous networks (lphn) [12] and gnetmine (gnm) [7]. in order to explain the regularization functions of both algorithms, let \(\mathbf {f}_{o_i} = \{f_{\textit{interest}}, f_{\textit{not-interest}}\}\) be the class information vector, which gives how much an object \(o_i\) belongs to each of the classes, and let \(\mathbf {y}_{o_i}\) be the real class information vector \(\mathbf {f}\), in which the position of the vector corresponding to its class is filled with 1. thus, only labeled objects have values for vector \(\mathbf {y}\). the term \( w_{o_i, o_j}\) is the weight of the edge connecting object \(o_i\) to object \(o_j\), and \(\mathcal {o}^{l}\) refers to the set of labeled objects. the regularization function to be minimized by lphn is given by: $$\begin{aligned} q(\mathbf {f}) = \sum _{\mathcal {d},\mathcal {t} \in \mathcal {o}} \frac{1}{2} \sum _{{o_i} \in \mathcal {o}_i} \sum _{{o_j} \in \mathcal {o}_j} w_{o_i, o_j}(\mathbf {f}_{o_i} \mathbf {f}_{o_j})^2 + \lim _{\mu \rightarrow \infty } \sum _{o_i \in \mathcal {o}^l} (\mathbf {f}_{o_i} \mathbf {y}_{o_i})^2. \end{aligned}$$ (3) we can observe that the different relations have the same importance and that \(\lim _{\mu \rightarrow \infty }\), which forces that \(\mathbf {f}_{i} = \mathbf {y}_{i},\, \forall o_{i} \in \mathcal {o}^{l}\), i.e., the class information of label object do not change. after inferring the \(\mathbf {f}\) vectors of the unlabeled objects, the class mass normalization (cmn) [23], and then the objects are classified according to the \(\arg \)-\(\max \) of the corresponding \(\mathbf {f}\) vector. in gnetmine [7], the relations between different objects have different importance. the label of an object \(o_{i} \in \mathcal {o}^{l}\) can be changed during the classification process if information from neighboring objects diverges from the class of the object initially labeled. the regularization performed by gnetmine is given by: $$\begin{aligned} q(\mathbf {f}) =&\sum _{\mathcal {o}_i,\mathcal {o}_j\subset \mathcal {o}} \lambda _{\mathcal {o}_i,\mathcal {o}_j} \sum _{o_k\in \mathcal {o}_i} \sum _{o_l\in \mathcal {o}j} w_{o_k,o_l} \left| \left| \frac{\mathbf {f}_{o_k}(\mathcal {o}_i)}{\sqrt{\displaystyle \sum _{o_m\in \mathcal {o}_j}w_{o_k,o_m}}} \frac{\mathbf {f}_{o_l}(\mathcal {o}_j)}{\sqrt{\displaystyle \sum _{o_m\in \mathcal {o}_i}w_{o_l,o_m}}} \right| \right| ^2 \nonumber \\&\quad \quad \qquad \qquad \qquad \qquad \qquad \qquad \quad \qquad \;\; +\sum _{o_j\in \mathcal {o}^l}\alpha _{o_j}(\mathbf {f}{o_j}-\mathbf {y}{o_j}) \end{aligned}$$ (4) in which \(\lambda _{\mathcal {o}_i,\mathcal {o}_j} (0 \ge \lambda _{\mathcal {o}_i,\mathcal {o}_j} \ge 1)\) is the importance given to the relationship between objects of the types \(\mathcal {o}_i\) and \(\mathcal {o}_j\), and \(\alpha _{o_i} (0 \ge \alpha _{o_i} \ge 1)\) is the importance given to the real class information of an object \(o_j \in \mathcal {o}^{l}\) (set of labeled objects). the documents are classified considering the arg-max of the final value of \(\mathbf {f}_{d_{i}}\) vectors for \(d_{i} \in \mathcal {d}^{u}\). 5 experimental evaluation in this section, we present the experimental configuration used in the experiments, the datasets and an analysis of the results achieved. our goal is to encourage the study of pul approaches to fake news detection, which perform well using little labeled data. furthermore, we want to demonstrate that with little labeled fake news data, structured in a network, it is possible to achieve reasonable classification performance, which can be improved by adding extra information, taken from the news textual content. table 3. detailed information about news datasets.full size table 5.1 news datasets in this paper, six news datasetsfootnote 1 were evaluated, four of them are in english and two are in portuguese. table 3 presents detailed information about language, subject and amount of real and fake news present in each dataset. fact-checked news (fcn) was collected from five brazilian journalistic fact-checking sites to evaluate our approach. the second portuguese dataset is fake.br (fbr), the first reference corpus in portuguese for fake news detection [17]. the third dataset was acquired from fakenewsnet repository (fnn) [16], which contains news of famous people fact-checked by the gossipcop website. fakenewsnet is the one with the greatest unbalance in the distribution of classes. the last three datasets are also written in english. the news was taken randomly from the fakenewscorpus datasetfootnote 2 (fnc0, fnc1, fnc2), an open source dataset composed of millions of news, collected from 1001 domains. fake and real news were selected, and stopwords were extracted that could induce the behavior of classification algorithms, such as links and names of serious publication vehicles. 5.2 experimental setup and evaluation criteria this section presents the experiment configuration and evaluation criteria for the pu-lp and the baseline algorithms. after pre-processing, feature vectors for news representation were obtained considering two strategies: (i) a traditional bow, with tf-idf as the term-weighting scheme; and (ii) d2v (paragraph vectors). in d2v we used the union of the models distributed memory and distributed bag-of-words to generate the document embeddings. for training each of these models, we consider the average and concatenation of the word vectors to create the hidden layer’s output. also, we employed the range of the maximum number of epochs \(\in \{100, 1000\}\), \(\alpha = 0.025\) and \(\alpha _{min} = 0.0001\), number of dimensions of each model \(= 500\), window size \(= 8\), and minimum count \(= 1\) [8]. for the k-nn matrix used in pu-lp, we used \(k = [5,6,7]\) and cosine as similarity measure. for the extraction of reliable interest and non-interest sets, we use: \(m = 2\), \(\lambda = [0.6, 0.8]\), and \(\alpha = [0.005, 0.01,0.02]\). these values were chosen as suggested in [9]. for the selection of representative unigrams and bigrams, we use \(\ell = 0.08\). the parameter \(\ell \) was chosen after a statistical analysis of the sample, indicating that about 25% of bag-of-words terms had tf-idf greater than 0.08. for the label propagation stage, as suggested in [14], we used a convergence threshold = 0.00005 and a maximum number of iterations = 1,000. for the gnm \(\alpha = \{0.1, 0.5\}\) and \(\lambda = 1\) was considered. a 10-fold cross-validation adapted to ocl and pul problems was used as a validation scheme. in this case, the set of fake news (\(\mathcal {d}^{+}\)) was randomly divided into 10 folds. in order to simulate a semi-supervised learning environment, in which the number of labeled examples is higher than the unlabeled examples, i.e., \(|\mathcal {d}^{+}| \gg |\mathcal {d}^u|\), we carried out different experiments considering as labeled data 1, 2 or 3 folds. the remaining folds and the real news are: (i) considered as test documents for the ocl algorithm; (ii) considered as unlabeled and test documents for the pul algorithms. we proposed a baseline approach using binary semi-supervised learning to assess the labeling procedure of reliable-interest and reliable-non-interest examples in pu-lp. for this analysis, the set of real news was randomly divided into ten subsets. in the cross-validation scheme, for each fake news fold used to train the algorithm, one fold of real news was used. from the network obtained by \(\mathcal {n}\) in pu-lp, and considering the training set as the set of labeled nodes, label propagation algorithms infer the class of the remaining news from the network. we considered the values of k ranging in the interval [5, 7]. the propagation algorithms and their respective configurations are the same as the experiments performed with pu-lp. as evaluation measure, we used \(f_{1} = (2 \cdot \textit{precision} \cdot \textit{recall})/(\textit{precision}+\textit{recall})\) considering the fake news as the positive class (interest-\(f_1\))footnote 3. in the next section, the results of the experiments are presented. 5.3 results and discussions tables 4 and 5 present the the best results for the interest-\(f_{1}\) that each network reached considering bow and d2v representation models for both pu-lp and binary semi-supervised baseline, the experimental configuration defined in sect. 5.2, and the news datasets. due to space limitations, we present the results obtained only with the gnm algorithm, which obtained better overall performance. the complete table of results is available in our public repository. 10%, 20%, and 30% indicate the percentage of news used to train the algorithms. networks 1 to 12 have a combination of the features proposed in sect. 4.3 (see table 2). table 6 presents the average ranking and standard deviation analysis of the proposed heterogeneous networks. table 4. interest-\(f_{1}\) of the pu-lp and binary baseline approaches with bag-of-words representation model, using gnetmine as label propagation.full size table on network 1 (net 1), built only with news, we can see that overall pu-lp behaves better than the binary baseline, even using half of the labeled news. when exceptions occur, the results obtained tend to be very close, with less than 2% of difference. this demonstrates that identifying reliable fake news and inferring a real news set using the initially interest labeled set and katz index, can be a promising strategy for news classification. in particular, using only 10% of labeled fake news and representation models that consider semantic relations such as d2v, can provide results as efficient as using larger training sets, which favors the dynamic scenario of news classification. including terms (net 2) in the news network tends to improve classification performance in general, especially when news are represented with bow (table 4 and table 5). table 6 also shows that net 2 obtained a better average ranking, with a low standard deviation, in relation to the other networks proposed. for bow with 20% and 30% labeled data, including also pausality achieved better results (net 7). the term selection considering the tf-idf measure above a threshold seems to contribute to distinguishing real and fake news by propagation algorithms, increasing the algorithm’s accuracy. table 5. interest-\(f_{1}\) of the pu-lp and binary baseline approaches with doc2vec representation model, using gnetmine as label propagation.full size table among the news datasets, only the fnn is unbalanced. fnn has only celebrity news, of which fake news is a minority making up only 24.3% of the dataset. for this dataset, the interest-\(f_{1}\) always tends to be lower. however, we can notice that our proposal performed better than the binary baseline, which also points out the utility of pul for fake news detection. lower results also occur in the fbr datasets. this behavior also happened in [5], in which 67% of the interest-\(f_1\) was obtained using 90% labeled fake news and one-class learning algorithms. the hypothesis is that since this collection in composed of 6 different subjects, fake news from different subjects are spread along with the space, making it difficult to infer reliable negative documents. therefore, we conclude that to achieve more promising results, the best way is to classify the news considering one subject at a time. although the results achieved with these datasets are inferior, we can observe that the addition of extra information in the heterogeneous network was beneficial, especially for the baseline, which increases its interest-\(f_{1}\) by more than 10%. pu-lp obtained great results for fcn, fnc0, fnc1 and fnc2 news datasets. using d2v as the representation model, the results ranged from 87 to 92% interest-\(f_{1}\) only with news and terms on the network. these results are relevant, mainly because they consider an initially small set of labeled data, different from the vast majority of methods proposed in the literature for news classification. table 6. average ranking and standard deviation analysis for the proposed heterogeneous networks.full size table 6 conclusion and future work this paper proposed a new approach for detecting fake news based on pu-lp. our main contributions were a heterogeneous network using content-based features resulting in a domain-independent fake news detection approach and a performance assessment of different linguistic features for fake news detection. pu-lp achieves good classification performance using a low amount of labeled data from the interest class and uses unlabeled data to improve classification performance, minimizing the news labeling effort. furthermore, since pu-lp is a pul approach based on network, it allows the easy incorporation of other features to perform learning. we assessed the performance of a heterogeneous network that incorporates combinations of four different features based on terms and linguistic characteristics. the results of our experimental analysis in six datasets from different languages and subjects indicate that our proposal can provide good classification performance even with a small amount labeled fake news. incorporating additional linguistic information such as representative terms and pausality into the network improved classification performance. our approach achieved performance similar to a binary classifier in all datasets, even requiring much lesser effort for data labeling. future work intends to consider more powerful representation models, such as context-based, and use more efficient label propagation algorithms, such as those based on neural networks. we also intend to apply our approach to new datasets, which will allow us to perform statistical significance tests to better assess the results. notes 1.all datasets and source codes used in this paper are available in our public repository: https://github.com/marianacaravanti/a-heterogeneous-network-based-positive-and-unlabeled-learning-approach-to-detecting-fake-news. 2.https://github.com/several27/fakenewscorpus. 3.we also evaluate the average of \(f_1\) for both classes (macro-averaging \(f_{1}\)). due to space limitations, the complete results are available in our public repository: https://github.com/marianacaravanti/a-heterogeneous-network-based-positive-and-unlabeled-learning-approach-to-detecting-fake-news/tree/main/results. references aggarwal, c.c.: machine learning for text. springer publishing (2018). https://doi.org/10.1007/978-3-319-73531-3 vargas, f.a., pardo, t.a.s.: studying dishonest intentions in brazilian portuguese texts. arxiv e-prints (2020) google scholar  bekker, j., davis, j.: learning from positive and unlabeled data: a survey. mach. learn. 109(4), 719–760 (2020). https://doi.org/10.1007/s10994-020-05877-5 article  mathscinet  math  google scholar  bondielli, a., marcelloni, f.: a survey on fake news and rumour detection techniques. inf. sci. 497, 38–55 (2019) article  google scholar  faustini, p., covões, t.f.: fake news detection using one-class classification. in: 2019 8th brazilian conference on intelligent systems, pp. 592–597. ieee (2019) google scholar  greifeneder, r., jaffe, m., newman, e., schwarz, n.: the psychology of fake news: accepting, sharing, and correcting misinformation. routledge, milton park (2021) google scholar  ji, m., sun, y., danilevsky, m., han, j., gao, j.: graph regularized transductive classification on heterogeneous information networks. in: european conference on machine learning and knowledge discovery in databases, pp. 570–586 (2010) google scholar  le, q., mikolov, t.: distributed representations of sentences and documents. in: international conference on machine learning, pp. 1188–1196 (2014) google scholar  ma, s., zhang, r.: pu-lp: a novel approach for positive and unlabeled learning by label propagation. in: 2017 ieee international conference on multimedia & expo workshops (icmew), pp. 537–542. ieee (2017) google scholar  pennebaker, j.w., boyd, r.l., jordan, k., blackburn, k.: the development and psychometric properties of liwc2015. university of texas, technical report (2015) google scholar  ren, y., wang, b., zhang, j., chang, y.: adversarial active learning based heterogeneous graph neural network for fake news detection. in: 2020 ieee international conference on data mining (icdm), pp. 452–461. ieee (2020) google scholar  rossi, r.g.: automatic text classification through network-based machine learning. ph.d. thesis, university of são paulo, doctoral thesis (2016). (in portuguese) google scholar  santos, r.l.s., pardo, t.a.s.: fact-checking for portuguese: knowledge graph and google search-based methods. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 195–205. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_19 chapter  google scholar  santos, b.n.: transductive classification of events using heterogeneous networks. master’s thesis federal university of mato grosso do sul (2018). (in portuguese) google scholar  heterogeneous information network analysis and applications. da. springer, cham (2017). https://doi.org/10.1007/978-3-319-56212-4_9 shu, k., mahudeswaran, d., wang, s., lee, d., liu, h.: fakenewsnet: a data repository with news content, social context, and spatiotemporal information for studying fake news on social media. big data 8(3), 171–188 (2020) article  google scholar  silva, r.m., santos, r.l., almeida, t.a., pardo, t.a.: towards automatically filtering fake news in portuguese. expert syst. appl. 146, 113–199 (2020) article  google scholar  singh, v.k., ghosh, i., sonagara, d.: detecting fake news stories via multimodal analysis. assoc. inf. sci. technol. 72(1), 3–17 (2021) article  google scholar  yu, j., huang, q., zhou, x., sha, y.: iarnet: an information aggregating and reasoning network over heterogeneous graph for fake news detection. in: 2020 international joint conference on neural networks (ijcnn), pp. 1–9. ieee (2020) google scholar  yu, s., li, c.: pe-puc: a graph based pu-learning approach for text classification. in: perner, p. (ed.) mldm 2007. lncs (lnai), vol. 4571, pp. 574–584. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-73499-4_43 chapter  google scholar  zhang, j., dong, b., philip, s.y.: deep diffusive neural network based fake news detection from heterogeneous social networks. in: big data 2019: international conference on big data, pp. 1259–1266. ieee (2019) google scholar  zhang, x., ghorbani, a.a.: an overview of online fake news: characterization, detection, and discussion. inf. process. manage. 57(2), 102025 (2020) google scholar  zhu, x., ghahramani, z., lafferty, j.d.: semi-supervised learning using gaussian fields and harmonic functions. in: proceedings of the 20th international conference on machine learning (icml-2003), pp. 912–919 (2003) google scholar  download references author information authors and affiliations department of computer science, university of são paulo, são carlos, brazil mariana c. de souza, ricardo m. marcacini & solange o. rezende federal university of mato grosso do sul, campo grande, brazil bruno m. nogueira federal university of mato grosso do sul, três lagoas, brazil rafael g. rossi authors mariana c. de souzaview author publications search author on:pubmed google scholar bruno m. nogueiraview author publications search author on:pubmed google scholar rafael g. rossiview author publications search author on:pubmed google scholar ricardo m. marcaciniview author publications search author on:pubmed google scholar solange o. rezendeview author publications search author on:pubmed google scholar corresponding author correspondence to mariana c. de souza . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de souza, m.c., nogueira, b.m., rossi, r.g., marcacini, r.m., rezende, s.o. (2021). a heterogeneous network-based positive and unlabeled learning approach to detect fake news. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_1 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords fake news one-class learning positive and unlabeled learning transdutive semi-supervised learning graph-based learning publish with us policies and ethics profiles ricardo m. marcacini view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature modeling eeg data into graphs for the prognostic of patients in coma using graph neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper modeling eeg data into graphs for the prognostic of patients in coma using graph neural networks conference paper first online: 31 january 2025 pp 223–237 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) modeling eeg data into graphs for the prognostic of patients in coma using graph neural networks download book pdf download book epub odelmo o. nascimento junior  orcid: orcid.org/0000-0001-8360-56009, dhara l. c. assis  orcid: orcid.org/0000-0002-3840-55949, joão b. destro filho  orcid: orcid.org/0000-0001-5306-80589, liang zhao  orcid: orcid.org/0000-0002-1502-660410 & … murillo g. carneiro  orcid: orcid.org/0000-0002-2915-89909,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 232 accesses abstract in this work we present a modeling approach using graph neural networks (gnns) to support the prognosis of patients in coma (ppc) based on electroencephalogram (eeg) exams. coma is a critical medical condition characterized by an extended state of unconsciousness. eeg is an exam that captures the electrical activity of the brain through electrodes positioned on the patient’s scalp. it provides valuable information about cerebral functions and can also be used for the monitoring and diagnosis of several diseases. in this study, we investigate four gnn architectures specifically designed for ppc analysis, leveraging the power of convolutional-based layers. in addition, we also develop a modeling strategy to represent eeg sequential data into a graph, named eegraph, effectively harnessing the spatial positions of electrodes. the main contribution of representing the eeg exam in a graph structure is also to explore all the inherent spatial information the exam has. as it is a highly sensitive exam with generally noisy results on the micro-volt scale, its spatial exploration may reveal the potential to improve the ppc task. experimental results considering real data showed that our proposed architecture outperformed the lstm model in terms of f1-score and accuracy. statistical tests evidenced the enhanced predictive performance achieved by exploiting the spatial aspects of eeg data using gnns. moreover, the findings highlight the significant potential of gnns to contribute to the ppc task from eeg data. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a graph theory analysis on distinguishing eeg-based brain death and coma chapter © 2017 eeg models and analysis chapter © 2021 eeg models and analysis chapter © 2023 1 introduction coma is a serious medical condition characterized by a prolonged state of unconsciousness in which an individual has little or no level of response [9, 10]. the electroencephalogram (eeg) is a medical exam that can record the brain’s electrical activity. it represents the summation effect of many excitatory and inhibitory postsynaptic potentials produced in the pyramidal layer of the cerebral cortex [29]. data from this exam are commonly used for monitoring and diagnosing various diseases, some of which are schizophrenia [24] and epilepsy [26]. the adoption of eeg data for disease detection tasks often considers well-known machine learning (ml) algorithms such as adaptive boosting, support vector machine and tree-based algorithms that compare portions of the exam for one or more electrodes [17, 22, 28]. to enable the application of ml models to examination data, traditional approaches involve the application of various filters to all channels to extract spectral attributes [5]. subsequently, these attribute vectors are formed and processed through the cited algorithms. in addition to disease-related monitoring and diagnosis, eeg data can be used for behavioral and emotional analysis of patients. some studies show that it is possible to obtain good results in predicting emotions applying feature extraction methods by examining time series followed by the application of ml models [20, 27]. these studies performed good results in predicting emotions by combining a series of attributes extracted from the time series resulting from the eeg examination, most of the time combining attributes from the time and frequency domains. to improve the results, other studies combine time series attributes with the result of facial expression detection [14, 30]. other studies show the impact that a musical stimulus can bring to the brain activity behavior [15, 25], and even the relationship between these musical stimuli with the behavioral recovery of comatose patients [7]. research on the therapeutic potential of musical stimulation in coma patient recovery often employs a comparative approach by analyzing the temporal signal from each electrode before and after the stimulus. these comparisons involve the extraction of statistical variables from the time series data. additionally, there are applications of ml models to comprehensively assess the impact generated by such stimuli on patients’ neurological responses and recovery trajectories [23]. most approaches to eeg analysis involve extracting information from descriptive statistical variables or using simple signal pre-processing methods [18, 19]. another well-known approaches include recurrent neural networks (rnn), such as echo state networks and long short-term memory (lstm) [1, 3, 4, 21, 25, 32, 33]. lstm is particularly effective for handling temporal data and managing relationships between states. in eeg analysis tasks, lstm often includes a regularization layer during the training process. this architectural choice enhances the model’s ability to accurately interpret and analyze the examination results. lstms yield satisfactory results when applied to eeg exams for the aforementioned applications. however, they miss a significant opportunity to explore the scalp position of each electrode and the relationships between channels. graph neural networks (gnns) have the potential to work with graph-structured data. recent works on applying gnns to emotion prediction and other general eeg classification tasks [8, 34] have obtained promising results. besides such positive outcomes, gnns can also explore connections involving the spatial positioning of the electrodes (nodes) and the correlations between the graph data features, which is a few explored topic in the literature. in this work, we investigate four gnns architectures for the prognosis of patients in coma (ppc) using eeg data. our graph classification approach incorporates convolutional-based layers specifically designed for graph data, allowing the exploration of spatial relationships between electrodes due to the representation of the eeg examination in the form of a graph structure. in order to represent eeg sequential data into graphs, we also develop eegraph, a modeling strategy in which each vertex in the graph corresponds to an electrode, and the edges are constructed from the electrodes position in the scalp. while the ppc task is continuously evolving, we recognize certain opportunities for improvement. for example, the topological structure of eeg channels is often underutilized in the formulation of eeg representations [2]. to address this, our study focuses on leveraging the graph structure to uncover more informative patterns and relationships among the electrodes, leading to enhanced prognostic accuracy. by addressing such a challenge, we aim to contribute in the ppc task from eeg signals, an approach compliant with most of the brazilian public hospitals in terms of cost and equipment availability. our experiments with real data shows that both the proposed gnns architectures and the graph modeling strategy improve the predictive results regarding the outcome of comatose patients. in summary, our main contributions can be stated as follows: development of four gnns architectures based on convolutional dynamics to predict ppc outcomes. the proposed architectures aim to explore both nonlinear transformations and local connections of the vertices. design of eegraph, a modeling strategy to transform raw eeg data into graph structures. eegraph seeks to exploit the connections between the vertices (electrodes) of graph structures as much as possible. development of an experimental setup to compare the proposed architectures against lstm, and provide some evidence about their satisfactory results. the remainder work is divided as follows: sect. 2 presents the background for the understanding of the proposed architectures. section 3 describes the proposed approaches. section 4 shows the experimental results, and sect. 5 brings the conclusion of the work as well as forthcoming investigations. 2 background this section presents a brief description of the mathematical concepts related to the gnn and lstm layers investigated in this paper. 2.1 graph neural networks for a better understanding of gnns and their respective convolutional layers used in this work, it is important to reiterate the graph data structure definition in which these networks operate. a graph \( \mathcal {g = (v, e)} \) is the data representation formed by connecting vertices through edges, which are denoted respectively by \( \mathcal {v} \) and \( \mathcal {e} \). each edge can contain a weight value \( \mathcal {w} \). an adjacency matrix \( \mathcal {a} \) can be used to represent an overview of the final structuring of the graph. graph convolutional network (gcn) have their layer propagation behavior governed by the following equation [16]: $$\begin{aligned} h^{(l + 1)} = \sigma ( \tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}} h^{(l)} w^{(l)})~, \end{aligned}$$ (1) with the variables related to a certain undirected graph \( \mathcal {g} \), where \( \tilde{a} \) is the adjacency matrix, \( \tilde{d} \) is the diagonal matrix of \( \tilde{a}\) governed by \(\tilde{d}_{ii} = \sum _{j} \tilde{a}_{ij}\), \( w ^l\) is the matrix of weights of neural network in layer l, \(\sigma ()\) function represents the activation functions at \(l^{th}\) layer, and finally \(h^{l + 1}\) is the resulting matrix at the end of \( l^{th}\) layer, where \(h^{0}\) is the initial inputs, called x, and \(h^{n}\), with n as the last layer, contains the results in the final step of our learning process, called y. equation (1) is the final representation of the summary equations of the three stages of gcn before the prediction, which are: feature propagation, linear transformation and nonlinear activation. the feature propagation stage can be updated for the entire graph represented by: $$\begin{aligned} \tilde{h}^{(l)} \leftarrow \tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}} h^{(l 1)}~. \end{aligned}$$ (2) after the first stage, eq. (2) will be combined with the weight matrix \( w ^l\) for each iteration, resulting in the linear transformation stage. finally, the function \(\sigma ()\) can be applied to complete the process: $$\begin{aligned} h^{(l)} \leftarrow \sigma ( \tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}} h^{(l-1)} w^{(l)})~. \end{aligned}$$ (3) at the end, gcn prediction is performed by applying a softmax layer in order to normalize the results and weights obtained during the update process. simple graph convolution (sgc) differs from gcn by the removal of the non-linear transformations sequence between the layers of the gcn structure and by exploring more the feature propagation stage [31]. the feature propagation stage for the sgc has as main objective to explore the potential of the relations between the neighboring vertices via configuration of the adjacency matrix. therefore, eq. (2) previously used in gcn layers can be rewritten for use in sgc architectures as follows: $$\begin{aligned} \tilde{h}^{(l)} \leftarrow (\tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}})^{l} h^{(0)}~. \end{aligned}$$ (4) from eq. (4), the product by the matrix of weights \( w ^l\) results is given by: $$\begin{aligned} \tilde{h}^{(l)} \leftarrow (\tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}})^{l} h^{(0)} w^{(l)}~. \end{aligned}$$ (5) to obtain the normalized value of the sgc predictions for the classes, a softmax function is applied over eq. (5). sample and aggregate convolutional network (sage) introduces an approach for aggregating feature information from a node’s local neighborhood, enhancing the representation learning for graph-structured data. the propagation behavior of the sage layer is given by eqs. (6) and (7), which collectively describe the sampling and aggregation processes integral to the model [12]. the first step in the sage layer involves sampling a fixed-size set of neighbors for each node, which can be described by: $$\begin{aligned} \mathcal {n}(v) = \text {sample}(\tilde{\mathcal {n}}(v), k)~, \end{aligned}$$ (6) where \(\mathcal {n}(v)\) represents the set of neighbors sampled for node \( v \), \(\tilde{\mathcal {n}}(v)\) is the full set of neighbors of \( v \), and \( k \) is the number of neighbors to sample. once the neighbors are sampled, the next step is to aggregate their features using an aggregation function \( \text {agg} \), which can be mean, lstm-based, or pooling-based. the aggregated feature vector is then concatenated with the node’s own feature vector and transformed through a weight matrix and non-linear activation function, as follows: $$\begin{aligned} h_v^{(l+1)} = \sigma \left( w^{(l)} \cdot \text {concat} \left( h_v^{(l)}, \text {agg} \left( \{ h_u^{(l)}, \forall u \in \mathcal {n}(v) \} \right) \right) \right) ~, \end{aligned}$$ (7) in which \( h_v^{(l)} \) denotes the feature vector of node \( v \) at layer \( l \), \( \text {concat} \) is the concatenation operation, \( \text {agg} \) represents the chosen aggregation function, \( w^{(l)} \) is the weight matrix for layer \( l \), and \( \sigma \) is the activation function. the final node representations can be used for downstream tasks such as node classification or link prediction, often incorporating a softmax layer for normalization and classification purposes. graph convolutional network ii (gcnii) extends the traditional gcn by introducing initial residual and identity mapping, which address the limitations of deeper gnns. the propagation behavior of gcnii is governed by eqs. (8) and (9), which incorporate these novel characteristics [6]. the initial residual connection allows each layer to directly utilize the input features of the first layer, enhancing information flow and mitigating the vanishing gradient problem. from eq. 1, this can be defined as follows: $$\begin{aligned} h^{(l+1)} = \sigma \left( (1 \alpha ) \left( \tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}} h^{(l)} w^{(l)} \right) + \alpha h^{(0)} \right) ~, \end{aligned}$$ (8) where \(\alpha \) is a hyperparameter controlling the balance between the initial residual connection and the current layer’s transformation. the identity mapping characteristic further stabilizes the training of deep graph networks by incorporating identity mappings, which preserve the information from the previous layer, such as follows: $$\begin{aligned} h^{(l+1)} = \sigma \left( (1 \alpha ) \left( \tilde{d}^{-\frac{1}{2}} \tilde{a} \tilde{d}^{-\frac{1}{2}} h^{(l)} w^{(l)} \right) + \alpha h^{(0)} + \beta h^{(l)} \right) ~, \end{aligned}$$ (9) where \(\beta \) is another hyperparameter that controls the contribution of the identity mapping term \(h^{(l)}\), further enhancing the model’s capacity to maintain feature representation throughout layers. the final node representations generated by gcnii typically involves a softmax layer for normalization and classification purposes. 2.2 long short-term memory lstm is a type of rnn architecture designed to address the vanishing gradient problem and capture long-term dependencies in sequential data [11, 13]. lstm uses specialized memory units to store and retrieve information selectively. unlike traditional rnns, lstm has a stronger memory mechanism that allows it to retain information for longer periods and overcome the vanishing gradient problem. the lstm model adopted for comparison purposes in the present work consists of five key functions [11]: 1. forget gate: denoted by \(f_t\), it computes the amount of previously stored information to forget. it is calculated by applying a sigmoid activation function (\(\sigma \)) to the linear transformation of the concatenated input (\([h_{t-1}, x_t]\)) using the weight matrix (\(w_f\)) and bias vector (\(b_f\)). $$\begin{aligned} f_t = \sigma (w_{xf}x_t + w_{hf}h_{t-1} + w_{cf}c_{t-1} + b_f) \end{aligned}$$ (10) 2. input gate: denoted by \(i_t\), it determines the amount of new information to be stored in the current memory state. it is calculated by applying a sigmoid activation function (\(\sigma \)) to the linear transformation of the concatenated input (\([h_{t-1}, x_t]\)) using the weight matrix (\(w_i\)) and bias vector (\(b_i\)). $$\begin{aligned} i_t = \sigma (w_{xi}x_t + w_{hi}h_{t-1} + w_{ci}c_{t-1} + b_i) \end{aligned}$$ (11) 3. candidate cell state: denoted by \(\widetilde{c}t\), it represents the new information that can be added to the memory. it is calculated by applying a hyperbolic tangent activation function to the linear transformation of the concatenated input (\([h{t-1}, x_t]\)) using the weight matrix (\(w_c\)) and bias vector (\(b_c\)). $$\begin{aligned} \widetilde{c}t = \tanh (w_{xc}x_t + w_{hc}h_{t-1} + b_c) \end{aligned}$$ (12) 4. cell state update: it combines the forget gate activation vector (\(f_t\)), the input gate activation vector (\(i_t\)), and the candidate cell state (\(\widetilde{c}_t\)) to update the current memory state (\(c_t\)). element-wise multiplication (\(\odot \)) is performed between the forget gate and the previous memory state, and between the input gate and the candidate cell state. $$\begin{aligned} c_t = f_tc_{t-1} + i_t\widetilde{c}_t \end{aligned}$$ (13) 5. output gate: denoted by \(o_t\), it controls the amount of information to be output from the memory state. it is calculated by applying a sigmoid activation function (\(\sigma \)) to the linear transformation of the concatenated input (\([h_{t-1}, x_t]\)) using the weight matrix (\(w_o\)) and bias vector (\(b_o\)). $$\begin{aligned} o_t = \sigma (w_{xo}x_t + w_{ho}h_{t-1} + w_{co}c_t + b_o) \end{aligned}$$ (14) the lstm model involves various parameters, including weight matrices (\(w_f\), \(w_i\), \(w_c\), \(w_o\)), bias vectors (\(b_f\), \(b_i\), \(b_c\), \(b_o\)), and the hidden state \(h_t\). these parameters are learned during the training process and play a crucial role in the network’s ability to capture and model complex temporal dependencies. 3 model description this section provides a detailed description of all the components developed to enable the gnn learning over eeg data. we also present the dataset adopted in this study, the formulation and pseudocode of the modeling approach to represent eeg data into graphs, the selection of hyper-parameters for the gnn networks, and the validation process employed. an overview about the complete pipeline of the model can be see in the fig. 1, and all its respective steps will be detailed in this section. fig. 1. complete pipeline fluxogram. (a) electroencephalogram before preprocessing step; (b) eeg exam as a graph, with each electrode’s respective temporal values as vertices, and the patient’s actual outcome stored as a target class of the graph; (c) graph structure after modeling the exam data: each electrode is a node, and all the edges have the same weights (w = 1); (d) graph neural network representation, created based on the convolutions principles; and (e) the final result (outcome probability) of the pipeline. full size image 3.1 eeg dataset the dataset used in this study is private. with approval of the research ethics committee of the federal university of uberlândia (ufu), eeg exams recorded from comatose patients in the intensive care unit of the ufu clinical hospital were collected. table 1 shows the class distribution. eeg exams considered the 10–20 electrode system, which the disposition along the scalp is shown by fig. 2. after the preprocessing, each eeg exam is denoted by 10 two-second segments selected through visual inspection by a specialized physician in the field. along with the results, the datasets were labeled regarding the outcome of the comatose patient, considering the following groups: favorable outcome is related to patients with positive outcome. unfavorable outcome is related to patients with clinical or brain death; table 1. outcomes final class distributionfull size table fig. 2. electrodes position on patients’ cortex. full size image eeg data were collected at a frequency of 100 to 400 hz for the given electrodes [fp1, fp2, f7, f3, fz, f4, f8, t3, c3, cz, c4, t4, t5, p3, pz, p4, t6, o1, o2]. to address the possible difference in the frequency at which data were collected, the amount of data collected for 400 hz was used as reference. 3.2 eegraph: modeling eeg data into graphs as illustrated by steps figs. 1(b) and 1(c), we develop eegraph, a modeling strategy to represent eeg exam data into graphs. the graph’s data structure, as seen in the background section, requires the definition of vertices and edges in order to leverage the power of gnns frameworks. therefore, eegraph is divided in two main steps: node and edge construction. in the node construction step, each electrode is denoted by a graph vertex (\( \mathcal {v} \)), whose attributes contain their respective temporal eeg records. figure 1(b) illustrates the step in eegraph responsible for modeling the eeg records for each electrode to vertices in the graph structure. at this stage, the temporal values for each electrode are assigned to the respective vertices in the form of an attribute vector, and simultaneously, the respective patient outcomes are also assigned as the target class of the final graph data structure. in the edge construction step, a matrix p, defined by (15), is built as a reference for the creation of the edges between nodes (electrodes), as exemplified in fig. 1(c). all links were created by considering the electrodes positions along the scalp, as shown in fig. 2, to effectively leverage the spatial relationships between the nodes and represent them as connections. the padding elements, denoted by \(\times \) in (15), were used only as a facilitator in the visual interpretation and also for the implementation of a neighboring heuristic described next. although they are not directly present in the construction of the graph structures, they have a great importance enabling the eegraph strategy not only to have a positional reference for each vertex in the edge construction process, but also to define different levels of neighboring relationships. $$\begin{aligned} p = \begin{bmatrix} \times & f_{p1} & \times & f_{p2} & \times \\ f_{7} & f_{3} & f_{z} & f_{4} & f_{8} \\ t_{3} & c_{3} & c_{z} & c_{4} & t_{4} \\ t_{5} & p_{3} & p_{z} & p_{4} & t_{6} \\ \times & o_{1} & \times & o_{2} & \times \\ \end{bmatrix} \end{aligned}$$ (15) we enhance our eegraph modeling strategy by designing an edge construction heuristic able take advantage of different levels of neighboring relationships among the nodes. such a heuristic is named h-eegraph, in which h defines the maximum geodesic distance to be considered in the creation of adjacent edges. figure 3 shows the graph constructed for 1-eegraph, in which only adjacent vertices are connected, ignoring the padding elements in matrix p. figure 4 shows the connection of the \(f_{p1}\) electrode/node considering one, two and three levels of neighborhood by setting maximum geodesic distance h respectively equal to 1, 2 and 3. formally, the edge construction step is given by: $$\begin{aligned} a(v_i,v_j) = {\left\{ \begin{array}{ll} 1, & \text {if } d(v_i, v_j) \le h~,\\ 0, & \text {otherwise,} \end{array}\right. } \end{aligned}$$ (16) in which a refers to the adjacency matrix obtained from our h-eegraph modeling, and \(d(v_i, v_j)\) is the geodesic distance between nodes \(v_i\) and \(v_j\). fig. 3. electrodes structured as a graph. full size image fig. 4. example of the fp1 channel method connections. full size image 3.3 neural networks techniques the gnns frameworks investigated in this study are gcn [16], sgc [31], sage [12] and gcnii [6]. for comparative purposes, we considered a lstm network [33], a widely adopted technique for sequential data analysis. section 2 presents a brief description about each one of the techniques under comparison. to be specific, all gnns models considered three convolutional layers. in the sage layers architecture, we adopted the mean aggregation function, and for the gcnii layers architecture, we adopted 0.1 and 0.5 values for the \(\alpha \) and \(\beta \) hyperparameters, respectively. they were trained with a number of 100 epochs, a batch size of 64 and adaptive movement estimation (adam) optimizer with a learning rate of \(10^{-3}\). the parameters and structures of the created gnns were similarly designed to ensure a fair comparison between them. regarding lstm architecture parameters, we identified the optimal configuration, which consisted of a single lstm layer with 20 neurons, trained over 100 epochs, yielding the best predictive performance. table 2. performance comparison: metric value/standard deviationfull size table 4 experimental results in order to ensure a reliable and fair comparison between the results generated by the network architectures used, the k-fold stratified method was applied. the dataset was partitioned into 10 folds maintaining proportionality between the classes, using 8 partitions for training, 1 for validation and 1 for test. table 1 made explicit a considerable imbalance between the number of registers for each outcome, a factor of great weight for the choice of metrics presented in the comparison of table 2. to deal with this imbalance, f1-score was used as the most important metric to both selection of hyper-parameters and evaluation of the predictive performance of the neural networks’ (nn for short in the table) architectures. the indicated recall and precision values were chosen to facilitate the understanding of the existing trade-off in each architecture, making evident any model fit problems with the data. and finally, the choice of accuracy was made so that we could always compare with the reference f1-score, ensuring that the balancing of the metrics was obeyed and that there were no problems related to the adaptation of the models to the class imbalance. table 3. statistical analysis using the wilcoxon test to evaluate the best configuration among the combinations of gnns and eegraph neighborhood levels. adopting a confidence level of 95% (\(\alpha \) = 0.05), the symbols >, <, and \(\approx \) indicate that the variable in the column is respectively better, worse, or equivalent to the variable in the row. inner h=4 results omitted without affecting the analysis.full size table the results obtained in the table 2 shows a significant improvement for all gnns proposed architectures in the target metric f1-score when compared to the reference lstm architecture. in addition to a significant improvement of almost 18% in the target metric of the study and a better balance between precision and recall, a better fit of the gnns architectures compared to the lstm is also noticeable since the precision of the lstm network has signs of bias. the eegraph strategy for the graph structure is shown in the table according to the number of neighborhood levels (h), considered from 1 to 4. with exception of gcnii, smaller neighborhood levels like 1 and 2 usually provide better results in terms of the target metric f1-score, as well as in terms of accuracy and recall. this means that a strategy to increment the number of edges based solely on neighborhood levels, without considering weight differences during the edge construction steps, does not contribute to an enhancement of network learning for the given problem. on the other hand, it also indicates that the relationships among adjacent electrodes contribute decisively to achieve better predictive performance. table 3 shows the wilcoxon statistical test between all the architectures used in the experiment, making explicit the superiority of all gnns techniques in comparison to lstm. the results of the sgc algorithm with h=1 are statistically superior to all other gnns and eegraph configurations. for h=2 and h=3, sgc and gcn are statistically equivalent and outperform all other architectures, demonstrating the potential of both techniques. table 4. statistical analysis using the wilcoxon test to evaluate the models regardless of the eegraph modeling strategy. adopting a confidence level of 95% (\(\alpha \) = 0.05), the symbols >, <, and \(\approx \) indicate that the variable in the column is respectively better, worse, or equivalent to the variable in the row.full size table table 5. statistical analysis using the wilcoxon test to evaluate the eegraph modeling strategy, regardless of the model architecture. adopting a confidence level of 95% (\(\alpha \) = 0.05), the symbols >, <, and \(\approx \) indicate that the variable in the column is respectively better, worse, or equivalent to the variable in the row.full size table when comparing the models regardless of the h-eegraph architecture, observed in table 4, the sgc model consistently outperforms gcn, sage, and gcnii, indicating it is the most effective model in this context. gcn also performs well, surpassing sage and gcnii. sage shows better performance than gcnii but is outperformed by both gcn and sgc. in addition to the previous statistical analysis, one can see in table 2 that gcnii model has very lower predictive performance in comparison to all other gnns. it is important to note that we used only a single hyperparameters configuration for the gcnii architecture, which could be a potential limitation in fully evaluating its performance, as it may indicate an stronger dependence about the choice of \(\alpha \) and \(\beta \) hyperparameters, related respectively to the initial residual and identity mapping contributions. because of that, we remove gcnii results from our next statistical analysis, as we are interested in identifying the contribution of different neighborhood levels in our eegraph modeling strategy. table 5 shows that one level neighborhood (h=1) outperforms the other neighborhood configurations across gcn, sgc and sage gnns. conversely, the h=4 configuration is less effective compared to the others, maybe due to the difficult to propagate salient information over a denser graph. in summary, the results presented in this section highlight the substantial potential of leveraging the links between electrodes in eeg classification tasks. furthermore, they also show that eegraph modeling strategy allow easy integration of eeg data with gnns, a powerful machine learning framework. overall, the results confirm that the sgc and 1-eegraph is the best-performing model. 5 conclusion in this study, we successfully developed four gnns architectures based on convolutional dynamics to predict ppc results. these architectures effectively explored both nonlinear transformations and local connections between the vertices. furthermore, we proposed eegraph, a modeling strategy to transform raw eeg data into graph structures, which leverages the connections between the vertices (electrodes) of the graph structures to the fullest extent. our experimental setup provided a comprehensive comparison between the proposed gnns architectures and an lstm architecture, demonstrating the satisfactory performance of our proposals. in conclusion, the results obtained from our study demonstrate that the proposed gnn architectures outperformed the lstm in terms of various performance metrics. the comparison was carried out considering different modeling architectures of eeg data as a graph. this accomplishment aligns with the primary objective of our article, which aimed to explore the spatial structure of eeg and establish its potential contribution to performance improvement.throughout the experiments, it was observed that the gnns architectures consistently achieved higher accuracy, f1-score, and recall values compared to the lstm. this suggests that, leveraging the topological characteristics of eeg data through graph-based approaches can effectively enhance the performance of eeg analysis tasks. looking forward, further studies will focus on developing novel layers tailored specifically for the gnn architecture to achieve even better performance on the prognostic of patients in coma. by customizing the gnn architecture to better suit the characteristics of eeg data, we can potentially uncover additional improvements in the accuracy and effectiveness of eeg analysis techniques. in the future, we will refine the modeling strategy to represent eeg data into graph structures and extend its application to other public datasets. this refinement will focus on employing network structural optimization techniques to enhance the accuracy and efficiency of the graph-based models, thereby improving the overall predictive performance and applicability in real-world scenarios. references alhagry, s., fahmy, a.a., el-khoribi, r.a.: emotion recognition based on eeg using lstm recurrent neural network. int. j. adv. comput. sci. appl. 8(10) (2017) google scholar  altıntop, ç.g., latifoğlu, f., akın, a.k., cetin, b.: a novel approach for detection of consciousness level in comatose patients from eeg signals with 1-d convolutional neural network. biocybern. biomed. eng. 42(1), 16–26 (2022) article  google scholar  baldo júnior, s., carneiro, m.g., destro-filho, j.b., zhao, l., tinós, r.: classification of coma etiology using convolutional neural networks and long-short term memory networks. in: 2023 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2023) google scholar  bissaro, l.z., junior, o.o.n., filho, j.b.d., jin, y., carneiro, m.g.: towards the prognosis of patients in coma using echo state networks for eeg analysis. procedia comput. sci. 222, 509–518 (2023) article  google scholar  carneiro, m.g., ramos, c.d., destro-filho, j.b., zhu, y.t., ji, d., zhao, l.: high-level classification for eeg analysis. in: 2023 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2023) google scholar  chen, m., wei, z., huang, z., ding, b., li, y.: simple and deep graph convolutional networks. in: international conference on machine learning, pp. 1725–1735. pmlr (2020) google scholar  davis, a.e., gimenez, a.: cognitive-behavioral recovery in comatose patients following auditory sensory stimulation. j. neurosci. nurs. 35(4), 202–209 (2003) article  google scholar  demir, a., koike-akino, t., wang, y., haruna, m., erdogmus, d.: eeg-gnn: graph neural networks for classification of electroencephalogram (eeg) signals. in: 2021 43rd annual international conference of the ieee engineering in medicine and biology society (embc), pp. 1061–1067. ieee (2021) google scholar  di perri, c., thibaut, a., heine, l., soddu, a., demertzi, a., laureys, s.: measuring consciousness in coma and related states. world j. radiol. 6(8), 589 (2014) article  google scholar  gerrard, p., zafonte, r., giacino, j.t.: coma recovery scale-revised: evidentiary support for hierarchical grading of level of consciousness. arch. phys. med. rehabil. 95(12), 2335–2341 (2014) article  google scholar  graves, a.: generating sequences with recurrent neural networks. arxiv preprint arxiv:1308.0850 (2013) hamilton, w., ying, z., leskovec, j.: inductive representation learning on large graphs. adv. neural info. process. syst. 30 (2017) google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  math  google scholar  huang, y., yang, j., liao, p., pan, j.: fusion of facial expressions and eeg for multimodal emotion recognition. comput. intell. neurosci. 2017(1), 2107451 (2017) google scholar  iwaki, t., hayashi, m., hori, t.: changes in alpha band eeg activity in the frontal area after stimulation with music of different affective content. percept. mot. skills 84(2), 515–526 (1997) article  math  google scholar  kipf, t.n., welling, m.: semi-supervised classification with graph convolutional networks. arxiv preprint arxiv:1609.02907 (2016) kumari, r.s.s., jose, j.p.: seizure detection in eeg using time frequency analysis and svm. in: 2011 international conference on emerging trends in electrical and computer technology, pp. 626–630. ieee (2011) google scholar  kustermann, t., et al.: electroencephalography-based power spectra allow coma outcome prediction within 24 h of cardiac arrest. resuscitation 142, 162–167 (2019) article  math  google scholar  lehembre, r., et al.: resting-state eeg study of comatose patients: a connectivity and frequency analysis to find differences between vegetative and minimally conscious states. funct. neurol. 27(1), 41 (2012) google scholar  liu, j., meng, h., nandi, a., li, m.: emotion detection from eeg recordings. in: 2016 12th international conference on natural computation, fuzzy systems and knowledge discovery (icnc-fskd), pp. 1722–1727. ieee (2016) google scholar  ma, s., cui, j., chen, c.l., xiao, w., liu, l.: an improved bi-lstm eeg emotion recognition algorithm. j. netw. intell 7(3), 623–639 (2022) math  google scholar  panda, r., khobragade, p., jambhule, p., jengthe, s., pal, p., gandhi, t.: classification of eeg signal using wavelet transform and support vector machine for epileptic seizure diction. in: 2010 international conference on systems in medicine and biology, pp. 405–408. ieee (2010) google scholar  portnova, g.v., atanov, m.s.: nonlinear eeg parameters of emotional perception in patients with moderate traumatic brain injury, coma, stroke and schizophrenia. aims neurosci. 5(4), 221 (2018) article  google scholar  sabeti, m., katebi, s., boostani, r., price, g.: a new approach for eeg signal classification of schizophrenic and control participants. expert syst. appl. 38(3), 2063–2071 (2011) article  math  google scholar  sheykhivand, s., mousavi, z., rezaii, t.y., farzamnia, a.: recognizing emotions evoked by music using cnn-lstm networks on eeg signals. ieee access 8, 139332–139345 (2020) article  google scholar  shoeb, a.h., guttag, j.v.: application of machine learning to epileptic seizure detection. in: proceedings of the 27th international conference on machine learning (icml-10), pp. 975–982 (2010) google scholar  soleymani, m., asghari-esfeden, s., fu, y., pantic, m.: analysis of eeg signals and facial expressions for continuous emotion detection. ieee trans. affect. comput. 7(1), 17–28 (2015) article  math  google scholar  temko, a., thomas, e., marnane, w., lightbody, g., boylan, g.: eeg-based neonatal seizure detection with support vector machines. clin. neurophysiol. 122(3), 464–473 (2011) article  google scholar  thakor, n.v., tong, s.: advances in quantitative electroencephalogram analysis methods. annu. rev. biomed. eng. 6, 453–495 (2004) article  math  google scholar  wang, s., qu, j., zhang, y., zhang, y.: multimodal emotion recognition from eeg signals and facial expressions. ieee access 11, 33061–33068 (2023) article  math  google scholar  wu, f., zhang, t., jr., a.h.s., fifty, c., yu, t., weinberger, k.q.: simplifying graph convolutional networks. arxiv preprint arxiv:1902.07153 (2019) zheng, w.l., et al.: predicting neurological outcome in comatose patients after cardiac arrest with multiscale deep neural networks. resuscitation 169, 86–94 (2021) article  math  google scholar  zheng, w.l., et al.: predicting neurological outcome from electroencephalogram dynamics in comatose patients after cardiac arrest with deep learning. ieee trans. biomed. eng. 69(5), 1813–1825 (2021) article  math  google scholar  zhong, p., wang, d., miao, c.: eeg-based emotion recognition using regularized graph neural networks. ieee trans. affect. comput. 13(3), 1290–1301 (2022) article  math  google scholar  download references acknowledgment authors thank the financial support given by brazilian national council for scientific and technological development cnpq (grant number 420212/2023-0), and the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (grants number 2019/07665-4 and 2024/02783-7) and by the ibm corporation. we also thank nvidia corporation by the donation of a titan v gpu used in this research. author information authors and affiliations faculty of computing, federal university of uberlândia, uberlândia, brazil odelmo o. nascimento junior, dhara l. c. assis, joão b. destro filho & murillo g. carneiro department of computing and mathematics, university of são paulo, ribeirão preto, brazil liang zhao & murillo g. carneiro authors odelmo o. nascimento juniorview author publications search author on:pubmed google scholar dhara l. c. assisview author publications search author on:pubmed google scholar joão b. destro filhoview author publications search author on:pubmed google scholar liang zhaoview author publications search author on:pubmed google scholar murillo g. carneiroview author publications search author on:pubmed google scholar corresponding author correspondence to odelmo o. nascimento junior . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper nascimento junior, o.o., assis, d.l.c., destro filho, j.b., zhao, l., carneiro, m.g. (2025). modeling eeg data into graphs for the prognostic of patients in coma using graph neural networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_16 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords graph neural networks electroencephalogram coma outcome sequential data graph-based learning eegraph publish with us policies and ethics profiles murillo g. carneiro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a data distribution-based ensemble generation applied to wind speed forecasting | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a data distribution-based ensemble generation applied to wind speed forecasting conference paper first online: 30 january 2025 pp 17–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a data distribution-based ensemble generation applied to wind speed forecasting download book pdf download book epub diogo m. almeida9, paulo s. g. de mattos neto9 & daniel c. cunha9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 474 accesses abstract variability in wind intensity and direction causes unstable electricity supply to the power system, making integrating this energy into the electrical system a significant challenge for operations and planning practices. ensembles can be used as an alternative to address the complex patterns over time in wind speed time series. the appropriate size of the training partition for ensemble models depends on dataset characteristics. and, to enhance model training efficiency, it is important to minimize repetitive data. by maintaining a concise training set, it becomes easier to meet the requirements of software and hardware constraints. mainly because, there is a growing interest in deploying machine learning models on edge devices. for these reasons, a new method called local distribution (locdist) has been introduced in this paper to predict wind speed, utilizing local pattern recognition based on data distribution. in testing with three wind speed time series, locdist created a compact training subset with less than 20% of the training data. the diebold-mariano hypothesis test was employed to assess the significance of the forecast errors of the proposal compared to individual and bagging methods that use the entire training set. the locdist method with long short-term memory (lstm) and gated recurrent unit (gru) won in 100% of the cases. additionally, the locdist with extreme learning machines (elm) and autoregressive integrated moving average (arima) won or tied in 83% and 66% of the cases, respectively. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others short-term forecasting of wind speed using time division ensemble of hierarchical deep neural networks article 04 march 2020 efficient training over long short-term memory networks for wind speed forecasting chapter © 2017 short-term wind power prediction based on data reconstruction and improved extreme learning machine article 12 february 2022 1 introduction the use of wind energy is increasing worldwide each year. over the past 20 years, wind energy has become the primary clean and cost-competitive energy globally [1]. in 2023, wind energy reached the historic milestone of 1 tw of installed capacity, and the coming years will mark a crucial transition period for the global wind industry. it took 40 years to reach this milestone, but the next 1 tw will take less than a decade [2]. the brazilian northeast is a region with some of the best winds in the world for wind energy production, as they are more constant, have stable speeds, and do not frequently change direction. therefore, 90% of brazilian wind farms are located in the northeast region [3]. although wind energy is considered an attractive option due to its abundance and ecological benefits, several challenges are still faced in its exploitation, such as the variability in wind intensity and direction, causing unstable electricity supply to the power system [4]. integrating this energy into the electrical system represents a significant challenge for operations and planning practices [5]. hence, wind energy systems’ planning, scheduling, maintenance, and control depend on wind speed forecasting, making it essential to obtain wind speeds in advance [4]. for these reasons, the analysis and evaluation of this type of energy have attracted the attention of researchers worldwide [6]. various machine learning methods have already been applied to wind speed forecasting, such as support vector regression (svr) [7,8,9], extreme learning machines (elm) [10], long short-term memory (lstm) [11, 12], and gated recurrent unit (gru) [12]. each model can recognize nonlinear patterns in a series and has its particularities. however, wind speed time series display complex patterns [6]. factors such as regions’ surface roughness, vegetation variability, and land use influence wind behavior [13]. furthermore, research shows that wind speed data can exhibit a wide range of distributions [14], and these time series may also reveal hidden patterns with chaotic characteristics [5]. as a result, accurately forecasting wind speed poses a significant challenge [6]. an alternative to modeling complex patterns over time in wind speed time series is using ensembles [6]. for example, applying traditional procedures to a time series with a mixed distribution can lead to misleading results [15]. therefore, we understand that ensembles offer superior prediction accuracy and reduced uncertainty compared to individual models, as they combine the different characteristics of individual prediction models [16,17,18,19,20], especially in the field of wind speed forecasting [6, 13, 21,22,23]. ensemble is a technique used to combine prediction models [6]. it involves at least two steps: generation and integration. as stated in [6], individual models (also known as base models) use time series as input to produce predictions in generation. in integration, the predicted outputs of the base models are combined to obtain the final prediction. diversity among predictions is crucial to recognizing different patterns in the time series, and it is one of the primary objectives of using an ensemble [20]. the authors in [17] have reported that the appropriate size of the training partition for ensemble models varies depending on dataset characteristics [17]. it is highlighted in [24] that to enhance model training efficiency, it is important to minimize repetitive data. by maintaining a concise training set, it becomes easier to meet the requirements of software and hardware constraints. additionally, efficient artificial intelligence (ai) everywhere is a goal pursued by humans. as a result, there is a growing interest in deploying machine learning models on edge devices (computation and data storage closer to the sources of data) that have stringent constraints on resources and energy [25]. for these reasons, studies on the ensemble generation stage that appropriately define partitions for the base models are desirable in wind speed prediction. especially because in [26], it has been shown that it is achievable to obtain a subset of wind speed data partitions that adequately represent the data dynamics throughout the entire analysis period. this fact suggests that wind speed time series exhibit redundancy in their patterns, rendering it unnecessary to utilize the entire training set for model training. so, the proposal of this paper introduces a new generation method for homogeneous ensembles, composed of a single type of base model, to predict wind speed, called local distribution (locdist), which utilizes local pattern recognition based on data distribution. this method generates partitions based on local data distribution using a divide-and-conquer strategy. instead of using a single prediction model to identify patterns in different distributions in the time series, base models of an ensemble are applied to separate partitions selected from the time series to identify distribution patterns. the predictions of the base models are then combined to aggregate the mapping of different distribution patterns of the time series into an ensemble. our hypothesis suggests that if expert models are generated in partitions containing data with different distributions among the partitions, it will result in models with diversity in their predictions. moreover, combining these models will create an ensemble of specialists, each well-versed in different distributions of the time series data, containing concise information about the data patterns. thus, the primary contribution of this proposal is to create a compact training subset that minimally represents the data distribution of the time series. the training phase can be executed more efficiently by reducing the amount of training data. the outline of the paper is organized as follows. section 2 describes the related works. section 3 introduces the proposal in detail. section 4 is the description of the processes through which results are generated. in sect. 5, results are presented by performing a comparison among individual models, ensembles and the proposal. finally, in sect. 6, conclusions are drawn. 2 related works the use of individual models for wind speed time series forecasting has been widespread [8, 10, 12, 27]. in [27], it was found that autoregressive moving average (arma) models are more effective than the persistence model in predicting future observations. the persistence model simply repeats the most recent past observation. furthermore, in [8], it was shown that support vector regression (svr) outperforms multilayer perceptron (mlp). in [10], the extreme learning machines (elm) demonstrated competitive prediction errors compared to mlp and svr while requiring lower computational costs. reference [12] introduced a new seasonal autoregressive integrated moving average (arima) model for predicting wind speed time series and compared it with long short-term memory (lstm) and gated recurrent unit (gru). despite the more advanced machine learning algorithms of lstm and gru, the seasonal arima model exhibited superior predictive performance and faster training. these individual models employ a global mapping of a time series, using the entire training set for model training. however, this approach is not the optimal choice due to the weaknesses, disadvantages, and limitations of modeling complex energy systems [6]. some methods for the generation stage in ensembles have been proposed to identify complex patterns in time series data. in [21,22,23], wind speed time series are decomposed into a finite and small number of intrinsic mode functions (imfs) using empirical mode decomposition (emd). these imfs contain information about local trends and fluctuations at different scales of the original signal, which is valuable for understanding the actual physical significance of the signal [23]. a homogeneous ensemble utilizing a resampling technique is developed in a different application in [19] and used on the m3-competition dataset. the time series is bootstrapped using the moving block bootstrap (mbb). this process creates new time series from the original, intending that the predictors of these time series will make the final forecast resilient to the uncertainty observed in the data [18]. this method has a significant computational cost, involving training 100 base models on the entire training set. however, some generation methods have been proposed focusing on identifying different local patterns in a time series, thus providing specialized local predictors [17]. therefore, in [17], the authors propose a homogeneous ensemble for various time series applications. they create training partitions of equal size, allowing for an overlap between two adjacent partitions, which enables the model to adapt to progressive pattern changes. the authors also investigate four different partition sizes. in [28], a generation strategy is used to obtain partitions of the training set with the most informative data through clustering. this approach can accelerate the learning phase without compromising accuracy or improving it. it is particularly beneficial when the training dataset is large, as it can significantly reduce computation time by avoiding the processing of redundant data [28]. however, in the proposed method of [28], it is initially necessary to train 200 prediction models on training partitions before applying clustering. the authors argue that the execution time of their method decreases because these 200 models can be run in parallel. in most of the previously cited works, excluding [28], no analysis is conducted on the partition data to identify different data patterns or to remove redundancy. there is no evidence that the constructed partitions have different data patterns to meet the requirement of generating diverse models, nor any analysis on the most appropriate size or region for the training partitions. in [25], it is reported that developing an automatic solution for model compression is challenging. the aim is to achieve a higher compression ratio while minimizing accuracy loss, posing a challenging trade-off. therefore, further research is desirable to investigate the ensemble generation step by analyzing the data patterns in the partitions. mainly because in [26], it was demonstrated that it is possible to obtain a subset of wind speed data partitions that reasonably reflect the data dynamics over the entire analysis period. this fact indicates that wind speed time series may exhibit redundancy in their patterns, making it unnecessary to use the entire training set for training a model. 3 proposed new ensemble fig. 1. training and testing phases of the proposed local pattern recognition based on data distribution. full size image the proposal is a homogeneous ensemble comprising two stages: generation and integration. we define partitions with different data distributions and eliminate other ones with redundant data distributions within the training set during the generation stage. therefore, a concise training set that reflects the data distribution patterns is obtained. this process enables the creation of a pool of trained models \(p=\{m_1,m_2,\ldots ,m_n\}\), that is, a group of n local expert predictors, which have been trained on partitions generated with different data distributions validated by the kolmogorov-smirnov (ks) hypothesis test. the null hypothesis of ks test states that the two samples share the same distribution [29]. subsequently, the predictors are combined in the integration stage to achieve the final forecasting of the ensemble. figure 1(a) shows the training phase of the locdist method. the inputs consist of the training and validation sets of the time series, the significance levels of the ks hypothesis test, \(\alpha _1\) and \(\alpha _2\), the minimum partition size t, the sliding window size w, the base model m with a technique for hyperparameter selection g, the evaluation measure e for the evaluation of m and, lastly, a combination method c for integration. the output is the pool of trained models \(p=\{m_1,m_2,\ldots ,m_n\}\) of n local expert models and combination method c. first, the ensemble generation step is employed through partitioning, which is constructed as follows: (i) windows \(w_t\) are created in the training set with a fixed size w. the difference between two adjacent windows \((w_t w_{t-1})\) is only one observation. the first window \(w_1\) is defined as the initial reference window \(ref_1\); (ii) subsequently, a significance level \(\alpha _1\) is defined for applying the ks hypothesis test between the reference window and each subsequent window sequentially; (iii) the method constructs the partition by joining adjacent windows that have a p-value \(>\alpha _1\), otherwise a new partition is initiated with a new reference window. in the end, we will have the defined partitions and their respective reference windows; (iv) next, partitions with a size smaller than t are eliminated; (v) return to step (iii) applying a grid search with different significance levels for \(\alpha _1\), aiming to maximize the expression $$\begin{aligned} \text {maximize } \frac{\sum _{i<j}^{u} ks(ref_i,ref_j)}{b_{u,2}} \end{aligned}$$ (1) $$\begin{aligned} \text {subject to: } ks(ref_i,ref_j) = \left\{ \begin{matrix} 1, & \text {if } \text {{ p}-value} \le 0.05;\\ 0, & \text {otherwise.} \end{matrix} \right. \end{aligned}$$ (2) where b is a combination in the field of combinatorial analysis and u is the number of reference windows at the end of step (iv). therefore, \(b_{u,2}\) represents the total number of possible two-by-two combinations between the reference windows of each partition. ks refers to the kolmogorov-smirnov (ks) hypothesis test. thus, \(ks(ref_i,ref_j)\) is equal to 1 if the reference windows i and j of partitions have different distributions (p-value \(\le 0.05\)), and equal to 0, otherwise. maximizing the expression (1) is a key approach for identifying the most significant number of partitions with different distributions in the training set; (v) after determining the best significance level for \(\alpha _1\) in the grid search to maximize the expression (1), the ks test is applied once again to eliminate partitions with redundant distributions at a significance level of \(\alpha _2=0.05\). at the end of step (v), the disjoint partitions d with solid evidence of different data distributions are obtained, and the partitioning procedure is concluded. succeeding generating disjoint partitions d with different distributions, time series prediction models m are trained on these disjoint partitions d using technique g and evaluation metric e for hyperparameter selection. linear models are chosen based on the lowest error e in the training partition, while nonlinear models are selected based on the lowest error e in the validation partition to avoid overfitting. this process creates a pool \(p=\{m_1,m_2,\ldots ,m_n\}\) consisting of n local expert predictors specialized in different distribution patterns of the time series. this pool p is combined using a combination method c, which can be a statistical measure, such as the average, or a nonlinear model m trained with technique g and evaluation metric e for hyperparameter selection. the best combination method c is selected based on the lowest error e in the validation set. finally, fig. 1(b) shows the testing phase of the locdist method. the input involves the testing set of the time series, the pool p of n local expert predictors, and the combination method c for integration. the output is the wind speed forecasting. this phase consists of testing the predictive capacity of the proposal with the models p and the combination method c on an out-of-sample set. 4 experimental protocol the databases considered in this work are from the brazilian institute of space research (inpe, in portuguese), obtained through sensors at a height of 50 m [30]. the northeastern region of brazil has some of the world’s best winds for wind energy production, which is why 90% of brazilian wind parks are located in this region [3]. database 1 refers to petrolina-pe, while databases 2 and 3 were obtained in triunfo-pe. the first database covers the period from july to september 2010. databases 2 and 3 cover the periods from october to december 2006 and march to may 2006, respectively. there are 13,248 instances at ten-minute intervals between consecutive instances for each database, and no missing values. every six instances were averaged, resulting in 2,208 instances for each time series with one-hour time intervals to reduce the size of the database. subsequently, each time series was fully normalized to the interval [0, 1]. thus, the input data for the neural networks are appropriately mapped according to the codomain of the sigmoid activation function. then, each time series was divided into three subsets: the first 50% of the data for training, then 25% for validation, and the last 25% for testing. table 1 presents the statistical summary of the three pre-processed and non-normalized databases. table 1. statistical summary of wind speed in the three time series databases.full size table the locdist method is a homogeneous ensemble and was separately analyzed with the following base models: arima, elm, svr, lstm, and gru. the value 24 was chosen for the sliding window size w due to the seasonality of the time series, which is 24 observations (24 h). the grid search for the values of \(\alpha _1\) was conducted in the range [0.05, 0.1, 0.15, ..., 0.3]. values of \(\alpha _1>0.3\) tend to construct minimal partitions, so the search ends at \(\alpha _1=0.3\). furthermore, the behavior of locdist was evaluated for different values of the variable t, which is the minimum allowed size for the generated partitions. in [31], mentions that 66 observations are considered a minimal amount for adjusting the parameters of a machine learning model. this manner, the variable t was evaluated with values 45, 60, and 75. for values above 60, such as 75, the generation method could not find partitions with this minimum size for all time series considered in this work. thus, the evaluation was limited to the values \(t=45\) and \(t=60\). the elimination of partitions with redundant data distributions was done so that the less recent distributions were discarded. it was found that maintaining the more recent distributions resulted in better locdist predictive performance compared to the approach that discards the more recent redundant partitions. the arima base model used the training partition for training and parameter adjustment. the elm, svr, lstm and gru models used the training partition for training and the validation set for parameter adjustment to avoid overfitting. at last, in the integration stage, the locdist used the simple average or the elm nonlinear model as a combination method. the simple average is a statistical measure known in the ensemble literature for its robustness [20]. the combination with a nonlinear model elm is a more sophisticated approach that can surpass the simple average combination in the presence of predictors with significant differences in accuracy [32]. the validation set was used to select the best combination approach according to the lowest rmse. firstly, we compared locdist with the respective individual models: arima, svr, elm, lstm, or gru. each model was globally trained on the complete training set. subsequently, the locdist was compared to the respective homogeneous ensemble using a resampling technique through bagging [19]. the resampling technique was applied to arima, elm, and svr base models. thus, three homogeneous ensembles were obtained, and the base models were globally trained one hundred times on resampled series [19]. this approach has a high computational cost, and because of that, lstm and gru were not used in this resampling technique. in the integration stage, the combination method of the ensemble with bagging was the simple average. the same parameter selection technique was applied to the individual and base models of the ensembles. for the arima model, the integer parameter d was chosen according to the augmented dickey-fuller test to identify stationarity. meanwhile, p and q were obtained in the range [0, 1, ..., 24], as there is a seasonality of 24 lags in the hourly time series. the conditional sum of squares method was applied for model fitting, and the model with the lowest corrected akaike information criterion was selected. the grid search technique was applied to the remaining models. the svr type was \(\epsilon \)-regression with kernel radial basis function (rbf). the input used the sliding window method whose width values were taken from [1, 6, 12, 18, 24]. the regularization parameter was adjusted in the range \([2^0,2^1,...,2^{10}]\), while the tolerance region \(\epsilon \) and the parameter \(\lambda \) from kernel rbf were obtained from the values \([2^{-10},2^{-9},...,2^{0}]\). for elm, the parameters were selected in the range [1, 2, ..., 24] for the input and hidden layer nodes. the activation functions for hidden and output layers were the sigmoid and the identity, respectively. however, during the integration stage with the elm model in the locdist, the number of neurons in the hidden layer was selected from the range [1, 2,..., 42] to enhance the combination. concerning lstm and gru, a stateful network with parameter values taken from the set [1, 6, 12, 18, 24] was assumed for the input nodes. the network parameters were obtained from the set [6, 12, 18, 24] for the hidden layer nodes. still, for the hidden layer, the activation function was the sigmoid. the lstm and gru were trained through 100 epochs and batch size equal to one. finally, the optimization solver adopted was adam. table 2. individual models and ensemble forecasting approaches used in this work with corresponding methods and respective acronyms.full size table table 2 shows all individual models and ensembles implemented and their respective acronyms. all simulations were performed for one-step-ahead forecasting using the r language with the forecast [33] library for arima, e1071 [34] for svr, elmnnrcpp [35] for elm, and keras [36] for lstm and gru. the grid search technique selected the nonlinear models parameters based on the smallest rmse in the validation set. the rmse is defined by equation (3), such that $$\begin{aligned} rmse = \root \of {\left( \frac{\sum _{t=1}^{l} (y_t-\widehat{y}_t)^2}{l}\right) }, \end{aligned}$$ (3) where l is the size of the series, \(y_t\) is the actual value of the time series in period t, and \(\widehat{y}_t\) is the predicted value of the time series in period t. the rmse was also employed to assess predictive performance on the test set, along with three other metrics: mean absolute error (mae), mean absolute percentage error (mape), and predicted change in direction (pocid). for mae and mape metrics, the lower their values, the better the model’s accuracy. in the case of pocid, the higher the value, the better the model’s performance. the mae is defined by $$\begin{aligned} mae = \frac{\sum _{t=1}^{l} |y_t-\widehat{y}_t|}{l}, \end{aligned}$$ (4) where \(|\cdot |\) is the absolute value operator. the mape metric is given by $$\begin{aligned} mape = \frac{100}{l} \sum _{t=1}^{l}\frac{|y_t-\widehat{y}_t|}{y_t}, \end{aligned}$$ (5) while the pocid is designated by $$\begin{aligned} pocid = 100 \frac{\sum _{t=1}^{l} trend_t }{l}, \end{aligned}$$ (6) where $$\begin{aligned} \begin{matrix} trend_t= \left\{ \begin{matrix} 1, & \text {if } (y_t {y}_{t-1})(\widehat{y}_t {\widehat{y}}_{t-1}) > 0;\\ 0, & \text {otherwise.} \end{matrix} \right. \end{matrix} \end{aligned}$$ (7) finally, the diebold-mariano (dm) hypothesis test [37] was applied between locdist and the global mapping methods of the respective base model. so, the rmse loss function is considered for the errors of both competing predictors. the null hypothesis \(h_0\) assumes equality in forecasting accuracy for both predictors. for \(h_0\) to be rejected, the p-value must be less than the statistical significance level \(\alpha =0.05\). some signals were used to interpret the results in tables 3, 4 and 5 of sect. 5. the sign “\(+\)” indicates that \(h_0\) was rejected and the proposed method outperforms the method used for comparison. for instance, if \(locdist_{arima}\) outperformed arima with statistical significance, the sign “\(+\)” will appear in the row of arima. the sign “−” indicates that the proposed method underperforms the method used for comparison. the sign “\(=\)” indicates that the null hypothesis was not rejected, i.e., equality in forecasting accuracy for both predictors. table 3. evaluation measurements and dm hypothesis test for the testing set obtained from the wind speed time-series database 1.full size table 5 results the best results were obtained for all three databases with the value \(t=60\), which is the minimum allowed size for the generated partitions. in database 1, locdist generated three partitions with the following observations: 69, 62, and 96. this corresponds to 20.6% of the entire training set, which consists of 1104 observations. table 3 shows the evaluation measurements and dm hypothesis test for the testing set obtained from the wind speed database 1. the first column indicates the five model types, arima, elm, svr, lstm, and gru, which were used in the methods of this research. the second column lists the constructed methods, such as individual, ensemble with bagging, and the proposed locdist, according to the respective model type. then, the four evaluation metrics, rmse, mae, mape, and pocid, are presented for each forecasting method. finally, the dm hypothesis test result and the respective p-value are shown. the best measure among the forecasting methods for each model type is highlighted in bold. in the case of arima, the \(locdist_{arima}\) achieved the best results in rmse and pocid, though with no statistical significance compared to individual arima and \(bagg_{arima}\). about elm, \(locdist_{elm}\) did not stand out in any metric, but the dm hypothesis test demonstrated no statistical significance in the errors compared to individual elm and \(bagg_{elm}\). in the case of svr, \(locdist_{svr}\) loses to individual svm and \(bagg_{svr}\) with statistical significance. concerning lstm, the \(locdist_{lstm}\) outperformed individual lstm in all metrics with statistical significance. the last type of model is gru, and the \(locdist_{gru}\) achieved the best results in rmse, mae, and mape with statistical significance compared to individual gru. the integration step selected the combination via elm for the arima, lstm, and gru approaches, and the combination by averaging for the elm and svm. table 4. evaluation measurements and dm hypothesis test for the testing set obtained from the wind speed time-series database 2.full size table regarding database 2, locdist generated two partitions with the following observations: 122 and 89. this corresponds to 19.1% of the entire training set. table 4 illustrates the evaluation measures and the dm hypothesis test for the wind speed database 2 testing set. about arima, \(locdist_{arima}\) achieved the best results in rmse and mae, with statistical significance compared to individual arima. in the case of elm, \(locdist_{elm}\) achieved the best results in all metrics but with no statistical significance compared to the individual elm and \(bagg_{elm}\). about svr, \(locdist_{svr}\) loses to individual svm and \(bagg_{svr}\) with statistical significance. concerning lstm, the \(locdist_{lstm}\) achieved the best results in rmse, mae, and mape with statistical significance compared to individual lstm. finally, the \(locdist_{gru}\) method outperforms the individual gru in all metrics and with statistical significance. the integration step selected the combination via elm for all models except for the svm, which used the average. table 5. evaluation measurements for the test set obtained from the wind speed time-series database 3.full size table in database 3, locdist generated three partitions with the following observations: 66, 65, and 71. this corresponds to 18.3% of the entire training set. table 5 exhibits the results for the testing set acquired from the wind speed time series database 3. \(locdist_{arima}\) reached the best result in pocid, though it had lost to individual arima and \(bagg_{arima}\) with statistical significance. regarding elm, the \(locdist_{elm}\) did not exhibit superior performance in any metric but showed statistical significance in the dm hypothesis test, indicating lower forecasting errors compared to the individual elm but higher errors compared to \(bagg_{elm}\). in the case of svr, the \(locdist_{svr}\) achieved the best result in rmse, mae, and mape, and with statistical significance. concerning lstm, the \(locdist_{lstm}\) achieved the best results in rmse, mae, and mape with statistical significance compared to individual lstm. at last, the \(locdist_{gru}\) method one more time outperformed the individual gru in all four metrics with statistical significance. the integration step selected the combination via elm for all models. concerning the results of the dm hypothesis test in tables 3, 4 and 5, we can see that the methods \(locdist_{lstm}\) and \(locdist_{gru}\) won 100% of the cases. in addition, the \(locdist_{elm}\) and \(locdist_{arima}\) won or tied in 83% and 66% of the cases, respectively. these results corroborate our hypothesis that the locdist method can generate a concise and representative training set compared to the complete training dataset, potentially enabling more appropriate recognition of local patterns. \(locdist_{svr}\) has lost in 66% of the cases. this reveals some limitations, indicating that it is not always the best approach for all models. fig. 2. one-step ahead prediction results for locdist and global mapping methods on dataset 1 considering 20 sequential observations from the test set. full size image fig. 3. one-step ahead prediction results for locdist and global mapping methods on dataset 2 considering 20 sequential observations from the test set. full size image fig. 4. one-step ahead prediction results for locdist and global mapping methods on dataset 3 considering 20 sequential observations from the test set. full size image we then chose some models, such as arima, elm, lstm, and gru, for graphical evaluation of the methods in each dataset. the svr was excluded from the analysis based on the results of the dm hypothesis test. figures 2(a) and 2(b) show, respectively, the one-step-ahead prediction using the method locdist built from gru and lstm models. the superiority of locdist is noticeable compared to the individual methods. in fig. 3, we have dataset 2, and the chosen models were arima and elm. we can observe that the local mapping of locdist exhibits similar or even smaller prediction errors than the global mapping of individual and bagging methods. finally, fig. 4 illustrates the results on dataset 3 for the lstm and gru models. it is also observed that locdist exhibits lower prediction errors compared to methods trained on the entire training set. 6 conclusions the locdist introduces a new generation method for ensembles to predict wind speed, which utilizes local pattern recognition based on data distribution. it creates a compact training subset that minimally represents the data distribution of the time series. the results show promise for the locdist applied to three wind speed time series. creating a training subset for the locdist ensemble was possible using less than 20% of the training data. the predictive performance of locdist was competitive with or even better than that of the individual and bagging models trained on the entire training set. concerning the diebold-mariano hypothesis test applied in the rmse, the locdist method with lstm and gru won in 100% of the cases. in addition, the locdist with elm and arima won or tied in 83% and 66% of the cases, respectively, while the proposed ensemble method with svr lost in 66% of the cases, revealing some limitations. for future work, it would be interesting to investigate the behavior of this new generation method with more sophisticated ensemble approaches, such as dynamic selection. references gwe council: gwec\(|\) global wind report 2021. global wind energy council, brussels, belgium (2019) google scholar  gwe council: gwec\(|\) global wind report 2023. global wind energy council, brussels, belgium (2023) google scholar  a. b. d. e. e. abeeólica, “abeeólica \(|\) infovento,” infovento 31, 15 de junho de 2023 (2023) google scholar  qu, z., mao, w., zhang, k., zhang, w., li, z.: multi-step wind speed forecasting based on a hybrid decomposition technique and an improved back-propagation neural network. renew. energy 133, 919–929 (2019) article  math  google scholar  jiang, p., wang, b., li, h., lu, h.: modeling for chaotic time series based on linear and nonlinear framework: application to wind speed forecasting. energy 173, 468–482 (2019) article  math  google scholar  ahmadi, m., khashei, m.: current status of hybrid structures in wind forecasting. eng. appl. artif. intell. 99, 104133 (2021) article  math  google scholar  mohandes, m.a., halawani, t.o., rehman, s., hussain, a.a.: support vector machines for wind speed prediction. renew. energy 29(6), 939–947 (2004) article  math  google scholar  salcedo-sanz, s., ortiz-garcı, e.g., pérez-bellido, á.m., portilla-figueras, a., prieto, l., et al.: short term wind speed prediction based on evolutionary support vector regression algorithms. exp. syst. appl. 38(4), 4052–4057 (2011) article  math  google scholar  kong, x., liu, x., shi, r., lee, k.y.: wind speed prediction using reduced support vector machines with feature selection. neurocomputing 169, 449–456 (2015) article  math  google scholar  saavedra-moreno, b., salcedo-sanz, s., carro-calvo, l., gascón-moreno, j., jiménez-fernández, s., prieto, l.: very fast training neural-computation techniques for real measure-correlate-predict wind operations in wind farms. j. wind eng. ind. aerodyn. 116, 49–60 (2013) article  math  google scholar  memarzadeh, g., keynia, f.: a new short-term wind speed forecasting method based on fine-tuned lstm neural network and optimal input sets. energy convers. manage. 213, 112824 (2020) article  math  google scholar  liu, x., lin, z., feng, z.: short-term offshore wind speed forecast by seasonal arima a comparison against gru and lstm. energy 227, 120492 (2021) article  google scholar  ferreira, m., santos, a., lucio, p.: short-term forecast of wind speed through mathematical models. energy rep. 5, 1172–1184 (2019) article  math  google scholar  wu, j., li, n.: impact of components number selection in truncated gaussian mixture model and interval partition on wind speed probability distribution estimation. sci. total environ. 883, 163709 (2023) article  google scholar  robinson, p.: analysis of time series from mixed distributions. ann. stat., 915–925 (1982) google scholar  santos júnior, d.s.o., de mattos neto, p.s.g., de oliveira, j.f.l., cavalcanti, g.d.c.: a hybrid system based on ensemble learning to model residuals for time series forecasting. inf. sci. 649, 119614 (2023) google scholar  de mattos neto, p.s., cavalcanti, g.d., firmino, p.r., silva, e.g., nova filho, s.r.v.: a temporal-window framework for modelling and forecasting time series. knowl. based syst. 193, 105476 (2020) google scholar  petropoulos, f., hyndman, r.j., bergmeir, c.: exploring the sources of uncertainty: why does bagging for time series forecasting work? eur. j. oper. res. 268(2), 545–554 (2018) article  math  google scholar  bergmeir, c., hyndman, r.j., benítez, j.m.: bagging exponential smoothing methods using stl decomposition and box-cox transformation. int. j. forecast. 32(2), 303–312 (2016) article  math  google scholar  sergio, a.t., de lima, t.p., ludermir, t.b.: dynamic selection of forecast combiners. neurocomputing 218, 37–50 (2016) article  math  google scholar  hu, j., wang, j., zeng, g.: a hybrid forecasting approach applied to wind speed time series. renew. energy 60, 185–194 (2013) article  math  google scholar  ruiz-aguilar, j.j., turias, i., gonzález-enrique, j., urda, d., elizondo, d.: a permutation entropy-based emd-ann forecasting ensemble approach for wind speed prediction. neural comput. appl. 33(7), 2369–2391 (2021) article  math  google scholar  jiang, z., che, j., wang, l.: ultra-short-term wind speed forecasting based on emd-var model and spatial correlation. energy convers. manage. 250, 114919 (2021) article  math  google scholar  bowden, g.j., maier, h.r., dandy, g.c.: optimal division of data for neural network models in water resources applications. water resour. res. 38(2), 2–1 (2002) article  math  google scholar  deng, l., li, g., han, s., shi, l., xie, y.: model compression and hardware acceleration for neural networks: a comprehensive survey. proc. ieee 108(4), 485–532 (2020) article  math  google scholar  ribeiro, r., fanzeres, b.: identifying representative days of solar irradiance and wind speed in brazil using machine learning techniques. energy ai 15, 100320 (2024) article  math  google scholar  torres, j.l., garcia, a., de blas, m., de francisco, a.: forecast of hourly average wind speed with arma models in navarre (spain). sol. energy 79(1), 65–77 (2005) article  math  google scholar  tetko, i.v., villa, a.e.: efficient partition of learning data sets for neural network training. neural netw. 10(8), 1361–1374 (1997) article  math  google scholar  hodges, j., jr.: the significance probability of the smirnov two-sample test. ark. mat. 3(5), 469–486 (1958) article  mathscinet  math  google scholar  inpe: rede do sistema de organização nacional de dados ambientais (2020). http://sonda.ccst.inpe.br/index.html. accessed 27 jul 2023 cerqueira, v., torgo, l., soares, c.: a case study comparing machine learning with statistical methods for time series forecasting: size matters. j. intell. inf. syst. 59(2), 415–433 (2022) article  math  google scholar  andrawis, r.r., atiya, a.f., el-shishiny, h.: forecast combinations of computational intelligence and linear models for the nn5 time series forecasting competition. int. j. forecast. 27(3), 672–688 (2011) article  math  google scholar  hyndman, r.j., et al.: “package ‘forecast’,” online (2024). https://cran.r-project.org/web/packages/forecast/forecast.pdf meyer, d., et al.: e1071: misc functions of the department of statistics, probability theory group (formerly: e1071), tu wien. r package version (2023) google scholar  mouselimis, l., gosso, a., jonge, e.: elmnnrcpp: the extreme learning machine algorithm. r package (2023) google scholar  chollet, f.: et al. “keras” (2015). https://keras.io diebold, f.x., mariano, r.s.: comparing predictive accuracy. j. bus. econ. stat. 13(3) (1995) google scholar  download references author information authors and affiliations centro de informática (cin), universidade federal de pernambuco (ufpe), recife, pe, brazil diogo m. almeida, paulo s. g. de mattos neto & daniel c. cunha authors diogo m. almeidaview author publications search author on:pubmed google scholar paulo s. g. de mattos netoview author publications search author on:pubmed google scholar daniel c. cunhaview author publications search author on:pubmed google scholar corresponding author correspondence to diogo m. almeida . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper almeida, d.m., de mattos neto, p.s.g., cunha, d.c. (2025). a data distribution-based ensemble generation applied to wind speed forecasting. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_2 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords time series forecasting ensembles data distribution wind speed publish with us policies and ethics profiles paulo s. g. de mattos neto view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a multi-level semantics formalism for multi-agent microservices | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a multi-level semantics formalism for multi-agent microservices conference paper first online: 30 january 2025 pp 48–63 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a multi-level semantics formalism for multi-agent microservices download book pdf download book epub alison r. panisson9 & giovani p. farias9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 486 accesses 1 citation abstract operational semantics is a fundamental approach to the formalisation of programming languages and almost a standard when it comes to agent-oriented programming languages. it helps ensure the correctness of interpreters, facilitates their implementation, and supports proofs of important properties. however, in the literature, there are few attempts of formalising multi-level operational semantics for multi-agent systems which allows the formalisation of all levels of abstraction in those systems. also, recently, authors have started to discuss a new approach to build multi-agent systems known as multi-agent microservices, arguing that microservices represent a potential point of convergence between modern software engineering and multi-agent systems. we agree that microservices could be a useful approach to think of multi-agent systems and, in this paper, we propose a multi-level semantics to formalise this new approach to build multi-agent systems based on the abstraction of microservices. we not only introduce a template for those formalisation but we also show how our operational semantics style promotes desired characteristics of modularity, reusability, and a clear specification of shared communication artifacts in multi-agent microservices. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a multi-level approach to the formal semantics of agent societies chapter © 2020 semantic collaboration for multi-agent: theory, framework, and prospects chapter © 2024 semantic web services for multi-agent systems interoperability chapter © 2019 1 introduction multi-agent systems involve core concepts such as loose-coupling, distribution, reactivity and isolated (local) state. thus, given also its characteristics, the concept of agents has been widely studied and a large range of tools, programming languages and methodologies developed [5]. however, the adoption of these tools, programming languages and methodologies is not largely seen within industry [10]. in the same time that multi-agent systems have been largely studied, the industry has evolved through a range of enterprise paradigms and models, that have shifted from centralised systems towards highly-decentralised systems, showing properties that were exclusively found in multi-agent systems [9]. nowadays, the industry has focused on the so-called microservice paradigm, in which systems are built from small, loosely-coupled services that maintain their own state [16]. with microservices, the industry has found a manner to develop systems towards an ecosystem of tools and components that are easy to extend, strong on fault tolerance and scaling [34]. studies on the convergence of software engineering for microservices and multi-agent systems brings many benefits for both areas [9]. on the one side, from the multi-agent systems perspective, the main benefits are: (i) the embracing of web models for agents; (ii) the decomposition of agent systems into self-contained subsystems, which can be independently tested, improving reliability and the creation of libraries of agents; and (iii) the alignment of multi-agent systems deployment with the best practice of software engineering into the industry. on the other side, from the microservices perspective, the main benefit is the seamless integration of agent technologies into microservices-based systems [9]. although the convergence between modern software engineering for microservices and multi-agent systems has been studied, for example, in [9], still it is missing a formalisation for such systems. in this paper, we propose a formal operational semantics to multi-agent microservices. thus, a formalisation for multi-agent microservices using our semantics can be directly implemented in any agent-oriented programming language that supports such components. the operational semantics provide means to specify programs which are independent of any particular programming language, taking in consideration the main components of multi-agent microservices. the main contributions of this work are: (i) a template to define the formal semantics of multi-agent microservices, which allows us to specify the evident multiple levels of abstraction present in multi-agent microservices; (ii) abstract instances of the multi-level semantic rules, showing how the operational semantics can be used to define formally the communication processes not only between microservices and agents but also for the treatment of external communication; (iii) a discussion on how the multi-level semantics provide a specification of multi-agent microservices that promote desired characteristics like modularity and reusability. also, we discuss how the multi-level components provide a clear specification for shared communication artifacts, which is a fundamental characteristic of microservices, given they reduce chattiness [28]. 2 background 2.1 multi-agent microservices (mams) in [9], the authors introduce multi-agent microservices (mams), which is a class of systems comprised of agent-oriented microservices (aoms) and plain-old microservices (poms). aoms are microservices built using mas methodology and they are exposed through well-defined interfaces modelled as a set of representation state transfer (rest). rest promotes a view in which all resources are isolated and treated as distinct services. in order to avoid an abundance of network calls, chattiness [28], a microservice may often play the role of a host to multiple resources. thus, microservices share some characteristics with agents. agents are not just resources; they are complex rational and decision-making entities. however, an agent can be seen itself as a resource as well as it can be seen as a host for others resources it has access to perceive and act. also, agents rarely work separately, and multi-agent systems are normally modelled by means of a society of agents that will collaborate, through message-passing, to reach an organisational goal [33]. another related concept present in microservices and multi-agent systems is visibility. as human organisations, some individuals are supposed to be more communicative than other. for example, in a company, sales and customers support staff are supposed to interact with the external public, while other roles within that organisation will implement internal functions necessary to keep the company running. concepts of interface agents for organisations are already discussed in the multi-agent system literature [12, 22]. with these ideas of convergent concepts between microservices and multi-agent systems technologies, the authors in [9] have proposed that an agent (or group of agents) can implement a microservice and an organisation of agents (or multi-agent system) could be seen as a service, an approach they named as organisation as a service (oaas). figure 1 shows an example of two multi-agent systems (services), each system configured through group/set of agents representing microservices within the multi-agent system. in mams, agents communicate within their groups, microservices interact through interfaces usually handled by an agent, and multi-agent systems (services) interact through interfaces usually handled by a service. fig. 1. multi-agent microservices full size image 2.2 multiple levels in multi-agent systems there are various approaches to developing multi-agent systems that consider different levels of abstraction. in this section, we discuss some of such approaches in order to clarify the existence of such multiple levels in mas, which supports the manner we are going to define the operational semantics for multi-agents microservices. one well-known practical multi-agent programming framework is jacamo [3, 4]. jacamo is built upon three existing platforms: jason [7] for programming autonomous agents, \(\mathcal {m}\textsc {oise}\) [19] for programming agent organisations, and cartago [27] for programming shared environments. jacamo emphasises the existence of three levels, in that work called “dimensions”, in a single framework so that all levels can be programmed using the languages provided by the combined platforms mentioned above. also, the development of mas based on electronic institutions [21, 32] considers the existence of multiple levels in mas, emphasising that, while electronic institutions focus on the so-called macro-level (societal) aspects, the so-called micro-level has its focus on the agents [30]. further, the poporg (populational-organisational) model [12] also aims to structure the mas in various levels of abstraction. the model focuses on the dynamics of the different internal organisational levels of the organisation of mas. in addition, the work in [11] introduces so-called social sub-systems, which account for the structuring of organisational entities into sub-systems within the mas. other aspects that emphasises the existence of multiple levels in mas can be found in the survey [18]. furthermore, multi-agent microservices inherit such characteristics of multiple levels of abstraction from multi-agent systems. in multi-agent microservices, the multiple levels of abstraction not only promote the system organisation but also promote the modularity and reusability of the system’s components, i.e., microservices. 3 operational semantics for multi-agent microservices the main idea behind structured operational semantics proposed by [26] is to define the behaviour of a program in terms of the behaviour of its parts, providing a structural, i.e., syntax-oriented and inductive, view on operational semantics. thus, specifying a structured operational semantics, it is possible to define the behaviour of a program in terms of a (set of) transition relation(s). the transition relation(s) take the form of a set of inference rules that define the valid transitions of composite tuples representing the transitions of its components. defining the operational semantics allows us not only a formal analysis of the behaviour of systems but also studies of relations between systems. furthermore, operational semantics has an intuitive look and easy-to-follow structure. in this section, we propose a style of formal semantics for multi-agent microservices, in which their components and interfaces are formalised through an operational semantics. updates at the system, microservices (group/set of agents), and agents internal state are given by an elegant transition system proposed by [26] extended to embrace multiple levels of abstraction. as will be discussed in this work, the operational semantics provide a formal specification of the system, guarantying the modularity necessary in those systems, encouraging the reuse of those formalisations. we argue that our style of operational semantics promotes the following desired characteristics of multi-agent microservices: (i) the modularity of the formalisation, which not only fits into methodologies of software engineering to systems based on services but also simplifies the specification of those systems through a multi-level composition, and clear interaction between components from those multiple levels. it also promotes the scalability, considering that the operational semantics provides a modular approach to formalise the modern forefront on the development of distributed computing towards containerisation. it allows us to specify containers individually; (ii) the reusability of the formalisation, in which components (or even entire levels of the formalisation) could be reused in the development of similar systems; (iii) the clear specification of shared communication artifacts, which are essential in systems built based on the paradigm of microservices, when it is necessary that agents, microservices and system share information, thus avoiding chattiness [28]. 3.1 the basis for mams operational semantics in our approach, a system will be represented as a tuple \(\langle s _{ id },\mathcal{m}\mathcal{s}_{s},\mathcal {d}_{ s },\mathcal {m}_{ s } \rangle \) with \( s _{ id }\) the service identifier, \(\mathcal{m}\mathcal{s}_{s}\) a set of microservices (the microservices populating the main service), \(\mathcal {d}_{ s }\) a set of data that service stores at the system level, and \(\mathcal {m}_{ s }\) is the message pooling component for that system. the microservices are represented by the tuple \(\langle ms _{ id },ag_{ms},\mathcal {d}_{ ms },\mathcal {m}_{ ms } \rangle \) with \( ms _{ id }\) the microservice identifier, \(ag_{ms}\) a set of agents (which are populating that microservice), \(\mathcal {d}_{ ms }\) a set of data that microservice stores at the microservice level, and \(\mathcal {m}_{ ms }\) is the message pooling component for that microservice. each agent in the multi-agent system (a member of the microservices) is represented by a tuple \(\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \) with \( ag _{ id }\) the agent identifier, \(\mathcal {g}_{ ag }\) the agent goals, \(\mathcal {b}_{ ag }\) the agent beliefs, and \(\mathcal {a}\) the set of actions that the agent is able to execute. in order to define the semantic rules, we are going to use a particular style of operational semantics introduced in [23, 24], in which multiple levels of a multi-agent system are considered. thus, we have the template to the operational semantic rules represented in the semantic rule below, named semanticrulename. in the semantic rule semanticrulename, we present a template to define the operational semantics of multi-agent microservices considering their multiple levels of abstraction. at the top of the semantic rule, we have conditions representing the preconditions necessary to apply such semantic rule, which also represents the context of the system at that moment. below the conditions, there are three transitions systems: \(\rightarrow _{s}\), \(\rightarrow _{ms}\), \(\rightarrow _{ag}\), representing the transitions occurring in the system, set of microservices, and set of agents, respectively. note that the set of agents being considered at that particular semantic rule represents the set of agents populating the microservice being considered from the set of microservices populating that system. we use the notation “\(\{\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \ldots \}\)” as a simplified representation for the set of agents, highlighting the particular agent being considered, i.e., \(ag_{id}\) is the agent currently being considered in that semantic rule. the same notation is used to highlight the microservice, from the set of microservices, being considered at that particular semantic rule. thus, using transitions in three levels, i.e., the service, the microservices, and the agents, we are able to define the operational semantics for multi-agent microservices which is modular and reusable. 3.2 abstract examples in this section, we show abstract examples of common operations regarding multi-agent microservices, for example, the creation of a new microservice in the system, and the treatment of external messagesfootnote 1. both, microservices approaches and multi-agent systems communicate through message passing and it should not be different in multi-agent microservices. in order to exemplify some operations in multi-agent microservices, we use the following format for messages: \(\langle \texttt{sender}, \texttt{receiver}, \texttt{performative}, \texttt{content} \rangle \), in which \(\texttt{sender}\) represents the sender of the message, \(\texttt{receiver}\) represents the receiver of the message (a system, a particular microservice, or a particular agent), \(\texttt{performative}\) represents the speech-act used, representing the intention of the sender, and \(\texttt{content}\) represents the content of the message, which together with the performative will provide the meaning of that message. for example, \(\langle \texttt{bob}, \texttt{alice}, \texttt{achieve}, \mathtt {closed(door)}\rangle \) has a different meaning than \(\langle \texttt{bob}, \texttt{alice}, \texttt{tell}, \mathtt {closed(door)}\rangle \), in the first message \(\texttt{bob}\)’s desires that \(\texttt{alice}\) closes the door, while in the second message \(\texttt{bob}\) is telling \(\texttt{alice}\) that the door is closed. this representation for messages is commonly used in the multi-agent literature, for example [25, 31]. in the semantic rules, we use a select message function, \(s_{m}\), which selects one of the messages from the message pooling being considered at that semantic rule, according to some priority for treating those messages. in the first semantic rule, \(\texttt{createmicroservice}\), we show how the system treats a request for a new microservice. in this semantics rule, the select message function, \(s_m()\), selects the message \(\langle \textit{sender}, s _{ id }, \mathtt {create\_microservice}, \langle ms _{ id },ag_{ms}, \) \( \mathcal {d}_{ ms },\mathcal {m}_{ ms } \rangle \rangle \), executing the creation of such microservice. we assume that the sender of the message has permission to create a new microservice. in this case, both levels, the system and the microservices levels suffer a transition, represented in (a) and (b), respectively. basically, the system updates the message pooling removing that message, considering that it already has been treated by the system, and creates a new microservice \(\langle ms _{ id },ag_{ms},\mathcal {d}_{ ms },\mathcal {m}_{ ms } \rangle \) that will populate the system, represented by “\(\{ \ldots \} \rightarrow _{ms} \{\langle ms _{ id },ag_{ms},\mathcal {d}_{ ms },\mathcal {m}_{ ms } \rangle \ldots \}\)”, i.e., the set of microservices is updated with that particular new microservice. such request should include all information necessary to create the microservice, including the microservice identifier \( ms _{ id } \), the set of agents, \(ag_{ms} \), that will populate that microservice, the relevant data to keep at the microservice level, \(\mathcal {d}_{ ms } \), and a message pooling for that microservice, \(\mathcal {m}_{ ms } \). in the semantic rule createagents, we show how the system treats a request for the creation of a new agent inside a particular microservice \( ms _{ id } \). considering that this request comes from a source that is external to the system, this request first will be treated by the system message pooling, as we show in the semantics rule createagents, which forwards the message to the corresponding microservice \( ms _{ id } \). note that, at (a), both components \(\mathcal{m}\mathcal{s}_{s} \) and \(\mathcal {m}_{ s } \) are updated, give that updates at (b) always will cause an update to the \(\mathcal{m}\mathcal{s}_{s} \) component, given it represents the set of microservices populating that system. when the microservice treats a request for creating an agent, represented in the semantic rule createagentms, by selecting the message \(\langle \textit{sender}, ms _{ id }, \) \(\texttt {create\_agent}, \langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \rangle \) from its message pooling \(\mathcal {m}_{ ms } \), it updates the message pooling component, \(\mathcal {m}_{ ms } \), and the set of agents populating that microservice, \(ag_{ms} \), including the agent \(\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \) to the set of agents populating that microservice, represented in (c) by “\(\{ \ldots \} \rightarrow _{ag} \{\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \ldots \}\)”. note that, similar to the previous semantic rule, both components at (b) are updated: the set of agents populating the microservice \( ms _{ id } \) represented by \(ag_{ms} \) and the messsage pooling \(\mathcal {m}_{ ms } \). we assume that external communication is treated by the system, in which an interface for authentication could be implemented through the message pooling component, \(\mathcal {m}_{ s } \). however, microservices communicate directly through shared artifacts, which reduces the chattiness [28]. in the next semantic rules, we present an example of communication between microservices, following the message format introduced above. in the semantic rule sendrequestinfoms, an agent \( ag _{ id } \) selects a goalfootnote 2 to be executed using the goal selection function \(s_g()\), which returns the goal \(\texttt {request(}\psi \texttt {,} ms _{ 2 } \texttt {)}\). consequently, \( ag _{ id } \) posts a message \(\langle ms _{ 1 }, ms _{ 2 }, \texttt {request}, \psi \rangle \) in the message pooling of the microservice \( ms _{ 2 } \), requesting the information \(\psi \) from the microservice \( ms _{ 2 } \). then, the agent \( ag _{ id } \) updates its goals, considering that it reached that goal. note that \( ag _{ id } \)’s goals are updated at (c), which causes an update to the set of agents, \(ag_{ ms _{ 1 }} \), populating the microservice \( ms _{ 1 } \) at (b). also, the message pooling of the microservice \( ms _{ 2 } \) is updated with that new message at (b). finally, given that both microservices \( ms _{ 1 } \) and \( ms _{ 2 } \) have been updated, consequently the set of microservices of the system \( s _{ id } \), \(\mathcal{m}\mathcal{s}_{s} \), also is updated at (a). note that, the operational semantics allows us to specify multiple elements from the sets of microservices and agents. in order to make it clear to which set of agents we refer at the transition (c), those sets are labelled with their respective microservices. in the semantic rule sendrequestinfoms, \(\{\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \ldots \}_{ ms _{ 1 }}\) represents the set of agents \(\{\langle ag _{ id },\mathcal {g}_{ ag },\mathcal {b}_{ ag },\mathcal {a} \rangle \ldots \}\) populating the microservice \( ms _{ 1 } \). in the semantic rule receiverequestinfoms, the microservice \( ms _{ 2 } \) selects the message \(\langle ms _{ 1 }, ms _{ 2 }, \texttt {request}, \psi \rangle \) from the microservice message pooling \(\mathcal {m}_{ ms _{ 2 } } \) through the select message function \(s_m()\). also, considering that \( ms _{ 2 } \) owns the requested information, i.e., \(\psi \in \mathcal {d}_{ ms _{ 2 } } \), it answers the request message posting a message \(\langle ms _{ 1 }, ms _{ 2 }, \texttt {answer}, \psi \rangle \) in the message pooling of the microservice \( ms _{ 1 } \). both messages pooling are updated at (b), there is no update at (c), and, given that two microservices of the system have been updated, then the set of microservices of that system, \(\mathcal{m}\mathcal{s}_{s} \), also is updated at (a). finally, in the semantic rule receiveinfoms, the microservice \( ms _{ 1 } \) selects the message \(\langle ms _{ 2 }, ms _{ 1 }, \texttt {answer}, \psi \rangle \) from its message pooling \(\mathcal {m}_{ ms _{ 1 } } \) through the select message function \(s_m()\). then, the message is removed from the message pooling \(\mathcal {m}_{ ms _{ 1 } } \), and the microservice data \(\mathcal {d}_{ ms _{ 1 } } \) is updated with that information. in that semantic rule, no agent is updated, thus there is no update at (c). at (b), both, the message pooling \(\mathcal {m}_{ ms _{ 1 } } \) and data \(\mathcal {d}_{ ms _{ 1 } } \) components of the microservice \( ms _{ 1 } \) are updated, and; therefore, the set of microservices of that system, \(\mathcal{m}\mathcal{s}_{s} \), also is updated at (a). 4 example taxi service application in this section, we discuss how our approach for multi-level semantics for multi-agent microservices can be used to specify systems that already have been implemented according to the mams paradigm. in particular, we discuss how some processes from a taxi service application, described in [2], could be formalised using our approach. in [2], the authors propose a multi-agent system that is responsible for providing a taxi allocation service, in which taxis drivers dynamically enter the system, becoming available for allocation. when a client requests the system for transportation, the microservice responsible for treating requests in that particular area that the client departs from allocates one of the available taxis drivers to that client. in the system, different areas are covered by different microservices according to an architecture based on mams [9], and the microservices are dynamically created according to the system’s demand. figure 2 shows the dynamism of the microservices according to the areas covered by the system. each area is covered by a microservice responsible for registering, receiving requests, and allocating taxis drivers. fig. 2. dynamic reorganisation for geometric treatment. it shows an evolution from the top-left configuration to the bottom-right configuration [2]. full size image in the system proposed by [2], one of the main operations is to create a new microservice always when a particular region (controlled by a microservice) has too many registered taxis drivers and requests, spliting such region into two small regions, one of them controlled by the new microservice. this situation can be observed in fig. 2, in which region a01 is divided in two regions, the old microservice remains responsible for a01, and a new microservice becomes responsible for that new region a04. first, the agent called controller01, which populates and is responsible for the region a01, requests to the taxi service system tx01 to create a new microservice a04, populated by controller04, which will be responsible for that region and its registered taxis, i.e., \(\vardelta _{a01'}\) contains all registered taxis in the region of a04. the rule createmicroservicea04 shows how tx01 processes this message creating the new microservice a04. in the application proposed in [2], each user’s application (clients and taxis drivers) runs an agent that interacts with the user. while clients’ application can request for taxi drivers, taxi drivers’ applications allow them to register in the systems, being notified always that a client is allocated to them. continuing our example, imagine that a client in region a01 requests a taxi driver to go from a particular address at region a01 to another address at region a04. also, imagine that the microservice responsible for a01 has no taxi drivers available, thus it requests to microservice a04 the closest taxi driver in the region a04. this request can be observed in the semantic rule sendrequestinfomsa04. in the semantic rule sendrequestinfomsa04, the controller agent, in the microservice a01, responsible for allocating taxi drivers to clients, i.e., controller01, selects the goal request(closest(td,a01),a04) from its goals, which can be achieved by sending a message to a04 requesting the information closest(td,a01). thus, a message of the type \(\langle a01, a04, \texttt {request}, closest(td,a01),a04) \rangle \) is posted in the message poll \(\mathcal {m}_{a04}\) and its goals are updated accordingly. in the semantic rule receiverequestinfomsa01, when the microservice a04 selects that message from its message pool \(\mathcal {m}_{a04}\), and the microservice a04 has that particular information in their data base \(\mathcal {d}_{a04}\), then it answer that particular request posting a message \(\langle a04, a01, \texttt {answer}, closest(tx245,a01),a04) \rangle \) into a01’s message poll \(\mathcal {m_{a01}}\), informing that the taxi tx245 is the closest taxi from a01. finally, in the semantic rule receiveinfomsa04, when the microservice a01 receives the requested information, selecting that message from the message poll \(\mathcal {m}_{a01}\), it updates its database \(\mathcal {d}_{a01}\) with that particular information, also removing the message \(\langle a04, a01, \texttt {answer}, closest(tx245,a01),a04) \rangle \) from its message poll. now, a01 microservice is able to allocate that taxi driver tx245 to that particular client. 5 related work in [15], the authors propose an agent infrastructure layer (ail) for bdi-style programming languages, aiming to (i) provide a common semantic foundation for various bdi languages, and (ii) support formal verification through an optimized model-checker. they design ail based on comprehensive operational semantics from [14], claiming it encompasses all key features of common bdi languages. in [13], the authors introduce operational semantics for modularisation, which they argue simplifies the implementation of agents, roles, and profiles—key aspects in structured programming. they provide operational semantics for creating, executing, testing, updating, and realizing module instances, essential for agent programming. in [17], operational semantics are provided for managing goals—dropping, aborting, suspending, and resuming them. the authors argue that these semantics clarify how an agent handles its goals and support the verification of correct agent behavior. they emphasize that this work contributes to a detailed specification of appropriate operational behavior when a goal is pursued, succeeds, fails, or undergoes changes by the agent. in [11], the author proposes an operational semantic framework for legal systems within agent societies, modeling their structure and dynamics. this work focuses on actions within legal systems (internal and external legal acts, social acts, etc.) under an action-based dynamics, where system transitions are determined by actions of legal organs and legal subjects. it exemplifies the use of operational semantics to formalize multi-level systems. in [29], the authors propose a semantic framework for \(\mathcal {m}\textsc {oise}^+\) [19] and a linear temporal logic (ltl) to express its properties. they argue that organizational specifications help agents achieve their goals and prevent undesired behavior by imposing organizational constraints on agents’ behavior, which they must consider in decision-making. the work defines the formal semantics of \(\mathcal {m}\textsc {oise}^+\), focusing on ensuring agents respect (i) preconditions for organizational actions and (ii) acquaintance and communication links. as mentioned before, the use of operational semantics has been quite extensive in multi-agent languages so we cannot cover all that literature here. our work differs from all those in that we present a new style for the semantics of multi-level systems which has been developed to the formalisation of multi-agent microservices. we are not focused on a special semantics for bdi languages as in [15], in semantics for module-related actions [13], in semantics for goals [17], in semantics for legal systems [11], or semantics for the \(\mathcal {m}\textsc {oise}\) model [29] specifically. we focus on proposing an operational semantics to the formalisation of multi-agent microservices that takes into account the desired characteristics of the software engineering based on microservices, i.e., modularity, scalability, reusability, and the specification of shared communication artifacts. to the best of our knowledge, we are the first to propose a formalisation for multi-agent microservices, providing a template that considers the multiple levels of such systems, and also promotes those desired characteristics. 6 conclusion in software engineering, there are many benefits of formal specification of systems [1, 20]: (i) it allows the proof of formal properties; (ii) it allows a concise description of high-level behaviour of the system; (iii) it eliminates imprecision and ambiguity; and (iv) it allows model checking the system. while operational semantics has been almost a standard method for the formalisation of agent-oriented programming languages, only recent research has been applied to make those formalisations adequate to the multiplicity of levels of abstraction (i.e., organisation, groups, agents, environment, etc.) present when engineering multi-agent systems, and there is no such work on the formalisation of multi-agent microservices. following those recent approaches, in this work, we introduced an approach to the formalisation of multi-agent microservices using operational semantics; we use an approach based on the multi-level semantics proposed by [24]. we introduced a template for semantic rules that provide separated transition systems to multiple levels of multi-agent microservices. we also demonstrate how our template can be used to formalise some operations commonly present in multi-agent microservices, detailing how different (sets of) components are represented through the semantic rules. our approach allows the representation of the interactions between components of different levels. given the complexity and ubiquity of such multiple levels in multi-agent microservices, the approach seems to allow for a clearer understanding of such complex semantics, improving its readability. furthermore, with the multi-level representation, our operational semantics style also provides independence between the specification of the various levels present in multi-agent microservices, so that each level is formalised through its own transition system. this promotes the modularity and reusability of formal models. using the multiple levels of formalisation, it is possible to make clear the representation of shared communication artifacts and how the different components of the system interact with such shared components, as we have illustrated in this paper through the formalisation of abstract operation for communication in multi-agent microservices. shared communication artifacts are considered an essential part of the paradigm based on microservices, given they avoid chattiness [28]. in our future work, we intend to explore other relations between the multiple levels of multi-agent microservices. also, we intend to investigate how our multi-level semantics and shared communication artifacts can play a part in research towards model checking of such systems [6], towards the verification of multi-agent microservices. notes 1.for all semantics rules presented in this section, when there is no update at some of the three levels, that transition is omitted from the rule. 2.note that, considering the practical reasoning to reach that goal is out of the scope of this paper, thus we assume that such goal is reached by the agent executing that particular action. for details about the mental attitudes we use in this paper from the bdi architecture we refer to bratman’s paper [8]. references beckert, b., et al.: intelligent systems and formal methods in software engineering. ieee intell. syst. 21(6), 71–81 (2006) article  math  google scholar  blanger, l., junior, v., jevinski, c.j., panisson, a.r., bordini, r.h.: improving the performance of taxi service applications using multi-agent systems techniques. in: encontro nacional de inteligência artificial e computacional (eniac) (2017) google scholar  boissier, o., bordini, r.h., hubner, j., ricci, a.: multi-agent oriented programming: programming multi-agent systems using jacamo (2020) google scholar  boissier, o., bordini, r.h., hübner, j.f., ricci, a., santi, a.: multi-agent oriented programming with jacamo. sci. comput. program. 78(6), 747–761 (2013) article  math  google scholar  bordini, r.h., dastani, m., dix, j., seghrouchni, a.e.f.: multi-agent programming: languages, 1st edn. tools and applications. springer publishing company, incorporated (2009) math  google scholar  bordini, r.h., fisher, m., visser, w., wooldridge, m.: verifying multi-agent programs by model checking. auton. agent. multi-agent syst. 12(2), 239–256 (2006) article  math  google scholar  bordini, r.h., hübner, j.f., wooldridge, m.: programming multi-agent systems in agentspeak using jason. john wiley & sons (2007) google scholar  bratman, m., et al.: intention, plans, and practical reason, vol. 10. harvard university press cambridge, ma (1987) google scholar  collier, r.w., o’neill, e., lillis, d., o’hare, g.m.: mams: multi-agent microservices. in: world wide web conference (2019) google scholar  collier, r.w., russell, s., lillis, d.: reflecting on agent programming with agentspeak (l). in: international conference on principles and practice of multi-agent systems, pp. 351–366. springer (2015). https://doi.org/10.1007/978-3-319-25524-8_22 rocha costa, a.c.: situated legal systems and their operational semantics. artif. intell. law 23(1), 43–102 (2015). https://doi.org/10.1007/s10506-015-9164-z article  math  google scholar  costa, a.c.r., dimuro, g.p.: introducing social groups and group exchanges in the poporg model. in: proceedings of the 8th international conference on autonomous agents and multiagent systems-volume 2, pp. 1297–1298. international foundation for autonomous agents and multiagent systems (2009) google scholar  dastani, m., steunebrink, b.r.: operational semantics for bdi modules in multi-agent programming. in: computational logic in multi-agent systems, pp. 83–101. springer (2010). https://doi.org/10.1007/978-3-642-16867-3_5 dennis, l.a.: agent infrastructure layer (ail): design and operational semantics v1. tech. rep., technical report ulcs-07-001, department of computer science, university of liverpool (2007) google scholar  dennis, l.a., farwer, b., bordini, r.h., fisher, m., wooldridge, m.: a common semantic basis for bdi languages. in: programming multi-agent systems, pp. 124–139. springer (2008). https://doi.org/10.1007/978-3-540-79043-3_8 dragoni, n., lanese, i., larsen, s.t., mazzara, m., mustafin, r., safina, l.: microservices: how to make your application scale. in: international andrei ershov memorial conference on perspectives of system informatics, pp. 95–104. (2017) google scholar  harland, j., morley, d.n., thangarajah, j., yorke-smith, n.: an operational semantics for the goal life-cycle in bdi agents. auton. agent. multi-agent syst. 28(4), 682–719 (2014) article  math  google scholar  horling, b., lesser, v.: a survey of multi-agent organizational paradigms. knowl. eng. rev. 19(4), 281–316 (2004) article  math  google scholar  hubner, j.f., sichman, j.s., boissier, o.: developing organised multiagent systems using the \(moise^+\) model: programming issues at the system and agent levels. int. j. agent-oriented softw. eng. 1(3–4), 370–395 (2007) article  math  google scholar  leveson, n.g.: guest editor’s introduction: formal methods in software engineering. ieee trans. software eng. 16(9), 929 (1990) math  google scholar  noriega, p.: agent mediated auctions: the fishmarket metaphor. institut d’investigació en intelligència artificial (1999) google scholar  panisson, a.r., ali, a., mcburney, p., bordini, r.h.: argumentation schemes for data access control. in: comma, pp. 361–368 (2018) google scholar  panisson, a.r., bordini, r.h., rocha, a.c.: an approach to the multi-level semantics of agent societies. in: brazilian conference on intelligent systems (2020) google scholar  panisson, a.r., bordini, r.h., da rocha costa, a.c.: multi-level semantics with vertical integrity constraints. in: 22st european conference on artificial intelligence (ecai), pp. 1708 – 1709 (2016) google scholar  panisson, a.r., meneguzzi, f., fagundes, m., vieira, r., bordini, r.h.: formal semantics of speech acts for argumentative dialogues. in: thirteenth int. conf. on autonomous agents and multiagent systems, pp. 1437–1438 (2014) google scholar  plotkin, g.d.: a structural approach to operational semantics (1981) google scholar  ricci, a., piunti, m., viroli, m.: environment programming in multi-agent systems: an artifact-based perspective. auton. agent. multi-agent syst. 23(2), 158–192 (2011) article  math  google scholar  richards, m.: microservices vs. service-oriented architecture. o’reilly media (2015) google scholar  van riemsdijk, m.b., hindriks, k.v., jonker, c.m., sierhuis, m.: formalizing organizational constraints: a semantic approach. in: proceedings of the 9th international conference on autonomous agents and multiagent systems: volume 1-volume 1, pp. 823–830. international foundation for autonomous agents and multiagent systems (2010) google scholar  sierra, c., rodriguez-aguilar, j.a., noriega, p., esteva, m., arcos, j.l.: engineering multi-agent systems as electronic institutions. europ. j. infor. profess. 4(4), 33–39 (2004) math  google scholar  vieira, r., moreira, a., wooldridge, m., bordini, r.h.: on the formal semantics of speech-act based communication in an agent-oriented programming language. j. artif. int. res. 29(1), 221–267 (2007) math  google scholar  vivanco, m.e., garcía, c.s.: electronic institutions: from specification to development. consell superior d’investigacions científiques, institut d’investigació en intelligéncia artificial (2003) google scholar  wooldridge, m.: an introduction to multiagent systems. john wiley & sons (2009) google scholar  zimmermann, o.: mircroservices tenets: agile approach to service development and deployment. in: proceedings of the symposium/summer school on service-oriented computing (2016) google scholar  download references author information authors and affiliations department of computing (dec), federal university of santa catarina (ufsc), araranguá, brazil alison r. panisson & giovani p. farias authors alison r. panissonview author publications search author on:pubmed google scholar giovani p. fariasview author publications search author on:pubmed google scholar corresponding author correspondence to alison r. panisson . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper panisson, a.r., farias, g.p. (2025). a multi-level semantics formalism for multi-agent microservices. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_4 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-agent systems microservices multi-level semantics publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a performance increment strategy for semantic segmentation of low-resolution images from damaged roads | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a performance increment strategy for semantic segmentation of low-resolution images from damaged roads conference paper first online: 30 january 2025 pp 80–95 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) a performance increment strategy for semantic segmentation of low-resolution images from damaged roads download book pdf download book epub rafael s. toledo9, cristiano s. oliveira9, vitor h. t. oliveira9, eric a. antonelo9 & … aldo von wangenheim9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 484 accesses abstract autonomous driving needs good roads, but 85% of brazilian roads have damages that deep learning models may not regard as most semantic segmentation datasets for autonomous driving are high-resolution images of well-maintained urban roads. a representative dataset for emerging countries consists of low-resolution images of poorly maintained roads and includes labels of damage classes; in this scenario, three challenges arise: objects with few pixels, objects with undefined shapes, and highly underrepresented classes. to tackle these challenges, this work proposes the performance increment strategy for semantic segmentation (pisss) as a methodology of 14 training experiments to boost performance. with pisss, we reached state-of-the-art results of 79.8 and 68.8 miou on the road traversing knowledge (rtk) and technik autonomer systeme 500 (tas500) test sets, respectively. furthermore, we also offer an analysis of deeplabv3+ pitfalls for small object segmentation (code available on https://github.com/tldrafael/pisss). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others paved and unpaved road segmentation using deep neural network chapter © 2020 visual scene understanding for autonomous driving using semantic segmentation chapter © 2019 pixel-wise detection of road obstacles on drivable areas by introducing maximized entropy to synboost framework chapter © 2023 1 introduction autonomous driving research is mainly based on developed countries with well-maintained infrastructure represented by many european urban streets datasets of high-res images, e.g., cityscapes [7], camvid [2], and kitti [9]. in brazil, 85% of the roads suffer from fatigue, cracks, holes, patches, and wavy surfaces [6]. this poor condition demands adjustments in the perception of autonomous driving. additionally, computational constraints in emerging countries may limit the input size for the deep learning models, constraining the usage of high-res images and forcing adapted solutions for low-res images. some datasets such as [17, 20, 22] represented emerging countries’ roads. among them, especially the rtk dataset [17] captures the brazilian countryside road featuring distinct maintenance conditions and surfaces. rtk consists of 701 annotated images of resolution 352x288 with 12 classes. the classes include surfaces (asphalt, paved, and unpaved), signs (markings, cat’s eyes, speed bumps, and storm drains), and damages (patches, water-puddles, potholes, and cracks). see fig. 1. fig. 1. examples of emerging countries’ roads in the rtk dataset. full size image this work raised three challenges for training a deep learning model when working with low-res images: objects with few pixels, objects with undefined shapes, and highly underrepresented classes. low-res images have small objects not only relative to other objects’ sizes but in pixels’ quantity; for example, 70% of the cat’s eyes blobs have edges with equal or less than 5 pixels, and 15% of the road markings blobs even have edges of a unique pixel. these tiny objects can easily vanish at the beginning of the forward pass given the stride of convolutional and pooling layers. the vanishing objects became a problem for deeplabv3+, as will be seen in sect. 4.2. other issues are undefined shape objects or multiscale elements presence in the image, e.g., road surfaces are broad and have a well-defined shape, whereas patches do not have a defined shape and size. this problem complexity increases with intraclass shape variations, like holes and cracks in the same image with multiple formats. lastly, small damages and signs become very underrepresented, e.g., background and road surfaces occupy 98.5% of pixels, while cat’s eye and storm-drain are just 0.02%. the imbalanced scenario raises the risk of overlooking small-sized classes. 1.1 contributions to meet the challenges presented above, we designed the performance increment strategy for semantic segmentation (pisss) that consists of a series of good training practices found in the state-of-the-art (sota) works that tackle semantic segmentation challenges of imbalanced datasets, small objects, and multiscale segmentation. with pisss, we raised the rtk benchmark to 79.8 miou and the tas500 to 68.8 miou, the best published results so far. furthermore, we also propose removing the resnet’s max-pooling (mp) layer to preserve small objects segmentation. 2 background in this section, we introduce the main topics covered in the pisss strategy, they are: training procedures, small objects segmentation, and multiscale segmentation. 2.1 guidelines and training procedures a significant part of deep learning success comes from adopting better training procedures. however, they are not usually the main research focus, and their details may be hidden in the implementation code. the bag of tricks for image classification [11] is a fundamental work for training recipes with convolutional neural networks, and it pushed resnet-50’s imagenet top-1 validation accuracy from 75.3% to 79.29%. [1, 24] also present common procedures like warm-up learning rate (lr), cosine lr decay, weight decay, label smoothing, stochastic depth, dropout, mixup, cutmix, and random resized cropping. furthermore, standard practices in semantic segmentation sota works [5, 14, 25,26,27] are cropping, resizing, and flipping as data augmentation; stochastic gradient descent (sgd) with a polynomial lr decay, momentum of 0.9, and weight decay of 5e-4 as optimizer; and multiscale ensemble predictions for testing. the sgd preference over adam is explained by its better generalization results [13, 28]. important takeaways are pointed out by [24] that there is no training procedure ideal for all models, and by [1] that training methods are more task-driven than architectures, and, hence, improvements from training methods do not necessarily generalize as well as architectural ones. 2.2 small objects segmentation small object segmentation is a challenge for low-res datasets. [10, 12] emphasized the importance of context for small object detection, underlining that even humans cannot recognize a small building in a satellite image without the context of roads, cars, or other buildings. hence, neurons with a large receptive field (rf) are essential for the task. a simpler option was found by [19] that noticed that the double sequence of stride 2 on resnet loses sensible feature information of small objects; they proposed replacing the resnet 7x7 convolution layer with stride 2 to a series of three 3x3 with stride only at the end. this same tactic was followed later by [15, 29]. 2.3 multiscale segmentation and deeplabv3+ some approaches for handling the multiscale segmentation challenge are to extract multiscale features in a layer level like res2net [8] and to add an attention mechanism that smartly combines predictions from different feature map scales to avoid scale pitfalls [4, 21]. [21] noted that feature maps of large scales predict better fine details, such as edges of objects or thin structures, whereas feature maps of small scales predict better large structures that demand global context. deeplabv3+ [3] proposed the atrous spatial pyramid pooling (aspp) module in charge of capturing multiscale features by simultaneously applying various dilated convolutions, hence, combining multiple receptive fields. aspp balances the trade-off between accurate localization (small receptive field) and context assimilation (large receptive field). 3 pisss performance increment strategy for semantic segmentation pisss is a methodology that consists of an additive series of ablation experiments. each experiment checks the best performance among a set of hypotheses; e.g., which augmentation operation works better for the rtk dataset? geometry operation? color operations? both together? each option is called a hypothesis. the ablation experiments answer the questions of the best hypothesis for part of the training setup. the next ablation experiment is built on top of the best setup until that moment. in total, the pisss applied on the rtk dataset sums 14 ablation experiments organized into four categories: baseline (b), prediction (p), technique (t), and architecture (a). baseline checks choices of the rtk authors training in [18], prediction checks the usage of prediction ensemble, technique checks training setup specificities, and architecture checks changes in the neural network structure. figure 2 shows the experiments tried in each category. fig. 2. pisss diagram. full size image the sequence of experiments for pisss starts on the baseline work, and we also adopted a different evaluation strategy as discussed next. 3.1 baseline we adopted the rtk authors’ solution [18] as the starting point for the sequence of experiments. we call it baseline. it consists of a u-net with resnet-34, adam optimizer with lr of 1e-4, batch size of 8, data augmentation with perspective distortion and horizontal flipping, named as geomrtk, and a two-stage training regime that first runs 100 epochs with cross-entropy (ce) and another 100 later with weighted cross-entropy (wce), summing 14k iterations at total. 3.2 evaluation methodology when monitoring our training experiments, we noticed slight miou variations after the loss convergence. in fig. 3, it is seen an oscillation around [0.733,0.743], which represents 1% of the miou scale of [0,1]; this variation is enough to lead to an incorrect conclusion when comparing the hypotheses. we adopted a workaround to reduce this noisy variation by averaging the last ten results steps of the validation set. fig. 3. miou oscillation after loss convergence. full size image 4 applying pisss on rtk in this section, we present the pisss application over the rtkfootnote 1. we split it into 5 main parts that most leverage performance, following the chronological order: 1) surpassing the baseline, 2) tuning deeplabv3+, 3) fancy approaches and cutmix, 4) loss functions, and 5) prediction ensemble. 4.1 part 1 surpassing the baseline in the first part, we covered the five first experiments: iterations (b), single stage training (b), data augmentation (t), encoder depths (a), and resnet variants (a). see table 1; the order follows from top to bottom. every subsequent experiment is built over the best hypothesis until that moment unless it is otherwise said, and the best hypothesis until that time has its miou value in bold. table 1. part 1 of pisss.full size table in the iterations experiment, we ensured that 14k iterations were short for training, and it proved necessary to push the iterations up to 200k. in the next experiment, single stage training, we discarded any benefit from the two-stage training regime or wce. later, in the data augmentation experiment, we adopted cropping of (224, 224), random edge resizing with scale [0.78, 2], and color augmentation with grayscale and jitter of 0.27. cropping alone provided the best result. the addition of resizing or color augmentation worsened the results, cutmix alone made little impact, and no augmentation hypothesis had a terrible performance, revealing the need for data augmentation. subsequently, we tried a deeper resnet version, which performed better. after, we tried resnet variants: res2net, which uses different scales within the resnet module; resnest, which adds an attention mechanism; and resnext, which works with grouped convolutions. no variant lifted the performance. 4.2 part 2 tuning deeplabv3+ in this section, we cover seven more experiments: model architecture (a), output stride (a), max-pooling removal (a), transposed convolutions (a), hybrid local feature extractor (a), cutmix (t), and optimizer (t). see table 2. table 2 experiment follows the best setup from table 1 experiments. in the model architecture experiment, we tried deeplabv3+ (dl3+). it did not outperform u-net in the first trials. however, after controlling the usage of the max-pooling (mp) layer and the output stride (os), it reached a higher performance. in the os experiment, in which we control the dimension ratio between the input and the encoder’s outcome, there was a clear trend that reducing os increases performance. later, in the experiment that we suggest the mp removal, this trend reverted, and a higher os reached the best performance. this behavior’s change in os is interpreted in the next section. table 2. part 2 of pisss. abbreviations: arch (architecture), wo/ mp (without mp layer), r50/101 (resnet-50/101).full size table interpreting deeplabv3+ patterns. it is essential to point out that the decoder of dl3+ concatenates high-level (hl) features from the aspp block and low-level (ll) features from the resnet’s stem. see fig. 4 for a vanilla example of the architecture and the control of the dimensions. the os experiment’s parameter just controls the final hl feature dimensions. on the other hand, the usage of the mp layer controls the ll dimensions; for example, without the mp layer, the ll features come just after the 7x7 conv layer with a stride of 2; otherwise, the features come just after the mp layer with a stride of 4. fig. 4. low and high-level features connections in deeplabv3+. full size image removing the mp layer positively impacted the os of 8 and 16, whereas it did not matter for os 4. a smaller ll stride seems more crucial than the hl stride for dl3+. thus, when concatenating a lower ll stride with a higher hl one, it joins the best of both contexts, explaining the results of table 2. another pattern found that is quantitatively unnoticeable but qualitatively impactful was that, with a higher os, the model randomly predicts small background blobs over the road and small road blobs over the background. this problem gradually vanished until the os decreased to 4. besides, the problem was also solved after the mp removal; see figs. 5 and 6. 4.3 part 3 fancy approaches and cutmix in this section, we cover four more experiments: transposed convolutions (a), hybrid local feature extractor (a), cutmix (t), and optimizer (t). see table 3. we tried two fancy approaches, and none brought a performance improvement. first, we tried replacing the non-parametric upsampling with a transposed convolution layer. next, we implemented a hybrid local feature extractor (hlfe) that joins the digressive dilation rates [10] and the hybrid dilation rates [23]. for hlfe, we implemented the following dilation rates of [1, 3, 5, 5, 3, 1] for block 3 and [1, 3, 1] for block 4 of the resnet. it is impossible to try hlfe for os 16 as it has no dilation rate. we also tried sgd over adam, and it neither presented any improvement. the sgd setup had lr of 1e-2 with a linear warm-up of 5k iterations, poly lr decay, and momentum of 0.9. on the other hand, trying the cutmix augmentation together with cropping had a quite effective impact, raising performance from 76.9 to 78.2 miou. we tested it with probability occurrence of 50% and 80%. fig. 5. comparing results of distinct os w/ or wo/ mp layer. the subtitles have the os and mp states and the prediction iou result. removing the mp layer avoids early spatial information loss for extracting small object features. full size image table 3. part 3 of pisss. abbreviations: convt (transposed convolution).full size table 4.4 part 4 loss functions we tried the surrogate losses of miou and dice. it did not present any advantage on a well-calibrated training setup, although it does help the baseline simpler training setup, see table 4. we noticed the surrogate losses alone degrade performance, whereas ce acts like a proxy for miou, optimizing it even better than its loss. fig. 6. comparing results of distinct os w/ or wo/ mp layer. full size image table 4. part 4 of pisss. abbreviations: r34/50 (resnet-34/101).full size table table 5. part 5 of the pisss experiments.full size table 4.5 part 5 prediction ensemble in contrast with the evaluation methodology applied so far, this experiment counts on the evaluation metrics of a single checkpoint (ckpt), either from the last training step or from the step with the best validation result; hence, the values presented in table 5 diverge from the previously reported values. we adopted the 288 \(\times \) 224 and 448 \(\times \) 352 resolutions for multiscale predictions besides the 352 \(\times \) 288 native one. ultimately, the flipped ensemble got the best results, with 78.9 and 79.8 miou for the last and best checkpoints. 5 rtk experiments’ analysis this section first analyzes how the classes’ size and pixel quantity impact performance and, next, shows what the trained model looks for in each class. the results used for the analysis are from the best pisss hypothesis, i.e., the checkpoint from the cutmix (t) experiment. 5.1 performance by classes and groups in fig. 7, we see a clear correlation between the miou metric and the object size, confirming a miou bias toward big objects, also pointed out in [7]. furthermore, the worst performances by class are from cracks, water puddles, and cats’ eyes, either classes of tiny objects or undefined shapes. fig. 7. relation between class characteristics and miou. (\(^a\)the median size of the shortest edge of the class object. \(^b\)the ratio of the number of pixels between the most popular class and the class i.) full size image 5.2 what does the neural network look for? one way to understand how the neural network perceives a dataset category is by optimizing the input neurons to maximize output probability. we optimized the network’s inputs using gradient ascent, shown in fig. 8. it highly noticed different textures, color distribution, and geometric patterns attached to each class; for example, storm-drain presents black holes, road-paved has the presence of polygon structures, and road-asphalt also seems to capture the cracks that happen over the road surface. fig. 8. classes optimized inputs. full size image 6 applying pisss on tas500 the tas500 dataset [16] is a dataset from 2021 that meets unstructured environments with annotations of fine-grained vegetation and terrain classes to distinct drivable surfaces and natural obstacles. tas500 has high-res (hr) images of 2026 \(\times \) 620, which turned prohibitive the experiments of mp layer removal and transposed convolutions due to the gpu 16 gb memory. besides, the batch size had to be reduced to 4. moreover, the images were trained with cropped parts of 1024 \(\times \) 512, a standard practice for training hr images. furthermore, we skipped the baseline experiments and only applied a subset of the pisss hypotheses used on rtk. so the experiments’ setup started with deeplabv3+, resnet50, os 16, adam with lr of 5e5 (a reduced value from the 1e-4 in the rtk experiments, given the reduced batch size). for validation evaluation, we followed the same methodology of averaging the results of the last ten steps. see tables 6 and 7. we found that resizing as data augmentation, ce+dice loss, sgd, and multiscale ensemble prediction were fundamental for raising the tas500 benchmark. the pisss raised the validation set results from 65.4 to 74.7 miou. furthermore, we checked our best hypothesis model on the outdoor semantic segmentation challenge, and it reached 68.8footnote 2 miou, surpassing the 2021 1st place of 67.5 miou. table 6. summary of the pisss experiments on tas500. abbreviations: res (resizing), cm (cutmix).full size table table 7. prediction ensemble on tas500.full size table 7 discussion and findings although pisss worked for rtk and tas500 datasets, a very different subset of hypotheses was the best in each case. for rtk, when using the mp layer, cutmix, os 4, ce loss, and adam worked better; while for tas500, resizing, os 16, ce+dice loss, and sgd worked better. the dissimilarity between these two training setups endorses the need for a custom solution, which also corroborates with the ideas [24] that there is no ideal training procedural for all models. the cutmix augmentation had a meaningful gain of 1.3 miou for rtk, but it did not help tas500; we suppose that cutmix is more helpful for tricky scenes of rough transitions between the road surfaces and damage classes. finally, we summarize all findings on the following items: segmenting small objects is problematic if the features’ dimensions are reduced before producing deep features; the model tends to overpredict small objects. the hardest damage classes to segment based on the rtk results are cracks, water puddles, and cat’s eyes. we confirmed the miou towards big objects. ce optimizes the miou metric better than miou and dice losses. an experiment performance gain depends on the initial setup that it was tested. the high dissimilarity between the best training setup for each dataset endorses the need for custom solutions. conventional setups usually bring better results than fancy procedures and should be the first attempt. 8 conclusions pisss was effective for rtk and tas500, reaching sota results and showing the importance of a well-tuning training setup besides the bare choice of a neural network architecture. moreover, we warned of the pitfalls of early stride on convolutional networks when working with tiny objects and road damage in low-res images. we also highlighted the potential cause of problems for false-positive blobs on deeplabv3+ predictions due to early large strides. notes 1.https://data.mendeley.com/datasets/hssswvmjwf/1. 2.results with user slow on https://codalab.lisn.upsaclay.fr/competitions/5637#results.. references bello, i., et al.: revisiting resnets: improved training and scaling strategies. adv. neural. inf. process. syst. 34, 22614–22627 (2021) math  google scholar  brostow, g.j., fauqueur, j., cipolla, r.: semantic object classes in video: a high-definition ground truth database. pattern recogn. lett. 30(2), 88–97 (2009) article  google scholar  chen, l.c., papandreou, g., kokkinos, i., murphy, k., yuille, a.l.: deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. ieee trans. pattern anal. mach. intell. 40(4), 834–848 (2017) article  math  google scholar  chen, l.c., yang, y., wang, j., xu, w., yuille, a.l.: attention to scale: scale-aware semantic image segmentation. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 3640–3649 (2016) google scholar  chen, l.c., zhu, y., papandreou, g., schroff, f., adam, h.: encoder-decoder with atrous separable convolution for semantic image segmentation. in: proceedings of the european conference on computer vision (eccv), pp. 801–818 (2018) google scholar  cnt: pesquisa cnt de rodovias 2021. sest senat (2021). https://pesquisarodovias.cnt.org.br/downloads/ultimaversao/ cordts, m., et al.: the cityscapes dataset for semantic urban scene understanding. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 3213–3223 (2016) google scholar  gao, s.h., cheng, m.m., zhao, k., zhang, x.y., yang, m.h., torr, p.: res2net: a new multi-scale backbone architecture. ieee trans. pattern anal. mach. intell. 43(2), 652–662 (2019) article  math  google scholar  geiger, a., lenz, p., stiller, c., urtasun, r.: vision meets robotics: the kitti dataset. int. j. robot. res. 32(11), 1231–1237 (2013) article  google scholar  hamaguchi, r., fujita, a., nemoto, k., imaizumi, t., hikosaka, s.: effective use of dilated convolutions for segmenting small object instances in remote sensing imagery. in: 2018 ieeed winter conference on applications of computer vision (wacv), pp. 1442–1450. ieee (2018) google scholar  he, t., zhang, z., zhang, h., zhang, z., xie, j., li, m.: bag of tricks for image classification with convolutional neural networks. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 558–567 (2019) google scholar  hu, p., ramanan, d.: finding tiny faces. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 951–959 (2017) google scholar  keskar, n.s., socher, r.: improving generalization performance by switching from adam to sgd. arxiv preprint arxiv:1712.07628 (2017) kirillov, a., wu, y., he, k., girshick, r.: pointrend: image segmentation as rendering. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 9799–9808 (2020) google scholar  li, y., peng, b., he, l., fan, k., li, z., tong, l.: road extraction from unmanned aerial vehicle remote sensing images based on improved neural networks. sensors 19(19), 4115 (2019) article  math  google scholar  metzger, k.a., mortimer, p., wuensche, h.j.: a fine-grained dataset and its efficient semantic segmentation for unstructured driving scenarios. in: 2020 25th international conference on pattern recognition (icpr), pp. 7892–7899. ieee (2021) google scholar  rateke, t., justen, k.a., von wangenheim, a.: road surface classification with images captured from low-cost camera-road traversing knowledge (rtk) dataset. revista de informática teórica e aplicada 26(3), 50–64 (2019) article  math  google scholar  rateke, t., von wangenheim, a.: road surface detection and differentiation considering surface damages. auton. robot. 45(2), 299–312 (2021). https://doi.org/10.1007/s10514-020-09964-3 article  math  google scholar  shen, z., liu, z., li, j., jiang, y.g., chen, y., xue, x.: dsod: learning deeply supervised object detectors from scratch. in: proceedings of the ieee international conference on computer vision, pp. 1919–1927 (2017) google scholar  shinzato, p.y., et al.: carina dataset: an emerging-country urban scenario benchmark for road detection systems. in: 2016 ieee 19th international conference on intelligent transportation systems (itsc), pp. 41–46. ieee (2016) google scholar  tao, a., sapra, k., catanzaro, b.: hierarchical multi-scale attention for semantic segmentation. arxiv preprint arxiv:2005.10821 (2020) varma, g., subramanian, a., namboodiri, a., chandraker, m., jawahar, c.: idd: a dataset for exploring problems of autonomous navigation in unconstrained environments. in: 2019 ieee winter conference on applications of computer vision (wacv), pp. 1743–1751. ieee (2019) google scholar  wang, p., et al.: understanding convolution for semantic segmentation. in: 2018 ieee winter conference on applications of computer vision (wacv), pp. 1451–1460. ieee (2018) google scholar  wightman, r., touvron, h., jégou, h.: resnet strikes back: an improved training procedure in timm. arxiv preprint arxiv:2110.00476 (2021) yu, c., gao, c., wang, j., yu, g., shen, c., sang, n.: bisenet v2: bilateral network with guided aggregation for real-time semantic segmentation. int. j. comput. vision 129(11), 3051–3068 (2021) article  math  google scholar  yuan, y., chen, x., chen, x., wang, j.: segmentation transformer: object-contextual representations for semantic segmentation. arxiv preprint arxiv:1909.11065 (2019) zhao, h., shi, j., qi, x., wang, x., jia, j.: pyramid scene parsing network. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2881–2890 (2017) google scholar  zhou, p., feng, j., ma, c., xiong, c., hoi, s.c.h., et al.: towards theoretically understanding why sgd generalizes better than adam in deep learning. adv. neural. inf. process. syst. 33, 21285–21296 (2020) google scholar  zhou, p., ni, b., geng, c., hu, j., xu, y.: scale-transferrable object detection. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 528–537 (2018) google scholar  download references author information authors and affiliations federal university of santa catarina, florianópolis, santa catarina, brazil rafael s. toledo, cristiano s. oliveira, vitor h. t. oliveira, eric a. antonelo & aldo von wangenheim authors rafael s. toledoview author publications search author on:pubmed google scholar cristiano s. oliveiraview author publications search author on:pubmed google scholar vitor h. t. oliveiraview author publications search author on:pubmed google scholar eric a. antoneloview author publications search author on:pubmed google scholar aldo von wangenheimview author publications search author on:pubmed google scholar corresponding author correspondence to rafael s. toledo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper toledo, r.s., oliveira, c.s., oliveira, v.h.t., antonelo, e.a., wangenheim, a.v. (2025). a performance increment strategy for semantic segmentation of low-resolution images from damaged roads. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_6 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords unstructured environment road segmentation damaged roads low-resolution deeplabv3+ publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature developing resource-efficient clinical llms for brazilian portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper developing resource-efficient clinical llms for brazilian portuguese conference paper first online: 31 january 2025 pp 46–60 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) developing resource-efficient clinical llms for brazilian portuguese download book pdf download book epub joão gabriel de souza pinto  orcid: orcid.org/0009-0006-7133-30749,10, andrey rodrigues de freitas  orcid: orcid.org/0009-0008-6511-32489, anderson carlos gomes martins  orcid: orcid.org/0009-0001-4637-030111, caroline midori rozza sawazaki  orcid: orcid.org/0009-0008-2251-11319, caroline vidal  orcid: orcid.org/0009-0003-6904-27199 & … lucas emanuel silva e oliveira  orcid: orcid.org/0000-0003-1811-50879,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 272 accesses 1 citation abstract in this study, we developed and evaluated two medical large language models, clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2, specifically designed for brazilian portuguese. utilizing the low-rank adaptation (lora) technique, our models achieved significant improvements in generating synthetic clinical text, particularly in terms of authenticity of format and structure, spelling accuracy, and clinical coherence. the evaluation, conducted by medical students using a 5-point likert scale, demonstrated the effectiveness of our approach compared to baseline models. the scores indicate superior performance compared to baseline models such as llama-2-7b and mistral-7b-v0.2. our results suggest that these resource-efficient models can effectively generate clinically relevant text, maintaining high standards of structure, accuracy, and coherence. future work will focus on expanding datasets, refining evaluation protocols, and enhancing model robustness to further improve performance across various medical tasks. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others human level information extraction from clinical reports with finetuned language models article open access 24 november 2025 evaluation and mitigation of the limitations of large language models in clinical decision-making article open access 04 july 2024 the diagnosis of typical medical cases through optimized fine-tuning of large language models chapter © 2025 1 introduction foundation models have gained significant attention for their strong ability to process various data types, particularly text. large language models (llms), a subset of these architectures, are central in natural language processing (nlp) research due to their capability to generate coherent and contextually relevant text. this makes them instrumental in applications from automated content creation to conversational agents. the versatility and scalability of llms highlight their potential to revolutionize interactions with digital information, driving advancements in ai. in the medical field, llms enhance the efficiency and accuracy of clinical decision-making processes [5]. these models excel in interpreting complex medical language and extracting relevant information from unstructured texts [29], aiding in the understanding of patient histories and treatment outcomes [29]. llms streamline administrative procedures and support diagnostic/therapeutic decisions [28], potentially improving patient outcomes and operational efficiencies [5]. for example, llms like med-palm can achieve passing scores on medical licensing exams, showcasing their potential in clinical knowledge and question-answering tasks [29]. they also extract structured information from clinical notes and reports in various languages, enhancing diagnostic accuracy and aiding in health-related information dissemination [28]. the lack of resources in languages other than english is a persistent issue in nlp research, and the development of llms is no exception. this gap is further worsened by the substantial computational power required to train llms from scratch, a resource predominantly accessible to large corporations. it is essential to find ways to make these technologies accessible to the research and academic community by leveraging publicly available multilingual models and adapting them to specific contexts. in this work, we focus on creating a useful medical llm for the brazilian portuguese language (pt-br) using minimal computational resources. our methodology involved continuing the pre-training (or unsupervised fine-tuning) of llama-2-7b and mistral-7b-v0.2 base models using three different datasets of clinical narratives. we applied the lora (low-rank adaptation) technique to enhance the efficiency of this process. the new models, called clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2, were evaluated on their ability to generate synthetic clinical data, and results showed that our models outperformed other baseline models in generating clinically relevant data. this initiative is part of a collaborative project between hailab and comsentimento, named med-llm-brfootnote 1, which aims to develop multiple medical llms for pt-br, including base models and task-specific models, with different sizes. to the best of our knowledge, this is the first clinical llm model publicly available for the portuguese language. our work addresses the specific needs and gaps in the brazilian health informatics community, providing a valuable resource for clinicians and researchers to utilize in their healthcare ai projects. by overcoming linguistic and technical challenges, we contribute to the broader field of natural language processing and medical informatics, highlighting the significance of domain-specific knowledge in medical language processing. 2 related work the development of large language models (llms) has revolutionized natural language processing (nlp), especially in the medical field. this section reviews advancements in transformer architectures, the importance of transfer learning and fine-tuning, and computational strategies for optimizing these models. we examine key initiatives in other languages, particularly portuguese, and highlight prominent medical llms to provide context for our resource-efficient medical llm for brazilian portuguese. the advent of transformers, introduced by vaswani et al. [26], revolutionized the field of natural language processing (nlp). this architecture uses self-attention mechanisms to handle long-range dependencies in text, significantly improving performance over previous recurrent neural network (rnn) and convolutional neural network (cnn) models. transformers can be configured as encoder or decoder models, supporting both generative and non-generative tasks. encoder models, like bert [7], focus on understanding and encoding input text into a dense representation. decoder models, such as gpt-3 [3], generate text based on an input sequence. encoder-decoder models, like the original transformer [26], combine both functionalities and are effective for tasks like translation and summarization. in the context of transfer learning, it’s crucial to distinguish between base models and task-specific models. base models, like bert and gpt-3, are pre-trained on large corpora to learn general language representations. these models can then be fine-tuned on specific tasks, such as sentiment analysis or named entity recognition, to create task-specific models that retain the general language understanding while being optimized for particular tasks. training and fine-tuning large language models pose significant computational challenges. parameter efficient fine-tuning (peft) methods, like low-rank adaptation (lora) [11], introduce trainable low-rank matrices into each layer of a pre-trained model, reducing trainable parameters while maintaining performance and lowering costs. models like llama 2 [25] and mistral [13] are designed for efficiency in training and inference. llama 2 and mistral 7b models offer superior performance and resource efficiency, with llama 2 excelling in benchmarks and being accessible for various applications due to its smaller size. mistral 7b’s mixtral architecture uses expert mixture techniques to enhance performance and scalability, making high-performance llms more accessible and practical for specialized tasks. by utilizing peft methods, it is possible to adapt large models like those aimed at pt-br more efficiently, enabling broader application despite computational constraints. these techniques are crucial for developing specialized models in resource-constrained environments, ensuring that advancements in nlp are accessible and applicable across different languages and domains. while most large language models have been developed for english, significant efforts have been made to create models for other languages. for example, mbert (multilingual bert) handles 104 languages by training on a multilingual corpus [27], and xlm-r [6] extends this idea by pre-training on a massive multilingual dataset. there are initiatives to develop models for pt-br, such as sabiá [20], sabiá-2 [1] and bode [9]. however, relatively few models exist for this language, despite the large portuguese-speaking population (almost 250 million speakers). in the medical field, models like biobert [16] and clinicalbert [12] are fine-tuned on domain-specific corpora to capture the unique terminology and context of medical texts. med-bert [21], trained on electronic health records (ehrs), predicts patient outcomes and assists in clinical decision-making. meditron and biomistral have shown significant performance gains by extending pre-training on curated medical corpora [4, 15]. gatortrongpt, trained from scratch using 277 billion words of mixed clinical and english text with a gpt-3 architecture, improves biomedical nlp for medical research [19]. openbiollm-70b leverages the meta-llama-3-70b-instruct model to achieve state-of-the-art performance on various biomedical tasks [2]. in the pt-br medical context, the two main resources are the biobertpt [23] and cardiobertpt [22]. biobertpt is a biomedical llm, fine-tuned from multilingual bert using clinical and biomedical data, showing promising results in clinical text tasks like named entity recognition and negation detection. while cardiobertpt was fine-tuned on clinical data specifically from cardiology. the only available generative model for pt-br is gpt2-bio-pt [24], which was trained exclusively on biomedical data from medical articles. the model has several limitations such as to be based on a model trained on automatically translated text, absence of clinical narratives knowledge and small context window (512 tokens). 3 method this chapter describes our methodology divided into three key steps: data acquisition, model architecture and training, and experimental setup. each subsection details the processes and techniques employed to build and evaluate our models. 3.1 data acquisition our project combined data from three distinct clinical datasets, totaling 2.4gb of text and 309 151 121 tokens. the first one comes from the same data sources used in the semclinbr project [18], which are composed of 2 100 546 clinical narrative entries from multiple brazilian hospitals. the dataset contains diverse document types (e.g., discharge summaries, ambulatory notes, nursing notes) and medical specialties (e.g., cardiology, nephrology, endocrinology). the electronic health record (ehr) data used in the study were de-identified and approved by the pucpr research ethics committee, certificate of presentation for ethical appreciation number 51376015.4.0000.0020. the brateca dataset [8] was collected as well, and it is composed of 73 040 admission notes from 10 brazilian hospitals and associated with multiple medical departments (e.g., obstetrics, surgery, emergency, covid-19, intensive care, ambulatory). all the data was anonymized and granted ethical approval by the national research ethics committee under the number 46652521.9.0000.5530. the dataset is available under physionet credentialed health data use [10]. finally, we utilized the data used in lopes et al. work [17], which consists mostly of neurology clinical cases collected from medical journals written in european portuguese. the dataset contains 3678 medical texts and it is publicly availablefootnote 2. 3.2 model architecture and training in this study, we fine-tuned the llama-2 and mistral base models using the llama-factory framework [30], focusing on 7 billion parameter (7b) models to minimize computational resources. the models were trained over 2 epochs with a learning rate of 2e−5, using low-rank adaptation (lora) for efficient weight updates. lora introduces trainable low-rank matrices into each layer, reducing trainable parameters and gpu memory requirements. to further optimize memory and computational usage, we applied lora with 16-bit precision on the q_proj and v_proj projections, setting lora r to 8, lora alpha to 16, and lora dropout to 0.1. we used the adamw optimizer with settings \(\beta _1 = 0.9\) and \(\beta _2 = 0.999\) to balance rapid convergence with stability during training. the main difference between the two fine-tuned models was the max_position_embeddings parameter configuration: 4096 for llama-2 and 32768 for mistral. training was conducted on google cloud platform (gcp) with an nvidia tesla a100 gpu, 12 vcpus, and 85 gb of ram. the resulting models, clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2, had their training loss shown in fig. 1. we used 2.4 gb of clinical data for this study. training the clinical-br-llama-2-7b model involved 309,151,121 tokens and took 47 h and 56 min, costing r$ 1250.94. the clinical-br-mistral-7b-v0.2 model took 50 h and 18 min, costing r$ 1202.73. fig. 1. training loss of models full size image 3.3 experimental setup the models trained in this work are base models, trained unsupervised on a text corpus to learn clinical language representations. they serve as foundations for further fine-tuning for specific tasks. evaluating base models typically involves benchmarks for tasks like language understanding, text generation, and question answering. however, these benchmarks can be imprecise, often relying on standard metrics that may not capture the nuances of language or generalization to unseen data. they might not reflect real-world performance, especially in specialized fields like healthcare, where domain-specific knowledge is crucial. for instance, most medical llms are evaluated on english question answering benchmarks like medqa [14], which do not measure the model’s ability to interpret clinical data in electronic medical records, the focus of this work. to overcome limitations, we used the models to generate synthetic clinical text and evaluated their performance on three criteria using a 5-point likert scale (fig. 2): authenticity of format and structure, spelling accuracy, and clinical coherence. fig. 2. assessment card example. the evaluators had to fill the scores for each criterion, based on the output of each model full size image criterion 1 authenticity of format and structure: evaluated if the model’s output adhered to clinical document norms, including layout and sectioning. the highest score was given to models that perfectly matched the standard. criterion 2 spelling accuracy: focused on linguistic correctness, including spelling and medical terminology in pt-br. models lost points for incorrect grammar or generating text in a different language. criterion 3 clinical coherence: assessed if the content logically correlated with the clinical history, focusing on relevance and precision. this criterion involved more subjectivity due to varying evaluator backgrounds. we selected 100 random clinical notes, not part of the training corpus, and used excerpts as input for five models to complete the notes. this aimed to measure the models’ proficiency in producing consistent clinical notes. medical students with experience in clinical notes evaluated the generated texts from our models (clinical-br-llama-2-7b, clinical-br-mistral-7b-v0.2) and three reference models (llama-2-7b, mistral-7b-v0.2, sabia-7b). they assigned scores from 1 to 5 on a likert scale for each of the three criteria. the evaluation was blind; students did not know which model generated each note. llama-2-7b and mistral-7b-v0.2 served as benchmarks to measure improvements from our training. sabia-7b, a portuguese-trained model, was included for comparison. this selection allowed a thorough performance analysis across benchmarks. gpt2-bio-pt was excluded due to its limited context size, which is inadequate for clinical text generation. 4 results this section presents the evaluation outcomes of clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2 models, alongside three baselines: llama-2-7b, mistral-7b-v0.2, and sabia-7b. the evaluation focused on authenticity of format and structure, spelling accuracy, and clinical coherence, using a 5-point likert scale. we analyze the average scores for each criterion, the frequency each criterion was met, and perform an error analysis. this helps us understand the strengths and weaknesses of each model, highlight improvements in our fine-tuned models, and identify areas for future enhancement. 4.1 clinical text generation the average score of the models for each criterion is presented in table 1 and stacked in fig. 3. in addition, table 2 presents the frequencies of score for each criterion in the evaluation of each model. in table 3, we can see some examples of clinical notes generated with our models that were successful in the three evaluation metrics used (scores greater than 4 in all metrics), that is, they are cohesive texts with regard to structure, spelling and clinical coherence. authenticity of format and structure: the models were assessed on their adherence to the format and structure typical of clinical documents. sabia-7b and mistral-7b-v0.2 scored 3.84 and 3.85, respectively, indicating moderate adherence. llama-2-7b demonstrated better consistency with a score of 4.45. our models, clinical-br-llama-2-7b and clinical-br-mistral-2-7b, achieved the highest scores of 4.60 and 4.62, respectively. these models also had a significant number of perfect scores (5), reflecting their superior performance in maintaining document authenticity. spelling accuracy: despite sabia-7b not achieving above 4 score average in criteria 1 and 3, it demonstrated excellent performance in orthographic correctness, likely due to its training on a large corpus of pt-br data. in contrast, the mistral-7b-v0.2 model showed significant difficulties, often generating random texts or words in english. llama-2-7b, although not trained with a large portuguese corpus, managed to perform well in this criterion. as indicated by the graph, our adjusted llama model (i.e., clinical-br-llama-2-7b) further improved upon the already good performance of llama-2-7b. the mistral-7b-v0.2 model showed a remarkable improvement, with its base model scoring an average of 3.82 and achieving an increase to 4.69, including 30 additional perfect scores (5) in the frequency distribution, indicating excellent performance. table 1. average score of the models in the three criteria evaluated. in bold the models with the best score for each criterion.full size table fig. 3. stacked average score of the models across the three evaluated criteria full size image table 2. score frequencies for each evaluated model. c1, c2 and c3 stands for criterion 1, 2 and 3 respectively.full size table table 3. examples of successfully generated clinical notesfull size table clinical coherence: in terms of clinical coherence, sabia-7b scored 3.54 and mistral-7b-v0.2 scored 3.52, both struggling to maintain logical correlation within the clinical context. llama-2-7b scored 4.19, indicating a better, though not optimal, performance. the models clinical-br-llama-2-7b and clinical-br-mistral-2-7b again led with scores of 4.45 and 4.46, respectively, demonstrating their ability to generate clinically coherent content. frequency analysis revealed that these models had the highest number of perfect scores (5), underscoring their superior coherence. 4.2 error analysis in this section, we perform an error analysis in the task of generating clinical texts, both from the models we trained and the original baseline models. table 4 presents some examples of generated clinical texts that obtained low scores for one or more evaluation criteria. all the evaluated models have demonstrated a behavior in which they repeat certain words or sequences multiple times in succession, like on error #2. this issue can probably be solved by adjusting some inference parameters. for example, when applying the temperature = 0.1 and repetition penalty = 1.2, the output for error #2 would be “por sensação de pressão no peito. a doença foi diagnosticada há mais de um ano atrás, mas o paciente não tomou nenhuma medicação para tratamento. o exame clínico revelou: pressão arterial sistólica de 140 mmhg; frequência cardíaca regular de 76 bpm; pulso normal; auscultação cardiovascular sem alterações\(\ldots \)”. although clinical texts are known for not having a standard formal structure, as each physician or institution may establish a different format, we expect a clinical note to contain at least a partial description of the patient’s condition or visit, either in narrative or semi-structured form. therefore, some notes generated contained an output more similar to a test question or teaching documentation than a clinical narrative (as shown in error #1). this was a more prevalent behavior of the sabia-7b model, most likely due to its training data related to teaching exams. regarding the problems in the spelling metric, most of the issues came from generating an output in english instead of portuguese (like on error #3), being a more recurrent problem in the mistral-7b-v0.2 model. as for the clinical coherence dimension, the most complex for models to achieve maximum scores, most of the problems are associated with the use of medical terms that would make sense to be in that text section, but not for the patient’s reported condition. for example, error #4 shows a text that suggests the use of atenolol, a drug widely used to treat hypertension, for the treatment of dyslipidemia. this is an expected problem, given that the standard behavior of models based on vector representations interprets all terms related to drugs as similar, since they are always used in similar contexts. we also had clinical coherence problems, in which the generated output uses keywords related to the patient’s context, but the construction of sentences with these words does not make sense, as in error #5, where the word “dialysis” makes sense, since the patient has chronic kidney disease, but the presented sentence does not make sense from a clinical point of view. table 4. examples of poorly generated clinical notesfull size table 5 discussion and future work in this study, we developed and evaluated two medical large language models, clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2, tailored for pt-br. our findings showed significant improvements in generating synthetic clinical text, particularly in structure, orthographic accuracy, and clinical coherence. these improvements are due to targeted pre-training and fine-tuning on clinical narratives using low-rank adaptation (lora) for efficient adaptation. the clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2 models produced outputs that adhered to clinical document norms, maintained high spelling accuracy, and generated clinically coherent content. clinical-br-mistral-7b-v0.2 handled long contexts better due to its larger context window, a critical attribute for long clinical documents. it is worth noting that the mistral-7b-v0.2 model performed worse than llama-2-7b in all evaluation criteria, maybe due to the low volume of pt-br data in its training (this information was not made available by the team that developed the model). however, after our training on clinical data, the clinical-br-mistral-7b-v0.2 model performed better or similarly to the clinical-br-llama-2-7b model. this may indicate that architectural details of the mistral-based models better handle adjustments to new contexts in a low volume of data scenario and with the use of peft techniques. our work focuses on medical llms for pt-br, addressing unique challenges compared to models trained in other languages. this distinction is critical as pt-br linguistic and clinical terminologies differ significantly, affecting model generalization. biomistral and meditron models, trained on extensive medical corpora like pubmed central, show performance gains in english medical qa tasks. however, our pt-br models face fewer resources and less standardized evaluations, complicating direct performance comparisons. gatortrongpt and openbiollm-70b handle extensive clinical data in english, showcasing improvements in biomedical nlp tasks. in contrast, our approach optimizes limited resources while maintaining high performance in pt-br clinical narratives, emphasizing computational efficiency with techniques like lora. this aspect is crucial for making advanced llms accessible to research communities with limited computational infrastructure. evaluation protocols often involve well-defined benchmarks in english. biomistral’s evaluation includes multilingual medical qa tasks, reflecting its capabilities. our evaluation metrics are tailored to pt-br, ensuring relevance but limiting direct comparisons with english-centric models. as another limitation in the evaluation protocol of our project, we can mention that the use of medical students may not have been an optimal solution, especially when evaluating the clinical cohesion of the notes, as students have limited expertise regarding some conditions and treatments. moreover, the length of the clinical notes, a critical factor in generating synthetic health data, was not included in our structure or cohesion assessment. our primary aim was to identify the impact of pt-br clinical training data on model performance, even in small volumes, compared to existing models. consequently, our models are not recommended for generating fully coherent synthetic data. despite high scores, we did not consider the note length or completeness, leading models to sometimes generate only a few words post-input. additionally, using other clinical notes as input references may complicate synthetic data generation in environments with limited real clinical data. while our models show promising results in pt-br, future work should aim to fine-tune and evaluate the models in clinical downstream tasks, and align evaluation protocols more closely with international benchmarks to facilitate better comparisons. in addition to also building more specific benchmarks for the context of clinical texts, which have a very peculiar format and structure, differing greatly from scientific articles or medical licensing exams. furthermore, expanding our dataset and incorporating more diverse clinical narratives can enhance model robustness and generalization. training a model with medical records data tends to perform better in tasks related to data extraction, interpretation, and generation in this context. however, we understand that several tasks in the medical field involve medical reasoning, and much of the knowledge needed for the model to perform this type of inference is found in biomedical texts and specific question-answering datasets. in this context, we intend to expand our models to the biomedical context as well, so that they are able to deal well with all types of medical data. by addressing these differences, we highlight the unique contributions of our work and set the stage for future advancements in developing and evaluating medical llms across diverse languages and resource settings. 6 conclusion this study focused on developing and evaluating two medical large language models, clinical-br-llama-2-7b and clinical-br-mistral-7b-v0.2, made for medical brazilian portuguese context. these models demonstrated significant improvements over baseline models in generating synthetic clinical text, particularly in terms of authenticity of format and structure, spelling accuracy, and clinical coherence, largely due to the efficient use of the lora technique and continued pretraining on a dataset of clinical notes. the models’ performance underscores the potential for integrating resource-efficient medical llms into clinical practice, facilitating the generation of clinical notes and supporting medical research in portuguese-speaking contexts. however, limitations such as the subjectivity of the evaluation protocol, the expertise of the evaluators, and the scope of the dataset highlight areas for further enhancement. future research should aim to perform the models fine-tuning for specific tasks, align evaluation protocols with international benchmarks, build specific benchmarks for clinical texts, and expand datasets to include diverse clinical narratives and biomedical texts. these efforts will help improve model robustness and generalization, enabling the models to perform better in various medical tasks. our work highlights the feasibility of creating resource-efficient medical llms for pt-br, paving the way for their broader adoption in healthcare. continued improvements and expansions will contribute to more accurate and efficient healthcare solutions, benefiting both patients and medical professionals. notes 1.https://github.com/hailab-pucpr/med-llm-br/. 2.https://github.com/fabioacl/portugueseclinicalner. references almeida, t.s., abonizio, h., nogueira, r., pires, r.: sabiá-2: a new generation of portuguese large language models (2024) google scholar  ankit pal, m.s.: openbiollms: advancing open-source large language models for healthcare and life sciences (2024). https://huggingface.co/aaditya/openbiollm-llama3-70b brown, t.b., mann, b., et al.: language models are few-shot learners (2020) google scholar  chen, z., cano, a.h., et al.: meditron-70b: scaling medical pretraining for large language models (2023) google scholar  clusmann, j., et al.: the future landscape of large language models in medicine. commun. med. 3(1) (2023) google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale (2020) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2019) google scholar  dias, h., ulbrich, a.h.d.p.d.: brateca (brazilian tertiary care dataset): a clinical information dataset for the portuguese language (2022). https://physionet.org/content/brateca/1.1/ garcia, g.l., et al.: introducing bode: a fine-tuned large language model for portuguese prompt-based task (2024) google scholar  goldberger, a.l., et al.: physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation 101(23), e215–e220 (2000). [online] google scholar  hu, e.j., et al.: lora: low-rank adaptation of large language models (2021) google scholar  huang, k., altosaar, j., ranganath, r.: clinicalbert: modeling clinical notes and predicting hospital readmission (2020) google scholar  jiang, a.q., et al.: mistral 7b (2023) google scholar  jin, d., pan, e., oufattole, n., weng, w.h., fang, h., szolovits, p.: what disease does this patient have? a large-scale open domain question answering dataset from medical exams. appl. sci. 11(14), 6421 (2021) article  google scholar  labrak, y., bazoge, a., morin, e., gourraud, p.a., rouvier, m., dufour, r.: biomistral: a collection of open-source pretrained large language models for medical domains (2024) google scholar  lee, j., et al.: biobert: a pre-trained biomedical language representation model for biomedical text mining. bioinformatics 36(4), 1234–1240 (2019) article  math  google scholar  lopes, f., teixeira, c., gonçalo oliveira, h.: contributions to clinical named entity recognition in portuguese. in: proceedings of the 18th bionlp workshop and shared task, pp. 223–233. association for computational linguistics, florence, italy (2019) google scholar  oliveira, l.e.s.e., et al.: semclinbr a multi-institutional and multi-specialty semantically annotated corpus for portuguese clinical nlp tasks. j. biomed. semant. 13(1) (2022) google scholar  peng, c., et al.: a study of generative large language model for medical research and healthcare. npj digit. med. 6(1) (2023) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models, pp. 226–240. springer, cham (2023) google scholar  rasmy, l., xiang, y., xie, z., tao, c., zhi, d.: med-bert: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction (2020) google scholar  schneider, e.t.r., et al.: cardiobertpt: transformer-based models for cardiology language representation in portuguese. in: 2023 ieee 36th international symposium on computer-based medical systems (cbms), pp. 378–381 (2023) google scholar  schneider, e.t.r., et al.: biobertpt a portuguese neural language model for clinical named entity recognition. in: proceedings of the 3rd clinical natural language processing workshop, pp. 65–72. association for computational linguistics, online (2020) google scholar  schneider, e.t.r., de souza, j.v.a., gumiel, y.b., moro, c., paraiso, e.c.: a gpt-2 language model for biomedical texts in portuguese. in: 2021 ieee 34th international symposium on computer-based medical systems (cbms), pp. 474–479 (2021) google scholar  touvron, h., martin, l., et al.: llama 2: open foundation and fine-tuned chat models (2023) google scholar  vaswani, a., et al.: attention is all you need (2023) google scholar  wu, s., dredze, m.: beto, bentz, becas: the surprising cross-lingual effectiveness of bert (2019) google scholar  yu, h., et al.: large language models in biomedical and health informatics: a bibliometric review (2024) google scholar  zheng, y., gan, w., chen, z., qi, z., liang, q., yu, p.s.: large language models for medicine: a survey (2024) google scholar  zheng, y., zhang, r., zhang, j., ye, y., luo, z., ma, y.: llamafactory: unified efficient fine-tuning of 100+ language models (2024) google scholar  download references acknowledgments we would like to thank comsentimento and the google for startups cloud program for providing infrastructure and support to develop this project. we acknowledge the use of generative ai tools, particularly chatgpt, in the preparation of this article. chatgpt was employed for tasks such as text revision, paragraph summarization, and enhancing the overall clarity and coherence of the manuscript. author information authors and affiliations pontifícia universidade católica do paraná (pucpr), curitiba, pr, brazil joão gabriel de souza pinto, andrey rodrigues de freitas, caroline midori rozza sawazaki, caroline vidal & lucas emanuel silva e oliveira comsentimento nlp lab, são paulo, sp, brazil joão gabriel de souza pinto & lucas emanuel silva e oliveira instituto federal de goiás (ifg), luziânia, go, brazil anderson carlos gomes martins authors joão gabriel de souza pintoview author publications search author on:pubmed google scholar andrey rodrigues de freitasview author publications search author on:pubmed google scholar anderson carlos gomes martinsview author publications search author on:pubmed google scholar caroline midori rozza sawazakiview author publications search author on:pubmed google scholar caroline vidalview author publications search author on:pubmed google scholar lucas emanuel silva e oliveiraview author publications search author on:pubmed google scholar corresponding author correspondence to lucas emanuel silva e oliveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de souza pinto, j.g., rodrigues de freitas, a., martins, a.c.g., sawazaki, c.m.r., vidal, c., silva e oliveira, l.e. (2025). developing resource-efficient clinical llms for brazilian portuguese. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_4 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords large language models medical nlp generative ai publish with us policies and ethics profiles joão gabriel de souza pinto view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an open source eye gaze tracker system to perform remote user testing evaluations | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an open source eye gaze tracker system to perform remote user testing evaluations conference paper first online: 12 october 2023 pp 192–207 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) an open source eye gaze tracker system to perform remote user testing evaluations download book pdf download book epub marc g. capdevila  orcid: orcid.org/0000-0003-3092-42079, karine aparecida p. rodrigues  orcid: orcid.org/0000-0003-4787-79379, camila f. jardim  orcid: orcid.org/0009-0006-7265-90419 & … renato m. silva  orcid: orcid.org/0000-0001-6687-89819  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 597 accesses 2 citations abstract eye gaze trackers are devices designed to identify an individual’s gaze fixation in relation to a screen or another reference point. these tools are widely applied in usability testing as they provide various metrics for studying how people interact with applications. in the past, these tools were expensive and required a controlled environment, as well as trained personnel for proper operation. although nowadays, new implementations do not require physical hardware to perform these tests, they often rely on license-based models instead of being open source. the objective of this work is to create a standalone system that enables any user to implement a low-cost eye gaze tracker using web technologies. the goal is to facilitate its use in remote and face-to-face studies in a simple way, requiring only a computer and a webcam. we evaluated the impact of three different calibration techniques on the performance of a regression-based prediction algorithm in eye-tracking. in our experiments, the best result of linear regression was obtained with a circular calibration system that uses 160 points. furthermore, we integrated the system with a web interface and an api, enabling users to record their usability sessions and analyze fixation points through heatmaps. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an analysis of accuracy requirements for automatic eyetracker recalibration at runtime chapter © 2016 surface recalibration as a new method improving gaze-based human-computer interaction chapter © 2018 efficiency in real-time webcam gaze tracking chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer application in social and behavioral sciences intervision oculomotor system psychometrics saccades vestibuloocular reflex 1 introduction usability and user experience (ux) have become used in most of the user-centered design methodologies [4] to understand and later improve the way that users interact with a system [18]. one of the well-known techniques is called “user testing”, which is a technique that involves evaluating a system with real users to see how easy it is to use and how well it meets their needs [3]. the versatility of the technique enables the use of different kinds of configurations based on the specific goals of the research [37]. they can be divided between moderated or unmoderated [19], and can be applied in a physical environment like a usability laboratory [37] to perform in-person tests, or in a remote environment [34] using multiple testing tools [46, 51]. they can also be applied in one-on-one testing procedures, where one moderator works with one user at a time, or in a group where a moderator addresses multiple users simultaneously. to capture information through these different configurations, multiple tools can be used, such as eye gaze tracker systems [11], which are the object of the study of this paper. eye gaze trackers and their remote extensions, also known as “remote eye gaze trackers” (regt), are devices capable of identifying a person’s gaze fixation. these devices are widely used in usability testing. generally, they are accompanied by native software that generates various metrics, such as fixation time, areas of interest, saccade speed, and gaze duration. however, these tools are expensive and require highly controlled spaces, typically found in usability facilities. consequently, the possibility for regular users to perform remote tests using these tools is almost null. the general objective of this study is to investigate, validate, and develop a software that can be used in usability studies with the purpose of capturing the eye gaze of a person and saving their fixation points to generate usability reports. the proposed system will be composed only of a computer and a conventional webcam, creating an open-source and low-cost project that can be used by individuals or groups, both in person or remotely. finding open-source contributions to this topic is a challenge, as highlighted by shehy et al. [47]. although the authors introduced several organizations and software options, we could not find a software that was still in active development, and incorporate both the open api specifications and make use of web technologies that would allow easy integration. for this reason, we developed our own system to be published under open source license. moreover, we also prepared the system to work standalone and also as a plugin for an open source organization called remote user experience lab (ruxlab)footnote 1 with the objective to be integrated as a tool in their user testing system. the development workflow to be implemented has to be capable of identifying the direction of the eye gaze based on the position of the iris and pupil in relation to the screen. several machine learning methods have already been proposed in the literature to accomplish this task, such as convolutional neural networks [25], recurrent neural networks [22], adversarial generative networks [28], and siamese neural networks [2]. however, although these models are used with success in gaze estimation and other related learning problems, they require a large amount of training data, and are computationally expensive. for an open-source and low-cost system like the one being proposed, it is important to use a predictive algorithm that is simple and fast. therefore, this study used a regression algorithm to predict the direction of the eye. the performance of this method or any other employed in eye tracking may be impacted by the calibration system. therefore, we also investigated three different calibration techniques to analyze their effect on the system’s performance. the calibration system is used to obtain personalized information from each user, according to the position of their head and also the level of brightness when they perform the calibration process. this information feed the model to improve the performance, giving better insights about the user environment and features. we also developed a graphical interface as a tool for users to easily use the model for analyzing and conducting their studies on their desired web pages. the remainder of this study is organized as follows. section 2 presents the context about user testing and usability alongside with the eye tracker systems. in sect. 3, we explore existing research in the field and examine various eye tracker systems that were tagged as open source. our proposed prototype is described in sect. 4, where we segmented it into the gaze tracker system, the web implementation, and the api architecture. finally, the conclusions and future work are presented in sect. 5. 2 context the following section describes the topics related to conducting a user testing evaluation, focusing on eye tracker tools. we describe the methods and necessary steps to conduct such evaluations, as well as the main features that these systems must contain. 2.1 user testing and usability the term usability was created to define the ease with which people interact with a tool or interface while performing a task [24]. it was defined by the iso/cd 9241-11 as “the ability of software to be understood, learned, used under specific conditions”footnote 2. to evaluate usability, different techniques can be applied [36], which can be grouped into three different categories: inspection methods, inquiry methods, and test evaluations. user testing belongs to the test evaluations technique and it can be performed during product development to assess the effectiveness, speed, user error tolerance, ease of learning, and user engagement of the product. the goal is to ensure that users experience satisfaction instead of frustration when using the solution [3]. user testing can be divided into three different steps or sections in which users have to complete different tasks based on provided information from the team of observers or evaluators. those sections are the pre-test, test phase, and post-test sections. inside the pre-test section, participants are typically presented with a consent form and a pre-test form to get insights, such as demographic information, about the user that will participate in the study. the test phase involves different tasks that the user must accomplish, which can be highly customized by the evaluators to gather as much qualitative or quantitative data as possible. each task can have additional forms attached, known as pre-task or post-task forms, that are typically completed before or after the task is finished. examples of these forms are the single ease question (seq) [45] or the nasa task load index (nasa tlx) [17]. finally, in the post-test section, a post-test form such as the system usability scale (sus) [7] can be shared. recruiting participants is another key challenge for user testing. it involves recruiting representations of the target user group for the product or service being tested. when a test is performed in physical facilities, the representative sample of the target group may not be heterogeneous enough, as highlighted by several authors [10, 34, 51]. after the participants are selected, a set of tasks are presented to them to test the product or service, while the evaluator observes and takes notes. when the different tasks of a user testing are created, the evaluation team must decide what type of data needs to be collected. these decisions can influence the nature of the results, for example, whether they are qualitative or quantitative. they can also determine whether it is possible to measure or calculate the effectiveness, efficiency, or user satisfaction of the system [15]. to achieve this, a variety of configurations can be used, such as recording tools for capturing time, voice, screen activities, and video [10]. in the last case, we can distinguish between face recognition, which enables the application of artificial intelligence (ai) for emotion recognition [33], and gaze tracking, which allows for the implementation of eye-tracking procedures combined with heat map technologies [21]. 2.2 eye tracker the detection and recognition of human faces are extensively studied tasks in the area of ai, and their applications can be expanded in several use cases, such as security, entertainment, marketing, identification of people in social networks, forensic investigations, participant control in events, and many more [5, 9, 11, 23, 33]. a face has very rich features that are essential for social interaction. while humans can effortlessly detect and interpret these characteristics, this task is not as simple for a computer [57]. initially, it is necessary for the computer to understand which features are related to a human face and differentiate them from other objects that may have similar visual characteristics [23]. in most face detection algorithms, the first point to be identified are the eyes [57], as they are usually the simplest features to detect. next, it follows other parts, such as the mouth, nose, eyebrows, and contours to increase confidence in the detected area being a face [31]. for better chances of success and to address a broad spectrum of human physiology and morphology [23], these models are usually trained with very large datasets of positive and negative images [47], since different factors can affect face detection, such as the pose, the presence or absence of structures (glasses and beard for example), facial expression, occlusion, orientation, and image quality [31]. eye tracking systems aim to discover where the eyes are focused or study how they move when interacting with a graphical interface or environment [5]. this technology enables people with physical or cognitive disabilities [9] to perform activities using only their eyes and opens up a wide range of studies related to the usability of interfaces [44], where eyes can be analyzed to discover how users interact with websites, desktop, and mobile applications [35]. eye tracking systems can be divided into multiple categories depending on the technology that they are based on, their final application, or their method of operation [47]. some of them are more intrusive than others, and regarding this configuration, they can be divided into two categories: (i) head-mounted and (ii) remote-based eye trackers. the first category consists of wearable devices that track the movements through a system mounted on a headset like glasses that can be worn. while these systems are more intrusive compared to the remote option, they offer more freedom of movement to the user. on the other hand, remote-based are a non-intrusive method, as they capture eye movements from a distance. in this case, cameras or other sensors that track the movements are normally coupled into a layout or screen. in such cases, the main problems that can arise are related to a lower accuracy because they can be affected by light conditions, as well as a more time-consuming calibration process [9, 29]. some of the most used analysis tools in these types of studies are heatmaps, gaze replays, and gazeplots. however many other options exist, and a combination of them might help to find more accurate information [43]. fig. 1. examples of different output solutions from eye tracker systems. full size image heatmaps (fig. 1a) are able to show areas where users fixed their gaze the most. they can be generated using information from a single person, or from all participants if the objective is to find the most fixed consensus point among all of them. in addition, different metrics can be chosen to generate hot and cold zones, such as the fixation time in an area or the number of fixations. gaze replays (fig. 1b) enable researchers to watch the test session and see exactly where the user was looking. usually, in this method, a cursor is placed on the screen to simulate the position of the user’s eyes, while the recorded screen video plays in the background. this helps to evaluate task flow and determine exactly the order of each fixation. gazeplots (fig. 1c) show where the user is fixing its attention, which order was followed, and for how long the fixation occurred at that point. in most representations, the size of two circles represents the fixation time (larger circles indicate longer durations), and their position indicates the location of the fixation on the canvas. the track represent the direction of the next fixing point, and they are typically numbered in sequential order. there are different eye tracking devices on the market, such as tobiiprofootnote 3 and pupillabs invisiblefootnote 4. these tools have multiple light sensors and cameras to record the user‘s gaze, and they come with native software that generates various metrics such as fixation time, areas of interest, saccade speed, and gaze duration. they also incorporate functionality for locating the eyes in the image and performing calibration processes. 3 related work usability tests with eye trackers have been studied since early 2000 ss. at the same time, ai started to become more affordable, and in the field of regt, it contributed to the development of systems that do not need special hardware and are released as open source [38]. in this study, an open source organization called ruxlab [8]footnote 5 has been investigated, which is dedicated to carrying out different usability tests, including user tests. this organization allows to implement different plugins to increase the functionalities that can be used when a user test is performed. for this reason, we studied different open-source regt systems that can be reused instead of developing a new one. we used the study performed by shehu et al. [47] as a starting point as they present a general overview of different configurations in which regt can be grouped. one of those presented configurations is based on mapping open-source regt. we have conducted in-depth research to analyze which platforms are still available and also find new options that might have appeared. table 1 summarises our finds. the first block represents the different organizations that were found by shehu et al. [47], while the second block presents organizations that were found in another studies. we tried to identify critical aspects that correspond with our objectives. specifically, we assessed whether the organization provided an open repository, if the code was available, also which kind of open source license they were using, if it contains open api specifications, and also if the final result was an executable or not. moreover, we have identified the technologies that were used during the development and the last updates that we were able to find in the code. table 1. list of gaze tracking software.full size table 4 prototype results we developed our prototype taking into account the main features encountered in the previous section. one of our objectives was to create a system that does not rely on specific hardware, thereby avoiding dependencies. this means that the system should be compatible with conventional webcams that are available in the market at a low cost. additionally, since we intended for the system to run on web browsers, we aimed to develop it using web technologies. moreover, it has to comply with open source compliance as the system will be licensed by an open source license and able to be integrated with ruxlab system. finally, communication layer must be created using open apifootnote 6 specification [6]. for this reason we created three different parts to be the most modular possible: (i) the gaze tracker with the calibration system with the use of an ai model (linear regression), (ii) the web application that allows to run tests and serves as the user interface, and (iii) the api that does all the communication between the parties and manages the created data using the open api specification. 4.1 gaze tracker system in order for the proposed model to accurately predict the direction in which the eyes focus, it is necessary to carry out a point calibration process [5, 9]. before starting a usability test on the platform, users need to undergo small exercises using their eyes. this step is essential before each session, as each user possesses unique physical characteristics, so it is possible to have a special analysis for each face instead of a pre-trained generic model with other people’s faces. environmental conditions, including factors such as luminosity, can also interfere with the access during eye tracking sessions [11]. therefore, each session will be trained considering the specific environmental conditions in which the participant is situated. the calibration process starts with the user positioning correctly within the camera coverage area, so that it is possible to capture the face and eyes without obstructions or poor lighting conditions. once positioned, the user will go through the exercises, which consist of following instantiated points on the screen with the eyes. the user has to do his best to follow the moving circle that goes around the whole screen. while the user performs them, the x and y position of the user’s pupil and the x and y position of the moving object are saved. pupil points are captured using the face landmarks detection modelfootnote 7 from the tensorflow.js packagefootnote 8 which has a map of the entire face, including the center of the pupils. this pre-trained model is based on media pipe’s facemesh packagefootnote 9. all points captured during this process are saved and stored along with the session. the calibration points are stored in a csv file containing the entire relationship between the position of the points and the corresponding pupil position. this data is later used to train the machine-learning model. the method chosen to perform this task was the linear regression [49], as it is ideal for cases where you need to predict a value based on the relationship between two or more variables. moreover, it is a light and fast model, which suits for the scenario considered in this work, that involves low-cost cameras and multiple browsers. to implement the linear regression method we used the linearregression function from the scikit-learn library [42] with default parameters. we performed the experiments using a holdout validation with 80% of the data in the training set and 20% in the test set. for the calibration process, we compared the three different techniques presented in fig. 2. the (a) 9-point coordinate system (fig. 2a) calibration system was implemented with a double forward and backward movement that increases the number of points to 36; the n-point calibration(figure 2b), where the number of points is determined by the width and height relation of the screen, and finally a system based on the design of four concentric circles of varying sizes, presented in descending order from largest to smallest which uses 160 points (fig. 2c) [9, 11]. fig. 2. calibration strategies. full size image we used the coefficient of determination and also the mean squared error to analyze the performance of the calibration strategies adopted. the formula for the coefficient of determination (r-squared) is given by: $$\begin{aligned} r^2 = 1 \frac{ss_{res}}{ss_{tot}} \end{aligned}$$ (1) where \(ss_{res}\) is the residual sum of squares and \(ss_{tot}\) is the total sum of squares. on the other hand, the formula for the mean squared error (mse) is given by: $$\begin{aligned} mse = \frac{1}{n} \sum _{i=1}^{n}(y_i \hat{y_i})^2 \end{aligned}$$ (2) where \(y_i\) is the actual value of the dependent variable, \(\hat{y_i}\) is the predicted value of the dependent variable, and n is the total number of observations. table 2 shows the different results for the three different techniques that were used. it can be observed that the (c) 160 points calibration technique obtained higher values for both x \(r^2\) and y \(r^2\). however, regarding mse, we obtained different results. for x mse, the lowest value was achieved with the (c) 160 points calibration technique. however, for y mse, the lowest value was obtained with the (a) 36 points calibration. the number of calibration points generated with the 160 points calibration technique is higher than the other two evaluated techniques. therefore, with this technique, more data was provided for training the linear regression, which may have contributed to this learning model capturing more of the variability in the relationship between eye movements and screen coordinates, which resulted in a better performance in terms of \(r^2\). table 2. comparative table of coefficient of determination (r-squared) and mean squared error (mse) coefficients.full size table 4.2 web system the proposed project has to run first standalone so it must be easily deployable into any environment. we have used docker to package our build and to facilitate dependencies between users. regarding dependencies, any browser can run the project as only will ask for webcam and screen recording permissions. functionalities implemented for the prototype include a landing page with a social authentication system, and a home page to perform create, read, update and delete operations, also known as crud. additionally, it incorporates a calibration system, screen and webcam recording system, as well as the iris capture model integration. fig. 3. screenshots from the gaze module implemented. full size image for the website development, the vuejs frameworkfootnote 10 was used along with the css framework called vuetifyfootnote 11 which provides ui components following material design guidelines as long as their iconsfootnote 12. for the recording of the screens were used the native apis for javascript media devicesfootnote 13 and media recorderfootnote 14. for the login feature, google sign-in using firebasefootnote 15 is used as the authentication provider. therefore, another important part of the proposed gaze tracker system is the generation of heatmaps that we have usedfootnote 16 to represent the focus of interest of a user on the studied page. this system needs to be integrated with an api, so that the calibration process, the usability session and the delivery of the results can be shown in a web application. 4.3 api the api has a set of crud endpoints to manage sessions for each logged-in user. table 3 contains the list of all routes defined in the api together with their descriptions and http methods correspondents. table 3. api endpoints and their descriptions.full size table those endpoints are used for the integration into the ruxlab system but also might be useful for others that might use the present work, as it presents a decoupled layer of interconnectivity between a user testing evaluation and an eye tracker system. the final version of our code can be found in the same organization umbrella where is attached the ruxlab in github repositoryfootnote 17. 5 conclusions and future work in this paper, we have presented a complete architecture and system that enables to perform eye tracking sessions that work standalone. we have used a linear regression along with three different calibration procedures and developed a graphical user interface to display data using heatmaps with an api specification to facilitate communication between both systems. one of the problems we encountered while analyzing previous solutions was that approximately 70% of the software applications were standalone executable programs with their own interfaces, whereas only 20% had a library or api that could allow integration. this topic has limited our research in terms of reusability, especially when we dive deep into the details of trying to use some of the identified items like pupillabs, which was identified as having an open source code [27], but requiring specific hardware to run. a similar situation occurred with eyetribe, as the project was acquired by facebook in 2016 and later discontinued. moreover, table 1 shows that around the 70% of the softwares identified as open source, have not received updates in the past three years. moreover we were not able to find repositories, partly because the cited links in the bibliography encountered were broken [1, 14, 30, 55, 56, 58] and also we do not find references or keywords to conduct a deep research. regarding technologies, we found that most of them are based on java, c, or python, and around only 10% use web pure technologies like javascript [41]. it has a direct correlation if the software is executable or not, being most of the applications or softwares developed using java or c# as an executable. as we could not identify a solution that met our requirements, we implemented our own solution taking into consideration different topics found in the other systems like ai model, calibration process, and data visualization technique. we have used different calibration systems based on girshick et al. [16] for their trade-off between simplicity and performance, similar to the choice of the linear regression as an artificial algorithm core due to that kind of unsupervised appearance-based method have demonstrated their high effectiveness in other works [47]. the results obtained from the calibration method suggest that there is room for improvement in achieving a high-accuracy and high-precision system. therefore, in future work, we intend to explore other techniques for the calibration process and alternative machine-learning models. still regarding the calibration method, different authors cited the importance of this step as it can help to greatly improve the final results [9]. we just implemented only three different models but we found gamification processes with better results than ours that also use linear regression [16]. in future work, we plan to study how different strategies can perform on different calibration methods to better analyze which configuration suits better for conventional webcam users. moreover, as the system was build as a standalone module, we pretend to integrate it with ruxlab to be used as a complementary tool to perform user testing evaluations. finally, regarding the visualization of the results, it was used a heatmap library to display that information. however, it exists other kinds of maps that can be created like the gaze-replay (fig. 1b) and gazeplot (fig. 1c), which might be implemented to improve the understanding of the results. also, we started with the integration of the plugin into ruxlab and we are working with the release of a new version that will allow the automatic generation of reports. notes 1.remote user experience lab (ruxlab). available at: https://github.com/uramakilab/remote-usability-lab. 2.iso/cd 9241-11: ergonomics of human-system interaction part 11: guidance on usability (1998). available at: https://www.iso.org/standard/63500.html. accessed date: september 15, 2023. 3.global leader in eye tracking for over 20 years tobii. available at: https://www.tobii.com/. accessed date: september 15, 2023. 4.pupil invisible eye tracking glasses for the real world pupil labs. available at: https://pupil-labs.com/. accessed date: september 15, 2023. 5.remote user experience lab. available at: https://retlab.web.app/. accessed date: september 15, 2023. 6.openapi specification version 3.0.3. available at: https://swagger.io/specification/. accessed date: september 15, 2023. 7.tensorflow.js: face landmarks detection. available at: https://github.com/tensorflow/tfjs-models/tree/master/face-landmarks-detection. accessed date: september 15, 2023. 8.tensorflow javascript. available at: https://www.tensorflow.org/js. accessed date: september 15, 2023. 9.mediapipe. available at: https://developers.google.com/mediapipe. accessed date: september 15, 2023. 10.vuejs. available at: https://vuejs.org/ accessed date: september 15, 2023. 11.vue component framework. available at: https://vuetifyjs.com/. accessed date: september 15, 2023. 12.material design. available at: https://m2.material.io/design/guidelines-overview accessed date: september 15, 2023. 13.mediadevices web apis | mdn. available at: https://developer.mozilla.org/es/docs/web/api/mediadevices. accessed date: september 15, 2023. 14.mediarecorder web apis | mdn. available at: https://developer.mozilla.org/en-us/docs/web/api/mediarecorder. accessed date: september 15, 2023. 15.firebase. available at: https://firebase.google.com/. accessed date: september 15, 2023. 16.dynamic heatmap for the web. available at: https://www.patrick-wied.at/static/heatmapjs/. accessed date: september 15, 2023. 17.available at: https://github.com/uramakilab/web-eye-tracker. accessed date: september 15, 2023. references agustin, j.s., et al.: evaluation of a low-cost open-source gaze tracker. in: proceedings of the 2010 symposium on eye-tracking research & applications, pp. 77–80 (2010) google scholar  ahn, h., jeon, j., ko, d., gwak, j., jeon, m.: contactless real-time eye gaze-mapping system based on simple siamese networks. appl. sci. 13(9) (2023). https://doi.org/10.3390/app13095374 bastien, j.c.: usability testing: a review of some methodological and technical aspects of the method. int. j. med. inform. 79(4), e18–e23 (2010) article  google scholar  bevan, n.: what is the difference between the purpose of usability and user experience evaluation methods. in: proceedings of the workshop uxem. vol. 9, pp. 1–4. citeseer (2009) google scholar  biedert, r., buscher, g., dengel, a.: the eye book. informatik-spektrum 33(3), 272–281 (2009) article  google scholar  braunschweig, k., eberius, j., thiele, m., lehner, w.: the state of open data. limits curr. open data platforms 1, 72–72 (2012) google scholar  brooke, j., et al.: sus-a quick and dirty usability scale. usability eval. indust. 189(194), 4–7 (1996) google scholar  capdevila, m.g., saltiveri, t.g.: heurísticas de usabilidad utilizando una plataforma abierta y colaborativa. v congreso internacional de ciencias de la computación y sistemas de información 2021 (2022) google scholar  carter, b.t., luke, s.g.: best practices in eye tracking research. int. j. psychophysiol. 155, 49–62 (2020) article  google scholar  castillo, j.c., hartson, h.r., hix, d.: remote usability evaluation: can users report their own critical incidents? in: chi 98 conference summary on human factors in computing systems, pp. 253–254 (1998) google scholar  chennamma, h., yuan, x.: a survey on eye-gaze tracking techniques. arxiv preprint arxiv:1312.6410 (2013) dalmaijer, e.s.: pygaze: an open-source, cross-platform toolbox for minimal-effort programming of eyetracking experiments. behav. res. methods 46(4), 913–931 (2014) article  google scholar  ferhat, onur e vilariño, f.: rastreamento ocular de baixo custo: o panorama atual. inteligência computacional e neurociência 2016 (2016) google scholar  ferhat, o., vilarino, f., sánchez, f.j.: a cheap portable eye-tracker solution for common setups. j. eye movement res. 7(3) (2014) google scholar  frøkjær, e., hertzum, m., hornbæk, k.: measuring usability: are effectiveness, efficiency, and satisfaction really correlated? in: proceedings of the sigchi conference on human factors in computing systems, pp. 345–352 (2000) google scholar  girshick, r., donahue, j., darrell, t., malik, j.: rich feature hierarchies for accurate object detection and semantic segmentation. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 580–587 (2014) google scholar  hart, s.g.: nasa-task load index (nasa-tlx); 20 years later. in: proceedings of the human factors and ergonomics society annual meeting. vol. 50, pp. 904–908. sage publications sage ca: los angeles, ca (2006) google scholar  hassan, h.m., galal-edeen, g.h.: from usability to user experience. in: 2017 international conference on intelligent informatics and biomedical sciences (iciibms), pp. 216–222. ieee (2017) google scholar  hertzum, m., borlund, p., kristoffersen, k.b.: what do thinking-aloud participants say? a comparison of moderated and unmoderated usability sessions. int. j. human-comput. interact. 31(9), 557–570 (2015) article  google scholar  holanda, corey e komogortsev, o.: eye tracking em tablets comuns não modificados: desafios e soluções. in: proceedings of the symposium on eye tracking research and applications, pp. 277–280 (2012) google scholar  holmqvist, k., nyström, m., mulvey, f.: eye tracker data quality: what it is and how to measure it. in: proceedings of the symposium on eye tracking research and applications, pp. 45–52 (2012) google scholar  hwang, b.j., chen, h.h., hsieh, c.h., huang, d.y.: gaze tracking based on concatenating spatial-temporal features. sensors 22(2), 545 (2022). https://doi.org/10.3390/s22020545 jain, a.k., li, s.z.: handbook of face recognition, vol. 1. springer (2011) google scholar  jordan, p.w.: an introduction to usability. crc press (1998) google scholar  kanade, p., david, f., kanade, s.: convolutional neural networks (cnn) based eye-gaze tracking system using machine learning algorithm. europ. j. electr. eng. comput. sci. 5(2), 36–40 (2021) article  google scholar  karlsson, h., berglund, e., larsson, j.: method and apparatus for eye tracking (2014). https://patents.google.com/patent/us8723875b2/, the eye tribe aps kassner, m., patera, w., bulling, a.: pupil: an open source platform for pervasive eye tracking and mobile gaze-based interaction. in: proceedings of the 2014 acm international joint conference on pervasive and ubiquitous computing: adjunctpublication, pp. 1151–1160 (2014) google scholar  kim, j.h., jeong, j.w.: gaze estimation in the dark with generative adversarial networks. in: acm symposium on eye tracking research and applications. etra ’20 adjunct, association for computing machinery, new york, ny, usa (2020). https://doi.org/10.1145/3379157.3391654 krafka, k., et al.: eye tracking for everyone. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2176–2184 (2016) google scholar  kuling, e., et al.: myeye: an open-source wearable gaze tracker. in: proceedings of the symposium on eye tracking research and applications, pp. 1–10. acm (2019) google scholar  lee, s.: understanding face detection with the viola-jones object detection framework. towards data science (2020) google scholar  li, d., babcock, j., parkhurst, d.j.: openeyes: a low-cost head-mounted eye-tracking solution. in: proceedings of the 2006 symposium on eye tracking research & applications, pp. 95–100 (2006) google scholar  lim, j.z., mountstephens, j., teo, j.: emotion recognition using eye-tracking: taxonomy, review and current challenges. sensors 20(8), 2384 (2020) article  google scholar  madathil, k.c., greenstein, j.s.: an investigation of the efficacy of collaborative virtual reality systems for moderated remote usability testing. appl. ergon. 65, 501–514 (2017) article  google scholar  manhartsberger, m., zellhofer, n.: eye tracking in usability research: what users really see. in: usability symposium. vol. 198, pp. 141–152 (2005) google scholar  martins, a.i., queirós, a., silva, a.g., rocha, n.p.: usability evaluation methods: a systematic review. human factors softw. develop. design 250–273 (2015) google scholar  nielsen, j.: usability laboratories. behav. inform. technol. 13(1–2), 3–8 (1994) article  google scholar  nielsen, j., pernice, k.: eyetracking web usability. new riders publishing (2003) google scholar  ooms, k., dupont, l., lapon, l., popelka, s.: accuracy and precision of fixation locations recorded with the low-cost eye tribe tracker in different experimental setups. j. eye movement res. 8(1) (2015) google scholar  palmero, c., selva, j., bagheri, m.a., escalera, s.: recurrent cnn for 3d gaze estimation using appearance and shape cues (2018) google scholar  papoutsaki, a., laskey, j., huang, j.: searchgazer: webcam eye tracking for remote studies of web search. in: proceedings of the 2017 conference on conference human information interaction and retrieval, pp. 17–26 (2017) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  pernice, k., nielsen, j.: how to conduct eyetracking studies. nielsen norman group 945397498 (2009) google scholar  poole, a., ball, l.j.: eye tracking in hci and usability research. in: encyclopedia of human computer interaction, pp. 211–219. igi global (2006) google scholar  sauro, j.: things to know about the single ease question (seq). measuring u 2012 (2012) google scholar  scholtz, j.: adaptation of traditional usability testing methods for remote testing. in: proceedings of the 34th annual hawaii international conference on system sciences, pp. 8-pp. ieee (2001) google scholar  shehu, i.s., wang, y., athuman, a.m., fu, x.: remote eye gaze tracking research: a comparative evaluation on past and recent progress. electronics 10(24), 3165 (2021) article  google scholar  sjöberg, a., rominger, m.: beyond hand-eye coordination: an exploration of eye-tracking and speech recognition as a navigation tool for interactive systems (2015) google scholar  skodras, e., kanas, v.g., fakotakis, n.: on visual gaze tracking based on a single low cost camera. signal process. image commun. 36, 29–42 (2015) article  google scholar  sogo, h.: gazeparser: an open-source and multiplatform library for low-cost eye tracking and analysis. behav. res. methods 45, 684–695 (2013) article  google scholar  thompson, k.e., rozanski, e.p., haake, a.r.: here, there, anywhere: remote usability testing that works. in: proceedings of the 5th conference on information technology education, pp. 132–137 (2004) google scholar  venugopal, d., amudha, j., jyotsna, c.: developing an application using eye tracker. in: 2016 ieee international conference on recent trends in electronics, information & communication technology (rteict), pp. 1518–1522. ieee (2016) google scholar  voßkühler, a., nordmeier, v., kuchinke, l., jacobs, a.m.: ogama (open gaze and mouse analyzer): open-source software designed to analyze eye and mouse movements in slideshow study designs. behav. res. methods 40, 1150–1162 (2008) article  google scholar  wisiecka, k., et al.: comparison of webcam and remote eye tracking. in: 2022 symposium on eye tracking research and applications, pp. 1–7 (2022) google scholar  wood, e., bulling, a.: eyetab: model-based gaze estimation on unmodified tablet computers. in: proceedings of etra (2014). http://www.cl.cam.ac.uk/research/rainbow/projects/eyetab/ zhang, m., bulling, a.: xlabs: a platform for rapid design, prototyping and evaluation of ubiquitous gaze interfaces. in: proceedings of the symposium on eye tracking research and applications, pp. 69–76. acm (2018) google scholar  zhao, w., chellappa, r., phillips, p.j., rosenfeld, a.: face recognition: a literature survey. acm comput. surv. (csur) 35(4), 399–458 (2003) article  google scholar  zielinski, p.: opengazer: open-source gaze tracker for ordinary webcams. samsung and the gatsby charitable foundation. http://www.inference.phy.cam.ac.uk/opengazer (2007) download references author information authors and affiliations facens university, sorocaba, são paulo, brazil marc g. capdevila, karine aparecida p. rodrigues, camila f. jardim & renato m. silva authors marc g. capdevilaview author publications search author on:pubmed google scholar karine aparecida p. rodriguesview author publications search author on:pubmed google scholar camila f. jardimview author publications search author on:pubmed google scholar renato m. silvaview author publications search author on:pubmed google scholar corresponding author correspondence to marc g. capdevila . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper capdevila, m.g., rodrigues, k.a.p., jardim, c.f., silva, r.m. (2023). an open source eye gaze tracker system to perform remote user testing evaluations. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_13 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords eye gaze tracking user testing user experience publish with us policies and ethics profiles marc g. capdevila view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese conference paper first online: 12 october 2023 pp 241–255 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese download book pdf download book epub lucelene lopes  orcid: orcid.org/0000-0003-0314-140x9, paulo fernandes  orcid: orcid.org/0000-0003-3959-043410, marcio l. inacio  orcid: orcid.org/0000-0002-0875-457411, magali s. duran  orcid: orcid.org/0000-0002-3843-46009 & … thiago a. s. pardo  orcid: orcid.org/0000-0003-2111-13199  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 519 accesses 1 citation abstract this paper explores methods to disambiguate part-of-speech (pos) tags for closed class words in brazilian portuguese corpora annotated according to the universal dependencies annotation model. we evaluate disambiguation methods of different paradigms, namely a markov-based method, a widely adopted parsing tool, and a bert-based language modeling method. we compare their performances with two baselines, and observe a significant increase of more than 10% over the baselines for all proposed methods. we also show that while the bert-based model outperforms the others reaching for the best case a 98% accuracy predicting the correct pos tag, the use of the three methods as an ensemble method offers more stable result according to the smaller variance for the numerical results we performed. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others top-down parsing error correction applied to part of speech tagging chapter © 2020 universal dependencies-based pos tagging refinement through linguistic resources chapter © 2021 a comparative study of language modeling to instance-based methods, and feature combinations for authorship attribution chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing natural language processing (nlp) portuguese research methods in language and linguistics 1 introduction to develop large annotated corpora is a challenging task in the area of natural language processing (nlp). it is a common strategy to rely on automatic analysis tools to pre-annotate the data and then to manually review the annotation, as direct human annotation requires large amounts of time, a strict discipline, and an homogeneous system of annotation rules that must be followed by annotators. in the case of treebanks (i.e., databases of sentences and their corresponding syntactic trees), this must be carried out for both part-of-speech (pos) tagging and syntactic parsing, at least. however, as the annotation of pos tags can be considered input for syntactic annotation, the better the pos annotation, the lower the risk of tagging errors leading to parsing errors. in the pre-annotation of pos, the hardest part is assigning tags to words from closed classes, as many of them may pertain to different classes, that is, they are ambiguous. these words are among the most frequent in any corpus, regardless of genre and domain. as they constitute marks for syntactic annotation decisions, it is important to annotate them accurately, thus offering safe annotation points to support the effort of producing large annotated treebanks. of particular interest to this paper is the initiative to perform pos tagging for portuguese language according to the widely adopted universal dependencies (ud) model [18, 20], which predicts the existence of “universal” pos tags and syntactic relations that may be applied to all languages in the world. there are currently nearly 200 treebanks in over 100 languages affiliated to ud. in portuguese, we highlight the previous work of rademaker et al. [22], that has produced a ud-annotated version of bosque treebank [1], and the recent efforts of silva et al. [25] that have explored pos-tagging for tweets, lopes et al. [17] that shows that there are non-ambiguous words and sequences of words that may significantly improve tagging accuracy, and souza et al. [3] that creates a ud corpus for domain specific documents. nevertheless, in practically all languages and domains there is a large number of challenging words that are ambiguous and require more efforts to be properly annotated. we focus here on investigating and proposing methods for solving the problem of pos tag disambiguation of words of a particular case in portuguese: the closed class words. such word classes include prepositions, conjunctions, determiners, numbers, pronouns and primitive adverbs (non-derived adverbs). the motivation for such choice relies on the fact that closed class words may be very ambiguous regarding pos tagging and, once solved, may serve as solid evidence to help disambiguating words of other word classes, improving tagging accuracy overall. for instance, in portilexicon-ud [16], which is a ud-based lexicon for portuguese, the word “que” (in english, it usually corresponds to “what”, “that” or “which”) may have up to 7 different word classes and represents a challenge for annotation, as discussed in the work of duran et al. [6]. to exemplify the challenges, we reproduce some examples from this work: the word “que” is: a pronoun in the sentence “as coisas com que sonhamos são pistas para o autoconhecimento”, a subordinate conjunction in “é obvio que quero votar”, a determiner in “ que maravilha”, and an adverb in “ que horrível foi esse acidente”. the work of silva et al. [25] evidences the practical difficulties in pos tagging. some of these classes, as pronouns and subordinate conjunctions, achieve some of the lowest accuracy values (below 90%) in the reported experiments. in our paper we explore three different methods to disambiguate the words of the selected classes: a markovian model similar to the one developed by assunção et al. [2] to disambiguate each ambiguous word, considering the pos tag sequences of a training set; the tagger/parser udpipe 2.0 [28] pretrained for portuguese is used to tag the whole sentences, and we observe the decision took for ambiguous words; a contextual word embedding-based solution using a pretrained portuguese model to individually disambiguate the pos tag for ambiguous words, in a similar approach to vandenbussche et al. [30] that uses bert (bidirectional encoder representations from transformers) [4] by a pretrained language model specific for brazilian portuguese, the bertimbau [26]. we perform a detailed error analysis in comparison with two baselines and also explore an ensemble solution considering a voting system taking into account the three presented models’ outcomes. our results show accuracy values ranging from 90% to 98% for the three proposed methods, as well as the ensemble one. while the bertimbau-based approach delivers the higher accuracy values for the best cases experimented, the ensemble solution presents more stable results since the accuracy of all experimented cases has a smaller standard deviation than all three methods applied individually. the achieved accuracy is relevant and the gains in terms of annotation effort are considerable, as we correct nearly 25% of all tokens of large portuguese corpora. this paper is organized as follows. the next section presents the main related work. the problem definition and the baselines for this study are presented in sect. 3. section 4 presents the three experienced models: the markovian model, the udpipe 2.0 application, and our contextual word embedding bertimbau-based approach, respectively. section 5 shows our experiments over an example corpus and discusses the achieved effectiveness. finally, the final section summarizes our contributions and suggests some future work. 2 related work we can find in the literature related works using varied techniques to deal with pos ambiguities. we comment on the main ones in what follows. ehsani et al. [7] perform the disambiguation of pos tags for turkish using conditional random fields (crf)  [14]. in such work the authors point out the processing burden associated using crf to perform the task, but they deliver pos tag accuracies around 97%. the authors also mention that this high pos tag accuracy is very beneficial to support reliable lemma and morphological annotations. a different approach is the work of assunção et al. [2] that uses a markovian model to perform pos tag disambiguation based on markovian models [13] and a partially annotated corpus. in this work, the disambiguation follows a basic probability computation of pos tag sequences, and therefore delivers faster computations than more sophisticated approaches. the work of hoya quecedo et al. [10] presents an effort to disambiguate pos tag and lemma for morphological rich languages, which is the case of portuguese, using a neural network model based on bilstm (bidirectional long short-term memory) [8]. hoya quecedo et al. build word embeddings to submit to the bidirecional model and, thus, estimate the probability of the pos tag and lemma options of each word. these experiments deliver a precision varying from 80% to 85% to corpora in finnish, russian, and spanish. another interesting work published by muñoz-valero et al. [19] also employs an lstm (long short-term memory) rnn (recurrent neural networks) [5] to disambiguate words in the american national corpus [11]. similar to this one, the works of shen et al. [24] and zalmout and habash [32] aim at the disambiguation of morphological features, both using lstm and rnn. 3 problem definition and baseline approaches the problem we are aiming to solve is how to disambiguate closed class words that can be tagged with different pos tags, for example, the word “por” that in portuguese can be either an adp (in english, “by”) or a verb (in english, “to put”). our proposed methods are based on working with a training set of fully annotated sentences using universal dependencies (ud) in the conll-u format [20] in order to disambiguate the pos tag annotation of all ambiguous words of a test set. the set of ud pos tags is formed by: adj adjectives, as “bonito” (“beautiful” in english); adp adpositions, as “de” (“of” in english); adv adverbs, as “não” (“no” in english); aux auxiliary verbs, as “foi” (“was” in english); cconj coordinating conjunctions, as “e” (“and” in english); det determiners, as “cujo” (“whose” in english); intj interjections, as “tchau” (“goodbye” in english); noun nouns, as “vida” (“life” in english); num numerals, as “cinco” (“five” in english); part particles, which is not employed in portuguese; pron pronouns, as “ele” (“he” in english); propn proper nouns, as “brasil” (“brazil” in english); punct punctuations, as “?”; sconj subordinating conjunctions, as “porque” (“because” in english); sym symbols, as “$”; verb verbs, as “jogamos” (“(we) play” in english); x others, as foreign words. the closed classes considered in this paper are adp, cconj, det, num, pron and sconj, plus subsets of the classes advfootnote 1, auxfootnote 2, and adjfootnote 3. table 1 shows the considered closed classes, the total number of words (#tot.), the number of ambiguous words (#amb.), and some examples. considering that many ambiguous tokens belong to several classes, the overall sum of ambiguous tokens is 368 different words. a full list of the considered closed class words is available at https://sites.google.com/icmc.usp.br/poetisa/publications. table 1. closed pos tags considered in our work.full size table the first baseline (baseline 1) that we evaluate takes into account the number of occurrences of each ambiguous word in the training set and uses the most common pos tag to all occurrences in the test set. for example, if the word “até” that can be an adp (“until” in english) or an adv (“even” in english) is found 189 times as adp and 77 times as adv, we will assume that all occurrences of the word “até” will be considered adp. however, to prevent decisions based on too sparse data at the training set, we will only consider a prediction when there are at least 3 occurrences of a word with the most frequent pos tag. additionally, to prevent decisions based on too close differences, the second more frequent pos tag has to be less than half frequent than the more frequent one. the second baseline considered is the implementation following the definitions made by the work of hoya quecedo et al. [10] that uses a neural network model based on bilstm as described before. we employ this baseline using the same hyperparameters defined in [10] and we will refer to this as baseline 2. 4 prediction methods in this section we present the three methods implemented in these paper experiments. the first one is a traditional markovian modeling approach inspired in the work of assunção et al [2] and it is presented in subsect. 4.1. the second method is the application of the udpipe 2.0 tool [27] and it is presented at subsect. 4.2. the third method is an approach based on bertimbau pretrained neural language model [26] to build a contextual word embedding model and it is presented at subsect. 4.3. 4.1 prediction through markovian models among the markovian approaches to stochastically predict language information, we decided to employ the one based on assunção et al. [2]. this approach is based on developing a markov chain model to each sentence of the training set describing the probabilities of pos tag sequences. for an illustration of the method, consider the sentence “com a rescisão, as provas apresentadas pelos delatores ainda podem ser usadas.”. this sentence will be annotated as described in fig. 1. this sentence corresponds to the attached markov chain that represents the sequences of pos tags. for example, the two words tagged as adp are followed by det, therefore the probability of transition from adp to det is 100%. similarly, the three occurrences of noun are followed by punct, verb, and adv tags, therefore, each existing arc in the chain has a transition probability of 33.3%. additionally, one additional node representing the boundary of the sentence (\(\star \)) links to the sentence start (before adp) and the sentence end (after punct). fig. 1. annotated sentence and corresponding markov chain of pos tag sequences. full size image this markov chain creation process is repeated to all sentences of the training set. once all sentences of the training set are used to create a markov chain, each possible word with ambiguous pos tags is disambiguated considering the pos tags preceding and succeeding the ambiguous word. 4.2 prediction through udpipe 2.0 the second method experimented is the use of the tagger/parser udpipe 2.0 [28] using one of the available portuguese training models, which is bosque-pt [29] in our experiment, composed by 9,364 sentences and 210,957 tokens. udpipe 2.0 uses a rnn based on bilstm to perform a full ud annotation based on a previously training set. our approach in this paper is to feed the whole sentences of the testing set to udpipe 2.0 and pinpoint the ambiguous words to observe the accuracy of disambiguation of the target closed class words. 4.3 prediction through bertimbau-based model a more recent approach, which has achieved state of the art results in many nlp tasks [9, 15], is based on the usage of pretrained neural language models (lm), such as bert [4]. a lm is capable of providing contextualized word embeddings, which can later be used in a downstream task. specifically, we use the language model bertimbau, a bert model pretrained for brazilian portuguese [26], in the smallest version, bertimbau-base, that, like bert-base, has 12 layers. all sentences are previously tokenized using the same bertimbau model, as it requires the words to be split into subtokens. for words consisting of more than one subtoken, the embedding used during classification represents the first one. for the pos disambiguation, our model has two inputs: the sentence duly tokenized and the position of the token to disambiguate. bertimbau is used to retrieve a sentence vector (which represents the whole context) and also the word embedding for the token to disambiguate (to include specific information about the token), which are later concatenated and passed through a linear layer in order to create a combined representation. finally, this vector is fed into another linear layer with softmax for classification into pos tags, plus a dropout layer between the combined representation and the final classification layer in order to tackle over fitting issues. figure 2 depicts the general process for the bertimbau-based method. fig. 2. bertimbau-based method process. full size image for the fine tuning of the model, the cross entropy loss function was optimized using adam [12] with learning rate \(\alpha = 10^{-5}\) for 2 epochs. the dropout rate used was 0.2 with batch size 8. the whole model was implemented using huggingface’s transformers library [31] alongside pytorch [21] and it is available as supplemental materialfootnote 4. 5 experiments the conducted experiments were performed over an annotated corpus in brazilian portuguese language with 8,420 sentences (168,397 tokens) extracted from the first five thousand news from folha kaggle data bank [23]. this corpus was manually annotated using ud tags by a group of 10 annotators with redundancy and supervised by a chief linguist. it contains (among its 168,397 tokens) 44,208 occurrences of 368 distinct ambiguous words. to allow performing the experiments and comparing all the methods, we randomly split the corpus into: train a training data set with approximately 80% of the tokens of the full corpus; dev a development data set with approximately 10% of the tokens of the full corpus; test a testing data set with approximately 10% of the tokens of the full corpus. however, to prevent bias in our experiments, we created 10 of such random splits as folds to perform cross-validation in order to establish the accuracy of each method. the application of baseline 1 and the markov-based model employs the train+dev data sets to training and test data set to testing. the application of udpipe 2.0 method employs test data set as testing and ignores the data sets train and dev. the application of baseline 2 and bertimbau-based model employ each split individually. table 2 presents the accuracy and f1 obtained for each fold, while table 3 summarizes the results obtained by each method for each fold in terms of the averagefootnote 5, standard deviation, minimum, and maximum results of the ten folds individually. in these tables, the methods are abbreviated as markov, for the markov-based method (subsect. 4.1), udpipe for the udpipe 2.0 application method (subsect. 4.2), and bert for the bertimbau-based method (subsect. 4.3). finally, in these tables we included the accuracy and f1 results of an ensemble method. the ensemble is obtained applying the markov-based, udpipe 2.0, and bertimbau-based methods, and deciding on the pos tag by a simple voting strategy among them. table 2. accuracy and f1 values of each method for each of the ten folds.full size table table 3. summarization of accuracy and f1 values considering the ten folds.full size table the first observation from the results in table 3 is that the three methods are significantly more accurate than the baselines. it is also noticeable the high accuracy achieved by the bertimbau-based approach that alone provides a better accuracy than the ensemble method. the second observation is the stability of the ensemble method delivering smaller standard deviation both for accuracy and f1. the accuracy achieved by the ensemble method is slightly smaller than the bertimbau-based one, but since the its variance of results is smaller, it is probably more reliable to employ the ensemble method in a real annotation where the gold standard is not previously known. to illustrate the difficulties of the phenomenon, we analyzed the number of mistaken decisions (wrong pos tags) made by each of the three methods, stated in the first three rows of table 4. we also present in the middle rows of this table the mistakes shared by the methods, two by two, and all together. the last rows stated the total mistakes by methods and the use of an ensemble of the three methods application. table 4. number of wrong pos tags by method for each of the ten folds of the pos tag errors.full size table fig. 3. number of wrong pos tags by each method for fold 4 full size image figure 3 shows a venn diagram of the pos tag errors made by the three methods for fold 4 (which is a fold with number of errors nearest to the average of all folds, as highlighted in table 4). for this fold, there were 319 pos tag errors by the markov-based method alone, 187 by the udpipe 2.0 method alone, and 9 by the bertimbau-based method alone. similarly, 50 pos tag errors were made by both markov-based and udpipe 2.0 methods, 18 by markov-based and bertimbau-based methods, and 16 by the udpipe 2.0 and bertimbau-based methods. finally, 29 errors were made by all three methods. these lead to 416 errors by markov-based (319+50+18+29), 282 by udpipe 2.0 (187+50+16+29), and 72 by bertimbau-based (9+18+16+29). since the ensemble method will always decide wrongfully when the three methods are wrong (29 errors), when any two methods are wrong (50+18+16), the errors of the ensemble method will appear on 113 pos tags (29+50+18+16). analyzing fold 4 testing set outcome, we observe that it is composed by 842 sentences and a total of 16,938 tokens. from these, 4,354 tokens are ambiguous words from closed classes, and only 72 were incorrect after applying bertimbau-based method, i.e., the method produced 4,282 correct tokens (25.21% of the total tokens), which correspond to 98.35% of the ambiguous tokens. to illustrate the errors of the methods, we use two words from the 29 mistakes in the testing set for fold 4. the token “segundo” (“second” in english) at the end of the sentence “flamengo e são paulo viraram 12 e 11 pontos de distância, mas não estavam em segundo.” is tagged as adj, but the methods delivered a noun prediction, because the methods had difficulty to handle the noun elipse, as the full form would be “segundo lugar” (“second place” in english). the first token “a” (“her” in english) at the sentence “ela recebeu uma mensagem certo dia chamando a a ir a cariri...” is tagged as pron. the three methods delivered adp prediction because a double token sequence “a a” is the usual tokenized form of crasis (“à”, in english “to the”), a very frequent form in portuguese that is tagged as adp det, respectively. in the training set for fold 4, the number of crasis occurrences is 276, while other occurrences of double “a” were never present, consequently the three methods were unable to learn tags pron adp for the sequence, assigning det instead of pron for it. one may see that both cases are difficult ones, which somehow explains why the methods failed. 6 conclusion we conducted an experiment with three proposed methods that show to be competitive and delivered an accuracy gain over the baselines, reaching average accuracy values ranging from 90.76% to 97.84%. the f1 average values were also impressive, ranging from 94.89% to 99.11%. it is important to mention that, since we experiment using a 10-fold cross-validation, our results are statistically sound, as the standard deviation of the results of each fold was always lowfootnote 6. it is particularly noticeable that the bertimbau-based approach brings a relevant correction power. recalling fold 4 testing set correction of 25.21% of the total tokens, we observe that, to a large corpus as the folha-kaggle [23] with around 84 million tokens, we can automatically deliver more than 20 million correct tokens for ambiguous words of closed classes. it is also noticeable that combining the three presented methods in an ensemble method has shown a small impact in the average accuracy with respect to bertimbau-based method, respectively 97.23% and 97.84%, but the ensemble solution offers more stable results, since the standard deviation of the 10 folds dropped from 0.0034 for bertimbau-based solution to 0.0024 for the ensemble one. as future work, we intend to perform a deeper analysis of the ensemble approach, perhaps adding other methods casting votes together with the three methods already developed. it is also possible to imagine the development of a similar technique to disambiguate other fields of the ud model, as lemma and morphological features, given the promising results achieved by our methods. notes 1.similarly to english with the ending -“ly”, in portuguese it is possible to turn adjectives into adverbs by adding -“mente” at the end. we disconsider those -ly adverbs, as only the primitive adverbs are a closed class. 2.the verbs ser and estar (“to be” in english) are always annotated as aux, either by being true auxiliary verbs, either by being copula verbs. the verbs “ir”, “haver”, and “ter” (“to go”, “to exist”, and “to have” in english) are sometimes annotated as verb, sometimes annotated as aux (as “going to” and “have” + a past participle in english). 3.while adjectives are not a closed class, the adjectives that are ordinal numbers are considered belonging to a closed subset of class adj. 4.https://github.com/nilc-nlp/pos-disambiguation. 5.the values stated as average are the macro average of the values of each fold, but since the folds have about the same size, the values for micro and macro average have are practically the same (less than 0.01% difference). 6.for reproducibility purposes, all data (including fold splits) and implementation of all methods are available at https://sites.google.com/icmc.usp.br/poetisa/publications. references afonso, s., bick, e., haber, r., santos, d.: floresta sintá(c)tica: a treebank for portuguese. in: proceedings of the third international conference on language resources and evaluation (lrec’02). elra, las palmas, canary islands spain (may 2002), http://www.lrec-conf.org/proceedings/lrec2002/pdf/1.pdf assunção, j., fernandes, p., lopes, l.: language independent pos-tagging using automatically generated markov chains. in: proceedings of the 31st international conference on software engineering & knowledge engineering, pp. 1–5. lisbon, portugal (2019). https://doi.org/10.18293/seke2019-097 de souza, e., freitas, c.: polishing the gold-how much revision do we need in treebanks? in: procedings of the universal dependencies brazilian festival, pp. 1–11 (2022). https://aclanthology.org/2022.udfestbr-1.2.pdf devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2018). https://doi.org/10.48550/arxiv.1810.04805, https://arxiv.org/abs/1810.04805 dipietro, r., hager, g.d.: chapter 21 deep learning: rnns and lstm. in: zhou, s.k., rueckert, d., fichtinger, g. (eds.) handbook of medical image computing and computer assisted intervention, pp. 503–519. the elsevier and miccai society book series, academic press (2020). https://doi.org/10.1016/b978-0-12-816176-0.00026-0 duran, m., oliveira, h., scandarolli, c.: que simples que nada: a anotação da palavra que em córpus de ud. in: proceedings of the universal dependencies brazilian festival, pp. 1–11 (2022). https://aclanthology.org/2022.udfestbr-1.3 ehsani, r., alper, m.e., eryiğit, g., adali, e.: disambiguating main pos tags for turkish. in: proceedings of the 24th conference on computational linguistics and speech processing (rocling 2012), pp. 202–213. the association for computational linguistics and chinese language processing (aclclp), chung-li, taiwan (2012). https://aclanthology.org/o12-1021 gers, f.a., schmidhuber, j.a., cummins, f.a.: learning to forget: continual prediction with lstm. neural comput. 12(10), 2451–2471 (2000). https://doi.org/10.1162/089976600300015015 article  google scholar  hoang, m., bihorac, o.a., rouces, j.: aspect-based sentiment analysis using bert. in: proceedings of the 22nd nordic conference on computational linguistics, pp. 187–196. linköping university electronic press, turku, finland (2019). https://aclanthology.org/w19-6120 hoya quecedo, j.m., maximilian, k., yangarber, r.: neural disambiguation of lemma and part of speech in morphologically rich languages. in: proceedings of the 12th language resources and evaluation conference, pp. 3573–3582. european language resources association, marseille, france (2020). https://aclanthology.org/2020.lrec-1.439 ide, n., suderman, k.: integrating linguistic resources: the american national corpus model. in: proceedings of the fifth international conference on language resources and evaluation (lrec’06). elra, genoa, italy (2006). http://www.lrec-conf.org/proceedings/lrec2006/pdf/560_pdf.pdf kingma, d.p., ba, j.: adam: a method for stochastic optimization. in: bengio, y., lecun, y. (eds.) proceedings of the 3rd international conference on learning representations (2015). http://arxiv.org/abs/1412.6980 kupiec, j.: robust part-of-speech tagging using a hidden markov model. comput. speech lang. 6(3), 225–242 (1992). https://www.sciencedirect.com/science/article/pii/088523089290019z lafferty, j.d., mccallum, a., pereira, f.c.n.: conditional random fields: probabilistic models for segmenting and labeling sequence data. in: proceedings of the eighteenth international conference on machine learning, pp. 282–289. icml ’01, morgan kaufmann publishers inc., san francisco, ca, usa (2001). https://dl.acm.org/doi/10.5555/645530.655813 liu, y., lapata, m.: text summarization with pretrained encoders. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 3730–3740. association for computational linguistics, hong kong, china (2019). https://doi.org/10.18653/v1/d19-1387 lopes, l., duran, m., fernandes, p., pardo, t.: portilexicon-ud: a portuguese lexical resource according to universal dependencies model. in: proceedings of the language resources and evaluation conference, pp. 6635–6643. european language resources association, marseille, france (2022). https://aclanthology.org/2022.lrec-1.715 lopes, l., duran, m.s., pardo, t.a.s.: universal dependencies-based pos tagging refinement through linguistic resources. in: proceedings of the 10th brazilian conference on intelligent system. bracis’21 (2021). https://link.springer.com/chapter/10.1007/978-3-030-91699-2_41 de marneffe, m.c., manning, c.d., nivre, j., zeman, d.: universal dependencies. comput. linguist. 47(2), 255–308 (2021). https://doi.org/10.1162/coli_a_00402, https://aclanthology.org/2021.cl-2.11 muñoz-valero, d., rodriguez-benitez, l., jimenez-linares, l., moreno-garcia, j.: using recurrent neural networks for part-of-speech tagging and subject and predicate classification in a sentence. int. j. comput. intell. syst. 13, 706–716 (2020). https://doi.org/10.2991/ijcis.d.200527.005 article  google scholar  nivre, j., et al.: universal dependencies v1: a multilingual treebank collection. in: proceedings of the tenth international conference on language resources and evaluation (lrec’16), pp. 1659–1666. elra, portorož, slovenia (2016). https://aclanthology.org/l16-1262 paszke, a., et al.: pytorch: an imperative style, high-performance deep learning library. in: wallach, h., larochelle, h., beygelzimer, a., d’alché-buc, f., fox, e., garnett, r. (eds.) advances in neural information processing systems 32, pp. 8024–8035. curran associates, inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf rademaker, a., chalub, f., real, l., cláudia freitas, bick, e., de paiva, v.: universal dependencies for portuguese. in: proceedings of the fourth international conference on dependency linguistics (depling), pp. 197–206 (2017) google scholar  santana, m.: kaggle news of the brazilian newspaper. https://www.kaggle.com/marlesson/news-of-the-site-folhauol, accessed: 2021-06-14 shen, q., clothiaux, d., tagtow, e., littell, p., dyer, c.: the role of context in neural morphological disambiguation. in: proceedings of coling 2016, the 26th international conference on computational linguistics: technical papers, pp. 181–191. osaka, japan (2016). https://aclanthology.org/c16-1018 silva, e., pardo, t., roman, n., fellipo, a.: universal dependencies for tweets in brazilian portuguese: tokenization and part of speech tagging. in: anais do xviii encontro nacional de inteligência artificial e computacional. pp. 434–445. sbc, porto alegre, rs, brasil (2021). https://doi.org/10.5753/eniac.2021.18273, https://sol.sbc.org.br/index.php/eniac/article/view/18273 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, october 20–23 (2020), https://link.springer.com/chapter/10.1007/978-3-030-61377-8_28 straka, m.: udpipe 2.0 prototype at conll 2018 ud shared task. in: proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pp. 197–207 (2018). https://aclanthology.org/k18-2020 straka, m., straková, j.: tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. in: proceedings of the conll 2017 shared task: multilingual parsing from raw text to universal dependencies, pp. 88–99. association for computational linguistics, vancouver, canada (2017), https://aclanthology.org/k17-3009 universal dependencies: ud portuguese bosque ud version 2. https://universaldependencies.org/treebanks/pt_bosque/index.html. accessed 14 jun 2021 vandenbussche, p.y., scerri, t., jr., r.d.: word sense disambiguation with transformer models. in: proceedings of the 6th workshop on semantic deep learning (semdeep-6), pp. 7–12. association for computational linguistics, online (2021) https://aclanthology.org/2021.semdeep-1.2 wolf, t., et al.: transformers: state-of-the-art natural language processing. in: proceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38–45. association for computational linguistics, online (2020). https://www.aclweb.org/anthology/2020.emnlp-demos.6 zalmout, n., habash, n.: don’t throw those morphological analyzers away just yet: neural morphological disambiguation for arabic. in: proceedings of the 2017 conference on empirical methods in natural language processing, pp. 704–713. association for computational linguistics, copenhagen, denmark (2017). https://aclanthology.org/d17-1073 download references acknowledgements this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp grant number 2019/07665-4) and by the ibm corporation. the project was also supported by the ministry of science, technology and innovation, with resources of law n. 8.248, of october 23, 1991, within the scope of ppi-softex, coordinated by softex and published as residence in tic 13, dou 01245.010222/2022-44. author information authors and affiliations icmc, university of são paulo, são carlos, brazil lucelene lopes, magali s. duran & thiago a. s. pardo merrimack college, north andover, ma, usa paulo fernandes cisuc, universidade de coimbra, coimbra, portugal marcio l. inacio authors lucelene lopesview author publications search author on:pubmed google scholar paulo fernandesview author publications search author on:pubmed google scholar marcio l. inacioview author publications search author on:pubmed google scholar magali s. duranview author publications search author on:pubmed google scholar thiago a. s. pardoview author publications search author on:pubmed google scholar corresponding author correspondence to lucelene lopes . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper lopes, l., fernandes, p., inacio, m.l., duran, m.s., pardo, t.a.s. (2023). disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_16 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords part-of-speech tagging universal dependencies disambiguation publish with us policies and ethics profiles marcio l. inacio view author profile magali s. duran view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature coarsening algorithm via semi-synchronous label propagation for bipartite networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper coarsening algorithm via semi-synchronous label propagation for bipartite networks conference paper first online: 28 november 2021 pp 437–452 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) coarsening algorithm via semi-synchronous label propagation for bipartite networks download book pdf download book epub alan demétrius baria valejo10, paulo eduardo althoff11, thiago de paulo faleiros11, maria lígia chuerubim12, jianglong yan13, weiguang liu13 & … liang zhao14  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 751 accesses 3 citations abstract several coarsening algorithms have been developed as a powerful strategy to deal with difficult machine learning problems represented by large-scale networks, including, network visualization, trajectory mining, community detection and dimension reduction. it iteratively reduces the original network into a hierarchy of gradually smaller informative representations. however, few of these algorithms have been specifically designed to deal with bipartite networks and they still face theoretical limitations that need to be explored. specifically, a recently introduced algorithm, called mlpb, is based on a synchronous label propagation strategy. in spite of an interesting approach, it presents the following two problems: 1) a high-cost search strategy in dense networks and 2) the cyclic oscillation problem yielded by the synchronous propagation scheme. in this paper, we address these issues and propose a novel fast coarsening algorithm more suitable for large-scale bipartite networks. our proposal introduces a semi-synchronous strategy via cross-propagation, which allows a time-effective implementation and deeply reduces the oscillation phenomenon. the empirical analysis in both synthetic networks and real-world networks shows that our coarsening strategy outperforms previous approaches regarding accuracy and runtime. this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp) under grant number: 2019/07665-4 and by the ibm corporation. this work is also supported in part by fapesp under grant numbers 2013/07375-0, 2015/50122-0 and 2019/14429-5, the brazilian national council for scientific and technological development (cnpq) under grant number 303199/2019-9, and the ministry of science and technology of china under grant number: g20200226015. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a review and comparative analysis of coarsening algorithms on bipartite networks article 07 june 2021 community detection in bipartite network: a modified coarsening approach chapter © 2018 analyzing the impact of coarsening on k-partite network classification chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. coarse-grained models complex networks dynamic networks network topology network models stochastic networks 1 introduction bipartite networks are a broadly pervasive class of networks, also known as two-layer networks, where the set of nodes is split into two disjoint subsets called “layers” and links can connect only nodes of different layers. these networks are widely used in science and technology to represent pairwise relationship between categories of entities, e.g. documents and terms, patient and gene expression (or clinical variables) or scientific papers and their authors [17, 19]. over the last years, there has been a growing scientific interest in bipartite networks given their occurrence in many data analytic problems, such as community detection and text classification. several coarsening algorithms have been proposed as a scalable strategy to address hard machine learning problems in networks, including network visualization [29], trajectory mining [15], community detection [30] and dimensionality reduction [21]. these algorithms build a hierarchy of reduced networks from an initial problem instance, yielding multiple levels-of-detail, fig. 1. it is commonly used for generating multiscale networks and, most notably, as a step of the well-known multilevel method. fig. 1. coarsening process. in (a), group of nodes are matched; in (b), the original network is coarsened, i.e., matched nodes are collapsed into a super-node and links incident in matched nodes are collapsed into super-edges; the coarsest network is illustrated in (c). the coarsening process is repeated, level by level, until the desired network size is reached. full size image however, only a few coarsening algorithms have been specifically designed to deal with bipartite networks, as showed in a recent survey [28], and they still face theoretical limitations that open for scientific investigation. specifically, a recently introduced algorithm, proposed in [24], called mlpb, is based on a label propagation strategy that uses the diffusion of information to build a hierarchy of informative simplified representations of the original network. it implements a high time-cost strategy that searches the whole two-hop neighborhood of each node, which limits its use to sparse networks with a low link-density. as an additional limitation, mlpb uses a synchronous strategy which is known to yield a cyclic oscillation problem in some topological structures, as bipartite components. to overcome these issues, we propose a novel fast coarsening algorithm based on the cross-propagation concept suitable for large-scale bipartite networks. specifically, two-fold contribution: we design a novel coarsening algorithm that uses a semi-synchronous strategy via cross-propagation, which only considers the direct neighbors of nodes, which implies a cost-efficient implementation and can deeply reduce the oscillation phenomenon. we improve the classical cross-propagation strategy using the multilevel process by adding two restrictions: the first defines the minimum number of labels at the algorithm convergence and the second enforces size constraints to groups of nodes belonging to the same label. these restrictions increase the potential and adaptability of cross-propagation to foster novel applications in bipartite networks and can foster future research. the empirical analysis, considering a set of thousands of networks (both synthetic and real-world networks), demonstrated that our coarsening strategy outperforms previous approaches regarding accuracy and runtime. the remainder of the paper is organized as follows: first, we introduce the basic concepts and notations. then, we present the proposed coarsening strategy. finally, we report results and summarize our findings and discuss future work. 2 background a network \(\mathcal {g}=(\mathcal {v}, \mathcal {e}, \sigma , \omega )\) is bipartite (or two-layer) if its set of nodes \(\mathcal {v}\) and \(\mathcal {v}^1 \cap \mathcal {v}^2 = \emptyset \). \(\mathcal {e}\) is the set of links, wherein \(\mathcal {e}\subseteq \mathcal {v}^1 \times \mathcal {v}^2\). a link (u, v) may have an associated weight, denoted as \(\omega (u, v)\) with \(\omega : \mathcal {v}^1 \times \mathcal {v}^2 \rightarrow \mathbb {r}\); and a node u may have an associated weight, denoted as \(\sigma (u)\) with \(\sigma : v \rightarrow \mathbb {r}\). the degree of a node \(u \in \mathcal {v}\) is denoted by \(\kappa (u) = \sum _{v \in \mathcal {v}}w(u, v)\). the h-hop neighborhood of u, denoted by \(\vargamma _h(u)\), is formally defined as the nodes in set \(\vargamma _h(u) = \{v\ | \) there is a path of length h between u and \(v\}\). thus, \(\vargamma _1(u)\) is the set of nodes adjacent to u; \(\vargamma _2(u)\) is the set of nodes 2-hops away from u, and so forth. 2.1 label propagation the label propagation (lp) algorithm is a popular, simple and time-effective algorithm, commonly used in community detection [20]. every node is initially assigned to a unique label, then, at each iteration, each node label is updated with the most frequent label in its neighborhood, following the rule: $$\begin{aligned} l_u^{'} = \mathop {{{\,\mathrm{arg\,max}\,}}}\limits _{{l \in \mathcal {l}}} \sum _{v \in \vargamma _1(u)} \delta (l_v, l), \end{aligned}$$ (1) wherein \(l_v\) is the current label of v, \(l_u^{'}\) is the new label of u, \(\mathcal {l}\) is the label set for all nodes and \(\delta \) is the kronecker’s delta. intuitively, groups of densely connected nodes will converge to a single dominant label. lp has been widely studied, extended and enhanced. the authors in [1] proposed a modified algorithm that maximizes the network modularity. the authors in [11] presented a study of lp in bipartite networks. the authors in [12], improved in [2], introduced a novel algorithm that maximizes the modularity through lp in bipartite networks. the authors in [7] presented a variation of this concept to k-partite networks. in the multilevel context, the authors in [14] proposed a coarsening algorithm based on lp and, recently, the authors in [24] extended this concept to handle bipartite networks. synchronous lp formulation can yield cyclic oscillation of labels in some topological structures, as bipartite, nearly bipartite, ring, star-like components and other topological structures within them. specifically, after an arbitrary step, labels values indefinitely oscillate between them, i.e. a node exchanges its label with a neighbor and, in a future iteration, this exchange is reversed. this problem is illustrated in fig. 2. fig. 2. oscillation phenomenon. in (a), labels are randomly assigned to nodes; in (b), a propagation process updates the labels; in the subsequent iterations, labels values indefinitely oscillate between them. full size image to suppress this problem, it is used the asynchronous [20] or semi-synchronous [4] strategy, in which a node or a group of nodes is updated at a time, respectively. for bipartite networks are common to apply the cross-propagation concept, a semi-synchronous strategy [11], in which nodes in a selected layer are set as propagators and nodes in the other layer are set as receivers. the process is initially performed from the propagator to the receivers, then it is performed in the reverse direction, as illustrated in fig. 3. fig. 3. cross-propagation in bipartite networks. in (a), labels are propagated from top layer to bottom layer; in (b), the process is performed in the reverse direction. full size image 2.2 coarsening in bipartite networks a popular strategy to solve large-scale network problems (or data-intensive machine learning problems) is through a multiscale analysis of the original problem instance, which can involve a coarsening process that builds a sequence of networks at different levels of scale. coarsening algorithms are commonly used as a step of the multilevel method, whose aim is to reduce the computational cost of a target algorithm (or a task) by applying it on the coarsest network. it operates in three phases [25]: coarsening phase: original network \(\mathcal {g}_0\) is iteratively coarsened into hierarchy of smaller networks \(\{\mathcal {g}_1, \mathcal {g}_2, \cdots , \mathcal {g}_\mathcal {h}\}\), wherein \(g_\mathcal {h}\) is the coarsest network. the process implies in collapsing nodes and links into single entities, referred to as super-node and super-link, respectively. solution finding phase: the target algorithm or a task is applied or evaluated in the coarsest representation \(g_\mathcal {h}\). uncoarsening phase: the solution obtained in \(g_\mathcal {h}\) is projected back, through the intermediate levels \(\{g_{\mathcal {h}-1}, g_{\mathcal {h}-2}, \cdots , g_{1}\}\), until \(g_0\). it is notable that the coarsening is the key component of the multilevel method, since it is problem-independent, in contrast to the other two phases that are designed according to the target task [25]. therefore, many algorithms have been developed and, recently, some strategies able to handle bipartite networks have gained notoriety. one of the first, proposed in [22, 23], called opm\(_{hem}\) (one-mode projection-based matching algorithm), decomposes the bipartite structure into two unipartite networks, one for each layer. although it increases the range of analysis options available (as classic and already established algorithms), this decomposition can lead to loss of information, reflecting in the performance of the algorithm. later, the authors in [27] introduced two coarsening algorithms, called rgmb (random greedy matching) and gmb (greedy matching), that uses directly the bipartite structure to select a pairwise set of nodes. they use the well-known and useful concept of a two-hop neighborhood. as a drawback, performing this search on large-scale bipartite networks with a high link-density can be computationally impractical. recently, the authors in [24] proposed a coarsening based on label propagation through the two-hop neighborhood. despite its accuracy, it uses a standard and synchronous propagation strategy that can lead to instability and it does not guarantee the convergence. the growing interest in coarsening algorithms for bipartite networks is recent and current strategies faces several theoretical limitations that remain mostly unexplored, consequently, open to scientific exploration. to overcome these issues, in the next section, we present a novel coarsening strategy. 3 coarsening via semi-synchronous label propagation for bipartite networks we design a coarsening strategy via semi-synchronous label propagation for bipartite networks (clpb). we use the cross-propagation concept to diffuse labels between layers. after the convergence, nodes in the same layer that belong to the same label will be collapsed into a single super-node. 3.1 algorithm a label is defined as a tuple \(\mathcal {l}_u(l, \beta )\), wherein l is the current label and \(\beta \in [0,1] \subset \mathbb {r}^+\) is its score. at first, each node \(u \in \mathcal {v}\) is initialized with a starting label \(\mathcal {l}_u=(u, \nicefrac {1.0}{\sqrt{\kappa (u)}})\), i.e. the initial \(\mathcal {l}_u\) is denoted by its id (or name) with a maximum score, i.e. \(\beta =1.0\). to reduce the influence of hub nodesfootnote 1, in all iteration, \(\beta \) must be normalized by its node degree, as follows: $$\begin{aligned} \mathcal {l}_u^{'}=\left( l_i, \frac{\beta _i}{\sqrt{\kappa (u)}}\right) \end{aligned}$$ (2) each step propagates a new label to a receiver node u selecting the label with maximum \(\beta \) from the union of its neighbors’ labels, i.e. \(\mathcal {l}_u = \cup \mathcal {l}_v\ \forall \ v \in \vargamma _1(u)\), according to the following filtering rules: 1. equal labels \(\mathcal {l}^{eq} \subseteq \mathcal {l}u\) are merged and the new \(\beta ^{'}\) is composed by the sum of its belonging scores: $$\begin{aligned} \beta ^{'} = \sum _{(l, \beta ) \in \mathcal {l}^{eq}} \beta , \end{aligned}$$ (3) 2. the belonging scores of the remaining labels are normalized, i.e.: $$\begin{aligned} \mathcal {l}_u = \{(l_1, \frac{\beta _1}{\beta ^{sum}}), (l_2, \frac{\beta _2}{\beta ^{sum}}), \dots , (l_\gamma , \frac{\beta _\gamma }{\beta ^{sum}})\}, \end{aligned}$$ (4) $$\begin{aligned} \beta ^{sum} = \sum _{i=1}^\gamma \beta _i, \end{aligned}$$ (5) where \(\gamma \) is the number of remaining labels. 3. the label with the largest \(\beta \) is selected: $$\begin{aligned} \mathcal {l}_u^{'} = \mathop {{{\,\mathrm{arg\,max}\,}}}\limits _{{(l,\beta ) \in \mathcal {l}_u}} \mathcal {l}_u. \end{aligned}$$ (6) 4. the size of the coarsest network is naturally controlled by the user, i.e. require defining a number of reduction levels, a reduction rate or any other parameter to fit a desired network size. here, the minimum number of labels \(\eta \) for each layer is a user-defined parameter. a node \(u \in \mathcal {v}^i\), with \(i\in \{1,2\}\) define a layer, is only allowed to update its label if, and only if, the number of labels in the layer \(|\mathcal {l}^i|\) remains equal to or greater than \(\eta ^i\), i.e.: $$\begin{aligned} |\mathcal {l}^i| \le \eta ^i. \end{aligned}$$ (7) 5. at last, a classical issue in the multilevel context is that super-nodes tend to be highly unbalanced at each level [25]. therefore, is common to constrain the size of the super-nodes from an upper-bound \(\mu \in [0, 1] \subset \mathbb {r}^+\), which limits the maximum size of a group of labels in each layer: $$\begin{aligned} \mathcal {s}^i = \frac{1.0 + (\mu * (\eta ^i 1)) * |\mathcal {v}^i|}{\eta ^i}, \end{aligned}$$ (8) wherein \(\mu =1.0\) and \(\mu =0\) implies highly imbalanced and balanced groups of nodes, respectively. therefore, a node u with weight \(\sigma (v)\) can update its current label l to a new label \(l^{'}\) if, and only if: $$\begin{aligned} \sigma (u) + \sigma (l^{'}) \le s^i \quad \mathrm {and}\quad \sigma (l^{'}) = \sum _{v \in l^{'}} \sigma (v). \end{aligned}$$ (9) if restrictions 3 or 4 are not attained, the algorithm returns to step 3; the label with the maximum \(\beta \) is removed and a new ordered label is selected. the process is repeated until a label that satisfies the restrictions 3 and 4 is obtained. figure 4 shows one step of clpb in a bipartite network using the previously defined strategy. the propagation process is repeated \(\mathcal {t}\) (user-defined parameter) times until the convergence or stops when there are no label changes. fig. 4. one step of the clpb algorithm in a bipartite network. in (a), the process is performed from the top layer, considering the propagators nodes \(\in \mathcal {v}^1\), to the bottom layer, considering the receiver nodes \(\in \mathcal {v}^2\). at first, represented in (b), equal labels are merged. in (c), second step, remaining labels are normalized. in third step, the label b is selected, as showed in (d). in (e), the restriction 4 and 5 are tested. finally, label b is propagated to the node in the bottom layer, as illustrated by the black dashed line. full size image after the cross-propagation convergence, the algorithm collapses each group of matched nodes (i.e. nodes with same label) into a single super-node. links that are incident to matched nodes are collapsed into the so-called super-links. figure 5 illustrates this process. note, the clpb process does not guarantee that the desired minimum number of labels \(\eta \) will be reached at the current level, i.e. the algorithm can stop with a number of labels greater than the desired one. however, the multilevel process naturally mitigates this problem, since clpb is performed, level by level, in the subsequent coarsened networks, until the desired number of nodes are reached. i.e., the original network \(\mathcal {g}_0\) is iteratively coarsened into a hierarchy of smaller networks \(\{\mathcal {g}_1, \mathcal {g}_2, \cdots , \mathcal {g}_\mathcal {h}, \cdots \}\), wherein \(\mathcal {g}_\mathcal {h}\) is an arbitrary level. table 1 summarizes three levels automatically achieved by clpb when evaluated in the ucforum [10]. fig. 5. contraction process. in (a), group of nodes are matched using clpb algorithm; in (b), the original network is coarsened, i.e., nodes that share labels are collapsed into a super-node and links incident to matched nodes are collapsed into super-edges. full size image table 1. ucforum: contains 899 users and 522 posts on forums. considering \(\eta ^1=30\) and \(\eta ^2=50\) as an input user-parameter, clpb automatically builds three levels to reach the desired network size.full size table naturally, users can control the maximum number of levels and the reduction factor \(\rho \) for each layer, rather than input the desired number of nodes in the coarsest network. in this case, the desired number of nodes for each layer and in each level can be defined as exemplified in eq. 10. alternatively, users can stop the algorithm in an arbitrary level. however, this is a technical decision and the stop-criterion in eq. 10 is commonly used in the literature [25]. $$\begin{aligned} \eta _i = (1 \rho ^i) * |\mathcal {v}^i| \end{aligned}$$ (10) 3.2 complexity the computational complexity of the lp is near-linear regarding the number of links, i.e., \(\mathcal {o}(|\mathcal {v}|+|\mathcal {e}|)\) steps are needed at each iteration. if a constant number of \(\mathcal {t}\) iterations are considered, then \(\mathcal {o}(\mathcal {t}(|\mathcal {v}|+|\mathcal {e}|))\). the contraction process (illustrated in fig. 5), first, iterates over all matched nodes \(\in \mathcal {v}_\mathcal {h}\) to create super-nodes \(\in \mathcal {v}_{\mathcal {h}+1}\), then, each link in \(\mathcal {e}_\mathcal {h}\) is selected to create super-links \(\in \mathcal {e}_{\mathcal {h}+1}\), therefore, \(\mathcal {o}(|\mathcal {v}|+|\mathcal {e}|)\). these complexities are well-known in the literature, and the expanded discussion can be found in [25] and [20]. based on these considerations, the clpb complexity is \(\mathcal {o}(\mathcal {t}(|\mathcal {v}|+|\mathcal {e}|))\) + \(\mathcal {o}(|\mathcal {v}|+|\mathcal {e}|)\) at each level. 4 experiments we compared the performance of clpb with four state-of-the-art coarsening algorithms, namely mlpb, opm\(_{hem}\), rgmb and gmb (discussed in sect. 2 and presented in the survey [28]). first, we conducted an experiment in a set of thousands of synthetic networks and, then, we test the performance of the algorithms in a set of well-known real networks. a common and practical approach to verify the quality of a coarsened representation is mapping each super-node as a group (community or cluster) and evaluate them using quality measures. this type of analysis is considered a benchmark approach in the literature, as discussed in the recent surveys [25, 28] and in other studies, as [6, 8, 9, 18, 24]. therefore, it is natural to use this analysis in our empirical evaluation. the following two measures were considered: normalized mutual information (nmi) [13], which quantifies the quality of the disjoint clusters comparing the solution found by a selected algorithm with the baseline (or ground truth), and murata’s modularity [16], which quantifies the strength of division of a network into communities. experiments were executed on a linux machine with an 6-core processor with 2.60 ghz and 16 gb main memory. 4.1 synthetic networks the benchmark analysis was conducted on thousands of synthetic networks obtained employing a network generation tool called bnoc, proposed in [26]. each network configuration was replicated 10 times to obtain the average and standard deviation. default parameters are presented in [26]. first, we evaluated the sensibility of the algorithms regarding the noise level in the networks. the noise level is a disturbance or error in the dataset (the proportion of links wrongly inserted), e.g., 0.5 means that half of the links are not what they should be. noise can negatively affect the algorithm’s performance in terms of accuracy. a set of 1000 synthetic bipartite networks with distinct noise level was generated, as follows: \(|\mathcal {v}|=2,000\) with \(|\mathcal {v}^1|=|\mathcal {v}^2|\), noise within the range [0.0, 1.0] and 20 communities for each layer. figure 6(a) depicts the nmi values for the evaluated algorithm as a function of the amount of noise. the algorithms exhibit distinct behaviors. mlpb and clpb obtained high nmi values with a low level of noise, however, nmi values for mlpb decrease quickly after 0.22 noise level, whereas clpb decreases slowly. therefore, mlpb revealed a high noise sensibility. although gmb, rgmb and opm\(_{hem}\) algorithms obtained the lowest nmi values, mainly, within the range [0.0, 0.4], their performances decrease slowly compared with mlpb. we also evaluated the sensibility of the algorithms regarding the number of communities in the networks. a set of 1000 synthetic bipartite networks with distinct number of communities was generated, as follows: \(|\mathcal {v}|=2,000\) with \(|\mathcal {v}^1|=|\mathcal {v}^2|\), communities within the range [1, 500] and 0.3 of noise level. figure 6(b) depicts the nmi values for the evaluated algorithm as a function of the number of communities. gmb, rgmb and opm\(_{hem}\) presented a high sensibility to a low number of communities in the network, specifically, within the range [1, 100], in contrast, clpb and mlpb obtained high nmi values in the same range. within the range [200, 500], all algorithms obtained close nmi values. fig. 6. nmi of the algorithms in relation to the noise level (a) and number of communities and (b) in 2, 000 synthetic networks. full size image a nemenyi post-hoc test [5] was applied to the results depicted in figs. 6(a) and 6(b) to detect statistical differences in the performances of the algorithms. the critical difference (cd) is indicated at the top of each diagram and algorithms’ average ranks are placed in the horizontal axis, with the best ranked to the left. a black line connects algorithms for which there is no significant performance difference. according to the nemenyi statistics, the critical value for comparing the mean-ranking of two different algorithms at 95 percentiles is 0.04, i.e. significant differences are above this value. clpb was ranked first followed by gmb, the pair mlpb and rgmb and then, in last, opm\(_{hem}\). furthermore, clpb performs statistically better than mlpb, rgmb and opm\(_{hem}\) algorithms. fig. 7. nemenyi post-hoc test applied to the results depicted in figs. 6(a) and 6(b). full size image fig. 8. runtime as a function of link-density in 1, 000 synthetic networks: (b) and (c) shows the total time spent for each algorithm on the experiments, absolute and relative values, respectively. full size image we assessed the scalability of the algorithms in terms of the absolute and relative total time spent. first, a set of 1000 synthetic networks with distinct link-density was generated, as follows: link-density within the range [0.01, 0.99], wherein 0.01 indicates very sparse networks and 0.99 indicates very dense networks with \(m\approx n^2\); \(|v|=5,000\) with \(|v^1|=|v^2|\) and 20 communities at each layer. figure 8 shows how each algorithm contributed to the total time, in both absolute values, figs. 8(a) and 8(b), and relative values, fig. 8(c) (values shown on top of the bars). the total time spent running the experiments was 419, 857.968 s, or nearly 116 h. clpb spent 4, 351.2 s, which is nearly \(1.0\%\) of the total time, furthermore, clpb ran 18 to 35 times faster than the other algorithms. gmb and opm\(_{hem}\) were the most expensive algorithms. a set of 1000 synthetic bipartite networks with distinct number of nodes was generated, as follows: the number of nodes within the range [1, 000, 40, 000] and communities as a percentage of the number of nodes, i.e. \(|v|*0.01\). figure 9 shows how each algorithm contributed to the total time, in both absolute values, figs. 9(a) and 9(b), and relative values, fig. 9(c) (values shown on top of the bars). the total time spent running the experiments was 128, 151.711 s, or nearly 35 h. clpb spent 5, 644.2 s, which is nearly \(4.4\%\) of the total time, furthermore, clpb ran 3, 3 to 8, 2 times faster than the other algorithms. gmb and opm\(_{hem}\) were the most expensive algorithms. 4.2 real-world networks we considered six real-world bipartite networks available at konect (the koblenz network collection) [10]. we took the largest connected component of each network. network properties are detailed in table 2(a). murata’s modularity was used to obtain the accuracy of the algorithms by reducing the networks to \(30\%\), \(50\%\) and \(80\%\) of its original sizes. fig. 9. runtime as a function of the number of nodes in 1, 000 synthetic networks: (b) and (c) show the total time spent for each algorithm on the experiments, absolute and relative values, respectively. full size image considering \(30\%\), \(50\%\) and \(80\%\) of network reduction, summarized in tables 2(b), 2(c) and 2(d), clpb yielded the best values on 4, 5 and 3 networks, respectively; and mlpb yielded the best values on 2 networks at each of the three cases. the pair gmb and opm\(_{hem}\) obtained the best value on one network with \(80\%\) of network reduction. a nemenyi post-hoc test was applied to the results, shown in figs. 10(a), 10(b) and 10(c), obtained from \(30\%\), \(50\%\) and \(80\%\) of network reduction, respectively. figure 10(d) summarizes the overall results. according to the nemenyi statistics, the critical value for comparing the average-ranking of two algorithms at the 95 percentile is 1.11. according to fig. 10(d), clpb was ranked best, followed by mlpb, the pair gmb and opm and, in last, rgmb. furthermore, clpb performs statistically better than gmb, rgmb and opm\(_{hem}\). the empirical investigation showed that clpb yielded more accurate and stable results compared to the standard algorithms and requires considerably lower execution time. it is a strong indicator of its performance on large-enough problem sizes and must foster the development of novel scalable solutions defined in bipartite networks, including network visualization, trajectory mining, community detection or graph partitioning, data dimension reduction and optimization of high-complexity algorithms [3, 15, 27]. table 2. modularity scores of the algorithms: (a) summaries the properties of the networks; (b), (c) and (c) presented modularity scores of the algorithms considering \(30\%\), \(50\%\) and \(80\%\) of network reduction, respectively.full size table fig. 10. nemenyi post-hoc test: (a), (b) and (c) represent the results depicted in tables (b), (d) and (d); alternatively, (d) summarizes the overall results. full size image 5 conclusion we have proposed a novel time-effective semi-synchronous coarsening algorithm to handle large-scale bipartite networks, called clpb. we introduce the cross-propagation concept in the model to overcome unstable issues, as the cyclic oscillations found state-of-the-art algorithms. furthermore, clpb employs a search strategy that only visits the immediate neighborhood of each node, which is more suitable to evaluate large-scale networks even with high link-density. consequently, the algorithm has low computational complexity. empirical analysis on thousands of networks provided compelling evidence that clpb outperforms the state-of-the-art algorithms regarding accuracy and demands considerably shorter execution times, specifically, clpb was 4 to 35 times faster as compared to the established algorithms. note, this study intended to present our algorithm and validate them from an empirical approach following the state-of-the-art guidelines [28]. we now intend to employ the clpb in real applications, e.g., in future work, we plan to extend the clpb to dimension reduction and semi-supervised classification tasks. another issue that deserves further attention is its application to network visualization. notes 1.the highest-degree nodes are often called hubs. references barber, m.j., clark, j.w.: detecting network communities by propagating labels under constraints. phys. rev. e 80, 026129 (2009) google scholar  beckett, s.j.: improved community detection in weighted bipartite networks. r. soc. open sci. 3(1), 140536 (2016) google scholar  cintra, d., valejo, a., lopes, a., oliveira, m.: visualization to assist interpretation of the multilevel paradigm in bipartite graphs. in: 15th international joint conference on computer vision, imaging and computer graphics theory and applications, pp. 133–140 (2019) google scholar  cordasco, g., gargano, l.: label propagation algorithm: a semi-synchronous approach. int. j. soc. netw. min. 1(1), 3–26 (2012) article  google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) mathscinet  math  google scholar  dias, m.d., mansour, m.r., dias, f., petronetto, f., silva, c.t., nonato, l.g.: a hierarchical network simplification via non-negative matrix factorization. in: proceedings of the conference on graphics, patterns and images (sibgrapi), pp. 119–126 (2017) google scholar  ding, c., li, t., wang, d.: label propagation on k-partite graphs. in: 2009 international conference on machine learning and applications, pp. 273–278. ieee (2009) google scholar  karypis, g., kumar, v.: metis unstructured graph partinioning and sparse matrix ordering system. technical report, university of minnesota, department of computer science (1995) google scholar  karypis, g., kumar, v.: a fast and high quality multilevel scheme for partitioning irregular graphs. siam j. sci. comput. 20(1), 359–392 (1998) article  mathscinet  google scholar  kunegis, j.: konect: the koblenz network collection. in: proceedings of the 22nd international conference on world wide web, pp. 1343–1350 (2013) google scholar  liu, x., murata, t.: how does label propagation algorithm work in bipartite networks? in: 2009 ieee/wic/acm international joint conference on web intelligence and intelligent agent technology, vol. 3, pp. 5–8. ieee (2009) google scholar  liu, x., murata, t.: an efficient algorithm for optimizing bipartite modularity in bipartite networks. j. adv. comput. intell. intell. inform. 14(4), 408–415 (2010) article  google scholar  meilă, m.: comparing clusterings—an information based distance. j. multivar. anal. 98(5), 873–895 (2007) google scholar  meyerhenke, h., sanders, p., schulz, c.: partitioning complex networks via size-constrained clustering. in: gudmundsson, j., katajainen, j. (eds.) sea 2014. lncs, vol. 8504, pp. 351–363. springer, cham (2014). https://doi.org/10.1007/978-3-319-07959-2_30 chapter  google scholar  minatel, d., valejo, a., lopes, a.: trajectory network assessment based on analysis of stay points cluster. in: brazilian conference on intelligent systems (bracis), pp. 564–569 (2018) google scholar  murata, t.: modularities for bipartite networks. in: proceedings of the 20th acm conference on hypertext and hypermedia, pp. 245–250 (2009) google scholar  newman, m.e.j.: scientific collaboration networks. ii. shortest paths, weighted networks, and centrality. phys. rev. e stat. nonlinear soft matter phys. 64(1), 016132 (2001) google scholar  noack, a., rotta, r.: multi-level algorithms for modularity clustering. in: vahrenhold, j. (ed.) sea 2009. lncs, vol. 5526, pp. 257–268. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-02011-7_24 chapter  google scholar  de paulo faleiros, t., valejo, a., de andrade lopes, a.: unsupervised learning of textual pattern based on propagation in bipartite graph. intell. data anal. 24(3), 543–565 (2020) article  google scholar  raghavan, n., albert, r., kumara, s.: near linear time algorithm to detect community structures in large-scale networks. phys. rev. e stat. nonlinear soft matter phys. 76, 036106 (2007) google scholar  sakellaridi, s., fang, h.r., saad, y.: graph-based multilevel dimensionality reduction with applications to eigenfaces and latent semantic indexing. in: proceedings of the international conference on machine learning and applications (icmla), pp. 194–200 (2008) google scholar  valejo, a., ferreira, v., de oliveira, m.c.f., de andrade lopes, a.: community detection in bipartite network: a modified coarsening approach. in: lossio-ventura, j.a., alatrista-salas, h. (eds.) simbig 2017. ccis, vol. 795, pp. 123–136. springer, cham (2018). https://doi.org/10.1007/978-3-319-90596-9_9 chapter  google scholar  valejo, a., lopes, a., filho, g., oliveira, m., ferreira, v.: one-mode projection-based multilevel approach for community detection in bipartite networks. in: international symposium on information management and big data (simbig), track on social network and media analysis and mining (snman), pp. 101–108 (2017) google scholar  valejo, a., faleiros, t.p., oliveira, m.c.r.f., lopes, a.: a coarsening method for bipartite networks via weight-constrained label propagation. knowl. based syst. 195, 105678 (2020) google scholar  valejo, a., ferreira, v., fabbri, r., oliveira, m.c.r.f., lopes, a.: a critical survey of the multilevel method in complex networks. acm comput. surv. 53(2), 35 (2020) article  google scholar  valejo, a., goes, f., romanetto, l.m., oliveira, m.c.f., lopes, a.a.: a benchmarking tool for the generation of bipartite network models with overlapping communities. knowl. inf. syst. 62, 1641–1669 (2019) google scholar  valejo, a., oliveira, m.c.r.f., filho, g.p., lopes, a.a.: multilevel approach for combinatorial optimization in bipartite network. knowl. based syst. 151, 45–61 (2018) article  google scholar  valejo, a.d.b., de oliveira dos santos, w., naldi, m.c., zhao, l.: a review and comparative analysis of coarsening algorithms on bipartite networks. eur. phys. j. spec. top. (4), 1–11 (2021). https://doi.org/10.1140/epjs/s11734-021-00159-0 walshaw, c.: a multilevel algorithm for force-directed graph drawing. in: proceedings of the international symposium on graph drawing, vol. 1984, pp. 171–182 (2001) google scholar  zhu, m., meng, f., zhou, y., yuan, g.: an approximate spectral clustering for community detection based on coarsening networks. int. j. adv. comput. technol. 4(4), 235–243 (2012) google scholar  download references author information authors and affiliations department of computing, federal university of são carlos (ufscar), são carlos, sp, brazil alan demétrius baria valejo university of brasília (unb), brasília, df, brazil paulo eduardo althoff & thiago de paulo faleiros federal university of uberlândia (ufu), uberlândia, brazil maria lígia chuerubim school of computer science, zhongyuan university of technology, zhengzhou, china jianglong yan & weiguang liu department of computing and mathematics (dcm), ffclrp, university of são paulo (usp), ribeirão preto, sp, brazil liang zhao authors alan demétrius baria valejoview author publications search author on:pubmed google scholar paulo eduardo althoffview author publications search author on:pubmed google scholar thiago de paulo faleirosview author publications search author on:pubmed google scholar maria lígia chuerubimview author publications search author on:pubmed google scholar jianglong yanview author publications search author on:pubmed google scholar weiguang liuview author publications search author on:pubmed google scholar liang zhaoview author publications search author on:pubmed google scholar corresponding author correspondence to alan demétrius baria valejo . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper valejo, a.d.b. et al. (2021). coarsening algorithm via semi-synchronous label propagation for bipartite networks. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_29 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords complex networks bipartite networks multilevel method multiscaling analysis publish with us policies and ethics profiles thiago de paulo faleiros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature low birth weight in brazil vulnerable groups: an analysis based on data mining and big data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper low birth weight in brazil vulnerable groups: an analysis based on data mining and big data conference paper first online: 31 january 2025 pp 209–222 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) low birth weight in brazil vulnerable groups: an analysis based on data mining and big data download book pdf download book epub josé gustavo cunha9, tarcísio daniel pontes lucas9, andreza daniela pontes lucas10 & … monaliza de oliveira ferreira10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 213 accesses abstract low birth weight (lbw) is the most important isolated risk factor for infant mortality. several studies are conducted to understand the features related to the increased of lbw rate in brazil. however, few works aggregate national and special vulnerable groups analysis. furthermore, some data mining methodologies with potential to discover new information from datasets, such as subgroup discovery, have not yet been applied for this purpose. thus, this study aims to identify and analysis features sets related to lbw in low-income families from brazil and comparing them with six more vulnerable groups: small towns, rural areas, illiterate mothers, adolescent mothers, indigenous and quilombolas. we also development a dashboard for publicize features set related to lbw in several brazilian cities. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others factors associated with low birth weight in nepal using multiple imputation article open access 20 february 2017 evaluation of artificial intelligence models for predicting low birth weight using brazilian real data article 04 december 2024 social inequalities in child development: analysis of low-birth-weight trends in brazil, 2010–2020 article 18 april 2024 1 introduction this work aims to identify features sets related low birth weight (lbw) in brazil low-income families, in some vulnerable groups and by city. the world health organization (who) defines lbw as the birth of children weighing less than 2,500g [28]. lbw is a global public health issue that significantly affects maternal and infant health, potentially leading to severe short and long-term consequences [7, 9, 12]. additionally, lbw is the most important isolated risk factor for infant mortality [7]. in brazil, the lbw rate is around 8% [7]. several studies are conducted to understand the features related to lbw in brazil. prenatal care quality and coverage, for example, have been identified as crucial factors to reducing the occurrence of lbw [12, 15, 38]. the mother’s age is also a significant risk factor for lbw, particularly in adolescent [4, 26] and older mothers [35]. the influence of alcohol consumption during pregnancy has also been studied [29]. even maternal oral health problems can affect lbw [31]. finally, the impact of some government programs in lbw, such as the bolsa família program (pbf), are also investigated [20]. there are also some studies based on big data that aim to identifying factors related to the lbw in brazil. a study using data from 2001 to 2015 on more than 100 million brazilians, for example, investigated factors related to low birth weight from poor or extremely poor families in brazil [10]. the results showed that the following features are more related to the lbw: low maternal education, black mothers, single mothers, few prenatal visits, maternal age between 35 and 49 years, first pregnancy, and female. other studies aim to identifying factors related to the lbw in local areas or in groups that require specific attention, such as indigenous [3, 14] and quilombolas [27]. despite the large amount of work dedicated to identifying factors related to lbw in brazil and some special areas or groups, few works aggregate national in special groups analysis in the same paper. some data mining areas, as subgroup discovery, also did not was applied in order to identify features sets related to lbw in brazil. subgroup discovery is a data mining area that aims to identify features sets related to an exaggerated presence of a specific target in relation to others [1, 13, 16]. thus, it looks for patterns that are significant only in certain portions of the data set (e.g. newborns with lbw, fraudulent transactions). subgroup discovery has been used in various applications, such as bioinformatics [19], medicine [5], marketing [6], learning [34] and depression [8]. in this context, this work has the following contributions: (1) the use of subgroup discovery as a tool to identify features sets related to lbw; (2) analysis of features sets considering the national scenario of low-income families and comparing them with six more vulnerable groups: small towns, rural areas, illiterate mothers, adolescent mothers, indigenous and quilombolas; and (3) development of a dashboard for publicize features set related to lbw in more than 4,000 brazilian cities. the remainder of this article is structured as follows. in sect. 2 we present the main concepts of the subgroup discovery. in sect. 3 we show the methodology, highlighting the data processing, the use of subgroup discovery, and the steps for creating the dashboard. in sect. 4 we present the results and discussions, followed by sect. 5, where we present the conclusions, future work, and limitations. 2 subgroup discovery problem the subgroup discovery problem can be formalized as follows. let d be a labeled dataset with a set a of categorical/discrete attributes. it can be partitioned into \(d^+: \{e_{1}^+, e_{2}^+, ..., e_{|d^+|}^+\}\), representing the target attribute (positive), and \(d^-: \{e_{1}^-, e_{2}^-, ..., e_{|d^-|}^-\}\) representing the other examples (negative). let \(dom(a_i)\) be the domain of possible values for attribute \(a_i \in a\). an feature \(f_i\) is defined as the pair (attribute, value), such that \(f_i \in f\) and \(f = \cup a_i \times dom(a_i)\). a subgroup s can be represented as a features set \(f' \subset f\), where the presence of positives examples are exaggerated in relation to the negatives. let table 1 be an example of a dataset with 10 examples of newborns with the attributes: mother’s age (\(<20\), [20, 40], \(>40\)), first pregnancy (yes or no), and lbw (yes or no). let be \(lbw = yes\) the target attribute in table 1. thus, \(d^+: \{e_{7}, e_{8}\}\), \(d^= \{e_{1}, e_{2}, e_{3}, e_{4}, e_{5}, e_{6}, e_{9}, e_{10},\}\) and \(a = \{mother age, first pregnancy\}\). the features set f is given by table 2. table 1. toy dataset representing newborns with and without low birth weight (lbw).full size table table 2. features universe of table 1.full size table in this context, \(s = \{f_1, f_4\}\) (table 2) is an interesting subgroup because it concentrates two cases of lbw (\(e_7\),\(e_8\)) and only one case of non-lbw (\(e_2\)). the proportion of \(lbw=yes\) in dataset is 30% and in the subgroup s is 66.67%. the concept of "interesting" in subgroup discovery can vary depending on the context, but generally involves identifying some significant deviation from the expected behavior of the dataset. this deviation is measured using evaluation metrics. one of the most commonly used metrics in the area of subgroup discovery is weight relative accuracy (wracc) [8, 22], presented in the eq. 1, where \(tp\) are the true positives and fp are the false positives. \(|d|\) is the total number of examples in the dataset and \(|d^+|\) is the total number of positive examples. this metric calculates the proportion of the subgroup relative to the dataset, adjusting the precision of the subgroup by the prevalence rate of the positive examples. $$\begin{aligned} wracc(s) = \frac{tp + fp}{|d|} \left( \frac{tp}{tp + fp} \frac{|d^+|}{|d|} \right) \end{aligned}$$ (1) , the \(ssdp+\) [21] is an evolutionary algorithm designed for subgroup discovery, particularly effective in high-dimensional databases. it automatically adjusts mutation and crossover rates based on the performance of the top-k subgroups in each generation. the algorithm iterates until a stopping criterion is met, defined as three consecutive generations without improvement in the top-k subgroups. additionally, \(ssdp+\) [21] incorporates a diversity operator to avoid redundancy among the top-k subgroups, ensuring greater variety and relevance in the final results. 3 methodology the methodology is divided as follows. the subsect. 3.1 presents the datasets and the preprocessing steps. subsection 3.2 shows how the subgroup discovery process was applied to the datasets of subsect. 3.1. finally, subsect. 3.3 details the development process of the dashboard. 3.1 dataset the dataset utilized in this work is a subset of 100m sinasc-sim [2, 30]. it is a linkage of big data sets from brazil that include: "cadastro unico" (cadu), newborn information system (sinasc) and "bolsa familia" ("family stipend") program (pbf), described below: cadu: it registers low-income families in brazil. it contains information on family composition, education, habitation and other data for analyzing families’ socioeconomic vulnerability. the information contained in the cadu is used to select and prioritize families to government assistance. pbf: it is the largest income transfer program in brazil. around 56 million brazilians (21.3 million families) were benefited for pbf in 2023. the pbf dataset includes the families benefit received. sinasc: it registers information about births in brazil, providing information about the newborn, his mother and childbirth. the sinasc coverage 90% of brazil’s births and is a fundamental tool for monitoring some health indicators. thus, for a given newborn in 100m sinasc-sim dataset, we have information about your birth (from sinasc), your residence and family (from cadu), and the benefits received from the pbf. the cohort 100m sinasc-sim includes 114, 001, 661 million individuals, about 50% of the brazilian population [2, 30]. the dataset used in this study is a subset of 100m sinasc-sim with 31,331,817 individuals, including mothers and newborns. we reshaped this dataset in order to match each newborn with his or her mother’s information, resulting in 16,448,931 newborn-mother pairs. after that, we retained only the newborns between 2011 and 2015, resulting in 5,246,874 records. we removed all redundant attributes, identifiers, and attributes with more than 50% missing values. we excluded cases of twins or triplets (multiple pregnancies), which commonly result in low birth weight. the final dataset has 5,242,215 newborns into low-income families with 42 attributes describing them. each record has also a label that indicates two possible value for lbw: true or false. we will refer to this dataset as lbw ds (low birth weight dataset). next, we created several subsets of lbw ds by the filters: (1) born in rural areas (lbw rural), (2) born in cities with fewer than 10,000 inhabitants (lbw 10k), (3) illiterate mothers (lbw illiterate), (4) mothers under 15 years old (lbw less15years), (5) born into indigenous families (lbw indigenous), and (6) born into quilombola families (lbw quilombolas). furthermore, we created a subset for each brazilian city with more than 10 records of lbw in the lbw ds, resulting in 4,400 cutouts. 3.2 subgroups discovery application we applied the ssdp+ algorithm for subgroup discovery task with the following parameters: \(k=10\), \(sim = 0.5\), \(k_s = 2\) and the evaluation metric wracc. the parameters sim and \(k_s\) control the subgroup diversity and were chosen based on preliminary tests. additionally, ssdp+ was configured to ignore features associated with missing values (e.g. \(mother age = nan\)). for each of the 4,407 generated datasets (subsect. 3.1), we ran ssdp+ twice. in the second run, features that appeared in the first iteration were excluded. as a result, the final top-10 subgroups were formed by combining the 5 most relevant subgroups from each of the two iterations. this approach aimed to ensure diversity in subgroup descriptions, as features present in the first five subgroups did not repeat in the subsequent five subgroups. the results were saved in csv files, one for each dataset. 3.3 dashboard the dashboard development was divided into 6 steps: 1. data collection: we collected the 4407 csv files containing the top-10 subgroups, one for each city (4,400) and for the datasets lbw ds, lbw rural, lbw 10k, lbw illiterate, lbw less15years, lbw indigenous, and lbw quilombolas. 2. data consolidation and correction: we consolidated all 4407 csv files into a single file. 3. data preparation: we used regular expressions to transform the result in the consolidate csv file into insertion commands for a relational dataset (postgresql [32]) and deployed the dataset using a cloud service (render [33]). 4. dashboard prototyping: we developed an initial prototypes of the dashboard project using figma [11] and based on concepts from storytelling with data [18]. 5. development: we created a rest api using java and spring boot [37] and deployed the api on the render service. finally, we used html [23], css [24], and javascript [25] to build the dashboard consuming data from our api and deployed it on the vercel platform [39]. 6. deployment of the dashboard: the dashboard was available by a public link. 4 results and discussions the table 3 shows some information about the datasets generated in this study: lbw ds, born in rural areas (lbw rural), born in cities with fewer than 10,000 inhabitants (lbw 10k), illiterate mothers (lbw illiterate), mothers under 15 years old (lbw less15years), born into indigenous families (lbw indigenous) and born into quilombola families (lbw quilombolas). the lbw rate in low-income families (lbw ds) was 7.25% (table 3). in the lbw illiterate and lbw less15years datasets, the lbw were 7.90% and 11.60%, respectively (greater than lbw ds dataset). this is expected, since low education and age are risk factors for lbw [7, 10]. in the other datasets (lbw 10k, lbw rural, lbw indigenous, lbw quilombolas), the lbw rates were lower than lbw ds, although these groups have, in general, worst medical and housing structure. this can be explained by "low birth weight paradox", which shows that the better the medical and home structure, the higher the lbw rate [17, 36]. this occurs because the probability of a newborn with low birth weight surviving increase as the mother has more ways to deal with the problem. table 3 also indicates that the lbw rural dataset covers more than 20% of the lbw ds, while the others cover less than 10%. table 3. profile of the databases generated in the study for the application and analysis of the returned subgroups.full size table the fig. 1 presents the most relevant subgroups according to the methodology used (sect. 3.2). each subgroup in fig. 1 is represented by a \( w_i \), where \( w_1 \) to \( w_{10} \) are subgroups associated with increased risk of lbw (in purple), and \( w_{11} \) to \( w_{20} \) are associated with reduced risk of lbw (in green). the percentage represents the increase or decrease of lbw rate in each subgroup. the size of each vertex is proportional to the number of newborns belonging to the subgroup and the color intensity is proportional to increase/decrease of lbw rate. the smaller gray circles represent common features in each subgroup. fig. 1. description of the subgroups considered most relevant by the methodology applied. (color figure online) full size image figure 1 also shows that the number of prenatal visits is an important feature related to the increase and reduction of lbw risk. five subgroups with reduced lbw risk (\(w_{11}\) to \(w_{15}\)) are associated with prenatal visits above six. on the other hand, four subgroups with increased lbw risk (\(w_{1}\), \(w_{3}\), \(w_{4}\), and \(w_{5}\)) are associated with prenatal visits below six. in subgroup \(w_3\), where mothers had between 1 and 3 visits, the lbw risk increases 66%. still on the fig. 1, single and young mothers were associated with increased lbw, as shown in subgroups \(w_9\) and \(w_{10}\), respectively. the same occurred with first pregnancies, cesarean delivery, and female baby gender (subgroups \(w_6\), \(w_7\), and \(w_5\)). finally, the absence of the pbf was related to a modestly increase of lbw rate (subgroup \(w_8\)). features related to better health and housing structure, such as piped water, paved streets, garbage collection and hospital birth (subgroups \(w_2\), \(w_4\), \(w_5\), \(w_7\), \(w_8\), and \(w_9\)) were associated with increased lbw. it again can be explained by "low birth weight paradox" [17, 36]. table 4 presents the description of the top 10 subgroups according to the methodology applied in the datasets lbw ds, lbw 10k, and lbw rural. for each subgroup, we present an identifier, description, number of positive examples (\(lbw=yes\)), number of negative examples (\(lbw=no\)) and lift. in this context, lift is calculated as the lbw rate in the subgroup divided by the lbw rate in the reference dataset. therefore, a lift 1.25 for a subgroup s means that it has lbw rate 25% higher than in the reference dataset. the table 4 shows that the descriptions related to top-10 subgroups of datasets lbw ds, lbw 10k, and lbw rural are similar in relation to the kind of features, such as prenatal visits, first pregnancies, single mothers, young mothers, female gender, good housing structure, and absence from pbf. however, the impacts of these features are different for each group. for example, the number of prenatal visits between four and six increased the lbw rate 24% in the lbw ds (subgroup 1), 20% in rural areas (subgroup 11) and 32% in small cities (subgroup 21). prenatal visits between one and three, increased lbw 66% in the lbw ds dataset (subgroup 3), 48% in rural areas (subgroup 15) and 70% in small cities (subgroup 26). thus, the relationship between prenatal visits and lbw rate is considerably higher in small cities and lower in rural areas. another example is the first pregnancies, witch increases 19% the lbw rate in lbw ds and lbw 10k datasets (subgroups 6 and 24) and 26% in rural areas (subgroup 12). therefore, in rural areas (lbw rural), first pregnancies are more strongly associated with lbw when compared to the general context (lbw ds) and small cities (lbw 10k). table 4. description of the top-10 most relevant subgroups according to the methodology applied in lbw ds, lbw rural, and lbw 10k. for each subgroup, we present an identifier, description, number of lbw (yes), non-lbw (no) and lift.full size table table 5 shows the description of the 10 most relevant subgroups according to the adopted methodology, for illiterate mothers (lbw illiterate), adolescent mothers (lbw less15years), indigenous (lbw indigenous), and quilombola (lbw quilombolas). the types of features, in general, are also similar in relation to the others datasets. however, again they have different levels of relevance in relation to lbw rate increase. table 5. top-10 subgroups returned for the lbw illiterate, lbw less15years, lbw indigenous, and lbw quilombolas datasetsfull size table the mother with low education (lbw illiterate) are even more susceptible to having newborns with lbw when she is young (subgroup 36) or during their first pregnancy (subgroup 32) (table 5). among adolescent mothers (lbw less15years) who have few prenatal consultations, the rate rises by 87% (subgroup 41). additionally, being a single mother in first pregnancy increases the lbw rate by 126% (subgroup 44). another noteworthy features is that among adolescent mothers, normal delivery was associated with an increased lbw rate, whereas for other groups analyzed, normal delivery reduced the lbw rate (fig. 1). the increase of lbw rate among indigenous first pregnancy was 40%, significantly higher than in any other analyzed group (table 5). furthermore, single indigenous mothers with female newborns have 26% higher lbw rate (subgroup 55), a similar impact to the low number of consultations in the same group (subgroup 54). the impacts of these highlighted features are atypical, when compared to the other groups presented. the increased of lbw rate in quilombolas groups in first pregnancy was 34% (subgroup 61). quilombolas with low prenatal consultations and in first pregnancy have 65% more risk of lbw rate (subgroup 63). finally, among newborns who are grandchildren in the registered family, the lbw rate increased by 53% (subgroup 67). considering the most relevant subgroups according to the methodology across different vulnerable groups, it is clear that the impact of features varies significantly between them. in some specific cases, a feature that increased the lbw rate in one group reduced it in another, such as the type of delivery. these kind of differences tend to occur also between brazilian cities. figure 2 presents the dashboard screen developed to show the 10 most relevant subgroups, according to the methodology applied, for all brazilian cities that have more than 10 cases of lbw in lbw sd dataset. on the left side, there is a sidebar that allows navigation between databases and searching by city. this sidebar is the main tool for accessing different sections and functionalities of the dashboard. in the center of the screen (fig. 2), we will find a detailed description of the top-10 subgroups, organized in a table format. additionally, two important graphs are presented. the first graph illustrates the relationship of births, divided into three columns: total births, low birth weight births, and normal birth weight births. the second graph displays data related to the selected dataset or city, comparing them with the general data. this pattern of description and graphs is repeated for all other options available on the dashboard, providing a consistent and comparative visualization. the last option on the sidebar is the city search, where specific data for desired cities can be researched. figure 2 shows where to perform this search; simply click on the "buscar por cidade" (search by city) field and type the city name followed by the state abbreviation in parentheses. for example, to search for information about caruaru, type "caruaru (pe)" and click the "buscar" (search) button. the data for the searched city will be displayed according to the explained pattern. fig. 2. developed dashboard screen. full size image for those accessing the dashboard for the first time, it’s important to note that the data may take up to 60 s to load automatically. this initial waiting time only occurs on the first access. after that, all information will be presented faster. these instructions aim to facilitate navigation and usage of the dashboard, allowing for a clear and comparative visualization of the presented data. the dashboard is available at the link: https://dashbpn.vercel.app/. 5 conclusion and future work this study applied subgroup discovery aiming to identify the main features sets related to the occurrence of lbw (low birth weight) in newborns from low-income families. we further extended the analysis to some specific low-income groups: small towns, rural areas, illiterate mothers, teenage mothers, indigenous and quilombolas. the results showed that the types of features related to lbw are considerably similar among different groups, such as low number of prenatal visits, first pregnancy, single mother, low educational, female gender and good habitation and medical structure. however, these features have different impacts in each group analyzed. quilombolas and indigenous people, for instance, showed a considerably higher risk increase in lbw during the first pregnancy compared to other groups analyzed. these and other examples suggest that public policies to reduce lbw maybe are more effective if adapted to different groups. to contribute in this direction, we also developed a dashboard to present in a user-friendly manner the groups of features related to lbw in all brazilian cities that had more than 10 cases of lbw between 2011 and 2015, resulting in information for 4,400 brazilian cities. this work is part of a larger study that will include the same type of analysis for features set related to prematurity, congenital malformation, and apgar score less than seven. these three criteria are also used by brazil to consider risk at birth. therefore, investigating and providing these new results is our main future work. another future task is to make the dashboard more user-friendly by improving the terms used in subgroup descriptions and incorporating new information. lastly, the same methodology could also be applied to more recent data. references atzmueller, m.: subgroup discovery. wiley interdiscip. rev. data min. knowl. discov. 5(1), 35–49 (2015) article  math  google scholar  barbosa, g.c., et al.: cidacs-rl: a novel indexing search and scoring-based record linkage system for huge datasets with high accuracy and scalability. bmc med. inform. decis. mak. 20, 1–13 (2020) article  math  google scholar  barreto, c.t.g., tavares, f.g., theme-filha, m., cardoso, a.m.: factors associated with low birth weight in indigenous populations: a systematic review of the world literature. revista brasileira de saúde materno infantil 19, 07–23 (2019) article  google scholar  belfort, g.p., santos, m.m.a.d.s., pessoa, l.d.s., dias, j.r., heidelmann, s.p., saunders, c.: determinants of low birth weight in the children of adolescent mothers: a hierarchical analysis. ciência & saúde coletiva 23, 2609–2620 (2018) google scholar  carmona, c.j., chrysostomou, c., seker, h., del jesus, m.: fuzzy rules for describing subgroups from influenza a virus using a multi-objective evolutionary algorithm. appl. soft comput. 13(8), 3439–3448 (2013) article  math  google scholar  carmona, c.j., ramírez-gallego, s., torres, f., bernal, e., del jesús, m.j., garcía, s.: web usage mining to improve the design of an e-commerce website: orolivesur.com. expert syst. appl. 39(12), 11243–11249 (2012) google scholar  da saúde, m.: (br) atenção à saúde do recém-nascido: guia para os profissionais de saúde (2014) google scholar  de medeiros azevedo, f.c., lucas, t.d.p.: depressão entre jovens brasileiros: uma investigação baseada em mineração de subgrupos. res. soc. develop. 11(1), e10511124547–e10511124547 (2022) google scholar  de oliveira lima, m.d., et al.: associação entre peso ao nascer, idade gestacional e diagnósticos secundários na permanência hospitalar de recém-nascidos prematuros. reme-revista mineira de enfermagem 26 (2022) google scholar  falcao, i.r., et al.: factors associated with low birth weight at term: a population-based linkage study of the 100 million brazilian cohort. bmc pregnancy childbirth 20(1), 1–11 (2020) google scholar  figma. figma: ferramenta de design de interface e prototipagem (2024) google scholar  gaíva, m.a.m., lopes, f.s.p., mufato, l.f., ferreira, s.m.b.: fatores associados à mortalidade neonatal em recém-nascidos de baixo peso ao nascer. revista eletrônica acervo saúde 12(11), e4831–e4831 (2020) article  google scholar  gamberger, d., lavrac, n.: expert-guided subgroup discovery: methodology and application. j. artif. intell. res. 17, 501–527 (2002) article  math  google scholar  geerts, l., brink, l.t., odendaal, h.j.: selecting a birth weight standard for an indigenous population in a lmic: a prospective comparative study. int. j. gynecol. obstet. (2024) google scholar  gonzaga, i.c.a., santos, s.l.d., silva, a.r.v.d., campelo, v.: atenção pré-natal e fatores de risco associados à prematuridade e baixo peso ao nascer em capital do nordeste brasileiro. ciência saúde coletiva 21, 1965–1974 (2016) google scholar  herrera, f., carmona, c.j., gonzález, p., del jesus, m.j.: an overview on subgroup discovery: foundations and applications. knowl. inf. syst. 29(3), 495–525 (2011) article  math  google scholar  juárez, s., ploubidis, g.b., clarke, l.: revisiting the low birthweight paradox using a model-based definition. gac. sanit. 28(2), 160–162 (2014) article  math  google scholar  knaflic, c.n.: storytelling with data: a data visualization guide for business professionals. wiley (2015) google scholar  liu, x., wu, j., gu, f., wang, j., he, z.: discriminative pattern mining and its applications in bioinformatics. brief. bioinform. 16(5), 884–900 (2015) article  math  google scholar  lucas, a.d.p., de oliveira ferreira, m., lucas, t.d.p., salari, p.: the intergenerational relationship between conditional cash transfers and newborn health. bmc public health 22(1), 201 (2022) google scholar  lucas, t., vimieiro, r., ludermir, t.: ssdp+: a diverse and more informative subgroup discovery approach for high dimensional data. in: 2018 ieee congress on evolutionary computation (cec), pp. 1–8. ieee (2018) google scholar  da fonseca lucena, a.h.b.: investigação do uso de computação evolucionária para descoberta de subgrupos em conjuntos de dados numéricos de alta dimensionalidade, master’s thesis, universidade federal de pernambuco (2019) google scholar  mdn web docs. noções básicas de html (2024) google scholar  mdn web docs. o que é css? (2024) google scholar  mdn web docs. o que é javascript? (2024) google scholar  oliveira, l.l.d., gonçalves, a.d.c., costa, j.s.d.d., bonilha, a.l.d.l.: fatores maternos e neonatais relacionados à prematuridade. revista da escola de enfermagem da usp 50, 382–389 (2016) google scholar  oliveira, s.k.m., pereira, m.m., freitas, d.a., caldeira, a.p.: saúde materno-infantil em comunidades quilombolas no norte de minas gerais. cadernos saúde coletiva 22, 307–313 (2014) article  google scholar  w.h.o. low birth weight (2024) google scholar  pereira, p.p.d.s., mata, f.a.f.d., figueiredo, a.c.m.g., silva, r.b., pereira, m.g.: maternal exposure to alcohol and low birthweight: a systematic review and meta-analysis. revista brasileira de ginecologia e obstetrícia 41, 333–347 (2019) article  math  google scholar  pita, r., et al.: on the accuracy and scalability of probabilistic data linkage over the brazilian 114 million cohort. ieee j. biomed. health inform. 22(2), 346–353 (2018) article  math  google scholar  porto, e.c.l., et al.: periodontite materna e baixo peso ao nascer: revisão sistemática e metanálise. ciência saúde coletiva 26, 5383–5392 (2021) article  google scholar  postgresql. postgresql: about (2024) google scholar  render. render: about us (2024) google scholar  romero, c., gonzález, p., ventura, s., del jesús, m.j., herrera, f.: evolutionary algorithms for subgroup discovery in e-learning: a practical application using moodle data. expert syst. appl. 36(2), 1632–1644 (2009) article  google scholar  shimamura, l.k.s., monteiro, d.l.m., da silva, c.r., morgado, f.e.f., de souza, f.m., rodrigues, n.c.p.: gestação tardia: impacto na prematuridade e peso do recém-nascido. rev. assoc. med. bras. 67(11), 1550–1557 (2021) google scholar  silva, a.a.m.d., et al.: the epidemiologic paradox of low birth weight in brazil. revista de saude publica 44, 767–775 (2010) article  math  google scholar  spring. spring: framework (2024) google scholar  vale, c.c.r., almeida, n.k.d.o., almeida, r.m.v.r.d.: association between prenatal care adequacy indexes and low birth weight outcome. revista brasileira de ginecologia e obstetrícia 43, 256–263 (2021) google scholar  vercel. vercel: develop. preview. ship (2024) google scholar  download references author information authors and affiliations universidade de pernambuco, caruaru, pe, brazil josé gustavo cunha & tarcísio daniel pontes lucas universidade federal e pernambuco, caruaru, pe, brazil andreza daniela pontes lucas & monaliza de oliveira ferreira authors josé gustavo cunhaview author publications search author on:pubmed google scholar tarcísio daniel pontes lucasview author publications search author on:pubmed google scholar andreza daniela pontes lucasview author publications search author on:pubmed google scholar monaliza de oliveira ferreiraview author publications search author on:pubmed google scholar corresponding author correspondence to tarcísio daniel pontes lucas . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cunha, j.g., lucas, t.d.p., lucas, a.d.p., ferreira, m.d.o. (2025). low birth weight in brazil vulnerable groups: an analysis based on data mining and big data. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_15 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords low birth weight vulnerable groups subgroup discovery publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ethics of ai: do the face detection models act with prejudice? | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ethics of ai: do the face detection models act with prejudice? conference paper first online: 28 november 2021 pp 89–103 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) ethics of ai: do the face detection models act with prejudice? download book pdf download book epub marcos vinícius ferreira10,11, ariel almeida10, joão paulo canario10, matheus souza12, tatiane nogueira10 & … ricardo rios10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1592 accesses 4 citations abstract this work presents a study on an ethical issue in artificial intelligence related to the presence of racist biases by detecting faces in images. our analyses were performed on a real-world system designed to detect fraud in public transportation in salvador (brazil). our experiments were conducted by taking into account three steps. firstly, we individually analyzed a sample of images and added specific labels related to the users’ gender and race. then, we used well-defined detectors, based on different convolutional neural network architectures, to find faces in the previously labeled images. finally, we used statistical tests to assess whether or not there is some relation between the error rates and such labels. according to our results, we had noticed important biases, thus leading to higher error rates when images were taken from black people. we also noticed errors are more likely in both black men and women. based on our conclusions, we recall the risk of deploying computational software that might affect minority groups that are historically neglected. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an explainable deepfake detection framework on a novel unconstrained dataset article open access 17 january 2023 decoding pictorial collections using faces chapter © 2024 ethical ai in facial expression analysis: racial bias article 09 may 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biometrics computer ethics computer vision machine learning philosophy of artificial intelligence artificial intelligence 1 introduction the face detection approaches have been widely adopted in several applications to create biometric markers in social networks, smart devices, surveillance systems, fraud detection, and so on [15, 20, 25]. in summary, face detection is a subarea of object detection, devoted to finding regions in images that contain faces [22, 25]. the advances in the artificial neural networks (ann) [9], especially with the convolutional neural networks (cnn) [14], have improved the face detection performances and made its usage more usual in many real-world scenarios. nowadays, state-of-the-art manuscripts report relevant performances by finding faces with different positions and rotations, occlusions, expressions, scales, and localization [23, 25]. despite the theoretical and scientific advances, scientists are concerned about the fairness in ai models, which is a current hot topic in computer vision, aiming at better understand the robustness of such ai models across important human features as, for example, age, gender, and race [5]. due to ethical issues, some commercial systems designed by important companies (e.g. ibm, microsoft, and clarifai) were discontinued due to the high error rates in specific groups as black women, possibly presenting a racist bias [7]. recently, in new york, scientists have asked to interrupt the usage of face recognition in some situations due to errors associated with gender, race, and ethnicity [4]. the most important factor related to this issue is that, even in systems with high accuracy rates, small errors may strongly affect the life of minorities who are historically oppressed [2, 15]. an alert was recently triggered when several researchers have emphasized that classification errors are more likely when people of color are under scrutiny [4]. the reason why such errors happen is still being assessed. according to the scientists, it can be driven by the own methods and/or the dataset used during the training process, thus requiring a careful evaluation when deployed in a scenario characterized by different groups  [16]. in our context, the problem came up during the development of an ai-based system to detect fraud in public transportation in salvador (brazil). before the pandemic outbreak, more than 1,1 million passengers used to take buses per day in salvador (average calculated between 2016 and 2019). our system analyzes all images taken when the passengers cross the turnstile by running two steps. firstly, we execute a face detection step in the usage images and, then, we compare the estimated face along with the users’ profiles. the fraud usually happens when users have some benefits like student discounts. in this situation, the students share their discount cards with others. in case of confirmed fraud, the card is blocked. by considering salvador is the brazilian city with the highest percentage of black people (about \(80\%\)), any error may affect a significant number of users, leading to a high number of false positives. in our scenario, due to the absence of images with appropriate labels to describe users by gender and race, we used pre-trained face detectors, published by the original authors. thus, we created an empirical setup to assess whether the detectors have gender and race biases, before deploying our system. aiming at reaching this goal, we firstly created a specialist committee to label our images. then, we analyzed the errors produced by every detector by taking into account different groups of genders and races. as discussed later, we have indeed found a race bias in our environment. this manuscript is organized as follows: sect. 2 shows a set of related work; sect. 3 introduces our methodology and tools considered to conduct our study; the experimental results are presented in sect. 4; finally, concluding remarks and future directions are given in sect. 5, which is followed by a list of references. 2 background this section presents studies related to the investigation of ethical issues caused by gender and race biases that influence the performance of artificial intelligence methods. in this sense, farinella and dugelay (2012) [6] published a manuscript focused on understand whether gender and ethnicity affect each other during the classification process. according to the authors, these features are not affected by themselves. furl, phillips, and o’toole (2002) [7] designed a study to verify the precision of face recognition models by dealing with different ethnicities. the results were focused on caucasians and asians. phillips et al. (2011) [16] have discussed the risks of using faces with different characteristics during the training and test phases. similar work was published by klare et al. (2012) [13], in which the main contribution was extend the analysis to different demographic features as gender, race, and age. according to the authors, black and young women have presented the highest error rates. to overcome this situation, the authors recommend balanced datasets with data sampled from the population where the system will be deployed. the authors also suggest an effort to create public datasets support systems that will be implemented in similar scenario. more recent manuscripts show contributions in the organization of datasets as well as investigation of the existence of a racist bias. in that sense, karkkainen and joo (2021) [12] focused on developing a balanced dataset, named fairface, which has information about seven race groups: white, black, indian, east asia, southeast asia, middle east, and latino. this dataset was compared with others in the literature and used in commercial apis, bringing improvements in the race classification task. buolamwini and gebru (2018) [2] explored three models created to detect faces used in commercial solutions (microsoft, ibm, face++) to evaluate the presence of gender and race prejudice. their most important contribution to the literature was the creation of a dataset, by integrating other public images, balanced by gender and ethnicity. results presented by the authors show higher errors were obtained from the group composed of black women. wang et al. (2019) [21] have evaluated the racist bias in races/ethnicities by developing a dataset and a model capable of reducing racist bias in facial recognition. the dataset created, named racial faces in-the-wild (rfw), has caucasian, indian, asian and african ethnicities. the experiments have shown that the facial recognition of africans has error rates twice as high as the facial recognition of caucasians. raji et al. (2020) [17] have investigated ethics in facial processing technologies developed by amazon, microsoft and clarifai, verifying the accuracy of their systems in classifying people according to race, gender, age, expression, and face detection. all experiments were performed on a dataset created by the authors, which is made up only of celebrities. the main result of such research is the alert about the importance of correctly analyzing ethical issues in the considered models. 3 methodology as previously mentioned, the main objective of this work is to assess whether or not the mostly-adopted face detection methods present similar performance to classify real-world images regardless of the users’ gender and race. the analyzed images were collected by the consortium integra, formed by the four companies responsible for public transportation in salvador (bahia – brazil). aiming at reaching this objective, we defined the following hypothesis: do the pre-trained face detection models act with prejudice? this hypothesis was analyzed by using validation metrics, usually adopted in ml problems, under a null hypothesis (\(h_0\)) that supports the statement there is no difference between metrics when compared users by gender and race. in turn, our alternative hypothesis was used to illustrate when \(h_0\) is rejected, which might call our attention to biases that affect the expected face detection behavior. to better define the scope of our analyses, we have selected images from the integra databases and three well-defined face detectors: i) yolo faces [18]; ii) multitask cascaded convolutional networks (mtcnn) [26]; and iii) resnet [10]. the following sections detail our scope, the experimental setup and the methodology considered to assess our hypothesis. 3.1 dataset the images considered in our experiments were collected from public transportation in salvador. when a passenger crosses the turnstile inside the buses, four pictures are taken and stored in a database. from this database, we have randomly selected 10, 000 images captured on december 10, 2019. this date was a usual workday and was considered without any relevant reason, but the fact of being before the covid-19 pandemic outbreak to avoid analyzing images of users wearing masks, which might affect the performance of the selected face detectors. we also created a tool to support specialists during the tasks of setting the gender and race labels and validating the face detector results, as shown in fig. 1. as one may notice, as a specialist sets the gender and race, the percentage of each label is updated to keep the dataset balanced. consequently, the final set of images contains a similar amount of users according to their gender and race. we emphasize that, although we are aware of the different classes of gender and race adopted by ibge (brazilian institute of geography and statistics), the scope of this study was limited to man/woman and black/white, respectively. the final dataset is composed of instances with the following class rates: i) black – 569 (\(56.50\%\)); ii) white – 438 (\(43.50\%\)); iii) male – 407 (\(40.42\%\)); and iv) female – 600 (\(59.58\%\)). fig. 1. gui interface for target imagens and models predictions. full size image another important feature implemented in this tool is the validation of the face detectors. once pre-trained models are used, we show to the specialists a face and the bounding boxes produced by all detectors, thus selecting their performances based on four possibilities. true positive (tp) happens when the bounding box highlights an actual face. in turn, if a bounding box is drawn on a region without a face, the specialists set it as false positive (fp). a true negative option is used to confirm when no bounding box is drawn on an image without a face. it usually happens when the camera is wrongly positioned or triggered. finally, false negative characterizes situations when the face detectors do not find an existing face. after performing this step, we have a contingency matrix, which is considered to assess our hypothesis. aiming at respecting the users’ privacy and due to the brazilian general personal data protection law (lgpd), the resultant dataset cannot be shared. we emphasize all images were manipulated by authorized employees from integra and the researchers associated with this researcher, respecting the lgpd requirements. if the reviewers deem it is necessary to audit the dataset, the program chairs of the conference can contact us to have access to images, bounding boxes, and labels, always respecting the lgpd requirements. 3.2 face detection models the detection of faces in images is a challenging task in the computer vision area and is considered an instance of object detection in general. recent advances in this area, especially with the advent of convolutional neural networks (cnns), have allowed performing this task with great performance, even in a situation characterized by, for example, face rotation, and occlusion (e.g. hat, cap, and glass). based on the literature, approaches designed to detect face are, usually, less complex than those considered to look for several objects in images. next, we show three models widely used to do so. the first approach considered in our experiments was yolo (you only look once) [18], which is a cnn with 26 convolutional, 4 pooling, and 2 full connected layers. the face detection is essentially performed by a fast regression task, which provides high performance. in summary, yolo analyzes the image as a grid, in which every piece is considered to predict a bounding box and define a confidence level to estimate the presence of objects. the pre-trained model considered in our evaluation was based on the wider face dataset [24]. the second approach is called multi-task cascaded convolutional neural network (mtcnn) [26], which is based on 3 steps. the fist one, referred to as p-net, contains three convolutional layers with 10, 16, and 32 filters and a pooling layer. the second one is called r-net and composed of three convolutional layers with 28, 48, and 64 filters, two pooling layers, and a full-connected layer. finally, the last step, known as o-net, is characterized for presenting four convolutional layers along with 32, 64, 64, and 128 filters. moreover, it has 3 pooling and 1 full-connected layers. p-net is designed to detect candidate faces initially and its results are refined by r-net that removes regions whose presence of faces is highly unlikely. then, o-net detects the faces and provides five reference points. in our scenario, we have considered a pre-trained model that was adjusted on the datasets face detection benchmark [11] and wider face [24]. the third approach used in our experiments was resnet-33 [10], which is based on region proposal network (rpn) to define bounding boxes and reference points. the architecture considered by resnet uses a shared memory strategy, aiming at making the face map available to all convolutional layers. firstly, there is a fast zf-net(zeiler and fergus model) with five convolutional and three full-connected layers. then, there is a vgg-16 architecture [19] with 13 convolutional and 3 full-connected layers. the model used in our analyses was previously trained on the dataset wider face, as well. 3.3 validation metrics aiming at assessing the contingency matrix produced by comparing the results obtained by the face detectors and specialists, we used six validation metrics widely adopted by the ml area: sensitivity, specificity, accuracy, matthews correlation coefficient (mcc), kappa coefficient, and f1-score. sensitivity, defined in eq. 1, calculates the true positive rate between the correctly-detected and missing faces. the true negative rate is measured by the specificity as shown in eq. 2, i.e., the total of images correctly classified as without faces and wrong bounding boxes. accuracy is responsible for calculating the total number of correct classifications, i.e. right bounding boxes and total of images without faces and bounding boxes, as depicted in eq. 3. in contrast, precision counts the total of correctly detected faces among all bounding boxes produced by the detectors, eq. 4. we also used f-score, eq. 5, which calculates a harmonic mean between precision and sensitivity (recall). in our analyses, we have considered precision and sensitivity to have the same weight and are equally important. $$\begin{aligned} \text {sensitivity} = \frac{tp}{tp + fn} \end{aligned}$$ (1) $$\begin{aligned} \text {specificity} = \frac{tn}{fp + tn} \end{aligned}$$ (2) $$\begin{aligned} \text {accuracy} = \frac{tp + tn}{fp + tn + tp + tn} \end{aligned}$$ (3) $$\begin{aligned} \text {precision} = \frac{tp}{fp + tp} \end{aligned}$$ (4) $$\begin{aligned} \text {f1-score} = \frac{2 * ( \text {precision} * \text {sensitivity})}{ \text {precision} + \text {sensitivity}} \end{aligned}$$ (5) the kappa coefficient (\(\kappa \)), eq. 6, is calculated between the observed \(p_o\) classification and the agreement that would be expected to occur by chance \(p_e\), as shown in eq. 7. for all metrics presented so far, the greater the value, the better the classification is. $$\begin{aligned} \text {kappa} = \kappa = (p_o p_e) / (1 p_e) \end{aligned}$$ (6) $$\begin{aligned} p{_{o}}=\sum _{i=1}^{c}p_{ii}, \quad p{_{e}}=\sum _{i=1}^{c}p_{i.}p_{.i} \end{aligned}$$ (7) the last measure used in our experiments is mcc (matthews correlation coefficient), eq. 8, which uses a contingency matrix to compare classifiers similarly to the pearson’s correlation coefficient. the coefficient can be interpreted by the interval \([-1,+1]\), in which \(+1\) represents a perfect mach between expected and predicted labels. when it is equal to 0, learning models are confirmed to provide random predictions. values approaching \(-1\) suggest a total disagreement between expected and predicted labels. $$\begin{aligned} mcc = \frac{tp \times tn fp \times fn}{\sqrt{(tp + fp)(tp + fn)(tn + fp)(tn + fn)}} \end{aligned}$$ (8) 3.4 hypothesis test the validation metrics aforementioned were considered to verify the agreement between the bounding boxes produced by the face detectors and the presence of actual faces. next, we need to verify whether or not the classifier performances vary depending on the users’ gender and race. in this way, we start from the assumption that the data distribution follows a discrete sequence of independent bernoulli random variables with probability p for class 1 and probability \(q = 1 p\) for class 0. thus, the expected value of the i-th bernoulli random variable is \(e(x_i) = p\) and its variance is defined by \(\text {var}(x_i) = p(1-p)\) [8]. in our context, by calculating the probability of success for the gender and race labels, the confidence interval approaches a binomial proportion, which is used to calculate the test statistics. from this perspective, we evaluate whether there are significant differences among outcomes yielded from the combination of different pairs of experiments considering users’ gender and race. in this sense, we used the two proportion z-test to assess the presence of biases affecting the face detection process. the null hypothesis (\(h_0\)) states there is no significant difference between proportions (\(p_1 p_2 = 0\)). on the other hand, the alternate hypothesis (\(h_1\)) states the proportions are not the same, i.e., we reject \(h_0\) with a significance level \(\alpha = 0.05\) in case of \(p_1-p_2>0\). equation 9 calculates the statistic used in our test. $$\begin{aligned} z=\dfrac{p_1-p_2}{\sqrt{\dfrac{p\cdot (1-p)}{n_1} + \dfrac{p\cdot (1-p)}{n_2}}};p=\dfrac{n_1\cdot p_1+n_2\cdot p_2}{n_1+n_2}; \end{aligned}$$ (9) as one may notice, we focused our analysis on the upper-tailed test, in which a z value greater than 1.645 leads to rejecting \(h_0\). by choosing \(\alpha = 0.05\), there is a \(5\%\) probability of getting pairs of value by chance and the results present, indeed, type i error (false positive) [3]. next section, we show the obtained results and our analyses performed to assess our hypothesis. 4 experimental results before proceeding with our statistical analyses, we have to confirm that the detectors are suitable to find faces in images from our database. as previously mentioned, we are processing real-world images collected from passengers while using public transportation in salvador. the analyses of such images are very challenging due to the dynamic environment, i.e., pictures taken inside the buses are affected by cameras with low quality, current weather, road condition, natural movements, and different illumination, thus producing, for example, underexposed, overexposed, and blurry images. firstly, we have run the three detectors on our complete dataset. as a consequence, every detector produced new images with bounding boxes drawn on regions where faces were estimated. figure 1 illustrates three images along with bounding boxes produced by the detectors. in this figure, there is also a button called “no clear face” that is used to remove the image from our analyses. this is especially important when, for example, the images have low quality or the specialists are not completely confident about the most appropriate label. in summary, if a human specialist has any doubt, then the image must be disregarded to avoid propagating a classification error. aiming at verifying the performances of the selected detectors, the human specialists had two main tasks. the first one was to check the bounding boxes, by choosing one of the following options for the contingency matrix: tp (true positive – the face was correctly detected), fn (false negative – no face was found), fp (false positive – a bounding box was drawn on a region with no face), and tn (true negative – no bounding box was drawn on an image with no face). the second task is to set gender and race labels for the users. therefore, the final dataset contains 1, 038 images correctly classified by a specialist committee. figure 2 presents all contingency matrices calculated based on the agreement between the face detectors (rows) and the specialists’ opinion (columns). fig. 2. contingency matrices for all face detectors. full size image by considering the resultant contingency matrices, we calculated the validation metrics (see sect. 3.3) as shown in table 1. according to such results, we notice the best overall result was produced by mtcnn. we emphasize that, as expected, our dataset is imbalanced, presenting more images with faces (positive). for this reason, we have also selected metrics that verify results by chance. concerning the classes based on gender and race, the dataset is balanced as presented in sect. 3.1. once we use pre-trained models published by the original authors, all these images were used as a test fold, whose results are presented in this section. table 1. validation metrics: positive – \(97.27\%\) and negative \(2.73\%\).full size table after demonstrating the general performance of the detectors, we start assessing our hypothesis that, essentially, raises the following question: “are the face detectors influenced by the users’ gender and race?”. to systematically answer this question, we have hypothesized the following assumptions: \(h_0\): the error proportions are the same regardless of the users’ gender and/or race: \(p_1-p_2=0\); \(h_1\): the error proportions are not the same, thus there is difference based on the users’ gender and/or race: \(p_1-p_2 > 0\). the first step towards performing this analysis was to find the error proportions p and the number n of sample per class, as shown in table 2. for example, by considering this table, we notice the detection error in black users is greater when compared to white people. however, can this difference be considered significant? table 2. error proportions (%) between classes.full size table in the next step, by only considering images with faces according to the specialists, we computed \(z_{obs}\), \(\alpha _{obs}\), and the (lower and upper) confidence intervals (ci) from the error proportions between a different combination of gender and races. in the following tables, results with significant differences were highlighted in bold. in table 3, we show the comparison between errors from different combinations of classes, based on the face detection task performed by mtcnn. as one may notice, although there are different error rates in table 2, the statistical tests reject the null hypothesis in three situations: black \(\times \) white, black/woman \(\times \) white/woman, and black/man \(\times \) white/man. therefore, by considering the alternative hypothesis, the error is greater when black users are the analyzed passenger. in this experiment, we have noticed no significant difference when the passengers are compared by gender. table 3. test statistic for mtcnn.full size table next, we repeated the same analyses on images processed by yolo. regardless of the classification metrics, we decided to assess whether the errors produced by this detector were also influenced by gender and race. according to the results summarized in table 4, we noticed the face detection errors have ratified the results produced by mtcnn. as shown in bold, there are significant differences when race is assessed. the face detection with yolo also produced worse results when the passengers are black. table 4. test statistic for yolo.full size table in our last analysis, we have also compared the results produced by resnet. although this detector has presented the worst results, we decided to perform a close examination on the error proportions produced by it according to the different classes. the final results, presented in table 5, also ratifies the previous analyses, in which the error proportions are significant when users are compared by their race. moreover, the errors produced by this detector were statistically different when black woman results are compared to a white man. despite the users’ race, we cannot affirm, at least in our scenario and experiments, there is a significant difference by comparing the users’ gender. table 5. test statistic for resnet.full size table 5 conclusion in this work, we decided to investigate an ethical issue that has been calling the attention of several ai researchers and practitioners. in 2020, a letter signed by several mathematicians had explicitly pointed out the risks of using ai-based technologies without carefully examining whether their decision processes are implicitly working as expected [5]. the mathematics community has not criticized the advances of ai, which are important and necessary to the wellness of our society as the contribution in areas as, for example, medicine and ecology. however, can we work in such direction without propagating to our algorithms (and data) historical errors as structural and explicit racism? this situation is strongly relevant in the computer vision area, especially by working on tasks related to face recognition. are our data and algorithms performing as expected regardless of users’ race? can the algorithm produce more errors when black people are under analysis? as discussed by nick bostrom in the preface of his book on superintelligence with the fable “the unfinished fable of the sparrows”, an important part of the ai advances is performed by researchers interested in studies about the risks of deploying them incautiously [1]. this issue motivated us during the design of a solution to identify fraud in the usage of benefit cards in public transportation in salvador (brazil). aiming at identifying whether or not the cards were being used by the actual owner, we created a fraud detection system that uses artificial neural networks to identify when the users’ pictures taken inside the bus were different from their registered profiles. the experimental setup created in this work was based on the application of models, published to detect faces in images, on our dataset, which is composed of millions of pictures daily taken inside the buses. in summary, we evaluated the errors produced by mtcnn, yolo, and resnet by comparing different groups according to the users’ gender and race. according to our analyses, we have noticed significant differences when the passengers’ race is analyzed. all results consistently highlighted detection errors were greater with black passengers. moreover, we have noticed this problem also happened when we subsampled the data by the users’ gender. in summary, errors with black woman and man were greater than those obtained from white woman and man, respectively. this bias found in our experiments was strongly important to emphasize the need for better models to detect fraud before deploying our system. the main limitation of our work is the usage of pre-trained model, although it is a common practice in commercial applications, especially when the availability of labeled data is limited. finally, we recall the results obtained in this work shows this undesired bias to our unspecific application and illustrates the risks, in which similar applications may suffer in real-world scenarios. thus, as future work, we plan to increase our dataset with more gender and race labels, aiming at retraining the face detectors, and updating their parameters and hyper-parameters. then, we will draw stronger conclusion about whether the problem happens due to the ann architectures, our data used in the test phase, or the requirement of retraining the models with more appropriated (fair) datasets. in the future, we also plan to evaluate other users’ labels. references bostrom, n.: superintelligence: paths, dangers, strategies (2014) google scholar  buolamwini, j., gebru, t.: gender shades: intersectional accuracy disparities in commercial gender classification. in: conference on fairness, accountability and transparency, pp. 77–91. pmlr (2018) google scholar  casella, g., berger, r.l.: statistical inference. cengage learning (2021) google scholar  castelvecchi, d.: is facial recognition too biased to be let loose?, vol. 587. nature, https://doi.org/10.1038/d41586-020-03186-4 (2020). https://www.nature.com/articles/d41586-020-03186-4 castelvecchi, d.: mathematicians urge colleagues to boycott police work in wake of killings. nature 582, 465 (2020). https://doi.org/10.1038/d41586-020-01874-9 article  google scholar  farinella, g., dugelay, j.l.: demographic classification: do gender and ethnicity affect each other? in: 2012 international conference on informatics, electronics & vision (iciev), pp. 383–390. ieee (2012) google scholar  furl, n., phillips, p., o’toole, a.j.: face recognition algorithms and the other-race effect: computational mechanisms for a developmental contact hypothesis. cogn. sci. 26(6), 797–815 (2002). https://doi.org/10.1016/s0364-0213(02)00084-8, https://www.sciencedirect.com/science/article/pii/s0364021302000848 grinstead, c.m., snell, j.l.: introduction to probability. am. math. soc. (2012) google scholar  haykin, s.: neural networks: a comprehensive foundation, 1st edn. prentice hall ptr, upper saddle river (1994) math  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  jain, v., learned-miller, e.: fddb: a benchmark for face detection in unconstrained settings. technical report, umass amherst technical report (2010) google scholar  karkkainen, k., joo, j.: fairface: face attribute dataset for balanced race, gender, and age for bias measurement and mitigation. in: proceedings of the ieee/cvf winter conference on applications of computer vision (wacv), pp. 1548–1558, january 2021 google scholar  klare, b.f., burge, m.j., klontz, j.c., vorder bruegge, r.w., jain, a.k.: face recognition performance: role of demographic information. ieee trans. inf. foren. secur. 7(6), 1789–1801 (2012). https://doi.org/10.1109/tifs.2012.2214212 article  google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2323 (1998) article  google scholar  najibi, a.: racial discrimination in face recognition technology. 24, october 2020. https://sitn.hms.harvard.edu/flash/2020/racial-discrimination-in-face-recognition-technology phillips, p.j., jiang, f., narvekar, a., ayyad, j., o’toole, a.j.: an other-race effect for face recognition algorithms. acm trans. appl. percept. (tap) 8(2), 1–11 (2011) article  google scholar  raji, i.d., gebru, t., mitchell, m., buolamwini, j., lee, j., denton, e.: saving face: investigating the ethical concerns of facial recognition auditing. https://doi.org/10.1145/3375627.3375820 (2020) redmon, j., divvala, s., girshick, r., farhadi, a.: you only look once: unified, real-time object detection. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 779–788 (2016) google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556 (2014) snow, b.j.: amazon’s face recognition falsely matched 28 members of congress with mugshots. 26, july 2018 (june 2018), technology & civil liberties attorney, aclu of northern california google scholar  wang, m., deng, w., hu, j., tao, x., huang, y.: racial faces in the wild: reducing racial bias by information maximization adaptation network. in: proceedings of the ieee/cvf international conference on computer vision, pp. 692–702 (2019) google scholar  yang, m.h., kriegman, d.j., ahuja, n.: detecting faces in images: a survey. ieee trans. patt. anal. mach. intell. 24(1), 34–58 (2002) article  google scholar  yang, s., luo, p., loy, c.c., tang, x.: from facial parts responses to face detection: a deep learning approach. in: proceedings of the ieee international conference on computer vision, pp. 3676–3684 (2015) google scholar  yang, s., luo, p., loy, c.c., tang, x.: wider face: a face detection benchmark. in: ieee conference on computer vision and pattern recognition (cvpr) (2016) google scholar  zhang, c., zhang, z.: a survey of recent advances in face detection. technical report. msr-tr-2010-66, june 2010. https://www.microsoft.com/en-us/research/publication/a-survey-of-recent-advances-in-face-detection/ zhang, k., zhang, z., li, z., qiao, y.: joint face detection and alignment using multitask cascaded convolutional networks. ieee signal process. lett. 23(10), 1499–1503 (2016) article  google scholar  download references acknowledgment this work was partially supported by capes (coordination for the improvement of higher education personnel – brazilian federal government agency). we gratefully acknowledge the support of nvidia corporation with the donation of the titan v gpu used for this research. finally, we also thank the company integra, responsible for the public transportation in salvador, for support this work. any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of capes, nvidia, and integra. author information authors and affiliations federal university of bahia, salvador, brazil marcos vinícius ferreira, ariel almeida, joão paulo canario, tatiane nogueira & ricardo rios neodados, salvador, brazil marcos vinícius ferreira integra association of public transportation companies, salvador, brazil matheus souza authors marcos vinícius ferreiraview author publications search author on:pubmed google scholar ariel almeidaview author publications search author on:pubmed google scholar joão paulo canarioview author publications search author on:pubmed google scholar matheus souzaview author publications search author on:pubmed google scholar tatiane nogueiraview author publications search author on:pubmed google scholar ricardo riosview author publications search author on:pubmed google scholar corresponding author correspondence to marcos vinícius ferreira . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper ferreira, m.v., almeida, a., canario, j.p., souza, m., nogueira, t., rios, r. (2021). ethics of ai: do the face detection models act with prejudice?. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_7 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords face detection racism ethic deep learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating contextualized embeddings for topic modeling in public bidding domain | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating contextualized embeddings for topic modeling in public bidding domain conference paper first online: 12 october 2023 pp 410–426 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) evaluating contextualized embeddings for topic modeling in public bidding domain download book pdf download book epub henrique r. hott9, mariana o. silva9, gabriel p. oliveira9, michele a. brandão9,10, anisio lacerda9 & … gisele pappa9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 644 accesses 6 citations abstract public procurement plays a crucial role in government operations by acquiring goods and services through competitive bidding processes. however, the increasing volume of procurement data has made manual analysis impractical and time-consuming. therefore, text clustering and topic modeling techniques have been widely used to uncover hidden patterns in unstructured text data. this paper leverages the power of bert-based models to overcome the challenges associated with analyzing public procurement data. specifically, we employ bertopic, a topic modeling technique based on bert, to generate clusters that capture the underlying topics in procurement data. additionally, we evaluate several sentence embedding models for representing procurement documents. by combining bert-based models and advanced sentence embeddings, we aim to enhance the accuracy and interpretability of topic modeling in public procurement analysis. our results provide valuable insights into the underlying topics within the data, aiding decision-making processes and improving the efficiency of procurement operations. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others unsupervised topic modeling with bertopic for coarse and fine-grained news classification chapter © 2023 sentence part-enhanced bert with respect to downstream tasks article open access 15 july 2022 the performance of bert as data representation of text clustering article open access 08 february 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization discourse research information model machine learning natural language processing (nlp) net literature 1 introduction public procurement is a vital aspect of government and corporate operations, involving acquiring goods and services through competitive bidding processes. with the increasing volume of procurement data, manual analysis becomes an impractical and time-consuming task. therefore, applying machine learning approaches, such as text clustering and topic modeling algorithms, has become increasingly essential to automate procurement analysis. text clustering and topic modeling techniques have proven effective in identifying hidden structures and patterns in unstructured text data [6, 17]. such methods aim to identify and extract hidden structures and patterns from a large corpus of unstructured text data. topic modeling, in particular, has been widely used to identify prevalent topics in a corpus of documents. by identifying these topics, analysts can quickly identify common themes and trends in the data, which can inform decision-making processes. however, traditional topic modeling algorithms such as latent dirichlet allocation (lda) [1] may not always produce accurate and interpretable results, especially when dealing with noisy and unstructured text data. in the context of public procurement data, characterized by its high technicality and specific jargon, processing and analyzing the data using traditional techniques can be challenging. therefore, there is a need for advanced methods that can effectively capture and represent the underlying topics in such complex data. in this paper, we leverage the power of bert-based models to address the challenges associated with analyzing public procurement data. specifically, we use bertopic [10], a topic modeling framework based on bert pre-trained model [5], to generate clusters that capture the underlying topics in procurement data. additionally, we evaluate several sentence embedding models to represent procurement documents. our main contributions are based on the following research questions (rqs): rq1. what is the comparative performance of different sentence embedding models in representing procurement documents? we conduct an internal evaluation based on topic coherence, topic diversity, and a weighted score (that combines both factors) to compare the performance of four different sentence embedding models and an additional model we trained, called libert-se. we aim to identify the most effective model for representing procurement documents’ complex and specialized language by addressing this research question. rq2. how well does bertopic capture the underlying topics in procurement data? we conduct an external evaluation to compare the bertopic results with the true document class labels, assessing how well the topics identified in each cluster align with the document labels. 2 related work like many other government documents, public procurement documents are frequently stored in portable document format (pdf) and consist of unstructured text [17]. the lack of attention to accessibility, usability, and data quality by governmental portals poses challenges in effectively accessing and extracting valuable information from these documents [13]. consequently, there is a growing need to employ machine learning techniques, specifically text clustering and topic modeling algorithms, to automate the analysis of procurement data. text clustering techniques aim to group similar documents together based on their content, enabling the efficient organization and exploration of large document collections [7]. these methods employ various algorithms, such as k-means, hierarchical clustering, and density-based clustering, to identify patterns and similarities in the text data. by grouping related procurement documents, these techniques may facilitate the discovery of common themes, topics, and trends within the data. on the other hand, topic modeling algorithms provide a means to uncover latent topics or thematic structures within a collection of documents. these algorithms, such as latent dirichlet allocation (lda), probabilistic latent semantic analysis (plsa), and, more recently, bertopic [10], assign topics to documents based on the distribution of words within the corpus. by extracting meaningful topics, topic modeling enables a deeper understanding of the content and allows for more targeted analysis and decision-making. previous research has explored the application of text clustering and topic modeling algorithms to unstructured text data [6, 17]. for instance, souza jr. at al. [19] evaluate different pre-processing methodologies in topic modeling for brazilian portuguese political discussion extracted from twitter and reddit. in particular, the authors applied three document representation models, including two new proposals based on the cluwords model adapted to portuguese. also, silva et al. [16] use topic models for analyzing and visualizing brazilian comments about legislation. specifically, the authors adapted the bertopic topic mining tool to extract topics from political comments. furthermore, advancements in deep learning and language modeling have provided new opportunities to enhance the representation of text data [12]. models like bert (bidirectional encoder representations from transformers) [5] and its variants have shown remarkable performance in various nlp tasks, including document classification and named entity recognition. these models can capture intricate semantic relationships between words and provide context-aware embeddings, thereby improving the quality of text representations. in this paper, we build upon these developments and leverage bert-based models to address the challenges of analyzing public procurement data. by applying bertopic, a topic modeling algorithm based on bert, we aim to generate clusters that capture the underlying topics and themes within procurement documents. while previous studies have focused on topic modeling or document clustering alone, we extend the analysis by considering the performance of different sentence embedding models in capturing the semantics and contextual information of procurement documents. fig. 1. overview of the topic modeling methodology. full size image 3 methodology the methodology followed in this work is summarized in fig. 1 and consists of two main phases: text preprocessing and topic modeling. first, a set of unstructured documents undergoes a sequence of preprocessing steps to transform the text into a more structured format (sect. 3.1). then, we apply bertopic, a topic modeling technique, to create dense clusters and uncover easily interpretable topics in our preprocessed documents (sect. 3.2). 3.1 preprocessing preprocessing plays a fundamental role in nlp tasks, as it generates a more structured representation of the text and reduces the final set of words (i.e., the vocabulary) that will be used as input for learning models. indeed, applying a preprocessing step in a learning pipeline directly impacts the models’ accuracy by reducing noise and improving the input quality. therefore, in this work, we apply a sequence of three distinct preprocessing steps for each document: text normalization, stop word removal, and numeral normalization. considering our focus on brazilian governmental documents, our preprocessing phase is specifically designed to handle texts in the portuguese language. hence, in this sequence, text normalization removes special characters, accents, city names, person names, and words with a sequence of repeated letters and lowercasing. city and person names are identified and eliminated based on an exact match against dictionaries containing over 5,000 brazilian cities and 7,000 common brazilian names. next, stop word removal involves removing stop words, emails, urls, and unit measures. finally, numeral normalization means identifying and removing hours, numbers, and number symbols from the text. 3.2 topic modeling the second phase of our methodology involves identifying the underlying topics within a set of documents. such an approach is especially beneficial in documents that do not present clear structural patterns. in this work, we use bertopic [10], an efficient topic model that leverages clustering techniques and a class-based variation of tf-idf to create consistent topic representations. overall, bertopic consists of four steps: text representation, dimensionality reduction, document clustering, and topic representation. first, a language model converts each preprocessed document to an embedding representation. then, we reduce the dimensionality of these embeddings to avoid problems in the clustering process. the last step from these clusters is extracting topic representations, which is our primary goal. all such steps are detailed next. text representation. mapping sentences into numeric vector spaces is an efficient method to generate richer text representations in nlp tasks [20]. such representations preserve semantic and syntactic information in sentences, leading to better performance in learning models that rely on the vector representation of the text as input. here, we use the sentence-bert (sbert) framework [14] to convert sentences and paragraphs into dense vector representations using pre-trained language models. one of the strengths of sbert is its scalability since it can be executed over large volumes of data, allowing topic analysis in relevant domains that comprises several long documents, such as governmental processes. dimensionality reduction. reducing the dimensionality of the embeddings is an essential step in preparing the data for clustering. high-dimensional data is difficult for clustering algorithms, as they suffer from the curse of dimensionality. when the data is highly dimensional, the concept of spatial location becomes ill-defined, and distance measures differ little. therefore, reducing the dimensionality of the embeddings is crucial to make the clustering process more efficient. umap (uniform manifold approximation and projection for dimension reduction) [11] is a popular dimensionality reduction method that stands out for preserving more local and global characteristics of high-dimensional data in lower projected dimensions. using umap allows for representing the information more condensed using fewer dimensions, reducing the noise caused by highly correlated variables. as a result, this step increases the efficiency of the clustering algorithm since it will have a better capacity to group documents. clustering. after reducing the embeddings (i.e., document representation), the next step is to cluster them semantically. to do so, we use hdbscan [3], a density-based clustering algorithm, due to its superior performance compared to other density-based clustering algorithms, such as dbscan, in terms of both accuracy and efficiency. hdbscan can handle clusters of different shapes and sizes, making it ideal for identifying topics within a set of documents. moreover, it can identify instances that do not belong to any cluster (i.e., outliers), reducing noise in the final topic representation. as a result, the detected topics are more coherent and representative of the underlying structure of the data. topic representation. after generating the document clusters, we apply a traditional tf-idf topic detection method variation to each cluster, called c-tf-idf [10]. such an adaptation considers that documents in a cluster share similar content and treats all documents in a cluster as a single document. therefore, one can calculate the terms’ importance and generate the topics’ distribution in each cluster. the more important words are within a cluster, the more it represents that topic. in other words, if one extracts the most important words per cluster, the result is the topics’ descriptions. 4 experimental design this section describes the experimental design to evaluate the effectiveness of bertopic and sentence embedding models. we first describe lipset, the dataset used in our experiments (sect. 4.1). then, we present the sentence embedding models used to represent procurement documents (sect. 4.2). finally, we detail the evaluation setup and metrics used in this work (sect. 4.3). 4.1 dataset we consider lipset [15], a dataset with labeled public bidding documents from the brazilian state of minas gerais, to evaluate the models. these documents are characterized by their technical language and specific jargon, making them an ideal candidate for evaluating the performance of the proposed clustering approach. the lipset comprises 9083 labeled documents, classified into four metaclasses: minutes, public notice, homologation, and others (includes files belonging to other types of documents, including erratum, annexes, contracts, and descriptive memorials). in the latest version of lipset, the documents were also assigned to classes using a hierarchical approach that consider their metaclasses. in other words, the documents were first labeled to a metaclass and then to one of the classes that belong to this metaclass. for example, minutes can be divided into price registration, waiver, and face-to-face auction. overall, the four metaclasses are divided into 12 classes. the list of classes for all metaclasses and the number of documents for each one are presented in table 1. table 1. list of classes for each metaclass within lipset.full size table table 2. transformer models used as a baseline in this work.full size table 4.2 sentence embedding models in addition to evaluating the effectiveness of bertopic, we also assess the performance of several sentence embedding models for representing procurement documents. specifically, we consider four models as baselines: multilingual universal sentence encoder (use) [21], language-agnostic bert sentence embedding [8] (labse), s-bertimbau [14, 18], and portuguese legislative sentence embedding (legalbertptbr) [16]. table 2 summarizes these four language models. additionally, we train a custom model called libert-se and explore a fine-tuned version using the simcse framework [9], described as follows. libert-se.footnote 1 our model is adjusted in the masked-language modeling (mlm) task for the domain of public procurements, using the bertimbau [18] model checkpoint. following the best practices of domain adaptation outlined in [5], we fine-tuned our model using a dataset of 300,000 official gazette segments sourced from various municipalities in minas gerais, as previously extracted in [4]. these segments encompass articles published in official gazettes, providing brief information about other documents related to the bidding process. to enhance the model’s vocabulary, we employed a tf-idf procedure, which allowed us to extract the most significant terms from the gazette segments. additionally, we manually incorporated domain-specific jargon commonly found in public procurement contexts. to ensure compatibility with the bertimbau model, we divided the segments into blocks of fixed sizes, ensuring that each block adhered to the maximum sentence length of 512 words supported by bertimbau. for training, we followed the same hyperparameter settings employed by bertimbau, except for the batch size, which we reduced to 16 due to computational resource limitations. by adhering to these settings, we aimed to maintain consistency and leverage the existing knowledge and insights gained from the bertimbau model while tailoring it to the specific domain of public procurements. libert-se + simcse. in addition to the initial fine-tuning step using the masked-language modeling (mlm) task, we further optimized our model by performing a new fine-tuning step specifically tailored for the task of generating contextualized sentence embeddings. for this purpose, we use the simcse framework, which is a self-supervised sentence embedding model designed to learn rich sentence representations through contrastive learning. the simcse framework utilizes augmented positive and negative pairs to train the model. augmented versions of the same sentence are considered positive pairs, and the model is trained to maximize their similarity. on the other hand, augmented versions of different sentences serve as negative pairs, and the model aims to minimize their similarity. by employing this contrastive learning approach, simcse effectively learns robust and discriminative sentence representations that capture essential semantic information. to fine-tune our model using the simcse framework, we used a dataset consisting of approximately 300,000 segments of official diaries, extracted and published in a previous work [4]. these segments were distinct from those used in the previous fine-tuning procedure for the mlm task. 4.3 evaluation evaluating clustering and topic modeling results can be challenging due to their unsupervised nature. popular approaches for evaluation involve either “internal” evaluation, where the results are summarized into a single quality score, or “external” evaluation, where the results are compared to an existing “ground truth” [7]. here, we adopt both internal and external evaluation methods to assess the performance of the proposed approach. internal evaluation. we used two primary metrics to evaluate the performance of models in the internal experiments. first, we use the topic coherence based on normalized pointwise mutual information (npmi) [2], which ranges from -1 to 1, where a higher value indicates stronger coherence between words in a topic. we also evaluate the topic diversity, which measures the percentage of unique words across all topics [6]. the topic diversity score ranges from 0 to 1, where 0 indicates redundant topics and 1 indicates more varied topics. additionally, to make a comprehensive evaluation, we combine the coherence and diversity scores by assigning weights to the normalized coherence (nc) and diversity (nd) based on their relative importance to our specific task. the weighted evaluation score is defined as the sum of the weighted coherence (\(wc_s\)) and weighted diversity (\(wd_s\)), where \(wc_s = 0.8 \times nc\) and \(wd_s = 0.2 \times nd\). by considering both coherence and diversity, we can better assess the overall performance of the topic modeling approach and ensure that the generated topics are not only coherent but also diverse. external evaluation. in external evaluation, results are evaluated based on data not used in the topic modeling, such as known class labels. to ensure optimal performance, we fine-tuned the hyperparameters of bertopic using the language model that achieved the best results in the internal evaluation. during the evaluation, we manually compared the topics generated by bertopic with the known class labels, which serve as ground truth for the evaluation. this comparison allowed us to gauge the accuracy and relevance of the generated topics in capturing the underlying themes and categories present in the procurement data. by conducting this manual comparison, we aimed to determine the extent to which the generated topics align with the known class labels, thereby providing insights into the effectiveness of bertopic in capturing and representing the true structure and content of the procurement documents. experimental setup. the internal evaluation metrics were calculated by averaging the results over 10 iterations, where we varied the number of topics from 10 to 50 with a step size of 10. this allowed us to assess the performance of bertopic across a range of topic numbers and identify the optimal number of topics for our analysis. in the external evaluation, we fine-tuned two specific hyperparameters of bertopic using the best-performing language model: nr_topics_list, which includes a list of potential numbers of topics,footnote 2 and min_topic_sizes,footnote 3 which represents the minimum size threshold for each topic. by fine-tuning these hyperparameters, we aimed to further optimize the performance of bertopic in generating meaningful and coherent topics. additionally, for the n_gram_ranges parameter, we set it to (1, 1), indicating that we considered only individual words (unigrams) during the topic modeling process. this choice focused on capturing the most important and distinct keywords within each topic without considering combinations of multiple words. the experiments were performed using 3 gpu models nvidia geforce rtx 3090 ti, nvidia geforce rtx 4090, and nvidia tesla grid v100d-32c. 5 experimental results in this section, we present the results of both internal and external evaluations, which aim to address the two research questions defined in sect. 1. the internal evaluation focuses on assessing the performance and effectiveness of different sentence embedding models in representing procurement documents (sect. 5.1), while the external evaluation examines the quality of the generated clusters using bertopic (sect. 5.2). table 3. comparison of sentence embedding models based on internal evaluation metrics. the best result for each metric is underlined.full size table fig. 2. internal evaluation. (a–c) distribution of internal evaluation metrics, varying across the different models and number of topics. the kruskal-wallis test is applied for the models’ mean comparison. (d) grouped distribution of the weighted score. the paired wilcoxon test is applied for grouped distributions’ mean comparison, referring to the number of topics = 10. full size image 5.1 internal evaluation the internal evaluation results are presented in table 3 and fig. 2, providing a comparative analysis of the performance of the evaluated models. table 3 summarizes the evaluation metrics, including topic coherence, diversity, and the weighted score. the results indicate that our trained model, libert-se, consistently outperforms the other five models across most internal evaluation metrics. our model performs similarly to the fine-tuned simcse version and labse regarding topic diversity, indicating that it can generate diverse and distinct topics. these findings underscore the efficacy of libert-se in representing procurement documents and capturing the underlying topics accurately. figure 2 complements the results’ evaluation, depicting the distribution of topic coherence (a), topic diversity (b), weighted score (c), varying across the different models and number of topics. furthermore, fig. 2(d) shows the grouped distribution of the weighted score, providing a concise overview of the performance of each model across the number of topics range. kruskal-wallis and paired wilcoxon tests were conducted to determine the statistical significance of the observed differences. these tests allow us to verify whether there are significant differences between the models considered, providing additional insights into their relative performance. figures 2(a–c) corroborate the findings presented in table 3, further reinforcing the superior performance of our model (libert-se) compared to the other models across most evaluated metrics. our model consistently achieved higher scores in terms of coherence across the different numbers of topics. however, regarding topic diversity, the performance of our model was comparable to the simcse and labse models. therefore, while our model excelled in topic coherence, it also maintained a high level of topic diversity, which is crucial for generating meaningful and varied topics. this indicates that our model balances coherence and diversity, producing coherent topics while avoiding redundancy. figure 2(d) provides a comprehensive overview of the models’ performance by showing the grouped distribution of the weighted score across the number of topics range. on average, considering ten topics resulted in a high general performance. however, as the number of topics increased, the performance varied across the models and slightly decreased. this finding suggests that the number of topics can impact the performance of the models. while a lower number of topics tends to yield higher overall performance, increasing the number of topics introduces more granularity but may result in a slight decrease in performance. fig. 3. comparative of (a) topic coherence, (b) topic diversity, and (c) the weighted score for each model when considering ten topics. the vertical dashed line represents the median value for each internal evaluation metric. the paired wilcoxon test is applied for the models’ mean comparison, referring to the libert-se model. full size image to conclude the internal evaluation, fig. 3 provides further analysis explicitly focusing on (a) topic coherence, (b) topic diversity, and (c) the weighted score for each model when considering ten topics. by examining these metrics individually, we can assess the strengths and weaknesses of each model in capturing the underlying themes and generating diverse topic representations. consistent with the previous findings, our model outperforms the other five evaluated models, and it consistently achieves higher topic coherence, indicating a more substantial alignment of words within each topic. additionally, our model exhibits competitive performance regarding topic diversity, suggesting that it can generate varied and distinct topics. considering only the weighted score, which combines coherence and diversity, the models that perform well after our model are s-bertimbau and labse. s-bertimbau’s high performance can be attributed to its design specifically for the portuguese language. as public procurement data often contain technical terms, s-bertimbau’s domain-specific embeddings enable it to capture the nuances and intricacies of the text more effectively. labse, on the other hand, is a language-agnostic model that leverages large-scale multilingual training data. its competitive performance can be explained by its ability to handle various languages, including portuguese. finally, our model’s superior performance can be attributed to its ability to leverage pre-training on large-scale datasets and capture the contextual understanding of the text, thereby enhancing the quality of the generated topic clusters. fig. 4. performance comparison based on the weighted score across different variations of nr_topics_list and min_topic_sizes hyperparameters. full size image 5.2 external evaluation to perform the external evaluation, we fine-tuned two specific hyperparameters of bertopic using the best-performing language model, i.e., libert-se. these hyperparameters are nr_topics_list and min_topic_sizes. figure 4 shows the performance based on the weighted score across different variations of both hyperparameters. to determine the statistically significant combination of nr_topics_list and min_topic_sizes, we applied the kruskal-wallis test, allowing us to identify the combination that achieved the best performance: 10 and 60, respectively. figure 5 shows the word clouds of the ten topics identified by bertopic, using the optimal hyperparameters. each word cloud visually represents the most significant keywords associated with a specific topic. the first topic (-1) represents the outlier documents that do not align with any particular theme or topic. these outlier documents may contain diverse or unrelated content that does not fit into the predefined categories. fig. 5. word clouds of the ten topics identified by bertopic. full size image the remaining topics capture distinct themes and content within the procurement documents. these topics range from specific domains or subfields within procurement, such as construction projects, supply chain management, legal regulations, or financial aspects. to compare the topics generated with the ground truth (i.e., metaclasses and class labels), we named each topic based on its top words. table 4 presents the mapping between the topic id and its given name, along with the top words and their frequency within the topic. table 4. mapping between the topic id and its given name.full size table fig. 6. summary of the topic-metaclass associations. full size image using the mapped names in table 4, we generate a heatmap to analyze each document’s occurrence between topics and metaclass labels (fig. 6). this heatmap provides a visual representation of the associations between the identified topics and the metaclass labels assigned to the procurement documents. the intensity of the colors in the heatmap reflects the frequency or occurrence of the topic-metaclass pair within the documents. darker colors indicate a higher frequency, while lighter colors represent a lower frequency or absence of the topic-metaclass pair. overall, the topic-metaclass associations do not perfectly align in the heatmap. while some topics show strong associations with specific metaclasses, indicating a clear thematic correspondence, others exhibit a more scattered or diverse distribution across different metaclasses. this misalignment can be attributed to several factors. firstly, the bertopic is based on the underlying patterns and co-occurrences of words within the documents, which may not always align precisely with the metaclass labels assigned to the documents. secondly, the complexity and diversity of the procurement domain can contribute to the variation and dispersion of topics across different metaclasses. consequently, topics may overlap or span multiple metaclasses, leading to a less direct one-to-one correspondence between topics and metaclasses. additionally, the quality and representativeness of the training data can influence the topic modeling results. if the training data does not fully cover the diversity of procurement documents or lacks sufficient labeled examples for each metaclass, it may affect the accuracy of the topic-metaclass associations. 6 conclusion in this paper, we presented a comprehensive methodology for topic modeling in brazilian procurement documents using the bertopic framework. our approach involved preprocessing steps to enhance the quality of the text and utilized hdbscan for clustering the documents. the resulting clusters were then analyzed using c-tf-idf to extract meaningful topic representations. through evaluating several sentence embedding models, we found that our trained model, libert-se, consistently outperformed the other models in terms of topic coherence. however, regarding the efficiency of bertopic in capturing the underlying topics in the purchase data, there is still room for improvement in aligning the topics with the predefined class labels. this finding highlights the inherent challenge of topic modeling in public procurement data, which is characterized by its complexity and diversity. limitations and future work. while our methodology provides valuable findings, it is important to acknowledge some limitations: the preprocessing steps and embedding models used in this study were specifically tailored for the portuguese language. applying the methodology to other languages may require adaptations and modifications. the focus of this work was on brazilian governmental documents. the methodology’s effectiveness and the models’ performance may vary when applied to different domains or document types. the evaluation and comparison of sentence embedding models were conducted on a specific dataset and task. as future work, we aim to conduct experiments on diverse datasets across various languages and domains to validate the generalizability and robustness of the methodology, models, and evaluation metrics. additionally, we plan to broaden our evaluation by testing a wider range of sentence embedding models, incorporating state-of-the-art models. notes 1.https://huggingface.co/dccmpmgfinalisticas/libert-se. 2.(10, 13, 14, 15, 16, 17, 19, 20, auto). 3.(10, 20, 30, 40, 50, 60, 70, 80, 90, 100). references blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3, 993–1022 (2003) google scholar  bouma, g.: normalized (pointwise) mutual information in collocation extraction. proc. gscl 30, 31–40 (2009) google scholar  campello, r.j.g.b., moulavi, d., sander, j.: density-based clustering based on hierarchical density estimates. in: pei, j., tseng, v.s., cao, l., motoda, h., xu, g. (eds.) pakdd 2013. lncs (lnai), vol. 7819, pp. 160–172. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-37456-2_14 chapter  google scholar  constantino, k., et al.: segmentação e classificação semântica de trechos de diários oficiais usando aprendizado ativo. in: sbbd, pp. 304–316. sbc (2022). https://doi.org/10.5753/sbbd.2022.224656 devlin, j., et al.: bert: pre-training of deep bidirectional transformers for language understanding. in: naacl-hlt, pp. 4171–4186. association for computational linguistics (2019). https://doi.org/10.18653/v1/n19-1423 dieng, a.b., ruiz, f.j.r., blei, d.m.: topic modeling in embedding spaces. trans. assoc. comput. linguistics 8, 439–453 (2020). https://doi.org/10.1162/tacl_a_00325 article  google scholar  feldman, r., sanger, j.: the text mining handbook advanced approaches in analyzing unstructured data. cambridge university press (2007) google scholar  feng, f., et al.: language-agnostic bert sentence embedding. in: proceedings of the 60th annual meeting of the association for computational linguistics (acl), pp. 878–891. association for computational linguistics (2022). https://doi.org/10.18653/v1/2022.acl-long.62 gao, t., yao, x., chen, d.: simcse: simple contrastive learning of sentence embeddings. in: proceedings of the 2021 conference on empirical methods in natural language processing (emnlp), pp. 6894–6910. association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.emnlp-main.552 grootendorst, m.: bertopic: neural topic modeling with a class-based tf-idf procedure. arxiv preprint arxiv:2203.05794 (2022) mcinnes, l., et al.: umap: uniform manifold approximation and projection. j. open source softw. 3(29), 861 (2018). https://doi.org/10.21105/joss.00861 naseem, u., et al.: a comprehensive survey on word representation models: from classical to state-of-the-art word representation language models. acm trans. asian low resour. lang. inf. process. 20(5), 74:1–74:35 (2021). https://doi.org/10.1145/3434237 nikiforova, a., mcbride, k.: open government data portal usability: a user-centred usability analysis of 41 open government data portals. telematics inform. 58, 101539 (2021). https://doi.org/10.1016/j.tele.2020.101539 article  google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: emnlp-ijcnlp, pp. 3980–3990. association for computational linguistics (2019). https://doi.org/10.18653/v1/d19-1410 silva, m., et al.: lipset: um conjunto de dados com documentos rotulados de licitações públicas. in: anais do iv dataset showcase workshop, pp. 13–24. sbc, porto alegre, rs, brasil (2022). https://doi.org/10.5753/dsw.2022.224925 silva, n.f.f., et al.: evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 104–120. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_8 chapter  google scholar  silveira, r., et al.: topic modelling of legal documents via legal-bert. ceur workshop proc. 1613, 0073 (2021) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  souza júnior, a.p., et al.: evaluating topic modeling pre-processing pipelines for portuguese texts. in: webmedia, pp. 191–201. acm (2022) google scholar  turian, j.p., ratinov, l., bengio, y.: word representations: a simple and general method for semi-supervised learning. in: proceedings of the 48th annual meeting of the association for computational linguistics (acl), pp. 384–394. the association for computer linguistics (2010) google scholar  yang, y., et al.: multilingual universal sentence encoder for semantic retrieval. in: proceedings of the 58th annual meeting of the association for computational linguistics: system demonstrations (acl), pp. 87–94. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.acl-demos.12 download references acknowledgments. this work was funded by the prosecution service of the state of minas gerais (ministério público do estado de minas gerais) through the analytical capabilities project (programa de capacidades analíticas) and by cnpq, capes, and fapemig. author information authors and affiliations universidade federal de minas gerais (ufmg), belo horizonte, mg, brazil henrique r. hott, mariana o. silva, gabriel p. oliveira, michele a. brandão, anisio lacerda & gisele pappa instituto federal de minas gerais (ifmg), ribeirão das neves, mg, brazil michele a. brandão authors henrique r. hottview author publications search author on:pubmed google scholar mariana o. silvaview author publications search author on:pubmed google scholar gabriel p. oliveiraview author publications search author on:pubmed google scholar michele a. brandãoview author publications search author on:pubmed google scholar anisio lacerdaview author publications search author on:pubmed google scholar gisele pappaview author publications search author on:pubmed google scholar corresponding author correspondence to gisele pappa . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper hott, h.r., silva, m.o., oliveira, g.p., brandão, m.a., lacerda, a., pappa, g. (2023). evaluating contextualized embeddings for topic modeling in public bidding domain. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_27 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords public bids topic modeling contextualized embeddings publish with us policies and ethics profiles mariana o. silva view author profile gisele pappa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature hybrid artificial intelligence model for detecting signs of delayed child development | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper hybrid artificial intelligence model for detecting signs of delayed child development conference paper first online: 31 january 2025 pp 196–208 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) hybrid artificial intelligence model for detecting signs of delayed child development download book pdf download book epub daniel leal souza  orcid: orcid.org/0000-0002-1345-67239,10, isadora mendes dos santos  orcid: orcid.org/0000-0003-3263-307510,12, caio johnston soares  orcid: orcid.org/0009-0000-5010-01719,10,11, josé pires de oliveira neto  orcid: orcid.org/0009-0008-6400-871010,12, lucas cassiano  orcid: orcid.org/0009-0000-7936-188510,12, marco aurélio proença neto  orcid: orcid.org/0009-0004-8631-31949,10, aline maria pereira cruz ramos  orcid: orcid.org/0000-0001-8812-292313, liliane afonso de oliveira  orcid: orcid.org/0000-0003-4581-995214, flávia luciana guimaraes marçal pantoja de araújo  orcid: orcid.org/0000-0002-2747-086814, fabrício almeida araújo  orcid: orcid.org/0000-0001-8134-596410,12, gilberto nerino de souza junior  orcid: orcid.org/0000-0002-8452-393810,12 & … marcus de barros braga  orcid: orcid.org/0000-0003-1621-446010,12  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 217 accesses abstract child development assessment is a multifaceted process that incorporates variables of diverse origins in order to identify developmental delays. the present study proposes a hybrid artificial intelligence model, combining first-order logic and fuzzy logic to identify delays in child development. the usage of first-order logic facilitates the integration of large volumes of data, promoting a holistic view. the usage of fuzzy logic enables the treatment of uncertainties and a detailed analysis of variables. the results indicate that the proposed model is effective in mapping delays in child development, as well as in using the data obtained to map the child’s evolution trend. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ai-based intellectual prediction in children: an exploration of early cognitive assessment for personalized educational strategies chapter © 2025 moving skills—a contributing factor in developmental delay chapter © 2022 a hybrid model based on fuzzy rules to act on the diagnosed of autism in adults chapter © 2019 1 introduction human development is constant, dynamic and age-related. child development is performed by a complex interaction between prenatal environmental, hereditary factors and postnatal biological, cultural, social and educational factors. periodic assessments of child development indicators are important since birth, with most of them having preventive purposes or for the early diagnosis of disorders that may affect child development. there are standardized indicators on motor, cognitive, language, functioning in daily activities, behavior and communication skills called developmental milestones that characterize the neuropsychomotor and psychosocial development of children in the first years of life [6]. this early childhood development is included on global political agenda as one of the targets in the sustainable development goals (sdgs) to guarantee quality access for all. in brazil, following the global trend, interest in promoting development in early childhood has also grown [8]. the reason for this is that preventing the causes and intervening early makes it possible to modify the effects and improve the general evolutionary results of children, promoting better quality of life and autonomy in daily tasks. both health and education professionals, as well as parents and caregivers of children, must be careful to identify risk factors for child development in advance, and make it possible to treat problems as soon as they are detected [4]. in this context, the present work aims to develop a hybrid artificial intelligence model using first-order logic and fuzzy logic that can track signs of delay in child development in a more automated way. 2 theoretical reference 2.1 development milestones assessment instrument several instruments have already been created for monitoring the child’s development and detecting possible deviations. in the 1920s and 1930s, scales were created and adapted to meet new professional demands. the pan american health organization published the manual for child development surveillance, based on the integrated management of childhood illnesses (imci) methodology, designed to train primary care professionals. its objective was to systematize care, facilitating guidance for parents on their children’s development and early detection of warning signs in development, so that they could be referred for appropriate assessment and treatment. in 1984, the ministry of health (ms) of brazil created the comprehensive child health care program (paisc). as of 2009, the child health handbook began to include the child development component based on the manual for surveillance of child development, in the imci context with some adaptations [3]. the child health booklet is delivered to the parents of all children born in brazil, while still in the maternity ward. it is currently the main instrument for monitoring the child’s health, to be filled out appropriately by the health professionals who accompany the child and carried by parents/guardians in any service. in addition to providing great knowledge of health history, growth and development, the instrument favors the monitoring of healthy evolution and the acquisition of new skills, promoting early interventions and preventing future damage [5] as shown in table 1. documents from the brazilian ministry of health, namely the child health primary care handbook: growth and development (2012), the child health handbook (2014), the early stimulation guidelines: children aged 0 to 3 years with neuropsychomotor delay (2016) and the national policy for comprehensive child health care (2018), present several milestones in child development, as well as possible warning signs about the development that must be achieved by the child in each age group. furthermore, these documents suggest some stimulation that can performed by those responsible as soon as any change in development is identified. it also contains guidelines for referring this child to specialized and multidisciplinary treatment [7]. table 1. developmental milestones at 6 months. the table is filled in according to the markers: p = present milestone; a = milestone missing, nv = milestone not checked.full size table 2.2 artificial intelligence approaches artificial intelligence is an area of computer science that aims to make machines perform intelligent tasks, such as learning and solving problems, in a way similar to the natural intelligence of humans and animals [9]. the growing role of intelligent systems in human society raises pertinent questions about the need to explain the behavior or outcome of these systems, in order to be able to motivate their decision and make the underlying decision process understandable to humans. at this point, subsymbolic artificial intelligence techniques, such as machine and deep learning, despite their efficiency, leave something to be desired. symbolic techniques, based on the formal representation of knowledge and its elaboration through explicit reasoning rules, end up becoming more attractive. this is especially relevant when artificial intelligence is exploited in the context of human organizations intended to provide public services, such as healthcare, diagnostic or counseling systems [1]. among the various approaches already developed, one of the oldest and most consolidated are expert systems, which are based on established rules, generating reasonable advice and suggestions for solving a problem. the use of expert system as a decision support tool is decisive in solving problems that cannot be solved based on analytical calculations [2]. however, expert systems have some disadvantages, such as the lack of ability and flexibility to adapt to changes in the environment, lack of ability to generate a creative response when there is no response, lack of ability to summarize their knowledge through an analogy and the impossibility of learning from experience, not to mention that many expert systems cannot automatically change their knowledge base, nor adjust existing rules or add new ones [11]. one of the ways to mitigate these deficiencies is a combination of expert systems with other artificial intelligence techniques. for example, fuzzy logic can be used to manage uncertainty in expert systems and solve problems that cannot be solved effectively by conventional methods [10]. the main goal of fuzzy expert systems is to use human knowledge to process uncertain and ambiguous data. 3 materials and methods this research is part of a broad study, covering the child’s first seven years. however, for elucidation purposes, a section was carried out, showing only the first semester of the infant’s life. to preserve children’s data and for methodological presentation purposes, data from fictitious children were artificially generated. the proposed model is divided into two main modules, one based on first-order logic, built with the prolog language, using the pytholog tool, and the other based on fuzzy logic, built with the python language, using the scikit-fuzzy framework. the process of classifying the child and their development is conducted by a health professional, father, mother or caregiver. in this study, the first step, which is currently manual, can also be performed by the proposed software, with the child’s correct data being provided in advance. the process occurs according to the following steps: find the child’s age range in the age columns (in months) of the developmental milestones table (similar to table 1); find the four colored lines of the same color corresponding to the developmental milestones of the age group; check the presence of development milestones or skills; fill in the corresponding spaces using the following commands: p = present milestone; a = missing milestone, nv = unverified milestone. this process is performed until the end of the child’s age group. the child is expected to have reached all anticipated milestones. but, if they have not yet reached a milestone in their age group, then an alternative process is performed, following the steps: go to previous age group; check whether the child meets the milestones in the previous range; consult the child classification instrument (table 2); classify the child’s development and adopt the indicated behavior. table 2. child development assessment instrumentfull size table in this work, only development milestones were considered, disregarding phenotypic or physical verifications. this is because the presence of a health professional would be necessary to assess it. therefore, this software does not produce a diagnosis, but only a partial assessment of the child, according to the defined development milestones. all codes can be found at: < https://github.com/projeto-sdia >. 3.1 proposed artificial intelligence approach in this article, a software is proposed as an alternative to the traditional booklet, offering a more efficient and practical approach to monitoring child development. it was based on the 2022 children’s handbook, as a guide for implementing the artificial intelligence model, focusing specifically on development milestones. first order logic modeling. during the process of classifying the child’s record book, the child’s development is checked based on well-defined rules. this fits into inference models consistent with first-order logic that use rules and quantifiers. to enable classification comparable to that of the booklet, using the categories adequate (ad), alert (al) and delayed (at), we continued by performing an empirical generation of fictitious examples of children categorized into three distinct groups: delayed development, development in alert and appropriate development. these examples, as shown below, will serve as a reference to demonstrate that the first part of the artificial intelligence model can help identify possible developmental delays, allowing for early and appropriate intervention. below are the data fields: name: name of the child for whom milestones are being recorded. age in months: age in months at time of assessment. milestone name: milestone being evaluated. assessment: indicates the outcome of the milestone assessment for the child in question. milestone assessment month: refers to the specific month in which the milestone was assessed for the child, which can be assessed as p present and a absent. starting month for milestone assessment: indicates the starting month in which the specific milestone can be evaluated. deadline month for milestone assessment: indicates the cut-off month up to which the specific milestone must be evaluated. after creating the datasets containing examples of fictional children, filtering was conducted to identify only developmental milestones compatible with each child’s age group. subsequently, another filtering step was performed with the objective of selecting exclusively the milestones that are present or absent for each child. the consistency of development milestones in relation to the child’s current and previous age group is also verified. these filtering aim to eliminate any noise or incorrect information that may be present in the data, ensuring that they do not negatively interfere with the inference and application of the rules. processing takes place to generate the knowledge base, with the facts and rules defined to be consulted via logical programming (prolog). prolog has advantages over complex problems through logical rules, facts and queries, which facilitate the clear expression of domain knowledge. furthermore, prolog employs a depth-first search pattern, exploring all possibilities until a solution is found, which is useful for problems with multiple alternatives. the knowledge base is generated iteratively (for each child and child’s milestone) according to the scheme in fig. 1. fig. 1. knowledge base structure. (a) knowledge base model; (b) example knowledge base facts and rules. full size image subsequently, all rules are consulted for each child, generating a checklist: whether each milestone was met or not. if the list elements return true indications, then the suitable classification is obtained. otherwise, a new list is computed only for the missing landmarks. this new listing consists of which missing milestones indicate’alerts’ or’delays’ in the child’s development, as described in the algorithm 1. algorithm 1 list of missing milestone full size image if the list generated by the algorithm has only one element as’delayed’, then the classification is assigned as delayed for a given child, otherwise the child is classified as alert. fuzzy system modeling. subsequent to the execution of the first module, based on first-order logic, the fuzzy machines receive output data referring to the months in which the child reached the milestones of a certain subset, according to the child development framework (i.e. green, pink, blue , yellow). such data are submitted to fuzzification procedures in order to be represented in linguistic terms, through concepts that express the presence and/or absence of the landmark within a time interval (age group). figure 2 shows the relationship between the fuzzified input variables (antecedents), their respective defuzzified outputs (consequent) and the meaning of the score for each membership function. the result of the defuzzification process consists of a score from zero to one hundred, organized into relevance functions that represent the child’s performance linked to a certain set of milestones, shown in fig. 2. at the end of the execution, the score obtained by the fuzzy control system is converted into a linguistic variable that classifies the child’s performance. it is important to highlight that the fuzzy sets of “very high", “high", “low" and “very low" scores refer to the number of late milestones present (low score), as well as milestones reached before the start of the track. age established by the booklet (high score) (see fig. 2). fig. 2. list of antecedents and consequents (sdia’s fuzzy inference machine). full size image regarding the inference rules, the following quantities were produced for each fuzzy machine associated with the development milestone sets: 16 (green); 12 (pink); 14 (blue); 9 (yellow). such quantities are justified by the nature of the problem, which does not require a high number of rules, thus eliminating the risk of redundancy, as well as deterioration of the capacity of the predictive model. some samples of inference rules are presented in algorithm 2. algorithm 2 samples from inference rules applied to sdia’s fuzzy inference machines. full size image after execution, the fuzzy module is capable of generating responses that indicate to the specialist professional the trend and status of child development in a given set of milestones, which allows it to contribute to a better understanding of the performance associated with the child’s evolution. 4 results to collect results, the hybrid artificial intelligence model was used to classify the child and developmental milestones based on data from 12 fictional children. in order to integrate the data from the hybrid model, the fuzzy system will use as input the markers of months in which the child presented a certain milestone, as well as the monitoring signal generated by prolog (i.e. adequate: ad, alert: al, delay: dl). table 3 presents the number of milestones achieved in a period of up to six months, as well as the score generated by the fuzzy system. table 3. number of milestones reached within six months and score generated by the fuzzy systemfull size table the children evaluated in  3 classified according to the score obtained in the score and described in intervals in the universe of discourse (see fig. 2): pink, blue and yellow fuzzy machines: between 0 and 40 (very low and low trapezoidal): presence of late milestones (below or above two milestones, respectively); between 20 and 80 (medium bell function): milestones achieved in the first months (over 50) or in the second half of the age group (under 50); between 60 and 100 (high and very high trapezoidal): presence of milestones reached before the predicted age range (below or above two milestones, respectively); green fuzzy machine: between 0 and 80 (attention and absence of milestones trapezoidal): set of functions that represent the number of milestones not reached by the child; between 80 to 100 (melterms achieved trapezoidal): all milestones have been achieved; after calculating the score, the system will classify the performance on each fuzzy machine based on the score and the intervals described above. at the end of the execution, the specialist will have a general overview of the child’s development, which is described by the code below: green fuzzy machine: a: normal development; b, from 1 to 4: development alert delay in basic milestone detected. level b1 represents one late milestone, while b4 represents all late milestones; blue, pink and yellow fuzzy machine: a, from 1 to 2: normal development. level a1 represents the milestone reached in the first half of the age group, while level a2 represents the milestone reached late; b, from 1 to 2: milestone delay detected. level b1 represents one milestone late, while b2 represents two milestones late; b, from 3 to 4: high number of milestone delays detected. level b3 represents three milestones late, while b2 represents all milestones late; c, from 1 to 4: milestone detected early (before the standard age range). level c1 represents one milestone advanced, while c4 represents all milestones advanced; based on the established parameters, it becomes possible to map the current state of development. when applied to the sampling described in table 3, we have the evolution profile of each child for each set of milestones. table 4 presents the interpretation of the results according to the score obtained by the child in each inference machine. table 4. development profile and interpretation of fuzzy outputsfull size table the results arising from this hybrid structure make it possible to establish a system capable of monitoring and evaluating progress in child development, providing a robust mechanism that addresses the complexity inherent in evaluating the growth and learning of children up to 6 months of age. 5 final considerations in this study, the implementation of an artificial intelligence system is presented that integrates the logical reasoning capabilities of prolog with the flexibility of fuzzy control systems to monitor child development. in this initial version, the development milestones of the first six months were addressed, which demonstrated satisfactory accuracy. however, the need for additional refinements in fuzzy inference rules is evident. the accuracy and relevance of the system are intrinsically linked to the quality of these rules. furthermore, expanding the model to include developmental milestones up to age 6 presents an opportunity to expand the scope and applicability of the system, ensuring its relevance and usefulness to a broader spectrum of child development. for future work, it is suggested to integrate generative ai into the fuzzy system to automatically predict and refine inference rules. this would increase the accuracy of the system by adapting it to emerging trends as the child grows. furthermore, it is recommended to implement a graph model to relate development milestones and improve system analysis. in summary, the initial success of this system in monitoring child development indicates a promising path for future innovations in this field. furthermore, it is also suggested a comparison with other similar systems present in the literature, in addition to the analysis of approaches such as non-monotonic reasoning and refutable reasoning, which can offer greater flexibility for solving the problem and developing new algorithms. references calegari, r., ciatto, g., denti, e., omicini, a.: logic-based technologies for intelligent systems: state of the art and perspectives. information 11(3), 167 (2020). https://doi.org/10.3390/info11030167 article  math  google scholar  dudnyk, o., sokolovska, z.: application of fuzzy expert systems in it project management. in: project management new trends and applications. intechopen (2023). https://doi.org/10.5772/intechopen.102439 de melo figueiras, a.c.: vigilância do desenvolvimento da criança. boletim do instituto de saúde bis 16(1), 77–83 (2015). https://doi.org/10.52753/bis.v16i1.37374 rodrigues, f.f.d.s.: construção de um website educativo sobre o acompanhamento dos marcos do desenvolvimento de crianças de 0 a 2 anos. dissertação (mestrado em saúde da família) – faculdade de farmácia, odontologia e enfermagem, universidade federal do ceará (2023). https://repositorio.ufc.br/handle/riufc/75230 de melo silva, r.c., de almeida, l.a., dos santos, a.m., de oliveira soares, a.c.: avaliação do preenchimento de cadernetas da criança de zero a três anos em um centro municipal de educação infantil. contribuciones a las ciencias sociales 17(1), 554–579 (2024). https://doi.org/10.55905/revconv.17n.1-032 teixeira, m.c.t.v., alckmin-carvalho, f., emeric, d.r., cevallos, p.v., de paula, c.s.: indicadores de atraso no desenvolvimento em crianças de creche advindas de famílias de baixa renda. estudos e pesquisas em psicologia 17, 1042 – 1062 (2017). http://pepsic.bvsalud.org/scielo.php?script=sci_arttext&pid=s1808-42812017000300014 &nrm=iso da silva velho, m.a., cardoso, f.b.: marcos do desenvolvimento motor de crianças de 0 a 3 anos numa visão neuropsicopedagógica: conhecendo o tipico para identificar o atípico. a psicologia em suas diversas áreas de atuação 3, pp. 64–74 (2020) google scholar  venancio, s.i., bortoli, m.c., frias, p.g., giugliani, e.r., alves, c.r., santos, m.o.: development and validation of an instrument for monitoring child development indicators. j. pediatr. 96(6), 778–789 (2020). https://doi.org/10.1016/j.jped.2019.10.008 article  google scholar  xiao, p.: artificial intelligence programming with python. wiley (2022). https://books.google.com/books/about/artificial_intelligence_programming_with.html zadeh, l.: the role of fuzzy logic in the management of uncertainty in expert systems. fuzzy sets syst. 11(1–3), 199–227 (1983). https://doi.org/10.1016/s0165-0114(83)80081-5 article  mathscinet  math  google scholar  fazel zarandi, m.h., sadat asl, a.a., sotudian, s., castillo, o.: a state of the art review of intelligent scheduling. artif. intell. rev. 53(1), 501–593 (2018). https://doi.org/10.1007/s10462-018-9667-6 article  math  google scholar  download references acknowledgement the authors would like to thank the fundação amazônia de amparo a estudos e pesquisa (fapespa) for funding this project. author information authors and affiliations centro universitário do estado do pará (cesupa-argo), av. alcindo cacela 1523, belém, brazil, 66065-219 daniel leal souza, caio johnston soares & marco aurélio proença neto núcleo de pesquisas em computação aplicada (npca), av. presidente vargas, s/n, paragominas, brazil, 68627-451 daniel leal souza, isadora mendes dos santos, caio johnston soares, josé pires de oliveira neto, lucas cassiano, marco aurélio proença neto, fabrício almeida araújo, gilberto nerino de souza junior & marcus de barros braga consulting department, international business machines corporation (ibm), virtual location, belém, brazil caio johnston soares universidade federal rural da amazônia (ufra), campus paragominas, av. presidente vargas, s/n, paragominas, brazil, 68627-451 isadora mendes dos santos, josé pires de oliveira neto, lucas cassiano, fabrício almeida araújo, gilberto nerino de souza junior & marcus de barros braga instituto de ciências da saúde da universidade federal do pará (ics-ufpa), av. tancredo neves 2501, belém, brazil, 66050-160 aline maria pereira cruz ramos instituto ciberespacial da universidade federal rural da amazônia (icibe-ufpa), av. tancredo neves 2501, belém, brazil, 66077-830 liliane afonso de oliveira & flávia luciana guimaraes marçal pantoja de araújo authors daniel leal souzaview author publications search author on:pubmed google scholar isadora mendes dos santosview author publications search author on:pubmed google scholar caio johnston soaresview author publications search author on:pubmed google scholar josé pires de oliveira netoview author publications search author on:pubmed google scholar lucas cassianoview author publications search author on:pubmed google scholar marco aurélio proença netoview author publications search author on:pubmed google scholar aline maria pereira cruz ramosview author publications search author on:pubmed google scholar liliane afonso de oliveiraview author publications search author on:pubmed google scholar flávia luciana guimaraes marçal pantoja de araújoview author publications search author on:pubmed google scholar fabrício almeida araújoview author publications search author on:pubmed google scholar gilberto nerino de souza juniorview author publications search author on:pubmed google scholar marcus de barros bragaview author publications search author on:pubmed google scholar corresponding author correspondence to marcus de barros braga . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper souza, d.l. et al. (2025). hybrid artificial intelligence model for detecting signs of delayed child development. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_14 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords child development hybrid intelligent system artificial intelligence publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluation of convolutional neural networks for covid-19 classification on chest x-rays | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluation of convolutional neural networks for covid-19 classification on chest x-rays conference paper first online: 28 november 2021 pp 121–132 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) evaluation of convolutional neural networks for covid-19 classification on chest x-rays download book pdf download book epub felipe andré zeiser10, cristiano andré da costa10, gabriel de oliveira ramos10, henrique bohn10, ismael santos10 & … rodrigo da rosa righi10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1228 accesses 8 citations abstract early identification of patients with covid-19 is essential to enable adequate treatment and to reduce the burden on the health system. the gold standard for covid-19 detection is the use of rt-pcr tests. however, due to the high demand for tests, these can take days or even weeks in some regions of brazil. thus, an alternative for detecting covid-19 is the analysis of digital chest x-rays (xr). changes due to covid-19 can be detected in xr, even in asymptomatic patients. in this context, models based on deep learning have great potential to be used as support systems for diagnosis or as screening tools. in this paper, we propose the evaluation of convolutional neural networks to identify pneumonia due to covid-19 in xr. the proposed methodology consists of a preprocessing step of the xr, data augmentation, and classification by the convolutional architectures densenet121, inceptionresnetv2, inceptionv3, movilenetv2, resnet50, and vgg16 pre-trained with the imagenet dataset. the obtained results for our methodology demonstrate that the vgg16 architecture presented a superior performance in the classification of xr, with an accuracy of \(85.11\%\), sensitivity of \(85.25\%\), specificity of \(85.16\%\), f1-score of \(85.03\%\), and an auc of 0.9758. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a deep learning-based framework for detecting covid-19 patients using chest x-rays article 22 march 2022 a light-weight convolutional neural network architecture for classification of covid-19 chest x-ray images article 07 january 2022 diagnosis of covid-19 on chest x-ray (cxr) images using cnn with transfer learning and integrated stacking ensemble learning chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer vision covid19 machine learning object recognition x-ray spectroscopy artificial intelligence 1 introduction severe acute respiratory syndrome coronavirus 2 (sars-cov-2) is a new beta-coronavirus first identified in december 2019 in wuhan province, china [3]. since the initial outbreak, the number of patients confirmed with covid-19 has exceeded 178 million in the world. more than 3.85 million people have died as a result of covid-19 (june 19, 2021) [11]. these numbers can be even higher due to asymptomatic cases and flawed tracking policies. in brazil, a study points to seven times more infections than that reported by the authorities [12]. sars-cov-2 shares \(79.6\%\) of the sars-cov base pairs genome [37]. despite its resemblance to the 2002 sars-cov virus, sars-cov-2 has rapidly spread across the globe challenging health systems [28]. the burden on healthcare systems is due to the high rates of contagion and how sars-cov-2 manifests itself in the infected population [22]. according to data from epidemiological analyses, about \(20\%\) to \(30\%\) of patients are affected by a moderate to severe form of the disease. in addition, approximately \(5\%\) to \(12\%\) of infected patients require treatment in the intensive care unit (icu). of those admitted to icus, about \(75\%\) are individuals with pre-existing comorbidities or older adults [8]. in brazil, the first case was officially notified on february 25, 2020 [5]. since then, due to the continental proportions of brazil, several measures to contain and prepare the health system have been carried out in the federative units [24]. brazil went through two delicate moments for the health system, with hospitals without beds and even the lack of supplies for patients in some regions of the country [33]. currently, reverse transcription polymerase chain reaction (rt-pcr) is the test used as the gold standard for the diagnosis of covid-19 [20, 32]. however, due to difficulties in purchasing inputs, increases in the prices of materials and equipment, the lack of laboratories and qualified professionals, and the high demands for rt-pcr tests, the diagnosis can take days or even weeks in some cities in brazil [20]. this delay can directly impact the patient’s prognosis and be associated with a greater spread of sars-cov-2 in brazil [7]. as alternatives to rt-pcr, radiological exams as computed tomography (ct) and digital chest radiography (xr) are being used as valuable tools for the detection and definition of treatment of patients with covid-19 [9]. studies show equivalent sensitivities to the rt-pcr test using ct images [2, 35]. lung alterations can be observed even in asymptomatic patients of covid-19, indicating that the disease can be detected by ct even before the onset of symptoms [19]. xr has lower sensitivity rates compared to ct. however, due to some challenges in ct use, health systems adopted xr in the front line of screening and monitoring of covid-19 [4]. the main challenges in using ct compared to xr are: (i) exam cost, (ii) increased risk for cross-infection, and (iii) more significant exposure to radiation [34]. furthermore, in underdeveloped countries, the infrastructure of health systems generally does not allow rt-pcr tests or the acquisition of ct images for all suspected cases. however, devices for obtaining radiographs are now more widespread and can serve as a fundamental tool in the fight against the epidemic in these countries [9]. in this perspective, artificial intelligence techniques have shown significant results in the processing of large volumes of information in various pathologies and have significant potential in aiding the diagnosis and prognosis of covid-19 [10, 16, 18, 36]. thus, this work aims to explore the application of deep learning (dl) techniques to detect pneumonia caused by covid-19 through xr images. in particular, our objective is to evaluate the performance and provide a set of pre-trained convolutional neural network (cnn) models for use as diagnostic support systems. the cnn architectures evaluated in this article are: densenet121 [15], inceptionresnetv2 [30], inceptionv3 [31], movilenetv2 [27], resnet50v2 [13], and vgg16 [29]. the study is organized into five sections. in sect. 2, we present the most significant related works for the definition of the work. section 3 presents the methodology of the work. section 4 details the results. finally, sect. 5 presents the conclusions of the work. 2 related work artificial intelligence has made significant advances motivated by large data sets and advances in dl techniques. these advances have allowed the development of systems to aid in analyzing medical images with a precision similar to healthcare specialists. furthermore, machine learning or data mining techniques extract relevant features and detect or predict pathologies in the images. therefore, this section describes some relevant works in the literature to detect pneumonia and covid-19 in xr images. since the initial outbreak of covid-19, studies applying cnns have been used to detect covid-19 in xr images. however, at the beginning of the outbreak, the lack of positive xr images was a problem. in [14], the performance of seven cnns using fine-tuning was compared in a set of 50 xr, 25 positive cases, and 25 negative cases for covid-19. however, the few images used, the lack of a test set, and the learning graphs presented indicate that the models could not generalize the problem. in the work proposed by [1], a cnn-based decompose, transfer, and compose (detrac) model is used for covid-19 xr classification. the model consists of using a pre-trained cnn for the imagenet set as a feature extractor. selected features go through principal component analysis (pca) to reduce dimensionality. these selected characteristics are then classified by a cnn into covid-19 or not. the proposed model reached an accuracy of 93.1% and sensitivity of 100% for the validation set. darkcovidnet, a convolutional model based on darknet, is used for the covid-19 detection task in xr [21]. the authors used seventeen darknet convolutional layers, achieving an accuracy of 98.08% for binary classification and 87.02% for multi-class classification. in [17] it is proposed to use the pre-trained cnn xception for the imagenet dataset for the classification of images into normal, bacterial, viral, and covid-19 pneumonia. the study achieved an average accuracy of 87% and an f1-score of 93%. another alternative for detecting covid-19 in xr is detection in levels using vgg-16 [6]. at the first level, xrs are analyzed to detect pneumonia or not. in the second level, the classification of xrs in covid-19 or not is performed. finally, on the third level, the heat maps of the activations for the covid-19 xrs are presented. the accuracy of the study, according to the authors, is 99%. in summary, several recent works have investigated the use of cnns for the covid-19 xr classification. however, most of these works were based on small datasets, performed evaluations directly on the validation sets, and only a few performed the multiclass classification. therefore, these studies lack evidence on their ability to generalize the problem, making it unfeasible to be used as an aid system for the radiologist. thus, our contribution to these gaps is the proposal to evaluate six convolutional models in a dataset with more than five thousand xr. we evaluated models on a set of xrs not used in training and validation. 3 materials and methods an overview of the methodology employed in this work is presented in fig. 1. the methodology is divided into four stages: preprocessing, data augmentation, training, and testing. image preprocessing consists of image resizing and contrast normalization (sect. 3.2). the data augmentation step describes the methods used to generate synthetic images (sect. 3.3). in the training stage, the cnn models and the parameters used are defined (sect. 3.4). finally, in the test step, we performed the performance evaluation of the cnn models (sect. 3.4). fig. 1. diagram of the proposed methodology. full size image 3.1 dataset we use xrs from the curated dataset for covid-19 [26]. the dataset is a combination of chest xr images from different sources. the dataset has xrs for normal lung, viral pneumonia, bacterial pneumonia, or covid-19. the authors performed an analysis of all images to avoid having duplicate images. in addition, for the final selection of images, the authors used a cnn to remove images with noise, such as distortions, cropped, and with annotations [26]. in table 1 we present the number of images used in our work. table 1. public dataset of chest xr used in this article.full size table 3.2 preprocessing in this step, we resize the xrs to \(512\times 512\) pixels. this size limitation is imposed by available computing power. to avoid distortions in the xr, a proportional reduction was applied to each of the dimensions of the images. we added a black border to complement the size for the dimension that was smaller than 512 pixels. as radiographic findings are often low contrast regions, contrast enhancement techniques can be used in xr images. the application of techniques such as contrast-limited adaptive histogram equalization (clahe) to breast and chest radiography images helped in the generalization of convolutional models and an increase in performance metrics [23, 36]. therefore, we use clahe to enhance xr images. clahe subdivides the image into sub-areas using interpolation between the edges. to avoid noise increase, uses a threshold level of gray, redistributing the pixels above that threshold in the image. clahe can be defined by: $$\begin{aligned} p = [p_{\text {max}} p_{\text {min}}]*g(f) + p_{\text {min}} \end{aligned}$$ (1) where p is the pixel’s new gray level value, the values \(p_{\text {max}} \) and \(p_{\text {min}} \) are the pixels with the lowest and highest values low in the neighborhood and g(f) corresponds to the cumulative distribution function [38]. in fig. 2 we present an example with the original and preprocessed image. fig. 2. (a) original xr image with covid-19; (b) preprocessed image. full size image finally, we use stratified k-fold cross-validation as a method for evaluating the models, with k=10. with a 10-fold we train the models 10 times for each model. at the end of the training, we calculated the mean and standard deviation of the results for the defined metrics. 3.3 data augmentation a technique that helps in the convergence and learning of cnn is the use of data augmentation. thus, we use the imagedatagenerator class from keras to perform on-the-fly data augmentation. we apply a horizontal flip, rotations of up to 20\(^\circ \)c, and shear in the training set for each fold. we do not apply transformations on xrs in the validation and test set. 3.4 convolutional architectures we use six cnn architectures for chest xr classification: densenet121 [15], inceptionresnetv2 [30], inceptionv3 [31], movilenetv2 [27], resnet50v2 [13], and vgg16 [29]. we use the pre-trained weights provided by keras for the imagenet dataset for each model. this process of initializing pre-trained weights speeds up the convergence process of the models. table 2 presents the hyper-parameters used for each of the architectures evaluated in this work. table 2. parameters used for each of the cnn architectures.full size table in the training stage, we use categorical cross-entropy as the loss function. the categorical cross-entropy measures the log-likelihood of the probability vector. to optimize the weights of the models, we use the adam algorithm. at the end of each epoch, we used the validation set to measure the model’s accuracy in an independent set and obtain the best training weights. 4 results and discussion in this section, we present and evaluate the results obtained using the proposed convolutional models. 4.1 models training and validation we train the models for 100 epochs for each fold. we evaluate each model at the end of training in the validation set. the choice of the best set of weights was performed automatically based on the error for the validation set. figure 3, presents the confusion matrices for each of the models. fig. 3. confusion matrices of each model for the best test fold set. dark colors represent a greater number of cases. light colors represent a smaller amount of cases. full size image analyzing the confusion matrices (fig. 3), we can see that all classifiers could correctly classify the vast majority of cases. we can highlight the tendency to classify cases of viral pneumonia as bacterial and bacterial pneumonia as viral. this trend may indicate that the number of viral and bacterial pneumonia cases was insufficient for an optimized generalization for these two classes. as for the classification of normal cases, the resnet50v2 model had the highest rate of misclassification. the classification of pneumonia due to covid-19 showed similar success rates. the highest false-negative rate for covid-19 was presented by the inceptionresnetv2 model, with 4 cases. the lowest rate of false negatives was presented by the inceptionv3, resnet-50, and vgg16 models, with 1 cases. from the confusion matrix, we can calculate the performance metrics of the models [25]. table 3 presents the values obtained for the evaluation metrics in the test fold based on the confusion matrices presented in fig. 3. table 3. results for the test fold for each model.full size table analyzing the results, it is clear that there was relative stability in the performance metrics analyzed for each model. for accuracy, the largest standard deviation was ±2.27%, and the largest difference between the models was 3.83% (vgg16 and densenet121). these results indicate an adequate generalization of each model for detecting pneumonia due to covid-19. as for sensitivity, which measures the ability to classify positive classes correctly, the models differ by 3.96%. the maximum variation between models for specificity was 3.83%. in general, for the chest xr classification, the resnet50v2 and vgg16 models showed the best results. this better performance can be associated with the organizations of the models. for resnet50v2, we can highlight the residual blocks that allow an adaptation of the weights to remove filters that were not useful for the final decision [13]. as for vgg16, the performance may indicate that the classification of xr resources from lower levels, such as more basic forms, is better to differentiate viral pneumonia, bacterial pneumonia, covid-19, and normal. however, the vgg16 is computationally heavier and requires more training time. also, vgg16 has a vanishing gradient problem. figure 4 presents the roc curves for each fold and model in the test fold. when comparing the accuracy, sensitivity, specificity, and f1-score metrics, the area under the roc curve (auc) showed the greatest stability, with a variation of only \(\pm 0.80\%\). fig. 4. classification performance for each fold and model in terms of roc curves in the test fold. full size image 4.2 comparison with related work a direct quantitative comparison with related works is difficult. due to private datasets, methodologies with different image cuts, selecting specific aspects of the datasets, removing some images, and with different classification objectives. in this way, we organized in table 4 a quantitative comparison with similar works. table 4. comparison with the related works. acc is the accuracy, sen is the sensitivity, spe is specificity, f1 is the f1-score, and auc is the area under the roc curve.full size table in general, the convolutional architectures evaluated in our work achieved performance similar to the works that perform multiclassification [1, 17, 21]. if compared to works that perform binary classification, the difference for accuracy is ±13%. however, the work with the best accuracy uses a reduced set of positive images for covid-19 (250 images) out of a total of 6523 [6]. thus, the dataset used in the work is not balanced and may indicate a bias in the study. this bias may be associated with a lower sensitivity (87.00%) of the study compared to other metrics. the sensitivity was similar to the other studies, except for the work with 100% sensitivity [1]. however, the size of the dataset used in the study (196 xr images) can influence the results of the method. in addition, the method proposed by [1] is more complex compared to the proposed in our study. in this way, we can highlight that our method use a balanced dataset. we also use stratified k-fold cross-validation for models training. in this way, we can assess the stability of the assessment metrics and the confidence interval. finally, the trained models and source code is available on github [hidden ref.]. 5 conclusion this article compared six convolutional architectures for the detection of pneumonia due to covid-19 in chest xr images. in order to improve the generalizability of the results, we apply a set of preprocessing techniques. we use several models with pre-trained weights for the imagenet dataset, and we propose the classification in normal cases, viral pneumonia, bacterial pneumonia, or covid-19. the main scientific contribution of this study was the performance comparison of the densenet121, inceptionresnetv2, inceptionv3, movilenetv2, resnet50, and vgg16 convolutional architectures for the detection pneumonia due to covid-19. these pre-trained models can serve as a basis for future studies and provide a second opinion to the radiologist during xr analysis. as future work, we intend to analyze the influence of datasets on the characteristics learning of covid-19 xr images, analyzing whether cnns can generalize characteristics for different datasets. furthermore, we hope to investigate explainable artificial intelligence approaches to convey specialists the features present in the images used to form the diagnostic suggestion. finally, using multimodal methodologies, for example, using clinical data and images, can be helpful in the transparency of diagnostic recommendations. references abbas, a., abdelsamea, m.m., gaber, m.m.: classification of covid-19 in chest x-ray images using detrac deep convolutional neural network. appl. intell. 51(2), 854–864 (2021) article  google scholar  ai, t., et al.: correlation of chest ct and rt-pcr testing for coronavirus disease 2019 (covid-19) in china: a report of 1014 cases. radiology 296(2), e32–e40 (2020). https://doi.org/10.1148/radiol.2020200642 article  google scholar  andersen, k.g., rambaut, a., lipkin, w.i., holmes, e.c., garry, r.f.: the proximal origin of sars-cov-2. nat. med. 26(4), 450–452 (2020) article  google scholar  borghesi, a., maroldi, r.: covid-19 outbreak in italy: experimental chest x-ray scoring system for quantifying and monitoring disease progression. la radiologia medica 125(5), 509–513 (2020) article  google scholar  brasil, m.d.s.: boletim epidemiológico especial: doença pelo coronavírus covid-19 (2020). https://saude.gov.br/images/pdf/2020/july/22/boletim-epidemiologico-covid-23-final.pdf brunese, l., mercaldo, f., reginelli, a., santone, a.: explainable deep learning for pulmonary disease and coronavirus covid-19 detection from x-rays. comput. methods program. biomed. 196, 105608 (2020) article  google scholar  candido, d.s., et al.: evolution and epidemic spread of sars-cov-2 in brazil. science 369(6508), 1255–1260 (2020) google scholar  cdc covid-19 response team: severe outcomes among patients with coronavirus disease 2019 (covid-19)-united states, february 12-march 16, 2020. mmwr morb. mortal wkly. rep 69(12), 343–346 (2020) google scholar  cohen, j.p., et al.: covid-19 image data collection: prospective predictions are the future (2020) google scholar  de fauw, j., et al.: clinically applicable deep learning for diagnosis and referral in retinal disease. nat. med. 24(9), 1342–1350 (2018) article  google scholar  dong, e., du, h., gardner, l.: an interactive web-based dashboard to track covid-19 in real time. lancet. infect. dis 20(5), 533–534 (2020) article  google scholar  epicovid: covid-19 no brasil: várias epidemias num só país (2020). http://epidemio-ufpel.org.br/uploads/downloads/276e0cffc2783c68f57b70920fd2acfb.pdf he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  hemdan, e.e.d., shouman, m.a., karar, m.e.: covidx-net: a framework of deep learning classifiers to diagnose covid-19 in x-ray images. arxiv preprint arxiv:2003.11055 (2020) huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4700–4708 (2017) google scholar  jatobá, a., lima, l., amorim, l., oliveira, m.: cnn hyperparameter optimization for pulmonary nodule classification. in: anais do xx simpósio brasileiro de computação aplicada à saúde, pp. 25–36. sbc, porto alegre, rs, brasil (2020). https://doi.org/10.5753/sbcas.2020.11499, https://sol.sbc.org.br/index.php/sbcas/article/view/11499 khan, a.i., shah, j.l., bhat, m.m.: coronet: a deep neural network for detection and diagnosis of covid-19 from chest x-ray images. comput. methods program. biomed. 196, 105581 (2020) article  google scholar  lakhani, p., sundaram, b.: deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. radiology 284(2), 574–582 (2017) article  google scholar  lee, e.y., ng, m.y., khong, p.l.: covid-19 pneumonia: what has ct taught us? lancet. infect. dis 20(4), 384–385 (2020) article  google scholar  marson, f.a.l.: covid-19 6 million cases worldwide and an overview of the diagnosis in brazil: a tragedy to be announced. diagn. microbiol. infect. dis. 98(2), 115113 (2020) article  google scholar  ozturk, t., talo, m., yildirim, e.a., baloglu, u.b., yildirim, o., rajendra acharya, u.: automated detection of covid-19 cases using deep neural networks with x-ray images. comput. biol. med. 121, 103792 (2020) article  google scholar  pascarella, g., et al.: covid-19 diagnosis and management: a comprehensive review. j. internal med. 288(2), 192–206 (2020) article  mathscinet  google scholar  pooch, e.h.p., alva, t.a.p., becker, c.d.l.: a deep learning approach for pulmonary lesion identification in chest radiographs. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 197–211. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_14 chapter  google scholar  rafael, r.d.m.r., neto, m., de carvalho, m.m.b., david, h.m.s.l., acioli, s., de araujo faria, m.g.: epidemiologia, políticas públicas e pandemia de covid-19: o que esperar no brasil?[epidemiology, public policies and covid-19 pandemics in brazil: what can we expect?][epidemiologia, políticas públicas y la pandémia de covid-19 en brasil: que podemos esperar?]. revista enfermagem uerj 28, 49570 (2020) google scholar  ruuska, s., hämäläinen, w., kajava, s., mughal, m., matilainen, p., mononen, j.: evaluation of the confusion matrix method in the validation of an automated system for measuring feeding behaviour of cattle. behav. process. 148, 56–62 (2018). https://doi.org/10.1016/j.beproc.2018.01.004 article  google scholar  sait, u., et al.: curated dataset for covid-19 posterior-anterior chest radiography images (x-rays). (september 2020). https://doi.org/10.17632/9xkhgts2s6.3 sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv 2: inverted residuals and linear bottlenecks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4510–4520 (2018) google scholar  shereen, m.a., khan, s., kazmi, a., bashir, n., siddique, r.: covid-19 infection: origin, transmission, and characteristics of human coronaviruses. j. adv. res. 24, 91–98 (2020) article  google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556 (2014) szegedy, c., ioffe, s., vanhoucke, v., alemi, a.: inception-v4, inception-resnet and the impact of residual connections on learning. in: proceedings of the aaai conference on artificial intelligence (2017) google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  tang, y.w., schmitz, j.e., persing, d.h., stratton, c.w.: laboratory diagnosis of covid-19: current issues and challenges. j. clin. microbiol. 58(6) (2020). https://doi.org/10.1128/jcm.00512-20, https://jcm.asm.org/content/58/6/e00512-20 taylor, l.: covid-19: is manaus the final nail in the coffin for natural herd immunity? bmj 372 (2021) google scholar  wong, h.y.f., et al.: frequency and distribution of chest radiographic findings in patients positive for covid-19. radiology 296(2), e72–e78 (2020) article  google scholar  xie, x., et al.: chest ct for typical coronavirus disease 2019 (covid-19) pneumonia: relationship to negative rt-pcr testing. radiology 296(2), e41–e45 (2020) article  google scholar  zeiser, f.a., et al.: segmentation of masses on mammograms using data augmentation and deep learning. j. digital imaging 1–11 (2020) google scholar  zhou, p., et al.: a pneumonia outbreak associated with a new coronavirus of probable bat origin. nature 579(7798), 270–273 (2020) article  google scholar  zuiderveld, k.: graphics gems iv. in: heckbert, p.s. (ed.) graphics gems, chap. contrast limited adaptive histogram equalization, pp. 474–485. academic press professional inc, san diego, ca, usa (1994) google scholar  download references acknowledgements we thank the anonymous reviewers for their valuable suggestions. the authors would like to thank the coordination for the improvement of higher education personnel capes (financial code 001), the national council for scientific and technological development cnpq (grant numbers 309537/2020-7), the research support foundation of the state of rio grande do sul fapergs (grant numbers 08/2020 ppsus 21/2551-0000118-6), and nvidia gpu grant program for your support in this work. author information authors and affiliations graduate program in applied computing, universidade do vale do rio dos sinos, são leopoldo, brazil felipe andré zeiser, cristiano andré da costa, gabriel de oliveira ramos, henrique bohn, ismael santos & rodrigo da rosa righi authors felipe andré zeiserview author publications search author on:pubmed google scholar cristiano andré da costaview author publications search author on:pubmed google scholar gabriel de oliveira ramosview author publications search author on:pubmed google scholar henrique bohnview author publications search author on:pubmed google scholar ismael santosview author publications search author on:pubmed google scholar rodrigo da rosa righiview author publications search author on:pubmed google scholar corresponding author correspondence to cristiano andré da costa . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper zeiser, f.a., costa, c.a.d., ramos, g.d.o., bohn, h., santos, i., righi, r.d.r. (2021). evaluation of convolutional neural networks for covid-19 classification on chest x-rays. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_9 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords covid-19 chest x-rays deep learning convolutional neural network publish with us policies and ethics profiles cristiano andré da costa view author profile gabriel de oliveira ramos view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature explainability of machine learning models with xgboost and shap values in the context of coping with disasters | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper explainability of machine learning models with xgboost and shap values in the context of coping with disasters conference paper first online: 31 january 2025 pp 152–166 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) explainability of machine learning models with xgboost and shap values in the context of coping with disasters download book pdf download book epub lucas teixeira9, augusto matos9, gabriel carvalho10, norma valencio10 & … heloisa camargo9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15415)) included in the following conference series: brazilian conference on intelligent systems 280 accesses 1 citation abstract a considerable part of brazilian municipalities experiences the recurrence of emergencies, related to different types of disasters, which weakens them – both the public administration and the citizens – while they are equally oriented to operate transitions to achieve the 2030 sustainable development goals (sdg) agenda. to mitigate the crises addressed by the 2030 agenda, intersectoral consultation guided by an integrated approach to social justice, accountability and sustainability is required to renew social dynamics and corresponding local public policies. the objective of this paper is to carry out an integrated analysis of such dynamics in their main social and economic components, modeling the indicators of some specific sgd and emergence decrees data as input variables to a classification problem and using explainable machine learning, namely shap values technique, to assess their intertwining and/or synergies. three classification models were tested on data from 2003 to 2020 and the model which presented the best accuracy was analyzed with shap values, revealing that different variables are decisive for each phenomenon associated with the disaster and their understanding allows elucidating critical points to be addressed by managers. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others machine learning based approach for sustainable social protection policies in developing societies article 07 january 2021 big data management for policy support in sustainable development chapter © 2022 inequality, water accessibility, and health impacts in chiapas, mexico article 07 december 2022 1 introduction in the first decades of this century, several nations on the planet – including brazil – have been facing a rapid materialization of events threatening the well-being of their population. many of these events, although classified as natural, are from different groups and subgroups, such as belonging to the geological (mass movement), hydrological (floods, torrents), meteorological and climatological (storms, droughts, dry season) and biological (viral infectious diseases) categories [1]. the greater the process of socioeconomic and environmental vulnerability to which local populations may be subject, the greater their fragility if exposed to such threatening factors, whether they are located in poorer or more affluent macro-regions of the country [2]. one aspect of this vulnerability is the asymmetry in the distribution of opportunities for decent work, income, education and safe housing; in other words, it concerns the structural challenges underlying the development model that the country has been following. another aspect to consider is the relative institutional unpreparedness to protect these vulnerable populations. systematically, there has been no success in improving preventive and preparedness civil protection actions in the country. this results in disasters, defined as a significant set of collective losses and damages that cause significant stress and social suffering [3,4,5], negatively affecting local socio-spatial dynamics, from wealth flows to difficulties in taking public action to recover damaged or destroyed infrastructure. in the face of disasters, there is an institutional need for municipal public administration to declare an emergency. when the local public authority declares an emergency associated with a given manifested event, this means that the damage caused by it exceeds the capacity of local actors to respond to the situation. emergency decrees associate the temporality of the event’s manifestation (generally short) with the bureaucratic temporality of the emergency (of medium duration) and point to the disasters themselves (an even longer social time of the disturbed daily life of the affected groups) [6]. emergency decrees duration allows the local administration to take or induce measures to repair the situation more strategically and urgently, enabling certain social and economic routines, territorial flows and infrastructures to be resumed, even if dealing with a certain precariousness for a longer period of time so that a full recovery can be envisaged. more than undermining well-being conditions in the affected place (city, state or country), disasters can also harm the achievement of the sustainable development goals by 2030. the sdgs are a multilateral agreement to incisively combat the main planetary socioeconomic and socio-environmental problems. this includes commitments to stop precarious work, expand access to basic sanitation, protect tropical forests and eliminate world hunger. in the sdg progress report [7], the united nations recognizes that more than half the world is being left behind, with disasters related to climate change, pandemics and loss of biodiversity as well as their adverse economic and social consequences on developing countries where there aren’t enough resources to reverse the situation. the use of explainable ai (xai) methods [8] with machine learning (ml) algorithms has become increasingly common over the last few years, even being considered by researchers as a mandatory step in critical domains, where it is expected that the model not only provides an answer but, also explains how and why the conclusion was reached. among the methods currently available, shapley additive explanations (shap) [9] stands out as one of the most used, for presenting several advantages such as ease of use, being model agnostic and allowing the presentation of analyzes through graphs that are both easy to interpret and informative. ensemble machine learning models, specifically xgboost [11] and random forest (rf) [10], associated to shap technique, have been successfully applied in a broad variety of fields including health [12], finances [13], urban environment [14], among many others. in the specific domain of threatening factors associated to disasters monitoring and assessment, the last few years witnessed a significant increase in the number of works that explore the combined ensemble model-shap approaches. in [15], the authors adopted an integrated approach using statistical modeling and machine learning techniques to assess the effect of urban expansion and deforestation on temperature rise in cajazeiras, brazil. the databases were created with time series representing the evolution of each biome and temperature from 1990 to 2020. rf with shap values unraveled the importance of urbanization and the absence of savanna formation near the city. a landslide susceptibility evaluation model was constructed in [16]. using the historical landslide information from three counties in chongqing, china as the base data, a model was constructed based on the xgboost algorithm and the shap algorithm was applied to quantify the contribution of the influencing factors on landslide occurrence at both global and local levels. the study in [17] proposes a wildfire severity mapping approach that incorporates feature selection techniques within the xai framework, aiming at enhancing the precision and providing insights into the factors contributing to model decisions. utilizing post-fire imagery, the authors developed a rf model and identified the most influential predictors to elucidate the qualitative and quantitative impacts on ml algorithm performance. in [18], an explainable ml pipeline using the xgboost model and shap was proposed, based on a comprehensive database of drought impacts in the u.s., both at national and state levels. data from 2011 to 2020 were obtained from online databases and processed to represent the occurrence of various drought indicators and drought impacts. the authors concluded that the patterns observed by the shap explainer were aligned with expert knowledge, revealing that the abnormal dryness positively contributes to the occurrence of drought impacts. the tropical cyclone disaster loss was evaluated in [19] using ml algorithms and identifying the impact of specific features on the prediction with shap values. the indicators of hazard, vulnerability, and resilience were incorporated into the system as input variables in a dataset of 492 disaster events caused by tropical cyclones that occurred from 2000 to 2020 in different provinces in china. the study presented in [20], uses a xgboost model and shap values to assess how urban morphology influences urban flooding susceptibility. the city of shenzhen in china, was chosen as a case study due to its rapid urbanization that led to increased residential population and ecosystem fragmentation. the findings underscore the varying impact of disaster variables on urban flooding, with morphological attributes becoming highly significant during severe inundations. in the aforementioned studies, the necessity of explaining the model outputs were highlighted and the benefits of ml models and shap values as a valuable xai tool for decision-makers when managing disasters were emphasized as the main contribution of the researches. although relevant, the majority of related work address specific problems. in this sense, our work brings a novel and broader approach that provides insights to reduce the lack of studies in brazil, and addresses the problem from a comprehensive level. based on the brazilian context, this study is a preliminary and original exercise on the connections between disasters associated with different types of events (geological, hydrological, meteorological, climatological and biological) and which have been recognized by local and federal authorities as emergencies and dozens of socioeconomic variables available in official open databases, covering the period 2003-2020. the main objective of the study is to identify, by means of ml models and shap values, which socioeconomic and socioenvironmental variables are relevant in their association with the different types of events that led to the declaration of emergency in brazil in the period 2003-2020. once identified, preliminary interpretations were made about this connection, to be tested in future studies. secondarily, the aim was to establish the link between the variables highlighted in each type of event (class) causing disaster and the sdgs, indicating the bottlenecks that must be addressed to comply with them. 2 materials and methods 2.1 dataset the dataset includes 44,285 examples created from emergency decrees in brazilian municipalities from 2003 to 2020, each with 55 features, such that 11 of them, extracted from s2idfootnote 1, are related to the decrees. the other 44 external features are categorized as follows: housing data, hdi indicators and occupational sector data, form the ibge censusfootnote 2; the share of value added in relation to the national gdp, from ibge’s sidra plataformfootnote 3; education level investment data, from inepfootnote 4. all external features are provided annually at the estate level, except for data from inep, which are provided at the national level. table 1 shows the variables, their corresponding abbreviated names used in the graphs and sources. besides the complete dataset, two subsets were created: one with data from 2003 to 2011, comprising 13,407 examples, and the other with data from 2012 to 2020, comprising 30,878 examples. this separation was considered to allow the emergence of different variables among the most important ones in different periods, due to the changes that occurred in 2012, such as the new classification of disasters [1] and the standardization of validity periods for emergency situations (90 days) and states of public calamity (180 days). table 1. selected variables.full size table 2.2 machine learning algorithms and shap values our problem was modeled as a classification task with the cobrade category of events [1] as the class, aiming at unraveling insights on the relationships among these events and several socio-economic indicators. we employed tree-based models such as xgboost, random forest, and decision tree [22] for our classification task due to their effectiveness in handling complex relationships within data and their ability to capture nonlinearities and interactions among features. evaluation of model performance included metrics such as accuracy, precision, recall, and f1 score. to enhance interpretability, we utilized the shap libraryfootnote 5 to compute average variable importance scores for each class separately. these insights were crucial for evaluation by a disaster specialist, considering both geographical and socio-political aspects. 3 experiments and analysis the algorithms were run on the entire dataset, spanning from 2003 to 2020, and on two separate datasets, covering the periods: period 1 (2003-2011) and period 2 (2012-2020). in all executions we employed an 80-20 split for this study, allocating 80% for training and 20% for testing and using the default parameters. throughout the process, our focus was on ensuring adequate class representation to allow the model to generalize common behaviors across event occurrences. we acknowledge the inherent imbalance in our dataset, however, introducing synthetic data was deemed inappropriate as it could distort variable importance and exacerbate biases. the performance measures for all algorithms and time periods are shown in table 2. the model with best accuracy (xgboost) was selected to perform the analyses of features importance by the shap method. table 2. model performance metrics across time periodsfull size table as not all classes have enough examples, only the classes with the highest frequency in the data set were considered for analysis (cobrade classes in parenthesis): landslide (1.1.3.2.1), storms and floods (1.2.1.0.0), dry season (1.4.1.1.0), drought (1.4.1.2.0), forest fires (1.4.1.3.1) and viral infectious diseases (1.5.1.1.0) (see fig. 1). the shap beeswarm plot is used to present the 11 most important variables for each class in the period chosen for analysis: the entire period (2003-2020) or separate periods (2002-2011 or 2012-2020), depending on the relevance identified in each case. the beeswarm plot gives the rank of features by their importance, and provides information on how the variation of the features values can impact each class. for each feature, all instances in the dataset are identified by a dot, with the horizontal position of the dot being the shap value of that feature. the blue color represents low values and, red, the high ones. it can be noted that date related features such as year or month of beginning and end of the decree validity, as well as ibge code, appear frequently among the most important ones. although important for the class determination, it is not possible to extract specific information based on the feature values, except for some cases where the decrees validity start in specific years, for example viral infectious diseases (1.5.1.1.0) that occurred mainly during 2020 covid-19 pandemic period as seen in fig. 1. for this reason, we do not comment on these variables in the analysis that follows. it is important to mention that ibge code is included as one of the attributes as an identifier of the municipalities. fig. 1. frequency of cobrade classes over the years, 2003-2020. full size image below, we comment on the variables that appear among the most important and provide relevant information for the objectives of the article, for each of the selected classes. fig. 2. most important variables for class landslide (1.1.3.2.1) full size image landslide. a landslide is defined as the movement of a mass on a slope due to the action of gravity and the effect of some force [21]. in the brazilian case, intense rainfall over these areas acts to produce the movement of soil sliding (in the form of mud) or falling rocks and debris (including natural or planted flora) causing considerable destruction to the form of human occupation in the path of their flow. because of this, it can directly affect the share of agriculture in the municipal/state contribution when this productive activity is carried out on this type of land. this explanation shapes both the low values of the agriculture share (3rd, fig. 2a) and the high values of the industry share (1st, fig. 2b). the importance of monetary value in high administration, observed more markedly in the second period, may be due to the significant volumes of extraordinary monetary resources accessed by the municipal executive power after the emergency decree when this event reaches the fullness of its local destructive manifestation. this is because the damage caused, especially on trafficable roads and in inhabited areas, requires expeditious and exceptional technical intervention – carried out by human staff and current public equipment, however, combined with that of private companies hired on an emergency basis – to unblock and repair the roads. reached as well as containing unstable slopes. it is surprising, however, that in the second period (2012-2020) hdi income and longevity, both at high values, emerge and become significant in the model. this may indicate that the topographic and geomorphological characteristics of certain municipalities belonging to wealthy regions make the location more susceptible to this event; however, even though the regional socioeconomic characteristics, of a more general nature, are prosperous, the most susceptible terrain in the affected locations may be those in the territories of the most socioeconomically disaffiliated communities, that is, prosperous regions with severe asymmetries in terms of socio-spatial susceptibility. storms and floods. these events, characterized by climatological/atmospheric and hydrological origins, are important in both periods (2003-2011 and 2012-2020), although they occur at different times in the five brazilian macro-regions. since this class of events has relative predictability regarding the period of the year the months in which it will occur, if the federative unit (uf) and the municipalities are (or remain) unprepared to dealing with this type of events, disaster become inevitable. the regularity of manifestation of these threats throughout the year is what would allow the local public administration to achieve the sdgs 11 (particularly, referring to sub-goals 11.5 and 11.7.b) and 13 (sub-goal 13.1), if it acted in non-seasonal periods. rainy seasons for planning and executing preventive and preparatory measures to face them in the next time and avoid a new disaster. the low and medium number of vulnerable people dependent on the elderly (1st and 6th, fig. 3) denotes that the locations currently susceptible to disasters associated with this class of events are socioeconomically prosperous; therefore, the chances are greater that the local administration will have its own human, technological and financial resources to adopt an agenda in accordance with the sub-objectives of sustainable development mentioned above. fig. 3. most important variables for class storms and floods (1.2.1.0.0), 2003-2020. full size image dry season. medium and high values for single mother heading households and high percentage of vulnerable people dependent on the elderly (1st and 4th, fig. 4) indicate that the predominant social and economic characteristics associated with the dry season are structural poverty. this is confirmed by the importance, also demonstrated in the model, of the low values in both variables, added value of agriculture and added value of industry ((3rd and 5th, fig. 4). thus, it is clear how far the localities experiencing disasters related to this class of event are from sdg 11 (sub goal 11.5: protect the poor and people in vulnerable situations). it is important to highlight the unusual aspect of the variable high investment value in basic education (10th, fig. 4) associated with disasters caused by dry season, in an apparent contradiction with the above interpretation. it turns out that, in less economically dynamic locations, the amounts transferred from the federal level to the municipalities as well as mandatory spending on education (25% of revenue) stands out in front of the dispersion of other expense items with which the local public administration must deal with. thus, the importance of high investments in education in the model signals a convergent direction with sdg 4, especially with regard to sub-goal 4.1 (ensure that all girls and boys complete primary and secondary education [.. .] that leads to relevant and effective learning outcomes). fig. 4. most important variables for class dry season (1.4.1.1.0), 2003-2020. full size image drought. in period 1, the triad of low values in percentage of employment in the manufacturing industry, monetary value added to administrative activities and percentage of investments in basic education (7th, 4th and 9th, fig. 5a) indicates that, to an inoperative public management corresponds a poorly dynamic economy and a precarious educational cultural standard for the child population. thus, the lack of commitment to the intellectual training of the new generations compromises the brightest perspectives in their occupational future and both make it impossible for these sub-citizens to exercise greater social control over the public policies. this context connects, in an unfavorable way, sdg 1 (eradication of poverty), particularly subgoal 1.5.b (support accelerated investments in actions to eradicate poverty) to sdg 8 (decent work and economic growth), especially subgoal 8.3 (promote development-oriented policies that support productive activities, generation of decent employment, entrepreneurship, creativity and innovation). period 2 reconfirms the meanings of this same triad, although it does so through the variables low value added by agriculture and low hdi (5th, 9th, fig. 5b), which clash with the same sub-goals of the sdgs mentioned above (1.5.b and 8.3). the low hdi, however, also points unfavorably to the possibility of achieving sdg 3 (health and well-being) and sdg 4 (quality education), distancing localities in drought emergencies from a local, state and regional development model effectively sustainable. fig. 5. most important variables for class drought (1.4.1.2.0) full size image forest fires. forest fires are well demarcated in the model in relation to the beginning and end months of the season in both periods. in the first one (2003-2011), it indicates direct correspondence with high percentages of employment in the service sector (5th, fig. 6a), these probably being harmed by atmospheric pollution and destruction of forests, especially if they were linked to tourism services, as in the brazilian pantanal. in the second period (2012-2020), the lowest percentage of vulnerable people dependent on the elderly (8th, fig. 6b) shows that these are economically prosperous locations. the regional characteristic (central-west) and the added value of agriculture (large-scale production) stand out (7th, fig. 6b), indicating the potential huge economic loss to the primary sector. if, on the one hand, the possibility of specifying the location of these events (by region and municipality) would facilitate prevention strategies, which would converge with sdg 15, sub-goal 15.2 (stop deforestation, restore degraded forests), on the other the recurrence of forest fires indicates that the relative ineffectiveness of the actions taken is far from achieving this goal. fig. 6. most important variables for class forest fires (1.4.1.3.1) full size image viral infectious diseases. viral infectious diseases gained prominence in the model only in the second period (2012-2020), mainly due to the covid-19 pandemic in 2020, which led to all regions of brazil declaring an emergency, which is why the event’s starting year is the best demarcated variable in the model (1st, fig. 7). the set of high values for variables related to economic activities (high percentage of jobs in the service sector and added value of agriculture, 9th and 10th, fig. 7) indicates the success of adaptive strategies in the face of a serious biological threat. despite the mandatory change in collective behavior, through preventive measures of social isolation, it was possible to keep the production and distribution of essential and consumer goods that society needed flowing, which is in line with the purposes of sdg 8.8 (promote safe and secure working environments for all workers) and 3.d (strengthen the capacity [...] particularly of developing countries, for early warning, risk reduction and management of national [...] health risks). in this case, even though the country’s health policies were unprepared to mitigate the situation that year, the health sector pursued sdg 3, sub-goal 3.9.b (support the research and development of vaccines and medicines for communicable and non-communicable diseases, which mainly affect developing countries). fig. 7. most important variables for class viral infectious diseases (1.5.1.1.0), 2012-2020. full size image 4 conclusions with only seven years left for nations, like brazil, to achieve the sdgs to which they had committed, united nations secretary antónio guterres stated: “unless we act now, the 2030 agenda will become an epitaph for a world that might have been.” the findings above indicate that the larger number of emergencies that occur in brazil indicative of weak institutional and social resilience to face geological, hydrological, climatic, biological and similar threats and prevent them from triggering disasters is intertwined with cultural vulnerabilities, historical economic and territorial issues, however, uncomfortably persistent in the face of multilateral aspirations. in other words, the brazilian pace of inducing sustainable inter-sectoral public policies, which can reduce the risks of disasters based on their discussion and training in the content of quality basic education, in the training processes to obtain employment, is slow. there is still a lack of good strategies to protect subsistence and small-scale agriculture in the face of climate setbacks as well as in the expansion of slope containment measures and the improvement of rainwater drainage systems, among others. during data modeling, we encountered problems with inconsistent and missing data for several indicators, which could significantly impact conclusions and prevent us from exploring more recent periods for an up-to-date warning study. despite an unbalanced dataset and the inability to create synthetic data, we obtained information that helped interpret events and achieved relevant performance using simple tree-based models. future studies could expand the analysis with a wider range of classes and variables at both state and municipal levels, enhancing the understanding of emergency decrees. improving the model would help express the association between emergencies and socioeconomic and socio-environmental variables. this would aid decision-makers in identifying underlying factors related to decrees, supporting a strong government commitment to investing in public institutions that collect, systematize, and disseminate updated open data, thus accelerating progress toward fulfilling the 2030 agenda. notes 1.https://www.gov.br/mdr/pt-br/assuntos/protecao-e-defesa-civil/sistema-integrado-de-informacoes-sobre-desastres. 2.http://www.atlasbrasil.org.br/consulta/planilha. 3.https://sidra.ibge.gov.br/tabela/21. 4.https://www.gov.br/inep/pt-br/acesso-a-informacao/dados-abertos/indicadores-educacionais/indicadores-financeiros-educacionais. 5.https://shap.readthedocs.io/en/latest. references brasil: classificação e codificação brasileira de desastres. secretaria nacional de proteção e defesa civil-sedec/mdr, brasília (2012) google scholar  kowarick, l.: viver em risco: sobre a vulnerabilidade socioeconômica e civil. editora 34, são paulo (2009) google scholar  fritz, c.: disaster. in: merton, r.. nisbet, r. (eds.) contemporary social problems. 1\(^{\underline{a}}\) ed., pp. 651–694. harcourt brace jovanovich, new york (1961) google scholar  quarantelli, e.: uma agenda de pesquisa do século 21 em ciências sociais para os desastres: questões teóricas, metodológicas e empíricas, e suas implementações no campo professional. o social em questão. rio de janeiro, vol. 18, pp. 25–56 (2015) google scholar  valencio, n.: para além do ‘dia do desastre’: o caso brasileiro. coleção ciências sociais. ed. appris, curitiba (2012) google scholar  valencio, n., valencio, a.: o assédio em nome do bem: dos sofrimentos conectados à dor moral coletiva de vítimas de desastres. lumina 12, 19–39 (2018) article  google scholar  united nations: the sustainable development goals report 2023, special united nations publications, new york (2023) google scholar  arrieta, a., et al.: explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. inf. fusion 58, 82–115 (2020) article  math  google scholar  lundberg, s., lee, s.-i.: a unified approach to interpreting model predictions. adv. neural. inf. process. syst. 30, 4765–4774 (2017) math  google scholar  breiman, l.: random forests. mach. learn. 45, 5–32 (2001) article  math  google scholar  chen, t., guestrin, c.: xgboost: a scalable tree boosting system. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794. acm, new york (2016) google scholar  yi, f., yang, h., chen, d., et al.: xgboost-shap-based interpretable diagnostic framework for alzheimer’s disease. bmc med. inform. decis. mak. 23(137), 1–14 (2023) math  google scholar  tan, b., gan, z., wu, y.: the measurement and early warning of daily financial stability index based on xgboost and shap: evidence from china. expert syst. appl. 227, 120325 (2023) article  google scholar  hatami, f., rahman, m., nikparvar, b., thill, j.-c.: non-linear associations between the urban built environment and commuting modal split: a random forest approach and shap evaluation. ieee access 11, 12649–12662 (2023) article  google scholar  andrade, j., de souza junior, t., silva, l. lucena, d., fernandes, b.: assessing the effect of urban expansion and deforestation on temperature rise in cajazeiras, brazil: a data-driven approach. in: 2023 ieee la-cci, pp. 1–6. ieee, recife (2023) google scholar  zhang, j., et al.: insights into geospatial heterogeneity of landslide susceptibility based on the shap-xgboost model. j. environ. manage. 332, 117357 (2023) article  google scholar  van, l., tran, v., nguyen, g., yeon, m., do, m., lee, g.: enhancing wildfire mapping accuracy using mono-temporal sentinel-2 data: a novel approach through qualitative and quantitative feature selection with explainable ai. eco. inform. 81, 102601 (2024) article  math  google scholar  zhang, b., salem, f., hayes, m., smith, k., tadesse, t., wardlow, b.: explainable machine learning for the prediction and assessment of complex drought impacts. sci. total environ. 898, 165509 (2023) article  google scholar  liu, s., et al.: evaluation of tropical cyclone disaster loss using machine learning algorithms with an explainable artificial intelligence approach. sustainability 15, 12261 (2023) article  google scholar  wang, m., et al.: an xgboost-shap approach to quantifying morphological impact on urban flooding susceptibility. ecol. ind. 156, 111137 (2023) article  math  google scholar  highland, l.m., bobrowsky, p.: the landslide handbook—a guide to understanding landslides. u.s. geological survey circular, vol. 1325 (2008) google scholar  wu, x., et al.: top 10 algorithms in data mining. knowl. inf. syst. 14(1), 1–37 (2008) article  mathscinet  math  google scholar  download references acknowledgments this study was funded by fapesp (processes 2022/09136-1 and 2023/03000-3), cnpq (processes 316828/2023-8 and 144465/2023-0) and capes. author information authors and affiliations computer science department, ufscar, são carlos, sp, brazil lucas teixeira, augusto matos & heloisa camargo department of environmental sciences, ufscar, são carlos, sp, brazil gabriel carvalho & norma valencio authors lucas teixeiraview author publications search author on:pubmed google scholar augusto matosview author publications search author on:pubmed google scholar gabriel carvalhoview author publications search author on:pubmed google scholar norma valencioview author publications search author on:pubmed google scholar heloisa camargoview author publications search author on:pubmed google scholar corresponding author correspondence to heloisa camargo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper teixeira, l., matos, a., carvalho, g., valencio, n., camargo, h. (2025). explainability of machine learning models with xgboost and shap values in the context of coping with disasters. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15415. springer, cham. https://doi.org/10.1007/978-3-031-79038-6_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79038-6_11 published: 31 january 2025 publisher name: springer, cham print isbn: 978-3-031-79037-9 online isbn: 978-3-031-79038-6 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning explainable ai disasters xgboost shap values sustainable development goals publish with us policies and ethics profiles heloisa camargo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature adaptive client-dropping in federated learning: preserving data integrity in medical domains | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper adaptive client-dropping in federated learning: preserving data integrity in medical domains conference paper first online: 30 january 2025 pp 111–126 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) adaptive client-dropping in federated learning: preserving data integrity in medical domains download book pdf download book epub arthur negrão9, guilherme silva9, rodrigo pedrosa9, eduardo luz9 & … pedro silva9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 483 accesses 1 citation abstract in this work, we address the challenge of training machine learning models on sensitive clinical data while ensuring data privacy and robustness against data corruption. our primary contribution is an approach that integrates conformal prediction (cp) techniques into federated learning (fl) to enhance the detection and exclusion of corrupted data contributors. by implementing a client-dropping strategy based on an adaptive threshold informed by the interval width metric, we dynamically identify and exclude unreliable clients. this approach, tested using the medmnist dataset with a resnet50 architecture, effectively isolates and discards corrupted inputs, maintaining the integrity and performance of the learning model. our findings demonstrate that this strategy prevents the potential 10% decrease in accuracy that can occur without such measures, confirming the efficacy of our cp-enhanced fl methodology in ensuring robust and private data handling in sensitive domains like healthcare. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others target informed client recruitment for efficient federated learning in healthcare article open access 18 december 2024 federated learning performance on early icu mortality prediction with extreme data distributions chapter © 2023 fedccw: a privacy-preserving byzantine-robust federated learning with local differential privacy for healthcare article 21 january 2025 1 introduction with advancements in computer systems and medicine, computer-assisted diagnoses have become more prevalent, aiding specialists in decision-making [19]. often, the data involved are sensitive, and their dissemination is constrained by privacy and security regulations. an example of such regulations is the brazilian general data protection law (lei geral de proteção de dados lgpd) [4], which reflects a common legislative approach among various countries. furthermore, the brazilian bank secrecy law (lei de sigilo bancário) [3] imposes additional restrictions on the circulation and disclosure of data, often resulting in a scarcity of large and accessible datasets [1]. this scenario presents a significant challenge for the training of deep learning models, which typically require extensive datasets. in this context, federated learning (fl) [21] offers a compelling strategy for building collaborative models among entities holding sensitive data without the needed of sharing it. fl achieves this by training multiple local models on decentralized datasets and then aggregating these models to form a comprehensive global model. this methodology permits all participating entities to benefit from the collectively derived insights while maintaining the confidentiality of their data [7]. moreover, as the models are trained locally, the overall demand for computational power is significantly diminished. this efficiency allows for the integration of larger datasets into the training process without the need for high-cost computational resources. by implementing federated learning, institutions can leverage diverse datasets for the enhancement of medical ai applications, thereby advancing healthcare innovations without breaching data privacy. addressing the data sharing challenge through federated learning (fl) is not straightforward. the difficulty arises because participants in a federated learning setup might contribute inconsistently, that is, some participants may not contribute effectively, while others could inadvertently or maliciously introduce corrupted data. managing such corrupted data is a significant challenge; it can severely degrade the performance and reliability of the overall model. ensuring that models maintain their robustness in the face of corrupted data is therefore a crucial obstacle that must be overcome [15]. previous approaches have faced challenges in balancing data privacy with model reliability [10]. this work integrates conformal prediction (cp) techniques [2, 16] within a federated learning framework, enhancing both the robustness and reliability of the model by providing a measure of uncertainty for each client. cp, a statistical approach that validates the confidence level of predictions, allows for the use of cp metrics to develop an adaptive threshold strategy. this strategy identifies and manages clients contributing corrupted data during aggregation rounds, thereby ensuring that central model training incorporates only reliable data. the primary objective of this work is to test the hypothesis that applying conformal prediction techniques within a federated learning context can enhance the robustness and reliability of trained models in medical applications, particularly in distributed scenarios where corrupted data may be present. two research questions arise from this hypothesis: (rq1) how does the introduction of corrupted data affect the precision and reliability of trained models in a federated learning context? (rq2) can a client-dropping strategy based on conformal prediction metrics effectively reduce the negative effects of corrupted data on a global model in a federated learning context? to address these research questions and the hypothesis, we conducted experiments using the medmnist dataset [20], comparing the results of a traditional neural network approach with those of a federated approach. additionally, we developed a strategy to manage clients with bad/corrupted data based on an adaptive conformal prediction threshold. this threshold uses the interval width metric to determine whether a client should be dropped during an aggregation round. the experimentation revealed that the proposed strategy is resilient in the presence of corrupted data, with an average accuracy difference of less than 2% between an expected scenario with no corrupted data and a scenario with a corrupted client using the proposed approach. in contrast, a scenario with corrupted data without the proposed strategy showed almost a 10% decrease in accuracy. furthermore, applying the proposed approach in a context with no corrupted clients resulted in a difference of less than 1% and even enhanced accuracy in some datasets. these results demonstrate that the proposed strategy is resilient against corrupted data and does not negatively impact scenarios without corrupted clients. this work is subdivided as follows: sect. 2 presents related works; the data used is presented in sect. 3; section 4 presents proposed methodology, followed by experiments and results in sect. 5; finally, conclusions and future researches are presented on sect. 6. 2 related works although studies on federated learning (fl) are relatively recent, they already show promising results. in [9], the authors conducted tests on three different datasets (mnist, mimic-iii, and ecg), and in all three cases, the models were able to maintain good predictive performance. additionally, the federated approach proved robust against data distortion and imbalanced distribution, demonstrating that, besides promoting data security, federated learning can bring various other benefits. other recent approaches combine graph neural networks (gnns) with federated learning [6]. it shows that this hybrid approach, in addition to the evident privacy gains, was able to maintain high-quality predictive performance, matching or even surpassing centralized learning in some cases. however, the authors noted that federated gnns perform worse on datasets with non independent and identically distributed data distribution (non-iid split) compared to centralized approaches. they suggest that future work should address this and other challenges in the field of fl. in [14], a study on various model aggregation techniques was conducted. using the mnist dataset, the authors tested federated averaging (fedavg), federated stochastic variance reduced gradient, and co-op, achieving the best results with fedavg. they also noted that fedavg is efficient with or without independent and identically distributed (i.i.d.) partitioning, and for i.i.d. partitioning, the performance is similar to the centralized approach. this reinforces the hypothesis that federated learning can avoid data exchange while maintaining the efficiency of trained models in various scenarios. related to the training and aggregation process, the folb algorithm was proposed in [13]. it promises to accelerate and improve the convergence of fl models. the algorithm analyzes the computation capacity, communication, and heterogeneity of the clients to perform client sampling in each training round. it assigns different weights to each client’s model during the aggregation process. the authors conducted empirical tests and demonstrated that the algorithm improves the convergence speed, accuracy, and stability of models across many tasks and datasets. regarding information security in federated learning, [12] discusses that, despite providing a learning environment without the need to share data, federated learning still presents some vulnerabilities that designers must be aware of. poisoning attacks and inference attacks threaten the learning environment by allowing malicious clients to distort the produced data, making gradient descent inefficient and hindering the construction of an effective model. additionally, analyzing data provided by the server, such as weights and/or loss from each training round, can infer input data and their labels from other clients in the federated network. in high-stakes decision-making scenarios (e.g., medical), the lack of model interpretability can lead to distrust among users. in [11], the authors propose conformal prediction (cp) analysis as an effective and simple way to provide statistical confidence in predictions, offering clear information about prediction uncertainty without requiring direct modification to the model training process. in their experimentation, they conducted cp analysis on a model trained under the fl paradigm using 6 subsets from medmnist (blood, derma, path, tissue retina, organ3d). they achieved up to \(97\%\pm 0.7\%\) coverage and found a correspondence between conformal uncertainty and prediction task difficulty (measured by class entropy). in summary, the studies discussed in this section highlight the growing importance and promising potential of federated learning. various works demonstrate, in different contexts and types of data, that federated approaches are robust and efficient. however, it is crucial to recognize and address the identified vulnerabilities, such as poisoning and inference attacks, to ensure the integrity and security of federated learning systems. unlike the work in [11], we not only analyze cp but also introduce a client-dropping strategy. 3 medmnist dataset the medmnist dataset [20] consists of a collection of 18 subsets of biomedical images standardized in a format similar to mnist. this dataset comprises 12 subsets of pre-processed data in \(28\times 28\) (2d) and six more subsets in \(28\times 28\times 28\) (3d). among these subsets, only the path, blood, derma, and retina datasets are in rgb format, while the others are in grayscale. this data collection was developed to address a variety of tasks, such as binary/multi-class classification, ordinal regression, and multi-label tasks, and it covers a wide range of data scales, ranging from 100 to 100,000 images. figure 1 presents examples of all the used datasets. fig. 1.adapted from [20]. used subsets of medmnist dataset. full size image considering the nature of the data in the medmnist datasets, some were excluded from this work. the chest subset was discarded because it is multi-label (the same image belongs to one or more classes simultaneously), and the 3d subsets—organ, nodule, adrenal, fracture, vessel, and synapse—were also excluded. therefore, the remaining 11 datasets used in this work are: blood, breast, derma, oct, organa, organc, organs, path, pneumonia, retina, and tissue. each dataset used in this study is from a medical domain. the descriptions of each dataset are as follows: the blood dataset regards the problem of classifying blood cells. the breast dataset comprises breast ultrasound images divided into two conditions: normal or benign (1), and malignant (2). the derma dataset contains dermatoscopic images of pigmented skin lesions, split into seven different diseases. the oct dataset includes optical coherence tomography (oct) images for retinal diseases, divided into four diagnoses. the organa/c/s datasets consist of computed tomography (ct) scan images of body organs from three viewing planes: axial, coronal, and sagittal. the path dataset is made up of histological images of colorectal cancer used to predict the survival of individuals, collected from two subsets: nct-crc-he-100k (training and validation) and crc-val-he-7k (testing). the pneumonia dataset consists of pediatric chest x-ray images classified as pneumonia or normal. the retina dataset contains fundus images used for ordinal regression to classify the severity of diabetic retinopathy into five levels. the tissue dataset consists of images of human renal cortex cells divided into eight classes. it is worth noting that the image formats were not altered, and no preprocessing was applied to the data. table 1 provides details for each of the datasets used. 4 methodology in this section, we outline the methodology adopted for this studyfootnote 1 and the metrics used for evaluation. first, we present the model used for the evaluation process, along with the federated learning strategy employed. next, we introduce and describe the proposed approach that utilizes conformal prediction for client-dropping. finally, we detail the metrics used to evaluate the model, including an in-depth description of the conformal prediction metrics used. table 1. details of the used datasets. adapted from [20]. mc = multi-class, bc = binary classification, or = ordinal regression, ct = computed tomography and oct = optical coherence tomography.full size table 4.1 model used on training process the authors of [20] used the resnet architecture [18] for their benchmark. for this reason, the resnet50 architecture was employed in this study without pre-trained weights, with modifications only to the input data format and the number of neurons in the last layer to suit each data subset. the resnet50 architecture consists of 49 convolutional/pooling layers, with the final layer being fully connected for classification. for comparison purposes, the same model used in federated training was also evaluated in a centralized context. the model was trained for a proportional number of epochs and evaluated on the same test data. in both cases, the validation data were used to identify the best model during training. the model with the lowest loss and highest accuracy rate during the global epochs was used to calculate the metrics and report the results. 4.2 federated learning federated learning (fl) [21] is a technique for training deep learning models that operates in a distributed environment, allowing for the collaborative learning of a global model while the data remains locally on the users’ devices. in contrast to centralized approaches, where all data is available in a single location for model training, fl allows diverse devices and servers to perform training in parallel without data centralization, avoiding data exchange [7]. figure 2 shows the workflow of the fl technique used in this work. fig. 2. a client, with his respective portion of data d (\(d^c\) all clients have the entire training dataset, \(d^f_i\) each client has a portion of the entire training dataset), trains a local model \(m^{l} _{i}\). those local models are aggregated generating a new global model \(m^{g}_{i}\), which is then distributed across all clients for a new training round. this process is repeated for a number of global epochs, and after that the final global model is evaluated. full size image for the federated model experimentation, the data was divided in two ways. in the first format, each client has access to all entries in the dataset. this strategy is referred to in the experiments as “complete dataset”. although this format is not realistic, it serves as an upper bound on accuracy performance. in the second format, referred to as “fragmented dataset” in the experiments and closer to a real-world scenario, each client has \(\frac{1}{q_c}\) percent of the dataset, where \(q_c\) is the total number of clients in a given execution. this second strategy allows the repetition of the same training/validation instance among clients. however, despite possible repetitions, there is considerable data diversity among clients. they can be described as: complete dataset: let d be the dataset, and \(m_g\) be the global model. in the complete dataset format, each client \(c_i\) with data \(d_i\) has access to the entire dataset d, that is $$\begin{aligned} d_i = d, \quad \forall i \in \{1, 2, \ldots , q_c\}. \end{aligned}$$ (1) fragmented dataset: in the fragmented dataset format, each client \(c_i\) has access to a \(\frac{1}{q_c}\) fraction of the dataset \(d_i\), that is $$\begin{aligned} d_i = \frac{d}{q_c}, \quad \forall i \in \{1, 2, \ldots , q_c\} \end{aligned}$$ (2) there are various strategies for manipulating and calculating the loss value during federated training. among the existing ones, federated averaging (fedavg) was chosen due to the good results presented by [14, 17]. in this context, the loss function value of a global epoch is calculated using the average error found on each client. this value is used to update the global model weights, which are then redistributed to all clients. it can be defined as: federated averaging: each client \(c_i\) trains a local model \(m_{l_i}\) on its local dataset \(d_i\) and computes the local loss \(l_{l_i}\). the local models and losses are then sent to the server to update the global model \(m_g\). the global loss (\(l_g\)) is defined as: $$\begin{aligned} l_g = \frac{1}{q_c} \sum _{i=1}^{q_c} l_{l_i}. \end{aligned}$$ (3) updating the global model weights: the global model weights are updated using the average of the local models’ parameters, described as: $$\begin{aligned} m_g = \frac{1}{q_c} \sum _{i=1}^{q_c} m_{l_i}. \end{aligned}$$ (4) it is worth pointing out that the updated global model \(m_g\) is then redistributed to all clients for the next round of training. the entire process can be seen in algorithm 1. besides, in both federated and local training, the test data is used only for metric calculation and not for training. algorithm 1 proposed federated model experimentation full size image 4.3 measuring uncertainty with conformal prediction for the conformal prediction [2] analysis, two metrics were used: interval width (|i|) and coverage rate (cr). the interval width can be mathematically defined as: $$\begin{aligned} |i| = 2*q_{\alpha }(|\hat{y}_i y_{i}|), \end{aligned}$$ (5) where \(q{\alpha }\) is the \(\alpha \) quantile of the residual distribution \(r = |\hat{y}_i y_i|\) (\(\hat{y}_i\) being the prediction and \(y_i\) the true label), indicating how far predictions are from true labels. the smaller the |i|, the less uncertainty there is in the predictions [5, 16]. the coverage rate can be mathematically defined as: $$\begin{aligned} cr = \frac{1}{n} \sum _{i}^{n} f(y_{i} \in i(x_{i})), \end{aligned}$$ (6) where n is the number of instances, \(i(x_i)\) is the prediction interval for instance \(x_i\), and f is a function that returns 1 if the condition is true and 0 otherwise. the higher the coverage rate, the more reliable is the predicted interval in containing the true values [8]. 4.4 conformal prediction based on strategy for dropping clients during the federated training process, clients can encounter issues and return incorrect parameters for global model merging during the global epoch. this issue can be the result of various factors, such as network problems, corrupted data, an so on. to address this issue, we propose simulating a corrupted client by replacing its data with a random array of the same dimensions. to mitigate the negative impact of a corrupted client, we introduce a client-dropping strategy based on the interval width metric. a client is excluded from a given aggregation round if its interval width is greater than a specified threshold t, that is, \(|i_{i}| > t\), where \(|i_{i}|\) is the interval width of the i-th client calculated in eq. (5). this threshold, which varies over the aggregation rounds, is calculated as $$\begin{aligned} t = \frac{\sum _{i=1}^{n} |i_{i}|}{q_c} + 0.05 \times \frac{\sum _{i=1}^{n} |i_{i}|}{q_c}, \end{aligned}$$ (7) where \(q_c\) is the number of clients. in practical terms, clients with interval widths 5% greater than the average are discarded and not used in computing the new global model weights. the 5% threshold was determined in preliminary runs. this strategy aims to enhance the robustness of federated learning by effectively identifying and excluding corrupted clients. it is worth to highlight that this strategy also can remove no-corrupted clients and a client is removed during a global epoch, although, it can be used in a next global epoch, since it can be fixed during the training. to the best of the authors’ knowledge, no other work that proposes a client drop strategy in the same way as described was found, that is, no other work used the interval width metric to drop clients in fl context. the metrics used, on the other hand, are derived from adaptations of existing works (sect. 4.3). 4.5 evaluation metrics besides the conformal prediction metrics, accuracy (acc) is used for evaluating the trained models. accuracy can be defined as the number of correct classifications (where the predicted class matches the expected class) divided by the total number of classifications made. accuracy ranges from 0 to 1 (or from 0% to 100%), where a value closer to one indicates a higher accuracy rate and better model predictions. 5 experiments and results in this section, we outline the experimental setup, as well as the results and their respective implications. 5.1 experiments setup for the experiments described below, the model was trained for 10 local epochs and 10 global epochs (rounds), with data divided into batches of 128 images each. the adam optimizer was used with a learning rate of \(10^{-3}\) to adjust the model parameters, and the adopted loss function was the cross-entropy loss. a 10% learning rate decay was applied at the 50th and 75th total epoch (i.e., at the first local epoch in the fifth round and the fifth local epoch in the seventh round). it is worth noting that the choice of these hyperparameters follows [20], aiming to keep all variables constant except for the “federated versus centralized” comparison. additionally, for each data subset and each form of data partitioning, the number of clients involved in the training process was varied, with tests conducted with three, four, five, and six clients. the experiments were conducted on three different systems, within the same local network, with the following configurations: system 1: intel i7-5820k 3.3 ghz; gpu titanxp 12 gb; ram ddr4 12 gb; system 2: intel i9-10900 2.80 ghz; gpu rtx 3090 24 gb; ram ddr4 128 gb; and system 3: amd ryzen threadripper 3960x 3.70 ghz; gpu rtx 3090 24 gb. the distribution of systems and clients was defined as follows: (i) for the scenario with three clients, one client was assigned to each system; (ii) for four clients, two clients were on system 2 and one on the others; (iii) for five clients, two clients were on systems 2 and 3, and only one on system 1; and (iv) for six clients, two clients were on each system. the server responsible for managing federated learning process and aggregation was always on system 3. the tensorflow framework was used to manipulate the model, the scikit-learn library for calculating metrics, and the flower framework to perform federated learning. in addition to the above experiments, cp-related experiments were also conducted in this study. the model described in sect. 4.1 was trained in the fl paradigm with three clients (one client per system) in both fragmented and complete dataset cases, using the same configuration as the previous experiments. for the cp-related configuration, the residual threshold was set to 0.3, and the conformal quantile was set to 0.95. these experiments were conducted in the following scenarios: s1 no data abnormalities; s2 one client with 100% corrupted data; s3 no data abnormalities along with the client-dropping strategy; s4 one client with 100% corrupted data along with the client-dropping strategy. 5.2 experiment results the accuracy results achieved in the evaluations conducted in each scenario, using the resnet50 model, are presented in table 2. these results underscore the effectiveness and reliability of the trained model, as demonstrated by the presented data. table 2. accuracy obtained by the federated approach, local approach, and the reference results (ref) obtained in the work [20]. cli = number of clients.full size table regarding table 2, it can be stated that clients who trained with the complete dataset generally achieved better results than those who trained with the subdivided dataset. despite the difference between the strategy using the entire dataset and the one using only a portion of the dataset, it is noticeable that this difference is relatively small, typically around 0.05 or 5% in terms of accuracy. when comparing the results of federated learning (fl) with those of local (centralized) training, it is evident that the federated model achieves comparable and, in some cases, even superior results. this phenomenon is particularly notable with fragmented datasets, where the federated model trains with different subsets of data on each client. this exposure to diverse scenarios during training enhances the model’s ability to generalize, contributing to its superior performance. in the context of complete datasets, although the same data are used in all clients, the randomness during training also contributes to data diversification. this federated training approach results in more robust and adaptable models to different scenarios, demonstrating the effectiveness and advantages of this strategy compared to centralized training. still on complete datasets, it is notable that the variation in the number of clients had little impact on the obtained results, especially concerning the accuracy metric (see table 2). this phenomenon suggests that, for this specific problem, the number of models in the aggregation had a less significant influence on the results than the amount of information available to each individual model. this observation highlights the importance of the quality and diversity of training data over the number of models used in the context of federated learning. when comparing the accuracy obtained in the federated context with the results presented by [20], it is evident that the former stands out in five cases compared to the latter and shows comparable results in the others. this confirms the effectiveness and feasibility of this strategy. 5.3 conformal prediction and client-dropping analysis in this section, we present the results of the conformal prediction experiments across four scenarios (s1, s2, s3, and s4) as described in the experimental setup. this section addresses the rq1 and the rq2 research questions defined in the introduction. the experiments were conducted using only three clients, as no significant differences were observed in table 2 when more clients were used, with each client utilizing the entire training dataset. table 3 shows a considerable reduction in accuracy when a corrupted client is included in the experiment (s2), compared to experiments without a corrupted client (s1) and those where the corrupted client removal strategy is adopted (s3 and s4) for both dataset types (complete and fragmented). for instance, in the path and oct subsets, the accuracy dropped by approximately 47% and 24%, respectively. conversely, in the derma subset, the drop was approximately 3%. table 3. results of conformal prediction (cp) experiments for all three clients, considering the four scenarios: s1: none corrupted clients; s2: one corrupted client; s3: none corrupted clients and applying the client-dropping strategy; and s4: one corrupted client and applying the client-dropping strategy. the best results in each scenario are highlighted in bold.full size table it is important to note from table 3 that in both complete and fragmented data scenarios, the client-dropping strategy does not impact training when no corrupted client is present (s3). however, it offers significant benefits when a corrupted client is included (s4). with the client-dropping strategy, the accuracy achieved was close to the ideal (s1) and, in some cases, even higher. this can be attributed to the global model being merged with local models that have less uncertainty, resulting in a more accurate and reliable model. even in scenarios where the drop in accuracy was mild, prediction uncertainty generally increased. notable examples include the derma and organa datasets, where slight drops in accuracy (3% and 2%, respectively) compared to scenario s1 were accompanied by significant increases in prediction uncertainty. specifically, there were increases of 0.39 and 0.56 in interval width and decreases of 0.28 and 0.18 in coverage rate for each respective dataset. additionally, results from s3 and s4 are very close to (and sometimes slightly better than) those from s1. 5.4 discussion as can be observed in fig. 3, all models trained with fragments of the original dataset tend to have lower accuracy performance compared to models trained with the entire dataset. it is also noticeable that the models trained with six clients had results closest to the reference benchmark values conducted in [20]. figure 4 graphically presents the accuracy results obtained by the models trained on fragmented datasets. as can be observed, in all federated models, the results were close to the reference benchmark values conducted in [20]. the federated strategy using six clients showed the best results across all datasets. more clients were not tested due to the lack of infrastructure to run the experiments. fig. 3. accuracy graph of all models trained with fragmented and complete datasets. full size image changing focus to the cp analysis, it is possible to see that prediction uncertainty metrics (i.e. interval width and coverage rate) can be very useful to detected and possibly discard corrupted clients and their bad contribution to the learning process. specially on cases where accuracy drops are mild, where the high uncertainty on corrupted client predictions would probably be masked, those metrics can provide good intel whether a client is corrupted or not. in conclusion, it is possible to state that the federated approach succeeded in avoiding data sharing while building competitive models, in terms of accuracy, when compared to the centralized model extensively tested with medmnist in [20]. in certain cases, especially in the tests conducted on the blood, breast, derma, oct, and pneumonia datasets, the federated approach outperformed the reference model, reinforcing the potential efficiency of the federated strategy. fig. 4. accuracy graph of the models trained with fragmented datasets. full size image 6 conclusion this study proposes a client-dropping strategy using conformal prediction in the federated learning context. the proposed approach preserves the positive characteristics of federated learning, such as data security, while maintaining predictive power and generalization capability in scenarios with and without clients with corrupted data, specifically in the context of the 2d-classification subsets from the medmnist dataset. additionally, it fosters a learning environment less likely to incorporate noise into the learned data representation. overall, the proposed strategy demonstrated effectiveness by producing results comparable to or even superior to local (centralized) training and normal federated learning scenarios without corrupted clients, with less than a 2% difference in accuracy on average. when the proposed approach was not used in scenarios involving a client contributing corrupted data, we observed an average accuracy reduction of nearly 10%. these results confirm the strategy’s validity and address the initial research questions. additionally, the consistency in performance across varying client numbers in fragmented dataset scenarios indicates that data can be effectively distributed across multiple sites without compromising computational efficiency. future research could focus on expanding federated learning strategies, assessing the impact of a higher proportion of corrupted clients, and exploring the challenges of non-iid data splits. notes 1.code available at https://www.dropbox.com/scl/fi/607e0jiy80zbxs7ug96i4/bracis_code_drop_strat.zip?rlkey=p2jq5n1li473itnjcx1tgst5f&st=o2yuoo8l&dl=0. references abad, g., picek, s., ramírez-durán, v.j., urbieta, a.: on the security & privacy in federated learning. arxiv preprint arxiv:2112.05423 (2021) angelopoulos, a.n., bates, s.: a gentle introduction to conformal prediction and distribution-free uncertainty quantification. arxiv preprint arxiv:2107.07511 (2021) congresso nacional: lei complementar n\(^{\underline{\text{o}}}\) 105, de 10 de janeiro de 2001 (2001). https://www.planalto.gov.br/ccivil_03/leis/lcp/lcp105.htm congresso nacional: lei n\(^{\underline{\text{ o }}}\) 13.709, de 14 de agosto de 2018 (2018). https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm dunn, p.k., smyth, g.k.: randomized quantile residuals. j. comput. graph. stat. 5(3), 236–244 (1996) article  math  google scholar  he, c., et al.: fedgraphnn: a federated learning system and benchmark for graph neural networks. arxiv preprint arxiv:2104.07145 (2021) kairouz, p., et al.: advances and open problems in federated learning. found. trends® mach. learn. 14(1–2), 1–210 (2021) google scholar  karimi, h., samavi, r.: quantifying deep learning model uncertainty in conformal prediction. in: proceedings of the aaai symposium series, vol. 1, pp. 142–148 (2023) google scholar  lee, g.h., shin, s.y.: federated learning on clinical benchmark data: performance assessment. j. med. internet res. 22(10), e20891 (2020). https://doi.org/10.2196/20891, http://www.jmir.org/2020/10/e20891/ li, s., ngai, e., ye, f., voigt, t.: auto-weighted robust federated learning with corrupted data sources. acm trans. intell. syst. technol. (tist) 13(5), 1–20 (2022) math  google scholar  lu, c., kalpathy-cramer, j.: distribution-free federated learning with conformal predictions. arxiv preprint arxiv:2110.07661 (2021) lyu, l., yu, h., yang, q.: threats to federated learning: a survey. arxiv preprint arxiv:2003.02133 (2020) nguyen, h.t., sehwag, v., hosseinalipour, s., brinton, c.g., chiang, m., poor, h.v.: fast-convergent federated learning. ieee j. sel. areas commun. 39(1), 201–218 (2020) article  google scholar  nilsson, a., smith, s., ulm, g., gustavsson, e., jirstrand, m.: a performance evaluation of federated learning algorithms. in: proceedings of the second workshop on distributed infrastructures for deep learning, pp. 1–8 (2018) google scholar  park, s., et al.: feddefender: client-side attack-tolerant federated learning. in: proceedings of the 29th acm sigkdd conference on knowledge discovery and data mining, pp. 1850–1861 (2023) google scholar  shafer, g., vovk, v.: a tutorial on conformal prediction. j. mach. learn. res. 9(3) (2008) google scholar  sun, t., li, d., wang, b.: decentralized federated averaging. ieee trans. pattern anal. mach. intell. 45(4), 4289–4301 (2022) article  math  google scholar  targ, s., almeida, d., lyman, k.: resnet in resnet: generalizing residual architectures. arxiv preprint arxiv:1603.08029 (2016) yanase, j., triantaphyllou, e.: a systematic survey of computer-aided diagnosis in medicine: past and present developments. expert syst. appl. 138, 112821 (2019) article  math  google scholar  yang, j., et al.: medmnist v2-a large-scale lightweight benchmark for 2d and 3d biomedical image classification. sci. data 10(1), 41 (2023) article  math  google scholar  zhao, y., li, m., lai, l., suda, n., civin, d., chandra, v.: federated learning with non-iid data. arxiv preprint arxiv:1806.00582 (2018) download references acknowledgements this work was carried out with the support of the coordination for the improvement of higher education personnel brazil (capes) financial code 001, the minas gerais state research support foundation (fapemig), the national council for scientific and technological development (cnpq), and the federal university of ouro preto (ufop/proppi). author information authors and affiliations departamento de computação, universidade federal de ouro preto (ufop), ouro preto, mg, brazil arthur negrão, guilherme silva, rodrigo pedrosa, eduardo luz & pedro silva authors arthur negrãoview author publications search author on:pubmed google scholar guilherme silvaview author publications search author on:pubmed google scholar rodrigo pedrosaview author publications search author on:pubmed google scholar eduardo luzview author publications search author on:pubmed google scholar pedro silvaview author publications search author on:pubmed google scholar corresponding author correspondence to arthur negrão . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper negrão, a., silva, g., pedrosa, r., luz, e., silva, p. (2025). adaptive client-dropping in federated learning: preserving data integrity in medical domains. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_8 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords federated learning medmnist conformal prediction cnn publish with us policies and ethics profiles eduardo luz view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature anomaly detection in brazilian federal government purchase cards through unsupervised learning techniques | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper anomaly detection in brazilian federal government purchase cards through unsupervised learning techniques conference paper first online: 28 november 2021 pp 19–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) anomaly detection in brazilian federal government purchase cards through unsupervised learning techniques download book pdf download book epub breno nunes  orcid: orcid.org/0000-0002-5854-928810, tiago colliri  orcid: orcid.org/0000-0001-9233-466211, marcelo lauretto  orcid: orcid.org/0000-0001-5507-236812, weiguang liu13 & … liang zhao  orcid: orcid.org/0000-0002-1502-660414  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1250 accesses 1 citation abstract the federal government purchase card (cpgf) has been used in brazil since 2005, allowing agencies and entities of the federal public administration to make purchases of material and provision of services through this method. although this payment system offers several advances, in the technological and administrative aspect, it is also susceptible to possible cases of card misuse and, consequently, waste of public funds, in the form of purchases that do not comply with the terms of the current legislation. in this work, we approach this problem by testing and evaluating unsupervised learning techniques on detecting anomalies in cpgf historical data. four different methods are considered for this task: k-means, agglomerative clustering, a network-based approach, which is also introduced in this study, and a hybrid model. the experimental results obtained indicate that unsupervised methods, in particular the network-based approach, can indeed help in the task of monitoring government purchase card expenses, by flagging suspect transactions for further investigation without requiring the presence of a specialist in this process. this research was funded by cepid-cemeai – center for mathematical sciences applied to industry (grant 2013/07375-0, sao paulo research foundation–fapesp), and was carried out at the center for artificial intelligence (c4ai-usp), with also support from the sao paulo research foundation (fapesp) under grant number: 2019/07665-4 and by the ibm corporation. this work is also supported in part by fapesp under grant numbers 2015/50122-0, the brazilian national council for scientific and technological development (cnpq) under grant number 303199/2019-9, and the ministry of science and technology of china under grant number: g20200226015 access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others wave hedges distance-based feature fusion and hybrid optimization-enabled deep learning for cyber credit card fraud detection article 24 july 2024 enhancing credit card fraud detection using knowledge graphs and centralities chapter © 2025 impact of boosting techniques in ai-based credit card fraud detection classifier chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. data mining and knowledge discovery data mining machine learning public economics public finance public sector studies 1 introduction in the last years, the credit card market has been rapidly increasing in brazil and worldwide, trading a total amount of more than r$ 1.1 trillion in 2019, only in brazil, which represents a growth of almost 20% over the previous year [1]. the availability of easy to use and practicality led government agencies from several countries to also adopt the use of this payment method. however, with the credit card market transacting increasingly larger volumes, the detection of fraud and anomalies has become one of the biggest challenges faced by companies in this branch [18]. it is estimated that, annually, fraud and anomalies in the use of credit cards generate losses of billions of dollars worldwide. in brazil, the federal government purchase card (cpgf, in portuguese) was instituted in 2001 by means of a presidential decree, but its use only started in 2005 when it was regulated by another presidential decree [17]. the cpgf allows agencies and entities of the federal public administration that are part of the fiscal budget and of the social security to make purchases of material and provision of services, under the terms of the current legislation. the cpgf is issued in the name of the management unit of each agency, with the nominal identification of its bearer. due to its specificity, the main concern regarding the cpgf is not related to fraud – in the sense of theft and/or cloning cases – but in the detection of possible anomalies that can help to identify expenses not covered by the current legislation, thus raising possible misuse of cpgf and, consequently, public funds. in this sense, in an attempt to bring transparency to cpgf expenses, the brazilian transparency portal [19] presents a series of information regarding its transactions, such as: expenses by type of card, by cardholders and favored. nevertheless, despite the efforts from the brazilian transparency portal to transform the public spending technical data, especially expenses made on the cpgf, into information intelligible to the society in general, this promotion of social control does not fully achieve its objective, since this information, as it is currently presented, does not point out possible anomalies in the application of public resources. therefore, studies detecting possible anomalies in these data can help to identify non-compliant transactions and potential misuse of the cpgf. machine learning techniques for detecting fraud/anomalies can be divided into two main approaches: supervised and unsupervised ones [18]. in the supervised approach, the dataset used to train the model needs to be labeled as legitimate or fraudulent transactions. in the case of credit cards, such labels often do not exist in the dataset, and the suggested approach, in this case, is the unsupervised one. several studies have been developed over the years, in different areas, on the identification of anomalies using unsupervised machine learning models. one of these works [14], which can be applied in different areas such as financial transactions and industrial machines, focus on the detection of anomalies in time series, proposing a fast and efficient anomaly detection scheme focusing on fluctuation features, thus being capable of handling non-extreme fluctuation anomalies involved in periodic patterns. in the specific area of fraud detection in financial transactions, an unsupervised deep learning model was used to classify brazilian exporters regarding the possibility of committing fraud in exports [16]. there is also a study [8] which combines unsupervised and supervised learning, in a complementary form: while supervised techniques learn from the fraudulent behaviors of the past, unsupervised techniques are aimed at detecting new types of fraud in the present. another particularly related study [3] proposes a methodology for detecting anomalies in government purchases, in the form of public bids, through the use of frequent pattern mining, temporal correlation and joint multi-criteria analysis, obtaining results that are in line with those previously and independently gathered by specialists from the office of the comptroller general (cgu), through former audits and inspections. in this work, we investigate anomalies detection in an unsupervised manner, by using real data from cpgf with the main objective of public transparency, i.e., in an attempt to identify purchases that do not comply with the current legislation for the set of goods and services that can be purchased through this payment method. to this end, four different machine learning techniques are tested: two clustering methods (k-means and agglomerative clustering), a network-based approach, which is also introduced in this work, and a hybrid model, based both on networks and the k-means method. we apply these techniques on historical cpgf expenses real data from brazilian public agencies, occurred in 2018 and 2019. the experimental results obtained indicate that unsupervised methods have the potential to help in the task of monitoring government purchase card expenses, by flagging suspect transactions for further investigation without requiring the presence of a specialist in this process. 2 materials and methods the dataset used in this study was built from original data obtained in the brazilian transparency portal [19], comprising a total amount of r$ 55,048,553 in cpgf expenses, from 211,896 different transactions, performed by 175 public agencies, in 2018 and 2019. we focus our analyses on data from two public agencies which were among the top largest spenders during this period: army command and ministry of agriculture, livestock and food supply (table 1). table 1. summary of the data used in the analysisfull size table in fig. 1, we show the frequency histogram of all values and also the ones from the 10 most frequent types of expense, from ministry of agriculture cpgf data. one can observe, from this figure, that the values are concentrated in smaller transactions. we also note that, overall, the values do not seem to follow a normal distribution, and may present different curve shapes, according to the expense type. fig. 1. frequency histogram of all values and of the 10 most frequent types of expense, from ministry of agriculture cpgf data. full size image for the anomaly detection task, four different unsupervised techniques are tested: k-means [15], agglomerative clustering [12], a network-based approach, also introduced in this study, and a hybrid model, based on complex networks and the k-means method. following, we describe each one of them in more details. 2.1 k-means method the idea of the k-means algorithm is to group n objects of a sample into k clusters by minimizing the moment of inertia of each cluster, i.e., by making the data within each cluster as close as possible. k-means uses the squared euclidean distance to calculate the inertia or, as it is also known, the within-cluster sum of square wcss. therefore, the total inertia of a dataset can be denoted as follows: $$\begin{aligned} \sum _{i=1}^k \sum _{x \in c_i} \left\| {x-\mu _i}\right\| ^2 \quad , \end{aligned}$$ (1) which means that for all points x that are in cluster \(c_i\), the square of the distances from each point to the center of the cluster \(\mu _i\) is summed up, and to find the total inertia, we just sum all the inertia values of each of the k clusters in the dataset. one of the problems of the k-means algorithm is that it is extremely sensitive to the centroids starting conditions. this problem can be addressed through the initialization scheme called k-means \(^{++}\) [6]. in this method, the first centroid is selected randomly, and the others are selected based on the distance to the first one. in this way, k-means \(^{++}\) reduces the likelihood of bad starting conditions by selecting the initial centroids that are usually far from each other. the best number of clusters k in this technique is found using the elbow method [4], and the initialization parameter is adjusted by the k-means \(^{++}\) technique. 2.2 agglomerative clustering method agglomerative clustering is a bottom-up hierarchical clustering algorithm where the hierarchy of clusters is represented as a tree, called a dendrogram. the root of the dendrogram is the single cluster that contains all the samples, whereas the leaves are the clusters where each sample would be considered as a cluster. in this study, the dendrogram is created by using the ward linkage method [20], that seeks to minimize the sum of the square of the differences within all clusters. this is an approach which minimizes the variance and, in this respect, is somewhat similar to the k-means function, but from a hierarchical perspective. for determining the number of clusters to be considered for this technique, we set a threshold value and draw a horizontal line that cuts the tallest vertical line in the dendrogram, and take the number of vertical lines which are intersected by the line drawn using the threshold as the number of clusters to be used in the analysis. in this case, as one can observe from fig. 2, the number of clusters has been defined as 3, both for ministry of agriculture and army command data. fig. 2. dendrograms generated from ministry of agriculture (left) and army command data (right), for the hierarchical agglomerative clustering analysis. full size image 2.3 network-based approach in the last years, driven by technological advances and by the increase in the number of data available to be analyzed, the area of complex networks has emerged as a topic capable of unifying complex systems, and is currently present in several branches of science [5]. a network can be defined as a graph \(\mathcal {g}=(\mathcal {v, e})\), where \(\mathcal {v}\) is a set of nodes, \(\mathcal {v} = \{ v_1, v_2, ..., v_n \}\), and \(\mathcal {e}\) is a set of tuples, \(\mathcal {e} = \{ (i,j) : i,j \in \mathcal {v} \}\), representing the edges between each pair of nodes. one of the most evident characteristics in complex networks is the presence of communities. the notion of community is straightforward: each community is defined as a subgraph (or subnet) whose vertices are densely interconnected and, at the same time, these vertices have little connection with the rest of the network. a traditional way of detecting communities in a network is through the use of the modularity measure, which, broadly speaking, compares the number of connections between vertices which share a same characteristic with the expected number of connections when occurred randomly [7]. the fast greedy algorithm [9], for example, determines the optimal number of communities in the network by maximizing the modularity score of the graph. although there are currently already different graph-based methods for detecting anomalies [2], for this study, we have opted for developing and testing a novel one for the considered task. the network-based approach introduced in this work is inspired on a technique originally conceived for detecting periodicity in time series, such as in weather-related data [13]. the same technique had also already inspired the developing of a network-based model to identify up and down price trend patterns in the stock market, using real data [10, 11] and, in this study, we adapt it in order to detect anomalies in federal government payment cards (cpgf) data. we start by grouping the daily cpgf expenses data, in the form of weekly averaged values, considering only the days where there were at least one card transaction. afterwards, we calculate the variations in the weekly time series, using a sliding window, in the form of an array x. this array is then sorted in ascending order, by weekly variations, and split into n ranges with the same length l. in the original study [13], l is defined as \(\sqrt{t}\), where t is the total length of x and, in this work, we opt to set \(l=\root 3 \of {t}\), thus increasing the number of ranges n in the series, to better adapt the technique for the outlier detection task. these ranges can be represented by a list \(r = [r_0, r_1, r_2 . . . , r_{n-1}]\) ordered from low to high ranges. more specifically, each range \(r_q \in r\) is an interval defined as follows: $$\begin{aligned} r_q = \left\{ \begin{array}{ll} [\text{ min }(x), x_{[1/n]}] &{} \text{ if } q=0;\\ ]x_{[(q-1)/n]}, x_{[q/n]}] &{} \text{ if } q \in (1,\ldots ,n-2); \\ ]x_{[(q-1)/n]}, \text{ max }(x)] &{} \text{ if } q=n-1, \end{array} \right. \end{aligned}$$ (2) where \(x_{[qt]}\) denotes the empirical qt-quantile of x, \(qt \in [0,1]\). every range \(r_q \in r\) will be mapped as a vertex \(v_q \in \mathcal {v}\) in the network \(\mathcal {g(v, e)}\). for this end, we label every range of r as a sequence of integers \([0, 1, 2 . . . , n-1]\) in the same order they appeared in r, that will be used to represent the indexes of an adjacency matrix a, and also to store the variation ranges in r as node attributes in the set of nodes \(\mathcal {v}\), from network \(\mathcal {g}\). initially, a is considered a null matrix, of size \(n \times n\). two vertices \(v_q\) and \(v_w\) will become connected if there are two consecutive observations \(x_i\) and \(x_{i+1}\) in the series located in the different ranges \(r_q\) and \(r_w\), i.e., if they ever appeared consecutively in the series, chronologically. in this case, we have that \(a_{q,w} = a_{w,q} = 1\). the graph resulting from repeating this process to all pairs of consecutive observations in x is an undirected one. the overall procedure for generating the network is described in algorithm 1. in fig. 3, we provide an illustration showing the application of the network-based technique on a simple time series, comprising only 12 observed weekly values. in this case, the array is divided into 6 equal parts with two observations in each of them, i.e., \(\root 3 \of {12}\), creating variation ranges \(r = [r_0, r_1, r_2, r_3, r_4, r_5]\). these values are then mapped as nodes in the network \(\mathcal {g}\), with the links between them being generated according to whether they ever appeared consecutively in the original time series, pairwise. please note that, according to this rule, the node 4 is the one with highest degree in the network. fig. 3. example demonstrating how the variation ranges in the original dataset x are defined and then mapped as nodes in the network \(\mathcal {g}\), in the network-based approach. for the sake of simplification, there are only 12 weekly observations in this time series, and the variations are calculated based on the initial value, instead of on a sliding window. (a) after sorting the series by variation values, in ascending order, the array is split into 6 equal parts, hence delimiting the ranges \(r_0\) to \(r_5\). (b) each range will become a node in the network \(\mathcal {g}\), and two nodes will be connected only if they ever appeared consecutively in the original time series, chronologically. full size image the resulting network \(\mathcal {g}\) can be used to identify anomalies and spending oscillation patterns in the cpgf time series, based on the topological structure of the data. given that two vertices in the network \(\mathcal {g}\) will be connected by a link only if they are immediately next to each other, chronologically, at any point in the weekly time series, then the variation ranges in \(\mathcal {g}\) will indicate how distant a certain weekly spending value is from the weekly spending value from a certain number of weeks ago, used as sliding window, and the network topological structure will reflect the overall spending oscillation pattern for the cpgf data, with subsequent and similar variation ranges tending to be connected and closer to each other. in this way, most weekly variation values in x will be less or equal the highest range returned by the nodes in \(\mathcal {g}\). when there is an exception to this rule, i.e., when a spending data instance in x presents a weekly variation bigger than the variation ranges represented in \(\mathcal {v}\), then it will be labeled as an outlier in the results output. this procedure is described in algorithm 2. it is also worth mentioning that, since the variation ranges in r are generated based on a sliding window, there is the possibility of regulating the model’s sensitivity when detecting sudden or progressive spending increases, with smaller window values resulting in a higher sensitivity to detect possible anomalies over time. another useful feature of the proposed approach is in the fact that the communities detected in the resulting network \(\mathcal {g}\) can be used as some sort of spending variation “thermometer”, so to speak, such that by looking at which community a certain spending value belongs to, then one can use it as an indicator of how far its variation is from the spending average from a certain number of weeks ago, used as sliding window. 2.4 hybrid approach we also test a hybrid model in the anomaly detection task on cpgf data, consisting of taking the weekly averaged values and variation ranges generated by the network-based approach, pairwise, and grouping them using the k-means algorithm. for this end, as preprocessing, all variation values are converted to the logarithmic scale. in this approach, the best number of clusters k is achieved using the elbow method [4] as well, being defined as \(k=4\). to reduce the sensibility to the centroids starting condition, the k-means \(^{++}\) method is also used to adjust the initialization parameter. 3 experimental results in this section, we present the results obtained when applying the four techniques to detect anomalies in the cpgf data. we start by showing the results obtained from the k-means and agglomerative clustering techniques, in fig. 4. in both of these methods, the identification of points of attention as possible anomalies in the use of cpgf had been pointed out by specialists from the business area, who are familiar with the purchasing process in this modality, through the analysis of the generated plots. in this manner, the points identified in red are those that can be further investigated as possible cases of cpgf misuse. as one can observe, the clusters and anomalies returned by the two grouping approaches for the considered period are very alike, with the main difference between them being in the number of clusters generated. from the results obtained by both k-means and agglomerative clustering techniques, the specialists from the business area were able to identify a total of 10 transactions as possible cases of cpgf misuse for the army command, with two of them amounting to around r$ 30,000 each. as for the results obtained for the ministry of agriculture, the specialists were able to identify 5 transactions as suspected, with values around r$ 2,000 each. it is also worth noting that the clusters generated by these techniques were more influenced by the transaction values than by the type of expenditure, as can be observed by their flat shape in fig. 4. with regard to the network-based approach, we present, in fig. 5, the variations range networks resulting from the application of this method on the cpgf time series, using a 12-weeks sliding window, i.e., around 3 months of card transactions. each node in these networks corresponds to one variation range in the time series. for detecting communities in the network, we use the fast greedy algorithm [9]. both networks, in this case, present 3 communities, which can be used as some sort of “thermometer” to indicate how far a certain weekly value is from the weekly average from 12-weeks ago. in this way, for monitoring purposes, values from communities with higher variation ranges would trigger more attention than values from lower and in-between communities. fig. 4. results obtained with the k-means and agglomerative clustering techniques. the highlighted values indicate possible anomalies in the use of cpgf, identified by specialists from the business area. full size image in fig. 6, we present the results obtained with the network-based approach, showing the cpgf weekly time series, for each public agency, and respective outliers detected during the considered period. the main difference we note in these results, when compared to the ones provided by the k-means and agglomerative clustering methods, is that this method was able to point out, regardless of the identification from specialists in the business area, values considered outliers in the time series, based on the chosen 12-weeks sliding window variation. this feature is important as it can contribute to optimize the anomalies-identification process in this area, thus allowing the specialists team to have more focus on monitoring and investigation activities. in fig. 7, we show the weekly variations and the communities identified in the cpgf time series, still according to the chosen 12-weeks sliding window. as mentioned earlier, this plot, with the identification of communities, can be used as an alert indicator for certain values of the time series, where communities with higher variation ranges would raise the alert level for possible suspect transactions, with values increasingly becoming more distant from the average of 12 weeks ago. hence, this type of representation, which is also provided in the results from the network-based approach, can be helpful, as an additional graphical resource, in the activity of monitoring the cpgf transactions. fig. 5. variation ranges networks resulting from cpgf data of army command and ministry of agriculture. full size image fig. 6. results obtained with the network-based approach, using a 12-weeks sliding window. the points in black indicate possible anomalies in the use of cpgf returned by the model for the considered period. full size image fig. 7. weekly variations in the network-based approach. the colors denote the communities, that can be used as an indicator of how much the transaction values deviate from those of a certain number of weeks ago (12 weeks, in this case, which is the chosen sliding window). full size image finally, we show, in fig. 8, the results obtained with the hybrid approach, which consists of taking the information generated by the network-based method, i.e., the weekly averaged values and variation ranges, pairwise, and grouping them using the k-means algorithm. in this technique, the values highlighted can be considered outliers. however, in the same manner of the grouping models, this identification had been carried out by specialists from the business area. 4 final remarks in this work, four different unsupervised learning techniques were tested and evaluated on detecting anomalies in data from brazilian federal government purchase cards (cpgf). the obtained experimental results indicate that the k-means and agglomerative clustering methods were not able to provide a clear output, in terms of information to help in the identification of values that can configure possible misuse of the card. on the other hand, the network-based approach, also introduced in this study, demonstrated to be more successful in this task, as it was the only technique able to automatically infer some points of attention that can be used as a starting point for further investigations. the hybrid model obtained more clear results when compared to the first two clustering techniques, and has potential to be used as well in the initial process of identifying misuse of cpgf. it is important to note that the results obtained in the anomaly detection task, in this case, depend heavily on the definition of some factors inherent to the knowledge of the business areas from the controller general office, such as: what are the more suitable sliding window values to make comparisons, what are the more suitable grouping periods for transactions (weekly, biweekly etc.) and what would be the threshold, for each public agency, to define which payments worth a deeper investigation and which ones can be neglected. all of these parameters have proved to be relevant to the results obtained in this study. we believe that one of the strengths of this research lies in its social contribution aspect, especially when we consider that currently only a few amount of people know how to properly use the brazilian transparency portal and gather relevant information from this source. therefore, we expect that our work can contribute to improve this scenario, by bringing more attention to the importance of this tool for the population in general, in order to increase the government accountability level in brazil. finally, from the values identified as outliers in the models, especially those returned by the network-based and hybrid approaches, one can evolve to develop a more detailed measurement in order to ascertain whether in fact these identified transactions represent a misuse of the cpgf. in this way, the overall process could hence evolve to a semi-supervised learning approach and, later, to supervised models. fig. 8. results obtained with the hybrid approach. the highlighted values indicate possible anomalies in the use of cpgf, identified by specialists from the business area. full size image references abecs: brazilian association of credit card and services companies. www.abecs.org.br. accessed 2 july 2020 akoglu, l., tong, h., koutra, d.: graph based anomaly detection and description: a survey. data mining knowl. disc. 29(3), 626–688 (2015) article  mathscinet  google scholar  de andrade, p.h.m.a., meira, w., cerqueira, b., cruz, g.: auditing government purchases with a multicriteria anomaly detection strategy. j. inf. data manage. 11(1), 50–65 (2020) google scholar  bholowalia, p., kumar, a.: ebk-means: a clustering technique based on elbow method and k-means in wsn. int. j. comput. appl. 105(9) (2014) google scholar  bornholdt, s., schuster, h.g.: handbook of graphs and networks. from genome to the internet, willey-vch (2003 weinheim) (2001) google scholar  bradley, p.s., fayyad, u.m.: refining initial points for k-means clustering. in: icml, vol. 98, pp. 91–99. citeseer (1998) google scholar  brandes, u., et al.: on modularity clustering. ieee trans. knowl. data eng. 20(2), 172–188 (2007) article  google scholar  carcillo, f., le borgne, y.a., caelen, o., kessaci, y., oblé, f., bontempi, g.: combining unsupervised and supervised learning in credit card fraud detection. inf. sci. 557, 317–331 (2019) google scholar  clauset, a., newman, m.e., moore, c.: finding community structure in very large networks. phys. rev. e 70(6), 066111 (2004) google scholar  colliri, t., zhao, l.: a network-based model for optimizing returns in the stock market. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 645–650 (2019). https://doi.org/10.1109/bracis.2019.00118 colliri, t., zhao, l.: stock market trend detection and automatic decision-making through a network-based classification model. nat. comput. 1–14 (2021). https://doi.org/10.1007/s11047-020-09829-9 day, w.h., edelsbrunner, h.: efficient algorithms for agglomerative hierarchical clustering methods. j. classif. 1(1), 7–24 (1984) article  google scholar  ferreira, l.n., zhao, l.: detecting time series periodicity using complex networks. in: 2014 brazilian conference on intelligent systems, pp. 402–407. ieee (2014) google scholar  li, j., di, s., shen, y., chen, l.: fluxev: a fast and effective unsupervised framework for time-series anomaly detection. in: proceedings of the 14th acm international conference on web search and data mining, pp. 824–832 (2021) google scholar  likas, a., vlassis, n., verbeek, j.j.: the global k-means clustering algorithm. pattern recogn. 36(2), 451–461 (2003) article  google scholar  paula, e.l., ladeira, m., carvalho, r.n., marzagao, t.: deep learning anomaly detection as support fraud investigation in brazilian exports and anti-money laundering. in: 2016 15th ieee international conference on machine learning and applications (icmla), pp. 954–960. ieee (2016) google scholar  republica, p.: decreto 5.355 de 25 de janeiro de 2005. www.planalto.gov.br/ccivil03/ato2004-2006/2005/decreto/d5355.htm. accessed 7 may 2021 rezapour, m.: anomaly detection using unsupervised methods: credit card fraud case study. int. j. adv. comput. sci. appl. 10(11), 1–8 (2019) google scholar  da uniao, b.c.g.: portal da transparencia. gastos por cartoes de pagamento. www.portaltransparencia.gov.br/cartoes?ano=2019. accessed 27 june 2020 ward, j.h., jr.: hierarchical grouping to optimize an objective function. j. am. stat. assoc. 58(301), 236–244 (1963) article  mathscinet  google scholar  download references author information authors and affiliations brazilian office of the comptroller general (cgu), brasilia, brazil breno nunes department of computer science, university of fortaleza (unifor), fortaleza, brazil tiago colliri school of arts, science and humanities, university of sao paulo, sao paulo, brazil marcelo lauretto school of computer science, zhongyuan university of technology, zhengzhou, china weiguang liu faculty of philosophy, science and letters, university of sao paulo, ribeirao preto, brazil liang zhao authors breno nunesview author publications search author on:pubmed google scholar tiago colliriview author publications search author on:pubmed google scholar marcelo laurettoview author publications search author on:pubmed google scholar weiguang liuview author publications search author on:pubmed google scholar liang zhaoview author publications search author on:pubmed google scholar corresponding author correspondence to tiago colliri . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper nunes, b., colliri, t., lauretto, m., liu, w., zhao, l. (2021). anomaly detection in brazilian federal government purchase cards through unsupervised learning techniques. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_2 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords anomaly detection government purchase cards unsupervised learning complex networks. publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature encoding physical conditioning from inertial sensors for multi-step heart rate estimation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper encoding physical conditioning from inertial sensors for multi-step heart rate estimation conference paper first online: 28 november 2021 pp 58–72 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) encoding physical conditioning from inertial sensors for multi-step heart rate estimation download book pdf download book epub davi pedrosa de aguiar  orcid: orcid.org/0000-0003-2286-584110 & fabricio murai  orcid: orcid.org/0000-0003-4487-638110  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1260 accesses abstract inertial measurement unit (imu) sensors are present in everyday devices such as smartphones and fitness watches. as a result, the array of health-related research and applications that tap onto this data has been growing, but little attention has been devoted to the prediction of an individual’s heart rate (hr) from imu data, when undergoing a physical activity. would that be even possible? if so, this could be used to design personalized sets of aerobic exercises, for instance. in this work, we show that it is viable to obtain accurate hr predictions from imu data using recurrent neural networks, provided only access to hr and imu data from a short-lived, previously executed activity. we propose a novel method for initializing an rnn’s hidden state vectors, using a specialized network that attempts to extract an embedding of the physical conditioning (pce) of a subject. we show that using a discriminator in the training phase to help the model learn whether two pces belong to the same individual further reduces the prediction error. we evaluate the proposed model when predicting the hr of 23 subjects performing a variety of physical activities from imu data available in public datasets (pamap2, ppg-dalia). for comparison, we use as baselines the only model specifically proposed for this task and an adapted state-of-the-art model for human activity recognition (har), a closely related task. our method, pce-lstm, yields over 10% lower mean absolute error. we demonstrate empirically that this error reduction is in part due to the use of the pce. last, we use the two datasets (ppg-dalia, wesad) to show that pce-lstm can also be successfully applied when photoplethysmography (ppg) sensors are available, outperforming the state-of-the-art deep learning baselines by more than 30%. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a new approach to hr monitoring using photoplethysmographic signals during intensive physical exercise article 30 april 2021 a comparative analysis on sensor-based human activity recognition using various deep learning techniques chapter © 2021 a motion heart-rate monitor watch with improved grey differential equation model based on reflective photoplethysmography article 20 june 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. neural encoding neural decoding physiological psychology predictive medicine predictive markers sport training 1 introduction in the recent years there has been an ever increasing usage of sensor-equipped devices, such as smartphones, smartwatches and fitness watches. these sensors can be used to track user behavior and health-related measurements. among the most common types are the inertial measurement units (imu), composed primarily of accelerometers and gyroscopes.the use of photoplethysmography (ppg) sensors to track heart rate (hr) is also becoming ubiquitous, especially in devices targeting fitness conscious consumers, such as the apple watch, fitbit and samsung simband [1]. to promote an effective fitness training, it is necessary to induce an optimal cardiovascular response, making it essential to model and predict individual hr responses [2]. several methods have been designed to predict the hr under the influence of physical activity. most of them are based on differential equations [3, 4], hammerstein and wiener models [5] or neural networks. among the latter, some predict many steps into the future [6, 7], and others use imu signals as input  [7, 8] but only the model proposed in [7] does both, to the best of our knowledge. despite the recent advances in neural networks, the problem of multi-step hr estimation from imu sensor data remains little explored. in this paper, we investigate neural architectures that could be used for this task. to benchmark our model, in addition to using the model from [7], we adapt a network proposed for a closely related task called human activity recognition (har), where the goal is to identify the activity being performed by a person (e.g., running, walking, swimming) given some sensor data (mostly, imu) as input. recurrent neural networks (rnns) are widely used for multi-step predictions because they can carry information regarding the “hidden” state of a sequence through time as vectors. although the performance of rnns can be highly dependent on the initialization of these vectors, most often they are simply set to zero. sometimes, the rnns are set up so that the state vectors go through some iterations (washout period) before the first prediction can be returned. in the context of aerial vehicle dynamics, [9] proposes an approach to initialize these vectors by using features and labels from a previous interval. in a similar spirit, based on the premise that different subjects will display different hr levels when performing the same exercises depending on their physical conditioning, we propose a novel neural architecture that attempts to encode that attribute given imu and hr data collected from a previous, short-lived, activity performed by an individual. this attribute is extracted by a cnn network as a vector dubbed physical conditioning embedding (pce). the pce is used, in turn, to set the initial values of the hidden and cell states of an lstm responsible for outputting the hr predictions. our proposed method, pce-lstm, differs from [9] in several ways, markedly, (i) it uses a discriminator to improve the ability of the network to capture the inherent characteristics of the subjects based on previous activities and (ii) it is able to improve predictions over windows of up to 2 h. this corroborates the idea that pce-lstm is encoding physical conditioning, rather than information about specific physical activities. although ppg sensors for tracking hr are becoming more common, they are prone to measurement errors due to motion-related artifacts. in this case, data from other sensors can be used to correct the hr measurements. this task, called ppg-based hr estimation, has been explored in [1, 10, 11]. to show that the proposed architecture can also be used for this task, we adapt pce-lstm to incorporate the ppg signal as an additional input and show that it can also outperform the state-of-the-art in this task. in sum, our main contributions are: a new model to predict the hr from imu-sensor signals, which outperforms state-of-the art baselines; an ablation study on the contribution of the pce subnetwork to the performance of the model; a demonstration that pce-lstm estabilishes the new deep learning sota of ppg-based hr estimation.footnote 1 outline. section 2 describes datasets and pre-processing used in this work. section 3 details the proposed method, pce-lstm, and the key hypothesis we investigate. section 4 discusses the reference methods for the hr estimation task. section 5 presents the evaluation results. the related work is reviewed in sect. 6. last, sect. 7 discusses the significance and impact of this work. 2 methodology this work addresses the problem of predicting the heart rate \(h_t\) of an individual at time \(t=1,\ldots \), given imu sensor data gathered up to time t and hr values from an initial, short lived period, \(h_1 \ldots h_{i}\), with \(i < t\). we refer to this task as imu-based multi-step hr estimation. for this study, we use the pamap2 and ppg-dalia datasets (sect. 2.1), which are among the very few publicly available sets containing both imu and hr signals from individuals performing a variety of activities. ppg-dalia also has data from ppg sensors, which is disregarded in this first task. we address the case where ppg sensors are available (in addition to imu) as a secondary prediction task, referred as ppg-based multi-step hr estimation. since ppg data are hr measurements that can be perturbed by movement, this task consists of correcting such measurements based on imu data. for this task, we use the (complete) ppg-dalia [1] and wesad [12] datasets. we show that it is possible to adapt the architecture proposed for the first to the second task with minor changes. below we describe the datasets and the pre-processing techniques used throughout this work. fig. 1. time snippet representation full size image 2.1 datasets the pamap2 dataset. [13] consists of data from 40 sensors (accelerometers, gyroscopes, magnetometers, thermometers and hr sensor) sampled 100 hz of 9 subjects performing 18 different activities (e.g., rope jumping, running, sitting). there is a single time series of sensor signals per subject, each performing a sequence of activities. later on we explain that the time series for 1 of 9 the individuals is too short for training the models. the ppg-dalia dataset. [1] is composed of signals from two devices, a chest-worn device which provides accelerometer and ecg data; and a wrist-worn device measuring the ppg and triaxial acceleration, sampled 32 hz. it also includes hr series computed from the ecg signals. this dataset contains a contiguous time series of sensor signals from 15 individuals performing 8 activities. we dub dalia, the subset of this dataset which does not include the ppg signals (to use in the imu-based multi-step hr estimation task). the wesad dataset. [12] consists of data from 15 subjects wearing the same sensors available in ppg-dalia, but the individuals remain seated/standing during the whole study while going through different affective states (neutral, stress, amusement). unlike ppg-dalia, wesad does not provide precomputed hr series, therefore we used the heartpy library [14] to extract hr measurements from the ecg signals. although subjects are indexed up to number 17, the dataset does not include subject identifiers 1 and 12. in total, we use data from 23 (resp. 30) subjects for imu-based (resp. ppg-based) multi-step hr estimation task. 2.2 pre-processing basic preprocessing. we upsample the hr signal using linear interpolation in all datasets to make its sampling rate consistent with the other signals. only pamap2 contains a few missing data points, which we handle by local averaging the data around the missing point using a 0.4s window [15]. to make the use of the pamap2 and ppg-dalia datasets more consistent, we use only the accelerometer signals of the chest and wrist, and downsampled the signals 32 hz. all signals \(\mathbf {s}\) are z-normalized, i.e., \(\hat{\mathbf {s}} = (s \mu (\mathbf {s}))/\sigma (\mathbf {s})\), where \(\mu \) stands for the mean operator and \(\sigma \) for the standard deviation operator. time snippet discretization. like most works based on this data, we discretize the time series signals into time snippets (ts), i.e., partially overlapping windows of fixed duration \(\tau _{ts}\) and overlap ratio \(r_{ts}\) (task-dependent). figure 1 illustrates this procedure for the case when \(r_{ts}=0\). each time snippet \(ts_t\) is a matrix where each row represents a sensor. time snippets determine the granularity of the predictions. accordingly, we define the average hr \({h}_{t}\) for each time snippet \(\text {ts}_t\) as the response to be predicted. time series segmentation. in order to create a fixed-length training set, we segment the time series of each individual in a sequence of n contiguous time snippets of fixed duration. each segment is partitioned into two smaller segments. the first, called initialization segment, contains the imu and hr signals for the first \(i=12\) time snippets, and can be used by a neural network (nn) to encode a “state” specific to that time series. the second, called prediction segment, contains the imu, but does not contain the hr, as it is used by a (possibly different) nn to output predictions for each time snippet. figure 1 illustrates the subdivision of a segment, in the case where \(r_{ts}=0\). note that the nn used for processing the first segment can also be used for processing the second segment by replacing the hr signal in the latter by zeroes. 3 the physical conditional embedding lstm model here we describe pce-lstm, our proposed neural network architecture for hr prediction. pce-lstm is composed of convolutional and lstm layers, similarly to the state-of-the-art techniques for the closely related task of har. the main novelty of this model is discussed below. recurrent neural networks (rnns) have been especially designed to work with time series data. they are composed by cells with shared parameters, which process units of the input sequentially, using one or more vectors to carry state information through time. in particular, the unidirectional lstm (long short-term memory) cell uses two vectors – the hidden state and the cell state – which are received from the previous iteration, updated based on the input for that time and passed onto the next iteration. the first iteration, however, receives these vectors as they were initialized, typically as zero vectors. the implicit assumption is that the network will gradually be able to encode the correct state from the inputs as the vectors are passed through the cells. for hr prediction, we argue that if some data on the relationship between the input and output signals is available prior to the prediction, it can be beneficial to use a specialized network to encode this relationship as the rnn initial state. this initial state should contain information about physical conditioning: a more fit individual is able to sustain similar movement levels with smaller increase in hr. thus, the main hypothesis investigated here can be stated as hypothesis: it is possible to encode information about physical conditioning from an individual’s sensor data as a vector and use it as the initial state of a rnn to improve hr predictions. this is the rationale behind the main difference between our approach and similar ones, namely the initialization of the rnn hidden vectors using a specialized network, which we call the physical conditioning encoder. figure 2 shows the high level structure of our architecture, which is made of five components: the time snippet encoder (ts encoder), a convolutional network which encodes the 2-dimensional ts into a vector; the physical conditioning encoder (pc encoder), a convolutional network which extracts, from signals of the initialization segment (including hr), a physical conditioning embedding (pce) used for initializing the lstm’s hidden vectors; the discriminator, used to force the pce of a given subject to be similar across time segments; the state updater, a lstm which maintains and updates the subject’s state, encoded in its hidden vectors; and the prediction decoder, a fully-connected network which decodes the hr prediction from the state tracked by the state updater. we emphasize that instances of the same module shares the same weights. each component is described in detail below. fig. 2. pce-lstm (proposed architecture). full size image ts encoder: extracts features from one time snippet. we reckon the role of each sensor in the description of the intensity of an activity as being equivalent to role of each rgb channel in the description of a picture. hence, it is reasonable to combine them early in the nn. to do so, we use 1d convolutions along the time dimension, with the sensors stacked along the channel dimension. this approach differs from the most well known architectures in the literature [1, 16, 17] which keep the transformations on each sensor’s signals separate during the convolutional section of their architectures by the use convolutional kernels spanning a single sensor, only combining them later on. the ts encoder comprises multiple layers, each made of a 1d convolution followed by a leaky rectified linear unit activation and a dropout layer of rate tse\(_{\text {dropout}}\). after each layer, the tensor length \(\ell \) is reduced to \(\lfloor \ell / 2 \rfloor \) by filters of size 3 and stride 2 (when \(\ell \) is even, we use \(\text {padding}=1\)), except when the the tensor length is 2, in which case we use a filter of size 2 without padding. all layers have tse\(_{f}\) filters, except the last layer, which has tse\(_\text {out}\) filters. the number of layers is \(\text {tse}_{n}=\lfloor \log _{2}(\text {ts}_l)\rfloor \), where ts\(_l\) is the length of the time dimension of the ts, so as to transform the size of the time dimension to one. the vectors extracted from each time snippet of the initialization segment are concatenated along the time dimension, and the hr of each ts is concatenated along the feature dimension before being passed onto the next component. pc encoder: takes the vectors concatenated in the previous step and extracts a physical conditioning embedding (pce). it is a multi-layer convolutional network that transforms the 2d-input ( \((\text {tse}_{\text {out}}+1) \times i\)) into a single vector of length pce\(_\text {out}\). its convolutional architecture is designed with the same principles as the ts encoder. figure 2 shows the pc encoder in detail. from the pce, the lstm’s hidden state and cell state vectors are computed using a single linear layer each, represented in fig. 2 by fc\(_h\) and fc\(_c\). state updater: is a standard lstm with both state vectors (cell and hidden state’s) of size lstm\(_h\), and input of size tse\(_{\text {out}}\). these state vectors are initialized by the pc encoder using only signals from the initialization segment. the lstm is then fed with the deep attributes extracted by the ts encoder from each time snippet. prediction decoder: takes the hidden state representation for each time snippet from the state updater and computes the prediction from these representations. it is made of three fully connected layers, where the first two layers have 32 neurons, each followed by a relu activation function, and the last layer have lstm\(_h\) neurons, without activation function, outputting the predicted hr. we use the mean absolute error (\(\ell _1\) loss) as the cost function \(l_\text {hr}\) associated with this output because using the \(\ell _2\) loss hampers training as larger differences between the predicted and actual hr have an out-sized impact on the loss, according to our preliminary experiments. discriminator: given the benefits of multi-task learning we propose the simultaneous training of the model on a constructed task in which the pce is used to distinguish between individuals. assuming a subject’s physical conditioning as constant in the short term, we reason that employing a network to discriminate whether two pces belong to same person will foster better embeddings, when trained jointly with pce-lstmfootnote 2. to train the discriminator, for each segment in the training set, we sample another segment from the same individual with probability 0.50 and from a different individual with probability 0.50. for each pair, we measure the cross entropy as the loss function (\(l_d\)). we set the weight of the discrimination loss to 10% of the total loss.footnote 3 hence, the total loss \(l_\text {total}\) is given by \(l_\text {total} = 0.9 l_\text {hr} + 0.1 l_d \). this component is the head of a siamese network and is completely optional, being used exclusively to improve the performance of the model. the discriminator’s chosen architecture is comprised of 5 fully connected linear layers, each with 64 neurons and followed by relu activation function and a dropout rate of 0.15, except for the last layer, which uses a sigmoid activation function. this network receives two pces (concatenated) and outputs a probability. 3.1 adaptations for ppg-based hr estimation the task of ppg-based hr estimation has 3 main distinctions from the imu-only hr estimation and hence requires a few adaptations to our method. first, the ppg signal is best represented in the frequency domain; second, the ppg signal is more important than the other signals since it is a rough estimate of the hr; third, for ppg-based hr estimation, a ground truth hr is not usually available. to deal with these differences, we modified the ts encoder sub-network to comprise two ts encoder architectures, one for the raw ppg and imu signals (\(\text {tse}_{\text {raw}}\)) and another for the fast fourier the transformed ppg signal (\(\text {tse}_{\text {ppg-fft}}\)). these outputs are concatenated and returned by the new sub-network. the \(\text {tse}_{\text {ppg-fft}}\) is smaller, with \([\text {tse}_{\text {ppg-fft}}]_{\text {out}} = 12\). we also modified the pc encoder slightly, so that it only receives as input the concatenated outputs of the tse (without the hr). figure 3 illustrates the changes to the pce-lstm architecture for the secondary task. 4 reference methods to the best of our knowledge, the task of predicting multiple steps of hr given imu signals has only been attempted by [7], which used in their experiments a dataset not publicly available. in addition to using their model as a baseline, we also adapt a model [17] designed for similar tasks (har) in order to benchmark our method. in this section we describe each of these models, the task they were designed to address and the minor changes required to adapt the models for the task at hand. more recent architectures for the har task, such as [16], were not used as baselines because they were not designed for multi-step predictions over time series and, as such, would require significant changes. fig. 3. ppg-adapted pce full size image ffnn. [7] is a feed-forward recursive architecture with skip connections using data from a wrist-worn 3-axial accelerometer. the model uses the average measurement of each sensor in a non-overlapping window of 30 s. as the architecture details were not reported, we set the layer size to 16 and relu as the activation function for each layer based on random search optimization. we adapted their architecture, using a time window of \(\tau _{\text {ts}}=4\) s, to make it more comparable to our method (testing with \(\tau _{\text {ts}} = 30\) s as in the original work yielded worse results). we train the network using adam, instead of the genetic algorithms used in that study. deepconvlstm. [17] performs a sequence of convolutions on the input series. the deep features from each time entry feed a lstm. from each lstm hidden vector, a prediction is computed using a single linear layer. in our adapted version, we add padding to the convolutions to maintain the length of the tensor and select only the outputs corresponding to the last time input of each time snippet, as we have one label per time snippet. as in [17], we downsampled the input signals 30 hz, and set time snippets’ length to \(\tau _{\text {ts}} = 3s\). 5 empirical evaluation in this section we describe the experiments conducted to evaluate pce-lstm and their results. we begin with an overview of the experimental setup. next, we present the imu-based hr prediction experiments (the primary prediction task) and an ablation study on the pce’s impact on performance. last, we describe the ppg-based hr estimation (the secondary prediction task). 5.1 experimental setup we evaluate the imu-based multi-step hr estimation results w.r.t. mean absolute error (mae), root mean squared error (rmse) and maximum error (\(\ell _1\)) as evaluation metrics. for the ppg-based hr estimation, however, we consider only mae, since we transcribe the results from the paper where the baseline method was proposed [1], which did not include another evaluation metric. train-test split. we create several train-test splits using a “leave one subject out strategy”, which mimics a realistic setting where the model would be applied to individuals not contained in the training set, as is the case for most models trained offline [18]. in a dataset with s subjects, each of the subjects is used once as the test subject and the remaining \(s-1\) subjects’ time series are split into training segments of \(n=50\) time snippets each and then randomly assigned to train/validation sets using an 80/20 split. for test, we use the whole series of each subject, which represents around one or two hours for most subjects in the pamap2 and dalia datasets respectively. with each of the s subjects as the test subject, we perform 7 executions, with different train/validation splits and different neural network weight initialization, as done in [1]. hyperparameter tuning. in order to choose pcm-lstm’ and the optimizer’s (adam) hyperparameters, we applied a random search using the results on pamap2’s subject 5 as reference, following [17]. the following hyperparameters were selected: pce\(_f = 64\), \(i = 12\), \(\tau _{ts} = 4\), \(r_{ts} = 0.5\), \(\text {tse}_{\text {out}} = 128\), \(\text {tse}_f = 16\), \(\text {lstm}_h = 64\), \(\text {pce}_{\text {out}} = 64\), \(\text {tse}_{\text {dropout}} = 0.15\), learning rate \(= 0.005\), weight decay \(= 0.00005\). training setup. all the models are optimized with adam using the \(\ell _1\) loss as the cost function associated with the hr predictions. in each epoch, we compute the validation loss. after training is complete, we load the model weights that yielded the lowest validation loss. training was done using a batch size of 64 over 100 epochs for pce-lstm and deepconvlstm. ffnn showed slower convergence and hence was trained for 200 epochs. subject 9 of the pamap2 dataset was not included in the analysis because the corresponding time series is shorter than the length of the training segment (102 s). mean vs. ensemble performance. each method is trained 7 times for each test subject (using different train-validation splits). we compute a “mean” performance by averaging the errors of individual models and an “ensemble” performance by averaging the predictions and then computing the resulting error. table 1. error metrics on the dalia dataset (best shown in bold)full size table 5.2 imu-based multi-step hr estimation we begin by analyzing the performance metrics. table 1 and 2 shows the mean and the ensemble performances w.r.t. mae by test subject and mae averages for pce-lstm and the baselines, on dalia and pamap2 respectively. due to space limitations, we only report \(\ell _1\) and rmse averages, but the complete results can be found in our technical report [19]. we note that the series of subjects 5 and 6 of dalia dataset can be regarded as outliers as none of the methods performed well on them. as expected, ensembles tend to outperform their standalone counterparts. considering the ensemble performances, out of 23 subjects, ffnn, deepconvlstm and pce-lstm achieve the lowest errors for 2, 4 and 17 subjects, respectively. the lowest average error is obtained by the pce-lstm ensemble, with over 11.5% lower mae and 7.3% lower rmse than the next best method. in relation to the maximum error, our method was over 18% better than the next best method in the dalia dataset, but just slightly better than the second best on the pamap2 dataset. table 2. error metrics on pamap2 (best shown in bold)full size table for a qualitative evaluation, we plot the predictions of each model. figure 4 shows the ensemble predictions for the complete series of five representative test subjects in the dalia and pamap2 datasets. we observe that deepconvlstm fails to capture the variance of the hr series: although the predictions are correlated with changes in hr, they tend to remain close to an “average” hr. ffnn, in turn, exhibits more variance, but cannot accurately capture the amplitude of the peaks and sometimes overestimates the hr. in contrast, pce-lstm (our method) can capture both peaks and valleys more accurately than the baselines. 5.3 performance impact of pce-lstm’s hidden state initialization in this section we conduct an ablation study to demonstrate that pce-lstm’s strategy for initializing hidden state vectors is key to boosting the model’s performance. for conciseness, we limit this study to the dalia dataset, which contains nearly twice the number of subjects, and a longer series per subject. a typical strategy to extract a hidden state is to use the lstm itself to encode the relationship between sensor data and hr. in contrast, our model initializes the lstm’s hidden state vectors by passing the sensor and hr data corresponding to the first \(i=12\) time snippets (26 s) of the series through the physical conditioning encoder (pc encoder). as we assume that the pce partially encodes an individual’s physical conditioning, we expect that the joint training of the pce-lstm regression and the discriminator will improve the regression predictions by promoting a better training of the pc encoder subnetwork. fig. 4. long-term imu-based hr estimation full size image to quantify the performance impact of the proposed initialization and of using the discriminator, we conduct additional experiments with alternative initialization strategies. in what follows, “with discr.” indicates the joint training with the discriminator, “without discr.” indicates that the discriminator is not used; and “lstm self-encode”, indicates that hidden state vectors are initialized by feeding the hr to the network as an additional input channel in the ts of the initialization segment (replaced by zeros during the prediction segment). table 3. ablation: error results for variations of the pce-lstm model on daliafull size table table 3 shows the prediction results for these experiments on dalia. pce-lstm’s outperforms an lstm self-encode initialization strategy even without a discriminator, but using the discriminator further reduces the error. 5.4 ppg-based hr estimation we adapt pce-lstm for ppg-based hr estimation. while similar to the imu-based hr estimation, the former task has been better explored in the literature and has, therefore, well established baselines. for this reason, we compare pce-lstm to the sota method based on deep learning [1]. this method is based on cnns and will be referred simply as cnn. to match the length and step size of each time snippet in [1], we set \(\tau _{\text {ts}} = 8\) and \(r_{\text {ts}} = 0.75\) for this task. table 4 shows the ensemble performance for each method on ppg-dalia and wesad, when the individual designated in the column is the test subject. the results of cnn were transcribed directly from [1]. the last column shows the row average. on both datasets, pce-lstm provides error reductions of approximately 32% when compared to the sota method. another advantage of pce-lstm is that it has 2 orders of magnitude less parameters than the cnn – approx. 120k parameters for pce-lstm vs. approx. 8.5m parameters for the cnn model. table 4. ppg-based hr estimation experimentsfull size table 6 related work few studies used imu sensors to predict the heart rate. [8] used a simple feed-forward neural network to predict the hr one step ahead, given its value and the average signal of each imu sensor on the previous step. quite similarly, [7] performed a multi-step hr prediction by repeatedly using the hr computed for step t to predict the hr for step \(t+1\). their experiments demonstrated promising results, but had some notable deficiencies: they used data from a single individual in his daily activities, hence without much variation in hr values. also on multi-step hr estimation, but using speed and acceleration as inputs, [6] proposed a bayesian combined predictor, where one estimator was a linear regression and the other, a neural network similar to that of [7]. in their experiments, data from multiple individuals performing running sessions was used, thus addressing some of the shortcomings in [7]. however, the proposed method required calibration with actual hr data every 90 s, hindering its practical use. for har, a task where the use of imu data is widespread, [20] studied the performance of deep feed-forward networks (dnns), cnns and lstms, demonstrating that cnn and lstm based models outperform dnns and, among them, the best performing model was very dependent on the dataset used. the hybrid cnn and lstm model proposed by [17], achieved good results but was surpassed by the cnn-based architecture of [16]. more recently, some works proposed the use of self-attention based architectures for this task [21, 22]. the model proposed by [16] outperformed the one proposed by [21] in the datasets that were common to both works. [22] proposed a deep attention model that was able to surpass [16] in some, but not all datasets. a more extensive survey was conducted by [23], where the authors note the variety of neural network architectures proposed for this task, such as cnns, dnns, rnns, stacked auto encoders and hybrid approaches. a few studies have proposed methods for initializing state vectors of rnns. to regularize and improve the dynamics of an rnn, [24] proposed initializing the rnn with a vector proportional to the backpropagated errors. [25] showed that using contextual information to initialize the rnn hidden state improved the convergence rate, but not the final accuracy of the networks on some constructed problems. in modeling an aerial vehicle dynamics, [9] showed that passing an initial segment of the data through a custom network to set the hidden state vectors reduces errors, especially in the first iterations of predictions. our work differs from theirs in several ways. most markedly, (i) it uses a discriminator to improve the ability of the network to capture the subjects’ inherent characteristics based on previous activities and (ii) it is able to improve predictions over windows of up to 2 h. this corroborates the idea that pce-lstm is encoding physical conditioning, rather than information about specific physical activities. 7 conclusions we investigated the much neglected task of predicting hr from imu sensor data. we started from the premise that, depending on their physical conditioning, different people will display different hr levels when performing the same exercises. we proposed a neural architecture dubbed physical conditioning embedding lstm (pce-lstm) that employs a convolutional network to extract vectors which carry information about the relationship between sensor measurements and the hr for a specific individual, thus representing his/her physical conditioning. these vectors are used as the initial state vectors for a lstm network that outputs hr predictions from sensor data. we evaluate the prediction accuracy of pce-lstm w.r.t. mae and rmse using public datasets (pamap2, ppg-dalia, wesad). moreover, we compare pce-lstm with 2 baselines: one model proposed for this task (ffnn) and one minimally-adapted state-of-the-art model (deepconvlstm) originally proposed for the closely related task known as human activity recognition. pce-lstm yields over 10% (resp. 30%) lower mae in the imu-based (resp. ppg-based) hr estimation task. last, we conduct additional experiments to show that the performance gains achieved by our method are, in part, due to the strategy used to initialize the hidden vectors. specifically, using the outputs of the pce network applied to the data from the previous 12 time snippets of the subject’s time series is helpful and works better than using the lstm itself to output hidden vectors from the same data, especially when using the pce discriminator during training. we have highlighted the potential of neural network approaches to tackle this problem. some questions are left open for future work, such as whether these results are accurate enough to be successfully used in real-world applications or whether the metrics used best represent the suitability of a model. we hope that this work can foster the development of new methods and metrics for this task, bringing about a practical use for the method in medical or fitness applications. notes 1.we released all our code at https://github.com/davipeag/heartrateregression. 2.we avoid overfitting by using disjoint sets of subjects for training and validation. 3.alternatively, losses’ weights can be set by hyperparameter tuning, but since we use a single subject for validation, we fixed the weights to (0.9, 0.1) to avoid overfitting to the validation subject. references reiss, a., indlekofer, i., schmidt, p., van laerhoven, k.: deep ppg: large-scale heart rate estimation with convolutional neural networks. sensors 19(14), 3079 (2019) article  google scholar  ludwig, m., hoffmann, k., endler, s., asteroth, a., wiemeyer, j.: measurement, prediction, and control of individual heart rate responses to exercise-basics and options for wearable devices. front. physiol. 9, 778 (2018) article  google scholar  cheng, t.m., savkin, a.v., celler, b.g., wang, l., su, s.w.: a nonlinear dynamic model for heart rate response to treadmill walking exercise. in: ieee iembs, pp. 2988–2991 (2007) google scholar  hunt, k.j., fankhauser, s.e.: heart rate control during treadmill exercise using input-sensitivity shaping for disturbance rejection of very-low-frequency heart rate variability. biomed. signal process. control 30, 31–42 (2016) google scholar  mohammad, s., guerra, t.m., grobois, j.m., hecquet, b.: heart rate control during cycling exercise using takagi-sugeno models. ifac proc. 44(1), 12783–12788 (2011) google scholar  zhang, h., wen, b., liu, j.: the prediction of heart rate during running using bayesian combined predictor. in: ieee iwcmc, pp. 981–986 (2018) google scholar  xiao, f., yuchi, m., ding, m., jo, j.: a research of heart rate prediction model based on evolutionary neural network. in: ieee icbmi, pp. 304–307 (2011) google scholar  yuchi, m., jo, j.: heart rate prediction based on physical activity using feedforwad neural network. in: ieee ichit, pp. 344–350 (2008) google scholar  mohajerin, n., waslander, s.l.: multistep prediction of dynamic systems with recurrent neural networks. 30(11), 3370–3383 (2019) google scholar  salehizadeh, s., dao, d., bolkhovsky, j., cho, c., mendelson, y., chon, k.: a novel time-varying spectral filtering algorithm for reconstruction of motion artifact corrupted heart rate signals during intense physical activities using a wearable photoplethysmogram sensor. sensors 16(1), 10 (2016) article  google scholar  schäck, t., muma, m., zoubir, a.m.: computationally efficient heart rate estimation during physical exercise using photoplethysmographic signals. in: eusipco, pp. 2478–2481 (2017) google scholar  schmidt, p., reiss, a., duerichen, r., marberger, c., van laerhoven, k.: introducing wesad, a multimodal dataset for wearable stress and affect detection. in: acm icmi, pp. 400–408, new york, ny, usa (2018). association for computing machinery google scholar  reiss, a., stricker, d.: introducing a new benchmarked dataset for activity monitoring. in: ieee iswc, pp. 108–109, june 2012 google scholar  van gent, p., farah, h., van nes, n., van arem, b.: analysing noisy driver physiology real-time using off-the-shelf sensors: heart rate analysis software from the taking the fast lane project. j. open res. softw. 7(1) (2019) google scholar  eyobu, o.s., han, d.s.: feature representation and data augmentation for human activity classification based on wearable imu sensor data using a deep lstm neural network. sensors 18(9), 2892 (2018) google scholar  rueda, f.m., grzeszick, r., fink, g.a., feldhorst, s., hompel, m.t.: convolutional neural networks for human activity recognition using body-worn sensors. informatics 5(2), 26 (2018) google scholar  ordóñez, f.j., roggen, d.: deep convolutional and lstm recurrent neural networks for multimodal wearable activity recognition. sensors 16(1), 115 (2016) google scholar  jordao, a., nazare jr, a.c., sena, j., schwartz, w.r.: human activity recognition based on wearable sensor data: a standardization of the state-of-the-art. arxiv e-prints, page arxiv:1806.05226, june 2018 de aguiar, d.p., murai, f.: am i fit for this physical activity? neural embedding of physical conditioning from inertial sensors (2021) google scholar  hammerla, n.y., halloran, s., plötz, t.: deep, convolutional, and recurrent models for human activity recognition using wearables. in: ijcai, pp. 1533–1540 (2016) google scholar  vaswani, a.: attention is all you need. in: neurips, pp. 5998–6008 (2017) google scholar  wu, n., green, b., ben, x., o’banion, s.: deep transformer models for time series forecasting: the influenza prevalence case. arxiv e-prints, page arxiv:2001.08317, january 2020 wang, j., chen, y., hao, s., peng, x., lisha, h.: deep learning for sensor-based activity recognition: a survey. patt. recogn. lett. 119, 3–11 (2019) article  google scholar  zimmermann, h., grothmann, r., schaefer, a., tietz, c.: 8 modeling large dynamical systems with dynamical consistent neural networks. in: haykin, s., principe, j.c., sejnowski, t.j., mcwhirter, j. (eds.) new directions in statistical signal processing: from systems to brains, chapter 8. the mit press (2006) google scholar  wenke, s., fleming, j.: contextual recurrent neural networks (2019) google scholar  download references author information authors and affiliations universidade federal de minas gerais, belo horizonte, brazil davi pedrosa de aguiar & fabricio murai authors davi pedrosa de aguiarview author publications search author on:pubmed google scholar fabricio muraiview author publications search author on:pubmed google scholar corresponding author correspondence to davi pedrosa de aguiar . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de aguiar, d.p., murai, f. (2021). encoding physical conditioning from inertial sensors for multi-step heart rate estimation. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_5 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords heart rate estimation imu sensors photoplethysmography neural networks publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature leveraging sign language processing with formal signwriting and deep learning architectures | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper leveraging sign language processing with formal signwriting and deep learning architectures conference paper first online: 12 october 2023 pp 299–314 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) leveraging sign language processing with formal signwriting and deep learning architectures download book pdf download book epub fernando de almeida freitas  orcid: orcid.org/0009-0008-4937-90979, sarajane marques peres  orcid: orcid.org/0000-0003-3551-64809, otávio de paula albuquerque  orcid: orcid.org/0000-0002-2504-37079 & … marcelo fantinato  orcid: orcid.org/0000-0001-6261-14979  show authors part of the book series: lecture notes in computer science ((lnai,volume 14197)) included in the following conference series: brazilian conference on intelligent systems 558 accesses abstract advances in sign language processing have not adequately kept pace with the tremendous progress that has been made in oral language processing. this fact serves as motivation for conducting research on the potential utilization of deep learning models within the domain of sign language processing. in this paper, we present a method that utilizes deep learning to build a latent and generalizable representation space for signs, leveraging formal signwriting notation and the concept of sentence-based representation to effectively address sign language tasks, such as sign classification. extensive experiments demonstrate the potential of this method, achieving an average accuracy of \(81\%\) on a subset of 70 signs with only 889 training data and \(69\%\) on a subset of 338 signs with 3, 871 training data. the authors of this work would like to thank the center for artificial intelligence (c4ai-usp) and the support from the são paulo research foundation (fapesp grant #2019/07665-4) and from the ibm corporation. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others deep learning-based sign language recognition and translation chapter © 2023 application of deep learning techniques on sign language recognition—a survey chapter © 2021 from rule-based models to deep learning transformers architectures for natural language processing and sign language translation systems: survey, taxonomy and performance evaluation article open access 29 august 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. language processing machine learning natural language processing (nlp) sign languages symbolic ai artificial intelligence 1 introduction the world health organization has projected that approximately 700 million individuals will experience moderate-to-severe hearing loss by 2050, with a majority being elderly due to aging population [38]. hearing loss brings challenges to the daily lives of individuals. the challenge is even more difficult for deaf babies born into families that are not familiar with sign language, as it is essential for them to gain literacy in sign language to facilitate appropriate cognitive and socioemotional development, as well as equitable and effective communication [38]. besides acquiring sign language, access to diverse forms of knowledge throughout their lives is essential for their intellectual and civic development. deaf individuals are inevitably placed in social environments where oral language predominates. although they can still use a gesture-visual language to communicate, acquiring information, whether routine or formal, is primarily accomplished through the use of the oral language writing system. this system is based on the transcription of the phonological aspects of oral language. due to this discrepancy, the acquisition of written proficiency in the local spoken language by deaf individuals is not only complex, but it also does not promote linguistic reflection in their first language. furthermore, this discrepancy prevents the complete dissemination of knowledge to this segment of the population. the availability of automatic translation tools between sign languages and oral languages could facilitate the social inclusion of the deaf community and enhance their access to knowledge. however, despite recent advances in natural language processing, tasks related to the automatic processing of sign languages still remain a challenging problem. one of the major obstacles in this task is the scarcity of labeled datasets [26]. although sign language videos are currently available in significant volume, labeling them is a laborious task and requires refined linguistic knowledge [25]. furthermore, the most modern resources for automatic resolution of tasks related to linguistic systems are based on the principle of a word sequence-based language. this aspect prevents the use of modern techniques of language processing on sign language due its organization based in gestural elements superimposed in a three-dimensional space. one alternative to overcome this gap is to use a writing system for sign languages, such as the well-known signwriting. developed by valerie sutton [33], signwriting is a universal notation for recording sign languages that uses the international signwriting alphabet 2010 to represent the parameters of sign languages. as a result, it provides a basis for writing gesture-visual signs. using a writing system for sign language opens up an avenue for using the transformer architecture [23, 35] a neural network architecture widely used for language modeling and solving various natural (oral) language tasks. in this paper we present a framework developed for using lexicons of sign language, described through signwriting, as input for deep learning architectures. we trained deep learning models using public datasets that contain sign descriptions in signwriting, and tested the resulting model on the classification task of signs from three sign languages: american, brazilian, and german. we contend that our work makes significant contributions to the field of automatic sign language processing. we propose a new conceptual model for sign language processing. this model creates a latent representation space for signs, and supports the development of solutions for tasks such as translation. the representation space is applied in classifier implemented with convolutional neural network and transformer architectures. we have made pre-trained model checkpoints publicly availablefootnote 1 to the research community, which can accelerate the development of solutions for various sign language processing tasks. additionally, as part of the conceptual model development process, we devised three mappings for adequate signwriting descriptions for using in deep learning architectures. this paper is organized as follows: sect. 2 provides a theoretical background; sect. 3 presents an overview of the related work; sect. 4 discusses the dataset design and the method proposal; sect. 5 presents the experimental results along with a discussions about them; sect. 6 presents the final remarks. 2 background 2.1 sign language natural languages have structure based on small, meaningless building blocks (smallest units called phonemes) and bigger meaningful building blocks (morphemes and lexemes that make up words). despite an unlimited number of meanings that a natural language can create, the limitation of the number of minimum units makes a descriptive study of languages possible [19]. sign languages are natural languages assumed to be primary for deaf individuals. this languages employ signs composed of hand configurations and movements combined with facial expressions and body postures to communicate meaning, as opposed to the spoken word used in oral languages [31]. for instance, the signs for “little house”, “house” and “big house” in brazilian sign language are built using a specific hand configuration carried out in different locations and applying different facial expressions (cf. fig. 1). fig. 1. examples of signs for “little house”, “house” and “big house” in brazilian sign language [1]. full size image notwithstanding common misconceptions, sign languages are not word-for-word translations of spoken languages. for instance, the relative sentence in the oral language portuguese “a garota que caiu de bicicleta? ... ela está no hospital!!”footnote 2 is uttered in brazilian sign language by using appropriate signs in the following sequence: <menina caiu bicicleta> ... <ela lá hospital>footnote 3. while there is a consensus among scholars regarding the notion of a finite number of minimal units generating an infinite number of meanings, there is no academic agreement on the standardized approach to describe and organize signs. this issue arises from the various signwriting-based annotation practices applied worldwide [5]. 2.2 writing systems for sign language and the signwriting the written recording of languages in a permanent and standardized form is essential for reflecting on communication content, recording memories, and thinking language itself, enabling the advancement and improvement of thought and cognitive processes. the absence of records makes it challenging to organize social and cultural activities within a society, leading to distortions and incomplete information about memories and traditions [8]. thereby, sign writing systems were proposed over the years. the oldest writing system was developed by auguste bébian in 1825. such a system, known as “mimographie”, has 190 symbols that describe five constituent parameters of a sign [4]. in 1965, stokoe published a more concise writing system specific to the american sing language consisting of 50 symbols [31]. based on stokoe’s notation, françois xavier neve proposed, in 1996, a system that uses characters from western writing as a way of representing the minimum units of sign languages [24]. the hamburg sign language notation system (hamnosys), created in 2004, is also based on stokoe’s system, but with more iconic symbols to facilitate its use with computational techniques [16]. finally, writing in sign language (elis) was created by the mariângela estelita barros in 2008, bringing the concept of finger configurations [3]. the writing system of interest in this paper is signwriting, developed by valerie sutton [33], mainly because it has the widest availability of collaborative databases, with adherence to deaf communities around the world, among all notations [30]. this is a universal notation for sign languages that represents the parameters of sign languages through a set of visual symbols hierarchically classified. it comprises seven categories, with 30 groups of symbols, and 652 icons called “visographemes”. each icon has varying degrees of freedom to describe a sign. they can be rotated on their central axis, mirrored, and contextualized with other parts of the sign, generating 35.023 unique symbols. the categories of symbols used in signwriting are: 1. hand shape: symbols to represent possibilities of finger articulation; 2. movements: symbols used to represent contact and small finger movements; 3. dynamics/timing: symbols use to describe intensity or movement cadence; 4. head & face: symbols used to represent head position, head/neck/facial parts movements, and facial expressions; 5. body: symbols used to represent torso/shoulders/limbs/hips movements; 6. location: contains 8 symbols to represent where, in the 3d-space, the sign is performedfootnote 4; 7. punctuation: symbol used when writing complete sentences. figure 2 depicts some examples of how to write the sign “rain” and “strong rain” in signwriting, considering the composition of the sign in american sign language. notice that the same sign can be written in multiple ways, and there exists the possibility of representing varying levels of intensity. fig. 2. examples of how to write “rain” and “strong rain” in signwriting, considering the composition of the sign in american sign language [33]. full size image formal signwriting. to facilitate the collection and dissemination of signwriting content, valerie sutton developed the signbank databasefootnote 5, which serves as a collaborative platform for users worldwide. the database encourages users to contribute by adding new vocabulary entries and content written in signwriting. among the available contents, there are dictionaries that link signs written in signwriting with their corresponding meanings utilizing words or expressions from the oral language of the respective country. within this database, all inputs are encoded using formal signwriting, a standard symbol representation system. formal signwriting employs ascii characters within a regular expression format, incorporating special tokens and \(x,y-\)coordinates to describe and position symbols within a designated area known as the signbox. tokens are organized in four groups: structural makers, the base symbol ranges, modifier indexes, and the numbers. the method for sign language processing introduced in this paper requires the use of formal signwriting for symbol representation. below, we present the regular expressionfootnote 6 employed in such formalism: $$ \begin{array}{c} (s[123][0-9a-f]{2}[0-5][0-9a-f][0-9]{3}x[0-9]{3}) \end{array} $$ the symbols have a unique identification, consisting of six characters, with the first character always being the letter s. the next three characters represent the base of a symbol; the fourth character identifies the filling of this symbol, which represents the plane in 3-dimensional space parallel to the palm of the hand, and the last character represents the rotation of the symbol. each symbol has x, y coordinates within the singbox. optionally, the regular expression can be used to describe a sequence of symbols, meaning that signs have modifications to their structure during their execution. this notation uses the letter “a” in place of the letter “s” in the symbol key definition. 2.3 deep learning basics convolutional neural network architecture: according to [15], the term convolutional neural network signifies the network’s utilization of the convolution operation, which proves particularly advantageous for processing data with grid-like structures like time series and images. the appeal of this neural network architecture stems from its capacity to extract meaningful features while disregarding the exact feature locations and focusing instead on their relative positions in relation to other features. the network comprises consecutive layers of convolution and pooling, culminating in a dense feedforward neural network. each convolution layer consists of multiple feature maps generated by sets of synapses sharing the same weights. maps are constructed through convolution followed by flattening carried out by sigmoid or relu functions [15, 17]. pooling layers follow each convolutional layer, reducing the feature map resolution through statistical operations. the final dense neural network is responsible for learning the mapping of features extracted by the preceding layers to a predefined set of classes corresponding to the input data labels. transformer architecture: in the field of natural language processing, the transformer [35] architecture has emerged as a prominent model renowned for its performance in text processing tasks. transformer utilizes a neural network with an encoder-decoder [23] architecture. considering the translation task, the network receives as input a sentence comprising words from the source natural language, encodes this information using attention layers, and subsequently decodes the encoded information, yielding a sentence translated into the target natural language. in this architecture, the so-called attention layers were introduced, which are responsible for identifying the importance of words as processing occurs. to accomplish this objective, every word is abstracted into vectors within the attention layer, and undergo processing through matrices of synaptic weights. consequently, an internal parameterizable dimension encoding is obtained. subsequently, a weighted normalization process is carried out using a softmax activation function. the weighting mechanism determines the significance of each encoding position. 3 related work previous attempts at automatic sign language processing for recognition, synthesis, translation, and application development relied on classification models using machine learning or hybrid approaches (using language parsers and rule-based reasoning) [2, 9, 12, 14, 26]. these efforts were restricted to specific languages and problems due to limitation of techniques and methods, and lack of labeled data [26]. recent advances in natural language processing have addressed these challenges, enabling end-to-end solutions and efficient representation learning, mainly using writing systems for sign language, mitigating the scarcity of labeled data and promoting transfer learning. this section lists recent work that uses deep learning architectures in the context of sign language processing. the initial endeavors aimed to enhance the automated processing of sign language by utilizing signwriting descriptions primarily focused on german sign language signs [13, 21], in 2012 and 2013. this initiative introduced an automated annotation process for sign subunits. later, in 2015, the same research group employed the rwth-phoenix-weather dataset to train machine learning models for classification tasks [20]. in 2019, the description of signs was used to optimize bert-type models, enabling the construction of a representation space [6]. subsequently, this space was combined with visual information, encompassing hand and body movements, to facilitate classification tasks. this research involved the use of bert models, convolutional neural networks, and long short-term memory networks for processing space-time information. in 2020, camgoz et al. [7] introduced embeddings to represent space-time information in frame sequences, using data from the montalbano ii [11], msr daily activity 3d [36], cad-60 [32] and ntu-60 datasets [29]. deep neural networks were employed to extract features, while textual information was provided as inputs in a bert model. both representations were used in a transformer training for sign language recognition. in the same year, li et al. [22] combined unstructured data (rgb images), bert-based models, and spatial information from finger joints for solving the problem of classifying signs over the word-level american sign language dataset. several researchers employed efforts to apply deep learning to sign language processing in 2021 [10, 18, 27, 34, 39]. word-level american sign language dataset was again used in the study of tunga, nuthalapati and wachs [34]. in this case, convolutional graph network (gcn) and bert model were combined to receive human poses as input and solve the glosses identification task. deep neural networks were studied for feature extraction by rastgoo et al. [27]. these authors combined the result of this representation learning with other traditionally used attributes (distances and angles). bert attention layers were studied by coster et al. [10] and proved to be effective in allowing the zero-shot classification process in the task of translating sign languages. pre-trained models of bert-base and mbart-50 were applied to the task of translating sign language videos to german text. a new model called signbert was proposed by zhou, tam and lam [39], using bert in conjunction with resnet to extract spatial information and apply it to the translation task. in this study, pose information in skeleton formats and hand configurations as inputs in the bert strategy. the (signbert) was also used by hu et al. [18] to compose another model, based on hand configurations and the sequence of gesture execution. the method presented in this paper draws inspiration from related work but offers novel contributions in the following aspects: leveraging information from collaborative sources for training deep learning models, such as utilizing public data from platforms like wikipedia to generate language models; introducing schema for mapping coding in formal signwriting for sentence-like representation, providing a strategy to create a versatile latent representation space applicable to various sign languages. 4 method figure 3 illustrates the method proposed herein. the objective of this method is to establish a latent representation space for signs in a sign language, enabling efficient processing by deep learning models, such as for constructing classifier models. in this scenario, the classifier model takes a sign instance as input and outputs the corresponding concept (or word) represented by that sign. fig. 3. proposed method for sign language processing: generation of a representation space and its application for constructing classifier models. full size image the top section of fig. 3 (above the dotted line) outlines the process of constructing the representation space. first, a collection of signs recorded in dictionaries is subjected to preprocessing to create a labeled dataset (sign representation \(\rightarrow \) concept). then, this representation is mapped to sentence-based structures and passed through a customized tokenizer. this tokenizer generates tensors for the sentences, which constitute the latent representation space. the custom tokenizer was trained using the regular expression from formal signwriting, which generates a customized vocabulary of tokens. the lower section of the figure illustrates the process of constructing classification models. instances within the representation space are organized into datasets for both training and testing purposes, which are then utilized to build and evaluate classifier models. the remainder of this section details the dataset used, the process of mapping the formal signwriting representation to a sentence-based representation, and implementation and evaluation strategies. 4.1 datasets the signbank database portal has content related to more than 80 sign languages. for the purpose of our research, three dictionaries of signs were usedfootnote 7: german sign language with 24, 777 entries, american sign language with 11, 647 entries, and brazilian sign language with 45, 412 entries. the dictionary entries form the vocabulary of signs being processed in our approach. each dictionary entry corresponds to a sign description representing a concept. multiple signs can be associated with the same concept (cf. example shown in fig. 2) and, not rarely, there are identical entries in the dictionaryfootnote 8. in addition, the dictionaries present some noise referring to data curation failures. during the data preprocessing stage, duplicated descriptions were removed, as well as noises were excluded. multiple signs associated with the same concept allowed for the creation of distinct subsets of data for model training and testing. these subsets are mutually exclusive in terms of descriptions and signs but share common concepts. this approach enables the evaluation of the deep learning model’s ability to learn the patterns of concept descriptions. for instance, consider the formal and logographic descriptions illustrated in fig. 4 for the concept “know”. the training set can consist of descriptions (a) to (e), while descriptions (f) and (g) are allocated to the test set. fig. 4. formal descriptions and logographic representations for the concept “know” using signwriting [33], considering the signs used in american sign language. full size image table 1 lists the number of minimum entries available in each dictionary when we consider the existence of a specific number of different descriptions for the same concept. each row in the table represents a situation for training and testing a model. for example, the second row of the table shows the count of concepts represented and the number of entries selected in each dictionary when the condition is to have a minimum of two unique descriptions for each concept. in row number 12, the criterion is set to have at least 12 distinct descriptions. in the last scenario, the american sign language dictionary meets this requirement only for 13 concepts and only 197 entries are available for building training and testing datasets. the first line counts the total concepts and entries of each dictionary after the data preprocessing stage. table 1. configuration of datasets for training and testing models. german german sign language; asl american sign language; libras brazilian sign language; #desc. number of different descriptions for the same concept.full size table 4.2 data mapping considering that transformer-based solutions for natural language processing operate on word sequences (sentences) as input, we present three schema to map formal signwriting descriptions to sentence-based representations, involving sequences of meaningful units. we proposed three different mapping schema: mapping m1: this scheme only represents the sequence of symbols, for instance, for representing a sign “know” in fig. 4a, the mapping m1 generates the following sequence: s15a11 s20600 s30007. mapping m2: the sentence provided by this mapping schema embeds an additional set of descriptive information about the sign. such a set of information comprises: what elements are involved in the sign description: hand, movement, dynamic, head, body or location; indication if there is variation in the hand configuration of the right hand (notation: staticityrtrue) and of left hand (notation: staticityltrue); indication of the number of hands used during sign signaling (notation: hands_qtt0, hands_qtt1 or hands_qtt2). the mapping m2 for a sign “know” generates the following sequence: head s300 s30007. hand right s15a s15a11. movement s206 s20600. staticityrtrue. staticityltrue. hands_qtt1. mapping m3: in this schema, the palm orientation and rotation information are presented separately. in accordance with the mapping m3 schema to describe the signs, we can represent the corresponding letters in fig. 4 as follows: (a) head s300 s3000 s30007. hand right s15a s15a1 s15a11. movement s206 s2060 s20600. staticityrtrue. staticityltrue. hands_qtt1. (b) movement s206 s2060 s20600. hand right s182 s1821 s18210. head s300 s3000 s30007. staticityrtrue. staticityltrue. hands_qtt1. (c) head s300 s3000 s30007. hand right s182 s1821 s18210. movement s205 s2050 s20500. staticityrtrue. staticityltrue. hands_qtt1. (d) movement s206 s2060 s20600. hand right s182 s1821 s18210. head s300 s3000 s30006. staticityrtrue. staticityltrue. hands_qtt1. (e) head s321 s3210 s32101. hand left s100 s1001 s10018. hand right s15a s15a1 s15a11. staticityrtrue. staticityltrue. hands_qtt2. (f) head s36a s36a0 s36a00. movement s205 s2050 s20500. hand left s182 s1824 s18248. staticityrtrue. staticityltrue. hands_qtt1. (g) head s36a s36a0 s36a00. hand left s182 s1821 s18218. movement s206 s2060 s20600. staticityrtrue. staticityltrue. hands_qtt1. 4.3 implementation details this project implemented the tokenizer method from distilbert [28] based-model using the huggingface transformers library [37]. the vocabulary generated from the regular expression used in formal signwriting (cf. sect. 2 and fig. 3) is concatenated with the words used in the sentences (hand, movement, dynamic, head, body, location, punctuation, stacityrtrue, stacityrfalse, stacityltrue, stacitylfalse, hands_qtt0, hands_qtt1, hands_qtt2), and with the special tokens [pad], [unk], [cls], [sep], and [mask]. the custom vocabulary has 73, 743 tokens. the transformer architecture used for contruction of classifier models was also developed using the huggingface transformers library [37]. the implementation of the convolutional neural network architecture was accomplished using libraries and frameworks provided by tensorflowfootnote 9. 4.4 classifier evaluation the evaluation of the classifier models involved analyzing the impact of the number of concepts (classes) on their performance. this choice of analysis perspective is justified by the dynamic and distributed nature of sign languages. due to the visual-spatial nature of these languages, signs can undergo numerous variations when used by different user communities. moreover, the continuous development of new signs reflects the need to incorporate corresponding signs for emerging concepts in societal evolution. thus, the models were generated using varying numbers of descriptions per concept (from one to eleven). we present the accuracy achieved through stratified cross-validation testing. 5 experiments and results table 2 presents the optimal outcomes achieved using the convolutional neural network architecture. the table provides the minimum, maximum, and average accuracies (along with their standard deviations) obtained through a comprehensive cross-validation process. notably, the highest performance was observed in folds with smaller number of descriptions and concepts, indicating that while the experiment yielded promising, there exists an upper limit in terms of the number of concepts these models can effectively recognize. the libras sign language models exhibited the lowest results, despite having a larger amount of available data. this outcome can be attributed to the experimental strategy employed, which involved a higher number of concepts for this language. further experiments focusing on a smaller set of concepts are necessary to determine whether additional complexities exist in processing this specific sign language. table 2. results for models using convolutional neural network. #d. number of different descriptions for the same concept; \(\mu \) average; \(\sigma \) standard deviationfull size table the results achieved using the transformer architecture are presented in table 3. on average, the results are slightly lower compared to the convolution-based architecture. while better performance in terms of maximum results was attainable for german and american sign languages, the model exhibited instabilities as indicated by the brazilian sign language results. table 3. results for models using transformer architecture. #d. number of different descriptions for the same concept; \(\mu \) average; \(\sigma \) standard deviationfull size table regarding the representation schemes, we observed a slight superiority for the mapping m3 schema when applying convolutional architecture, and for the mapping m1 and mapping m2 schemes when applying the transformer architecture. these findings underscore the necessity for further investigations to harness the learning capabilities of each architecture, as they appear to encompass distinct features in the representation spaces. 6 final remarks in this paper, we investigate a new sign language modeling approach based on signwriting descriptions. the obtained classifier models demonstrate the ability to recognize signs based on detailed and unprecedented descriptions of their execution. our method exhibits adaptability across sign languages from various countries, indicating the potential for utilizing a representation scheme that bridges the gap between signwriting coding and sentence-based representation in automatic sign language processing research. as future work, exploration of approaches involving texts and sentences in signwriting could be conducted, enabling the incorporation of contextual information in sign execution. this would facilitate the development of models for various tasks within the same domain. it is worth emphasizing that these models still require significant improvement, as their current capacity does not encompass a wide range of concepts. therefore, conducting a detailed investigation into the errors that arise when introducing a larger set of concepts to the model can provide valuable insights for enhancing the latent representation space. notes 1.https://github.com/fernandoafreitas/bracis_2023_signwriting. 2.translation to english: “the girl who fell from bike? ... she is in the hospital!”. 3.mapping for english words: <girl fell bike> ... <she there hospital>. 4.position symbols were not found in the databases used in this paper. 5.https://www.signbank.org/. 6.detailed information about the regular expression is available at https://datatracker.ietf.org/doc/html/draft-slevinski-signwriting-text-05#section-2.3. 7.accessed on 2023-01-31. 8.the same sign can be represented in multiple ways, either due to slight variations in the execution of the gesture or by taking contextual factors into account during the signaling process. 9.https://www.tensorflow.org/. references de almeida freitas, f.: reconhecimento automático de expressões faciais gramaticais na língua brasileira de sinais. master’s thesis, universidade de são paulo, brasil (2015) google scholar  de almeida freitas, f., peres, s.m., de moraes lima, c.a., barbosa, f.v.: grammatical facial expressions recognition with machine learning. in: proceedings of the 27th international florida artificial intelligence research society conference, pp. 180–185 (2014) google scholar  barros, m.e.: elis-escrita das línguas de sinais: proposta teórica e verificação prática. ph.d. thesis, tese (doutorado em linguística)-universidade federal de santa catarina (2008) google scholar  bébian, a.: mimographie, ou essai d’écriture mimique propre á régulariser le langage des sourds-muets. l. colas (1825) google scholar  bertoldi, n., et al.: on the creation and the annotation of a large-scale italian-lis parallel corpus. in: proceedings of 7th international conference on language resources and evaluation, valletta, malta, pp. 19–22. european language resources association (2010) google scholar  bilge, y.c., ikizler-cinbis, n., cinbis, r.g.: zero-shot sign language recognition: can textual data uncover sign languages? arxiv preprint arxiv:1907.10292 (2019) camgoz, n.c., koller, o., hadfield, s., bowden, r.: sign language transformers: joint end-to-end sign language recognition and translation. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 10023–10033 (2020) google scholar  capovilla, f., raphael, w., viggiano, k., neves, s., luz, r.: sign writing: implicações psicológicas e sociológicas de uma escrita visual direta de sinais, e de seus usos na educação do surdo. revista espaço 33–39 (2000) google scholar  de araújo cardoso, m.e., peres, s., de almeida freitas, f., venância barbosa, f., de moraes lima, c.a., hung, p.: automatic segmentation of grammatical facial expressions in sign language: towards an inclusive communication experience. in: proceedings of the 53rd hawaii international conference on system science, pp. 1499–1508 (2020) google scholar  de coster, m., et al.: frozen pretrained transformers for neural sign language translation. in: proceedings of the 1st international workshop on automatic translation for signed and spoken languages, pp. 88–97. association for machine translation in the americas (2021) google scholar  escalera, s., et al.: multi-modal gesture recognition challenge 2013: dataset and results. in: proceedings of the 15th acm on international conference on multimodal interaction, pp. 445–452. acm, new york (2013) google scholar  farooq, u., rahim, m.s.m., sabir, n., hussain, a., abid, a.: advances in machine translation for sign language: approaches, limitations, and challenges. neural comput. appl. 33(21), 14357–14399 (2021) article  google scholar  forster, j., et al.: rwth-phoenix-weather: a large vocabulary sign language recognition and translation corpus. in: international conference on language resources and evaluation, istanbul, turkey, vol. 9, pp. 3785–3789. european language resources association (2012) google scholar  freitas, f.a., peres, s.m., lima, c.a., barbosa, f.v.: grammatical facial expression recognition in sign language discourse: a study at the syntax level. inf. syst. front. 19, 1243–1259 (2017) article  google scholar  goodfellow, i., bengio, y., courville, a.: convolutional networks. in: deep learning, vol. 2016, pp. 330–372. mit press, cambridge (2016) google scholar  hanke, t.: hamnosys-representing sign language data in language resources and language processing contexts. in: 4th international conference on language resources and evaluation, vol. 4, pp. 1–6 (2004) google scholar  haykin, s.: neural networks and learning machines, 3rd edn. pearson, london (2009) google scholar  hu, h., zhao, w., zhou, w., wang, y., li, h.: signbert: pre-training of hand-model-aware representation for sign language recognition. in: proceedings of the ieee international conference on computer vision, pp. 11087–11096 (2021) google scholar  karnopp, l.b.: aquisição fonológica na língua brasileira de sinais: estudo longitudinal de uma criança surda. ph.d. thesis, universidade federal do rio grande do sul (ufrgs) (1999) google scholar  koller, o., forster, j., ney, h.: continuous sign language recognition: towards large vocabulary statistical recognition systems handling multiple signers. comput. vis. image underst. 141, 108–125 (2015) article  google scholar  koller, o., ney, h., bowden, r.: may the force be with you: force-aligned signwriting for automatic subunit annotation of corpora. in: 10th ieee international conference and workshops on automatic face and gesture recognition, pp. 1–6. ieee (2013) google scholar  li, d., opazo, c.r., yu, x., li, h.: word-level deep sign language recognition from video: a new large-scale dataset and methods comparison. in: proceedings of the ieee winter conference on applications of computer vision, pp. 1459–1469 (2020) google scholar  luong, m.t., pham, h., manning, c.d.: effective approaches to attention-based neural machine translation. in: proceedings of the 2015 conference on empirical methods in natural language processing, lisbon, portugal. acl (2015) google scholar  nève, f.x.: essai de grammaire de la langue des signes française, vol. 271. librairie droz (1996) google scholar  polat, k., saraclar, m.: unsupervised term discovery for continuous sign language. in: proceedings of the 9th workshop on the representation and processing of sign languages: sign language resources in the service of the language community, technological challenges and application perspectives, marseille, france, pp. 189–196. european language resources association (2020) google scholar  rastgoo, r., kiani, k., escalera, s.: sign language recognition: a deep survey. expert syst. appl. 164, 113794 (2021) article  google scholar  rastgoo, r., kiani, k., escalera, s., sabokrou, m.: multi-modal zero-shot sign language recognition. arxiv preprint arxiv:2109.00796 (2021) sanh, v., debut, l., chaumond, j., wolf, t.: distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arxiv preprint arxiv:1910.01108 (2019) shahroudy, a., liu, j., ng, t.t., wang, g.: ntu rgb+ d: a large scale dataset for 3d human activity analysis. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1010–1019 (2016) google scholar  stiehl, d., addams, l., oliveira, l.s., guimarães, c., britto, a.: towards a signwriting recognition system. in: 13th international conference on document analysis and recognition, pp. 26–30. ieee (2015) google scholar  stokoe, w.c.: sign language structure. annu. rev. anthropol. 9(1), 365–390 (1980) article  google scholar  sung, j., ponce, c., selman, b., saxena, a.: unstructured human activity detection from rgbd images. in: ieee international conference on robotics and automation, pp. 842–849. ieee (2012) google scholar  sutton, v.: signwriting. sl: sn, p. 9 (2009) google scholar  tunga, a., nuthalapati, s.v., wachs, j.p.: pose-based sign language recognition using gcn and bert. in: ieee winter conference on applications of computer vision workshops, pp. 31–40 (2021) google scholar  vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, vol. 30 (2017) google scholar  wang, j., liu, z., wu, y., yuan, j.: mining actionlet ensemble for action recognition with depth cameras. in: ieee conference on computer vision and pattern recognition, pp. 1290–1297. ieee (2012) google scholar  wolf, t., et al.: huggingface’s transformers: state-of-the-art natural language processing. arxiv preprint arxiv:1910.03771 (2019) world health organization: world report on hearing. technical report, world health organization (2021) google scholar  zhou, z., tam, v.w., lam, e.y.: signbert: a bert-based deep learning framework for continuous sign language recognition. ieee access 9, 161669–161682 (2021) article  google scholar  download references author information authors and affiliations school of arts, sciences and humanities, university of são paulo, são paulo, brazil fernando de almeida freitas, sarajane marques peres, otávio de paula albuquerque & marcelo fantinato authors fernando de almeida freitasview author publications search author on:pubmed google scholar sarajane marques peresview author publications search author on:pubmed google scholar otávio de paula albuquerqueview author publications search author on:pubmed google scholar marcelo fantinatoview author publications search author on:pubmed google scholar corresponding author correspondence to fernando de almeida freitas . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de almeida freitas, f., peres, s.m., de paula albuquerque, o., fantinato, m. (2023). leveraging sign language processing with formal signwriting and deep learning architectures. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14197. springer, cham. https://doi.org/10.1007/978-3-031-45392-2_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45392-2_20 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45391-5 online isbn: 978-3-031-45392-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords sign language processing formal signwriting deep learning convolutional neural network transformer publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature on the generalizations of the choquet integral for application in frbcs | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper on the generalizations of the choquet integral for application in frbcs conference paper first online: 28 november 2021 pp 498–513 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) on the generalizations of the choquet integral for application in frbcs download book pdf download book epub giancarlo lucca10, eduardo n. borges  orcid: orcid.org/0000-0003-1595-767610, rafael a. berri  orcid: orcid.org/0000-0002-5125-275610, leonardo emmendorfer  orcid: orcid.org/0000-0002-6950-394710, graçaliz p. dimuro  orcid: orcid.org/0000-0001-6986-988810,11 & … tiago c. asmus  orcid: orcid.org/0000-0002-7066-715610,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13073)) included in the following conference series: brazilian conference on intelligent systems 762 accesses abstract an effective way to cope with classification problems, among others, is by using fuzzy rule-based classification systems (frbcss). these systems are composed by two main components, the knowledge base (kb) and the fuzzy reasoning method (frm). the frm is responsible for performing the classification of new examples based on the information stored in the kb. a key point in the frm is how the information given by the fired fuzzy rules is aggregated. precisely, the aggregation function is the component that differs from the two most widely used frms in the specialized literature. in this paper we provide a revision of the literature discussing the generalizations of the choquet integral that has been applied in the frm of a frbcs. to do so, we consider an analysis of different generalizations, by t-norms, copulas, and by f functions. also, the main contributions of each generalization are discussed. supported by pnpd/capes (process. 464880/2019-00), fapergs (19/2551-0001279-9, 19/2551-0001660), cnpq (301618/2019-4), the spanish ministry of science and technology (tin2016-77356-p, pid2019-108392gb i00 (aei/10.13039/501100011033)). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an alternative to power measure for fuzzy rule-based classification systems chapter © 2020 a fuzzy reasoning method based on ensembles of generalizations of the choquet integral chapter © 2020 application of the sugeno integral in fuzzy rule-based classification chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms functional clustering knowledge based systems special functions stochastic integrals integral transforms and operational calculus 1 introduction a classification problem [1] is a research field in the area of data mining [2], which can be tackled in two different ways. an approach to deal with this problem is known as supervised learning, where a function (classifier) is generated from the available and labeled data (classes). then, when a new example needs to be classified, the learned classifier is responsible to perform the prediction. in the literature it is possible to find several methods that aim to cope with these problems using supervised learning, such as support vector machines (svm) [3], decisions trees [4] and neural networks [5]. here, the focus is on fuzzy rule-based classification systems (frbccs) [6], because they provide the user with interpretable models by using linguistic labels [7] in their rules. another reason is related with their accurate results and versatility, as shown in the many different fields where they have been applied like health [8], security [9], economy [10], food industry [11]. an important role in any frbcs is played by the fuzzy reasoning method (frm) [12]. this method is responsible to perform the classification of new examples. for that, it makes usage of the information available in the rule base and the database. moreover, in order to perform the classification, this mechanism uses an aggregation operator in order to aggregate, by classes, the information provided by the fired fuzzy rules when classifying new examples. a widely used frm considers the function maximum as aggregation operator. by using this aggregation function, for each class, the frm performs the selection of the best fired rule since it has the highest compatibility with the example [13]. the issue of this inference method is that the information provided by the remainder fired fuzzy rules is ignored. the maximum is an averaging aggregation operator, since the obtained result is within the range between the minimum and the maximum of the aggregated values (in this case, obviously, the result is always the maximum). to avoid the problem of ignoring information, it was proposed a frm that applies the normalized sum [12] to perform the aggregation of the available information given by the fired rules. in this way, for each class, all information is taken into account in the aggregation step. this aggregation operator is considered as non-averaging since the result of this function can leave the range minimum–maximum. in [14] the authors introduced a frm considering the usage of the choquet integral (ci) [15], which is an averaging operator. in this way, this approach mixes the characteristics of the previous frms considering an averaging operator that uses the information provided by all the fired rules of the system. moreover, the ci is defined in terms of a fuzzy measure, which provides it with the nice properties to take into account the interaction among the data to be aggregated [15]. the objective of this paper is to discuss different methodologies that change the aggregation step performed in the frm, when considering different generalizations of the ci, which are supported by solid theoretical studies [16], varying from the generalization by t-norms (\(c_t\)-integrals) [17], by copulas (cc-integrals [18,19,20]) and functions f(\(c_f\)-integrals [21] and \(c_{f1f2}\)-integrals [22, 23]). moreover, for each generalization it is provided provide a discussion of the main obtained results of each study (we highlight that our focus here are related with the main conclusions and not the specific obtained results of each approach). this paper is organized as follows. section 2 present the main components of a frbc, showing an example of how the aggregation function is used in this context. sections 3–6 discuss the theoretical and applied contributions of different generalizations of the ci. section 8 is the conclusion. 2 the role of aggregation functions in the frm fuzzy rule-based classification systems (frbcss) [6] are extensions of the rule-based system by using fuzzy sets in the antecedents of the rules. the best-known frbcss are the ones defined by takagi-sugeno-kang (tsk) [24] and mamdani [25], which is the one that it is adopted. the standard architecture of the mamdani method is presented in fig. 1. fig. 1. a structure of frbcs of the mamdani type. full size image where the knowledge base (kb) is composed by: data base (db) – stores the membership functions associated with the linguistic labels considered in the fuzzy rules. rule base (rb) – is composed by a collection of linguistic fuzzy rules that are joined by a connective (operator and). here we consider that a classification problem ins composed by t training patterns \(x_p = (x_{p1}, \ldots , x_{pm}), p = 1, 2, \ldots , t.\) where \(x_{pi}\) is the i-th attribute and with the rules having the following structure: $$\begin{aligned} \text{ rule } \; r_j : \text{ if } \, x_1\, \text{ is }\, a_{j1} \, \text{ and } \,\ldots \, \text{ and } \, x_n \, \text{ is } \, a_{jn} \,&\\ \text{ then } \, \text{ class } \text{ is } \, c_j \, \text{ with }\, rw_j, \nonumber \end{aligned}$$ (1) where \(r_j\) is the label of the j-th rule, \(a_{ji}\) is a fuzzy set modeling a linguistic term, modeled by a triangular shaped function. \(c_j\) is the class label and \(rw_j \in [0, 1]\) is the rule weight [26]. the fuzzyfication interface converts the inputs (real values) into fuzzy values. in case of categorical variables, each value is modeled by a singleton and, consequently, its membership value is either 1 or 0. once the input is fuzzified, the inference process is the mechanism responsible for the use of the information stored in the kb to determine the class in which the example will be classified. the generalizations discussed in this paper are applied at this point. once the knowledge has been learnt and a new example \(x_p = {x_{p1}, \ldots , x_{pn}}\) has to be classified, the frm [27] is applied to perform this task, where m is the number of classes of the problem and l is the number of rules that compose the rb. the stages of the frm are: matching degree: it represents the importance of the activation of the if-part of the rules for the example to be classified \(x_p\), using a t-norm as conjunction operator: $$\begin{aligned} \mu _{a_{j}}(x) = t(\mu _{a_{j1}}(x_{1}), \ldots , \mu _{a_{jn}}(x_{n})). \end{aligned}$$ (2) with \(j = 1,\ldots , l.\) and \(\mu _{a_{j1}}\) as the membership function with relation to a membership function. association degree: for each rule, the matching degree is weighted by its rule weight: $$\begin{aligned} b^{k}_{j} (x)= \mu _{a_{j}}(x)\cdot rw^{k}_{j}, \end{aligned}$$ (3) with \(k = class(r_{j})\) and \(j=1, \ldots , l\). example classification soundness degree for all classes: for each class k, the positive information \(b_j^k(x) > 0\), given by the fired fuzzy rules of the previous step, is aggregated by an aggregation function \(\mathbb {a}\): $$\begin{aligned} s_{k} (x)= \mathbb {a}_{{k}} \left( b^{k}_{1}(x), \ldots , b^{k}_{l}(x) \right) , \end{aligned}$$ (4) with \(k = 1, \ldots , m.\) in what follows, three different well-known frms are presented. observe that their main difference is in the use of a different aggregation function to perform the aggregation of the information provided by the rules: winning rule (wr) – for each class, it only considers the rule having the maximum compatibility with the example. $$\begin{aligned} s_{k} (x) = \underset{r_{j_k} \in rb;}{\max }b_j(x). \end{aligned}$$ (5) additive combination (ac) – it aggregates all the fired rules, for each class k, by using the normalized sum. $$\begin{aligned} s_{k} (x)= \displaystyle \frac{\sum _{j = 1}^{r_{j_k} \in rb}{b_j(x)}}{f_{1_{max}}}, \end{aligned}$$ (6) where \(f_{1_{max}} = \max _{k=1,\ldots ,m} \sum _{j = 1}^{r_{j_k} \in rb}{b_j(x)}\). the choquet integral (ci) – it is the function \(\mathfrak {c}_{\mathfrak {m}} : [0,1]^n \rightarrow [0,1]\), defined, for all of \(\boldsymbol{x} \in [0,1]^n\), by: $$\begin{aligned} \mathfrak {c}_{m} (\boldsymbol{x}) = \sum _{i=1}^{n} \left( x_{(i)} x_{(i-1)} \right) \cdot m\left( a_{(i)} \right) , \end{aligned}$$ (7) where \(n=\{0,\ldots , n\}\), \(\mathfrak {m}: 2^{n} \rightarrow [0,1]\) is a fuzzy measurefootnote 1, \(\left( x_{(1)}, \ldots , x_{(n)}\right) \) is an increasing permutation on the input \(\boldsymbol{x}\), that is, \(0 \le x_{(1)} \le \ldots \le x_{(n)}\), with \(x_{(0)} = 0\), and \(a_{(i)} = \{(i), \dots , (n) \}\) is the subset of indices corresponding to the \(n-i+1\) largest components of \(\boldsymbol{x}\). then: $$\begin{aligned} s_{k} (x)= \sum _{j = 1}^{r_{j_k} \in rb}\mathfrak {c}_{\mathfrak {m}}(b_j(x)). \end{aligned}$$ (8) where \(\mathfrak {c}\) is the standard ci and \(\mathfrak {m}\) the fuzzy measure. classification: for the final decision, the class that maximizes all the example classification soundness degrees is considered, using the function \(f:[0,1]^m \rightarrow \{1, \ldots , m\} \): $$\begin{aligned} f (s_{1}, \dots , s_{m}) = arg~\max _{k=1,\ldots ,m}(s_{k}). \end{aligned}$$ (9) to exemplify the role of different aggregation operator in the frm, consider a classification problem composed by 3 classes (\(c_1\), \(c_2\) and \(c_3\)). for each one, 3 generic fuzzy rules, \(r_a\), \(r_b\) and \(r_c\) are fired when classifying a new example (they can be different for each class). we present the information about this problem in table 1. notice that the numbers in this table represent the positive association degree (step 2 of the frm) obtained for each fired rule. having into account that three fuzzy rules are fired for each class, by columns, three aggregations have to be computed (one for each class). table 1. association degrees for each class.full size table since the ci is defined with respect to a fuzzy measure, in this example the standard cardinality (see [28]) is considered as fuzzy measure. the values computed for each class using these three frms are the following ones: c\(_1\) \(\bullet \): wr = 0.94 \(\bullet \): ac = \(\frac{0.94\,+\,0.1\, +\, 0.25}{2.62}\) = 0.49 \(\bullet \): choquet = \(((0.1 0)\) \(\frac{3}{3}\))  +  \(((0.25 0.1)\) \(\frac{2}{3}\)) + \( ((0.94 0.25)\) \(\frac{1}{3}\)) = 0.43 c\(_2\) \(\bullet \): wr = 0.4 \(\bullet \): ac = \(\frac{0.15\,+\, 0.4 \,+\, 0.1}{2.62}\) = 0.24 \(\bullet \): choquet =0 \(((0.1 0)\) \(\frac{3}{3}\))  +  \(((0.15 0.1)\) \(\frac{2}{3}\))  +  \(((0.4 0.15)\) \(\frac{1}{3}\)) = 0.21 c\(_3\) \(\bullet \): wr = 0.89 \(\bullet \): ac = \(\frac{0.89\,+\, 0.88\, +\, 0.85}{2.62}\) = 1.0 \(\bullet \): choquet = \(((0.85 0)\) \(\frac{3}{3}\))  +  \(((0.88 0.85)\) \(\frac{2}{3}\))  +  \(((0.89 0.88)\) \(\frac{1}{3}\)) = 0.87 once the example classification soundness degree for each class has been computed, the predicted class is the one associated with the largest value (step 4 of the frm): wr = arg \(\max \)[0.94, 0.4, 0.89] = c\(_1\) ac = arg \(\max \)[0.49, 0.24, 1.0] = c\(_3\) choquet = arg \(\max \)[0.43, 0.21, 0.87] = c\(_3\) it is observable that the usage of the maximum as an aggregation operator predicts class 1, since it only considers the information provided by one fuzzy rule (having the maximum compatibility). however, if we look in detail at the association degrees presented in table 1, this prediction may not be ideal, since that class 1 has one rule having high compatibility whereas class 3 has three rules having high compatibilities (slightly less than that of class 1). then, class 3 seems to be the most appropriated option. this fact is taken into account by the ci and the ac, since the information given by all the fuzzy rules and not only by the best one is considered and, consequently, the prediction assigns class 3. in this example, it is noticeable the non-averaging behavior of ac. observe that the result of this function for class c\(_3\) is greater than the maximum value. this fact does not occur for averaging functions. in the case of wr, the result is always the maximum, meanwhile for the ci the result is a value between the minimum and the maximum. another interesting point that raises with this example, is that the usage of different aggregation in the frm is directly related with the performance of the classifier. 3 the \(c_t\)-integral and pre-aggregations this study was originally based on [14], where the authors modified the frm of the chi et al. algorithm [29] by applying the ci to aggregate all available information for each class. furthermore, they introduced a learning method using a genetic algorithm in which the most suitable fuzzy measure for each class was computed. we highlight that this fuzzy measure is considered in all the applications of the generalizations of the choquet integral. for the first proposed generalization, the product operator of the standard ci was replaced by different other t-norms [30]. in this way, the manner how the information was aggregated would be different, consequently leading into different frms that could present performances even more accurately. the choquet integral generalized by t-norms t, known as \(c_t\)-integral [17], is defined as: definition 1  [17] let \(\mathfrak {m}: 2^{n} \rightarrow [0,1]\) be a fuzzy measure and \(t: [0,1]^2 \rightarrow [0,1]\) be an t-norm. a \(c_t\)-integral is the function \(\mathfrak {c}_{\mathfrak {m}}^{t}: [0,1]^n \rightarrow [0,1]\), defined, for all \({\boldsymbol{x}} \in [0,1]^n\), by $$\begin{aligned} \mathfrak {c}_{\mathfrak {m}}^t ({\boldsymbol{x}}) = \sum _{i=1}^{n}t\left( x_{(i)} x_{(i-1)} , \mathfrak {m}\left( a_{(i)} \right) \right) , \end{aligned}$$ (10) where \( x_{(i)} \) and \(a_{(i)}\) are defined as in eq. (7). observe that some \(c_t\)-integrals are not aggregation function. e.g., take the minimum t-norm \(t_{m} (x, y) = \min (x, y)\) and the cardinality measure (see [17, 28]), and consider \(\mathbf {x_1} = (0.05, 0.2, 0.7, 0.9)\) and \(\mathbf {x_2} = (0.05, 0.1, 0.7, 0.9)\), where \(\mathbf {x_1} > \mathbf {x_2}\). however, \(\mathfrak {c}_{\mathfrak {m}}^{t_m} (\mathbf {x_1}) = 0.7\) and \(\mathfrak {c}_{\mathfrak {m}}^{t_m} (\mathbf {x_2}) = 0.8\). thus, the primordial condition of increasingness of any aggregation function is not fulfilled by \(\mathfrak {c}_{\mathfrak {m}}^{t_m}\). yet, it is noticeable that the monotonicity property is not crucial for aggregation functions. take for example a well-known statistical tool, the mode. it is not considered as an aggregation since the monotonicity of this function is not fulfilled, although it is useful. in [31], bustince et al. introduced the notion of directional monotonicity, which allows monotonicity to be fulfilled along (some) fixed ray. so, with this in mind, the concept of pre-aggregation functions was introduced in [17]. these functions respect the boundary condition as any aggregation function, however, they are directional increasing: definition 2 [31] let \(\vec {r}=(r_1,\dots ,r_n)\) be a real n-dimensional vector, \(\vec {r} \ne \vec {0}\). a function \(f:[0,1]^n \rightarrow [0,1]\) is directionally increasing with respect to \(\vec {r}\) (\(\vec {r}\)-increasing, for short) if for all \((x_1,\dots ,x_n) \in [0,1]^n\) and \(c>0\) such that \((x_1+cr_1,\dots ,x_n+cr_n) \in [0,1]^n\) it holds that $$\begin{aligned} f(x_1+cr_1,\dots ,x_n+cr_n) \ge f(x_1,\dots ,x_n).\end{aligned}$$ (11) similarly, one defines an \(\vec {r}\)-decreasing function. now, as the chi algorithm it is not a state-of-the-art fuzzy classifier, the \(c_t\)-integrals were applied in the frm of a powerful fuzzy classifier like farc-hd [32]. the quality of the proposal was analyzed by applying these generalizations to cope with 27 classification problems. the considered datasets are available in keel [33] dataset repository. when comparing the different generalizations among themselves, it can be noticed that the one based on hamacher t-norm was superior to the remaining ones. this fact occurred with four out the five considered fuzzy measures. the best accuracy was obtained when combining the hamacher product with the power measure. to evaluate the quality of this best generalization, the study has compared it against the classical frm of wr, since both frms apply averaging aggregation functions. in this comparison, it was empirically demonstrated that this generalization is statistically superior to wr and the standard ci. 4 copulas and cc-integrals the usage of the generalizations of the ci in a powerful fuzzy classifier has produced satisfactory results to cope with classification problems. however, these generalizations were pre-aggregation functions, that is, the monotonicity is not satisfied. then, with this in mind, generalizations that are idempotent and averaging aggregation functions were developed. for that, in eq. (7), firstly the distributivity property of the product operation is considered with the subtraction and then replaced the two instances of the product by copulas [30], obtaining the cc-integrals [18]: definition 3 let \(\mathfrak {m}: 2^{n} \rightarrow [0,1]\) be a fuzzy measure and \(c: [0,1]^2 \rightarrow [0,1]\) be a bivariate copula. the cc-integral is defined as a function \(\mathfrak {c}_{\mathfrak {m}}^c : [0,1]^n \rightarrow [0,1]\), given, for all \({\boldsymbol{x}} \in [0,1]^n\), by $$\begin{aligned} \mathfrak {c}_{\mathfrak {m}}^c ({\boldsymbol{x}}) = \sum _{i=1}^{n} c \left( x_{(i)}, \mathfrak {m}\left( a_{(i)} \right) \right) c \left( x_{(i-1)}, \mathfrak {m}\left( a_{(i)} \right) \right) , \end{aligned}$$ (12) where \( x_{(i)} \) and \(a_{(i)}\) are defined as in eq. (7). to demonstrate the efficiency of the cc-integrals to tackle classification problems, an experimental study considering 30 numerical datasets is considered. this study was conducted in two different ways. the first one was focused on comparisons per family of copulas (t-norms, overlap functions [34, 35] and specific copulas), in order to find the function that presented the best generalization. then, this best generalization is compared with 1) the classical frm of wr (considering that both functions are averaging); 2) to the standard ci and 3) the best pre-aggregation function achieved in the previous study (\(c_t\)-integral), the one based on the hamacher t-norm. the best cc-integral is the cmin-integral, constructed with the minimum copulafootnote 2. the obtained results showed that the cmin-integral is statistically equivalent to the ci and the \(c_t\)-integral and superior than the wr. 5 \(c_f\)-integrals the acquired knowledge from the previous studies shows that the function responsible to generalize the ci is very important. at this point only generalizations with averaging characteristics were presented. having this in mind, the ci was generalized by special functions, in order to produce more competitive generalizations, allowing to produce non-averaging integrals. to achieve it, its used a family of left 0-absorbing aggregation functions f, which satisfy: (lae) \(\forall y \in [0,1]: f(0,y) = 0\). moreover, the following two basic properties are also important: (rne) right neutral element: \(\forall x \in [0,1]: f(x,1) = x\); (lc) left conjunctive property: \(\forall x,y \in [0,1]: f(x,y)\le x\); any bivariate function \(f:[0,1]^2 \rightarrow [0,1]\) satisfying both (lae) and (rne) is called left 0-absorbent (rne)-function. then, the so-called \(c_f\)-integral [21] is defined as: definition 4 [21] let \(f:[0,1]^2 \rightarrow [0,1]\) be a bivariate function and \(\mathfrak {m}: 2^{n} \rightarrow [0,1]\) be a fuzzy measure. the \(c_f\)-integral is the function \(\mathfrak {c}_{\mathfrak {m}}^{f} : [0,1]^n \rightarrow [0,1]\), defined, for all \({\boldsymbol{x}} \in [0,1]^n\), by $$\begin{aligned} \mathfrak {c}_{\mathfrak {m}}^{f} ({\boldsymbol{x}}) = \min \left\{ 1, \sum _{i=1}^{n} f\left( x_{(i)} -x_{(i-1)}, \mathfrak {m}\left( a_{(i)} \right) \right) \right\} , \end{aligned}$$ (13) where \( x_{(i)} \) and \(a_{(i)}\) are defined as in eq. (7). in [21, theorems 1 and 2], it was proved that the set of conditions that the function f should fulfill for the \(c_f\)-integral to be a pre-aggregation function is one of the following ones: theorem 1 ((lae) and (rne)) or theorem 2 ((lae), \(f(1,1)=1\) and (1, 0)-increasingness). moreover, for the \(c_f\)-integral to be averaging f must satisfy (rne) and (lc). this means that there exist a lot of non-averaging \(c_f\)-integrals. the quality of the \(c_f\)-integrals to cope with classification problems was tested considering 33 different datasets. the experimental study was conducted considering \(c_f\)-integrals with and without averaging characteristics. considering the non-averaging functions, six \(c_f\)-integrals were studied. in order to support the quality of this approach, a comparison with the best non-averaging \(c_f\)-integral with the frm of ac and a frm considering the probabilistic sum ps (since it is an operator with non-averaging characteristics) is provided. the results showed that the non-averaging \(c_f\)-integrals-integrals, as expected, offer a performance superior than the averaging ones, and the best \(c_f\)-integral, based on the function fna2footnote 3 provides results that are statistically superior than all classical frms, and also, very competitive with the classical non-averaging frms like ac or ps. 6 \(c_{f_1f_2}\)-integrals the previous study demonstrated that the generalization of the standard choquet integral by functions f resulted in satisfactory results. then, this study combine the ideas of previous approaches, precisely, it take the same idea of cc-integrals, generalizing the each of the two instances of copulas by a pair of functions f, called \(f_1\) and \(f_2\), as consequence obtaining the \(c_{f_1f_2}\)-integrals [22]: definition 5 let \(\mathfrak {m}: 2^{n} \rightarrow [0,1]\) be a symmetric fuzzy measure and \(f_1, f_2:[0,1]^2 \rightarrow [0,1]\) be two fusion functions fulfilling: (i) \(f_1\)-dominance (or, equivalently, \(f_2\)-subordination): \(f_1 \ge f_2\); (ii) \(f_1\) is (1, 0)-increasing, a \(c_{f_1f_2}\)-integral is defined as a function \(\mathfrak {c}_{\mathfrak {m}}^{(f_1,f_2)} : [0,1]^n \rightarrow [0,1]\), given, for all \({\boldsymbol{x}} \in [0,1]^n\), by (14) where \( x_{(i)} \) and \(a_{(i)}\) are defined as in eq. (7). in this paper, twenty-three different functions, f, were considered. as consequence, 201 different pairs of functions that could be used as \(f_1\) and \(f_2\) could be combined, respecting the dominance property. an important question that could appear is related to the choice of the function to be selected as \(f_1\) and the one to act as \(f_2\). therefore, a methodology to reduce the scope of the study have been proposed by using the concept of dominance and subordination strength degree, dst and sst respectively. definition 6 let \(\mathcal {f} = \{f_1, \ldots , f_m\}\) be a set of m fusion functions. the dominance and subordination strength degrees, dst and sst, of a fusion function \(f_i \in \mathcal {f}\) are defined, respectively, for \(j \in \{1, \ldots , m\}\), by as follows: $$\begin{aligned} dst (f_i)= & {} \frac{1}{m} \sum _{j=1}^m \left\{ \begin{array}{ll} 1 &{} \text{ if } \; f_i \ge f_j, \\ 0 &{} \text{ otherwise } \end{array} \right. \cdot 100\% \\ sst (f_i)= & {} \frac{1}{m} \sum _{j=1}^m \left\{ \begin{array}{ll} 1 &{} \text{ if } \; f_i < f_j, \\ 0 &{} \text{ otherwise. } \end{array} \right. \cdot 100\% \end{aligned}$$ the generalizations provided in this study are non averaging. moreover, they satisfy the boundary conditions of any (pre) aggregation function. however, considering the monotonicity, we observed that these functions are neither increasing nor directional increasing. in fact, they are ordered directionally (od) monotone functions [36]. these functions are monotonic along different directions according to the ordinal size of the coordinates of each input. the \(c_{f_{1}f_{2}}\)-integrals were used to cope with classification problems in 33 different datasets. when analyzing the results that were obtained by the usage of these generalizations, it is noticeable that the combination of a function having a high dominance as \(f_1\) combined with a function with high subordination as \(f_2\) presented the best results of this study (from the top ten of the best global accuracies from the 81 pairs, eight have this characteristic). we also observed that the opposite, for each function \(f_2\), is also true and that its best results are achieved when using a \(f_1\) with a high dominance. the performance of this proposal is analyzed by comparing them against distinct state-of-the-art frbcss, namely: farc-hd [32], furia [37], ivturs [38], a classical non-averaging aggregation operator like the probabilistic sum, \(p^*\), and, the best \(c_f\)-integral that was selected from the previous study, \(f_{na2}\). in this comparison, furia was the fuzzy classifier that achieved the highest accuracy mean, however, our new approach achieved a close classification rate. furthermore, the number of specific datasets where the performance of our generalization is the worst among all the methods in the comparison is less than that of furia. the function representing the \(c_f\)-integrals also achieved good results, meanwhile the remainder cases (ivturs, \(p^*\) and farc-hd) where inferior and similar among themselves. the 81 pairs of combinations considered to construct \(c_{f_{1}f_{2}}\)-integrals were compared against ivturs, \(p^*\), farc-hd and \(f_{na2}\). the results highlighted the quality of our new method because an equal or greater average result was obtained by 39, 36, 34 and 12 different combinations in these comparisons. finally, from the considered pairs, it was observed that five different \(c_{f_{1}f_{2}}\)-integrals were considered as control variable in the statistical test in which all methods are compared, including furia. the last generalization only presented statistical differences with respect to farc-hd. however, for any remaining pair, it is statistically equivalent when compared to furia and to \(f_{na2}\) and superior to ivturs, \(p^*\) and farc-hd. 7 detailed results   in this section the results obtained by the usage of different aggregation operator are shown. we highlight that these results consider the same 33 datasets as in [21, 22] and [28]. also, the results are related with the power measure, as mentioned previously, take into consideration the 5-fold cross validation technique [2] and are applied in the frm of the farc-hd [32] fuzzy classifierfootnote 4. the results are provided in table 2 where each cell correspond to the mean accuracy among all folds, the rows are related with the different considered datasets and the columns are the results obtained by classical frms such as: of the additive combination (ac), probabilistic sum (ps), winning rule (wr), choquet integral (ci), \(c_t\)-integrals (due to lack of space were summarized to int, in all integrals) with is defined by the hamacher product t-norm, cc-integral that use the copula of the minimum, \(c_f\)-integral considering the \(fna_2\) function and \(c_{f_{1}f_{2}}\)-integrals using the pair gm–fbpc. table 2. detailed results achieved in test by different generalizations of the ci.full size table from the detailed results, we can noticed that classic frm of the wr is the one that achieved the lowest global mean, indicating that the usage of all information related with the problem is an interesting alternative. moreover, it is also observable that all non-averaging generalizations (ac, ps, \(c_f\)-integrals and \(c_{f_{1}f_{2}}\)-integrals) presents superior results when compared against the averaging ones (wr, ic, \(c_t\)-integral and cc-integral). the results also showed that the generalizations of the ci (\(c_t\), cc, \(c_f\), \(c_{f_{1}f_{2}}\)-integrals) provided a superior performance in comparison to the standard ci. finally, as mentioned before, the largest performance is obtained when the \(c_{f_{1}f_{2}}\)-integral is used to cope with classification problems. 8 conclusions the application of the choquet integral (ci) in the fuzzy reasoning method (frm) of fuzzy rule-based classification systems (frbcss) modified the way in which the information was used and enhanced the system quality. after that, many generalizations of the ci were proposed and also applied in frm, obtaining success as well. in this paper the main contributions, theoretical and applied, of the generalizations are summarized and discussed. the first generalization was built by the replacement of the product operator of the standard ci by different t-norms. these generalizations were supported by an important theoretical concept known as pre-aggregation functions. differently from a simple aggregation function, a pre-aggregation function is monotonic only in a determined direction. this first generalization produced averaging functions and its applications to cope with classification problems showed that the generalization by the hamacher product t-norm was superior than the frm of the winning rule (wr) and the ci. the second step aimed in generalizations of the ci that produce aggregation functions. to do so, the ic was used in its expanded form and generalized by copula functions, introducing the concept of choquet-like copula-based aggregation functions, the so called cc-integrals. these functions also present averaging characteristics. the results of their applications demonstrated that the classical wr was statistically overcame. it is observable that up to this point only generalizations with averaging characteristics were presented. on the otter hand, fuzzy classifiers known as state-of-the-art take into account the usage of non-averaging functions. thus, to produce more competitive generalizations, a family of fusion functions f were introduced. the generalization of the choquet integral by f functions introduced the concept of \(c_f\)-integrals. this generalization has averaging and non-averaging characteristics, it depends on the considered function. it was observed that the application of any non-averaging function statistically overcome any averaging one. also, the developed operators outperforms the classical wr and additive combination (ac). the generalization of the expanded ci by two functions f, \(f_1\) and \(f_2\), introduced the concept of \(c_{f_1f_2}\)-integrals. these functions present an ordered directional increasing functions (od increasing) and, therefore, represent a different level of aggregation operators. the summit of the performance in the classification problems was reached in this generalization. to do so, a methodology to select different functions as \(f_1\) and \(f_2\) were presented, based on the concept of degrees of dominance and subordination. for the considered \(c_{f_1f_2}\)-integrals, in five different cases the generalizations are equivalent, or even superior, in comparison with fuzzy classifiers found in the literature. taking as basis the analysis provided by this paper, some interesting research points emerge. for example, the application of these generalizations in the frm of different fuzzy classifiers. also, considering that the generalizations are based on the choquet integral, the usage of a different operator, such as the sugeno integral can produce even more powerful operators. finally, the combinations with different fuzzy measures are an alternative with great potential. notes 1.a fuzzy measure \(\mathfrak {m}\) is an increasing function on \(2^{n}\) such that \(\mathfrak {m} (\emptyset ) = 0\) and \(\mathfrak {m} (n) = 1\). 2.examples of special cc-integrals were studied in [19, 20]. 3.the function \(fna2: [0,1]^2\) is defined, for all \(x,y \in [0,1]\) by \(f(0,y)= 0\), \(f(x,y) = \frac{x+y}{2}\) if \(0 < x \le y\) and \(f(x,y) = \min (\frac{x}{2},y)\), otherwise, which satisfies the conditions of [21, theorems 2]. 4.the considered datasets and the fuzzy classifier are available in keel repository. available at https://www.keel.es. references alpaydin, e.: introduction to machine learning, 2nd edn. mit press, cambridge (2010) math  google scholar  tan, p.-n., steinbach, m., kumar, v.: introduction to data mining, 1st edn. addison-wesley longman publishing co., inc., boston (2005) google scholar  cortes, c., vapnik, v.: support vector networks. mach. learn. 20, 273–297 (1995). https://doi.org/10.1007/bf00994018 article  math  google scholar  quinlan, j.: c4.5: programs for machine learning. morgan kauffman, san francisco (1993) google scholar  garcía-pedrajas, n., garcía-osorio, c., fyfe, c.: nonlinear boosting projections for ensemble construction. j. mach. learn. res. 8, 1–33 (2007) mathscinet  math  google scholar  ishibuchi, h., nakashima, t., nii, m.: classification and modeling with linguistic information granules, advanced approaches to linguistic data mining. advanced information processing, springer, heidelberg (2005). https://doi.org/10.1007/b138232 book  math  google scholar  zadeh, l.a.: the concept of a linguistic variable and its application to approximate reasoning i. inf. sci. 8(3), 199–249 (1975) article  mathscinet  google scholar  unold, o.: diagnosis of cardiac arrhythmia using fuzzy immune approach. in: dobnikar, a., lotrič, u., šter, b. (eds.) icannga 2011, part ii. lncs, vol. 6594, pp. 265–274. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-20267-4_28 chapter  google scholar  vitoriano, b., rodrígues, j.t., tirado, g., mart ín-campo, f.j., ortuão, m.t., montero, j.: intelligent decision-making models for disaster management. hum. ecol. risk assess. int. j. 21(5), 1341–1360 (2015) article  google scholar  sanz, j.a., bernardo, d., herrera, f., bustince, h., hagras, h.: a compact evolutionary interval-valued fuzzy rule-based classification system for the modeling and prediction of real-world financial applications with imbalanced data. ieee trans. fuzzy syst. 23(4), 973–990 (2015) article  google scholar  goel, n., sehgal, p.: fuzzy classification of pre-harvest tomatoes for ripeness estimation an approach based on automatic rule learning using decision tree. appl. soft comput. 36, 45–56 (2015) article  google scholar  cordon, o., del jesus, m.j., herrera, f.: analyzing the reasoning mechanisms in fuzzy rule based classification systems. mathw. soft comput. 5(2–3), 321–332 (1998) math  google scholar  chi, z., yan, h., pham, t.: fuzzy algorithms: with applications to image processing and pattern recognition. world scientific publishing co., inc., river edge (1996) book  google scholar  barrenechea, e., bustince, h., fernandez, j., paternain, d., sanz, j.a.: using the choquet integral in the fuzzy reasoning method of fuzzy rule-based classification systems. axioms 2(2), 208–223 (2013) article  google scholar  choquet, g.: “theory of capacities,” annales de l’institut fourier, vol. 5, pp. 131–295, (1953–1954) google scholar  dimuro, g.p., et al.: the state-of-art of the generalizations of the choquet integral: from aggregation and pre-aggregation to ordered directionally monotone functions. inf. fusion 57, 27–43 (2020) article  google scholar  dimuro, g., et al.: pre-aggregation functions: construction and an application. ieee trans. fuzzy syst. 24(2), 260–272 (2016) article  google scholar  lucca, g., et al.: cc-integrals: choquet-like copula-based aggregation functions and its application in fuzzy rule-based classification systems. knowl-based syst. 119, 32–43 (2017) article  google scholar  dimuro, g.p., lucca, g., sanz, j.a., bustince, h., bedregal, b.: cmin-integral: a choquet-like aggregation function based on the minimum t-norm for applications to fuzzy rule-based classification systems. in: torra, v., mesiar, r., de baets, b. (eds.) agop 2017. aisc, vol. 581, pp. 83–95. springer, cham (2018). https://doi.org/10.1007/978-3-319-59306-7_9 chapter  math  google scholar  lucca, g., sanz, j.a., dimuro, g.p., bedregal, b., bustince, h.: a proposal for tuning the the \(\alpha \) parameter in \(c_{\alpha }c\)-integrals for application in fuzzy rule-based classification systems. nat. comput. 19, 533–546 (2020) article  mathscinet  google scholar  lucca, g., sanz, j.a., dimuro, g.p., bedregal, b., bustince, h., mesiar, r.: cf-integrals: a new family of pre-aggregation functions with application to fuzzy rule-based classification systems. inf. sci. 435, 94–110 (2018) article  google scholar  lucca, g., dimuro, g.p., fernandez, j., bustince, h., bedregal, b., sanz, j.a.: improving the performance of fuzzy rule-based classification systems based on a nonaveraging generalization of cc-integrals named \(c_{f_1f_2}\)-integrals. ieee trans. fuzzy syst. 27(1), 124–134 (2019) article  google scholar  dimuro, g.p., et al.: generalized cf1f2-integrals: from choquet-like aggregation to ordered directionally monotone functions. fuzzy sets syst. 378, 44–67 (2020) article  google scholar  takagi, t., sugeno, m.: fuzzy identification of systems and its applications to modeling and control. ieee trans. syst. man cybern. smc-15(1), 116–132 (1985) google scholar  mamdani, e.h.: application of fuzzy algorithms for control of simple dynamic plant. in: proceedings of the institution of electrical engineers, vol. 121, issue number 12, pp. 1585–1588 (1974) google scholar  ishibuchi, h., nakashima, t.: effect of rule weights in fuzzy rule-based classification systems. ieee trans. fuzzy syst. 9(4), 506–515 (2001) article  google scholar  cordón, o., del jesus, m.j., herrera, f.: a proposal on reasoning methods in fuzzy rule-based classification systems. int. j. approximate reasoning 20(1), 21–45 (1999) article  google scholar  lucca, g., sanz, j.a., dimuro, j.a., borges, e.n., santos, h., bustince, h.: analyzing the performance of different fuzzy measures with generalizations of the choquet integral in classification problems. in: 2019 ieee international conference on fuzzy systems (fuzz-ieee), pp. 1–6, june 2019 google scholar  chi, z., yan, h., pham, t.: fuzzy algorithms: with applications to image processing and pattern recognition. world scientific, singapore (1996) book  google scholar  alsina, c., frank, m.j., schweizer, b.: associative functions: triangular norms and copulas. world scientific publishing company, singapore (2006) book  google scholar  bustince, h., fernandez, j., kolesárová, a., mesiar, r.: directional monotonicity of fusion functions. eur. j. oper. res. 244(1), 300–308 (2015) article  mathscinet  google scholar  alcala-fdez, j., alcala, r., herrera, f.: a fuzzy association rule-based classification model for high-dimensional problems with genetic rule selection and lateral tuning. ieee trans. fuzzy syst. 19(5), 857–872 (2011) article  google scholar  alcalá-fdez, j., et al.: keel: a software tool to assess evolutionary algorithms for data mining problems. soft comput. 13(3), 307–318 (2009). https://doi.org/10.1007/s00500-008-0323-y article  google scholar  bustince, h., fernandez, j., mesiar, r., montero, j., orduna, r.: overlap functions. nonlinear anal. theory methods appl. 72(3–4), 1488–1499 (2010) article  mathscinet  google scholar  dimuro, g.p., bedregal, b.: on the laws of contraposition for residual implications derived from overlap functions. in: 2015 ieee international conference on los alamitos fuzzy systems (fuzz-ieee), pp. 1–7. ieee, august 2015 google scholar  bustince, h., et al.: ordered directionally monotone functions. justification and application. ieee trans. fuzzy syst. 1 (2017, in press). corrected proof google scholar  hühn, j., hüllermeier, e.: furia: an algorithm for unordered fuzzy rule induction. data min. knowl. discov. 19(3), 293–319 (2009) article  mathscinet  google scholar  sanz, j., fernández, a., bustince, h., herrera, f.: ivturs: a linguistic fuzzy rule-based classification system based on a new interval-valued fuzzy reasoning method with tuning and rule selection. ieee trans. fuzzy syst. 21(3), 399–411 (2013) article  google scholar  download references author information authors and affiliations universidade federal do rio grande, rio grande, rs, brazil giancarlo lucca, eduardo n. borges, rafael a. berri, leonardo emmendorfer, graçaliz p. dimuro & tiago c. asmus universidad publica de navarra, pamplona, navarra, spain graçaliz p. dimuro & tiago c. asmus authors giancarlo luccaview author publications search author on:pubmed google scholar eduardo n. borgesview author publications search author on:pubmed google scholar rafael a. berriview author publications search author on:pubmed google scholar leonardo emmendorferview author publications search author on:pubmed google scholar graçaliz p. dimuroview author publications search author on:pubmed google scholar tiago c. asmusview author publications search author on:pubmed google scholar corresponding author correspondence to giancarlo lucca . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper lucca, g., borges, e.n., berri, r.a., emmendorfer, l., dimuro, g.p., asmus, t.c. (2021). on the generalizations of the choquet integral for application in frbcs. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13073. springer, cham. https://doi.org/10.1007/978-3-030-91702-9_33 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91702-9_33 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91701-2 online isbn: 978-3-030-91702-9 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords choquet integral generalizations choquet integral pre-aggregation function publish with us policies and ethics profiles giancarlo lucca view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature code autocomplete using transformers | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper code autocomplete using transformers conference paper first online: 28 november 2021 pp 211–222 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) code autocomplete using transformers download book pdf download book epub gabriel t. meyrer  orcid: orcid.org/0000-0003-2589-057510, denis a. araújo  orcid: orcid.org/0000-0002-4319-594x10 & sandro j. rigo  orcid: orcid.org/0000-0001-8140-562111  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1455 accesses 1 citation abstract in software development, code autocomplete can be an essential tool in order to accelerate coding. however, many of these tools built into the ides are limited to suggesting only methods or arguments, often presenting to the user long lists of irrelevant items. since innovations introduced by transformer-based models that have reached the state of the art performance in tasks involving natural language processing (nlp), the application of these models also in tasks involving code intelligence, such as code completion, has become a frequent object of study in recent years. in these paper, we present a transformer-based model trained on 1.2 million java files gathered from top-starred github repositories. our evaluation approach was based on measuring the model’s ability to predict the completion of a line, proposing a new metric to measure the applicability of the suggestions that we consider better adapted to the practical reality of the code completion task. with a recently developed java web project as test set, our experiments showed that in 55.9% of the test cases the model brought at least one suggestion applicable, while the best baseline model presented this in 26.5%. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others statistical approach to increase source code completion accuracy chapter © 2018 parameter-efficient fine-tuning of pre-trained code models for just-in-time defect prediction article 03 june 2024 on source code completion assistants and the need of a context-aware approach chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automation models of computation software engineering programming techniques reverse engineering software testing 1 introduction when building digital solutions, combining good practices of code quality and standardization with fast development is one of the challenges in the field of software engineering. in this context of productivity, associated with integrated development environments (ides) and often used by developers, code completion can be an essential tool in order to accelerate coding [10]. many traditional code completion systems works recommending all possible methods or attributes. although, the list of suggestions is often long, requiring the user to scroll through many irrelevant items. to supply this, intelligent code completion systems propose to reduce these lists to relevant items given the context being edited [1, 10, 13]. however, many of them work exclusively suggesting only methods or arguments [12], that is, not able to complete longer code sequences by suggesting them both together, increasing productivity. since innovations introduced by transformer-based solutions such as bidirectional encoder representations from transformers (bert) [2], generative pre-trained transformer (gpt) [11] and xlnet [17], natural language processing (nlp) tasks has significantly expanded its range of possibilities in recent years. dismissing recurrence or convolutions, the transformers model architecture has reached state of the art results in translation tasks mainly based on attention mechanisms [16]. thus, the application of these models also in tasks involving code intelligence such as code completion has become a frequent target [3, 5, 7, 12, 14]. even though it has aroused great interest and stimulated much work, there are some open research points in this area. the appropriate use of meaningful datasets for the tasks, modeling and evaluation aspects are among the observed challenges. so our work was developed with this context in perspective. the first contribution of this paper is a transformer-based model trained on 1.2 million java files focused on code completion. as the second contribution, we point out a new method of evaluating code completion systems, proposing a new metric that we consider closer to measuring the applicability of the suggestion in the real world of software development. the paper is structured as follows: sect. 2 discusses the context of related works. after that, sect. 3 describes our approach for building the corpus, model definition and training. in sect. 4, we define our model evaluation methodology, and propose a new code completion evaluation metric by arguing how the standard metrics that we are aware of can be problematic in this context. section 5 presents the results of the experiments and, finally, sect. 6 concludes this work. 2 related work this section summarizes selected papers in the area, highlighting their objectives, results, techniques, and parameters used. the papers were obtained in a non-systematic review considering the main topics involved in machine learning applied to code autocomplete tasks. a challenge when applying standard language models to tasks like code completion consists of out-of-vocabulary (oov) words, which can be understood as words that were not observed during training and become unpredictable for many models. this problem can strongly affect the performance since the software developers can create any identifier needed on the code, which is not a frequent problem in natural language tasks [4]. jian li et al. [6] proposes a solution for that presenting what they called a pointer mixture network. based on the context, the proposed model learns to regenerate an oov word locally repeated through a pointer copy mechanism, and generate a within vocabulary word through an rnn component. to build the training corpus, the source codes were represented in the form of abstract syntax tree (ast). it was evaluated different approaches with javascript and python datasets, totaling 150,000 files each stored in the ast format. although, as demonstrated by rafael-michael karampatsis et al. [4], advanced approaches such as variations of the byte-pair encoding (bpe) algorithm can not only deal well with the oov problem, but also reduce the vocabulary size. having the bpe as a principle of the tokenization strategy, the gpt-2 [11] is a 1.5b parameter transformer [16] that performs state of the art results in several natural language tasks. recently, this model started to be widely explored also for code intelligence tasks. in this context, alexey svyatkovskiy et al. [12] introduces gpt-c, which is a variant of gpt-2 trained from scratch using a large set of data files that comprises over 1.2 billion lines of source code in javascript, typescript, c# and python. as the evaluation of the model’s quality, besides the perplexity, was used another two metrics to capture string similarity between suggested and expected code: rouge-l, which consists of a variant of the recall-oriented understudy for gisting evaluation (rouge) based on the longest common subsequence (lcs), and levenshtein edit similarity. the authors consider the last one a critical metric in this context by affirming that developers tend to accept suggestions even partially matching the expectation, that is, requiring edits. the best model presented was trained on python, reaching an average perplexity of 1.82, rouge-l of 0.80 and an edit similarity of 86.7%. in addition, shuai lu et al. [7] also proposes a gpt model for code completion and text-to-code generation, named codegpt. the authors present pretrained models for both python and java, with datasets that includes respectively 1.1m and 1.6m functions. for each programming language, they have used two training approaches. the first approach trains the model from scratch, which means that weights are randomly initialized and the vocabulary is newly obtained. the other approach uses gpt-2 model as starting point, taking advantage of the predefined weights and keeping the same vocabulary. the evaluation of the models is made both in token-level, measuring the accuracy in predicting the next token, and line-level, measuring the exact match (em) and the edit similarity. their best model reaches a score average of 71.28. finally, based on the gpt-2 model as well, we find tabnine [14]. available for several ides, this is a plugin that displays code sequences as suggestions according to the context. however, we are not aware about performance metrics of this solution, as much as architecture details. 3 approach in this session, we describe the construction of our training corpus, as well as detail the model applied and the training process. 3.1 corpus we built our corpus selecting public java projects on github. in order to increase the probability of obtaining a selection of projects with well-written code, we limited our search for repositories with at least 1500 stars, indicating a good relevance in the community. the corpus is structured in a single text file, using special tokens to delimit the beginning and end of a file, as well as the end of each line. after processing these repositories, we reached 8.12g of total data size, comprising over 1.4 million files and 230 million lines of java source code. 3.2 model the transformers [16] is a type of deep neural network that relies on attention mechanisms for sequence processing, dispensing with recurrence or convolutions. this architecture applies multiple heads of what the authors called self-attention mechanism, which enables the model to draw complex correlations between tokens in different levels, taking this data into account when predicting the next tokens in a sequence. this type of network overcomes the state of the art approaches such as recurrent neural networks and long short-term memory in many tasks in the field of natural language processing (nlp) and natural language understanding (nlu). in addition, it requires significantly less time to train as it is able to parallelize. the architecture of our model is the same as the gpt-2 [11], which is an auto-regressive pre-trained model based on 12 layers of transformer decoders. in order to take advantage of the pre-defined weights and vocabulary, we used the gpt-2 small version pretrained in english as a starting point. this indicates that the resulting model has the original gpt-2 vocabulary and persists the natural language processing skills. the process of model fine-tuning with the java corpus was done with 1 epoch, 1.6 million steps and 96 h of computing time in a single gpu v100. throughout this article, we’ll refer to our model as java8g. 4 evaluation in this section, we define our evaluation methodology, where we initially propose the creation of a new metric to measure the applicability of the model’s suggestions in code completion, and then we detail how we proceed with the experiments. 4.1 dg evaluation metric this far we already know that language models can also be applied to problems involving code intelligence. however, when looking for the best metric to evaluate the model code suggestions, we found that the application of standard metrics (e.g. levenshtein and bleu) to evaluate natural language tasks could be problematic when applied to code completion tasks, meaning that they may not measure the real applicability of a code hint. motivated by this, we propose a new metric we consider better adapted to the context of code completion. given two strings expected and prediction, the dg metric, as formally described in algorithm 1, calculates the sum of equal characters until there is a divergence between expected and prediction, rated by the expected length. to support the proposed metric, in fig. 1, we present two practical cases simulating the string comparison between an expected code and the model prediction by applying edit similarity, one of the metrics used to evaluate the intellicode models [12], against dg metric. as we can see, in the case a, the strings are relatively close by the edit similarity metric, but there’s a big difference between them when we talk about the algorithm logic. in such case, we understand that there’s a higher chance of the user skipping the suggestion than accept and edit it to meet the expectation, since it tends to be as less productive as writing the entire sequence. in the other way, we have a lower dg score, showing that only 20% of the prediction could be applied without requiring adaptation. now, in the case b, we have a situation where the model effectively suggests part of the expected code, meaning that the user could fully accept the suggestion, but we don’t have it reflected on the edit similarity metric. even more problematic, we have two very different situations for the practical reality of the code completion presenting the same edit similarity score (67.6%). that’s not the case for dg score where we have a full match, meaning that the suggestion is perfectly applicable. fig. 1. edit similarity and dg metric in code completion full size image 4.2 methodology the dg metric creation and evaluation methodology we defined was based on the attempt to get as close as possible to the practical reality of using a code completion tool. for this purpose, our approach was based on submitting the model to try to complete a line given specific previous contexts, and evaluate the applicability of the predictions in the current line using the dg metric. to base the execution of the tests we selected a real project of an web api recently developed with spring boot, one of the most popular java frameworks in web development. in order to get more insights from the evaluation, we decided to observe the performance in more specific contexts of the project. thus, we categorized the files in the following groups, which are also often found in other web api projects: controller, which comprises implementation of classes that handles http requests; service, consisting of classes that implements functionalities and system business rules; test, defining the context of unit testing; generic, for everything else. from this project, we generated 4075 test cases for evaluation. to define the test cases we worked with the concept of triggers, which can be understood as points of opportunity to display suggestions, like the insertion of characters such as white space, “.", “(" or “@". we consider this examples as triggers for the java language because we understand them as points where the developer is more likely to seek help from a code completion tool. therefore, our evaluation system iterates through the project files lines, seeking for triggers to generate test cases, taking what’s before the trigger as input and what’s next as reference to measure the applicability of the predictions. to exemplify this, the fig. 2 shows how we structured the test cases. highlighted in dark gray, we have the marker endofcontext right after the trigger “@", meaning that everything before this marker, in white, is the input sent to the model and, in dark green, what will be used as reference for evaluation. fig. 2. example of a test case full size image for each test case, we request twice the model to generate two sequences of 16 tokens. for each model prediction, we create two suggestions: the long suggestion (l) and the short suggestion (s). in the fig. 3 we show highlighted a demonstration of the model suggestions and their types in a plugin developed for idea intellij. therefore, the l suggestions are the complete predictions of the model within the boundary of a line. as an attempt to increase the performance of the model, the short suggestions presents only a part of the predicted sequence, since we understand that there’s a higher probability of the model to correctly predict a short next part of the code than trying to complete the entire line. as shown, the s suggestions are the first alphabetical part of the l suggestions. fig. 3. java8g suggestions on intellij full size image 4.3 baseline we selected the microsoft codegpt [7] java model as baseline, since our model is also based on the gpt architecture. we ran the tests with both models versions available on huggingface [15]: the codegpt [8], which is trained from scratch, and the codegpt-adapted [9], that is pretrained in english as well as our model. 5 results in this section we present different analyzes and interpretations of the results obtained in the experiments. in fig. 4 we present the models long and short suggestions score average in two metrics: dg, on the left, and edit similarity on the right. as we can see, unlike the dg metric, the l suggestions performed better than s in edit similarity. this highlights the problem of using this metric when you want to measure the applicability of the model’s predictions. since s is part of the l suggestion, if l is applicable, then s should be as well. the opposite happens with the edit similarity because it considers the length of the target to calculate the score, penalizing the suggestion score even with it fully matching a initial part of the target. therefore, with dg metric, the s suggestions must have a superior or at least equivalent performance compared to the l suggestion. and, as we can see, the model tends to predict shorter sequences more accurately than longer sequences, meeting the initial expectation regarding the creation of short suggestions. fig. 4. suggestions dg and edit similarity score average full size image by analysing the fig. 5, in which we again demonstrate the score average of both dg metric and edit similarity, but this time with the individual results for each code category, we can observe a pattern: in most of the analysis the performance in the controller category presented the best results. intuitively, by analyzing the code pattern applied in these contexts, we can deduce that, since it is usually formed by blocks of codes that are remarkably often repeated, we understand that it tends to become more easily predictable by the model. the same could explain the lower performance on the service category, where there tend to have fewer patterns or repetitions that can increase the predictive capacity of the model. for ease of reading and comparison, we also present this data in tabular form, in table 1, adding the average of the metric for each tested model. fig. 5. suggestions dg and edit similarity score by category full size image table 1. suggestions and average for dg and edit similarity score by categoryfull size table table 2. presence of full matching suggestions in each test case and averagefull size table fig. 6. presence of full matching suggestions in each test case full size image analyzing the average performance of the metrics can give us a good overview of the model’s performance. however, since we understand that the developer tends to ignore suggestions that require corrections, the analysis considering partially matching suggestions can be limited. this way, looking deeper into the data, in fig. 6 we try to answer an important question: of all the test scenarios applied, how many presented at least one full matching suggestion? as we can see, our model in 55% the tests brought at least one full match suggestion, while the best codegpt model present this in 26.5% of the cases. again, we present the view in tabular form in table 2. 6 conclusion we presented a gpt-2 based model trained on 1.2 million java files focused on code completion. we also introduced the dg metric, a new approach that we consider closer to measure the applicability of the suggestion in the real world of the software development compared to standard metrics such as levenshtein that, as we demonstrated, can be problematic. we also look to run tests on a real and newly developed project, seeking to get a real idea of the model’s completion capability in a situation as close to reality as we could simulate. among different analyzes shown, we demonstrated that our model presented at least one fully applicable suggestion in more than half of the test cases, outperforming in 2 times the baseline models. the results corroborate, along with other studies in this field, that transformers-based models have great potential in code intelligence tasks. in the future, as also proposed by alexey svyatkovskiy et al. [12], we plan to finetune the model for user personalization, making the model sensitive to specific standards of each project, enhancing the performance of the suggestions. references bruch, m., monperrus, m., mezini, m.: learning from examples to improve code completion systems. in: proceedings of the 7th joint meeting of the european software engineering conference and the acm sigsoft symposium on the foundations of software engineering, esec/fse ’09, pp. 213–222. association for computing machinery, new york (2020). isbn 9781605580012, https://doi.org/10.1145/1595696.1595728 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, minneapolis, minnesota, vol. 1 (long and short papers), pp. 4171–4186. association for computational linguistics (2019). https://doi.org/10.18653/v1/n19-1423, https://www.aclweb.org/anthology/n19-1423 feng, z., et al.: codebert: a pre-trained model for programming and natural languages. in: findings of the association for computational linguistics: emnlp 2020, pp. 1536–1547. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.139, https://www.aclweb.org/anthology/2020.findings-emnlp.139 karampatsis, r.m., babii, h., robbes, r., sutton, c., janes, a.: big code != big vocabulary. in: proceedings of the acm/ieee 42nd international conference on software engineering (2020). https://doi.org/10.1145/3377811.3380342, http://dx.doi.org/10.1145/3377811.3380342 kim, s., zhao, j., tian, y., chandra, s.: code prediction by feeding trees to transformers. in: 2021 ieee/acm 43rd international conference on software engineering (icse), pp. 150–162 (2021). https://doi.org/10.1109/icse43902.2021.00026 li, j., wang, y., lyu, m.r., king, i.: code completion with neural attention and pointer networks. in: proceedings of the twenty-seventh international joint conference on artificial intelligence (2018). https://doi.org/10.24963/ijcai.2018/578, http://dx.doi.org/10.24963/ijcai.2018/578 shuai, l., et al.: a machine learning benchmark dataset for code understanding and generation, codexglue (2021) google scholar  microsoft code gpt small java (2021). https://huggingface.co/microsoft/codegpt-small-java, accessed 18 may 2021 microsoft code gpt small java adapted gpt-2 (2021). https://huggingface.co/microsoft/codegpt-small-java-adaptedgpt2, accessed 18 may 2021 proksch, s., lerch, j., mezini, m.: intelligent code completion with bayesian networks. acm trans. softw. eng. methodol. 25(1) (2015). issn 1049–331x, https://doi.org/10.1145/2744200 radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i.: language models are unsupervised multitask learners (2019). https://openai.com/blog/better-language-models/ svyatkovskiy, a., deng, s.k., fu, s., sundaresan, n.: intellicode compose: code generation using transformer. in: proceedings of the 28th acm joint meeting on european software engineering conference and symposium on the foundations of software engineering, esec/fse 2020, new york, ny, usa, pp. 1433–1443. association for computing machinery (2020). isbn 9781450370431, https://doi.org/10.1145/3368089.3417058 svyatkovskiy, a., zhao, y., fu, s., sundaresan, n.: pythia: ai-assisted code completion system. in: proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining (2019). https://doi.org/10.1145/3292500.3330699 tabnine. code faster with ai completions (2021). https://www.tabnine.com/, accessed 18 may 2021 the ai community building the future (2021). https://huggingface.co/, accessed 18 may 2021 vaswani, a., et al.: attention is all you need. in: proceedings of the 31st international conference on neural information processing systems, nips’17, red hook, ny, usa, pp. 6000–6010. curran associates inc. (2017). isbn 9781510860964 google scholar  yang, z., dai, z., yang, y., carbonell, j., salakhutdinov, r.r., le, q.v.: xlnet: generalized autoregressive pretraining for language understanding. in: wallach, h., larochelle, h., beygelzimer, a., d’alché-buc, f., fox, e., garnett, r. (eds.) advances in neural information processing systems, vol. 32. curran associates inc. (2019). https://proceedings.neurips.cc/paper/2019/file/dc6a7e655d7e5840e66733e9ee67cc69-paper.pdf download references author information authors and affiliations cwi software, são leopoldo, brazil gabriel t. meyrer & denis a. araújo applied computing graduate program, unisinos, são leopoldo, brazil sandro j. rigo authors gabriel t. meyrerview author publications search author on:pubmed google scholar denis a. araújoview author publications search author on:pubmed google scholar sandro j. rigoview author publications search author on:pubmed google scholar corresponding author correspondence to gabriel t. meyrer . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper meyrer, g.t., araújo, d.a., rigo, s.j. (2021). code autocomplete using transformers. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_15 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords code completion deep learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature long-term map maintenance in complex environments | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper long-term map maintenance in complex environments conference paper first online: 28 november 2021 pp 146–161 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) long-term map maintenance in complex environments download book pdf download book epub josias oliveira10, filipe mutz11, avelino forechi12, pedro azevedo10, thiago oliveira-santos10, alberto f. de souza10 & … claudine badue10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1333 accesses abstract as changes in external environments are inevitable, a lifelong mapping system is desirable for autonomous robots that aim at long-term operation. capturing external environment changes into internal representations (for example, maps) is crucial for proper behavior and safety, especially in the case of autonomous vehicles. in this work, we propose a new large-scale mapping system for our autonomous vehicle or any other. the new mapping system is based on the graph slam algorithm, with extensions to deal with the calibration of odometry directly in the optimization of the graph and to address map merging for long-term map maintenance. the mapping system can use sensor data from one or more robots to build and merge different types of occupancy grid maps. the system’s performance is evaluated in a series of experiments carried out with data captured in complex real-world scenarios. the experimental results indicate that the new large-scale mapping system can provide high-quality occupancy grid maps for later navigation and localization of autonomous vehicles that use occupancy grid maps. this study was financed in part by coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001; conselho nacional de desenvolvimento científico e tecnológico brasil (cnpq); and fundação de amparo à pesquisa do espírito santo brasil (fapes) – grants 75537958 and 84412844. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others glans: gis based large-scale autonomous navigation system chapter © 2018 global map generation using local feature grid maps for autonomous vehicles in frequently changing off-road environments chapter © 2024 vector maps: a lightweight and accurate map format for multi-robot systems chapter © 2016 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. applied dynamical systems brain mapping cartography long-term potentiation long-term memory geographical information system 1 introduction autonomous vehicles (avs) typically use maps of the environment to localize themselves and safely navigate in it. in this context, avs can localize themselves in relation to the map using their sensor measurements (localization problem). when there is no map, they can create a new one also using their sensors. vehicle poses can be known when creating the map (mapping problem) or can be estimated as the map is built (simultaneous localization and mapping slam problem [4]). one of the requirements imposed on avs is the ability to operate for a long period of time without human intervention. therefore, avs must display a life-long mapping capability and promptly update their internal maps whenever the environment changes. in addition, when operating outdoors, avs face highly dynamic objects (for example, pedestrians and other vehicles) and slightly dynamic objects (for example, parked vehicles and traffic cones). environments may undergo slower modifications to the infrastructure and, therefore, the avs’ internal maps must be updated accordingly. these issues, altogether, pose the life-long mapping problem, which usually requires merging the current map already built with the new ones from another surveying mission with a single robot or multiple robots at the same time. the map merging problem consists of combining two or more individual maps without a common frame of reference into a larger global map. fig. 1. example of map merging. the lack of loop closure treatment between the sessions results in misalignment between the maps. full size image one of the main challenges in the map merging problem is illustrated in fig. 1, where red and green pixels represent obstacles reached by a lidar sensor and projected in the ground. red pixels represent obstacles in a map while green pixels represent obstacles in another. considering the final map after improper merging, there is a prominent misalignment between the two individual maps. we have developed a self-driving car (fig. 2), whose autonomy system follows the typical architecture of self-driving cars [1]. our self-driving car is based on a ford escape hybrid tailored with a variety of sensors and processing units. its autonomy system is composed of many modules, which include a localizer [6], a mapper [13], a moving obstacle tracker [15], a traffic signalization detector [14, 20], a route planner, a path planner for structured urban areas, a behavior selector, a motion planner [5], an obstacle avoider [10], and a controller [11], among other modules. fig. 2. our self-driving car and its autonomy software system. in light blue, perception modules. in orange, decision making modules. tsd denotes traffic signalization detection and mot, moving objects tracking. red arrows show the state of the car, produced by its localizer module and shared with most modules; blue arrows show the car’s internal representation of the environment, jointly produced by several perception modules and shared with most decision making modules. see details in badue et al. (2020). full size image in this work, we propose a new mapping system based on graph slam for solving the map merging problem in large outdoor environments. the proposed mapping system aims to provide high quality grid maps for autonomous driving. it is also able to calibrate the vehicle odometry using a new method applied in the graph optimization. the system receives as input one or more log files containing the surveying missions (or sessions) from a single or multiple vehicles, and it outputs the avs’ poses in a global coordinate frame, the parameters of the odometry bias calibration for each vehicle and different types of grid maps. we evaluated the performance of the proposed mapping system on logs containing high and low dynamics in real-world scenarios. the logs were collected using the available sensors in the av over a period of two years. the new odometry bias calibration method was compared against an odometry calibration based on particle swarm optimization (pso) proposed by mutz et al. [13]. the results indicate that the new method provides better calibration values in all logs. the new mapping features were also analyzed through mapping and map merging experiments. the results show that the proposed system is able to correctly estimate the av poses in a global reference frame, and to correctly merge local maps without drifts and misalignment for all logs. the remaining sections are organized as follows. in sect. 2, we present related work. in sect. 3, we detail the proposed mapping system. in sect. 4, we describe the experimental methodology and, in sect. 5, we discuss experimental results. finally, in sect. 6, we close with our conclusions and directions for future work. 2 related work progress on the simultaneous localization and mapping (slam) problem can be measured in terms of a wide variety of algorithms, techniques and theoretical frameworks that have been developed in last decades. however, unresolved questions remain such as robustness and outlier rejection for long-term operations [4]. the map merging problem addresses theses open issues in different contexts of application, either with multiple robots running synchronously or with only one robot gathering data during multiple sessions. while the first is focused in collaborative strategies to efficient mapping, the later is oriented to long-term map maintenance in order to accommodate infrastructural changes along time. both share the same main goal which is to produce a final aggregated map alongside the robot trajectories for all sessions. the common approach in multi-robot scenarios is to fetch all maps using map-matching to find the relative map transformations and merge all maps into one final map alongside the corresponding trajectory of each robot in that map. a system for real-time multi-robot collaborative slam is presented in deutsch et al. [7]. the distributed robots use their own slam system to build local maps, but they also share information (features from images) with a central system, which builds a global pose graph including information from all robots. after the graph optimization, the central system sends the results to all robots, so they keep a consistent local position and map estimates. in contrast, in our solution, maps are built before autonomous operation on outdoor environments. ours can handle any map size as it depends only in the available storage. sun et al. [19] convert the map merging problem into an image registration problem. when two grid maps contains overlapping areas, their approach tries to find the best transformation which aligns the maps. the algorithm extract features from both maps using the harris corner detector and it uses the selected features to calculate a initial transformation matrix which is later refined by an iterative algorithm. in contrast, our method finds the common reference frame through the graph optimization by making usage of gps and lidars sensors, hence the proper alignments are achieved before the map construction. the approach in single-robot multi-session scenarios is to use the all maps using map-matching to find the relative map transformations and merge all maps into one final map alongside the corresponding trajectory of each robot in that map a slam system using normal distribution transform (ndt) maps is presented in einhorn and gross [9]. the authors propose a modification in the ndt mapping to handle free-space measurements which makes them better suited for slam. the map merging is achieved by the alignment of the ndt maps through registration techniques. bonanni et al. [3] present a slam method that considers the map as a graph of point clouds. in order to merge two maps, it visits each node in the first graph and tries to find the best match in the second graph. normal iterative closest point (nicp) [17] is used to find the registration between the point clouds. the method is similar to our solution as point clouds are used to provide the correct relationship between sessions. we use gps and a nearest neighbor algorithm to find the best candidates in the second graph. the work of ding et al. [8] is meant for large scale outdoor environments, as ours, and use 3d lidar for loop closure detection whilst we use gps. their system is capable of building maps using multi-session data as ours. their approach splits maps into submaps and use an efficient method for loop closure detection using feature vectors and a loop closure database using the kd-tree algorithm. their method also takes care of loop closure outliers and actively detects changes in the environment (high and low dynamics). 3 large-scale mapping system in this section, we describe the proposed full slam-based mapping system. despite being able to build many types of grid maps, the next sections will focus only on ogms. figure 3 shows the architecture of the proposed mapping system. the pre-processing and the hypergraph builder modules are responsible for handling low-level information coming from sensors, removing undesired measurements, and building the hypergraph. the hypergraph optimizer is responsible for estimating the av poses along the sessions while also calibrating the odometry bias. the mapper module builds and merges the maps. fig. 3. system overview. full size image 3.1 pre-processing the pre-processing module sorts all measurements by their timestamps and also discards measurements in which the av is below a minimum speed ms. this module also removes a specific error on gps measurements as illustrated in fig. 4. figure 4(b) shows the poor results after increasing the gps error variance in a try to solve this issue. our solution is illustrated on fig. 4(c). the gps measurements are clustered together if the distance between immediate neighbors is less than neighbordistance. after clustering, all groups containing fewer elements than neighborquantity are removed. fig. 4. (a) gps error. (b) increasing variance from left to right. (c) clustering gps and removing small groups. full size image 3.2 hypergraph building nodes in the hypergraph represent av poses and odometry biases. edges represent constraints between the nodes. a pose node \(\boldsymbol{n}_{i}\) is added to the hypergraph for each sensor measurement \(\boldsymbol{z}_{i}\). \(\boldsymbol{n}_{i}\) refers to an unknown pose \(\boldsymbol{x}_{i}\): $$\begin{aligned} \boldsymbol{n}_{i} = \boldsymbol{x}_{i} = \left( x_{i}, y_{i}, \theta _{i} \right) \end{aligned}$$ (1) where \((x_{i}, y_i)\) is the position of the vehicle in the ground plane and \(\theta _{i}\) is its heading. for each log, the hypergraph builder creates odometry edges connecting each pair \(\boldsymbol{n}_{i}\) and \(\boldsymbol{n}_{i+1}\). this procedure guarantees that all nodes in the hypergraph are connected by odometry measurements, therefore, there is no need to synchronize the sensors since the odometry is used o join all nodes from the same log. while traversing the nodes in the graph, the system maintains the current command \(\boldsymbol{u}_{i} = (v_{i}, \varphi _{i}, \vardelta t)\). \(\vardelta t\) is the time difference between \(\boldsymbol{n}_{i}\) and \(\boldsymbol{n}_{i+1}\). the velocity \(v_{i}\) and steering angle \(\varphi _{i}\) are obtained from the last odometry measurement and they are updated at each new odometry measurement. the odometry edge indicates a constraint between the nodes \(\boldsymbol{n}_{i}\) and \(\boldsymbol{n}_{i+1}\) through the error function \(\boldsymbol{e}_{}^{o}(\boldsymbol{x}_i, \boldsymbol{u}_i)\). we also simplify the notation by encoding the involved quantities in the indices of the error function as in [12]: $$\begin{aligned} \boldsymbol{e}_{}^{o}(\boldsymbol{x}_i, \boldsymbol{u}_i) {\mathop {=}\limits ^{def.}} \boldsymbol{e}_{i}^{o}(\boldsymbol{x}, \boldsymbol{u}) \end{aligned}$$ (2) the odometry constraint between each node pair is defined as: $$\begin{aligned} \boldsymbol{e}_{i}^{o}(\boldsymbol{x}, \boldsymbol{u}) = (\boldsymbol{x}_{i + 1} \ominus \boldsymbol{x}_{i}) \ominus k(\boldsymbol{u}_{i}) \end{aligned}$$ (3) where \(\ominus \) is the inverse of the motion composition operator \(\oplus \) as defined in [18], and k(.) uses the ackerman steering geometry with understeer corrections [2] to compute a relative movement given the command \(\boldsymbol{u}_{i}\in \boldsymbol{u}\), and \(\boldsymbol{u}\) is a vector containing all commands. all odometry edges are combined in: $$\begin{aligned} \boldsymbol{j}^{o} \left( \boldsymbol{x}, \boldsymbol{u}\right) = \sum _{i}^{n} \boldsymbol{e}_{i}^{o}(\boldsymbol{x}, \boldsymbol{u}^{\intercal })^{\intercal } \boldsymbol{\hat{\varomega }}_{i}^{o} \boldsymbol{e}_{i}^{o}(\boldsymbol{x}, \boldsymbol{u}^{\intercal }) \end{aligned}$$ (4) where \(\hat{\varomega }_{i}^{o}\) is the odometry information matrix at node \(\boldsymbol{n}_{i}\) projected through the non-linear function k(.) via the unscented transformation [22]. each node \(\boldsymbol{n}_{i}\) created by a gps measurement \(\boldsymbol{z}_{i}^{gps}\) receives a unary edge containing the error: $$\begin{aligned} \boldsymbol{e}_{i}^{gps} \left( \boldsymbol{x}\right) = \boldsymbol{x}_{i} \ominus \boldsymbol{z}_{i}^{gps} \end{aligned}$$ (5) all gps edges constraints are combined in the total gps error: $$\begin{aligned} \boldsymbol{j}^{gps} \left( \boldsymbol{x}\right) = \sum _{i} \boldsymbol{e}_{i}^{gps}(\boldsymbol{x})^{\intercal } \boldsymbol{\varomega }_{i}^{gps} \boldsymbol{e}_{i}^{gps}(\boldsymbol{x}) \end{aligned}$$ (6) where \(\varomega _{i}^{gps}\) is the inverse of the gps covariance matrix at node \(\boldsymbol{n}_{i}\). we use lidar and gps to compute intra-session loop closures as in [13]. however, we use an ad-hoc method to reduce the number of edges in the graph by only considering a new loop closure if it is at a certain distance from the previous one. we also extend this approach to handle inter-session loop closures. inter-session loop closures provide the required relationships between the av poses in multiple logs. a loop closure edge computes the following error: $$\begin{aligned} \boldsymbol{e}_{i,j}^{l}(\boldsymbol{x}) = (\boldsymbol{x}_{j} \oplus \boldsymbol{s}^{l}) \ominus (\boldsymbol{x}_{i} \oplus \boldsymbol{s}^{l}) \ominus icp_{i, j}^{l} \end{aligned}$$ (7) where \(\boldsymbol{x}_{i}\) and \(\boldsymbol{x}_{j}\) are the av poses that are considered as a loop closure, \(\boldsymbol{s}^{l}\) is the pose of the lidar sensor in the av reference frame and \(icp_{i,j}^{l}\) is a linear transform that aligns the point cloud \(\boldsymbol{z}_{i}^{l}\) with the point cloud \(\boldsymbol{z}_{j}^{l}\). we use a custom method based on the generalized iterative closest point (gicp) [16] algorithm to find the transform. the error below combines the constraints of all loop closure edges: $$\begin{aligned} \boldsymbol{j}^{l} \left( \boldsymbol{x}\right) = \sum _{i} \boldsymbol{e}_{i,j}^{l}(\boldsymbol{x})^{\intercal } \boldsymbol{\varomega }_{i,j}^{l} \boldsymbol{e}_{i,j}^{l}(\boldsymbol{x}) \end{aligned}$$ (8) where \(\varomega _{i}^{l}\) is the inverse gicp covariance matrix. mutz et al. [13] showed that odometry measurements are subject to biases. this error can be propagated to the edges in the graph if not addressed. the authors proposed a pso-based optimization method to calibrate the odometry bias before building the graph. the calibration is approximated by the following linear functions: $$\begin{aligned} v_{i}^{'}= v_{i} b_{mult}^{v} \end{aligned}$$ (9) $$\begin{aligned} \varphi _{i}^{'} = \varphi _{i}b_{mult}^{\varphi }+ b_{add}^{\varphi } \end{aligned}$$ (10) where \(b_{mult}^{v}\) is the velocity multiplicative bias, \(b_{mult}^{\varphi }\) and \(b_{add}^{\varphi }\) are the steering wheel angle multiplicative and additive bias. the authors employed a pso-based optimization method using a fitness function, which computes the difference between gps and dead reckoning poses. each particle contains the odometry bias parameters, \(\boldsymbol{b} = \left( b_{mult}^{v}, b_{mult}^{\varphi }, b_{add}^{\varphi }\right) \), and the best parameters are used to update all odometry measurements before the av poses optimization. we propose a new method, which is performed directly in the hypergraph optimization. a bias calibration node \(\boldsymbol{n}^{b}_{j:k}\) contains the parameters of the eqs. 9 and 10: $$\begin{aligned} \boldsymbol{n}^{b}_{j:k} = (b_{mult}^{v}, b_{mult}^{\varphi }, b_{add}^{\varphi })_{j:k} \end{aligned}$$ (11) the range subscript \(j:k\) indicates that a single bias calibration node \(\boldsymbol{n}^{b}_{j:k}\) can be related to any subgroup of av poses \(\boldsymbol{x}\) containing all consecutive poses from j to k. as each session generates its hypergraph and each hypergraph have at least one bias calibration node, the system can calibrate the odometry bias on each session. each pair \(\left( \boldsymbol{n}_{i}, \boldsymbol{n}_{i+1} \right) \) is linked to a calibration node, forming a hyperedge. we use the following error to calibrate the odometry bias: $$\begin{aligned} \boldsymbol{e}_{i, j:k} \left( \boldsymbol{x}, \boldsymbol{b}, \boldsymbol{u} \right) = \left( \boldsymbol{x}_{i+1} \ominus \boldsymbol{x}_{i} \right) \ominus \overline{k}(\boldsymbol{b}_{j:k}, \boldsymbol{u}_{i}, \vardelta t) \end{aligned}$$ (12) where \(\overline{k}(.)\) is a function that takes the bias parameters \(\boldsymbol{b}_{j:k}\) at \(\boldsymbol{n}^{b}_{j:k}\) to update the command \(\boldsymbol{u}_{i}\) through eqs. 9 and 10, and then it calls k(.) to estimate a relative displacement. the global odometry bias calibration error is defined as: $$\begin{aligned} \boldsymbol{j}^{b}&(\boldsymbol{x}, \boldsymbol{b}, \boldsymbol{u}) \nonumber \\&=\,\sum _{j:k} \sum _{i = j}^{k-1} \boldsymbol{e}_{i, j:k} \left( \boldsymbol{x}, \boldsymbol{b}, \boldsymbol{u} \right) ^{\intercal } \boldsymbol{\hat{\varomega }}_{i}^{b} \boldsymbol{e}_{i, j:k} \left( \boldsymbol{x}, \boldsymbol{b}, \boldsymbol{u} \right) \end{aligned}$$ (13) where \(\hat{\varomega }_{i}^{b}\) is the odometry information matrix projected through \(\overline{k}(.)\) via the unscented transformation. in this case, the projection must be updated when the bias changes substantially. fig. 5. graph containing edges from odometry, gps and loop closure. full size image fig. 6. odometry bias calibration node and hyperedges. full size image the resulting hypergraph contains all av pose nodes, the odometry bias calibration nodes and all required edges to impose the constraints defined in eqs. 4, 6, 8 and 13. figure 5 illustrates the relationships between the elements of the hypergraph, including pose nodes, odometry edges, a gps edge, and a loop closure edge. figure 6 displays an odometry bias calibration node and some calibration hyperedges represented by regions with different colors for better visualization. 3.3 hypergraph optimization the hypergraph optimization is an iterative process limited by a pre-defined number of iterations and it is separated in two main steps. the first step maximizes only the likelihood of the av poses by combining eqs. 4, 6 and 8 while excluding eq. 13: $$\begin{aligned} \boldsymbol{j}_{1} \left( \boldsymbol{x}, \boldsymbol{u}\right) = \boldsymbol{j}^{o} \left( \boldsymbol{x}, \boldsymbol{u}\right) + \boldsymbol{j}^{gps} \left( \boldsymbol{x}\right) + \boldsymbol{j}^{l} \left( \boldsymbol{x}\right) \end{aligned}$$ (14) the second step swaps the node update strategy by handling only the odometry bias calibration: $$\begin{aligned} \boldsymbol{j}_{2} = \boldsymbol{j}^{b} (\boldsymbol{x}, \boldsymbol{b}, \boldsymbol{u}) \end{aligned}$$ (15) when the second step reaches a defined number of iterations, the new odometry biases are used to update all edges related to eq. 7 by first updating the corresponding commands \(\boldsymbol{u}\) using eqs. 9 and 10. after these updates, the optimization process restarts from the first step. after repeating the two steps for a desired number of iterations, the optimization returns the avs poses and the odometry calibration biases. 3.4 large-scale environment mapping the map is represented in log-odds to avoid numerical instabilities [21]. the log-odds of a given cell \(\boldsymbol{m}_{c}\) in the map is defined as: $$\begin{aligned} l_{c} = \log {\frac{p \left( \boldsymbol{m}_{c}| \boldsymbol{x}, \boldsymbol{z}\right) }{1 p \left( \boldsymbol{m}_{c}| \boldsymbol{x}, \boldsymbol{z}\right) }} \end{aligned}$$ (16) where \(p \left( \boldsymbol{m}_{c}| \boldsymbol{x}, \boldsymbol{z}\right) \) is the probability of \(\boldsymbol{m}_{c}\) being occupied given \(\boldsymbol{x}\) and \(\boldsymbol{z}\). initially, all grid cells are set to an invalid state (e.g. -1.0) and, at each time step, if a cell \(\boldsymbol{m}_{c}\) is in the lidar perceptual field then it’s current log-odds is updated: $$\begin{aligned} l_{i,c} = l_{i-1,c} + ism \left( \boldsymbol{m}_{c}, \boldsymbol{x}_{i}, \boldsymbol{z}_{i}^{l}\right) l_{0} \end{aligned}$$ (17) where \(l_{i-1,c}\) is the previous log-odds of the cell \(\boldsymbol{m}_{c}\), \(l_{0}\) is the map prior probability, \(\boldsymbol{z}_{i}^{l}\) is a lidar measurement and ism(.) is the inverse sensor model function [21]. 3.5 large-scale map merging the mapper module merges the grids maps in an incremental approach. a first map is created using any of the resulting output files from the hypergraph optimization. while the mapping process evolves, the system saves how many times each cell \(\boldsymbol{m}_{c}\) is reached by the lidar rays in the current session. after the first map is complete, the system changes the rule defined in eq. 17 and then only the cells with the hit counter below a threshold value k are updated: $$\begin{aligned} l_{i, c} = {\left\{ \begin{array}{ll} l_{i-1,i} + ism \left( \boldsymbol{m}_{c}, \boldsymbol{x}_{i}, \boldsymbol{z}_{i}^{l}\right) l_{0}, &{} hits \left( \boldsymbol{m}_{c}\right) < k \\ l_{i-1,i}, &{} otherwise \end{array}\right. } \end{aligned}$$ (18) we assume that the lidar sensor is reliable and after enough hits the system can freeze the resulting log-odds. this procedure can unintentionally keep the low dynamics fixed in a previous map, but we can easily return to the original update rule defined in eq. 17 if the difference in time between the sessions are greater than that of parameterized in sect. 4. assuming that the main changes in the environment infrastructure takes a time greater than that, then the system can update the current map while discarding most of the undesired changes in the next sessions. 4 experimental methodology in this section, we describe each one of the recorded sessions, and the metrics used to compare the odometry calibration methods and to evaluate the estimated av poses. we also indicate all the experiments that evaluates the main features of the new proposed large-scale mapping system. fig. 7. satellite view of the environments. (a) university campus beltway (red), parking lot 1 (yellow), parking lot 2 (dark blue) and parking lot 3 (cyan). (b) route from the university campus to another city (light blue). (color figure online) full size image we collected data using the iara platform. the av was conducted by a human along the beltway of the university campus and within parking lots in it’s vicinity. figure 7(a) shows the beltway (red) and the parking lots 1 (yellow), 2 (dark blue) and 3 (cyan). log ub1 was acquired along the beltway at night, in the clockwise direction and without moving obstacles. log ub2 was recorded in the morning, in counterclockwise direction, with moving objects and many parked vehicles. the logs pl1, pl2 and pl3 are related to the parking lots. finally, fig. 7(b) shows the route from the campus to another city in which the av crosses highways, rural and urban streets and a long bridge. the trvl log was recorded along this route which has about 72 km of extension. 4.1 experiments we carried out a series of experiments to evaluate the odometry calibration and the map building and merging proposed methods. the main parameters of the proposed mapping system used in our experiments are the minimum speed (ms) set to 0.01 ms in the speed filtering, the neighbor distance (nd) and the neighbor quantity (nq) thresholds used in the gps clustering are set to 1.2 m and 40, respectively. the cell hits threshold (k) represented in eq. 18 is set to 64. the time distance between sessions (istd) is set to 1month for low dynamics consideration. odometry bias calibration. in this experiment, we compare the pso-based optimization [13] and the proposed hypergraph optimization methods for calibrating odometry biases. we use the mean absolute error (mae) metric to compare the paths obtained by both calibration methods (pso-based and hypergraph) against the gps. a better odometry calibration should produce lower mae values as the resulting path should better approximate gps measurements. map building and merging. in this experiment, we compare the estimated av poses with the gps measurements for each log using the mae metric. we constructed a single map using all logs in order to verify the merging capabilities of the proposed system. we illustrate same obstacles viewed in different visits to the same region with different colors in order to qualitatively evaluate the results of the map merging. 5 results and discussions 5.1 odometry bias calibration table 1 compare the mae results of the calibration using both methods. in all cases, the novel approach has found better parameters since it produces lower mae values and the recovered paths are closer to gps than both raw odometry and pso. figure 8 compares gps and raw odometry with the paths computed by the two odometry bias calibration methods in the trvl log. table 1. odometry bias calibration using pso and hypergraph optimization.full size table fig. 8. odometry calibration trvl. full size image fig. 9. map merging. (a) errors in the ogm (ub1 and ub2). (b) enabling inter-session loop closures. (c) complete map. full size image 5.2 map building and merging given that our new system exploits all relevant sensor data, additional filtering strategies and lidar odometry, it was capable of handling the trvl dataset and providing the maps in that large-scale context. figure 9(a) illustrates the absence of a common reference frame while merging ub1 and ub2 without the inter-session loop closures while also using 18. red pixels are the obstacles found in ub1 and green pixels are obstacles in ub2. similar errors occur when we merge the remaining logs without using the inter-session loop closure to estimate a good reference frame. figure 9(b) illustrates the removal of the errors after activating the inter-session loop closures between ub1 and ub2. figure 9(c) presents the final merged map including all parking lots. 6 conclusions and future work in this work, we presented a new system for long-term map maintenance. the system uses multi-session data with the purpose of building and updating grid maps for posterior autonomous navigation and localization. the system deals with different types of sensors and can handle typical errors found in noisy sensor data. the hypergraph builder module includes intra-session and inter-session loop closures in order to estimate the av poses in a global coordinate frame. the system also uses a new odometry bias calibration method to turn the odometry more reliable in the entire process. the mapper module uses a custom method to freeze highly observed cells and so handle high and low dynamics in the map merging. we evaluated the system using datasets collected in an university campus and in a travel to another city. experiments indicate that the new odometry bias calibration method surpass a previous pso-based and also shows that the system can build and merge submaps into a global map using inter-session loop closures. equation 7 is used in kümmerle et al. [12] to estimate the 2d pose of some laser scanner sensors directly in the hypergraph optimization. we did not obtain same quality results for the velodyne hdl-32e in our av. in a future work, we can investigate proper methods to calibrate the extrinsic of our lidar and cameras. references badue, c., et al.: self-driving cars: a survey. expert syst. appl. 165, 113816 (2020) article  google scholar  bergman, w.: the basic nature of vehicle understeer-oversteer. sae trans. 74, 387–422 (1966) google scholar  bonanni, t.m., corte, b.d., grisetti, g.: 3-d map merging on pose graphs. ieee robot. autom. lett. 2(2), 1031–1038 (2017) article  google scholar  cadena, c., et al.: past, present, and future of simultaneous localization and mapping: toward the robust-perception age. ieee trans. robot. 32(6), 1309–1332 (2016) article  google scholar  cardoso, v., et al.: a model-predictive motion planner for the iara autonomous car. in: 2017 ieee international conference on robotics and automation (icra), pp. 225–230 (2017) google scholar  de paula veronese, l., et al.: a light-weight yet accurate localization system for autonomous cars in large-scale and complex environments. in: 2016 ieee 19th international conference on intelligent transportation systems (itsc), pp. 520–525 (2016) google scholar  deutsch, i., liu, m., siegwart, r.: a framework for multi-robot pose graph slam. in: 2016 ieee international conference on real-time computing and robotics, rcar 2016, pp. 567–572 (2016) google scholar  ding, x., wang, y., yin, h., tang, l., xiong, r.: multi-session map construction in outdoor dynamic environment. in: 2018 ieee international conference on real-time computing and robotics, rcar 2018, pp. 384–389 (2019) google scholar  einhorn, e., gross, h.m.: generic ndt mapping in dynamic environments and its application for lifelong slam. robot. auton. syst. 69, 28–39 (2015) article  google scholar  guidolini, r., badue, c., berger, m., de paula veronese, l., de souza, a.f.: a simple yet effective obstacle avoider for the iara autonomous car. in: 2016 ieee 19th international conference on intelligent transportation systems (itsc), pp. 1914–1919 (2016) google scholar  guidolini, r., de souza, a.f., mutz, f., badue, c.: neural-based model predictive control for tackling steering delays of autonomous cars. in: 2017 international joint conference on neural networks (ijcnn), pp. 4324–4331 (2017) google scholar  kümmerle, r., grisetti, g., burgard, w.: simultaneous parameter calibration, localization, and mapping. adv. robot. 26(17), 2021–2041 (2012) article  google scholar  mutz, f., veronese, l., oliveira-santos, t., aguiar, e., cheein, f.a., de souza, a.: large-scale mapping in complex field scenarios using an autonomous car. expert syst. appl. 46, 439–462 (2016) google scholar  possatti, l., et al.: traffic light recognition using deep learning and prior maps for autonomous cars, pp. 1–8 (july 2019) google scholar  sarcinelli, r., et al.: handling pedestrians in self-driving cars using image tracking and alternative path generation with frenét frames. comput. graph. 84, 173–184 (2019) article  google scholar  segal, a., hähnel, d., thrun, s.: generalized-icp (june 2009) google scholar  serafin, j., grisetti, g.: nicp: dense normal based point cloud registration. in: 2015 ieee/rsj international conference on intelligent robots and systems (iros), pp. 742–749 (2015) google scholar  smith, r., self, m., cheeseman, p.: estimating uncertain spatial relationships in robotics. in: cox i.j., wilfong g.t. (eds.) autonomous robot vehicles, vol. 1, pp. 167–193. springer, new york (july 1990) google scholar  sun, y., sun, r., yu, s., peng, y.: a grid map fusion algorithm based on maximum common subgraph. in: 2018 13th world congress on intelligent control and automation (wcica), vol. 2018-july, pp. 58–63. ieee (july 2018) google scholar  tabelini torres, l., et al.: effortless deep training for traffic sign detection using templates and arbitrary natural images. 2019 international joint conference on neural networks (ijcnn), pp. 1–7 (july 2019) google scholar  thrun, s., burgard, w., fox, d.: probabilistic robotics (intelligent robotics and autonomous agents). the mit press, cambridge (2005) google scholar  uhlman, j.: dynamic map building and localization for autonomous vehicles. ph.d. thesis, university of oxford (1994) google scholar  download references author information authors and affiliations departamento de informática, universidade federal do espírito santo, av. fernando ferrari 514, goiabeiras, vitória, 29075-910, brazil josias oliveira, pedro azevedo, thiago oliveira-santos, alberto f. de souza & claudine badue coordenação de informática, instituto federal do espírito santo, es-010 km-6.5, manguinhos, serra, 29173-087, brazil filipe mutz coordenadoria de engenharia mecânica, instituto federal do espírito santo, av. morobá 248, bairro morobá, aracruz, 29192-733, brazil avelino forechi authors josias oliveiraview author publications search author on:pubmed google scholar filipe mutzview author publications search author on:pubmed google scholar avelino forechiview author publications search author on:pubmed google scholar pedro azevedoview author publications search author on:pubmed google scholar thiago oliveira-santosview author publications search author on:pubmed google scholar alberto f. de souzaview author publications search author on:pubmed google scholar claudine badueview author publications search author on:pubmed google scholar corresponding authors correspondence to josias oliveira , alberto f. de souza or claudine badue . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper oliveira, j. et al. (2021). long-term map maintenance in complex environments. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_11 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords mapping slam map maintenance map merging autonomous vehicles publish with us policies and ethics profiles alberto f. de souza view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature the future of ai: neat or scruffy? | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper the future of ai: neat or scruffy? conference paper first online: 28 november 2021 pp 177–192 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) the future of ai: neat or scruffy? download book pdf download book epub bernardo gonçalves  orcid: orcid.org/0000-0003-2794-847810 & fabio gagliardi cozman  orcid: orcid.org/0000-0003-4077-493510  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1377 accesses 4 citations abstract the “neat” and “scruffy” portraits have long been painted to describe viewpoints, styles of reasoning and methodologies in ai research. essentially, the neats defend techniques based on first principles and grounded in mathematical rigor, while the scruffies advocate diversity within cognitive architectures, sometimes meant to be models of parts of the brain, sometimes just kludges or ad-hoc pieces of engineered code. the recent success of deep learning has revived the debate between these two approaches to ai; in this context, some natural questions arise. how can we characterize, and how can we classify, these positions given the history of ai? more importantly, what is the relevance of these positions for the future of ai? how should ai research be pursued from now on, neatly or scruffly? these are the questions we address in this paper, resorting to historical analysis and to recent research trends to articulate possible ways to allocate energy so as to take the field to maximal fruition. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others quo vadis artificial intelligence? article open access 07 march 2022 can artificial intelligence lead us to genuine virtue? a confucian perspective chapter © 2021 artificial intelligence—an overview of the legal status chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence logic in ai intelligence philosophy of artificial intelligence symbolic ai artificial intelligence 1 introduction which architecture should be implemented in a machine to make it best reproduce human intelligence in relevant intellectual tasks? in the history of ai, two answers have often been given to this question by different researchers, who have been labeled either as “neats” or “scruffies.” capturing viewpoints, styles of reasoning and methodologies, these labels have often been used as caricatures of the ai researcher—the analytical, sensible, dry vs. the empirical, messy, creative. but is this a fair account of what is at stake? the debate about which methodologies are best for ai and what ai actually is or should be is an old one. but the advent of deep learning brought it back to the front stage of the discussion about the present and the future of ai. with considerable success in image, speech and natural language processing, ai has increased its social impact. arguably the question about which architecture is most promising for ai underwent a twist, and it now comes with an attached social concern about the future. which future is best for ai? is it one developed around a (super)intelligence that emerges through learning and that makes its judgements with little oversight? or is it one based on a useful yet more predictable ai that is managed more closely? possible answers to this question are strongly tied to different research directions that have been proposed by the neats, on one side, and by the scruffies, on the other side. recently hybrid or the so-called neurosymbolic ai has also been proposed as an attempt to leverage on the strengths of both sides. this approach to ai needs to be tested by time and deserves further study. in any case, it shall also benefit from a reflection upon its neat vs. scruffy origins. in this paper we contribute to the discussion with: 1. an in-depth examination of the past through a review of notable neat and scruffy positions in the history of ai up to the present; 2. the identification of three types of attitudes towards ai research from neats and scruffies; 3. the formulation of their implications for the future of ai from the point of view of both research and its social impact. in sects. 2, 3 we shall study positions and uses of the “neat” and the “scruffy” terminology through the history of ai in order to arrive in sect. 4 at three types of attitude towards ai research. our contributed scheme is aimed at clarifying which notions of the terms have been available. in sect. 5 we study implications of the three approaches for the future of ai. we conclude the paper in sect. 6. 2 the “neats” vs. “scruffies” debate in the history of ai we start with a review that partly extends a recent survey paper [3]. our exposition is not chronological. 2.1 marvin minsky (1985–1995) minsky did not explicitly refer to the “neat” vs. “scruffy” dichotomy yet he posited an influential position in the discussion. in his 1985 book the society of mind and later, minsky appealed to the complexity of the human mind and/or brain to justify that his explanations “rarely go in neat, straight lines from start to end.” he associated it with the very nature of the mind: perhaps the fault is actually mine, for failing to find a tidy base of neatly ordered principles. but i’m inclined to lay the blame upon the nature of the mind: much of its power seems to stem from just the messy ways its agents cross-connect. if so, that complication can’t be helped; it’s only what we must expect from evolution’s countless tricks. [18, p. 18] a few years later minsky developed the point in connection with the brain: the brain’s functions simply aren’t based on any small set of principles. instead, they’re based on hundreds or perhaps even thousands of them. in other words, i’m saying that each part of the brain is what engineers call a kludge — that is, a jury-rigged solution to a problem, accomplished by adding bits of machinery wherever needed, without any general, overall plan: the result is that the human mind — which is what the brain does — should be regarded as a collection of kludges. the evidence for this is perfectly clear: if you look at the index of any large textbook of neuroscience, you’ll see that a human brain has many hundreds of parts — that is, subcomputers — that do different things. why do our brains need so many parts? surely, if our minds were based on only a few basic principles, we wouldn’t need so much complexity [19]. so according to minsky, one is led to think, ai systems shall be untidy like the brain. but did minsky suggest that ai as a discipline shall be messy as well? this question will be revisited in a discussion with yann lecun in subsect. 2.6. 2.2 nils nilsson (2009) nilsson addressed minsky’s position that the brain is a kludge so an ai system should be likewise. in his 2009 book the quest for artificial intelligence, nilsson acknowledged the wide disagreement in the field about what ai research should be like. he observed: of course, just because the brain is a kludge does not mean that computer intelligences have to be. nevertheless, some ai researchers favored systems consisting of collections of experimentally derived, ad hoc routines designed to solve specific problems. these people called themselves “scruffies” to distinguish themselves from the “neats” who favored programs based on theoretically based principles. (these terms were apparently first used by roger schank in the 1970s to contrast his approach to building natural language processing systems with the more theoretically based work of mccarthy and others.) in his keynote address at the 1981 annual meeting of the cognitive science society, robert abelson compared the two camps by saying “the primary concern of the neat is that things should be orderly and predictable while the scruffy seeks the rough-and-tumble of life as it comes [...].” [20, p. 417] nilsson thus resumed to put his own opinion over the issue: i believe that both neats and scruffies are needed in a field as immature as ai is. scruffies are better at exploring frontiers outside the boundaries of well-established theory. neats help codify newly gained knowledge so that it can be taught, written about, and thus remembered. (ibid.) so it seems that nilsson supported the coexistence of both approaches in ai research, each one in its own camp. he suggested that this is particularly important while ai is still a young endeavor. 2.3 herbert simon (1972) simon had received in 1978 the nobel prize in economics for identifying the limited and messy nature of decision making as opposed, say, to idealized metrics of utility. pamela mccorduck interviewed simon extensively. she reported: simon also took up the problem that had divided the ai (and cognitive science) community from the beginning: is thinking best viewed as a process of reasoning from premises (the neats?) or as a process of selective search through a maze (the scruffies)? he had no final answer to that when he died in 2001, though we can assume he leaned toward the scruffy point of view, given his book, human problem solving, that he’d published with allen newell in 1972. [15, p. 452-3] in his theory of human problem solving with allen newell, simon emphasized the role of heuristics. he thus described their famous logic theorist (lt) program: there were important differences between lt’s processes and those used by human subjects to solve similar problems. nevertheless, in one fundamental respect that has guided all the simulations that have followed lt, the program did indeed capture the central process in human problem solving: lt used heuristic methods to carry out highly selective searches, hence to cut down enormous problem spaces to sizes that a slow, serial processor could handle. selectivity of search, not speed, was taken as the key organizing principle [...]. heuristic methods that make this selectivity possible have turned out to be the central magic in all human problem solving that has been studied to date. [29, p. 147] from a mathematical point of view, heuristics may be considered ad-hoc techniques as it is hard to assign to them accurate theoretical guarantees. this is probably related to what mccorduck meant when classing simon (and his lt co-designed with newell) as an example of scruffiness in ai. now, if the event in the northern summer of 1956 in dartmouth marks the birth of ai as a discipline, then the scruffy lt can be considered the first ai program. 2.4 john mccarthy (1958) mccarthy was the primary organizer of the dartmouth workshop [15]. but it was newell and simon who seem to have taken most of the attention at dartmouth, as they were the ones that had something real to show—the lt. mccarthy’s reaction would come soon. in 1958, he compared the lt with his own program: the advice taker is a proposed program for solving problems by manipulating sentences in formal languages. the main difference between it and other programs or proposed programs for manipulating formal languages (the logic theory machine of newell, simon and shaw and the geometry program of gelernter) is that in the previous programs the formal system was the subject matter but the heuristics were all embodied in the program. in this program the procedures will be described as much as possible in the language itself and, in particular, the heuristics are all so described. [14, no emphasis added] mccarthy’s reservations with respect to newell and simon’s lt had to do with the organization of the knowledge manipulated by the program. for mccarthy, this knowledge—about the world and about the problems the program is expected to solve—should be held transparent in an expressive language. so the language should be “most likely a part of the predicate calculus” (ibid.). mccarthy’s concern with the knowledge representation language to be used in an ai program seems to have been the core of his research agenda. further on in his 1958 text he outlined five features that, in his opinion, a system “which is to evolve intelligence of human order” should have at least. he then summarized: “we base ourselves on the idea that: in order for a program to be capable of learning something it must first be capable of being told it” (no emphasis added). mccarthy had also expressed back then his hope to collaborate with minsky: the design of this system [so-called “the advice taker”] will be a joint project with marvin minsky, but minsky is not to be held responsible for the views expressed here. [14] later on, as of 1996, mccarthy updated his text with a note “[t]his was wishful thinking,” for “minsky’s approach to ai was quite different.” in fact, mccarthy is seen as perhaps the main advocate of neatness in ai. his divergence with minsky seems strongly related with our main subject in this paper. and yet it is still hard to pinpoint what exactly their differences were. before we conclude this review and proceed to address the issue, we shall refer to another ai textbook and to the advent of deep learning as the most recent success in ai. 2.5 russell and norvig (1995–2020) in the 1995 (first) edition of their textbook [24], russell and norvig reported that ai research had just seen a “sea change in both [its] content and [its] methodology” (p. 25). they referred to “the field of speech recognition” as a paradigmatic example, as it went through a shift from the use of “ad-hoc and fragile architectures and approaches” to eventually find in hidden markov models (hmm’s) an approach that is “based on a rigorous mathematical theory” and whose models “are generated by a process of training on a large corpus of real speech data.” this, the authors observed, allowed “speech researchers to build on several decades of mathematical results developed in other fields,” and ensured “that the performance is robust.” they further remarked: some have characterized this change as a victory of the neats — those who think that ai theories should be grounded in mathematical rigor — over the scruffies — those who would rather try lots of ideas, write some programs, and then assess what seems to be working. both approaches are important. a shift towards increased neatness implies that the field has reached a level of stability and maturity. (whether that stability will be disrupted by a new scruffy idea is another question.) [24, p. 25, note 17, no emphasis added] the last phrase, as appearing in this 1995 edition of the book, has been replaced in the 2020 (fourth and latest) edition by this note: “the present emphasis on deep learning may represent a resurgence of the scruffies” [25, p. 24, note 14]. it seems that, for these authors, the advent of deep learning is a result of scruffiness while that of hmm and bayesian networks are results of neatness. but to what extent is deep learning not based on mathematical theory? 2.6 yann lecun (2018) with the success and also the hype around deep learning, there has been a criticism that machine learning techniques are pseudo-science and that their empirical results are not explainable nor reproducible. a report in science magazine [9] covered the nips 2017 “test-of-time award” keynote address by ai researchers ali rahimi and benjamin recht [23], who made the case that various aspects of machine learning algorithms are so badly understood, even by engineers, that they amount to alchemy. this has triggered a polemic with co-recipient of the 2018 turing award and facebook’s vp and chief ai scientist yann lecun, who reacted to their talk on the internet: in the history of science and technology, the engineering artifacts have almost always preceded the theoretical understanding: the lens and the telescope preceded optics theory, the steam engine preceded thermodynamics, the airplane preceded flight aerodynamics, radio and data communication preceded information theory, the computer preceded computer science. why? because theorists will spontaneously study “simple” phenomena, and will not be enticed to study a complex one until there [is] a practical importance to it. [11] along the same lines, lecun is reported by the science magazine reporter to have said that “shifting too much effort away from bleeding-edge techniques toward core understanding could slow innovation and discourage ai’s real-world adoption.” he concluded that “it’s not alchemy, it’s engineering;” and added: “engineering is messy.” now, is engineering messy? it seems that it is only in the eyes of the scruffies that it is. we can think of researchers and practitioners who contribute to the field with standards and specification, design theory and so on, and altogether with the tidy application of principles in engineering. in any case, lecun suggested that deep learning is engineering. and yet there has been no question that deep learning is ai. we shall then make a detour to inquire about the nature of ai. 3 is ai a science of intelligence or a branch of engineering? there is something odd going on here—for minsky and simon seem to have thought of ai more as a science of intelligence and less as a field of engineering. minsky had a point about the possibility of intelligence itself and the brain being messy as products of evolution by natural selection. for him, neuroscience—and ai —, as scientific disciplines, would be accordingly untidy as well. now, is this reasonable? let us consider an analogy. just like the brain, the human organism as a whole is a (messy) biological product of evolution. yet that did not stop the discovery of the molecular structure of dna as a unifying principle of all life. so minsky’s point may be seen rather as an a priori assumption. simon also seems to have thought that brain processes were to some extent messy. but he found in heuristics sort of a unifying principle. given the initial success achieved by newell and simon’s heuristics, minsky himself felt compelled to refer to them in his 1959 paper [16] and in his follow-up survey [17]. moreover, simon presented a view of the design of artificial systems as an empirical science, and strove to manage the complexity of large systems [28]. later in 1995 he even wrote a dedicated piece to show that ai is an empirical science [27]. and yet it is unlikely that he would endorse minsky’s view that ai should be messy. for simon, ai is a lawful empirical science just as physics is. he wrote: the natural laws that determine the structure and behavior of an object, natural or artificial, are its internal constraints. an artificial system, like a natural one, produces empirical phenomena that can be studied by the methods of observation and experiment common to all science. [27, p. 99] in fact, simon accepted no significant distinction between natural and artificial objects from the point of view of empirical studies. but lecun’s point looks really different. he seems to be talking about ai as an engineering discipline, say, such as aerospace engineering. and this connects to an older point about ai but also about computer science more generally. in a 1984 talk “the threats to computer science” [5], the physicist and computer programming pioneer edsger dijkstra set out one of his ingenious metaphors: the fathers of the field had been pretty confusing: john von neumann speculated about computers and the human brain in analogies sufficiently wild to be worthy of a medieval thinker and alan m. turing thought about criteria to settle the question of whether machines can think, a question of which we now know that it is about as relevant as the question of whether submarines can swim [5]. ten years later a variant of the same metaphor was caught by noam chomsky and appeared (with no source given) in the may 1994 lectures that further composed his 1995 mind paper [2]. many of these debates “over such alleged questions as whether machines can think,” chomsky referred (p. 9), “trace back to the classic paper by alan turing.” they fail to take note, he objected, that turing himself declared to believe that the question “can machines think?” was “too meaningless to deserve discussion.” chomsky thus concluded: it is not a question of fact, but a matter of decision as to whether to adopt a certain metaphorical usage, as when we say (in english) that airplanes fly but comets do not [...] similarly, submarines set sail but do not swim. there can be no sensible debate about such topics; or about machine intelligence, with the many familiar variants. [2, p. 9] overall, chomsky denied turing’s question to have a seat within the empirical sciences. given minsky’s position as we have quoted above, it is unlikely that he would agree with dijkstra and chomsky. but perhaps lecun and others would do it, and then the discussion must go back to turing’s 1950 vision of ai [31]. in an influential keynote lecture at ijcai 1995 [8], patrick hayes and kenneth ford indirectly associated themselves with chomsky’s variant of dijkstra’s metaphor to argue against “the turing test vision” of ai (p. 974–5). they complained that ai systems back then (e.g., expert systems) were sufficiently successful as task-specific cognitive artifacts and yet were seen as a failure because of “turing’s ghost” (p. 976). they then urged: [i]f we abandon the turing test vision, the goal naturally shifts from making artificial superhumans which can replace us, to making superhumanly intelligent artifacts which we can use to amplify and support our own cognitive abilities, just as people use hydraulic power to amplify their muscular abilities. [8, p. 974] according to the vision laid out by hayes and ford, the primary concern of ai should be the engineering of intelligent systems. now, it is interesting to note that patrick hayes hardly fits in the scruffy portrait. in fact, hayes’ intellectual project in ai had significant overlapping with mccarthy’s and can be best seen as a neat one. this shows that the discussion about the nature of ai as a discipline is present among both scruffies and neats. among neats, however, the science v. engineering question seems much less pronounced. this is perhaps because neats tend to agree on the value of understanding—the logic of intelligent systems must be well-understood. in fact, lack of principles and understanding is the core of most critiques of machine learning today, e.g., rahimi and recht’s at nips 2017 (cf. subsect. 2.6). in 2012, chomsky claimed that ai departed from the tradition of modern science as it gave up the understanding of cognitive phenomena and their rendering in artificial systems. a reporter mentioned ai’s recent shift from the so-called “good old fashioned ai” to the use of “probabilistic and statistical models.” he was trying to get chomsky’s opinion on what could explain that shift and whether or not it was a step in the right direction. chomsky answered: chomsky: [an] approach, which i think is the right approach, is to try to see if you can understand what the fundamental principles are that deal with the core properties, and recognize that in the actual usage, there’s going to be a thousand other variables intervening — kind of like what’s happening outside the window, and you’ll sort of tack those on later on if you want better approximations [...that] is what science has been since galileo, that’s modern science. the approximating unanalyzed data kind is sort of a new approach, not totally, there’s things like it in the past. it’s basically a new approach that has been accelerated by the existence of massive memories, very rapid processing, which enables you to do things like this that you couldn’t have done by hand. but i think, myself, that it is leading subjects like computational cognitive science into a direction of maybe some practical applicability... interviewer: “... in engineering?” chomsky:... but away from understanding. [10]. chomsky’s opinion is worth quoting also because google’s director of research peter norvig bothered to reply it. thus wrote norvig: i agree that engineering success is not the goal or the measure of science. but i observe that science and engineering develop together, and that engineering success shows that something is working right, and so is evidence (but not proof) of a scientifically successful model. science is a combination of gathering facts and making theories; neither can progress on its own. i think chomsky is wrong to push the needle so far towards theory over facts; in the history of science, the laborious accumulation of facts is the dominant mode, not a novelty. the science of understanding language is no different than other sciences in this respect [21]. norvig’s view seems strongly related with lecun’s. we shall now be well positioned to characterize types of past and present attitudes towards ai in view of its future as a science and engineering discipline. 4 types of neat and scruffy’s attitudes in ai considering neat and scruffy approaches to ai and the further heterogeneity that is found within each camp, we shall distinguish three types of attitudes. although there is some overlapping of concerns in between them, the primary commitment in each type is different and looks clear, as we elaborate next. 4.1 scruffy type i: the empirical scientists for the scruffies type i, ai should be above all an empirical science of intelligence and of the mind and/or the brain. in short, their primary goal is: goal: to understand and reproduce the phenomenon of intelligence. this attitude was quite strong among influential figures in the early phase of the discipline. we identify it in minsky and simon but first, very early on in turing himself. after world war ii, turing was recruited by the national physical laboratory (npl) near london to build a digital computer that would match us-based initiatives such as the construction of the eniac. turing’s first technical report proposed electronic calculator in late 1945 is an outcome of his task of specifying a computer architecture. he was engaged in the job of building a computer system (“a machine”). but the historical sources show that, unlike some of his colleagues at the time (e.g., the physicist and computer pioneer douglas hartree [7]), turing’s goal was not to enable scientific computing applications but rather to try to imitate the human brain [6, p. 237]. in fact, when he joined the npl after the northern summer of 1945 he said that he was going to “build a brain” [6, p. 233]. it is possible to identify turing’s shift from neatness to scruffiness—from his seminal on computable numbers paper in 1936 and his 1938 doctoral thesis in mathematical logics to codebreaking and empirical problem solving ever since his wartime service in 1939 on [6, p. 230-1]. this illustrates the ambition of the scruffy type i, who will resort to systems building as a means to empirically study natural intelligence and the brain. the same inclination can be found in minsky and simon’s intellectual projects. minsky opened his 1985 society of mind by positing his goal of explaining “how minds work” straightforwardly and asking “[h]ow can intelligence emerge from non-intelligence?” to answer that, he added, “we’ll show that you can build a mind from many little parts, each mindless by itself.” [18, p. 17]. as known, minsky’s experimental approach to pursue that project was based on computer techniques and systems. this is described, for example, in his well-known 1961 survey of ai techniques [17]. simon’s take is another example of the scruffy type i. together with allen newell, he developed plenty of computer techniques and software systems in order to test his theories of intelligence. the logic theorist as we have seen was the first of the line, and was followed by the general problem solver and others. to mention an active ai researcher that we identify with this type, rodney brooks has recently called upon ai to get back at taking the human brain as reference [1]. in short, among scruffies, the empirical scientist seems to aim at discovering whatever cognitive architecture and kludges that can be shown to emulate human intelligence. emphasis is given on the best knowledge then available about the human brain as a product of evolution by natural selection over billions of years. but in fact, as we have seen (sect. 3), this is not the only kind of attitude among scruffies. we shall discuss a second position next. 4.2 scruffy type ii: the system builders for this other class of scruffies, ai should not be primarily concerned with the deeper mysteries that surround human intelligence. the system builders seem to aim at inventing whatever techniques to achieve or surpass human-level intelligence. if some contribution to the understanding of human intelligence is made in the process, the better. emphasis is given on leveraging the best computer resources then available. in short, we can write their goal as: goal: to build intelligent systems that are as autonomous as possible. we identify this view in lecun and norvig as quoted above. both posed their commentaries while being in charge of two of the biggest companies that invest in ai today. their wish to reply to criticisms can also be understood as a defense of the research strategy and methodology underlying the ai systems that have been deployed in society by the companies they represent. another ai researcher that seems to express this view is richard sutton, distinguished research scientist of the company deepmind. in a 2019 web commentary [30], he wrote that “[t]he biggest lesson that can be read from 70 years of ai research is that general methods that leverage computation are ultimately the most effective.” sutton called this “the bitter lesson.” he was implicitly referring to the recent success of deep learning and suggested that “[t]he ultimate reason for this is moore’s law.” he completed: “researchers seek to leverage their human knowledge of the domain, but the only thing that matters in the long run is the leveraging of computation.” now, there is no doubt that the human brain has huge computation power. yet sutton’s line of thought seems to depart from a concern with discovering the true nature of human intelligence and with imitating it closely. it characterizes most clearly the goal of resorting to whatever techniques are available in order to achieve performance that is compatible with the output of human intelligence. 4.3 neats: the computer epistemologists we have examined the varieties of discourse among neats and found no significant differences. this class may even include logicians and analytic philosophers, as it closely resembles the field of epistemology (or the theory of knowledge) in modern philosophy. but it also includes builders of knowledge bases (kb’s), as long as there seems to be a concern with the soundness of both the knowledge that gets into the kb and the knowledge that is derived out of it. the computer epistemologists seem to aim at understanding and developing knowledge management and inference techniques to imitate the rational aspects of human-level intelligence. emphasis is given on the soundness and effectiveness of reasoning. in short, we can write their goal as: goal: to understand and reproduce rational aspects of intelligence. as mentioned, mccarthy advocated the need to address commonsense knowledge representation and management in view of human-level intelligence early on in the history of ai. he was perhaps the main champion of the neats in ai research. but there is a lot more diversity in this class. the related tradition of expert systems is also a notable example of the neats at work. douglas lenat and ed feigenbaum thus tried to summarize their view: we articulate the three major findings and hypotheses of ai to date: (1) the knowledge principle: if a program is to perform a complex task well, it must know a great deal about the world in which it operates. in the absence of knowledge, all you have left is search and reasoning, and that isn’t enough. (2) the breadth hypothesis: to behave intelligently in unexpected situations, an agent must be capable of falling back on increasingly general knowledge and analogizing to specific but superficially far-flung knowledge. (this is an extension of the preceding principle.) (3) ai as empirical inquiry: premature mathematization, or focusing on toy problems, washes out details from reality that later turn out to be significant. thus, we must test our ideas experimentally, falsifiably, on large problems. [12, p. 185, no emphasis added] considering all three “findings and hypotheses,” one may note lenat and feigenbaum’s concern with justified reasoning through sound knowledge in real-world scenarios. also, they address the caricature of the neat profile which is often criticized for tackling toy problems. lenat’s iniative to develop the large-scale commonsense kb he called cyc was based on an expressive knowledge representation language “involving first-order predicate calculus plus zf set theory, meta-level assertions, contexts, and modal operators” [13, p. 38]. it did not pay much attention to uncertainty management. but several other approaches to knowledge representation and reasoning have been developed by the neats based on, say, multi-valued logics and probability theory. we identify as neats thinkers such as isaac levi and henry kyburg, who developed theories of uncertain and approximate reasoning; but also thinkers such as david lewis who studied causaility and counterfactuals. other neats are ronald fagin and joseph halpern, who theorized on reasoning about beliefs; but also judea pearl and adnan darwiche who contributed to reasoning through graphical models and probability distributions. darwiche posited that the results achieved by “function-based” approaches to ai such as deep learning are closer to animal-like abilities” than to “human-level intelligence.” he pointed out that the latter requires a “model-based approach” [4]. for pearl, true ai can only come when a machine is able to test cause-and-effect statements, which would allow it to explain events [22]. 5 implications for the future of ai related to the goals of scruffy types i and ii and the neats are implications for the future of ai research in society. the discussion between researchers from different categories is often heated as if there was no room for the fellow’s approach in ai. in dialogue with the scruffy type ii, e.g., ali rahimi said at the 2017 of nips: we are building systems that govern healthcare and mediate our civic dialogue. we would influence elections. i would like to live in a society whose systems are built on top of verifiable, rigorous, thorough knowledge, and not on alchemy [23]. but also, in dialogue with the neats simon had written in 1995: artificial objects, including computer programs, are what they are because they were designed to be that way. this fact has led some to claim that there can be no science of artificial objects, but only an engineering technology. those who hold the most extreme form of this view look to the discovery and proof of mathematical theorems about intelligent systems as the only genuine route to a science of ai, and denigrate the role of system building and experiment as “only engineering.” [27, p. 98-9] now, in acknowledgement of the values underlying each of the three attitudes let us pack their core values and implications for the future of ai in society. scruffy type i. the empirical scientists hold the promise to contribute to an in-depth understanding of the phenomenon of intelligence, perhaps in cross-fertilization with neuroscience, cognitive psychology and the behavioral sciences, psychiatry and human development. this attitude towards ai research was present early on from the beginning with turing, minsky and simon. there is hardly any ethical concern to be brought about in connection with it, and this is particularly true if we consider its software-based (abstract and non-invasive) methodology.footnote 1 as intelligence is distinctive of the human, this approach to ai can in principle deliver an improved understanding of our own nature. scruffy type ii. the system builders are primarily committed to deliver ai systems that can learn for themselves from experience and change their environment, be it physical or virtual. this form of ai needs essentially two skills: perception or the ability to recognize things, and control or the ability to do things in its environment. so the recent success in image, speech and natural language processing, that is, in the semantic interpretation of opaque data, is for sure a step forward towards it. this attitude towards ai research is more recent and arguably flourished with the emergence of large ai projects in the big tech companies. it holds promise to deliver value by pushing the amount of automation in industry to the next level. it is key for applications such as tumor detection, face and speech recognition, language translation and self-driving cars. it can also replace workers in dangerous jobs. however, hand in hand with this great value there is increasing social concern that it shall lead to significant job losses, privacy risks and concentration of power. neats. the computer epistemologists are primarily concerned with the study of reasoning and the delivery of ai systems that can truly interact with us humans in our own language. this form of ai needs essentially one skill: knowing or the ability to understand and form new ideas and judgements, perhaps even structured theories, which must be communicated and explained in a conversation. this approach to ai is key for achieving non-shallow chatbots and personal assistants, question answering and domain-specific reasoning and decision making in, say, law, science and engineering, healthcare and so on. it is also important, of course, in teaching ai. it has cross-fertilization with analytic philosophy and logics. we hope that this description of types and their implications for the future of ai can be helpful for the ai community in its reflection about methodologies. 6 conclusion in the history of ai as a discipline, researchers tended to adopt either a neat or a scruffy (of whatever type) approach and seldom both. it is important to recognize that the forms of ai derived from each of them are in fact essentially different and hard to integrate. in spite of that, recently an approach to ai called hybrid or neurosymbolic ai has been proposed. this purportedly hybrid approach to ai needs to be tested by time and deserves a dedicated study. in this paper we have striven to describe the state of affairs within ai research. hopefully this descriptive effort can be of value for the ai community to make a step forward in its own reflection towards the future. notes 1.and yet, as mentioned, these types of attitude towards ai do have some overlapping. it is worth noting that simon’s rand-corporation collaboration with darpa during the cold war has something of the scruffy type ii as well (e.g., cf. [26]). references brooks, r.: is the brain a good model for machine intelligence? nature 482, 462–3 (2012). https://doi.org/10.1038/482462a article  google scholar  chomsky, n.: language and nature. mind 104(413), 1–61 (1995). https://doi.org/10.1093/mind/104.413.1 article  google scholar  cozman, f.: no canal da inteligência artificial: nova temporada dos desgrenhados e empertigados. estudos avançados 35(101), 7–20 (2021). https://doi.org/10.1590/s0103-4014.2021.35101.002 article  google scholar  darwiche, a.: human-level intelligence or animal-like abilities? commun. acm 61(10), 56–67 (2018) article  google scholar  dijkstra, e.: the threats to computing science. in: talk delivered at the acm 1984 south central regional conference, november 16–18, austin, texas (november 1984). http://www.cs.utexas.edu/users/ewd/transcriptions/ewd08xx/ewd898.html. accessed 10 jun 2021 gonçalves, b.: machines will think: structure and interpretation of alan turing’s imitation game. ph.d. thesis, faculty of philosophy, languages and human sciences, university of são paulo, são paulo (march 2021). http://dx.doi.org/10.11606/t.8.2021.tde-10062021-173217 hartree, d.: calculating instruments and machines. university of illinois press, champaign (1949) google scholar  hayes, p., ford, k.: turing test considered harmful. in: proceedings of the 14th international joint conference on artificial intelligence (ijcai 1995), pp. 972–7 (1995) google scholar  hudson, m.: ia researchers allege that machine learning is alchemy. science (3 may 2018). http://dx.doi.org/10.1126/science.aau0577 katz, y.: noam chomsky on where artificial intelligence went wrong: an extended conversation with the legendary linguist. the atlantic (1 nov 2012) (2012). http://www.theatlantic.com/technology/archive/2012/11/noam-chomsky-on-where-artificial-intelligence-went-wrong/261637/ lecun, y.: my take on ali rahimi’s "test of time" award talk at nips. https://www.facebook.com/yann.lecun/posts/10154938130592143. accessed 3 june 2021 lenat, d., feigenbaum, e.: on the thresholds of knowledge. artif. intell. 47(1–3), 185–250 (1991). https://doi.org/10.1016/0004-3702(91)90055-o article  mathscinet  google scholar  lenat, d.: cyc: a large-scale investment in knowledge infrastructure. commun. acm 38, 33–8 (1995) article  google scholar  mccarthy, j.: programs with common sense. in: proceedings of the teddington conference on the mechanization of thought processes, her majesty’s stationery office, london (december 1958). http://www-formal.stanford.edu/jmc/mcc59.pdf. accessed 3 june 2021 mccorduck, p.: machines who think: a personal inquiry into the history and prospects of artificial intelligence. a. k. peters, second edn. crc press, boca raton (2004 [1979]) google scholar  minsky, m.: some methods of heuristic programming and artificial intelligence. in: blake, d.v., uttley, a.m. (eds.) proceedings of the symposium on mechanisation of thought processes, vol. 2, h. m. stationery office, london (1959) google scholar  minsky, m.: steps toward artificial intelligence. proc. ire 49, 8–30 (1961). https://doi.org/10.1109/jrproc.1961.287775 article  mathscinet  google scholar  minsky, m.: the society of mind. simon & schuster, new york (1985) google scholar  minsky, m.: smart machines. in: brockman, j. (ed.) the third culture: beyond the scientific revolution, chap. 8. simon & schuster, new york (1995) google scholar  nilsson, n.: the quest for artificial intelligence. cambridge university press, cambridge (2009) google scholar  norvig, p.: on chomsky and the two cultures of statistical learning (2012). http://norvig.com/chomsky.html. accessed 3 june 2021 pearl, j.: the book of why. basic books, new york (2019) google scholar  rahimi, a., recht, b.: nips "test-of-time award" keynote address (2017). http://www.youtube.com/watch?v=qi1yry33tqe. accessed 3 june 2021 russell, s., norvig, p.: artificial intelligence: a modern approach. 1st edn. prentice hall, hoboken (1995), isbn 0-13-103805-2 google scholar  russell, s., norvig, p.: artificial intelligence: a modern approach. pearson series in artificial intelligence, pearson, 4th edn. (2020). isbn 9781292401133 google scholar  sent, e.m.: herbert a. simon as a cyborg scientist. perspect. sci. 8(4), 380–406 (2000). https://doi.org/10.1162/106361400753373759 simon, h.: artificial intelligence: an empirical science. artif. intell. 77(1), 95–127 (1995). https://doi.org/10.1016/0004-3702(95)00039-h article  google scholar  simon, h.: the sciences of the artificial. 3rd edn. mit press, cambridge (1996 [1969]) google scholar  simon, h., newell, a.: human problem solving: the state of the theory in 1970. am. psychol. 26(2), 141–59 (1971). https://doi.org/10.1037/h0030806 article  google scholar  sutton, r.: the bitter lesson. http://incompleteideas.net/incideas/bitterlesson.html. accessed 15 june 2021 turing, a.m.: computing machinery and intelligence. mind lix (236), 433–60 (1950). https://doi.org/10.1093/mind/lix.236.433 download references author information authors and affiliations escola politécnica, universidade de são paulo, são paulo, brazil bernardo gonçalves & fabio gagliardi cozman authors bernardo gonçalvesview author publications search author on:pubmed google scholar fabio gagliardi cozmanview author publications search author on:pubmed google scholar corresponding author correspondence to bernardo gonçalves . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper gonçalves, b., cozman, f.g. (2021). the future of ai: neat or scruffy?. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_13 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords neat vs. scruffy history of ai future of ai scientific method styles of scientific reasoning publish with us policies and ethics profiles bernardo gonçalves view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an instance level analysis of classification difficulty for unlabeled data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an instance level analysis of classification difficulty for unlabeled data conference paper first online: 30 january 2025 pp 141–155 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) an instance level analysis of classification difficulty for unlabeled data download book pdf download book epub patricia s. m. ueda  orcid: orcid.org/0009-0003-2200-59029, adriano rivolli  orcid: orcid.org/0000-0001-6445-300710 & ana carolina lorena  orcid: orcid.org/0000-0002-6140-571x9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 497 accesses abstract instance hardness measures allow us to assess and understand why some observations from a dataset are difficult to classify. with this information, one may curate and cleanse the training dataset for improved data quality. however, these measures require data to be labeled. this limits their usage in the deployment stage when data is unlabeled. this paper investigates whether it is possible to identify observations that will be hard to classify despite their label. for such, two approaches are tested. the first adapts known instance hardness measures to the unlabeled scenario. the second learns regression meta-models to estimate the instance hardness of new data observations. in experiments, both approaches were better at identifying instances lying in borderline regions of the dataset, which pose a greater difficulty when the label is unknown. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others measuring difficulty of learning using ensemble methods chapter © 2022 measuring instance hardness using data complexity measures chapter © 2020 active learning using difficult instances chapter © 2022 1 introduction the machine learning (ml) literature extensively provides algorithmic developments focused on model hyperparameter tuning and related model-centric tasks. more recently, the community of data-centric artificial intelligence (ai) is lighting the focus on the effort to understand more the data and its quality improvement than on developing more complex ml models [13]. paving the way for such a data-centric approach is a more fine-grained analysis of the data and classification performance. herewith, aggregated measures applied for classification problems, such as accuracy, precision, or similar metrics, restrict the understanding of the particularities in the data the algorithms are modeling. those aggregated metrics do not provide information about misclassification at the level of an instance or why they are misclassified. however, a more reliable usage of ml algorithms must reveal for which particular instances a model struggles to classify correctly and why. one way to achieve such understanding is leveraging knowledge from correlating data characteristics extracted by a set of meta-features [12] to the predictive performance of multiple algorithms, in a meta-learning (mtl) approach [2]. one particular set of meta-features is the set of data complexity measures, previously proposed by ho and basu [5] to explore the overall complexity of solving the classification problem given the dataset available for learning, providing a global perspective of the difficulty of the problem [1]. since these measures can fail to provide information at the instance-level [8], the instance hardness measures (ihms) were introduced by smith et al. [14] to characterize the difficulty level of each individual instance of a dataset, giving information on which particular instances are misclassified and why. these developments attend to a trending interest in responsible ai that has emerged in recent years, making researchers focus on the reliability and trustfulness of the predictions obtained by ml models. nonetheless, the current ihms need the instance label to be computed, which restricts their use for analyzing and curating ml training datasets. in the use of ml in production, where the class of an instance is unknown, adaptations are needed. this paper proposes alternative instance hardness measures when the instances do not have a label. the idea is to leverage the knowledge of the hardness of the training dataset, which is labeled, to assess the hardness level of new unlabeled instances. this knowledge can support reject-option strategies in the future so the ml model might opt for abstaining from some predictions that will be uncertain [4]. firstly, a set of ihms is adapted to disregard the labels of the new instances in their computation. another strategy tested was generating regression meta-models to estimate the ihms for new unlabeled observations in a meta-learning approach at the instance level. both approaches are compared experimentally using one synthetic dataset and four datasets of the health domain, known for presenting hard instances. instances with characteristics making them lie in overlapping or borderline regions of the classes are highlighted as hard to classify by both approaches. the adapted measures show an increased correlation to the original values of the instance hardness measures and prove to be an adequate alternative to estimate instance hardness in the deployment stage, driving the solutions to a more refined level and contributing toward a more trustful use of ml models. the paper is organized as follows: sect. 2 details the hardness measures to apply to unlabeled data and how they were modified from the original measures. section 3 presents the materials and methods used in experiments, whose results are presented in sect. 4. finally, sect. 5 presents the conclusions of this work. 2 instance hardness measures the concept of instance hardness was introduced in the seminal work of smith et al. [14] as an alternative for a fine-grained analysis of classification difficulty. they define an instance as hard to classify if it gets consistently misclassified by a set of classifiers of different biases. they also define a set of measures to explain possible reasons why an instance is difficult to classify, which are regarded as instance-level meta-features in the literature [7]. the base ihms adopted in this work are presented next, along with their adaptations, which are indicated by the “\({\text {adj}}\)” (adjusted) extension. in their definition, let \(\mathcal d\) be a training dataset with n pairs of labeled instances \((\textbf{x}_i,y_i)\), where each \(\textbf{x}_i \in \mathcal x\) is described by m input features and \(y_i \in \mathcal y\) is the class of the instance in the dataset. the number of classes is denoted as c. and let \(\textbf{x}\) be a new instance for which the label is unknown. to illustrate the concepts, consider the dataset in fig. 1 containing two classes, red and blue. two instances are highlighted: \(\textbf{x}_1\) and \(\textbf{x}_2\). the instance \(\textbf{x}_1\) is in a borderline area of the classes and might be difficult to classify despite its class. the instance \(\textbf{x}_2\) is more aligned to the blue class. if the label registered for it in the dataset is blue, it will be easily classified. otherwise, it will have a hardness level higher than \(\textbf{x}_1\). standard ihms need to know these labels, so that both \(\textbf{x}_1\) and \(\textbf{x}_2\) are contained in the labeled dataset \(\mathcal d\). this work introduces adaptations to estimate the hardness level of an instance in the absence of its label, meaning \(\textbf{x}_1\) and \(\textbf{x}_2\) are not in the labeled dataset \(\mathcal d\) used to estimate the hardness levels. please note there are differences between the two estimations. based on the characteristics of \(\textbf{x}_2\), it will probably be easily classified as blue. in contrast, \(\textbf{x}_1\) will probably be considered hard to classify in both scenarios. fig. 1. example of dataset with highlighted instances: \(\textbf{x}_1\) is in a borderline region and can be difficult to classify despite its class; \(\textbf{x}_2\) might be easy or hard to classify depending on its registered label. (color figure online) full size image 2.1 neighborhood-based ihm the hardness level of the instance can be obtained considering its neighbourhood in the dataset. in the original ihms, instances surrounded by elements sharing the same label as themselves can be considered easier to classify. for new data without labels, our approach seeks the neighbourhood of the instance in the labeled dataset \(\mathcal d\) and assigns a higher hardness level when there is a mix of different classes in this region. k-disagreeing neighbors kdn: the original kdn measure computes the percentage of the k nearest neighbors of \(\textbf{x}_i\) in the dataset \(\mathcal d\) that have a different label than the refereed instance: $$\begin{aligned} \text {kdn}(\textbf{x}_i,y_i) = \frac{\sharp \{\textbf{x}_j | \textbf{x}_j \in \text {knn}(\textbf{x}_i) \wedge y_j \ne y_i\}}{k}, \end{aligned}$$ (1) where \(\text {knn}(\textbf{x}_i)\) represents the set of k-nearest neighbors of the instance \(\textbf{x}_i\) in the dataset \(\mathcal d\). an instance will be considered harder to classify when the value of kdn\((\textbf{x}_i,y_i)\) is higher. values close to 1 represent an instance surrounded by examples from a different class of itself. this would be \(\textbf{x}_2\)’s case in fig. 1 when labeled red in \(\mathcal d\). intermediate values of kdn\((\textbf{x}_i,y_i)\) are found for borderline instances. easier instances are those surrounded by elements sharing their class label, which would correspond to \(\textbf{x}_2\) when it has a blue label. in the absence of an instance’s label, an alternative way to measure the mixture of classes in its neighbourhood is to compute an entropy measure. specifically, the entropy is computed based on the proportion of the classes found in the instance’s neighbourhood. higher entropy values represent the new instance is in regions from \(\mathcal d\) near elements from different classes. this corresponds to the \(\textbf{x}_1\) case in fig. 1. in contrast, \(\textbf{x}_2\) will be regarded as easy to predict, as it is surrounded by elements of the blue class. $$\begin{aligned} \text {kdn}_{\text {adj}}(\textbf{x}) = -\sum _{i=1}^{c} p(y_j = c_i) \log p(y_j = c_i), \text {for} \textbf{x}_j \in \text {knn}(\textbf{x}), \end{aligned}$$ (2) where \(p(y_j = c_i)\) are the proportions of the classes of the k-nearest neighbours of \(\textbf{x}\) in the dataset \(\mathcal d\). ratio of the intra-class and extra-class distances. n2\(_{\text {ihm}}\): the original measure takes the complement of the ratio of the distance of \(\textbf{x}_i\) to the nearest example from its class in \(\mathcal d\) to the distance it has to the nearest instance from a different class (nearest enemy) in \(\mathcal d\) with a normalization as presented next: $$\begin{aligned} \text {n2}_{\text {ihm}}(\textbf{x}_i,y_i) = 1 \frac{1}{\textrm{intrainter}(\textbf{x}_i)+1}, \end{aligned}$$ (3) where: $$\begin{aligned} \textrm{intrainter}(\textbf{x}_i,y_i) = \frac{d(\textbf{x}_i,\text {nn}(\textbf{x}_i\in y_i) )}{d(\textbf{x}_i,\text {ne}(\textbf{x}_i))}, \end{aligned}$$ (4) where d is a distance function, nn\((\textbf{x}_i\in y_i)\) represents the nearest neighbor of \(\textbf{x}_i\) from its class and ne\((\textbf{x}_i)\) is the nearest enemy of \(\textbf{x}_i\) (ne\((\textbf{x}_i) = \text {nn}(\textbf{x}_i \in y_j \ne y_i\))). in this formulation, when an instance is closer to an example from another class than another from its own class, the n2\(_{\text {ihm}}\) values will be larger, indicating that this instance is harder to classify. this would correspond to the case where \(\textbf{x}_2\) in fig. 1 has the red label. the alternative measure for unlabeled instances can be obtained by taking the ratio of the minimum distance from \(\textbf{x}\) and the closest element in \(\mathcal d\), denoted as \(\textbf{x}_j\) in eq. 5, to the distance from \(\textbf{x}\) and the closest element from another class in \(\mathcal d\), that is, a class different from that of \(\textbf{x}_j\). this ratio will assume value close to 1 when the instance is almost equally distant from different classes. this will happen more probably for borderline instances, such as \(\textbf{x}_1\) in fig. 1. $$\begin{aligned} \text {n2}_{\text {adj}}(\textbf{x}) = \frac{\min (d(\textbf{x},\textbf{x}_j))}{\min (d(\textbf{x},\textbf{x}_k)|y_k \ne y_j)} \end{aligned}$$ (5) 2.2 class likelihood ihm this type of measure captures if the instance is well situated in its class, considering the general patterns of this class. the likelihood can be estimated for that, considering the input features are independent for simplifying the computations. class likelihood difference] cld: the original measure takes the complement of the difference between the likelihood that \(\textbf{x}_i\) belongs to its class \(y_i\) and the maximum likelihood it has to any other class. this complement is taken to standardize the interpretation of the direction of hardness since the confidence of an instance belongs to its class is larger than that of any other class [9]: $$\begin{aligned} \text {cld}(\textbf{x}_i,y_i) = \frac{1 -\left( p(\textbf{x}_i|y_i)p(y_i) {\max }_{y_j \ne y_i}[p(\textbf{x}_i |y_j)p(y_j)]\right) }{2}, \end{aligned}$$ (6) where \(p(y_i)\) is the prior of class \(y_i\), set as \(\frac{1}{c}\) for all data instances. \(p(\textbf{x}_i|y_i)\) represents the likelihood \(\textbf{x}_i\) belongs to class \(y_i\) and it can be estimated considering the input features independent of each other, as in naïve bayes classification. for example, if \(\textbf{x}_2\) in fig. 1 is labeled as blue in \(\mathcal d\), it will be easy according to this measure, as its likelihood to the blue class will be higher than to the red class. when the class of an instance cannot be defined in advance, the hardness measure can be estimated by the difference between the two higher likelihoods of all possible classes in the dataset. like in the original measure, the complement of the difference is taken to keep the interpretation that higher values are found for instances harder to classify. the values of this measure will tend to be higher for borderline instances since their likelihood of being in different classes will be similar. $$\begin{aligned} \text {cld}_{\text {adj}}(\textbf{x}) = \frac{1 -\left( {\max }_{y_i}[p(\textbf{x}|y_i)p(y_i)] {\max }_{y_j \ne y_i}[p(\textbf{x}|y_j)p(y_j)]\right) }{2}. \end{aligned}$$ (7) 2.3 tree-based ihm decision trees (dts) can be used to estimate the hardness level of an instance based on the number of splits necessary to classify it. if many splits are required, the instance’s classification will be harder. the dt is built based on the labeled dataset \(\mathcal d\). unlabeled instances are input to the built dt, and the measure can be computed based on where it is classified. disjunct class percentage dcp: from a pruned decision tree (dt) using \(\mathcal d\), the leaf node where the instance is classified is considered the disjunct of \(\textbf{x}_i\). the complement of the percentage of instances in this disjunct that shares the same label as \(\textbf{x}_i\) gives the original dcp measure: $$\begin{aligned} \text {dcp}(\textbf{x}_i,y_i) = 1\frac{\sharp \{\textbf{x}_j | \textbf{x}_j \in \text {disjunct}(\textbf{x}_i) \wedge y_j = y_i\}}{\sharp \{\textbf{x}_j|\textbf{x}_j \in \text {disjunct}(\textbf{x}_i)\}}, \end{aligned}$$ (8) where disjunct\((\textbf{x}_i)\) represents the instances contained in the disjunct (leaf node) where \(\textbf{x}_i\) is placed. for easy instances, according to this measure, larger percentages of examples sharing the same label as the instance will be found in their disjunct. for example, if \(\textbf{x}_2\) in fig. 1 has the red label in \(\mathcal d\), it will probably be placed in a leaf node containing many elements of the blue class, making it harder to classify according to the interpretation of this measure. in scenarios where the instance’s class is unknown, we take the entropy of the disjunct where the instance is placed as a hardness measure, similarly to what has been done for kdn. $$\begin{aligned} \text {dcp}_{\text {adj}}(\textbf{x}) = \sum _{i=1}^{c} p(y_j = c_i) \log p(y_j = c_i), \text {for} \textbf{x}_j \in \text {disjunct}(\textbf{x}), \end{aligned}$$ (9) where the proportions of the classes are taken based on the disjunct where \(\textbf{x}\) is placed in the dt built using the dataset \(\mathcal {d}\). tree depth td: the original measure gives the depth of the leaf node that classifies \(\textbf{x}_i\) in a dt built using all labeled dataset \(\mathcal d\), normalized by the maximum depth of the tree: $$\begin{aligned} \text {td}(\textbf{x}_i,y_i) = \frac{\text {depth}_{\text {dt}}(\textbf{x}_i)}{\max (\text {depth}_{\text {dt}}(\textbf{x}_j \in \mathcal d))}, \end{aligned}$$ (10) where \(\text {depth}_{\text {dt}}(\textbf{x}_i)\) gives the depth where the instance \(\textbf{x}_i\) is placed in the dt. instances harder to classify tend to be placed at deeper levels of the tree, making td higher. there are two versions of this measure. one derives from a pruned tree (\(\text {td}_{\text {p}}\)) and the other from an unpruned tree (\(\text {td}_{\text {u}}\)). for unlabeled instances, the procedure for hardness estimation is the same as in dcp, where the dt is built from the labeled set \(\mathcal d\), and next, the unlabeled instance is submitted to the built dt. the depth of the leaf node where this instance is classified by the dt is taken and used in the equation: $$\begin{aligned} \text {td}_{\text {adj}}(\textbf{x}) = \frac{\text {depth}_{\text {dt}}(\textbf{x})}{\max (\text {depth}_{\text {dt}}(\textbf{x}_j \in \mathcal d))}, \end{aligned}$$ (11) 2.4 using meta-models to estimate ihm meta-learning is a traditional ml task that uses data related to ml itself [2]. here, mtl is designed to predict ihm values without considering their labels. this is done using the original input features from the dataset \(\mathcal d\) to learn the expected ihm values in a regression task. therefore, in this approach regression meta-models are induced to estimate the ihm values of new instances. their training datasets comprise the original input features of \(\mathcal d\) and a label corresponding to an ihm estimated from \(\mathcal d\) in its original formulation. there is one regression model per ihm. the estimation of the ihm values for unlabeled data with this meta-learning approach is compared to the usage of the adjusted ihm values. 3 materials and methods in this section, we describe the materials and methods used in experiments performed to analyze the behaviour of ihm for unlabeled data in classification problems. 3.1 datasets five datasets are employed in the experiments. the first dataset was created synthetically, containing three classes with some overlap. the other four datasets are from the health domain, for which some instances are hard to classify due to the overlap of attribute values for different classes or inconsistencies. two of them are from the uci public repository [6] and have been employed in previous related work [8, 11]. the last two are related to severe covid-19 cases in two large hospitals from the são paulo metropolitan area [15]. the main characteristics of the five datasets are presented in table 1, including the number of instances, classes and input features. table 1. summary of the datasets used in the study.full size table the dataset blobs was generated synthetically using the make_blobs package from the scikit-learn library [10], which can generate isotropic gaussian blobs in space. the standard deviation between the centers of the classes was set as 2 to create some overlap between the input features and regions where the difficulty in classifying the instances is harder than others. figure 4 presents this dataset, where it is possible to notice some overlap in the borderline regions of the classes. fig. 2. illustration of the blobs dataset. full size image the diabetes dataset is related to the incidence of diabetes in female patients of pima indian heritage who are at least 21 years old. the objective is to identify the presence of diabetes. the predictive variables record blood indices and patient characteristics, such as number of pregnancies and age [6]. the heart dataset registers heart disease in patients and has features collected during the exercise test, others reflecting blood indices and personal characteristics of the patients, such as age and gender [6]. the last two datasets, named hospital, were extracted from the raw public database provided by fapesp covid data sharing initiative [3]. the binary response categorized patients as severe when hospital stay was greater than or equal to 14 days or patients who progressed to death. the features collected in those datasets were related to blood indices, age and gender [15]. 3.2 methodology the adjusted ihm measures proposed in this paper were applied to the datasets, considering each instance unlabeled once at a time, and the remaining instances labeled, resembling a leave-one-out (loo) cross-validation scheme. the same procedure is used to generate the meta-models to predict the ihm values, where one instance is left out as unlabeled at a time. the ihm of the other instances is calculated using their original formulations. next, a meta-dataset is built, mapping the original features of the instances to the computed ihm values. regression meta-models are induced to learn this relationship and predict the expected ihm value of the left-out instance. one meta-model is induced per ihm measure considered. we used the random forest regressor (rf) available in the scikit-learn library [10] with default hyperparameters’ values to generate these meta-models. we also computed the original ihms for the entire datasets, which regard the labels of all instances. next, we compare the association of the ihm values of the original measures to those of the estimated measures, where the estimation is taken by the adjusted measures or the induced meta-models. spearman’s correlation provides a non-parametric estimation of the association (monotonic relationship) of the modified measures with the original measure. this correlation captures if the direction of the adjusted/estimated ihm is the same as the value obtained from the original ihm. higher values of the spearman’s correlation indicate more association between the estimated and original ihm. we expect medium to high correlations, although there can be deviations of values, since they do not strive to deliver identical ihm values. indeed, instances with noisy labels in the training datasets have characteristics that make them aligned to another class and are expected to show a lower correlation to the original ihm values. but for most cases, we expect the hardness directions to be maintained. all codes and analyses are implemented in python. the original ihms are computed using the pyhard package [7, 9]. codes of the adjusted measures are in a public repository https://anonymous.4open.science/r/adj-ihm-bf75. the k value in kdn was set as 10, default value in the pyhard package. 4 results the results of the experiments performed are presented and discussed next. 4.1 meta-models first, we present the performance of the meta-models in the regression task. table 2 presents the mean squared error (mse) obtained in predicting the ihms using the regression meta-models. lower values are indicative of better performances in predicting the original ihm values. table 2. mse obtained for the rf algorithm concerning predicting the ihms to different datasets.full size table for some measures, the mses are lower, demonstrating a better approximation of the original ihm values. this happens mostly for tree-depth measures. for others, the approximations are not as good (e.g. for kdn and dcp). one possible explanation is that the tree depth measures do not depend as much on the labels of the instances as the others. the only difference between the original tree depth measures and their estimated counterparts is excluding one instance from the decision tree induction, which affects less the results. for other measures, if an instance is incorrectly labeled, the original measures will point them as very hard to classify. but this instance might be easily classified into another class, making it easy without the label information. 4.2 correlation analysis table 3 shows spearman’s correlation coefficient between the original ihms and the measures obtained using the meta-learning approach. values higher than 0.5 are highlighted in bold. the values of the estimated tree depth measures are the highest, especially for the pruned version of the measure (td\(_\text {u}\)). this happens because, in the pruned version of the tree, noisy and outlier instances tend to be placed in nodes which have undergone pruning. therefore, the label of the particular instance seems to matter less in the original ihm formulation. in contrast, the formulation of the original cld, dcp and kdn measures is highly influenced by the label of each instance where they are measured. this decreases the correlations, especially in datasets with many instances with feature values akin to a class, despite being originally labeled into another class in the dataset. this is the case for hospital 1 and 2 datasets, where situations such as instances wrongly labeled or with overlapping feature values are more common. table 3. spearman coefficient obtained for the rf algorithm concerning predicting the ihms to different datasets.full size table table 4. spearman coefficient obtained for adjusted measures compared to the original ihms in different datasets.full size table table 4 presents the same results for the adjusted ihms: their spearman correlation to the original ihms. as in table 3, values higher than 0.5 are boldfaced. more boldfaced correlations are observed here. similar observations concerning the higher correlation values for tree depth-based measures are observed in table 4 too. the correlations observed for the adjusted measures are generally higher than those observed for the measures estimated by the meta-regressors. to make the differences clearer, fig. 3 plots the spearman’s correlations for the adjusted ihm and the meta-models compared to the original values. blue bars represent the correlation of the adjusted measures, while orange bars denote the meta-learning approach. only for the blobs dataset and for the dcp-diabetes combination were correlations of the meta-models higher than those of the adjusted ihms. the blobs dataset has difficult instances concentrated on the border of the classes, while the other datasets may pose other sources of difficulties which are not captured when the labels are absent, such as label noise. fig. 3. spearman’s correlation applied to the adjusted ihm vs. the original ihm (blue bars) and the predicted vs. expected values from mtl (orange bars). (color figure online) full size image figure 4 shows the instances in the blobs dataset colored by the hardness of the original ihm (in the left) followed by the adjusted ihm (in the center) and the meta-learning approach (in the right). this can be done for this dataset, as it is bi-dimensional. the harder the instance is to classify, the more intense it is colored in red. in contrast, instances that are easier to classify are filled with darker blue. the central areas of the plots contain the overlapping region between the three classes (see fig. 2) and, therefore, are harder to classify. the first row corresponds to the kdn measure, while the second is the td\(_\text {u}\) measure. for kdn, it is clear that the hardest instances are those in the border of the classes. for td\(_\text {u}\) the pattern observed in the three approaches shows that the harness level is related to partition derived from the decision tree classification. all measures show similar behaviors. however, for the adjusted kdn measure, more central instances have higher ihm values compared to the other measures. it is important to note that since the adjusted measures can vary on a different scale from the original ihm, the results presented in the plots were normalized between 0 and 1 to allow a direct comparison. fig. 4. visualization of the measures kdn (top) and td\(_\text {u} \) (bottom): original ihm (left), adjusted ihm (middle) and meta-learning approach (right) for the blobs dataset. (color figure online) full size image 4.3 discussion considering the difference in the nature of the datasets, where the blobs were artificially designed with three classes and the other are real-world health data, the spearman’s correlation in fig. 3 shows that the mtl achieves more convergent result than the adjusted ihm in the blobs dataset for great part of the measures. conversely, for real datasets, the adjusted ihm is more associated with the original measure for almost all measures. the tree-depth measures had the highest correlations to the original measures for the adjusted ihm and the mtl approach. this is mostly related to the fact that the original tree-depth measures do not depend so directly on the label of the instances. the other measures all regard whether the labels of some vicinity are in accordance with the registered label of the instance. this makes them deviate more for instances that are mislabeled, for instance. this can be observed in fig. 5, where the original and estimated ihms kdn (in the top) and td\(_\text {u}\) (in the bottom) are contraposed for all instances of the blobs dataset. in the x-axis, we have the original measures, whilst, in the y-axis, the proposed counterparts are taken. the adjusted kdn is normalized between 0 and 1 for direct comparison. fig. 5. the adjusted ihm vs the original ihm and meta-learning prediction vs original ihm for the blobs dataset. full size image for the measure kdn, one can observe that as the hardness to classify the instances grows, both the adjusted and the original ihm increase their values, reaching their peak in the middle of the scale. after that, the value of the estimated ihm assumes the opposite direction of the original measure. this result is expected considering the unlabeled data, given that instances harder to classify without a label will be predicted as belonging to any other class rather than being an outlier from a specific class. conversely, in the td unpruned graphs, the results indicate that the hardness in classification is independent of the class being known or not. for both adjusted ihm and meta-learning approaches, there is some linearity between those measures and the original ihm. it means that the proposed measures for unlabeled data capture the increase of hardness for classification problems equivalent to the increase in hardness when the class is given for this measure. this result can be expected considering the nature of the measure. the cld measure, the only measure using likelihood as the metric, performed more closely to the original measure with the adjusted ihm for all datasets. especially for datasets with two classes, in many cases, both estimates might agree when the first and second classes with maximum likelihoods are the same. overall, adjusted and meta-learning ihms were able to assess the hardness level of the unlabeled instances, with some prominence of the adjusted measures, which showed larger correlations to the original measures in most cases. they are also simpler to compute, as they do not need to induce a ml model as in the meta-learning approach. in the absence of labels, most measures are more effective in pointing borderline instances as posing a higher difficulty of posterior classification. 5 conclusion and future work this research analyzed alternative ways to measure the hardness of instances for classification problems in scenarios where the label of an instance is unknown, that is, in the deployment stage. standard ihms from the literature were adapted to this scenario. their results were compared to the alternative of generating regression meta-models to predict the ihm values. both alternatives were effective on their behalf, correlating to the original ihms that need to know the label of each instance. the correlations were higher for some measures that do not rely as much on the labels, but the results for other measures are expected as their original formulation allows one to identify noise and outliers on data regarding their labels. the results encourages the usage of the adjusted measures in the deployment of ml models, allowing the identification of instances that ml models might struggle to classify. in future work, we will explore the patterns found in the comparisons between the original and adjusted ihm not presented in this work and alternative measures for unlabeled data not addressed in this research. we will expand the application of the adjusted measures and meta-learning to more datasets, and tuning the meta-models could lead to new findings about the characteristics of the instances. another fruitful direction will be to explore the usage of the adjusted measures for designing classification rejection options. references al hosni, o.s., starkey, a.: investigating the performance of data complexity & instance hardness measures as a meta-feature in overlapping classes problem. in: iccbdc 2023, manchester, united kingdom (2023). https://doi.org/10.1145/3616131.3616132 brazdil, p., van rijn, j.n., soares, c., vanschoren, j.: metalearning: applications to automated machine learning and data mining. springer, cham (2022). https://doi.org/10.1007/978-3-030-67024-5 book  math  google scholar  fapesp: fapesp covid-19 data sharing/br (2020). https://repositoriodatasharingfapesp.uspdigital.usp.br franc, v., prusa, d., voracek, v.: optimal strategies for reject option classifiers. j. mach. learn. res. 24(11), 1–49 (2023) mathscinet  math  google scholar  ho, t.k., basu, m.: complexity measures of supervised classification problems. ieee trans. pattern anal. mach. intell. 24(3), 289–300 (2002). https://doi.org/10.1109/34.990132 article  math  google scholar  kelly, m., longjohn, r., nottingham, k.: the uci machine learning repository (2023). https://archive.ics.uci.edu lorena, a.c., paiva, p.y., prudêncio, r.b.: trusting my predictions: on the value of instance-level analysis. acm comput. surv. 56(7), 1–28 (2024) article  math  google scholar  martínez-plumed, f., prudêncio, r.b., martínez-usó, a., hernández-orallo, j.: item response theory in ai: analysing machine learning classifiers at the instance level. artif. intell. 271, 18–42 (2019) article  mathscinet  math  google scholar  paiva, p.y.a., moreno, c.c., smith-miles, k., valeriano, m.g., lorena, a.c.: relating instance hardness to classification performance in a dataset: a visual approach. mach. learn. 111(8), 3085–3123 (2022) article  mathscinet  math  google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  prudêncio, r.b., silva filho, t.m.: explaining learning performance with local performance regions and maximally relevant meta-rules. in: xavier-junior, j.c., rios, r.a. (eds.) bracis 2022. lncs, vol. 13653, pp. 550–564. springer, cham (2022). https://doi.org/10.1007/978-3-031-21686-2_38 chapter  math  google scholar  rivolli, a., garcia, l.p., soares, c., vanschoren, j., de carvalho, a.c.: meta-features for meta-learning. knowl.-based syst. 108101 (2022) google scholar  schweighofer, e.: data-centric machine learning: improving model performance and understanding through dataset analysis. in: legal knowledge and information systems: jurix 2021, vol. 346, p. 54. ios press (2021) google scholar  smith, m.r., martinez, t., giraud-carrier, c.: an instance level analysis of data complexity. mach. learn. 95(2), 225–256 (2014) article  mathscinet  math  google scholar  valeriano, m.g., et al.: let the data speak: analysing data from multiple health centers of the são paulo metropolitan area for covid-19 clinical deterioration prediction. in: 2022 22nd ieee international symposium on cluster, cloud and internet computing (ccgrid), pp. 948–951. ieee (2022) google scholar  download references acknowledgements this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior—brasil (capes)—finance code 001. the authors thank fapesp for its support under grant 2021/06870-3. author information authors and affiliations universidade tecnológica federal do paraná, cornélio procópio, brazil patricia s. m. ueda & ana carolina lorena instituto tecnológico de aeronáutica, são josé dos campos, brazil adriano rivolli authors patricia s. m. uedaview author publications search author on:pubmed google scholar adriano rivolliview author publications search author on:pubmed google scholar ana carolina lorenaview author publications search author on:pubmed google scholar corresponding author correspondence to patricia s. m. ueda . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ueda, p.s.m., rivolli, a., lorena, a.c. (2025). an instance level analysis of classification difficulty for unlabeled data. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_10 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning instance hardness measures unlabeled data deployment of models publish with us policies and ethics profiles ana carolina lorena view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature analyzing the impact of coarsening on k-partite network classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper analyzing the impact of coarsening on k-partite network classification conference paper first online: 30 january 2025 pp 156–168 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) analyzing the impact of coarsening on k-partite network classification download book pdf download book epub thiago de paulo faleiros9, paulo eduardo althoff9 & alan demétrius baria valejo10  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 480 accesses abstract the ever-expanding volume of data presents considerable challenges in storing and processing semi-supervised models, hindering their practical implementation. researchers have explored reducing network versions as a potential solution. real-world networks often comprise diverse vertex and edge types, leading to the adoption of k-partite network representation. however, existing methods have mainly focused on reducing uni-partite networks with a single vertex type and edges. this study introduces a novel coarsening method designed explicitly for k-partite networks, aiming to preserve classification performance while addressing storage and processing issues. we conducted empirical analyses on synthetically generated networks to evaluate their effectiveness. the results demonstrate the potential of coarsening techniques in overcoming storage and processing challenges posed by large networks. the proposed coarsening algorithm significantly improved storage efficiency and classification runtime, even with moderate reductions in the number of vertices. this led to over one-third savings in storage space and a twofold increase in classification speed. moreover, the classification performance metrics exhibited low variation on average, indicating the algorithm’s robustness and reliability in various scenarios. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others coarsening effects on k-partite network classification article open access 05 december 2023 coarsening algorithm via semi-synchronous label propagation for bipartite networks chapter © 2021 effective k-vertex connected component detection in large-scale networks chapter © 2017 1 introduction semi-supervised learning has emerged as a practical approach for leveraging labeled data to guide a supervisor’s response. these methods harness both labeled and unlabeled data to facilitate the learning process. such algorithms consider the connections between labeled and unlabeled data to compensate for the absence of labels [1]. presenting data in the form of networks further enhances the potential of semi-supervised techniques, as it enables the extraction of relationships from the topological attributes of labeled and unlabeled data [2]. while the semi-supervised approach has offered a solution to reduce the reliance on human intervention, a significant challenge persists. as the volume of data grows, the storage cost and computational processing time required to train a semi-supervised model can become prohibitive, rendering it impractical for specific applications [3]. to address these limitations, researchers have extensively explored a technique involving reduced versions of networks instead of the original ones [4]. this approach minimizes storage requirements, improving algorithm performance compared to a full-sized network. one notable category within these techniques is the employment of coarsening algorithms, which group similar vertices together, effectively reducing redundant information. coarsening has long been established in visualization and graph partitioning [5], and more recently, it has demonstrated its efficacy in solving classification problems in homogeneous networks [6]. however, most of these methods concentrate on analyzing networks with a singular type of vertex and edge, referred to as uni-partite networks. in reality, information networks often exhibit heterogeneity, comprising various types of vertices and edges [7]. leveraging the information richness of the heterogeneous k-partite network allows the model to learn more resilient and generalizable data representations, leading to improved performance in downstream tasks like classification and prediction. as interest in techniques for heterogeneous networks grows, as evidenced by studies such as [4, 8, 9], research on coarsening methods has also gained momentum, particularly for bipartite networks [10,11,12,13,14]. the methods explicitly designed for heterogeneous networks still need to be explored [5]. in this study, we aim to assess the accuracy of coarsened heterogeneous networks in classifying vertices based on their relationships to other vertices within a semi-supervised context. it is important to note that coarsening can result in information loss and a potential decrease in classification accuracy. therefore, evaluating the trade-off between computational efficiency and classification accuracy becomes crucial when employing coarsening in heterogeneous network classification tasks. in this context, this study introduces the development of a novel coarsening method designed explicitly for k-partite networks. our proposed method utilizes a technique that organizes partitions and selects paths in the schema, resulting in improved coarsening performance compared to random approaches used in other studies [15]. our results demonstrate the potential of coarsening techniques in resolving storage and processing issues for large networks. this paper is organized as follows. section 2 discusses the background and foundational concepts necessary for understanding the research, including an overview of k-partite networks and their formal descriptions. section 3 introduces a novel coarsening algorithm specifically designed for k-partite networks, detailing its methodology and the theoretical underpinnings that ensure its efficiency and effectiveness. section 4 presents the experimental results, showcasing the performance of the proposed algorithm through various datasets. finally, sect. 5 concludes the paper by summarizing the findings and discussing their implications. 2 background a network, denoted as \(g=(v, e)\) or graph g, is referred to as k-partite if its vertex set v consists of k disjoint sets: \(v= \mathcal {v}_1 \cup \mathcal {v}_2 \cup ... \cup \mathcal {v}_k\). here, each \(\mathcal {v}_i\) and \(\mathcal {v}_j\) (\(1 \le i,j \le k\)) represent sets of vertices, and the edge set e is a subset of pairs from \(\bigcup _{i \ne j} \mathcal {v}_i \times \mathcal {v}_j\). in other words, every edge \(e=(a,b)\) connects vertices from different sets, where \(a \in \mathcal {v}_i\) and \(b \in \mathcal {v}_j\), with \(a i \ne j\). additionally, each edge (a, b) in the graph may be associated with a weight, denoted as \(\omega (a, b)\), where \(\omega : e \rightarrow \mathbb {r}^*\). moreover, individual vertices may have associated weights, represented as \(\sigma (a)\), where \(\sigma : v \rightarrow \mathbb {r}^*\). fig. 1. illustration of the coarsening process performed by the proposed algorithm in a heterogeneous network. in figure (a), the target partition is highlighted in red, while in figure (b), the other partitions are highlighted in blue, and the coarsening process reduces them. (color figure online) full size image in this context, a heterogeneous network is considered a specific type of k-partite network, where vertices of the same type form a partition, and connections between vertices of different types exist. these connections are undirected, and the relationships between nodes are symmetric. the weight of a vertex \(a \in \mathcal {v}_i\), represented as \(\kappa _a\), is defined as the total weight of its adjacent edges, expressed as \(\kappa _a = \sum _{b \in v} \omega (a, b)\). the h-hop neighborhood of vertex a, denoted as \(\vargamma _h(a)\), is formally defined as the set of vertices such that \(\vargamma _h(a) = \{b\ |\) there exists a path of length h between vertex a and vertex \(b \}\). thus, the 1-hop neighborhood of a, denoted \(\vargamma _1(a)\), consists of the vertices directly adjacent to a. similarly, the 2-hop neighborhood, \(\vargamma _2(a)\), comprises the vertices that are reachable from a in exactly 2 hops, and so on for higher values of h. in a k-partite network context, the network schema refers to the topological structure that links the k partitions together. formally, the network schema of a k-partite network g can be represented by the network \(s(g) = (v_s, e_s)\), where \(v_s\) is the set of k vertices associated with each partition, and \(e_s\) is the set of edges connecting these vertices. for any edge \((a, b) \in e_s\), vertex a belongs to a partition \(\mathcal {v}_i\), and vertex b belongs to a different partition \(\mathcal {v}_j\), where \(i \ne j\) and \(1 \le i, j \le k\). a metapath, in this context, is defined as a sequence of edges that connects vertices from different partitions in the network schema. our proposed technique employs a label propagation scheme that disseminates labeled vertices from a specific partition, known as the target partition \(\mathcal {v}_t\), to all other vertices within the network. let \(\mathcal {v}^l \in v_t\) denote the set of labeled vertices in the target partition, and \(\mathcal {v}^u \in v\) represent the set of unlabeled vertices. each vertex in \(\mathcal {v}^l\) is associated with a label from a set \(c=\{c_1, c_2, \ldots , c_m \}\) comprising m classes. the matrix \(\mathcal {y} \in \mathbb {r}^{|v| \times m}\) represents the labels for the corresponding vertices in v. for simplicity, we denote \(\mathcal {y}_{a, i}\) as the weight of label \(c_i\) assigned to a vertex a, and \(\mathcal {y}_{\mathcal {v}_i}\) as the labels assigned to a subset of vertices in partition \(\mathcal {v}_i\). the transductive learning algorithm inputs a labeled training set of vertices \(\mathcal {v}^l\) and a set of unlabeled test vertices \(\mathcal {v}^u\). it outputs a transductive learner f that assigns a label \(c_i \in c\) to each vertex a in \(\mathcal {v}^u\), denoted as \(f(a) = \mathop {\mathrm {arg\,max}}\limits _i \mathcal {y}_{a,i}\), signifying the label with the highest weight assigned to vertex a by the transductive learner. 3 coarsening algorithm for k-partite network this section presents our proposed coarsening algorithm designed to reduce k-partite networks, facilitating subsequent classification tasks. the algorithm leverages labeled vertices from the target partition \(\mathcal {v}_t\) to guide the reduction process. the k-partite network is initially decomposed into a series of bipartite networks, where pairs of partitions are selected from the original network. subsequently, an adaptation of the clpb (coarsening strategy via semi-synchronous label propagation for bipartite networks) [16] coarsening algorithm is applied to these partition pairs, wherein one partition acts as the propagator partition, denoted as \(\mathcal {v}_p\), and the other as the receptor partition, denoted as \(\mathcal {v}_r\). the coarsening process is executed semi-synchronously, grouping vertices into super-vertices from \(\mathcal {v}_r\) with identical labels. an illustration of the coarsening process in a k-partite network can be seen in fig. 1. this coarsening procedure aims to reduce the overall training time for transductive learning, as network-based methods generally exhibit complexity associated with the number of vertices and edges. after establishing label propagation as a matching approach for each bipartition, the next crucial step is determining the strategy for selecting the pairs of partitions and the order in which the clpb algorithm will be applied. when considering all possible partition pairs, networks with high connectivity schemas can lead to numerous procedures, resulting in a quadratic complexity concerning the number of vertices [15]. moreover, cycles in the network schema would cause information repetition. to address these challenges, our objective was to identify, for each partition, the most suitable neighboring partition to serve as a pair in the bipartite coarsening procedure. this neighboring partition, acting as the pair, is called the “guide partition.” one approach to reducing the number of partition pairs used is to identify paths in the schema of the k-partite network [17]. given that only the target partition possesses label information initially, a logical strategy for the coarsening procedure is to propagate the information from the target partition to others, leveraging the label information during the matching phase. shorter paths tend to indicate stronger relationships between vertices [18], making the selection of the shortest metapath between the two partitions preferable. the objective is to perform coarsening on all non-target partitions following a metapath. the procedure is executed synchronously, one partition pair at a time, with the partition pairs selected radially, starting from the target partition. initially, guide partitions at a 1-hop distance from the target are coarsened, followed by those at a 2-hop distance, and so on. when reducing each guide partition, the pair used is the neighboring partition within the metapath that leads from the partition being reduced towards the target partition. this chosen order serves two purposes: it propagates the label information initially present in the target partition, and it ensures that the selected neighbor partition has already undergone a reduction in the previous coarsening process (except in the first iteration). the process is illustrated in fig. 2. fig. 2. the coarsening process is executed radially, beginning from the target partition \(s_t = s_1\). the first step involves performing coarsening on the immediate neighbors of \(s_1\) (in (b)), which are solely influenced by the target partition. subsequently, coarsening is applied to partitions that are farther away from \(s_t\) (in (c)), with these partitions being influenced only by their respective guide partitions, which carry structural information from \(s_a\). full size image 3.1 algorithm description the coarsening procedure for a k-partite network is outlined in algorithm 1. it takes as input a k-partite network \(g=(v, e)\), a schema s(g) of the k-partite network, and a vertex \(\mathcal {s}^t\) in schema s(g) corresponding to the target partition \(\mathcal {v}_t\). the algorithm’s output is a coarsened version of the k-partite network g. consider \(p_i\) as the shortest metapath that originates from the labeled target partition \(s^t\) and terminates at the non-target partition \(s^i\). the algorithm chooses the first one found in cases where multiple shortest metapaths exist. for each \(s^i\), a partition \(s_{j}\) at a 1-hop distance from \(p_i\) is selected as the “guide partition.” the coarsening process begins with a breadth-first search, starting from \(\mathcal {v}_t\), to select a non-target partition \(\mathcal {v}_i\) using a guide partition \(\mathcal {v}_j\). the objective is to replace the bipartite subnetwork \(g_i=(\mathcal {v}_j, \mathcal {v}_i, e_i)\) in g with its condensed version \(g_i^c=(\mathcal {v}_j, \mathcal {v}_i^c, e_i^c)\), consisting of super-nodes and super-edges obtained during coarsening. it is essential to note that although the guide partition \(\mathcal {v}_j\) supports the bipartite coarsening process, only the partition \(\mathcal {v}_i\) is updated in the network g at a time (see line 17 in algorithm 1). the algorithm then returns the coarsened network \(g^c\), obtained by applying coarsening to all non-target partitions of the original k-partite network. 4 experimental results experimental studies were conducted on synthetic and real datasets using transductive classification to evaluate the effectiveness of the proposed coarsening algorithm. the principal reduction objectives, namely memory savings and classification runtime, were analyzed as the number of vertices increased. additionally, the accuracy of various metrics, such as accuracy, precision, recall, and f-score, was compared for each reduction level concerning the original uncoarsened network. the subsequent sections present the results of these experiments, along with insights drawn from the findings. algorithm 1 coarsening procedure for non-target partitions full size image fig. 3. distinct topologies networks schemes generated in the experiments. hierarchical star – figure (a), hierarchical web – figure (b), and bipartite topology – figure (c). full size image 4.1 synthetic network generation given the scarcity of standard datasets for analyzing heterogeneous data in various fields, we utilized a synthetic network generator called hnoc [19] to address this limitation and create k-partite networks. hnoc was selected for its flexibility in adjusting partition size, the number of potential classes, classification probability, noise, and dispersion levels. initially developed for community detection, the hnoc tool’s concept of communities can be extended to perform data classification in a semi-supervised setting. initially, the tool assigns each vertex precisely to its designated community. subsequently, each vertex within a community is connected to all other vertices in the same community and those in different partitions. edges are selectively removed based on the dispersion parameter to control community density. lower dispersion values create sparser communities, while higher dispersion values lead to denser communities. the noise parameter is utilized for edges between different communities, also known as inter-community edges. network noise impacts the identification of community boundaries and increases overlap, making finding communities within the network more complex. higher noise levels can significantly decrease classification accuracy and increase complexity. generally, noise values greater than 0.5 result in networks with poorly defined and sparse community structures, where inter-community edges outnumber intracommunity edges. optimal noise values typically range from 0.1 to 0.4, striking a balance between increasing class detection difficulty while preserving the overall network structure [19]. it is crucial to note that no edges connect vertices within the same partition. diverse object types and topological structures are necessary in heterogeneous networks. it is essential to generate various network topologies to simulate real-world scenarios and complex systems effectively. in addition to the dispersion and noise parameters, the type of topological structure in the network was also varied. three specific topologies were selected for the study, namely the hierarchical star (fig. 3(a)), the hierarchical web (fig. 3(b)), and the bipartite topology (fig. 3(c)). 4.2 experimental setup the experiments were categorized into two groups: the first utilized synthetic data and the second involved a real dataset. in the first group of experiments, a wide range of parameter configurations was considered to generate synthetic k-partite networks with diverse class structures randomly [20]. the number of vertices varied among 2, 000 and 15, 000 for each scheme (fig. 3). the number of classes ranged from 4 to 10 with increments of 1, the dispersion varied from 0.1 to 0.9 with increments of 0.1, and the noise value was fixed at 0.3 to create more challenging networks for classification without excessive degradation. ten distinct random networks were generated for each configuration, resulting in over 13, 000 networks. the vertices were evenly distributed among the non-target partitions. to accurately reflect the memory savings achieved by the coarsening algorithm, the non-target partitions were set to be five times larger than the target partition. the second group of experiments involved a real-world heterogeneous network using the dblp datasetfootnote 1 for the classification task. the dblp dataset contains open bibliographic information from major computer science journals and proceedings. in this study, the task was to classify the authors into four areas of knowledge, and the authors served as the target partition. the non-target partitions included the articles written by each author, the conferences where the authors have published, and the terms found in these articles. the dblp dataset comprises 14, 475 authors, 14, 376 articles, 20 conferences, and 8, 920 terms, resulting in 37, 791 vertices and 170, 795 edges, with a dispersion value of approximately 0.002. the main objective of this study is to evaluate the effectiveness of the proposed coarsening algorithm in transductive classification tasks involving k-partite networks. for the experiments, the gnetmine algorithm [21] was chosen, as it is a widely used reference algorithm in heterogeneous classification and has been utilized as a benchmark in numerous previous studies [7, 17, 22,23,24,25,26]. all generated networks were classified using gnetmine, and metrics such as precision, recall, f-score (macro variant), accuracy, and classification time were recorded. the proposed coarsening algorithm was applied to the networks, with \(20\%\) and \(80\%\) reduction levels. the transductive classification was evaluated by varying the number of labeled vertices, using \(1\%\), \(10\%\), \(20\%\), and \(50\%\) of the vertices from the target partition. the obtained results serve as a comparative reference for classification metrics and the storage size of the networks. 4.3 results on synthetic data the analysis of memory economies was conducted concerning the dispersion parameter. the detailed results in table 1 demonstrates that networks with higher edge densities incur increased storage and classification runtime costs. consequently, coarsening is more effective in such scenarios, yielding more substantial benefits. the relative economy of memory and classification runtime for each reduction level compared to the original unreduced network is presented in table 1. interestingly, there is a notable increase in the relative economy during the early stages of coarsening, leading to satisfactory results with just a \(20\%\) decrease in the number of vertices. this observation may be attributed to the initial levels already clustering many vertices with more shared connections. table 1. compared to their original forms, reducing networks can lead to savings in both storage size and classification time. for both metrics, the values are represented as percentages.full size table table 2 presents the relationship between network reduction and classification metrics. the findings indicate that as the networks grow, the impact of coarsening on classification metrics becomes more pronounced. for instance, when dealing with a network of 15, 000 vertices, reducing it by \(20\%\) and \(80\%\) results in a relative f-score loss of \(1.69\%\) and approximately \(11.97\%\), respectively. additionally, the precision metric remains relatively stable, suggesting that the proposed coarsening algorithm is better suited for applications seeking to reduce the number of false positives. furthermore, the relatively low f-score loss achieved by reducing the dataset by about \(20\%\) is particularly noteworthy, especially considering the significant savings in storage and classification runtime. table 2. the loss incurred during network reduction becomes more pronounced as the network size increases.full size table 4.4 results on real dataset the results in table 3 indicate that despite a \(67\%\) reduction in the network, there was no significant decrease in classification metrics. however, the reduction in storage size was not as effective, as shown in table 4. these findings align with the observations from the synthetic experimental evaluation. the network \(g_{dblp}\) exhibits a low dispersion level of approximately 0.002. as previously discussed, such low dispersion levels, nearing zero, lead to a minimal loss in classification quality and offer limited storage savings. table 3. metrics of accuracy, precision (macro), recall (macro) and f-score (macro) according to \(g_{dblp} \) reductionfull size table table 4. storage (megabyte – mb) and time (seconds s) savings analysis for \(g_{dblp} \)full size table 5 conclusion the primary objective of this study was to evaluate a proposed algorithm for reducing the size of k-partite networks while maintaining classification performance. the study successfully obtained metrics related to resource savings and classification performance, confirming the efficacy of the proposed coarsening algorithm for k-partite networks. however, it is essential to acknowledge certain limitations. using synthetic networks instead of real-world networks may introduce some bias, especially considering the high assortativity levels generated by the hnoc tool, which might only be representative of some real-world scenarios. furthermore, the study’s experiment design involved a limited number of k-partite network schemes, which should be considered a potential limitation. nevertheless, various experiments and parameter configurations thoroughly assessed the algorithm’s performance, showcasing its potential for diverse network applications. the researchers have made the entire source code used in the experiments availablefootnote 2, enabling future works to test different parameters and networks. the study’s findings highlight that the proposed coarsening algorithm achieved significant savings in storage and classification runtime, even with modest reduction levels. for instance, a \(20\%\) reduction in the number of vertices resulted in over 1/3 savings in storage and twice faster classifications. moreover, the classification performance metrics exhibited low average levels of variation, demonstrating the algorithm’s stability and reliability. notes 1.dblp dataset available at https://dblp.org. 2.url with code and experiments repository, https://github.com/pealthoff/coarseklass. references silva, m.m.: uma abordagem evolucionária para aprendizado semi-supervisionado em máquinas de vetores de suporte. master’s thesis, escola de engenharia ufmg, belo horizonte (2008) google scholar  aggarwal, c.c.: machine learning for text. springer, cham (2018). https://doi.org/10.1007/978-3-319-73531-3 walshaw, c.: multilevel refinement for combinatorial optimisation problems. ann. oper. res. 131(1), 325–372 (2004). https://doi.org/10.1023/b:anor.0000039525.80601.15 liu, y., safavi, t., dighe, a., koutra, d.: graph summarization methods and applications: a survey. acm comput. surv. 51(3) (2018). http://arxiv.org/abs/1612.04883 valejo, a.d.b.: métodos multi-nível em redes bi-partidas. ph.d. dissertation, instituto de ciências matemáticas e de computação usp, são carlos (2019) google scholar  liang, j., gurukar, s., parthasarathy, s.: mile: a multi-level framework for scalable graph embedding (2020) google scholar  romanetto, l.d.m.: classificação transdutiva em redes heterogêneas de informação, baseada na divergência kl. ph.d. dissertation, instituto de ciências matemáticas e de computação usp, são carlos (2020) google scholar  zhou, j., et al.: graph neural networks: a review of methods and applications. ai open 1, 57–81 (2020). http://arxiv.org/abs/1812.08434 wu, z., pan, s., chen, f., long, g., zhang, c., yu, p.s.: a comprehensive survey on graph neural networks. ieee trans. neural netw. learn. syst. 32(1), 4–24 (2021). https://ieeexplore.ieee.org/document/9046288/ valejo, a., ferreira, v., filho, g.p.r., oliveira, m.c.f.d., lopes, a.d.a.: one-mode projection-based multilevel approach for community detection in bipartite networks. in: annual international symposium on information management and big data simbig. ceur-ws (2017) google scholar  valejo, a., ferreira, v., de oliveira, m.c.f., de andrade lopes, a.: community detection in bipartite network: a modified coarsening approach. in: lossio-ventura, j.a., alatrista-salas, h. (eds.) simbig 2017. ccis, vol. 795, pp. 123–136. springer, cham (2018). https://doi.org/10.1007/978-3-319-90596-9_9 chapter  math  google scholar  valejo, a., ferreira de oliveira, m.c., filho, g.p., de andrade lopes, a.: multilevel approach for combinatorial optimization in bipartite network. knowl.-based syst. 151, 45–61 (2018). https://www.sciencedirect.com/science/article/pii/s0950705118301539 valejo, a., faleiros, t., de oliveira, m.c.f., de andrade lopes, a.: a coarsening method for bipartite networks via weight-constrained label propagation. knowl.-based syst. 195, 105678 (2020). https://www.sciencedirect.com/science/article/pii/s0950705120301180 valejo, a., et al.: coarsening algorithm via semi-synchronous label propagation for bipartite networks. in: anais da x brazilian conference on intelligent systems, porto alegre, rs, brasil, sbc (2021). https://sol.sbc.org.br/index.php/bracis/article/view/19047 zhu, l., ghasemi-gol, m., szekely, p., galstyan, a., knoblock, c.a.: unsupervised entity resolution on multi-type graphs. in: groth, p., et al. (eds.) iswc 2016. lncs, vol. 9981, pp. 649–667. springer, cham (2016). https://doi.org/10.1007/978-3-319-46523-4_39 chapter  math  google scholar  valejo, a.d.b., et al.: coarsening algorithm via semi-synchronous label propagation for bipartite networks. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13073, pp. 437–452. springer, cham (2021). https://doi.org/10.1007/978-3-030-91702-9_29 chapter  math  google scholar  luo, c., guan, r., wang, z., lin, c.: hetpathmine: a novel transductive classification algorithm on heterogeneous information networks. in: de rijke, m., et al. (eds.) ecir 2014. lncs, vol. 8416, pp. 210–221. springer, cham (2014). https://doi.org/10.1007/978-3-319-06028-6_18 chapter  math  google scholar  gupta, m., kumar, p., bhasker, b.: heteclass: a meta-path based framework for transductive classification of objects in heterogeneous information networks. expert syst. appl. 68, 106–122 (2017). https://linkinghub.elsevier.com/retrieve/pii/s0957417416305462 valejo, a., góes, f., romanetto, l., ferreira de oliveira, m.c., de andrade lopes, a.: a benchmarking tool for the generation of bipartite network models with overlapping communities. knowl. inf. syst. 62(4), 1641–1669 (2020). https://doi.org/10.1007/s10115-019-01411-9 valejo, a., góes, f., romanetto, l., ferreira de oliveira, m.c., de andrade lopes, a.: a benchmarking tool for the generation of bipartite network models with overlapping communities. knowl. inf. syst. 62(4), 1641–1669 (2019). https://doi.org/10.1007/s10115-019-01411-9 article  google scholar  ji, m., sun, y., danilevsky, m., han, j., gao, j.: graph regularized transductive classification on heterogeneous information networks. in: balcázar, j.l., bonchi, f., gionis, a., sebag, m. (eds.) machine learning and knowledge discovery in databases, pp. 570–586. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-15880-3_42 zhi, s., han, j., gu, q.: robust classification of information networks by consistent graph learning. in: appice, a., rodrigues, p.p., santos costa, v., gama, j., jorge, a., soares, c. (eds.) ecml pkdd 2015. lncs (lnai), vol. 9285, pp. 752–767. springer, cham (2015). https://doi.org/10.1007/978-3-319-23525-7_46 chapter  math  google scholar  bangcharoensap, p., murata, t., kobayashi, h., shimizu, n.: transductive classification on heterogeneous information networks with edge betweenness-based normalization. in: proceedings of the ninth acm international conference on web search and data mining (2016) google scholar  faleiros, t., rossi, r., lopes, a.: optimizing the class information divergence for transductive classification of texts using propagation in bipartite graphs. pattern recogn. lett. 87, 04 (2016) math  google scholar  luo, j., ding, p., liang, c., chen, x.: semi-supervised prediction of human mirna-disease association based on graph regularization framework in heterogeneous networks. neurocomputing 294, 29–38 (2018). https://www.sciencedirect.com/science/article/pii/s0925231218302674 ding, p., shen, c., lai, z., liang, c., li, g., luo, j.: incorporating multisource knowledge to predict drug synergy based on graph co-regularization. j. chem. inf. model. 60(1), 37–46 (2019) article  google scholar  download references acknowledgment this work is supported by são paulo research foundation (fapesp) under grant numbers 2022/03090-0. author information authors and affiliations university of brasília (unb), brasília, df, brazil thiago de paulo faleiros & paulo eduardo althoff department of computing, federal university of são carlos (ufscar), são carlos, sp, brazil alan demétrius baria valejo authors thiago de paulo faleirosview author publications search author on:pubmed google scholar paulo eduardo althoffview author publications search author on:pubmed google scholar alan demétrius baria valejoview author publications search author on:pubmed google scholar corresponding author correspondence to thiago de paulo faleiros . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de paulo faleiros, t., althoff, p.e., valejo, a.d.b. (2025). analyzing the impact of coarsening on k-partite network classification. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_11 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords graph network coarsening k-partite network node classification transductive learning publish with us policies and ethics profiles thiago de paulo faleiros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature transfer learning of shapelets for time series classification using convolutional neural network | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper transfer learning of shapelets for time series classification using convolutional neural network conference paper first online: 28 november 2021 pp 325–339 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) transfer learning of shapelets for time series classification using convolutional neural network download book pdf download book epub alexandre felipe muller de souza  orcid: orcid.org/0000-0003-3090-031x10, mariane r. s. cassenote  orcid: orcid.org/0000-0003-0489-223x10 & fabiano silva  orcid: orcid.org/0000-0001-5453-617510  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1339 accesses 1 citation abstract time series classification has a wide variety of possible applications, like outlines of figure types, signs of movement and sensor signals. this diversity may present different results with the application of machine learning techniques. among the different ways to classify time series, two, in particular, are explored in this paper: the shapelet primitive and the neural network classification. a shapelet is a sub-sequence of the time series symbolic, like a high-level descriptor, that are representative for the class to which they belong. these features act as common knowledge used by domain experts. this paper proposes a cnn training protocol to achieve better results in the classification of time series. the idea consists of decomposing the original time series into shapelets and noise. then, the shapelets are used to train a classifier while the no-shapelets (“signal noise”) is used to train another classifier. the original time series is then used to train two final classifiers starting with the weights of shapelet and no-shapelet classifiers. this previous extraction of this representation can improve classification ability in a convolutional neural network using transfer learning. the experimental evaluation shows that the pre-selection of shapelets before the network training changes the classifier results for several databases and, consequently, improves classification accuracy. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others learning multivariate shapelets with multi-layer neural networks for interpretable time-series classification article 04 march 2021 learning perceptual position-aware shapelets for time series classification chapter © 2023 temporal representation learning for time series classification article 21 july 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization learning algorithms machine learning shape analysis statistical learning time series analysis 1 introduction with the technology advancement and the mobile devices popularization, systems that generate real-time data have become disseminated. the amount of data generated is continuously growing and meaning attributions are needed to extract relevant information from this data. in addition to sensors, many other types of data can be represented in structures called time series. these structures can be handled as shapes, that can be projected in a one-dimensional plane or spectrogram [3]. simply, any string of numerical values whose order forms a sequence [10] can be called a time series. naturally, the processing of these time series has applications in the most diverse areas of knowledge [7]. the scope of this work is limited to the problem of automated classification of time series: inputs are divided into two or more classes to be determined. in particular, the focus in this analysis is supervised classification, i.e., given a series whose classification is not known, the class to which it belongs must be inferred. this is accomplished through other examples previously classified among these classes, not exactly the same, but similar. naturally, the series repository [3] already makes available these pre-classified (and other unclassified) examples by the domain expert. although there are several methods proposed for classifying time series, we will cover two of them in more details: symbolic representations, in particular the shapelet primitive extraction, and convolutional neural networks (cnn). in the shapelet method, are extracted the sub-sequences of the time series that are partial and representative attribute (thus, symbolic) for the class to which they belong. the idea is to extract features from domain problem that allow to perform the classification. in contrast to classification using symbolic representations paradigm, there are classifiers based on neural networks. these classifiers use a data model representing the memory in the form of weights of a network. neural classifiers are known for their efficiency, as their model has already been trained, but learning is based purely on adjusting the weights of the network. this adjustment is given by an error minimization algorithm that can achieve different results depending on many factors such as network architecture, number of layers, filters, among others. cnn-based classifiers have many applications and are referred to by the term “deep learning” [11]. fig. 1. diagram of experiments. full size image this paper proposes a pre-training method for time series classifiers using cnn. figure 1 outlines the basic process of our method. usually the network is started with random weights. however, in a different way, our proposal is to carry out an initial phase that extracts the relevant shapelets and separates the symbolic representations from the rest of the series, generating two new databases: one with only the parts of the series that represents shapelets, and other with only the remaining parts of the series. these databases are then used to train different models. then a final model is generated with the original database, but starting the network with the information learned before by the previous trained models. the motivation of this approach refers to the duality between these symbolic and neural classifiers and how their performance improve by exchanging information. through this study, we observe that the pre-initialization of the network increases the accuracy of the final classification for most of the tested bases. there are two methods of improvement. the first consists in extracting the symbolic representations, in this case shapelets, and training only with these parts of the original series and then transferring the learned weights to the final model, which allows highlighting the symbolic representations. this case is being called “classifier 4” in fig. 1. another method removes the shapelets and trains only with the “noisy part” of the original series and then apply the transfer the weights to the final model, called “classifier 5” in the figure. the rest of this paper is organized as follows: related works are commented on in sect. 2. the materials and method used are presented in sect. 3. section 4 contains the results and discussions, and finally, sect. 5 concludes the paper and list some future works. 2 related work in order to obtain the necessary conditions to compare classification models, researchers often use an openly available database of ucr time series [3]. this is a popular time series benchmark due to its diversity. the time series classification is divided into several paradigms for generating models, whether symbolic or statistical. most works based on symbolic paradigms use shapelets due to the intelligibility of the model. according to [20], shapelets can provide results that should help domain experts better understand their own data. in [12] is presented an interface to visualize the extracted shapelets in an interactive way, which can also be used at different extraction methods. the first method was proposed by the same authors [13] and is based on intermediate symbolic representations, called symbolic aggregate approximation (sax). this concept is widely explored and was presented for the first time in [14]. it consists of discretizing the series into bands that become symbols of an alphabet and, consequently, each sub-sequence generate a word. in [13] the sax words are converted into shapelets through a structure of weighted bitmaps of the sax words to determine the quality of the shapelet candidates, transforming the shapelet selection problem into a heuristic search process. according to the authors, this process makes finding shapelets more efficient. the main shapelet-based classification methods cited in [12] are: hive-cote (hierarchical vote collective of transformation based ensembles) [16], elis (efficient learning interpretable shapelets) [4] and lts (learning time-series shapelet) [8]. in addition, the boss method (bag of sfa symbols) of [17] is also referenced by [19] and [6]. each of these will be explained in detail below. one of the most popular time series classification algorithms is called boss. it generates a set of symbolic sfa representations (symbolic fourier approximation) with different sliding window sizes and creates an ensemble based on a template similar to bag of words (bow). this model is a set with occurrences counting of symbols, disregarding other elements such as the order in which they appear. compared to shapelets, these representations are extremely short and have a wide range of sizes. its classification is based on a histogram of occurrences rather than a decision tree. the boss is noise tolerant, scale independent, and position invariant. one of the works that supports several others in the field of time series classification is the use of a collection of classifiers called cote [1]. in this type of approach there are 35 classifiers [16]. each classifier is trained individually and then the results are combined by assigning weights to the classifiers [15]. a later variation of this method, called hive-cote, is based on a hierarchical vote between classifiers of the same nature. in this method there are a total of five modules. for example, one module would be based on shapelets and another module would be based on bow. superior results were obtained through this hierarchical voting process. however, despite both works showing positive results, they use classifiers from different paradigms, such as based on examples, symbolic, statistical and connectionist. this makes the final classification unintelligible. another work related to this paper that presents a time series classifier using shapelets is elis [4], which has a faster extraction method than the traditional one and shows good results. this method works in two phases: shapelet discovery and shapelet adjustment. in the first stage, the generation of sub-sequences occurs in a process called paa (piecewise aggregation approximation). this process is like a discretization that reduces the resolution of the series. for each class the paas, words are ranked and this information helps to adjust the shapelet numbers to be automatically extracted. is created a model that tries to adjust the shapelet to the best shape. the main idea is that true shapelets shouldn’t appear in exactly the best shape in the training set [4]. finally, in the second stage, several classifiers are built, each with its specific shapelets. then a regression process is used to better adjust the shapelet in the series and generate a final classifier. elis presents limitations in small databases, as well as the number of parameters that need to be adjusted. in order to outline these limitations, the authors recently proposed the elis++ method [21], which in addition to using data augmentation, also adjusts its parameters automatically. another very relevant work in the area of shapelets is lts[8]. in that work, the authors obtain the shapelets not by candidate calculations, but through a gradient descent algorithm. in this way, the process becomes very similar to adjusting the weights of a neural network. although efficient, all these classifiers do not completely replace the use of a classifier based on a cnn. one work that managed to take cnn comparatives in time series was [6]. in this time series classification review, the authors reproduced the main implementations and compared deep learning methods with other methods. according to the authors, hive/cote is currently considered the state-of-the-art algorithm for time series classification when evaluated in the 85 datasets of the ucr file. through the empirical study carried out by the authors, resnet came closest to cote/hive in terms of accuracy. on the other hand, when talking about time series, is a consensus that there is a lack of research on the use of cnn in this context[2, 5]. therefore, we have an important area to be explored, as learned symbolic representations (as they are in images) offer an important intelligibility that can also be used for time series. the implementation of the cnn presented in this paper was derived from [19], which has already obtained results close to the state-of-the-art in 2016. considering all this context, there seems to be an interesting research line to correlate classifiers based on cnn and shapelets. the motivation for this proposal is not new. several authors focus on this theme, although each one has its own methodology and presentation. for example, in [2] the authors assume that learning lts shapelets is a particular case of learning in convolutional networks. the authors explain that the measure of distance from shapelets can be transformed into a convolutional form and, thus, there is a relation between both methods. however, this specific study only aims to propose a multivalued classifier based on cnn. thus, the study justifies its positive result by the ability to learn these representations similar to shapelets. also in this work, the authors present a theoretical formalization, correlating the layers of a cnn with shapelets and calculating the euclidean distance. although the authors present this theoretical formalization, they do not deepen this relationship, showing it experimentally. although not directly related, some other knowledge is involved with this issue. another relevant work to be cited is [5], which explores the issue of transfer learning. this concept is realized when the weights of a network previously trained in another series are used as initial training weights, instead of randomly assigned weights. this learning process is faster and has better generalization, which is due to the transfer of representations between the original series and those that received the knowledge. the article even mentions series based on symbols, as a particular case of good generator of models. thus, it is evident that, when transferring knowledge from this series, the convergence of the neural network is faster, reinforcing the hypothesis that the neural network model learns symbolic representations (shapes). this experiment influences the development of our proposal, as we will see in sect. 3. 3 materials and method in this section, some implementation details of the experiments performed in this work are presented. the classifier, the source code and the method used are available for consultationfootnote 1. this work aims to serve as a basis for derivative works, whether for reproduction of procedures, analysis or inspiration for other classifiers based on cnn. the experiment that generates the classifier began by choosing the series that make up the database. the choice of the 20 series represented in table 1 was due to their diversity, as well as to the results published in previous works of classifiers based on symbolic representations. the data are presented in text files with different formats, but already normalized and standardized. the chosen database has training and testing examples, i.e., each dataset has pre-classified examples in predetermined quantities to be used in model generation and validation. table 1. selected time series from [3].full size table 3.1 shapelet extraction this step is the differential of this work in relation to a conventional training using a classifier based on neural networks. the extraction of shapelets is a task that usually demands a lot of processing time and neural network training. finding these sub-sequences is an additional step that allows to transfer them to the model to be used in the original series. given the set of series, the most relevant shapelets from each one were extracted. for this, we used the traditional method with a sliding window and candidate dictionary [20] with sizes from 5 to 18. this choice of size was adjusted to balance the proportion of what was shapelet and what was not in similar proportions in our database. in this method, an sorted list of all representations is generated with the computation of the gain. if we had to choose only one shapelet per class, choosing the first element from this list would be enough. in our case, the three most relevant are selected, discarding intersections. this is done by looking at the position in which the shapelet was extracted in the original series: if there is a conflict in the position (however different examples are involved), the deletion occurs. one to three most relevant shapelets from these candidates were chosen. the choice criterion is based on the gain and on having no intersection for most series. the resulting shapelets serve to generate two extra supporting databases, totaling three databases named below: original series; shapelet series: where only the parts corresponding to the shapelets is mantained from the original series and everything else is replaced by the central value of the series. a central value is understood as the mean value of the extremes of the complete series; noise series: where only the parts that do not match the shapelets were mantained from the original series, with the part corresponding to the shapelets replaced by the central value. the process to generate these two databases is shown as example in fig. 2. if we ignore the parts filled by the central value, the join between the shapelet series and the noise series corresponds to the original series. fig. 2. example of shapelet and noise series generation with 2 shapelets. full size image 3.2 neural network training as described in sect. 2, the one-dimensional convolution training process is performed from the three databases. the initial training is based on the implementation of [19], which does not use the concept of shapelet. the only pre-processing performed was to shuffle the training base to improve neural network training. the network architecture has been empirically adjusted to 3 one-dimensional layers. the first convolutional layer of the network has 32 filters and kernel length 8 (window equivalent concept), the second layer has 64 filters and kernel length 5 and the third convolutional layer has 32 filters and kernel length 3. at the end, there is a pooling layer and the output corresponds to the number of classes, thus obtaining the probability of classification of each one. regarding [19], the removal of “dropout” was performed. this technique consists of randomly removing some neurons to improve generalization and avoid overfitting [18]. to verify the impact of the transfer learning on classifier performance, we conducted the following experiments: 0. process that precedes the training: extraction of shapelets and generation of support databases; 1. original series training starting with random weights; 2. shapelet series training starting with random weights; 3. original series training starting with model imported from experiment 2; 4. noise series training starting with random weights; 5. original series training starting with model imported from experiment 4; 6. cross validation, shapelet series training, importing model from 4; 7. cross validation, noise series training, importing model from 2. the objective of item 1 is to create a baseline comparison for the following tests. in this way, the results of this item should be similar to those obtained in [19]. items 2 and 4 aim to obtain a training model formed by shapelets or by the rest of the series without shapelets (noise). items 3 and 5 are the most important, as they aim to show the relevance of transferring the previously trained model. as all series have the same length filled with central value of the series, changing the initial model is possible. finally, items 6 and 7 serve to elucidate and ratify the importance of transfer learning in each situation. as the classification model is based on a neural network, if the same information is not put into training again, the weights suffer from loss of information during training. this means that if experiment 6 gives better results than 7, the shapelets are the most relevant piece of information. thus, although all experiments have the same network architecture, only experiments 1, 3 and 5 use the complete data. all others use partial data. 3.3 training and transfer learning the extraction of symbolic representations, is performed by a deterministic algorithm, so its result is not changed in different executions. however, as the training process contains random factors related to weight adjustment, neural network training may vary. therefore, experiments resulting from neural network training need to be statistically evaluated. the selected databases from table 1 are separated between training and validation (or testing). during neural network training, weights are adjusted with examples from the training base. at the end of each epoch (which corresponds to the end of the adjustment of the weights), the accuracy of the model in the validation base is verified, but this result is only a consultation. while training takes place, the error in the training base continues to be adjusted and decreased, but at the same time the error in the validation base increases. this difference is basically a form of over-adjustment. given the size of the time series bases, learning in 300 epochs was determined, with the best model between these epochs being saved for the validation step. for most of the tested series, the overfitting starts to happen between 150 and 250 epochs, remaining saved for the best model. we chose to run for 300 epochs because later models are discarded. as a rule, the model validated only on the training basis achieves 100% accuracy in this way. the best model is given by a checkpoint, i.e., as training continues and the error increases, the best model is saved until a better one is found. therefore, there is no stopping criterion in this 250 epoch range because the best model is already saved. as we transfer the model within the domain itself, it is expected that every experiment that receives a pre-trained model will rapidly evolve in number of epochs to reach a satisfactory result. the behavior in this transfer learn situation is the faster evolution of the model’s accuracy [5]. for this reason, the greatest expectation in defining the experiments was that experiment 3, in addition to evolving quickly, would have better accuracy. this expectation is also due to the theoretical context exposed in sect. 2. 4 experimental evaluation as explained in sect. 3, a total of 30 training rounds were carried out. for comparison with other works, it is expressed as means in table 2. fig. 3. accuracy evolution for experiments 1, 2 and 4. full size image the average accuracy of experiment 1 is equivalent to that presented in [19] in the databases used in both works. this validate our similar implementation of the proposed model. experiment 4 outperformed experiment 2 (fig. 3). this suggests that there is still a lot of information relevant to classification that is comprised outside of extracted shapelets. in part, this is due to the size of the extracted shapelets that have absolute lengths of at most 18 values, which for most bases is a small length. an increase in this length would adjust this result, but at the cost of not making sense to highlight a very large segment. as the experiment extracts the segment from the original series, making the shapelet segment too large would make the initial premise of putting the shapelet in evidence compromised, as the shapelet could be almost the entire length of the series. it becomes evident that setting parameters like the size of the extracted shapelets, the quantity of the extracted shapelets and the relevance of the shapelet is a difficult task. fig. 4. accuracy evolution for experiments 1, 6 and 7. full size image another indicator of shapelet quality are the two cross-validation experiments 6 and 7. both are inferior to experiment 1 at all epochal amounts (fig. 4). even so, experiment 6 was superior to experiment 7. fig. 5. critical difference diagram for nemenyi test. full size image experiments 3 and 5 with transfer learn from the shapelet and noise series constitute the final objective. the result of the accuracy in the validation basis for experiments 1, 3 and 5 is described in table 2 for each series (fig. 6). for comparison, other 4 related classifiers in evidence were chosen, as described before. table 2 is expressed as means (of 30 measurements), each series of measurements has a standard variation in the order of \(10^{-5}\). the student’s t-test of both between experiments 1 vs 3 and separately experiments 1 vs 5 obtained p-values in the order of \(10^{-15}\) each, so it is a high degree of reliability that the series present statistically relevant differences along the training epochs. after rejecting the null hypothesis, we then proceed with a post-hoc analysis in the best classifier as possible to see in diagram fig. 5. the significance level of the tests is alpha=0.050 and critical distance of 0.76. we can see that on final result experiment 3 and experiment 1 have no significant differences and experiment 5 is significant different. table 2. accuracy of the main methods with the best result in bold.full size table an important result is that of these 20 sets, at least 15 sets (in experiments 3 and 5) were better when compared to experiment 1 (baseline). another relevant factor is that the accuracy of the classification overtaken what is equivalent to the state-of-the-art in half of the cases. when evaluating the state-of-the-art on the theme of symbolic representations, it was expected that the model trained with the shapelets series would be similar to the training model of the original series started from scratch. however, there were situations where transfer learning showed superior results when started from shapelets and in some other cases where the final result was better when training started by the noise set. in these, it seems that there is an external knowledge towards symbolic representations that support a more precise classification with less overfitting. in many cases, the results obtained in terms of accuracy of the time series classifier reach the reference works. the same results in simple base training, using the implementation of [19], would be repeatedly similar. however, by including the symbolic representations (which, in most cases, are the classification criteria of domain experts), the weights adjust more quickly (which is expected) and improve the generalization of learning, two relevant benefits. figures 3, 4, 6, 7 and 8 show the accuracy evolution graphs of all experiments in comparison with experiment 1 as baseline. fig. 6. accuracy evolution for experiments 1, 3 and 5. full size image fig. 7. accuracy evolution for experiments 1, 2 and 3. full size image fig. 8. accuracy evolution for experiments 1, 4 and 5. full size image 5 conclusion by exploring shapelets we obtained important results for time series classification in most of the tested databases. we believe that the objectives of this work have been achieved, since the performed experiments presented better results than training purely started from scratch. thus, the importance of extracting symbolic representations in time series classifiers is evident, as they can be useful even in classifiers based on convolutional networks. learning based on neural networks is not trivial, as it requires the data analyst to adjust their experiments with different parameters. only the continued training of the model, without parameter changes, would invariably lead to an increase in error. the proposed method, therefore, has a great advantage in this type of classifier. throughout this work, some points identified as limitations have already been pointed. among them are experiments, whose results were less than expected, such as specific domains. furthermore, there seems to be a need to resolve a question not yet fully answered: are there non-symbolic components relevant to classification that can be captured by a neural network? the quality of the extracted shapelets still needs to be separated from the external factors in every symbolic representation. by pointing out the limitations of this work, it is evident that derived studies are still needed. the results from the experiments allow us to believe that the quality of the extracted shapelets can be improved, making the proposed method more refined. furthermore, due to the number of parameters, the tests need to be replicated in order to confirm that the results persist in different contexts, such as other neural network architectures. the analysis of the model already trained in a convolutional neural network is also a necessary development. given the complexity of an already adjusted set of weights, unsupervised learning can be applied to segment the concepts in the learned model, such as identifying symbolic representations in the activated weights of a neural network. this would give a non-experimental, more analytical perspective on what happens during the transfer learn process. this can lead to the suppression of the training stage of the shapelet series using a neural network and creating a new model from scratch. another interesting future work would be comparison with [9], the authors search for a shapelet using neural network architecture. the implementation presented in this paper extracts shapelets using neural network adjustment directly. according to authors this hybrid approach have similar results than extracting shapelets using traditional extraction. likely mutual contributions are possible to get better results. notes 1.https://github.com/alexandrefelipemuller/timeseries_shapelet_transferlearning, last accessed 16 aug 2021. references bagnall, a., lines, j., hills, j., bostrom, a.: time-series classification with cote: the collective of transformation-based ensembles. ieee trans. knowl. data eng. 27(9), 2522–2535 (2015) article  google scholar  cui, z., chen, w., chen, y.: multi-scale convolutional neural networks for time series classification. corr abs/1603.06995 (2016). http://arxiv.org/abs/1603.06995 dau, h.a., et al.: the ucr time series classification archive (october 2018). https://www.cs.ucr.edu/~eamonn/time_series_data_2018/ fang, z., wang, p., wang, w.: efficient learning interpretable shapelets for accurate time series classification. in: 2018 ieee 34th international conference on data engineering (icde), pp. 497–508 (april 2018). https://doi.org/10.1109/icde.2018.00052 fawaz, h.i., forestier, g., weber, j., idoumghar, l., muller, p.: transfer learning for time series classification. corr abs/1811.01533 (2018). http://arxiv.org/abs/1811.01533 ismail fawaz, h., forestier, g., weber, j., idoumghar, l., muller, p.-a.: deep learning for time series classification: a review. data min. knowl. disc. 33(4), 917–963 (2019). https://doi.org/10.1007/s10618-019-00619-1 article  mathscinet  math  google scholar  gamboa, j.c.b.: deep learning for time-series analysis. arxiv preprint arxiv:1701.01887 (2017) grabocka, j., schilling, n., wistuba, m., schmidt-thieme, l.: learning time-series shapelets. in: proceedings of the 20th acm sigkdd international conference on knowledge discovery and data mining, kdd 14, pp. 392–401. acm (2014). https://doi.org/10.1145/2623330.2623613, http://doi.acm.org/10.1145/2623330.2623613 guijo-rubio, d., gutiérrez, p.a., tavenard, r., bagnall, a.: a hybrid approach to time series classification with shapelets. in: yin, h., camacho, d., tino, p., tallón-ballesteros, a.j., menezes, r., allmendinger, r. (eds.) ideal 2019. lncs, vol. 11871, pp. 137–144. springer, cham (2019). https://doi.org/10.1007/978-3-030-33607-3_16 chapter  google scholar  längkvist, m., karlsson, l., loutfi, a.: a review of unsupervised feature learning and deep learning for time-series modeling. pattern recogn. lett. 42, 11–24 (2014) article  google scholar  lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  google scholar  li, g., choi, b., bhowmick, s.s., wong, g.l.h., chun, k.p., li, s.: visualet: visualizing shapelets for time series classification. in: proceedings of the 29th acm international conference on information & knowledge management, pp. 3429–3432 (2020) google scholar  li, g., choi, b.k.k., xu, j., bhowmick, s.s., chun, k.p., wong, g.l.: efficient shapelet discovery for time series classification. ieee trans. knowl. eng. (2020) google scholar  lin, j., keogh, e., wei, l., lonardi, s.: experiencing sax: a novel symbolic representation of time series. data min. knowl. disc. 15(2), 107–144 (2007) article  mathscinet  google scholar  lines, j., bagnall, a.: time series classification with ensembles of elastic distance measures. data min. knowl. disc. 29(3), 565–592 (2014). https://doi.org/10.1007/s10618-014-0361-2 article  mathscinet  math  google scholar  lines, j., taylor, s., bagnall, a.: time series classification with hive-cote: the hierarchical vote collective of transformation-based ensembles. acm trans. knowl. discov. data 12(5), 52:1–52:35 (july 2018). https://doi.org/10.1145/3182382, http://doi.acm.org/10.1145/3182382 schäfer, p.: the boss is concerned with time series classification in the presence of noise. data min. knowl. disc. 29(6), 1505–1530 (2015) article  mathscinet  google scholar  srivastava, n., hinton, g., krizhevsky, a., sutskever, i., salakhutdinov, r.: dropout: a simple way to prevent neural networks from overfitting. j. mach. learn. res. 15(1), 1929–1958 (2014) mathscinet  math  google scholar  wang, z., yan, w., oates, t.: time series classification from scratch with deep neural networks: a strong baseline. corr abs/1611.06455 (2016). http://arxiv.org/abs/1611.06455 ye, l., keogh, e.: time series shapelets: a new primitive for data mining. in: proceedings of the 15th acm sigkdd international conference on knowledge discovery and data mining, kdd 2009, pp. 947–956. acm, new york (2009). https://doi.org/10.1145/1557019.1557122, http://doi.acm.org/10.1145/1557019.1557122 zhang, h., wang, p., fang, z., wang, z., wang, w.: elis++: a shapelet learning approach for accurate and efficient time series classification. world wide web 24(2), 511–539 (2021) article  google scholar  download references author information authors and affiliations informatics department, federal university of paraná, curitiba, brazil alexandre felipe muller de souza, mariane r. s. cassenote & fabiano silva authors alexandre felipe muller de souzaview author publications search author on:pubmed google scholar mariane r. s. cassenoteview author publications search author on:pubmed google scholar fabiano silvaview author publications search author on:pubmed google scholar corresponding author correspondence to alexandre felipe muller de souza . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de souza, a.f.m., cassenote, m.r.s., silva, f. (2021). transfer learning of shapelets for time series classification using convolutional neural network. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_23 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning shapelets time series transfer learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature aroeira: a curated corpus for the portuguese language with a large number of tokens | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper aroeira: a curated corpus for the portuguese language with a large number of tokens conference paper first online: 30 january 2025 pp 185–199 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) aroeira: a curated corpus for the portuguese language with a large number of tokens download book pdf download book epub thiago lira9, flávio cação9, cinthia souza9, joão valentini9, edson bollis9, otavio oliveira9, renato almeida9, marcio magalhães9, katia poloni9, andre oliveira9 & … lucas pellicer9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 554 accesses 1 altmetric abstract the emphasis on constructing extensive datasets for training large language models (llm) has recently increased, and current literature predominantly features datasets for high-resource languages such as english and chinese. however, there is a notable scarcity of high-quality corpora for the portuguese language. to address this limitation, we propose aroeira, a curated corpus explicitly designed for training large language models in the portuguese language, with a focus on the brazilian portuguese one. the aroeira corpus consists of 100 gb of texts from various internet platforms, processed through a comprehensive pipeline to ensure superior quality. the pipeline handles downloading, text extraction, language identification, application of quality and bias filters, and storage, all tailored for the portuguese language. the resulting corpus contains 35.3 million documents and over 15.1 billion tokens, surpassing the largest previously available corpus in this domain. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others sabiá: portuguese large language models chapter © 2023 historical portuguese corpora: a survey article open access 18 july 2024 mapping languages: the corpus of global language use article 08 april 2020 1 introduction most modern natural language processing (nlp) methodologies, including large language models (llms), rely on extensive text corpora for precise training and weight adaptation [18]. large-scale training corpora, or pre-training corpora, are fundamental for developing foundational models, which serve as the basis for numerous task-specific adaptations [7]. state-of-the-art llm training pipelines utilize various types of datasets: (i) pre-training corpora to acquire language structure, syntax, and semantics; (ii) instruction fine-tuning datasets to enhance the model’s capability to follow instructions; (iii) preference datasets to rank responses; and (iv) evaluation datasets to measure model performance [23]. recent research shows that the size and diversity of pre-training corpora significantly impact llm performance [14, 18]. most pre-training datasets are available in english and chinese, which are high-resource languages, while other languages have significantly fewer tokens [23]. although multilingual corpora can help mitigate data scarcity for low-resource languages, these datasets are often unbalanced, favoring high-resource languages [17]. this imbalance affects the performance of multilingual models for less-represented languages and models trained on multilingual corpora do not perform as well as those trained on monolingual corpora [34]. therefore, it is essential to train or fine-tune models in the target languages to capture linguistic nuances, structures, and domain-specific or cultural knowledge [28]. a direct implication of this scenario is the necessity of making high-quality plain-text corpora available to encourage research on specific model languages and the development of better-performing approaches. therefore, we introduce aroeira: a curated portuguese language-specific corpus composed of approximately 100 gb of text. the content was extracted from recent common crawlfootnote 1 (cc) web pages (up until 2023) and fully curated to remove web tags, ensuring quality and bias filtering. to the best of our knowledge, aroeira is the largest highly-curated portuguese corpus available to date. it has the potential to influence new instruction fine-tuning and evaluation dataset studies while guiding the development of preference datasets and large models. aroeira was created based on a double-pipeline inspired by [30]. the pipeline comprises two key steps: data quality management and content safety assurance. these steps ensure the size and quality necessary for a portuguese corpus to train safe llms effectively. as part of the content safety step, we investigated techniques for filtering hazardous content and mitigating biases in our corpora [16]. this effort resulted in a custom portuguese word dictionary, which encompasses offensive words, as well as terms, expressions, and phrases that include sexism, homophobia, ableism, racism, hate speech, and political, religious, and regional prejudice [13, 24, 26]. we highlight our main contributions: introduction of aroeira, a 100 gb portuguese corpus from diverse internet sources. our dataset surpasses the largest currently available corpus for training language models in portuguese in terms of size, quality, and representativeness. development of a parameterizable double-pipeline, which includes: downloading, extracting, language identification, quality filtering, and text storage in the data step; filtering sexual content, toxic data, and bias in the content safety step. creation of a dictionary to filter biased terms and mitigate social bias in the portuguese language. this paper is organized as follows. in sect. 2, we present related work in corpus extraction. sections 3 and 4, we describe the methodology for generating the corpus and the configuration of hyperparameters used in the quality filters, respectively. in sect. 5, we analyze the volumetry of aroeira in terms of year distribution, knowledge domains, document length, and other relevant results. finally, sect. 6 presents conclusions and future works. 2 related work the largest portuguese language corpus is brwac [35] which has approximately 25 gb of textual data distributed in 3.53 mi documents totaling 2.68 bi tokens. another large corpus is the carolina 1.2 ada [11], which contains approximately 2.11 mi documents and a total of 11 gb of textual data. when we compare this corpus with the corpora of other languages, the gaps become evident. gao et al. [14], for example, propose the pile, a corpus with 825 gb of texts in english. the corpus is derived from various data sources, including scientific articles, patent documents, and forums. an inspiring work for aroeira is colossal clean crawled corpus (c4) [30], a curated english-only corpus. c4 was created using common crawl (cc) data extracted in april 2019 and comprises approximately 750 gb of clean english text. similar to our approach, they apply filters to the raw data. cluecorpus2020 [36] was constructed using cleaned data from cc, resulting in a high-quality chinese pre-training corpus of 100 gb and 36 bi tokens. massivetext [29] is a collection of large english datasets created with data from different sources. massivetext contains 2.35 bi documents, equivalent to 10.5 tb of text. more recently, sabiá [28] applied a similar filtering methodology of massivetext to the portuguese section of clueweb dataset [27] and managed to retrieve a curated dataset. wudaocorpora [38] is a 3 tb chinese corpus with 1.08 tri of hanzi characters collected from 822 mi web pages. it is also worth mentioning that a current trend is the proposal of multilingual corpora. the c4corpus authors [17] present the construction of a 12 mi web page corpus containing more than 50 languages, including portuguese. english has a volume of 7.7 mi (64.2%) documents while portuguese has only 0.3 mi (2.5%). redpajama [10] is a large multilingual corpus, containing 100 bi text documents extracted from 84 cc snapshots. quality signals were applied to 30 billion documents, and deduplication was performed on 20 billion documents. it claims to have english (69.8%), deutch (9.2%), spanish (8.8%), french (7.8%), and italian (4.4%). as we can see, the corpora available in english and chinese have a massive data amount, easily surpassing corpora in portuguese and other languages. however, we know that the amount of internet information available in english and chinese is greater than in portuguese. another important aspect is the biases present in corpora and texts. language is a highly relevant avenue for manifesting social hierarchies, pre-established concepts, and standard forms of treatment [6]. various efforts are being made to evaluate data biases and how they impact the behavior of language models. the paper [24] analyzed 93 social groups that receive stigmatized treatment by nlp models. work [25] created stereoset to measure stereotypical treatment in certain ethnic groups, and paper [26] developed a benchmark dataset for measuring biases related to gender, race, age, sexual orientation, and others. these aspects are relevant in a context with strong normative motivations and the need to create responsible ai. many ways exist to mitigate text biases, such as data augmentation, content filtering, rebalancing, masking, and many others [13]. our work uses the concept studied by [16], where filtering sensitive content can result in models with more equitable treatment of different ethnic groups. the work specifically uses word co-occurrence in the filtering process. based on these past works, we can see that, in general, they focus on creating corpora for training language models for high-resource language tasks. thus, there is a necessity for creating a portuguese corpus since the amount of large brazilian portuguese models has drastically increased recently. we can cite bertimbau [33], ptt5 [9], bertaú [12], sabiá [4, 28], cabrita [22], and bode [15]. 3 aroeira in this section, we detail the steps of the corpus creation (double-pipeline) which is divided into two objectives, (i) collect (data pipeline) and (ii) ensure content safety (content safety pipeline). our whole pipeline contains nine steps: data collection and sampling, text extraction, language identification, deduplication, and quality filters in data pipeline, and sexual content filter, toxic data filter, bias filter, and categorization in content safety pipeline. figure 1 presents the entire workflow. fig. 1. double pipeline: data pipeline contains collection and sampling, text extraction, language identification, deduplication, and quality filters; and content safety pipeline encompasses sexual content filter, toxic data filter, bias filter, and categorization. full size image 3.1 data collection and sampling the data collection and sampling step involves downloading and extracting portuguese text from raw web archive (warc) in files. all data is sourced from common crawl (cc), which contains petabytes of scraped internet content from millions of web pages. we use the raw http files as the initial material, from which we extract and filter portuguese text as detailed in subsects. 3.2 and 3.3. cc organizes its datasets by date, each comprising thousands of individual shards of scraped content. we sampled shards from datasets ranging from 2015 to 2023, prioritizing more recent data. 3.2 text extraction this computational step involves using multiple cloud machines in parallel. these instances download and process raw files by extracting text from the html and filtering for portuguese text. the resulting data is then processed further using a single machine containing a key-value database for deduplication, as explained in subsect. 3.4. we opted to work with warc files to ensure better quality text, which includes handling raw html files and extracting texts ourselves. a python library called trafilatura [5] extracted only natural language text from the html files. metadata from webpages were saved for later use in the pipeline. 3.3 language identification roughly 0.2% of the pages in each shard are in portuguese. to filter these pages, it is necessary to detect the language they were written in automatically. as utilized by [14] and [1], we employ meta ai’s pre-trained fasttext model, which can detect 176 languages. for each downloaded page, the text is extracted using the trafilatura library [5] and the fasttextfootnote 2 [20] models used to determine the language. pages identified as portuguese with the highest probability by fasttext were selected. 3.4 deduplication the purpose of this step is to remove duplicated data from the corpus. to achieve this goal, we use two deduplication approaches. the first is a page-level approach, which identifies and removes pages with duplicate urls. the second is the document-level approach, which aims to remove significant overlapping documents. we employ the minhashlsh algorithm to calculate the jaccard similarity between documents, considering whether two document similarity exceeds 0.7 [29]. 3.5 quality filters a significant amount of data available on the internet may be insufficient in terms of quality for linguistic model formation. some examples include automatically generated text and text not written for human consumption [29]. this step aims to retain only pages written by humans for humans. to achieve this, we applied a series of ten quality filters: number of tokens: removes pages with fewer than a minimum number of tokens (in this work, we used the same tokenizer employed by gpt-2), as texts with low token counts are generally not informative; number of words: removes pages that do not attend specified upper and lower word limits, excluding punctuation and special characters; type token ratio (ttr): the ratio of unique words (types) to total words (tokens) [31]. ttr [37] serves as an indicator of text quality; symbols-word ratio: removes pages whose symbol word percentages exceed limits. any special character is considered a symbol; symbols at the beginning of the text: removes pages with an excessive number of symbols at the beginning of the text; stopwords: the presence of stopwords may indicate text coherence [29]; n-gram repetition: excessive repetition of sentences, paragraphs, or n-grams indicates low informational content [29]; number of sentences: removes pages with fewer than a specified number of sentences; lorem ipsum: removes pages containing the term “lorem ipsum” [30]; valid words: removes pages whose percentage of words found in a language dictionary is below a specified threshold. the thresholds for each filter are detailed in sect. 4. 3.6 sexual content filter to maintain the integrity of the corpus, a filter was applied to remove sexual content from the data. we verified whether a url was present in the université toulouse 1footnote 3 (ut1) blocklist for each page collected. as noted by [2], the ut1 blocklist is an extensive compilation of block lists frequently used for internet access control at schools. it was developed with the help of automated systems and human contributors and currently includes 3.7 million entries. for this work, we utilized a filtered version of this blocklist tailored for brazilian websites. it should be noted that this filter only excludes websites marked as adult content. for the remaining content, we randomly selected 25,000 examples and used a mistral 7b [19] model to extract pejorative sexual terms. these terms were then reviewed by humans and used as the final sexual content filter. 3.7 toxic data filter in this step, our objective is to identify and remove potential toxic content. toxicity definition is a rude, disrespectful, or unreasonable comment likely to incite an argument [13]. our filter comprises a dictionary of insults and pejorative terms. we evaluated exact matches of dictionary words with document terms and removed documents that exceeded a specified threshold percentage of words in the dictionary. the dictionary used for this filter was created by merging two lists of wordsfootnote 4\(^,\)footnote 5. we reinforce this filter does not aim to eliminate all the data containing toxic words, but rather to remove content with a significant toxic content proportion. 3.8 bias filter our work involves a step to identify and eliminate potential biases in the text based on contextual cues. we construct a dictionary of portuguese expressions used in biased contexts. when compiling this dictionary, it is crucial to consider the society dynamic and its relationship with linguistics [6]. we filtered the corpus by checking for exact matches between the dictionary words and those in the text [16]. various types of social biases were mapped, including gender, religion, race, sexist expressions, xenophobia, homophobia, ableism, fatphobia, and politics [24, 26]. 3.9 categorization this phase aims to categorize each page into a specific knowledge domain. we identified 27 knowledge domains, covering several categories and subjects, such as: blog posts, news articles, marketing, movies, social media, health, culinary recipes, books, scientific articles, politics, etc. the information regarding each domain can be leveraged to balance the data for specific tasks or augment datasets where knowledge is lacking. essentially, we map the urls of different pages to each knowledge scope and assign a topic to each url. a pt-pt text category has been introduced to differentiate between brazilian portuguese and european portuguese, as brazilian portuguese predominates in the dataset. the identified knowledge domains and their distribution are discussed in the subsect. 5.2. 4 qualitative configuration test we generated a 1 gb sample of texts and analyzed the distribution of metrics such as the number of tokens, word count, and ttr. we use this sample to find the best configuration for our double-pipeline to produce the resultant datasets. different value sets were empirically tested, and for each one, we checked the correctness of page removals and recorded the number of pages excluded after the filters were applied. analysis was carried out on this sample to find the optimal configuration. moreover, we perform qualitative analyses to verify the removal appropriateness. this qualitative test helped us define satisfactory values that we should filter to obtain content with good textual quality, i.e., a text that is diverse in words, fluid, with few repetitions, and with semantically relevant content. it is worth noting that we also evaluated the number of potentially toxic words and possible biases contained in the texts. table 1 shows the optimal configuration to obtain texts that exceed our minimum quality requirements. table 1. double-pipeline final configuration.full size table 5 results the created corpus was evaluated concerning five groups of requirements. the first requirement is the created corpus must be larger than the existing corpus for the portuguese language (see subsect. 5.1). the second requirement is the corpus must be diverse, i.e., containing data from different sources (see subsect. 5.2). the third requirement is the corpus covers the most recent to the least recent information (see subsect. 5.3). the fourth requirement is that the corpus presents high-quality text indicators (see subsect. 5.4). finally, the fifth requirement is that the corpus avoids introducing or increasing bias. due to the large corpus size, subsects. 5.4 and 5.5 utilize a 10% randomly generated sample to present the results. consequently, fig. 4 and table 3 were created based on this sample size. 5.1 corpus size the first requirement evaluated was the corpus size. we collected terabytes of data from different cc dumps. each dump has approximately 0.2% of texts in portuguese. it is worth noting that the documents may be of poor quality, contain inappropriate or biased content, and be duplicated due to the cc not filtering the data. therefore, a corpus cleaning step was necessary to ensure that the final corpus was composed only of non-duplicated documents to respect quality criteria. at the end of this process, we obtained a corpus of 100 gb. table 2 presents the created corpus statistics alongside other portuguese corpora. aroeira surpasses brwac [35] and carolina 1.2 ada [11] in size, document quantity, and token number. thus, our corpus is potentially a more diverse resource regarding texts and tokens than the available resources. table 2. corpora size comparison.full size table 5.2 knowledge domains the second requirement evaluated was the distribution of knowledge domains within the created corpus. a mapping of different urls to their respective knowledge domains was conducted. each base url was verified against a dictionary. when there were no matches, keywords were used to determine the document’s domain (subsect. 3.9). figure 2 illustrates the document distribution across these domains. the complexity of the corpus strongly correlates with downstream data performance [3]. therefore, an extensive representation of knowledge domains can contribute to the generation of more robust models, potentially improving in-context few-shot learning performance [32]. most documents could not be assigned to a specific domain and are marked as nr (not recognized). among those that were categorized, blog posts and news articles were the most frequent, although other categories such as institutional texts, e-commerce, and internet forums were also found. knowledge domains are essential for evaluating the quality of the data in the corpus and for filtering data used in the pre-training phase of domain-specific language models. 5.3 distribution of documents over time our third analysis is the distribution of the corpus documents over time. this temporal analysis is important to identify possible temporal biases such as outdated texts. our corpus presents a recent data distribution, which indicates more up-to-date texts. figure 3 illustrates that our data set comprises documents spanning up to 7 years, beginning in 2017. the bulk of the data is from 2017 to 2019, but a notable portion of recent data is from 2021, 2022, and 2023. this distribution meets the need for both recent and extensive data. as a result, models can train on up-to-date information and include recent and common terms used in portuguese. fig. 2. distribution of knowledge domains. full size image fig. 3. distribution of documents over time. full size image fig. 4. quality indicators. higher ttr and percentage of valid words indicate better corpus quality, while lower values for other indicators also signify better quality. full size image 5.4 quality indicators we used ttr value, symbol word percentage, stopword percentage, valid words, and toxic content as quality indicators. figure 4 shows the results obtained. we have two metrics that indicate the diversity of the content present in our corpus. the ttr indicates the variability of tokens in a sentence, and we aim for this value to be as high as possible, as it is a sign of texts composed of varied tokens with low word repetition. a ttr threshold of 0.5 is a quality parameter for the text. we obtained a distribution curve with the first quartile close to 0.5 ttr, a median of 0.57, and the third quartile above 0.65. thus, most of the data in our corpus reaches a satisfactory ttr value, which is a strong indicator of non-repetitive texts. another indicator of variability is the percentage of valid tokens or keywords. this indicator measures the frequency of contextually relevant words within the text, and we also want higher values for more diverse and fluid texts. we achieved an excellent distribution in this indicator, with most data above 0.7. this distribution value is another strong sign of lexically diverse texts. in contrast, the metrics for the percentage of symbols and the percentage of stopwords are indicators of less fluid texts, with many symbols interrupting the text or commonly used words that do not add semantic value to the documents (e.g., “to”, “for”, or “the”). our goal is to minimize these metrics as much as possible. we achieved our goal of reducing these values, obtaining distributions with low values, with the third quartile below 0.2 in both metrics. this result is a strong indication that the texts in the corpus are fluid. finally, we aim to minimize the percentage of potentially toxic words, decreasing to close to 0 (no potentially toxic words). the results show that we achieved this goal in almost all the texts in the corpus, except for some outliers that do not exceed 0.2% of toxic content. 5.5 bias we performed a word co-occurrence analysis to identify biases in our corpus. this technique has proven effective in demonstrating stereotypical treatment of a particular social group [13], which we do not desire. it is worth noting that this method is one of several mitigation approaches, and the corpus may still exhibit bias. we analyzed three groups of biases as shown in table 3, they are: (i) gender, (ii) religion, and (iii) race. we selected different words for each group and analyzed the context in which these words were inserted. we have chosen representative terms indicative of social groups and conducted an analysis focusing on those with the highest co-occurrence frequencies. we isolate the terms “man” and “woman” to evaluate gender bias. we selected “atheist”, “christian”, “buddhist”, “evangelical”, “jewish”, “muslim” and “umbandist” for religious bias. finally, we highlighted the words “white”, “black”, “asian” and “hispanic” for racial bias. the tables 3 respectively represent the results for gender, religious, and racial bias. table 3. related word co-occurrence.full size table no stereotypical treatment or hazardous behaviors described in the literature exist in the analyzed groups, such as associations with crime, income, and others [16]. furthermore, the words selected among the various social groups are very similar, which suggests a more equanimous treatment in our proposed corpus. 6 conclusion recent studies have shown significant improvements in the performance of language models trained on large corpora [8, 14]. consequently, the interest in creating large datasets has grown. most existing research focuses on high-resource languages like english and chinese, with considerable efforts made to develop multilingual corpora. however, there is a pressing need to develop large datasets for lower-resource languages. this work aims to address this gap by developing language models for lower-resource languages, specifically portuguese. we created the largest curated corpora for training or pre-training portuguese language models. to achieve this, we implemented a double-pipeline process to extract data while ensuring content safety. the process includes downloading, text extraction, language identification, deduplication, quality filtering, filtering for sexual content and toxicity, bias filtering, categorization, and storage. this effort involved collecting terabytes of data, resulting in a curated dataset of approximately 100 gb for aroeira’s construction. our results demonstrate that our corpus fulfills the requirements of corpus size, knowledge domains, document distribution over time, quality indicators, and bias mitigation. we conducted statistical analyses of the corpus to better comprehend the size of the collected documents in terms of the number of tokens, words, and sentences. additionally, we analyzed bias to recognize potential harms in the created corpus, a distinguishing factor in building models free from social biases. our findings conclude that the corpus created is of high quality and diversity, with minimal bias. we are eager to advance our work by training models with encoder architectures, such as bert models [21]. furthermore, we plan to pre-train language models with aroeira to obtain higher-quality models in portuguese. we intend to research existent instruction and evaluation datasets. finally, conducting comparative bias analyses on models trained with aroeira relative to other available corpora [13] will be highly valuable. notes 1.available at: https://commoncrawl.org/. 2.available at: https://fasttext.cc/. 3.https://dsi.ut-capitole.fr/blacklists/index_en.php. 4.https://github.com/ldnoobw/list-of-dirty-naughty-obscene-and-otherwise-bad-words/blob/master/pt. 5.https://github.com/dunossauro/chat-detox/blob/main/palavras.txt. references abadji, j., ortiz suárez, p.j., romary, l., sagot, b.: ungoliant: an optimized pipeline for the generation of a very large-scale multilingual web corpus (2021). https://doi.org/10.14618/ids-pub-10468 abadji, j., suarez, p.o., romary, l., sagot, b.: towards a cleaner document-oriented multilingual crawled corpus, january 2022 google scholar  agrawal, a., singh, s.: corpus complexity matters in pretraining language models, pp. 257–263, january 2023. https://doi.org/10.18653/v1/2023.sustainlp-1.20 almeida, t.s., abonizio, h., nogueira, r., pires, r.: sabi\(\backslash \)’a-2: a new generation of portuguese large language models. arxiv preprint arxiv:2403.09887 (2024) barbaresi, a.: trafilatura: a web scraping library and command-line tool for text discovery and extraction. in: proceedings of the joint conference of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: system demonstrations, pp. 122–131. association for computational linguistics (2021). https://aclanthology.org/2021.acl-demo.15 blodgett, s.l., barocas, s., daumé iii, h., wallach, h.: language (technology) is power: a critical survey of “bias” in nlp. in: proceedings of the 58th annual meeting of the association for computational linguistics. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.485 bommasani, r., et al.: on the opportunities and risks of foundation models. arxiv preprint arxiv:2108.07258 (2021) brown, t., et al.: language models are few-shot learners. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 1877–1901. curran associates, inc. (2020). https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-paper.pdf carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data. arxiv preprint arxiv:2008.09144 (2020) computer, t.: redpajama: an open source recipe to reproduce llama training dataset (2023). https://github.com/togethercomputer/redpajama-data crespo, m.c.r.m., et al.: carolina: a general corpus of contemporary brazilian portuguese with provenance, typology and versioning information, march 2023 google scholar  finardi, p., viegas, j.d., ferreira, g.t., mansano, a.f., caridá, v.f.: berta\(\backslash \)’u: ita\(\backslash \)’u bert for digital customer service. arxiv preprint arxiv:2101.12015 (2021) gallegos, i.o., et al.: bias and fairness in large language models: a survey. comput. linguist. 1–79 (2024). https://doi.org/10.1162/coli_a_00524 gao, l., et al.: the pile: an 800 gb dataset of diverse text for language modeling, december 2020 google scholar  garcia, g.l., et al.: introducing bode: a fine-tuned large language model for portuguese prompt-based task. arxiv preprint arxiv:2401.02909 (2024) garimella, a., mihalcea, r., amarnath, a.: demographic-aware language model fine-tuning as a bias mitigation technique. in: he, y., et al. (eds.) proceedings of the 2nd conference of the asia-pacific chapter of the association for computational linguistics and the 12th international joint conference on natural language processing (volume 2: short papers), pp. 311–319. association for computational linguistics, online only, november 2022. https://aclanthology.org/2022.aacl-short.38 habernal, i., zayed, o., gurevych, i.: c4corpus: multilingual web-size corpus with free license. in: calzolari, n., et al. (eds.) proceedings of the tenth international conference on language resources and evaluation (lrec 2016), pp. 914–922. european language resources association (elra), portorož, slovenia, may 2016. https://aclanthology.org/l16-1146 hoffmann, j., et al.: an empirical analysis of compute-optimal large language model training. in: koyejo, s., mohamed, s., agarwal, a., belgrave, d., cho, k., oh, a. (eds.) advances in neural information processing systems, vol. 35, pp. 30016–30030. curran associates, inc. (2022). https://proceedings.neurips.cc/paper_files/paper/2022/file/c1e2faff6f588870935f114ebe04a3e5-paper-conference.pdf jiang, a.q., et al.: mistral 7b. arxiv preprint arxiv:2310.06825 (2023) joulin, a., grave, e., bojanowski, p., mikolov, t.: bag of tricks for efficient text classification. in: proceedings of the 15th conference of the european chapter of the association for computational linguistics: volume 2, short papers, pp. 427–431. association for computational linguistics, april 2017 google scholar  kenton, j.d.m.w.c., toutanova, l.k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of naacl-hlt, vol. 1, p. 2 (2019) google scholar  larcher, c., piau, m., finardi, p., gengo, p., esposito, p., caridá, v.: cabrita: closing the gap for foreign languages. arxiv preprint arxiv:2308.11878 (2023) liu, y., cao, j., liu, c., ding, k., jin, l.: datasets for large language models: a comprehensive survey. arxiv preprint arxiv:2402.18041 (2024) mei, k., fereidooni, s., caliskan, a.: bias against 93 stigmatized groups in masked language models and downstream sentiment classification tasks. in: 2023 acm conference on fairness, accountability, and transparency, facct 2023. acm, june 2023. https://doi.org/10.1145/3593013.3594109 nadeem, m., bethke, a., reddy, s.: stereoset: measuring stereotypical bias in pretrained language models. in: proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing (volume 1: long papers). association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.acl-long.416 nangia, n., vania, c., bhalerao, r., bowman, s.r.: crows-pairs: a challenge dataset for measuring social biases in masked language models. in: proceedings of the 2020 conference on empirical methods in natural language processing (emnlp). association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.emnlp-main.154 overwijk, a., xiong, c., callan, j.: clueweb22: 10 billion web documents with rich information. in: proceedings of the 45th international acm sigir conference on research and development in information retrieval, pp. 3360–3362 (2022) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models. in: brazilian conference on intelligent systems, pp. 226–240 (2023) google scholar  rae, j.w., et al.: scaling language models: methods, analysis & insights from training gopher, december 2021 google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21(140), 1–67 (2020). http://jmlr.org/papers/v21/20-074.html richards, b.: type/token ratios: what do they really tell us? j. child lang. 14(2), 201–209 (1987). https://doi.org/10.1017/s0305000900012885 article  mathscinet  math  google scholar  shin, s., et al.: on the effect of pretraining corpora on in-context learning by a large-scale language model (2022) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  virtanen, a., et al.: multilingual is not enough: bert for finnish, december 2019 google scholar  wagner filho, j.a., et al.: the brwac corpus: a new open resource for brazilian portuguese. in: calzolari, n., et al. (eds.) proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). european language resources association (elra), miyazaki, japan, may 2018. https://aclanthology.org/l18-1686 xu, l., zhang, x., dong, q.: cluecorpus2020: a large-scale chinese corpus for pre-training language model. arxiv preprint arxiv:2003.01355 (2020) youmans, g.: measuring lexical style and competence: the type-token vocabulary curve. style 24(4), 584–599 (1990). http://www.jstor.org/stable/42946163 yuan, s., et al.: wudaocorpora: a super large-scale chinese corpora for pre-training language models. ai open 2, 65–68 (2021) article  math  google scholar  download references acknowledgements we thank instituto de ciência e tecnologia itaú-unibanco (icti) and itaú-unibanco sa for the technical support, resources, and financial aid in the development of the aroeira corpus. it’s also noteworthy the fact that chatgpt (openai) was employed in the writing process, contributing to thorough grammatical and semantic reviews. author information authors and affiliations instituto de ciência e tecnologia itaú-unibanco (icti), são paulo, brazil thiago lira, flávio cação, cinthia souza, joão valentini, edson bollis, otavio oliveira, renato almeida, marcio magalhães, katia poloni, andre oliveira & lucas pellicer authors thiago liraview author publications search author on:pubmed google scholar flávio caçãoview author publications search author on:pubmed google scholar cinthia souzaview author publications search author on:pubmed google scholar joão valentiniview author publications search author on:pubmed google scholar edson bollisview author publications search author on:pubmed google scholar otavio oliveiraview author publications search author on:pubmed google scholar renato almeidaview author publications search author on:pubmed google scholar marcio magalhãesview author publications search author on:pubmed google scholar katia poloniview author publications search author on:pubmed google scholar andre oliveiraview author publications search author on:pubmed google scholar lucas pellicerview author publications search author on:pubmed google scholar corresponding author correspondence to lucas pellicer . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations data availability our aroeira corpus is available for download in the hugging face repository: https://huggingface.co/datasets/itau-unibanco/aroeira and is under the cc-by-nc 4.0 license. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper lira, t. et al. (2025). aroeira: a curated corpus for the portuguese language with a large number of tokens. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_13 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature deep learning approach to temporal dimensionality reduction of volumetric computed tomography | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deep learning approach to temporal dimensionality reduction of volumetric computed tomography conference paper first online: 30 january 2025 pp 296–309 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) deep learning approach to temporal dimensionality reduction of volumetric computed tomography download book pdf download book epub lucas almeida da silva  orcid: orcid.org/0000-0001-7511-29109, eulanda miranda dos santos  orcid: orcid.org/0000-0002-5671-581x9 & rafael giusti  orcid: orcid.org/0000-0002-1288-71269  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 488 accesses abstract a common approach for analyzing medical images on volumetric data is to employ deep 2d convolutional neural networks (2d cnn) on each individual slice of the volume. this is largely attributed to the challenges posed by the nature of three-dimensional data: variable volume size, high gpu and ram requirements, costly parameter optimization, etc. however, handling the individual slices independently in 2d cnns deliberately discards the temporal information that is contained within the depth of the volumes, which tends to result in poor performance. in order to maintain temporal information, current solutions reduce the temporal dimensionality of the data using non-adaptive sampling processes, such as taking slices at given intervals or by means of interpolation. however, although this allows to keep some temporal information, it may discard important slices. in this paper, we propose a method based on gradcam to select meaningful slices in computed tomography volumes by evaluating the activation map to reduce temporal data dimensionality. extensive experiments demonstrate the effectiveness of our method when compared to the current state of the art. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others enhancement and evaluation for deep learning-based classification of volumetric neuroimaging with 3d-to-2d knowledge distillation article open access 28 november 2024 flexible conditional image generation of missing data with learned mental maps chapter © 2019 domain-specific cues improve robustness of deep learning-based segmentation of ct volumes article open access 01 july 2020 1 introduction x-ray is a form of electromagnetic radiation, like visible light. it is less energetic than gamma rays and more energetic than ultraviolet light. while the human body is mostly opaque to visible light, x-rays easily pass through soft tissue, such as organs and muscles, but not as easily through hard tissue, such as bones and teeth. consequently, x-ray imaging is well-suited for examining skeletal structures, but not so much for soft tissue, as it is the case of the brain. computed tomography (ct) addresses this limitation, allowing thorough observation of internal structures, including the brain, the lungs, and other organs. the technique is non-invasive and provides good cross-sectional visualization. additionally, ct exams are usually quick, with a single imaging session often completed in less than a few minutes. ct scans are obtained from multiple shots at different angles, and with complex geometric transformations they produce volumes of data called slices. a set of slices is comparable to a collection of images that provide information on body sections. in a sense, ct scans are considered a type of three-dimensional data, as each slice can be transformed into a gray-scale image in a two-dimensional space and a full ct exam can be transformed into a sequence of images. they are used by medical professionals to diagnose a multitude of diseases and conditions, and may provide a rich source of information to artificial intelligence (ai) methods. however, many ai engines are unable to make the best use of ct data. the reason is the often large volume of data, which renders the computational cost involved in training and inference infeasible for many information systems [1]. instead, researchers often resort to two-dimensional approaches, which naturally disregard the sequential aspect of the volumetric data. while this reduces the computational cost and allows models to be trained and employed without the requirement of supercomputers, performance is usually also reduced. furthermore, even when approaches designed to cope with three-dimensional (3d) data are used, such as 3d convolutional neural networks (cnn3d), performance tends to be lower than expected, especially when compared to two-dimensional approaches. this is primarily due to the difficulty of adjusting hyper-parameters, which require very time-consuming validation steps, and the necessity of reducing the resolution of the scans in favor of temporal data. state-of-the-art approaches attempt to reduce the data’s temporal dimensionality while preserving image resolution as much as possible. however, most techniques rely on non-adaptive sampling processes, such as taking slices at given intervals or interpolating through slices. as a result, slices that are irrelevant to the application may be selected while important slices are discarded. in this work, we address the problem of selecting slices that are more relevant for machine learning models, while simultaneously discarding slices containing less information about a given decision problem (e.g., detecting signs of a disease in volumetric ct). our approach is based in the grad-cam technique [25]. ideally, this should result in accurate models with shorter training times and lower hardware demands. as a case study, two datasets addressing two different decision problems are investigated in this paper: a set of lung cts to determine if the patient is infected with the covid-19 virus; and a dataset for intracranial hemorrhage detection. the highlights of our paper are: we employ gradient-weighted class activation mapping (grad-cam) to eliminate the manual task of selecting meaningful slices. we use depthwise convolutions [7] and grad-cam on volumetric ct data to preserve temporal order and facilitate relevant slice selection through adaptation of cnn [27]. our proposed method reduces the temporal dimensionality of volumetric ct data, providing an advantage when a low amount of processing power is available. the gss method consistently shows the best results compared to other slice selection methods, both in terms of auc and f1 score. this is true for all configurations and for both deep learning models (c3d and 3dcnn-c) investigated in this paper. therefore, the gss method is the most promising for the task of selecting meaningful slices on ct volumes. 2 related work deep learning has been utilized in various medical domains [3, 5, 19, 24, 31]. many of these studies concentrate on techniques developed for two-dimensional (2d) data, such as images. since volumetric cts are inherently 3d, one common approach is to analyze each ct slice individually using algorithms designed for 2d data, primarily state-of-the-art 2d cnns [10, 15, 18, 23, 26, 28]. however, there is evidence suggesting that utilizing 3d data from ct scans leads to improved results [2, 14, 16, 17, 21], as it maintains the depth properties of the ct scans. several challenges arise when processing ct scans with volumetric data. generally, data complexity increases exponentially with each added dimension [20]. in machine learning, this often results in substantially larger demands for memory, computation, and training data [12] because the complexity of the model must grow to more properly represent the complexity of the date. in the case of deep neural networks (dnn), learning from 3d data typically demands more layers and neurons than learning from 2d data. in other words, working with 3d data is significantly more computationally expensive, and one might not have enough resources to train with 3d data. to provide some insight into this computational cost, we summarize the requirements for three machine learning methods employed in this study when classifying examples of the mosmed data set. the mosmed data set contains volumetric ct scan data that we resize in both width (image resolution) and depth (number of slices per instance). our analysis is presented in table 1. in all instances, the ct scan slice numbers were standardized to either 12 or 30, and the resolution of each scan remained at 512\(\,\times \,\)512 pixels or was scaled down to 224\(\,\times \,\)244 pixels. we measured the model size in memory, the number of trainable parameters, the number of floating-point operations (flops), and the gpu memory needed to perform tasks with each model. the number of flops directly influences the model’s execution time, and the required gpu memory determines whether the model can be used with the available computational resources. for example, squeezenet is a compact neural network designed for low-end devices, requiring minimal gpu memory. conversely, the traditional 3d cnn architecture demands nearly 100 gigabytes of gpu memory to train and test 30 slices at full resolution. at the time of writing this article, this memory requirement exceeds the capacity of all consumer-grade devices on the market. table 1. computational cost of three 3d cnn models when classifying instances from the mosmed data set. the input width and depth correspond, respectitvely, to the width of the input, and the number of slices of each instance. the total cost was measured empirically in terms of memory, number of trainable parameters, number of floating point operations (flops), and the amount of gpu memory required to perform the classification.full size table typical ct exam equipments can produce volumes ranging from 2 to 640 slices. selecting the most meaningful slices is the optimal approach for analyzing this data without discarding temporal information. even if one possesses sufficient computational power to process all 640 slices, reducing data complexity is usually preferable, as less complex models tend to generalize better and decrease the risk of overfitting [4]. slice selection can be achieved using adaptive or non-adaptive approaches. adaptive approaches examine the slice content to determine which ones contain the most relevant information, while non-adaptive methods select subsets of slices regardless of their content. there are five primary techniques found in the literature, described in this section. the first three are examples of non-adaptive methods, the fourth is an adaptive approach, while the remaining one is also non-adaptive. discard by slice similarity (dss) is a category of methods where a similarity-based algorithm (e.g., structural similarity index measure (ssim), mean squared error (mse), euclidean distance) compares each slice to its successor. based on a defined threshold, the most similar slices are eliminated [29]. this approach aims to reduce redundancy in the data while preserving unique and informative slices. subset slice selection (sss) is a method proposed in [34]. in this approach, slices are sampled from three specific positions of a volumetric ct: start, middle, and end. to achieve this, the volume is first divided into three equal parts. then, a desired number of slices are extracted from each part. this method aims to capture the representative information from different regions of the ct volume. even slice selection (ess), also known as uniform slice selection, is a technique commonly used in video processing. in ess, a “spacing factor” is calculated to enable the selection of equidistant slices [33]. given a volume with d slices and a desired number of \(k << d\) slices, ess works by dividing the volume into disjoint subsets of approximately \(\frac{d}{k}\) slices. then, only the first slice of each subset is retained. in contrast to sss, this technique mitigates semantic losses of temporal data. algorithm 1 summarizes the ess method. slice selection by object detection (ssod) consists of methods in which an algorithm scans each slice and determines whether an object of interest is present. for instance, an ai model may be employed on ct scans to detect which slices display segments of organs such as lungs, brain, or other targeted organs [8]. the slices displaying the desired organ are then selected, while the others are discarded. while this method adapts to the data, it overlooks the inherent temporal relationships between slices. often, it discards the initial and final subsets of slices, as the largest portion of the target organ is typically located in the middle of the volume. as previously mentioned, ssod is an adaptive method, whereas dss, sss, and ess are non-adaptive. in some applications, employing non-adaptive strategies to discard slices may lead to loss of essential information (e.g., a critical segment of an organ may not be present in any of the selected slices). spline interpolation zoom (siz) is a non-adaptive strategy aimed at reducing these losses. in siz, the temporal dimension is reduced by interpolating all slices. the volume is enlarged or compressed by replicating the closest pixel of each slice along the depth/z-axis. although each resulting slice may be less precisely “located” in time, this technique may be advantageous for various applications since it does not produce “gaps” in the temporal axis [13, 32]. as summarized in algorithm 2, given k as the desired number of slices and d as the total number of slices of a volume, zoom is performed along the z-axis by a factor of \(\frac{1}{d/k}\) using interpolation by splines [9]. our work primarily focuses on comparing non-adaptive techniques against our proposed method because, although our method is adaptive in nature, many of the available adaptive methods, including ssod, do not leverage the temporal relationship present in the slices when selecting. this omission may result in a loss of critical information. furthermore, most adaptive slice selection methods in the medical domain currently focus on object detection rather than preserving temporal information. we believe our adaptive approach that emphasizes temporal relationships offers a unique perspective in the slice selection domain, warranting its comparison with more traditional non-adaptive methods. 3 materials and methods in this section we consider the definitions provided by tran et al. [30], a primary reference for 3d learning, to help us to explain our proposed method. in that work, tran et al. provide empirical evidence that 3d convolutions are effective feature extractors when modelling appearance and motion or depth simultaneously. this makes them well-suited for learning sequence and spatiotemporal data. for instance, in 3d cnns, even the output of each layer is a volume. in 2d convolution, the kernel is a 3-dimensional matrix and both the input layer and the filters have the same depth (channel number equals their kernel number). however, the 3d filter only moves in two directions, as in over the height and the width of the image. in other words, it deals with only the spatial dimensions of the input. therefore, even when using 3d filters, the output is a 2d array. conversely, 3d convolution not only involves a 3d filter, but the filter can move in all three directions—height, width and time/sequence. at each position, multiplication and elementary addition provide a number. as the filter slides through the 3d space, the output numbers are also arranged in a 3d space, i.e. the output is 3d data. cnns exploit the spatially-local correlation by enforcing a local connectivity pattern between neurons of adjacent layers. such 3d convolutions also take temporal features into account, typically through a sliding window, which is a filter with trainable weights over the input, and producing as output a weighted sum of weights and input. the weighted sum is the feature space used as the input for the next layers. in this work we employ depthwise convolution, which is a type of convolution where a single convolutional filter is applied for each input channel. in regular 2d convolution performed over multiple input channels, the filter is as deep as the input and mixes channels to generate each element of the output feature map. in contrast, depthwise convolution keeps each channel separated. in our case, it maintains each slice separated, preserving the temporal relationship at different timestamps. we summarize the steps involved in this process bellow: 1. the input tensor of 3 dimensions (volumetric ct, in our case) is divided into separate channels/slices. 2. each slice is convolved with its respective filter. 3. the obtained convolved output are stacked together to provide the output as an entire 3d tensor. figure 1 exemplifies this process. at the top part of the image an instance containing three slices (a volumetric ct) is shown. each slice is trated separated and convolution is only performed within each slice. it is necessary to maintain the depth size until the final convolution so that the feature maps of each slice is preserved to allow the extraction of grad-cam heatmap, which is performed in our proposed method, as detailed in next sections. fig. 1. illustration of the depthwise convolution used in the proposed method on a volume with 3 slices. full size image 3.1 inference method the proposed method consists of a 3dcnn designed to capture the activation map of a volumetric ct. the objective is to obtain as output a 3d matrix representing the regions of interest in each slice of the volume. this is achieved by adapting the grad-cam technique, which uses the gradients obtained when evaluating the class of interest by flowing to the last convolutional layer in order to map the activation regions of an image. grad-cam requires no re-training and is broadly applicable to any cnn-based architectures. additionally, the generated heatmaps may provide explanations, making the predictions understandable to users, consequently helping the users to trust the predictions made by the system, especially in medical applications. as previously mentioned, we use a model that preserves the depth dimension up to the last convolutional layer, performing depthwise convolutions on the volume so as to maintain the order of each slice. in this way, a feature map is created for each slice. however, since ct are temporal high-dimensional data, a critical point here is the viability of training a 3dcnn model using all the slices of the volume, depending on the chosen hardware. therefore, in order to allow this training, we focus on providing a condition of lower computational cost to train the model by applying image compression techniques to identify macro-regions of interest. thus, the resizing of the width and height dimensions is a fundamental aspect. the following steps were performed to obtain our proposed grad-cam slice selection method (gss), in which grad-cam is applied to obtain the meaningful slices: 1. normalize the volumetric data with one of the techniques described in sect. 2. siz (sect. 2) was used in this work. 2. train a cnn3d in the classification problem represented by the volumetric data preserving the depth dimension, e.g. the binary classification of data as covid-19 or not. 3. remove the layers stacked after the last convolutional layer after finishing the network training, since they are no longer needed. 4. extract the representative matrix of heatmaps. these steps are also summarized in algorithm 3 and are further visualized in fig. 2 that shows that the last conv layer will be analyzed for the heatmap acquisition. fig. 2. inference pipeline of the gss model that will be used for heatmap acquisition on the last conv layer. full size image 4 experimental results in this section, we present the experimental results obtained using two 3dcnn architectures: c3d and 3dcnn-c. the experiments were conducted using the canonic houldout partitions established as training and test splits of the two datasets investigated in order to provide state-of-the-art comparison. the datasets employed are mosmeddata [22] and cq500 [6], detailed below. one notable aspect of our experimental design is the conscious decision to not employ any data augmentation techniques. data augmentation is often employed to artificially expand the dataset size by introducing variations such as rotation, flipping, and scaling. while this can improve the model’s generalization ability, it also introduces an additional layer of complexity that could potentially skew results or make direct comparisons with other methodologies more challenging. in our study, we aim to evaluate the efficacy of the deep learning architectures in recognizing patterns in unaltered, real-world medical imaging data. by excluding data augmentation, we seek to establish a baseline performance metric that reflects the model capabilities to generalize from the raw, original data, thereby ensuring that any observed performance differences are attributable solely to the architectures themselves, rather than to external manipulations of the data. mosmeddata contains anonymized human lung ct scans with covid-19 related findings, as well as those without such findings. the ct scans were obtained between 1st march 2020 and 25th april 2020, provided by medical hospitals in moscow, russia. cq500 is a dataset that contains 491 scans with 193,317 slices, provided as anonymized dicoms by the centre for advanced research in imaging, neurosciences, and genomics (caring), new delhi, india. the reads were performed by three radiologists with 8, 12, and 20 years of experience in cranial ct interpretation, respectively. the 3d data from these datasets are in nifti format, which are volumetric (3d) images. after pre-processing, all images were set to a size of 512 with variable depth. due to the nature of ct datasets, the images were imported as color grayscale images, demanding modifications according to each network’s input for data compatibility. in addition, the datasets were partitioned following the holdout procedure into 70-15-15 for training, test and validations subsets, respectively. the two 3dcnn models investigated were trained end-to-end on both datasets for 100 epochs with early stopping method so as to better tackle potential overfitting. moreover, overfitting was also addressed by incorporating dropout and l1/l2 regularization techniques. the adam optimizer was used in both architectures with a learning rate of \(10^{-4}\), determined experimentally. weights were initialized using glorot normal initialization [11], with convolution layers featuring the activation function rectified linear activation function (relu) and the final layer using the activation function sigmoid for the binary problem. training was performed in an environment with a tesla v100-sxm2 video card with 16 gb memory. as previously mentioned, all four non-adaptive slice selection methods described in sect. 2 are used in this work as baselines. the experiments were divided into two comparison scenarios according to the number of selected slices, precisely 30 and 12 slices were selected with each method and compared against the proposed gss. table 2 presents the results of this comparison on the two distinct datasets, mosmed and cq500, representing binary classification problems. besides the number of selected frames (30 and 12), the four investigated slice selection methods (sss, ess, siz, and gss) are also compared considering two different image resolutions: 512\(\,\times \,\)512 and 224\(\,\times \,\)224. finally, two 3dcnn models are used: c3d and 3dcnn-c adapted from [27]. the evaluation metrics used are auc (area under the roc curve) and f1 score. analyzing the results, the following conclusions are observed: gss consistently presents the best results compared to other methods, both in terms of auc and f1 score. this is true across all configurations and for both deep learning models (c3d and 3dcnn-c). therefore, gss is the most promising method for selecting the most relevant slices from tomography volumes. the baseline siz outperforms sss and ess in general, but still cannot surpass the results from gss. the results show that selecting 30 frames generally performs better than selecting 12 frames. this indicates that having more frames available for the model can lead to better model performance. the 512\(\,\times \,\)512 resolution generally produces better results compared to the 224\(\,\times \,\)224 resolution, suggesting that a higher resolution provides more valuable information for the model. the 3dcnn-c model adapted from [27] tends to achieve better results than the c3d model across all configurations and datasets. selecting slices without decreasing accuracy is a significant advancement towards using ai as an assistant in clinical management with lower computational power requirements. therefore, the development of a clinical prognostic model based on our ai system, utilizing ct parameters and clinical data, marks an essential step forward in using ai to assist clinical management in this kind of scenario. table 2. results comparing the proposed method to three non-adaptive slice selection methods when varying the number of selected slices (12 or 30) on two different ct datasets and using two different 3dcnn models.full size table 5 conclusions in this paper we propose gss (grad-cam slice selection), a technique to deal with the problem of classifying complex 3d spatiotemporal data in the form of volumetric computer tomography (ct) imagery. by selecting slices from each ct in a way that takes into account the classification problem at hand, we are able to dynamically select the slices that provide more information, and reduce the computational cost required to train and apply a model. we compared different deep learning techniques under the same conditions when applied to classification problems with two well-known data sets. by applying the proposed gss to each deep learning model before training and inference, we achieved accuracies on par with or surpassing the state-of-the-art and, in most cases, better than other agnostic sampling techniques. this was especially true for edge cases requiring a greater number of slices to be selected. our findings emphatically underline the superiority of the gss approach combined with 30 frames at 512\(\,\times \,\)512 resolution, leveraging the cnn model cited in [27]. as anticipated, this high-resolution configuration with an optimal number of frames yielded the most promising results in the domain of tomography volume anomaly detection. however, the strategic selection between resolution and the number of frames remains pivotal. for detecting nuanced or minute pathologies, it is imperative to lean towards higher spatial resolutions. conversely, for challenges deeply rooted in the three-dimensional intricacies, a richer frame selection is recommended. these findings support the hypothesis that using grad-cam in deep learning models for learning simple and macro-relevant features in a ct volume dataset is effective and warrants further investigation. subsequently, transferring this knowledge to another model to learn complex patterns proves fruitful. references ahmed, e., et al.: a survey on deep learning advances on different 3d data representations. arxiv preprint arxiv:1808.01462 (2018) alebiosu, d.o., dharmaratne, a., lim, c.h.: improving tuberculosis severity assessment in computed tomography images using novel davou-net segmentation and deep learning framework. expert syst. appl. 213, 119287 (2023) article  google scholar  becker, a.s., et al.: detection of tuberculosis patterns in digital photographs of chest x-ray images using deep learning: feasibility study. int. j. tuberculosis lung disease 22(3), 328–335 (2018) google scholar  bejani, m.m., ghatee, m.: a systematic review on overfitting control in shallow and deep neural networks. artif. intell. rev. 54(8), 6391–6438 (2021). https://doi.org/10.1007/s10462-021-09975-1 article  math  google scholar  chen, w.w., et al.: a deep learning approach to classify fabry cardiomyopathy from hypertrophic cardiomyopathy using cine imaging on cardiac magnetic resonance. int. j. biomed. imaging 2024(1), 6114826 (2024) google scholar  chilamkurthy, s., et al.: development and validation of deep learning algorithms for detection of critical findings in head ct scans. arxiv preprint arxiv:1803.05854 (2018) chollet, f.: xception: deep learning with depthwise separable convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1251–1258 (2017) google scholar  da cruz, l.b., et al.: kidney segmentation from computed tomography images using deep neural network. comput. biol. med. 123, 103906 (2020) article  google scholar  de boor, c.: bicubic spline interpolation. j. math. phys. 41(1–4), 212–218 (1962) article  mathscinet  math  google scholar  gao, x.w., hui, r., tian, z.: classification of ct brain images based on deep learning networks. comput. methods programs biomed. 138, 49–56 (2017) article  math  google scholar  glorot, x., bengio, y.: understanding the difficulty of training deep feedforward neural networks. in: proceedings of the thirteenth international conference on artificial intelligence and statistics, pp. 249–256. jmlr workshop and conference proceedings (2010) google scholar  goodfellow, i., bengio, y., courville, a.: deep learning. mit press (2016) google scholar  gordaliza, p.m., vaquero, j.j., sharpe, s., gleeson, f., munoz-barrutia, a.: a multi-task self-normalizing 3d-cnn to infer tuberculosis radiological manifestations. arxiv preprint arxiv:1907.12331 (2019) grewal, m., srivastava, m.m., kumar, p., varadarajan, s.: radnet: radiologist level accuracy using deep learning for hemorrhage detection in ct scans. in: 2018 ieee 15th international symposium on biomedical imaging (isbi 2018), pp. 281–284. ieee (2018) google scholar  hamadi, a., cheikh, n.b., zouatine, y., menad, s.m.b., djebbara, m.r.: imageclef 2019: deep learning for tuberculosis ct image analysis. in: clef (working notes) (2019) google scholar  huang, x., shan, j., vaidya, v.: lung nodule detection in ct using 3d convolutional neural networks. in: 2017 ieee 14th international symposium on biomedical imaging (isbi 2017), pp. 379–383. ieee (2017) google scholar  ji, s., xu, w., yang, m., yu, k.: 3d convolutional neural networks for human action recognition. ieee trans. pattern anal. mach. intell. 35(1), 221–231 (2012) article  math  google scholar  kavitha, s., nandhinee, p., harshana, s., s, j.s., harrinei, k.: imageclef 2019: a 2d convolutional neural network approach for severity scoring of lung tuberculosis using ct images. in: clef (working notes) (2019) google scholar  lakhani, p., sundaram, b.: deep learning at chest radiography: automated classification of pulmonary tuberculosis by using convolutional neural networks. radiology 284(2), 574–582 (2017) article  math  google scholar  lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  math  google scholar  li, b., zhang, t., xia, t.: vehicle detection from 3d lidar using fully convolutional network. arxiv preprint arxiv:1608.07916 (2016) morozov, s.p., et al.: mosmeddata: data set of 1110 chest ct scans performed during the covid-19 epidemic. digit. diagn. 1(1), 49–59 (2020) article  math  google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: international conference on medical image computing and computer-assisted intervention, pp. 234–241. springer (2015). https://doi.org/10.1007/978-3-319-24574-4_28 santosh, k., allu, s., rajaraman, s., antani, s.: advances in deep learning for tuberculosis screening using chest x-rays: the last 5 years review. j. med. syst. 46(11), 82 (2022) article  google scholar  selvaraju, r.r., cogswell, m., das, a., vedantam, r., parikh, d., batra, d.: grad-cam: visual explanations from deep networks via gradient-based localization. in: proceedings of the ieee international conference on computer vision, pp. 618–626 (2017) google scholar  shah, a.a., malik, h.a.m., muhammad, a., alourani, a., butt, z.a.: deep learning ensemble 2d cnn approach towards the detection of lung cancer. sci. rep. 13(1), 2987 (2023) article  math  google scholar  da silva, l.a., et al.: spatio-temporal deep learning-based methods for defect detection: an industrial application study case. appl. sci. 11(22), 10861 (2021) article  math  google scholar  sultana, a., et al.: a real time method for distinguishing covid-19 utilizing 2d-cnn and transfer learning. sensors 23(9), 4458 (2023) article  math  google scholar  thasneem, a.h., sathik, m.m., mehaboobathunnisa, r.: a fast segmentation and efficient slice reconstruction technique for head ct images. j. intell. syst. 28(4), 533–547 (2019) google scholar  tran, d., bourdev, l., fergus, r., torresani, l., paluri, m.: learning spatiotemporal features with 3d convolutional networks. in: proceedings of the ieee international conference on computer vision, pp. 4489–4497 (2015) google scholar  wajgi, r., et al.: optimized tuberculosis classification system for chest x-ray images: fusing hyperparameter tuning with transfer learning approaches. engineering reports, e12906 (2024) google scholar  yang, j., et al.: reinventing 2d convolutions for 3d images. ieee j. biomedical health inform. (2021) google scholar  zhu, h., liu, x., mao, x., wong, t.t.: real-time deep video deinterlacing. arxiv preprint arxiv:1708.00187 (2017) zunair, h., rahman, a., mohammed, n., cohen, j.p.: uniformizing techniques to process ct scans with 3d cnns for tuberculosis prediction. in: rekik, i., adeli, e., park, s.h., valdés hernández, m.c. (eds.) predictive intelligence in medicine: third international workshop, prime 2020, held in conjunction with miccai 2020, lima, peru, october 8, 2020, proceedings, pp. 156–168. springer international publishing, cham (2020). https://doi.org/10.1007/978-3-030-59354-4_15 chapter  google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes-proex) finance code 001. this work was partially supported by amazonas state research support foundation fapeam through the project pdpg/capes. author information authors and affiliations instituto de computação, universidade federal do amazonas, manaus, brazil lucas almeida da silva, eulanda miranda dos santos & rafael giusti authors lucas almeida da silvaview author publications search author on:pubmed google scholar eulanda miranda dos santosview author publications search author on:pubmed google scholar rafael giustiview author publications search author on:pubmed google scholar corresponding author correspondence to rafael giusti . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, l.a.d., santos, e.m.d., giusti, r. (2025). deep learning approach to temporal dimensionality reduction of volumetric computed tomography. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_21 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords pattern recognition feature selection ct classification ct slice selection publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature aggressive language detection using vgcn-bert for spanish texts | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper aggressive language detection using vgcn-bert for spanish texts conference paper first online: 28 november 2021 pp 359–373 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) aggressive language detection using vgcn-bert for spanish texts download book pdf download book epub errol mamani-condori  orcid: orcid.org/0000-0001-8481-036610 & josé ochoa-luna  orcid: orcid.org/0000-0002-8979-378510  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1254 accesses 1 citation abstract the increasing influence from users in social media has made that aggressive content disseminates over the internet. to tackle this problem, recent advances in aggressive language detection have demonstrated a good performance of deep learning techniques. recently transformer based architectures such as bidirectional encoder representations from transformer (bert) outperformed previous aggressive text detection baselines. however, most of the transformers-based approaches are unable to properly capture global information such as language vocabulary. thus, in this work, we focus on aggressive content detection using the combination of vocabulary graph convolutional network (vgcn) to capture global information and bert to model local information. this combined approach called vgcn-bert allows us to improve the feature level representation in spanish aggressive language detection. our experiments were performed on a benchmark called mex-a3t aggressiveness dataset which is composed of aggressive and non-aggressive tweets written in the mexican spanish variant. we report 86.46% in terms of f1-score using this vgcn-bert approach which allows us to obtain comparable results with the current state-of-the-art, ensemble bert, so as to detect aggressive content regarding the track mex-a3t 2020. this research was supported by national fund for scientific and technological development and innovation (fondecyt-perú) within the framework of the “project of 50 e038-2019-01-bm improvement and expansion of the services of the national system of science, technology and technological innovation” [grant 028-2019-fondecyt-bm-inc.inv]. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others vgcn-bert: augmenting bert with graph embedding for text classification chapter © 2020 enhancing intent detection through chatgpt-driven data augmentation chapter © 2025 diagnosing bert with retrieval heuristics chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. aggression computational linguistics languages language processing language and gender romance languages 1 introduction the exponential growth of social media such as twitter, community forums, and blogging platforms has revolutionized communication and content publishing, but it has also increased the dissemination of hate speech [6, 32]. thus, nowadays offensive and aggressive language is pervasive in social media. the content which has profanity, abusive, aggressive, or any kind of words that disparages a person or a group is considered hate speech. seemingly, the language is considered aggressive if encourages any kind of hostility or intention to be harmful, conveying a violent message. social media platforms and technology companies have heavily invested in ways to cope with this offensive language to prevent abusive behavior in social media [29]. one of the first approaches to tackle this problem was human control on text content. this manual filtering is very time-consuming as it can cause post-traumatic stress disorder-like symptoms to human annotators. thus, the most effective strategy is to use computational methods to identify offense, aggression, and hate speech in user-generated content. this topic has attracted significant attention as evidenced in recent publications [11, 28]. in this context, we address the automated task of aggressive language identification in spanish texts. to do so, we use the combination of bert model and vocabulary graph convolutional network which improves local information encoded in bert embeddings by adding global information between words and concepts (vocabulary gcn). this combination has been previously applied to other tasks [20, 30]. we argue that vgcn-bert is better suited to identify aggressive language in spanish because it allows integrating the global notion of a specific domain language. that is, in addition to being able to capture the context of a word within a sentence with bert, we can also catch the semantic relationships between words which can be replaced with other words in the same category using vgcn. basically, we incorporate the feature of a word and its relationship to other words that may replace it in meaning. for example, the phrase “read a book” would be replaced with “study a book” or “understand a book”. it is worth noting that some aggressive words are composed by two or more tokens. one single word rarely appears as aggressive in the corpus, and most of the deep learning models cannot even find the word in their pre-trained vocabulary. whereby, adding an extra feature of the connected “read a book” to “study a book” meaning will add a piece of global context information to the model. connecting those expressions more explicitly with their meanings allows to take into account the knowledge about the language (vocabulary). for instance, let consider the following tweet of aggressive content: las fans de odisseo se ven bien bonitas en sus fotos de twitter y estan bien feas en persona. (odisseo fans look really pretty in their twitter photos and they are pretty ugly in person.) the sentence “they are pretty ugly” can be related to “disgusting” and “nasty” through the connections in a gcn graph. in this example, both positive and negative words appear in the sentence and the component “odisseo fans look really pretty” denotes a strong positive opinion. thus, the overall text would be wrongly classified as non-aggressive due to the first part overweights the classifier. on the contrary, if we connect explicitly the last part of the sentence “pretty ugly” with the meaning of “really nasty” taking into account the knowledge of the aggressive tweet content (vocabulary), then it would be classified more accurately. this last correlation between words and concepts in gcn has demonstrated to have a good performance capturing global information [30]. in brief, the combination of the local (embedding) and global information (graph) through self-attention is feed into a bert model. in this sense, the word and graph embeddings interact with each other through the attention mechanism while learning the classifier. in our experiments using vgcn-bert, we report 0.8646 in terms of macro f1-score. this result outperforms slightly the state-of-the-art ensemble bert model and other deep learning techniques [15], including the baselines reported during the mexican spanish aggressive detection (mex-a3t) task [3]. overall, the contribution of the paper can summarized as: combining the local an global information using vgcn-bert without any external word embeddings or knowledge which is suitable and novel to detect the aggressiveness words with a global notion for the spanish language. the combination of vgcn with a bert model (a spanish pre-trained version called beto [9]) allows to obtain comparable results regarding the ensemble bert models in spanish aggressiveness detection. the paper is organized as follows. related works are presented in sect. 2. the methodology is described in sect. 3. experiments and conclusions are presented in sects. 5 and 6. 2 related work the english offensive language detection problem has been increasingly researched in the last years [11, 31], this task is closely related to spanish aggressive identification. although many classification approaches have been applied to hate speech [5], offensive language, and aggressive, there still very few approaches applied to the spanish aggressive language [2] compared to english [31]. among them, the first attempts to detect offensive language were mostly based on the surface features techniques such as uni-grams and a larger n-grams. later badjatiya et al. [6] found out that the character n-grams had better performance than word tokens for hate speech detection. unlike feature extraction approaches, classification methods for offensive language detection have been predominantly supervised [23]. first works focused on manual feature engineering used for machine learning algorithms such as support vector machine (svm) [11], naive bayes [11], and logistic regression [29]. recent deep learning approaches [21] have demonstrated that automated feature learning representation allows to obtain better results. also, pre-trained word embeddings have been applied successfully [6]. nowadays, the best approach for the task and related nlp tasks has been the bidirectional encoder representation from transformer called bert [12]. although all of those techniques have been applied to the english language, they can also be “transferred” to other languages such as spanish. recently, to encourage the nlp community to develop this task for the spanish language, ibereval and iberlef (iberian languages evaluation workshops) released the aggressive identification taskfootnote 1. related works on spanish are mostly based on svm and convolutional neural network (cnn) [2]. it is also worth noting that good results were obtained using svm and bag of words [4] and long short term memory (lstm) and gate recurrent unit [14]. 2.1 transfer learning with bert with the recent emergence of the attention mechanism in 2017 [26], many improvements began to be noted in the field of natural language processing (nlp). the same could be observed with the proposal of the revolutionary bert [12]; that proved to be good and obtained outstanding results for 10 nlp downstream tasks. using bert for identifying offensive language in the english language has proven to be effective [31], obtaining results that far exceed traditional models. however, since the transformers architecture and the attention mechanism appeared, attempts to combine bert with one extra layer of attention were proposed [22]. in more recent related works multi-task bert and ensemble bert models showed good results to the detection of aggressive, offensive, and misogyny language in english, as reported by trac in aclfootnote 2 [18]. seemingly, the last mex-a3t 2020 recently reported the use of bert transformer-based models for aggressive language in spanish. the best results were obtained using a pre-trained bert version for the spanish called beto. fine-tuning beto for the aggressive has proved to be useful especially if we use an ensemble of different beto [9] models for classification. furthermore, beto has been integrated with autoencoder representation [27], and a xgboost classifier [10]. tanase et al. [25] tried to fine-tune beto with different dataset and multilingual transformer models. additionally, as evidenced in the last mex-a3t at iberlef 2020, 9 teams used bert, especially beto, and 5 obtained the best results [3]. 2.2 graph convolutional networks (gcn) throughout the development of research in natural language processing, graphs have served as a representation of documents and texts. very recently, this idea inspired to search for the relationship of words in a language. there are many studies combining graphs with neural networks which is commonly called graph neural networks (gnn) [7]. graphs proved to be good at capturing general knowledge about words in a language. thus, numerous studies and variations of gnns have been proposed and applied to text classification tasks [17, 24]. one of the most outstanding recent works showed the use of graph convolutional networks (gcn) based on the spectrum of graph theory [17]. kipf built a symmetric adjacency matrix denoting relationships in graphs. thus, the representation of a node is also affected by its neighbors and its relationship in the graph during the convolution. based on kipf’s work, yao et al. [30] recently proposed a special case of gcn for text classification called text gcn. the authors denoted words and documents as nodes in a graph and their relationships as edges. weights in edges can be calculated using three alternatives: the co-occurrence relationship between the words, tf-idf between documents and words and the similarity between documents. the work showed that gcns are good at convolving global information from an input graph. [20] would later demonstrate the effectiveness of combining bert with gcn using the co-occurrence measure and the relationship between words which is called vocabulary graph convolutional networks (vgcn) at the feature level. 3 methodology the goal of this paper is to identify whether a spanish tweet content is aggressive or not. to do that, we carefully fine-tune the bidirectional encoder representation from transformers (bert-spanish version) and combine that with the vgcn. our design choice for bert is due to its good performance in the english offensive detection [31]. thus, to identify aggressive language in spanish we have used the bert model. however, we have noted this model is not suitable to lead with spanish vocabulary which took us to explore other ways to improve it. bert is good at capturing the words’ order and the local context information, but it also unable to capture global information. this missing information could be relevant to recognize aggressiveness specially in long text sequences. starting from this point, we have already seen that gcns for text classification are good at extracting global information [30]. intuitively we thought of combining both approaches to join local and global text information. fortunately, yao et al. [30] focused on the feature level of bert adding gcn features. in this sense, to address the missing global information in bert, an additional embeddings graph would be needed. having said that, in this work we improve the bert (beto) model with a gcn. we aim to capture global notion and semantic correlation between words using the graph as a vocabulary of words following the previous proposals for text classification [16]. thereby, the gcn-bert architecture based on a vocabulary graph was deemed into the pipeline of our system. the vocabulary graph enables the lexical relationship between words in a language. the graph is built using the co-occurrence of words with documents such as lu did [20]. the local information of a tweet is captured by bert and is enriched by adding a graph embedding. we also must highlight that the construction of the vocabulary graph is based on the vocabulary of bert, in such a way that the size is reduced. at the same time, the added embedding can have an efficient interaction and improve local context information of bert. by doing so, the relevant part of the global vocabulary graph is selected according to the input sentence and transformed into an embedding representation. then vgcn-bert uses multiple layers of attention mechanism to concatenate both embeddings representations allowing the interaction between bert embeddings and graph embeddings. finally, the classifier uses the fully connected layers, more details are depicted in fig. 1. fig. 1. illustration of vgcn-bert. (blue) represents the input words, (green) are the related words in the graph, and the (blue-green) blend is the convolutional representation. the embeddings of the input sentence (phase 1) are combined with the vocabulary graph (phase 2) to produce a graph embedding, which is concatenated to the input sentence (phase 3). phase 2 produces three (hyperparameter) graph embeddings which have the same shape as the word embeddings. note that from the vocabulary graph, the relevant part from input is extracted and embedded. in this case (3,5,8) will be convoluted but (2,6) will not. (phase 4), several layers of self-attention are applied to the concatenated representation, allowing interactions between word embeddings and graph embeddings. the final embedding at the last layer is fed in a fully connected layer (phase 5) for classification [20]. (color figure online) full size image by applying the vgcn-bert we aim to add carefully the global information of a language using vocabulary (of bert) graph embedding to the classical embedding bert (spanish pre-trained beto) in the features level. the system we used obtained good results improving the detection of the aggressiveness words in contrast to the other ensemble models using multiple bert models at the same time. the vocabulary graph knowledge from global information and its background concepts of the words will be added to the local information throughout multiple self-attention mechanisms. the vgcn and the integration to bert will further explained in next sections. 3.1 vgcn our system follows the implementation and the model proposed by lu [20], which uses a vocabulary graph based on the word co-occurrences in documents. this graph is built using normalized point-wise mutual information (npmi) [8] to measure the relationship between words in a document. then, they create an edge between two words if their npmi is larger than a threshold and the performance is better when the threshold is between 0.0 and 0.3. the gcn is generally based on kipf’s model [17], i.e., it has two multi-layer neural networks that convolve directly on the graph and create embedding vectors of nodes based on the neighborhoods. in this case, the convolution is made from related words instead of documents and the gcn is constructed in the vocabulary. thus, for a single document, assuming the document is a row vector x consisting of words in the vocabulary, a layer of convolution is defined as (1). $$\begin{aligned} h = ( ax^{t} )^{t}w = xaw \end{aligned}$$ (1) where \(a^{t} = a\) is the vocabulary graph. xa extracts the part of vocabulary graph relevant to the input sentence x. w holds the weights of the hidden state vector for the single document, whose dimension is |v|h. we could define the two layers vgcn with relu as follows: $$\begin{aligned} vgcn = relu(x_{mv} a_{vv} w_{vh}) w_{hc} \end{aligned}$$ (2) where m is the number of documents in mini-batch size. v is the vocabulary size, h is the hidden layer size, c the class size or sentence embedding size [17]. 3.2 integrating vgcn into bert in this section, we want to integrate vgcn embeddings with bert and take advantage of its attention mechanism [26]. when using the attention mechanism one gets weighted vectors that encode the context information. thus, instead of using only word embeddings of the input sentences, we feed both the vocabulary graph embedding obtained with the eq. (2) and the sequence word embeddings to the bert transformer. in this sense, not only the order of the words in the sentence is retained (local context information), but also the background information obtained by vgcn (global information of the language). after that, the local and global embedding are fully integrated through layers interacting them with 12-layers and 12-heads of the self-attention encoder. 4 dataset since our goal is to identify the aggressiveness content in social media for spanish, we worked with the mexican spanish dataset mex-a3t trackfootnote 3. we also noted that this track on detecting aggressive language is not new; however, it is still a challenge for the spanish language as shown in the previous workshops: trac2020footnote 4, iberlef2020footnote 5 and the track at ivereval2018. the dataset was collected considering twitter as a main source media since it is open and its anonymity allows people to write judgments about others. while building the corpus they use some rude words and controversial hashtags to narrow the search that ranged from august until november of 2017. they also used around 143 terms that served as a seed to extract tweets [13]. additionally, the mex-a3t contains 10.475 tweets 3143 for testing and 7332 for training (see table 1) and we also noted that the test set does not contain labels in the dataset. it is worth noting that the data (tweets) contains a dictionary of mexican words “mexicanisms” with at least one of vulgar or insulting words; then they were manually labeled with “aggressive” and “non aggressive”. the criteria used while tagging was on the promise that offensive content is disparaging or humiliating a person or a group of persons [3]. the linguistic criteria are the approach of using “vulgar, aggressive and offensive” as an identifier. diaz et al. [13] used a new annotation scheme based on the linguistic characteristics and intent of the message. table 1. mex-a3t tweets with aggressive data set distribution of classes.full size table the aggressive content samples are shown below, we use # symbol to hide the aggressive word in purpose to avoid any offensive words for the reader. aggresssive samples: “sólo a las pu#..as mocos#s gordas y feas les gusta ese al%man xd” “profe hijo de las mil pu#..as 6 de calificación como es posible. ” non aggressive samples: “put#s madres ahora comprendo todo, tu tan lava y yo tan frio ” “ segunda vez que me pasa. estoy hasta la madre” we follow [15] to split the data, i.e., 80% for training and 20% for testing. the picture below describes this distribution. we can see that the amount of non-aggressive content is greater which means our data is imbalanced (fig. 2), but also it reflects the true distribution of data (more detailed in [1]). fig. 2. mex-a3t dataset with 5222 labeled distribution, left: 28.8% (green) aggressive, 71.2% no aggressive (pink), right: bar shows the majority class of the “non-aggressive” (blue) class 0 and 2110 for “aggressive” (orange) class 1. (color figure online) full size image 5 experiments this section presents the baselines models, pre-processing techniques, experiments, evaluation methodology, and results. we also discuss how we address the issue of vocabulary performance within aggressive identification. 5.1 baselines and task in order to investigate the aggressiveness detection for mexican spanish language, we studied the previous techniques and research lines. according the literature and the best performance models, we re-implement the current state of the art model proposed during the mex-a3t 2020 track [15]. we also took other baselines [3]. besides bert there are others neural network models that we will detail below: bow with svm (base-line): the approach is presented as a traditional machine learning approach. the model uses a bag-of-words representation with tf and tf-idf weights. one of their good results is obtained when applying a svm and other neural network [4]. bi-lstm (base-line): this deep learning model approach is a bi-directional gru model that uses words as inputs. the model also combines the predictions of gender and occupation of users obtained by a reference model, it uses a simple concatenation and one-hot-encodings [14]. transformers (state-of-the-art): the approach uses the novel pre-trained bert and performs fine-tuning to accomplish the detection of aggression for spanish. the bert variation used is the spanish transformer-based bert called betofootnote 6. there are also beto improvements using multiple ensembles of these models, weighting vote schemes and different data augmentation techniques [15]. most of the models above work well; however, we also explore deep learning classification models in other languages. thus, we found a good performance of the vgcn-bert for english text classification task. of course, there are other attempts to combine vgcn with bert (see sect. 2). we will briefly describe the vanilla model we based on: vanilla-vgcn-bert: our system is mainly based on this model which combines bert and vgcn. first, the model produces two separate representations through bert and gcn. both features are concatenated. then, fully connected layer and rectified linear unit (relu) are applied to classify [16]. there is a large number of models that obtained good results for the task of aggressive content detection on twitter. we were also able to describe some other scopes that were given for the classification of text in the english language. however, we noted that multi-tasking models or combined models have not yet been proposed to improve the detection of aggressive language especially for spanish language. thereby, we took as a baseline the works described above for the aggressive language detection for mexican spanish. 5.2 pre-processing and model setting when using a dataset that came from social media usually has a lot of noise and every tweet needs a pre-processing technique. to implement this work we first noted that there are url strings to be removed as well as “@”, i.e., we retain word text content and remove all unnecessary symbols. after cleaning, we also convert all the text into lower case and tokenized using social media tweettokenizer library from nltkfootnote 7. we also use the bert tokenizer to split the text, so the vocabulary of gcn is always a subset of the bert vocabulary. we fine-tune vgcn-bert model and train the model for the spanish aggressive language detection because we want to capture the global context. to get the best performance we are using vgcn with npmi for measuring the correlation between words in the vocabulary such we reduce the range of the meaningful relationship up to 0.2. the graph embedding size used in the vgcn is set 16 and the hidden graph embedding dimension is 128. whereas, the pre-trained bert model use 200 length max of the sequence. then, the whole model is trained in 8 epoch with a dropout of 0.2. furthermore, due to the limitation of our gpu memory we use mini batches of 8 to seed the data and a learning rate of 8e−6, \(l_2\) loss weight decay of 0.01. it is worth to mention that all these parameters were defined based on our experimental test results. we also follow the parameter setting of \(l_2\) loss weight and the learning rate according to the paper [20]. we also opted to use cross-entropy to optimize the classification. on the other hand, adam optimizer and class weights were estimated to handle the unbalanced data using compute_class_weightfootnote 8. 5.3 evaluation metrics as we are going to compare vgcn bert against baselines approaches, we adopt the widely used metric f1-score for text classification, which was also used by [13]. the weighted average f1-score and the macro f1-score were defined as follows [19]. $$\begin{aligned} f1= 2*\frac{precision * recall}{precision + recall} \end{aligned}$$ (3) $$\begin{aligned} \text {weigthed avg f1} = \sum _{i=1}^{c}f1_{ci}*w_{ci}, \quad \quad \text {macro f1} = \frac{1}{c}\sum _{i=1}^{c} f1_{ci} \end{aligned}$$ (4) where c: is the each individual different class. 5.4 experimental results in this section, we show the results of our vgcn-bert system and to compare it against the best models for the mexican spanish aggressive detection task. we carefully selected the previous outstanding approaches considered as baselines for this task. we also highlight the implementation of the bert (multi-language) combined with vgcn comparing to the best (sota) spanish pre-trained bert (beto). we also found that 5 of the best winning models were based on beto and clearly outperformed traditional deep learning methods [3]. those methods are references to compare results under the same criteria. however, to the best of our knowledge, there are very few bert models combined with other models. also, the bert multi-lingual was not much used during the mex-a3t aggressiveness analysis track at iberlef 2020 competition. thereby, our bert with vgcn system becomes a good alternative for the detection of aggressiveness, since our results show that it clearly outperformed the baselines models as shown in table 2. the table shows the “macro f1-score” evaluated with the test set for every model. to compare our proposal against the others, we use the average f1-macro score. we have also shown the scores for the aggressiveness and non-aggressiveness detection tasks. in addition, we also report the best model performed using ensembled beto models. we noted that the closest score to ours is the bert ensemble (20 betos). another good classifier is beto+xgboost referred to as beto+msg. on the other hand, we also re-implemented the single bert-multi-language, the single bert (spanish pre-trained beto), and the aforementioned ensembled beto following guzman et al. paper’s [15]. this last model varies from 1 beto to 20 betos and was used a voting scheme to compare them against our results. table 2. results of the aggressive detection using f1-score in test set.full size table those results demonstrate that our system which combined vgcn with the fine-tuned bert (beto-uncased) effectively identifies whether a given tweet contains aggressive content or not. furthermore, we see that all models (except ours) have low values when they classify the minority aggressive class. it is worth emphasizing that even in the case we are using a multi-language bert model (not beto) with vgcn, referred to as bert multi*, we are still able to outperform beto+xgboost. that shows the improvement over pre-trained bert model for spanish text. we also noted that our system obtained a high f1-score on the aggressive class. we obtained the third score on the non-aggressive class. we argue that those results are due to the global information which adds attention to the aggressive class with the vocabulary. one interesting thing that we found is that the vgcn-bert, without any additional external embedding, obtained comparable results to bert (20 betos). preliminary studies for english text classification did not show impressive improvements using vgcn and bert. however, in this paper we have shown that this model is suitable and fits better in the multi-language scenario. despite all apparent disadvantage, we have shown that global information captured using vgcn can contribute to the classification task. likewise, if we add global to the local context information from bert then the result is very good at detecting aggressive language. 6 conclusion in this paper, we have explored the tight combination of the transfer learning fine-tuning bert (spanish pre-trained beto) model and the graph convolutional network to identify aggressive content for spanish tweets. we have shown that adding a vocabulary graph, even in a small set of vocabulary for spanish bert, could improve the context local information. that was possible since the gcn has proven to be good at capturing the global information of a language. our experimental results allowed us to state that using vgcn-bert could be beneficial to detect aggression content, specially for the spanish language. on the other hand, although our system used the original classic bert version, with disadvantages compared to ensembled betos, it is still able to perform well when combined with another neural network such as gcn. thus, bert could be improved using the vocabulary graph convolutional network as our promising results show. we have also demonstrated that it is still possible to improve the original bert with models that increase the global knowledge information about the domain-specific language. based on our results, we also argue that, for detecting the aggressiveness in tweets, a vocabulary and global information of a language is a path for improving the bert model as a classifier. furthermore, we believe that a word within a sentence could contribute a lot if its relationship within a tweet contributes to detect aggressiveness. we also leave open another way to explore gcn for text classification. notes 1.mex-a3t: authorship and aggressiveness analysis in the mexican spanish case study. 2.association for computer language conference. 3.mex-a3t site: https://mexa3t.wixsite.com/home. 4.ii trolling aggressive and cyberbylling workshop. 5.iberian languages evaluation forum 2020. 6.beto is a pre-trained bert on a large spanish corpus [9]. 7.http://www.nltk.org/api/nltk.tokenize.html. 8.https://scikit-learn.org/.../sklearn.util..._weight.compute_class_weight.html. references álvarez-carmona, m.á., et al.: overview of mex-a3t at ibereval 2018: authorship and aggressiveness analysis in mexican spanish tweets. in: notebook papers of 3rd sepln workshop on evaluation of human language technologies for iberian languages (ibereval), seville, spain, vol. 6 (2018) google scholar  aragón, m.e., álvarez-carmona, m.á., montes-y gómez, m., escalante, h.j., villasenor-pineda, l., moctezuma, d.: overview of mex-a3t at iberlef 2019: authorship and aggressiveness analysis in mexican spanish tweets. in: notebook papers of 1st sepln workshop on iberian languages evaluation forum (iberlef), bilbao, spain (2019) google scholar  aragón, m., et al.: overview of mex-a3t at iberlef 2020: fake news and aggressiveness analysis in mexican spanish. in: notebook papers of 2nd sepln workshop on iberian languages evaluation forum (iberlef), malaga, spain (2020) google scholar  arce-cardenasa, s., fajardo-delgadoa, d., álvarez-carmonab, m.á.: tecnm at mex-a3t 2020: fake news and aggressiveness analysis in mexican spanish (2020) google scholar  plaza-del arco, f.m., molina-gonzález, m.d., ureña-lópez, l.a., martín-valdivia, m.t.: comparing pre-trained language models for spanish hate speech detection. expert syst. appl. 166, 114120 (2021) google scholar  badjatiya, p., gupta, s., gupta, m., varma, v.: deep learning for hate speech detection in tweets. in: proceedings of the 26th international conference on world wide web companion, pp. 759–760. international world wide web conferences steering committee (2017) google scholar  battaglia, p.w., et al.: relational inductive biases, deep learning, and graph networks. arxiv preprint arxiv:1806.01261 (2018) bouma, g.: normalized (pointwise) mutual information in collocation extraction. in: proceedings of gscl, pp. 31–40 (2009) google scholar  canete, j., chaperon, g., fuentes, r., pérez, j.: spanish pre-trained bert model and evaluation data. in: pml4dc at iclr 2020 (2020) google scholar  casavantes, m., lópez, r., gonzález, l.: uach at mex-a3t 2020: detecting aggressive tweets by incorporating author and message context. in: notebook papers of 2nd sepln workshop on iberian languages evaluation forum (iberlef), malaga, spain (2020) google scholar  davidson, t., warmsley, d., macy, m., weber, i.: automated hate speech detection and the problem of offensive language. in: eleventh international aaai conference on web and social media (2017) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) díaz-torres, m.j., morán-méndez, p.a., villasenor-pineda, l., montes, m., aguilera, j., meneses-lerín, l.: automatic detection of offensive language in social media: defining linguistic criteria to build a mexican spanish dataset. in: proceedings of the second workshop on trolling, aggression and cyberbullying, pp. 132–136 (2020) google scholar  garrido-espinosa, m., rosales-pérez, a., lópez-monroy, a.: gru with author profiling information to detect aggressiveness. in: notebook papers of 2nd sepln workshop on iberian languages evaluation forum (iberlef), malaga, spain (2020) google scholar  guzman-silverio, m., balderas-paredes, a., lópez-monroy, a.: transformers and data augmentation for aggressiveness detection in mexican spanish. in: notebook papers of 2nd sepln workshop on iberian languages evaluation forum (iberlef), malaga, spain (2020) google scholar  jeong, c., jang, s., park, e., choi, s.: a context-aware citation recommendation model with bert and graph convolutional networks. scientometrics 124(3), 1907–1922 (2020) article  google scholar  kipf, t.n., welling, m.: semi-supervised classification with graph convolutional networks. arxiv preprint arxiv:1609.02907 (2016) kumar, r., et al.: proceedings of the second workshop on trolling, aggression and cyberbullying (2020) google scholar  lever, j., krzywinski, m., altman, n.: classification evaluation (2016) google scholar  lu, z., du, p., nie, j.-y., et al.: vgcn-bert: augmenting bert with graph embedding for text classification. in: jose, j.m. (ed.) ecir 2020. lncs, vol. 12035, pp. 369–382. springer, cham (2020). https://doi.org/10.1007/978-3-030-45439-5_25 chapter  google scholar  park, j.h., fung, p.: one-step and two-step classification for abusive language detection on twitter. arxiv preprint arxiv:1706.01206 (2017) samghabadi, n.s., patwa, p., srinivas, p., mukherjee, p., das, a., solorio, t.: aggression and misogyny detection using bert: a multi-task approach. in: proceedings of the second workshop on trolling, aggression and cyberbullying, pp. 126–131 (2020) google scholar  schmidt, a., wiegand, m.: a survey on hate speech detection using natural language processing. in: proceedings of the fifth international workshop on natural language processing for social media, pp. 1–10 (2017) google scholar  shang, j., ma, t., xiao, c., sun, j.: pre-training of graph augmented transformers for medication recommendation. arxiv preprint arxiv:1906.00346 (2019) tanase, m.a., zaharia, g.e., cercel, d.c., dascalu, m.: detecting aggressiveness in mexican spanish social media content by fine-tuning transformer-based models (2020) google scholar  vaswani, a., et al.: attention is all you need. arxiv preprint arxiv:1706.03762 (2017) villatoro-tello, e., ramırez-de-la rosa, g., kumar, s., parida, s., motlicek, p.: idiap and uam participation at mex-a3t evaluation campaign. in: notebook papers of 2nd sepln workshop on iberian languages evaluation forum (iberlef), malaga, spain (2020) google scholar  waseem, z., davidson, t., warmsley, d., weber, i.: understanding abuse: a typology of abusive language detection subtasks. arxiv preprint arxiv:1705.09899 (2017) waseem, z., hovy, d.: hateful symbols or hateful people? predictive features for hate speech detection on twitter. in: proceedings of the naacl student research workshop, pp. 88–93 (2016) google scholar  yao, l., mao, c., luo, y.: graph convolutional networks for text classification. in: proceedings of the aaai conference on artificial intelligence, vol. 33, pp. 7370–7377 (2019) google scholar  zampieri, m., malmasi, s., nakov, p., rosenthal, s., farra, n., kumar, r.: semeval-2019 task 6: identifying and categorizing offensive language in social media (offenseval). arxiv preprint arxiv:1903.08983 (2019) zhang, z., luo, l.: hate speech detection: a solved problem? the challenging case of long tail on twitter. semant. web 10(5), 925–945 (2019) google scholar  download references author information authors and affiliations department of computer science, universidad católica san pablo, arequipa, peru errol mamani-condori & josé ochoa-luna authors errol mamani-condoriview author publications search author on:pubmed google scholar josé ochoa-lunaview author publications search author on:pubmed google scholar corresponding author correspondence to errol mamani-condori . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper mamani-condori, e., ochoa-luna, j. (2021). aggressive language detection using vgcn-bert for spanish texts. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_25 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords aggressiveness detection bert graph convolutional networks publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature erasmo: leveraging large language models for enhanced clustering segmentation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper erasmo: leveraging large language models for enhanced clustering segmentation conference paper first online: 30 january 2025 pp 414–429 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) erasmo: leveraging large language models for enhanced clustering segmentation download book pdf download book epub fillipe dos santos silva9,10,11, gabriel kenzo kakimoto9,10,12, julio cesar dos reis9,11 & … marcelo s. reis9,10,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 512 accesses 1 citation abstract cluster analysis plays a crucial role in various domains and applications, such as customer segmentation in marketing. these contexts often involve multimodal data, including both tabular and textual datasets, making it challenging to represent hidden patterns for obtaining meaningful clusters. this study introduces erasmo, a framework designed to fine-tune a pretrained language model on textually encoded tabular data and generate embeddings from the fine-tuned model. erasmo employs a textual converter to transform tabular data into a textual format, enabling the language model to process and understand the data more effectively. additionally, erasmo produces contextually rich and structurally representative embeddings through techniques such as random feature sequence shuffling and number verbalization. extensive experimental evaluations were conducted using multiple datasets and baseline approaches. our results demonstrate that erasmo fully leverages the specific context of each tabular dataset, leading to more precise and nuanced embeddings for accurate clustering. this approach enhances clustering performance by capturing complex relationship patterns within diverse tabular data. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others information overload in crisis management: bilingual evaluation of embedding models for clustering social media posts in emergencies chapter © 2025 optimizing embedding-based video recommendation in spanish using llms for automated model and parameter selection chapter © 2026 evaluating the impact of content deletion on tabular data similarity and retrieval using contextual word embeddings chapter © 2024 1 introduction tabular data is ubiquitous in various fields such as finance, healthcare, and marketing, where it serves as a primary source of information for machine learning (ml) tasks [15]. despite its widespread use, extracting meaningful insights from tabular data remains a complex challenge, particularly in clustering tasks [6]. these challenges include handling heterogeneous feature types, dealing with high-dimensional spaces, and ensuring meaningful distance metrics. these can significantly impact the effectiveness of clustering algorithms in identifying natural groupings within the data [14]. recent researchers have explored traditional statistical methods and modern deep learning approaches [14, 19, 21] to address these challenges. existing studies have utilized large language models (llms) like openai’s gpt and llama to create embeddings from textual datasets, enhancing data representation and analysis [6, 14]. in addition, a method combining llms and deterministic,independent-of-corpus embeddings (dice) has been proposed to generate consistent embeddings across datasets, improving segmentation accuracy [19]. however, these approaches often fail to fully leverage the specific context of each tabular dataset, resulting in less nuanced embeddings for precise clustering. this study originally introduces erasmo, our proposed framework designed to generate high-quality embeddings from tabular data using transformer-based language models. these embeddings excel in clustering analysis, revealing hidden patterns and groupings. our solution can also be used in retrieval-augmented generation (rag) systems and other tasks to gain deeper insights from context information [7]. erasmo operates through two stages: (1) fine-tuning a pretrained language model on textually encoded tabular data; and (2) generating embeddings from the fine-tuned model. using techniques like random feature sequence shuffling and number verbalization, erasmo produces contextually rich and structurally representative embeddings, outperforming all clustering strategies from the literature based on internal metrics. our experimental evaluation used three clustering quality metrics to compare erasmo with state-of-the-art methods: silhouette score (ss), calinski-harabasz index (chi), and davies-bouldin index (dbi). these metrics comprehensively assess clustering effectiveness results by measuring cohesion, separation, and overall cluster structure. we extensively evaluated erasmo on real-world datasets without true labels, including banking marketing targets, e-commerce public dataset by olist, yelp reviews, petfinder.my, and women’s clothing reviews. these datasets encompass diversified information and present various challenges, rigorously testing erasmo’s clustering capabilities. this article provides the main contributions as follows: we introduce erasmo, a novel framework that leverages transformer-based language models to generate high-quality embeddings from tabular data, enhancing clustering analysis. we demonstrate that our framework significantly improves clustering performance by capturing the complex relationships within tabular data through random feature sequence shuffling and number verbalization techniques. we experimentally achieve state-of-the-art clustering results with erasmo, showcasing its effectiveness in identifying patterns and groupings within diverse tabular datasets. to the best of our knowledge, erasmo is the first framework to fully integrate and fine-tune transformer-based language models specifically for generating embeddings from tabular data, resulting in superior clustering outcomes. the remainder of this article is organized as follows. section 2 presents a synthesis of related work. section 3 details the erasmo framework. section 4 outlines our experimental methodology. section 5 presents the results of our evaluations. section 6 discusses the implications of our findings. finally, sect. 7 provides conclusions and directions for future work. 2 related work several studies have explored the application of llms to transform tabular data for clustering tasks, demonstrating the potential to enhance user segmentation and data analysis [6, 14, 19, 21, 23, 24]. zhu et al. [24] proposed a novel method named word embedding of dimensionality reduction (werd) for document clustering. their approach integrates pre-trained word embeddings with dimensionality reduction techniques. in their work, sentence-bert embeds them into high-dimensional vectors after preprocessing documents, which pacmap then reduces. spectral clustering is applied, followed by non-negative matrix factorization to extract keywords. clusterllm  [23], a novel text clustering framework, leverages feedback from llms such as chatgpt. this method enhances clustering by utilizing llms to refine clustering perspectives and granularity through two stages: a triplet task for fine-tuning embedders based on user preferences and a pairwise task for determining cluster granularity. extensive experiments on fourteen datasets demonstrated that clusterllm consistently improves clustering quality and is cost-effective, outperforming traditional clustering methods. both werd [24] and clusterllm [23] presented limitations compared to the erasmo framework (our proposal). werd might not fully capture the contextual nuances of each dataset due to its focus on dimensionality reduction techniques. at the same time, clusterllm’s reliance on general-purpose llms for guidance may overlook specific dataset characteristics. a method demonstrating that llms enables few-short learning applied to clustering tasks was proposed in [21]. their study showed how llms can perform clustering tasks with minimal labeled data by leveraging their extensive pretraining, significantly reducing the need for large annotated datasets and achieving reasonable clustering performance with few-shot learning. similarly, tipirneni et al. [18] explored context-aware clustering using llms, highlighting how these models can utilize contextual information to enhance clustering accuracy. both methods, however, may not fully leverage the dataset-specific nuances as effectively as erasmo because we employ a fine-tuning step, allowing the model to capture better and utilize dataset-specific details, leading to more accurate and reliable clustering results. tissera, asanka, & rajapakse developed  [19] an approach to enhancing customer segmentation using llms and dice. their method combined llms with dice to generate consistent and deterministic embeddings across different datasets, improving segmentation accuracy and robustness. their approach may not fully leverage the context-specific nuances of each dataset as effectively as erasmo. our fine-tuning process proposal allows it to adapt to the unique characteristics of the input data, providing more contextually rich and detailed embeddings that can result in more precise and meaningful clusters. a comparative analysis of llm embeddings for effective clustering was explored in  [6]. as an extension, the study on text clustering with llm embeddings [14] delves deeper, exploring additional models and datasets to demonstrate improvements in text data clustering. while existing approaches effectively capture complex semantic relationships and handle categorical, numerical, and textual data, they lack the fine-tuning specificity of erasmo, our key originality aspect. erasmo’s tailored embeddings for tabular datasets and integration of feature order permutation provide more precise and contextually relevant clusters, offering superior versatility and robustness in various clustering applications. 3 erasmo this section introduces erasmo, our framework that leverages transformer-based language models to generate high-quality embeddings from tabular data. these embeddings are particularly effective for clustering analysis, allowing for identifying patterns and groupings within the data that might not be immediately apparent. the process involves two main stages: (1) fine-tuning a pretrained llm on a textually encoded tabular dataset; and (2) utilizing the fine-tuned model to generate embeddings, which are used by a clustering algorithm. these designed stages were inspired by  [2]. subsection 3.1 details the fine-tuning phase, whereas subsect. 3.2 reports on the embedding generation processes. 3.1 phase 1: fine-tuning standard pretrained generative llms expect sequences of words as inputs. hence, we convert each row of our dataset into a textual representation to apply an llm to tabular data, which can contain categorical, numerical, and textual information. definition 1 (textual converter). given a tabular dataset with m columns with feature names \(f_1, f_2, \ldots , f_m\) and n rows of samples \(s_1, \ldots , s_n\), let the entry \(v_{i,j}\), \(i \in \{1, \ldots , n\}\), \(j \in \{1, \ldots , m\}\) represent the value of the j-th feature of the i-th data point. taking the feature name and value into account, each sample \(s_i\) of the table is transformed into a textual representation \(t_i\) using the following subject-predicate-object transformation: $$\begin{aligned} t_{i,j} &= [f_j, \text {``is''}, v_{i,j}, \text {``,''}], \quad \forall i \in \{1, \ldots , n\}, j \in \{1, \ldots , m\} \end{aligned}$$ (1a) $$\begin{aligned} t_i &= [t_{i,1}, t_{i,2}, \ldots , t_{i,m}], \quad \forall i \in \{1, \ldots , n\}, \end{aligned}$$ (1b) where \(t_{i,j}\), the textually encoded feature, is a clause with information about a single value and its corresponding feature name, and \([ \cdot ]\) denotes the concatenation operator. by transforming a tabular feature vector into a sequence using the textual subject-predicate-object encoding scheme, pseudo-positional information is artificially introduced into the transformed tabular data sample. however, there is no spatial ordering relationship between features in tabular datasets. we randomly permute the encoded short sentences \(t_{i,j}\) of the full textual representation \(t_i\) to reconstruct the feature order independence. definition 2 (random feature sequence shuffle). let \(t_i, i \in \{1, \ldots , n\}\), be a textual representation. consider a sequence \(k = ( k_1, \ldots , k_m )\) that is a permutation of the sequence of indices \(( 1, \ldots , m )\). a random feature sequence shuffle is defined as \(t_i(k) = [t_{i,k_1}, t_{i,k_2}, \ldots , t_{i,k_m}]\). we fine-tune our generative language model on samples without order dependencies when using shuffled orders of the textually encoded features. moreover, such permutations are highly beneficial as they allow for arbitrary conditioning in tabular data generation. in our experiments, we refer to erasmobase as the baseline model, utilizing only the textual converter and random feature sequence shuffle. in addition, there is evidence that verbalizing numerical tokens can enhance effectiveness in specific scenarios [8]. in this sense, we explore this approach, naming it erasmonv, as follows. definition 3 (number verbalizer). let \(t_i, i \in \{1, \ldots , n\}\), be a textual representation, and \(t_{i,j}\) be the set of words of the j-th feature of \(t_i\). a number verbalizer is a function v that receives as input a word w of \(t_{i,j}\) and is defined as: $$\begin{aligned} v(w) &= {\left\{ \begin{array}{ll} w, \text { if }w is not numerical,\\ \text {verbalized }w otherwise. \end{array}\right. } \end{aligned}$$ (2) by applying this transformation on every token of every textual representation, we ensure that any numerical information in the text is verbalized. in some nlp tasks, such as clustering with embeddings, sentiment analysis, and text classification, verbalizing numbers can improve the model’s understanding of the context and meaning of numerical values, leading to more accurate and meaningful results. this transformation might not be beneficial in some cases, depending on the specific nature of the data and the task at hand [8, 9]. fine-tuning a pretrained auto-regressive language model : we describe the fine-tuning procedure of a pretrained llm on the encoded tabular data for generation tasks. we suppose a textually encoded tabular dataset \(t = \{t_i(k)\}_{i=1, \ldots , n}\) that was transformed into text by the proposed encoding scheme. let k be a randomly drawn permutation, and n denote the number of rows. based on user choice, the pipeline can proceed directly to fine-tuning the llm to generate erasmobase, or it can first apply a number verbalizer to convert numerical tokens into their verbal representations before fine-tuning the llm to generate erasmonv. to be processed with an llm, the input sentences \(t \in t\) must be encoded into a sequence of tokens from a discrete and finite vocabulary w. these tokens can be character, word, or subword encodings such as the byte-pair-encodings (bpe). thus, \(t \in t\) is represented by a sequence of tokens \((w_1, \ldots , w_j) = \text {tokenize}(t)\) with tokens \(w_1, \ldots , w_j \in w\), where j denotes the number of tokens required to describe the character sequence t. commonly, the probability of natural-language sequences is factorized in an auto-regressive manner in llms. it is represented as a product of output probabilities conditioned on previously observed tokens: $$\begin{aligned} p(t) = p(w_1, \ldots , w_j) = \prod _{k=1}^j p(w_k \mid w_1, \ldots , w_{k-1}). \end{aligned}$$ (3) as a result, an end-user can choose any existing generative language model for tabular data modeling and exploit the vast amount of contextual knowledge presented in these models. fine-tuning enables the model to leverage this contextual information with the feature and category names to enhance the model’s capabilities. figure 1 presents the pipeline for erasmo’s fine-tuning step. 3.2 phase 2: embedding generation and clustering analysis we generate embeddings from the model after fine-tuning the llm on the textually encoded tabular dataset. these embeddings capture the contextual relationships and features encoded during the training phase. we start by feeding the test dataset, transformed into its textual representation, into the fine-tuned llm. the model generates embeddings for each input sequence, providing a high-dimensional representation for each sample. this process ensures that the embeddings preserve the contextual and feature relationships learned during fine-tuning. depending on the user’s choice in the pipeline, the embeddings are generated from either erasmobase or erasmonv models, reflecting whether the number verbalizer step was applied. to generate these embeddings, the input sentences \(t \in t_{test}\) are encoded into sequences of tokens and processed by the fine-tuned llm. the embeddings are obtained from the final hidden states of the model, resulting in rich and informative representations of the data. these embeddings can then be utilized for various downstream tasks, including clustering analysis, to gain deeper insights into the data structure (cf. figure 2). 4 experimental methodology our experiments evaluated the quality assessment for the best-performing clustering algorithms for each dataset and approach (model) combination (cf. table 1 for the obtained results). subsection 4.1 describes the datasets used for training and testing. subsection 4.2 presents an overview of the clustering algorithms. subsection 4.3 describes the approaches used for comparison as baselines in our experiments. subsection 4.4 reports on the evaluation metrics. subsection 4.5 presents the implementation details. each experimental setup for a given dataset assessed the different language models fine-tuned and pretrained. each model considers the several clustering algorithms and their configuration. fig. 1. the erasmo data pipeline for the fine-tuning phase. first, a textual converter step transforms tabular data into meaningful text (1). next, a random feature order permutation step is applied (2). then, based on user choice, the pipeline diverges: it can proceed directly to fine-tuning a llm (3a) to generate erasmobase, or apply a number verbalizer (3b) before fine-tuning the llm (4b) to generate erasmonv. full size image fig. 2. the erasmo pipeline for generating embeddings and cluster analysis. the input test tabular data is first transformed into text sequences (1). next, a random feature order permutation step is applied (2). for erasmonv, a number verbalizer step follows (3) before processing by the fine-tuned llm to generate embeddings (4). for erasmobase, the pipeline goes directly from step (2) to step (4). the embeddings are subsequently used for clustering analysis. full size image 4.1 datasets we selected a diversified set of datasets to encompass a variety of challenges related to text categorization and clustering, and we used them to evaluate text clustering algorithms. banking marketing targets: composed of data from direct marketing campaigns of a banking institution, which includes client attributes like age and job, along with the response to the campaign [10]. e-commerce public dataset by olist: a brazilian e-commerce dataset with over 100,000 orders from 2016 to 2018 across multiple marketplaces [11]. it includes 72,794 training and 18,199 testing samples. the recency, frequency, monetary (rfm) model was used for customer segmentation, as described in [19]. yelp: comprises reviews from yelp businesses, including text reviews, star ratings, and business attributes, offering a rich resource for sentiment analysis and review classification tasks [4]. petfinder.my: features adoption records from the petfinder.my website, encompassing various pet attributes, descriptions, and adoption status, valuable for text classification and clustering related to animal welfare [5]. women clothing reviews: contains reviews of women’s clothing, with each review detailing text feedback, ratings, and customer information, suitable for sentiment analysis and recommendation system research [3]. each unlabeled dataset was processed through the proposed pipeline, which involves training a pretrained llm. this approach ensures that the clustering algorithms can perform optimally across diverse textual inputs, enhancing their ability to effectively identify and group related data points. 4.2 clustering algorithms the clustering algorithms chosen are well-suited for handling complex patterns in structured and textual data, ensuring efficient categorization. we used the k-means algorithm for its simplicity and effectiveness with large datasets and k-means++ for its strategic centroid initialization to enhance clustering efficiency and quality [12]. unlike k-means, which assigns each data point to a single cluster, fuzzy c-means (fuzzycm) employs a probabilistic membership approach, effectively capturing the nuances and polysemy typical of textual data. we used agglomerative hierarchical clustering (ahc) to uncover hierarchical structures and spectral clustering for its proficiency in recognizing clusters based on the data’s graph structure, effectively identifying non-convex shapes. for k-means, the parameters were: initialization method set to random, number of initializations (ninit) set to 10, and the random seed set to 0. the k-means++ algorithm utilized k-means++ for initialization, ninit was 1, and the seed was 0. agglomerative hierarchical clustering (ahc) employed the euclidean metric with ward linkage. for fuzzy c-means (fuzzycm), no initialization method was specified, the fuzziness parameter (m) was 2, error tolerance was set to 0.005, and the maximum number of iterations (maxiter) was 1000. spectral clustering used ’discretize’ for label assignment and a random seed of 10. implementations for these algorithms were sourced from the scikit-learn library [13], except for fuzzycm, which used the scikit-fuzzy package [22]. for k-means and k-means++, init specifies the initial cluster centroid method, ninit indicates the number of algorithm runs with different seeds, and seed sets the random number for centroid initialization. in ahc, metric is the metric used for linkage computation, and linkage is the criterion measuring the distance between observation sets. we used euclidean distance to measure point similarities and a nearest centroid approach to associate clusters. for fuzzycm, init is the initial fuzzy c-partitioned matrix (random if none), m is the fuzziness degree, error is the stopping criterion, and maxiter is the iteration limit. in spectral clustering, assign_labels specifies the labeling strategy in the embedding space, and seed is the pseudorandom number for initializing the eigenvector decomposition. for all datasets, the number of clusters (\( k \)) was determined using the silhouette score to optimize cluster cohesion and separation. 4.3 approaches (baselines) we utilized various embedding techniques from state-of-the-art llms, including openai, falcon, llama 2, gpt-2 medium, and an mpnet-based model, each enhancing text representation by capturing contextual nuances. for the mpnet-based model, we used sentence-transformers/all-mpnet-base-v2 (mpnet-v2) [17]. for the openai model, we utilized text-embedding-3-large, and for the falcon model, we used tiiuae/falcon-7b [1]. additionally, we employed llama-2-7b-chat-hf [20] for chat applications and gpt2-medium [16] for faster textual representations. we used the embeddings from the last layer of all models for the most contextually rich text representations. we integrated an additional baseline from a recent study that explores customer segmentation using llms combined with dice [19]. they used the paraphrase-multilingual-mpnet-base-v2 model from sentence transformers to generate 768-dimensional sentence embeddings. this model, based on the mpnet architecture, has about 278 million parameters and is designed for clustering and semantic search. 4.4 evaluation metrics we use a set of metrics to evaluate our proposed framework and baselines thoroughly. specifically, we employed the ss, chi, and dbi metrics to assess the cohesion, compactness, and separation of clusters, ensuring a robust analysis of their structural integrity. the ss metric, which assesses the separation and cohesion of clusters, is calculated for each data point \(i\) as: $$\begin{aligned} s(i) = \frac{b(i) a(i)}{\max \{a(i), b(i)\}}, \end{aligned}$$ where \(a(i)\) measures the average intra-cluster distance, and \(b(i)\) is the minimum inter-cluster distance for the point \(i\). chi measures the ratio of between-cluster dispersion to within-cluster dispersion, providing insights into the overall clustering structure. the index is formulated as: $$\begin{aligned} chi = \frac{\text {tr}(b_k) / (k-1)}{\text {tr}(w_k) / (n-k)}, \end{aligned}$$ where \(\text {tr}(b_k)\) is the trace of the between-group dispersion matrix, and \(\text {tr}(w_k)\) the trace of the within-group dispersion matrix, thus evaluating both the separation and compactness of the clusters. the dbi evaluates the average similarity ratio of each cluster with its most similar one, offering a measure of cluster separation. the index is calculated as: $$\begin{aligned} dbi = \frac{1}{k} \sum _{i=1}^{k} \max _{j \ne i} \left( \frac{\sigma _i + \sigma _j}{d_{ij}} \right) , \end{aligned}$$ where \(\sigma _i\) and \(\sigma _j\) represent the average distance of all elements in clusters \(i\) and \(j\) to their respective centroids, and \(d_{ij}\) is the distance between centroids of clusters \(i\) and \(j\). lower values of dbi indicate better cluster separation. 4.5 our implemented setup we compare the described baselines (cf. subsection 4.3) using a pretrained transformer-decoder llm model, \(gpt-2\) medium [16], which has 355 million trainable parameters, 24 layers, 16 attention heads, an embedding size of 1024, and a context size of 1024. to facilitate this comparison, we convert the tabular dataset into text for all baselines, applying the random feature sequence shuffling function and comparing the results with erasmobase and erasmonv. both models were trained with a batch size of 8 over 60 epochs. we applied a dropout rate of 0.1 and utilized 500 warmup steps. the models incorporated a weight decay of 0.01 and used the adam optimizer with \(\epsilon \) set to 1e–8 and \(\beta \) values of [0.7, 0.9]. the initial learning rate was set to 5e–5, with a schedule starting at 1e–8, ranging from a minimum of 1e–5 to a maximum of 4e-5. the model was developed using pytorchfootnote 1 and is made available at a github repositoryfootnote 2. it ran on a system equipped with five nvidia rtx a6000, each having 48 gb of random access memory (ram). 5 experimental results we present key results obtained organized by datasets. table 1 presents the ss, chi, and dbi metrics results for the test dataset for all evaluated approaches; values in bold indicate the best outcomes. the best algorithm was determined by choosing the algorithm with the highest ss. banking. in the banking dataset, the erasmobase strategy outperformed all other strategies, achieving the highest  ss of 0.75, the highest chi of 12, 038.44, and the lowest dbi of 0.37. this indicates well-defined and compact clusters. erasmonv also performed strongly with an ss of 0.71 and chi of 7, 570.38, though it had a slightly higher dbi of 0.43 compared to erasmobase. other strategies like mpnet-v2 and falcon showed moderate performance with ss values of 0.27 and 0.23, respectively, while openai had the lowest ss at 0.11 and the highest dbi at 2.73, indicating poor clustering. olist. for the olist dataset, erasmonv achieved the highest ss of 0.77, indicating the best clustering quality. it reached the highest chi of 62, 036.87 and a low dbi of 0.32. erasmobase followed closely with an ss of 0.75 and a chi of 54236.31, along with the lowest dbi of 0.30. other strategies like llama-2 and falcon performed reasonably well, with ss values of 0.71 and 0.66, respectively. however, openai and mpnet-v2 showed lower ss values, with openai achieving an ss of 0.19 and mpnet-v2 an ss of 0.24. yelp. in the yelp dataset, erasmonv and erasmobase both demonstrated superior performance, with ss values of 0.79 and 0.78, respectively. erasmonv also achieved the highest chi of 8, 410.94 and tied with erasmobase for the lowest dbi of 0.28. other strategies, such as gpt2 medium and llama-2, showed moderate clustering performance with ss values of 0.39 and 0.29, respectively. openai, with an ss of 0.07, and mpnet-v2, with an ss of 0.23, indicated less effective clustering. table 1. this table shows the clustering quality assessment results for the top-performing algorithms across each dataset and approach combination. the best algorithm was chosen based on the highest ss. we provide the optimal number of clusters (k) and the results for ss, chi, and dbi. bold values indicate the best results for each metric.full size table petfinder.my. in the petfinder.my dataset, erasmonv slightly outperformed erasmobase with an ss of 0.73 compared to 0.72. erasmobase had the highest chi of 3, 351.95 and a low dbi of 0.40, while erasmonv had a chi of 3, 063.55 and the lowest dbi of 0.34. other strategies, such as gpt2 medium and falcon, showed moderate results with ss values of 0.55 and 0.20, respectively. mpnet-v2 had the lowest ss of 0.14, indicating poor clustering performance. clothings. for the clothings dataset, erasmobase achieved the highest ss of 0.72 and the highest chi of 6, 208.52, along with the lowest dbi of 0.39. erasmonv also performed well with an ss of 0.71 and a chi of 5, 916.57, matching the lowest dbi of 0.39. other strategies like gpt2 medium and llama-2 showed moderate performance, with ss values of 0.52 and 0.37, respectively. openai and mpnet-v2 had the lowest ss values of 0.07 and 0.12, respectively, indicating poor clustering quality. figure 3 shows a 2d t-distributed stochastic neighbor embedding (t-sne) visualization for the yelp dataset across all strategies. the yelp dataset was chosen for its rich and diverse user reviews, making it ideal for clustering evaluation. our models, erasmobase and erasmonv, display distinct and well-separated clusters, reflecting their lowest dbi scores. this highlights the superior clustering performance of our models. fig. 3. t-sne visualization of embedding representations on the yelp dataset for different models: (a) mpnet-v2, (b) openai, (c) llama-2, (d) falcon, (e) gpt2 medium, (f) pmv2 + dice, (g) erasmobase, and (h) erasmonv. full size image 6 discussion the ss results consistently highlight the superior clustering effectiveness of the erasmobase and erasmonv strategies across all datasets. erasmobase achieved the highest ss in the banking and clothing datasets, while erasmonv led in the olist, yelp, and petfinder.my datasets. this indicates that both strategies help to form well-defined clusters, with erasmobase having a slight edge in some datasets. the higher ss values for these models suggest they are better at creating distinct clusters than other strategies like mpnet-v2, openai, and falcon, which showed significantly lower ss values, indicating poorer representation relevance for clustering quality. the dbi results further support the effectiveness of erasmobase and erasmonv. both strategies consistently achieved the lowest dbi values, particularly excelling in the banking, olist, and yelp datasets. a lower dbi indicates more compact and well-separated clusters, affirming that our proposed models form tight and distinct clusters. erasmobase generally showed slightly better dbi scores, suggesting it produces more compact clusters than erasmonv. in contrast, approaches like openai and mpnet-v2 had significantly higher dbi values, reflecting less adequate cluster compactness and separation. the chi results corroborate the trends observed in ss and dbi. erasmobase and erasmonv achieved the highest chi scores across most datasets, with erasmobase leading in the banking and clothing datasets, and erasmonv in the olist and yelp datasets. high chi scores indicate that our proposed models create clusters that are not only well-separated but also highly dense. these superior results across multiple metrics underscores the robustness and effectiveness of the erasmo approach. other models like mpnet-v2, openai, and falcon demonstrated lower chi scores, indicating less effective clustering effectiveness. erasmo has limitations, including reliance on high-quality data, challenges with ambiguous datasets, and significant computational demands for fine-tuning. random sequence shuffling can affect reproducibility, and verbalizing numerical tokens does not consistently improve outcomes, warranting further investigation. furthermore, applying traditional metrics such as ss, chi, and dbi to analyze clusters from various representation spaces appears as an open research challenge yet to be resolved in the literature. while these metrics are designed for uniform representation spaces, they are frequently applied to different representation spaces, as evidenced by previous studies [6, 14]. we recognize these limitations and stress the relevance of investigating and developing further refined metrics capable of accurately assessing embeddings from varied dimensional spaces. despite these challenges, erasmo significantly advances clustering techniques by enhancing data representation. this innovation sets the stage for the potential standardization of metrics and improves clustering quality. erasmo stands as a crucial development in pursuing more robust clustering strategies. 7 conclusion exploring structured and textual data simultaneously for clustering analysis is a challenging problem. this study presented and evaluated the erasmobase and erasmonv clustering approaches, demonstrating their superior empirical effectiveness across multiple datasets. the results, based on ss, dbi, and chi, consistently revealed that our proposed models help to create well-defined, compact, and dense clusters, outperforming all strategies. despite their high computational demands and dependency on data quality, we found the erasmo models suited for practical clustering tools. future work might focus on optimizing computational efficiency and enhancing robustness for diverse and noisy datasets. notes 1.pytorch.org. 2.erasmo github. references almazrouei, e., et al.: falcon-40b: an open large language model with state-of-the-art performance (2023) google scholar  borisov, v., seßler, k., leemann, t., pawelczyk, m., kasneci, g.: language models are realistic tabular data generators. arxiv preprint arxiv:2210.06280 (2022) brooks, n.: women’s e-commerce clothing reviews (2017). https://www.kaggle.com/datasets/nicapotato/womens-ecommerce-clothing-reviews dataset, y.: yelp dataset (2014). http://www.yelp.com/dataset_challenge kaggle, petfinder.my: petfinder.my adoption prediction (2019). https://www.kaggle.com/c/petfinder-adoption-prediction/data/ keraghel, i., et al.: beyond words: a comparative analysis of llm embeddings for effective clustering. in: int. symposium on intelligent data analysis, pp. 205–216. springer, heidelberg (2024) google scholar  lewis, p., perez, a., petroni, w.t., rocktäschel, t., et al.: retrieval-augmented generation for knowledge-intensive nlp tasks. in: nips (2020) google scholar  liu, y., he, x., zhong, t., et al.: understanding llms: a comprehensive overview from training to inference. arxiv preprint arxiv:2401.02038 (2024) min, b., ross, d., others.: recent advances in natural language processing via large pre-trained language models: a survey. acm comput. surv. 1–40 (2023) google scholar  moro, s. rita, p., cortez, p.: bank marketing. uci machine learning repository (2012). https://doi.org/10.24432/c5k306 olist: brazilian e-commerce public dataset by olist (2023). https://doi.org/10.34740/kaggle/dsv/195341. acessado em: 03 de outubro de 2023 ourabah, o., et al.: large scale data using k-means. mesopotamian j. big data 2023, 36–45 (2023) math  google scholar  pedregosa, f., varoquaux, m., prettenhofer, v., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  petukhova, a., matos-carvalho, j.p., fachada, n.: text clustering with llm embeddings. arxiv preprint arxiv:2403.15112 (2024) pitafi, s., anwar, t., sharif, z.: a taxonomy of machine learning clustering algorithms, challenges, and future realms. appl. sci. 13(6), 3529 (2023) article  math  google scholar  radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i., et al.: language models are unsupervised multitask learners. openai blog 1(8), 9 (2019) google scholar  song, k., et al.: mpnet: masked and permuted pre-training for language understanding. adv. neural inf. process. syst. 33, 16857–16867 (2020) math  google scholar  tipirneni, s., adkathimar, n., hiranandani, v.n., yuan, c., reddy, c.k.: context-aware clustering using large language models. arxiv (2024) google scholar  tissera, m., asanka, p., et al.: enhancing customer segmentation using llms and deterministic, independent-of-corpus embeddings. in: 2024 4th (icarc) (2024) google scholar  touvron, h., martin, l., stone, k., et al.: llama 2: open foundation and fine-tuned chat models. arxiv preprint arxiv:2307.09288 (2023) viswanathan, v., gashteovski, k., lawrence, c., wu, t., neubig, g.: large language models enable few-shot clustering. arxiv preprint arxiv:2307.00524 (2023) warner, j., sexauer, j., et al.: jdwarner/scikit-fuzzy: scikit-fuzzy version 0.4.2. zenodo (2019). https://doi.org/10.5281/zenodo.3541386 zhang, y., wang, z., shang, j.: clusterllm: large language models as a guide for text clustering. arxiv preprint arxiv:2305.14871 (2023) zhu, p., lang, q., liu, x.: word embedding of dimensionality reduction for document clustering. in: 2023 35th chinese control and decision conference (ccdc), pp. 4371–4376. ieee (2023) google scholar  download references acknowledgements this project was supported by the brazilian ministry of science, technology and innovations, with resources from law n\(^{\underline{{\textbf {o}}}}\) 8,248, of october 23, 1991, within the scope of ppi-softex, coordinated by softex and published arquitetura cognitiva (phase 3), dou 01245.003479/2024 -10. author information authors and affiliations hub de inteligência artificial e arquiteturas cognitivas (h.iaac), campinas, brazil fillipe dos santos silva, gabriel kenzo kakimoto, julio cesar dos reis & marcelo s. reis artificial intelligence laboratory (recod.ai), campinas, brazil fillipe dos santos silva, gabriel kenzo kakimoto & marcelo s. reis instituto de computação, universidade estadual de campinas (unicamp), campinas, brazil fillipe dos santos silva, julio cesar dos reis & marcelo s. reis faculdade de engenharia mecânica, universidade estadual de campinas (unicamp), campinas, brazil gabriel kenzo kakimoto authors fillipe dos santos silvaview author publications search author on:pubmed google scholar gabriel kenzo kakimotoview author publications search author on:pubmed google scholar julio cesar dos reisview author publications search author on:pubmed google scholar marcelo s. reisview author publications search author on:pubmed google scholar corresponding author correspondence to fillipe dos santos silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper dos santos silva, f., kakimoto, g.k., dos reis, j.c., reis, m.s. (2025). erasmo: leveraging large language models for enhanced clustering segmentation. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_29 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords clustering segmentation transformer-based models tabular data embeddings publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature detection of pathological regions of the gastrointestinal tract in capsule images using efficientnetv2 and yolov8 | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper detection of pathological regions of the gastrointestinal tract in capsule images using efficientnetv2 and yolov8 conference paper first online: 30 january 2025 pp 324–339 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) detection of pathological regions of the gastrointestinal tract in capsule images using efficientnetv2 and yolov8 download book pdf download book epub anderson lopes silva  orcid: orcid.org/0009-0003-7721-07009, hellen guterres frança  orcid: orcid.org/0009-0002-7128-66789, carlos mendes dos santos neto  orcid: orcid.org/0009-0002-1500-28459, alexandre césar pinto pessoa  orcid: orcid.org/0000-0003-4995-89099, darlan bruno pontes quintanilha  orcid: orcid.org/0000-0001-8134-48739, aristófanes corrêa silva  orcid: orcid.org/0000-0003-0423-25149 & … anselmo cardoso de paiva  orcid: orcid.org/0000-0003-4921-06269  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 492 accesses abstract diseases of the gastrointestinal (gi) tract are among the most common pathologies in the world population and are responsible for thousands of deaths every year. this work proposes an automatic method for detecting regions with gi tract abnormalities, intending to reduce the number of lesions missed in wireless capsule endoscopy (wce) video exams by expert endoscopists. by taking advantage of convolutional neural networks (cnns) and yolo detection models, the proposed method not only increases the reliability of pathological detection in wce images, but also sets a new benchmark in this field. our results for binary classification between healthy and pathological images are promising, with an accuracy of 87.8%, precision of 91.6%, recall of 89.1% and f1-score of 90.3%. in addition, the detection model showed an intersection over union (iou) of 31.33% among all the images classified as pathological. the impact of this research is significant, as it provides a method capable of detecting gi tract diseases in wce images and contributing to better clinical decision-making and patient care. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others human visual system consistent model for wireless capsule endoscopy image enhancement and applications article 01 july 2020 combined deep convolutional neural networks for abnormality classification in wireless capsule endoscopy images article 26 march 2025 classification of gastrointestinal diseases of stomach from wce using improved saliency-based method and discriminant features selection article 26 june 2019 1 introduction the gastrointestinal (gi) tract is a tubular system whose main task is digestion. its structure is made up of the mouth, pharynx, esophagus, stomach, small intestine, large intestine, rectum and anal canal. inflammatory diseases associated with the gi tract have a high incidence rate in the world population, especially in developing countries, where colon, liver, and stomach cancer are among the most recurrent diseases [15]. according to the world health organization (who), in 2022, there were approximately 5.1 million cases of gi tract cancers worldwide, and, in the americas region alone, 431,116 deaths were recorded [27]. based on the need to combat the harmful effects of gi tract pathologies, endoscopy is one of the most widely used techniques for analyzing gi tract pathologies, but it is characterized by being a painful and invasive process [12]. in contrast, videos can be taken using wireless capsule endoscopy (wce), a less aggressive and uncomfortable method. during this procedure, the patient swallows a capsule with a light source, a micro-camera, and a signal emitter that travels painlessly throughout the gi tract until it is naturally expelled by the patient’s body. however, as they scan the entire gi tract, the videos acquired by the capsule are around 8 h long. therefore, analyzing this video is an arduous and tedious task, making it possible to miss abnormalities during the examination due to its dependency on the analyst’s constant concentration on the images [4]. for this reason, it is important to use automation to help the specialist analyze the video. from this perspective, machine learning techniques, which consist of using a dataset to develop a computer-aided diagnosis (cad) system, have been used in the analysis of medical images since the 1990s [13]. these techniques are widely used in medical tasks due to their ability to automatically learn discriminative features from raw data and to reduce the need for pre-processing and feature extraction [14]. an example of this type of method is convolutional neural networks (cnns), a computational resource inspired by the workings of the human brain commonly used in the processing and analysis of digital images [26]. another well-known method is yolo (you only look once), which stands out as a real-time object detection algorithm and a single-stage detector that can predict all objects in a single pass through the image. in this way, the adoption of yolo can help in the early detection and diagnosis of gi tract diseases [19]. furthermore, the importance of identifying the region of the pathology lies in enabling a more precise diagnosis, potentially avoiding additional invasive diagnostic procedures [12]. it allows for the comparison of images from subsequent exams to evaluate disease progression or treatment response and informing patients about the exact location of their condition can improve their understanding of the disease and increase treatment adherence. thus, it becomes crucial to enhance the capability of tools to automatically detect and characterize pathologies, increasing the efficiency of cads. therefore, this work aims to introduce detection methodologies for wce images of the gi tract, providing a solid foundation for subsequent research efforts to build upon the automated detection of diseases in wce images. thus, this study developed the application of a method to classify and detect the pathologies using the best cnn architecture experimented and yolo. next, sect. 2 highlights the related work, sect. 3 describes the proposed method, and sect. 4 presents the results, experiments, comparisons, and discussions of the proposed method. 2 related works in recent years, neural networks based on deep learning methods have proven to be a powerful tool for image analysis, including medical imaging tasks [3]. this has led to several advances in the field, including computer-aided classification and localization of pathologies from the gi tract. this section presents some of the work that has been done on this subject. xu et al. (2019) introduced a multi-task anatomy detection convolutional neural network (mt-ad-cnn) designed to evaluate esophagogastroduodenoscopy inspection quality. the model classifies images and distinguishes between informative and non-informative frames, thus displaying detection boxes only on informative frames and reducing false positives. the mt-ad-cnn achieves a mean average precision (map) of 93.74% for the detection task and an accuracy of 98.77% for the classification task. the work by oukdach et al. (2024) presented a new framework that synergizes the strengths of cnns and vision transformers (vits). this approach incorporates an attention mechanism within a cnn to extract local features. experiments performed on the kvasir capsule, a large-scale dataset of wce images [21], showed a promising result of 97% in accuracy, precision, recall, and f1-score. muruganantham and balakrishnan (2022) proposed a method for classifying images with ulcers, bleeding, polyps, or no abnormality using resnet-50 associated with a self-attention mechanism aggregate spatial characteristics globally. the method was also tested on the kvasir-capsule dataset and achieved a classification accuracy of 94.7%. in addition, srivastava et al. (2022) proposed a focal modulation network called focalconvnet. focalconvnet makes use of focal modulation and an added convolutional block to extract features in order to improve detection. the study used 13 classes from anatomy and luminal findings from the kvasir-capsule dataset, obtaining f1-score, recall, and mcc (matthews correlation coefficient) of 67.34%, 63.73%, and 29.74, respectively. as videos are sequences of captured images and the mentioned methods are applied per image (individual video frame), some work [5, 7, 17, 18, 22] do not clearly specify how sampling is done to split the training and test sets. ideally, all video frames for a patient should be in the same set. applying a random sampling of video frames can result in “data leakage” [9], as similar frames (neighbors) from the same patient can end up in both the training and test sets, compromising the results’ reliability. in addition, although there are many published studies to assist in the diagnosis of gi pathologies in wce images, there is a notable lack of studies focused on the detection of the region in which these anomalies are found. to this end, a pipeline was developed to classify abnormal images using the best-performing cnn architecture in the conducted experiments, followed by the detection of the region with the presence of the disease using yolov8 [8], a recent model for detecting objects from images in real-time. this work’s contribution lies in the proposal of an automated method that not only identifies the presence of different types of anomalies in wce images but also indicates the region affected by the pathology, reducing the time and work involved in analyzing videos of wce exams. 3 materials and method this section describes the proposed method. firstly, the image dataset used to train and evaluate the proposed method is determined, and then the best performing cnn architecture is selected to classify the images in the dataset. then, the model that has shown the best performance during the experiments sends the images classified as abnormal for detection. finally, the detection models trained for a specific class by yolov8, make predictions based on the images received. the predicted region is the one with the highest confidence between the model’s predictions. a representation of the process can be seen in fig. 1. fig. 1. proposed method full size image 3.1 dataset table 1. kvasir-capsule images by classfull size table in this work, the experiments were carried out using the kvasir-capsule dataset, which was built by collecting wce scans from a norwegian hospital [21]. currently, kvasir-capsule is the largest and most diverse wce dataset with expert-labeled data and publicly available. it comprises a total of 117 videos collected from wce exams, totaling around 4.7 million frames extracted from these videos. in addition, the dataset features 47.238 frames labeled by specialists, 336\(\times \)336 in size and divided into two categories: anatomical findings, consisting of pylorus, ileocecal valve and ampulla of vater, and luminal findings, composed of angiectasia, blood fresh, blood hematin, erosion, erythema, foreign body, polyp, lymphangiectasia, reduced mucosal view and ulcer, in addition to the normal clear mucosa class, which represents normal images. figure 2 shows the sample image for each class. the division of the dataset, the number of patients, and the availability of the bounding boxes per class can be seen in table 1. fig. 2. sample of labeled images in kvasir-capsule full size image 3.2 classification stage in this stage, we previously implemented and evaluated cnn architectures to choose the model to be used in the method, defined the metrics to be used to evaluate the binary classification models, and trained and tested the dataset with the architecture that obtained the best performance. selection of cnn model. after defining the dataset, the best-performing cnn is selected to classify the images. for this task, cnns widely used in the literature in the field of image classification were chosen. the architectures evaluated in this stage were vgg [20], resnet [6], inception [23], xception [1], efficientnet [24] and efficientnetv2 [25]. the architectures mentioned were used according to their respective original authors and the implementation provided by the keras library [2]. binary classification training and evaluation. after choosing the best architecture to perform the binary classification, the model is evaluated during and after the training process using the following metrics: $$\begin{aligned} \text {accuracy } (acc):(tp + tn) / (tp + tn + fp + fn)\end{aligned}$$ (1) $$\begin{aligned} \text {precision } (pre): tp / (tp + fp)\end{aligned}$$ (2) $$\begin{aligned} \text {recall } (rec): tp / (tp + fn)\end{aligned}$$ (3) $$\begin{aligned} \text {f1-score } (f1): 2 * (pre * rec) / (pre + rec)\end{aligned}$$ (4) where tp and tn represent correctly classified images with pathology and healthy images. in addition, fp represents the healthy images wrongly classified as having abnormalities, and fn means the classifications of pathological images as healthy images. in the context of the proposed method’s objective, accuracy is useful for giving an overview of the model’s performance, but it is misleading in unbalanced datasets, as in the case of kvasir-capsule, where one class is more frequent than the other. thus, precision is also used to ensure that images classified as containing pathologies actually do have pathologies, thus reducing false positives that can overload the region detection stage. in addition, recall is crucial to ensure that as many images with pathologies as possible are identified, minimizing false negatives that could result in undetected pathologies. the f1-score will be the most important metric to evaluate due to its balance between precision and recall, providing a single performance measure that considers both false positives and false negatives. furthermore, although specificity is a useful metric in many contexts, in this work, the main focus is ensuring that all images with pathologies are identified (maximizing recall) and that the identified images contain pathologies (maximizing precision). thus, using the metrics presented, the model selects the images with pathologies and sends them to the detection phase to determine the region where the classified pathology appears. 3.3 detection stage during this stage, the detection models based on yolov8 were trained and the pathological area was predicted based on the model with the highest prediction confidence. yolov8 architecture. yolov8 was unveiled in january 2023 by ultralytics, providing five versions: yolov8n (nano), yolov8s (small), yolov8m (medium), yolov8l (large), and yolov8x (extra-large) [8]. yolov8 provides support for classification, segmentation, and object detection tasks. in this work, yolov8s was used as a model to detect the regions where anomalies appear in the dataset images. the yolov8 architecture can be seen in fig. 3. fig. 3. yolov8 architecture [8] full size image among the differences in yolov8 compared to previous versions is the c2f (partial inter-stage bottleneck with two convolutions) module, which combines high-level features to improve detection accuracy based on contextual information. in addition, yolov8 has an approach that allows each branch to be focused on its task in order to improve the overall accuracy of the model. in the output layer of yolov8, the sigmoid function is used as the activation function to represent the probability of a bounding box containing an object. in addition, the softmax function is used for class probabilities. yolov8 uses the ciou (complete intersection over union) and dfl (distribution focal loss) loss functions for bounding box loss and binary cross-entropy for classification loss. these losses improved object detection performance, especially when dealing with smaller objects [11]. yolov8’s architecture uses a modified backbone cspdarknet53 combined with an sppf layer (spatial pyramid pooling fast), which speeds up the calculation by grouping features into a fixed-size map. furthermore, each convolution has batch normalization and silu activation (sigmoid linear unit), and the head of the architecture is decoupled, so the objectivity, classification, and regression tasks are processed independently [16]. detection model training. after classifying the images with pathologies, the next step is to train the detection models with yolov8. the aim was to train a dedicated model to detect each class so that each model could predict based on the abnormal images received from the previous stage. yolov8s was therefore used to train the detection models. along with accuracy and sensitivity, the following metrics were also used to measure the assertiveness of the predicted regions in relation to the true region with the pathology: average precision (ap): \(\int _{0}^{1} p(r) \, dr \) mean average precision (map): \((\frac{1}{n} \sum _{i=1}^{n} \text {ap}_i )\) intersection over union (iou): \(\frac{\text {intersection area}}{\text {union area}} \) map is a widely used metric in object detection tasks, as it helps calculate average accuracy at various iou thresholds. thus, it provides a comprehensive assessment of the algorithm’s accuracy in object detection. the map calculation considers the ap, the average of the accuracies calculated at various recovery points. the recovery point is where the rate of true positives is calculated in relation to the total number of positive examples. additionally, the iou metric was used to measure how close the predicted bounding boxes are to the real box described in the dataset. this metric is calculated as the area of intersection over the area of the union of the prediction with the true area. it is commonly used to assess the accuracy of object detection. thus, after training the dedicated models for each class, all these metrics were calculated and applied to a test set to evaluate the performance of each model. pathology region detection. after training the dedicated model for each class, the images classified as abnormal are passed through all the detection models to assess the confidence in the region predicted in each image. from the results stored in text files in the yolo format, we will have access to the following prediction information: detected class, x and y coordinates of the center of the bounding box, width, and height of the predicted bounding box, and the prediction confidence. so this step consists of identifying the prediction with the highest confidence and then collecting the coordinates of the predicted bounding box. then, the given coordinates are translated into cartesian format on top of the original image size and the region of intersection between the prediction and the true location of the pathology is stipulated. once all the images have been passed through and the region of interest has been identified, the iou corresponding to all the intersection regions is calculated in order to obtain an overall evaluation of the proposed method. 4 results and discussion this section presents the results achieved using the proposed method for detecting regions containing gi tract pathologies. the experiment carried out, and the impact of the results on the advancement of detection models in gi tract images are also described. 4.1 dataset preparation initially, the dataset was divided by patient, given that the images were taken from wce videos and depict frames of the same patient. this division was then designated to prevent data leakage, the influence of extremely similar images of the same patient being present in the same set (training, validation, or test). as a result, only classes with at least three patients were used, since each set must contain at least one different patient with each pathology. in addition, only classes of luminal findings in which the bounding boxes were also available in the dataset were considered. therefore, based on table 1, the classes used were angiectasia, erosion, foreign body, lymphangiectasia, and ulcer. as a result, there was an imbalance between the number of pathological images belonging to each patient. so, in order to increase the difference in the number of samples in each set, the patient with the most images was sent to the training set, the second to the validation set, the third to the test set, and so on, in that order, until all the patients in the class were distributed. the number of images with pathology belonging to each of the sets can be seen in table 2. table 2. distribution of images by class for each setfull size table in the healthy images used for classification, it was observed that many images were from the same patient. therefore, efforts were made to diversify the number of normal images obtained from each patient, ensuring a balance with the set of abnormal images. thus, 200 images from 13 patients were used for the training set, 80 images from 10 patients for validation, and 30 images from 14 patients for testing, totaling 2600 images for training, 800 images for validation, and 420 images for testing. 4.2 abnormal image classification this section describes the results achieved in the stage of selecting the best-performing cnn architecture. all cnn architectures were trained with 100 epochs and the following hyperparameters: adam [10] optimizer with a learning rate of 0.001, batch size of 32 and early stopping with patience of 25 epochs. moreover, the images were normalized and resized to 224 \(\times \) 224. in order to evaluate the models to determine the best performing one, experiments were carried out on the following architectures: vgg16, resnet50, inceptionv3, xception, efficientnetb0, efficientnetb1, efficientnetb2, efficientnetv2b0, efficientnetv2b1 and efficientnetv2b2. all the architectures were trained with pre-trained weights from imagenet. table 3 shows the results achieved with each architecture and the number of parameters in each model. table 3. comparison of cnn architecturesfull size table although vgg16 has perfect precision, its accuracy, recall and f1-score are low, which indicates that the model makes few positive predictions and is not ideal for applications that require balanced detection. something similar occurs in efficientnetb0, as the model has high accuracy but fails to detect part of the true positives. xception and efficientnetv2b2 presented modest results in all metrics, demonstrating that neither model was well-suited to the proposed task. therefore, resnet50, inceptionv3 and efficientnetv2b1 have good accuracy, but their precisions are relatively low, resulting in moderate f1-score values, 78.7% and 70.9%, respectively. efficientnetb1 and efficientnetb2 presented relatively good and balanced precision and recall values, although efficientnetb2 presented low accuracy compared to the other models. as a result we have 82.7% and 84.5% f1-score, respectively. for these two architectures, the model does not usually miss a positive instance and also does not make many wrong predictions. table 4. efficientnetv2b0 confusion matrixfull size table however, efficientnetv2b0 is the model that presents the best combination of accuracy (87.8%), precision (91.6%), recall (89.1%) and f1-score (90.3%). this indicates that it makes many correct predictions and detects most true positives. in other words, it performs the best correct classification of pathologies among all detections made by the system, thus being the most suitable model to undertake the proposed task of classifying pathological images. table 4 shows the confusion matrix for the classification with efficientnetv2b0. 4.3 detection of regions with anomalies to train the detection models, data augmentation was performed on the images of each class. the operations applied were: vertical and horizontal flip and blur variations with filter size varying from 1\(\,\times \,\)1 to 7\(\,\times \,\)7 pixels. the results of training the dedicated detection models for each class are shown in table 5. table 5. metrics of the models trained with yolov8full size table the map50 returns the average precision of the detections with an iou threshold of 0.5, while the map50-95 returns the results of the map with iou thresholds of 0.5 to 0.95. finally, the iou indicates the extent to which the prediction box actually intersected the region with the anomaly. of particular note were the angiectasia, foreign body and ulcer classes, which achieved 67.3%, 87.2% and 73.1% iou, respectively, demonstrating the models’ ability to detect the majority of pathological regions in their respective class. at the end of training the detection models, the next step is to determine the coordinates of the predicted box with the highest confidence prediction in the images classified as abnormal. thus, in order to evaluate the model’s prediction in relation to the true region with the pathology, the iou was used. the results obtained from the predictions per model and the overall evaluation of all the models can be seen in table 6. table 6. evaluation of detection regionsfull size table to evaluate overall performance, it should be noted that for normal images, which have passed through the classification filter for detection, a 100% true positive value will be considered if there are no predictions for these images since there are no abnormalities to be found. otherwise, if there is any kind of prediction in an image without pathology, a 0% true positive value will be considered, as there are no pathologies to be found in such images. in addition, abnormal images that have been classified as normal automatically count as a 0% hit in the method’s overall performance since the detection models won’t have the chance to predict the pathologies present in these images. some examples of the acquired results can be seen in fig. 4. the red boxes represent the model’s prediction, while the green boxes show the true anomaly region. in addition, the iou value of each prediction is highlighted in the top left corner of the images. although the overall result was not satisfactory, it should be noted that this work represents an initial step in the detection of pathological regions in wce images, especially with regard to the criteria for dividing up the dataset. fig. 4. sample results of pathology region prediction full size image 4.4 discussion the results shown in tables 5 and 7 demonstrate the superiority of the dedicated models for each class compared to the single model for all classes in detecting pathologies. for the angiectasia classification, the dedicated model obtained a precision of 94.6% and a map50 of 79.3%, in contrast to the single model, which showed a precision of only 21.1% and a map50 of 29.9%. although the recall of the single model is slightly higher, the combination of high precision and map50 of the dedicated model indicates a more accurate and reliable identification of angiectasia images. in erosion detection, the dedicated model was also superior, with a precision of 46.6% and a map50 of 16.9%, while the single model showed substantially lower values. table 7. metrics of the model trained for all classes with yolov8full size table for the foreign body class, although both models showed high performances, the dedicated model was still superior in its performance. in the lymphangiectasia and ulcer classes, the dedicated models significantly outperformed the single model in all evaluated metrics. as for the iou, the dedicated models were superior in detecting regions containing anomalies in all classes. especially in the erosion and ulcer classes, where the model trained for all classes obtained insignificant results, while the dedicated models could detect more than half of the pathological regions. these results highlight that the approach of training dedicated models for each class not only improves efficiency in detecting specific pathologies, but also reduces the false positive rate and increases confidence in the detections made. therefore, using dedicated models for pathology detection in wce is a more effective solution, providing more reliable results than a single model trained for all classes. in addition, in table 8 a comparison was made with the binary classification methods of works related to the study carried out in this paper, all using kvasir-capsule. each study contributed different methodologies and made significant advances in the field. however, based on the division criteria adopted by the works that relate to the classification stage carried out in this study, the proposed method still showed good performance compared to state-of-the-art models. in comparison, muruganantham et al. [17] achieved 94.70% accuracy, fonseca et al. [5] obtained 97.00% precision, oukdach et al. [18] achieved f1-score of 97.00% and jain et al. [7] achieved 98.00% across all metrics with a hybrid network approach and residual attention, all using the same random division of data. table 8. comparison of binary classification works using kvasir-capsulefull size table it is emphasized that the patient splitting criterion adopted in the proposed method is crucial to avoid data leakage between training and testing, ensuring more reliable results when evaluating the model’s performance. this method ensures the integrity of the results by mitigating potential biases introduced by less rigorous data-splitting practices. it is worth noting that during the search for research related to this work, no other work was identified that utilized the kvasir-capsule dataset and proposed or evaluated a method to classify abnormalities and detect the region in which the pathology is found. as such, this work aims to propose advances in the area of pathology detection in wce images of the gi tract, presenting some baseline results for future work in the area. 5 conclusion in conclusion, early diagnosis is essential so that the necessary treatment can be carried out, but the detection of gi tract pathologies in wce images remains a challenging task due to the complex nature of the gastrointestinal system and the low quality of the images generated by this exam. the proposed method proved to be a promising tool for detecting gi tract pathologies present in wce images. the f1-score of 90% obtained in the classification stage indicates that the efficientnetv2b0 model is able to help in the early identification of pathological images, allowing the detection models dedicated to each class to work on the images that present the diseases. thus, the detection showed the ability to find some of the pathological regions based on the strategy of treating the prediction region as the one with the highest confidence among the trained detection models. as future work, in order to build on the results obtained in this study, an expansion of the proposed method is intended, with pre-processing methods to improve the models’ ability to identify pathologies. in addition, it is proposed to investigate multi-class classification methods to serve as a step between the first classification and detection, to use the best detection model for the frame analyzed. it is also planned to test the method on other datasets in order to validate the robustness of the proposed method further. therefore, it is understood that this work can help health specialists in the task of identifying pathologies in wce images, potentially reducing the workload on doctors and improving patient outcomes through early and accurate diagnosis. references chollet, f.: xception: deep learning with depthwise separable convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1251–1258 (2017) google scholar  chollet, f.: keras (2015). https://keras.io costa, c.l., lima, d.a., zorzo barcelos, c.a., travençolo, b.a.: ensemble architectures and efficient fusion techniques for convolutional neural networks: an analysis on resource optimization strategies. in: brazilian conference on intelligent systems, pp. 107–121. springer (2023) google scholar  delagah, b., hassanpour, h., et al.: feature extraction for polyp detection in wireless capsule endoscopy video frames. j. healthc. eng 2023, 6076514 (2023) article  google scholar  fonseca, f., nunes, b., salgado, m., cunha, a.: abnormality classification in small datasets of capsule endoscopy images. procedia comput. sci. 196, 469–476 (2022) article  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  jain, s., seal, a., ojha, a.: a hybrid convolutional neural network with meta feature learning for abnormality detection in wireless capsule endoscopy images. arxiv preprint arxiv:2207.09769 (2022) jocher, g., chaurasia, a., qiu, j.: yolo by ultralytics (2023). https://github.com/ultralytics/ultralytics. accessed 28 feb 2023 kaufman, s., rosset, s., perlich, c., stitelman, o.: leakage in data mining: formulation, detection, and avoidance. acm trans. knowl. discov. data 6(4) (2012). https://doi.org/10.1145/2382577.2382579 kingma, d.p., ba, j.: adam: a method for stochastic optimization. arxiv preprint arxiv:1412.6980 (2014) li, x., et al.: generalized focal loss: learning qualified and distributed bounding boxes for dense object detection. adv. neural. inf. process. syst. 33, 21002–21012 (2020) google scholar  lima, d.l.s., pessoa, a.c.p., de paiva, a.c., da silva cunha, a.m.t., júnior, g.b., de almeida, j.d.s.: classification of video capsule endoscopy images using visual transformers. in: 2022 ieee-embs international conference on biomedical and health informatics (bhi), pp. 1–4. ieee (2022) google scholar  litjens, g., et al.: a survey on deep learning in medical image analysis. med. image anal. 42, 60–88 (2017) article  math  google scholar  mendes, a.c., pessoa, a.c.p., de paiva, a.c.: multi-label classification of pathologies in chest radiograph images using densenet. in: brazilian conference on intelligent systems, pp. 167–180. springer (2023) google scholar  milivojevic, v., milosavljevic, t.: burden of gastroduodenal diseases from the global perspective. curr. treatment opt. gastroenterol. 18, 148–157 (2020) article  math  google scholar  mmyolo: yolov8 by mmyolo (2023). https://github.com/open-mmlab/mmyolo/tree/main/configs/yolov8. accessed 15 mar 2024 muruganantham, p., balakrishnan, s.m.: attention aware deep learning model for wireless capsule endoscopy lesion classification and localization. j. med. biol. eng. 42(2), 157–168 (2022) article  google scholar  oukdach, y., kerkaou, z., el ansari, m., koutti, l., fouad el ouafdi, a., de lange, t.: vitca-ndet: a framework for disease detection in video capsule endoscopy images using a vision transformer and convolutional neural network with a specific attention mechanism. multimedia tools appl. 1–20 (2024) google scholar  qureshi, r., et al.: a comprehensive systematic review of yolo for medical object detection (2018 to 2023). authorea preprints (2023) google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556 (2014) smedsrud, p.h., et al.: kvasir-capsule, a video capsule endoscopy dataset. sci. data 8(1), 142 (2021) article  google scholar  srivastava, a., tomar, n.k., bagci, u., jha, d.: video capsule endoscopy classification using focal modulation guided convolutional neural network. in: 2022 ieee 35th international symposium on computer-based medical systems (cbms), pp. 323–328. ieee (2022) google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: international conference on machine learning, pp. 6105–6114. pmlr (2019) google scholar  tan, m., le, q.: efficientnetv2: smaller models and faster training. in: international conference on machine learning, pp. 10096–10106. pmlr (2021) google scholar  valentim, n.a., dorça, f.a., asnis, v.p., elias, n.c.: the artificial intelligence as a technological resource in the application of tasks for the development of joint attention in children with autism. in: brazilian conference on intelligent systems, pp. 306–320. springer (2023) google scholar  world health organization: the global cancer observatory (2024). https://gco.iarc.fr/en. accessed 15 mar 2024 xu, z., et al.: upper gastrointestinal anatomy detection with multi-task convolutional neural networks. healthcare technol. lett. 6(6), 176–180 (2019) article  math  google scholar  download references acknowledgments the authors acknowledge the coordenação de aperfeiçoamento de pessoal de nível superior (capes), brazil finance code 001, conselho nacional de desenvolvimento científico e tecnológico (cnpq), brazil, and fundação de amparo à pesquisa desenvolvimento científico e tecnológico do maranhão (fapema) (brazil), empresa brasileira de serviços hospitalares (ebserh) brazil (grant number 409593/2021-4) for the financial support. author information authors and affiliations núcleo de computação aplicada, universidade federal do maranhão (ufma), caixa postal, são luís, ma, 65.085-580, brazil anderson lopes silva, hellen guterres frança, carlos mendes dos santos neto, alexandre césar pinto pessoa, darlan bruno pontes quintanilha, aristófanes corrêa silva & anselmo cardoso de paiva authors anderson lopes silvaview author publications search author on:pubmed google scholar hellen guterres françaview author publications search author on:pubmed google scholar carlos mendes dos santos netoview author publications search author on:pubmed google scholar alexandre césar pinto pessoaview author publications search author on:pubmed google scholar darlan bruno pontes quintanilhaview author publications search author on:pubmed google scholar aristófanes corrêa silvaview author publications search author on:pubmed google scholar anselmo cardoso de paivaview author publications search author on:pubmed google scholar corresponding author correspondence to anderson lopes silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, a.l. et al. (2025). detection of pathological regions of the gastrointestinal tract in capsule images using efficientnetv2 and yolov8. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_23 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords wireless capsule endoscopy efficientnet yolo detection of gastrointestinal diseases publish with us policies and ethics profiles alexandre césar pinto pessoa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature enhancing graph data quality by leveraging heterogeneous node features and embeddings | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper enhancing graph data quality by leveraging heterogeneous node features and embeddings conference paper first online: 30 january 2025 pp 383–398 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) enhancing graph data quality by leveraging heterogeneous node features and embeddings download book pdf download book epub silvio fernando angonese  orcid: orcid.org/0009-0001-6441-43209 & renata galante  orcid: orcid.org/0000-0003-3589-16199  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 478 accesses abstract heterogeneous graphs are important data sources due to their rich representation of knowledge, primarily based on node features and relationships. it is common for these graphs to have significant data gaps, particularly in the nodes. graph neural networks are state-of-the-art solutions that achieve excellent results by extracting information based on node relationships. however, they suffer from severe limitations when there is no available information in the graph elements, weakening their representation. this paper proposes the specifications and an algorithm to process different types of node features, such as text, images, and subgraphs, generating both single and composition embeddings. to evaluate the effectiveness of the proposed algorithm, experiments were conducted to generate the features and their respective node embeddings in a heterogeneous graph. the achieved performance was measured using the average of accuracy, f1-score, and their standard deviations based on the recommender system tasks applied to the embeddings generated in the experiments. we can highlight the performance achievement in the experiments as the node classification task, using the composition of aggregated features with metapaths embedding, which achieved an f1-score of 83.66% overcoming the 60.70% achieved by the approach without embeddings. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a tutorial of graph representation chapter © 2019 data augmentation for graph convolutional network on semi-supervised classification chapter © 2021 a gated graph attention network based on dual graph convolution for node embedding article 22 march 2023 1 introduction with the exponential growth of data, the challenge lies in acquiring raw data and transforming it into valuable information. although the data exists, extracting and utilizing it is not a simple task. complex business cases, for example, can be effectively modeled as graphs, which may contain rich data about the entities they represent. enhancing learning from graph representations signals a viable path to uncovering the hidden knowledge in raw data [6]. deep learning and machine learning excel at discovering hidden knowledge and characteristics, enabling the use of specific techniques to generate rich data from each graph node. the semantically rich data generated can then be used in various downstream applications, such as recommender systems (recsys) [1, 17]. however, an important restriction is associated with the common low level of information in the graphs. if graph elements, especially nodes, have more data attached, the representation of knowledge will increase and enhance the subjacent performance of applications. node embedding is a technique that maps graph nodes to low-dimensional vectors, preserving the graph structure and node features, representing the nodes [16]. thus, node embeddings are an excellent option for collecting and aggregating data from nodes, resulting in high-quality node representation. some work [10, 18] introduces the generation of embeddings based on the neighboring nodes, increasing their expressiveness and recommendation performance. meanwhile, the works of [17, 19] evaluate the heterogeneity of data types embeddings, opening up a new perspective. researches [7, 16] propose a new approach for generating embeddings from metapaths, capturing semantics based on node relationships. vision gnn [11] and superpixel image classification [3] are examples of the few works where the graph is an image and the nodes are parts of it, demonstrating the feasibility of having images as nodes. none of the related work addresses the use of heterogeneous graphs with heterogeneous data types, such as texts, images, and subgraphs. this paper proposes the specifications and a new algorithm that leverages deep learning techniques to extract information from images. also, the feature generation is provided by specialized autoencoders, which are machine learning models designed to generate embeddings by mapping high-dimensional data to a compact and meaningful latent representation, preserving the essential data from the node features. the experiment results clearly demonstrate that the performance of embeddings, particularly when a composition of different types of embeddings is created, can yield superior results compared to evaluations conducted without them. in general, this paper contributes to providing an unexplored way of generating information and embeddings on nodes in heterogeneous graphs, through the definition of a modeling approach using texts, images, and subgraphs as representations of the graph nodes. in previous work aghe [2], we proposed an approach for generating and processing heterogeneous embeddings. in this paper, we specify each step of the approach, with the main contributions as follows: a) specification of procedures for processing and generating different types of embedding; b) proposes an algorithm based on procedures for processing and generating heterogeneous embeddings; c) introduce composition embeddings as an alternative to generate high semantic nodes and present the evaluation of their performances achieved by the proposed algorithm; d) share the public datasets, including the final recommended graph in json and csv formats, with the research community, particularly those focused on heterogeneous graphs and recsys. the remainder of this paper is organized as follows: sect. 2 conceptualizes the background techniques applied in this paper. section 3 describes the related work. section 4 presents the approach aghe. section 5 defines the specification of the procedures that serve as the foundation for the proposed algorithms shown in this section. section 6 conducts some experiments and evaluates the results achieved while sect. 7 exposes the conclusions and future works. 2 background the aim of this section is to present the machine learning and deep learning concepts and techniques adopted in this paper. heterogeneous graph and embeddings. in heterogeneous graphs the nodes and edges can be of different types, e.g., a person group represented by a heterogeneous graph can have nodes with different types. the bipartite graph is a special, commonly used type of heterogeneous graph, where edges exist between nodes of two different types. thus, multiple types of nodes and different relationships contain comprehensive information and rich semantics [8]. a recurring problem in representation learning in heterogeneous graphs is over-smoothing issue. this implies that, after too many aggregations of the features in a graph, the node embeddings start to converge to nearly the same or even the very same value. losing the capacity to discriminate between the different types of nodes and their various characteristics. during the aggregation process, if the neighboring nodes have very similar features or if the aggregation process is repeated many times, the local differences between the nodes are smoothed out, leading to a homogenization of the features. two simple techniques used to minimize over-smoothing are discarding similar features from nodes during the embedding generation step and removing random edges during each training epoch [12]. embedding captures the graph topology, node features, node-to-node relationships, and other relevant information about graphs, subgraphs, and nodes. hence, embedding represents the nodes, and the similarity between node embeddings indicates their similarity in the graph [10, 14]. metapath2vec. it is a heterogeneous graph embedding model that formalizes metapath based on random walk to construct the heterogeneous neighborhood of a node and then leverages a heterogeneous skip-gram model to perform node embedding. maximizing the probability of preserving both the structures and semantics of a given heterogeneous network, being able to learn desirable node representations in heterogeneous networks [7]. autoencoders. they are useful for incorporating structural graph information from nodes, providing specific implementations. autoencoder (ae) is a kind of neural network architecture that imposes a bottleneck on the network that forces a compressed knowledge representation of the original input. if the input resources were independent of each other, the compression and subsequent reconstruction would be a very difficult task. thus, aes are neural networks that aim to copy their input to their output, compressing the input in a latent space representation, called encoder \(h = f(x)\), and after that, rebuilding the output through this representation, called decoder \(r = g(h)\) [5]. autoencoder may be modified or combined to form new models for various applications such as generative models, classification, clustering, anomaly detection, recommendation, dimensionality reduction, and capture information [5]. 3 related work heterogeneous graph can be traced back to generate data embedding from node features based on random walk approach citing representation learning on graphs [10, 18] improving the node expressivity. more close to the aims of our proposal is [19] which defines of heterogeneous graph neural network with the processing of embedding. the survey graph neural networks in recommender systems [17] shows gnns have been widely used in downstream applications essentially because graph structure and gnn have superiority in graph representation learning, citing graphsage [10] as an important work regarding generating node embedding from node feature information. metapath2vec [7] captures the structure of heterogeneous graph, guiding random walks to generate sequences of heterogeneous nodes with rich semantics. hence, metapath plays an important role in this paper capturing vital information by leveraging the relationships among heterogeneous nodes, transforming it into a form of node embedding. vision gnn [11] and superpixel image classification [3] works are other sources of inspiration that illustrate image representation in the form of a graph. in this context, each node corresponds to a distinct part of the same image, implying that every node encapsulates an image. adopting a similar conceptualization, we can extend this idea to employ an image for representing node content. heterogeneous graph can be integrated with some applications. however, one usually needs to carefully consider two factors: the first is how to construct heterogeneous graph for a specific application, and the second is what information or domain knowledge should be incorporated into a heterogeneous graph to ultimately benefit the application [16]. in the recsys, the interaction between the user and items can be naturally modeled as a heterogeneous graph with two types of nodes. thus, the application of heterogeneous graph embedding to recsys constitutes an important research area, as highlighted in the survey on heterogeneous graph embedding [16]. hence, our proposal uses recommendations as an assessment to validate the value of performance by employing various types of node embeddings from the heterogeneous graph. 4 aghe approach for generating enhanced heterogeneous embeddings from heterogeneous graphs this section presents the approach for generating enhanced heterogeneous embeddings from heterogeneous graphs (aghe) [2] shown in fig. 1. aghe generates heterogeneous embeddings through the processing of texts, images, and subgraphs represented in the nodes of heterogeneous graphs, such as the following steps: 1. graph creation generates the heterogeneous graph along with all its components, such as nodes, edges, and node features. this step is critical because all other steps and results depend on it; 2. generating text node embeddings is the process of creating node embeddings from their corresponding node features or extracted from images embedded in the nodes; 3. metapath and aggregated node embeddings generates of aggregated feature embeddings using the random walks approach. this includes defining metapaths that represent the business rules through the relationships among the nodes, followed by the generation of their embeddings using the metapath2vec algorithm; 4. graph enhancement with recsys tasks represents the experiments conducted in this paper, aiming to predict the type of nodes, predict some links, and cluster the nodes based on the heterogeneous graph generated in the first step; 5. rebuilding the graph involves incorporating the generated embeddings and predictions saved into the graph nodes. fig. 1. steps of aghe approach for generating enhanced heterogeneous embeddings from heterogeneous graphs. full size image in the following sections, we describe the main contributions of this paper, specifying the key procedures to be performed at each step, serving as the foundation for building the proposed algorithm. 5 specification for enhancing graph data quality this section builds the aghe approach [2] defining the procedures to be used in the proposed algorithms 1 and 2 as highlighted in fig. 1, where each of them is described as follows. 5.1 graph creation graph creation is the first step and there are two parts, the first one regards adding nodes and defining their types, characterizing a heterogeneous graph. subsequently, node features should be attached using a set of short descriptions or simple words. in this context, edge features are not considered as part of the heterogeneous graph in this paper. the second part involves mapping relationships between nodes by adding edges to the heterogeneous graph, which enables navigation between the nodes. if the node type is image, then another important operation is needed that involves uploading the node image from an image database provided by the application that understands the graph. as a result, the heterogeneous graph is created and ready to be used in subsequent steps. figure 2(a) shows a heterogeneous graph \(\mathcal{h}\mathcal{g}(v, e)\) where v is a set of nodes, and e is a set of edges, which the node \(v \in v\) can have different types of content with various data type features, such as text, image, and subgraph. figure 2(b) shows the same graph simplified with heterogeneous data features embedded into the nodes, after the graph creation is done. fig. 2. heterogeneous graph within heterogeneous data types. full size image 5.2 embeddings creation and rebuilding the graph after the graph is ready for use, the following steps describe the creation of embeddings from the processing of heterogeneous features, detailing the techniques used in the proposed algorithm: 1. specialized autoencoder for processing image feature data type nodes of the type image are processed in two steps. first, the convolutional neural network (cnn) classifies the image, and second, extracts the characteristics from the image. both results, class, and characteristics are merged and saved into the nodes as data text features. cnns can be represented like the approximation function \(f^{*}: \mathbb {r}^{\alpha \times \beta \times \gamma } \rightarrow \{1,...,c\}\) that takes as input an image from unknow distribution \(\mathbb {r}^{\alpha \times \beta \times \gamma }\) where \(\alpha \times \beta \) are pixels of image and \(\gamma \) is the number of color bands and determines which of class the image is from a set of classes \(\{1,...,c\}\). our proposed algorithm uses the resnet50 model, a cnn with 50 layers and imagenet weights, to implement the specialized autoencoder for extracting classes and characteristics from nodes where the types are images. the extracted data was saved into the respective node feature as a text vector; 2. specialized autoencoder for generating aggregated embedding the text features of nodes are captured on k interaction over their neighbors and combined with the information embedding on the before \(k-1\) interaction. the approach has two phases, message-passing and readout [9, 13]. the message-passing phase or propagation step runs for t time steps and contains two subfunctions: message function \(m_{t}\) and a node update function \(u_{t}\). during the message passing phase, hidden states \(h_{v}^{t}\) at each node in the graph are updated based on messages \(m_{v}^{t + 1}\) according to: $$\begin{aligned} \begin{aligned} m_{v}^{t + 1} = \sum _{w \in n_{(v)}} m_{t}(h_{v}^{t},h_{w}^{t}), \ \ \ \ h_{v}^{t + 1} = u_{t}(h_{v}^{t},m_{v}^{t + 1}), \end{aligned} \end{aligned}$$ (1) where the sum \(n_{(v)}\) denotes the neighbors of node v in graph g, \(h_{w}^{t}\) node embedding from node v to w, and function \(u_{t}\) updates the hidden states \(h_{v}^{t}\). the readout phase uses function r to compute an embedding vector representation for the entire graph as follows: $$\begin{aligned} \hat{y} = r(\{ h_{v}^{t} \mid v \in g \}), \end{aligned}$$ (2) where t denotes the total time steps. the message-passing phase in our algorithm was implemented through random walks, which accessed k neighboring nodes, extracted their features, and aggregated them with the local node. in an attempt to avoid the over-smoothing issue in the first layer, the node features were saved into a data structure of type set, removing duplicate features. the readout phase in our proposed algorithm used a word2vec model to generate the vector node embeddings from the node features produced by the message-passing phase; 3. specialized autoencoder for generating metapaths embedding the first step of the process is to read the original graph as input, and through the metapath2vec algorithm using the pre-defined metapaths, within a sequence of nodes dependent on their node type, walking transversely in the graph captures the semantics between the nodes. the information of nodes is saved into the latent vector space represented by the function \(f: v \rightarrow \mathbb {r}^{d}\) where v is the node embedding, and \(\mathbb {r}\) is the vector space. the specific nodes of the data type graph already have an adjacency matrix linked, like a subgraph embedded, where the metapath2vec exploits it and generates the corresponding metapath embedding [15]. metapath2vec uses random walk to generate the metapaths embeddings, where skip-gram or node2vec model maintains the proximity of node v and its neighbors in the random walk sequences, and heterogeneous graph \(\mathcal{h}\mathcal{g} = (v,e,t)\) is considered, where \(t_{v}\) represents the set of node types and \(\mid t_{v} \mid \ > \ 1\) maximizing the probability of having heterogeneous context \(n_{t}(v),t \in t_{v}\) given a node v: $$\begin{aligned} arg \ \underset{\theta }{max}\ \sum _{v \in v} \sum _{t \in t_{v}} \sum _{c_{t} \in n_{t}(v)} \log p(c_{t} \mid v;\theta ), \end{aligned}$$ (3) where \(n_{t}(v)\) denotes \(v's\) neighbourhood with the \(t^{th}\) type of nodes and \(p(c_{t} \mid v;\theta )\) is defined as a softmax function, and $$\begin{aligned} p(c_{t} \mid v;\theta ) = \frac{e^{x_{ct} \cdot x_{v}}}{\sum _{u \in v} {e^{x_{u} \cdot x_{v}}}}, \end{aligned}$$ (4) where \(x_{v}\) is the \(v^{th}\) row of x, representing the embedding vector for node v [7]. the proposed algorithm implements the metapath2vec algorithm by performing walks on the heterogeneous graph based on a set of predefined metapaths. the hyperparameters “length” and “walks” define the length of each random walk and the number of random walks to generate from each node, respectively. by tracking valid walks based on node types, a node2vec model is built to generate the metapath vector embeddings for each node; 4. consolidating data embedding and rebuilding the graph within node features and embeddings each autoencoder \(a:\mathbb {r}^{n} \rightarrow \mathbb {r}^{p}\) generates its data vectors output that is used as input per the decoder \(b:\mathbb {r}^{p} \rightarrow \mathbb {r}^{n}\) to rebuild the entire graph, associating the nodes with their data embedding generated, that satisfy $$\begin{aligned} arg \ min_{a,b} \ e[\vardelta (x,b \circ a(x)], \end{aligned}$$ (5) where e is the expectation over the distribution of x, \(\vardelta \) is the reconstruction loss function, which measures the distance between the output of the decoder and the input, and a and b are neural networks [4, 5]; based on the data saved during the entire process, the proposed algorithm rebuilds the heterogeneous graph with the original nodes and edges, adding the features generated, vector embeddings, and prediction information to the respective nodes and edges. 5.3 algorithms the algorithm 1 aims to process heterogeneous graphs with features of different data types as input, according to the procedures described in sect. 5.2. it produces node representations in latent vector spaces, consolidating various data type features into a unified format and generating the respective node embeddings. the algorithm takes a heterogeneous graph \(\mathcal{h}\mathcal{g}\) as input data, and it consists of several main blocks. lines 1 to 3 locate the subgraphs in the \(\mathcal{h}\mathcal{g}\) and call itself to solve the generation of features and embeddings. lines 4 to 7 iterate over the nodes, uploading the images and extracting their features using a specialized image autoencoder based on a resnet50 cnn model with imagenet weights. line 8 generates node embeddings from node features using specialized text autoencoder with word2vec algorithm. lines 9 and 10 iterate over the set of hyperparameter metapaths m, generating metapath embeddings into the local node from its neighbors according to the metapath2vec algorithm, using node2vec model to create the respective embeddings. lines 11 to 15 generate aggregated embeddings from direct neighbors and create compositions of embeddings, such as features with metapaths and aggregated features with metapaths. thus, \(\mathcal{h}\mathcal{g}\) has all the features and embeddings returned in line 17, upon completion of the execution of algorithm 1. algorithm 1 delivers a heterogeneous graph \(\mathcal{h}\mathcal{g}\) with vector embeddings \(v_{h}\), where \(v_{h} \in \{features, aggregated, metapaths, features+metapaths, aggregated+metapaths\}\). thus, the next step is to evaluate the performance of the downstream application using \(\mathcal{h}\mathcal{g}\) with embeddings \(v_{h}\) to determine if it achieves better results compared to without node embeddings. algorithm 2 implements recsys tasks as an example of downstream application, where lines 1 to 5 compute the recsys tasks based on the set h of embeddings. in line 3, the link prediction task can define a target node \(v_{v}\) to predict the new edges using cosine-similarity as the applied technique. in line 4, the node classification task has hyperparameters iterations = 5 and k-fold cross-validation = 10 to calculate performance based on the average metrics and evaluate their standard deviations. line 5 calls the node clustering task, defining 3 clusters using the kmeans algorithm. the generation of json and csv files aims to provide a new public dataset with the raw data for the research community. the time complexity of the algorithms is determined by the loops that iterate over the graph nodes, which is o(n) where n is the number of nodes in the graph in the worst case, \(o(m \times n)\) where m is the number of metapaths, and \(o(n)^{2}\) generated in the nested loops for generating embeddings composed of features neighbors of each node, which dominate over the other complexities. therefore, the total time complexity of the proposed algorithms are polynomial \(o(n)^{2}\) in the worst case. 6 experiments this section presents the experiments aimed at applying recsys tasks to the graph with and without embeddings, and evaluating the resulting metrics in both cases. special focus is given to the composition of embeddings, such as features+metapaths and aggregated+metapaths, as introduced in this paper. the experiments were guided by the methodology based on the approach for generating enhanced heterogeneous features and embeddings using the algorithms 1 and 2. figure 3(a) shows the main pipeline of the methodology used to execute the entire set of experiments, where they are guided by the follows steps: algorithm 1 genprochetembedding: generating and processing heterogeneous data type features and embeddings. full size image algorithm 2 recsys: running recsys tasks over the heterogeneous graph. full size image 1. graph generation responsible for creating heterogeneous graphs, including nodes, edges, and node features when available; 2. embeddings generation process of creating embeddings, where features are generated from text node features; aggregated embeddings are generated from the text node features of neighboring nodes; metapath embeddings are generated from the defined set of metapaths; features+metapaths and aggregated+metapaths are compositions of those embeddings. 3. recsys and performance metrics iterates over the set of each embedding, applying recsys tasks to each type of embedding and generating respective performance metrics. in essence, the experiments involve the generation of features and embeddings from nodes, and the validation of recsys performance metrics based on the embeddings generated. to evaluate the generalization capability and robustness of xgboost models used in the experiments, performance accuracy and f1-score metrics should be calculated using 10 stratified k-folds and the average of 5 iterations, along with their respective standard deviations. 6.1 heterogeneous graph data model figure 3(b) defines the heterogeneous graph employed in the experiments, containing features of different data types, such as person, car, and pet. each node has its own heterogeneous features, which may include texts, images (e.g., cars and pets), or subgraphs, such as the family of mary embedded into the mary node, each with its corresponding embeddings. fig. 3. methodology and heterogeneous graph within heterogeneous data types model. full size image based on the scope of experiments and the use case shown in fig. 3(b), the graph data model was defined according to table 1. the entire heterogeneous graph used in the experiments is available at https://github.com/silviofernandoangonese/datasets/blob/main/experiments_initial_het_graph.json. table 1. heterogeneous graph data model used in the experiments.full size table 6.2 execution of experiments the experiments were guided according to the methodology shown in fig. 3(a) and the pre-definition of the metapaths used in the experiments, which were \(\mathcal {m}\ \leftarrow \{(car, person), (pet, person), (car, person, person), (pet, person, pet), (person, person, car), (person, person, pet), (person, person, person)\}\). these 13 metapaths represent the interactions between different types of nodes, providing the business semantics for the respective node embeddings. the second step was to generate single and composed node embeddings, which were defined as features, aggregated features, metapaths, a composition of features and metapaths, and a composition of aggregated features and metapaths, which guided the entire experiments and analysis shown in table 2. the third and fourth methodological steps aim to validate the graph data quality by applying link prediction, node classification, and node clustering tasks to all the previously defined types of embeddings. link prediction was calculated using cosine similarity with a threshold of 80% based on the target node. the algorithm supports selecting a specific node or all the nodes as a target. for node classification, the node type was used as the class. link prediction from “no embeddings” was calculated using the jaccard algorithm based on the intersection and union set operation \(j(a, b) = \frac{|a \cap b|}{|a \cup b|}\). for node clustering, we pre-defined three clusters c0, c1, c2. “no embeddings” clusters were calculated using the louvain algorithm based on the nodes community where nodes without community identified have no cluster assigned. the experiments results are collected and presented in table 2, where each column represents the following: “type node embeddings” indicating the type of generated embeddings, using an ablation perspective, where the experiments aim to identify which embedding has the greatest impact on the performance of the models; “prediction links” the count of links predicted by recsys link prediction task; “classification avg acc and f1-score” are average of accuracy and f1-score metrics, and “stds” are the standard deviation from each average of metrics respectively, illustrating the overall accuracy of the entire node graph classification; “cor” is the count of correctly predicted links applied to the final model over the entire graph; and “inc” is the count of incorrectly predicted links. columns “clusters c0 c1 c2” display the count of nodes clustered by the recsys task. table 2. performance metrics achieved from different types of node embeddings.full size table 6.3 evaluation of results the assumption that enriching the heterogeneous graph within heterogeneous embeddings, generated from processing the available data within the graph, could enhance the performance of downstream applications, was validated. table 2 illustrates the evolution of the performance starting with features embeddings and progressing to the best performance achieved by the composition of aggregated+metapaths node embeddings, as indicated by the average of accuracy and f1-score metrics. link prediction count using features embedding is so high, it can indicate data homogeneity with a lack of distinctive features. although it varies based on certain factors, predictions from metapath and features+metapaths may be deemed more reliable. node clustering already reveals similar cluster distributions independent of the embedding used, except aggregated embedding. the best results were achieved with the composition aggregated+metapaths embeddings, where the results indicate that combining aggregated features from neighbors and metapaths embeddings led to better performance in all tested recsys tasks. the experiments reveal that the best average accuracy was 71.90% with a 1.98% standard deviation, which means the accuracy could range from 69.92% to 73.88%. the average f1-score is already at 83.66% with a 1.95% of standard deviation, thus the f1-score can vary between 81.68% and 85.61%. a higher standard deviation indicates a greater spread of results, suggesting that the model is more sensitive to variations in input data or other training conditions. on the other hand, a lower standard deviation indicates greater consistency in results, which may indicate that the model is more stable and robust. this suggests that combining information from node content with information from semantics defined by features and structural relationships defined by metapaths can lead to more powerful graph node representation. the final recommended heterogeneous graph json file is available at https://github.com/silviofernandoangonese/datasets/blob/main/experiments_final_recommended_het_graph.json and csv file at https://github.com/silviofernandoangonese/datasets/blob/main/experiments_final_recommended_het_graph.csv. the failure cases were due to a high number of links predicted from the features embedding, specifically 866 out of 872 nodes. this result is not acceptable, indicating the need for a deeper analysis of the model to understand the reason for the discrepancy in values. additionally, another topic related to the features embedding is the smallest cluster, c0, which contains only 1 node. this raises an important question: why is there only 1 node in this cluster, and what does it mean? future investigations should explore this anomaly to enhance our understanding of the clustering behavior and improve the accuracy model. 7 conclusion this paper proposed an algorithm based on the aghe approach for generating enhanced heterogeneous embeddings from heterogeneous graphs, enhancing the graph as a dataset for downstream applications. the performance achieved by the experiments conducted, especially when we compare “no embeddings” with “aggregated+metapaths” embeddings demonstrates how the proposed algorithm effectively collaborates with the data enhancements in heterogeneous graph. represented by the recsys, which was used as a downstream application reference in this paper. developing effective and efficient graph analytics from information embedding, can greatly help to better understand complex graphs, and provide innovative solutions for data models. based on the obtained results, we believe that the conducted studies can open the doors for its use in different downstream applications as demonstrated. some specific evaluations can be achieved, where the choice of appropriate embeddings plays a crucial role in the performance of downstream tasks. the results indicate that a one-size-fits-all embedding approach is not necessarily the best for all tasks and datasets. an important lesson learned from the experiments is the significance of exploring a variety of embedding generation techniques and considering the unique characteristics of the data and tasks at hand. combining information from different sources, such as node features and structural relationships defined by metapaths, can lead to more comprehensive and informative node representations in the graph. this highlights the importance of exploring hybrid approaches. future works include: aggregating edges data features to enhance the nodes data embedding; evaluating the performance of recommendations using a tabulated dataset and the same dataset modeled as a heterogeneous graph with heterogeneous embeddings; automatic detection of subgraphs and embedded it into the interconnected node; evaluation of the impact of the embeddings vectors elements normalization; references alslaity, a., tran, t.: towards persuasive recommender systems. in: 2019 ieee 2nd international conference on information and computer technologies (icict) on proceedings, pp. 143–148. publisher (2019) google scholar  angonese, s. f., galante, r.: aghe: approach for generating enhanced heterogeneous embeddings from heterogeneous graphs. in: 2024: proceedings of the 51st integrated software and hardware seminar (semish) on proceedings, pp. 252–263. publisher (2024) google scholar  avelar, p.h.c., tavares, a.r., da silveira, t.l.t., jung, c.r., lamb, l.c.: superpixel image classification with graph attention networks. in: 2020 33rd sibgrapi conference on graphics, patterns and images (sibgrapi), pp. 203-209. publisher (2020) google scholar  baldi, p.: autoencoders, unsupervised learning and deep architectures. in: proceedings of the 2011 international conference on unsupervised and transfer learning workshop volume 27 on proceedings, pp. 37-50. publisher (2011) google scholar  bank, d., koenigstein, n., giryes, r.: autoencoders. publisher (2021) google scholar  barret, n., gauquier, a., law, j.j., manolescu, i.: pathways: entity-focused exploration of heterogeneous data graphs. in: the semantic web: eswc 2023 satellite events on proceedings, pp. 91–95. publisher (2023) google scholar  dong, y., chawla, n.v., swami, a.: metapath2vec: scalable representation learning for heterogeneous networks. in: proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining on proceedings, pp. 135-144. publisher (2017) google scholar  fu, x., zhang, j, meng, z., king, i.: magnn: metapath aggregated graph neural network for heterogeneous graph embedding. in: proceedings of the web conference 2020 on proceedings, pp. 2331–2341. publisher (2020) google scholar  gilmer, j., schoenholz, s.s., riley, p.f., vinyals, o., dahl, g.e.: neural message passing for quantum chemistry. publisher (2017) google scholar  hamilton, w.l., ying, r., leskovec, j.: inductive representation learning on large graphs. in: proceedings of the 31st international conference on neural information processing systems on proceedings, pp. 1025–1035. publisher (2017) google scholar  han, k., wang, y., guo, j., tang, y., wu, e.: vision gnn: an image is worth graph of nodes. in: advances in neural information processing systems, pp. 8291–8303. publisher (2022) google scholar  li, j., zhang, q., liu, w., chan, a. b., fu, y-g. koishekenov, y.: another perspective of over-smoothing: alleviating semantic over-smoothing in deep gnns. ieee trans. neural networks learn. syst. proc., 1–14. publisher (2024) google scholar  liu, z., zhou, j.: introduction to graph neural networks. publisher (2020) google scholar  rozemberczki, b., davies, r., sarkar, r., sutton, c.: gemsec: graph embedding with self clustering. in: gemsec: graph embedding with self clustering on proceedings, pp. 65-72. publisher (2020) google scholar  sun, y., han, j.: mining heterogeneous information networks: principles and methodologies. publisher (2012) google scholar  wang, x., bo, d., shi, c., fan, s., ye, y., yu, philip s.: a survey on heterogeneous graph embedding: methods, techniques, applications and sources. in: ieee transactions on big data on proceedings, pp. 415–436. publisher (2023) google scholar  wu, s., fei, s., wentao, z., xie, x, cui, b.: graph neural networks in recommender systems: a survey. publisher (2023) google scholar  ying, r., he, r., chen, k., eksombatchai, hamilton, p.w., leskovec, j.: graph convolutional neural networks for web-scale recommender systems. in: proceedings of the 24th acm sigkdd; data mining on proceedings, pp. 974–983. publisher (2018) google scholar  zhang, c., song, d., huang, c., swami, a., chawla, n.: heterogeneous graph neural network. in: proceedings of the 25th acm sigkdd; data mining on proceedings, pp. 793–803. publisher (2019) google scholar  download references author information authors and affiliations institute of informatics, federal university of rio grande do sul (ufrgs), ave. bento gonçalves, 9500, porto alegre, rio grande do sul, brazil silvio fernando angonese & renata galante authors silvio fernando angoneseview author publications search author on:pubmed google scholar renata galanteview author publications search author on:pubmed google scholar corresponding author correspondence to silvio fernando angonese . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper angonese, s.f., galante, r. (2025). enhancing graph data quality by leveraging heterogeneous node features and embeddings. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_27 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords heterogeneous embedding graph embedding heterogeneous graph representation knowledge graph neural networks publish with us policies and ethics profiles silvio fernando angonese view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature comparing contextual embeddings for semantic textual similarity in portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper comparing contextual embeddings for semantic textual similarity in portuguese conference paper first online: 28 november 2021 pp 389–404 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) comparing contextual embeddings for semantic textual similarity in portuguese download book pdf download book epub josé e. andrade junior10,11, jonathan cardoso-silva12,13 & leonardo c. t. bezerra10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1281 accesses 1 citation abstract semantic textual similarity (sts) measures how semantically similar two sentences are. in the context of the portuguese language, sts literature is still incipient but includes important initiatives like the assin and assin 2 shared tasks. the state-of-the-art for those datasets is a contextual embedding produced by a portuguese pre-trained and fine-tuned bert model. in this work, we investigate the application of sentence-bert (sbert) contextual embeddings to these datasets. compared to bert, sbert is a more computationally efficient approach, enabling its application to scalable unsupervised learning problems. given the absence of sbert models pre-trained in portuguese and the computational cost for such training, we adopt multilingual models and also fine-tune them for portuguese. results showed that sbert embeddings were competitive especially after fine-tuning, numerically surpassing the results of bert on assin 2 and the results observed during the shared tasks for all datasets considered. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others sentiment analysis in portuguese tweets: an evaluation of diverse word representation models article 28 june 2023 cosbert: a cosine-based siamese bert-networks using for semantic textual similarity chapter © 2023 sense representations for portuguese: experiments with sense embeddings and deep neural language models article 28 february 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics machine translation machine learning portuguese semantic memory 1 introduction semantic textual similarity (sts) is the task of measuring how semantically similar a pair of sentences is [11]. the importance of this task to the natural language processing (nlp) field is endorsed by the creation of sts shared tasks, such as the ones proposed by the international workshop on semantic evaluation (semeval [5, 12]). shared tasks like semeval led to great advancements in sts, but there is still a reduced number of studies in the context of the portuguese language. to foster this research, the assin [9] and assin 2 [15] workshops hosted shared tasks for sts and natural language inference (nli) in portuguese. the models developed during assin 2 used more recent nlp approaches, including contextual embeddings like bert [7]. recent works addressing these datasets have since been continuously proposed, with the state-of-the-art being a bert model pre-trained in portuguese fine-tuned for sts [8]. though competitive, training bert models on sts demands that all pairs of sentences be used as input to the network. this approach can cause a massive computational overhead, even for a moderate-size corpus. to address this problem, reimers and gurevych [17] proposed sentence-bert (sbert), a siamese architecture with shared weights that can reduce the cost of bert models. since its proposition, sbert has been used successfully in tasks where bert had obtained good results [17, 18]. however, to the best of our knowledge, these applications do not yet include sts in portuguese. the goal of this work is to evaluate contextual embeddings generated by sbert models for sts in portuguese, which we investigate in two stages. first, we compare the performance of pre-trained sbert models with the state-of-the-art bert models for the assin datasets [8]. in addition, we include other baseline models, such as the best-performing works assessed in the workshops and other multilingual contextual embeddings. later, we evaluate the benefits of fine-tuning sbert models for sts in portuguese, also comparing them with all baseline and state-of-the-art models. results from the first part of our investigation showed that multilingual sbert models are competitive, outperforming the best results of the assin shared task. even if at this stage fine-tuning was not considered, sbert results were second only to the results achieved by the state-of-the-art bert model [8]. in the second part of our work, results were improved with the fine-tuning, numerically surpassing the performance of the state-of-the-art model for the assin 2 dataset, at a much lower computational cost. for the assin datasets, even if the state-of-the-art results were not matched, the contextual embeddings generated by multilingual sbert models with fine-tuning remained competitive and, as mentioned before, required a reduced computational cost. finally, we discuss the impacts of fine-tuning and language variants with a qualitative assessment that demonstrates that results can be further improved in the future. the remainder of this work is structured as follows. section 2 briefly reviews preliminary concepts, namely contextual embeddings and the most relevant architectures used to produce them, and deep learning training approaches such as fine-tuning and knowledge distillation. next, sect. 3 defines the sts problem, details the assin and assin 2 shared tasks, and briefly discusses the state-of-the-art for these datasets. in sect. 4, we detail the experimental setup adopted in this investigation, and discuss results in sects. 5 and 6. we conclude and discuss future work in sect. 7. 2 background in this section, we briefly review the main preliminary concepts required to understand contextual embeddings, an important tool for problem solving in nlp. initially, we discuss transfer learning [25], the training paradigm that motivated the proposal of embeddings in general. next, we discuss contextual embeddings and the main algorithms adopted in this work, namely bert [7] and sentence-bert [17]. finally, we briefly detail knowledge distillation [10], the deep learning training paradigm that enables the multilingual training used to produce the models we adopt in this work. 2.1 transfer learning transfer learning is a training paradigm that enables models fit to a general problem to be reused for a separate, and usually more specific problem [25]. when solving the new problem, two main strategies can be adopted. pre-trained models can be applied directly to the problem at hand when adjustments to match different input and/or output dimensionalities are feasible. though pre-trained models are not expected to perform as well as problem-specific models, they are reusable across different problems. fine-tuned models are pre-trained models that are subject to additional training on a target specific problem. in detail, fine-tuning uses the pre-trained model as a starting point and runs additional training iterations on the new data. given the benefits of transfer learning, pre-trained models have become increasingly publicly available. combined with fine-tuning, the same pre-trained model can become multiple specialized models without training from the ground up, saving both computing and time resources. in the context of nlp problems, this has motivated researchers to propose general models and make their pre-trained versions available, the most relevant example being the contextual and word embeddings we discuss next. 2.2 contextual and word embeddings a statistical language model is a probability distribution function of word sequences in a given language. training language models is a very computationally intensive task, easily plagued by the curse of dimensionality, for two main reasons. first, the number of word co-occurrence is typically large. second, the word sequences vary considerably across multiple text datasets, and even from training to test sets. these challenges motivated the distributed word representation [3], models in which both the distributed representation of each word and the probability function for the word sequence are learned simultaneously. distributed representations can achieve better generalization since these models are able to attribute a high probability of occurrence to word sequences it had never seen before if similar sequences have been provided during training. popular word embeddings like word2vec [13], glove [14], and fasttext [4] consider all sentences in which a given word appear to build a global vector representation for each word in the corpus. however, a word can have different meanings depending on the sentence and context it is in. to solve this problem, different contextual embeddings were proposed. next, we discuss the two contextual approaches we use in this work, namely bert and sentence-bert. 2.3 bert the bidirectional encoder representations from transformers (bert) architecture brought great advances to the nlp field, lately becoming the current state-of-the-art in different tasks [7]. bert is a language model designed to create deep bidirectional vector representations from unsupervised texts. the bidirectional approach models context both to the right and to the left of each token in the input sequence. the bert architecture can be seen on fig. 1, where the pre-trained scenario is seen on the left and the fine-tuning scenario is illustrated for different tasks on the right. fig. 1. the bert architecture [7] for pre-training (left) or fine-tuning (right). full size image an improvement over other architectures is that bert can be applied to tasks that take as input either individual sentences or pairs thereof. in the latter case, both sentences are separated by a special token dubbed [sep]. whether for individual or sentence pairs, the first token fed to the network is a special token dubbed [cls], which serves as encoding for supervised tasks after training. alternatively, a bert embedding can be obtained by aggregating the three types of internal bert embeddings, namely token, segment, and position. 2.4 sentence-bert (sbert) although bert has become the state-of-the-art for several nlp tasks, the algorithm faces scalability issues in tasks such as sts where there is the need to feed the network with sentence pairs. according to reimers and gurevych [17], even a moderate sized training corpus of 10000 sentences requires 50 million computational inferences, rendering bert impractical for sts. as an alternative, sbert was proposed as a more efficient approach to this type of tasks. sbert [17] is a siamese architecture in which the sentence pairs are fed to two parallel but connected network pathways to produce an estimate of the similarity between the sentences (fig. 2a). the first layers consist of regular bert networks followed by a pooling operation to ensure embeddings produced for each sentence have the same vector size. the network outputs the cosine similarity between these embeddings. since both bert networks take a single sentence and their weights are shared, the number of computational inferences is much reduced [17]. fig. 2. sbert architecture (left) and multilingual knowledge distillation (right). full size image 2.5 knowledge distillation another concept relevant to this study is knowledge distillation, one of the deep learning training approaches where one model is trained to learn the knowledge encoded by another [10]. in this training approach, a student model attempts to emulate a teacher model with similar (or even higher) performance. a knowledge distillation-based approach has been proposed by reimers and gurevych [17] to produce multilingual sbert models and is illustrated in fig. 2b. the teacher model is an english pre-trained sbert model and the student model is trained to produce multilingual embeddings. training is structured as follows. first, the input for teacher and student models are different, since the original sentence fed to the teacher is enriched with parallel, translated versions for the student. second, the objective function is to reduce the mean squared error losses comparing every student embedding produced with the original embedding produced by the teacher. effectively, translated sentences are mapped to the same location in the euclidean space of the original sentence. two other aspects of that work are further worth mentioning. first, the student models are trained with parallel data for 50 languages including portuguese. second, the student model is learned by a xlm-r algorithm [6], a cross-lingual version of bert. in detail, authors claim xlm-r is better suited for multilingual learning than bert given its ability to handle non-latin alphabets. as reviewed in this section, the literature on contextual embeddings has matured over recent years. still, models are usually developed and made available for nlp applications in english. in the next section, we discuss the sts problem in the context of the portuguese language. 3 semantic textual similarity in portuguese as previously discussed, semantic textual similarity (sts) is a relevant nlp application with international efforts to foster its research. sts can be defined as a regression task where one wants to numerically measure the similarity between two sentences. higher scores indicate a stronger similarity between a sentence pair. among the main international efforts concerning sts, the international workshop on semantic evaluation (semeval [1]) is a series of workshops focused on semantic nlp problems. each year, several shared tasks are hosted, in which high-quality datasets are used to benchmark submissions from participating research teams. several similarity tasks have been defined at semeval, including cross-lingual and multilingual sts [5]. the first dataset that included semantic textual similarity between pairs of sentences in portuguese was assin [9]. later, assin 2 [15] proposed a new dataset, based on sentences from the sick-br dataset [16]. next, we describe each shared task, its associated datasets, and the best results observed for each. assin stands for evaluation of semantic textual similarity and natural language inference [9], a shared task organized at the propor 2016 conference. the assin datasets comprise (i) pt-pt, containing sentences in european portuguese, and (ii) pt-br, containing sentences in brazilian portuguese. each dataset (pt-pt and pt-br) has 5,000 sentences: 2,500 for training, 500 for validation, and 2,000 for testing. sentences in each dataset were obtained from google news, processed in three steps. first, vectorial space models [21] were used to select similar sentences from different documents. second, these sentences were filtered manually, and noisy pairs were removed. finally, filtered sentence pairs were annotated by four human judges. scoring rules are describe below: 1. sentences are completely different. they may talk about the same fact, but this cannot be determined without context. 2. sentences refer to different facts and are not similar, but are about the same subject (e.g. a soccer match or elections). 3. the sentences present some similarity; can refer to the same fact or not. 4. sentence content is very similar, but one (or both) has some exclusive information (e.g. a date or locality). 5. sentences have practically the same meaning, possibly with a minimal difference (e.g. an adjective that does not change interpretation). assin 2 was the second edition of assin [15], held at the stil 2019 symposium in brazil. the dataset is a translation and manual adaptation of the sick dataset [12], the sick-br dataset [16]. in addition, assin 2 also includes manually-generated sentence pairs, which were reviewed by human judges. each pair of sentences in assin 2 was annotated by at least four native brazilian portuguese speakers with linguistic training background. for sts, the final score was the average of the scores of each annotator. the final dataset has 10000 pairs of sentences: 6500 used for training, 500 for validation, and 3000 for testing. baseline results for the assin and assin 2 datasets are listed at the top of table 1. pearson correlation coefficient (to be maximized) is considered as the primary metric for the comparison, complemented by the mean squared error (mse, to be minimized). the three topmost works are the best results obtained during the shared tasks, whereas the following two models were proposed in a work published after assin 2 [8].footnote 1 for brevity, the models we assess in this work are also given in table 1, and are described in the next section. baseline results. the work of fialho, coheur, and quaresma [8] considerably advanced the state-of-the-art for both assin pt-pt and assin pt-br, and to a lesser extent also for assin 2. specifically, the authors experimentally assessed bert models for sts and nli in portuguese using the datasets assin, assin 2, and sick-br. for sts, two approaches were considered: bert as a regressor or bert embeddings as input for regression algorithms. three models fine-tuned for sts were evaluated, namely (i) one multilingual and (ii) two monolingual versions pre-trained in brazilian portuguese differing in size (ptbert-base\(_\textit{ft}\) and ptbert-large\(_\textit{ft}\)). ptbert-large\(_\textit{ft}\) used as a regressor achieved the best results. table 1. results using baseline, pre-trained, and fine-tuned models.full size table as discussed in this section, assin and assin 2 are relevant portuguese sts datasets for which bert models are the state-of-the-art. yet, the computational overhead incurred for bert training limits its applicability to problems with a moderate size sentence collection. in the next section, we discuss how we employ and assess sbert models on the portuguese sts datasets reviewed here. 4 experimental assessment sbert models have been successfully employed on nlp applications where bert models had state-of-the-art performance [17]. furthermore, multilingual pre-trained sbert models have been made available in an open source sbert repository (https://www.sbert.net). in this section, we describe our experimental analysis in which we assess the performance of sbert models with and without fine-tuning for sts in portuguese. initially, we detail the models we consider, following the order given in table 1.footnote 2 later, we discuss the two stages of our experiments, differing as to whether fine-tuning is employed. 4.1 models assessed as listed in table 1, the top-most five models we consider as baseline are (i) the best-performing algorithms of the assin and assin 2 shared tasks, and (ii) the state-of-the-art bert models [8], already described in the previous section. in addition, we also include as baseline two portuguese pre-trained bert models called bertimbau [20]. remaining models given in table 1 are used within the sbert architecture. the first two reuse bertimbau weights, given the compatibility between bert and sbert. the remaining three models are multilingual pre-trained models open sourced by sbert proponents. we choose to employ pre-trained models instead of training models from scratch for two major reasons. first, the computational cost required for training deep learning nlp models from scratch is considerable. second, the pre-trained models have often been trained on very rich corpora, whether in portuguese or multilingual. below, we describe the bertimbau and multilingual models. bertimbau models were trained by neuralmind and are available in two sizes, namely bertimbau-base and bertimbau-large. pre-training bert requires a vocabulary and a training corpus. neuralmind produced a vocabulary from the brazilian portuguese version of the wikipedia, and used the brwac corpus [22]. pre-training followed the original bert paper [7]. bertimbau-base and bertimbau-large were respectively warm-started with the multilingual bert-base and english bert-large checkpoints. sbert models pre-trained in portuguese are not openly available. instead, we adopt two sets of models, given in table 1 on the five rows that follow baseline models. the first two models, labeled bertimbau-base (sbert) and bertimbau-large (sbert), reuse the trained weights provided by bertimbau models. the remaining three models are multilingual models open-sourced by sbert proponents, having been trained using the knowledge distillation process. among the models available, we used the ones for which the student-model was learned by the xlm-r algorithm [6]. different teacher-models were used, described below under the notation \(\langle \textsf {student, teacher, task}\rangle \), indicating the student-model and the task for which the teacher-model was originally trained. \(\langle \textsf {xlm-r, sbert, nli+sts}\rangle \): xlm-r learned a multilingual model from an sbert model fine-tuned for nli and sts in english. \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \): xlm-r learned a multilingual model from an sbert model trained on an english paraphases dataset. \(\langle \textsf {xlm-r, sbert, cross-en-de}\rangle \): fine-tuned \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) model for the stsbenchmark [5], with english and/or german sentences. training multilingual versions used a range of datasets, and so contextual embeddings produced by sbert were generalized by xlm-r for 50 different languages. further information about their training is available in the original paper [18]. 4.2 experimental setup as previously discussed, we split our experiments in two parts to isolate the effect of fine-tuning. this is reflected in table 1, where we label results obtained from a given model after fine-tuning with the subscript \(\textit{ft}\). each part of the investigation is further detailed next. experiments without fine-tuning. in the first part of our experiments, we employed the pre-trained models on assin and assin 2 datasets without fine-tuning. concretely, labels provided by the datasets were used only to validate the results. for the baseline bertimbau models (bertimbau-base and bertimbau-large), each input sentence was fed individually to the model and the corresponding embedding was obtained from the [cls] token. the similarity score for a pair of sentences was calculated as the cosine distance between the sentence embeddings produced. then, we compare the cosine similarity directly with the labels of assin and assin 2 using pearson correlation. though cosine distance ranges differs from the sts score range, normalization and scaling is not necessary since pearson correlation only considers the linear relationship between the variables. experiments with fine-tuning. in the second part of our experimental analysis, we assessed the impact in performance when pre-trained models are fine-tuned for the assin and assin 2 datasets. in detail, we use the available validation labels to fine-tune the pre-trained models. the developers of sbert made available a standard fine-tuning script, with recommended hyperparameters we adopted. for each dataset and model, we ran fine-tuning ten times and report mean and standard deviation results. regarding baseline bertimbau results, fine-tuning would be computationally unfeasible in the context of this work. yet, we remark that the state-of-the-art models [8] are ptbert-base and ptbert-large networks that have been fine-tuned for the assin and assin 2 datasets. as such, these models serve as reference for bert results, though mean and standard deviation have been reported only for five repetitions. given the large number of models we consider, our complete set of results is provided as supplementary material [2]. in the next sections, we discuss the most relevant insights we observe in our assessment. 5 results as previously discussed, contextual embeddings are usually produced from pre-trained or fine-tuned language models. in our setup, we have isolated experiments without and with fine-tuning for sts in portuguese. below, we discuss the most relevant insights observed for each part of our investigation. 5.1 pre-trained models without fine-tuning for sts in portuguese in order of appearance in table 1, literature results are given at the top, as previously detailed. the following two models are also taken as baseline. in detail, bertimbau-base and bertimbau-large are the bertimbau models that have been pre-trained in portuguese. remaining models are sbert approaches. given the compatibility between bert and sbert, we include sbert results where weights were obtained from bertimbau models, which we refer to as ptbert-base (sbert) and ptbert-large (sbert). finally, table 1 lists the multilingual sbert models, namely \(\langle \textsf {xlm-r, sbert, nli+sts}\rangle \), \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) and \(\langle \textsf {xlm-r, sbert, cross-en-de}\rangle \). next, we discuss the main insights observed in this part of our investigation. portuguese pre-trained models did not produce competitive results w.r.t. workshop or literature results. a direct comparison between models that use bertimbau weights showed that sbert models in general achieved better results than their bert counterparts. this result is interesting considering that the only difference between them is that embeddings from bert models are obtained from the [cls] special token, whereas sbert is average pooling all bert output embeddings. another important insight is that not always the large version outperformed its base counterpart. this is likely explained by the checkpoints used by neuralmind for bertimbau model training. as discussed in sect. 4, a multilingual bert checkpoint was used to warm start base training, whereas large models were warm-started with an english bert checkpoint. multilingual pre-trained models produced competitive results w.r.t. results from the participating teams of the shared tasks. this is true for nearly all datasets and multilingual models considered, with two major exceptions. first, the \(\langle \textsf {xlm-r, sbert, nli+sts}\rangle \) model is unable to match workshop results for any of the datasets considered, though its teacher-model had been originally trained for nli and sts tasks. second, no model is able to match the performance of assin 2 workshop results. among the sbert versions, the one that shows the most consistent performance is the one fine-tuned for cross-en-de sts. this indicates that fine-tuning for multilingual tasks is helpful, as expected. yet, without a specific fine-tuning for sts in portuguese, even this model cannot match state-of-the-art results [8]. indeed, in this part of the investigation all sbert results fall short of the state-of-the-art for all datasets considered. 5.2 pre-trained models with fine-tuning results obtained with fine-tuned models are displayed at the bottom part of table 1. models are given in the same order as the pre-trained models. below, we discuss the main insights from this analysis. benefits of fine-tuning. the performance gains provided by fine-tuning are considerable for all algorithms. in fact, all models now (numerically) outperform the best results achieved during the assin and assin 2 tasks. to evidence the performance gains provided by fine-tuning, the boxplots in fig. 3 show the average performance of all sbert models with and without fine-tuning, grouped by dataset. in addition to improving average performance, fine-tuning also reduces variability to a significant extent, especially on assin pt-pt. comparison with the state-of-the-art. results differ as a function of shared task. multilingual sbert models numerically surpass the state-of-the-art results according to pearson correlation and/or mse on assin 2. remaining models also get closer to the state-of-the-art for this task, confirming the expected benefits from sbert and fine-tuning. for assin pt-br and assin pt-pt, the best results showed by sbert models are not yet competitive. homogeneity of results. in the absence of fine-tuning (pre-trained section in table 1) results were very contrasting among different models. conversely, all fine-tuned models have produced results that are much closer to each other. we illustrate this increased homogeneity for assin 2 results in fig. 4, where average pre-trained model performance is given in red and average fine-tuned model performance is given in blue, grouped by model and sorted by fine-tuned performance. for this dataset, even bertimbau models lead to a competitive performance after fine-tuning. fig. 3. performances of pre-trained models with and without fine-tuning. full size image fig. 4. homogeneity of results with (blue) and without (red) fine-tuning (color figure olnine) full size image overall, results discussed in this section confirmed the benefits of adopting sbert and fine-tuning for sts in portuguese. for the assin 2 dataset, multilingual models even numerically surpass the current state-of-the-art. in the next section, we further investigate results to understand these benefits. 6 further analysis to better understand some of the insights observed in the previous section, we initially conduct a qualitative assessment, regarding both (i) fine-tuning and (ii) language variant effects. moreover, we run additional experiments to assess the benefits of multiple language variants in the fine-tuning setup. table 2. assin 2 sentence pair examples with their scores illustrating changes in cosine similarity before and after fine-tuning \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \).full size table 6.1 fine-tuning effects we select a few sentence pairs to illustrate fine-tuning effects on \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) results, the sbert model that performed best in our experiments. table 2 gives six sentence pair examples with their expected scores and cosine similarities before and after fine-tuning for sts in portuguese. for the first three pairs, results were improved by fine-tuning. indeed, fine-tuned results become very close to the expected score, whereas pre-trained results were very off. by contrast, results obtained with fine-tuning for the latter three sentence pairs in table 2 are worse than when fine-tuning is not adopted. in the first deterioration example, we believe the model misinterprets the terms “colocando” (putting) e “tirando” (taking), which can be used as antonyms but not in the context “tirando uma foto” (taking a picture). in the second example, we believe the model misinterprets a negation quantifier “não tem nenhuma mulher” (no woman) with an opposition adverb (not a woman), which renders both sentences less semantically different. future analysis could also investigate the impact of double negatives in this case, as multilingual models need to deal with such differences between languages. finally, in the last example the only differences between sentences regard (i) active versus passive sentence form, and (ii) whether “montando” refers to getting on or riding an animal. since both interpretations are possible, we believe that the assin 2 example could be considered noisy. 6.2 language variant effects with the exception of the state-of-the-art, all algorithms assessed for assin datasets in the literature and also here are unable to perform their best for the european and brazilian portuguese variants at the same time. we illustrate this with sentence pairs given in table 3. the two top-most sentence pairs are obtained from assin pt-br, and the two bottom-most from assin pt-pt. sentences selected in european portuguese respectively illustrate vocabulary and grammatical differences between the language variants. beside expected scores, similarities computed by \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) when fine-tuned for assin pt-br and assin pt-pt are also given. results for the model fine-tuned for assin pt-br are better on that dataset. the symmetrical situation is observed for the assin pt-pt dataset. the better performance of models fine-tuned for the given language variant is expected. we then conduct additional experiments to understand the impact of fine-tuning \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) for the three assin and assin 2 datasets altogether. we report mean values for ten repetitions, and provide standard deviations as supplementary material [2] given the very low variability observed in the results. results differ as a function of the target testing dataset. for assin pt-br and assin 2, results are not changed, except for mse on the latter, reduced from 0.43 to 0.41. regarding assin pt-pt, results are strongly improved, with pearson coefficient correlation increasing from 0.77 to 0.81 and mse reducing from 0.52 to 0.49. table 3. assin pt-br (top) and pt-pt (bottom) pairs with their scores illustrating fine-tuning \(\langle \textsf {xlm-r, sbert, paraphrases}\rangle \) for different language variants.full size table the qualitative assessment conducted in this section helped further understand the benefits of fine-tuning and the potential impacts of different language variants. even if improved results did not match the state-of-the-art for the assin pt-pt dataset, they shed light into promising future work directions. 7 conclusions semantic textual similarity (sts [11]) is a core research problem in natural language processings (nlp) studies. this is evidenced by the (i) international efforts promoting sts shared tasks [5, 12] and (ii) the fact that several novel contextual embedddings recently proposed have been benchmarked on this problem [17]. in the context of the portuguese language and its variants, assin and assin 2 are the most relevant shared tasks identified [9, 15], for which bert embeddings comprise the current state-of-the-art [8]. in this work, we have investigated the application of the sentence-bert (sbert [17]) architecture to sts in portuguese. sbert addresses scalability issues in bert networks, and has been successfully applied and open-source also for multilingual applications [18]. our contributions were two-fold. in the first part of this work, we assessed portuguese and multilingual sbert models, demonstrating that the latter present competitive performance w.r.t. shared task results even without fine-tuning. this can be very useful to data professionals who need to address this problem in unsupervised learning scenarios, as typical in real-world applications. in the second part, we demonstrated that fine tuning sbert led to improvements in all previous results. the multilingual models of sbert once again stood out and this time surpassed the state-of-the-art for the assin 2 dataset. further analysis indicated promising future work possibilities. in detail, we have observed situations where fine-tuning worsened results, an indication that pre-training sbert models in portuguese could lead to yet better results. more importantly, we have discussed the impact of language variants, which we believe should be taken into consideration either for pre-training or fine-tuning. we evidenced this possibility with additional experiments that further improved the performance on european portuguese for the best-performing sbert multilingual model. the contributions discussed in this paper further highlight that other contextual embeddings such as xlnet [24] need to be assessed in the context of sts in portuguese. in this sense, we believe our work is instrumental to motivate more research groups addressing nlp problems in portuguese. this is the purpose of shared tasks, especially given the computational cost of the experimental campaigns involved in deep learning assessment for nlp applications. even more interesting is stirring the interest of researchers proposing novel nlp models to the portuguese language, given that the differences between portuguese variants impacts on the results for most models considered. notes 1.for brevity, other relevant works such as [19] comparing word2vec, fasttext, elmo, and bert on assin are not included as their results are surpassed by [8]. 2.though models based on other relevant architectures such as the multilingual universal sentence encoder [23] were available at the sbert repository, we did not include them in our work due to the lack of training setup details. references agirre, e., diab, m., cer, d., gonzalez-agirre, a.: semeval-2012 task 6: a pilot on semantic textual similarity. in: semeval, pp. 385–393. acl, usa (2012) google scholar  andrade, j., bezerra, l.c.t., cardoso-silva, j.: comparing contextual embeddings for semantic textual similarity in portuguese (supplementary material) (2021). https://github.com/andradejunior/bracis-2021-supp-material bengio, y., ducharme, r., vincent, p., janvin, c.: a neural probabilistic language model. j. mach. learn. res. 3, 1137–1155 (2003) math  google scholar  bojanowski, p., grave, e., joulin, a., mikolov, t.: enriching word vectors with subword information. trans. acl 5, 135–146 (2017) google scholar  cer, d., diab, m., agirre, e., lopez-gazpio, i., specia, l.: semeval-2017 task 1: semantic textual similarity multilingual and crosslingual focused evaluation. in: semeval, pp. 1–14. acl, vancouver (2017) google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale. in: acl, pp. 8440–8451. acl (2020) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: naacl, pp. 4171–4186. acl, minneapolis (2019) google scholar  fialho, p., coheur, l., quaresma, p.: benchmarking natural language inference and semantic textual similarity for portuguese. information 11, 484 (2020) article  google scholar  fonseca, e.r., borges dos santos, l., criscuolo, m., aluísio, s.m.: visão geral da avaliação de similaridade semântica e inferência textual. linguamática 8(2), 3–13 (2016) google scholar  gou, j., yu, b., maybank, s.j., tao, d.: knowledge distillation: a survey. int. j. comput. vis. 129(6), 1789–1819 (2021) article  google scholar  jurafsky, d., martin, j.h.: speech and language processing, 2nd edn. prentice-hall inc., upper saddle river (2009) google scholar  marelli, m., bentivogli, l., baroni, m., bernardi, r., menini, s., zamparelli, r.: semeval-2014 task 1: evaluation of compositional distributional semantic models on full sentences through semantic relatedness and textual entailment. in: semeval, pp. 1–8. acl, dublin (2014) google scholar  mikolov, t., sutskever, i., chen, k., corrado, g., dean, j.: distributed representations of words and phrases and their compositionality. in: neurips, pp. 3111–3119. curran associates inc., red hook (2013) google scholar  pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: emnlp, pp. 1532–1543. acl, doha, qatar (2014) google scholar  real, l., fonseca, e., gonçalo oliveira, h.: the assin 2 shared task: a quick overview. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 406–412. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_39 chapter  google scholar  real, l., et al.: sick-br: a portuguese corpus for inference. in: villavicencio, a. (ed.) propor 2018. lncs (lnai), vol. 11122, pp. 303–312. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_31 chapter  google scholar  reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: emnlp, pp. 3973–3983. acl (2019) google scholar  reimers, n., gurevych, i.: making monolingual sentence embeddings multilingual using knowledge distillation. in: emnlp, pp. 4512–4525. acl (2020) google scholar  rodrigues, r.c., rodrigues, j., de castro, p.v.q., da silva, n.f.f., soares, a.: portuguese language models and word embeddings: evaluating on semantic similarity tasks. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 239–248. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_23 chapter  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  turney, p.d., pantel, p.: from frequency to meaning: vector space models of semantics. j. artif. int. res. 37(1), 141–188 (2010) mathscinet  math  google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: lrec. elra, miyazaki, japan (2018) google scholar  yang, y., et al.: multilingual universal sentence encoder for semantic retrieval. in: acl: system demonstrations, pp. 87–94. acl, online, july 2020 google scholar  yang, z., dai, z., yang, y., carbonell, j., salakhutdinov, r.r., le, q.v.: xlnet: generalized autoregressive pretraining for language understanding. in: wallach, h., larochelle, h., beygelzimer, a., d’ alché-buc, f., fox, e., garnett, r. (eds.) neurips, vol. 32. curran associates, inc. (2019) google scholar  yosinski, j., clune, j., bengio, y., lipson, h.: how transferable are features in deep neural networks? in: ghahramani, z., welling, m., cortes, c., lawrence, n., weinberger, k.q. (eds.) nips, vol. 27. curran associates, inc. (2014) google scholar  download references author information authors and affiliations imd, universidade federal do rio grande do norte, natal, rn, brazil josé e. andrade junior & leonardo c. t. bezerra ifood, osasco, sp, brazil josé e. andrade junior data science brigade, porto alegre, rs, brazil jonathan cardoso-silva london school of economics and political science, london, uk jonathan cardoso-silva authors josé e. andrade juniorview author publications search author on:pubmed google scholar jonathan cardoso-silvaview author publications search author on:pubmed google scholar leonardo c. t. bezerraview author publications search author on:pubmed google scholar corresponding author correspondence to josé e. andrade junior . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado 1 electronic supplementary material below is the link to the electronic supplementary material. supplementary material 1 (pdf 22 kb) rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper andrade junior, j.e., cardoso-silva, j., bezerra, l.c.t. (2021). comparing contextual embeddings for semantic textual similarity in portuguese. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_27 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning natural language processing semantic textual similarity word embeddings publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature how to generate synthetic paintings to improve art style classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper how to generate synthetic paintings to improve art style classification conference paper first online: 28 november 2021 pp 238–253 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) how to generate synthetic paintings to improve art style classification download book pdf download book epub sarah pires pérez  orcid: orcid.org/0000-0003-0771-231010 & fabio gagliardi cozman  orcid: orcid.org/0000-0003-4077-493510  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1313 accesses 4 citations abstract indexing artwork is not only a tedious job; it is an impossible task to complete manually given the amount of online art. in any case, the automatic classification of art styles is also a challenge due to the relative lack of labeled data and the complexity of the subject matter. this complexity means that common data augmentation techniques may not generate useful data; in fact, they may degrade performance in practice. in this paper, we use generative adversarial networks for data augmentation so as to improve the accuracy of an art style classifier, showing that we can improve performance of efficientnet b0, a state of art classifier. to achieve this result, we introduce class-by-class performance analysis; we also present a modified version of the sagan training configuration that allows better control against mode collapse and vanishing gradient in the context of artwork. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others style recommendation and simulation for handmade artworks using generative adversarial networks article open access 14 november 2024 creativity and style in gan and ai art: some art-historical reflections article 01 may 2024 research on the innovation mechanism and development strategy of digital media art in the context of artificial intelligence era chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. art history fine art machine learning symbolic ai visual culture artificial intelligence 1 introduction no society, however low it may have been its level of material accomplishment, has ceased to produce art [15]. art has been, through the ages, a mirror reflecting society; the purpose of art has varied but its importance in capturing society has been constant. one way to discuss the content of artwork is through style categories. such categories indicate connections between paintings and help us to better understand their meanings. even if many art theorists look at style categories with reservations,footnote 1 categories provide an important guideline for beginner art aficionados and the general crowd within museums and galleries. besides, style categories can be used by recommendation tools to improve a museum visitor’s experience, either in a traditional physical museum location or in one of the increasingly common digital galleries. alas, the current quantity of online art has now surpassed our manual indexing abilities. the field of computer vision has developed artwork style classification tools using a diversity of techniques from feature-based machine learning [2, 32] to deep learning [3, 10, 23]. beyond the challenge of class imbalance, style classification is further complicated by aesthetic diversity, a problem that is not usually met in most image classification models. for instance, the baroque is directly linked to the historic moment of the catholic church struggling against the protestant reformation; in an attempt to combat the protestant faith, the church announced during the council of trent (1545–1563) the dissemination of religious ideas through images to awaken religious fervor—therefore, the baroque works are very concentrated on the religious theme. minimalism, on the other hand, does not have a single theme, but it employs a few elements in a distinctive manner [15]. the use of image augmentation is well established in computer vision so as to enhance the content of a training dataset. but usual techniques do not produce good results when applied to artwork; for instance, many of the common image augmentation techniques modify the resulting color of the image, which can be problematic in the case of artwork as colors often have or convey particular meaning. in this paper we propose an artwork style classifier that resorts to generative adversarial networks (gans) as an image augmentation tool for oversampling. artwork generation with gans is not a direct matter; even high performance gan architectures did not do well in preliminary experiments, so we had to propose techniques to enhance image quality and diversity. we describe in this paper a methodology for data augmentation that is tuned to the requirements of artwork classification; we also describe a specialized network architecture and an adjusted loss function that both contribute to the overall performance of our proposal. in short, our contributions are both in adopting gans for data augmentation in the particular context of artwork classification, and in proposing novel techniques that may be of broad interest whenever similar classification problems are met. the paper is organized as follows. section 2 analyzes related work on art style classification and generative adversarial networks. section 3 describes our approach; results and analysis are shown in sect. 4. finally, conclusions are presented in sect. 5. 2 related work in this section, we explore relevant efforts on artwork classification. we also present the architectural development of generative adversarial networks (gans), its use in the art domain and its relevance to image augmentation. 2.1 artwork classification initial research on artwork classification was focused on feature-based machine learning [2, 32]. since the initial success of a convolutional neural network (cnn) in object classification for imagenet [25], several studies in artwork classification have explored these networks [3, 10, 16, 23]. to improve the performance of the classification of art styles with cnns, researchers have investigated the benefits of pre-trained weights from the imagenet challenge instead of random startups [26, 36]. there has been significant activity in artwork classification in recent years. cetinic et al. expanded artwork classification beyond style, genre, artist and time period classification, examining also nationality and testing scene recognition and sentiment analysis techniques [6]. chu et al. studied the style classification task focusing on describing image texture with deep learning. they investigated the intra-layer and inter-layer correlations in order to create deep features for style classification. [9]. the work of elgammal et al. analyzed the learned representations of a fine-tuned resnet-152, noting that some of the style patterns designed by heinrich wölfflin (1846–1945) correlate with the pca decomposition of these learned representations [14]. in rodriguez et al., five image patches of painting were used for training and weights for each patch were optimized in order to improve accuracy of the final model [29]. sandoval et al. also worked with image patches, but in a two-stage deep learning approach, in which these five patches are trained independently at a first step. at the second stage, the outcome of these patches are fused to a second shallow neural network for the final decision [31]. zhong et al. presented a two-channel dual path network and two inputs are used: the rgb image and four-directional gray-level co-occurrence matrix for detecting the brush stroke information [44]. the work of zhu et al. not only trained the inception v3 network for classifying nine artistic movements, but also used grad-cam heat map for visualizing the areas of the images the model was focusing for class prediction [46]. chen et al. presented an adaptive cross-layer correlation for artwork classification, in which it adaptively weights features in different spatial locations based on similarity [7]. bianco et al. studied the advantages of training a model with the full image of the paintings and also its crop at different resolutions [4]. 2.2 generative adversarial networks generative adversarial networks (gans) are built by training two networks with a minimax game framework, where we have a model capable of generating synthetic data (model g) and another model that evaluates whether the data is real or synthetic (model d) [19]. in the work of mirza and osindero, an extra parameter was introduced in the gan architecture so as to allow the generator model to create images according to class labels [27]. chen et al. developed a gan architecture to learn disentangled representations in an unsupervised manner; they introduced a representation learning algorithm called information maximizing generative adversarial networks (infogan), in which an information-regularized minimax game is used in order to train a multi class generative model without the label information [8]. some work on gans has focused on the impact of the loss function. arjovsky et al. used the concept of the earth mover (em) distance, also known as wasserstein-1 distance, as a loss function [1]. gulrajani et al. improved the wasserstein function loss adding a gradient penalty [20]. an important concept in the evolution of the gan architecture is the self-attention mechanism. the convolutional architecture processes information in local neighborhoods and it has no mechanism to deal with long distance dependencies. the self-attention enables both the generator and the discriminator to deal with widely separated spatial regions. the work of zhang et al. presented this concept as self-attention gan (sagan) [43] and other proposals have used this architecture as a reference [5, 11]. the use of synthetic images to improve classifiers has been adopted particularly in the medical field, for instance in simulating lung nodules [21], ecg [42], liver lesions [17], chromosomes [40], skins lesions [28] and covid-19 results [12, 38]. we also mention the work of suh et al., where classifier loss is included in the gan training process in order to reduce ambiguity between classes: their classification enhancement generative adversarial networks (cegan) consist of three independent networks – a discriminator, a generator and a classifier – using wgan-gp for classification under imbalanced data conditions [34]. generative adversarial networks have already been used in the art domain. for instance, one can find a generator for image style transfer (the cyclegan [45]), an image-to-image translator from art to real images [18, 37], and a model specialized in creating chinese landscapes [41]. the first, and to the best of our knowledge the only, work aimed at creating artwork with the wikiart dataset appeared in ref. [13]. there, elgammal et al. developed the creative adversarial network to creatively generate artwork by maximizing deviation from established styles and by minimizing deviation from art distribution [13]. the authors found that the way to encourage the generator “to be creative" was to penalize it any time in which it was too easy for the discriminator to identify the synthetic image as being art from a certain style. 3 our proposal simply put, our main goal is to enhance the accuracy of artwork style classification. in pursuing this, we were led to study the potential benefits of synthetic images generated by generative adversarial networks. as noted before, synthetic art generation has been employed only by elgammal et al. (2017), but not with the purpose of enhancing classification; to the best of our knowledge, the latter task has not been investigated yet. in short, our specific goal here is not to make art, but to improve art style classification. we introduce a strategy that we refer to as class-by-class performance analysis. in order to start up learning, a baseline model is trained without image augmentation techniques. the next step is to explore the benefit of geometric transformations. we then concentrate on classes with the lowest performance; we want to maximize the information about these classes. such information is enhanced by a version of gans with a self-attention mechanism for image diversity that is trained with the wasserstein with gradient penalty loss function for avoiding vanishing gradient. in addition, classes with low performance may or may not contain a small number of images. hence, we suggest two strategies when we sample either: low quantity classes: add a multiple of the number of original images; high quantity classes: add a fraction of the number of original images. there are many decisions that must be set in implementing this strategy; some of them depend on the particular artwork collection is dealing with. in the following subsections, all relevant aspects of our strategy are discussed: our study of image augmentation, our choices regarding the gan architecture and the loss function and, finally, the model architecture for our classification task. 3.1 image augmentation data augmentation is a natural solution to the problem of limited data [33]. with respect to images, augmentation techniques consist of geometric transformations, such as rotation, image cropping, flipping and color conversions [34]. we studied these techniques in the context of art style classification, a setting with many classes and serious class imbalance. our first experiment was to ascertain the benefits of rotation, image cropping and flipping. results can be seen in table 6. improvement was obtained by rotating the image between -10 and 10 degrees, horizontal flips and random crops. alas, traditional image augmentation techniques often generate simple and redundant copies of the original data in many cases [34]. in some cases, data augmentation may not be a profitable idea. for many domains, color conversions offer image diversity; for instance, the object class “bicycle" can be represented with red or blue bicycles. in artwork, however, colors are meaningful and altering them without care can result in an image that lies in a category different from the original one. for example, pablo picasso’s blue period paintings (1901–1904) should always be represented by gloomy shades of grayish blue. the true atmosphere color for édouard manet should always be violet and many of his impressionist colleagues used violet as their main color [24]. 3.2 generative adversarial network one of the most challenging tasks for generative adversarial networks is producing a diverse set of synthetic images. wang et al. compare the most influential gan architectures with respect to image quality, performance against vanishing gradient and ability for mode diversity. in order to create synthetic artwork, where shape must greatly vary and colors are very relevant, we chose to focus on a mechanism proven to excel in diversity; we adopted self-attention generative adversarial network (sagan) with some modifications which will be explained later. we chose this architecture as the basis of our gan as the self-attention mechanism helps the gan to learn global and long-range dependencies across multi-class images [39]. fig. 1. the self-attention mechanism [43] full size image figure 1 shows a self-attention module. transformers are used to create the key f(x), query g(x) and value h(x): $$ f(x) = w_{f}x, \qquad g(x) = w_{g}x, \qquad h(x) = w_{h}x. $$ the attention map is created after applying a softmax to the dot product of the key and the query (eq. 1). another dot product is taken between the attention map and the value; the attention map is applied to the value in order to create a self-attention map (\(o_{j}\)) (eq. 2). $$\begin{aligned} \alpha _{j,i} = \mathrm {softmax}(f(x_{i})^{\top } g(x_{j})), \end{aligned}$$ (1) $$\begin{aligned} o_{j} = w_{v}\left( \sum _{i=1}^{n}\alpha _{j,i}h(x_{i})\right) . \end{aligned}$$ (2) the parameters \(w_{f}\), \(w_{g}\), \(w_{h}\) and \(w_{v}\) are the learned weight matrices. 3.3 adversarial loss function the original version of the sagan training configuration used hinge loss. for our dataset, we experienced frequent mode collapse and vanishing gradient with this loss, likely due to our dataset’s diversity. moreover, in order to guarantee this to be k-lipschitz, as needed to prevent mode collapse, a gradient penalty was added to the loss function. for this reason, we used the wasserstein with gradient penalty (wasserstein-gp) loss function so as to have better control of the values for feedback. that is, the loss is: $$\begin{aligned} l = \mathop {\mathbb {e}}_{\tilde{x}\sim \mathbb {p}_{g}} \left[ d(\tilde{x}) \right] \mathop {\mathbb {e}}_{x\sim \mathbb {p}_{r}} \left[ d(x) \right] + \lambda \mathop {\mathbb {e}}_{\hat{x}\sim \mathbb {p}_{\hat{x}}} \left[ (||\nabla _{\hat{x}} d(\hat{x})||_{2} 1)^2\right] , \end{aligned}$$ (3) where \(\mathbb {p}_{r}\) is the real images distribution, \(\mathbb {p}_{g}\) is the generated images distribution and \(\mathbb {p}_{\hat{x}}\) is the sampling uniformly along straight lines between pairs of points sampled from the data distribution \(\mathbb {p}_{r}\) and the generator distribution \(\mathbb {p}_{g}\). in our experiments, the interpolation between a batch of real images and fake images was enforced and the gradient norm of its output was limited at 1. the value of the penalty coefficient (\(\lambda \)) was 10, following the original paper [20]. 3.4 efficientnet the efficientnet b0 architecture was chosen for these experiments due to its high performance and relatively small size. efficientnet b0 belongs to a family of models known as efficientnet, presented by tan and le [35]. the efficientnet b0 is the efficientnet family’s baseline, from which all the models from b1 to b7 are scaled up uniformly in dimensions of depth, width and resolution. table 1 summarizes the stages in the efficientnet b0. each stage consists of one or more layers built with blocks of mbconv [30], which are combined with a squeeze-and-excitation optimization [22]. table 1. summary of the composition of the efficientnet b0full size table 4 experimental results the methods described previously are only useful if they do lead to improvement in realistic circumstances. we developed and tested them by dealing with real artwork and existing datasets. in this section, we present the dataset we used and its characteristics, the training configurations for both gan and classifier and the results for each step of the class-by-class performance analysis. 4.1 the wikiart dataset we used the wikiart dataset in our experiments. in fact, the version we used was the one discussed by elgammal et al. [14], from which we adopted the following conventions: new realism and contemporary realism were added to realism; action painting was added to abstract-expressionism; synthetic cubism and analytical cubism were added to cubism. table 2. dataset used in experiments.full size table a total of 63,659 images are available there; 10% of them were used for testing and 10% of the remaining dataset was used for validation. the training volumetry of the image distribution is presented in the last column of table 2. the training dataset is used both in classifier training and in gan training. the test dataset is the same for all experiments to ensure comparability of results. the class imbalance is apparent with the volumetry varying between 10,566 and 940 images. 4.2 gan training configuration in order to accommodate the training process in 2 gpus geforce gtx 1080 ti (12 gb) to generate images of 128 \(\times \) 128, we lowered the batch size to 32 images (the sagan was trained with batch size of 256 images). the process ran for 200.000 epochs (approximately 34 h). each class was trained independently, using the equivalent training dataset of the classifier. the optimizer setup follows the original sagan article: adam optimizer with \(\beta _{1} = 0\) and \(\beta _{2} = 0.9\). the learning rate is constant but specific for each model: for the discriminator is 0.0004 and for the generator is 0.0001 [43]. 4.3 efficientnet b0 training configuration using the imagenet pretrained model, the fine-tuning for our classification purpose was done by unfreezing each block until there was no further improvement on the validation dataset. the stochastic gradient descent optimizer was used with decay 0.9 and momentum 0.9; initial learning rate of 0.01 with decay after the fifth epoch (\(lr = lr*e^{-0.1}\)). images were resized to 224 \(\times \) 244 and batch size was 32 images. 4.4 baseline results the result of the efficientnet b0 baseline trained model and the efficientnet b0 trained model with geometric augmentation are shown in table 6 (first and second lines). the performance for each class of the latter is shown in fig. 2. it is important to highlight that low performance is not correlated to image quantity: the ukiyo-e movement – the japanese style – had the least amount of images and the best f1-score. still, the worst performance was obtained by the pop art class with only 1205 images. on the other hand, the second and third worst performances belong to two classes with the largest number of images expressionism and romanticism. fig. 2. analysis of the trained efficientnet b0 with geometric augmentation full size image 4.5 sampling low quantity classes pop art is the fourth lowest class in terms of quantity and images, so the low quantity class sampling strategy was applied. the results are shown in table 3. the classifier that obtained the best accuracy score is not the same as that obtained the best f1-score for the pop art class. this shows that other classes have benefited from the information obtained from the images generated (tables 4 and 5). table 3. summary of experiments with added synthetic pop art imagesfull size table 4.6 sampling high quantity classes both expressionism and romanticism classes have a similar behavior to that of our reference model: they have many more images than most of the classes – more than four times pop art’s image quantity – but low performance. it is interesting to observe that for both classes, the quantity of images that generated the best results was 1/4 of its original training data quantity. the synthetic romantic class had a better f1-score when 3/4 of the quantity of the original data was added, but still didn’t produce a better classifier. table 4. summary of experiments with added synthetic expressionist imagesfull size table table 5. summary of experiments with added synthetic romantic imagesfull size table 4.7 summary of results figure 3 shows how each class performed in the best experiment for each class (pop art, expressionism and romanticism). although the classification problem has many classes to be able to point out correlations, it is possible to observe that the art nouveau class had its best performance with synthetic images from the pop art artistic movement. it is also observed that the pop art class itself performs even better when we have the information of the synthetic images of the artistic movements romanticism and expressionism. fig. 3. the evolution of performance for each class during experiments. full size image table 6 also shows the best result of each efficientnet b0 model trained with synthetic artwork class. the strategy of class-by-class performance analysis allowed us to improve the classifier accuracy by almost 2%. table 6. efficientnet b0 trained models.full size table fig. 4. first row of images contains real pop art painting and second row contains generated pop art painting. full size image fig. 5. first row of images contains real expressionist painting and second row contains generated expressionist painting. full size image fig. 6. first row of images contains real romantic painting and second row contains generated romantic painting. full size image 4.8 generated images figures 3, 4 and 5 show a sample of the images generated in our experiments. to help in understanding the quality of these generated images, real images of corresponding artistic styles were added above. by visual inspection, it is noticeable that the generated images retain general properties of the styles. for the three depicted artistic styles, it is clear that the models are flawed in terms of shape definition, but they are competent in the choice of colors, even in the variety of colors that a style contains and correctly separating grayscale images from colored images (examples in figs. 4 and 5) (fig. 6). 5 conclusion artwork style classification is quite challenging: class imbalance and high diversity within classes and similarities between classes are inevitable given existing art styles. besides, classification performance does not correlate with dataset size. it is thus natural to look for data augmentation strategies; however, straightforward techniques do not generate images that can help distinguish between class styles. in this paper we explored gans for data augmentation in this setting; this is a little explored avenue and one that cannot be taken without some care. to be able to actually use gans to generate artwork-like images with the desirable properties and diversity, we had to develop a working methodology, a specialized network architecture, and an adapted loss function. these technical contributions should be valuable in other settings where artwork is to be classified; moreover, they should be useful in data augmentation scenarios with many classes that are hard to differentiate and that are sensitive to popular existing techniques such as coloring and rotation. we found that our approach and methodology can generate better classifiers. in particular, the combination of self-attention for mode diversity and the wasserstein-gp loss function against vanishing gradient enabled us, in our experiments, to improve the augmentation of the images and consequently increase the accuracy of the model by almost 2 %. notes 1.this feeling can be found in articles on the study of art: https://www.artsy.net/article/alina-cohen-art-movements-matter. references arjovsky, m., chintala, s., bottou, l.: wasserstein gan. arxiv (2017) google scholar  arora, r.s., elgammal, a.: towards automated classification of fine-art painting style: a comparative study. in: proceedings international conference on pattern recognition, pp. 3541–3544 (2012) google scholar  bar, y., levy, n., wolf, l.: classification of artistic styles using binarized features derived from a deep neural network. in: agapito, l., bronstein, m.m., rother, c. (eds.) eccv 2014. lncs, vol. 8925, pp. 71–84. springer, cham (2015). https://doi.org/10.1007/978-3-319-16178-5_5 chapter  google scholar  bianco, s., mazzini, d., napoletano, p., schettini, r.: multitask painting categorization by deep multibranch neural network. expert syst. appl. 135, 90–101 (2019). https://doi.org/10.1016/j.eswa.2019.05.036 article  google scholar  brock, a., donahuey, j., simonyany, k.: large scale gan training for high fidelity natural image synthesis, pp. 1–35 (2018). arxiv google scholar  cetinic, e., lipic, t., grgic, s.: fine-tuning convolutional neural networks for fine art classification. expert syst. appl. 114, 107–118 (2018). https://doi.org/10.1016/j.eswa.2018.07.026 article  google scholar  chen, l., yang, j.: recognizing the style of visual arts via adaptive cross-layer correlation. in: mm 2019 proceedings of the 27th acm international conference on multimedia, pp. 2459–2467 (2019). https://doi.org/10.1145/3343031.3350977 chen, x., duan, y., houthooft, r., schulman, j., sutskever, i., abbeel, p.: infogan: interpretable representation learning by information maximizing generative adversarial nets. in: neural information processing systems, pp. 2180–2188 (2016) google scholar  chu, w.t., wu, y.l.: image style classification based on learnt deep correlation features. ieee trans. multimedia 20, 2491–2502 (2018). https://doi.org/10.1109/tmm.2018.2801718 article  google scholar  condorovici, r.g., florea, c., vertan, c.: automatically classifying paintings with perceptual inspired descriptors. j. visual commun. image represent. 26, 222–230 (2015). https://doi.org/10.1016/j.jvcir.2014.11.016 article  google scholar  daras, g., odena, a., zhang, h., dimakis, a.g.: your local gan: designing two dimensional local attention mechanisms for generative models. proceedings of the ieee computer society conference on computer vision and pattern recognition, pp. 14519–14527 (2020). https://doi.org/10.1109/cvpr42600.2020.01454 eldeen, n., khalifa, m.: detection of coronavirus (covid-19) associated pneumonia based on generative adversarial networks and a fine-tuned deep transfer learning model using chest x-ray dataset (2020). http://www.egyptscience.net elgammal, a., liu, b., elhoseiny, m., mazzone, m.: can: creative adversarial networks, generating “art" by learning about styles and deviating from style norms, pp. 1–22 (2017). arxiv google scholar  elgammal, a., liu, b., kim, d., elhoseiny, m., mazzone, m.: the shape of art history in the eyes of the machine. in: 32nd aaai conference on artificial intelligence, aaai 2018, pp. 2183–2191 (2018) google scholar  farthing, s.: tudo sobre arte. 2 edn. (2018) google scholar  florea, c., toca, c., gieseke, f.: artistic movement recognition by boosted fusion of color structure and topographic description. in: proceedings 2017 ieee winter conference on applications of computer vision, wacv 2017, pp. 569–577 (2017). https://doi.org/10.1109/wacv.2017.69 frid-adar, m., diamant, i., klang, e., amitai, m., goldberger, j., greenspan, h.: gan-based synthetic medical image augmentation for increased cnn performance in liver lesion classification. neurocomputing 321, 321–331 (2018). https://doi.org/10.1016/j.neucom.2018.09.013 article  google scholar  gao, x., tian, y., qi, z.: rpd-gan: learning to draw realistic paintings with generative adversarial network. ieee trans. image process. 29, 8706–8720 (2020). https://doi.org/10.1109/tip.2020.3018856 article  google scholar  goodfellow, i., et al.: generative adversarial networks. commun. acm 63, 139–144 (2014). https://doi.org/10.1145/3422622 article  google scholar  gulrajani, i., ahmed, f., arjovsky, m., dumoulin, v., courville, a.: improved training of wasserstein gans, vol. 2017-decem, pp. 5768–5778 (2017) google scholar  han, c., et al.: synthesizing diverse lung nodules wherever massively: 3d multi-conditional gan-based ct image augmentation for object detection. in: proceedings 2019 international conference on 3d vision, 3dv 2019, pp. 729–737 (2019). https://doi.org/10.1109/3dv.2019.00085 hu, j., shen, l., sun, g.: squeeze-and-excitation networks. in: proceedings of the ieee computer society conference on computer vision and pattern recognition, pp. 7132–7141 (2018). https://doi.org/10.1109/cvpr.2018.00745 karayev, s., et al.: recognizing image style. in: bmvc 2014 proceedings of the british machine vision conference 2014, pp. 1–20 (2014). https://doi.org/10.5244/c.28.122 kastan, d.s., farthing, s.: on color. yale university press, new haven (2018) google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. commun. acm 60, 84–90 (2012). https://doi.org/10.1145/3065386 article  google scholar  lecoutre, a., negrevergne, b., yger, f.: recognizing art style automatically in painting with deep learning. j. mach. learn. res. 77, 327–342 (2017) google scholar  mirza, m., osindero, s.: conditional generative adversarial nets, pp. 1–7 (2014) google scholar  qin, z., liu, z., zhu, p., xue, y.: a gan-based image synthesis method for skin lesion classification. comput. methods prog. biomed. 195 (2020). https://doi.org/10.1016/j.cmpb.2020.105568 rodriguez, c.s., lech, m., pirogova, e.: classification of style in fine-art paintings using transfer learning and weighted image patches. in: 2018, 12th international conference on signal processing and communication systems, icspcs 2018 proceedings, pp. 1–7 (2019). https://doi.org/10.1109/icspcs.2018.8631731 sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv 2: inverted residuals and linear bottlenecks. in: proceedings of the ieee computer society conference on computer vision and pattern recognition, pp. 4510–4520 (2018) google scholar  sandoval, c., pirogova, e., lech, m.: two-stage deep learning approach to the classification of fine-art paintings. ieee access 7, 41770–41781 (2019) article  google scholar  shamir, l., macura, t., orlov, n., eckley, d.m., goldberg, i.g.: impressionism, expressionism, surrealism: automated recognition of painters and schools of art. acm trans. appl. percept. 7 (2010). https://doi.org/10.1145/1670671.1670672 shorten, c., khoshgoftaar, t.m.: a survey on image data augmentation for deep learning. j. big data 6, 1–48 (2019) article  google scholar  suh, s., lee, h., lukowicz, p., lee, y.o.: cegan: classification enhancement generative adversarial networks for unraveling data imbalance problems. neural netw. 133, 69–86 (2021) article  google scholar  tan, m., le, q.v.: efficientnet: rethinking model scaling for convolutional neural networks. in: 36th international conference on machine learning, icml 2019 2019-june, pp. 10691–10700 (2019) google scholar  tan, w.r., chan, c.s., aguirre, h.e., tanaka, k.: ceci n’est pas une pipe: a deep convolutional network for fine-art paintings classification. in: proceedings international conference on image processing, icip 2016-august, pp. 3703–3707 (2016). https://doi.org/10.1109/icip.2016.7533051 tomei, m., cornia, m., baraldi, l., cucchiara, r.: art2real: unfolding the reality of artworks via semantically-aware image-to-image translation, vol. 2019-june, pp. 5842–5852 (2019). https://doi.org/10.1109/cvpr.2019.00600 waheed, a., goyal, m., gupta, d., khanna, a., al-turjman, f., pinheiro, p.r.: covidgan: data augmentation using auxiliary classifier gan for improved covid-19 detection. ieee access 8, 91916–91923 (2020). https://doi.org/10.1109/access.2020.2994762 article  google scholar  wang, z., she, q., ward, t.e.: generative adversarial networks: a survey and taxonomy, pp. 1–41 (2019). arxiv google scholar  wu, j., huang, z., thoma, j., acharya, d., van gool, l.: wasserstein divergence for gans. in: ferrari, v., hebert, m., sminchisescu, c., weiss, y. (eds.) eccv 2018. lncs, vol. 11209, pp. 673–688. springer, cham (2018). https://doi.org/10.1007/978-3-030-01228-1_40 chapter  google scholar  xue, a.: end-to-end chinese landscape painting creation using generative adversarial networks, pp. 3863–3871 (2020). arxiv google scholar  özal yıldırım, pławiak, p., tan, r.s., acharya, u.r.: arrhythmia detection using deep convolutional neural network with long duration ecg signals (2018) google scholar  zhang, h., goodfellow, i., metaxas, d., odena, a.: self-attention generative adversarial networks. in: chaudhuri, k., salakhutdinov, r. (eds.) proceedings of the 36th international conference on machine learning. proceedings of machine learning research, vol. 97, pp. 7354–7363. pmlr (2019) google scholar  zhong, s., huang, x., xiao, z.: fine-art painting classification via two-channel dual path networks. int. j. mach. learn. cybern. 11, 137–152 (2020) article  google scholar  zhu, j.y., park, t., isola, p., efros, a.a.: unpaired image-to-image translation using cycle-consistent adversarial networks, vol. 2017-octob, pp. 2242–2251 (2017) google scholar  zhu, y., ji, y., zhang, y., xu, l., zhou, a.l., chan, e.: machine: the new art connoisseur (2019). http://arxiv.org/abs/1911.10091 download references acknowledgements this work is part of the center for data science with funding by itaú unibanco. the first author thanks itaú unibanco for its generosity in authorizing research activities that led to this work. the second author was partially supported by conselho nacional de desenvolvimento científico e tecnológico (cnpq), grant 312180/2018-7, and by são paulo research foundation (fapesp), grant 2019/07665-4. author information authors and affiliations escola politécnica, universidade de são paulo, são paulo, brazil sarah pires pérez & fabio gagliardi cozman authors sarah pires pérezview author publications search author on:pubmed google scholar fabio gagliardi cozmanview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper pérez, s.p., cozman, f.g. (2021). how to generate synthetic paintings to improve art style classification. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_17 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computer vision gan art style classification publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature explaining biomarker response to anticoagulant therapy in atrial fibrillation: a study of warfarin and rivaroxaban with machine learning models | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper explaining biomarker response to anticoagulant therapy in atrial fibrillation: a study of warfarin and rivaroxaban with machine learning models conference paper first online: 30 january 2025 pp 475–487 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) explaining biomarker response to anticoagulant therapy in atrial fibrillation: a study of warfarin and rivaroxaban with machine learning models download book pdf download book epub adriano veloso  orcid: orcid.org/0000-0002-9177-49549,10, gianlucca zuin  orcid: orcid.org/0000-0002-0429-32809,10 & luan sena  orcid: orcid.org/0009-0000-2849-083911  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 494 accesses abstract atrial fibrillation (af) is a common arrhythmia that originates in the heart’s upper chambers and can lead to serious complications like strokes and systemic embolism due to atrial thrombi. to mitigate these risks, anticoagulants such as warfarin and rivaroxaban are frequently prescribed. this study examines how known biomarkers behave under treatment with either warfarin or rivaroxaban. we developed high-performance models (auroc > 0.94) to distinguish patient subpopulations based on their treatment. additionally, we built models (auroc > 0.97) to differentiate individuals with af from a control group without the condition. using synthetic data generation for training augmentation and explainable machine learning techniques, we analyzed biomarker behavior, uncovering distinct patterns based on whether patients received warfarin or rivaroxaban. our approach provides valuable insights into the critical factors influencing biomarker variations across different treatments, enhancing our understanding of their roles in af management. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others rivaroxaban as an effective alternative to warfarin in a patient with atrial fibrillation, thrombophilia, and left atrial appendage thrombus: a case report article open access 09 april 2017 the safety and efficacy of rivaroxaban compared with warfarin in patients with atrial fibrillation and diabetes: a systematic review and meta-analysis article 08 june 2020 comparative effectiveness of rivaroxaban versus warfarin or dabigatran for the treatment of patients with non-valvular atrial fibrillation article open access 06 september 2017 1 introduction atrial fibrillation (af) is the most prevalent cardiac arrhythmia, impacting millions of people worldwide [31]. its prevalence escalates with age, affecting approximately 10% of individuals by the age of 80 [12]. additionally, underlying cardiac conditions, such as coronary artery disease, significantly heighten the risk of developing af [12]. af is characterized by erratic electrical impulses in the atria, leading to an irregular and rapid heart rhythm that disrupts blood flow, causing stasis and increasing the risk of thromboembolic events such as ischemic stroke and systemic embolism [16]. this condition is associated with a fourfold increase in mortality risk, as evidenced by a population-based study in korea [19]. the primary goal in managing af is to reduce the risk of thromboembolism and stroke, primarily through anticoagulation therapy [3, 17, 20]. warfarin, a traditional anticoagulant, has been extensively used for this purpose but poses challenges due to its food and drug interactions and the need for frequent monitoring [26]. maintaining therapeutic levels of warfarin requires regular international normalized ratio (inr) checks and frequent dose adjustments to mitigate bleeding risks. recently, direct oral anticoagulants (doacs), such as rivaroxaban, have emerged as more favorable alternatives due to their improved pharmacological profiles [1, 26]. machine learning has become a valuable tool in healthcare [9], aiding in the development of robust risk prediction algorithms and enhancing our understanding of disease-related features to improve patient outcomes [33]. these techniques are particularly effective in uncovering patterns and associations in large datasets that may be challenging to discern manually [28]. however, the application of machine learning in healthcare demands a focus on explainability due to the high-stakes nature of medical decision-making. explainability involves articulating the internal mechanisms and outputs of an algorithm in a way that is comprehensible to humans. for healthcare professionals to trust and effectively utilize these tools in clinical settings, they must understand the rationale behind a model’s predictions. for example, when a machine learning model forecasts a high risk of stroke in an af patient, clinicians need to grasp the factors influencing this prediction to make well-informed treatment decisions [14, 21]. trust in these models is built through transparency and interpretability, aligning model predictions with clinical knowledge and patient data [6]. similarly, clear and transparent communication of these predictions fosters patient adherence to treatment plans [10]. in this study, we built a dataset comprising biomarkers from 195 individuals, including 109 in the control group (without af) and 86 patients with af. among the af patients, 47 were using warfarin and 39 were using rivaroxaban. the biomarkers used comprise patient data from characterization, blood count, lipid profile, coagulation, inflammatory and cardiac diseases of the individuals. we built high-performance machine learning models using this data (i) to identify biomarkers that differentiate individuals with af from those without, and (ii) to compare biomarker responses between patients on different anticoagulant therapies (warfarin \(\times \) rivaroxaban). additionally, we conducted ablation studies to better understand the significance of two major biomarker groups (coagulation and inflammatory) given their known strong correlation with af [11, 23]. in summary, we utilize synthetic data generation [32], machine learning algorithms [18], and explainable machine learning [22] to create a framework that provides insights into af pathophysiology and the effects of anticoagulant therapy. 2 related work machine learning is changing healthcare by enabling the analysis of vast amounts of patient data to uncover patterns and insights that improve diagnosis, treatment, and patient outcomes [2, 29, 34]. in [25], authors used convolutional neural networks to predict the evolution of alzheimer’s disease, and explainable machine learning to explain patterns within this evolution. in [5], authors employed raman spectroscopy to identify traces of proteins and developed machine learning models to find which proteins are more related to melanoma. in [4], authors developed a machine learning model for predicting the risk of death in covid-19 patients. these models employ blood count data and present high prognostic performance. authors in [7] developed a new ensemble algorithm to predict the evolution of pain relief in patients suffering from chronic pain. specific patterns in patients with af were already studied using typical procedures in the biomedical literature. in [11], authors showed that platelet and coagulation activation profiles in patients with af were reduced by warfarin or rivaroxaban use and that patients with af using rivaroxaban were less hypocoagulable than patients using warfarin. this means that endogen thrombin potential was less reduced in patients using rivaroxaban compared to those taking warfarin. similarly, authors in [23] showed that patients with af (both warfarin and rivaroxaban groups) presented increased levels of inflammatory cytokines in comparison with controls. the use of rivaroxaban was associated with decreased levels of inflammatory cytokines in comparison with warfarin. on the other hand, patients with af using rivaroxaban presented increased levels of the chemokines (mcp-1 in comparison with warfarin users; mig and ip-10 in comparison with controls). finally, authors in [8] showed that hp levels and hp1-hp2 polymorphism are not associated with af, and authors in [24] provided a basis for the study of inflammatory markers that had not yet been well addressed in af, especially ip-10, besides supporting evidence about molecules that had previously been associated with the disease. we employ the aforementioned findings to select the biomarkers to be included in our models. to the best of our knowledge, our study is the first to use machine learning models to provide a deep understanding of af in terms of its known biomarkers and given the possible therapies. 3 materials and methods let d be a dataset comprising biomarkers from patients, including a control group without af and patients diagnosed with af. specifically, d consists of individuals in the control group and af patients who are either on warfarin or rivaroxaban. further, let \(x = \{ x_1, x_2, \ldots , x_n \}\) be a feature vector containing the biomarker values for each individual. missing values in numerical features are imputed using the mean of the feature within the corresponding class, and categorical features are imputed using the mode of the feature within the same class. formally, for a numerical feature \(x_i\) in class \( c_k \), the imputation is given by: $$\begin{aligned} x_i^{(j)} = \frac{1}{|c_k|} \sum _{x_i \in c_k} x_i \end{aligned}$$ (1) for a categorical feature \( x_i \) in class \( c_k \), the imputation is: $$\begin{aligned} x_i^{(j)} = \text {mode}(x_i \in c_k) \end{aligned}$$ (2) the task is formulated as a binary classification problem, in which we are concerned with two distinct contexts: classifying individuals based on the presence or absence of af, and classifying af patients based on their medication (warfaring \(\times \) rivaroxaban). the goal is to learn a function \(f(x; \phi ) \rightarrow \{0, 1\}\), where x represents the biomarker features and \(\phi \) represents the learning parameters of the model. we perform ablation experiments to evaluate and better understand the impact of different biomarker subsets on classification performance. thus, the evaluated models are trained on three different sets of variables: all biomarkers are used in our models. all biomarkers, excluding coagulation biomarkers. all biomarkers, excluding inflammatory biomarkers. this ablation approach allows us to understand how different groups of biomarkers contribute to the classification task and identify which features are more informative for distinguishing between the patient groups. 3.1 data the dataset for our analysis comprises biomarkers from 195 individuals, including 109 in the control group (without af) and 86 patients with af. among the patients, 47 were using warfarin and 39 were using rivaroxaban. the biomarkers used comprise patient data from characterization, blood count, lipid profile, coagulation, inflammatory and cardiac diseases of the individuals and can be found in table 1. table 1. biomarkers used to build our machine learning models.full size table prior to further analysis, missing feature values for patients were imputed based on their respective classes (control, rivaroxaban, warfarin). our data was divided into two distinct datasets for separate analysis. the first subset \( d_1 \) segregates individuals based on the presence or absence of atrial fibrillation: $$\begin{aligned} y_i = {\left\{ \begin{array}{ll} 1 & \text {if af is present} \\ 0 & \text {if af is absent} \end{array}\right. } \end{aligned}$$ (3) the second subset, \( d_2 \), includes only af patients and further classifies them based on their medication: $$\begin{aligned} y_i = {\left\{ \begin{array}{ll} 1 & \text {if the patient is on warfarin} \\ 0 & \text {if the patient is on rivaroxaban} \end{array}\right. } \end{aligned}$$ (4) as previously mentioned, for numerical features imputation was performed using the mean value of the feature within the class in either \( d_1 \) or \( d_2 \). for categorical features, the mode of the values within the class was used. by imputing missing values based on class-specific statistics we obtain results more closely related with the underlying data distribution, resulting in more realistic and reliable values. synthetic data and training augmentation: to improve the visualization and analysis of explanatory factors, we generate synthetic data and create an expanded dataset \(d'\) using two techniques: gaussian copula [27, 30] and tabular variational autoencoder (tvae) [32]. the gaussian copula method models the dependency structure between variables by transforming the original data into a space where the variables follow a multivariate normal distribution. this involves computing the empirical cumulative distribution function for each variable and transforming them using their inverse of the standard normal to produce data that follows a standard normal distribution. we then proceed to estimate the covariance matrix of these transformed variables to capture their dependencies. new samples are generated from a multivariate normal distribution and are transformed back to the original space using the inverse of the initial transformations. formally, let \(x = (x_1, x_2, \ldots , x_n)\) represent the original data and \(u = (u_1, u_2, \ldots , u_n)\) be the transformed variables that follow a standard normal distribution. the gaussian copula c is defined as: $$\begin{aligned} c(u_1, u_2, \ldots , u_n) = \phi _{\sigma }(\phi ^{-1}(u_1), \phi ^{-1}(u_2), \ldots , \phi ^{-1}(u_n)) \end{aligned}$$ (5) in which \(\phi \) represents the cdf of the standard normal distribution, \(\phi ^{-1}\) its inverse, and \(\phi _{\sigma }\) is the cdf of the multivariate normal distribution with the covariance matrix. in contrast, tvae uses a variational autoencoder to learn the underlying data distribution and generate synthetic samples. it consists of an encoder and a decoder structure, in which the encoder \(q_{\phi }(z|x)\) maps the input data x to a latent representation z using a neural network characterized by a mean \(\mu \) and variance \(\sigma ^2\). formally: $$\begin{aligned} q_{\phi }(z|x) = \mathcal {n}(z; \mu (x), \sigma ^2(x)) \end{aligned}$$ (6) the decoder \(p_{\theta }(x|z)\) then maps the latent representation z back to the original data space, attempting to reconstruct the original data \(\hat{x}\): $$\begin{aligned} p_{\theta }(x|z) = \mathcal {n}(x; \hat{\mu }(z), \hat{\sigma }^2(z)) \end{aligned}$$ (7) we create synthetic data points by sampling from the latent space and feeding it to the decoder. for each technique, synthetic samples equivalent to \(33\%\) of the total number of individuals in each subset are generated, resulting in the expanded dataset \(d'\). while synthetic data generation can raise concerns about the representativeness of the data distribution and the potential for introducing bias in real-world applications [15], validating the model on real datasets helps mitigate these issues and ensures its reliability in real-world scenarios. 3.2 model training for each comparison, the model is trained on three different sets of variables: one including all biomarkers, one excluding coagulation markers, and one excluding inflammatory markers. the target variable corresponds to the group that the individual belongs to. our models were built using an implementation of the lightgbm algorithm [18]. it works by generating a model made up of hundreds of simple decision trees, these trees are integrated into a unified model through boosting [13]. explainability: as a key point in any health domain application, we employ explainability tools to better understand the decision-making process of the model. specifically, we employ shap [22]. we represent how model \(f(x; \phi )\) explains a phenomenon as a d-dimensional vector \(e(f(x)) = {e_1, e_2, . . . , e_d }\) showing which features are contributing to the model’s prediction. specifically, \(e_i\) takes a value that corresponds to the influence that the respective feature \(x_i\) had on the model decision. this influence is quantified by calculating shap values, which provide a unified measure of feature importance across different instances. shap values are calculated using shapley values from cooperative game theory, ensuring that each feature’s contribution to the prediction is fairly attributed. for each prediction, the shap values are computed by considering all possible combinations of feature inputs, thus capturing the complex interactions between features. in particular, shap contains three desirable properties: additivity: the explanations are truthfully explaining the model and each evaluated instance. this means that the sum of the shap values for all features equals the model output for that instance, ensuring that the attributions are consistent with the model’s actual predictions. consistency: if a model changes such that some feature’s contribution increases or stays the same regardless of the other features, that feature’s attribution should not decrease. this property ensures that shap values remain stable and interpretable even if the underlying model changes. missingness: a missing feature gets an attribution of zero. that is, a feature not present in a model does not impact its output or explanation. 4 experimental results model performance was evaluated by the area under the receiver operating characteristic curve (auroc). it is used to show how much the model is capable of distinguishing between competing classes (i.e., patient groups). in our experiments we employed stratified 5-fold cross-validation by dividing the dataset into five parts and, at each iteration, we use four folds for training and the remaining one as the testing set for the model. no hyperparameter tuning was performed. following, we discuss the results of each study. 4.1 atrial fibrillation and control group table 2 shows the model performance in differentiating af patients from the control group, with all models achieving high auroc values. literature suggests that physiological patterns in af patients are markedly different from those in the control group, which the trained models were able to learn. given the high performance of our models, our main interest lies in understanding these distinctive patterns. we used shap values for this analysis. table 2. model performance on atrial fibrillation and control group classification.full size table when using shap to find the most significant features using all biomarkers, we found a great impact of il-4, peak, mpes/ul, il-2, etp, f1+2, ldlc, ifn, trig, il-10, tgf\(\beta \), and hdlc in model prediction (fig. 1 (top)). if we exclude the coagulation biomarkers the most important features are il-4, il-10, ifn, total cholesterol, il-6, sicam-1, ldlc, mcp-1, il-2, il-8, gdf-15, and hdlc (fig. 1 (middle)). furthermore, without inflammation biomarkers the most impacting features are peak, tafi, f1+2, lagtime, trig, mpes/ul, pai-1, total cholesterol, sicam-1, gdf-15, mpp/ul and etp (fig. 1 (bottom)). these findings suggest that inflammation, lipid metabolism, and coagulation processes are all critical in the model’s predictions. without coagulation biomarkers, cytokines and lipid measures remain important, indicating that inflammation and lipid metabolism are still significant. additional markers like sicam-1 and mcp-1 become more prominent, highlighting the role of inflammation and immune response. without inflammation biomarkers, coagulation and lipid measures (such as tafi, f1+2, lagtime, pai-1) are significant. this shift underscores the importance of blood clotting mechanisms and lipid metabolism. fig. 1. (top) all features. (middle) all, but coagulation biomarkers. (bottom) all, but inflammatory biomarkers full size image 4.2 warfarin \(\times \) rivaroxaban table 3 shows the model performance in differentiating af patients on warfarin from af patients on rivaroxaban. again, all models achieved very high auroc values, and given the high performance of our models, our main interest lies in understanding the distinctive patterns in both groups. we used shap values for this analysis. table 3. model performance on rivaroxaban and warfarin group classification.full size table fig. 2. (top) all features. (middle) all, but coagulation biomarkers. (bottom) all, but inflammatory biomarkers full size image when using shap to find the most significant features using all biomarkers, we found a great influence of il-2, f1+2, etp, gdf-15, sicam-1, rantes, tnf, t-pa, il-4, tgf\(\beta \), plt, and il-8 in model prediction (fig. 2 (top)). when excluding the coagulation biomarkers the most impacting features became il-2, il-4, genotype hp (1), il-6, tgf\(\beta \), ifn, haptoglobin (hp), mcp-1, plt, dyslipidemia (hypercholesterolemia), trig and has (fig. 2 (middle)). furthermore, without inflammation biomarkers the most impating features are f1+2, etp, ttpeak, t-pa, peak, dyslipidemia (hypercholesterolemia), trig, plt, total cholesterol, hdlc, ldlc, and tafi (fig. 2 (bottom)). 5 conclusion this study underscores the potential of machine learning models in providing a deeper understanding of atrial fibrillation (af). by leveraging ml models, we successfully identified and analyzed key biomarkers that differentiate af patients from healthy individuals and elucidate the effects of different anticoagulant therapies. our high-performance models demonstrate robust predictive capabilities, offering significant insights into the distinct biomarker patterns associated with warfarin and rivaroxaban treatments. the application of explainable machine learning methods enabled us to uncover critical factors influencing biomarker behavior, providing a deeper understanding of the pathophysiological processes in af and the impacts of specific treatments (we can clearly see patients using rivaroxaban as having lower inflammation markers in contrast to those using warfarin). these insights contribute to a more nuanced view of how anticoagulant therapy affects biomarker profiles, which can inform clinical decision-making and optimize treatment strategies. finally, synthetic data generation proved invaluable in augmenting our dataset, enhancing model robustness, and ensuring reliable predictions despite potential limitations in real-world data availability. this approach highlights the efficacy of combining data augmentation techniques with machine learning to overcome data scarcity issues common in clinical research. references alberts, m., chen, y.w., lin, j.h., kogan, e., twyman, k., milentijevic, d.: risks of stroke and mortality in atrial fibrillation patients treated with rivaroxaban and warfarin. stroke 51(2), 549–555 (2020) article  google scholar  amador, t., saturnino, s., veloso, a., ziviani, n.: early identification of icu patients at risk of complications: regularization based on robustness and stability of explanations. artif. intell. med. 128, 102283 (2022) article  math  google scholar  amin, a., houmsse, a., ishola, a., tyler, j., houmsse, m.: the current approach of atrial fibrillation management. avicenna j. med. 6(01), 8–16 (2016) article  google scholar  araújo, d.c., veloso, a.a., borges, k.b.g., das graças carvalho, m.: prognosing the risk of covid-19 death through a machine learning-based routine blood panel: a retrospective study in brazil. int. j. med. inf. 165, 104835 (2022) google scholar  araújo, d.c., et al.: finding reduced raman spectroscopy fingerprint of skin samples for melanoma diagnosis through machine learning. artif. intell. med. 120, 102161 (2021) google scholar  bayer, s., gimpel, h., markgraf, m.: the role of domain expertise in trusting and following explainable ai decision support systems. j. decis. syst. 32(1), 110–138 (2022) article  math  google scholar  costa, a.b.d., moreira, l., andrade, d.c.d., veloso, a., ziviani, n.: predicting the evolution of pain relief: ensemble learning by diversifying model explanations. acm trans. comput. heal. 2(4), 36:1–36:28 (2021) google scholar  costa, l.b.x., et al.: evaluation of serum haptoglobin levels and hp1-hp2 polymorphism in the haptoglobin gene in patients with atrial fibrillation. molec. biol. rep. 49, 7359–7365 (2022) google scholar  deo, r.c.: machine learning in medicine. circulation 132(20), 1920–1930 (2015) article  math  google scholar  dimatteo, m.r., et al.: physicians’ characteristics influence patients’ adherence to medical treatment: results from the medical outcomes study. health psychol. 12(2), 93 (1993) article  google scholar  duarte, r.c.f., et al.: thrombin generation and other hemostatic parameters in patients with atrial fbrillation in use of warfarin or rivaroxaban. j. thrombosis thrombolysis 51, 47–57 (2021) google scholar  fang, m.c., chen, j., rich, m.w.: atrial fibrillation in the elderly. am. j. med. 120(6), 481–487 (2007) article  math  google scholar  freund, y., schapire, r.e.: a decision-theoretic generalization of on-line learning and an application to boosting. j. comput. syst. sci. 55(1), 119–139 (1997) article  mathscinet  math  google scholar  gerlings, j., jensen, m.s., shollo, a.: explainable ai, but explainable to whom? an exploratory case study of xai in healthcare. in: handbook of artificial intelligence in healthcare: practicalities and prospects, vol. 2 pp. 169–198 (2022) google scholar  giuffrè, m., shung, d.l.: harnessing the power of synthetic data in healthcare: innovation, application, and privacy. npj dig. med. 6(1), 186 (2023) google scholar  hindricks, g., et al.: esc scientific document group: 2020 esc guidelines for the diagnosis and management of atrial fibrillation developed in collaboration with the european association for cardio-thoracic surgery (eacts): the task force for the diagnosis and management of atrial fibrillation of the european society of cardiology (esc) developed with the special contribution of the european heart rhythm association (ehra) of the esc. eur. heart j. 42(5), 373–498 (2021) article  google scholar  joglar, j.a., et al.: 2023 acc/aha/accp/hrs guideline for the diagnosis and management of atrial fibrillation: a report of the american college of cardiology/american heart association joint committee on clinical practice guidelines. circulation 149(1), e1–e156 (2024) article  mathscinet  math  google scholar  ke, g., et al.: lightgbm: a highly efficient gradient boosting decision tree. in: guyon, i., luxburg, u.v., bengio, s., wallach, h., fergus, r., vishwanathan, s., garnett, r. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017) google scholar  lee, e., et al.: mortality and causes of death in patients with atrial fibrillation: a nationwide population-based study. plos one 13(12), e0209687 (2018) google scholar  lip, g.y., tse, h.f.: management of atrial fibrillation. lancet 370(9587), 604–618 (2007) article  math  google scholar  lötsch, j., kringel, d., ultsch, a.: explainable artificial intelligence (xai) in biomedicine: making ai decisions trustworthy for physicians and patients. biomed. inf. 2(1), 1–17 (2021) math  google scholar  lundberg, s.m., lee, s.i.: a unified approach to interpreting model predictions. in: guyon, i., luxburg, u.v., bengio, s., wallach, h., fergus, r., vishwanathan, s., garnett, r. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017) google scholar  martins, g.l., et al.: comparison of inflammatory mediators in patients with atrial fibrillation using warfarin or rivaroxaban. front. cardiovasc. med. 7, 47–57 (2020) google scholar  martins, g.l., et al.: evaluation of new potential inflammatory markers in patients with nonvalvular atrial fibrillation. int. j. molec. sci. 24, 3326 (2023) google scholar  nigri, e., ziviani, n., cappabianco, f.a.m., antunes, a., veloso, a.: explainable deep cnns for mri-based diagnosis of alzheimer’s disease. in: 2020 international joint conference on neural networks, ijcnn 2020, pp. 1–8. ieee (2020) google scholar  patel, m.r., et al.: rivaroxaban versus warfarin in nonvalvular atrial fibrillation. new engl. j. med. 365(10), 883–891 (2011) google scholar  patki, n., wedge, r., veeramachaneni, k.: the synthetic data vault. in: 2016 ieee international conference on data science and advanced analytics (dsaa), pp. 399–410 (2016) google scholar  phillips, p.j., et al.: four principles of explainable artificial intelligence (2021) google scholar  silcox, c., et al.: the potential for artificial intelligence to transform healthcare: perspectives from international health leaders. npj dig. med. 7(1) (2024) google scholar  sun, y., cuesta-infante, a., veeramachaneni, k.: learning vine copula models for synthetic data generation (2018) google scholar  wyndham, c.r.: atrial fibrillation: the most common arrhythmia. tex. heart inst. j. 27(3), 257 (2000) google scholar  xu, l., skoularidou, m., cuesta-infante, a., veeramachaneni, k.: modeling tabular data using conditional gan. in: wallach, h., larochelle, h., beygelzimer, a., d’alché-buc, f., fox, e., garnett, r. (eds.) advances in neural information processing systems, vol. 32. curran associates, inc. (2019) google scholar  zuin, g., et al.: prediction of sars-cov-2-positivity from million-scale complete blood counts using machine learning. commun. med. 2(1), 72 (2022) article  google scholar  zuin, g., et al.: a modified louvain approach for medical community detection using geographic data. in: 2023 ieee 36th international symposium on computer-based medical systems (cbms), pp. 143–148. ieee (2023) google scholar  download references author information authors and affiliations computer science department, universidade federal de minas gerais, belo horizonte, brazil adriano veloso & gianlucca zuin kunumi, belo horizonte, brazil adriano veloso & gianlucca zuin electrical engineering department, universidade federal de minas gerais, belo horizonte, brazil luan sena authors adriano velosoview author publications search author on:pubmed google scholar gianlucca zuinview author publications search author on:pubmed google scholar luan senaview author publications search author on:pubmed google scholar corresponding author correspondence to gianlucca zuin . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper veloso, a., zuin, g., sena, l. (2025). explaining biomarker response to anticoagulant therapy in atrial fibrillation: a study of warfarin and rivaroxaban with machine learning models. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_33 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_33 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords atrial fibrillation explainable machine learning medical data analysis publish with us policies and ethics profiles gianlucca zuin view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature incorporating text specificity into a convolutional neural network for the classification of review perceived helpfulness | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper incorporating text specificity into a convolutional neural network for the classification of review perceived helpfulness conference paper first online: 28 november 2021 pp 480–495 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) incorporating text specificity into a convolutional neural network for the classification of review perceived helpfulness download book pdf download book epub beatriz lima10 & tatiane nogueira10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1248 accesses abstract reviews are valuable sources of information to support the decision making process. therefore, the task of classifying reviews according to their helpfulness has paramount importance to facilitate the access of truly informative content. in this context, previous studies have unveiled several aspects and architectures that are beneficial for the task of review perceived helpfulness prediction. the present work aims to further investigate the influence of the text specificity aspect, defined as the level of details conveyed in a text, with the same purpose. first, we explore an unsupervised domain adaptation approach for assigning text specificity scores for sentences from product reviews and we propose an evaluation measure named specificity prediction evaluation (spe) in order to achieve more reliable specificity predictions. then, we present domain-oriented guidelines on how to incorporate, into a cnn architecture, either hand-crafted features based on text specificity or the text specificity prediction task as an auxiliary task in a multitask learning setting. in the experiments, the perceived helpfulness classification models embodied with text specificity showed significant higher precision results in comparison to a popular svm baseline. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others sentiment analysis of text classification using rnn algorithm chapter © 2021 active neural learners for text with dual supervision article 01 january 2020 text feature extraction and classification based on convolutional neural network (cnn) chapter © 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization computational linguistics machine learning natural language processing (nlp) predictive markers artificial intelligence 1 introduction reviews written by consumers on e-commerce platforms are valuable resources in the decision making process of product purchase. nowadays, it is hard to imagine buying a product without reading at least some reviews first. moreover, online product reviews are an essential part of the acquisition of business intelligence and market advantage for the retailers [17, 22] once they can retrieve informative evaluations about several aspects of a product. due to the great number of reviews constantly posted, many websites provide a voting system through which consumers can express whether they perceive a review as helpful or not. these votes are frequently summarized to the users with a message next to the review, similar to “h of t people found this helpful”. this message means that from t people who evaluated this comment, h found the review helpful. from the helpfulness votes, the perceived helpfulness of a review can be derived, which is characterized as the majority impression of the consumers about the review utility during the purchase decisions. the name “perceived helpfulness” is used to emphasize that we are tackling a definition obtained from the helpfulness votes rather than some expert judgment [1, 12, 17]. given this concept, the helpfulness score of a review is calculated as \(\theta = \frac{h}{t}\). having all posted comments with a helpfulness score assigned is beneficial for consumers and merchants in several ways, including as a dimension to rank them. since most of the reviews do not receive any votes at all [13], especially for less popular and recently released products, several reviews cannot be graded with a perceived helpfulness score. therefore, this scenario has been fruitful for researchers to study what makes a review helpful and how to build intelligent systems that can distinguish reviews with different levels of helpfulness. these studies have mainly concentrated their efforts on feature engineering since several complex factors can contribute to the performance of a model in the helpfulness prediction task. subjectivity [1, 6, 8] and readability [6, 7] are some of the many aspects that have been investigated in the last years. more recently, hand-crafted features based on text specificity, which is defined as the level of details about a subject expressed in a text [14], also proved to be interesting attributes for the classification of reviews according to their helpfulness [12]. text specificity has been widely used to assess the writing quality of a text. in the field of automatically generated summaries, [15] argued that the quantity of specific and general sentences, as well as how sequences of them are disposed, affect the writing quality of a text. the quality of a text is also related to how easy it is to understand its content, i.e. what is its level of readability. this aspect has also been considered important for the helpfulness prediction task [6, 7]. based on the aforementioned points, we argue that text specificity may play an important role in helpfulness prediction. the first part of our proposal aims to combine the specificity based features proposed in [12] into a well known convolutional neural network (cnn) architecture [9]. cnns have been successfully employed in some earlier helpfulness prediction studies by reducing the need for labor-intensive feature engineering and by achieving good performance [1, 3, 4, 20]. secondly, we propose to integrate the specificity prediction as an auxiliary task in a multitask learning (mtl) version of the same cnn architecture used in the first part. mtl can be used to improve model generalization and learning efficiency by sharing underlying common structures between complementary tasks [2]. the multitask paradigm has recently also shown promising results in the helpfulness prediction task [4]. the specificity based features are derived from the specificity degree, which is assigned to each review using an unsupervised domain adaptation technique [10]. although [12] proposed these features, the study did not discuss how different experimental setups, beyond the recommended by the authors of the domain adaptation method, can affect the specificity degree predictions of online product reviews. therefore, the present study also aims to bridge this gap by empirically finding more suitable experimental settings for our domains. to evaluate the specificity scores predicted in this unsupervised context, we also propose a measure named specificity prediction evaluation (spe), which is calculated as the average length of general sentences over the average length of specific sentences. in a nutshell, our contribution is three-fold: 1) a better understanding of how to obtain more reliable specificity prediction results in the domain of product reviews; 2) proposing a method for the evaluation of specificity predictions of unlabeled sentences, and 3) presenting domain-oriented guidelines on how to incorporate the specificity aspect into a cnn architecture. the experiments showed that the cnn-based models embodied with text specificity could achieve significant higher precision results in comparison to a popular svm baseline. the rest of this paper is structured as follows. section 2 presents some of the related studies on perceived helpfulness prediction. section 3 explains how did the review specificity predictions were obtained and how they were evaluated using the spe. our proposed perceived helpfulness classification model using a cnn architecture and text specificity is presented in sect. 4. details about the experiments are defined in sect. 5 and their results are discussed in sect. 6. section 7 summarizes the concluding remarks and future work. 2 related work one of the first studies to investigate the helpfulness prediction task is in [8]. this work proposed a regression task in which the target was the helpfulness score based on the votes received by the review. one of its main goals was to analyze the performance of five categories of features for this task: structural, lexical, syntactic, semantic, and metadata. they reported that the best combination of features was the length of the review, the lemmatized unigrams, and the star rating score given by the review author to the product. the effect of review length on perceived helpfulness prediction has also been much explored by past researches [6,7,8, 17]. according to [8], in the reviews used in their experiments, among the longest reviews the average helpfulness score was 0.82 while the average score among the shortest ones was only 0.23. [17] also empirically demonstrated that reviews with more words have a higher helpfulness degree. therefore, the researchers have been arguing that longer reviews are likely to contain more information and, consequently, can influence more readers to vote them as “helpful” [18]. besides the review extensiveness, how well the text is organized and written, to facilitate the understanding of its content, has also been considered an important factor. in [7] the behavior of the review length as a determinant of review helpfulness was further investigated and they revealed some factors that moderate the relation between length and helpfulness. according to their experiments, review length has a positive influence on helpfulness only when the readability is high. additionally, [6] reported that not only review readability has an impact on review helpfulness, but also it influences the product sales. in their experiments, for some types of products, having reviews with higher readability scores is related to higher sales of such products. this elucidates the importance of studying the influence of factors related to writing quality on review helpfulness. based on the previous studies, [12] started to investigate the influence of text specificity in the perceived helpfulness context. this work reported that combining lemmatized unigrams with three hand-crafted features, derived from text specificity, either outperformed or showed similar performance on more than 80% of the results, when compared to models using only the unigrams. at the opposite side, using only the features based on sentence specificity achieved the lowest results. the ideas developed in this work have inspired the present study and, therefore, the features proposed by its authors are used to evaluate our proposal to incorporate them into a cnn architecture. several machine learning algorithms have been used to model the helpfulness prediction task, including cnns. [3] proposed a novel end-to-end cross-domain architecture for helpfulness prediction using word-level and character-based representations. their cnn architecture is based on the one designed by [9] which they further adjusted with auxiliary domain discriminators to transfer knowledge between domains, using large datasets as the source domains and small ones as the target domains. in their experiments, the cnn models yielded better results than known baselines which used hand-crafted features. likewise, [1] proposed a helpfulness classification model by combining convolutional layers and gated recurrent units. instead of using embeddings, they selected 100 linguistic and meta data features, based on prior work, to form the input review representation. their approach outperformed other classification methods by 0.04 and 0.02 in terms of f1-measure and accuracy, respectively. due to the successful usage of mtl on a wide range of tasks, [4] introduced an end-to-end mtl architecture to learn at the same time the helpfulness classification of online product reviews and the star rating given by the reviewer to the product. the architecture consists of convolutional layers, equipped with an attention mechanism, which receives as input the concatenation of word embeddings and their character-level encoding. the main purpose of this model is to jointly learn the helpfulness classification task (main task) by using the hints learned from the star rating prediction task (auxiliary task). using amazon reviews, the study empirically validated that the model achieved a superior performance than single-task learning (stl) models similar to the proposed mtl model, as well as prior techniques using hand-designed features. 3 specificity prediction on sentences from reviews the text specificity prediction task aims to build machine learning models that can learn, from examples, to assign a specificity degree to a text representation, usually on sentence-level [10, 11, 14, 15]. even though different approaches have been proposed in the last years, the majority of them had tackled the problem in a supervised way, which requires extensive manual annotation. to overcome this matter, a robust unsupervised domain adaptation solution was proposed in [10] to predict the specificity score of unlabeled domains. the unsupervised domain adaptation technique consists of using labeled sentences from a source domain to assign reliable specificity scores (between 0 and 1) to unlabeled sentences from a different target domain, without access to any labeled example from the target domain. [10] reported good generalization results when evaluated their approach with social media posts, restaurant reviews, and movie reviews as the target domains. the source data is a publicly available dataset with about 4,300 labeled news sentences [14]. this approach is based on the self-ensembling of a single base model, which combines hand-crafted features (sentence length, average word length, percentage of strongly subjective words, etc.), word embeddings and bilstm (bidirectional long short-term memory) to encode the input sentences. because of the good performance of this technique on review domains (restaurant and movie reviews), it was employed by the present work to infer the specificity degree of the unlabeled sentences from our product reviews. although this solution was also exploited in [12] to predict the specificity of reviews’ sentences, the study did not report any broader analysis about these experiments. in summary, for each dataset (for example, reviews about books), they have sampled 4,300 sentences as the training data (the same amount of sentences of the source news data) and trained the model for 30 epochs, which is the same value that [10] reported for their best results. however, during our experiments, we detected more robust specificity degrees when increasing the training set and the number of epochs. the best settings regarding these two aspects for our domains are further discussed in sect. 6, along with more details about the experiments. one of the features used in the base model introduced in [10] is the sentence length. their labeled and unlabeled datasets (source and target data) have sentences with lengths varying between 15 and 23 tokens, on average. therefore, despite coming from different domains, the sentences used in their experiments have close lengths. likewise, the datasets used in our experiments contain reviews with sentences varying, on average, between 17 and 22 tokens. however, while the longest sentence from the target domains used by [10] has 202 tokens, there are many much longer sentences within our datasets. this may be caused by noises in the reviews, which, even after several preprocessing techniques, may be harming the sentence splitting process. still, we are assuming, for now, this is an intrinsic characteristic of these datasets. we noticed that this aspect was negatively affecting the specificity prediction results when training for few epochs or with few sentences. in many cases, the longest sentences were scored with the lowest specificity degrees and the opposite for the shortest sentences. after trying different experimental settings, the models could achieve more robust results by increasing the training size and the number of epochs. given the problems observed in the experiments, to systematically choose a trained model that could designate the most reliable specificity scores for each target domain of product reviews, we propose a measure called specificity prediction evaluation (spe). considering \(\sigma \) as the specificity degree of a sentence, general sentences having \(\sigma < 0.5\), and specific sentences having \(\sigma \ge 0.5\). spe is based on one assumption: on average, reliable specific sentences tend to be longer than reliable general sentences [11]. this assumption is supported by related past studies and by examining the datasets used in the experiments performed in [10]. therefore, spe is calculated as follows: $$\begin{aligned} spe = {\left\{ \begin{array}{ll} 0,&{} \text {if } |g| = 0 \text { or } |s| = 0 \\ \frac{l_{avg}(g)}{l_{avg}(s)}, &{} \text {otherwise} \end{array}\right. } \end{aligned}$$ (1) where g is the set of general sentences, s is the set of specific sentences, |g| denotes the quantity of general sentences, |s| refers to the quantity of specific sentences, \(l_{avg}(g)\) denotes the average length of general sentences, and \(l_{avg}(s)\) is the average length of general sentences. we assume \(spe < 1\) is a satisfactory value because, in this case, the average length of specific sentences is longer than the general ones. in fact, the datasets from [10] have spe values varying between 0.52 and 0.74. in addition to that, it is important to highlight that the text specificity aspect is composed by several other text characteristics beyond the length. for that reason, spe is a first effort to evaluate the specificity scores without having labeled sentences and, in future endeavors, it should be applied along with other evaluation approaches. after the improvements performed to get more reliable specificity predictions, we extracted the same three hand-crafted features proposed in [12]: specificitydegree, the review-level specificity degree; specificsents%, the percentage of specific sentences in the review; and specificitybalance, which is a balance measure between the number of specific and general sentences in the review. the values of all these features lie in the range [0, 1]. the next section explains how these features were incorporated into a cnn architecture for the perceived helpfulness classification task. 4 perceived helpfulness classification model for a review r, let \(h_r\) be the number of helpful votes received by the review and \(t_r\) be the total number of votes given to r. the helpfulness ratio of the review r is calculated as \(\theta _r = \frac{h_r}{t_r}\). to transform the helpfulness ratio into binary labels, we set the threshold value to 0.5, which is a common value used on related studies [4]. if \(\theta _r > 0.5\), r is categorized as “very helpful”, otherwise if \(\theta _r \le 0.5\), r is said to be “poorly helpful”. in summary, the task consists of classifying reviews into two categories: “very helpful” and “poorly helpful”. based on the successful application of cnn architectures on previous related works [1, 3, 4, 20], this type of neural network was employed here to validate the influence of text specificity on perceived helpfulness. the usage of convolutional and pooling layers is known to be successful in several text classification tasks when there are strong local clues (sequence of tokens) that could appear in different locations of the texts. for example, in the context of product reviews, expressions such as “i did not like” or “i recommend” could be frequent sequences that are important to distinguish reviews with different levels of helpfulness. our helpfulness classification models have a base cnn classification architecture which is the single channel architecture proposed in [9] for sentence classification. this architecture has achieved good performance across many different tasks, including sentiment classification. it consists of a single convolutional layer, with 3 types of filter of kernel sizes 3, 4, and 5 and 100 filters each, followed by a max-pooling layer. each filter processes all the possible windows of words, according to its kernel size, from the input review representation, and generates a feature map. then, the max-pooling operation captures, for each feature map, the feature with the highest value, which is presumably the most important feature in the map. finally, the learned pooled vector is passed to a fully connected layer that computes the model output. figure 1 illustrates a review representation at different steps of the classification process using this cnn design. the input representation is a \(l \times d\) matrix where d is the dimension of the embedding vectors used to represent each word in the review and l is the text length which is a fixed value for all the reviews. truncation and zero padding is applied to the reviews when necessary. fig. 1. the input, outputs, and intermediate representations in the cnn architecture proposed on [9] and adjusted for our proposal. full size image to incorporate the text specificity aspect into the base cnn structure, we propose two approaches: i) using the specificity based features, and ii) jointly learning the specificity prediction and the helpfulness classification task in a mtl setting. concerning the features, they are simply concatenated into the pooled vector. then, the classification output is computed using not only the cnn learned representation but also the specificity hand-crafted features. regarding the mtl scheme, the goal of the model is to learn all the network parameters by minimizing the joint loss presented on eq. 2. here the weights \(\lambda _h\) and \(\lambda _s\) were empirically set to 1 after some initial experiments. $$\begin{aligned} \mathcal {l}= \lambda _h \mathcal {l}_h(\hat{y}_h, y_h) + \lambda _s \mathcal {l}_s(\hat{y}_s, y_s) \end{aligned}$$ (2) where \(\mathcal {l}_h(\hat{y}_h, y_h)\) is the loss of the helpfulness classification task and \(\mathcal {l}_s(\hat{y}_s, y_s)\) is the loss of the specificity prediction task. review specificity prediction is a regression task that consists of learning to predict a specificity score between 0 and 1 for each review. we tackled this problem as a supervised task by using, as the true labels, the review specificity degrees obtained with the approach explained in sect. 3. after assigning specificity scores for each sentence in the review, the review specificity degree is calculated as the average sentence specificity degree. 5 experimental setup 5.1 data and preprocessing the experiments were performed with public datasets of amazon reviews written in english from 9 different product categories [16]. each dataset originally contains at least five reviews per product and per user. we decided to use amazon reviews in the experiments to be consistent with most of the studies from the related literature [1, 3,4,5,6,7,8, 12, 17, 20]. table 1 shows some relevant characteristics of these datasets. table 1. characteristics of the amazon datasets after preprocessing. the column # reviews is the total amount of reviews, v is the vocabulary size, \(l_{avg}\) is the average review length, \(l_{min} l_{max}\) is the range of review length, \(p_{avg}\) is the average amount of sentences, \(p_{min} p_{max}\) is the minimum and maximum amount of sentences in the reviews, and % very helpful is the percentage of reviews with the class “very helpful”. all values, except from the last column, were computed after data balancing.full size table before using these collections as inputs to our classification models, some cleaning procedures were applied to reduce the negative effect of noisy data in the experiments. this preprocessing step is specially important to improve tokenization and splitting the comments into sentences, which were done here with the help of the spacy libraryfootnote 1. the procedures employed to clean the datasets and to improve sentence boundaries are listed below: replace urls by the placeholder token “url”; remove ascii symbols, double quotes, simple emoticons (e.g. “=)”), and empty brackets (e.g. “()”); replace ellipsis by a single period mark (“...” \(\longrightarrow \) “.”) add a blank space after every comma or period mark when it was followed by a word, a special symbol or a number (if the token is a number, the token before the comma or the period must not be a number as well); replace dots by hyphens on enumerations (e.g.: “1. item” \(\longrightarrow \) “1 item”). besides the previous steps, invalid reviews with more helpful votes than the total amount of votes (\(h > t\)) were discarded. also, it was removed the reviews with \(t = 0\) because the helpfulness score \(\theta \) cannot be calculated for such reviews. regarding the datasets with the largest sizes in their original form (books, cds and vinyl, electronics, and movies and tv), reviews with \(t < 5\) were removed in order to increase the confidence in the perceived helpfulness scores as more people has evaluated the selected reviews. this type of cut-off is a known approach among the prior related studies [8]. additionally, reviews with \(0.3< \theta < 0.7\) were removed because this approach seems to avoid dealing with reviews that received almost the same number of helpful and not helpful votes, i.e. the reviews with a more uncertain perception of helpfulness [12]. these amazon datasets are known to be highly imbalanced [12, 13] in which the class “very helpful” is the majority one. handling the challenges related to learning on imbalanced scenarios is out of the scope of the present work, so data balancing was applied by undersampling the majority class. 5.2 training even though all collections have product reviews from amazon, each dataset was analyzed as a distinct domain because past studies showed that the product type influences how people perceive helpfulness [17]. therefore, in the experiments, the models were independently trained for each domain. the code was developed with the aid of kerasfootnote 2, tensorflowfootnote 3 and scikit-learnfootnote 4 libraries. all the random seeds were set to 42 during the experiments. the results reported in sect. 6 were obtained using stratified 5-fold cross-validation. the classifiers embodied with the text specificity aspect were also statistically compared against two baselines with the non-parametric friedman test, considering \(p-value < 0.05\), and the post-hoc nemenyi test. regarding the cnn hyperparameters, valid padding was performed and the stride was set to 1. in comparison to the original proposal [9], we used much less regularization, a dropout of 0.1 in the penultimate layer and l2-norm regularization of 0.01, because of observed performance improvements after some initial experiments. consequently, the number of training epochs was decreased to two since the models would overfit beyond this point. training was performed with the adam optimizer using \(\eta = 0.001\). for the helpfulness classification task, it was used the binary cross entropy loss and the relu activation function. for the auxiliary specificity prediction task, it was used mean squared error and sigmoid. relu was also the chosen activation in the convolutional layers. for every cnn-based model, similar to experiments performed in [4, 9], we tested two ways of initializing the 300-dimensional embeddings: randomly and by using pretrained glove [19] embeddings which were trained on a large 42 billion token corpus. in either approach, the word embeddings were fine-tuned during training time. the dimension size of 300 was defined because of its good performance when using cnn on other text classification tasks [21]. due to the broad range of review sizes among the amazon datasets, it was also tested two maximum input length values for each domain: the mean review length and 512 tokens. in fact, other values were tested but we noticed more interesting results when comparing these two values. the batch size was set to 32 when using the mean length and to 16 when using a length of 512 due to memory constraints. 5.3 model variations the following models were compared during our experiments: baselines: \(\bullet \): svm: a linear svm model trained with all the lemmatized unigrams weighted with tf-idf. this is a popular baseline based on previous works on helpfulness prediction [1, 4, 8, 12, 13]. \(\bullet \): stl cnn: the single-task cnn model, without the specificity features. cnn combined with text specificity (cnn+specificity): \(\bullet \): mtl cnn: the multitask cnn model, without the specificity features. \(\bullet \): specificitydegree: the cnn model embodied with the specificitydegree feature. \(\bullet \): specificsents%: the cnn model embodied with the specificsents% feature. \(\bullet \): specificitybalance: the cnn model embodied with the specificitybalance feature. \(\bullet \): all features: the cnn model embodied with all the three features (specificitydegree, specificsents%, specificitybalance). with respect to the models specificitydegree, specificsents%, specificitybalance, and all features, both stl and mtl architectures were tested. 6 results and discussion 6.1 perceived helpfulness classification table 2 summarizes the best f1 results obtained with the stl cnn and the mtl cnn models. together with the f1 measure, we also report which combination of embeddings initialization approach (randomly or with glove pre-trained vectors) and maximum input length (mean length or 512 tokens) yielded such performance. finally, the table also shows the glove embeddings coverage for each domain, i.e. the percentage of all words for which a pre-trained glove embedding was found. the closer this coverage is to 100%, the smaller is the amount of out-of-vocabulary (oov) words, which may affect the performance. table 2. comparison results of the stl cnn and mtl cnn modelsfull size table the results indicate that jointly learning the text specificity prediction task with the perceived helpfulness classification task could enhance the classifier performance for most of the datasets. this reveals some evidence regarding the benefit of incorporating text specificity into a cnn architecture, at least for 5 out of 9 domains. apart from this performance comparison between stl cnn and mtl cnn, the purpose of this first part of the experiments was to investigate the effect of embeddings initialization and maximum input length. learning the word embeddings from the glove pre-trained weights seemed overall a good strategy for both stl and mtl approaches, even for domains with a considerable proportion of oov words, such as the cds and vinyl dataset (35%). besides that, except for health and personal care and sports and outdoors, the choice of embeddings initialization did not appear to be influenced by the learning paradigms (single-task or multitask). likewise, the results in table 2 also reveal that the maximum input length seemed to be more related to the domain than to the type of learning paradigm. for most of the datasets, the best input length was the same for stl cnn and mtl cnn. additionally, even though there is not a clear relation between these best input lengths to the data characteristics presented on table 1, the datasets with higher average review length showed better performance when using 512 as the maximum input length. all of these observations demonstrate that there may be a suitable domain-oriented choice of embeddings initialization and maximum input length, considering the single-task and multitask cnn approaches here presented. this finding may be used as a starting point on future experiments. even though having a good f1 score is desirable, having a high precision is crucial for the helpfulness classification problem. considering a real-world application, either for recommendation [5] or for automatic summarization of the opinions expressed on helpful reviews [13], it is preferable to correctly identify very helpful reviews, even if they are few, and to avoid misclassifying poorly helpful reviews. in other words, we expect the number of false positive instances to be as close to zero as possible. that’s why the best results achieved with the cnn+specificity models were compared against the baseline models using both f1 (table 3) and precision (table 4). as most of the best results from the first part of the experiments (table 2) were obtained with glove initialization, and to simplify the amount of settings to vary, the subsequent experiments were conducted without testing the random embeddings initialization. table 3. comparison f1 results of svm, stl cnn and the best of all cnn+specificity modelsfull size table table 4. comparison precision results of svm, stl cnn and the best of all cnn+specificity modelsfull size table even though the svm baseline seemed to outperform the other two approaches, the f1 results are not statistically significantly different, according to friedman test at \(p-value < 0.05\). on the other hand, when comparing the models using precision, cnn+specificity greatly outperformed svm with significant difference. this might indicate that incorporating the text specificity into a cnn classifier, whether as features or as an auxiliary prediction task, can specially reduce the amount of misclassified poorly helpful reviews. all the five cnn+specificity model variations appeared in the best results, therefore any of them should be worth testing when doing experiments with perceived helpfulness classification on other domains. regarding the best settings, the results suggest that they seemed to vary according to both the domain and the classifier architecture, which do not indicate a clear domain-oriented behavior. nevertheless, multi-task was a popular approach among most of the best results, across several datasets. 6.2 sentence specificity prediction besides analyzing the performance of the helpfulness classification models with and without incorporating the text specificity aspect, we also present here the main points regarding our experiments with the sentence specificity prediction approach proposed in [10]. therefore, the spe (explained in sect. 3) results are reported considering the specificity predictions on sentence-level. in other words, for each dataset containing product reviews, their sentences were extracted, then some or all of them were used as the target source to train the unsupervised domain adaptation model, and finally, the trained model was used to assign the specificity scores for all of these sentences. concerning the size of the training set, two variations were tested: a random sample of 100k sentences and all of the sentences in the dataset. after some preliminary experiments, better results were observed using a bigger training set rather than just 4.3k instances, as it was set in [12]. we decided to experiment with 100k sentences because this is about the amount of unlabeled sentences in the restaurant reviews domain used in the experiments performed in [10]. by increasing the training set, superior spe results were also observed when training the models for more epochs. therefore, using a training set with 100k instances, the model was trained for 80, 100, 120, 140, 160, and 180 epochs, and with all the sentences, it was trained for 60, 80, 100, and 120 epochs. the best and worst spe results for each dataset are summarized in table 5. beauty and books domains were the only ones that achieved a truly satisfactory spe score (\(< 1\)), according to what was discussed in sect. 3. nevertheless, when comparing the worst and the best results we can see that testing different training settings was important to achieve more reliable specificity predictions in our product domains. in addition to that, it is interesting to note that the size of the training set was crucial for the performance because, for every dataset, when the sample of 100k sentences achieved the preferred result, using all sentences to train the model presented an inferior performance, and vice-versa. table 5. the best and the worst spe results obtained in the specificity prediction experiments. the settings consist of the amount of sentences used for training + the quantity of training epochs.full size table some datasets, even the ones with more than 1m sentences, required all the sentences to be used as the target source to train the domain adaptation model. furthermore, the results show that training with more data requires fewer epochs to achieve better results. however, choosing a suitable training set size for this specificity prediction approach depends on the domain. 7 conclusion the present study aims to contribute to the perceived helpfulness prediction literature by exploring the text specificity aspect of product reviews. it is not a trivial activity to access truly valuable online reviews due to an overwhelming volume of poor quality comments. this elucidates the need for intelligent systems with the ability to identify helpful reviews. in this sense, we argue that text specificity may play an important role to improve such systems. therefore, we empirically validated the usefulness of different ways to incorporate the text specificity aspect into a cnn architecture. in summary, the results revealed that the proposed models seem to have the desired ability to reduce the amount of false positive (increasing precision) in comparison to a svm baseline. in future endeavors, we plan to carry out analysis about the influence of the text specificity in the helpfulness prediction, using, among other solutions, explainability methods. besides that, we aim to empirically evaluate incorporating the text specificity aspect into other modern architectures (e.g. transformers) due to their successful application on a variety of text classification problems. notes 1.https://spacy.io/. 2.https://keras.io/. 3.https://www.tensorflow.org/. 4.https://scikit-learn.org/. references basiri, m.e., habibi, s.: review helpfulness prediction using convolutional neural networks and gated recurrent units. in: 2020 6th international conference on web research (icwr), pp. 191–196 (2020) google scholar  caruana, r.: multitask learning. mach. learn. 28(1), 41–75 (1997) article  mathscinet  google scholar  chen, c., yang, y., zhou, j., li, x., bao, f.s.: cross-domain review helpfulness prediction based on convolutional neural networks with auxiliary domain discriminators. in: proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, vol. 2 (short papers), pp. 602–607 (2018) google scholar  fan, m., feng, y., sun, m., li, p., wang, h., wang, j.: multi-task neural learning architecture for end-to-end identification of helpful reviews. in: 2018 ieee/acm international conference on advances in social networks analysis and mining (asonam), pp. 343–350 (2018) google scholar  fan, m., feng, c., sun, m., li, p.: reinforced product metadata selection for helpfulness assessment of customer reviews. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, pp. 1675–1683 (2019) google scholar  ghose, a., ipeirotis, p.g.: estimating the helpfulness and economic impact of product reviews: mining text and reviewer characteristics. ieee trans. knowl. data eng. 23(10), 1498–1512 (2011) article  google scholar  kang, y., zhou, l.: longer is better? a case study of product review helpfulness prediction. in: americas conference on information systems (2016) google scholar  kim, s.m., pantel, p., chklovski, t., pennacchiotti, m.: automatically assessing review helpfulness. in: conference on empirical methods in natural language processing, pp. 423–430 (2006) google scholar  kim, y.: convolutional neural networks for sentence classification. in: proceedings of the conference on empirical methods in natural language processing, pp. 1746–1751 (2014) google scholar  ko, w.j., durrett, g., li, j.j.: domain agnostic real-valued specificity prediction. in: proceedings of the aaai conference on artificial intelligence (2019) google scholar  li, j.j., nenkova, a.: fast and accurate prediction of sentence specificity. in: proceedings of the aaai conference on artificial intelligence, pp. 2281–2287 (2015) google scholar  lima, b., nogueira, t.: novel features based on sentence specificity for helpfulness prediction of online reviews. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 84–89 (2019) google scholar  liu, j., cao, y., lin, c.y., huang, y., zhou, m.: low-quality product review detection in opinion summarization. in: conference on empirical methods in natural language processing, vol. 7, pp. 334–342 (2007) google scholar  louis, a., nenkova, a.: automatic identification of general and specific sentences by leveraging discourse annotations. in: proceedings of 5th international joint conference on natural language processing, pp. 605–613 (2011) google scholar  louis, a., nenkova, a.: text specificity and impact on quality of news summaries. in: proceedings of the workshop on monolingual text-to-text generation, pp. 34–42 (2011) google scholar  mcauley, j., targett, c., shi, q., van den hengel, a.: image-based recommendations on styles and substitutes. in: proceedings of the acm sigir conference on research and development in information retrieval, pp. 43–52 (2015) google scholar  mudambi, s.m., schuff, d.: what makes a helpful online review? a study of customer reviews on amazon.com. mis q. 34(1), 185–200 (2010) google scholar  ocampo diaz, g., ng, v.: modeling and prediction of online product review helpfulness: a survey. in: proceedings of the 56th annual meeting of the association for computational linguistics, vol. 1: long papers, pp. 698–708 (2018) google scholar  pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: proceedings of the 2014 conference on empirical methods in natural language processing, pp. 1532–1543 (2014) google scholar  saumya, s., singh, j.p., dwivedi, y.k.: predicting the helpfulness score of online reviews using convolutional neural network. soft comput. 24(15), 10989–11005 (2019). https://doi.org/10.1007/s00500-019-03851-5 article  google scholar  zhang, y., wallace, b.: a sensitivity analysis of (and practitioners’ guide to) convolutional neural networks for sentence classification. in: proceedings of the eighth international joint conference on natural language processing, vol. 1: long papers, pp. 253–263. asian federation of natural language processing (2017) google scholar  zhang, y., lin, z.: predicting the helpfulness of online product reviews: a multilingual approach. electron. commer. res. appl. 27, 1–10 (2018) article  google scholar  download references author information authors and affiliations federal university of bahia, salvador, brazil beatriz lima & tatiane nogueira authors beatriz limaview author publications search author on:pubmed google scholar tatiane nogueiraview author publications search author on:pubmed google scholar corresponding author correspondence to beatriz lima . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper lima, b., nogueira, t. (2021). incorporating text specificity into a convolutional neural network for the classification of review perceived helpfulness. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_33 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_33 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords review helpfulness text specificity opinion mining publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating cnn-based classification models combined with the smoothed pseudo wigner-ville distribution to identify low probability of interception radar signals | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating cnn-based classification models combined with the smoothed pseudo wigner-ville distribution to identify low probability of interception radar signals conference paper first online: 30 january 2025 pp 444–459 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) evaluating cnn-based classification models combined with the smoothed pseudo wigner-ville distribution to identify low probability of interception radar signals download book pdf download book epub edgard b. alves  orcid: orcid.org/0009-0009-2138-29299, jorge a. alves  orcid: orcid.org/0009-0004-0766-858810 & ronaldo r. goldschmidt  orcid: orcid.org/0000-0003-1688-05869  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 483 accesses 1 citation abstract fundamental in defense, radar electronic warfare (rew) requires adaptation to current threats. automatic recognition algorithms for intrapulse modulations (atr) of low probability of interception (lpi) radar signals are essential in rew. existing lpi signal atr methods combine the choi-williams distribution (cwd) pre-processing technique with convolutional neural networks (cnn). this work proposes two new atr combinations, based on squeezenet and googlenet cnn. both used the smoothed pseudo-wigner-ville distribution (spwvd) pre-processing technique as an alternative to cwd. replacing cwd by spwvd was based on the hypothesis that the latter usually provides higher resolutions than the former. the proposed atr overcame the sota atr, achieving a 99.06% accuracy, under noisy environments and providing evidence to the hypothesis raised. experiments involved two datasets with 13 types of modulations and 806,000 samples each. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others radar signal recognition based on triplet convolutional neural network article open access 13 november 2021 deep learning-based lightweight radar target detection method article open access 27 may 2023 detection of rolling element bearing defects using spwvd-cnn chapter © 2025 1 introduction currently, radar electronic warfare (rew)footnote 1 has assumed a fundamental role in defense of countries around the world. increasingly, rew support measures (esm) have become essential to improve detection, identification, and protection processes against enemy equipment such as missiles, for instance. in the context of missile detection, efforts have been made to incorporate electronic intelligence systems (elint) into esm equipment aimed at identifying low probability of interception (lpi) radar signals [5]. due to the use of robust automatic recognition algorithms of intrapulse modulations (atr automatic target recognition) of lpi radar signals, elint systems have good performance, even in environments with low signal-to-noise ratio (snr) [6]. for text simplification purposes, in this article, the acronym atr will be used as the expression atr model. in recent years, several atr systems for low probability of intercept (lpi) radar signals have been developed using different classification algorithms, such as classical artificial neural networks (ann) [9], decision trees (dt) [10], and convolutional neural networks (cnn) [6], among others [5]. to apply these atr systems, the signals need to be pre-processed using time-frequency analysis (tfa) techniquesfootnote 2. the main tfa techniques employed in lpi radar signal atr include the choi-williams distribution (cwd) [6], short-time fourier transform (stft) [13], and variations of the wigner-ville distribution (wvd) [14, 15], such as the smoothed pseudo-wigner-ville (spwvd). the spwvd is one of the most effective tfa techniques for estimating various temporal and spectral parameters of lpi radar signals, especially in noisy environments [7]. notably, literature works that have combined the tfa-cwdfootnote 3 technique with cnn have achieved the best performances in identifying lpi radar signals [3, 5, 6]. when analyzing these studies, it becomes evident that although they all employed different time-frequency analysis (tfa) techniques for signal pre-processing, there was no initiative to optimize the cnn’s hyperparameters according to the pre-processing technique. efforts to optimize hyperparameters in machine learning applications are justified because, in general, classification model performance varies based on the data and the pre-processing applied to it [2]. nevertheless, all the mentioned studies relied on the default parameterization of the cnn implementations. in light of the above mentioned, this study raises the following hypothesis: using a cnn with hyperparameters optimized accordingly to the tfa technique used in the pre-processing of lpi radar signals can lead to better performance when combining tfa+cnnfootnote 4. moreover, when comparing tfa+cnn with spwvd as the tfa technique to cases where cwd is used, the former technique can lead to superior results. to gather pieces of evidence that support the hypothesis raised above, this study aimed to analyze the results produced by the following combinations: \(spwvd\) \(+\) \(squeezenet_{opt}\); \(cwd\!+\!squeezenet_{opt}\); \(spwvd\!+\!googlenet_{opt}\); and \(cwd\!+\!googlenet_{opt}\). all these combinations were tested using 13 types of intrapulse modulations in lpi radar signals. the following combinations, \(spwvd\!+\!squeezenet_{opt}\), \(cwd\!+\!squeezenet_{opt}\), \(spwvd\!+\!googlenet_{opt}\) and \(cwd\!+\!googlenet_{opt}\), refer to versions of the cnn squeezenet and cnn googlenet with optimized hyperparameters, trained after applying tfaspwvd and tfa-cwd to the data, respectively. the 13 tested modulations were generated by considering a range of random values for their respective parameters, resulting in two time-frequency image (tfi) databases: one derived from the application of tfa-spwvd (spwvd-tfi) and the other from the application of tfa-cwd (cwd-tfi). each of these databases contains 403,000 tfi. remarkably, the \(spwvd\!+\!googlenet_{opt}\) combination achieved an average classification accuracy of 99.06% at \(0db\) snr, surpassing both previous related works and the results obtained by the \(cwd\!+\!googlenet_{opt}\) combination, thus confirming the considered hypothesis. the present text is structured into five sections. section 2 introduces the time-frequency analysis (tfa) techniques used to generate the time-frequency images (tfi) employed in the experiments. section 3 discusses state-of-the-art studies in lpi radar signal identification. section 4 outlines the experimental methodology, including tfi creation and the structure of classifiers based on squeezenet and googlenet. section 5 describes the obtained results and compares them with those from related works. finally, sect. 6 presents the research reflections, highlighting the main contributions and suggesting avenues for future investigations. 2 fundamental theory various pre-processing techniques for low probability of intercept (lpi) radar signals have been developed. among these, we can highlight tfa methods such as the wvd, its variations, and the cwd. these techniques play a crucial role in enhancing atr performance. the wvd allows optimal time-frequency concentration compared to other tfa methods [11]. however, it introduces cross-terms (ct)footnote 5. to mitigate the influence of ct, the pseudo-wigner-ville distribution (pwvd) was introduced, incorporating a sliding analysis window along the frequency axis [11]. subsequently, as an enhancement of the pwvd, the cwd was developed to reduce the ct influence both in frequency and time domains. the cwd is defined based on the fourier transform x(\(\omega \)) of x(t), as indicated in eq. 1. here, \(t\) represents the time variable, \(\omega \) is the angular frequency variable, \(*\) denotes complex conjugation, and \(\sigma = 1\) serves as a scale factor, critical for ct suppression by smoothing the cwd distribution. the kernel \(\phi \) acts as a low-pass filter for processing the two-dimensional fourier transform into a cohen class ambiguity function [11]. typically, an exponential kernel \(\phi (\xi ,\tau )=e^{-\xi ^2\tau ^2/\sigma }\) is adopted in this distribution [1]. $$\begin{aligned} \begin{aligned} cwd_x(t,\omega )=&\frac{1}{2\pi }\int _{\xi =-\infty }^{\infty }e^{-j\xi {t}}\int _{\mu =-\infty }^{\infty }\sqrt{\frac{\sigma }{4\pi \xi ^2}}e^{\frac{(\mu -\omega )^2}{4\xi ^2/\sigma }}x(\mu +\frac{\xi }{2})x^*(\mu -\frac{\xi }{2})d\mu {d\xi } \end{aligned} \end{aligned}$$ (1) another approach proposed to mitigate the interference caused by ct in the wvd is the creation of the spwvd. the spwvd introduces smoothing windows both in the time and frequency domains. mathematically, the spwvd is defined by eq. 2 [7]. in this equation, h(\(\tau \)) and g(\(\nu \)) represent the time and frequency window functions, respectively. the signal x(t) corresponds to the analytic signal of r(t), and \(*\) denotes complex conjugation. specifically, the analytic signal x(t) follows the equation \(x(t) = r(t) + jh[r(t)]\), where \(h[r(t)]\) represents the hilbert transform of the real signal r(t). $$\begin{aligned} \begin{aligned} & spwvd_x(t,f) = \int \int {x}(t-\nu +\tau /2){x^*}(t-\nu -\tau /2)h(\tau )g(\nu )e^{-j2\pi {f\tau }}d\nu {d\tau } \end{aligned} \end{aligned}$$ (2) the visual effect of interference resulting from ct in the time-frequency domain can be observed in the examples shown in fig. 1. these examples depict the results of applying different tfa techniques to an lpi radar signal. the resulting tfi from the wvd fig. 1a and the pwvd fig. 1b exhibit pronounced ct values (visible as highlighted vertical bars). in contrast, the tfi produced by the cwd fig. 1c and the spwvd fig. 1d show reduced ct effects. notably, although the tfi appears similar, the image in fig. 1d seems to have a slightly higher resolution than that in fig. 1c, suggesting a potential advantage of using spwvd over cwd. more details about each tfa can be obtained in [11]. fig. 1. lpi costas-4 signal pre-processed by different tfa full size image 3 related work currently, the state-of-the-art studies in the field of lpi radar signal classification, specifically regarding modulation type, include those by [3, 5, 6]. table 1 presents main characteristics of these works compared to the present proposal. notably, among these studies, only [3] explored different tfa techniques and compared results across classification models. however, unlike the current proposal, the three lpinet cnn evaluated in [3] experiments were generated using default hyperparameter values, regardless of the tfa technique used for signal pre-processing. additionally, our proposal uniquely assessed and compared the effects of employing tfa-spwvd and tfa-cwd on the same signals. this strategic choice was based on the expectation that even minor resolution differences, as illustrated in figures 1d and 1c, could lead to improved classification results with tfa-spwvd compared to tfa-cwd. table 1. main characteristics of related works and this proposalfull size table 4 proposed methodology to validate the hypothesis proposed in this study, we employed the methodology depicted in fig. 2. this methodology comprises the following steps: creation of lpi radar signal instances base, generated by adding additive white gaussian noise (awgn) and simulated channel loss to the noise-free lpi radar signal; signal pre-processing, generating the spwvd-tfi and cwd-tfi bases by pre-processing the signals using spwvd and cwd tfa techniques, respectively; model training, validating and testing, where the classification models were trained, validated and tested based on the lpi modulation type using the pre-processed signals; and evaluation of results, where the performance of our proposed approach is evaluated. notably, our method considered both tfa-spwvd and tfa-cwd on the same signals, aiming to leverage their distinct characteristics for improved classification results. the next paragraphs detail each of these steps. fig. 2. flowchart of the methodology adopted in the experiments full size image to create the base of lpi radar signal instances, we modeled the receiver of a radar system. we considered that the complex sample of an intercepted lpi radar signal is perturbed by additive white gaussian noise (awgn) and channel loss, as indicated by the equation: \(y(k) = x(k) {\circledast } h(k) + n(k)\). in this equation: \(x(k)\) represents the signal generated during the signal generation step, which is noise-free; \(h(k)\) corresponds to the channel interference resulting from the channel loss generation step; \(n(k)\) characterizes the noise introduced during the awgn generation step; \(k\) denotes the sample index for each \(t_s\) (sampling period), considering a sampling frequency \(f_s\). table 2. tfi databases generated in this workfull size table it’s important to emphasize that this study aimed to employ a data creation mechanism identical to that used in related works. to achieve this, we utilized the source code for generating lpi radar signals provided by the authors of [6] after corresponding via email. additionally, we created the database using the same intrapulse modulations and parameter ranges specified in table 1. consequently, we generated 13 different types of lpi radar signal modulations, including linear frequency modulation (lfm), unmodulated (rectangular) signals, costas modulation (frequency shift key fsk), binary barker coding, five polyphase codes (frank, p1, p2, p3, and p4), and four polytemporal codes (t1, t2, t3, and t4). similar to [3], we introduced noise during the awgn generation step, varying the snr from –20 db to +10 db with a 1.0 db increment. in the same way, channel loss interference was modeled, using rayleigh fading during the channel loss generation step. the different signal instances constituting the lpi signal base were created by randomly varying the specific parameters for each intrapulse modulation, following the specifications outlined in [3] and detailed in fig. 3. fig. 3. parameter ranges used in generating lpi signal instances [3] full size image as specified in [3], to generate the cwd-tfi and spwvd-tfi bases, we initially pre-processed each lpi radar signal using the spwvd and cwd techniques. during this step, we applied kaiser filters with a size of \(63\) samples and a shape factor of \(0.5\) to smooth the time and frequency windows. subsequently, we obtained the corresponding tfi, capturing images with \(256\) grayscale levels and dimensions of (50 \(\times \) 50) pixels using bicubic interpolation for resizing. figure 4 illustrates examples of tfi obtained from tfa-spwvd with an snr of +10 db. fig. 4. tfi examples from the spwvd-tfi base full size image as indicated in table 2, two databases were generated, each containing \(403.000\) tfi (\(31\) snr levels * \(13\) lpi signals * \(1000\) lpi signal instances). during the data partitioning step, each tfi base was sorted into five disjoint sets to facilitate cross-validation. in each cross-validation round, the data was stratified into training, validation, and test datasets in a 70-15-15 ratio, following a process and proportion similar to that used by [3]. the atr proposed in this work was evaluated using classifiers based on two cnn models: squeezenet and googlenet. each of these cnn models was selected for distinct reasons. squeezenetfootnote 6 was chosen primarily because it delivers strong performance comparable to alexnet in image classification tasks, yet with \(50\) times fewer parameters and a model size \(510\) times smaller. this compactness makes squeezenet suitable for memory-constrained devices [4]. additionally, as indicated in table 1, squeezenet falls within a medium-sized network category compared to other cnn used in related works. its topological architecture features unique fire modules, comprising squeeze layers with 1 \(\times \) 1 convolution filters and expand layers with 1 \(\times \) 1 and 3 \(\times \) 3 convolution filters. you can visualize this architecture in fig. 5. in this work, we have chosen version 1.1 of the squeezenet network because it requires 2.4 times fewer computations than version 1.0 while maintaining the same accuracyfootnote 7. fig. 5. squeezenet cnn [4] full size image googlenetfootnote 8 was chosen primarily for its performance, which approaches the sota in classification [12]. this network architecture consists of 27 deep layers, formed by stacking 9 inception modules linearly, as illustrated in fig. 6. these modules utilize 1 \(\times \) 1, 3 \(\times \) 3, and 5 \(\times \) 5 convolution filters, along with maximum pooling layers. this structure enables image identification and categorization, even when limited information is available [12]. fig. 6. inception module [12] full size image the fine-tuning of the squeezenet and the googlenet was made through a matlab app known as deep designerfootnote 9. the optimization process of the hyperparameter values of the squeezenet was made for the cwd-tfi (\(cwd\!+\!squeezenet_{opt}\)) and for the spwvd-tfi (\(spwvd\!+\!squeezenet_{opt}\)). in the same way, the optimization process of the hyperparameter values of the googlenet was made for the cwd-tfi (\(cwd\!+\!googlenet_{opt}\)) and for the spwvd-tfi (\(spwvd\!+\!googlenet_{opt}\)). these optimization processes were possible due to the use of the matlab app known as experiment managerfootnote 10 using bayesian searches. these searches employed stochastic gradient descent with momentum (sgdm) as the optimizer. the optimal hyperparameter values and their corresponding search ranges during training are summarized in table 3. additionally, a minibatch size of 128 was used for all models. table 3. search ranges and optimal values found for squeezenet and googlenet hyperparameters during cross-validation processfull size table finally, the evaluation of results step was made by comparing the performance of the \(spwvd\!+\!squeezenet_{opt}\), \(cwd\!+\!squeezenet_{opt}\), \(spwvd\!+\!googlenet_{opt}\), and \(cwd\!+\!googlenet_{opt}\) combinations with each other and with the results reported by state-of-the-art studies. all experiments were conducted on a hardware platform with an intel® core™ i5-12500h 12th gen cpu running at 2.50 ghz, 16 gb of ram, and a single nvidia geforce rtx 3070 ti gpu with 8 gb of memo 5 results to obtain evidence supporting the validity of the hypothesis raised in this study, three analyses were conducted based on the evaluation of the obtained results. the first analysis was conducted in two stages. in the first stage, the results of the \(cwd\!+\!squeezenet_{opt}\) and \(spwvd\!+\!squeezenet_{opt}\) combinations obtained during the cross-validation process were compared. in the second stage, the results of the \(cwd\!+\!googlenet_{opt}\) and \(spwvd\!+\!googlenet_{opt}\) combinations, also obtained during cross-validation, were compared. both stages present precision values for each tested combination considering the 13 lpi signal types and varying snr between –20 db and 10 db. notably, regardless of the combination used, precision approaches 100% for higher snr values (above 0 db) across all lpi signal types. the results of the first analysis using squeezenet can be visualized in fig. 7. comparing fig. 7a and fig. 7b, it can be noticed that, at –20 db, the barker signal achieved 97.4% precision with \(cwd\!+\!squeezenet_{opt}\) and 93.2% precision with \(spwvd\!+\!squeezenet_{opt}\). conversely, at the same snr, the t2 signal achieved 31.2% precision with \(cwd\!+\!squeezenet_{opt}\) and 55.5% with \(spwvd\!+\!squeezenet_{opt}\). another important observation is the favorable results for signals t1 to t4 when using \(spwvd\!+\!squeezenet_{opt}\), especially at negative snr values, compared to results obtained with \(cwd\!+\!squeezenet_{opt}\). however, at –15 db snr, \(cwd\!+\!squeezenet_{opt}\) outperformed \(spwvd\!+\!squeezenet_{opt}\) for the costas and lfm signals. fig. 7. comparison of classification precision tested by squeezenetopt full size image the results of the first analysis using the googlenet are presented in fig. 8. comparing fig. 8a and fig. 8b, it can be noticed that, at –20 db, the barker signal achieved 97.1% precision with \(cwd\!+\!googlenet_{opt}\) and 95.2% precision with \(spwvd\!+\!googlenet_{opt}\). similarly, at the same snr, the t1 signal achieved 66.7% precision with the \(spwvd\!+\!googlenet_{opt}\) against 27.6% with the \(cwd\!+\!googlenet_{opt}\). another notable result is the good performance of the lfm signal, using the \(spwvd\!+\!googlenet_{opt}\), and the costas signal, with the \(cwd\!+\!googlenet_{opt}\). fig. 8. comparison of classification precision tested by googlenetopt full size image in summary, based on the results obtained in the first analysis, it is clear that both combinations yielded comparable outcomes. specifically, each combination performed better for certain lpi radar signal types and worse for others. consequently, there is no single combination that outperforms the other across all 13 lpi radar signal types and the entire snr range. the second analysis aimed to compare the average accuracy results obtained by considering all 13 lpi radar signal types together, across the 5 rounds of cross-validation adopted, and the four types of atr combinations. accordingly, the average accuracy value was calculated for each combination across the entire snr range, as illustrated in fig. 9. it can be noticed in fig. 9 that the combinations using the cnn googlenetopt achieved higher accuracy. another conclusion drawn from the results presented in fig. 9 is that networks composed of the spwvd-tfi datasets, specifically \(spwvd\!+\) \(squeezenet_{opt}\) and \(spwvd\!+\!googlenet_{opt}\), exhibited superior average accuracy across the entire snr range when compared to combinations using the cwd-tfi datasets (\(cwd\!+\!squeezenet_{opt}\) and \(cwd\!+\!googlenet_{opt}\)). additionally, it is noteworthy that for high snr values, average accuracy approaches 100%, regardless of the combination used. for instance, at 0 db, \(spwvd\!+\!squeezenet_{opt}\) achieved an accuracy of 97.83%, while \(spwvd\!+\!googlenet_{opt}\) reached 99.06%. another noteworthy aspect from the curves in fig. 9 that reflects the superior performance of \(spwvd\!+\!squeezenet_{opt}\) combination compared to \(cwd\!+\!squeezenet_{opt}\) occurred at an snr of –15 db. at this point, \(spwvd\!+\!squeezenet_{opt}\) outperforms \(cwd\!+\!squeezenet_{opt}\) by 4.4 p.p. similarly, the optimal performance point for the \(spwvd\!+\!googlenet_{opt}\) combination, comparing with \(cwd\!+\!googlenet_{opt}\), occurs at an snr of –16 db, with a margin of 2.7 p.p above \(cwd\!+\!googlenet_{opt}\). ultimately, the spwvd-tfi based combinations consistently achieve more prominent accuracy results when compared to the cwd-tfi based combinations, mainly at negative snr values. fig. 9. average accuracy obtained with the four proposed atr combinations full size image table 4 provides the mean, minimum, and maximum accuracy values and the standard deviation obtained by each atr combination. it also shows the p-values and the results of the wilcoxon signed rank test applied to the atr combination pairs (\(cwd+squeezenet\), \(spwvd+squeezenet\)) and (\(cwd+googlenet\), \(spwvd+googlenet\)) with significance level \(\alpha = 0.05\) and null hypothesis \(h_0\) stating that the means are statistically identical for each pair of atr combinations. the wilcoxon test was run in matlab with the statistics and machine learning toolbox. table 4. accuracy values and results of wilcoxon signed rank testfull size table the results obtained, shown by table 4, suggest a slight superiority of spwvd-tfi over cwd-tfi when combined with an optimized cnn for classification. on average, the worst result of \(spwvd\!+\!squeezenet_{opt}\) \(80.9\%\) outperformed the best result of \(cwd\!+\!squeezenet_{opt}\) \(80.25\%\). similarly, the worst result of \(spwvd\!+\!googlenet_{opt}\) \(83.04\%\) surpassed the best result of \(cwd\!+\!googlenet_{opt}\) \(82,8\%\). additionally, its important to note that the standard deviation of mean accuracy for \(spwvd\!+\!squeezenet_{opt}\) \(0.22\) was significantly lower than for \(cwd\!+\!squeezenet_{opt}\) \(0.34\), indicating more consistent accuracy distributions for the spwvd-tfi-based combinations, while both \(cwd\!+\!googlenet_{opt}\) and \(spwvd\!+\!googlenet_{opt}\) standard deviation of average accuracy’s have been similar and rather small values, \(0.13\) and \(0.17\) respectively. it is also important to highlight that the wilcoxon test rejected the null hypothesis in both atr combination pairs, indicating statistical evidence that spwvd can lead to more accurate classification models than cwd. in fig. 10, we present the confusion matrices, at –8 db snr, for the two combinations that achieved the highest accuracy, considering their validation rounds, the \(spwvd\!+\!googlenet_{opt}\) and \(cwd\!+\!googlenet_{opt}\). in both confusion matrices, certain modulation types are strongly confused with others. for instance, p1 signals are often misclassified as p4, while t1 signals are confused with t2. these misclassifications occur regardless of the tfa technique employed (i.e. cwd or spwvd). however, a noticeable advantage lies in the classification capability of \(spwvd\!+\!googlenet_{opt}\) over \(cwd\!+\!googlenet_{opt}\). this conclusion is based on two observations: the higher accuracy achieved by \(spwvd\!+\!googlenet_{opt}\) \(90.00\%\) compared to \(cwd\!+\!googlenet_{opt}\) \(89.38\%\); and the better precision provided by \(spwvd\!+\!googlenet_{opt}\), which outperformed \(cwd\!+\!googlenet_{opt}\) when classifying 5 lpi radar signal types (barker, lfm, p3, p4, and t1), and losing only at classifying 3 lpi signal types (p1, t2, and t3). these findings underscore the effectiveness of spwvd-tfi in enhancing radar waveform recognition, particularly in low snr environments. fig. 10. confusion matrices full size image the results observed in the first and second analyses provide evidence supporting the validity of the hypothesis proposed in this study. specifically, the use of the spwvd tfa technique, combined with squeezenet and googlenet cnn trained with optimal hyperparameter values, appears to be a promising option for identifying the modulations of lpi radar signals. finally, the third analysis was conducted to ensure the promising performance of the atr combinations proposed in this article compared with the related studies listed in table 1. this comparison was feasible because all these studies evaluated their combinations using cross-validation, and the lpi radar signal instance base used in our study was generated following the same procedures and execution conditions as those adopted by the related works in their experiments. the graph in fig. 11 indicates the mean accuracy values obtained by our study and related works. from this figure, it is possible to note that the proposed combinations outperform all combinations from related studies across the entire tested snr range. notably, accuracy remains consistently high for snr values above \(0db\). considering the snr range from –6 db to 10 db, the proposed combinations achieve an average accuracy of 97.41% with \(spwvd\!+\!squeezenet_{opt}\) and 98.05% with \(spwvd\!+\!googlenet_{opt}\), while the combination proposed by [3] achieves 96.33%. even at the lower end of the snr spectrum (e.g., –18 db), our approach achieves an accuracy of 39%, surpassing the 25% accuracy reported by [3]. this 14 p.p difference represents a 56% improvement, demonstrating robust performance even in noisy conditions. the gain achieved by the present work, especially under low snr values, gives the elint, incorporated into an esm system, a better capacity for an early reaction against threats that employ guidance technology based on lpi radars, such as, missiles. it should be noted that the radar signals transmitted by a missile equipped with an lpi radar have very low power and are immersed in noise. therefore, the highest detection range of an esm will be limited to the lowest snr value at which that esm is capable of correctly detecting an lpi signal. fig. 11. mean accuracy values of the proposed atr and related works full size image finally, given the results of the three above analyses, it is possible to observe the effectiveness of our proposed atr combinations across different snr levels. 6 conclusions in the context of automatic identification of lpi radar signals, this article presents the following contributions: (i) providing experimental evidence that employing the spwvd tfa technique as a preprocessing step can lead to better classification model performance compared to models built using the cwd tfa technique, current state-of-the-art solution for this problem; (ii) proposing two new atr combinations, where the pre-processing was based on spwvd and the classifiers were based on cnn squeezenet and googlenet. these atr combinations outperform existing literature approaches across a wide snr range from –18 db to +10 db, considering 13 distinct intrapulse modulations; (iii) providing two large image databases, each containing 403,000 samples, to be used in further experiments in this field. as part of future work, in our quest for further evidence to validate the hypothesis proposed in this article and to enhance the state-of-the-art in lpi radar signal modulation detection, we intend to evaluate the performance of new atr combinations and different signal types. for example, in the future, we intend to explore the use of stft as the pre-processing tfa technique combined with different cnn models and with different signal input datasets, such as with acoustic signals. notes 1.set of actions that aim to ensure the use of the electromagnetic spectrum by friendly forces and prevent, reduce or prevent its use by enemy forces [8]. 2.a tfa technique consists of converting signals from the time domain to the time-frequency domain [11]. 3.also, to simplify the discourse, throughout the article, the expression tfa-x technique will be adopted to denote the expression tfa x technique, where x is a tfa technique. 4.in this work, the tfa+cnn notation will be used to denote the application of the tfa technique in data pre-processing, followed by the application of the indicated cnn as the classifier. 5.ct are spurious terms that arise in some time-frequency domain transformations. 6.https://github.com/forresti/squeezenet. 7.https://github.com/forresti/squeezenet/tree/master/squeezenet_v1.1. 8.https://github.com/bvlc/caffe/tree/master/models/bvlc_googlenet. 9.https://www.mathworks.com/help/deeplearning/ref/deepnetworkdesigner-app.html. 10.https://www.mathworks.com/help/matlab/ref/experimentmanager-app.html. references choi, h.i., williams, w.j.: improved time-frequency representation of multicomponent signals using exponential kernels. ieee trans. acoust. speech signal process. 37(6), 862–871 (1989) article  math  google scholar  faceli, k., lorena, a.c., gama, j., almeida, t.a.d., carvalho, a.c.p.d.l.f.d.: inteligência artificial: uma abordagem de aprendizado de máquina, vol. 1, p. 100. ieee (2021) google scholar  huynh-the, t., et al.: accurate lpi radar waveform recognition with cwd-tfa for deep convolutional network. ieee wirel. commun. lett. 10(8), 1638–1642 (2021) article  math  google scholar  iandola, f.n., et al.: squeezenet: alexnet-level accuracy with 50x fewer parameters and 0.5 mb model size. arxiv preprint arxiv:1602.07360 (2016) kong, s.h., et al.: automatic lpi radar waveform recognition using cnn. ieee access 6, 4207–4219 (2018) article  math  google scholar  liu, z., et al.: a method for lpi radar signals recognition based on complex convolutional neural network. int. j. numer. model. electron. netw. dev. fields 37(1), e3155 (2024) article  google scholar  ma, n., wang, j.: dynamic threshold for spwvd parameter estimation based on otsu algorithm. j. syst. eng. electron. 24(6), 919–924 (2013) article  math  google scholar  md: política de ge de defesa. portaria 333/md, art. 4 (2004) google scholar  milczarek, h., et al.: automatic classification of frequency-modulated radar waveforms under multipath conditions. ieee sens. j. (2023) google scholar  niranjan, r., rama rao, c., singh, a.: fpga based identification of frequency and phase modulated signals by time domain digital techniques for elint systems. defence sci. j. 71(1) (2021) google scholar  pace, p.e.: detecting and classifying low probability of intercept radar. artech house (2009) google scholar  szegedy, c., et al.: going deeper with convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1–9 (2015) google scholar  walenczykowska, m., kawalec, a., krenc, k.: an application of analytic wavelet transform and convolutional neural network for radar intrapulse modulation recognition. sensors 23(4), 1986 (2023) article  math  google scholar  wan, c., si, w., deng, z.: research on modulation recognition method of multi-component radar signals based on deep convolution neural network. iet radar sonar navigat. (2023) google scholar  willetts, b., ritchie, m., griffiths, h.: optimal time-frequency distribution selection for lpi radar pulse classification. in: 2020 ieee international radar conference (radar), pp. 327–332 (2020) google scholar  download references author information authors and affiliations instituto militar de engenharia, rio de janeiro, rj, 22.290-270, brazil edgard b. alves & ronaldo r. goldschmidt escola naval, rio de janeiro, rj, 20.010-060, brazil jorge a. alves authors edgard b. alvesview author publications search author on:pubmed google scholar jorge a. alvesview author publications search author on:pubmed google scholar ronaldo r. goldschmidtview author publications search author on:pubmed google scholar corresponding author correspondence to edgard b. alves . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alves, e.b., alves, j.a., goldschmidt, r.r. (2025). evaluating cnn-based classification models combined with the smoothed pseudo wigner-ville distribution to identify low probability of interception radar signals. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_31 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords cnn automatic lpi radar signal recognition spwvd publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature pseudonymization in legal texts according to the lgpd: a named entity recognition approach | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper pseudonymization in legal texts according to the lgpd: a named entity recognition approach conference paper first online: 30 january 2025 pp 309–323 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) pseudonymization in legal texts according to the lgpd: a named entity recognition approach download book pdf download book epub marcelo anselmo  orcid: orcid.org/0009-0005-1145-35019 & bruno césar ribas  orcid: orcid.org/0000-0001-6314-15119  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 379 accesses 2 citations abstract this study explores the application of named entity recognition (ner) for the pseudonymization of data in legal texts, aiming to protect personally identifiable information (pii) in compliance with brazil’s general data protection law (lgpd). the research highlights the challenge of balancing data privacy and utility, presenting a methodology that uses artificial intelligence technologies to effectively identify and obscure pii in legal documents. in this study, we propose a transformer model along with regex techniques to identify entities in a text. to test the model, we created a new dataset from the existing lenerbr. we also used a function and prompt engineering applied to the llama 8b version 3 model to generate synthetic data. tests showed the need for further adjustments in the proposed new model. future work will focus on improving the model’s accuracy and efficiency, as well as enhancing the identification of sensitive data and learning from user interactions. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others name pattern recognition: a model proposal applied to the anonymization of unstructured data chapter © 2025 fostering judiciary applications with new fine-tuned models for legal named entity recognition in portuguese chapter © 2022 on the assessment of deep learning models for named entity recognition of brazilian legal documents chapter © 2023 1 introduction in today’s digital age, where technology and computing power have grown exponentially, personal data has become an extremely valuable asset. companies capture this information from online interactions, such as websites and social networks, to create detailed user profiles. these profiles range from consumer habits to sensitive information like health conditions and personal preferences. this practice not only allows for the personalization of online experiences, but it also exposes users to privacy risks, including the illegal sale of such data in dark markets, potentially for use in criminal activities [3]. this scenario highlights the critical importance of protecting personal data, not only for individual privacy, but also for the overall security of society. the solution to these issues has manifested through regulations and laws dedicated to data protection, such as the general data protection regulation (gdpr) in the european union and the general data protection law (lgpd) in brazil [9]. the main problem addressed in this work is the difficulty of balancing individual privacy with the need to use their information in legal and administrative texts. this issue deepens when dealing with identifying and hiding pii without compromising data integrity and utility. using resources, such as transformer models and regex techniques, an effective balance between privacy and utility is sought, as already accomplished in other studies, such as the work of g. m. gr, s. abhi, and r. agarwal [5]. the aim of this study is to investigate and demonstrate the effectiveness of named entity recognition (ner) applied to pii pseudonymization in legal documents. it proposes to develop and test a model capable of efficiently identifying and hiding named entities that correspond to pii. the fig. 1 illustrates the pipeline adopted in our methodology. another important note is the cnmp resolution no. 281, dated december 12, 2023footnote 1, in its article 79, presents pseudonymization as an alternative to protecting the personal data of natural persons in procedures or processes within the brazilian public prosecutor’s office. fig. 1. pipeline adopted in our methodology full size image legal texts are often lengthy and complex, prioritizing formality over readability [12]. however, the implementation of advanced techniques, such as artificial intelligence models, often relies on large volumes of annotated data for training, which can be challenging in terms of cost and time [13]. with this in mind, we face a fundamental challenge: the need for data to be reliably and efficiently labeled. traditionally, this labeling is done manually, a process that, despite improving the accuracy of artificial intelligence models, is notoriously time-consuming and expensive [4]. this scenario imposes a high cost in terms of time, money, and effort, limiting the scalability of solutions. this paper is structured as follows: sect. 2 presents the background concepts, covering fundamental concepts, like the related to personal data protection and pseudonymization. section 3 describes related work, highlighting previous studies that addressed similar issues. section 4 presents the proposed methodology, detailing the data used, tools, and terms to be found. section 5 discusses the expected results for each test set. finally, sect. 6 presents the conclusions and suggestions for future work. 2 background concepts 2.1 general data protection law (lgpd) the general data protection law (lgpdfootnote 2, law no. 13,709, dated august 14, 2018) of brazil is a regulation that establishes guidelines for the collection, use, processing, and storage of personal data. the law grants individuals more control over their personal information while balancing privacy rights with technological advancement. the lgpd requires that any operation involving personal data be based on clear legal justifications and respect strict principles, such as purpose, adequacy, and necessity [3]. 2.2 personal data (pd) personal data (lgpd, art. 5, i) refers to information that identifies or can directly or indirectly identify a natural person. this definition is broad and includes a variety of types of information, ranging from names and digital identities to physical and electronic characteristics, such as photos and location data. such data is crucial in digital society, being used in various contexts, from identity verification to service personalization. however, handling it involves significant privacy risks, which requires robust protection measures to ensure data security and confidentialityfootnote 3. 2.3 anonymization vs. pseudonymization anonymization and pseudonymization are two fundamental concepts in data privacy management. anonymization (lgpd, art. 5, xi) refers to the process whereby personal data is processed to remove the ability to permanently associate this data with a specific individual, without using additional information. this means that once anonymized, the data cannot be linked back to the owner, even with the use of reasonable technical means. this process is irreversible, ensuring a high level of protection for the individual’s identity. on the other hand, pseudonymization (lgpd, art. 13, \({\breve{\textrm{g}}}\) 4) is a technique that modifies personal data so that identification of the owner cannot be done without using additional information, which is maintained separately and under strict security measures by the data controller. unlike anonymization, pseudonymization is reversible, provided that the additional information is made availablefootnote 4. this method will be adopted in this work as it allows the data to be protected without losing its utility for analysis and research. 2.4 retrieval-augmented generation (rag) rag is an advanced approach in natural language processing (nlp) models that integrates information retrieval with text generation [6]. this technique allows the model to retrieve relevant documents that are used to inform subsequent text generation. the unique ability of rag to combine parametric and non-parametric memory enables it to adapt both retrieval and content generation for specific nlp tasks, improving the accuracy and relevance of the generated informationfootnote 5. by utilizing rag, we can incorporate a vast amount of pre-existing legal information to enhance the quality of our generated dataset. 3 related work in the article by patsakis and lykousas (2023) [11], the authors explore the challenge of effectively anonymizing text in the age of large language models. this study questions the effectiveness of these approaches and assesses their ability to prevent identification, especially with the use of ai on large datasets. an experiment is conducted using gpt on anonymized texts of well-known personalities to verify whether these language models can re-identify individuals. they argue that despite technological advances, there are still significant obstacles in protecting data privacy in texts processed by these tools. in the work of de andrade et al. (2023) [1], the authors present an approach called promptner, which focuses on ner in sensitive data using automatically labeled instances. the authors propose an approach that employs llms to identify entities in complaints and then trains simpler models like the spert method. the improved ner model shows substantial improvements in f-score, ranging from 41% to 129% compared to models using only manually labeled data. this study is crucial because it combines artificial intelligence methods to improve efficiency in identifying and handling personal data. mota et al. (2021) [7] investigate the use of neural networks for named entity recognition in legal documents in portuguese. the authors used spacy and flair libraries. their results demonstrate the ability of these advanced technologies to handle the linguistic complexity of portuguese, providing a technical foundation for the development of more effective solutions. nunes and dos santos (2023) [9] discuss the application and impact of the lgpd in the brazilian context, focusing on a technological and agnostic approach. they emphasize the need for organizations to adapt to legal requirements and the importance of implementing effective measures for protecting personal data. this study highlights that implementing the lgpd requires a comprehensive technical and organizational approach. multiples tools, such as data management systems (dms), anonymization, pseudonymization, encryption, and auditing, are highlighted as essential for compliance. techniques like big data and machine learning can improve compliance, while privacy by design and ediscovery are emphasized as crucial for protecting data. gr, shinu, and agarwal (2023) [5] presented a hybrid model that combines regex and ner for resume analysis and matching. this work illustrates the applicability of ner techniques along with regular expressions to efficiently extract and process information in structured documents like resumes. the proposed methodology is relevant to our study as it demonstrates the effectiveness of integrating ner and regex in data identification and pseudonymization tasks. luz de araujo et al. (2018) [2] developed lener-br, a dataset specifically for named entity recognition in brazilian legal texts. their results provide an essential foundation for validating ner models in the brazilian legal context. the dataset they proposed serves as a valuable resource for training and testing new methodologies, including the approach of our study. oliveira et al. (2024) [10] explored the combination of prompt-based language models and weak supervision to label the task of named entity recognition (ner) in legal documents. they highlighted the effectiveness of their techniques in improving the accuracy and automation of the ner process, which is crucial for data pseudonymization in compliance with privacy regulations. 4 methodology 4.1 personal data used we categorized terms according to some types of personal data (pd). it is important to note that the spectrum of terms is much broader than those discussed in this study, however, we highlight some popular terms. note that the cpf is a brazilian id number. the terms used to generate synthetic data are presented in table 1 below. table 1. selected terms for identification of personal data (pd)full size table 4.2 dataset for this study, we used the lenerbr dataset derived from the study by luz de araujo and pedro henrique [2] and available atfootnote 6. this dataset consists exclusively of legal documents. it includes labels for people, places, temporal entities, and organizations, as well as specific tags for legal entities and judicial processes. we used the functions of the faker libraryfootnote 7 to generate synthetic data. the faker library is a python tool that generates fake data as per the function used. it is useful for creating synthetic data for testing and prototyping. we created a python function that generates random data for one to two items per term using the faker library. figure 2 below show examples of data generated by our function. fig. 2. example of data generated by running our function twice. full size image the entities in portuguese and its respective translation to english, hidden the entities that has the same translation, are: “nome” (name), “data” (date), “endereco” (address), “telefone” (phone), “dinheiro” (money), “cep” (zip code). as seen in the previous images, we generated synthetic data for the term cpf. for this term, being a personal identifier, we changed the values of the check digit. according to the study by [8], the check digit is a number calculated from the other numbers of the cpf. it is used to verify whether the number is mathematically consistent with the standards defined by the competent brazilian agencies. this action aims to ensure that invalid cpf is generated, thus preventing the exposure of sensitive data. we applied prompt engineering techniques with rag to create synthetic data. for this purpose, we used the llama model version 3 with 8 billion parameters, a large-scale open-source language model from meta. it is freely available for research purposesfootnote 8. some of the most relevant parameters for running the llama model were: “temperature = 0.2” and “max_new_tokens = 512”. the testing environment was a local machine with a gpu of 24 gb vram and an intel i9 10th generation processor. for each generated text, the time was around 12 s. these parameters were used to generate the synthetic data. our code is available at a github repositoryfootnote 9. an example of a prompt to generate a synthetic text and the response to this prompt is shown in fig. 3 below. fig. 3. prompt engineering to generate a synthetic example. full size image using this prompt model from fig. 3, we managed to generate 600 synthetic data of legal texts. these examples were used to evaluate the new proposed ner model for lgpd. we noticed that, even with the command passed to the prompt to use all synthetic data, the model did not use the value “29566719” for “cep”. therefore, we created a function that removes these data from the generated json already displayed in fig. 2. 4.3 ner model for evaluation we used a pre-trained bert model for named entity recognition in portuguese, provided by pierrefootnote 10. this model leveraged the same dataset previously mentioned, the lenerbr. however, our study changed the names of some entities displayed in the original model, removed others, and also added new entities. the original model had 6 entities, while our model has 8 entities. the mentioned changes are shown in table 2 below. table 2. changes in entities for the new ner model.full size table for the new entities, we employed regular expressions (regex) to enhance ner accuracy. regex proved to be a valuable technique for identifying and replacing specific text patterns (highlighted in the study [5]), such as phone numbers, email addresses, and other personal identifiers, which are not always captured by standard ner methods. this complementary approach allowed for more granular filtering of added data, strengthening the effectiveness of identification when dealing with a wide range of formats. 5 results 5.1 evaluation of the new ner model based on the generated dataset, we conducted evaluations using the f1-score. the below table 3 contains the results obtained. table 3. evaluation resultsfull size table table 4. evaluation results with regex corrections.full size table we observed that the best-performing entity was “email” with an f1-score of 0.995, while the worst was “address” with an f1-score of 0.377. it’s important to note that the first is identified through regex, while the latter by the ner model. with a simple analysis of the “address”, our assumption is that it contains text that could be confused with names of people, animals, among many others. there is also the fact that during the synthetic data generation process, the model may have generated addresses that are uncommon, which could have made correct identification more difficult. the model achieved an overall f1-score of 0.832, which is a satisfactory result for an ner model. the entity “zip code” had an f1-score of 0.69, which is below expectations. this result can be attributed to the model’s difficulty in correctly identifying zip codes, which consist of 8 digits and a hyphen. the hyphen was not captured by the initially created regex pattern, as we had made the hyphen mandatory, hence a zip code like “12345678” was not recognized. to address this issue, we added a regular expression to also identify zip codes without a hyphen. the table 4 contains the results obtained after this correction. by comparing the tables 3 and 4, we can see that the model correction with the addition of regular expressions for zip code identification significantly improved the f1-score for this entity. the f1-score for zip code increased from 0.69 to 0.99, which is an excellent result. the overall f1-score of the model also improved, from 0.83 to 0.86. this demonstrates that adding regular expressions to correct entity identification errors can significantly enhance identification accuracy. it is interesting to note that terms identified using regex, such as “cpf”, “phone”, “email”, “money”, and “zip code”, achieved f1-scores that were not always perfect. upon further investigation, we observed some interesting behaviors. for this analysis, let’s focus on the “entity” column and the values identified with regex. analyzing the “recall” value for “cpf”, we see a 100% score, meaning the model correctly identified all cases where “cpf” should not appear. however, for “precision”, the value was 98.2%, indicating the percentage correctly identified. upon analyzing the cases where the model missed, we observed some instances like the one shown below in fig. 4. fig. 4. issues regarding the model’s “precision”. full size image this indicates that, even though we had instructed the llm to generate text without creating new terms, it did. this behavior repeats for the other regex terms about “precision”. as for “recall”, the model correctly identified all cases, except for “phone”, which scored 94.4%. in this case, our model made an error in just one instance, resulting in a false positive, as shown in the example in fig. 5. it is still valid to say that our proposed model also ended up correctly identifying an additional data entity. in this way, we see a kind of inversion of our proposal, where our model ends up correcting the new dataset, improving its quality. this is a positive point, as it demonstrates that the model can be used to enhance data quality, even if that is not its primary objective. fig. 5. issues regarding the model’s “recall” full size image 5.2 usage example of the new ner model for the following example, we used the synthetic text generated as shown in fig. 3. we created an interface using python to facilitate the pseudonymization of the text. thus, we established one input field and two output fields. the input field is for the original text, and the output fields are for the identified terms and another for the pseudonymization secret. the fig. 6 below contains the original text, while figs. 7 and 8 contain data related to the generated outputs. fig. 6. input data full size image as shown in figs. 7 and 8 below, here again is the translation of the entities from portuguese to english: “nome” (name), “data” (date), “endereco” (address), “telefone” (phone), “dinheiro” (money), “cep” (zip code). fig. 7. output data entity recognition full size image fig. 8. output data pseudonymization secret full size image 5.3 limitations as with any automated system, there are limitations that must be considered to ensure the effectiveness and accuracy of pseudonymization. one of these limitations is the need for human review. our model may occasionally make errors in the pseudonymization of named entities. this requires a subsequent human review process to correct possible errors and ensure that the data is properly hidden, according to legal requirements. another significant limitation is the lack of cross-referencing between terms previously identified in the document. for example, if the system identifies “maria silva” as a person in one part of the text, it may not recognize “maria” in a subsequent reference as the same person, potentially labeling it differently, such as “name_2”. this problem, which seems simple to solve, may involve complex issues, such as homonyms and the overall context of the text. additionally, the model can generate false positives, which are errors where non-personal data are erroneously categorized with a different label. for example, a street name may be incorrectly identified as a person’s name. finally, the model struggles to identify and correct typographical errors in critical information, such as cpf or phone numbers. such typographical errors, which may include spaces or incorrect characters inserted between numbers, are not recognized by the system, potentially leading to a failure in properly concealing these sensitive data. this issue is likely to be addressed in subsequent work through optimization of the transformer model. 6 conclusion this study addressed the challenge of pseudonymizing personally identifiable information (pii) in legal texts to comply with brazil’s general data protection law (lgpd). we employed artificial intelligence technologies, including a transformer model and regex techniques, to effectively identify and conceal pii. tests revealed that although our model showed good initial results, adjustments are necessary to enhance its precision and effectiveness. the conclusions highlight that the combined approach of ner and regex is promising, allowing for more accurate entity identification. however, methodological limitations include the need for ongoing adjustments to the model to handle atypical cases and the reliance on high-quality training data to maintain accuracy. moreover, the technique faces challenges in consistently identifying all pii categories, with some entities like addresses showing lower results compared to others, such as cpfs and emails. in summary, while initial results on synthetic data were promising, tests on real data are necessary to ensure the model’s effectiveness in practical environments. the main advantage of the proposed model is its lightweight nature, allowing it to operate on systems with less computational capacity. this feature makes the model particularly valuable for applications in resource-limited environments or where processing speed is crucial. 7 future work for future work, one of the main intentions is to mitigate the limitations of the current model, especially in terms of consistency in entity identification and error reduction, such as false positives and the non-cross-referencing of named entities. the idea is to refine the existing model and incorporate more precise feedback in the model training phase to improve its accuracy and efficiency. plans also include developing an interactive model that learns from corrections made by users. this active learning approach will allow the system to continuously refine its identification and categorization of pii, adjusting to the peculiarities of legal text and the specific pseudonymization preferences of the user. lastly, one of the focuses will be on identifying new terms related to pii that are continually emerging with changes in data collection practices and legislation. recognizing that new types of personal data may arise, the model needs to be prepared and ready to respond to these new demands, enabling all forms of pii to be adequately identified and protected in accordance with the latest legal and ethical standards. notes 1.https://www.cnmp.mp.br/portal/images/calj/resolucoes/resoluo-281-de-2023.pdf. 2.https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm. 3.https://www.serpro.gov.br/lgpd/menu/protecao-de-dados/dados-pessoais-lgpd. 4.https://www.cnmp.mp.br/portal/images/calj/resolucoes/resoluo-281-de-2023.pdf. 5.https://huggingface.co/transformers/model_doc/rag.html. 6.https://huggingface.co/datasets/lener_br. 7.https://faker.readthedocs.io/en/master/. 8.https://llama.meta.com/llama3. 9.https://github.com/celiudos/paper_bracis_2024. 10.https://huggingface.co/pierreguillou/ner-bert-large-cased-pt-lenerbr. references de andrade, c.m., et al.: promptner: uma abordagem para reconhecimento de entidades nomeadas em dados sensíveis a partir de instâncias rotuladas automaticamente. in: anais do xxxviii simpósio brasileiro de bancos de dados, pp. 269–281. sbc (2023) google scholar  luz de araujo, p.h., de campos, t.e., de oliveira, r.r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: villavicencio, a., et al. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 313–323. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_32 chapter  math  google scholar  de dados pessoais, p.: lgpd (2020) google scholar  fredriksson, t., mattos, d.i., bosch, j., olsson, h.h.: data labeling: an empirical investigation into industrial challenges and mitigation strategies. in: product-focused software process improvement: 21st international conference, profes 2020, turin, italy, 25–27 november 2020, proceedings, pp. 202–216. springer, heidelberg (2020). https://doi.org/10.1007/978-3-030-64148-1_13 murthy, g., abhi, s., agarwal, r.: a hybrid resume parser and matcher using regex and ner. in: 2023 international conference on advances in computation, communication and information technology (icaiccit), pp. 24–29. ieee (2023) google scholar  lewis, p.s.h., et al.: retrieval-augmented generation for knowledge-intensive nlp tasks. corr abs/2005.11401 (2020). 2005.11401 google scholar  mota, c.c., nascimento, a.c., miranda, p.b., mello, r.f., maldonado, i.w., coelho filho, j.l.: reconhecimento de entidades nomeadas em documentos jurídicos em português utilizando redes neurais. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 130–140. sbc (2021) google scholar  nascimento, b.a.r., botto filho, m., gomes, v.g., menezes, h.k.a., silva, a.n.m., et al.: breve histório do cadastro de pessoa física–cpf e sua relação com a teoria dos números. caderno de graduação-ciências exatas e tecnológicas-unit-sergipe 2(3), 125–135 (2015) google scholar  nunes, l.f.p., dos santos, j.c.f.: lgpd-uma visão de tecnologia e agnóstica. revista direito & paz 2(49), 218–237 (2023) google scholar  oliveira, v., nogueira, g., faleiros, t., marcacini, r.: combining prompt-based language models and weak supervision for labeling named entity recognition on legal documents. artificial intelligence and law, pp. 1–21 (2024) google scholar  patsakis, c., lykousas, n.: man vs the machine: the struggle for effective text anonymisation in the age of large language models. arxiv preprint arxiv:2303.12429 (2023) sakhaee, n., wilson, m.c.: information extraction framework to build legislation network. corr abs/1812.01567 (2018). 1812.01567 google scholar  zhang, s., he, l., dragut, e., vucetic, s.: how to invest my time: lessons from human-in-the-loop entity extraction. in: proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining, kdd 2019, pp. 2305–2313. association for computing machinery, new york, ny, usa (2019). https://doi.org/10.1145/3292500.3330773 download references acknowledgments the authors acknowledge the use of the llama version 3 model for generating data to enhance the dataset utilized in this research. the synthetic data generation included sensitive information to ensure the robustness of the model. the authors take full responsibility for the content of the article, including ensuring the originality and correctness of all text, and have taken all necessary precautions to ensure that no real sensitive information is used or exposed. author information authors and affiliations department of computer science, university of brasilia, brasilia, brazil marcelo anselmo & bruno césar ribas authors marcelo anselmoview author publications search author on:pubmed google scholar bruno césar ribasview author publications search author on:pubmed google scholar corresponding author correspondence to marcelo anselmo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper anselmo, m., ribas, b.c. (2025). pseudonymization in legal texts according to the lgpd: a named entity recognition approach. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_22 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords data pseudonymization named entity recognition information privacy legal texts lgpd transformer publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature govbert-br: a bert-based language model for brazilian portuguese governmental data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper govbert-br: a bert-based language model for brazilian portuguese governmental data conference paper first online: 30 january 2025 pp 19–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) govbert-br: a bert-based language model for brazilian portuguese governmental data download book pdf download book epub mariana o. silva  orcid: orcid.org/0000-0003-0110-99249, gabriel p. oliveira  orcid: orcid.org/0000-0002-7210-64089, lucas g. l. costa  orcid: orcid.org/0009-0002-8898-42379 & … gisele l. pappa  orcid: orcid.org/0000-0002-0349-44949  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 389 accesses 3 citations abstract given the growing interest in natural language processing (nlp) for governmental applications, particularly in brazil, where vast amounts of governmental data are processed daily, the need for specialized nlp models tailored to the nuances of brazilian portuguese and the legal and administrative domains has become increasingly apparent. however, existing models may struggle to accurately interpret the complexities of governmental texts, often leading to suboptimal performance in document classification and analysis tasks. to address these challenges, we introduce govbert-br, a pre-trained language model tailored to the brazilian governmental context, covering legal and administrative domains. leveraging insights from diverse governmental texts, govbert-br addresses the challenges of accurately interpreting brazilian portuguese and the unique legal and bureaucratic terminology prevalent in governmental documents. we present the pre-training process and experimental evaluation of govbert-br, comparing its performance against baseline models across various text classification tasks relevant to the brazilian public sector. our findings demonstrate that govbert-br outperforms existing models in document and short-text classification tasks, showcasing its efficacy in accurately analyzing governmental text data. furthermore, our analysis reveals insights into the convergence behavior of govbert-br during fine-tuning, highlighting its rapid adaptation to downstream tasks. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others legalbert-pt: a pretrained language model for the brazilian portuguese legal domain chapter © 2023 bert self-learning approach with limited labels for document classification chapter © 2022 ulysses tesemõ: a new large corpus for brazilian legal and governmental domain article 18 july 2024 1 introduction in recent years, the application of natural language processing (nlp) tools to government data has surged significantly. this trend is largely driven by the promise of nlp to enhance public administration’s efficiency, transparency, and accountability. in brazil, where a vast volume of governmental data is generated and processed daily [4], nlp tools can streamline various administrative processes by automating the analysis and interpretation of textual data, thereby facilitating improved decision-making [8, 10, 20]. however, the effective utilization of this data is often constrained by the need for specialized nlp models capable of accurately interpreting the specificities of brazilian portuguese and the unique legal and bureaucratic terminology. existing nlp models are predominantly generic [19], trained on datasets that do not adequately reflect the particularities of brazilian governmental documents. this mismatch usually leads to suboptimal performance when these models are applied to the brazilian public sector [20]. in this context, some efforts have been made to develop domain-specific models focused on tackling legal [8, 18, 20] and administrative [10] tasks. although these endeavors represent important steps forward, they are often limited in scope, addressing only specific sub-contexts of governmental data. the interdisciplinary nature of governmental documents requires models capable of seamlessly integrating insights from multiple domains. legal documents, for example, frequently intersect with administrative procedures, and understanding these interrelations is crucial for effective decision-making within public administration. therefore, we introduce govbert-br, a pre-trained language model specifically tailored to the brazilian governmental context. it integrates both administrative and legal domains, providing a multifaceted perspective to better align with the diverse nature of governmental text data. in this paper, we present the pre-training process and the experimental setup of govbert-br. our evaluation involves a series of text classification tasks relevant to the brazilian governmental context, comparing the performance of govbert-br against several baseline models. our main contributions are summarized as follows. we introduce a novel pre-trained language model designed explicitly for the brazilian governmental domain, incorporating insights from both administrative and legal domains to enhance its adaptability and effectiveness. we provide a comprehensive evaluation framework, including a diverse set of text classification tasks pertinent to the brazilian governmental context. this allows for a thorough assessment of govbert-br’s performance across different types of governmental text data. our experimental results demonstrate govbert-br’s superior or competitive performance compared to baseline models across various datasets. this underscores its efficacy in accurately classifying governmental text, highlighting its potential for improving text understanding and decision-making within the brazilian public sector. 2 related work using language models in government applications has changed how governments interact with citizens and manage information. indeed, such models offer advanced capabilities in nlp, enabling automation in understanding public inquiries, analyzing policies, and other applications using open government data (ogd) [7, 14]. they can be applied in several areas, including consulting services [9], topic modeling [10], and sentiment analysis related to government policies [23]. in addition, language is crucial in these models, as the model’s language must align with the languages used in the target domain to ensure more efficient adaptation. therefore, recent works have proposed distinct models to tackle government-related tasks in several languages [1, 3, 11, 15, 22]. in the brazilian context, most models that address government tasks are built from bertimbau, a general-purpose model pre-trained on a large corpus of portuguese texts scraped from the web [19]. such government-related models can be divided into two distinct categories according to their purpose: legal domain and administrative domain models. the first group includes models focusing mainly on juridical and legislative tasks, including texts from the brazilian legal system into their pre-training phase to handle legal language and concepts effectively. for example, legalbertpt-br [18] is a model for mining topics within comments on brazilian draft laws. legalbert-pt [20], jurisbert [21] and clsjur.br [12] use texts exclusively from the brazilian judiciary branch for tasks such as named entity recognition (ner), semantic textual similarity (sts), and document summarization. finally, robertalexpt [8] uses a large corpus from the judiciary and legislative bodies over ner and classification tasks. the main difference between such models is that robertalexpt’s corpus contains texts in both european and brazilian varieties of the portuguese language, whereas the others consider only texts in brazilian portuguese. on the other hand, administrative domain models are designed to support various governmental functions outside the legal sphere, such as managing bidding processes and processing information from the official gazettes. the development of solutions for this specific domain is relatively recent, with few models dedicated exclusively to this context. one notable example is libert-se [10], a model developed using a corpus of brazilian official gazette segments and evaluated in topic detection tasks within bidding documents. in light of the aforementioned related work, our goal is to propose a model that addresses the legal and administrative domains in brazilian portuguese. govbert-br bridges the gap between these two domains, providing a comprehensive solution for handling diverse governmental text data. by integrating insights from both legal and administrative contexts, govbert-br aims to enhance the efficiency and effectiveness of nlp applications within the brazilian public sector, ultimately contributing to improved decision-making. 3 govbert-br governmental data is a rich and diverse source of information that contains many documents, including public bidding documents, legal texts, legislative bills, administrative reports, and more. however, the complexity and specialized nature of this data pose significant challenges for generic nlp models, which often fail to capture the nuances and specific terminology used in governmental texts. moreover, existing domain-specific models tend to focus on narrow sub-contexts, failing to address the interdisciplinary nature inherent in governmental documents comprehensively. to overcome these challenges, we introduce govbert-br, a pre-trained language model specifically tailored to the brazilian governmental context.footnote 1 it considers both legal and administrative domains, providing a multifaceted perspective that can better align with the diverse nature of governmental text data. in this section, we describe the pre-trained model used as the foundation for govbert-br (sect. 3.1) and detail the methodologies employed in its development (sect. 3.2). fig. 1. overview of govbert-br pre-training methodology full size image 3.1 pre-trained model govbert-br is built upon the architecture of bertimbau [19], the first large-scale portuguese language model. bertimbau, based on the bert (bidirectional encoder representations from transformers) [6] architecture, has been fine-tuned on various tasks to enhance its performance in understanding portuguese text. govbert-br inherits this architecture and extends it to focus specifically on the nuances of brazilian governmental and legal documents. 3.2 pre-training the govbert-br pre-training process involves further training bertimbau using two distinct corpora, each tailored to capture the intricacies of governmental and legal texts in brazilian portuguese. we believe such a multi-domain approach allows govbert-br to provide a comprehensive perspective that captures the interplay between legal and administrative language, offering a powerful resource for interpreting brazilian governmental documents. figure 1 shows the overview of the govbert-br pre-training methodology, which involves a two-phase approach. initially, the model is pre-trained using data from the administrative domain to equip the model with an understanding of bureaucratic terminology and administrative procedures commonly found in governmental texts. next, the model undergoes further pre-training using a dataset comprised of legal documents related to extraordinary appeals received by the brazilian supreme court (stf). table 1. overview of the baseline modelsfull size table administrative data. the initial pre-training corpus includes public bidding documents and segments from the official gazette, which contain summarized texts published to provide information regarding the bidding process [5]. it comprises 1,578,107 sentences and encompasses various document types and administrative data, including announcements, notices, and bidding terms. legal data. the second pre-training corpus includes legal documents related to extraordinary appeals received by the stf. it contains a subset of 1,760,862 pages sourced from the victor dataset [13]. task. the pre-training task employed for govbert-br is the masked language model (mlm) task. in this task, a percentage of words in a sequence (15%) are masked, and the model is trained to predict these masked tokens based on the surrounding context. this method enables the model to learn contextual relationships and develop a deep understanding of the language patterns specific to the domain. setup. each pre-training session is limited to 10 epochs, with checkpoints saved at the end of each epoch. we use a batch size of 64 sequences, each containing a maximum of 300 tokens. both pre-training steps involved over 245,000 steps, ensuring thorough exposure to the diverse dataset. 4 experimental design to evaluate the effectiveness of the pre-trained govbert-br language model, we conduct a series of experiments across various text classification tasks relevant to the brazilian governmental context. these experiments aim to assess govbert-br’s performance compared to baseline models and existing nlp approaches, demonstrating its ability to analyze and interpret governmental documents accurately. in this section, we describe the baselines considered (sect. 4.1), the downstream tasks (sect. 4.2) and datasets (sect. 4.3), and the evaluation metrics used (sect. 4.4). 4.1 baselines our main objective is to assess whether integrating legal and administrative pre-training data into govbert-br leads to better performance on governmental text classification tasks. therefore, we compare our model against several baseline models, including general-purpose and domain-specific models (table 1). regarding the general domain, we consider the bertimbau-base model, a widely used general-purpose language model pre-trained on a large-scale portuguese corpus and used as the basis for the govbert-br model. in addition to bertimbau, we include domain-specific models trained on legal and administrative data separately. for the legal domain, we consider the legalbert-pt model [20], which is also built upon bertimbau and pre-trained on a vast corpus of legal documents, enabling it to capture the nuances of legal language and terminology. as govbert-br builds upon bertimbau, comparing both models allows us to evaluate the impact of additional pre-training on legal and administrative texts. for the administrative domain, we consider libert-se [10], a pre-trained language model tailored for the public bidding context. libert-se is built upon bertimbau using the mlm task and a dataset of 300,000 official gazette segments sourced from various municipalities in minas gerais, as previously extracted in [5]. as it focuses on administrative language and procedures, such a model is a relevant benchmark for evaluating govbert-br’s performance in administrative tasks. 4.2 downstream tasks as downstream tasks, we focus on two distinct text classification tasks within the governmental domain: document classification and short text classification. document classification. this task involves categorizing documents, such as legal texts, legislative bills, administrative reports, and public bidding documents, into predefined classes. document classification provides a high-level understanding of the content and purpose of each document, facilitating efficient document retrieval and organization within governmental systems. short text classification. unlike document classification, this task focuses on classifying shorter text segments, such as item descriptions, summaries, or motions. short text classification is particularly useful for extracting relevant information from large volumes of textual data, enabling quick decision-making and analysis in governmental processes. table 2. datasets used for the downstream tasksfull size table 4.3 datasets for each domain, three datasets are selected to evaluate the performance of govbert-br and the baseline models. table 2 summarizes each dataset’s key characteristics, further described as follows. lipset [17]. contains 9,761 labeled public bidding documents from minas gerais, brazil. such documents are classified into 12 classes: ata registro preços (price registration), ata dispensa (minutes of waiver), ata pregão presencial (face-to-face auction), outras atas (other minutes), edital (public notice), contrato (contract), aditamento (amendment), aviso (notice), ratificação (ratification), errata (erratum), homologação (homologation), and outros (others). napex. contains 583,174 labeled public expenditure items from minas gerais, brazil.footnote 2 each item presents a short text description and is classified into five expenditure nature classes: obras (construction), serviços (services), material de consumo (consumables), material permanente (permanent materials), and locação (rent). prodserv. represents the same public expenditure items as napex, but in this dataset, each item description is further classified as either products or services. svic [13]. it is a subset of the victor dataset, containing 94,267 legal documents with 339,478 labeled pages. instances are labeled as acórdão (lower court decisions under review), recurso extraordinário (appeal petitions), agravo de recurso extraordinário (motions against the appeal petition), sentença (judgments), despacho (court orders), and others. motions [16]. contains 6,449 legal proceedings, each with an individual and variable number of motions, labeled by lawyers. legal proceedings refer to formal legal actions or cases brought before a court of law. motions, in turn, are formal requests made to a judge during legal proceedings, typically seeking a decision or ruling on a specific aspect of the case. instances in this dataset are labeled as arquivado (archived), ativo (active), or suspenso (suspended), indicating the status of the legal proceedings. rriop [2]. contains rhetorical annotations within the legal domain, focusing on sentences extracted from judicial sentences from the court of justice of mato grosso do sul (tjms), brazil. it comprises 10,784 petitions from 70 civil lawsuits filed in tjms court and was manually labeled with rhetorical roles specifically tailored for petitions. instances in this dataset are labeled as identificação das partes (identification), fatos (facts), argumentos (arguments), fundamentação legal (legal basis), jurisprudência (precedents), pedidos (requests), valor da causa (remedy), and others. 4.4 evaluation setup in our experimental setup, each dataset is partitioned into distinct subsets for training, validation, and testing, maintaining the distribution of label values across partitions. given that all datasets exhibit class imbalance, we perform the split in a stratified manner by class. this stratified partitioning scheme allocates 70% of the data to training, 20% to validation, and the remaining 10% to testing. however, it is important to note that we do not use cross-validation in this study, which may introduce bias into the results as the model’s performance is evaluated on a single split rather than multiple folds. this choice limits the ability to generalize the findings across different data partitions. table 3. performance comparison on document and short text classification tasks in the administrative domain. f1-macro scores are reported with standard deviationsfull size table moreover, the class imbalance present across all datasets poses a challenge. while we refrain from using oversampling or undersampling techniques to reflect real-world scenarios, balancing techniques could be applied during training without compromising the integrity of the imbalanced test scenario. such techniques might help mitigate the impact of class imbalance on model performance, leading to more reliable and interpretable outcomes. we leave the exploration of these balancing strategies for future work, as it could further enhance the robustness of the models in handling skewed data distributions. all evaluated language models are trained for a fixed number of five epochs without extensive hyperparameter search. models are trained until convergence on the validation set loss is achieved, ensuring stable performance. following convergence, or at the last epoch, model performance is evaluated using the f1-macro metric. to ensure the robustness of our evaluations, we conduct five experiments for each model, varying the random seeds. next, the results are averaged to produce a more reliable estimation of model performance. 5 experimental results in this section, we present the experimental results obtained from evaluating govbert-br across various text classification tasks relevant to the brazilian governmental domain. we analyze govbert-br’s performance in terms of overall effectiveness (sect. 5.1), the impact of multi-domain pre-training (sect. 5.2), and its convergence across different governmental domains (sect. 5.3). table 4. performance comparison on document and short text classification tasks in the legal domain. f1-macro scores are reported with standard deviationsfull size table 5.1 overall performance we first assess how govbert-br’s performance compares to other state-of-the-art nlp models. tables 3 and 4 present the results of document and short text classification tasks in the administrative and legal domains, respectively. the statistical significance is reported using the wilcoxon rank-sum test to compare the distributions of f1-macro scores. in document classification, govbert-br achieves the highest f1-macro on the lipset dataset, with an average score of 0.948, outperforming significantly the general-purpose bertimbau (\(w=25\), \(p<0.05\)) and the domain-specific libert-se models (\(w=22\), \(p\le 0.05\)). however, regarding legalbert-pt, the difference in performance is not statistically significant (\(w=15\), \(p>0.05\)). similarly, in the svic dataset, govbert-br achieves an f1-macro score of 0.794, demonstrating its competitive performance. it significantly outperforms libert-se (\(w=23\), \(p<0.05\)), but the difference in performance compared to bertimbau (\(w=21\), \(p>0.05\)) and legalbert-pt (\(w=20\), \(p>0.05\)) is not statistically significant. these results demonstrate that pre-training on both legal and administrative data provides govbert-br with a solid understanding of the nuances of brazilian governmental text, enabling it to perform well across diverse datasets and domains. regarding short text classification, govbert-br shows competitive performance across all datasets. it achieves the highest f1-macro score on the napex dataset, tying with bertimbau at 0.839, and performs competitively on the motions and rriop datasets, indicating its robustness in handling varied short-text data within governmental contexts. although legalbert-pt slightly outperforms govbert-br on the rriop dataset, govbert-br’s balanced performance across all datasets highlights its versatility. finally, our results reveal no statistical difference between all models in the prodserv dataset. overall, govbert-br demonstrates superior or competitive performance compared to baseline models across multiple datasets, underscoring its efficacy in accurately classifying both document-level and short-text data within the governmental domain. its robust performance highlights its versatility and effectiveness in handling various types of textual governmental data. fig. 2. distribution of f1-macro scores on single domains (libert-se and legalbert-pt) and multi-domains (govbert-br) across administrative and legal text classification tasks. (a) administrative and (b) legal domains. the horizontal dashed line represents the mean f1-macro score for each dataset full size image 5.2 multi-domain pre-training to assess the impact of incorporating both administrative and legal domain pre-training data, we compare govbert-br’s performance to models pre-trained on single domains. figure 2 shows the distribution of f1-macro scores for models pre-trained on a single domain only (libert-se and legalbert-pt) and multi-domains (govbert-br), for text classification tasks in (a) administrative and (b) legal domains. the horizontal dashed line represents the mean f1-macro score for each dataset. in the administrative domain (fig. 2a), for most datasets, govbert-br achieves a high mean f1-macro score compared to the single-domain libert-se model. the exception is the prodserv dataset, where the performance difference between govbert-br and libert-se is minimal, indicating that the benefit of multi-domain pre-training may vary depending on the dataset. this suggests that while multi-domain pre-training generally enhances performance, its impact can be less pronounced in certain administrative contexts. similarly, in the legal domain (fig. 2b), govbert-br outperforms the single-domain legalbert-pt model across most datasets, except for the rriop. in this particular dataset, legalbert-pt shows a slight edge, possibly due to its specialized training in legal texts, which are more aligned with the content of rriop. however, the difference in performance is not statistically significant, indicating that govbert-br’s inclusion of administrative domain data does not detract from its legal domain capabilities. overall, the results highlight the advantages of multi-domain pre-training. govbert-br’s consistent performance across various datasets, covering both administrative and legal texts, demonstrates the robustness and versatility of multi-domain pre-training. this approach leverages a broader contextual understanding, making the model more adaptable to different types of text classification tasks within governmental contexts. this versatility underscores the potential of govbert-br as a valuable resource for interpreting diverse governmental documents, ultimately supporting more efficient and accurate information processing in governmental operations. fig. 3. convergence analysis. (a) average performance, in terms of f1-macro, of each model across all datasets, highlighting convergence over training epochs. (b) regression lines of the average performance trajectory during fine-tuning sessions full size image 5.3 convergence analysis to further evaluate the models’ performance and stability, we analyzed the f1-macro scores across epochs during fine-tuning on all downstream tasks. figure 3a presents the average performance, in terms of f1-macro, of each model across the six datasets, highlighting how each model converges over time. figure 3b shows the regression lines for the average performance of each model across all datasets during fine-tuning sessions. the linear regression lines provide insights into the overall performance improvement or degradation trend throughout the fine-tuning process over epochs. in fig. 3a, we observe that bertimbau initially exhibits lower average performance than the domain-specific models, indicating a slower convergence rate in the early epochs. such a result is expected as the model undergoes significant adjustments in the initial epochs, transitioning from broad pre-training to more specialized tasks. as training progresses, bertimbau’s performance steadily increases, eventually reaching comparable levels to the domain-specific models, albeit with a slower rate of improvement. this suggests that bertimbau may need more epochs to achieve full adaptation. overall, all domain-specific models demonstrate rapid convergence compared to the bertimbau baseline, with govbert-br notably achieving the highest f1-macro scores early in the training process. libert-se and legalbert-pt also converge quickly, demonstrating the effectiveness of domain-specific pre-training. this allows the models to quickly adjust to the nuances of the target tasks and achieve high-performance levels early in the training process. however, govbert-br consistently maintains a higher f1-macro score, suggesting that multi-domain pre-training provides a significant advantage across various governmental text classification tasks. figure 3b corroborates these observations, providing a straightforward visual representation of the trajectory of performance improvement over epochs. the steeper the slope of the line, the faster the model’s performance improves during fine-tuning. conversely, a shallower slope indicates a slower rate of improvement. although bertimbau shows the steepest slope among the models, indicating relatively rapid improvement, it starts from a lower performance level compared to the domain-specific models. conversely, govbert-br, libert-se, and legalbert-pt, while exhibiting slightly gentler slopes, start from higher initial performance levels, reflecting the benefits of domain-specific pre-training. 6 conclusions and future work this paper introduces govbert-br, a specialized pre-trained language model tailored explicitly for the brazilian governmental context, encompassing legal and administrative domains. through pre-training on diverse governmental texts, govbert-br effectively addresses the challenges of interpreting brazilian portuguese and governmental documents’ intricate legal bureaucratic terminology. our experimental evaluation demonstrates govbert-br’s superior performance compared to existing models across various document and short-text classification tasks relevant to the brazilian public sector, highlighting its efficacy in accurately analyzing governmental text data. moreover, our analysis of govbert-br’s convergence behavior during fine-tuning reveals its rapid adaptation to downstream tasks, highlighting its versatility and effectiveness in handling diverse text classification challenges within governmental contexts. by offering a comprehensive solution for processing governmental text data, govbert-br significantly improves the efficiency and effectiveness of nlp applications within the brazilian public sector, directly contributing to more informed decision-making and governance processes. in summary, govbert-br represents a significant advancement in applying natural language processing to the brazilian governmental context. our model bridges the gap left by more generic models by addressing the specific linguistic and terminological nuances of brazilian portuguese and the distinct domains of legal and administrative documents. our comprehensive pre-training approach, incorporating diverse governmental texts, ensures that govbert-br can handle the complexities inherent in public administration tasks more effectively than previously available models. looking ahead, future research could explore the potential extensions and applications of govbert-br in other domains within the brazilian governmental landscape. moreover, ongoing efforts to refine and optimize govbert-br’s performance could further enhance its utility and impact in real-world governmental applications. finally, exploring ways to integrate govbert-br with other existing nlp models or frameworks could lead to synergistic effects, potentially improving its overall performance and expanding its capabilities in addressing complex governmental challenges. notes 1.available at https://huggingface.co/dccmpmgfinalisticas/govbert-br. 2.https://dadosabertos.tce.mg.gov.br/. references al-qurishi, m., alqaseemi, s., souissi, r.: aralegal-bert: a pretrained language model for arabic legal text. in: nllp@emnlp, pp. 338–344. association for computational linguistics (2022) google scholar  aragy, r., fernandes, e.r., caceres, e.n.: rhetorical role identification for portuguese legal documents. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 557–571. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_38 chapter  google scholar  bogdanovic, m., kocic, j., stoimenov, l.: srberta a transformer language model for serbian cyrillic legal texts. information 15(2), 74 (2024) article  google scholar  brandão, m.a., et al.: plus: a semi-automated pipeline for fraud detection in public bids. digit. gov. res. pract. 5(1), 5:1–5:16 (2024). https://doi.org/10.1145/3616396 constantino, k., et al.: segmentação e classificação semântica de trechos de diários oficiais usando aprendizado ativo. in: sbbd, pp. 304–316. sbc (2022).https://doi.org/10.5753/sbbd.2022.224656 devlin, j., chang, m., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: naacl-hlt, pp. 4171–4186. association for computational linguistics (2019).https://doi.org/10.18653/v1/n19-1423 gao, s., gao, l., li, q., xu, j.: application of large language model in intelligent q &a of digital government. in: cncit, pp. 24–27. acm (2023) google scholar  garcia, e.a.s., et al.: robertalexpt: a legal roberta model pretrained with deduplication for portuguese. in: propor, pp. 374–383. acl (2024) google scholar  han, j., lu, j., xu, y., you, j., wu, b.: intelligent practices of large language models in digital government services. ieee access 12, 8633–8640 (2024) article  math  google scholar  hott, h.r., silva, m.o., oliveira, g.p., brandão, m.a., lacerda, a., pappa, g.l.: evaluating contextualized embeddings for topic modeling in public bidding domain. in: bracis. lncs, vol. 14197, pp. 410–426. springer, cham (2023). https://doi.org/10.1007/978-3-031-45392-2_27 licari, d., comandé, g.: italian-legal-bert models for improving natural language processing tasks in the italian legal domain. comput. law secur. rev. 52, 105908 (2024) article  math  google scholar  lins, a.a., carvalho, c.s., das chagas jucá bomfim, f., de carvalho bentes, d., pinheiro, v.: clsjur.br a model for abstractive summarization of legal documents in portuguese language based on contrastive learning. in: propor, pp. 321–331. acl (2024) google scholar  luz de araujo, p.h., de campos, t.e., braz, f.a., da silva, n.c.: victor: a dataset for brazilian legal documents classification. in: lrec, pp. 1449–1458. elra (2020) google scholar  mamalis, m.e., kalampokis, e., karamanou, a., brimos, p., tarabanis, k.a.: can large language models revolutionalize open government data portals? a case of using chatgpt in statistics.gov.scot. in: pci, pp. 53–59. acm (2023) google scholar  miyazaki, k., yamada, h., tokunaga, t.: cross-domain analysis on japanese legal pretrained language models. in: aacl/ijcnlp (findings), pp. 274–281. association for computational linguistics (2022) google scholar  polo, f.m., ciochetti, i., bertolo, e.: predicting legal proceedings status: approaches based on sequential text data. in: icail, pp. 264–265. acm (2021). https://doi.org/10.1145/3462757.3466138 silva, m.o., et al.: lipset: um conjunto de dados com documentos rotulados de licitações públicas. in: dsw, pp. 13–24. sbc (2022). https://doi.org/10.5753/dsw.2022.224925 silva, n.f.f., et al.: evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 104–120. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_8 chapter  math  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  de v. silveira, r., ponte, c., almeida, v., pinheiro, v., furtado, v.: legalbert-pt: a pretrained language model for the brazilian portuguese legal domain. in: bracis. lncs, vol. 14197, pp. 268–282. springer, cham (2023). https://doi.org/10.1007/978-3-031-45392-2_18 viegas, c.f.o., costa, b.c., ishii, r.p.: jurisbert: a new approach that converts a classification corpus into an sts one. in: iccsa. lncs, vol. 13956, pp. 349–365. springer, cham (2023) google scholar  xiao, c., hu, x., liu, z., tu, c., sun, m.: lawformer: a pre-trained language model for chinese legal long documents. ai open 2, 79–84 (2021) article  math  google scholar  yulita, i.n., wijaya, v., rosadi, r., sarathan, i., djuyandi, y., prabuwono, a.s.: analysis of government policy sentiment regarding vacation during the covid-19 pandemic using the bidirectional encoder representation from transformers (bert). data 8(3), 46 (2023) article  google scholar  download references acknowledgments this work was funded by the prosecution service of state of minas gerais (in portuguese, ministério público do estado de minas gerais, or simply mpmg) through its analytical capabilities project and by cnpq, capes, fapemig and the partnership project between amazon web services (aws) and cnpq. author information authors and affiliations universidade federal de minas gerais (ufmg), belo horizonte, mg, brazil mariana o. silva, gabriel p. oliveira, lucas g. l. costa & gisele l. pappa authors mariana o. silvaview author publications search author on:pubmed google scholar gabriel p. oliveiraview author publications search author on:pubmed google scholar lucas g. l. costaview author publications search author on:pubmed google scholar gisele l. pappaview author publications search author on:pubmed google scholar corresponding author correspondence to gisele l. pappa . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, m.o., oliveira, g.p., costa, l.g.l., pappa, g.l. (2025). govbert-br: a bert-based language model for brazilian portuguese governmental data. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_2 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords natural language processing brazilian governmental data legal texts bert language models publish with us policies and ethics profiles mariana o. silva view author profile gisele l. pappa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature predicting bull and bear markets: a deep learning and linear regression study in cryptocurrencies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper predicting bull and bear markets: a deep learning and linear regression study in cryptocurrencies conference paper first online: 30 january 2025 pp 281–295 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) predicting bull and bear markets: a deep learning and linear regression study in cryptocurrencies download book pdf download book epub joão paulo costa e souza9, rodolfo i. meneguette10, vinícius p. gonçalves9, fábio l. l. de mendonça9, francisco airton silva11 & … geraldo p. rocha filho9,12  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 408 accesses abstract despite the growing popularity and increasingly widespread use of cryptocurrencies in contemporary financial markets, understanding market trends and predicting their future movements is a formidable challenge in financial analysis. regardless of the numerous tools and techniques available, accurately forecasting market behavior remains complex due to financial data’s inherent volatility and noise. this work proposes a novel hybrid model for market trend classification and trend slope regression to address this gap. the proposed approach leverages technical indicators, autoencoders, a bilstm-cnn architecture, and the use of linear regression to label the trends, enhancing prediction accuracy. the core innovation lies in the integration of these advanced techniques to extract meaningful and useful patterns from the data. this methodology allows the model to better capture the complexities of financial time series, thus improving its predictive capabilities. as a proof of concept, the proposed model was evaluated using historical bitcoin data spanning two years, with performance metrics such as accuracy, precision, recall for trend classification, and mean squared error (mse), mean absolute error (mae), and mean absolute percentage error (mape) for trend slope regression. the results are compelling, demonstrating that the combination of linear regression labeling and the architecture proposed achieves a remarkable level of accuracy both in-sample and out-of-sample. this accuracy significantly surpasses the 50% random benchmark in the predictions of bull and bear markets. ultimately, this research presents a promising approach to enhancing the precision of trend predictions in the volatile and unpredictable cryptocurrency market. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others machine learning algorithm for cryptocurrencies price prediction chapter © 2021 predicting stock price movement with multiple data sources and machine learning models chapter © 2021 forecasting and trading cryptocurrencies with machine learning under changing market conditions chapter © 2025 1 introduction cryptocurrency refers to electronic or virtual currency used for online transactions and asset transfers, secured through cryptographic techniques that regulate the creation of new units and safeguard transactions, ensuring secure transfers [3, 18]. unlike conventional currencies, cryptocurrencies operate on a decentralized framework without central banking systems. due to their uncontrolled and untraceable nature, the cryptocurrency market has experienced nearly exponential growth [4, 12, 14]. as cryptocurrencies gain recognition as a new electronic option for currency exchange, they have become increasingly important in emerging economies and the global financial landscape [11]. they are now integral to numerous financial transactions, making them a promising investment avenue. however, the market is characterized by significant volatility, price fluctuations, and significant randomness. predicting cryptocurrency trends is a formidable challenge in time-series forecasting due to the multitude of unpredictable variables, pronounced volatility in pricing, the lack of stationarity, and the fact that they are close to a random process [6, 10]. researchers have proposed various market rules to guide investors in their decisions, using historical price movements and trading volume in the form of technical analysis. this analysis helps forecast the continuation or reversion of market trends by identifying patterns in past price data. nevertheless, manual trading remains difficult due to market uncertainty, emotional influences on traders [16], and the infinitude of indicators and other financial data [1]. algorithmic trading, incorporating machine learning techniques, offers a solution. deep learning techniques, particularly convolutional neural networks (cnn) and long short-term memory (lstm) layers, have proven effective for financial forecasting in the cryptocurrency market. lstm layers efficiently capture sequential patterns in both longand short-term dependencies, whereas convolutional layers serve to eliminate noise from intricate time-series datasets and extract valuable patterns [9]. the literature presents a variety of models aimed at predicting the dynamics of financial markets, often achieving impressive results in price prediction. for instance, selvin et al. compare classical econometric methods with cutting-edge deep learning techniques, concluding that the latter are superior [15]. similarly, zhanhong he et al. study different architectures for predicting gold prices, identifying a combination of cnn, lstm, and attention mechanisms as the most effective [5]. zhuorui zhang et al. develop a robust price prediction regressor using memory and convolutional layers that consider the interrelations between cryptocurrencies [19]. in another study, ioannis e. livieris et al. create a cnn-lstm based neural network to predict gold prices, which shows promising results in regression but falls short in trend classification, with accuracy close to randomness [9]. livieris et al. also propose a non-sequential cnn-lstm neural network for price regression and trend classification, again finding trend classification results near randomness [8]. yanhui liang et al. take a different approach by decomposing gold prices into various frequencies before feeding them into a cnn-lstm structure, though they do not test the model in trend classification [7]. sima siami-namini et al. compare bidirectional long short term momory (bilstm) and lstm for price prediction, showing that bilstm achieves lower errors, but they do not test the model in trend classification [17]. iromie k. samarasekara et al. focus on risk management rather than trading, developing a dynamic stop-loss tool that uses a cnn-lstm trend classifier to feed a price regressor, the study showed good results compared to other tools of risk management [13]. faraz et al. enhance an lstm price regressor with an lstm autoencoder, achieving good results in price prediction but not testing it in trend classification [2]. despite these successes in price prediction, trend prediction often yields poor results. to address this gap, the current study proposes a novel trend classification method using a robust bilstm-cnn model and suggests linear regression for better sample labeling in trend slope regression and classification tasks. to effectively manage the inherent noise in financial markets, reduce input dimensionality, and eliminate outliers, an encoding structure is applied to the features. this compression process, handled by bilstm layers, forces market noise filtering, which is essential when dealing with financial data. once encoded, the features are fed into a bilstm-cnn neural network. the bilstm layers track shortand long-term dependencies in both directions of the time series, while the cnn layers capture significant patterns. the model is then trained using labels produced by linear regression over past and future price candles. trends are identified by their slope, with an ascending slope indicating a bull market and a descending one signaling a bear market. this approach enhances accuracy in identifying and predicting market trends. the point-by-point contributions are listed below: a novel autoencoder architecture to filter market noise based on bilstm. proposed a bilstm-cnn to handle trend prediction, both in market classification and slope regression tasks. a novel trend labeling system based on linear regression slope. the rest of this paper is organized as: sect. 2 describes the related work related to financial forecasting, both in price and trend prediction. section 3 shows the proposed model. section 4 discusses the results. section 5 brings a conclusion, followed by references. 2 related work this session aims to discuss several models and strategies proposed by various researchers. these proposals cover various architectures as well as methods for pre-processing the data and variables that feed the input layer of the various models. sreelekshmy selvin et al. [15] conducted a comparative analysis of three machine learning approaches: recurrent neural network (rnn), long short-term memory, and convolutional neural network, focusing on their performance in predicting financial market price. employing a sliding window methodology with a window size of 100 min, wherein 90 min of historical data is utilized to forecast the price of the subsequent 10 min. the method used to evaluate the models is mape. their study revealed that cnn outperformed rnn and lstm. the superior performance of cnn is attributed to its ability to adapt to the dynamic nature of the market by prioritizing current information over past patterns, which may not always be indicative of future trends. unlike rnn and lstm, which rely heavily on historical data, cnn’s emphasis on current information enables it to identify sudden shifts in market trends more effectively. this study underscores the efficacy of cnn in financial forecasting, particularly in scenarios characterized by rapid market changes and volatility. in the research conducted by zhanhong he et al. [5], a novel deep learning model for predicting gold prices is introduced, integrating long short-term memory and convolutional neural networks with an attention mechanism. their study reveals that arranging the model with the lstm layer preceding the cnn layer yields superior results compared to the reverse order. additionally, incorporating an attention mechanism after the lstm layer and before the cnn layer further enhances predictive accuracy. the author attributes this improvement to the prevention of information loss in the attributes, which would occur if cnn were placed before lstm. furthermore, the performance evaluation of the lstm-cnn hybrid model against cnn and lstm individually demonstrates a reduction in error, highlighting the efficacy of the combined approach. the author proposes that employing bidirectional lstm (bilstm) could potentially yield even more favorable outcomes, indicating avenues for future research and refinement in forecasting gold prices. introducing a new approach to cryptocurrency price prediction, the work of zhuorui zhang et al. [19] emphasizes volatility and strong intercorrelations among different cryptocurrencies. their predictive system comprises three key modules: the attentive memory module, leveraging a gated recurrent unit (gru) layer and self-attention mechanism to capture both shortand long-term dependencies; the channel weighting module, explicitly modeling interdependencies between cryptocurrency channels to account for their correlated nature; and the convolutional and pooling module, identifying attribute forms crucial for prediction. while the attentive memory module effectively captures dependencies, it struggles to assess the true importance of temporally delayed instances, addressed by the self-attention mechanism, which selects significant information across various time intervals. by explicitly modeling interdependencies between cryptocurrency channels, the channel weighting module proves essential, given the inherent correlations among cryptocurrency prices. the convolutional module further enhances predictive accuracy by identifying attribute forms. evaluation metrics, including average absolute percentage error and accuracy, demonstrate superior performance of the proposed model over traditional methods like cnn, lstm, or gru, indicating its effectiveness in cryptocurrency price prediction. in the research conducted by ioannis e. livieris et al. [9], the focus lies on leveraging convolutional neural networks and long short-term memory neural networks for gold price and movement prediction. the proposed model capitalizes on cnn’s capability to extract crucial information and learn internal representations from time series data, complemented by lstm’s proficiency in capturing both longand short-term dependencies. while the study yielded highly accurate results for price prediction with minimal errors, the performance for trend prediction was noted to be average. this observation suggests that while the model effectively captures and predicts the price movements of gold, it may encounter challenges in accurately forecasting the directional trends over time. further refinements or alternative approaches may be necessary to enhance the model’s efficacy in trend prediction, potentially through additional feature engineering, model adjustments, or the incorporation of complementary techniques. in another study by ioannis e. livieris et al. [8], an advanced non-sequential neural network architecture is proposed to predict cryptocurrency prices. this model capitalizes on the intricate interrelations between different cryptocurrencies by using data from multiple cryptocurrencies as inputs. the research focuses on the three leading cryptocurrencies by market capitalization: bitcoin, ethereum, and ripple. the model is applied to both price regression and trend classification tasks, with its performance evaluated using a range of metrics: mean absolute error (mae), root mean squared error (rmse), and r squared (\(r^2\)) for regression tasks, as well as accuracy, geometric mean, sensitivity, and specificity for classification tasks. while the model achieved relatively low errors in regression, its accuracy in trend classification was unfortunately close to randomness. yanhui liang et al. [7] also concentrate on predicting gold prices through a multi-step approach. initially, they decompose gold prices into distinct frequency components utilizing the improved complete ensemble empirical mode decomposition with adaptive noise (iceemdan) algorithm. subsequently, each of these decomposed components undergoes neural network layers such as cnn and lstm networks to refine the prediction. the study reveals that decomposing prices into various frequencies significantly enhances prediction accuracy, outperforming alternative decomposition methods. this approach underscores the importance of considering diverse frequency components in price prediction models, highlighting its potential to improve forecasting accuracy in financial markets, particularly for assets like gold characterized by complex and multifaceted price dynamics. sima siami-namini et al. [17]conducted a comparative study on the performance of lstm and bilstm in price prediction tasks. while bilstm is known to excel in tasks like predicting the next word in a sentence, its superiority in time series prediction was uncertain. the study revealed that bilstm outperformed lstm by 37.78% in mse for price prediction tasks. however, bilstm requires more effort to train compared to lstm. the key advantage of bilstm in financial data analysis is its bidirectional training, allowing the model to process information both forward and backward, enhancing its predictive capabilities. iromie k. samarasekara et al. [13] deviate from conventional approaches in financial forecasting by focusing their research on stop-loss dynamics. while traditional studies center on developing price prediction systems to aid buying and selling decisions, this work directs attention toward creating a prediction system to optimize stop-loss placement. the authors employ convolutional neural networks and long short-term memory networks to forecast future trends, utilizing these predictions as inputs for a subsequent price prediction system. subsequently, both the future trend predictor and price predictor are fed into a regression neural network, also leveraging cnn and lstm architectures, tasked with determining optimal stop-loss positions. this neural network is trained according to a specific algorithm. results demonstrate that losses incurred using the proposed strategy are substantially lower than losses experienced with traditional strategies employing fixed stop-loss and trailing stop mechanisms. faraz et al. [2] employed an autoencoder in their study to compress information extracted from technical indicators. both the encoder and decoder components of the autoencoder utilized long short-term memory cells. following the encoding process and reduction in input dimensionality, the data was then fed into a neural network consisting of lstm cells for price prediction. comparative analysis with the generative adversarial networks (gan) model demonstrated a reduction in error, indicating the superior performance of the lstm-based autoencoder approach. this suggests the effectiveness of leveraging lstm cells within the autoencoder framework for information compression and subsequent price prediction tasks, showcasing its potential for enhancing accuracy in financial forecasting models. fig. 1. pipeline of the model, showing the flow of market information carried out by technical indicators from the preprocessing stage, passing through encoding, and reaching the forecasting neural network. full size image fig. 2. architectural scheme of the neural network used for trend market classification. the model uses three bilstm layers, one convolutional layer, and two dense layers. full size image fig. 3. architectural scheme of the neural network used for trend slope regression. the model uses a prior convolutional layer, two bilstm layers, one convolutional layer, and two dense layers. full size image 3 proposed method in this section, we elucidate the proposed models and main concepts about trend classification and trend slope regression, which are the aims of this study. figure 1 illustrates the pipeline of the models of trend prediction. first, the technical indicators—which contain valuable trend information—go through the normalization process. then, the normalized data passes through an autoencoder to reduce the dimension of features, filter out some inherent market noise, and extract outliers. finally, the data is sent to a bilstm-cnn neural network to forecast the trends. in this step, linear regression is used to train the neural net both in the trend classifier and trend slope regressor. a set of candles is used in the trend forecasters’ training. candles are a type of financial instrument that displays four different prices during a given period: the opening, closing, minimum, and maximum prices. half of this candle set is composed of past candles, and the other half is composed of future candles. the closing prices of this set of candles are then used to perform a linear regression. in conclusion, regarding the trend classifier, a positive linear regression slope indicates a bull market, while a negative linear regression slope indicates a bear market. regarding the trend slope regressor, the linear regression slope measure provides an approximation of the trend’s strength. the trend classifier’s architecture is depicted in fig. 2. it is made up of three bilstm layers with a dropout layer to prevent overfitting, a convolutional layer, a max pooling layer, and two dense layers. bilstm layers have the advantage of being able to obtain time dependences from the past to the future as well as from the future to the past. additionally, because they come before the cnn layer, they ensure that information crucial to classification is not improperly pruned. furthermore, the design of the slope regressor is depicted in fig. 3. two cnn layers, two bilstm layers, a cnn layer, a max pooling layer, and two dense layers make up this structure. 3.1 normalization neural networks are sensitive to the size of the input data, so learning may be dominated by features with wider numerical ranges if normalization isn’t applied. this could lead to the model skewing in favor of some features and possibly ignoring others. in that regard, normalization was used to stabilize the training process by scaling the data to a common range, \([-1,1]\). this accelerates the convergence of the gradient descent technique and increases its efficiency. additionally, the uniform scaling of normalization helps to achieve better and more balanced weight adjustments during training, which improves the model’s performance and generalization on fresh data. it also prevents numerical instability. $$\begin{aligned} norm(x) = \frac{x \min (x)}{\max (x) \min (x)} \end{aligned}$$ (1) in eq. 1, x represents the value of the sample in a particular feature, \(\min (x)\) represents the minimum value the feature takes, whereas \(\max (x)\) represents its maximum value. 3.2 autoencoder an autoencoder is a kind of artificial neural network that is designed to automatically learn effective coding from input data without supervision. an autoencoder is a structure that employs two structures to learn a compressed representation of data: an encoder that codifies the information using the function \(h = f(x)\), and a decoder that tries to match the encoder’s input with its output \(x' = g(h)\). bilstm layers were used to build the autoencoder, which was based on the premise of dimensionality reduction and filtering outliers and market noise. the number of bilstm layers in the encoder and decoder components is equal, and each layer has an innate dropout mechanism that keeps the model from overfitting. 3.3 convolutional neural networks convolutional neural networks (cnns) are engineered to autonomously discern and internalize intricate patterns from singleor multi-dimensional datasets. the distinctive feature of cnns lies in their convolutional layer, where convolutional operations are applied, enabling the identification of patterns such as edges and shapes. the application of cnns in financial time series analysis offers significant potential for generating valuable insights into market forecasting, identifying investment opportunities, and recognizing price patterns. these networks excel in handling complex sequential data and extracting pertinent information, rendering them indispensable tools in financial analytics. unlike densely connected neural networks, cnns leverage weight sharing in convolutional layers, enhancing scalability and enabling the network to learn specific types of patterns more efficiently. this characteristic contributes to the adaptability and effectiveness of cnns in capturing and interpreting intricate patterns inherent in financial data, empowering traders and analysts with enhanced decision-making capabilities in dynamic market environments. $$\begin{aligned} f * g (x) = \sum _{-\infty }^{\infty } f(k)g(x-k) \end{aligned}$$ (2) equation 2 shows the convolution operation applied to one dimension, where f is the function to pass through the operation, g is the convolution filter and \((*)\) represents the convolution operation. 3.4 lstm neural networks long short-term memory networks (lstms) represent a specialized variant of recurrent neural networks (rnns) designed specifically for processing sequential information. lstms incorporate three essential channels for managing and processing sequential data. the first channel, termed the “forgetting gate,” regulates the extent to which information is discarded from the long memory, enabling the network to forget irrelevant or outdated information. the second channel, known as the “input gate,” receives new data and assesses its significance in the context of previous information, determining what information to retain and what to discard. this gate plays a crucial role in updating the network’s internal state based on incoming data. finally, the third channel, the “output gate,” determines which information stored in the long-term memory should be utilized to predict the current output, effectively filtering and selecting relevant memory components for the current context. these three channels interact dynamically, allowing lstms to capture and preserve long-term temporal dependencies within sequential data (which financial prices are an example) while mitigating issues such as gradient vanishing or exploding, which are common in traditional rnn architectures. $$\begin{aligned} i_t = \sigma _g (w_i[x_t,h_{t-1}] + b_i) \end{aligned}$$ (3) $$\begin{aligned} f_t = \sigma _g(w_f[x_t,h_{t-1}] + b_f) \end{aligned}$$ (4) $$\begin{aligned} o_t = \sigma _g(w_o[x_t,h_{t-1}] + b_o) \end{aligned}$$ (5) $$\begin{aligned} c_t = f_t \odot c_{t-1} + i_t \odot \sigma _c(w_c[x_t,h_{t-1}]+b_c) \end{aligned}$$ (6) $$\begin{aligned} h_t = o_t \odot \sigma _c(c_t) \end{aligned}$$ (7) in the equations above, \(i_t\), \(f_t\) and \(o_t\) represent respectively the input, forget and output gates at the time step t; w and b represent which one the weights and the bias of the lstm cell, it’s noted that each gate has their own weights and biases; \(x_t\) is the input of the cell, and \(h_{t-1}\) the hidden state at the previous time step. the operation [y, z] represents concatenation. \(\sigma _g\) is the sigmoid function, whereas \(\sigma _c\) is the hyperbolic tangent function. thus, memory \(c_t\) is formed taking the value of previous memory \(c_{t-1}\) weighted by the forget gate \(f_t\) added the new information given by \(\sigma _c(w_c[x_t,h_{t-1}]+b_c)\) weighted by the input gate \(i_t\). the symbol \(\odot \) represents the hadamard’s multiplication. finally, the actual hidden state is formed by the memory \(c_t\) transformed by the function \(\sigma _c\). 4 experiments and results this section assesses the effectiveness of our suggested market trend classifier and trend slope regressor models using various parameter values. while the accuracy, precision, and recall metrics are used to assess the market trend classifier’s performance, the mean squared error, mean absolute error, and mean absolute percentage error are used to evaluate the trend slope regressor’s performance. table 1. metrics for the market trend classifierfull size table table 2. metrics for the trend slope regressorfull size table in this study, bitcoin data was used to perform the training and testing of the trend predictors. bitcoin was chosen as it’s the first and most widely recognized cryptocurrency, and holds significant representativity in the market, often serving as the benchmark for other digital assets. it commands the largest market capitalization, influencing market trends and investor sentiment. bitcoin’s dominance also reflects its role as a store of value and a leading indicator of the cryptocurrency sector’s health and maturity. said that, the dataset consists of the last 30’000 candles of the btcusdt (bitcoin price given in dollars) in the period between january 01, 2022, and january 01, 2024. opening, high, low, close, and volume of each candle were used to calculate technical indicators. the technical indicators were: exponential moving average(ema), weighted moving average(wma), moving average convergence divergence(macd), momentum (mom), commodities channel index(cci), detrended price oscillator(dpo), relative strength index(rsi), average true range(atr), stochastic k% and stochastic d%. the results for training and test of the market trend classifier are shown in table 1. the number of candles taken for the linear regression calculation was varied between two to twelve, where half of the candles are future candles and half are past candles. the greater the number of candles, the more the trend is consolidated. it’s clear from the data that market forecasting is possible, given that if the market was all random, the accuracy of all models would be near 50%. furthermore, as shown in the table, the models trained with more candles show a better performance than the models trained with fewer candles, which suggests that market forecasting is more sympathetic to more consolidated trends. the results for training and testing of the trend slope regressor are shown in table 2 and in fig. 4. visually, the graphs show, like in the case of the market trend classifier, that the greater the number of candles used in linear regression, the more likely the curves of true and predicted values will match. this phenomenon can be explained by the higher volatility of linear regression calculated with fewer candles. nevertheless, table 2 shows bigger values of errors as the number of candles increases. the smallest values of errors come from fig. 4. trend slope over time, for the trend slope predictor built with (a) 2 candles, (b) 4 candles, (c) 6 candles, (d) 8 candles, (e) 10 candles, and (f) 12 candles. the curves in purple show the true values of the slope, whereas the curves in blue show the predicted values. the slopes are shown for train and test samples (color figure online) full size image the case in which only 2 candles were used for linear regression calculation, and we see clearly from the pictures that was the worst scenario. this occurs because the model can be stuck in local minima in the gradient descent part of training, delivering small values of error, whereas the overall predicted curve does not match the real curve. 5 conclusion this paper presents a novel strategy for training financial predictors using linear regression, focusing on cryptocurrencies, especially bitcoin, due to its market significance and trend forecasting complexities. the proposed architecture employs bidirectional long short-term memory (bilstm) and convolutional layers to capture spatial and temporal patterns in the data. unlike traditional methods, this approach uses linear regression over a variable set of 2 to 12 candles -half future and half past candlesto better represent market trends. the trend classifier shows promising potential, achieving significantly higher accuracy than the 50% random benchmark. increasing the number of candles in the linear regression set enhances accuracy by providing more consolidated trends, which are easier to forecast. consolidated trends exhibit stable patterns, while less consolidated ones are more volatile and random. thus, using more candles in the linear regression set leads to more reliable predictions by effectively capturing market dynamics. the trend slope regressor shows discrepancies between predicted trend slopes and error metrics, with better matches for more consolidated trends. future research could explore a broader set of indicators and the maximum number of candles predictors can handle before accuracy degrades. such advancements could enhance the model’s predictive capabilities. the trend classification and slope regression techniques in this study offer valuable tools for investors, enabling precise identification of trend reversals and continuations, potentially leading to greater profits. references derbentsev, v., matviychuk, a., soloviev, v.n.: forecasting of cryptocurrency prices using machine learning. advanced studies of financial technologies and cryptocurrency markets, pp. 211–231 (2020) google scholar  faraz, m., khaloozadeh, h., abbasi, m.: stock market prediction-by-prediction based on autoencoder long short-term memory networks. in: 2020 28th iranian conference on electrical engineering (icee), pp. 1–5. ieee (2020) google scholar  garcia, d., tessone, c.j., mavrodiev, p., perony, n.: the digital traces of bubbles: feedback cycles between socio-economic signals in the bitcoin economy. j. r. soc. interface 11(99), 20140623 (2014) article  math  google scholar  goundar, s.: introduction to blockchains and cryptocurrencies. in: blockchain technologies, applications and cryptocurrencies: current practice and future trends, pp. ix–xix. world scientific (2021) google scholar  he, z., zhou, j., dai, h.n., wang, h.: gold price forecast based on lstm-cnn model. in: 2019 ieee intl conf on dependable, autonomic and secure computing, intl conf on pervasive intelligence and computing, intl conf on cloud and big data computing, intl conf on cyber science and technology congress (dasc/picom/cbdcom/cyberscitech), pp. 1046–1053. ieee (2019) google scholar  lamberton, d., lapeyre, b.: introduction to stochastic calculus applied to finance. chapman and hall/crc (2011) google scholar  liang, y., lin, y., lu, q.: forecasting gold price using a novel hybrid model with iceemdan and lstm-cnn-cbam. expert syst. appl. 206, 117847 (2022) article  math  google scholar  livieris, i.e., kiriakidou, n., stavroyiannis, s., pintelas, p.: an advanced cnn-lstm model for cryptocurrency forecasting. electronics 10(3), 287 (2021) article  math  google scholar  livieris, i.e., pintelas, e., pintelas, p.: a cnn-lstm model for gold price time-series forecasting. neural comput. appl. 32, 17351–17360 (2020) article  math  google scholar  livieris, i.e., stavroyiannis, s., pintelas, e., pintelas, p.: a novel validation framework to enhance deep learning models in time-series forecasting. neural comput. appl. 32(23), 17149–17167 (2020) google scholar  nasir, m.a., huynh, t.l.d., nguyen, s.p., duong, d.: forecasting cryptocurrency returns and volume using search engines. financial innovation 5(1), 1–13 (2019) google scholar  nobre, r.a., et al.: aurora: an autonomous agent-oriented hybrid trading service. neural comput. appli., 1–16 (2022) google scholar  samarasekara, i.k., mendis, o.k., ahangama, s., atukorale, a.s.: dynamic stop-loss approach for short term trades using deep learning. in: 2022 ieee international conference on big data (big data), pp. 3537–3546. ieee (2022) google scholar  schmitz, m., immich, r., pessin, g., rocha filho, g.p.: towards the categorization of brazilian financial market headlines. ieee lat. am. trans. 20(2), 344–351 (2021) article  math  google scholar  selvin, s., vinayakumar, r., gopalakrishnan, e., menon, v.k., soman, k.: stock price prediction using lstm, rnn and cnn-sliding window model. in: 2017 international conference on advances in computing, communications and informatics (icacci), pp. 1643–1647. ieee (2017) google scholar  shefrin, h., statman, m.: the disposition to sell winners too early and ride losers too long: theory and evidence. j. financ. 40(3), 777–790 (1985) article  math  google scholar  siami-namini, s., tavakoli, n., namin, a.s.: the performance of lstm and bilstm in forecasting time series. in: 2019 ieee international conference on big data (big data), pp. 3285–3292. ieee (2019) google scholar  souza, f.m.d.c., filho, g.p.r., guimarães, f.g., meneguette, r.i., pessin, g.: navigating market sentiments: a novel approach to iron ore price forecasting with weighted fuzzy time series. information 15(5), 251 (2024) google scholar  zhang, z., dai, h.n., zhou, j., mondal, s.k., garcía, m.m., wang, h.: forecasting cryptocurrency price using convolutional neural networks with weighted and attentive memory channels. expert syst. appl. 183, 115378 (2021) article  google scholar  download references author information authors and affiliations university of brasilia, brasilia, brazil joão paulo costa e souza, vinícius p. gonçalves, fábio l. l. de mendonça & geraldo p. rocha filho university of são paulo, são carlos, brazil rodolfo i. meneguette federal university of piauí, picos, brazil francisco airton silva state university of southwest bahia, vitória da conquista, brazil geraldo p. rocha filho authors joão paulo costa e souzaview author publications search author on:pubmed google scholar rodolfo i. meneguetteview author publications search author on:pubmed google scholar vinícius p. gonçalvesview author publications search author on:pubmed google scholar fábio l. l. de mendonçaview author publications search author on:pubmed google scholar francisco airton silvaview author publications search author on:pubmed google scholar geraldo p. rocha filhoview author publications search author on:pubmed google scholar corresponding author correspondence to joão paulo costa e souza . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper costa e souza, j.p., meneguette, r.i., gonçalves, v.p., de mendonça, f.l.l., silva, f.a., rocha filho, g.p. (2025). predicting bull and bear markets: a deep learning and linear regression study in cryptocurrencies. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_20 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial intelligence cryptocurrency linear regression cnn lstm bilstm machine learning deep learning time series forecasting publish with us policies and ethics profiles rodolfo i. meneguette view author profile francisco airton silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature iris-cv: classifying iris flowers is not as easy as you thought | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper iris-cv: classifying iris flowers is not as easy as you thought conference paper first online: 28 november 2021 pp 254–264 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) iris-cv: classifying iris flowers is not as easy as you thought download book pdf download book epub itamar de paiva rocha filho10, joão pedro vasconcelos teixeira10, joão wallace lucena lins10, felipe honorato de sousa10, ana clara chaves sousa10, manuel ferreira junior10, thaís ramos11, cecília silva12, thaís gaudencio do rêgo10, yuri de almeida malheiros10 & … telmo silva filho10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1275 accesses abstract the iris flower dataset is a ubiquitous benchmark task in machine learning literature. with its 150 instances, four continuous features, and three balanced classes, of which one is linearly separable from the others, iris is generally considered an easy problem. hence researchers usually rely on other datasets when they need more challenging benchmarks. a similar situation happens with computer vision datasets such as mnist and imagenet, which have been widely explored. the state of the art models essentially solves these problems, motivating the search for more challenging tasks. therefore, this paper introduces a new computer vision toy dataset featuring iris flowers. users of a nature photography application took the pictures, thus they include noisy background information. additionally, certain desirable features are not guaranteed, such as single, similarly-sized objects at the center of each picture, which makes the task more challenging. our benchmark results show that the dataset can be challenging for traditional machine learning algorithms without any pre-processing steps, while state of the art deep learning architectures achieve around 82% accuracy, which means some effort will be necessary to drive this accuracy closer to what has been accomplished for mnist and imagenet. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others assessment of iris flower classification using machine learning algorithms chapter © 2021 end-to-end automated iris segmentation framework using u-net convolutional neural network chapter © 2020 flower species detection system using deep convolutional neural networks chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization computer vision data science data mining machine learning object recognition 1 introduction since fisher’s original publication in the context of linear discriminant analysis [4], the iris flower dataset turned into one of the most well-known and explored datasets in statistical classification and machine learning (ml), with over 18,000 citations until the date of publication of this work. the dataset consists of 150 observations, equally divided into three classes (iris setosa, iris virginica and iris versicolor), which are described by four features: the length and the width of the sepals and petals, in centimeters. the measurements were taken by edgar anderson [1], who was interested in measuring the morphological variation of these species, while fisher was the first to use the dataset in a statistical learning context. iris is generally considered an easy classification problem and is frequently used as ml’s “hello world” (i.e., as the first example one comes into contact with as a beginner in this area). most classification techniques have no trouble achieving accuracies well above 90% for iris with various hyperparameter configurations, as shown by multiple benchmark results available at openml [17]. in this paper, we introduce a new iris flower toy dataset, which breaks away from the original dataset’s simplicity by turning the problem of iris classification into a computer vision (cv) task. since the rise of deep learning (dl), the literature of cv has seen many of its well-known problems, including mnist [9], imagenet [3], and the newer fashion-mnist [18], being mostly solved, i.e., there have been models which achieved near-human or even better-than-human accuracy for these tasks. hence, this work introduces a dataset which features cv challenges such as fine grained categorization, background noise, real world environment conditions (e.g. lighting variation), as well as different scales and non-centered objects. the main idea is to make this dataset available so that it can be used to validate machine/deep learning approaches considering difficult cv scenarios. the paper also contains the classification results of traditional machine learning algorithms, as well as some state-of-the-art deep learning architectures using the iris-cv dataset. the remainder of this paper is organized as follows: sect. 2 shows some related works using the inaturalist dataset, sect. 3 describes our new dataset, sect. 4 presents benchmark results, including experiments with traditional ml algorithms and state of the art deep neural networks, and finally, sect. 5 contains our final remarks. 2 related works the inaturalist is a dataset proposed by [16] and has 675,170 images from more than 5 thousand different species of animals and plants. the category of plants has almost 200 thousand images. those species images were captured from all over the world, with different cameras and image qualities. also, it has a large class imbalance [16]. this dataset is constantly being updated. although inaturalist is a toy dataset that was used in kaggle competitions, it was already used in published papers. some works were focused on detection [16] and others’ purposes were to make classifications such as this work. plant classification was the focus of a study using the inaturalist dataset [12], where the authors used a convolutional neural network to classify the plant’s subclasses. data augmentation was used to reduce overfitting and balance the classes. then, a transfer learning approach based on resnet50 was utilized. this work classifies different plant species rather than only iris unlike our work. another study used the entire inaturalist dataset to make both classification and detection [16]. for classification, the following deep network architectures were performed: resnets, inception v3, inception resnetv2 and mobilenet. from those models, inception resnetv2 se had the best performance. this work also classifies different plant species rather than focusing on iris flowers, and their experiments did not include classic algorithms unlike ours. this work uses the inaturalist dataset as a single source, but some other researchers used it combined with different datasets. an example of it was the work proposed by [7]. after selecting different species of plant images from three datasets, they applied deep learning techniques for plant classification. their goal was to achieve at least 50 percent accuracy as a baseline classification and resnet50 was able to classify almost half of the inaturalist observations. the inaturalist dataset performed better than the portuguese flora dataset, but the google image search observations were better than both. as the purpose of this paper is to classify the different species of iris, the inaturalist dataset provided the images necessary to do that. the paper differs from works described in this section since it focuses on benchmark analysis, thus it covers a variety of algorithms and computational cost for each of them. in addition, we focus specifically on iris flowers, whereas the other works used plants in general. 3 a new iris dataset our new iris dataset, called iris-cv, consists of 5,139 examples extracting from inaturalist (september 10th 2020). each example representing an rgb image associated with a label corresponding to five different species: bearded iris (be) (iris x germanica, 928 images), douglas iris (do) (iris douglasiana, 944 images), dwarf crested iris (dw) (iris cristata, 1290 images), western blue iris (we) (iris missouriensis, 1036 images), and yellow iris (ye) (iris pseudacorus, 941 images), as shown in table 1. all images were gathered from inaturalist [10], a website that provides creative commons-licensed pictures of fauna and flora taken by users worldwide. table 1. dataset size per classfull size table after downloading the images, we manually removed photos that had too much noise, i.e., pictures of many different flowers, human hands covering the majority of the frame, and blurry images, which could confuse learning models, resulting in a harder problem. the original images have different sizes, therefore we resized them to a resolution of 256x256, which maintained the images’ main features while avoiding high memory requirements. we also kept the color information, which is coded in three rgb channels, as it can be important to differentiate the classes. table 2 shows resized examples of each class. the classes show different color patterns, particularly the yellow iris, which is appropriately named, and the dwarf crested iris, with its white and orange crests. table 2. class names and examples from the iris-cv dataset.full size table the pictures also show heavy background information, thus models must learn how to separate the flowers from the background to perform well. images may contain multiple and/or non-centered flowers. additionally, due to the different image and flower sizes, some individuals may become small compared to others after we resize the pictures, as seen in some examples in table 2. there can also be very different lighting conditions across pictures. finally, successful models will have to learn that each species can show different colors, for instance, dwarf crested irises can be lavender, lilac, pale blue, purple, white, or pink. as a result of all these features, iris-cv can be a challenging computer vision problem. 4 benchmark results we begin our experimental analysis by validating our dataset with eight different classic algorithms using scikit-learn and xgboost libraries, as listed below: decision tree (dt); extra tree (et); gradient boosting (gb); extreme gradient boosting (xgb); multilayer perceptron (mlp); perceptron; random forest (rf); k-nearest neighbors (knn). since these algorithms are not originally equipped to receive pixel matrices as input, we flattened the 256x256x3 images, obtaining vectors with 196,608 dimensions which can then be used as input. after obtaining the input vectors, we rescaled the images by dividing each pixel by 255 before training and testing the algorithms. experiments were carried out using 5-fold stratified cross-validation, which maintains class proportions across folds. the code was implemented using python’s scikit-learn library [11]. the algorithms and their hyperparameter values were chosen based on their performances on fashion-mnist’s benchmark [18] and we used several machines to streamline the execution of tasks. despite each machine has different settings (i.e. processor, graphic cards and memory), it is important to notice that this only affects the training time, not the algorithms overall performance. the best results achieved per classifier can be seen in table 3. we tested different values (table 3) for the following hyperparameters: criterion; objective; splitter; max_depth; loss; n_estimators; activation; hidden_layer_sizes; penalty; n_neighbors; weights; and p. see scikit-learn’s documentation [11] for more details about these parameters. in addition, table 4 describes the hardware resources for each classifier. table 3. results with standard deviation (std) of classic algorithms for the iris-cv dataset using 5-fold cross validation. the time column refers only to training time. hyperparameter names are shown as they appear in scikit-learn’s documentation.full size table table 4. hardware settings for training and evaluate each classifierfull size table even though most results surpassed the expected accuracy of a random classifier (0.251), the overall performance shows how difficult this problem is for classic methods without any extra preprocessing specifically designed to improve their results. best results were obtained by xgboost with 500 estimators, multi:softmax as the split objective, and 3 as the max depth, with a mean accuracy of 0.614. results were also poor compared to fashion-mnist [18] and mnist [9], where the same algorithms can reach accuracy values over 0.85 and 0.95, respectively. as mentioned in sect. 3, our dataset’s difficulty can be explained by some factors, such as somewhat high-dimensional images, rgb channels, non-centered images, noisy backgrounds, and petals with different colors within each species. thus, this problem cannot be tackled by simple approaches. 4.1 deep neural net results we now turn to state-of-the-art convolutional neural network (cnn) architectures, implemented using tensorflow 2 [5], to see how these techniques performing in the classification of iris-cv images. we chose to the hyperparameters used in the densenet [8] paper, since this is a state-of-the-art network. we trained each network with the stochastic gradient descent (sgd) optimizer, using 0.9 for the nesterov momentum and 40 epochs. regarding the learning rate, we chose a starting value of 0.1, decaying at a pace of \(10^\frac{epoch}{20}\) as a way of tuning network performance, which was observed empirically. in addition, we used different state-of-the-art architectures such as efficientnet, mobilenet and resnet50 to verify if it has any performance improvement regarding state-of-the-art network architectures. also, we performed data augmentation to balance the class proportions. we tested several combinations of the parameters of the imagedatagenerator provided by tensorflow and keras, viewing the images we had as a result and choosing the ones that kept most of the original information. below we list the augmentation parameters: rescale: 1/255 rotation range: 20\(^{\circ }\) width shift range: 0.1 height shift range: 0.1 horizontal flip: true shear range: 0.1 zoom range: 0.4 0.5 fill mode: nearest more information about these settings and the process used by imagedatagenerator is available in the official tensorflow documentationfootnote 1. the trained architectures are listed below: densenet121. proposed by huang [8], this deep net aims to optimize the flow of information between the layers of the network, making a dependency link between them, trying to minimize convergence time through shorter paths. efficientnetb0. this architecture, introduced by mingxing tan [15], is a neural network obtained through a compound scaling method of convnets’s depth, width, and resolution. inceptionv3. this architecture is a refinement of its antecessors, firstly by the introduction of batch normalization, and later by additional factorization ideas in the third iteration [14]. mobilenetv2. this network uses light convolution layers to filter the features in the intermediate layer, is based on a residual inverted structure [13]. resnet50. resnet [6] is an abbreviation for residual networks. this type of deep convolutional neural network uses residual blocks and can work with many layers of depths, avoiding the vanishing gradient problem. specifically, resnet50 is a 50-layer residual network. xception. xception [2] is a novel deep convolutional neural network architecture inspired by inception [14], where inception modules have been replaced with depthwise separable convolutions. the models were evaluated using 5-fold stratified cross-validation. table 5 shows that all cnns outperformed almost all classic algorithms, as expected. most architectures achieved accuracies over 0.74, except for efficientnetb3, which achieved the worst performance among the deep learning architectures. results also show that mobilenetv2 holds the highest accuracy and can be considered the state of the art for the iris-cv dataset. table 5. results with standard deviation (std) of deep learning architectures for the iris-cv dataset using 5-fold cross-validation. columns b and e refer to batch size and number of epochs, respectively. we used different batch sizes considering hardware resources of each machine. the densenet121, efficientnetb0, and mobilenetv2 architectures were trained using rtx 2060 super, resnet 50 was trained with rtx 2070, while the remaining ones were trained with a gtx 1660ti.full size table in addition to assessing accuracy, we analyzed the confusion matrix that corresponds to the best mobilenetv2 result, so that we can determine which classes the models have the most difficulty in predicting. the confusion matrix is shown in table 6. table 6. confusion matrix corresponding to mobilenetv2’s best test accuracy – each class is represented by the first two letters in its common name.full size table the mobilenetv2 confusion matrix shows that the network was good at differentiating western iris (we), yellow iris (ye), and dwarf crested iris (dw) from the other classes, as their precision scores corresponded to approximately 88%, 89%, and 89%, respectively, as shown in table 7. the other two classes – bearded iris (be) and douglas iris (do) – were not as well discriminated, and the algorithm had some trouble distinguishing them correctly since the flowers belonging to these two classes often have the same color, although they have different petal shapes. the similarity between these two classes can be observed in table 8. douglas iris can also sometimes be confused with dwarf crested irises, due to their white petal markings. as a result, douglas irises were the hardest flowers to classify correctly, with only 69% precision. table 7. precision, recall and f1-score corresponding to mobilenetv2’s best results in test dataset each class is represented by the first letters in its common namefull size table table 8. examples of bearded iris and douglas iris flowers showing their similar colors, but different petal structures.full size table 5 conclusion this paper introduced a new computer vision dataset, called iris-cv, consisting of five classes of iris flowers. the images show many features that make this dataset a challenging task, such as non-centered flowers, different lighting conditions, multiple flowers per image, and classes that naturally appear with different petal colors. due to all of these reasons, iris-cv proved to be too hard for traditional machine learning algorithms, with poorer results than those observed for established benchmark datasets, such as mnist and fashion-mnist. state-of-the-art deep neural nets also performed worse than current results for mnist, fashion-mnist, and imagenet, with mobilenetv2 achieving 82% accuracy, which is the best cross-validated result so far. additionally, an analysis of the best confusion matrix produced by mobilenetv2 showed that three of the classes are more easily classified, namely dwarf crested iris, western iris, and yellow iris. the remaining two classes – bearded iris and douglas iris – offered harder challenges, with the latter being the toughest to discriminate. since this paper main objective is to propose a new iris dataset considering computer vision common problems (e.g. occlusion, background noise and fine grained features), we performed a baseline of most used and state-of-the-art algorithms. future works include further exploration of deep neural net architectures and regularization techniques since the architectures used in this work focused on learning large datasets and some of them overfitted. in addition, hyper parameter tuning is also possible through optimization techniques such as bayesian optimization, grid search or random search, specially classic machine learning approaches which returned worse performed in this dataset. therefore, our next goal is to improve the current benchmark results, collect more data from different sources, and extract an object detection dataset based on the same images. notes 1.see tensorflow documentation for more details. references anderson, e.: the species problem in iris. ann. mo. bot. gard. 23(3), 457–509 (1936). http://www.jstor.org/stable/2394164 chollet, f.: xception: deep learning with depthwise separable convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1251–1258 (2017) google scholar  deng, j., dong, w., socher, r., li, l.j., li, k., fei-fei, l.: imagenet: a large-scale hierarchical image database. in: 2009 ieee conference on computer vision and pattern recognition, pp. 248–255. ieee (2009) google scholar  fisher, r.a.: the use of multiple measurements in taxonomic problems. ann. eugenics 7(2), 179–188 (1936). https://doi.org/10.1111/j.1469-1809.1936.tb02137.x, https://onlinelibrary.wiley.com/doi/abs/10.1111/j.1469-1809.1936.tb02137.x goodfellow, i., et al.: tensorflow: large-scale machine learning on heterogeneous systems (2015). https://www.tensorflow.org/software available from tensorflow.org he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  heredia, i.: large-scale plant classification with deep neural networks. in: proceedings of the computing frontiers conference, pp. 259–262 (2017) google scholar  huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4700–4708 (2017) google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  google scholar  loarie, s.: a community for naturalists (2008). https://www.inaturalist.org/ pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  r. al-qurran, m.a.a., shatnawi, a.: plant classification in the wild: a transfer learning approach. international arab conference on information technology (acit) (2018) google scholar  sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv 2: inverted residuals and linear bottlenecks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4510–4520 (2018) google scholar  szegedy, c., vanhoucke, v., ioffe, s., shlens, j., wojna, z.: rethinking the inception architecture for computer vision. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 2818–2826 (2016) google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: international conference on machine learning, pp. 6105–6114. pmlr (2019) google scholar  van horn, g., et al.: the inaturalist species classification and detection dataset. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 8769–8778 (2018) google scholar  vanschoren, j., van rijn, j.n., bischl, b., torgo, l.: openml: networked science in machine learning. sigkdd explor. 15(2), 49–60 (2013). https://doi.org/10.1145/2641190.2641198, http://doi.acm.org/10.1145/2641190.2641198 xiao, h., rasul, k., vollgraf, r.: fashion-mnist: a novel image dataset for benchmarking machine learning algorithms (2017). arxiv preprint arxiv:1708.07747 download references author information authors and affiliations universidade federal da paraíba, joão pessoa, pb, brazil itamar de paiva rocha filho, joão pedro vasconcelos teixeira, joão wallace lucena lins, felipe honorato de sousa, ana clara chaves sousa, manuel ferreira junior, thaís gaudencio do rêgo, yuri de almeida malheiros & telmo silva filho universidade federal do rio grande do norte, natal, rn, brazil thaís ramos universidade federal de pernambuco, recife, pe, brazil cecília silva authors itamar de paiva rocha filhoview author publications search author on:pubmed google scholar joão pedro vasconcelos teixeiraview author publications search author on:pubmed google scholar joão wallace lucena linsview author publications search author on:pubmed google scholar felipe honorato de sousaview author publications search author on:pubmed google scholar ana clara chaves sousaview author publications search author on:pubmed google scholar manuel ferreira juniorview author publications search author on:pubmed google scholar thaís ramosview author publications search author on:pubmed google scholar cecília silvaview author publications search author on:pubmed google scholar thaís gaudencio do rêgoview author publications search author on:pubmed google scholar yuri de almeida malheirosview author publications search author on:pubmed google scholar telmo silva filhoview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de paiva rocha filho, i. et al. (2021). iris-cv: classifying iris flowers is not as easy as you thought. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_18 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computer vision dataset machine learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature quaternion-valued convolutional neural network applied for acute lymphoblastic leukemia diagnosis | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper quaternion-valued convolutional neural network applied for acute lymphoblastic leukemia diagnosis conference paper first online: 28 november 2021 pp 280–293 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) quaternion-valued convolutional neural network applied for acute lymphoblastic leukemia diagnosis download book pdf download book epub marco aurélio granero  orcid: orcid.org/0000-0002-1993-189810, cristhian xavier hernández  orcid: orcid.org/0000-0001-8530-988011 & marcos eduardo valle  orcid: orcid.org/0000-0003-4026-511012  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1298 accesses 9 citations 2 altmetric abstract the field of neural networks has seen significant advances in recent years with the development of deep and convolutional neural networks. although many of the current works address real-valued models, recent studies reveal that neural networks with hypercomplex-valued parameters can better capture, generalize, and represent the complexity of multidimensional data. this paper explores the quaternion-valued convolutional neural network application for a pattern recognition task from medicine, namely, the diagnosis of acute lymphoblastic leukemia. precisely, we compare the performance of real-valued and quaternion-valued convolutional neural networks to classify lymphoblasts from the peripheral blood smear microscopic images. the quaternion-valued convolutional neural network achieved better or similar performance than its corresponding real-valued network but using only 34% of its parameters. this result confirms that quaternion algebra allows capturing and extracting information from a color image with fewer parameters. this work was supported in part by coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others acute-lymphoblastic leukemia detection through deep transfer learning approach of neural network chapter © 2021 efficient convolutional neural networks for acute lymphoblastic leukaemia prediction in computer vision article open access 16 december 2025 an effective approach for detecting acute lymphoblastic leukemia using deep convolutional neural networks chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer vision flow cytometry linear algebra machine learning neural decoding artificial intelligence 1 introduction in recent years, machine learning has influenced how we solve a variety of real-world problems. indeed, artificial neural networks (nn) outperformed many state-of-the-art approaches in several applications with the development of deep neural networks (dnn) and convolutional neural networks (cnn) architectures. most neural network architectures are real-valued neural networks (rvnn). in such architectures, the input data is arranged into real-valued vectors, matrices, or tensors to be processed by the neural network. in some sense, this approach assumes that all the input data components have equal importance and, thus, they are all evaluated in the same way. however, in some cases, the data sets contain multidimensional information that requires a specific approach to treat them as single entities. for example, a pixel’s color is obtained by combining the red, green, and blue components in image processing. the three coordinates position in the color space represents a plethora of colors such as pink or brown, and the color information is lost if the components are treated separately [38]. in some practical image recognition tasks, the complexity of the color space needs to be captured by the neural networks to generalize well and represent the multidimensional nature of the colors [20, 27]. indeed, parcollet et al. showed that rvnns might fail to capture the color information [39]. also, matsui et al. remarked that rvnns are not able to preserve the 3d shape of an input object when transformed into the 3d space [30]. from these remarks, neural networks based on hypercomplex numbers, such as complex and quaternions, have been proposed and extensively investigated in the last years. 1.1 complex and quaternion-valued neural networks a complex-valued neural network (cvnn) is based on the algebra of complex numbers, which allows preserving or treating the relationship between magnitude and the phase information during the learning [38]. furthermore, the algebraic structure of complex numbers yields cvnns better generalization capability [17] besides being easier to train [35]. as long as the processed information are correlated two-dimensional data, cvnns mostly outperformed or at least matched the real-valued ones [3, 4, 16, 29, 48]. the encouraging performance of cvnns inspired the development of quaternion-valued neural networks (qvnns). qvnns use quaternion algebra and can represent colors efficiently, with the advantage of fully representing colors through unique structures [6]. as far as we know, the first qvnn has been introduced by arena et al. [6], who developed a specific backpropagation algorithm able to learn the local relations that exist between quaternions. furthermore, like the real-valued neural networks, single hidden layer qvnns are universal approximators [6]. an extensive list of applications and investigations with different qvnn architectures can be found in references [7, 11, 36,37,38, 40, 46]. a detailed up-to-date review on quaternion-valued neural networks, including some of their successful applications, can be found at [38]. in contrast to rvnn, which represented color channels as independent variables, qvnn can benefit from representing colors as single quaternions. for example, greenblatt et al. applied a qvnn model to prostate cancer [13]. gaudet and maida investigated the use of quaternion-valued convolutional neural networks (qvcnn) for image processing [11]. pavllo et al. modeled human motion using qvnns [41]. zhu et al. proposed a qvcnn for color image classification and denoising tasks [51]. the localization of color image splicing by a fully quaternion-valued convolutional network was explored by chen et al. [9]. a deformable quaternion gabor convolutional neural network for recognition of color facial expression was proposed by jin et al. [22]. takahashi et al. have merged histograms of oriented gradients (hog) for human detection with a qvnn to determine human facial expression [47]. quaternion multi-layer perceptron has been successfully applied to polarimetric synthetic aperture radar (polsar) land classification [24, 43]. 1.2 contributions and the organization of the paper corroborating with the development of hypercomplex-valued neural networks, we present a quaternion-valued convolutional neural network (qvcnn) development to classify isolated white cells as lymphoblasts. precisely, the qvcnn receives a white cell image like the one shown in fig. 1 and classifies it as a lymphoblast or not. the classification of lymphoblast is essential for diagnosing acute lymphoblastic leukemia, a kind of blood cancer. the performance of the qvcnn is compared with a real-valued convolutional neural network with a similar architecture. fig. 1. candidate cell to be a lymphoblast, from all-idb dataset [28]. full size image the paper is structured as follows: sect. 2 presents the medical problem of acute lymphoblastic leukemia and presents a literature review on the computer-aided diagnosis of leukemia. section 3 addresses real-valued and quaternion-valued convolutional neural networks. the experimental results are detailed in sect. 4. the sect. 5 presents the concluding remarks and future works. 2 acute lymphoblastic leukemia (all) according to the national cancer institute of the united states, acute lymphoblastic leukemia (all) is a type of leukemia, cancer in the blood, that appears and multiplies rapidly [32]. all is characterized by the presence of many lymphoblasts in the blood and also in the bone marrow. in this context, a lymphoblast is an immature cell that can be converted into a mature lymphocyte [33]. there are several methods used for the diagnosis of all that can be found in the literature [34], including the peripheral blood smear technique [23]. the peripheral blood smear technique allows observing the information of a blood sample taken from the patient through a microscope. a specialist (hematologist) counts the number of lymphoblasts observed by microscope and, based on that, makes a diagnosis [45]. figure 2 shows a picture of a blood smear that a hematologist sees for analysis. it is worth mentioning that the white cells appear stained with a bluish-purple coloration, which serves as a guide to find lymphoblasts. fig. 2. blood smear image from all-idb dataset [28]. full size image the manual counting of lymphoblasts under the microscope is a somewhat dull task that takes much time from a professional who could be more productive in other matters. in effect, the time spent analyzing the microscope image has an economical cost because a specialist has significant value in the labor market. in addition, the analysis can be affected by human factors such as tiredness and stress. the operator’s experience also plays an important role, and therefore, there is a subjectivity component affecting the results of the lymphoblast count. for these reasons, computational models to perform automatic lymphoblast counting in a blood smear image have been proposed in the literature [42]. many methods divide the problem of automatic lymphoblast counting into two stages. the first stage, usually called the identification phase, aims to find white cells to be lymphoblast. labeling a candidate cell as a lymphoblast or healthy cell is performed in the second stage, referred to as the classification phase. in this paper, we use real-valued and quaternion-valued convolutional neural networks to classify white cells, that is, in the second stage of the blood smear image analysis. in the following sections, we review real-valued and quaternion-valued neural networks. before, however, we provide a literature review on automatic leukemia diagnosis methods. 2.1 computer-aided diagnosis of leukemia: literature review current literature has shown a large number of studies on computer-aided leukemia diagnosis with different approaches, including support vector machines (svm), k-nearest neighbor (k-nn), principal component analysis (pca), naive bayes classifier, and random forest [8]. in [26], the authors used 60 sample images to develop a model to detected all using knn and naive bayes classifier with 92.8% accuracy. a method to extract features of microscopic images using discrete orthogonal stockwell transform (dost) and linear discriminant analysis (lda) has been proposed in [31]. the paper [50] applies three pre-trained cnn architectures to extract features for image classification. in [2], a cnn reached 88.25% of accuracy in classifying all versus healthy cells. to distinguish between the four subtypes of leukemia, this cnn hits 81.74% accuracy. using all-idb dataset, [1] presents a k-medoids algorithm with 98.60% accuracy to classify white blood cells. furthermore, a method based on generative adversarial optimization (gao) [49], a neural network with statistic features [5], and a deep cnn with chronological sine-cosine algorithm (sca) [21] have been proposed for all detection with 93.84%, 97.07%, and 98.70% accuracy, respectively. a table summarizing the results from 16 papers on automated detection of leukemia and its subtypes can be found in [8]. this reference also presents a framework for automated leukemia diagnosis based on the resnet-34 [15] and the densenet-121 [19]. the accuracy reported was 99.56% for the resnet-34 and 99.91% for the densenet-121 [8]. 3 convolutional neural networks in many machine learning applications, identifying appropriate representations of a large amount of data is usually challenging. a successful model must efficiently encode local relations within the input resources and their structural relations. moreover, an adequate representation of data also offers a positive side effect by reducing the number of neural parameters needed to well-learn the input features, leading to a natural solution to the overfitting phenomenon [38]. convolutional neural networks (cnn) are feed-forward neural networks with a robust feature representation method widely applied in machine learning. for example, the resnet set a milestone in 2015 by outperforming humans in the imagenet competition [10, 15]. the successful alexnet [25] also inspired the development of many novel cnns including the vgg [44] and the densenet [19]. in addition, deep neural networks have been successfully used, for example, for segmentation tasks as well as for the automatic classification of objects in images [14, 18]. one crucial aspect of the deep networks is the convolution layer, which extracts features from high-dimensional data through a set of convolution kernels [51]. although convolutions perform well in many practical situations, it has some drawbacks in color image processing tasks. firstly, a convolution layer sums up the outputs corresponding to different channels and ignores their complicated interrelationships. as a consequence, it may eventually lose important information of a color image. secondly, simply summing up the outputs gives too many degrees of freedom, and thus, the network has a high risk of overfitting even when imposing heavy regularization terms [51]. accordingly, garcía-retuerta et al. argue that quaternion-valued neural networks may have a significant advantage in color image processing tasks because of quaternion’s four-dimensional algebraic structure [10]. the following section reviews the basic concepts of quaternion-valued convolutional neural networks. 3.1 quaternion-valued convolutional neural networks quaternions are a four-dimensional extension of complex numbers. developed by hamilton in 1843, the set of all quaternions is defined by $$\begin{aligned} \mathbb {h} = \{q = {q}_0 + {q}_1 \boldsymbol{i}+ {q}_2 \boldsymbol{j}+ {q}_3 \boldsymbol{k}: q_0,q_1,q_2,q_3 \in \mathbb {r} \} \end{aligned}$$ (1) where \(q_0\) is the real part of a quaternion, \(q_1\), \(q_2\), and \(q_3\) denote the imaginary components while \(\boldsymbol{i}\), \(\boldsymbol{j}\), \(\boldsymbol{k}\) are the hypercomplex units. the product of the hypercomplex units is governed by the following identities, knows as hamilton rules: $$\begin{aligned} \begin{array}{c} \boldsymbol{i}^2 = \boldsymbol{j}^2 = \boldsymbol{k}^2 = \boldsymbol{i}\boldsymbol{j}\boldsymbol{k}= -1. \end{array} \end{aligned}$$ (2) alternatively, a quaternion can be written as $$\begin{aligned} q = (q_0+q_1\boldsymbol{i}) + (q_2+q_3\boldsymbol{i})\boldsymbol{j}= z_0 + z_1 \boldsymbol{j}, \end{aligned}$$ (3) where \(z_0 = q_0 + q_1\boldsymbol{i}\) and \(z_1 = q_2+q_3\boldsymbol{j}\) are complex numbers. the addition of quaternions is performed adding the real and imaginary components. precisely, given \(p = {p}_0 + {p}_1 \boldsymbol{i}+ {p}_2 \boldsymbol{j}+ {p}_3 \boldsymbol{k}\) and \(q ={q}_0 + {q}_1 \boldsymbol{i}+ {q}_2 \boldsymbol{j}+ {q}_3 \boldsymbol{k}\), their sum is $$\begin{aligned} p+q = (p_0+q_0) + (p_1+q_1)\boldsymbol{i}+ (p_2+q_2)\boldsymbol{j}+ (p_3+q_3)\boldsymbol{k}. \end{aligned}$$ (4) the main result in quaternion algebra is the hamilton product between two quaternions \(p = {p}_0 + {p}_1 \boldsymbol{i}+ {p}_2 \boldsymbol{j}+ {p}_3 \boldsymbol{k}\) and \(q = {q}_0 + {q}_1 \boldsymbol{i}+ {q}_2 \boldsymbol{j}+ {q}_3 \boldsymbol{k}\), denoted by \(p \otimes q\) and defined by $$\begin{aligned} \begin{array}{rl} p \otimes q &{}= (p_0 q_0 p_1 q_1 p_2 q_2 p_3 q_3) + \, (p_0 \, q_1 + p_1 \, q_0 + p_2 \, q_3 p_3 \, q_2) \, \boldsymbol{i}\\ &{} \quad + \, (p_0 \, q_2 p_1 \, q_3 + p_2 \, q_0 + p_3 \, q_1) \, \boldsymbol{j}+ \, (p_0 \, q_3 + p_1 \, q_2 p_2 \, q_1 + p_3 \, q_0) \, \boldsymbol{k}\end{array} \end{aligned}$$ (5) quaternions and quaternion algebra allow building processing entities composed of four elements that share information via the hamilton product. according to gaudet and maida [11], a quaternion-valued convolutional layer is obtained convolving a quaternion-valued filter matrix \(\boldsymbol{w} = {\boldsymbol{w}}_0 + {\boldsymbol{w}}_1 \boldsymbol{i}+ {\boldsymbol{w}}_2 \boldsymbol{j}+ {\boldsymbol{w}}_3 \boldsymbol{k}\) by a quaternion-valued vector \(\boldsymbol{h} = {\boldsymbol{h}}_0 + {\boldsymbol{h}}_1 \boldsymbol{i}+ {\boldsymbol{h}}_2 \boldsymbol{j}+ {\boldsymbol{h}}_3 \boldsymbol{k}\). here, \(\boldsymbol{w}_0\), \(\boldsymbol{w}_1\), \(\boldsymbol{w}_2\), and \(\boldsymbol{w}_3\) are real-valued matrices while \(\boldsymbol{h}_0\), \(\boldsymbol{h}_1\), \(\boldsymbol{h}_2\), and \(\boldsymbol{h}_3\) are real-valued vectors. details on the implementation of quaternion-valued convolutional layers can be found in [11]. fig. 3. rvcnn and qvcnn architectures. full size image 4 computational experiments let us compare real-valued and quaternion-valued convolutional neural networks’ performance for classifying a white cell image as a lymphoblast. both real-valued and quaternion-valued neural networks have been implemented in python using the keras and tensorflow libraries. the real-valued model is a sequential feed-forward network composed of three convolutional layers, three max-pooling layers, and a dense layer. precisely, the first convolutional layer has 32 filters with a (3, 3) kernel and relu activation function. a max-pooling follows the convolutional layer with a (2, 2) kernel. the second and third two-dimensional convolutional layers have 64 and 128 filters and have relu activation functions. furthermore, they are also followed by max-pooling layers with (2, 2) kernels. figure 3 shows the architecture of the real-valued convolutional neural network. the total number of trainable parameters of the real-valued convolutional neural network is 106, 049. the quaternion-valued convolutional neural network has been designed similarly. precisely, to maintain the same parameter budget among the real and quaternion-valued models, the number of filters per layer of the real-valued network was divided by four to build a quaternion-valued convolution. thus, the quaternion-valued convolutional neural network has the same structure as the real-valued network depicted in fig. 3, but with a quarter of the number of filters per layer. the number of trainable parameters of the quaternion-valued cnn model is 36, 353. table 1 summarizes the number of trainable parameters of both neural networks per layer. table 1. parameters of real and quaternion-valued neural networksfull size table the dense layer of both real-valued and quaternion-valued networks has a single output neuron without activation function. such a single neuron is used to classify the input image as a lymphoblast or not. moreover, the parameters of all layers have been initialized according to glorot and bengio [12]. the optimizer used was adam, an algorithm based on the stochastic gradient descent method with adaptive estimation of first-order and second-order moments. to evaluate the performance of the rvcnn and qvcnn classifiers, we used the all-idb: the acute lymphoblastic leukemia image database for image processing provided by the “università degli studi di milano” [28]. this image database contains 260 images of white blood cells with \(257 \times 257\) pixels, labeled by experts and evenly distributed among lymphoblast and health cells. figure 1 shows an example of a color image used in the computational experiment. we resized the \(257 \times 257\) white blood cells images to \(100 \times 100\) pixels. also, the set of 260 color images was randomly divided into training and test images with different ratios. data augmentation has been applied on the training set to improve the accuracy of the convolutional neural networks. precisely, the images used for training were all submitted to a pre-processing data generation, which consists of obtaining new images through horizontal and vertical flips. in our experiments, images were converted to rgb (red, green, and blue) and hsv (hue, saturation, and value) color spaces and used as input to neural networks. as a consequence, we performed the four experiments detailed in table 2. the first experiment considers a real-valued cnn whose input is obtained by concatenating the three rgb channels in a single tensor with values in the unit interval [0, 1]. the second experiment also considers real-valued cnns, but the input is obtained by concatenating the three hsv channels. here, hue is arranged in a radial slice \(h \in [0,2\pi )\) while saturation and value belong to the unit interval, i.e., \(s, \ v \in [0,1]\). table 2. experiments with real and quaternion-valued neural networksfull size table the last two experiments were performed using quaternion-valued cnn. specifically, in the third experiment, the rgb image is encoded in a quaternion structure with real part null, and each channel as one imaginary part of a quaternion as follows: $$\begin{aligned} q = 0 + r \ \boldsymbol{i}+ g \ \boldsymbol{j}+ b \ \boldsymbol{k}. \end{aligned}$$ (6) finally, in the fourth experiment, a color is encoded in a quaternion through the following expression using the hsv representation: $$\begin{aligned} q = s \cos (h) + s \sin (h) \ \boldsymbol{i}+ v \cos (h) \ \boldsymbol{j}+ v \sin (h) \ \boldsymbol{k}. \end{aligned}$$ (7) the dataset has been divided into training and test sets with 5 different training/test ratios and trained by 100 epochs. one hundred simulations were performed for each different training/test ratio and, the average and standard deviation of the accuracy was calculated. figure 4 presents the average accuracy of both real-valued and quaternion-valued convolutional neural networks for different percentages used for testing the networks in the four experiments. this figure also presents the interval between the \(25\%\) and the \(75\%\) quantiles of accuracy as shaded area. fig. 4. accuracy by the percentage of the dataset used for testing. full size image note from fig. 4 that the quaternion-valued convolutional neural network with images in hsv color space (qvcnn-hsv) obtained the best performance, reaching 98.2% of accuracy in the test phase with 10% of training/test ratio. the real and quaternion-valued networks with rgb encoded images exhibited similar performance, with accuracy between \([93.6\%, \ 97.1\%]\) and \([94.4\%,\) \(97.3\%]\), respectively, depending on the ratio training/test. the real and quaternion-valued cnn models with rgb encoded images exhibited statistically equivalent performances. the real-valued neural network with hsv encoded images yielded the worst performance, reaching an average accuracy of 95.3% in the best case. concluding, the qvcnn-hsv exhibits a better generalization capability than the qvcnn-rgb, rvcnn-rgb, and rvcnn-hsv models. moreover, the performance of the quaternion-valued convolutional neural network with images encoded using the hsv color space and (7) compares well with the results reported in the literature (see sect. 2.1). however, the quaternion-valued convolutional neural network is much simpler than many of the architectures considered previously. 5 concluding remarks and future works acute lymphoblastic leukemia is characterized by many lymphoblasts in the blood and the bone marrow. such disease can be diagnosticated by counting the number of lymphoblasts in a blood smear microscope image. this paper investigated the application of convolutional neural networks for classifying a white cell as lymphoblast or not. precisely, we compared the performance of real-valued and quaternion-valued models. the qvcnn with input images encoded using the hsv color space showed the best result in our experiments. also, the performance of the qvcnn is comparable with other deeper neural networks from the literature, including the resnet and the densenet [8]. this computational experiment suggests that quaternion-valued neural networks exhibit better generalization capability than the real-valued convolutional neural network, possibly because it treats colors as single quaternion entities. furthermore, it is noticeable that the quaternion-valued convolutional neural network has about 34% of the parameters of the corresponding real-valued model. we plan to develop neural networks that segment and classify white blood cells on a blood smear microscope image as future work. further research can also address the application of qvcnn for the classification of other types of leukemia. references acharya, v., kumar, p.: detection of acute lymphoblastic leukemia using image segmentation and data mining algorithms. med. biol. eng. comput. 57(8), 1783–1811 (2019). https://doi.org/10.1007/s11517-019-01984-1 article  google scholar  ahmed, n., yigit, a., isik, z., alpkocak, a.: identification of leukemia subtypes from microscopic images using convolutional neural network. diagnostics (basel) 9(3) (2019). https://doi.org/10.3390/diagnostics9030104 aizenberg, i., alexander, s., jackson, j.: recognition of blurred images using multilayer neural network based on multi-valued neurons. in: 2011 41st ieee international symposium on multiple-valued logic, pp. 282–287 (2011) google scholar  aizenberg, i., gonzalez, a.: image recognition using mlmvn and frequency domain features. in: 2018 international joint conference on neural networks (ijcnn), pp. 1–8 (2018). https://doi.org/10.1109/ijcnn.2018.8489301 aljaboriy, s., sjarif, n., chuprat, s., abduallah, w.: acute lymphoblastic leukemia segmentation using local pixel information. pattern recogn. lett. 125, 85–90 (2019). https://doi.org/10.1016/j.patrec.2019.03.024 arena, p., fortuna, l., muscato, g., xibilia, m.g.: multilayer perceptrons to approximate quaternion valued functions. neural netw. 10(2), 335–342 (1997). https://doi.org/10.1016/s0893-6080(96)00048-2 article  google scholar  bayro-corrochano, e., lechuga-gutiérrez, l., garza-burgos, m.: geometric techniques for robotics and hmi: interpolation and haptics in conformal geometric algebra and control using quaternion spike neural networks. robot. auton. syst. 104, 72–84 (2018) article  google scholar  bibi, n., sikandar, m., din, i.u., almogren, a., ali, s.: iomt-based automated detection and classification of leukemia using deep learning. j. healthc. eng. (2020). https://doi.org/10.1155/2020/6648574 article  google scholar  chen, b., gao, y., xu, l., hong, x., zheng, y., shi, y.q.: color image splicing localization algorithm by quaternion fully convolutional networks and superpixel-enhanced pairwise conditional random field. math. biosci. eng. 6(16), 6907–6922 (2019). https://doi.org/10.3934/mbe.2019346 article  mathscinet  math  google scholar  garcía-retuerta, d., casado-vara, r., martin-del rey, a., de la prieta, f., prieto, j., corchado, j.m.: quaternion neural networks: state-of-the-art and research challenges. in: analide, c., novais, p., camacho, d., yin, h. (eds.) ideal 2020. lncs, vol. 12490, pp. 456–467. springer, cham (2020). https://doi.org/10.1007/978-3-030-62365-4_43 chapter  google scholar  gaudet, c.j.; maida, a.: deep quaternion networks. in: international joint conference on neural networks (ijcnn), pp. 1–8 (2018) google scholar  glorot, x., bengio, y.: understanding the difficulty of training deep feedforward neural networks. in: teh, y.w., titterington, m. (eds.) proceedings of the thirteenth international conference on artificial intelligence and statistics. proceedings of machine learning research, vol. 9, pp. 249–256. pmlr, chia laguna resort, sardinia, italy (13–15 may 2010). http://proceedings.mlr.press/v9/glorot10a.html greenblatt, a., mosquera-lopez, c., agaian, s.: quaternion neural networks applied to prostate cancer gleason grading. in: 2013 ieee international conference on systems, man, and cybernetics, pp. 1144–1149 (2013). https://doi.org/10.1109/smc.2013.199 he, k., gkioxari, g., dollár, p., girshick, r.: mask r-cnn (2018) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: 2016 ieee conference on computer vision and pattern recognition (cvpr), pp. 770–778 (2016). https://doi.org/10.1109/cvpr.2016.90 hirose, a.: complex-valued neural networks. studies in computational intelligence, 2nd edn. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-27632-3 hirose, a., yoshida, s.: generalization characteristics of complex-valued feedforward neural networks in relation to signal coherence. ieee trans. neural netw. learn. syst. 23(4), 541–551 (2012). https://doi.org/10.1109/tnnls.2012.2183613 article  google scholar  hu, j., shen, l., sun, g.: squeeze-and-excitation networks. in: 2018 ieee/cvf conference on computer vision and pattern recognition, pp. 7132–7141 (2018). https://doi.org/10.1109/cvpr.2018.00745 huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: 2017 ieee conference on computer vision and pattern recognition (cvpr), pp. 2261–2269 (2017). https://doi.org/10.1109/cvpr.2017.243 isokawa, t., matsui, n., nishimura, h.: quaternionic neural networks: fundamental properties and applications. in: complex-valued neural networks: utilizing high-dimensional parameters, pp. 411–439 (2009) google scholar  jha, k.k., sekhar dutta, h.: mutual information based hybrid model and deep learning for acute lymphocytic leukaemia detection in single cell blood smear images. comput. methods program. biomed. 179, 104987 (2019). https://doi.org/10.1016/j.cmpb.2019.104987 jin, l., zhou, y., liu, h., song, e.: deformable quaternion gabor convolutional neural network for color facial expression recognition. in: 2020 ieee international conference on image processing (icip), pp. 1696–1700 (2020). https://doi.org/10.1109/icip40778.2020.9191349 kasvi: hematologia: como é realizada a técnica de esfregaço de sangue? https://kasvi.com.br/esfregaco-de-sangue-hematologia/ (2021). accessed 18 feb 2021 kinugawa, k., shang, f., usami, n., hirose, a.: isotropization of quaternion-neural-network-based polsar adaptive land classification in poincare-sphere parameter space. ieee geosci. remote sens. lett. 15(8), 1234–1238 (2018). https://doi.org/10.1109/lgrs.2018.2831215 article  google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. commun. acm 60(6), 84–90 (2017). https://doi.org/10.1145/3065386 article  google scholar  kumar, s., mishra, s., asthana, p.: pragya: automated detection of acute leukemia using k-mean clustering algorithm. in: bhatia, s.k., mishra, k.k., tiwari, s., singh, v.k. (eds.) advances in computer and computational sciences, pp. 655–670. springer, singapore (2018). https://doi.org/10.1007/978-981-10-3773-3_64 chapter  google scholar  kusamichi, h., isokawa, t., matsui, n., ogawa, y., maeda, k.: a new scheme for color night vision by quaternion neural network. in: proceedings of the 2nd international conference on autonomous robots and agents (icara 2004), pp. 101–106 (2004) google scholar  labati, r.d., piuri, v., scotti, f.: all-idb: the acute lymphoblastic leukemia image database for image processing. in: 2011 18th ieee international conference on image processing (2011). https://doi.org/978-1-4577-1303-3 mandic, d.p., goh, v.s.l.: complex valued nonlinear adaptive filters: noncircularity, widely linear and neural models, vol. 59. wiley, new york (2009) book  google scholar  matsui, n., isokawa, t., kusamichi, h., peper, f., nishimura, h.: quaternion neural network with geometrical operators. j. intell. fuzzy syst. 15(3), 149–164 (2004) math  google scholar  mishra, s., majhi, b., sa, p.k.: texture feature based classification on microscopic blood smear for acute lymphoblastic leukemia detection. biomed. sig. process. control 47, 303–311 (2019). https://doi.org/10.1016/j.bspc.2018.08.012 nci: acute lymphoblastic leukemia. https://www.cancer.gov/publications/dictionaries/cancer-terms/def/acute-lymphoblastic-leukemia (2021). accessed 18 feb 2021 nci: lymphoblast. https://www.cancer.gov/publications/dictionaries/cancer-terms/def/lymphoblast (2021). accessed 18 feb 2021 nhs: acute lymphoblastic leukemia diagnosis. https://www.nhs.uk/conditions/acute-lymphoblastic-leukaemia/diagnosis/ (2021). accessed 18 feb 2021 nitta, t.: on the critical points of the complex-valued neural network. in: proceedings of the iconip 2002 9th international conference on neural information processing: computational intelligence for the e-age, pp. 411–439. singapore (2002) google scholar  ogawa, t.: neural network inversion for multilayer quaternion neural networks. comput. technol. appl. 7, 73–82 (2016) google scholar  onyekpe, u., palade, v., kanarachos, s., christopoulos, s.r.: a quaternion gated recurrent unit neural network for sensor fusion. information (2021). https://doi.org/10.3390/info12030117 parcollet, t., morchid, m., linarès, g.: a survey of quaternion neural networks. artif. intell. rev. 53(4), 2957–2982 (2020). https://doi.org/10.1007/s10462-019-09752-1 article  google scholar  parcollet, t., morchid, m., linarès, g.: quaternion convolutional neural networks for heterogeneous image process. (2018). https://doi.org/arxiv:1811.02656v1 parcollet, t., et al.: quaternion convolutional neural networks for end-to-end automatic speech recognition. in: proceedings of the interspeech 2018, pp. 22–26 (2018). https://doi.org/10.21437/interspeech.2018-1898 pavllo, d., feichtenhofer, c., auli, m., grangier, d.: modeling human motion with quaternion-based neural networks. int. j. comput. vis. 128(4), 855–872 (2019). https://doi.org/10.1007/s11263-019-01245-6 shafique, s., tehsin, s.: computer-aided diagnosis of acute lymphoblastic leukaemia. comput. math. methods med. 2018, 6125289 (2018). https://doi.org/10.1155/2018/6125289 shang, f., hirose, a.: quaternion neural-network-based polsar land classification in poincare-sphere-parameter space. ieee trans. geosci. remote sens. 52, 5693–5703 (2014) article  google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition (2015) google scholar  terwilliger, t., abdul-hay, m.j.b.c.j.: acute lymphoblastic leukemia: a comprehensive review and 2017 update. blood cancer j. (2017). https://doi.org/10.1038/bcj.2017.53 takahashi, k., isaka, a., fudaba, t., hashimoto, m.: remarks on quaternion neural network-based controller trained by feedback error learning. in: ieee/sice international symposium on system integration, pp. 875–880 (2017) google scholar  takahashi, k., takahashi, s., cui, y., hashimoto, m.: remarks on computational facial expression recognition from hog features using quaternion multi-layer neural network. in: mladenov, v., jayne, c., iliadis, l. (eds.) eann 2014. ccis, vol. 459, pp. 15–24. springer, cham (2014). https://doi.org/10.1007/978-3-319-11071-4_2 chapter  google scholar  trabelsi, c., et al.: deep complex networks (may 2017) google scholar  tuba, m., tuba, e.: generative adversarial optimization (goa) for acute lymphocytic leukemia detection. stud. inf. control 28, 245–254 (2019). https://doi.org/10.24846/v28i3y201901 article  google scholar  vogado, l.h., veras, r.m., araujo, f.h., silva, r.r., aires, k.r.: leukemia diagnosis in blood slides using transfer learning in cnns and svm for classification. eng. appl. artif. intell. 72, 415–422 (2018). https://doi.org/10.1016/j.engappai.2018.04.024 zhu, x., xu, y., xu, h., chen, c.: quaternion convolutional neural networks. in: ferrari, v., hebert, m., sminchisescu, c., weiss, y. (eds.) eccv 2018. lncs, vol. 11212, pp. 645–661. springer, cham (2018). https://doi.org/10.1007/978-3-030-01237-3_39 chapter  google scholar  download references author information authors and affiliations instituto federal de educação, ciência e tecnologia de são paulo, são paulo, brazil marco aurélio granero escuela superior politécnica del litoral, guayaquil, ecuador cristhian xavier hernández universidade estadual de campinas, campinas, brazil marcos eduardo valle authors marco aurélio graneroview author publications search author on:pubmed google scholar cristhian xavier hernándezview author publications search author on:pubmed google scholar marcos eduardo valleview author publications search author on:pubmed google scholar corresponding author correspondence to marcos eduardo valle . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper granero, m.a., hernández, c.x., valle, m.e. (2021). quaternion-valued convolutional neural network applied for acute lymphoblastic leukemia diagnosis. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_20 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords quaternion algebra hypercomplex number neural network quaternion neural network classification problem publish with us policies and ethics profiles marcos eduardo valle view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature heact: hybrid evolutionary algorithm for the multi-region multi-objective cloud task scheduling problem. a study of workflow scheduling in aws ec2 | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper heact: hybrid evolutionary algorithm for the multi-region multi-objective cloud task scheduling problem. a study of workflow scheduling in aws ec2 conference paper first online: 30 january 2025 pp 49–64 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) heact: hybrid evolutionary algorithm for the multi-region multi-objective cloud task scheduling problem. a study of workflow scheduling in aws ec2 download book pdf download book epub vinicius renan de carvalho  orcid: orcid.org/0000-0002-4623-72449 & jaime simão sichman  orcid: orcid.org/0000-0001-8924-96439  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 402 accesses 1 citation abstract cloud computing has revolutionized the provisioning and access of computing resources, offering scalable and flexible alternatives to traditional infrastructure. however, defining how to use these computational resources may be challenging. this paper addresses the challenge of workflow scheduling in cloud environments, focusing on amazon web services (aws) elastic compute cloud (ec2). we present heact, a novel approach that integrates a multi-objective evolutionary algorithm with a specialist scheduling heuristic. the evolutionary algorithm is responsible for generating an initial set of machines (with their performance capability and cost information). the set is sent to the specialist scheduling heuristic for efficient task assignment in these machines. our approach considers fourteen aws regions, accurate pricing information from aws, and employs simgrid to simulate task execution. the proposed method was benchmarked considering established heuristics (heft, peft, hsip, mpeft) and meta-heuristics (nsga-ii, agemoea2). results demonstrated that the combinations of agemoea2 with mpeft and agemoea2 with heft yield the best performance, indicating agemoea2’s efficacy as a state-of-the-art meta-heuristic for workflow scheduling. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-objective workflow scheduling in cloud computing: trade-off between makespan and cost article 10 october 2021 hybrid meta-heuristic genetic algorithm: differential evolution algorithms for scientific workflow scheduling in heterogeneous cloud environment chapter © 2023 a hybrid genetic algorithm for scientific workflow scheduling in cloud environment article 11 may 2020 1 introduction cloud computing changed how computational resources are provisioned and accessed, offering a scalable and flexible alternative to traditional on-premises infrastructure. moreover, elasticity allows users to dynamically adjust their resource usage based on demand, ensuring that users can scale their applications up or down, paying only for the resources they consume. amazon web services (aws) is an infrastructure as a service (iaas) that provides virtualized computing resources over the internet. in this context, with aws ec2footnote 1, users can provision virtual machines (vms) on demand according to their requirements. aws ec2 vms come in various instance types, from smaller to high-performance computing instances. their computational power correlates with their capabilities and costs. effective resource management and scheduling are important in maximizing the benefits of cloud computing, especially when processing workflows. workflow scheduling in cloud computing, which is known as np-complete, consists of orchestrating complex sequences of tasks to be processed. workflows are commonly represented as directed acyclic graphs (dags), where nodes represent tasks and edges denote dependencies between tasks. several approaches have been proposed over the years in order to schedule tasks to optimize performance. the list-based scheduling algorithm is the most popular scheduling paradigm for the computation of complex workflow applications [1]. among them, we can cite heft [26], peft [2], hsip [27] and mpeft [20] (all described in sect. 2) as heuristic specialized in assigning tasks to processors (or machines) focusing on reducing the total time for executing all tasks. this total time is named as makespan. over the years, genetic algorithms (ga) were also considered for scheduling tasks in this domain. in [15], a single objective genetic algorithm combined with heft. in this approach, the ga considers single-point crossover and mutation operators. the chromosome was defined as an ordered list of tasks. as commonly performed in heuristics for this problem, a fixed set of processors was considered. nasonov et al. [17] proposed linewise earliest finish time (left) by combining the heft heuristic and the genetic algorithm to optimize the makespan for cloud workload optimization. in [24], the authors introduced two hybrid heuristics based on genetic algorithms for task scheduling. both algorithms enhance the initial population of the ga by incorporating guided chromosomes using novel prioritization equations and improved mutation strategies. similar to [15], these algorithms consider as chromosomes an ordered list of tasks, use two-point crossover and proposed mutation, use makespan as a single-objective function, and consider a specialist heuristic, in that case, hltsd [25]. these mentioned works are, in fact, effective. however, an overall literature review reveals an evident lack of workflow scheduling algorithms that optimize equally important makespan and monetary cost functions [13]. despite of that, durillo et al. [11] introduced moheft, a multi-objective workflow scheduling algorithm based on heft. moheft aims to optimize two key objectives simultaneously: minimizing makespan and enhancing energy efficiency. however, there are significant challenges in directly applying these algorithms to cloud environments because most of them are still based on traditional heterogeneous environments such as grid [30]. in order to tackle this challenge, zhu et al. [30] proposed using an evolutionary multi-objective optimization algorithm to address the cloud workflow scheduling problem as a multi-objective problem. this approach generates a series of schedules with different tradeoffs between cost and time so that users can choose acceptable schedules that fit their preferences. the chromosome is defined using three arrays: one responsible for ordering the tasks, the other responsible for assigning the tasks to a machine, and the other for defining the type of the machine. they applied nsga-ii [9], spea2 [31], and moea/d [29]. the case study revolves around amazon ec2, utilizing instance specifications and pricing schemes, focusing on the us east region, considering only a single region and a small number of machine types. chen and menzies [7] proposed riot (randomized instance order types). this heuristic method considers aspects from heft to group tasks in the workflow, considering a probability model and a surrogate-based method. this paper used the exact chromosome representation considered by [30] and a similar setup. qin et al. [21] approach the same problem, focusing on three key objectives: minimizing makespan and cost while maximizing reliability. this problem is situated within the context of multi-cloud systems, where machines from both azure and aws were utilized. to address this task, they proposed a variation of the memetic algorithm [16] for multi-objective problems. the studied case study considered the same workflow scheduling problem but in the multi-cloud scenario. motivated by research that combines genetic algorithms with specialist heuristics and approaches that consider the problem as a multi-objective problem, this work proposes heact, hybrid evolutionary algorithm for the multi-region multi-objective cloud task scheduling problem. contributions: this research has as contributions 1. proposes a hybrid algorithm that considers a multi-objective evolutionary algorithm and a specialist heuristic in order to solve the multi-objective task scheduling problem considering makespan and cost as objectives; 2. considers multi-region: instances located in different places in the world have different associated costs. moreover, regions may have a different set of machine types; 3. uses realistic aws information to instantiate the simulator (simgrid); 4. considers state-of-the-art scheduling heuristic (specially hsip and mpeft) with state-of-art meta-heuristic (especially agemoea2 [19]). as far as we know, mpeft was never studied together with a ga, while agemoea was never applied to the cloud scheduling problem. this paper is organized as follows: first, in sect. 2, we provide some background necessary for the understanding of our proposal. then, in sect. 3, we present our proposal. then, in sect. 4, we detail our experiments and provide results to demonstrate the effectiveness of our approach. finally, in sect. 5, we present our conclusions and further work. 2 background in this section, we provide the background necessary for understanding our proposal. first, we define the cloud scheduling problem and then detail the specialized heuristics used to solve it. finally, we introduce meta-heuristics briefly. 2.1 scheduling problem formulation the cloud task scheduling problem can be modeled as a dag \(g = (v, e)\) composed by a set of nodes \(t_i \in v\), which represents an application task to be executed, and a set of edges e, which represents dependency between tasks and have an associated cost. thus, given \(e(i,j) \in e\), a task \(t_i\) should be wholly executed before starting task \(t_j\). the scheduling problem consists of assigning tasks to processors considering dependency between n tasks presented in the dag and information regarding the processors. in this paper, we commonly use the name processor and, in our context, consider a processor to be the same as a virtual machine. equation 1 calculates the objective function of the scheduling problem. $$\begin{aligned} makespan = \max \{aft(t_{exit})\} \end{aligned}$$ (1) the actual finish time (aft) represents when all the tasks and dependencies were executed and considers the last task \(t_{exit}\). in order to calculate an aft, we have to calculate the earliest starting time (est) and the earliest finish time (eft) for all the tasks. $$\begin{aligned} eft(t_i,p_j) = w_{t_i,p_j} + est(t_i,p_j) \end{aligned}$$ (2) thus, to calculate when a task will finish, we sum the starting time with the computational cost \(w_{t_i,p_j}\). this is performed for a given task \(t_i\) considering a given processor \(p_j\). the computation cost matrix w of size \(nt \times np\), where nt is the number of tasks, and np is the number of processors. each \(w_{i,j} \in w\) provides the estimated execution time to complete task \(t_i\) on processor \(p_j\). the earliest starting time (est in the previous equation) is calculated according to eq. 3: $$\begin{aligned} est(t_i, p_j) = \max \{t_{avaliable}(p_j), \max _{t_m \in pred(t_i)} \{eft(t_m) + c_{m,i}\}\} \end{aligned}$$ (3) in this equation, \(t_{avaliable}(p_j)\) is the time at which processor \(p_j\) is available and \(pred(t_i)\) represents all tasks that \(t_i\) is dependent. we first calculate the latest time we can get all dependencies executed plus communication time. thus, we guarantee that all dependencies will be respected. this scheduling problem considers that a given task \(t_m\) could depend on another task \(t_i\), which is located in another processor. hence, it considers how much time will be necessary for the data generated by \(t_i\), which is an input in \(t_m\), arrives at the processor that \(t_m\) will be processed. this is what we call as a communication cost and it is represented by \(c_{m,i}\), which is the communication cost for tasks \(t_i\) and \(t_j\) and it is defined by: $$\begin{aligned} c_{m,i} = l_{t_m} + \frac{data_{i,m}}{b_{t_m,t_i}} \end{aligned}$$ (4) in this equation, b represents the transfer rates between the processors. this matrix has a size of \(np \times np\), where np is the number of processors. l represents the processors’ communication startup costs. \(data_{i,m}\) represents the amount of data to be transferred and processed. if \(t_i\) and \(t_m\) are at the same machine, then \(c_{m,i}=0\). the mono-objective scheduling problem considers only the makespan as an objective to be minimized. this is a realistic scenario only when we have a static set of processors waiting to receive tasks to run, so in this case, we are just trying to use the resources we already have (on-premises servers) better. recently, with the advent of cloud providers, we can use the machines we need. by doing this, costs are reduced when using other companies’ computational resources. in the context of task scheduling, we can also consider the cost to be optimized and then treat this problem as a multi-objective optimization problem (mop). equation 5 presents this function; there, we convert the makespan into a fraction of an hour and then multiply it by the cost of every machine (processor) we are considering. each machine has a type (\(p_{type}\)), which defines how powerful the machine is and how much it will cost. the same happens for the region where the machine is located (\(p_{region}\)). usually, the same machine type in different regions may have different prices. $$\begin{aligned} totalcost = \sum _{p \in p} \frac{makespan}{3600} * cost_{(p_{type}, p_{region})} \end{aligned}$$ (5) 2.2 heuristics to deal with the problem several of the specialized heuristics in the scheduling problem consist of sorting the tasks according to a criterion and then iterating through this ranking in order to assign tasks to processors considering the shorter starting time available. for example, in the heterogeneous earliest finish time algorithm (heft) [26], tasks are ranked decreasingly according to the upward rank value (eq. 6), which is based on mean computation and mean communication costs. $$\begin{aligned} rank_u(t_i) =\bar{ w_i} + \max _{t_j \in succ(t_i)} (\bar{c_{i,j}} + rank_u(t_j)) \end{aligned}$$ (6) in this equation, succ is the set of immediate successors of task \(t_i\) \(\bar{c_{i,j}}\) is the average communication cost of an edge (i, j) and \(\bar{w_i} \) is the average computation cost of task \(t_i\) among the processors. similarly, the predict earliest finish time algorithm (peft) [2] also ranks tasks but considers the optimistic cost table (oct) for the ranking phase. the oct (eq. 7) is a matrix of size \(np \times nt\) where each element represents, for a given task \(t_i\) and processor \(p_k\), the maximum of the shortest paths of \(t_i\) dependencies tasks to the exit node. \(\bar{c_{i,j}} = 0\) if \(p_w=p_k\). $$\begin{aligned} oct(t_i, p_k) = \max _{t_j \in succ(t_i)} \left[ \min _{pw \in p} \{oct(t_j,p_w) + w(t_j, p_w) + c_{i,j}\} \right] \end{aligned}$$ (7) the oct represents the maximum optimistic processing time of the children of task \(t_i\) because it considers that dependencies tasks are executed in the processor that minimizes processing time (communications and execution) independently of processor availability [2]. with the values of oct calculated, we can then rank the values decreasingly, according to eq. 8. $$\begin{aligned} rank_{oct}(t_i) = \frac{\sum _{k=1}^{qp} oct(t_i, p_k)}{qp} \end{aligned}$$ (8) in order to select a processor for a task, the algorithm considers the lowest optimistic eft (oeft eq. 9), which sums to the estimated finish time of a given task to the computation time of the longest path to the exit node (oct). the aim is to guarantee that the tasks ahead will finish earlier, which is the purpose of the oct table. $$\begin{aligned} oeft(t_i,p_j) = eft(t_i, p_j) + oct(t_i, p_j) \end{aligned}$$ (9) in [27], the authors proposed the heterogeneous scheduling algorithm with improved task priority (hsip). this algorithm is considered a prioritization strategy based on computational cost standard deviation and total communication cost for a task (eq. 10). \(\sigma _i\) is the standard deviation of the computational cost of the given task \(t_i\) all over the processors while \(\bar{w_i}\). $$\begin{aligned} rank_{u}(t_i) = \max _{t_j \in succ(t_i)} \left\{ \bar{w_i} \times \sigma _i + occw(t_i) + rank_u(t_j) \right\} \end{aligned}$$ (10) moreover, the occw term is defined by eq. 11, where \(occw(t_{exit})=0\). $$\begin{aligned} occw(t_i) = \sum _{t_j \in succ(t_i)} c_{i,j} \end{aligned}$$ (11) hsip also implements an entry task duplication in order to improve the makespan and uses an improved idle time slot insertion-based optimizing policy to make the task scheduling more efficient. recently, the modified predict earliest finish time algorithm (mpeft) [20] was proposed. equation 12 presents how this algorithm ranks tasks. $$\begin{aligned} rank_{ap} (t_i) = dct(t_i) + \sum _{t_j \in os(t_i)} dct(t_j) \end{aligned}$$ (12) the mpeft ranking equation considers all direct and indirect successor tasks. thus, instead of only using succ as the other algorithms, mpeft also considers all indirect tasks according to eq. 13. $$\begin{aligned} os(t_i) = {\left\{ \begin{array}{ll} \bigcup _{t_j \in succ(t_i)} os(t_j) \iff t_j \not t_{exit} \\ \emptyset \text { otherwise} \end{array}\right. } \end{aligned}$$ (13) mpeft ranking considers dct, where the minimal computational time for a given task \(t_i\) (represented by \(w^*(t_i)\)) is the sum with the summation of the average transfer costs \( c_{i,j}\). this is expressed by eq. 14. $$\begin{aligned} dct(t_i) = w^*(t_i) + \sum _{t_j \in succ(t_i)} c_{i,j} \end{aligned}$$ (14) similarly to peft, mpeft does not directly consider the estimated finish time (eft) but incorporates it into another task’s estimated finish, represented by meft (eq. 15). $$\begin{aligned} meft(t_i,p_k) = eft(t_i,p_k) + oct(t_j,p_k) \times k(t_i,p_k) \end{aligned}$$ (15) the meft equation also considers the oct value (eq. 7) as performed in peft but also uses a weight for the oct value. this weight is calculated according to eq. 16. $$\begin{aligned} k(t_i, p_k) = {\left\{ \begin{array}{ll} \frac{rank_{ap} (cps(t_i, p_k))}{\sum _{t_j \in succ(t_i)} rank_{ap}(t_j) + c_{i,j}} \iff |succ(t_i)| > |p| + 1 \\ 1\text { otherwise} \end{array}\right. } \end{aligned}$$ (16) where the cps is essentially the \(\mathop {\textrm{argmax}}\limits \) of eq. 7, this is given by equation 17, which is pretty similar to eq. 7. $$\begin{aligned} cps(t_i, p_k) = \mathop {\textrm{argmax}}\limits \nolimits _{t_j \in succ(t_i)} \left[ \min _{pw \in p} \{oct(t_j,p_w) + w(t_j, p_w) + c_{i,j}\} \right] \end{aligned}$$ (17) in all presented heuristics, tasks are ordered according to the heuristic criterion. based on that order, tasks are assigned to the processor that is the most early available. the best available processor is found by considering the time that the processor will be available (last task executing in the processor + task duration) + the amount of time to send the task to that processor. 2.3 meta-heuristics meta-heuristics are algorithms, solution methods that orchestrate an interaction between local improvement procedures (heuristics) and higher-level strategies to create a process capable of escaping from local optima and performing a robust search of a solution space [12]. thus, heuristics usually specializes in solving problems for one particular domain, while meta-heuristics are more generic and adaptive in several domains. one of the most used meta-heuristic is the evolutionary algorithm (eas) [8]. eas are meta-heuristics that employ darwin’s theory of the survival of the fittest as their inspiration. this kind of algorithm generates solutions using the crossover and mutation heuristic operators. it employs an objective function to choose which individuals (solutions for the problem) will compose the next population. multi-objective evolutionary algorithm (moea) has been successfully applied to the solution of mops (multi-objective optimization problems) [5]. these algorithms are heuristic techniques that allow a flexible representation of the solutions and do not impose continuity conditions on the functions to be optimized. moreover, moeas are extensions of eas for multi-objective problems that usually apply the concepts of pareto dominance. these algorithms typically differ in their replacement strategies. in a replacement strategy, a population of current solutions p that was used to generate offspring solutions o is combined, generating \(p \cup o\). so, a replacement strategy (rs) to generate a new population of solutions \(p'\) by \(p' \leftarrow rs(p \cup o)\). the non-dominated sorting genetic algorithm-ii [9] (nsga-ii) performs the replacement strategy considering pareto dominance and crowding distance selection. this selection evaluates how close solutions are to their neighbors, giving a better evaluation of large values allows a better diversity in the population. thus, nsga-ii selects surviving solutions from \(p \cup o\), it first takes non-dominated solutions to compose \(p'\), and if the set size is smaller than the maximum population size, it iteratively adds dominated solutions with higher crowding distance values until \(p'\) is complete. if the set size exceeds the maximum population size, nsga-ii discards the solutions with lower crowding values until \(p'\) is complete. the adaptive geometry estimation based moea (agemoea) [18] inherits the overall framework of nsga-ii. however, it replaces the crowding distance with a survival score that combines diversity and proximity of the non-dominated fronts. first, this algorithm uses a fast heuristic with \(o(m \times n)\) computational complexity to estimate the geometry of the front in each generation. then, proximity is computed as the distance between each population member and the ideal point, while the diversity is measured using the distance among population members. the distance used to calculate proximity and diversity corresponds to the lp norm that is associated with the estimated geometry. a second version of the age algorithm (agemoea2) [19] incorporated the newton-raphson iterative method and the geodesic distance into the agemoea framework. this was performed by proposing a novel method for non-dominated front modeling using the newton-raphson iterative method. second, they compute the distance (diversity) between each pair of non-dominated solutions using geodesics, which are generalizations of the distance on riemann manifolds (curved topological spaces). in this paper, we considered nsga-ii because this algorithm is probably the most popular moea in the literature and has had good results in recent works on cloud scheduling. agemoea2 is one state-of-the-art algorithm for continuous multi-objective optimization benchmarks and was never applied to the cloud scheduling problem. 3 proposal figure 1 presents the heact workflow. the workflow to be scheduled is sent to the data preparation process. this process is also responsible for getting real data about ec2 instance types from aws (using the boto3 library). then, with that information, for each region, the data preparation calculates how long each task \(t \in t\) demands to run in each machine. this creates the table w with size \(nr \times np \times nt\), where nr is the number of regions, np is the number of machines (this number varies in each region), and nt is the number of tasks in t. to create w, we considered two pieces of information about ec2 instances taken from boto3: (1) the network performance and (2) the estimated compute unit (ecu) of the instance type. according to aws, “ecu provides the equivalent cpu capacity of a 1.0–1.2 ghz 2007 opteron or 2007 xeon processor” [22]. in order to use simulation of distributed computer systems (simgrid) [6] as the simulator, we follow [14] and convert ecu to gigaflops using the formula \(gigaflops=ecu*4.2\). once w is created, the ‘data preparation’ process disseminates this information to nr distinct processes, each tasked with finding solutions for a specific region. this is where the moea and a specialist scheduling heuristic come into play, running in parallel. moeas (e.g. nsga-ii) are responsible for defining the set of machines to be used and heuristics (e.g. heft) are responsible for assigning tasks to the selected machine. after completion, each process sends its final population to the ‘pareto dominance’ analysis, eliminating dominated and repeated (same decision variables) solutions. the resulting solution is then stored as a json file. fig. 1. heact architectural representation full size image an essential part of an moea is how chromosomes are designed. in our approach, we used a fixed-size length integer array. each array element represents an instance type, numbered from zero to np. when a number is negative, it means that this slot will not be used. that is a logical way to have a variable-size array. this is important because it allows us to use more straightforward crossover operators in the moea. figure 2 presents two examples of how the solution is defined. fig. 2. moea chromosomes definition illustration full size image after generating a solution through crossover and mutation, the partial solution is sent to a specialized heuristic, which will be responsible for allocating tasks in machines specified by the previous partial solution. thus, every task \(t \in t\) must be assigned to a machine considering the performance characteristics of each ec2 instance type, such as network performance and how long it takes to solve a task (available in table w). tables 1 and 2 show an example of eight tasks being assigned to the solutions represented in fig. 2. notice that not every machine will be used; this is something particular to the heuristic because it considers the cost of transferring gigabytes of data through the network or just waiting for a dependency task at the same machine to finish and having a network transfer cost of zero. table 1. solution #1full size table table 2. solution #2full size table finally, the solution is updated by taking all the scheduling information (which task will run in each machine) and which machine will, in fact, be used. thus, the solution [0,3,3,−1,4,−1,−1,−1,−1,−1] ([0,3,3,4]) becomes [0,3,−1,−1,−1,−1,−1,−1,−1,−1] ([0,3]) and [5,12,3,−1,13,−1,4,0,−1,−1] ([5,12,3,13,4,0]) becomes [5,12,−1,−1,−1,−1,4,−1,−1,−1] ([5,12,4]). with the complete solution, scheduling + machine types, we are able to calculate the objectives makespan and cost. 4 experiments we run experiments to evaluate how heact performs. for this purpose, we considered several real-world workflows, such as montage, cybershake, epigenomics, inspiral, and sipht [3]. we named the problems in the format problem_nodes. so, montage_25 is an instance of montage, which has 25 tasks. figure 3 shows the different workflows. fig. 3. real-world workflows used to evaluate heact full size image we created eight different instances by combining the meta-heuristics nsga-ii and agemoea2 with scheduling heuristics heft, peft, hsip, and mpeft. the population size was set as 50. the number of generations was set to 700, totaling 35000 heuristic execution and solution evaluation, defined empirically. we considered 14 aws regions. these are the regions that have ec2 instances with complete data about ecu. each aws region was assigned to a process that would run for 50 generations (or 2500 solution evaluations). for nsga-ii and agemoea2, we considered a two-point crossover with a probability of 90% and flip mutation with a probability of 10%. all algorithms were coded using python 3.11. we used pymoo [4] for meta-heuristics, pysimgrid [23] and simgrid 3.24 for the cloud simulation, and aws boto3 1.28 to get aws pricing data. after running each of the eight different algorithm instances twenty times (for each problem), we took the resulting population and removed all the dominated solutions. after that, as commonly performed in the literature, we calculated the normalized hypervolume [28] to get the pareto front’s quality for each execution. after getting twenty hypervolume values (for each problem and each algorithm instance), we calculated the average and submitted them to the kruskal-wallis statistical test with 99% confidence, as commonly performed in the literature. table 3 presents hypervolume results; highlighted values mean the highest hypervolume average, while bold values mean that the value is statistically tied with the best value. ns means nsga-ii while age means agemoea2. we can see that no single algorithm statistically overcomes all the others. we can also see that instances that use peft as a heuristic have, in general, the poorest performance. however, regarding cybershake_100, age+peft and ns+peft were indeed the best-performing algorithms. ns+hsip have similar performance compared with peft instances by being the best at one problem and have statistically tied results with the best algorithm in one problem. age+hsip is a bit better because it has the best average at one problem, but it is tied with the best in five problems and is overcome in nine problems. table 3. hypervolume averagesfull size table based on the statistical results, we sum how many problems each algorithm instance got the best hypervolume average, tied with the best, and got the averages statistically worse when compared with the best algorithm. table 4 presents this analysis. moreover, we follow [10] and perform the ranking of algorithms considering the friedman ranking. according to friedman ranking, age+mpeft is the best algorithm. this happens especially because it is the algorithm that has been the best more times. the ranking states that ns+heft, age+heft, and ns+mpeft are in sequence. however, an interesting aspect is that ns+heft is the fourth algorithm in the ranking, but this is the algorithm with fewer overcomes (qtd worse column). by this analysis, we can conclude that age+mpeft and age+heft are the most suitable algorithms in general. table 4. average rankings of the algorithms (friedman)full size table another interesting aspect to evaluate is how solutions are spreading across the world. due to this fact, we looked for solutions for each region independently and then selected the surviving ones using the pareto non-dominance criterion. one region may have more solutions than another. figure 4 presents this percentual. we can see that always us-east-2 (us east (ohio)) have more solutions. the second best is us-west-2 (us west (oregon)) followed by ap-south-1 (asia pacific (mumbai)) and us-east-1 (us east (virginia)). of course, these values can change depending on the price aws assigns to instances. fig. 4. percentual of non-dominated solutions considering regions full size image 5 conclusions and further work planning the execution of a workflow is a challenging task. over the years, several approaches have been proposed in order to tackle this problem. this present paper introduced heact, a novel approach that finds solutions for workflow scheduling by first running a multi-objective evolutionary algorithm in order to create the set of machines and then submitting this setup to a specialist heuristic responsible for the task assigning. this process was performed considering different regions, each one with its own set of instance types. our proposal considered accurate pricing information from aws and simulated the workflow execution considering the simgrid simulator. moreover, fourteen aws regions were considered. we instanced our approach by considering heuristics, such as heft, peft, hsip, and mpeft, as well as meta-heuristic nsga-ii and agemoea2. results showed that agemoea2 + mpeft and agemoea2 +heft were the best combinations. thus, agemoea2, a recent state-of-art meta-heuristic, seems promising for this kind of problem. moreover, mpeft, a recent work, seems a really interesting heuristic for scientific workload problems, as also verified in [20]. further work may focus on combining these heuristics and meta-heuristics to create a hyper-heuristic aiming for the best overall performance. this is important because no single algorithm was the best across all the problems. moreover, information about carbon emissions should also be considered in future work. notes 1.aws elastic compute 2, the virtual machine product of aws. references ahmad, d., gautam, g., alam, b., bhati, b.: an analytical review and performance measures of state-of-art scheduling algorithms in heterogenous computing enviornment. arch. comput. meth. eng. (2024) google scholar  arabnejad, h., barbosa, j.g.: list scheduling algorithm for heterogeneous systems by an optimistic cost table. ieee trans. parallel distrib. syst. 25(3), 682–694 (2014) article  math  google scholar  bharathi, s., chervenak, a., deelman, e., mehta, g., su, m.h., vahi, k.: characterization of scientific workflows. in: 2008 3rd workshop on workflows in support of large-scale science, pp. 1–10 (2008) google scholar  blank, j., deb, k.: pymoo: multi-objective optimization in python. ieee access 8, 89497–89509 (2020) article  math  google scholar  boussaid, i., lepagnot, j., siarry, p.: a survey on optimization metaheuristics. inf. sci. 237, 82–117 (2013) article  mathscinet  math  google scholar  casanova, h., giersch, a., legrand, a., quinson, m., suter, f.: versatile, scalable, and accurate simulation of distributed applications and platforms. j. parallel distrib. comput. 74(10), 2899–2917 (2014) article  math  google scholar  chen, j., menzies, t.: riot: a stochastic-based method for workflow scheduling in the cloud. in: ieee international conference on cloud computing, cloud 2018-july, pp. 318–325 (2018) google scholar  coello, c.: evolutionary algorithms for solving multi-objective problems. springer, new york (2007). https://doi.org/10.1007/978-0-387-36797-2 book  math  google scholar  deb, k., pratap, a., agarwal, s., meyarivan, t.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 6(2), 182–197 (2002) article  math  google scholar  derrac, j., garcía, s., molina, d., herrera, f.: a practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. swarm evol. comput. 1(1), 3–18 (2011) article  math  google scholar  durillo, j.j., fard, h.m., prodan, r.: moheft: a multi-objective list-based method for workflow scheduling. in: 4th ieee international conference on cloud computing technology and science proceedings, pp. 185–192 (2012) google scholar  gendreau, m., potvin, j.y.: handbook of metaheuristics, 2nd edn. springer, heidelberg (2010) book  math  google scholar  hosseini shirvani, m., noorian talouki, r.: bi-objective scheduling algorithm for scientific workflows on cloud computing platform with makespan and monetary cost minimization approach. complex intell. syst. 8 (2021) google scholar  iosup, a., ostermann, s., yigitbasi, m.n., prodan, r., fahringer, t., epema, d.: performance analysis of cloud computing services for many-tasks scientific computing. ieee trans. parallel distrib. syst. 22(6), 931–945 (2011) article  math  google scholar  keshanchi, b., souri, a., navimipour, n.j.: an improved genetic algorithm for task scheduling in the cloud environments using the priority queues: formal verification, simulation, and statistical testing. j. syst. softw. 124, 1–21 (2017) article  google scholar  moscato, p.: on evolution, search, optimization, genetic algorithms and martial arts: towards memetic algorithms. technical report 826, caltech concurrent computation program, c3p (1989) google scholar  nasonov, d., visheratin, a., butakov, n., shindyapina, n., melnik, m., boukhanovsky, a.: hybrid evolutionary workflow scheduling algorithm for dynamic heterogeneous distributed computational environment. j. appl. logic 24, 50–61 (2017) article  mathscinet  google scholar  panichella, a.: an adaptive evolutionary algorithm based on non-euclidean geometry for many-objective optimization. in: proceedings of the genetic and evolutionary computation conference, gecco 2019, pp. 595–603. acm, new york (2019) google scholar  panichella, a.: an improved pareto front modeling algorithm for large-scale many-objective optimization. in: proceedings of the genetic and evolutionary computation conference, gecco 2022, pp. 565–573. acm, new york, ny, usa (2022) google scholar  pu, j., meng, q., chen, y., sheng, h.: mpeft: a novel task scheduling method for workflows. front. environ. sci. 10 (2023). https://doi.org/10.3389/fenvs.2022.996483 qin, s., pi, d., shao, z., xu, y., chen, y.: reliability-aware multi-objective memetic algorithm for workflow scheduling problem in multi-cloud system. ieee trans. parallel distrib. syst. 34(4), 1343–1361 (2023) article  math  google scholar  amazon web services: amazon ec2 instance-types (2024). https://www.amazonaws.cn/en/ec2/instance-types/. accessed 21 may 2024 sukhoroslov, o.: toward efficient execution of data-intensive workflows. j. supercomput. 77(8), 7989–8012 (2021) article  math  google scholar  sulaiman, m., halim, z., lebbah, m., waqas, m., tu, s.: an evolutionary computing-based efficient hybrid task scheduling approach for heterogeneous computing environment. j. grid comput. 19 (2021) google scholar  sulaiman, m., halim, z., waqas, m.m., aydm, d.: a hybrid list-based task scheduling scheme for heterogeneous computing. j. supercomput. 77, 10252–10288 (2021) article  math  google scholar  topcuoglu, h., hariri, s., wu, m.y.: performance-effective and low-complexity task scheduling for heterogeneous computing. ieee trans. parallel distrib. syst. 13, 260–274 (2002) article  math  google scholar  wang, g., wang, y., liu, h., guo, h.: hsip: a novel task scheduling algorithm for heterogeneous computing. sci. program. 2016, 1–11 (2016) math  google scholar  while, l., bradstreet, l., barone, l.: a fast way of calculating exact hypervolumes. ieee trans. evol. comput. 16(1), 86–95 (2012) article  math  google scholar  zhang, q., liu, w., li, h.: the performance of a new version of moea/d on cec09 unconstrained mop test instances. technical report, ces-491, school of cs & ee, university of essex, february 2009 google scholar  zhu, z., zhang, g., li, m., liu, x.: evolutionary multi-objective workflow scheduling in cloud. ieee trans. parallel distrib. syst. 27(5), 1344–1357 (2016) article  math  google scholar  zitzler, e., laumanns, m., thiele, l.: spea2: improving the strength pareto evolutionary algorithm for multiobjective optimization. in: evolutionary methods for design optimization and control with applications to industrial problems. pp. 95–100. international center for numerical methods in engineering, barcelona, spain (2001) google scholar  download references acknowledgments we gratefully acknowledge financial support from the são paulo research foundation (fapesp) grant 2023/15949-8. author information authors and affiliations escola politécnica, universidade de são paulo (usp), são paulo, brazil vinicius renan de carvalho & jaime simão sichman authors vinicius renan de carvalhoview author publications search author on:pubmed google scholar jaime simão sichmanview author publications search author on:pubmed google scholar corresponding author correspondence to vinicius renan de carvalho . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de carvalho, v.r., sichman, j.s. (2025). heact: hybrid evolutionary algorithm for the multi-region multi-objective cloud task scheduling problem. a study of workflow scheduling in aws ec2. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_4 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords cloud scheduling multi-objective optimization evolutionary algorithms scheduling heuristics publish with us policies and ethics profiles vinicius renan de carvalho view author profile jaime simão sichman view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature classification of non-alcoholic fatty liver disease in thermal images of the liver using a siamese neural network | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper classification of non-alcoholic fatty liver disease in thermal images of the liver using a siamese neural network conference paper first online: 30 january 2025 pp 260–269 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) classification of non-alcoholic fatty liver disease in thermal images of the liver using a siamese neural network download book pdf download book epub maxwell pires silva  orcid: orcid.org/0009-0004-3237-52349, aristófanes corrêa silva  orcid: orcid.org/0000-0003-0423-25149 & anselmo cardoso de paiva  orcid: orcid.org/0000-0003-4921-06269  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 476 accesses abstract non-alcoholic fatty liver disease (nafld) is a prevalent and severe condition that requires effective and non-invasive diagnostic methods. this work presents an innovative approach using siamese neural networks to classify thermal images of the liver in order to identify the presence of nafld. the research is motivated by the need to use artificial intelligence to analyze thermal images, especially when the number of images is limited, making it difficult for ordinary neural networks to learn, a difficulty that the siamese network already faces with ease. the proposed method involves three stages: extraction of the region of interest (roi), image pre-processing and classification using the siamese neural network, which compares pairs of images to determine their similarity and, consequently, the presence or absence of nafld. the preliminary results, with an accuracy of 71%, precision of 57% and recall of 96%, indicate that this approach could offer a promising tool for the non-invasive diagnosis of nafld, contributing a promising method to the field of machine learning applied to medicine. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others improving nonalcoholic fatty liver disease classification performance with latent diffusion models article open access 07 december 2023 rule extraction for fatty liver detection using neural networks article 28 june 2017 automated thermal imaging for the detection of fatty liver disease article open access 23 september 2020 1 introduction non-alcoholic fatty liver disease (nafld) is increasingly diagnosed worldwide. it is the most common cause of abnormal liver function tests and chronic liver disease in both developed and developing countries [1]. nafld refers to fat accumulation, mainly triglycerides, in hepatocytes so that it exceeds 5% of the liver’s weight. primary nafld results from insulin resistance and often occurs as part of the metabolic changes accompanying obesity, type 2 diabetes and dyslipidemia. the histological damage in nafld is very similar to that seen in patients with alcoholic liver disease, but nafld is not alcohol-induced by definition [2]. so, it is important to exclude secondary causes of steatosis. furthermore, nafld affects approximately 1.5 billion people worldwide, making it the most common liver disease [3]. given the seriousness of this context, the diagnosis of nafld must be rapid, accurate and efficient. currently, liver biopsy is the most specific method for diagnosing nafld, and it also makes it possible to assess the severity of fatty infiltration in the liver. however, this method is an invasive procedure that poses a risk to the patient and is expensive. therefore, alternative pathology detection methods are desirable in this context [4]. most patients are asymptomatic, and the diagnosis is considered suspicious of nafld after finding elevated transaminases in routine tests. this disease is also a frequent incidental finding on ultrasound (us) performed for other reasons, such as suspected gallstones [5]. hepatic us is currently the most widely available, simple and inexpensive non-invasive method for detecting hepatic steatosis in clinical practice. however, the accuracy of us is highly operator-dependent, and its sensitivity is reduced when steatosis infiltration is less than 30% or in morbidly obese patients. in addition, the quantification of hepatic steatosis is subjective and can be influenced by the heterogeneity observed in some patients with nafld [6]. on the other hand, thermographic studies have been widely included in medical practice to obtain additional data for the diagnosis of various diseases and determine the best methods and the effectiveness of treatment. the main advantages of thermographic research are the relatively low cost, absence of ionizing radiation or electromagnetic fields, absence of contraindications, safety, and the ability to diagnose the disease at an early stage [7]. thus, thermography in medicine can be used to identify the dynamics of pathological processes. specifically in patients with nafld, it is observed that the development of the disease causes a decrease in the surface temperature of the liver [8], making it a promising method for diagnosis. even so, the capture quality of the infrared emission that thermal cameras use to generate surface thermograms is influenced by the environment, which attenuates the thermographic effect, making its detection difficult. consequently, a controlled environment is required for image acquisition and pre-processing to reduce noise and refine image characteristics [9]. in this context, few studies have been carried out using medical thermal images in conjunction with neural networks for detection or classification. pinto et al. (2021) and farias et al. (2023) developed a study to classify thermal images of the liver into two groups: healthy and nafld. the first used the alexnet convolutional neural network with satisfactory results, achieving an accuracy of 96%. the second study used the temporal analysis technique with a temporal convolutional network, achieving an accuracy of 88%. both prove the great potential that the analysis of medical thermal images has if combined with machine learning, enabling professionals to be supported by neural networks during the diagnosis of this disease. in machine learning, convolutional neural networks (cnns) are important architectures that enable the resolution of some problems, such as facial recognition, autonomous vehicles, and intelligent medical treatment [12]. furthermore, in deep learning, cnns are widely known for their ability to achieve high accuracy in classifying medical images. however, these models that are not pre-trained, i.e., with the initial weights of the network starting with random values, have some limitations. training the deep learning model on a large number of images requires huge computational resources, and for proper training of the model, it needs a very significant amount of standard training datasets, which is the biggest problem for medical images; after all, data can be expensive and difficult to obtain. furthermore, there are still ethical privacy issues [13]. with this perspective, siamese neural networks (snns) emerge as an architecture of artificial neural networks composed of two identical neural networks (they share the same weights), united using perceptrons whose function is to calculate the similarity between the response of the two networks [14]. they were proposed to attenuate the data scarcity problem since this type of architecture requires small samples for training and learning the model. therefore, considering the importance of a non-invasive diagnosis of nafld together with the contribution that machine learning is capable of bringing to the classification of images within the scope of computer vision, the present work aims to classify thermal images of the liver region as affected by nafld or healthy, using a siamese neural network architecture. this work makes the following contributions: a study for the processing of medical thermal images, a study to detail the structure of the siamese network and the application of the siamese neural network in the classification of medical thermal images. this study is structured as follows: sect. 2 presents the proposed method and the image dataset used. section 3 presents the result and discusses what was obtained. finally, sect. 4 presents the conclusions of this work. 2 materials and method this section presents the image dataset used and the proposed method for classifying thermal images of the liver (fig. 1). the method consists of three steps: the first is the extraction of the region of interest (roi), followed by the pre-processing of the roi and, finally, the classification of the images by the siamese network. below, each step is detailed. fig. 1. proposed method full size image 2.1 dataset the dataset comprises images from 40 patients, 18 diagnosed with nafld and 22 healthy. in total, there are 103 images of patients with this disease and 132 images of healthy patients. the diagnosis of these patients for nafld was confirmed via ultrasound of the abdominal region by the specialist, serving to determine and label the patients as affected or not by nafld. a thermograph with an infrared sensor flir model s650c, which has a resolution of 640\(\,\times \,\)480 pixels, has been used to acquire the images. as mentioned, a controlled environment is required, so the specialist has developed a thorough protocol for acquiring the thermal images, which consists of certain steps: initially, the environment is acclimatized, where the patient is kept for 15 min at 23\(^\circ \)c and a relative humidity of 65%. at this location, the patients were positioned in dorsal decubitus (lying with their chests up) and the images were then taken with the camera above the patient, forming a \(90^\circ \) angle (fig.  2). five images were captured per patient. in addition, throughout the process of acclimatization and taking the images, the patients had their abdominal region exposed. fig. 2. examples of thermal images obtained from the dataset. full size image 2.2 extraction of the liver region (roi) in this study, the region of interest (roi) comprises the location of the liver in the acquired images. ozougwu (2017) defines the liver region as the upper right quadrant of the abdomen. the roi extraction process was carried out using the proprietary software flir toolsfootnote 1, provided by the manufacturer of the thermograph used to acquire the images. then, with the help of the specialist, rois of different dimensions were obtained from the dataset to capture the liver region better since each patient has a different physiology. the roi was exported to a temperature table in degrees celsius, where each cell shows a value for the temperature at each specific point. 2.3 pre-processing once the roi had been delineated, its temperature table was transformed into thermal images, which use a gray scale for their representation (fig 3(a)). the process described in eq. 1 was used to map the temperature values read t to other desired domains f(t). $$\begin{aligned} f(t) = \frac{(t*1000 oldmin)(newmax newmin)}{(oldmax oldmin)} + newmin. \end{aligned}$$ (1) in eq. 1, oldmin represents the lowest temperature among all the images in the dataset, multiplied by 1000, while oldmax is the highest temperature among all the images multiplied by 1000. newmin and newmax are constant values that represent the minimum and maximum values of the scale adopted (0 and 255 for gray). in this way, all the base images were converted into intermediate thermal images. fig. 3. roi pre-processing steps full size image filters were applied to the images to enhance them. the median filter and adaptive histogram equalization (clahe) [16] (fig 3(b)) were used and demonstrated a significant improvement in network performance in preliminary studies [10, 11]. finally, to solve the problem of the different roi dimensions obtained for each patient, all the roi images were centrally cropped to 200 by 200 pixels. 2.4 classification siamese networks have a high potential to learn, even from a small set of data, and require relative computing power compared to what they can deliver. with this in mind, a siamese neural network architecture was developed for this context. this architecture works with two input images and a single output, which provides a number of how similar these two images are by means of a similarity function. the architecture is composed of a pair of convolutional networks that share the same weights and are responsible for extracting features from the input image pair. the internal structure of the pair of convolutional networks comprises four convolutional layers, with kernel size of 2 by 2 pixels, each containing a maxpooling layer and a dropout layer. the two outputs of the convolutional networks are processed by the euclidean distance layer, converging their result to a single neuron, activated by a sigmoid function, therefore, this last neuron is also trained, generating the final result of the similarity of the two images (fig.  4). thus, the closer the value is to 1.0, the network identifies that the two input images are similar, i.e. they should belong to the same class; otherwise, for values close to 0.0, the network identifies that the input images are not very similar, i.e. they should belong to different classes. fig. 4. siamese neural network architecture full size image 2.5 evaluation metrics as a result of the classification, there are four possibilities: true positives (tp), which are patients with nafld who have been correctly classified as sick, and true negatives (tn), which are healthy patients who have been correctly classified as healthy. there are also false positives (fp), which are healthy patients who were wrongly classified as having nafld, and false negatives (fn), which are patients with nafld who were wrongly classified as healthy. based on these values, it was possible to calculate the metrics chosen to evaluate the network: accuracy, precision and recall. each metric is calculated according to the eqs. 2, 3, and 4 below. $$\begin{aligned} precision = \frac{tp}{tp+fp}, \end{aligned}$$ (2) $$\begin{aligned} recall = \frac{tp}{tp+fn}, \end{aligned}$$ (3) $$\begin{aligned} accuracy = \frac{tp+tn}{tp+tn+fp+fn}. \end{aligned}$$ (4) in medical applications, one of the most important metrics is the recall metric since the most significant risk to the patient is that they have the disease, and the model classifies them as healthy. recall, therefore, provides the model’s effectiveness in detecting the disease where it exists. precision calculates how effective the model is at detecting healthy patients where there are healthy ones. finally, accuracy reveals how effective the model is at getting its classifications right, whether positive or negative. 3 results and discussion this section presents the experimental setup and the analysis of the results achieved at each stage of the proposed method. 3.1 experimental setup the dataset was divided into three different sets: train, test, and validation. the images were separated per patient so that each set was as balanced as possible since the number of images per patient was different. therefore, the final training set had 126 (66 healthy and 60 nafld) images, the test set had 89 (56 healthy and 33 nafld) images, and the validation set had 20 (10 healthy and 10 nafld) images. there was a careful separation so that the images of the same patient belong to a single set. the training was designed as follows: as the siamese network receives pairs of images as input, two image pairs were formed for each image in the training set, the first with a positive example (same class) and the other with a negative example (different classes), so that the image completing the pair was chosen at random. as a result, the network could have one positive and one negative example for each image, and a total of 252 image pairs could be provided. the network’s hyperparameters were batch size 10, adam optimizer with a learning rate of 0.001, dropout layers of 0.3 and a binary cross-entropy chosen as the loss function since the problem is one of binary classification (similar or not similar). data augmentation was not used, as the aim was to work with the original data set, even with a few examples, and observe the behaviour of the siamese network, which was proposed precisely for this reason. training was carried out over 200 epochs on the google colab platform in conjunction with the keras library. 3.2 image classification results the test stage was designed specifically for the context of this work. the siamese network receives two images as input, but in a real scenario, the specialist treats each image individually, using the neural network to classify the patient’s image as healthy or nafld. given this, it was necessary to devise a test strategy to meet this context. the strategy adopted was to find the two best training images, one of a healthy patient and the other of a patient with nafld. this choice was made by analyzing the output of the network that had already been trained. for all the training pairs, among the pairs of the same class, the healthy pair and the nafld pair with the best classification values were extracted, and at the end of the process, the first image from each pair was chosen. in this way, each image in the test set could be tested at the same time with the most representative images of the healthy and nafld classes. the result with the highest network response, as a similarity function, will be the final classification of the image tested. the results of the evaluation metrics obtained are shown in table  3.2, along with the results of the related works carried out by [10, 11]. table 1. results obtained by the proposed method and comparison with related worksfull size table compared to the few studies already carried out in the area [10, 11], the siamese network, despite being an early-stage study, shows promising results. recall, as the most relevant metric, reached 96%, demonstrating that the model is effective at detecting sick people among all the sick people. this represents a great improvement compared to the work of pinto et al. (2021), and very close to what was achieved by farias et al. (2024). the siamese network has an advantage in that it does not require the use of data augmentation and has a much simpler architecture than those used in the related works. this means less computing power for the training and testing phase, and less time is required for pre-processing the images. accuracy was the lowest of the three metrics, but this is to be expected for a work still in progress. the lowest metric recorded was accuracy, meaning that the model still classifies many healthy people as sick, increasing the number of revisions to be carried out by the specialist assisted by the network. analyzing the architectures of the networks compared, it is clear that the convolutional part of the proposed network, used for feature extraction, is much simpler than the others, which may cause low accuracy and low precision. the network learned better the characteristics of the images of sick patients than the images of healthy patients, revealing a point of attention in the images of this group in the used dataset. figure  5 shows an example of each network classification, where (a) is a true positive, (b) is a true negative, (c) is a false positive and (d) is a false negative. fig. 5. images classified by the proposed method full size image the results obtained show that the potential of the siamese network is notorious. it represents a promising step in the classification of thermal images for non-alcoholic hepatic steatosis precisely because of the low quantity of images available for training the network, a situation in which siamese networks acquire their greatest value. 4 conclusion this study presented an approach using siamese neural networks (snn) to classify non-alcoholic fatty liver disease (nafld). the use of thermal images has shown promise, highlighting the importance of adequate pre-processing to improve image quality and reduce noise. the proposed architecture achieved an accuracy of 71%, precision of 57% and recall of 96%, revealing that snns are a viable alternative for nafld classification, especially in scenarios similar to the context presented where there is a limited amount of training data. for future research, it is suggested to explore additional image enhancement techniques, as well as using the architecture already presented, but with modifications in the feature extraction stage, replacing the convolutional layers with a network specialized in extracting features such as, for example, different versions of efficientnet. notes 1.https://www.flir.eu/browse/professional-tools/thermography-software/. references yalmaz, y.: review article: is non-alcoholic fatty liver disease a spectrum, or are steatosis and non-alcoholic steatohepatitis distinct conditions? aliment. pharmacol. ther. 36(9), 815–823 (2012). https://doi.org/10.1111/apt.12046 article  google scholar  angulo, p.: gi epidemiology: nonalcoholic fatty liver disease. aliment. pharmacol. ther. 25(8), 883–889 (2007). https://doi.org/10.1111/j.1365-2036.2007.03246.x article  google scholar  schwabe, r.f., tabas, i., pajvani, u.b.: mechanisms of fibrosis development in nonalcoholic steatohepatitis. gastroenterology 158(7), 1913–1928 (2020). https://doi.org/10.1053/j.gastro.2019.11.311 article  google scholar  santana, j.t., et al.: perfil metabólico e antropométrico dos pacientes obesos e não obesos portadores de esteatose hepática não alcoólica. revista eletrônica acervo saúde 13(2), e5525 (2021). https://doi.org/10.25248/reas.e5525.2021 article  google scholar  dowman, j.k., tomlinson, j.w., newsome, p.n.: systematic review: the diagnosis and staging of non-alcoholicfatty liver disease and non-alcoholic steatohepatitis. aliment. pharmacol. ther. 33(5), 525–540 (2010) article  google scholar  stern, c., castera, l.: non-invasive diagnosis of hepatic steatosis. hepatol. int. 11(1), 70–78 (2017). https://doi.org/10.1007/s12072-016-9772-z article  math  google scholar  olaru, a.: infrared thermographic evaluation of patients with metastatic vertebral fractures after combined minimal invasive surgical treatment. moldovan med. j. 60, 22–25 (2017) math  google scholar  farooq, m.a., corcoran, p.: infrared imaging for human thermography and breast tumor classification using thermal images. in: 2020 31st irish signals and systems conference (issc), pp 1-6. ieee (2020) google scholar  abdul wahab, a., mohamad salim, m.i., yunus, j., ramlee, m.h.: comparative evaluation of medical thermal image enhancement techniques for breast cancer detection. j. eng. technol. sci. 50(1), 40–52 (2018). https://doi.org/10.5614/j.eng.technol.sci.2018.50.1.3 article  math  google scholar  pinto, d.m., souza, j.c., silva, a.c., de araujo martins filho, h.m., de paiva, a.c., zangaro, r.a.: classificação de esteatose hepática não alcoólica em imagens termicas da região do fígado utilizando redes neurais convolucionais. in: anais do xxi simposio brasileiro de computação aplicada à saúde, pp. 302–312. sbc (2021) google scholar  farias, m., et al.: method for detecting non-alcoholic fatty liver disease in abdominal thermography time series using temporal convolutional networks. in: elsevir to be published google scholar  li, z., liu, f., yang, w., peng, s., zhou, j.: a survey of convolutional neural networks: analysis, applications, and prospects. in: ieee transactions on neural networks and learning systems (2021) google scholar  mehmood, a., maqsood, m., bashir, m., shuyuan, y.: a depp siamese convolution neural network for mult-class classification of alzheimer disease. brain sci. j. (2020) google scholar  bromley, j., guyon, i., lecun, y., et al.: signature verification using a siamese time delay neural network. in: cowan, j.d., tesauro, g., alspector, j. (eds) advances in neural information processing systems, voll 6, [7th nips conference, denver, colorado, usa, 1993]. morgan kaufmann, pp. 737–744 (1993) google scholar  ozougwu, j.c.: physiology of the liver. int. j. res. pharm. biosci., 13–24 (2017) google scholar  zuluaga-gomez, j.., al masry, z.., benaggoune, k.., meraghni, s.., zerhouni, n..: a cnn-based methodology for breast cancer diagnosis using thermal images. comput. methods biomech. biomedical eng. imaging vis. 9(2), 131–145 (2021). https://doi.org/10.1080/21681163.2020.1824685 article  google scholar  download references acknowledgments the authors acknowledge the coordenação de aperfeiçoamento de pessoal de nível superior (capes), brazil finance code 001, conselho nacional de desenvolvimento científico e tecnológico (cnpq), brazil, and fundação de amparo à pesquisa desenvolvimento científico e tecnológico do maranhão (fapema) (brazil), empresa brasileira de serviços hospitalares (ebserh) brazil (grant number 409593/2021-4) for the financial support. author information authors and affiliations núcleo de computação aplicada universidade federal do maranhão (ufma), av. dos portugueses, 1966 vila bacanga, são luís, ma, 65080-805, brazil maxwell pires silva, aristófanes corrêa silva & anselmo cardoso de paiva authors maxwell pires silvaview author publications search author on:pubmed google scholar aristófanes corrêa silvaview author publications search author on:pubmed google scholar anselmo cardoso de paivaview author publications search author on:pubmed google scholar corresponding author correspondence to maxwell pires silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, m.p., silva, a.c., de paiva, a.c. (2025). classification of non-alcoholic fatty liver disease in thermal images of the liver using a siamese neural network. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_18 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords non-alcoholic fatty liver disease thermal imaging siamese neural network classification publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature baswe: balanced accuracy-based sliding window ensemble for classification in imbalanced data streams with concept drift | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper baswe: balanced accuracy-based sliding window ensemble for classification in imbalanced data streams with concept drift conference paper first online: 30 january 2025 pp 231–246 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) baswe: balanced accuracy-based sliding window ensemble for classification in imbalanced data streams with concept drift download book pdf download book epub douglas amorim de oliveira  orcid: orcid.org/0009-0009-3476-32029, karina valdivia delgado  orcid: orcid.org/0000-0002-9120-89879 & marcelo de souza lauretto  orcid: orcid.org/0000-0001-5507-23689  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 507 accesses 1 citation abstract in the wake of the exponential growth in data generation witnessed in recent decades, the binary classification task within data streams presents inherent challenges due to their continuous, real-time flow and dynamic nature. this paper introduces the balanced accuracy-based sliding window ensemble (baswe) algorithm that leverages balanced accuracy, sliding windows, and resampling techniques to effectively handle imbalanced classes and concept drifts, ensuring robust performance even as data patterns evolve. in experiments conducted on 40 datasets, comprising 16 real-world and 24 synthetic datasets generated under three configurations-no drift, gradual drift, and sudden drift-and with varying imbalance ratios, baswe demonstrated superior performance compared to seven other state-of-the-art algorithms in terms of f1 score and the kappa statistic. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-window based ensemble learning for classification of imbalanced streaming data article 08 march 2017 rose: robust online self-adjusting ensemble for continual learning on imbalanced drifting data streams article 20 april 2022 ensemble framework for concept drift detection and class imbalance in data streams article 04 may 2024 1 introduction the rapid increase in data generation has led to the emergence of data streams, characterized by continuous, real-time data flow that challenges the static nature of traditional datasets [2, 11]. in supervised learning tasks, this dynamic environment presents the dual issues of concept drift, also known as data shift [3], where underlying data distributions shift over time, and class imbalance, where one class vastly outnumbers the others. in data streams, this imbalance is not static but can also evolve, intensifying the difficulty of the learning task. the interplay between the dynamic nature of imbalance and the shifting paradigms of concept drift demands a rethinking of traditional approaches to maintain classifier performance [12, 15]. many algorithms have been proposed to address the challenges in data streams classification with imbalanced data, such as calmid [17], csarf [18], rose [8], oob [19], smote-ob [4] and uob [19]. four of these algorithms, calmid, csarf, rose, and smote-ob, implement an explicit approaches to handle concept drift. csarf, rose, oob, and uob implement an ensemble approach to handle imbalanced data. additionally, csarf also implements a cost-sensitive approach. one ensemble algorithm that showed robust results for binary classification in data streams with concept drift is the kappa updated ensemble (kue) [7]. by implementing the very fast decision tree (vfdt) [13] algorithm as a basis for its experts and creating policies for model updates, vote abstention, and model assessment using the kappa statistic, the kue outperformed other state-of-the-art algorithms [7]. however, kue was not designed for data streams with imbalanced classes. this paper proposes an algorithm for binary classification named the balanced accuracy-based sliding window ensemble (baswe), inspired by kue. baswe uses two sliding windows, resampling techniques, and replaces the kappa statistic with the balanced accuracy metric in ensemble updates. these modifications aim to achieve a more robust ensemble with higher performance in the scenario of classification in data streams with imbalanced classes and concept drifts. the contributions of this work can be summarized as follows: introduction of baswe, an ensemble algorithm designed to manage binary classification in data streams with imbalanced classes and concept drift. the adoption of balanced accuracy as a pivotal metric in guiding ensemble updates, encompassing voting strategies, model performance measurement, and expert substitution. the use of class-specific sliding windows and a resampling step in pre-processing. this strategy is tailored to address imbalanced data streams, effectively reducing the imbalance ratio during the model training phase with new data chunks. a comprehensive experimental evaluation comparing baswe against state-of-the-art algorithms. 2 data stream classification data streams can be understood as data instances generated at high speed and arriving continuously, presenting challenges to computational systems for storage and processing [10]. however, if efficiently analyzed, they provide an important source of information for real-time decision-making support. a data stream is characterized as a sequence denoted by \( s = \langle s_{1}, s_{2}, \ldots , s_{n}, \ldots \rangle \), where each \( s_{j} \) represents a collection of instances with a size \( n \ge 1 \). in the particular case where \( n=1 \), the context is referred to as online learning [10], otherwise, it is called learning by chunks. typically, each instance \( s_t \) within each set \( s_j \) is independently and randomly produced following a stationary distribution \( d_{j} \). the data stream classification task aims to predict the correct label \(y_t\), for each incoming instance \(s_t \in s_j\). each instance is described by a set of attributes \(x_t=\{x_{t1}, x_{t2}, ..., x_{tn}\}\). a classifier, represented as f, takes the set of instance attributes \(x_t\) as input and generates the predicted label \(\hat{y}_t\), as output. in our research, we focus on fully labeled binary classification, where the correct label \( y_t \) is restricted to one of two possible outcomes. additionally, every instance \( s_t \) within the data stream is pre-labeled. in several classification problems, the distribution \(d_{j}\) is subject to changes over time, as the characteristics and definitions of the data stream evolve. this phenomenon is called concept drift [12, 20]. concept drifts in the pattern of class distribution can be segmented into 4 categories [12]: sudden, incremental, gradual, or recurrent. in the realm of data streams, concept drift can be addressed either explicitly (using a dedicated drift detector) or implicitly. class imbalance in data stream classification often occurs for various reasons. it can be due to the intrinsic nature of the data scope being processed, such as a medical database that includes sporadic occurrences of a rare disease. alternatively, it could occur purely by chance if the batch of data being processed lacks a representative diversity of instances for each class. to address the scenario of imbalanced classes in machine learning within data streams, three main methods are commonly employed [14]: (i) data-level methods, where the data is pre-processing using, for example, resampling techniques; (ii) algorithmic-level methods where the learning algorithms are adapted to deal with imbalance classes; and (iii) hybrid methods which represent a combination of the previous two. 3 baswe this section introduces the balanced accuracy-based sliding window ensemble (baswe), an algorithm designed specifically for binary classification in imbalanced data streamsfootnote 1. this algorithm is inspired by kue algorithm [6]. baswe has three main characteristics. the first one is the use of dynamic ensemble techniques, which address dynamic substitutions and provide real-time ensemble updates. additionally, baswe implements an abstention policy to address relevance modifications of the experts. the second crucial characteristic is the use of balanced accuracy as the primary metric for model evaluation and updating. this strategy is specifically tailored to address the imbalanced data sets that baswe is designed to handle. the third key feature is the application of sliding windows as a data-level method that can effectively manage class imbalance in the data stream. these latter two characteristics are explained in sects. 3.1 and 3.2, respectively. baswe’s experts in the ensemble implement the very fast decision trees (vfdt) algorithm [13]. this algorithm is recognized for its computational efficiency, making it ideal for handling real-time data streams. it operates by building decision trees incrementally, accommodating massive volumes of data while minimizing memory usage. 3.1 balanced accuracy approach to handle ensemble update balanced accuracy (ba) (eq. 1) provides a robust measure for binary classification problems, especially when dealing with imbalanced datasets. $$\begin{aligned} \text {balanced accuracy} = \frac{1}{2}\left( \frac{tpr}{tpr+fnr}+\frac{tnr}{tnr+fpr}\right) \end{aligned}$$ (1) where tpr is the true positive rate and tnr is the true negative rate. in the context of the baswe algorithm, balanced accuracy is used as a driving mechanism for two key operations: expert substitution within the ensemble. with every new chunk of data, q new experts are trained outside the ensemble. the ensemble itself consists of k experts \((\gamma _1,...,\gamma _{k})\), and if the balanced accuracy of the newly trained experts exceeds that of the worst-performing ones in the ensemble, a substitution takes place. abstention of voting from ensemble experts. the abstention policy operates by controlling the voting rights of the ensemble experts based on their balanced accuracy. when the ensemble computes the majority vote to determine the final classification, the ba of each expert is assessed against a specific threshold (0.5 in our implementation). any expert with a ba below this threshold is prevented from voting, meaning their classification decision does not influence the ensemble’s final output. this abstention policy ensures that the ensemble’s decision-making process is not adversely affected by the subpar performance of any of its experts, thereby maintaining the integrity and accuracy of the ensemble’s output. 3.2 sliding window approach for handling imbalanced data streams in baswe, the sliding window approach is adopted to manage a sort of data cache, aiming to decrease the disproportion among the classes over time. this methodology contributes to handling the data imbalance and providing a more accurate and representative sample of the current data for model training. to implement this approach, the algorithm establishes two sliding windows, \(w_0\) and \(w_1\), each corresponding to one class. the size of each sliding window is set to be half of the chunk size that is used for model processing. as each new chunk of data arrives from the data stream, the instances are added to the corresponding sliding window. if a sliding window is full, meaning the number of instances equals the predefined size, the oldest instance is discarded to make room for the newest one. this policy of expelling older instances helps accommodate concept drift, as instances naturally exit the cache over time, thereby reflecting the changing nature of the data stream. algorithm 1 shows the fillwindow function. in this function, the algorithm takes as input the current sliding window, the target class covered by this sliding window (desiredclass), the current chunk of data (\(s_i\)), and the window max size (cs). the chunk is traversed searching for instances of the desired class (lines 2-3), and upon finding an instance, it is added to the sliding window (line 4). after adding a new instance to the sliding window, it checks whether its size has exceeded cs, and if so, removes the oldest instance from the sliding window (lines 5-7). in the end, the function returns the window vector incremented by the new instances of the desiredclass contained in the i-th chunk. algorithm 1 fillwindow function full size image when the baswe model is trained, if any of the sliding windows is not fully populated, oversampling is performed. the oversampling function iterates over the vector from the most recent to the oldest instance, replicating instances until the number of instances required to complete the chunk size is achieved. in this way, the sliding window approach within the baswe algorithm not only ensures an up-to-date representation of the data for model training but also plays an instrumental role in managing class imbalance and concept drift. 3.3 baswe’s pseudocode baswe’s pseudocode is delineated in algorithm 2, and is segmented into four stages: ensemble initialization, ensemble experts update, training of the new \( q \) components, and replacement of the weakest \( \gamma \) expert in the ensemble \( \epsilon \). algorithm 2 baswe: balanced accuracy-based sliding window ensemble full size image the main loop (lines 3-34) represents the algorithm over the chunks that arrive throughout the data stream at each iteration. in lines \(4-5\), two sliding windows \(w_0\) and \(w_1\) are filled, each for one class. oversampling is then performed by the oversampling function in lines \(6-7\), creating \(w'_0\) and \(w'_1\), which are joined to form the chunk \(s'_i\) in line 8. this chunk will be used by the experts in the training phase, while the original chunk \(s_i\) will be used to compute the performance. upon receiving the first chunk, \(s_1\), we have the process of creating and initializing the ensemble of k experts, represented in lines \(9-15\). for each of the \( k \) experts in the ensemble, a random integer number \( r \) is selected (line 11), drawn from uniform probability \([1,f]\). this value, \( r \), determines the size of the feature subspace that will be used for the \( j \)-th expert. in line 12, the choosefeaturessubspace method selects a \( r \)-dimensional random feature subspace for the \( j \)-th expert, with this information stored in the variable \( \varphi _j \), that will be used to generate diversity in the ensemble and avoid noisy data. in line 13, the filterfeaturessubspace method receives two parameters, the \( \varphi _j \) and the chunk \(s_1'\), and then returns the chunk \(s_1'\) filtering only the features contained in the \( \varphi _j \) subspace. the classifier \( \gamma _{j} \) is then initialized, training with the chunk \(s_1'\) filtered by the filterfeaturessubspace method. the balanced accuracy of the \( j \)-th expert is then computed and stored in the variable \(ba_j\) (line 14). the ensemble model update is performed in lines 17-20 considering the subsequent chunks. within this process, the algorithm filters the feature subspace \( \varphi _j \) from chunk \( s'_i \), conducts incremental training of \( \gamma _{j} \) (line 18), and computes the balanced accuracy of this expert (line 19). from lines \(21-26\), \( q \) new experts are trained. if these new experts exhibit superior performance based on balanced accuracy compared to the \( q \) least effective experts in the ensemble, they will replace the latter. the higher the value of \( q \), the more intense the behavioral change of the ensemble tends to be, as more components can be replaced in each data chunk. this presents both an advantage of rapid adaptation and a risk of undesirable abrupt changes caused by anomalous chunks. in line 26, the minbalancedaccuracy method conducts a search for the expert in the ensemble \( \epsilon \) with the lowest performance. the result returned by this method is then stored in \(ba_{min}\), the balanced accuracy of the expert, and the position of the expert within the ensemble, represented by w. this substitution takes place in lines \(27-31\) of algorithm 2, where we compare the balanced accuracy of the new expert, denoted as \( ba' \), to that of the expert with the lowest balanced accuracy, \(ba_{min}\). if the new expert demonstrates superior performance, the expert at position \( w \) (represented by \( \gamma _w \)) is substituted with the new \( \gamma ' \). we also update the feature subspace, \( \varphi _w \), and the balanced accuracy, \( ba_w \), of the weakest expert with \( \varphi ' \) and \( ba' \), respectively. 4 experimental results in this section, we compare baswe with seven state-of-the-art algorithms. 4.1 benchmark algorithms baswe was compared with different ensemble classifiers: kue [7], calmid [17], csarf [18], rose [8], oob [19], uob [19], and smote-ob [4]. calmid, csarf, rose, uob, and smote-ob were selected as the top 5 algorithms by a recent survey on imbalanced data streams [1]. additionally, we included kue, as it served as the inspiration for baswe, and oob, which was proposed by the same authors as uob and demonstrated good performance in other recent works [8]. among the selected algorithms, kue is the only one that was not designed to deal with imbalanced data streams. all those algorithms are ensembles; however, they implement distinct approaches to address the replacement of new base classifiers. kue and calmid algorithms execute, at most, one replacement per new chunk when employed with their default parameters. conversely, baswe, csarf, and rose allow the substitution of multiple experts with each new chunk (or instance, in the case of rose). on the other hand, the oob, smote-ob, and uob algorithms do not replace their base classifiers throughout the data stream. these algorithms are implemented in moa and we use its default parameters, as specified in their respective original studies. kue is the only exception; it has an implementation with \(q = 1\) and an additional implementation that permits the substitution of two experts (referred as kue (q = 2) in this section). in baswe, we make use of the same kue’s default parameters (\(k=10\)), except q, which is equal to 2. baswe has one additional parameter, cs, that represents the sliding window’s cache size, we use the half of chunk size as the value of cs. 4.2 datasets experiments were conducted using 24 binary data stream generators in the moa (massive online analysis) environment [5], featuring varying degrees of data imbalance. of these, 8 data streams have sudden concept drift, 8 data streams have gradual concept drift, and the remaining 8 do not have concept drift. additionally, evaluations were carried out on 16 real-world datasets. all experiments in this study treated the data streams as fully labeled. synthetic datasets: a series of synthetic datasets was generated utilizing the moa environment. the chosen generators for this process were as follows: agrawal, asset negotiation, hyperplane, mixed, random rbf, random tree, sea, and sine. each of these synthetic datasets was synthesized under specified conditions, with a designated random seed value set at 42, a chunk size of 500, and comprising 200,000 instances. all synthetic datasets were designed for binary classification. for a comprehensive analysis, these datasets were crafted with five distinct imbalance ratios: 90:10, 95:5, 97.5:2.5, 99:1, and 99.5:0.5. in these ratios, the value before the colon denotes the proportion of the majority class, while the value after represents the proportion of the minority class. additionally, the synthetic data streams were generated in the two contexts, with and without concept drift. in the data streams with concept drift, we analyzed sudden and gradual concept drift, the two types of concept drift commonest in the studies [1]. real benchmark datasets: sixteen real benchmark datasets were utilized for the experimental analysis: adult, airlines, bridges-1vsall, census, covtypenorm-1-2vsall, credit.g, dermatology, diabetes, electricity, gmsc, mushroom, sick, sonar, vehicle, vote, and vowelfootnote 2. 4.3 metrics the performance of the classifiers was evaluated using the kappa metric, which was used in [1, 7, 8], and the f1-score, which was used in [9, 15, 16]. for each set of experiments, described in sect. 4.4, the following measures are computed: absolute best (ab) that represents the number of experiments in which the algorithm achieved the best absolute metric (f1-score or kappa) value. equivalent best using t-test (eb) that indicates the number of experiments where, despite the algorithm not having achieved the highest metric (f1-score or kappa) value, no significant difference was found (indicating a t-test p-value greater than 0.05) between its metric value and the highest metric value among all other algorithms. total best (tb) that is the sum of the absolute best and equivalent best values, representing the number of experiments where the algorithm demonstrated the best or equivalent to the best metric (f1-score or kappa) value. avg. metric (f1-score or kappa) that represents the average of metric (f1-score or kappa) value from all experiments. the algorithms are ranked by the total best value and, in case of a tie, the avg. metric (f1-score or kappa) is used to rank the algorithms. we prioritize the total best value over avg. metric (f1-score or kappa) since averages can be skewed by the outcomes from specific datasets. in certain scenarios, an algorithm might exhibit a significantly higher or lower avg. metric (f1-score or kappa) due to unique characteristics inherent to a particular dataset. hence, it is possible for an algorithm to rank first in total best while manifesting the least favorable results when evaluated by avg. metric (f1-score or kappa), due to notably poorer performance in a few experiments. 4.4 experiments configuration in this study, we define an experiment as the execution of an algorithm on a data stream. to compare the algorithms, experiments were performed with synthetic and real data streams. for the synthetic data, the number of streams was 40 without concept drift, 40 with gradual concept drift, and 40 with sudden concept drift. this is because there are 8 synthetic data streams generators, and each has one experiment for every ratio of data imbalance: 90:10, 95:5, 97.5:2.5, 99:1, and 99.5:0.5. for the real data streams, the number of experiments was 16 which corresponds to the number of real data streams. since 9 algorithms were evaluated across 136 data streams, there are 1,224 experimental configurations. to mitigate potential impacts of performance anomalies associated with specific chunk characteristics or random initialization of the experts, each experiment was executed 10 times, resulting in 12,240 experimental runs. to calculate the f1-score and kappa values for each algorithm on each data stream, the average value of the metrics was calculated over all the chunks of each run, using the “moa.tasks.evaluateprequential” configuration of moa. each new chunk is processed in two steps: (i) evaluate the current ensemble on the chunk, computing the f1-score and kappa in line 18 of algorithm 2; and (ii) update the ensemble with the new chunk in line 19 of algorithm 2. the time limit for each experiment execution was 60 min. the experiments were conducted on an apple m1 pro cpu with 16 gb of memory, running macos ventura 13.3. 4.5 synthetic imbalanced data streams without concept drift in this section, we address the results obtained from 40 experiments (8 synthetic imbalanced data streams without concept drift and 5 different imbalance rates). baswe achieved the highest total best value for both f1-score (15 of 40 experiments) and kappa (20 of 40 experiments). the second-best total best value for both measures was achieved by the oob ensemble classifier. conversely, the csarf algorithm recorded the lowest total best value, securing the total best f1-score value in just 2 of 40 experiments and kappa in 3 of 40. smote-ob did not finish executing the experiments within the time constraint of 60 min. 4.6 synthetic imbalanced data streams with concept drift in this section, we address the results obtained from 80 experiments (5 different imbalance rates with 8 imbalanced synthetic data streams with sudden concept drift and 8 with gradual concept drift). for experiments with gradual drift, the datasets followed a consistent pattern until the 40,000th instance. from this point, a gradual drift began, continuing over a span of 20,000 instances until the 60,000th instance. for experiments with sudden drift, the datasets maintained a stable pattern until the 40,000th instance, where an immediate and abrupt change occurred. in both scenarios, with gradual and sudden drifts, smote-ob did not finish the experiments’ execution within the time limit constraint (60 min). gradual concept drift. table 1 shows f1-score and kappa for the experiments over the imbalanced data streams with gradual concept drift. baswe emerged as a top-performing algorithm, securing the highest ‘total best’ value in both f1-score (14 out of 40 experiments) and kappa (19 out of 40 experiments). for synthetic imbalanced data streams without concept drift, oob had the highest average metric, whether f1-score or kappa. however, in the scenario of synthetic imbalanced data streams with gradual concept drift, baswe took the lead with the highest average f1-score and rose had the highest average kappa. table 1. f1-score (left) and kappa (right) obtained by the algorithms for the eight synthetic datasets with gradual concept driftfull size table fig. 1. f1-score obtained by the algorithms for the eight synthetic datasets with gradual concept drift full size image figure 1 shows the f1-score achieved by each algorithm under different imbalance ratios. the results of kappa were also plotted but are not shown here. we can see that in several data streams, the algorithms exhibit somewhat similar behavior. however, uob and csarf tend to diverge more from the other algorithms as the imbalance increases. this might suggest that, even though both algorithms are designed to handle imbalanced classes and perform reasonably well in that scope, they are more sensitive to highly imbalanced data contexts, such as ratios of 97.5:2.5 or higher. this behavior is observed for both kappa and f1-score. in the case of uob, which employs undersampling, the reduced performance can be attributed to its aggressive discarding of instances from the majority class at high imbalance ratios, thereby losing valuable information beneficial for learning. sudden concept drift. table 2 shows the compilation of results measured by f1-score and kappa, respectively, for the experiments over the imbalanced data streams with sudden concept drift. the best algorithm in terms of f1-score was calmid, which achieved the ‘total best’ in 16 experiments. in terms of kappa, the top spot went to rose, also with a ‘total best’ value of 16 experiments. both algorithms, calmid and rose, implement explicit techniques to handle concept drift. these performances highlight that, in the specific scenario of sudden concept drift, the presence of dedicated mechanisms for detecting concept drifts makes a significant difference, ensuring the best performance in our experiments. this distinction was not as pronounced in experiments with gradual concept drift, where baswe, which uses an implicit approach to handle concept drift, secured the highest ‘total best’ values, for both f1-score and kappa. table 2. f1-score (left) and kappa (right) obtained by the algorithms for the eight synthetic datasets with sudden concept driftfull size table baswe, while not securing top performance, still showed strong results, tied in the second-highest ‘total best’ value in f1-score (11 out of 40 experiments), and the second in kappa (14 out of 40 experiments). this suggests that, even with its implicit approach to concept drift, baswe exhibited notable robustness within this data scope. plots for f1-score and kappa for different imbalance rates, analogous to those in fig. 1, were also analyzed under the sudden concept drift scenario but are not shown here. it was observed that, similar to the experiments with gradual concept drift, uob deteriorates more quickly as the imbalance increases. csarf, which previously showed rapid deterioration like uob in the gradual concept drift experiments, demonstrated more robustness in the experiments with sudden concept drift. this can be attributed to mechanisms designed to explicitly handle the occurrence of concept drift. 4.7 real datasets in this section, we discuss the results from 16 experiments on real data streams, some of which exhibit concept drifts and feature various imbalance ratios. unlike synthetic data streams, which are generated based on specific probability distributions and are thus well-behaved, real data streams might not follow these distributions. this can introduce unique challenges, such as data chunks that contain instances of only one class. table 3. f1-score (left) and kappa (right) obtained by the algorithms for the sixteen real datasetsfull size table table 3 shows the f1-score and kappa achieved by the algorithms compared across all real datasets. baswe consistently delivered the highest total best value of f1-score, achieving this in 8 of the 16 real datasets. in terms of kappa, there was a tie among csarf, rose, and baswe algorithms, that have total best value equal to 5 experiments each. just as observed in experiments with synthetic data streams, smote-ob exhibited a high computational cost and was unable to complete the execution of experiments within the 60-minute time constraint for 5 of the 16 data sets. table 4. ‘total best’ of f1-score for total experimentsfull size table table 5. ‘total best’ of kappa for total experimentsfull size table 4.8 discussion tables 4 and 5 present the overall performance of the algorithms measured by f1-score and kappa, respectively. experimental results suggest that the baswe is a promising choice for binary classification tasks involving class imbalance with and without concept drift. baswe achieved superior results when measured by f1-score, achieving the best performance or equivalent to the best in 48 out of 136 general experiments: 15 out of 40 in imbalanced data streams without concept drift, 14 out of 40 for the set with gradual concept drift, 11 out of 40 for the set with sudden concept drift, and 8 out of 16 for the real datasets. similarly, it attained superior results when measured by the kappa statistic, achieving the best performance or equivalent to the best in 58 out of 136 general experiments: 20 out of 40 in imbalanced data streams without concept drift, 19 out of 40 for the set with gradual concept drift, 14 out of 40 for the set with sudden concept drift, and 5 out of 16 for the real datasets. the experimental data highlighted in sect. 4.6 suggest that the effectiveness of baswe decreases in the presence of sudden concept drifts. in such cases, the algorithms that recorded the highest ‘total best’ scores for each metric (calmid for f1-score and rose for kappa) utilize drift detection mechanisms, which directly address concept drift. this could suggest that baswe’s implicit method of handling concept drift may be less effective during rapid shifts in data distribution, resulting in a slower adaptation to these changes. 5 conclusion this paper presents baswe, an algorithm tailored specifically for binary classification in imbalanced data streams with and without concept drift. baswe incorporates sliding windows paired with class-specific oversampling. rather than processing all data uniformly, baswe prioritizes the minority class instances, ensuring their proportional representation within the model. class-specific oversampling entails replicating instances of the minority class, effectively countering the prevalent class imbalance. furthermore, baswe leverages balanced accuracy for ensemble updates and vote abstention. this not only ensures that the algorithm remains unbiased towards the majority class, offering a fair evaluation of its performance across both classes, but also measures the quality of experts within the ensemble. by evaluating and subsequently updating or replacing underperforming experts based on balanced accuracy, baswe implicitly addresses concept drift over time. this continuous updating also enhances the diversity within the ensemble, which is further augmented by selecting a random subset of features for each expert in the ensemble. its consistently high performance across a variety of datasets, coupled with its robustness to different data conditions, makes baswe a versatile and reliable choice for handling classification tasks in imbalanced data streams, with or without concept drift. notes 1.code and datasets available on https://github.com/baswe-paper/baswe. 2.code and datasets are available on https://github.com/baswe-paper/baswe. references aguiar, g., krawczyk, b., cano, a.: a survey on learning from imbalanced data streams: taxonomy, challenges, empirical study, and reproducible experimental framework. mach. learn. 113(7), 4165–4243 (2023) article  mathscinet  math  google scholar  bahri, m., bifet, a., gama, j., gomes, h.m., maniu, s.: data stream analysis: foundations, major tasks and tools. wires data mining knowl. discov. 11(3) (2021) google scholar  barrabés, m., mas montserrat, d., geleta, m., giró-i nieto, x., ioannidis, a.: adversarial learning for feature shift detection and correction. in: advances in neural information processing systems, vol. 36 (2024) google scholar  bernardo, a., valle, e.d.: smote-ob: combining smote and online bagging for continuous rebalancing of evolving data streams. in: 2021 ieee international conference on big data, pp. 5033–5042. ieee, orlando, fl, usa (2021) google scholar  bifet, a., holmes, g., pfahringer, b., kranen, p., kremer, h., jansen, t., seidl, t.: moa: massive online analysis, a framework for stream classification and clustering. in: proceedings of the first workshop on applications of pattern analysis, vol. 11, pp. 44–50 (2010) google scholar  cano, a., krawczyk, b.: evolving rule-based classifiers with genetic programming on gpus for drifting data streams. pattern recogn. 87, 248–268 (2019) article  math  google scholar  cano, a., krawczyk, b.: kappa updated ensemble for drifting data stream mining. mach. learn. 109(1), 175–218 (2020) article  mathscinet  math  google scholar  cano, a., krawczyk, b.: rose: robust online self-adjusting ensemble for continual learning on imbalanced drifting data streams. mach. learn. 111(7), 2561–2599 (2022) article  mathscinet  math  google scholar  du, h., palaoag, t.d.: dynamic weighted majority based on over-sampling for imbalanced data streams. in: ciis 2021: the 4th international conference on computational intelligence and intelligent systems, november 20–22, 2021, pp. 87–95. acm, tokyo, japan (2021) google scholar  gaber, m.m.: advances in data stream mining. wires data mining knowl. discov. 2(1), 79–85 (2012) article  math  google scholar  gama, j.: knowledge discovery from data streams. crc press, boca raton, fl, usa, chapman and hall / crc data mining and knowledge discovery series (2010) book  math  google scholar  gama, j., zliobaite, i., bifet, a., pechenizkiy, m., bouchachia, a.: a survey on concept drift adaptation. acm comput. surv. 46(4), 1–37 (2014) article  math  google scholar  hulten, g., spencer, l., domingos, p.m.: mining time-changing data streams. in: proceedings of the seventh acm sigkdd international conference on knowledge discovery and data mining, pp. 97–106. acm, san francisco, ca, usa (2001) google scholar  krawczyk, b.: learning from imbalanced data: open challenges and future directions. prog. artif. intell. 5(4), 221–232 (2016) article  math  google scholar  krawczyk, b., minku, l.l., gama, j., stefanowski, j., wozniak, m.: ensemble learning for data stream analysis: a survey. inf. fusion 37, 132–156 (2017) article  math  google scholar  li, z., huang, w., xiong, y., ren, s., zhu, t.: incremental learning imbalanced data streams with concept drift: the dynamic updated ensemble algorithm. knowl. based syst. 195, 105694 (2020) article  google scholar  liu, w., zhang, h., ding, z., liu, q., zhu, c.: a comprehensive active learning method for multiclass imbalanced data streams with concept drift. knowl. based syst. 215, 106778 (2021) article  math  google scholar  loezer, l., enembreck, f., barddal, j.p., de souza britto jr., a.: cost-sensitive learning for imbalanced data streams. in: the 35th acm/sigapp symposium on applied computing, pp. 498–504. acm, brno, czech republic (2020) google scholar  wang, b., pineau, j.: online bagging and boosting for imbalanced data streams. ieee trans. knowl. data eng. 28(12), 3353–3366 (2016) article  math  google scholar  webb, g.i., hyde, r., cao, h., nguyen, h., petitjean, f.: characterizing concept drift. data min. knowl. discov. 30(4), 964–994 (2016) article  mathscinet  math  google scholar  download references acknowledgments this study was supported by cepid-cemeai-center for mathematical sciences applied to industry (grant 2013/07375-0, são paulo research foundation-fapesp). author information authors and affiliations school of arts, sciences and humanities, university of são paulo, são paulo, brazil douglas amorim de oliveira, karina valdivia delgado & marcelo de souza lauretto authors douglas amorim de oliveiraview author publications search author on:pubmed google scholar karina valdivia delgadoview author publications search author on:pubmed google scholar marcelo de souza laurettoview author publications search author on:pubmed google scholar corresponding author correspondence to douglas amorim de oliveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de oliveira, d.a., delgado, k.v., lauretto, m.d.s. (2025). baswe: balanced accuracy-based sliding window ensemble for classification in imbalanced data streams with concept drift. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_16 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords ensemble concept drift imbalanced datasets publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature time-dependent item embeddings for collaborative filtering | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper time-dependent item embeddings for collaborative filtering conference paper first online: 28 november 2021 pp 309–324 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) time-dependent item embeddings for collaborative filtering download book pdf download book epub pedro r. pires10, amanda c. pascon10 & tiago a. almeida10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1238 accesses 1 citation abstract collaborative filtering recommender systems are essential tools in many modern applications. their main advantage compared with the alternatives is that they require only a matrix of user-item interactions to recommend a subset of relevant items for a given user. however, the increasing volume of the data consumed by these systems may lead to a representation model with very high sparsity and dimensionality. several approaches to overcome this problem have been proposed, neural embeddings being one of the most recent. since then, many recommender systems were made using this representation model, but few consumed temporal information during the learning phase. this study shows how to adapt a pioneering method of item embeddings by adding a sliding window over time, in conjunction with a split in the user’s interaction history. results indicate that considering temporal information when learning neural embeddings for items can significantly improve the quality of the recommendations. this study was financed by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) finance code 88882.426978/2019-01, and fundação de amparo à pesquisa do estado de são paulo (fapesp) grant #2020/09354-3. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others bilateral neural embedding for collaborative filtering-based multimedia recommendation article 25 september 2017 learning continuous user and item representations for neural collaborative filtering chapter © 2019 interact2vec: neural item and user embedding for collaborative filtering chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. data storage representation learning algorithms learning theory machine learning neural encoding neural decoding 1 introduction personalized recommendations have become an increasingly common practice in our digital lives [5]. with the popularization of technology and ease of making information available, users have access to many items such as movies, music, and news, making it challenging for them to discover everything available and explore their interests. recommender systems then emerged in the 1990s to overcome this problem, employing filtering techniques to direct the user to a restricted subset of relevant items [2]. among several types of recommender systems, collaborative filtering is currently the most prolific in scientific literature and practical applications. their popularity can be credited to its ease of application, since it depends only on a matrix of interactions between users and items to generate recommendations based on similar tastes among users [5]. in early studies, the general approach uses item-based neighborhood algorithms for generating the recommendation [7]. however, the representation commonly adopted grows fast in size as the number of users and items increases; in real-world scenarios, these numbers grow constantly and much faster than the number of user-item interactions. consequently, the traditional representation model became highly sparse and with an inflated dimensionality, demanding more and more computational power and reducing the quality of the results [18]. many approaches have been presented to overcome this problem, such as matrix factorization [20] and neural models [36]. in the latter, methods based on neural embeddings have been gaining ground in recent literature. these methods aim to represent items or users as distributed vectors, resulting in a representation with reduced dimensionality while keeping its latent meaning [4, 13]. although many studies have proposed different methods based on neural embeddings [11, 31], most of these studies do not consider the moment when the user interacted with the item [6]. the static approach – as it is known – while very present in the literature, has proven flaws [21]. taking into account the timing and order of interactions can lead to significant improvements in results [3, 6, 33]. in this context, this study shows how to adapt item2vec [4] – a pioneering method of neural embeddings for a recommendation context – to consider temporal information during training by outlining an approach based on a sliding window over time, a strategy commonly employed in similar scenarios [33]. 2 related work neighborhood algorithms became very popular in the first collaborative filtering recommender systems, given their relative low complexity and ability to produce satisfying results. early neighborhood algorithms represented users as vectors and yielded recommendations based on the interactions history of the users closest to the target user [7]. subsequent approaches began to represent items as vectors [28]. this strategy achieved better results, the training was more efficient, and the results were more stable over time having less need for retraining [17]. gradually, the data used by recommender systems has grown, bringing new challenges to the area such as high sparsity and dimensionality of the commonly adopted representation [18]. many algorithms for dimensionality reduction have been proposed to overcome this problem. among different strategies, matrix factorization techniques have become state-of-the-art and are still very relevant to this day [25]. in this approach, users and items are represented as matrices of reduced dimensionality, and the recommendation is made using user-item similarities [20]. with the growth in research of neural networks, the use of neural models for recommender systems has increased significantly in recent years. among several proposals for using neural networks for recommendation, neural embeddings have been gaining ground in the literature [13]. inspired by natural language processing (nlp) neural models capable of generating vector representations of words [24], these models were adapted to learn low-dimensional embeddings for items and users [27]. grbovic et al. [13] addressed an ad recommendation problem based on receipts collected by email and proposed the prod2vec and user2vec methods. the former is based on a skip-gram architecture [24] learning product embeddings and recommending through item similarity, while the latter is based on paragraph vector [22], learning embeddings of users and items concurrently and recommending through the similarities between users and items. with an approach similar to prod2vec, barkan & koenigstein [4] proposed item2vec in the following year. the method was also inspired by consolidated techniques from the nlp area [24] and achieved promising results, surpassing state-of-the-art matrix factorization methods in different tasks. in the ensuing years, different strategies for learning neural embeddings in recommender systems were proposed, such as (i) adapting existing models to leverage item metadata [11, 31]; (ii) using content information during the embeddings training phase [14, 35]; (iii) employing more complex neural models, e.g., lstms or cnns [29, 30]; (iv) training unaltered nlp models over textual data of items, users, or interactions [9, 15]; and (v) considering different types of user-item interaction [37]. few studies on neural embeddings in recommender systems explored the impact of using temporal information, i.e., when the interaction happened. the most common approach is session-based recommendation, in which only items consumed by the user in a short period are used for training the embeddings, considering an ordered relationship [12] or not [13]. the static approach is still customary in recommender systems as a whole, even though it has flaws [21]. user’s interests change over time, and by not accounting for it, important information can be lost [6, 33]. it is known that considering temporal aspects tends to improve the quality of the recommendation and generate greater confidence for the user [3]. additionally, static algorithms trained in offline scenarios can suffer a drop in performance when applied in online and real scenarios with temporal properties [21] recommender systems capable of learning relevant information from temporal data have been proposed since the beginning of the field [1], but this has gained relevance when timesvd++ [19], a temporal adaptation of the matrix factorization method svd++, won one of the rounds of the netflix prizefootnote 1. two approaches can be applied to temporal information: using time as context, based on the idea of cyclical phenomena as it is done in time-aware recommender systems (tars); or use time as a sequence, viewing temporal information as a chronologically ordered sequence, the strategy used in time-dependent recommender systems (tdrs) [33]. in tdrs two data interpretation methodologies are commonly adopted: the use of decay functions, which make newer interactions have more weight during the learning phase than older interactions [10]; and the use of sliding windows (also known as “forgetting techniques”) in which interactions are sorted chronologically, and only a slice of the data is consumed during training, with the remaining information receiving less weight or being ignored [33]. in the last decade, some studies showed promising results using sliding window strategies for recommender systems. for instance, vinagre & jorge [32] employed two different sliding window techniques in a neighborhood-based recommender. matuszyk et al. [23] proposed five forgetting strategies for matrix factorization using temporal windows or fading factors during data preprocessing and model training. wang et al. [34] trained a neural network to predict shopping baskets in an e-commerce environment using a unit-size sliding window over past baskets. all studies above surpassed baseline algorithms or improved scalability without compromising accuracy. in line with the positive results that have been presented in the literature, we propose seqi2v, a sequential item2vec [4] adaptation using sliding windows over time. in our proposal, the embeddings learning phase is influenced by when interactions occur, increasing the context these item representations carry. the final recommendation is statically computed using an item-based neighborhood approach, just like the original method. 3 the proposed method methods based on neural embeddings are becoming state-of-the-art for recommender systems, presenting results comparable to established approaches, such as neighborhood techniques and matrix factorization. the item2vec was a pioneering technique in introducing neural embedding techniques from natural language processing to the recommendation scenario, adapting the skip-gram network to generate item embeddings. as stated by the authors, the method outperformed the well-known svd in different evaluated tasks. some subsequent studies adapted item2vec to specific scenarios or improved the embeddings training phase [11, 31]. however, temporal information is still underexplored, limiting the application of item2vec to offline scenarios, in which pattern variations over time are ignored. to fill this important gap, we propose using sliding windows over time to introduce temporal information in the learning stage while keeping the elegant simplicity of item2vec. in the following, we explain the traditional item2vec model and present seqi2v – a sequential item2vec adaptation using sliding windows over time. 3.1 item2vec item2vec is an artificial neural network composed of three layers: an input, a hidden and an output layer, as shown in fig. 1. given a recommender system composed of a set u of users, a set i of items and multiple sets \(i_u\) of items consumed by user u, the input layer \(\mathcal {j}\) has a size |i|, as is comprised of items i encoded as one-hot vectors (all elements in the array have a value of 0, except for the one that represents the item, which has a content value of 1). the middle layer \(\mathcal {h}\) has size m, which corresponds to the desired dimensionality for the final vector representation of each item. between the input and the middle layers, there is a weight matrix \(\mathcal {w}\), of size \(|i| \times m\), used to activate the initial layer through the operation \(\mathcal {h} = \mathcal {j} \cdot \mathcal {w}\). the output layer \(\mathcal {o}\) is made up of \(|i_u|-1\) vectors of size |i| connected to the middle layer by the transpose of the same weight matrix \(\mathcal {w}\) and a softmax activation function. thus, \(\mathcal {o} = \sigma (\mathcal {h} \cdot \mathcal {w}^t)\), where \(\sigma \) represents the softmax function shown in eq. 1. $$\begin{aligned} \sigma (u, i) = \frac{e^{(\mathcal {w}_u \mathcal {w}^t_i)}}{\sum _{j \in i}{e^{(\mathcal {w}_u \mathcal {w}^t_j)}}} \end{aligned}$$ (1) fig. 1. standard item2vec architecture full size image the network is trained over multiple epochs. for each epoch, each item \(i \in i_u\) for every user u is used as input. in the output layer \(\mathcal {o}\), we have \(|i_u|-1\) one-hot encoded vectors (the number of items consumed by the user removing the target item), each associated with an item \(j \in i_u \mid j \ne i\). in other words, the network receives an item and must predict another item consumed by the user, approximating them in the vector space. for every pair of input and output items, the network calculates the prediction error and updates the weights. the main goal of the network is to maximize the objective function given by eq. 2. $$\begin{aligned} \frac{1}{|u|}\sum _{u\in u}\sum _{i\in i_u} \sum _{\overset{j\in i_u}{j\ne i}} log \; \sigma (i, j) \end{aligned}$$ (2) at the end of the learning phase, the weight matrix \(\mathcal {w}\) will contain dense vector representations of dimensionality m for every item processed. item2vec come with some drawbacks that would be impractical in scenarios with many items, since it must predict a related item (when output is 1) and all unrelated items (when output is 0) for each target item. thus, two strategies commonly used in nlp are adopted [24]: (i) negative sampling, i.e., the error is calculated over a small subset of n negative items, randomly drawn so that frequent items are more likely to be selected according to eq. 3; and (ii) subsampling of frequent items, i.e., a subset of the interactions of popular items are discarded with a probability of being kept shown in eq. 4. in both equations, function z(i) returns the frequency of item i, and the hyperparameters \(\gamma \) and \(\rho \) can be fine-tuned. $$\begin{aligned} p(i \mid \gamma ) = \frac{z(i)^\gamma }{\sum _{j \in i}{z(j)^\gamma }} \end{aligned}$$ (3) $$\begin{aligned} p(i \mid \rho ) = \left( \sqrt{\frac{z(i)}{\rho }} + 1 \right) \cdot \frac{\rho }{z(i)} \end{aligned}$$ (4) 3.2 sequential item2vec standard item2vec assumes that all items consumed by a user are equally related, shuffling them at the beginning of training. therefore, given a particular item, the network considers that another item consumed in the distant past (e.g., years ago) has the same similarity to the target item as an item consumed in the recent past (e.g., same day). in other words, all other items are considered contextually equal to the target item, as illustrated in fig. 3 for the example shown in fig. 2. fig. 2. user interactions in a chronological order. full size image we have adopted a sliding window of fixed and parameterizable size during training to restrict the context. in this way, only items consumed in sequence are considered to be related to the target item. in a first approach, we are not interested in the moment the interaction occurred, but only in the chronological order, as shown in fig. 4. the neural architecture for seqi2v is very similar to item2vec, the only difference taking place in the number of vectors present in the output layer: instead of a vector for each of the other \(|i_u|-1\) items consumed by the user, the network must predict only s items, where s corresponds to the window size (fig. 5). fig. 3. standard item2vec. every item consumed by the user is equally related to the target item. full size image fig. 4. temporal sliding window during training. full size image fig. 5. sequential item2vec architecture full size image although this model considers the chronological order in which items are consumed, it fails to capture changes in users who spend long periods without interacting with the system. to overcome this problem, we propose splitting a user’s interaction history into various chronological sequences using absolute and pre-defined time intervals to decide where to split. this approach allows that when two continuous interactions of the same user have an interval greater than d days, their chronological sequence is then split into two. as we are only interested in the relationship between items, the network would interpret this single-user as different ones, restricting learning to its behavior in a defined time interval, as illustrated by fig. 6. the value of the parameter controlling the number of days to define the binding time window is flexible and can be adjusted for each specific application scenario a user might want to use seqi2v on. in this work, we opt for an absolute value of days when splitting as it is a less complex and humanly understandable strategy. fig. 6. sequential item2vec. the user interaction history is split to capture the changes in its behavior. full size image during the learning phase and when generating the final recommendation, seqi2v also employs all techniques used by standard item2vec, such as negative sampling and item subsampling during training, and also an item-based k-nearest neighbor when recommending. 4 experiments and results in order to make the experiments reproducible, this section describes the experimental protocol conducted during the study. first, we introduce the datasets used, in conjunction with a description of how the training was carried out. then, we present the achieved results. 4.1 datasets the datasets used in the experiment are listed in table 1. all of them are publicly available and were selected based on their relevance within literature. deliciousbookmarks and movielens belong to grouplens repositoryfootnote 2; bestbuyfootnote 3 and netflixprizefootnote 4 were datasets used in challenges conducted by, respectively, association for computing machinery (acm) and netflix, inc.; ciaodvdfootnote 5 was crawled from ciao website; retailrocket is the database of the retention management platform with the same namefootnote 6. in all these datasets, we removed users or items with a single interaction since most of the trained algorithms aim to learn relations between same-user or item interactions. 4.2 experimental protocol we sorted the interactions chronologically in each dataset and separated them into a train, validation, and test partitions, with an 8:1:1 ratio, respectively. then, we optimized the hyperparameters using grid search on the validation partition, aiming to maximize ndcg in a top-15 recommendation scenario. finally, the final results were obtained on the test partition. due to hardware limitations, we performed the parameter optimization/final experiment of larger datasets over a small and randomly comprised subset: 25%/100% for movielens and 5%/25% for netflixprize. additionally, we removed cold-start cases, i.e., users or items with no interaction on the training set, since this is a problem beyond the scope of our study. table 1. description of the datasets in terms of number of users (|u|), number of items (|i|), number of interactions (|r|), sparsity (e), and application domain.full size table although our main goal is to improve the results of item2vec, we also compared the performance achieved by the proposed sequential item2vec (seqi2v) with classical recommender algorithms. for this, we have implemented the established item-based version of k-nearest neighbors (knn) [28] and two state-of-the-art matrix factorization methods: implicit singular value decomposition (als) [16] and bayesian personalized ranking (bpr) [26]. in addition to the standard item2vec (i2v), we have also implemented user2vec (u2v) [13], a user-item neural embeddings-method. for knn we used the implementation of turicreate library, without requiring hyperparameter choices. both matrix factorization methods were implemented using implicit library, with latent factors ranging in \(\{50, 100, 200\}\), regularization factor in \(\{10^{-6}, 10^{-4}, 10^{-2}\}\), number of epochs in \(\{10, 20, 50, 100\}\), and learning rate in \(\{0.0025, 0.025, 0.25\}\). all neural embeddings models, including sequential item2vec, were implemented using gensim library, with hyperparameter being tuned according insights proposed by caselles-duprés et al. [8]. thus, the dimensionality m was fixed in 100 and 5 negative samples were randomly drawn for each item, with the number of epochs ranging in \(\{50, 100, 150\}\), negative exponent for the negative sampling in \(\{-1.0, -0.5, 0.5, 1.0\}\), and the subsampling factor in \(\{10^{-5}, 10^{-4}, 10^{-3}\}\). additionally, for the sequential adaptation, we tested different window sizes ranging in \(\{1,3,5,7,9\}\), and the interval of days for splitting in \(\{182, 365, 730, \text {no split}\}\). table 2. f1-score obtained in a top-15 recommendation problem.full size table table 3. ndcg obtained in a top-15 recommendation problem.full size table 4.3 results we evaluated the algorithms in a top-15 recommendation task, i.e. for each user, the methods must recommend 15 items in a ranked list. then, we calculated the f1-score (table 2) and the normalized discounted cumulative gain, or ndcg (table 3). all results are in grayscale, with darker tones representing better performances. the best value for each dataset is highlighted in bold. to facilitate comparing the seqi2v with its baseline (item2vec), table 4 presents the improvements and worsenings resulting from the use of seqi2v concerning the results obtained by item2vec in each database and performance measure. in both metrics, the results obtained by sequential item2vec are promising. in all datasets, except for bestbuy, the seqi2v outperformed the i2v. on the retailrocket dataset, the performance of seqi2v was approximately 150% higher. the f1-score achieved by seqi2v was 52% and 43% higher than item2vec, in average and median, respectively. moreover, the ndcg achieved by seqi2v was 48% and 55% higher, in average and median. when compared to the other established recommender algorithms, the results of sequential item2vec remain auspicious. for the f1-score, it attained the best result in 66% of the datasets. on the other hand, as bpr optimizes the ranking, it achieved the best performance when ndcg is used as an evaluation metric. still, for both metrics, seqi2v was the best or second-best in 83% of the cases. it is important to highlight that regarding the number of days used to split the temporal sequences, retailrocket was the only dataset in which sequences with no split achieved the best performance. for all the others datasets, the best performance was attained using sequences split between 182 and 730 days. these results evidence the effectiveness of splitting user sequences and the need to adapt the interval for each specific application. moreover, even when the split was not necessary, using sliding windows significantly improved the results. it is also important to highlight that, although seqi2v demands an extra step of data preparation, training the embeddings is less costly than in item2vec. for each target item, it makes predictions only for a reduced set of items (the window size) instead of all items previously consumed by the user. thus, the proposed method is more computationally efficient and presents better results without increasing implementation complexity. table 4. improvements and worsenings provided by sequential item2vec compared to item2vec.full size table the main limitation of the proposed method relies on exceptional cases in which the user always takes a long period between two interactions, greater than the defined window size. in this case, consumption contexts would be composed of unitary relationships, allowing sliding windows but making the interactions sequence splitting unfeasible. 5 conclusion item2vec is a well-known method capable of learning neural embeddings for items in a recommender system. this study presented sequential item2vec – an adaptation of the original approach that uses a sliding window over time, giving temporal context awareness for the item embeddings. we also proposed splitting the user’s interaction history into multiple sequences so that the network would not learn relationships between items consumed in considerably different moments, thus capturing changes in the user’s behavior. in order to assess whether the proposed approach was able to improve the results obtained with the standard item2vec, we compared both models with other established methods in the field, over six well-known and publicly available databases. metrics of hit and ranking were calculated in a top-15 recommendation scenario. the results indicate that sequential item2vec outperformed the results obtained by item2vec. it was superior to its predecessor in almost all of the datasets. the proposed approach was also competitive compared to the other methods, matching the bayesian personalized ranking, a state-of-the-art matrix factorization method. although giving time awareness to the item embeddings, the recommendation generated by sequential item2vec is still statically. in future research, we aim to consider time in the recommendation step, analyzing its impact. additionally, we suggest studying other ways to include time during the learning phase. one promising approach can be the use of decay functions instead of or in conjunction with sliding windows. notes 1.the well-known competition of recommender systems, organized by the streaming company netflix from 2006 to 2009: https://www.netflixprize.com/rules.html. 2.grouplens. available at: www.grouplens.org/datasets/. access in august 31, 2021. 3.data mining hackathon on big data (7 gb) best buy mobile web site. available at: www.kaggle.com/c/acm-sf-chapter-hackathon-big. access in august 31, 2021. 4.netflix prize data. available at: www.kaggle.com/netflix-inc/netflix-prize-data. access in august 31, 2021. 5.ciaodvd movie ratings. available at: www.konect.cc/networks/librec-ciaodvd-movie_ratings/. access in august 31, 2021. 6.retailrocket recommender system dataset. available at: www.kaggle.com/retailrocket/ecommerce-dataset. access in august 31, 2021. references adomavicius, g., tuzhilin, a.: multidimensional recommender systems: a data warehousing approach. electron. commer. 2232, 180–192 (2001). https://doi.org/10.1007/3-540-45598-1_17 article  math  google scholar  adomavicius, g., tuzhilin, a.: toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. ieee trans. knowl. data eng. 17(6), 734–749 (2005). https://doi.org/10.1109/tkde.2005.99 article  google scholar  baltrunas, l., amatriain, x.: towards time-dependant recommendation based on implicit feedback. in: proceedings of the recsys 2009 workshop on context-aware recommender systems, recsys 2009, pp. 1–5. association for computing machinery, new york (2009) google scholar  barkan, o., koenigstein, n.: item2vec: neural item embedding for collaborative filtering. in: ieee 26th international workshop on machine learning for signal processing, mlsp 2016, pp. 1–6. ieee, piscataway, nj, usa (2016). https://doi.org/10.1109/mlsp.2016.7738886 bobadilla, j., ortega, f., hernando, a., gutiérrez, a.: recommender systems survey. knowl.-based syst. 46, 109–132 (2013). https://doi.org/10.1016/j.knosys.2013.03.012 article  google scholar  de borba, e.j., gasparini, i., lichtnow, d.: time-aware recommender systems: a systematic mapping. in: kurosu, m. (ed.) hci 2017. lncs, vol. 10272, pp. 464–479. springer, cham (2017). https://doi.org/10.1007/978-3-319-58077-7_38 chapter  google scholar  breese, j.s., heckerman, d., kadie, c.: empirical analysis of predictive algorithms for collaborative filtering. in: proceedings of the 14th conference on uncertainty in artificial intelligence, uai 1998, pp. 43–52. morgan kaufmann publishers inc., san francisco, ca, usa (1998) google scholar  caselles-duprés, h., lesaint, f., royo-letelier, j.: word2vec applied to recommendation: hyperparameters matter. in: proceedings of the 12th acm conference on recommender systems, recsys 2018, pp. 352–356. association for computing machinery, new york (2018). https://doi.org/10.1145/3240323.3240377 collins, a., beel, j.: document embeddings vs. keyphrases vs. terms for recommender systems: a large-scale online evaluation. in: proceedings of the 2019 acm/ieee joint conference on digital libraries, jcdl 2019, pp. 130–133. ieee, new york (2019). https://doi.org/10.1109/jcdl.2019.00027 ding, y., li, x.: time weight collaborative filtering. in: proceedings of the 14th acm international conference on information and knowledge management, cikm 2005, pp. 485–492. association for computing machinery, new york (2005). https://doi.org/10.1145/1099554.1099689 peng, f.u., lv, j.h. and li, b.j.: attr2vec: a neural network based item embedding method. in: proceedings of the 2nd international conference on computer, mechatronics and electronic engineering, cmee 2017, pp. 300–307. destech publications, lancaster, pa, usa (2017). https://doi.org/10.12783/dtcse/cmee2017/19993 grbovic, m., cheng, h.: real-time personalization using embeddings for search ranking at airbnb. in: proceedings of the 24th acm sigkdd international conference on knowledge discovery and data mining, kdd 2018, pp. 311–320. association for computing machinery, new york (2018). https://doi.org/10.1145/3219819.3219885 grbovic, m., et al.: e-commerce in your inbox: product recommendations at scale. in: proceedings of the 21th acm sigkdd international conference on knowledge discovery and data mining, kdd 2015, pp. 1809–1818. association for computing machinery, new york (2015). https://doi.org/10.1145/2783258.2788627 greenstein-messica, a., rokach, l., friedman, m.: session-based recommendations using item embedding. in: proceedings of the 22nd international conference on intelligent user interfaces, iui 2017, pp. 629–633. association for computing machinery, new york (2017). https://doi.org/10.1145/3025171.3025197 hasanzadeh, s., fakhrahmad, s.m., taheri, m.: review-based recommender systems: a proposed rating prediction scheme using word embedding representation of reviews. comput. j. 1–10 (2020). https://doi.org/10.1093/comjnl/bxaa044 hu, y., koren, y., volinsky, c.: collaborative filtering for implicit feedback datasets. in: proceedings of the 8th ieee international conference on data mining, icdm 2008, pp. 263–272. ieee computer society, washington, d.c., usa (2008). https://doi.org/10.1109/icdm.2008.22 karypis, g.: evaluation of item-based top-n recommendation algorithms. in: proceedings of the 10th international conference on information and knowledge management, cikm 2001, pp. 247–254 (2001). https://doi.org/10.1145/502585.502627 khusro, s., ali, z., ullah, i.: recommender systems: issues, challenges, and research opportunities. in: information science and applications (icisa) 2016. lnee, vol. 376, pp. 1179–1189. springer, singapore (2016). https://doi.org/10.1007/978-981-10-0557-2_112 chapter  google scholar  koren, y.: collaborative filtering with temporal dynamics. in: proceedings of the 15th acm sigkdd international conference on knowledge discovery and data mining, kdd 2009, pp. 447–456 (2009). https://doi.org/10.1145/1557019.1557072 koren, y., bell, r., volinsky, c.: matrix factorization techniques for recommender systems. computer 42(8), 30–37 (2009). https://doi.org/10.1109/mc.2009.263 article  google scholar  lathia, n., hailes, s., capra, l.: temporal collaborative filtering with adaptive neighbourhoods. in: proceedings of the 32nd international acm sigir conference on research and development in information retrieval, sigir 2009, pp. 796–797. association for computing machinery, new york (2009). https://doi.org/10.1145/1571941.1572133 le, q., mikolov, t.: distributed representations of sentences and documents. in: proceedings of the 31st international conference on machine learning, icml 2014, pp. 1188–1196. jmlr.org (2014). https://doi.org/10.5555/3044805.3045025 matuszyk, p., ao vinagre, j., spiliopoulou, m., jorge, a.m., ao gama, j.: forgetting methods for incremental matrix factorization in recommender systems. in: proceedings of the 30th annual acm symposium on applied computing, sac 2015, pp. 947–953. association for computing machinery, new york (2015). https://doi.org/10.1145/2695664.2695820 mikolov, t., sutskever, i., chen, k., conrado, g., dan, j.: distributed representations of words and phrases and their compositionality. in: proceedings of the 26th international conference on neural information processing systems, nips 2013, pp. 3111–3119. curran associates inc., red hook, ny, usa (2013). https://doi.org/10.5555/2999792.2999959 rendle, s.: factorization machines. in: proceedings of the 10th ieee international conference on data mining, icdm 2010, pp. 14–17. ieee, new york (2010). https://doi.org/10.1109/icdm.2010.127 rendle, s., freudenthaler, c., gantner, z., schmidt-thieme, l.: bpr: bayesian personalized ranking from implicit feedback. in: proceedings of the 25th conference on uncertainty in artificial intelligence, uai 2009, pp. 452–461. auai press, arlington, va, usa (2009). https://doi.org/10.5555/1795114.1795167 rudolph, m., ruiz, f.j.r., mandt, s., blei, d.m.: exponential family embeddings. in: proceedings of the 30th international conference on neural information processing systems, nips 2016, pp. 478–486. curran associates inc., red hook, ny, usa (2016). https://doi.org/10.7916/d8nz9rht sarwar, b.m., karypis, g., konstan, j.a., riedl, j.t.: item-based collaborative filtering recommendation algorithms. in: proceedings of the 10th international conference on world wide web, www 2001, pp. 285–295. association for computing machinery, new york (2001). https://doi.org/10.1145/371920.372071 sidana, s., trofimov, m., horodnytskyi, o., laclau, c., maximov, y., amini, m.-r.: user preference and embedding learning with implicit feedback for recommender systems. data min. knowl. disc. 35(2), 568–592 (2021). https://doi.org/10.1007/s10618-020-00730-8 article  google scholar  tang, j., wang, k.: personalized top-n sequential recommendation via convolutional sequence embedding. in: proceedings of the 11th acm international conference on web search and data mining, wsdm 2018, pp. 565–573. association for computing machinery, new york (2018). https://doi.org/10.1145/2939672.2939673 vasile, f., smirnova, e., conneau, a.: meta-prod2vec: product embeddings using side-information for recommendation. in: proceedings of the 10th acm conference on recommender systems, recsys 2016, pp. 225–232. association for computing machinery, new york (2016). https://doi.org/10.1145/2959100.2959160 vinagre, j., jorge, a.m.: forgetting mechanisms for scalable collaborative filtering. j. braz. comput. soc. 18(4), 271–282 (2012). https://doi.org/10.1007/s13173-012-0077-3 article  google scholar  vinagre, j., jorge, a.m., gama, j.: an overview on the exploitation of time in collaborative filtering. wires data min. knowl. disc. 5, 195–215 (2015). https://doi.org/10.1002/widm.1160 article  google scholar  wang, p., guo, j., lan, y., xu, j., wan, s., cheng, x.: learning hierarchical representation model for nextbasket recommendation. in: proceedings of the 38th international acm sigir conference on research and development in information retrieval, sigir 2015, pp. 403–412. association for computing machinery, new york (2015). https://doi.org/10.1145/2766462.2767694 zhang, f., yuan, n.j., lian, d., xie, x., ma, w.y.: collaborative knowledge base embedding for recommender systems. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, kdd 2016, pp. 353–362. association for computing machinery, new york (2016). https://doi.org/10.1145/2939672.2939673 zhang, s., yao, l., sun, a., tay, y.: deep learning based recommender system: a survey and new perspectives. acm comput. surv. 52(1), 5:1–5:35 (2019). https://doi.org/10.1145/3285029 zhao, x., louca, r., hu, d., hong, l.: the difference between a click and a cart-add: learning interaction-specific embeddings. in: companion proceedings of the web conference 2020, www 2020, pp. 454–460. association for computing machinery, new york (2020). https://doi.org/10.1145/3366424.3386197 download references author information authors and affiliations department of computer science, federal university of são carlos, sorocaba, brazil pedro r. pires, amanda c. pascon & tiago a. almeida authors pedro r. piresview author publications search author on:pubmed google scholar amanda c. pasconview author publications search author on:pubmed google scholar tiago a. almeidaview author publications search author on:pubmed google scholar corresponding author correspondence to pedro r. pires . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper pires, p.r., pascon, a.c., almeida, t.a. (2021). time-dependent item embeddings for collaborative filtering. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_22 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords collaborative filtering distributed vector representation item embeddings temporal data sliding window publish with us policies and ethics profiles tiago a. almeida view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature enriching portuguese word embeddings with visual information | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper enriching portuguese word embeddings with visual information conference paper first online: 28 november 2021 pp 434–448 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) enriching portuguese word embeddings with visual information download book pdf download book epub bernardo scapini consoli  orcid: orcid.org/0000-0003-0656-511x10 & renata vieira  orcid: orcid.org/0000-0003-2449-547711  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1202 accesses 1 citation abstract this work focuses on the enrichment of existing portuguese word embeddings with visual information in the form of visual embeddings. this information was extracted from images portraying given vocabulary terms and imagined visual embeddings learned for terms with no image data. these enriched embeddings were tested against their text-only counterparts in common nlp tasks. the results show an increase in performance for several tasks, which indicates that visual information fusion for word embeddings can be useful for word embedding based nlp tasks. financially supported by the brazilian national council for scientific and technological development (cnpq) and by the portuguese foundation for science and technology (fct) under the projects ceecind/01997/2017, uidb/00057/2020. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others enriching portuguese word embeddings with visual information chapter © 2022 visual word embedding for text classification chapter © 2021 learning word sense embeddings from word sense definitions chapter © 2016 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. embodiment instructional design language processing natural language processing (nlp) portuguese data and information visualization 1 introduction language modelling technologies have been dominated by semantic embedding models ever since mikolov et al. (2013b) and mikolov et al.’s (2013a) [11, 12] popularization of word embeddings, a concept which revolutionized the field of natural language processing (nlp). the architecture presented by the authors, word2vec, has been used as basis for many works across the spectrum of nlp tasks, as attested by nearly 45,000 citations when accounting both of the aforementioned papers (as recorded by google scholar), mainly because of the fact that training this architecture only requires raw text, and no human-made annotation (the main obstacle in training machine learning models). many architectures based on the original intuition behind word2vec have become popular since 2013. the most prevalent, besides the original word2vec, are fasttext [7] and glove [14]. an evolution upon the concept, taking into account the current context of a word, not just an amalgamation of all contexts with which it was trained, was introduced by peters et al. (2018) [15], with their elmo architecture, and popularized by devlin et al.’s (2019) [4] bert architecture. all of the mentioned embedding architectures have at least one model trained on portuguese language corpora. the núcleo interinstitucional de linguística computacional (nilc), from the universidade de são paulo (usp), for example, has several word2vec, fasttext and glove models for the portuguese language available within their word embedding repositoryfootnote 1. the allen institute for ai maintains an elmo model repository which includes a portuguese language modelfootnote 2. bertimbau [19], a portuguese language bert model, was recently developed and added to the hugging facefootnote 3 library. these models, and others, have been used to advance the state-of-the-art in several portuguese language nlp tasks [6, 9, 17]. beyond these efforts to further enhance the usage of text in the training of word embedding models, be it portuguese language text or otherwise, an effort to enrich these embeddings with other modes of information also arose. the most studied modes of information used to enhance word embeddings are the visual mode (composed of images and video), and the aural mode (composed of sounds, spoken language, music, etc.). these efforts spurred the creation of multimodal embedding fusion architectures, used to join embeddings of disparate modes into a single embedding representing all fused knowledge. an example of this is the concatenation based architecture of bruni et al. (2014) [1], which arrived at promising results after proposing that multiple embeddings of different modes could be concatenated, resulting in a higher-dimensional space, for them to be enhanced for better use in nlp tasks. the goal of this work is to study the usage of visual data to enrich textual data within word embeddings for nlp tasks in the portuguese language. the main hypothesis is that fusing textual information with visual information will enhance results for text-only tasks. to test it, experiments in four nlp tasks were performed using text-only and multimodal embedding models. these tasks were: word relatedness, analogy prediction, semantic similarity in short sentences, and named entity recognition. the following sections are arranged in the following manner: sect. 2 delves into related work within the literature; sect. 3 explains the methodology used in creating the multimodal embeddings; sect. 4 explains the testing methodology and presents the results for these tests; sect. 5 presents the conclusions and future work that can be done to expand upon this topic. 2 related work the literature reveals two main ways in which multimodal embeddings are constructed: individually and simultaneously. that is, either learning is performed individually (an embedding is learned for each modality, and then these are fused) [3], or simultaneously (all modalities are learned at the same time in the same space). henceforth, the former method will be referred to as post-learning fusion, while the latter method will be referred to as simultaneous learning. post-learning fusion is divided into two further methods: early fusion and late fusion. early fusion is performed at the representation level, and three methods of early fusion were found in the literature: feature concatenation, auto-encoder fusion, and cross-modal mapping. feature concatenation is performed through the concatenation of all single modality fusion embedding vector pairs (that is, a textual feature vector representing a concept will be concatenated with a visual feature vector representing that same context) into a single, longer, multimodal feature vector [8]. auto-encoder fusion is performed through the use of auto-encoders fed with pre-trained single modality embeddings, thus generating a single feature vector which can then be extracted from the auto-encoder’s last hidden layer [18]. cross-modal mapping is performed through the learning of a certain amount of pre-mapped multimodal inputs and predicting those that do not have examples in both modalities [3]. late fusion is performed at the level of prediction scores, and it is performed through an averaging of single modality predictions [8]. lazaridous et al. (2015) [10] introduced the first instance found during the review of simultaneous learning semantic embedding model, based on mikolov et al.’s (2013) [12] skip-gram architecture. they extended mikolov et al.’s (2013) models to present relevant visual feature vectors alongside textual data during training for a subset of target words. this model has been shown to further propagate visual information to representations of words which were not trained with visual features. as for evaluation methodologies, most of the literature consisted of using multimodality to improve the performance of downstream tasks. as such, the evaluation of the embeddings was extrinsic. that is, the evaluation metric was whether or not its addition to the systems performing the downstream task affected their performance. lazaridou et al. (2015) [10] were the only ones to perform intrinsic tests, using general semantic benchmarks such as concept relatedness (also known as semantic relatedness) [1] or semantic similarity. these are usually used to evaluate word embeddings, but multimodal embeddings were shown by lazaridou et al. (2015) to outperform word embeddings on these tasks. 3 resources and methodology several choices had to be made prior to developing this work’s methodology. it was decided that these first tests would use static word embeddings based on the word2vec and fasttext architectures, and use a translated version of the image embeddings released by collell et al. (2017) [3]. these readily available resources made post-learning fusion methods an obvious choice for developing our own fusion architectures. since the tasks we intended to use these models for were strictly textual, late post-learning fusion was not an option, as there would be no visual data to input into the system during tests, which left early post-learning fusion techniques. the resources and architectures mentioned in the above paragraph are elaborated upon in the following subsections. this section also explains the multimodal model development methodology. 3.1 unimodal embeddings the text embeddings used for this work were nilc’s word embeddingsfootnote 4 [9] and bbp corpus word embeddingsfootnote 5 [17]. three versions of nilc’s embeddings were used: the 100 feature word2vec version and the 100 and 300 feature fasttext versions. these three were deemed to be adequate for studying the effect of different parameters when adding multimodality to textual models. only the 300 feature fasttext version of bbp was used, as it was the only one readily available for download. this final bbp model was chosen as a means to study how different text embedding training corpora within the same domain affected multimodal fusion. table 1. corpora and token totals for each of the text corpora used for training text embedding models.full size table the visual embedding, henceforth referred to as the imagenet embedding, is derived from collell et al.’s (2017) [3] work, as they made their original visual embeddings created using imagenet freely availablefootnote 6. the individual embeddings were paired with english language terms from the english language wordnet, however, and so needed to be translated before use with portuguese language textual embeddings. in order to translate the english terms, openwordnet-pt [13], an open brazilian wordnet available onlinefootnote 7, was used. since the codes used to refer to each term in both wordnets were the same, and collell et al. (2017) also shared the wordnet code for each term, about 5000 of the  18000 original text-visual embedding pairs were successfully translated into brazilian portuguese unigrams. this resulted in what we believe to be the first visual embedding dataset paired with brazilian portuguese terms, made available through our github pagefootnote 8. 3.2 dealing with the information gap the great imbalance between visual embeddings and text embeddings becomes clear when comparing the roughly 5000 terms of the imagenet embedding to the textual embedding vocabularies shown in table 1. in order to ameliorate this problem, the “imagined embeddings” architecture described in collell et al. (2017) [3] was used. as exemplified in fig. 1, textual embedding-visual embedding pairs are created for the terms present in the visual embedding vocabulary, w, and used to train a feed-forward neural network. it does this by inputting the textual embedding \(\overrightarrow{l_x}\) into the nn, and expecting the visual embedding \(\overrightarrow{v_x}\) as an output, where the \(w_x\) is the term being learned. once this textual-visual translation, f, is learned by the network, it can be extrapolated into terms without visual counterparts, creating “imagined” visual embeddings for the entire vocabulary represented by the textual embedding that was translated. fig. 1. example of the architecture used by collell et al. (2017). the imagined representations are the outputs of a text-to-vision mapping, f. image created by collell et al. (2017) [3] full size image our imagined embedding networks used the following parameters: 0.25 dropout; 0.1 learning rate, sgd optimization; mse based loss; 1 hidden layer with 200 nodes; and a tanh activation function. as was done in collell et al. (2017), we chose certain epoch thresholds to test. these were 25, 50, 100, and 150 epochs for each model. the translated imagenet embeddings were used to train these imagined embeddings for each text model. notably, the work discusses that while these imagined embeddings are valuable aggregates to common embeddings, substituting the textual embeddings completely with these “imagined embeddings” yields worse results. additionally, in a follow-up paper, collell et al. (2018) [2] highlighted several problems with this architecture, such as the fact that they do not fully mimic the behaviour of proper visual embeddings to the desired degree. it remains, however, that when combined with the original textual embeddings, these “imagined embeddings” do positively affect results in intrinsic tasks such as word relatedness. 3.3 fusion techniques two fusion techniques were tested in this work: concatenation and auto-encoding. concatenation was performed as detailed in collell et al. (2017), but for the portuguese language, while auto-encoding was inspired by the work of silberer and lapata (2014) [18] and adapted to work with our resources. furthermore, in order to perform this kind of fusion, it is helpful to ensure that all fused embeddings are in the same scale, so that none can overly influence the result simple because it is presented in a larger scale than another. to do this, a mathematical process called standardization was performed on the embeddings, making it so all features were scaled according to a standard deviation of 1 and had a mean of 0. concatenation fusion. this technique is simple: concatenate one mode’s embeddings to the end of another mode’s embeddings. this effectively packages all necessary data into a single vector space by expanding the dimensionality of that space. this fusion technique’s greatest weakness is the fact that should one embedding in a certain mode not have a counterpart in another mode (as often happens with text-image multimodality, e.g. the presence of a textual embedding with no counterpart visual embedding) it is not possible to create the multimodal embedding. this problem is solved with the use of imagined embeddings, which, though not perfectly representative of an actual image embedding, allows for the concatenation of the entire vocabulary. as such, the development of this embedding required the prediction of an imagined visual embedding for each word in the vocabulary, which was then concatenated with its originating word embedding. this resulted in multimodal embeddings with larger feature pools with which to draw from. figure 2 presents the architecture of the concatenated fusion used for every word embedding in this work. fig. 2. simplified concatenation fusion architecture. full size image auto-encoding fusion. this technique is performed by a neural network trained to predict an output by using the output itself as an input. once this is done, one of the hidden layers of this network with less features than the original input is extracted to serve as an embedded version of the input. this serves to both shorten the final embedding, and to fuse several embeddings together. this fusion, in theory, keeps the most important features and fuses less important features together to make them more impactful. this architecture has been used to lessen the impact of the gap between textual and visual information in the literature [18]. in this instance, whenever there was no visual pair for the textual embedding, a zeroed vector was appended to the textual embedding for the purposes of auto-encoding. the architecture presented below is a bit different, as it offers a new possibility: using imagined embeddings to fill the knowledge gap and offer complete feature vectors for auto-encoding. figure 3 presents the architecture of the auto-encoded fusion used for every multimodal word embedding in this work fig. 3. simplified auto-encoding fusion architecture. full size image the hidden layers are divided into two encoding layers and two decoding layers. the first encoding layer has the initial input node size of the concatenated textual-visual feature vector and an output node size of the feature vector of the textual model plus half the feature vector of the visual model. the second layer has the input node size of the previous output, and an output the size of the feature vector of the textual embedding. the output of this second layer is extracted and used as the auto-encoded textual-visual embedding. the decoder is used only during training, and its two hidden layers are the same as the encoder’s, but in reverse order. the auto-encoding networks used the following parameters: 0.001 learning rate; adam optimization; mse based loss; four hidden layers, as explained above; and relu activation funtions between layers with a tanh function as a final output. 3.4 multimodal embeddings several different textual-visual multimodal embeddings were created using the unimodal embeddings and multimodal fusion techniques explained above. four text corpora were used for training: a 300-dimensional fasttext model using the bbp text corpus (bbpft300); a 300-dimensional fasttext model using the nilc text corpus (nilcft300); a 100-dimensional fasttext model using the nilc text corpus (nilcft100); and a 100-dimensional word2vec model using the nilc text corpus (nilcft100). the imagined embeddings were trained from the  5000 available word-image pairs to four epoch thresholds: 25, 50, 100, and 150. finally, each of the four text corpora were fused with all imagined embedding epoch threshold models, meaning a total of 16 models, four for each text corpus. since we used two different fusion architectures, this total is doubled to a final roster of 32 tested models, with 16 for the concatenation fusion architecture and 16 for the auto-encoding fusion architecture. note that the act of training to different epochs was simply due to a lack of time and computational resources that would be required to train the best model for each individual task. as such, the best performing model out of each group can be taken to best represent the capabilities of the multimodal embedding fusion in question. 4 tests and results four tests were performed on the 32 models created for his work. these were word relatedness, analogy prediction, semantic similarity in short sentences, and named entity recognition. these tests and their results are presented in that order within this section. note that for each model, all four training epochs were tested for each fusion architecture of each model, but only the results for the best performing training epoch for each model are displayed for comparison in the below tables. 4.1 word relatedness word relatedness is the intrinsic task of giving a score to how closely related two terms are. these tasks are usually scored via spearman correlation, which assigns a real number score between −1 and 1. the score approaches -1 the more inversely correlated the predictions are to the annotation; it approaches 0 the more unrelated the predictions are to the annotation; and it approaches 1 the more directly correlated the predictions are to the annotation. the more representative of human understanding of the terms an embedding is, the closer the spearman score comes to 1. a custom code was written for this task. it uses the gensim python library to extract the cosine distance between each word pair as a relatedness measurement, and compares them to their respective annotated relatedness scores using the spearman correlation method. the code can be accessed in the github page for this projectfootnote 9. the test set is a collection of 3000 word pairs annotated with a relatedness score from 50 (most related) to 0 (least related). the objective of the semantic models is to score each word pair in order to rank them from most related to least related. the closer to the original ranking the model gets, the higher its spearman correlation, the chosen method for scoring these kinds of tests. table 2 presents some examples from this test set. table 2. four examples of word pairs from the translated men corpus.full size table table 3. the best spearman correlation results for each multimodal model and the results for their text-only counterparts for the men test set.full size table the test corpus used for word relatedness testing, men [1], was translated from the english language to the portuguese language with the help of deepl translatefootnote 10. the machine translations were checked individually to ensure some degree of uniformity, but the corpora should be considered silver standard nonetheless. table 3 shows the results of this test, with the best overall model and the best architecture for each model marked in bold. this table shows that multimodal fusion enhances model performance for this task by an average of 3% points, and that the best overall architecture is the auto-encoded architecture. it should be said, however, that between the multimodal models, the average difference is 0.5% point in favor of the auto-encoded architecture, so neither fusion architecture has an overwhelming advantage over the other in this task. 4.2 analogy prediction hartmann et al. (2017) [9] published an analogy prediction test set, divided into brazilian portuguese and european portuguese halves, alongside their initial publication of their nilc word embeddings. the test gives a related word pair and a single word from which it must predict a pair analogous to the first. the code used to run these tests was made available alongside the test set itself. it can be found in the associated paper’s github pagefootnote 11. it measures accuracy by counting how many correct predictions were achieved by the model against the total number of predictions. this test focuses on two kinds of analogies: semantic and syntactic. these are each divided into a brazilian portuguese set and an european portuguese set. to reiterate, the objective of this task is to accurately predict the second word of a pair, when given an example pair and the first word of the prediction pair (e.g. berlin/germany, prediction: paris/?). the accuracy of the model is then measured in a percentage, from 0 (completely inaccurate) to 100 (completely accurate). table 5 shows the results of this test, with the best overall model and the best architecture for each model marked in bold. table 4 presents some examples for this test set. table 4. four examples, two semantic and two syntactic, of word pairs from the analogy prediction corpus, translated to english.full size table the results, presented in table 5, show that the two language-specific test sets were corroborative, with similar results being achieved in all test set pairs. on average, there was a 0.01% point difference between text-only and the best multimodal results, which leads to the conclusion that the task of analogy prediction was mostly unaffected by the addition of visual information through the multimodal fusion methods tested herein. that said, the auto-encoding architecture tended to underperform when compared to the concatenated architecture in all cases but those in which the nilcft100 semantic model was used. these results were not outside expectations, as the image data used for the visual embeddings focused mostly on objects, while the analogy prediction tests focused on abstracts such as parentage, countries, currency and word forms. it is promising, however, that the previously mentioned best overall model achieved the best multimodality results when compared to their text-only counterpart. perhaps with further testing, it might be ascertained that the better the original text-embedding, the more effective the imagined visual embedding fusion is. table 5. the best accuracy results for each multimodal model and the results for their text-only counterparts for the brazilian portuguese and european portuguese analogy prediction test sets.full size table 4.3 semantic similarity in short sentences semantic similarity requires that a model give a numerical value to how semantically similar two sentences are, with the lower similarity extreme being that the sentences are completely different, and the higher similarity extreme being that the sentences are paraphrases. the assin [5] sentence similarity corpus was used for this task in this work. the code used for the tests is the same as was used by hartmann et al. (2017) [9], available in the publication’s github pagefootnote 12. the architecture uses a linear regression algorithm trained on two features: the cosine similarity of the tf-idf of each sentence and the cosine similarity between the sum of each sentence’s word embeddings. the assin semantic similarity dataset is divided into two tracks, european portuguese and brazilian portuguese. the objective of the task is to predict a number between 1 (unrelated sentences) and 5 (paraphrasing sentences) to represent the similarity between two short sentences. the task was evaluated using pearson’s correlation and mean standard error (mse), as it was during the original assin task. table 6 shows the results of this test, with the best overall model and the best architecture for each model marked in bold. figure 4 presents some examples for this test set. fig. 4. example of sentence pairs from the assin corpus. full size image the results found with these semantic similarity tests are very similar to those found during the word relatedness tests presented previously in this section. these tests found that multimodal models outperform text-only models by an average of 2% points and that the auto-encoding architecture is generally superior to the concatenation architecture within these test conditions. table 6. the best results for the brazilian portuguese and european portuguese tracks of the assin semantic similarity task.full size table 4.4 named entity recognition named entity recognition (ner) requires that, given a set of classes for named entities, a model recognize and classify these entities within raw text, usually by use of tags. word embeddings can be used to parse the text input into the model, using the feature vectors in its tagging predictions. the harem [16] ner corpus was used to test the models. the code used for the ner tests was developed by santos et al. (2019) [17], available in the paper’s github pagefootnote 13. it uses an lstm-crf neural network architecture to train a sequence tagger using the flair toolkit to perform a ner task based on the supplied training and test corpora. the harem test set is composed of two tracks: the selective track; and the total track. the selective track is the smaller of the two, including only the five most populated named entity categories within the test set. the total track is the larger of the two, including all ten named entity categories present in the first harem test set. table 7 shows the results of this test, with the best overall model and the best architecture for each model highlighted in bold. figure 5 presents some examples for this test set. fig. 5. a snippet of a sentence from the first harem, to exemplify its annotation. full size image table 7. the best results for the selective and total tracks of the harem task.full size table on average, the multimodal models outperform text-only models by 0.25% point in the selective track, and are outperformed by the text-only models by 0.08% point in the total track. these results show that the addition of visual information through multimodal fusion does not have much effect on the results for our models in this test set. 5 conclusion and future work this work presented the results of a study into the usefulness of visual data when used in conjunction with textual data for nlp tasks in a general news domain. it involved the development of word embedding models which were then put through a test battery for multimodal word embedding models which included the following tasks: word relatedness, sentence similarity, analogy prediction and named entity recognition. these results revealed some aspects of textual-visual multimodal fusion for word embeddings within nlp tasks for the portuguese language, a field in which it is most common to study purely textual word embedding models. it took inspiration from the works of bruni et al. (2014) [1], and their concatenation based multimodal fusion architecture; silberer et al. (2014) [18], and their auto-encoding multimodal fusion architecture; and collell et al. (2017) [3], and their imagined embeddings cross-modal mapping neural network, for visual vocabulary expansion. it took a different tack from previous work by exploring the possibility of use of this technology beyond the english language, using resources for the portuguese language and in previously unexplored combinations. testing revealed that in tasks which require broader semantic meaning judgements, such as word relatedness and semantic similarity, multimodal fusion with visual information enhances results. for the specific test sets within these tasks explored in this work, the average increase in correlation with human scoring was of 2% points. for the tasks of analogy prediction and named entity recognition, however, the fusion resulted in little to no impact in the final results. this might be explained by the fact that these annotations in particular make use of knowledge that is not present within the visual modality, and thus were not enhanced by its addition. as future work, we have planned testing multimodal fusion techniques on contextual embeddings such as bert and elmo. we also plan to expand testing into inherently multimodal tasks such as text-image pairing and cross-modal retrieval, both for the portuguese language. notes 1.http://www.nilc.icmc.usp.br/embeddings. 2.https://allennlp.org/elmo. 3.https://huggingface.co/. 4.http://www.nilc.icmc.usp.br/embeddings. 5.https://github.com/jneto04/ner-pt. 6.https://liir.cs.kuleuven.be/software_pages/imagined_representation_aaai.php. 7.http://wn.mybluemix.net/. 8.https://github.com/bsconsoli/enriching-portuguese-word-embeddings-with-visual-information. 9.https://github.com/bsconsoli/enriching-portuguese-word-embeddings-with-visual-information. 10.https://www.deepl.com/translator. 11.https://github.com/nathanshartmann/portuguese_word_embeddings. 12.https://github.com/nathanshartmann/portuguese_word_embeddings. 13.https://github.com/jneto04/ner-pt. references bruni, e., tran, n., baroni, m.: multimodal distributional semantics. j. artif. intell. res. 49, 1–47 (2014) article  mathscinet  google scholar  collell, g., moens, m.: do neural network cross-modal mappings really bridge modalities? in: proceedings of the 56th annual meeting of the association for computational linguistics, pp. 462–468 (2018) google scholar  collell, g., zhang, t., moens, m.: imagined visual representations as multimodal embeddings. in: proceedings of the 31st aaai conference on artificial intelligence, pp. 4378–4384 (2017) google scholar  devlin, j., chang, m., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 17th conference of the north american chapter of the association for computational linguistics on human language technologies, pp. 4171–4186 (2019) google scholar  fonseca, e.r., santos, l.b., criscuolo, m., aluísio, s.m.: visão geral da avaliação de similaridade semântica e inferência textual. linguamática 8, 3–13 (2016) google scholar  gomes, d.s.m., et al.: portuguese word embeddings for the oil and gas industry: development and evaluation. comput. ind. 124, 1–44 (2021) article  google scholar  grave, e., mikolov, t., joulin, a., bojanowski, p.: bag of tricks for efficient text classification. in: proceedings of the 15th conference of the european chapter of the association for computational linguistics, pp. 427–431 (2017) google scholar  habibian, a., mensink, t., m., s.c.g.: video2vec embeddings recognize events when examples are scarce. ieee trans. pattern anal. mach. intell. 39, 2089–2103 (2017) google scholar  hartmann, n., fonseca, e.r., shulby, c., treviso, m.v., rodrigues, j.s., aluísio, s.m.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: proceedings of the 11th brazilian symposium in information and human language technology, pp. 122–131 (2017) google scholar  lazaridou, a., pham, n.t., baroni, m.: combining language and vision with a multimodal skip-gram model. in: proceedings of the 13th conference of the north american chapter of the association of computational linguistics on human language technologies, pp. 153–163 (2015) google scholar  mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space. in: proceedings of the 1st international conference on learning representations, p. 12 (2013) google scholar  mikolov, t., sutskever, i., chen, k., corrado, g.s., dean, j.: distributed representations of words and phrases and their compositionality. in: proceedings of the 27th annual conference on neural information processing systems, pp. 3111–3119 (2013) google scholar  paiva, v., rademaker, a., melo, g.: openwordnet-pt: an open brazilian wordnet for reasoning. in: proceedings of the 24th international conference on computational linguistics, pp. 353–360 (2012) google scholar  pennington, j., socher, r., manning, c.d.: glove: global vectors for word representation. in: proceedings of the 19th conference on empirical methods in natural language processing, pp. 1532–1543 (2014) google scholar  peters, m.e., neumann, m., iyyer, m., gardner, m., clark, c., lee, k., zettlemoyer, l.: deep contextualized word representations. in: proceedings of the 16th conference of the north american chapter of the association for computational linguistics on human language technologies, pp. 2227–2237 (2018) google scholar  santos, d., cardoso, n.: a golden resource for named entity recognition in portuguese. in: proceeding of the 7th international conference on the computational processing of portuguese, pp. 69–79 (2007) google scholar  santos, j., consoli, b.s., santos, c.n., terra, j., collovini, s., vieira, r.: assessing the impact of contextual embeddings for portuguese named entity recognition. in: proceedings of the 8th brazilian conference on intelligent systems, pp. 437–442 (2019) google scholar  silberer, c., lapata, m.: learning grounded meaning representations with autoencoders. in: proceedings of the 52nd annual meeting of the association for computational linguistics, pp. 721–732 (2014) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: proceedings of the 9th brazilian conference on intelligent systems, pp. 403–417 (2020) google scholar  download references author information authors and affiliations pontifical catholic university of rio grande do sul, pucrs. school of technology, porto alegre, brazil bernardo scapini consoli cidehus, university of évora, évora, portugal renata vieira authors bernardo scapini consoliview author publications search author on:pubmed google scholar renata vieiraview author publications search author on:pubmed google scholar corresponding author correspondence to bernardo scapini consoli . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper consoli, b.s., vieira, r. (2021). enriching portuguese word embeddings with visual information. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_30 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords word embedding models multimodality portuguese language publish with us policies and ethics profiles renata vieira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature humanities and ai: ethical education in technology careers | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper humanities and ai: ethical education in technology careers conference paper first online: 30 january 2025 pp 81–93 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) humanities and ai: ethical education in technology careers download book pdf download book epub laura simões camargo  orcid: orcid.org/0000-0002-9685-89709,10, enio alterman blay  orcid: orcid.org/0000-0001-7097-940010 & gabriela soares schmidt  orcid: orcid.org/0009-0005-3187-108110  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 437 accesses 2 citations abstract this paper presents a quantitative and qualitative study of the teaching of ethics and related subjects in undergraduate courses in computing of 17 top-ranked universities evaluated in three different ranking systems. the analysis was performed using documents from the selected universities and semi-structured interviews with professors and a head of department. this article discusses the relevance of ethical and social-oriented teaching and classifies the syllabuses of the sampled disciplines, assessing the type of focus that is apparently given. the overall results show considerable variability in the availability of data, ranging from courses whose content is very detailed to those that don’t present their syllabus at all. considering that the development of critical reasoning based on ethics is extremely relevant to the education of future artificial intelligence developers, the authors concluded that universities are not fulfilling their role adequately. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ai ethics in higher education: a review of ethical challenges chapter © 2026 ethics and ai in higher education: a study on students’ perceptions chapter © 2024 the use of generative ai tools in higher education: ethical and pedagogical principles article 14 february 2025 1 introduction the world has changed since the release of openai’s generative conversational tool at the end of 2022 [5]. the consequent concern about regulation and ethical behavior in the face of artificial intelligence (ai) has become the order of the day worldwide [7]. however, little attention seems to be paid to critical education for ai developers, especially in the global south. unesco itself [9, 10] has published reports on the impacts of ai on education at basic levels and on society at large but has not addressed the professional education of those who create and employ the algorithms themselves. in addition, studies such as [4, 6, 14, 16] and [11] are at the frontier of the discussion on ethics, education and ai, but address issues outside the brazilian context. in the various areas of computing, higher education is the natural path for both scientists and technical professionals who work in ai. in order to focus on education as a possible way of mitigating the risks and negative social impacts of ai, this article maps out the humanistic subjects offered in bachelor’s degree programs in the field of computing at main brazilian universities and classifies their approaches. despite the worldwide impact of the social challenges posed by recent generative technologies, one might consider that the discussion of ethics in the field of computing has been going on for more than 20 years. not only was the association of computing machinery’s (acm) code of ethics [1] published in the 1990s, but some academics have been dedicated to teaching this subject long before the contemporary phenomenon of ai [8], even presenting the same type of dilemmas, such as the reproduction of racism, the misappropriation of intellectual property, among others. however, through interviews conducted in this research, it was possible to gather evidence that these efforts have been and continue to be individual, rather than systemic or structural. although there are groups concerned with the social impacts of ai within computing, they are still small and their initiatives are insufficient in the face of current challenges. there are more than 50 million places offered by around 22,000 bachelor’s courses in computer science, information systems, software engineering, and computer engineering registered with the ministry of education (mec). this data demonstrates the magnitude of the area and reinforces the relevance of the discussion on the critical education of these students. considering the size of the universe to be researched, it would be beyond the scope of this work to analyze all the courses. the authors, therefore, opted for a subset of 17 universities, chosen for their importance in terms of publications in the area of ai, verified on the scopus platform and in a fapesp publication [12], and a ranking system, presented by the specialized publication folha de são paulo university ranking 2023, as detailed in the sect. 3 section. in addition to this introduction, this article is composed of five other sections. the second section presents aspects of the field of computing related to ethics. the third section presents the method used to collect the data. the next section discusses the quantitative and qualitative results of the subjects. the fifth section systematizes the syllabuses according to their classifications. finally, the final section presents some of the authors’ conclusions. 2 the construction of the idea of ethics in computing professional activities have always involved ethical issues. this goes back to the most ancient civilizations whose history is documented, as well as the rules of native peoples whose traditions are lost in time. of today’s professions, the medical field has the oldest tradition of ethical values. this can be traced back to the hippocratic oath, for example, which deals with commitments such as secrecy and beneficence in the doctor-patient relationship. created more than 2.5 millennia ago and introduced into brazilian medical practice in 1771, the oath is still taken at the end of undergraduate medical courses. perhaps that’s why the biological sciences have such well-structured practices and regulations, as well as a network of 890 research ethics councils (cep) throughout brazil. in the case of computing, although some discussions about the ethical use of technology goes way back, systematic discussions in congresses and seminars date to end of the 1980s and begin of the 1990s [15]. in addition to the fact that there are no research ethics councils in computing, it is not customary for professionals in the field to submit their projects to other councils, which should be the case, especially when involving human beings. this means that there is no formal impediment to computer professionals carrying out their work in an unethical manner. 3 data collection and analysis having explained the context of computer science education and its dimension in brazil, this section details how the data was collected and analyzed in this research. in order to build the sample, the universities analyzed were chosen from three sources: i) the article “scientific publications on artificial intelligence” by fapesp [12], ii) folha de são paulo university ranking 2023 (ruf) and iii) searches carried out by the authors on the scopus platform. these sources were chosen to combine the relevance of the academic output of these universities with their other characteristics, such as teaching, innovation, and internationalization. after defining the sources, the ten institutions with the greatest relevance in each of them were selected. once the data had been cross-referenced and overlaps excluded, the final sample was consolidated in table 1, presented below. a detailed explanation of the selection process is described in the following subsections. table 1. sample composition. source: author’s elaboration, 2024.full size table 3.1 fapesp and scopus rankings according to the article “scientific publications on artificial intelligence” [12], the 10 leading institutions in ai publications between 2014 and 2018 were, respectively: usp, unicamp, ufpe, ufmg, ufrn, ufrgs, ufpr, ufscar, unesp, and utfpr. although international rankings like edurank [17] and times higher education [18] evaluate more metrics to achieve universities classification, they provide a generic and global ranking on all areas. as this article focuses on ai, the authors simplified the criteria aiming at this specific area in brazil. fig. 1. ai publications by author affiliation. source: plataform scopus, 2024. full size image based on scopus searches using the keywords “artificial intelligence”, “machine learning”, “deep learning” and “natural language processing”, the 10 brazilian institutions with the largest number of publications on each topic were listed. after eliminating overlaps, 11 universities were added. they were: usp, unicamp, ufpe, ufmg, ufrgs, ufrj, ufsc, unb, puc-rj, uff, and puc-rs. it is worth noting that just by searching for “artificial intelligence” as a keyword, there were more than 65,000 articles published in brazil between 1974 and 2024, i.e. in the last 50 years. as shown in fig. 1, the universities with the most publications in the area are, respectively: usp, unicamp, ufrj, ufmg, ufrgs, ufpe, puc-rj, ufsc, uff, and unb. 3.2 ruf according to the folha de são paulo university ranking 2023 (ruf) [13], an annual assessment of higher education in brazil that considers teaching, research, market orientation, innovation and internationalization indexes, the best-placed universities are, respectively: usp, unicamp, ufrgs, ufrj, ufmg, unesp, ufsc, ufpr, unb and ufpe. 3.3 consolidating the field and building the database once the sample of universities to be analyzed had been selected, the bachelor’s degree courses were chosen. during the initial phase of the research, the authors held conversations with researchers from usp’s center for artificial intelligence (c4ai) about the ai field. a university degree in computing was presented as the main qualification among ai developers in the private sector. in addition, resolution no. 5 of november 16, 2016 [3], which establishes the national curriculum guidelines for undergraduate courses in the area of computing, states that the area is composed of the following courses: computer science, information systems, computer engineering and software engineering. these were the bachelor’s degree courses chosen by the authors for analysis. in order to identify and analyze the subjects related to ethics and social sciences offered by these 17 universities in the four bachelor’s degree courses described above, the authors created a database containing the following variables for each subject: university where the course was offered, institute or faculty, course, name of the subject, whether it is obligatory or optional course, and syllabus. to search for the subjects, the authors used the curricular matrices available on the official websites of the faculties and institutes or the pedagogical political projects, depending on which document was more up-to-date. the subjects were searched for using the following keywords: ethics, society, social, sociology, diversity, human rights, humanism and humanities. the terms were chosen from the bibliography, resolution no. 5 of november 16, 2016, and a pre-analysis of the curricula. 3.4 categorization of the syllabuses after creating the categories for analysis, adapted from the literature review by [2], each syllabus was read and classified by at least two authors independently. both classifications were then checked, and, in cases of disagreement, the authors debated until a consensus was reached. 3.5 semi-structured interviews to complement the understanding of ethics teaching in the context of higher education, three semi-structured interviews were carried out, one with a professor of ethics in computer science, one with a professor of ethics in engineering, and one with a professor of ethics and head of a computer science department. in all cases, the interviewees were part of the teaching staff of the universities that make up the sample studied. 4 analysis of ethics-related subjects in computing courses this section presents two analyses. firstly, considerations on the survey of ethics-related subjects offered in the courses evaluated. these are quantitative assessments. in the next subsection, qualitative analyses are made. the authors define “ethics-related subjects” as the set of subjects identified using the selected keywords: ethics, society, social, sociology, diversity, human rights, humanism, and humanities. 4.1 descriptive analysis of ethics-related disciplines this section deals with the research analysis into the quantitative evaluation of ethics-related disciplines in the computer science, computer engineering, software engineering, and information systems courses of the 17 universities studied. to the assessment, quantity and the compulsory nature of those disciplines were considered. this analysis made it possible to understand the interviewees’ accounts of the ideological disputes in the field, which prioritize certain areas of knowledge to the detriment of others, and prompt a broader debate about the need for critical education of future ai professionals. although the research sample included 17 universities, not all have the four elected bachelor’s degree courses. table 2 below shows the relationship between the courses and the universities where they are offered. it is important to take this information into account when analyzing the number of ethics-related subjects below, since the number of disciplines was calculated in absolute numbers and not proportional to the number of courses. table 2. courses offered by each university. source: author’s elaboration, 2024.full size table based on a survey of ethics subjects and related topics (such as social sciences and human rights) in curricular matrices and political pedagogical plans, a total of 204 subjects were found. of these, 83 were part of the bachelor’s degree courses in computer science, 67 belonged to the computer engineering courses, 12 to the software engineering courses, and 42 to the information systems courses. as seen in fig. 2, utfpr is the university with the largest number of ethics-related disciplines. it offers all four degrees, having 55 disciplines, with computer engineering having the most (19) and software engineering the least (4). at the other end, unicamp and unesp have only one discipline each, and none was found at ufpr. fig. 2. number of disciplines related to ethics, per course and university. source: author’s elaboration, 2024. full size image the disciplines were classified as compulsory or optional/ elective/ complementary by analyzing the above-mentioned documents. the optional/ elective/ complementary disciplines were identified in the curricular matrices and political pedagogical plans and were considered as a single block here treated as “optional disciplines”. furthermore, as non-compulsory subjects can be elective (compulsorily offered by the university, regardless of the number of applicants) or optional (they may not be offered), it was not possible to determine their classification in the surveyed data and whether they have been, in fact, offered or not. in the interviews, the authors found out that there are registered disciplines in university systems that are not, and have never been, offered. as shown in fig. 3, 132 disciplines are optional, which represents a large majority compared to the 62 compulsory ones. thus, from the total of 204 ethics-related disciplines, only around 30.4% are obligatory, and it is up to the students individually to choose from the remaining 69.6%. fig. 3. number of obligatory vs. optional disciplines per university. source: author’s elaboration, 2024. full size image 4.2 analysis of the interviews the findings about the disparity between compulsory and optional subjects are aligned with the information from the interviews conducted with computer science and engineering professors. according to what was reported, disciplines related to ethics, social responsibility, and social issues linked to technology are usually the result of personal initiatives by teachers who are interested in the subject and who, often in a self-taught way, begin to study philosophy in particular, incorporating these themes into their disciplines. they usually do this informally in technical classes, discussing real cases involving ethical issues, for example. less often, they opt to formalize specific subjects on the topic. when asked about the legal resolution no. 5 of november 16, 2016, that makes humanistic teaching compulsory, the teachers and the head of department interviewed said they had no prior knowledge of it. in addition, the head of department said that, during a visit to inspect the course, the mec did not analyze compliance with the aforementioned resolution. in addition, the interviewees said that there was a certain amount of demise or even resistance from colleagues to the introduction of this kind of disciplines. according to them, there is an ideological dispute in the field about the essential nature of higher education for professionals in the field of computing. although not explicit, the neglect of ethics-related in favor of technical disciplines is based on a traditional and market-oriented discourse. as for student interest in ethical issues, the interviewees described the profile of engineering and computing students as more introspective and less participative in class. for one of the interviewees, this attitude is due to the context of what is valued by companies as well as a lack of interest in social impact. however, the other interviewees attributed this behavior to typical characteristics of the social group and a more traditional hierarchy between teachers and students that is ingrained in these bachelor’s degrees, and not necessarily due to the discipline itself. furthermore, one of the interviewees considered quota students to be more politicized and interested in discussions about ethics. 5 analysis of the syllabuses 5.1 creation of categories ethics teaching covers a wide range of concepts and has roots in various knowledge fields. reading the syllabuses of the collected subjects showed themes related to the fields of sociology, philosophy, and law. the categories used to analyze the syllabuses were adapted from the review by [2]. in this review, the articles were identified according to the presence or absence of an explicit description of the conception of ethics, and how these concepts appeared (exclusively or mixed) in them. [2] present three main conceptions of ethics found in the literature review: relative to philosophical frameworks, computing applications, and social impacts of computing. in general, the authors state that these conceptions used to mention “normative ethical theories or situate ethics in relation to concepts of computing, social impact and professionalism” [2] (p. 12). about the philosophical conception, [2] ethics is identified as a western philosophical theory, including issues such as consequentialism, virtue, and duty, as well as a discussion of decision-making processes. on the other hand, the articles that dealt with ethics from a computational perspective did so from debates close to the legal field, addressing issues of security, privacy, and responsibility, for example. the third approach focused on the relationship between technology design and its social impacts, as well as issues such as accessibility. as mentioned in the sect. 3 section, the selection of subjects from computing courses was broader than just those specific to ethics. sociological and humanistic subjects were also taken into account. for this methodological reason, an adapted version of the three conceptions of ethics by [2] was adopted. the computing applications and social impacts of computing categories were partly merged and partly fragmented. on the one hand, the issue of social impacts was expanded to cover not only the relationship between computing and society but also typical sociology topics that go beyond technology, such as social science theories, racism, gender inequalities and others. on the other hand, all the topics related to the profession of future computing professionals, including both legal and responsibility debates, were compiled as a second block. based on the inspiration provided by the work of [2] and the particular reading of the set of syllabuses in this research, three ways of classifying the issues linked to ethics and society found in computing subjects were drawn up: a) philosophical framework: syllabuses that refer to theories, concepts, thoughts, and thinkers from the philosophical tradition; epistemology of ethics; human values; principles of justice, truth, good, freedom, etc.; mention of the notions of consequentialism (ethics based on consequences); deontology (ethics based on duties), and ethics based on virtues. b) applied to computing: syllabuses that refer to ethics in relation to the concepts and areas of application of computing; everyday dilemmas faced by professionals; responsibility of professionals towards the community and society; ethics in decision-making; best practices and procedures; quality and security of computer systems; quality of scientific knowledge production; topics related to privacy, intellectual property, legislation, and codes of professional ethics. c) social issues: syllabuses introduce the concept of ethics through the foundations of sociological thought and address the social impacts of computing; reference to the social and historical character of the technological artifact; social repercussions and changes in how society is organized, such as in work, motivated by technology; the problem of reproducing inequalities, and stereotypes (of gender, race, class, sexuality, etc.); notions of risks and threats to the well-being of society; notions of citizenship. 5.2 systematization of syllabuses all the syllabuses found in the curricular matrices and political pedagogical projects of the four types of courses considered were analyzed based on the three categories mentioned in subsect. 5.1. this showed 180 non-repeated disciplines. it is important to note that, of these 180 subjects, 48 did not have their syllabus available. among those that did, there was a great disparity in the amount of information available. while some had detailed content in several descriptive paragraphs, others provided simplified generalizations in just one line. thus, although there was one example with more than 200 words, the average was 50 words. with this in mind, about the classification, each discipline was placed in one to three categories, depending on the syllabus. table 3 shows the following scenario: table 3. courses offered by each university. source: author’s elaboration, 2024.full size table as can be seen in table 3, all three categories have a significant presence in all disciplines. however, disciplines that deal with social issues are more frequent and those with a philosophical framework make up a smaller group. furthermore, as shown in table 4, the category “social issues” appeared more often alone than the others. when combined, it is usually linked to the “applied to computing” category. only 10 disciplines were categorized under “philosophical framework” and “applied to computing”, and only eight had all three approaches. table 4. classification. source: author’s elaboration, 2024.full size table 6 conclusions the emergence of generative ai and the widespread use of ai-based technologies have intensified the debate on ethical issues, both nationally and internationally. the establishment of professional ethical values through education is an important step towards mitigating the negative social impacts generated by the use of ai tools. the topic’s relevance is also demonstrated by the size of the field of computing, which in brazil offers thousands of courses and millions of bachelor’s degree places. for this reason, this article presents an introductory overview of ethics-related teaching in bachelor’s degree courses in computer science, computer engineering, software engineering, and information systems at the 17 main brazilian universities. based on the bibliography, an analysis of university documents and semi-structured interviews, the authors found an important disparity in the provision of ethics-related subjects/disciplines between universities. if, on the one hand, the federal technological university of paraná (utfpr) and the federal university of rio de janeiro (ufrj) were the only institutions to have more than 50 humanistic subjects on their courses, 13 universities out of the 17 evaluated offered less than 10 subjects of this type; the state university of campinas (unicamp) and the são paulo state university (unesp) with just one and the case at the federal university of paraná (ufpr) with none. a significant number of subjects had no syllabus, around 24%. however, among those that did, at least one reference to some philosophical framework, applications in computing or social issues was identified. as in the review work by [2] this does not mean that clear conceptualizations of ethics were found in the syllabuses. it just shows that, in terms of the delimitation suggested in this work, it was possible to analyze the distribution of references to fields of knowledge and to some sets of themes that the disciplines refer to in order to address the problem of ethics and social dilemmas in the context of computing education. the balanced distribution of categories suggests no predominance of one approach. one possible explanation is that, since there is still no “canon” for the teaching of ethics in the area, there is a plurality of approaches depending on the proposal of each subject. the semi-structured interviews, although only three, showed a tendency toward relative pessimism and frustration, with little belief in the possibility of greater academic space for such ethical discussions and devaluation of the subject in relation to technological aspects. the authors recognize the limitations of the data, which does not allow them to draw any major conclusions about the state of ethics teaching in brazil. this article was intended to record the first findings of a more extensive research agenda, which could expand both to a bibliographical analysis of the disciplines offered by universities and to an understanding of the way they are taught by lecturers and understood by students. in addition, it is proposed to deepen the gathering and interpretation of discourses on the importance of critical education in computing and its relationship with the development of ai for the social good. finally, encouraging scientific debate on the ethical education of future ai developers could stimulate the application of the resolution no. 5 of november 16, 2016 and suggest approaches to make this education effective. references anderson, r.e.: acm code of ethics and professional conduct. commun. acm 35(5), 94–99 (1992) article  math  google scholar  brown, n., xie, b., sarder, e., fiesler, c., wiese, e.s.: teaching ethics in computing: a systematic literature review of acm computer science education publications. acm trans. comput. educ. 24(1), 36 (2024). https://doi.org/10.1145/3634685 article  google scholar  câmara de educação superior, conselho nacional de educação, ministério da educação. resolução cne/ces 5/2016. diário oficial da união, brasília, 17 de novembro de 2016, seção 1, pp. 22–24 (2016) google scholar  fiesler, c., garrett, n., beard, n.: what do we teach when we teach tech ethics? a syllabi analysis. in: proceedings of the 51st acm technical symposium on computer science education (sigcse 2020), pp. 289–295. association for computing machinery, new york (2020). https://doi.org/10.1145/3328778.3366825 george, a.s., george, a.h.: a review of chatgpt ai’s impact on several business sectors. partn. univ. int. innov. j. 1(1), 9–23 (2023) math  google scholar  gorur, r., hoon, l., kowal, e.: computer science ethics education in australia – a work in progress. in: 2020 ieee international conference on teaching, assessment, and learning for engineering (tale), takamatsu, japan, 2020, pp. 945–947 (2020). https://doi.org/10.1109/tale48869.2020.9368375 hacker, p., engel, a., mauer, m.: regulating chatgpt and other large generative ai models. in: proceedings of the 2023 acm conference on fairness, accountability, and transparency, pp. 1112–1123 (2023) google scholar  masiero, p.c.: ética em computação. são paulo, edusp (2000) google scholar  organização das nações unidas para a educação, a ciência e a cultura. currículos de ia para a educação básica: um mapeamento de currículos de ia aprovados pelos governos. paris (2022). author proof humanities and ai: ethical education in technology careers 13 google scholar  organização das nações unidas para a educação, a ciência e a cultura. recomendações sobre a ética da inteligência artificial. tradução de tikinet edições ltda. paris (2022) google scholar  paltiel, m., cheong, m., coghlan, s., lederman, r.: approaches and models for teaching digital ethics in information systems courses – a review of the literature. australas. j. inf. syst. 27 (2023). https://doi.org/10.3127/ajis.v27i0.4517 “publicações científicas sobre inteligência artificial”. pesquisa fapesp, mai. (edição 291) (2020). https://revistapesquisa.fapesp.br/publicacoes-cientificassobre-inteligencia-artificial1/. accessed 10 may 2024 ranking de universidades. https://ruf.folha.uol.com.br/2023/ranking-deuniversidades/principal/. accessed 10 may 2024 saltz, j.s., dewar, n.i., heckman, r.: key concepts for a data science ethics curriculum. in: proceedings of the 49th acm technical symposium on computer science education (sigcse 2018), pp. 952–957. association for computing machinery, new york (2018). https://doi.org/10.1145/3159450.3159483 stahl, b.c., ess, c.m.: 20 years of ethicomp: time to celebrate? j. inf. commun. ethics soc. 13(3/4), 166–175 (2015) article  google scholar  stavrakakis, i., et al.: the teaching of computer ethics on computer science and related degree programmes. a european survey. int. j. ethics educ. 7, 101–129 (2022). https://doi.org/10.1007/s40889-021-00135-1 article  math  google scholar  university ranking. https://edurank.org/methodology/. accessed 28 aug 2024 university ranking. https://www.timeshighereducation.com/. accessed 28 aug 2024 download references acknowledgments the authors would like to thank the interviewees for their important insights. author information authors and affiliations josé luiz egydio setúbal foundation, av. angélica, 2071, são paulo, 01227-200, brazil laura simões camargo university of são paulo, r. da reitoria, 374, são paulo, 05508-220, brazil laura simões camargo, enio alterman blay & gabriela soares schmidt authors laura simões camargoview author publications search author on:pubmed google scholar enio alterman blayview author publications search author on:pubmed google scholar gabriela soares schmidtview author publications search author on:pubmed google scholar corresponding authors correspondence to laura simões camargo , enio alterman blay or gabriela soares schmidt . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper camargo, l.s., blay, e.a., schmidt, g.s. (2025). humanities and ai: ethical education in technology careers. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_6 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial intelligence ethics human rights computing education publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature when external knowledge does not aggregate in named entity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper when external knowledge does not aggregate in named entity recognition conference paper first online: 28 november 2021 pp 616–627 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) when external knowledge does not aggregate in named entity recognition download book pdf download book epub pedro ivo monteiro privatto10 & ivan rizzo guilherme10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1214 accesses abstract in the different areas of knowledge, textual data are important sources of information. this way, information extraction methods have been developed to identify and structure information present in textual documents. in particular there is the named entity recognition (ner) task, which consists of using methods to identify named entities, such as person, place, among others, in texts, using techniques from natural language processing and machine learning. recent works explored the use of external sources of knowledge to boost the machine learning models with sets of domain specific relevant information for the ner task. this work aims to evaluate the aggregation of external knowledge, in the form of gazetter and knowledge graphs, for ner task. our approach is composed of two steps: i) generation of embeddings, ii) definition and training of the machine learning methods. the experiments were conducted on four english datasets, and their results show that the applied strategies for external knowledge integration did not bring great gains to the models, as expressed by f1-score metric. in the performed experiments, there was an f1-score increase in 17 of the 32 cases where external knowledge was used, but in most cases the gains were lesser than 0.5% in f1-score. in some scenarios the aggregated external knowledge does not capture relevant content, thus not being necessarily beneficial to the methodology. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others prompt-based data augmentation framework for few-shot named entity recognition chapter © 2024 robustness of named entity recognition: case of latvian chapter © 2021 token is a mask: few-shot named entity recognition with pre-trained language models chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining and knowledge discovery expertise studies ontology knowledge management knowledge based systems 1 introduction with the evolution of digital technologies and the internet users profile, the generation of unstructured data had a huge increase. according to [22], unstructured data are those that do not show a clear syntactic and semantic, machine-processable structure. this way, methods have been developed to identify and structure information present in textual documents. in these methods, the data structuring process make use of natural language processing, an area that uses concepts of linguistics and artificial intelligence to process data and automate language related tasks. some of the natural language processing tasks that we can mention are machine translation, speech recognition, automatic text summarization, information extraction, among others. information extraction methods have been developed to identify and structure information present in textual documents. in particular there is the named entity recognition task, which consists of using methods to identify named entities, such as person, place, among others, in texts written in natural language and categorize them according to their nature. a named entity is a concept, formed by one or more words, that belongs to a previously defined semantic group [9,10,11, 25, 28, 33]. traditionally, ner approaches have used many techniques from linguistics, such as syntactic labels, word lemmas, affixes, among others, to extract information present in texts. the use of traditional techniques is quite laborious, as it involves several stages of data preparation. in order to ease the work of traditional ner approaches, machine learning methods have been developed. recently, numeric techniques that captures syntactic and semantic aspects of words has been gaining space for their similar results, sometimes even better, when compared to classical techniques, without requiring an extensive process of feature engineering. as examples of machine learning approaches to named entity recognition task we can cite [5, 6, 12, 14, 15, 26], among many others. distributed representations for words are some of the reasons for the popularization of machine learning methods in nlp, and as example of these representations there are word2vec [23, 24], glove [27], fasttext [3], flair [1], bert [7], and many others. as stated by ratinov and roth [29], ner is knowledge-intensive task. therefore, there are approaches that still perform feature engineering with machine learning methods in order to include features that cannot be captured by word vectors or texts. even though part of the problems are solved by feature engineering, there is still a semantic gap that cannot be solved with lexical and syntactic attributes alone. in this sense, semantic repositories have been adopted. in [5] external information is added to the cnn-lstm model through the use of a lexicon. this lexicon is made of wikipedia data and is used to add new features in the shape of inside-outside-beginning tags to the input vectors. a work using external knowledge from ontology is presented in [19], where the authors perform the named entity recognition on bridge inspection reports. in order to do this, they use a semi-supervised conditional random fields (crf) whose inputs consists of syntactic features, like stems and part of speech tags, and semantic features from the domain ontology, named bridgeonto. liu et al. [20] add external information from gazetteers, but instead of using tags indicating the pertinence of the word to one or more gazetteers, it uses hybrid semi-markov crf to generate a numeric value that express the degree of pertinence of the word to a gazetteer. the reason for this is to find a better alternative to the hard match representation, which is commonly used when gazetteers use is adopted. in [18] the authors present an approach using external knowledge through use of lexicons together with syntactic features like part of speech tags and n-grams. ratinov and roth [29] enrich the named entity recognition task by using a external knowledge in the form of gazetteers on conll03, muc7, and a smaller dataset of webpages assembled and annotated by them. they also use other features, such as context aggregation, extended prediction history, among others, to boost the ner performance of their model. seyler et al. [30] divide external features in four categories to quantify the impact of each one in the ner task, using a linear-chain crf on conll03 and muc7 datasets. the four categories are: i) knowledge agnostic using only local features; ii) name-based knowledge using a list of named entities; ii) knowledge-base-based knowledge using features extracted from a knowledge base or an annotated corpus; iv) entity-based knowledge using the results of a named entity disambiguation. authors show that incrementally adding more categories of knowledge yields better effectiveness, but sometimes at the cost of efficiency, stating that there is a trade-off between them. in order to add external knowledge to a neural model, ding et al. [8] propose the use of an adapted gated graph sequence neural network to capture the information contained in multiple gazetteers. the role of the gated graph sequence neural network is to serve as a embedding layer that learn how to combine the knowledge present in more than one gazetteer of the same or different type. the resulting embeddings are then fed to a standard bilstm-crf to fulfill the ner task on weibo-ner and ontonotes 4, both in chinese. the gazetteer knowledge is aggregated to an attentive neural network by lin et al. [17] for the nested named entity recognition task. they leverage the knowledge contained in gazetteers by finding a representation for entity candidates through what they call a gazetteer network, that is concatenated with the representation learned by a region encoder. the experiments show that this strategy improves the model performance on ace2005 dataset. xiaofeng et al. [32] propose a method to incorporate dictionary features to a bilstm-crf model in order to evaluate their impact. their differential is that the dictionaries used during the training phase are obtained from the train split data, whereas the dictionaries used in the testing phase are from senna. their experiments are conducted on conll2003 dataset, and show that the size of the dictionaries (partial or oversized) may lead to inferior results in some cases. the purpose of this work is to evaluate the aggregation of external knowledge, in form of gazetter and knowledge graphs from yago and freebase, for named entity recognition task using bilstm, bigru and crf. the paper is organized as follows: sect. 2 presents our approach, details of the models, the input vectors and the external knowledge used. in sect. 3 the experimental protocol and the datasets are explained, and the results are discussed. section 4 presents conclusions and future work. 2 approach this work aims to investigate the use of external knowledge in some commonly used neural models for ner. the first step of our approach is to generate the embeddings for the datasets utilized by the methods, and add the sources of external knowledge. after the first step, the next one is to define the neural networks, with the architecture inspired by [5]. this section introduces the neural models used, as well as the sources of the external knowledge and how these knowledge sources are added to our model. 2.1 neural models in this work we used two neural models: bidirectional long short-term memory (bilstm) combined with conditional random fields (crf) [6], and bidirectional gated recurrent units (bigru) combined with crf. the aim of the models is to find a sequence of labels \(\mathcal {y}=\{\mathcal {y}_1,\mathcal {y}_2,...,\mathcal {y}_n\}\) for a given sequence of inputs \(\mathcal {x}=\{\mathcal {x}_1,\mathcal {x}_2,...,\mathcal {x}_n\}\) of length n. in this work, \(\mathcal {x}_i\) are vector representations of each word and its features in a sentence. these vectors are used as input to the bilstm/bigru layers, and the purpose of these layers is to extract features and create a new feature vector for each word represented by \(\mathcal {x}_i\) while considering the surrounding words present in the same sentence. the idea of bidirectional layers is to use the same input vectors for two lstm/gru layers, one layer with the word sequence from left to right, generating \(\overrightarrow{h_i}\), and the other the sequence from right to left, generating \(\overleftarrow{h_i}\), which are concatenated into one feature vector, \(h_i = [\overrightarrow{h_i}: \overleftarrow{h_i}]\), that is used as input by the classification layer. following [14] and other works, we chose crf as the classification layer of our models because it takes into account the previously assigned labels. this way, combining bilstm/bigru with crf, the models make good use of the sequential characteristic of texts. 2.2 embeddings inspired by [5], this work make use of common embeddings: character embeddings, casing embeddings and word embeddings. further, we also aggregate the aim of validation: external knowledge embeddings. all these embedding techniques are explained below, with exception of the external knowledge embedding that is shown in the next subsection. character embeddings: to generate the character embeddings we first create a vector, randomly initialized with \(u(-0.5,0.5)\), for each character in a set of 135 characters. after initialization, these vectors are then retrieved for each character on each word and used as input to a convolutional neural network with max-pooling layer to generate a single vector with information from all the characters of a word, named character embedding. to better capture character features, character vectors are trained with the rest of the model, and we use a dropout layer to avoid character embeddings overfitting. casing embeddings: for the casing embeddings generation, each input word is categorized in one of eight possible categories according to their composition, such as their capitalisation and presence of numbers and special characters. then, each category is initialized as a one-hot vector that is further trained with the model. word embeddings: as word embeddings we decided to use pre-trained glove [27] word embeddings with 50 dimensions. other candidates were word2vec, fasttext, flair and bert. some of these maybe would yield better final results, but as the objective of the work is to analyze the impact of external knowledge in the chosen neural architecture, results below the state of the art do not invalidate this work. 2.3 external knowledge to aggregate external information to the input vectors, we chose to use two distinct sources: gazetteers made from version 3.1 of yet another great ontology (yago) [21]; and knowledge embeddings from freebase, generated by transe method [4] using openke framework [13] with the latest freebase dumpfootnote 1. to create the gazetteers we picked all entities referring to four types of entities in yago (person, organization, foundation, place), which correspond to three of conll2003 types (person, organization, place). besides the chosen types, we also picked their sub-types (e.g. abstract painters is a sub-type of people, presidents is a sub-type of people) by the end of this process, we had three gazetteer lists whose number of entities is shown in table 1. table 1. number of entities in each gazetteer list.full size table with the purpose of adding gazetteer information to the input vectors, a strategy similar to the casing embedding was used, but instead of the eight categories related to the composition of the word, we used another eight categories that express the pertinence of the word in one or more gazetteer. to do this, we did a tagging stage where each word of the dataset was tagged according to their pertinence to a gazetteer (e.g. washington received per/loc tag, kilimanjaro received only loc tag). then one-hot vectors are generated for each of the categories, which are then trained with the model, just as we did with casing embeddings. as for the knowledge embeddings from freebase, we chose to use only single-word topics, but to compensate it we did not filter the topics by types. this way we used the knowledge embeddings of all words in the text, and if the word doesn’t have a knowledge embedding we used a vector full of zeros. regardless of the source, the addition of external knowledge is done by concatenating the semantic feature vector, that was generated by one of the method described above, to the other vectors (word, casing, character) in order to present the resulting vector as input to the neural models used. 3 experiments we conducted experiments on four datasets in order to check the impact of external knowledge on chosen neural model for ner task. we executed each experimental setting a total of 10 times due to the stochastic elements present in each model initialization. for evaluation we decided to adopt the f1-score metric, shown in eq. 1 that is the harmonic mean of precision (p) and recall (r), shown in eq. 2 and eq. 3, where tp stands for true positive, fp for false positive, and fn for false negative. $$\begin{aligned} f_1-score = \frac{2 * p * r}{(p + r)} \end{aligned}$$ (1) $$\begin{aligned} p = \frac{tp}{tp + fp} \end{aligned}$$ (2) $$\begin{aligned} r = \frac{tp}{tp + fn} \end{aligned}$$ (3) so, the f1-score results presented in this section are the mean of 10 executions for each setting of the model. 3.1 datasets in this work we chose four distinct datasets, all of them in english, with different sizes and types of entities in order to validate our approach on different scenarios. the chosen dataset are conll2003 [31], ontonotes5, mit movies [18], and mit restaurants [18]. we only used train/test split, leaving validation sets out of our experiments, thus not conducting a hyper-parameter optimization. all of the datasets use inside-outside-beginning as entity annotation scheme. table 2 shows in details the number of sentences and entities contained in datasets. table 2. quantification of datasets.full size table 3.2 experimental settings the experiments consisted in using external embeddings, described on previous sections, to check their impact on the results of our models. to verify this, we conducted experiments in the same model with and without the external knowledge. the model without the external knowledge was used as baseline for our modifications, this way, we executed both network models, bilstm-crf and bigru-crf, with and without the addition of the external embeddings and compared their f1-score. the parameters used are shown in table 3. the parameters are the same for all datasets, except for ontonotes 5 dataset, which we decided to use a lower learning rate and fewer epochs due its large amount of data. table 3. parameters values. values with \(*\) symbol refer to parameters used only for ontonotes5.full size table 3.3 experimental results table 4 shows the results of the carried out experiments and the state-of-art (as stated by the authors in their papers) f1-score for the given dataset. the results show a an increase in all cases when we added gazetteer information to bilstm-crf. however, the addition of knowledge embedding to bilstm-crf decreased the f1-score on every situation. on the experimental results using bigru there are two scenarios of increase in f1-score with the addition of gazetteer information, and the use of knowledge embeddings also increased the f1-score in two scenarios. regardless of whether the external knowledge increase or decreased the f1-score, the difference was not significant, representing on average 0,4%, with exception of ontonotes 5 dataset, where the differences were very accentuated, averaging 6,76%. even though other works show an increase in f1-score when adding external knowledge to the models, our addition of external knowledge doesn’t always brings positive impacts on f1-score, as shown by table 4, and even when it does it not necessarily achieve the best result of our models. furthermore, even our best results are far from the state of the art in some datasets. table 4. comparison of f1-score between our models and state-of-art models, where gaz stands for gazetteer, and ke stands for knowledge embeddings. each column represents the f1-score for that dataset, and the bold values represent the best result for that dataset.full size table when compared to the results of other works, such as [29, 30] our strategy doesn’t seem to bring gains, however it is worth noting that the models we chose as baseline achieve good values of f1-measure (88.57% and 88.15% for conll2003 using bilstm and bigru, respectively), which are very close, respectively, to their best and second best results. this choice of a good baseline may be one of the reasons behind the small gains. 4 conclusion this work aimed to evaluate the use of external knowledge used in machine learning methods for named entity recognition. the methodology was composed of two steps: i) generation of embeddings, ii) definition and training of machine learning methods. the defined models were trained and tested in four english dataset of different sizes and different types of entities in order to evaluate our methodology. as our experiments show, in spite of an increase of f1-score in 17 of the 32 cases, the way external knowledge was integrated to the model did not bring much gains, most of them being lesser than 0.5%, and in some datasets the results were a way below the state-of-art methods (for the values stated by the authors in their papers). this may be explained because we haven’t made any hyper-parameter optimization, which may have led the model to suffer from overfit, and underfit in the case of ontonotes5, where there was an increase of 11.8% in one scenario. another point to consider in the gap between our results and the state-of-art is the choice of word embeddings: while our choice was to use glove word embeddings, most recent works use embeddings that better capture the context of words. it is important to note that we haven’t reproduced the state-of-art methods. although we did not achieve state-of-art results, we were still able to check the impact of the addition of external knowledge in the used neural models. the results and discussion show that the results of adding external knowledge are strongly linked to what information is used, as well as how it is used. we conjecture that in some cases the information present in external bases may be already integrated on the word representations, especially when the embeddings training set and the knowledge base share common data. this way, adding external knowledge to the models does not always improve the results, and can even lead to performance decreases. so, in order to integrate external knowledge, a deep analysis is needed to capture all the semantic present in external knowledge bases. notes 1.https://developers.google.com/freebase/. references akbik, a., blythe, d., vollgraf, r.: contextual string embeddings for sequence labeling. in: proceedings of the 27th international conference on computational linguistics, santa fe, new mexico, usa, pp. 1638–1649. association for computational linguistics (august 2018). https://www.aclweb.org/anthology/c18-1139 baevski, a., edunov, s., liu, y., zettlemoyer, l., auli, m.: cloze-driven pretraining of self-attention networks. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), hong kong, china, pp. 5360–5369. association for computational linguistics (november 2019). https://doi.org/10.18653/v1/d19-1539. https://www.aclweb.org/anthology/d19-1539 bojanowski, p., grave, e., joulin, a., mikolov, t.: enriching word vectors with subword information. trans. assoc. computat. linguist. 5, 135–146 (2017). https://doi.org/10.1162/tacl_a_00051. https://www.aclweb.org/anthology/q17-1010 bordes, a., usunier, n., garcia-durán, a., weston, j., yakhnenko, o.: translating embeddings for modeling multi-relational data. in: proceedings of the 26th international conference on neural information processing systems, nips 2013, vol. 2, pp. 2787–2795. curran associates inc., red hook (2013). https://doi.org/10.5555/2999792.2999923 chiu, j.p., nichols, e.: named entity recognition with bidirectional lstm-cnns. trans. assoc. comput. linguist. 4, 357–370 (2016). https://www.aclweb.org/anthology/q16-1026 collobert, r., weston, j., bottou, l., karlen, m., kavukcuoglu, k., kuksa, p.: natural language processing (almost) from scratch. j. mach. learn. res. 12, 2493–2537 (2011). https://doi.org/10.5555/1953048.2078186 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, minneapolis, minnesota, volume 1 (long and short papers), pp. 4171–4186. association for computational linguistics (june 2019). https://doi.org/10.18653/v1/n19-1423. https://www.aclweb.org/anthology/n19-1423 ding, r., xie, p., zhang, x., lu, w., li, l., si, l.: a neural multi-digraph model for chinese ner with gazetteers. in: proceedings of the 57th annual meeting of the association for computational linguistics, florence, italy, pp. 1462–1467. association for computational linguistics (july 2019). https://doi.org/10.18653/v1/p19-1141. https://aclanthology.org/p19-1141 freire, n., borbinha, j., calado, p.: an approach for named entity recognition in poorly structured data. in: simperl, e., cimiano, p., polleres, a., corcho, o., presutti, v. (eds.) eswc 2012. lncs, vol. 7295, pp. 718–732. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-30284-8_55 chapter  google scholar  gorinski, p.j., et al.: named entity recognition for electronic health records: a comparison of rule-based and machine learning approaches. corr abs/1903.03985. arxiv:1903.03985 (2019) goyal, a., gupta, v., kumar, m.: recent named entity recognition and classification techniques: a systematic review. comput. sci. rev. 29, 21–43 (2018). https://doi.org/10.1016/j.cosrev.2018.06.001 habibi, m., weber, l., neves, m., wiegandt, d.l., leser, u.: deep learning with word embeddings improves biomedical named entity recognition. bioinformatics 33(14), i37–i48 (2017). https://doi.org/10.1093/bioinformatics/btx228 han, x., et al.: openke: an open toolkit for knowledge embedding. in: proceedings of the 2018 conference on empirical methods in natural language processing: system demonstrations, brussels, belgium, pp. 139–144. association for computational linguistics (november 2018). https://doi.org/10.18653/v1/d18-2024. https://www.aclweb.org/anthology/d18-2024 lample, g., ballesteros, m., subramanian, s., kawakami, k., dyer, c.: neural architectures for named entity recognition. in: proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, san diego, california, pp. 260–270. association for computational linguistics (june 2016). https://doi.org/10.18653/v1/n16-1030. https://www.aclweb.org/anthology/n16-1030 lange, d., böhm, c., naumann, f.: extracting structured information from wikipedia articles to populate infoboxes. in: proceedings of the 19th acm international conference on information and knowledge management, cikm 2010, pp. 1661–1664. association for computing machinery, new york (2010). https://doi.org/10.1145/1871437.1871698 li, x., sun, x., meng, y., liang, j., wu, f., li, j.: dice loss for data-imbalanced nlp tasks. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 465–476. association for computational linguistics (july 2020). https://doi.org/10.18653/v1/2020.acl-main.45. https://www.aclweb.org/anthology/2020.acl-main.45 lin, h., lu, y., han, x., sun, l., dong, b., jiang, s.: gazetteer-enhanced attentive neural networks for named entity recognition. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), hong kong, china, pp. 6232–6237. association for computational linguistics (november 2019). https://doi.org/10.18653/v1/d19-1646. https://aclanthology.org/d19-1646 liu, j., pasupat, p., cyphers, s., glass, j.: asgard: a portable architecture for multilingual dialogue systems. in: 2013 ieee international conference on acoustics, speech and signal processing, pp. 8386–8390 (2013). https://doi.org/10.1109/icassp.2013.6639301 liu, k., el-gohary, n.: ontology-based semi-supervised conditional random fields for automated information extraction from bridge inspection reports. autom. constr. 81 (2017). https://doi.org/10.1016/j.autcon.2017.02.003 liu, t., yao, j.g., lin, c.y.: towards improving neural named entity recognition with gazetteers. in: proceedings of the 57th annual meeting of the association for computational linguistics, florence, italy, pp. 5301–5307. association for computational linguistics (july 2019). https://doi.org/10.18653/v1/p19-1524. https://www.aclweb.org/anthology/p19-1524 mahdisoltani, f., biega, j., suchanek, f.m.: yago3: a knowledge base from multilingual wikipedias (2013). https://hal-imt.archives-ouvertes.fr/hal-01699874/ manning, c.d., raghavan, p., schütze, h.: introduction to information retrieval. cambridge university press (2008). https://doi.org/10.1017/cbo9780511809071 mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space. in: bengio, y., lecun, y. (eds.) 1st international conference on learning representations, workshop track proceedings, iclr 2013, scottsdale, arizona, usa, 2–4 may 2013 (2013). arxiv:1301.3781 mikolov, t., sutskever, i., chen, k., corrado, g., dean, j.: distributed representations of words and phrases and their compositionality. corr abs/1310.4546. arxiv:1310.4546 (2013) nadeau, d., sekine, s.: a survey of named entity recognition and classification. lingvisticæ investigationes 30(1), 3–26 (2007). https://doi.org/10.1075/li.30.1.03nad. https://www.jbe-platform.com/content/journals/10.1075/li.30.1.03nad nurdin, a., maulidevi, n.u.: 5w1h information extraction with cnn-bidirectional lstm. j. phys. conf. ser. 978, 012078 (2018). https://doi.org/10.1088/1742-6596/978/1/012078 pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), doha, qatar, pp. 1532–1543. association for computational linguistics (october 2014). https://doi.org/10.3115/v1/d14-1162. https://www.aclweb.org/anthology/d14-1162 rais, m., lachkar, a., lachkar, a., ouatik, s.e.a.: a comparative study of biomedical named entity recognition methods based machine learning approach. in: 2014 3rd ieee international colloquium in information science and technology (cist), pp. 329–334 (october 2014). https://doi.org/10.1109/cist.2014.7016641 ratinov, l., roth, d.: design challenges and misconceptions in named entity recognition. in: proceedings of the 13th conference on computational natural language learning, conll 2009, usa, pp. 147–155. association for computational linguistics (2009) google scholar  seyler, d., dembelova, t., del corro, l., hoffart, j., weikum, g.: a study of the importance of external knowledge in the named entity recognition task. in: proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), melbourne, australia, pp. 241–246. association for computational linguistics (july 2018). https://doi.org/10.18653/v1/p18-2039. https://aclanthology.org/p18-2039 tjong kim sang, e.f., de meulder, f.: introduction to the conll-2003 shared task: language-independent named entity recognition. in: proceedings of the 7th conference on natural language learning at hlt-naacl 2003, conll 2003, usa, vol. 4, pp. 142–147. association for computational linguistics (2003). https://doi.org/10.3115/1119176.1119195 xiaofeng, m., wei, w., aiping, x.: incorporating token-level dictionary feature into neural model for named entity recognition. neurocomputing 375, 43–50 (2020). https://doi.org/10.1016/j.neucom.2019.09.005. https://www.sciencedirect.com/science/article/pii/s0925231219312652 yadav, v., bethard, s.: a survey on recent advances in named entity recognition from deep learning models. in: proceedings of the 27th international conference on computational linguistics, santa fe, new mexico, usa, pp. 2145–2158. association for computational linguistics (august 2018). https://www.aclweb.org/anthology/c18-1182 download references author information authors and affiliations institute of geosciences and exact sciences, unesp são paulo state university, rio claro, sp, brazil pedro ivo monteiro privatto & ivan rizzo guilherme authors pedro ivo monteiro privattoview author publications search author on:pubmed google scholar ivan rizzo guilhermeview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper privatto, p.i.m., guilherme, i.r. (2021). when external knowledge does not aggregate in named entity recognition. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_42 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_42 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords named entity recognition information extraction knowledge embeddings publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature embedding representations for automl pipelines | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper embedding representations for automl pipelines conference paper first online: 30 january 2025 pp 368–382 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) embedding representations for automl pipelines download book pdf download book epub camila santana braz  orcid: orcid.org/0000-0002-9238-53959, matheus cândido teixeira  orcid: orcid.org/0000-0003-1103-53839 & gisele lobo pappa  orcid: orcid.org/0000-0002-0349-44949  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 487 accesses abstract the area of automated machine learning (automl) emerged to automate the tedious process of manually testing different sets of algorithms hyperparameters and other data engineering tasks involved in the process of solving a machine learning (ml) problem. while many researchers focus on developing and refining techniques, there have been few advances in understanding the models and how the optimization works. one way to tackle this problem is to investigate the fitness landscape and analyze the distance between solutions to describe this environment. these techniques require calculating distances between solutions in the search space. this is a problem, as in a diverse range of automl methods, machine learning pipelines are represented by a tree structure, which has limitations in computational time for calculating distances and does not account for the semantics for the solutions. in this direction, this paper proposes a new way to represent ml pipelines using embeddings. we use a transformer model to generate embeddings of machine learning pipelines, and then evaluated the embeddings using the correlation between the distances calculated when using the two representations. we also perform a qualitative and a visual analyses to compare both representations. developing this representation allows researchers to improve current automl methods by providing a better understanding of how difficult it is to search for them. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others incremental search space construction for machine learning pipeline synthesis chapter © 2021 a literature review on automated machine learning article open access 11 november 2025 machine learning—automated machine learning (automl) for disease prediction chapter © 2023 1 introduction the area of automated machine learning (automl) emerged to automate the tedious process of manually testing different sets of algorithms hyperparameters and other data engineering tasks involved in the process of solving a machine learning (ml) problem. automl methods reduce or eliminate the need of specialized human intervention in building and tuning models. given its importance, nowadays all major tech companies like google, microsoft, and amazon offer automl techniques on their platforms. an ml pipeline is defined as a set of ml operations that include data preprocessing, data classification and post-processing steps, which can be executed sequentially or in parallel. most automl tasks are tackled as an optimization problem, with a search space that includes a range of algorithms and hyperparameters used to build machine learning task pipelines. the search method optimizes a metric of learning quality, such as the accuracy or f-measure [11]. different optimization methods have been used in automl, including bayesian optimization, evolutionary algorithms [5] and hybrids techniques, but few have looked at how difficult this optimization problem is. one way to measure problem difficulty is to look at the fitness landscape of a problem [12]. the fitness landscape is defined by the set of viable solutions within the search space associated with the quality metric being optimized. investigating the fitness landscape is important because it contributes to understanding the space from an optimization perspective, such as whether it has multiple global/local optima, saddle points, plateaus, or regions of low variability. understanding these characteristics of the space leads to the construction of more effective automl methods, allowing for a more informed exploration of the space. the analysis of fitness landscapes requires calculating distances between solutions to determine neighbourhoods of solutions with, for example, the same quality (plateaus) or peaks that may contain global or local optima, in the case of maximization problems. calculating these distances will depend on how solutions are represented. the current canonical representation of an automl pipeline is a tree. this representation is used because it suits the complex and hierarchical search spaces of automl [9]. however, this model presents two important limitations: (i) high computational complexity to calculate distance between solutions and (ii) difficulty in considering the semantic aspects of the solutions. with these drawbacks in mind, this work proposes to represent these pipelines as embeddings. an embedding is essentially a numerical vector, generated, in our case, from training a neural network with data from a range of ml pipelines. the network is able to capture latent features of the original space and map it to the weights of the network, which can be later used to present these pipelines. calculating the distance between embeddings is simple and has low computational complexity. in addition, embeddings have the capability to preserve the semantic aspects of the data they represent, which can be observed when they are plotted in spaces of a lower number of dimensions. being able to plot ml pipelines in 2d spaces also allows for the visualization of both the path taken during optimization and the distance between different model configurations, facilitating the analysis of these aspects and, consequently, promoting a more efficient exploration of the search space. finally, many state-of-art works in machine learning have been using embeddings representations of spaces, suggesting them as a promising representation solution [15] [6]. inspired by the work of [3] – which developed an embedding representation for symbolic regression trees and showed them to be effective to measure the distance between trees and its potential to capture semantics – this paper proposes to generate embeddings to represent automl pipelines. the embeddings generated are evaluated by comparing distance metrics of different representations. moreover, visual and qualitative analysis of the search space and pipelines distance are performed to better asses the proposed representation. the main contributions of this study are: comparison of two models to represent automl pipelines representations: tree and embeddings; analysis and evaluation of the use of this linear representation in the context of automl; investigation of semantic preservation in the representation through embeddings; development of a method for visualizing the search spaces. in the long term, this research may contribute to the development of more robust, effective and efficient automl models, starting from the analysis of fitness landscapes. this work is organized as follows. section 2 reviews relevant literature, positioning the research within the state-of-the-art . section 3 outlines the methodology, detailing the techniques, metrics, and tools used to achieve the objectives. section 4 discusses the results, analyzing the effectiveness of embeddings as representations of automl pipelines and visualizations of their search spaces. finally, sect. 5 presents the conclusions from the experiments and future work directions. 2 related work a few studies have proposed new representations for automl pipelines and different visualization tools to better understand how small changes affect the fitness landscape. concerning pipeline representation, research in [10] employs deep learning techniques in embeddings to optimize automl pipelines. the method involves creating latent representations for pipeline configurations through deeppipe, which uses bayesian optimization to find ideal configurations. the system improves its generalization and prediction capabilities through meta-learning. experiments on three meta-datasets show state-of-the-art results, surpassing existing methods like oboe, smac and autoprognosis. with a different objective, the approach in [4] integrates nlp techniques with automl methods. the authors use embeddings to represent metadata from datasets and algorithm documentation, allowing automl systems to recommend machine learning pipelines based on textual descriptions. their system increases pipeline optimization efficiency and shows significant improvements over frameworks like oboe, autosklearn, alphad3m and tpot, recommending solutions in under a second without prior runs. they also made their data, models and code publicly available. turning to visualization techniques, study [8] presents pipelineprofiler, an interactive visualization tool for exploring and comparing machine learning pipelines generated by different automl systems. integrated into jupyter notebook, it facilitates pipeline analysis, including hyperparameters and evaluation metrics, and helps identify patterns that lead to better results. in [16], autoaiviz is presented as an interactive visualization tool that uses conditional parallel coordinates (cpc) to explore model and pipeline generation steps. cpc visualization helps users understand how automl decisions impact fitness. a usability study showed the tool’s effectiveness in increasing the comprehensibility of automl processes, highlighting the importance of transparency in automl systems. the authors in [14] introduce atmseer, a visualization tool that enhances transparency and controllability in automl processes. it allows users to refine the search space and analyze results interactively. atmseer offers multi-level visualizations, enabling real-time monitoring and modifications. the methodology proposed in this paper can be coupled to any of the visualization tools aforementioned to represent ml pipelines in a 2d space. it can also help improve search by using informed decisions based on the shape of the search space. 3 methodology the main objective of this paper is, given a set of machine learning pipelines represented as trees, map these trees to a new representation space, where the pipelines are represented as embeddings. the embeddings are generated by training a neural network, and then extracting the weights of the final encoder layer. the proposed methodology can be divided into 3 steps: (i) generate a training set, which involves preparing machine learning pipelines to be given as input to neural networks; (ii) configure and train the neural network; (iii) given a new tree, extract the embedding from the network (fig. 1). fig. 1. methodology steps to generate the pipeline represented as an embedding. full size image 3.1 generating the training set as previously explained, an ml pipeline includes preprocessing steps (data cleaning, feature selection), a machine learning model (classification or regression) and post-processing steps. the search space used in this paper was borrowed from [9], where pipelines were generated by following a context-free grammar with 38 production rules, 92 terminals, and 45 non-terminals. these include three dimensionality reduction algorithms (pca, select k-best, standard scaler) and five classification methods (logistic regression, mlp, knn, random forest, adaboost), with parameter counts ranging from two to seven. default values from scikit-​learn were used for some continuous parameters while relative values were employed for parameters that depend on the number of attributes. this search space was chosen because it has size 69,960, allowing all solutions to be enumerated. this facilitates testing a new representation methodology, as we know which solutions should be more similar to each other. these generated pipelines are initially structured as a tree. they must be converted into a linear, non-hierarchical format of fixed size to be used as input for the neural network. a depth-first search (dfs) is performed on the graph structure to investigate the connected nodes and leaves [3], and a grammar-based parser generates a token sequence representing the original pipeline in the appropriate format. 3.2 neural network configuration the neural network purpose is to learn a function to encode the linearized pipeline string into an embedding by mapping each token of this string into an embedding space. the trained neural network is a transformer [13], an architecture developed for both supervised and unsupervised machine learning tasks, primarily applied to natural language processing (nlp) to capture semantics in large datasets. transformers revolutionized nlp by implementing self-attention mechanisms, allowing the encoder to extract information from the entire input sequence and enabling the decoder to give more importance to specific elements of the input, depending on the token being processed at the moment. with multiple self-attention heads, it is possible to map various pieces of information to a single word, allowing each position in the input sequence to attribute importance to all others, improving the understanding of context and semantic relationships. table 1 presents the transformer parameters and tested values. two tree representation strategies will be examined: a detailed one with tags indicating algorithm types (i.e., pca points to <features_dim>, <whiten> and <svd_solver> and these nodes points to the values of each parameter ) and a straightforward one with only algorithm names and parameters (i.e., pca points directly to the values of each parameter). for the first one, the grammar vocabulary is comprised of a total of 137 tokens. in the second, 69 tokens are included. for both of them three of the tokens are for specific scenarios: <pad_token> (used for padding sequences to the same length), <unk_token> (represents unknown words or parameters) and <sos_token> (marks the start of a sequence). table 1. transformer parameters.full size table finally, the loss function measures the difference between the model’s predictions and the actual tokens and the labelsmoothing strategy helps make the model less confident and more general. the loss is normalized by the number of non-padding tokens to ensure fair evaluation across sequences of varying lengths. 3.3 extracting embeddings from network layers the encoding function learned by the transformer in the previous step generates embeddings for each token in the input sequence. these embeddings are extracted from the final encoder layer, after passing through the attention and feed-forward sublayers and undergoing normalization. to create the complete tree embedding, it is necessary to aggregate the individual mapped tokens to obtain the full tree representation. this can be done using three aggregation functions: sum: summing the value of each dimension of the tokens across all tokens and dimensions. mean: calculating the mean value of each dimension of the tokens across all tokens and dimensions. concat: concatenating all dimensions of the tokens across all tokens and dimensions. the three aggregation functions proposed above will be tested to determine the most appropriate one. 4 generating tree embeddings given the proposed methodology, the next step was to train the proposed transformer to generate tree representations. these datasets were generated from 3 datasets, listed in table 2. they are available on kaggle and the uci machine learning repository, and were selected considering their number of attributes and number of classes. table 2. datasets characteristics.full size table for each dataset, the trees previously defined in sect. 3.1 are used as input to train the transformer. the only differences in tree representation are in the parameters that depend on the input variables, such as the number of features in the selectkbest algorithm. for this type of parameter, relative values were set. having the datasets, we tuned the number of embedding dimensions (16, 32, 64, 128 and 256) and learning rate (0.001 and 0.0001), while other parameters were kept with their default values presented in table 3. the network was trained for up to 50 epochs, with an early stopping mechanism. if the loss did not decrease more than a predefined threshold of \(1e-8\) for five consecutive epochs, it was considered that the model had converged, and training was stopped. table 3 presents the configuration and test set loss for each trained neural network. we ran each model five times to generate the variance and confidence interval. table 3. transformer configuration and test loss for each model.full size table figure 2 shows the evolution per epoch of the loss function of each model that resulted from the best configuration. the training loss and the test loss decrease rapidly within the first few epochs, which indicates that the model is learning quickly and effectively adjusting its parameters early in the training process. fig. 2. learning curve. full size image 5 contrasting tree-based and embedded representations one of the difficulties of the proposed methodology is how to evaluate the embedded representations when compared to the tree representation. we start by looking at the correlations between the distances considering trees and embedding representations. the traditional way to calculate distances between tree representations is to use the tree edit distance, defined in equation (1): $$\begin{aligned} d(t_1, t_2) = {\left\{ \begin{array}{ll} 0 & \text {if } t_1 = t_2\\ \min {\left\{ \begin{array}{ll} d(t_1 l_1, t_2 l_2) + \text {cost}(l_1, l_2)\\ d(t_1 l_1, t_2)\\ d(t_1, t_2 l_2)\\ \end{array}\right. } & \text {otherwise.} \end{array}\right. } \end{aligned}$$ (1) the distances between the embeddings will be measured using two measures: cosine distance (1 the cosine similarity) and euclidean distance. the goal is to compare whether the embeddings are suitable representations for the pipelines, evaluating the pearson and spearman correlation between the two representation distances. by calculating the correlation between these distances, evidence of the suitability of using embeddings for automl can be obtained. additionally, as the embeddings are encoded in high-dimensional spaces, we use umap  [7] to reduce their dimensionality and plot the solutions in a two-dimensional space, allowing the visualization of the search space and a qualitative analysis of the relationships between pipelines. both are reported in the next section. to calculate the distances between solutions using tree edit distance metrics, euclidean distance and cosine distance, five samples with 5000 trees each were collected through random and stratified sampling to ensure diversity and representativeness in the sampled population. the use of samples is justified by the complexity of calculating tree edit distances [1, 2]. figures 3 and 4 present the results of spearman correlations between edit tree distance and embedding distances. the pearson correlations are omitted due to space restrictions, but spearman presented better results. this indicates that the relation between the tree edition and the other metrics might be monotonic but not necessarily linear. results showed a strong correlation between embedding distances and tree edit distances, especially in models without tags and using "sum" as the aggregation function. euclidean distances consistently outperformed cosine distances. the ml-prove and raisin datasets exhibited the highest correlations, ranging from 0.73 to 0.79, with the ml-prove model (64 dimensions, learning rate 0.0001) performing better. the mushrooms dataset showed lower correlations but still reached values between 0.66 and 0.71 in certain configurations. in summary: spearman correlation with euclidean distance: spearman correlations were slightly higher, with no-tag models using mean and sum functions achieving correlations up to 0.78. the mushrooms dataset showed significantly lower correlations. spearman correlation with cosine distance: results were slightly inferior to euclidean distance, with maximum correlations of 0.79 for no-tag models in ml-prove. the mushrooms dataset also showed lower correlations. fig. 3. average across samples of spearman correlations for euclidean distance. full size image fig. 4. average across samples of spearman correlations for cosine distance. full size image 6 visual analysis of the search space as previously mentioned, the embeddings also underwent a dimensionality reduction process to enable plotting them on a cartesian plane and generating a coordinate chart. umap was trained using a greedy algorithm that optimized the number of neighbours, minimum distance and distance metric. based on the correlation results, plots generated with the aggregation functions mean and sum in the no-tag strategy are plotted. additionally, the datasets ml-prove and mushrooms were selected for visualizations as they demonstrated the best and worst performance, respectively. the objective is to investigate how visualizations occur in datasets with different performance levels, understanding the potential of semantic preservation in scenarios of high correlations (0.78) and lower correlations (0.66). in the coordinate plot in fig. 5, each point represents a pipeline, and the set of points is the search space. since embeddings have the potential to preserve the semantic aspects of the data they represent, similar pipelines should be closer together in this space, while different pipelines should be farther apart. the following analysis highlights this potential for semantics preservation, as pipelines with similar algorithms and hyperparameters are grouped closely together. figure 5a shows the visualization of the ml-prove search space. note that similar algorithms were clustered together. random forest and k-nearest neighbors display a clear boundary of separation in the search space, while the other algorithms mlp, logistic regression and ada boost are concentrated in the lower right region of the search space. furthermore, a cluster for each of these algorithms is identifiable, although logistic regression is slightly more spread out than the other two. since the search space is predominantly composed of random forest and k-nearest neighbors, it is consistent with the results that the transformer has acquired more information about these algorithms to classify them with higher accuracy. figure 5b presents the search space of the mushrooms dataset. although this model showed the worst correlation results, an analysis of the images reveals an acceptable algorithm clustering, with the locations of each algorithm in the space being easily visualized. again, random forest and k-nearest neighbors occupy most of the space and are in the most identifiable regions. however, the model also managed to group the other classification algorithms into perceptible regions. comparing this visualization with the previous one, it is noteworthy that this model was better at grouping identical algorithms into smaller more concise regions, revealing similarity by both algorithm and parameters. fig. 5. search space by classification algorithm. full size image the search space divided by preprocessing algorithms is presented in fig. 6. it is worth highlighting that this type of algorithm was not the best choice to primarily cluster the pipelines but can still be useful if analyzed together with the plot showing classification algorithm 5. as observed with the classification algorithms, the mushrooms model better separated the pipelines into groups. fig. 6. search space grouped by preprocessing algorithm. full size image 6.1 investigating clustering two types of clustering are worth investigating: regions where the model incorrectly grouped the pipelines, such as the lower right region in the ml-prove dataset, and regions well-separated, such as the clustering observed in the mushrooms search space. when analyzing fig. 5a, observe that the logistic regression, mlp, and adaboost algorithms appear in locations where they should not be, mixed with the knn region in the lower right part of the search space. by analyzing fig. 6a, notice that the preprocessing algorithm also fails to differentiate the pipelines. although the clustering is not pure, the model still managed to separate the groups to some extent. one possible reason for the poor grouping of these three classification algorithms could be the number of pipelines that use them. with fewer training data on these three, the model has less information and does not learn as well how to classify them. another important investigation is to analyze the composition of clusters. it is expected that subgroups are composed of pipelines with similar algorithms and hyperparameters. figure 7a provides a closer look at a region within one of the random forest clusters. in this figure, we observe sub-clusters with similar preprocessing parameters and algorithms: preprocessing algorithm: pca with 8, 15 and 21 principal components; classification algorithm: random forest with entropy as criterion and sqrt as max_features hyperparameters. figure 7b presents another group with similar hyperparameters: preprocessing algorithm: standar scaler; classification algorithm: random forest with gini as criterion and sqrt as max_features hyperparameters fig. 7. search space by preprocessing algorithm. full size image 6.2 qualitative analysis finally, to provide a qualitative analysis of the embedding distance, a set of four pipelines from the best and worst-performing models as done in the previous subsection were selected for this experiment. the goal is to choose a pair of pipelines that appear similar and a pair of very different pipelines and then investigate their distances and locations in the search space. table 4 presents these results for the ml-prove dataset. note that the similar pairs are closer than the different ones, which is expected. the similar pairs use the same preprocessing algorithm with different parameters and the same classification algorithm with different parameters. table 4. qualitative analysis for ml-provefull size table for the mushrooms dataset, results are shown in table 5. as expected, the similar pair is closer than the different one. the comparison between the sum without the tag model and the mean with the tag model indicates that the former better captures the structures and semantics of the pipelines, as evidenced by the greater distance between the pipelines. nevertheless, the second model can satisfactorily identify distance relationships. table 5. qualitative analysis for mushrooms.full size table figure 8 illustrates the positions of each pair of pipelines. the points were jittered to disperse overlapping regions. the visual analysis suggests that umap successfully preserved the high-dimensional structure and mapped it into a two-dimensional space. however, it is worth noting that the euclidean distance between the worst and best pipelines is greater in the ml-prove dataset than in the mushrooms dataset, yet, in the plot, it appears to be smaller. fig. 8. location of each pair of pipeline. stars represent similar pipelines and circles mark different ones. full size image 7 conclusion this paper developed and evaluated a novel linear representation for automl pipelines. by transforming traditional tree-based representations into embeddings, we addressed the challenges of computational complexity to calculate distances between solutions and semantic preservation inherent in the canonical methods. the experimental results indicate that different embedding models exhibit strong correlations with traditional tree edit distances, suggesting that embeddings can effectively capture the underlying relationships between different pipeline configurations. the visualizations of the search space validate the potential of embeddings as a suitable representation for pipelines. analysis of the plots reveals distinct algorithm regions and clusters and qualitative analysis shows that the embeddings’ distances align with the apparent pipeline distances. future work will focus on further studies regarding embeddings, such as investigating the preservation of local optima networks (lons) and using neighborhood plots to explore these relationships more deeply. overall, this research contributes to the advancement of automl by proposing a robust, efficient, and semantically meaningful representation for pipelines, enhancing the explicability of the optimization process and paving the way for more effective and transparent automl solutions. references bille, p.: a survey on tree edit distance and related problems. theoret. comput. sci. 337(1–3), 217–239 (2005) article  mathscinet  math  google scholar  bringmann, k., gawrychowski, p., mozes, s., weimann, o.: tree edit distance cannot be computed in strongly subcubic time (unless apsp can). acm trans. algorithms (talg) 16(4), 1–22 (2020) article  mathscinet  math  google scholar  caetano, v., teixeira, m.c., pappa, g.l.: symbolic regression trees as embedded representations. in: proceedings of the genetic and evolutionary computation conference, pp. 411–419 (2023) google scholar  drori, i., liu, l., nian, y., koorathota, s.c., li, j.s., moretti, a.k., freire, j., udell, m.: automl using metadata language embeddings. arxiv preprint arxiv:1910.03698 (2019) hutter, f., kotthoff, l., vanschoren, j.: automated machine learning: methods, systems, challenges. springer nature (2019) google scholar  kim, h., park, m., cho, s.: learning representations for medical images with deep embeddings. med. image anal. 78, 102343 (2023) math  google scholar  mcinnes, l., healy, j., melville, j.: umap: uniform manifold approximation and projection for dimension reduction. arxiv preprint arxiv:1802.03426 (2018) piazentin ono, j., castelo, s., lopez, r., bertini, e., freire, j., silva, c.: pipelineprofiler: a visual analytics tool for the exploration of automl pipelines. arxiv e-prints pp. arxiv–2005 (2020) google scholar  pimenta, c.g., de sá, a.g., ochoa, g., pappa, g.l.: fitness landscape analysis of automated machine learning search spaces. threshold 10(6), 103 google scholar  pineda arango, s., grabocka, j.: deep pipeline embeddings for automl. in: proceedings of the 29th acm sigkdd conference on knowledge discovery and data mining, pp. 1907–1919 (2023) google scholar  smith, j., brown, k., williams, d.: automated machine learning: hyperparameter optimization in practice. j. artif. intell. res. 65, 101–120 (2023) math  google scholar  stadler, p.f.: fitness landscapes. in: biological evolution and statistical physics, pp. 183–204. springer (2002) google scholar  vaswani, a., et al.: attention is all you need. advances in neural information processing systems 30 (2017) google scholar  wang, q., ming, y., jin, z., shen, q., liu, d., smith, m.j., veeramachaneni, k., qu, h.: atmseer: increasing transparency and controllability in automated machine learning. in: proceedings of the 2019 chi conference on human factors in computing systems, pp. 1–12 (2019) google scholar  wang, y., liu, s., yang, j.: embedding-based approaches for recommender systems: a survey. ieee trans. knowl. data eng. 35(4), 1234–1248 (2023) math  google scholar  weidele, d.k.i., et al.: autoaiviz: opening the blackbox of automated artificial intelligence with conditional parallel coordinates. in: proceedings of the 25th international conference on intelligent user interfaces, pp. 308–312 (2020) google scholar  download references acknowledgments a bold run-in heading in small font size at the end of the paper is used for general acknowledgements, for example: this study was funded by x (grant number y). the authors have no competing interests to declare that are relevant to the content of this article. author information authors and affiliations computational science department, federal university of minas gerais, belo horizonte, brazil camila santana braz, matheus cândido teixeira & gisele lobo pappa authors camila santana brazview author publications search author on:pubmed google scholar matheus cândido teixeiraview author publications search author on:pubmed google scholar gisele lobo pappaview author publications search author on:pubmed google scholar corresponding author correspondence to matheus cândido teixeira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper braz, c.s., teixeira, m.c., pappa, g.l. (2025). embedding representations for automl pipelines. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_26 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning automl embedding space machine learning pipelines publish with us policies and ethics profiles matheus cândido teixeira view author profile gisele lobo pappa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature named entity recognition for brazilian portuguese product titles | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper named entity recognition for brazilian portuguese product titles conference paper first online: 28 november 2021 pp 526–541 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) named entity recognition for brazilian portuguese product titles download book pdf download book epub diego f. silva10, alcides m. e silva10, bianca m. lopes10, karina m. johansson10, fernanda m. assi10, júlia t. c. de jesus10, reynold n. mazo10, daniel lucrédio10, helena m. caseli10 & … livy real11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1280 accesses 2 citations abstract improving the interaction between consumers and marketplaces, focusing on reaching higher conversion rates is one of the main goals of e-commerce companies. offering better results for user queries is mandatory to improve user experience and convert it into purchases. this paper investigates how named entity recognition can extract relevant attributes from product titles to derive better filters for user queries. we conducted several experiments based on mitie and bert applied to smartphones/cellphones product titles from the largest brazilian retail e-commerce. both of our strategies achieve outstanding results with a general f1 score of around 95%. we concluded that using a classical machine learning pipeline is still more useful than relying on large pre-trained language models, considering the model’s throughput and efficiency. future work may focus on evaluating the scalability and reusability capacity of both approaches. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others mining user’s data based on customer’s rating for prediction and recommendation—a comparative analysis chapter © 2020 an evaluation of nerc learning-based approaches to discover personal data in brazilian portuguese documents article open access 30 november 2023 using machine learning and nlp for the product matching problem chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization e-business e-commerce and e-business escrt machine learning object recognition 1 introduction improving the interaction between consumers and marketplaces, focusing on reaching higher conversion rates is one of the most significant challenges faced by e-commerce companies. in other words, the primary goal in this scenario is to guide the users to purchase the items they are searching in the platform. therefore, due to the increasing specificity of search, the shopping experience becomes a bottleneck in this scenario, even for big e-commerce companies. e-commerce shopping becomes a more daily activity than ever. in the pandemic scenario, brazilian e-commerces saw an increase of 47% of purchases, according to ebit|nelson 2020 reportfootnote 1. besides, the product assortment, particularly for marketplaces, is getting more complicated since e-commerce is replacing traditional retail settings, like groceries and health care supply. a traditional marketplace strategy to improve product assortment is to introduce more sellers into the marketplace platform. on the one hand, it makes the consumers’ lives easier, since they can acquire their needs in only one platform. on the other hand, it makes the product assortment to be increasingly non-uniform. product titles broadly vary, either by their sellers or their nature/category. consider the following examples from our data to illustrate these problems: 1. smartphone samsung galaxy a01, 32 gb, 2 gb ram, tela infinita de 5.7\(^{\prime \prime }\) , câmera dupla traseira 13mp (principal) + 2mp (profundidade), frontal de 5mp, bateria de 3000 mah, dual chip, android azulfootnote 2 2. smartphone moto g9 play 64 gb dual chip android 10 tela 6.5\(^{\prime \prime }\) qualcomm snapdragon 4g câmera 48mp+2mp+ 2mp verde turquesafootnote 3 3. iphone se 128 gb preto ios 4g wi-fi tela 4.7\(^{\prime \prime }\) câmera 12mp + 7mp applefootnote 4 the first title offers the following information: witfootnote 5 (smartphone); brand (samsung); model (galaxy a01); internal memory (32 gb); display size (5.7\(^{\prime \prime }\)); quality of the cameras (13mp and 5mp); sim card capacity (dual chips) and color (azul – “blue” in brazilian portuguese). in the second example, the brand name is not explicit, but we can find the operational system (android 10) and processor (qualcomm snapdragon) specifications. the last example shows: brand (apple); model (se); internal memory (128 gb); color (preto – “black” in portuguese); display size (4,7\(^{\prime \prime }\)) and quality of the cameras (12mp and 7mp). from these examples, we can see that the titles from a quite common category (smartphone) do not follow a standard structure and are not even offering the same information. as a consequence, it is very difficult to specify standard search filters that could fit any title or query in this category. in this scenario, increasing the products’ retrieval becomes a difficult challenge. a syntax-based search engine is not enough to make all the assortment accessible to the customer. commonly, e-commerces web pages offer facets to filter customer queries. it becomes a critical task for offering useful filters, having a uniform, vast, and high-quality product attribute extraction in place [11]. hence, motivated by improving the human interaction with our e-commerce platform, we investigated how named entity recognition could be applied to identify attributes in product titles of the smartphone categoryfootnote 6. we investigated two approaches: mitiefootnote 7 and bert [6]. from our experiments we found that models trained with mitie and bert had very similar results, achieving around 95% in f1 scores. mitie was chosen as the best one for our scenario due to other advantages such as its prediction latency and required infra-structure. thus, the main contributions of this work are: the description of an end-to-end approach of corpus annotation, model building and evaluation for attribute-value pairs extraction from product titles; the careful comparison of two state-of-the-art models for named entity recognition – mitie and bert – in a not well explored domain (e/commerce); the addressing of a poorly explored scenario of e-commerce language for brazilian portuguese. this paper is organized as follows. in sect. 2 we describe the task of named entity recognition and the approaches proposed to perform it. in sect. 3 we describe the set of attribute products that we found relevant for our problem. the experiments and results are presented in sect. 4 and sect. 5 finishes the paper with some final remarks. 2 background and related work named entity recognition. (ner) is a subtask of natural language processing (nlp) or information extraction (ie) that tries to identify and classify “named entities” present in unstructured text. named entities are well-known predefined categories such as people’s names (e.g., bill), organizations (e.g., united nations), locations (e.g., brazil), time expressions (e.g., yesterday), quantities (e.g., liter), monetary values (e.g., dollar), among others. in the scope of e-commerce, our named entities will be renamed to attributes, and the attributes that are of our interest will be specified in sect. 3. as noted by more [11], extracting attributes from product titles in e/commerce is a difficult task since titles, in this context, lack syntactic structure. for this purpose, the author proposes a ner-based method and demonstrates results on the particular attribute “brand”. more’s proposal consists of combining structured perceptron and conditional random fields to predict bio (b = begin, i = in, o = out) tags in product titles in english. the author used an automatically labeled (by regex) dataset to train the models and increased its volume and quality by a post-processing step, which includes augmenting a normalization dictionary and manual feedback from domain experts. the author reports a f1-score of 92% for the attribute “brand”. xu et al. [18] proposed an approach to support value extraction, scaling up to thousands of attributes with no loss of performance. the approach proposed by those authors uses: (i) a global set of bio tags, (ii) a model of semantic representation for attributes and titles, and (iii) an attention mechanism to capture the semantic relations. the proposed model processes up to 8,906 attributes with an overall f1 score of 79.12%. cheng et al. [4] explored ner in the e-commerce context, proposing an end-to-end procedure to query annotation and extraction for english language, using bio as a sequence tagging format and establishing two important entities to investigate: “brand” and “product type”. the annotation process was carried out with 16k queries. the proposed model reached a f1 score of 99.49% for the validation data and 93.30% for the test data. real et al. [14] presented the use of ner in the process of title generation in the fashion e-commerce context. the experiments showed that using a context-specific human-annotated data set for training, even though small (only 358 examples), achieved an overall f1 score of 71%. the solution proposed uses mitie. recently, one of the greatest revolutions in nlp area was the development of the contextualized language models such as the bidirectional encoder representations from transformers (bert) [6]. this language model encodes the text to a vector space where words with similar meanings are close to each other through a series of bidirectional self-attention heads. unlike other well-known word embedding models, such as glove [13] and word2vec [10], bert is capable of embedding words considering their specific contexts. bert has been used successfully in several nlp tasks as an off-the-shelf solution. to achieve these results, bert was trained in an unsupervised procedure, using corpora that sum up billions of words. souza et al. [15] trained a bert-crf to the ner task on the portuguese language, and explored feature-based and fine-tuning training strategies for the bert model. their fine-tuning approach obtained new state-of-the-art results on the harem i dataset [1], improving the f1-score by 1 point on the selective scenario (5 named entities classes) and by 4 points on the total scenario (10 classes of named entities). besides bert, we also investigate how the well-known mit information extraction tool (mitie) perform for the extraction of attributes from product titles. geyer et al. [7] used mitie to recognize named entities on a corpus consisting of 5,991 tweets in english, which have been annotated on seven different categories: person, organization, location, event, product, character, and thing. they varied the number of training documents and tested on the remainder of the documents, utilizing 5-fold cross-validation and only in-domain data. their results showed that training with in-domain data began to show diminishing returns with 500 training documents across all of the entity types other than character and thing. to the best of our knowledge this is the first work to investigate, in depth, the attribute-value extraction for the e-commerce domain in brazilian portuguese. 3 attribute set to define which attributes are relevant in smartphone category, after an initial analysis of smartphone titles, 10 attributes were selected for being investigated in this work: 1. wit: wit stands for “what is this” and is attached to the expression that better represents the product. in this work, the possible values for this attribute are: “smartphone”, “celular” (cell phone), “iphone” and other spelling variations. for examplefootnote 8: 2. brand: the brand is one of the most relevant attributes in a customer query. common values for this attribute are: “motorola”, “apple”, “samsung”, “xiaomi”, “blu”, among others. for example: 3. model: the model is also an important attribute for queries in this category and should be extracted in its maximum extension. for example: 4. internal memory: the internal memory capacity is usually an important attribute for this category and must be extracted together with its unit measure in gigabytes (gb). common values for this attribute are: 4 gb, 8 gb, 16 gb, 32 gb, 64 gb, 128 gb, 256 gb and 512 gb. for example: 5. display size: the display size should also be extracted with its unit measure in inches (”). the values for this attribute generally range from 4.7” to 6.1”. for example: 6. processor: common values for this attribute are: “dual-core”, “quad-core”, “hexa-core”, “octa-core”, “helio”, among others. for example: 7. camera quality: the quality of the smartphone camera is usually an attribute of great interest to customers. subsequently, the resolution in megapixels (mp) must be extracted with its unit measure. the values for this attribute usually range from 5.0 mp to 18.0 mp. for example: 8. color: when the color of the product appears in the title, it must be extracted as a relevant attribute. for example: 9. operating system: the operational system is an extremely relevant attribute for the smartphone category. the version of the operational system must also be extracted. common values for this attribute are: “android”, “ios”, “windows phone”, “blackberry os”, among others. for example: 10. sim card capacity: when the sim card capacity is present in the title, it must be extracted. the most common value for this attribute is “dual chip”. for example: these are the ten most relevant attributes for the smartphone category that we found in our database. from this set, we built a small annotated sample and trained supervised named entity recognition approaches, aiming at automatically extracting attribute-value pairs from product titles for the smartphone category as explained in the sect. 4. 4 experiments and results in this section we describe the corpus used in our experiments and the annotation process (sect. 4.1) carried out to feed the supervised named entity recognition (ner) approaches we investigated in this work. these approaches are detailed in sect. 4.2 and their results presented in sect. 4.3. it is worth mentioning that we used the source code available in the official open repositories of each specific tool applied here and that the corpus is not freely available due to the americanas s.a.’s police we worked with. 4.1 corpus pre-processing and annotation to analyze the viability of ner to extract attributes from product titles of the smartphone category, we defined a criterion to select a subset from the original data set of around 7 million products of general categories. we removed from the data set all the products with wit not labeled as “smartphone”, “celular” (cell phone), or “iphone”. this selection resulted in a data set of 7,432 products. from this set, we extracted only the product titles. the titles were, then, pre-processed by performing the following tasks: (i) conversion of the text to lowercase; (ii) removal of special characters such as: bars, parentheses, brackets and hyphens; (iii) tokenization by white space; (iv) stop wordsfootnote 9 removal; and (v) removal of duplicate titles. then, we removed the outliersfootnote 10 by excluding all titles that did not contain “smartphone”, “celular” or “iphone” and some possible misspelling variations (such as “smatphone”). in the end, 58 outliersfootnote 11 were removed. most of them were battery cells, biological cells or money bills probably wrongly categorized as “celular” due to the orthographic similarity between the word “celular” (cell phone) and the words “célula” (cell) or “cédula” (bill). the remaining 7,374 product titles were then clusterized. to start the clustering task, the tokens “smartphone”, “celular” and related spelling variants were removed from the data setfootnote 12. then, the titles were vectorized using the tf-idf vectorizer, using default parameter values. using the elbow method, the number of clusters was set to be 16 and the mini batch k-means clustering method was executed to group the product titles of the same brand, with a few deviations. the clustering was performed using scikit-learn [12]. from each cluster, 26 items were selected: 13 being the closest to the centroid and 13 being other ones randomly selected. as a result, a total of 416 product titles were selected to be annotated by two linguists. the annotation was performed using the prodigy [3] tool. a third linguist was responsible for resolving the discrepancy cases. the percentagesfootnote 13 of agreement between the two annotators can be seen in table 1. the annotation of the title “smartphone blu grand m dual sim 3g 5.0” 5mp 3.2mp prata” is illustrated in fig. 1 using the prodigy [3] tool. as a result, the following attributes were marked: wit (smartphone), brand (blue grand m), sim card capacity (dual sim), internal memory (3g), display size (5.0”), camera quality (5mp 3.2mp) and color (prata). fig. 1. annotation example using the annotation tool prodigy [3] full size image table 1. agreement between the two annotators for each attributefull size table as table 1 makes evident, wit was the attribute with the highest agreement, being labeled equally in 386 of the 414 instances. on the other hand, processor, model and display size were the attributes with the highest disagreement between annotators, with about 73% to 76% of the instances annotated equally. 4.2 experimental setup in this section, we describe the tools for ner investigated in this work: bert and mitie. bert is a recent approach which has achieved significant results in many nlp tasks [6]. mitie, in turn, is an earlier well-known and widely used tool for ner and binary relation extraction. while the former is a neural network based approach, which relies on contextualized word vectors, the latter applies traditional machine learning algorithms and static word embeddings to identify named entities. bert. (bidirectional encoder representations from transformers) [6] is a neural approach which outputs deep contextualized language representations derived from unlabeled text. although training a big bert model from scratch can be too computationally expensive, pre-trained bert models can be fine-tuned to several nlp tasks without substantial task-specific architecture modifications. some freely available, pre-trained bert models are: (i) the bert-base multilingual casedfootnote 14, which is a pre-trained model for 104 languages including portuguese; and (ii) bertimbau-basefootnote 15 [16], which was trained with the brazilian portuguese corpus brwac [17], widely used in open-source projects. besides bertimbau-base, some pre-trained models are also available for ner [15], which use bertimbau-base as a start point and were optimized for ner using haremfootnote 16 dataset. two approaches were applied for ner by the bertimbau authors: the feature-based and the fine-tuning. in the feature-based approach, a pre-trained model is used as a starting point, but the weights for the bert’s network layers did not change. a long short term memory (lstm) layer, when possibly combined with a conditional random fields (crf) layer, is trained and used to classify the sentences.footnote 17 in the fine-tuning approach, a pre-trained model is also used as a starting point, but its weights are adjusted during the training to achieve better results for ner. following the fine-tuning approach, it is possible to add a crf layer at the end of the network to help with sentence classification. in this work we fine-tuned two models for ner in our dataset: one fine-tuning only with bertimbau-base (bertimbau) and another one starting from the bertimbau-base trained for ner with harem dataset and also adding a crf layer (bertimbau-harem).footnote 18 to evaluate the robustness of these methods to random variations, two random seeds (seed1 and seed2) were extracted from the data set for generating training (50%), validation (25%) and test (25%) partitions. in total, 4 models were trained using the bert approach: bertimbau in seed1, bertimbau in seed2, bertimbau-harem in seed1 and bertimbau-harem in seed2. for both pre-trained bert models, the hyperparameters used were the default values. mitie. (mit information extraction) frameworkfootnote 19 [9] was designed to be a simple, easy and intuitive tool with no parameters. the current version of mitie includes tools for performing ner, binary relationship extraction, as well as tools for training custom models. the mitie ner model uses the wordrep benchmark for building word representations (word embeddings), word morphology and traditional machine learning algorithms such as structural support vector machines [8]. mitie comes with some previously generated word embeddings for english, spanish and german. in our experiments we generated new word embeddings, using wordrep, which is publicly available in the mitie repository [2]. we used a dataset containing around 7 thousand titles from the smartphone category. this word representation was called celular. the same seeds (seed1 and seed2) used for training bert ner models were also used for training mitie ner models with one difference: mitie does not explicitly use a validation set. thus, the seeds were divided only into training and test, following the proportion of 75% for training and 25% for testing. in this way, the training instances of mitie correspond to the training and the validation instances used by bert, and the test cases are the same for both models. in addition to the celular word representation created in this work, existing ones for english (english) and spanish (spanish) available with mitie were also used. we also generated a huge word representation from 7.5 million titles for products from diverse categories (not only smarthphone). this representation was called all. finally, we also generated two versions of the celular word representation removing the titles from seed1 (cel-seed1) and seed2 (cel-seed2). ergo, ten different models were trained in total using these 6 word representations. 4.3 results in this section we present the results for bert and mitie separately, then a comparison of both approaches will be made in the next section by drawing some conclusions about their performances on solving the investigated problem. bert results. as described in sect. 4.2, two bert models were fine-tuned for ner (attribute extraction) in product titles: the original bertimbau (bertimbau) and the bertimbau trained for ner with the harem dataset and a crf layer (bertimbau-harem). table 2 shows that the bert models present f1 scores of around 94–95%, proving to be very robust independently of the random seed, indicating the absence of overfitting. table 2. f1 scores for bert models in test setfull size table it is interesting to notice that, although by a little difference, the model fine-tuned from the bertimbau-base performed better than the one fine-tuned from a version trained for ner in harem dataset. table 3 provides the detailed results for the best scenario: the bertimbau model in the test set of seed2. the “support” column indicates the amount of evaluated instances in each class. the classes with the worst f1 scores were model and processor. the former one, probably due to labelling errors for the boundaries of the entity and the latter one maybe due to the small amount of training instances (less than 40). mitie results. as described in sect. 4.2, ten mitie models were trained using 6 different word representations: english, spanish, all, celular, cel/seed1 and cel/seed2. table 4 shows the f1 scores for these models. table 3. detailed evaluation of bertimbau fine-tuned with seed2 in the test setfull size table table 4. f1 scores for the mitie models in test setfull size table from f1 scores we can see, again, that the values obtained for seed1 and seed2 are very close – again around 94–95%. regarding the wordreps (word embeddings), the english and celular were the ones with best results and removing the titles from seed1 or seed2 did not significantly impact the results. thus, it is possible to conclude that having the titles used in training for the generation of the word representations did not significantly bias the model. therefore, the model trained with celular for seed2 was the one chosen for further analysis. in order to do that, table 5 presents the detailed results for the evaluation of the mitie model trained with celular and seed2 in the test set. it can be noticed that, as happened with the bert models, the mitie model also performed poorly in the model class. again, we believe that this is due to entity labelling errors. example. to finish our results and discussion section, table 6 shows the mitie’s predictions output by the model trained with celular and seed2 for the three product titles: 1. smartphone samsung galaxy j6, 32 gb, dual chip, android, tela 5.6 pol, octa core, 4g, 13mp, tv prat; 2. iphone 7 apple, ouro rosa, tela 4.7\(^{\prime \prime }\), 4g+wifi, ios 11, 12mp, 32 gb; 3. celular up 3 chip quad cam mp3 4 fm preto cinza p3274 multilaser table 5. detailed evaluation of the mitie using celular and seed2 in the test setfull size table the only differences between bert and mitie’s outputs for these examples were: for title 1, bert did not output a value for color attribute; for title 2, in the internal memory, the only output of bert was “32 gb” and the “ios 11” was tagged as operating system. for title 3, “quad” token was tagged as brand, the display size attribute was left empty, and the model attribute values were “up” and “p3274”. table 6. examples of mitie’s predictions using celular and seed2full size table 4.4 comparison between the models figure 2 shows a graph of f1 scores for the best bert (bertimbau in seed2) and mitie (trained with celular and seed2) models. fig. 2. comparison of f1 scores for the best models (bertimbau and mitie trained with celular) trained with seed2 in the test set full size image as we can see from fig. 2, the performances are very close with little highlights to: bertimbau in color class and mitie celular in processor class. in fact, it is worth highlighting the good performance of the mitie model in the processor class, which has only less than 40 training instances. regarding the training time, we can point out that , when using the cpu, both mitie and bert took a considerable amount of time, however mitie was still faster than bert. bert took around 7 h against 5 h from mitie to train each model. however, the bert training can be speed up using gpus as accelerators: with a median gpu, the fine-tuning of bertimbau took only 28 min. regarding the prediction latency (that is, the time for inferring new entities), the bertimbau model took 97 min and 15 s to infer all the entities in the dataset containing 7,374 titles running (only on cpu) in a machine with a intel(r) xeon(r) silver 4210 cpu @ 2.20 ghz processor and 40 cores. using a gpu gigabyte nvidia geforce rtx 2060 windforce oc (6 gb), the time needed to finish all inferences decreased considerably: the model took only 11 min e 52 s to complete the inference for the same set of 7,374 instances. on the other hand, for the same dataset and the same cpu machine, the mitie model trained with celular and seed2 took approximately 3.64 s to make all the inferences, showing an extremely better throughput when compared to bert model. this mean 4 s from mitie \(\times \) 11 min from bert. so, mitie offers not only accuracy, but also efficiency as had already been pointed out as one of its great advantages by its developers. regarding the size of the model when stored on disk, bert uses an average of 1.3 gb to store the model as a whole. for mitie, the storage size of the model varies a lot and it is directly proportional to the size of the word representation used. for english, spanish and all the model sizes are around 338 mb, for celular, cel-seed1 and cel-seed2, the final size of the models is about 29 mb. in light of that, the mitie model wins again when the size of the model is considered. in general, both bert and mitie models achieved satisfactory results on the pursued task. the difference in f1 score between these methods is not significant. however, a few points need to be considered. first, mitie performs better where bert achieved the worst scores. specifically in the processor and model classes. in the classes where bert performs slightly better than mitie, the differences are minor, and the scores obtained by mitie were considered adequate. moreover, the computational resources required by mitie are notably lower than those demanded by bert. considering these points and, particularly, the fact that the prediction latency is crucial for the application, we considered the mitie model trained on celular and seed2 as the most appropriate to generate the standardization dictionary that will give rise to search filters for the smartphone category. 5 final remarks this paper addressed the problem of attribute-value extraction from product titles of the smartphone category in brazilian portuguese. by automating this process, the e-commerce companies are able to have a unified treatment to their product assortment, improving query matching, product recommendation and the use of facets to enhance product discoverability. several models were investigated in this paper based on bert and mitie approaches. after a careful analysis based on accuracy and efficiency, we concluded that, for our scenario, a solution based on mitie is still more interesting than one based on pre-trained language models. as future work, we want to investigate how scalable our approach is: would a mitie-based model still get competitive accuracy when applied to different categories? considering a sparsely annotated data scenario, which approach would be quick to deploy, keeping high-quality predictions? all in all, a question that remains is, across the 35mi products americanas s.a. sells now, will a classical machine learning approach still be competitive with deep learning approaches? notes 1.https://company.ebit.com.br/webshoppers/webshoppersfree. 2.in english: samsung galaxy a01 smartphone, 32 gb, 2 gb ram, 5.7\(^{\prime \prime }\) infinite screen, 13mp dual rear camera (main) + 2mp (depth), 5mp front, 3000 mah battery, dual chip, android blue. 3.in english: moto g9 play 64 gb smartphone dual chip android 10 screen 6.5\(^{\prime \prime }\) qualcomm snapdragon 4g camera 48mp+2mp+2mp turquoise green. 4.in english: iphone se 128 gb black ios 4g wi-fi screen 4.7\(^{\prime \prime }\) 12mp + 7mp camera apple. 5.wit stands for “what is this” and is used to define the product being sold. 6.the smartphone category encompasses cell phones, smartphones and iphones sold by the largest brazilian e-commerce marketplace, b2w digital. 7.https://github.com/mit-nlp/mitie. 8.in this paper, the tags associated with the attributes are presented between brackets and the values associated to them, in bold. 9.stop words list in portuguese: de, a, o, que, e, do, da, em, os, no, na, por, as, dos, ao, das, á, ou, ás, com. 10.outliers refers to those product titles that do not belong to the smartphone category. 11.examples of removed outliers, in portuguese: “célula vegetal ampliada aproximadamente 20 mil vezes”, “celula carga 10 kg sensor peso arduino” and “cédula foleada ouro 100 euros coleção notas moedas euro”. 12.we didn’t remove the token“iphone” and its spelling variations since the format of iphone titles is very different from the other items in the data set and the experiments removing “iphone” led to results worse than those not removing it. 13.since how to measure the inter-annotator agreement of named entities annotation is a debatable task [5], we followed [1] and considered the percentages of agreement as our main metric. 14.https://huggingface.co/bert-base-multilingual-cased/tree/main. 15.https://github.com/neuralmind-ai/portuguese-bert. 16.https://www.linguateca.pt/harem/. 17.from previous experiments, we concluded that the feature-based models as well as the ones trained with the multilingual bert did not reach results as good as the fine-tuned models trained with bertimbau-base, so these options are not described here. 18.these two configurations were the ones which achieved the best results in previous experiments with product tiles from fashion category. 19.https://github.com/mit-nlp/mitie. references harem linguateca datasets. https://www.linguateca.pt/harem/. accessed 08 jan 2021 repositório oficial mitie. https://github.com/mit-nlp/mitie prodigy: an annotation tool for ai, machine learning, and nlp (2017). https://prodi.gy. accessed 08 jan 2021 cheng, x., bowden, m., bhange, b.r., goyal, p., packer, t., javed, f.: an end-to-end solution for named entity recognition in ecommerce search. arxiv preprint arxiv:2012.07553 (2020) deleger, l., et al.: building gold standard corpora for medical natural language processing tasks. in: amia annual symposium proceedings (2012) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: conference of the north american chapter of the association for computational linguistics, pp. 4171–4186 (2019) google scholar  geyer, k., greenfield, k., mensch, a.c., simek, o.: named entity recognition in 140 characters or less (2016) google scholar  joachims, t., finley, t., yu, c.j.: cutting-plane training of structural svms. mach. learn. 77, 27–59 (2009) article  google scholar  king, d.e.: dlib-ml: a machine learning toolkit. j. mach. learn. res. 10, 1755–1758 (2009) google scholar  mikolov, t., sutskever, i., chen, k., corrado, g.s., dean, j.: distributed representations of words and phrases and their compositionality. in: burges, c., bottou, l., welling, m., ghahramani, z., weinberger, k. (eds.) advances in neural information processing systems 26, pp. 3111–3119 (2013) google scholar  more, a.: attribute extraction from product titles in ecommerce. in: workshop on enterprise intelligence acm sigkdd conference on knowledge discovery and data mining (2016) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  pennington, j., socher, r., manning, c.d.: glove: global vectors for word representation. in: empirical methods in natural language processing, vol. 14, pp. 1532–1543 (2014) google scholar  real, l., johansson, k., mendes, j., lopes, b., oshiro, m.: generating e-commerce product titles in portuguese. in: anais do xlviii seminário integrado de software e hardware, pp. 299–304. sbc, porto alegre, rs, brasil (2021). https://doi.org/10.5753/semish.2021.15835. https://sol.sbc.org.br/index.php/semish/article/view/15835 souza, f., nogueira, r., lotufo, r.: portuguese named entity recognition using bert-crf. arxiv preprint arxiv:1909.10649 (2019). http://arxiv.org/abs/1909.10649 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, october 20–23 (2020) google scholar  wagner, j., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: international conference on language resources and evaluation, pp. 4339–4344 (2018) google scholar  xu, h., wang, w., mao, x., jiang, x., lan, m.: scaling up open tagging from tens to thousands: comprehension empowered attribute value extraction from product title. in: annual meeting of the association for computational linguistics, pp. 5214–5223 (2019). https://doi.org/10.18653/v1/p19-1514 download references acknowledgment this paper and the research behind it would not have been possible without the support of americanas s.a. digital lab, specially josé pizani, ester campos and thiago gouveia nunes, who closely followed this research. this work is part of the project “dos dados ao conhecimento: extração e representação de informação no domínio do e-commerce” (projeto de extensão ufscar #23112.000186/2020-97). author information authors and affiliations federal university of são carlos, são carlos, sp, brazil diego f. silva, alcides m. e silva, bianca m. lopes, karina m. johansson, fernanda m. assi, júlia t. c. de jesus, reynold n. mazo, daniel lucrédio & helena m. caseli americanas s.a. digital lab, são paulo, sp, brazil livy real authors diego f. silvaview author publications search author on:pubmed google scholar alcides m. e silvaview author publications search author on:pubmed google scholar bianca m. lopesview author publications search author on:pubmed google scholar karina m. johanssonview author publications search author on:pubmed google scholar fernanda m. assiview author publications search author on:pubmed google scholar júlia t. c. de jesusview author publications search author on:pubmed google scholar reynold n. mazoview author publications search author on:pubmed google scholar daniel lucrédioview author publications search author on:pubmed google scholar helena m. caseliview author publications search author on:pubmed google scholar livy realview author publications search author on:pubmed google scholar corresponding author correspondence to diego f. silva . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silva, d.f. et al. (2021). named entity recognition for brazilian portuguese product titles. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_36 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_36 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords product titles named entity recognition mitie bert portuguese publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature experiments on kaldi-based forced phonetic alignment for brazilian portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper experiments on kaldi-based forced phonetic alignment for brazilian portuguese conference paper first online: 28 november 2021 pp 465–479 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) experiments on kaldi-based forced phonetic alignment for brazilian portuguese download book pdf download book epub cassio batista  orcid: orcid.org/0000-0001-6799-605810 & nelson neto  orcid: orcid.org/0000-0003-0408-418710  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1254 accesses 1 citation abstract forced phonetic alignment (fpa) is the task of associating a given phonetic unit to a timestamp interval in the speech waveform. phoneticians are able mark the boundaries with precision, but as the corpus grows it becomes infeasible to do it by hand. for brazilian portuguese (bp) in particular, only three tools appear to perform fpa: easyalign, montreal forced aligner (mfa), and ufpalign. therefore, this work aims to develop resources based on kaldi toolkit for ufpalign, including their release alongside all scripts under open licenses; and to bring forth a comparison to the other two aforementioned aligners. evaluation took place in terms of the phone boundary metric over a dataset of 385 hand-aligned utterances, and results show that kaldi-based aligners perform better overall, and that ufpalign models are more accurate than mfa’s. furthermore, complex deep-learning-based approaches did not seem to improve performance compared to simpler models. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others free resources for forced phonetic alignment in brazilian portuguese based on kaldi toolkit article open access 19 february 2022 towards a free, forced phonetic aligner for brazilian portuguese using kaldi tools chapter © 2020 forced phonetic alignment in brazilian portuguese using time-delay neural networks chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics lisp orthography speech and audio processing speech and audio signal processing phonology and phonetics 1 introduction the analysis of the prosodic structure of speech very often requires the alignment of the speech recording with a phonetic transcription of the speech, a task known as forced phonetic alignment (fpa). however, transcribing and aligning several hours of speech by hand is very time-consuming, even for experienced phoneticians. as several approaches have been applied to automate this process, some of them brought from the automatic speech recognition (asr) domain, the combination of hidden markov models (hmm) and gaussian mixture models (gmm) has been for long the most widely explored for fpa. with respect to asr-based frameworks, we found only three forced aligners that provide pre-trained models for brazilian portuguese (bp): easyalign [9], montreal forced aligner (mfa) [16] and ufpalign [6, 32]. to the best of our knowledge, easyalign is the only htk-based aligner that ships with a model for bp, mfa is the only kaldi-based one, and ufpalign has been evolving through time to work with both htk and kaldi as back-end. it should be remarked that ufpalign was born in [32] as an early effort to mitigate the gap for brazilian portuguese, providing a package with grapheme-to-phoneme (g2p) converter, syllabification system and gmm-based acoustic models trained over the htk toolkit [34]. as usual, tests comparing the automatic versus manual segmentations were performed. an extra comparison was made to easyalign [9], which to our knowledge was the only aligner that supported bp at that moment. it was observed that the tools achieved equivalent behaviors, considering two metrics: boundary-based and overlap rate. later on, following kaldi’s success as the de facto open-source toolkit for asr [25] due to its efficient implementation of deep neural networks (dnn) for hybrid hmm-dnn acoustic modeling, ufpalign was updated in [6] w.r.t. its older htk-based version, yielding better results with both monophone and triphone gmm-based models, as well as with a standard feed-forward network trained using nnet2 recipes. both htkand kaldi-based versions of ufpalign were then evaluated over a dataset containing 181 utterances spoken by a male speaker, whose phonemes were manually aligned by an expert phonetician. therefore, as nnet2 recipes became outdated, this work builds upon [6] by updating training scripts to kaldi’s nnet3 recipe, which contains the current state-of-the-art scripts for asr. up-to-date versions of the acoustic models, phonetic and syllabic dictionaries were released to the public under the mit license on falabrasil’s github accountfootnote 1, as well as the scripts to generate them. assuming kaldi is pre-installed as a dependency, ufpalign pipeline’s works fine under linux environments via command line, but also provides a graphical interface as a plugin to praat [3], a popular free software package for speech analysis. additionally, some intraand inter-evaluation procedures were performed, the former considering all acoustic models trained within the kaldi’s default gmm and dnn pipeline, while the latter applied the htk former version of ufpalign [32], easyalign [9], and mfa [16] aligners over the same dataset for the sake of a fair comparison. the evaluation dataset was extended from 193 utterances spoken by a male individual to include 192 sentences spoken by a female speaker, i.e., 385 manually aligned audio files in total. the similarity measure is given by the absolute difference between the forced alignments with respect to manual ones, which is called phonetic boundary [16]. in summary, the contributions of this work include: release of monophone-, triphone-, and dnn-based (nnet3) acoustic models, which comprise a total of five pre-trained, kaldi-compatible models included as part of ufpalign. scripts used to train such models are also available. generation of multi-tier textgrid files for praat, based on phonetic and syllabic dictionaries built over a list of words in bp collected from multiple sources and post-processed by gnu aspell [2] spell checker. comparison to the only two asr-based phonetic aligners that exist for brazilian portuguese (to the best of our knowledge), regarding the phone boundary metric [16] over a dataset of 385 hand-aligned utterances. the remainder of this paper is as follows. section 2 presents the fpa procedure with kaldi, and some other resources used for training and evaluation. evaluation tests and results are reported and discussed on sects. 3 and 4, respectively. finally, sect. 5 presents the conclusion and plans for future work. 2 methodology this section details the forced phonetic alignment process within ufpalign, which is similar to a traditional decoding stage in speech recognition where one needs an acoustic model and a phonetic dictionary (or lexicon) to decide among senones, except the language model is not necessary in such case. ufpalign uses kaldi as the asr back-end, and falabrasil’s grapheme-to-phoneme (g2p) and syllabification tools to provide phonemes and syllables from regular words (also known as graphemes), given that users themselves provide such transcriptions as input alongside with the corresponding audio file. the output is stored in a textgrid file—a well-known file format for praat users. 2.1 kaldi, grapheme-to-phoneme and syllabification tools kaldi [25] is an open-source toolkit developed to support speech recognition researchers. the dnn training framework is provided by kaldi in three distinct setupsfootnote 2: nnet1 [14], nnet2 [27, 35] and nnet3. unlike nnet1 and nnet2, nnet3 offers an easier access to more specialized kinds of networks other than simple feed-forward ones, including long short-term memory (lstm) [21] and time-delay neural networks (tdnn) [22, 24], for example. scripts in kaldi’s nnet3 setup use factorized time-delay neural networks (tdnn-f) as default architecture [22], which are a type of feed-forward network that has a behavior similar to recurrent topologies like lstms in the sense of capturing past and future temporal contexts w.r.t. the current speech frame to be recognized, but with an easier procedure for parallelization. this opposes to previous nnet2 recipes, for instance, which are pure vanilla networks. as kaldi requires a lexicon to serve as the target being modeled by hmms, this work uses a g2p converter provided by the falabrasil group as an open-source library written in java [18, 30]. this tool relies on a stress determination system to provide only one pronunciation per word, which means it does not consider co-articulation between words (i.e., cross-word events are ignored). the phonetic alphabet is composed by 38 phonemes plus a silence phone, inspired by the speech assessment methods phonetic alphabet (sampa) [7]. the syllabification tool, on the other hand, is not a requirement when training acoustic models for asr, but rather just a feature of ufpalign for composing another tier in the textgrid output file. it is also provided by the falabrasil group within the same library as the g2p [19]. 2.2 training speech corpora and lexicon to build an effective acoustic model (am), a relatively large amount of labeled data is required, apart from a language model (lm) and a pronunciation model (a.k.a. phonetic dictionary or lexicon). an lm is necessary for speech recognition despite not being explicitly used during phonetic alignment itself. the model used here was built in [18] using srilm [33] toolkit over approximately 1.5 million sentences from the cetenfolha dataset [12]. the falabrasil speech corpora, on the other hand, consists of seven datasets in brazilian portuguese with a total of approximately 170 h of transcribed audio, the same as in [6]. finally, the phonetic dictionary was created via falabrasil g2p tool [18, 30] based on a list of words collected from multiple sources on the internet, including university of minho’s projecto natura [1], libreoffice’s vero dictionary [17], nilc’s cetenfolha dataset [12], and frequencywords repository based on subtitles from opensubtitles [8, 20]. gnu aspell [2] is responsible for checking out the spelling and consequently filtering the huge number of words collected, resulting in approximately 200,000 words in the final list. 2.3 acoustic models the deep-learning-based training approach in kaldi actually uses the gmm training as a pre-processing stage. for this work, ams were trained by adapting the recipe for mini-librispeech dataset [23]. for details on the gmm training pipeline, the reader is referred to [6]. the dnn is trained on the top of the last gmm model of the pipeline, which comprises a speaker-adapted training (sat). fig. 1. stages for training a tdnn-f following kaldi’s mini-librispeech recipe. on the left side, high-resolution, cepstral-normalized mfccs (40 features instead of 13) are extracted from an augmented corpora after applying speed and volume perturbation [15], as are the speaker-related 100-dimensional i-vector features [5, 31]; to be used as input to the neural network. on the right side, training labels are provided by a gmm tri-sat acoustic model. full size image figure 1 details how the dnn model is obtained as a final-stage am by using the neural network to model the state likelihood distributions as well as to input those likelihoods into the decision tree leaf nodes [10]. the implementation in kaldi uses a sub-sampling technique that avoids the whole computation of a feed-forward’s hidden activations at all time steps and therefore allows a faster training of tdnns. the “factorized” term distinguishes a tdnn-f from a traditional tdnn architecture by a singular value decomposition (svd) that is applied at the hidden layer’s weight matrices in order to reduce the number of model parameters without degrading performance [24]. 2.4 kaldi forced phonetic alignment kaldi’s fpa procedure performs several steps for obtaining the time-marked conversation (ctm) files, which contains a list of numerical indices corresponding to phonemes with both their start times and durations in seconds. after kaldi scripts extract some features from time-domain audio data, the forced alignment step, that employs the aforementioned pre-trained acoustic models, is computed by kaldi using viterbi beam search algorithm [11]. figure 2 shows an overview of the stages within ufpalign. fig. 2. pipeline followed by ufpalign. when a user feeds the system with an audio (.wav) and its respective transcription (.txt), they should expect a praat’s textgrid file (.tg) as output. time marks are provided by kaldi, which relies on the knowledge of the acoustic model (am) and tokens of the lexicon (l.fst). full size image the data and language preparation stage in particular also creates some “data files” on the fly, which contain information regarding the specifics of the audio file and its transcription, namely text, wav.scp, utt2spk, and spk2utt. the language preparation stage, on the other hand, is given by a script provided by kaldi to create another set of important files, the main one being the lexicon parsed into a finite-state transducer (fst) format, called l.fst. for data preparation, the first step consists in checking whether there are any new words in the input data that were not seen during the acoustic model training. if any word in the transcriptions is not found in the pronunciation dictionary (lexicon), it calls the grapheme-phoneme conversion module (g2p) [18, 30] to extend the lexicon with each new word along with its respective phonemic pronunciation. for praat’s final visualization purposes, the word is also divided into syllables through the embedded syllabification tool [19]. as the original phonetic and syllabic dictionaries contain approximately 200,000 entries, they both become lex 200k+ and syll 200k+ files after the insertion of missing words. the last block of the phonetic alignment process handles the conversion of both ctm files to a praat’s textgrid (.tg), a text file containing the alignment information. therefore, ctm files are read by a python script that in the conversion process uses the lex 200k+ and syll 200k+ extended dictionaries to generate the output five-tier textgrid that can be displayed by praat’s editor. 3 evaluation tests the evaluation procedure takes place by comparing a bunch of textgrid files: the hand-aligned reference and the ones automatically annotated by the forced aligners (i.e., by inference), as the phone boundary metric considers the absolute difference between the ending time of both phoneme occurrences [16]. the calculation is performed for each acoustic model, and it takes place over all utterances from the evaluation dataset composed by one male and one female speaker. 3.1 evaluation speech corpus the automatic alignment was estimated on the basis of the manual segmentation. the original dataset used for assessing the accuracy of the phonetic aligner is composed of 200 and 199 utterances spoken by a male and a female speaker, in a total of 15  min and 32 s of hand-aligned audio, as shown in table 1. praat’s textgrid files, whose phonetic timestamps were manually adjusted by a phonetician, are available alongside audio and text transcriptions. table 1. speech corpus used to evaluate the automatic phonetic aligners. actual duration and number of files after discard are shown between parentheses, as well as the number of unique words.full size table this dataset was aligned with a set of phonemes inspired by the sampa alphabet, which in theory is the same set used by the falabrasil’s g2p software that creates the lexicon during acoustic model training. nevertheless, there are some problems of phonetic mismatches, and some cross-word phonemes between words, which makes the mapping between both phoneme sets challenging, given that falabrasil’s g2p only handles internal-word conversion [30]. fig. 3. evaluation takes place by comparing the output of all forced aligners to a hand-aligned ground-truth. the m2m mapping is applied to make different phone sets match the sampa version used by falabrasil’s g2p, which is provided by the lexicon generated over transcriptions of the corpus (lex m/f). full size image the example in table 2 shows the phonetic transcription for a sentence given by the original dataset (top) and the acoustic model (bottom) which then suppress vowel sounds altogether due to cross-word rules (usually elision and apocope) when they occur at the end of the current word and at the beginning at the next. such mismatches occur because the dataset was aligned by a phonetician considering acoustic information (i.e., listening), which cannot be done by the g2p tool that creates the acoustic model’s lexicon, since it is provided only with textual information. situations like these of phonetic information loss led to the removal of such audio files from the dataset before evaluation. table 2. cross-word mismatches between transcriptions manually aligned by a phonetician (top) vs. generated by our g2p software (bottom). word boundary losses are represented by the empty set symbol (\(\varnothing \)).full size table in the end, fourteen files were excluded from the dataset, so about 34 s of audio was discarded, and 193 and 192 utterances remained in the male and female datasets, respectively. the filtering also ignored intraand inter-word pauses and silences, resulting in 2,518 words (686 unique, since the utterances’ transcriptions are identical for both speakers, i.e., they speak the very same sentences) and 10,537 phonetic segments (tokens) (c.f. table 1). 3.2 simulation overview figure 3 shows a diagram of the experiments where easyalign, ufpalign and mfa forced aligners receive the same input of audio files (.wav) with their respective textual transcriptions (.txt). these are the files whose manual annotation is available. all three aligners output one textgrid file (.tg) for each audio given as input, which then serve as the inference inputs to the phone boundary calculation. the reference ground-truth annotations, on the other hand, are provided by the 385 textgrid files that contain the hand-aligned phonemes corresponding to the transcriptions in the evaluation dataset. however, for computing phone boundaries, there must exist a one-to-one mapping between the reference and the inference phones, which was not possible at first due to the nature of the phonetic alphabets: ufpalign and easyalign share the same sampa-inspired lexicon generated by falabrasil’s g2p tool, while mfa is based on arpabet [29]. furthermore, the hand-aligned utterances fall on a special case where the phonetic alphabet used (referred here as “original”) is also sampa-inspired, but not exactly the same as falabrasil’s. apart from the fact that cross-word rules can insert or delete phones, some phonemes do not have an equivalent, such as /ts/ and /dz/. besides, there are also usual swaps between phonetically similar sounds: /h//, , /h/ and /4/, for instance, might be almost deliberately mapped to either /r/, /r/ or /x/. this is worse in mfa, where the set of phonemes is entirely different. thus, since the situation seemed to require a smarter approach than a simple one-to-one tabular, static mapping, it was necessary to employ a many-to-many (m2m) mapping procedure (c.f. dashed blocks on fig. 3) based on statistical frequency of occurrence, e.g., how many times phones /t/ and /s/ from the original evaluation dataset were mapped to a single phone /ts/ in the lex m/f file representing falabrasil’s g2p sampa-inspired alphabet. this mapping also works when dealing with mfa’s arpabet phonemes, and will be further discussed in sect. 3.3. 3.3 many-to-many (m2m) phonetic mapping by taking another look at table 2, one might have also reasoned that the mapping between the two sets of phonemes is not always one-to-one. the usual situation is where a pair of phonemes from the dataset (original) is merged into a single one for the am (falabrasil g2p), such as /i\(\sim \) / /n/ \(\rightarrow \) /i\(\sim \)/ and /t/ /s/ \(\rightarrow \) /ts/. however, a single phoneme can also be less frequently split into two or more, such as /u/ /s/ \(\rightarrow \) /u/ /j/ /s/. to deal with these irregularities, we used the many-to-many alignment (m2m-aligner) software [13] in the core of a pipeline that converts the original textgrid from the evaluation dataset to a textgrid that is compatible with the falabrasil’s lexicon used to train the acoustic models. we took advantage of the same pipeline to convert mfa’s arpabet-based phonemes to sampa as well. the m2m-aligner works in an unsupervised fashion, using an edit-distance-based algorithm to align two strings from a file in the news format, in order for them to share the same length [13]. all 385 textgrid files from our evaluation dataset (.tg) are used to compose a single news file, as exemplified in table 3. notice the file is composed by the phonemes of the whole sentence rather than by isolated words, in order to mitigate the effects of the cross-word boundaries. the string mapping is finished after a certain number of iterations when the m2m-aligner provides a one-to-one mapping in a file we called m2m that joins some phonemes together, as shown by shades of gray in table 3. table 3. example of a single news file with phonemes from three out of 385 textgrid files for sentences “é bom pousar” and “os lindos jardins”. each line contains a whole phonetic sentence to be converted, and different phone sets are separated into two distinct columns divided by a tabular character, so every other token is separated by a single space. groups of phonemes which are supposed to be later merged by m2m-aligner in the m2m file are shaded in gray.full size table finally, as the m2m-aligner provides the mapping for phonemes, another script provides the time stamps calculations prior to creating the converted textgrid file. table 4 illustrates how the phonetic time stamps, in milliseconds, are mapped accordingly. basically if two or more phonemes are mapped into a single one (merging), as in /o\(\sim \)/ /n/ \(\rightarrow \) /o\(\sim \)/ or /d/ /z/ \(\rightarrow \) /dz/ (marked with an \(*\)), the time stamp of the last phoneme is considered. however, if one phoneme is mapped to two or more (splitting) as in /e\(\sim \)/ \(\rightarrow \) /e\(\sim \)/ /j\(\sim \)/, then linearly spaced time stamps are generated in between the phone to be split (\(\dagger \)) and its immediate predecessor (\(\ddagger \)). 4 results and discussion results will be reported in terms of statistics such as mean (\(\mu \)), median and standard deviation (\(\sigma \)) over the distribution of phone boundary values, and a tolerance threshold that shows how many phonetic tokens were more precisely aligned with respect to the manual alignments. numerical values, in milliseconds, are presented in table 5. the best ones are highlighted in bold. table 4. conversion of time stamps for the sentence “onde existem”.full size table as far as mfa train-and-align (t&a) feature is concerned, roughly only 1% of phoneme tokens aligned by kaldi-based aligners are off the 100 ms tolerance, against 3% of tokens aligned by htk-based tools. in fact, approximately 96%–97% of phonemes were under the 50 ms tolerance when aligned by acoustic models trained with mfa and ufpalign, considering an average of all models. unfortunately, this is not true for mfa’s pre-trained model for brazilian portuguese (in align-only mode), which on the other hand, for larger tolerance threshold values, performed a little worse than htk. among htk-based aligners, easyalign performed best considering all statistics and tolerance thresholds for both male and female speakers. however, as already pointed out in [32], the same ground-truth dataset used for evaluation in this work was also used to train the bp acoustic model shipped with easyalign, so this might have had some bias during the comparison. overall, ufpalign (htk) achieved very similar values across metrics for both speakers of the dataset, while easyalign’s behavior shows a greater accuracy on the female voice. nevertheless, the parcel of phonetic tokens whose difference to the manual segmentation was less than 10 ms stayed below the 40% even for easyalign. in align-only (a) mode, mfa models performed slightly better until 10 ms than easyalign’s, but increasingly worse for larger values of tolerance for both male and female speakers. these poor results may be due to the nature of the dataset used to generate mfa’s pre-trained acoustic models (globalphone [28]), which contains only 22 h of transcribed audio. in contrast, training and aligning (t&a) on the same evaluation dataset with mfa proved better than htk for the male speaker, and the results are similar for the female speaker. the monophoneand triphone-based gmm models we trained with kaldi for ufpalign achieved the best performance with respect to phone boundary when compared to both mfa and htk-based aligners. on average, approximately 45% of tokens were accurately aligned within the 10 ms margin for all gmm models. mean and median values are the lowest (except for tri-sat on the male dataset, which was greater than mfa’s t&a) and at most \(\sim \)4 ms distant from each other. with respect to the speakers’ gender, ufpalign (kaldi) performed approximately 4% better for the woman’s voice until the 50 ms of tolerance, and about 2 ms more accurate according to the average mean. table 5. results regarding mean (\(\mu \)), median (med.), standard deviation (\(\sigma \)), and cumulative percentage below a tolerance threshold, in milliseconds, of the differences between forced aligned audio and ground-truth (hand aligned) phonemes, also known as phone boundary. notations on mfa stand for align-only (a) and train-and-align (t&a) procedures, while on ufpalign they denote either the nature of the toolkit or the acoustic model.full size table finally, tdnn-f simulation was definitely disappointing. we expected that results from a nnet3 dnn-based setup would be at least similar to gmm-based ones, as it was in [6] with nnet2, but cumulative tolerance values were instead just slightly better than easyalign. therefore, even though one can say that the best result was achieved by tri-delta (\(\updelta \)) models on both male and female datasets, since it holds the rows with most boldface values in table 5 (except mfa was better off after 50 ms on the man’s voice, but the values compared to ufpalign’s tri-\(\updelta \) model are fairly and virtually the same), we would rather prefer to state that all gmm-based ams in ufpalign achieved similar results. even monophone models, the simplest ones, had a close performance on tri-sat, the most complex. 4.1 discussion a possible reason for such a difference between htkand kaldi-based aligners might be that htk uses baum-welch algorithm for training hmms while kaldi uses viterbi training [4]. on the other hand, among kaldi models, tri-\(\updelta \) stands out as being virtually the best one. however, with just a \(\sim \)1–3% difference in tolerance, and \(\sim \)1 ms difference in both mean and median values, we cannot tell whether it is significant enough to classify one model into being better than the others, as they appear pretty close at glance. the linear sequence of model training just does not result in lower errors in phonetic boundaries as it resulted in lower word error rates for speech recognition. the somewhat shocking results were produced by the dnn. for the state of the art for asr to perform so poorly in phonetic alignment problems, it certainly needs careful investigation. we suspect the hmm topology used in nnet3 chain models, which can be traversed in one frame rather than in three on the traditional left-to-right [26], may have had some unfavorable influence. moreover, data insufficiency could even have been the problem for the dnn in the first place, since the \(\sim \)171 h in our training dataset are far from the ideal volume to train a neural network efficiently. other reasons include the possible high number of hidden layers in the tdnn-f, and the use of frame subsampling, which requires an extra normalization value to be passed to kaldi’s ali-to-phones script for compensation. besides, navigating through all the burden to train a dnn model with kaldi (which requires at least one gpu card) may not be the more appropriate move if the final task’s goal is to align phonemes rather than to recognize speech. as mfa seem to have dropped support to dnn models, and our previous results with a nnet2 neural network setup only took tolerance values so far as to match tri-\(\updelta \) models [6], we feel discouraged to invest so much time computer power to train a dnn model. nevertheless, conjectures still need to be experimented. 5 conclusion this paper presented contributions for the problem of forced phonetic alignment (fpa) in brazilian portuguese (bp). an update to ufpalign [6] was offered by providing adapted kaldi recipes for training acoustic models on bp datasets, as well as properly releasing all the acoustic models for free under an open-source license on the github of the falabrasil groupfootnote 3. ufpalign works either via command line (linux) or in a graphical interface as a plugin to praat. up-to-date phonetic and syllabic dictionaries created over a list of 200,000 words for bp are also provided, as well as standalone grapheme-to-phoneme and syllabification systems for handling out-of-vocabulary words. for evaluation, a comparison among the kaldi-based acoustic models trained with an updated version of the scripts from [6] was performed, as well as a comparison to an outdated htk-based version of ufpalign from [32]. results regarding the absolute difference between forced and manual aligned utterances (phone boundary metric) showed that the htk-based aligner performed worse when compared to any of the kaldi-based models, and that our acoustic models we trained from scratch performed better than mfa’s pre-trained models. 5.1 future work as future work, there are a couple of experiments to be investigated. the simplest one would be to train gmm-based tri-\(\updelta \), tri-lda, tri-sat and even monophone-based acoustic models with a higher number of gaussian mixtures per senone. training a dnn on the top of tri-\(\updelta \), since that was the one that yielded the most accurate results according to phone boundary, should be also worth trying. besides, training a dnn on the top of context-independent monophones does not sound so absurd either, given the proximity of the results. regarding the dnn, one thing to verify is whether removing the i-vectors and leaving just normalized mfccs as input features would result in more accurate alignments. splicing cepstral features with lda would also be a valid test. by the way, the tdnn-f setup has not been altered from mini-librispeech’s default recipe, which means some parameters such as layer dimension, number of layers, context width, and the application of frame subsampling could still undergo tuning. finally, other architectures like lstms should have its use evaluated. another idea might be the employment of transfer learning techniques to take advantage of models pre-trained on larger volumes of audio data and just make some adaptations to make it work on our evaluation dataset. that way, an acoustic model trained over librispeech dataset, for example, could be downloaded from openslr [23] to serve as a starting point, and gmm-based models would be trained from scratch over the male/female evaluation dataset to play the role of the new tri-sat reference alignments. one impediment, however, is that most of the pre-trained tdnn-f-based models available on the internet are chain models (i.e., a simplified hmm topology is used to model phonemes), which suggests a new, chain-free model would have to be trained from scratch on english data, which is also freely available. at last, although ufpalign can be used as a plugin to praat, we plan in the future to train models compatible with mfa under the same licensing, as to avoid open-source competition. the provision of a train-and-align feature for ufpalign is also an ongoing plan. notes 1.https://github.com/falabrasil. 2.http://www.kaldi-asr.org/doc/dnn.html. 3.https://github.com/falabrasil. references almeida, j.j., simões, a.: projecto natura (2021). https://natura.di.uminho.pt/wiki/doku.php atkinson, k.: gnu aspell (2021). https://aspell.net boersma, p., weenink, d.: praat: doing phonetics by computer (version 6.1.15) [computer program] (2020). https://www.fon.hum.uva.nl/praat/ buthpitiya, s., lane, i., chong, j.: a parallel implementation of viterbi training for acoustic models using graphics processing units. in: 2012 innovative parallel computing (inpar), pp. 1–10 (2012). https://doi.org/10.1109/inpar.2012.6339590 dehak, n., kenny, p.j., dehak, r., dumouchel, p., ouellet, p.: front-end factor analysis for speaker verification. ieee trans. audio speech lang. process. 19(4), 788–798 (2011). https://doi.org/10.1109/tasl.2010.2064307 article  google scholar  dias, a.l., batista, c., santana, d., neto, n.: towards a free, forced phonetic aligner for brazilian portuguese using kaldi tools. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 621–635. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_44 chapter  google scholar  gibbon, d., moore, r., winski, r.: sampa computer readable phonetic alphabet (2021). https://www.phon.ucl.ac.uk/home/sampa/ github: frequencywords (2020). https://github.com/hermitdave/frequencywords goldman, j.p.: easyalign: an automatic phonetic alignment tool under praat. in: proceedings of the annual conference of the international speech communication association, interspeech, pp. 3233–3236 (2011) google scholar  guiroy, s., de cordoba, r., villegas, a.: application of the kaldi toolkit for continuous speech recognition using hidden-markov models and deep neural networks. in: iberspeech’2016 on-line proceedings, iberspeech 2016, lisboa, portugal, pp. 187–196 (2016) google scholar  huang, x., acero, a., hon, h.w.: spoken language processing: a guide to theory, algorithm, and system development, 1st edn. prentice hall ptr, upper saddle river (2001) google scholar  interinstitutional center for computational linguistics: cetenfolha dataset (2021). https://www.linguateca.pt/cetenfolha/index_info.html jiampojamarn, s., kondrak, g., sherif, t.: applying many-to-many alignments and hidden markov models to letter-to-phoneme conversion. in: human language technologies 2007: the conference of the north american chapter of the association for computational linguistics; proceedings of the main conference, pp. 372–379. association for computational linguistics, rochester, new york (2007). http://www.aclweb.org/anthology/n/n07/n07-1047 vesely, k., ghoshal, a., burget, l., povey, d.: sequence-discriminative training of deep neural networks. in: interspeech 2013, pp. 2345–2349 (2013) google scholar  ko, t., peddinti, v., povey, d., khudanpur, s.: audio augmentation for speech recognition. in: proceedings of interspeech (2015) google scholar  mcauliffe, m., socolof, m., mihuc, s., wagner, m., sonderegger, m.: montreal forced aligner: trainable text-speech alignment using kaldi. in: proceedings of interspeech, pp. 498–502 (2017). https://doi.org/10.21437/interspeech.2017-1386 moura, r.: libreoffice’s vero dictionary (2021). https://github.com/libreoffice/dictionaries/tree/master/pt_br neto, n., patrick, c., klautau, a., trancoso, i.: free tools and resources for brazilian portuguese speech recognition. j. braz. comput. soc. 17(1), 53–68 (2010). https://doi.org/10.1007/s13173-010-0023-1 article  google scholar  neto, n., rocha, w., sousa, g.: an open-source rule-based syllabification tool for brazilian portuguese. j. braz. comput. soc. 21(1), 1–10 (2015). https://doi.org/10.1186/s13173-014-0021-9 article  google scholar  opensubtitles.org: opensubtitles (2021). https://www.opensubtitles.org/ peddinti, v., wang, y., povey, d., khudanpur, s.: low latency acoustic modeling using temporal convolution and lstms. ieee signal process. lett. 25(3), 373–377 (2018). https://doi.org/10.1109/lsp.2017.2723507 article  google scholar  peddinti, v., povey, d., khudanpur, s.: a time delay neural network architecture for efficient modeling of long temporal contexts. in: proceedings of interspeech, pp. 3214–3218 (2015) google scholar  povey, d.: openslr: open speech and language resources (2021). https://openslr.org/index.html povey, d., et al.: semi-orthogonal low-rank matrix factorization for deep neural networks. in: proceedings of interspeech 2018, pp. 3743–3747 (2018). https://doi.org/10.21437/interspeech.2018-1417 povey, d., et al.: the kaldi speech recognition toolkit. in: in ieee 2011 workshop (2011) google scholar  povey, d., et al.: purely sequence-trained neural networks for asr based on lattice-free mmi. in: interspeech 2016, pp. 2751–2755 (2016). https://doi.org/10.21437/interspeech.2016-595 povey, d., zhang, x., khudanpur, s.: parallel training of dnns with natural gradient and parameter averaging (2015) google scholar  schultz, t., vu, n.t., schlippe, t.: globalphone: a multilingual text speech database in 20 languages. in: 2013 ieee international conference on acoustics, speech and signal processing, pp. 8126–8130 (2013). https://doi.org/10.1109/icassp.2013.6639248 shoup, j.e.: phonological aspects of speech recognition. trends speech recogn., 125–138 (1980) google scholar  siravenha, a., neto, n., macedo, v., klautau, a.: uso de regras fonológicas com determinação de vogal tônica para conversão grafema-fone em português brasileiro (2008) google scholar  snyder, d., garcia-romero, d., povey, d., khudanpur, s.: deep neural network embeddings for text-independent speaker verification. in: proceedings of interspeech 2017, pp. 999–1003 (2017). https://doi.org/10.21437/interspeech.2017-620 souza, g., neto, n.: an automatic phonetic aligner for brazilian portuguese with a praat interface. in: silva, j., ribeiro, r., quaresma, p., adami, a., branco, a. (eds.) propor 2016. lncs (lnai), vol. 9727, pp. 374–384. springer, cham (2016). https://doi.org/10.1007/978-3-319-41552-9_38 chapter  google scholar  stolcke, a.: srilm an extensivle language modeling toolkit. in: proceedings of the 7th international conference on spoken language processing (icslp), vol. 2, pp. 901–904 (2002) google scholar  young, s., ollason, d., valtchev, v., woodland, p.: the htk book. cambridge university engineering department, version 3.4, cambridge, uk (2006) google scholar  zhang, x., trmal, j., povey, d., khudanpur, s.: improving deep neural network acoustic models using generalized maxout networks. in: 2014 ieee international conference on acoustics, speech and signal processing (icassp), pp. 215–219 (2014). https://doi.org/10.1109/icassp.2014.6853589 download references acknowledgment we gratefully acknowledge nvidia corporation with the donation of the titan xp gpu used for this research. the authors also would like to thank capes for providing scholarships and fapespa (grant 001/2020, process 2019/583359) for the financial support. author information authors and affiliations computer science graduate program, falabrasil group, federal university of pará, augusto corrêa 1, belém, 66075–110, brazil cassio batista & nelson neto authors cassio batistaview author publications search author on:pubmed google scholar nelson netoview author publications search author on:pubmed google scholar corresponding author correspondence to cassio batista . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper batista, c., neto, n. (2021). experiments on kaldi-based forced phonetic alignment for brazilian portuguese. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_32 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_32 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords forced phonetic alignment speech segmentation acoustic modeling kaldi brazilian portuguese publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a combinatorial optimization model and polynomial time heuristic for a problem of finding specific structural patterns in networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a combinatorial optimization model and polynomial time heuristic for a problem of finding specific structural patterns in networks conference paper first online: 12 october 2023 pp 33–47 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a combinatorial optimization model and polynomial time heuristic for a problem of finding specific structural patterns in networks download book pdf download book epub igor m. sampaio  orcid: orcid.org/0000-0002-0929-47339 & karla roberta p. s. lima  orcid: orcid.org/0000-0001-5483-54669  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 522 accesses abstract the development of tools based on robust mathematical models to deal with real-world, computationally intractable problems has increasingly aligned with combinatorial optimization and integer linear programming techniques due to enormous technological advances and the real need to deal with large volumes of data. in this paper, the maximum tropical path problem on graphs (mtpp), which is known to be np-hard, was investigated. it is a problem of searching for specific structural patterns in networks that can represent various applications, among them biological interactions, such as metabolic, neurological or protein interaction networks. the main result of this work consists of a polynomial-time heuristic algorithm that, according to experimental results, finds good solutions in practice. furthermore, an integer linear programming model was developed and implemented so that it could be compared with the heuristic presented. to conclude, an empirical analysis was performed through computational experiments on both random and real-word instances to evaluate the results presented in this paper. this work was partially supported by grant 2021/07080-6, são paulo research foundation (fapesp) and by the graduate support program (proap)-capes. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others target controllability of linear networks chapter © 2016 fixed parameter algorithms and hardness of approximation results for the structural target controllability problem chapter © 2018 reconfiguring shortest paths in graphs article open access 27 august 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. combinatorics complex networks computational complexity discrete optimization graph theory optimization 1 introduction the tropical path search problem in graphs represents a search problem for specific structural patterns in networks. different types of pattern search have been presented in the literature, taking into account the peculiarities of each application; the one used in this work originates from the problem of occurrence of the motifs in graphs and was proposed by lacroix et al. (2006) [10]. the initial problem was investigated in the context of metabolic network analysis [5, 9], in which the set of reactions involved in the synthesis and degradation of certain molecules is represented through a network and the motifs correspond to the “modules” into which this network can be decomposed in order to facilitate the interpretation of the functions that each of these modules plays in cellular metabolism. in graph theory terms, the problem of searching for colored motifs in graphs consists of finding a subgraph whose set of colors matches the multicolored set given as input. contributions. this work aims to present results in the line of combinatorial optimization for a problem with applications in bioinformatics that represents the problem of searching for structural patterns in graphs. specifically, the main contributions of this work are as follows: polynomial time heuristic algorithm for the mtpp that transforms a simple graph instance into a cactus graph instance and subsequently runs an algorithm that finds the path of interest. formulation of an integer linear programming (ilp) model for, given a graph colored at the vertices, finding a path that maximizes the number of colors used. tests and analysis of the quality and performance of the proposed algorithm based on the ilp model through large instances, both real-world and random. this paper is organized as follows: sect. 2 presents some initial concepts and a brief review of the literature on the tropical path problem in graphs; sect. 3 presents the ilp model formulation; sect. 4 presents the proposed heuristic algorithm; sect. 5 presents the experimentation and results; and sect. 6 presents the concluding remarks. 2 concepts and related work a simple graph g is an ordered pair (v, e), where v is a finite set of elements called vertices and e is a set of elements called edges, where each edge is an unordered pair of distinct vertices (u, v). if \(u,v \in v\) is an edge, then u and v are adjacent. a path in a graph g is a sequence of distinct vertices \(p=(v_1,v_2,\ldots ,v_k)\), such that \(v_iv_{i+1}\in e\) for \(i=1 ,\ldots ,k-1\). a graph g is connected, if for any pair of distinct vertices u and v, there is a path from u to v in g. a coloring of a graph \(g = (v,e)\) is a function \(c: v \rightarrow \mathcal {c}\), where \(\mathcal {c}\) is a set of colors. the coloring defined here corresponds to a simple assigning of colors to the vertices of the graph, without any restrictions. a color graph \(g^c\) consists of a graph g and a coloring c of the vertices of g. given a graph g colored at the vertices, a path p in g is said to be tropical if each color, assigned to some vertex in the input graph, appears at least once in the p. a cactus graph is a connected graph in which any two simple cycles have at most one vertex in common; equivalently, each edge belongs to at most one simple cycle. a cactus g in which a vertex is defined as a root is called a rooted cactus. the root of a cactus g is denoted by root(g). the problem of interest consists of, given a colored graph of vertices, finding a path whose vertices use the greatest number of distinct colors. this problem will be referred to as maximum tropical path problem (mtpp). note that any solution to the mtpp is optimal if the path found is tropical and not every optimal solution is a tropical path (fig. 1). furthermore, if each vertex in the input graph has a color distinct from the others, mtpp reduces to the classical hamiltonian path problem. cohen et al. (2017) [4] proved that mtpp is np-hard even on dags (acyclic directed graph), cactus graphs, and interval graphs, and it can be solved in polynomial time for some specific classes, such as trees, block graphs, proper interval graphs, bipartite chain graphs and threshold graphs. chapelle et al. (2017) [2] proposed exact algorithms for the problem of searching for a tropical subset of minimum size. anglès et al. (2018) [1] studied tropical subsets in graphs from a structural and algorithmic point of view and obtained results on the line of inapproximability. in general the problem of searching for tropical subgraphs in vertex colored graphs has been explored extensively for some classes of graphs; for more details see [1, 3, 6]. fig. 1. an optimal solution with 7 vertices and 6 colors. the instance has 7 colors. see that in the solution it is allowed to repeat colors. full size image 3 an ip model for mtpp in this section, the integer linear programming model developed for the mtpp problem is presented. the initial purpose of developing an integer linear programming model for the mtpp was to explore the combinatorial structures of the problem and obtain algorithms involving branch-and-bound and branch-and-cut techniques, for example. however, the model was satisfactorily used to directly obtain the optimal solution for large instances to be compared to the heuristics proposed in this paper. to formulate the mtpp as an integer linear program, an instance of this problem is considered to consist of a graph colored at the vertices \(g^c = (g, c)\). in the presented formulation, each edge \((u,v) \in e(g)\) is replaced by two directed arcs uv and vu. variables \(x_{uc}\), \(y_{uv}\), and \(z_c\), \(\forall u \in v, \forall uv \in a\), and \(\forall c\in \mathcal {c}\) are used and are associated with vertices, arcs, and colors, respectively. the variable \(x_{u,c}= 1\) indicates that the vertex u of color c is in the solution, the variable \(y_{uv} = 1 \) indicates that the arc uv is in the solution, and the variable \(z_c = 1\) imposes that the color c is in the solution. additionally, the variable \(l_v \ge 0 ~\forall v\in v \) is created, which allows us to number (label) the vertices along the path in increasing order. this variable is used in the constraint proposed by miller, tucker, and zemlin (1960) [12] to eliminate circuits in the solution. 3.1 ilp for general graphs in summary, the objective function maximizes the number of colors that should compose the solution. the constraint (1) prevents the variable \(z_c\) from assuming value 1 without at least one vertex of color c being in the solution. given a vertex v, the constraints (2) and (3) guarantee that if a vertex v is in the solution, the number of incoming and outgoing arcs of v will be at most 1, respectively; the constraint (4) makes use of the function \(l_v\) that allows to number the vertices along the path in increasing order. this inequality is essential to avoid cycles in the solution. the equality (5) guarantees that the difference between the number of vertices and the number of edges that make up the solution is exactly 1. the constraints (6) and (7) guarantee the integrality of variables. $$\begin{aligned} \!\!\!\!\!\!\!\!\!{} & {} & \,\,\,\,\,\,\,\,\,\,\text {max} \, \sum _{c \in \mathcal {c}} z_{c} \nonumber \\ {} & {} \text {s. t.} \, \,{} & {} {} & {} \nonumber \\ {} & {} & z_{c} \le \sum _{u \in v} x_{uc}, {} & {} \forall c \in \mathcal {c} \end{aligned}$$ (1) $$\begin{aligned} {} & {} & \sum _{c \in \mathcal {c}} x_{vc} \sum _{u \in \delta ^-(v)} y_{uv} \ge 0, {} & {} \forall v \in v \end{aligned}$$ (2) $$\begin{aligned} {} & {} & \sum _{c \in \mathcal {c}} x_{vc} \sum _{w \in \delta ^+(v)} y_{vw} \ge 0, {} & {} \forall v \in v \end{aligned}$$ (3) $$\begin{aligned} {} & {} & (l_v l_u) \ge _{} y_{uv} (1 y_{uv})n, {} & {} \forall uv \in a \end{aligned}$$ (4) $$\begin{aligned} {} & {} & \sum _{u \in v} \sum _{c \in \mathcal {c}} x_{uc} \sum _{u \in v} \sum _{v \in v} y_{uv} = 1, {} & {} \end{aligned}$$ (5) $$\begin{aligned} {} & {} & x_{uc}, y_{uv} \in \left\{ 0,1\right\} _{}, {} & {} \forall u \in v, \forall uv \in a, \forall c\in \mathcal {c} \end{aligned}$$ (6) $$\begin{aligned} {} & {} & z_c \in \left\{ 0,1\right\} , l_v \ge 0; {} & {} \forall v\in v ,\forall c \in \mathcal {c} \end{aligned}$$ (7) note that the proposed formulation has a linear amount of inequalities added to the model as the (1) constraints add \(|\mathcal {c}|\) inequalities to the model, while the (2) and (3) constraints each add |v| inequalities, and the (4) constraints add 2|a| inequalities. last, the (5) adds only one inequality to the model, totaling a set of \(|\mathcal {c}| + 2|v| + 2|a| + 1\) linear inequalities. theorem 1 the restricted solution x of the integer solutions (x, y, z) for the formulation from (1) to (7) defines a path on the graph (g, c) that uses \(|\mathcal {c'}|\) colors, where \(\mathcal {c'} \subseteq \mathcal {c}\). proof for any feasible integer solution (x, y, z) of the formulation, let \(v_{p} = \{u: x_{uc}=1\}\), \(a_{p} = \{uv: y_{uv}=1\}\), and \(c_p = \{c: z_c = 1\}\). note that if a vertex \(u\in v_{p}\), then \(c \in c_p\). let \(\mathcal {c'}\) be the set formed by the colors of vertices where \(x_{uc}=1\). if an arc \(uv \in a_p\), by the constraint (2), \(\sum _{c \in \mathcal {c}} x_{vc} = 1\), and by the constraint (3), \(\sum _{c \in \mathcal {c}} x_{uc} = 1\), so \(u, v \in v_p\). by the same conditions imposed in constraints (2) and (3), for each \(u \in v_p\), there is at most one outgoing arc in u and at most one incoming arc in u. note that the subgraph \(g = (v_p, a_p,c_p)\) is well defined but not necessarily connected. if an arc \(uv \in a_p\), by the constraint (4) \(y_{uv}=1\) and the difference between the labels of vertices u and v has to be at least one, which prohibits circuits, finally, the connectivity of g is established by adding the constraint (5) and thus x indicates a path that uses \(\mathcal {c'}\) colors, which completes the proof. although not done in this paper, one way to evaluate the strength of the ilp model would be to compare the value of the integer optimal solution to the value of the optimal solution of the model without the integrality conditions. this is the integrality gap. 4 polynomial-time heuristic for mtpp it is known that mtpp is np-hard for simple graphs in general, especially for dags, cactus graphs, and interval graphs. the inherent intractability of mtpp encourages the development of heuristics that are designed to find bounds, in particular lower bounds, for the problem. in this section, it is presented a polynomial-time heuristic called the tropical-cactus algorithm for finding maximum tropical paths in vertex-colored graphs. for this the algorithm makes use of the algorithm presented in sect. 4.2. existing literature does not provide any exact, approximation, or heuristic algorithms for the mtpp. tropical-cactus algorithm is inspired by a linear-time algorithm proposed by markov et al. (2012) [11] for calculating the longest paths in cactus graphs. cleverly, the algorithm finds a path whose number of colors empirically comes very close to an optimal solution. additionally, an auxiliary algorithm, presented in sect. 4.1 called the cactus-graph algorithm was developed to construct a cactus graph from a simple graph. 4.1 cactus-graph algorithm given a simple graph g, the cactus-graph algorithm extracts from g a cactus graph \(g'\) that will be the input of the tropical-cactus algorithm. to perform this process, it is necessary to find long cycles in the graph. given an integer \(k \ge 3\), a cycle in an undirected graph is called long if it has at least k edges. the cactus-graph algorithm uses as a subroutine the algorithm presented in [7] that finds a path of length \(exp(\varomega (\sqrt{log \ l}))\), where l is the length of the longest path (the details of this algorithm will be omitted here). this subroutine will be called in this paper findcycle(g, v) and receives as input a cactus graph g and a vertex v. extraction step of a cactus graph from a graph g: given a simple graph g, any vertex \(v \in g\), and a positive integer \(k \ge 3\), the constructed graph is defined by \(a_k\). two types of components form \(a_k\). a component in \(a_k\) is either a singular component or a circuit component. a singular component corresponds to a single vertex \(u \in g\) and defines a vertex in \(a_k\), while a circuit component corresponds to a circuit \(c\in g\) and defines a circuit in \(a_k\). the fig. 2 demonstrates an example of transforming a simple graph g into a cactus graph \(a_k\), where k is the minimum circuit size. note that the set of vertices circled by the red line represents a circuit component in the graph \(a_k\) and has a length of 9. the vertices circled by the blue line represent singular components in the graph \(a_k\). note that vertex 7 could be a neighbor of 2, 6, or 11 in the graph \(a_k\), and in a second execution of the algorithm, a circuit of different size could be found. the first vertex to become part of \(a_k\) is the given vertex v, and after the execution of the algorithm, v may be a singular component or become part of a circuit-component in \(a_k\). it is also generally defined in this work that the minimum cycle size k is equal to 3, as such variation did not present a significant change in the final results of the paths found. note that the cactus-graph algorithm is linear, and at each step of the algorithm either a cycle or a vertex is extracted from g which in turn is connected to another cycle or another vertex, maintaining the properties of a cactus graph. fig. 2. example of transforming g into \(a_k\) for \(k \ge 3\) full size image 4.2 labelling algorithm definition 1 given a vertex u in a cactus graph g that is root, the children of u are all vertices adjacent to u or that are in a cycle that contains u. definition 2 given a cactus graph g and a vertex \(u \in v(g)\), \(f^u\) denotes the set of all vertices that are children of u, and \(h^u\subseteq f^u\) denotes the subset of \(f^u\) consisting only of the children of u that belong to some cycle containing u. definition 3 a vertex u is a bridge vertex if u has degree two, belongs to a cycle, and is the child of some other visited vertex. definition 4 a vertex \(u'\), child of a root u, has no children if \(u'\) is a bridge vertex or a leaf vertex. the fig. 3 demonstrates a cactus graph rooted in u, and presents an example of the formation of the sets \(f^u\) and \(h^u\). in the figure, \(f^u\) is highlighted by the red line and, in this example, is formed by the vertices \(f^u = v_1, v_2, v_3, v_4, v_5, v_6, v_7\), while \(h^u\) is highlighted by the blue line and, in this example, is formed by the vertices \(h^u = v_3, v_4, v_5, v_6, v_7\). fig. 3. a rooted cactus graph at u. full size image step of calculating the vertex labels of a cactus graph: by associating an integer value to each vertex of a cactus graph, one obtains a labeled cactus graph. in this algorithm, two labels are associated with each vertex \(u \in v\): the partial label and the total label, both defined below. definition 5 the value of the partial label of u (\(r_p(u)\)) represents the largest number of colors on a path in a cactus subgraph rooted at u. definition 6 the value of the total label of u (\(r_t(u)\)) represents the largest number of colors on a path that contains u and is contained in cactus subgraphs rooted at u. algorithm 1 label-calculation full size image to compute the label of the vertices of the cactus graph, each vertex \(u \in v\) receives a color list l(u) that will initially contain the color of u. this list l, after execution of the algorithm, will contain the set of colors of the vertices of a path of a subgraph with a root at u that contains the greatest number of colors. the label of each vertex, except the leaf and bridge vertices which have labels \(= 1\), is calculated based on the color lists of its children. note that the label-calculation algorithm labels either leaf-type or bridge-type vertices first, and so on until it reaches the root. the algorithm label-calculation is executed for all vertices of the set \(f^{u}\), children of the root u. the vertex label u is calculated only after all vertices in \(f^{u}\) already have a label. it is easy to see that the algorithm can be adapted to return the vertices of the best path found in linear time. 4.3 tropical-cactus algorithm given any vertex u defined as a root of g, the main tropical-cactus algorithm uses the auxiliary label-calculation algorithm to label all vertices of the cactus graph g. for each root \(u'\ne u\), the tropical-cactus algorithm is run, and the vertices of g are given a label. in short, the tropical-cactus algorithm returns the largest found value of the labels \(r_t(u)\) that is calculated for each \(u'\in v\) possible root of g. by the definition of \(r_t(u)\), this is equivalent to the largest number of colors of a found path that passes through u. algorithm 2 tropical-cactus full size image the following result shows the complexity of the tropical-cactus algorithm. the idea in this context is to show that the execution of the algorithm takes polynomial time. lemma 1 the label-calculation algorithm runs in linear time \(\vartheta \left( \left| v\left( g \right) \right| \right) \). proof the algorithm is initially run for the root vertex u. in line 4, the algorithm is called recursively for each child \(u'\) of u, which will either be a bridge or a leaf, or there will be a cactus subgraph rooted in \(u'\), of which u is no longer part, thus the total number of recursive calls of the algorithm is the number of vertices in the graph. additionally, note that the operations in lines 1 and 2 and lines 6–18 take o(1) time, which shows that the algorithm has linear complexity in input size. proposition 1 the tropical-cactus algorithm runs in polynomial time \(\vartheta \left( \left| v\left( g \right) \right| ^2 \right) \) proof note that the tropical-cactus algorithm makes \(\left| v\left( g \right) \right| \) calls to the label-calculation algorithm, one call for each vertex that will be taken as the root of the graph. by the lemma 1, the label-calculation algorithm runs on \(\vartheta \left( \left| v\left( g \right) \right| \right) \) and therefore the result follows. 5 experimental results this section presents an analysis of the experimental results performed in this work. in general terms, the performance of the heuristic algorithm was evaluated in relation to the integer linear programming model for mtpp in general graphs, and in cactus graphs. although the problem is np-hard in all these classes, the idea was to obtain the optimal solution for instances of various dimensions, in terms of number of vertices, edges, and colors, and compare it with the result of the heuristic algorithm for this same set of instances. for this purpose, instances were generated randomly and also obtained from real-world data sets. implementation details. the computational experiments were carried out on a pc with the following specifications: intel(r) core (tm) i7-8550u cpu 1.99 ghz processor, 8 gb of ram, and windows 10 home operating system version 20h2. for the implementation of the lp models, the commercial optimization software gurobi optimizer version 9.5.0 was used. the gurobi parameters were configured with their respective default values. to consolidate the results, all instances were run 10 times, and the final results of each instance were defined by the average of the valid results. benchmark instances. real-world instances were obtained from a dataset repository [8]. in some instances, adaptations were necessary due to the different characteristics of the networks in relation to the problem studied. the adaptations included removing isolated vertices, parallel edges, and loops. to study the impact of varying the number of colors in an instance, three random colorings were set for all networks that did not have a previous coloring. to explore the behavior of both the heuristic algorithm and the ilp model in terms of the number of vertices, edges, and colors, random graphs were generated. for each n vertices, instances were generated with the following variations in the number of edges: \(n + (n/10)\), 2n, \((n*(n-1))/4\), and \((n*(n-1))/3\). additionally, for each number of vertices, three variations in the number of colors were generated, representing 10%, 50%, and 90% of the total number of vertices. it was noted that the ilp model exhibited performance limitations in instances with around 800 vertices, although, an optimal integer solution was obtained for isolated instances with more than 1200 vertices. therefore, for comparative purposes, instances with 200, 400, 600, and 800 vertices were considered. forty-eight random instances will be presented in this analysis, which are identified in the tables 2 and 3. the tables use an index for each variation in the number of vertices, colors, and edges. the index follows the format \(a_{ikj}\), where i represents the variation in the number of vertices, k represents the variation in the number of colors, and j represents the variation in the number of edges. the instance identification table is grouped by the number of vertices |v|, and each set contains the columns: id, number of colors \(|\mathcal {c}|\), and number of edges |e|. since the cactus graph has a limited number of edges compared to a traditional graph, two variations on the number of edges were defined for each cactus graph of n vertices and k colors (table 1). table 1. identification of real-world instances for graphs in generalfull size table table 2. identification of the random instances for graphs in generalfull size table table 3. identifying random instances of cactus graphsfull size table 5.1 computational results the results of the computational experiments for the ilp model and the heuristic algorithm are presented in the following tables: table 4, table 5 and table 6 and present the following nomenclatures, from left to right: the instance (id); the cpu time spent, in seconds, to obtain the integer optimal solution (\(t_{ilp}(s)\)); the value of the integer optimal solution (\(sol_{ilp}\)); the cpu time spent, in seconds, of the heuristic algorithm (\(t_{alg}(s)\)); the value of the solution returned by the heuristic algorithm (\(sol_{alg}\)); and finally the difference in percentage between the integer optimal solution and the approximate solution of the heuristic algorithm (\(gap_{sol}\)). table 4. results of the computational experiments on random instances for graphs in generalfull size table from an individual analysis of the ilp model, it was noted that even though some instances exceeded the proposed execution time limit of 60 min, over \(77\%\) of the executed instances obtained optimal integer solutions in less than seven minutes. some isolated tests obtained an optimal integer solution for instances of up to 1250 vertices, 1125 colors, and 1375 edges. table 5. results of the computational experiments on real-world instances for graphs in generalfull size table from the perspective of random graph instances in general, the heuristic algorithm found the integer optimal solution for \(28.26\%\) of the instances; \(86.96\%\) of the instances presented a gap, that is, a difference, in percent, between the integer optimal solution presented by the ilp model and the solution of the heuristic algorithm, of less than \(30\%\); and \(60.87\%\) of the instances presented a gap of less than \(10\%\). the worst case had a gap of \(38.16\%\). it was also noted that, on average, the solution of the heuristic algorithm diverges from the optimal solution presented by the ilp model by \(11.37\%\). table 6. results of the computational experiments for cactus graphsfull size table for the real instances, the heuristic algorithm found the integer optimal solution in \(42.86\%\) of the instances; \(64.29\%\) of the instances presented a gap less than \(30\%\); and \(50\%\) of the instances presented a gap smaller than \(10\%\). the worst case presented a solution gap of \(56.64\%\). for this data set, the solution of the heuristic algorithm diverges, on average, by \(19.04\%\) from the optimal solution presented by the ilp model. for the random cactus graph instances, the heuristic algorithm found the optimal solution for \(83.33\%\) of the instances; \(100\%\) of the instances presented a gap less than \(10\%\). the worst-case instance had a gap of \(9.80\%\), and on average, this gap was \(1.04\%\). we noticed an improved behavior of the heuristic algorithm on graphs where the number of possible circuits is limited; moreover, in real-world instances, the heuristic algorithm performed better than in random instances, which concludes that in general, the algorithm is a good tool to obtain lower bounds for the mtpp. 6 concluding remarks in this article, a contribution to the state of the art of the problem of tropical paths in graphs is proposed from the perspective of combinatory optimization. specifically, an integer linear optimization model for mtpp has been developed. this model contains, in order, o(nk) variables, being n the number of vertices, and k the number of colors, and o(n) linear inequalities. due to being an np-hard problem, its execution is impractical for large instances. additionally, a polynomial-time heuristic algorithm was also presented, which, as far as it is known, is the first result of this line of research. the differential of the optimization model was the association of two important inequalities that together guaranteed both the connectivity and the absence of circuits in the solution, unlike the commonly used models in which an exponential number of subsets of graphs are exploited for this purpose. the proposed heuristic algorithm is a recursive algorithm that, for a given random root, finds the path with the largest number of colors containing that root. all vertices are visited as roots, and the best path in these terms is returned. the proposed heuristics were tested on a benchmark consisting of 93 instances, among them random and real-world, of various dimensions. the results were compared with the respective entire optimal solutions of the model, and the experimental results pointed out that the average percentage of the difference between the optimal solution and the heuristic algorithm solution is less than \(11\%\), which proves experimentally the strength of the proposed heuristic algorithm. in practice, when faced with computationally intractable problems, it is natural to sacrifice optimality in favor of computational time to obtain a competitive solution. in this regard, the recursive algorithm proved to be an advantageous tool for obtaining good solutions for the mtpp. references anglès d’auriac, j.a., et al.: tropical dominating sets in vertex-coloured graphs. j. disc. algor. 48, 27–41 (2018). https://doi.org/10.1016/j.jda.2018.03.001. https://www.sciencedirect.com/science/article/pii/s1570866718300595 chapelle, m., cochefert, m., kratsch, d., letourneur, r., liedloff, m.: exact exponential algorithms to find tropical connected sets of minimum size. theor. comput. sci. 676, 33–41 (2017). https://doi.org/10.1016/j.tcs.2017.03.003. https://www.sciencedirect.com/science/article/pii/s0304397517301883 cohen, j., manoussakis, y., phong, h., tuza, z.: tropical matchings in vertex-colored graphs. electron. notes disc. math. 62, 219–224 (2017). https://doi.org/10.1016/j.endm.2017.10.038, https://www.sciencedirect.com/science/article/pii/s1571065317302779 cohen, j., italiano, g.f., manoussakis, y., nguyen, k.t., pham, h.p.: tropical paths in vertex-colored graphs. in: gao, x., du, h., han, m. (eds.) cocoa 2017. lncs, vol. 10628, pp. 291–305. springer, cham (2017). https://doi.org/10.1007/978-3-319-71147-8_20 chapter  google scholar  deville, y., gilbert, d.r., van helden, j., wodak, s.j.: an overview of data models for the analysis of biochemical pathways. brief. bioinf. 4(3), 246–259 (2003). https://doi.org/10.1093/bib/4.3.246 article  math  google scholar  foucaud, f., harutyunyan, a., hell, p., legay, s., manoussakis, y., naserasr, r.: the complexity of tropical graph homomorphisms. disc. appl. math. 229(c), 64–81 (2017). https://doi.org/10.1016/j.dam.2017.04.027 article  mathscinet  math  google scholar  gabow, h.n., nie, s.: finding a long directed cycle. acm trans. algor. (talg) 4(1), 1–21 (2008) article  mathscinet  math  google scholar  gephi: conjuntos de dados gephi (2009). https://github.com/gephi/gephi/wiki/datasets. accessed 27 sept 2021 kelley, b.p., et al.: conserved pathways within bacteria and yeast as revealed by global protein network alignment. proc. natl. acad. sci. 100(20), 11394–11399 (2003). https://doi.org/10.1073/pnas.1534710100. https://www.pnas.org/content/100/20/11394 lacroix, v., fernandes, c.g., sagot, m.f.: motif search in graphs: application to metabolic networks. ieee/acm trans. comput. biol. bioinf. 3(4), 360–368 (2006). http://doi.ieeecomputersociety.org/10.1109/tcbb.2006.55 markov, m., ionut andreica, m., manev, k., tapus, n.: a linear time algorithm for computing longest paths in cactus graphs. serdica j. comput. 6(3), 287p–298p (2012) article  mathscinet  math  google scholar  miller, c.e., tucker, a.w., zemlin, r.a.: integer programming formulation of traveling salesman problems. j. acm (jacm) 7(4), 326–329 (1960) article  mathscinet  math  google scholar  download references author information authors and affiliations university of sao paulo usp, sao paulo, brazil igor m. sampaio & karla roberta p. s. lima authors igor m. sampaioview author publications search author on:pubmed google scholar karla roberta p. s. limaview author publications search author on:pubmed google scholar corresponding author correspondence to igor m. sampaio . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sampaio, i.m., lima, k.r.p.s. (2023). a combinatorial optimization model and polynomial time heuristic for a problem of finding specific structural patterns in networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_3 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords optimization combinatorial integer linear programming graphs publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature text classification in legal documents extracted from lawsuits in brazilian courts | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper text classification in legal documents extracted from lawsuits in brazilian courts conference paper first online: 28 november 2021 pp 586–600 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) text classification in legal documents extracted from lawsuits in brazilian courts download book pdf download book epub andré aguiar  orcid: orcid.org/0000-0001-5437-094510, raquel silveira  orcid: orcid.org/0000-0001-7445-605x11, vládia pinheiro  orcid: orcid.org/0000-0002-9851-830410, vasco furtado  orcid: orcid.org/0000-0001-8721-430810 & … joão araújo neto  orcid: orcid.org/0000-0002-0690-244910  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1412 accesses 15 citations abstract recently, brazil’s national council of justice (cnj) highlighted the importance of robust solutions to perform automated lawsuit classification. a correct lawsuit classification substantially improves the assertiveness of (i) distribution, (ii) organization of the agenda of court hearing and sessions, (iii) classification of urgent measures and evidence, (iv) identification of prescription and (v) prevention. this paper investigates different text classification methods and different combinations of embeddings, extracted from portuguese language models, and information about legislation cited in the initial documents. the models were trained with a golden collection of 16 thousand initial petitions and indictments from the court of justice of the state of ceará, in brazil, whose lawsuits were classified in the five more representative cnj’s classes common civil procedure, execution of extrajudicial title, criminal action ordinary procedure, special civil court procedure, and tax enforcement. our best result was obtained by the bert model, achieving 0.88 of f1 score (macro), in the experiment scenario that represents the lawsuit in an embedding formed by concatenating the texts of all the petitions that contain at least one citation to one legislation. legal documents have specific characteristics such as long documents, specialized vocabulary, formal syntax, semantics based on a broad specific domain of knowledge, and citations to laws. our interpretation is that the representation of the document through contextual embeddings generated by bert, as well as the architecture of the model with bidirectional contexts, makes it possible to capture the specific context of the domain of legal documents. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others using topic modeling in classification of brazilian lawsuits chapter © 2022 evaluating text classification in the legal domain using bert embeddings chapter © 2023 impact of text specificity and size on word embeddings performance: an empirical evaluation in brazilian legal domain chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization common tort law escrt legal history machine translation machine learning 1 introduction recently, brazil’s national council of justice (cnj) highlighted in a public call [1] the need for implementing procedural automation with robotization resources and the use of artificial intelligence (ai) techniques, such as machine learning in legal proceedings in brazil. these applications allow the replication of human activity, in order to save human resources that could be relocated to tasks that demand greater creativity and expertise. particularly in the field of jurisdictional efficiency, the cnj calls the ai community in brazil to develop robust solutions to perform automated lawsuit classification. a correct lawsuit classification substantially improves the assertiveness of (i) distribution, (ii) organization of the agenda of court hearing and sessions, (iii) classification of urgent measures and evidence, (iv) identification of prescription and (v) prevention, considering, also, issues such as the categorization in the cnj’s unified procedural tables. the classification of lawsuits is initially assigned by the lawyers and attorneys, manually, and recorded in the initial lawsuit documents. in brazilian justice, these documents are commonly classified as initial petitions (civil lawsuits) and indictment (criminal lawsuits). some benefits of the classification of petitions and indictimens, highlighted by the cnj in [1], are: simplifying procedures for access to justice by citizens and/or their representatives, freeing them from carrying out the procedural classification; reduce the necessary structure of human resources in notary offices and departments responsible for processing the cases, in the different spheres of justice; drastically reduce the need for human intervention in the distribution and processing of cases; provide greater detail in the lawsuit classification, resulting in better organization of the judgment guidelines and opening the way for the development of productivity improvement tools. in order to automate this task, text classification methods should be developed from language models [2, 3] from the natural language processing (nlp) area and learning models [4, 5] from the ai area, and trained from a golden collection of documents extracted from lawsuits. text classification, i.e., the process of assigning one or multiple categories from a set of options to a document [6], is a prominent and well-researched task in natural language processing (nlp) and text mining. text classification variants include simple binary classification (for example, deciding if a document is spam or not spam), multi-class classification (selection of one from a number of classes), and multi-label classification (where multiple labels can be assigned to a single document). text classification methods have been successfully applied to a number of nlp tasks and applications ranging from plagiarism [7] and pastiche detection [8], in order to estimating the period in which a text was published [9], with commercial or forensic goals (e.g. identifying potential criminals [10], crimes [11, 12], or antisocial behavior [13]). however, in the legal domain, to the best of our knowledge, is relatively under-explored, mainly for the brazilian lawsuits. text classification in the legal domain is used in a number of different applications. katz et al. [14] use extremely randomized trees and extensive feature engineering to predict if a decision by the supreme court of the united state would be affirmed or reversed. in a similar fashion, [15] trained a model to predict, given the textual content of a case from the european court of human rights, if there has been a violation of human rights or not. in [16], the authors trained traditional classifiers on text descriptions of cases from the french supreme court, in order to predict with high accuracy the ruling of the french supreme court (six classes), the law area to which a case belongs to (eight classes), and the influence of the time period in which a ruling was made. undavia et al. [17] evaluated a series of classifiers trained on a dataset of cases from the american supreme court. in the brazilian legal domain, araújo et al. [18] present a novel dataset built from brazil’s supreme court digitized legal documents, containing labeled text data and supports two types of tasks: document type classification; and theme assignment, a multilabel problem. specifically for the problem stated by the cnj in [1], we did not find any work that considered initial documents of lawsuits (petitions and complaints), especially those filed in lower courts of brazilian justice. considering the above, this paper investigates different text classification methods and different combinations of embeddings extracted from portuguese language models, like bertimbau [19], and information about legislation cited in the initial documents, validated by a brazilian legal knowledge graph, containing brazilian federal and regional legislation. the models were trained with a golden collection of 16 thousand initial petitions and indictments from the court of justice of the state of ceará, in brazil, whose lawsuits were classified in the five more representative cnj’s classes common civil procedure, execution of extrajudicial title, criminal action ordinary procedure, special civil court procedure, and tax enforcement. the lawsuits categorized in these classes represent more than 80% of lawsuits processed in 2019 in the state of ceará court of justice. in addition, the golden collection contains a class “others” with a mix of initial petitions and indictments from several classes, which were selected to provide negative examples for the learning model. our best result was obtained by the bert model, achieving 0.88 of f1 score (macro), in the experiment scenario that represents the lawsuit in an embedding formed by concatenating the texts of all the petitions that contain at least one citation to one legislation. legal documents have specific characteristics such as long documents, specialized vocabulary, formal syntax, semantics based on a broad specific domain of knowledge, and citations to laws. our interpretation is that the representation of the document through contextual embeddings generated by bert, as well as the architecture of the model with bidirectional contexts, makes it possible to capture the specific context of the domain of legal documents. we also emphasize that by representing a lawsuit with only the text of petitions that have legislation, we are reducing the size of the text, but keeping only the excerpts that present reasons for the content of the lawsuit, which are therefore relevant for the classification of lawsuits. 2 related works text classification methods are investigated and applied with commercial or forensic goals (e.g. identifying potential criminals [10], crimes [11, 12], or antisocial behavior [13]). however, in the legal domain, these methods have been under-explored, mainly in brazilian lawsuits. katz et al. [14] use extremely randomized trees and extensive feature engineering to predict if a decision by the supreme court of the united state would be affirmed or reversed, achieving an accuracy of 69.7%. aletras et al. [15], in a similar fashion, trained a model to predict, given the textual content of a case from the european court of human rights, if there has been a violation of human rights or not. the paper employed ngrams and topics as inputs to a svm, reaching an accuracy of 79%. sulea et al. [16] trained a linear svm on text descriptions of cases from the french supreme court in order to predict with high accuracy the ruling of the french supreme court and the law area to which a case belongs to. the authors also investigate the influence of the time period in which a ruling was made. they report results of 98% average f1 score in predicting a case ruling, 96% f1 score for predicting the law area of a case, and 87.07% f1 score on estimating the date of a ruling. undavia et al. [17] proposed to classify us supreme court documents into fifteen different categories, comparing various combinations between feature representation and classification models, with better results through the application of convolutional neural networks (cnn) to word2vec [3] representations and was able to achieve an accuracy of 72.4% when classifying the cases into 15 broad categories and 31.9% when classifying over 279 finer-grained classes. in the brazilian legal domain, araújo et al. [18] present a novel dataset built from brazil’s supreme court digitized legal documents, containing labeled text data and supports two types of tasks: document type classification; and theme assignment, a multilabel problem. a similar model was applied to documents received by the supreme court in brazil and reported in silva et al. [20], which obtained a classifier using an embedding layer and a cnn applied to a problem involving six classes. 3 corpus and data preparation in this section, we describe the preparation and structuring of data used in the experiments that will be demonstrated in this paper. initially, we present the dataset collected from lawsuits of the court of justice of the state of ceará, in brazil. then, we presented the knowledge graph of federal and regional legislations, the named entity recognition (ner) of these legislations and the validation process of legislations recognized in the legal documents. 3.1 corpus and golden collection data processing was divided into phases, consisting of: (i) data cleaning phase, and (ii) data extraction phase. at the end of the data cleaning and extraction phases, a sample was produced with the lawsuits completed in 2019. during the data cleaning phase, corrupt and inaccurate records were detected in their metadata, where an action to correct or remove these records was taken. in the metadata we can obtain the information of document date, document type, lawsuit number and lawsuit page. these data were relevant to organize the data extraction phase. in the data extraction phase, amazon’s optical character recognition (ocr) service was used, which refers to the branch of computer science that allows the translation of an optically digitized image of a printed or written text into manageable and editable text. we identified some errors after the data extraction phase, such as: special characters and additional spacing. with this, we carry out a data cleaning process to minimize these errors. finally, 7,103 lawsuits were selected, of which 16,668 petitions were used to build the dataset, distributed in 6 selected classes, which were defined by experts of the court of justice of the state of ceará, in brazil. these classes express the type of lawsuits and were defined by the national council of justice of brazilfootnote 1. 3.2 integration with the brazilian legal knowledge graph a characteristic of legal documents is that they present citations to legislation, which are used to argue and present legal theories and arguments about the legal themes supporting the lawsuit. considering that legislation can be important for classifying lawsuits, we use a named entity recognizer (ner) to identify legislation in the text of the lawsuits. more specifically, we use a nerfootnote 2, structured in the conditional random field model (crf) classifier of the stanford javanlp apifootnote 3 and trained with the lener-br dataset [21] (a dataset for named entity recognition in brazilian legal text). additionally, a knowledge graph was built with federal and regional legislation of the state of ceará, in brazil. the laws were taken from the federal government legislation portalfootnote 4, legislative and legal information networkfootnote 5 and from the legislative assembly of cearáfootnote 6. table 1 demonstrates the legislation types, year of creation and quantity of legislation in the actual brazilian legal knowledge graph. table 1. statistics and information about the brazilian legal knowledge graph.full size table the legislations extracted from the texts of the lawsuits by ner were validated in the knowledge graph of legislation, so that we consider only those present in the knowledge base as an entity “legislation”. from the identification and validation of the “legislation” entities in the text of the lawsuits, the lawsuits that have at least one citation to the legislation were selected, totaling 6,283 lawsuits, with 11,131 petitions in all. in table 2 we can see the name of the lawsuit classes of the dataset and the number of the national council of justice of brazil (informed in parentheses in the class column), and the distribution of dataset lawsuits in the 6 classes mentioned in sect. 3.1. table 2. distribution of lawsuits in legal classes.full size table we investigate the legislation cited in the lawsuits for each class individually. table 3 shows the amount of different legislations mentioned and in how many classes these legislations occur. table 3. number of classes and the amount of legislation mentioned differently.full size table in the text of the lawsuits’ petitions, were identified 110,044 citations to brazilian legislation. according to table 3, these citations represent 895 different brazilian laws and legal norms. we verified that only 1.2% (10 legislations) of these legislations are mentioned in lawsuits of all classes, being characterized as more generic legislations that do not distinguish the class of lawsuit, such as the brazilian civil code (law 10.406, of january, 01/10/2002), brazilian criminal code (decree-law 2,848, of 07/12/1940), code of civil procedure (law 13.105, of 03/16/2015) and national tax code (law 5.172, of 10/25/1966). while, 71.84% (643 legislations) are cited in lawsuits of a single class, which can be considered legislations that characterize the theme and, consequently, the class of the lawsuit. as, for example, law 970, of 12/16/1949, which provides for the attributions, organization and functioning of the national economy council, is mentioned only in the lawsuits of the “common civil procedure” class. table 4 shows the total of citations of legislation, the total of distinct legislations that occur in citations and the total of distinct legislations that occur by legal class. we analyzed the amount of different legislation mentioned in each of the classes of lawsuits and verified that there are laws that occur only in each class. for example, the 59,720 citations to the legislation present in the lawsuits of the “common civil procedure” class correspond to 536 different legislations. in cases of this class, law 8,078, of 09/11/1990, for example, is cited 8,560 times. we emphasize that, of the 536 distinct legislations cited in this class, 329 legislations are cited only in this class. likewise, for the “criminal action ordinary procedure” class, the 12,610 citations that occur in this class correspond to 125 different legislations. for example, law 11,343, of 23/08/2006, is cited 1,910 times in lawsuits of this class. we highlight that 43 different legislations are mentioned only in this class. table 4. distribution of citations of legislation in total numbers and distinctly by legal class.full size table to complement the interpretation of citations in the lawsuits, we created a graph (shown in fig. 1) that shows the relationship of the most frequent legislations with the lawsuits in which these legislations are cited. the nodes of this graph represent a law or a lawsuit and the edges represent the citation of the law in the lawsuit. we reinforce that some legislations are cited in many lawsuits, while other legislations are cited in few lawsuits, which suggests that the legislations cited in petitions can help in the identification of the class of lawsuits. fig. 1. graph with nodes being legislations or lawsuits and edges being a lawsuit that cites legislation. full size image 4 lawsuit classification in this section, initially, we define the scenarios and algorithms used in the experiments for classifying lawsuits. then, we present and discuss the results achieved. 4.1 experiment scenarios lawsuits classification experiments were conducted in different scenarios. in each scenario, we choose different features to represent a lawsuit. below we present each scenario used in the experiments. embeddings of the lawsuit/case text (s1). each lawsuit is made up of a set of petition-type text documents. we define that the text of a lawsuit corresponds to the concatenation of the texts of all the petitions that comprise it. in this scenario, the lawsuit is represented by the embedding of the lawsuit text, generated from the pre-trained model bertimbau [19] (a pre-trained model bert for brazilian portuguese). embeddings of the lawsuit/case text with citation (s2). we assume that petitions that have citations to the legislation are more relevant to the lawsuit. in this way, for each petition in a lawsuit, we verify if it contains a citation for brazilian legislation (according to the procedure described in sect. 3.2). then, a text is formed by concatenating the texts of all the petitions that contain at least one citation to one legislation. in this scenario, the lawsuit is represented by the embedding of this text, generated from the pre-trained bertimbau model [19]. embeddings of the cited laws (s3). we assume that in the case of legal documents, legislations are cited to represent and substantiate the subject of the document, therefore, the content of these legislations is relevant to define the class of a lawsuit. in this context, a text is formed by the concatenation of the legislation cited in the petitions of a lawsuit. in this scenario, the lawsuit is represented by the embedding of the text of the summary of the cited legislations (these summaries were retrieved from brazilian legal knowledge graph), generated from the pre-trained model bertimbau [19]. embeddings of the lawsuit/case text with citation and cited laws (s4). we assume that the laws cited are as important as the text of the lawsuit. in this scenario, a text is formed by the concatenation of the texts of all the petitions that contain at least one citation to the legislation and the texts of the summary of the cited legislations. the embedding of this text, generated from the pre-trained model bertimbau [19], is used to represent the lawsuit. embeddings of the lawsuit/case text with citation and embeddings of cited laws (s5). in this scenario, a lawsuit is represented by two embeddings: (i) embedding of the lawsuit text, generated according to s2, and (ii) embedding of the summary of the cited legislations, generated according to s3. tf-idflaw (s6). we assume that the relevance of cited legislations in lawsuit texts is an important information for the learning model: legislation cited only in lawsuits of the same class should be considered more important than legislation cited in lawsuits of several classes. based on this, we define a feature named tf-idflaw, a variant of tf-idf (term frequency inverse document frequency), which aims to measure the relevance of legislation for a lawsuit, according to the frequency of citations of legislation in the lawsuit and in other lawsuits in the corpus. tf-idflaw(l,d) is calculated according to eq. (1) below: $$ tf idf_{law} \left( {l,d} \right) = \frac{{f_{l, d} }}{{\sigma_{ l \in d} f_{l, d} }} \times log\frac{n}{{\left| {\left\{ {d \in d : l \in d} \right\}} \right|}} $$ (1) where fl,d, the frequency of citation of the law l in the lawsuit d, is divided by the total number of citations to the law in the lawsuit. d. then, the number of lawsuits n in the corpus d is divided by the total frequency of the law l in all lawsuits of the corpus. in this scenario, we represent a lawsuit as a bag-of-words with tf-idflaw features. in other words, given a vocabulary v = {cl1, cl2, …, clv} formed by the laws mentioned in the n lawsuits of the train set, we represent each lawsuit di as a length-|v| vector l(di), where l(di)j = tf-idflaw (clj, di), with j = 0 to |v|–1 and \({cl}_{j}\in v\). embeddings of the lawsuit/case text with citation and tf-idflaw (s7). in this scenario, a lawsuit is represented by: (i) embedding of the lawsuit text with citation, generated according to s2, and (ii) bag-of-words with tf-idflaw features, generated according to s6. embeddings of the cited laws and tf-idflaw (s8). in this scenario, a lawsuit is represented by: (i) embedding of the menus of the aforementioned laws, generated according to s3, and (ii) bag-of-words with tf-idflaw features, generated according to s6. embeddings of the lawsuit/case text with citation and cited laws and tf-idflaw (s9). in this scenario, a lawsuit is represented by: (i) embedding of the lawsuit text and the summary of the cited laws, generated according to s4, and (ii) bag-of-words with tf-idflaw features, generated according to s6. embeddings of the lawsuit/case text with citation, embeddings of cited laws and tf-idflaw (s10). in this scenario, a lawsuit is represented by: (i) embedding of the lawsuit text, generated according to s2, (ii) embeddings of the summary of the cited laws, generated according to s3, and (iii) bag-of-words with tf-idflaw features, generated according to s6. embeddings of the lawsuit/case text and topics (s11). given a set of text documents, a topic model is applied to find out interpretable semantic concepts, or topics, present in documents. we assume that the topics can suggest the main theme of the document, making it possible, therefore, to help in inferring the class of lawsuits. we choose latent dirichlet allocation (lda) [22] as the method for topic generation of the lawsuits. lda is a probabilistic model generator of a corpus, where each document is represented as a mixture of latent topics. each topic is, in turn, a distribution of words. in this way, we run lda on corpus d to find 6 topics (we set the number of topics as the same number of classes). to each document, we choose the most likely topic for each lawsuit and the top-10 words more representative to the lawsuit’s topic. in this scenario, each lawsuit is represented by: (i) embedding of the lawsuit text, generated according to s1, and (ii) bag-of-words with the word distribution of the topics. 4.2 models we train four models on split-of-the-data training in the scenarios presented in sect. 4.1: (i) random forest [23], (ii) support vector machine (svm) [24], (iii) extreme gradient boosting (xgboost) [25], and (iv) bidirectional encoder representations from transformers (bert)footnote 7 [2]. we train each model as a multiclass classification task, using the following parameters (obtained from an empirical evaluation): random forest. we applied the random forest algorithm using scikit-learn’s random forest classifier packagefootnote 8. we trained 1,000 trees with a maximum of 200 features to consider when looking for the best split. support vector machine (svm). we applied the svm algorithm using scikit-learn’s svm packagefootnote 9 with default parameters. extreme gradient boosting (xgboost). we applied the xgboost algorithm using xgboost classifier packagefootnote 10. we trained 1,000 trees with a maximum depth of 4 and a learning rate of 0.1. bidirectional encoder representations from transformers (bert). we used the fine-tuning-based approach with the pre-trained model bertimbau [19], 4 epochs and batch size of 4 samples for the classification task. for the optimizer, we used the adam optimizer with a learning rate of 1e − 5. 4.3 results and discussion to carry out the experiments in the scenarios and models described in sects. 4.1 and 4.2, we divided the dataset into 85% for the train set and 15% for the test set. table 5 shows the performance of the models in each experiment scenario, in terms of f1 score (macro) for the test set. table 5. results in terms of f1 score macro for lawsuit classification in different scenarios and models.full size table based on the kruskal-wallis statistical test, we are able to claim that the results are statistically significant at a 99% confidence level, with p-value equal to 0.0008. the bert model outperforms the other models in all scenarios, achieving the best result of 0.88 of f1 score (macro) in the experiment scenario that represents the lawsuit from the petition texts that have citation. legal documents have specific characteristics such as long documents, specialized vocabulary, formal syntax, semantics based on a broad specific domain of knowledge, and citations to laws. our interpretation is that the representation of the document through contextual embeddings generated by bert, as well as the architecture of the model with bidirectional contexts, makes it possible to capture the specific context of the domain of legal documents. we also emphasize that by representing a lawsuit with only the text of petitions that have legislation citation, we are reducing the size of the text, but keeping only the excerpts that present reasons for the content of the lawsuit, which are therefore relevant for the classification of lawsuits. the second best model is xgboost in s7, s9 and s10 scenarios. these scenarios use the feature tf × idflaw, in addition to the embeddings of the texts of the petitions with legislation (in s7, s9 and s10) and the summary of the cited legislation (in s9 and s10). the characterization of the lawsuit with the petitions that have citations is complemented by the relevance of the legislation for the lawsuits, through the feature tf × idflaw. this suggests that the legislation cited informs about the class, helping to classify the lawsuits. finally, in order to evaluate the results in a cross-validation approach, we employ a stratified 5-fold cross-validation setup for the best result experiment (model bert with scenario s2) and obtained the f1 macro score of 0.88 (the same result for the dataset divided into 85% for training and 15% for testing). figure 2 shows the confusion matrix for the best result achieved in the experiments, a scenario with the representation of text embeddings of the petitions of the lawsuits that have citations with the model trained with bert. we observe some patterns which may help us understand these results. the confusion matrix shows us that the lawsuits of the “tax enforcement” class have the best results, with a recall close to 1.0, while the lawsuits of the “others” class are the most difficult to predict, with a recall close to 0.71. lawsuits of class “others” correspond to lawsuits of mixed classes grouped into a single class, thus making it difficult to classify these lawsuits. the second most difficult class to predict is the “execution of extrajudicial title” class. this class contains the smallest number of samples in the dataset. fig. 2. confusion matrix for the lawsuit classification in the best-case scenario and model (model bert with scenario s2). full size image table 6 presents the f1 score results of the bert model trained with the texts of the petitions of the lawsuits without (s1) and with (s2) citation to the legislation, for each class. table 6. results in terms of f1 score for bert model in s1 and s2 experiments scenario, considering the text of the lawsuits with and without citations to legislation, respectively.full size table the classes’ f1 scores show variability from 0.78 to 1.00 in both scenarios. s2 scenario presents better f1 score results for the “execution of extrajudicial title” and “tax enforcement” classes, with an increase of 3.6% and 2.0%, respectively. whereas the s1 scenario has better results for the “special civil court procedure” and “criminal action ordinary procedure” classes, with an increase of 2.4% and 1.5%, respectively. in the other classes, “common civil procedure” and “others”, the f1 score values are the same for both scenarios. 5 conclusions in this paper we investigate the application of different models and scenarios for classifying legal texts using a dataset of lawsuits from the court of justice of the state of ceará, in brazil. the best results are achieved by the bert model (using the pre-trained bertimbau model [19]) in the scenario in which the lawsuit is represented by the text of the petitions that have citations to brazilian legislation. the legal text has specific characteristics, in this way, we represent the text contextually from the bertimbau (pre-trained model for brazilian portuguese) and provide only the petitions of the lawsuit that mention a brazilian law. we argue that the contextual representation of the text and the citation to legislation help to identify the classes of a lawsuit. as future work, we intend to investigate other features to characterize the lawsuits and to use other models, such as recurrent neural network (rnn) and convolutional neural network (cnn) for lawsuit classification. we also intend to develop a specific language model for the brazilian legal domain, which can be used to improve the contextual representation of the texts of proceedings in the specific domain of the legal area. we also intend to investigate the accuracy of the model in lawsuits from other courts of justice in different brazilian states. notes 1.https://www.cnj.jus.br/sgt/consulta_publica_classes.php 2.https://github.com/mpmg-dcc-ufmg/m02 3.https://nlp.stanford.edu/nlp/javadoc/javanlp/edu/stanford/nlp/ie/crf/crfclassifier.html 4.http://www4.planalto.gov.br/legislacao/ 5.https://www.lexml.gov.br/ 6.https://www.al.ce.gov.br/index.php/tividades-legislativas/leis 7.given the specifics of the bert’s original architecture, this model was trained only in scenarios: (s1), (s2), (s3) and (s4). 8.https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.randomforestclassifier.html 9.https://scikit-learn.org/stable/modules/generated/sklearn.svm.svc.html 10.https://xgboost.readthedocs.io/en/latest/python/python_api.html references conselho nacional de justiça: convocação nº 01/2021 – desenvolvimentopiloto de soluções para a automação processual e uso de técnicas de inteligência artificial no poder judiciário. https://acessoexterno.undp.org.br/public/jobs/18062021164751_resultado%20para%20publica%c3%a7%c3%a3o_sinapses.pdf. accessed 20 june 2021 devlin, j., chang, m.-w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, vol. 1 (long and short papers), pp. 4171–4186. association for computational linguistics, minneapolis (2019) google scholar  mikolov, t., chen, k., carrado, g., dean, j.: efficient estimation of word representations in vector space. http://arxiv.org/pdf/1301.3781.pdf. accessed 20 nov 2015 hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(12), 1735–1780 (1997) article  google scholar  conneau, a., schwenk, h., barrault, l., lecun, y.: very deep convolutional networks for text classification. in: proceedings of the 15th conference of the european chapter of the association for computational linguistics, vol. 1, long papers, pp. 1107–1116. association for computational linguistics, valencia, spain (2017) google scholar  shaheen, z., wohlgenannt, g., filtz, e.: large scale legal text classification using transformer models. arxiv preprint arxiv:2010.12871 (2020) barrón-cedeño, a., vila, m., martí, m.a., rosso, p.: plagiarism meets paraphrasing: insights for the next generation in automatic plagiarism detection. comput. linguist. 39(4), 917–947 (2013) article  google scholar  dinu, l.p., niculae, v., sulea, o.-m.: pastiche detection based on stopword rankings: exposing impersonators of a romanian writer. in: proceedings of the workshop on computational approaches to deception detection (2012) google scholar  niculae, v., zampieri, m., dinu, l.p., ciobanu, a.m.: temporal text ranking and automatic dating of texts. in: proceedings of eacl (2014) google scholar  sumner, c., byers, a., boochever, r., park, g.j.: predicting dark triad personality traits from twitter usage and a linguistic analysis of tweets. in: proceedings of icmla (2012). https://doi.org/10.1109/icmla.2012.218 pérez-rosas, v., mihalcea, r.: experiments in open domain deception detection. in: lluís, m., chris, c.b., jian, s., daniele, p., yuval, m. (eds.) proceedings of emnlp. association for computational linguistics (2015). https://aclweb.org/anthology/d/d15/d15-1133.pdf pinheiro, v., pequeno, t., furtado, v., nogueira, d.: information extraction from text based on semantic inferentialism. in: andreasen, t., yager, r.r., bulskov, h., christiansen, h., larsen, h.l. (eds.) fqas 2009. lncs (lnai), vol. 5822, pp. 333–344. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-04957-6_29 chapter  google scholar  cheng, j., danescu-niculescu-mizil, c., leskovec, j.: anti-social behavior in online discussion communities. in: proceedings of icwsm (2015) google scholar  katz, d.m., bommarito, m.j.i., blackman, j.: predicting the behavior of the supreme court of the united states: a general approach. in: arxiv e-prints, page arxiv:1407.6333 (2014) aletras, n., tsarapatsanis, d., preotiuc-pietro, d., lampos, v.: predicting judicial decisions of the european court of human rights: a natural language processing perspective. peer j. comput. sci. 10 (2016) google scholar  sulea, o.m., zampieri, m., vela, m., vangenabith, j. predicting the law area and decisions of french supreme court cases. in: proceedings of the international conference recent advances in natural language processing, ranlp, pp. 716–722. incoma ltd. (2017) google scholar  undavia, s., meyers, a., ortega, j.e.: a comparative study of classifying legal documents with neural networks. in: federated conference on computer science and information systems (fedcsis), pp. 515–522 (2018) google scholar  araújo, p.h.l., campos, t.e., braz, f.a.; silva, n.c.: victor: a dataset for brazilian legal documents classification. in: proceedings of the 12th conference on language resources and evaluation (lrec 2020), 11–16 may, pp. 1449–1458. marseille (2020) google scholar  fabio, s., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, 20–23 october. rio grande do sul, brazil (2020) google scholar  silva, n., braz, f., campos, t.: document type classification for brazil’s supreme court using a convolutional neural network. in: the tenth international conference on forensic computer science and cyber law-icofcs, vol. 10, pp. 7–11 (2018) google scholar  luz de araujo, p.h., de campos, t.e., de oliveira, r.r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: villavicencio, a., et al. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 313–323. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_32 chapter  google scholar  blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3, 993–1022 (2003) math  google scholar  breiman, l.: random forests. mach. learn. 45(1), 5–32 (2001) article  google scholar  hearst, m.a.: support vector machine. ieee intell. syst. 13(4), 18–28 (1998) article  google scholar  chen, t., guestrin, c.: xgboost: a scalable tree boosting system. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794. new york (2016) google scholar  download references author information authors and affiliations university of fortaleza, fortaleza, brazil andré aguiar, vládia pinheiro, vasco furtado & joão araújo neto federal institute of education, science and technology of ceará, fortaleza, brazil raquel silveira authors andré aguiarview author publications search author on:pubmed google scholar raquel silveiraview author publications search author on:pubmed google scholar vládia pinheiroview author publications search author on:pubmed google scholar vasco furtadoview author publications search author on:pubmed google scholar joão araújo netoview author publications search author on:pubmed google scholar corresponding author correspondence to raquel silveira . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper aguiar, a., silveira, r., pinheiro, v., furtado, v., neto, j.a. (2021). text classification in legal documents extracted from lawsuits in brazilian courts. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_40 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_40 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords text classification legal domain brazilian lawsuits language models publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature rhetorical role identification for portuguese legal documents | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper rhetorical role identification for portuguese legal documents conference paper first online: 28 november 2021 pp 557–571 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) rhetorical role identification for portuguese legal documents download book pdf download book epub roberto aragy  orcid: orcid.org/0000-0002-4872-704410,11, eraldo rezende fernandes  orcid: orcid.org/0000-0002-7039-244710 & edson norberto caceres  orcid: orcid.org/0000-0001-6471-346210  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1286 accesses 4 citations 7 altmetric abstract in this paper, we present a new corpus for rhetorical role identification in portuguese legal documents. the corpus comprises petitions from 70 civil lawsuits filed in tjms court and was manually labeled with rhetorical roles specifically tailored for petitions. since petition documents are created without a standard structure, we had to deal with several issues to clean the extracted textual content. we assessed classic and deep learning machine learning methods on the proposed corpus. the best performing method obtained an f-score of 80.50. at the best of our knowledge, this is the first work to deal with rhetorical role identification for petitions, given that previous works focused only on judicial decisions. additionally, it is also the first work to tackle this task for the portuguese language. the proposed corpus, as well as the proposed rhetorical roles, can foster new research in the judicial area and also lead to new solutions to improve the flow of brazilian court houses. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ulysses tesemõ: a new large corpus for brazilian legal and governmental domain article 18 july 2024 appellate court modifications extraction for portuguese article 12 july 2019 building a relevance feedback corpus for legal information retrieval in the real-case scenario of the brazilian chamber of deputies article 18 august 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics discourse research discourse analysis escrt speech act theory 1 introduction brazilian judicial system has usually adopted a conservative stance regarding its lawsuit bureaucracy. however, in the last two decades, it started to adopt information technology systems to improve its underlying procedures. in 2006, brazilian congress passed a lawfootnote 1 to rule electronic lawsuit systems. since then, all brazilian courts started to digitize their lawsuit bureaucracy. this process has greatly accelerated the filing and intermediary procedures of a lawsuit which, for instance, allowed a 22% increase in the number of cases filed in 2019 when compared to 2009 [18]. although this effect is mostly beneficial, it presents its own challenges. one issue is that the main bottleneck of brazilian judicial system has shifted from the intermediary procedures performed by civil registries to judges’ offices. nowadays, most documents in a lawsuit in brazil are either originally digital or digitized before inclusion in the judicial system. nevertheless, several tasks within a lawsuit procedure still depend on direct human intervention. in general, every time a party (judge, complainant, defendant, etc.) is requested to pronounce within a case, this party may need to carefully analyse different documents. in particular, the petition (petição inicial, in portuguese) is the first document presented by the complainant in order to file a civil lawsuit in brazilian courts. in this document, the complainant needs to identify the involved parties, expose facts that are relevant to the case, present the legal basis and relevant prior cases, describe the requested remedy, among other aspects. after a petition is filed, it needs to be carefully analysed by staff of a judge’s office in order to verify different aspects. for instance, it is necessary to check if the petition presents the minimal requirements specified in the brazilian civil procedure (código de processo civil) [4]. ultimately, the petition must be fully considered by the assigned judge, when another careful analysis is required. although it is virtually impossible to mechanize all these manual analytic steps, in most of these cases if not all, machine learning (ml) and natural language processing (nlp) techniques can be leveraged to assist humans in such analyses. rhetorical role identification (rri) is an nlp task that consists of labeling the sentences of a document according to a given set of semantic functions (rhetorical roles). this task is useful to applications like document summarization [11, 19,20,21, 26], semantic search [15], document analysis [22], among others. different approaches have been proposed to solve this task for court decisions [2, 20, 21]. these previous works have proposed to segment a court decision into rhetorical roles such as facts, court arguments, prior related cases, ruling, among others. as far as we know, all previous rri methods have tackled only court decisions and are limited to the english language. although the benefits of analytic tools of judicial decisions are undisputed, it is also important to provide similar tools for other legal documents and languages, particularly petitions in brazilian legal system. as mentioned before, a judge’s office in brazil handle a high volume of work, and petition analysis represents a substantial portion of it. therefore, rhetorical role identification for petitions represents an opportunity to improve brazilian judges’ offices efficiency. in this paper, we propose to segment petitions into eight rhetorical roles by mainly considering the analytic needs of judges’ offices in brazil. we present a corpus of 70 petitions comprising more than 10 thousand sentences manually labeled with the proposed rhetorical roles. these petitions were taken from civil lawsuits filed in the court of the brazilian state of mato grosso do sul (tjms). additionally, we report on experiments to assess classic and modern machine learning methods on the created corpus. in order to propose relevant rhetorical roles, we considered the brazilian civil procedure [4] which defines the minimal requirements of a petition. the proposed rhetorical roles are: identification of parties, facts, arguments, legal basis, precedents, requests, remedy, and others. the corpus was anonymized and is publicly available. we trained and evaluated six machine learning methods, including classic and deep learning models, on the developed corpus. the best performing method was an end-to-end-trained bert-based [6] method that obtained a sentence-level micro-averaged f-score of 80.50. this method outperformed a fixed-representation bert-based model by five points and the best classic algorithm (svm with tf-idf representation) by 25 points of f-score. as far as we know, this is the first paper to approach rhetorical role identification of petitions, instead of court decisions. moreover, this is also the first work on this task that considers the portuguese language and legal domain. we hope this paper will foster research about this important subject, especially in brazil. 2 related work one of the first works to deal with text segmentation based on rhetorical function was proposed by teufel [25]. in this work, teufel proposed a segmentation approached called argumentative zoning that is based on the rhetorical function that an excerpt plays within a scientific article. this approach was later applied to extractive summarization [5, 12, 26]. hachey and grover [11], inspired by teufel’s work, proposed a method to legal text summarization based on rhetorical role text segmentation. the proposed roles were: fact, proceedings, background, framing, disposal, textual, and other. they assessed the proposed method on the holj dataset [10] which consists of 40 manually-labeled decisions from the house of lords, the highest court in the united kingdom until 2009. they employed a c4.5 classifier that achieved an accuracy of \(65.4\%\). saravanam [19] also followed teufel’s work to study the generation of headnotes (a type of summary) for decisions from indian courts. he argued that rhetorical roles proposed by teufel were not appropriate in the legal domain. thus, he proposed the following rhetorical roles for legal decisions: case identification, facts, arguments, chronology, argument analysis, ratio decidendi, and final decision. later, based on these roles, saravanam et al. [21] created a corpus out of 200 decisions from the indian court of justice spanning three different domains. they developed and evaluated a conditional random field (crf) classifier to perform sentence-level rhetorical role identification which obtained the following f-scores on the considered domains: 84.9 on rent control, 81.7 on income tax, and 78.7 on sales tax domain. nejadgholi et al. [15] leveraged rhetorical role identification to perform semantic search. they defined eight rhetorical roles and manually labeled the sentences in 150 decisions taken from immigration and refugee cases from canada’s federal and supreme court. they assessed a simple classifier based on fasttext that obtained an accuracy of 90%. walker et al. [28] proposed six rhetorical roles to aid a system for argument mining. they produced a corpus of 50 decisions extracted from the u.s. board of veterans’ appeals comprising 6,153 sentences. they evaluated three different classification algorithms on this sentence classification corpus. the best performing classifiers (based on logistic regression and svm) achieved 85.7% accuracy. bhattacharya et al. [2] annotated 50 decisions from the supreme court of india using the same rhetorical roles proposed by saravanam [19]. they trained a hierarchical bilstm model with crf on this corpus, obtaining an f-score of 82.0. yamada et al. [29] leveraged the argumentation structure in legal decisions from japanese justice system to improve a summarization system. they manually annotated a corpus of 209 decisions based on seven rhetorical roles. by means of bilstm-crf model, they obtained a macro-averaged f-score of 65.4 on this corpus. all papers mentioned above are based on legal decisions. although this kind of documents is of high importance, there are other types of documents that are also relevant to different applications in the legal domain. another aspect that is worth mentioning is that all previous works on rhetorical role identification were performed on documents from legal systems based on the common law paradigm. however, the brazilian legal system is based on the civil law paradigm, which implies some key differences in the structure of the documents within a legal case. therefore, it is worth studying rhetorical role identification within the brazilian legal system. a few previous works [7,8,9] have tackled rhetorical role identification in portuguese. however, as far as we know, there is no rri work for legal documents in this language. recently, some papers have proposed resources and systems to tackle nlp problems involving portuguese legal documents, mainly for document classification [1] and named entity recognition [13, 24]. as far as we know, our work is the first one on rhetorical role identification for portuguese legal documents, and also the first one to perform this task on petitions regardless of language. moreover, we propose some novel rhetorical roles that are specific for petitions. 3 corpus and rhetorical roles we created a corpus of petitions based on the court of the brazilian state of mato grosso do sul (tjms) which receives hundreds of thousands lawsuits every year. during 2017, for example, more than 340 thousand cases have been filed in tjms among different types such as civil, criminal, fiscal, and others. we initially considered civil cases that were judged between 2014 and 2018 and are not under judicial secrecy. we then took the petitions of such cases which are usually written by a lawyer that represents the complainant. the system used to file cases in tjms accepts a petition only as a pdf file. however, there is no requirement as how this document is created. it can even be produced by scanning a paper document. therefore, such documents present great variation regarding encoding, content structure, background watermark, headings, footers, among others. since these aspects may generate a lot of noise in the content extracted from the petitions, we selected 70 petitions and, after extracting its textual content by means of pdfboxfootnote 2, a manual inspection was performed in order to remove noisy content. we split each petition into sentences and tokens by means of the nltk library [3]. we obtained a total of 10,784 sentences comprising 249,105 tokens (20,914 unique tokens). the average length of a petition is 154 sentences, and the average length of a sentence is 23.1 tokens. in fig. 1, to provide a better understanding of this corpus, we present the histogram of sentence length. fig. 1. frequency of sentences per length full size image for all documents, personal information were manually identified and anonymized by means of random procedures. more specifically, names of people, institutions, streets and cities were replaced by random names taken from a database containing two thousand names. personal identification numbers and telephone numbers were replaced by random numbers. in order to define the rhetorical roles used to label sentences, we took some inspiration from the brazilian civil procedure (código de processo civil) [4]. more specifically, we considered articles 319, 320 and 321 from this law, which state the minimum requirements regarding the content of a petition within brazilian legal system. in addition to these legal requirements, we considered two possible applications for this corpus: (i) support to staff of a judge’s office (including the judge) when analysing a filed petition; and (ii) petition summarization to support the generation of headnotes that need to be included in the judge’s pronouncement within the legal case. in the following, we list the eight proposed rhetorical roles for sentences within a petition. the terms between parenthesis correspond to portuguese translations of the rhetorical roles. for each rhetorical role, we also present some exemplary sentences. 1. identification (identificação das partes): identification of the parties (complainant, appellant, judge, among others) involved in the case. usually such sentences occur at the beginning of the document. ademir cantareira e paula, brasileiro, solteiro, auxiliar de produção, portador do cpf n\(^{\underline{\mathrm{o}}}\). 179.939.452 04 e do rg n\(^{\underline{\mathrm{o}}}\). 103673421 ssp/ms, residente e domiciliado sito á rua do carapicuibano, n\(^{\underline{\mathrm{o}}}\). 2. facts (fatos): factual account that gave rise to the lawsuit. usually these sentences occur right after the identification, but they can be presented interspersed with the arguments. no dia 28/01/2013 o funcionário da requerida, sr. jorge, realizou serviços de substituição de válvulas no apartamento do síndico do condomínio, sr. ailton cesar, que aproveitando o ensejo contratou verbalmente o profissional para verificar os hidrômetros do prédio e, se necessário, consertar ou substituir dois registros de água do condomínio. 3. arguments (argumentos): pronouncement to persuade the judge to adopt a certain line of thought. these are the sentences that present the greatest variation in linguistic styles. it is common to interconnect these sentences with facts and legal basis. o presente pleito tem como objetivo, entre tantos outros, ressarcir o condomínio autor pelos danos morais, coibindo a atitude abusiva da requerida, em razão do constrangimento sofrido pelo síndico, sr. leandro musk, o qual se sente lesado porque ficou extremamente constrangido ao ter que pagar e explicar perante os condôminos do prédio um gasto exorbitante de r$ 1.660,00 para trocar apenas dois castelos de registros. 4. legal basis (fundamentação legal): description of legal doctrine that gives the ground for the decision making process. it usually includes citations to laws and texts by jurisprudential authors. this is usually interspersed with the arguments and precedents, but we decided to create this specific label since its nature is more objective than arguments. cc. art. 186. aquele que, por ação ou omissão voluntária, negligência ou imprudência, violar direito e causar dano a outrem, ainda que exclusivamente moral, comete ato ilícito. 5. precedents (jurisprudência): this part is based on citations to headnotes of similar or relevant previous cases. some sentences within this part are in capital letters since this is the typical format of headnotes in brazilian legal system. it also occurs interspersed with the arguments. (stj resp 927727 / mg, recurso especial, 2007/0038830-3, dês. rel. ministro josé delgado, primeira turma, 06/05/2008). 6. requests (pedidos): this corresponds to the main requests by the complainants about which the judge must pronounce. it is usually the last by one section of the petition. ante o exposto com supedâneo nos motivos de fato e nos fundamentos do direito pátrio, requer a vossa excelência se digne de. a) citar a ré, na pessoa de seu representante legal, para que, assim desejando, compareça á audiência designada e apresente defesa, sob pena de revelia e confissão (art. 7. remedy (valor da causa): normally contained in one or two sentences in the end of the petition, this part provides a monetary estimate to remedy the complainant’s claims. this is usually the last section of the petition. dá-se á presente o valor de r$ 4.450,00 (quatro mil quatrocentos e cinquenta reais), para fins de alçada e fiscais. 8. others (outros): other textual elements like section and page headings. dos requerimentos. do direito. in table 1, we present the number of sentences and the average number of tokens per sentence for each rhetorical role in our corpus. in the first column of this table, we provide a label for each rhetorical role to ease further references. table 1. basic statistics per rhetorical rolefull size table by means of the doccano tool [14], an expert labeled each sentence of a petition with one of the eight rhetorical roles aforementioned. the sequence of sentences in a petition was preserved in the corpus, so that future approaches can leverage this structure. 4 text representation and machine learning approaches rhetorical role identification is usually modeled as a sentence classification task. following this modeling, there are two broad types of methods in literature: the ones that treat each sentence independently, and the ones that treat a document as a sequence of sentences leveraging the inter-dependencies among sentences. although the later type considers a relevant piece of information that is ignored by the former, in this work we focus only on the former type of methods. our motivation for this decision is that our main goal here is to investigate the feasibility of the proposed rri task. and we indeed experimentally show that, even ignoring the sequential aspect of the given sentences, it is possible to achieve a highly effective rhetorical role classifier (f-score over 80%) for sentences within petitions from a brazilian court. besides that, we also make progress on assessing different text representations and machine learning algorithms on this task. 4.1 baseline methods we considered some standard text classification approaches as baseline methods. more specifically, we combined naive bayes (nb) and support vector machine (svm) models with two standard text representation techniques, namely bag of words (bow) and term frequency-inverse document frequency (tfidf). bow and tfidf are among the most used text representation techniques. although deep learning representations have dominated the latest advances in text processing tasks, bow and tfidf can still be competitive, mainly in text classification problems. thus, these classic representations represent strong baselines in order to evaluate the incremental contribution of more complex methods, like the ones based on deep learning. nb is a probabilistic ml model specially appropriate for text classification tasks. while linear svm is an effective model to tackle problems with sparse input data like texts. both models offer competitive performance in many text classification problems. 4.2 deep learning methods in the last decade, deep learning models have represented the most prominent approaches for nlp problems. and, following the proposal of the seminal transformer architecture [27], models based on attention mechanisms [6, 16, 17] became the standard approach for text tasks. bert [6] is probably the most popular model among such approaches. for instance, among the more than 10 thousand nlp models available in the popular hugging face libraryfootnote 3, approximately 25% are based on bert or some of its variants. among those bert-based models, there are 18 models were trained with portuguese data. in this work, we assessed two bert-based models for rhetorical role identification by training and evaluating them on our proposed corpus. both models are based on the bertimbau base model [23], a bert language model pre-trained on portuguese texts which is available in the hugging face libraryfootnote 4. in fig. 2, we depict these two models. both of them employ a bert language model (lm) to compute a sentence representation (black part of the figure). this part, which is the same for both models, is represented in black at the bottom of the figure. before applying a bert lm, the input sentence is first tokenized into n tokens \((t_1, \ldots , t_n)\), then two special tokens [cls] and [sep] are added, respectively, at the beginning and in the end of the tokenized sentence. the resulting sequence of tokens ([cls], \(t_1, \ldots , t_n\), [sep]) is then fed in the language model, which outputs a 768-dimensional representation for each input token: \((e_\mathrm {[cls]}, e_{t_1}, \ldots , e_{t_n}, e_\mathrm {[sep]})\). when dealing with sentence classification tasks, like is the case for rhetorical role identification, one needs to compute a fixed-length representation for the variable-length input sentence. in this work, we use a common sentence representation approach which consists of simply taking \(e_\mathrm {[cls]}\), thus ignoring the representations of all remaining tokens. since bert is composed of a sequence of self-attention layers, the output \(e_\mathrm {[cls]}\) is directly connected to all tokens of the sentence. therefore, this representation is robust for sentence classification tasks, regardless of which part of the input sentence is more important for the task at hand. fig. 2. two bert-based models for sentence classification. black: bert language model computes a sentence representation \(e_\mathrm {[cls]}\). this part is identical in both models, except that, for the model on the left, the bert lm is fine tuned during supervised training. on the other hand, for the model on the right, the bert lm is kept fixed and only the remaining layers are updated during supervised training. blue: the bert-based sentence representation \(e_\mathrm {[cls]}\) is fed to a single linear layer followed by a softmax function that outputs a probability distribution over the eight labels. this model is trained in an end-to-end fashion, i.e., the bert model is fine-tuned together with the linear layer. red: the sentence representation is fed to a mlp comprising two 512-neuron layers followed by a softmax function which outputs the label distribution. when training this model, the bert lm is kept fixed and only the mlp parameters are updated. (color figure online) full size image in the upper part of fig. 2, we present the remaining layers of the two sentence classification models used in our work. in blue (upper-left of the figure), we represent the model that takes the bert-based sentence representation \(e_\mathrm {[cls]}\), passes it through a linear layer and then applies a softmax activation function which outputs a probability distribution over the eight rhetorical roles proposed in our work. this corresponds to the original bert architecture for text classification problems. this model is trained in an end-to-end fashion, i.e., during supervised training the parameters of both the output linear layer and the bert language model are jointly learned. this approach performs the so called lm fine tuning because the bert lm parameters are updated using labeled data for the final task. the red part of fig. 2 represents a model that feeds the bert-based sentence representation \(e_\mathrm {[cls]}\) to a two-layer multi-layer perceptron (mlp). the output of this mlp is then passed to a softmax function that provides the output label distribution. when training this model, we keep fixed the parameters of the bert language model. in that way, we can evaluate the benefit of fine tuning the bert lm in the rhetorical role identification task. 5 experimental evaluation in this section, we describe the experimental evaluation of the developed ml models on the proposed rri corpus. we model rri as a sentence classification problem. in order to perform model selection and still report meaningful performance estimates, we randomly split the proposed corpus into three splits: train containing 6, 901 sentences, validation containing 1, 726 sentences, and test containing 2, 157 sentences. we use naive bayes and svm models, combined with bag of words and tfidf representations, to define meaningful baselines for this task. in these approaches, each sentence was preprocessed with nltkfootnote 5, the natural language toolkit, to tokenize and remove stopwords.we represent input sentences using bow and tfidf representations by means of the scikit-learn libraryfootnote 6. then, we train and evaluate nb and svm models using both input representations, again making use of scikit-learn. the performances of these four baseline methods on our test set are reported in table 2. we can observe that the best performing baseline method is svm with tfidf representation which obtains a f-score of 60.66. but this method is only slightly better than nb with the same representation. table 2. test performances for different models and input representations. the column labeled f corresponds to micro-averaged f-score over the eight classes, and the remaining columns correspond to per-class f-score.full size table we also evaluated the four nb/svm-based models when the input comprised all words (including stopwords), but the achieved results were substantially worse (f-score values were 5% lower on average). thus, we do not report these performances. in order to train and evaluate the two bert-based methods, we did not employ any preprocessing steps but the corresponding tokenization. by means of the hugging face (hf) library, we trained and evaluated the end-to-end-trained bert-based model. for the mlp-based classifier, we again employed the hf library to extract sentence representations, but then we used the keras library to train the mlp model, since the bert lm is kept fixed. in table 2, we refer to this fixed bert lm as bert\(^f\). the mlp network comprises two hidden layers with 512 neurons each. this network was trained by the nadam optimizer for 500 epochs using a batch size of 32 sentences and the categorical cross-entropy loss. the original (end-to-end) model was trained by the default hf adamw optimizer for 10 epochs using a batch size of 4 sentences. from the results in table 2, we can observe that fine-tuning the bert lm brings a 5 point increase in f-score. the best performing method achieved an f-score of 80.50, which is almost 20 points higher than the performance achieved by the best baseline method. in table 2, we also report per-class f-scores. untypically, the class with best results is remedy, which is the least frequent class. although there are only 71 sentences labeled as remedy in the whole corpus (including the three splits), such sentences have a very specific vocabulary (“dá-se o valor da causa ...”, for instance). this characteristic obviously facilitates the identification of these sentences. besides that, we also observe that some frequent classes – namely facts (2), arguments (3), and legal basis (4) – present the worst performances. in order to investigate this fact, we present the confusion matrix for our best model predictions in fig. 3. first, we can see that there are 73 sentences labeled as facts (fac) but classified by the model as arguments (arg). and other 73 sentences labeled as arguments (arg) but classified as facts (fac). we also see that there are 181 sentences labeled as arguments (arg), legal basis (lb) or precedents (pre) that are misclassified among these same three classes. as mentioned in sect. 3, sentences from these four roles (fac, arg, lb and pre) are frequently interspersed among them. thus, these interspersed sentences are probably more difficult to classify. fig. 3. confusion matrix full size image 6 conclusions and future work in this paper, we present a new corpus comprising 70 petitions (petição inicial) from civil lawsuits filed in tjms. based on the brazilian civil procedure, we proposed eight categories representing the rhetorical roles of text sentences within the petitions. we manually labeled the sentences of the proposed corpus using these eight categories. since tjms does not require any specific structure for the digital files of petitions, we had to deal with several issues to extract the textual content from these files. as far we know, this is the first work to consider rhetorical role identification of petitions in portuguese language. the proposed corpus is availablefootnote 7 and we hope that it can foster research on this important subject. we also analysed the performance of classic and deep learning machine learn methods on the proposed rhetorical role identification corpus. the best performing method was an end-to-end-trained bert-based sentence classifier. this model achieved 80.50 of f-score on our test set. as future work, we intend to pre-train a bert language model on judicial documents. the models evaluated in this work classify sentences independently, i.e. disregarding the sequence of sentences within a petition. thus, we also intend to improve the obtained classification performance by employing deep learning models that consider the sequential structure of sentences. nevertheless, the obtained results are promising. therefore, we are working on the development of a new corpus for summarization of petitions, which will leverage the rhetorical role labels in the proposed corpus. as we have stated, the contributions of this paper can open new research possibilities for judicial nlp. additionally, our work can lead to solutions with potential to relieve the current bottleneck in brazilian judges’ offices. notes 1.leifederal11.419de19/12/2006. 2.https://pdfbox.apache.org/. 3.https://huggingface.co/models. 4.https://huggingface.co/neuralmind/bert-base-portuguese-cased. 5.https://www.nltk.org/. 6.https://scikit-learn.org/. 7.https://bit.ly/rhetoricalrole. references de araujo, p.h.l., de campos, t.e., braz, f.a., da silva, n.c.: victor: a dataset for brazilian legal documents classification. in: proceedings of the 12th language resources and evaluation conference, marseille, france, pp. 1449–1458. european language resources association (may 2020). https://www.aclweb.org/anthology/2020.lrec-1.181 bhattacharya, p., paul, s., ghosh, k., ghosh, s., wyner, a.: identification of rhetorical roles of sentences in indian legal judgments. corr abs/1911.05405 (2019). http://arxiv.org/abs/1911.05405 bird, s., loper, e., klein, e.: natural language processing with python. o’reilly media inc. (2009) google scholar  brasil: lei n. 13.105 de 16 de março de 2015 (código de processo civil) google scholar  contractor, d., guo, y., korhonen, a.: using argumentative zones for extractive summarization of scientific articles. in: proceedings of coling 2012, mumbai, india, pp. 663–678. the coling 2012 organizing committee (december 2012). https://www.aclweb.org/anthology/c12-1041 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), minneapolis, minnesota, pp. 4171–4186. association for computational linguistics (june 2019). https://doi.org/10.18653/v1/n19-1423. https://www.aclweb.org/anthology/n19-1423 feltrim, v.d., aluísio, s.m., nunes, m.g.v.: analysis of the rhetorical structure of computer science abstracts in portuguese. in: corpus linguistics (2003) google scholar  feltrim, v.d., nunes, m.g.v., aluísio, s.m.: um corpus de textos científicos em português para a análise da estrutura esquemática (2001) google scholar  feltrim, v.d., teufel, s., das nunes, m.g.v., aluísio, s.m.: argumentative zoning applied to critiquing novices’ scientific abstracts. in: shanahan, j.g., qu, y., wiebe, j. (eds.) computing attitude and affect in text: theory and applications. the information retrieval series, vol. 20. springer, dordrecht (2006). https://doi.org/10.1007/1-4020-4102-0_18 grover, c., hachey, b., hughson, i.: the holj corpus: supporting summarisation of legal texts. in: proceedings of the 5th international workshop on linguistically interpreted corpora, geneva, switzerland, pp. 47–54. coling, 29 august 2004. https://www.aclweb.org/anthology/w04-1907 hachey, b., grover, c.: a rhetorical status classifier for legal text summarisation. in: text summarization branches out, barcelona, spain, pp. 35–42. association for computational linguistics (july 2004). https://www.aclweb.org/anthology/w04-1007 liu, h.: automatic argumentative-zoning using word2vec. corr abs/1703.10152 (2017). http://arxiv.org/abs/1703.10152 luz de araujo, p.h., de campos, t.e., de oliveira, r.r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: villavicencio, a., moreira, v., abad, a., caseli, h., gamallo, p., ramisch, c., gonçalo oliveira, h., paetzold, g.h. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 313–323. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_32 chapter  google scholar  nakayama, h., kubo, t., kamura, j., taniguchi, y., liang, x.: doccano: text annotation tool for human (2018). software available from https://github.com/doccano/doccano nejadgholi, i., bougueng, r., witherspoon, s.: a semi-supervised training method for semantic search of legal facts in canadian immigration cases. in: wyner, a.z., casini, g. (eds.) the 30th annual conference on legal knowledge and information systems, jurix 2017. frontiers in artificial intelligence and applications, luxembourg, 13–15 december 2017, vol. 302, pp. 125–134. ios press (2017). https://doi.org/10.3233/978-1-61499-838-9-125 radford, a., narasimhan, k., salimans, t., sutskever, i.: improving language understanding by generative pre-training. openai (2018) google scholar  radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i.: language models are unsupervised multitask learners. openai (2019) google scholar  rotta, m.j.r., vieira, p., rover, a.j., sewald, e., jr.: aceleração processual e o processo judicial digital: um estudo comparativo de tempos de tramitação em tribunais de justiça. democracia digital e governo eletrônico 1(8), 125–154 (2013) google scholar  saravanan, m.: ontology-based retrieval and automatic summarization of legal judgments. ph.d. thesis, indian institute of technology madras (2008) google scholar  saravanan, m., ravindran, b.: identification of rhetorical roles for segmentation and summarization of a legal judgment. artif. intel. law 18(1), 45–76 (2010) article  google scholar  saravanan, m., ravindran, b., raman, s.: automatic identification of rhetorical roles using conditional random fields for legal document summarization. in: proceedings of the 3rd international joint conference on natural language processing: volume-i (2008). https://www.aclweb.org/anthology/i08-1063 savelka, j., ashley, k.d.: segmenting u.s. court decisions into functional and issue specific parts. in: palmirani, m. (ed.) the 31st annual conference on legal knowledge and information systems, jurix 2018. frontiers in artificial intelligence and applications, groningen, the netherlands, 12–14 december 2018, vol. 313, pp. 111–120. ios press (2018). https://doi.org/10.3233/978-1-61499-935-5-111 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, 20–23 october (2020, to appear) google scholar  souza, f., nogueira, r.f., de alencar lotufo, r.: portuguese named entity recognition using bert-crf. corr abs/1909.10649 (2019). http://arxiv.org/abs/1909.10649 teufel, s.: argumentative zoning: information extraction from scientific text. ph.d. thesis, university of edinburgh (1999). http://www.cl.cam.ac.uk/users/sht25/az.html teufel, s., moens, m.: sentence extraction and rhetorical classification for flexible abstracts. in: intelligent text summarization, pp. 16–25 (1998) google scholar  vaswani, a., et al.: attention is all you need. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017). https://proceedings.neurips.cc/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-paper.pdf walker, v.r., pillaipakkamnatt, k., davidson, a.m., linares, m., pesce, d.j.: automatic classification of rhetorical roles for sentences: comparing rule-based scripts with machine learning. in: ashley, k.d., et al. (eds.) proceedings of the 3rd workshop on automated semantic analysis of information in legal texts co-located with the 17th international conference on artificial intelligence and law, icail 2019, montreal, qc, canada, 21 june 2019. ceur workshop proceedings, vol. 2385. ceur-ws.org (2019). http://ceur-ws.org/vol-2385/paper1.pdf yamada, h., teufel, s., tokunaga, t.: neural network based rhetorical status classification for japanese judgment documents. in: araszkiewicz, m., rodríguez-doncel, v. (eds.) the 32nd annual conference on legal knowledge and information systems, jurix 2019. frontiers in artificial intelligence and applications, madrid, spain, 11–13 december 2019, vol. 322, pp. 133–142. ios press (2019). https://doi.org/10.3233/faia190314 download references author information authors and affiliations universidade federal de mato grosso do sul, campo grande, brazil roberto aragy, eraldo rezende fernandes & edson norberto caceres tribunal de justiça de mato grosso do sul, campo grande, brazil roberto aragy authors roberto aragyview author publications search author on:pubmed google scholar eraldo rezende fernandesview author publications search author on:pubmed google scholar edson norberto caceresview author publications search author on:pubmed google scholar corresponding author correspondence to roberto aragy . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper aragy, r., fernandes, e.r., caceres, e.n. (2021). rhetorical role identification for portuguese legal documents. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_38 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_38 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords rhetorical role identification legal sentence classification natural language processing corpus publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature embracing data irregularities in multivariate time series with recurrent and graph neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper embracing data irregularities in multivariate time series with recurrent and graph neural networks conference paper first online: 12 october 2023 pp 3–17 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) embracing data irregularities in multivariate time series with recurrent and graph neural networks download book pdf download book epub marcel rodrigues de barros9, thiago lizier rissi9, eduardo faria cabrera9, eduardo aoun tannuri9, edson satoshi gomi9, rodrigo augusto barreira10 & … anna helena reali costa9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 633 accesses 1 citation abstract data collection in many engineering fields involves multivariate time series gathered from a sensor network. these sensors often display differing sampling rates, missing data, and various irregularities. to manage these issues, complex preprocessing mechanisms are required, which become coupled with any statistical model trained with the transformed data. modeling the motion of seabed-anchored floating platforms from measurements is a typical example for that. we propose and analyze a model that uses both recurrent and graph neural networks to handle irregularly sampled multivariate time series, while maintaining low computational cost. in this model, each time series is represented as a node in a heterogeneous graph, where edges depict the relationships between each measured variable. the time series are encoded using independent recurrent neural networks. a graph neural network then propagates information across the time series using attention layers. the outcome is a set of updated hidden representations used by the recurrent neural networks to create forecasts in an autoregressive manner. this model can generate forecasts for all input time series simultaneously while remaining lightweight. we argue that this architecture opens up new possibilities as the model can be integrated into low-capacity systems without needing expensive gpu clusters for inference. we gratefully acknowledge the support from anp/petrobras, brazil (project n. 21721-6), cnpq (grants 310085/2020-9 and 310127/2020-3), capes (finance code 001) and c4ai-usp (fapesp grant 2019/07665-4 and ibm corporation). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others recurrent neural networks integrate multiple graph operators for spatial time series prediction article 18 august 2023 time series forecasting model for non-stationary series pattern extraction using deep learning and garch modeling article open access 02 january 2024 adaptive graph recurrent network for multivariate time series imputation chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. data science dynamic networks network models probabilistic data networks stochastic networks time series analysis 1 introduction although the study of multivariate time series (mts) spans over 40 years, this area remains relatively unexplored due, in part, to its distinct nature compared to conventional statistical theory. classical methods primarily rely on random samples and independence assumptions, whereas mts are characteristically highly correlated [19]. despite this, the field has recently garnered attention from machine learning (ml) researchers, motivated by the recent achievements in token sequence modeling, particularly in tasks related to natural language processing (nlp). models like the transformer [17] have generated groundbreaking results in tasks such as question answering and text classification, thereby prompting the question: can such successes be replicated in the context of time series? for the majority of applications, ml algorithms currently underperform when compared to traditional statistical methods for univariate time series [9]. one of the main hurdles is that deep learning models, largely responsible for the ml revolution in nlp, are particularly susceptible to overfitting in univariate scenarios. however, multivariate problems exhibiting complex interactions are not easily modeled by classical approaches. this makes it an appealing area for ml research exploration. this work introduces a cost-effective encoder-decoder architecture founded on recurrent and graph neural networks (rnns and gnns), coupled with vector time representation capable of forecasting mts. moreover, this proposed model can process data from sensors with differing sampling rates and missing data profiles without resorting to data imputation techniques. our dataset consists of motion data from floating production, storage, and offloading units (fpsos)—floating sea platforms used for oil extraction. moored to the seafloor, fpsos display intricate oscillatory behavior influenced by environmental factors such as wind, currents, and waves. the key contributions of this work are fourfold: we demonstrate that time encoding enhances forecasting results for complex oscillatory dynamics, especially when large missing data windows are present. we provide evidence that heterogeneous graph neural networks can effectively propagate information between abstract representations of sensor data. we define a comprehensive end-to-end system capable of modeling fpso hydrodynamic behavior at low computational costs and without the need for elaborate data preprocessing routines. additionally, we argue that the proposed architecture can be successful in a wide range of applications. in light of the above, this article is organized as follows: section 2 presents the foundational concepts underpinning this work and provides a synopsis of how data irregularities can affect common ml architectures. section 3 outlines the proposed architecture, while sect. 4 offers a more detailed description of the dataset and experimental setups. sections 5 and 6, respectively, present the experimental results and offer some conclusions, as well as directions for future work. 2 background this section introduces grounding concepts for this work, namely: mts definitions and terminology, rnns and gnns main characteristics as means to restrict the search space for the parameter optimization process, and how fpso motion forecast context is centered in dealing with data irregularities. 2.1 multivariate time series formally, we can define a multivariate time series context as a (finite) set of time series \({\mathcal {s} = \{\mathbf {z^i_t}\}^s_{i=1}}\). each element of this set is a sequence of events of length \(t_z\), \({\mathbf {z_t} = [z_1, z_2, \cdots , z_{t_z}]}\), such that \(z_t \in \mathbb {r}^{k_z}\). note that we choose to represent a time series as a boldface matrix \(\mathbf {z_t}\) with \(t_z\) observations of \(k_z\) features. this work treats each feature vector \(z_t\) as an event associated with a scalar timestamp representing t. the set \(\mathbf {\mathcal {s}}\) is defined over \(\mathbf {\chi }\) which is the input space of the model. let’s suppose that, in the simplest case, one wishes to forecast the features associated with a single future event \(z_{t_z+d}\) from a time series \(\mathbf {z^i_t}\), but making use of all information contained in \(\mathbf {\mathcal {s}}\). in other words, in this scenario we wish to approximate an unknown function \(f : \chi \rightarrow \mathbb {r}^{k_z}\) such that \(f(\mathbf {\mathcal {s}}) = z_{t_z+d}\) for all \(\mathbf {\mathcal {s}}\). we call the approximated function \(\tilde{f} \in \mathcal {f}\). an important concept in this work is the choice of function class \(\mathcal {f}\) that contains the function \(\tilde{f}\). the typical deep learning approach is to use a parametrized function class \(\mathcal {f} = \{f_\theta \in \vartheta \}\). an adequate parametrization guarantees the existence of a function \(\tilde{f}\) that is arbitrarily close to the true function f [6]. this is called the interpolating regime i.e. \(\tilde{f}(\mathbf {\mathcal {s}}) = f(\mathbf {\mathcal {s}})\) for all \(\mathbf {\mathcal {s}}\). usually, the stochastic gradient descent algorithm is used to optimize the parameters by minimizing a loss function \(l(\tilde{f}(\mathbf {\mathcal {s}}), f(\mathbf {\mathcal {s}}))\). however, each data point in a time series context typically has little to no information since the process is mainly driven by strong correlations and white noise processes. in this scenario, simple deep neural networks such as multilayer perceptrons (mlps) rapidly pick-up spurious patterns and are unable to generalize to unseen data. thus, to learn these complex interactions from the data available it is necessary to reduce the size of \(\mathcal {f}\) by restricting it. 2.2 rnns and gnns as means to restrict \(\mathcal {f}\) many successful deep learning models can be described as restricted optimization problems in function classes that represent some kind of symmetry. for example, convolutional neural networks, which is the main architecture for image processing tasks, implement translational symmetry through the convolution operation on grids. other analogous interpretations can be derived for gated rnns, transformers, and others [3]. in this work, we apply gated rnns, which exhibit time warping symmetry invariance [16]. this symmetry is defined as any monotonically increasing differentiable mapping \(\tau : \mathbb {r}^+ \rightarrow \mathbb {r}^+\) between times. being invariant to this symmetry means that gated rnns can associate patterns in time series data even if they are distorted in time as long as order is preserved. we also apply gnns to propagate information between different \(\mathbf {z^i_t}\). the gnn architecture [14] can be described by the message-passing mechanism: $$\begin{aligned} \textbf{x}_i^{(k)} = \gamma ^{(k)} \left( \textbf{x}_i^{(k-1)}, \square _{j \in \mathcal {n}(i)} \, \phi ^{(k)}\left( \textbf{x}_i^{(k-1)}, \textbf{x}_j^{(k-1)}\right) \right) , \end{aligned}$$ (1) where \(\textbf{x}_i^{(k)} \in \mathbb {r}^{d}\) is the new representation of the features of node i at layer k, \(\phi ^{(k)}\) is a differentiable message function that produces a hidden representation that is sent from node j to node i. \(\square \) denotes a differentiable, permutation invariant aggregation function that aggregates all messages from neighbour nodes. \(\gamma ^{(k)}\) is a differentiable function that updates the node’s hidden representation. once again the crucial fact is the permutation invariance that restricts the function class \(\mathcal {f}\) with the fact that isomorphic graphs have the same representation. there are several variants of gnns and this work does not intend to detail their differences. a comprehensive survey can be found at [21]. in this work we use the heterogeneous graph attention network architecture. that means we consider the graph that models the mts as being heterogeneous, i.e., as having different types of nodes and edges. this may become clearer in sect. 3 that details how these rnns and a gnn are combined in an end-to-end system. 2.3 fpso dynamics and irregularities in mts while the contributions of this work apply to a general mts context, it was developed to address the challenge of maintaining stable positions of floating platforms in deep water. fpso units have been employed for years to safely extract offshore oil reserves. these floating platforms enable the extraction and storage of crude oil while being held in place using mooring lines anchored to the ocean floor. mooring systems play a critical role in ensuring the positional stability, personnel safety, and smooth functioning of various operations on a platform, such as extraction, production, and oil offloading. the constant exposure of floating structures to offshore environmental conditions like waves, currents, and wind leads to continuous stresses and strains on the platforms. consequently, the structural integrity and mooring lines of these platforms degrade over time. understanding and modeling the complex dynamics of the floating platform allows for real-time detection of unexpected oscillatory patterns, which could assist in assessing the mooring lines’ integrity. in this work, we analyze four variables: utmn and utme, which are north and east coordinates from a gps, heading, which measures the rotation movement around the vertical axis, and roll, which measures the rotation along the longitudinal axis of the platform. these four variables are the most significant to characterize the platform movement. the ever-increasing amount of data generated by sensors can be used to train ml algorithms for this task, but the same environment characteristics that deteriorate the mooring system also affect the sensors which are reflected in the measurements. typical sensor data from fpsos suffers from long-missing data windows, different polling rates between sensors, and other deformities. several data imputation approaches to achieve grid-like data structure have been proposed [5]. a regular grid structure allows for the usage of simple rnns or even mlps [13], but also causes the model behavior to be highly dependent on the data imputation technique used. furthermore, achieving grid-like structure in complex sensor networks becomes rapidly unfeasible due to missing data points and shifted time series as shown in fig. 1. our proposal aims to offer a simple and robust solution that does not rely in any imputation process. fig. 1. white balls denote observations from sensors. \(\mathbf {z^1_t}\) is represented by a single feature and has two missing data points, denoted in dark gray, while \(\mathbf {z^2_t}\) also has one feature, but a different frequency, and is shifted. red balls denote data that would have to be imputed to achieve a grid-like structure. (color figure online) full size image 3 architecture the proposed architecture consists of a set of single-layered rnns, a single layer of a heterogeneous graph attention network [2, 18], a time encoding module, and a linear projection layer. each input time series is processed by a different rnn instance into a fixed-size representation as shown in fig. 2. in this work, we use gated recurrent units (grus) [4] which is a proven gated rnn architecture. this is a design choice that allows both \(t_z\) and \(k_z\) to vary between time series since rnns can process arbitrary sequence lengths and each rnn instance has its width predefined based on the number of features of its input time series. fig. 2. encoding each time series with a different rnn instance provides flexibility to handle irregularities without the need for data imputation. each time series is encoded into a fixed-size representation. full size image a central aspect in this work is the encoding of timestamps. instead of feeding rnns with only feature values, we also encode the time scalar for each observation using the positional encoding mechanism from [17]. the process can be visualized in fig. 3. fig. 3. detailed view of encoding of a single time series. an rnn can ingest a sequence of arbitrary length \(t_z\). instead of just feeding it with \(k_z\) features (1 in this example), the model adds the encoded timestamps as additional information. rnns generate an output for each element in the input sequence, but our architecture only makes use of the last as a fixed-size representation of the whole input sequence. full size image more formally, the time encoder can be defined as a function \(t: \mathbb {r}^{t_z} \rightarrow \mathbb {r}^{t_z \times \mathcal {t}}\) that encodes each scalar \(t-t_0\), where \(t_0\) represents the instant of inference, into a representation of size \(\mathcal {t}\). each rnn encoder can be defined as \(e^i: \mathbb {r}^{t_z \times (k_z + \mathcal {t})} \rightarrow \mathbb {r}^{\mathcal {h}}\) that encodes a sequence of \(t_z\) vectors with \(k_z\) features enriched by \(\mathcal {t}\) time features into a single hidden representation of size \(\mathcal {h}\). once each input time series has now a fixed-size representation, the information is shared between nodes using the message-passing mechanism defined in eq. 1. note that this expression denotes a homogeneous message passing. in a pure heterogeneous case, a different pair of functions \(\phi \) and \(\gamma \) would be trained for each edge type. that means for a graph with n nodes there would be \(n^2\) pairs of learnable parametrized message-passing functions. to prevent overfitting due to this quadratic increase of parametrized functions the regularization technique proposed in [15] is applied. in it instead of training a new pair of functions for each new edge type a fixed set of function pairs is trained and each edge type applies a different learnable linear combination of this set of bases. the structures of \(\phi \) and \(\gamma \) are determined by the choice of gnn architecture, this work applies the graph attention network v2 [2] that calculates attention masks for the incoming messages of each node. this allows nodes to incorporate the most relevant neighborhood information dynamically. the gnn part of the encoder can be defined as \(\mathcal {g}: (\mathbb {r}^{n \times \mathcal {h}}, \mathcal {e}) \rightarrow \mathbb {r}^{n \times \mathcal {h}}\) that updates the hidden representation of each node \(i \in n\) based on the set of edges \(\mathcal {e}\). a fully-connected graph representation for our problem can be visualized in fig. 4. fig. 4. heterogeneous gnn takes as input the fixed-size representation of each time series and propagates the information through the graph using the message-passing mechanism. outputs are the updated vector representations for each node. full size image finally, the decoding process uses the same rnns used for encoding, but now in an autoregressive configuration coupled with a linear layer. the initial hidden state \(h^i_{t_0}\) is given by the updated hidden representation obtained from \(\mathcal {g}\). each forecasted data point \(y^i_{t+1}\) is obtained by projecting the hidden representation that is iteratively updated. during this process, we also enrich the input features for the rnn with the encoded representations of the distance to the target timestamp \((t+1) t_0\). the whole decoding process can be visualized in fig. 5. formally, each decoding step \(\mathcal {d}^i: (\mathbb {r}^{\mathcal {h}},\mathbb {r}^{k_z},\mathbb {r}) \rightarrow (\mathbb {r}^{\mathcal {h}},\mathbb {r}^{k_z})\) receives a hidden state \(h^i_{t}\), the last forecasted label \(y^i_t\) and a target time \(t+1\). it outputs an updated state \(h^i_{t+1}\) and its related output forecast \(y^i_{t+1}\). it is important to note that each module-specific implementation could be replaced. adopting alternatives for time encoding such as time2vec [8], different rnn architectures, such as lstms [7], or different gnn architectures would not change the grounding principles behind this work but could bring further performance gains. fig. 5. the decoder uses the same rnn instances from the encoder. it takes the updated representations as its initial state and uses a linear projection to produce \(y^i_{t+1}\) in an autoregressive manner. full size image 4 experimental setup three variants of the model were trained: a complete version including the timestamp encoding method and a fully connected graph (named full model); a second version without the timestamp encoding strategy (named no time enconding model); and a third version with a totally disconnected graph (named disconnected graph model). with these comparisons, we intend to evaluate the effectiveness of gnn in propagating information between abstract representations of sensor data and the improvement in regression performance provided by timestamp encoding. for each model, 3 sets of tests were evaluated, varying the proportion of missing data points: 40%, 20% and 0%, in order to verify the model’s robustness in addressing the irregularities that usually occur in real sensor data. long windows of 5% missing data were randomly inserted into the test dataset before inference. often, missing data in sensor networks appear in the form of long missing data windows instead of isolated missing data points [1]. the dataset is obtained from numerical simulations that applies partial differential equations to model environmental conditions acting onto a 3d model of an existing fpso. the simulator generates three-hour mts with 1 s frequency for 7003 different environmental conditions, of which we separated 50% for training, 20% for validation and 30% to test. this represents 14,006 h of simulated data. in this work validation data is only used for training convergence plots, i.e., no hyperparameter optimization was performed. all experiments used the following settings: \(\mathcal {t}=50, \mathcal {h}=200\). for the heterogeneous gnn, the number of bases for regularization is 6 and the number of attention heads is 16. a single 10% dropout layer is applied to gnn outputs during training. optimizing these hyperparameters may be beneficial, but it is not in the scope of this work. to prepare the data, we applied a differentiation in each time series such that \(ds_{t} = s_{t} s_{t-1}\). this is a common technique in autoregressive models and is especially relevant for gps data, since fpso initial offset can vary widely with environmental conditions. in addition, we applied a z-score normalization to the differential time series. the instant of inference is denoted \(t_0\). the model builds the input mts \(\mathcal {s}\) by filtering measurements within a certain context window c in seconds so that \(t_0 c \le t < t_0\). this architecture allows for each time series to adopt a different context window length c based on its characteristics. longer context windows of 1800 s are chosen for utmn and utme since their signals present lower frequencies, while shorter 600 s windows are enough to characterize the spectrum of heading and roll motions, which mostly consist of high frequencies. such as the input, the model builds the target mts \(\mathbf {\mathcal {s}}_{o}\) by filtering the measurements within a certain forecast window d in seconds so that \(t_0 \le t < t_0 + d \). for all the four variables we adopted forecast windows of \(d=\) 100 s. the loss function \(l(\mathbf {\mathcal {s}_o}, \hat{\mathbf {\mathcal {s}_o}})\) used for training was based on the index of agreement [20] that provides a fit measurement between two time series that falls within the unit interval. this metric had been used successfully in the ml for mts context by [10] since it provides a common range for all time series losses. for a mts context it can be precisely defined as: $$\begin{aligned} l(\mathbf {\mathcal {s}_o}, \hat{\mathbf {\mathcal {s}_o}}) = 1 {ioa} = 1 \frac{1}{s}\sum _{i=1}^{s}(\frac{\sum _{t}(\mathbf {z^i}-\mathbf {\hat{z^i}})^2}{\sum _{t}(|\mathbf {\hat{z^i}}-\bar{\mathbf {z^i}} |+|\mathbf {z^i}-\bar{\mathbf {z^i}} |)^2}) \end{aligned}$$ (2) in each training epoch, the model chooses randomly a \(t_0\) from the three-hour mts for doing the forecasting. with this, the model avoids becoming specialized in forecasting at some specifics inference times. for testing, we determined a set of inference times beforehand so each model can be consistently compared. all models were trained for 2000 epochs. for the full model that corresponds to 22 h of training in a single nvidia rtx 3060ti gpu. this inexpensive training is possible since the model is only four layers deep and contains 8.35m parameters taking up only 33.4 mb of disk space. the inference can be run in a consumer-level cpu in around 0.2 s while consuming 100 mb of memory. 5 results and analysis figure 6 displays each model’s results for the time series in a randomly selected environmental condition and \(t_0\), for different levels of missing data. the full model proved to be resistant to the increase of missing data points, and managed to maintain a consistent performance as the prediction horizon increased. fig. 6. illustration of the three models predictions (full model, no time enconding model, and disconnected model) with distinct missing data proportions for the four time series (utmn, utme, roll, and heading) in the same random environmental condition, considering a forecast window of 100 s. full size image additionally, the model using disconnected graphs performed worst than both the full model and no-time encoding model, since in several cases the model was not able to correctly learn the movement tendency, and produced forecasts with significant offsets. in other cases, it failed to represent high frequency components of the mts. from this qualitative view it is noticeable that the full model also outperforms the no-time encoding model, especially as missing data windows increase. this is the hypothesized behavior since the time encoding provides a way for the rnn to recognize a time gap and adjust for it. thus, the model was able to better extrapolate temporal patterns of motion. box plots in fig. 7 present the evolution of the regression’s mean absolute error in the forecast window using the full model. for better visualization we omit the outliers, limiting the whiskers to comprehend the range of 5% to 95% percentiles. it is observed that as the temporal distance increases the error increases, as expected. fig. 7. boxplots of the absolute error versus the time distance to target in the forecast window for all the test environmental conditions. the whiskers are limited to comprehend the range of 5% to 95% percentiles. full size image numerical results are presented in table 1, displaying absolute errors’ statistics calculated upon the last predicted point in the inference horizon of 100 s. the results confirm the analysis for fig. 6, showing that the full model outperformed the other two models in the vast majority of the cases, except for the 25% and 50% percentiles for the roll series without missing data, that were better predicted by the no-time encoding model. table 1. table containing all numeric results calculated for the last predicted point in the inference horizon for all different forecast windows and environments conditions. the 25, 50 and 75 columns refer to the absolute error that represents the respective percentiles, while the max column refers to the maximum error obtained in the forecast of any environmental condition and \(t_0\).full size table here it is emphasized the resilience of the full model to the increase in missing data. this occurs because missing windows are recognized by the time encoding mechanism and missing information can be filled in by neighbouring nodes in a latent space, allowing for an “intrinsic data imputation” to occur. this effect of feature completion in gnns was previously described by [12]. all three models displayed forecast outliers where errors were several times larger than the 95% percentile. preventing or recognizing these outliers is an important task to enable these type of models for operating in mission critical tasks. nevertheless, the full model also exhibited fewer and less extreme outliers. to conclude, the evolution of the ioa during training calculated over the validation dataset is provided in fig. 8. the results are evidence that both graph connectivity and time encoding act together to improve convergence speed and stability during training. fig. 8. evolution of index of agreement loss during training. this plot denotes ioa loss running mean for all time series for the last 30 epochs calculated over the validation dataset. full size image 6 conclusion results provide evidence that heterogeneous gnns can effectively propagate information between abstract representations of sensor data, exploiting the relationship between them. this work also demonstrates that gnns can be associated with rnns and time encoders to tackle missing data windows in mts. this general architecture is a simple, cost-effective, yet robust solution for restricting the function class \(\mathcal {f}\) and allowing the stochastic gradient optimization to converge faster and more accurately in complex multivariate time series contexts. we argue, based on the results and conveniences of this model, that this architecture could be successful in a multitude of applications making it a strong one-fits-all model in the mts context. as future research directions, we note that other works have been successful on modeling mts as abstract networks. we believe that approaches treat each single measurement as a different node such as [22] can be combined with latent graph inference techniques, allowing for dynamic inference of an underlying optimized graph topology. creating specialized functions that are able to store relevant information from hidden states into persistent vector representations for nodes between different inference times such as proposed by [11] is also an interesting direction for long-term predictions. finally, we believe a more comprehensive study on the choice of loss function and how known mts losses affect this architecture could bring other insights on its capabilities and weaknesses. this work was financed in part by anp/petrobras, brazil (project n. 21721-6). we also gratefully acknowledge partial support from cnpq (grants 310085/2020-9 and 310127/2020-3). references adhikari, d., jiang, w., zhan, j.: imputation using information fusion technique for sensor generated incomplete data with high missing gap. microprocess. microsyst., 103636 (2021). https://doi.org/10.1016/j.micpro.2020.103636 brody, s., alon, u., yahav, e.: how attentive are graph attention networks? corr abs/2105.14491 (2021). https://arxiv.org/abs/2105.14491 bronstein, m.m., bruna, j., cohen, t., velickovic, p.: geometric deep learning grids, groups, graphs, geodesics, and gauges. corr abs/2104.13478, 160 (2021) google scholar  cho, k., et al.: learning phrase representations using rnn encoder-decoder for statistical machine translation, september 2014. https://doi.org/10.48550/arxiv.1406.1078. arxiv:1406.1078 [cs, stat] emmanuel, t., maupong, t., mpoeleng, d., semong, t., mphago, b., tabona, o.: a survey on missing data in machine learning. j. big data 8(1), 140 (2021). https://doi.org/10.1186/s40537-021-00516-9 hartman, e.j., keeler, j.d., kowalski, j.m.: layered neural networks with gaussian hidden units as universal approximations. neural comput. 2(2), 210–215 (1990). https://doi.org/10.1162/neco.1990.2.2.210 hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9, 1735–80 (1997). https://doi.org/10.1162/neco.1997.9.8.1735 kazemi, s.m., et al.: time2vec: learning a vector representation of time (2019). https://doi.org/10.48550/arxiv.1907.05321. arxiv:1907.05321 [cs] makridakis, s., spiliotis, e., assimakopoulos, v.: statistical and machine learning forecasting methods: concerns and ways forward. plos one 13(3), e0194889 (2018). https://doi.org/10.1371/journal.pone.0194889. https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0194889. publisher: public library of science netto, c.f.d., et al.: modeling oceanic variables with dynamic graph neural networks, june 2022. https://doi.org/10.48550/arxiv.2206.12746. arxiv:2206.12746 [cs] rossi, e., chamberlain, b., frasca, f., eynard, d., monti, f., bronstein, m.: temporal graph networks for deep learning on dynamic graphs. arxiv:2006.10637 [cs, stat], october 2020. arxiv: 2006.10637 rossi, e., kenlay, h., gorinova, m.i., chamberlain, b.p., dong, x., bronstein, m.: on the unreasonable effectiveness of feature propagation in learning on graphs with missing node features, may 2022. https://doi.org/10.48550/arxiv.2111.12128. arxiv:2111.12128 [cs] saad, a.m., et al.: fpso mooring line failure detection based on predicted motion. in: proceedings of the asme 2021 40th international conference on ocean, offshore and arctic engineering. american society of mechanical engineers digital collection, october 2021. https://doi.org/10.1115/omae2021-62413 scarselli, f., gori, m., tsoi, a.c., hagenbuchner, m., monfardini, g.: the graph neural network model. ieee trans. neural netw. 20(1), 61–80 (2009). https://doi.org/10.1109/tnn.2008.2005605. http://ieeexplore.ieee.org/document/4700287/ schlichtkrull, m., kipf, t.n., bloem, p., van den berg, r., titov, i., welling, m.: modeling relational data with graph convolutional networks. in: gangemi, a., et al. (eds.) eswc 2018. lncs, vol. 10843, pp. 593–607. springer, cham (2018). https://doi.org/10.1007/978-3-319-93417-4_38 chapter  google scholar  tallec, c., ollivier, y.: can recurrent neural networks warp time? in: international conference on learning representation 2018, february 2018. https://openreview.net/forum?id=sjckhk-ab vaswani, a., et al.: attention is all you need. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017) google scholar  veličković, p., cucurull, g., casanova, a., romero, a., lió, p., bengio, y.: graph attention networks. arxiv:1710.10903 [cs, stat], february 2018. http://arxiv.org/abs/1710.10903 wei, w.w.s.: multivariate time series analysis and applications. wiley, p. 528 (2019) google scholar  willmott, c.j.: on the validation of models. phys. geogr. 2(2), 184–194 (1981). https://doi.org/10.1080/02723646.1981.10642213. publisher: taylor & francis _eprint wu, z., pan, s., chen, f., long, g., zhang, c., yu, p.s.: a comprehensive survey on graph neural networks. ieee trans. neural netw. learn. syst. 32(1), 4–24 (2021). https://doi.org/10.1109/tnnls.2020.2978386. https://ieeexplore.ieee.org/document/9046288/ zhang, x., zeman, m., tsiligkaridis, t., zitnik, m.: graph-guided network for irregularly sampled multivariate time series, march 2022. https://doi.org/10.48550/arxiv.2110.05357. http://arxiv.org/abs/2110.05357 [cs] download references author information authors and affiliations universidade de são paulo, são paulo, sp, brazil marcel rodrigues de barros, thiago lizier rissi, eduardo faria cabrera, eduardo aoun tannuri, edson satoshi gomi & anna helena reali costa petróleo brasileiro s.a., rio de janeiro, brazil rodrigo augusto barreira authors marcel rodrigues de barrosview author publications search author on:pubmed google scholar thiago lizier rissiview author publications search author on:pubmed google scholar eduardo faria cabreraview author publications search author on:pubmed google scholar eduardo aoun tannuriview author publications search author on:pubmed google scholar edson satoshi gomiview author publications search author on:pubmed google scholar rodrigo augusto barreiraview author publications search author on:pubmed google scholar anna helena reali costaview author publications search author on:pubmed google scholar corresponding author correspondence to marcel rodrigues de barros . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de barros, m.r. et al. (2023). embracing data irregularities in multivariate time series with recurrent and graph neural networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_1 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multivariate time series machine learning artificial intelligence graph neural networks recurrent neural networks data imputation publish with us policies and ethics profiles marcel rodrigues de barros view author profile eduardo aoun tannuri view author profile anna helena reali costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature comparing neural network encodings for logic-based explainability | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper comparing neural network encodings for logic-based explainability conference paper first online: 30 january 2025 pp 281–295 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) comparing neural network encodings for logic-based explainability download book pdf download book epub levi cordeiro carvalho9, saulo a. f. oliveira9 & thiago alves rocha9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 476 accesses 9 altmetric abstract providing explanations for the outputs of artificial neural networks (anns) is crucial in many contexts, such as critical systems, data protection laws and handling adversarial examples. logic-based methods can offer explanations with correctness guarantees, but face scalability challenges. due to these issues, it is necessary to compare different encodings of anns into logical constraints, which are used in logic-based explainability. this work compares two encodings of anns: one has been used in the literature to provide explanations, while the other will be adapted for our context of explainability. additionally, the second encoding uses fewer variables and constraints, thus, potentially enhancing efficiency. experiments showed similar running times for computing explanations, but the adapted encoding performed up to 18% better in building logical constraints and up to 16% better in overall time. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ontology-based explanations of neural networks: a user perspective chapter © 2024 correction of neural network solutions by logical analysis methods chapter © 2023 differentiable logics for neural network training and verification chapter © 2022 1 introduction artificial neural networks (anns) are widely applied in tasks like computer vision, speech recognition, and pattern recognition [13]. despite their success, anns are often considered black-box algorithms. such a lack of interpretability poses risks in critical domains such as medical and financial applications, where understanding model decisions is crucial. additionally, the presence of adversarial examples highlights the need for explainability in machine learning algorithms, including neural networks. an adversarial example is an instance misclassified by a machine learning model and also slightly different from another correctly classified instance [6]. in this work, an explanation for a prediction made by an ann is a subset of features and their values that alone suffice for the prediction. if an instance has the features in this subset, the ann makes the same prediction, regardless of the values of other features. for example, given an instance \(\{sneeze=true\), \(weight=70 \ kg\), \(headache=true\), \(age=40 \ years\}\) and its ann output flu, a possible explanation could be \(\{sneeze=true, headache=true\}\). that is, if an instance has the features \(sneeze=true\) and \(headache=true\), the ann prediction is flu, regardless of weight and age values. a minimal explanation avoids redundancy by including only essential information. an explanation is considered minimal when removing any feature results in the loss of assurance that every instance satisfying the explanation maintains the same output. then, a minimal explanation avoids redundancy, providing only essential information. heuristic methods, such as anchor [15] and lime [14], have been used to provide explanations for machine learning models. however, these approaches explore the instance space locally, not resulting in explanations that have minimal sizes and formal guarantees of correctness. correctness guarantees are provided when there are no instances with the values specified in the explanation such that the ann makes a different prediction. moreover, minimal explanations are desired since they do not contain redundancy, making them easier to understand and interpret. some approaches aim to provide explanations for machine learning models with formal guarantees of correctness [1, 4, 7, 8, 16, 18]. ignatiev et al. [8] proposed a logic-based algorithm that gives minimal and correct explanations for anns, utilizing logical constraints originally designed for finding adversarial examples fischetti and jo [5]. these constraints include linear equations, inequalities, and logical implications, solved using a mixed integer linear programming (milp) solver. however, scalability issues arise, particularly with large anns, necessitating further development before deployment in large-scale production environments. this work explores two different encodings to improve the scalability of providing correct minimal explanations for anns, building upon [8]. in addition to the logical constraints of [5], we adopt the encoding proposed by tjeng et al. [17], which uses fewer variables and constraints, and excludes logical implications. by reducing variables and constraints compared to [5], our approach aims to enhance explanation computation performance. to adapt the approach of [17] for explanations, we introduce new constraints to ensure correctness. in line with the encodings proposed by fischetti and jo [5] and tjeng et al. [17], we also compute lower and upper bounds for each neuron. these bounds are found through optimization using a milp solver. moreover, these bounds can aid the solver in computing explanations more rapidly. in this manner, we compare the time required for constructing logical constraints with lower and upper bounds of each neuron, along with the time needed for computing explanations. we conducted experiments to evaluate both encodings. our adaptation of the encoding proposed in [17] exhibits a better running time in building encodings for anns with two layers and tens of neurons, showing an improvement of up to 18%. surprisingly, both methods exhibit similar running times for computing explanations. furthermore, our adaptation outperforms the other encoding in the overall time, encompassing both building logical constraints and computing explanations. in this case, the results indicate an improvement of up to 16%. in summary, our main contributions are described in the following: adaptation of the encoding proposed in [17] to provide explanations for anns; additional constraints were incorporated to address the problem of computing explanations. comparative analysis of the running time for building the logical constraints between the two approaches. additionally, we analyze the time for generating explanations using both encodings. publicly available implementations of both encodings for finding explanations for annsfootnote 1. in the next section, we review some concepts and terminologies about logic, milp and anns. sections 3 and 4 show how to compute explanations with and without implications, respectively. section 4 describes our adaptation of the encoding proposed in [17]. experiments and results are presented in sect. 5. finally, conclusions and future work are described in sect. 6. 2 background in this section, we introduce some initial concepts and terminology to understand the rest of this work. 2.1 first-order logic over lra in this work, we use first-order logic (fol) to give explanations with guarantees of correctness. we use quantifier-free first-order formulas over the theory of linear real arithmetic (lra). then, first-order variables are allowed to take values from the real numbers \(\mathbb {r}\). for details, see [11]. therefore, we consider formulas as defined below: $$\begin{aligned} \begin{aligned} f, g &:= p \mid (f \wedge g) \mid (f \vee g) \mid (\lnot f) \mid (f \rightarrow g),\\ p &:= \sum ^n_{i=1} w_i x_i \le b \mid \sum ^n_{i=1} w_i x_i < b, \end{aligned} \end{aligned}$$ (1) such that f and g are quantifier-free first-order formulas over the theory of linear real arithmetic. moreover, p represents the atomic formulas such that \(n \ge 1\), each \(w_i\) and b are fixed real numbers, and each \(x_i\) is a first-order variable. observe that we allow the use of other letters for variables instead of \(x_i\), such as \(s_i\), \(z_i\), \(q_i\). for example, \((2.5x_1 + 3.1x_2 \ge 6) \wedge (x_1=1 \vee x_1=2) \wedge (x_1=2 \rightarrow x_2 \le 1.1)\) is a formula by this definition. observe that we allow standard abbreviations as \(\lnot (2.5x_1 + 3.1x_2 < 6)\) for \(2.5x_1 + 3.1x_2 \ge 6\). since we are assuming the semantics of formulas over the domain of real numbers, an assignment \(\mathcal {a}\) for a formula f is a mapping from the first-order variables of f to elements in the domain of real numbers. for instance, \(\{x_1 \mapsto 2.3, x_2 \mapsto 1\}\) is an assignment for \((2.5x_1 + 3.1x_2 \ge 6) \wedge (x_1=1 \vee x_1=2) \wedge (x_1=2 \rightarrow x_2 \le 1.1)\). an assignment \(\mathcal {a}\) satisfies a formula f if f is true under this assignment. for example, \(\{x_1 \mapsto 2, x_2 \mapsto 1.05\}\) satisfies the formula in the above example, whereas \(\{x_1 \mapsto 2.3, x_2 \mapsto 1\}\) does not satisfy it. a formula f is satisfiable if there exists a satisfying assignment of f. to give an example, the formula in the above example is satisfiable since \(\{x_1 \mapsto 2, x_2 \mapsto 1.05\}\) satisfies it. as another example, the formula \((x_1 \ge 2) \wedge (x_1 < 1)\) is unsatisfiable since no assignment satisfies it. given formulas f and g, the notation \(f \models g\) is used to denote logical consequence or entailment, i.e., each assignment that satisfies f also satisfies g. as an illustrative example, let \(f = (x_1 = 2 \wedge x_2 \ge 1)\) and \(g = (2.5x_1 + x_2 \ge 5) \wedge (x_1=1 \vee x_1=2)\). then, \(f \models g\). the essence of entailment lies in ensuring the correctness of the conclusion g based on the given premise f. in the context of computing explanations, as presented in [8], logical consequence serves as a fundamental tool for guaranteeing the correctness of predictions made by anns. therefore, our adaptation of the encoding proposed by tjeng et al. [17] also incorporates the principles of entailment for computing explanations. the relationship between satisfiability and entailment is a fundamental aspect of logic. it is widely known that, for all formulas f and g, it holds that \(f \models g\) iff \(f \wedge \lnot g\) is unsatisfiable. for instance, \(( x_1 = 2 \wedge x_2 \ge 1) \wedge \lnot ((2.5x_1 + x_2 \ge 5) \wedge (x_1=1 \vee x_1=2))\) has no satisfying assignment since an assignment that satisfies \((x_1 = 2 \wedge x_2 \ge 1)\) also satisfies \((2.5x_1 + x_2 \ge 5) \wedge (x_1=1 \vee x_1=2)\) and, therefore, does not satisfy \(\lnot ((2.5x_1 + x_2 \ge 5) \wedge (x_1=1 \vee x_1=2))\). since our approach builds upon the concept of logical consequence, we can leverage this connection in the context of computing explanations for anns. 2.2 mixed integer linear programming in mixed integer linear programming (milp), the objective is to optimize a linear function subject to linear constraints, where some or all of the variables are required to be integers [2]. milp is a crucial technique in our work for determining the lower and upper bounds of each neuron in the anns. for example, we utilize a minimization problem to determine the lower bound of neurons within anns. this process involves formulating an objective function that seeks to minimize the lower bound, subject to constraints that reflect the behaviour of anns. to illustrate the structure of a milp, we provide an example below: $$\begin{aligned} \begin{aligned} \min \quad & y_1 \\ \text {s.t.} \quad & 1 \le x_1 \le 3\\ & 3x_1 + s_1 2 = y_1 \\ & 0 \le y_1 \le 3x_1 2 \\ & 0 \le s_1 \le 3x_1 2\\ & z_1 = 1 \rightarrow y_1 \le 0 \\ & z_1 = 0 \rightarrow s_1 \le 0 \\ & z_1 \in \{0, 1\} \end{aligned} \end{aligned}$$ (2) in the milp in (2), we want to find values for variables \(x_1, y_1, s_1, z_1\) minimizing the value of the objective function \(y_1\) among all values that satisfy the constraints. variable \(z_1\) is binary since \(z_1 \in \{0, 1\}\) is a constraint in the milp, while variables \(x_1, y_1, s_1\) have the real numbers \(\mathbb {r}\) as their domain. the constraints in a milp may appear as linear equations, linear inequalities, and indicator constraints. indicator constraints can be seen as logical implications of the form \(z = v \rightarrow \sum ^n_{i=1} w_i x_i \le b\) such that z is a binary variable, v is a constant 0 or 1 [3]. an important observation is that a milp problem without an objective function corresponds to a satisfiability problem, as discussed in sect. 2.1. given that the approach for computing explanations relies on logical consequence, and considering the connection between satisfiability and logical consequence, we employ a milp solver to address explanation tasks. additionally, throughout the construction of the milp model, we utilize optimization, specifically employing a milp solver, to determine tight lower and upper bounds for the neurons of anns. 2.3 classification problems and artificial neural networks in machine learning, classification problems are defined over a set of n features \(\mathcal {f} = \{x_1, ..., x_n\}\) and a set of \(\mathcal {n}\) classes \(\mathcal {k} = \{c_1, c_2,...,c_\mathcal {n}\}\). in this work, we consider that each feature \(x_i \in \mathcal {f}\) takes its values \(v_i\) from the domain of real numbers. moreover, each feature \(x_i\) has an upper bound \(u_i\) and a lower bound \(l_i\) such that \(l_i \le x_i \le u_i\), and its domain is the closed interval \([l_i, u_i]\). this is represented as a set of domain constraints or feature space \(d = \{l_1 \le x_1 \le u_1,\text l_2 \le x_2 \le u_2, ..., l_n \le x_n \le u_n \}\). for example, a feature for the height of a person belongs to the real numbers and may have lower and upper bounds of 0.5 and 2.1 meters, respectively. furthermore, \(\{x_1 = v_1, x_2 = v_2, ..., x_n = v_n\}\) represents a specific point or instance of the feature space such that each \(v_i\) is in the domain of \(x_i\). an ann is a function that maps elements in the feature space into the set of classes \(\mathcal {k}\). a feedforward ann is composed of \(l+1\) layers of neurons. each layer \(l \in \{0, 1, ..., l\}\) is composed of \(n_l\) neurons, numbered from 1 to \(n_l\). layer 0 is fictitious and corresponds to the input of the ann, while the last layer, k corresponds to its outputs. layers 1 to \(l-1\) are typically referred to as hidden layers. let \(x^l_i\) be the output of the ith neuron of the lth layer, with \(i \in \{1,...,n_l\}\). the inputs to the ann can be represented as \(x^0_i\) or simply \(x_i\). moreover, we represent the outputs as \(x^l_i\) or simply \(o_i\). the values \(x^l_i\) of the neurons in a given layer l are computed through the output values \(x^{l-1}_j\) of the previous layer, with \(j \in \{1,...,n_{l-1}\}\). each neuron applies a linear combination of the output of the neurons in the previous layer. then, the neuron applies a nonlinear function, also known as an activation function. the output of the linear part is represented as \(\sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_{j} + b^{l}_{i}\) where \(w^{l}_{i,j}\) and \(b^{l}_{i}\) denote the weights and bias, respectively, serving as parameters of the ith neuron of layer l. in this work, we consider only feedforward anns with the rectified linear unit (\(\textrm{relu}\)) as activation function because it can be represented by linear constraints due to its piecewise-linear nature. this function is a widely used activation whose output is the maximum between its input value and zero. then, \(x^{l}_{i} = \textrm{relu}(\sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_{j} + b^{l}_{i})\) is the output of the \(\textrm{relu}\). for classification tasks, the last layer l is composed of \(n_l = \mathcal {n}\) neurons, one for each class. moreover, it is common to normalize the output layer using a softmax layer. consequently, these values represent the probabilities associated with each class. the class with the highest probability is chosen as the predicted class. however, we do not need to consider this normalization transformation as it does not change the maximum value of the last layer. thus, the predicted class is \(c_i \in \mathcal {k}\) such that \(i = \mathop {\mathrm {arg\,max}}\nolimits _{j \in \{1, ..., \mathcal {n}\}} x^l_j\). 3 explanations for anns with logical implications ignatiev et al. [8] proposed an algorithm that computes minimal explanations for anns, yielding a subset of the input features sufficient for the prediction. this approach is based on logic with guarantees on the correctness and minimality of explanations. a flowchart for computing explanations using such an algorithm is shown in fig. 1. fig. 1. flowchart for calculating explanations. full size image first, the ann and the feature space \(\{l_1 \le x_1 \le u_1,\text l_2 \le x_2 \le u_2, ..., l_n \le x_n \le u_n \}\) are encoded as a formula f, an instance \(\{x_1 = v_1, x_2 = v_2, ..., x_n = v_n\}\) of the feature space is encoded as a conjunction in a formula c, and the associated prediction by the ann is encoded as a formula e. then, it holds that \(c \wedge f \models e\). the minimal explanation \(c_m\) of c is calculated removing feature by feature from c. for example, given a feature \(x_i\) with value v in c, if \(c \setminus \{x_i=v\} \wedge f \models e\), feature \(x_i\) may be considered as irrelevant in the explanation and is removed from c. otherwise, if \(c \setminus \{x_i=v\} \wedge f \not \models e\), then \(x_i\) is kept in c since the same class cannot be guaranteed. this \(c \setminus \{x_i=v\}\) notation represents the removal of \(x_i=v\) from formula c. this process is described in algorithm 1 and is performed for all features. then, \(c_m\) is the result at the end of this procedure. this means that for the values of the features in \(c_m\), the ann makes the same classification, whatever the values of the remaining features. since to check entailments \(c \wedge f \models e\) is equivalent to test whether \(c \wedge f \wedge \lnot e\) is unsatisfiable and f, c and \(\lnot e\) are encoded as linear constraints and indicator constraints, such a entailment can be addressed by a milp solver. the encoding of anns used in [8] and originally proposed by fischetti and jo [5] uses implications to represent the behavior of the \(\textrm{relu}\) activation function. we encode an ann with \(l+1\) layers as in eqs. (3)–(5). in the following, we explain the notation. the encoding uses variables \(x^{l}_{i}\) and \(o_n\) with the same meaning as in the notation for anns. auxiliary variables \(s^{l}_{i}\) and \(z^{l}_{i}\) control the behaviour of \(\textrm{relu}\) activations. variable \(z^{l}_{i}\) is binary and if \(z^{l}_{i}\) is equal to 1, the \(\textrm{relu}\) output \(x^{l}_{i}\) is 0 and \(s^{l}_{i}\) is equal to the linear part. otherwise, the output \(x^{l}_{i}\) is equal to the linear part and \(s^{l}_{i}\) is equal to 0. the constant \(ub^{l}_{s,i}\) is the upper bound of variable \(s^{l}_{i}\), and the constant \(ub^{l}_{x,i}\) is the upper bound of variable \(x^{l}_{i}\). each variable \(x^{0}_{i}\) has also lower and upper bounds \(l_i\), \(u_i\), respectively, defined by the domain of the features. $$\begin{aligned} \left. \begin{aligned} &\sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_i + b^{l}_{i} = x^{l}_{i} s^{l}_{i}\\ &z^{l}_{i} = 1 \rightarrow x^{l}_{n} \le 0 \\ &z^{l}_{i} = 0 \rightarrow s^{l}_{n} \le 0 \\ &z^{l}_{i} \in \{0, 1\} \\ &0 \le x^{l}_{i} \le ub^{l}_{x,i} \\ &0 \le s^{l}_{i} \le ub^{l}_{s,i} \\ \end{aligned} \right\} l = 1, ..., l-1, \ i = 1, ..., n_l \end{aligned}$$ (3) $$\begin{aligned} &l_i \le x_{i} \le u_i,\quad i = 1, ..., n_0\end{aligned}$$ (4) $$\begin{aligned} &o_i = \sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_i + b^{l}_{i},\quad i = 1, ..., n_l \end{aligned}$$ (5) the constraints in (3)–(5) represent the formula f. the bounds \(ub^{l}_{x,i}\) are defined by isolating variable \(x^{l}_{i}\) from other constraints in subsequent layers. then, \(x^{l}_{i}\) is maximized to find its upper bound. a similar process is applied to find the bounds \(ub^{l}_{s,i}\) for variables \(s^{l}_{i}\). this optimization is possible due to the bounds of the features. furthermore, these bounds can assist the solver in accelerating the computation of explanations. therefore, the time required for this process must be considered when building f. to check the unsatisfiability of the expression \(c \wedge f \wedge \lnot e\), we still need to take into account the formula \(\lnot e\), referring to the prediction of the ann. given an input c predicted as class \(c_i\) by the ann, formula e must be equivalent to \(\bigwedge _{j=1, j \ne i}^{\mathcal {n}} o_i > o_j\). this formula asserts that the maximum value of the last layer is in output \(o_i\). therefore, \(\lnot e\) must ensure that \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\). since milp solvers can not directly represent disjunctions, we use implications (6) and a linear constraint (7) over binary variables to define \(\lnot e\). $$\begin{aligned} &q_j = 1 \rightarrow o_i \le o_j, \quad j \in \{1, ..., \mathcal {n}\} \setminus \{i\} \end{aligned}$$ (6) $$\begin{aligned} &\sum _{j=1, j \ne i}^{\mathcal {n}} q_j \ge 1\end{aligned}$$ (7) $$\begin{aligned} &q_{j} \in \{0, 1\}, \quad j \in \{1, ..., \mathcal {n}\} \setminus \{i\} \end{aligned}$$ (8) if an assignment \(\mathcal {a}\) satisfies \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\), then \(o_i \le o_j\) is true under \(\mathcal {a}\) for some j. therefore, the assignment \(\mathcal {a} \cup \{q_j \mapsto 1\}\) satisfies the eqs. (6)–(8). conversely, if an assignment \(\mathcal {a}\) satisfies eqs. (6)–(8), it clearly also satisfies \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\). we conclude this section with a proposition regarding the number of variables and constraints of the encoding discussed above. proposition 1 let c be an instance predicted as a class \(c \in \mathcal {k}\) by an ann, the formula \(c \wedge f \wedge \lnot e\) has \(n_0 + n_l + 2\sum _{l=1}^{n_{l-1}} n_l\) real variables and \(n_l 1 + \sum _{l=1}^{n_{l-1}} n_l\) binary variables. also, the formula has \(n_0 + 2n_l + 5\sum _{l=1}^{n_{l-1}}\) constraints. 4 explanations for anns without implications in this section, we present an adaptation of the encoding proposed by tjeng et al. [17] for logic-based explainability. in such a work, the authors originally used the encoding to find adversarial examples without using logical implications. even more importantly, such an encoding uses fewer variables and constraints compared to [5]. then, we expect that our adaptation can lead to a better execution time for both building the logical constraints and computing explanations. adapting the encoding in [17] to the context of computing explanations requires incorporating additional constraints that were not part of the original work. these new constraints represent the class predicted by the ann as a formula e, as seen in sect. 3. however, to maintain the concept of the original encoding without implications, we define these additional constraints without implications. in the following, we apply the same algorithm from fig. 1, but replacing the encoding of f as in [5] with the one in [17]. we encode an ann with \(l+1\) layers as in eqs. (9), (4) and (5). the variables \(x^{l}_{i}\) and \(o_i\) have the same meaning as in eqs. (3)–(5). furthermore, auxiliary variables \(s^l_i\) are not required, as observed in the encoding by fischetti and jo [5]. constants \(lb^{l}_{i}\) and \(ub^{l}_{i}\) are, respectively, the lower and upper bounds of \(\sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_j + b^{l}_{i}\). again, we find such bounds via a milp solver. the behavior of \(\textrm{relu}\) is modeled using these bounds and binary variables \(z^{l}_{i}\). if \(z^{l}_{i}\) is equal to 0, the \(\textrm{relu}\) output \(x^{l}_{i}\) is 0. otherwise, \(x^{l}_{i}\) is equal to \(\sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_j + b^{l}_{i}\). the bounds \(lb^{l}{i}\) and \(ub^{l}{i}\) are necessary to maintain the integrity of the set of constraints for the entire feature space. regardless of the value of \(z^{l}{i}\), the bounds ensure that the constraints remain valid for the entire feature space. $$\begin{aligned} \left. \begin{aligned} &x^{l}_{i} \le \sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_j + b^{l}_{i} lb^{l}_{i} (1 z^{l}_{i})\\ &x^{l}_{i} \ge \sum _{j=1}^{n_{l-1}} w^{l}_{i,j} x^{l-1}_j + b^{l}_{i} \\ &x^{l}_{i} \le ub^{l}_{i} z^{l}_{i} \\ &z^{l}_{i} \in \{0, 1\} \\ &x^{l}_{i} \ge 0 \\ \end{aligned} \right\} l = 1, ..., l-1, \ i = 1, ..., n_l \end{aligned}$$ (9) in our proposal for computing explanations, constraints in eqs. (9), (4) and (5) represent the formula f. as in sect. 3, an instance is a conjunction c, and the associated prediction by the ann is a formula e. given an input c predicted as class \(c_i\) by the ann, again formula \(\lnot e\) must ensure that \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\). therefore, we must add new constraints to represent \(\lnot e\). maintaining the concept of the original encoding in [17] without implications, we define these additional constraints accordingly. we employ binary variables \(q_j\) and the upper and lower bounds \(ub_j\) and \(lb_j\) of variables \(o_j\). as for \(lb^{l}_{i}\) and \(ub^{l}_{i}\), we find the bounds \(ub_j\) and \(lb_j\) through a milp solver. we recall such elements are not originally present in [17]. however, they are necessary for the context of computing explanations for anns. in equations (10)-(12) we represent our proposal for encoding formula \(\lnot e\), where the prediction associated with an input c is class \(c_i\). $$\begin{aligned} &o_i o_j \le (ub_i lb_j) (1 q_j) , \quad j \in \{1, ..., \mathcal {n}\} \setminus \{i\} \end{aligned}$$ (10) $$\begin{aligned} &\sum _{j=1, j \ne i}^{\mathcal {n}} q_j \ge 1 \end{aligned}$$ (11) $$\begin{aligned} &q_{j} \in \{0, 1\}, \quad j \in \{1, ..., \mathcal {n}\} \setminus \{i\} \end{aligned}$$ (12) in what follows, we prove that equations (10)-(12) correctly ensure that \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\). proposition 2 let \(\lnot e\) be defined as in equations (10)-(12). let \(i \in \{1, ..., \mathcal {n}\}\) be fixed and \(ub_i\) be such that \(o_i \le ub_i\). let \(lb_j\) be such that \(lb_j \le o_j\), for \(j \in \{1, ..., \mathcal {n}\} \setminus \{i\}\). therefore, $$ \lnot e \text { is satisfiable iff } \bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j \text { is satisfiable.} $$ proof if an assignment \(\mathcal {a}\) satisfies \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\), then \(o_i \le o_j'\) is true under \(\mathcal {a}\) for \(j' \ne i\). let \(\mathcal {a}' = \mathcal {a} \cup \{q_{j} \mapsto v \mid v=1 \text { if } j=j', \text { else } v=0, \text { for } j \in \{1, ..., \mathcal {n}\} \setminus \{i\}\}\) be an assignment. then, \(\mathcal {a}'\) imposes that \(o_i \le o_j'\) in equation (10) for \(j = j'\), which in clearly true under this assignment. for \(j \ne j'\), it follows that \(o_i o_j \le (ub_i lb_j)\) must hold, which is also true under \(\mathcal {a}'\) since \(lb_j \le o_j\) and \(o_i \le ub_i\). conversely, if an assignment \(\mathcal {a}\) satisfies equations (10)-(12), it satisfies some \(q_j'\), for \(j' \ne i\) by equation (11). moreover, \(\mathcal {a}\) satisfies \(o_i \le o_{j'}\) by equation (10), for \(j = j'\). therefore, \(\mathcal {a}\) also satisfies \(\bigvee _{j=1, j \ne i}^{\mathcal {n}} o_i \le o_j\). finally, we give a proposition on the number of variables and constraints in our adaptation of the encoding in [17]. proposition 3 let c be an instance predicted as a class \(c \in \mathcal {k}\) by an ann, then formula \(c \wedge f \wedge \lnot e\) has \(n_0 + n_l + \sum _{l=1}^{n_{l-1}} n_l\) real variables and \(n_l 1 + \sum _{l=1}^{n_{l-1}} n_l\) binary variables. moreover, the formula has \(n_0 + 2n_l + 4\sum _{l=1}^{n_{l-1}} n_l\) constraints. therefore, this encoding has \(\sum _{l=1}^{n_{l-1}} n_l\) fewer variables than the one presented in sect. 3. additionally, this encoding has \(\sum _{l=1}^{n_{l-1}} n_l\) fewer constraints. consequently, one would expect a reduction in running time for both building the logical constraints and computing explanations. 5 experiments in this section, we detail the experiments conducted to compare our proposal against the encoding presented in [5]. our evaluation consists of two main experiments. in the first one, we compare the two encodings using 12 datasets. in the second one, we conduct a detailed comparison using a single dataset. we vary the architecture of the trained anns to explore the effect of the number of layers and neurons. we evaluate the performance of each encoding in terms of time for building logical constraints and time for computing explanations. we explained all instances in a given dataset and calculated the average time and standard deviation to compare times for computing explanations. in the other case, given a trained ann on a dataset, we built the logical constraints 10 times and calculated the average time and standard deviation. next, we present the experimental setup, describing technologies, datasets, the trained anns and hyperparameters. after that, we discuss the results providing a comparative analysis of the running times for building logical constraints and computing explanations. finally, we highlight specific improvements observed in our proposal. 5.1 experimental setup we used python to implement the approaches and to run the experiments. tensorflow was used to manipulate anns, including the training and testing steps. cplex was used as the milp solver and accessed by the docplex library. we used 12 datasets from the uci machine learning repositoryfootnote 2 and penn machine learning benchmarksfootnote 3, each ranging from 9 to 32 integer, continuous, categorical or binary features. the number of instances in the selected datasets ranges from 156 to 691. the types of classification problems related to these datasets are binary and multi-class classification. the preprocessing performed on the datasets included one-hot encoding of the categorical data and normalization of the continuous features to the range [0, 1]. this normalization was not applied to the integer features to avoid transforming their space into continuous, which could compromise formal guarantees on the correctness of the algorithm. as far as we know, such a methodology was not considered in earlier works. the anns training was accomplished using a batch size of 4 and a maximum of 100 epochs, applying early stopping regularization with 10 epochs based on validation loss. the optimization algorithm used was adam and the learning rate was 0.001. the datasets split was \(80\%\) for training and \(20\%\) for validation. the ann architectures were limited to 2 layers to reduce the total running time, because many solver calls were performed in the experiments due to the large number of instances. each solver call deals with an np-complete problem, therefore, impacting the experiments running time. the first experiment compared the two encodings presented on the 12 datasets. the architecture of the trained anns is two hidden layers with 20 neurons each. for each dataset and the associated ann, the explanation of each instance was obtained using the algorithm 1 and both presented encodings. the second experiment performed the comparison of the two encodings presented using the voting dataset of the first experiment. this experiment was conducted in two cases. in the first case, the trained anns consist of one hidden layer with the number of neurons ranging from 10 to 100. in the second case, the anns consist of two hidden layers such that both layers contains the same number of neurons, ranging from 10 to 40. in both cases, the number of neurons in the layers increases in increments of 5. again, the explanations of each instance was obtained using algorithm 1 and both presented encodings. the objective of this experiment is to verify the influence of the number of layers and the number of neurons in both encodings. 5.2 results table 1. comparison of both encodings.full size table the results of the first experiment are shown in table 1. for each dataset, its number of features is indicated in parentheses. the column exp (s) refers to the average running time for computing explanations in seconds, and the standard deviation is also presented. the column build (s) refers to the average running time, in seconds, for building the logical constraints of the trained ann. the running time for finding the bounds of variables is included in the time for building the encodings. despite the encoding by fischetti and jo [5] achieved a better average running time for computing explanations in 7 out of 12 cases, both encodings generally perform similarly when considering the variability of the time. for instance, in the spect dataset, the average execution time of the encoding by fischetti and jo [5] (2.64 seconds) falls within the range of the average minus the standard deviation of our approach (\(3.21 1.81\)). this pattern is observed across several datasets. in summary, the results indicate that both encodings generally perform similarly in terms of running time for computing explanations, with minor variations across different datasets. with respect to the average running time for building the logical constraints, our adaptation generally outperformed the encoding by fischetti and jo [5]. however, there were exceptions noted, such as in the spect dataset. overall, our adaptation achieved an improvement of up to \(18\%\) compared to the other one, as seen in the heart-statlog dataset. in summary, the results indicate that our adaptation is consistently more efficient than the other approach for building logical constraints. the variability in the results, as shown by the standard deviations, further supports this conclusion. for instance, the average running time of our proposal is less than the average minus the standard deviation of the other approach in 9 out of 12 datasets. these cases are highlighted in bold in table 1. it is important to note that, in the spect dataset, the average running time of the encoding proposed by fischetti and jo [5] is less than the average minus the standard deviation of our adaptation. this indicates that, while our proposal appears generally more efficient, specific dataset characteristics can influence which encoding is more advantageous. our adaptation yielded notably superior results not only in the average time for building logical constraints but also in the overall time, which includes both computing explanations and constructing logical constraints. for instance, it achieved an improvement of up to \(16\%\) compared to the other approach, as seen in the heart-statlog dataset. fig. 2. comparison of average running time for computing explanations, using anns with one hidden layer and the voting dataset. full size image fig. 3. comparison of average running time for computing explanations, using anns with two hidden layers and the voting dataset. full size image the results of the second experiment are shown in figs. 2 and 3. the x-axis shows the number of neurons in each hidden layer in both figures. while the y-axis shows the average running time for computing explanations. furthermore, the standard deviation is indicated by the shaded region in both figures. figure 2 refers to anns with one hidden layer, while fig. 3 refers to anns with two hidden layers. figure 2 suggests that our proposal, for anns with one hidden layer, achieved a superior average running time for computing explanations in the voting dataset. this encoding outperforms the other approach with percentage improvements ranging from approximately \(7.69\%\) to \(40.82\%\). the most significant improvements are observed with higher neuron counts. for instance, with 85 neurons per layer, our proposal shows an improvement of around \(40.82\%\), and with 95 neurons, the improvement is about \(31.65\%\). moreover, our proposal generally exhibits similar or lower standard deviations, indicating more consistent performance across different number of neurons. on the other hand, fig. 3 depicts comparable results between both encodings for anns with two hidden layers. moreover, similar standard deviations were achieved for both encodings, indicating comparable levels of consistency in performance. the variability increases significantly with more complex networks. for example, with 35 and 40 neurons per layer, both encodings exhibit significant variability. furthermore, with 30 neurons per layer, our adaptation shows higher variability compared to the other approach. 6 conclusions and future work explanations for the outputs of anns are fundamental in many scenarios, due to critical systems, data protection laws, adversarial examples, among others. therefore, several heuristic methods have been developed to provide explanations for the decisions made by anns. however, these approaches lack guarantees of correctness and may also produce redundant explanations. logic-based approaches address these issues but often suffer from scalability problems. in this work, we compare two logical encodings of anns: one has been used in the literature to provide explanations [5, 8], and another [17] that we have adapted for our context of explainability. our experiments indicate that both encodings have similar running times for computing explanations, even as the number of neurons and layers increases. furthermore, our experimental results suggest that our adaptation is generally more efficient for anns with one hidden layer, while the performance advantage diminishes for anns with two hidden layers. however, our proposal achieved a better running time for building the encoding for anns with two layers, showing an improvement of up to \(18\%\). this can help to decrease the scalability issue for building the logical constraints given an ann. furthermore, this encoding obtained better results also in the overall time, i.e., the time for computing explanations plus the time for building the logical constraints, showing an improvement of up to \(16\%\). in the experiments of this work, we considered all instances of the datasets used, which considerably increased the experiments running time. as future work, we can change the design of experiments, using only a subset of the datasets, to allow the use of larger anns. more experiments are necessary, especially with additional layers and neurons, to further validate our findings and understand the performance of these encodings. furthermore, others encodings [9, 10] can be evaluated for computing logic-based minimal explanations for anns. moreover, in order to improve the scalability of computing logic-based explanations, the anns can be simplified, before or during building their encodings, via pruning or slicing as proposed by [12]. this results in equivalent anns with smaller sizes. notes 1.https://github.com/levicc8/explications-anns. 2.https://archive.ics.uci.edu/ml/. 3.https://github.com/epistasislab/penn-ml-benchmarks/. references audemard, g., lagniez, j.m., marquis, p., szczepanski, n.: computing abductive explanations for boosted trees. in: 26th aistats (2023) google scholar  bénichou, m., gauthier, j.m., girodet, p., hentges, g., ribière, g., vincent, o.: experiments in mixed-integer linear programming. math. program. 1, 76–94 (1971) article  mathscinet  math  google scholar  bonami, p., lodi, a., tramontani, a., wiese, s.: on mathematical programming with indicator constraints. math. program. 151(1), 191–223 (2015). https://doi.org/10.1007/s10107-015-0891-4 article  mathscinet  math  google scholar  choi, a., shih, a., goyanka, a., darwiche, a.: on symbolically encoding the behavior of random forests. in: 3rd fomlas (2020) google scholar  fischetti, m., jo, j.: deep neural networks and mixed integer linear optimization. constraints 23(3), 296–309 (2018). https://doi.org/10.1007/s10601-018-9285-6 article  mathscinet  math  google scholar  goodfellow, i.j., shlens, j., szegedy, c.: explaining and harnessing adversarial examples. in: 3rd iclr (2015) google scholar  gorji, n., rubin, s.: sufficient reasons for classifier decisions in the presence of domain constraints. in: 36th aaai (2022) google scholar  ignatiev, a., narodytska, n., marques-silva, j.: abduction-based explanations for machine learning models. in: 33rd aaai (2019) google scholar  katz, g., barrett, c., dill, d.l., julian, k., kochenderfer, m.j.: reluplex: an efficient smt solver for verifying deep neural networks. in: majumdar, r., kunčak, v. (eds.) computer aided verification: 29th international conference, cav 2017, heidelberg, germany, july 24-28, 2017, proceedings, part i, pp. 97–117. springer international publishing, cham (2017). https://doi.org/10.1007/978-3-319-63387-9_5 chapter  math  google scholar  katz, g., et al.: the marabou framework for verification and analysis of deep neural networks. in: 31st cav (2019) google scholar  kroening, d., strichman, o.: decision procedures: an algorithmic point of view. springer, berlin, heidelberg (2016). https://doi.org/10.1007/978-3-662-50497-0 book  math  google scholar  lahav, o., katz, g.: pruning and slicing neural networks using formal verification. in: 21st fmcad (2021) google scholar  liu, w., wang, z., liu, x., zeng, n., liu, y., alsaadi, f.e.: a survey of deep neural network architectures and their applications. neurocomputing 234, 11–26 (2017) article  math  google scholar  ribeiro, m.t., singh, s., guestrin, c.: why should i trust you?: explaining the predictions of any classifier. in: 22nd kdd (2016) google scholar  ribeiro, m.t., singh, s., guestrin, c.: anchors: high-precision model-agnostic explanations. in: 32th aaai (2018) google scholar  shih, a., choi, a., darwiche, a.: a symbolic approach to explaining bayesian network classifiers. in: 27th ijcai (2018) google scholar  tjeng, v., xiao, k.y., tedrake, r.: evaluating robustness of neural networks with mixed integer programming. in: 7th iclr (2019) google scholar  wu, m., wu, h., barrett, c.: verix: towards verified explainability of deep neural networks. in: 37th neurips (2023) google scholar  download references author information authors and affiliations instituto federal do ceará (ifce), fortaleza, brazil levi cordeiro carvalho, saulo a. f. oliveira & thiago alves rocha authors levi cordeiro carvalhoview author publications search author on:pubmed google scholar saulo a. f. oliveiraview author publications search author on:pubmed google scholar thiago alves rochaview author publications search author on:pubmed google scholar corresponding author correspondence to thiago alves rocha . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper carvalho, l.c., oliveira, s.a.f., rocha, t.a. (2025). comparing neural network encodings for logic-based explainability. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_20 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial neural networks explainable artificial intelligence logic-based explainable ai publish with us policies and ethics profiles saulo a. f. oliveira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an ensemble of llms finetuned with lora for ner in portuguese legal documents | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an ensemble of llms finetuned with lora for ner in portuguese legal documents conference paper first online: 30 january 2025 pp 127–140 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) an ensemble of llms finetuned with lora for ner in portuguese legal documents download book pdf download book epub rafael oleques nunes  orcid: orcid.org/0009-0007-8842-421x9, letícia maria puttlitz  orcid: orcid.org/0009-0002-0313-10179, antonio oss boll  orcid: orcid.org/0009-0004-5440-61269, andre spritzer  orcid: orcid.org/0009-0002-4232-15859, carla maria dal sasso freitas  orcid: orcid.org/0000-0003-1986-84359, dennis giovani balreira  orcid: orcid.org/0000-0002-0801-93939 & … anderson rocha tavares  orcid: orcid.org/0000-0002-8530-64689  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 528 accesses abstract given the high computational costs of traditional fine-tuning methods and the goal of improving performance,this study investigate the application of low-rank adaptation (lora) for fine-tuning bert models to portuguese legal named entity recognition (ner) and the integration of large language models (llms) in an ensemble setup. focusing on the underrepresented portuguese language, we aim to examine the reliability of extractions enabled by lora models and glean actionable insights from the results of both lora and llms operating in ensembles. achieving f1-scores of 88.49% for the lener-br corpus and 81.00% for the ulyssesner-br corpus, lora models demonstrated competitive performance, approaching state-of-the-art standards. our research demonstrates that incorporating class definitions and counting votes per class substantially improves llm ensemble results. overall, this contribution advances the frontiers of ai-powered legal text mining, proposing small models and initial prompt engineering to low-resource conditions that are scalable for broader representation. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others scaling and adapting large language models for portuguese open information extraction: a comparative study of fine-tuning and lora chapter © 2025 fostering judiciary applications with new fine-tuned models for legal named entity recognition in portuguese chapter © 2022 on the assessment of deep learning models for named entity recognition of brazilian legal documents chapter © 2023 1 introduction with the advances in natural language processing (nlp), the domain of legal nlp also follows in constant evolution, facing similar challenges and obstacles. one of the main areas of work in the legal nlp field is named entity recognition (ner), which aims to identify and extract entities in text, such as petitions, bills, and sentences from legal documents. the process of recognizing these entities has become challenging due to the lack of structure and predefined entities in corpora from the legal system. another obstacle is to work in the brazilian legal system domain, as it does not have the same number of documents and models as the english language. to address these problems, researchers have created multiple corpora to tackle the issue of ner in the brazilian legal domain, such as the lener-br [5] and ulyssesner-br [3], adding entities to multiple phrases and helping to create a legal ner-friendly corpus. additionally, many bert models were created to facilitate the brazilian legal ner, including legalbert-pt [28], bertikal [22], and many others. in this work, we propose a comprehensive investigation centered on two primary objectives: first, to analyze the efficacy of deploying lora to fine-tune bert architectures for ner tasks within the legal domain, in contrast with relevant literature; and second, to present a study on the design and implementation of prompt engineering tactics for llm ensemble modeling, aiming to enhance robustness. our core contributions are (i) delivering a study on the practicality and advantages of coupling lora and bert models within the specialized context of legal ner, supplemented by comparison with related works, and (ii) an exposition on exploiting assorted prompt ingredients for building llm ensembles. 2 related work natural language processing is continuously evolving with the introduction of new concepts. each new technique opens doors to several fields of work, each with its unique modeling approach and output. introduced by mikolov in 2013 [17], the word embeddings gave the area of natural language processing (nlp) a new view, allowing words to be represented in a vectorized manner. after that, in 2017, the transformer model [30] revolutionized much of the work in the nlp area, inspiring the creation of several models based on its ideas, such as bert [12] and gpt [23]. bert, for instance, utilizes only the encoder part of the transformer and can understand context and work on several other tasks that require text comprehension. meanwhile, gpt, utilizing the decoder, can generate text, among other tasks. another model worth mentioning is the elmo, introduced in 2018 [21], which represents terms in a vector space and makes them context-sensitive. researchers have been creating several domain-specific models in the last few years due to bert’s ability to pre-train on data. bertimbau [29], for instance, was developed specifically for making inferences with portuguese data, focusing mainly on brazilian portuguese corpora. it leverages the transformer architecture to achieve high performance in various natural language processing tasks such as sentence textual similarity and named entity recognition. for training, bertimbau utilized the brwac [31] corpus, which consists of a large web corpus for brazilian portuguese. in the legal domain, several models have been developed for various languages, including english [9], german [11], arabic [2], french [13], and portuguese [28]. the corpora is also a crucial step in obtaining the results, as it needs to be from the domain and have the correct entities tied to it. a few of the legal corpora from the brazilian legal system for named entity recognition are lener-br [5], ulyssesner-br [3], brazilian supreme court corpus [10] and cdjur-br [8]. concerning advances in models and the improvement of ner, previous studies have explored how transformer models can enhance ner performance. we focus on lener-br and ulyssesner-br, as the former is predominantly a judiciary corpus, and the latter is a legislative corpus. this approach provides an interesting perspective on two different aspects of the legal domain. regarding the lener-br corpus, bonifácio et al. (2020) [6] found that using a domain-specific corpus during the pre-training of multilingual bert can enhance the recognition of named entities. additionally, zanuz et al. (2022) [34] achieved state-of-the-art results by fine-tuning the lener-br corpus using bertimbau. for the ulyssesner-br corpus, albuquerque et al. (2023) [4] fine-tuned bertimbau and conducted the first evaluation of this corpus using bert models. nunes et al. (2024) [19] investigated how a semi-supervised technique can improve bertimbau’s performance in the legislative domain, demonstrating that such techniques can enhance model results. a recent study [20] focused on how a generative language model (glm) specialized in brazilian portuguese performs on both corpora. the authors found that in-context learning can provide initial results and demonstrated the potential of these models to extract entities. however, they noted that further studies are needed, as bert models have shown better performance. to the best of our knowledge, our work is the first to analyze the efficacy of using lora to fine-tune bert models specifically for legal ner tasks and to explore the design and implementation of prompt engineering techniques for using llm to ensemble legal ner models. these contributions offer a novel perspective on the study of legal ner and the practical advantages of combining lora and bert models within this specialized domain. 3 domain corpora this section presents two types of corpora that recognize legally named entities, each focusing on a different type of data. they are called lener-br [5] and ulyssesner-br [3], both sourced from the brazilian legal system. we chose these two corpora because they can access different fronts of legal domains by leveraging judiciary and legislative texts. both corpora are embedded within the legislative context. while sentences in lener-br are extracted from court and legislative texts, those in ulyssesner-br come from legislative inquiries and bills from the brazilian chamber of deputies. the first corpus, introduced in 2018 and called lener-br [5], focuses on data from the brazilian justice system. it is composed of a total of 70 legal documents from various types of courts and legislations, containing a total of 10,392 sentences and 318,073 tokens. the categories of the named entities include “person”, “legal cases”, “time”, “location”, “legislation” and “organization”. focusing on the legislative side of legal data, ulyssesner-br [3] consists of two types of data from the brazilian chamber of deputies: legislative consultations (st) and bills (pl). the st database has 790 sentences and 77,441 tokens, whereas the pl corpus contains 9,526 sentences with 138,741 tokens. both datasets include the following categories of named entities: “fundamento”, “organizacao,” “produtodelei,” “local,” “data,” and “evento.” 4 models and fine-tuning with lora this section provides an overview of the models used to recognize legally named entities. these models are variations of the bert model [12], each pre-trained with a specific type of data, along with the addition of a large language model. bertimbau [29], launched in 2020, is one of the first brazilian bert models. it was pre-trained using brwac [32], a large corpus of data from the brazilian web. two types of bertimbau were pre-trained: bertimbau base, with 12 layers, a hidden size of 768, 12 attention heads, and 110 million parameters; and bertimbau large, with 24 layers, a hidden size of 1024, 16 attention heads, and 330 million parameters. the maximum length for a sentence contains 512 tokens for both models. we used bertimbau base for the analysis. it was fine-tuned separately on the ulyssesner-br and lener-br corpora. the hyperparameters used for this model are specified in sect. 6.2. aiming to develop a legal-focused model, legalbert-pt [28] is a bert model pre-trained on brazilian legal corpora. the authors pre-trained two types of models: legalbert-pt sc and legalbert-pt fp. the sc model is formulated by pre-training a bert model from scratch, with the same configuration as the bertimbau base. differently, the fp model pre-trains a bertimbau-base with the domain-specific corpora. the corpora used in the article consist of multiple brazilian court cases and, for the sc model, portuguese wikipedia articles, totaling 1,500,000 legal documents and a vocabulary of 36,345 words. legalbert-pt fp was selected as the best-performing model. in this study, legalbert-pt fp was fine-tuned using the corpora described in sect. 3, with its hyperparameters detailed in sect. 6.2. finishing the bert variant models, bertikal [22] is a bert model trained on corpora from clippings, court cases and motions in the legal brazilian data. the previously described models are fine-tuned on lener-br and ulyssesner-br (see sect. 3) and used hyperparameters described in sect. 6.2. we perform our fine-tuning with lora [15], which is a method to reduce the number of trainable parameters and hence the training computational cost using matrix algebra. the method freezes the pre-trained parameters of the model and optimizes the rank-decomposition of the dense layer matrices. the only large language model (llm) we use is mistral 7b [16]. it is composed of 7 billion parameters, has a dimension size of 4096, 32 layers, a vocabulary size of 32000 and multiple other parameters and hyperparameters. we obtained the training corpus from the open web. as an llm, the mistral 7b model is capable of a diverse gamma of tasks, including knowledge, math, reasoning, comprehension, and code, among several others. we use this model for the ensemble technique explained in sect. 5 and for a zero-shot learning method. 5 an ensemble of language models ensemble learning aims to improve the predictive performance of a single model by training multiple models and combining their predictions [25]. a simple form of ensemble is voting, where base classifiers are presented with an input and each makes a prediction. subsequently, the prediction that receives the most votes is selected as the final output. we apply ensemble learning to combine multiple language models on the legal named entity recognition task. in our approach, bert models serve as the base classifiers for named entity recognition. after the bert models generates their predictions, mistral acts as an aggregator in the ensemble, combining the outputs. this method provides a unified result that incorporates and values all the predictions from the bert models. we based our prompt engineering on the following elements: persona, definitions, votes, answer format, sentence, and query. the prompt starts with a persona, which is used to impersonate a linguist specialized in political science to achieve the best possible performance [26]. we chose this role for the persona because it is a linguistic task that requires legal domain knowledge. we used the definitions of the entity classes to alleviate the problem that some categories have non-intuitive meanings based on their names. for instance, the category fundamentofootnote 1 does not seem directly linked with legal norms or bills. thus, we also give the entity class definitions in the prompt. definitions were sourced from corpora [3, 5] and translated into portuguese. however, we adapted the definitions for person and location from harem [27] because both corpora were influenced by harem and adhered to its class definitions. we designed the votes to provide information to the model regarding the classes given more or fewer votes from each classifier. we present the vote in the format class: x votes, where class is the name of the entity class, and x is the number of votes. in the answer format, we encourage the model to use the chain of thought (cot) [33] by explaining the answer before providing the entity class. we divided the answers into explanation and answer to facilitate post-processing to obtain the class. finally, at the end of the prompt, we provide the sentence and ask the model to assign the right entity class to a term or say that it does not have a class in question-answer format. we provided entities from the classifiers in question, definitions, and votes. we also tested using all the entities for the definitions and questions. all classes used in the prompt are given with the first letter in uppercase and the rest in lowercase. we also use space to divide n-gram terms, as in produto de lei. we opt to use this format so that the names remain in a more fluid language, providing a better context and avoiding unnecessary splitting in more tokens of the terms. sometimes, the model gives answer classes near the goals but with slightly different names (e.g. typos). we used sentence-bert (sbert) [24] for post-processing, along with the classes [20]. after comparison, we converted the answer into a vector annotating the entity of the term in the sentence. the vector is incrementally annotated because we use a prompt for each term. 6 experimental evaluation this section provides an overview of the experimental environment, encompassing the principal libraries utilized, hyperparameters, and a thorough explanation of the hyperparameter tuning carried out for the lora models. in addition, we discuss the performance metrics utilized to evaluate our models. 6.1 setup we conducted the experiments on a computer with 12 gb ram and an nvidia geforce rtx 4070 gpu. owing to its broad support for libraries related to machine learning and natural language processing, we decided to implement our experiments using python 3.7.6. quantization, performed using the bitsandbytes libraryfootnote 2, was employed to reduce memory requirements and computational costs. the bert and mistral models were obtained from the huggingface hub. the functions and methods of lora and running the models were used from huggingface. 6.2 hyperparameters in this section, we describe the experiment hyperparameters. we first present the hyperparameter search and final values for lora training. here, we present the values used to train the bert models. finally, we present the hyperparameters for quantization and generate responses to the llm. lora. we used optuna [1] to perform the hyperparameter search. we set the optimization range to drop out around 0.1 and 0.5, and the range of r and alpha between 16, 32, 64, and 128, with 100 iterations. the final hyperparameters were r = 32 alpha = 1.747406, dropout = 0.1 (without optimization) and bias = “all”. bert. we followed previous studies that used the same legal corpora to set the hyperparameters for the bert classifier [6, 19, 34]. the set of values was learning_rate = 1e−3, batch_size = 10, and weight_decay_size = 0.01. llm. we use 4-bit quantization to load the model in our machine, setting the attributes load_in_4bit = true, bnb_4bit_compute_dtype = torch.bfloat16, torch_dtype = torch.float16, device_map = “auto”. to generate the answer, we used the standard value of the method generate, setting max_new_tokens = 800 and pad_token_id = model.config.eos_token_id. 6.3 metrics we used seqeval [18] to compute the metrics. this library assesses the results by considering the sequence of tags assigned to each entity in a complete sentence, clearly recognizing entities beyond individual tokens. our primary metric is the f1-score, calculated for each class and overall. additionally, precision and recall were computed for each class, whereas accuracy was determined for the overall results. 7 results and discussion in this section, we present the outcomes of our experiments involving lora applied to bert models and llm as an ensemble approach. initially, we delve into the performance of individual lora classifiers, considering any potential connections to the pretraining domain of the underlying base models. subsequently, we reveal the combined results achieved through the llm ensemble technique and scrutinize the relationships between the selected models and prompts. finally, we compare our results with those existing state-of-the-art (sota) baselines. 7.1 lora classifiers table 1. f1-score for each corpus to the lora classifiers.full size table we present the final f1-score of the classifiers to each corpus in table 1. legalbert-pt presented the best result for lener-br, obtaining 88%, followed by bertimbau, with 87%. for ulyssesner-br, we obtained the inverse result, where bertimbau had the best result at 81%, followed by legalbert-pt with 76%. this result probably occurred because legalbert-pt follows bertimbau pretraining in judiciary documents (more information in sect. 4), which is the data domain of lener-br, whereas ulyssesner-br uses legislative documents, which can explain the lower results in this case. however, it is important to note that we cannot express the consistency of the difference between the close results because we did not use k-fold cross-validation to obtain the average result and variance around the folds. furthermore, we did not compute tests to obtain statistical significance. another observation regards the response format, which we discovered during manual inspection. specifically, instances exist where the generated outputs fail to adhere strictly to the prescribed formats. given this discrepancy, there is a risk that accurate responses might go undetected due to unexpected presentation, consequently contributing to reduced f1 scores. 7.2 llms applied as ensemble table 2. overall f1-score for ulyssesner-br using mistral 7b as ensembler.full size table table 3. overall f1-score for lener-br using mistral 7b as ensembler.full size table the performance resulting from employing mistral 7b as an ensemble approach can be found in tables 2 and 3. we conducted experiments covering diverse combinations of prompts and models to evaluate the effect of each model-prompt pairing. surprisingly, the combination of multiple models did not yield improved outcomes. the top three scores across both datasets featured bertimbau and legalbert-pt employed independently, echoing the trend seen in table 1, wherein bertimbau outperformed ulyssesner-br, whereas legalbert-pt proved superior when working with lener-br. regarding prompt engineering, our investigation revealed that utilizing definitional descriptions paired with voting yielded the most successful outcomes. this strategy offers the model clear class definitions and individual class preferences as indicated by the participating models. although the label distribution appears scattered throughout the findings, the majority converge near 50% for lener-br and fall below this threshold for ulyssesner-br. these observations underscore the significance of defining a constrained solution space when using llms, facilitating enhanced categorization capabilities. considering the exploratory findings presented thus far, it is important to highlight the necessity of performing more deliberate examinations to acquire robust statistical comparisons between various prompt structures and models, as discussed in sect. 7.1. furthermore, embracing a multifaceted validation strategy, including repetitive experimentation (k iterations), calculation of mean values, and determination of standard deviations, constitutes another key element in fostering greater certainty surrounding the obtained outcomes. 7.3 our results and the state-of-the-art table 4. overall f1-score for each corpus for our classifiers and sota classifiers. the results for ulyssesner-br are from nunes et al. (2024) [19], and the results for lener-br are from zanuz et al. (2022) [34].full size table table 5. f1-score for each class in ulyssesner-br for our classifier and from nunes et al. (2024) [19]. the underlined values are those near the established results.full size table table 6. f1-score for each class in lener-br for our classifier and from zanuz et al. (2022) [34]. the underlined values are those near the established results.full size table comparing our results with those of the sota, table 4 reveals that the lora classifiers achieved similar performance to previous studies [19, 34]. specifically, our lora classifier obtained results closely aligned with a classifier solely utilizing bertimbau [19] on the ulyssesner-br corpus and approached the performance of the classifier proposed by zanunz et al. (2022) [34] on the lener-br corpus. the classes that achieved results similar to those of nunes et al. (2024) [19] were pessoa, fundamento, evento, and data, as shown in table 5. notably, the structuring of entities, such as data (date) and pessoa (person), presents a plausible explanation for their relatively higher performance. dates typically adhere to specific formats, facilitating their distinction from other entities, whereas the names of individuals often follow recognizable patterns similar to those observed in fundamento (laws). conversely, evento (event) posed a more significant challenge because of the limited training and test examples available for this class. furthermore, a comparative analysis with zanuz et al. (2022) [34] revealed that classes such as pessoa, organizacao, and local demonstrate similar performance trends, as shown in table 6. this alignment could be attributed to factors such as the extensive training, validation, and test data available for each class and the inherent structural complexities associated with entities in these categories, as discussed previously. these results show the power of lora for the ner task, with the capacity to achieve good results by training a smaller number of parameters, i.e., we decrease the number of training params from 109,532,186 to only 1,291,789 params. thus, this approach results in small and faster models that can be used with less computer power and stored in less space. 8 conclusion in this work, we presented a study of using lora to fine-tune bert portuguese models to ner tasks in the legal domain. the lora models could achieve near results compared to state-of-art for each corpora tested. another point is that we have the advantage of training smaller models, which is an advantage to storing and running the model in machines with low processing. we also tested using mistral 7b applied to the ensemble, which was unproductive. the results were lower compared to the lora models. however, we recognize that some outputs do not respect the format requested, which can be a point that the right answers are not recognized. in future work, we aspire to incorporate hyperparameter tuning tailored to each model augmented with lora, potentially amplifying the overall performance. moreover, we intend to utilize k-fold cross-validation to derive more authoritative and comparable results than existing literature. regarding ensembling, we envision expanding our scope by exploring alternative prompts engineered using prompt manipulation techniques. complementarily, we seek to develop post-processing routines accommodating minor fluctuations in the output format, thereby minimizing false negatives and boosting performance indicators. testing other multilingual llms (e.g., llama2) and trialing retrieval mechanisms, such as in-context learning, are other possible ways to improve results. another interesting future approach is to compare our results with classic ensemble strategies, such as bagging [7] and boosting [14]. simple fusion schemes such as averaging and maximum selection may provide promising avenues warranting investigation in this dynamic domain. notes 1.in english: foundation. 2.https://github.com/timdettmers/bitsandbytes. references akiba, t., sano, s., yanase, t., ohta, t., koyama, m.: optuna: a next-generation hyperparameter optimization framework. in: proceedings of the 25th acm sigkdd international conference on knowledge discovery and data mining, pp. 2623–2631 (2019) google scholar  al-qurishi, m., alqaseemi, s., soussi, r.: aralegal-bert: a pretrained language model for arabic legal text (2022) google scholar  albuquerque, h.o., et al.: ulyssesner-br: a corpus of brazilian legislative documents for named entity recognition. in: pinheiro, v., et al. (eds.) propor 2022. lncs, vol. 13208, pp. 3–14. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_1 chapter  math  google scholar  albuquerque, h.o., et al.: on the assessment of deep learning models for named entity recognition of brazilian legal documents. in: moniz, n., vale, z., cascalho, j., silva, c., sebastião, r. (eds.) progress in artificial intelligence epia 2023. lecture notes in computer science(), vol. 14116, pp. 93–104. springer, cham (2023) math  google scholar  luz de araujo, p.h., de campos, t.e., de oliveira, r.r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: villavicencio, a., et al. (eds.) propor 2018. lncs (lnai), vol. 11122, pp. 313–323. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_32 chapter  math  google scholar  bonifacio, l.h., vilela, p.a., lobato, g.r., fernandes, e.r.: a study on the impact of intradomain finetuning of deep language models for legal named entity recognition in portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020, part i. lncs (lnai), vol. 12319, pp. 648–662. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_46 chapter  math  google scholar  breiman, l.: bagging predictors. mach. learn. 24, 123–140 (1996). https://doi.org/10.1007/bf00058655 article  math  google scholar  brito, m., et al.: cdjur-br-uma coleção dourada do judiciário brasileiro com entidades nomeadas refinadas. in: anais do xiv simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 177–186. sbc (2023) google scholar  chalkidis, i., fergadiotis, m., malakasiotis, p., aletras, n., androutsopoulos, i.: legal-bert: the muppets straight out of law school (2020) google scholar  correia, f.a., et al.: fine-grained legal entity annotation: a case study on the brazilian supreme court. inf. process. manag. 59(1), 102794 (2022) article  mathscinet  math  google scholar  darji, h., mitrović, j., granitzer, m.: german bert model for legal named entity recognition. in: proceedings of the 15th international conference on agents and artificial intelligence. scitepress science and technology publications (2023). https://doi.org/10.5220/0011749400003393 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2019) google scholar  douka, s., abdine, h., vazirgiannis, m., hamdani, r.e., amariles, d.r.: juribert: a masked-language model adaptation for french legal text (2022) google scholar  freund, y., schapire, r.e.: experiments with a new boosting algorithm. in: machine learning: proceedings of the thirteenth international conference, vol. 96, pp. 148–156 (1996) google scholar  hu, e.j., et al.: lora: low-rank adaptation of large language models (2021) google scholar  jiang, a.q., et al.: mistral 7b (2023) google scholar  mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space (2013) google scholar  nakayama, h.: seqeval: a python framework for sequence labeling evaluation (2018). https://github.com/chakki-works/seqeval, software available from https://github.com/chakki-works/seqeval nunes, r.o., balreira, d.g., spritzer, a.s., freitas, c.m.d.s.: a named entity recognition approach for portuguese legislative texts using self-learning. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 290–300 (2024) google scholar  oleques nunes., r., spritzer., a., dal sasso freitas., c., balreira., d.: out of sesame street: a study of portuguese legal named entity recognition through in-context learning. in: proceedings of the 26th international conference on enterprise information systems volume 1: iceis, pp. 477–489. insticc, scitepress (2024). https://doi.org/10.5220/0012624700003690 peters, m.e., et al.: deep contextualized word representations (2018) google scholar  polo, f.m., et al.: legalnlp-natural language processing methods for the brazilian legal language. in: anais do xviii encontro nacional de inteligência artificial e computacional, pp. 763–774. sbc (2021) google scholar  radford, a., narasimhan, k., salimans, t., sutskever, i.: improving language understanding by generative pre-training. openai technical report (2018) google scholar  reimers, n., gurevych, i.: making monolingual sentence embeddings multilingual using knowledge distillation. in: proceedings of the 2020 conference on empirical methods in natural language processing. association for computational linguistics (2020). https://arxiv.org/abs/2004.09813 sagi, o., rokach, l.: ensemble learning: a survey. wiley interdisc. rev. data min. knowl. discov. 8(4), e1249 (2018) article  math  google scholar  salewski, l., alaniz, s., rio-torto, i., schulz, e., akata, z.: in-context impersonation reveals large language models’ strengths and biases. in: thirty-seventh conference on neural information processing systems (2023). https://openreview.net/forum?id=cbsj53ldkc santos, d., cardoso, n.: a golden resource for named entity recognition in portuguese. in: vieira, r., quaresma, p., nunes, m.g.v., mamede, n.j., oliveira, c., dias, m.c. (eds.) propor 2006. lncs (lnai), vol. 3960, pp. 69–79. springer, heidelberg (2006). https://doi.org/10.1007/11751984_8 chapter  math  google scholar  silva, n., et al.: evaluating topic models in portuguese political comments about bills from brazil’s chamber of deputies. in: anais da x brazilian conference on intelligent systems. sbc, porto alegre, rs, brasil (2021). https://sol.sbc.org.br/index.php/bracis/article/view/19061 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese, pp. 403–417 (2020). https://doi.org/10.1007/978-3-030-61377-8_28 vaswani, a., et al.: attention is all you need (2017) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation (lrec 2018) (2018) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: calzolari, n., et al. (eds.) proceedings of the eleventh international conference on language resources and evaluation (lrec 2018). european language resources association (elra), miyazaki, japan (2018). https://aclanthology.org/l18-1686 wei, j., et al.: chain-of-thought prompting elicits reasoning in large language models. in: advances in neural information processing systems, vol. 35, pp. 24824–24837 (2022) google scholar  zanuz, l., rigo, s.j.: fostering judiciary applications with new fine-tuned models for legal named entity recognition in portuguese. in: pinheiro, v., et al. (eds.) propor 2022. lncs, vol. 13208, pp. 219–229. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_21 chapter  math  google scholar  download references acknowledgements this work has been partially funded by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. we also acknowledge financial support from the brazilian funding agency cnpq. author information authors and affiliations federal university of rio grande do sul, porto alegre, brazil rafael oleques nunes, letícia maria puttlitz, antonio oss boll, andre spritzer, carla maria dal sasso freitas, dennis giovani balreira & anderson rocha tavares authors rafael oleques nunesview author publications search author on:pubmed google scholar letícia maria puttlitzview author publications search author on:pubmed google scholar antonio oss bollview author publications search author on:pubmed google scholar andre spritzerview author publications search author on:pubmed google scholar carla maria dal sasso freitasview author publications search author on:pubmed google scholar dennis giovani balreiraview author publications search author on:pubmed google scholar anderson rocha tavaresview author publications search author on:pubmed google scholar corresponding author correspondence to rafael oleques nunes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper nunes, r.o. et al. (2025). an ensemble of llms finetuned with lora for ner in portuguese legal documents. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_9 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords named entity recognition large language models lora publish with us policies and ethics profiles rafael oleques nunes view author profile antonio oss boll view author profile andre spritzer view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature logic-based explanations for linear support vector classifiers with reject option | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper logic-based explanations for linear support vector classifiers with reject option conference paper first online: 12 october 2023 pp 144–159 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) logic-based explanations for linear support vector classifiers with reject option download book pdf download book epub francisco mateus rocha  orcid: orcid.org/0009-0002-2261-37679, thiago alves rocha  orcid: orcid.org/0000-0001-7037-96839, reginaldo pereira fernandes ribeiro  orcid: orcid.org/0009-0000-0250-57139 & … ajalmar rêgo rocha  orcid: orcid.org/0000-0002-4512-55319  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 531 accesses 2 citations 2 altmetric abstract support vector classifier (svc) is a well-known machine learning (ml) model for linear classification problems. it can be used in conjunction with a reject option strategy to reject instances that are hard to correctly classify and delegate them to a specialist. this further increases the confidence of the model. given this, obtaining an explanation of the cause of rejection is important to not blindly trust the obtained results. while most of the related work has developed means to give such explanations for machine learning models, to the best of our knowledge none have done so for when reject option is present. we propose a logic-based approach with formal guarantees on the correctness and minimality of explanations for linear svcs with reject option. we evaluate our approach by comparing it to anchors, which is a heuristic algorithm for generating explanations. obtained results show that our proposed method gives shorter explanations with reduced time cost. furthermore, although our approach is demonstrated with linear svcs, it can be easily adapted to other classifiers with reject option, such as neural networks and random forests. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others classic machine learning methods chapter © 2023 supervised learning chapter © 2024 legitimate and spam sms classification employing novel ensemble feature selection algorithm article 29 july 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. logical analysis linear logic learning algorithms logic machine learning symbolic ai 1 introduction it is undeniable that artificial intelligence is increasingly being inserted into the daily lives of people [14], influencing the most complex decision taking tasks [22]. consequently, the most varied classification models in machine learning may come across instances that are difficult to correctly classify, be it due to lack of good data, to feature bias (color, size, gender) [7] or to noise present on the given instances [22]. the svc is one of the most well-known ml models. the linear support vector machine (svm), specifically, has been used in a variety of classification problems [2, 20, 23]. however, this model can also fall into the same previously said pitfalls, failing to give a correct classification. as such, the reject option (ro) strategy depicted in [3] can be used to remedy such cases. classification with ro is a paradigm that aims to improve reliability in decision support systems by handling the more complex cases and avoiding a higher error rate. it involves withholding and rejecting a sufficiently ambiguous classification result, i.e. when the instance in question is situated too close to the decision boundary of the classifier. these cases are separated to be dealt with in another specialized way, be it through other models or with the assistance of a human specialist [17]. since classification with ro comprises a set of techniques, in this work we consider the strategy where the classifier is trained as usual and the rejected cases are determined after the training phase. usually, this process requires finding a trade-off between the costs of misclassifications and rejections. the linear svc can already be globally interpreted to a certain extent based on the decision function, where the most important features are often associated with the highest weights [6]. such analysis, however, is not capable of giving decisive answers on more specific cases [12]. an example of this is depicted and more thoroughly explored in sect. 3. this gives a margin for questionable explanations, especially when ro is present due to the added level of complexity. therefore the need for a more thorough, instance-based explanation method. then, in this work, we consider instance-based explanations [7]. specifically, the objective of instance-based explanations is to provide interpretable insights by highlighting relevant features that influenced the output of an ml model on a given instance. by linking the output to specific instances and their features, users can gain a better understanding of the reasons behind the predictions of ml models. the popularization of concepts of explainable artificial intelligence (xai) increased efforts to explain most complex models [10]. these have mostly been done through heuristic methods, the more prevalent approach, such as lime [18], shap [15], and anchors [19]. these tend to be model-agnostic and not able to guarantee a trustworthy explanation with regard to the characteristics of the model or the correctness of the answers [10]. then, their explanations can be often proven wrong through counterexamples that expose their contradictions, leading to even more doubts regarding how much the ml model can be trusted. thus, the importance of trustworthy ml models increases the need for logic-based approaches to explain the decisions made by these models [9, 10]. the computation of explanations in these approaches rigorously explores the entire feature space. due to this, such explanations are provably correct and hold for any point in the space, which therefore makes them trustworthy [7]. moreover, logic-based approaches can guarantee minimality of explanations. minimality is important since succinct explanations seem easier to be interpreted by humans. recent years have seen a surge in research dedicated to investigating logic-based xai for ml models, such as neural networks, naive bayes, random forests, decision trees, and boosted trees [1, 8, 9, 11, 16]. due to the importance of reject option and logic-based explainability for trustworthy ml models, this work proposes a logic-based approach to explain linear svcs with reject option. given that rejected instances may be further analyzed by specialists, explanations regarding the causes of rejections can reduce the workload of these professionals. our proposal builds on work from the literature of logic-based explainability for traditional ml models, i.e. without reject option [9]. we compute explanations for linear svcs with ro by solving a set of logical constraints, specifically, boolean combinations of linear constraints. despite the fact that we consider a linear svc with ro in this work, our approach can be easily adapted for other ml models with ro, such as neural networks. we conducted experiments to compare our approach against anchors, a heuristic method that generates explanations by locally exploring the feature subspace close to a given instance [19], through six different datasets. the explanations generated by anchors are expressed as rules designed to highlight important features. therefore, the resulting explanations of anchors are similar to the ones computed by our approach. the results of our experiments show that our approach is capable of generating succinct explanations up to 286 times faster than anchors, in scenarios with and without the presence of reject option. 2 background 2.1 machine learning and binary classification problems in machine learning, binary classification problems are defined over a set of features \(\mathcal {f} = \{f_1, ..., f_n\}\) and a set of two classes \(\mathcal {k} = \{-1, +1\}\). in this paper, we consider that each feature \(f_i \in \mathcal {f}\) takes its values \(x_i\) from the domain of real numbers. moreover, each feature \(f_i\) has an upper bound \(u_i\) and a lower bound \(l_i\) such that \(l_i \le x_i \le u_i\). then, each feature \(f_i\) has domain \([l_i, u_i]\). we represent this as a set of domain constraints \(d = \{l_1 \le f_1 \le u_1, l_2 \le f_2 \le u_2, ..., l_n \le f_n \le u_n \}\). besides, the notation \(\textbf{x} = \{f_1 = x_1, f_2 = x_2, ..., f_n = x_n\}\) represents a specific point or instance such that each \(x_i\) is in the domain of \(f_i\). a binary classifier \(\mathcal {c}\) is a function that maps elements in the feature space into the set of classes \(\mathcal {k}\). for example, \(\mathcal {c}\) can map instance \(\{f_1 = x_1, f_2 = x_2, ..., f_n = x_n\}\) to class \(+1\). usually, the classifier is obtained by a training process given as input a training set \(\{ \textbf{x}_i,y_i\}^{l}_{i=1}\), where \(\textbf{x}_i \in \mathrm{i\!r}^n\) is an input vector or instance and \(y_i \in \{-1, +1\}\) is the respective class label. then, for each input vector \(\textbf{x}_i\), its input values \(x_{i, 1}\), \(x_{i, 2}\), ..., \(x_{i, n}\) are in the domain of corresponding features \(f_1\), ..., \(f_n\). a well-known classifier and its training process are presented in subsect. 2.2. 2.2 support vector machine the svm [4] is a supervised machine learning model often used for classification problems. it uses the concept of an optimal separating hyperplane [4] to separate the data. on a \(\mathrm{i\!r}^n\) space, such a hyperplane is defined by a set of points \(\textbf{x}\) that satisfies \(\textbf{w}_o \cdot \textbf{x} + b = 0\), where \(\textbf{w}_o \in \mathrm{i\!r}^n\) is the optimal weight vector, \(\textbf{x} \in \mathrm{i\!r}^n \) is a feature vector with n features and an intercept (bias) \(b \in \mathrm{i\!r}\). a given training set \(\{ \textbf{x}_i,y_i\}^{k}_{i=1}\) is said to be linearly separable if there is \(\textbf{w}_o \in \mathrm{i\!r}^n\) and \(b \in \mathrm{i\!r}\) that guarantees the separation between positive and negative class instances without error. in other words, the following inequalities $$\begin{aligned} \text {for }i \in \{1, ..., k\}, {\left\{ \begin{array}{ll} \textbf{w}_o \cdot \textbf{x}_i + b \ge +1,\quad &{} \text {if } y_i = +1,\\ \textbf{w}_o \cdot \textbf{x}_i + b \le -1,\quad &{} \text {if } y_i = -1, \end{array}\right. } \end{aligned}$$ (1) must be satisfied to obtain the optimal hyperplane \(h_o = \{ \textbf{x} \mid \textbf{w}_o \cdot \textbf{x} + b = 0\}\). a hard margin svm (svm-hm) [21] can be used when the data is linearly separable and misclassifications are not allowed, maximizing the margin between two hyperplanes, \(h_+\) and \(h_-\), parallel to \(h_o\). there can be no training instances between \(h_+\) and \(h_-\). once maximizing the margin is similar to minimizing \(\frac{1}{2}||\textbf{w}||^2\), the optimization problem for obtaining the optimal parameters for the svm can be described as follows: $$\begin{aligned} \begin{aligned} \min _{\textbf{w},b} & \quad {\frac{1}{2}||\textbf{w}||^2} \\ \text {s.t.} & \quad y_i(\textbf{w} \cdot \textbf{x}_i + b) \ge 1,\quad \text {for }i \in \{ 1,...,k\}. \end{aligned} \end{aligned}$$ (2) thus, the decision function used for classifying input instances \(\textbf{x}\) is defined as \(d(\textbf{x}) = \textbf{w} \cdot \textbf{x} + b\), while the predicted label \(\hat{y} \in \mathcal {k}\) of an input \(\textbf{x}\) is given by $$\begin{aligned} \hat{y}= {\left\{ \begin{array}{ll} +1, \quad \text {if } d(\textbf{x}) \ge 0,\\ -1, \quad \text {if } d(\textbf{x}) < 0. \end{array}\right. } \end{aligned}$$ (3) however, in real-world problems, the training instances from the two classes can not be linearly separated by a hyperplane due to data overlapping. in order to overcome this situation, one must use soft-margin svms (svm-sm) in which misclassifications are allowed to happen. to do so, slack variables can be used to relax the constraints of the svm-hm [4, 21]. given this, the optimization problem for soft-margin svms is described as $$\begin{aligned} \begin{aligned} \min _{\textbf{w},b,\mathbf {\xi }} & \quad \frac{1}{2}||\textbf{w}||^2 c\sum ^{k}_{i=1}\xi _i \\ \text {s.t.} & \quad y_i(\textbf{w} \cdot \textbf{x}_i + b) \ge 1 \xi _i,\quad & \text {for } i \in \{1,...,k\}\\ & \quad \xi _i \ge 0,\quad & \text {for } i \in \{1,...,k\} \end{aligned} \end{aligned}$$ (4) where c is a trade-off between \(\frac{1}{2}||\textbf{w}||^2\) and \(\sum _{i}^n \xi _i\). thus, for a high enough value of c, minimizing the sum of errors while maximizing the separation margin leads toward the optimal hyperplane. this can be the same as the one found through svm-hm if the data is linearly separable. moreover, the decision function and predicted label for svm-hm and svm-sm are the same. 2.3 reject option classification reject option for classification problems, as depicted in [3], is a set of techniques that aim to improve reliability in decision support systems. in the case of a binary problem, it consists in withholding and rejecting a classification result that is ambiguous enough, i.e. when the instance is too close to the decision boundary of the classifier. for the linear svc, it is when the instance is too close to the separating hyperplane. then, these rejected instances are analyzed through another classification method or even by a human specialist [17]. in applications where a high degree of reliability is needed and misclassifications can be too costly, rejecting to classify an instance can be more beneficial than the risk of a higher error rate due to wrong classifications [5]. according to [3], the optimal classifiers that best handle such a relation can be achieved by the minimization of \(\hat{r}\) through $$\begin{aligned} \begin{aligned} \hat{r} = e + w_r r \end{aligned} \end{aligned}$$ (5) where r is the ratio of the number of rejected training instances to the number of instances in the entire training dataset; e is the ratio of the number of misclassified instances to the number of all the training instances without including those ones rejected; and \(w_r\) is a weight denoting the cost of rejection. a lower \(w_r\) gives room for a decreasing error rate at the cost of a higher quantity of rejected instances, with the opposite happening for a higher \(w_r\). a method is presented in [17] for single, standard binary classifiers that do not provide probabilistic outputs. for svcs, the proposed rejection techniques are based on the distance of instances to the optimal separating hyperplane. if the distance value is lower than a predefined threshold, then the instance is rejected. as such, a rejection region is determined after the training step of the classifier, with a threshold containing appropriate values being applied to the output of the classifier. therefore, applying this strategy to a standard binary svc leads to the following prediction cases: $$\begin{aligned} \hat{y} = {\left\{ \begin{array}{ll} &{}+1, \quad \text {if } f(\textbf{x}) > t_+,\\ &{}-1, \quad \text {if } f(\textbf{x}) < t_-,\\ &{}\ \ 0, \quad \text {otherwise}, \end{array}\right. } \end{aligned}$$ (6) where \(t_+\) and \(t_-\) are the thresholds for the positive class and negative class, respectively, and 0 is the rejection class. furthermore, these thresholds are chosen to generate the optimal reject region, which corresponds to the region that minimizes \(\hat{r}\) by producing both the ratio of misclassified instances e and the ratio of rejected instances r. 2.4 first-order logic in order to give explanations with guarantees of correctness, we use first-order logic (fol) [13]. we use quantifier-free first-order formulas over the theory of linear real arithmetic. then, first-order variables are allowed to take values from the real numbers. therefore, we consider formulas as defined below: $$\begin{aligned} \begin{aligned} \varphi , \psi &:= s \mid (\varphi \wedge \psi ) \mid (\varphi \vee \psi ) \mid (\lnot \varphi ) \mid (\varphi \rightarrow \psi ),\\ s &:= \sum ^n_{i=1} a_i z_i \le e \mid \sum ^n_{i=1} a_i z_i < e, \end{aligned} \end{aligned}$$ (7) such that \(\varphi \) and \(\psi \) are quantifier-free first-order formulas over the theory of linear real arithmetic. moreover, s represents the atomic formulas such that \(n \ge 1\), each \(a_i\) and e are concrete real numbers, and each \(z_i\) is a first-order variable. for example, \((2.5z_1 + 3.1z_2 \ge 6) \wedge (z_1=1 \vee z_1=2) \wedge (z_1=2 \rightarrow z_2 \le 1.1)\) is a formula by this definition. observe that we allow standard abbreviations as \(\lnot (2.5z_1 + 3.1z_2 < 6)\) for \(2.5z_1 + 3.1z_2 \ge 6\). since we are assuming the semantics of formulas over the domain of real numbers, an assignment \(\mathcal {a}\) for a formula \(\varphi \) is a mapping from the first-order variables of \(\varphi \) to elements in the domain of real numbers. for instance, \(\{z_1 \mapsto 2.3, z_2 \mapsto 1\}\) is an assignment for \((2.5z_1 + 3.1z_2 \ge 6) \wedge (z_1=1 \vee z_1=2) \wedge (z_1=2 \rightarrow z_2 \le 1.1)\). an assignment \(\mathcal {a}\) satisfies a formula \(\varphi \) if \(\varphi \) is true under this assignment. for example, \(\{z_1 \mapsto 2, z_2 \mapsto 1.05\}\) satisfies the formula in the above example, whereas \(\{z_1 \mapsto 2.3, z_2 \mapsto 1\}\) does not satisfy it. a formula \(\varphi \) is satisfiable if there exists a satisfying assignment for \(\varphi \). then, the formula in the above example is satisfiable since \(\{z_1 \mapsto 2, z_2 \mapsto 1.05\}\) satisfies it. as another example, the formula \((z_1 \ge 2) \wedge (z_1 < 1)\) is unsatisfiable since no assignment satisfies it. the notion of satisfiability can be extended to sets of formulas \(\vargamma \). a set of first-order formulas is satisfiable if there exists an assignment of values to the variables that makes all the formulas in \(\vargamma \) true simultaneously. for example, \(\{(2.5z_1 + 3.1z_2 \ge 6), (z_1=1 \vee z_1=2), (z_1=2 \rightarrow z_2 \le 1.1)\}\) is satisfiable given that \(\{z_1 \mapsto 2, z_2 \mapsto 1.05\}\) jointly satisfies each one of the formulas in the set. it is well known that, for all sets of formulas \(\vargamma \) and all formulas \(\varphi \) and \(\psi \), $$\begin{aligned} \begin{aligned} \vargamma \cup \{\varphi \vee \psi \} \text { is unsatisfiable iff } & \vargamma \cup \{\varphi \} \text { is unsatisfiable and }\\ & \vargamma \cup \{\psi \} \text { is unsatisfiable.} \end{aligned} \end{aligned}$$ (8) given a set \(\vargamma \) of formulas and a formula \(\varphi \), the notation \(\vargamma \,\models \,\varphi \) is used to denote logical consequence, i.e., each assignment that satisfies all formulas in \(\vargamma \) also satisfies \(\varphi \). as an illustrative example, let \(\vargamma \) be \(\{z_1 = 2, z_2 \ge 1\}\) and \(\varphi \) be \((2.5z_1 + z_2 \ge 5) \wedge (z_1=1 \vee z_1=2)\). then, \(\vargamma \,\models \,\varphi \) since each satisfying assignment for all formulas in \(\vargamma \) is also a satisfying assignment for \(\varphi \). moreover, it is widely known that, for all sets of formulas \(\vargamma \) and all formulas \(\varphi \), $$\begin{aligned} \begin{aligned} \vargamma \,\models \,\varphi \text { iff } \vargamma \cup \{\lnot \varphi \} \text { is unsatisfiable.} \end{aligned} \end{aligned}$$ (9) for instance, \(\{ z_1 = 2, z_2 \ge 1, \lnot ((2.5z_1 + z_2 \ge 5) \wedge (z_1=1 \vee z_1=2)) \}\) has no satisfying assignment since an assignment that satisfies \((z_1 = 2 \wedge z_2 \ge 1)\) also satisfies \((2.5z_1 + z_2 \ge 5) \wedge (z_1=1 \vee z_1=2)\) and, therefore, does not satisfy \(\lnot ((2.5z_1 + z_2 \ge 5) \wedge (z_1=1 \vee z_1=2))\). finally, we say that two first-order formulas \(\varphi \) and \(\psi \) are equivalent if, for each assignment \(\mathcal {a}\), both \(\varphi \) and \(\psi \) are true under \(\mathcal {a}\) or both are false under \(\mathcal {a}\). we use the notation \(\varphi \equiv \psi \) to represent that \(\varphi \) and \(\psi \) are equivalent. for example, \(\lnot ((z_1 + z_2 \le 2) \wedge z_1 \ge 1)\) is equivalent to \((\lnot (z_1 + z_2 \le 2) \vee \lnot (z_1 \ge 1))\). besides, these formulas are equivalent to \(((z_1 + z_2 > 2) \wedge z_1 < 1)\). 2.5 xai and related work some ml models are able to be interpreted by nature, such as decision trees [18]. others, such as neural networks and boosted trees, have harder-to-explain outputs, leading to the use of specific methods to get some degree of explanation [10]. one of the most predominant ways to achieve this is through the use of heuristic methods for generating instance-dependent explanations, which can be defined as a local approach. these analyze and explore the sub-space close to the given instance [7, 10]. some of the most well-known heuristic methods are lime, shap, and anchors. these approaches are model-agnostic, generating local explanations while not taking into account the instance space as a whole [15, 18, 19]. this, in turn, allows the explanations to fail when applied, since they can be consistent with instances predicted in different classes. moreover, they can include irrelevant elements which could be otherwise removed while still maintaining the correctness of the answer [10]. explanations with irrelevant elements may be harder to understand. anchors have been shown as a superior version to lime, having a better accuracy with the resultant explanations [19]. the explanations obtained by anchors are decision rules designed to highlight which parts (features) of a given instance are sufficient for a classifier to make a certain prediction while being intuitive and easy to understand. for example, consider an instance \(i = \{\)sepal_length \( = 5.0\), sepal_width \( = 2.3\), petal_length \( = 3.3\), petal_width \( = 1.0\}\) of the well known iris dataset predicted as class versicolor by a classifier \(\mathcal {c}\). an example of an explanation obtained by anchors is the decision rule: $${\textbf {if }} {sepal\_width} \le 2.8 {\textbf { and }} {petal\_length} > 1.6 {\textbf { then }} {versicolor}.$$ however, due to the heuristic characteristic of anchors, there is still room for wrong explanations, which can lead to cases where, for the same set of given rules, different classes are predicted [10]. therefore, both explanation correctness and size can be set as of questionable utility if they can not be fully relied upon. since heuristic-based explanations are unreliable, logic-based approaches have been proposed recently for a variety of ml models [1, 8, 9, 11, 16]. logic-based approaches can guarantee the correctness of explanations, which makes them trustworthy. although most of the related work on logic-based explanations focused on traditional ml models, there appears to be a gap in the literature regarding the provision of such explanations specifically in the presence of ro. then, in this work we are considering instance-based explanations for linear svcs with ro, i.e. given a trained linear svc with ro and an instance \(i = \{f_1 = x_1, f_2 = x_2, ..., f_n = x_n\}\) classified as class \(c \in \{-1, +1, 0\}\), an explanation e is a subset of the instance \(e \subseteq i\) sufficient for the prediction c. therefore, by fixing the values of the features in e, the prediction is guaranteed to be c, regardless of the values of the other features. in other words, features in \(i \setminus e\) may take any value in their respective domains without changing the prediction. moreover, since succinct explanations seem easier to understand, one would like explanations e to be minimal, that is, for all subset \(e' \subset e\), it follows that \(e'\) does not guarantee the same prediction. informally, a minimal explanation provides an irreducible explanation that is sufficient for guaranteeing the prediction. 3 linear svcs may not be instance-based interpretable efforts have been made to bring explanations to linear svcs prediction outputs. these have often been done through the analysis of the weights that compose the decision function, where the most important features are associated with the highest weights [6]. however, this may not be enough to enable correct interpretations. although feature weights can give a rough indication of the overall relevance of features, they do not offer insight into the local decision-making process for a specific set of instances [12]. assume a binary classification problem where \(\mathcal {f} = \{f_{1}, f_{2}\}\), and a linear svc where \(\textbf{w} = \{w_1 = -0.8, w_2 = 2\}\) and \(b = 0.05\). the features \(f_i\) can take values in range [0, 1]. a visual representation is depicted in fig. 1. analyzing solely through the values of the weights, it could be assumed that the feature \(f_{2}\) is determinant and \(f_{1}\) is not, since \(|w_2| > |w_1|\). this would mean that, for any instance, feature \(f_2\) is more important than feature \(f_1\). however, for the instance \(\{f_{1}=0.0526, f_2 = 0.3\}\) predicted as class \(+1\), feature \(f_2\) is not necessary for the prediction, since for any value of \(f_{2}\) the class will not change. therefore, feature \(f_1\) is sufficient for the prediction of this instance. moreover, feature \(f_2\) is not sufficient for the prediction of this instance, since for \(f_1 = 1.0\) and \(f_2 = 0.3\) the prediction would change to \(-1\). therefore, for instance \(\{f_{1}=0.0526, f_2 = 0.3\}\), feature \(f_{1}\) would be determinant and \(f_{2}\) would not. fig. 1. classification through features \(f_{1}\) and \(f_{2}\). full size image while it is trivial to observe if feature \(f_{1}\) and \(f_{2}\) can be determinant, due to the simplicity of the given decision function, it becomes harder to do so as the number of features increases. furthermore, the presence of ro adds another layer of complexity to the problem, turning the interpretation task very non-trivial. hence, explanations given by weights evaluation may give room for incorrect interpretations. this raises the need for an approach to compute explanations with guarantees of correctness for linear svcs with reject option. 4 explanations for linear svms with reject option as depicted before, most methods that generate explanations for ml models are heuristic, which can bring issues about how correct and trustworthy the generated answers are. added that they tend not to take into account whether ro is present, these problems are further aggravated. as such, an approach that guarantees the correctness of explanations for ml models with ro is due. our approach is achieved by encoding the linear svc with ro as a first-order formula, following the ideas of [9] for traditional neural networks, i.e. without ro. here we need to take into account the reject option and its encoding as a first-order formula. for example, given a trained linear svc with ro defined by w, b, \(t_+\) and \(t_-\) as in eq. 6, and an instance i classified as class \(c = 0\), i.e. in the rejection class, we define the first-order formula p: $$\begin{aligned} \begin{aligned} p = (\sum ^n_{i=1} w_i f_i + b \le t_+) \wedge (\sum ^n_{i=1} w_i f_i + b \ge t_-). \end{aligned} \end{aligned}$$ (10) therefore, given an instance \(i = \{f_1 = x_1, f_2 = x_2, ..., f_n = x_n\}\) such that its features are defined by \(d = \{l_1 \le f_1 \le u_1, l_2 \le f_2 \le u_2, ..., l_n \le f_n \le u_n \}\), an explanation is a subset \(e \subseteq i\) such that \(e \cup d\,\models \,p\). therefore, each assignment that satisfies all the formulas in \(e \cup d\) must also satisfy p. in other words, prediction is guaranteed to be in the rejection class. moreover, since our goal is to obtain minimal explanations, e must be such that, for all subset \(e' \subset e\), it follows that \(e' \cup d \not \models p\). instances predicted in other classes are treated in a similar way. for example, if instance i is classified as class \(c = +1\), p is defined as \(\sum ^n_{i=1} w_i f_i + b > t_+\). it is worth noting that, in these formulas, each \(f_i\) is a first-order variable, and b, \(t_+\), \(t_-\) and each \(w_i\) are concrete real numbers. note that each first-order variable \(f_i\) is analogous to the \(z_i\) in sect. 2.4. moreover, each first-order variable \(f_i\) represents a feature. earlier work [9] outlined algorithm 1 for computing minimal explanations for traditional neural networks, i.e. without ro. by leveraging the insights from this earlier work, we can effectively employ algorithm 1 to find a minimal explanation for linear svcs with ro given i, d and p. if instance i is predicted as a class c by the linear svc with ro, then p is defined accordingly to reflect c. the minimal explanation e of i is calculated by setting e to i and then removing feature by feature from e. for example, given a feature \(f_i = x_i\) in e, if \(e \setminus \{f_i = x_i\}, d\,\models \,p\), then the value \(x_i\) of feature \(f_i\) is not necessary to ensure the prediction, and then it is removed from e. otherwise, if \(e \setminus \{f_i = x_i\}, d \not \models p\), then \(f_i = x_i\) is kept in e since it is necessary for the prediction. this process is performed for all features as described in algorithm 1. then, at the end of algorithm 1, for the values of features in e, the prediction is the same and invariant with respect to the values of the remaining features. by the result in 9, verifying entailments of the form \((e \setminus \{f_i = x_i\}) \cup d\,\models \,p\) can be done by testing whether the set of first-order formulas \((e \setminus \{f_i = x_i\}) \cup d \cup \{\lnot p\}\) is unsatisfiable. if p is a conjunction \(p_1 \wedge p_2\) as in 10, then \(\lnot p\) is equivalent to a disjunction \(\lnot p_1 \vee \lnot p_2\). therefore, we must check whether \((e \setminus \{f_i = x_i\}) \cup d \cup \{\lnot p_1\}\) is unsatisfiable and \((e \setminus \{f_i = x_i\}) \cup d \cup \{\lnot p_2\}\) is unsatisfiable, by the result in 8. algorithm 1 computing a minimal explanation full size image moreover, since \((e \setminus \{f_i = x_i\}) \cup d\) is a set of linear constraints and \(\lnot p_1\) and \(\lnot p_2\) are linear constraints, the unsatisfiability checkings can be achieved by two queries answered by a linear programming (lp) solver. therefore, if, for example, the set of linear constraints \((e \setminus \{f_i = x_i\}) \cup d \cup \{\lnot p_1\}\) has a solution, then \((e \setminus \{f_i = x_i\}) \cup d \not \models p\). from a computational complexity viewpoint, the linear programming problem is solvable in polynomial time. then, our approach for computing minimal explanations for linear svcs with reject option can also be solved in polynomial time. this is achieved by a linear number of calls to an lp solver, which further contributes to the efficiency and feasibility of our approach. 5 experiments in this paper, a total of 6 well known datasets are used. the vertebral column and the sonar datasets are available on the uci machine learning repositoryfootnote 1. the pima indians diabetes dataset is available on kagglefootnote 2. the iris, the breast cancer wisconsin, and the wine datasets are available through the scikit-learn packagefootnote 3. as a standard procedure, all features were scaled to range [0, 1]. to maintain the \(\{-1, +1\}\) class domain in our context of binary classification, the iris and the wine dataset (each with three classes) were binarized, changed to setosa-versus-all and class_0-versus-all, respectively. the other datasets are already binary. the classes were changed to follow values in \(\{-1, +1\}\). a summary of the datasets is presented in table 1. table 1. datasets details.full size table the classifiers. for each dataset, a linear svc was trained based on 70% of the original data, to simulate how a model would be trained for usage in real-world problems, with a regularization parameter \(c=1\) and stratified sampling. for finding the rejection region, a value of \(w_r = 0.24\) was used together with the decision function outputs based on training data. although the value itself is not too important for our purposes, it was chosen to ensure that rejections are neither penalized too softly nor too harshly, resulting in some rejected outputs. the selected rejection thresholds were obtained by minimizing \(\hat{r}\), as described in subsect. 2.3. for defining the values of \(t_+\) and \(t_-\), we determine a range of thresholds \(t = \{(t_{+}^{1}, t_{-}^{1}), ..., (t_{+}^{p},t_{-}^{p})\}\) containing the respective possible candidates. the maximum absolute value for \(t_+\) and \(t_-\) in t, respectively, is the highest and lowest output value of the decision function, i.e. the \(upper\_limit\) and the \(lower\_limit\), based on the training instances. we use decision function output values to achieve a reasonable reject region, that is able to properly reject ambiguous classification results for both known and unknown data. thus, the attainable thresholds are achieved through \(t = \{(i \cdot 0.01 \cdot upper\_limit,\text { }i \cdot 0.01 \cdot lower\_limit) \mid i \in \{1, ..., 100\} \}\). hence, the selected values of \(t_+\) and \(t_-\) are the pair that minimizes \(\hat{r}\). for each dataset, we obtained the best reject region defined by \(t_+\) and \(t_-\), the test accuracy of the classifier with ro, and the rejection ratio based on test data. the test accuracy of a classifier with ro is the standard test accuracy applied only to non-rejected instances. afterward, we determine the number of instances per class from both training and test data, i.e. the entire datasets. table 2 details these results and also the test accuracy of the classifier without reject option. table 2. reject region thresholds using training data. classifier accuracy without reject option, accuracy with reject option, and rejection ratio for test data. instances by class for each entire dataset.full size table observe that the reject region for the iris dataset did not return any rejected instances in the dataset. this is likely due to the problem being linearly separable. therefore, since our experiments rely on explaining the instances present in the dataset, this case of the iris dataset is treated as if reject option is not present. the reject region obtained for the wine dataset did not reject any of the test instances, therefore having no change in accuracy, likely due to the fact that the value chosen for \(w_r\) penalizes rejections too harshly. for the other datasets, the presence of reject option lead to a higher accuracy in all cases. this is expected since the rejected instances are not taken into account for evaluation. in addition, there was a substantial increase for both the vrtc and the pima dataset at the cost of rejecting roughly 48% and 60% of all the test instances, respectively. different values for \(w_r\) can be used to achieve desirable results, depending on how much a rejection must be penalized in each dataset. anchors. we compared our approach against the heuristic method anchors for computing explanations. anchors was designed to work with traditional multiclass classification problems, i.e. classifiers without reject option. since it was proposed as a model-agnostic method for explaining complex models, it should be more than enough for simpler models such as linear svcs. moreover, since we are explaining predictions of a linear svc with ro, we used the anchors explanation algorithm to treat the classifier with reject option as if it were a traditional classifier with classes in \(\{-1, 0, +1\}\). our approach. the prototype implementation of our approachfootnote 4 is written in python and follows algorithm 1. as an lp solver to check the unsatisfiability of sets of first-order sentences, we used coin-or branch-and-cut (cbc)footnote 5. moreover, the solver is accessed via the python api pulpfootnote 6. evaluation. all instances were used for generating explanations, including both the ones used for training and testing. this was done to simulate scenarios closer to real-world problems, where explanations for both previously known (training) and unknown (test) data may prove valuable. moreover, the main focus is on explaining classification results rather than model evaluation. since a per-instance explanation is done for each dataset, both mean elapsed time (in seconds) and the mean of the number of features in explanations are used as the basis for comparison. the results were separated by class to better evaluate and distinguish the rejected class from the others. moreover, time to compute and size of explanations were chosen as metrics to better reflect the importance of fast and concise explanations when used to assist specialists in better analyzing rejected instances. 5.1 results the results based on the running time and the size of explanations are presented in table 3 and table 4, respectively. following the iris dataset description in table 2, there are no results for the rejected class. our approach has shown to be up to, surprisingly, roughly 286 times faster than anchors. this happened for the sonr dataset, where the mean elapsed time of our method is 0.86 seconds against 246.14 seconds for anchors. these results show how much harder it can be for anchors to give an explanation as the number of features increases. furthermore, it is worth noting that our approach needs much less time to explain positive, negative, and rejected classes overall. table 3. time comparison between our approach and anchors (seconds). less is better.full size table table 4. explanation size comparison between our approach and anchors. less is better.full size table while our approach obtained more extensive explanations in certain cases, it also demonstrated the ability to be more succinct, generating explanations that were up to \(43\%\) smaller in other cases. in addition, our method is able to maintain the correctness of explanations. this highlights an expressive advantage of our approach over the heuristic nature of anchors. while anchors may provide explanations with fewer features in some cases, this is achieved due to the lack of formal guarantees on correctness. it is important to note that cases similar to what we discussed in sect. 3 occurred for the iris dataset. through weights \(\textbf{w} = \{w_1 = 0.8664049, w_2 = -1.42027753, w_3 = 2.18870793, w_4 = 1.7984087\}\) and \(b = -0.77064659\), obtained from the trained svc, it could be assumed that feature \(f_1\) might possibly be not determinant for the classification. however, for the instance \(\{f_1 = 0.05555556, f_2 = 0.05833333, f_3 = 0.05084746, f_4 = 0.08333333\}\) in class \(-1\) for example, feature \(f_1\) was in the explanation, while feature \(f_2\) was not present. furthermore, similar cases happened where \(f_2\) was in the explanation while \(f_4\) was not, even though \(|w_4| > |w_2|\). similar occurrences are present in other datasets, reinforcing that the points discussed in sect. 3 are not uncommon. as one more illustrative example, consider the linear svc with reject option trained on the vrtc dataset such that \(\textbf{w} = \{w_1 = 0.72863148,\) \(w_2 = 1.97781269,\) \(w_3 = 0.85680605,\) \(w_4 = -0.32466632, \) \(w_5 = -3.42937211,\) \(w_6 = 2.43522629\}\), \(b = 1.10008469\), \(t_= -0.3334\) and \(t_+ = 0.8396\). for the instance \(\{f_1 = 0.25125386, f_2 = 0.4244373, f_3 =0.7214483, f_4 = 0.20007403, f_5 = 0.71932466, f_6 = 0.15363128\}\) in the reject class, feature \(f_3\) was not in the explanation, while \(f_4\) is present in the explanation, despite the fact that \(|w_3| > |w_4|\). this is substantial for our approach since it demonstrates its capability of finding such cases. 6 conclusions in this paper, we propose a logic-based approach to generate minimal explanations for a classifier with ro while guaranteeing correctness. a trained linear svc with ro is used as a target model for explainability. our approach is rooted in earlier work on computing minimal explanations for standard machine learning models without ro. therefore, we encode the task of computing explanations for linear svcs with ro as a logical entailment problem. moreover, we use an lp solver for checking the entailment since all first-order sentences are linear constraints with real variables, and at most one disjunction occurs. our method is compared against anchors, one of the most well-known heuristic methods, through six different datasets. anchors was originally proposed as a model-agnostic method, and therefore expected to perform well for simpler models as the linear svc. nonetheless, we found that not only our approach takes considerably less time than anchors, but it also has reduced size of explanations for many instances. our approach achieved astonishing results in terms of efficiency, surpassing anchors by an impressive factor of up to approximately 286 times. however, our approach has limitations. since we guarantee correctness, the size of explanations may considerably increase, as observed for some cases in table 4. moreover, since we only assure minimal explanations, i.e. irreducible explanations that are sufficient for guaranteeing the prediction, it is possible that there are explanations with fewer features that our method does not guarantee to find out. our approach can be further improved in future work. for example, it can be easily adapted to other classifiers with ro, such as neural networks and random forests. another improvement is to provide more general explanations. for example, its possible that some features in an explanation may change values and still lead to the same prediction. then, a more general explanation may consider a range of values for each feature, such that the prediction does not change for any instances whose feature values fall within such ranges. this may enable a more comprehensive understanding of the model. notes 1.https://archive.ics.uci.edu/ml/datasets.php. 2.https://www.kaggle.com/datasets/uciml/pima-indians-diabetes-database. 3.https://github.com/scikit-learn/scikit-learn/tree/main/sklearn/datasets/data. 4.https://github.com/franciscomateus0119/logic-based-explanations-for-linear-support-vector-classifiers-with-reject-option. 5.https://github.com/coin-or/cbc. 6.https://github.com/coin-or/pulp. references audemard, g., bellart, s., bounia, l., koriche, f., lagniez, j.m., marquis, p.: on preferred abductive explanations for decision trees and random forests. in: 31st ijcai, pp. 643–650 (2022) google scholar  chlaoua, r., meraoumia, a., aiadi, k.e., korichi, m.: deep learning for finger-knuckle-print identification system based on pcanet and svm classifier. evol. syst. 10(2), 261–272 (2019) article  google scholar  chow, c.: on optimum recognition error and reject tradeoff. ieee trans. inf. theory 16(1), 41–46 (1970) article  math  google scholar  cortes, c., vapnik, v.: support-vector networks. mach. learn. 20, 273–297 (1995) article  math  google scholar  de oliveira, a.c., gomes, j.p.p., neto, a.r.r., de souza, a.h.: efficient minimal learning machines with reject option. in: 5th bracis, pp. 397–402. ieee (2016) google scholar  guyon, i., weston, j., barnhill, s., vapnik, v.: gene selection for cancer classification using support vector machines. mach. learn. 46, 389–422 (2002) article  math  google scholar  ignatiev, a.: towards trustable explainable ai. in: ijcai, pp. 5154–5158 (2020) google scholar  ignatiev, a., izza, y., stuckey, p.j., marques-silva, j.: using maxsat for efficient explanations of tree ensembles. in: aaai, vol. 36, pp. 3776–3785 (2022) google scholar  ignatiev, a., narodytska, n., marques-silva, j.: abduction-based explanations for machine learning models. in: aaai, vol. 33, pp. 1511–1519 (2019) google scholar  ignatiev, a., narodytska, n., marques-silva, j.: on formal reasoning about explanations. in: rcra (2020) google scholar  izza, y., ignatiev, a., marques-silva, j.: on explaining decision trees. arxiv preprint arxiv:2010.11034 (2020) krause, j., dasgupta, a., swartz, j., aphinyanaphongs, y., bertini, e.: a workflow for visual diagnostics of binary classifiers using instance-level explanations (2017) google scholar  kroening, d., strichman, o.: decision procedures. springer, heidelberg (2016). https://doi.org/10.1007/978-3-662-50497-0 lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  google scholar  lundberg, s.m., lee, s.i.: a unified approach to interpreting model predictions. in: advances in neural information processing systems, vol. 30 (2017) google scholar  marques-silva, j., gerspacher, t., cooper, m., ignatiev, a., narodytska, n.: explaining naive bayes and other linear classifiers with polynomial time and delay. neurips 33, 20590–20600 (2020) google scholar  mesquita, d.p., rocha, l.s., gomes, j.p.p., rocha neto, a.r.: classification with reject option for software defect prediction. appl. soft comput. 49, 1085–1093 (2016) google scholar  ribeiro, m.t., singh, s., guestrin, c.: “why should i trust you?” explaining the predictions of any classifier. in: 22nd acm sigkdd, pp. 1135–1144 (2016) google scholar  ribeiro, m.t., singh, s., guestrin, c.: anchors: high-precision model-agnostic explanations. in: aaai, vol. 32 (2018) google scholar  richhariya, b., tanveer, m., rashid, a.h., initiative, a.d.n., et al.: diagnosis of alzheimer’s disease using universum support vector machine based recursive feature elimination (usvm-rfe). biomed. signal process. control 59, 101903 (2020) article  google scholar  vapnik, v.: statistical learning theory. wiley (1998) google scholar  weld, d.s., bansal, g.: intelligible artificial intelligence. corr abs/1803.04263 (2018) google scholar  yang, w., si, y., wang, d., guo, b.: automatic recognition of arrhythmia based on principal component analysis network and linear support vector machine. comput. biol. med. 101, 22–32 (2018) article  google scholar  download references acknowledgments the authors thank funcap and cnpq for partially supporting our research work. author information authors and affiliations instituto federal de educação, ciência e tecnologia do ceará (ifce), fortaleza, brazil francisco mateus rocha, thiago alves rocha, reginaldo pereira fernandes ribeiro & ajalmar rêgo rocha authors francisco mateus rochaview author publications search author on:pubmed google scholar thiago alves rochaview author publications search author on:pubmed google scholar reginaldo pereira fernandes ribeiroview author publications search author on:pubmed google scholar ajalmar rêgo rochaview author publications search author on:pubmed google scholar corresponding author correspondence to francisco mateus rocha . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper rocha, f.m., rocha, t.a., ribeiro, r.p.f., rocha, a.r. (2023). logic-based explanations for linear support vector classifiers with reject option. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_10 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords logic-based explainable ai support vector machines classification with reject option publish with us policies and ethics profiles reginaldo pereira fernandes ribeiro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an empirical study of text features for identifying subjective sentences in portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an empirical study of text features for identifying subjective sentences in portuguese conference paper first online: 28 november 2021 pp 374–388 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) an empirical study of text features for identifying subjective sentences in portuguese download book pdf download book epub miguel de oliveira10 & tiago de melo10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1208 accesses 3 citations abstract studies in sentiment analysis have examined how different features are effective in identifying subjective sentences. several studies on sentiment analysis exist in the literature that have already performed some evaluation of nlp classifications. however, the vast majority of them did not handle texts in the brazilian portuguese language, and there is no one to consider the combination of sets of text features of nlp tasks with classifiers. therefore, in our investigation, we combined empirical features to identify subjective sentences in portuguese and provide a comprehensive analysis of each set of features’ relative importance using a representative set of user reviews. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others evaluating methods of different paradigms for subjectivity classification in portuguese chapter © 2020 a survey of sentiment analysis in the portuguese language article 06 july 2020 natural language processing techniques for hate speech evaluation for brazilian portuguese chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing linguistics machine translation natural language processing (nlp) 1 introduction a vast quantity of user reviews is constantly published on the web every day in recent years, thanks to the rapid development of social media. these reviews assist other users in making decisions, such as purchasing new products, as they are also useful for companies to improve their products. sentiment analysis (sa) is an area of study that aims to find computational techniques to extract and analyze people’s opinions and to identify the sentiment expressed in those opinions. subjectivity identification is an essential task of sa because most polarity detection tools are optimized for distinguishing between positive and negative texts. subjectivity identification hence ensures that factual information is filtered out and only opinionated information is passed on to the polarity classifier [9]. several studies have demonstrated that information provided by some textual features is valuable for sentiment classification [2, 11, 20]. many types of text features have been proposed and evaluated in the literature, such as syntactic features and part-of-speech features. sa for the english language is well advanced. however, the linguistic resources available for sentiment analysis in other languages are still limited [23]. portuguese is one of the languages with few linguistic resources available, despite being among the top five languages used online. thus, the small number of specific works and resources for portuguese-focused sentiment analysis provides a scenario with great challenges and opportunities. given this scenario, our study aims to present research regarding the use of machine learning algorithms for subjectivity identification of user reviews published in portuguese. in addition, we will also investigate which text features are more important to address during this task. in particular, we will show that certain features play a more important role than others under different circumstances. in this way, this work contributes to an analysis of these combinations for the sa task from texts written in the portuguese language, with the objective of answering the following research questions: rq1: is there a single machine learning algorithm for subjectivity identification that is always the best for a diversified set of datasets? rq2: given a set of feature groups, is there a group that is always the best regardless of the algorithm used? rq3: how do different feature combinations affect the predictive performance of a classifier? we carried out an exhaustive evaluation of combinations considering five classifiers for four sentiment analysis datasets in the brazilian portuguese language, with the aim being to answer these research questions. we chose four well-known algorithms used in text classification: gradient boosting trees (gbt), maximum entropy (me), random forest (rf), and support vector machine (svm). we also used automl, a machine learning algorithm where the model finds the solution to the given data and produces results automatically. specifically, we used autogluonfootnote 1 because it was one of the most recently released automl frameworks [14]. we didn’t focus on presenting the best combinations or even improving the state-of-the-art predictive performance for each dataset. instead, the goal was to raise the importance of text features for subjectivity identification. our main contribution is to report an extensive set of experiments aimed to evaluate the relative effectiveness of different linguistic features for subjectivity identification. the remainder of this paper is organized as follows. section 2 provides a brief description of the main related works. then, the method and materials are described in sect. 3. experiments are reported in sect. 4, where we also describe the evaluation and discuss the results. finally, we draw the conclusion and future works in sect. 5. 2 related work in this section, we discuss prior works related to subjectivity identification on user reviews. the main approach is using machine learning techniques with different linguistic features. there are many types of classifiers for this task using supervised learning algorithms, but the most popular classifiers are support vector machine (svm), naive bayes (nb), and maximum entropy [2, 3]. previous studies [2, 11] also have explored the use of english features, such as adjectives, nouns and phrases, and have applied these features to identify subjective sentences. chenlo and losada [11] in their survey paper evaluated a wide range of features that have been employed to date for subjectivity classification. some of the prominent features include pos, lexicon, number and proportion of subjective terms in the sentence, and so on. the subjectivity classification part is relevant to our study. moraes et al. [19] investigate the problem of subjectivity classification for the portuguese language. they created a corpus of tweets on the area of technology called computerbr. inspired by the methods for english, the evaluated methods were based on the use of sentiment lexicons and machine learning algorithms, such as svm and naive bayes. carvalho et al. [7] proposed a comparative study test between three learning algorithms (naive bayes, svm and maximum entropy) and three feature selection methods for classifying texts of election-related news in brazil. belisário et al. [5] focused specifically on the task of subjectivity classification. the authors reported the study and comparison of machine learning methods of different paradigms (naive-bayes and svm) to perform subjectivity classification of book review sentences in portuguese. the authors also evaluated lexical and discourse features. in summary, most methods for subjectivity identification are supervised and use different subsets of text features for english. therefore, there is not a clear picture of the impact of every feature set for the most common machine learning algorithms. thus, there is room for further studies in portuguese. in our work, we evaluate a large set of text features with machine learning algorithms, including a recent automatic machine learning approach, for subjectivity identification from texts written in the brazilian portuguese language. we evaluated all features and algorithms across four different datasets. 3 materials and methods we deal with a binary classification task: factual versus subjective sentences. this task can be performed by text classifiers composed of training data in a supervised strategy. 3.1 features the characteristics of user reviews will be encoded as features in vector representation. these vectors and the corresponding labels feed the classifiers. to build our classifiers, we considered the following sets of feature groups. pos features. part-of-speech (pos) information is used to locate different types of information of interest inside text documents. for instance, adjectives usually represent opinions, while adverbs are used as modifiers to represent the degree of expressiveness of opinions. for the extraction of pos tags, the sentences were initially tokenized. then the tokens were tagged with their respective word classes using spacyfootnote 2 pos tagger [16]. the tokens were then lowercased. the following pos featuresfootnote 3 were extracted from the processed sentences: adjective (adj), conjunction (adp), adverb (adv), verb auxiliary (aux), coordinating conjunction (cconj), determiner (det), interjection (intj), noun (noun), cardinal number (num), pronoun (pron), proper noun (propn), punctuation mark (punct), subordinating conjunction (sconj), symbol (sym), verbs (verb), comparative (comp), superlative (sup), and other tags (x). the latter is used for words that can not be assigned a real pos category for any reason. in table 1, each pos feature is described with an example and its feature values (fv). table 1. set of pos features.full size table syntactic patterns. natural language texts are based on syntactic patterns which create a sequence of words based on grammar rules. in our study, we used syntactic patterns as indicative of subjective sentences. the patterns were originally proposed by turney [25] and we adapted them to use spacy toolkit. in table 2, each syntactic pattern is described with an example and its feature values (fv). the rules are described in table 2 with arrows (\(\rightarrow \)) denoting immediate sequence between two pos tags, while not indicates that the absence of a given tag is expected. in feature 23, the verb can be in different forms such as participles (pcp), gerund (ger), perfect simple (ps), and imperfect (impf). table 2. set of syntactic patterns.full size table sentiment lexicons. some of the most important indicators in the analysis of subjective text are sentiment lexicons. these features are based on counting the sentiment-bearing terms that occur in the sentence [1, 10]. we consider the following terms: subjectives, positives, and negatives. the sentiment lexicon is a portuguese translation of the lexicon from pennebaker et al. [22], a well-known subjective classifier [4]. we include the number and percentage of opinionated terms in a sentence as features for our classifiers. we also consider the number and percentage of exclamation marks “!” and question marks “?” in the sentences because they usually indicate the strength of sentiment. in table 3, each sentiment lexicon feature is described with an example and its feature values (fv). table 3. set of lexicon features.full size table concept-level features. concept-level features can help with a semantic analysis of text through web ontologies or semantic networks. hence, they allow for aggregating conceptual and affective information associated with natural language opinions [24]. we adopted the same set of concept-level features used in [11], which in turn, measured different aspects of these concepts, including: pleasantness, attention, aptitude, sensitivity, and polarity. to do this, we used the senticnet corpus [6]. we have represented this group of features considering only each concept’s absolute score since highly opinionated concepts tend to have higher absolute values. in table 4, each concept-level feature is described with an example and its feature values (fv). table 4. set of concept-level features.full size table structural features. structural features encode the number of words, words with an upper letter, and the number of upper letters. intuitively, the number of words could be indicative of subjectivity. for instance, factual sentences may be shorter [11]. we include the ratio of the number of uppercase words and uppercase characters in a sentence as features for our classifiers. in table 5, each structural feature is described with an example and its feature values (fv). table 5. set of text structural features.full size table twitter features. because the language used on twitter is often informal and differs from traditional text types, we used the number of urls, mentions and elongated words as features [18]. we also include emoji sentiment scores provided in [17] and emoticons sentiment scores provided in [15]. in table 6, each twitter feature is described with an example and its feature values (vof). table 6. set of twitter features.full size table miscellaneous features. we adopted a set of miscellaneous features that did not belong to the aforementioned groups. as adopted by palshikar et al. [21], we include the number of named entities and datetimes. the intuition is that factual information generally has lots of dates and entities. we also include the number of words that are present in a dictionary [18]. therefore, misspelled words were not counted. we used a list of more than 300,000 wordsfootnote 4. in table 7, each miscellaneous feature is described with an example and its feature values (fv). table 7. set of miscellaneous features.full size table 3.2 models we chose four conventional methods frequently used in text classification when the features set was selected to investigate which feature groups are best between: gradient boosting trees (gbt), maximum entropy (me), random forest (rf), and support vector machine (svm). we also used autogluon because it is one of the most recently released automl frameworks. we used the python scikit-learnfootnote 5 library to build conventional methods. 3.3 recursive feature elimination recursive feature elimination (rfe) is considered a kind of wrapper feature selection method [8]. initially, all the features are selected for feature selection. at every iteration, some features will be removed from the full features set, based on the inference of the training model. the goal is to repeatedly build a model and choose the best-performing feature. this procedure is repeated until all features in the dataset are evaluated. features are then ranked according to the elimination order. as such, it is a greedy optimization for finding the best performing subset of features. the coefficients of features are generally used as the index of importance evaluation. the closer the coefficient to 0, the smaller effect on the target variable is. the importance ranking of the candidate features is finally achieved by following the sequence order of elimination of features during every iteration. 4 experiments 4.1 setup table 8 presents the datasets we have used in our experiments. dataset reli contains 2,000 sentences from user reviews of books taken from [12]. dataset computerbr consists of 2,281 sentences of tweets regarding computers taken from [19]. dataset tripadvisor contains 1,049 sentences from user reviews of restaurants taken from [20]. dataset hotel consists of 840 sentences from user reviews of hotel services taken from [13]. note that only dataset reli is well-balanced. table 8. summary of the datasets.full size table 4.2 metrics for a particular sentence, the class predicted by a model is considered correct if it matches the class assigned in the datasets described previously. we use traditional metrics for evaluating the performance of models macro-averaged precision (p), recall (r) and f-score (f1). for macro-averaged metrics, we compute the metrics for each class separately, then take their average to prevent bias towards high-frequency classes. 4.3 comparison of classifier performances to answer the research question rq1, we conducted experiments to confirm whether there is a single, best-suited machine learning algorithm for subjectivity identification for a diversified set of datasets. table 9 shows each classifier’s performance in terms of f-measure (f\(_1\)) and accuracy (a) along with its standard deviation. the bold values indicate the best scores. in this experiment, each result denotes an average of 5-fold cross-validation. table 9. classification performance of each classifier on the four datasets.full size table the results in table 9 reveal the following trends. overall, the gbt model achieved the best average f\(_1\) among all the datasets. the difference in the results with the best classifier (rf) in terms of the accuracy measure is only 0.001 on average. we also note that although the models used in our experiments were considerably different, there is minimal difference in their final results. for example, the difference between the best (gbt) and worst (me and ag) classifier was only 0.014 for f\(_1\) average. meanwhile, the difference between the best (gbt) and worst (svm) classifier was only 0.021 for accuracy average. to ensure that the results were not achieved by chance, the friedman test was performed on the results. the friedman test is a non-parametric test used to measure the statistical differences of methods over multiple datasets. for each dataset, the methods were ranked based on the results of classifier metrics. we adopted f\(_1\) metrics to rank the results. let m be the number of models evaluated and n be the number of datasets. the friedman test is distributed according to the fisher distribution with \(m 1\) and (\(m 1\))(\(n 1\)) degrees of freedom. the null hypothesis in the friedman test means that all methods perform equivalently at the significance level \(\alpha \). the null hypothesis is accepted when the value of fisher distribution is less than the critical value; otherwise, it is rejected. for a confidence interval \(\alpha \)=0.05, m=5, and n=4, the critical value was 3.259. therefore, the null hypothesis is accepted as 1.764. 4.4 feature ablation study to answer the research question rq2, we conducted a feature ablation study to evaluate the importance of the proposed features in the performance of the classifiers. an ablation study is performed by removing sets of features individually and evaluating the classifier’s performance to identify which of those features impact the most on the results. for each feature group, we retrained and retested the best performing classifier overall (gbt, as shown in sect. 4.3), evaluating its performance with 5-fold cross-validation and f\(_1\) score as a metric. in table 10, each row labeled as all -f represents the results obtained by training the classifier with all groups of features, except f. the row labeled as all shows the results for all groups of features together. for each dataset, we show the average f\(_1\) achieved between the two classes (factual and subjectivity), and the f\(_1\) for each class. we also provide the difference of each f\(_1\) measure from the score obtained by all groups of features together, labeled as loss. a positive loss indicates that a group f of features contributes significantly to the overall performance. a negative loss indicates that removing f from the features set, in fact, improves the classifier’s performance. we observed that different groups of features perform differently in each dataset. however, it is noticeable that the pos features always contribute to all datasets. this may indicate that part-of-speech tags are very relevant information for subjectivity classification in the portuguese language, being the only group in our experiments shown to be neither structure nor domain-specific. it is interesting to mention that the syntactic patterns group have a low contribution for reli, tripadvisor, and hotel datasets, and a negative performance for computerbr which contains only tweets. since syntactic patterns tend to be language-specific, many of the chosen rules potentially do not represent opinions in portuguese texts. likewise, the whole group of concept-level features only contributed to the hotel dataset. interestingly, this is the smallest dataset used in our experiments. thus, concept-level features may work better in smaller scopes. we also observed that, as expected, the most contributing group in the computerbr dataset was the twitter group. this indicates that extracting specific features of twitter texts is a better strategy than relying on language-specific resources such as sentiment lexicons and language patterns. table 10. results obtained by the feature ablation study.full size table 4.5 feature ranking to answer the research question rq3, we performed the recursive feature elimination (rfe) algorithm to evaluate how different combinations of features affect the predictive performance of a classifier. firstly, all 54 features are used to train the gbt model. the performance of each feature is separately maintained. gbt was chosen because it achieved the best performance as a learning algorithm (see sect. 4.3). the evaluation is performed using 5-fold cross-validation and macro f\(_1\) score as the metric. secondly, the features with the lowest coefficients will be omitted during every epoch and retrained after removing each feature from the input set and continued until the required number of features are retained. as an outcome of rfe, features with the worst coefficients are eliminated individually during every iteration of the training process. figure 1 shows the performance of the gbt model using different sizes of feature subsets. the best result in each dataset is identified with a star. first of all, we can see that using all the features does not produce the best results. therefore, using a technique of feature selection is necessary to remove unnecessary features. interestingly, the best results in the hotel dataset were achieved with just 8 features. in addition, we observed an improvement in the results when using feature selection. for example, using feature selection yielded gains of f\(_1\) of approximately 10.9% for computerbr and 2.6% for the hotel dataset. fig. 1. number of features against the f\(_1\) score plot for each dataset. full size image figure 2 lists the top 10 features for each dataset. individually, the number of adjectives (feat. 1 table 1) is the most important feature in three datasets (computerbr, reli, and tripadvisor). this feature is still the eleventh most important feature in the hotel dataset. concept-level features (table 4) are the most important group of features. concept-level features represent 65% among the top 10 features of the four datasets. fig. 2. ranking of the top 10 features for each dataset. full size image it can be concluded that rfe as a feature selection assists in increasing the classification accuracy by generating a more significant feature-weighted ranking. the number of features plays an important role in text classification. a large number of features does not guarantee the best classification performances and vice-versa. yet, the best number of features will generate optimum classification accuracy. 5 conclusion in this paper, we have presented an empirical study of a vast set of text features for identifying subjective sentences for the portuguese language. we examined the performance of these features within supervised learning methods along with four different datasets. firstly, we examined the performance of well-known machine learning algorithms for subjectivity identification. the experimental results showed that, although the classifiers used in our experiments were considerably different, there is minimal difference between the results obtained. this indicates that choosing the correct set of features is the most important factor for identifying subjective sentences. secondly, we conducted a feature ablation study to evaluate the importance of the proposed features in the performance of the classifiers. the experimental results showed that pos features always contribute to all datasets. this indicates that pos tags are the most relevant feature group for subjectivity classification. finally, we performed the recursive feature elimination algorithm to evaluate how different combinations of individual features affect the performance of a classifier. the experimental results showed that the number of adjectives is the most important feature in three out of four datasets. it was also possible to conclude that feature selection assists in increasing the classification accuracy. in future works, we will try to evaluate the same set of text features against other nlp tasks, such as identifying comparative sentences and calculating the polarity of subjective sentences. notes 1.https://auto.gluon.ai. 2.the python package version was v3.0.0 with model pt_core_news_lg, except for the comp and sup pos tags, which were extracted with v2.2.0 and model pt_core_news_sm. 3.https://universaldependencies.org/u/pos. 4.https://www.ime.usp.br/~pf/dicios/index.html. 5.https://scikit-learn.org/stable. references agarwal, a., xie, b., vovsha, i., rambow, o., passonneau, r.j.: sentiment analysis of twitter data. in: proceedings of the workshop on language in social media (lsm 2011), pp. 30–38 (2011) google scholar  almatarneh, s., gamallo, p.: linguistic features to identify extreme opinions: an empirical study. in: yin, h., camacho, d., novais, p., tallón-ballesteros, a.j. (eds.) ideal 2018. lncs, vol. 11314, pp. 215–223. springer, cham (2018). https://doi.org/10.1007/978-3-030-03493-1_23 chapter  google scholar  almatarneh, s., gamallo, p.: comparing supervised machine learning strategies and linguistic features to search for very negative opinions. information 10(1), 16 (2019) article  google scholar  balage filho, p., pardo, t.a.s., aluísio, s.: an evaluation of the brazilian portuguese liwc dictionary for sentiment analysis. in: proceedings of the 9th brazilian symposium in information and human language technology (2013) google scholar  belisário, l.b., ferreira, l.g., pardo, t.a.s.: evaluating richer features and varied machine learning models for subjectivity classification of book review sentences in portuguese. information 11(9), 437 (2020) article  google scholar  cambria, e., li, y., xing, f.z., poria, s., kwok, k.: senticnet 6: ensemble application of symbolic and subsymbolic ai for sentiment analysis. in: proceedings of the 29th acm international conference on information & knowledge management, pp. 105–114 (2020) google scholar  carvalho, c.m.a., nagano, h., barros, a.k.: a comparative study for sentiment analysis on election brazilian news. in: proceedings of the 11th brazilian symposium in information and human language technology, pp. 103–111 (2017) google scholar  chai, y., lei, c., yin, c.: study on the influencing factors of online learning effect based on decision tree and recursive feature elimination. in: proceedings of the 10th international conference on e-education, e-business, e-management and e-learning, pp. 52–57 (2019) google scholar  chaturvedi, i., cambria, e., welsch, r.e., herrera, f.: distinguishing between facts and opinions for sentiment analysis: survey and challenges. inf. fusion 44, 65–77 (2018) article  google scholar  chenlo, j.m., losada, d.e.: a machine learning approach for subjectivity classification based on positional and discourse features. in: lupu, m., kanoulas, e., loizides, f. (eds.) irfc 2013. lncs, vol. 8201, pp. 17–28. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-41057-4_3 chapter  google scholar  chenlo, j.m., losada, d.e.: an empirical study of sentence features for subjectivity and polarity classification. inf. sci. 280, 275–288 (2014) article  google scholar  freitas, c., motta, e., milidiú, r., césar, j.: vampiro que brilha... rá! desafios na anotaçao de opiniao em um corpus de resenhas de livros. encontro de linguística de corpus 11, 22 (2012) google scholar  freitas, l.a.d., vieira, r.: ontology-based feature-level sentiment analysis in portuguese reviews. int. j. bus. inf. syst. 32(1), 30–55 (2019) google scholar  ge, p.: analysis on approaches and structures of automated machine learning frameworks. in: 2020 international conference on communications, information system and computer engineering (cisce), pp. 474–477. ieee (2020) google scholar  hogenboom, a., bal, d., frasincar, f., bal, m., de jong, f., kaymak, u.: exploiting emoticons in polarity classification of text. j. web eng. 14(1 & 2), 22–40 (2015) google scholar  honnibal, m., montani, i.: spacy 2: natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. to appear 7(1), 411–420 (2017) google scholar  kimura, m., katsurai, m.: investigating the consistency of emoji sentiment lexicons constructed using different languages. in: proceedings of the 20th international conference on information integration and web-based applications & services, pp. 310–313 (2018) google scholar  mansour, r., hady, m.f.a., hosam, e., amr, h., ashour, a.: feature selection for twitter sentiment analysis: an experimental study. in: gelbukh, a. (ed.) cicling 2015. lncs, vol. 9042, pp. 92–103. springer, cham (2015). https://doi.org/10.1007/978-3-319-18117-2_7 chapter  google scholar  moraes, s.m.w., santos, a.l.l., redecker, m., machado, r.m., meneguzzi, f.r.: comparing approaches to subjectivity classification: a study on portuguese tweets. in: silva, j., ribeiro, r., quaresma, p., adami, a., branco, a. (eds.) propor 2016. lncs (lnai), vol. 9727, pp. 86–94. springer, cham (2016). https://doi.org/10.1007/978-3-319-41552-9_8 chapter  google scholar  oliveira, m., melo, t.: investigando features de sentenças para classificação de subjetividade e polaridade em português do brasil. in: anais do xvii encontro nacional de inteligência artificial e computacional, pp. 270–281. sbc (2020) google scholar  palshikar, g., apte, m., pandita, d., singh, v.: learning to identify subjective sentences. in: proceedings of the 13th international conference on natural language processing, pp. 239–248 (2016) google scholar  pennebaker, j.w., francis, m.e., booth, r.j.: linguistic inquiry and word count: liwc 2001. mahway: lawrence erlbaum assoc. 71(2001), 2001 (2001) google scholar  pereira, d.a.: a survey of sentiment analysis in the portuguese language. artif. intell. rev. 54(2), 1087–1115 (2021) article  google scholar  poria, s., cambria, e., winterstein, g., huang, g.b.: sentic patterns: dependency-based rules for concept-level sentiment analysis. knowl.-based syst. 69, 45–63 (2014) article  google scholar  turney, p.: thumbs up or thumbs down? semantic orientation applied to unsupervised classification of reviews. in: proceedings of the 40th annual meeting of the association for computational linguistics, pp. 417–424 (2002) google scholar  download references acknowledgments the authors are thankful for the financial and material support provided by the amazonas state research support foundation (fapeam) through project ppp 04/2017 and for the assistance of the intelligent systems laboratory (lsi) of the amazonas state university (uea). we also gratefully acknowledge the support provided by the gratificação de produtividade acadêmica (gpa) of the amazonas state university (portaria 086/2021). author information authors and affiliations intelligent systems laboratory, superior school of technology, amazonas state university, manaus, am, brazil miguel de oliveira & tiago de melo authors miguel de oliveiraview author publications search author on:pubmed google scholar tiago de meloview author publications search author on:pubmed google scholar corresponding author correspondence to miguel de oliveira . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de oliveira, m., de melo, t. (2021). an empirical study of text features for identifying subjective sentences in portuguese. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_26 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords sentiment analysis natural language processing data mining publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a deep learning approach for aspect sentiment triplet extraction in portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a deep learning approach for aspect sentiment triplet extraction in portuguese conference paper first online: 28 november 2021 pp 343–358 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) a deep learning approach for aspect sentiment triplet extraction in portuguese download book pdf download book epub josé meléndez barros  orcid: orcid.org/0000-0003-3035-116210 & glauber de bona  orcid: orcid.org/0000-0002-6795-988710  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1284 accesses 1 citation abstract aspect sentiment triplet extraction (aste) is an aspect-based sentiment analysis subtask (absa). it aims to extract aspect-opinion pairs from a sentence and identify the sentiment polarity associated with them. for instance, given the sentence “large rooms and great breakfast”, aste outputs the triplet t = {(rooms, large, positive), (breakfast, great, positive)}. although several approaches to asba have recently been proposed, those for portuguese have been mostly limited to extracting only aspects without addressing aste tasks. this work aims to develop a framework based on deep learning to perform the aspect sentiment triplet extraction task in portuguese. the framework uses bert as a context-awareness sentence encoder, multiple parallel non-linear layers to get aspect and opinion representations, and a graph attention layer along with a biaffine scorer to determine the sentiment dependency between each aspect-opinion pair. the comparison results show that our proposed framework significantly outperforms the baselines in portuguese and is competitive with its counterparts in english. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a pairing enhancement approach for aspect sentiment triplet extraction chapter © 2023 dess: deberta enhanced syntactic-semantic aspect sentiment triplet extraction chapter © 2026 unifying aspect-based sentiment analysis bert and multi-layered graph convolutional networks for comprehensive sentiment dissection article open access 25 june 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. applied linguistics computational linguistics corpus linguistics machine translation natural language processing (nlp) portuguese 1 introduction sentiment analysis, also known as opinion mining, is a field of natural language processing (nlp) that analyzes people’s opinions, sentiments, and attitudes towards entities such as products, services, organizations, individuals, among others [21]. there are three granularity levels in sentiment analysis: document level, sentence level and aspect level. aspect-based sentiment analysis (absa) is the term used to denote a set of tasks that aim to resolve aspect level granularity problems [21, 25], i.e., given a review, the model should select specific words corresponding to aspect terms and opinion terms and identify their sentiment polarity. in turn, absa subtasks may focus only on extracting/classifying one or more of these elements. in this work, we deal with aspect sentiment triplet extraction (aste), which is a relatively new subtask [24]. consider for instance the sentence “large rooms and great breakfast”. aspects are word sequences describing attributes or features of the targets (e.g., rooms, breakfast), opinions are those expressions carrying subjective attitudes or (un)desirable characteristics (e.g., large, great) and sentiments are the polarities associated with aspect-opinion pair, which can be positive, negative or neutral. aste aims to extract not only the aspects and its sentiment, but also the corresponding opinion spans expressing the sentiment for each aspect, as the example in fig. 1 illustrates. fig. 1. differences among absa subtasks. full size image absa tasks tackled in portuguese include aspect term extraction [2, 23], aspect sentiment classification [1, 26] and aspect opinion co-extraction [4], but the proposed solutions employs methods that are considerably behind the state of the art in reference languages [25]. besides, these works are focused on identifying aspects or opinions separately. our work aims to extract aspects, opinions and sentiment polarity simultaneously. this not only provides richer context output, but also explains why a specific sentiment polarity was assigned. we develop a framework based on deep learning to perform the aspect sentiment triplet extraction task in portuguese. our framework is based the state-of-the-art models for english, building on the work of zhang et al. [32]. the main improvements proposed by our work are: we use bert instead of glove for sentence encoding, with word-vectors capturing a richer context; we add a graph attention layer to capture more complex relationships between two word vectors. the comparison results show that our proposed framework outperforms the baselines by more than 10%. besides, the ablation study demonstrates the effectiveness of graph attention layers and bilstm layers for dimensionality reduction. the main contributions of this paper can be summarized as follows: we propose the first model to deal with aste tasks in portuguese. to the best of our knowledge, all works in portuguese-absa have been limited to aspect extraction. we provide a new dataset in portuguese to work with aste tasks. additionally, we adapt an existing dataset for this task (reli corpus [12]). other datasets [11, 13, 28] related to absa in portuguese do not have the proper labeling for aste tasks. 2 related work absa includes several subtasks that vary according to the domain of the target term. xu et al. [31] developed a model based on bilstm networks, attention and conditional random fields (crf), aiming to extract an opinion term given an aspect term as input. their main contribution is the model’s capability of capturing variable-length opinion spans. this model cannot manage overlapped aspects/opinions. he et al. [17] work on aspect-sentiment pair extraction task. they created a model using cnn layers and an attention mechanism. it incorporates two document-level classification tasks to be jointly trained with aspect extraction and aspect-level sentiment classification, allowing aspect-level tasks to benefit from document-level information. this model requires outputs intensively annotated to validate the results. aspect/target term sentiment classification is another subtask commonly addressed in absa, approached for instance by cui and maojie [5] and han et al. [16]. both works employed bigru/lstm and attention mechanisms to attenuate irrelevant information, but neither of them can manage overlapped sentiments. meanwhile, fan et al. [10] focused their work on target-oriented opinion words-extraction (towe), which aims to extract aspect-opinion tuples given an aspect(s) as input. to achieve that, they proposed an inward-outward lstm to get information from the left and the right contexts of the target. this work can handle overlapped aspect/opinion. aspect sentiment triplet extraction was recently proposed by peng et al. [24], and they put forward a two-stage framework to extract opinion triplets. in the first stage, they initially used a neural network based on bilstm and graph convolutional networks to extract aspects-sentiments pairs and opinion terms separately. then, to detect the relationship between aspect terms and opinion terms in the second stage, a lstm network and pretrained word embeddings are employed. this pipeline approach might suffer from error propagation. to handle error propagation problems in multi-stage frameworks, wu et al. [30] proposed a tagging scheme model, which is implemented in three variants, using cnn, bilstm or bert. advantages of this approach include the following: only raw reviews are required as input; this model can extract all opinion factors of opinion pair extraction in one step, instead of pipelines; and it is easily extended to other pair/triplet extraction tasks from text. the most important limitation is again that outputs must be intensively annotated for the results to be validated. zhang et al. [32] developed a framework that can handle overlapping sentences, only requires raw review as input, and that is not multi-stage like the above one. the model structure consists of a word embedding (glove) attached to a bilstm layer, which they use as a context sentence encoder. the bilstm output h is passed to relu layers, which apply dimension-reduction to strip away irrelevant features. finally, two independent softmax layers obtain distributions over the labels that denote an aspect-opinion term. a biaffine scorer [9] to determine if the sentiment dependency between each word pair is neutral, negative, positive, or null. unlike the works developed for english, in portuguese, no available models were found in aste. the most remarkable works are focused on aspect extraction and aspect-sentiment pair extraction. cardoso and pereira [4] used crf and oplexicon to deal with aspect sentiment classification. aires et al. [1] developed two models, one based on svm and one employing an lstm network. saias et al. [26] opted for three methods: maximum entropy classifier, sentiment lexicon sentilex-pt and rule-based methods, with the last two focusing on aspect-sentiment pair extraction. finally, balage f. [2] used clustering techniques, word2vec embeddings and crf to implement an unsupervised model. 3 proposed framework our frameworkfootnote 1 (bert for opinion triplet extraccion bote) consists of 4 modules. the overall architecture is shown in fig. 2. the sentence encoder module generates a set of word vectors, which encode semantic and context information. the aspect-opinion representation module extracts the aspect and opinion features from word vectors. then, the aspect-opinion tagging module takes as input this feature vector to label the word as an aspect or an opinion or neither of them. finally, the sentiment dependency between aspect and opinion vectors is detected by the dependency parsing module. fig. 2. architectures of our model. full size image 3.1 problem definition our aspect sentiment triplet extraction (aste) [24] problem is defined as, given a sentence \(s=\{w_{1},w_{2},w_{3},...,w_{n}\}\) consisting of n words, extracting all possible triplets \(t=\{(a,o,p)_{m}\}_{m=1}^{|t|}\) from s, where a, o and p respectively denote an n-gram aspect term, an n-gram opinion term and a sentiment polarity; \(a_{m}\) and \(o_{m}\) can be represented as their start and end positions (\(s_{m}\), \(e_{m}\)) in s and \(p_m \in \{positive, negative, neutral\}\). note that this task involves dealing with one-to-one, one-to-many and many-to-many relationships between aspects and opinions, as well as overlapped sentences. 3.2 sentence encoding we adopt bert [7] to encode sentences. given a sentence \(s=\{w_{i}\}_{i=1}^{|s|}\) we get a word-vectors set \(v=\{v_{i}|v_i \in \mathbb {r}^{d_{b}} \}_{i=1}^{|s|}\) from the \(n^{th}\) bert hidden state, where \(d_b\) denote the dimensionality of the bert word-vector. the capability of bert to generate different word embeddings depending on the context allows us to obtain feature vectors with better semantic information; for example, bert vectors can deal with homonyms. additionally, bert can handle long reviews better than lstm networks, thanks to attention mechanisms. the bert model works with wordpiece [7], therefore, from the sentence s, we obtain token vectors \(t_i\) instead of word vectors \(v_i\). a word can be formed by two or more tokens. to obtain a vector \(v_i\) of a word \(w_i\), we average the token vectors \((u_{i1},u_{i2},...,u_{il})\) that form that word; formally, \(v_i = \frac{1}{m}\sum _{l=1}^{m} u_{i,l} \) for all \(v_i \in v\). a custom average pooling2d layer calculates the averages. finally, we apply a bilstm dimension-reducing layer to the word vectors. since a bert vector can have 758 or 1024 dimensions and machine learning does not understand causality, models map any input feature to the target variable, even if there is no causal relation. many features demand a more complex model and increase the risk of overfitting. dimensionality reduction removes multi-collinearity and irrelevant features, making our model more straightforward, less data-hungry, and reducing the risk of overfitting. 3.3 aspect and opinion representation following zhang et al. [32], we apply linear layers and nonlinear functions on v before the opinion and aspect tagging operation in order to attenuate the noise of irrelevant information. formally, let \( \mathbf {r}_i^{(ap)} \) and \( \mathbf {r}_i^{(op)}\) (\( \mathbf {r}_i^{(ap)\prime } \) and \( \mathbf {r}_i^{(op)\prime } \)) denote the aspect and opinion representations employed in aspect and opinion tagging (sentiment dependency parsing), with \(\mathbf {r}_i^{(ap)}, \mathbf {r}_i^{(op)}, \mathbf {r}_i^{(ap)\prime }, \mathbf {r}_i^{(op)\prime } \in \mathbb {r}^{d_r} \), where \(d_r\) is the dimensionality of the representation. the linear layers employ a relu family function g(.) and have parameters \( \mathbf {w}_r^{(ap)}, \mathbf {w}_r^{(op)}, \mathbf {w}_r^{(ap)\prime }, \mathbf {w}_r^{(op)\prime } \in \mathbb {r}^{d_r \times d_b} \) and \( \mathbf {b}_r^{(ap)}, \mathbf {b}_r^{(op)}, \mathbf {b}_r^{(ap)\prime }, \mathbf {b}_r^{(op)\prime } \in \mathbb {r}^{d_r}\): $$\begin{aligned} \mathbf {r}_i^{(ap)} = g(\mathbf {w}_r^{(ap)}v_i + \mathbf {b}_r^{(ap)}) \end{aligned}$$ (1) $$\begin{aligned} \mathbf {r}_i^{(op)} = g(\mathbf {w}_r^{(op)}v_i + \mathbf {b}_r^{(op)}) \end{aligned}$$ (2) $$\begin{aligned} \mathbf {r}_i^{(ap)\prime } = g(\mathbf {w}_r^{(ap)\prime }v_i + \mathbf {b}_r^{(ap)\prime }) \end{aligned}$$ (3) $$\begin{aligned} \mathbf {r}_i^{(op)\prime } = g(\mathbf {w}_r^{(op)\prime }v_i + \mathbf {b}_r^{(op)\prime }) \end{aligned}$$ (4) 3.4 aspect and opinion tagging words are tagged by using two taggers built on a softmax layer as follows: $$\begin{aligned} \mathbf {p}_i^{(l)} = softmax(\mathbf {w}_t^{(l)}\mathbf {r}_i^{(l)} + \mathbf {b}_t^{(l)}) \end{aligned}$$ (5) above, \( l \in \{ap, op\}\), \(\mathbf {w}_t^{(l)} \in \mathbb {r}^{3 \times d_r}\) and \( \mathbf {b}_t^{(l)} \in \mathbb {r}^3 \) are trainable parameters. the softmax outputs two series of distributions over {b, o, i}, tagging aspect and opinion terms via the probability of each word \(w_i\) being the begining/inside/outside of an aspect/opinion term. for example, in the sentence “large rooms and great breakfast. room service was awesome.”, after decoding \( \mathbf {p}_i^{(ap)}\) and \( \mathbf {p}_i^{(op)}\) outputs, the result should be as in table 1. table 1. tagging result.full size table 3.5 sentiment dependency parsing in this module, the aim is to identify the sentiment polarity of every word pair \((w_{i}, w_{j})\). our model uses four tags (neu, neg, pos, no-dep) to denote these dependencies. the tags respectively indicate neutral, negative, positive, and non-existent sentiment dependency. there are \(|s|^2\) possible word pairs in each sentence since, during the training process, the target triplets induce ordered pairs. to avoid redundant relations, following [3, 32], sentiment dependency between an aspect and opinion term is assigned via the pair formed by their last word. formally, we have a 3d-tensor \( \mathbf {t} \in \mathbb {r}^{|s| \times |s| \times 4}\), where each element \(t_{ijk}\) denotes the probability that the polarity in \((w_{i}, w_{j})\) is k, with \(k \in \) {neu, neg, pos, no-dep}. finally, we get an asymmetric square matrix \( \mathbf {d} \in \mathbb {r}^{|s| \times |s|}\) as shown in fig. 3. fig. 3. a parsing example for sentiment dependency. full size image a biaffine scorer [9] is used to obtain word-level sentiment dependencies. this mechanism is widely used and has shown success in various parsing-related tasks [19, 20]. the score \( s_{ijk} \) for a pair \((w_{i}, w_{j})\) that have a dependency k is computed using syntax-aware vectors \( \mathbf {h}_i^{(ap)\prime } \) and \( \mathbf {h}_j^{(op)\prime } \) obtained from a graph attention layer (gal). using ; to represent the concatenation operation and \(\mathbf {w}^{(k)}\) and \( \mathbf {b}^{(k)} \) to denote trainable weight and bias, the scores can be defined as: $$\begin{aligned} s_{ijk} = [\mathbf {w}^{(k)} \mathbf {z}_i^{(ap)\prime } + \mathbf {b}^{(k)}]^\top \mathbf {z}_j^{(op)\prime } \end{aligned}$$ (6) $$\begin{aligned} \mathbf {z}_i^{(ap)\prime } = [\mathbf {r}_i^{(ap)\prime }; \mathbf {h}_i^{(ap)\prime }], \quad \mathbf {z_i} \in \mathbb {r}^{2d_r} \end{aligned}$$ (7) $$\begin{aligned} \mathbf {z}_j^{(op)\prime } = [\mathbf {r}_j^{(op)\prime }; \mathbf {h}_j^{(op)\prime }], \quad \mathbf {z_j} \in \mathbb {r}^{2d_r} \end{aligned}$$ (8) an example of how to calculate \( s_{ijk} \) is shown in fig. 4. the aspect-vector \(\mathbf {z}_i\) representing the word \(w_i\) is inputted into the linear layer \( \mathbf {w}^{(k)} (.) + \mathbf {b}^{(k)} \) (eq. 6). the purpose of this linear layer is to generate a vector \(\mathbf {z}_i\) for each k-polarity. therefore, by multiplying \(\mathbf {z}_{ik}\) and \(\mathbf {z}_j\), we obtain a score that measures the polarity between \(w_i\) and \(w_j\). in our example, the score vector of \( (\mathbf {w}_{1}, \mathbf {w}_0)\) = (“rooms”, “large”) is \( s_{10} = \langle s_{100}, s_{101}, s_{102}, s_{103} \rangle = \langle -0.32, -0.39, -0.55, 1.98 \rangle \). once we apply the softmax function, we obtain the probability vector \( s_{13} = \langle 0.08, 0.07, 0.06, 0.79 \rangle \), indicating that the dependency between “quartos” and “rooms” is positive. although bert contains implicit syntactic information, its ability to capture explicit syntactic features is limited [15]. a common problem in absa when feature vectors are generated from a sequence, without considering explicitly the knowledge about the language, is that they may incorrectly locate specific targets in sentences with multiple aspects and opinions [22]. fig. 4. a biaffine scoring example for sentiment dependency. full size image 3.6 syntax-aware vector unlike zhang et al., we concatenate a syntax-aware vector \( \vec {\mathbf {h}} \) with \( \vec {\mathbf {r}} \) before inputting the features vector into the biaffine scorer. as mentioned earlier, this vector \( \vec {\mathbf {h}} \) is obtained from a graph attention layer (gal), which can to capture linguistic structures like dependency trees other than only sequential data [18]. the gal receives the vector \( \vec {\mathbf {r}} \) and an adjacency matrix that represents the syntactic dependency graph of the sentencefootnote 2. gal then embed the graph structure into the vector by performing masked attention – it computes self-attention between node i and node j if and only if j is a neighbor of i. figure 5 shows the syntactic dependency of “the hotel has an awesome room service” and its adjacency matrix. in \( \vec {\mathbf {r}}_6\), for example, self-attention is applied between \( \vec {\mathbf {r}}_6\) and its neighbors (green region) instead of all tokens. in a dependency tree, the aspect-opinion pair will generally be related. by applying the mechanism of attention following the connections of the tree, more important nodes receive higher weight during neighborhood aggregation. therefore, there will be more chances of identifying a sentiment dependency relationship in the selected pair. although there are some useless dependency relationships for aste, aspect and opinion tagging outputs serve to counteract the noise produced by these irrelevant dependencies. we take the syntactic dependency as an undirected graph. in directed graphs, self-attention is computed following the direction of the edge, however, edges do not always connect aspects and opinions directly, and if it does, the direction is not necessarily aspect \(\rightarrow \) opinion. if we use a directed graph, we could lose information. fig. 5. a biaffine scoring example for sentiment dependency. full size image table 2. information regarding the composition of the rehol dataset.full size table 4 experiments 4.1 datasets we conduct experiments in portuguese and english datasets. portuguese datasets include reli corpus [12], which is composed of 1600 reviews from 14 different books with about 200 comments each. besides, a new dataset was produced by us from hotel reviews collected from tripadvisor (table 2). both datasets were manually annotated. these datasets are referenced in this work as reli and rehol and they are available at https://dx.doi.org/10.21227/0ej1-br13. since there are no portuguese models for aste tasks, english baselines were adapted to portuguese-aste to compare with our model. additionally, our model was trained in english to compare with the actual performance of these baselines. we used four english datasetsfootnote 3 in the laptop and restaurant domain from semeval 2014, 2015 and 2016 (task 4, 12 and 5 respectively). 4.2 baselines existing portuguese-absa solutions do not provide enough information to reproduce their models/experiments. the lack of information includes: the source code is not available, there is insufficient information on the hyperparameters used, or the datasets are not available. the baselines were adapted to portuguese by replacing the english pre-trained word embedding with a portuguese pre-trained word embeddingfootnote 4. no further changes were made. ote-mtl [32] is a model that encodes the sentences using glove and bilstm networks, then, it applies dimension-reducing linear layers and non-linear functions on the hidden states to identify aspects and opinion and finally, the biaffine scorer [9] is used to get aspect-opinion pairs along with the associated sentiment with each other. cmla-mtl [29] is an aspect-opinion co-extraction system, which is based on multi-layer attentions. it was adapted by zhang et al. for aste task. 4.3 experiment settings for the bert encoder, we use the pre-trained cased bert developed by neuralmind [27]. to compare our model with the baselines in english, we use uncased bert base model developed by google researchfootnote 5. the maximum sequence length is 512 with a batch size of 32. we train our model for a maximal of 60 epochs using adam optimizer. the learning rate is set to 1e−5. bilstm dimension reducing vector is 300-dimensional. aspect and opinion representation vectors and syntax-aware vectors are 150-dimensional with a single hidden layer. dropout is applied to avoid overfitting and the drop rate is 0.3. the development set is used for early stopping. 4.4 evaluation following the previous works [24, 30, 32], f1-score, recall, and precision measurements were used to evaluate the results and compare our model to the baselines. a triplet is correct if and only if the aspect span, the opinion span, the parsing and the sentiment polarity are all correct. the \(5\times 2\)-nested cross-validation technique [8] was used to generate sufficient data from the selected metrics. lilliefors, anderson – darling, and shapiro – wilk tests were used to validate the assumption of normality, and the mauchly’s test for the assumption of sphericity [6]. because cross-validation process generates repeated measures, appropriate tests should be used to deal with within-subjects. based on the evaluation of the assumptions, non-parametric tests were selected. comparing our model with the baselines, we used friedman’s test to identify if there was a statistically significant difference between the models [6]. finally, we applied hommel post-hoc procedure to identify if our model was better than the baselines [14]. the significance level used in all tests was 0.05. 5 results and analysis 5.1 comparison with baselines the results are shown in table 3. our framework consistently achieves the best scores, both for the aspect and opinion extraction task and triplet extraction task in portuguese. the average difference between our model and the portuguese baselines in aspect extraction is 6.68%, opinion extraction is 7.87% and triplet extraction is 12.58%. in aspect extraction in english, the results are not conclusive. however, in opinion and triplet extraction in english, the framework outperforms the baselines significantly in three of the four datasets. in rest 14, there is no statistically significant difference between bote and ote-mtl. detailed information about precision, recall, and how the errors affect the application can be found at https://dx.doi.org/10.21227/0ej1-br13. table 3. results f1-score (%). * indicates difference is not statistically significant between flagged models. bold indicates statistically significant superior performance.full size table additionally, experiments were carried out to test the models in domains for which they were not trained. the tests were performed in both portuguese and english. the models in portuguese were trained in the books domain and tested in the hotels domain. the english models were trained in the restaurant domain and tested in the laptop domain. the results are displayed in table 4. bote shows significantly better performance than baselines in all three tasks, although lower when compared with bote results in the same domain. this indicates that, comparing with the baselines, our model responds better to new information, relationships and objects that were shown to it during training. table 4. f1-score (%) transfer learning between different domains.full size table finally, several tests were run to evaluate the performance of the proposed model based on the number of triples per sentence (table 5). the results indicate that the greater the number of triplets in a sentence, the higher the hit rate (except between 3-triplets and 4-triplets in rehol). since the greater the number of triplets, the more complex are the relationships between aspects and opinion terms, further experiments are needed to identify the underlying cause of this behavior. table 5. bote f1-score (%) according to number of triplets per sentence.full size table 5.2 ablation study we examine the effectiveness of two components of our bote model, namely the reduction layer and graph attention layer (gal). table 6 presents the ablation results on reli and rehol datasets. bote-noreduction does not use any type of reduction, bote-linear uses a linear layer, bote-relu uses a relu layer and bote-nograph uses a bilstm layer but does not use graph attention. the original bote uses a bilstm layer to reduce the size of the word vectors and graph attention layer to parse aspect and opinion terms. regarding aspect extraction (at) and opinion extraction (ot) tasks, there is no statistically significant difference between linear, nonlinear and no reduction. however, we can observe a significant improvement when the model uses a bilstm layer to reduce the dimensionality of the word vectors. since gal is not involved in at and ot, it is not expected to be any significant difference between bote-nograph and bote. regarding triplet extraction (tr), nonlinear and bilstm reduction show a significant improvement in the sentiment parsing process compared to no reduction. likewise, when we use gal we obtain better results compared to the other variants. table 6. ablation study f1-score (%).full size table 6 conclusions and future work in this work, we proposed a machine learning framework to deal with aspect sentiment triplet extraction in portuguese. our model used a bert model to obtain context-aware word vectors, a graph attention layer to exploit the syntactic information contained in the sentence, and a biaffine scorer as a sentiment parser. the model achieved an average f1-score of 62.07% in the datasets in aste portuguese. when the model was trained in one domain and tested in another, it achieved an f1-score of 37.78% compared to 17.15% of the best baseline performance. besides, the results showed the higher the number of triplets, the more accurate the model. the ablation study proved that performance is better when a dimensionality reducer is applied to bert word vectors. the experimental results verify the effectiveness of our framework compared to the baselines. to the best of our knowledge, we have developed the most fine-grained model to deal with aspect sentiment tasks in portuguese. the proposed model even outperforms its english counterparts in some cases. ablation study validates the effectiveness of applying dimensionality reduction in bert word vectors, especially when using a bilstm layer. besides, syntactic information improves sentiment parsing. finally, our model shows a greater capacity of transfer learning across different domains, responding better to new information. future works include developing an opinion dependency tree where the relationship between words is based not on syntax, but on sentiment, as syntax trees usually ignore many connections between aspects and opinion words. other possible research directions could aim at improving the classification ability of the model in multiple domains or determining the reason why the higher the number of triplets per sentence, the better is the model performance. notes 1.source code available at https://github.com/josemelendezb/bote. 2.we obtain the syntactic dependency graph of a sentence using spacy parser. 3.datasets are available by zhang et al. at https://github.com/genezc/ote-mtl. 4.http://nilc.icmc.usp.br/nilc/index.php/repositorio-de-word-embeddings-do-nilc. 5.https://github.com/google-research/bert. references aires, j.p., padilha, c., quevedo, c., meneguzzi, f.: a deep learning approach to classify aspect-level sentiment using small datasets. in: 2018 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2018) google scholar  balage filho, p.p.: aspect extraction in sentiment analysis for portuguese language. ph.d. thesis, universidade de sao paulo (2017) google scholar  bekoulis, g., deleu, j., demeester, t., develder, c.: joint entity recognition and relation extraction as a multi-head selection problem. expert syst. appl. 114, 34–45 (2018) article  google scholar  cardoso, b., pereira, d.: evaluating an aspect extraction method for opinion mining in the portuguese language. in: anais do viii symposium on knowledge discovery, mining and learning, pp. 137–144. sbc (2020) google scholar  cui, z., maojie, z.: aspect level sentiment classification based on double attention mechanism. in: proceedings of the 2019 2nd international conference on e-business, information management and computer science. ebimcs 2019. association for computing machinery, new york (2019). https://doi.org/10.1145/3377817.3377834 demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7(1), 1–30 (2006). http://jmlr.org/papers/v7/demsar06a.html devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics, minneapolis, minnesota, pp. 4171–4186. association for computational linguistics, june 2019. https://doi.org/10.18653/v1/n19-1423 dietterich, t.g.: approximate statistical tests for comparing supervised classification learning algorithms. neural comput. 10(7), 1895–1923 (1998) article  google scholar  dozat, t., manning, c.d.: deep biaffine attention for neural dependency parsing. in: 5th international conference on learning representations, iclr 2017, toulon, france, 24–26 april 2017, conference track proceedings. openreview.net (2017). https://openreview.net/forum?id=hk95pk9le fan, z., wu, z., dai, x.y., huang, s., chen, j.: target-oriented opinion words extraction with target-fused neural sequence labeling. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics, minneapolis, minnesota, pp. 2509–2518. association for computational linguistics, june 2019. https://doi.org/10.18653/v1/n19-1259 farias, d.s., matsuno, i.p., marcacini, r.m., rezende, s.o.: opinion-meter: a framework for aspect-based sentiment analysis. in: proceedings of the 22nd brazilian symposium on multimedia and the web, pp. 351–354. acm (2016). https://doi.org/10.1145/2976796.2988214 freitas, c., motta, e., milidiú, r., césar, j.: vampiro que brilha... rá! desafios na anotaçao de opiniao em um corpus de resenhas de livros. encontro de linguística de corpus 11, 22 (2012) google scholar  freitas, l.a., vieira, r.: exploring resources for sentiment analysis in portuguese language. in: brazilian conference on intelligent systems, pp. 152–156. ieee (2015) google scholar  garcía, s., herrera, f.: an extension on “statistical comparisons of classifiers over multiple data sets” for all pairwise comparisons. j. mach. learn. res. 9(89), 2677–2694 (2008). http://jmlr.org/papers/v9/garcia08a.html goldberg, y.: assessing bert’s syntactic abilities. arxiv (2019). http://arxiv.org/abs/1901.05287 han, h., li, x., zhi, s., wang, h.: multi-attention network for aspect sentiment analysis. in: proceedings of the 2019 8th international conference on software and computer applications, icsca 2019, pp. 22–26. association for computing machinery, new york (2019). https://doi.org/10.1145/3316615.3316673 he, r., lee, w.s., ng, h.t., dahlmeier, d.: an interactive multi-task learning network for end-to-end aspect-based sentiment analysis. in: proceedings of the 57th annual meeting of the association for computational linguistics, florence, italy, pp. 504–515. association for computational linguistics, july 2019 google scholar  huang, b., carley, k.: syntax-aware aspect level sentiment classification with graph attention networks. in: proceedings of the 2019 conference on empirical methods in natural language processing, hong kong, china, pp. 5469–5477. association for computational linguistics, november 2019. https://doi.org/10.18653/v1/d19-1549 li, y., li, z., zhang, m., wang, r., li, s., si, l.: self-attentive biaffine dependency parsing. in: proceedings of the twenty-eighth international joint conference on artificial intelligence, ijcai, pp. 5067–5073. ijcai.org (2019). https://doi.org/10.24963/ijcai.2019/704 li, z., et al.: dependency or span, end-to-end uniform semantic role labeling. in: proceedings of the aaai conference on artificial intelligence, vol. 33, no. 01, pp. 6730–6737 (2019). https://doi.org/10.1609/aaai.v33i01.33016730 liu, b.: sentiment analysis and opinion mining. synthesis lect. hum. lang. technol. 5(1), 1–167 (2012) article  google scholar  lu, z., du, p., nie, j.-y., et al.: vgcn-bert: augmenting bert with graph embedding for text classification. in: jose, j.m. (ed.) ecir 2020. lncs, vol. 12035, pp. 369–382. springer, cham (2020). https://doi.org/10.1007/978-3-030-45439-5_25 chapter  google scholar  machado, m.t., pardo, t.a.s., ruiz, e.e.s.: analysis of unsupervised aspect term identification methods for portuguese reviews. in: anais do xiv encontro nacional de inteligncia artificial e computacional (eniac), sbc, pp. 239–249 (2017) google scholar  peng, h., xu, l., bing, l., huang, f., lu, w., si, l.: knowing what, how and why: a near complete solution for aspect-based sentiment analysis. in: proceedings of the aaai conference on artificial intelligence, vol. 34, no. 05, pp. 8600–8607 (2020). https://doi.org/10.1609/aaai.v34i05.6383 pereira, d.a.: a survey of sentiment analysis in the portuguese language. artif. intell. rev. 54(2), 1087–1115 (2020). https://doi.org/10.1007/s10462-020-09870-1 article  google scholar  saias, j., mourão, m., oliveira, e.: detailing sentiment analysis to consider entity aspects: an approach for portuguese short texts. trans. mach. learn. artif. intell. 6, 26–35 (2018) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  vargas, f.a., pardo, t.a.s.: hierarchical clustering of aspects for opinion mining: a corpus study. in: finatto, m.j.b., rebechi, r.r., sarmento, s., bocorny, a.e.p. (eds.) linguística de corpus: perspectivas, pp. 69–91 (2018) google scholar  wang, w., pan, s.j., dahlmeier, d., xiao, x.: coupled multi-layer attentions for co-extraction of aspect and opinion terms. in: proceedings of the aaai conference on artificial intelligence, vol. 31 (2017) google scholar  wu, z., ying, c., zhao, f., fan, z., dai, x., xia, r.: grid tagging scheme for aspect-oriented fine-grained opinion extraction. in: findings of the association for computational linguistics, pp. 2576–2585. association for computational linguistics, november 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.234 xu, l., bing, l., lu, w., huang, f.: aspect sentiment classification with aspect-specific opinion spans. in: proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pp. 3561–3567. association for computational linguistics, november 2020. https://doi.org/10.18653/v1/2020.emnlp-main.288 zhang, c., li, q., song, d., wang, b.: a multi-task learning framework for opinion triplet extraction. in: findings of the association for computational linguistics, pp. 819–828. association for computational linguistics, november 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.72 download references acknowledgments this work was supported by capes and ministerio de ciencia from colombia. author information authors and affiliations universidade de são paulo, são paulo, sp, 05508-010, brazil josé meléndez barros & glauber de bona authors josé meléndez barrosview author publications search author on:pubmed google scholar glauber de bonaview author publications search author on:pubmed google scholar corresponding author correspondence to josé meléndez barros . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper barros, j.m., de bona, g. (2021). a deep learning approach for aspect sentiment triplet extraction in portuguese. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_24 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning natural language processing aspect sentiment triplet extraction publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature assessing adversarial effects of noise in missing data imputation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper assessing adversarial effects of noise in missing data imputation conference paper first online: 30 january 2025 pp 200–214 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) assessing adversarial effects of noise in missing data imputation download book pdf download book epub arthur dantas mangussi  orcid: orcid.org/0000-0003-2086-532x9,10, ricardo cardoso pereira  orcid: orcid.org/0000-0003-1735-077111,12, pedro henriques abreu  orcid: orcid.org/0000-0002-9278-819412 & … ana carolina lorena  orcid: orcid.org/0000-0002-6140-571x9,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 497 accesses 1 citation abstract in real-world scenarios, a wide variety of datasets contain inconsistencies. one example of such inconsistency is missing data (md), which refers to the absence of information in one or more variables. missing imputation strategies emerged as a possible solution for addressing this problem, which can replace the missing values based on mean, median, or machine learning (ml) techniques. the performance of such strategies depends on multiple factors. one factor that influences the missing value imputation (mvi) methods is the presence of noisy instances, described as anything that obscures the relationship between the features of an instance and its class, having an adversarial effect. however, the interaction between md and noisy instances has received little attention in the literature. this work fills this gap by investigating missing and noisy data interplay. our experimental setup begins with generating missingness under the missing not at random (mnar) mechanism in a multivariate scenario and performing imputation using seven state-of-the-art mvi methods. our methodology involves applying a noise filter before performing the imputation task and evaluating the quality of the imputation directly. additionally, we measure the classification performance with the new estimates. this approach is applied to both synthetic data and 11 real-world datasets. the effects of noise filtering before imputation are evaluated. the results show that noise preprocessing before the imputation task improves the imputation quality and the classification performance for imputed datasets. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others performance analysis of missing values imputation methods using machine learning techniques chapter © 2019 mi2ami: missing data imputation using mixed deep gaussian mixture models chapter © 2023 on the use of multivariate medians for nearest neighbour imputation chapter © 2023 1 introduction real-world data often presents multiple problems, which can jeopardize the performance of machine learning (ml) classifiers [12]. renggli et al. [16] advocate that data quality issues influence several stages of the ml pipeline. one common type of inconsistency is missing data (md), which can be described as the absence of values in the data observations [17]. the literature categorizes md mechanisms into the following categories [14]: missing completely at random (mcar): missingness occurs randomly without any dependency on specific features within the dataset; missing at random (mar): a dependency between existent features determines the missingness nature; missing not at random (mnar): the missing values depend on observed and/or other unobserved data (i.e., features not available on the dataset). to address the md issue, the literature presents different missing value imputation (mvi) methods, from basic, such as mean, median, and mode, to more sophisticated strategies, including using ml methods to estimate the missing values [15]. santos et al. [17] describe the classical experimental setup for evaluating md imputation algorithms with four main steps. the first concerns data collection, where a complete dataset (i.e., without missing values) is considered. subsequently, the next step is amputation, which introduces artificial missing values, following the characteristics of mcar, mar, or mnar mechanisms. then, the mvi algorithms need to be selected and applied. finally, the last step is evaluation. in general, studies in the md field often evaluate new mvi techniques by measuring the difference between the original (i.e., the ground truth) and imputed data, a process known as direct evaluation. conversely, there is indirect evaluation, which measures the classification performance using the imputed datasets. however, the literature rarely focuses on evaluating mvi methods using both approaches [6, 9]. another data quality issue is the presence of noisy instances on the dataset. according to zhu and wu [26], there are two distinct types of noise: attribute and class noise. the former type of noise affects input features, while the latter affects the labels registered for the observations. both are present in real-world scenarios, emphasizing that aggregate noise identification strategies should be considered for both. nonetheless, the label noise is potentially more harmful than attribute noise [22, 26]. the literature outlines several techniques for identifying potential noise and addressing it to build more reliable ml models from data [3]. our work will focus on noise filters (nfs), which scan the training data for potentially noisy instances. regarding the interplay of md and noise inconsistencies, when initial noisy data is used to extract patterns for missing data imputation, whether through simple statistics or more sophisticated strategies, the harmful effects of noise can propagate to other instances. nonetheless, this interaction has received little attention in the missing data literature, and only zhu et al. [25], fangfang et al. [8], and hulse and khoshgoftaar [23] have investigated the interaction between these two data inconsistencies. none of these works analyze the application of noise pre-processing before the imputation task. thus, this work investigates the interaction between missing and noisy data inconsistencies, considering pre-processing on noisy instances before the imputation task and investigating how noisy data impacts the results of mvi methods. the remainder of this work is organized as follows: sect. 2 presents related work on the md field, and the interplay of missing values and noise data. section 3 describes the methodology and the experimental setup. the results are presented in sect. 4. section 5 outlines the conclusions and future directions of this work. 2 related work this section presents a literature review of imputation techniques, noise in ml, and the interplay of both data inconsistencies. 2.1 imputation techniques there are several strategies for handling md. replacing the missing values with a predetermined estimate (i.e., imputing the missing values) is a common approach in the literature. the simplest way to perform the imputation task is through a single imputation, where missing values in quantitative features can be replaced with the mean or median of all available non-missing values. in contrast, missing values in qualitative attributes are replaced with the mode [9]. to solve the limitations of single imputation, the literature provides multiple imputation strategies, which employ approximate values that reflect the uncertainty around the actual value from the observed data [2]. the multivariate imputation by chained equation (mice) is the most widely used multiple imputation technique [1]. the mice algorithm is a gibbs sampler that estimates the posterior distribution of a vector of unknown parameters by sampling iteratively from conditional distributions [20]. another way to impute md is using matrix completion methods, such as softimpute. the key idea is to perform imputation by approximating the original data with a low-rank matrix. this process typically involves matrix decomposition to identify latent features that best describe the available values [14]. random forests (rf) are used for mvi in the missforest algorithm [19]. the missforest algorithm is an iterative imputation scheme that trains an rf on observed values in the first step, predicts the missing values and proceeds iteratively. another common ml algorithm used in data imputation is the k-nearest neighbor (knn), which finds the nearest neighbors of instances with missing values and uses them for imputation [6]. the mvi community has recently been using deep learning (dl) methods for mvi [10]. an example is the generative adversarial imputation nets (gain) [24], which has two main components: generator and discriminator. the generator component imputes the missing values on the observed data. it outputs a complete vector, and the discriminator attempts to validate which element in the output vector is imputed [18, 20]. autoencoders (aes) are another example of dl-based methods employed in mvi. aes are a neural network architecture that learns from incomplete data (input layer) and tries reproducing this input at the output layer, generating new plausible values for imputation [13, 14]. with the development of new mvi techniques, it is essential to evaluate their effectiveness. the literature presents an interesting behavior wherein most works only use the direct evaluation for mvi techniques, where the imputed values are compared to real values in datasets where some values are amputed. but it is also important to evaluate how the imputed values influence classification performance, in an indirect evaluation. very few works use both approaches. pereira et al. [14] evaluate the siamese autoencoder-based approach for missing data imputation (saei) for imputation tasks in direct and indirect ways. for the direct evaluation, the authors used the mean absolute error (mae) metric, and for the indirect evaluation, they measured the f1-score performance with three different classifiers: knn, rf, and extreme gradient boosting (xgb). this was done for 14 datasets. the results show that saei outperforms other state-of-the-art mvi methods under mnar assumption and induces the best classification results, improving the f1-scores for 50% of the used datasets. luengo, garcía, and herrera [11] analyze the behavior of 23 classification methods and 14 different mvi approaches in an indirect evaluation. moreover, this methodology was applied to 21 real-world datasets, and all of them have their proper md. they found that using certain mvi techniques could improve the accuracy obtained for the classification methods, facilitating an explanation of how imputation may be a helpful tool to overcome the negative impact of md. 2.2 noise noise is frequent in real-world datasets and can harm the predictive performance of ml classifiers. although most ml algorithms have some internal mechanisms to avoid focusing on noisy instances (e.g. pruning mechanisms in decision trees), cleansing such instances can be beneficial [4]. the literature shows various methods to identify and address attribute and label noise. according to saez et al. [21], noise filter (nf) techniques are widely used in a data pre-processing step for cleansing the training data [3]. their strategy consists of identifying potential noise and removing these unreliable examples. however, these inconsistencies can also be corrected [5]. nfs can be divided into two main categories: similarity-based and ensemble-based filters [21]. the similarity-based or distance-based filters employ the knn algorithm to evaluate whether an example is closest to others within its class; otherwise, it is an unreliable and potentially noisy instance. various knn-based methods have emerged in the literature [3]. a well-known example is the edited nearest neighbor (enn) nf, which eliminates samples whose class differs from most of its k nearest neighbors [5, 21]. our focus in this work will be using the enn algorithm to identify potential noisy instances. these noisy instances will be disregarded during the imputation process to avoid noise from propagating to the imputed data. 2.3 interplay of missing and noisy data a few works have investigated the relationship between noise and md. hulse and khoshgoftaar [23] evaluated the impact of noise in software measurement data on the imputation process. the authors have used five imputation methods using real-world software measurement datasets. the amputation process was made only in the dependent variable covering the mcar, mar, and ni (non-ignorable) mechanisms until they achieved 5%, 10%, 15%, and 20% of missing rates. they used five imputation methods on real-world software measurement datasets. the amputation process was applied only to the dependent variable, covering the mcar, mar, and ni (non-ignorable) mechanisms, until they achieved missing rates of 5%, 10%, 15%, and 20%. they also considered four different noisy scenarios: inherent noise only (i.e., noise present in the original dataset [23]), no noisy instances, and inherent noise with an additional 5% and 10% of injected noise. for each experimental setup and combination of factors, they conducted five independent random selections and compared the imputation accuracy of the five imputation techniques. their experiments demonstrated that data quality plays a crucial role in the effectiveness of imputation techniques. moreover, for the four missing rates used in their experimental setup, an increase in missing rate was not found to be significant for all imputation techniques. robust imputation based on the group method of data handling (ribg) is an mvi method proposed by zhu et al. [25] for predicting missing values in noisy environments. the authors use the group method of data handling (gmdh), a heuristic self-organizing data mining technique known for its noise immunity, to develop the ribg method. ribg operates as follows: given an incomplete dataset, it first performs a preliminary imputation using the mean for numerical features and mode for categorical features to create an initial complete dataset. then, ribg applies the gmdh mechanism to iteratively predict and update these initial missing value estimates. to evaluate the effectiveness of the ribg method, the authors tested it on nine datasets from the uci repository, with missing rates of 5%, 10%, and 20%, under varying noise levels. the missing data was generated under mcar, mar, and mnar assumptions. the results indicate that noise significantly impacts mvi methods, particularly at high noise levels. li et al. [8] present a noise-aware missing data multiple imputation (npmi) algorithm designed to handle missing data in noisy environments. the npmi algorithm uses the random sample consensus (ransac) method to estimate the initial parameters of the multiple imputation algorithm. this approach enhances the robustness of multiple imputations and ensures accuracy even when noise is present. the proposed method was validated on four datasets: two real and two synthetic. for the synthetic data, random gaussian noise was simulated at different noise levels: 5%, 10%, 20%, 30%, 40%, and 50%. for the real data, some values were randomly designated as missing, with missing rates corresponding to the noise level percentages. the accuracy of imputation was evaluated using the root mean square error (rmse). the experimental results demonstrated that noise significantly affects the data quality of the entire dataset, with higher noise levels leading to a greater degradation in data quality. therefore, previous work has shown that the quality of observed data significantly impacts the imputation task and needs to be addressed. as suggested in studies by [8, 25], one option is to use algorithms that are robust to noise for the imputation task. however, those works do not analyze if a simple pre-processing for treating noisy instances beforehand impacts missing value imputation. overall, a comprehensive exploratory analysis of nf use before the imputation task is still needed. thus, this work aims to investigate the interaction of missing and noisy data inconsistencies and to what extent noisy data impacts the results of mvi methods. from the authors’ knowledge, this constitutes the first work that has tried employing an nf before the imputation, investigating the direct and indirect impact of this procedure on imputation quality. 3 methodology fig. 1. methodology overview illustrating the sequential steps followed in the experimental design in the case of synthetic datasets. full size image the methodology for this work consists of evaluating the interplay between noise and missing data in two points: the imputation quality (direct evaluation) and the classification performance for imputed datasets (indirect evaluation). we perform two types of experiments to achieve our goal: using synthetic data and real-world datasets. firstly, we use the synthetic datasets to conduct an initial analysis and have greater control over the experiments. as shown in fig. 1, we generate synthetic data and introduce artificial noise into the attributes or labels. afterward, we insert missing data in an amputation process (i.e., generate artificial missing values). then, with a dataset that contains both data inconsistencies, we use an nf and perform the imputation of the missing values, disregarding the potentially noisy instances. the next section describes how these datasets were generated. a direct evaluation of the imputation quality in these datasets with and without noise filtering is done. once the impact of noise on the imputation results of missing values in synthetic data has been assessed, the experiments are extended to real-world datasets currently employed in the related literature, assuming that real-world data already contains some noise level [8]. therefore, in this case, the artificial insertion of noise at the beginning of fig. 1 is disregarded. here, both direct and indirect evaluations of the imputation results are assessed. we have selected eleven real-world heterogeneous benchmark datasets that are currently employed in the md field and are available on the university of california irvine machine learning repositoryfootnote 1 and kagglefootnote 2. table 1 overviews dataset characteristics. each dataset is identified by its acronym name and information on the number of instances, types of features, and classes (i.e., the number of possible output variable values). categorical features were converted to quantitative values with a one-hot encoding when needed. as seen in table 1, the eleven real-world datasets are all binary classification problems. we selected this type of problem due to the methodology employed in this work. we split each dataset according to the two classes to ensure that both classes have the same amount of missing values and to prevent the noise filter from removing an entire class. section 3.1 will describe this process in more detail. table 1. overview of datasets characteristics.full size table for the implementation, we used python version 3.11 and several additional libraries: pandas, numpy, scikit-learn, mdatagen, and imbalance-learn. all the experiments were conducted on a machine with 60gb ram, gpu nvidia geforce rtx 4090 24gb, and linux, ubuntu version 22.04.4. 3.1 experimental setup as outlined in fig. 1, the first step in our methodology was to create synthetic data. we have used the “make_classification” function from scikit-learn to generate the synthetic data. our base dataset consists of 500 observations with 5 numerical input features and two classes. to introduce attribute noise, gaussian noise is added to each feature. this noise is generated using the mean and standard deviation of each feature, ensuring the values remain within the feature’s minimum and maximum range. label noise is simulated by flipping the labels. the following rates are tested: 5%, 10%, and 20%. we introduce noise to the entire dataset to obtain a more realistic dataset, as real-world data already contains some noise. once noise is inserted, we used a stratified cross-validation strategy with five folds to perform the amputation process (i.e., artificially generating missing values in the dataset) and imputation task. we used the python library mdatagenfootnote 3 for each fold to generate artificial md under mnar mechanism in a multivariate scenario. to ensure that both classes would receive md, we split the training set by the outcome and conducted independent procedures for training and testing sets to keep the same missing rates for both. the multivariate mnar strategy deleted the lowest values for dataset features more related to the classes up to 5%, 10% and 20%. these missing rates were selected from [25]. using such corrupted datasets, we employ the edited nearest neighbor (enn) algorithm to filter the potentially noisy instances in the training sets, with \(k=5\). the enn algorithm cleans the dataset by deleting samples that are close elements from other classes. this tends to remove data in overlapping, borderline and noisy areas of the dataset [7]. in this work, we used a more conservative approach to undersample the majority class, where most of the neighbors must belong to the same class as the examined sample for it to be retained, and the default distance metric in imbalanced-learn package. the identified noisy instances are not used in data imputation afterwards. for the imputation task, we chose seven state-of-the-art mvi to address the generated md. the algorithms chosen are the mean of each feature, knn, mice, pmivae, missforest, softimpute, and gain. the knn, imputation by the mean, missforest, and mice were used directly from the scikit-learn library. the remaining algorithms are available in different github repositoriesfootnote 4. the knn was used with \(k=5\) and the euclidean distance, mice was run with 100 iterations, and the parameterization of the architecture of deep learning methods followed the authors’ recommendations from the original articles. as aforementioned, we saved the imputed test data for each fold in the stratified cross-validation strategy. we measured the imputation quality with the mean absolute error (mae) between the predicted values and the ground truth for the multivariate scenario in the test sets. next, at the end of the cross-validation process, we combined all folds, obtaining an imputed dataset with original data dimensions without bias. we selected an rf classifier to measure the classification performance of the new datasets. the experiments used the same cross-validation strategy for amputation and imputation processes. we tuned the rf hyperparameters by randomized search in the training sets using the f1 score. 4 results this section presents our analysis of the effect of employing an nf based on similarity before the imputation process under mnar mechanism. we begin by investigating the overall impact of enn in synthetic data. subsequently, we analyze the effect of enn filtering on the imputation task for real-world datasets in direct and indirect evaluations. we also discuss if the pattern found in the experiments using synthetic data is verified in real-world datasets. 4.1 impact of nf in imputation table 2 presents differences in mae imputation results for synthetic data under the mnar mechanism. the differences are taken from the baseline, where no noise filtering is applied. therefore, positive values represent better estimates of the missing values after nf, while negative values denote the opposite. the negative values are boldfaced in the table. the datasets named attx contain x% of noise in the attributes, where x is a noise rate, while datasets named labelx have x% of label noise. for each one of them, missing rates are also varied. in all cases, except some specific scenarios with the softimpute method, using an nf has improved the mae results in imputation. although the differences are small, this demonstrates that even by employing a simple noise filter as enn before imputation, the imputation quality can be improved. for the mean, mice, pmivae and missforest imputers, the results are improved more for increased missing rates, indicating that a reliable dataset is especially needed for better imputation results when there are large missing rates. in other cases, there is no clear tendency. there is also no clear tendency on which type of noise or noise ratio affects more the mvi results. regardless of the characteristics of the softimpute method, its primary goal is to find a low-rank matrix that approximates the original dataset [14]. however, the data distribution is altered when we apply an nf that removes specific observations based on the enn criterion. these new estimates may no longer accurately represent the observed data, leading to the deterioration in the mvi results shown in tables 2 and 3. table 2. differences between the average mae for the baseline where no nf is applied and the application of an nf before imputation under mnar multivariate conditions for the synthetic datasets. ‘att’ is an acronym for attribute noise and ‘label’ for label noise. ‘att05’ means a 5% noise level introduced as attribute noise, and the same applies for ‘label 05’.full size table based on the findings in table 2, we extend our analysis to real-world datasets, assuming they already have noisy instances. 4.2 imputation results for real-world datasets table 3 demonstrates the differences in mae results of employing the previous methodology in real-world datasets under the mnar mechanism in a multivariate scenario. again, the differences are taken from the baseline of not employing an nf. negative differences are boldfaced and indicate the estimates of the missing values are worse when nf is applied. again, this happens mostly for the softimpute algorithm. but now gain, missforest, mice and knn were also impaired in some specific cases. in most cases, there are improvements, which tend to be higher for larger missing rates. the softimpute method shows worse mae for 44.12% instances and gain for 41.18%. for the remaining methods, the percentage of worse results is less than 10%. therefore, the applied methodology enhances the imputation quality for most mvi techniques on these real-world datasets under the mnar mechanism. as observed for synthetic data, when the missing rate is increased, the results tend to be improved more when enn is applied as a pre-processing step. table 3. differences between the average mae for the baseline where no nf is applied and the application of an nf before imputation under mnar multivariate conditions for the real-world datasets.full size table table 4 gives a general overview of how each imputation method performs in the datasets by averaging the mae results per imputation method and missing rate. the best results are boldfaced. the missforest algorithm outperforms the remaining methods, followed by knn and mice. the gain method was the worst method in this experimental setup, and the pmivae only surpassed the imputation results by mean. the complexity of those methods may justify the need for larger datasets to obtain better results in the experiments. since the enn algorithm further reduces the datasets in imputation, this might have degraded their performance more. table 4. average mae results obtained for each imputation method, grouped by missing rate. the highlighted bold results present the best mae results for each missing rate.full size table taking the three best imputation methods, missforest, knn, and mice, respectively, we also analyze the average difference between applying or not the methodology of this work, for different missing levels. the results are shown in table 5. the average difference increases for higher missing rates, confirming the results observed for synthetic data. table 5. average differences of mae grouped by missing rate for the three best-performing imputation methods.full size table indirectly evaluating the imputation quality, we trained an rf classifier for the newly imputed data and compared its f1 score with the baseline (i.e., the f1-score for original datasets, without amputed values). figure 2 illustrates the overall average results across all imputed real-world datasets and missing rates, where the average f1-score achieved for the original complete datasets is shown as a dashed red line. mice outperformed the remaining mvi methods and got closer average to the baseline, followed by knn and missforest. those methods were the best three imputation algorithms in our previous evaluation and benefited more from noise filtering. therefore, we can observe an interplay with better quality of imputation and classification performances. in contrast, the pmivae method presents a worse f1-score than the other methods. also, pmivae only outperforms the mean and gain in mae imputation quality. we believe that the nature of enn’s undersampling process, combined with the high dependency on parameterization for deep learning methods such as pmivae and gain, explains the results observed in our study. therefore, we plan to conduct an optimization search to identify the best parameters for these methods in the future. additionally, due to its specific characteristics, the pmivae imputation method may not yield good results with the rf classifier. we intend to extend our methodology to include other classification algorithms to address this. fig. 2. overall f1 score of all imputation methods for random forest classification model. full size image in conclusion, most imputation strategies benefited from noise filtering, estimating new values with higher quality. however, the results are impaired for some specific algorithms. mae is reduced for softimpute, gain, and pmivae. pmivae is especially impaired in the indirect evaluation. these algorithms may require more data to obtain a proper fit of values to be imputed. since noise filtering reduces the dataset, this can explain the observed behavior. 5 conclusions this study investigated the interaction between missing and noisy data, focusing on pre-processing noisy instances before imputation and how this procedure impacts mvi methods. this is the first comprehensive analysis of its kind, addressing a gap in the related literature. we used the edited nearest neighbors (enn) noise filter and seven imputation methods: mean, knn, mice, pmivae, softimpute, gain, and missforest. the filtering takes place before imputation and the hypothesis is that disregarding the unreliable instances from the imputation process can be beneficial, as it prevents noise from propagating to the new estimated values. applying the noise filter has generally improved estimates of the missing values for both synthetic and real-world datasets. the imputation methods that benefited the most from this approach were missforest, knn, and mice. and the results are improved more for datasets with higher missing rates. nonetheless, the reduction in the datasets used for adjusting the models seems to have impaired the results for more complex techniques. however, this must be confirmed for larger datasets with more instances. for future considerations of this work, we want to extend the methodology for the mar multivariate mechanism and cover other classifiers and noise filters more complex than enn. notes 1.https://archive.ics.uci.edu/datasets. 2.https://www.kaggle.com/datasets. 3.https://pypi.org/project/mdatagen/. 4.https://github.com/travisbrady/py-soft-impute, https://github.com/jsyoon0823/gain https://github.com/ricardodcpereira/pmivae. references buuren, s., groothuis-oudshoorn, c.: mice: multivariate imputation by chained equations in r. j. stat. softw. 45 (2011). https://doi.org/10.18637/jss.v045.i03 emmanuel, t., maupong, t., mpoeleng, d., semong, t., mphago, b., tabona, o.: a survey on missing data in machine learning. j. big data 8(1) (2021). https://doi.org/10.1186/s40537-021-00516-9. funding information: this work received a grant from the botswana international university of science and technology. publisher copyright: 2021, the author(s) frenay, b., verleysen, m.: classification in the presence of label noise: a survey. ieee trans. neural netw. learn. syst. 25(5), 845–869 (2014). https://doi.org/10.1109/tnnls.2013.2292894 article  math  google scholar  garcia, l.p., de carvalho, a.c., lorena, a.c.: effect of label noise in the complexity of classification problems. neurocomputing 160, 108–119 (2015) article  math  google scholar  garcia, l.p., lehmann, j., de carvalho, a.c., lorena, a.c.: new label noise injection methods for the evaluation of noise filters. knowl.-based syst. 163, 693–704 (2019). https://doi.org/10.1016/j.knosys.2018.09.031, https://www.sciencedirect.com/science/article/pii/s0950705118304829 hasan, m.k., alam, m.a., roy, s., dutta, a., jawad, m.t., das, s.: missing value imputation affects the performance of machine learning: a review and analysis of the literature (2010–2021). informatics in medicine unlocked 27 (2021). https://doi.org/10.1016/j.imu.2021.100799 lemaître, g., nogueira, f., aridas, c.k.: imbalanced-learn: a python toolbox to tackle the curse of imbalanced datasets in machine learning. j. mach. learn. res. 18(17), 1–5 (2017). http://jmlr.org/papers/v18/16-365 li, f., sun, h., gu, y., yu, g.: a noise-aware multiple imputation algorithm for missing data. mathematics 11, 73 (2022). https://doi.org/10.3390/math11010073 lin, w.c., tsai, c.f.: missing value imputation: a review and analysis of the literature (2006–2017). artif. intell. rev. 53, 1487–1509 (2020). https://doi.org/10.1007/s10462-019-09709-4 liu, m., et al.: handling missing values in healthcare data: a systematic review of deep learning-based imputation techniques. artif. intell. med. 142, 102587 (2023). https://doi.org/10.1016/j.artmed.2023.102587, https://www.sciencedirect.com/science/article/pii/s093336572300101x luengo, j., garcía, s., herrera, f.: on the choice of the best imputation methods for missing values considering three groups of classification methods. knowl. inf. syst. 32, 77–108 (2012). https://doi.org/10.1007/s10115-011-0424-2 article  math  google scholar  nakhaei, a., sepehri, m.m., khatibi, t.: a promising method for correcting class noise in the presence of attribute noise. int. j. hosp. res. 12(1) (2023). https://ijhr.iums.ac.ir/article_171438.html pereira, r.c., abreu, p.h., rodrigues, p.p.: siamese autoencoder-based approach for missing data imputation. in: international conference on computational science, pp. 33–46. springer, cham (2023). https://doi.org/10.1007/978-3-031-35995-8_3 pereira, r.c., abreu, p.h., rodrigues, p.p.: siamese autoencoder architecture for the imputation of data missing not at random. j. comput. sci. 78, 102269 (2024). https://doi.org/10.1016/j.jocs.2024.102269, https://www.sciencedirect.com/science/article/pii/s1877750324000620 pereira, r.c., rodrigues, p.p., figueiredo, m.a.t., abreu, p.h.: automatic delta-adjustment method applied to missing not at random imputation. comput. sci. iccs 2023, 481–493 (2023). https://doi.org/10.1007/978-3-031-35995-8_34 article  math  google scholar  renggli, c., rimanic, l., gürel, n.m., karlaš, b., wu, w., zhang, c.: a data quality-driven view of mlops. arxiv preprint arxiv:2102.07750 (2021) santos, m.s., pereira, r.c., costa, a.f., soares, j.p., santos, j., abreu, p.h.: generating synthetic missing data: a review by missing mechanism. ieee access 7, 11651–11667 (2019) article  math  google scholar  shahbazian, r., greco, s.: generative adversarial networks assist missing data imputation: a comprehensive survey and evaluation. ieee access 11, 88908–88928 (2023). https://doi.org/10.1109/access.2023.3306721 article  google scholar  stekhoven, d., bühlmann, p.: missforest? non-parametric missing value imputation for mixed-type data. bioinformatics (oxford, england) 28, 112–118 (2012). https://doi.org/10.1093/bioinformatics/btr597 sun, y., li, j., xu, y., zhang, t., wang, x.: deep learning versus conventional methods for missing data imputation: a review and comparative study. expert syst. appl. 227, 120201 (2023). https://doi.org/10.1016/j.eswa.2023.120201 sáez, j.a.: noise models in classification: unified nomenclature, extended taxonomy and pragmatic categorization. mathematics 10(20) (2022). https://doi.org/10.3390/math10203736, https://www.mdpi.com/2227-7390/10/20/3736 sáez, j.a., galar, m., luengo, j., herrera, f.: analyzing the presence of noise in multi-class problems: alleviating its influence with the one-vs-one decomposition. knowl. inform. syst. 38, 179–206 (2014). https://doi.org/10.1007/s10115-012-0570-1 van hulse, j., khoshgoftaar, t.: a comprehensive empirical evaluation of missing value imputation in noisy software measurement data. j. syst. softw. 81, 691–708 (2008). https://doi.org/10.1016/j.jss.2007.07.043 yoon, j., jordon, j., van der schaar, m.: gain: missing data imputation using generative adversarial nets. in: international conference on machine learning (icml), pp. 5689—5698 (2018) google scholar  zhu, b., he, c., liatsis, p.: a robust missing value imputation method for noisy data. appl. intell. 36, 61–74 (2012). https://doi.org/10.1007/s10489-010-0244-1 zhu, x., wu, x.: class noise vs. attribute noise: a quantitative study. artif. intell. rev. 22, 177–210 (2004). https://doi.org/10.1007/s10462-004-0751-8 download references acknowledgments the authors gratefully acknowledge the brazilian funding agencies fapesp (fundação amparo à pesquisa do estado de são paulo) under grants 2022/10553 -6, 2023/13688-2 and 2021/06870-3. this research was also supported by the portuguese recovery and resilience plan (prr) through project c645008882-00000055 center for responsible ai. author information authors and affiliations computer science division, aeronautics institute of technologies, praça marechal eduardo gomes, 50, são josé dos campos, 12228-900, brazil arthur dantas mangussi & ana carolina lorena science and technology institute, federal university of são paulo, talim street 330, são josé dos campos, 12231-280, brazil arthur dantas mangussi & ana carolina lorena miguel torga institute of higher education, 3000-132, coimbra, portugal ricardo cardoso pereira centre for informatics and systems of the university of coimbra, department of informatics engineering, university of coimbra, 3030-290, coimbra, portugal ricardo cardoso pereira & pedro henriques abreu authors arthur dantas mangussiview author publications search author on:pubmed google scholar ricardo cardoso pereiraview author publications search author on:pubmed google scholar pedro henriques abreuview author publications search author on:pubmed google scholar ana carolina lorenaview author publications search author on:pubmed google scholar corresponding author correspondence to arthur dantas mangussi . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper mangussi, a.d., pereira, r.c., abreu, p.h., lorena, a.c. (2025). assessing adversarial effects of noise in missing data imputation. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_14 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords missing data imputation noise filtering publish with us policies and ethics profiles ana carolina lorena view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature applying transformers for anomaly detection in bus trajectories | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper applying transformers for anomaly detection in bus trajectories conference paper first online: 30 january 2025 pp 169–184 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) applying transformers for anomaly detection in bus trajectories download book pdf download book epub michael cruz9 & luciano barbosa9  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 509 accesses abstract trajectory anomaly detection is essential in understanding traffic behavior, especially when external factors such as congestion, accidents, and poor weather conditions occur. most of the previous approaches for this task highly rely on handcrafted features and physical trajectory characteristics, which can be costly to calculate in a large volume of data. in this paper, we propose a novel trajectory anomaly detection approach that relies on language modeling to learn well-formed gps bus trajectories and, based on it, identifies anomalous trajectories and pinpoints their abnormal points (sub-trajectory anomaly detection). our solution uses a deep generative encoder-decoder transformer that learns relationships between the sequential points in the trajectories based on the self-attention mechanism. it does not require manual feature extraction and can be easily adapted to any type of trajectory (e.g., cars, people, and vessels). we have performed an extensive experimental evaluation that shows: (1) our approach is effective for both trajectory and sub-trajectory anomaly detection; and (2) it outperforms the baselines in most evaluation scenarios. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others detecting anomalous bus-driving behaviors from trajectories article 30 september 2020 ts-dbscan: to detect trajectory anomaly for transportation vehicles chapter © 2020 anomalous trajectory detection between regions of interest based on anpr system chapter © 2018 1 introduction with the advances in gps devices and the smart city’s infrastructure, more and more mobile data has been generated [23]. vehicles, smartwatches, traffic signals, and mobile phones, only to name a few, generate a massive volume of trajectory data. these data offer an unprecedented opportunity to discover rich information about traffic behavior, people’s mobility, and weather impact [3, 21]. anomaly detection is the task of finding observations that stand out as dissimilar to all others [4]. particularly in the traffic context, which is easily influenced by external factors (e.g., accidents, detours, events, and weather conditions), trajectory anomaly detection is crucial to understand traffic behavior and to support better decision-making by transit authorities. although trajectory anomaly detection has been a research hotspot [1], some challenges remain. for example, most solutions rely on handcrafted features and physical trajectory characteristics such as density [8], distance [9], and isolation [5]. such features can be costly when dealing, for instance, with a high volume of trajectories, or might not work well in the case of data sparseness [20], which occurs when the period between consecutive trajectory’s reported points is too large. in addition, little attention has been paid to finding anomalous regions in online way [5], which can benefit real applications as follows: example 1: a car ridesharing company might store thousands of daily trajectories to suggest the best routes to its riders and drivers. therefore, finding anomaly trajectories is an essential feature since it can be helpful to recommend alternative paths, free of anomalies such as traffic congestion, and alert riders or penalize drivers that do not follow the recommended paths by its app. example 2: public transportation agencies face traffic problems such as congestion, accident, and poor weather conditions in large cities worldwide. at the same time, these agencies need to serve the population by making real-time decisions to deal with these issues, such as proposing alternative routes or releasing more buses. given that, an online trajectory anomaly detection approach can identify problematic bus trips to allow transit authorities to intervene as quickly as possible. in this paper, we propose an approach that uses language modeling, commonly used in natural language processing tasks, to: (1) detect anomalous trajectories in bus trajectories and (2) pinpoint the abnormal points in these trajectories (sub-trajectory anomaly detection). our solution allows these tasks to be performed either offline or online, i.e., as the buses move along their route. in addition, it can also easily be adapted to other types of trajectories (e.g., cars, people, and vessels) since our model does not use any specific aspect from the bus domain (e.g., bus stops). the key idea of our model is to learn the language of well-formed trajectories and then identify erroneous (ill-formed) trajectories and the trajectories’ points where the errors occur. for that, given an input trajectory t mapped by our solution to a sequence of tokens, our language model (lm) generates \(\hat{t}\), the most likely (common) sequence of points for t, which represents t supposedly without anomalies. our assumption is, therefore, that since anomalous points are typically few and different from the others [5], there is a small chance that abnormal points are present in \(\hat{t}\). based on that, our solution produces an anomaly score for t and pinpoints anomalous regions in t by comparing t with \(\hat{t}\). we build our language model using a deep generative encoder-decoder transformer [16] to learn the relationships between the sequential points in trajectories. more specifically, our solution first maps the input raw trajectory’s points into a geographical grid and uses each grid cell’s id as a token to represent the trajectory points. then, this token-based trajectory representation feeds the transformer encoder, which applies the self-attention mechanism to relate tokens in the sequence. the decoder receives the encoder’s output and leverages previously predicted tokens to generate the next one based on the self-attention mechanism. we have conducted an extensive experimental evaluation on real-world bus datasets from recife (brazil) and dublin (ireland) cities. the results show that our language model effectively detects whether trajectories are anomalous and, at the same time, finds anomalous regions without any handcrafted features. the rest of this paper is organized as follows. section 2 reviews the state of the art of anomaly detection in gps trajectory. in sect. 3, we present some background concepts and the problem statement. section 4 delineates our model, and sect. 5 describes the datasets and the setup experimentation. we compare our results with the state-of-the-art algorithms and previous research in sect. 6. finally, the conclusion and future work are drawn in sect. 7. 2 related work a recent survey [1] provides a comprehensive summary of the state-of-the-art solutions in trajectory anomaly detection. most of them are based on distance [9] and pattern mining [5]. in addition, some approaches use machine learning on a supervised [6] and semi-supervised way [13]. in this section, we discuss some of these approaches in detail. iboat (isolation-based online anomalous trajectory detection) [5] detects anomalies point-by-point by isolating trajectories that are “few” and “different” from historical trajectories on the same route. it uses a “support” function to count how many historical trajectories share points with an ongoing trajectory (called a window). if the support value falls below a certain threshold, the points in the window are marked as anomalous. an anomaly score is then assigned using a logistic function, giving high values to anomalous points and low values to non-anomalous ones. iboat operates online, generating anomaly scores and detecting abnormal points in real-time, but it requires trajectories with the same departure and destination points to function. in gmvsae [13], the authors propose a deep learning encoder-decoder approach to detect whether the trajectory is anomalous. the approach uses an lstm (long short-term memory) architecture to encode and decode trajectories. the approach uses the encoder to model trajectories by a gaussian distribution mixture. then, the proposed method uses each distribution as the decoder’s input information to calculate the probability of an ongoing trajectory belonging to one of the distributions. the trajectory is anomalous if the probability is below a certain threshold. stod (spatial-temporal outlier detector) [6] uses a supervised deep-learning model to detect trajectory anomalies in bus routes. it classifies bus trajectories based on predefined routes and considers them anomalous if the model’s confidence in the classification is low. each trajectory point is represented with features like gps timestamp and a pre-trained embedding vector, which are fed into a bi-gru (gated recurrent unit) architecture followed by a multi-layer perceptron (mlp) with a softmax function. stod calculates entropy over the softmax output’s probability distribution to determine prediction confidence. a trajectory is marked as anomalous if the confidence score falls below a threshold. similar to stod, [2] proposes a multi-class convolutional neural network to classify trajectories by route ids and to identify anomalies based on misclassification or low-class probability. both approaches require route id labels, do not perform online anomaly detection, and do not detect anomalous regions. lastly, ctss (continuous trajectory similarity search) [19] presents an online trajectory search method based on similarity scores to detect anomalous trajectories. for this, the authors proposed an approach that considers the current point of an input trajectory and all possible paths to arrive at the trajectory destination. given all those future trajectories and a reference trajectory (ground truth), the approach calculates similarity scores between each pair (future trajectory and the reference one) and returns the one with minimum distance. then, if the score is greater than a threshold, the input trajectory is anomalous. ctss needs ground truth trajectories to calculate the similarity scores and knowledge of the trajectory destination in advance. 3 problem formulation in this section, we provide some background concepts and state the problem we deal with in this work. definition 1 [bus trajectory]. we define a bus trajectory t as a sequence of consecutive gps points collected from a bus trip, denoted as \(t=\{p_1,p_2,...,p_n\}\) where n is the length of the trajectory. each point \(p_i=\{lat_i,lng_i\}\in \mathbb {r}^{2}\) is composed of latitude (\(lat_i\)) and longitude (\(lng_i\)) is associated with a bus trip and ordered by the points’ timestamp, i.e., \(tsp_i < tsp_{i+1}\). definition 2 [spatial anomaly trajectory]. we consider a trajectory t as anomalous if some of its points spatially diverge from regular trajectories of t’s assigned bus route. definition 3 [problem statement]. given a bus trajectory t, we aim to calculate the spatial anomaly score of t and detect the anomalous points of t. 4 method in this section, we introduce our approach to discovering abnormal trajectories and localizing anomaly regions in trajectories. as fig. 1 shows, our solution is composed of three main components: grid mapping, transformer language model, and anomaly detector. they work as follows. given an input bus trajectory, the grid mapping discretizes it by mapping each of its points to a geographical grid cell, represented by a token, generating a sequence of grid cell tokens. this sequence is passed to a language model, a deep generative encoder-decoder transformer, that produces a series of predicted grid cell tokens. finally, the anomaly detector compares the original token sequence with the predicted one to calculate the trajectory’s anomaly score and identify anomalous points if they exist. in the remainder of this section, we provide further details about each one of these components. fig. 1. the trajectory anomaly detection solution proposed in this work. full size image 4.1 grid mapping the first step of our approach, grid mapping, maps the trajectories, which are multivariate times series, into a univariate sequence of tokens. for this purpose, we use the h3 (hexagonal hierarchical geospatial indexing system)footnote 1 library to create a grid system based on hexagonal cells and a hierarchical index. more specifically, given a raw trajectory \(t=(p_1, p_2,..,p_n)\), where \(p_i\) is each trajectory point, represented by its latitude and longitude (lat, long), and a grid of cell g, we use the geotoh3 function in the h3 library to map points of trajectories into grid cell locations \(c_i\), generating \(t'=(c_i, c_2,...,c_n)\). as a result, every point that falls into the same cell has the same identifier (token). therefore, this mapping reduces the complexity of dealing with a continuous multi-dimensional domain to a discrete uni-dimensional one, which language models can adequately process. 4.2 transformer encoder the first component of our language model is the transformer encoder which maps the sequence of trajectory tokens \(t'\) into a set of vectors that feeds the transformer decoder. for that, it encodes the tokens in \(t'\) based on the other tokens (points) in \(t'\) by applying the self-attention strategy. more concretely, as fig. 1 depicts, the encoder first generates a sequence of embeddings from the trajectory tokens. an embedding is a vector representation of a token in an n-dimension spacefootnote 2. next, a positional encoding adds position information to the embeddings. similar to [16], our position encoder is calculated as: $$\begin{aligned} pe(pos,2_i) = sin(pos/1000^{2_i/d_{model}}) \end{aligned}$$ (1) $$\begin{aligned} pe(pos,2_{i+1}) = cos(pos/1000^{2_i/d_{model}}) \end{aligned}$$ (2) where sin and cos are the trigonometric functions sine and cosine, respectively, pos is the position of the point in the trajectory, and \(d_{model}\) is the dimension in the embedding vector. to create the final embedding representation for each input token, the model performs an element-wise addition of the token embedding with the positional encoding vector. the model then passes these embeddings to the transformer block with four identical encoder layers. it uses the so-called multi-head self-attention to allow the network to attend different input sequence positions and learn which points in the sequence are relevant to the current one. multiple heads create multiple representation subspaces to learn a set of queries q and keys k of dimension \(d_k\), and values v dimension \(d_v\) weight matrices. each head computes the attention weights for a given token embedding j (\(embedding_j\)) as follows: $$\begin{aligned} q_{i,j} = w^{q}_i \cdot embedding_j \end{aligned}$$ (3) $$\begin{aligned} k_{i,j} = w^{k}_i \cdot embedding_j \end{aligned}$$ (4) $$\begin{aligned} v_{i,j} = w^{v}_i \cdot embedding_j \end{aligned}$$ (5) $$\begin{aligned} head = softmax(\dfrac{q_ik_i}{\sqrt{d_k}})v_i \end{aligned}$$ (6) with parameters matrices \(w^q \in \mathbb {r}^{d_{model}xd_k}\), \(w^k \in \mathbb {r}^{d_{model}xd_k}\), and \(w^v \in \mathbb {r}^{d_{model}xd_v}\). the multi-head attention combines the individual heads as follows: $$\begin{aligned} multi\_head=concat(head_1, head_2,...,head_n)w^o \end{aligned}$$ (7) where \(w^o \in \mathbb {r}^{hd_{v}xd_v}\) is a weight matrix learned during training. on top of the multi-head attention, there are two skip connections and two normalization layers interspersed with fully connected feed-forward networks. the residual connection helps the encoder to combine features from different layers, merging different levels of representations [7]. the normalization layers standardize the residual connection and the feed-forward outputs, giving numerical stability to the model. the model calculates it as follows: $$\begin{aligned} \bar{x}_i = \frac{x_i \mu _b}{\sqrt{\sigma _b^2 + \epsilon } } \end{aligned}$$ (8) $$\begin{aligned} z_i = \gamma \cdot \bar{x} + \beta \end{aligned}$$ (9) where \(\mu _b\) and \(\sigma ^2\) are respectively the batch mean and standard deviation, \(\epsilon \) is a stability factor added to variance to avoid a division by zero, \(\gamma \) and \(\beta \) are learning parameters, and \(z_i\) is the normalized value of \(x_i\). note that \(x_i\) is the concatenation between \(multi\_head\) vector and positional embeddings (skip connection) and the normalization vector along with feed-forward output as shown in fig. 1. lastly, the feed-forward network has two layers on the top of the encoder. their goal is to project the normalization of the multi-head attention to another dimension space and add non-linearities between them. the encoder’s final output is the matrix \(z=(k, v)\), where k are key vectors and v value vectors of the tokens in the input sentence. the decoder uses this matrix to focus on appropriate tokens in the input sentence to generate the predicted sequence. 4.3 transformer decoder the transformer decoder is the second component of our language model. its goal is to produce a grid cell token sequence from the input sentence encoded by the transformer encoder. for that, it uses the auto-regressive method, i.e., it predicts each token in the sequence based on the previous ones produced by the model. similar to the encoder, the decoder is also composed of transformer blocks. the decoder self-attention works, however, in a slightly different way. while the self-attention in the encoder considers all tokens from the trajectory to generate the attention weights, the decoder only considers tokens preceding the current one to predict the next. for that, the transformers mask future positions using the look-ahead mask approach [16]. in addition, the first decoder multi-head attention layer learns a query matrix \(q_{dec}\) from the previously predicted tokens. first, the decoder receives the output from the previous layer (embeddings). then, similar to the encoder, the decoder augments it with a positional embedding layer and feeds it to multi-head attention to generate \(q_{dec}\). after, the query vector and the residual connection feed a normalization layer similar to the decoder. next, a second multi-head attention layer receives the learned decoder \(q_{dec}\) and the matrix \(z=(k, v)\) from the encoder output to guide the query/search process. this second multi-head layer allows the decoder to focus on which trajectory points from the encoder are relevant to predict the next token/point. after the second multi-head layer generates the encoder-decoder attention vector, the decoder passes it to a feed-forward layer, followed by another normalization to add non-linearities and stability to the values. finally, the last layer implements a feed-forward neural network that projects the decoder vectors in a large dimension (vocabulary size) to represent the logit vectorfootnote 3. each logit represents a token/cell score, which the softmax function turns into a probability. the model outputs the highest probability token for each position in our decoding strategy, generating the predicted sequence \(\hat{t}\). 4.4 training we use the sparse categorical cross-entropy loss to train our model since the labels are integers. the loss is described as follows: $$\begin{aligned} l(y,\hat{y}) = \sum _{j=0}^{m}\sum _{i=0}^{n} (y_{ij}\cdot log(\hat{y}_{ij})) \end{aligned}$$ (10) where \(y_{ij}\) is the target, and \(\hat{y}_{ij}\) represents the prediction. to train the model, we use the adam optimizer (\(\beta _1=0.9\), \(\beta _2=0.9\), and \(\epsilon =1e^{-9}\)) with a flexible learning rate that increases at the beginning of training and decreases slowly in the remaining training steps conform [16]. we also apply residual dropout with a rate of 0.1 for each layer in the encoder and decoder. it is worth mentioning that our approach learns to generate the input trajectory, then input and targets are the same for training. in addition, during training, we use the teacher-forcing, i.e., we pass the true output to each successive step in the decoder. finally, in the inference step, we provide the input to the encoder and a starting token to the decoder that outputs prediction one token at a time. 4.5 anomaly detector given the encoder’s input token sequence and the decoder’s output, as aforementioned, our solution produces two outputs for a given bus trajectory: its anomaly score and the regions where the anomaly occurs in the trajectory. our primary assumption is that our trained language model predicts the correct sequence. any token in the input sequence (trajectory) that diverges from the predicted ones is considered an anomaly. thus, to calculate the trajectory’s anomaly score, the detector compares the sentence predicted by the decoder with the encoder’s input sentence by aligning them and computing their hamming distance [14], as follows: score = 1 (hamming(t,\(\hat{t}\))/n), where \(\hat{t}\) is the decoder’s predicted sequence, t is the language model input sequence, and n represents their size. we consider the anomalous regions in the input sequence t the trajectory points represented by the unmatched tokens between t and \(\hat{t}\). 5 data description and setup 5.1 experimental setup in this section, we provide details about the setup of our experimental evaluation. datasets. we conducted our experiments in two real-world bus trajectory datasets. the first dataset is from recife, brazil. it comprises 19,290 trajectories (100 points on average) from 82 bus lines generated by 238 buses from october 2017 to november 2017. each bus reports points at intervals of 30 s, containing longitude, latitude, timestamp, route id, vehicle id, instantaneous velocity, and travel distance (from the beginning of the trip). the second dataset is from dublinfootnote 4, ireland. it contains 60,084 trajectories (206 points on average) and 68 bus lines. each trajectory point is reported between 20 and 50 s. in total, there 12,497,472 points collected from jan 01 2013 to jan 04 2013. each point contains the attributes: latitude, longitude, timestamp, line id, journey id, and vehicle id. pre-processing. as mentioned in sect. 4.1, we map the trajectories into a geographical grid. table 1 presents the statistics of trajectories before and after the grid-mapping transformation using the h3 parameter resolution 10 (16 is the maximum resolution), which we chose by experimentationfootnote 5. as can be seen, this transformation greatly reduces the dimensionality of both datasets. table 1. number of unique points before and after the grip mapping.full size table ground truth. since there is no label available in our datasets, one can try manually labeling anomalies as [17, 18] or generate artificial ones as [10, 12, 22]. we chose to generate synthetic anomalies, as manual labels are time-consuming, by adding some perturbation in the real trajectories. we do so by randomly choosing the first point in an actual trajectory t and shifting it along with the following n points in t sequentially. in the experiments, we use two parameters to create different anomaly trajectories from real ones: d (the distance in kilometers from the real points) and p (the percentage of shifted points). for example, using \(d=0.5\) and \(p=0.1\), 10% of trajectory points are moved 500 m from the real point. we generate anomalous trajectories for our experiments considering the values of \(p=[0.1, 0.2, 03]\) and \(d=1.0\). table 2. values of hyper-parameters of transformer.full size table baselines. we evaluate the following anomaly detection methods in our experiments: riobusdata [2] is a supervised method to detect anomalous bus trajectories by classifying them in bus routes. it uses a convolutional neural network (cnn) fed by raw bus trajectories. on the top of cnn, a softmax function outputs a vector of probability where each value is the probability of class membership for each route/label. a trajectory is abnormal if its highest-class probability is below a given threshold. stod [6] is also a supervised method that detects anomalous bus trajectories and learns to classify bus trajectories in their routes using a deep-learning network. the model outputs the routes’ class distribution of a given trajectory. from this distribution, it calculates the uncertainty of the classifier using entropy as a measure of anomaly degree. the higher the classifier uncertainty, the higher the entropy. a trajectory is anomalous if the entropy of the classifier’s probability distribution output for it is higher than a threshold. gm-vsae [13] uses an encoder-decoder strategy to detect anomalous trajectories. to perform that, firstly, the encoder infers a disentangled latent space to discover the distribution of each trajectory based on this space. this distribution is then fed to the decoder that generates a trajectory. the method calculates a score comparing the generated trajectory with the input trajectory. a high score \(\approx \) 1.0 means that the input trajectory has a high probability of being an anomaly. iboat [5] is based on the isolation mechanism [11], and an adaptive windows approach. it performs the detection of both trajectory and sub-trajectory anomaly detection. iboat calculates the frequency of points mapped into a grid cell to isolate “few and different” points. based on that, trajectories that visit cells with low frequency get small scores, meaning that those points are highly likely to be an anomaly. conversely, trajectories with high-frequent visited cells have high scores of non-anomaly. similar to the previous methods, iboat also needs a threshold to detect anomalies. transformer is our proposed approachfootnote 6. we train our model on both datasets with the hyper-parameter values shown in table 2. it is worth pointing out that all those methods identify anomalous trajectories, but only our approach (transformer) and iboat detect anomalous sub-trajectories. we randomly selected 8,200 trajectories from the recife dataset and 6,800 from dublin to evaluate the approaches. based on a data analysis, we defined the maximum number of points as 100 for the recife dataset and 208 for dublin. evaluation metrics. we use f1-measure, precision, recall, and pr-auc as evaluation metrics since they are usually applied to evaluate outlier detection methods [1, 13]. to verify whether the f1-measure values of our model are statistically different from the baselines, we execute the wilcoxon statistical test [15]. the test verifies whether two paired samples (f1-measure values of our solution vs. a baseline) come from the same distribution. given that, we set the significance level \(\alpha =5\%\). in our context, the null hypothesis \(h_0\) considers that the median difference between the f1 values of a pair of models is zero. we performed this statistical test on the instances in the test set. 6 results and discussion in this section, we first present the evaluation of the trajectory outlier identification and, subsequently, the region anomaly detection. table 3. results of anomaly trajectory detection on the dublin dataset.full size table table 4. results of anomaly trajectory detection on the recife dataset.full size table 6.1 trajectory anomaly detection table 3 presents the results for the trajectory anomaly detection task on the dublin dataset. transformer outperforms the baselines in all scenarios and metrics. for example, considering the best results on f1, our approach is at least \(17\%\) better than all baselines. to confirm this, table 6 depicts the p-values for the results of each baseline in comparison to transformer on the dublin dataset. all the p-values are smaller than the significance level (0.05), which supports that our transformer network has, in fact, superior performance than the baselines on this dataset. regarding the results on the recife dataset, presented in table 4, our method obtains better f1-measure values than stod and riobusdata, and comparable ones with gmvsae and iboat. the p-values of the hypothesis tests on this dataset for f1-measure, table 7, confirm this: the p-values of gmvsae and iboat versus transformer in all scenarios are higher than the significance level of 0.05, meaning there is no statistical difference in terms of f1-measure between our approach and them. regarding riobusdata and stod, however, the p-values are lower than 0.05 in two of the three anomaly cases. looking at precision values, transformer achieved the best overall results but lower recall than gmvsae and iboat. in practice, better precision can be an advantage since an anomaly detection model can be considered a filter that identifies possibly a few anomalous trajectories in a large set of trajectories. the more precise this filter is, the few false negative anomalies need to be inspected. overall, the methods built to detect anomalies (transformer, gmvsae and iboat) outperformed in almost all scenarios the ones that try to do this indirectly (stod and riobusdata), i.e., learning to classify routes instead of anomalies. for example, for \(p=0.1\), riobusdata obtained the lowest f1 (0.651) on dublin and recife (0.533). however, on the dublin dataset, stod shows better f1 results than iboat for \(p=0.2\) (0.69 vs 0.673 respectively), and for \(p=0.3\) obtained the f1 second best value (0.724) only behind transformer (0.854). we also observed that stod is more sensible than our method over the percentage outlier variation p. for example, for \(p=0.1\), and \(p=0.3\), the f1-measure is 0.58 and 0.66 on recife (difference of 0.8), respectively. in contrast, our method is more stable regarding the outlier level. for instance, for \(p=0.1\) and \(p=0.3\), the f1-measure is respectively 0.84 and 0.85 on dublin and 0.66 and 0.67 on recife, i.e., there is not much difference. fig. 2. pr-auc of the approaches for route 1 on the dublin dataset and route 54 on the recife dataset. full size image table 5. results for the region anomaly detection models.full size table to provide a detailed analysis of the approaches on individual routes, fig. 2 shows the pr-auc curves of all methods on both datasets in two different routes, one from each dataset with \(p=0.3\). in route 54 from recife, our approach has the highest area under curve \(\approx 0.98\), outperforming both the unsupervised methods (gmvsae \(\approx 0.94\) and iboat \(\approx 0.77\)) and the supervised ones (stod \(\approx 0.54\) and riobusdata \(\approx 0.67\)). looking at route 1 from dublin, we observe that the encoder-decoder methods have almost the perfect curve auc \(\approx 0.99\), i.e., the models can adequately distinguish anomalous trajectories from non-anomalous ones. conversely, the riobusdata has the worst pr-auc curve \(\approx 0.70\). finally, we can see that the precision of iboat degrades with a recall close to 1. table 6. hypothesis test for f1 on the dublin dataset.full size table table 7. hypothesis test for f1 on the recife dataset.full size table 6.2 region anomaly detection table 5 shows the results between transformer and iboat on region anomaly detection. we observe that transformer outperforms iboat on the recife dataset in all scenarios. this occurs mainly because our model achieved the high values of precision (0.988, 0.975, 0.961). the methods, however, are similar regarding recall for \(p=0.1\), for instance, transformer’s recall is 0.992, and iboat 0.990. on the dublin dataset, the methods are qualitatively similar. note that the most difference between the models occurs for \(p=0.1\): transformer’s f1 is 0.986, and iboat’s f1 is 0.978. we applied the wilcoxon test to verify whether there is a statistical difference between the f1-measure values of the methods on this task. on the recife dataset, the models are statistically different, with p-value lower than our significance level of 0.05 for all scenarios. on the dublin dataset, however, there is no statistical evidence to reject the \(h_0\), since the p-values are higher than 0.05 and, therefore, both models are statistically equivalent in terms of f-1 measure. to present concrete examples of the detection of region anomalies on real trajectories by our model, fig. 3a shows the expected trajectories of dublin route 1, and fig. 3b depicts the anomalous regions identified (represented by the red dots) by transformer. one can see from these plots that our approach identifies the anomalous regions in this trajectory very precisely. fig. 3. example of anomaly detection inference. full size image 7 conclusion in this paper, we propose a solution that applies an encoder-decoder transformer language model in bus trajectory data to solve two problems: trajectory and sub-trajectory anomaly detection. our solution transforms a trajectory into a discrete token sequence by mapping its points to tokens representing geographical grid cells. this sequence is then passed to the transformer language model that outputs a predicted sequence, supposedly without anomalies. finally, our solution calculates the trajectory’s anomaly score by applying the hamming distance between the two sequences and identifies the anomalous regions by looking at the unmatched tokens between them. experiments in two real-world bus trajectory datasets demonstrate that our approach is effective for anomalous trajectory detection and anomalous region detection tasks. in future work, we intend to train our approach in multiple trajectory datasets to verify whether it can learn general trajectory patterns (deep representation). once our approach learns those patterns, we want to exploit other tasks, such as trajectory similarity and classification, using transfer learning. notes 1.https://eng.uber.com/h3/. 2.we use embeddings of 256 dimensions in our solution. 3.the logit vector dimension is the total number of grid cell tokens (vocabulary). 4.https://data.gov.ie/dataset/dublin-bus-gps-sample-data-from-dublin-city-council-insight-project. 5.the resolution allows the library to increase or decrease the size of grid cells. thus, the higher resolution, the smaller the cell areas. 6.https://github.com/michaeloc/its_research. references belhadi, a., djenouri, y., lin, j.c.w., cano, a.: trajectory outlier detection: algorithms, taxonomies, evaluation, and open challenges. acm trans. manage. inf. syst. (tmis) 11(3), 1–29 (2020) article  math  google scholar  bessa, a., silva, f.d.m., nogueira, r.f., bertini, e., freire, j.: riobusdata: outlier detection in bus routes of rio de janeiro. arxiv preprint arxiv:1601.06128 (2016) bouritsas, g., daveas, s., danelakis, a., thomopoulos, s.c.: automated real-time anomaly detection in human trajectories using sequence to sequence networks. in: 2019 16th ieee international conference on advanced video and signal based surveillance (avss), pp. 1–8. ieee (2019) google scholar  chalapathy, r., chawla, s.: deep learning for anomaly detection: a survey. arxiv preprint arxiv:1901.03407 (2019) chen, c., et al.: iboat: isolation-based online anomalous trajectory detection. ieee trans. intell. transp. syst. 14(2), 806–818 (2013) article  math  google scholar  cruz, m., barbosa, l.: learning gps point representations to detect anomalous bus trajectories. ieee access 8, 229006–229017 (2020) article  math  google scholar  huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4700–4708 (2017) google scholar  kong, x., song, x., xia, f., guo, h., wang, j., tolba, a.: lotad: long-term traffic anomaly detection based on crowdsourced bus trajectory data. world wide web 21(3), 825–847 (2018) article  google scholar  lee, j.g., han, j., li, x.: trajectory outlier detection: a partition-and-detect framework. in: 2008 ieee 24th international conference on data engineering, pp. 140–149. ieee (2008) google scholar  li, x., zhao, k., cong, g., jensen, c.s., wei, w.: deep representation learning for trajectory similarity computation. in: 2018 ieee 34th international conference on data engineering (icde), pp. 617–628. ieee (2018) google scholar  liu, f.t., ting, k.m., zhou, z.h.: isolation-based anomaly detection. acm trans. knowl. discov. data (tkdd) 6(1), 1–39 (2012) article  math  google scholar  liu, s., ni, l.m., krishnan, r.: fraud detection from taxis’ driving behaviors. ieee trans. veh. technol. 63(1), 464–472 (2013) article  google scholar  liu, y., zhao, k., cong, g., bao, z.: online anomalous trajectory detection with deep generative sequence modeling. in: 2020 ieee 36th international conference on data engineering (icde), pp. 949–960. ieee (2020) google scholar  luu, v.t., forestier, g., weber, j., bourgeois, p., djelil, f., muller, p.a.: a review of alignment based similarity measures for web usage mining. artif. intell. rev. 53(3), 1529–1551 (2020) article  google scholar  siegel, s.: nonparametric statistics for the behavioral sciences (1956) google scholar  vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, pp. 5998–6008 (2017) google scholar  wang, j., et al.: anomalous trajectory detection and classification based on difference and intersection set distance. ieee trans. veh. technol. 69(3), 2487–2500 (2020) article  math  google scholar  zhang, d., li, n., zhou, z.h., chen, c., sun, l., li, s.: ibat: detecting anomalous taxi trajectories from gps traces. in: proceedings of the 13th international conference on ubiquitous computing, pp. 99–108 (2011) google scholar  zhang, d., chang, z., wu, s., yuan, y., tan, k.l., chen, g.: continuous trajectory similarity search for online outlier detection. ieee trans. knowl. data eng. (2020) google scholar  zhang, y., ning, n., zhou, p., wu, b.: ut-atd: universal transformer for anomalous trajectory detection by embedding trajectory information. in: proceedings of the 27th international conference on distributed multimedia systems (2021) google scholar  zhao, x., rao, y., cai, j., ma, w.: abnormal trajectory detection based on a sparse subgraph. ieee access 8, 29987–30000 (2020) article  google scholar  zheng, g., brantley, s.l., lauvaux, t., li, z.: contextual spatial outlier detection with metric learning. in: proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 2161–2170 (2017) google scholar  zheng, y.: trajectory data mining: an overview. acm trans. intell. syst. technol. (tist) 6(3), 1–41 (2015) article  math  google scholar  download references acknowledgment this work is partially supported by ines (www.ines.org.br), cnpq grant 465614/2014-0, facepe grants apq-0399-1.03/17 and apq/0388-1.03/14, capes grant 88887.136410/2017-00. author information authors and affiliations universidade federal de pernambuco centro de informática, recife, brazil michael cruz & luciano barbosa authors michael cruzview author publications search author on:pubmed google scholar luciano barbosaview author publications search author on:pubmed google scholar corresponding author correspondence to michael cruz . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cruz, m., barbosa, l. (2025). applying transformers for anomaly detection in bus trajectories. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_12 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords anomaly detection trajectory language model transformer publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature deployment of ibm federated learning platform and aggregation algorithm comparison: a case study using the mnist dataset | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deployment of ibm federated learning platform and aggregation algorithm comparison: a case study using the mnist dataset conference paper first online: 30 january 2025 pp 310–323 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) deployment of ibm federated learning platform and aggregation algorithm comparison: a case study using the mnist dataset download book pdf download book epub hans herbert schulz  orcid: orcid.org/0009-0003-5342-84579 & benjamin grando moreira  orcid: orcid.org/0000-0002-0339-40129  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 475 accesses 1 citation abstract with the exponential growth in artificial intelligence, privacy and data acquisition concerns have surged, prompting stricter data protection laws. federated learning (fl) addresses these issues by enabling model training without accessing private data, allowing geographically dispersed clients to participate without sharing data. this study deployed an ibm fl platform using docker containers across two clients and conducted training with the mnist dataset using the two most common fl strategies: federated stochastic gradient descent (fedsgd) and federated averaging (fedavg). the results validated the platform’s deployment and assessed the performance of each strategy in terms of model accuracy and client hardware capabilities. performance metrics, including cpu and ram usage, network traffic, and model accuracy, were collected. despite the higher resource demands, both strategies achieved satisfactory model accuracy, with fedavg showing slightly better efficiency for the small-scale deployment. the results emphasize the potential of fl for predictive maintenance in industrial applications, enabling decentralized data utilization while ensuring data privacy and security. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others federated learning client selection algorithm based on gradient similarity article 26 november 2024 model poisoning attack against federated learning with adaptive aggregation chapter © 2024 towards effective device-aware federated learning chapter © 2019 1 introduction traditional machine learning methods typically rely on centralized datasets to train models. however, this centralized approach carries significant risks, especially concerning data breaches that could expose all the training data [1, 9]. in response to these concerns, federated learning (fl) has emerged as a promising solution. fl enables model training across geographically dispersed clients without the need to share their private data, thereby preserving data privacy and security. this work is part of an industrial solution to enhance predictive maintenance within production engineering. the project focuses on developing a multi-sensory platform capable of acquiring and transmitting air parameters (e.g., saturation, temperature, and flow speed) from geographically apart industrial air exchange systems. the collected data will be used to train machine learning models to improve predictive maintenance routines. the primary goal of this study is to deploy an fl platform using docker container technology to train models and check its functionality by comparing the performance of machine learning models using different fl strategies. the deployment process involves several tasks aimed at installing a pre-developed application into its intended operational environment [5]. after a successful deployment, performance tests are conducted utilizing the usual fl techniques for machine-learning model algorithms and the results are discussed. 2 federated learning overview as industries grow, more technology is being required and developed. among those technologies, federated learning is proposed as a pioneering approach to some common issues faced in machine learning, such as privacy and data storage. in [7] a review application in industrial engineering is presented. with the advent of deep learning (dl), the need for data acquisition in several science instances has increased. this is mainly because dl models are usually interpolators, meaning they present the best results when the input is similar to the data used for training models. therefore, a huge amount of heterogeneous data is required to achieve decent accuracy [16]. in light of this characteristic, google presented in 2016 the concept of federated learning (fl), which can be described as a machine learning technique to provide decentralized collaborative learning across different nodes [10]. the fl architecture comprises multiple parties and an aggregator agent. firstly, a global model is generated and sent to each party. then, the model will be trained in each party with its own data set, meaning each model will have a different result by the end of the process [9]. an update that contains the difference between the base model and the new one is then generated by the parties and sent to the aggregator agent. the agent then merges the updates and creates a new global model sent to each party, and so on [10]. to understand federated learning comprehensively, we will explore its key components and methodologies. the following subsections will delve into data distribution methods, training models, and essential algorithms that form the backbone of federated learning. 2.1 data distribution data can be distributed and used in the fl context in different ways. the conventional way consists of having multiple parties training a model independently, and the data in all parties possess the same features. such distribution results in the so-called horizontal federated learning (hfl) approach [8]. in contrast to hfl, one can find the vertical federated learning (vfl) which consists of training a model where each party’s data contains different features of the same set of individuals [8]. essentially, all parties have data related to the same entities but with different features. for example, in a healthcare scenario, one party might have medical records (features) for a set of patients (entities), while another party might have lifestyle information (different features) for the same set of patients. to summarize, the main difference between hfl and vfl lies in how the training data is distributed among parties. the hfl will focus on training multiple data samples sharing the same features, whereas the vfl will use the same sample but different features [13]. 2.2 fl training design from a system perspective, the fl scenarios aim to train machine learning models using disparate data to maintain privacy and overall model performance. it is imperative that data is not moved across parties and not even visualized by the central server. to achieve that, there are mainly two system designs: cross-device and cross-silo [9]. the objective of cross-device fl is minimizing the function f(w), which is defined by eq. 1 [9]: $$\begin{aligned} f(w) = \sum _{k=1}^n p_kf_k(w) \end{aligned}$$ (1) in eq. 1, \(f_k(w)\) is the local objective function for the device k with model weights w. the term \(p_k\) represents the importance given to the contribution of the k device to the global model objective function. usually, in the cross-device setting, the parties are user-owned iot (internet of things) instruments such as cell phones, tablets, edge devices, and others [8]. figure 1 depicts the covered steps of the cross-device learning system whose main steps are [8]: 1. initially, the model \( g_0 \) (represented as a neural network image in the upper rectangle) is untrained and initialized with random weights. a subset of the available devices is selected at the beginning of each round; 2. after selecting \( k \) devices, the initial weights \( w \) of the global model \( g_n \) are sent to the devices; 3. each device trains the model on its dataset \( d_k \), resulting in local models \( g_{nk} \); 4. privacy mechanisms anonymize the local models, which are then sent to the aggregator; 5. the aggregator combines local models using an aggregation algorithm to create a new global model \( g_{n+1} \). the process repeats from step 1 until a stopping condition, like the maximum number of rounds or model convergence, is met. fig. 1. cross-device federated learning system overview. source: [8] full size image on the other hand, the main objective of cross-silo federated learning can be formally described as minimizing the l(w) function described by eq. 2. $$\begin{aligned} l(w) = \sum _{j=1}^m \sum _{i=1}^n l(w_j, x_j^i) \end{aligned}$$ (2) where l(w) is the global loss function for the global model w, i represents the client, j is the data provider (i.e., silos) with the local model \(w_j\) trained from data feature \(x^i_j\) [8]. in the context of cross-silo, the data silos used as parties tend to be of commercial grade, providing a less heterogeneous environment and generally having better computational resources than iot devices [8]. figure 2 depicts the chain of events for the cross-silo system whose main steps are [8]: 0. participants align datasets using techniques like secure multi-party communication or key-sharing, ensuring data alignment by id. unmatched data is discarded; 1. a third-party aggregator distributes encryption key pairs to each party and initializes partial models. the last party usually holds the labels, while others have training features; 2. the first party trains on a mini-batch of local data, encrypts the output with homomorphic encryption, and sends it to the next party; 3. the second party performs a forward pass, calculates the loss (since it holds the labels), and sends intermediate outputs to the first party. the loss is sent to the aggregator; 4. both parties compute partial gradients, add encryption, and send them to the aggregator to prevent data leakage; 5. the aggregator decrypts the gradients, calculates exact gradients using the loss, and sends them back to the parties for model updates. the process repeats from step 1. fig. 2. cross-silo federated learning system overview. source: [8] full size image the training procedure for cross-silo fl also happens in rounds but with one significant difference: all silos take part in every round of the training. therefore, the choice of targeted devices significantly influences the system selection. in cross-device fl, user-owned iot devices like cellphones, tablets, and edge devices are common, leading to a diverse range of specifications. conversely, cross-silo fl typically involves commercial-grade data silos, offering a more uniform environment with superior computational resources compared to iot devices [8]. thus, the type of devices used as parties in training will determine the most suitable fl system. 2.3 federated learning algorithms after establishing the main concepts of fl and its benefits and challenges, it becomes clear that the next step is optimization to address common issues, such as unbalanced data sets and limited communication. two algorithms used to contour these challenges are the federated stochastic gradient descent (fedsdg) and the federated averaging (fedavg) as they are the most common in the fl context [8]. to apply the fedsdg in fl, a c fraction of clients is chosen in each round and the gradient of the loss over all the data held by the clients is computed. therefore, c refers to the global batch size with \(c = 1\) representing the full batch setup [11]. a typical implementation is with \(c=1\) and a fixed learning rate \(\eta \) [11]. in this scenario, each client k will compute \(g_k = \nabla f_k(w_t)\) (the average gradient on its local data for model \(w_t\)). then, the central server aggregates these gradients and applies the update \(w_{t+1} \leftarrow w_t \eta \sum _{k=1}^k \dfrac{n_k}{n} g_k\) since that \(\sum _{k=1}^k \dfrac{n_k}{n} g_k = \nabla f (w_t)\). an equivalent update is found by \(\forall k, w_{t+1}^k \leftarrow w_t \eta g_k\) followed by \(w_{t+1} \leftarrow \sum _{k=1}^k \dfrac{n_k}{n} w_{t+1} ^ k\). this means that each client will locally take one step of gradient descent on the current model using its local data. the server then takes a weighted average of the resulting models [11]. conversely, the federated averaging algorithm, fedavg, is an extension of the fedsdg. the main concept is following the same steps motioned above, but also iterating the local update \(w_k \leftarrow w^k \eta \nabla f_k(w^k)\) in each client multiple times before the averaging step [11]. besides the previous hyperparameters c and \(\eta \), the fedavg will count with two more. b represents the local mini-batch size and e is the number of iterations through the local data before the global model is updated [14]. it is worth noting that both algorithms work with cross-silo and cross-device training design. for this work, cross-silo was used since it is an industry-oriented project. 3 methodology this section outlines the sequential steps taken in this study, including the selection of the federated learning (fl) library, deployment mechanism, hardware setup, dataset, neural network model, fl parameters, network configuration, and evaluation criteria. 3.1 fl library given the enterprise-oriented nature of the application, which prioritizes privacy, security, and rapid model specification, the ibm fl library was selected for this project. this library was chosen for its robust cryptographic methods, extensive range of fl strategies, and support for various machine learning models. according to [15], from the most notable libraries, only flower and ibm fl are production-ready libraries. a comparison between fl platforms (pysyft, flower, ibm fl, tff and fedml) was performed and is available at [17]. 3.2 hardware the fl platform’s main components include a central server (aggregator) and client devices. two devices were chosen as silos for this study. table 1 presents the hardware configurations for each device (silo). table 1. hardware configurations of federated learning client devicesfull size table 3.3 dataset to test the platform, the modified national institute of standards and technology (mnist) dataset [3] was used. this dataset contains 28\(\,\times \,\)28 pixel grayscale images of handwritten digits (0 through 9) and corresponding labels. it is a standard benchmark for machine learning algorithms. 3.4 neural network overview for pattern recognition tasks involving images, convolutional neural networks (cnn) are commonly used [4]. the training model used in the experiment is a keras cnn classifier, a built-in feature in the fl library. since this model aims to test the platform, the suggested model in the ibm guide was used. it was configured to possess two convolutional layers with activated relu, one pooling layer, one dropout, flatten, and two dense layers to compose the fully-connected layers segment. 3.5 fl parameters two categories of tests were conducted to explore the platform’s capabilities, each repeated five times. the main variable between the tests was the fl strategy used: fedsgd and fedavg. table 2 summarizes the training parameters. table 2. federated learning training parametersfull size table 3.6 network configuration the platform’s deployment involved creating a private vpn connection to ensure secure data transmission. a websocket connection was established between the fl client app and the web app backend, along with a flask connection between the aggregation service and local training services. 3.7 evaluation criteria in terms of model training, threshold metrics were used. a threshold metric quantifies classification prediction errors, meaning that it is a fraction or ratio of a class classification and its expected classification. they are usually used when it is expected to minimize the number of errors [2]. the platform’s performance was evaluated based on hardware metrics (cpu and ram usage), network traffic, and model training metrics. threshold metrics, which measure classification prediction errors, were used to assess accuracy, loss, precision, recall, f1 score, and the confusion matrix for each global model. 4 results this section addresses the achieved results and discusses how they compare to the expected output from the platform setup. data was collected from each fl strategy (fedavg and fedsdg) through a python script using the psutil and time libraries. 4.1 resource usage for fedavg strategy the project’s initial proposition was to use edge devices such as raspberry pi devices to perform the role of parties. therefore, the main concern with the deployment was the resources required to execute a full fl round, given that edge devices are known for their limited hardware capabilities. figure 3(a) and fig. 3(b) show the cpu usage while using the fedavg strategy from clients ipt-n-0007 and ipt-n-0311 respectively. each cpu presents the same 20-peak behavior, indicating that model training occurred as expected, with each peak representing one training round. fig. 3. (a) cpu usage from device ipt-n-0007 in fedavg. (b) cpu usage from device ipt-n-0311 in fedavg. full size image figures 4(a) and 4(b) show the ram usage for both devices during the experiment. the scale indicates that device ipt-n-0311 uses almost 600 mb/s more ram than ipt-n-0007, likely due to having more available memory. this may jeopardize using edge devices as parties due to their limited ram resources. fig. 4. (a) ram usage from device ipt-n-0007 in fedavg. (b) ram usage from device ipt-n-0311 in fedavg. full size image the next criterion to analyze is the network traffic from both parties. figure 5(a) and 5(b) depicts both devices network traffic. since the training procedure encloses receiving and sending the model parameters, peak patterns like cpu usage are also expected. fig. 5. (a) network traffic from device ipt-n-0007 in fedavg. (b) network traffic from device ipt-n-0311 in fedavg. full size image both figures show a very distinctive exponential-shaped curve from the network traffic. that can happen due to a phenomenon entitled communication overhead.footnote 1 after every round, the model gets more accurate, which means that some parameters will no longer be altered. therefore, the size of the exchanged communication keeps getting smaller after every round. 4.2 resource usage for fedsdg strategy the same analysis was conducted for the fedsdg strategy. figure 6 show the cpu usage for both parties. fig. 6. (a) cpu usage from device ipt-n-0007 in fedsdg. (b) cpu usage from device ipt-n-0311 in fedsdg. full size image the presented behavior is almost identical to the one observed while using the fedavg strategy. however, since the fedavg algorithm is a variation of the fedsdg, a similar behavior was expected. ram usage in the experiment has also been tracked and is shown in fig. 7 from devices ipt-n-0311 and ipt-n-0007 respectively. once again, the observed behavior is similar to the one observed within the fedavg strategy. fig. 7. (a) ram usage from device ipt-n-0007 in fedsdg. (b) ram usage from device ipt-n-0311 in fedsdg. full size image finally, the network traffic is analyzed and the results are shown in fig. 8(a) and 8(b) for both ipt-n-0311 and ipt-n-0007 parties respectively. fig. 8. (a) network traffic from device ipt-n-0007 in fedsdg. (b) network traffic from device ipt-n-0311 in fedsdg. full size image 4.3 training results for fedavg the training results of the 5 experiments for fedavg is presented in table 3 and a confusion matrix is depicted in fig. 9. table 3. global model metrics achieved with fedavg strategyfull size table the f1 score for all five experiments was above 0.87, indicating a decent classification model. since the main objective of the experiments was testing the platform, the enhancement of the model performance by using extra layers, for instance, was neglected. the confusion matrix in fig. 9 shows clearly defined diagonal squares. fig. 9. confusion matrix for experiment 1 using fedavg full size image table 4. global model metrics achieved with fedsdg strategyfull size table fig. 10. confusion matrix for experiment 1 using fedsdg full size image 4.4 training results for fedsdg table 4 shows the training metrics for the fedsdg strategy, with the corresponding confusion matrix depicted in fig. 10. similar to the fedavg experiments, the metrics for the fedsdg strategy were within the established thresholds. the confusion matrix in fig. 10 shows well-defined diagonal squares. the results evaluated the performance and resource usage of fedavg and fedsdg federated learning strategies using multi-sensory device data. both strategies achieved high accuracy and f1 scores, indicating effective classification models, with fedavg showing slightly better resource efficiency. cpu and ram usage were significant, particularly for devices with lower ram capacity, and network traffic increased with model accuracy, with fedsdg consuming more bandwidth and taking longer. scalability issues and data heterogeneity challenges were noted, underscoring the need for careful consideration in practical implementations. 5 conclusion this work aimed to deploy a federated learning library into an industrial platform and test its functionalities so that it can later be adapted to train with data acquired and transmitted from multi-sensory devices. to achieve the main objective, the library and several other features such as the dataset itself, were containerized and deployed in each device that worked as a client in the training process, through a docker container. afterwards, the websocket and flask connections were established, and minor configurations such as export model function buttons were added. with the deployment process completed, two sets of five different global model training were executed using two different fl strategies, the fedavg and fedsdg. nevertheless, when comparing the strategies hardware-wise, the results have shown that the fedsdg strategy consumes higher network bandwidth, ranging from 30% to almost 70% more, depending on the analyzed experiment. moreover, the fedsdg took an average of 1,8 more minutes to finish the training procedure than the fedavg. for a test training with only two parties, two minutes does not impact the final results, however, the full concept of fl is to be able to train on dozens or hundreds of devices, meaning, this network consumption can exceed the available network traffic and impact on several more minutes of delay. although the testing was a proof of concept made with only two parties, when extending to dozens of clients, some other challenges may appear such as the heterogeneity of data. according to [12], since each company is physically apart, the way data is sampled can vary for each party, and achieving a good performance model requires similar data sets in all parties. as the project unfolded, the federated learning approach seemed even more fitting since one company could train a model with data from multiple branches, enhancing the predictive maintenance schedule and mitigating production costs. moreover, this approach affords the company the ability to retain all data within its storage infrastructure, thereby ensuring the privacy and governance of each client’s data. future work could explore scalability testing with larger networks, incorporating more diverse datasets beyond mnist, and investigating advanced aggregation techniques to improve performance and privacy. additionally, comparing the ibm fl platform with other frameworks, examining the impact of communication delays, and customizing federated learning algorithms for specific use cases could further refine the platform’s effectiveness and applicability. notes 1.communication overhead is defined as the total number of packets to be transferred or transmitted from one node to another [6]. references anthem pays ocr \$16 million in record hipaa settlement following largest u.s. health data breach in history (2020). https://www.hhs.gov/guidance/document/anthem-pays-ocr-16-million-record-hipaa-settlement-following-largest-us-health-data-breach brownlee, j.: imbalanced classification with python: better metrics, balance skewed classes, cost-sensitive learning. mach. learn. mast. (2020) google scholar  cohen, g., afshar, s., tapson, j., van schaik, a.: emnist: an extension of mnist to handwritten letters (2017) google scholar  herman, r.l.: introduction to partial differential equations. r.l. herman (2015). https://doi.org/10.1007/978-3-030-96896-0, https://math.libretexts.org/bookshelves/differential_equations/introduction_to_partial_differential_equations_(herman)/09%3a_transform_techniques_in_physics/9.06%3a_the_convolution_operation heydarnoori, a., mavaddat, f.: reliable deployment of component-based applications into distributed environments. third international conference on information technology: new generations (itng’06), pp. 52–57 (2006).https://doi.org/10.1109/itng.2006.112, https://ieeexplore.ieee.org/document/1611570 kumar, n., singh, y.: routing protocols in wireless sensor networks, p. 86–128. igi global (2017). https://doi.org/10.4018/978-1-5225-0486-3.ch004, http://dx.doi.org/10.4018/978-1-5225-0486-3.ch004 li, l., fan, y., tse, m., lin, k.y.: a review of applications in federated learning. comput. ind. eng. 149, 106854 (2020). https://doi.org/10.1016/j.cie.2020.106854, https://www.sciencedirect.com/science/article/pii/s0360835220305532 ludwig, h., baracaldo, n. (eds.): federated learning. springer (2022). https://doi.org/10.1007/978-3-030-96896-0 ludwig, h., et al.: ibm federated learning (2020). https://doi.org/10.48550/arxiv.2007.10987 manias, d.m., shami, a.: making a case for federated learning in the internet of vehicles and intelligent transportation systems. ieee network 35(3), 88–94 (2021). https://doi.org/10.1109/mnet.011.2000552 article  math  google scholar  mcmahan, h.b., moore, e., ramage, d., hampson, s., y arcas, b.a.: communication-efficient learning of deep networks from decentralized data (2023). https://doi.org/10.48550/arxiv.1602.05629 laydner de melo rosa, g., mohanram, p., gilerson, a., schmitt, r.h.: architecture for edge-based predictive maintenance of machines using federated learning and multi sensor platforms (2023). https://doi.org/10.20944/preprints202305.1563.v1 nguyen, d.c., ding, m., pathirana, p.n., seneviratne, a., li, j., vincent poor, h.: federated learning for internet of things. ieee commun. surv. tutor. 23(3), 1622–1658 (2021). https://doi.org/10.1109/comst.2021.3075439 article  math  google scholar  nilsson, a., smith, s., ulm, g., gustavsson, e., jirstrand, m.: a performance evaluation of federated learning algorithms. in: proceedings of the second workshop on distributed infrastructures for deep learning (2018). https://doi.org/10.1145/3286490.3286559, https://dl.acm.org/doi/pdf/10.1145/3286490.3286559 saidani, a.: a systematic comparison of federated machine learning libraries. master’s thesis, technische universität münchen (2023). https://wwwmatthes.in.tum.de/pages/1giyhi2qf7es2/master-s-thesis-ahmed-saidani sarma, k.v., et al.: federated learning improves site performance in multicenter deep learning without data sharing. j. am. med. inform. association: jamia 28, 1259–1264 (2021) article  math  google scholar  schulz, h.h.: decentralized brilliance: deploying a federated learning platform and evaluating aggregation algorithms (2024). https://repositorio.ufsc.br/handle/123456789/255868 download references author information authors and affiliations joinville technological center (ctj), federal university of santa catarina (ufsc), r. dona francisca, 8300, cep 89219-600 joinville, santa catarina, brazil hans herbert schulz & benjamin grando moreira authors hans herbert schulzview author publications search author on:pubmed google scholar benjamin grando moreiraview author publications search author on:pubmed google scholar corresponding author correspondence to hans herbert schulz . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper schulz, h.h., moreira, b.g. (2025). deployment of ibm federated learning platform and aggregation algorithm comparison: a case study using the mnist dataset. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_22 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords federated learning data privacy ibm federated learning platform publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature efficient density-based models for multiple machine learning solutions over large datasets | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper efficient density-based models for multiple machine learning solutions over large datasets conference paper first online: 12 october 2023 pp 48–62 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) efficient density-based models for multiple machine learning solutions over large datasets download book pdf download book epub natanael f. dacioli batista  orcid: orcid.org/0009-0004-8489-58999, bruno leonel nunes  orcid: orcid.org/0009-0006-5479-43469 & murilo coelho naldi  orcid: orcid.org/0000-0002-3107-82369  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 526 accesses 1 citation abstract unsupervised and semi-supervised machine learning is very advantageous in data-intensive applications. density-based hierarchical clustering obtains a detailed description of the structures of clusters and outliers in a dataset through density functions. the resulting hierarchy of these algorithms can be derived from a minimal spanning tree whose edges quantify the maximum density required for the connected data to characterize clusters, given a minimum number of objects, minpts, in a given neighborhood. core-sg is a powerful spanning graph capable of deriving multiple hierarchical solutions with different densities with computational performance far superior to its predecessors. however, density-based algorithms use pairwise similarity calculations, which leads such algorithms to an asymptotic complexity of \(o(n^2)\) for n objects in the dataset, impractical in scenarios with large amounts of data. this article enables hierarchical machine learning models based on density by reducing the computational cost with the help of data bubbles, focusing on clustering and outlier detection. it presents a study of the impact of data summarization on the quality of unsupervised models with multiple densities and the gain in computational performance. we provide scalability for several machine learning methods based on these models to handle large volumes of data without a significant loss in the resulting quality, enabling potential new applications like density-based data stream clustering. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others fast sparse representative tree splitting via local density for large-scale clustering article open access 11 august 2025 cost-effective clustering by aggregating local density peaks chapter © 2023 an efficient multicore cpu implementation of the databionic swarm chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. big data data science data analysis and big data data mining functional clustering machine learning 1 introduction massive amounts of data are generated daily [3, 23]. unsupervised machine learning is advantageous in such scenarios as it finds significant relationships between the data and does not require labels, as the last can be rare or expensive in big data scenarios [2]. among the various unsupervised techniques, data clustering algorithms are worth mentioning, especially those based on density, as they can find clusters and outliers without any information external to the data. furthermore, these algorithms are not bound by format constraints for clusters and do not require a prior definition of several clusters like parametric clustering algorithms. its applications are present in high-impact journals across various research areas, including human behavior [9], chemistry [14], genetics [19], robotics [21], and others. density-based algorithms estimate that objects of a dataset are chainlike connected if there is a minimum number of objects in their neighborhood (known as the smoothing parameter minpts) to characterize a region dense enough to form a cluster. otherwise, objects are considered outliers. given a value of minpts, it is possible to establish the minimum density and similarity necessary to connect any pair of objects in the data set, forming a complete graph g in which each node represents an object and each edge a measure of the density of mutual reach that connects them. for example, the hdbscan [6] algorithm, state-of-the-art in clustering by density, derives a hierarchy between objects and their clusters from the minimum spanning tree (mst) of the complete graph g. this enables transmission of clusters by density, detection, and evaluation of outliers, and other helpful information in various areas such as passing bridge approval [8], cancer research [15], shipping systems [16], among others. later, gertrudes et al. [10] demonstrated that several other unsupervised and semi-supervised algorithms could also be explained from a perspective based on instances or variations of utilizing this mst over g, considering the resulting minimum density of minpts. however, you may need to find the most appropriate value for minpts, which can be done by analyzing multiple msts over g, with each mst requiring an asymptotic computational complexity of \(o(n^2.\ log\ n)\) for a dataset containing n objects. in some cases, it is necessary to analyze different minimum densities, which can be computationally prohibitive given the re-execution of the mst extraction algorithm of the complete graph g. recently, neto et al. [18] proposed the replacement of the graph g by a much smaller graph called core-sg which has as its main characteristic to contain all the msts of g for any value of minpts less than a given maximum value \(minpts_{max}\). an extraction of mst from core-sg has an asymptotic complexity of \(o(n. log\ n)\), with \(n >> minpts_{max}\), which is a very significant reduction compared to the extraction over g. core-sg is very advantageous, as it allows the analysis of results for multiple values of minpts at the computational cost of obtaining a single result since the construction of the core-sg requires the mst over g with density estimated as \(minpts_{max}\). in scenarios where the amount of data generated is massive, using core-sg to extract results with different densities is even more computationally advantageous. however, in these scenarios, its construction is prohibitive due to the intrinsic complexity of density estimation methods using pairwise calculations, which remains \(o(n^2)\). because of their complexity, density-based methods might be limited to smaller datasets. breunig et al. [4] proposed the use of data bubbles, a data summarization method data for hierarchical clustering by density, which makes it possible to preserve the quality of the result obtained and, at the same time, considerably increase the computational performance of the algorithm. furthermore, data bubbles are advantageous when compared to other feature vectors (clustering feature cf) as they were explicitly designed by some of the authors of hdbscan* [6] to be used accurately in the calculation of reachability distances based on density. contributions. this work enables the construction of the core-sg on large datasets, which allows for the direct application of a variety of unsupervised and semi-supervised algorithms in tasks like clustering, outlier detection, classification, and others [10]. density-based algorithms are known for the quality of their results, and their computational cost is traditionally limited to \(o(n^2)\). this work proposes summarization of the data in a significantly smaller amount of data bubbles and, subsequently, the construction of the core-sg on the data bubbles. once built, core-sg can be used to obtain hierarchical results extracted with different values for its density parameter with computational performance o(n), which enables its application in large volumes of data. far as the authors are aware, this is the only work in the literature that allows the construction of multiple hierarchical models with different density estimates from a single model of summarized data. the present work analyzes the impact of summarization on the quality and performance of hierarchical models based on densities to measure to what extent it can obtain significant results without degenerating these models. the presented results, in addition to benefiting a variety of existing algorithms in the literature and organized in [10], also enables the potential for new density-based applications that include scenarios with continuous data streams. 2 related works data summarization has been successfully used to enable machine learning tasks on large data sets or data streams, which are potentially infinite. for hierarchical clustering, one of the best-known algorithms is birch [24], which builds and maintains a hierarchy of structures called clustering features (cf). each cf stores the number of summarized data, the linear and square sum of the data. the birch algorithm structure of cfs forms a tree of subclusters and is built incrementally, as objects are inserted sequentially in the cf that best represents it. parameters define when new cfs are added to the tree to increase the hierarchy. using cfs statistics improves the algorithm’s performance without significantly impacting the quality of the result. traditional cf summarize data in such a way as to lose information about its original placement and its nearest neighbors, creating a distortion when represented only by its central objects. this loss of information impacts the quality of density-based algorithms. to solve this problem, data bubbles (dbs) [4] were proposed to add information to cf necessary for the calculation of density estimates in the region covered by db. this information is a “representative” vector to indicate the placement of db; its radius and estimation of average distances between objects within the db, essential for density calculations. dbs have been successfully used in a variety of jobs [13, 20, 26] related to new summarization approaches and density calculations. additionally, dbs were proposed for application in hierarchical algorithms, as was done with the optics [1] in [5] where the use of dbs obtained a great increase in computational performance, maintaining the quality of the final result. the potential of applying dbs for analysis and visualization of large databases is also discussed in [5]. a characteristic of dbs is that they are incremental, which means they can be updated with new data. this characteristic is explored in [17], where dbs are used to provide incrementality for the method that the authors call incremental and effective data summarization (ieds), designed for large volumes of data dynamic, where new data is continually add in a hierarchical clustering tree. the ieds method speeds up the incremental construction of summarized data using triangular inequalities for inserting and deleting dbs. furthermore, the algorithm analyzes the dbs to keep those with high-quality compressed data. however, when initializing the summarization, defining which data will be summarized in each db is necessary. this choice can generate different and unsatisfactory results, mainly if data separated by low-density regions are summarized in the same db. to avoid the performance of the algorithm not so sensitive to the compression rate and the location of the seeds that initialize the dbs, a method of data summarization and initialization of dbs is proposed in [25]. in it, a sampling based on density for selecting a subset of data to be used in the construction of dbs is used, thus avoiding the choice of data separated by regions of low density for the same db. recently, dbs have been used as summary frameworks along with framework mapreduce for data summarization and [20] scalability. they were applied to enable the scalable version of hdbscan* [6] to summarize data (and even other dbs) into a hierarchy of clusters by density. this approach is similar to the one proposed in this article, as it enables the application of a hierarchical algorithm based on density over large amounts of data. however, unlike the approach used in [6], our work considers the summarization of a structure that results in multiple hierarchies with different densities and not just a single one, which is quite advantageous when one does not have prior knowledge of the fittest values of the density parameters. 3 proposed method our proposal is divided into two steps: summarizing the data and obtaining hierarchies with multiple densities. the original db [4] uses the parameter \(\epsilon \), which is incompatible with hdbscan* and core-sg. therefore, we propose an adaptation and new definitions to enable the joint application of db and core-sg. 3.1 density-based summarization given a dataset x with n objects of d dimensions, an integer value for the density parameter minpts that defines the number of neighbors needed for an object to be considered core, our method starts by applying a o(n) computational complexity method to divide x into m subsets, where \(m << n\). clustering into m in subsets can be adopted in this step or a guided initialization [25]. definition 1 data bubbles: the data subset \(x_i \subseteq x\) summarized in a db which is defined as a tuple \(b_i = (rep_i, n_i, extent_i, nndist_i)\), where \(rep_i\) is the position of the representative of \(x_i \), \(n_i\) is the \(|x_i|\) cardinality, \(extent_i\) is a natural number that defines an extended radius around the \(rep_i\) that contains most objects of \(x_i\), and \(nndist_i\) is a function that given an integer value \(k \le minpts < n_i\) that denotes the estimated average distance of the k-nearest neighbor within the set of objects \(x_i\). in a euclidean space, the statistics linear and quadratic sums of the objects in \(x_i\), \(ls_i\), and \(ss_i\), respectively, can be used to calculate the characteristics of \(b_i\). with these statistics, \(rep_i=\frac{ls_i}{n_i}\), \(extent_i=\sqrt{\frac{2 \cdot n_i \cdot ss_i 2 \cdot ls^2_i}{n_i \cdot (n_i 1)}}\) is calculated and the expected distance from the k-nearest neighbor is \(nndist_i(k) = (\frac{k}{n_i})^{\frac{1}{d}} \cdot extension_i\). definition 2 distance between dbs: if \(b_i\) and \(b_j\) are two dbs, the direct distance between \(b_i\) and \(b_j\) is defined as: $$\begin{aligned} dist(b_i, b_j)= \left\{ \begin{array}{c} 0,\; if \;b_i=b_j, \\ \\ dist(rep_i, rep_j)-(extend_i + extend_j) \\ + nndist_i(1) + nndist_j(1), \\ \text {if } dist(rep_i, rep_j)-(extend_i + extend_j) \ge 0, \\ \\ max(nndist_i(1), nndist_j(1)),\ \text {otherwise}. \end{array} \right. \end{aligned}$$ (1) in other words, if \(b_i = b_j\), their distance will be zero. if they do not overlap, their distance is given by the distance between their representatives minus their radii plus their expected distances to the nearest neighbor. finally, if the dbs overlap, the distance is the maximum between their expected nearest neighbor distances. the described cases are illustrated in fig. 1. fig. 1. distance between data bubbles [4]. full size image definition 3 core distance from a db: given a value of \(minpts < n\), the function \(nn(b_i,k)\) that returns the k-th closest db to \(b_i\) or \(b_i\) if \(k=0\), a function that returns the minimum number of neighbors of \(b_i\) needed to summarize minpts objects, i.e., \(snn(b_i,minpts) = \arg \min _k \sum _{j=0}^k n_j \ge minpts \,|\) \( b_j = nn(b_i,j)\), the core distance of \(b_i\) is: $$\begin{aligned} core_{minpts}(b_{i})= \left\{ \begin{array}{c} nndist_i(minpts),\; if \, snn(b_i,minpts) = 0 \\ \\ dist(rep_i, rep_k) extend_k + \\ nndist_k(minpts \sum _{l=0}^{k-1} n_l|b_l = nn(b_i,l)), \text {otherwise} \end{array} \right. \end{aligned}$$ (2) where \(b_k = nn(b_i,snn(b_i,minpts))\) is the kth nearest neighbor of \(b_i\). the core distance is the minimum distance from a db that summarizes at least minpts objects. it is important to point out that \(core_{a}(b_{i}) \ge core_{b}(b_{i}) \; \forall \; a \ge b\), i.e., increasing the value of minpts cannot reduce the core distance of a db. we can calculate the reachability distances used in density estimates using the distance between two dbs and their core distances. definition 4 mutual reachability distance (mrd) between dbs: given the dbs \(b_i\) and \(b_j\), the minimum number of objects minpts, the mutual reachability distance between the bubbles \(b_i\) and \(b_j\), i.e., the distance that make them density reachable within each other is: $$\begin{aligned} mrd_{minpts}(b_i, b_j)=max(core_{minpts}(b_i), core_{minpts}(b_j), dist(b_i, b_j)) \end{aligned}$$ (3) 3.2 core-ssg: core summarized spanning graph the mrd reflects the minimum density needed to connect two dbs. with it, we can estimate the distance that connects all pairs of dbs in the data set by density, forming a virtual graph g that represents all the density relationships between the data, given a minimum value minpts of points needed to obtain the desired density. definition 5 mutual reachability graph g: given a minimum density value minpts, the \(g_{minpts} = (v, e)\) is a complete graph in which the set of vertices v represent the dbs and the set of edges is defined as \(e = \{e(b_i, b_j)\; | \; b_i, b_j \in v \text { with weights } w(e) = mrd_{minpts}(b_i, b_j)\}\). the main idea behind density-based hierarchical machine learning algorithms is to extract a minimal graph that connects all objects in the dataset by density [6, 10]. that is, directly or indirectly, these algorithms obtain the \(mst_{minpts}\) of the graph \(g_{minpts}\) and, with it, build a hierarchy that allows obtaining clusters, identification of outliers and applications. however, different minpts values tend to derive different results, which would require multiple extractions of msts at a high computational cost. to avoid this, given an upper bound for minpts defined here as \(minpts_{max}\), we can replace the complete graph g with \(o(n^2)\) edges by the graph core-sg\( _{minpts_{max}}\) with o(n) edges when applied to a dataset with n objects, if \(minpts \le minpts_{max} << n\) [18], which is a natural assumption. however, obtaining the core-sg of g over n objects using a traditional mst extraction algorithm is \(o(n^2 \cdot log \ n)\), prohibitive for large values of n. in the present work, we summarize the data into dbs and consider the complete graph g over m dbs, which makes the cost to build the core-sg\(_{minpts_{max}}\) \(o(m^2 \cdot log \ m)\) and the cost of each extraction of \(mst_{minpts}\) \(o(m\cdot log \ m)\). in a scenario with a large volume of data, it is natural to consider that \(minpts_{max} << m << n\) and, therefore, to limit the necessary computational cost for our proposal to o(n). in scenarios where this assumption is invalid, the data can be used normally, and summarization is unnecessary. as it is applied over dbs, the summarized core-sg will be referred to here as core summarized spanning graph (core-ssg) and has definitions and properties different from its original version [18]. next, we will present its new concepts and proprieties. definition 6 minimum nearest neighbor graph \(mnng_{minpts}\): given a minimum density value minpts, the graph \(mnng_{minpts} = (v, e)\) has the generated dbs as a set of vertices v over the data and the set of edges \(e = \{e(b_i, b_j)\; | \; b_i, b_j \in v \wedge b_j = nn(b_i,k) \ \forall k = [0,snn(b_i,minpts)] \}\). this graph connects the dbs to guarantee a minimum number of minpts connected summarized objects. we see that \(mnng_{minpts}\) has two important properties, as shown in lemmas 1 and 2. lemma 1: \(\forall \ a \le b\), \(mnng_{a} \subseteq mnng_{b}\). proof: since \(mnng_{a} = (v_a, e_a)\), \(mnng_{b} = (v_b, e_b)\) and \(v_a = v_b = v\), it is necessary to prove that \(e_a \subseteq e_b\) so that lemma 1 be true. assuming that \(e_a \not \subseteq e_b\), there must be an edge in \(e_a\) that is not in \(e_b\). by definition, \(snn(b_i,a) \le snn(b_i,b) \ \forall \ a \le b, b_i \in v\). therefore, the set of integer values of the interval \([0,snn(b_i,a)]\) is contained in \([0,snn(b_i,b)]\), which makes false the statement that there is an edge \(e_a\) that is not in \(e_b\). lemma 2: \(\forall \ a \le b: mrd_a(b_i,b_j) = max(core_a(b_i), core_a(b_j)) \rightarrow e(b_i, b_j) \in mnng_{b}\). proof: assuming that \(mrd_a(b_i,b_j) = \max (core_a(b_i), core_a(b_j))\), given the definition of the mrd in definition 4, it follows that \(core_a(b_i) \ge dist(b_i,b_j)\) or \(core_a(b_j) \ge dist(b_i,b_j)\), or both. hence, at least one of these dbs must be in the neighborhood of the other, i.e., \(b_j = nn(b_i,k), k = [0,snn(b_i,a)]\) or/and \(b_i = nn(b_j,k), k = [0,snn(b_j,a)]\) must hold and implies that \(e(b_i,b_j) \in mnng_a\). since \(a \le b\), it follows by lemma 1 that \(mnng_{a} \subseteq mnng_{b}\) and, accordingly, \(e(b_i,b_j) \in mnng_{b}\). definition 7 summarized core-sg (core-ssg): given a maximum value for the minimum density parameter \(minpts_{max}\), the core-ssg\(_{minpts_{max}} \) = \( mnng_{minpts_{max}} \cup mst_{minpts_{max }} \) is a non-directional graph that contains the edges of all \(mst_{minpts}\) of \(g_{minpts}\) for each value of \(minpts < minpts_{max}\). this feature allows replacing g by core-ssg\(_{minpts_{max}}\) in the process of extracting a \(mst_{minpts}\), simply replacing the weight of the edges \(e(b_i, b_j)\) in core-ssg\(_{minpts_{max}}\) by \(w(e) = mrd_{minpts}(b_i, b_j)\) to \(minpts < minpts_{max}\). theorem 1: for two values p and q for the density parameter minpts, let \(m_p\) be the set of all possible msts of \(g_p\), and let \(mst_q\) be any mst of \(g_q \), so \(\forall \ p < q\): \(\exists mst_p \in m_p\): \(mst_p \subseteq mst_q \cup mnng_q\). proof: let’s consider an edge \(e(b_i,b_j)\) in a \(mst_{p}^{*} \in m_p\) that connects two subgraphs i and j. the edge weight \(e(b_i,b_j)\) is defined by \(mrd_{p}(b_i,b_j)\), which is the maximum between \(dist(b_i, b_j)\) and their core distances \(core_p(b_i) \) and \(core_p(b_j)\), according to definition 4. due to the cutting property of msts, \(e(b_i,b_j)\) must have the lowest weight among all the edges that connect the sets i and j in \(g_p\). there are two scenarios: (1) \(e(b_i,b_j) \in mst_q\) (2) \(e(b_i,b_j) \notin mst_q\) in scenario (1), since \(e(b_i,b_j) \in mst_q\), it trivially follows that \(e(b_i,b_j) \in mst_q \cup mnng_q\). in scenario (2), \(mst_q\) must have a different edge \(e(c_i,c_j)\) that connects the dbs \(c_i\) and \(c_j\) that are contained in the graphs i and j, respectively, having the lowest weight among the edges that connect these graphs (msts are connected graphs). thus, the following two statements must be true (i) \(mrd_p(b_i, b_j) \le \) \(mrd_p(c_i, c_j)\) (otherwise \(e(b_i, b_j)\) could not be the edge with the smallest weight connecting i and j in \(mst_p^{*}\)) and (ii) \(mrd_q(b_i, b_j) \ge \) \(mrd_q (c_i, c_j)\) (otherwise \(e(c_i, c_j)\) could not be the least weighted edge connecting i and j in \(mst_q\)). we saw in definition 3 that by increasing the value of minpts, the core distances \(core_{minpts}(\cdot )\) can only grow and the distance \(dist(\cdot ,\cdot )\) between two dbs is constant. therefore, the weight of an edge, given by \(mrd_{minpts}(\cdot ,\cdot )\), cannot decrease when the value of minpts increases. consequently, the mutual reachability distances (edge weights), defined in eq. 3, can only remain the same or increase. therefore, when minpts increases from p to q (\(p < q\) by assumption in the theorem), one of the following must be true: (a) \(mrd_p(b_i, b_j) < mrd_q(b_i, b_j)\) (b) \(mrd_p(b_i, b_j) = mrd_q(b_i, b_j)\) if (a) is true, as the distance \(dist(\cdot ,\cdot )\) between two dbs does not depend on minpts, the mutual reachability distance \(mrd_q(b_i,b_j)\) must be determined by the core distance of \(b_i\) or \(b_j\), i.e., \(mrd_q(b_i, b_j) = max(core_q(b_i), core_q(b_j))\). from lemma 2 with \(minpts = q\) it then follows that \(e(b_i, b_j) \in mnng_q\) and therefore \(e(b_i, b_j) \in mst_q \cup mnng_q\). if (b) is true, then from statements (i) and (ii) above, we have \(mrd_q (c_i, c_j) \le mrd_q(b_i, b_j)\) = \(mrd_p(b_i, b_j) \le mrd_p(c_i , c_j)\). however, since \(p < q\) and \(mrd_{minpts}\) cannot decrease when minpts increases, it follows that \(mrd_q(c_i, c_j) = mrd_p(b_i, b_j) = mrd_p(c_i, c_j)\). in this case, replacing \(e(b_i, b_j)\) in \(mst_p^{*}\) with \(e(c_i, c_j)\) will result in another \(mst_p{'} \in m_p\), as the total weight of \(mst_p ^{*}\) and \(mst_p{'}\) are the same. \(mst_p^{*}\) includes the edge \(e(c_i, c_j)\) and by assumption \(e(c_i, c_j) \in mst_q\), and therefore \(e(c_i, c_j) \in mst_q \cup mnng_q\). it follows that, given a mst of \(g_q\), \(mst_q\), and a mst of \(g_p\), \(mst_p^{*} \in m_p\), with \(q > p\), we can construct a weighted graph \(mst_p{'}\) as follows: (1) include all edges and edge weights \(e(b_i, b_j) \in mst_p\) that belong to \(mst_q \cup mnng_q\) (according to scenarios (1) or (2)-a); (2) replace all edges \(e(b_i, b_j) \in mst_i\) that do not belong in \(mst_q \cup mnng_q\) with edges \(e(c_i, c_j)\) that must exist in \(mst_q \cup mnng_q\)(as scenario (2)-b), connecting the same two subsets of points as \(e(b_i, b_j)\) and having the same edge weight as \(e(b_i, b_j)\). this graph \(mst_p{'}\) is a mst of \(g_p\), that is, \(mst_p{'} \in m_p\) and \(mst_p{'} \subseteq mst_q \cup mnng_q\). thus there exists a \(mst_p \in m_p\) such that \(mst_p \subseteq mst_q \cup mnng_q\). theorem 1 supports that the complete graph \(g_{minpts_{max}}\) can be replaced by \(mst_{minpts_{max}} \cup mnng_{minpts}\) when calculating additional results of msts with \(minpts < minpts_{max}\), with potential for greater execution time and memory savings. 4 experiments in this section, we evaluate the use of core-ssg for the clustering task to illustrate its utility, although it can be used in various applications [10]. the core-ssgfootnote 1 was implemented in python/cython, and the experiments were performed on computer with 16gb of ram and 8 processing cores. the experimental evaluations are divided into three parts: assessment of the quality of the hierarchies and extracted partitions from core-ssg; the clustering solutions from core-sg and core-ssg are compared; an analysis of the runtime and speed-up of the compared algorithms is made. 4.1 evaluation of the quality loss resulted from core-ssg summarization three bi-dimensional datasets with 38k, 43k, and 49k objects were synthesized for the quality evaluation experiments and divided into 13, 10, and 12 clusters with noise. after that, each dataset was divided into subsets of the same volume so that the number of subsets totalized n/10, i.e., 10% of the datasets’ sizes. then, the dbs were created over these subsets, and a core-ssg with \(minpts = 200\) was built. from the core-ssg\(_{200}\), we extracted density-based msts for minpts ranging from 200 to 2 in decreasing steps of 2. using the extracted msts, we built one hundred hdbscan* clustering solutions and calculated their pairwise similarity using the hai index [12]. based on their similarities, we have chosen four of the most representative clustering solutions, i.e., we selected four solutions that are most similar to the others and different within themselves, according to the hai index. these four hierarchical clustering results are presented in the clustering trees of fig. 2, where the most prominent clusters, according to the fosc framework [7], are circled in red. the fosc framework uses the concept of excess of mass to extract the most prominent partition from a hierarchical clustering solution. as done in the hdbscan* article, we assumed the minimum cluster size value equals minpts. fig. 2. cluster hierarchies of core-ssg’s most representative solutions. full size image the clustering hierarchies presented in fig. 2 accurately reflect the behavior of hdbscan* clustering for a range of minpts values. for small minpts values, the hierarchy is full of small, highly detailed clusters that are hard to visualize but can be analyzed using cluster validation indexes [22]. in contrast, as minpts values increase, the hierarchies comprise a smaller number of more robust clusters that reflect denser structures. the results perfectly mimic what is expected from density-based clustering algorithms. the difference resides in the fact that core-ssg does not have the hierarchical structure of the summarized data, i.e., within the dbs. in most cases, the summarized data has a level of detail of minor importance for large volumes of data. however, if the user requires the structure within the dbs, the core-sg can be applied over the summarized data of each dbs independently, and the results (which are graphs) combined. if the summarized data is still extensive, the core-ssg can be applied recursively. although a hierarchy of clusters is a more complete and richer way of representing the data structure, they can be too complex and extensive for very large datasets. in such scenarios, the automatic extraction of a partition of clusters may be preferred. fosc is an excellent [7] choice for this job. in fig. 3, partitions selected (circled) by fosc in fig. 2 are presented. it is possible to note that fosc successfully extracted prominent clusters in the datasets and correctly separated noise. additionally, different minpts values resulted in different cluster selections, where small values favor the selection of broader clusters that resides in the higher levels of the hierarchy, as they are more present in the hierarchy than their several smaller nested child clusters. in contrast, a higher minpts value prevents smaller cluster structures at the bottom of the hierarchy, favoring the selection of denser clusters nested into the broader ones. the results explicit the importance of the analysis of multiple results with different minpts values, as all of them have the potential to be considered “the best” and validate the main objective of the present work. furthermore, summarizing the data did not impact the choice of clusters made by fosc, as the framework has detected the most prominent ones seamlessly for different minpts values. fig. 3. cluster chart by representative minpts in datasets. full size image 4.2 comparison between core-sg and core-ssg partitions as the hierarchies of core-sg and core-ssg are not comparable due to summarization, we compared the fosc extracted partitions between core-sg and core-ssg to check if the prominent clusters are similar. we used the adjusted rand index (ari) [11] to obtain the similarities between the extracted partitions where one indicates that the partitions are identical and zero is adjusted to reflect randomness. figure 4 presents the ari heatmap resulting from the pairwise comparison among partitions from core-sg and core-ssg with different minpts values. core-sg partitions are presented in the horizontal axis, and core-ssg partitions are in the vertical axis. the heatmaps show that, for small minpts values, the core-sg has a much more detailed partition, and core-ssg tends to unify small clusters, which is expected because the information within the dbs is summarized and cannot be split. that is reflected in the darker area over small minpts values. for large datasets, small values of minpts tend to over-fragment the clustering structure into a hard-to-interpret solution. according to the ari, for medium and high values of minpts, most heatmaps present bright areas that reflect a similarity over 0.85. this result shows that the main cluster structure of the data is present in the results of both graphs, sometimes for distinct values of minpts, which also support the importance of a multi-density analysis of the dataset, the objective of this paper. fig. 4. comparing partitions with and without dbs in datasets. full size image 4.3 runtime analysis to assess the computational performance of core-ssg in relation to core-sg, we created datasets with a variety of sizes, dimensions, and different levels of summarization, as described in table 1. to evaluate each characteristic separately, default values were fixed and presented in bold in table 1. additionally, we also considered a variety of \(minpts_{max}\) values used to build the graphs. the runtime necessary to obtain the graphs, core-sg and core-ssg, added to the time needed for the hierarchical density solutions extraction, are presented in fig. 5. table 1. dataset characteristics and experimental setup.full size table fig. 5. runtime to build the graphs and extract the hierarchical density solutions for core-sg and core-ssg. full size image figure 5 (a) shows that the parameter minpts has a linear correlation with the processing time needed to run the experiment for both graphs, but core-ssg presented a speedup close to 20 times better than core-sg. considering the dataset size n and \(m << n\), fig. 5 (b) clearly shows the quadratic behavior of core-sg as the dataset size increases up to one million and how the computational cost is amortized when using dbs, i.e., for core-ssg. the present results support the main research hypothesis of this work: data summarization enables scalability for density-based machine learning solutions, clustering algorithms in particular. summarization also had an impact on the algorithms’ total runtime when the data’s dimensionality increases, as shown in fig. 5 (c), with speedups up to 15 times. last but not least, the total time needed for the data summarization, presented in fig. 5 (d), reflects the linear increase of the computational time related to the proportion of dbs used, which reflects the expected behavior for our proposal and shows that summarization requires a small fraction of the total runtime. 5 conclusion and future work the main goal of the present research, enabling the use of core-sg over larger datasets, was achieved as we showed that a single core-ssg permits the accurate extraction of multiple density-based hierarchical solutions for different densities parameters at a subquadratic computational cost. that was shown theoretically, with the proof that core-ssg holds the main characteristics of core-sg, and empirically through experiments. moreover, the experiments also supported the importance of the analysis of multiple density-based results, as datasets can have nested density structures with different characteristics, each structure with its relative importance, which justifies the application of core-sg in a pleura of machine learning tasks [10], including clustering. the most prominent clusters (according to the fosc framework) present in the original version of core-sg are also present in our core-ssg solution, obtaining a significant computational cost reduction. the downside of our approach is the loss of small density-based structures summarized inside the dbs, which were rarely considered prominent by the fosc framework as they connect a very small fraction of the dataset. thus, our solution is indicated for large volumes of data. even so, if these small structures are indeed important for the application, we recommend the use of core-sg (or core-ssg) over the data within each dbs, followed by the combination of the resulted graphs. in addition to the direct benefits of our findings for the machine learning applications previously referred in [10], we believe that core-ssg has much potential to innovate in scenarios with continuous data streams. that is because parameters of density-based applications for data streams usually must be set (fixed) as an algorithm input, which may be hard to define and require changes during the evolution of the data (concept drift). core-ssg enables the extraction of different density-based parameters, even multiple results, from a single induced model, which can be analyzed online to adapt the model for different density levels and concept drift. indeed, this is a future work worth exploring. notes 1.implementation in: https://github.com/natanaelbatista99/core-ssg. references ankerst, m., breunig, m.m., kriegel, h.p., sander, j.: optics: ordering points to identify the clustering structure. sigmod rec. 28(2), 49–60 (1999) article  google scholar  barlow, h.: unsupervised learning. neural comput. 1(3), 295–311 (1989) article  google scholar  blazquez, d., domenech, j.: big data sources and methods for social and economic analyses. technol. forecast. soc. chang. 130, 99–113 (2018) article  google scholar  breunig, m.m., kriegel, h.p., kröger, p., sander, j.: data bubbles: quality preserving performance boosting for hierarchical clustering. in: proceedings of the 2001 acm sigmod international conference on management of data, pp. 79–90 (2001) google scholar  breunig, m.m., kriegel, h.-p., sander, j.: fast hierarchical clustering based on compressed data and optics. in: zighed, d.a., komorowski, j., żytkow, j. (eds.) pkdd 2000. lncs (lnai), vol. 1910, pp. 232–242. springer, heidelberg (2000). https://doi.org/10.1007/3-540-45372-5_23 chapter  google scholar  campello, r.j.g.b., moulavi, d., zimek, a., sander, j.: hierarchical density estimates for data clustering, visualization, and outlier detection. acm trans. knowl. discov. data 10(1), 1–51 (2015) article  google scholar  campello, r.j., moulavi, d., zimek, a., sander, j.: a framework for semi-supervised and unsupervised optimal extraction of clusters from hierarchies. data min. knowl. disc. 27, 344–371 (2013) article  mathscinet  math  google scholar  cheema, p., alamdari, m.m., chang, k., kim, c., sugiyama, m.: a drive-by bridge inspection framework using non-parametric clusters over projected data manifolds. mech. syst. signal process. 180, 109401 (2022) article  google scholar  djonlagic, i., et al.: macro and micro sleep architecture and cognitive performance in older adults. nat. hum. behav. 5(1), 123–145 (2021) article  google scholar  gertrudes, j.c., zimek, a., sander, j., campello, r.j.g.b.: a unified view of density-based methods for semi-supervised clustering and classification. data min. knowl. discov. 33(6), 1894–1952 (2019) article  mathscinet  math  google scholar  hubert, l., arabie, p.: comparing partitions. j. classif. 2, 193–218 (1985) article  math  google scholar  johnson, d., xiong, c., gao, j., corso, j.: comprehensive cross-hierarchy cluster agreement evaluation. acm tkdd. 10, 1–51 (2013) google scholar  liu, b., shi, y., wang, z., wang, w., shi, b.: dynamic incremental data summarization for hierarchical clustering. in: yu, j.x., kitsuregawa, m., leong, h.v. (eds.) waim 2006. lncs, vol. 4016, pp. 410–421. springer, heidelberg (2006). https://doi.org/10.1007/11775300_35 chapter  google scholar  miccio, l.a., schwartz, g.a.: mapping chemical structure-glass transition temperature relationship through artificial intelligence. macromolecules 54(4), 1811–1817 (2021) article  google scholar  minussi, d.c., et al.: breast tumours maintain a reservoir of subclonal diversity during expansion. nature 592(7853), 302–308 (2021) article  google scholar  murray, b., perera, l.p.: an ais-based deep learning framework for regional ship behavior prediction. reliab. eng. syst. saf. 215, 107819 (2021) article  google scholar  nassar, s., sander, j., cheng, c.: incremental and effective data summarization for dynamic hierarchical clustering. in: proceedings of the 2004 acm sigmod international conference on management of data, pp. 467–478. sigmod 2004, association for computing machinery, new york, ny, usa (2004) google scholar  neto, a.c.a., naldi, m.c., campello, r.j.g.b., sander, j.: core-sg: efficient computation of multiple msts for density-based methods. in: 2022 ieee 38th international conference on data engineering (icde), pp. 951–964 (2022) google scholar  norman, t.m., et al.: exploring genetic interaction manifolds constructed from rich single-cell phenotypes. science 365(6455), 786–793 (2019) article  google scholar  dos santos, j.a., syed, t.i., naldi, m.c., campello, r.j., sander, j.: hierarchical density-based clustering using mapreduce. ieee trans. big data 7(1), 102–114 (2019) article  google scholar  savoie, w., et al.: a robot made of robots: emergent transport and control of a smarticle ensemble. sci. robot. 4(34), eaax4316 (2019) article  google scholar  vendramin, l., campello, r.j., hruschka, e.r.: relative clustering validity criteria: a comparative overview. statist. anal. data mining asa data sci. j. 3(4), 209–235 (2010) article  mathscinet  math  google scholar  zerhari, b., lahcen, a.a., mouline, s.: big data clustering: algorithms and challenges. in: proceedings of international conference on big data, cloud and applications (bdca-5) (2015) google scholar  zhang, t., ramakrishnan, r., livny, m.: birch: an efficient data clustering method for very large databases. sigmod rec. 25(2), 103–114 (1996) article  google scholar  zhang, y., cheung, y., liu, y.: quality preserved data summarization for fast hierarchical clustering. in: 2016 international joint conference on neural networks (ijcnn), pp. 4139–4146 (2016) google scholar  zhou, j., sander, j.: data bubbles for non-vector data: speeding-up hierarchical clustering in arbitrary metric spaces. in: freytag, j.c., lockemann, p., abiteboul, s., carey, m., selinger, p., heuer, a. (eds.) proc. 2003 vldb conf., pp. 452–463. morgan kaufmann, san francisco (2003) chapter  google scholar  download references acknowledgement the authors would like to thank cnpq mai/dai program, capes, and fapesp for their financial support. we also thank prof. jörg sander for his insight and support. author information authors and affiliations department of computing science, federal university of são carlos (ufscar) rod, washington luís, km 235 pbox 676, são carlos, sp, brazil natanael f. dacioli batista, bruno leonel nunes & murilo coelho naldi authors natanael f. dacioli batistaview author publications search author on:pubmed google scholar bruno leonel nunesview author publications search author on:pubmed google scholar murilo coelho naldiview author publications search author on:pubmed google scholar corresponding author correspondence to natanael f. dacioli batista . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper batista, n.f.d., nunes, b.l., naldi, m.c. (2023). efficient density-based models for multiple machine learning solutions over large datasets. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_4 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords density-based data models unsupervised learning clustering data sumarization big data publish with us policies and ethics profiles natanael f. dacioli batista view author profile murilo coelho naldi view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a monte carlo algorithm for time-constrained general game playing | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a monte carlo algorithm for time-constrained general game playing conference paper first online: 12 october 2023 pp 97–111 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a monte carlo algorithm for time-constrained general game playing download book pdf download book epub victor scherer putrich9, anderson rocha tavares10 & felipe meneguzzi9,11  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 527 accesses abstract general game playing (ggp) is a challenging domain for ai agents, as it requires them to play diverse games without prior knowledge. in this paper, we develop a strategy to improve move suggestions in time-constrained ggp settings. this strategy consists of a hybrid version of uct that combines sequential halving and , favoring information acquisition in the root node, rather than overspend time on the most rewarding actions. empirical evaluation using a ggp competition scheme from the ludii framework shows that our strategy improves the average payoff over the entire competition set of games. moreover, our agent makes better use of extended time budgets, when available. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others testing hybrid computational intelligence algorithms for general game playing chapter © 2020 self-adaptive mcts for general video game playing chapter © 2018 towards human-competitive game playing for complex board games with genetic programming chapter © 2016 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence continuous optimization general economical equilibrium game theory games studies optimization 1 introduction general game playing (ggp) is a research area focused on developing intelligent agents capable of playing a wide variety of games without prior knowledge of any specific game being played [5]. ggp agents receive the rules of potentially unknown games and must play them effectively. this prevents the creation of game-specific heuristics. the upper confidence for trees (uct) [7] algorithm has been effectively utilized in ggp environments. uct is based on building a search tree using monte carlo tree search (mcts). mcts employs monte carlo simulations to iteratively build a game tree. uct guarantees asymptotic optimal convergence. however, it may spend too much time on the most promising choice so far, instead of attempting other less explored options to potentially find a better one. this results in uct taking too long to produce high-quality recommendations in certain scenarios. a significant challenge in ggp is designing algorithms that can efficiently find solutions in a timely manner, particularly in competitive contexts, where the time required to find a solution is critical to the agent’s performance. in this paper, we tackle the problem of ggp with scarce time resources. specifically, we focus on the following question: is it uct the best option for ggp environments with strict time constraints? in response, we develop uct\(_{{\sqrt{\text {sh}}}}\)footnote 1 (presented at sect. 4). our algorithm is based on a hybrid scheme for building game-trees though mcts [15], which is presented at sect. 3. uct\(_{{\sqrt{\text {sh}}}}\) aims to be more exploratory in the root node than uct, avoiding repeatedly probing the currently most promising action, and using the budget to find potentially better alternatives. we conduct two distinct experiments (sect. 5). first we present the prize box selection experiment, which is a simplified multi armed bandit (mab) problem to compare how selection polices allocate their resources under scenarios with high and low reward variance. the second experiment aims to measure the agents’ performance relative to uct under time constraints. for this purpose, we use the ludii ggp environment [12]. specifically, we use the kilothon tournament, one of the tracks of ludii’s ggp competitionfootnote 2. such international competitions have a crucial role in motivating ggp research [14]. the main contributions of this paper are as follows: (i) the uct\(_{{\sqrt{\text {sh}}}}\) algorithm, a new decision-making method that attempts to use less budget on the greedy choice to favour less-explored ones; (ii) the clock bonus time (cbt) approach, which enhances the estimation of thinking time in a ggp environment; (iii) the prize box selection experiment, which highlights the resilience of uct\(_{{\sqrt{\text {sh}}}}\) allocation criteria on decision-making problems with high and low variance, compared to \(\text {ucb}_1\) and other selection policies (examined in our work at sect. 2 and 3); and (iv) empirical evidence of the improved performance of uct\(_{{\sqrt{\text {sh}}}}\) over uct on a ggp environment, suggesting its effectiveness as a selection policy. 2 monte carlo methods monte carlo techniques employ random sampling via simulations to acquire information to address problems that are otherwise intractable. in game-playing, monte carlo methods can be used to evaluate a game-tree node by computing the expected outcome of its actions by sampling a sufficient large number of random completions of a game, also called playouts. this section presents useful concepts regarding monte carlo methods, that are useful throughout the remainder of the paper. 2.1 regret on bandit problem multi-armed bandit (mab) problems [1] represent a category of decision-making scenarios in which the outcomes of chosen actions are unknown. imagine a casino slot machine (bandit) with k distinct arms, each with its own reward and probability of winning. the gambler’s objective is to plan a strategy that maximizes their overall profit in a previously unknown bandit. the challenge lies in determining the number of times to pull each arm to maximize returns (exploitation) while learning rewards and probability distributions (exploration). one way to measure performance in the multi-armed bandit problem is via regret, defined as the difference in the reward obtained from the pulled versus the optimal arm. we use two important measures of regret adapted from the definitions of pepels [10] and bubeck [3]. specifically, we use cumulative and simple regret from definitions 1 and 2. definition 1 cumulative regret is the regret accumulated over a number n of arm pulls. let \(\mu ^\star \) be the expected reward of the best arm, \(\mu _i\) be the expected reward of the arm pulled in the i-th trial. then, the cumulative regret \(r_n\) can be defined as: $$\begin{aligned} r_n &= \sum _{i=1}^{n} (\mu ^\star \mu _i) \end{aligned}$$ (1) an alternative experimental setup involves just finding the optimal arm without the need to maximize the reward during this process. then, the gambler makes a final pull in this arm. simple regret then measures the sub-optimality of this final pull, as per definition 2. definition 2 simple regret is the difference between the expected reward of the best arm \(\mu ^\star \) and the expected reward of the arm of the final pull \(\mu _n\): $$\begin{aligned} r_n = \mu ^\star \mu _n \end{aligned}$$ (2) bubeck et al. [3] showed that there is a trade-off between minimizing cumulative and simple regret. specifically, they found that a smaller upper bound on cumulative regret (\(r_n\)) leads to a larger lower bound on simple regret (\(r_n\)), meaning that when an algorithm performs well in terms of cumulative regret (worst-case scenario), it is likely to have a higher minimum simple regret (best-case scenario), and vice-versa. 2.2 upper confidence bound upper confidence bound (ucb) [1] is a selection policy in mab problems and mcts. the policy optimizes cumulative regret over time at a logarithmic rate over the number of trials performed. a widely adopted variant, \(\text {ucb}_1\), is favored for its simplicity and its ability to consistently deliver robust performance outcomes. \(\text {ucb}_1\) establishes a statistical confidence interval for the estimated mean action-value. the \(\text {ucb}_1\) equation, adapted from auer et al. [1], is presented in eq. 3: $$\begin{aligned} \text {ucb}_1(s,a) = q_{s,a} + \sqrt{\dfrac{2\ln {n_s}}{n_{s,a}}} . \end{aligned}$$ (3) here, \(q_{s,a}\) is the mean reward from action a, when selected from state s. \(n_s\) is the number of visits on state s, while \(n_{s,a}\) is the number of times action a has been selected in state s. the square-root term quantifies the upper-confidence bound, i.e., the uncertainty in the estimate of taking action a on state s. it serves as a bonus to encourage exploring less visited actions. \(\text {ucb}_1\) offers a desirable property: the discovery process can be interrupted at any time, providing an estimate of each option’s quality based on collected samples. this anytime property allows for more flexibility in managing computational resources. 2.3 sequential halving sequential halving [6] is a flat, non-exploiting approachfootnote 3 for the mab problem. the algorithm evenly distributes a pre-defined budget (representing the number of simulations to be conducted) among all actions and progressively eliminates the bottom half in terms of performance. while effective at reducing simple regret compared to \(\text {ucb}_1\), it reduces cumulative regret at slower rates. sequential halving cannot be interrupted prematurely and offers less robust assurances regarding the quantity of suboptimal selections made [6]. algorithm 1 depicts sequential halving, employing a tree-like structure for compatibility with subsequent algorithms. each node, denoted as v, retains the following data: a state s, total reward q, number of visits n, and a list of child nodes originated from v (produced by the \(\textsc {children}{}\) function if it’s not a leaf node). we use k to restrict the quantity of possible actions, and the \(\textsc {head}{}\) function gives the first k children of \(v_{root}\). the sequential halving formula in line 7 divides the budget \(\mathcal {b}\) by the number of times \(v_{root}\)’s children can be halved, given by \(\log _2{\textsc {children}{v_{root}}}\). to distribute the budget over the remaining children in the current iteration, \(\mathcal {b}\) is also divided by k. 2.4 monte carlo tree search monte carlo tree search (mcts) [13] employs monte carlo simulations to iteratively build a game tree. mcts is designed to progressively converge on the best action as it gathers more statistical information about the domain. mcts is based on two principles: (1) with sufficient time, the sampled average reward from random simulations converges to the true state value, and (2) previous samples can guide future searches. algorithm 2 outlines the mcts process, starting instantiating a root node, denoted as \(v_{root}\). a node v consists of a state s, the list of applicable actions in s, the parent node, a list of children, and the q and n values for cumulative reward and visits count, respectively. the search process involves the following four steps: selection: beginning at tree root, the selection phase traverses the tree using a tree policy (\(\pi \)) that guides the search towards promising nodes, until finding a node with untried actions. expansion: a node is expanded by applying a random untried action to its state, resulting in a new child node. this new node is initialized with the new state, a list of applicable actions, an empty child list, and its parent reference. simulation: a playout evaluates the potential reward r of the new node. this is done by following a default policy (\(\pi _{\vardelta }\)), which usually applies random actions until reaching a terminal state. backpropagation: each node from \(v_{k+1}\) up to the root are updated: their q is updated by rw and n increases by 1. the search process continues until the algorithm uses up a specified resource \(\mathcal {r}\), which can be time or a number of iterations. \(\textsc {recommend}{}\) function selects a move according to one of three criteria: max child, with the highest q value; robust child, with the highest n value; or max-robust child, combining both q and n values. the most popular tree policy for the selection phase is the upper confidence bound (\(\text {ucb}_1\), sect. 2.2), which considers each node as an individual mab problem. when used in mcts, the algorithm is called upper confidence bound applied to trees (uct). mcts and uct exhibit an anytime property, allowing them to recommend useful actions even if the search execution is interrupted. 3 alternatives to uct simple regret (sr) minimization is strictly related to choosing a child node from the root at the recommendation phase, and the cumulative regret (cr) is related to the searching process. \(\text {ucb}_1\) has optimal bounds on cumulative regret recommendation, but it is penalized in terms of simple regret. at the root node, sampling in mcts/uct typically focuses on finding the best move with high confidence. once \(\text {ucb}_1\) identifies such a move, it continues to spend time on it, possibly resulting in low information gain [15]. in time-sensitive situations, not considering other options and continuing with the current best choice may be a potential flaw that could be improved. bubeck et. al [3] shows that \(\text {ucb}_1\) exhibits a slow decrease in terms of simple regret, with the best-case scenario being a polynomial rate decrease. this can be problematic when fast recommendations are required. karning et. al [6] suggest that the more exploratory policies have better bounds on simple regret minimization, like the sequential halving algorithm (sect. 2.3). 3.1 and sr+cr tolpin and shimony [15] modify \(\text {ucb}_1\)’s policy into . this policy adjusts the \(\text {ucb}_1\) formula using a quicker-growing sublinear function, leading to a faster increase in the uncertainty bonus. the new policy changes the \(\ln n_s\) term in \(\text {ucb}_1\) to \(\sqrt{n_s}\), aiming to narrowing the gap between the selections of non-greedy nodes. tolpin and shimony also point out that nodes closer to the root and those deeper in the tree have different goals. the former is more crucial for move recommendations. as a result, the search strategy near the root should prioritize reducing simple regret more quickly, while nodes deeper in the tree should aim to match the value of taking the optimal path, aligning more with cumulative regret minimization. the simple regret plus cumulative regret (sr+cr) scheme proposed by tolpin and shimony integrates two different policies to strike a balance between minimizing simple regret and cumulative regret. they introduced two specific algorithms, both of which combine the uct policy with more exploratory strategies. the first one, \(\text {ucb}{\sqrt{\text {}}} + \text {uct}\), operates by applying the \(\text {ucb}{\sqrt{\text {}}}\) policy at the root node and \(\text {ucb}_1\) to all child nodes. their second algorithm, \(\frac{1}{2}\text {-greedy} + \text {uct}\), operates by randomly selecting a node at the root with a 50% probability. 3.2 hybrid monte carlo tree search hybrid monte carlo tree search (h-mcts) [10] is a sr+cr algorithm that combines the sequential halving applied on trees (shot) algorithm [4] with uct. shot is a recursive adaptation of sequential halving, used for constructing game-trees using a non-exploiting policy. in h-mcts, shot is used not only at the root node but also deeper within the tree to address simple regret minimization. the proposed method switches from uct to shot when the computational budget spent in the node achieves a certain threshold, after considering changing the policy to be safe (when a subset of good moves are already identified and evaluated). the algorithm shifts its focus from minimizing cumulative regret to minimizing simple regret after beginning to expand shot within uct regions that have been sufficiently visited. since the computational budget per node is initially small, the simple regret tree remains shallow, as shot eliminate nodes from selection, the budget spent increases, causing the shot tree to grow deeper. h-mcts outperforms uct for various exploration coefficients [10] and is highly effective in games with large branching factors, as it prunes low-promising nodes and directs the search towards the most promising areas. 4 improving uct in time-restricted scenarios in this section, we present a new sr+cr algorithm, and a new method to calculate the amount of time the agent should spend in each move, assuming a time constraint for the entire game. 4.1 \(\text {uct}_{{\sqrt{\text {sh}}}}\) although h-mcts is promising at balancing simple and cumulative regret, it requires a predefined budget for the shot portion, which is not possible to estimate for previous unknown domains. furthermore, by neglecting the exploitation of nodes, the agent becomes prone to excessive resource allocation in unpromising regions. to enhance the performance of uct under a ggp environment with rigid time constraints, we propose a different sr+cr method, using sequential halving and , as shown in algorithm 3. uct\(_{{\sqrt{\text {sh}}}}\) prioritize simple regret minimization at root node by combining with sequential halving eliminations, and the cumulative regret component uses uct. in uct\(_{{\sqrt{\text {sh}}}}\), the aim of sequential halving is not to converge to the single best move, but rather to limit the number of children to search, which allows to explore the most promising areas. we set a lower limit, \(k_{min}\), on the number of child nodes necessary for the elimination process to occur. the halving operation is executed by dividing k by two. during the child selection process at the root, we use k to constrain the selection to the first k child nodes. upon reaching a new elimination point, the algorithm arranges the root’s child nodes in descending order according to their anticipated reward. we employ an iterative methodology (presented at line 11) to ascertain when to halve the number of children. we compute a ratio representing the fraction of halving stages already completed. for that, we divide the halve counter h by the maximum number of halving operations \(\log _2{n}\). we compute the resource allocation required for the next halving operation multiplying this ratio with the total resource \(\mathcal {r}\). after the used resource r surpasses this value, we increment h by one. this method ensures that the same portion of \(\mathcal {r}\) is equally divided across all halving stages. a key distinction from traditional mcts lies in the separate treatment of root selection. the root selection, depicted at line 13, iterates over the first k-th children, by calling the \(\textsc {head}{ch, k}\) function. the selected child is the one which maximizes \(\pi _{ucb_{\sqrt{\text {}}}}\). notice that rather than eliminating moves based on the number of visits a node has, we determine when to apply eliminations based on \(\mathcal {r}\), which can be time or number of simulations. 4.2 clock bonus time ggp agents play games without prior knowledge, usually with a time limit for the entire game. agents must allocate time for each move. a common strategy is to use a predefined fixed time budget, which can lead to inefficient time management, either by exhausting the time on long games, or leaving unused time in shorter games. we propose a method for estimating the time to spend on each move in a ggp environment, using a certain number of simulations during the search to gather information about the game itself. our model employs a minimum thinking time, and for games where the agent can spend more time, it provides a thinking time bonus. the clock bonus time (cbt) formula is as follows: $$\begin{aligned} cbt = \max (\tau _{min}, \min (\tau _{max}, g/\overline{m})) \tau _{min} . \end{aligned}$$ (4) in eq. 4, g is the total time budget, \(\tau _{min}\) and \(\tau _{max}\) are the minimum and maximum allowed thinking times per move, respectively. the bonus is given by g divided by the estimated number of moves left to finish the game \(\overline{m}\), which we compute using playouts. the max and min functions ensure that the agent performs at least the minimum thinking time and avoids overestimating the time it has. we then discount the new time by \(\tau _{min}\) because it is a bonus increased to the minimum thinking time. one way to integrate cbt with mcts consists of calling cbt after half of \(\tau _{min}\) has passed, which is when \(r \ge \mathcal {r}/2\). 5 experiments to evaluate uct\(_{{\sqrt{\text {sh}}}}\), we conduct two experiments. first, we examined the agent’s decision-making in two different scenarios of reward variance. this design simulate game situations where making a suboptimal choice significantly affects the outcome (high variance), as well as those where suboptimal choices have a milder effect and are less harmful (low variance). however, in these latter scenarios, a series of misjudgments due to lack of confidence in the most rewarding decision can potentially lead to an overall loss. the second test examine our agent’s performance within a practical context. we utilized a game competition called kilothon, hosted within the ludii environment. this competition served as a benchmark on uct\(_{{\sqrt{\text {sh}}}}\) performance in a ggp environment. 5.1 prize box selection experiment the prize box selection experiment is a simplified mab where there are k boxes containing a deterministic amount of money. the money for each box is pre-selected from a gaussian distribution \(n(\mu ,\sigma )\). we test different policies for a given number of trials and number of boxes, recording how often the policy selects each box during the experiment. we compare \(\text {ucb}_1\), sequential halving, , and ucb\(_{\sqrt{\text {sh}}}\) (root policy of uct\(_{{\sqrt{\text {sh}}}}\) ), at a scenario with high and low variance in reward’s distributions. both scenarios with 10000 trials for distribute across 30 boxes. in the low variance case, we set \(\mu =0.3\) and \(\sigma = 0.05\), limited to [−0.5, 0.5]. for the high variance case, we set \(\mu = 0.3\) and \(\sigma = 0.5\), limited to [−1,1]. fig. 1. box selection frequency under low variance reward distribution. full size image figure 1 depicts the low variance scenario, where the boxes are arranged in descending order, displaying only the 20 boxes with the highest rewards (i.e., boxes 1–10 have the lowest reward and are omitted). in this scenario,  presents the most dispersed trials among all boxes, with \(\text {ucb}_1\) following a similar pattern. the use of eliminations in this specific scenario guides the policies that adopt them towards more focused selections, ucb\(_{\sqrt{\text {sh}}}\) and sequential halving concentrated a higher quantity of resources on a smaller subset of boxes than \(\text {ucb}_1\) and . in the high variance test at fig. 2, both \(\text {ucb}_1\) and ucb\(_{\sqrt{\text {sh}}}\) exhibit a stronger preference for the highest rewarded box. while sequential halving does not change its selection distribution no matter the reward distribution is, due to its non-exploiting nature. and sequential halving share more similar frequencies of selection between them, indicating their preference for exploration over \(\text {ucb}_1\). using ucb\(_{\sqrt{\text {sh}}}\) avoids overspending trials on the best box in high-variance scenarios, which can be a desirable characteristic from the perspective of simple regret minimization, although it has a clear preference for the best rewarding box. fig. 2. box selection frequency under high variance reward distribution. full size image our analysis emphasizes that while exploiting (i.e. allocating budget to the most promising option) is valuable for reward maximization, exploration is crucial in game playing as it leads to the rapid discovery of beneficial moves. in this context, ucb\(_{\sqrt{\text {sh}}}\) displays a particularly desirable quality of enhanced exploration in our experiments. essentially, the primary objective is to identify and execute the optimal move in the game. the frequency of selecting the best move during the search is not of primary concern. furthermore, ucb\(_{\sqrt{\text {sh}}}\) appears to be less sensitive to varying rewards distributions. this resilience stems from its ability to not overlook exploitation in high variance scenarios, and to focus resources on low variance situations through the application of sequential halving eliminations. 5.2 ggp competition experiment general game playing (ggp) is a research area focused on developing intelligent agents capable of playing a wide variety of games without prior knowledge of any specific game being played [5]. ludii is a system for general game research, which has contributed significantly to the field. games are implemented using ludii’s game description language (gdl). ludii’s gdl is robust and straightforward, it allows researchers and game designers to create new games and even reproduce historical ones [12]. ludii hosted a ggp competition, where games chosen for the competition were turn-based, adversarial, sequential, and fully observable, including deterministic and stochastic games. kilothon was one of the competition tracks, where participants play 1094 games against an implementation of uct algorithm, native from ludii (which we will refer to as kilothon agent). each agent has a strict one-minute time limit to play each game in its entirety. when the one-minute time limit is reached, the agent must resort to random moves until the game concludes. the kilothon agent uses a fixed thinking time of 0.5 s per move, and incorporates two modifications to the pure uct algorithm: tree reuse enables the agent to store the search tree from previous plays and reusing it in the future, and open loop [11] for dealing with stochastic games. experimentation. as a baseline, we implemented our uct version, without tree reuse neither open loop, to compete against the kilothon agent, and compare results with uct\(_{{\sqrt{\text {sh}}}}\) . for both, we use 0.5 s of thinking time, and we differentiate agents when using the cbt method (which also uses 0.5 s as its \(\tau _{min}\) and 2 s as \(\tau _{max}\)). we conduct 10 kilothon trials for each agent, computing the average payoff of our agents to evaluate their effectiveness in kilothon. table 1. average payoff ± standard deviation and maximum payoff of each agent in kilothon.full size table table 1 presents the average payoff ± standard deviation of our tested agents across all games, along with the maximum payoff achieved by each of them. our results highlights the performance of uct\(_{{\sqrt{\text {sh}}}}\) method over uct, which achieves better scores than uct including after adding the cbt method. uct\(_{{\sqrt{\text {sh}}}}^{cbt}\) had the highest score, that could achieve second place in the official competition, where the first place achieved 0.231, and the second 0.031. the board games category, contains the vast majority of games in kilothon contest. ludii board games are classified according the following classifications [2, 8]: hunt, where a player controls more pieces and aims to immobilize the opponent; race, where the first to complete a course, with moves controlled by dice or other random elements, wins; sow or mancala, where players sow seeds to specific positions and capture opponent seeds; space, where players place and/or move pieces to achieve a specific pattern, with possibility of blocks and captures; and war, where the goal is to control territory, immobilize or capture all opponent’s pieces. fig. 3. winning percentage in board categories for each agent, after running 10 kilothon tournaments each. full size image figure 3 showcases the winning rate of our agents under the five board game categories. the win ratio is computed as win/(win+loss), not including draws. uct\(_{{\sqrt{\text {sh}}}}\) outperforms uct in sow (+6%), space (+7%), and war (+4%) games. against the kilothon agent, uct\(_{{\sqrt{\text {sh}}}}\) secures the respective win ratios in hunt, race, sow, space and war, respectively: 56%, 53%, 57%, 55%, 51% . sow and space games shows the highest variability among agents, with the highest scores achieved by uct\(_{{\sqrt{\text {sh}}}}^{cbt}\) of 69% and 63%. both these games display significant performance boosts via the cbt method for \(\text {uct}^{cbt}\) and uct\(_{{\sqrt{\text {sh}}}}^{cbt}\) , both surpassing a 60% win rate. our evaluations reveal that the uct\(_{{\sqrt{\text {sh}}}}\) strategy, especially with the cbt method, outperforms baseline uct. the uct\(_{{\sqrt{\text {sh}}}}^{cbt}\) agent had the highest score, showcasing its improvement over the baseline. ggp subset: five board games. while kilothon competition encompassed an extensive variety of games, we examine a subset that fell within the board game category. to this end, we selected games that were also part of a study conducted by pepels [9]. we compare uct\(_{{\sqrt{\text {sh}}}}\) vs uct, where both agents had 0.5, 1, and 2 s of thinking time for each move. each experiment running over 1000 matches. table 2 showcases the results. table 2. win rate of uct\(_{{\sqrt{\text {sh}}}}\) against uct.full size table our study reveals the advantages of uct\(_{{\sqrt{\text {sh}}}}\) over uct across various game domains. uct\(_{{\sqrt{\text {sh}}}}\) makes a better use of the time budget, as its win rate over uct increases significantly when the budget increases. in pentalath, the win rate began at 51.7% and rose to 66.8% as the thinking time increased, while nogo achieved the highest final win rate of 79.6%. both breakthrough and amazons exhibited a significant increase in win rates, escalating from less than 50% to 66.9% and 70.1%, respectively. the overall result shows that uct\(_{{\sqrt{\text {sh}}}}\) improve its advantage over uct after both had increased its thinking time, achieving win rates of 53% up to 71%. 6 conclusion this work addresses two drawbacks in uct, the base method of most general game playing (ggp) agents: (i) uct exploitation factor guarantees asymptotic optimality but suffers from simple regret minimization; (ii) the use of fixed time-budget for search per move may be an overestimate and an underestimate for long and short games, respectively. for (i), we introduce uct\(_{{\sqrt{\text {sh}}}}\), a new mcts method, which foregoes the asymptotic optimality in exchange for a timely response. uct\(_{{\sqrt{\text {sh}}}}\) uses the simulation budget more exploratively than traditional uct, since during the search time the goal is to find the best possible move and return it to the game. for (ii), we present the clock bonus time (cbt) strategy to better allocate the search time per move, given a fixed time budget to play the entire game. we use two experiments to empirically evaluate uct\(_{{\sqrt{\text {sh}}}}\) against uct. the prize box experiment indicates that uct\(_{{\sqrt{\text {sh}}}}\) is less sensible to changes in the distribution of rewards as \(\text {ucb}_1\) and do. these latter two have a more spread-out allocation when rewards have minimal variation. however, when there’s a lot of variation in the rewards, uct\(_{{\sqrt{\text {sh}}}}\) tends to favor the best option, although not as consistently as \(\text {ucb}_1\), which almost always selects the top choice. although it may appear that constantly selecting the known best option would maximize rewards, as proposed in tolpin and shimony’s study [15], they argue differently. they suggest that policies that promote more exploration at the root level can actually lead to faster identification of better moves. in the kilothon ggp competition, our method exceeded the performance of the baseline uct. the implementation of the cbt strategy more than doubled the scores for both agents. uct\(_{{\sqrt{\text {sh}}}}^{cbt}\) could achieve the second place in the official kilothon competition, according to the final competition results. this achievement is remarkable, considering our agent relies solely on monte carlo simulations and does not utilize any other enhancements or parallelism. our study also showed that uct\(_{{\sqrt{\text {sh}}}}\) uses increased time budgets significantly better than uct. moreover, our findings suggest that using 0.5 s of thinking time, as the default uct kilothon agent does, the time constraint imposed might be too unrealistic for agents to play in many game domains. notes 1.https://github.com/schererl/finalyearproject. 2.https://github.com/ludeme/ludiiaicompetition. 3.in contrast with exploiting policies, that allocate most resources to the most promising choice, non-exploiting policies allocate resources uniformly among choices, iteratively discarding the poorly-performing ones. references auer, p., cesa-bianchi, n., fischer, p.: finite-time analysis of the multiarmed bandit problem. mach. learn. 47(2), 235–256 (2002) article  math  google scholar  brice, w.c.: a history of board-games other than chess. by h. j. r. murray. oxford: clarendon press, 1952. pp. viii 287, 86 text figs. 42s. j. hellenic stud. 74, 219–219 (1954). https://doi.org/10.2307/627627 bubeck, s., munos, r., stoltz, g.: pure exploration in finitely-armed and continuous-armed bandits. theoret. comput. sci. 412(19), 1832–1852 (2011) article  mathscinet  math  google scholar  cazenave, t.: sequential halving applied to trees. ieee trans. comput. intell. ai games 7(1), 102–105 (2014) article  google scholar  genesereth, m., love, n., pell, b.: general game playing: overview of the aaai competition. ai mag. 26(2), 62–62 (2005) google scholar  karnin, z., koren, t., somekh, o.: almost optimal exploration in multi-armed bandits. in: international conference on machine learning, pp. 1238–1246. pmlr (2013) google scholar  kocsis, l., szepesvári, c., willemson, j.: improved monte-carlo search. univ. tartu, estonia, tech. rep. 1, 1–22 (2006) google scholar  parlett, d.: the oxford history of board games. oxford university press, oxford (1999) google scholar  pepels, t.: novel selection methods for monte-carlo tree search. master’s thesis, department of knowledge engineering, maastricht university, maastricht, the netherlands (2014) google scholar  pepels, t., cazenave, t., winands, m.h., lanctot, m.: minimizing simple and cumulative regret in monte-carlo tree search. in: workshop on computer games, pp. 1–15. springer (2014) google scholar  perez liebana, d., dieskau, j., hunermund, m., mostaghim, s., lucas, s.: open loop search for general video game playing. in: proceedings of the 2015 annual conference on genetic and evolutionary computation, pp. 337–344 (2015) google scholar  piette, é., soemers, d.j.n.j., stephenson, m., sironi, c.f., winands, m.h.m., browne, c.: ludii the ludemic general game system. in: giacomo, g.d., catala, a., dilkina, b., milano, m., barro, s., bugarín, a., lang, j. (eds.) proceedings of the 24th european conference on artificial intelligence (ecai 2020). vol. 325, pp. 411–418. ios press (2020) google scholar  świechowski, m., godlewski, k., sawicki, b., mańdziuk, j.: monte carlo tree search: a review of recent modifications and applications. artificial intelligence review, pp. 1–66 (2022) google scholar  świechowski, m., park, h., mańdziuk, j., kim, k.j.: recent advances in general game playing. sci. world j. 2015 (2015) google scholar  tolpin, d., shimony, s.: mcts based on simple regret. in: proceedings of the aaai conference on artificial intelligence, vol. 26.1, pp. 570–576 (2012) google scholar  download references author information authors and affiliations pontifical catholic university of rio grande do sul, porto alegre, brazil victor scherer putrich & felipe meneguzzi federal university of rio grande do sul, porto alegre, brazil anderson rocha tavares university of aberdeen, aberdeen, scotland felipe meneguzzi authors victor scherer putrichview author publications search author on:pubmed google scholar anderson rocha tavaresview author publications search author on:pubmed google scholar felipe meneguzziview author publications search author on:pubmed google scholar corresponding author correspondence to victor scherer putrich . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper putrich, v.s., tavares, a.r., meneguzzi, f. (2023). a monte carlo algorithm for time-constrained general game playing. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_7 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords general game playing monte carlo regret publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature dual-bandwidth spectrogram analysis for speaker verification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper dual-bandwidth spectrogram analysis for speaker verification conference paper first online: 30 january 2025 pp 340–351 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) dual-bandwidth spectrogram analysis for speaker verification download book pdf download book epub rafaello virgilli9, arnaldo candido junior10, augusto seben da rosa11, frederico s. oliveira9 & … anderson da silva soares9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 483 accesses abstract the variability of the human voice is a challenge for speaker verification systems, influenced by individual traits and environmental conditions. this research introduces a novel approach that uses dual-bandwidth spectrograms with the fast resnet-34 neural network architecture for speaker verification. dual-bandwidth spectrograms are data structures similar to multi-channel images, generated by stacking spectrograms derived from the same audio segment using two different window sizes. in this study, we employed window sizes of 5 ms and 30 ms. this approach captures a wider range of voice features across multiple temporal and spectral resolutions. our findings demonstrate a statistically significant improvement in system performance, achieving an equal error rate (eer) of 1.64% ±0.13%. this represents a 26% enhancement over the previously reported benchmark eer of 2.22% ±0.05%, validating our hypothesis that dual-bandwidth spectrograms offer a more detailed and comprehensive representation of voice features for accurate speaker verification. analysis of individual bandwidth contributions reveals that narrowband spectrograms carry more relevant features for speaker verification, while the combination with broadband spectrograms provides complementary information. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others speech bandwidth extension using spectral data masking chapter © 2025 detecting speaker-discriminative spectral content in wideband for automatic speaker recognition chapter © 2016 conclusions and future work chapter © 2016 1 introduction speaker verification (sv) is the task of determining whether two voice samples belong to the same speaker. unlike speaker identification [1], which attempts to identify a speaker from a closed set of individuals, sv compares the features extracted from any pair of voices to assess their similarity. this open-set approach offers greater flexibility and applicability in biometric systems [2]. the variability of the human voice poses a significant challenge for sv systems. individual traits such as age, accent, emotional state, intonation, and language, as well as environmental factors like background noise, reverberation, and recording devices, can influence the voice signal [3, 4]. despite these variations, the voice possesses unique attributes that remain relatively stable, primarily due to the anatomical characteristics of the speaker’s vocal tract. in the context of machine learning, sv is typically addressed by training a model to extract individualized voice characteristics and compare them to determine a similarity score between two voice samples. this score is then compared to a decision threshold to classify the samples as originating from the same speaker or different speakers. the equal error rate (eer) is a commonly used metric for calibrating the decision threshold. it balances the rate of false positives and false negatives. in this work, our primary contribution is the exploration of dual-bandwidth spectrograms for speaker verification. we propose adapting an existing deep neural network architecture for sv to process a combination of two spectrograms generated from the same audio segment. our hypothesis is that the integration of narrowband and broadband spectrograms will significantly enhance the accuracy of sv tasks by leveraging the complementary nature of these two spectrogram types. our dual-bandwidth approach combines narrowband spectrograms, generated with longer analysis windows (approximately 30 ms), and broadband spectrograms, generated with shorter windows (around 5 ms). narrowband spectrograms offer higher frequency resolution, allowing better visualization of harmonic structures related to the speaker’s fundamental frequency. conversely, broadband spectrograms provide better temporal resolution, capturing rapid acoustic events and broader spectral patterns like formants. by exploring this novel combination of representations, we aim to capture a more comprehensive set of speaker-specific features across different temporal and spectral resolutions. this dual-bandwidth approach potentially allows the model to leverage both fine-grained frequency information and broader spectral patterns, leading to more robust speaker verification performance. while combined spectrograms have shown success in related tasks such as keyword detection, voice activity detection, and emotion recognition, our work is the first to thoroughly investigate their potential in speaker verification. we chose the model presented in [5], which achieved state-of-the-art sv performance in 2020, as the reference to validate our hypothesis. although more recent models with better performance exist, the simplicity of this model’s construction facilitates the isolation of the input’s influence during experiments. moreover, its relatively low computational cost enables training on available hardware resources. the remainder of this paper is organized as follows. section 2 presents related work, positioning our research within the current landscape of speaker verification techniques. section 3 describes our proposed method in detail, elaborating on the dual-bandwidth spectrogram approach and the neural network model created to validate our hypothesis. section 4 outlines the experimental setup, including the dataset, hardware configuration, and training procedure. section 5 presents and discusses the results, covering the evaluation protocol, performance metrics, and a comparison of the proposed model with the baseline. this section also addresses the limitations of our current approach and outlines potential directions for future work. finally, sect. 6 concludes the paper, summarizing the main findings and their implications for the field of speaker verification. 2 related work speaker verification has seen significant advancements through various techniques including data augmentation, novel neural network architectures, and multi-bandwidth spectrogram approaches. in terms of data augmentation for speaker recognition, [6] proposed using a voice conversion model to generate additional data. they also applied a bandwidth extension to augment narrowband speech, generating missing frequency bands from lowband information. regarding neural network architectures, [7] and [8] introduced different attention layers on top of the feature extractor to better encode variable-length vector representations compared to global average pooling. further advancing this concept, [9] adapted selective kernel attention (ska) modules with a multi-scale frequency and channel module, modifying feature extractor blocks to use an attention mechanism that selects from multiple 1d convolutional kernel sizes. the use of multiple spectrograms as input to neural network models has been explored by [10]. they combined mel spectrograms, gammatone spectrograms, and spectrograms extracted with continuous wavelet transform into a single 3d-channel spectrogram for a convolutional neural network (cnn). multi-bandwidth spectrograms, which combine multiple frequency bandwidths to capture a broader range of audio features, have shown promise in various speech and audio processing tasks. [11] analyzed combined spectrograms and proposed a new representation obtained by computing the pixel-wise geometric mean of narrowband and broadband spectrograms from the same audio signal. in the work of [12], multi-bandwidth spectrograms were applied to voiced and unvoiced sound detection. the authors used different window analyses depending on the frequency range, resulting in performance improvements compared to classical spectrogram generation approaches. while multi-bandwidth spectrograms focus on combining different frequency representations, other researchers have explored combining different types of features to improve speaker recognition performance. feature fusion techniques have been explored in speaker recognition. [13] proposed fusing mel frequency cepstral coefficients (mfccs) with new features based on temporal domain statistical indicators such as mean, median, and standard deviation. their model, trained and evaluated on different subsets of librispeech [14], outperformed the baseline models they evaluated. these studies demonstrate the potential of multi-bandwidth spectrograms and feature fusion techniques in various speech and audio processing tasks, motivating further exploration in speaker verification applications. our research indicates that the use of dual-bandwidth spectrograms, as defined in this work, has not been previously applied to speaker verification tasks. while studies such as [10] and [11] have explored various combinations of spectral representations in speech processing, they have primarily focused on speech recognition or general audio processing tasks. the specific application of dual-bandwidth spectrograms to speaker verification appears to be novel. this approach, combining narrowband and broadband spectrograms, extends the existing work on spectral analysis in speaker recognition systems. by leveraging complementary information from different spectral resolutions, our method aims to enhance the accuracy of speaker verification tasks. our approach addresses limitations in previous works by providing a simple yet effective means of improving speaker verification performance using readily available spectral information. 3 method this study introduces a novel approach to sv analysis, leveraging a deep neural network model that processes combined narrowband and broadband spectrograms. narrowband spectrograms are generated with longer analysis windows of 30ms. they capture finer harmonic details. whereas broadband spectrograms are generated with shorter windows of 5ms. they better represent rapid temporal variations and spectral envelopes. the window lengths of 30ms for narrowband and 5ms for broadband spectrograms are defined in [15] and [16]. the baseline for this study is the fast resnet-34 architecture proposed by [5]. we have implemented it using the original code provided by the authors. this model is a streamlined version of the traditional resnet-34 [17], characterized by a significant reduction in complexity. fast resnet-34 has been optimized to use only 1.4 million parameters, while the classical resnet-34 comprises 63.5 million parameters. this efficient design greatly enhances computational efficiency and simplifies the training process, enabling us to explore our hypothesis without requiring high-end hardware resources. the baseline for this study is the fast resnet-34 architecture proposed by [5]. we have implemented it using the original code provided by the authors. this model is a streamlined version of the traditional resnet-34 [17], characterized by a significant reduction in complexity. fast resnet-34 has been optimized to use only 1.4 million parameters, while the classical resnet-34 comprises 63.5 million parameters. this efficient design allows us to explore our dual-bandwidth hypothesis while maintaining reasonable computational requirements, even with the additional preprocessing step of generating two spectrograms per audio sample. 3.1 adaptation for dual-bandwidth spectrograms to accommodate our dual-bandwidth spectrogram approach, we modified the baseline fast resnet-34 model. this adaptation primarily involved changing the initial convolutional layer to accept two input channels instead of one, corresponding to the narrowband and broadband spectrograms. despite this modification, the number of output channels in this layer remained the same as in the original model. consequently, the dimensions of the tensor after the first layer, and throughout the rest of the network, remain unchanged. this allows the subsequent layers to process the combined information from both spectrograms seamlessly, without requiring further architectural changes. our modified model processes and combines features from both narrowband and broadband spectrograms from the very first layer, enabling it to simultaneously capture fine spectral details and broader temporal patterns. additionally, we incorporated self-attentive pooling (sap) [7] for temporal aggregation, utilizing attention mechanisms to emphasize critical segments in the sv process. despite these adaptations, the modified model maintains the parameter efficiency of the original fast resnet-34, retaining a total of 1.4 million parameters. this efficient design allows us to investigate the potential of dual-bandwidth spectrogram input for improved sv performance without significantly increasing computational requirements. 3.2 angular prototypical loss function in this study, we employ the angular prototypical loss function for metric learning, as introduced by [5]. this method involves using m audio clips from each speaker per training batchfootnote 1. these m clips are divided into two sets: s clips for the support set and q clips for the query set, with q set to 1, following the approach in [5]. as described in [5], a class prototype, or centroid, is calculated from the support set to represent each speaker class using the formula: $$\begin{aligned} \displaystyle c_j = \frac{1}{m-1}\sum _{m=1}^{m-1} {\textbf {x}}_{j, m} \end{aligned}$$ here, \({\textbf {x}}_{j, m}\) is the feature from the m-th audio of the j-th speaker, and \(m-1\) represents the number of clips in the support set. the similarity between each class centroid, \(c_j\), and the query audio feature, \(x_{j, m}\), is then measured using: $$\begin{aligned} \displaystyle s_{j, k} = w \cdot \cos (x_{_{j, m}}, c_k) + b \end{aligned}$$ (1) where w and b are learnable parameters. finally, the angular prototypical loss is calculated by comparing these similarity scores across all classes within a batch, as defined in [5]: $$\begin{aligned} \displaystyle l_p = -\frac{1}{n}\sum _{j = 1}^{n}\log \frac{e^{s_{j, j}}}{\sum _{k=1}^n e^{s_{j, k}}} \end{aligned}$$ here, \(s_{j, j}\) is the similarity between the centroid and the query vector of the same class, as defined in eq. 1. the remaining terms \(s_{j, k}\) represent comparisons with centroids of other classes. 3.3 dual-bandwidth spectrogram representation we represent the input audio as a dual-bandwidth spectrogram, which is a data structure similar to a multi-channel image with dimensions of width, height, and number of channels, i.e., \(x \in \mathbb {r}^{c \times h \times w}\). the way we do so is the following, we combine narrowband and broadband spectrograms extracted from the same audio segment. we apply the hamming window function [18] with a window width of 30 ms for the narrowband spectrogram and 5 ms for the broadband spectrogram. the step size between consecutive windows is set to 6.25 ms. considering the audio data sampling rate of 16,000 hz, the window lengths correspond to 480 samples for the narrowband spectrogram, 80 samples for the broadband spectrogram, and a step size of 100 samples. we then apply the mel-function to both narrowband and broadband spectrograms, resulting in 40 mel channels each. hence, let \(x_n \in \mathbb {r}^{m \times t}\) and \(x_b \in \mathbb {r}^{m \times t}\) denote the narrowband and broadband mel-spectrograms, respectively, where m is the number of mel channels, and t is the variable length dimension determined by the audio segment duration. to construct the dual-bandwidth spectrogram, \(x_n\) and \(x_b\) are concatenated along a new channel dimension c, yielding \(x \in \mathbb {r}^{c \times m \times t}\) with \(c=2\). figure 1 illustrates this construction process for a 2-s audio segment, resulting in a dual-bandwidth spectrogram. fig. 1. construction of the dual-bandwidth spectrogram by concatenating narrowband and broadband spectrograms along a new channel dimension. full size image 4 experiments we conducted three distinct experiments to evaluate sv models using the voxceleb2 [19] dataset’s development subset. this dataset features 5,994 unique speakers and over one million audio clips from youtube videos, varying in quality, presenting a gender imbalance, and predominantly in english. for each experiment, the models were initialized with random weights. the training was conducted on a setup with a single v100 gpu with 64gb of memory and 8 cpu cores, allowing for batch sizes of 240 audio samples. we used the adam optimizer [20] with an initial learning rate of 0.001, which was decreased by a factor of 0.95 every 10 epochs over 200 epochs. we observed a significant reduction in training loss early on, leading to stabilized training progress. the average training duration was 1.8 h per epoch for single spectrogram models and 2.21 h for dual-bandwidth spectrogram models. this sums to 360 h for single spectrogram models and 442 h for dual-bandwidth models. we performed three experiments with the models detailed in [5] 1. we have explored broadband mel-spectrograms with an 80-sample window 2. narrowband mel-spectrograms with a 480-sample window 3. dual-bandwidth mel-spectrograms. the core of our study is the exploration of a dual-bandwidth mel-spectrogram approach that combines both 80 and 480 sample windows, highlighting our innovative proposal. in the context of metric learning, accuracy is defined by calculating class centroids (prototypes) from the support set and classifying feature vectors inferred by the model from the query set based on the most similar centroid for each batch. at the end of the training, the model based on broadband spectrograms achieved an accuracy of 81.94%, the narrowband-based model obtained 85.19% accuracy, and the model using both spectrograms resulted in an accuracy of 88.22%. 5 results and discussion this section presents the main findings of our study, including the evaluation protocol, performance metrics, and a comparison of the proposed models with the baseline. we also discuss the significance of the results and their implications for sv. 5.1 evaluation protocol the trained models were evaluated using the test set and following the protocol proposed in [5], which utilizes a list of test cases developed by the authors of voxceleb1 [21]. each test case specifies two audio files and a label indicating if they belong to the same or different speakers. the test files are from the voxceleb1 test set, comprising 40 classes and 4,874 files. the list encompasses 8 tests for each file, with an equal division between intra-class and inter-class comparisons. for each audio file in the list of test cases, 10 two-second segments were extracted, evenly distributed (overlap may occur). a feature vector was extracted from each segment using the trained model, resulting in 10 vectors per audio, which were then normalized. the similarity between two audios was calculated using the average euclidean distance among all 100 possible vector pairs. the decision threshold l, set between 0 and 2, classifies audios as same or different classes based on similarity values being less than/greater than l. 5.2 performance metrics we evaluated the models using two main performance metrics: equal error rate (eer) and minimum detection cost function (mindcf) [22]. table 1. equal error rate and mindcf from the trained models and from the baseline [5]. to calculate the 95% confidence intervals for the eers of our trained models, we considered the eer as a random variable following a bernoulli distribution, approximated by a normal distribution for our large test set (37,720 instances) [23]. in [5], the authors repeated all experiments three times and reported the mean and standard deviation to account for random initialization.full size table the system is calibrated to equate the costs of both errors, setting the prior probability of the target speaker at 5% for defining mindcf. figure 2 presents the probability distributions of similarity scores obtained for the audio test cases using the model trained with dual-bandwidth spectrograms. it demonstrates the overlap between same-speaker and different-speaker cases. fig. 2. distribution of similarity scores obtained for the test cases from the list provided in [21] using the model trained with dual-bandwidth spectrograms. full size image figure 3 displays the variation of true positive rate (tpr), true negative rate (tnr), and detection cost function (dcf) with respect to the decision threshold for the model trained with dual-bandwidth spectrograms, as observed in our experiments. fig. 3. success rate curves for positive cases (tpr) and for negative cases (tnr) and equal error rate (red arrow); (3(a)) and detection cost [22] with minimum mindcf (red arrow) for the model trained with dual-bandwidth spectrograms. (color figure online) full size image the eer and mindcf values monitored during the training of each model are shown in fig. 4. 5.3 results and comparison table 1 presents the eer and mindcf values obtained for the trained models, along with the results from the baseline [5]. among the single spectrogram models, the narrowband mel-spectrogram model (eer 2.01% ±0.14%) significantly outperformed the broadband mel-spectrogram model (eer 2.65% ±0.16%). the minimal overlap in their confidence intervals suggests that this difference is statistically significant. this indicates that the narrowband representation likely carries more relevant features for sv, aligning with the baseline model [5] which used a 25 ms window (eer 2.22%). the superior performance of narrowband spectrograms suggests that fine-grained frequency resolution may be more critical for speaker verification than high temporal resolution, as it better captures certain speaker-specific characteristics in speech signals. the dual-bandwidth mel-spectrogram model achieved the lowest eer of 1.64% ±0.13%, representing a 26% relative improvement compared to the baseline. this further improvement demonstrates that while narrowband information is more relevant, the temporal resolution provided by broadband spectrograms contributes valuable complementary information. the statistically significant differences between these models, evidenced by the minimal overlap in confidence intervals, validate the potential of multi-resolution approaches in capturing the complex nature of speaker-specific information in speech signals. these confidence intervals were calculated considering the eer as a random variable following a bernoulli distribution, approximated by a normal distribution given our large test set (37,720 instances). this statistical approach allows us to confidently state that the observed improvements, particularly with the dual-bandwidth model, are statistically significant and unlikely due to random chance. 5.4 limitations and future work while our study demonstrates the effectiveness of dual-bandwidth spectrograms in enhancing sv performance, there are areas that could benefit from further exploration to optimize and expand upon the current findings. our investigation primarily focused on two specific bandwidth configurations-5ms and 30ms windows-which were selected based on their promising results. however, exploring a wider range of window sizes could reveal more effective combinations for various sv tasks. additionally, integrating our dual-bandwidth approach with more recent sv architectures might yield further performance enhancements, as the current comparisons were primarily against a single baseline model from 2020. moreover, the potential of multi-bandwidth spectrograms, extending beyond two bandwidths, offers a promising direction for future research. this could involve investigating spectrograms with three or more bandwidths to assess potential performance gains and identify points of diminishing returns. alternative techniques for combining spectrograms may also present opportunities for more optimal representations of multi-bandwidth information. fig. 4. eer (4(a)) and mindcf (4(b)) evolution during the training. full size image finally, while our experiments were conducted on the widely used voxceleb2 dataset, extending the evaluation to include more diverse datasets and real-world scenarios could help to establishing the robustness and broader applicability of the approach. by addressing these considerations, future research can continue to advance the understanding and application of multi-bandwidth spectrogram approaches in speaker verification. 6 conclusion this study investigated the impact of using dual-bandwidth spectrograms on the performance of speaker verification (sv) models. our research validated the hypothesis that combining narrowband and broadband spectrograms provides complementary information, leading to improved sv accuracy. the proposed dual-bandwidth spectrograms model achieved an equal error rate (eer) of 1.64% ±0.13%, outperforming the reference model from [5] which had an eer of 2.22% ±0.05%. this represents a 26% relative improvement in performance. the statistical analysis, based on 95% confidence intervals, supports the robustness of these findings, indicating that the performance differences observed are unlikely to be due to random variation. our analysis revealed that narrowband spectrograms (eer 2.01% ±0.14%) carry more relevant features for sv compared to broadband spectrograms (eer 2.65% ±0.16%). this suggests that when using conventional spectrograms, opting for analysis windows around 30 ms, which produce narrowband spectrograms, is preferable. however, the superior performance of the dual-bandwidth approach demonstrates that while narrowband information is more relevant, the temporal resolution provided by broadband spectrograms contributes valuable complementary information. the limitations of our study, including the exploration of only two bandwidth configurations and the use of a single dataset, have been acknowledged. these limitations, along with the promising results obtained, pave the way for future research. these include exploring additional bandwidth configurations, integrating our approach with more recent state-of-the-art models, and investigating the potential of multi-bandwidth spectrograms beyond dual-bandwidth. in summary, this study demonstrates the effectiveness of dual-bandwidth spectrograms in enhancing sv performance. the statistically significant improvements achieved by our approach highlight its potential to advance the state-of-the-art in sv systems. notes 1.here, a “batch” refers to a subset of data for one training iteration, sometimes called a “mini-batch” in other studies. references reynolds, d.a.: an overview of automatic speaker recognition technology. in: 2002 ieee international conference on acoustics, speech, and signal processing, vol. 4, pp. iv–4072. ieee (2002) google scholar  jain, a.k., flynn, p., ross, a.a.: handbook of biometrics. springer (2007) google scholar  spazzapan, e.a., cardoso, v.m., fabron, e.m.g., berti, l.c., brasolotto, a.g., marino, v.c.d.c.: acoustic characteristics of healthy voices of adults: from young to middle age. in: codas, vol. 30. scielo brasil (2018) google scholar  geoffrey, s.m., ewald, e., ramos, d., gonzález-rodríguez, j., lozano-díez, a.: statistical models in forensic voice comparison. in: handbook of forensic statistics, pp. 451–497. chapman and hall/crc (2020) google scholar  chung, j.s., et al.: in defence of metric learning for speaker recognition. in: interspeech (2020) google scholar  yamamoto, h., lee, k.a., okabe, k., koshinaka, t.: speaker augmentation and bandwidth extension for deep speaker embedding. in: interspeech, pp. 406–410 (2019) google scholar  cai, w., chen, j., li, m.: exploring the encoding layer and loss function in end-to-end speaker and language recognition system (2018) google scholar  kye, s.m., kwon, y., chung, j.s.: cross attentive pooling for speaker verification [c]\(\parallel \) 2021 ieee spoken language technology workshop (slt). 19–22 january 2021, shenzhen, china (2021) google scholar  mun, s.h., jung, j.-w., han, m.h., kim, n.s.: frequency and multi-scale selective kernel attention for speaker verification. in: ieee spoken language technology workshop (slt). ieee 2023, pp. 548–554 (2022) google scholar  arias-vergara, t., klumpp, p., vasquez-correa, j.c., nöth, e., orozco-arroyave, j.r., schuster, m.: multi-channel spectrograms for speech processing applications using deep learning methods. pattern anal. appl. 24, 423–431 (2021) article  google scholar  cheung, s., lim, j.: combined multiresolution (wide-band/narrow-band) spectrogram. ieee trans. signal process. 40(4), 975–977 (1992) article  math  google scholar  annabi-elkadri, n., hamouda, a.: automatic silence/sonorant/non-sonorant detection based on multi-resolution spectral analysis and anova method. in: international workshop on future communication and networking. szczecin, poland (2011) google scholar  jahangir, r., et al.: text-independent speaker identification through feature fusion and deep neural network. ieee access 8, 32 187-32 202 (2020) article  math  google scholar  panayotov, v., chen, g., povey, d., khudanpur, s.: librispeech: an asr corpus based on public domain audio books. in: 2015 ieee international conference on acoustics, speech and signal processing (icassp), pp. 5206–5210 (2015) google scholar  boersma, p., weenink, d.: praat: doing phonetics by computer (version 6.1.48) (2021). http://www.praat.org styler, w.: using praat for linguistic research. university of colorado at boulder phonetics lab (2013) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition (2015) google scholar  smith, j.o.: spectral audio signal processing. w3k (2011) google scholar  chung, j.s., nagrani, a., zisserman, a.: voxceleb2: deep speaker recognition. in: interspeech 2018 (2018). http://dx.doi.org/10.21437/interspeech.2018-1929 kingma, d., ba, j.: adam: a method for stochastic optimization. in: international conference on learning representations (2014) google scholar  nagrani, a., chung, j.s., zisserman, a.: voxceleb: a large-scale speaker identification dataset. interspeech 2017 (2017). http://dx.doi.org/10.21437/interspeech.2017-950 omid.sadjadi@nist.gov. nist 2018 speaker recognition evaluation (2018). https://www.nist.gov/itl/iad/mig/nist-2018-speaker-recognition-evaluation agresti, a., coull, b.a.: approximate is better than “exact’’ for interval estimation of binomial proportions. am. stat. 52(2), 119–126 (1998). https://doi.org/10.1080/00031305.1998.10480550 article  mathscinet  math  google scholar  download references author information authors and affiliations universidade federal de goiás, goiânia, brazil rafaello virgilli, frederico s. oliveira & anderson da silva soares universidade estadual paulista, são josé do rio preto, brazil arnaldo candido junior universidade tecnológica federal do paraná, medianeira, brazil augusto seben da rosa authors rafaello virgilliview author publications search author on:pubmed google scholar arnaldo candido juniorview author publications search author on:pubmed google scholar augusto seben da rosaview author publications search author on:pubmed google scholar frederico s. oliveiraview author publications search author on:pubmed google scholar anderson da silva soaresview author publications search author on:pubmed google scholar corresponding author correspondence to rafaello virgilli . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper virgilli, r., candido junior, a., da rosa, a.s., oliveira, f.s., soares, a.d.s. (2025). dual-bandwidth spectrogram analysis for speaker verification. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_24 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords speaker verification feature fusion dual-bandwidth spectrogram broadband narrowband publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature deep active-self learning applied to named entity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deep active-self learning applied to named entity recognition conference paper first online: 28 november 2021 pp 405–418 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) deep active-self learning applied to named entity recognition download book pdf download book epub josé reinaldo c. s. a. v. s. neto10 & thiago de paulo faleiros10  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1274 accesses 4 citations abstract deep learning models have been the state-of-the-art for a variety of challenging tasks in natural language processing, but to achieve good results they often require big labeled datasets. deep active learning algorithms were designed to reduce the annotation cost for training such models. current deep active learning algorithms, however, aim at training a good deep learning model with as little labeled data as possible, and as such are not useful in scenarios where the full dataset must be labeled. as a solution to this problem, this work investigates deep active-self learning algorithms that employ self-labeling using the trained model to help alleviate the cost of annotating full datasets for named entity recognition tasks. the experiments performed indicate that the proposed deep active-self learning algorithm is capable of reducing manual annotation costs for labeling the complete dataset for named entity recognition with less than 2% of the self labeled tokens being mislabeled. we also investigate an early stopping technique that doesn’t rely on a validation set, which effectively reduces even further the annotation costs of the proposed active-self learning algorithm in real world scenarios. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others investigation of deep active self-learning algorithms applied to named entity recognition chapter © 2023 a framework of data augmentation while active learning for chinese named entity recognition chapter © 2021 deep entity matching with adversarial active learning article 28 april 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms categorization learning algorithms machine learning sequence annotation statistical learning 1 introduction the task of named entity recognition (ner) is widely used for information extraction with trained models being able to directly extract named entities from unstructured text. the named entities can be used to create databases of structured information, as well as to improve the performance of models in more challenging natural language processing tasks. deep neural models have been the state of the art for solving ner tasks, but to be trained to achieve it they often require big labeled sets of data. in many practical scenarios it’s not realistic to expect big amounts of data to be manually annotated. active learning (al) algorithms are often used to reduce the cost of annotation by identifying a small and representative subset of samples to be labeled. it’s expected that a model trained on this representative subset, and on the whole dataset, should achieve similar performances. thus, by labeling the small representative subset, it is possible to reduce the annotation costs while training a good machine learning model. deep active learning. (dal) applied to sequence tagging tasks (e.g. ner, pos-tagging) has only recently appeared in the literature [15], even though traditional active learning strategies applied to such tasks have appeared much earlier [14]. the use of neural-based models in active learning scenarios, instead of classic shallow models, brings with it challenges that need to be addressed. for once, deep learning models are slow to be retrained from scratch for each active learning iteration when compared to shallow models. shen et al. [15] proposes the first dal algorithm applied to a sequence tagging task. it proposes to use iterative training where the neural model’s training continues from where it stopped in the previous iteration of the active learning process. this iterative training coupled with early stopping based on the validation set incurs in a reduction of execution time. this may be justified by the fact that as the subset of labeled samples becomes more representative of the whole dataset, new unlabeled samples are less likely to bring new relevant information to it. thus, the iteratively trained model is more likely to require fewer training epochs to assimilate the newer samples that were recently labeled. they also propose the cnn-cnn-lstm model, a neural model which is lightweight to train and the maximum normalized log-probability (mnlp) sampling function which normalizes the confidence of the model’s predictions based on the sentence’s length. the al algorithm of shen et al. [15] trained a neural model to peak performance using just over 25% of labeled data from the training set of the ontonotes 5.0 [11] dataset. siddhant and lipton [16] extend the previous work by shen et al. [15]. they rely on the bayesian active learning through disagreement (bald) [4] sampling technique, which queries unlabeled samples that generate the most disagreement from multiple passes on bayesian neural models and apply it to sequence tagging problems. they also use early stopping based on the performance of the model on the validation set, but limit the size of the validation set for it to be proportional to the size of the labeled set used to train the model. the experiments performed investigate the performance of the bayesian dal algorithm proposed using the cnn-cnn-lstm [15] and the cnn-bilstm-crf [7] models on the ontonotes 5.0 and conll 2003 [13] datasets. the results indicate that the proposed bayesian sampling functions consistently outperform the mnlp function, but with marginal improvement. current works on deep active learning applied to ner rely on validation sets for early stopping of the model training. in real world scenarios, active learning algorithms are used to reduce annotation costs, and separating a part of the labeled dataset to be used as a validation set seems to go against the core idea of al in general. active-self learning. (asl) combines traditional active learning with self-training techniques, where an oracle (i.e. human annotator) annotates the most informative samples queried by a sampling function (e.g. least confidence, mnlp) while the trained model labels samples for which it has the highest confidence in its predictions. the asl algorithm comes to further reduce the costs of labeling a dataset, when compared to the pure active learning process. tran et al. [17] shows that the symbiotic relationship created by both the human and a trained conditional random field model, labeling samples from the unlabeled set, increases the performance of the model while significantly reduce labeling costs. one major drawback of the asl algorithm appears in its sensitivity to the quality of the initial labeled set, where a model trained on an ill-sampled initial labeled set can have its performance significantly reduced throughout the asl process. in this work, we propose a deep active-self learning algorithm, that builds upon previous works from the literature [6, 17]. we propose some key changes to the algorithm from the literature in order to try and minimize its sensitivity to the initial set of labeled samples. we also propose an early stopping strategy based on the model’s confidence on an unlabeled set, which renders the use of a validation set unnecessary. 2 methodology this section presents the proposed deep active-self learning algorithm, which extends previous works from the literature [6, 17] with the use of deep learning models and key changes to alleviate the sensitivity of the self learning process to the initial labeled set. along with the asl algorithm, we also propose an early stopping criterion based on the model’s confidence on its predictions for the unlabeled samples, effectively dismissing the use of a validation set throughout the asl procedure. sections 2.1 and 2.2 present the proposed asl algorithm and the early stopping criterion, respectively. section 2.3 presents the experiments to be performed to evaluate the proposed strategies, as well as the baselines for comparison and hardware setup used for simulations. 2.1 deep active-self learning algorithm the active-self learning algorithms from the literature require the human and the trained model to cooperatively annotate samples from the unlabeled set. this process is very sensitive to the initial labeled set, which is used to train the initial machine learning model [17]. we argue that this sensitivity comes from the fact that samples annotated by the model in early rounds of the asl algorithm may add a permanent bias to the labeled set, if poorly annotated. this stems from the fact that the samples annotated by the model are considered as reliable as those annotated by the oracle. based on that, we propose an active-self learning algorithm that separates samples labeled by the model from those labeled by the human annotator, with the former having less impact on the model’s parameters during training. we also propose to return the samples labeled by the model to the unlabeled set at the end of each iteration of the asl algorithm. we expect these two changes to reduce the risk of adding permanent bias to both the trained model and the labeled sets. the proposed algorithm can be thought as an iterative semi-supervised learning process, as for each iteration it identifies a set of unlabeled samples that can be reliably used for self-training, an approach that resembles current semi-supervised works from the literature [1]. figure 1 presents a comparison between the labeling process and training of the model by the asl algorithms presented in the literature and the one proposed here. algorithm 1 presents a more detailed explanation of the proposed active-self learning algorithm. fig. 1. in the diagrams, u and l represent the unlabeled and labeled sets, respectively. a.l. is the active labeled set, which contains samples labeled by the oracle. s.l. stands for self-labeled, meaning the set containing the samples labeled by the trained model. figure (a) presents the asl algorithm found in the literature, where both the oracle and the trained model cooperatively annotate unlabeled samples and add them to the same labeled set, which is then used to train the machine learning model. figure (b) shows the proposed asl algorithm, where samples labeled by the oracle and by the trained model are separated into different labeled sets, which are used for further training of the model, but with the self-labeled set having a lesser impact on the model’s parameters during training. it also shows that after training, all samples labeled by the model are returned to the unlabeled set, thus reducing the risk of introducing permanent bias to the model throughout the asl process. full size image note that in the algorithm 1, m represents the machine learning model, q is the query budget, \(min\_confidence\) is the minimum confidence the model must have to annotate an unlabeled sample, a.l. stands for active labeled set which contains samples annotated by the oracle, s.l. contains the samples labeled by the trained model, and u represents the set of unlabeled data. the active learning procedure, represented by the \(active\_learning\_query(\cdot )\) function in algorithm 1, identifies the most informative unlabeled samples to be annotated by the oracle. for this work we selected the maximum normalized log-probability (mnlp) [15] as the sampling function, as it was shown to be competitive with more sophisticated sampling techniques [16] while being less expensive to be computed. the mnlp of a sequence x of length n can be computed as: $$\begin{aligned} mnlp(x) = \max _{y_1, ..., y_{n-1}}\frac{1}{n}\sum ^n_{i=0}log\ p(y_i\vert x_i, y_0, y_1, ..., y_{i-1}). \end{aligned}$$ (1) for the experiments, we follow previous works from the literature [16] and define a query budget as the number of tokens that can be selected by the sampling function to be manually annotated by the human annotator in a single iteration of the active learning process. we chose a query budget of approximately 2% of the tokens from the whole training set. thus, the query budget was 20,000, 6,000 and 4,000 words for the datasets ontonotes5.0, aposentadoria, and conll 2003, respectively. the \(self\_learning\_query(\cdot )\) function in algorithm 1 identifies the samples for which the current trained model has a confidence higher than a predefined threshold. these samples are considered to be reliable and, therefore, can be used for self-training. a minimum confidence of 0.99 was selected in this work as the threshold, based on experiments performed using the validation set. the learning rate for training the model with self-labeled samples is one tenth of the learning rate used for training with the active-labeled samples, with this value being selected from initial experiments using the validation set. for the model training, represented as \(train\_model(\cdot )\) in algorithm 1, we used classic supervised training. the single difference is that during training, we alternate using minibatches from the a.l. and s.l. sets. 2.2 dynamic update of training epochs dute inspired by the overall uncertainty stopping criterion for active learning algorithms, proposed by zhu et al. [18], we propose the dynamic update of training epochs (dute), an early stopping strategy that reduces the number of training epochs based on the mean confidence of the trained model on its predictions for the unlabeled set. we hypothesize that the model’s mean confidence indicates how well the labeled set represents the whole dataset, meaning that adding more unlabeled samples to the labeled set incurs in little change of the labeled set’s label distribution. because of that, we argue that fewer training epochs are required as the model’s confidence on the unlabeled set increases. to measure the model’s confidence on the unlabeled set, we propose to use the harmonic mean of the model’s normalized confidence for each unlabeled sample. the harmonic mean was used to emphasize those unlabeled samples for which the trained model has lower confidence in its predictions. the normalized confidence of a sequence of elements x can be computed as $$\begin{aligned} nc(x) = e^{mnlp(x)}, \end{aligned}$$ (2) where mnlp(.) is the maximum normalized log-probability, presented in eq. 1. the harmonic mean for the model’s confidence on the unlabeled set u is computed as (3) given the mean confidence of the trained model on the unlabeled set, the dute strategy computes the number of training epochs to be used at the k-th iteration of the active learning algorithm, as presented in eq. 4. $$\begin{aligned} \begin{aligned} epoch(k) =&round\bigg [(1.0 momentum) \times (1.0 mc(u)) \times epoch(k-1) \\ +\&(momentum) \times epoch(k-1)\bigg ] \end{aligned} \end{aligned}$$ (4) in this work, a momentum of 0.9 was selected based on initial experiments performed using the validation sets of the datasets presented in sect. 2.3. 2.3 experiments this section describes the experiments designed in order to validate the proposed asl algorithm and dute strategy. this validation was done in two separate experiments. the first experiment compares the proposed asl algorithm with the dute strategy to the al algorithm proposed by shen et al. the second experiment is an ablation study, where we investigate the impact that the dute strategy has on both the execution time of the algorithm and the trained model’s performance. the ner datasets, neural models and baselines used on both experiments are presented in the following sections. datasets for the experiments, we used two english ner datasets frequently used in the literature, namely the ontonotes 5.0 [11] and the conll 2003 [13]. additionally, we also experimented on a novel legal domain ner dataset in portuguese, named aposentadoria (retirement, in direct translation) dataset. the aposentadoria dataset contains named entities from 10 distinct classes related to retirement acts of public employees published in the diário oficial do distrito federal (brazilian federal district official gazette, in direct translation). it’s a part of a major dataset created by collaborators of project knedlefootnote 1. more information about the dataset as well as a download link to it are available at the github repositoryfootnote 2. table 1 presents the datasets used in the experiments, along with relevant information such as domain area and language. table 1. datasets to be used for experiments on ner.full size table for the experiments, all datasets were preprocessed by converting them to the iobes notation which is frequently adopted by works on ner in the literature [1, 5, 7] for being capable of significantly improving the model’s performance [12]. also, numeric characters were replaced by # and 0 characters for english and portuguese datasets, respectively. both replacements were made in order to enable the use of pretrained word embeddings used by the neural models, which are better described next. neural models the models to be used are those that appear in the work by siddhant and lipton [16], namely the cnn-cnn-lstm introduced by shen et al. [15] and the cnn-bilstm-crf designed by ma and hovy [7]. the hyperparameters used for both models were similar to those from their original papers for the experiments performed in the english ner datasets, and a grid-search was performed using the validation set on the aposentadoria dataset to find the best performing parameters for both models. the parameters for the cnn-cnn-lstm to be trained on the english ner datasets were similar to those presented in the experiments of siddhant and lipton [16], where the character-level cnn uses 25 dimensional character embeddings, the 1-d convolution layer has 50 filters, kernel size of 3, stride and padding of 1. the word-level cnn has two blocks, where each block has a 1-d convolutional layer with 800 filters, kernel size of 5, stride of 1 and padding of 2. the lstm tag decoder consists of an lstm layer of size 256. all dropout layers for the cnn-cnn-lstm model have probability 0.5. for the portuguese ner dataset, a grid-search was performed using the validation set and the final architecture chosen had the following hyperparameters: the character-level cnn had the same parameters as those used for the english datasets. the word-level cnn has one block, consisting of a 1-d convolution layer with 400 filters, kernel size of 5, stride of 1 and padding of 2. the lstm tag decoder has size 128. all dropout probabilities are still of 0.5. the cnn-bilstm-crf model trained on the english ner datasets has a character-level cnn with 30 dimensional character embeddings, a 1-d convolution layer with 30 filters, kernel size of 3, padding of 1 and stride of 1. the bilstm layer has size 300. all dropout probabilities are set to 0.5. for the portuguese dataset, a grid-search was employed to define the model’s best parameters. the chosen model had the same parameters for the character-level cnn as that used for english datasets, while the bilstm layer has size 256. the pretrained embeddings used for both models for the english datasets were the glove embeddings of 100 dimensions pretrained on an english newswire corpus [10]. for the portuguese dataset, the glove embeddings of 300 dimensions pretrained on a multi-genre portuguese corpus [2] was used. for optimization of the models throughout training, we used stochastic gradient descent of the cross entropy loss for the cnn-cnn-lstm and the negative log-likelihood for the cnn-bilstm-crf. a constant momentum of 0.9 was selected and gradient was clipped at 5.0. learning rates were 0.015 for both models trained on english datasets. for the portuguese dataset, learning rates of 0.010 and 0.0025 were selected for the cnn-cnn-lstm and cnn-bilstm-crf models, respectively. baselines and evaluation metrics for the first experiment, the baseline to compare the performance of the proposed asl algorithm with the dute strategy is the deep active learning algorithm proposed by shen et al. [15], using the mnlp sampling function. one key change is that we do not employ early stopping based on a validation set as the original work, instead we train the machine learning model for the full number of training epochs at each iteration of the dal process. this change is justified by the fact that the early stopping is applied mainly to avoid overfitting of the models during training, and throughout all the experiments performed this phenomenon didn’t occur. the f1-scores will be computed on the test set at the end of the training process. query budget, number of training epochs, and initial labeled set size will be the same as those used by the asl algorithm. for the second experiment, the ablation study, we wish to investigate the impact of the proposed dute strategy. as such, the baseline will be the proposed asl algorithm without the dute strategy. a trained model’s performance will be evaluated using the exact-match (i.e. span-based) micro-averaged f1-score on the test set. experiment setup all experiments reported were implemented and executed on the google colab platform with a pro subscription using the pytorch [9] framework. gpus available were the tesla p100, the tesla v100, and the tesla t4 all with 16gb. fig. 2. comparison between the proposed asl algorithm with the dute strategy (asl_dute) and the deep active learning algorithm (al). experiments were performed across all datasets with both neural models. the x axis represents the percentage of the whole training set that has been annotated by the oracle. full size image 3 results the results for the experiments on the proposed deep active-self learning algorithm along with its baselines are presented in this section. we also show an ablation study conducted to investigate the impact of the dute strategy on the performance and execution time of the proposed asl algorithm. 3.1 deep active-self learning with dute strategy figure 2 shows the performance curves comparing the traditional deep active learning algorithms found in the literature and the proposed deep active-self learning algorithm with dute strategy. each experiment was executed six times with unique initial labeled sets, and we report the mean performance curve for each of them. the same six initial labeled sets were used for experiments on the same dataset for all training algorithms and neural models. the algorithms were executed until at least 50% of the training set had been annotated by the oracle, in a similar setup used in previous works [15, 16]. it has been observed that the proposed asl algorithm achieved similar performance to the deep active learning baseline for most experiments. one advantage of the active-self learning algorithm is its capabilities of allowing for self-labeling, thus potentially reducing human annotation efforts. figure 3 reports the performance of the self-labeling process throughout the active-self learning algorithm. the figures present the percentage of self-labeled samples by the percentage of actively labeled samples (i.e. human annotation) and discriminates the amount of correctly and wrongly self-labeled tokens. in general, the trained model predicted most tokens correctly at all iterations of the algorithm. across all experiments performed, \(1.31\%\) was the highest percentage of tokens to be mislabeled by the trained model in any given iteration of the asl process. even though most tokens were correctly self-labeled, little impact has been observed on the trained model’s performance due to self-training. this may be justified by the fact that ner datasets are imbalanced, with most tokens not being a part of a named entity, and entity level annotations being unreliable at earlier rounds. fig. 3. comparison between the correctly and wrongly self-annotated samples, at token level, for the experiments performed on all three datasets. note that the y-axis on each graph presents the percentage of the whole training set that has been self-labeled by the trained model in a given iteration of the asl algorithm, while the x-axis presents the percentage of the whole training set that has been labeled by the human annotator. full size image 3.2 ablation study next we conduct an ablation study to investigate the impact of the proposed dute strategy on the performance and execution time of the proposed deep active-self learning algorithm. differently from the previous experiment, in this ablation study we executed the algorithms until at least 30% of the training set had been annotated by the oracle. figure 4 compares the performance of the asl algorithm with and without the dute strategy, while table 2 reports the mean and standard deviation for the execution times of the simulations. it has been observed that for all the experiments conducted, the dute strategy is capable of significantly reducing execution time, while maintaining similar model performance. fig. 4. comparison of the performance of a model trained through the proposed deep active-self learning algorithm with and without the dute strategy. the x axis represents the percentage of the whole training set that has been annotated by the oracle. full size image table 2. mean and standard deviation for the execution time of the proposed active-self learning algorithm with and without the dute strategy, reported in units of minutes. execution times are measured with algorithms being stopped when the set of samples annotated by the human annotator reaches 30% of the entire training set. asl_dute and asl refer to the proposed active-self learning algorithm with and without the dute strategy, respectively.full size table 4 conclusion from the experiments conducted, we noticed that the proposed deep active-self learning algorithm is capable of self-annotating unlabeled samples reliably. the self-training hasn’t, however, shown significant impact on the model’s performance. the proposed dute strategy has also shown promising results, being capable of significantly reducing the execution times for simulations of the proposed algorithm with little impact on the model’s performance, while not relying on a validation set. one of the main limitations of the proposed asl algorithm still continues to be its reliance on labeled data for hyperparameter tuning. while our dute strategy effectively avoids using validation sets throughout the asl process, we still used training and validation sets for tuning of the hyperparameters for the neural models and others such as learning rates and batch sizes. this still poses one of the biggest limitations to the use of active learning algorithms in real-world low-resource scenarios. another observation made is that the self-training technique employed here wasn’t able to improve the model’s performance. future works may investigate more sophisticated self-training techniques, such as consistency regularization techniques, which are often employed by current state-of-the-art semi-supervised methods [1]. these more recent self-training techniques also tend to use soft targets for training, instead of hard targets (i.e. one hot encoded vectors) [1, 8], in a setup similar to knowledge distillation [3]. notes 1.http://nido.unb.br/. 2.https://avio11.github.io/resources/aposentadoria/. references clark, k., luong, m.t., manning, c.d., le, q.: semi-supervised sequence modeling with cross-view training. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 1914–1925. association for computational linguistics, brussels, october–november 2018. https://doi.org/10.18653/v1/d18-1217 hartmann, n.s., fonseca, e.r., shulby, c.d., treviso, m.v., rodrigues, j.s., aluísio, s.m.: portuguese word embeddings: evaluating on word analogies and natural language tasks. in: anais do xi simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 122–131. sbc, porto alegre, rs, brasil (2017). https://sol.sbc.org.br/index.php/stil/article/view/4008 hinton, g., vinyals, o., dean, j.: distilling the knowledge in a neural network (2015) google scholar  houlsby, n., huszár, f., ghahramani, z., lengyel, m.: bayesian active learning for classification and preference learning (2011) google scholar  lample, g., ballesteros, m., subramanian, s., kawakami, k., dyer, c.: neural architectures for named entity recognition. in: proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 260–270. association for computational linguistics, san diego, june 2016. https://doi.org/10.18653/v1/n16-1030, https://www.aclweb.org/anthology/n16-1030 lin, y., sun, c., xiaolong, w., xuan, w.: combining self learning and active learning for chinese named entity recognition. j. softw. 5, may 2010. https://doi.org/10.4304/jsw.5.5.530-537 ma, x., hovy, e.: end-to-end sequence labeling via bi-directional lstm-cnns-crf. in: proceedings of the 54th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1064–1074. association for computational linguistics, berlin, august 2016. https://doi.org/10.18653/v1/p16-1101 miyato, t., dai, a.m., goodfellow, i.: adversarial training methods for semi-supervised text classification. in: international conference on learning representations (iclr) (2017) google scholar  paszke, a., et al.: pytorch: an imperative style, high-performance deep learning library. in: wallach, h., larochelle, h., beygelzimer, a., d’alché-buc, f., fox, e., garnett, r. (eds.) advances in neural information processing systems 32, pp. 8024–8035. curran associates, inc. (2019). http://papers.neurips.cc/paper/9015-pytorch-an-imperative-style-high-performance-deep-learning-library.pdf pennington, j., socher, r., manning, c.: glove: global vectors for word representation. in: proceedings of the 2014 conference on empirical methods in natural language processing (emnlp), pp. 1532–1543. association for computational linguistics, october 2014. https://doi.org/10.3115/v1/d14-1162 pradhan, s., et al.: towards robust linguistic analysis using ontonotes. in: proceedings of the seventeenth conference on computational natural language learning, pp. 143–152. association for computational linguistics, sofia, august 2013 google scholar  ratinov, l., roth, d.: design challenges and misconceptions in named entity recognition. in: proceedings of the thirteenth conference on computational natural language learning (conll-2009), pp. 147–155. association for computational linguistics, boulder, june 2009. https://www.aclweb.org/anthology/w09-1119 sang, e.f.t.k., meulder, f.d.: introduction to the conll-2003 shared task: language-independent named entity recognition. in: proceedings of the seventh conference on natural language learning at hlt-naacl 2003, pp. 142–147 (2003) google scholar  settles, b., craven, m.: an analysis of active learning strategies for sequence labeling tasks. in: proceedings of the conference on empirical methods in natural language processing, emnlp 2008, pp. 1070–1079. association for computational linguistics, usa (2008) google scholar  shen, y., yun, h., lipton, z., kronrod, y., anandkumar, a.: deep active learning for named entity recognition. in: proceedings of the 2nd workshop on representation learning for nlp, pp. 252–256. association for computational linguistics, vancouver, august 2017. https://doi.org/10.18653/v1/w17-2630 siddhant, a., lipton, z.c.: deep bayesian active learning for natural language processing: results of a large-scale empirical study. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 2904–2909. association for computational linguistics, brussels, october–november 2018. https://doi.org/10.18653/v1/d18-1318 tran, v.c., nguyen, n.t., fujita, h., hoang, d.t., hwang, d.: a combination of active learning and self-learning for named entity recognition on twitter using conditional random fields. knowl.-based syst. 132, 179–187 (2017). https://doi.org/10.1016/j.knosys.2017.06.023 article  google scholar  zhu, j., wang, h., hovy, e., ma, m.: confidence-based stopping criteria for active learning for data annotation. acm trans. speech lang. process. 6(3), april 2010. https://doi.org/10.1145/1753783.1753784 download references acknowledgements the authors are supported by the fundação de apoio a pesquisa do distritio federal (fap-df) as members of the knowledge extraction from documents of legal content (knedle) project from the university of brasilia. author information authors and affiliations university of brasilia, brasilia, df, brazil josé reinaldo c. s. a. v. s. neto & thiago de paulo faleiros authors josé reinaldo c. s. a. v. s. netoview author publications search author on:pubmed google scholar thiago de paulo faleirosview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper neto, j.r.c.s.a.v.s., faleiros, t.d.p. (2021). deep active-self learning applied to named entity recognition. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_28 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep active learning self learning named entity recognition deep learning publish with us policies and ethics profiles thiago de paulo faleiros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ensemble of cnns for enhanced leukocyte classification in acute myeloid leukemia diagnosis | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ensemble of cnns for enhanced leukocyte classification in acute myeloid leukemia diagnosis conference paper first online: 30 january 2025 pp 399–413 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) ensemble of cnns for enhanced leukocyte classification in acute myeloid leukemia diagnosis download book pdf download book epub leonardo p. sousa9, romuere r. v. silva9, maíla l. claro10, flávio h. d. araújo9, rodrigo n. borges9, vinicius p. machado9 & … rodrigo m. s. veras9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 475 accesses 3 citations abstract acute myeloid leukemia (aml) is one of the most lethal and aggressive forms of hematological cancer, characterized by the rapid proliferation of immature leukocytes. this disease is diagnosed by highly trained specialists who meticulously analyze microscopic images of blood smears. this work explores the feasibility and effectiveness of using eight convolutional neural network (cnn) architectures to form specialized ensembles capable of accurately differentiating between mature and immature leukocytes. we used voting ensemble techniques and the bagging method to integrate cnns that achieved the best individual performances. the bagging strategy, explicitly using the efficientnet b3 cnn, stood out by achieving an accuracy of 96.62%, a precision of 98.11%, and a kappa index of 92.27% on a dataset of 48,100 blood cell images. this performance enhancement highlights the superior diagnostic capabilities of this approach compared to the individual architectures of cnns in identifying cell types in the context of aml diagnosis. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others detection of acute myeloid leukemia from peripheral blood smear images using transfer learning in modified cnn architectures chapter © 2023 acute myeloid leukemia subtype classification using single blood cell images article open access 09 december 2025 a convolutional neural network–based learning approach to acute lymphoblastic leukaemia detection with automated feature extraction article 06 november 2020 1 introduction blood plays a crucial role in transporting oxygen and nutrients to all organs and comprises three main types of cells: erythrocytes, platelets, and leukocytes. erythrocytes are responsible for the transport of gases such as oxygen and carbon dioxide; platelets are essential in the blood clotting process [16]; and leukocytes, or white cells, are tasked with defending the organism. additionally, leukocytes can be classified as mature (fully developed cells) or immature (cells in the development process). all these cells originate in the bone marrow and are regularly released into the bloodstream. the healthy development of blood cells can be compromised by the disproportionate increase in abnormal blood cells, becoming the primary cause of blood cancer. leukemia originates when the malignancy of blood precursor cells manifests through their uncontrolled proliferation, primarily affecting leukocytes [27]. leukemias can be categorized as either acute or chronic. the acute form manifests rapidly, requiring immediate therapeutic intervention, while chronic leukemias progress more slowly, often not requiring immediate treatment after diagnosis. they can also be grouped based on the types of white blood cells they affect: lymphoid or myeloid [15]. thus, the main types of leukemia are acute lymphoblastic leukemia (all), acute myeloid leukemia (aml), chronic myeloid leukemia (cml), and chronic lymphocytic leukemia (cll). aml stands out as the most lethal variant among the four leukemia subtypes, with approximately 20,380 new cases and about 11,300 deaths in the united states in 2023, according to data from the national cancer institute. the five-year average survival rate is 31.7%footnote 1. from 2009 to 2019, it was the most prevalent leukemia in brazil, with 10,554 casesfootnote 2. the disease carries an unfavorable prognosis, and any delay in accurate diagnosis has a severe and negative impact on the patient’s survival capacity. currently, no specific screening methods are available for early detection of leukemia before the onset of symptoms. physicians advise that vulnerable individuals undergo regular medical examinations, including physical assessments and routine blood tests. the detection of an increase in leukocyte count during these tests may indicate the presence of various factors, such as infection, stress, inflammation, or, in some cases, bone marrow disorders, including the possibility of leukemia. subsequently, the blood smear examination is conducted by trained technical personnel to carry out morphological analyses, count, and identify anomalies in blood cells. this process prioritizes the detection of specific characteristics associated with each type of leukemia. according to sadek et al. [19], the diagnostic criterion for aml is established when the count of blasts (immature leukocytes) reaches or exceeds 20%. figure 1 displays examples of mature and immature leukocytes. biermann et al. [3] described that mature leukocytes typically exhibit more segmented and ring-shaped nuclei, while immature ones have larger, more rounded, and less segmented nuclei. considering the low survival rate, especially among adults diagnosed with aml, and the inherent challenges of manual leukocyte classification for diagnosing aml, it becomes crucial and indispensable to develop tools that utilize computational technology and advances in machine learning methods. these tools aim to assist specialist technicians in validating and supporting the diagnostic process. convolutional neural networks (cnns) have demonstrated exceptional performance in image classification. in this study, in addition to analyzing individual networks, we explored various ensemble techniques, including majority voting, weighted voting and bagging. we chose to use distinct architectures, selected for their topological innovations and methods, to optimize image classification. specifically, we propose an ensemble using the bagging technique with the efficientnet b3 architecture to classify mature and immature leukocytes in blood smear images aimed at detecting aml. for analysis, we constructed a dataset by combining information from three public datasets, totaling 48,100 images, enhancing our model’s diversity and robustness. fig. 1. examples of (a) mature and (b) imature leukocytes. full size image this paper is organized as follows: sect. 2 presents related work, citing the most relevant studies. section 3 introduces the image dataset, the techniques applied, and the evaluation metrics adopted in the study. section 4 presents the results and discussions, and sect. 5 discusses the conclusions and future work. 2 related work in recent years, numerous studies in the literature have explored image datasets to classify various types of leukemia, employing machine learning (ml) and deep learning (dl) techniques. the principal methodologies identified are discussed below. an example of leukemia classification is found in works such as ahmed et al. [2] and aftab et al. [1], which address the four basic types of leukemia: aml, cml, cll, all, in addition to a class representing images without leukemia. on the other hand, there are papers like claro et al. [5] that focus exclusively on classifying the two types of acute leukemias (aml and all), along with a class representing images without leukemia. another approach for diagnosing aml involves classifying and counting different types of leukocytes, focusing on immature leukocytes in blood samples. however, this approach still needs to be explored, with few publications addressing binary classification. some of the works using this approach are discussed below. khanam et al. [10] presents an advanced approach for the diagnosis of aml using the ancom convolutional neural network, aiming to identify all aml subtypes and healthy leukocytes automatically. the study utilized an image set from johns hopkins hospital, totaling 18,365 images. random sampling and data augmentation techniques were employed to balance the dataset, achieving an accuracy of 95% for binary classification. rahman and ahmad [18] propose a methodology to precisely detect immature leukocytes using the convolutional neural network architectures alexnet, resnet50, densenet161, and vgg-16. after training and validation with the optimization of various parameters, the best-performing model was the modified alexnet, achieving an accuracy of 96.52%, an area under the curve (auc) of 94.94%, and an f1-score of 97.00%. they utilized a dataset from johns hopkins hospital with 18,365 images, consisting of 3,532 immature leukocytes and 14,833 mature leukocytes. dasariraju et al. [7] began their research with binary classification distinguishing between mature and immature leukocytes. positive results for immature leukocytes were further subdivided into four classes: erythroblasts, monoblasts, promyelocytes, and myeloblasts. the lmu-db dataset (university hospital of munich), consisting of 15,192 images, was used. the procedure started with the segmentation of the nucleus using multi-otsu thresholding. subsequently, features were extracted, and an svm-based classifier was applied. the achieved results show an accuracy of 92.99% for binary classification. the three studies, as presented in table 1, address the binary classification of leukocytes using different deep-learning techniques and methodologies. khanam et al. [10] utilize the ancom neural network, achieving an accuracy of 95%. rahman and ahmad [18] implement transfer learning with a modified alexnet, achieving an accuracy of 96.52%. meanwhile, dasariraju et al. [7] apply an svm-based classifier, obtaining an accuracy of 92.99%. the main deficiencies include data imbalance, model complexity, lack of comprehensive comparisons with other methodologies, and the use of only a single fold in testing, which may limit the robustness and generalization of the results. table 1. comparative summary of related works.full size table 3 materials and methods to conduct a comparative analysis of acute myeloid leukemia (aml) image classifications, we evaluated pre-trained convolutional neural networks. we selected eight distinct architectures: densenet201, efficientnet b3, inceptionv3, resnet50, resnet101, vgg16, vgg19, and xception. this choice was based on a meticulous review of the literature, in which each of these architectures demonstrated remarkable capabilities in image classification tasks. the diversity of the architectures allows for the exploration of a wide range of characteristics of leukocyte images, essential for enhancing the precision of classification between mature and immature cells. 3.1 proposed method after analyzing the results obtained from both individual cnns and committees, the proposed approach was developed as illustrated in fig. 2, where we adopted a bagging ensemble technique to improve image classification using the efficientnet b3 architecture, which achieved the best results in leukocyte classification. initially, all images are standardized to 300\(\,\times \,\)300 pixels, ensuring consistency in the input data. we used the bootstrap method to train ten distinct models (n = 10), each on randomly selected data subsets with replacement, processed over 100 epochs in batches of 64 images. we also implemented an early stopping mechanism that monitors validation loss to prevent overfitting. if loss does not improve after a specific number of epochs, training is stopped, and the weights of the best-performing model are saved. this strategy saves training time and ensures that the models are optimally trained without overfitting. during the testing phase, each model in the ensemble makes independent predictions, and the final classification is determined by averaging these predictions using a decision threshold of 0.5. the final classification distinguishes between mature and immature leukocytes. this approach leverages the diversity of the models in the committee and enhances the robustness of the classification system, ensuring an accurate analysis of leukocyte maturation. fig. 2. proposed method using a bagging ensemble. full size image 3.2 image dataset this study combined three datasets, resulting in 48,100 images categorized into mature and immature classes. table 2 illustrates the combination of images used. the johns hopkins hospital dataset, as described in sidhom et al. [20], contains images of blood smears featuring different types of individual leukocytes (.jpg format, 360\(\,\times \,\)360 pixels) from 106 patients diagnosed with aml and acute lymphoblastic leukemia (all) across various age groups and genders. initially, the images were organized by the patient, but they were reorganized by type of leukocyte since the leukocyte identification is present in each patient’s folder. this reorganization process resulted in a total of 18,365 images. in the dataset from the university hospital of munich, as referenced in matek et al. [14], there are 15,192 images of blood smears with individual leukocytes (.tiff format, 400\(\,\times \,\)400 pixels) from 100 patients diagnosed with aml. these data were collected between 2014 and 2017, covering 13 classes of images. the dataset from the central laboratory of the clinical hospital of barcelona, described by boldu et al. [4], consists of 14,543 images of individual cells. the dataset is organized into eight distinct groups: neutrophils, eosinophils, basophils, lymphocytes, monocytes, immature granulocytes (promyelocytes, myelocytes, and metamyelocytes), erythroblasts, and platelets or thrombocytes. the images are 360\(\times \)363 pixels in size, in .jpg format, and have been annotated by specialized clinical pathologists. table 2. summary of the image datasets used.full size table 3.3 evaluated cnn architectures cnns have been widely employed in the field of machine learning. thanks to their deep architectures, cnns can map image features at various levels of abstraction, which has significantly contributed to the development of more accurate medical diagnostic tools. in many cases, cnns surpass the accuracy of conventional feature extraction methods [21], thus demonstrating their potential and effectiveness in analyzing and interpreting medical images. cnns designed for the imagenet large scale visual recognition challenge (ilsvrc) were evaluated. in kornblith et al. [11], the authors concluded that the better a cnn architecture adapts to the imagenet dataset, the better it will transfer to other datasets. the architectures evaluated for ensemble formation are indicated in table 7, which refers to the topological depth of the network, number of parameters, input resolution, and year of publication for each (table 3). table 3. characteristics of the evaluated architectures.full size table 3.4 transfer learning transfer learning is a method where knowledge gained by a neural network on one task is used to improve performance on another related task. this technique reduces the need to re-adjust all the parameters of the cnn from scratch [22]. the proposed method utilizes pre-trained convolutional neural networks to identify mature and immature leukocytes. during development, eight neural networks were evaluated using shallow fine-tuning (sft), which involves freezing layers from the beginning of the cnn to extract features from the input images and classify them. the network architecture was first loaded to implement this, excluding the dense layers at the top. subsequently, the pre-trained convolutional layers were frozen. custom dense layers were then added to the convolutional layers to tailor the model to the specific needs of binary leukocyte classification. finally, the model was compiled using the adam optimizer, the binary cross entropy loss function, and the accuracy metric, preparing it for training. 3.5 ensembles of classifiers the ensemble approach, also known as the ensemble technique, is a method that existed long before the advent of the deep learning paradigm [6]. the theory behind it is pretty simple and is based on the well-known notion of the “wisdom of crowds”: instead of relying on just one model for prediction, a set of multiple (pre-trained) models is created. the results of these models are combined into a final classification through a voting mechanism. the original idea was developed to reduce the variance of classifiers, aiming to achieve better overall performance [8]. constructing an ensemble of classifiers involves three main phases: generating base classifiers, selecting ensemble members, and defining the decision mechanism [26]. in addition to the individual analysis of convolutional neural networks (cnns), this study created ensembles using the voting method, which incorporates both majority and weighted voting. the bagging technique was also employed to enhance the diversity and robustness of the classifiers in the tests. voting is a fundamental ensemble technique that aggregates the predictions of several models to improve classification performance. in this approach, each model independently predicts the output, and the final decision is made based on either the majority vote or the weighted vote of all models. in the majority voting scheme, each classifier votes for a specific class, and the class receiving the most votes is selected as the final output. this technique is particularly effective when the individual classifiers are diverse, and their errors are uncorrelated [8]. in the weighted voting scheme, different weights are assigned to each classifier based on their performance. this ensures that more reliable classifiers have a greater influence on the final decision, potentially improving overall accuracy [12]. bagging is an algorithm framework that trains several different models respectively and then lets all models vote to test the output of samples [17]. as shown in fig. 3 of zhang et al. [23], bagging adopts a sampling with replacement to generate multiple training subsets, which are employed to train classifiers [25]. each training process is independent, so parallel computing could accelerate the process [9]. particularly, the training subset in bagging is selected randomly, meaning different subsets can contain the same data. additionally, bagging introduces randomization in the training process of each classifier. after training, all classifiers are combined to reduce the variance in prediction results. fig. 3. structure of bagging classifier. adapted from zhang et al. [23]. full size image 3.6 evaluation metrics the technique of stratified k-fold cross-validation was used. this approach randomly distributes instances from the dataset into k subsets (or folds), which are mutually exclusive and of approximately equal size while maintaining the same proportion observed in the original dataset. thus, the cnn model is fitted and tested k times, and in each iteration, a different subset is reserved for evaluation. in contrast, the remaining k-1 subsets are used for fine-tuning the network parameters. this technique is valuable for assessing the model’s performance across different datasets and can provide a more reliable estimate of its generalization ability. the image dataset was divided into five folds (k = 5), resulting in 20% of the instances comprising the test subset. this subset is not used during the network training but is employed in the final evaluation of the classifiers’ performance. the images in the remaining folds were divided into two subsets: the training subset, used to adjust the network weights based on the loss observed during training, and the validation subset, which comprises 20% of the total training subset. the latter monitors the training process and identifies potential overfitting issues. values from the confusion matrix were used to evaluate the methodology’s performance. from these values, five metrics widely recognized in the literature were calculated: accuracy (a), precision (p), recall (r), f1-score (f), and kappa index (k). the selection of these metrics is justified by their importance in providing a holistic view of the model’s performance, as highlighted by zhang et al. [24]. these metrics are crucial for comprehensively testing machine learning models, enabling not only the assessment of overall effectiveness (accuracy) but also the precision and the model’s ability to identify all relevant positive instances (recall), balance these measures (f1-score), and measure agreement adjusted for chance (kappa). the kappa index is recommended as an appropriate measure of accuracy because it can adequately represent the confusion matrix. this index considers all elements of the confusion matrix, not just those on the main diagonal, unlike overall classification accuracy, which considers only the main diagonal elements. the kappa coefficient is calculated using the formula 1: $$\begin{aligned} k = \frac{(observed expected)}{1-expected} \, \times \, 100 \end{aligned}$$ (1) according to [13], the value of k can be interpreted in the following way: k \(\le 20\%\): poor; \(20\%\) < k \(\le 40\%\): fair; \(40\%\) < k \(\le 60\%\): good; \(60\%\) < k \(\le 80\%\): very good; and k > \(80\%\): excellent. 4 results and discussions from the fine-tuning techniques presented, experiments were conducted to determine the most effective approach for the problem at hand. during the experiments, the selected evaluation metrics were considered. the k-fold cross-validation method was adopted with k equal to 5 to ensure a robust evaluation of the models. through empirical evaluation, different architectures were tested to optimize the obtained results. all experiments were performed on a computer with an intel(r) core(tm) i5 processor at 2.90 ghz, 16 gb of ram, and an nvidia zotac geforce rtx 3060 12 gb graphics card with 3584 cores. 4.1 individual classification this section presents the individual results obtained with the eight evaluated architectures. the networks were trained for 100 epochs using a batch size of 128 images. a parameter was used to monitor the network’s learning progress, setting a patience value of 15 epochs. if accuracy did not improve after this period, the training was terminated. additionally, during the cnns’ training, the binary cross-entropy cost function and the adam optimizer were employed. the dense layers at the network’s top were not included, and the pre-trained convolutional layers were frozen. two custom dense layers were added for the binary classification step: the first composed of 256 neurons with relu activation, followed by an output layer with one neuron and sigmoid activation. the results obtained are described in table 4, highlighting efficientnet b3 as the architecture with the best performance, achieving an accuracy of 94.21%, precision of 95.74%, recall of 95.76%, f1-score of 95.75%, and a kappa index of 86.69%, which is considered excellent. including the squeeze-and-excitation (se) attention mechanism in efficientnet b3 likely contributed significantly to its superior performance. this mechanism enhances the network’s ability to focus on the most relevant features within an image by adaptively recalibrating channel-wise feature responses, leading to more effective feature representation and extraction. the standard deviation analysis showed values close to 0, indicating little variation in results across the folds. these results surpassed those of the other individual architectures, demonstrating the effectiveness of the se module in improving the network’s accuracy and robustness. table 4. results of the evaluated architectures.full size table 4.2 ensemble of cnns we conducted 120 experiments to evaluate the robustness of various convolutional neural network (cnn) architectures in an ensemble context. we explored combinations of three networks, selected from eight possibilities, using majority and weighted voting methods. additionally, we implemented the bagging method on all eight networks evaluated to enhance the robustness of the classifications. the consolidated results indicated that the ensemble approach significantly improved performance metrics compared to individual classifiers. however, the bagging strategy stood out, surpassing the ensembles that employed voting methods and achieving superior results. the analysis of table 5 reveals that forming ensembles by combining different convolutional neural network (cnn) architectures and using voting methods significantly increased classification metrics. notably, the ensemble composed of densenet 201, efficientnet b3, and resnet 50, using majority voting, achieved the best results: an accuracy of 94.93%, a precision rate of 96.21%, a recall of 96.30%, an f1-score of 96.27%, and a cohen’s kappa index of 88.35%. these results substantially surpass the metrics achieved by the individual networks, highlighting the superior efficacy of the ensemble approach. table 5. best results for the evaluated cnn ensemble.full size table table 6 illustrates the best results achieved with bagging ensemble, applied to various individual neural networks. among them, the set that used the efficientnet b3 network stood out notably, achieving an accuracy of 96.62%, precision of 98.11%, recall of 96.89%, f1-score of 97.50%, and a kappa coefficient of 92.27%. the superiority of efficientnet b3 can be largely attributed to its attention layer, which allows the model to focus on the most relevant features within complex images, significantly enhancing classification precision. furthermore, a low standard deviation in the results obtained through bagging must be considered, as it indicates high consistency and reliability of the classification metrics. by combining multiple instances of the same network, the bagging method forms a more stable model that better resists data variability and minimizes the chances of overfitting. this stability is enhanced by the attention layer of efficientnet b3, which enables each model in the ensemble to concentrate on different nuances of the data efficiently. this combination of bagging and focused attention contributes to stronger generalization and more accurate classification. table 6. best results for the bagging ensemble.full size table figure 4 illustrates a comparative that synthesizes the top performances from individual, ensemble, and bagging approaches, providing a clear visual representation of how different methodologies enhance the effectiveness of convolutional neural network (cnn) architectures. the chart highlights that the bagging approach using efficientnet b3 stands out across all metrics, achieving superior accuracy, precision, recall, f1-score, and kappa coefficient. this method exhibits exceptional stability and minimal variability, underscoring its robustness against data inconsistencies and ability to maintain high performance. fig. 4. performance comparison of cnn models across different methods. full size image the data presented in table 7 provide a clear comparative view of the advancements achieved in the methodologies for classifying leukocyte images. our research, which implemented a bagging strategy using a combination of three public datasets, achieved an impressive accuracy of 96.62%. this not only surpasses the accuracy levels reported in other studies but also marks a significant milestone by utilizing a considerably larger dataset of 48,100 images. unlike previous studies, such as khanam et al. [10], who used the ancom architecture on 18,365 images to achieve a 95% accuracy, or rahman and ahmad [18] who modified alexnet to achieve a slightly higher accuracy of 96.52%, our approach significantly expands the size and diversity of the dataset. the use of machine learning by dasariraju et al. [7] with random forest on the same number of images as the first two studies, but achieving only 92.99% accuracy, further underscores the effectiveness of our method. the substantial increase in the number of images in our study not only contributes to the robustness and generalization of the classification model but also enhances the heterogeneity and diversity of the data. table 7. comparison of state-of-the-art and the proposed method: architectures used, number of images, and accuracy achieved.full size table when considering computational efficiency, it’s clear that despite its higher accuracy, the cnn ensemble naturally requires more computational resources for training and inference. in comparison, efficientnet b3, the top-performing individual network, provides a more resource-efficient and faster option. the decision between these approaches should weigh the need for diagnostic accuracy against the available resources, especially in clinical environments where speed and efficiency are critical. 5 conclusion the implementation of ensemble methods using bagging with the efficientnet b3 architecture has shown substantial promise in improving the accuracy of leukocyte classification for the early detection of aml. our study demonstrated significant improvements in performance metrics compared to individual models. using a diverse dataset comprising over 48,100 images from three different public image collections provided robustness in the training and validation of the models. this extensive dataset allowed a thorough evaluation of the ensemble’s performance under various image conditions, ensuring the generalization and reliability of our findings. the promising results obtained in this study suggest that ensemble methods can be a significant step in developing more accurate diagnostic tools in medical imaging. integrating multiple networks into a bagging ensemble reduced variance and enhanced predictive reliability, which is crucial for clinical applications. while our research utilizing a bagging strategy with the efficientnet b3 architecture achieved impressive precision, there are several opportunities for enhancement. despite the substantial size of our dataset, which includes 48,100 images, further diversification of data sources is essential to improve the model’s robustness and its ability to generalize across different clinical scenarios. integrating generative networks or large vision models (lvms) during the transfer learning stage could significantly enhance feature robustness. additionally, employing advanced ensemble techniques such as boosting and stacking may further refine model accuracy and adaptability. these techniques focus on improving training on challenging instances and synthesizing predictions from diverse models, which could lead to more precise and reliable diagnostic tools in medical imaging. notes 1.https://seer.cancer.gov/statfacts/html/amyl.html. 2.https://www.htct.com.br/pt-leucemia-mieloide-aguda-perfil-clinico-epidemiologico-articulo-resumen-s253113792101018x. references aftab, m.o., awan, m.j., khalid, s., javed, r., shabir, h.: executing spark bigdl for leukemia detection from microscopic images using transfer learning. in: 2021 1st international conference on artificial intelligence and data analytics (caida). pp. 216–220. ieee (2021) google scholar  ahmed, n., yigit, a., isik, z., alpkocak, a.: identification of leukemia subtypes from microscopic images using convolutional neural network. diagnostics 9(3), 104 (2019) article  google scholar  biermann, h., pietz, b., dreier, r., schmid, k.w., sorg, c., sunderkötter, c.: murine leukocytes with ring-shaped nuclei include granulocytes, monocytes, and their precursors. j. leukoc. biol. 65(2), 217–231 (1999) article  google scholar  boldú, l., merino, a., acevedo, a., molina, a., rodellar, j.: a deep learning model (alnet) for the diagnosis of acute leukaemia lineage using peripheral blood cell images. comput. methods programs biomed. 202, 105999 (2021) article  google scholar  claro, m., vogado, l., veras, r., santana, a., tavares, j., santos, j., machado, v.: convolution neural network models for acute leukemia diagnosis. in: 2020 international conference on systems, signals and image processing (iwssip). pp. 63–68. ieee (2020) google scholar  dasarathy, b.v., sheela, b.v.: a composite classifier system design: concepts and methodology. proc. ieee 67(5), 708–713 (1979) article  math  google scholar  dasariraju, s., huo, m., mccalla, s.: detection and classification of immature leukocytes for diagnosis of acute myeloid leukemia using random forest algorithm. bioengineering 7(4), 120 (2020) article  math  google scholar  dietterich, t.g.: ensemble methods in machine learning. in: international workshop on multiple classifier systems. pp. 1–15. springer (2000) google scholar  khan, f.h., saadeh, w.: an eeg-based hypnotic state monitor for patients during general anesthesia. ieee transactions on very large scale integration (vlsi) systems 29(5), 950–961 (2021) google scholar  khanam, a.s., khan, m.h., tabassum, f., ripa, t.a., jabid, t., ali, m.s., islam, m., ovi, j.a., hasan, m.: classification of acute myeloid leukemia using convolutional neural network (2023) google scholar  kornblith, s., shlens, j., le, q.v.: do better imagenet models transfer better? in: proceedings of the ieee/cvf conference on computer vision and pattern recognition. pp. 2661–2671 (2019) google scholar  kuncheva, l.i.: combining pattern classifiers: methods and algorithms. john wiley & sons (2014) google scholar  landis, j.r., koch, g.g.: the measurement of observer agreement for categorical data. biometrics 33(1), 159–174 (1977) article  math  google scholar  matek, c., schwarz, s., spiekermann, k., marr, c.: human-level recognition of blast cells in acute myeloid leukaemia with convolutional neural networks. nature machine intelligence 1(11), 538–544 (2019) article  google scholar  mrozek, k., heerema, n.a., bloomfield, c.d.: cytogenetics in acute leukemia. blood rev. 18(2), 115–136 (2004) article  math  google scholar  nascimento, m.d.l.p.: a interação entre as células hematológicas durante as suas atividades. concluído em (2017) google scholar  nguyen, t.t., pham, x.c., liew, a.w.c., pedrycz, w.: aggregation of classifiers: a justifiable information granularity approach. ieee transactions on cybernetics 49(6), 2168–2177 (2019) article  math  google scholar  rahman, j.f., ahmad, m.: detection of acute myeloid leukemia from peripheral blood smear images using transfer learning in modified cnn architectures. in: proceedings of international conference on information and communication technology for development: icictd 2022. pp. 447–459. springer (2023) google scholar  sadek, n.a., abd-eltawab, s.m., assem, n.m., hamdy, h.a., el-sayed, f.m., ahmad, m.a.r., elhamammy, r.h.: prognostic value of absolute lymphocyte count, lymphocyte percentage, serum albumin, aberrant expression of cd7, cd19 and the tumor suppressors (pten and p53) in patients with acute myeloid leukemia. asian pacific journal of cancer biology 5(4), 131–140 (2020) google scholar  sidhom, j.w., siddarthan, i.j., lai, b.s., luo, a., hambley, b.c., bynum, j., duffield, a.s., streiff, m.b., moliterno, a.r., imus, p., et al.: deep learning for diagnosis of acute promyelocytic leukemia via recognition of genomically imprinted morphologic features. npj precision oncology 5(1), 38 (2021) article  google scholar  tajbakhsh, n., shin, j.y., gurudu, s.r., hurst, r.t., kendall, c.b., gotway, m.b., liang, j.: convolutional neural networks for medical image analysis: full training or fine tuning? ieee trans. med. imaging 35(5), 1299–1312 (2016) article  google scholar  yosinski, j., clune, j., bengio, y., lipson, h.: how transferable are features in deep neural networks? advances in neural information processing systems 27 (2014) google scholar  zhang, h., zhou, t., xu, t., hu, h.: remote interference discrimination testbed employing ai ensemble algorithms for 6g tdd networks. sensors 23, 2264 (02 2023) google scholar  zhang, j.m., harman, m., ma, l., liu, y.: machine learning testing: survey, landscapes and horizons. ieee trans. software eng. 48(1), 1–36 (2020) article  math  google scholar  zhang, l., suganthan, p.n.: benchmarking ensemble classifiers with novel co-trained kernel ridge regression and random vector functional link ensembles [research frontier]. ieee comput. intell. mag. 12(4), 61–72 (2017) article  google scholar  zhou, z.h.: ensemble methods: foundations and algorithms (2012) google scholar  zolfaghari, m., sajedi, h.: a survey on automated detection and classification of acute leukemia and wbcs in microscopic blood cells. multimedia tools and applications 81(5), 6723–6753 (2022) article  math  google scholar  download references author information authors and affiliations federal university of piauí, teresina, brazil leonardo p. sousa, romuere r. v. silva, flávio h. d. araújo, rodrigo n. borges, vinicius p. machado & rodrigo m. s. veras federal institute of piauí, paulistana, brazil maíla l. claro authors leonardo p. sousaview author publications search author on:pubmed google scholar romuere r. v. silvaview author publications search author on:pubmed google scholar maíla l. claroview author publications search author on:pubmed google scholar flávio h. d. araújoview author publications search author on:pubmed google scholar rodrigo n. borgesview author publications search author on:pubmed google scholar vinicius p. machadoview author publications search author on:pubmed google scholar rodrigo m. s. verasview author publications search author on:pubmed google scholar corresponding author correspondence to leonardo p. sousa . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sousa, l.p. et al. (2025). ensemble of cnns for enhanced leukocyte classification in acute myeloid leukemia diagnosis. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_28 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords convolutional neural networks (cnns) leukocyte classification acute myeloid leukemia diagnosis publish with us policies and ethics profiles rodrigo m. s. veras view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature going bananas! unfolding program synthesis with origami | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper going bananas! unfolding program synthesis with origami conference paper first online: 30 january 2025 pp 3–18 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) going bananas! unfolding program synthesis with origami download book pdf download book epub matheus campos fernandes  orcid: orcid.org/0000-0002-5118-23999, fabrício olivetti de frança  orcid: orcid.org/0000-0002-2741-87369 & emilio francesquini  orcid: orcid.org/0000-0002-5374-25219  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 386 accesses abstract automatically creating a computer program using input-output examples can be a challenging task, especially when trying to synthesize computer programs that require loops or recursion. even though the use of recursion can make the algorithmic description more succinct and declarative, this concept creates additional barriers to program synthesis algorithms such as the creation and the attempt to evaluate non-terminating programs. one reason is that the recursive function must define how to traverse (or generate) the data structure and, at the same time, how to process it. in functional programming, the concept of recursion schemes decouples these two tasks by putting a major focus on the data processing. this can also help to avoid some of the pitfalls of recursive functions during program synthesis, as argued in the paper that introduced the origami technique. the authors showed how this technique can be effective in finding solutions for programs that require folding a list. in this work, we incorporate other recursion schemes into origami, such as accumulated folding, unfolding, and the combination of unfolding and folding. we evaluated it on the 29 problems of the standard general program synthesis benchmark suite 1, obtaining favorable results against other well-known algorithms. overall, origami achieves the best results in \(25\%\) more problems than its predecessor (hotgp) and an even higher increase when compared to other approaches. not only that, but it can also consistently find a solution to problems for which other concurrent algorithms report a low success rate. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others origami: (un)folding the abstraction of recursion schemes for program synthesis chapter © 2024 abstract origami chapter © 2020 introduction to active origami structures chapter © 2019 1 introduction program synthesis (ps) is the task of automatically generating a computer program written in a programming language of choice using some form of specification [6]. this can be framed as a search problem that explores the space of possible programs under a set of constraints, usually as a grammar and a maximum program size, that remove undesirable solutions from the search space. a convenient format for the specification is a set of input-output examples that demonstrate the expected outputs for several different input cases. in this particular case, the task is called inductive synthesis or programming-by-example (pbe). the main advantage of this approach is that sets of examples are easy to create, and often do not require deep knowledge of the problem in question. however, the lack of corner or special cases in the set allows for the generation of programs that do not follow the original intent of the user. among the vast selection of possible search algorithms, we highlight genetic programming (gp) [10]. gp is a search algorithm that tries to balance the exploration and exploitation of the search space using recombination and perturbation of a bag of solutions. it employs selective pressure at every step, inspired by the evolution of species, to favor the fittest solutions when replicating and applying such operators. some benchmark problems extracted from common programming tasks have been successfully solved by recent variations of the original gp algorithm, such as pushgp [8], cbgp [15], ge [13], g3p [3], and hotgp [1]. the representation of the solutions and the imposed constraints play a major role in search algorithms as they describe the navigability in this space and the coverage of possible programs. for example, the push language [17] is a stack-based programming language in which the operations are executed sequentially storing and retrieving values from the stack corresponding to the types of the operation. this provides additional flexibility during the execution of a computer program as any invalid operation can be skipped and, thus, the operation at any given timestep is not constrained by output type of the previous step. this increases the possibilities of navigation of the search space. on the other hand, in grammatical evolution [13] the search space is defined by the grammar being evolved. this can limit the size of the search space and ensure that each step of the generated program is valid. likewise, type-safe representations, such as in cbgp [15] and hotgp [1], carry at each node the information of the input and output types, constraining the space to valid programs (i.e., that execute without errors) and solutions that follow the type specification. a common challenge to gp algorithms is how to handle recursion or loops. the problem here lies in the fact that it is possible to generate an unnecessarily long or even unbounded recursion/loop. this is alleviated by the use of programming patterns that hide the loop or recursion behind a declarative language. popular examples of such patterns are the map, filter, and fold functions. in particular, fold describes a recursive pattern capable of expressing recursive algorithms that traverse a structure aggregating the partial results. examples of the use of this pattern are sum, product, and even insertion sort. a complementary pattern that describes the recursion that generates or builds a structure is known as unfold. for example, unfold can be used to generate the list of fibonacci numbers, or reversing the order of the elements of a list. although many algorithms can be described using folds or unfolds, this approach has some limitations such as working only on lists, or the inability to store partial results. a more general set of recursive patterns that involves folding and unfolding is known as recursion schemes [11]. the recursion schemes extend the common folding and unfolding operations to work on any inductive type and include additional mechanisms to handle a wider variety of recursive function patternsfootnote 1. these recursion schemes can be used to guide the synthesis of recursive programs, as their structure is well-defined and can be used as scaffolding, with variations only in certain parts of the program. origami [2] is an algorithm capable of synthesizing typed, pure, functional programs that support different recursion schemes. origami’s authors presented a proof-of-concept implementation, capable of partially synthesizing one single scheme. this implementation was evaluated, yielding promising preliminary results. in this paper, we provide and evaluate the first complete implementation of origami, following the description given in the original paper. this represents a meaningful step in assessing the effectiveness of integrating recursion schemes into program synthesis. we evaluate the performance of origami using the problems described in the general program synthesis benchmark suite 1 (psb1) [9], comparing the obtained results to well-known gp-based program synthesis algorithms. when comparing the success rate of each problem, this new version of origami achieves the best results in \(25\%\) more problems than its predecessor (hotgp) and an even higher increase when compared to other approaches. additionally, origami was capable of achieving a high success rate (higher than \(70\%\)) in problems that most algorithms achieve less than \(50\%\). this text is organized as follows. section 2 brings a brief literature review of functional program synthesis and recursion schemes. section 3 presents origami and describes the details of our implementation. an analysis of the results and comparisons of origami to hotgp and other well-known methods is shown in sect. 4. finally, sect. 5 concludes our work. 2 related work there have been several attempts to use functional programming as the basis of program synthesis. one of the earliest attempts employed types to guide the search [12]. more recently, the use of recursion schemes [18] and higher-order functions [1, 5, 14] to represent recursion were proposed. out of these approaches, the one that is more closely related to this work is hotgp [1], which is a gp algorithm that synthesizes pure, typed, and functional programs. its approach to recursion includes support for higher-order functions, \(\lambda \)-functions, and parametric polymorphism. notably, in [18] the authors presented the first algorithm for synthesizing programs that exploit recursion schemes. their work focuses only on catamorphisms over natural numbers using peano representation (i.e., the inductive type of natural numbers). the authors evaluate their approach with variations of the fibonacci sequence, successfully obtaining the correct programs. origami was originally proposed in [2] where the authors evaluated the feasibility of using recursion schemes to synthesize recursive programs. in that work, they showed that the entire psb1 benchmark can be solved by one of four different recursion schemes: catamorphism, accumulation, anamorphism, and hylomorphism. it also described preliminary experiments with catamorphism showing that, for the problems that are solvable with this scheme, the use of scaffolding improved the success rate when compared to hotgp. these results brought evidence that the synthesis process is simplified once the correct recursion scheme is determined, as it just needs to evolve non-recursive expressions. large language models as code assistants have also been employed for program synthesis. in [16], the authors use github copilot to synthesize programs from a textual description of the problem, obtaining correct programs more often than a selection of gp based synthesizers in 50% of the evaluated problems. 3 origami program synthesis origami’s implementation follows a koza-style genetic programming [10] (tree representation). the main distinctions to traditional approaches are the introduction of immutable nodes (ensuring a certain recursion scheme); and the type-safety of the genetic operators (the same approach taken by hotgp [1]). the implementation is based on patterns, which are used to represent different recursion schemes. a pattern is composed of immutable nodes and a set of evolvable slots that, when replaced with expressions, can be evaluated. the immutable nodes describe the main definition of the recursion scheme (see sect. 3.1) and are fixed once we choose the pattern, while the evolvable slots represent the inner mechanisms that need to be synthesized to correspond to the expected behavior described by the dataset. these slots have a well-defined output type (inferred from the problem description), and a well-defined set of bindings to which the expression has access. in this work we focus on the six different patterns that comprehend the minimal set required to solve psb1. naturally, origami is not limited to these patterns, and more could be included as needed. section 3.1 details these six patterns. due to space constraints, we assume the reader has a basic understanding of recursion schemes [11], how they can be used to solve psb1 [2], as well as the haskell language notation, which is similar to the ml notation. 3.1 patterns noscheme. this is the simplest pattern in origami, as it does not employ any recursion at all. it is represented by the following code: this pattern has just a single slot, which has all the arguments in scope and returns a value of the same type as the output of the program. its main use is to accommodate for problems that do not require any recursion. catamorphism over indexed list. this pattern captures the most common recursion scheme observed in psb1, and arguably in practical scenarios as well, i.e., folding a list from the right. in meijer-notation [11], this would be represented by the banana brackets \((\!|b, \oplus |\!)\), where b is the initial value and \(\oplus \) is the combining function. in the context of origami, it can be represented as: in a problem with arguments of type i\(_\texttt {0}\) \(\ldots \) i\(_\texttt {n}\) and of output type o, where 0 \(\equiv \) [e]footnote 2, this pattern’s slots are typed as follows:  : : o, with nothing in scope;  : : o, with scope { i : : int; x  : :  e; acc  : :  o; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\ldots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)}. this pattern will be referred simply as cata in the remainder of this paper. curried catamorphism over indexed list. this pattern captures a common variation of the catamorphism, and can be represented by the following code: as a problem of type i\(_\texttt {0}\) \(\texttt {->}\) i\(_\texttt {1}\) \(\texttt {->}\) o can also be seen in its curried form as i\(_\texttt {0}\) \(\texttt {->}\) (i\(_\texttt {1}\)  \(\texttt {->}\) o), we can employ catamorphism to accumulate a function over the first argument, and then apply this function to the second argument. this is useful when we need to apply a catamorphism over the zip of two lists [2]. in a problem with arguments of type i\(_{0}\) , \(\texttt {i}_\texttt {1}\)footnote 3 and of output type o, where i\(_\texttt {0}\) \(\equiv \) [e], this pattern’s slots are typed as follows:  : : o, with scope { ys  : :  i\(_\texttt {1}\) };  : : o, with scope { i : : int; x  : :  e; f  : :  i\(_\texttt {1}\)-> o; ys : : i\(_\texttt {1}\) }. for brevity, this will be referred to as simply curriedcata. note that both this and the previous pattern use indexed linked list as the data structure, allowing the program to access each element’s index and value. for the remaining patterns we employ a regular list since it is enough to solve their problems (as shown by the canonical solutions presented in [2]). anamorphism to a list. this pattern is commonly used in haskell as unfold, which is used to generate a list. in meijer-notation [11], this would be represented by the concave lenses where g is the generator function, and p is the predicate. in the context of origami, it can be represented by the following code: in a problem with arguments of type i\(_{\texttt {0}}\) \(\ldots \) i\(_\texttt {n}\) and of output type o, where o \(\equiv \) [e], this pattern’s slots are typed as follows:  : : i\(_\texttt {0}\), with scope { arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\ldots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : bool, with scope { seed  : :  i\(_\texttt {0}\) ; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\ldots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : e, with scope { seed  : :  i\(_\texttt {0}\) ; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\ldots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : i\(_\texttt {0}\), with scope { seed  : :  i\(_\texttt {0}\) }. note that while we do not enforce arg\(_\texttt {0}\) to be used in , it must be of the same type as arg\(_\texttt {0}\), as all of the solutions for psb1 respected this constraint. for brevity, this will be referred to as simply ana in the rest of this paper. accumulation over a list. this pattern captures using an accumulation strategy before using a foldr, and can be represented by the following code: in a problem with arguments of type i\(_\texttt {0}\) \(\ldots \) i\(_\texttt {n}\) and of output type o, where i\(_\texttt {0}\) \(\equiv \) [e], and given a type a, this pattern’s slots are typed as follows:  : : a, with scope { arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\ldots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : a, with scope { x  : :  e; xs  : :  [e]; s  : :  a; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : o, with scope { s  : :  a; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : o, with scope { x  : :  e; acc  : :  o; s  : :  a; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)}. this is the first pattern whose types are not fully determined by the type of the arguments and the expected output type: the accumulator type a. types such as this will be referred to as unbound types. to keep the implementation simple, we assume unbound types are known and provided by the user. the exploration of different types is an interesting challenge that warrants dedicated research. this pattern will be referred to as simply accu in the rest of this paper. hylomorphism through a list. this pattern captures an anamorphism followed by a catamorphism, such as applying foldr to the result of unfold, in haskell. in meijer-notation [11], this would be represented by the envelopes . in origami, it is represented by the following code: in a problem with arguments of type i\(_\texttt {0}\) \(\ldots \) i\(_\texttt {n}\) and of output type o, and given a type a, this pattern’s slots are typed as follows:  : : bool, with scope { seed  : :  i\(_\texttt {0}\) ; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : a, with scope { seed  : :  i\(_\texttt {0}\) ; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : i\(_\texttt {0}\), with scope { seed  : :  i\(_\texttt {0}\) ; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)};  : : o, with nothing in scope;  : : o, with scope { x  : :  a; acc  : :  o; arg\(_\texttt {0}\)  : :  i\(_\texttt {0}\) \(\dots \) arg\(_\texttt {n}\)  : :  i\(_\texttt {n}\)}. this pattern also contains an unbound type: the intermediary list has elements of type a. this pattern will be referred to as hylo. 3.2 genetic programming origami synthesizes the evolvable slots using a genetic programming (gp) [10] algorithm. since the patterns require more than a single slot, we represent each solution as a collection of programs represented as expression trees. each element of this collection corresponds to one of the slots. the gp starts with an initial random population of \(1\,000\) individuals, and iterates by applying either crossover to a pair of parents, or mutation to a single parent, generating \(1\,000\) new individuals in total. the entire population is replaced by the offspring population. the initial population is generated using a ramped half-and-half where half of the individuals are generated using the full method and half using the grow method. the maximum depth for each method varies between 1 and 5. the parental selection is performed using a tournament selection of size 10. following a simple gp algorithm, in origami the mutation randomly selects one of the evolvable slots, then picks one point in the tree at random to be replaced by a new subtree generated at random using the grow method, with a maximum depth of \(5 d_ current \). crossover also starts by picking one of the slots at random, then performing one of these two actions with equal probability: i) swap the entire slot of one parent with the same slot of the other parent; ii) swap two subtrees of the same output type from each parent. 3.3 grammar in hotgp [1], the selected grammar focused on providing a minimal set of operations, including higher-order functions, that would enable the synthesis of programs under the functional programming paradigm. with origami, however, the main focus is assessing recursion schemes as the only means of synthesizing recursive programs. therefore, we designed our grammar to avoid implicitly recursive functions, like map, filter, sum, and product. we acknowledge that this might remove shortcuts and potentially make the synthesis of certain problems harder. notice that the recursion happens in the immutable nodes that describe the recursion schemes, so the recursion is provided rather than evolved. additionally, the set of operations includes functions equivalent to those used by other methods, in particular those implemented by pushgp [8]. as a result, origami has a larger set of operations than hotgp. the full grammar is presented in table 1. once an execution is finished, the champion’s slots are refined using the same procedure that was used in hotgp [1]. to refine a tree, we pick the root node and check if replacing it with any of its children leads to a correctly-typed solution with an equal or better fitness. if so, we replace it with the best child; otherwise, we keep the original node. this process continues recursively, traversing the tree and greedily replacing nodes with their children when needed. this procedure applies occam’s razor to choose a simpler solution [7], making sure the fitness in the training set is never worse. 4 experimental results to evaluate our approach we conducted experiments to perform an automatic search for different patterns in the psb1 [9] context. for each of the 29 datasets, we sequentially tried each pattern in increasing order of complexity: noscheme; cata, if arg\(_\texttt {0}\) is a list; curriedcata, if the problem has two arguments and arg\(_\texttt {0}\) is a list; ana, if the return type is a list; accu, if arg\(_\texttt {0}\) is a list; hylo. for each dataset, we executed 30 seeds of each pattern starting from the simplest and testing other patterns if none of the seeds succeeded in finding a solution (i.e., the success rate was \(0\%\)). each seed followed the instructions provided by psb1, using the recommended number of training and test instances, and included the fixed edge cases in the training data, as well as using the fitness functions described in [9]. we also made the same adaptations to the benchmarks as in [2], and similar to [1] and [16]. specifically, we changed the input of the grade benchmark from (arg\(_\texttt {0}\) , arg\(_\texttt {1}\) , arg\(_\texttt {2}\) , arg\(_\texttt {3}\) , arg\(_\texttt {4}\) ) to ([(arg\(_\texttt {0}\) , ’a’), (arg\(_\texttt {1}\) , ’b’), (arg\(_\texttt {2}\) , ’c’), (arg\(_\texttt {3}\) , ’d’)], arg\(_\texttt {4}\) ); and, since we only generate pure programs, we adapted the output to return the results instead of printing them on: checksum, digits, even-squares, for-loop-index, grade, pig-latin, replace-space-w-nl., string-differences, syllables, and word-stats. note that we deliberately placed the patterns with unbound types at the end of the sequence. therefore, the unbound type in both accu and hylo is only decided after all other patterns have failed. for the benchmarks that origami failed to find a solution with the other patterns, we applied one of these two patterns choosing the type that was known to be correct according to the canonical solutions [2]. for the cases in which the canonical solutions did not use accu or hylo, we chose a reasonable type as needed (see table 2). table 1. the complete set of operations available for origami. each dataset only had access to the operations that involved its allowed types according to [9].full size table table 2. the chosen types for the unbound types in accu and hylo. the type is colored in blue when the decision was guided by the canonical solution.full size table the maximum tree depth was set to 5 for each slot. as origami is based in hotgp, which was empirically shown to be robust to changes to the crossover rate, we set it to the same value as hotgp (\(50\%\)). we allowed a maximum of \(300\,000\) evaluations with an early stop whenever the algorithm finds a perfectly accurate solution according to the training data. for patterns in which termination is not guaranteed, namely ana and hylo, a maximum number of iterations was imposed (empirically set to \(10\,000\)). non-termination is also an issue that can occur with curriedcata. specifically, origami was synthesizing solutions with the slot , essentially creating a “fork bomb”. to tackle this, a maximum execution budget is enforced when the evaluation of a single iteration of a slot executes more than \(10\,000\) operations. in this case, the program is assigned an infinitely bad fitness. this limit was reached by less than \(0.5\%\) of the individuals. table 3 shows the percentage of executions in which origami was able to synthesize a solution that completely solved the test set (i.e., success rate). origami found a solution for all of the problems that were canonically solved by noscheme as well as cata. surprisingly, it was also able to synthesize a solution for for-loop-index by using noscheme, even though the canonical solution used ana, and for grade by using cata when the canonical solution used curriedcata. nonetheless, we also ran these problems with their canonical patterns and discovered origami was also able to synthesize solutions, albeit less often. moreover, origami was able to find the solutions for 3 out of the 4 canonical curriedcata problems, and 2 out of the 3 ana problems. accu and hylo, however, appear to be the most difficult patterns to synthesize, as no solution for problems that canonically involve these patterns was found. table 3. success rates obtained by origami for each pattern in each benchmark. the “best” column shows the highest success rate for that benchmark across all patterns, which is also underlined. we also show in blue the pattern of the canonical solution.full size table considering the 4 canonical accu problems, checksum and word-stats are historically hard, with few methods ever finding a solution. the same can be said for hylo in the wallis-pi and collatz-numbers problems. in vector-average, the canonical solution involved using accu to compute both the sum and the count as a pair in the st slots, and using the alg slots to perform the division as a post-processing step, finally obtaining the average. the solution that got closer to the intended result was the following: origami took a different approach from the canonical solution, by storing the length of the input in the second element of the tuple while having no use for the first element. the st section had no other purpose than to transmit this pre-processing step to the alg section. this solution got a perfect score during training but failed in testing for certain cases. if we were to replace min 0 (last arg\(_\texttt {0}\) ) by 0 and max (x acc) x by x, then this solution would be correct. the hylo solution for sum-of-squares employed coalg to generate a list of all the numbers from 0 to arg \(_{\texttt {0}}\), and then used alg to square each number and accumulate the sum. even though this was the simplest hylo solution, as hylo has 5 different slots, it has an increased search space in relation to other patterns, which seems to be a big challenge for the algorithm. table 4 compares origami’s results to hotgp’s. there was a substantial increase (\(>30\)) in the success rate in 6 problems. in the 17 problems where the absolute difference is \(<30\), we highlight syllables, double-letters and even-squares problems, as those were problems for which hotgp was not able to synthesize a solution, whereas origami was successful at least once. the two problems with a more noticeable decrease are replace-space-w-nl. and vector-average. these can be explained by the change in grammar between the two algorithms, as hotgp’s solutions were arguably simpler due to having map and filter for replace-space-w-nl. and sum for vector-average. in a practical scenario, the inclusion of these functions would likely lead to a correct solution but, as previously noted, removing them was a conscious decision to enable the proper assessment of the impact of recursion schemes in ps. it would also allow for composite solutions, such as using ana with a map inside instead of relying on hylo to find the entire pattern, which might be easier to synthesize. table 4. origami’s success rates compared to hotgp’s on solved problems. the \(\vardelta \) column shows the relative success rate of origami with respect to hotgp.full size table table 5. success rate with the best values underlined. the last row displays the ratio of victories of each algorithm against origami by the amount of tested problems.full size table to assess how origami fares with relation to the best methods found in the literature, we compare its results to those obtained by pushgp [9], grammar-guided genetic programming (g3p) [3], the extended grammar version of g3p (here called g3p+) [4], code building genetic programming (cbgp) [14], g3p with haskell and python grammars (g3phs and g3ppy) [5], as well as with github copilot [16]. in some of those works, only a subset of the problems was chosen, often avoiding the most difficult ones (i.e., not previously solved by any other method), or problems not solvable by the proposed method itself. the results are reported in table 5 (“–” indicates the authors did not test their method on that specific problem), and summarized in table 6. table 6. number of problems with a success rate \(\ge \) a certain % for each method.full size table among the other gp algorithms, the best performer (dsls) achieves a higher success rate than origami in only 10 problems. considering that they both get a 0 in pig-latin, origami outperforms dsls in 14 problems. notably, origami frequently outperforms cbgp and the g3p variants. it also has the highest number of problems solved with \(100\%\), \(\ge 75\%\), and \(\ge 50\%\), and is second-place in \(\ge 25\%\). when we consider problems to which origami found at least one solution, we note that it outperforms hotgp, cbgp, and all the g3p variations, placing origami at the fourth place. it is also worth noting that origami outperforms hotgp in both the number of best results and amount of problems above all thresholds, which demonstrates it is a substantial improvement over hotgp. we also compare with the results obtained using copilot on psb1, as reported by [16]. in that paper, the authors tested copilot with a different formulation of the program synthesis problem: instead of receiving example of input-outputs, they made the problem description and the function signature available to copilot. this input format can be more difficult to process, as it requires the extraction of useful information from a textual description, but contains additional information that may be implicit in the input-output format. out of the 29 problems, copilot had better results in 14 of them, and origami is equivalent or outperforms copilot in 15 problems. while copilot solves more problems (at least once) than origami, it struggles with consistency, and does not achieve \(100\%\) success rate on any of the tested problems. 5 conclusion this work is the first full implementation of origami, a gp algorithm proposed in [2], and builds on its previous work, hotgp [1]. origami’s main differential is the use of recursion schemes, well-known constructs in functional programming that enable recursive algorithms to be defined in a unified manner. the main motivation for using these in the ps context is enabling recursive programs to be synthesized in a controlled manner, without sacrificing expressiveness. we evaluate our approach in the 29 problems in the psb1 dataset, which is known to be solvable by just a handful of recursion schemes. in general, origami performs better than other similar methods, synthesizing the correct solution more often than other methods in most problems. it was also able to obtain the highest count of problems with success rate \(=100\%\), \(\ge 75\%\) and \(\ge 50\%\) among the gp methods. furthermore, origami achieved comparable results to github copilot, solving some problems that the llm achieved \(0\%\) score. we should stress that the problem formulation is different for both approaches, indicating that combining llm with gp and recursion schemes could be beneficial to improve the results. these experimental results suggest that using recursion schemes to guide the search is a promising research avenue. currently, the main challenge of origami appears to be dealing with harder recursion schemes, such as accumulation and hylomorphism. different evolutionary mechanisms, such as other selection methods and mutation/crossover operators, should be evaluated in this context to understand if they can positively impact the search process. notes 1.the authors represent these recursive functions using a notation called bananas, lenses, envelopes, and barbed wires [11]. 2.the notation i\(_\texttt {0}\) \(\equiv \) [e] is a restriction such that i\(_{\texttt {0}}\) can be decomposed into the type [e], which is the type of a list with elements of some type e. 3.notice that this pattern can only be applied to problems with exactly two arguments. references fernandes, m.c., de frança, f.o., francesquini, e.: hotgp higher-order typed genetic programming. in: proceedings of the genetic and evolutionary computation conference, gecco 2023, pp. 1091-1099. association for computing machinery, new york (2023). https://doi.org/10.1145/3583131.3590464 fernandes, m.c., de franca, f.o., francesquini, e.: origami: (un)folding the abstraction of recursion schemes for program synthesis, pp. 263–281. springer nature singapore, singapore (2024).https://doi.org/10.1007/978-981-99-8413-8_14 forstenlechner, s., fagan, d., nicolau, m., o’neill, m.: a grammar design pattern for arbitrary program synthesis problems in genetic programming. in: mcdermott, j., castelli, m., sekanina, l., haasdijk, e., garcía-sánchez, p. (eds.) eurogp 2017. lncs, vol. 10196, pp. 262–277. springer, cham (2017). https://doi.org/10.1007/978-3-319-55696-3_17 chapter  math  google scholar  forstenlechner, s., fagan, d., nicolau, m., o’neill, m.: extending program synthesis grammars for grammar-guided genetic programming. in: auger, a., fonseca, c.m., lourenço, n., machado, p., paquete, l., whitley, d. (eds.) ppsn 2018. lncs, vol. 11101, pp. 197–208. springer, cham (2018). https://doi.org/10.1007/978-3-319-99253-2_16 chapter  google scholar  garrow, f., lones, m.a., stewart, r.: why functional program synthesis matters (in the realm of genetic programming). in: proceedings of the genetic and evolutionary computation conference companion, gecco 2022, pp. 1844–1853. association for computing machinery, new york (2022). https://doi.org/10.1145/3520304.3534045 gulwani, s.: dimensions in program synthesis. in: proceedings of the 12th international acm sigplan symposium on principles and practice of declarative programming, ppdp 2010, pp. 13–24. association for computing machinery, new york (2010). https://doi.org/10.1145/1836089.1836091, https://doi.org/10.1145/1836089.1836091 helmuth, t., mcphee, n.f., pantridge, e., spector, l.: improving generalization of evolved programs through automatic simplification. in: proceedings of the genetic and evolutionary computation conference, gecco 2017, pp. 937-944. association for computing machinery, new york(2017).https://doi.org/10.1145/3071178.3071330, https://doi.org/10.1145/3071178.3071330 helmuth, t., mcphee, n.f., spector, l.: program synthesis using uniform mutation by addition and deletion. in: proceedings of the genetic and evolutionary computation conference, pp. 1127–1134 (2018) google scholar  helmuth, t., spector, l.: general program synthesis benchmark suite. in: proceedings of the 2015 annual conference on genetic and evolutionary computation, pp. 1039–1046 (2015) google scholar  koza, j.r.: genetic programming: on the programming of computers by means of natural selection. bradford books, cambridge, ma (dec, complex adaptive systems (1992) google scholar  meijer, e., fokkinga, m., paterson, r.: functional programming with bananas, lenses, envelopes and barbed wire. in: hughes, j. (ed.) fpca 1991. lncs, vol. 523, pp. 124–144. springer, heidelberg (1991). https://doi.org/10.1007/3540543961_7 chapter  math  google scholar  montana, d.j.: strongly typed genetic programming. evol. comput. 3(2), 199–230 (1995) article  math  google scholar  o’neill, m., ryan, c.: grammatical evolution. ieee trans. evol. comput. 5(4), 349–358 (2001) article  math  google scholar  pantridge, e., helmuth, t., spector, l.: functional code building genetic programming. in: proceedings of the genetic and evolutionary computation conference, gecco 2022, pp. 1000–1008. association for computing machinery, new york (2022). https://doi.org/10.1145/3512290.3528866 pantridge, e., spector, l.: code building genetic programming. in: proceedings of the 2020 genetic and evolutionary computation conference, pp. 994–1002 (2020) google scholar  sobania, d., petke, j., briesch, m., rothlauf, f.: a comparison of large language models and genetic programming for program synthesis. ieee trans. evolutionary comput. (2024).https://doi.org/10.1109/tevc.2024.3410873 spector, l., robinson, a.: genetic programming and autoconstructive evolution with the push programming language. genet. program evolvable mach. 3, 7–40 (2002) article  math  google scholar  swan, j., krawiec, k., kocsis, z.a.: stochastic synthesis of recursive functions made easy with bananas, lenses, envelopes and barbed wire. genetic programm. evolvable mach. 20(3), 327-350 (2019).https://doi.org/10.1007/s10710-019-09347-3 download references acknowledgments this research was partially funded by fundação de amparo à pesquisa do estado de são paulo (fapesp), grant numbers #2021/12706-1, #2019/26702-8, #2021/06867-2, #2023/00811-0, #2024/01115-0, cnpq #301596/2022-0, cnpq-aws #421787/2022-8 and coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes). author information authors and affiliations federal university of abc (ufabc), são paulo, santo andré, brazil matheus campos fernandes, fabrício olivetti de frança & emilio francesquini authors matheus campos fernandesview author publications search author on:pubmed google scholar fabrício olivetti de françaview author publications search author on:pubmed google scholar emilio francesquiniview author publications search author on:pubmed google scholar corresponding author correspondence to matheus campos fernandes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper fernandes, m.c., frança, f.o.d., francesquini, e. (2025). going bananas! unfolding program synthesis with origami. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_1 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords program synthesis genetic programming recursion publish with us policies and ethics profiles fabrício olivetti de frança view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature predicting energy consumption data using deep learning: an lstm approach | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper predicting energy consumption data using deep learning: an lstm approach conference paper first online: 30 january 2025 pp 296–308 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) predicting energy consumption data using deep learning: an lstm approach download book pdf download book epub julio macedo chaves  orcid: orcid.org/0009-0007-1939-97009,10, elene firmeza ohata  orcid: orcid.org/0000-0001-8386-24559,11, matheus araujo dos santos  orcid: orcid.org/0000-0003-3854-71789,11, josé daniel de alencar santos  orcid: orcid.org/0000-0002-0649-80449,10, matheus jardim bernardes12, daniel seleme dora12 & … pedro pedrosa rebouças filho  orcid: orcid.org/0000-0002-1878-54899,10,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 369 accesses 1 citation abstract nowadays, most of the energy produced globally comes from fossil fuels. however, this type of energy generation harms the environment by emitting various toxic residues in natural bodies. in recent years, an area that has gained strength is producing and consuming clean energy. one that stands out is solar energy production because it is easy to implement and relatively cheap compared to other clean energy production. therefore, several projects were created focusing on generating energy from sunlight, and using it. due to this growing number of enterprises in the area, the creation of applications to help manage production and its use has increased. the use of deep learning techniques to help this industry has also gained strength; predictive models for energy consumption have been widely studied to help enterprises in future planning. in this work, we developed a deep learning model using long short-term memory (lstm) capable of predicting energy consumption from solar power plant data, using the data to train the model and make inferences in the future. we explored configuration combinations such as data filtering with smoothing techniques, model hyperparameters, number of layers, number of neurons, and optimal prediction horizon. the achieved results demonstrate the validity and effectiveness of the implemented methodology. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a comparison of hourly solar energy generation forecasting using rnn and lstm network chapter © 2022 solar pv power forecasting approach based on hybrid deep neural network chapter © 2021 deep learning based framework for forecasting solar panel output power chapter © 2024 1 introduction the growth of the global population has generated an increasing demand for energy [24]. however, most of the energy currently generated comes from fossil fuels [14], which are known for their high emissions of toxic elements into the environment. this results in several problems, such as global warming, pollution of water bodies and the atmosphere, and the decrease of green areas [7, 21]. in response to this situation, there has been a significant increase in the search for clean energy sources in recent years [24]. data collected by a study by the international renewable energy agency in 2020 point to a 7% growth in the use of clean energy sources worldwide. of this increase, 90% is related to the increased use of solar energy [13]. therefore, solar power plays a crucial role in the search for energy sources that do not harm the environment. another point that draws attention to this form of energy generation is that it is relatively cheap when compared to other sources [31], which is why the industry field has turned its eyes to the horizon of investments in this area. another area highlighted in the industry field is machine learning [3]. machine learning techniques have been used in several ways to solve various problems in the industry, such as fault diagnosis [12], quality detection [15, 22], performance optimization [5, 18], process control [20], signal processing [25], outcome prediction [16]. among these applications, one that stands out in solar energy is the forecast of the amount of energy generated in a given period [17, 33]. predictions can be used for financial planning, low-generation detection, electrical crisis management, and energy distribution planning. this information can be used for the general planning of power generation plants, providing a service with excellent customer stability and higher quality [17]. furthermore, another point investigated in the field of solar energy is the prediction of energy consumption [1]. this type of application, as well as generation, brings insights and better planning in the management of a power plant, mainly in managing and avoiding energy crises [26]. therefore, consumption prediction applications are of paramount importance for the field of solar energy producers. based on that, we proposed a methodology using deep learning techniques to predict the amount of energy consumed by a solar plant based on time. we use data transformation techniques, search for optimal hyperparameters, and prediction horizons to create a robust tool with a clear purpose. with this tool, we aim to provide an artifact that brings valuable information to industries related to solar energy consumption. based on the data obtained with a device, enterprises can get insights and manage energy crises in addition to other business planning. 2 related works this section will examine previous studies and works related to this research topic, providing a broader and deeper view of the subject addressed. predicting future energy consumption is an issue frequently explored in the scientific community. this is evident in the study conducted by amasyali et al. [2], providing a comprehensive analysis of the most commonly employed methods in temporal forecasting of energy consumption. their conclusions highlight the strong results obtained by various forecasting approaches. in the literature on the subject, ekononomou et al. [9] stands out for using a multi-layer perceptron (mlp) and an artificial neural network (ann) to predict energy consumption, incorporating factors such as temperature and gross domestic product (gdp). in contrast, hu et al. [11] presented the adaboost-esn model, developed to predict the monthly consumption of industrial electricity. in addition to considering previous consumption, this model included factors such as sales volume and import value, achieving promising results. a notable approach was proposed by du et al. [8]. they introduced a deep learning framework based on an encoder-decoder framework with bi-lstm to predict multivariate energy consumption time series. the experiments, controlled on five different datasets, revealed that the model achieved robust performance in shortand long-term predictions. furthermore, it is essential to highlight studies such as mahjoub et al. [19] and olu et al. [23], which go beyond conventional approaches by exploring deep learning for time series forecasting. these studies stand out notably in the results achieved, showing the substantial potential of these techniques to address this problem. within this scenario, the long short-term memory (lstm) architecture has gained prominence, as evidenced by the study by wang et al. [29]. this work concludes that the lstm model has a significant potential to predict future energy consumption. such a model stands out for its ability to accurately capture the data’s periodicity and exhibits a predictive performance superior to traditional models. in the literature, smoothing techniques have been explored to improve the accuracy of time series predictions. a notable example is the study by wibawa et al. [30], in which the application of smoothing in the data significantly enhanced the performance of his convolutional neural network (cnn), surpassing several state-of-the-art models. in a study conducted by xiao et al. [32], the impact of data pre-processing and smoothing techniques on predicting time series regarding energy consumption in buildings was investigated. the results showed that the smoothing method resulted in a 0.91% reduction in the evaluated metric. furthermore, a gradual decline in model performance over time was observed, suggesting the need for updating every seven weeks. 3 methodology in this section, we detail the methodology that supports our study. our proposed approach is visually captured in fig. 1, which encompasses three main blocks: input data, pipeline, and the model evaluation. the following subsections will describe each of the steps in this process. fig. 1. methodology applied in this work. full size image 3.1 dataset the data used in this study was collected from a power plant, which we refer to by its pseudonym: sri. the information was collected for approximately 390 days. these data were registered daily by specialized equipment, reflecting the energy consumption of each station during this period. the data used is in fig. 2. fig. 2. power consumption data collected from the station, covering 390 days. full size image figure 3 illustrates the decomposition of consumption data into its seasonality, trend, and residual components. we observed that the data trend presents variations, not following a continuous linear trajectory. these oscillations suggest the existence of patterns of growth followed by periods of decline in the data, possibly influenced by external factors such as climate change or economic events. furthermore, in fig. 3, it is possible to notice a marked seasonality repeated approximately every 30 days, indicating a monthly seasonality. it is also possible to observe many residuals, showing that we could not solve the problem with linear models. fig. 3. seasonal, trend, and residuals for the studied dataset. full size image 3.2 data preparation we split the dataset into two subsets, one for training and the other for testing. we performed this split temporally; we separated 80% of the dataset for training, equivalent to approximately 330 days, and we separated the remaining 20% for model validation, about 60 days. for prediction purposes, we divided the test set into four groups, which covered the horizons of 14, 30, 45 and 60 days. we performed this division of the test set to evaluate the model’s efficiency in different prediction horizons. the first 14-day horizon is chosen because it encompasses two complete weeks, which helps the model capture the typical patterns of weekdays and weekends. the second horizon, representing a whole month, allows the model to account for monthly variation, such as regular events that occur once per month. the third horizon represents a month and a half, making it possible to observe trends that may not be visible at shorter intervals. finally, the fourth horizon represents two whole months, providing an even broader view and enabling the capture of more seasonal effects or periodic changes. we used the z-score method to standardize the data. this procedure recalibrates an attribute to align with 0, and its values are readjusted by dividing them by their standard deviation. as a result, the mean of the attribute is normalized to 0 and its standard deviation to 1. we apply the eq. 1 to perform this transformation on the data. $$\begin{aligned} x_{i_{std}}=\frac{x_i mean(x)}{std(x)} \end{aligned}$$ (1) in eq. 1, x denotes a value of a specific attribute taken from the training set. here, mean(x) represents the mean of these attribute values, and std(x) indicates their standard deviation. before the preprocessing step, this attribute’s value is expressed as \(x_i\). after undergoing standardization, it transitions to its standardized form, denoted by \(x_{i_{std}}\). it is important to emphasize that, while both training and test data undergo standardization, the parameters—specifically mean(x) and std(x)—used in eq. 1 are derived solely from the training set. 3.3 regression within the scope of this study, we evaluated different state-of-the-art models to find the architecture that best fits the problem. the models were autoregressive integrated moving average (arima) [4], prophet [27], neuralprophet [28], extreme gradient boosting (xgboost) [6], and lstm [10]. this model selection represents a comprehensive approach encompassing traditional methods and recent innovations. it aims to achieve a holistic understanding of these models’ performance regarding the complexities of the analyzed time series. we trained all models initially using default hyperparameters and tested their performance over a thirty-day prediction horizon. then, we submit the model that obtained the best results for optimization by the grid search method. 3.4 metrics this subsection will present the metrics used to evaluate the trained models. mean absolute error (mae). the mae quantifies the mean error in terms of the magnitude of discrepancies. the corresponding mathematical formulation is given by the eq. 2. $$\begin{aligned} mae = \frac{1}{n} \sum _{i=1}^{n} |y_i \hat{y_i}| \end{aligned}$$ (2) in this context, n denotes the total number of observations, \(y_i\) refers to the observed value, and \(\hat{y}_i\) refers to the value predicted by the model. the mae, often used in regression problems, stands out for its direct interpretation: high values indicate a higher average of absolute errors, indicating a lower model performance. furthermore, the mae uses the same unit as the analyzed variable. in this study, for example, an mae of 0.1 implies that, on average, the model estimates differ by 0.1 kwh from the actual values. root mean square error (rmse). the rmse is calculated from the square root of the mean squared error, as defined by the eq. 3. $$\begin{aligned} rmse = \sqrt{\frac{1}{n} \sum _{i=1}^{n} (y_i \hat{y_i})^2} \end{aligned}$$ (3) unlike the mae, the rmse incorporates the squared error in its formula, amplifying the influence of more significant errors. this characteristic makes the rmse especially valuable in situations where it is crucial to penalize more expressive deviations. thanks to the inclusion of the square root, rmse maintains the same unit as the original data. like mae, an rmse of 0 represents a perfect fit where all model predictions are exact. higher rmse values indicate a declining performance of the model. 3.5 system configuration the device used for the tests has the following hardware configurations: ryzen 5 5600x processor, 12 gb rtx 3060 graphics card, and 16 gb of ram. table 1 details the versions of libraries and software adopted in this study. table 1. specifications of the softwares and libraries used in this study.full size table 4 results and discussion in this section, we address the results achieved by the proposed methodology to forecast energy consumption; our proposal seeks to find the combination between the best method, use the appropriate pre-processing technique on the data, and adjust the hyperparameters. 4.1 modeling of the regression method in table 2, we present the results of energy consumption prediction using different models; we used the default settings for all models. the model that demonstrated the best performance was the ltsm due to its lower values of mae and rmse, indicating its effectiveness in the prediction task. the arima model showed the worst results, with an mae of 84.40 kwh and rmse of 96.67 kwh. the other models showed similar metrics but were significantly lower than the lstm. table 2. results of energy consumption prediction using different models.full size table 4.2 smoothing and horizon we set out for specific tests and adjustments for the ltsm model. we analysed two hypotheses: applying the smoothing technique to remove noise, and searching for an optimal prediction horizon threshold. for the smoothing step, we used a convolution-based smoothing, using a window of 30 days. the forecast horizon is analyzed to observe the limitation of how many steps (days) the model can predict without knowing more data, thus simulating a real environment. the results of applying the method in combination with different prediction horizons are shown in table 3. table 3. comparison of mae and rmse metrics for different prediction horizons, with and without data smoothing.full size table table 3 shows that the data without smoothing showed a significant increase in the mae in the 30-day and 45-day horizons. however, the mae remained relatively stable when the prediction horizon was extended to 45 and 60 days. based on these results, we used the 30-day threshold as the optimal prediction horizon. beyond this point, accumulated noise and errors have a more significant impact on the non-smoothing data. furthermore, in table 3, it is noted that data with smoothing were less affected by the increase in the prediction horizon, allowing the choice of longer prediction horizons. we, therefore, chose to use a 45-day forecast horizon with smoothed data, as they demonstrate the lowest mae among all configurations. 4.3 hyperparameters selection later, as the last step, we searched for the best hyperparameters in the network; we used grid search techniques and calculated the model metrics for the 45-day horizon with smoothed data. the hyperparameters used were the number of layers and the number of neurons per hidden layer. the results are displayed in the table 4. the configuration that obtained the best result was using five hidden layers with 100 neurons in each layer. this configuration achieved an mae of 45.52 kwh and rmse of 54.57 kwh, demonstrating greater accuracy than the other tested configurations. table 4. model performance with hyperparameter variations.full size table analyzing the other results, we found that the variation in the number of hidden layers and neurons per layer had different impacts on the model’s performance. configurations with five hidden layers and 25 neurons obtained results close to those of the highlighted layer. meanwhile, the increase to 20 hidden layers resulted in lower performance, likely due to overfitting during the training step. adding more layers to the model increases its learnability, which can lead to better adaptation to the training data. however, in some cases, this greater flexibility can cause the model to memorize the training data instead of learning more general patterns, impairing generalization to new data. based on these results, the configuration with five hidden layers and 100 neurons in each layer stood out as the most suitable for use in a 45-day horizon and smoothed data. this configuration has struck a balance between the network’s learnability and generalization ability to new data, resulting in good forecasting performance for the specific scenario. figure 4 shows the results obtained for the plant. fig. 4. prediction results using the model with the best settings found in development. (color figure online) full size image 5 conclusion during the modeling development, we can explore different combinations of configurations such as data transformation techniques, hyperparameter search, and search for an optimal prediction horizon. we built an artificial intelligence model from this exploration using deep learning techniques that can predict solar plants’ energy consumption. the results found with the training from the database were satisfactory, with the model being able to generalize and make predictions, achieving good metrics. the best model found has achieved an mae of 45.52 kwh and an rmse of 54.57 kwh and uses five hidden layers with 100 neurons. the optimal prediction horizon seen was 45 days, which provides a good generalization for a satisfactory time window. from this, a user can get good results for forecasting energy consumption up to 45 days into the future of the current day. such a window provides a good window for insights and planning decisions for companies and industries. therefore, industries in the field of solar energy can take advantage of this artifact for future planning based on forecasts created by the model. therefore, we hope that the modeling developed in this study can help the field of solar energy with helpful information. in addition, studies with different databases and exploration of other solutions to the problem of predicting solar energy consumption are needed. for future work, we plan to use other databases to test the model, explore other modeling techniques, such as classical regression, and seek to use other deep learning models for better results. references abd el-aziz, r.m.: renewable power source energy consumption by hybrid machine learning model. alex. eng. j. 61(12), 9447–9455 (2022) article  math  google scholar  amasyali, k., el-gohary, n.m.: a review of data-driven building energy consumption prediction studies. renew. sustain. energy rev. 81, 1192–1205 (2018) article  math  google scholar  bertolini, m., mezzogori, d., neroni, m., zammori, f.: machine learning for industrial applications: a comprehensive literature review. expert syst. appl. 175, 114820 (2021) article  math  google scholar  box, g.e., pierce, d.a.: distribution of residual autocorrelations in autoregressive-integrated moving average time series models. j. am. stat. assoc. 65(332), 1509–1526 (1970) article  mathscinet  math  google scholar  chan, s.l., lu, y., wang, y.: data-driven cost estimation for additive manufacturing in cybermanufacturing. j. manuf. syst. 46, 115–126 (2018). https://doi.org/10.1016/j.jmsy.2017.12.001, https://www.sciencedirect.com/science/article/pii/s0278612517301577 chen, t., guestrin, c.: xgboost: a scalable tree boosting system. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 785–794 (2016) google scholar  dincer, i.: environmental impacts of energy. energy policy 27(14), 845–854 (1999) article  math  google scholar  du, s., li, t., yang, y., horng, s.j.: multivariate time series forecasting via attention-based encoder-decoder framework. neurocomputing 388, 269–279 (2020) article  math  google scholar  ekonomou, l.: greek long-term energy consumption prediction using artificial neural networks. energy 35(2), 512–517 (2010) article  math  google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  math  google scholar  hu, h., wang, l., peng, l., zeng, y.r.: effective energy consumption forecasting using enhanced bagged echo state network. energy 193, 116778 (2020) article  math  google scholar  iqbal, r., maniak, t., doctor, f., karyotis, c.: fault detection and isolation in industrial processes using deep learning approaches. ieee trans. industr. inf. 15(5), 3077–3084 (2019) article  google scholar  irena, a.: renewable capacity highlights. proc. int. renew. energy agency (irena), 1–8 (2020) google scholar  jefferson, m.: world energy outlook to 2100. in: world petroleum congress, pp. wpc–26015. wpc (1994) google scholar  jennings, c., wu, d., terpenny, j.: forecasting obsolescence risk and product life cycle with machine learning. ieee trans. compon. packag. manuf. technol. 6(9), 1428–1439 (2016). https://doi.org/10.1109/tcpmt.2016.2589206 article  math  google scholar  ji, s., wang, x., zhao, w., guo, d.: an application of a three-stage xgboost-based model to sales forecasting of a cross-border e-commerce enterprise. math. probl. eng. 2019 (2019) google scholar  lima, m.a.f., carvalho, p.c., fernández-ramírez, l.m., braga, a.p.: improving solar forecasting using deep learning and portfolio theory integration. energy 195, 117016 (2020) article  google scholar  maggipinto, m., terzi, m., masiero, c., beghi, a., susto, g.a.: a computer vision-inspired deep learning architecture for virtual metrology modeling with 2-dimensional data. ieee trans. semicond. manuf. 31(3), 376–384 (2018). https://doi.org/10.1109/tsm.2018.2849206 article  math  google scholar  mahjoub, s., chrifi-alaoui, l., marhic, b., delahoche, l.: predicting energy consumption using lstm, multi-layer gru and drop-gru neural networks. sensors 22(11), 4062 (2022) article  math  google scholar  mezzogori, d., zammori, f.: an entity embeddings deep learning approach for demand forecast of highly differentiated products. procedia manuf. 39, 1793–1800 (2019). https://doi.org/10.1016/j.promfg.2020.01.260, https://www.sciencedirect.com/science/article/pii/s2351978920303243, 25th international conference on production research manufacturing innovation: cyber physical manufacturing, 9–14 august 2019|chicago, illinois (usa) mirzaliev, s., sharipov, k.: a review of energy efficient fluid power systems: fluid power impact on energy, emissions and economics, vol. 30 (2020) google scholar  oh, y., ransikarbum, k., busogi, m., kwon, d., kim, n.: adaptive svm-based real-time quality assessment for primer-sealer dispensing process of sunroof assembly line. reliab. eng. syst. saf. 184, 202–212 (2019). https://doi.org/10.1016/j.ress.2018.03.020, https://www.sciencedirect.com/science/article/pii/s0951832017303861, impact of prognostics and health management in systems reliability and maintenance planning olu-ajayi, r., alaka, h., sulaimon, i., sunmola, f., ajayi, s.: building energy consumption prediction for residential buildings using deep learning and other machine learning techniques. j. build. eng. 45, 103406 (2022) article  google scholar  rabaia, m.k.h., et al.: environmental impacts of solar energy systems: a review. sci. total environ. 754, 141989 (2021) article  google scholar  scime, l., beuth, j.: using machine learning to identify in-situ melt pool signatures indicative of flaw formation in a laser powder bed fusion additive manufacturing process. addit. manuf. 25, 151–165 (2019). https://doi.org/10.1016/j.addma.2018.11.010, https://www.sciencedirect.com/science/article/pii/s2214860418306869 shamshirband, s., rabczuk, t., chau, k.w.: a survey of deep learning techniques: application in wind and solar energy resources. ieee access 7, 164650–164666 (2019) article  google scholar  taylor, s.j., letham, b.: forecasting at scale. am. stat. 72(1), 37–45 (2018) article  mathscinet  math  google scholar  triebe, o., hewamalage, h., pilyugina, p., laptev, n., bergmeir, c., rajagopal, r.: neuralprophet: explainable forecasting at scale. arxiv 2021. arxiv preprint arxiv:2111.15397 wang, j.q., du, y., wang, j.: lstm based long-term energy consumption prediction with periodicity. energy 197, 117197 (2020) article  math  google scholar  wibawa, a.p., utama, a.b.p., elmunsyah, h., pujianto, u., dwiyanto, f.a., hernandez, l.: time-series analysis with smoothed convolutional neural network. j. big data 9(1), 44 (2022) article  google scholar  wilberforce, t., baroutaji, a., el hassan, z., thompson, j., soudan, b., olabi, a.g.: prospects and challenges of concentrated solar photovoltaics and enhanced geothermal energy technologies. sci. total environ. 659, 851–861 (2019) article  google scholar  xiao, z.: impacts of data preprocessing and selection on energy consumption prediction model of hvac systems based on deep learning. energy build. 258, 111832 (2022) article  math  google scholar  yagli, g.m., yang, d., srinivasan, d.: automatic hourly solar forecasting using machine learning models. renew. sustain. energy rev. 105, 487–498 (2019) article  google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. also pedro pedrosa rebouças filho acknowledges the sponsorship from the brazilian national council for research and development (cnpq) via grant 301455/2022-8. author information authors and affiliations laboratory for processing images, signals and computer science (lapisco), fortaleza, ceará, brazil julio macedo chaves, elene firmeza ohata, matheus araujo dos santos, josé daniel de alencar santos & pedro pedrosa rebouças filho federal institute of education, science and technology of ceará (ifce), fortaleza, ceará, brazil julio macedo chaves, josé daniel de alencar santos & pedro pedrosa rebouças filho federal university of ceará (ufc), fortaleza, ceará, brazil elene firmeza ohata, matheus araujo dos santos & pedro pedrosa rebouças filho pontifical catholic university of rio grande do sul (pucrs), porto alegre, brazil matheus jardim bernardes & daniel seleme dora authors julio macedo chavesview author publications search author on:pubmed google scholar elene firmeza ohataview author publications search author on:pubmed google scholar matheus araujo dos santosview author publications search author on:pubmed google scholar josé daniel de alencar santosview author publications search author on:pubmed google scholar matheus jardim bernardesview author publications search author on:pubmed google scholar daniel seleme doraview author publications search author on:pubmed google scholar pedro pedrosa rebouças filhoview author publications search author on:pubmed google scholar corresponding author correspondence to julio macedo chaves . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper chaves, j.m. et al. (2025). predicting energy consumption data using deep learning: an lstm approach. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_21 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords solar energy deep learning lstm publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating large language models for tax law reasoning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating large language models for tax law reasoning conference paper first online: 30 january 2025 pp 460–474 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) evaluating large language models for tax law reasoning download book pdf download book epub joão paulo cavalcante presa  orcid: orcid.org/0009-0004-4160-64959, celso gonçalves camilo junior  orcid: orcid.org/0000-0003-2553-87909 & sávio salvarino teles de oliveira  orcid: orcid.org/0009-0002-1203-52469  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 545 accesses abstract the ability to reason over laws is essential for legal professionals, facilitating interpreting and applying legal principles to complex real-world situations. tax laws are crucial for funding government functions and shaping economic behavior, yet their interpretation poses challenges due to their complexity, constant evolution, and susceptibility to differing viewpoints. large language models (llms) show considerable potential in supporting this reasoning process by processing extensive legal texts and generating relevant information. this study evaluates the performance of llms in legal reasoning within the domain of tax law for legal entities, utilizing a dataset of real-world questions and expert answers in brazilian portuguese. we employed quantitative metrics (bleu, rouge) and qualitative assessment using a solid llm to ensure factual accuracy and relevance. a novel dataset was curated, comprising genuine questions from legal entities in tax law, answered by legal experts with corresponding legal texts. the evaluation includes both open-source and proprietary llms, providing a assessment of their effectiveness in legal reasoning tasks. the strong correlation between robust llm evaluator metric and bert score f1 suggests these metrics effectively capture semantic aspects pertinent to human-perceived quality. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others evaluating open and proprietary large language models in law interpretation: the case of the eu vat directive chapter © 2026 leveraging llms for legal terms extraction with limited annotated data article open access 14 march 2025 transformational question-answering system based on ukrainian legal documents for online lean education utilizing large language models chapter © 2026 1 introduction the ability to reason over laws is essential for legal professionals, enabling them to interpret and apply legal principles to complex real-world situations. legal questions often lack straightforward answers, requiring thorough analysis, comprehensive research, and synthesis of multiple sources to develop well-founded arguments or solutions. tax law, in particular, is crucial because it influences how governments fund public services and impacts economic activity by shaping investment decisions and individual spending. however, interpreting tax law presents significant challenges for natural language processing (nlp) due to the inherent complexity and ambiguity of legal language, constant updates, amendments, and the need to contextualize regulations within specific jurisdictions. large language models (llms) show significant potential in enhancing the legal reasoning process [23]. these models can process extensive legal texts, including statutes, case law, and legal opinions, to extract relevant information and address the complexities of tax law. by leveraging advanced generation techniques, llms can answer legal questions using specific legal datasets, such as court cases and legal precedents, thus providing comprehensive and relevant information to legal professionals [29]. however, there is a gap in understanding how llms reason over legal texts, as existing question-answering tasks typically contain answers directly extractable from the provided texts, whereas legal reasoning often requires deeper comprehension and application of legal principles to nuanced scenarios [2]. to address this gap, we developed a novel dataset comprising real questions posed by legal entities in the domain of tax law, answered by legal experts with supporting legal texts (gold passages). this dataset allows us to assess the legal reasoning abilities of llms, focusing on their capacity to understand complex legal questions, use relevant law articles, and generate accurate and coherent responses. the evaluation compares llm-generated answers to expert responses using metrics such as rouge [19], bleu [27], and semantic similarity [40], alongside assessments by a strong llm [41], contributing to a understanding of llms’ legal reasoning capabilities. this research evaluates both open-source and proprietary llms in scenarios requiring comprehensive understanding and application of the law, distinct from the extractive approaches used in datasets like squad [28] and triviaqa [14]. our dataset requires llms to comprehend and apply the law to generate appropriate answers, often involving complex vocabulary and contexts not directly mirrored in the texts [2]. this paper presents two significant contributions to the field of legal nlp, particularly within the challenging domain of tax law. first, it introduces a novel dataset consisting of real-world tax law questions, expert-crafted answers, and supporting legal texts, moving beyond extractive question-answering tasks and requiring models to demonstrate legal reasoning abilities. using this dataset, we evaluate how well llms understand complex tax law questions and generate accurate, well-supported answers, providing a better understanding of current llm capabilities and limitations in handling legal reasoning tasks. 2 related works while there are numerous works utilizing large language models (llms) in the legal domain [5, 22, 23, 25, 39], our interest lies in those that apply llms for question and answer (q&a) tasks. these works can be classified into three main categories: those using retrieval-augmented generation, those evaluating llms based on prior knowledge, and those performing fine-tuning and testing the models on q&a tasks in the legal domain. below, we discuss the key works found in each of these categories. 2.1 tax law applications of llms evaluating q&a llms with retrieval-augmented generation. the lleqa [21] dataset includes 1,868 legal questions annotated by experts, containing answers and legal references. this work applies the retrieval-augmented generation (rag) technique, retrieving statutory articles from an extensive corpus of belgian legislation. the model’s effectiveness is evaluated using the meteor metric, demonstrating the feasibility of integrating information retrieval with llms to enhance the accuracy of legal responses. chatlaw [5] addresses the creation of a large-scale language model for the legal domain, specifically in the chinese context. this work combines vector database retrieval methods with keyword-based retrieval to increase the accuracy of responses. integrating these techniques enables the model to provide more precise and contextually relevant answers. evaluating q&a legal reasoning of llms. laiw [6] proposes a benchmark for evaluating the capabilities of llms in the chinese legal context. the aim of this work is to test how well models can handle specific legal tasks. the results show that some legal-specific llms perform better than their general counterparts, although there remains a significant gap compared to gpt-4 [26]. lawbench [8] offers a comprehensive assessment of llm capabilities in legal tasks, including q&a. this work extensively tested 51 popular llms, including 20 multilingual, 22 focused on chinese, and 9 specific to law. the conclusion is that while fine-tuning llms on specific legal texts brings some improvements, the models still need to be usable and reliable for complex legal tasks. fine-tuning and evaluating q&a large language models. fedjudge [37] uses federated learning (fl) to overcome data privacy challenges. this framework optimizes federated legal llms, allowing the models to be trained locally on clients, with their parameters aggregated and distributed on a central server. fedjudge is evaluated on q&a tasks using metrics such as rouge, bleu, and bertscore to compare the quality of generated answers. this work demonstrates that the model provides more precise and relevant answers in different legal contexts. disc-lawllm [38] employs large language models trained on supervised datasets in the legal domain and incorporates a retrieval module to access and utilize external legal knowledge. this system assesses objective and subjective perspectives using disc-law-eval, a benchmark that includes legal question answering. additionally, subjective evaluation is carried out using the gpt-3.5 model as a judge. 3 metodology this section outlines the methodology utilized in our study, with a particular emphasis on the model selection process. we also detail the data collection process, the creation of a relevant corpus, and the experimental setups of the selected models, including the specific prompts and parameters used. furthermore, we detail the evaluation approach, discussing both the metrics employed and the strategy for subjective evaluation. 3.1 dataset collection our dataset consists of a series of tax law questions related to legal entities. the questions were selected from a collection that is annually updated by the general coordination of taxation (cosit) [9] of the brazilian federal revenue service. the dataset includes over a thousand question-answer pairs, with most answers being supported by a relevant normative or legal basis. the granularity of the references in the answers is as detailed as possible, citing the specific articles of law or other regulations used to formulate the responses. the questions represent real taxpayer doubts, and experts in the brazilian tax field craft the answers. below, we will discuss how the dataset was created. selection of questions. we extracted a subsample from the comprehensive set of questions and answers provided by cosit. in this selection process, we focused on questions that included responses with legal references rather than the entire regulation. although the majority of responses included legal references, they were often elaborated by experts in a way that extended beyond the scope of the question or included excessive details such as tables and numerous examples. this complexity made them unsuitable for use in contexts like retrieval-augmented generation (rag). we excluded these overly detailed responses to ensure a fair evaluation with the llms. the initial outcomes of this selection process are depicted in the first three columns of table 1. table 1. questions and answers with legal references and gold passagesfull size table collection of regulations (gold passages). after selecting the questions and their corresponding legal references, laws, and articles, we gathered each regulatory document referenced by the experts in their responses to the questions posed by legal entities. although this task was time-intensive, it was essential for assessing the reasoning capabilities of llms in relation to legal texts. upon completion of this stage, the dataset comprised the question, answer, reference to the regulation, and the regulation itself (gold passages). table 1 presents the final dataset. legislation corpus. in this stage, we collected over 30 documents, which included laws, instructions, decrees, and opinions. each document contains up to thousands of articles comprising multiple provisions. these documents represent a fraction of the brazilian tax legislation and include the regulations that underpin the experts’ responses in the dataset. it is important to note that these regulations are constantly being amended, and many provisions have been revoked. all revoked provisions were excluded up to the dataset creation date to ensure a high-quality corpus. additionally, any questions that had their regulatory basis revoked were eliminated during the question selection phase. figure 1 shows the corpus documents. 3.2 experimental setup in this study, we conducted a comprehensive evaluation of large language models (llms) in terms of their ability to reason about laws, specifically focusing on corporate taxation for legal entities. we evaluated the llms using the datasets created in this paper, which were created from real-world questions and answers about the taxation of legal entities, which were provided by subject-matter experts. we selected over 20 llms for evaluation, encompassing both proprietary and open-source models. the chosen models include notable examples such as mistral ai, llama, gemma, qwen, various community fine-tuned versions of these models, and a proprietary model. each model possesses unique characteristics and capabilities, providing a diverse range of perspectives for our assessment. in order to maintain consistency in our evaluations, we standardized the temperature parameter at 0.1 for all chosen models. this low-temperature setting was chosen to reduce randomness in the output, thereby encouraging more deterministic responses. additionally, we did not impose a maximum token limit, allowing the models to generate responses without any constraints on their length. a specific prompt (see prompt question answer in appendix a) was crafted to guide the models in reasoning about the law and generating appropriate responses. the prompt explicitly instructs the models to reason through the legal context provided and formulate an answer. if a model is unable to generate a satisfactory response, it is instructed to state that it does not know the answer. fig. 1. documents from the legislative corpus full size image the legal information needed to answer each question, like an article from a law or a legal document, is included in the prompt. this information is the same one used by the experts to create the reference answers, ensuring a fair basis for comparison. by using standardized prompts and incorporating relevant legal provisions, it ensures that the models have access to the same information as human experts. this enables a thorough evaluation of their reasoning capabilities. it is important to note that the questions and the reference answers are presented in brazilian portuguese. this aspect of the study tests the models’ reasoning abilities and evaluates their proficiency in generating accurate and contextually appropriate responses in the portuguese language. given that many llms are primarily trained on english-language datasets, assessing their performance on brazilian portuguese legal texts for understanding the applicability and limitations of these models in non-english-speaking jurisdictions. although the dataset used in this experiment contains a corpus suitable for retrieval-augmented generation (rag), our evaluation focused solely on the tasks of generation and reasoning. this decision was inspired by other prominent datasets, such as squad 2.0 and hotpotqa, which also provide the expected passages alongside the ground truth answers, allowing for a direct assessment of the model’s generation capabilities without the retrieval step. by concentrating on these aspects, we aimed to isolate and thoroughly evaluate the llms’ ability to generate accurate and reasoned responses based solely on the provided legal context. 3.3 evaluation metrics our study evaluated large language models (llms) using a comprehensive approach integrating quantitative and qualitative methods. for the quantitative assessment, we employed the bleu (bilingual evaluation understudy) and rouge (recall-oriented understudy for gisting evaluation) metrics [19, 27]. in the field of natural language processing, these metrics are crucial for assessing the quality of text generation by comparing the models’ responses to a predefined set of reference answers. specifically, in the domain of questions and answers related to corporate taxation, these metrics provide a quantitative measure of how closely the generated responses align with the ideal answers. despite their widespread use, metrics such as bleu, rouge [37], and meteor [21], which are widely used for evaluating language models, they primarily provide a quantitative perspective and may not fully capture the accuracy of responses in question-answering scenarios [20]. this limitation arises because these metrics do not adequately assess the factual accuracy or the relevance of the generated responses, which are critical in determining whether the questions were answered correctly. in order to address this gap, we adopted a more nuanced qualitative approach, utilizing the capabilities of a powerful language model as a surrogate for human judgment. specifically, we employed gpt-4 to evaluate the performance of other models. this approach is premised on the notion that a robust llm, such as gpt-4, can effectively emulate human judgment in evaluating responses [7, 10, 17, 20, 31, 35, 41] to open-ended questions, thereby providing a closer approximation to human evaluative criteria. for the qualitative evaluation, we used a carefully designed prompt to assess the factual accuracy of the models’ responses. the accuracy of each model was then calculated based on this assessment. the specific prompt used for this evaluation can be found in prompt evaluation in the appendix a for further details. table 2. model performance metricsfull size table 4 results in this section, we will present the results of the experiments described in the experimental setup section. we used the metrics outlined in the metrics section, both of which are situated in the methodology (sect. 3). we selected the main language models operating in portuguese to evaluate how well they can reason and answer questions in the context of tax law, with the aim of identifying potential improvements (table 2). 4.1 model performance analysis the latest versions of the llama, qwen, and mistral families exhibit significant advancements compared to their predecessors. these models incorporate several architectural enhancements, including swiglu activation [30] and grouped query attention (gqa) [4]. both the qwen2-72b-instruct [1] and llama-3-70b-chat-hf [3] models benefited from these improvements, particularly the modifications to the tokenizer and the inclusion of gqa, leading to notable performance gains. as a result, the qwen2-72b-instruct  [1] model achieved the highest accuracy. similar results have been observed in other llm evaluation benchmarks  [1], highlighting the superior performance of models incorporating these techniques. the performance analysis of the models revealed that model size significantly impacts the results, but this impact is not always straightforward. larger models, such as qwen2-72b-instruct [1] and mixtral-8x22b-instruct-v0.1 [13], achieved superior performance, exhibiting the highest rouge-l, bleu, bert score f1, and gpt-4 evaluated accuracy metrics. however, we observed that smaller models, such as mistral-7b-instruct-v0.3 [12] and openhermes-2p5-mistral-7b [32], outperformed some larger models in specific metrics. for instance, mistral-7b-instruct-v0.3 attained a bert score f1 of 0.71, surpassing several larger models, and openhermes-2p5-mistral-7b demonstrated remarkable performance with accuracy comparable to significantly larger models. these findings suggest that while larger models generally deliver better results due to their ability to capture more complex information, well-trained and fine-tuned smaller models can offer competitive performance in specific contexts. this trend indicates that the training quality and the model’s suitability to the particular dataset are crucial factors that can mitigate the size disparity among models. although the volume of portuguese data used in training these models has yet to be verified, the architectural and training improvements suggest the enhanced performance of the llms in q&a tasks in corporate tax law. in the mistral family, the mixtral-8x22b-instruct-v0.1 [24] model stood out with the highest scores in rouge-l, bleu, and bert score f1, indicating the potential of the mixture of experts architecture [13] for legal texts in portuguese. the analysis of fine-tuned open-source models reveals significant improvements over the base models. the openchat-3.5-1210 [34] and openhermes-2p5-mistral-7b [32], both derived from the mistral-7b-v0.1 [12], showed notable increases in accuracy after fine-tuning. similarly, the vicuna-13b-v1.5 and vicuna-7b-v1.5 models [41], fine-tuned from llama 2 [33], also demonstrated advances in response accuracy. furthermore, models such as wizardlm-13b-v1.2 [36], solar-10.7b-instruct-v1.0 [15, 16], and platypus2-70b-instruct [18], derived from llama 2, improved the results of their base models. notably, these fine-tuning processes were conducted on diverse datasets, not on the experimental dataset itself, yet still led to enhanced metrics within the experimental dataset. these improvements suggest that fine-tuning can effectively enhance the capabilities of q&a and legal text generation tasks when applied to specific datasets. 4.2 evaluation metrics analysis the traditional metrics like bleu and rouge may not fully capture the nuances needed for accurate question answering in q&a tasks. the bert score f1 metric is extensively recognized for its alignment with human evaluation due to its capacity to capture deep semantic similarities between texts [40], surpassing the lexical matching capabilities of traditional metrics like rouge-l and bleu [40]. while this study does not aim to prove that llm as a judge for evaluation is aligned with human evaluation, recent studies have been exploring this alignment [20, 31, 35, 41]. our research evaluates the quality of responses generated by llms in legal domain q&a tasks. the strong correlation between the llm (gpt-4) accuracy evaluation and bert score f1, as evidenced by the pearson (0.657) and kendall (0.491) correlations (see table 3), suggests that both metrics capture semantic aspects relevant to human-perceived quality. the results are in line with studies [40] recommending using pearson and kendall correlations to evaluate metric quality. table 3. correlation matrices for evaluation metricsfull size table fig. 2. bland-altman plots for metrics vs llm (gpt-4) accuracy evaluation full size image furthermore, the bland-altman analysis, which is particularly suitable for comparing measurement methods [11], confirms that bert score f1 and llm (gpt-4) accuracy evaluation are more concordant, as shown by the lower variation in point dispersion and the narrower width of the limits of agreement (see fig. 2). in contrast, rouge-l and bleu demonstrated higher mean differences and wider limits of agreement. bert score f1 exhibited a mean difference close to zero and narrower limits of agreement, indicating better concordance with llm accuracy evaluation measurements. these findings imply that llm (gpt-4) accuracy evaluation, similar to bert score f1, could be a valuable and representative metric for assessing the actual performance of language models. while rouge-l and bleu show higher correlations with bert score f1, the stronger correlation and concordance of llm accuracy evaluation with bert score f1 indicate its potential alignment with human evaluation. this supports the development of evaluation metrics that more accurately reflect human-perceived quality, aligning with the direction of current research investigating the potential of llms used to align with human evaluation [20, 31, 35, 41]. 5 conclusion this study underscores the importance of tax law in society and the potential of language models to assist in its understanding and application. we developed a novel dataset of real-world tax law questions and expert answers in brazilian portuguese and conducted a rigorous evaluation of various language models. while our findings suggest that these models show promise in comprehending and reasoning about complex legal texts, further research is necessary to fully demonstrate their effectiveness in legal reasoning across a broader range of scenarios and tasks. our evaluation showed that advancements in model architecture have a noticeable impact on performance, and fine-tuning open-source models, even when done on diverse datasets rather than those specific to the legal domain, can still improve their ability to generate relevant and accurate responses. this suggests that continuous improvements and adaptations are valuable in enhancing the capabilities of language models in legal tasks. for assessing model performance, we used bert score f1, known for its strong correlation with human evaluations in tasks involving descriptive and structural understanding, and a newer metric, llm accuracy evaluation. while bert score f1 is already established as an effective measure aligned with human judgment, especially in descriptive tasks, our results showed that llm accuracy evaluation also demonstrated strong correlation with bert score f1 through pearson and kendall correlations. the bland-altman analysis further confirmed that the llm metric aligns closely with bert score, suggesting its potential as a reliable alternative in evaluations. however, it is important to note that while these findings are encouraging, the use of these metrics for reasoning-based tasks, such as those in this study, still requires further validation. the llm metric is a promising tool, but more research is needed to establish its effectiveness fully, particularly in capturing the nuances of legal reasoning. limitations and future work. a limitation of our study is that while it focused on evaluating the generation and reasoning capabilities of llms, it did not require the models to identify specific legal provisions as part of their responses. our dataset includes a comprehensive corpus containing the necessary laws, enabling the application of retrieval-augmented generation (rag) techniques. this allows models to retrieve relevant legal provisions and incorporate them into their answers, which is essential for a more complete statutory reasoning process. by providing the relevant legal articles, which do not directly mirror the answers to the questions, our study did assess a portion of the statutory reasoning by testing the models’ ability to apply the law to generate accurate responses. future work could leverage this corpus to explore the integration of rag, aiming to enhance the models’ ability to not only generate correct answers but also to identify and cite the appropriate legal provisions, thereby achieving a more robust and comprehensive statutory reasoning. dataset and code availability. the dataset used in this study, as well as the code for reproducing the experiments and analyses, are publicly available. the dataset can be accessed at the following link: https://github.com/joaopaulopresa/dataset. the code can be found here: https://github.com/joaopaulopresa/code. references qwen2 blog (2024). https://qwenlm.github.io/blog/qwen2/. accessed 08 june 2024 abdallah, a., piryani, b., jatowt, a.: exploring the state of the art in legal qa systems. j. big data 10(1), 127 (2023) article  math  google scholar  ai@meta: llama 3 model card (2024). https://github.com/meta-llama/llama3/tree/main ainslie, j., lee-thorp, j., de jong, m., zemlyanskiy, y., lebron, f., sanghai, s.: gqa: training generalized multi-query transformer models from multi-head checkpoints. in: proceedings of the 2023 conference on empirical methods in natural language processing, pp. 4895–4901 (2023) google scholar  cui, j., li, z., yan, y., chen, b., yuan, l.: chatlaw: open-source legal large language model with integrated external knowledge bases. arxiv preprint arxiv:2306.16092 (2023) dai, y., et al.: laiw: a chinese legal large language models benchmark (a technical report). arxiv preprint arxiv:2310.05620 (2023) du, y., wei, f., zhang, h.: anytool: self-reflective, hierarchical agents for large-scale api calls. arxiv preprint arxiv:2402.04253 (2024) fei, z., et al.: lawbench: benchmarking legal knowledge of large language models. arxiv preprint arxiv:2309.16289 (2023) general coordination of taxation (cosit): questions and answers for legal entities (2023). https://www.gov.br/receitafederal/pt-br/assuntos/orientacao-tributaria/declaracoes-e-demonstrativos/ecf/perguntas-e-respostas-pj-2023.pdf. accessed 11 nov 2023 hackl, v., müller, a.e., granitzer, m., sailer, m.: is gpt-4 a reliable rater? evaluating consistency in gpt-4’s text ratings. in: frontiers in education. vol. 8, p. 1272229. frontiers media sa (2023) google scholar  haghayegh, s., kang, h.a., khoshnevis, s., smolensky, m.h., diller, k.r.: a comprehensive guideline for bland-altman and intra class correlation calculations to properly compare two methods of measurement and interpret findings. physiol. meas. 41(5), 055012 (2020) article  google scholar  jiang, a.q., et al.: mistral 7b (2023). arxiv preprint arxiv:2310.06825 jiang, a.q., et al.: mixtral of experts (2024). arxiv preprint arxiv:2401.04088 joshi, m., choi, e., weld, d.s., zettlemoyer, l.: triviaqa: a large scale distantly supervised challenge dataset for reading comprehension. in: proceedings of the 55th annual meeting of the association for computational linguistics, vol. 1: long papers, pp. 1601–1611 (2017) google scholar  kim, d., et al.: sdpo: don’t use your data all at once (2024) google scholar  kim, d., et al.: solar 10.7b: scaling large language models with simple yet effective depth up-scaling (2023) google scholar  koutcheme, c., dainese, n., sarsa, s., hellas, a., leinonen, j., denny, p.: open source language models can provide feedback: evaluating llms’ ability to help students using gpt-4-as-a-judge. arxiv preprint arxiv:2405.05253 (2024) lee, a.n., hunter, c.j., ruiz, n.: platypus: quick, cheap, and powerful refinement of llms (2023) google scholar  lin, c.y.: rouge: a package for automatic evaluation of summaries. in: text summarization branches out, pp. 74–81 (2004) google scholar  liu, y., iter, d., xu, y., wang, s., xu, r., zhu, c.: g-eval: nlg evaluation using gpt-4 with better human alignment. in: the 2023 conference on empirical methods in natural language processing (2023) google scholar  louis, a., van dijck, g., spanakis, g.: interpretable long-form legal question answering with retrieval-augmented large language models. arxiv preprint arxiv:2309.17050 (2023) ma, s., chen, c., chu, q., mao, j.: leveraging large language models for relevance judgments in legal case retrieval. arxiv preprint arxiv:2403.18405 (2024) martin, l., whitehouse, n., yiu, s., catterson, l., perera, r.: better call gpt, comparing large language models against lawyers. arxiv preprint arxiv:2401.16212 (2024) mistral.ai: introducing the mixtral-8x22b-instruct-v0.1 model (2024). https://mistral.ai/news/mixtral-8x22b/. accessed 15 may 2024 niklaus, j., et al.: flawn-t5: an empirical examination of effective instruction-tuning data mixtures for legal reasoning. arxiv preprint arxiv:2404.02127 (2024) openai: gpt-4 technical report (2023). https://cdn.openai.com/papers/gpt-4.pdf. accessed 10 jan 2024 papineni, k., roukos, s., ward, t., zhu, w.j.: bleu: a method for automatic evaluation of machine translation. in: proceedings of the 40th annual meeting of the association for computational linguistics, pp. 311–318 (2002) google scholar  rajpurkar, p., zhang, j., lopyrev, k., liang, p.: squad: 100,000+ questions for machine comprehension of text. in: proceedings of the 2016 conference on empirical methods in natural language processing, pp. 2383–2392 (2016) google scholar  lai, j., gan, w., wu, j., qi, z., philip, s.y.: large language models in law: a survey (2023). https://arxiv.org/abs/2312.03718 shazeer, n.: glu variants improve transformer. arxiv preprint arxiv:2002.05202 (2020) sottana, a., liang, b., zou, k., yuan, z.: evaluation metrics in the era of gpt-4: reliably evaluating large language models on sequence to sequence tasks. in: the 2023 conference on empirical methods in natural language processing (2023) google scholar  team, i.: openhermes-2-5-mistral-7b (2024). https://github.com/inferless/openhermes-2-5-mistral-7b. accessed 15 may 2024 touvron, h., a.: llama 2: open foundation and fine-tuned chat models (2023) google scholar  wang, g., cheng, s., zhan, x., li, x., song, s., liu, y.: openchat: advancing open-source language models with mixed-quality data. in: the twelfth international conference on learning representations (2023) google scholar  wei, f., chen, x., luo, l.: rethinking generative large language model evaluation for semantic comprehension. arxiv e-prints pp. arxiv–2403 (2024) google scholar  xu, c., et al.: wizardlm: empowering large language models to follow complex instructions. arxiv preprint arxiv:2304.12244 (2023) yue, l., et al.: fedjudge: federated legal large language model. arxiv preprint arxiv:2309.08173 (2023) yue, s., et al.: disc-lawllm: fine-tuning large language models for intelligent legal services. arxiv preprint arxiv:2309.11325 (2023) zhang, r., et al.: evaluation ethics of llms in legal domain. arxiv preprint arxiv:2403.11152 (2024) zhang, t., kishore, v., wu, f., weinberger, k.q., artzi, y.: bertscore: evaluating text generation with bert (2020) google scholar  zheng, l., et al.: judging llm-as-a-judge with mt-bench and chatbot arena. in: thirty-seventh conference on neural information processing systems datasets and benchmarks track (2023) google scholar  download references author information authors and affiliations federal university of goias (ufg), goiânia, brazil joão paulo cavalcante presa, celso gonçalves camilo junior & sávio salvarino teles de oliveira authors joão paulo cavalcante presaview author publications search author on:pubmed google scholar celso gonçalves camilo juniorview author publications search author on:pubmed google scholar sávio salvarino teles de oliveiraview author publications search author on:pubmed google scholar corresponding author correspondence to joão paulo cavalcante presa . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri appendix: list of prompts used appendix: list of prompts used this appendix provides a list of the prompts utilized throughout the experiments detailed in this study. the inclusion of these prompts ensures the transparency and reproducibility of the experimental procedures, allowing other researchers to replicate the study and verify the findings. prompt question answer: prompt evaluation: rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper presa, j.p.c., camilo junior, c.g., oliveira, s.s.t.d. (2025). evaluating large language models for tax law reasoning. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_32 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_32 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords legal reasoning large language models (llms) legal question answering tax law publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature joint event extraction with contextualized word embeddings for the portuguese language | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper joint event extraction with contextualized word embeddings for the portuguese language conference paper first online: 28 november 2021 pp 496–510 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) joint event extraction with contextualized word embeddings for the portuguese language download book pdf download book epub anderson da silva brito sacramento  orcid: orcid.org/0000-0002-2288-689910 & marlo souza  orcid: orcid.org/0000-0002-5373-727110  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1279 accesses 2 citations abstract event extraction (ee) is the task of identifying mentions of particular event types and their arguments in text, and it constitutes an important and challenging task within the area of information extraction (ie). however, in the context of the portuguese language, very little work has been conducted on this topic. in this paper, we propose a neural-based method for ee, as well as a data resource to mitigate this research gap. we also present a data augmentation strategy for ee, employing an open information extraction (oie) system, aiming to overcome the shortage in annotated data for the problem in the portuguese language. our experimental results show that our method is able to predict event types and arguments automatically, and the proposed method of data augmentation, in one of the two evaluated samples, contributes to the performance of the tested models in the subtask of argument role prediction. further, an implementation of our method is available to the community, as the models trained in our experiments (https://github.com/formas/tefe). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others event extraction for portuguese: a qa-driven approach using ace-2005 chapter © 2023 dptoie: a portuguese open information extraction based on dependency analysis article 05 december 2022 a new entity extraction model based on journalistic brazilian portuguese language to enhance named entity recognition chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing machine translation natural language processing (nlp) sequence annotation 1 introduction event extraction (ee) is an important and challenging task of information extraction (ie) in the field of natural language processing (nlp). the task comprises identifying the mentions of particular event types and their arguments, i.e. mentions to entities participating in the event, and the roles of these arguments in relation to an event type. the term event has different, definitions in the literature [6, 23]. while events are undeniably temporal entities, they also may possess a rich non-temporal structure that is important for intelligent information access systems [1]. in this work, we consider events as things that happen or occur involving participants and attributes, most notably spatio-temporal attributes. to be more precise, we adopt the definition presented in the ace 2005 annotation guidelines [6]: an event is a specific occurrence involving participants; it is something that happens and can frequently be described as a change of state. based on the event annotation guideline ace 2005 [6], and related literature on ee [18, 20], ee can be divided into two main subtasks: event detection (ed), i.e. the task of identifying and classifying event triggers, and argument role prediction (arp), i.e. the task of identifying the arguments of an even and labeling their roles. in this context, an event trigger consists of an expression denoting the occurrence of the event and each event mention in a sentence is identified by a trigger term. an event trigger may be expressed primarily through verbs and nominalizations but also by other word classes such as adjectives and prepositions. as such, one of the challenging aspects of the ee task is that the same trigger term can express different events in different contexts. that is, trigger words may be ambiguous. to illustrate, consider the following sentences: \(s_1\): “... os árabes teriam de apoiar o iraque numa luta contra o seu inimigo comum israelita.”footnote 1 \(s_2\): “ambas as empresas são aliadas da navigation mixte na luta contra uma opa hostil ...”footnote 2 both of \(s_1\) and \(s_2\) contain a word, “luta” (fight), functioning as an event trigger. however, in s1, “luta” means a hostile encounter between opposing forces, which denotes a hostile encounter event. in s2, “luta” means a dispute between parties with incompatible opinions, and denotes an event of type quarreling [2]. on the other hand, the same event can be described by various different expressions, by means of different trigger words. for example, we observe in the annotated timebankpt event frame annotation (tefa) corpus that the following trigger words denote an occurrence of an event of type statement: “dito”, “disse”, “referiu”, “anunciado”, “acrescentou”, etc.footnote 3 furthermore, when dealing with multiple event mentions in the same sentence, it is possible that different events share arguments with different roles. consider the following sentence: “a meridian national corp. disse que vendeu 750.00 ações oridinárias ao grupo da família mcalpine, por 1 milhão de dólares, ou 1,35 dólares por ação.”footnote 4 the entity mention “meridian national corp.” takes two different (and not related) argument roles: speaker and seller, respectively, in the events statement and commerce_sell denoted by the event triggers “disse” (said) and “vendeu” (sold). event mentions in natural language text are ubiquitously present in different domains and genres and encompass a great amount of information encoded in a sentence. as noted by ahn [1], solutions for ee can benefit the development of other nlp applications, such as text summarization, question answering and information retrieval. notably, there are two variations of the task of ee regarding the types of target event that are considered [30]: open domain, when no schema and list of target event types are defined for the task; and closed domain, when the set of target event types and their respective schemata are pre-defined. in this paper, we will discuss a method for closed domain ee for the portuguese language. we developed a method to identify and classify a closed set of event types whose arguments and their roles were previously specified based on machine learning. as there is no publicly available dataset and corpus for the task of closed domain ee for the portuguese language that we are aware, we built a dataset by enriching the timebankpt [8] corpus with event annotation schemata from the framenet project [2]. timebankpt is the first corpus of portuguese with rich temporal annotations, i.e. it includes annotations not only of temporal expressions but also about events and temporal relations [8]. in this work, we propose a method to the task of ee developed and evaluated over a newly created data set with event types and argument roles annotations, labeled over the timebankpt corpus, from which we maintain only the event trigger annotations. our method jointly performs ed and arp by learning a shared intermediary representation for the ed and arp subtasks. our main contributions are: (1) we propose a novel method for the ee task, simultaneously tackles the ed and arp subtasks. (2) we demonstrate the usefulness of the application of oie systems as a strategy to data augmentation for ee annotated sentences. (3) we have conducted experiments on a proposed enriched timebankpt corpus, achieving modest performance with low feature engineering effort. 2 related work most work on the literature on ee have focused on languages such as english and chinese, for which there are standard corpora for the task [30], such as the ace 2005 corpus [27]. regarding the approaches adopted in the literature, there are two main categorizations with respect to the input features and the modularization of the methods, respectively divided between discriminative featured-based [1, 15] and representation learning-based methods [10, 17], and the pipeline [15] versus the joint approach [20]. recently, however, methods based on the application of deep neural networks, more specifically, the use of representation learning to address the task of ee [4, 19], have achieved good performance improvement for the task compared to earlier feature-based approaches and, as such, we will focus on a neural-based approach in this work. while ed is a much well-established task, if we consider the arp subtask, very few works on the literature focused on both argument identification and argument role labeling [20]. most of the related work assumes that argument identification is a previously resolved problem and focus only on the task role classification. among the work on argument identification and role labeling, the works of nguyen et al.  [18, 20] propose the use of recurrent neural networks (rnns) for the arp, achieving good performance for the task. these works will be the main inspiration for our method. it is also important to notice that the use of contextualized word embeddings is also a recent development [26]. to our knowledge, very little work has been done on event extraction for the portuguese language. notably, this task has been addressed in a limited scope in the harem [3] named entity recognition evaluation, which included temporal named entities of the type event. from a more general perspective, the work of costa and branco [7] and quaresma [22] are some of the only works on the literature on this problem for the portuguese language, to our knowledge. costa e branco [7] proposed a method for event identification and classification using an approach based on feature engineering and a decision tree trained over the timebankpt corpus. the corpus, however, only contains annotation on a very broad event classification typology, e.g. reporting, occurrence, state, i_state, and i_action, and does not contain annotation on event structure, i.e. its arguments and their roles in the event. quaresma et al. [22] focus on the task of event extraction proposing an algorithm that relies on the output of a semantic role labeling (srl) system. in doing so, the authors assumed that only predicates are event trigger candidates. in their work, the authors did not classify events by type, and they also limited the classification of event arguments, considering only the roles provided by srl schemas. unfortunately, the data set provided by the authorsfootnote 5 contains only the information on trigger output predictions of their systems, without a gold reference that would allow comparison at least for event identification. 3 task definition in this work, we are concerned with the task of event extraction for the portuguese language. throughout this work, we will employ the following terminology: event mention: a phrase or sentence in which an event occurs, including one trigger and an arbitrary number of arguments. event trigger: the main word that most clearly expresses an event occurrence. event argument: an entity mention, temporal expression or value (e.g. money) that servers as a participant or attribute with a specific role in an event mention. argument role: the relationship between an argument to the event in which it participates. the corpus that we employ in our experiments contains 514 event types (e.g., statement, commerce_buy, firing) that correspond to the types of event triggers associated with each event mention. each event type has its own set of roles that can be performed by one of the entities participating in the event, i.e. its arguments. for instance, some of the roles of the event type statement are speaker, message, topic, addresse and time. the total number of roles for all the event types on the annotated corpus is 1936. following this terminology, we explicitly define the standard evaluation procedures as follows, as presented by [30]: trigger identification: a trigger is correctly detected if its offsets (viz., the position of the trigger word in the text) match a reference trigger trigger classification: an event type is correctly classified if both the trigger’s offset and event type match a reference trigger and its event type. argument identification: an argument is correctly identified if its offsets match any of the reference argument mentions (viz., correctly recognizing participants in an event) argument role classification: an argument role is correctly classified if its event type, offsets, and role match any of the reference argument mentions. 4 model we propose a model that jointly performs ed and arp at the sentence level. as such, let \(w = \langle w_1, w_2, ..., w_n \rangle \) be a sentence, where n is the number of words/tokens and \(w_i\) is its i-th token. we will model the ed problem, as a multi-class word classification problem, following prior works [10, 16, 26] in assuming that event triggers are single words/tokens in the sentences. this definition leads to a sequence classification problem on the input sentence w, in which we predict the event type \(t_i\) for each \(w_i \in w\) where \(t_i\) can be “none” to indicate the word \(w_i\) is not a trigger, or it is not any of the target event types. it is assumed that a trigger candidate cannot refer to more than one event mentioned in the same sentence. for event argument identification and role prediction, we need to recognize the entity mentions that act as an argument for each of the event mentions occurring in w (argument identification) and classify their associated role. so, for the sake of explanation, considering that each sentence has only one event mention, this task is cast as a sequence labeling task in which for each word/token in the sentence, we predict a sequence of labels \(e = e_1, e_2, ..., e_n\). in summary, for each trigger candidate word \(w_i\) we must predict the corresponding argument role labels for each word/token in w. an illustration of the architecture of our model with the joint prediction of event mentions and their arguments is depicted in fig. 1. fig. 1. the joint ee model for the two subtasks with an input sentence. “subiram” (risen) is the current trigger word candidate, and all the other word tokens are transformed into vector representations to predict their roles given the current trigger candidate. full size image 4.1 trigger and sentence encoding in the component of trigger encoding, every trigger candidate word \(w_i \in w\) is transformed into a word vector \(c_i\) using the pre-trained word embedding \(d_i\) of \(w_i\). we employ the bertimbau model [25], a transformer model trained over portuguese language data, for obtaining these representations. for the sentence encoding step, we only use the respective word embedding for each word/token of the sentence. since bertimbau applies wordpiece [28] tokenizations, possibly generating subword units, we choose to compose the respective word embeddings by summing the vector representation of each word’s subword units. 4.2 sentence representation after the sentence encoding step, the input sentence w is transformer in a sequence of word embedding vectors \(x = \langle x_1, x_2, ..., x_n\rangle \). to produce the sentence representation, we feed the sequence x into a bidirecional recurrent neural network [24], which produces the hidden representation vector for each input vector \(x_t\) from x. we obtain the sequence \(h = \langle [\overrightarrow{h_1};\overleftarrow{h_1}], [\overrightarrow{h_2};\overleftarrow{h_2}], \cdots , [\overrightarrow{h_n};\overleftarrow{h_n}]\rangle \) of two hidden states at each time-step t, one for the left-to-right propagation \(\overrightarrow{h_t}\) and another for the right-to-left propagation \(\overleftarrow{h_t}\). at each step t, we employ a lstm network [13] that accepts a current input \(x_t\) and a previous hidden state \(h_{t-1}\) to compute the current hidden state \(h_t = lstm(x_t, h_{t-1})\). 4.3 trigger representation for the trigger representation, we feed the word embedding vector of the current trigger candidate \(c_i\) into a feed-forward layer \(ff(c_i)\). the feature representation from this layer is shared between the two next prediction states to perform ed and arp simultaneously. the value of this layer activations is computed as: \({r_i^{ed}} = relu(w_{tr}c_i + b_{tr})\) where \(w_{tr}\) is a weight matrix and \(b_{tr}\) a bias term. 4.4 event detection and argument role prediction the predictions of the event trigger classifier and argument role classifier components are done sequentially over the sentence from left to right. at the current word/step i, we attempt to compute the probability $$p(a_i,t_i| \vargamma ) = p(t_i|\vargamma ) p(a_{i,1}|\vargamma ,t_i) p(a_{i,2}|\vargamma ,a_{i,<1},t_i) \cdots p(a_{i,n}|\vargamma ,a_{i,<n},t_i),$$ where \(\vargamma = \langle w,a_{<i},t_{<i}\rangle \), and \(a_{i<j} = a_{i,1},a_{i,2},\cdots ,a_{i,j-1}\). in the argument role prediction classifier, at each time step j, we generate the feature representation vector \(r_{i,j}^{arp}\) for the word/token \(w_j\) by concatenating the hidden layer representation \(h_j\) of the sentence representation stage with the shared vector of activations of the current trigger candidate, the trigger representation vector \(r_i^{ed}\), to obtain: \(r_{i,j}^{arp} = [h_j, r_i^{ed}]\). in order to compute the probability \(p(t_i|w,t_{<i})\), we feed the feature representation \(r_{i}^{ed}\) into a feed-forward neural network with a softmax layer \(ff^{ed}\), resulting in the a probability distribution over the possible event types: \(p(t_i|w,t_{<i})\) \(=\) \(ff^{ed}(r_{i}^{ed})\). the event type of the trigger candidate i is selected by a greedy decoder: \(t_{i}^{p} = argmax p(t_i|w,t_{<i})\). for the argument role distribution \(p(a_{i,j}|w,a_{i,<j},t_{<i+1})\), we follow similar procedure. we feed the feature representation \(r_{i,j}^{arp}\) into a feed-forward neural network with a relu activation function, followed by a softmax layer: \(p(a_{i,j}|w,a_{i,<j},t_{<i+1})\) \(=\) \(ff^{arp}(relu(r_{i,j}^{arp}))\). we apply the greedy strategy to predict the argument role: \(a_{i,j}^p\) \(=\) \(argmax p(a_{i,j}|w,a_{i,<j},t_{<i+1})\). 4.5 training we train the network by minimizing the loss function values. the optimization objective function is defined as multi-class cross-entropy loss: $$\begin{aligned} c(a,t,w) = -\sum _{i=1}^{n}\log p(t_i|w,t_{<i}) \sum _{i=1}^{n}\sum _{j=1}^{n}\log p(a_{i,j}|w,a_{i,<j},t_{<i+1}). \end{aligned}$$ the gradients are computed with backpropagation with the stochastic gradient descent with mini-batches applying rmsprop [12] weights-average. 5 corpus to our knowledge, there is no currently publicly available dataset for ee in the portuguese language that encompasses annotation both on event occurrences and their arguments, which include event types and argument roles. between the two public available corpora found in the literature [8, 22], the timebankpt meets most of our requirements, as it was annotated following a well-defined guideline, which is well-known and used to guide the annotation of corpora in other languages [23]. also, it provides others annotation information, besides event mentions, that can be used to perform related tasks, such as event ordering. it is important to notice that the notion event in timebankpt guidelines agrees with that of ace 2005 guidelines, which we adopt in this work. thus, we consider this corpus as an interesting resource to base our work. while timebankpt describes classes attributes, it does not provide a fine-grain distinction of the semantics of events by means of a semantically relevant typology of events. more yet, it does not contain annotations of event arguments and their roles, which are necessary for our problem. as such, we enrich the timebankpt corpus with annotation about event type and argument roles. in this annotation, we employ the event type schemata from the framenet project to define event types and argument roles. we adopted a semi-automatic annotation process, in which firstly, the lome [29] frame parser automatically annotated all the sentences in the corpus, followed by a manual validation and correction by two human annotators. from the 7887 events originally annotated in timebankpt, 6220 (79%) were automatically identified by lome and analyzed by the human annotators. we measured the inter-annotator over 10% of a common set of event type annotations to assess the reliability of the annotation process. the annotators achieved an inter-annotator agreement of 0.78 as measured by the cohen’s kappa [5] coefficient, which is considered a good value. we evaluate the proposed model for ee on this tefa (timebankpt event frames annotation) dataset obtained as a result of our annotation process. the tefa corpus contains annotation of 6620 event mentions of 514 different types and 1936 arguments. 6 experiments in the following, we describe our empirical investigation on ee for the portuguese language, based on the joint identification and classification model proposed in sect. 4 and evaluated on the tefa corpus described in the previous section. 6.1 dataset, parameters, and resources for all the experiments, in the trigger and sentence encoding phase, we use 768 dimensions for the word embeddings. we extracted these embeddings from the bertimbau base model, where we compose the resulting wordpiece sub-tokens by adding them. we use 350 units in the hidden layers for the lstms. for the trigger representation, during training, we use a dropout layer of 25% rate over the input word embedding followed by a feed-forward neural network with 700 units with relu activation function. after this layer, we employed a batch normalization layer [14], and another dropout layer of 50% rate. regarding the argument prediction phase, at each time step of the bilstm, of the sentence representation phase, we concatenate the trigger representation with the two hidden unit vectors, constructing a vector representation of 1400 activations, and applied a time distributed feed-forward layer with 700 units with relu activation function followed by a final softmax layer for probability distribution prediction for the argument role labels. for the ed part of the model, we apply a softmax dense layer over the shared event trigger representation. finally, during training, we use a mini-batch of size 128. these parameter values were selected according to the validation data. we applied the pre-trained language representations from bertimbau-base in a feature-based approach. we follow devlin et al. [9], applying a feature-based approach by extracting the activations from one or more layers without fine-tuning any parameters of bertimbau-base. during the model development, we evaluated three different strategies: capturing the last hidden layer activations, summing the last four, and summing all the twelve. from now on we will refer to these strategies as last hidden, sum last 4, and sum all 12, respectively. regarding the event types and argument roles labels, we consider a set of 514 event types from framenet [2] plus the none type, which identifies a word as not denoting an event, and a set of 1936 argument roles types as defined in framenet [2]. we adopted the bio annotation schema to assign argument role labels to each token in the sentence, achieving a total of 3873 labels (two times the number of argument role types, corresponding to the labels begin and in for each argument role type, plus one outside label). to evaluate our method, we defined two scenarios considering the amount of annotated examples for each event type. the first consisted only of event mentions of target types containing more than 100 examples in the training corpus, consisting of 7 event types with 93 argument roles. the second consisted of all event types annotated in the corpus, containing 514 event types with 1936 argument roles. the purpose of this procedure was to investigate the influence of the number of training examples in the performance of the method and possibly infer the effort to further support new event types. we use the criteria for evaluation described on sect. 3. to provide a meaningful comparison with possible future work, we employ the precision (p), recall (r), and \(f_1\) score (\(f_1\)) evaluation metrics. we report the micro-average scores for event type classification and argument role prediction as in previous work [18]. we also show the \(f_1\) micro-average score for argument role label prediction on the test set. 6.2 data augmentation via oie in order to obtain more annotated data and possibly increase the robustness of our method, we explore the use of a data augmentation method for the task of ee using new sentences formed by open information extraction (oie) triples over the event annotated sentences from the original corpus. to the best of our knowledge, this strategy of data augmentation for the task of ee has not been mentioned in the literature. oie is the task of extracting facts, or simple propositions, from text without limiting the analysis to a predefined set of relationships or predicates [11]. as described by oliveira et al. [21], oie systems usually extract facts from a sentence in the form of triples \(t = (arg1, rel, arg2)\), where arg1 and arg2 are two arguments and rel describes a semantic relationship between arg1 and ag2. we argue that this technique of ee data augmentation can boost the model performance because it provides simpler syntactic realizations of event mentions to model the relationship between an event trigger and their arguments. to make sense of the previous argument, consider the following sentence: “na bolsa de valores americana, as ações da citadel fecharam ontem nos 45,75 dólares, descendo 25 cêntimos.”footnote 6. one of the triples obtained by the execution of dptoie [21]footnote 7, an oie system for the portuguese language, over the previous sentence was: (“as ações da citadel”, “fecharam”, “nos 45,74 dólares”). if we consider as a new sentence, which we refer to as a synthetic sentence, simply the concatenation of arg1, rel and arg2 from the extracted triple, we obtain the sentence: “as ações da citadel fecharam nos 45,74 dólares”footnote 8. in most cases, this simple approach produces grammatical and semantic correct sentences. the synthetic sentences obtained from the triples are at most the same length that the original sentences, and those that are equal we do not count as new ones. our hypothesis is that this data augmentation approach will be beneficial when dealing with long sentences for which the synthetic sentences are shorter and include the relevant arguments because the new sentence will help reduce the number of long-term dependencies in the input sequence the model needs to remember. also, as we maintain the original sentence it will not prohibit the model from learning these dependencies from more complex and real examples. we create the augmented dataset by iterating into the extractions from the oie system, producing the synthetic sentences and selecting the one where the event trigger is part of the relation descriptor in the obtained triple. following that, we transfer the annotations of the extracted sentence to the synthetic one. the application of this data augmentation strategy over the original dataset produced an increase of 192% of annotated sentences (i.e. from 1962 to 5747 examples), 70% of event triggers, i.e. from 5345 to 9130 examples, and 56% of arguments, i.e. from 10727 to 16779 examples. 6.3 word embedding layers selection we evaluate different approaches to extract the activations from layers of bertimbau to obtain the pretrained word embeddings, which we employ as input features to the joint model for ee. table 1 presents the performance on the development set of the augmented training set when last hidden, sum last 4, and sum all 12 layers extractions are used to obtain the word embeddings from the same corpus over the two different scenarios (more than 100 examples and more than 0 examples). the development set also contains sentences resulting from the data augmentation via oie. table 1. average performance of models over development set with augmentation using 10-fold cross-validation evaluation.full size table we can observe from the results depicted in table 1, that the last hidden input word embeddings performs best for the task of arp, and the sum all 12 performs best for the ed subtask. we observe the same pattern on the original development set evaluation, i.e. without data augmentation. from these results, we chose to prioritize the arp subtask by choosing the last hidden combination as the input word embeddings to produce the final models on both original and augmented datasets. 6.4 test evaluation table 2 shows the results of the comparison between the proposed method trained in four different scenarios, where we compare the performance of the models trained on the original tefa data (blstme) and over the augmented training set (blstmea). we notice that data augmentation strategy was beneficial to the task of arp when the data per event type and argument role were more scarce, and the bio tags labeling task performance does not match the performance on argument role classification. however, we did not notice the same benefit on event types and, consequently, arguments roles with a greater number of examples in the corpus. more than that, the augmentation appears to diminish the \(f_1\) performance of the model, but in contrast, it raises the arp individual labels classification. table 2. performance of the four models over test set of trained over the original (blstme) and augmented training sets (blstmea) over two event types distributions.full size table 7 error analysis we examine the outputs of blstme-0 on the test set to find out the contributions of each event type to event detection errors. following nguyen et al. [20], we highlight two types of errors: (i) missing an event mention in the test set (called missed), and (ii) incorrectly detecting an event mention (called incorrect). in table 3, we list the top five event types appearing in these two types of errors and their corresponding percents over the total number of errors, and the number of available event mentions on the training and testing data sets. the total number of event types present on the test set amounts to 244. these top five event types amount to 8.1% of the missed errors and 31.1% of the incorrect errors. the missed errors are more evenly distributed between the event types than the incorrect errors. examples from the statement event type appears in both top five type of errors, and it is ranked in the first place primarily due to the greater amount of examples on the test set. when we observe one of the missed errors from the statement event type, even the trigger word “relata” being available on the training set, due to a different and somehow confusing context, the model failed to detect the event mention in the following sentence: “a correspondente da bbc karyn coleman relata do kosovo.”footnote 9. when we change the word order of this sentence: “do kosovo, a correspondente da bbc karyn coleman relata.” the model was able to detect the statement event mention. despite this example, the missed errors were detected more frequently in cases where the trigger words are not present in the training data. from the analysis of the incorrect errors, we notice that similar context, when the sentence contains event arguments common to different event types, the most predominant example event type hinder the performance of similar event types. as an example of such a case, in the sentence: “os funcionários americanos afirmam que já veem sinais de que saddam hussein está a ficar nervoso.”footnote 10, the model detected a statement event type, instead of affirm_or_deny event type, despite a very indicative trigger word. this may indicate an over-reliance of the model on lexical information, as opposed to the semantic information we expect to be encoded in the word and sentence representation stages of our architecture. another explanation may reside in the much fine-grained schemata that we employed and the imbalance between these two event type annotated examples. table 3. top five event types for ed errors.full size table 8 conclusion in this paper, we have presented a method for event extraction in the portuguese language over an enriched event frame annotated corpus constructed from the timebankpt corpus. our approach requires less effort on feature engineering, using as input only contextualized word embeddings, and jointly performs ed and arp. we also proposed and evaluated a method of data augmentation for the ee task using an oie system. the annotation process required to produce the corpus turns out to be tested by this experimentation evaluation, validating the process of annotation, using three language resources already available: the timebankpt corpus, the lome frame parser, and the framenet schemata. our experimental results show that our method is able to learn and automatically predict with good precision some event types and argument roles over the annotated corpus. also, we notice an improvement in the performance on the argument role prediction for the models that used the augmented data from oie when dealing with event types and argument roles with scarce annotated examples, showing that this strategy may be valuable to improve the performance of ee systems for languages with little training data for the task, as is the case of the portuguese language. notes 1.s1: “... the arabs would have to support iraq in a fight against their common israeli enemy.”. 2.s2: “both companies are allies of navigation mixte in its fight against a hostile takeover bid ...”. 3.“said”, “referred”, “announced”, “added”. 4.“meridian national corp. said it sold 750,000 shares of its common stock to the mcalpine family interests, for $1 million, or $1.35 a share.” 5.https://github.com/kraiyani/automated-event-extraction-model-for-multiple-linked-portuguese-documents. 6.“in american stock exchange composite trading, citadel shares closed yesterday at $45.75, down 25 cents”. 7.available at: https://github.com/formas/dptoie. 8.“citadel shares closed at $45.75”. 9.“bbc correspondent karyn coleman reports from kosovo.”. 10.“u.s. officials claim they already see signs saddam hussein is getting nervous.”. references ahn, d.: the stages of event extraction. in: proceedings of the workshop on annotating and reasoning about time and events, pp. 1–8 (2006) google scholar  baker, c.f., fillmore, c.j., lowe, j.b.: the berkeley framenet project. in: 36th annual meeting of the association for computational linguistics and 17th international conference on computational linguistics, vol. 1, pp. 86–90 (1998) google scholar  carvalho, p., gonçalo oliveira, h., santos, d., freitas, c., mota, c.: segundo harem: modelo geral, novidades e avaliaçao. quot; in cristina mota; diana santos (ed) desafios na avaliação conjunta do reconhecimento de entidades mencionadas: o segundo harem linguateca 2008 (2008) google scholar  chen, y., xu, l., liu, k., zeng, d., zhao, j.: event extraction via dynamic multi-pooling convolutional neural networks. in: proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 1: long papers), pp. 167–176 (2015). https://doi.org/10.3115/v1/p15-1017 cohen, j.: a coefficient of agreement for nominal scales. educ. psychol. meas. 20(1), 37–46 (1960). https://doi.org/10.1177/001316446002000104 article  google scholar  consortium, l.d.: ace (automatic content extraction) english annotation guidelines for events. version (5.4.3) (2005) google scholar  costa, f., branco, a.: lx-timeanalyzer: a temporal information processing system for portuguese (2012). http://hdl.handle.net/10451/14148 costa, f., branco, a.: timebankpt: a timeml annotated corpus of portuguese. in: lrec, pp. 3727–3734 (2012) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2018). https://doi.org/10.18653/v1/n19-1423 ding, r., li, z.: event extraction with deep contextualized word representation and multi-attention layer. in: gan, g., li, b., li, x., wang, s. (eds.) adma 2018. lncs (lnai), vol. 11323, pp. 189–201. springer, cham (2018). https://doi.org/10.1007/978-3-030-05090-0_17 chapter  google scholar  glauber, r., de oliveira, l.s., sena, c.f.l., claro, d.b., souza, m., et al.: challenges of an annotation task for open information extraction in portuguese. in: villavicencio, a. (ed.) propor 2018. lncs (lnai), vol. 11122, pp. 66–76. springer, cham (2018). https://doi.org/10.1007/978-3-319-99722-3_7 chapter  google scholar  hinton, g., srivastava, n., swersky, k.: neural networks for machine learning lecture 6a overview of mini-batch gradient descent. cited on 14(8), 2 (2012) google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997). https://doi.org/10.1162/neco.1997.9.8.1735 article  google scholar  ioffe, s., szegedy, c.: batch normalization: accelerating deep network training by reducing internal covariate shift. in: international conference on machine learning, pp. 448–456. pmlr (2015) google scholar  ji, h., grishman, r.: refining event extraction through cross-document inference. in: proceedings of acl-08: hlt, pp. 254–262 (2008). https://aclanthology.org/p08-1030 li, q., ji, h., huang, l.: joint event extraction via structured prediction with global features. in: proceedings of the 51st annual meeting of the association for computational linguistics (volume 1: long papers), pp. 73–82. association for computational linguistics (2013). https://www.aclweb.org/anthology/p13-1008 liu, j., chen, y., liu, k., zhao, j.: event detection via gated multilingual attention mechanism. in: proceedings of the aaai conference on artificial intelligence, vol. 32 (2018) google scholar  nguyen, t.h., cho, k., grishman, r.: joint event extraction via recurrent neural networks. in: proceedings of the 2016 conference of the north american chapter of the association for computational linguistics: human language technologies, pp. 300–309. association for computational linguistics (2016). https://doi.org/10.18653/v1/n16-1034 nguyen, t.h., grishman, r.: event detection and domain adaptation with convolutional neural networks. in: proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing (volume 2: short papers), pp. 365–371. association for computational linguistics (2015). https://doi.org/10.3115/v1/p15-2060 nguyen, t.m., nguyen, t.h.: one for all: neural joint modeling of entities and events. in: proceedings of the aaai conference on artificial intelligence, vol. 33, pp. 6851–6858 (2019) google scholar  oliveira, l.d., claro, d.b.: dptoie: a portuguese open information extraction system based on dependency analysis (2019) google scholar  quaresma, p., nogueira, v.b., raiyani, k., bayot, r.: event extraction and representation: a case study for the portuguese language. information 10(6), 205 (2019). https://doi.org/10.3390/info10060205 article  google scholar  saurı, r., littman, j., knippen, b., gaizauskas, r., setzer, a., pustejovsky, j.: timeml annotation guidelines version 1.2. 1 (2006) google scholar  schuster, m., paliwal, k.k.: bidirectional recurrent neural networks. ieee trans. sig. process. 45(11), 2673–2681 (1997). https://doi.org/10.1109/78.650093 article  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, october 20–23 (2020). https://doi.org/10.1007/978-3-030-61377-8_28 wadden, d., wennberg, u., luan, y., hajishirzi, h.: entity, relation, and event extraction with contextualized span representations. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 5788–5793 (2019). https://doi.org/10.18653/v1/d19-1585 walker, c., strassel, s., medero, j., maeda, k.: ace 2005 multilingual training corpus. linguist. data consortium philadelphia 57, 45 (2006) google scholar  wu, y., et al.: google’s neural machine translation system: bridging the gap between human and machine translation. arxiv preprint arxiv:1609.08144 (2016) xia, p., et al.: lome: large ontology multilingual extraction. in: proceedings of the 16th conference of the european chapter of the association for computational linguistics: system demonstrations, pp. 149–159. association for computational linguistics (2021). https://aclanthology.org/2021.eacl-demos.19 xiang, w., wang, b.: a survey of event extraction from text. ieee access 7, 173111–173137 (2019) article  google scholar  download references acknowledgments anderson da silva brito sacramento would like to thank coordenação de aperfeiçoamento de pessoal de nível superior capes for financial support (88887. 467864/2019-00). author information authors and affiliations federal university of bahia ufba, salvador-bahia, brazil anderson da silva brito sacramento & marlo souza authors anderson da silva brito sacramentoview author publications search author on:pubmed google scholar marlo souzaview author publications search author on:pubmed google scholar corresponding author correspondence to marlo souza . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper sacramento, a.d.s.b., souza, m. (2021). joint event extraction with contextualized word embeddings for the portuguese language. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_34 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_34 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords event extraction information extraction natural language processing portuguese language publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature growing self-organizing maps for multi-label classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper growing self-organizing maps for multi-label classification conference paper first online: 30 january 2025 pp 33–48 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) growing self-organizing maps for multi-label classification download book pdf download book epub pedro henrique casarotto9 & ricardo cerri  orcid: orcid.org/0000-0002-2582-169510  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 399 accesses 1 citation abstract in machine learning, multi-label classification is the problem of simultaneously classifying an instance into two or more labels. it is a challenging problem since each label has its specialty, and correlations between them must be considered. a self-organizing map (som) is a neural network where neurons organized in a grid are tuned to represent the input instances in self-organization. after tuning, similar instances in the input space are mapped to closer neurons in the grid. soms have already been used for multi-label problems, obtaining competitive results with other methods. however, the static nature of their grid of neurons is a disadvantage since it is difficult to define the optimized grid size for each problem. the growing self-organizing maps (gsom) extends the soms, allowing the network to grow during execution based on the data characteristics. this paper proposes a gsom to predict multi-label data. the experiments showed that gsom obtained better or more competitive results in most of the datasets investigated compared to som and had a competitive performance compared to other methods. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others incorporation of neighborhood concept in enhancing som based multi-label classification chapter © 2019 improving self-organizing maps with unsupervised feature extraction chapter © 2020 enhancing visual clustering using adaptive moving self-organizing maps (amsom) chapter © 2017 1 introduction in the machine learning context, the most common classification problem is called single-label. this problem involves classifying a data input in a single class l from a set of classes l, where \(|l| > 1\). if the data input is expected to be classified over two or more classes simultaneously, then the problem is considered a multi-label classification problem. in the literature, this problem is presented over diverse areas, such image classification and recognition [8, 22], text and document classification [9, 12], music classification [33, 40], in biomedicine, such in genome classification [39, 42] and protein function prediction [29, 30]. there are many strategies to deal with multi-label problems, divided into two main approaches. the first is called the local approach, in which traditional classification algorithms independently predict each label. the other strategy is global, which considers label correlations by creating specific algorithms to deal with the problem and all labels simultaneously. a hybrid of these two approaches is also possible [19]. colombini et al. [17] proposed using self-organizing maps [21] to solve the multi-label problems. self-organizing maps (som), or kohonen maps, is an unsupervised neural network model containing a one-layer grid of neurons. each neuron has a discriminant function measuring the similarity of the neuron weights to an input instance. in addition, colombini et al. associated a prototype vector to each neuron, which is the mean of the input label vectors of the instances mapped to the neuron. when the model receives a new instance, it is compared with all weights of the neurons of the grid. the neuron that shows more similarity with the instance is selected as the winner neuron, and his prototype vector is returned as the prediction for that instance. the growing self-organizing maps (gsom) [6] are an extension of soms in which the number of neurons in the grid increases during training. the increase rate is established by the growth threshold, a parameter that compares the total error of the neuron with the input data. a new neuron is created if the error value exceeds the growth threshold. the benefit of this adaptation is that the number of neurons is more precise, and thus, the results are better. in this paper, we propose a method for multi-label classification using growing self-organizing maps. the method is called growing self-organizing maps multi-label learning (gsom-mll), and improves the som-mll proposed in the work of colombini et al. [17]. the remainder of this paper is organized as follows. section 2 contains a literature review of the most common algorithms for multi-label problems. section 3 presents an overview of the kohonen maps and introduces the som-mll proposed by colombini et al. [17]. our proposal, gsom-mll, is presented in sect. 4, while our methodology is presented in sect. 5. section 6 presents and discusses our experiments and results. finally, sect. 7 concludes the paper and points to future work. 2 multi-label classification this section presents the two main approaches to classifying multi-label data: local (problem transformation) and global (algorithm adaptation). 2.1 local approach the local approach reduces the original problem into many different single-label problems. each problem is individually solved, and the corresponding predictions are combined to form the final multi-label classification. the simplest local-based strategy is called binary-relevance (br) [35] in which l binary classifiers are trained, each associated with one of the l labels. one advantage of this strategy is that the classifiers can be run independently from each other (in parallel, for example), and the peculiarities of each label independently influence its corresponding classifier. another advantage is that traditional algorithms can be used. however, the disadvantage is that this strategy ignores the correlations between labels, a fundamental characteristic of multi-label problems. many proposed improvements in the br algorithm have been explored over the last few years. classifier chains (cc) [27] modifies br, adding a dependency mechanism in the labels. this is done by a chain that randomly connects the binary classifiers, considering the previous labels/predictions as input for training a classifier in the chain. it has the advantage of maintaining br’s simplicity while learning the labels’ relationships. a similar strategy is meta staking (ms) [20], which indirectly considers these relations by adding the number of examples based on the previous classification. in cherman et al. [15], br was explored using decision trees and naïve bayes to consider label relations. dembczyński et al. [14] is another approach that also uses the naïve bayes algorithm to optimize predictions. the brknn [32] is another approach that uses the knn algorithm conceptually equivalent to br, with the advantage of being faster. another common local-based strategy is called label-powerset (lp) [11] in which all classes assigned to each instance are turned into a new and unique class. this is a form to consider the correlations between classes, something br could not do. however, this method can escalate quickly and create many classes with few positive instances. a modified version called random k-labelsets (rakel) [37] creates combinations of label-powerset classifiers iteratively, which considers label correlations without escalating the problem. pruned sets (ps) [26] is a strategy that identifies the relation of the labels and eliminates the ones with less usage, grouping them based on their density and frequency while trying not to lose data. the ensemble of pruned sets (eps) is also presented as a version of the algorithm that avoids the model overfitting. 2.2 global approach the global approach creates classifiers that take into account label dependency in a direct way, dealing with all labels simultaneously. it uses only one classifier to predict the l labels of the problem. the advantage is that it considers the relationships between labels and is usually faster to execute. however, the approach does not give too much weight to each individual label’s peculiarity. additionally, the algorithm is more difficult to parallelize, bringing more computational cost to the algorithms and spending more training time. one of the first global-based strategies is c4.5m [16], a decision-tree-based method created by adapting the c4.5 algorithm to multi-label problems. this was done by altering the entropy formula. the algorithm was also changed to represent a set of labels in their tree leaf nodes. the mlknn method [41, 43] is based on the knn algorithm where, for each instance, the k closest neighbors classes are compared. based on this, the instance is labeled. in zhang and zhou [42], the back-propagation multi-label learning was proposed, in which a multi-label error measure is proposed and used in the output layer of an artificial neural network. in tsoumakas et al. [34], the hierarchy of multi-label classifiers (homer) is proposed to deal with large label sets, focusing on being effective and computationally efficient. this method uses a “divide and conquer” label hierarchy strategy. it was able to have linear training and logarithmic testing complexity. colombini et al. [17] proposed som-mll (self-organizing maps multi-label learning), implementing kohonen maps to predict multi-label data. this algorithm was also used by alshanqiti and namoun [7] to compare the method with other baseline models, like linear regression and matrix factorization. in saini et al. [28], a different structure of som for multi-label data was presented and called ml-som, where the label vector is determined using the closest and neighboring neurons, obtaining better results compared to som-mll in some cases. 3 multi-label classification with self-organizing maps this section provides an overview of self-organizing maps (kohonen maps) and the self-organizing map multi-label learning (som-mll) [17]. 3.1 self-organizing maps self-organizing maps (som) [21] are unsupervised neural networks with a fixed-size bi-dimensional grid of neurons. when an instance is mapped to the grid, its best-matching neuron is selected as the winning neuron, which strengthens its proximity to the instance. the mapping process is illustrated in fig. 1. fig. 1. mapping an input instance in the kohonen map full size image the training starts by calculating the euclidean distance between the input instance and all neuron weight vectors. the winning neuron is the one with the minimum distance according to eq. 1. in this equation, \(d_j\) is the distance between the input data \(\textbf{x}\) and the neuron j, with a the dimension of the input instance. each neuron weight vector is initialized with small random values. $$\begin{aligned} d_j (x) = \sqrt{\sum _{i=1}^{a} {(x_i w_{j,i})^2}} \end{aligned}$$ (1) as the winner neuron is defined, the value of its weights is updated based on the input data and the weights of the neurons in its neighborhood, according to eq. 2. the hyperparameter \(\alpha \) is the learning rate of the model, which typically decreases with training time. $$\begin{aligned} w_j (t+1) = w_j(t) + \alpha (t) \times h_{j,i} \times (\textbf{x} w_j(t)) \end{aligned}$$ (2) the \(h_{j,i}\) is the influence that the winning neuron j has in its neighborhood of neurons. the most commonly used formula is the gaussian, which can be seen in eq. 3. the \(\sigma \) hyperparameter defines the influence’s size, i.e., the neighborhood width. $$\begin{aligned} h_{j,i} = \exp \biggl (\, -\frac{d_{j}^2(x)}{2\sigma ^2} \biggl ) \end{aligned}$$ (3) the self-organizing map multi-label learning (som-mll) is trained similarly to the common som. the difference is that in the prediction process, multiple labels are considered instead of only one label. 3.2 predicting new data in the som-mll, every instance \(\textbf{x}_i\) is associated with a binary vector \(\boldsymbol{v}_i\), which has the value 1 in position j if the instance is classified in class \(c_j\), and 0 otherwise. after training, these binary vectors are used to construct a prototype vector \(\overline{\boldsymbol{v}}_i\) for each neuron. the prototype of a neuron n is the column-wise average of the matrix formed by all binary label vectors associated with the training instances that had the neuron n as their winning neuron. this calculation is presented in eq. 4, where \({s_{n,j}}\) is the set of training instances mapped to the neuron n classified in the class \(c_j\) and \({s_n}\) is the total instances mapped to neuron n. $$\begin{aligned} \overline{\textbf{v}}_{i,j} = \frac{|s_{n,j}|}{|s_n|} \end{aligned}$$ (4) the prototype vector represents the probability that the mapped instance \(x_i\) will be classified in each of the problem’s l labels. a threshold is applied to the prototype vector to obtain a final binary prediction. the common threshold used is 0.5, so that if \(\overline{\textbf{v}}_{i,j}\) is greater or equal to 0.5, \(x_i\) is classified in class \(c_j\). figure 2 shows an example of applying a threshold to a prototype vector. fig. 2. obtaining a final prediction by applying a threshold to a prototype vector full size image 4 growing self-organizing maps for multi-label classification this section introduces the growing self-organizing maps (gsom) and our proposal for a multi-label classification method called growing self-organizing maps multi-label learning (gsom-mll). 4.1 growing self-organizing maps the growing self-organizing maps [6] is an extension of the kohonen maps able to increase its number of neurons during training based on the characteristics of the input data. the algorithm has two phases: the growing phase, where new neurons are added to the map, and the smooth phase, where the weights of neurons are faceted by their neighbors. the algorithm has a hyperparameter called spread factor sf, \(0 \le sf \le 1\), which controls the number of grid nodes. lower sf values create more populated maps, while higher values create more concentrated maps. 4.2 growing phase the growing phase is responsible for creating new nodes on the map by means of a growth threshold (gt), a spread factor (sf), and the dimension d of the training data. this can be seen in eq. 5. $$\begin{aligned} gt = d \times \log {sf} \end{aligned}$$ (5) the euclidean distance (eq. 1) is used for each input training instance to find the corresponding closest neuron. after this winning neuron i is found, the weights are adjusted for all neurons j of the grid in the neighborhood influence (\(h_{j,i}\)) of neuron i. this adjustment procedure is the same one used in the original som (eqs. 1, 2 and 3). as an extension of soms, each neuron in gsom has an associated error value e used to compute the accumulated error in the neuron. this is shown in eq. 6, where \(\textbf{x}\) is the input sample, and a is the number of attributes. $$\begin{aligned} e_j(t+1) = e_j(t) + \sqrt{\sum _{i=1}^{a} {(x_i w_{j,i})^2}} \end{aligned}$$ (6) the largest error value e is called \(h_{e_{rr}}\), and is updated every time a new input instance is given to the model. if \(h_{e_{rr}} > gt\), the model creates a new node and sets \(h_{e_{rr}} = gt/2\). 4.3 node generation a new node will always be connected to a boundary node, a neuron with at least one of its four immediate neighboring positions free. figure 3 shows the process of a node generation, being (i) the original grid, (ii) the grid where a neuron has an error larger than gt, and (iii) the two possible positions where a new neuron can be inserted in the grid. fig. 3.adapted from alahakoon et al. [6] insertion of a new neuron in a grid. full size image the weights vector \(\textbf{w}_{new}\) of a new neuron is based on the weight vectors \(\textbf{w}1\) and \(\textbf{w}2\) of its neighbors so that certain regions have similar characteristics. figure 4 illustrates the four cases that dictate the weight initialization of a new neuron. all comparisons and operations in the four cases are executed individually for each weight of the vectors \(\textbf{w}1\) and \(\textbf{w}2\). 1. the new neuron has two consecutive older nodes on one of its sides (fig. 4(a)). in this case, the following rules are applied: if \(w2 > w1\), then \(w_{new} = w1 (w2 w1)\); if \(w1 > w2\), then \(w_{new} = w1 + (w1 w2)\). 2. the new neuron is in between two older nodes (fig. 4(b)). in this case: \(w_{new} = \frac{w1 + w2}{2}\). 3. the new neuron has only one direct neighbor: an older neuron (fig. 4(c)). in this case, the following rules are applied: if \(w2 > w1\), then \(w_{new} = w1 (w2 w1)\); if \(w1 > w2\), then \(w_{new} = w1 + (w1 w2)\). 4. the new neuron has only one direct neighbor, which is an old isolated neuron (fig. 4(d)). in this case: \(w_{new} = (r1 + r2)/2\), where r1 and r2 are the lower and upper values of the range of the weight vector distribution. fig. 4.adapted from alahakoon et al. [6] initializing weights of a new neuron in a grid. full size image 4.4 smoothing phase this phase adjusts the neuron’s weights, tuning the grid smoother. at each epoch, the learning rate \(\alpha \) is decreased according to eq. 7. $$\begin{aligned} \alpha (e+1) = \alpha \times v (e) \times \alpha (e) \end{aligned}$$ (7) in alahakoon et al. [6], v is defined as shown in eq. 8, where r is experimentally set to 3.8 in cases of a four nodes initialization grid, and n(e) the number of nodes in the beginning of that epoch. $$\begin{aligned} v (e) = 1 \frac{r}{n(e)} \end{aligned}$$ (8) the neighborhood around the winning neuron must also be reduced with training time. this is guaranteed decreasing the neighborhood width according to eq. 9. in the equation, \(\sigma _0\) is the neighborhood width in the algorithm’s initialization, while \(\tau \) is a time constant. $$\begin{aligned} \sigma (t) = \sigma _0 \exp \biggl (-\frac{t}{\tau }\biggl ) \end{aligned}$$ (9) 4.5 predicting new data after finishing training, each neuron is associated with a prototype vector \(\overline{\boldsymbol{v}}_i\) calculated the same way as som-mll using eq. 4. a threshold is then applied to obtain a binary final prediction vector (fig. 2). 5 materials and methods this section presents the datasets used in the experiments, the baseline algorithms used for comparison, the evaluation measures employed, and the tuning procedure employed to define the hyperparameters of gsom-mll. 5.1 datasets the datasets used are available in the mulan repositoryfootnote 1. table 1 shows some characteristics of each dataset: domain, the number of instances, the number of labels and distinct combinations of labels, and the label cardinality and label density. label cardinality (lc) is a metric of the average number of labels per instance, while label density (ld) is the lc metric over the total number of labels. equations (10) and (11) show ld and lc, where \(y_i\) is the set of positive labels for instance i, m is the total number of instances, and q is the total number of labels. table 1. characteristics of the datasetsfull size table $$\begin{aligned} lc = \frac{1}{m} \sum _{i=1}^{m} {|y_i|} \end{aligned}$$ (10) $$\begin{aligned} ld = \frac{1}{m} \sum _{i=1}^{m} \frac{|y_i|}{q} \end{aligned}$$ (11) 5.2 baseline classifiers the proposed gsom-mll was compared with the original som-mll proposed by colombini et al. [17], where a hexagonal grid of 25 neurons was used, the learning rate was set to decrease linearly from 0.05 to 0.01 at each epoch, and the gaussian function (eq. 3) was used as the neighborhood with \(\sigma = 2/3\). gsom-mll was also compared with different local and global-based methods. they are listed below. binary relevance (br) and label powerset (lp) [35], with the following algorithms as base classifiers: support vector machine (svm) [38], j48 decision tree induction [24] and k-nearest neighbors (knn) [5]; random k-labelsets (rakel) [37] and classifiers chains (cc) [27], both also using svm, j48 and knn as base classifiers; hierarchy of multi-label classifiers (homer) [23], using lp and br, with svm, j48 and knn as base classifiers; back-propagation multi-label learning (bpmll) [42], multi-label k-nearest neighbors (mlknn) [43], instance-based learning by logistic regression (iblr-ml and iblr-ml+) [13], and predictive clustering trees (clus) [10]. all the baseline classifiers were executed within mulan, a library for multi-label learning [36], using their default hyperparameter values. the knn-based methods used \(k = 3\). bpmll was executed with one hidden layer with its number of neurons equal to 80% of the number of input features. 5.3 evaluation measures we evaluated all classifiers using the well-known f1-score multi-label measure [20], which is the harmonic mean of precision and recall. its calculation is presented in eq. 12, with \(z_i\) the set of predicted labels, \(y_i\) the set of true labels, and m the number of instances. $$\begin{aligned} \text { f1-score} = \frac{1}{m} \sum _{i=1}^m \frac{2 \times |y_i \cap z_i| }{|y_i| + |z_i|} \end{aligned}$$ (12) 5.4 hyperparameters and tuning there are many gsom implementations available in the literature. adeu et al. [4], for example, evaluated three of them: pygsom python package [3], gsom python package [2] and the growingsom r package [1]. these are all implementations based on alahakoon et al. [6], and according to adeu et al., all have different implementation details focusing on specific applications that can slightly change the results. because of this, they chose to implement their own packagefootnote 2 focusing on their specific application. in the same direction, we created a new python package to meet our needs here. gsom-mllfootnote 3 is an adaptation of adeu et al. for multi-label classification. we used the area under the precision-recall curve (auprc) to find the best hyperparameter values for our method. a precision-recall curve is obtained by applying threshold values in the interval [0, 1] to the classifier’s outputs. after interpolation all the precision-recall points (pr-points), and posterior connecting them, we can calculate the area under the pr-curve [18]. we performed a grid search to investigate different ranges of values for the hyperparameters. for the initial learning rate, we tested 20 values ranging from 0.005 to 0.195; for the spread factor, we evaluated 20 values ranging from 0.05 to 0.95; and for the smoothing and growing factors, we tested values ranging from 1 to 5 and 1 to 10, respectively. we chose the combination of values that led to the best auprc value. the gaussian function with \(\sigma = 1\) for both growing and smoothing phases was used as the neighborhood influence. after obtaining the best hyperparameter values, the threshold of 0.5 was applied to the prototype vectors in order to obtain the final binary predictions. all grids were initialized with 4 neurons. 6 experiments and discussion all the experiments were performed using the iterative stratified 10-fold cross-validation proposed for multi-label data [31]. tables 2, 3, and 4 present, respectively, the average f1-score values and standard deviations over the 10 executions for each dataset and method. the best results are highlighted in boldface. table 2. comparing gsom-mll with the strategies br, lp, and ccfull size table table 3. comparing gsom-mll with the methods rakel and homerfull size table table 4. comparing gsom-mll with som-mll, mlknn, bpmll, iblr and clusfull size table the overall results show that gsom-mll is superior to som-mll and competitive with the performances of the other methods investigated. the gsom-mll f1-score was superior to som-mll, especially in the dataset flags and scene. we can observe that gsom-mll performed surprisingly badly in the birds dataset. this dataset has the smallest label density, meaning it has a very sparse distribution of labels assigned for each dataset instance. our tuning procedure may not have been effective in maximizing performance in these circumstances. overall, the experiments show us that dynamically increasing the neuron grid dimension is better than working with a fixed grid size. although the f1-score results were not as satisfactory as expected, gsom-mll is competitive with the other methods and is a promising strategy, especially for data visualization, since we can observe how instances are grouped in the grid. we can have a 2d visualization, keeping the original properties of the input space. table 5 presents the mean values for each hyperparameter after the tuning over the 10-fold cross-validation procedure: the number of growing and smooth epochs, the spread factors, the learning rates, the time in seconds to run the model, and the auprc values. table 5. mean values of the gsom hyperparameters after tuningfull size table table 5 also shows that a high spread factor is preferred for most datasets, indicating a large grid instead of a grid with few neurons. another observation is that the learning rate is well-suited to most datasets, except for birds, which is very close to the maximum value used during the tuning stage (0.195). table 6 shows each dataset’s final number of neurons in the grid. the results of the gsom-mll experiment suggest that the number of neurons in the grid significantly influenced the outcomes. this is because all the datasets in the original som-mll experiment had a fixed grid of 25 neurons, while in gsom-mll experiment, all datasets started with a grid of 4 neurons. table 6. final number of neurons in the gsom-mll gridfull size table another point to consider is that the time spent on each iteration of the algorithm correlates with the number of neurons on the final grid. this is because more neurons can lead to increased computational costs. 7 conclusion and future work in this work, we presented gsom-mll, a growing self-organizing map tailored for multi-label classification. the model’s overall performance proved to be an improvement over the original som-mll. the high values of the spread factor show that the datasets benefit from a higher number of neurons in a dynamic grid. unfortunately, gsom-mll performs poorly in datasets with balanced label cardinality and small label density with many distinct label sets. however, it shows consistent and competitive results in other scenarios. in future works, we can explore the neighborhood’s influence on the creation of the prototype vector. exploring different learning rate ranges also allows for analyzing the model’s performance, as many datasets have their optimal value near the limits tested when considering error rates. another way to expand the research would be to utilize hierarchical growing self-organizing maps [25]. this involves creating a hierarchy of gsoms based on a hierarchical threshold as a direct extension of som and gsom. notes 1.https://mulan.sourceforge.net/datasets-mlc.html. 2.https://github.com/rodrigosales/gsom. 3.https://github.com/pedrohcasarotto/gsom-mll. references growingsom r package. https://github.com/alexhunziker/growingsom gsom the growing self organizing map implementation on python. https://github.com/anantadata/gsom pygsom a gsom (growing self-organizing map) implementation for python. https://github.com/philippludwig/pygsom adeu, r.s., ferreira, k.r., andrade, p.r., santos, l.: evaluating growing self-organizing maps for satellite image time series clustering. in: geoinfo, 20 years after! p. 243 (2019) google scholar  aha, d.w., kibler, d., albert, m.k.: instance-based learning algorithms. mach. learn. 6(1), 37–66 (1991) article  math  google scholar  alahakoon, d., halgamuge, s.k., srinivasan, b.: dynamic self-organizing maps with controlled growth for knowledge discovery. ieee trans. neural netw. 11(3), 601–614 (2000) article  math  google scholar  alshanqiti, a., namoun, a.: predicting student performance and its influential factors using hybrid regression and multi-label classification. ieee access 8, 203827–203844 (2020) article  google scholar  baltruschat, i.m., nickisch, h., grass, m., knopp, t., saalbach, a.: comparison of deep learning approaches for multi-label chest x-ray classification. sci. rep. 9(1), 1–10 (2019) article  google scholar  banerjee, s., akkaya, c., perez-sorrosal, f., tsioutsiouliklis, k.: hierarchical transfer learning for multi-label text classification. in: proceedings of the 57th annual meeting of the association for computational linguistics, pp. 6295–6300 (2019) google scholar  blockeel, h., raedt, l.d., ramon, j.: top-down induction of clustering trees. in: proceedings of the fifteenth international conference on machine learning, icml 1998, pp. 55–63. morgan kaufmann publishers inc., san francisco, ca, usa (1998) google scholar  boutell, m.r., luo, j., shen, x., brown, c.m.: learning multi-label scene classification. pattern recogn. 37(9), 1757–1771 (2004) article  math  google scholar  chen, q., et al.: multi-label classification for biomedical literature: an overview of the biocreative vii litcovid track for covid-19 literature topic annotations. database 2022, baac069 (2022) google scholar  cheng, w., hüllermeier, e.: combining instance-based learning and logistic regression for multilabel classification. mach. learn. 76(2–3), 211–225 (2009) article  math  google scholar  cheng, w., hüllermeier, e., dembczynski, k.j.: bayes optimal multilabel classification via probabilistic classifier chains. in: proceedings of the 27th international conference on machine learning, icml-10, pp. 279–286 (2010) google scholar  cherman, e.a.: aprendizado de máquina multirrótulo: explorando a dependência de rótulos e o aprendizado ativo. ph.d. thesis, universidade de são paulo (2013) google scholar  clare, a., king, r.d.: knowledge discovery in multi-label phenotype data. in: de raedt, l., siebes, a. (eds.) pkdd 2001. lncs (lnai), vol. 2168, pp. 42–53. springer, heidelberg (2001). https://doi.org/10.1007/3-540-44794-6_4 chapter  math  google scholar  colombini, g.g., de abreu, i.b.m., cerri, r.: a self-organizing map-based method for multi-label classification. in: 2017 international joint conference on neural networks (ijcnn), pp. 4291–4298. ieee (2017) google scholar  davis, j., goadrich, m.: the relationship between precision-recall and roc curves. in: international conference on machine learning, pp. 233–240 (2006) google scholar  gatto, e.c., valejo, a.d.b., ferrandin, m., cerri, r.: community detection for multi-label classification. in: naldi, m.c., bianchi, r.a.c. (eds.) intelligent systems, bracis 2023. lncs, vol. 14195, pp. 78–93. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_6 godbole, s., sarawagi, s.: discriminative methods for multi-labeled classification. in: dai, h., srikant, r., zhang, c. (eds.) pakdd 2004. lncs (lnai), vol. 3056, pp. 22–30. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-24775-3_5 chapter  math  google scholar  kohonen, t.k.: the self-organizing map. proc. ieee 78(9), 1464–1480 (1990) article  math  google scholar  li, q., peng, x., qiao, y., peng, q.: learning category correlations for multi-label image recognition with graph networks. arxiv preprint arxiv:1909.13005 (2019) papanikolaou, y., tsoumakas, g., katakis, i.: hierarchical partitioning of the output space in multi-label data. data knowl. eng. 116, 42–60 (2018) article  math  google scholar  quinlan, j.r.: c4.5: programs for machine learning. morgan kaufmann publishers inc., san francisco, ca, usa (1993) google scholar  rauber, a., merkl, d., dittenbach, m.: the growing hierarchical self-organizing map: exploratory analysis of high-dimensional data. ieee trans. neural netw. 13(6), 1331–1341 (2002) article  math  google scholar  read, j., pfahringer, b., holmes, g.: multi-label classification using ensembles of pruned sets. in: 2008 8th ieee international conference on data mining, pp. 995–1000. ieee (2008) google scholar  read, j., pfahringer, b., holmes, g., frank, e.: classifier chains for multi-label classification. in: buntine, w., grobelnik, m., mladenić, d., shawe-taylor, j. (eds.) ecml pkdd 2009. lncs (lnai), vol. 5782, pp. 254–269. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-04174-7_17 chapter  math  google scholar  saini, n., saha, s., bhattacharyya, p.: incorporation of neighborhood concept in enhancing som based multi-label classification. in: deka, b., maji, p., mitra, s., bhattacharyya, d.k., bora, p.k., pal, s.k. (eds.) premi 2019. lncs, vol. 11941, pp. 91–99. springer, cham (2019). https://doi.org/10.1007/978-3-030-34869-4_11 chapter  math  google scholar  santos, b.z., nakano, f.k., cerri, r., vens, c.: predictive bi-clustering trees for hierarchical multi-label classification. in: hutter, f., kersting, k., lijffijt, j., valera, i. (eds.) ecml pkdd 2020. lncs (lnai), vol. 12459, pp. 701–718. springer, cham (2021). https://doi.org/10.1007/978-3-030-67664-3_42 chapter  math  google scholar  schietgat, l., vens, c., struyf, j., blockeel, h., kocev, d., džeroski, s.: predicting gene function using hierarchical multi-label decision tree ensembles. bmc bioinf. 11(1), 1–14 (2010) article  math  google scholar  sechidis, k., tsoumakas, g., vlahavas, i.: on the stratification of multi-label data. in: gunopulos, d., hofmann, t., malerba, d., vazirgiannis, m. (eds.) ecml pkdd 2011, part iii. lncs (lnai), vol. 6913, pp. 145–158. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-23808-6_10 chapter  math  google scholar  spyromitros, e., tsoumakas, g., vlahavas, i.: an empirical study of lazy multilabel classification algorithms. in: darzentas, j., vouros, g.a., vosinakis, s., arnellos, a. (eds.) setn 2008. lncs (lnai), vol. 5138, pp. 401–406. springer, heidelberg (2008). https://doi.org/10.1007/978-3-540-87881-0_40 chapter  math  google scholar  trohidis, k., tsoumakas, g., kalliris, g., vlahavas, i.p., et al.: multi-label classification of music into emotions. in: ismir, vol. 8, pp. 325–330 (2008) google scholar  tsoumakas, g., katakis, i., vlahavas, i.: effective and efficient multilabel classification in domains with large number of labels. in: proceeding of the ecml/pkdd 2008 workshop on mining multidimensional data, mmd 2008, vol. 21, pp. 53–59 (2008) google scholar  tsoumakas, g., katakis, i., vlahavas, i.: mining multi-label data. in: data mining and knowledge discovery handbook, pp. 667–685 (2010) google scholar  tsoumakas, g., spyromitros-xioufis, e., vilcek, j., vlahavas, i.: mulan: a java library for multi-label learning. j. mach. learn. res. 12, 2411–2414 (2011) mathscinet  math  google scholar  tsoumakas, g., vlahavas, i.: random k-labelsets: an ensemble method for multilabel classification. in: kok, j.n., koronacki, j., mantaras, r.l., matwin, s., mladenič, d., skowron, a. (eds.) ecml 2007. lncs (lnai), vol. 4701, pp. 406–417. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-74958-5_38 chapter  math  google scholar  vapnik, v.n.: the nature of statistical learning theory. information science and statistics, springer, new york (1999). https://doi.org/10.1007/978-1-4757-3264-1 book  math  google scholar  vens, c., struyf, j., schietgat, l., džeroski, s., blockeel, h.: decision trees for hierarchical multi-label classification. mach. learn. 73(2), 185–214 (2008) article  math  google scholar  wu, b., zhong, e., horner, a., yang, q.: music emotion recognition by multi-label multi-layer multi-instance multi-view learning. in: proceedings of the 22nd acm international conference on multimedia, pp. 117–126 (2014) google scholar  zhang, m.l., zhou, z.h.: a k-nearest neighbor based algorithm for multi-label classification. in: 2005 ieee international conference on granular computing, vol. 2, pp. 718–721. ieee (2005) google scholar  zhang, m.l., zhou, z.h.: multilabel neural networks with applications to functional genomics and text categorization. ieee trans. knowl. data eng. 18(10), 1338–1351 (2006) article  math  google scholar  zhang, m.l., zhou, z.h.: ml-knn: a lazy learning approach to multi-label learning. pattern recogn. 40(7), 2038–2048 (2007) article  math  google scholar  download references acknowledgment this study was financed by the national council for scientific and technological development (cnpq) and the são paulo research foundation (fapesp) grant #2022/02981-8. author information authors and affiliations department of computing, federal university of são carlos, são carlos, sp, brazil pedro henrique casarotto institute of mathematics and computer science, university of são paulo, são carlos, sp, brazil ricardo cerri authors pedro henrique casarottoview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar corresponding author correspondence to ricardo cerri . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper henrique casarotto, p., cerri, r. (2025). growing self-organizing maps for multi-label classification. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_3 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords growing self-organizing maps multi-label classification publish with us policies and ethics profiles ricardo cerri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature euclidean alignment for transfer learning in multi-band common spatial pattern | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper euclidean alignment for transfer learning in multi-band common spatial pattern conference paper first online: 30 january 2025 pp 430–443 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) euclidean alignment for transfer learning in multi-band common spatial pattern download book pdf download book epub marcelo m. amorim  orcid: orcid.org/0009-0003-1181-09229, leonardo prata  orcid: orcid.org/0009-0007-2976-36339, joão stephan maurício  orcid: orcid.org/0009-0002-0176-040x9, alex borges  orcid: orcid.org/0000-0003-0821-126x9, heder bernardino  orcid: orcid.org/0000-0003-2012-78029 & … gabriel de souza  orcid: orcid.org/0000-0002-5931-654x9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 488 accesses 1 citation abstract stroke is a major cause of death and disability worldwide. motor-imagery based brain-computer interface (mi-bci) models offer a post-stroke rehabilitation option. existing studies for mi-bci use transfer learning techniques like euclidean alignment (ea) but lose important brain information due to bandpass filtering. this study introduces new bci architecture with multi-band temporal filters and ea. the methods considered here are filter bank (fb), empirical mode decomposition (emd), and continuous wavelet transform (cwt). results show performance improvements, especially with ea being applied before filter bank. these models offer promise for post-stroke rehabilitation, particularly when using ea before the multi-band filter. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others session-independent subject-adaptive mental imagery bci using selective filter-bank adaptive riemannian features article 03 june 2024 optimal selection of frequency band and wavelet scattering for feature extraction in multi-class eeg classification article 06 february 2025 multiclass motor imagery classification with riemannian geometry and temporal-spectral selection article 09 may 2024 1 introduction stroke is a leading cause of disability and mortality worldwide, demands urgent attention due to its deep impact on individuals and healthcare systems [18]. over the years, significant strides have been made in stroke treatment, revolutionizing patient care and outcomes. one of the most recent methods of treatment is the utilization of brain-computer interface models. brain-computer interface (bci) is a system that allows for direct communication between the brain and an electronic device without peripheral muscles [3]. it translates the electrical signal of the brain in a task to a computational system. invasive and non-invasive acquisition methods can collect the brain’s signal. the invasive ways have a better signal-to-noise ratio (snr), being between 10 to 100 times better than non-invasive methods, but are more dangerous for the subject, since they carry risks such as infection, tissue damage, and implant rejection due to their surgical nature. the non-invasive equipment is safer than the invasive ones, but they have a more imprecise signal. electroencephalography (eeg) is the most used equipment for bci as it is not invasive, has a satisfactory temporal resolution in the range of milliseconds (same as the invasive ones), and is portable, light, and 10 times cheaper than the invasive ones [2]. the bci models are sensitive to the subject who provided the brain’s signals to train the model. due to the non-stationary behavior of the eeg signals and the anatomic difference between people’s brains, the model is usually subject-dependent. transfer learning (tl) techniques can decrease the signal difference between subjects and reduce that problem. this way, we can use data from a set of subjects (source domain) to train a model to another one (target domain). riemannian alignment (ra) and euclidean alignments (ea) are the most used transfer learning methods [7]. however, those models only use a single band in the temporal filtering step, losing some temporal patterns. in this case, the alignment transforms the signals of the source and the target subjects to the same space. we propose different ways to use multi-band models with ea, and the evaluated temporal filtering was filter bank (fb), empirical mode decomposition (emd), and continuous wavelet transform (cwt). using the physionetmi dataset, a well-known dataset from literature, the results obtained pointed out that multi-band with ea improves the results of bci classifiers in this scenario. this is an advantage for the bci models dedicated to post-stroke motor rehabilitation, as it reduces the calibration time and makes the procedure straightforward to be used with any patient. 2 related work the variations in eeg signals between individuals are remarkable, reflecting the distinct nuances of their brain activities [12]. this condition is intensified in individuals who have suffered some form of stroke. as a result, these differences significantly impact the training phase in bci, requiring a personalized approach. therefore, in most cases, it is crucial to train a distinct model for each person. to deal with this situation, transfer learning methods were proposed to use many subjects in a single training step. one of the most used methods is the riemannian alignment (ra), which is a method to align the covariance matrix of the signals from each subject during their resting state. however, riemannian space makes the model more computationally expensive and limited since the classifiers must perform efficiently in that space. to outperform that problem, the euclidean alignment (ea) was proposed [7]. ea aligns the eeg trials for each subject in euclidean space using the mean of the covariance matrix of the subject’s trials as a reference matrix. thereafter, the transformed reference matrix regarding each subject becomes the identity matrix. this technique reduces the difference between data of distinct subjects, similarly than ra, but without transferring the data to riemannian space. therefore, it is possible to use non-riemannian classifiers, such as linear discriminant analysis (lda) and support vector machine (svm) [14, 15]. moreover, the computational cost is reduced by operating ea instead of ra. when comparing models, ea points out better results than ra when transfer learning is used for bci applications [7]. after its release, different approaches were proposed to improve ea. for instance, the unification of ea and ra alignment methods using a hybrid (euclidean and riemannian) space [12]. in addition, a selection of the subjects with data similar to that of the target one can be considered. [12]. another approach to improve ea is weighting the subject importance based on the distance between the source and target data [5]. comparisons of transfer learning methods applied to deep learning classifiers pointed out that ra and ea present better results than normalization techniques [13, 21, 22]. those articles do not use ea with multi-band techniques. in this work, we propose novel pipelines for ea using multi-bands and parallel application of ea after signal decomposition. 3 brain-computer interfaces for post-stroke motor rehabilitation stroke is a medical emergency that leaves sequels, including cognitive and motor deficits and oropharynxes dysphagia [10]. the rehabilitation of post-stroke patients has great importance to the recovery of their cognitive and motor functions. also, bci for post-stroke rehabilitation is an alternative to induce neural plasticity. bcis support this process by delivering targeted stimuli that create a closed loop for neural reorganization. engaging directly with external devices activates brain regions linked to movement, strengthening pertinent neural pathways and fostering positive plasticity. reinforcement learning promotes bci for post-stroke motor rehabilitation, since it involves adapting behavior based on the consequences of actions. within bcis, individuals can receive instant and personalized feedback corresponding to their rehabilitation endeavors. this feedback creates a positive reinforcement, fortifying the neural connections linked to targeted movement patterns. furthermore, bcis aim to close the motor planning loop, aiding in the reconnection of brain circuits responsible for motor control. by converting brain signals into actionable commands for external devices, bcis support the rehabilitation of motor planning, promoting iterative practice and the gradual enhancement of motor skills. furthermore, studies on upper-limb rehabilitation using bci indicate that the accuracy of the bci system can improve motor function. other studies also showed a relationship between the bci accuracy and clinical gains [2, 4, 16]. this relationship is based on the hypothesis that higher bci accuracy may improve confidence and motivation in patients, which may better promote plasticity [16]. moreover, another reason for the success of bci rehabilitation can be the correlation between engagement and higher levels of patient attention. figure 1 represents the bci cycle for post-stroke motor rehabilitation. first, brain signals are collected through sensors, such as electroencephalogram (eeg). the collected signals are filtered to remove interference and increase the signal-to-noise ratio. techniques such as temporal and spatial filtering can be applied to extract relevant information. then, feature extraction and selection steps are performed, and the chosen features are used to classify the signal. with the signals already classified, the system executes commands based on the label. for post-stroke motor rehabilitation, the feedback is usually presented as an image on a screen and an electrical stimulus in the muscle. in the context of bci systems for patient rehabilitation, it is relevant to mention that a smaller number of electrodes brings benefits in terms of ease of treatment and patient comfort [20]. furthermore, data alignment is emphasized for synchronizing stimuli and their responses and ensuring more accurate temporal analysis. data alignment also proves to be effective in enhancing system efficiency and reducing interference and signal noise. by utilizing data alignment, we can reduce the number of trials needed to train the model, resulting in faster and less exhausting treatment. fig. 1. bci cycle for post-stroke motor rehabilitation. full size image 4 methods this section presents the methods from the literature used as a baseline to develop the proposed bci pipelines. we describe ea, the temporal filter methods used in the experiments, and the common spatial pattern. 4.1 euclidean alignment euclidean alignment (ea) was proposed as an alternative to riemannian alignment (ra) [7]. the advantages of ea are the reduction of the alignment time when compared to ra, and the maintenance of the data in the euclidean space. ea uses a reference matrix, calculated through the eeg trials, to align the data from different subjects. a reference matrix \(\bar{r}_s\) is the mean of the trial’s covariance for the s-th subject, which is defined by $$\begin{aligned} \bar{r}_{s} = \frac{1}{n_s}\sum ^{n_s}_{n=1}x^{n}_{s}(x^{n}_{s})^{\text {t}} \end{aligned}$$ (1) where \(n_s\) is the number of trials of subject s, \(x_s^n\) is the n-th trial of subject s. the matrix \(\bar{r}_{s}\) reduces the difference between the subjects as $$\begin{aligned} e^{n}_{s}=\bar{r}^{-\frac{1}{2}}_{s} \times x^{n}_{s} \end{aligned}$$ (2) where \(e^n_s\) is the signal of subject s after alignment. the new reference matrix for all the transformed subjects is equal to the identity matrix. therefore, the new space is the same for all the subjects. 4.2 temporal filtering temporal filtering is a technique used in data processing to extract or manipulate temporal information in data sets. its purpose is to enhance signal quality by improving the signal-to-noise ratio. temporal filtering removes or attenuates unwanted temporal components in the data, allowing the identification of more significant patterns or the reduction of noise. the choice of a temporal filtering technique relies on the particular context of the problem and the goals of the analysis. this section presents four temporal filters: bandpass, filter bank (fb), empirical mode decomposition (emd), and continuous wavelet (cwt). bandpass and filter bank: bandpass filters are widely employed in various signal-processing applications, including wireless communication systems, audio and video processing, and eeg. then, they are useful for isolating signals of specific frequencies and eliminating unwanted noise [1]. a bandpass filter allows for a specific range of frequencies to pass through the filter while attenuating frequencies outside that range. a filter bank is a group of bandpass filters that divide an input signal into sub-bands. figure 2 shows a diagram of the analysis filter bank. fig. 2. filter bank. full size image empirical mode decomposition: empirical mode decomposition [9] is a method that decomposes non-stationary and non-linear data in different components to extract information that is not explicit in the original signal. the core of this method lies in the empirical identification of the oscillatory modes inherent in the data, distinguished by their characteristic time scales. after this identification, the data is decomposed accordingly. the method uses the sifting process [9] to decompose the data into intrinsic mode functions (imfs) and a residual. figure 3 presents a schematic representation of an emd algorithm, with 3 imfs and a residual as outputs. the sifting process of an x signal consists of the following steps: the local extrema of the signal are identified, and all the local maxima and minima are connected by a cubic spline line as the upper envelope. their mean is denominated \(m_{i}\), and the difference between the data and \(m_{i}\) is the first component \(h_{i}\). if the component \(h_{i}\) is an imf, this is separated from the remaining data and is designated as \(c_{i}\). the result is a component \(r_{n}\). otherwise, step (1) is repeated with \(h_{i}\) as the input signal. if \(r_{n}\) is a residual, the original dataset x is decomposed into n imfs \(c_{i}\) and a residual \(r_{n}\), which can be either the mean trend or a constant. otherwise, steps (1) and (2) are repeated. the edm procedure can be represented as the following equation $$\begin{aligned} x = \sum _{i=1}^{n}c_{i} + r_{n} \end{aligned}$$ (3) fig. 3. empirical mode decomposition with 3 levels. full size image continuous wavelet: it is an application of the wavelet transform that employs arbitrary scales and nearly arbitrary wavelets [19]. wavelets are brief oscillations localized in time and frequency. in this transformation, the signal is decomposed into a collection of wavelets through the wavelet method, corresponding to a distinct frequency band. the continuous wavelet transform (cwt) generates a time-frequency representation of a signal with best location found by the model in both the time and frequency dimensions. the cwt generates a graphical representation called a spectrogram that shows the intensity of different frequencies along time. each point in the spectrogram reflects the intensity of the associated frequency component at a specific time. cwt is especially useful for analyzing dynamic signals, where frequency characteristics can vary over time, and the choice of wavelet and scale parameters is critical in cwt, as it determines the sensitivity of the transform to various signal characteristics. cwt function of an input signal x(t) can be represented as $$\begin{aligned} w(a,b) = \frac{1}{\sqrt{a}}\int _{-\infty }^{\infty }x(t)\psi ^{*}(\frac{t b}{a})dt \end{aligned}$$ (4) where a controls the length of the wavelet, b controls the position of the wavelet along the time axis, and \(\psi ^{*}(\frac{t b}{a})\) is the complex conjugate version of the wavelet \(\psi (\frac{t b}{a})\). 4.3 common spatial pattern common spatial pattern (csp) is a technique for spatial filtering to extract features from signals. csp aims to find a set of spatial filters that maximize the difference between signals from two classes based on their covariance matrices. its application is given by $$\begin{aligned} z_{i}=w^{\text {t}}\times e_{i} \end{aligned}$$ (5) where \(z_{i}\in \mathbb {r}^{s\times t_{s}}\) is the matrix obtained when we multiply the i-th trial \(e_{i}\in \mathbb {r} ^{s\times t_{s}}\) with the transformation matrix \(w^{t}\in \mathbb { r}^{s\times s}\). s and t represent the number of electrodes and the size of the signal collected per electrode in each trial. the correlation matrices of each label are used to find the transformation matrix w, and it is defined by $$\begin{aligned} {\varsigma }^{(c)} = \frac{1}{n_c}\sum ^{n_c}_{n}e^{(c)}_{n}{e^{(c)}_{n}}^t \end{aligned}$$ (6) where \({\varsigma }^{(c)}\in \mathbb {r}^{s\times s}\) is the correlation matrix between the electrodes, \(n_c\) is the number of training samples of class c, and \(e^{(c)}_{n}\) is the n-th training sample of class c. maximization is restricted so that the sum of the diagonal matrices formed by the eigenvalues is equal to the identity matrix, that is, \({\varlambda }^{(1)} + {\varlambda }^{(2)} = i\) this problem is equivalent to solving the generalized eigenvalue problem, defined as $$\begin{aligned} {{\varsigma }^{(1)}}w = ({{\varsigma }^{(1)}}+{{\varsigma }^{(2)}})w\varlambda \end{aligned}$$ (7) finally, the following equation is applied to z to extract its features: $$\begin{aligned} f_{i}=log\bigg (\frac{diag(z_{i}z_{i}^t)}{tr(z_{i}z_{i}^t)}\bigg ) \end{aligned}$$ (8) where \(f_{i}\) represents the feature vector of \(z_i\), and \(diag(\cdot )\) and \(tr(\cdot )\) represent the main diagonal and trace of the matrix, respectively. the number of csp filters is equal to the number of electrodes. however, only the first and last filters are used. as csp maximizes the variance of class 1 in the first eigenvectors and class 2 in the last, a feature vector \(v_i\) is created where its elements are the m/2 first components and m/2 final components of \(f_i\). 4.4 proposed multi band pipelines with euclidean alignment a temporal filter improves the signal-to-noise ratio and, when applied parallel to the data, can create a new set of filtered data. in this work, we propose these filters with ea to evaluate their combinations. csp is adopted as a base model, as it is a standard classification approach for mi [7]. we propose here two ways to use ea with multi-band filtering: using the ea before and after the multi-band filter. also, one can observe that many multi-band filters can be used. this work focuses on the temporal filter models presented in sect. 4.2. the pipelines described in fig. 4 are in order: the baseline model with ea [7], the use of a temporal filter before the ea step, and the application of ea before the temporal filter. as shown in fig. 4, we used lda as the classifier [14, 15]. fig. 4. literature and proposed architectures using euclidean alignment. full size image 5 computational experiments we performed the experiments to evaluate the proposals in sect. 4. a 5-fold stratified cross-validation was performed to evaluate the proposed approaches. the non-parametric kruskall-wallis statistical test was used as a multi-group test followed by dunn’s test as post-hoc with the bonferroni correction. the codes implemented for this work are publicly availablefootnote 1. 5.1 datasets the physionetmi dataset [6] was used to perform the experiments. that dataset has the recorded data of 109 subjects with 14 sessions per subject. the acquisition was performed using an eeg with a sample rate of 160 hz and 64 monopolar electrodes based on the 10–10 electrode system [17]. for the first two sessions, a baseline with eyes open and closed was recorded without any task to execute. for other sessions, there are two classes per session, which is sessions: (i) left-fist and right-fist movement: 3, 7, 11; (ii) left-fist and right-fist motor imagery: 4, 8, 12; (iii) both-fists and both-feet movement: 5, 9, 13; and (iv) both-fists and both-feet motor imagery: 6, 10, 14. each session with a task has 15 trials, and each trial has 8 s. the cue for the task is presented for 1 s, followed by 3 s of the task execution. moreover, there is a 4-second break between each trial. the steps of each trial are shown in fig. 5. due to the binary classification nature of the models presented here, we only used left-fist and right-fist motor imagery in our experiments. fig. 5. timing scheme of a trial from the physionetmi dataset. full size image 5.2 parameters used the computational experiments were performed with two subsets of electrode positions: (i) 3 electrodes: c3, cz, c4; and (ii) 8 electrodes: fc3, c3, cp3, fcz, cpz, fc4, c4, cp4. we reduced the number of electrodes in those 2 cases, as it is proven that a smaller number of electrodes makes the treatment more comfortable for the patient, faster to prepare the treatment, and less expensive [11, 20]. in the future, we will be able to use this electrode parameter to acquire data from more individuals, without losing model accuracy. the positions of the electrodes above the brain motor area are: (i) above the primary motor area when using 3 electrodes, and (ii) the supplementary motor and premotor areas in addition to those in (i) when using 8 electrodes [8]. furthermore, we created two new classes: left-fist and right-fist, by combining the movement and motor imagery of each one. only those 2 new classes were used as it is the most common approach for post-stroke rehabilitation bcis [20]. all acquired data was down-sampled to 128 hz using a cubic spline. when analyzing the dataset, the eeg trials of subjects 88, 94, and 101 were unavailable, resulting in these subjects being removed from our computational experiments. considering this, the experiments were performed using data from the remaining 106 subjects. the data window for training started 0.5 s after the cue and lasted for 2 s. the same window bounds were used for the testing phase. we created 11 models to evaluate our proposal. the first models serve as baselines, namely: csp, ea-csp, and \((\cdot )\)-csp, where \((\cdot ) \in \) {fb, emd, cwt}. after that, the models with ea applied after the multi-band have the form \((\cdot )\)-ea-csp. finally, we have the models where ea is applied before the multi-band in the form ea-\((\cdot )\)-csp. we performed preliminary experiments, and after that, we set the hyperparameters of the models as: the bandpass was implemented using the fast fourier transform (fft), between 4 and 40 hz. for csp, we used \(m=4\). the filter bank used nine bandpass filters: (4, 8), (8, 12), (12, 16), (16, 20), (20, 24), (24, 28), (28, 32), (32, 36), (36, 40). emd used the smallest amount of imf less 1, with nine being the maximum limit of this value, and cwt was implemented with five levels. for each target subject, the source data for ea had all other subjects except the target, and the solver for lda was svd. 5.3 results using 3 electrodes the experiments were performed with the settings presented in session 5.2. table 1 presents the results for the three-electrode case, where the accuracy for ea-csp and mb-csp was worse than ea-mb-csp and mb-ea-csp for all mb methods. filter bank was the best temporal filter method regardless of the use of ea. ea-fb-csp outperformed the remaining approaches. the statistical analysis presented a significant difference p-value \(<0.001\) for all methods when compared to ea-fb-csp. furthermore, fb-ea-csp was the second-best model, with a statistical difference concerning the remaining models. these results pointed out that filter bank is the best temporal filtering evaluated here to be used with ea, regardless the order of its application in the bci pipeline. it is also possible to note that ea before multi-band reached better results in all tested cases. table 1. accuracy for the scenario with 3 electrodes.full size table 5.4 results using 8 electrodes for the 8 electrodes case, and their results are presented in table 2. the results are similar to those observed in the 3 electrodes case. ea-fb-csp outperformed the remaining approaches. the use of ea before multi-band performed better than that applying ea after it, and filter bank is the temporal filter model with the best result. such as in the 3-electrode case, the p-value for all methods compared to ea-fb-csp was lower than 0.001. when comparing both electrode settings, 3 electrodes reached better results than 8 electrodes for all cases. that difference between 3 and 8 electrodes was statistically significant for all models. in the same way as the 3-electrode case, the ea-mb-csp pipeline was more robust than the other two pipeline models. the results pointed out that ea as the first step of the bci pipeline makes the model more robust between subjects. table 2. accuracy for the scenario with 8 electrodes.full size table 5.5 performance profile analysis fig. 6. performance profiles of the results. each problem is a combination of subject and number of electrodes. full size image the performance profiles (pps) were used to conduct an overall analysis concerning the accuracy value [20], where each pair (subject, electrode case) is a problem. pps present how many problems were solved with less than \(\tau \) times the best value for each problem. figure 6 contains pps regarding the results of the 11 methods evaluated in this work. the following conclusions can be highlighted: (i) ea-fb-csp was the model with the best result for most subjects, since it had the largest value for \(\tau = 1\) (about 90% of the 212 problems). (ii) ea-fb-csp obtained the best overall performance in the average case, as this approach achieved the biggest area under the pp curve; and (iii) ea-fb-csp is the most robust method, with at more than 10% worse than the best case. other results in pps are: (i) fb-ea-csp was the second method for the average result but the 4-th most robust method; (ii) the three most robust cases used ea before the multi-band; and (iii) ea-csp was better and more robust than all cases which did not use ea. 6 discussion all models in the 3-electrode case obtained better results than their counterparts in the 8-electrode case. one can observe that the amount of electrode raises the dimensionality for lda with no relevant information, leading to worse results. lda performs better when the number of trials per area in the search space is high enough to fill it. when the dimension increases, the search space gets bigger and less dense. that explains that models in the 3-electrode case have better results than the 8-electrode case. the standard bci pipeline is formed by the following steps: temporal filtering, spatial filtering, feature extraction, feature selection, and classifier. ea can be represented as a spatial filter due to its linear transformation on the input signal. therefore, better results are expected when ea is used after the temporal filtering, such as in the bci pipeline. however, the results found in the experiments demonstrate the opposite, leading to questions regarding the bci stage’s order. spatial filters before the temporal step can result in new configurations for the bci pipeline. filter bank was the best temporal filter tested here. this result is expected, as it takes the signal in frequency sub-bands, being able to select separately \(\mu \) (8–13) and \(\beta \) (13–30) rhythms, which is crucial for motor imagery activities [23]. the other temporal filtering techniques applied in this work do not use the signal spectrum as effectively, which explains their results in comparison with filter bank. 7 conclusion bci-based mi models can be used for post-stroke rehabilitation. there are some difficulties despite that, such as long calibration periods during the training phase and low resolution of the signal obtained. we proposed euclidean alignment (ea) with multi-band temporal filters to reduce the impact of these two conditions. the models were created using filter bank, empirical mode decomposition, and continuous wavelet as temporal filter techniques. each model uses one of these temporal filters, either before or after ea. we evaluated the models in 2 cases: with 3 and 8 electrodes in the physionetmi dataset. the models used in this work demonstrate how ea and multi-band temporal filters can be used together to acquire more robust models. they also demonstrate that changing the steps’ order can improve the quality of the solutions. the best model in the computational experiments performed here is ea-fb-csp-lda, with an improvement of approximately 44% for three electrodes and 43% for eight electrodes when compared to ea-csp-lda, reaching an accuracy of 80% and 73.33%, respectively. ea before the temporal filtering presented results better than those obtained by models with ea after the temporal filtering. new experiments using feature selection or subject selection can be performed to improve those results. our results also pointed out that spatial filtering before temporal filtering has prominent potential for new bci pipelines. notes 1.https://github.com/ghdesouza/bci. references ang, k.k., chin, z.y., wang, c., guan, c., zhang, h.: filter bank common spatial pattern algorithm on bci competition iv datasets 2a and 2b. front. neurosci. (2012). https://doi.org/10.3389/fnins.2012.00039 ang, k.k., guan, c.: eeg-based strategies to detect motor imagery for control and rehabilitation. ieee trans. neural syst. rehabil. eng. (2017). https://doi.org/10.1109/tnsre.2016.2646763 bashashati, a., fatourechi, m., ward, r.k., birch, g.e.: a survey of signal processing algorithms in brain-computer interfaces based on electrical brain signals. j. neural eng. (2007). https://doi.org/10.1088/1741-2560/4/2/r03 biasiucci, a., et al.: brain-actuated functional electrical stimulation elicits lasting arm motor recovery after stroke. nat. commun. (2018). https://doi.org/10.1038/s41467-018-04673-z gao, y., li, m., peng, y., fang, f., zhang, y.: double stage transfer learning for brain-computer interfaces. ieee trans. neural syst. rehabil. eng. (2023). https://doi.org/10.1109/tnsre.2023.3241301 goldberger, a.l., et al.: physiobank, physiotoolkit, and physionet: components of a new research resource for complex physiologic signals. circulation (2000). https://doi.org/10.1161/01.cir.101.23.e215 he, h., wu, d.: transfer learning for brain-computer interfaces: a euclidean space data alignment approach. ieee trans. biomed. eng. (2020). https://doi.org/10.1109/tbme.2019.2913914 homan, r.w., herman, j., purdy, p.: cerebral location of international 10–20 system electrode placement. electroencephalogr. clin. neurophysiol. (1987). https://doi.org/10.1016/0013-4694(87)90206-9 huang, n.e., et al.: the empirical mode decomposition and the hilbert spectrum for nonlinear and non-stationary time series analysis. proc. math. phys. eng. sci. (1998). https://doi.org/10.1098/rspa.1998.0193 kelly-hayes, m., beiser, a., kase, c.s., d’agostino, r.b., scaramucci, a., wolf, p.a.: the influence of gender and age on disability following ischemic stroke: the framingham study. j. stroke cerebrovasc. dis. (2003). https://doi.org/10.1016/s1052-3057(03)00042-9 leeb, r., lee, f., keinrath, c., scherer, r., pfurtscheller, g., bischof, h.: brain-computer communication: motivation, aim, and impact of exploring a virtual apartment. ieee trans. neural syst. rehabil. eng. (2007). https://doi.org/10.1109/tnsre.2007.906956 li, y., wei, q., chen, y., zhou, x.: transfer learning based on hybrid riemannian and euclidean space data alignment and subject selection in brain-computer interfaces. ieee access (2021). https://doi.org/10.1109/access.2020.3048683 liang, y., ma, y.: calibrating eeg features in motor imagery classification tasks with a small amount of current data using multisource fusion transfer learning. biomed. signal process. control (2020). https://doi.org/10.1016/j.bspc.2020.102101 lotte, f., et al.: a review of classification algorithms for eeg-based brain-computer interfaces: a 10 year update. j. neural eng. (2018). https://doi.org/10.1088/1741-2552/aab2f2 lotte, f., congedo, m., lécuyer, a., lamarche, f., arnaldi, b.: a review of classification algorithms for eeg-based brain-computer interfaces. j. neural eng. (2007). https://doi.org/10.1088/1741-2560/4/2/r01 mane, r., chouhan, t., guan, c.: bci for stroke rehabilitation: motor and beyond. j. neural eng. (2020). https://doi.org/10.1088/1741-2552/aba162 nuwer, m.r.: 10–10 electrode system for eeg recording. clin. neurophysiol. (2018). https://doi.org/10.1016/j.clinph.2018.01.065 pacheco-barrios, k., et al.: burden of stroke and population-attributable fractions of risk factors in latin america and the caribbean. j. am. heart assoc. (2022) google scholar  samar, v.j., bopardikar, a., rao, r., swartz, k.: wavelet analysis of neuroelectric waveforms: a conceptual tutorial. brain lang. (1999). https://doi.org/10.1006/brln.1998.2024 de souza, g.h., bernardino, h.s., vieira, a.b., barbosa, h.j.c.: differential evolution based spatial filter optimization for brain-computer interface. in: proceedings of the genetic and evolutionary computation conference (2019). https://doi.org/10.1145/3321707.3321791 wang, x., yang, r., huang, m.: an unsupervised deep-transfer-learning-based motor imagery eeg classification scheme for brain-computer interface. sensors (2022). https://doi.org/10.3390/s22062241 article  math  google scholar  xu, l., xu, m., ma, z., wang, k., jung, t.p., ming, d.: enhancing transfer performance across datasets for brain-computer interfaces using a combination of alignment strategies and adaptive batch normalization. j. neural eng. (2021). https://doi.org/10.1088/1741-2552/ac1ed2 yu, h., ba, s., guo, y., guo, l., xu, g.: effects of motor imagery tasks on brain functional networks based on eeg mu/beta rhythm. brain sci. (2022). https://doi.org/10.3390/brainsci12020194 download references acknowledgements the authors thank the support provided by capes, cnpq, fapemig, ufjf, and openbci. author information authors and affiliations federal university of juiz de fora (ufjf), juiz de fora, mg, brazil marcelo m. amorim, leonardo prata, joão stephan maurício, alex borges, heder bernardino & gabriel de souza authors marcelo m. amorimview author publications search author on:pubmed google scholar leonardo prataview author publications search author on:pubmed google scholar joão stephan maurícioview author publications search author on:pubmed google scholar alex borgesview author publications search author on:pubmed google scholar heder bernardinoview author publications search author on:pubmed google scholar gabriel de souzaview author publications search author on:pubmed google scholar corresponding author correspondence to marcelo m. amorim . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper amorim, m.m., prata, l., maurício, j.s., borges, a., bernardino, h., de souza, g. (2025). euclidean alignment for transfer learning in multi-band common spatial pattern. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_30 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords electroencephalogram motor imagery brain-machine interface medical applications publish with us policies and ethics profiles marcelo m. amorim view author profile alex borges view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature classifying graphs of elementary mathematical functions using convolutional neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper classifying graphs of elementary mathematical functions using convolutional neural networks conference paper first online: 30 january 2025 pp 270–280 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) classifying graphs of elementary mathematical functions using convolutional neural networks download book pdf download book epub joaquim viana  orcid: orcid.org/0009-0008-0755-64119, helder matos  orcid: orcid.org/0000-0002-5632-79489, marcelle mota  orcid: orcid.org/0000-0001-9226-90209 & … reginaldo santos  orcid: orcid.org/0000-0002-0456-85479  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 470 accesses abstract the classification of images of elementary mathematical function graphs presents a significant challenge in computer vision; this is due to the varied shapes and formats of each functions curves. this classification is crucial for identifying function graphs, which have important applications in text and mathematical symbol recognition technologies, aiding visually impaired individuals by providing access to printed content. in educational environments, this identification helps obtain the analytical expression of drawn graphs, facilitating the extraction of information from educational materials. this article investigates various convolutional neural network (cnn) architectures to identify the most suitable model for classifying images of elementary mathematical function graphs. we compare our model with other renowned architectures, such as resnet, mobilenet, and efficientnet, using a custom dataset of function graphs. our experiments show that the proposed architecture significantly outperforms networks of general purpose, achieving an accuracy of \(98.51\%\) in classifying elementary mathematical function graphs. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others efficient design of pruned convolutional neural networks on fpga article 14 november 2020 pushing data into cp models using graphical model learning and solving chapter © 2020 fast evolution of cnn architecture for image classification chapter © 2020 1 introduction classifying images of elementary mathematical function graphs is challenging in computer vision because of the varied shapes and formats of these curves [7]. identifying the type of graph of a function is a preliminary step toward identifying the analytic expression of the graph in question. determining the analytic expression of graphs of elementary mathematical functions can be used in technologies for recognizing text and mathematical symbols, assisting visually impaired individuals by enabling them to access information contained in printed material [10]. in educational environments, such identification can aid in extracting information from educational materials, helping research, and the study of mathematical concepts. in this article, we propose a solution for classifying images of graphs of elementary mathematical functions using convolutional neural networks (cnns). our goal is to find a cnn architecture that best suits this problem. this article is structured as follows: sect. 2 discusses related work regarding the use of cnns for image classification; sect. 3 describes the proposed methodology; sect. 4 presents results and discussions; and sect. 5 lists the study’s conclusions. 2 related work in the literature, there are few works demonstrating techniques for extracting elements within images of graphs of elementary mathematical functions. in [9], a method is proposed that divides the graph image into regions: regions outside the axes (such as titles, axis labels, and scales) and contour regions of the line. scale information is obtained from the regions outside the axes, and the coordinate values of each line graph are also extracted from these regions. finally, this information is integrated to convert graph data into numerical information. however, the method in [9] was used only for graphs with a single configuration regarding the positioning of graph elements within the image. in [21], an offline image segmentation approach based on the technique from [9] is proposed. images are segmented into connected components and characters. the characters represent the analytical expressions typically accompanying the image, while the connected components encompass everything from the axes to the function curves. subsequently, elements of connected components are grouped to identify the function curve. thus, the author managed to separate 15 of a universe of 20 images and group 104 of a total of 112 components within the images. the study highlights the need to improve the accuracy of their results. another study, aimed at developing an automatic system for transforming function graphs into tactile graphics that can be used by visually impaired individuals, was conducted by [5]. the authors present a method for extracting graphical elements in mathematical function graphs. the method involves identifying all components of the images (curves, lines, axes, equations, dashed lines, points, etc.) and dividing them into primitive elements (x and y axes, main lines, and curves). after identifying the primitive elements, these elements are combined to form graphical elements. in an experiment with 33 scanned images from science textbooks, the authors were able to extract 502 primitive elements. of these, 373 were successfully combined into 115 graphical elements, while another 129 were not. another approach suggested by [18] uses the mathgraphreader, a tool that extracts data from graphs and creates alternative textual descriptions. using text processing, image processing, plotting, and mathematical concepts, the authors built a tool that allows visually impaired students to access graph information interactively. the technique involves first detecting the origin of the axes and then tracing the intersection points of the axes with the function curve to find inflection points. however, the study does not address cases where the dashed line of the graph curve is displayed without intersecting the axes. the methods mentioned so far propose offline extraction techniques, where the extraction of elements is performed on previously provided images. a different method was suggested by [19], which proposes obtaining the analytic expression from a function constructed online, i.e., when the graph curve is drawn in real-time. this method captures the pixels of the curve on the screen and then uses polynomial regression to find a curve that best fits the pixel points. although the method yields good results, it has not been evaluated on a large dataset to verify accuracy. additionally, it is limited to polynomial expressions and is restricted to the range \( x \in [-1, 1] \). cnns have revolutionized the field of computer vision and have been widely used in a broad range of applications [1, 3], such as image recognition, object detection, image segmentation, among others [2, 15, 16]. however, the use of cnns for classifying images of graphs of elementary mathematical functions has been limited due to the scarcity of datasets [4]. the proposed methodology encompasses not only the creation of a dataset that contains different types of functions and their variations but also conducts an investigation into cnn architectures applied to the recognition of mathematical function patterns in graphs. additionally, it compares these results with a set of renowned architectures applied to more generalizable computer vision tasks. 3 methodology this section describes the generation of a dataset of images of elementary functions graphs, the rationale behind the chosen deep learning model architecture, and how the model’s performance was assessed and benchmarked against other existing methods. 3.1 process of data generation the lack of open public datasets addressing the problem of classifying function graphs motivated the development of a systematic dataset generation process. figure 1 illustrates the process of creating a dataset containing images of various elementary functions to be modeled. fig. 1. image processing in function graphs dataset. full size image fig. 2. graphs of elementary mathematical functions. full size image to create the dataset, we first listed various types of elementary functions, as depicted in fig. 2: linear (2a), quadratic (2b), cubic (2c), exponential (2d), logarithmic (2e), square root and cubic root (2f), sine (2g), cosine (2h), tangent (2i), and cotangent (2j). we randomly generated analytic expressions that could plot visible graphs within the range \(y \in [-5, 5]\) and for the domain x the interval was set to \(x \in [-2\pi , 2\pi ]\) for trigonometric functions and \(x \in [-5, 5]\) for other functions. subsequently, a script was generated to automate the construction of these functions using winplot [20], a free software for plotting two-dimensional and three-dimensional graphs of mathematical functions, including explicit, implicit, parametric, and polar function graphs. some functions, such as exponential and logarithmic functions, do not offer many variations within the specified range and therefore could generate a limited number of images. to avoid class imbalance, we limited the number of generated images to 1,450 per class. in total, 14,500 images were generated with a resolution of 128\(\,\times \,\)128 pixels and saved with names like \(\texttt {<class>\_<expression>.png}\). as for generated images, we divided them into training (80%, 11,600 images), validation (10%, 1,450 images), and test sets (10%, 1,450 images). each group contains all classes with a balanced number of samples. the images were flattened into a one-dimensional vector and labeled using a one-hot encoding format. finally, the dataset was shuffled and saved in serialized pickle files. 3.2 selection and definition of the cnn architecture due to the scarcity of studies focusing on constructing deep neural network architectures for classifying images in elementary function graphs data, we will compare the performance of the proposed architecture with architectures that have excelled on broader datasets. three cnn architectures were selected for comparison, known for their strong performance in image classification tasks on datasets such as imagenet [8] and cifar-10 [14]: resnet [11]: designed to ensure that learning is not hindered by adding more layers, using shortcuts or residual connections to facilitate optimization. variants include resnet-50 (50 layers), resnet-101 (101 layers), resnet-152 (152 layers). resnet-50 was chosen for comparison as it achieved 93% to 94% accuracy on cifar-10 dataset. mobilenet [12]: designed for mobile and low-power devices, known for its lightweight and low computational cost. variants include v1 (depthwise separable convolutions), v2 (inverted residual blocks and linear bottlenecks), and v3 (automated neural architecture search and additional optimizations). mobilenetv3 was chosen for comparison as it achieved 91% to 92% accuracy on cifar-10 dataset. efficientnet [22]: designed to optimize the balance between accuracy and computational efficiency, using a compound scaling method that simultaneously scales depth, width, and resolution of the network. variants include efficientnet-b0 (more efficient) to b7 (more layers, filters, and resolution). efficientnet-b0 was chosen for comparison as it achieved approximately 95% accuracy on cifar-10 dataset. the cnn architecture proposed for classifying elementary function graph images was named f-graphs. the hyperband algorithm [17] was used to perform an optimized search of neural network hyperparameters that enhance its performance in image classification tasks. from the initializations made, the hyperband algorithm returned the cnn architecture illustrated in fig. 3. the network consists of three convolutional layers activated with the rectified linear unit (relu) function, combined with max pooling layers after the convolutional layers. in the fully connected layer, only one hidden layer with 128 neurons activated with the sigmoid function was included. at the output, a dense layer with 10 neurons corresponding to the 10 classes of the dataset and activated with the softmax function produces the probability distribution for the output classes. fig. 3. f-graphs’ network architecture. full size image 3.3 model training and evaluation f-ggraphs was created using the keras library for python [6], which simplifies the construction and training of neural networks, often used to create and experiment with deep learning models. figure 4 shows the loss behavior and accuracy evolution during the training of the proposed model. training included five different initializations for each architecture and 40 epochs using early stopping based on monitoring loss on the validation data as the stopping criteria. fig. 4. loss and accuracy values for training and validation of the proposed model. full size image to optimize training, the adaptive moment estimation (adam) optimizer [13] and a learning rate of \(9 \times 10^{-5}\) were used. loss estimation during training for the 10 classes was performed using categorical cross-entropy. the metric used to evaluate performance during training was accuracy. the results will be shown through comparisons between the accuracy, precision, recall, and f1-score metrics averaged over the 5 runs of the model fit on the data, as well as the memory size used to load the model and the total number of parameters for each architecture. visualizations have also been built to show the model’s predictions between classes. 4 results the average behavior results for accuracy, precision, recall, f1-score, size, and number of parameters for the five runs of the compared models are presented in table 1. the proposed architecture achieved its best performance with the configuration described in subsect. 3.2, reaching \(98.51\%\) accuracy on the test data. in contrast, resnet-50, mobilenet-v3, and efficientnet-b0 returned low accuracies of \(10.57\%\), \(13.02\%\), and \(11.09\%\), respectively. table 1. mean and standard deviation results for training on the function graphs dataset.full size table the proposed model f-graphs also achieved better results in precision, recall, and f1-score, with values of \(98.43\%\), \(98.41\%\), and \(98.41\%\), respectively. however, f-graphs is surpassed in terms of memory size and number of parameters by mobilenet-v3, which has the smallest memory footprint at 38.63mb and the lowest number of parameters at 10,126,560. as expected, the latter presents a lower space complexity, due to the fact that it is specifically designed for mobile devices, which are constrained by limited computational resources. resnet-50 is a deep cnn with multiple convolutional layers designed to extract more features from images [11]. the number of convolutional layers influences the models learning process, leading to inferior performance in classifying images of function graphs. in terms of memory size, it occupies approximately 138mb, making it heavier due to the large number of network parameters. similarly, the number of convolutional layers heavily influenced the generalization power of mobilenet-v3 and efficientnet-b0, although these architectures outperformed resnet-50. to improve the performance of resnet-50, mobilenet-v3, and efficientnet-b0 on the dataset of elementary function images, parameter tuning would be necessary. although the proposed model f-graphs achieved an accuracy of 98.51%, it struggled with misclassifying some functions, as illustrated in fig. 5. for example, the misclassification of the cosine function with sine occurs quite often, since these functions are very similar, especially when translated horizontally. another important fact is that the f-graphs model tends to generalize linear functions more frequently. functions such as the exponential function (eq. 1), shown in fig. 6a, and the cubic function (eq. 2), shown in fig. 6b, were often misclassified as linear. this is because the coefficients of the functions were generated randomly, forming analytical expressions that generate functions of this type. to address this issue, we replaced some of these functions that tended to be confused with other less ambiguous functions. $$\begin{aligned} f(x)=1.25^x-1 \end{aligned}$$ (1) $$\begin{aligned} g(x)=x^3+2x^2+5x+0 \end{aligned}$$ (2) fig. 5. misclassifications among classes. full size image fig. 6. exponential and cubic functions that resemble linear ones. full size image 5 conclusion in this work, we aimed to find an optimal architecture for the classification of elementary mathematical function graphs. cnns are excellent for solving image classification tasks. however, different datasets with different characteristics may require a specific architecture for the problem being solved. images of graphs of elementary mathematical functions tend to have few features when compared to images of other objects in the real world. in an image of a mathematical function graph, the convolution process involves extracting features to identify the vertical line (y-axis), horizontal line (x-axis), and the curve of the function. with fewer features, the model does not require many convolutional layers for feature extraction, making the learning process lighter and faster. despite having a larger size and number of parameters compared to mobilenet-v3, the proposed model f-graphs could be beneficial for incorporation into mobile assistive technologies that aid in extracting information from printed texts for visually impaired people. the proposed model f-graphs demonstrated good performance on the test data. however, although the model made errors in classifying some graphs that had ambiguities, we think that this will not be an issue in deployment environments, since scenarios of this kind are unlikely to occur. one limitation of this model arises when presented with two or more graphs sketched on the same axes. in some real-world scenarios, the model tends to misclassify functions when there are multiple analytical expressions in the image. future work should focus on extracting additional information from images, such as identifying the analytical expressions of functions, to provide more comprehensive details about graph behaviors. references algan, g., ulusoy, i.: image classification with deep learning in the presence of noisy labels: a survey. knowl.-based syst. 215, 106771 (2021). https://doi.org/10.48550/arxiv.1912.05170 bakhtiarnia, a., zhang, q., iosifidis, a.: efficient high-resolution deep learning: a survey. acm comput. surv. 56 (2024). https://doi.org/10.1145/3645107 bhatt, d., et al.: cnn variants for computer vision: history, architecture, application, challenges and future scope. electronics 10(20) (2021). https://doi.org/10.3390/electronics10202470 cao, p., zhu, z., wang, z., zhu, y., niu, q.: applications of graph convolutional networks in computer vision. neural comput. appl. 34(16), 13387–13405 (2022). https://doi.org/10.1007/s00521-022-07368-1 article  math  google scholar  chen, j., nagaya, r., takagi, n.: development of a method for extracting graph elements from mathematical graphs. in: 2012 ieee international conference on systems, man, and cybernetics (smc), pp. 2921–2926. ieee (2012) google scholar  chollet, f., et al.: keras (2015). https://keras.io dai, j., lin, s.: image recognition: current challenges and emerging opportunities (2018). https://www.microsoft.com/en-us/research/lab/microsoft-research-asia/articles/image-recognition-current-challenges-and-emerging-opportunities/. accessed 20 jun 2024 deng, j., dong, w., socher, r., li, l.j., li, k., fei-fei, l.: imagenet: a large-scale hierarchical image database. in: 2009 ieee conference on computer vision and pattern recognition, pp. 248–255. ieee (2009).https://doi.org/10.1109/cvpr.2009.5206848 fuda, t., omachi, s., aso, h.: recognition of line graph images in documents by tracing connected components. syst. comput. japan 38(14), 103–114 (2007). https://doi.org/10.1002/scj.10615 goncu, c., marriott, k.: accessible graphics: graphics for vision impaired people. in: cox, p., plimmer, b., rodgers, p. (eds.) diagrams 2012. lncs (lnai), vol. 7352, pp. 6–6. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-31223-6_5 chapter  math  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016). https://doi.org/10.1109/cvpr.2016.90 howard, a.g., et al.: mobilenets: efficient convolutional neural networks for mobile vision applications. comput. res. repository (corr) (2017). https://doi.org/10.48550/arxiv.1704.04861 kingma, d.p., ba, j.: adam: a method for stochastic optimization. arxiv (2014). https://doi.org/10.48550/arxiv.1412.6980 krizhevsky, a., hinton, g.: learning multiple layers of features from tiny images. tech. rep. 0, university of toronto, toronto, ontario (2009). https://www.cs.toronto.edu/~kriz/learning-features-2009-tr.pdf krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. commun. acm 60(6), 84–90 (2017). https://doi.org/10.1145/3065386 lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015). https://doi.org/10.1038/nature14539 li, l., jamieson, k., desalvo, g., rostamizadeh, a., talwalkar, a.: hyperband: a novel bandit-based approach to hyperparameter optimization. j. mach. learn. res. (2016). https://doi.org/10.48550/arxiv.1603.06560 nazemi, a., fernando, c., murray, i., a. mcmeekin, d.: accessible and navigable representation of mathematical function graphs to the vision-impaired. comput. inf. sci. 9(1), 31 (2015). https://doi.org/10.5539/cis.v9n1p31 phyo, y.k.: graphical functions made from an effortless sketch. ms paint that returns equations. https://towardsdatascience.com/graphical-functions-made-from-an-effortless-sketch-266ccf95c46d (2020). accessed 8 apr 2024 souza, s.d.a.: usando o winplot. http://www.mat.ufpb.br/~sergio/winplot/winplot.html (2004). accessed 08 apr 2024 takagi, n.: mathematical figure recognition for automating production of tactile graphics. in: 2009 ieee international conference on systems, man and cybernetics, pp. 4651–4656 (2009). https://doi.org/10.1109/icsmc.2009.5346749 tan, m., le, q.v.: efficientnet: rethinking model scaling for convolutional neural networks. computing research repository (corr) (2019). https://doi.org/10.48550/arxiv.1905.11946 download references acknowledgments this research was funded by the brazilian federal agency for support and evaluation of graduate education (capes) and developed with the support of the center for high-performance distributed computing of pará (ccad) of the federal university of pará (ufpa). author information authors and affiliations instituto de ciências exatas e naturais, universidade federal do pará, belém-pa, brazil joaquim viana, helder matos, marcelle mota & reginaldo santos authors joaquim vianaview author publications search author on:pubmed google scholar helder matosview author publications search author on:pubmed google scholar marcelle motaview author publications search author on:pubmed google scholar reginaldo santosview author publications search author on:pubmed google scholar corresponding author correspondence to joaquim viana . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. dataset availability the dataset for this project is publicly available to facilitate transparency, reproducibility, and further research through the following link: https://github.com/jvianna07/fgraphs_dataset/. the script used is available here: https://github.com/jvianna07/functions_graphs/ rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper viana, j., matos, h., mota, m., santos, r. (2025). classifying graphs of elementary mathematical functions using convolutional neural networks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_19 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords graph of a function curve recognition image classification convolutional neural networks deep learning publish with us policies and ethics profiles joaquim viana view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature performance analysis of yolov3 for real-time detection of pests in soybeans | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper performance analysis of yolov3 for real-time detection of pests in soybeans conference paper first online: 28 november 2021 pp 265–279 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) performance analysis of yolov3 for real-time detection of pests in soybeans download book pdf download book epub fábio amaral godoy da silveira  orcid: orcid.org/0000-0002-8187-892810, everton castelão tetila  orcid: orcid.org/0000-0001-5396-766x10, gilberto astolfi  orcid: orcid.org/0000-0003-2565-182211,12, anderson bessa da costa  orcid: orcid.org/0000-0003-4339-300310 & … willian paraguassu amorim  orcid: orcid.org/0000-0002-3456-548610  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1282 accesses 5 citations abstract in this work, we evaluate the you only look once (yolov3) architecture for real-time detection of insect pests in soybean. soybean crop images were collected on different days, locations, and weather conditions between the phenological stages r1 to r6, considered the period of the high occurrence of soybean pests. for training and testing the neural network, we used 5-fold cross-validation analyzing four metrics to evaluate the classification results: precision, recall, f-score, and accuracy; and three metrics to evaluate the detection results: mean absolute error (mae), root mean square error (rmse) and coefficient of determination (r\(^2\)). the experimental results showed that the yolov3 architecture trained with batch size 32 leads to higher classification and detection rates than batch sizes 4 and 16. the results indicate that the evaluated architecture can support experts and farmers in monitoring pest control action levels in soybean fields. we thank the centro nacional de desenvolvimento científico e tecnológico (cnpq), the coordenação de aperfeiçoamento de pessoal de nível superior (capes), nvidia corporation for the graphics card donation, and the fundação de apoio ao desenvolvimento do ensino, ciência e tecnologia do estado de mato grosso do sul (fundect). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others identification of wormholes in soybean leaves based on multi-feature structure and attention mechanism article 20 december 2022 weed detection in soybean crop using yolo algorithm chapter © 2023 implementation of yolov7 for pest detection chapter © 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. agronomy computer vision motion detection model plants object vision 1 introduction soy (glycine max) is a vegetable crop belonging to the fabaceae family. it is mainly used for human nutrition (in the form of soybean oil, tofu, soy sauce, soy milk, soy protein, soybeans) and animal nutrition (in the preparation of animal feed). additionally, it has an excellent nutritional profile and essential economic support [1]. in the 2019–2020 season, brazil has once again become the world’s largest producer of soybeans, surpassing its main competitor, the united states. for the brazilian crop in 2020–2021 is expected a record production of 134.45 million tons, representing an increase of 7.7% over the previous crop [2]. the state of mato grosso is the largest brazilian producer, followed by the rio grande do sul, paraná, goiás, and mato grosso do sul [3]. it is estimated that most of the direct costs for soybean production are concentrated in fertilizers (27.82%), pesticides (18.24%), machinery operations (9.10%), seeds (7.35%) and depreciation of agricultural machinery and equipment (6.76%) [4]. therefore, one of the highest costs for soybean production is applying pesticides to control pests that can cause damage to the end product. monitoring pests from the beginning of soybean development allows a more efficient application of pesticides. agricultural inputs can be applied in the correct amount and places, reducing production costs and the environmental impact from the overuse of chemical control. additionally, contributing to human health and food safety [5]. sampling methods, visual examination of plants, and examination of soil samples have been used in monitoring pests, and their damage in the crop [6]. regardless of the method used to assess field pest infestation, the number of insects should be recorded at each sampling point, allowing to determine the level of infestation in each area of the field. the higher the number of samples taken in an area, the more confidence in predicting the level of pest infestation. it is recommended at least six samplings for plots up to 10 ha, eight considering plots up to 30 ha, and ten considering plots up to 100 ha. for larger farms, a division into 100 ha plots is recommended [6]. digital images collected from the field are an increasingly used alternative to manual sampling methods. computer vision systems fed with these digital images perform real-time detection of pest insects. automatically analyzing images is crucial for making these applications viable since many images are captured. in addition, high spatial resolution cameras can be embarked on unmanned aerial vehicles (uavs) to capture images of the crop on data collection missions, allowing experts and farmers to make better pest management decisions [7]. uavs can also be programmed to perform automatic flight plan operations, with an autonomy of over 1 h depending on the model, covering average areas that can exceed 1,000 ha per flight [8]. however, few works address the use of digital images to identify insect pests in soybeans. furthermore, no reports have been found in the literature of pest monitoring in soybean fields using state-of-the-art object detection methods such as yolov3 [9], faster r-cnn [10], and retinanet [11]. this paper proposes to evaluate the you only look once (yolov3) architecture to detect pest insects in an actual field environment under different lighting conditions, object size, and complex background variations. yolov3 is one of the most popular series of object detection models. its advantage has been to provide real-time detections while approaching the accuracy of state-of-the-art object detection models. soybean crop images were collected on different days, locations, and weather conditions between the phenological stages r1 to r6, considered the period of the high occurrence of soybean pests. for training and testing the neural network, we used 5-fold cross-validation analyzing four metrics to evaluate the classification results: accuracy, recall, f-score, and precision; and three metrics to evaluate the detection results: mean absolute error (mae), root mean square error (rmse) and coefficient of determination (r\(^2\)). to the best of our knowledge, no papers were found that address state-of-the-art object detection methods for pest detection in soybean. 2 related work by using deep learning, the authors in [12] created a framework that identifies ten types of pests of different crops using a manually collected dataset. the dataset was collected by downloading 5,629 images from search engines (google, baidu, yahoo, and bing) and taking outdoor footage using an iphone 7 plus. in the pre-processing phase, data augmentation was used to expand the dataset. further, the grabcut and watershed algorithms were implemented to remove complex background from the images. in the training phase, vgg-16, vgg-19, resnet50, resnet152 and googlenet were used. the googlenet model outperformed other models in terms of accuracy, model complexity, and robustness in the experiments. the original model achieved an accuracy of 93%, while the fine-tuned model achieved an accuracy of 98.91%. on the other hand, googlenet required more computational resources. as a drawback, the complex architecture of inception-v3 makes it challenging to adjust the layer structure for a specific dataset. a novel pest detection approach called deeppest, based on two-stage mobile vision, was proposed in [13] for small-sized pest recognition and multiple species classification in an unbalanced dataset. first, deeppest extracts multi-scale contextual information from images. next, a multi-projection pest detection model (mdm) is proposed and trained by crop-related pest images. the mdm function can combine pest contextual information from low-level convolutional layers with high-level layers to generate the super-resolved feature of deeppest. experimental results show better performance of deeppest compared with other state-of-the-art methods. although the dataset contains 17,192 pest images captured in a field environment with 76,595 pest annotations, the dataset is restricted to wheat and rice crops. through the use of convolutional neural network (cnn), the authors in [14] developed a mobile application for detecting pests and diseases in rice. an inception-v3 model was trained using transfer learning with the weights obtained from imagenet. the final layer of the model was retrained from scratch using the provided images. experiments showed that the model achieved an accuracy of 90.9%. the mobile app can display the appropriate description of the pest, its possible damage, and how to control it. nonetheless, no information about the level of infestation or the number of pests in the crop can be found. the authors in [15] created and made available a large-scale dataset, called ip102, for pest insect recognition, including over 75,000 images of 102 species. initially, the performance of classification with shallow approaches was evaluated, comparing six methods: color histogram, gabor filter, gist, scale invariant feature transform (sift), speeded up robust features (surf), local color histogram; and two classifiers: support vector machine (svm), k-nearest neighbors (k-nn). next, four deep learning approaches were compared: alexnet, googlenet, vggnet, and resnet. as a result, the best shallow approach achieved 19.5% accuracy using the combination of surf and the svm classifier. resnet stood out among the deep approaches. last, five state-of-the-art detection methods were evaluated on ip102: faster r-cnn, fpn, ssd300, refinedet, and yolov3. the results show that the performance of the two-stage detector (fpn) was better than the one-stage detector (ssd300, refinedet, and yolov3). the experimental results showed low accuracy (<50%) in almost all scenarios, indicating that the shallow and deep methods cannot yet handle pest recognition well on data sets with inter-class and intra-class variance along with data imbalance. similarly, using convolutional neural networks (cnn), the authors in [16] evaluated three detection and classification methods to identify five species of insects collected in traps in a factory environment. with a set of 200 original photos of 3,026 insects, the following methods were compared: (1) adaptive thresholding combined with vgg-16, (2) single shot multibox detector (ssd) developed over vgg16 network but replacing the fully connected layers, (3) vgg16 with sliding window approach. ssd produced the best results among the three methods in the experiments, achieving detection and classification rates of 84% and 86%, respectively. in the paper, data augmentation was performed on the training set and test set. note, however, that data augmentation should be done on the training set only. otherwise, the visual patterns of an original image learned during training may be the same as those of an augmented image in the test set, causing overfitting and, ultimately, making the data set unreliable. in reference [17], an improved network architecture based on vgg19 was implemented for the detection and classification of 24 insect species collected in crop fields such as rice and soybean. the proposed method was implemented by combining vgg19 models and a particular network called region proposal network (rpn). the experimental results showed that the proposed method achieved an accuracy of 89.2% on the mpest dataset and is superior to the traditional state-of-the-art ssd (85.3%) and fast r-cnn (79.6%) methods. however, there are still some problems with this method, such as target detection errors. although the paper proposes a cnn to solve the problem of multi-classification of agricultural insects, species frequently found on soybean, such as euschistus heros, edessa meditabunda, nezara viridula, anticarsia gemmatalis, spodoptera spp. and diabrotica speciosa, were not present in the mpest dataset. the deep residual learning method was used in [18] for identifying ten classes of agricultural pests in images with complex backgrounds. the method’s performance was improved after optimizing with deep residual learning the pre-trained resnet101 and resnet50 models in imagenet. compared with traditional support vector machines (svm) and convolutional neural networks (cnn), such as alexnet, the resnet-101 model based on deep residual learning achieved 98.67% accuracy in agricultural pest recognition, being significantly higher than traditional svm and cnn. the deep residual network could be combined with object detection methods such as faster r-cnn or r-fcn to track pest targets in real-time, providing a practical value of the method in agricultural pest control tasks. the work in [19] aims to select the best approach for detection and identification of the most harmful pests in greenhouse tomato and pepper crops, bemisia tabaci and trialeurodes vaporariorum. a dataset with many images of infected tomato plants was created to train and evaluate machine learning models (mlp and k-nn) and deep learning models (ssd and faster r-cnn). results show that the deep learning models offer a better solution as they achieve the highest accuracy. results also indicate that egg detection and identification are a significant challenge, as they are similar in color and shape at an early stage. moreover, adding other metrics to evaluate the models would be interesting, such as accuracy and f-score. in order to achieve accurate detection and identification of pyralidae pests, the authors in [20] designed an intelligent robot on an autonomous vehicle to acquire images in the natural farm scene and also presented a pyralidae pest recognition algorithm to be used on the robot. precisely, by employing the color and shape features of pyralidae pests, they proposed a segmentation algorithm using inverse histogram mapping and spatial-constrained otsu (sc-otsu) to segment pests. further, they designed a recognition approach based on hu’s invariant moment. the experimental results show that the robot vehicle can automatically capture images of pyralidae pests in the natural farm scene, achieving 94.3% accuracy in pest recognition. this approach, however, is limited by the time complexity required. the average processing time for each frame is longer than 1 s, leading to a delay in responding to the observation results. in addition, the detection of pests under non-uniform illumination was also not efficient. the local color of the images being too light or too dark made it difficult to determine a stable threshold range in the algorithm. with the purpose of rapid detection and recognition of ten insect species affecting tea fields in china, the authors in [21] created the sift-hmax model inspired by the mechanism of human visual attention. first, the statistical model saliency using natural (sun) generated saliency maps and detected the region of interest (roi) in a pest image. to extract the attributes representing the appearance of the pest, the hierarchical model and x (hmax) was extended. sift was integrated into the hmax model to increase invariance to rotational changes. meanwhile, non-negative sparse coding (nnsc) was used to simulate the simple cell responses. texture attributes were extracted based on the local configuration pattern (lcp) algorithm. finally, the extracted attributes were fed into an svm to perform the recognition task. the proposed method performed well with a recognition rate of 85.5%, showing an advantage over the compared methods hmax, sparse coding, and nimble, but being slightly inferior to a cnn such as matconvnet (86.9%). in the work [22] a method based on svm, mser and hog was developed for aphid identification in wheat fields. the method uses a maximally stable extreme regions (mser) descriptor to simplify the complex background of the images containing the aphids, and then histogram oriented gradient (hog) attribute vectors are used in an svm for aphid classification. this method was compared with five other commonly used methods for aphid detection; its performance was analyzed using images with different aphid densities, colors, and plants’ locations. the results showed that the method provides average identification and error rates of 86.81% and 8.91%, respectively, higher than other compared methods including k-means, thresholding, svm-hog, svm, svm-hog-haar, and adaboost-haar-hog. the authors did not compare the proposed method with state-of-the-art deep learning models. the authors in [7] evaluated deep learning models trained with different fine-tuning and transfer learning parameters for the tasks of classification and counting of pest insects in soybean. first, an image segmentation stage with the slic superpixels method was considered to segment the images’ insects. in the classification stage, three deep learning models were compared: inception-resnet-v2, resnet-50 and densenet-201. the approach with the best result was densenet-201 (94.89%) using 100% fine-tuning. in the counting experiment, the authors used the weights of the cnn best rated for accuracy in a computer vision system to classify each superpixel segment of the image. the count of pest insects was obtained by adding up the superpixels of each class, thus calculating the infestation level of a crop area. the count results showed that the accuracy decreases to 90.86% when an insect is segmented into two different superpixels. it should be noted that this is an area of research that has been explored but not yet resolved. moreover, the authors did not compare the results with other state-of-the-art detection methods, such as yolov3, faster r-cnn, and retinanet. 3 materials and methods 3.1 image acquisition an experimental area of 2 ha was sown with conventional soybean cultivars and no pesticide application to create a reference collection of soybean pest insects. the agricultural area shown in fig. 1 is located at the experimental farm of the federal university of grande dourados, located in the municipality of dourados-ms, brazil, with geographic coordinates 22\(^{\circ }\)13’57.52" south latitude and 54\(^{\circ }\)59’17.93" west longitude. fig. 1. (color online) aerial view of the experimental area used for soybean planting. full size image in order to collect images of insects present in the experimental area, a sony dsc-hx300 camera with 20.4 megapixels resolution and a sm-g930f camera with 12.2 megapixels resolution were used. a total of 1,800 images (600x800 pixels) in jpg format were collected from sep/2017 to feb/2018 crop during the phenological stages r1 to r6 of the soybean reproductive phase, on different days and weather conditions, between 8 am to 10 am and 5 pm to 6:30 pm. the exposure of pests at the plant’s top usually occurs early in the day or late in the afternoon. the recommendation is to sample insects of the aerial part be performed, preferably, in the cooler weather and more humid periods of the day, as reported in [6]. the researcher captured images on-site using a camera 50 cm away from the target of interest and a 90-degree angle of the camera to the ground. the targets, in this case, correspond to the defoliating insects that cause economic damage when found at high levels in soybean fields. next, each image was annotated in labelimgfootnote 1 with support from an entomologist biologist. thus building a reference collection for the system’s training and test image set (see fig. 2), named insect12c-dataset and available at [23]. table 1 presents the total number of insects annotated by species. the unequal number of samples used in the training and test image set reflects the number of occurrences of each pest species under actual field conditions. fig. 2. (color online) sample images from our dataset, divided into 12 soybean pest species. the images were collected under actual field conditions, which include lighting conditions, object size and positioning, occlusion, complex background variations, and developmental stages. full size image table 1. the total number of insects noted per species.full size table 3.2 you only look once (yolov3) the yolov3 detection method evaluated in this paper is briefly described in the following. the source code used as a basis for our implementation can be found at https://github.com/qqwweee/keras-yolo3. unlike faster-rcnn, which has a region proposal stage, yolov3 [9] approaches object detection as a direct pixel regression problem for bounding box coordinates and class probabilities. the input image is divided into \(s \times s\) blocks. for each block, yolov3 predicts bounding boxes using dimension groups as anchor boxes. the objectivity score is predicted using logistic regression for each bounding box, which indicates the chance that the bounding box has an object of interest. in addition, the probabilities of class c are estimated for each bounding box, indicating the classes it may contain. in our case, each bounding box may contain the species of an insect or the bottom of the plantation (uninteresting object). thus, each prediction in yolov3 comprises four parameters for the bounding box (coordinates), the objectivity score, and the c class probabilities. yolov3 uses darknet-53 as its backbone to provide high accuracy, requiring fewer operations than other architectures. 3.3 experimental design we adopted 5-fold cross-validation with validation, and test sets in the insect12c [23] dataset. in each fold, we set aside the 20% allocated for testing. we then split the 80% allocated for training into two subsets, 60% to be the actual training set and 20% to be the validation set. thus, in each fold, the dataset was divided into 60% for training, 20% for validation, and 20% for testing, allowing the deep learning model to be iteratively trained and validated on different sets. finally, the classifier’s output is given by the average of the five folds in the test set. the classification task consists of predicting the class of the object in the image. the problem of identifying the location of the object along with its class is called object detection. instead of predicting the object class from an image, we now have to train a multi-label classifier to determine the location of objects and predict the class of each object (circumscribed in a rectangle called the bounding box). we consider four metrics to evaluate the classification results: precision, recall, f-score, and accuracy; and three metrics to evaluate the detection results: mean absolute error (mae), root mean square error (rmse), and coefficient of determination (r\(^2\)). we have calculated the average results of the evaluation metrics for the model studied and the standard deviation. we used the anova hypothesis test to determine statistical significance in the average performance between groups with different batch sizes. we report the p-value found for each metric, and the significance level was set at 5%. in our experiments, we used the open-source implementation of the yolov3 [9] recognized by the imagenet competition [24]. the following input parameters were used: input image width and height set to 608 \(\times \) 800 pixels, batch size by 4, 16, 32 samples, and training with 24,000 iterations. we also used the sgd optimizer [25] with a learning rate set at 0.001 and momentum at 0.9. we employed the data augmentation technique to supplement the training data by applying the random rotation operations between 0\(^{\circ }\) and 90\(^{\circ }\) clockwise/anticlockwise and changing image brightness from –10% to 10%. this procedure aims to enhance rotation invariance and illumination invariance during detection. we used colab, a google research service that allows writing code through a browser while providing free gpu resources in all of our experiments. we use the google colab pro version for our work, which provides priority access to more powerful gpu resources and high-memory virtual machines over the free colab. additionally, the lifecycle of virtual machines is increased from 12 to 24 h. 3.4 evaluation metrics to evaluate the classification results, we use four metrics: accuracy eq. (1), precision eq. (2), recall eq. (3) and f-score eq. (4). we account for true positive (tp) when the insect is detected in the correct class with a detection score \(\ge \) 0.3; and false positive (fp) when some object that is not of interest (leaves or background) is identified as an insect. in cases where the insect was not detected or was detected in the incorrect class, we say it is a false-negative (fn). a true negative (tn) occurs when the network does not detect any insect in the image where there is no insect. since all images have at least one insect, we have no true negative case in our dataset. $$\begin{aligned} accuracy =\frac{tp + tn}{tp + fp + fn + tn} \end{aligned}$$ (1) $$\begin{aligned} precision =\frac{tp}{tp + fp} \end{aligned}$$ (2) $$\begin{aligned} recall =\frac{tp}{tp + fn} \end{aligned}$$ (3) $$\begin{aligned} f-score =\frac{2*(recall*precision)}{(recall+precision)} \end{aligned}$$ (4) furthermore, we consider three metrics to evaluate the detection results: mean absolute error (mae) eq. (5), root mean square error (rmse) eq. (6) and coefficient of determination (r\(^2\)) eq. (7). $$\begin{aligned} mae = \frac{1}{n}\sum _{i=1}^{n}\left| y_{i} \hat{y}_{i} \right| \end{aligned}$$ (5) $$\begin{aligned} rmse = \sqrt{\frac{1}{n}\sum _{i=1}^{n}\left( y_{i} \hat{y}_{i} \right) ^2} \end{aligned}$$ (6) $$\begin{aligned} r^2 = 1-\frac{\sum _{i=1}^{n} (y_{i} \hat{y}_{i})^2}{\sum _{i=1}^{n} ( y_{i} \bar{y} )^2} \end{aligned}$$ (7) where \(y_{i}\) is the observed value, \(\hat{y}_{i}\) is the predicted value, \(\bar{y}\) is the mean of the observation, and n is the number of observations. 4 results and discussion classification evaluation. the classification results obtained by yolov3 for precision, recall, f-measure, and accuracy are presented in table 2. the values in percentages represent the average of the five folds in the test set. regarding accuracy, the experiment with batch size 4 obtained a higher result than size 32, resulting in fewer false positives during detection. however, yolov3 achieved the best results with batch size 32 for the other metrics, showing that the proportion of predicted insects were true-positives in most cases. note that batch size 32 results in fewer false negatives than batch size 4 and batch size 16. overall, this is the most critical measure for the studied problem since its occurrence implies the lack of agricultural product application in an area with the presence of pest insects. table 2. classification results obtained by yolov3 in average percentage on the insect12c dataset.full size table the results of the anova test indicate that there is evidence of a statistical significance in mean performance between the groups of batch sizes at a significance level of 5%, using the precision (p-value = \(2. 46e^{-04}\)), recall (p-value = \(7.99e^{-09}\)), f-score (p-value = \(2.3e^{-08}\)) and accuracy (p-value = \(1.37e^{-08}\)) as metrics. fig. 3. (color online) examples of pest-insect detection in the yolov3 architecture with batch size set at 32 samples. the cases are true positive with a detection score \(\ge \) 0.3. full size image fig. 4. (color online) examples of pest insect detection failures in the yolov3 architecture with batch size set at 32 samples. subfigures in the first row have examples of false negatives and in the second row false positives. full size image insect pest detection in soybeans. table 3 presents the measured values of the three configurations proposed. in the experiments, batch size 32 produced the best results among the sizes evaluated, obtaining mae and rmse rates of 0.41 and 0.83, respectively. the results show low mean error rates in almost all scenarios, indicating that the yolov3 architecture can handle multi-species pest detection well, even in unbalanced datasets with interclass and intraclass variance. some examples of insect-pest detection in yolov3 are shown in fig. 3. the bounding boxes are true positive with a detection score \(\ge \) 0.3. table 3. insect pest detection on the insect12c dataset for yolov3.full size table however, yolov3 presented false negatives (insects not detected or detected incorrectly) and false positives (when some object that is not of interest is identified as an insect), showing that detections fail under certain field conditions such: non-uniform illumination on the insect, complex insect-like background (e.g., herbivory and lesions on leaves), partial occlusion of the insect, and mainly low representativity of species with few samples (fig. 4). 5 conclusion in this paper, we evaluated the performance of the yolov3 architecture for the real-time detection of soybean pest insects. we also defined a baseline for soybean pest detection by publishing a new dataset called insect12c, consisting of 2,758 annotated insects from 12 species. experimental results showed that yolov3 architecture trained with batch size 32 leads to higher classification and detection rates compared to batch sizes 4 and 16; however, the method failed in some cases: areas with complex lighting conditions, herbivory and leaf injuries, partial insect occlusion, and low species representativeness with few samples. the ratio of true positives to the total predicted positives achieved did not show significant distortions, indicating that yolov3 allows tracking pest targets in real-time, offering additional practical value of the method in agricultural pest control tasks. we plan to evaluate new state-of-the-art object detection architectures with higher resolution cameras onboard the uav as future work. we also intend to make use of the oversampling technique in order to solve class unbalances. finally, we plan to implement automatic counting of detected insects by species. notes 1.labelimg is a graphical image annotation tool and label object bounding boxes in images. references hou, j., et al.: association analysis of vegetable soybean quality traits with ssr markers. plant breed. 130(4), 444–449 (2011). https://doi.org/10.1111/j.1439-0523.2011.01852.x article  google scholar  conab, acompanhamento da safra brasileira grãos v. 8 safra 2020/21 n. 3 terceiro levantamento (2020). https://www.conab.gov.br/info-agro/safras/graos/boletim-da-safra-de-graos. issn: 2318–6852 —, acompanhamento da safra brasileira grãos v. 6 safra 2018/19 n. 12 décimo segundo levantamento (2019). https://www.conab.gov.br/info-agro/safras/graos/boletim-da-safra-de-graos/item/download/28484_9a9ee12328baa359b3708d64e774e5d8. issn 2318–6852 —, compêndio de estudos conab: evolução dos custos de produção de soja no brasil, 2 (2016). https://www.conab.gov.br/institucional/publicacoes/compendio-de-estudos-da-conab/item/download/2512_c2638f76696e3b926ab22e93f9549d21. issn 2448–3710 tetila, e.c., machado, b.b., belete, n.a., guimarães, d.a., pistori, h.: identification of soybean foliar diseases using unmanned aerial vehicle images. ieee geosci. remote sens. lett 14(12), 2190–2194 (2017). https://doi.org/10.1109/lgrs.2017.2743715 article  google scholar  hoffmann-campo, c.b., corrêa-ferreira, b.s., moscardi, f.: soja: manejo integrado de insetos e outros artrópodes-praga., embrapa soja (2012). http://www.cnpso.embrapa.br/artropodes/capitulo9.pdf. isbn 978-85-7035-139-5 tetila, e.c., machado, b.b., menezes, g.v., de souza belete, n.a., astolfi, g., pistori, h.: a deep-learning approach for automatic counting of soybean insect pests. ieee geosci. remote sens. lett 17, 1–5 (2019). https://doi.org/10.1109/lgrs.2019.2954735 article  google scholar  tetila, e.c.: detecção e classificação de doenças e pragas da soja usando imagens de veículos aéreos não tripulados e técnicas de visão computacional. ph.d. dissertation, universidade católica dom bosco (2019). http://repositorio.ufgd.edu.br/jspui/handle/prefix/2385 redmon, j., farhadi, a.: yolov3: an incremental improvement (2018). corr, vol. abs/1804.02767, http://arxiv.org/abs/1804.02767 ren, s., he, k., girshick, r.b., sun, j.: faster r-cnn: towards real-time object detection with region proposal networks (2015). corr, vol. abs/1506.01497, http://arxiv.org/abs/1506.01497 lin, t.-y., goyal, p., girshick, r., he, k., dollár, p.: focal loss for dense object detection. ieee trans. pattern anal. mach. intell. 42(2), 318–327 (2020) article  google scholar  li, y., wang, h., dang, l.m., sadeghi-niaraki, a., moon, h.: crop pest recognition in natural scenes using convolutional neural networks. comput. electron. agric. 169,(2020). https://doi.org/10.1016/j.compag.2019.105174. issn 0168–1699 wang, f., wang, r., xie, c., yang, p., liu, l.: fusing multi-scale context-aware information representation for automatic in-field pest detection and recognition. comput. electron. agric. 169,(2020). https://doi.org/10.1016/j.compag.2020.105222. issn 0168–1699 mique, e.l., palaoag, t.d.: rice pest and disease detection using convolutional neural network. in: proceedings of the 2018 international conference on information science and system, series iciss ’18, pp. 147–151. association for computing machinery, new york (2018). https://doi.org/10.1145/3209914.3209945. isbn 9781450364218 wu, x., zhan, c., lai, y., cheng, m., yang, j.: ip102: a large-scale benchmark dataset for insect pest recognition. in: 2019 ieee/cvf conference on computer vision and pattern recognition (cvpr), pp. 8779–8788 (2019). https://doi.org/10.1109/cvpr.2019.00899 nam, n.t., hung, p.d.: pest detection on traps using deep convolutional neural networks. in: proceedings of the 2018 international conference on control and computer vision, series icccv ’18, pp. 33–38. association for computing machinery, new york (2018). https://doi.org/10.1145/3232651.3232661 xia, d., chen, p., wang, b., zhang, j., xie, c.: insect detection and classification based on an improved convolutional neural network. sensors 18(12), 4169 (2018). https://doi.org/10.3390/s18124169. issn 1424–8220 cheng, x., zhang, y., chen, y., wu, y., yue, y.: pest identification via deep residual learning in complex background. comput. electron. agric. 141, 351–356 (2017). https://doi.org/10.1016/j.compag.2017.08.005 article  google scholar  gutierrez, a., ansuategi, a., susperregi, l., tubío, c., rankić, i., lenža, l.: a benchmarking of learning strategies for pest detection and identification on tomato plants for autonomous scouting robots using internal databases (2019). https://doi.org/10.1155/2019/5219471 liu, b., hu, z., zhao, y., bai, y., wang, y.: recognition of pyralidae insects using intelligent monitoring autonomous robot vehicle in natural farm scene (2019). corr, vol. abs/1903.10827, http://arxiv.org/abs/1903.10827 deng, l., wang, y., han, z., yu, r.: research on insect pest image detection and recognition based on bio-inspired methods. biosyst. eng. 169, 139–148 (2018). https://doi.org/10.1016/j.biosystemseng.2018.02.008. issn 1537–5110 liu, t., chen, w., wu, w., sun, c., guo, w., zhu, x.: detection of aphids in wheat fields using a computer vision technique. biosyst. eng. 141, 82–93 (2016). https://doi.org/10.1016/j.biosystemseng.2015.11.005. issn 1537–5110 tetila, e.c.: insect12c-dataset conjunto de imagens de insetos e outros invertebrados da cultura da soja, ufgd (2021). http://evertontetila.ws.ufgd.edu.br/insect12c-dataset.zip russakovsky, o., et al.: imagenet large scale visual recognition challenge. int. j. comput. vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y article  mathscinet  google scholar  bottou, l., bousquet, o.: the tradeoffs of large scale learning. in: platt, j., koller, d., singer, y., roweis, s. (eds.) advances in neural information processing systems, vol. 20. curran associates inc. (2008). https://proceedings.neurips.cc/paper/2007/file/0d3180d672e08b4c5312dcdafdf6ef36-paper.pdf download references author information authors and affiliations federal university of grande dourados, facet, dourados, 79825-070, brazil fábio amaral godoy da silveira, everton castelão tetila, anderson bessa da costa & willian paraguassu amorim federal institute of mato grosso do sul, informática/eng. de software, campo grande, 79100-510, brazil gilberto astolfi federal university of mato grosso do sul, facom, campo grande, 79070-900, brazil gilberto astolfi authors fábio amaral godoy da silveiraview author publications search author on:pubmed google scholar everton castelão tetilaview author publications search author on:pubmed google scholar gilberto astolfiview author publications search author on:pubmed google scholar anderson bessa da costaview author publications search author on:pubmed google scholar willian paraguassu amorimview author publications search author on:pubmed google scholar editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper silveira, f.a.g.d., tetila, e.c., astolfi, g., costa, a.b.d., amorim, w.p. (2021). performance analysis of yolov3 for real-time detection of pests in soybeans. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_19 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning object detection precision agriculture pests in soybeans publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature portuguese emotion detection model using bertimbau applied to covid-19 news and replies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper portuguese emotion detection model using bertimbau applied to covid-19 news and replies conference paper first online: 30 january 2025 pp 265–280 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) portuguese emotion detection model using bertimbau applied to covid-19 news and replies download book pdf download book epub francisco bráulio oliveira  orcid: orcid.org/0000-0003-2926-82149 & jaime simão sichman  orcid: orcid.org/0000-0001-8924-96439  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 390 accesses 1 citation abstract in this study, we developed an emotion detection (ed) model for brazilian portuguese social media comments and used it to analyze emotion distribution in replies to covid-19-related news on twitter. we translated the goemotions dataset into portuguese using a large language model, preserving the original social media writing style, and fine-tuned the bertimbau model for detecting ekman’s six basic emotions plus neutral emotion. we collected tweets related to the covid-19 pandemic from major brazilian news portals and analyzed user replies for expressed emotions. chi-square tests indicated that emotion distribution depended on both the news topic and the media outlet, while the augmented dickey-fuller (adf) test showed that some emotions’ distributions were influenced by the publication date. among replies with identifiable emotions, anger (21.1%), joy (17.2%), and surprise (11.6%) were most prevalent. the brazilian pandemic inquiry (cpi) prompted the highest anger prevalence (41.9%), while news from uol notícias had the highest anger (40.0%) and g1 led in sadness (13.6%). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others using social media categorical reactions as a gateway to identify hate speech in covid-19 news article 16 october 2022 applying and understanding an advanced, novel deep learning approach: a covid 19, text based, emotions analysis study article 25 june 2021 emotions, morals and resilience: the consumption of news in ibero-america during the covid-19 pandemic chapter © 2021 1 introduction emotions constitute a fundamental aspect of human cognition and behavior. they play a critical indicator of psychological state and social interaction. understanding the emotional expressions within text not only provides insights into individual sentiment, but also unveils collective trends and behavioral patterns within communities. therefore, analysis of textual data using artificial intelligence (ai) techniques has became an important activity across multiple domains. in recent years, the intersection of natural language processing (nlp) and machine learning has facilitated significant advancements in emotion analysis. notably, the utilization of deep learning architectures, such as bert (bidirectional encoder representations from transformers), achieve significant results. social media platforms provide a significant amount of data to be analyzed by text classification models. also, sentiments and topics on social media can be a good proxy for public opinion. for instance, data from social media, such as twitter, replicate consumer confidence and presidential job approval polls [28] and contribute to understand the response to terrorist attacks [6]. recently, it helps elucidate covid-19-related discourse, providing valuable insights into public sentiment, reactions to health policies, and the dissemination of information during crises. however, while numerous studies have explored emotion analysis in english-language contexts, research in brazilian portuguese remains limited. this gap in the literature restricts comprehensive analyses of emotions expressed by portuguese-speaking communities, particularly in response to significant events like the covid-19 pandemic. addressing this gap is crucial for promoting inclusive research and understanding the unique cultural nuances present in brazilian portuguese discourse. in this study, we aim to contribute to the advancement of natural language processing (nlp) research in brazilian portuguese by developing and applying a bert-based emotion classification model, using the bertimbau language model. specifically, we seek to answer the following research questions: rq1: given an ekman-based labeled emotion corpus in english, is it feasible to use llms to generate a similarly labeled corpus in brazilian portuguese while maintaining the language style? moreover, can we build an efficient emotion transformer-based classifier in brazilian portuguese using this new corpus? rq2: what are the predominant emotions expressed in comments on brazilian media news about the covid-19 pandemic on twitter? rq3: how do these emotions distribute across different press outlets, news themes, and over time? to address these questions, we employ a multi-step methodology involving corpus translation, model fine-tuning, dataset collection, and emotion detection. this study also aims to provide insights for policymakers, healthcare experts, and marketers by explaining the emotional landscape of covid-19 discourse in brazilian portuguese. it hopes to enable more informed decision-making and communication strategies in the context of a major event. the remainder of the paper is organized as follows. section 2 gives an overview of the related work. section 3 describes our methodology, and sect. 4 presents our findings. we conclude the paper in sect. 5 with some general remarks. 2 background and related work human emotions are expressed in a multi-modal fashion, encompassing facial expressions, voice, and language [23]. however, classification strategies focusing only on textual data exhibit performance comparable to multimodal approaches [1]. a crucial aspect in developing emotion classifiers is the establishment of an emotion model, defining how emotions are represented. the main kinds of emotion models include discrete and dimensional models [46]. discrete models categorize emotions into a limited set of categories. paul ekman’s study suggests six fundamental emotions—anger, sadness, disgust, joy, surprise, and fear—originating from distinct neural systems [11]. in contrast, robert plutchik’s model suggests that emotions occur in opposing pairs at varying intensities, augmenting ekman’s set with acceptance/trust and anticipation [32]. meanwhile, dimensional models propose emotions as non-independent entities, represented within a continuous multi-dimensional space. plutchik’s two-dimensional wheel [32] and russell’s circumplex model [34] differentiate emotions based on arousal (activation vs. deactivation) and valence (pleasantness vs. unpleasantness). furthermore, the pleasure-arousal-dominance (pad) model delineates emotions across three axes [22]. the field of research known as affective computing emerged with the recognition that computational systems could potentially classify a user’s emotional state [30], from where emerged the text-based emotional detection (tbed). annotated collections of documents, or texts, with distinct emotions facilitated the development of artificial intelligence models capable of supervised emotion detection in new documents, as well as performance evaluation and comparison with other models. the first annotated corpus for emotions was the isear dataset, encompassing 7 distinct emotions [35]. subsequently, wordnet was developed as one of the primary lexical databases specifically constructed for natural language processing (nlp) in english [12]. lexical databases consist of structured sets of terms aiming to approximate the terms native speakers associate with a given concept. early nlp applications focused on emotion detection used lexical databases to identify the predominant emotion in a document [13, 38, 40, 45]. following this, word2vec [25] enabled the usage of word embeddings, opening the way for neural networks and deep learning in a variety of nlp tasks, including emotion detection. in 2017, the introduction of transformers [42] enabled the development of language models, including bert [10], representing a qualitative leap in emotion analysis tasks. presently, tbed research indicates that lexicon-based applications are still in use; however, classifiers based on machine learning and deep learning are becoming more prevalent [2, 15, 16]. recent works have employed large language models (llms) for tbed activities [3, 44]. among works focused on the portuguese language, notable efforts include the development of wordnetaffectbr, one of the earliest and most significant lexical databases in brazilian portuguese [29]. another contribution constructed a corpus of short news items (headlines and brief descriptions) and utilized latent semantic analysis to classify documents among ekman’s six emotions, along with neutral emotion. more recently, a bert-based language model named bertimbau was developed [37]. bertimbau underwent fine-tuning using a version of the goemotions corpus [9] translated by google translate to generate a model specialized in emotion classification in portuguese [14]. this model was utilized to evaluate the polarity of comments related to political discourse, and it was compared to chatgpt, demonstrating superior performance [36]. in contrast, other techniques, such as classification models based on lexical databases, were employed to identify emotions in tweets related to the covid-19 pandemic in portuguese [4]. regarding tweets related to the covid-19 pandemic written in english, multiple studies used machine learning and deep learning techniques to investigate the distribution of emotions among social media users, finding a prevalence of negative emotions such as anger [18, 27, 47]. 3 methodology in this study, we construct a model for classifying emotions in social media comments, applied to a dataset of covid-19-related tweets collected from twitter. we adapt the goemotions dataset [9] to portuguese using chatgpt to ensure that we keep the writing style of social media comments. then, we fine-tune a pre-trained portuguese language model, bertimbau, for emotion detection (ed), employing ekman’s model of six basic emotions plus neutral emotion. following this, we gather tweets related to the covid-19 pandemic from major brazilian news portals on twitter, and their user replies. the fine-tuned model is used to detect the emotions expressed in these replies. finally, we conduct statistical analysis on the distribution of these emotions relating them to the news topic, the news media, and the date when the news was published. 3.1 model training table 1. sample comments from goemotions, their translation to portuguese using google translate, and chatgpt (goemotions-br).full size table language model (lm). language models are artifacts capable of performing language-related tasks automatically. typically, they are generated through machine learning or statistical modeling. recently, lms have been build using deep neural network (dnn) models trained on large volumes of data, utilizing a transformer architecture [42]. one of the most commonly used lm architectures is the bidirectional encoder representations from transformers (bert) [10]. this modeling approach provides contextual language features, enabling lms to have generalist capabilities. that is, they can undergo fine-tuning processes, generating new models specialized in specific tasks, including sentiment analysis, machine translation, question-answering systems, and more. in this study, we utilize a lm called bertimbau [37], trained using the bert architecture on a dataset collected in brazilian portuguese. bertimbau was trained on the brwac corpus [43], compiled from web pages, consisting of 3.53 million documents and a total of 2.68 billion tokens. fine-tuning. fine-tuning a language model (lm) typically involves retraining one of its layers using an annotated dataset tailored to a specific task. this study aimed to develop an lm specialized in emotion detection (ed) using the goemotions dataset, which includes about 58,000 reddit comments manually annotated into 27 emotions, later aggregated into ekman’s six basic emotions plus neutral. hammes and freitas [14] previously translated this dataset using google translate but noted its inability to preserve the social media writing style. recent studies [8, 17] have used chatgpt for translating corpora to fine-tune lms for underrepresented languages. following this approach, we created goemotions-br by translating the goemotions corpus with chatgpt-3.5-turbo, maintaining the original writing style and localizing slang (see table 1). the translation prompt used was: “translate the following message to portuguese. preserve the writing style, using equivalent slang and abbreviations, if needed.” for example, “looks better imo”’ was accurately translated to “parece melhor na minha opinião,” with chatgpt recognizing “imo” as “na minha opinião”. similarly, “i brought my love a cherrrryyyyy” was translated as “eu trouxe umaaaa cerreeeejjjaaaaa pro meu amoooorrrrrr”, preserving the original’s expressive style, unlike google translate. the goemotions-br corpus was then used to fine-tune the bertimbau-base model, creating bertimbau-ekman-br, specialized in ed. we followed the code and configurations provided by hammes and freitas [14]. table 2. statistics of the twitter accounts of the selected media outlets.full size table 3.2 emotion distribution the corpus. we utilize the reuters institute’s digital news report 2023 [26] to identify the most prominent digital media outlets in brazil. then, we search for the highly active twitter accounts that best represent each media outlet, as presented in table 2. we utilize the twitter api to collect tweets related to the covid-19 pandemic from these accounts. the tweets are identified through contextual annotations provided by twitter itself [41], a tool that employs artificial intelligence and manual annotations to determine the topic of posts on the social network. in total, 169,281 news-related tweets and 4,861,704 replies on these news items were collected, originated from 971,176 distinct users. topic modeling. we employ a technique called latent dirichlet allocation (lda) [5] to identify the topics present in the news tweets. lda is a statistical method that discovers latent topics from a corpus of documents. before training, we exclude stopwords (i.e., very common words such as articles, prepositions, and others), web links, punctuation marks, numbers, and named entities of geographical type (e.g., city names, countries). additionally, other terms that are either too common (i.e., appearing in more than one-third of the documents) or too rare (i.e., appearing in less than 0.05% of the documents) are removed. we also create tokens from bigrams—pairs of words that occur frequently together. at the end, the dictionary have 56,444 unique tokens, used in 169,281 documents, with an average of 7.56 tokens per document. we use the implementation of the machine learning for language toolkit (mallet) library [20] for the lda algorithm to topic modeling. we identify the appropriate number of topics (as the lda input) by analyzing the coherence scores ranging from 3 to 39, with a step of 3. the coherence of a topic, used as an indicator for topic quality, is based on the distributional hypothesis that words with similar meaning tend to co-occur within a similar context [39]. the coherence score of a model is the average of the coherence scores of all topics. a coherence score of 0.38 was achieved for 12 topics, in our dataset. statistical analysis. we use the chi-square test of independence [21] to assess whether a pair of categorical variables (e.g., emotion and media outlet) are related (alternative hypothesis) or independent (null hypothesis). this is a non-parametric test, appropriate for our dataset because some of the variables (i.e., emotion, topic, and media outlet) are nominal in nature. by analyzing the test statistic and the degrees of freedom, we can determine the p-value. when the p-value is greater than the significance level (\(p > 0.05\)), we do not reject the null hypothesis, indicating insufficient evidence to conclude that the categorical variables are related. additionally, the effect size measures the strength of the relationship between the variables; larger effect sizes suggest a stronger association. we also use the augmented dickey-fuller (adf) test [19] to assess whether a time series (e.g., the monthly prevalence of emotion in replies) is stationary (alternative hypothesis) or non-stationary (null hypothesis). a non-stationary time series exhibits some time-dependent structure (e.g., variance that changes over time). when the test statistic exceeds the critical value at a 5% significance level, this is reflected in the p-value (\(p > 0.05\)), leading us not to reject the null hypothesis, indicating insufficient evidence to conclude that the time series is stationary. 4 results in this work, our aim is to train a model for emotion detection (ed) in brazilian portuguese. we usecchatgpt for dataset translation, enabling the model to classify social media comments within the scope of ekman’s basic emotions model. we then utilize this model to analyze the distribution of emotions in replies to brazilian news tweets related to the covid-19 pandemic, considering variations related to the news outlet, the topic of the news, and the time of publication. 4.1 emotion detection model table 3 presents the f1-score results for each emotion detected by the bert model used in the goemotions study [9], as well as the performance achieved with our model for the goemotions-br corpus. although the results are not directly comparable, since they were calculated on different corpora, they provide an indicative of expected outcome. our results demonstrate a similar order of magnitude and consistency across emotion classes, indicating that the developed model satisfactorily meets the conditions for analyzing emotion distribution in responses to covid-19-related news tweets in brazil. table 3. f1-score comparison of ekman emotion for goemotions and bertimbau-ekman-br models.full size table 4.2 emotion distribution table 4. summary of chi-square test of independence results for multiple pairs of categorical variables. the test statistics is presented in thousands. a p-value lower than 5% rejects the null hypothesis that the two variables are independent. the table presents the rule of thumb over cohen’s \(\omega \) measure [7] as the effect size result.full size table we conducted a multi-stage statistical analysis to investigate how the distribution of emotions varies according to different factors. firstly, we examine the global distribution of emotions in replies to news tweets, how reply volume varies over time, and how the distribution of emotions also changes over time. subsequently, we used latent dirichlet allocation (lda) to identify latent topics within the corpus of covid-19-related news tweets. then, we assess how emotions are distributed among these topics. finally, we evaluate how emotions are distributed among different news outlets. with that, we are able to map different scenarios impacting the distribution of emotions in replies, and statistically assess their impact. the results for multiple pairs of variables are summarized in table 4, along with the effect size estimated using cohen’s \(\omega \) [7]. both the topic and the news outlet are hypothesized to influence the distribution of emotions, as the statistical test fails to reject the null hypothesis that they are independent in both cases. however, the effect in both instances is small. fig. 1. the prevalence of emotions in user replies. full size image distribution over time. in total, 4,861,704 replies to news tweets were collected over a three-year period, from january 2020 to december 2022. the volume of replies varied considerably throughout the observed period, peaking at over 300,000 posts per month in the first quarters of 2020 and 2021, but becoming virtually irrelevant from february 2022 onward. conversely, fig. 1 shows the distribution of emotions across the entire corpus of news tweet replies. just under half (40.9%) of the replies exhibit no detectable emotion, categorized as neutral. among the replies with detectable emotions, anger (21.1%) emerges as the most prevalent, followed by joy (17.2%), surprise (11.6%), sadness (6.7%), disgust (1.3%), and fear (1.2%), which represents the least prevalent emotion. fig. 2. monthly distribution of emotions in the corpus. full size image furthermore, fig. 2 presents the distribution of emotions over time in replies. this figure considers only the replies with detectable emotions, thus excluding those categorized as neutral. over the course of time, slightly more than a third of the replies with detectable emotions were associated with anger, while just under a third were linked to joy. the remainder was distributed among the other emotions, with sadness experiencing a significant peak in may 2021, surpassing 20% of the replies with detectable emotions. table 5. augmented dickey-fuller test for monthly prevalence of emotions in user comments. a p-value higher than 5% fails to reject the null hypothesis that the time series of the daily amount of user comments with each emotion is non-stationary, thus indicating that the time series probably have time-dependent structures.full size table table 5 indicates that joy and neutral are likely the only emotions that are non-stationary, whereas anger, disgust, fear, surprise, and sadness are likely stationary. this suggests that the monthly prevalence of these emotions in user replies is not dependent on time. a more negative test statistic indicates a stronger rejection of the null hypothesis. consequently, we can infer with greater confidence that joy is non-stationary, while disgust is stationary. table 6. the topics identified by lda.full size table table 7. prevalence and chi-square values (in thousands) of emotions in user comments per topic. a higher chi-square value indicates a more significant deviation from the expected (random) prevalence. for example, anger appears in 20.5% of comments on cases & deaths news, with a chi-square value of \(5.9x10^3\).full size table table 8. prevalence and chi-square values (in thousands) of emotions in user comments per media outlet. a higher chi-square value indicates a more significant deviation from the expected prevalence. for example, anger appears in 29.7% of comments on bbc news brasil, with a chi-square value of \(1.1x10^3\).full size table distribution over topics. using lda, we identify 12 topics in the news tweets, shown in table 6. each topic is represented by the ten most related words, a sample tweet, its prevalence among the news tweets, coherence score, and label (e.g., cases & deaths”). for instance, the first topic is more likely to be chosen by the lda model if a news tweet contains keywords like coronavírus,” saúde,” or pacientes.” this topic has a coherence score of 38.94%, appears in 15% of news tweets, and is labeled as “impact on health system.” the chi-square test (\(\chi ^2\)) shows a significant relationship between topics and the prevalence of emotions in user comments (table 4), although with a small effect size. table 7 provides \(\chi ^2\) results and the prevalence of emotions in comments according to the topic. a higher \(\chi ^2\) value indicates a greater deviation from the mean (e.g., anger in responses to vaccination news). the highest \(\chi ^2\) is observed for anger in replies to pandemic investigation news (41.9%), higher than the mean anger prevalence (35.7%). this topic also has the lowest joy (25.4%) and sadness (9.2%). replies to cases & deaths news show the lowest anger (29.4%) but the highest sadness (16.3%), surprise (21.5%), and fear (3.5%). news on preventive measures have the second-highest sadness (14.3%) and lowest surprise (17.6%). joy is prevalent in replies to almost any topic, with the highest rates in vaccine distribution (34.6%) and international impact news (32.3%). distribution over media outlet. media outlet and emotion are also two categorical variables that hold a significant relationship, according to the \(\chi ^2\) test result (table 4), although with a small effect size. table 8 displays the prevalence of each emotion in user comments as well as the \(\chi 2\) value for a given media outlet. the coverage of uol has the highest prevalence of anger (40.0%) and the lowest of joy (26.1%) among user replies. meanwhile, the highest prevalence of sadness (13.6%) and the lowest of surprise (18.0%) are observed in user replies to the news from g1, with a significant deviation from the global mean. the highest prevalence of joy is observed in user replies to bbc news brasil (32.9%), and surprise reach the highest prevalence in user replies to the news published by sbt (22.3%). 5 conclusions and future work in this study, we developed a specialized emotion detection (ed) model for brazilian portuguese social media comments. while goemotions [9] is widely used, there have been no prior efforts to create emotion classifiers in brazilian portuguese that preserve the original social media writing style. to address this, we translated goemotions into goemotions-br, with chatgpt-3.5-turbo effectively capturing nuanced expressions. the bertimbau-base model, fine-tuned on goemotions-br, enabled the detection of ekman’s six basic emotions and neutral emotion in portuguese. our model achieved a f1-score in the 0.5–0.6 range, close to state-of-the-art results, with consistent performance across emotions, indicating evenly distributed errors and minimal bias. although there is room for improvement, the model’s performance is significantly better than chance, given the number of emotion classes. we analyzed factors influencing emotion distribution in replies to covid-19-related news, using 169,281 tweets and 4,861,704 replies. lda identified the top 12 topics, and chi-squared tests confirmed that emotion distribution depended on both the news topic and the publishing outlet. the adf test revealed non-stationarity in joy prevalence, while other emotions were stationary. most replies were neutral, with anger (21.1%) being the most prevalent identifiable emotion, followed by joy (17.2%) and surprise (11.6%). the 2021 brazilian pandemic inquiry had the highest prevalence of anger (41.9%), preventive measures news led to the most sadness (14.3%), and vaccine production news elicited the most joy (32.3%). uol notícias had the highest anger (40.0%) and lowest joy (26.1%), while g1 had the highest sadness (13.6%). future research should focus on creating ed datasets that reflect brazilian cultural nuances and evaluating emotion classifiers with other language models, such as albertina [33], pelle [24], and sabiá [31]. additionally, comparing the results on goemotions-br with other strategies, like bag-of-words, could help assess the proposed model’s quality. finally, the methodology could be extended to other tasks, such as hate speech detection. data availability statement the data and models developed in this study will be made available upon request. references akhtar, m.s., chauhan, d.s., ghosal, d., poria, s., ekbal, a., bhattacharyya, p.: multi-task learning for multi-modal emotion recognition and sentiment analysis. arxiv preprint arxiv:1905.05812 (2019) al maruf, a., khanam, f., haque, m.m., jiyad, z.m., mridha, f., aung, z.: challenges and opportunities of text-based emotion detection: a survey. ieee access (2024) google scholar  amin, m.m., mao, r., cambria, e., schuller, b.w.: a wide evaluation of chatgpt on affective computing tasks. arxiv preprint arxiv:2308.13911 (2023) bernardi, f.a., lima, v.c., rijo, r.p.c.l., alves, d.: mais do que palavras: uma análise das emoções brasileiras durante a covid-19. revista ibérica de sistemas e tecnologias de informação e40, 526–541 (2021) google scholar  blei, d.m., ng, a.y., jordan, m.i.: latent dirichlet allocation. j. mach. learn. res. 3, 993–1022 (2003) google scholar  cheong, m., lee, v.c.: a microblogging-based approach to terrorism informatics: exploration and chronicling civilian sentiment and response to terrorism events via twitter. inf. syst. front. 13, 45–59 (2011). https://doi.org/10.1007/s10796-010-9273-x article  google scholar  cohen, j.: statistical power analysis for the behavioral sciences. routledge (2013) google scholar  cui, y., yang, z., yao, x.: efficient and effective text encoding for chinese llama and alpaca. arxiv preprint arxiv:2304.08177 (2023) demszky, d., movshovitz-attias, d., ko, j., cowen, a., nemade, g., ravi, s.: goemotions: a dataset of fine-grained emotions. arxiv preprint arxiv:2005.00547 (2020) devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) ekman, p.: an argument for basic emotions. cognit. emotion 6(3–4), 169–200 (1992) article  math  google scholar  fellbaum, c.: wordnet. in: theory and applications of ontology: computer applications, pp. 231–243. springer (2010) google scholar  gill, a.j., french, r.m., gergle, d., oberlander, j.: identifying emotional characteristics from short blog texts. in: proceedings for the 30th annual meeting of the cognitive science society, pp. 2237–2242. cognitive science society (2008) google scholar  hammes, l.o.a., de freitas, l.a.: utilizando bertimbau para a classificação de emoções em português. in: anais do xiii simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 56–63. sbc (2021) google scholar  hung, l.p., alias, s.: beyond sentiment analysis: a review of recent trends in text based sentiment analysis and emotion detection. j. adv. comput. intell. intell. inform. 27(1), 84–95 (2023) article  math  google scholar  kusal, s., patil, s., choudrie, j., kotecha, k., vora, d., pappas, i.: a systematic review of applications of natural language processing and future challenges with special emphasis in text-based emotion detection. artif. intell. rev. 56(12), 15129–15215 (2023) article  google scholar  larcher, c., piau, m., finardi, p., gengo, p., esposito, p., caridá, v.: cabrita: closing the gap for foreign languages. arxiv preprint arxiv:2308.11878 (2023) lwin, m.o., et al.: global sentiments surrounding the covid-19 pandemic on twitter: analysis of twitter trends. jmir public health surveill. 6(2), e19447 (2020) article  math  google scholar  mackinnon, j.g.: approximate asymptotic distribution functions for unit-root and cointegration tests. j. bus. econ. statist. 12(2), 167–176 (1994) article  mathscinet  math  google scholar  mccallum, a.k.: mallet: a machine learning for language toolkit (2002). http://mallet.cs.umass.edu mchugh, m.l.: the chi-square test of independence. biochemia medica 23(2), 143–149 (2013) article  math  google scholar  mehrabian, a.: basic dimensions for a general psychological theory: implications for personality, social, environmental, and developmental studies (1980) google scholar  mehrabian, a.: communication without words. in: communication theory, pp. 193–200. routledge (2017) google scholar  de mello, g.l., finger, m., carpi, m.d.m., jose, m.m., domingues, p.h., cavalim, p., et al.: pelle: encoder-based language models for brazilian portuguese based on open data. arxiv preprint arxiv:2402.19204 (2024) mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space. arxiv preprint arxiv:1301.3781 (2013) newman, n., fletcher, r., eddy, k., robertson, c.t., nielsen, r.k.: digital news report 2023 (2023) google scholar  oliveira, f.b., haque, a., mougouei, d., evans, s., sichman, j.s., singh, m.p.: investigating the emotional response to covid-19 news on twitter: a topic modeling and emotion classification approach. ieee access 10, 16883–16897 (2022) article  google scholar  o’connor, b., balasubramanyan, r., routledge, b., smith, n.: from tweets to polls: linking text sentiment to public opinion time series. in: proceedings of the international aaai conference on web and social media (icwsm), vol. 4(1), pp. 122–129 (2010). https://doi.org/10.1609/icwsm.v4i1.14031 pasqualotti, p.r., vieira, r.: wordnetaffectbr: uma base lexical de palavras de emoções para a língua portuguesa. revista novas tecnologias na educação 6(1) (2008) google scholar  picard, r.w.: affective computing. mit press (2000) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models. in: brazilian conference on intelligent systems, pp. 226–240. springer (2023) google scholar  plutchik, r.: emotions: a general psychoevolutionary theory. approach. emot. 1984, 197–219 (1984) google scholar  rodrigues, j., et al.: advancing neural encoding of portuguese with transformer albertina pt. in: epia conference on artificial intelligence, pp. 441–453. springer (2023). https://doi.org/10.1007/978-3-031-49008-8_35 russell, j.a.: a circumplex model of affect. j. pers. soc. psychol. 39(6), 1161 (1980) article  math  google scholar  scherer, k.: profiles of emotion-antecedent appraisal: testing theoretical predictions across cultures. cogn. emot. 11(2), 113–150 (1997) article  math  google scholar  silva, l.g., caseli, h.m.: aspect-based sentiment analysis in comments on political debates in portuguese: evaluating the potential of chatgpt. propor 2024, 312 (2024) math  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  strapparava, c., mihalcea, r.: learning to identify emotions in text. in: proceedings of the 2008 acm symposium on applied computing, pp. 1556–1560 (2008) google scholar  syed, s., spruit, m.: full-text or abstract? examining topic coherence scores using latent dirichlet allocation. in: 2017 ieee international conference on data science and advanced analytics (dsaa), pp. 165–174 (2017). https://doi.org/10.1109/dsaa.2017.61 sykora, m., jackson, t., o’brien, a., elayan, s.: emotive ontology: extracting fine-grained emotions from terse, informal messages (2013) google scholar  twitter, i.: twitter context annotations: list of entities (2022). https://github.com/twitterdev/twitter-context-annotations vaswani, a., et al.: attention is all you need. adv. neural inform. process. syst. 30 (2017) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation (lrec 2018) (2018) google scholar  wake, n., kanehira, a., sasabuchi, k., takamatsu, j., ikeuchi, k.: bias in emotion recognition with chatgpt. arxiv preprint arxiv:2310.11753 (2023) wang, x., zheng, q.: text emotion classification research based on improved latent semantic analysis algorithm. in: conference of the 2nd international conference on computer science and electronics engineering (iccsee 2013), pp. 210–213. atlantis press (2013) google scholar  wang, y., et al.: a systematic review on affective computing: emotion models, databases, and recent advances. inform. fusion 83, 19–52 (2022) article  math  google scholar  xue, j., et al.: twitter discussions and emotions about the covid-19 pandemic: machine learning approach. j. med. internet res. 22(11), e20550 (2020) article  google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001. author information authors and affiliations laboratório de técnicas inteligentes (lti), escola politécnica (ep), universidade de são paulo (usp), são paulo, brazil francisco bráulio oliveira & jaime simão sichman authors francisco bráulio oliveiraview author publications search author on:pubmed google scholar jaime simão sichmanview author publications search author on:pubmed google scholar corresponding author correspondence to francisco bráulio oliveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper oliveira, f.b., sichman, j.s. (2025). portuguese emotion detection model using bertimbau applied to covid-19 news and replies. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_19 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords nlp emotion detection deep learning topic modeling covid-19 news publish with us policies and ethics profiles jaime simão sichman view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature heuristic solutions for the 2d bin-packing problem with varied size | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper heuristic solutions for the 2d bin-packing problem with varied size conference paper first online: 30 january 2025 pp 65–80 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) heuristic solutions for the 2d bin-packing problem with varied size download book pdf download book epub eduardo augusto militão fernandes9, thiago ferreira de noronha9 & amadeu almeida coco10  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 400 accesses abstract this work addresses the 2d bin-packing problem with varied size and proposes heuristic solutions for it. an extensive literature review on the problem is carried out, and the state-of-the-art literature algorithm for its variation with guillotine constraint is reproduced in order to validate the reported results. it is also used to build a baseline which is compared with a proposed improvement for this algorithm in order to enable the addition of a new constraint to the problem that meets an industry demand for its practical application. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others adaptive sequence-based heuristic for two-dimensional non-guillotine packing problems chapter © 2021 applying triple-block patterns in solving the two-dimensional bin packing problem article 26 june 2017 comparative study of heuristics for the one-dimensional bin packing problem chapter © 2023 1 introduction given one-dimensional boxes and items of varying lengths, the bin packing problem (bpp) seeks to determine who to allocate items to each box in order to pack all items while minimizing the number of boxes used. such a problem is famously np-hard, which means that a solution to the bpp is also a solution to every problem in the np-hard class, and, going further, there is no known algorithm of polynomial complexity to solve it. the bin-packing problem is one of the most studied classical problems in computing, with several heuristic algorithms having already been proposed to quickly find good, but not necessarily optimal, solutions for it. in this context, the 2d bin-packing problem with varied size (2dvsbpp) is a variation of the original bpp, with rectangular two-dimensional boxes and items, with such boxes having different sizes, and the objective is to pack all items minimizing the total area spent for this (or, in the same way, maximizing the use of the total area). more formally, given n rectangular items of widths \(w_i\) and heights \(h_i\), with \(i\in [1, n]\), and m rectangular boxes of widths \(w_j\) and heights \(h_j\), with \(j \in [1, m]\), one wants to define the position of each item in exactly one box such that the items are entirely within a box, there is no overlapping of items and the sides of an item are parallel to the sides of its respective box, in order to minimize the sum of the areas of the boxes used. as it is a generalization of bpp, 2dvsbpp is also in the class of np-hard problems, and methods to solve it become even more complex and elaborate, given the exponentially greater number of possible combinations of items with boxes due to the addition of a extra dimension. furthermore, 2dvsbpp is of great relevance to the industry [2, 4], having as its immediate application the determination of the layout for cutting metal sheets in order to minimize material waste. in this context, another way of describing the problem (which will be used throughout the work) is to determine a sequence of vertical and horizontal cuts in the base material in order to obtain rectangular items from it. since these cuts have to be from one end of the base material to the opposite one, this version of the problem is known as 2dvsbpp with guillotine constraint [9]. due to its direct applicability in the industry, it is of great interest to study this problem and develop more efficient solutions for it, including ones that fit in strict industry requirements [2, 4, 5]. the objectives of this work are, thus, three-fold: to carry out an extensive literature review for the 2d bin-packing problem, focusing on its version with varied size and guillotine constraint; reproduce the state-of-the-art algorithm for the 2dvsbpp with guillotine constraint and analyse the results obtained by [3]; propose an improvement to this algorithm, taking into account not only the use of the total area spent but also the real practicality of the generated solutions by reducing the number of changes of cut orientations in the solution and enable the introduction of a new constraint to the problem in order to increase its applicability in the industry, by setting a hard limit on such number of cut orientation changes. the remainder of this paper is organized as follows: sect. 2 presents the literature review carried out for the 2dvsbpp. section 3 introduces the methodology used in the work, specifying the process of collecting test instances, implementing the baseline algorithm, proposing an improvement to the algorithm, adding a new constraint to the problem and the corresponding results obtained. finally, sect. 4 concludes the work. 2 literature review in [9], the authors proposed a notation to define 4 different variations of the 2d bin packing problem: 2bp|o|g: items cannot be rotated and must be positioned to allow guillotine cutting 2bp|r|g: items can be rotated and must be positioned to allow guillotine cutting 2bp|o|f: items cannot be rotated and the cutting shape is free 2bp|r|f: items can be rotated and the cutting shape is free a guillotine cut is a straight, vertical or horizontal cut that must be made from one side of the box to the other. thus, the guillotine cutting constraint implies finding a configuration for the arrangement of items in such a way as to allow such items to be extracted only through cuts of this type. such constraint is very common in industrial applications. still in this work, a different heuristic is presented for each of the four variations of the problem, and an algorithm based on tabu search that allows using the four heuristics to solve the corresponding variation was also presented, and became the state-of-the-art of the era. in the work of [12], an exact solution for the 2dvsbpp is presented. the authors used an integer programming (ip) formulation for the problem which, according to the text, is very difficult to be solved by standard solvers. they then presented four lower-bounds for the problem, and then implement a “branch-and-price” algorithm that applies these bounds to optimally solve the problem. a new set of instances for the problem, with 5 bin sizes and number of items equal to 20, 40, 60, 80 and 100, was also proposed. it is shown that one of the presented lower-bounds, which makes use of the dantzig-wolfe decomposition, is extremely accurate, capable of providing a bound of 3.89% on average at the root node of the branch-and-cut tree. such lower-bound is found through a mixed-integer programming model. in [11], the 2dvsbp|o|f is discussed, that is, 2bp|o|f with items and boxes of varying sizes. the authors presented a two-phase algorithm for the problem. the first step consists of solving consecutive strip packing problems, using the width of the current box as the width of the strip, until all items have been packed. in the second step, a repacking process is carried out on each box, in the hope of repacking all of its items into a box of smaller area, through successive attempts on increasingly larger boxes. the algorithm was tested on instances from the literature and new proposed instances, whose generation process is described in detail. in [8], the authors presented two algorithms based on dynamic programming (dp) to solve the 2dvsbpp with a limited number of each bin size. the first algorithm, h1, only solves the 2dvsbpp without guillotine cuts and with orientation constraints. the second, h2, is capable of incorporating or not each of these constraints. the dp does not solve this problem optimally, since it has an exponential number of states. in this way, the dp is used to solve the problem approximately, by uniting several states into just one, thus losing information but gaining time and space. on average, the h1 algorithm is able to obtain considerably better solutions than h2, when the constraints that h2 can incorporate are not present. alvarez-valdés et al. [1] tackled the 2d and 3d variable sized and variable cost bin packing problems. the authors proposed a two-phase algorithm to solve the problem. the first phase consists of a grasp, which generates a set of local optimal solutions. the second phase is a path relinking algorithm, responsible for combining the solutions from the first phase into a better solution. hong et al. [6] addressed the 2dvsbp|o|g variant and introduced a heuristic called btvs, which selects the next box size using a procedure based on backtracking. for each size chosen, a semi-greedy algorithm is used to try to fill the box with the largest area of items possible, considering different item orderings. such heuristic makes use of two new algorithms, one semi-greedy and the other greedy, described by the authors, and in the end the best solution returned by one of the two methods is chosen. the authors also presented a heuristic based on simulated annealing, where the neighborhood function simply consists of selecting two random items and swapping their order. this method is designed to improve the solution found by btvs. in [2], the authors proposed the two-dimensional three-stage guillotine cutting problem with batch precedence constraints, which is similar to 2dvsbpp, but with an additional constraint that solutions can have a maximum depth of three that is, they must start with a set of vertical cuts, followed by a set of horizontal cuts, followed by a final set of vertical cuts and the order of the items being obtained follows a pre-defined sequence. such restrictions are motivated by demands from the glass cutting industry. three heuristics are presented for this problem, and experiments are carried out on publicly available instances. similarly, in [4] the same authors propose constructive heuristic methods, whereas in [5], a biased random-key genetic algorithm is proposed for the same problem. in [13], the authors presented a matheuristic approach for 2dvsbp|o|g. in this algorithm, feasible solutions are successively constructed by solving entire schedules iteratively, one sub-problem each time. three different integer programming models were presented, each one better but more expensive than the other. the developed algorithm surpasses the then state of the art, both in the problem in question and in the 2d single size bpp problem. in the latter, the procedure was able to prove the optimality of the solution in 415 of the 500 instances tested. recently, gardeyn and wauters [3] proposed a method for \(2dvsbp|*|g\), that is, the orientation constraint can be applied or not. the authors used a rooted tree to represent a solution, where leaf nodes are empty items/spaces, and inner nodes define which direction child nodes are cut (vertical or horizontal). the proposed algorithm is based on the ruin-and-recreate paradigm, and computational experiments show that the algorithm is the new state-of-the-art for \(2dvsbp|*|g\), and equals the state-of-the-art for \(2dbp |*|g\), although it has considerably longer execution times. finally, [10] carried out an extensive bibliographical review of the original bin packing problem, exposing its formal notation, applications and, mainly, proposes a notation note to catalog all the variations and restrictions of the bpp. furthermore, it also presents statistics that illustrate how the problem has been more or less researched over the years. 3 methodology and contribution based on the literature review, it was decided to collect and use the test instances proposed in [7, 12] and [11], for a total of 855 instances. the first set contains 500 instances separated into 10 groups, namely mc_b1, up to mc_b10. the second set consists of 340 instances divided into two groups of 170 each, where the first, nice, contains instances of items of homogeneous size, while the second, path, contains items of vastly different and extreme sizes and dimensions. the last set contains 15 instances divided into 3 groups: m1, m2 and m3. 3.1 literature baseline implementation the algorithm chosen for reproduction and implementation was the one proposed in [3], for the \(2dvsbp|*|g\), as it is an extremely recent work, is the current state of the art for this problem and is well specified in the original article. the entire description of the algorithm presented by the authors was meticulously followed, and the parts of the approach not detailed in the original article were implemented with a focus on minimizing the time and memory complexity of the algorithm, while maintaining the expected behavior. the algorithm is based on a goal-driven heuristic of ruin and recreate. a rooted tree represents a solution, where leaf nodes are empty items/spaces, and internal nodes define in which direction child nodes are cut. more specifically, if an internal node represents cuts in a certain direction, its child node, which is also internal, represents cuts in the opposite direction to that of its parent (that is, an internal node represents a sequence of cuts in the same direction). the procedure uses two values to guide the search for better solutions: the total area of boxes used and the total area of unpacked items. this is because the procedure addresses invalid solutions rather than discarding them. the ruin phase consists of deleting random nodes (item or internal) from the solution tree until a certain maximum number of nodes have been deleted and as long as the area of the modified solution is greater than an area limit. the recreate phase tries to insert each unpacked item into the solution, whether or not it may open new boxes to do so. each item is inserted in the best empty space for it, considering both orientations for it and for the cut that will generate it. however, to introduce a stochastic element, there is a probability that one of the options is ignored, which could lead to insertion in a sub-optimal space. finally, to decide whether the new generated solution will be accepted or not, the late-acceptance hill-climbing metaheuristic is used. in this metaheuristic, a new solution is accepted or not if, and only if, it is better than the current solution x iterations ago. algorithm 1 presents a high-level overview of the algorithm. algorithm 1 . gardeyn and wauters 2021 ruin and recreate heuristic full size image results. our implementation of the algorithm was run for all 855 instances described in the beginning of this section. the code was developed in c++11 and compiled with gcc v9.4.0. the tests were run on an ubuntu 18.04 machine with an intel core i7 980 and 24 gb of ram. in the original article, each instance ran for 10 min with 8 threads. as multi-threading was not implemented in this work and to analyze the impact of running time on the algorithm’s performance, it was decided to run our version for 25 min, then reporting the final result and the result obtained in the first 10 min of each execution. the tables 1, 2 and 3 display the results obtained in the instances of [7, 11] and [12], respectively. each line presents the average value of utilization, in percentage, obtained in each instance subset, for our algorithm with 10 and 25 min of execution, the result reported in [3] and the difference between the result in the article original and our 25-minute performance. the final line presents the averages for each column. the use is defined as the sum of the area of all items divided by the sum of the area of all boxes used in the solution found. therefore, its maximum value is 1 (100%). table 1. obtained vs original results on instances of [7] on 2vsbp|r|gfull size table table 2. obtained vs original results in instances of [11] in 2vsbp|r|gfull size table table 3. obtained vs original results in instances of [12] on 2vsbp|r|gfull size table through an analysis of the results, it can be seen that the results obtained by our implementation are close to those reported in the original article, and, therefore, it is a faithful reproduction of the original algorithm, in addition to a positive validation of such reported results in [3]. this implementation, then, served as a baseline for the following stages of this work, since it is possible to compare the performance between the algorithm and proposed modifications. 3.2 proposed improvement to the baseline algorithm although the main objective of the 2dvsbpp and, consequently, of the gardeyn algorithm, is to maximize the percentage of box area used by the solution, in practical applications this is not the only priority taken into account when one needs to produce good solutions to the problem. more specifically, in industrial applications where 2dvsbpp is used to define a sequence of guillotine cuts in metal/glass sheets in order to obtain smaller pieces, one of the criteria applied to find a feasible solution is the number of cuts to be made or, in more particular scenarios, the number of exchanges between a sequence of cuts in one orientation (for example, vertical) to a sequence of cuts in the other orientation (for example, horizontal), as addressed in [2]. this is mainly due to the fact that, in such scenarios, changing the orientation of machine cuts requires performing slow and costly procedures, such as rotating all the base material or parts of the machine itself. so, a solution that minimizes such number of exchanges of cut orientation can drastically save a lot of time and money for a manufacture. in this scenario, this last criterion can be measured as the maximum depth among the trees that represent a solution constructed by the baseline algorithm, since, as already described, an internal node of a solution tree represents a sequence of cuts in the same direction, and the children nodes represent cuts in the opposite direction. therefore, it was decided to measure the maximum depth of the solutions generated by our reproduction of the algorithm described above. the results can be seen in tables 4, 5 and 6. table 4. maximum depth of solutions obtained in instances of [11] in 2vsbp|r|gfull size table table 5. maximum depth of solutions obtained in instances of [12] in 2vsbp|r|gfull size table table 6. maximum depth of solutions obtained in instances of [7] in 2vsbp|r|gfull size table it was noticed that in the instances of [12] and, mainly, of [11], many solutions had depths considered impractical by industry standards (in [2], the industry requirement is a max depth of 3). this result motivated the elaboration of a modification to the original algorithm, which not only aims at improving the use of the area used in the solution, but also focus on generating solutions of smaller depths, less variable and with no extreme values (in certain instances, the original method generated solutions with depth above 20). description of the proposed algorithm. while analysing the solutions generated by reproducing the original algorithm, it was noticed that the high depth of the solutions was ofter concentrated in some specific regions of the box where many items were grouped with cutting directions that alternated with high frequency. therefore, the proposed solution to mitigate this problem, in order to generate solutions of less depth but maintaining or possibly increasing the use of boxes, is to use a dynamic programming (dp) algorithm. given an empty space in the box, choose a set of items to be arranged side by side in this space in order to maximize the sum of the area of the selected items. such dp is equivalent to solving a knapsack problem, where items with weight \(p_i\) and value \(v_i\) must be chosen in order to maximize the sum of their values without the sum of their weights exceeding a pre-defined limit. in this case, \(v_i\) is the area of the item i and \(p_i\) is either the width or the height of the item, depending on how it is arranged in the empty area. more formally, let i be the set of items not yet added to the solution such that each item would fit individually in the empty space to be filled, h be the height of the space and w its width. let \(h_i\) also be the height of item \(i \in i\) and \(w_i\) be the width of \(i \in i\). if \(w > h\), then the items will be arranged from top to bottom in the empty space. otherwise, left to right, side by side. without loss of generalization, considering that \(w > h\), the items are chosen according to the following recurrence equation: $$\begin{aligned} \small f(k, l)= {\left\{ \begin{array}{ll} 0, & \text {if}\,\, k=0.\\ -\infty , & \text {if}\,\, l < 0.\\ max\{f(k-1, l), f(k-1, l-w_k) + (w_k\cdot h_k)\}, & \text {if}\,\, w_k\le h_k \cup w_k > h.\\ max\{f(k-1, l), f(k-1, l-h_k) + (w_k\cdot h_k)\}, & \text {if}\,\, h_k\le w_k \cup h_k > h. \end{array}\right. } \end{aligned}$$ (1) the initial call of the equation is \(f(\vert i\vert , w)\). the asymptotic time and memory complexity of this algorithm are equal to \(o(\vert i\vert \cdot w)\), since the values of f(k, l) are computed and stored for every pair of values \((k \in \{1 ... \vert i\vert \}, l \in \{1 ... w\})\). therefore, in order to preserve the core functioning of the original algorithm and maintain its stochastic character, it was decided to introduce such dp filling as part of the recreate step of the original algorithm. this way, after each insertion of an item during this step, one of the new empty spaces generated by such inclusion (randomly selected with equal chances) is filled with the new proposed method with probability \(p = 1 \gamma ^{d }\), where d is the depth of the node corresponding to the empty space in question and \(\gamma \in \mathbb {r}\) is a parameter between 0 and 1. this means that this method is more likely to be called when the current branch of the tree is deep. after experimentation, it was decided to set \(\gamma =0.6\). furthermore, after dp defines which items should be inserted, they are cut from highest to lowest, from left to right, and the direction of their cuts is defined using the same criteria as the original article. results. since solutions with excessive depth were observed in test instances of the [11] and [12] sets, the algorithm with the proposed modification was executed in these instances, on the same machine and same parameters described in sect. 3.2.1, for 25 min each. the results are displayed in tables 7 and 8. furthermore, in order to visualize the difference in the variation in the depth of the solutions found within the same set of instances, boxplots, available in figs. 1 and 2, were generated with the distribution of such depth. it can be seen that in both ortmann and pisinger instances, no significant difference was observed between the percentage of use of the solutions generated by the two methods. however, when observing the boxplot with the distribution of solution depths, it is evident that, especially in the most challenging sets of instances in ortmann’s dataset nice300i, nice400i, nice500i, path300i, path400i and path500i -, the solutions found by the proposed algorithm have depths smaller, less varied and without extreme values than those displayed by the original algorithm. such a difference, however, is not present in the pisinger set, where the results displayed by the two methods are extremely close. table 7. results obtained vs original in instances of [11] at 2vsbo|r|gfull size table fig. 1. difference in the depth of solutions between the original algorithm and the proposed modification, in the instances of [11] full size image table 8. results obtained vs original in instances of [12] in 2dvsbp|r|gfull size table fig. 2. difference in the depth of solutions between the original algorithm and the proposed modification, in the instances of [12] full size image 3.3 insertion of additional constraint to 2dvsbpp with the aim of obtaining greater control over the depth of the solutions found in order to obey to rigid industry limitations aforementioned, it was decided to implement a rigid constraint on the maximum depth of the generated solution, parameterized as input to the algorithm, with the proposed modification. this constraint was implemented in order to minimize the impact on the quality of solutions found without it. thus, the dp filling method is only run in empty spaces whose corresponding node has a depth less than the maximum minus 1, in order to prevent the choice of an item whose cut increases the depth of the solution beyond what is allowed. in other steps of the algorithm, when selecting the order of cuts to be added and the items to be inserted, these choices are also restricted to avoid breaking the constraint. results. the proposed algorithm with the strict maximum depth constraint was executed for the instances of [11] and [12], on the same machine and parameters described in sect. 3.2.1, with the same value of gamma mentioned in sect. 3.3.1, and maximum depth of 7. the results comparing the performance of the algorithm with and without the additional constraint are in tables 9 and 10. furthermore, figs. 9 and  10 show the distribution of the solution maximum depths of both proposed methods. it can thus be seen that it was possible to add the new restriction with minimal impact on the quality of the solutions found, since the average difference between the use of the area of solutions with and without the constraint is less than 1%. even in the most challenging sets of ortmann’s dataset, where the original solutions are quite deep, the constrained algorithm was able to find solutions of practically the same quality as those without the constraint (fig. 3 and 4). table 9. results obtained vs original in instances of [11] at 2dvsbp|r|gfull size table table 10. results obtained vs original in instances of [12] in 2dvsbp|r|gfull size table fig. 3. difference in the depth of solutions between the proposed algorithm without restrictions and with restrictions, in the instances of [11] full size image fig. 4. difference in the depth of solutions between the proposed algorithm without restrictions and with restrictions, in the instances of [12] full size image 4 conclusion this work presents an extensive review of the literature on the 2d bin packing problem, focusing on its variable size version, the re-implementation of the state-of-the-art algorithm for the 2d bin packing problem with guillotine cuts constraint, the proposal of a modification to this method in order to improve the real practicality of the solutions found by it and the insertion of an additional constraint to the 2dvsbpp and, correspondingly, to the proposed algorithm in order to deal with this. through the results obtained by the reproduced code, it was possible to verify the results reported in [3]. furthermore, the proposal for a modification to the original algorithm by using a dp to more optimally choose the items that fill empty spaces in the boxes in order to reduce the final depth of the solution and, so, make it more practical in industrial applications made this method more stable under this criterion, meaning it can find solutions without extreme maximum depth values and with smaller depth variation between similar instances. finally, the addition of a hard constraint on the maximum depth of solutions gave more control over the algorithm and made it even more applicable in practical situations that require such constraint. references alvarez-valdés, r., parreño, f., tamarit, j.m.: a grasp/path relinking algorithm for two-and three-dimensional multiple bin-size bin packing problems. comput. oper. res. 40(12), 3081–3090 (2013) article  mathscinet  math  google scholar  bogue, e.t., guimarães, m.v., noronha, t.f., pereira, a.h., carvalho, i.a., urrutia, s.: the two-dimensional guillotine cutting stock problem with stack constraints. in: 2021 xlvii latin american computing conference (clei), pp. 1–9. ieee (2021) google scholar  gardeyn, j., wauters, t.: a goal-driven ruin and recreate heuristic for the 2d variable-sized bin packing problem with guillotine constraints. eur. j. oper. res. 301(2), 432–444 (2022) article  mathscinet  math  google scholar  guimaraes, m., bogue, e., pereira, a., carvalho, i., noronha, t., urrutia, s.: heurısticas construtivas para o problema de corte guilhotinado bidimensional em 3 estágios com restriçoes de precedência. anais do li simpósio brasileiro de pesquisa operacional 2 (2019) google scholar  guimarães, m.v., bogue, e.t., carvalho, i.a., pereira, a.h., noronha, t.f., urrutia, s.: a biased random-key genetic algorithm for the 2-dimensional guillotine cutting stock problem with stack constraints. in: dorronsoro, b., yalaoui, f., talbi, eg., danoy, g. (eds.) international conference on metaheuristics and nature inspired computing, pp. 155–169. springer, heidelberg (2021). https://doi.org/10.1007/978-3-030-94216-8_12 hong, s., zhang, d., lau, h.c., zeng, x., si, y.w.: a hybrid heuristic algorithm for the 2d variable-sized bin packing problem. eur. j. oper. res. 238(1), 95–103 (2014) article  mathscinet  math  google scholar  hopper, e., turton, b.: problem generators for rectangular packing problems. stud. inform. univ. 2(1), 123–136 (2002) math  google scholar  liu, y., chu, c., wang, k.: a dynamic programming-based heuristic for the variable sized two-dimensional bin packing problem. int. j. prod. res. 49(13), 3815–3831 (2011) article  math  google scholar  lodi, a., martello, s., vigo, d.: heuristic and metaheuristic approaches for a class of two-dimensional bin packing problems. informs j. comput. 11(4), 345–357 (1999) article  mathscinet  math  google scholar  mezghani, s., haddar, b., chabchoub, h.: the evolution of rectangular bin packing problem—a review of research topics, applications, and cited papers (2023) google scholar  ortmann, f.g., ntene, n., van vuuren, j.h.: new and improved level heuristics for the rectangular strip packing and variable-sized bin packing problems. eur. j. oper. res. 203(2), 306–315 (2010) article  math  google scholar  pisinger, d., sigurd, m.: the two-dimensional bin packing problem with variable bin sizes and costs. discret. optim. 2(2), 154–167 (2005) article  mathscinet  math  google scholar  polyakovskiy, s., m’hallah, r.: a lookahead matheuristic for the unweighed variable-sized two-dimensional bin packing problem. eur. j. oper. res. 299(1), 104–117 (2022) article  mathscinet  math  google scholar  download references author information authors and affiliations department of computer science, universidade federal de minas gerais, belo horizonte, mg, 31270-901, brazil eduardo augusto militão fernandes & thiago ferreira de noronha orkad, cristal, umr 9189, cnrs, centrale lille, université de lille, lille, france amadeu almeida coco authors eduardo augusto militão fernandesview author publications search author on:pubmed google scholar thiago ferreira de noronhaview author publications search author on:pubmed google scholar amadeu almeida cocoview author publications search author on:pubmed google scholar corresponding author correspondence to eduardo augusto militão fernandes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper fernandes, e.a.m., de noronha, t.f., coco, a.a. (2025). heuristic solutions for the 2d bin-packing problem with varied size. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_5 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords bin-packing problem heuristics metaheuristics publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature sea state estimation with neural networks based on the motion of a moored fpso subjected to campos basin metocean conditions | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper sea state estimation with neural networks based on the motion of a moored fpso subjected to campos basin metocean conditions conference paper first online: 28 november 2021 pp 294–308 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) sea state estimation with neural networks based on the motion of a moored fpso subjected to campos basin metocean conditions download book pdf download book epub gustavo a. bisinotto  orcid: orcid.org/0000-0002-5495-593210, lucas p. cotrim  orcid: orcid.org/0000-0001-5288-181310, fabio gagliardi cozman  orcid: orcid.org/0000-0003-4077-493510 & … eduardo a. tannuri  orcid: orcid.org/0000-0001-7040-413x10  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1254 accesses 4 citations abstract important information for the design and operation of oceanic systems can be obtained by assessing local sea state parameters such as significant height, peak period and incidence direction. techniques for motion-based inference and their possible drawbacks have been extensively discussed in the literature (their motivation coming from the simplicity of the required instrumentation when compared to traditional measuring systems), and machine learning approaches are now appearing in a few investigations. this paper addresses the estimation problem through supervised learning, using time series with the movement of a moored vessel to train neural networks models so as to estimate the sea state. such time series are obtained through simulations, that consider a model of a spread-moored fpso (floating production storage and offloading) platform with constant draft, out of a set of metocean conditions observed at brazil’s campos basin. a sensitivity analysis for different classes of neural networks was run, based on the significant height estimation, to choose the network architecture with the best results with respect to the mean absolute error metric. that topology was trained and employed in the estimation of the remaining sea state parameter, separately. the outcomes of the proposed models were confronted with other neural networks-based methods and showed up a comparable or slightly better performance in the error metrics. a preliminary discussion of the ability of the approach to deal with some classical issues on motion-based estimation is presented. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others deep learning based ship movement prediction system architecture chapter © 2019 motion simulation of moorings using optimized lstm neural network article 24 july 2023 systematic literature review of various neural network techniques for sea surface temperature prediction using remote sensing data article 22 august 2023 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biooceanography coastal sciences marine engineering ocean sciences offshore engineering physical oceanography 1 introduction the operation of ships and other oceanic structures, such as offshore platforms, is affected by loads induced by waves, wind and currents. therefore, reliable ways to obtain information about those environmental effects provide essential data for the design and engineering of installations, and they are also useful to support decision-making processes, assisting the crew in the evaluation of a particular operation, or in the need to interrupt it [17]. over the last decades, wave measurement has been carried out mainly by moored wave-buoys, whose motions are measured so as to recover the conditions around the buoys (due to their negligible dynamics). however, the setup is prone to damage and loss, and suffers from deep water mooring setbacks [17]. most marine vessels are equipped with sensors that gather vast amounts of data, such as inertial measuring units (imu), accelerometers, anemometers. in this sense, those crafts are inherently equipped with sea state measuring systems, as their sensor data can be used to infer their on-site condition, similarly to what is done with traditional buoys [2]. when compared to other wave monitoring systems, estimation of sea state based on vessel’s motions presents the main advantage of the simplicity of the required instrumentation (composed basically of accelerometers and rate-gyros), already available or easily installed on-board. on the other hand, the limitation is clear: only waves that impose a reasonable level of motion may be inferred, which means that the vessel acts as a low-pass filter, filtering the high-frequency components that do not excite its first-order response. the filter analogy also indicates that the range of frequencies where the estimation can be properly performed will depend on the size of the vessel. large-displacement vessels, such as the very large crude carriers (vlccs) on which the fpso (floating production storage and offloading) units are usually based, will have lower cut-off frequencies if compared to smaller vessels [16]. during the past years, motion-based methods for wave inference have been widely studied, mainly focusing on the parametric and the bayesian approaches. the parametric method as applied to an fpso platform can be found in ref. [19]. the formulation considers the characterization of the directional wave spectrum by a set of parameters and a linear relation between the incident wave and the movements’ spectra, which is modeled by the response amplitude operators (raos). thereafter, a nonlinear optimization problem can be defined to minimize the absolute difference between the measured spectra of motion and the one computed from crossing the raos and the parametrized wave spectrum. the bayesian approach can be found in ref. [14], with application to a container ship with forward speed, and in ref. [17] and ref. [16], to a moored fpso, among others, with a derivation that leads to a quadratic optimization problem. in this case the linearity assumption remains, but no closed-form is imposed on the wave spectrum. the estimation is performed by maximizing the product of the likelihood function by the prior distribution. the likelihood function represents the conditional probability of occurrence of a given measurement, given the directional spectrum, while the prior distribution corresponds to the previous information about the unknown spectrum coefficients. since then, variations have been made to the initial proposals, especially to the bayesian method. for instance, take ref. [4], with the analysis of the influence of the hyperparameters associated with the priors, along with a procedure for their selection, and ref. [5], with a combination of data from wave probes to the bayesian modeling, to attenuate the effect of the filtering problem. more recent years have seen machine learning-based proposals. for instance, in ref. [1], where a quadratic discriminant analysis classification and least square regression were used to estimate the set of wave parameters and in [20] with the definition of a classification problem based on random forests. some other efforts based on neural networks can be found in the literature. in ref. [6], time series of movements from an in-service frigate type vessel and data from numerical simulation were considered to output wave height, period and direction. in ref. [3], a classification problem was analyzed with the definition of a sea state from discretized wave heights and directions adding up to a total of 40 possibilities, that were classified from zigzag motions of a research vessel. from the previous discussion, it can be noticed that a large interest has been devoted to motion-based wave estimation methods in the last couple of decades, with several applications in simulations, experiments and field campaigns. however, few works have been yet carried out with machine learning techniques, particularly when taking into account large moored vessels such as fpsos. this paper aims to address sea state estimation from motions of a moored fpso using neural networks. the motion data was obtained through simulations of the vessel subjected to a dataset of environmental conditions observed at brazil’s campos basin. the next section presents an explanation of ocean waves and their parameters, where directional and power wave spectra are introduced, and also the sea state. this is followed by a brief discussion of different classes of neural networks in sect. 3. the sea state estimation procedure is detailed in sect. 4, with a description of the data and the process to select the architecture of the estimation model. in sect. 5, the outcomes of the proposed method are displayed and discussed. finally, some conclusions drawn from this study are presented along with possible further steps. 2 background: the description of ocean waves there are two main types of waves affecting the behavior of oceanic structures: wind waves and swell. in both cases, the waves start as small ripples but increase in size due to the sustained energy from the wind. wind waves result from the local wind blowing over a fetch of water, while a swell is created by winds that are no longer blowing; such waves are been generated elsewhere from a distance and are not significantly influenced by the local wind [12, 15]. the most applied theory of waves in the dynamical analysis of marine systems is the linear wave theory, whose basic element is the linear regular wave [12]. its dimensions associated with the propagation through space are depicted in fig. 1. besides those parameters, others that should be mentioned are the main direction of propagation and the wave period, which defines the time interval between the arrival of consecutive crests at a stationary point, and that can be associated with the wave length by a dispersion relation [13]. fig. 1. main space dimensions associated with linear regular waves [12] full size image fig. 2. illustration of the filtering problem with a regular incident wave full size image the regular wave is also useful to illustrate the filtering problem on motion-based wave inference, as shown in fig. 2. where for low-frequency incident waves (on bottom) oscillatory motions are generated in the wave frequency (first-order response). while it can be noticed that for high-frequency waves (on top) the vessel is mostly not responsive to the wave excitation, and hence little movement is induced, making the estimation more challenging. however, that description is not faithful to the actual ocean environment, as real ocean waves are irregular and random. despite that, they can be modeled by a suitable distribution of regular waves. under a linearity assumption, a superposition method can be applied, in which irregular waves can be described as the sum of regular components with different amplitudes and periods [12, 13]. with this simplification, waves can be represented by a power spectrum, \(s(\omega )\), that gives the distribution of energy among different frequencies, \(\omega \). to also account for the propagation of the components of the irregular wave in different directions, \(\theta \), the power spectrum can be extended to the directional spectrum, \(s(\omega , \theta )\), that defines the distribution of energy over frequencies and directions. examples of power and directional spectra are shown in fig. 3. fig. 3. power and directional wave spectra full size image due to the irregular nature of waves, it is not reasonable to assume a single wave height or wave period. instead, wave statistics can be defined, such as the significant wave height (\(h_s\)), that is computed as the average height of the highest one-third of the waves in the record, and the peak period (\(t_p\)), that is the period of maximum energy of the spectrum, among other possible characteristic periods [10]. from the analysis of the power spectrum, fig. 3 (left), those parameters can be obtained as follows: $$\begin{aligned} h_s = 4\sqrt{\int _{0}^{\infty } s(\omega )d\omega } \quad \text{ and } \quad t_p = \frac{2\pi }{\omega _p}, \;\; \omega _p = \underset{\omega }{\mathrm {argmax}} \; s(\omega ). \end{aligned}$$ (1) in the directional spectrum fig. 3 (right), a contour plot is presented. where the angular positions indicate the propagation direction, in degrees; the concentric lines correspond to periods, in seconds, of the wave components; and colors express the energy density of those components. because wind-generated waves are not uni-directional, an angular spread is adopted, thus defining a distribution of energy along the directions. hence, two angular statistics can be defined to describe the wave propagation through space: the mean wave direction (\(\beta \)) and the relative wave direction (\(\beta _m\)). the mean wave incidence direction is usually measured clockwise from the vertical axis (from north), while the relative direction is a function of the incidence direction and the heading of the vessel (\(\beta _{heading}\)), depicted in fig. 3 (right) and according to the following expression: $$\begin{aligned} \beta _m = \pi + \beta _{heading} \beta . \end{aligned}$$ (2) with the set of parameters \((h_s, t_p, \beta )\) or \((h_s, t_p, \beta _m)\), the sea state for a unimodal sea can be described. for multimodal seas, superposition can be applied with the individual wave spectra. and the sea state can be characterized by multiples sets of significant height, peak period and incidence direction. 3 background: neural networks artificial neural networks (anns or simply nns) essentially aim to approximate a function \(y=f^* (x)\), that maps an input x to a continuous value or a categorical output y, defining a relation \(y = f(x;\vartheta )\). from data, one must learn the value of the parameters \(\vartheta \) that results in the best function approximation [7]. convolutional neural networks (cnns) are a particular class of neural networks specialized in processing data with a known grid-like topology. those networks use multiple sets of shared weights, called filters, to respond to different patterns in the data. to this end, the general matrix multiplication between input and weights is replaced by a convolution operation in the formulation of the neurons [7]. the basic structure of a cnn [11] consists of a series of convolutional layers, with different depths and kernel dimensions, each associated with an activation function (usually nonlinear). and pooling layers that are responsible for reducing the size of the data by keeping local information of the mapping, such as maximum or average values. in this process, it is possible to extract features from the inputs by combining patterns underlined by each set of filters, which can be concatenated to feed an mlp for the final step of classification or regression. fig. 4. cnn basic structure full size image recurrent neural networks (rnns) are a family of neural networks for sequential data processing, whose connections between layers model a dynamic temporal behavior for time-varying input series. however, traditional rnns suffer from significant issues, the “vanishing or exploding gradients” effects, since it becomes difficult to backpropagate errors through a long-time span [7]. long short-term memory (lstm) rnns [9] deal with this problem by creating paths through time that have derivatives that neither vanish nor explode, with the application of a system of gating units to controls that data flow. in such a way that they can store information over extended time intervals. fig. 5. fpso platform at campos basin full size image 4 an estimation procedure for sea parameters the proposed estimation procedure was formulated as a supervised learning regression problem, where the goal is to compute each one of the sea state parameters (significant height, peak period and incidence direction) of a unimodal sea from time series of motion of a moored platform. table 1. information about the fpso platform at campos basinfull size table those time series were obtained from simulations of the model in 6\(^\circ \)c of freedom (dofs) of a typical spread-moored fpso unit, fig. 5, with one loading condition (single draft) under a dataset of metocean conditions that were observed at brazil’s campos basin. for data generation, the simulation model, along with the platform information, table 1, and the environmental conditions were considered as inputs to the dynasim simulator, a hydro-dynamical numerical simulator developed by a partnership between the numerical offshore tank laboratory of the university of são paulo (tpn-usp) and petrobras. 4.1 metocean data the dataset of metocean information was observed from 2003 to 2009 and consists of 18006 different groups of environmental conditions with both unimodal and bimodal seas, defined by a total of up to 10 parameters by example: wind velocity and direction, current velocity and direction and up to 2 sets of wave related parameters. as previously stated, in this paper only unimodal seas were considered, which are associated with just one set of parameters \((h_s, t_p, \beta )\) to specify the sea state, and represent a total of over 5000 observations. the distribution of each of those parameters over the dataset is shown in figs. 6, 7 and 8. the histograms indicate the most common ranges of values for each of the parameters. as a result, the training of the neural networks may favor those values over the less frequent ones, given that some of those sets have considerably fewer examples than others. on the other hand, the models may learn to identify patterns related to the statistical description of waves of that geographical region. fig. 6. histogram of significant height full size image fig. 7. histogram of peak period full size image fig. 8. histogram of incidence direction full size image in the case of the wave significant height, most occurrences are around 1.5 m to 2.5 m, while examples with height less than 1 m or greater than 4.5 m are rarely observed. taking into account the wave peak periods, a substantial amount of waves with periods lower than \(7-8\,\text {s}\) can be noticed, which correspond to high-frequency incident waves for a large vessel like the fpso under analysis. this condition may be an issue for the estimation method, due to the already mentioned filtering problem. for the incidence directions, the most remarkable aspect is the almost absence of occurrences over \(200^\circ \), which can be justified by the geographical characteristic of the vessel’s location. that is, waves with those incidence angles would be coming from the west, according to the polar plot of the directional spectrum in fig. 3 (right), which would indicate waves coming from the brazilian coast, and those situations are less frequent. 4.2 data processing before the actual training of estimators, a data treatment procedure was carried out on the time series obtained through the simulations. the output of the dynasim simulator, for each set of conditions, is composed by time series of position, velocity and acceleration in the 6 dofs (surge, sway, heave, roll, pitch, yaw), which correspond to a time span of over 3 h (11400 s) with a sampling time of 1 s. from those series, just the 6 positional motions were selected, with respect to a fixed reference frame, and a time window of 30 min (1800 s), which is consistent with the regular dynamic of changes in the environment, was sampled in the middle of the total duration. thereafter, each time series was centered by subtracting its mean value, and a dataset of over 5000 examples, each with 6 series of 1800 points, could be built. the data was then divided into 3 different datasets, following a 70/20/10 split: the training data, to be used in the training of the network, the validation data, used for the evaluation of the model with the desired metrics and the test data, to verify the performance of the system on previously unseen data. finally, each dataset was normalized using the statistical properties (mean value and standard deviation) of the training dataset. 4.3 network architecture in this paper, a single neural network architecture was sought to address, separately, the estimation problems of each of the sea state parameters. from the literature review, it was observed that the significant height estimation was more challenging than period and direction. therefore, a sensitivity study with different classes of neural networks and their hyperparameters was run to select the network configuration with best performance, whose architecture would be then repeated to estimate peak period and incidence direction. in this way, the overall estimation was divided into 3 distinct problems, each with the same network structure trained independently with distinct targets. a general structure for the pursued network was chosen as being composed of one or more feature extracting blocks (fe blocks) followed by a regression block. the latter was kept unchanged during the evaluations, while the former varied according to the following scenarios: mlp: each fe block was built from multilayer perceptrons with batch normalization and relu activation functions. the number of layers (or blocks) was chosen from 1 to 3 and the number of neurons on each layer was selected from the set \(\{64, 128, 256, 512\}\). lstm: each block was composed of lstm layers with returning sequences. the number of layers varied from 2 to 4, and the number of units was picked from \(\{32, 64, 128, 256, 512\}\). cnn1d single kernel: 1d convolutional layers were employed, each convolution was followed by batch normalization and relu activation. the number of blocks was either 2 or 3, the number of filters was selected from \(\{32, 64, 128, 256, 512\}\), and the kernel dimension of those filters from \(\{1, 3, 5\}\). cnn1d multiple kernels: this scenario was motivated by the structure proposed in ref. [18], where convolutions with different kernels were combined to extract features in image classification. the number of layers was either 2 or 3, two sets of kernel dimensions were evaluated \(\{1, 3, 5\}\) and \(\{1, 3, 5, 7\}\) and the number of filters in the first layer was selected from \(\{64, 128, 256, 512\}\), for the remaining layers the number of filters was half of the previous layer. cnn1d multiple kernels with residual connections: residual connections, inspired by ref. [8], were added to the cnn1d with multiple kernels, where the input information was passed forward in the data flow. in all scenarios, the regression block was the same, with 3 mlp layers—the first two with batch normalization and relu activation functions, and the last with a linear activation. the number of neurons in the layers was, respectively, 128, 64 and 1. all experiments were carried out within a computational environment with python 3.7.6. the layers were implemented by keras 2.4.3, using tensorflow 2.4.0 as the backend. the training process used the adam optimizer, a batch size of 32 and a number of epochs of 1000. the selected loss function was the mean absolute error (mae) between the actual significant wave height and the estimated one. the best model (based on the validation set) was stored during training. the results of the sensitivity study are summarized in table 2, where the best models obtained in each category, with respect to the validation mae, along with their parameters, are presented. table 2. sensitivity study for network parametersfull size table from the analysis, it can be noted that the approaches based on convolutions presented a better performance compared to both mlp and lstm. furthermore, the association of filters with different kernels in a layer generated improvements, while the addition of residual connections did not bring benefits. those improvements can be justified by the definition of the convolutions, that were performed over windows of motions instead of time windows. the computations of each filter take into account all time samples and a number of movements equal to the kernel size; as a result, features related to the coupling of different combinations of motions can be extracted with multiple kernels. the proposed architecture is depicted in fig. 9. fig. 9. proposed network architecture full size image 5 motion-based estimation results and discussion after selecting the network architecture through sensitivity analysis, the performance of the model was evaluated on the test dataset, with respect to the following metrics: mean value (\(\mu \)) and standard deviation (\(\sigma \)) of the error, mean absolute error (mae), mean absolute percentual error (mape), and the coefficient of determination (\(r^2\)). table 3. comparison of wave height estimation resultsfull size table fig. 10. regression results wave height full size image moreover, the results attained by the proposed architecture were compared to other neural networks available in the literature for sea state estimation, that were implemented and trained with the same dataset. in particular, three networks were employed: ssenet, from ref. [3], based on convolutions with residual connections; mlstm-cnn, from ref. [6], combining cnn and lstm layers; and sliding puzzle, also from ref. [6], with a purely convolutional architecture. although the ssenet was designed for a classification task, its structure was adapted for the considered regression problem, which may provide illustrative results for comparison purposes. the mlstm-cnn and the sliding puzzle remained unchanged since they were already designed for regression. the summary of the results for the wave height estimation is presented in table 3. there it can be observed that, except for mean value of the error, the architectures built only with convolutional layers (proposed and sliding puzzle) presented a better performance when compared to the others, with a slightly better result of the proposed in the mae and mape, and slightly worse on the standard deviation and \(r^2\). it is also worth pointing out that in both cases the average errors were low, with less than \(2\%\) error, 0.04 m of absolute error, and the \(r^2\) was over 0.99. a further analysis is enabled by the (percentage) absolute difference plot and the comparison between actual and estimated values in fig. 10. table 4. comparison of wave period estimation resultsfull size table fig. 11. regression results wave period full size image from these plots, two main effects can be outlined: the filtering problem, given that the largest errors occurred at the low periods, and that true and estimated values on high height values, over 4 m, did not present large differences, despite few occurrences. similarly, for the peak period, the comparative results are shown in table 4. from there it can be verified that all the architectures are able to present significant results, less than \(0.8\%\) of percentual error, and no more than 0.06 s of absolute error on average. however, for this sea state parameter, the proposed network was outperformed by the other approaches. the plots in fig. 11 show the same tendency of the height estimation: presence of filtering affecting the performance in low periods, and even though few examples with high periods are available, their estimation had small errors. for the incidence direction, some differences from the previous cases are worth discussing. given the circular characteristics of the target, a custom loss function was adapted considering the distance of two points \((p_1, p_2)\) in the unitary circle with angular coordinates \(\beta _1\) (true value) and \(\beta _2\) (estimation), which is stated as: $$\begin{aligned} d(p_1,p_2) = \sqrt{(\sin \beta _1 \sin \beta _2)^2 + (\cos \beta _1 \cos \beta _2)^2}. \end{aligned}$$ (3) and then, the absolute angular difference in the range \([0, \pi ]\) can be obtained with the cosine law: $$\begin{aligned} d(p_1,p_2)^2 = 1^2 + 1^2 2cos(\beta _1 \beta _2) \rightarrow |\beta _1 \beta _2| = \arccos \left( 1 \frac{d(p_1,p_2)^2}{2}\right) . \end{aligned}$$ (4) from the expression, it can be noted that the absolute angular error increases if \(d(p_1,p_2)\) increases. thus, the loss function was defined as the distance itself, in order to avoid computations of derivatives of the inverse cosine function. table 5. comparison of wave direction estimation resultsfull size table fig. 12. regression results wave direction full size image the results of the direction estimation are shown in table 5. as in the previous cases, the error metrics displayed low values in all methods—less than \(1.5^\circ \) in the mae, for example. but for most of the metrics, except the mean value of the error, the proposed model generated better outcomes than other methods. the plots in fig. 12 highlight once again the difficulty in the estimation at low periods, with large deviations from the mean in those situations. and also the limited number of directions over \(200^\circ \), which could still be properly estimated. overall, the proposed architecture produced significant results in the sea state estimation based on motions of a moored fpso obtained through simulations; low errors metrics were attained and even less frequently observed parameters could be estimated. the well-known filtering problem was also observed in all evaluations. however, in most cases, either the errors in the high-frequency range were acceptable, less than \(19\%\) for height and period, or the number of occurrences with large deviations was very restricted, just 2 cases over \(7^\circ \) of absolute error in the direction. this may suggest that patterns were extracted from data related to the movement generated by those high-frequency waves, which may have led the network to use information not only from the first-order response but also from the slow varying motions (due to higher-order wave-induced effects) in the inference; or that the models were able to learn about the statistical distribution of the dataset of environmental conditions of the region—or some other feature that requires further analysis in the continuation of the work. 6 conclusions a supervised learning approach for motion-based sea state estimation with neural networks has been presented in this paper. time series of movements, generated through simulations of a spread-moored fpso with constant draft under environmental conditions observed at brazil’s campos basin, were applied to train the networks so as to estimate each of the sea state parameters. a single network architecture was obtained from a sensitivity study run on different classes of neural networks and their hyperparameters, based on the significant height estimation. convolutional layers with multiple kernels concatenated were selected, allowing us to analyze coupling of motions among different sets of dofs. results obtained by the proposed architecture were compared to other networks designed for sea state estimation available in the literature and presented comparable or slightly better performance in the evaluated error metrics. the examination of the estimation outcomes in the individual examples displayed the classical filtering problem, in which high-frequency waves are filtered by the vessel. despite that, the performance loss, although significant, did not completely compromise the estimation, which may lead one to question whether or not the network is able to extract features related to those settings—by the slow varying information present in the time series, by learning patterns of the statistical distribution of the environmental conditions of the region, or by some other aspect. those circumstances motivate further developments on the work; for instance, the analysis for bimodal seas and multiple loading conditions, and the continuous search for possibly more suitable estimation models. references arneson, i.b., brodtkorb, a.h., sørensen, a.j.: sea state estimation using quadratic discriminant analysis and partial least squares regression. ifac-papersonline 52(21), 72–77 (2019) article  google scholar  brodtkorb, a.h., nielsen, u.d., sørensen, a.j.: sea state estimation using vessel response in dynamic positioning. appl. ocean res. 70, 76–86 (2018) article  google scholar  cheng, x., li, g., ellefsen, a.l., chen, s., hildre, h.p., zhang, h.: a novel densely connected convolutional neural network for sea-state estimation using ship motion data. ieee trans. instrum. meas. 69(9), 5984–5993 (2020) article  google scholar  da silva bispo, i.b., simos, a.n., tannuri, e.a., da cruz, j.j., et al.: motion-based wave estimation by a bayesian inference method: a procedure for pre-defining the hyperparameters. in: the twenty-second international offshore and polar engineering conference. international society of offshore and polar engineers (2012) google scholar  de souza, f.l., tannuri, e.a., de mello, p.c., franzini, g., mas-soler, j., simos, a.n.: bayesian estimation of directional wave-spectrum using vessel motions and wave-probes: proposal and preliminary experimental validation. j. offshore mech. arctic eng. 140(4), 041102 (2018). https://doi.org/10.1115/1.4039263. issn 0892-7219 duz, b., mak, b., hageman, r., grasso, n.: real time estimation of local wave characteristics from ship motions using artificial neural networks. in: okada, t., suzuki, k., kawamura, y. (eds.) prads 2019. lnce, vol. 65, pp. 657–678. springer, singapore (2021). https://doi.org/10.1007/978-981-15-4680-8_45 chapter  google scholar  goodfellow, i., courville, a., bengio, y.: deep learning, vol. 1. mit press, cambridge (2016) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  hochreiter, s., schmidhuber, j.: long short-term memory. neural comput. 9(8), 1735–1780 (1997) article  google scholar  journée, j.m., massie, w.w.: offshore hydromechanics. delft university of technology, delft, the netherlands (2001) google scholar  lecun, y., bengio, y., et al.: convolutional networks for images, speech, and time series. handb. brain theor. neural netw. 3361(10), 1995 (1995) google scholar  ma, k.t., luo, y., kwan, c.t.t., wu, y.: mooring system engineering for offshore structures. gulf professional publishing, houston (2019) google scholar  newman, j.n.: marine hydrodynamics. the mit press, cambridge (1977) google scholar  nielsen, u.d.: estimations of on-site directional wave spectra from measured ship responses. mar. struct. 19(1), 33–69 (2006) article  google scholar  pecher, a., kofoed, j.p. (eds.): handbook of ocean wave energy. oeo, vol. 7. springer, cham (2017). https://doi.org/10.1007/978-3-319-39889-1 book  math  google scholar  simos, a.n., tannuri, e.a., da cruz, j.j., filho, a.n.q., da silva bispo, i.b., carvalho, r.c.: development of an on-board wave estimation system based on the motions of a moored fpso: commissioning and preliminary validation. in: international conference on offshore mechanics and arctic engineering, vol. 44922, pp. 259–270. american society of mechanical engineers (2012) google scholar  simos, a.n., tannuri, e.a., sparano, j.v., matos, v.l.: estimating wave spectra from the motions of moored vessels: experimental validation. appl. ocean res. 32(2), 191–208 (2010) article  google scholar  szegedy, c., et al.: going deeper with convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1–9 (2014) google scholar  tannuri, e.a., sparano, j.v., simos, a.n., da cruz, j.j.: estimating directional wave spectrum based on stationary ship motion measurements. appl. ocean res. 25(5), 243–261 (2003) article  google scholar  tu, f., ge, s.s., choo, y.s., hang, c.c.: sea state identification based on vessel motion response learning via multi-layer classifiers. ocean eng. 147, 318–332 (2018) article  google scholar  download references acknowledgments authors acknowledge petrobras for providing long-term support and motivation to this work. the authors also thank the center for artificial intelligence (c4ai-usp) and the support from the são paulo research foundation (fapesp grant #2019/07665-4) and from the ibm corporation. the first and second authors acknowledge the higher education personnel improvement coordination (capes) for the scholarship. the third author was supported in part by brazilian national council for scientific and technological development (cnpq) under grant 312180/2018-7. the last author acknowledges the cnpq for the research grant (310127/2020-3). author information authors and affiliations escola politécnica of university of são paulo, são paulo, brazil gustavo a. bisinotto, lucas p. cotrim, fabio gagliardi cozman & eduardo a. tannuri authors gustavo a. bisinottoview author publications search author on:pubmed google scholar lucas p. cotrimview author publications search author on:pubmed google scholar fabio gagliardi cozmanview author publications search author on:pubmed google scholar eduardo a. tannuriview author publications search author on:pubmed google scholar corresponding author correspondence to gustavo a. bisinotto . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper bisinotto, g.a., cotrim, l.p., cozman, f.g., tannuri, e.a. (2021). sea state estimation with neural networks based on the motion of a moored fpso subjected to campos basin metocean conditions. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_21 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords sea state estimation neural networks moored fpso metocean conditions publish with us policies and ethics profiles eduardo a. tannuri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature entity relation extraction from news articles in portuguese for competitive intelligence based on bert | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper entity relation extraction from news articles in portuguese for competitive intelligence based on bert conference paper first online: 28 november 2021 pp 449–464 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) entity relation extraction from news articles in portuguese for competitive intelligence based on bert download book pdf download book epub daniel de los reyes  orcid: orcid.org/0000-0003-0495-598210, douglas trajano  orcid: orcid.org/0000-0001-7801-893810, isabel harb manssour  orcid: orcid.org/0000-0001-9446-675710, renata vieira  orcid: orcid.org/0000-0003-2449-547711 & … rafael h. bordini  orcid: orcid.org/0000-0001-8688-990110  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1353 accesses 3 citations abstract competitive intelligence (ci) is a relevant area of a corporation and can support the strategic business area by showing those responsible, helping decision making on how to position an organization in the market. this work uses the bidirectional transformer encoding representations (bert) to process a sentence and its named entities and extract the parts of the sentences that represent or describe the semantic relationship between these named entities. the approach was developed for the portuguese language, considering the financial domain and exploring deep linguistic representations without using other lexical-semantic resources. the results of the experiments show a precision of 73.5% using the jaccard metric that measures the similarity between sentences. a second contribution of this work is the manually constructed dataset with more than 4.500 tuples (phrase, entity, entity) annotated. financially supported by the brazilian national council for scientific and technological development (capes) and the by portuguese foundation for science and technology (fct)under the projects ceecind/01997/2017, uidb/00057/2020. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others chinese named entity recognition method in history and culture field based on bert article open access 22 september 2021 divergent semantic integration (dsi): extracting creativity from narratives with distributional semantic modeling article open access 17 october 2022 semantic embedding-based entity alignment for cybersecurity knowledge graphs chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing machine translation natural language processing (nlp) market research and competitive intelligence 1 introduction the relation extraction (re) task aims to identify and classify the semantic relationships that occur between (pairs of) recognized entities in a given text. extracting relationships between named entities from text is a major challenge in information extraction (ie), given the necessary language knowledge and the sophistication of the language processing techniques employed. at the same time, this task can contribute to the development of several areas, such as question and answer systems, summarization, among others [7]. there is a growing interest in re, mainly motivated by the exponential growth of the information available on the web, which can make the task of searching and using such a large amount of data manually unfeasible. this context makes re an even more complex and relevant research area [10]. in relation to the financial market, a domain addressed in this work, the news brings information about sectors of the economy, industrial policies, acquisitions, and company partnerships, among others. automating the process of analyzing this data, in the form of financial reports, headlines, and corporate announcements can support personal and corporate economic decision making [30]. thus, for example, it is possible to extract an acquisition relationship between entities of the organization type, where a certain organization (the first entity) was purchased (relationship) by another organization (second entity) [20]. therefore, re research among financial entities is the basis for the automatic extraction of financial information that can be used to assist with individual economic activities and national economic decision making [30]. research on re for the english language is at a more advanced stage than for portuguese. while much work can be found in the literature on re for english [4, 12, 15, 16, 22, 25], very few papers focus on re for portuguese. unlike the english language, which has a larger number of datasets available to produce research, the portuguese language lacks this type of resource; in fact, we were unable to find any large dataset prepared for this task. considering that we had to build such a dataset for the validation of our experiments, it should be clear the difficulties for advancement in this area of research. there are numerous techniques aimed at the area of ie and among them, deep learning techniques have recently stood out, mainly due to their ability to discover patterns implicit in the data. the literature has presented deep learning algorithms such as transformers [23], recurrent neural networks [15, 24] and convolutional neural networks [28, 29] as good alternatives, as they have been efficiently applied in several sequential text processing tasks, including the re task. in this context, this work aims to propose an approach to the extraction of any semantic relationship between named entities (nes) in the financial market domain for the portuguese language. to achieve this goal, we use bert, a pre-trained model with the transformers [23] architecture, so it is possible to harness the power of bert and get sentence semantics without the use of enhanced feature selection or other external resources. for the validation of the experiment, we also created a dataset composed of 4,641 sentences and semantic relations annotated manually. therefore, the contributions of this work are: an entity relation extraction approach for the portuguese language in the financial context, based on bert. a corpus with more than 4,500 manually annotated tuples formed from financial market news, available on githubfootnote 1. this work is organized as follows. section 2 analyzes the context of automated information processing for the area of competitive intelligence (ci) and brings the concept about the transformers bert. section 3 discusses related work. section 4 provides a detailed description of the proposed solution. the experimental process and its results are detailed in sects. 5 and 6, respectively. finally, sect. 7 presents our conclusions, as well as possibilities for future work. 2 background this section presents a theoretical basis on the main concepts related to the work: financial market news processing as a problem addressed, presented in subsect. 2.1; bert, presented in subsect. 2.2. 2.1 competitive intelligence and news processing nowadays, the largest companies in the financial segment have a ci sector and, through it, information from different sources is strategically worked, allowing them to anticipate market trends and enabling the evolution of the business in relation to its competitors. this sector is usually formed by one or more professionals dedicated specifically to monitoring the competitors’ moves. ci is a structured survey model for unstructured facts and data analysis to support the company’s decision-making in its strategic planning. essentially, ci, involves the legal collection of information on competitors and the overall business environment. the knowledge gained from this information is then used to enhance the organization’s own competitiveness [26]. in times of competitiveness based on knowledge and innovation, ci allows companies to exercise pro-activity. the conclusions obtained in this process allow the company to know if it really remains competitive and if there is sustainability for its business model. ci can bring some advantages to companies that use it, such as: minimizing surprises from competitors, identifying opportunities and threats, obtaining relevant knowledge for the formulation of strategic planning, understanding the repercussions of their actions on the market, among others. the process of capturing information through news still requires a lot of manual effort. most of the information in the financial field appears in the form of free text, relying on manual processing of these data, far from keeping up with the growth rate of data, and unable to make full use of this information [30]. it often depends on a professional responsible for carefully reading numerous news about organizations to highlight possible market movements, and this professional can also keep this knowledge for himself. it is expected then that, with a system that automatically filters the relationships between financial market entities, the effort and time spent on those activities can be reduced. another benefit achieved is that this same system can feed business intelligence (bi) systems and establish a historical database with market events. thus, knowledge about market movements can be stored and organized independently of the particular ci analysts currently working in the company. 2.2 bert bert is a large pre-trained language model proposed by google [9] in 2018. since then, bert has achieved leading-edge results in various natural language processing (nlp) tasks [14, 27]. bert consists of a bidirectional multi-layer transformer encoder based on the original implementation described in vaswani et al. [23]. each layer has two sub-layers. the first is a multi-head self-attention mechanism, and the second is a simple, position-wise fully connected feed-forward network. residual connections are applied to both sub-layers. the model trains by combining two tasks: masked lm and next sentence prediction. the first, masked lm, involves masking 15% of the words in a given sentence, whose original value must then be predicted, considering the context of the sentence in which they appear. out of these 15%, 80% is replaced by the symbol [mask], 10% by a random word, and 10% is kept with the original word. in the bert training process, the model takes pairs of sentences as input and learns to predict whether the second sentence in the pair is the subsequent sentence in the original document. during training, 50% of the entries are pairs where the second sentence is the subsequent sentence in the original document, while in the other 50% a random sentence from the corpus is chosen as the second sentence. the assumption is that the random phrase will be disconnected from the first phrase. 3 related work with the advancement of information and communication technology in recent times, it is possible to store data from different sources and massively process it, which can facilitate research. due to its feasibility of use in several areas such as the one explored in this study, re has become the subject of numerous studies in the literature [2, 3, 13, 17, 30]. those studies differ in regards to the method used, the language of the trained dataset, and also the theme of the dataset itself. thus, this section presents some of these works, exploring some peculiarities. recently, deep learning algorithms have been targeted for application in a wide range of areas, such as nlp, image processing, bioinformatics, crowd simulation, and many others. in the area of nlp, for example, there is work on named entities recognition (ner) (e.g., [19]) and also the tool used in this study for ner, spacyfootnote 2, which is based on deep learning. several approaches have been proposed for extracting relationships from unstructured sources. for some languages, such as english, there is extensive research and literature [8]. to get around the problem of lack of memory for very large sequences in convolutional neural networks (cnn), some authors [12, 15, 16] adopted an approach using long short-term memory (lstm), which use memory cells for neural networks. in this direction, the work by qingqing li [15] uses recurrent neural networks bidirectional long short-term memory (bi-lstm), which are an extension of the traditional lstms for multitasking model, and presents a version with an attention mechanism that considerably improves the results in all tested datasets. more recent work in the literature [13, 17, 30] uses attention mechanisms to improve the performance of their neural network models. these mechanisms assist in the automatic information filtering step that helps to find the most appropriate sentence passage to describe the relationship between the named entities (nes). in [30], the authors also implement a model based on the recurrent neural network (rnn) bi-gru with an attention mechanism to focus on the most important assumptions of the sentences for the financial market. despite having great importance, the financial domain is little explored in the literature. the authors also created a corpus by manually collecting 3000 sentence records from major news sites, which was used for entity recognition and extraction of relationships such as learning and training as a whole. approaches to re for the portuguese language are still very limited and are often based on rules such as [2, 3]. such approaches applied simple heuristics that explore fragments of evidence of relationships between nes in texts, comprising lexical, syntactic, and semantic analysis, types of entities, and information from external sources. chaves [3] proposes an extraction system that deals with the ner task only for the local category and its relations. the work by batista et al. [1] proposes a distance-supervised re approach between two entities in portuguese-written wikipedia articles. more recently collovini et al. [5] developed a ci approach using re to collect and organize unstructured data in the portuguese language. the authors proposed a joint framework for named entity recognition and also their semantic relationships. for the entity recognition task, a neural network was used, and for the relation extraction, a model based in conditional random fields (crf) algorithm was used. this approach has shown good results on a small dataset, but it needs some additional lexical-semantic features of the sentences. most studies present re solutions for texts in english, thus it is possible to find a greater number of datasets in that language. there are few datasets available in the portuguese language, such as the golden collection harem. harem is a joint portuguese language assessment event, organized by linguateca [18]. the lack of this type of resource forces researchers to develop their own research corpora. in most cases, it is necessary to first create a set of sentences and manually annotate them when the classification is supervised in order to proceed with the re task. in addition, the lack of public datasets also makes it difficult to compare with related work fairly, as well as requiring more time and effort on the part of the researcher. it can also be noted that there is work related the re task which already uses machine learning techniques for this purpose. however, although we have found some papers on the re task, few of them are for the portuguese language, and only one of them is related to the financial context [5]. considering other languages, the work by zhou [30] has similar goals to ours but proposes a joint model for ner and classifies the type of relationship between such named entities. thus, it is clear that there is a gap in the literature for work that address such tasks using deep learning techniques and having portuguese as the main language, especially in the economic-financial context as addressed in this work. 4 relationship extractor model through the analysis of the records collected and used in the experiment, it is possible to note that the data from the financial field contain relationships composed of many tokens, in addition to the fact that the sentences can be long. there are many ways to express relationships, and the same semantics can be expressed in many ways. thus, the problem we address can benefit from the use of an attention mechanism. in this section, we present our model based on transformers [23] bert. we used the pre-trained model in portuguese based on the brwac corpus provided by souza et al. [21]. the implementations of this version of transformers are provided by huggingfacefootnote 3; we have added and trained the last layers of the model in order to extract a sub-sequence of the input text. as in fig. 1, the model architecture is composed of two main components, encoders and decoders. this approach provides a slightly different representation for the words as they are used in the sentences and their relationship to other words. this allows the model to understand the word itself and the context in which it is inserted. in general terms, the sentences enter through the encoder, which has two layers: an attention mechanism which then feeds into a feed forward neural network; the output of the encoder is then sent to the decoder. the decoder also has both those layers, but between them there is additionally an attention mechanism that helps focusing on the relevant parts of the input sentence. below, we describe in more detail some of these concepts. fig. 1. model architecture full size image attention mechanism: an attention mechanism can be described as mapping a query and a set of key-value pairs to an output, where the query, keys, values, and output are all vectors. as per eq. 1, the output is calculated as a weighted sum of values, where the weight assigned to each value is calculated by a query matching function with the corresponding key. in the bert pre-trained model, multi-head attention is used, which allows the model to serve together information from different representations in different positions. $$\begin{aligned} multihead(q,k,v) = concat(head_1, ..., head_h) w^o, \end{aligned}$$ (1) where $$\begin{aligned} head_i = attention(qw_1^q, kw_i^k, vw_i^v). \end{aligned}$$ (2) feed-forward layer: in addition to the attention mechanism, each of the layers in our encoders and decoders contains a fully connected feed-forward layer, which is applied to each position separately and identically. this consists of two linear transformations with a relu activation between them which are calculated according to eq. 3. $$\begin{aligned} ffn(x) = max(0, xw_1 + b_1)w_2 + b_2. \end{aligned}$$ (3) 5 experiments this section aims to evaluate the performance of the proposed model in experiments using the corpus for the financial domain. the proposed approach follows the knowledge discovery in databases (kdd) process created by fayyad [11]. this process contains five steps ranging from data collection and creation, as described in sect. 5.1, to the evaluation of results. the next sections aim to explain how each of these steps was employed in this work. 5.1 selection as in the selection step it is necessary to indicate which data will be used during the experiments for the re [11] task, we start by searching for corpora. however, there was no evidence of open datasets in the context of extracting relationships in the financial area in the portuguese language. therefore, for this work, a corpus with 4,641 manually annotated tuples was created. more than 1,500 financial market news from 2018 was provided by a partner company. those news articles were collected from various media outlets, such as financial market websites, newspapers, and corporate balance sheets. those articles gave rise to more than 10,000 sentences that were analyzed manually. another 7,097 tweets with dates from january 2021 were also collected. those tweets were selected from 10 communication media users focused on the financial market and the economy in general. the selected users were: infomoney, estadaoeconomia, uoleconomia, g1economia, oglobo_economia, folha_mercado, investingbrasil, leiamoneytimes, valoreconomico, br_economico. sentences that contained co-references were removed because handling them would require additional processing. 5.2 pre-processing the pre-processing step is intended to apply cleaning, correction, or removal procedures for inconsistent or missing data. in this work, we carried out a manual spell-checking process for each sentence, using excel’s spell-checker. acronyms have also been expanded, such as bc being replaced by banco central. standardization can be done manually, but in a real work scenario, this task becomes massive and can be automated by creating a base of named entities and their acronyms. thus, it is possible to design a process that validates the acronyms contained in the sentence and replaces them with their extensions or even an approach that focuses only on some specific entities informed by the ci analyst. in this same phase, the named entities are also identified, through a single ner tool, called spacyfootnote 4, ensuring that the same criteria were used for all sentences. the data cleaning process was done automatically through a script available at github (see footnote 1) that removes special characters and acronyms that follow the description itself. phrases that have only one recognized named entity or none at all are removed from the dataset. at the end of this cleaning step, just over 5,000 sentences were selected for the next step. the named entities in question are those related to the person, place, and organization categories, as they were considered the most relevant to the domain of organizations [6]. the focal point is information about organizations as well as their relationships with other organizations, people, and places. 5.3 transformation the data transformation or formatting step analyzes the data obtained from the previous step and reorganizes them in a specific way so that they can be analyzed and interpreted by the algorithm in the next step. in the case of this work, after identifying the named entities in the previous phase, the first step of the transformation step is to combine all entities present in the sentence in order to create a tuple (sentence, entity, entity) for each combination. it is important to emphasize that sentences with more than two entities generate more than one tuple for the same sentence, as it is necessary to create a pair for each combination of entities. then, the semantic relationships between the highlighted named entities are manually annotated. in this case, only positive tuples are considered and negative combinations are discarded. a tuple is considered positive when there is any semantic relationship between two named entities from the categories defined in sect. 5.2. finally, the two named entities are concatenated at the end of the sentence. finally, after removing sentences without the semantic relationship between the named entities, the corpus was left with a total of 4,641 records, containing a sentence, its entities, and the semantic relationship between them. table 1 exemplifies some records that have combinations of entities that can generate more than one tuple per sentence and also examples of tuples with positive annotations that contain relationships between named entities of the organization type. table 1. examples of positive tuples with annotations showing the relationships between named entities. the entities to be evaluated appear in bold font and the text representing the semantic relationship between them is underlined.full size table as sentences are naturally composed of words and characters, the transformation step of the methodology also includes the transformation of tokens into numerical representations by the bert encoder. as bert is a pre-trained model that expects input data in a specific format, this step also adds the special tokens [cls] and [sep] properly encoded in each sentence, finishing the transformation of the natural language sentence into the input for the extractor model. the special token [sep] is used to mark the end of a sentence or the separation between two sentences. the special token [cls] is used to mark the beginning of our text. 5.4 mining the mining step contemplates the prediction task, in which a behavioral pattern of the data is searched in order to predict the behavior of a future entity [11]. the corpus was randomly divided into two parts, 90% of which were used to train the model and 10% for testing; this proportion rate was chosen so that it would be possible to provide more samples for training the two models used in the approach. the first set was used so that the algorithm could search for the particular pattern in the data related to the relationship label. thus, after the training stage, from which the model is able to recognize this pattern, it is possible to apply it to the test set data. the adjustment of the bert hyper-parameters used was done using the combination of all the values indicated by jacob devlin when he proposed the new bert language model in [9]. in that work, jacob devlin used most hyper-parameters with default values, except for batch size, learning rate, and a number of training epochs. we also added two more test values, referring to the optimizer parameter. the dropout rate was always kept at 0.1. thus, the values analyzed for this task were: batch size: 16, 32; learning rate: 2e−5, 3e−5, 5e−5; epochs: 2, 3, 4, 5; optimizer: adamw; in the end, we ran a total of 24 experiments with all possible combinations of the parameters described above. after analyzing the results, the model that presented the best results was selected according to the parameters in table 2. table 2. combination of hyper-parameters that presented the best results.full size table 5.5 evaluation the last step of kdd methodology presented in this section aims to evaluate the performance of the model we built. the experimental evaluation was carried out by applying, in the test data, the model based on a deep neural network built in the learning stage based on the defined parameters. for the implementation of the neural network, the tensorflow kerasfootnote 5 library was used to adapt the last layer of bert. the evaluation metric used in this work is the jaccard similarity coefficient or simply jaccard coefficient. jaccard’s coefficient measures the similarity between two sets and is defined as the size of the intersection divided by the size of the union of the sample sets. we chose this metric as it can present a more realistic view of the model’s efficiency when recognizing the tokens present in the relationship. if we only considered completely correct extractions, the model would be penalized by not extracting just a token for example. equation 4 shows how the coefficient is calculated and table 3 an example of evaluation. $$\begin{aligned} jaccard(a, b) = \frac{|a \bigcap b|}{ |a\bigcup b|} \end{aligned}$$ (4) for the above two relations, we get a jaccard similarity of \(3 / (3 + 1) = 0.75\), which is the size of the intersection of the set divided by the total size of the set. table 3. evaluation example for extracting relationships between named entities. the evaluated named entities are in bold font.full size table 6 results and discussion after the training step, the model was applied to the test data set. in this evaluation step, the model obtained good results, indicating an overall jaccard metric of 73.5% for all the prediction cases. in terms of correctly extracted relationships, it was possible to extract 212 completely correct relationships from a total of 465 samples available for testing; that is when the extracted relationship is exactly the same as that manually annotated. we consider a relationship partially correct when the extracted relationship reaches a jaccard score of at least 50%. as for partially correct relationships, our approach was able to extract 176 relationships. in only 40 occurrences the model was not able to extract any token present in the relation to be extracted. these results are summarized in table 4. table 4. results obtained by the proposed model.full size table in general, the extracted relationships express information relevant to the ci focus, for example, partnership formation relationships between organizations; financial investment relationships between organizations; relations between people and organizations; entry of organizations into the market in certain locations, among others. table 5 illustrates instances of extracted relationships that are completely correct. from this re approach, it is possible to create a historical knowledge base, easily accessible and interpreted by ci analysts. thus, it is possible to provide information so that everyone can extract value quickly and clearly. table 5. completely correct relationships extracted by the proposed model.full size table fig. 2. average score grouped by the number of words contained in the relation to be extracted. full size image we claim that the model was able to respond well to the complexity of sentence length and the size of the relation to be extracted. it was able to recognize patterns and indicate which tokens belong to the semantic relations contained in the same sentence in the financial domain. figure 2 illustrates the average score according to sentence length. it can also be inferred that the model has greater difficulty when trying to extract relations composed of only one token, which ends up penalizing the model and its performance. this particular type of relationship is the third largest group within the test dataset, and in most cases there are only two score possibilities, 0 or 1. on these prediction errors, the model had more difficulty in inferring composite relationships just by a preposition (e.g. da, de, na, no) as shown in table 6. table 6. examples of relationships not extracted by the model. in bold and underlined, the semantic relationship to be extracted.full size table 7 conclusion and future work in this work, we provide an approach to the re task and a manually annotated corpus from news provided by a market intelligence company and also news collected from twitter on the subject of competitive intelligence. more than 1,500 news about the financial market and more than 7,000 tweets were selected to build a knowledge base. the model proved to be able to recognize semantic relationships and is therefore useful for discovering events and facts related to the financial market. from the section on related work, it is possible to note that there is little research on techniques for extracting relationships between named entities for the financial domain in portuguese. this domain lacks practical solutions, in part because of the large amount of information in the financial area, so manual analysis becomes difficult to meet the needs and make full use of this information. due to this lack of approaches to the re task, it is also difficult to present comparative results at this stage, since there is still no dataset available for comparison. as the main contributions of this work, we can mention the development of a relation extraction model between named entities based on bert, which replaces the explicit linguistic resources, required by other methods, such as rule-based approaches or that use parts-of-speech (pos). this approach becomes much simpler as it only needs the phrase information and the pair of concatenated entities. thus, it allows more than one entry to be sent, as a sentence can have n pairs of named entities. therefore, the approach adopted allows us to infer that the sentence and the pair of entities are sent separately. another contribution of this work is found in the development of a large corpus related to the financial market, with manually annotated text from tweets and news provided by ci analysts to support decision making. the results demonstrate that the approach used has reasonable results, reaching a jaccard score of 73.5%. this score is interesting since the model was able to extract relationships of different sizes. as shown in sect. 6 it was more penalized when extracting relations from just a token, usually formed by prepositions. as future work, a method in pipeline format containing two models for extracting relationships between named entities will be developed. the first model will determine whether an entry containing the phrase and entity pair has a semantic relationship between the named entities to be tested. the output of this model, when it indicates that there is a semantic relationship, will be the input of the model here presented, which will highlight the parts of the sentences that represent or describe the relationship between the selected named entities. notes 1.https://github.com/danielreeyes/financial-market-corpus. 2.https://spacy.io/. 3.https://github.com/huggingface/transformers. 4.https://spacy.io/. 5.https://www.tensorflow.org/. references batista, d.s., forte, d., silva, r., martins, b., silva, m.: extracçao de relaçoes semânticas de textos em português explorando a dbpédia e a wikipédia. linguamatica 5(1), 41–57 (2013) google scholar  bruckschen, m., de souza, j.g.c., vieira, r., rigo, s., mota, c., santos, d.: sistema serelep para o reconhecimento de relaçoes entre entidades mencionadas (2008) google scholar  chaves, m.: geo-ontologias e padrões para reconhecimento de locais e de suas relações em textos: o sei-geo no segundo harem. cristina mota; diana santos (ed) desafios na avaliação conjunta do reconhecimento de entidades mencionadas: o segundo harem linguateca 2008 (2008) google scholar  cheng, w., xiong, j.: entity relationship extraction based on bi-channel neural network. in: 2020 2nd international conference on machine learning, big data and business intelligence (mlbdbi), pp. 349–352. ieee (2020) google scholar  collovini, s., gonçalves, p.n., cavalheiro, g., santos, j., vieira, r.: relation extraction for competitive intelligence. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 249–258. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_24 chapter  google scholar  collovini, s., grando, f., souza, m., freitas, l., vieira, r.: semantic relations extraction in the organization domain. in: proceedings of iadis international conference on applied computing, rio de janeiro, pp. 99–106 (2011) google scholar  collovini, s., pugens, l., vanin, a.a., vieira, r.: extraction of relation descriptors for portuguese using conditional random fields. in: bazzan, a.l.c., pichara, k. (eds.) iberamia 2014. lncs (lnai), vol. 8864, pp. 108–119. springer, cham (2014). https://doi.org/10.1007/978-3-319-12027-0_9 chapter  google scholar  cruz, c.g.a., weitzel, l.: evaluation of relation extraction systems for portuguese language pt-br. in: 2018 13th iberian conference on information systems and technologies (cisti), pp. 1–6. ieee (2018) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. preprint arxiv:1810.04805 (2018) etzioni, o., fader, a., christensen, j., soderland, s., et al.: open information extraction: the second generation. in: twenty-second international joint conference on artificial intelligence, pp. 3–10 (2011) google scholar  fayyad, u., piatetsky-shapiro, g., smyth, p.: from data mining to knowledge discovery in databases. ai mag. 17(3), 37 (1996) google scholar  florez, e., precioso, f., pighetti, r., riveill, m.: deep learning for identification of adverse drug reaction relations. in: proceedings of the 2019 international symposium on signal processing systems, pp. 149–153 (2019) google scholar  gan, t., gan, y., he, y.: subsequence-level entity attention lstm for relation extraction. in: 2019 16th international computer conference on wavelet active media technology and information processing, pp. 262–265. ieee (2019) google scholar  han, x., wang, l.: a novel document-level relation extraction method based on bert and entity information. ieee access 8, 96912–96919 (2020) article  google scholar  li, q., et al.: a multi-task learning based approach to biomedical entity relation extraction. in: 2018 ieee international conference on bioinformatics and biomedicine (bibm), pp. 680–682. ieee (2018) google scholar  pandey, c., ibrahim, z., wu, h., iqbal, e., dobson, r.: improving rnn with attention and embedding for adverse drug reactions. in: proceedings of the 2017 international conference on digital health, pp. 67–71 (2017) google scholar  qin, p., xu, w., guo, j.: designing an adaptive attention mechanism for relation classification. in: 2017 international joint conference on neural networks (ijcnn), pp. 4356–4362. ieee (2017) google scholar  santos, d., cardoso, n.: reconhecimento de entidades mencionadas em português: documentação e actas do harem, a primeira avaliação conjunta na área (2007) google scholar  santos neto, j.f.d, et al.: reconhecimento de entidades nomeadas para o português usando redes neurais (2019) google scholar  sarawagi, s.: information extraction. now publishers inc. (2008) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, october 20–23 (to appear) (2020) google scholar  tang, y.: an extended sequence labeling approach for relation extraction. in: 2019 ieee international conference on power, intelligent computing and systems (icpics), pp. 121–124. ieee (2019) google scholar  vaswani, a., et al.: attention is all you need. preprint arxiv:1706.03762 1 (2017) wang, c., wang, y., mo, j., wang, s.: end-to-end relation extraction based on part of speech syntax tree. in: 2020 2nd international conference on machine learning, big data and business intelligence (mlbdbi), pp. 5–9. ieee (2020) google scholar  wang, y., sun, y., ma, z., gao, l., xu, y., wu, y.: a method of relation extraction using pre-training models. in: 2020 13th international symposium on computational intelligence and design (iscid), pp. 176–179. ieee (2020) google scholar  weiss, a.: a brief guide to competitive intelligence: how to gather and use information on competitors. bus. inf. rev. 19(2), 39–47 (2002) google scholar  xue, k., zhou, y., ma, z., ruan, t., zhang, h., he, p.: fine-tuning bert for joint entity and relation extraction in chinese medical text. in: 2019 ieee international conference on bioinformatics and biomedicine (bibm), pp. 892–897. ieee (2019) google scholar  yin, b., sun, y., wang, y.: entity relation extraction method based on fusion of multiple information and attention mechanism. in: 2020 ieee 6th international conference on computer and communications (iccc). ieee (2020) google scholar  yu, h., cao, y., cheng, g., xie, p., yang, y., yu, p.: relation extraction with bert-based pre-trained model. in: 2020 international wireless communications and mobile computing (iwcmc), pp. 1382–1387. ieee (2020) google scholar  zhou, z., zhang, h.: research on entity relationship extraction in financial and economic field based on deep learning. in: 2018 ieee 4th international conference on computer and communications (iccc), pp. 2430–2435. ieee (2018) google scholar  download references author information authors and affiliations pontifical catholic university of rio grande do sul, pucrs, school of technology, porto alegre, brazil daniel de los reyes, douglas trajano, isabel harb manssour & rafael h. bordini cidehus, university of évora, évora, portugal renata vieira authors daniel de los reyesview author publications search author on:pubmed google scholar douglas trajanoview author publications search author on:pubmed google scholar isabel harb manssourview author publications search author on:pubmed google scholar renata vieiraview author publications search author on:pubmed google scholar rafael h. bordiniview author publications search author on:pubmed google scholar corresponding author correspondence to daniel de los reyes . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper reyes, d.d.l., trajano, d., manssour, i.h., vieira, r., bordini, r.h. (2021). entity relation extraction from news articles in portuguese for competitive intelligence based on bert. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_31 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords competitive intelligence entity relation classification relation extraction publish with us policies and ethics profiles renata vieira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature beyond audio signals: generative model-based speaker diarization in portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper beyond audio signals: generative model-based speaker diarization in portuguese conference paper first online: 30 january 2025 pp 247–259 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) beyond audio signals: generative model-based speaker diarization in portuguese download book pdf download book epub antônio oss boll  orcid: orcid.org/0009-0004-5440-61269, letícia maria puttlitz  orcid: orcid.org/0009-0002-0313-10179, heloísa oss boll  orcid: orcid.org/0000-0001-8121-200210 & … rodrigo mor malossi  orcid: orcid.org/0000-0002-3100-007610  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 499 accesses abstract speaker diarization, the task of automatically identifying different speakers in audio and video, is frequently performed using probabilistic models and deep learning techniques. however, existing methods usually rely on direct analysis of the audio signal, which presents challenges for languages that lack established diarization methodologies, such as portuguese. in this article, we propose a new approach to speaker diarization that leverages generative models for automatic speaker identification in portuguese. we employed two generative models: one for refining the transcribed audio and another for performing the diarization task, as well as a model for initially transcribing the audio. our method simplifies the diarization process by capturing and analyzing speaker style patterns from transcribed audio and achieves high accuracy without depending on direct signal analysis. this approach not only increases the effectiveness of speaker identification but also extends the usefulness of generative models to new domains. it opens a new perspective for diarization research, especially for the development of accurate systems for under-researched languages in audio and video applications. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others revolutionizing speaker recognition and diarization: a novel methodology in speech analysis article 20 january 2025 recurrent neural network based speaker change detection from text transcription applied in telephone speaker diarization system chapter © 2018 unraveling the techniques for speaker diarization chapter © 2024 1 introduction diarization is the task of automatically identifying different speakers in speech, whether from audio or video [21]. it is based on two main processes: the recognition of who is speaking and the initial segmentation of the speech. its applications are wide-ranging, including meeting transcriptions [3], call center assistance [5], and the development of personalized voice assistants [23]. traditional methods for diarization employ probabilistic models, such as gaussian mixture models (gmm) and hidden markov models (hmm), which are effective at clustering audio according to voice patterns [12, 25]. subsequently, other techniques based on deep learning (dl) have brought significant advances to the field. examples include recurrent neural networks (rnns), which are adapted for sequential data [29], and convolutional neural networks (cnns), which are capable of analyzing audio directly from spectrograms [15]. most recently, methods based on the transformer architecture have demonstrated state-of-the-art performance across various domains [27, 28]. this success is largely attributed to self-attention, a mechanism that allows the model to focus on different parts of the input sequence when producing output while also capturing long-range dependencies in the input, regardless of their distance in the sequence. large language models (llms) are built upon this architecture, utilizing very deep neural networks with stacked multi-head attention layers. models such as gpt-4 [19], llama 3 [1], and claude [4], exhibit impressive capabilities in natural language understanding and in solving complex tasks via text generation. despite these advancements, there remains a gap in applying such approaches to audio and video tasks. exploring the use of transformer-based architectures for diarization presents a promising opportunity for achieving similar breakthroughs in audio and video processing. in addition, language variations are also a challenge in diarization tasks. most models are adapted to processing english content due to the abundance of materials in the language. although portuguese is a high-resource language for text-based tasks, it is considerably under-researched in audio and video applications, especially regarding models, transcription tools, and new diarization methods. in this article, we investigate existing diarization methods and introduce a novel approach that addresses identified gaps in the field, leveraging generative models for automatic speaker identification in portuguese videos and audio recordings. unlike traditional methods that require direct analysis of audio signals, our approach captures and analyzes speaker style patterns from transcribed audio with high accuracy. this method not only simplifies the process but also extends the applicability of generative models beyond their traditional domains, opening up a promising new line of research. furthermore, our approach is language-agnostic and can be extended to any language supported by both a generative language model and a transcription model. 2 background 2.1 speech data speech data contains a variety of information that reflects the identity of the speaker, including distinctive characteristics of the vocal tract, excitation source, and behavioral features. these cues are important for speaker recognition [13] and form the basis for the features of a speaker. in our approach, the model identifies the speaker’s traits from the text. processing such information involves a set of challenges, such as high variability due to factors like accent, background noise, and recording quality [24]. furthermore, the audio quality can be significantly affected when not recorded in controlled environments, such as outdoor settings, rooms with echo, or audio with simultaneous speech. our approach aims to mitigate these issues by employing advanced deep learning-based preprocessing techniques for audio transcription and refinement, as detailed in sect. 4. another issue is labeling, as accurate annotation of speech data is crucial but can be demanding and costly. techniques like large-scale weak supervision are employed to address these challenges [24]. in our case, we used the labeling from the brazilian chamber of deputies as a foundation, manually correcting some of its errors for better accuracy. the process is detailed in subsect. 4.1. finally, considering that handling speech data must comply with privacy regulations and ethical guidelines due to its sensitive nature [18], it is important to note that we used already transcribed examples of sessions that are required to be public. hence, the data already complied with the expected privacy standards. 2.2 speech to text speech to text (stt) is the process of converting speech signals to a sequence of words. the main technique based on stt is speech recognition [13]. there are several speech recognition models available, such as whisper large v2 [24], deepgram nova-2, amazon, microsoft azure batch v3.1, and the universal speech model [30]. in the present work, whisper large v2 was employed to transcribe public sections from the brazilian chamber of deputies. the whisper model, a state-of-the-art method for speech transcription, is a pre-trained model designed for automatic speech recognition (asr) and speech translation. it was trained on 680,000 h of labeled speech data, utilizing large-scale weak supervision for annotation. in its process, all audio is resampled to 16,000 hz and converted into an 80-channel log-magnitude mel spectrogram. whisper encoder handles this input representation, whereas the decoder utilizes learned position embeddings and tied input-output token representations. it employs an encoder-decoder transformer architecture, and both the encoder and decoder have the same width and number of transformer blocks. the architecture was trained to handle multiple tasks simultaneously, including voice activity detection and inverse text normalization. stt approaches have influenced numerous fields. in medicine, these models can document interactions between patients and doctors [9]. for customer service, stt enables real-time transcription of customer interactions, resulting in enhanced service quality [16]. additionally, stt offers tools for individuals with disabilities, improving accessibility [7]. in our study, we extended the application of stt to the political domain, focusing on diarizing speeches from the chamber of deputies. furthermore, transcription captures the spoken words without distinguishing between speakers. our goal task, diarization, goes beyond transcription by segmenting speech according to different speakers. 2.3 diarization diarization involves assigning audio segments to the appropriate speaker. it can be accomplished through various techniques, but typically involves voice activity detection (vad) to identify speech regions and clustering these segmented audio portions to associate them with individual speakers [8]. diarization also often includes speaker embedding extraction, where speaker characteristics are encoded into numerical representations. these embeddings are then utilized in the clustering process to group audio segments based on speaker similarity [22]. our methodology is an innovative approach to solving the challenges of the diarization task. it uses audio transcription and advanced prompt engineering steps, which are detailed in sect. 4. 2.4 large language models large language models (llms) based on transformers [27] are an essential part of our study, as they were used for refining the transcriptions and identifying the speakers, as detailed in sect. 4. these very deep neural networks are trained on extensive datasets, enabling them to learn complex patterns. each token within the model, whether it represents a whole word or part of one, is interpreted as a numerical embedding that encapsulates its semantic meaning [17]. similar words are mapped to embeddings that are close to each other in a high-dimensional space, facilitating the model’s ability to comprehend semantic relationships. the breadth and quality of the training dataset influence the model’s knowledge and vocabulary. therefore, a vast and high-quality training dataset is essential to ensure the model’s comprehension and accuracy. 3 related work there are several diarization models that support the english language, including pyannote [8]. pyannote works by implementing tasks such as voice activity detection (vad), speaker change detection (scd), and overlapped speech detection (osd), which assign a value of 1 if the event occurs and 0 otherwise. additionally, re-segmentation is used to refine the boundaries and labels of speech segments, assigning a label of 0 when no speaker is active and the corresponding speaker’s label when they are speaking. google also offers a model widely used for speech diarization on its cloud platform, the speech-to-text api. both versions, v1 and the recently introduced v2, can be used for this task in english. unfortunately, all aforementioned tools lack support for the portuguese language. therefore, we explored other multilingual options, such as oracle speech ai or assembly ai, which we tested and compared against our approach. the oracle speech ai service [20] offers a range of functionalities, including transcriptions utilizing the whisper model, which supports over 50 languages. additionally, it provides features such as text normalization, profanity filtering, and confidence scoring per word. moreover, it has the ability to recognize between 2 and 16 distinct speakers. however, the lack of detailed information about the training of their diarization model makes it difficult to understand the internal mechanisms of the service. on the other hand, the assembly ai service [6] also includes stt functionalities such as speech recognition and diarization. the speech recognition models supported by this service are best and nano. however, there is also not much information available about the methodology employed in their diarization process. 4 materials and methods 4.1 dataset as mentioned, there are very few public videos with identified speakers in portuguese. hence, the resource used in our study was one of the few available, obtained from the “ordem do dia” part of the sessions of the brazilian chamber of deputies [10]. given the source’s relevance, the data is from a trusted source, which makes it suitable for evaluating our method. another reason why this dataset was chosen is due to its sudden changes in speakers. while a number of videos have speaker identification, most of these instances are singular. this means that when one person speaks, there is usually a pause before another person responds. on the other hand, our dataset does not follow a specific order when people are speaking; anyone can interrupt and start talking at any time. this characteristic makes it ideal for our purposes, as we want to develop a method that can skillfully handle complex conversational dynamics. it is worth mentioning that some video transcripts may not be entirely accurate, as some instances may not follow the exact order of the speakers, having slightly different lines from the original ones. for example, in the session entitled “2nd ordinary legislative session of the 57th legislature, 109th session”, at the 13:56 mark, the audio says: “pelo rio de janeiro no rio de janeiro, nós vamos ouvir ela, que foi senadora, governadora, deputada federal, que é a nossa deputada.” the transcript’s meaning remains the same, but with some new words and is organized differently: “pelo rio de janeiro com a ex-senadora, ex-governadora e deputada federal.” given the inconsistencies between the actual audio and the written transcription, we used each of whisper’s transcriptions as the ground truth. we then manually inserted the real speakers and their labels. that resulted in a better evaluation of our method and ensured that the transcriptions were adequate. despite these discrepancies, our final goal was speaker diarization. therefore, we focused solely on identifying the speakers rather than ensuring perfect correspondence between the audio and transcripts, even though we manually inserted the speaker labels. 4.2 diarization strategy our approach involves two key stages: transcribing audio from videos and processing the resulting text with generative models to identify the speakersfootnote 1. we utilized whisper large-v2 [24] to transcribe eight parts of sessions from the brazilian chamber of deputies held in 2024. each part of the session lasted for 4 min, providing substantial audio data for analysis. for text refinement and speaker diarization, we employed the advanced gpt-4o generative model. furthermore, we developed a prompt engineering method to guide the models in both refining the transcriptions and generating the speaker predictions, aiming to refine the output and ensure speaker identification accuracy. fig. 1. pipeline of the proposed architecture. gm = generative model. full size image an overview of the process can be found in fig. 1. it begins by transcribing the video with whisper, and its output is then fed into a generative model. the model is then instructed to refine the text, removing misspelled words and correcting misplaced or contextually incorrect words. this refinement is achieved through prompt engineering, through which a detailed instructional prompt guides the model to make such changes and improve the quality of the text generated by whisper. finally, the refined text is passed on to another generative model to identify the speaker in the diarization task. although this second model’s architecture is the same as the model used for refinement, it is given a more robust prompt, allowing it to accurately identify the speaker. each speaker presents not only variations in their voice but also in how they communicate, using unique vocabulary and expressions. when applying a generative model to perform diarization on a transcribed text, it takes into account the individual characteristics of each speaker. additionally, the model leverages the knowledge acquired during training to identify patterns in the dialogue structure, such as recognizing that a question followed by an answer typically indicates that one person asked the question and another provided the response. 4.3 prompts generative models can perform poorly if they are not given precise instructions on the task they need to solve. therefore, this work also developed a prompt engineering strategy to ensure robustness. the task instructions used in this work are specified in figs. 2 and 3 for each generative model, respectively, text refinement and speaker identification. the specific prompts are available on our github. each of these tasks is further elaborated in the following subsections. fig. 2. design of the first prompt for the text refinement model. full size image text refinement. as mentioned, the first model functions as a “text refiner”. the prompt used for this task is detailed in fig. 2. it is important to firstly include the context in which the model will operate. this is specified in the ‘model role’, where the model is assigned a function or task at a higher level. subsequently, it requires instructions regarding the data that needs refinement. in our example, we explained the context of the video, instructing the model to act as a stenographer of sessions in the chamber of deputies and detailing the type of content of the transcript. in addition, we asked it to output an improved text, correcting portuguese errors and editing wrong words. for this step, to ensure consistent and accurate word correction, it is essential that the model has extended deterministic characteristics, avoiding variations in its responses. we therefore set the temperature to zero to eliminate any chance of variability. with both of these settings, the generative model could be effectively guided to provide the necessary text refinement. fig. 3. design of the second prompt for the text refinement model. full size image speaker diarization. the second model developed is significantly more complex. the prompt used for the diarization task includes almost all the refinement instructions involved in the first model, but with the addition of steps, results, and examples, providing much more detailed guidance. the prompt used in this task is detailed in fig. 3. specifically, the diarization prompt consists of three segments. firstly, an example of a conversation relevant to the task is provided. in our case, we simulated a dialog between individuals in a courtroom setting. if a different context is needed, a more contextually relevant example could be employed. for example, in a restaurant scenario, it could involve a conversation between a waiter, a customer, and the restaurant owner. the second segment comprises the transcript itself, which serves as the base text for identifying the speaker. finally, the third segment includes a set of instructions that the model must follow to perform the diarization task. they include both general instructions about the task and specific instructions for each of the potential speakers. with these three segments, we can achieve state-of-the-art performance in speaker identification tasks. another essential factor that influences the model’s prediction is the temperature setting. unlike the first model, we made this one slightly less deterministic since it needs to “understand” and accurately predict the number of speakers. we, therefore, increased the temperature and sacrificed some of the quality of the response at each iteration. however, this compensation significantly improves performance, with the model obtaining nearly perfect responses every two iterations. 4.4 evaluation there are several methods to evaluate speaker diarization, including the diarization error rate (der) [11], word error rate (wer), jaccard error rate (jer) [26], and e-wer [2], among others, such as the extensions discussed by galibert [14]. however, since our approach primarily uses text, we focused on evaluating the quality of diarization as a text classification method. in this work, we used accuracy and f1-score as our primary metrics for assessing the quality of speaker prediction. accuracy measures how often speakers are correctly labeled compared to the actual speaker for each sentence or word, reflecting the overall correctness of the classification across all speakers. the f1-score, being the harmonic mean of recall and precision, considers each correctly classified speaker. we specifically used the weighted f1-score to account for any imbalance among speakers. we evaluated these metrics at both the sentence level and the word level, comparing speaker predictions with the ground truth for each phrase and word, respectively. this separation between levels was implemented to enhance the accuracy of classifications. evaluating only at the sentence level may allow for multiple speakers within a sentence without significant penalties. therefore, we incorporated word-level evaluation to address this issue and provide a more granular assessment. to assess the variability and robustness of the models, we also utilized confidence intervals to statistically demonstrate these differences. additionally, we employed the wilcoxon signed-rank test to show that the differences between models are statistically significant. in this test, we specified the “greater” mode, focusing on superior performance rather than using a two-sided test. this approach ensured that our evaluation emphasized the best-performing model. 5 results our main results are shown in table 1. it displays the average accuracy and f1-score for the proposed method and the baseline models, as well as the respective 95% confidence intervals. table 1. comparison of our approach and baseline models for diarization.full size table the oracle model exhibited the lowest performance, with both accuracy and f1-score consistently around 0.48 for both sentence-level and word-level evaluations. the assembly model performed moderately well, with scores of approximately 0.81, yet it still fell short when compared to our model’s performance. our model demonstrated a significant improvement in accuracy, achieving 0.95 and 0.98 for sentence-level and word-level evaluations, respectively. this represents a substantial increase over both the oracle and assembly models. the improvement is similarly reflected in the f1-scores, where our model achieved 0.96 and 0.98 for sentence-level and word-level evaluations, respectively. additionally, the evaluation of the confidence intervals reveals that our approach exhibits the lowest variability, indicating high precision and consistency in the results. this suggests that our method is both robust and effective, especially when compared to the other models, which show greater variability in their confidence intervals. the superior performance of our method across both evaluation metrics indicates that it generalizes well across different audio types, performing effectively on both high and low conversational recordings within our dataset. to further validate these differences, we performed a wilcoxon signed-rank test to compare the results across the different models. the outcomes of this test are presented in table 2. table 2. comparison of our approach and baseline models using the wilcoxon signed-rank test.full size table as shown in table 2, all results are statistically significant, providing strong evidence of differences between the models. the wilcoxon test results allow us to state with 95% confidence that our model performs significantly better than both the assembly model and the oracle model at both the sentence and word levels, in terms of both accuracy and f1-score. additionally, our method has the unique ability to identify speakers based on their roles within a simple hierarchy (e.g., president and deputies). this functionality could be integrated into speaker-identified transcription tasks if required, though it would necessitate a robust example for effective implementation. all transcriptions and predictions generated by our model are available on our github. 5.1 limitations our model is aligned with the whisper-v2 model, which can transcribe audio segments up to 25 mb in size. consequently, our diarization method achieves state-of-the-art results only for audio segments within this size limit. although enhancing the model’s robustness is a potential area for future work, this study operates within the specified constraint. furthermore, since we leveraged generative models, the outputs may slightly vary with each prompt. we recommend iterating the process a few times to ensure the effectiveness of the task. 6 conclusion in this work, we introduced a new method for the diarization of audio and video. our approach leverages the power of large language models to excel in this task, particularly in languages that lack established diarization methodologies. by prioritizing an understanding of textual context and cues over traditional audio signal-based speech diarization techniques, we achieved state-of-the-art results for the portuguese language. our primary objective was to compare our method with other available methods for portuguese, such as the oracle model and assemblyai. the results demonstrated that our method significantly outperformed baseline methods, achieving higher accuracy and f1-scores (0.95 and 0.98 for sentence-level and 0.96 and 0.98 for word-level evaluations). additionally, the narrow confidence intervals indicated high precision and consistency. the statistical significance confirmed by the wilcoxon signed-rank test further supported the superior performance of our model. moreover, its ability to identify speakers based on their roles within a hierarchy, such as distinguishing between a president and deputies, adds another layer of functionality that could be integrated into more complex diarization tasks. for future work, we aim to investigate its behavior on longer audio segments and expand its evaluation by validating it on a wider variety of data. additionally, we see potential in extending it to other languages, especially those lacking well-established diarization tools, such as spanish and italian, and including multilingual evaluations in our comparisons. furthermore, we aim to adapt the method for real-time applications and explore the integration of retrieval-augmented generation (rag). by connecting the generative model to a website, rag could enable the model to selectively search for relevant examples and content. in addition, we plan to perform ablation studies to better understand the contributions of each component within the proposed framework. finally, llms, with their vast parameter-rich architectures, hold the potential to tackle a wide array of challenges. in this sense, we believe this work aligns with the emerging trend of leveraging generative models for tasks beyond text, extending into areas such as video, audio, and beyond. notes 1.in this article, we focus on diarizing videos; however, the process can be equally applied to audio inputs. references ai, m.: llama 3 (2024). https://llama.meta.com/. accessed 28 aug 2024 ali, a., renals, s.: word error rate estimation for speech recognition: e-wer. in: gurevych, i., miyao, y. (eds.) proceedings of the 56th annual meeting of the association for computational linguistics (volume 2: short papers), pp. 20–24. association for computational linguistics, melbourne, australia (2018). https://doi.org/10.18653/v1/p18-2004, https://aclanthology.org/p18-2004 anguera, x., bozonnet, s., evans, n., fredouille, c., friedland, g., vinyals, o.: speaker diarization: a review of recent research. ieee trans. audio speech lang. process. 20(2), 356–370 (2012) article  google scholar  anthropic: claude ai (2024). https://www.anthropic.com/claude. accessed 28 aug 2024 aronowitz, h.: speaker diarization using a priori acoustic information. in: interspeech, pp. 937–940 (2011) google scholar  assemblyai: assemblyai (2024). https://www.assemblyai.com/. accessed 30 aug 2024 bain, k., basson, s., faisman, a., kanevsky, d.: accessibility, transcription, and access everywhere. ibm syst. j. 44(3), 589–603 (2005) article  google scholar  bredin, h., et al.: pyannote. audio: neural building blocks for speaker diarization. in: icassp 2020-2020 ieee international conference on acoustics, speech and signal processing (icassp), pp. 7124–7128. ieee (2020) google scholar  chiu, c.c., et al.: speech recognition for medical conversations. arxiv preprint arxiv:1711.07274 (2017) câmara dos deputados do brasil: discursos e notas taquigráficas (2024). https://www.camara.leg.br/internet/sitaqweb/discursodireto.asp. accessed 10 jun 2024 fiscus, j.g., ajot, j., michel, m., garofolo, j.s.: the rich transcription 2006 spring meeting recognition evaluation. in: renals, s., bengio, s., fiscus, j.g. (eds.) mlmi 2006. lncs, vol. 4299, pp. 309–322. springer, heidelberg (2006). https://doi.org/10.1007/11965152_28 chapter  google scholar  fox, e.b., sudderth, e.b., jordan, m.i., willsky, a.s.: the sticky hdp-hmm: bayesian nonparametric hidden markov models with persistent states. arxiv preprint 2 (2007) google scholar  gaikwad, s.k., gawali, b.w., yannawar, p.: a review on speech recognition technique. int. j. comput. appl. 10(3), 16–24 (2010) google scholar  galibert, o.: methodologies for the evaluation of speaker diarization and automatic speech recognition in the presence of overlapping speech. in: interspeech (2013). https://doi.org/10.21437/interspeech.2013-303 hrúz, m., zajíc, z.: convolutional neural network for speaker change detection in telephone speaker diarization system. in: 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 4945–4949 (2017). https://doi.org/10.1109/icassp.2017.7953097 meng, j., zhang, j., zhao, h.: overview of the speech recognition technology. in: 2012 fourth international conference on computational and information sciences, pp. 199–202. ieee (2012) google scholar  mikolov, t., chen, k., corrado, g., dean, j.: efficient estimation of word representations in vector space. arxiv preprint arxiv:1301.3781 (2013) nautsch, a., et al.: preserving privacy in speaker and speech characterisation. comput. speech lang. 58, 441–480 (2019) article  math  google scholar  openai: gpt-4o (2023). https://openai.com/index/hello-gpt-4o/. accessed 28 aug 2024 oracle corporation: oracle ai speech (2024). https://www.oracle.com/artificial-intelligence/speech/. accessed 30 aug 2024 park, t.j., kanda, n., dimitriadis, d., han, k.j., watanabe, s., narayanan, s.: a review of speaker diarization: recent advances with deep learning. arxiv preprint arxiv:2101.09624 (2021) park, t., koluguri, n.r., jia, f., balam, j., ginsburg, b.: nemo open source speaker diarization system. in: interspeech, pp. 853–854 (2022) google scholar  ponraj, a.s., et al.: speech recognition with gender identification and speaker diarization. in: 2020 ieee international conference for innovation in technology (inocon), pp. 1–4. ieee (2020) google scholar  radford, a., kim, j.w., xu, t., brockman, g., mcleavey, c., sutskever, i.: robust speech recognition via large-scale weak supervision. in: international conference on machine learning, pp. 28492–28518. pmlr (2023) google scholar  reynolds, d.a.: speaker identification and verification using gaussian mixture speaker models. speech commun. 17(1), 91–108 (1995). https://doi.org/10.1016/0167-6393(95)00009-d, https://www.sciencedirect.com/science/article/pii/016763939500009d ryant, n., et al.: the second dihard diarization challenge: dataset, task, and baselines. arxiv preprint arxiv:1906.07839 (2019) vaswani, a., et al.: attention is all you need. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017). https://proceedings.neurips.cc/paper_files/paper/2017/file/3f5ee243547dee91fbd053c1c4a845aa-paper.pdf wang, d., xiao, x., kanda, n., yoshioka, t., wu, j.: target speaker voice activity detection with transformers and its integration with end-to-end neural diarization. arxiv preprint arxiv:2208.13085 (2022) weninger, f., wöllmer, m., schuller, b.: automatic assessment of singer traits in popular music: gender, age, height and race. in: proceedings 12th international society for music information retrieval conference, ismir 2011 (2011) google scholar  zhang, y., et al.: google usm: scaling automatic speech recognition beyond 100 languages. arxiv preprint arxiv:2303.01037 (2023) download references author information authors and affiliations institute of mathematics and statistics, university of sao paulo, rua do matao, 1010 cidade universitaria, sao paulo, sp, 05508-090, brazil antônio oss boll & letícia maria puttlitz institute of informatics, universidade federal do rio grande do sul, av. bento gonçalves, 9500, porto alegre, rs, 91501-970, brazil heloísa oss boll & rodrigo mor malossi authors antônio oss bollview author publications search author on:pubmed google scholar letícia maria puttlitzview author publications search author on:pubmed google scholar heloísa oss bollview author publications search author on:pubmed google scholar rodrigo mor malossiview author publications search author on:pubmed google scholar corresponding author correspondence to antônio oss boll . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper boll, a.o., puttlitz, l.m., boll, h.o., malossi, r.m. (2025). beyond audio signals: generative model-based speaker diarization in portuguese. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_17 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords speaker diarization generative models natural language processing publish with us policies and ethics profiles antônio oss boll view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature impact of parent selection operator on the fdea algorithm | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper impact of parent selection operator on the fdea algorithm conference paper first online: 30 january 2025 pp 94–109 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) impact of parent selection operator on the fdea algorithm download book pdf download book epub joão pedro a. de oliveira9 & olacir r. castro jr.9  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 366 accesses abstract many-objective optimization problems (maops) present a significant challenge due to the increased complexity associated with optimizing more than three conflicting objectives. traditional pareto-based algorithms often struggle to scale effectively with the number of objectives, leading researchers to explore alternative approaches. this study investigates the impact of parent selection strategies on the performance of the fuzzy decomposition-based multi/many-objective evolutionary algorithm (fdea). the fdea uses a fuzzy decomposition approach to divide a maop into subproblems, each solved individually to improve the adaptability and precision of solutions. the study compares the fdea with three selection methods: binary tournament based on pareto dominance, binary tournament using crowding distance, and a combination of pareto dominance and crowding distance. empirical evaluations were conducted using the dtlz and wfg benchmark problem sets, with 2, 3, 5, 8, 10, and 15 objectives. the results indicate that incorporating quality indicators into the selection process improves the efficiency and effectiveness of the fdea, especially in high-dimensional optimization scenarios. however, no variant was superior in all contexts. the study highlights the potential for methodological advances in indicator-based mating selection, emphasizing the need for robust and scalable algorithms capable of handling the complexities of maops. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others hybrid selection based multi/many-objective evolutionary algorithm article open access 27 april 2022 rode: ranking-dominance-based algorithm for many-objective optimization with opposition-based differential evolution article 20 may 2020 a new angle-based preference selection mechanism for solving many-objective optimization problems article 03 february 2018 1 introduction multi-objective optimization problems (mops) are those that require the simultaneous optimization of two or more objective functions. these objectives are usually conflicting, resulting in a set of solutions rather than a single optimal solution. to find this set of solutions, pareto optimality theory is generally used [1]. when a mop has more than three objectives, it is called a many-objective optimization problem (maop). maops present greater complexity compared to traditional mops due to the increase in the number of objectives [1]. this added complexity makes it harder to accurately approximate the pareto front, making it challenging to differentiate between high-quality and low-quality solutions. in addition, maintaining the convergence and diversity of solutions becomes more challenging, requiring advanced mechanisms to prevent premature convergence and ensure diversity. in the context of multi-objective evolutionary algorithms (moeas), the fuzzy decomposition-based multi/many-objective evolutionary algorithm [11] stands out by combining problem decomposition with the principles of fuzzy logic. the fdea employs a fuzzy decomposition approach to divide the mop into a set of subproblems, each solved individually. an essential component of the fdea, as well as other evolutionary algorithms, is crossover selection. crossover selection is a crucial step aimed at identifying pairs of individuals that can generate offspring with desirable characteristics [14]. this process is fundamental to ensure compatibility and complementarity of genetic traits, thereby enhancing the generation of superior solutions. related research indicates that only a few moeas utilize the crossover selection strategy, highlighting the need for more studies and methodological advances in this area. indicator-based methods, such as r2-ibea and maoea/igd, demonstrate the effectiveness of crossover selection based on quality indicators to improve algorithm performance in various optimization scenarios. this paper presents a study on using different crossover selection mechanisms in the fdea algorithm. four variants of the fdea were compared, each with different selection strategies, to assess their impact on the algorithm’s efficiency and effectiveness in solving complex maops. the variants are: fdea, the original variant that uses random selection; fdea_tb, which employs binary tournament selection based on pareto dominance; fdea_cd, which uses binary tournament selection based on crowding distance; and fdea_cdtb, which combines pareto dominance and crowding distance to make the final selection. the results indicate that incorporating quality indicators into the selection process enhances the efficiency and effectiveness of the fdea, particularly in high-dimensional optimization scenarios. however, it is important to note that no single variant proved superior across all tested contexts, suggesting that the optimal selection strategy may vary depending on the specific characteristics of the problem at hand. this study underscores the potential for methodological advancements in indicator-based mating selection. the remaining of this paper is organized as follows: some backgroud concepts are presented in sect. 2. section 3 presents the related work. in sect. 4 an empirical study comparing the modification of the crossover selection method on the fdea algorithm and sect. 5 presents the conclusions. 2 background 2.1 many-objective optimization a multi-objective optimization problem (mop) is a type of problem that requires the simultaneous optimization of two or more objective functions. these objectives are usually in conflict, so these problems do not present only one optimal solution but a set of them. this set of solutions is usually found using pareto optimality theory. a solution \({\textbf {x}} \in \varomega \) is optimal relative to \(\varomega \) if and only if there is no \({\textbf {x'}} \in \varomega \) that dominates x in all objectives [1]. in other words, a solution is pareto optimal if there is no other solution that is better or equal in all objectives and better in at least one simultaneously. in multi-objective optimization, it is inappropriate to use the term “optimal solution” since individual objectives often conflict. instead, a set of sufficiently good solutions is sought, representing a balance between the different objectives. when a multi-objective optimization problem (mop) has more than three objectives, it is classified as a many-objective optimization problem (maop). maops are more complex than mops due to the increased number of objectives, which makes approximating the pareto front more difficult and reduces selective pressure, complicating the distinction between high and low-quality solutions. additionally, maintaining convergence and solution diversity becomes more challenging, requiring advanced mechanisms to prevent premature convergence and preserve diversity [1]. recently, there has been a growing interest in enhancing the performance of moeas in maops [2]. the most popular methodologies include: (1) moeas that use relaxed pareto dominance relations, applying alternative preferences to increase selection pressure [3]; (2) decomposition-based moeas, which transform an mop into several single-objective optimization problems using scalarization functions; and (3) indicator-based moeas, which use quality indicators to evaluate solution sets and define selection mechanisms, aiming to optimize the population indicator value throughout the evolutionary process. 2.2 quality indicators evaluating the performance of a multi-objective optimizer is challenging, but there are three main requirements: convergence, which seeks results close to the true pareto set; diversity, which looks for solutions widely distributed in the objective space to allow comprehensive choices; and pertinence, which directs convergence and diversity to relevant areas of the search space where the decision-maker has the most interest. these characteristics are desired to ensure the optimizer is efficient and provides meaningful options to the decision-maker in multi-objective problems [4]. quality indicators are functions that evaluate approximation sets and can be used to define selection mechanisms in multi-objective evolutionary algorithms (moeas). the idea is to optimize the value of the population indicator throughout the evolutionary process [2]. indicator-based selection involves identifying good parental solutions based on quality indicator values. this type of selection mechanism does not aim to solve or approximate the subset selection problem based on indicators; instead, it attempts to produce promising solutions to accelerate the evolutionary process. crowding distance is a metric that estimates the density of solutions around a specific solution. it is calculated as the average of the normalized absolute distances between the objective function values of two adjacent solutions in each objective. the population is sorted according to the values of the objective function, and frontier solutions are given an infinite distance to ensure their selection. the distances of intermediate solutions are normalized between the minimum and maximum values of each objective, ensuring that all solutions have the same impact on the calculation of the metric. this metric is crucial for comparing solutions that belong to the same level of non-dominance, favoring those in less “crowded” regions [12]. igd. inverted generational distance (igd) measures the average distance from each reference point in the objective space to the nearest solution in the approximation set. this metric considers both convergence and the diversity of solutions, ensuring that the solutions not only approach the true pareto front but are also well distributed along it. igd is useful to assess how well the solutions cover the entire pareto front [5]. hipervolume (hv) is a crucial metric in multi-objective optimization used to evaluate the quality of solution sets by measuring the volume of the objective space dominated by the solutions. it assesses both convergence to the pareto front and the dispersion of solutions within the objective space, with higher hv values usually indicating better performance. calculating hv can be computationally intensive and requires a carefully chosen reference point, as this significantly impacts the results. overall, hv provides a comprehensive measure of an optimization algorithm’s effectiveness by considering both the proximity of solutions to the pareto front and their distribution across the objective space [2]. 2.3 benchmark problems benchmark problems are widely used in the literature to evaluate the performance of multi-objective optimizers and compare algorithms. they are specifically designed for evaluation purposes and offer several advantages, such as known true pareto fronts for precise assessment and control over difficulty levels, objectives, and decision variables, which facilitates comparisons between algorithms [9]. by isolating the optimizer’s performance, these problems eliminate potential real-world interferences, providing an objective evaluation of the algorithms’ capabilities, strengths, and limitations. however, it is important to note that benchmark problems are simplifications of real-world challenges, and the results obtained may not be directly transferable to real-world situations. among the most commonly used benchmark problems in academia are the dtlz and wfg problem sets, which are widely recognized and accepted for evaluating multi-objective optimization algorithms. dtlz. the dtlz benchmark problem set, formulated by deb et al. [10], was designed to provide a comprehensive and rigorous evaluation of multi-objective optimization algorithms, particularly those based on evolutionary principles. these problems are scalable, allowing researchers to adjust the number of objectives and decision variables to test an algorithm’s ability to manage increasing complexities. this scalability is essential because challenges in multi-objective optimization typically increase with the problem’s dimensionality, both in terms of search space and the complexity of the pareto front. each function within the dtlz set is constructed to evaluate distinct characteristics of evolutionary algorithms. for example, dtlz1 tests the convergence of algorithms to a linear pareto front, serving as an effective starting point for initial efficacy assessments. conversely, dtlz2 examines the ability of algorithms to converge to a concave pareto front, necessitating a robust search for optimal solutions within a more complex solution space. additionally, dtlz3 increases complexity by incorporating multiple local optima into the dtlz2 function. these problems, featuring non-convex and discontinuous pareto front surfaces, challenge algorithms to find suitable solutions while navigating the complex topology of the solution space. consequently, the dtlz problems are invaluable to the scientific community, providing a robust foundation for systematic benchmarking and objective comparison of diverse multi-objective optimization strategies. wfg. the wfg (walking fish group) benchmark problem set is designed to evaluate the efficiency of multi-objective optimization algorithms. these problems are particularly valuable because they encompass a wide range of pareto front shapes, allowing for the assessment of algorithm performance in various complex scenarios. wfg problems utilize a vector of underlying parameters, x, linked to a simple base problem that defines the fitness space. this vector results from sequential transformations from an initial parameter vector, z, adding layers of complexity such as multi-modality and non-separability. all wfg problems follow a specific format that involves minimizing a set of functions with parameters, z and x, where m, represents the number of objectives. the set x defines underlying parameters related to distance and position, while z comprises working parameters divided into position-related elements and distance. the constants \(a_1\) to \(a_{m-1}\) act as degeneracy factors, reducing the dimensionality of the pareto front. the functions \(h_1\) to \(h_m\) define the shape, and \(s_1\) to \(s_m\) are scaling constants. these transformations and structures provide a robust framework for systematic benchmarking and comparing different multi-objective optimization strategies. 2.4 fdea in the domain of multi-objective evolutionary algorithms (moeas), the complexity of managing high-dimensional solution spaces presents unique challenges concerning algorithm complexity and performance. the fuzzy decomposition-based multi/many-objective evolutionary algorithm (fdea) distinguishes itself by combining problem decomposition with the principles of fuzzy logic. this algorithm employs a fuzzy decomposition approach to divide a multi-objective optimization problem (mop) into a set of subproblems, each solved individually to enhance the adaptability and precision of solutions in multi-objective problems [11]. algorithm 1 . fdea(\(n, m, g_{max}\)) full size image the fdea begins by generating an initial random population and creates a new population by applying variation operators (crossover and mutation) to the parent population. the algorithm then performs the fuzzy decomposition process, which is divided into two main stages: fuzzy prediction and weight vector extraction. fuzzy prediction: in this phase, the solution space is partitioned into subregions using fuzzy logic. each solution is assigned to one or more subregions based on a membership function that determines the proximity of the solution to different reference vectors. this approach allows the fdea to capture a variety of solutions representing different parts of the pareto front. weight vector extraction: after fuzzy prediction, the algorithm extracts weight vectors that define the preferred directions in the objective space for each subproblem. these weight vectors are used to guide the search in each subregion, encouraging exploration in directions that promote a uniform coverage of the pareto front. following the fuzzy decomposition, an elite selection procedure is implemented to form the next generation. this selection strategy ensures that high-quality solutions, which best represent the desired characteristics of the objective space (such as convergence and diversity), are preserved and propagated to subsequent generations. elite selection in the fdea uses a combination of pareto dominance and crowding distance to select the most promising solutions from each subregion. the fdea adopts a normalization procedure derived from the nsga-iii algorithm to ensure uniform treatment of objectives during the optimization process. this normalization adjusts the solutions so that all objectives are treated equitably, preventing some objectives from unduly dominating others during the selection process. evolutionary operators, such as simulated binary crossover (sbx) and polynomial mutation (pm), are applied to pairs of parent solutions that are randomly selected or chosen based on indicator-based strategies. this normalization is crucial for the effective performance of the fdea, especially in problems with highly conflicting objectives. 2.5 crossover selection crossover selection in evolutionary algorithms aims to identify pairs of individuals that can produce offspring with desirable characteristics. this process is crucial to ensure compatibility and complementary genetic traits, which enhance the generation of better solutions. producing promising offspring is critical for efficient convergence to optimal or near-optimal solutions [14]. according to falcón-cardona and coello [2], crossover selection involves using quality indicator values to identify good parental solutions. this selection mechanism focuses on producing high-quality offspring to accelerate the evolutionary process rather than solving the subset selection problem based on indicators. by prioritizing the generation of promising offspring, the mechanism aims to optimize the search in the solution space, increasing the probability of finding superior solutions in fewer generations [14]. indicator-based mating selection. multi-objective evolutionary algorithms (moeas) traditionally use pareto dominance to identify a set of optimal solutions. however, quality indicators can provide additional insight that reflects user preferences and enhances the conventional concept of optimality. within this context, ib-mating selection leverages these indicators to select parental solutions, aiming to improve the generation of high-quality offspring [2]. ib-selection, a broader category within indicator-based mechanisms, includes ib-environmental selection, ib-density estimation, and ib-archiving. ib-environmental selection prioritizes solutions based on their quality and contribution to the population’s environment, promoting diversity and overall set quality. ib-density estimation assesses the density of solutions in the search space, guiding algorithms to explore less dense regions or refine high-quality solutions in denser areas. ib-archiving uses indicators to maintain a diverse and high-quality archive of solutions representing the pareto front. specifically, ib-mating selection focuses on selecting \(\mu \) parental solutions that, through genetic operations, produce \(\lambda \) offspring with high potential for quality and diversity. by prioritizing high-quality offspring, ib-mating selection drives the evolutionary process efficiently, ensuring a constant flow of superior solutions across generations. 3 related work according to falcón-cardona and coello [2], only three multi-objective evolutionary algorithms (moeas) use the crossover selection strategy: r2-ibea, maoea/igd, and ar-moea. these algorithms represent distinct approaches to optimizing the selection of parent solutions, each with its own methodologies and specific focuses. the r2-ibea [15] is one of the algorithms that uses the binary r2 indicator to select solutions. this method ensures that the chosen solutions outperform others in terms of weak pareto dominance, prioritizing high-quality solutions. the main advantage of this approach is the maintenance of a robust and competitive population of solutions, improving the overall performance of the algorithm in finding optimal solutions. the maoea/igd [16] implements a binary tournament where solutions are evaluated based on two main criteria: the dominance rank and the relative distance to the hyperplane, which serves as the reference set for the igd indicator. initially, the solutions are ranked by their dominance, with a preference for those with lower ranks. if there is a tie in the rank, the distance is used to determine the final selection. this method is effective in identifying solutions that not only have good convergence but also present an adequate distribution in the objective space. the ar-moea [18] uses the igd-ns metric to evaluate the solutions in the tournament, selecting those that offer the highest contribution in terms of fitness value. the solution with the highest contribution is selected for the parent set, ensuring that the generated solutions are optimal in terms of convergence. this focus on maximizing the contribution of each solution helps to enhance the efficiency of the evolutionary process, resulting in superior algorithm performance in multi-objective optimization problems. falcón-cardona and coello [2] observe a significant gap regarding the emphasis and in-depth study of indicator-based crossover selection, highlighting its underestimation and potential for new methodological advances. indicator-based crossover selection methods integrate quality indicators (qis) as a central selection criterion, quantifying specific characteristics of solutions or the population for more informed and targeted evaluations. 4 empirical study this section presents the results of the modification of the crossover selection operator in the fdea algorithm. the study involved adapting the fdea to use different selection strategies to select the parents, which were then subjected to evolutionary operators sbx and pm. an experimental study was carried out to compare the original fdea selection (random) and the new selection strategies to assess their effectiveness. the following sections detail the standard and modified parameters, as well as the evaluation criteria used to assess the results. table 1. population and objective numbersfull size table table 2. \(igd_p\) results for dtlz problemsfull size table 4.1 experimental setup in this study, the fdea algorithm was executed with the same parameters as those outlined in the original article [11], as shown in table 1, to maintain result consistency and comparability. the number of objectives (m) ranged from 2 to 15, addressing various levels of mop complexity. the population size (n) was modified based on the number of objectives. the maximum number of generations (gmax) was set between 300 and 1500, depending on the objective numbers, to ensure adequate convergence. these parameters were kept to provide a thorough and comparable assessment of the fdea, aiming to validate the proposed hypothesis and evaluate performance across different multi-objective optimization scenarios. the study used four variants of the fdea algorithm to assess the impact of different selection methods on its performance. the original fdea variant uses random selection to choose pairs of individuals for crossover and mutation. this simple approach involves randomly selecting two solutions and ensuring that they are different if the set contains more than one solution. the second variant, fdea_tb, employs binary tournament selection based on pareto dominance. in this method, two solutions are randomly selected and compared using a dominance comparator, if there is a domination relation between them, the non-dominated one is selected, otherwise, one of them is selected randomly. the third variant, fdea_cd, also uses binary tournament selection but compares solutions based on crowding distance, a metric that helps maintain diversity among non-dominated solutions by selecting the solution in the less crowded area. the fourth variant, fdea_cdtb, combines the approaches of fdea_tb and fdea_cd. the solutions are first compared using pareto dominance and, if they are equally good, crowding distance is used to make the final selection. the objective of exploring these variants is to determine how different selection strategies impact the algorithm’s effectiveness, focusing on enhancing the evolutionary process by incorporating different criteria for parent selection. the results of modifying the crossover selection method in the fdea algorithm were obtained through a total of 30 independent runs for each variant of the algorithm, for each problem and number of objectives. after each run, the igd of each obtained front was calculated. the results were subjected to a kruskal-wallis test with a significance level of 5%. in cases where there was no significant difference between any pairs, a post-hoc test was used to determine which pairs were equivalent. the best results are displayed in table 2 for the dtlz problems and in table 4 for the wfg problems, showing the indicator values for the analyzed algorithms. the tables are organized by the results of the statistical tests, highlighting the best results, and, in cases where there is no significant difference, those that are not significantly different from the best are also highlighted. 4.2 results and discussion for dtlz problems for the dtlz1 problem, known for its multi-modal decision space and linear pareto front, all variants achieved no statistical difference, except for fdea_cd which was outperformed by the other variants in two objectives. in the dtlz2 problem with a concave pareto front, fdea_cdtb had inferior performance in the 2-objective case. however, as objectives increased, fdea_cd achieved good results in 3, 5, 8, and 15 objectives. fdea and fdea_tb were outperformed in the 8-objective setup, but all variants performed well and were statistically equivalent otherwise. for the dtlz3 problem, which includes multiple local pareto fronts, fdea and fdea_tb had the best performance regardless of objective number. conversely, fdea_cd and fdea_cdtb were outperformed for 5 and 15 objectives. for 2, 3, 8, and 10 objectives, all variants were statistically equivalent. in the dtlz4 problem, challenging for maintaining a diverse solution set, fdea_tb was outperformed by the other variants for 2 objectives. fdea_cdtb had the worst results in the 5 and 10-objective configurations, and both fdea_cd and fdea_cdtb performed poorly for 15 objectives. for 3 and 8 objectives, all variants were statistically equivalent. a possible explanation for this poor performance of cdtb over the others is that is may favor dominance resistant solutions, hence, lowering diversity. for the dtlz5 problem, which is aconcave and degenerate problem, fdea_cd and fdea_cdtb performed worst in the 15-objective setup. for other configurations, all variants were statistically equivalent. in the dtlz6 problem, which is degenerate and presents convergence challenges, all variants were statistically equivalent for two objectives. for 3, 5, 8, 10, and 15 objectives, fdea_cd and fdea_cdtb performed better than the others. finally, in the dtlz7 problem, characterized by a disconnected pareto front, all variants performed equivalently for 2, 3, and 8 objectives. for 5 objectives, fdea and fdea_tb are tied for the best performance. for 10 and 15 objectives, fdea_cd and fdea_cdtb tied for the best results. table 3. lowest-performing of fdea variants for dtlz problemsfull size table based on the results summarized in table 3, a few patterns can be identified, first, in the non-degenerate problems that impose convergence challenge (dtlz1 and dtlz3) using the crowding distance that favors diversity does not seem to be a good choice. second, in the easier problems (dtlz1, dtlz2 and dtlz5) there are fewer differences between the variants, while in the dtlz6 and dtlz7 there are more differences. third, original fdea with random selection performs very well for two objectives, however its performance starts deteriorating as the number of objectives increase. table 4. \(igd_p\) results for wfg problemsfull size table 4.3 results and discussion for wfg problems for the wfg1 problem, which presents a convex geometry with a mixture of pareto shapes, the fdea_cd and fdea_cdtb variants had the best performances for 2, 3, and 5 objectives. as the number of objectives increases, all variants show similar performances. in the wfg2 problem, characterized by a convex and disconnected pareto front, the results show that, regardless of the number of objectives, all variants had statistically equivalent performance. in the wfg3 problem, characterized by a linear and degenerate geometry, the fdea and fdea_tb variants had the worst performance for 3 objectives. however, for 2, 5, 8, 10, and 15 objectives, all variants showed equivalent performances. for the wfg4 problem, which has a concave pareto front and is multi-modal, increasing the complexity of the problem, the fdea and fdea_tb variants had the worst performance for 10 and 15 objectives. however, for 2, 3, 5, and 8 objectives, all variants had equivalent performance. in the wfg5 problem, which tests convergence ability, the fdea and fdea_tb variants had the worst performances for 2, 10, and 15 objectives. for 3, 5, and 8 objectives, all variants had equivalent performance. in the wfg6 problem, a simple non-separable problem, all variants had statistically equivalent performance. for the wfg7 problem, which is a biased problem, the fdea variant had the worst performance for 10 objectives. for 2, 3, 5, 8, and 15 objectives, all variants had statistically similar performance. for the wfg8 problem, characterized by distance parameters dependent on position parameters, resulting in a biased and highly complex problem, the fdea and fdea_tb variants had the worst performance for 2, 10, and 15 objectives. the fdea_cdtb variant had the worst performance for 3 objectives, while for 5 objectives, the fdea_cd and fdea_cdtb variants showed the worst results. for 8 objectives, all variants had statistically equivalent performance. finally, for the wfg9 problem, which is a multi-modal, deceptive and biased problem, the fdea variant had the worst results for 8 and 15 objectives. for the other numbers of objectives, all variants had statistically equivalent performances. table 5. lowest-performing of fdea variants for wfg problemsfull size table in the wfg problems, where the results are summarized in table 5, the introduction of the new operators had one of three effects: it made the results worse than the original algorithm, which only happened on wfg8 for 3 and 5 objectives, hence, two instances (problem/objective number); it does not make a significant difference, which happened for all instances of the wfg2 and wfg6 problems, and a total of 37 instances considering all problems; it improves the results of the original algorithm, which happened in 15 instances along the problems. these results indicate that the operators proposed are beneficial when solving problems with the characteristics presented in the wfg suite. 5 conclusion this paper investigated the impact of different crossover selection strategies on the performance of the fdea algorithm for multi-objective optimization. four variants of the fdea were employed: fdea (random selection), fdea_tb (binary tournament with pareto dominance), fdea_tb (binary tournament with crowding distance), and fdea_cdtb (combination of fdea_tb and fdea_cdtb). the performance was evaluated using the igd metric on dtlz and wfg problems with 2, 3, 5, 8, 10 and 15 numbers of objectives. the results demonstrate that the choice of crossover selection strategy is capable of influencing the performance of the fdea. fdea_tb and fdea_cdtb showed good performance across various problems, especially in configurations with a higher number of objectives. fdea performed well in problems with multiple local pareto fronts, and random selection (original fdea) can be a simpler and computationally cheaper alternative, though with inferior results in some cases. despite the modifications, most variants maintained statistically equivalent performances in many of the tested scenarios. this suggests that while certain variants may excel in specific situations, there is no single variant that is superior in all configurations. this finding highlights the importance of adapting the choice of the fdea variant based on the specific problem and its characteristics. in conclusion, modifying the crossover selection method in the fdea was effective in improving the algorithm’s performance in different multi-objective optimization scenarios. these results encourage further studies on the hybridization of evolutionary algorithms, incorporating indicator-based selection methods to further explore the adaptive capabilities of algorithms in complex optimization environments. future work includes exploring other variants of crossover selection, utilizing additional quality indicators, and conducting comprehensive tests in various multi-objective optimization scenarios. this will allow the identification of the most effective strategies for different problem characteristics and further refine the selection parameters to achieve consistent and robust performance across a wide range of situations. references coello, c.a.c., et al.: evolutionary algorithms for solving multi-objective problems, 5th edn. springer, new york (2007) google scholar  falcón-cardona, j.g., coello, c.a.c.: indicator-based multi-objective evolutionary algorithms: a comprehensive survey. acm comput. surv. 53(2), 1–35 (2020). https://doi.org/10.1145/3376916 article  math  google scholar  liefooghe, a., derbel, b.: a correlation analysis of set quality indicator values in multiobjective optimization. association for computing machinery, new york (2016). https://doi.org/10.1145/2908812.2908906 adra, s.f.: improving convergence, diversity and pertinency in multiobjective optimization. uk (2007) google scholar  ishibuchi, h., masuda, h., tanigaki, y., nojima, y.: modified distance calculation in generational distance and inverted generational distance. in: gaspar-cunha, a., henggeler antunes, c., coello, c.c. (eds.) emo 2015. lncs, vol. 9019, pp. 110–125. springer, cham (2015). https://doi.org/10.1007/978-3-319-15892-1_8 chapter  math  google scholar  falcón-cardona, j.g., coello coello, c.a.: towards a more general many-objective evolutionary optimizer. in: auger, a., fonseca, c.m., lourenço, n., machado, p., paquete, l., whitley, d. (eds.) ppsn 2018. lncs, vol. 11101, pp. 335–346. springer, cham (2018). https://doi.org/10.1007/978-3-319-99253-2_27 chapter  math  google scholar  luke, s.: essentials of metaheuristics, 2nd edn (2012) google scholar  derrac, j., garcía, s., herrera, f.: a practical tutorial on the use of nonparametric statistical tests as a methodology for comparing evolutionary and swarm intelligence algorithms. swarm evol. comput. 3–18 (2011) google scholar  huband, s., et al.: a review of multiobjective test problems and a scalable test problem toolkit. ieee trans. evol. comput. 477–506 (2006) google scholar  deb, k., et al.: scalable multi-objective optimization test problems. ieee congr. evol. comput. 825–830 (2002) google scholar  liu, s., et al.: a fuzzy decomposition-based multi/many-objective evolutionary algorithm. ieee trans. cybern. 3495–3509 (2022) google scholar  deb, k., et al.: a fast and elitist multiobjective genetic algorithm: nsga-ii. ieee trans. evol. comput. 182–197 (2002) google scholar  deb, k., jain, h.: an evolutionary many-objective optimization algorithm using reference-point-based nondominated sorting approach, part i: solving problems with box constraints. ieee trans. evol. comput. 18, 577–601 (2014) article  math  google scholar  esquivel, s.c., et al.: selection mechanisms in evolutionary algorithms. fundamenta informaticae 17–33 (1998) google scholar  phan, d.h., suzuki, j.: r2-ibea: r2 indicator based evolutionary algorithm for multiobjective optimization. ieee congr. evol. comput. 1836–1845 (2013) google scholar  sun, y., yen, g.g., yi, z.: igd indicator-based evolutionary algorithm for many-objective optimization problems. ieee trans. evol. comput. 173–187 (2019) google scholar  hochstrate, n., naujoks, b., emmerich, m.: sms-emoa: multiobjective selection based on dominated hypervolume. eur. j. oper. res. 1653–1669 (2007) google scholar  tian, y., et al.: an indicator-based multiobjective evolutionary algorithm with reference point adaptation for better versatilit. ieee trans. evol. comput. 609–622 (2018) google scholar  download references author information authors and affiliations universidade federal do acre, rio branco, ac, brazil joão pedro a. de oliveira & olacir r. castro jr. authors joão pedro a. de oliveiraview author publications search author on:pubmed google scholar olacir r. castro jr.view author publications search author on:pubmed google scholar corresponding author correspondence to joão pedro a. de oliveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de oliveira, j.p.a., castro, o.r. (2025). impact of parent selection operator on the fdea algorithm. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_7 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords indicator-based mating selection many-objective parent selection strategies publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature assessing european and brazilian portuguese llms for ner in specialised domains | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper assessing european and brazilian portuguese llms for ner in specialised domains conference paper first online: 30 january 2025 pp 215–230 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) assessing european and brazilian portuguese llms for ner in specialised domains download book pdf download book epub rafael oleques nunes  orcid: orcid.org/0009-0007-8842-421x9, joaquim santos  orcid: orcid.org/0000-0002-0581-409210, andre spritzer  orcid: orcid.org/0009-0002-4232-15859, dennis giovani balreira  orcid: orcid.org/0000-0002-0801-93939, carla maria dal sasso freitas  orcid: orcid.org/0000-0003-1986-84359, fernanda olival  orcid: orcid.org/0000-0003-4762-345111, helena freire cameron  orcid: orcid.org/0000-0001-7719-699411 & … renata vieira  orcid: orcid.org/0000-0003-2449-547712  show authors part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 494 accesses 2 citations abstract this paper discusses the impact of portuguese variants in large language models for the task of named entity recognition (ner) in specialised domains. the tests were made on a brazilian portuguese legal and a european portuguese historical corpora. the models taken into account are bertimbau (pt-br), albertina (pt-pt and pt-br), and xml-r (multilingual). the impact was more evident in the portuguese historical corpus, which resulted in higher f1 measures compared to previous works that did not consider the same language variant. additionally, the study underscores the impact of model architecture on performance, highlighting the critical role of both linguistic alignment and model size in enhancing ner in specialised domains. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a study on the impact of intradomain finetuning of deep language models for legal named entity recognition in portuguese chapter © 2020 legalbert-pt: a pretrained language model for the brazilian portuguese legal domain chapter © 2023 portuguese named entity recognition using lstm-crf chapter © 2018 1 introduction named entity recognition (ner) is a crucial task in natural language processing (nlp). the objective of ner is to identify and classify specific terms in a sentence, such as the names of people, organisations, locations, dates, and other entities. this task is fundamental for various nlp applications, including information extraction, question answering, and automatic text summarisation. although many studies and models have been developed with good results, most have focused on general entities, relying on datasets such as conll 2002 for english [23] and harem for portuguese [17]. however, specialised domains often cannot utilise these general classifiers effectively because of domain-specific language or the need to identify different types of entities not commonly present in general datasets. for instance, while entities such as gene names and diseases are crucial in the biomedical domain, in legal texts, entities such as law citations and court names are more relevant. previous studies have addressed these challenges by delving into specific domains and proposing tailored datasets and models [13, 18, 20]. as specialised models on portuguese variants became recently available, [15, 21], this study evaluates the performance of these models on specialised texts written in two variants of the portuguese ner annotated corpus (european and brazilian). we aim to compare how these models perform in these variants, considering two specialised domains, historical and legislative, that present unique challenges and require domain-specific knowledge. by advancing ner capabilities in domain-specific portuguese texts in two variations of the language, we aim to facilitate more accurate and contextually relevant information extraction, supporting various applications, from historical research to legal document processing. the legislative and historical fields have the advantage of being comprehensive despite some specialization. this study contributes to the academic understanding of domain-specific ner and provides practical insights and tools for improving nlp applications in specialised fields. our main contributions are: (i) a comprehensive evaluation of different models on ner tasks within portuguese texts, (ii) an analysis of the differences in model performance when pre-trained with european and brazilian variants of the portuguese language, and (iii) insights into applying these models across different specialised domain corpora: legal and historical. 2 related work named entity recognition (ner) involves identifying and classifying entities such as locations, organisations, and persons within a text. general entities have been extensively explored in various languages and corpora. popular datasets for evaluating ner in different languages include conll 2002 for english [23] and harem for portuguese [17]. these datasets have been instrumental in advancing ner research and establishing model performance benchmarks. however, the classifiers trained on these datasets are not always sufficient for all domains. for example, silva et al. (2023) [20] proposed a dataset focused on cachaça, a distilled spirit made from sugarcane juice, where important entity categories include classification, price, storage time, and sensory characteristics, such as colour. these particular entities cannot be adequately represented using only the traditional ner categories. the legal domain has also been found to benefit from the use of specialised entities, such as legal citations from legislation or court cases [3]; products of law and legal basis [2]; courts, origins of legal procedure, and trial dates [7]; and decisions and sentences [5]. recent studies have explored these areas using various techniques with transformer models, including domain adaptation [4, 9], fine-tuning [25], in-context learning [14], and self-learning [13]. historical texts represent another important domain with unique characteristics. studies focusing on ner tasks in historical texts [10, 18, 24] have highlighted the challenges posed by different word spellings and sentence structures compared with contemporary texts in the same language. all the works discussed so far have focused on ner tasks within the context of portuguese language texts. in this study, we extend this focus by evaluating the impact of portuguese-specific models on specialised texts, specifically within the historical and legal domains, exploring both european and brazilian language variants. 3 corpora 3.1 european portuguese annotated historical texts the first sub-corpus considered in this study is a subset of the historical pt-eu corpus, the parish memories collection. it is an 18\(^{th}\)-century corpus composed of the answers to a survey sent in 1758 to the portuguese priests to obtain feedback about the state of the territory after the 1755 lisbon big earthquake and to gather information to form a geographical dictionary of portugal. the sub-corpus under analysis contains 71 memories of the municipalities of the main cities in the largest region in portugal, alentejo, i.e., portalegre, évora and beja. we also added the memories from the municipality of vila viçosa, a historically relevant municipality, dominated by the house of bragança. the original manuscripts have been transcribed and normalised to contemporary standard european portuguese spelling. the sub-corpus was annotated with named entities, customised to the historical reality. in a first approach (anonymous, 2021) three basic categories were considered (person, local, organisation), as, for a historian, they aim to answer the main questions: who, where, and when. after this initial approach, considering the need to describe past realities better, a corpus-based study was conducted to define extensions of these categories (anonymous, 2022) according to their relevance to the historian’s inquiry. for that, the main category of person was broken down into several subcategories. the category person (per) refers to references made by name, first name, and family name (per_nam); occupation (per_occ); or social category (per_cat). all these attributes reflect the hierarchical structure of the 18\(^{th}\)-century portuguese society as, frequently, titles and occupation positions were almost part of a person’s name and identity. an example of mentions to persons by occupation is reitor da universidade de évora (rector of the university of évora). still related to the per category, and because they constitute specific details of person, we established other subcategories to label saints (per_saint); divinities (per_div); groups of persons (per_pgrp); and authors (per_aut). the subcategory for groups of persons is used to annotate organic groups, families, and members of an organisation, among others. monges cartuxos (carthusians monks) and sarracenos (the saracens) are examples of this category. concerning the local category, we generalised it to place (plc). this category includes geopolitical entities (plc_gpe), aquifers(plc_aqu), mountains (plc_mount), facilities (plc_fac), and one extra subcategory for other locations (plc_loc) as for instance, bispado de portalegre (bishopric of portalegre, plc_gpe) and rio guadiana (guadiana river, plc_aqu). geopolitical entities were included to avoid ambiguities among locations and organisations, as this categories aggregates them indistinctly. other references to geographical points, such as rivers and mountains, are essential for geo-references. the remaining categories are for organisation, time, and authored work. the org category labels several organisations as, for example, companhia de jesus (society of jesus), and universidade de coimbra (university of coimbra). table 1. frequency of named entities in the parish memories for each type.full size table for tim_cron, we only annotated specific references to dates, for instance, o ano de 1755 (the year of 1755). we also established a category, autwork, that allows us to treat the text sources mentioned in the corpus to recognise the text sources mentioned by priests. an extended set of ne categories to account for past ages also implies more complexity in annotation and their computational processes. all the documents of the sub-corpus were manually annotated based on the consensual judgment of four annotators, and it was made using the inception platformfootnote 1. as can seen in table 1, we have 5,031 annotated nes. the major classes are geo-political entities, person names, and saints. persons are referenced only by category, and mountains are the least represented. for training, development, and testing, the distribution is 70, 10, and 20%. 3.2 brazilian portuguese annotated legislative texts table 2. frequency of named entities in ulyssesner-br for each type.full size table the legislative corpus used in this work is ulyssesner-br [2]. it comprised 150 bills from the brazilian chamber of deputies (bcod). annotations were performed in three phases by two undergraduate students and one graduate student as a curator. ulyssesner-br [2] included both coarse and fine-grained levels. the coarse-grained level comprises seven entity categories, whereas the fine-grained level includes eighteen entity types. the entities follow the traditional harem [16] entities (person, location, organisation, event, and date) with the addition of domain-specific entities such as law foundations and law products. the entity pessoa (person) was specialised in three types: pessoaindividual (individual), pessoacargo (occupation), and pessoagrupocargo (group of occupations). in this division, it is possible fine-grained levels of person citations, such as deputado hildo rocha (deputy hildo rocha), and 16 de março de 2011 [march 16, 2011]. fundamento (law foundation) refers to various legal entities such as laws, bills, and legislative consultations requested by congressmen. this category includes fine-grained entities, such as fundlei (legal norm), fundapelido (legal norm nickname), and fundprojetodelei (bill). examples of fine-grained entities include art. 34 do estatuto do idoso (art. 34 of the elderly statute), and código brasileiro de trânsito (brazilian traffic code). the final category, produtodelei (law product), pertains to anything created due to legislation. this class also includes three fine-grained types: produtosistema (system product), produtoprograma (program product), and produtooutros (other products). examples of each fine-grained class are sistema único de saúde (unified health system), and salário mínimo (minimum wage). 4 framework and models we adopted the flair [1] framework, a ner library for multiple languages developed in pytorchfootnote 2. this framework provides pre-trained language models, named entity recognition models, and neural networks for language model training and sequence tagging. with flair, we can construct pipelines for training token classifiers and feed them with various types of language models, such as word embeddings, transformer-based models, and flair embeddings itself. herein, we analyse four versions of language models. bertimbau [21] is a pre-trained transformer-based language model trained specifically for brazilian portuguese. it was trained on the brwac corpus [8], which amounts to a total of 2.6 billion tokens, resulting in 17.5 gb of preprocessed data. bertimbau was trained using token masking in input sentences. in other words, it is a masked language model (mlm). we chose this model because the current state-of-the-art [22] in ner for portuguese uses this model. we used the large version of bertimbaufootnote 3. albertina pt-* [15] is a large language model designed explicitly for portuguese. it functions as an encoder within the bert family and is built upon the deberta model using the transformer neural architecture. albertina pthas two variants: albertina pt-pt and albertina pt-br. both variants are distributed free of charge, under a permissible license. albertina pt-pt is the european portuguese version. this model is available in three sizes, specifically with 1.5 billion parameters, 900 and 100 million parameters. the pre-training corpora of the albertina pt-pt 1.5b comprises general and legislative domains. albertina pt-br focuses on brazilian portuguese. its largest version, albertina 1.5b pt-br [19], has a more permissive licensed model without using the brwac dataset, consisting of a 36 billion token dataset compiled from a multilingual corpus. since this corpus includes both european and brazilian portuguese, additional filtering was applied to retain only documents with metadata indicating brazil’s internet country code top-level domain. xlm-roberta [6] is a multilingual model designed to understand 100 languages without requiring language-specific tensors, as it can identify the language directly from the input identifiers. it incorporates techniques from roberta [12] into the xlm framework [11], focusing solely on masked language modeling for single-language sentences, and does not employ translation language modeling. table 3 compares the four described models according to some of their features. table 3. comparison of the four portuguese large language models.full size table 5 experimental evaluation we conducted our experiments on a gpu a100 with 80gb of ram and an rtx 4090 with 64gb of ram. the experiments used python 3.7.6 and the flair library to use pre-trained models. hyperparameters were set to the default values recommended by the library: a learning rate of 5e-5, a mini-batch size of 4, and training for 10 epochs. truncation was applied to the maximum length, and padding was set to true. we employed standard metrics for model evaluation, including accuracy, precision, recall, and micro f1-score using the conll-2002 script [23]. these metrics provided a comprehensive assessment of the model’s performance across different classes, ensuring a thorough evaluation of its effectiveness. 6 results and discussion 6.1 overall results first, we detail the results for each corpus individually. then, we combine the results for comparative analysis, highlighting the influence of linguistic context and textual domain on the performance of specialised ner models. table 4. parish memories models results.full size table parish memories corpus. table 4 presents the overall results, highlighting that albertina, trained in european portuguese, achieved superior f1-score results compared to other models. this success is likely not solely due to albertina having more parameters than previous models but primarily due to its pre-training on european portuguese texts. given that, although the corpus consists of 18\(^{th}\)-century portuguese texts, they were used in their normalized version in european portuguese, and this linguistic alignment may likely have contributed significantly to albertina’s performance. another notable observation concerns the differences in f1-scores, as indicated in the column \(\updelta \) \(\uparrow \). the largest discrepancy among the models’ results is observed with european portuguese albertina, which markedly outperforms its brazilian portuguese counterpart. the recall metric also yields insightful conclusions. bertimbau-large exhibits the highest recall among brazilian portuguese and multilingual models, surpassing albertina and xlm-r-large, suggesting that its smaller architecture can achieve comparable entity identification. however, precision trends align with model size, indicating that larger models tend to identify true positive instances more precisely. leveraging a model trained on the same portuguese variant as the corpus consistently yields the best results across all metrics. table 5. ulyssesner-br types level models results.full size table brazilian legislative texts. table 5 presents the results for the fine-grained level in the ulyssesner-br corpus. it highlights that larger models can achieve better results, but the contribution of language specificity to enhanced performance remains inconclusive. the challenge of language specificity is evident in table 5, where the increment in f1-score for the two best models (the variations of albertina) is only 0.08, possibly due to the legislative data during the pre-training of european portuguese albertina (see table 3), or possibly to random classifier initialisation and result fluctuations. comparing the models, a significant finding is bertimbau’s high recall compared to larger models, underscoring its effectiveness in accurately identifying positive instances within specific legislative contexts, even if it is the smallest model. however, its lower precision suggests potential misses in positive instances. larger models, such as xlm-r and albertina, excel in capturing fine-grained categories. while bertimbau demonstrates superior recall in fine-grained analysis, the trade-off between precision and recall results in larger models achieving better f1-score results with comparable recall. the albertina models achieve the best balance between precision and recall. this balance indicates their effectiveness in both capturing a large number of relevant instances and maintaining accuracy. comparative analysis. the linguistic variation between brazilian portuguese and european portuguese differed significantly across the two corpora, highlighting how the parish memories corpus could leverage models tailored to its specific variant more effectively than brazilian legislative texts. one plausible explanation lies in the domain and structure of the texts. the parish memories corpus comprises letters written by priests with varying levels of academic background and formality. these texts differ from contemporary european portuguese but maintain significant linguistic proximity, especially in their textual references. while the entities mentioned refer to things, people, and events of the 18\(^{th}\)-century, many of these references are still used and mentioned today, such as place names like coimbra and évora and local parish names and devotions that continue in the country. therefore, it is reasonable to argue that a model trained on european portuguese is better equipped to extract entities from this corpus. conversely, even though the brazilian legislative corpus contains contemporary texts, many of these details were not necessarily learned from general texts in either portuguese variant by the structure and specific jargon of legislative texts. thus, it becomes that, for this corpus, the model size was more decisive than the linguistic variation. our results and analysis illustrate how linguistic context and textual domain play a crucial role in shaping the selection and performance of ner models. it underscores the importance of meticulously evaluating the corpus to select a suitable model that optimizes the extraction of named entities within specific contexts. 6.2 results by categories table 6. parish memories results per entity.full size table we present the results for each corpus at the entity level. we discuss how each model learned each entity and what is the influence of specialised entities. subsequently, we present a comparative analysis to conclude how different levels of entities can help or affect the final model results. parish memories. table 6 shows the results at the entities level. albertina in european portuguese tends to consistently achieve the best results in f1-score in most categories, pointing out that the specific features of the language in the model can be advantageous to the corpus. models like albertina (pt) and (br) show a better balance between precision and recall, leading to higher f1 scores across many categories. this balance is important for practical applications where false positives and negatives must be minimised. the table also highlights that albertina (pt) achieved the highest results, with no f1-scores near 50%, unlike a random classifier. in contrast, bertimbau had three classes with scores near 50% and one with even lower results. additionally, it is noteworthy that albertina (br) had five classes with results near 50% (the highest number among the models). however, the overall result compensated for the higher number of classes with f1 scores of 70% or more. we observe specific trends and outliers in the f1-scores in a bird’s eye analysis of the categories in table 6. in autwork and plc_mount, the result increased when the model size was increased (see table 3 for reference on model sizes), and the best result was in the portuguese european albertina model. the albertina (pt) model consistently performs well in identifying specialised persons. however, an outlier emerges in the per_cat (social category), where bertimbau exhibits a notable increase of 17.93 points compared to the second-best result achieved by albertina (pt). this unexpected outcome could be attributed to bertimbau’s pre-training on the brwac corpus [8], which includes a diverse range of content potentially encompassing social category data. this may have enhanced bertimbau’s ability to accurately recognize entities within social contexts. additionally, while bertimbau shows a slight improvement in the per_saint category with a marginal increase of 0.45 points, this improvement is not significant enough to decisively conclude it performs better than albertina in this category. similarly, place entities consistently performed well with the albertina (pt) model, often achieving results that were either the best or very close to it. for instance, in plc_aqu, plc_fac, and plc_gpe, albertina (pt) demonstrated high precision and recall, with marginal differences from the top performer ranging from 0.57 to 1.38 (the largest difference observed). this suggests albertina (pt) can identify and classify various place-related entities within the corpus. however, it’s notable that bertimbau occasionally surpassed albertina (pt) in specific categories, such as tim_cron, indicating its capability to excel in contexts where temporal references are crucial. these nuances of performance highlight the strengths of each model in different entity categories. it emphasises the importance of considering specific contexts and the balance in the results distributions in evaluating effectiveness across varied entity types (table 7). table 7. ulyssesner-br type level results per entity.full size table brazilian legislative texts. table 5 shows the results at the entity level. similar to the conclusions drawn in sect. 6.1, we observe that top results are well-distributed among the models bertimbau, xlm-r, albertina (br), and albertina (pt), with 4, 6, 6, and 8 instances of achieving the best scores, respectively. additionally, while there are several instances of closely competitive results across models, having the worst result in a category does not imply poor performance, as evidenced by five instances where the worst result still exceeds 89%. this demonstrates the models’ effective learning of the representations of many target entities. regarding the specific classes, date (data) is notably well learned by all the models, with the minimum f1-score varying between 97.02% and 98.99%. it is an expected result since date is the largest class, with 433 examples in the training set, and carries specific patterns in its format. the class event (evento) was also learned by all models, although results exhibited greater variability compared to the date class, ranging from 75.00% to 87.50%. event is a minority class in the training set, comprising only 9 examples. bertimbau large stood out by achieving the highest performance for this class. this contrasts with previous findings [13], which demonstrated that bertimbau base failed to learn event, resulting in an f1-score of 0%. this observation supports the hypothesis discussed in sect. 6.1 that model size, rather than idiomatic variation, plays a crucial role in performance on this corpus. the location (local) classes exhibited varying results and trends. first, concrete places (localconcreto) were well-learned entities, with f1 scores ranging from 89.25% to 92.31%, demonstrating an increase in performance with larger models. concrete places have specific semantics related to geographical landmarks, which aid the models in identifying context and patterns. in contrast, virtual places (localvirtual) performed worse, with f1 scores between 29.41% and 51.61%. a virtual place is a more diverse entity encompassing different types of locations, which do not necessarily share the same meaning and can include entities such as newspapers and internet pages [16], making it more challenging for the models to learn. additionally, annotations in this broader category may lead to semantic conflicts. for example, the term jornal diário catarinense (diário catarinense newspaper) in the sentence destaco que nos anos 90, o jornal diário catarinense realizou uma pesquisa popular que colocou entre os 20 catarinenses do séculofootnote 4 could be understood as both a location and a general organisation [14], which is challenging for the model to discern. organisations (org) showed a slight variation in results across different models. the results of the specialised classes of organisations highlight the relationship between the number of training examples in each class and their respective f1 scores. as seen in table 2, governmental organisations (orggovernamental), non-governmental organisations (orgnaogovernamental), and political parties (orgpartido) had 324, 88, and 23 examples in the training set, respectively. despite the number of examples, the results were inversely related, with the minority class achieving the highest f1-score and the majority class also achieving a high score, as shown in table 5. one possible explanation for this behaviour is the specificity of the class. for instance, although the governmental organisation class pertains to a specific domain, it encompasses a wide range of entities, from municipal guards to the chamber of deputies. the same applies to non-governmental institutions. conversely, political parties typically appear in more specific contexts or share similar characteristics in their names, such as following the name of a deputy or including the word “party” in their titles. the classes related to law foundation legal norm nickname (fundapelido), legal norm (fundlei), and law proposals (fundprojetodelei) exhibited varying levels of learning. the discrepancies in results are likely attributed to the number of training examples available for each class. for example, legal norm nickname and legal norm achieved high f1 scores of 97.00% and 94.38%, respectively, with 123 and 359 examples. in contrast, law proposals, despite having a specific format, such as the example “pec 187/2016”, had only 8 examples in the training set. the highest results were concentrated in bertimbau in law foundation classes. the specialised classes of law products (produto) exhibited a trend of improved performance with larger models. this was particularly evident in the other products class (produtooutros), which achieved its best results with albertina (br). this class showed strong performance, specifically in the portuguese models, with the highest result being the brazilian portuguese version of albertina. regarding the program product class (produtoprograma), the best results were shared among xlm-r and both variations of albertina, with identical scores across all metrics. this consistency necessitates a more detailed investigation into the class distribution and model learning. lastly, the system product class (produtosistema) had only one example in the test set, and the same term appeared in the training set, making it difficult to determine whether the 100% f1-score reflects a well-learned class or if the model specialised in recognising this specific term. comparative analysis. regarding the models, albertina (pt) demonstrated superior performance in the parish memories corpus due to its european portuguese training, particularly in recognising person and place entities. bertimbau showcased strengths in identifying entities influenced by its diverse pre-training data, achieving comparable results to larger models, especially in the brazilian legislative texts. xlm-r and albertina (br) also showed competitive performance across various entity types, highlighting their versatility. the focus on european portuguese-specific entities allowed albertina (pt) to outperform other models in the parish memories corpus. additionally, increasing the model size significantly improved performance in this dataset. in the brazilian legislative texts corpus, the diversity and specificity of the entities, coupled with the number of training examples, greatly influenced the models’ performance. larger models generally performed better, especially in minority and specific entity classes. moreover, the inclusion of legal data in albertina’s (pt) pre-training likely contributed to its strong performance in this corpus. 7 conclusion we investigated the influence of portuguese language variants in pre-trained language models on named entity recognition (ner) within specialised domains. our experiments used two distinct corpora: a historical corpus in european portuguese and a legislative corpus in brazilian portuguese. we evaluated models pre-trained specifically for brazilian portuguese (bertimbau and albertina pt-br), european portuguese (albertina pt-pt), and multilingual contexts (xlm-r). our analysis delved qualitatively into specific class examples, semantics, and compositions. several conclusions can be drawn based on our findings. models tailored to the textual and linguistic context, such as albertina for european portuguese, demonstrated superior performance in ner tasks, particularly in the parish memories corpus. albertina excelled in precision and recall, extracting entities from 18th-century texts because of its linguistic alignment. conversely, larger models such as xlm-r and albertina showed an enhanced precision-recall balance in the brazilian legislative texts, underscoring the critical role of model size in handling fine-grained categories. future research should focus on granular error analysis and interpretation. understanding model improvements and challenges is crucial, particularly in ambiguity and linguistic complexity. exploring the performance of techniques such as semi-supervised learning and transfer learning methods will further advance the field of ner in specific textual domains. notes 1.https://inception-project.github.io. 2.https://pytorch.org/. 3.https://huggingface.co/neuralmind/bert-large-portuguese-cased. 4.english translation: i highlight that in the 90 s, jornal diário catarinense carried out a popular survey that placed it among the 20 santa catarina citizens of the century. references akbik, a., bergmann, t., blythe, d., rasul, k., schweter, s., vollgraf, r.: flair: an easy-to-use framework for state-of-the-art nlp. in: naacl 2019, 2019 annual conference of the north american chapter of the association for computational linguistics (demonstrations), pp. 54–59 (2019) google scholar  albuquerque, h.o., et al.: ulyssesner-br: a corpus of brazilian legislative documents for named entity recognition. in: international conference on computational processing of the portuguese language, pp. 3–14. springer (2022) google scholar  luz de araujo, p.h., de campos, t.e., de oliveira, r.r., stauffer, m., couto, s., bermejo, p.: lener-br: a dataset for named entity recognition in brazilian legal text. in: computational processing of the portuguese language: 13th international conference, propor 2018, canela, brazil, september 24–26, 2018, proceedings 13, pp. 313–323. springer (2018) google scholar  bonifacio, l.h., vilela, p.a., lobato, g.r., fernandes, e.r.: a study on the impact of intradomain finetuning of deep language models for legal named entity recognition in portuguese. in: intelligent systems: 9th brazilian conference, bracis 2020, rio grande, brazil, october 20–23, 2020, proceedings, part i 9, pp. 648–662. springer (2020) google scholar  brito, m., pinheiro, v., furtado, v., neto, j.a.m., bomfim, f.d.c.j., da costa, a.c.f., silveira, r.: cdjur-br-uma coleção dourada do judiciário brasileiro com entidades nomeadas refinadas. in: anais do xiv simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 177–186. sbc (2023) google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale. arxiv preprint arxiv:1911.02116 (2019) correia, f.a., et al.: fine-grained legal entity annotation: a case study on the brazilian supreme court. inf. process. manage. 59(1), 102794 (2022) article  mathscinet  math  google scholar  filho, j.a.w., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the 11th international conference on language resources and evaluation, pp. 4339–4344 (2018). http://www.lrec-conf.org/proceedings/lrec2018/summaries/599.html garcia, e.a., et al.: robertalexpt: a legal roberta model pretrained with deduplication for portuguese. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 374–383 (2024) google scholar  grilo, s., bolrinha, m., silva, j., vaz, r., branco, a.: the bdcamoes collection of portuguese literary documents: a research resource for digital humanities and language technology. in: proceedings of the twelfth language resources and evaluation conference, pp. 849–854 (2020) google scholar  lample, g., conneau, a.: cross-lingual language model pretraining. arxiv preprint arxiv:1901.07291 (2019) liu, y., et al.: roberta: a robustly optimized bert pretraining approach. arxiv preprint arxiv:1907.11692 (2019) nunes, r.o., balreira, d.g., spritzer, a.s., freitas, c.m.d.s.: a named entity recognition approach for portuguese legislative texts using self-learning. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 290–300 (2024) google scholar  oleques nunes., r., spritzer., a., dal sasso freitas., c., balreira., d.: out of sesame street: a study of portuguese legal named entity recognition through in-context learning. in: proceedings of the 26th international conference on enterprise information systems volume 1: iceis, pp. 477–489. insticc, scitepress (2024). https://doi.org/10.5220/0012624700003690 rodrigues, j., gomes, l., silva, j., branco, a., santos, r., cardoso, h.l., osório, t.: advancing neural encoding of portuguese with transformer albertina pt. in: epia conference on artificial intelligence, pp. 441–453. springer (2023) google scholar  santos, d., cardoso, n.: a golden resource for named entity recognition in portuguese. in: international workshop on computational processing of the portuguese language, pp. 69–79. springer (2006) google scholar  santos, d., seco, n., cardoso, n., vilela, r.: harem: an advanced ner evaluation contest for portuguese. in: calzolari, n., et al. (eds.) proceedings of the 5th international conference on language resources and evaluation (lrec’2006)(genoa italy 22–28 may 2006) (2006) google scholar  santos, j., cameron, h.f., olival, f., farrica, f., vieira, r.: named entity recognition specialised for portuguese 18th-century history research. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 117–126 (2024) google scholar  santos, r., et al.: fostering the ecosystem of open neural encoders for portuguese with albertina pt-* family (2024) google scholar  silva, p., franco, a., santos, t., brito, m., pereira, d.: cachacaner: a dataset for named entity recognition in texts about the cachaça beverage. lang. resour. eval. 58(4), 1315–1333 (2023) article  google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: proceedings of the 9th brazilian conference on intelligent systems, bracis (2020) google scholar  souza, f., nogueira, r.f., de alencar lotufo, r.: portuguese named entity recognition using bert-crf. corr arxiv:1909.10649 (2019) tjong kim sang, e.f.: introduction to the conll-2002 shared task: language-independent named entity recognition. in: coling-02: the 6th conference on natural language learning 2002 (conll-2002) (2002). https://aclanthology.org/w02-2024 vieira, r., olival, f., cameron, h., santos, j., sequeira, o., santos, i.: enriching the 1758 portuguese parish memories (alentejo) with named entities. j. open humanit. data 7, 20 (2021) article  math  google scholar  zanuz, l., rigo, s.j.: fostering judiciary applications with new fine-tuned models for legal named entity recognition in portuguese. in: international conference on computational processing of the portuguese language, pp. 219–229. springer (2022) google scholar  download references acknowledgements this work has received funds from the coordenação de aperfeiçoamento de pessoal de nível superior brazil (capes) finance code 001, the brazilian funding agency cnpq, and the portuguese science foundation fct, in the context of the projects ceecind/01997/2017 and uidb/00057/2020 https://doi.org/10.54499/uidb/00057/2020. author information authors and affiliations federal university of rio grande do sul, porto alegre, brazil rafael oleques nunes, andre spritzer, dennis giovani balreira & carla maria dal sasso freitas university of vale do rio dos sinos, são leopoldo, brazil joaquim santos cidehus portalegre polytechnic university, portalegre, portugal fernanda olival & helena freire cameron cidehus university of évora, évora, portugal renata vieira authors rafael oleques nunesview author publications search author on:pubmed google scholar joaquim santosview author publications search author on:pubmed google scholar andre spritzerview author publications search author on:pubmed google scholar dennis giovani balreiraview author publications search author on:pubmed google scholar carla maria dal sasso freitasview author publications search author on:pubmed google scholar fernanda olivalview author publications search author on:pubmed google scholar helena freire cameronview author publications search author on:pubmed google scholar renata vieiraview author publications search author on:pubmed google scholar corresponding author correspondence to rafael oleques nunes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper nunes, r.o. et al. (2025). assessing european and brazilian portuguese llms for ner in specialised domains. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_15 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords named entity recognition large language models portuguese language variants publish with us policies and ethics profiles rafael oleques nunes view author profile andre spritzer view author profile renata vieira view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature dynamicity analysis in the selection of classifier ensembles parameters | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper dynamicity analysis in the selection of classifier ensembles parameters conference paper first online: 30 january 2025 pp 352–367 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) dynamicity analysis in the selection of classifier ensembles parameters download book pdf download book epub jesaías carvalho pereira silva  orcid: orcid.org/0000-0002-6586-83409, anne magaly de paula canuto  orcid: orcid.org/0000-0002-3684-38149 & araken de medeiros santos  orcid: orcid.org/0000-0002-6407-475x10  part of the book series: lecture notes in computer science ((lnai,volume 15412)) included in the following conference series: brazilian conference on intelligent systems 477 accesses 1 citation abstract over the years, significant progress has been made in the realm of classifier ensembles research. several methods to enhance their efficiency have been proposed, applicable to both homogeneous and heterogeneous ensemble structures. a key challenge in employing classifier ensembles lies in determining their structure (hyper-parameters). basically, the ensemble structure selection can be done in two different ways, static and dynamic selection. unlike static selection, dynamic selection defines the ensemble structure for each testing instance. different dynamic selection methods have been proposed in the literature, mainly for ensemble members and features, but very little effort has been done to propose dynamic selection methods for combination methods. therefore, it is important to evaluate the impact of a dynamic selection of combination methods or both (methods and members) in the creation of robust classifier ensembles. in this paper, an exploratory analysis of dynamic selection of the main ensemble structure parameters will be performed. in order to do this, three different scenarios will be assessed: full static ensemble, partial dynamic ensemble and full dynamic ensemble. finally, an empirical analysis of these scenarios will be carried out. our findings show that the use of a full dynamic selection provides more robust classifier ensembles, in most of the analyzed cases. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others the dynamic selection of combination methods in classifier ensembles by region of competence chapter © 2023 dynamic ensemble selection using discriminant functions and normalization between class labels – approach to binary classification chapter © 2016 homogeneous ensemble selection experimental studies chapter © 2017 1 introduction in a classifier ensemble, the individual classifiers work together to create a more robust and accurate system for classifying patterns. these individual classifiers, known as ensemble members, operate in parallel, each one receiving the same input pattern and independently producing its own output. a combination method receives the members outputs and provides the global output of the system [1]. classifier ensemble is a two-step classification structure in which the main parameters are the individual classifiers (first step) and the combination method (second step). in these systems, one important aspect is the definition of the ensemble structure. several studies have proposed different ways to define the ensemble structure such as: optimization techniques, meta-learning, among others [2, 3]. however, most of these studies are related to the selection of individual classifiers. we can find some studies that investigate efficient methods for combining classifiers in ensemble systems, such as in [4,5,6,7]. the selection of ensemble parameters (classifiers and/or combination methods) can be static or dynamic. in static selection, the ensemble structure is defined during the training phase and remains fixed throughout. in contrast, dynamic selection adapts the ensemble structure for each test instance, often enhancing predictive performance [4, 8]. dynamic selection can be applied to both classifiers and combination methods, with most studies focusing on the dynamic selection of classifiers [6, 7, 9]. this is often based on techniques such as region of competence, hyper-boxes, or meta-learning. however, very little has been done to define a fully dynamic selection, making the selection of the ensemble parameters an automatic process. to advance the design of efficient classifier ensembles, this paper conducts an exploratory analysis of integrating dynamic selection into the main ensemble parameters. in this investigation, the dynamic selection will be applied in one ensemble parameter (classifier or combination) and in both parameters. the main aim of this analysis is to assess the impact of the dynamic selection in the two most important parameters of a classifier ensemble. in other words, to analyze whether the dynamic selection leads to more efficient ensembles when applied to individual classifiers, combination methods or in both parameters at the same time. in this analysis, three different scenarios will be evaluated. in the first scenario, the individual classifiers will be selected dynamically while the combination method will be selected statically. for this scenario, three well-known des (dynamic ensemble system) methods are used, which are: knora-eliminate (knora-e [4]) fh-des [10] and meta-des [5]. in the second scenario, the combination method is selected dynamically while the individual classifiers will be selected statically. for this scenario, a dynamic fusion method is presented. finally, in the last scenario, both individual classifiers and combination method will be selected dynamically. as baseline, a full static ensemble structure will also be investigated in order to assess the impact of the dynamic selection in the performance of the classifier ensembles. all the ensemble structures will be evaluated using 20 classification datasets. this paper is divided into 6 sections and its organization is defined as followed. section 2 describes the theoretical concepts and related work of this paper, while sect. 3 describes in more detail the ensemble structures to be used in the exploratory analysis of this paper. section 4 presents the experimental methodology of the empirical analysis, while its results are presented in sect. 5. finally, sect. 6 describes the final remarks of this paper. 2 theoretical concepts and related work 2.1 state of the art there are several studies that investigate the dynamic selection of ensemble structure, mainly for ensemble members [6, 7] and features [11] and both of them [9, 12]. regarding ensemble members, in [6], for instance, a new method for dynamic ensemble member selection is presented. in this method, the confidence of the base classifiers during the classification and its general credibility is used as selection criterion. another interesting way is to use region of competence as selection criterion, making it possible to improve the combination of classifiers, in which the most competent ones in a certain region are selected. the use of region of competence as selection criterion helps to maximize results by focusing only on the most competent classifiers, and examples can be found in knora-e [4] and meta-des [5]. in terms of dynamic feature selection, in [11], a dynamic feature selection approach was proposed. the main aim of this approach is to select a different subset of features for one instance or a group of instances. the main goal of this approach is to explore the full potential of all instances in a classification problem. in [9], an initial study on how to combine these two dynamic selection techniques was performed. according to the authors, an improvement in performance was detected with the use of this integrated dynamic selection technique. already in [13], the authors presented an initial method of classifier fusion using k-nearest neighbors (knn). although the results are promising, there is no general comparison with static ensembles. although there are several studies to propose dynamic selection of ensemble members and feature selection, very little has been done in order to propose efficient dynamic selection of combination methods. this paper tries to bridge this gap and it proposes a dynamic selection method based on region of competence. 2.2 classifier ensembles it is well-known that there is not a single classifier which can be considered optimal for all problem domains [1]. therefore, it is difficult to select a good single classifier which provides the best performance in practical pattern classification tasks [14]. in this context, classifier ensembles have emerged as an efficient classification structure since it combines the advantages and overcomes the limitations of the individual classifiers. thus, studies have shown that classifier ensemble provides better generalization and performance ability, when compared to the individual classifiers [14, 15]. in a classifier ensemble, an input pattern is presented to all individual classifiers [16, 17], and a combination method combines their outputs to produce the overall output of the system [1]. the machine learning literature has ensured that diversity plays an important role in the design of ensembles, contributing to their accuracy and generalization [1]. one important issue regarding the design of classifier ensembles involves the appropriate selection of its structure (individual classifiers and combination methods) [18]. as previously mentioned, there are basically two main selection approaches, static and dynamic. in this paper, we will focus on the dynamic approach. the next subsection will describe some existing dynamic selection methods that will be used in this paper. 2.3 dynamic ensemble member selection the dynamic ensemble selection (des) methods perform the dynamic selection of a subset of classifiers to classify each test instance. the selection of the classifier subset is done through the use of a selection procedure and each des method has its own procedure. there are several des methods proposed in the literature. in this paper, we will use two well-known des methods, knora-e and meta-des. knora-e. knora [4] is a well-known des method and it seeks to find the best subset of classifiers for a given test instance. it applies a k-nearest neighbors method. the neighbors of a testing instance are selected from the validation set and the competence of each classifier is calculated. based on a certain selection criterion, the classifier subset is selected. knora-e is a knora-based method, and the selection criterion is to select a set of classifiers formed only by the classifiers that correctly classify all k neighbors of a testing instance. in the case where no classifier can correctly classify all k neighbors, the k value is decremented by one and this is done until at least one classifier can be selected [4]. meta-des. the meta-des [5] is a des method that uses the idea of selection using meta-learning. in this method, a meta-problem is created to determine whether a classifier is competent for a given test instance. according to [12], the meta-des method uses five criteria for extracting meta-features in order to establish the new region of a meta-problem. after that, a meta-classifier is trained, based on the defined meta-features. this meta-classifier is then used to identify whether a classifier is competent or not to classify a testing instance. classifiers that are labeled as competent will be selected to compose the ensemble to classify the test instance. fh-des. the fh-des is also a des method, but based on fuzzy hyperboxes [19] to solve the local sensitivity problem using knn. hyperboxes represent a group of samples using maximum and minimum corners. they are formed based on regions that classifiers work well or areas of competence, but can also be applied to regions that present poor classifications or areas of incompetence. in the latter case, the classifiers whose hyperboxes have a lower degree of relevance will be further away from the query sample, therefore, they will be more competent to classify the query sample [10]. the tool can be applied in three ways, using a sum rule based on hyperbox weights, performing only the selection of competent classifiers, and hybrid, in which it uses the weights and selection of classifiers. 3 proposal 3.1 dynamic selection scenarios in order to carry out the exploratory analysis of the dynamic selection in classifier ensemble, three scenarios are defined, which are described as follows. 1. full static ensemble selection (fses): in the first scenario, we will have full static ensembles. in other words, all ensemble parameters are selected statically. 2. partially dynamic ensemble selection (pdes): in the second scenario, we will have partial dynamic ensembles, having the dynamic selection in only one parameter: ensemble members or fusion, but not both of them. therefore, this leads to two sub-scenarios: 1. partial dynamic member selection (p-dms): in this case, only the ensemble members (individual classifiers) will be dynamically selected. 2. partial dynamic fusion selection (p-dfs): in this cases, only the combination method will be dynamically selected. the dynamic fusion selection method is presented in sect. 3.2. 3. full dynamic ensemble selection (fdes): in this scenario, full dynamic ensembles, both parameters (ensemble member and combination methods) will be chosen dynamically for each new instance of teste, as well as which fusion. these three scenarios were selected because they gradually increase the dynamicity of the selection of the ensemble parameters. therefore, we aim at evaluating the impact of the dynamic selection in the design of robust classifier ensembles. 3.2 the dynamic fusion selection method in this paper, in order to achieve dynamicity in the selection of combination methods, we present the dfs (dynamic fusion selection) method, which is an algorithm that dynamically selects the combination methods from a set of methods. in other words, for each test instance, the most appropriate combination methods is selected. the selection is carried out in the testing phase. dfs calculates the competence of each combination method with respect to the presented test instance. algorithm 1 presents the main steps of the dfs processing. as it can be observed, dfs has two main parameters: the number of neighbors and the set of combination methods, which comes from the algorithm input. the number of neighbors determines the size of the neighborhood used to calculate the selection criterion. the set of combination methods define the methods that will be used in the dynamic selection method. competence is calculated based on the accuracy of the combination methods using the neighbors of the test instance (line 5). if there is a tie in the local accuracy, the number of neighbors is increased by 1 (k = k + 1) (line 2) until one combination method is selected. there is still a tie, the combination methods with accuracy equal to the maximum accuracy are selected (line 12). finally, dfs can be applied to a pool of classifiers selected statically or dynamically. in this paper, this method will be used in two dynamic selection scenarios, partially dynamic selection and the full dynamic selection one. 4 the experimental methodology in this section, the main aspects of the empirical analysis will be described, mainly the used datasets, as well as its methods and materials. 4.1 datasets this paper uses datasets extracted from the uci machine learning repository. table 1 presents some characteristics of these datasets, including the number of instances (inst), the number of attributes (att) and the number of classes (class). each dataset is divided into training, validation1, validation2 and testing sets, in a proportion of 50%, 16.7%, 16.7%, and 16.6%, respectively. the training set is used to train the pool of classifiers (ensemble members). the testing set is used to assess the performance of the classifier ensembles. the validation2 set is used to train the trainable combination methods (neural networks and naive bayes) while the validation1 set is used to obtain the selection criteria of the presented dynamic member selection methods. table 1. description of the used datasets.full size table this division is performed 30 times and the presented results of each ensemble configuration represent the average values over these 30 values. 4.2 methods and materials in this paper, all classifier ensembles used decision trees as individual classifiers, generated through the bagging method. in all analyzed scenarios, 6 different pool sizes are used, which are: 5, 10, 15, 20, 25, and 30 individual classifiers. the remaining parameter values were defined through extensive grid search experimental evaluation. for the fses scenario, the classifier ensembles will be evaluated using 12 different combination methods, which are: majority vote, sum, max, min, geometric mean, naive bayes, weigthed sum, weigthed vote, edge and three multilayer perceptron (mlp) versions: hard, soft, and soft-class. the three neural networks (nn) versions differ in the input information received by the ensemble members. in the hard version, the ensemble member provides only the winner class for the testing instance. in other words, this mlp version is trained and tested using only the winner class of each ensemble member. in the other two mlp versions, the prediction probability for each class is used. in this sense, the prediction probability for each class is provided for both mlp versions. additionally, the weighted sum and weighted vote methods use weights in their functioning. the used weight is 1/(distance-of-classes), and it is applied to the voting procedure in the weighted vote as well as to the outputs of the classifiers in the weighted sum method. for the first case of the pdes scenario (p-dms), three well-known methods are used knora-e [4], meta-des [5] and fh-des [19]. in the second case of the pdes scenario (p-dfs), the dynamic fusion method presented in sect. 4.2 is used to dynamically select the most suitable combination method for each testing instance. for the fdes scenario, all three methods used in the f-dms will be combined with the method for the p-dfs case, leading to 3 pdes variations (knora-e with dynamic fusion selection, meta-des with dynamic fusion selection, and fh-des with dynamic fusion selection in fusion). it is important to highlight that the dfs method, and some of the dms methods (knora-e and meta-des) use the idea of region of competence. in this sense, the same number of neighbors are used for all cases, in the selection of the combination method (dfs) and in the dynamic member selection (knora-e and meta-des), being defined in each iteration by means of a grid search between the values of 3, 7 and 11, using knn in the dataset validation 1. for the fh-des, there is no idea of a region of competence, but rather the use of hyper-boxes, which were created based on the areas of incompetence as presented by the authors in [10]. the results of all analyzed methods will be evaluated using the friedman statistical test [20]. the friedman test is used to be able to state the hypothesis that the k-related observations derive from the same population (similar performance) or not (superiority in performance). in this test, the significance level used was set to 0.05. hence, if the p-value is less than the established value, the null hypothesis is rejected, with a confidence level greater than 95%. in cases where a statistically significant difference is detected, the nemenyi post-hoc test is applied [21]. in order to present the obtained results by the post-hoc test, the critical difference diagram (cd) [21] is used. this diagram was selected in order to have a visual illustration of the statistical test, making it easier to interpret the obtained results. 5 the obtained results this section presents the results obtained by the empirical analysis, in terms of accuracy levels. first, all full static configurations is evaluated. then we evaluate separately the two cases of the pdes scenario, along with the best full static configuration. finally the fdes scenario is evaluated, along with the best full static configuration, the best dynamic member fusion configuration, and the best dynamic fusion configuration. in order to define the best configuration, the critical difference diagram is used. if there is no statistical difference between the best techniques, then the one with the highest average is selected. 5.1 full static ensembles tables 2, 3, 4, 5 and 6 present the accuracy results of all analyzed methods. as mentioned previously, 6 configurations were made for pool sizes; each configuration was executed 30 times. therefore, the values in tables 2, 3, 4, 5 and 6 represent the average of all 180 results. additionally, the last row of all tables represent the overall accuracies for all 20 datasets. finally, the numbers in bold represent the highest accuracy for each data set. for the fses scenario (table 2), it can be seen that all methods (except edge fusion) provided the highest accuracy levels in at least three datasets. in a general perspective, it can be observed that the ensembles combined by sum (second column) presented the highest overall accuracy (89.09%) and highest accuracy in 8 out of 20 datasets. they were followed by all three nns (6 data sets) and majority vote (5 data sets). in order to evaluate the obtained results from a statistical point of view, the friedman test [20] was applied to verify if there are statistical differences among all ensemble classifiers. the friedman test was applied to all 12 fses configurations. therefore, the statistical test detected statistical differences among all analyzed methods, with a p-value < 0.05. in this sense, the post-hoc test was applied, and the results are presented in the critical difference diagram [21], depicted in fig. 1. table 2. score of fusion methods in full static ensemble.full size table fig. 1. critical difference diagram for full static ensemble full size image as it can be seen in this figure, the cd diagram detected no statistically significant difference in the accuracy of sum, mlp soft, and mlp soft class. for the remaining configuration, they provided superior performance, detected by the statistical test. since no method stood out, sum-combined ensemble is selected the remaining tests since it achieved the best overall highest accuracy level. 5.2 the pdes scenario in this subsection, the results for the pdes scenario are presented. for a comparative analysis, in each table, the accuracy of the best fses configuration (fses-sum) is also presented. as mentioned previously, three well dynamic member selection are used. in this paper, table 3, 4 and 5 represent the values of knora-e, fh-des and meta-des, respectively. thus, for each method, the result of the p-dfs case is also presented, leading to a total of 14 analyzed methods. knora-e. for knora-e (table 3), it can be seen that the p-dfs method presented the best overall accuracy level (89.88%) and obtained the best results in 4 out of 20 datasets. however, the knora-e method combined by the majority vote method achieved the best results in 9 datasets, presenting the same average accuracy as knora-e combined by sum (89.85%) with 8 best results. weight fusion methods showed the worst results. the friedman test was then applied, which identified statistical differences with a p-value < 0.05. table 3. score of fusion methods for partial dynamic ensembles in knora-efull size table when applying the post-hoc test, the cd diagram, in fig. 2, results showed that knora-e sum and knora-e vote provided the best performance. however, these methods provide similar performance from a statistical point of view. for the other methods, the statistical test detected a superiority, in terms of accuracy, of these methods. the p-dfs and knora-e edge methods also showed good results, being superior to the remaining methods, from a statistical point of view. as there was not just one method that stood out, the knora-e vote method is selected for the best knora-e configuration. when comparing the p-dfs method and the best knora-e method (p-dms) method, although the p-dfs method provided the best overall accuracy, the statistical test showed that the use of dynamic selection in the ensemble members provided more robust ensembles. fig. 2. critical difference diagram for partial dynamic ensembles in knora-e. full size image fh-des. for fh-des, in table 4, it can be seen that fh-des sum presented the best overall result (90.51%) and obtained the best results in 4 of the 20 datasets. followed by the fh-des majority vote method with 90.49%, which obtained the best results in 3 datasets. table 4. score of fusion methods for partial dynamic ensembles in fh-des.full size table the fh-des mlp hard and weighted sum methods also presented the best results in 4 datasets; however, the results were slightly lower than the best fh-des methods. fusion by min and geometric mean provided the worst results. in order to check whether the accuracy levels derive from the same population, the friedman test identified statistical differences with a p-value < 0.05. in the post-hoc test, the critical difference diagram (fig. 3) in accuracy of fh-des sum, majority vote, and mlp soft class did not present statistically significant differences. the fh-des min and geometric mean presented the worst results, detected by the statistical test. once again, as there was not just one fh-des method that stood out in the statistical test, the fh-des sum method was selected to be the best fh-des method. fig. 3. critical difference diagram for partial dynamic ensembles in fh-des. full size image when comparing the p-dfs method and the best fh-des method (p-dms) method, we can observe a superiority of the p-dms case, showing that the use of dynamic selection in the ensemble members provided more robust ensembles. meta-des. in table 5, for meta-des, we can observe that meta-des sum and majority vote presented the best overall results (90.33%). meta-des sum obtained the best results in 8 of the 20 datasets. the meta-des vote method obtained the best results in 7 datasets, presenting a slightly lower accuracy than sum. the friedman test was then applied, which identified statistical differences, with a p-value < 0.05 in the tests. when applying the post-hoc test, the results of the cd diagram (fig. 4) did not show any statistically significant difference in the accuracy of meta-des sum and majority vote again, but providing higher levels of accuracy, when compared to the other methods, from a statistical point of view. table 5. score of fusion methods for partial dynamic ensembles in meta-des.full size table fig. 4. critical difference diagram for partial dynamic ensembles in meta-des. full size image when comparing the p-dfs method and the best meta-des method (p-dms) method, once again, we can observe a superiority of the p-dms case, showing that the use of dynamic selection in the ensemble members provided more robust ensembles. 5.3 the fdes scenario for the fdes scenario, the dynamic fusion methods were combined with meta-des, knora-e and fh-des, leading to 3 fdes configurations. for comparison purposes, the best 3 pdes configurations (one for meta-des, one for knora-e and one for fh-des) are also presented, along with p-dfs and the best fses configuration. table 6 presents the results of all evaluated methods. table 6. score of fusion methods in full dynamic ensembles.full size table from table 6, we can see that fdes (fh-des) achieved the best overall accuracy levels (90.86%), closely followed by fdes (meta-des) (90.75%), and then fdes (knora-e) (90.52). these methos delivered the best result in 5 out of 20 datasets each. then, the fh-des sum method is the best pdes case (90.51%), followed by meta-des vote and knora-e vote and then the p-dfs method. finally, the worst result was obtained by the fses sum method. as it can be seen, the use of dynamic selection on both ensemble members and combination methods provides the most robust classifier ensembles. figure 5 presents the cd diagram of the post-hoc test on the results of table 6. from this figure, it was possible to detect that the accuracy of all three fdes configurations showed the most accurate classifier ensembles. additionally, the statistical test detected superiority, in terms of accuracy of the fdes configurations, when compared to the remaining analyzed methods. the results obtained in fig. 5 corroborates the results of table 6, in which the use of dynamic selection on both ensemble members and combination methods provides the most robust classifier ensembles. when using dynamicity in the selection of one parameter, the dynamic selection of ensemble members provided the best results. finally, the static selection delivered the worst results, detected by the statistical test. fig. 5. cd diagram for full dynamic ensemble full size image 6 final remarks this paper proposed an exploratory analysis of the dynamic selection of the most important parameters of a classifier ensemble. in order to perform this analysis, three dynamic selection scenarios are defined, which are: fses (full static ensemble selection); pdes (partial dynamic ensemble selection) with two cases: p-dfs (partial dynamic fusion selection) and p-dms (partial dynamic member selection); and fdes (full dynamic ensemble selection). the main aim is to assess the impact of dynamic selection in the performance of classifier ensembles. through this exploratory analysis, it can be observed that the use of dynamic selection on both ensemble members and combination methods provides the most robust classifier ensembles. when using dynamicity in the selection of one parameter, the dynamic selection of ensemble members provided the best results. finally, the static selection delivered the worst results, detected by the statistical test. as future analysis, it is necessary to expand this analysis using different dynamic selection approach that are not based on region of competence. a statistical analysis comparing methods across different strategies is also needed. finally, this empirical study was limited to 20 classification datasets. it is important to perform an analysis with more robust datasets. references kuncheva, l.i.: combining pattern classifiers: methods and algorithms. [s.l.]: wiley (2004) google scholar  feitosa, a.a., canuto, a.m.p.: an exploratory study of mono and multi-objective metaheuristics to ensemble of classifiers. appl. intell. 48(2), 416–431 (2018) article  google scholar  kordík, p., cerný, j., frýda, t.: discovering predictive ensembles for transfer learning and meta-learning. mach. learn. 107(1), 177–207 (2018) article  mathscinet  google scholar  ko, a.h.r., sabourin, r., britto, a.s., jr.: from dynamic classifier selection to dynamic ensemble selection. pattern recogn. 41(5), 1718–1731 (2008) article  math  google scholar  cruz, r.m.o., et al.: meta-des: a dynamic ensemble selection framework using meta-learning. pattern recogn. 48(5), 1925–1935 (2015) article  math  google scholar  nguyen, t.t., et al.: ensemble selection based on classifier prediction confidence. pattern recogn. 100, 107104 (2020) article  math  google scholar  cruz, r.m.o., et al.: deslib: a dynamic ensemble selection library in python. j. mach. learn. res. 21(1), 283–287 (2020) math  google scholar  woods, k., kegelmeyer, w. p., bowyer, k.: combination of multiple classifiers using local accuracy estimates. [s.l.]: ieee (1997). 405–410 p. google scholar  dantas, c., et al.: instance hardness as a decision criterion on dynamic ensemble structure. in: 2019 8th brazilian conference on intelligent systems (bracis). ieee (2019) google scholar  davtalab, r., cruz, r.m.o., sabourin, r.: dynamic ensemble selection using fuzzy hyperboxes. in: 2022 international joint conference on neural networks (ijcnn), pp. 1–9. ieee (2022) google scholar  nunes, r.o., et al.: an unsupervised-based dynamic feature selection for classification tasks. in: 2016 international joint conference on neural networks (ijcnn). ieee (2016) google scholar  dantas, c. a.: an integration analysis of dynamic selection techniques for a classification system building. (ph.d. thesis) federal university of rio grande do norte (2021) google scholar  silva, j.c.p., de paula canuto, a.m., de medeiros santos, a.: the dynamic selection of combination methods in classifier ensembles by region of competence. in: international conference on artificial neural networks. springer, cham (2023) google scholar  dietterich, t.g.: ensemble methods in machine learning. in: kittler, j., roli, f. (eds.) mcs 2000. lncs, vol. 1857, pp. 1–15. springer, heidelberg (2000). https://doi.org/10.1007/3-540-45014-9_1 chapter  math  google scholar  silva, j. c. .p, et al.: ensemble classifiers in a serious game for medical students in clinical cases. in: proceedings of the 10th euro-american conference on telematics and information systems. association for computing machinery, pp. 1–5 (2020) google scholar  kuncheva, l.i.: a theoretical study on six classifier fusion strategies. ieee trans. pattern anal. mach. intell. 24(2), 281–286 (2002) article  math  google scholar  brown, g., et al.: diversity creation methods: a survey and categorisation. inf. fus. 6(1), 5–20 (2005) article  math  google scholar  canuto, a.m.p.: investigating the influence of the choice of the ensemble members in accuracy and diversity of selection-based and fusion-based methods for ensembles. pattern recogn. lett. 28(4), 472–486 (2007) article  math  google scholar  simpson, p.k.: fuzzy min-max neural networks-part 1: classification. ieee trans. neural networks 3(5), 776–786 (1993) article  math  google scholar  friedman, m.: a comparison of alternative tests of significance for the problem of m rankings. ann. math. stat. 11(1), 86–92 (1940) article  mathscinet  math  google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) mathscinet  math  google scholar  download references acknowledgement the authors would like to thank the financial support provided by the coordenação de aperfeiçoamento de pessoal de nıvel superior do brasil (capes), during the development of this work. author information authors and affiliations federal university of rio grande do norte, natal, rn, 59.078-970, brazil jesaías carvalho pereira silva & anne magaly de paula canuto federal university of rural do semi-árido, angicos, rn, 59.515-000, brazil araken de medeiros santos authors jesaías carvalho pereira silvaview author publications search author on:pubmed google scholar anne magaly de paula canutoview author publications search author on:pubmed google scholar araken de medeiros santosview author publications search author on:pubmed google scholar corresponding author correspondence to jesaías carvalho pereira silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, j.c.p., canuto, a.m.d.p., santos, a.d.m. (2025). dynamicity analysis in the selection of classifier ensembles parameters. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15412. springer, cham. https://doi.org/10.1007/978-3-031-79029-4_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79029-4_25 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79028-7 online isbn: 978-3-031-79029-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords classifier ensembles dynamic ensemble selection individual classifiers combination methods publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature deepagé: answering questions in portuguese about the brazilian environment | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deepagé: answering questions in portuguese about the brazilian environment conference paper first online: 28 november 2021 pp 419–433 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) deepagé: answering questions in portuguese about the brazilian environment download book pdf download book epub flávio nakasato cação  orcid: orcid.org/0000-0003-4771-600910, marcos menon josé  orcid: orcid.org/0000-0003-4663-438610, andré seidel oliveira  orcid: orcid.org/0000-0001-6551-691110, stefano spindola  orcid: orcid.org/0000-0002-3604-934610, anna helena reali costa  orcid: orcid.org/0000-0001-7309-452810 & … fabio gagliardi cozman  orcid: orcid.org/0000-0003-4077-493510  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1256 accesses 6 citations abstract the challenge of climate change and biome conservation is one of the most pressing issues of our time—particularly in brazil, where key environmental reserves are located. given the availability of large textual databases on ecological themes, it is natural to resort to question answering (qa) systems to increase social awareness and understanding about these topics. in this work, we introduce multiple qa systems that combine in novel ways the bm25 algorithm, a sparse retrieval technique, with ptt5, a pre-trained state-of-the-art language model. our qa systems focus on the portuguese language, thus offering resources not found elsewhere in the literature. as training data, we collected questions from open-domain datasets, as well as content from the portuguese wikipedia and news from the press. we thus contribute with innovative architectures and novel applications, attaining an f1-score of 36.2 with our best model. f. n. cação and m. m. josé—these authors contributed equally to the work. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others tracking environmental policy changes in the brazilian federal official gazette chapter © 2022 climate change and the individual: the brazilian perspective chapter © 2021 large-scale differences in microbial biodiversity discovery between 16s amplicon and shotgun sequencing article open access 31 july 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. machine translation machine learning natural language processing (nlp) portuguese problem solving artificial intelligence 1 introduction developing technologies and public policies to address the challenges of climate change is a multifaceted task. the united nations (un) resolution with 17 sustainable development goals, whose focus is on the protection of the global environment and increased general prosperity, contains a call for “climate action”, with five urgent goals to fight climate change; raising collective awareness belongs to these goals. brazil has a central role in that debate as it has some of the richest and most important biomes in the world, and at the same time, it faces difficulties in curbing deforestation and illegal forest fires [20]. there is much to be done still in informing a poorly educated but interested population on issues related to the environment [16]. we can take advantage of the latest advances in natural language processing (nlp) and question answering (qa) so as to better inform the population on environmental issues. while natural language processing of textual databases has been used to deal with environmental challenges [3, 8, 12], exploration of such databases in brazilian portuguese remains underdeveloped, to say the least. on the dataset availability side, [1] recently released pirá, the first bilingual portuguese-english crowdsourced qa dataset about the ocean and, in particular, the brazilian coast, based on un reports and abstracts from scientific papers. in english, climate-fever is intended to serve as a fact-checking dataset for claims related to climate change [5]. for topic detection, climatext is a dataset that leverages manual annotation of labels on texts extracted from wikipedia and the official u.s. documents via active learning [17]. furthermore, there is a large gap in the availability of annotated nlp datasets focused on environmental or climate issues, despite the urgency of the topic. in this work, we start to fill this gap by putting together qa systems that enhance existing architectures and that are built from a knowledge base (kb) consisting of 17k wikipedia articles in portuguesefootnote 1 and 29k news.footnote 2 the combination of encyclopedic knowledge and recent news lets us capture the public’s changing interests as they are reflected in newspapers. due to the scarcity of qa datasets specifically related to environment and climate in brazil and the high cost of manually generating such a dataset, we have also filtered the paq (probably asked questions) dataset [11], a massive open-domain qa dataset with 65 million automatically-generated question/answer pairs (qa-pairs); we thus obtained a large set of qa-pairs that was then translated. we leverage insights from the transformer architecture [18] and self-attention mechanisms [2], as well as insights on domain-specific fine-tuning [14]. our reader model is based on ptt5 architecture [4], a sequence-to-sequence t5 network [14] model pre-trained in a corpora in portuguese. to incorporate the databases in the production of answers, we used the bm25 algorithm [15], the state-of-the-art in sparse information retrieval, to add context blocks to the question then submitted to t5 for fine-tuning. this dual system, composed of a retrieval module and a neural reader model, has been explored in the qa literature [7, 10] and has some advantages over systems with only a neural module. first, it reduces the document load that the reader needs to process because the retrieval system pre-selects the k best passages in the corpus before submitting them further. second, the possibility of including new factual information as input from the subsequent reader. third, it potentially generates less hallucination, a very common problem in text generated by neural networks. in short, our main contribution is a qa system focused on the environment in brazil in portuguese, developed in two schemes. one scheme consists of only a reader module containing a language model; the other consists of a reader equipped with a powerful retriever with access to documents indicated previously (wikipedia, news, etc.). another contribution is the filtered and translated corpus of qa-pairs related to our domain; the fundamental idea there was to extract these pairs from a massive open-domain dataset, thus avoiding the manual creation of a new set of qa-pairsfootnote 3. as far as we know, we are the first to design a qa system and a dataset on the brazilian environment in this way. in the following sections, we explain how the models work and which settings we used in our experiments. we describe the process of building and filtering the knowledge bases: wikipedia categories in portuguese, and newspaper content. we also explain how we enhanced our qa dataset on the environment of brazil using a massive open-domain qa dataset. then we present the different experiments we ran, discuss the main results, and analyze possible improvements as future work. fig. 1. qa system with only ptt5 as a reader, without a retriever module. two versions of this reader were tested: (a) without fine-tuning, in which the model could only rely on knowledge saved in the network’s own weights (b) another with fine-tuning on the qa-pair training set. full size image fig. 2. qa system with both the retriever (bm25) and the reader (ptt5) modules working together. again, two versions of this system were tested: (a) without fine-tuning (b) with fine-tuning on the qa-pair training set. full size image 2 models and architectures in this work, we have two basic model architectures. the first one is illustrated in fig. 1: without any additional supporting documents, the model answers the posed question by only accessing the information stored in its own parameters. we call this architecture a reader-only one. so, for each question \(q_j\), the reader generates another sequence, the answer, \(a_j\). the reader can thus answer questions without text support; in our implementation, it is based on t5, a language model that can generate its own responses from scratch (that is, it does not rely on extracting it from a passage of text). in the second architecture, we add a retriever module before the reader. in this case, the reader has access to an external knowledge base, in addition to the information saved in its parameters. we resorted to wikipedia and news from mainstream newspapers. for each question \(q_j\), the retriever searches for the most relevant k passages \(\{p_1, p_2, ..., p_k\}\) in the corpus. then the original question is concatenated to these passages, producing a reformulated question \(q_j'= [q_j, p_1, p_2, ..., p_k]\), which, finally, serves as input to the reader, in place of \(q_j\), to generate an answer \(a_j\). the scheme is represented in fig. 2. in either case, the reader may or may not be fine-tuned with the qa-pairs \((q_j, a_j)\) from the problem domain. this is expressed by options (b) and (a) presented at the bottom of the figs. 1 and 2. we next discuss in more detail the technical aspects of the reader and the retriever we have implemented. 2.1 bm25 as the retriever bm25 [15] is an algorithm that estimates the relevance of documents from a set given a query. it is the state-of-the-art sparse retrieval technique, defined as a function of query terms frequency, document length, average document length, and the number of documents containing the query term. we applied bm25 to retrieve sentences of about 100 words from all sentences of the kb. the query is defined as the posed question \(q_j\). 2.2 ptt5 as the reader ptt5 [4], an encoder-decoder transformer, was pre-trained on the brwac brazilian portuguese website corpus [19] for the task of masked language modeling, where tokens from the corpus are masked so that the model has to predict them. the model is based on t5, which is derived from the original encoder-decoder transformer by vaswani et al. [18], characterized by several blocks of self-attention layers concatenated to feed-forward networks. we applied the “base” version of ptt5 with a portuguese vocabularyfootnote 4, with 12 layers and 12 attention heads, with a total of 220m trainable parameters. 3 dataset generation we built a dataset based on a set of textual documents and a set of qa-pairs. the kb has two main sources of texts, the brazilian portuguese wikipedia, and newspapers. we describe next how they were collected (as depicted in fig. 3). fig. 3. to build the kbs (upper blue section), wikipedia articles in portuguese from the category “environment of brazil” were processed, filtered and loaded into a database (1); newspaper news were filtered by keyword, scraped and also loaded into a database (2). finally, the two databases were integrated and shuffled to produce a third database (3), the “full” version. all three were used in different experiments. at the bottom of the figure (orange section), the paq filtering process, qa massive open-domain dataset, is described. using regex with key phrases related to the environment in brazil, we filtered 14k qa-pairs and then translated them into portuguese using google translation api (4). (color figure online) full size image 3.1 filtering articles from wikipedia in portuguese wikipedia is divided in such a way that articles are associated with categories. a category, on the other hand, is associated with articles and other categories that restrict even more the subject, called subcategories. to access this information, a sql table associating article and subcategory identifiers to category names is available for download at wikimedia’s dumps page.footnote 5 thus, to obtain several articles associated with the brazilian environment, we applied a recursive script that performs a breadth-first search of articles on subcategories, starting from an arbitrary category title. the algorithm stops when the desired number of articles is reached. we searched by taking the starting point “environment of brazil” (freely translated from “meio ambiente do brasil”) and obtained 17k wikipedia articles associated with the subject (hereafter abbreviated as “wiki” for simplicity). 3.2 scraping and filtering news from the biggest brazilian newspapers to build the news base, we scraped and processed news linked to pre-selected keywords in the three biggest newspapers in the country: folha de s.paulo, estadão and o globo. due to the limited value of this type of text over time, we kept only news from january 2018 on, as this is the beginning of the current federal government in brazil. we downloaded the headline and body of each article and, after final cleaning and pre-processing, we ended up with 29k news (we will refer to this database as “news”). to select the news that would be scraped, we first carefully crafted a list of keywords that are strongly related to the environment in the country. we then use the native search engines on each newspaper’s website to inject these keywords, list the search results, and download them via webscraping techniques. to minimize the number of false positives, i.e. news that is related to a certain keyword but is linked to news from a different subject that is not of our interest, we also excluded articles related to a set of specific words for some keywords. more details on this selection can be found in appendix b. due to latencies and limitations inherent to the scraping process of websites, there is no guarantee that all news related to a particular term has been downloaded. however, we obtained large numbers related to each keyword, which suggests good coverage, as illustrated in the fig. 4. fig. 4. citation count of each environmental category per year since 2018, considering the three biggest newspapers in brazil. full size image 3.3 filtering and translating the paq qa dataset to obtain an appreciable number of qa-pairs to fine-tune the models, we chose to filter a large-scale open-domain qa dataset with keywords that should and should not be in the questions or answers; this query with multiple rules was carefully hand-crafted and searches were done with regex expressions.footnote 6 initially, we applied this query to filter the ms marco v1 [13] training dataset, which is composed of questions from real users made in bing, with human-annotated answers and contained 80.142 qa-pairs after eliminating unanswered questions. however, due to many constraints we imposed on the filter to avoid false positive and false negative pairsfootnote 7, we obtained a return rate of only 0.037%, which corresponded to 30 pairs, an insufficient amount of data to fine-tune our models. relaxing the qa-pair filter did not generate a significant increase in this value. assuming this rate would be similar to other qa datasets based on user queries on search engines, such as google’s natural questions [9], we decided to filter the paq dataset, as it would be the only one capable of providing several training pairs of about three orders of magnitude greater than the one obtained from ms marco v1, which was what we desired at least. in fact, with a rate of return of about 0.024%, we got 14,386 qa-pairs after the filter. as shown in the plot 5, the filtering process did not generate a significant loss of quality for the qa-pairs when compared to the original paq. in addition to the quantitative evaluation that demonstrates that our filtered qa-pairs are as good as those in the general paq base (which already have high-quality [11]), we also performed a manual, qualitative inspection on a sample of 50 instances of our set of qa-pairs. they were evaluated by a human annotator, who answered three questions for each filtered and translated qa-pair: “is the domain adherent?”, “does the qa-pair make sense?” and “is the answer correct?”. the annotator could answer all questions with only one of the following possibilities: “yes”, “admissible” and “no”. as shown in fig. 6, the results show that got at least admissible were: 70% for domain adherence, 82% for sense, and 80% for the correctness of the answer, and even considering just perfect qa-pairs, all categories got more than 50% of occurrence. fig. 5. comparison between the distribution of passage scores of a random sample of the paq with the ones of qa-pairs filtered for the brazilian environmental domain. the passage score is a logprob score calculated in the paq dataset that measures how likely a given qa-pair is in practice; the closer to zero, the better. note that the filtering process did not cause a significant loss of quality. full size image finally, to fine-tune our qa system entirely on the same language, we applied the google translate api (application programming interface) to translate these pairs into portuguese. fig. 6. manual evaluation of a sample of 50 qa-pairs from our domain-specific qa dataset, which got at least 70% of admissible and 50% perfect results in all three categories full size image 4 experiments to measure the impact of the retrieval module, each base of supporting documents, and the fine-tuning of the system, we performed three groups of experiments: 1. reader-only and retriever+reader, both without fine-tuning; 2. reader-only, with fine-tune; 3. retriever+reader, with fine-tune. the filtered qa-pairs dataset was randomly split into 3 groups: 70% for training, 15% for validation, and 15% for the test. the models in experiments 2 and 3, which depended on a training phase, were trained for 30 epochs, with a batch size equal to 16, weight decay equal to 0.01 and a learning rate of 2e−5; the same training and validation sets were used for the fine-tuning. for all models, we report the f1-score, the exact match (em), and the rouge-l (r-l) metrics, also obtained in the same test set. in all cases, we used the ptt5-base-portuguese-vocab t5 pre-trained model in portuguese, since it was the recommended one by the original work [4] in comparison to the other versions of ptt5, including its large version. in cases where we used the bm25-based retrieval module, we preprocessed all the supporting documents by removing special characters, eliminating line breaks, and splitting them into chunks of 100-word passages. 4.1 experiment 1: systems without fine-tune experiment 1 aimed to provide a baseline and demonstrate the impact of the lack of a fine-tuning step with the filtered qa base for the problem domain. therefore, we place the two models, reader-only and retriever+reader, directly to answer the test set questions, without any previous fine-tune, as indicated in figs. 1(a) and 2(a), respectively. 4.2 experiment 2: reader-only, with fine-tune as in experiment 1, in this case, we abdicated the retrieval module but performed the fine-tuning of ptt5 in our domain-specific qa dataset. all other parameters are identical to those of the reader in the previous subsection. figure 1(b) illustrates the procedure. 4.3 experiment 3: retriever+reader with fine-tune in this experiment, the model was composed of the two modules, retriever and reader, as in the second case discussed in experiment 1, with the difference that now ptt5 is submitted to a fine-tuning on our domain-specific qa dataset, as shown in fig. 2(b). in this experiment, we explore the impact on the quality of model answers to: 1. a larger number of passages retrieved by the retriever; 2. each kb (wiki, news and wiki+news). thus, we trained the model with \(k = 5\) retrieved passages and 512 entry tokens for ptt5 three times, once with each distinct kb (wiki, news, and wiki+news). then, we repeated the same 3 pieces of training but considering \(k = 10\) and 1024 entry tokens for ptt5. 5 results and discussion the results of all the experiments described in the previous section are consolidated in table 1. as the metrics are highly correlated, we will focus on the f1-score from now on. information on training and inference times, as well as the machine settings used, can be found in appendix c. table 1. main results of the tests conducted analysed by the metrics f1-score, exact match and rouge-l. the best model was the one with a reader and a retriever backed only on the wiki database, with 10 passages retrieved.full size table 5.1 importance of fine-tune on the specific domain as expected, the models from experiment 1, without fine-tune, had the worst results of all. this demonstrates the importance of this training phase and the construction of the domain-specific dataset we performed: the reader-only from experiment 2, fine-tuned in our qa dataset, performed 11 times better than the reader+retriever system from experiment 1, which had access to a kb composed of all the news and wikipedia articles collected, and achieved scores comparable to models that were fine-tuned and had access to one kb. then, in short, the presence of the retrieval module does not compensate for the absence of the model’s fine-tune. 5.2 impact of different kbs on scores experiment 3 allowed us to compare the effect of each kb on the quality of the systems’ responses with the retriever. for both \(k = 5\) and for \(k = 10\), we observed that the systems supported only by the wiki kb performed better than those that have access to the expanded kb with newspaper news (wiki+news), which was contrary to what we expected at the beginning. also, in both cases, the least competitive results occurred when the kb is composed only by the news kb, still, however, surpassing the reader-only model. a possible explanation for this phenomenon may lie in the paq construction process, which is automatically generated on wikipedia passages in english. this can generate a bias in favor of the kb formed by a specific category of wikipedia, even though it is, here, in portuguese. 5.3 influence of distractors on the reader finally, it is remarkable that doubling the number of retrieved passages to \(k = 10\) improves the performance of the models with wiki, even when integrated with the news, but simultaneously the worst model among those that have access to a kb is the one composed solely of news. it is slightly inferior even to the same model configured with \(k = 5\). arguably, this is because, when we concatenate the initial question q with the 10 passages instead of 5, we end up diluting the weight of q in the reformulated question \(q'\) (a much longer text), making the original question more diffuse among heterogeneous news articles, which does not compensate for the information gain brought by the retrieved passages \(\{p_1, p_2, ..., p_k\}\). still, the news kb can be helpful due to the contemporaneity it aggregates to the kb, which can be particularly relevant for reasons of explainability. table 2. comparison between a case in which the presence of the retriever is useful to prevent the reader from making mistakes (first case), and another in which the retrieved passages otherwise mislead it due to the presence of distractors.full size table the same probably does not occur as often with the wiki kb perhaps because of the generational bias of paq. nevertheless, it does not make it immune to distractors. table 2 illustrates two emblematic cases, comparing a qa system with only the reader and another with the retriever+reader. in the first question, whose correct answer is “2001”, the reader-only was wrong, but the system composed also of a retriever was right. the retrieved passage that contains the answer is: “...today fernando de noronha’s economy depends on tourism (...) in 2001 the archipelago was declared a world heritage site, including the atol das rocas, as sítio das ilhas...”. thus, we see that the information extracted by the retriever was essential for the reader not to incur the same error again. however, in the second question, the opposite occurs: the reader gets it right, but the retrieve+reader system gets it wrong. when observing the recovered passages, we noticed a potential distractor that could have induced the system to error: ...on march 8 of that year, marabá was practically submerged. occupying an area of 803 250 square kilometers, it is the largest hydrographic basin entirely in brazil, even though it belongs to the amazon basin (...) the tocantins, the main river in this basin, rises in the north of goiás and flows into the marajoara gulf...”. despite these occasional problems, all tests we performed indicated that a system consisting of a retriever and a reader always surpasses one with only a reader. 6 conclusion we presented the first qa system focused on environmental issues in brazil—more importantly, a qa system based on the portuguese language, a language that has received remarkably low attention when it comes to automatic question answering. we combined ptt5 as the reader, the state-of-the-art among pre-trained language models, and bm25 as the retriever, the state-of-the-art sparse retrieval technique. also, we collected documents and qa-pairs by filtering articles related to “environment of brazil” in the wikipedia dump in portuguese; scraped environmental news from january 2018 to june of 2021 in the three most important newspapers in the country; filtered and translated a recently released massive open-domain qa dataset to obtain a substantial domain-specific set of qa pairs. despite potential generation biases found in this last step, our trained qa systems demonstrated competitive scores. we hope that this work can stimulate similar initiatives on a topic that is so relevant to brazilian environmental efforts. to deepagé increase social awareness and understanding about the environment and the climate of brazil, the system must be tested with human subjects. integration with other modules such as a social chatbot can certainly make the system more appealing for users [6]. another further improvement would be the construction of a system that gives more complete and elaborate answers. as our training was ran using the paq dataset, with a majority of factual and short responses, the system is not prepared to give long and detailed answers. also, generative models such as the ptt5 can hallucinate when giving answers, especially when there is no retriever. hence, another useful future extension to deepagé would be a filtering module to avoid absurd answers. notes 1.articles about the “environment of brazil” taken from the following category of wikipedia: https://pt.wikipedia.org/wiki/categoria:meio_ambiente_do_brasil. 2.published between january 2018 to june 2021, and scraped from the three biggest newspapers in circulation in brazil on different topics related to environmental issues in the country (such as deforestation, the status of indigenous peoples). 3.all relevant code, checkpoints, and data (except for the scrapped news, in respect of copyright laws) are available or referenced in the github repository: https://github.com/c4ai/deepage. 4.the pre-trained model is available at https://huggingface.co/unicamp-dl/ptt5-base-portuguese-vocab. 5.available at https://dumps.wikimedia.org/. 6.the query we developed is described in the appendix a. 7.for example, if the word “biome” were a substring of a certain question or its answer, the word “brazil” or other names of states in the country should also be so, so as not to include qa-pairs about biomes from other countries in our selection. references paschoal, a.f.a., et al.: a bilingual portuguese-english dataset for question-answering about the ocean. in: 30th acm international conference on information and knowledge management (cikm 2021) (2021). https://doi.org/10.1145/3459637.3482012 bahdanau, d., cho, k.h., bengio, y.: neural machine translation by jointly learning to align and translate. in: 3rd international conference on learning representations, iclr 2015 conference track proceedings, pp. 1–15 (2015) google scholar  bhatia, s., lau, j.h., baldwin, t.: automatic classification of neutralization techniques in the narrative of climate change scepticism, pp. 2167–2175 (2021) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data, pp. 1–12 (2020). http://arxiv.org/abs/2008.09144 diggelmann, t., boyd-graber, j., bulian, j., ciaramita, m., leippold, m.: climate-fever: a dataset for verification of real-world climate claims, pp. 1–16 (2020). http://arxiv.org/abs/2012.00614 gao, j., galley, m., li, l.: neural approaches to conversational ai. in: acl 2018–56th annual meeting of the association for computational linguistics, proceedings of the conference tutorial abstracts, pp. 2–7 (2018). https://doi.org/10.18653/v1/p18-5002 guu, k., lee, k., tung, z., pasupat, p., chang, m.w.: realm: retrieval-augmented language model pre-training. in: 37th international conference on machine learning, icml 2020 part f16814, pp. 3887–3896 (2020) google scholar  kim, y., sohn, j., bang, s., kim, h.: deep learning-based question answering system for proactive disaster management. in: proceedings of the 37th international symposium on automation and robotics in construction (isarc) (isarc), 2020 (2020). https://doi.org/10.22260/isarc2020/0181 kwiatkowski, t., et al.: natural questions: a benchmark for question answering research. trans. assoc. comput. linguistics 7, 453–466 (2019). https://doi.org/10.1162/tacl_a_00276 lewis, p., et al.: retrieval-augmented generation for knowledge-intensive nlp tasks. arxiv (neurips) (2020) google scholar  lewis, p., wu, y., liu, l., minervini, p., küttler, h., piktus, a., stenetorp, p., riedel, s.: paq: 65 million probably-asked questions and what you can do with them (2021). http://arxiv.org/abs/2102.07033 luccioni, a., baylor, e., duchene, n.: analyzing sustainability reports using natural language processing (2020). http://arxiv.org/abs/2011.08073 nguyen, t., rosenberg, m., song, x., gao, j., tiwary, s., majumder, r., deng, l.: ms marco: a human generated machine reading comprehension dataset. in: ceur workshop proceedings 1773 (nips 2016), pp. 1–11 (2016) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. arxiv 21, pp. 1–67 (2019) google scholar  robertson, s., zaragoza, h.: the probabilistic relevance framework: bm25 and beyond, vol. 3 (2009). https://doi.org/10.1561/1500000019 severo, e.a., de guimarães, j.c.f., dellarmelin, m.l.: impact of the covid-19 pandemic on environmental awareness, sustainable consumption and social responsibility: evidence from generations in brazil and portugal. j. clean. prod. 286 (2021). https://doi.org/10.1016/j.jclepro.2020.124947 varini, f.s., boyd-graber, j., ciaramita, m., leippold, m.: climatext: a dataset for climate change topic detection, pp. 1–13 (2020). http://arxiv.org/abs/2012.00483 vaswani, a., et al.: attention is all you need. advances in neural information processing systems 2017-decem(nips), pp. 5999–6009 (2017) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: lrec 2018–11th international conference on language resources and evaluation, pp. 4339–4344, may 2019 google scholar  west, t.a.p., börner, j., fearnside, p.m.: climatic benefits from the 2006–2017 avoided deforestation in amazonian brazil. front. forests global change 2(september), 1–11 (2019). https://doi.org/10.3389/ffgc.2019.00052 article  google scholar  download references acknowledgements this work was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior (capes, finance code 001) and by the itaú unibanco s.a., through the programa de bolsas itaú (pbi) of the centro de ciência de dados (c\(^2\)d) of escola politécnica of universidade de são paulo (usp). we also gratefully acknowledge support from conselho nacional de desenvolvimento cientiífico e tecnológico (cnpq) (grants 312180/2018-7 and 310085/2020-9) and the center for artificial intelligence (c4ai-usp), with support by the fundação de amparo à pesquisa do estado de são paulo (fapesp, grant 2019/ 07665-4) and by the ibm corporation. author information authors and affiliations escola politécnica, universidade de são paulo, sao paulo, brazil flávio nakasato cação, marcos menon josé, andré seidel oliveira, stefano spindola, anna helena reali costa & fabio gagliardi cozman authors flávio nakasato caçãoview author publications search author on:pubmed google scholar marcos menon joséview author publications search author on:pubmed google scholar andré seidel oliveiraview author publications search author on:pubmed google scholar stefano spindolaview author publications search author on:pubmed google scholar anna helena reali costaview author publications search author on:pubmed google scholar fabio gagliardi cozmanview author publications search author on:pubmed google scholar corresponding author correspondence to flávio nakasato cação . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado appendices 7 appendix a to filter qa-pairs in paq, we created four sets of keywords: m (from “must have”: “brazil*” and other states names), g (from “good to have”; e.g. “deforestation”), u (from “unique expressions”; e.g. “ibama” or “amazon rainforest”) and e (from “to exclude”; e.g. “soccer”). for a pair to be selected, it must contain, in its question or answer, at least one expression belonging to m or u. if it was from u, it would already be selected; if it was from m, it could not contain any e expressions. finally, if the qa-pair contains any keyword from g, it should also contain at least one from m, but none from e either. 8 appendix b we performed queries associated with the following keywords (here, translated into english): amazon rainforest, cerrado, climate change, deforestation, environmental conduct adjustment agreement, extractive (from “extractivism”), forest fires, funding for conservation, green economy, land grabbing, mining and protected areas. it should be noted that the decision to exclude some words is only understandable in portuguese, since there is no reasonable parallel in english, as with the word “fist” for the keyword “cerrado” in portuguese, the expression “punho cerrado” (“clenched fist”) is common, but it is clearly not directly related to the cerrado biome. also for more significant results, we eliminated news related to “agrobusiness” in the green economy keyword and “militias” in the land grabbing one. 9 appendix c to perform the experiments, we leverage a machine with an amd ryzen 9 3950x processor with 32 cpus, 64 gb of ram, 2 nvidia rtx 3090 gpus of 24gb each, on an ubuntu 20 lts. under these conditions, the training of each model lasted between 5h and 8h, with the best model being trained in about 7h40. the inference time – that is, the time it took to a model to generate an answer to a single question from the test set – was around 0.006s. rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper cação, f.n., josé, m.m., oliveira, a.s., spindola, s., costa, a.h.r., cozman, f.g. (2021). deepagé: answering questions in portuguese about the brazilian environment. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_29 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords question answering environment in brazil natural language processing in portuguese publish with us policies and ethics profiles anna helena reali costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature universal dependencies-based pos tagging refinement through linguistic resources | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper universal dependencies-based pos tagging refinement through linguistic resources conference paper first online: 28 november 2021 pp 601–615 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) universal dependencies-based pos tagging refinement through linguistic resources download book pdf download book epub lucelene lopes  orcid: orcid.org/0000-0003-0314-140x10, magali s. duran  orcid: orcid.org/0000-0002-3843-460010 & thiago a. s. pardo  orcid: orcid.org/0000-0003-2111-131910  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1220 accesses 2 citations abstract this paper presents a technique that employs linguistic resources to refine pos tagging using the universal dependencies (ud) model. the technique is based on the development and use of lists of non-ambiguous single tokens and non-ambiguous co-occuring tokens in portuguese (regardless of whether they constitute multiword expressions or not). these lists are meant to automatically correct the tags for such tokens after tagging. the technique is applied over the output of two well-known state of the art systems udpipe and udify and the results for a real data set have shown a significant improvement of annotation accuracy. overall, we improve tagging accuracy by up to 1.4%, and, in terms of the number of fully correct tagged sentences, our technique produces results that are 13.9% more accurate than the corresponding original system. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others building powerful dependency parsers for resource-poor languages chapter © 2016 disambiguation of universal dependencies part-of-speech tags of closed class words in portuguese chapter © 2023 tagsets and datasets: some experiments based on portuguese language chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics language processing lisp natural language processing (nlp) research methods in language and linguistics 1 introduction the importance of the part of speech (pos) tagging task is beyond discussion, as often the other natural language processing (nlp) tasks rely on the outcome of pos tagging. furthermore, the current nlp applications are designed to process large amounts of texts, making the use of taggers a necessity. for well resourced languages, pos tagging processes are usually dealt with consistent high accuracy [12]. the reason for such high accuracy relies mostly on the existence of large annotated corpora and powerful techniques, usually based on neural network inspired implementations [5]. for poorly resourced languages, or even specific dialects, pos tagging may become challenging because of the lack of an appropriate training model [8]. this can be coped with reduced or slightly inadequate training models and the use of very powerful taggers/parsers based on performant solutions as the use of word embeddings and bidirectional rnns [4]. one example of this is implemented in udpipe, a trainable pipeline that performs sentence segmentation, tokenization, lemmatization, pos tagging and dependency parsing [11]. other performant similar solutions are the ones based on the use of cutting edge technology as bert [9], which is implemented in udify, a multilingual multi-task model capable of accurately predicting universal part-of-speech, morphological features, lemmas, and dependency trees simultaneously for several universal dependencies (ud) treebanks [3]. such approaches are very effective, basically because they are flexible enough to detect complex characteristics necessary to classify words and grammatical constructions that are present in the training set. unfortunately, these high performant pos taggers still make basic mistakes and misclassify simple words (as those belonging to closed classes). that leads to errors in pos annotation that could be easily tackled by solutions that rely on linguistic resources that are not explicitly covered by current machine learning models [13]. one thing in common to the udpipe and udify taggers/parsers is the use of ud formalism for encoding the output representation [6]. ud is a de-facto standard for cross-linguistically comparable morphological and syntactic annotation that keeps evolving and provides a steadily growing and heavily multilingual collection of corpora [1]. ud has reshaped the initiatives in the area of tagging and parsing in the world, causing a significant movement from constituency to dependency analysis in nlp. for portuguese texts, the pos tagging task using tools as udpipe and udify present some challenges because of the low availability of appropriate training sets with the right language style and a reasonable size. one of the few options is the bosque-ud corpus, a portuguese treebank based on the constraint grammar converted version of bosque, which is part of the floresta sintá(c)tica treebank [7] that contains both european and brazilian variants [14]. the bosque-ud corpus has 9,364 sentences, totalizing 210,957 tokens. in order to contribute to ud-based pos tagging of brazilian portuguese texts, we propose a linguistic-based technique to improve tagging accuracy, which we specifically test over the output of udpipe and udify systems, which use bosque-ud as training set. the proposed technique relies on building and employing linguistic resources to automatically correct the pos tags produced by the cited systems. our research claim is that explicit linguistic knowledge does help improving the result of modern machine learning-based models. to support our claim, we describe the process of building our linguistic resources for brazilian portuguese, and we conduct experiments illustrating our attempt of quality improvement. specifically, we propose the automatic correction of tags wrongfully assigned to non-ambiguous single tokens and non-ambiguous co-occuring tokens (regardless of whether they constitute mwe multiword expressions or not) from closed pos classes in portuguese. we consider in our linguistic resources three types of phenomena: single functional words (the pronoun você “you” in english, for example); mwes acting as functional words (the prepositional phrase por meio de “through”, for example); and co-occurring functional words that, although in isolation are ambiguous, together are not ambiguous (for example, a may be a determiner (det), a preposition (adp), or a pronoun (pron), but in the co-occurrence à de may be unambiguously tagged as adp pron adp). it is important to call the reader attention that, being functional words, they are naturally non-ambiguous, independently of the domain. we show that the achieved gains by our proposed technique is significant and that this is a strategy worth to follow. overall, we improve tagging accuracy in up to 1.4%, and, in terms of the number of fully correct tagged sentences, our technique produces results that increases the accuracy by 13.9% from the corresponding original system. considering that pos tagging annotation is the basis for other text analysis levels, such improvement may be really impactful for producing better nlp tools and applications. this paper is organized as follows: the next section describes the process and the result of producing our linguistic resources composed by lists of non-ambiguous single tokens and non-ambiguous co-occuring tokens belonging to closed pos tag classes, highly frequent in portuguese written text; the third section presents the evaluation of our proposed correction technique on a data set of 2,000 sentences (43,483 tokens) from the folha-kaggle corpus [10] annotated by udpipe [11] and udify [3] systems; finally, the last section presents our concluding remarks and future works. 2 linguistic analysis of non-ambiguous tokens in portuguese in this section we analyze the possibility to establish a set of tokens that can only be annotated by one single possible pos tag. usually, very performant pos taggers based on a training set knows no such boundaries as they try to capture very subtle interdependencies among the tokens and their representation. linguistic knowledge of the target language, however, can provide such boundaries by establishing the specific tokens or expressions that are unambiguously capable to receive only one single pos tag. the knowledge of these non-ambiguous tokens can be a powerful and fully automatic tool to improve the quality for a very performant pos tagger based on a training set. in this work, we adopt two linguistic groups to improve the pos tagging: the single tokens belonging to a closed pos tag class (in ud): adp (adposition), adv (adverbsfootnote 1), cconj (coordinating conjunction), det (determiner), num (numeral just cardinals), pron (pronoun), sconj (subordinating conjunction), a subset of the class aux (verbs ser and estar) and adj (ordinal numbers); common co-occuring tokens that have always the same pos tag classes, which include: mwes acting as functional words; co-occurring functional words that in isolation are ambiguous, but together become non-ambiguous. 2.1 non-ambiguous single tokens for closed pos classes to establish this group, we started listing all tokens tagged to each of the targeted pos classes (adp, adv, cconj, det, num, pron and sconj) in the bosque-ud corpus [14], and the equivalent tags in the macmorpho corpus [2]. the lists for each pos class were analyzed and inappropriate tokens were removed throughout a manual linguistic analysis of each token individually. similarly, some tokens were added to the lists, despite not being found in the corpora above mentioned. this was the case of ordinal and cardinal numerals, additional forms of auxiliary verbs (ser and estar), and functional rare words, as the adverb adentro (“inwards”). table 1 presents the overall number of listed tokens and the final number of tokens retained for each class. in this table, only the tokens for ordinal numerals (annotated as adj in ud) are absent, since they were not searched in the bosque-ud and macmorpho as they are not distinguishable from other open class adjectives (all these tokens were added by the linguistic analysis only). table 1. tokens for closed pos classes.full size table the main reason for a token being considered inappropriate was misannotation due to different annotation principles. an example of such difference of annotation principles can be found in macmorpho corpus, where several mwes, acting as functional words, were tagged as a whole and not by the words that compose them. this is the case of the por fim expression that is tagged in macmorpho by assigning a tag adv to por and another tag adv to fim, since this expression plays the role of an adverb (this expression can be translated into english as “finally”). however, according to the ud guidelines, each token has to be tagged individually, not considering mwes. following this principle, por will be clearly tagged as an adposition (prep in macmorpho tagset, adp in ud tagset). therefore, the tokens por and fim were removed from the list of adv despite appearing as such in the macmorpho corpus. this is exemplified in the sentence of fig. 1. fig. 1. example of the difference between macmorpho and ud annotation principles. full size image the next step was to cross-analyze the lists among themselves. for example, the token se can be found in the pron and sconj lists, and, therefore, it was considered an ambiguous token regarding the pos tag. in english, se could be translated to “yourself” for pron or to “if” for sconj. figure 2 exemplifies two sentences where the se token is employed either way. fig. 2. example of ambiguous token within closed pos classes (pron and sconj). full size image after crossing the tokens for the closed pos tag classes, each token was also individually analyzed in relation to its other possible open classes. for example, the token entre can be an adp, but also a conjugation of the verb entrar (therefore, a verb). in english entre could be translated to “between” for adp or to “enter” for verb. figure 3 presents two sentences where entre token is employed both ways. after all those analyses, it was possible to identify all single tokens for the target closed pos classes, as well as which ones are ambiguous or not. tables 2 and 3 indicate the non-ambiguous tokens in bold face (269 tokens) and the ambiguous ones in italic. table 2. tokens of closed pos classes.full size table fig. 3. example of ambiguous token within a closed class (adp) and as an open class pos tag (verb). full size image table 3. tokens of closed pos classes (cont.).full size table 2.2 non-ambiguous co-occurring tokens the process to obtain the group of co-occurring tokens to be considered in our technique was somewhat similar to the one for single tokens of closed pos classes. we started by searching in the available corpora the occurrences of a list of functional expressions that are commonly employed in portuguese such as mwes acting as functional words, as the prepositional phrase de acordo com (“according to”). this list was suggested by a linguist dedicated to annotating a corpus following ud guidelines and it includes highly frequent functional expressions such as desde que (“since”), annotated as adp sconj, and mais ou menos (“more or less”), annotated as adv cconj adv. the linguistic analysis filtered out those expressions that appeared with different sets of pos tags in the corpora. in other words, expressions that need context to be disambiguated were disregarded. the goal was to determine once more if that was an ambiguity due to context or just a misannotation due to different annotation principles. table 4 shows some examples of non-ambiguous expressions and co-occurring tokens with their respective tags and number of occurrences in the bosque-ud and macmorpho corpora. table 4. examples of non-ambiguous co-occuring tokens and respective pos tags in different corpora.full size table observing the expressions in table 4, we have simple cases as o bastante (“enough”), where all annotations converged to the same tags (det adj), as macmorpho uses art for article, which is always mapped as det in ud. however, other examples required the analysis of different annotation principles, for example, de longe (“from afar”), which is annotated in macmorpho as an expression that plays a role of an adverb, but, following the ud principles to tag each token individually, de (“from” in this context) is undoubtedly an adposition (adp), as agreed by bosque-ud annotation. furthermore, our decision to adopt the tag adv to the word longe (“afar”) relies on the fact that it is a non-ambiguous place adverb in portuguese. similar decisions based on linguistic concepts were adopted to each expression individually. some expressions, on the contrary, were considered ambiguous because they correctly appear with different pos tags. figure 4 illustrates two sentences with different tags associated to the co-occurring tokens a mais that can be tagged as adp adv or det adv according to the context. in english, a mais could be translated to “more” (or “extra”) for adp adv or to “the more” (or “the most”) for det adv. the co-occuring tokens considered ambiguous were left out of our technique. fig. 4. example of sentences with ambiguous co-occuring tokens. full size image this process of analysis of ambiguity was carried out to each candidate individually and, even thought we are aware that the resulting list is non-exhaustive, the 110 non-ambiguous expressions and co-occurring tokens with their respective pos tags are representative of usual and highly frequent lexical phenomena in portuguese given the observed corpora. the full list of expressions and co-occuring tokens used in our technique is presented in table 5, where each expression is followed by its non-ambiguous associated pos tags. table 5. chosen non-ambiguous co-occuring tokens and their tags.full size table 3 evaluation the test data set. once the set of non-ambiguous single and co-occuring tokens for closed pos tag classes are defined, our proposed technique is experimented over a data set composed by 2,000 sentences randomly picked from folha-kaggle corpus [10]. the limit of 2,000 sentences for the data set is due to the fact that we proceeded a laborious manual annotation of these 2,000 sentences to serve as a gold standard for our experiments. the folha-kaggle corpus is composed by news published at the electronic version of the brazilian newspaper folha de são paulo from january 2015 and september 2017. this corpus holds 167,053 news articles with an approximate average of 22 sentences per news and 23 tokens per sentence, thus more than 3.6 million sentences and about 84 million tokens. the major concern choosing the sentences to form our testing data set is to have sentences with at least one of the chosen non-ambiguous single or co-occurring tokens. additionally, we restricted our choices to sentences from 7 to 40 tokens long. the full random process consisted to pass sequentially for the sentences of the folha-kaggle corpus, until obtaining 2,000 sentences performing the following steps: to test if the sentence had from 7 to 40 tokens; to test if the sentence had either at least one token or one expression among the non-ambiguous ones (tables 2, 3 or 5); to randomize a chance of 5% to keep the sentence in the test data set (so that the several sentences come from a wide variety of texts). the resulting data set is composed by 2,000 sentences totalizing 43,483 tokens, therefore, an average of 22 tokens per sentence. the total number of non-ambiguous single tokens of closed pos classes (tables 2 and 3) within this data set is 8,544 tokens, and the total number of non-ambiguous co-occuring tokens (table 5) is 273. the automatic annotation. the test data set was processed by udpipe [11] and udify [3] systems using as training set the bosque-ud [14]. the goal is to observe how each system annotated each token of the test data set and how our proposed technique could improve the tagging accuracy. it is important to notice that udpipe and udify are considered to be among the state of the art systems in the area for ud annotation. the manually produced gold standard. a team of linguists performed the annotation of the test data set to provide a gold standard to evaluate the automatic annotation accuracy of each system. the annotation was conducted independently by 10 highly trained linguists, and the overall result was adjudicated by a chief linguist to provide a correct and homogeneous annotation. 3.1 evaluation of the non-ambiguous single tokens from closed pos classes corrections our goal here is to improve the tagging accuracy by applying an automatic correction of all non-ambiguous single tokens from closed pos classes, by identifying and replacing the pos annotation of all these tokens (hence, this process is denoted as acs for automatic correction of single tokens). the test data set has 8,544 tokens that are non-ambiguous tokens from closed pos classes, and they are fairly distributed among all 2,000 sentences, since all sentences have at least one of those tokens. comparing the systems’ output with the gold standard, we observe a good overall accuracy for these tokens as stated in table 6. udpipe annotation was wrong in 5.6% of these tokens, while udify was wrong in 2.8%. however, considering that those tokens belong to sentences, and a wrongfully annotated token may jeopardize the whole sentence annotation because of a possible ripple effect, it is concerning that udpipe and udify outputs contain errors in 22.2% and 11.0% of the sentences, respectively. by running our automatic correction (acs), we successfully corrected all wrong annotations mentioned in table 6 for both systems’ outputs, as all tokens wrongfully annotated (482 by udpipe, 239 by udify) were corrected by us in accordance to the gold standard. this represented an improvement of, respectively, 22.2% and 11.0% in the number of correct sentences annotated by udpipe and udify concerning the non-ambiguous single tokens for closed pos classes. table 6. automatic annotation of the test data set for the non-ambiguous single tokens from closed pos classes compared to the gold standard (acs).full size table 3.2 evaluation of the non-ambiguous co-occurring tokens corrections similarly to the process for the single tokens of closed pos classes, we conducted an automatic correction of the chosen non-ambiguous functional expressions and co-occuring functional words (table 5) (hence, this process is denoted as acc for automatic correction of co-occuring tokens). the 2,000 sentences test data set holds 273 occurrences of the chosen non-ambiguous co-occurring tokens that are distributed over 263 sentences. both udpipe and udify have several expressions incorrectly annotated as shown in table 7. table 7. automatic annotation of the test data set for the non-ambiguous co-occurring tokens compared to the gold standard (acc).full size table the observation of the results described in table 7 shows in scale the same need for correction as the one for single tokens. an impressive number of wrongfully annotated co-occurring tokens (39.2% for udpipe and 32.6% for udify) reflects in annotation errors of 39.9% and 33.8% of sentences, respectively. after applying the automatic correction (acc) to both systems’ outputs, we were pleased to verify that all 107 and 89 expressions, respectively, were correctly reannotated by our technique, eliminating all system errors for the chosen non-ambiguous co-occurring tokens. it is important to point out that the acc carried out by our technique had a low overlap with acs. for udpipe, only 7 out of 107 co-occurring tokens corrections had an overlap with the corrections carried out for the single tokens: the token de in the expression a ponto de (twice); and the token desde in the expression desde que (five times). for udify, only 2 out of 89 co-occurring tokens corrections had overlap and they both were for the token desde in the expression desde que (twice). 3.3 evaluation of all non-ambiguous corrections as seen in sects. 3.1 and 3.2, both automatic corrections brought improvements considering the targeted tokens and expressions. also, our experiments indicated an overall improvement in the accuracy of the systems’ annotation when compared to the gold standard. table 8 presents the obtained accuracy starting from the original system annotation, followed by the application of our automatic correction of non-ambiguous single tokens (acs) and automatic correction of non-ambiguous co-occurring tokens (acc). however, in this table, we indicate not only the numbers concerned by the tokens belonging to the target set (single and co-occurring tokens), but all tokens of the 2,000 sentences test data set, providing an overall picture of the situation. table 8. automatic annotation of the test data set: overall analysis considering the original system annotation and our proposed automatic corrections (acs and acc) compared to the gold standard.full size table observing the results in table 8, it is clear that both automatic corrections presented consistent improvements by correcting wrongfully annotated tokens (1.4% and 0.8% for udpipe and udify, respectively). however, the gains in terms of the number of correctly annotated sentences is even more noticeable, as the initial system accuracy for fully correct tagged sentences was 44.3% and 40.0% for udpipe and udify, respectively, and the combined application of acs and acc brought an increase around 13.9% of fully correct sentences for udpipe (from 44.3% to 58.2% of sentences) and 5.4% for udify (from 40% to 45.4% of sentences). such values are very relevant, as tagging results directly influence other automatic analyses that parsers and other nlp systems perform. therefore, producing an error reduction in this scale is significant. another interesting observation from table 8 is that the acs and acc corrections bring rather complementary gains. for instance, observing the reduction of wrongfully annotated tokens for udpipe, the application of acc over the udpipe output corrected the tags of 118 tokens (1935 minus 1817), while the application of acs corrected the tags of 482 tokens (1935 minus 1453), and the combined application of both acs and acc corrected 583 tokens (1935 minus 1352). this shows that the intersection of corrected tags between acc and acs was just over 17 tokens (which is the difference between 118+482 and 583). a similar relation was found for udify output, and also for the gains in terms of sentences. these numbers reinforce our findings of the small overlap between acs and acc gains mentioned in sect. 3.2. 4 concluding remarks we proposed a technique for universal dependencies-based pos tagging refinement through linguistic resources. our goal was to show that such refinement could be accomplished by improving the annotation accuracy of two well-known systems (udpipe and udify) through tag corrections based on non-ambiguous single and co-occurring tokens. the overall impact of our proposed automatic corrections lead to a tagging accuracy improvement of up to 1.4%. more than this, in relation to the number of fully correct annotated sentences, the improvement was up to 13.9%. we performed a verification over the entire folha-kaggle corpus with 3.6 million sentences and 84 million tokens, and we found nearly 28 million tokens that would be detected by our proposed approach. given the results of our experiments, which indicate that about 2.5 million tokens would be wrongfully tagged by udpipe and would be corrected by our approach, this represents the correction of about 20% of the sentences of the whole folha-kaggle corpus. another important contribution of our work is the construction and availability of the linguistic resources with the 269 non-ambiguous single token words from closed pos classes and 110 non-ambiguous functional expressions and co-occuring functional words for portuguese. these resources may help several linguistic efforts in portuguese, which is still an under resourced language. for the interested reader, all the resources and the gold standard data are available at the poetisa project webpagefootnote 2. finally, tackling pos tagging under the ud formalism is also an important contribution for the computational processing of the brazilian portuguese language, as ud model has become the standard in tagging and parsing in the world. nowadays, the ud project has been adopted by over 100 languages, including portuguese. this paper helps fostering ud research for this language. future work includes to possibly refine the current lists, in particular, to enlarge the lists of non-ambiguous tokens and functional expressions, as well as to look for possible disambiguation rules for some tokens from closed pos classes and to include nominal and verbal mwes. notes 1.the adv class is not exactly a closed one as, similarly to english with the ending -ly, in portuguese it is possible to turn adjectives into adverbs by adding -mente at the end (for example, the adjective final can be turned into the adverb finalmente “finally”), but, for the purpose of our technique, we ignore such adverbs. 2.https://sites.google.com/icmc.usp.br/poetisa. references droganova, k., zeman, d.: towards deep universal dependencies. in: proceedings of the 5th international conference on dependency linguistics (depling, syntaxfest), pp. 144–152 (2019) google scholar  fonseca, e.r., g rosa, j.l., aluísio, s.m.: evaluating word embeddings and a revised corpus for part-of-speech tagging in portuguese. j. braz. comput. soc. 21(1), 1–14 (2015) google scholar  kondratyuk, d., straka, m.: 75 languages, 1 model: parsing universal dependencies universally. in: proceedings of the conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 2779–2795 (2019) google scholar  ling, w., et al.: finding function in form: compositional character models for open vocabulary word representation. in: proceedings of the conference on empirical methods in natural language processing (emnlp), pp. 1520–1530 (2015) google scholar  nivre, j., fang, c.t.: universal dependency evaluation. in: proceedings of the nodalida 2017 workshop on universal dependencies (udw), pp. 86–95 (2017) google scholar  nivre, j., et al.: universal dependencies v1: a multilingual treebank collection. in: proceedings of the 10th international conference on language resources and evaluation (lrec), pp. 1659–1666 (2016) google scholar  rademaker, a., chalub, f., real, l., freitas, c., bick, e., de paiva, v.: universal dependencies for portuguese. in: proceedings of the 4th international conference on dependency linguistics (depling), pp. 197–206 (2017) google scholar  rehbein, i., hirschmann, h.: pos tagset refinement for linguistic analysis and the impact on statistical parsing. in: henrich, v., hinrichs, e., de kok, d., osenova, p., przepiorkowski, a. (eds.) proceedings of the 13th international workshop on treebanks and linguistic theories (tlt13), pp. 172–183. university of tübingen (2018) google scholar  rogers, a., kovaleva, o., rumshisky, a.: a primer in bertology: what we know about how bert works. trans. assoc. computat. linguist. 8, 842–866 (2020) article  google scholar  santana, m.: kaggle news of the brazilian newspaper. https://www.kaggle.com/marlesson/news-of-the-site-folhauol. accessed 14 june 2021 straka, m.: udpipe 2.0 prototype at conll 2018 ud shared task. in: proceedings of the conll 2018 shared task: multilingual parsing from raw text to universal dependencies, pp. 197–207 (2018) google scholar  straka, m., straková, j.: tokenizing, pos tagging, lemmatizing and parsing ud 2.0 with udpipe. in: proceedings of the conll 2017 shared task: multilingual parsing from raw text to universal dependencies, pp. 88–99 (2017) google scholar  sulubacak, u.: implementing universal dependency, morphology, and multiword expression annotation standards for turkish language processing. turk. j. electr. eng. comput. sci. 26(3), 1662–1672 (2018) google scholar  universal dependencies: ud portuguese bosque ud version 2. https://universaldependencies.org/treebanks/pt_bosque/index.html. accessed 14 june 2021 download references acknowledgments this work was carried out at the center for artificial intelligence (c4ai-usp), with support of the são paulo research foundation (fapesp grant #2019/07665-4) and the ibm corporation. author information authors and affiliations interinstitutional center for computational linguistics (nilc), institute of mathematical and computer sciences, university of são paulo, são carlos, brazil lucelene lopes, magali s. duran & thiago a. s. pardo authors lucelene lopesview author publications search author on:pubmed google scholar magali s. duranview author publications search author on:pubmed google scholar thiago a. s. pardoview author publications search author on:pubmed google scholar corresponding author correspondence to lucelene lopes . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper lopes, l., duran, m.s., pardo, t.a.s. (2021). universal dependencies-based pos tagging refinement through linguistic resources. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_41 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_41 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords pos tagging linguistic resource universal dependencies publish with us policies and ethics profiles magali s. duran view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature portuguese neural text simplification using machine translation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper portuguese neural text simplification using machine translation conference paper first online: 28 november 2021 pp 542–556 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) portuguese neural text simplification using machine translation download book pdf download book epub tiago b. de lima  orcid: orcid.org/0000-0002-0707-522x10, andré c. a. nascimento10, george valença10, pericles miranda10, rafael ferreira mello10 & … tapas si11  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1272 accesses 3 citations abstract automatic text simplification (ats) has played a significant role in the natural language processing (nlp) field. ats is a sequence-to-sequence problem aiming to create a new version of the original text removing complex and domain-specific words. it can improve communication and understanding of documents from specific domains, as well as support second language learning. this paper presents an empirical study on the use of state-of-the-art ats methods to simplify texts in portuguese. it is important to remark that the literature reports the challenge in analyzing portuguese texts due to the lack of resources compared to other languages (i.e., english). more specifically, this work evaluated different neural machine translation (nmt) techniques for ats in portuguese. the experiments showed that nmt achieved promising results in portuguese texts, obtaining 40.89 bleu score using multiple parallel corpora and raising the overall readability score by more than 5 points. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others neural machine translation for low resource assamese–english chapter © 2021 a low-resource language translation: french to mooré chapter © 2025 neural network machine translation model based on deep learning technology chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics escrt language processing machine translation natural language processing (nlp) portuguese 1 introduction automatic text simplification (ats) aims to transform complex sentences into simpler ones, which supports second language learners and improves communication with people with poor literacy, among other benefits [4]. ats can be applied to multiple domains. for instance, it could simplify legal documents transforming jargon-heavy and juridic terms into more accessible vocabulary, making the document more understandable by the general public [9]. hence, there a lot of opportunities for ats techniques in institutions from the public sector, which needs to make their public documents more accessible and reinforce their relevance and transparency. ats has been an active research topic over the years with several applications [28]. in this context, josé and finatto expressed that the demand for text simplification in brazil has increased over the years [16]. one of the main reasons for that is the growing need to make specific concepts accessible for a wider range of people. the authors also made an investigation about the language used in the documents provided by the ministry of health of brazil. they found two kinds of documents describing the same disease. the first one is directed to health professionals and, therefore, has a particular domain-specific vocabulary. in the other class of documents, focused on the general public, a “simplified version” is provided. nevertheless, there are still terms that are not usual among the less educated, such as “mucus”. besides that, the manual simplification of complex text is not scalable, thus, ats alternatives should be investigated thoroughly [16]. in the past decade, portuguese ats had a significant expansion with systems using lexical and syntactical simplification and statistical machine translation (smt) methods [3, 4, 30]. for instance, specia [30] proposed a smt model to simplify text in portuguese using only a few examples. it achieved acceptable adequacy and fluency [30]. however, these studies were developed more than ten years ago, before the advent of neural networks architectures for nlp tasks. therefore, even though deep learning algorithms are a trend of the field, they still have limited applications in portuguese ats. for instance, neural machine translation (nmt) is a recent method for text simplification which directly transforms complex sentences in simpler ones without any need of syntactical or lexical analysis [2]. nmt has gained popularity due to its well succeed results on simplification in a variety of domains [2]. moreover, different companies have used nmt methods in their services, such as google translate, microsoft translator, ibm watson language translator [13, 27, 37]. mt-based ats methods, such as nmt, usually use a parallel corpus to map hard-to-read sentences to simpler ones. besides, it is domain-independent, i.e., one can use a larger parallel corpus to train an ats model and then apply the same model on texts from a different domain with regards to its performance [8, 12, 32]. it is important to highlight that nmt methods have presented successful results and outperformed consolidated statistical methods [2]. according to our knowledge, no research has investigated the application of nmt methods to simplify brazilian portuguese texts. based on this scenario, this study investigates and assesses the use of state-of-the-art methods based on nmt to simplify documents written in brazilian portuguese automatically. this paper presents an empirical evaluation of nmt models in a parallel corpus extracted from complex and simplified translations of the bible to reach this goal. the results demonstrated that the use of nmt in portuguese text simplification is promising, with a wide range of practical applications. these findings can improve text accessibility for more people, fostering the democratisation of information. this paper is organised as follows. section 2 introduces basic concepts about nmt and the methods adopted in this work. section 3 presents works related to the ats problem. in sect. 4, materials and methods are detailed. section 5 presents the results and discussion. finally, sect. 6 states the conclusion and future works. 2 background this paper explores the use of nmt models using mono language translation to simplify texts in portuguese. herein, we briefly introduce recurrent neural networks (rnns), nmt, and the methods considered in this work, i.e., an attention-based model and a bidirectional recurrent neural network (b-rnn) with an attention layer. a rnn main advantage is to learn temporal information, even though it can be used in a non-temporal context [20]. in the case of machine translation, it usually works with a combination of encoder-decoder architectures in which both of them use rnn [20, 34]. an encoder is responsible for transforming the input in a context vector summarising its information after t recursive updates  [25]. the decoding process takes the dummy input and generates recursively the output feeding the next output with the previous output generated [25]. in this paper, we consider a specific rnn type called bidirectional recurrent neural network (b-rnn), in which the encoder is not only able to predict based on the past inputs but also in the future ones [20]. it produces a feed-forward sequence \((f_1, f_2, \dots , f_n)\) and a backward sequence \((b_1, b_2, \dots , b_n)\) such that h = [f, b] is the concatenation of them [20]. a significant advancement on rnn was the proposal of the attention mechanism [6]. this mechanism allows a sequence-to-sequence model to pay attention to key parts of the target sequence. consequently, it permits the model to learn the correct alignment of the sentences [6]. studies have stated that attention mechanisms significantly improve the model performance on long sentences and improve the model soft-alignment [6]. consequently, it had a considerable impact on improving the results of machine translation [6]. in this work, we used a b-rnn using an attention layer as one of the algorithms to be analysed in the ats problem for portuguese. more recently, it was proposed a method called transformer, which is based solely on attention layers, dispensing with recurrence and convolutions entirely  [33]. the transformer follows a general sequence-to-sequence architecture based on encoder-decoder [33]. the basic encoder format is a stack of n layers followed by two sub-layers, the self-attention, and the multi-head attention. also, the encoder has a normalization layer and a residual connection [33]. the decoder follows a similar design using a stack of n layers with sub-layers and normalization with multi-head attention and residual connections [33]. the attention implementation proposed by [33] consists of a scaled dot product attention where the key (k), queries (q) are vectors (v) of dimension \(d_k\) and values are vectors of dimension \(d_v\) [33]. therefore, the attention to each output is calculated as given by eq. 2 where \(\frac{1}{\sqrt{d_k}}\) is the scaling factor. $$\begin{aligned} attention(q,k,v) = softmax(\frac{qk^{t}}{\sqrt{d_k}})v. \end{aligned}$$ (1) in practice, the model makes use of multi-head attention to learn different parts of the representation and in different positions [33]. besides, attention is used both in decoder and encoder and in a self-attention manner [33]. in addition to attention, the models also use a fully connected position-wise feed-forward with positional encoding layer also [33]. the transformer experimental results showed that only attention models overcame rnns in quality and required significantly less time to train as in the on two machine english-french translation tasks [33]. the development of transformer allowed the development of new promising models such as bidirectional encoder representations from transformers (bert) and many others [33]. due to its relevant results, the transformer was considered in our experiments as one of the algorithms to be analysed in the ats problem for portuguese. 3 related works ats is a relevant task, with a growing interest in natural language processing field in recent years. this section presents relevant methods developed, over the last decade, for the ats problem for brazilian portuguese. recently, the ats problem has been addressed as a monolingual machine translation problem, where a given text is translated into a simpler one. there are two relevant machine translation approaches: statistical machine translation (smt) and neural machine translation (nmt). in the smt, the translation of the original sentence f (called the translation model) into a sentence e (called the language model) is modelled on the bayes theorem as detailed in [1]. the research carried out by [30] treated the ats problem of portuguese texts as a translation task. the authors adopted the smt approach to learning how to translate complex sentences into simple ones. the smt was trained with a parallel corpus of original and simplified texts, aligned at the sentence level. the translations produced were evaluated using the metrics (bilingual evaluation understudy) bleu and manual inspection. according to both evaluations, the results were promising, and the overall sentence quality is not harmed. it was observed that some types of simplification operations, mainly lexical, are correctly captured. in summary, despite all the advancements, there is still a gap in studies on the applications of nmt to portuguese. nmt is a recently developed deep learning technique that has reached significant results over several complex tasks [2, 6, 23, 34, 39]. according to [2], nmt based methods have shown a better alternative than smt techniques for translation problems. although several works employed nmt for text simplification [2, 10, 23, 26, 32], no work has applied it in portuguese texts. to the best of our knowledge, the last works in automatic text simplification in portuguese was done more than ten years ago [3, 4, 30], and used smt. thus, this paper aims to explore state-of-the-art nmt methods for text simplification in portuguese. 4 materials and methods this section details the dataset, methods, experimental methodology and evaluation metrics of this work. 4.1 data description this work adopted a parallel corpus based on different versions of the bible to evaluate the nmt methods. the first one is a traditional version called almeida revista e corrigida (arc), published in 1997, with a complex text style. the newer version, called nova almeida atualizada (naa), was launched in 2017, simplifying the traditional versionfootnote 1. in this paper, we also evaluated other versions of the bible, such as the nova tradução linguagem de hoje (nlth), nova bíblia viva (nbv) and the nova versão internacional (nvi). considering a diverse range of versions might provide different simplification types as explored in the porsimples project [4]. each dataset has 29070 aligned verses, which will be used to create the proposed sequence-to-sequence model. table 1 provides more information about the corpora used in this study including number of tokens, sentences, readability and lexical diversity (extracted using pylinguistic library [7]). in that library, the readability ease (fre) score was proposed by  [17] and adapted to portuguese by  [22] (see eq. 2). $$\begin{aligned} fre = 206.835 (1.015 * asl) (84.6 * asw) \end{aligned}$$ (2) it adds 42 more points to the equation proposed by [22] because portuguese words has more syllable then english ones and therefore, the score would be too penalised. asl means the average sentence length and aws means average number of syllables per word. complex and simplified portuguese texts tend to have key differences. one of them is the size of each sentence and the number of tokens per sentence, as pointed out by a previous work, the porsimples project, which manually simplified a newspapers corpus. the study found that, in most cases, the simplified versions had fewer words per sentence [4]. this was also observed in both versions of the bible. in a random sample of 292 pairs of verses (0.01% of the original dataset), we analysed different aspects of the versions of the bible. the table 1 and fig. 1 shows the difference between the traditional arc versions and the other ones. table 1. the table shows descritive statistics on texts using the pylinguistic library in a random sample of 303 pairs of verses (0.01% of the original dataset). it is possible to see that simplified text has, in general, fewer tokens per sentence and more sentences.full size table figure 1 shows the histograms stating the frequency of the number of tokens per sentence of each bible. the simplified versions of the bible have fewer tokens per sentence, with more sentences under the median. the distribution of tokens per sentence in the arc is smoother, and it has a prevalence of longer sentences (i.e., more tokens per sentence). it is considerably different from the other versions, specially nlth and nbv. the difference is even greater from the arc to the nlth version in almost all aspects, such as average tokens per sentence and median sentence length. thus, both features indicate that the nlth and the other versions are easier to read because it has fewer tokens per sentences and more sentences. fig. 1. the histogram shows the distribution of tokens per sentences in both versions of the bible. in a random sample of 303 pairs of verses (0.01% of the original dataset), the split in sentences and tokens was made by using the portuguese sentencizer and tokenizer from spacy [15]. full size image table 2 exemplifies the aligned versions of arc and nlth bibles used in the experiments. it is important to remain that arc is considered the complex version to be simplified. the other versions are considered a target in separated experiments and combined in a unique dataset afterward. table 2. the table exemplifies the aligned versions of the bible used in the experiments. the passage is from genesis chapter 1.1-4.full size table 4.2 automatic text simplification this section presents the details about the automatic text simplification architecture adopted in this study. we applied the transformer and bidirectional recurrent neural network (b-rnn) architectures to simplify texts in portuguese which are state-of-the-art methods. table 3. the table shows the hyperparameters used in both models.full size table to perform the evaluation, we used the algorithms implemented at opennmt framework [18]. in opennmt, it is possible to train the model using different datasets. we considered each pair of source-target a distinct corpus and assigned different weights based on the difference between the median sentence length from the arc version to the target version of the corpus presented at table 1. further, the validation content received target examples in the same proportion of the weights. the main objective is to avoid over-fitting by increasing the training data. also, it might allow the model to learn different simplification styles, which may improve the model generalisation. the combined corpus can be identified in the following tables as “multi-corpus”. finally, the dataset was split into 17441 verses for training, 8139 verses for validation, and 3489 verses for test parallel examples. the experiments considered different target corpus where the arc bible version is always the input. also, different encoder-decoder architecture were considered (see table 5). two different experiments were performed for each model: with and without pre-trained word embeddings (portuguese glove embedding with 300 dimensions [14]). at last, a total of 20 different experiments were performed: 5 to each encoder-decoder architecture for each corpus. all the experiments considered a shared embedding and vocabulary and allowed the execution of 10000 training epochs. detailed information on the experiment setup is given in tables 3 and 4. table 4. the table shows the hyperparameters specific for each model.full size table 4.3 evaluation the evaluation used two metrics for translation and text simplification evaluation. the first one is the bilingual evaluation understudy score (bleu score) [24]. the bleu score is a widely used metric to evaluate machine translation between two languages based on a reference corpus. it also has been extensively used to assess automatic text simplification, especially the models based on mono language translation [5, 35]. another metric is the system output against references and against the input (sari score) [35]. unlike the bleu score, the initial purpose of the sari score is to evaluate text simplification, considering the system output and references and the source sentence. in summary, sari score measure how well the words are maintained or changed by the system [35]. herein, it was used the sari and bleu score implementationfootnote 2 proposed by [5]. 5 results this section presents the results obtained in the experiments. first, we discuss quantitative aspects of the supervised metrics, and then, a more in-depth discussion on the quality of the predictions is given. table 6 synthesize the results from table 5. table 5 shows the detailed results of the text simplification for each architecture and pair of datasets analyzed. in this paper context, as long as it was not possible to find a massive, parallel corpus of portuguese simplified texts and the experiment training time constraints, the b-rnn and b-rnn+embedding achieved the best results. despite the poor performance when compared with the brnn model, the transformer model might improve its performance when trained with more epochs and with a larger corpus [11, 21, 29]. table 5. the table shows the result of the text simplification to the different experiments. the b-rnn model outperforms all the other models when both metrics are considered.full size table table 6. the summary of metrics of table 5.full size table table 7. the multi-corpus prediction was produced using the b-rnn model with and without pre trained embeddings. he both models removed a specific part of the sentence to try to make it shorter which is in bold text.full size table 5.1 simplification quality one particular insight is that the simplification using multiple targets achieves a much higher bleu score but has a lower sari score in almost all experiments. this difference is due to the diverse nature of both metrics, i.e., the bleu score measures the number of unigrams of the system prediction is part of the references. in other words, it calculates a “modified precision score”, which decreases the incentive of an over-generation of a particular word to obtain a high score [24]. therefore, the high bleu score might mean that the model is sharing a significant overlap with the references in the predictionfootnote 3. on the other hand, the sari score rewards the words that are maintained in both reference and source sentence [5, 36]. it also scores the addition of new words as long as they belong to at least one reference [5, 36]. further, the metric showed to be intuitive on how the simplification gain is calculated [5, 36]. fig. 2. text readability scores, calculated by the pylinguistics library, of the text simplification made by the b-rnn model in different corpus. full size image table 8. different scores produced by the simplification made over different corpus from the best performing methods which are b-rnn with and without pre trained embeddings.full size table figure 2 show that in the single training corpora, the metrics of readability is improve and it is even higher than the readability of the reference corpus. this means that the model was able to learn the style of the target corpus as it was pointed out by previous works [19, 38]. besides, although the predicted texts have more tokens per sentence in average, the high readability score might mean that it is predicting short words as it is one of the aspects considered in the readability metric [7]. finally, the model trained with a single corpus achieved a higher sari score, indicating a better simplification. nonetheless, it could not produce a sentence with the same level of grammatical correctness and semantic meaning in this particular example as the one produced by the multi-corpus training approach. it was pointed by [31] that even though sari scores can represent the quality of the simplified sentences, the bleu score performs better on scoring the grammatical meaning of the sentences. table 7 presents an example of the outcome of the best text simplification method. it shows details about the potential of the application of the proposal in practice. as presented, the text after applying the algorithm contains more general words than the original text. even though it did not produce an exact translation of the tokens, the model is able to maintain the original meaning of the sentence and grammatical correctness. 6 conclusion neural machine translation (nmt) methods have achieved successful results for the text simplification problem in different languages, overcoming traditional statistical approaches. to the best of our knowledge, no research has investigated the application of nmt methods to simplify brazilian portuguese texts. the main contribution of this paper is the application of nmt methods for the simplification of portuguese text. two different state-of-the-art nmt methods were considered: the transformer and bidirectional recurrent neural network (b-rnn). the results demonstrated that the b-rnn was able to obtain the best results, in average (bleu = 21.84 without pre trained embedding and sari = 45.34 with pre trained embedding), despite the small corpus size and limited training epochs constraints. another significant improvement was the use of multiple corpora presenting different possible simplifications for the same input. it achieved an improvement of over 8 points on the bleu score. despite of a lower sari score, the higher bleu score might mean the ability to preserve the sentence meaning and grammatical correctness. as future works, we intend to: (i) perform an analysis on the parameters of each algorithm evaluated; (ii) use different embedding models, such as bert [11, 21, 29]; (iii) apply the nmt methods in text from different domains, such as law and health; (iv) explore the use of other methods such as lexical and syntactical simplification and pre-trained models for a mono-lingual translation approach. notes 1.versions are available at: http://altamiro.comunidades.net/biblias. 2.sari and bleu score implementation: https://github.com/feralvam/easse. 3.bleu score scale: https://cloud.google.com/translate/automl/docs/evaluate. references al-onaizan, y., et al.: statistical machine translation. in: final report, jhu summer workshop, vol. 30 (1999) google scholar  al-thanyyan, s.s., azmi, a.m.: automated text simplification: a survey. acm comput. surv. (csur) 54(2), 1–36 (2021) article  google scholar  aluisio, s., specia, l., gasperin, c., scarton, c.: readability assessment for text simplification. in: proceedings of the naacl hlt 2010 5th workshop on innovative use of nlp for building educational applications, pp. 1–9 (2010) google scholar  aluísio, s.m., gasperin, c.: fostering digital inclusion and accessibility: the porsimples project for simplification of portuguese texts. in: proceedings of the naacl hlt 2010 young investigators workshop on computational approaches to languages of the americas, pp. 46–53. association for computational linguistics (2010) google scholar  alva-manchego, f., martin, l., scarton, c., specia, l.: easse: easier automatic sentence simplification evaluation. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp): system demonstrations, hong kong, china, pp. 49–54. association for computational linguistics (november 2019). https://www.aclweb.org/anthology/d19-3009 bahdanau, d., cho, k., bengio, y.: neural machine translation by jointly learning to align and translate. arxiv preprint arxiv:1409.0473 (2014) castilhos, s., woloszyn, v., barno, d., wives, l.k.: pylinguistics: an open source library for readability assessment of texts written in portuguese. revista de sistemas de informação da fsma 18, 36–42 (2016) google scholar  chu, c., wang, r.: a survey of domain adaptation for neural machine translation. arxiv preprint arxiv:1806.00258 (2018) collantes, m., hipe, m., sorilla, j.l., tolentino, l., samson, b.: simpatico: a text simplification system for senate and house bills. in: proceedings of the 11th national natural language processing research symposium, pp. 26–32 (2015) google scholar  cooper, m., shardlow, m.: combinmt: an exploration into neural text simplification models. in: proceedings of the 12th language resources and evaluation conference, pp. 5588–5594 (2020) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) freitag, m., al-onaizan, y.: fast domain adaptation for neural machine translation. arxiv preprint arxiv:1612.06897 (2016) gao, y., et al.: ibm mastor system: multilingual automatic speech-to-speech translator. technical report, ibm thomas j watson research center yorktown heights, ny (2006) google scholar  hartmann, n., fonseca, e., shulby, c., treviso, m., rodrigues, j., aluisio, s.: portuguese word embeddings: evaluating on word analogies and natural language tasks. arxiv preprint arxiv:1708.06025 (2017) honnibal, m., montani, i., van landeghem, s., boyd, a.: spacy: industrial-strength natural language processing in python (2020). https://doi.org/10.5281/zenodo.1212303 josé, m., finatto, b.: acessibilidade textual e terminológica: promovendo a tradução intralinguística. estudos linguísticos (são paulo. 1978) 49(1), 72–96 (2020). https://doi.org/10.21165/el.v49i1.2775 kincaid, j.p., fishburne, r.p., jr., rogers, r.l., chissom, b.s.: derivation of new readability formulas (automated readability index, fog count and flesch reading ease formula) for navy enlisted personnel. technical report, naval technical training command millington tn research branch (1975) google scholar  klein, g., kim, y., deng, y., senellart, j., rush, a.: opennmt: open-source toolkit for neural machine translation. in: proceedings of acl 2017, system demonstrations, vancouver, canada, pp. 67–72. association for computational linguistics (july 2017). https://www.aclweb.org/anthology/p17-4012 krishna, k., wieting, j., iyyer, m.: reformulating unsupervised style transfer as paraphrase generation. arxiv preprint arxiv:2010.05700 (2020) liu, b., lane, i.: attention-based recurrent neural network models for joint intent detection and slot filling. arxiv preprint arxiv:1609.01454 (2016) liu, y., et al.: multilingual denoising pre-training for neural machine translation. trans. assoc. comput. linguist. 8, 726–742 (2020) article  google scholar  martins, t.b., ghiraldelo, c.m., das graças volpe nunes, m., de oliveira junior, o.n.: readability formulas applied to textbooks in brazilian portuguese. icmsc-usp (1996) google scholar  nisioi, s., štajner, s., ponzetto, s.p., dinu, l.p.: exploring neural text simplification models. in: proceedings of the 55th annual meeting of the association for computational linguistics (volume 2: short papers), pp. 85–91 (2017) google scholar  papineni, k., roukos, s., ward, t., zhu, w.j.: bleu: a method for automatic evaluation of machine translation. in: proceedings of the 40th annual meeting of the association for computational linguistics, pp. 311–318 (2002) google scholar  park, s.h., kim, b., kang, c.m., chung, c.c., choi, j.w.: sequence-to-sequence prediction of vehicle trajectory via lstm encoder-decoder architecture. in: 2018 ieee intelligent vehicles symposium (iv), pp. 1672–1678 (2018). https://doi.org/10.1109/ivs.2018.8500658 qiang, j.: improving neural text simplification model with simplified corpora. arxiv preprint arxiv:1810.04428 (2018) rescigno, a.a., vanmassenhove, e., monti, j., way, a.: a case study of natural gender phenomena in translation a comparison of google translate, bing microsoft translator and deepl for english to italian, french and spanish. in: association for machine translation in the americas (amta): workshop on the impact of machine translation, impact 2020, p. 62. workshop on the impact of machine translation (impact 2020) at association (2020) google scholar  sikka, p., singh, m., pink, a., mago, v.: a survey on text simplification. arxiv preprint arxiv:2008.08612 (2020) souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: 9th brazilian conference on intelligent systems, bracis, rio grande do sul, brazil, 20–23 october (2020, to appear) google scholar  specia, l.: translating from complex to simplified sentences. in: pardo, t.a.s., branco, a., klautau, a., vieira, r., de lima, v.l.s. (eds.) propor 2010. lncs (lnai), vol. 6001, pp. 30–39. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-12320-7_5 chapter  google scholar  sulem, e., abend, o., rappoport, a.: semantic structural evaluation for text simplification. arxiv preprint arxiv:1810.05022 (2018) sulem, e., abend, o., rappoport, a.: simple and effective text simplification using semantic and neural methods. arxiv preprint arxiv:1810.05104 (2018) vaswani, a., et al.: attention is all you need. arxiv preprint arxiv:1706.03762 (2017) wang, t., chen, p., amaral, k., qiang, j.: an experimental study of lstm encoder-decoder model for text simplification. arxiv preprint arxiv:1609.03663 (2016) xu, w., napoles, c., pavlick, e., chen, q., callison-burch, c.: optimizing statistical machine translation for text simplification. trans. assoc. comput. linguist. 4, 401–415 (2016). https://cocoxu.github.io/publications/tacl2016-smt-simplification.pdf xu, w., napoles, c., pavlick, e., chen, q., callison-burch, c.: optimizing statistical machine translation for text simplification. trans. assoc. comput. linguist. 4, 401–415 (2016) article  google scholar  yamada, m.: the impact of google neural machine translation on post-editing by student translators. j. specialised transl. 31, 87–106 (2019) google scholar  yang, z., hu, z., dyer, c., xing, e.p., berg-kirkpatrick, t.: unsupervised text style transfer using language models as discriminators. arxiv preprint arxiv:1805.11749 (2018) zaremba, w., sutskever, i., vinyals, o.: recurrent neural network regularization. arxiv preprint arxiv:1409.2329 (2014) download references author information authors and affiliations universidade federal rural de pernambuco, rua dom manuel de medeiros, recife, pernambuco, 52171-900, brazil tiago b. de lima, andré c. a. nascimento, george valença, pericles miranda & rafael ferreira mello bankura unnayani institute of engineering, subhankar nagar, bankura, pohabagan, 722146, west bengal, india tapas si authors tiago b. de limaview author publications search author on:pubmed google scholar andré c. a. nascimentoview author publications search author on:pubmed google scholar george valençaview author publications search author on:pubmed google scholar pericles mirandaview author publications search author on:pubmed google scholar rafael ferreira melloview author publications search author on:pubmed google scholar tapas siview author publications search author on:pubmed google scholar corresponding authors correspondence to tiago b. de lima , andré c. a. nascimento or rafael ferreira mello . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper de lima, t.b., nascimento, a.c.a., valença, g., miranda, p., mello, r.f., si, t. (2021). portuguese neural text simplification using machine translation. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_37 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_37 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords text simplification machine translation deep learning natural language processing publish with us policies and ethics profiles andré c. a. nascimento view author profile rafael ferreira mello view author profile tapas si view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature mrat-sql+gap: a portuguese text-to-sql transformer | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper mrat-sql+gap: a portuguese text-to-sql transformer conference paper first online: 28 november 2021 pp 511–525 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) mrat-sql+gap: a portuguese text-to-sql transformer download book pdf download book epub marcelo archanjo josé  orcid: orcid.org/0000-0001-7153-040210 & fabio gagliardi cozman  orcid: orcid.org/0000-0003-4077-493511  part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1360 accesses 16 citations 5 altmetric abstract the translation of natural language questions to sql queries has attracted growing attention, in particular in connection with transformers and similar language models. a large number of techniques are geared towards the english language; in this work, we thus investigated translation to sql when input questions are given in the portuguese language. to do so, we properly adapted state-of-the-art tools and resources. we changed the rat-sql+gap system by relying on a multilingual bart model (we report tests with other language models), and we produced a translated version of the spider dataset. our experiments expose interesting phenomena that arise when non-english languages are targeted; in particular, it is better to train with original and translated training datasets together, even if a single target language is desired. this multilingual bart model fine-tuned with a double-size training dataset (english and portuguese) achieved 83% of the baseline, making inferences for the portuguese test dataset. this investigation can help other researchers to produce results in machine learning in a language different from english. our multilingual ready version of rat-sql+gap and the data are available, open-sourced as mrat-sql+gap at: https://github.com/c4ai/gap-text2sql. supported by ibm and fapesp (são paulo research foundation). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a multilingual translator to sql with database schema pruning to improve self-attention article 23 june 2023 text-to-sql for non-english languages: realizations and limitations chapter © 2026 a survey on text-to-sql parsing: from rule-based foundations to large language models chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. database management system languages language translation lisp machine translation slavic and baltic languages 1 introduction a huge number of data is now organized in relational databases and typically accessed through sql (structured query language) queries. the interest in automatically translating questions expressed in natural language to sql (often referred to as nl2sql) has been intense, as one can observe through a number of excellent surveys in the literature [1,2,3]. figure 1 depicts the whole flow from a natural language question to a sql query result; the sql query refers to database tables and their columns, using primary and secondary keys as appropriate. existing approaches for nl2sql can be divided into entity-based and machine learning ones, the latter dominated by techniques based on deep learning [3]. fig. 1. from a natural language question to a sql query and to the database query result. database table names appear in blue (singer) and red (song), while the primary key appears in green (singer_id). (color figure online) full size image entity-based approaches focus on the interpretation of input text based on rules so as to translate it to a sql query. the translation often goes first to an intermediary state and later to a final sql query. relevant systems are bela [4], soda [5], nalir [6,7,8], tr discover [9], athena [10, 11], athena++ [12] and duoquest [13, 14] . machine learning approaches are based on supervised learning, in which training data contains natural language questions and paired sql queries [3]. several architectures can be trained or fine-tuned so as to run the translation. relevant systems are echoquery [15], seq2sql [41], sqlnet [16], dialsql [17], typesql [18], syntaxsqlnet [19], augmentedbi [21], irnet [22], rat-sql [23], rat-sql+gap [24], grappa [25], bridge [26], dbpal [27,28,29], hydranet [30], dt-fixup [32] and lgesql [31]. there are also hybrid approaches that combine entity-based and machine learning [3]; relevant examples are aqqu [34], means [35] and quest [33]. the previous paragraphs contain long lists of references that should suffice to demonstrate that translation from natural language to sql is a well explored research topic. the field is relatively mature and benchmarks for nl2sql are now widely used, containing training and testing data and ways to evaluate new proposals. table 1 shows a few important datasets in the literature, reporting their number of questions, number of sql queries, number of databases, number of domain and tables per database [20]. particularly, the spider dataset is a popular resource that contains 200 databases with multiple tables under 138 domains.footnote 1 the complexity of these tables allows testing complex nested sql queries. a solid test suite evaluation package for testing against spider [46] is available;footnote 2 in addition, there is a very active leaderboard rank for tests that use spider.footnote 3 table 1. nl2sql benchmarks.full size table currently, the best result in the spider leaderboard for exact set match without values, whereby a paper and code are available, is the entry by rat-sql+gap [24].footnote 4 this system appears in the 6th rank position with dev 0.718 and test 0.697.footnote 5 note that the spider leaderboard, as of august 2021, displays in the 1st rank position lgesql [31] with dev 0.751 and test 0.720 for exact set match without values. thus rat-sql+gap is arguably at the state-of-art in nl2sql. rat-sql+gap is based on the rat-sql package (relation-aware transformer sql) [23]. rat-sql was proposed in 2019 as a text to sql parser based on the bert language model [47]. package rat-sql version 3 with bert is currently the 14th entry in the spider leaderboard rank. rat-sql+gap adds generation-augmented pre-training (gap) to rat-sql. gap produces synthetic data to increase the dataset size to improve pre-training; the whole generative models are trained by fine-tuning a bart [48] large model. despite the substantial number of techniques, systems and benchmarks for nl2sql, most of them focus on the english language. very few results can be found for input questions in the portuguese language, for example. the study by silva et al. [51] presents an architecture for nl2sql in which natural language questions in portuguese are translated to the english language on arrival, and are then shipped to nl2sql existing packages. the goal of this paper is simple to state: we present a translator for queries in portuguese natural language into sql. we intend to study the effect of replacing the questions in the spider dataset with translated versions, and also to investigate how to adapt the rat-sql+gap system to the needs of a different language. using a new version of spider with rat-sql+gap to train models, we produce inferences and compare results so as to understand the difficulties and limitations of various ideas. what we found is that, by focusing on portuguese, we actually produced methods and results that apply to any multilingual nl2sql task. an important insight (and possibly the main contribution of this paper) is dealing with a non-english language, such as portuguese, we greatly benefit from taking a multilingual approach that puts together english and the other language—in our case, english and portuguese. we later stress this idea when we discuss our experiments. we thus refer to our “multilingual-ready” version of rat-sql+gap as mrat-sql+gap; all code and relevant data related to this system are freely availablefootnote 6. 2 preliminary tasks the adaptation to language other than english demands at least the translation of the dataset and changing the code to read and write files utf-8 encoding. 2.1 translating the spider dataset the translation to the portuguese language in the nl2sql task evolves the natural language part, the questions. the sql queries must remain the same to make sense. to translate the questions, it is important to extract them from specific .json files. the spider dataset has three files that contain input questions and their corresponding sql queries: dev.json, train_others.json, and train_spider.json. we extracted the questions and translated them using the google cloud translation api.footnote 7. table 2 presents the number of questions and number of characters per file (just for the questions). the code that reads the original files and that generates translated versions relies on the utf-8 encoding so as to accept portuguese characters; several files were generated in the process (.txt just for the translated questions, .csv for the sql queries and original/translated questions, .json for the translated questions). table 2. number of questions and characters per file.full size table we then conducted a revision process by going through the text file and looking for questions in the csv file (together with the corresponding sql queries). after the revision, a new .json file was generated with these translated and revised questions. table 3 shows four examples of translated questions. table 3. translation examples.full size table adapting the rat-sql+gap system. we had to change the rat-sql+ gap code to allow multilingual processing. for instance, the original python code is not prepared to handle utf-8 files; thus, we had to modify the occurrences of “open” and “json.dump” commands, together with a few other changes. we ran a rat-sql+gap test and checked whether all the characters employed in portuguese were preserved. we also noticed lemmatization errors in preprocessed files. as the original code for rat-sql+gap relies on the stanford corenlp lemmatization tool that currently does not support portuguese, it was replaced by simplemma.footnote 8 the latter package supports multilingual texts, and particularly supports portuguese and english. training. the original language model at the heart of rat-sql+gap is bart-largefootnote 9 [48], a language model pretrained for the english language. we had to change that model to another one that was pretrained for the portuguese language. a sensible option was to work with a multilingual sequence-to-sequence bart model; the choice was mbart-50footnote 10 [50] because it covers portuguese and english languages (amongst many others). another language model we investigated was the bertimbau-basefootnote 11 [49], as rat-sql works with bert; the move to berttimbau, a portuguese-based version of bert, seemed promising. dataset. a total of 8,659 questions were used for training (7,000 questions in train_spider.json and 1,659 questions in train_others.json). the 1,034 questions in dev.json were used for testing. we later refer to three scenarios: english train and test: questions are just in english for training and testing. portuguese train and testfootnote 12: questions are just in portuguese for training and testing. english and portuguese (double-size) train and test(see footnote 12): questions are in english and portuguese for training and testing (in this case, we thus have twice as much data as in each of the two previous individual scenarios). evaluation metrics. the main evaluation metric with respect to spider is exact set match (esm). we here present results for exact set match (esm) without values, as most results in the spider leaderboard currently adopt this metric. some spider metrics are also used to classify the sql queries into 4 levels: easy, medium, hard and extra hard. table 4 shows an example for each level; these queries correspond to the four questions in table 3 in the same order. to evaluate the results, we used the spider test suite evaluation [46]. an aside: the suite must receive a text file with the sql query generated and another one with the gold-standard sql query. these files obviously do not change when we move from english to any other input language. as a digression, note that it is possible to plug values during evaluation. a query with a value looks like this: select count(*) from airlines join airports where airports.city = “abilene" a query without values has the word “terminal" instead of the value: select count(*) from airlines join airports where airports.city = “terminal" table 4. sql query levels: easy/medium/hard/extra.full size table 3 experiments experiments were run in a machine with amd ryzen 9 3950x 16-core processor, 64 gb ram, 2 gpus nvidia geforce rtx 3090 24 gb running ubuntu 20.04.2 lts. figure 2 shows the architecture of the training, inference and evaluation processes described in this section. results can be found in table 5. this table shows the results of exact set match without values for rat-sql+gap trained locally for all models. we have 3 datasets, 5 trained model checkpoints and 7 distinct relevant results. the first line corresponds to the original model bart and original questions in english. note that the result in line #1 achieved the same performance reported by [24] for exact set match without values in spider: 0.718 (all) for dev. this indicates that our testing apparatus can produce state-of-the-art results. moreover, line #1 shows a well-tuned model in english that can be attained. fig. 2. architecture of the training, inference, evaluation. results related to table 5: each line in that table appears here as a square at the bottom of the figure. full size image our first experiment was to change the questions to portuguese and the model to bertimbau-base which is pretrained in portuguese. in table 5 the result in line #2 for bertimbau 0,417(all) is quite low when compared to the result in line #1. this happens for many reasons. the model is bert and the best result uses bart. another important difference is that the portuguese content used in fine-tuning has a mixture of portuguese and english words because the sql query inevitably consists of english keywords, see examples of sql queries in table 4. in fact, some questions demand untranslated words to make sense, for example “boston red stockings” in the translated question: qual é o salário médio dos jogadores da equipe chamada "boston red stockings"?. this suggested that a multilingual approach might be more successful. the mbart-50 language model was then tested within the whole architecture. mbart-50 was in fact fine-tuned in three different ways: with questions only in english, only in portuguese, and with questions both in english and in portuguese (that is, a dataset with questions and their translations). inferences were run with english test questions and portuguese test questions, while the model was fine-tuned with the corresponding training questions language. for the mbart-50 model fine-tuned with the train dataset with two languages, three inferences were made: only english, only portuguese, and the combined english portuguese test dataset. mbart-50 fine-tuned with questions in english in line #3 achieved 0.651 (all) when tested with questions in english. the same model mbart-50 fine-tuned with questions in portuguese in line #4 achieved 0.588(all) when tested with questions in portuguese. a multilingual model such as mbart-50 can be trained with the two languages at the same time. this is certainly appropriate for data augmentation and to produce a fine-tuning process that can better generalize. the results in table 5 lines #5, #6 and #7 were obtained with mbart-50 fine-tuned with the double-size training dataset (english and portuguese); the three inferences were made using the same model checkpoint. the test datasets were in english for line #5, in portuguese for line #6, and the double-size test dataset in english and portuguese for line #7. the results demonstrate improvements, if we compare inferences with english test dataset lines #3 and #5. results went up from 0.651 (all) to 0.664 (all), better for all levels of questions. if we compare inferences with the portuguese test dataset in lines #4 and #6, the results went from 0.588 (all) to 0.595 (all). however, they are better just for an easy level of questions; this was enough to influence the overall results for line #6. table 5. results.full size table fig. 3. examples of keywords in the prediction of the sql query in english and portuguese languages. top pair: question in english and corresponding sql query predicted from it. bottom pair: question in portuguese and corresponding sql query predicted from it. full size image the inference made with the double-size test dataset in english and portuguese, in line #7, cannot be compared with the other inferences because they used just one language. nevertheless, the model mbart-50 trained with english and portuguese (double-size training dataset) produced good results with this rather uncommon testing dataset. all the detailed results presented in this paper are openly availablefootnote 13. 4 analysis and discussion these experiments indicate that multilingual pretrained transformers can be extremely useful when dealing with languages other than english. there is always the need to integrate english processing with the additional languages of interest, in our case, with portuguese. overall, questions in the english language have a closer similarity with sql queries, thus simplifying inferences. conversely, questions in portuguese require further work. figure 3 show a real example of correct predictions in english (table 5 line #5) and in portuguese (table 5 line #6). in fig. 3, words such as song, names, singers, average in the english question are keywords needed to resolve the query, and they are very close to the target word in the query. in portuguese, the same keywords are músicas, nomes, cantores, média that must respectively match song, name, singer, and avg. this introduced an additional level of difficulty that explains the slightly worse results for inference with questions in portuguese: 0.595 (table 5 line #6). this is to be compared with questions in english: 0.664 (table 5 line #5). in any case, it is surprising that the same model checkpoint resolved the translation with such different questions. fig. 4. example of words that represent the value in the prediction of the sql query. en q: english question, pt q: question translated to portuguese, p: sql query predicted, and g: gold sql query. full size image some translations actually keep a mix of languages, because some words represent the value and cannot be translated. figure 4 shows an example. there the show name “the rise of the blue beetle” should not be translated to maintain the overall meaning of the question; these words in english must be part of the portuguese question. this successful sql query inference was produced with mbart-50 fine-tuned with the english and portuguese training dataset (table 5 line #6) and mbart-50 fine-tuned in the portuguese-only training dataset (table 5 line #4). the show name was then replaced with “terminal” during the rat-sql+gap prediction process, as it is processed through the spider exact set match without values evaluation. in any case, the show name is part of the input and will introduce difficulties in the prediction. in addition, for real-world databases, it is a practice, at least in brazil, to name tables and columns with english words even for databases with content in portuguese. this practical matter is another argument in favor of a multilingual approach. figure 5 shows a sample of failed translations evaluated by the spider exact set match for inferences using mbart-50 (table 5 line #6). it is actually difficult to find the errors without knowing the database schema related to every query. the objective of this figure is to show that even when the query is incorrect, it is not composed of random or nonsensical words. our manual analysis indicates that this is true for queries failing with all other models. fig. 5. sample of failed queries evaluated by spider exact set match for inferences from mbart-50 (table 5 line #6) . q: question, p: sql query predicted, and g: gold sql query. full size image 5 conclusion and future work in sum, we have explored the possible ways to create a translator that takes questions in the portuguese language and outputs correct sql queries corresponding to the questions. by adapting a state-of-the-art nl2sql to the portuguese language, our main conclusion is that a multilingual approach is needed: it is not enough to do everything in portuguese; rather, we must simultaneously work with english and portuguese. in table 5, our best result is in line #5 0.664 (all) whereby we test with questions in english (original test set from spider) using mbart-50 model fine-tuned with a double-size training dataset (english and portuguese). this yields 92% of the english-only performance of 0.718 (all) in line #1. testing with questions in portuguese (using our translation) with the same bart-50 model fine-tuned with a double-size training dataset (english and portuguese), we achieve instead 0.595 (all) line #6. now this is 83% of the english-only performance of 0.718 (all). these results should work as a baseline for future nl2sql research in portuguese. our multilingual rat-sql+gap, or mrat-sql+gap for short, the translated datasets, the trained checkpoint, and the results; are open-source availablefootnote 14. future work should try other multilingual transformers (and possibly other seq-to-seq models), always seeking ways to use english and portuguese together. another possible future work is to fine-tune bertimbau-largefootnote 15 [49] so as to better understand the effect of the size of the language model. lastly, a translation of the spider dataset to other languages so as to work with several languages at the same time should produce valuable insights. notes 1.spider dataset: https://yale-lily.github.io/spider. 2.spider test suite evaluation github:https://github.com/taoyds/test-suite-sql-eval. 3.spider leaderboard rank: https://yale-lily.github.io/spider. 4.rat-sql+gap github: https://github.com/awslabs/gap-text2sql. 5.dev results are obtained locally by the developer; to get official score and test results, it is necessary to submit the model following guidelines in “yale semantic parsing and text-to-sql challenge (spider) 1.0 submission guideline” at https://worksheets.codalab.org/worksheets/0x82150f426cb94c17b861ef4162817399/. 6.mrat-sql+gap github: https://github.com/c4ai/gap-text2sql. 7.cloud translation api: https://googleapis.dev/python/translation/latest/index.html. 8.simplemma: a simple multilingual lemmatizer for python at https://github.com/adbar/simplemma. 9.facebook bart-large: https://huggingface.co/facebook/bart-large. 10.facebookmbart-50manyfordifferentmultilingualmachinetranslations: https://huggingface.co/facebook/mbart-large-50-many-to-many-mmt. 11.bertimbau-base: https://huggingface.co/neuralmind/bert-base-portuguese-cased. 12.spider dataset translated to portuguese and double-size (english and portuguese together): https://github.com/c4ai/gap-text2sql. 13.mrat-sql+gap github: https://github.com/c4ai/gap-text2sql. 14.mrat-sql+gap github: https://github.com/c4ai/gap-text2sql. 15.bertimbau-large: https://huggingface.co/neuralmind/bert-large-portuguese-cased. references kim, h., so, b.h., han, w.s., lee, h.: natural language to sql: where are we today? proc. vldb endow. 13, 1737–1750 (2020). https://doi.org/10.14778/3401960.3401970 affolter, k., stockinger, k., bernstein, a.: a comparative survey of recent natural language interfaces for databases. vldb j. 28, 793–819 (2019). https://doi.org/10.1007/s00778-019-00567-8 ozcan, f., quamar, a., sen, j., lei, c., efthymiou, v.: state of the art and open challenges in natural language interfaces to data. in: proceedings of the acm sigmod international conference on management of data, pp. 2629–2636 (2020). https://doi.org/10.1145/3318464.3383128 walter, s., unger, c., cimiano, p., bär, d.: evaluation of a layered approach to question answering over linked data. in: cudré-mauroux, p., et al. (eds.) iswc 2012. lncs, vol. 7650, pp. 362–374. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-35173-0_25 chapter  google scholar  blunschi, l., jossen, c., kossmann, d., mori, m., stockinger, k.: soda: generating sql for business users. proc. vldb endow. 5, 932–943 (2012). https://doi.org/10.14778/2336664.2336667 li, f., jagadish, h. v: constructing an interactive natural language interface for relational databases. proc. vldb endow. 8, 73–84 (2014). https://doi.org/10.14778/2735461.2735468 li, f., jagadish, h. v.: nalir: an interactive natural language interface for querying relational databases. in: proceedings of the 2014 acm sigmod international conference on management of data, new york, pp. 709–712. acm (2014). https://doi.org/10.1145/2588555.2594519 li, f., jagadish, h.v.: understanding natural language queries over relational databases. acm sigmod rec. 45, 6–13 (2016). https://doi.org/10.1145/2949741.2949744 article  google scholar  song, d., et al.: tr discover: a natural language interface for querying and analyzing interlinked datasets. in: arenas, m., et al. (eds.) iswc 2015. lncs, vol. 9367, pp. 21–37. springer, cham (2015). https://doi.org/10.1007/978-3-319-25010-6_2 chapter  google scholar  saha, d., floratou, a., sankaranarayanan, k., minhas, u.f., mittal, a.r., özcan, f.: athena: an ontology-driven system for natural language querying over relational data stores. proc. vldb endow. 9, 1209–1220 (2016). https://doi.org/10.14778/2994509.2994536 lei, c., et al.: ontology-based natural language query interfaces for data exploration. ieee data eng. bull. 41, 52–63 (2018) google scholar  sen, j., et al.: athena++: natural language querying for complex nested sql queries. proc. vldb endow. 13, 2747–2759 (2020). https://doi.org/10.14778/3407790.3407858 baik, c., arbor, a., arbor, a., arbor, a., jagadish, h.v: constructing expressive relational queries with dual-specification synthesis. in: proceedings of the 10th annual conference innovations data systems research (cidr 2020) (2020) google scholar  baik, c., jin, z., cafarella, m., jagadish, h. v.: duoquest: a dual-specification system for expressive sql queries. in: proceedings of the acm sigmod international conference on management of data, pp. 2319–2329 (2020). https://doi.org/10.1145/3318464.3389776 lyons, g., tran, v., binnig, c., cetintemel, u., kraska, t.: making the case for query-by-voice with echoquery. in: proceedings of the acm sigmod international conference on management of data, 26-june-20, pp. 2129–2132 (2016). https://doi.org/10.1145/2882903.2899394 xu, x., liu, c., song, d.: sqlnet: generating structured queries from natural language without reinforcement learning. \({\rm arxiv}\). pp. 1–13 (2017) google scholar  gur, i., yavuz, s., su, y., yan, x.: dialsql: dialogue based structured query generation. in: acl 2018–56th annual meeting of the association for computational linguistics proceedings of the conference (long paper 1), pp. 1339–1349 (2018). https://doi.org/10.18653/v1/p18-1124 yu, t., li, z., zhang, z., zhang, r., radev, d.: typesql: knowledge-based type-aware neural text-to-sql generation. in: naacl hlt 2018 2018 conference of the north american chapter of the association for computational linguistics: human language technologies proceedings of the conference, vol. 2, pp. 588–594 (2018). https://doi.org/10.18653/v1/n18-2093 yu, t., yasunaga, m., yang, k., zhang, r., wang, d., li, z., radev, d.r.: syntaxsqlnet: syntax tree networks for complex and cross-domain text-to-sql task. in: proceedings of the 2018 conference on empirical methods in natural language processing, pp. 1653–1663. association for computational linguistics, brussels, belgium (2018) google scholar  yu, t., et al.: spider: a large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task. arxiv:1809.08887v5 (2018) francia, m., golfarelli, m., rizzi, s.: augmented business intelligence. in: ceur workshop proceedings, vol. 2324 (2019) google scholar  guo, j., et al.: towards complex text-to-sql in cross-domain database with intermediate representation (2019) google scholar  wang, b., shin, r., liu, x., polozov, o., richardson, m.: rat-sql: relation-aware schema encoding and linking for text-to-sql parsers. arxiv. (2019). https://doi.org/10.18653/v1/2020.acl-main.677 shi, p., et al.: learning contextual representations for semantic parsing with generation-augmented pre-training (2020) google scholar  yu, t., et al.: grappa: grammar-augmented pre-training for table semantic parsing (2020) google scholar  lin, x.v., socher, r., xiong, c.: bridging textual and tabular data for cross-domain text-to-sql semantic parsing, pp. 4870–4888 (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.438 utama, p., et al.: dbpal: an end-to-end neural natural language interface for databases (2018) google scholar  basik, f., et al.: dbpal: a learned nl-interface for databases. in: proceedings of the acm sigmod international conference on management of data, pp. 1765–1768 (2018). https://doi.org/10.1145/3183713.3193562 weir, n., et al.: dbpal: a fully pluggable nl2sql training pipeline. in: proceedings of the 2020 acm sigmod international conference on management of data, new york, pp. 2347–2361. acm (2020). https://doi.org/10.1145/3318464.3380589 lyu, q., chakrabarti, k., hathi, s., kundu, s., zhang, j., chen, z.: hybrid ranking network for text-to-sql. arxiv. pp. 1–12 (2020) google scholar  cao, r., chen, l., chen, z., zhao, y., zhu, s., yu, k.: lgesql: line graph enhanced text-to-sql model with mixed local and non-local relations, pp. 2541–2555 (2021). https://doi.org/10.18653/v1/2021.acl-long.198 xu, p., et al.: optimizing deeper transformers on small datasets, pp. 2089–2102 (2021). https://doi.org/10.18653/v1/2021.acl-long.163 bergamaschi, s., guerra, f., interlandi, m., trillo-lado, r., velegrakis, y.: combining user and database perspective for solving keyword queries over relational databases. inf. syst. 55, 1–19 (2016). https://doi.org/10.1016/j.is.2015.07.005 bast, h., haussmann, e.: more accurate question answering on freebase. in: proceedings of the international on conference on information and knowledge management, 19–23-october 2015, pp. 1431–1440 (2015). https://doi.org/10.1145/2806416.2806472 ben abacha, a., zweigenbaum, p.: means: a medical question-answering system combining nlp techniques and semantic web technologies. inf. process. manag. 51, 570–594 (2015). https://doi.org/10.1016/j.ipm.2015.04.006 article  google scholar  iyer, s., konstas, i., cheung, a., krishnamurthy, j., zettlemoyer, l.: learning a neural semantic parser from user feedback. in: acl 2017–55th annual meeting of the association for computational linguistics proceeding conference (long paper 1), pp. 963–973 (2017). https://doi.org/10.18653/v1/p17-1089 giordani, a., moschitti, a.: translating questions to sql queries with generative parsers discriminatively reranked. in: coling, pp. 401–410 (2012) google scholar  popescu, a.m., etzioni, o., kautz, h.: towards a theory of natural language interfaces to databases. in: international conference on intelligent user interfaces, proceedings of the iui, pp. 149–157 (2003). https://doi.org/10.1145/604050.604070 zelle, j.m., mooney, r.j.: learning to parse database queries using inductive logic programming. in: proceedings of the national conference on artificial intelligence, pp. 1050–1055 (1996) google scholar  zettlemoyer, l.s., michael, c.: learning to map sentences to logical form: structured classification with probabilistic categorial grammars. in: proceedings of the 21st conference on uncertain artificial intelligence, uai 2005, pp. 658–666 (2005) google scholar  zhong, v., xiong, c., socher, r.: seq2sql: generating structured queries from natural language using reinforcement learning. arxiv:1709.00103v7. pp. 1–12 (2017) zettlemoyer, l.s., collins, m.: online learning of relaxed ccg grammars for parsing to logical form. in: emnlp-conll 2007 proceedings of the 2007 joint conference on empirical methods in natural language processing and computational natural language learning, pp. 678–687 (2007) google scholar  price, p.j.: evaluation of spoken language systems. in: proceedings of the workshop on speech and natural language hlt 1990, pp. 91–95. association for computational linguistics, morristown, nj, usa (1990). https://doi.org/10.3115/116580.116612 dahl, d.a., et al.: expanding the scope of the atis task 43 (1994). https://doi.org/10.3115/1075812.1075823 hemphill, c.t., godfrey, j.j., george, r.d.: the atis spoken language systems pilot corpus. in: proceedings of the darpa speech and natural language workshop., hidden valley, pennsylvania (1990) google scholar  zhong, r., yu, t., klein, d.: semantic evaluation for text-to-sql with distilled test suites. arxiv. (2020). https://doi.org/10.18653/v1/2020.emnlp-main.29 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: naacl hlt 2019–2019 conference of the north american chapter of the association for computational linguistics: human language technologies proceedings of the conference, vol. 1, pp. 4171–4186 (2019) google scholar  lewis, m., et al.: bart: denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension (2019) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  tang, y., et al.: multilingual translation with extensible multilingual pretraining and finetuning. arxiv:2008.00401 (2020) da silva, c.f.m., jindal, r.: sql query from portuguese language using natural language processing. in: garg, d., wong, k., sarangapani, j., gupta, s.k. (eds.) iacc 2020. ccis, vol. 1367, pp. 323–335. springer, singapore (2021). https://doi.org/10.1007/978-981-16-0401-0_25 chapter  google scholar  download references acknowledgment this work was carried out at the center for artificial intelligence (c4ai-usp), supported by the são paulo research foundation (fapesp grant #2019/07665-4) and by the ibm corporation. the second author is partially supported by the conselho nacional de desenvolvimento científico e tecnológico (cnpq), grant 312180/2018-7. author information authors and affiliations instituto de estudos avançados, universidade de são paulo and center for artificial intelligence (c4ai), são paulo, brazil marcelo archanjo josé escola politécnica, universidade de são paulo and center for artificial intelligence (c4ai), são paulo, brazil fabio gagliardi cozman authors marcelo archanjo joséview author publications search author on:pubmed google scholar fabio gagliardi cozmanview author publications search author on:pubmed google scholar corresponding author correspondence to marcelo archanjo josé . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper josé, m.a., cozman, f.g. (2021). mrat-sql+gap: a portuguese text-to-sql transformer. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_35 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_35 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords nl2sql deep learning rat-sql+gap spider dataset bart bertimbau publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature exploring text decoding methods for portuguese legal text generation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper exploring text decoding methods for portuguese legal text generation conference paper first online: 12 october 2023 pp 63–77 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) exploring text decoding methods for portuguese legal text generation download book pdf download book epub kenzo sakiyama  orcid: orcid.org/0000-0002-5284-47459, raphael montanari  orcid: orcid.org/0000-0003-2281-36469, roseval malaquias junior  orcid: orcid.org/0000-0002-6005-05159, rodrigo nogueira  orcid: orcid.org/0000-0002-2600-603510 & … roseli a. f. romero  orcid: orcid.org/0000-0001-9366-27809  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 552 accesses 4 citations abstract in recent years, there has been considerable growth in the volume of legal proceedings in brazil. in this context, there is a lot of potential in using recent advances in natural language processing to automate tasks and analysis in the legal domain. in this article, we investigate text decoding methods for automating the writing of keyphrases, a sequence of key terms present in documents used in courts throughout brazil. for this purpose, a text-to-text framework based on generative transformers is used to generate keyphrases and evaluate three decoding techniques: greedy, top-k, and top-p. since the keyphrases are designed to improve retrieval tasks, we evaluated keyphrases generated by the decoding methods in legal document retrieval. traditional retrieval methods (tf-idf and bm25) were used to evaluate the quality of the generated keyphrases. the results obtained (in terms of ir metrics) were statistically significant, and they indicate that greedy decoding generates high-quality keyphrases for the dockets used in this work, providing keyphrases close to the ones generated by human specialists. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a deep extraction model for an unseen keyphrase detection article 12 november 2019 from fundamentals to recent advances: a tutorial on keyphrasification chapter © 2022 a comparative assessment of state-of-the-art methods for multilingual unsupervised keyphrase extraction chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics escrt machine translation natural language processing (nlp) neural decoding 1 introduction based on data from the national council of justice conselho nacional de justiça (cnj) [11], there were 77.3 million cases in transit in the brazilian judiciary at the end of 2021, indicating a 10.4% increase from the previous year. the analysis of such cases contributes to the slowness of the brazilian legal system due to the human effort required to both write and analyze the legal cases. in this context, dockets consist of a special type of document, that aims to provide a summary of a legal case. they are used in courts all around brazil and are designed to provide a summarised representation of judicial decisions. figure 1 presents an example of a docket. fig. 1. example of a docket. keyphrases are highlighted in bold text. full size image the dockets usually follow a pre-defined structure composed of two components: keyphrases and enumerated paragraphs. the keyphrases consist of a header present at the beginning of the docket and are composed of sequences of capitalised key terms that highlight the key subjects present in the document. this header is created to improve the search and retrieval of jurisprudences (precedents) [7]. the enumerated paragraphs discuss the themes (or topics) present in the document. by analysing the form and linguistic style present in the keyphrases, it’s possible to note similarities between the writing of keyphrases and two natural language processing (nlp) tasks: summarization and key terms extraction. however, keyphrases are not written in a fluid and natural manner such as summaries. in addition, most of the terms present in their text are not present in the remainder of the docket which originated the keyphrase, which makes it difficult to treat its writing as an extractive task. given the predictable structure and availability of dockets, it would be possible to prepare input-output pairs in order to generate keyphrases using the enumerated paragraphs as inputs, by employing a supervised approach. transformers, such as gpt [19], were already proven effective in various text-to-text generative natural language processing (nlp) tasks [6] (such as translation, question answering and summarization). also, the availability of pre-trained language models [3, 25, 33] presents a lot of opportunities to automate nlp tasks. thus, in this work, we aim to investigate the usage of state-of-the-art generative transformers to automate the writing of keyphrases. specifically, we seek to investigate text decoding methods in order to generate keyphrases that aid retrieval in the legal domain. this study is unprecedented in brazil and can be used to automate keyphrase generation in courts around the country. at last, the main contributions of this work are: 1. investigation of a novel approach to generate keyphrases from brazilian dockets, using a sequence-to-sequence transformer; 2. comparison of three different text decoding methods for the proposed task (greedy and sampling methods); 3. quantitative and qualitative analysis of the generated keyphrases. this paper is organised as follows. section 2 presents related works. section 3 discussed the methodology applied for the keyphrase generation. next, sect. 4 presents and discusses the obtained results. at last, sect. 5, presents conclusions and future works. 2 related works in this section, we will present studies related to the main objectives of this proposal. the transformer [29] consists of a deep neural network architecture that achieved the state of the art in several nlp tasks. it consists of an encoder-decoder architecture used originally for translation. however, the context-aware representations generated by the model can be used for a large variety of tasks. following the success of the generative pretrained transformers (gpt) models [19, 25, 33], there is a predominance of decoder-only models in nlp tasks that can be approached as text generation (such as question answering and summarization) [6]. in addition, recent studies showed the great potential in using such models in zero and few-shot scenarios [2]. other studies [20, 31] investigate text generation using the full transformer architecture (encoder-decoder) for some nlp tasks. the t5 [20] transformer proposes the unification of a series of nlp tasks in a single text-to-text framework and xue et al.  [31] expanded the original work to add multilingual support. although the presented transformer approaches for text generation are different in terms of architecture and scale (number of parameters), they all deal with common issues concerning the quality of the artificially generated text. generated texts are often simplistic, inconsistent, or end up being repetitive [8]. there is also the possibility of hallucination, generating contradictory texts, meaningless and without foundation or evidence [10]. in order to mitigate the challenges (repetitive and predictable texts), there were initiatives aimed at making text generation non-deterministic [4, 8]. such proposals arise as an alternative to simpler text generation methods (also called greedy decoding), arguing that choosing most probable words (or tokens) is one of the main causes of repetitive texts. another example of a study aimed at mitigating repetitive texts is the work from su et al.  [28]. proposed in 2022, the contrastive-search consists of a modification in the choosing words (or tokens) predicted by a textual generator, which aims to increase the variability of the text while maintaining its coherence. for this purpose, the authors suggest penalising, during decoding or unsupervised training of the language model, the softmax scores of the most likely tokens by their similarity to other tokens within the context. the importance given to the similarity is controlled by a parameter alpha. at last, we will present examples of studies employing transformers to generate text in the legal domain. keyphrases such as the ones used in this work are exclusive to brazil and, for the best of our knowledge, this is the first in depth study of decoding methods for brazillian keyphrase generation. feijo and moreira [5] and yoon et al. [32] applied transformer models to summarise rulings from the brazilian supreme court and korean legal cases, respectively. peric et al.  [16] proposed the use of transformers to generate opinions about legal cases originating in the u.s circuit court, by employing an encoder-decoder architecture. huang et al.  [9] proposed a solution to automate the legal judgment prediction (pjl) subtasks using the t5 text-to-text framework. at last, althammer et al.  [1] investigated the use of summaries (generated by transformer) as part of an information retrieval pipeline for the legal domain as part of the 2021 competition on legal information extraction/entailment (coliee). 3 methodology the methodology used in this work is composed of: i) data collection and preprocessing, ii) keyphrase generator training, iii) decoding methods evaluation and iv) qualitative analysis. these components will be discussed below. 3.1 data collection and preprocessing in 2022, the brazilian supremo tribunal de justiça (stj) supreme court of justice made available the dados abertosfootnote 1 platform. the platform consists of a public website, sharing legal decisions from various courts in brazil. the published documents comprise a large variety of topics in brazil’s legal domain, such as crimes in general, commerce, taxes, etc. we collected a total of 712,161 documents from the platform in august 2022. after the data collection, we extracted the dockets from the documents metadata and preprocessed the text of the decisions. we removed duplicated examples and removed urls from the text. 111,964 dockets remained after the preprocessing described. with the remaining examples, we extracted the keyphrases and enumerated paragraphs from the dockets, identifying and extracting capitalised sentences present in the header of the collected decisions. by extracting the inputs (enumerated paragraphs) and expected outputs (keyphrases), the original keyphrases (written by specialists) compose the reference set used for supervised training and evaluation. as a final preprocessing step, we divided the corpus (111,964 examples) into training (70%), validation (10%), and test (20%) splits. from the examples of the training set, we observed that enumerated paragraphs and keyphrases have a mean of 203.26 and 55.84 space-separated tokens, respectively. we used the splits to train and evaluate a supervised deep learning text generator. 3.2 keyphrase generation in this section is described the methodology employed for keyphrase training and generation. transformers for text generation. based on the dockets collected, we noted that most of the terms in the keyphrases are not directly present in the dockets. by further analysing examples from the validation set, we noted that only \(\sim \)10% of the terms present in the keyphrases are in fact present in the input text. thus, we decided to approach writing keyphrases as generation rather than extraction of text. for this purpose, a sequence-to-sequence (or text-to-text) transformer model was chosen. we choose ptt5 [3] as our keyphrase generator. ptt5 was pretrained in a large brazilian portuguese corpus (brwac [30]) with 2.7 billion tokens and the base version of the model (220m parameters) was used in our experiments. we experimented with other state-of-the-art multilingual generative transformer models (such as mt5 [31], bloom [25] and opt [33]), but the portuguese model (ptt5) performed better. previous works [3, 22, 27] observed that models pretrained for the task language tend to outperform multilingual models on the same tasks, and the same trend was observed in our experiments. fig. 2. train and validation losses obtained for the ptt5 model are shown in the left y-axis. the plot also shows the validation bleu scores in the right y-axis. full size image training details. we fixed the input (enumerated paragraphs) and output (keyphrases) sizes to 512 and 256 sentence-piece tokens, respectively. we padded shorter sequences of tokens and truncated longer sequences (to the maximum length). we fine-tuned the method ptt5 using a fixed learning rate of \(1\times 10^{-3}\), batch size equal to 256 and 20 maximum training epochs. for the sequence-to-sequence training, the cross-entropy loss function was adopted. the bleu score [17] metric was considered to evaluate the text generation quality. we used early-stopping during training, monitoring the bleu metric in the validation set. the training process is stopped after two epochs without improving the bleu score. for evaluation, we repeated the training process with five different seeds (1000, 2000, 3000, 4000 and 5000) and obtained a \(37.254 \pm 0.783\) bleu score. the best performing model achieved 38.607 bleu on the test set. the fine-tuning was done using the huggingfacefootnote 2 library, and a tesla p100 gpu with 16 gb vram. figure 2 shows the train and validation losses for the best execution of the ptt5 model in addition to the validation bleu scores. the model was trained for 17 epochs, totaling 5219 iterations. 3.3 decoding methods evaluation for the evaluation of the generated text and to compare the decoding methods, we have concatenated the generated keyphrases to their original document and used a real use case of retrieval to extract ir metrics. we opted to use an ir task to evaluate the generated keyphrases (created using different decoding methods) to evaluate them in their intended use: improving retrieval tasks. the details of the evaluations will be presented to follow. decoding methods. decoding techniques are used to guide neural text generation, in order to generate meaningful and coherent text. the methods are used to generate human-readable text from the internal representations of language models. in this work, we evaluate three decoding methods: greedy, top-k [4] and top-p [8]. top-k and top-p are sampling decoding methods, that, during text generation, sample tokens from finite sets. a brief description of the methods will be presented next. greedy: greedy decoding always selects the most probable token (highest softmax score) during generation. top-k: consists of filtering the most probable k tokens at a given instant, and redistributing their probabilities among them before sampling. top-p: limits the set of selectable tokens to a set of more probable tokens whose summed probabilities are lower than the established threshold p. note that the number of tokens that can be chosen is dynamic, since the probability distributions vary at each instant. task formulation. we have used the themes (categorical information), present in the dockets’ metadata, to simulate a retrieval task in which a specialist seeks to retrieve documents similar to a query document using a search engine. the themes are unique identifiers, that are mapped to common questions of law. thus, by using the binary relevance definition: given a query document q, the relevant documents r to q must have the same theme as q. note that, in the real scenario, the documents consists in dockets containing both keyphrases and enumerated paragraphs. from the collected decisions, only 801 have themes. these documents were removed from the training set and used to prepare query relevant document pairs for ir evaluation. the query set consists of dockets whose themes occur at least twice. from those, we prepared 482 query relevant document pairs (pairs of same theme documents). to prepare the final retrieval corpus, we combined the test set presented in sect. 3.1 with the dockets with themes and obtained a total of 23,194 documents. we increased the retrieval corpus to make the retrieval task more challenging. in the worst case, the documents without a theme may introduce false negatives (documents with the same theme of the query, but considered non-relevant), hindering the ir metrics. experimental setup. two different experiments were performed during the ir evaluation and they are described in the following. 1. studying sampling-based decoding methods: this experiment aims to investigate the generation of multiple keyphrases from a single docket using sampling decoding. by concatenating multiple keyphrases to a single docket, we expect to see improvements in the ir metrics since we are adding more text variations. we generated up to 10 keyphrases for each example in the search corpus, using top-k and top-p sampling, and concatenated them to the original input (enumerated paragraphs) to generate the ir metrics. we repeated the text generation five times with seeds of different random numbers (1,000, 2,000, 3,000, 4,000, and 5,000) and aggregated the results for comparisons. the effects of the k of the top-k, and the p of the top-p sampling methods were also evaluated in this experiment, varying the values of both k and p. we choose k and p from the following sets of values: \(k \in \{15, 50, 100\}\) and \(p \in \{0.1, 0.5, 0.9\}\). note that for this experiment, we are not interested in determining the best number of repetitions, nor the best value for k or p. the goal is to investigate the effect of the parameters on the proposed ir task, but the results may indicate the best parameter ranges. 2. decoding methods comparison: in order to compare the decoding methods, we extracted ir metrics for dockets using keyphrases generated using top-k and top-p sampling. we used the generated ones in place of the originals in this experiment. for reference, we also evaluated ir metrics considering documents with and without the original keyphrases (for both query and corpus documents) and using simple greedy decoding. we choose to use only one keyphrase generated by each method based on the results of the previous experiment and to evaluate the decoding methods in similar scenarios. for this experiment, we used the following parameters for the sampling-based decoding methods: \(k=15\) and \(p=0.9\) (based on the performances obtained in the previous experiment). the experiments with sampling decoding methods were inspired by the work doc2query [14]. in this work, for each example in a corpus, the authors generated several queries related to the example’s content using a sequence-to-sequence transformer model. the authors used top-k sampling in order to generate several queries per example. then the queries were concatenated to the input documents in order to improve ir metrics. considering both experiments with sampling methods, we used contrastive-search with \(alpha=0.6\), based on the original paper [28]. we choose the k and p values based on previous works with top-k and top-p sampling [8, 14]. information retrieval methods and metrics. to evaluate the proposed ir task, we choose two traditional methods: tf-idf and bm25 [21]. the methods were chosen due to their popularity in search engines (such as lucenefootnote 3) and competitive performance [18, 23]. previous works [12, 13, 15] also discussed that sparse representation methods (such as the chosen ones) tend to perform better in similar tasks in the legal domain. as an additional preprocessing for the ir methods, the documents were tokenized and portuguese stop-words and punctuation were removed. for tf-idf, we utilized a vocabulary size of 10,000 tokens (that appeared at least three times), and n-grams from 1 to 3. to sort documents during retrieval using tf-idf, we used the cosine similarity between queries and documents. considering bm25, the documents were sorted by the probability ranking principle, estimating the relevance of a document to the presented query. the additional preprocessing was done using spacyfootnote 4 and sklearnfootnote 5. for bm25, we used the implementation and default parameters from rank-bm25footnote 6. at last, we evaluated the performance in the proposed ir task using two traditional ir metrics: mean reciprocal rank (mrr) and recall. the metrics were chosen by their use in similar works in the legal domain [24, 26]. we used a threshold of 10 documents (top-10 ranked documents) to compute the metrics. according to russel et al. [24], law professionals tend to analyze, for the most part, up to 50 documents in their searches. therefore, we are evaluating an even more challenging scenario than the described by the authors. 3.4 qualitative analysis as a final analysis, for all decoding methods evaluated (greedy, top-k and top-p), we sampled examples generated using all methods and performed a qualitative analysis on them. for this analysis, we compared the generated keyphrases to the references and discussed the similarities between them and the effect of the sampling methods. 4 results and discussions these sections discuss the results obtained for each experiment described in sect. 3.3. in all experiments, we aim not to compare the retrieval methods (tf-idf and bm25), but to use them to evaluate the quality of the generated keyphrases using different decoding methods. 4.1 studying sampling-based decoding methods tables 1 and 2 present the ir metrics obtained, varying both the number of repetitions and k e p parameters. the metrics consist in the mean of five different executions (using five different seeds). table 1. top-k experiments evaluation metrics. n represents the number of samples included at the beginning of each docket.full size table table 2. top-p experiments evaluation metrics. n represents the number of samples included at the beginning of each docket.full size table when carrying out this experiment, the expectation was to observe a logarithmic growth as more different keyphrases were concatenated to the dockets (similar to doc2query [14]), since we are using more variations of keyphrases. however, this result was not observed in any of the evaluated metrics. contrary to expectations, in the worst cases, there was a decay in the metrics as new variations were added to the input texts for both top-k and top-p decoding methods. the decay is more noticeable for tf-idf method, with reductions between 2% (top-k) and 12% (top-p) in all observed metrics. the mentioned behaviors were observed for all evaluated k and p values. the worst performances were observed in increasing repetitions for \(p=0.1\) (top-p experiments). the most probable explanation is that the low p value is too restrictive, reducing the set of selectable tokens. hence, the top-p tends to generate similar keyphrases with low text variation (more similar or equal keyphrases). this way, the repetitive text hindered the performance of both ir methods evaluated. by increasing the k and p values, we increase the variability of the generated text, since the tokens to be predicted are chosen from a larger set. a positive effect on the metrics was also expected due to the possibility of adding more discriminative terms in the generated keyphrases, which is beneficial for the evaluated sparse methods. however, we observed deterioration of the performance and, at the best case, similar metrics by varying the k and p values. we suspect that even with the increase in variability, the generated keyphrases remained similar to each other, resulting in addition of repetitive texts to the dockets. the conclusion from these results is that there is no evidence that using more keyphrases (by using sampling decoding) is beneficial to the evaluated task. also, there is no benefit in using k values above 15, and p values lower than 0.9. we will discuss the results of the sampling methods further in sect. 4.3. 4.2 decoding methods comparison in table 3 is presented the results comparing decoding methods. for both tfidf and bm25, a single keyphrase using greedy was generated for both top-k and top-p decoding. we adopted \(k=15\) and \(p=0.9\) for the top-k and top-p decoding, respectively, based on the results from the previous experiment. table 3 also presents the results obtained performing the proposed retrieval task with and without the original (reference) keyphrases for comparisons. table 3. ir metrics obtained for each decoding method evaluated. superscript characters denote a pairwise statistically significant difference, according to a paired t-test (p-value \(< 0.05\)).full size table we observe that the keyphrases are, indeed, beneficial to retrieval tasks by comparing the metrics obtained by using documents with and without the keyphrases. for both metrics, we observed statistically significant differences. since both sparse methods (tf-idf and bm25) benefit from the existence of discriminative terms in the documents, these results were already expected. note that the metrics obtained using the original keyphrases act as an upper bound to our experiments. considering the tf-idf retrieval, we observed an increment in all metrics by using the generated texts (compared to not using any). the differences in all metrics are statistically significant (see table 3a). for the bm25 method, we observe similar results (see table 3b). however, no significant differences were observed when considering the r@10 metric. note that by using generated keyphrases, there is the possibility of introducing false positives (false similar) and false negatives (false non-similar) in the search corpus, originated by noisy keyphrases. the ir metrics obtained by the bm25 method suggest that the method was sensitive to these noisy examples. by comparing the decoding methods, we note small increments for the sampling methods in relation to greedy decoding. however, considering a paired t-test using a threshold of 5%, there is no significant difference between the decoding methods. thus, there is not enough evidence to reject the null hypothesis (metrics have the same mean) by observing the comparisons between the metrics of all three decoding approaches. therefore, there is no evidence that justifies choosing to sample decoding methods over a simpler greedy decoding approach, considering the proposed task. 4.3 qualitative analysis fig. 3. examples generated using greedy decoding. full size image greedy decoding. examples of keyphrases generated by ptt5, using greedy decoding, are presented in fig. 3. we can note that with bleu scores close to 40%, although being generated by a model trained in a modest training set (less than 100k examples), the generated keyphrases do not present spelling and lexical errors. they captured the writing style of the reference keyphrases and are very similar to the keyphrases written by humans. fig. 4. histogram comparing the number of space-separated tokens between the reference keyphrases and the ones generated using greedy decoding. full size image a comparison between the number of tokens of the original and the generated keyphrases using greedy decoding is shown in fig. 4. it is possible to observe that, although the distributions presented by the two histograms are similar, the generated keyphrases have a higher concentration of examples below 60 tokens. the average of space-separated tokens of the generated keyphrases is lower than the average of the tokens presented by the references (42.34 compared to 48.28). therefore, we identified that the keyphrases generated with greedy decoding tend to have a smaller length (in tokens) than the originals. we also observed the same pattern for the keyphrases generated using sampling decoding. sampling decoding. in fig. 5 is shown keyphrase examples generated using top-k and top-p decoding. we used \(k=15\) and \(p=0.9\) based on the results from the previous analysis. from the examples, it is possible to note that the main effect of using sampling is the generation of paraphrases of the original keyphrase. we also observe examples of reordering of the phrases present in the keyphrases. hence, the generated keyphrases tend to be similar to each other. the described behaviours are justified by the working of language models based on transformers since, during text generation, they tend to generate tokens that appear in similar contexts. by using sampling-based methods, we observed an increase in text variability. however, the possibility of the model generating text not related to the input also increases, which may have harmed the ir methods studied. in addition, when concatenating multiple variations of keyphrases similar to each other, we added many repeated terms to the documents, which may influence negatively the sparse ir methods evaluated. in addition to the justifications presented, the amount of training data may also have affected the sampling methods. although the results for greedy generation were better, the lack of variability in the training examples (due to their small size), may have harmed the decoding using top-k and top-p sampling. fig. 5. examples of keyphrases generated by top-k (\(k=15\)) and top-p (\(p=0.9\)) decoding. full size image 5 conclusion and future works in this paper, we successfully trained a sequence-to-sequence transformer to generate keyphrases and investigated three different text decoding methods. the results showed us that the keyphrases bring significant increments to ir metrics when used in combination with the dockets. this result was observed for all the keyphrases evaluated: the references and the generated ones (using greedy, top-k, and top-p decoding). although we have evaluated different parameters and concatenated multiple variations of keyphrases generated using sampling decoding (top-k and top-p), the simpler greedy decoding performed similarly to these methods. we presented and discussed possible justifications for such behaviour, and the results suggest that greedy decoding is enough for keyphrase generation considering legal dockets. as future works, we intend to experiment pre-training language models on legal documents in order to improve keyphrase generation. furthermore, we aim to improve the quality of the training by collecting more dockets from more sources around brazil. at last, this work can also be used to inspire other works aiming to automatise text writing in the legal domain. notes 1.https://dadosabertos.web.stj.jus.br/. 2.huggingface.co/. 3.https://lucene.apache.org/. 4.https://spacy.io/. 5.https://scikit-learn.org. 6.https://pypi.org/project/rank-bm25/. references althammer, s., askari, a., verberne, s., hanbury, a.: dossier@ coliee 2021: leveraging dense retrieval and summarization-based re-ranking for case law retrieval. arxiv preprint arxiv:2108.03937 (2021) brown, t., et al.: language models are few-shot learners. adv. neural. inf. process. syst. 33, 1877–1901 (2020) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data. arxiv preprint arxiv:2008.09144 (2020) fan, a., lewis, m., dauphin, y.n.: hierarchical neural story generation. corr abs/1805.04833 (2018). http://arxiv.org/abs/1805.04833 feijo, d., moreira, v.: summarizing legal rulings: comparative experiments. in: proceedings of the international conference on recent advances in natural language processing (ranlp 2019), pp. 313–322 (2019) google scholar  floridi, l., chiriatti, m.: gpt-3: its nature, scope, limits, and consequences. mind. mach. 30(4), 681–694 (2020) article  google scholar  guimarães, j.a.c., santos, j.c.g.: a ementa jurisprudencial como resumo informativo em um domínio especializado: aspectos estruturais. braz. j. inf. sci. 10(3), 32–43 (2016) google scholar  holtzman, a., buys, j., forbes, m., choi, y.: the curious case of neural text degeneration. corr abs/1904.09751 (2019). http://arxiv.org/abs/1904.09751 huang, y., shen, x., li, c., ge, j., luo, b.: dependency learning for legal judgment prediction with a unified text-to-text transformer. arxiv preprint arxiv:2112.06370 (2021) ji, z., et al.: survey of hallucination in natural language generation. acm comput. surv. 55(12), 1–38 (2022) article  google scholar  de justiça cnj, c.n.: conselho nacional de justiça justiça em números (2023). https://www.cnj.jus.br/pesquisas-judiciarias/justica-em-numeros/. accessed 08 may 2023 lima, j.p., costa, j.a., araújo, d.c.: comparison of feature extraction methods for brazilian legal documents clustering. in: 2021 ieee latin american conference on computational intelligence (la-cci), pp. 1–5. ieee (2021) google scholar  mandal, a., ghosh, k., ghosh, s., mandal, s.: unsupervised approaches for measuring textual similarity between legal court case reports. artif. intell. law 29(3), 417–451 (2021). https://doi.org/10.1007/s10506-020-09280-2 article  google scholar  nogueira, r., yang, w., lin, j., cho, k.: document expansion by query prediction. arxiv preprint arxiv:1904.08375 (2019) pedroso, d.d.s.c., ladeira, m., de paulo faleiros, t.: does semantic search performs better than lexical search in the task of assisting legal opinion writing? in: 2019 18th ieee international conference on machine learning and applications (icmla), pp. 680–685. ieee (2019) google scholar  peric, l., mijic, s., stammbach, d., ash, e.: legal language modeling with transformers. in: proceedings of the fourth workshop on automated semantic analysis of information in legal text (asail 2020) held online in conjunction with 33rd international conference on legal knowledge and information systems (jurix 2020) 9 december 2020, vol. 2764. ceur-ws (2020) google scholar  post, m.: a call for clarity in reporting bleu scores. arxiv preprint arxiv:1804.08771 (2018) pradeep, r., et al.: h2oloo at trec 2020: when all you got is a hammer... deep learning, health misinformation, and precision medicine. corpus 5(d3), d2 (2020) google scholar  radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i., et al.: language models are unsupervised multitask learners. openai blog 1(8), 9 (2019) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21(140), 1–67 (2020) mathscinet  google scholar  robertson, s.e., walker, s.: okapi/keenbow at trec-8. in: trec, vol. 8, pp. 151–162. citeseer (1999) google scholar  rosa, g.m., bonifacio, l.h., de souza, l.r., lotufo, r., nogueira, r.: a cost-benefit analysis of cross-lingual transfer methods. arxiv preprint arxiv:2105.06813 (2021) rosa, g.m., rodrigues, r.c., lotufo, r., nogueira, r.: yes, bm25 is a strong baseline for legal case retrieval. arxiv preprint arxiv:2105.05686 (2021) russell-rose, t., chamberlain, j., azzopardi, l.: information retrieval in the workplace: a comparison of professional search practices. inf. process. manag. 54(6), 1042–1057 (2018) article  google scholar  scao, t.l., et al.: bloom: a 176b-parameter open-access multilingual language model. arxiv preprint arxiv:2211.05100 (2022) souza, e., et al.: assessing the impact of stemming algorithms applied to brazilian legislative documents retrieval. in: anais do xiii simpósio brasileiro de tecnologia da informação e da linguagem humana, pp. 227–236. sbc (2021) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  su, y., lan, t., wang, y., yogatama, d., kong, l., collier, n.: a contrastive framework for neural text generation. arxiv preprint arxiv:2202.06417 (2022) vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, vol. 30 (2017) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation (lrec 2018) (2018) google scholar  xue, l., et al.: mt5: a massively multilingual pre-trained text-to-text transformer. arxiv preprint arxiv:2010.11934 (2020) yoon, j., junaid, m., ali, s., lee, j.: abstractive summarization of korean legal cases using pre-trained language models. in: 2022 16th international conference on ubiquitous information management and communication (imcom), pp. 1–7. ieee (2022) google scholar  zhang, s., et al.: opt: open pre-trained transformer language models. arxiv preprint arxiv:2205.01068 (2022) download references acknowledgement this study was supported by the coordenação de aperfeiçoamento de pessoal de nível superior brazil (capes) finance code 001. we thank cemeai for granting access to the euler cluster for the experiments. also, this work is partially funded by fundação de amparo à pesquisa do estado de são paulo (fapesp), grant 2022/01640-2. we would like also to thank inct (capes concessão 88887.136349/2017-00, cnpq 465755/2014-3 and fapesp 2014/50851-0) for the support. author information authors and affiliations universidade de são paulo icmc, são paulo, brazil kenzo sakiyama, raphael montanari, roseval malaquias junior & roseli a. f. romero universidade de campinas neuralmind, campinas, brazil rodrigo nogueira authors kenzo sakiyamaview author publications search author on:pubmed google scholar raphael montanariview author publications search author on:pubmed google scholar roseval malaquias juniorview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar roseli a. f. romeroview author publications search author on:pubmed google scholar corresponding author correspondence to kenzo sakiyama . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sakiyama, k., montanari, r., malaquias junior, r., nogueira, r., romero, r.a.f. (2023). exploring text decoding methods for portuguese legal text generation. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_5 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords text generation information retrieval legal texts publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature speech2phone: a novel and efficient method for training speaker recognition models | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper speech2phone: a novel and efficient method for training speaker recognition models conference paper first online: 28 november 2021 pp 572–585 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2021) speech2phone: a novel and efficient method for training speaker recognition models download book pdf download book epub edresson casanova  orcid: orcid.org/0000-0003-0160-717310, arnaldo candido junior  orcid: orcid.org/0000-0002-5647-089111, christopher shulby  orcid: orcid.org/0000-0001-9637-965712, frederico santos de oliveira  orcid: orcid.org/0000-0002-5885-674713, lucas rafael stefanel gris  orcid: orcid.org/0000-0002-2099-500411, hamilton pereira da silva  orcid: orcid.org/0000-0001-8709-876511, sandra maria aluísio  orcid: orcid.org/0000-0001-5108-263010 & … moacir antonelli ponti  orcid: orcid.org/0000-0003-2059-946310  show authors part of the book series: lecture notes in computer science ((lnai,volume 13074)) included in the following conference series: brazilian conference on intelligent systems 1234 accesses 1 citation abstract in this paper we present an efficient method for training models for speaker recognition using small or under-resourced datasets. this method requires less data than other sota (state-of-the-art) methods, e.g. the angular prototypical and ge2e loss functions, while achieving similar results to those methods. this is done using the knowledge of the reconstruction of a phoneme in the speaker’s voice. for this purpose, a new dataset was built, composed of 40 male speakers, who read sentences in portuguese, totaling approximately 3h. we compare the three best architectures trained using our method to select the best one, which is the one with a shallow architecture. then, we compared this model with the sota method for the speaker recognition task: the fast resnet–34 trained with approximately 2,000 h, using the loss functions angular prototypical and ge2e. three experiments were carried out with datasets in different languages. among these three experiments, our model achieved the second best result in two experiments and the best result in one of them. this highlights the importance of our method, which proved to be a great competitor to sota speaker recognition models, with 500x less data and a simpler approach. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi speaker text-to-speech synthesis using generalized end-to-end loss function article 13 january 2024 the influence of errors in phonetic annotations on performance of speech recognition system chapter © 2018 robust recognition of conversational telephone speech via multi-condition training and data augmentation chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. lisp signal, speech and image processing speech and language therapy speech and audio processing speech and audio signal processing speech perception 1 introduction voice recognition is widely used in many applications, such as intelligent personal assistants [14], telephone-banking systems [4], automatic question response [12], among others. in several of these applications, it is useful to identify the speaker, as is the case in voice-enabled authentication and meeting loggers. speaker verification can be done in two scenarios: open-set and closed-set. in both scenarios, the objective is to verify that two audio samples belong to the same speaker. however, in the closed-set scenario, the verification is restricted only to speakers seen during the training of the models. on the other hand, in the open-set scenario, verification occurs with speakers not seen in model training [11, 17]. the verification of speakers in an open-set scenario is especially desired in applications such as meeting loggers, since in these applications speakers can be added frequently, thus, the use of closed-set models would imply the retraining of the model after the insertion of new speakers. the first works to use deep neural networks in speaker recognition in an open-set scenario learned speaker embeddings were [30, 31], using the softmax function. although the softmax classifier can learn different embeddings for different speakers, it is not discriminatory enough [8]. to work around this problem, models trained with softmax were combined with back-ends built on probabilistic linear discriminant analysis (plda) [16] to generate scoring functions [27, 31]. on the other hand, softmax angular [19] was proposed and it uses cosine similarity as a logit entry for the softmax layer, and it proved to be superior to the use of softmax only. thereafter, additive margins in softmax (am-softmax) [35] was proposed to increase inter-class variance by introducing a cosine margin penalty in the target logit. however, according to [8] training with am-softmax and aam-softmax [10] proved to be a challenge, as they are sensitive to scale and margin value in the loss function. the use of contrastive loss [7] and triplet loss [5, 28] also achieved promising results in speaker recognition, but these methods require careful choice of pairs or triplets, which costs time and can interfere with performance [8]. finally, the use of prototypical networks [36] for speaker recognition was proposed. prototypical networks seek to learn a metric space in which the classification of open-sets of speakers can be performed by calculating distances to prototypical representations of each class. generalized end-to-end loss (ge2e) [34] and angular prototypical (angleproto) [8] follow the same principle and achieved state-of-the-art (sota) results recently in speaker recognition. parallel with this work, [15] proposed the use of the angleproto loss function in conjunction with softmax, presenting a result superior to the use of angleproto only. the authors proposed a new architecture presenting sota results. in this work, we propose a new method for training speaker recognition models, called speech2phone. this method was trained with approximately 3.5 h of speech and surpassed a model trained with 2.000 h of speech using the ge2e loss function. our method is based on the reconstruction of the pronunciation of a specific phoneme and has shown promise in scenarios with few available resources. in addition, the simplicity of its architecture makes our method suitable for real-time applications with low processing power. finally, to simplify the reproduction of this work, python code and download links for the datasets used to reproduce all experiments are publicly available on the github repositoryfootnote 1. this work is organized as follows. section 2 details the datasets used as well as the preprocessing performed to attend the proposed experiments. section 3 presents the speech2phone method and experiments carried out to find the best model trained with this method for the identification of speakers in an open-set scenario. section 4 compares the best model trained with the speech2phone method with the state of the art in the literature. finally, sect. 5 shows the conclusions of this paper. 2 datasets and pre-processing section 2.1 presents the datasets used, as well as describes a new dataset created to attend the needs of our experiments. sections 2.2 and 2.3 detail the pre-processing performed on the datasets to allow the execution of all the proposed experiments. 2.1 audio datasets to train our method, it was necessary to build a specific dataset, which we call the speech2phone dataset. this dataset includes 40 male speakers, aged between 20 and 50 years. the dataset includes only portuguese utterances, because that is the native language of the speakers. we chose to focus only on male speakers, because during the collection phase of our dataset we were able to collect only voices from 5 female speakers. to collect the data, each speaker was given a phonetically balanced text, according to the work of [29], which was comprised of 149 words. the reading time ranged from 42 to 95 s, totaling approximately 43 min of speech. additionally, we asked each speaker to say the phoneme /a/ for approximately three seconds. the central second of each capture was extracted and then used as expected output for the embedding models. the phoneme /a/ was chosen because it is simple to articulate and very frequent in the portuguese language. the speech2phone dataset is publicly available on the github repositoryfootnote 2 to evaluate our method and compare it with related works, we use the vctk [33] and common voice (cv) [2] datasets. the vctk is an english language dataset with a total of 109 speakers. during its creation, the 109 speakers spoke approximately 400 sentences. the same phrases are spoken by all speakers. the dataset has approximately 44 h of speech and is sampled at 48 khz. common voice is a massively multilingual transcribed speech dataset for research and development of speech technology. cv has 54 subsets and each of these sets have data from a language, currently the dataset has a total of 5,671 h. in this work, we use version 5.2 of the corpus. 2.2 preprocessing of the speech2phone dataset to preprocess the speech2phone dataset we extracted five-second speech segments from the original audio length. the five-second window was defined after preliminary experiments, varying the input duration. in order to maximize the number of speech segments, we used the overlapping technique, in which the window was shifted one second each time and an instance was extracted during the total audio duration. the main dataset resulted in 2,394 speech segments totaling 3 h and 23 min of speech. the next step was to divide the dataset into smaller sets to attend the needs of each proposed experiment. therefore, the original dataset was divided into four subsets. the partitions \(a_1\) and \(b_1\) each have 20 different speakers and have approximately 1,097 samples each. partitions \(a_2\) and \(b_2\) have approximately 100 speech segments each and have, respectively, speakers from the \(a_1\) and \(b_1\) partition. thus, a partitions do not have speakers in common with b partitions. 2.3 pre-processing of speaker verification datasets we preprocessed the vctk and cv datasets in order to use them for speaker verification. for the vctk we chose to use the entire dataset and for cv we used the test subsets of the dataset in portuguese (pt), spanish (es) and chinese spoken in china (zh). the vctk dataset has, in many of its samples, long initial and final silences. in order to ensure that this feature does not affect our analysis, we chose to remove these silences. so, we applied voice activity detection (vad) using the python implementation of the webrtcvad toolkitfootnote 3. we used vctk and cv to test our models. the datasets were used to build audio pairs. the positive class is composed from audio pairs from the same speaker, while negative class has pairs from different speakers. in this scenario, it is possible to build more examples from the negative class. to avoid class imbalance issues, we defined the maximum number of negative pairs analyzed as the number of positive samples divided by the number of speakers. we also removed speakers with less than two samples. table 1 shows the number of speakers, language and number of positive and negative speech segments of the datasets used to verify the speaker in our experiments. the python code used for the preprocessing of the dataset, as well as the link to download the versions of the vctk and cv datasets used are available in the github repositoryfootnote 4. table 1. preprocessed speaker verification datasetsfull size table 3 speech2phone section 3.1 presents the experiments carried out to choose the best model trained with the speech2phone method to identify speakers in the open-set scenario. on the other hand, sect. 3.2 presents the results of speech2phone experiments. 3.1 proposed method the goal of open-set models is to be speaker independent; additionally, a desirable feature is to be multilingual and text independent. in pursuit of these goals, we propose that the neural network training uses five second speech fragments as input and, as expected output, the reconstruction of a second of a simple phoneme (/a/ in our experiments). as the phoneme sounds differ according to each speaker, a good reconstruction would allow the model to distinguish between speakers. focusing on a single phoneme allows for dimensionality reduction in the embedding layer. in the speech2phone experiments, the model’s input and the expected output is represented in the form of mel frequency cepstral coefficients (mfccs) [20]. we extract mfccs using the librosa [21] library. the default sampling rate (22 khz) was used and 13 mfccs (empirically defined) were extracted. windowed frames were used as defined by the default parameters in librosa 0.6, namely, a 512 hop length and 2,048 as the window length for the fast fourier transform [25]. in addition, as the models must reconstruct an mfcc segment they were induced using mean square error (mse) [13]. several models using the speech2phone method, and consequently the speech2phone dataset, have been tested. we report the three most interesting ones in this section. to evaluate these experiments, we used the accuracy of the speaker identification. to calculate the accuracy, we randomly chose and entered an embedding of a sample of each of the speakers in a database. for the calculation it is necessary to search the extracted embedding in an embeddings database. this is done by running the knn [22] algorithm with \(k=1\) and using the euclidean distance. if the embedding of a speaker has a euclidean distance closer to its embedding registered in the database than the embedding of all other speakers, this will be counted as a hit, otherwise it will be counted as an error. in the experiments, we compared the performance of the open-set models in a closed-set scenario. all of these experiments were trained with the speech2phone dataset \(a_1\) subset, described previously in sect. 2.2. therefore, the models were trained with only 1,037 speech segments of 20 different speakers, approximately 1.5 h of speech. tensorflow [1] and tflearn [32] were used to generate the neural networks for all speech2phone experiments. all models were trained using the adam optimizer [18]. the convolutional layers in all experiments have a stride of 1. hyperparameters used to train each model are presented in table 2. table 2. experiments hyperparametersfull size table to try to find the best topology for the speech2phone method, we propose the following experiments: experiment 1 (dense): this experiment consists of a fully connected feed-forward neural network with one hidden layer for embedding extraction. the architecture of the model used in this experiment is shown in the fig. 1. experiment 2 (conv): this experiment is based on experiment 1, but it uses a fully convolutional neural network. this network has only convolutional layers, in addition the decoder uses upsampling layer. following [8] we use 2d convolutions. the speech segments can be seen as a bidimensional image matrix where columns are time steps and the rows are cepstral coefficients. convolutions have the advantage of being translationally invariant in this matrix. as we try to reconstruct a specific phoneme, this is a desired property, since the location of the instance may contain the target phoneme may occur in different parts of the window. the model architecture used in this experiment is shown in the fig. 2. experiment 3 (recurrent): this experiment is based on experiment 1, but it consists of a recurrent fully-connected neural network for embedding generation. the five-second window is split into five segments containing one second each, in which the model analyzes one at a time. considering that the recurrence window is small, we used classical recurrence instead of long term models like lstm [13], since vanishing gradients are less prone to occur. if the phoneme of interest happens in one of the five fragments, the recurrence network should store it in its memory before reconstructing it in the final step, potentially improving the reconstruction accuracy. the approach reduces the number of learned parameters and, consequently, also improves training times. the model architecture used in this experiment is shown in the fig. 3. fig. 1. fully connected shallow neural network (experiment 1) full size image fig. 2. fully convolutional neural network (experiment 2) full size image fig. 3. recurrent neural network (experiment 3) full size image 3.2 speech2phone results table 3 shows the results of the speech2phone experiments, showing accuracy obtained in the open-set scenario, that is, evaluations using different speakers for training and testing the models and closed-set, where the training and test speakers are the same. in this set of experiments, the neural models received 5 s of audio from a specific speaker and were induced to reconstruct 1 s of the phoneme /a/ in the voice of that speaker. the goal is to obtain good results with little training data in contrast to the ge2e and angleproto loss functions, which need a large data set for good performance. to conduct the proposed experiments, samples from 20 speakers were used for training and samples from another 20 speakers for testing, as previously discussed in sect. 3.1. experiment 1 used a fully connected neural network. in an open-set scenario, it obtained an accuracy of 76.96%, the best accuracy in the open-set scenario for all experiments. on the other hand, in the closed-set scenario, it obtained 77.50% accuracy. this was the worst result for all closed-set experiments. we believe that the fully connected model achieved the best results on the open-set due to the low number of parameters, thus being less prone to overfit. in addition, the dataset is very small and in this way, deeper models are very likely to memorize features dependent on the speaker or noise artifacts. apparently, the recurrent and convolutional models specialized in extracting particular features for the speakers in the training set in order to reconstruct the output. in this way, as the deeper models learn specific characteristics of the speakers, their generalization for new speakers is impaired, having good performance in the closed-set scenario and a drop in performance in the open-set. table 3. results of speech2phone experimentsfull size table experiment 2, which explored the use of a fully convolutional neural network for generating embeddings, presented the second best result (64.43%) in the open-set scenario. on the other hand, in the closed-set, the model achieved the best result obtaining an accuracy of 100%. translational invariance is a useful feature from convolutional networks and can be used to detect specific phonemes independently of where their occur in the audio. we believe that convolutional models are suitable for this task; however, due to the low amount of data the models learn characteristics dependent on speakers, which leaves this model at a disadvantage in open-set scenario, having a worse performance than experiment 1. experiment 3 explored the use of a recurrent neural network with fully connected layers for generating embeddings, resulting in the worst accuracy in the open-set scenario (50.28%) and the second worst in the closed-set scenario (88.75%). recurrent models can perform a more detailed analysis on the input audio, searching patterns in one input fragment at a time. however, a problem may happen when this pattern is split in a different analysis window. the simple recurrent model tested could not overcome this issue. we also evaluated, in preliminary experiments, a recurrent lstm network, but it did not perform as well as simple recurrence, as there is no need for long-term memory in a 5-step analysis process. in the open-set scenario, the superiority of the fully connected model (experiment 1) is noticeable. this is because they are able to generalize better for new speakers and have proven to be less prone to overfitting for the task. in addition, the fully connected model was able to maintain very close accuracy in the closed-set scenario (77.50%), compared to the open-set scenario (76.96%), while the other models had a great drop in performance. on the other hand, the fully convolutional model (experiment 2) also showed promising results with a performance 12.53% below the fully connected model. 4 application: speaker verification section 4.1 presents the proposed experiments to compare our method with the state of the art in the literature. on the other hand, sect. 4.2 presents the results of speaker verification experiments. 4.1 speaker verification experiments an important question is how well a model trained with the speech2phone method performs in speaker verification. to answer this question we compared speech2phone with the fast resnet–34 model proposed by [8] trained using the angular prototypical [8] and ge2e [34] losses function. we chose this model because it presents state-of-the-art results in the voxceleb [23] dataset, in addition the authors made the pre-trained models available in the github repositoryfootnote 5. to compare the models, we chose two datasets, one of which is multi-language, presented in sect. 2.3. as speech2phone receives 22 khz sample rate audio, while fast resnet–34 16 khz audio input. we cannot directly compare the results of the models in the voxceleb test dataset. as voxceleb is sampled at 16 khz, it would not be a fair comparison to resample audios from 16 khz to 22 khz. therefore, we chose other datasets with a sample rate greater than 22 khz. in addition, using the dataset test that the fast resnet–34 model was trained on could put the speech2phone model at a disadvantage. the audios for each dataset were resampled to 16 khz for the test with the fast resnet–34 model and to 22 khz for the test with speech2phone, thus making a fair comparison between the models. for comparison, we use the metric equal error rate (eer) [6], the lower the eer is, the greater the accuracy becomes. to compare the best model trained with the speech2phone method with the sota we propose the following experiments: experiment 4 (speech2phone): this experiment uses the same model and hyperparameters as experiment 1; however, the model was trained with the entire speech2phone dataset. therefore, the model was trained with 2,394 speech segments of 40 speakers totaling approximately 3 h and 23 min of speech. experiment 5 (fast angleproto): this experiment uses the fast resnet–34 model, proposed in [8], this model is trained with the angular prototypical loss function in the voxceleb dataset, which has approximately 2,000 h of speech by approximately 7,000 speakers. this model achieves an eer of approximately 2.2 in the voxceleb dataset test set, as reported in [8]. experiment 6 (fast ge2e): this experiment also uses the fast resnet–34 model, and was also trained with the same number of speakers and hours of experiment 5. however the model is trained using the ge2e loss function. for the evaluation of the fast resnet–34 model we followed the original work and sample ten 4-s temporal crops at regular intervals from each test segment. as the model accepts audios of variable size when the audio was less than 4-s, like the original work, we only used the audio without sample crops. unlike the fast resnet–34 model, which accepts audios of varying sizes, our model receives an mfcc of just 5 s of speech as input. however, in the vctk and cv datasets we have audios with variable sizes. therefore, for a fair comparison and each model to have access to all the audio content for calculating the eer of the speech2phone model, we proceed as follows. for audios longer than 5 s, we use the overlapping technique, as described in sect. 2.2. therefore, after applying the overlapping technique for a six-second audio, two five-second samples are obtained, the resulting embedding for that sample is the average between the predicted embedding for these two five-second samples. on the other hand, for sample less than 5 s, we repeat the audio frames until reaching at least 5 s, for example, a three-second audio is repeated once, thus obtaining a six-second audio, where the resulting audio applies the overlapping technique, since we audios longer than 5 s. 4.2 speaker verification results in this set of experiments we compared the performance of our best model trained with the speech2phone method with the fast resnet–34 model proposed by [8], trained with approximately 2,000 h of speech using the angular prototypical loss function [8] and ge2e [34]. table 4 shows the eer of these experiments in english using the vctk dataset, in portuguese (pt), spanish (es) and chinese (zh) using the common voice dataset. table 4. results for speaker verification experimentsfull size table experiment 4 consisted of the best speech2phone result trained using the entire speech2phone dataset (approximately 3 h and 23 min of speech from 40 different speakers). this achieved the best eer of all experiments in the vctk dataset. for the common voice dataset, the model achieved the second best result for the pt and zh subsets, being surpassed only by the fast resnet–34 model trained with the angular prototypical loss function (experiment 5). on the other hand, in the es subset, this experiment had the worst performance of all experiments. experiment 5 used fast resnet–34 model trained with the angular prototypical loss function, obtaining the second best eer in the vctk dataset, only surpassed by speech2phone. in addition, this experiment achieved the best result in all subsets of the common voice dataset. experiment 6, which consisted of using fast resnet–34 model trained with the ge2e loss function, obtained the worst eer in the vctk dataset. this experiment also had the worst eer in the zh and pt subsets of common voice. however, the model achieved the second best eer in the es subset of common voice. experiment 4 showed the potential of the speech2phone approach, which despite having been trained with only 3 h and 23 min of speech from only 40 speakers, obtained better results than fast resnet–34 model trained with 2000 h of speech and approximately 7,000 speakers using the ge2e loss function. in addition, this experiment achieved better results than both experiments 5 and 6 in the vctk dataset. in experiment 5 we observed better results in 3 of the 4 evaluated subsets, when compared with experiment 4. this was expected due to the model in experiment 5 having access to more than 1.996 h of speech and 6.960 speakers. in addition, the experiment 4’s training dataset has only male voices, as the reconstruction of a female voice is different and the vctk and common voice datasets have female speakers this should cause a decrease of performance at test time. another point is that the training dataset used in experiment 4 is a high quality dataset with low background noise, so the model probably did not learn to ignore or filter out noise, probably harming the reconstruction of the /a/ phoneme. despite audio quality and low noise facilitate learning during training, the model may be impaired in a noisy situation which is a scenario observed in other domains such as images [24]. its better overall performance in the vctk dataset, also high quality and low noise, built for speech synthesis and voice transfer applications, is yet another evidence for this behavior. the performance of the model drops, compared to experiment 5, when the model is used in audios recorded in uncontrolled environments such as common voice. in experiments 5 and 6 we verified what has already been shown in [8], that the fast resnet–34 model trained with the angleproto loss function in the voxceleb dataset obtains an eer higher than the same model trained with the ge2e loss function. however, the authors in their work compared the models only in the voxceleb test dataset, we on the other hand, made a comparison using 4 different languages and different datasets, thus making a broader comparison. an important consideration is that given the way we propose our experiments, we cannot say that language is a factor that decreases or increases the performance of the models. the high values of eer in the english language, for example, are due to the way the speaker verification dataset was set up. the vctk has only 109 speakers and many samples were considered for each speaker as can be seen in table 1. a greater number of test instances make the task more difficult and tend to increase the eer values. therefore, we can only compare the individual performance of each model in the datasets and we cannot discuss decrease or increase in performance with the language change. 5 conclusions and future work in this article, we proposed a novel training method for speaker recognition models, called speech2phone. to enable the training of this method, we also built a novel dataset. fully connected, fully convolutional and recurring models were explored. we observed that the fully connected models have a better performance in open-set scenarios, although they have had the worst performance for the closed-set scenario, while convolutional models have a better performance in the closed-set scenario, but they do not generalize well for the open-set scenario. the best model in our experiments was trained on 3 h and 23 min of speech and compared to two sota models in the voxceleb dataset, which were trained with approximately 2000 h of speech. the results obtained were comparable to sota even with an amount of data approximately 500 times smaller. this work contributes directly to the area of speaker recognition, presenting a promising method for training speaker recognition models. in addition, the model proposed here can be used in tasks such as speech synthesis [26], voice cloning [3] and multilingual speech conversion [37]. in these tasks, the speaker recognition system embeddings are used to represent the speaker. in addition, an advantage of speech2phone in relation to the models proposed in [8] is the speed of execution, as our best model is a fully connected shallow neural network, making it faster. this feature is very desirable for applications due to the need to run in real time. as our model demands a specific dataset format, we were not able to evaluate its training in a large dataset. we plan to address this issue in future works. for this we intend to increase the model’s training dataset as much as possible, and make it public. additionally, we intend to explore the use of multispeaker speech synthesis [9] and voice cloning [3] to generate a dataset with more speakers and a larger vocabulary. on the other hand, we intend to investigate the possibility of a hybrid method that uses a speech2phone technique and angular prototypical loss function, thus being able to learn from a larger amount of data and at the same time guide the model’s learning with the reconstruction of a phonemefootnote 6. evaluating the limits of generalization across different languages is also a matter of future investigation. in addition, we intend to explore the insertion of noise in the training dataset in order to make the model more robust to noisefootnote 7. notes 1.https://github.com/edresson/speech2phone. 2.https://github.com/edresson/speech2phone. 3.https://github.com/wiseman/py-webrtcvad. 4.https://github.com/edresson/speech2phone/tree/master/paper/eer-experiments. 5.https://github.com/clovaai/voxceleb_trainer. 6.https://cyberlabs.ai/. 7.https://www.pti.org.br/. references abadi, m., et al.: tensorflow: large-scale machine learning on heterogeneous distributed systems. arxiv preprint arxiv:1603.04467 (2016) ardila, r., et al.: common voice: a massively-multilingual speech corpus. in: proceedings of the 12th language resources and evaluation conference, pp. 4218–4222 (2020) google scholar  arik, s., chen, j., peng, k., ping, w., zhou, y.: neural voice cloning with a few samples. in: advances in neural information processing systems, pp. 10019–10029 (2018) google scholar  bowater, r.j., porter, l.l.: voice recognition of telephone conversations. us patent 6,278,772 (21 august 2001) google scholar  bredin, h.: tristounet: triplet loss for speaker turn embedding. in: 2017 ieee international conference on acoustics, speech and signal processing (icassp), pp. 5430–5434. ieee (2017) google scholar  cheng, j.m., wang, h.c.: a method of estimating the equal error rate for automatic speaker verification. in: 2004 international symposium on chinese spoken language processing, pp. 285–288. ieee (2004) google scholar  chopra, s., hadsell, r., lecun, y.: learning a similarity metric discriminatively, with application to face verification. in: 2005 ieee computer society conference on computer vision and pattern recognition, cvpr 2005, vol. 1, pp. 539–546. ieee (2005) google scholar  chung, j.s., et al.: in defence of metric learning for speaker recognition. in: proceedings of the interspeech 2020, pp. 2977–2981 (2020) google scholar  cooper, e., et al.: zero-shot multi-speaker text-to-speech with state-of-the-art neural speaker embeddings. in: 2020 ieee international conference on acoustics, speech and signal processing (icassp), icassp 2020, pp. 6184–6188. ieee (2020) google scholar  deng, j., guo, j., xue, n., zafeiriou, s.: arcface: additive angular margin loss for deep face recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4690–4699 (2019) google scholar  ertaş, f.: fundamentals of speaker recognition. pamukkale üniversitesi mühendislik bilimleri dergisi 6(2–3) (2011) google scholar  ferrucci, d., et al.: building watson: an overview of the deepqa project. ai mag. 31(3), 59–79 (2010) google scholar  goodfellow, i., bengio, y., courville, a.: deep learning. mit press (2016). http://www.deeplearningbook.org gruber, t.: siri, a virtual personal assistant-bringing intelligence to the interface (2009) google scholar  heo, h.s., lee, b.j., huh, j., chung, j.s.: clova baseline system for the voxceleb speaker recognition challenge 2020. arxiv preprint arxiv:2009.14153 (2020) ioffe, s.: probabilistic linear discriminant analysis. in: leonardis, a., bischof, h., pinz, a. (eds.) eccv 2006. lncs, vol. 3954, pp. 531–542. springer, heidelberg (2006). https://doi.org/10.1007/11744085_41 chapter  google scholar  kekre, h., kulkarni, v.: closed set and open set speaker identification using amplitude distribution of different transforms. in: 2013 international conference on advances in technology and engineering (icate), pp. 1–8. ieee (2013) google scholar  kingma, d.p., ba, j.: adam: a method for stochastic optimization. arxiv preprint arxiv:1412.6980 (2014) liu, w., wen, y., yu, z., li, m., raj, b., song, l.: sphereface: deep hypersphere embedding for face recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 212–220 (2017) google scholar  logan, b., et al.: mel frequency cepstral coefficients for music modeling. in: ismir, vol. 270, pp. 1–11 (2000) google scholar  mcfee, b., et al.: librosa: audio and music signal analysis in python. in: proceedings of the 14th python in science conference, pp. 18–25 (2015) google scholar  michalski, r.s., carbonell, j.g., mitchell, t.m.: machine learning: an artificial intelligence approach. springer science & business media (2013). https://doi.org/10.1007/978-3-662-12405-5 nagrani, a., chung, j.s., zisserman, a.: voxceleb: a large-scale speaker identification dataset. in: proceedings of the interspeech 2017, pp. 2616–2620 (2017) google scholar  nazaré, t.s., da costa, g.b.p., contato, w.a., ponti, m.: deep convolutional neural networks and noisy images. in: mendoza, m., velastín, s. (eds.) ciarp 2017. lncs, vol. 10657, pp. 416–424. springer, cham (2018). https://doi.org/10.1007/978-3-319-75193-1_50 chapter  google scholar  nussbaumer, h.j.: the fast fourier transform. in: fast fourier transform and convolution algorithms, vol. 2, pp. 80–111. springer, heidelberg (1981). https://doi.org/10.1007/978-3-662-00551-4_4 ping, w., et al.: deep voice 3: scaling text-to-speech with convolutional sequence learning. in: international conference on learning representations (2018) google scholar  ramoji, s., krishnan v, p., singh, p., ganapathy, s.: pairwise discriminative neural plda for speaker verification. arxiv preprint arxiv:2001.07034 (2020) schroff, f., kalenichenko, d., philbin, j.: facenet: a unified embedding for face recognition and clustering. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 815–823 (2015) google scholar  seara, i.: estudo estatístico dos fonemas do português brasileiro falado na capital de santa catarina para elaboração de frases foneticamente balanceadas. ph.d. thesis, dissertação de mestrado, universidade federal de santa catarina ... (1994) google scholar  snyder, d., garcia-romero, d., povey, d., khudanpur, s.: deep neural network embeddings for text-independent speaker verification. in: interspeech, pp. 999–1003 (2017) google scholar  snyder, d., garcia-romero, d., sell, g., povey, d., khudanpur, s.: x-vectors: robust dnn embeddings for speaker recognition. in: 2018 ieee international conference on acoustics, speech and signal processing (icassp), pp. 5329–5333. ieee (2018) google scholar  tang, y.: tf.learn: tensorflow’s high-level module for distributed machine learning. arxiv preprint arxiv:1612.04251 (2016) veaux, c., yamagishi, j., macdonald, k., et al.: superseded-cstr vctk corpus: english multi-speaker corpus for cstr voice cloning toolkit. university of edinburgh, the centre for speech technology research (cstr) (2016) google scholar  wan, l., wang, q., papir, a., moreno, i.l.: generalized end-to-end loss for speaker verification. in: 2018 ieee international conference on acoustics, speech and signal processing (icassp), pp. 4879–4883. ieee (2018) google scholar  wang, f., cheng, j., liu, w., liu, h.: additive margin softmax for face verification. ieee sig. process. lett. 25(7), 926–930 (2018) article  google scholar  wang, j., wang, k.c., law, m.t., rudzicz, f., brudno, m.: centroid-based deep metric learning for speaker recognition. in: 2019 ieee international conference on acoustics, speech and signal processing (icassp), icassp 2019, pp. 3652–3656. ieee (2019) google scholar  zhou, y., tian, x., xu, h., das, r.k., li, h.: cross-lingual voice conversion with bilingual phonetic posteriorgram and average modeling. in: 2019 ieee international conference on acoustics, speech and signal processing (icassp), icassp 2019, pp. 6790–6794. ieee (2019) google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior – brasil (capes) – finance code 001, as well as cnpq (national council of technological and scientific development) grant 304266/2020-5. also, we would like to thank the cyberlabs and itaipu technological park (parque tecnológico itaipu—pti) for financial support for this paper. we also gratefully acknowledge the support of nvidia corporation with the donation of the gpu used in part of the experiments presented in this research. author information authors and affiliations university of são paulo, são carlos, brazil edresson casanova, sandra maria aluísio & moacir antonelli ponti federal university of technology paraná, medianeira, brazil arnaldo candido junior, lucas rafael stefanel gris & hamilton pereira da silva definedcrowd corp, seattle, usa christopher shulby federal university of goiás, goiânia, brazil frederico santos de oliveira authors edresson casanovaview author publications search author on:pubmed google scholar arnaldo candido juniorview author publications search author on:pubmed google scholar christopher shulbyview author publications search author on:pubmed google scholar frederico santos de oliveiraview author publications search author on:pubmed google scholar lucas rafael stefanel grisview author publications search author on:pubmed google scholar hamilton pereira da silvaview author publications search author on:pubmed google scholar sandra maria aluísioview author publications search author on:pubmed google scholar moacir antonelli pontiview author publications search author on:pubmed google scholar corresponding author correspondence to edresson casanova . editor information editors and affiliations universidade federal de sergipe, são cristóvão, brazil andré britto universidade de são paulo, são paulo, brazil karina valdivia delgado rights and permissions reprints and permissions copyright information © 2021 springer nature switzerland ag about this paper cite this paper casanova, e. et al. (2021). speech2phone: a novel and efficient method for training speaker recognition models. in: britto, a., valdivia delgado, k. (eds) intelligent systems. bracis 2021. lecture notes in computer science(), vol 13074. springer, cham. https://doi.org/10.1007/978-3-030-91699-2_39 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-030-91699-2_39 published: 28 november 2021 publisher name: springer, cham print isbn: 978-3-030-91698-5 online isbn: 978-3-030-91699-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords speaker verification speaker recognition speaker identification publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving llms’ reasoning and planning with finite-state machines | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving llms’ reasoning and planning with finite-state machines conference paper first online: 30 january 2025 pp 110–124 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) improving llms’ reasoning and planning with finite-state machines download book pdf download book epub eduardo faria cabrera9, marcel rodrigues de barros9 & anna helena reali costa9  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 435 accesses 1 citation abstract large language models (llms) have shown remarkable capabilities across various applications but struggle with sequential decision-making tasks like planning. this work demonstrates that integrating llms with finite-state machines (fsms) can enhance their reasoning and planning abilities, while also offering increased reliability. several setup variants are compared, providing a better understanding of how validations, feedback loops, and restrictions enhance robustness and effectiveness. a comparison to the well known chain-of-thoughts approach is also provided. our methods improve planning capabilities of all analysed llms, consistently increasing the success rate in solving tasks of varying complexities. a detailed analysis of the two best variants are provided, highlighting their respective strengths and weaknesses. the authors thank the centro de ciência de dados (c2d-poli-usp) supported by itaú unibanco s.a., as well as the center for artificial intelligence (c4ai-usp), supported by fapesp (grant 2019/07665-4) and ibm corporation. this work is partially supported by cnpq (grant number 312360/2023-1) and capes (finance code 001). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others agent can say no: robot task planning by natural language feedback between planner and executor chapter © 2024 applying large language model in survey extraction chapter © 2026 comparative analysis of methodologies and approaches in recommender systems utilizing large language models article open access 17 april 2025 1 introduction recently, large language models (llms) have surged in development, revolutionizing ai capabilities. scaling up these models has unlocked new feats, from crafting narratives and summarizing text to coding and enhancing reasoning skills [2, 3, 18]. despite these advancements, llms still face challenges in basic logical reasoning, often producing implausible claims and struggling with rudimentary planning tasks [2, 15]. moreover, the opacity of their decision-making processes poses significant obstacles in interpreting and validating their outputs, especially in critical domains like healthcare and finance. thus, there is an urgent need for methodologies to validate, guide, and interpret the decision pathways of llms. one extensively studied form of reasoning task in the ai community is planning and making sequential decisions. planning involves executing a series of valid actions within a defined context to transition from an initial state to a goal state [11]. even simple planning tasks pose challenges for llms, as they often struggle to achieve success rates and generate valid plans due to the requirement of validating each action’s feasibility in every state, backtracking previous states and actions, and employing long-term reasoning [15]. enhancing planning, reasoning, and introducing guarantees in multi-step decision making problems is crucial for improving llms’ reliability, which would open up their usage in some risk averse applications. the research community is actively developing new methods to address multi-step problems with llms by enhancing their reliability, interpretability, and robustness. one such technique is chain-of-thought (cot) [19], a prompt-based approach that incorporates intermediate reasoning steps, or “thoughts," alongside task input/output pairs. by decomposing the original problem into these intermediate steps, cot facilitates structured reasoning, thereby improving llms’ ability to tackle procedural problems. empirical evidence has demonstrated cot’s effectiveness in enhancing problem-solving capabilities [19]. furthermore, advancements have extended cot’s capabilities, including techniques like self-consistency with cot (cot-sc) [17] and causalcot [9], as well as more complex and flexible approaches like tree-of-thoughts [20] and graph-of-thoughts [1]. however, these techniques primarily focus on refining prompting strategies without explicitly leveraging the logical structure of problems to validate and guide llms’ reasoning processes. our approach associates llms to finite-state machines (fsms), commonly used in systems that require reliability and predictability, as a guiding structure to the decision-making processes of llms. this association restricts the model’s decision at each step, providing several additional guarantees enabling applications with tighter restrictions to benefit from modern ai capabilities. the key contributions of our work are: novel approach: we introduce a novel method that incorporates fsms as a guiding structure for llm decision-making processes. enhanced problem-solving: we provide evidence that validations, feedback loops, and restrictions enabled by fsms, derived from standard planning language definitions, enhance llms’ ability to solve planning problems. comprehensive comparison: we present a detailed comparison of our most effective setups, thoroughly analyzing their strengths and weaknesses in the context of planning problems. 2 background this section covers essential concepts for this work. it briefly explains how auto-regressive llms function and their advantages in solving planning problems. it then provides a formal definition of a planning problem. finally, it defines a mealy machine, a type of finite-state machine, which will be used to guide the llm in addressing planning problems. 2.1 large language models for planning large language models are built upon a transformer architecture [16], which undergoes training on a vast corpus of text samples sourced from the web. formally, we define a collection of examples \(x = (x_1, x_2, \dots , x_n)\), where each \(x_i = (\alpha _1, \alpha _2, \dots \alpha _{n_i})\) and every token \(\alpha _i\) is drawn from a predefined vocabulary \(\textbf{v}\). each llm is equipped with a tokenizer responsible for converting raw input text into a sequence of tokens, each belonging to \(\textbf{v}\). llms typically undergo training in an auto-regressive manner, where the probability of each token is solely dependent on its preceding tokens in the sequence, expressed as \(p(x_i) = \prod _{j=1}^{n_i}p(\alpha _j|\alpha _{0:j-1})\). during training, the model parameters are learned by maximizing the likelihood of the entire dataset, formulated as \(p(x) = \prod _{i=1}^{n}p(x_i)\). addressing planning problems with pre-trained llms is highly advantageous as these models can leverage their extensive world knowledge, commonsense understanding, and logical reasoning abilities. the aim is to generate a sequence of actions \((a_1, a_2, \dots , a_n)\) to reach a desired final state. graph structures have been widely used to enhance the reasoning capabilities of llms, with empirical evidence showing significant improvements in solving complex problems [13, 17, 19]. the cot approach is fundamental, modeling the inference process as a linear chain and explicitly capturing intermediate reasoning steps. building on cot, methods like tree-of-thoughts and graph-of-thoughts introduce more complex topologies, including branches, backtracking, and aggregating reasoning steps. nonetheless, incorporating prompt topologies alone is insufficient to tackle planning problems. studies such as [15] underscore the difficulties llms face in generating valid and optimal plans. recent research, exemplified in [10, 12], suggests that integrating feedback loops can significantly improve llms’ accuracy in planning tasks. consequently, implementing a mechanism for validating the model’s outputs and providing feedback emerges as a promising strategy for enhancing its performance. 2.2 planning task definition and finite-state machines a planning problem can be defined as a tuple \(\mathcal {p} = (\mathcal {s}, \mathcal {a}, s_0, \mathcal {g})\) [11], where: \(\mathcal {s}\) is a finite set of states. each state \(s \in \mathcal {s}\) is a complete assignment of values to a set of state variables (or fluents). \(s_0 \in \mathcal {s}\) is the initial state. \(\mathcal {g} \subset \mathcal {s}\) is the set of goal states where the goal condition is satisfied. this condition is specified as a conjunction of literals. \(\mathcal {a}\) is the set of actions, where each action \(a \in \mathcal {a}\) is defined by its preconditions and effects: preconditions \(\textit{pre}(a)\) are a set of literals that must be satisfied for the action to be applied. effects \(\textit{eff}(a)\) are a set of literals that describe how the state changes after applying the action. we define a transition model \(\gamma : \mathcal {s} \times \mathcal {a} \rightarrow \mathcal {s}\) that describes deterministic state transitions in the planning problem. if a state \(s \in \mathcal {s}\) satisfies \(\textit{pre}(a)\), then \(\gamma (s, a) = s'\), where \(s' \in \mathcal {s}\) is the new state resulting from applying \(\textit{eff}(a)\) to s. if s does not satisfy \(\textit{pre}(a)\), then the action cannot be applied. the planning domain definition language (pddl) is commonly used to represent planning problems, allowing for the encoding of a problem by defining its domain (types of objects, predicates, and actions) and its problem instance (initial state and goal state). there is extensive literature on planning algorithms for various tasks. an ai planner takes a pddl domain and problem as input and produces a solution. most state-of-the-art planners [6, 7] are variants of heuristic-based forward search, where pddl encodings are used to automatically derive a heuristic function. another common approach is the integration of markov decision processes (mdps) into automated planning. mdps are best suited for modeling stochastic processes where maximizing rewards in uncertain contexts is crucial [4, 11]. however, many mission-critical applications are modeled using deterministic finite-state machines due to reliability and predictability requirements. building on this foundation, our work investigates whether simply imposing an alphabet of interactively validated symbols can enhance llms’ ability to solve planning tasks. this restriction of possible actions associated with deterministic transitions is best encapsulated by the fsms formalism. a mealy machine is an fsm defined by a 6-tuple \((\mathcal {s}, s_0, \varsigma , \varlambda , \mathcal {t}, \mathcal {o})\), where: \(\mathcal {s}\) is a finite set of states. \(s_0 \in \mathcal {s}\) is the initial state. \(\varsigma \) is a finite set of input alphabet. \(\varlambda \) is a finite set of output alphabet. \(\mathcal {t}: \mathcal {s} \times \varsigma \rightarrow \mathcal {s}\) is a transition function. \(\mathcal {o}: \mathcal {s} \times \varsigma \rightarrow \varlambda \) is the output function. 3 associating llms to fsms in this setup, we denote the llm as a function \(l(p) = t\), where a prompt \(p \in \mathbf {v^*}\) is provided, and the output \(t \in \mathbf {v^*}\) is a text response. note that both input and output can be any sequence of tokens \((\alpha _1,\dots ,\alpha _n) \in \mathbf {v^*}\), however, only a small fraction of this domain constitute valid actions. the fsm can be derived from the problem’s pddl and is designed to receive an action \(a \in \varsigma \subset \mathbf {v^*}\) and return an output \(o \in \varlambda \subset \mathbf {v^*}\), which is appended to the next llm prompt p. this establishes an interactive loop between the llm and the fsm, where the language model proposes actions based on its context p, while the fsm models preconditions and effects. note that in this setup there’s no guarantee that the llm output t is a valid action, which means the llm can send invalid actions (i.e. symbols) to the fsm. however, it is possible to project t onto \(\varsigma \) by providing a validation function f such that $$\begin{aligned} &f:(\mathbf {v^*},\mathcal {s}) \rightarrow \varsigma \end{aligned}$$ (1) $$\begin{aligned} &f(t_i, s_j) = {\left\{ \begin{array}{ll} t_i, \ t_i \text { is a valid action in state } s_j \\ -1, \text { otherwise } \end{array}\right. } \end{aligned}$$ (2) this function checks whether t corresponds to a valid action for the current state, producing a common symbol \(-1\) for invalid actions. consequently, instead of the fsm’s transition and output functions receiving t, they use the output from the validation function: \(\mathcal {t}(s_j, f(t_i, s_j))\) and \(\mathcal {o}(s_j, f(t_i, s_j))\). to assess how fsm modeling can enhance the capabilities of llms, we evaluate seven distinct methods. these range from a straightforward preplanned input/output setup to more sophisticated approaches that utilize fsms to receive information and provide feedback on their actions. additionally, we examine the cot approach, as outlined in [15], where the initial prompt induces the model to generate structured reasoning based on a similar example. by analysing these variations we aim to measure three key aspects: 1. which method proves more effective: generating the entire plan at once, or generating each action individually while interacting with the fsm? 2. does integrating validation functions and feedback loops improve llms’ effectiveness in solving planning tasks? 3. what methods can be employed to restrict the range of actions available to llms, thereby guiding them towards optimal outcomes? initially, all methods receive the same context prompt \(p_c\) sourced from [14]. this prompt includes a detailed description of the planning task, a rulebook outlining all predicates and actions, and information about the current and goal states, as shown in fig. 1. each setup has its own request prompt \(p_r\), which solicits either the entire plan or individual actions, enabling customization based on each setup’s specific approach. when the inference begins, the epsilon-transition \(\mathcal {t}(s_{\text {start}},\epsilon ) = s_0\) occurs and the respective output \(\mathcal {o}(s_{\text {start}},\epsilon ) = p_c + p_r\) produces the initial prompt. thus, the state space becomes \(\mathcal {s} = \{s_{\text {start}}, s_{\text {end}}\} \cup \mathcal {s_{\text {task}}}\), where \(\mathcal {s_{\text {task}}}\) is the set of all possible states of the task being planned. the methods analyzed are combinations of three characteristics: preplanned vs interactive: preplanned methods involve requesting the whole sequence of actions to the llm before performing any kind of validation. this setup was assessed in [15], revealing a low success rate and a high prevalence of invalid plans. in this setup, the llm generates the entire plan \((t_1, t_2, \dots , t_n)\) solely based on \(p_c+p_r\). on the other hand, interactive methods require one action at a time, performing an interaction between fsm and llm at each step, allowing for invalid actions to be recognized earlier, preventing a reset back to \(s_0\). no validation vs validated vs restricted: three different schemes of validation are analyzed, namely, no validation, validated, and restricted. the no validation scheme simply halts the process in the case of invalid plans or actions. the validated scheme makes use of f to guarantee llm outputs are projected onto \(\varsigma \) and provides feedback in the case of invalid actions. finally, restricted methods provide all valid actions (obtained from the pddl) for a certain state to the llm through \(\mathcal {o}\). in this case the llm is asked to choose the number \(t_i\) corresponding to the selected action, but can still output any \(t \in \mathbf {v^*}\). to handle invalid choices and to convert the selected action into a valid symbol we make use of a modified f $$\begin{aligned} f(t_i, s_j) = {\left\{ \begin{array}{ll} a_{t_i}^{s_j} & t_i \in p_a(s_j), \\ -1 & \text {otherwise}. \end{array}\right. } \end{aligned}$$ (3) where \(a_{t_i}^{s_j}\) is the action that corresponds to option \(t_i\) at state \(s_j\). standard vs chain-of-thoughts: the standard prompting setup simply describes the goal and requests a plan (or the next action). on the other hand, cot prompting induces the llm to perform a step-by-step reasoning process for each interaction with the fsm. prompting strategy can be modified by changing the prompts produced by \(\mathcal {o}\). fig. 1. (left) context prompt used in all proposed methods. (right) preplanned no validation: we prompt the llm to generate the complete plan in one step. full size image fig. 2. (left) preplanned validated: we prompt the llm to generate the entire plan in a single step. if the plan is invalid, we provide feedback and prompt the llm to generate a new plan. (right) interactive no validation: we prompt the llm to generate one action at a time and check if the goal is reached after each action. this process continues until the llm indicates that the goal has been reached or the maximum number of iterations is exceeded. full size image fig. 3. (left) interactive validated: we prompt the llm to generate one action at a time and provide feedback on whether the action is valid or not. this process continues until the goal is reached or the maximum number of iterations is exceeded. (right) interactive restricted: we prompt the llm to choose one of all possible actions at the current state. this process continues until the goal is reached or the maximum number of iterations is exceeded. full size image 4 experimental setup the dataset is obtained from [14], which provides a platform to insert a valid pddl domain and generate instances of it. for this study, we utilized the blockworld domain, which captures common-sense block manipulations and consists of a set of blocks. each block is uniquely identified by color and can be placed either on a table or on top of other blocks. in each instance, the initial state is a specific block arrangement, and the goal state is to arrange some of these blocks in a stack in a particular order (fig. 2). the blockworld domain includes four different actions: pick up, put down, stack, and unstack. the preconditions for each action are as follows: to pick up a block, it must be clear (no block stacked on it), on the table, and the hand must be free. to put down a block, it must be on hand. to stack a block on another, the first block must be on hand and the second block must be clear. to unstack a block from another, the first block must be clear, stacked on the second block, and the hand must be free. this particular domain is widely adopted in the planning literature due to its simplicity and alignment with common-sense reasoning [5, 8]. in this work, we generated a dataset with 768 instances, each containing 4 blocks. the dataset is balanced in terms of optimal plan length, which ranges from 2 to 12 actions. we use the optimal plan length as an indicator of the instance’s complexity. for each instance, we use the same context prompt as in [14]. all prompts are zero-shot, except for methods using a cot approach, where a one-shot approach is required to induce structured reasoning. we evaluated three different llms in this work: mixtral:8x22b, llama3:70b, and gpt-4o. the temperature parameter is set to 0 for all models, ensuring that each experiment is deterministic. our goal is to verify the effectiveness of the proposed methods independently of the specific llm used. additionally, a maximum of 8 tries for the preplanned validated approach and 16 invalid actions for the interactive approaches are enforced (fig. 3). the following metrics are calculated for each experiment: success plan: indicates if the generated plan successfully reaches the goal without any errors. valid plan: indicates if the generated plan is valid, meaning it does not contain any invalid actions. optimal plan: indicates if the generated plan is the optimal plan. relative plan length: the total number of actions in the generated plan divided by the optimal plan length. tokens: the number of tokens processed during inference, measured in thousands. distance to initial state: the number of actions from the initial state to the final state of the generated plan. distance to goal state: the number of actions from the final state of the generated plan to the goal state. 5 results numerical results are presented in table 1, displaying average values of accuracy, optimal plan percentage, number of processed tokens and relative plan length metrics for all methods and models. the validated and restricted interactive methods consistently outperform other setups in most cases. they achieve higher accuracy than their preplanned and unvalidated counterparts regardless of the model. an important finding is that the cot approach achieved good results exclusively with the gpt-4o model, which is specifically trained to handle cot reasoning. in contrast, the mixtral and llama models performed worse with the cot scheme in both the preplanned and the interactive validated methods. this provides evidence that fsm-based validation provides increased accuracy in a more generic sense, being independent of how the particular llm was trained. table 1. numerical results for all methods and models. the metrics include accuracy percentage, number of tokens processed, and relative plan length, calculated based on all generated plans. the optimal plan percentage is calculated only for the successful plans. these results represent the average values across the entire dataset. std refers to standard prompt and cot to chain of thought prompt.full size table the optimal plan percentage, which is calculated for successful plans, is higher for preplanned setups for two main reasons: first, since preplanned methods generate fewer successful plans, often for simpler tasks, they tend to show a higher optimal plan percentage on average. second, interactive validated setups often generate longer successful plans, because these methods have a deterministic function to hard stop inference when the goal state is reached, instead of relying on the llm to recognize it. this allows for longer plans to eventually achieve the goal state in situations that other methods fail. a downside of using interactive methods is their increased computational cost. as also shown in table 1, the interactive setup processes an order of magnitude more tokens than the preplanned setups. this is because interactive schemes involve a sequence of prompts where all previous history is used as input, causing the size of each prompt to increase with each iteration. special cases are those with the cot scheme, which significantly increases the number of processed tokens for both preplanned and interactive setups. it’s important to note that there is potential for prompt optimization in these schemes, which could make them faster and cheaper. as displayed in table 2 the preplanned and interactive methods without validation struggle with a low percentage of valid plans, as they are not robust against llm hallucinations. in contrast, the interactive validated and restricted methods, by design, handle invalid inputs from the llm, ensuring that all generated plans are valid. once again, it is relevant to note that using cot with mixtral and llama models frequently returned invalid actions, particularly in unfamiliar formats, demonstrating that such prompt engineering techniques can be sensitive to various factors and may work unpredictably across different models. table 2. percentage of valid plans for methods that cannot handle invalid input. these results represent the average values across the entire dataset.full size table considering the notably superior performance of the validated and restricted interactive methods compared to others, our analysis will primarily focus on these methods. figure 4 displays the average accuracy for each optimal plan length, serving as a complexity measure for each task in the dataset. as the optimal plan length increases, accuracy decreases, indicating that planning capabilities are sensitive to the complexity of the tasks. furthermore, the two setups show different behaviors: the interactive validated setup demonstrates less sensitivity to optimal plan length, achieving a significant number of correct plans for more complex tasks, particularly with gpt-4o. in contrast, the interactive restricted setup shows more pronounced sensitivity, achieving higher accuracy for simpler tasks but experiencing a decrease in accuracy for more complex ones. fig. 4. the evaluation of average accuracy for the interactive validated and restricted methods is divided by optimal plan length, which represents the minimal number of actions needed to achieve the goal state. full size image this behavior is further analysed in figs. 5 and 6, which represent the distance from the final state of the generated plan to the goal and initial state, respectively. these metrics are only calculated for generated plans that didn’t achieve the goal. figure 5 shows that the two methods have a similar behavior: when they produce plans that don’t achieve the goal state, these plans tend to finish far from the goal state. on average, the final state of the predicted plans is as far as the initial state of each task, indicating that the final state is not a good approximation of the goal state. however, in fig. 6, one can see the distance from the final state of the produced plan to the initial state. this visualization shows that the distances for the interactive validated method are greater than the distances for the interactive restricted method. thus, validated methods tends to explore more of the state space, resulting in final states farther away from the initial state, while restricted methods tend to perform local searches. this behavior is more pronounced for larger and more complex models, such as gpt-4o and llama3. fig. 5. the distance of the generated plan’s final state from the goal state is measured in terms of the minimum number of actions required to transition from the final state of the generated plan to the goal state. it was only calculated for plans that didn’t achieve the goal. full size image fig. 6. the distance of the generated plan’s final state from the initial state is measured in terms of the minimum number of actions required to transition from the final state of the generated plan to the initial state. it was only calculated for plans that didn’t achieve the goal. full size image fig. 7. we calculate the number of revisited states within a generated plan by summing up the occurrences of each state that appears more than once. this calculation is performed for all the plans generated. full size image figure 7 presents further evidence for the difference in exploratory behavior between the methods. it shows the average number of times each model revisited states within a generated plan. we observe a significant difference: the interactive validated method exhibits a lower number of repeated states, indicating a tendency to explore diverse states. conversely, the interactive restricted method frequently revisits states, suggesting a propensity to get stuck in loops. this looping behavior is likely inherent to the restricted setup. restricted models receive previous actions as input, but often fail to recognize when they’re trapped in a cycle. this potentially encourages short-term decision-making, hindering their ability to analyze past actions and states. we suspect this issue is exacerbated by the requirement to return action numbers instead of names in the restricted setting. this implicit task progress representation might make it challenging for the model to consider the entire historical trajectory when choosing the next action. consequently, the restricted version tends towards local optimizations, leading to frequent loop formation. 6 conclusion our results provide evidence that incorporating fsms as a guiding structure enhances llms’ ability to solve planning problems. by interacting with the fsm and leveraging integrated validations, feedback loops, and restrictions, llms demonstrably generate plans with increased accuracy across tasks of varying complexity. although the accuracy achieved is not yet ideal, the proposed method demonstrates a valuable tool for improving model inference in reasoning tasks, offering benefits independent of other prompt engineering techniques. fsm integration was also compared to the cot prompting technique in both preplanned and interactive validated schemes. cot showed superior performance exclusively for the gpt-4o model, while fsm integration greatly increased accuracy for all models. this provides evidence that fsm integration is a more consistent solution than prompt engineering techniques that can behave unpredictably across different scenarios. the proposed interactive validated and restricted setups offer a crucial advantage by being inherently designed to handle invalid inputs, thus guaranteeing that all generated plans are valid. this is a substantial improvement over traditional approaches, where llms frequently generate invalid plans. we observed that incorporating validation encourages exploratory behavior in the models, leading to better performance on complex tasks across all three compared models. conversely, while incorporating restrictions can enhance performance on simpler tasks, it can also cause the models to enter loops, resulting in poorer performance on more complex tasks. a significant downside of interactive approaches is their increased computational cost. results show that the number of tokens processed in these schemes is an order of magnitude higher than in preplanned setups, underscoring the necessity of prompt optimization in these scenarios. despite the simplicity of the experimental context, the findings open up new avenues for future research aimed at enhancing reasoning capabilities using logical structures that constrain the action space in problem-solving. the proposed method also introduces greater predictability and robustness into the system. our findings suggest promising avenues for future research. one direction involves developing approaches that can unify validation with feedback and restrictions within the llm’s decision-making process. this could lead to more reliable and robust models capable of both short-term and long-term reasoning. references besta, m., et al.: graph of thoughts: solving elaborate problems with large language models. in: proceedings of the aaai conference on artificial intelligence, vol. 38, no. 16, pp. 17682–17690 (2024). https://doi.org/10.1609/aaai.v38i16.29720. https://ojs.aaai.org/index.php/aaai/article/view/29720 bubeck, s., et al.: sparks of artificial general intelligence: early experiments with gpt-4 (2023) google scholar  chen, m., et al.: evaluating large language models trained on code (2021) google scholar  ghallab, m., nau, d., traverso, p.: automated planning: theory and practice. the morgan kaufmann series in artificial intelligence, morgan kaufmann, amsterdam (2004). http://www.sciencedirect.com/science/book/9781558608566 guan, l., valmeekam, k., sreedharan, s., kambhampati, s.: leveraging pre-trained large language models to construct and utilize world models for model-based task planning. in: oh, a., naumann, t., globerson, a., saenko, k., hardt, m., levine, s. (eds.) advances in neural information processing systems, vol. 36, pp. 79081–79094. curran associates, inc. (2023) google scholar  helmert, m.: the fast downward planning system. j. artif. intell. res. 26, 191–246 (2006). https://doi.org/10.1613/jair.1705 article  math  google scholar  hoffmann, j.: ff: the fast-forward planning system. ai mag. 22(3), 57 (2001). https://doi.org/10.1609/aimag.v22i3.1572. https://ojs.aaai.org/aimagazine/index.php/aimagazine/article/view/1572 jiménez, s., de la rosa, t., fernández, s., fernández, f., borrajo, d.: a review of machine learning for automated planning. knowl. eng. rev. 27(4), 433–467 (2012). https://doi.org/10.1017/s026988891200001x article  math  google scholar  jin, z., et al.: cladder: a benchmark to assess causal reasoning capabilities of language models (2023) google scholar  raman, s.s., cohen, v., rosen, e., idrees, i., paulius, d., tellex, s.: planning with large language models via corrective re-prompting. in: neurips 2022 foundation models for decision making workshop (2022). https://openreview.net/forum?id=cmdmrbe1tks russell, s., norvig, p.: artificial intelligence: a modern approach, 3 edn. prentice hall (2010) google scholar  stechly, k., marquez, m., kambhampati, s.: gpt-4 doesn’t know it’s wrong: an analysis of iterative prompting for reasoning problems. in: neurips 2023 foundation models for decision making workshop (2023). https://openreview.net/forum?id=pmtzjdyb68 suzgun, m., et al.: challenging big-bench tasks and whether chain-of-thought can solve them (2022) google scholar  valmeekam, k., marquez, m., olmo, a., sreedharan, s., kambhampati, s.: planbench: an extensible benchmark for evaluating large language models on planning and reasoning about change. in: thirty-seventh conference on neural information processing systems datasets and benchmarks track (2023). https://openreview.net/forum?id=yxogl4uquo valmeekam, k., marquez, m., sreedharan, s., kambhampati, s.: on the planning abilities of large language models a critical investigation. in: thirty-seventh conference on neural information processing systems (2023). https://openreview.net/forum?id=x6deqxisew vaswani, a., et al.: attention is all you need. in: guyon, i., et al. (eds.) advances in neural information processing systems, vol. 30. curran associates, inc. (2017) google scholar  wang, x., wei, j., schuurmans, d., le, q.v.: h. chi, sharan narang, aakanksha chowdhery, and denny zhou. self-consistency improves chain of thought reasoning in language models. in: the eleventh international conference on learning representations, vol. 1 (2023) google scholar  wei, j., et al.: h. chi, tatsunori hashimoto, oriol vinyals, percy liang, jeff dean, and william fedus. emergent abilities of large language models. trans. mach. learn. res. 2022 (2022) google scholar  wei, j., et al.: chain-of-thought prompting elicits reasoning in large language models. in: koyejo, s., mohamed, s., agarwal, a., belgrave, d., cho, k., oh, a. (eds.) advances in neural information processing systems, vol. 35, pp. 24824–24837. curran associates, inc. (2022) google scholar  yao, s., et al.: tree of thoughts: deliberate problem solving with large language models. in: oh, a., naumann, t., globerson, a., saenko, k., hardt, m., levine, s. (eds.) advances in neural information processing systems, vol. 36, pp. 11809–11822. curran associates, inc. (2023) google scholar  download references author information authors and affiliations universidade de são paulo, são paulo, sp, brazil eduardo faria cabrera, marcel rodrigues de barros & anna helena reali costa authors eduardo faria cabreraview author publications search author on:pubmed google scholar marcel rodrigues de barrosview author publications search author on:pubmed google scholar anna helena reali costaview author publications search author on:pubmed google scholar corresponding author correspondence to eduardo faria cabrera . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cabrera, e.f., de barros, m.r., costa, a.h.r. (2025). improving llms’ reasoning and planning with finite-state machines. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_8 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords large language models machine learning planning graph representations publish with us policies and ethics profiles marcel rodrigues de barros view author profile anna helena reali costa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature specifying preferences over policies using branching time temporal logic | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper specifying preferences over policies using branching time temporal logic conference paper first online: 12 october 2023 pp 128–143 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) specifying preferences over policies using branching time temporal logic download book pdf download book epub warlles carlos costa machado9, viviane bonadia dos santos10, leliane nunes de barros10 & … maria viviane de menezes11  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 521 accesses abstract automated planning is the subarea of ai devoted to developing algorithms that can solve sequential decision making problems. by taking a formal description of the environment, a planning algorithm generates a plan of actions (also called policy) that can guide an agent to accomplish a certain task. classical planning assumes the environment is fully-observed and evolves in a deterministic way considering only simple reachability goals (e.g. a set of states to be reached by a plan or policy). in this work, we approach fully-observed non-deterministic planning (fond) tasks which allow the specification of complex goals such as the preference over policy quality (weak, strong or strong-cyclic) and preferences over states in the paths generated by a policy. to solve this problem we propose formulae in \(\alpha \)-ctl (branching time) temporal logic and use planning as model checking algorithms based on \(\alpha \)-ctl to generate a solution that captures both, agent’s preferences and the desired policy quality. to evaluate the effectiveness of the proposed formulae and algorithms, we run experiments in the rovers benchmark domain. up to our knowledge, this is the first work to solve non-deterministic planning problems with preferences using a ctl temporal logic. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others research and development of a temporal model of branching time for the intelligent systems of real-time chapter © 2024 branching time logics with multiagent temporal accessibility relations article 27 may 2021 collaborative human-agent planning for resilience chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. logic in ai formal languages and automata theory formal logic linear logic symbolic ai policy formulation 1 introduction automated planning [9] is the subarea of artificial intelligence that studies the process of automatically generating a plan of actions that, when executed, enables an agent to accomplish a task. a state refers to a particular configuration of the agent’s environment and can be described by a set of properties. the initial state is the environment configuration at the beginning of the plan execution. a planning problem is defined by means of a formal description of the environment (called planning domain) plus an initial state and a goal description. the planning domain models the environment based on the set of actions the agent can perform. thus an action can be seen as a manifestation of the agent’s intent to make the environment to change from a state to another. finally, the goal description can vary from a simple reachability goal formula (e.g. a propositional formula that must be satisfied by a set of goal states) to complex temporal goal formulae [22, 26], capable to represent the agent’s preferences over the plan solution and its quality as well. fig. 1. examples of state transition systems representing planning domains. full size image figure 1a shows a planning domain represented by a state transition system given by a graph with vertices representing the states labeled by propositional atoms and directed edges representing the actions labeled by action names. to represent states we make the closed world assumption, i.e. a state is described only by the propositions that are true and any proposition that is false is not included in the state label. for example, in the state \(s_0\) (fig. 1a) p is true, q is false and r is false. finding a solution to a planning problem corresponds to select the best action at each state, which is a computationally hard problem [11]. therefore many research has been devoted to find efficient solutions for planning problems mainly focusing on making different assumptions about the environment and the capability to express goal formulae. deterministic planning. research in this area assumes the world is fully-observed and there are no uncertainties about the effects of the actions [9]. thus, when an action is executed in a state, there is only one possible successor state. the solution of a deterministic planning problem is a sequence of actions that takes the environment from the initial state to a state that satisfies a simple reachability goal description. examples of efficient algorithms for deterministic planning are based on heuristic search [10] and boolean satisfiability [13]. example 1 (deterministic planning). suppose the environment (system) depicted in fig. 1a is currently in the initial state \(s_0\) and the agent goal is to reach the state where property \(\varphi = \lnot p \wedge q \wedge r\) holds. the sequence of actions \(\langle a_1,a_4\rangle \) is a plan solution since it deterministically takes the agent from state \(s_{0}\) to state \(s_{5}\), where \(\varphi \) is satisfied. other possible plans are: \(\langle a_2,a_6\rangle \) e \(\langle a_3,a_7\rangle \). non-deterministic planning. also called fond planning fully observed non-deterministic planning, where we make the assumption the environment evolves in a non-deterministic way, due to the occurrence of natural events without the agent’s control (called exogenous events). in this case, the execution of an action can lead to a set of possible future states [22]. a solution for a non-deterministic planning problem is policy: a mapping from states to actions [9]. a policy can be classified into three quality categories: (i) a weak policy is a solution that may achieve the goal but offers no guarantee due to the non-determinism of the actions; (ii) a strong policy is a solution that guarantees the goal achievement, regardless of non-determinism of the actions; and (iii) a strong-cyclic policy is a solution that ensures the goal achievement, but its execution may involve infinite cycles (assuming that the agent will eventually exit all cycles) [6]. efficient algorithms for non-deterministic planning are based on model checking [6, 18, 21, 28] and determinization of the actions to obtain relevant policies [19]. example 2 (non-deterministic planning). consider the non-deterministic planning domain depicted in fig. 1b, which is similar to the domain shown in fig. 1a except for the fact that the actions \(a_2\) and \(a_4\) are non-deterministic. these actions can eventually take the agent, respectively, to the states {\(s_2\), \(s_3\)} and {\(s_4\), \(s_5\)}. suppose the agent is in state \(s_{0}\) and its goal is to reach state where the goal formula holds: \(\varphi = \lnot p \wedge q \wedge r\). a solution is a policy: \(\{(s_{0},a_1), (s_{1},a_4)\}\). this policy indicates that if the agent starts at \(s_{0}\) and performs the deterministic action \(a_1\), it will reach state \(s_{1}\). subsequently, if it takes the non-deterministic action \(a_4\) in \(s_{1}\), the agent will end up in either \(s_{4}\) or \(s_{5}\) (weak policy). other solutions are strong policies \(\{(s_0,a_2),(s_2,a_6),(s_3,a_7)\}\) and \(\{(s_0,a_3),(s_3,a_7)\}\). planning with preferences. this form of planning allows the specification of complex goal formulae to represent properties that should be preserved or avoided along the path to finally reach a goal state (also called extended reachability goal). there are two types of preferences: quantitative preferences and qualitative preferences. in the first type, preferences represent a subset of desirable goals when it is not possible to achieve all goals [25]. in the second type, preferences deal with additional properties that must hold along the path to achieve a goal state. qualitative preferences can be used to express safety and liveness properties [16]. informally, a safety property indicates that “something (bad) should not happen” during the execution of a plan, while a liveness property expresses that eventually “something (good) should happen” during the plan execution [15]. qualitative preferences can be expressed by operators in pddl such as: always, sometime, sometime-before and at-most-once [3, 8]. example 3 (planning with qualitative preferences in deterministic environments). consider the planning domain depicted in fig. 1a and a problem where the initial state is \(s_{0}\) and the goal is to reach a state where the propositional formula \(\varphi = \lnot p \wedge q \wedge r\) holds. suppose that the agent may prefer to visit, on the path to reach the goal, only states where p is true (liveness property). in this case, possible solutions are \(\langle a_{1}, a_{4} \rangle \), \(\langle a_{2}, a_{6} \rangle \) and \(\langle a_{3}, a_{7} \rangle \). this work focuses on the qualitative preference “always (p)”, which aims to find a plan where the property p is true in all states along the path to reach the goal. example 4 (planning with qualitative preferences in non-deterministic environments). consider the planning domain depicted in fig. 1b and the agent is at state \(s_{0}\) and its goal is to reach a state where the property \(\varphi = \lnot p \wedge q \wedge r\) is satisfied, but with the additional requirement that the agent must pass only through states where property p is true before reaching the goal. the solution \(\pi \) = \(\{(s_{0},a_1), (s_{1},a_4)\}\) is a weak police that satisfies also the user preference, once it corresponds to a path that possibly reaches the goal \(\varphi \) and passes only through states where p is satisfied. the solutions \(\pi \) = \(\{(s_{0},a_2), (s_{2},a_6), (s_3, a_7) \}\) and \(\pi \) = \(\{(s_{0},a_3), (s_3, a_7) \}\) are strong polices that satisfies the user preference, once it reaches the goal \(\varphi \) for sure and passes only through states where p is satisfied. most algorithms for planning with qualitative preferences work on deterministic domains [2, 12, 14, 20]; and the few works on fond planning do not include preferences over the policy quality. this is because they are based on ltl linear time logic [23] which can not have path quantifiers. to the best of our knowledge, this is the first step toward extending the notion of qualitative preferences in non-deterministic environments to also include preferences over policy quality. in order to do so, we apply the notion of extended reachability goals in model checking based on the branching time temporal logica called \(\alpha \)-ctl [27] to express qualitative preferences as extended reachability goals and solve existing algorithms for \(\alpha \)-ctl model checking. in this paper, we propose solutions for planning problems that address two aspects present in real-world environments: (i) we consider that the actions can have non-deterministic effects and; (ii) we consider that the user wants to express its preferences over the policy paths, and over the policy quality as well. this paper is organized as follows: sect. 2 presents the related works; sect. 3 describes the foundations; sect. 4 details how policy preferences can be expressed using the \(\alpha \)-ctl temporal logic [22]; sect. 5 presents our experimental analysis and, sect. 6 presents the conclusions and future work. 2 related work the work in [5] presents an approach to non-deterministic planning in which the qualitative preferences are expressed as extended (reachability) goals in linear time logic (ltl) [23]. let’s call this a problem p and to solve it, the authors proposed the following extra-logical approach: (i) build a buchi automaton for the extended goal expressed as an ltl formula; (ii) perform the determinization of non-deterministic actions (iii) build a new problem \(p'\) without extended goals; (iv) use a non-deterministic planner to produce a policy for \(p'\) and; (v) convert the resulting policy into a solution for p. in our work, we rely on a solution based on ctl (a branching time propositional logic) able to solve non-deterministic planning without the need to perform determinizations. the solution for planning with qualitative preferences in deterministic domains, presented in [20], involves transforming a problem with preferences p into a problem without preferences \(p'\). for each preference in p, a new goal, which is false in the initial state, is added in \(p'\). the new goal can be achieved by an action with zero cost, but requires the preference to be satisfied, or an action with a cost equal to the utility of the preference that can only be performed if the preference is false. due to this transformation method, any classical planner can be used to solve problems with preferences. the work in [4] introduces a method for solving generalized planning problems, where the same plan can be used for multiple problem instances. it generates policies by transforming multiple concrete problems into a single abstract (lifted) problem that captures their common structure. the global structure of the problems can be captured through qualitative preferences expressed as formulas in ltl logic. in addition, the authors demonstrate that for a wide class of problems, path constraints can be compiled, which reduces generalized planning into non-deterministic planning. the work in [27] presents a planning algorithm that aims to solve non-deterministic planning problems with temporally extended goals (complex goals), while also considering the quality of the policy (weak, strong, or strong-cyclic). the proposed planner utilizes the \(\alpha -\)ctl model checking framework to tackle these kind of problems. our work is an application of such planner for problems with complex goals involving qualitative preferences. 3 foundations 3.1 non-deterministic planning in the real-world situations, the nature can cause unpredictability in the effects the actions. as a result, when an agent is in a state \(s_i\) and chooses an action \(a_i\) with the intention of reaching a particular state \(s_j\), the interference of nature can cause the agent to end up in a different state \(s_k\) instead. a non-deterministic planning domain can be characterized as a state transition system, as showed in definition 1. in this kind of domain, when an action is performed in one state, it can lead to more than one successor state. definition 1 [non-deterministic planning domain] given a set of propositional atoms \(\mathbb {p}\) and a set of actions \(\mathbb {a}\), a non-deterministic planning domain is defined by a tuple \(\mathcal {d} = \langle s, l, t\rangle \) where states are labeled by elements of \(\mathbb {p}\) and actions are labeled by elements of \(\mathbb {a}\) [9]: s is a finite set of states; \( l : s \rightarrow 2^{\mathbb {p}}\) is a state label function; and \( t : s \times \mathbb {a} \rightarrow 2^{s}\) is a non-deterministic state transition function. we assume that the set \(\mathbb {a}\) contains the trivial action \(\tau \) and that \(t(s,\tau ) = \{s\}\), for every final state \(s \in s\). intuitively, this action represents that the agent may choose to do nothing. as the number of propositions increases, it becomes unfeasible to explicitly represent planning domains. therefore, the planning community uses action languages (such as the planning domain description language pddl) to concisely describe domains. in this representation, actions are defined by their preconditions and effects. preconditions are propositions that must be true in a state for an action to be executed, while effects are the literals that are modified in a state to produce a set of possible next states. after defining the domain, the next step is to formalize the planning problem. definition 2 (non-deterministic planning problem). given a non deterministic planning domain \(\mathcal {d}\), a non-deterministic planning problem is defined by a tuple \(p = (\mathcal {d}, s_0, \varphi )\) where: \(\mathcal {d}\) is a non-deterministic planning domain; \(s_0\) is the initial state, and \(\varphi \) is a propositional formula representing the goal. due to uncertainties about the effects of the actions, a solution to a non-deterministic problem is a mapping from states to actions, is called policy [9]. definition 3 (policy). let p = \(\langle \mathcal {d}, s_{0}, \varphi \rangle \) be a planning problem in the domain \(\mathcal {d} = \langle s, l, t\rangle \) with non-deterministic actions. a policy \(\pi \) is a partial function \(\pi : s \rightarrow a\) which maps states to actions; such that, for all state \(s \in s\) if \(\pi \) is set to s then [21]. the quality of a policy can be: weak are policies that can reach a goal state, but there is no guarantee due to the non-determinism of actions; strong are policies that reach a goal state, independently of the non-determinism and, strong-cyclic are policy that guarantees to reach a goal state, under the assumption that the agent will eventually exit cycles [6]. definition 4 (set of states reachable by a policy). let \(p = \langle \mathcal {d},s_{0},\varphi \rangle \) be a non-deterministic planning problem and \(\pi \) a policy to a planning problem p. the set of states reachable by a policy \(\pi \), denoted by \(s_{reach[\pi ]} \) is defined by \( \{ s:(s,a) \in \pi \} \cup \{ s{'} : (s,a) \in \pi \,e \, s{'} \in \textrm{t}(s,a) \}\) [21]. definition 5 (execution structure of a policy). let \(p = \langle \mathcal {d},s_{0},\varphi \rangle \) be a non-deterministic planning problem and \(\pi \) be a policy for a planning domain \(\mathcal {d}\). the execution structure induced by \(\pi \) from \(s_0 \in s\) is a tuple \(\langle s_{reach[\pi ]} ,t \rangle \) (\(s_{reach[\pi ]} \subseteq s\) and \(t \subseteq s \times \mathbb {a} \times s\)) which contains all the states and transitions that can be reached when executing policy \(\pi \). 3.2 planning as model checking model checking [7] is a formal technique that explores all possible states of a transition system to verify whether a given property holds. applying model checking involves: modeling the system; specifying the property to be verified via a logical formula; and verifying the property automatically in the model. definition 6 (action labelled transition system). let \(\mathbb {p}\) be a not-empty set of atomic proposition and \(\mathbb {a}\) a set of action names, an action labelled transition system is a tuple \( \mathcal {m}= \langle s,l,t \rangle \) where: s is a finite nonempty set of states; \(l : s \rightarrow 2^\mathbb {p}\) is the state labeling function and; \(t \subseteq s \times \mathbb {a} \times s\) is the state transition relation. definition 7 (path in an action labelled transition system). a path \(\rho \) in an action labelled transition system \(\mathcal {m}\) is a sequence of states \(s_{0}, s_{1}, s_{2}, s_{3}, \ldots ,\) such that \(s_i \in s\) and \((s_{i}, a, s_{i+1}) \in t\), for all \( i \ge 0 \). we can combine the model checking and planning approaches to find solutions to planning problems. in this context, the domain represents the model to be checked, and the temporal logic formula expresses the planning goal to be satisfied. 3.3 the temporal logic \(\alpha \)-ctl the branching time temporal logic \(\alpha \)-ctl [21] is an extension of ctl logic able to consider actions by labeling state transitions. with such extension it is possible to solve fond planning problems without to appeal to extra-logical procedures. in this logic, temporal operators are represented by “dotted” symbols (definition 8). the formulae of \(\alpha \)-ctl are composed by atomic propositions, logical connectives(\(\lnot ,\, \wedge \, \vee \,\)), path quantifiers (\(\exists \) and \(\forall \)) and temporal operators: , , and . definition 8 (\(\alpha \) -ctl’s syntax). let \(p \in \mathbb {p}\) be an atomic proposition. the syntax of \(\alpha \)-ctl [21] is inductively defined as: the temporal operators derived from are and . in \(\alpha \)-ctl, the temporal model \(\mathcal {m}\) with signature (\(\mathbb {p},\mathbb {a}\)) is labeled transition system, i.e., a states transition system whose states are labeled by propositions and whose the transitions are labeled by actions (definition 9): definition 9 (\(\alpha \) -ctl’s model). a temporal model \(\mathcal {m}\) with signature (\(\mathbb {p},\mathbb {a}\)) in the logic \(\alpha \)-ctl is state transition system \(\mathcal {d} = \langle s,l,t \rangle \), where: s :  is a non-empty finite set of states; l : s \(\rightarrow 2^{\mathbb {p}}\) is a state labeling function; \(t:s \,\times \,\mathbb {a} \rightarrow 2^{s}\) is states transition function labeling by actions. the semantics of the local temporal operators ( and ) is given by preimage functions, while the semantics of the global temporal operators (, , ) is derived from the semantics of the local temporal operators, by using least (\(\nu \)) and greatest (\(\mu \)) fixpoint operations. definition 10 (weak preimage in \(\alpha \) -ctl formula). let \(y \subseteq s \) be a set of states. the weak preimage of y, denoted by \(\mathcal {t}^{-}_\exists (y)\) is the set . definition 11 (strong preimage in \(\alpha \) -ctl formula). let \(y \subseteq s \) be a set of states. the weak preimage of y, denoted by \(\mathcal {t}^{-}_\forall (y)\) is the set . definition 12 (intension of an \(\alpha \) -ctl formula). let \(\mathcal {d} = \langle s, l, t\rangle \) be a temporal model (or a non-deterministic planning domain) with signature (\(\mathbb {p}, \mathbb {a}\)). the intension of an \(\alpha \)-ctl formula \(\varphi \) in \(\mathcal {d}\) (or the set of states satisfying \(\varphi \) in \(\mathcal {d}\)), denoted by \(\llbracket \varphi \rrbracket {\mathcal {d}}\), is defined as: \(\llbracket p \rrbracket _{\mathcal {d}} = \{ s \, : \, p \, \in \, l(s)\}(by \, definition, \llbracket \top \rrbracket _{\mathcal {d}} = s \, and \, \llbracket \bot \rrbracket _{\mathcal {d}} = \emptyset ) \) \(\llbracket \lnot p \rrbracket _{\mathcal {d}} = s \smallsetminus \llbracket p \rrbracket _{\mathcal {d}} \) \(\llbracket \varphi _1 \wedge \varphi _2 \rrbracket _{\mathcal {d}} = \llbracket \varphi _1 \rrbracket _{\mathcal {d}} \cap \llbracket \varphi _2 \rrbracket _{\mathcal {d}}\) \(\llbracket \varphi _1 \vee \varphi _2 \rrbracket _{\mathcal {d}} = \llbracket \varphi _1 \rrbracket _{\mathcal {d}} \cup \llbracket \varphi _2 \rrbracket _{\mathcal {d}}\) figure 2 shows the semantics of operators used in this work: (a) considering each effect of action a when applied in \(s_0\), p is globally true; (b) considering each effect of action a when applied in \(s_0\), p is true until q is true and; (c) considering some effect of action a when applied in \(s_0\), p is true until q is true. fig. 2. semantics of the temporal operators of the logic \(\alpha \)-clt. full size image 3.4 reasoning about non-deterministic actions the representation of a planning domains through of a action language, e.g. pddl, allow a compact representation of the state space. thus, we can start from a initial state and progressively advance, determining a sequence of actions and states that lead achievement of the goal state progressive search [10] or from goal state and regressively determining the sequence of the actions and state that lead achievement of the initial state regressive search [24]. from the representation of the actions through of the preconditions an effect, is possible compute the set of states y that precede a set x through of the regression operations of a set of states. the regression of a set of states x by action a leads to a set y of predecessor states. however, due to the non-determinism of actions, to the applying an action a on y, the states on x can be required (strong regression) or possibly (weak regression) achieved [18]. using the representation of states and actions as propositional formulas [18], is possible to compute the set of predecessor states (weak and strong regression operations). these operations was implemented using binary decision diagram [1] and incorporated to a planner capable of performing symbolic model checking using directly the actions representation with preconditions and effects. 3.5 planning as \(\alpha -\)ctl model checking in this section, we briefly describe the planning as \(\alpha -\)ctl model checking algorithms used in this work. the algorithms receive the planning problem, whose domain is given by actions with preconditions and effects; an initial state \(s_0\); a goal formula \(\varphi \); and a propositional atom p, representing the qualitative preference to be satisfied in all states along the path to achieve the goal. the algorithm sat-au computes the submodel of the domain that satisfies the \(\alpha \)-ctl formula . it performs strong regression [17], computing a path that reaches a goal \(\varphi _2\) while each state along the path towards \(s_0\) also satisfies the formula \(\varphi _{1}\). this operation computes the predecessors states directly using actions defined in terms of preconditions and effects (fig. 3). fig. 3. sat-au computes the set of states satisfying full size image figure 4 provides an example of how to compute the set of states that satisfies , where \(\varphi \) is the goal formula \(\lnot p \wedge q \wedge r\). the algorithm starts by computing the set of pairs state-action whose states satisfies \(\varphi \) (\(y = \{(s_5, \tau \}\)), as showed in fig. 4(a). in the first iteration (fig. 4b), the algorithm computes the strong regression of the set of states satisfying \(\varphi \) and performs the intersection with the set of states satisfying p, obtaining the set of pairs state-action \(y = \{(s_5\), \(\tau \)), \((s_2,a_6)\), \((s_3,a_7)\}\). in the second iteration (fig. 4c), the strong regression of the set of states y is computed, the intersection with the set of states satisfying p is done, resulting in the set of pairs state-action \(y = \{(s_5\), \(\tau \)), \((s_2,a_6), (s_3, a_7), (s_0,a_2), (s_0, a_3) \}\). the fixed point is reached after the second iteration, once no new pair state-action is obtained with weak regression operation in the y set. similarly, the algorithm sat-eu compute the set of states that satisfy the formula , but using weak regression operations [17]. in addition, the algorithm sat-ag computes a submodel satisfying the \(\alpha \)-ctl formula . it performs regressive search from goal states (states that satisfies \(\varphi _1\)) towards the initial state \(s_0\) by using strong regression operations [17]. fig. 4. computing the always preference for strong solutions. full size image 4 specifying preferences over policies in \(\alpha \)-ctl in this section we show how to specify preferences about policies in \(\alpha -\)ctl temporal logic. to the best of our knowledge, this work is the first step toward extending the notion of qualitative preferences in non-deterministic environments to also include the quality of policy (weak, strong or strong-cyclic). in order to do so, we apply the notion of extended goals in \(\alpha \)-ctl [27] to express qualitative preferences. in this work, we will specify the qualitative preference always(p), which expresses that the property p must occur in all states in the path to achieve the goal. consider the planning problem \(p = <\mathcal {d}, s_0, \varphi >\) where \(\mathcal {d}\) is the domain in figure 4a, \(s_0\) is the initial state and the goal \(\varphi = \lnot p \wedge q \wedge r\) is satisfied in state \(s_5\). the non-deterministic actions are \(a_2\) and \(a_4\). notice that: the solution \(\{(s_0, a_1), (s_1, a_4)\}\) is a weak policy that satisfies the preference always(p) on the path to reach the goal state; the solution \((s_0, a_2), (s_2, a_5), (s_3,a_7)\) is a strong solution that satisfies the preference always(p) on the path to reach the goal state; the solution \((s_0, a_3), (s_3, a_7)\) is also a strong solution that satisfies the preference always(p). 4.1 specifying the always preference for weak policies weak policies can reach a goal state, but there is no guarantee due to the non-determinism of actions. in this section, we define the qualitative preference \(\texttt {always}\) for weak policies and present an \(\alpha \)-ctl formula for it. definition 13 (preference always for weak policies). let \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \) be a non-deterministic planning problem, \(\pi \) a weak policy for p, \(s_{\mathcal {d}[\pi ]}\) the execution structure induced by weak policy \(\pi \). the preference always(p) (\(p \in \mathbb {p}\)) for a weak policy is satisfied in \(s_{\mathcal {d}[\pi ]}\) iff there is some execution path \(\mathcal {p}_{\pi }\) where \(\forall k, 0 \le k \le i-1\), \(s_{k} \in \mathcal {p}_\pi \) and . definition 14 (specifying the always preference for weak policies in \(\alpha \) -ctl). let \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \) be a non-deterministic planning problem with signature \((\mathbb {p}, \mathbb {a})\) and always(p) \((p\in \mathbb {p})\) a preference over p. this preference for a weak policy can be expressed using the following \(\alpha -\)ctl formula:                                            . to prove that the \(\alpha \)-ctl formula specifies the always(p) preference for a weak solution \(\pi \) for a planning problem \(p = \langle \mathcal {d}, s_0, \varphi \rangle \) we must show that it is possible to obtain a submodel \(s_{\mathcal {d}[\pi ]} \subseteq \mathcal {d}\) induced by \(\pi \), where: (1) \(s_0 \in s_{\mathcal {d}[\pi ]}\); (2) there is an execution path \(\mathcal {p}_{\pi } \in s_{\mathcal {d}[\pi ]}\) that reaches the goal state , from the initial state \(s_0\), and \(\forall k, 0 \le k \le i-1\), \( s_{k} \in {\mathcal {p}_\pi }\) and . proof consider the non-deterministic planning problem \(p = <\mathcal {d}, s_0, \varphi > \). assume that . according to \(\alpha \)-ctl semantics, there is an action \(a \in \mathbb {a}\) such that, for a path \(\mathcal {p}_\pi \) started at \(s_0\) there is a state \(s_i\) (for \(s_i \ge 0\)) along this path such that and, for every \(0 \le k < i\), we have . then there is an execution path for a policy \(\pi \),i.e., that lead the goal \(\varphi \). consequently, there is an execution structure \(s_{\mathcal {d}[\pi ]}\) induced by the policy \(\pi \) (definition 5) such that \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]}\). thus, we have \(s_0 \in s_{\mathcal {d}[\pi ]}\), satisfying condition 1. since there is a \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]}\), which reaches the goal state , from the initial state \(s_0\) and \(\forall k, 0 \le k < i\), \( s_{k} \in {\mathcal {p}_\pi }\) we have , the condition 2 is satisfied. 4.2 specifying the always preference for strong policies strong policies guarantees that all sequences of states, obtained after execution of the policy, reach the goal. in this section we define how one can obtain a strong solution for a non-deterministic planning problem with \(\texttt {always}\) preference. definition 15 (preference always for strong policies). let be \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \) a non-deterministic planning problem with signature \((\mathbb {p}\), \(\mathbb {a})\), \(\pi \) a strong policy for p, \(s_{\mathcal {d}[\pi ]}\) the execution structure induced by \(\pi \). the always(p) \((p \in \mathbb {p})\) preference for a strong policy is satisfied in \(s_{\mathcal {d}[\pi ]}\) iff for all execution path \(\mathcal {p}_{\pi }\), we have and \( \forall k, 0 \, \le k\le \,i-1, s_{k} \in {\mathcal {p}_\pi }\) with . definition 16 (specifying the always preference for strong policies in \(\alpha \) -ctl). let be \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \)a non-deterministic planning problem with signature \((\mathbb {p}, \mathbb {a})\) and always(p) \((p\in \mathbb {p})\) a qualitative preference over p. this preference for a strong policy can be expressed by the following \(\alpha -\)ctl formula:                                            . in order to prove that \(\alpha \)-ctl formula specifies the always(p) preference for a strong policy for a non-deterministic planning problem \(p = \langle \mathcal {d}, s_0, \varphi \rangle \) we have to show that is possible to obtain a submodel \(s_{\mathcal {d}[\pi ]} \subseteq \mathcal {d}\) induced by the strong policy \(\pi \) that satisfies the following conditions: (1) \(s_0 \in s_{\mathcal {d}[\pi ]}\); (2) for all execution paths \(\mathcal {p}_{\pi } \in s_{\mathcal {d}[\pi ]}\), we have \(\forall k, 0 \le k \le i-1\), \( s_{k} \in {\mathcal {p}_\pi }\) and . proof consider a non-deterministic planning problem \(p = <\mathcal {d}, s_0, \varphi > \). assume that . according \(\alpha \)-ctl semantics, there is an action \(a \in \mathbb {a}\) such that, for all execution path \(\mathcal {p}_\pi \), starting in \(s_0\), there is a state \(s_i\) (\(s_i \ge 0\)) in each path such that and, for each \(0 \le k < i\), we have . thus, we can affirm that \(\mathcal {p}_\pi \) is a execution path. consequently, there is a execution structure \(s_{\mathcal {d}[\pi ]}\) induced by the policy \(\pi \) (definition 5) such that all \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]}\). thus, \(s_0 \in s_{\mathcal {d}[\pi ]}\), satisfying condition 1. since all \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]}\) reach a goal state , from initial state \(s_0\), and \(\forall k, 0 \le k < i\), \( s_{k} \in {\mathcal {p}_\pi }\) e , we have that the condition 2 é satisfied. 4.3 specifying the always preference for strong-cyclic policies strong cyclic policies guarantees that all sequence of states, obtained after policy execution, reach the goal under the assumption that its execution will eventually exit all existing cycles. in this section, we define how to specify the \(\texttt {always}\) preference for a strong-cyclic policy. definition 17 (preference always for strong-cyclic policies). let be \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \) a non-deterministic planning problem, where \(\mathcal {d}\) has signature \((\mathbb {p}\), \(\mathbb {a})\), \(\pi \) a strong cyclic policy for p, \(s_{\mathcal {d}[\pi ]}\) a execution structure induced by \(\pi \). the always(p) preference \((p \in \mathbb {p})\) for a strong cyclic policy is satisfied in \(s_{\mathcal {d}[\pi ]}\) iff for all execution paths \(\mathcal {p}_{\pi }\), we have \( \forall k, 0 \, \le k\le \,i-1, s_{k} \in {\mathcal {p}_{{\pi }_i}}\) e . furthermore, it is possible to reach a goal state from each \(s_k \in \mathcal {p}_{\pi }\). definition 18 (specifying the always preference for strong cyclic policies in \(\alpha \) -ctl). let be \(p=\langle \mathcal {d}, s_{0}, \varphi \rangle \) a non-deterministic planning problem, where \(\mathcal {d}\) has signature \((\mathbb {p}, \mathbb {a})\) and always(p) \((p\in \mathbb {p})\) a qualitative preference over p. this preference for a strong cyclic police can be expressed by the following \(\alpha -\)ctl formula:                                            . in oder to show that \(\alpha \)-ctl formula specify the always(p) preference for a strong solution for a non-deterministic planning problem \(p =\langle \mathcal {d},s_0,\varphi \rangle \) we have to show that it is possible to obtain a submodel \(s_{\mathcal {d}[\pi ]} \subseteq d\), such that: (1) \(s_0 \in s_{\mathcal {d}[\pi ]} \); (2) in all execution path \(\mathcal {p}_{\pi } \in s_{\mathcal {d}[\pi ]}\) we have \(\forall k, 0 \le k \le i-1\), \( s_{k} \in {\mathcal {p}_\pi }\) and and; (3) it is possible to exit the all cycles and reach the goal state from any state in execution path. proof consider a non-deterministic planning problem \(p = \langle \mathcal {d}, s_0, \varphi \rangle \). assume that . according \(\alpha \)-ctl semantic, there is an action \(a \in \mathbb {a}\) such that for some path \(\mathcal {p}_\pi \), starting in \(s_0\) there is a state \(s_i\) \((s_i \ge 0)\) in this path such that and, for each \(0 \le k < i\), we have . thus, each path \(\mathcal {p}_\pi \) is a execution path. consequently, there is a execution structure \(s_{\mathcal {d}[\pi ]}\), induced by \(\pi \), such that \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]} \). thus, we have \(s_0 \in s_{\mathcal {d}[\pi ]}\) satisfying the condition 1. since there is a \(\mathcal {p}_\pi \in s_{\mathcal {d}[\pi ]}\), reaching a goal state , from \(s_0\) and \(\forall k, 0 \le k < i\), \( s_{k} \in {\mathcal {p}_\pi }\) and , attending condition 2. furthermore, according \(\alpha \)-ctl semantic, there is an action \(a \in \mathbb {a}\) such that, in all states \(s_i\) (\(i \ge 0\)) in all paths, starting in \(s_0\), we can reach a goal state by visiting only states where property p is true. thus, the condition 3 is also satisfied. 5 experimental analysis our preliminary experimental analysis aims to verify the feasibility of finding policies that satisfy the qualitative preferences specification. for this, we use the deterministic rover domain with preferences from \(5^{th}\) international planning competition, modified to include non-deterministic actions (to the best of our knowledge, there are currently no benchmark domain with these characteristics). rovers models the mission of planetary exploration on mars, where rovers are equipped with devices including cameras, sample collectors, and data transmitters. the goal of the rovers is to explore the planet surface and send the collected data to a space station. the qualitative preference always entails preventing the rover from sampling soil and rock at a specific waypoint. we use the planner pactl [28] to obtain the set of states satisfying the \(\alpha \)-ctl formulas specifying the qualitative preference \(\texttt {always}\) and also the quality of the policy. the experiments were performed on a xeon quad core server with 16gb of ram. a timeout of 45 min was set for solving each instance. table 1 presents, for each instance and qualitative preference specified, the execution time and number of steps to obtain a policy. the symbol “-” indicates that timeout was reached. the instances have different levels of difficulty, which are characterized by factors such as the number of regions that need to be traversed, the number of goals to be achieved, and the number of rovers involved. the planner found strong, weak, and strong-cyclic policies for all instances from 1 to 8 (except for instance 6). starting with instance 9, the planner was able to find solutions to problems 14 and 19. in cases where the planner fails to find a solution, it is difficult to distinguish either there is no solution to the problem, or the planner was unable to find the solution without exceed the timeout. table 1. time and number of steps to compute policies with preferences.full size table 6 conclusion and future work in this work, we propose using the \(\alpha \)-ctl temporal logic to specify the always qualitative preference in non-deterministic planning problems. in order to obtain weak, strong, or strong cyclic policies for non-deterministic problems with qualitative preferences, we employ planning as \(\alpha \)-ctl model checking algorithms. to demonstrate the suitability of these algorithms for solving non-deterministic problems with qualitative preferences, we modify the existing deterministic rover domain with preferences by incorporating non-deterministic effects on the actions. the instances of this modified domain were used in our experiments, which aimed to show that planning as model checking algorithms can effectively provide solutions for non-deterministic problems with qualitative preferences while considering the policy quality. to the best of our knowledge, this is the first work that addresses non-deterministic problems with preferences while incorporating a preference over the policy quality. as future work, we aim to investigate the suitability of the \(\alpha \)-ctl logic in expressing others types of qualitative preferences and, consequently, if planning as model checking algorithms are appropriated to solve non-deterministic planning problems with these others kinds of preferences. furthermore, we aim to conduct experiments using different benchmark domains. references akers, s.b.: binary decision diagrams. ieee trans. comput. 27(06), 509–516 (1978) article  math  google scholar  baier, j.a., bacchus, f., mcilraith, s.a.: a heuristic search approach to planning with temporally extended preferences. in: proceedings of the 20th international joint conference on artifical intelligence, pp. 1808–1815 (2007) google scholar  baier, s., mcilraith, s.a.: htn planning with preferences. in: 21st int. joint conf. on artificial intelligence, pp. 1790–1797 (2009) google scholar  bonet, b., de giacomo, g., geffner, h., rubin, s.: generalized planning: non-deterministic abstractions and trajectory constraints. arxiv preprint arxiv:1909.12135 [s.l] (2019) camacho, a., triantafillou, e., muise, c., baier, j.a., mcilraith, s.a.: non-deterministic planning with temporally extended goals: ltl over finite and infinite traces. in: thirty-first aaai conference on artificial intelligence (2017) google scholar  cimatti, a., pistore, m., roveri, m., traverso, p.: weak, strong, and strong cyclic planning via symbolic model checking. artif. intell. 147(1–2), 35–84 (2003) article  mathscinet  math  google scholar  edmund, m., clarke, j., grumberg, o., peled, d.a.: model checking. mit press. [s.l] p. 314 (1999) google scholar  gerevini, a.e., haslum, p., long, d., saetti, a., dimopoulos, y.: deterministic planning in the fifth international planning competition: pddl3 and experimental evaluation of the planners. artif. intell. 173(5–6), 619–668 (2009) article  mathscinet  math  google scholar  ghallab, m., nau, d., traverso, p.: automated planning: theory and practice. elsevier (2004) google scholar  hoffmann, j.: ff: the fast-forward planning system. ai mag. 22(3), 57–57 (2001) google scholar  hoffmann, j.: everything you always wanted to know about planning. in: annual conference on artificial intelligence, pp. 1–13. springer (2011) google scholar  hsu, c.w., wah, b.w., huang, r., chen, y.: constraint partitioning for solving planning problems with trajectory constraints and goal preferences. in: ijcai, pp. 1924–1929 (2007) google scholar  kautz, h., selman, b.: unifying sat-based and graph-based planning. in: ijcai, vol. 99, pp. 318–325 (1999) google scholar  kim, j., banks, c.j., shah, j.a.: collaborative planning with encoding of users’ high-level strategies. in: thirty-first aaai conference on artificial intelligence (2017) google scholar  kindler, e.: safety and liveness properties: a survey. bull. eur. assoc. theor. comput. sci. 53(268–272), 30 (1994) google scholar  lamport, l.: proving the correctness of multiprocess programs. ieee trans. software eng. 2, 125–143 (1977) article  mathscinet  math  google scholar  de menezes, m.v., de barros, l.n., do lago pereira, s.: symbolic regression for non-deterministic actions (2014) google scholar  menezes, maria viviane de: mudanças em problemas de planejamento sem solução. 2014. 127 f. ph.d. thesis, universidade de são paulo, são paulo (2014) google scholar  muise, c.j., mcilraith, s.a., beck, c.: improved non-deterministic planning by exploiting state relevance. in: twenty-second international conference on automated planning and scheduling (2012) google scholar  percassi, f., gerevini, a.e.: on compiling away pddl3 soft trajectory constraints without using automata. in: proceedings of the international conference on automated planning and scheduling, vol. 29, pp. 320–328 (2019) google scholar  pereira, s.d.l.: planejamento sob incerteza para metas de alcancabilidade estendidas. ph.d. thesis, universidade de são paulo, são paulo (2007) google scholar  pereira, s.l., de barros, l.n.: a logic-based agent that plans for extended reachability goals. auton. agent. multi-agent syst. 16(3), 327–344 (2008) article  google scholar  piterman, n., pnueli, a.: temporal logic and fair discrete systems. in: handbook of model checking, pp. 27–73 (2018) google scholar  rintanen, j.: regression for classical and nondeterministic planning. in: ecai 2008, pp. 568–572. ios press (2008) google scholar  santhanam, g.r., basu, s., honavar, v.: representing and reasoning with qualitative preferences: tools and applications. synthesis lectures artif. intell. mach. learn. 10(1), 1–154 (2016) article  math  google scholar  santos, r.m.d.: especificação de preferências de planos usando metas estendidas na lógica alpha-ctl (2019) google scholar  santos, v.b.d., barros, l.n.d., pereira, s.d.l., menezes, m.v.d.: symbolic fond planning for temporally extended goals. in: workshop on knowledge engineering for planning and scheduling (2022) google scholar  dos santos, v.m.b., de barros, l.n., de menezes, m.v.: symbolic planning for strong-cyclic policies. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 168–173. ieee (2019) google scholar  download references author information authors and affiliations instituto federal do piauí, teresina, brazil warlles carlos costa machado universidade de são paulo, são paulo, brazil viviane bonadia dos santos & leliane nunes de barros universidade federal do ceará campus quixadá, quixadá, brazil maria viviane de menezes authors warlles carlos costa machadoview author publications search author on:pubmed google scholar viviane bonadia dos santosview author publications search author on:pubmed google scholar leliane nunes de barrosview author publications search author on:pubmed google scholar maria viviane de menezesview author publications search author on:pubmed google scholar corresponding author correspondence to warlles carlos costa machado . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper machado, w.c.c., dos santos, v.b., de barros, l.n., de menezes, m.v. (2023). specifying preferences over policies using branching time temporal logic. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_9 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords non-deterministic planning preferences temporal logic publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature regulation and ethics of facial recognition systems: an analysis of cases in the court of appeal in the state of são paulo | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper regulation and ethics of facial recognition systems: an analysis of cases in the court of appeal in the state of são paulo conference paper first online: 12 october 2023 pp 18–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) regulation and ethics of facial recognition systems: an analysis of cases in the court of appeal in the state of são paulo download book pdf download book epub cristina godoy b. de oliveira9, otávio de paula albuquerque10, emily liene belotti9, isabella ferreira lopes11, rodrigo brandão de a. silva12 & … glauco arbix12  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 581 accesses abstract context: the use of artificial intelligence (ai) in various sectors of the economy is already a reality in brazil. consequently, since 2019, the number of cases in the judiciary involving ai has increased. cases involving facial recognition systems (frs) for contracting bank credit are increasing annually, so it is necessary to analyze how the judiciary handles the issues. problem: why is the são paulo court of appeal ruling in favor of banks in all cases involving taking out credit through facial recognition technology? methodology and methods: data were collected and processed using automated computer programs. the qualitative analysis used the analytical, comparative and monographic methods. results: the court of appeal of são paulo considers it difficult to deceive an ai system, therefore, the burden of proof is on the author, even if there is a consumer relationship. that is, the decisions are contrary to the general rule of the code of consumer protection in brazil, which consists of reversing the burden of proof in consumer relations when one of the parties is underprivileged. contributions and solutions: the research points to the path of jurisprudence in cases involving the contracting of credit through frs, and the judiciary is deciding against the bank’s customers, dispensing with the production of evidence by the banking sector. therefore, it is necessary to alert the national council of justice and the central bank regarding this situation so that it is disciplined adequately since the frs is fallible and does not guarantee the absence of fraud. this work was carried out at the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp grant #2019/07665-4) and by the ibm corporation. this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others trust in facial recognition systems: a perspective from the users chapter © 2023 implementation of artificial intelligence and chatbot for the enhancement of new age banking systems: a systematic review chapter © 2025 fear of missing out and the future of facial recognition payments in the blue economy chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biometrics courts law and religion philosophy of artificial intelligence privacy artificial intelligence 1 introduction the regulation of artificial intelligence (ai) has garnered the attention of various stakeholders worldwide. in brazil, the subject gained visibility between 2019 and 2021 when three bills concerning the regulation of ai began to be discussed in the national congress. in 2022, the discussion on the regulation of ai in the country became even more intense due to the creation, by the federal senate, of a commission of jurists responsible for gathering information to create a regulatory framework for ai. amidst these topics, many researchers have overlooked the fact that the legal regulation of ai depends, among other elements, on decisions by the judiciary regarding disputes involving the different applications of the technology in question, rather than solely relying on general laws that directly deal with regulation. in addition to this gap in the discussions on the legal regulation of ai, there is another gap in the discussions on the use of ai by the brazilian public power: it is still not completely clear to researchers how ai applications have been used by national public offices. in the case of the brazilian judiciary, salomão et al. [22] attempted to map ai use cases. investigations such as this are valuable but do not provide us with a clear understanding of how this power actually deals with ai in concrete situations where the technology in question may have caused objective harm to specific individuals. to partially address these issues, our research question is: when analyzing cases involving facial recognition systems, does the judiciary of the state of são paulo evaluate the transparency of these systems and their cybersecurity level against fraud? guided by this question, we applied web scraping techniques to collect data from the court of justice web portal and analyzed the arguments presented by the judiciary power to justify its decisions involving the use of facial recognition (fr) systems for credit loan contracts. we also assessed how clearly the use of such technologies is understood by both the judiciary power and the contracting party of the credit. we chose to study facial recognition systems because, as daly [6] points out, they are an example of an ai application that tangibly affects individuals’ lives, allowing them to conceive the technology in a more concrete manner, even though it may sometimes seem abstract. the choice of the state of são paulo is justified by its status as the most prosperous state in the country, potentially making it the state with the most frequent use of ai applications. finally, we discuss the challenges of transparency in ai, as we believe that ethical ai is impossible without it. in addition to this introductory section, the article comprises three additional sections. section 2 presents the methodology used to scrape data on the decisions of the judiciary in the state of são paulo and includes the methodology employed to analyze the collected data. section 3 presents and discusses the obtained results, highlighting the dependence of facial recognition systems on sensitive personal data and noting that the judiciary of the state of são paulo has given little importance to the requirement of free and informed consent for the treatment of such data. section 4 concludes the article. 2 materials and methods an exploratory research was conducted to investigate the factors that influence the decision-making of the state of são paulo’s court of appeal regarding subjects related to artificial intelligence [10]. the study was based on data collected from the decisions stored in the portal of services (e-saj) of the court of appeal portalfootnote 1 using a web scraping technique, which consists of extracting online data to obtain content of interest in a systematic and structured manner [11]. in this research, most of the data collection and treatment procedures were automated using computer programs. the technologies used can be defined into two categories: (1) web scraping tools and (2) data treatment coding. 2.1 web scraping tools and frameworks to scrape the data from the court of appeal processes, an application programming interface (api) named tjsp was used. it is a community open-source application published on github [8]. the main goals of the tjsp tool are to collect, organize, and structure data from process judgments of the first and second instances of são paulo’s court of appeal, using different methods for various data extraction proposals. this tool was developed using the programming language r, and the integrated development environment (ide) called rstudio was used to manipulate the code and instantiate the api for our research, defining the terms and authenticity properties. the api was used to extract data from the court of appeal web portal proceedings by searching for all decision documents containing the term “artificial intelligence” and was improved to deliver the data in a structured and organized file. after defining the search terms, the following steps were taken: (1) download the corresponding html pages with all the processes from the são paulo’s state court of appeal web portal; (2) based on the obtained html files, identify the information about the processes, returning information such as class, subject, rapporteur, court district, judging body, date of judgment, date of publication, case, summary, and code of the decision; (3) store the retrieved information in a data frame, where each column represents specific information and each row represents a process; (4) convert the data frame into a spreadsheet for qualitative and quantitative analysis. the collected decisions in this document refer to 190 (one hundred ninety) matching results, comprising data from 2012 to 2023, related to the most general terms in the ai context: “inteligência artificial” and “artificial intelligence”; “aprendizado de máquina” and “machine learning”; “aprendizado profundo” and “deep learning”. the search included the terms searched in portuguese and english, respectively, and in singular and plural. these results correspond to the first decision document registered on the portal on june 24th, 2010, to the latest scraping round performed on april 20th, 2023. 2.2 data treatment between the information retrieved from the decisions, there are different types of data, including integral, date-time, and character strings. given the goal of our study and to help facilitate the qualitative analysis, we applied different data treatment techniquesfootnote 2, divided into preand post-processing approaches. these approaches were applied to the “jurisprudence” column, which contains the decision text. cleaning techniques were developed using the python programming language and deployed in the google colab ide to take advantage of the performance benefits offered by the platform and make them available to the research team. data pre-processing. the pre-processing and cleaning of data are the most critical and time-consuming parts of a web scraping project. to obtain a good visualization of the data collection, it usually requires multiple iterations of cleaning, transforming, and visualizing the data before it is in a suitable form [27]. there are different techniques, ranging from the simplest to the most advanced, to preprocess a data collection, i.e., prepare, clean, and transform data from an initial form to a structured and context-specific form. in our research, we applied data mining techniques to clean and search for repetitive or redundant parts in the “jurisprudence” column content to improve the qualitative analysis. the decision content follows a standard structure containing repetitive or redundant sections. these sections include: (1) titles, headers, references, and terms that are identical for all documents, such as “judiciary power,” “court of appeal of the state of são paulo,” “electronic signature,” “decision,” among others; (2) appeal and vote numbers, which differ for each decision but will not be used in the analysis. these numbers are identified by being preceded by terms such as “appeal n,” “civil appeal n,” “appeal.,” “record:,” “vote n,” and others; and (3) document page identifiers, such as current page numbers, referenced section page numbers, or total page numbers of the decisions. these identifiers are preceded by sections such as “(fls.,” “(fl.” using the previously mentioned cleaning techniques, these expressions were removed from the results spreadsheets. data post-processing. once the data cleaning stage is complete, we move on to post-processing the information. in order to derive insights from the data, it is imperative that the data is presented in a format that is readily analyzable. this requires the data to be not only clean and consistent but also well-structured, aiding in addressing inconsistencies in scraped data [14]. this stage involves structuring the collected data according to the research context, or in other words, highlighting and grouping the most important parts of each decision to streamline and aid in the accuracy of manual work done by analysts. in the context of data post-processing, automated procedures were developed to improve the qualitative analysis by identifying relevant terms/contents that could make it easier to discover the topic addressed in each process and determine whether or not it should be part of the ongoing research analysis scope. (1) from each decision, all paragraphs in which the searched term appears were selected and added in a new column; (2) it was identified that some decisions refer to others using the number of the respective process. thus, for each decision, references to its process number in other documents were searched, and this information was also stored in a new column of the results spreadsheet. another procedure developed was the grouping of decisions based on rapporteurs, also using python modules for data manipulation, this improve made it more practical and efficient to perform a qualitative analysis of the results in a critical and embracing way. also, aiming to continuing collecting data, to improve the useful information, and to make the research dynamic by keeping the results up to date, a periodicity of repetition of the scraping procedures was established. for this, other methods were developed in python programs. they are: (1) a results counting mechanism, which compares the number of terms with corresponding matches and the number of matches for each term, comparing these numbers with the respective ones from the last scraping performed. these data are inserted into a new spreadsheet and updated every month; (2) in each results spreadsheet, the new scraped processes, compared to the ones from previous scraping, are marked with the character ’*’, so that analysts can easily identify new results, avoiding redundancies. 2.3 qualitative analysis method the main procedural methods adopted were analytical, comparative, and, especially, monographic. this was revealed through a longitudinal study of the specificities of each case, investigating the factors that influenced decision-making in each unique event, based on sectoral analysis and making generalizations about similar events. this article examined all decisions of the state of são paulo court of appeal that contain the following six terms: “inteligência artificial” and “artificial intelligence,” “aprendizado de máquina” and “machine learning,” and “aprendizado profundo” and “deep learning.” during the search, 190 items were retrieved, out of which 41 were deeply analyzed for the purpose of this ongoing research, totaling 21.58% of the results. these analyses focused specifically on banking matters, with particular attention to disputes involving the validity of digital payroll loan contracts carried out by facial biometrics. the first decision analyzed was issued and published on august 30, 2021, and the last decision (according to data scraping conducted in april 2023) was on april 20, 2023. using the monographic method, this research will present the standard arguments issued by each of the rapporteurs in the decisions referred to, justifying the validity of the loan contracts. firstly, the existence of a legal relationship between the bank and the consumer will be analyzed, as well as the alleged lack of voluntary consent. this will provide a generalized framework of the views adopted by judges regarding facial recognition technology (frt) and its use in digital banking contracts in brazil. finally, the analysis of the appeal court’s decisions was divided into 13 subcategories: (i) case number; (ii) class (type of appeal analyzed in the decision); (iii) rapporteur; (iv) judicial district; (v) judging body; (vi) subject; (vii) trial date; (viii) publication date; (ix) disputing parties; (x) reasoning; (xi) final decision; (xii) context of the use of the term “inteligência artificial” (a.i.); (xiii) jurisprudence. 3 results and discussion from the web portal of the state of são paulo’s court of justice, 190 decisions covering a span of 12 years (from 2012 to 2023) were analyzed. the results showed that 177 decisions involved the term “inteligência artificial,” 4 decisions referred to the term “artificial intelligence” (from october 2022 to april 2023), 3 decisions related to “aprendizado de máquina” (from august 2022 to february 2023), and 6 decisions mentioned “machine learning” (from september 2018 to november 2022). however, no decisions were found for the terms “deep learning” or “aprendizado profundo” through web scraping on the appellate court’s website. figure 1 illustrates the exponential growth trend in the number of decisions containing these six terms. the graph covers the period from 2012 to 2023, showing relative stability until 2019 and a significant increase from 2020 onwards. several factors contribute to this trend, including: (1) the enactment of the “lei geral de proteção de dados pessoais” (lgpd), law no. 13,709, on august 14, 2018, which came into force in brazil in september 2020. this has led to an increase in disputes related to the implementation of ai systems in various sectors of society, such as banking, service provision through applications, and compliance with the new standards. (2) as mentioned in the “qualitative analysis method” section, 37.3% of the “inteligência artificial” results are related to banking matters, including disputes involving credit card operations, atm fraud, nighttime bank robberies with threats for consecutive withdrawals, phone or sms scams where scammers pretend to be ai virtual assistants to obtain passwords/personal data, and payroll loan cases involving facial recognition (the focus of this article). the occurrence of facial recognition-related cases is also expected to grow rapidly, as depicted in fig. 1. fig. 1. the number of occurrences related to 6 term searched “inteligência artificial”,“artificial intelligence”, “aprendizado de máquina”, “machine learning”, “aprendizado profundo” and “deep learning”, between 2012 and 2023. full size image the apparent drop in 2023 does not indicate an actual decrease in the number of decisions. it is important to note that this research only compiled data up until april of that year. the growth trend remains high, as the number of decisions analyzed up until april 2023 alone has already more than doubled the entire year of 2019. our results will focus solely on decisions related to the term “artificial intelligence,” totaling 41 in number. the other terms do not involve disputes regarding the validity of digital payroll loan contracts carried out through facial biometrics. in fact, they do not pertain to banking matters at all. these decisions cover topics such as “health plans,” “higher education,” “contractual readjustment,” “business management,” “indemnity for moral damages,” and “provision of transport application services,” among others. none of the appellants in these cases are banking parties. out of the 41 decisions analyzed, in 40 of them, it was determined that a legal relationship existed between the contracting bank and the customer. therefore, the validity of the electronic payroll loan agreement was confirmed by the judges who recognized the legitimacy of the debt and the legality of the bank. furthermore, the court of appeal acknowledged that there is no unenforceability of the loan refinancing agreement. in cases where the bank files an appeal, it is granted in terms of the contract’s validity and the absence of unenforceability for material and moral damages (when claimed by the plaintiffs). when the judgment is unfavorable to the plaintiff, their appeal is dismissed. the existence of the loan and the allegation of bad faith litigation (when claimed) are recognized or partially dismissed (removing the bad faith litigation while maintaining the validity of the loan). figure 2, it is possible to point out that the exponential growth trend in the occurrence of payroll loan cases involving facial biometrics over time. the graph covers the period from 2020 to 2023. however, the registered cases in the facial biometrics contexts emerge from 2021. based on these data, it can be inferred that such an event occurs in this way, since the regulation of facial recognition technology (frt) is still quite incipient in the country, both in terms of the legislative framework and its implementation. in the lgpd, for example, there is not even a mention of the term “facial recognition” or “facial biometrics”, restricting itself only to classifying in its art. 5, item ii, as “sensitive” biometric data (relating to health, sex life, genetic data, religious/political/philosophical group, etc.). in brazil, there are only three bills dealing with frt: pl 2392/22, which prohibits the use of frt for identification purposes in the public and private sectors without a prior report on the impact on people’s privacy; pl 3069/22, which regulates the use of automated facial recognition by public security forces in criminal investigations or administrative procedures; and pl 2338/23, which regulates artificial intelligence and allows the use of facial recognition systems for public security purposes in public spaces only with legal and judicial authorizations. therefore, deprived of a normative framework that minimally regulates not only the frt implementation procedures in the country but also the rights and duties of consumers and banks regarding such technology, judges end up issuing judicial decisions that could be more grounded, complete, and divergent. fig. 2. the number of occurrences related to facial recognition between 2020 and 2023. full size image meanwhile, the four main arguments presented by judges to justify the existence of the digital payroll loan agreement involving facial recognition technology (frt) are listed below: (i) non-challenge of the documents presented by the bank. the plaintiff did not challenge his photo displayed by the bank (defendant) nor the receipts for withdrawing the money used by the plaintiff, nor did he require the production of an expert report at the appropriate procedural time to prove possible tampering with the evidence presented by the opposing party. (ii) article 3 of normative instruction no. 28/2008 of the inss/pres allows the contracting of a loan to take place electronically. therefore, the signature is not a requirement for the existence or even the validity of the contract, as it would only serve to confirm the expression of will. this is indeed essential and indispensable and was granted by the author when he agreed to collect his facial biometrics for the specific purpose of formalizing the contract. therefore, a contract carried out electronically dispenses with the formal rigors of a standard contract, precisely to the extent that the electronic form can be carried out anywhere, as long as the consumer consents. (iii) in consumer relations, according to the brazilian consumer protection code (cdc), there is a reversal of the burden of proof: presentation, by the bank, of a bank credit note (signed by the plaintiff through the capture of facial biometrics through ’selfie’, similar to its id photo); proof of transfer of the contracted amount made to the applicant’s account, confirmation of address at the same domicile declared by the plaintiff, digital signature, presentation of original documents, geolocation, digital contract indicating that it was signed by mobile application, telephone model cell phone used for hiring, in addition to time and ip address. (iv) all decisions are based on a judgment that considered: “as for electronic hiring and facial biometrics, i understand that such a form of hiring is perfectly appropriate, given the technological resources available and widely used [...] “the electronic medium is valid. falsifying a signature is something quite common, unfortunately, but falsifying a person’s face to ”fool” the artificial intelligence, even if it is possible, requires more sophisticated fraud, and the burden of proving the occurrence of this fraud was on the author, even if it were it is a consumer relationship, since it is a constitutive fact of the right alleged by itself.” 3.1 consent payroll loan contracts have a history of problems in brazil. in 2020, complaints about such credit doubled [3], totaling 20,564 records, half of the total in 2019 (39,012 records), according to records available in the government database. likewise, the consumer protection and defense foundation (procon) in são paulo recorded a 50% increase in complaints against banking institutions [21]. idec brazilian institute of consumer protection evaluated more than 300 reports of complaints sent on the consumer portal, which show, in general, the easy access of banks to confidential bank data of consumers with the inss (national institute of social security), such as confirmation of bank operations without the direct involvement of the person responsible for the account or the release of the payroll loan without the consumer’s consent [4]. such data can be explained by a conjuncture of phenomena, among them, the “esteira invertida”, in which fraudsters—usually bank correspondentsfootnote 3 or financial market operators—deposit an uncontracted loan in the account of the retiree or pensioner, without their authorization, to receive up to 6% of the transaction amount as commission. the fraudster voluntarily chooses victims who have previously carried out payroll loans, reusing the personal data of these consumers. thus, in a situation of vulnerability, such retirees often end up using the money without knowing its origin. therefore, the argument used by the reporting judges that the plaintiff did not challenge the withdrawal receipts presented by the bank does not hold up, as it is entirely plausible that the retiree, a hypo sufficient consumer in relation to the financial institution, really believed that the money in his account was his, not knowing that he had been the victim of fraudulent activity. the payroll law determines that it can only be deducted from the benefit “when expressly authorized by the beneficiary” (law no. 8213, art. 5, inc.v). if there is no express authorization, the discount will be undue. likewise, article 39, item iii of the consumer defense code expressly prohibits the supplier of products or services from sending to the consumer, without prior request, any product or taking advantage of the condition of ignorance, social situation, age or health to sell its products or services (article 39, item iv). such devices refer to the “consent” provided for in the lgpd (brazilian general law for the protection of personal data, law no. 13.709/2018), which is inspired by the general data protection regulation of the european union (gdpr): natural person holder of the data, or by their legal guardian, and must be expressed clearly and unequivocally, in writing or not”. consent must be informed, that is, before obtaining it, it is essential to provide information to data subjects so that they can understand what they agree with, for example: the identity of the person responsible for processing the data, the purpose of each of the processing operations for which consent is sought, what types of data will be analyzed, etc. [26]. article 4, \(\textrm{n}^{{\underline{\text {o}}}}\) 11 of the gdpr clearly states that consent requires an “unequivocal positive statement or act” from the data subject, which means that it has to be given using positive action or explicit statement, mainly in situations where there is a severe risk to data protection and, therefore, it is appropriate to have a high level of security, such as biometric data and/or financial information both used in payroll loan contracts. therefore, such an explicit manifestation may be through contracts or electronic forms, but it is necessary to stick to adhesive contracts with pre-signed adhesion conditions, in which the data subject’s intervention is necessary to prevent acceptance. thus, the argument of the reporting judge that “contracts executed in electronic form dispense with the formal rigors of a standard contract, precisely to the extent that the electronic form can be carried out anywhere” does not hold, since the generalized acceptance of general conditions does not match the aforementioned definition of unequivocal, explicit and informed consent. likewise, even though article 107 of the brazilian civil code (ccb) provides for freedom of form for contracting, using the electronic contract which dispenses with a traditional signature, any new technology used must prove the explicit demonstration of consent, guaranteeing the reliability of the protection of consumer data, ensuring unequivocal fraud prevention mechanisms—deficient characteristics in the use of facial biometrics, presented below. 3.2 facial biometrics the general data protection regulation (gdpr) clarifies what biometric data is in its art. 4th [7]: article 4 definitions for the purposes of this regulation, it is understood by: [...] “biometric data”, personal data resulting from a specific technical treatment relating to the physical, physiological or behavioral characteristics of a natural person that allows or confirm the unique identification of that natural person, namely facial images or dactyloscopic data; in the brazilian legal framework, decree no. 10,046/2019, which “provides for governance in data sharing within the scope of the federal public administration and establishes the base citizen register and the central data governance committee” defines biometric attributes [1] as: art. 2nd for the purposes of this decree, it is considered: [...] ii biometric attributes measurable biological and behavioral characteristics of the natural person that can be collected for automated recognition, such as the palm of the hand, fingerprints, retina or iris of the eyes, face shape, voice and way of walking. biometric processing corresponds to the processing of biometric data. it can “be referred to interchangeably as recognition, identification, authentication, detection or other related terms, as well as (often opaque) ways of collecting and storing biometric data, even if the data is not processed immediately” [19]: generally speaking, a facial recognition system operates by using biometrics to capture a person’s facial features, comparing the information to a database. they usually work through similar steps: a image of the person’s face is captured from a photo or video; then, the facial recognition software using a machine learning algorithm, verifies a series of facial characteristics such as the distance between the eyes, the curvature of the mouth, etc., to extract “facial signatures” based on the identified facial characteristics patterns. this signature would then be compared to a database of pre-registered faces. finally, there is the determination, in which the verification of the analyzed face can occur [16]. from a technical point of view, facial recognition technology (frt) is a subcategory within artificial intelligence (ai). frt is less accurate than, for example, fingerprinting, mainly when used in real-time or in large databases [9]. several factors influence the probability/accuracy of a match, such as: the quality of image, represented by the physical characteristics, external accessories usage, equipment and format usage, and environment conditions; quality of the dataset, size and proportion of the data; and the quality of the model by the choose of threshold, parameters, and fine-tuning [20, 23]. determining the accuracy level of facial recognition can be challenging and result in both false positives (a face is mistaken for another in a database where it shouldn’t be) and false negatives (a face doesn’t match in a database in which it should be registered). based on machine learning nature, the algorithms will never bring a definitive result, only probabilities. this means that identical twins can be misidentified. in addition, cutouts of ethnic, gender or age groups can also generate false results. debora lupton [15], based on the studies of donna haraway [13] and annemarie mol [18], explains that digital data is never a “tabula rasa” (superficial), but must be understood and experienced as something generated by structures, through processes of different characteristics, but above all, perhaps, social and cultural. artificial intelligence technology is not created”by itself”, nor is it devoid of the structural constraints that shape it. it was found that, on average, only 10 to 20% of the group responsible for developing artificial intelligence technologies is made up of women [25]—the data refer to the largest technology companies in the usa, a global hub in the development of this field. thus, darker people, ethnic groups, women, transgender people and people with physical disabilities are more likely to incur false negatives or positives due to less inaccuracy than other technologies (fingerprinting) and the low diversification of facial databases. furthermore, the quality of facial recognition (fr) models can be measured by their error rate, that is, the number of times such models fail to compare images of person’s face. known technically as false non-match rate, the error rate varies according to the types of images presented to fr algorithms. it tends to be smaller when the comparison is based on images taken in controlled environments, where variables such as the position of the face and the incidence of light can be manipulated. international visa photos are an example of this type of image. compared to images obtained in real situations, the error rate of fr systems tends to be higher. in this context, it is worth mentioning that, “as of 2022, the top-performing models on all of the [. . . ] datasets [that integrate the national institute of standards and technology (nist)’s face recognition vendor test], with the exception of wild photos, each posted an error rate below 1%, and as low as a 0.06% error rate on the visa photos dataset” [17]. in this way, the justification given by the judge rapporteur for recognizing the validity of the frt is highly generic and devoid of foundation. faking the frt does not necessarily require more sophisticated fraud, and automatic erroneous matching may occur depending on the accuracy level of a specific recognition system. not limited to occasional errors, like any other computational technologies, biometric data is also at risk of violation and misuse, either by outsiders (hackers) or insiders, employees who may use the pre-collected data for their own use [5]. an example of insiders is the banking correspondents of payroll loans, which can improperly renew the loans or activate them without the consumer’s authorization, or even the bank attendants themselves who use the wide range of personal data of consumers stored by the institution to defraud loans. in brazil, studies such as the one by nist are lacking. therefore, we need to find out what are, in general, the error rates of the systems used in the country. for this reason, brazilian banks must be able not only to inform the public about the error rates of the systems they develop and use, but also whether they are as low as those identified by the us agency. brazilian banks must also be able to inform if the error rates of their systems are equally distributed among different social groups or if they are concentrated in specific ones. this requirement is necessary because recent studies have shown that different fr systems make more errors when confronted with the faces of non-white people [2, 12, 24]. finally, and most importantly, any institution that decides to use fr systems must have clear protocols for using the outputs generated by the technology. in the case of taking out credit, for example, banks should investigate with users whether they are aware that automated facial recognition is a crucial step in the process. 4 conclusions and future work in the context of payroll loans, it is clear that the motivation and grounds for decisions are very generic, incomplete or even erroneous. the fact that the judge uses facial recognition as an infallible means of proof of hiring is wrong, since in addition to several factors influencing its accuracy, the algorithms do not bring a definitive result, based on alternative solutions for situations that do not appear or are not frequent in their databases. likewise, presenting the digital contract as evidence indicating that it was signed by mobile application, cell phone model used for contracting, in addition to time and ip address, is not sufficient to demonstrate the existence of free consent. the geolocation mechanism, for example, is susceptible to fraudulent mechanisms both for the one based on ip (vpns, proxies, tor, tunneling) and for the one based on gps (through fake gps applications). as for the electronic signature, despite art. 107 of the brazilian civil code provides for free forms of contract, an electronic contract presupposes the same free, explicit and informed consent provided for in the lgpd and the gdpr. therefore, when analyzing disputes involving hiring through facial recognition, it is recommended that the judge try to base his decisions on the greatest possible number of consolidated references, both in the area of science and technology, as well as in the area of sociology and public policies, given that this is a field still undergoing implementation and adaptation in the most diverse instances of society. from this, it is noted that it is still necessary to improve the standards that encompass facial recognition technology. therefore, within the scope of financial institutions, it is recommended that the brazilian central bank regulate this matter and define the measures that other banks must implement to guarantee the reliability, security, transparency, responsibility, diversity and effectiveness of the frt systems, in addition to defining essential minimum conditions for the use of such technology in its service contracts, guaranteeing unequivocal, explicit and informed consent to its customers/consumers regarding the new facial biometrics technologies. finally, the cnj (national court of justice), according to article 102 of resolution 67 of 2009, may edit normative acts, resolutions, ordinances, instructions, administrative statements and recommendations. additionally, resolutions and statements will have binding force on the judiciary power once approved by the majority of the cnj plenary. thus, it is recommended that the cnj pass a resolution that governs the criteria to be met by magistrates for the production of evidence in cases involving ai systems. transparency must be guaranteed in all cases, and it is necessary not to assume that the ai is infallible. in any case related to ai, the absence of production of evidence concerning to the existence of fraud should be accepted and the review of the decisions made by the ai should be enforced. if this situation in the judiciary continues, we will watch the weakening of due process of law and human dignity. this research carried out the web scrap data using umbrella terms from the field of artificial intelligence both in english and portuguese such as “artificial intelligence”, “inteligência artificial”, “machine learning”, “aprendizado de máquina”, “deep learning” and “aprendizado profundo”, which may have limited the amount of returned decisions. as future work, we will analyze a list of terms related to applied artificial intelligence and its subareas, and their applications. notes 1.são paulo’s state court of justice web portal jurisprudence consultations full consultation: https://esaj.tjsp.jus.br/cjsg/consultacompleta.do?f=1. 2.project coding and data treatment methods applied to this work: https://github.com/isabellalopes-usp/tjsp_data_treatment. 3.individuals who acquire certification to work in the sale of banking services, with emphasis on credit operations and credit card granting. references brasil: decreto no̱ 10.046, de 9 de outubro de 2019: dispõe sobre a governança no compartilhamento de dados no âmbito da administração pública federal e institui o cadastro base do cidadão e o comitê central de governança de dados. diário oficial da república federativa do brasil (2019). https://www.planalto.gov.br/ccivil_03/_ato2019-2022/2019/decreto/d10046.htm buolamwini, j., gebru, t.: gender shades: intersectional accuracy disparities in commercial gender classification. in: conference on fairness, accountability and transparency, pp. 77–91. pmlr (2018) google scholar  camargo, c.h., et al.: um ano de pandemia da covid-19: diversidade genética do sars-cov-2 no brasil. bepa. boletim epidemiológico paulista 18(207), 12–33 (2021) article  google scholar  camarotto, m.: reclamaçães explodem e governo decide olhar de perto o crédito consignado (2020). https://valorinveste.globo.com/produtos/credito/noticia/2020/10/16/reclamacoes-explodem-e-governo-decide-olhar-de-perto-o-credito-consignado.ghtml. accessed 1 may 2023 cheng, l., liu, f., yao, d.: enterprise data breach: causes, challenges, prevention, and future directions. wiley interdisc. rev. data mining knowl. disc. 7(5), e1211 (2017). https://doi.org/10.1002/widm.1211 article  google scholar  daly, a.: everyday ai ethics: from the global to local through facial recognition. osf preprints (2022). https://doi.org/10.31219/osf.io/bhm3w european union: eur-llex: access to european union law. micro-, smalland medium-sized enterprises: definition and scope (2016). https://eur-lex.europa.eu/legal-content/pt/txt/?uri=celex:32016r0679. accessed 1 may 2023 filho, j.j.: pacote tjsp: coleta e organização de dados do tribunal de justiça de são paulo (2020). https://github.com/jjesusfilho/tjsp. accessed 20 nov 2022 garvie, c.: the perpetual line-up: unregulated police face recognition in america. center on privacy & technology, georgetown law (2016) google scholar  gil, a.c.: como elaborar projetos de pesquisa, 4th edn. editora atlas, são paulo (2002) google scholar  glez-peña, d., lourenço, a., lópez-fernández, h., reboiro-jato, m., fdez-riverola, f.: web scraping technologies in an api world. brief. bioinform. 15(5), 788–797 (2014). https://doi.org/10.1093/bib/bbt026 article  google scholar  hao, k.: a us government study confirms most face recognition systems are racist (2019). https://www.technologyreview.com/2019/12/20/79/ai-face-recognition-racist-us-government-nist-study/. accessed 10 may 2023 haraway, d.j.: the companion species manifesto: dogs, people, and significant otherness. prickly paradigm press, chicago (2003) google scholar  lan, h.: covid-scraper: an open-source toolset for automatically scraping and processing global multi-scale spatiotemporal covid-19 records. ieee access 9, 84783–84798 (2021). https://doi.org/10.1109/access.2021.3085682 article  google scholar  lupton, d.: digital companion species and eating data: implications for theorising digital data-human assemblages. big data soc. 3(1), 1–5 (2016). https://doi.org/10.1177/2053951715619947 article  google scholar  lynch, j.: face off: law enforcement use of face recognition technology. elect. front. found. (eff) (2020). https://doi.org/10.2139/ssrn.3909038 article  google scholar  maslej, n., et al.: artificial intelligence index report 2023. technical report, institute for human-centered ai stanford university, stanford (2023). https://aiindex.stanford.edu/wp-content/uploads/2023/04/hai_ai-index-report_2023.pdf mol, a.: i eat an apple on theorizing subjectivities. subjectivity 22, 28–37 (2008). https://doi.org/10.1057/sub.2008.2 article  google scholar  montag, l., mcleod, r., de mets, l., gauld, m., rodger, f., pełka, m.: the rise and rise of biometric mass surveillance in the eu: a legal analysis of biometric mass surveillance practices in germany, the netherlands, and poland. edri (european digital rights) and eiji (edinburgh international justice initiative) (2021) google scholar  phankokkruad, m., jaturawat, p.: influence of facial expression and viewpoint variations on face recognition accuracy by different face recognition algorithms. in: 2017 18th ieee/acis international conference on software engineering, artificial intelligence, networking and parallel/distributed computing (snpd), pp. 231–237 (2017). https://doi.org/10.1109/snpd.2017.8022727 procon: procon tem reclamaçães de aposentados que não pediram empréstimos consignados (2020). https://globoplay.globo.com/v/8970817/. accessed 1 may 2023 salomão, l.f., outros: inteligëncia artificial: tecnologia aplicada á gestão dos conflitos no âmbito do poder judiciário brasileiro (2020). https://ciapj.fgv.br/sites/ciapj.fgv.br/files/estudos_e_pesquisas_ia_1afase.pdf. accessed 17 april 2023 sharif, m., naz, f., yasmin, m., shahid, m.a., rehman, a.: face recognition: a survey. j. eng. sci. technol. rev. 10(2), 1–12 (2017) article  google scholar  da silva, t.: visão computacional e racismo algorítmico: branquitude e opacidade no aprendizado de máquina. revista da associação brasileira de pesquisadores/as negros/as (abpn) 12(31), 428–448 (2020) article  google scholar  simonite, t.: ai is the future-but where are the women (2018). https://www.wired.com/story/artificial-intelligence-researchers-gender-imbalance. accessed 30 april 2023 união europeia: grupo de trabalho para o artigo 29\(^\circ \) para a proteção de dados (2017), https://edpb.europa.eu/about-edpb/more-about-edpb/article-29-working-party_pt. accessed 1 may 2023 vanderplas, j.: python for data science handbook. o’reilly media, sebastopol (2016) google scholar  download references author information authors and affiliations faculty of law of ribeirão preto, university of são paulo (usp), são paulo, brazil cristina godoy b. de oliveira & emily liene belotti school of arts, sciences and humanities, usp, são paulo, brazil otávio de paula albuquerque institute of mathematical and computer sciences of são carlos, usp, são paulo, brazil isabella ferreira lopes department of sociology, usp, são paulo, brazil rodrigo brandão de a. silva & glauco arbix authors cristina godoy b. de oliveiraview author publications search author on:pubmed google scholar otávio de paula albuquerqueview author publications search author on:pubmed google scholar emily liene belottiview author publications search author on:pubmed google scholar isabella ferreira lopesview author publications search author on:pubmed google scholar rodrigo brandão de a. silvaview author publications search author on:pubmed google scholar glauco arbixview author publications search author on:pubmed google scholar corresponding author correspondence to cristina godoy b. de oliveira . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper godoy b. de oliveira, c., de paula albuquerque, o., liene belotti, e., ferreira lopes, i., brandão de a. silva, r., arbix, g. (2023). regulation and ethics of facial recognition systems: an analysis of cases in the court of appeal in the state of são paulo. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_2 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords facial recognition systems and legal cases artificial intelligence and law ai and jurisprudence publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature allocating dynamic and finite resources to a set of known tasks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper allocating dynamic and finite resources to a set of known tasks conference paper first online: 12 october 2023 pp 193–208 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) allocating dynamic and finite resources to a set of known tasks download book pdf download book epub joão da silva  orcid: orcid.org/0009-0003-0101-87049, sarajane peres  orcid: orcid.org/0000-0003-3551-64809, daniel cordeiro  orcid: orcid.org/0000-0003-4971-73559 & … valdinei freire  orcid: orcid.org/0000-0003-0330-39319  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 521 accesses abstract we consider a generalization of the task allocation problem. a finite number of human resources are dynamically available to try to accomplish tasks. for each assigned task, the resource can fail or complete it correctly. each task must be completed a number of times, and each resource is available for an independent number of tasks. resources, tasks, and the probability of a correct response are modeled using item response theory. the task parameters are known, while the ability of the resources must be learned through the interaction between resources and tasks. we formalize such a problem and propose an algorithm combining shadow test replanning to plan under uncertain knowledge, aiming to allocate resources optimally to tasks while maximizing the number of completed tasks. in our simulations, we consider three scenarios that depend on knowledge of the ability of the resources to solve the tasks. results are presented using real data from the mathematics and its technologies test of the brazilian baccalaureate examination (enem). this study was partially supported by the coordenação de aperfeiçoamento de pessoal de nível superior (capes) finance code 001, by the são paulo research foundation (fapesp) grant #2021/06867-2 and the center for artificial intelligence (c4ai-usp), with support by fapesp (grant #2019/07665-4) and by the ibm corporation. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an adaptive resource allocation protocol for dynamic environments chapter © 2022 research on task allocation and resource scheduling method in cloud environment chapter © 2020 dynamic task scheduling with advance reservation of resources to minimize turnaround time for computational grid article 12 march 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. dynamic networks human resource management mental training operations research and decision theory theory and algorithms for application domains working memory 1 introduction the allocation of human resources to solve tasks is part of the daily routine of the most diverse institutions. going from the allocation of employees in a company to the fulfillment of complex tasks [23], to the assignment of public lawyers to defendants by a court [1]. in the previous examples, if the decision-maker, which allocates resources to perform each set of tasks, is well aware of the abilities of each resource to solve the tasks, one can use the hungarian algorithm to obtain the optimal solution [15]. however, this is not always the case. there are numerous scenarios where the abilities of the resources are not known beforehand, for example, in a non-governmental organization allocating tasks to sporadic volunteers [5, 14]. another example is crowdsourcing, the recent tendency in certain companies to allocate some tasks to non-specialized outsource workers [26, 28]. some works use previous information about the resource to estimate their ability to solve the tasks [8, 11]. to guarantee the quality of the solutions given by unknown resources, it is necessary to estimate their ability to accomplish such tasks. the ability estimation can be done by the conduction of a test before the allocation of the tasks [25]; or by dynamically estimating the abilities during the allocation process, depending on the results of previous assignments [31]. here, the resources’ abilities are dynamically estimated during the process, being updated after each response. many works consider a threshold so that weaker resources do not receive any task [9, 25, 31]; instead, we consider the resources scarce and try to use the maximum of their availability. we allocate the dynamic and finite resources that arrive sequentially to try to solve a set of known tasks. we consider a finite number of human resources arriving sequentially; tasks are assigned to resources dynamically, and the resources can either complete them correctly or fail. each task must be completed a number of times, and each resource can receive an independent number of tasks. no task is submitted twice to the same resource; each task can be submitted as many times as needed to different resources. resources, tasks, and the probability of a correct response are modeled using item response theory [6]. the task parameters are known, while the ability of the resources must be learned through the interaction between resources and tasks. we propose an algorithm that combines shadow test replanning [17] to replan under uncertain knowledge, with the goal of optimally allocating resources to tasks while maximizing the number of completed tasks. each replanning consists in solving an optimization problem through linear programming. the results are compared with two baselines, the random selection and the easier-first selection that selects the easier task available as the resources arrive sequentially. in summary, the contributions of this paper are: (i) a novel allocation of dynamic finite resources to a set of finite binary tasks problem is formalized; and (ii) algorithms are proposed to solve the problem in different scenarios by using most of the resources’ availability. the rest of this paper is structured as follows: the proposed allocation problem is defined in sect. 2. the proposed framework, with the definition of three algorithms, is presented in sect. 3. section 4 gives an overview of related works on the state of the art of task allocation. we present and discuss the results in sect. 5, before concluding in sect. 6. 2 the allocation of dynamic and finite resources to a set of binary tasks problem we consider the problem of repeatedly allocating resources to fulfill one of a set of tasks when finite resources are available dynamically. first, the decision-maker knows a set of tasks, in which each task must be solved an arbitrary number of times. second, resources are available dynamically to receive a number of tasks to try to solve. third, the decision-maker allocates the current resource to tasks, one after the other, until the resource’s availability ends. fourth, for each allocated task, the resource may fail or may fulfill it. finally, the decision-maker must allocate tasks to fulfill the largest number of tasks. we consider the following process: 1. while there is time, wait for resource: (a) a resource arrives. while the resource is available: i chooses and presents a task to the resource under two constraints: (i) the task was solved successfully less than a desired level; and (ii) the task has not been presented to the current resource; ii the resource tries to solve the task; and it is observed whether the task was solved or not. this process is formalized mathematically by the following definition: definition 1 (dynamic allocation of resources to solve tasks multiple times dart-mt). let \(\mathcal {t}\) be a set of tasks and \(\mathcal {r}\) a set of human resources. consider the function \(n: \mathcal {t} \rightarrow \mathbb {n}\), n(t) indicates the number of times the task t must be solved. also, consider the function \(m: \mathcal {r} \rightarrow \mathbb {n}\), m(r) indicates the number of tasks that can be assigned to resource r. \(\zeta _{t}\) is a vector of parameters for task \(t \in \mathcal {t}\). \(\theta _{r}\) is a vector of parameters for resource \(r \in \mathcal {r}\). \(f(\zeta _{t}, \theta _{r})\) is a function that represents the probability of the task t be fulfilled by the resource r. there is an order of resources arriving \(q = (r_{1}, r_{2}, \ldots , r_{|\mathcal {r}|})\), where \(r_{i} \in \mathcal {r}\) and \(r_{j} \ne r_{i}\) for all \(i \ne j\). the problem of dynamic resource allocation to solve tasks multiple times is defined by the tuple \((\mathcal {t}, \mathcal {r}, n, m, \{\mathbf {\zeta }_{t}\}, \{\mathbf {\theta }_{r}\}, f, q)\) and the objective is to maximize the expected number of solved tasks. because the result of a resource r trying to solve a task t is stochastic, an optimal solution is clearly contingent on past results; in the markov decision process jargon, policies. here, we considered different scenarios of knowledge about the problem dart-mt, where optimal solutions are contingent on information collected during the process. we consider three different scenarios. in every scenario probability function (f), tasks parameters and demands (\(\mathcal {t}\), n, and \(\{\mathbf {\zeta }_{t}\}\)) are known beforehand; they may be obtained from calibration in another set of resources. information about the resources (\(\mathcal {r}, m, \{\mathbf {\theta }_{r}\}\) and q) may be revealed directly or indirectly during the process. we differentiate knowledge of the current resource and future resources to define the following scenarios: (kk) known-known: fully-revealed, i.e., resources \(\mathcal {r}\), availability function m, parameters \(\{\mathbf {\theta }_{r}\}\) and arrival order q are known before hand; (ku) known-unknown: the number of resources \(|\mathcal {r}|\) and the current resource r is fully-revealed (availability m(r) and parameter \(\mathbf {\theta }_{r}\)); while future resources are known to be drawn from an a priori distribution on parameters \(\vartheta \) and availability m; and (uu) unknown-unknown: the number of resources \(|\mathcal {r}|\) and the a priori distribution on parameters \(\vartheta \) and availability m are revealed; the current resource r reveals its availability m(r), while its parameters must be learnt from results after submitting tasks to the resource. we note that the objective of managing to solve a task gets more difficult when knowledge is hidden from the decision maker, i.e., a decision maker should solve the largest number of tasks in scenario kk while solving the smallest number of tasks in scenario uu. to obtain an upper bound to the dart-mt problem, we relax the problem regarding the allocation and results of submitting a task to a resource. first, we allow a resource r to be partially allocated to a task t (decision \(s_{r,t}\)). second, a resource r solves a task deterministic and partially, given by \(p_{r, t} = f(\zeta _{t},\theta _{r})\). third, \(x_{r,t}=s_{r, t} p_{r, t}\) indicates the amount of task t to be solved by resource r. then, the following linear programming obtains an upper bound to the dart-mt problem: $$\begin{aligned} \max \quad & \sum _{t \in \mathcal {t}} \sum _{r \in \mathcal {r}} x_{r, t} & \end{aligned}$$ (1) $$\begin{aligned} \text {subject to} \quad & s_{r, t} \in [0,1] & \forall r \in \mathcal {r}, t \in \mathcal {t} \end{aligned}$$ (2) $$\begin{aligned} & x_{r, t} = s_{r, t} p_{r, t} & \forall r \in \mathcal {\text {subject to} r}, t \in \mathcal {t} \end{aligned}$$ (3) $$\begin{aligned} & \sum _{t \in \mathcal {t}} s_{r, t} \le m(r) & \forall r \in \mathcal {r} \end{aligned}$$ (4) $$\begin{aligned} & \sum _{r \in \mathcal {r}} x_{r, t} \le n(t) & \forall t \in \mathcal {t} \end{aligned}$$ (5) 3 \(\mathbf {[sta]^2o}\): a shadow test approach to skill-based task allocation optimization to solve the dart-mt problem in each of the three scenarios (kk, ku, and uu), we consider three levels of time abstractions: episodes, rounds, and steps. an episode comprehends the whole process of dart-mt problems when each resource \(r\in \mathcal {r}\) tries to solve m(r) tasks. a round comprehends the whole interaction with a resource r when the resource r tries to solve m(r) tasks. a step comprehends the interaction of a resource r and a task t. for each of the three scenarios, we define three different algorithms; all of them are based on the solution of an lp similar to lp in eqs. 1–5. then, at any step j of a round i consider: the current resource \(r_{i}\) and the set of tasks \(t_{i,j}\) that was presented to resource \(r_{i}\) before the step j or has already been completely solved as demanded by the function n(j); then, at step j a task t is drawn with probability: $$\begin{aligned} p_{t} = \frac{s_{r_{i},t}}{\sum _{t'\in \mathcal {t}\setminus t_{i,j}}s_{r_{i},t'}}, \end{aligned}$$ (6) where \(s_{r,t}\) for any \(r\in \mathcal {r},t\in \mathcal {t}\) is the solution of the lp. to solve the dart-mt problem in scenario kk, an lp is solved for each episode. to solve the dart-mt problem in scenario ku, an lp is solved for each round. to solve the dart-mt problem in scenario uu, an lp is solved for each step. in scenarios ku and uu, where knowledge is not full-revealed a priori, lps are defined based on population information, and when new information is revealed, per round or per step, a new lp is solved. this strategy of fully planning based on population information and replanning when information is renewed is used in the shadow test approach (sta) [18, 19, 30]. the algorithm 1 shows a general solution to dart-mt problems conditioned in one of the three scenarios. algorithm 1 the general solution to dart-mt conditioned for any scenario full size image kk scenario and per episode algorithm lppe. because the scenario kk knowledge is fully revealed a priori, the lp to be solved is the one described in eqs. 1–5. the solution to the lp problem provides us with a set of (partial) tasks to be presented for each resource. note that tasks are selected stochastically, but guarantee that a task t cannot be submitted repeatedly to the same resource and cannot be submitted beyond its demand n(t). since planning occurs only once, the only adaptation occurs through set \(t_{i,j}\) to avoid violating the constraints of the dart-mt problem. ku scenario and the per round algorithm lppr. let \(\hat{\mathcal {r}}\) be the set of resources that have already been selected from \(\mathcal {r}\), \(\tilde{\mathcal {r}}\) be the set of \(|\mathcal {r}| |\hat{\mathcal {r}}| 1\) resources drawn from distributions \(\vartheta \) and m, and \(r_{i}\) be the current resource. an adaptation of the previous lp problem is modeled as follows: $$\begin{aligned} \max \quad & \sum _{t\in \mathcal {t}} \sum _{r \in \tilde{\mathcal {r}} \cup \{r_{i}\}} x_{r, t} & \end{aligned}$$ (7) $$\begin{aligned} \text {subject to}\quad & s_{r, t} \in [0,1] & \forall r \in \tilde{\mathcal {r}} \cup \{r_{i}\}, t \in \mathcal {t} \end{aligned}$$ (8) $$\begin{aligned} & x_{r, t} = s_{r, t} p_{r,t} & \forall r \in \tilde{\mathcal {r}} \cup \{r_{i}\}, t \in \mathcal {t } \end{aligned}$$ (9) $$\begin{aligned} & \sum _{t \in \mathcal {t}} s_{r, t} \le m(r) & \forall r \in \tilde{\mathcal {r}} \cup \{r_{i}\} \end{aligned}$$ (10) $$\begin{aligned} & \sum _{r \in \tilde{\mathcal {r}} \cup \{r_{i}\}} x_{r, t} + \sum _{r \in \hat{\mathcal {r}}} \hat{ x}_{r,t} \le n(t) & \forall t \in \mathcal {t} \end{aligned}$$ (11) with \(\hat{x}_{r, t} = 1\) if the resource r succeeded on the already presented task t and 0 otherwise. instead of just solving an lp problem once for the episode, the lppr algorithm solves an lp problem after arriving and being revealed the next resource. replanning is done considering previous results on tasks submitted to previous resources and the knowledge regarding the current resource, whereas future resources are drawn from a priori distributions. uu scenario and the per step algorithm lpps. let \(\hat{\mathcal {r}}\) be the set of resources that have already been selected from \(\mathcal {r}\), \(\tilde{\mathcal {r}}\) be the set of \(|\mathcal {r}| |\hat{\mathcal {r}}| 1\) resources drawn from distributions \(\vartheta \) and m, \(r_{i}\) be the current resource, \(t_{i}\) the set of tasks already submitted to resource \(r_{i}\), and \(\hat{p}_{r_{i},t}\) be the estimation probability of success given current resource \(r_{i}\) and task t. an adaptation of the previous lp problem is modeled as follows: $$\begin{aligned} \max \quad & \sum _{t\in \mathcal {t}} \sum _{r \in \tilde{\mathcal {r}}} x_{r, t} + \sum _{t\in \mathcal {t}\setminus t_{i}} x_{r_{i}, t} & \end{aligned}$$ (12) $$\begin{aligned} \text {subject to}\quad & s_{r, t} \in [0,1] & \forall r \in \tilde{\mathcal {r}}, t \in \mathcal {t} \end{aligned}$$ (13) $$\begin{aligned} & s_{r_{i}, t} \in [0,1] & \forall t \in \mathcal {t}\setminus t_{i} \end{aligned}$$ (14) $$\begin{aligned} & x_{r, t} = s_{r, t} p_{r,t} & \forall r \in \tilde{\mathcal {r}}, t \in \mathcal {t } \end{aligned}$$ (15) $$\begin{aligned} & x_{r_{i}, t} = s_{r_{i}, t} \hat{p}_{r_{i},t} & \forall t \in \mathcal {t}\setminus t_{i} \end{aligned}$$ (16) $$\begin{aligned} & \sum _{t \in \mathcal {t}} s_{r, t} \le m(r) & \forall r \in \tilde{\mathcal {r}} \end{aligned}$$ (17) $$\begin{aligned} & |t_{i}| + \sum _{t \in \mathcal {t} \setminus t_{i}} s_{r_{i}, t}\le m(r_{i})& \end{aligned}$$ (18) $$\begin{aligned} & x_{r_{i},t} + \sum _{r \in \tilde{\mathcal {r}}} x_{r, t} + \sum _{r \in \hat{\mathcal {r}}} \hat{ x}_{r,t} \le n(t) & \forall t \in \mathcal {t}\setminus t_{i} \end{aligned}$$ (19) $$\begin{aligned} & \hat{x}_{r_{i},t} + \sum _{r \in \tilde{\mathcal {r}}} x_{r, t} + \sum _{r \in \hat{\mathcal {r}}} \hat{ x}_{r,t} \le n(t) & \forall t \in t_{i} \end{aligned}$$ (20) with \(\hat{x}_{r, t} = 1\) if the resource r succeeded on the already presented task t and 0 otherwise. here, the lpps algorithm solves an lp problem on every step (after each task submission). again, replanning is done considering previous results on tasks submitted to previous resources and the knowledge regarding the current resource, whereas future resources are drawn from a priori distributions. however, knowledge regarding the current resource is updated after every trial of solving a task, depending on the parameters of the task already submitted and the given results, success or failure. we postpone the discussion of estimating parameters of the current resource to sect. 5. 4 related work in this work, a new problem (dart-mt) is defined with specific settings. therefore, there is not an obvious state-of-the art method to be compared to ours. we used as baselines a random selection and the heuristic easier first, even though methods applicable to similar problems are briefly compared to our approach in the following paragraphs. many works make use of resource skills to improve the task allocation problem [4, 10, 14, 20, 27, 29]. some of them use this approach to improve allocation in project management or in crowdsourcing context, where the tasks are allocated to more than one resource to receive a number of solutions. however, these works assume the ability of the resources are given or can be estimated from a content-based approach, which is not always the case. our work stands on a cold start, i.e., we do not have any previous information about the resources. in recent works, social search engines have become very useful as it uses information about the resources so as to improve task allocation [2, 3, 8, 11]. these studies use techniques like ranking function and information retrieval in the context of crowdsourcing or the q &a problem. although similar to our work in the sense of optimizing task allocation, we do not use previous information about the resources and estimate their skills from the interaction with the tasks. also, we do not use a threshold to reject weaker resources. the sta is largely discussed in the context of adaptive tests [18, 19, 30]. in this context, many works address the problem of task allocation by learning information about the resources [12, 24]. they repeatedly administer tasks to resources; however, the objective of these works is restricted to better estimating the resources’ abilities. we use the same approach in a different context and with a different purpose since we use resource characterization to optimize the allocation of tasks by maximizing the number of solutions, not having resource characterization as an end in itself. in some works [9, 25, 31], resources and tasks are dynamically characterized. the tasks are allocated to a resource if the probability of a correct answer is above a threshold. our study does not limit the allocation of tasks by a threshold. also, the purpose is different, in their case, each resource receives only one task, and each task needs only one solution. 5 experiments and results to evaluate the proposed algorithms, we adapt real data from an exam of mathematics to emulate tasks (questions) and resources (students). within this data, we construct different dart-mt problems to identify the difference in the performance of our algorithms in the three scenarios kk, ku, and uu. finally, we present the improvement of our algorithms against two baselines: uniformly random and easier first. 5.1 real world database evaluating in real world data. real data can be obtained, basically, in two ways: online or offline. the online test consists of presenting tasks to human resources while the process is running, whereas the offline test consists of using a database that was previously collected. the advantage of the online test is to apply the method in the real world with resources interacting directly with the allocation process. the downside is that access to such resources is very costly, so getting a reasonable amount of human resources available to receive tasks is very complicated. the advantage of the offline test is precisely that it can take advantage of a database, but the interaction of the process with the real world is lost. to use an offline test, the database need to meet certain criteria: (i) the independence of task solution conditioned on the resource, and (ii) all tasks have been submitted to all resources. the success or failure of each task submission is obtained from the history taken from the database. we can choose any task for a resource and check in the history if he has solved the task correctly or not. one can arbitrarily choose resources, their arrival order, and their availability without loss of generality. availability must always be at most equal to the total amount of existing tasks in the database. it is equally possible to choose tasks and the number of solutions for them arbitrarily. the number of solutions demanded by each task must be equal to or less than the total amount of resources in the database. with historical data in hand, it remains to model the parameters \(\zeta \), \(\theta \), and the success probability function f. enem database. a database of the 2012 brazilian baccalaureate examination (enem) containing responses from ten thousand people (resources) and 185 questions (tasks) was used. the data are public and can be downloaded from the transparency portal [13]. enem uses item response theory to model tasks (\(\zeta \)), resources (\(\theta \)), and the success probability function (f). this exam is used to assess the level of knowledge (skills) of students in four areas of knowledge: human sciences; natural sciences; languages; and mathematics. in this work, only the exam of mathematics was considered. we calibrate task parameters on data from 10,000 students and make use of the three-parameter logistic model, where parameter b indicates the difficulty of a question. we make use of such parameters to construct a baseline policy where the current resource always receives the easiest allowed task, the heuristic easier first. 5.2 defining dart-mt problems from the database with the answers of ten thousand enem students, the average chance of the task t to be correctly solved is estimated by: $$\begin{aligned} k(t) = \frac{\text {number of resources that solved the task } t \text { correctly}}{\text {number of resources that tried to solve the task } t} \end{aligned}$$ (21) the responses of all students in the database were used to calculate k(t) and \(k_{mean} = \frac{\sum _{t\in \mathcal {t}}k(t)}{|\mathcal {t}|}\). however only one hundred of them are sampled to evaluate the method at each episode. so, from now on we consider \(|\mathcal {r}| = 100\) and the 45 math items from the enem’s mathematics test as the tasks, so \(|\mathcal {t}| = 45\). depending on the number of solutions for each task, n(t), and the availability of each resource, m(r), solving all the required tasks can be more or less difficult. in this way, we define the following difficulty levels: level 1: $$\begin{aligned} n(t) = \lceil {|\mathcal {r}| \times k_{mean}}\rceil \quad \forall t \in {\mathcal {t}}\end{aligned}$$ (22) $$\begin{aligned} m(r) = \lceil |\mathcal {t}| \times k_{mean} \rceil \quad \forall r \in \mathcal {r} \end{aligned}$$ (23) level 2: $$\begin{aligned} n(t) = \lceil {|\mathcal {r}| \times k(t)}\rceil \quad \forall t \in \mathcal {t}\end{aligned}$$ (24) $$\begin{aligned} m(r) = \lceil |\mathcal {t}| \times k_{mean} \rceil \quad \forall r \in \mathcal {r} \end{aligned}$$ (25) level 3: n(t) is defined as the number of correct solutions for the task t and m(r) as the number of tasks the resource r solved correctly. in levels 1, 2, and 3, we have that \(\sum _{t\in \mathcal {t}} n(t) \approx \sum _{r\in \mathcal {r}} m(r)\). note that we can only get all the desired solutions for difficulty levels 1 and 2 if all resources have the same ability or/and if all tasks have the same difficulty. to obtain all the solutions for level 3, we have to present for each resource exactly the tasks that user can solve correctly. one way to relax the levels is to decrease the number of solutions desired for each task, for example, \(n'(t) = n(t) \times 0.5\). we apply such relaxed version to level 3 to obtain the level 4. another more-relaxed level, let’s call level 0, is: given \(k_{mean}\), we compute n(t) and m(r) as constant values such that $$\begin{aligned} |\mathcal {r}| \times m(r) \times k_{mean} = |\mathcal {t}| \times n(t) \end{aligned}$$ (26) for this level, we choose \(m(r) = 16 \approx \sum _{t \in \mathcal {t}} k(t)\) and we get \(n(t) = 12\) from the eq. 26 with \(k_{mean} = 0.34\). empirical experiment. for each difficulty level and algorithm, we run 100 episodes. for each episode, one hundred resources are used from a previous random sample among the ten thousand available students. all simulations were performed in a google colaboratory environment. all codes were written in python 3 language. to solve lp problems, we used the pulp module and the open-source solver coin-or branch and cut (cbc) [22]. 5.3 results using real world data we use the five levels defined before to evaluate our proposed algorithms. the results obtained for each level and in each scenario are shown in the table 1. the proposed approaches are compared against two baselines: the uniform random selection and the heuristic easier first. the comparison is by the number of tasks solved correctly. besides solving each scenario with the corresponding time abstraction, we also apply per round algorithm to the scenario kk to be contingent on trial results. table 1. results for all scenarios and algorithms.full size table discussion. regardless of the knowledge of the resource’s ability to solve the tasks, the algorithms lppe, lppr, and lpps obtained quantities of solutions greater than the baselines (random and easier first) for all levels. in the case resources’ abilities are well known, the algorithms lppe and lppr are better than the baselines and can be computed in a feasible time (less than three minutes per episode). if we do not know the ability of the resources to whom the tasks should be submitted, we use the algorithm lpps, which furnished a better allocation than the baselines for all five levels. the algorithm lpps takes about forty minutes per episode, and the resources need to wait no more than 1.7 s to receive a task to try to solve (it was the case of the level 0, with the results shown on table 1). note that the algorithms only make sense to be applied in certain scenarios. for example, in scenario uu, it doesn’t make sense to use the per episode algorithm since we will allocate the tasks based on the solution of an lp problem whose resources’ abilities were unknown. also, if we are in the scenario kk, it is not necessary to solve an lp problem at each step if the resources’ abilities are well known from the beginning. to better visualize the results, box plot graphics are shown for all scenarios, levels, and algorithms in fig. 1. the name of the algorithms are abbreviated: random (rand), easier first (e-f), per episode (ep), per round (res), and per step (sub). fig. 1. box plots results for all scenarios and levels full size image to better analyze the results, we used statistical tests to verify which algorithm produced the higher number of solutions. the first step was to use the shapiro-wilk test to determine the data normality. with \(95\%\) of significance, the data do not deviate from a normal distribution, except for the results for some algorithms of level 0. as we have a lot of samples (100 per episode), we used the analysis of variance (anova) test to compare the algorithms for each scenario in all levels, including level 0. the results of the anova tests determined that for all scenarios in all levels, there are statistically significant differences, with \(95\%\) of significance. from the anova test results, the number of solutions is not statistically equivalent between the algorithms used in each scenario and level. then, we used the tukey pairwise multiple comparisons (post hoc) test to verify which means differ, with \(95\%\) of significance. specific algorithms are used in each scenario: kk: random; easier first; per episode; and per round. ku: random; easier first; and per round. uu: random; easier first; and per step. from the results of the tukey post hoc test, the proposed algorithms per episode, per round and per step are better than the baselines (random and easier first) for all scenarios in all levels, except for the algorithm lpps and the baseline easier first in the scenario uu at level 3. in this comparison, the tukey test resulted in no statistical difference in the number of solutions. figure 2 shows (for the first episode) the number of submissions and solutions, the mean of the difficulties of allocated tasks, and the ability of the resources for these two algorithms for scenario uu at level 3. from this figure, resources with higher abilities are available to receive more tasks at level 3. this is a very appropriate case for the easier first algorithm since the more tasks the resources receive following the easier first heuristic, the more difficult tasks received. fig. 2. algorithms at level 3, on the first episode. full size image the algorithms per episode and per round in the scenario kk are statistically equivalent. it is expected since in the scenario kk, all the resources’ abilities are well known beforehand, then the gain in doing replannings after each round is not significantly better than planning once at the beginning of the episode. the heuristic easier first is better than the random selection at most levels, except at level 0. figure 3 shows (for the mean of all episodes) the number of submissions and solutions, the mean of the difficulties of allocated tasks, and the ability of the resources for these two algorithms at level 0. the mean of all episodes is used so that the influence of the ability of the resources is diminished. we can see that the mean of the abilities goes to zero. in the easier first allocation, it is clear the difficulty of tasks increases over the steps, whereas the number of solutions decreases. while in the random allocation, the difficulty of the tasks is approximately uniform. the last resources receive tasks slightly more difficult because the easier already received the needed number of solutions. these characteristics from both algorithms explain why the random made more allocations (approximately \(4\%\)) than the easier first, which led to receiving more solutions (approximately \(1\%\)). fig. 3. algorithms at level 0, mean of all episodes. full size image 6 conclusion and future work we can compare how far is the algorithm lpps from the baselines and from the optimal case, that is, when we have the most information: scenario kk and the solution of the lp problems is updated per round. we have an increase in the number of solutions compared with the baselines on all levels. compared with the baseline easier first, we have up to almost \(9\%\) more solutions; compared with the random, we have up to \(49\%\) more solutions. when we compare with the optimal case, the uu scenario using the lpps algorithm reached up to \(96\%\) of the number of solutions of the optimal case. these processes could be used to submit fewer tasks to resources without losing important information. for example, when asking people to answer polls or to generate databases to be used in important applications such as q &a, intelligent tutoring systems, or natural language processing. in addition to the abilities of the resources being unknown, there is also the case in which the complexity of solving the tasks is unknown. therefore, estimating parameters that characterize such tasks become necessary [7, 21]. the task parameters can be obtained from the solutions of the tasks or from a description of them [16]. we plan on considering scenarios where the resource is not continuously available to receive tasks, but he arrives and goes. also, we plan on developing a real experiment with human resources receiving tasks and not only using real data from other applications. references abrams, d.s., yoon, a.h.: the luck of the draw: using random case assignment to investigate attorney ability. univ. chic. law rev. 74(4), 1145–1177 (2007). http://www.jstor.org/stable/20141859 ali, i., chang, r.y., hsu, c.h.: soqas: distributively finding high-quality answerers in dynamic social networks. ieee access 6, 55074–55089 (2018) article  google scholar  aydin, b.i., yilmaz, y.s., demirbas, m.: a crowdsourced “who wants to be a millionaire?’’ player. concurr. comput. pract. exp. 33(8), e4168 (2017) article  google scholar  ben rjab, a., kharoune, m., miklos, z., martin, a.: characterization of experts in crowdsourcing platforms. in: vejnarová, j., kratochvíl, v. (eds.) belief 2016. lncs (lnai), vol. 9861, pp. 97–104. springer, cham (2016). https://doi.org/10.1007/978-3-319-45559-4_10 chapter  google scholar  bezerra, c.m., araújo, d.r., macario, v.: allocation of volunteers in non-governmental organizations aided by non-supervised learning. in: 2016 5th brazilian conference on intelligent systems (bracis), pp. 223–228 (2016) google scholar  birnbaum, a.: some latent trait models and their use in inferring an examinee’s ability. in: lord, f.m., novick, m.r. (eds.) statistical theories of mental test scores, reading, charlotte, nc, pp. 397–479. addison-wesley (1968) google scholar  bock, r.d., aitkin, m.: marginal maximum likelihood estimation of item parameters: application of an em algorithm. psychometrika 46(4), 443–459 (1981) article  mathscinet  google scholar  difallah, d.e., demartini, g., cudré-mauroux, p.: pick-a-crowd: tell me what you like, and i’ll tell you what to do: a crowdsourcing platform for personalized human intelligence task assignment based on social networks. in: www 2013 proceedings of the 22nd international conference on world wide web, pp. 367–377 (2013) google scholar  ekman, p., bellevik, s., dimitrakakis, c., tossou, a.: learning to match. in: 1st international workshop on value-aware and multistakeholder recommendation (2017) google scholar  fan, j., li, g., ooi, b.c., tan, k.l., feng, j.: icrowd: an adaptive crowdsourcing framework. in: proceedings of the 2015 acm sigmod international conference on management of data, pp. 1015–1030. association for computing machinery (2015) google scholar  horowitz, d., kamvar, s.d.: the anatomy of a large-scale social search engine. in: proceedings of the 19th international conference on world wide web www 2010, raleigh, north carolina, usa, p. 431. acm press (2010) google scholar  huang, y.m., lin, y.t., cheng, s.c.: an adaptive testing system for supporting versatile educational assessment. comput. educ. 52(1), 53–67 (2009) article  google scholar  inep: instituto nacional de educação e pesquisas educacionais anísio teixeira entenda sua nota no enem (2012). http://download.inep.gov.br/educacao_basica/enem/guia_participante/2013/guia_do_participante_notas.pdf. accessed 19 june 2021 krstikj, a., esparza, c.r.m.g., mora-vargas, j., escobar, h.l.: volunteers in lockdowns: decision support tool for allocation of volunteers during a lockdown. in: regis-hernández, f., mora-vargas, j., sánchez-partida, d., ruiz, a. (eds.) humanitarian logistics from the disaster risk reduction perspective: theory and applications, pp. 429–446. springer, cham (2022). https://doi.org/10.1007/978-3-030-90877-5_15 chapter  google scholar  kuhn, h.w.: the hungarian method for the assignment problem. nav. res. logist. (nrl) 2(1), 83–97 (1955) article  mathscinet  math  google scholar  benedetto, l., cappelli, a., turrin, r., cremonesi, p.: r2de: a nlp approach to estimating irt parameters of newly generated questions. in: proceedings of the 10th international conference on learning analytics and knowledge (2020) google scholar  van der linden, w.j.: constrained adaptive testing with shadow tests. in: van der linden, w.j., glas, g.a. (eds.) computerized adaptive testing: theory and practice, new york, boston, dordrecht, london, moscow, pp. 27–52. kluwer academic publishers (2000) google scholar  van der linden, w.j., jiang, b.: a shadow-test approach to adaptive item calibration. psychometrika soc. 85(2), 301–321 (2020) article  mathscinet  math  google scholar  van der linden, w.j., veldkamp, b.p.: constraining item exposure in computerized adaptive testing with shadow tests. j. educ. behav. stat. 29(3), 273–291 (2004) article  google scholar  liu, c., gao, x., wu, f., chen, g.: qita: quality inference based task assignment in mobile crowdsensing. in: pahl, c., vukovic, m., yin, j., yu, q. (eds.) icsoc 2018. lncs, vol. 11236, pp. 363–370. springer, cham (2018). https://doi.org/10.1007/978-3-030-03596-9_26 chapter  google scholar  mislevy, r.j.: bayes modal estimation in item response models. psychometric 51, 177–195 (1986) article  mathscinet  math  google scholar  mitchell, s., kean, a., mason, a., o’sullivan, m., phillips, a., peschiera, f.: optimization with pulp (2009). https://coin-or.github.io/pulp/index.html. accessed 20 june 2021 munkres, j.: algorithms for the assignment and transportation problems. j. soc. ind. appl. math. 5(1), 32–38 (1957) article  mathscinet  math  google scholar  mwamikazi, e., fournier-viger, p., moghrabi, c., barhoumi, a., baudouin, r.: an adaptive questionnaire for automatic identification of learning styles. in: ali, m., pan, j.-s., chen, s.-m., horng, m.-f. (eds.) iea/aie 2014. lncs (lnai), vol. 8481, pp. 399–409. springer, cham (2014). https://doi.org/10.1007/978-3-319-07455-9_42 chapter  google scholar  negishi, k., ito, h., matsubara, m., morishima, a.: a skill-based worksharing approach for microtask assignment. in: 2021 ieee international conference on big data (big data), pp. 3544–3547 (2021) google scholar  paschoal, a.f.a., et al.: pirá: a bilingual portuguese-english dataset for question-answering about the ocean. in: demartini, g., zuccon, g., culpepper, j.s., huang, z., tong, h. (eds.) cikm 2021: the 30th acm international conference on information and knowledge management, virtual event, queensland, australia, 1–5 november 2021, pp. 4544–4553. acm (2021). https://doi.org/10.1145/3459637.3482012 shekhar, g., bodkhe, s., fernandes, k.: on-demand intelligent resource assessment and allocation system using nlp for project management. in: amcis 2020 proceedings, vol. 8 (2020) google scholar  tran-thanh, l., stein, s., rogers, a., jennings, n.: efficient crowdsourcing of unknown experts using bounded multi-armed bandits. artif. intell. 214, 89–111 (2014) article  mathscinet  math  google scholar  tu, j., cheng, p., chen, l.: quality-assured synchronized task assignment in crowdsourcing. ieee trans. knowl. data eng. 33(3), 1156–1168 (2021) google scholar  veldkamp, b.p.: bayesian item selection in constrained adaptive testing using shadow tests. psicologica 31(1), 149–169 (2010) google scholar  yu, d., wang, y., zhou, z.: software crowdsourcing task allocation algorithm based on dynamic utility. ieee access 7, 33094–33106 (2019) article  google scholar  download references author information authors and affiliations school of arts, sciences and humanities, university of são paulo, são paulo, brazil joão da silva, sarajane peres, daniel cordeiro & valdinei freire authors joão da silvaview author publications search author on:pubmed google scholar sarajane peresview author publications search author on:pubmed google scholar daniel cordeiroview author publications search author on:pubmed google scholar valdinei freireview author publications search author on:pubmed google scholar corresponding author correspondence to joão da silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, j., peres, s., cordeiro, d., freire, v. (2023). allocating dynamic and finite resources to a set of known tasks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_13 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords dynamic resource allocation adaptive allocation shadow test publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature community detection for multi-label classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper community detection for multi-label classification conference paper first online: 12 october 2023 pp 78–93 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) community detection for multi-label classification download book pdf download book epub elaine cecília gatto  orcid: orcid.org/0000-0002-3322-64079, alan demétrius baria valejo  orcid: orcid.org/0000-0002-9046-94999, mauri ferrandin  orcid: orcid.org/0000-0002-4248-120710 & … ricardo cerri  orcid: orcid.org/0000-0002-2582-16959  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 565 accesses 1 citation 1 altmetric abstract exploring label correlations is one of the main challenges in multi-label classification. the literature shows that prediction performances can be improved when classifiers learn these correlations. on the other hand, some works also argue that the multi-label classification methods cannot explore label correlations. the traditional multi-label local approach uses only information from individual labels, which makes it impractical to find relationships between them. in contrast, the multi-label global approach uses information from all labels simultaneously and may miss more specific relationships that are relevant. to overcome these limitations and verify if improving the prediction performances of multi-label classifiers is possible, we propose using community detection methods to model label correlations and partition the label space into partitions between the local and global ones. these partitions, here named hybrid partitions, are formed of disjoint clusters of correlated labels, which are then used to build multi-label datasets and train multi-label classifiers. since our proposal can generate several hybrid partitions, we validate all of them and choose the one that is considered the best. we compared our hybrid partitions with the local and global approaches and an approach that generates random partitions. although our proposal improved the predictive performance of the used classifier in some datasets compared with other partitions, it also showed that, in general, independent of the approach used, the classifier still has difficulties learning several labels and predicting them correctly. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-label classification with label clusters article 04 november 2024 partial multi-label learning with instance correlations chapter © 2024 extracting label importance information for multi-label classification chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining functional clustering machine learning mixed methods multivariate analysis 1 introduction multi-label classification is the task of simultaneously assigning multiple labels to an instance. thus, a classifier must be induced to predict a set of labels for new instances. as real-world applications, we can mention recognition of sensor activities [13], protein function prediction [39], and drug-target interactions [20]. a multi-label problem can be solved mainly through local and global approaches. the local approach divides the original multi-label dataset into many binary ones, so labels are learned individually. one classifier per label is trained, and the individual predictions are gathered for the final evaluation. any classification algorithm can be used, and traditional methods are binary relevance [25, 36] and classifier chains [23]. the global approach creates new methods or modifies existing ones to deal with all classes of the problem at the same time. therefore, only one multi-label classifier is trained and is responsible for predicting all labels of an instance. traditional methods in this approach include back propagation multi-label learning (bpmll) [37] and multi-label k-nearest neighbors (mlknn) [38]. among the challenges of multi-label classification, we can highlight class imbalance [33], high dimensionality [3], and label correlations [4]. learning label correlations can help to predict labels that would probably not be predicted using traditional methods. works in the literature have shown that modeling label correlations is an important aspect of multi-label classification and can help to build better classification models [12, 19]. the literature presents different strategies to model label correlations, and community detection methods are one of them [7] [32]. the motivation to use them is that the network topology can encode interactions of the data systematically and find relationships between them [17, 31]. community detection methods are also graph partitioning algorithms: they divide vertices into clusters to minimize the number of edges between them. roughly speaking, a community (cluster) is a set of vertices with many edges inside and a few edges outside the community, a desired characteristic for this study [31]. in multi-label problems, labels can be considered vertices and the correlations between them, the edges. hence, we can build a graph of correlated labels, which can then be partitioned to find label communities. in this paper, we verify if the performance of a classifier can be improved, and if it can better learn label correlations, using a different approach from the conventional global and local ones. we propose an approach that generates and validates several partitions from the label space which explores the label correlations and chooses the best one. such partitions are composed of disjoint correlated label clusters and are called hybrid partitions. figure 1 presents our idea of multi-label partitions, where squares are the partitions, circles are label clusters, and diamonds are labels. figure 1a shows a global partition (global approach) composed of only one cluster with all labels. we induce one global classifier for global partitions, and this classifier simultaneously learns and outputs predictions for all labels. figure 1b shows a local partition (local approach) composed of one cluster for each label. one local classifier per cluster is induced in local partitions, also with train and test sets, and then we gather those predictions to make the final evaluation for the dataset. finally, fig. 1c shows an example of a hybrid partition, where correlated labels are clustered. we need to induce a global-based classifier for clusters with more than one label, in order to learn correlations between those labels. in contrast, we induce a local-based classifier for clusters with a single label. the final prediction is then obtained combining the predictions performed by all global and local-based classifiers. fig. 1. illustration of the global, local and hybrid partitions. full size image to get consistent and coherent results, we must use the same classifier in all clusters; otherwise, we cannot observe how much the classifier learned. therefore, we emphasize that using a multi-label classifier with local and global versions is necessary for our strategy since we want to compare the improvements from hybrid partitions in relation to local, global, and even random generated partitions. therefore, the same classifier must be used in all partitions. as an example, it does not make sense to use a global-based classifier such as ml-c4.5 [8] since it cannot handle clusters containing a single label. in our strategy, using binary relevance in clusters with more than one label is not possible, as we would be re-dividing the problem and learning separate labels when we want to learn the clustered labels. accordingly, the classifier used in our strategy has to deal with all types of partitions. for this reason, here we used the clus framework [34], which induces binary and multi-label decision trees based on predictive clustering trees (pct), and is considered to be one of the state-of-the-art methods in the literature. we train a binary pct for each label and combine their outputs to form the final multi-label prediction for the local partition. for the global partition, only one multi-label pct is necessary. to test our hybrid partitions, a set of multi-label pcts, or a combination of binary and multi-label pcts, can be applied, depending on how many labels are in the partition clusters. the individual outputs are then combined to form the final multi-label prediction. as already pointed out by basgalupp et al. [1], finding a suitable partition in a set with n labels is a hard task, whose difficulty increases with the number of labels. considering a set with n labels, the number of possible partitions consisting of k separate and non-empty subsets is given by \(\sum _{k=0}^{n} \left\{ n \atop k\right\} \) [16]. thus, the number of possible partitions drastically increases with the number of labels. as an example, with \(n=12\) labels, 44.213.597 partitions can be generated. our experiments were conducted using 20 benchmark multi-label datasets from five application domains with different characteristics. first, we used jaccard index and rogers-tanimoto similarity measures to model the label correlations, which are used to build the label co-occurrence graphs. then, we applied 7 community detection methods that partition the graphs and generate several hybrid partitions. next, those hybrid partitions were validated, and the best one tested. finally, the performance results were compared with the global, local, and random generated partitions using the same base classifier. our results showed that our approach could improve the classifier’s prediction, be competitive with the local and random generated partitions, and outperform the global approach. however, the results also showed that independent of the partitioning used, the overall performance did not improve much, i.e., we noticed that the average performance remained competitive for most methods and datasets. thus, the valuable insight we had with our results is that the multi-label classification methods need to improve because regardless of the partitioning used or if the correlations were (or not) explored, we cannot state with absolute certainty that they are correctly learning the labels. the remainder of this paper is organized as follows: sect. 2 presents some related studies; sect. 3 presents our proposed approach; sect. 4 explains how we conducted our experiments; sect. 5 presents and discusses our results; and finally, sect. 6 presents our conclusions and future works. 2 related work a related and similar study was presented by melo and paulheim [15]. the authors used local and global feature selection approaches with binary relevance to compare flat and hierarchical problems. they showed that when using multi-label transformation methods, the overall predictive performance of the local feature selection approach was better and superior to the global. however, the authors also elucidated that, in some cases, the results of the best general performance between adaptation or transformation methods are inconclusive. although the authors used the instance space and feature selection, the results presented go in the same direction as our work. rivolli et al. [25] presented a hypothesis that base algorithms can have a more substantial influence than the binary transformation strategies on the predictive performance of multi-label models. the authors demonstrated that despite the performance improvements, all investigated strategies and base algorithms could not predict some labels or mispredict labels for many datasets. therefore, the conclusions are aligned with our study. on the problem of learning label correlations, luaces et al. [14] demonstrated that the level of dependency between labels in benchmark datasets is deficient, so the methods cannot explore the correlations. that was one of the reasons that led them to build a synthetic dataset generator, combining synthetic data with benchmark datasets to better analyze the behavior of multi-label classification methods. they pointed out the difficulty of extracting useful conclusions using only benchmark datasets. in gatto et al. [12], the authors modeled label correlations using the jaccard index and partitioned the label space using agglomerative hierarchical clustering, which was able to find hybrid partitions. they used average, complete, and single linkage as an agglomerative method to build dendrograms and cut them into levels. different hybrid partitions were generated and validated using the clus framework, which can deal with local and global partitions. the highest macro-f1 performance was used to choose the more suitable hybrid partition, compared with local, global, and random generated partitions. the authors concluded that the results from hybrid partitions were competitive, or better in some cases, compared to the local and global approaches. similar to our work, the authors, in the same direction as melo and paulheim [15], observed that the local approach was superior to the global one in most cases. also trying to find partitions in-between the global and local ones, the study presented by basgalupp et al. [1] investigated whether it was possible to obtain better results by learning multi-target models on several partitions of the targets. they used a genetic algorithm (ga) and an exhaustive search to find alternate partitions between the global and local ones. they tested those partitions with random forests and decision trees in multi-target regression and multi-label problems and then compared them with the local and global ones. an oracle experiment was also conducted, proving that it is possible to find partitions beyond traditional ones and obtain superior performances. although the global approach led to better results for the multi-label case with decision trees, the results showed no statistically significant differences among local, global, and alternate partitions, which matches our conclusions. in our work, we also found hybrid partitions that led to better results than the local and global ones for some datasets, and the conclusions from this study are similar to the ones from melo and paulheim [15]. from the revised works we can find many open research questions in multi-label classification. our objective here is not to answer or find solutions for all of them. instead, we focused on finding partitions between global and local ones and verifying whether they could improve the classifier’s predictions. 3 proposed approach our approach consists of the following steps: 1) data preprocessing, which divides the dataset into train, validation, and test sets; 2) modeling label correlations, where we build graphs computing jaccard index and rogers-tanimoto similarity measures on the label space of the train set; 3) applying community detection methods to find hybrid partitions; 4) building the hybrid partitions and the corresponding datasets for each partition; 5) validate the hybrid partitions and choose the best one using the silhouette coefficient; and 6) test the best hybrid partition with the test set. figure 2 shows an overview of our approach. the complete process is presented in the following subsections. fig. 2. overview of the proposed approach. full size image 3.1 steps 1 and 2: preprocess and model label correlations step 1 divides the dataset into the train, validation, and test sets and also splits the label space from the train set, while step 2 models label correlations. according to silva and zhao [31], we need a similarity or dissimilarity matrix to build a network from vector-based data, which makes it possible to establish links between pairs of labels with weights according to that matrix. however, links with small weights can lead to poor results and be considered noises that could provide wrong information to the machine learning algorithm. to avoid that, sparsification must be applied, a preprocess that cuts those edges and can improve the learning stage. one traditional sparsification is based on the concept of k nearest neighbors, where the cuts occur in the links that are not part of the neighborhood. another type allows a threshold that cuts some percentage of the links [31]. we use both types in our study. we choose jaccard index (eq. 1) and rogers-tanimoto (eq. 2) as similarities measures to build label co-occurrence graphs, as they are well-known and popular measures [6, 11, 35]. in eqs. 1 and 2, a stands for the number of times labels \(l_{p}\) and \(l_{q}\) occur together, while d stands for the number of times those two labels never occur together. variables b and c indicate the number of times labels \(l_{p}\) and \(l_{q}\) occur alone. each measure is applied in the label space of the train set, resulting in similarity matrices. each matrix represents a complete graph modeled with the label correlations, where an edge (similarity value) connects every pair of distinct vertices (labels). 3.2 step 3: apply community detection methods in this step, we applied seven community detection methods to each graph. the methods are divided into two categories: a) hierarchical: waktrap [21], fast greedy [9] and edge betweenness [18]; and b) non-hierarchical: louvain [2], infomap [27], spinglass [24] and label propagation [22]. hierarchical methods provide dendrograms that can be used to construct several hybrid partitions. the procedure is similar to hierarchical clustering but uses different techniques to agglomerate vertices: walktrap, which is based on random walks; fast greedy based on modularity gain, and edge betweenness based on the number of shortest paths through the edge. unlike hierarchical methods, non-hierarchical ones provide only one hybrid partition, constructed using different concepts. louvain and infomap are multilevel methods and differ in the optimization function used: the former is based on fastgreedy and the latter on map equation. spinglass is based on potts spin glass, which uses rotation models to perform clustering, while in label propagation, each vertex is assigned to the most frequent label among its neighbors. 3.3 step 4: build hybrid partitions after applying community detection methods, only one is chosen to generate hybrid partitions. this choice is performed using a modularity criterion that measures the separation among vertices, quantifying the density of links within communities compared to links between communities [18, 31]. finally, the method with the highest modularity value is chosen. as an example, consider that between non-hierarchical methods, louvain obtained the highest modularity value. hence, each identified community can be considered a cluster of correlated labels. then, we build a corresponding dataset for each cluster using the train instances assigned to the labels of the cluster, induce a classifier, and test it in the test set. the same occurs with hierarchical methods, but as they generate more than one hybrid partition, we must validate all generated partitions from the chosen community detection method and then choose only one hybrid partition to test. 3.4 step 5: validate hybrid partitions this step is executed only in the partitions generated by hierarchical community detection methods since these generate several hybrid partitions. each hybrid partition created using hierarchical methods in step 4 is now evaluated in a validation dataset. here we used the silhouette coefficient [28, 30] cluster validation measure. it measures the quality of partitions based on the proximity between, in this case, the labels of a particular cluster and the distance between them and the closest cluster. the partition with the highest silhouette coefficient is then evaluated in a test dataset. 3.5 step 6: testing hybrid partitions finally, the last step is to test the best-validated hybrid partition. as already explained, the clus framework was used in our experiments, configured with the hyper-parameters listed below. for a detailed explanation of clus and its hyper-parameters, the reader can consult the clus webpagefootnote 1. general: compatibility = mlj08 attributes: reducememorynominalattrs = yes and weights = 1 tree: heuristic = variancereduction and ftest = [0.001,0.005,0.01,0.05,0.1,0.125] model: minimalweight = 5.0 output: writepredictions = test 4 experimental setup we evaluated our approach using 20 multi-label datasets from five different application domains, which were chosen to diversify the domain and number of labels. their characteristics are summarized in table 1 with the name of each dataset, domain, the total number of instances, attributes, and labels. cardinality is the average of labels per instance; density is the average frequency of labels, and tcs (theoretical complexity score) [5] is a measure that computes how difficult it is to learn a predictive model from the dataset. table 1. characteristics of the datasets.full size table our source code and all necessary materials to replicate the experiments are freely availablefootnote 2. we executed our experiments within the 10-fold cross-validation strategy, using an iterative stratification algorithm proposed for multi-label data [29]. our proposal was compared with three approaches to generate partitions: local, global, and random. global and local partitions were already explained in sect. 1. random partitions were generated similarly to hybrid partitions. the main difference is in step-2: given an adjacency matrixfootnote 3 obtained from the label space, k-nn sparsification is executed with a random chosen k value. to evaluate the results, we focused mainly on two multi-label measures: a) macro-f1 (maf1), which is the harmonic mean of the macro-precision (map) and macro-recall (mar), and b) missing label problem (mlp). maf1 is a good measure for multi-label problems since it considers the individual performances in each class. furthermore, its analysis is complemented by mlp, which calculates the proportion of labels that are never predicted [26]. analyzing the results with the mlp measure is important since traditional multi-label measures cannot correctly identify when some individual labels are not correctly predicted for any given instance. however, it is possible to identify the predictive performance of each label for each instance individually. thus, mlp is an important measure to identify labels the classifiers could not learn and thus are never predicted. equations 3, 4, 5, and 6 show the evaluation measures used in the experiments where: \(\mathcal {l}\) is the set of all labels in the label space, and \(tp_{j}\), \(fp_{j}\) and \(fn_{j}\) are, respectively, the number of true positives, false positives, and true negatives for class \(y_j\). 5 results, analysis and discussion we generated 20 different variations of our proposal (table 2). considering k-nn sparsification, we experimented three values for k (k = 1, k = 2 and k = 3). considering threshold sparsification, we considered two alternatives: t0, which cuts only the self-loops from the graphs, and t1, which cuts 10% of edges from the graphs. table 2 also shows the global and local methods and two variations of random methods, one using hierarchical community detection methods and another using non-hierarchical community detection methods. table 2. characteristics of the compared methods and variations.full size table 5.1 community detection methods selected for the rogers-tanimoto similarity measure, the most chosen hierarchical community detection methods were edge betweenness for threshold sparsification and walktrap for \(k-\)nn sparsification. in contrast, for jaccard index, walktrap was the most chosen in both sparsification types. in the case of the non-hierarchical methods, infomap was the most chosen community detection method for both similarity measures and sparsification types. finally, for random communities, the hierarchical method most chosen was walktrap, while infomap was the most chosen for non-hierarchical methods. 5.2 hybrid partitions selected for birds, emotions, eukaryotepseaac, and yeast datasets, the best partitions chosen were close to the local partition. in contrast, eisen, gnegativego, gpostivigo, scene, and yelp datasets had the best partitions chosen close to the global partition. considering the langlog dataset, the partition most frequently chosen was similar to the local. in contrast, in pheno dataset, there is a balance between partitions near the global or local ones. finally, the best partitions for the other datasets vary, but a partition close to the global partition configuration was more frequently chosen. our experiments observed that the best hybrid partitions generated by the chosen community detection methods are similar to the global partition. this can be one reason our performance results are competitive compared with other partitions, overcome the global, and are not superior to the local ones for some datasets. 5.3 classifier’s performance figure 3 presents the number of wins, losses, and ties considering the 24 methods and 20 datasets. in the figure, blue indicates wins, purple indicates losses, and green indicates ties. considering the performances over the 20 datasets in both maf1 and mlp measures, the local approach tends to obtain the best results, followed by the results obtained with the hybrid partitions. on the other hand, the global approach obtained the worst results. furthermore, the theoretical complexity score (table 1) presents high values for all datasets except emotions and flags. this indicates that it is hard to learn label correlations in most of the datasets. fig. 3. graphical representations of wins, losses, and ties for each method. full size image the classifier’s performance resultsfootnote 4, in several datasets, using random partitions outperformed its performance when using global partitions. when comparing h.ra and nh.ra with lo, the random partitions led to competitive results. regarding the sparsification methods, k-nn obtained slightly better performances when compared to threshold sparsification. the clusters of the partitions found by the community detection methods for each dataset were similar and sometimes the same. by all indications, our method is not significantly affected by sparsification with k-nn. we also notice that the results from graphs built with the jaccard index and hierarchical methods outperformed the ones with the rogers-tanimoto similarity, while in non-hierarchical methods, rogers-tanimoto were generally better than the jaccard index. finally, in a method-by-method pairwise comparison, the local partitions performed better than all other partitions on an average of 16 datasets. meanwhile, hybrid and random partitions achieved the best results on an average of 10 datasets. for the mlp measure, the resulting values for many datasets are about 0.9. as the best value for this measure is 0.0, a value near 1.0 indicates that not all labels were correctly learned nor predicted by the classifier. the pheno dataset had the worst result when using the local partition, 0.977, which means that the classifier did not correctly predict 97% of all labels. in contrast, at least 40% of the labels were predicted by the other methods, indicating that the correlated labels were helpful in the learning process. considering the maf1 average values over the 20 datasets, most of the methods obtained performance values between 0.3 and 0.2, which is considered low. the best results (close to 1.0) were obtained in the datasets gpositivego and gnegativego, which have few labels. similar results can also be found in related studies, where the authors showed that multi-label classifiers could not correctly learn and predict the labels and correlations. finally, our results show that the traditional local and global approaches cannot correctly learn label correlations since random generated partitions led to very similar results in many cases. 5.4 statistical tests to verify the statistical significance of our results, we executed the friedman test (\(\alpha = 0.05\)), followed by the nemenyi post-hoc test. they are adequate when comparing many classifiers over multiple datasets [10]. the respective friedman p-values were 5.75e-09 and 9.27e-14, and the critical distances are shown in figs. 4 and 5. no statistically significant differences were detected between the results obtained with the local approach and those obtained with the random partitions generated with non-hierarchical community detection methods. there were also no significant differences when comparing the global partition and the random partitions generated with hierarchical community detection methods. fig. 4. critical diagram for the maf1 results. full size image fig. 5. critical diagram for the missing label problem results. full size image when comparing local and hybrid partitions from hierarchical community methods, there were no significant differences, meaning that hybrid partitions led to competitive results. the same occurs with global partitions and hybrid partitions from non-hierarchical community methods. given that random generated partitions led to results equal or superior to the local and global approaches, we can conclude that local and global-based methods still need many improvements. thus, our results show that the conventional and widely-used local and global approaches may not correctly learn label correlations. 6 conclusions and future works in this paper, we presented an approach that aims to improve multi-label classifiers’ performances by partitioning the multi-label dataset into in-between local and global partitions. these so-called hybrid partitions are obtained with community detection methods by learning label correlations in the label space. first, we modeled label correlations with the jaccard index and the rogers-tanimoto similarity measures and used the resulting similarity matrices to model graphs. next, we applied k-nn and threshold sparsification methods to prune edges from the graphs. we then applied seven community detection methods, three hierarchical and four non-hierarchical. for the non-hierarchical methods, we generated hybrid partitions directly using a training set and tested them with a test set. for hierarchical methods, we obtained dendrograms that were cut into levels to generate several hybrid partitions, also using a training set. then, we validated those partitions with a silhouette coefficient using a validation set. finally, the hybrid partition with the highest coefficient value was tested. our results showed that our proposed approach could obtain better or more competitive results in several datasets when compared to conventional local and global approaches. we also compared the results obtained by the global and local approaches with the ones obtained using random generated partitions. we could verify if the partitioning improved the prediction power of the classifier. we concluded that there is no vast improvement besides our competitive results, and independently of the partitioning used. according to the results obtained when using the missing label problem measure, this probably occurred because most labels were not learned by the classifier, even by traditional approaches. therefore, we can conclude that the local and global approaches still need many improvements and may not correctly learn label correlations. in future works, we intend to explore other multi-label evaluation measures, such as wrong label problem and constant label problem [26], and also use other base classifiers to test the generated partitions. with this, we will be able to compare the results from different classifiers, verify possible improvements, and recommend the best classifier and label partition for a given dataset. notes 1.https://dtai.cs.kuleuven.be/clus/. 2.https://github.com/cissagatto/bracis2023/. 3.computes the number of instances classified in all pairs of labels. 4.for the sake of space and better presentation, the tables with all the performance values were not included in this paper but are available in the repository. references basgalupp, m., cerri, r., schietgat, l., triguero, i., vens, c.: beyond global and local multi-target learning. inf. sci. 579, 508–524 (2021) google scholar  blondel, v.d., guillaume, j.l., lambiotte, r., lefebvre, e.: fast unfolding of communities in large networks. j. stat. mech. theory exp. 2008, p10008 (2008) google scholar  bogatinovski, j., todorovski, l., džeroski, s., kocev, d.: comprehensive comparative study of multi-label classification methods. expert syst. appl. 203, 117215 (2022) google scholar  chang, w., yu, h., zhong, k., yang, y., dhillon, i.s.: a modular deep learning approach for extreme multi-label text classification. corr abs/1905.02331 (2019) google scholar  charte, f., rivera, a., del jesus, m.j., herrera, f.: on the impact of dataset complexity and sampling strategy in multilabel classifiers performance. in: hybrid artificial intelligent systems, pp. 500–511. springer international publishing (2016) google scholar  choi, s., cha, s., tappert, c.c.: a survey of binary similarity and distance measures. j. systemics cybern. inform. 8, 43–48 (2010) google scholar  chu, y., et al.: dti-mlcd: predicting drug-target interactions using multi-label learning with community detection method. briefings in bioinform. 22, bbaa205 (2020) google scholar  clare, a., king, r.d.: predicting gene function in saccharomyces cerevisiae. in: proceedings of the european conference on computational biology (eccb 2003), september 27–30, 2003, paris, france, pp. 42–49 (2003) google scholar  clauset, a., newman, m.e.j., moore, c.: finding community structure in very large networks. phys. rev. e 70, 066111 (2004) google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) google scholar  garg, a., enright, c.g., madden, m.g.: on asymmetric similarity search. in: 2015 ieee 14th international conference on machine learning and applications (icmla) (2015) google scholar  gatto, e.c., ferrandin, m., cerri, r.: exploring label correlations for partitioning the label space in multi-label classification. in: 2021 international joint conference on neural networks (ijcnn) (2021) google scholar  lin, s.c., chen, c.j., lee, t.j.: a multi-label classification with hybrid label-based meta-learning method in internet of things. ieee access 8, 42261–42269 (2020) google scholar  luaces, o., díez, j., barranquero, et al., j.: binary relevance efficacy for multilabel classification. progress in artificial intelligence (2012) google scholar  melo, a., paulheim, h.: local and global feature selection for multilabel classification with binary relevance an empirical comparison on flat and hierarchical problems (2017) google scholar  mezo, i.: the r-bell numbers. j. integer sequences 14, a11 (2011) google scholar  mittal, r., bhatia, m.p.s.: classification and comparative evaluation of community detection algorithms. archives of computational methods in engineering (2020) google scholar  newman, m.e.j., girvan, m.: finding and evaluating community structure in networks. phys. rev. e 69, 026113 (2004) google scholar  nguyen, t.t., nguyen, t.t.t., luong, a.v., nguyen, q.v.h., liew, a.w.c., stantic, b.: multi-label classification via label correlation and first order feature dependance in a data stream. pattern recogn. 90, 35–51 (2019) google scholar  pliakos, k., vens, c., tsoumakas, g.: predicting drug-target interactions with multi-label classification and label partitioning. ieee/acm trans. comput. biol. bioinform. 18 1596–1607 (2021) google scholar  pons, p., latapy, m.: computing communities in large networks using random walks (long version) (2005) google scholar  raghavan, u.n., albert, r., kumara, s.: near linear time algorithm to detect community structures in large-scale networks. phys. rev. e 76, 036106 (2007) google scholar  read, j., pfahringer, b., holmes, g., frank, e.: classifier chains: a review and perspectives. j. artif. intell. res. 70, 683–718 (2021) google scholar  reichardt, j., bornholdt, s.: statistical mechanics of community detection. phys. rev. e 74, 016110 (2006) google scholar  rivolli, a., read, j., soares, c., pfahringer, b., de leon ferreira de carvalho, a.c.p.: an empirical analysis of binary transformation strategies and base algorithms for multi-label learning. machine learning (2020) google scholar  rivolli, a., soares, c., carvalho, a.c.p.d.l.f.d.: enhancing multilabel classification for food truck recommendation. expert systems (2018) google scholar  rosvall, m., bergstrom, c.t.: maps of random walks on complex networks reveal community structure. proceedings of the national academy of sciences 105 (2008) google scholar  rousseeuw, p.: silhouettes: a graphical aid to the interpretation and validation of cluster analysis. j. comput. appl. math. 20, 53–65 (1987) google scholar  sechidis, k., tsoumakas, g., vlahavas, i.: on the stratification of multi-label data. in: gunopulos, d., hofmann, t., malerba, d., vazirgiannis, m. (eds.) ecml pkdd 2011. lncs (lnai), vol. 6913, pp. 145–158. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-23808-6_10 chapter  google scholar  shahapure, k.r., nicholas, c.: cluster quality analysis using silhouette score. in: 2020 ieee 7th international conference on data science and advanced analytics (dsaa) (2020) google scholar  silva, t.c., zhao, l.: machine learning in complex networks. springer publishing company, incorporated (2016) book  math  google scholar  szymański, p., kajdanowicz, t., kersting, k.: how is a data-driven approach better than random choice in label space division for multi-label classification? entropy 18 (2016) google scholar  tahir, m.a.u.h., asghar, s., manzoor, a., noor, m.a.: a classification model for class imbalance dataset using genetic programming. ieee access 7, 71013–71037 (2019) google scholar  vens, c., struyf, j., schietgat, l., džeroski, s., blockeel, h.: decision trees for hierarchical multi-label classification. mach. learn. 73, 185–214 (2008) google scholar  warrens, m.j.: similarity coefficients for binary data: properties of coefficients, coefficient matrices, multi-way metrics and multivariate coefficients. master’s thesis, leiden university (2008) google scholar  zhang, m.l., li, y.k., liu, x.y., geng, x.: binary relevance for multi-label learning: an overview. front. comput. sci. 12, 191–202 (2018) google scholar  zhang, m.l., zhou, z.h.: multi-label neural networks with applications to functional genomics and text categorization. ieee trans. knowl. data eng. 18, 1338–1351 (2006) google scholar  zhang, m.l., zhou, z.h.: ml-knn: a lazy learning approach to multi-label learning. pattern recogn. 40, 2038–2048 (2007) google scholar  zhou, j.p., chen, l., guo, z.h., hancock, j.: iatc-nrakel: an efficient multi-label classifier for recognizing anatomical therapeutic chemical classes of drugs. bioinformatics 36, 1391–1396 (2020) google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. the authors also thank the brazilian research agencies fapesp and cnpq for financial support. author information authors and affiliations department of computer science, federal university of são carlos, rodovia washington luis, km 235, são carlos, sp, brazil elaine cecília gatto, alan demétrius baria valejo & ricardo cerri federal university of santa catarina, rua joão pessoa, blumenau, 2750, santa catarina, brazil mauri ferrandin authors elaine cecília gattoview author publications search author on:pubmed google scholar alan demétrius baria valejoview author publications search author on:pubmed google scholar mauri ferrandinview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar corresponding author correspondence to elaine cecília gatto . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper gatto, e.c., valejo, a.d.b., ferrandin, m., cerri, r. (2023). community detection for multi-label classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_6 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-label partitions multi-label learning label correlations label communities label problems publish with us policies and ethics profiles elaine cecília gatto view author profile mauri ferrandin view author profile ricardo cerri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature bluex: a benchmark based on brazilian leading universities entrance exams | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper bluex: a benchmark based on brazilian leading universities entrance exams conference paper first online: 12 october 2023 pp 337–347 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) bluex: a benchmark based on brazilian leading universities entrance exams download book pdf download book epub thales sales almeida  orcid: orcid.org/0009-0006-9568-93319,10, thiago laitz  orcid: orcid.org/0000-0001-7205-20949,11, giovana k. bonás  orcid: orcid.org/0009-0001-9460-83539 & … rodrigo nogueira  orcid: orcid.org/0000-0002-2600-60359,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 536 accesses 13 citations abstract one common trend in recent studies of language models (lms) is the use of standardized tests for evaluation. however, despite being the fifth most spoken language worldwide, few such evaluations have been conducted in portuguese. this is mainly due to the lack of high-quality datasets available to the community for carrying out evaluations in portuguese. to address this gap, we introduce the brazilian leading universities entrance exams (bluex), a dataset of entrance exams from the two leading universities in brazil: unicamp and usp. the dataset includes annotated metadata for evaluating the performance of nlp models on a variety of subjects. furthermore, bluex includes a collection of recently administered exams that are unlikely to be included in the training data of many popular lms as of 2023. the dataset is also annotated to indicate the position of images in each question, providing a valuable resource for advancing the state-of-the-art in multimodal language understanding and reasoning. we describe the creation and characteristics of bluex and establish a benchmark through experiments with state-of-the-art lms, demonstrating its potential for advancing the state-of-the-art in natural language understanding and reasoning in portuguese. the data and relevant code can be found at https://github.com/portuguese-benchmark-datasets/bluex. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others where am i? where am i going, and how do i get there?: increasing learner agency through large-scale self assessment in language learning chapter © 2019 designing llm-based study guides and personalized feedback in brazilian higher education: a skill-focused approach to learning and assessment perceptions chapter © 2025 case study involving interdisciplinary academic projects for reading fluency assessment and analysis chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics languages lisp machine translation machine learning predictive markers 1 introduction recent advances in language models (lms) have generated significant interest due to their demonstrated capabilities on a wide range of language tasks, including text classification, language translation, and text generation [3, 7]. lm performance has been particularly impressive on standardized tests, which present challenging questions requiring high levels of domain-specific knowledge and reasoning. for instance, recent benchmarks on gpt-4 [16] showed that it can achieve human-level performance on a variety of graduate-level benchmarks. despite the impressive performance of lms on standardized tests, few evaluations have been performed in portuguese [15], partially due to the lack of available datasets in the language. this lack of high-quality, standardized datasets presents a significant challenge for researchers interested in developing and evaluating lms in portuguese. to address this gap for brazilian portuguese, we introduce bluex, a dataset consisting of entrance exams for the two leading universities in brazil. our dataset offers a rich source of high-quality high school-level questions annotated with their respective subjects, as well as flags indicating the required capabilities necessary to respond accurately to the questions, such as knowledge of brazilian culture and the application of mathematical reasoning. these annotations can be used to evaluate the performance of lms on a variety of subjects and capabilities such as domain-specific knowledge and reasoning. additionally, bluex includes a collection of recently administered entrance exams that are unlikely to be included in the training data of many currently popular lms. in anticipation of the emergence of multimodal models that combine text and image understanding, we have annotated bluex to indicate the position of images in each question. additionally, we have included all necessary images with the dataset to facilitate research on multimodal language tasks. we believe that this resource will be essential in evaluating the performance of models that reason with both text and image inputs to solve complex problems. in this paper, we describe the creation and characteristics of bluex and establish a benchmark through experiments with state-of-the-art lms. our findings suggest that bluex provides a valuable resource for benchmarking and advancing the state-of-the-art in natural language understanding and reasoning in portuguese. this is particularly relevant since even the current state-of-the-art models, such as gpt-4, still have considerable room for improvement and do not achieve the highest cutoff grades for both universities. 2 related work in the realm of portuguese natural language processing (nlp) datasets, there appears to be a limited availability. for question-answering tasks, faquad [21] is available, which exhibits an extractive style akin to squad [18]. it features questions concerning brazilian higher education institutions, with documents sourced from a federal university and supplemented by wikipedia articles. another option is the multilingual knowledge questions and answers (mkqa) dataset, which covers 26 languages [12]. this dataset was generated by selecting 10,000 queries from the natural questions dataset [10] and acquiring new passage-independent answers for each question. subsequently, human translators translated the questions and answers into 25 non-english, typologically diverse languages, including portuguese. regarding sentence entailment tasks, assin 1 and 2 [5, 19] are available. these datasets encompass recognizing textual entailment (rte), also referred to as natural language inference (nli), and semantic textual similarity (sts) tasks. the former involves predicting if a given text (premise) implies another text (hypothesis), while the latter quantifies the semantic equivalence between two sentences. the portuguese language understanding evaluation (plue) benchmark [6] provides portuguese translations of the glue [26], snli [1], and scitail [8] datasets. these translations have been generated using automatic translation tools including google translate and opusmt [24]. the winograd schema challenge (wsc) dataset [9] contains pairs of sentences with minimal differences, featuring an ambiguous pronoun that is resolved divergently between the two sentences. melo et al. [13] manually translated and adapted this dataset to portuguese. for sentiment analysis tasks, the tweetsentbr dataset [2] consists of 15,000 tweets related to the tv show domain, collected between january and july 2017. the tweets were manually annotated by seven annotators into three classes: positive, neutral, and negative. the multilingual amazon slu resource package (slurp) for slot-filling, intent classification, and virtual assistant evaluation (massive) [4] is a 1m-example dataset containing realistic virtual utterances in 51 languages, including portuguese. professional translators translated the dataset from english, and it is annotated for slot (55 classes) and intent (60 classes) prediction tasks. a dataset more closely related to bluex is the enem-challenge dataset [22], which includes the editions of the brazilian national exam, exame nacional do ensino medio (enem), from 2009 to 2017. additionally, nunes et al. [15] introduced a dataset containing the enem exam of 2022, the same paper evaluated the performance of lms such as gpt-3.5-turbo and gpt-4 on both the enem-challenge and the enem 2022 datasets. 3 the bluex dataset 3.1 dataset creation bluex is a dataset comprising more than 1,000 multiple choice questions from the entrance exams of the two leading universities in brazil, unicamp and usp, administered between 2018 and 2023. the dataset was created by automatically extracting each question text, alternatives, and related images using scripts, and subsequently each example was manually annotated to correct extraction errors and provide additional metadata such as image positioning. 3.2 annotated question metadata the annotated metadata is described below. prior knowledge (prk) indicates whether the question requires knowledge from outside of what has been provided in the question, such as familiarity with a particular author’s work or a specific mathematical formula. text understanding (tu) indicates whether the question requires understanding of a particular text. image understanding (iu) indicates whether the question requires understanding of an image. it should be noted that not all questions with images require their understanding to answer the question. mathematical reasoning (mr) indicates whether the question requires mathematical reasoning, such as the ability to perform calculations and symbolic manipulations. multilingual (ml) indicates whether the question requires knowledge of two or more languages, such as questions designed to test english skills of portuguese speakers. brazilian knowledge (bk) indicates whether the question involves knowledge specific to brazil, such as brazilian history, literature, geography, or culture. subjects a list of subjects related to the question, such as geography, physics, etc. related images a list of all the related images for the question. alternative type indicates whether the answer choices are presented as text or as images. this is important because some questions may use images as answer choices, which requires different processing techniques than questions with only textual answers. by providing such annotations along with the questions we aim to facilitate research into language understanding and reasoning in portuguese for both pure language models and multimodal models. we believe that bluex will be a valuable resource for researchers to evaluate and improve the performance of future language models in the context of portuguese-language standardized tests. 3.3 image positioning many of the questions in the exams require a contextual or informational understanding of images. despite active research in the field of multimodal models, models that can adeptly process both text and image data and yield satisfactory results remain scarce in the public domain. we believe that bluex can serve as an essential evaluation tool for such models. anticipating the use of models that will process images and text in an interleaved manner, we also provide precise information regarding the placement of images within the question, as illustrated in fig. 1. fig. 1. example of image annotation in bluex. full size image 3.4 dataset distribution the bluex dataset covers a wide range of high school subjects, including mathematics, physics, chemistry, biology, history, geography, english, philosophy and portuguese, as well as multidisciplinary questions that involve two or more subjects. the distribution of questions is shown in table 1, where we also provide the distribution for the subset of questions without images, which accounts for approximately 58% of the total dataset. furthermore, table 2 shows the distribution of the dataset across annotated categories, as explained in sect. 3.2. we observe that the majority of questions require specific knowledge and the ability to comprehend text, two expected capabilities in students taking these exams. note that any given question can be part of multiple categories. table 1. distribution over subjects.full size table table 2. distribution over categories.full size table 4 results to enable future comparisons, we evaluated our dataset using several language models, ranging from 6b to 66b parameters, including openai’s gpt-4 and gpt-3.5-turbo models. our experiments were conducted using large language models with no specific training for this task. each model was provided with one example in the input and then asked to answer a question from the test set. the example was randomly selected from an exam of the same university as the current question, but from a different year. for example, if the current question is from unicamp 2019, the example provided in the prompt would be a question from a unicamp exam, but not from 2019. we excluded all questions containing images from our experiments since the language models we used can only process text. this resulted in a total of 638 questions being used, which corresponds to approximately 60% of the dataset. table 3. accuracy in the bluex dataset.full size table table 3 summarizes our experimental findings, including the mean score achieved by exam-taking students, as well as the mean cutoff score of the most competitive major, which is medicine in both universities.footnote 1 the bluex column shows the accuracy of the whole subset used in the evaluation, while the unicamp and usp columns account for only the questions from the respective universities. the mr and bk columns account only for questions that include those categories. among the language models tested in the 7b-parameter range, sabiá [17], a model further pre-trained in portuguese, consistently outperformed all other models, coming close to matching the average human score. among the open-source models in the 60b-parameter range, llama 65b [25] significantly outperformed opt 66b [30] and achieved similar performance to gpt-3.5-turbo. sabiá 65b achieved better performance than gpt-3.5-turbo but still lagged behind gpt-4 by ten points. gpt-4 was by far the best model in our evaluations but did not achieve an average score high enough to pass in medicine, the most competitive major. it is worth noting that the average and cutoff scores provided in table 3 are computed taking into account the whole exam, including questions with images, while the scores obtained by the language models utilize only the subset of questions with no images. we also conducted a more detailed analysis of the models’ performance by examining their ability to handle specific question types. table 3 presents the findings for questions that required mathematical reasoning (mr) and brazilian knowledge (bk). we observe that, with the exception of gpt-4, all models struggled to perform significantly better than random chance in questions that required mathematical reasoning. even gpt-4 only achieved an accuracy of 44% in mr questions. on the other hand, when considering questions that require brazilian knowledge, sabiá greatly outperformed all the other models in the 7b-parameter range, indicating that the extra pretraining in portuguese provided the model with additional regional knowledge. in the 60b-parameter range, sabiá also showed improvement over llama, increasing the accuracy in these questions by 10 points and slightly outperforming gpt-3.5-turbo. nevertheless, it could not match the remarkable performance of gpt-4. fig. 2. accuracy of the best models over the years of the exams. full size image moreover, fig. 2 displays the performance of the top four models on the exams conducted each year. it can be observed that the models have a small variance between the years, which is expected as the difficulty of each exam and the number of questions in the subset vary across years. a surprising result, however, is the increased performance that all models seem to exhibit in 2023. the average and highest cutoff scores also increased slightly over the years, indicating that the exams became slightly easier in recent years. since the 2023 exams were very recently administered, it is unlikely that they are part of any of the studied models’ training data. therefore, since the models’ performance in the most recent years is comparable to that in older exams, it is reasonable to assume that the models are not merely memorizing the answers for the questions in the dataset. 5 conclusion this work introduced bluex, a new dataset that consists of 13 college entrance exams applied between 2018 and 2023 from two of the leading brazilian universities, unicamp and usp. each question of these exams was extensively annotated to help measure different abilities across multiple subjects in portuguese. beyond that, by providing images and their corresponding positions within the text, bluex is one of the few portuguese datasets that are ready to evaluate multimodal models. we provide results from multiple lms as baselines and reference scores based on students performance to facilitate future comparisons. we believe that bluex will be a important benchmark in the evaluation of the portuguese capabilities of future models. 6 future work the models used in this study employed a single in-context example. however, there’s room for further investigation, such as determining whether increasing the number of few-shot examples could boost the performance of each model, as well as assessing their zero-shot performance. furthermore, nunes et al. [15] showed that gpt-4’s performance on enem questions was significantly boosted when chain-of-thought prompts [28] were used. adopting a similar approach here could potentially lead to performance improvement. finally, regarding multimodal models, their performance can be assessed utilizing the bluex dataset. this provides an opportunity for researchers to investigate the models’ capabilities in integrating visual and textual information to address high school level questions. notes 1.the average and cutoff scores are reported by the entities responsible for administering the exams. the results presented in table 3 are the average of all the exams contained in the bluex dataset. references bowman, s., angeli, g., potts, c., manning, c.d.: a large annotated corpus for learning natural language inference. in: proceedings of the 2015 conference on empirical methods in natural language processing, pp. 632–642 (2015) google scholar  brum, h.b., das graças volpe nunes, m.: building a sentiment corpus of tweets in brazilian portuguese (2017) google scholar  chowdhery, a., et al.: palm: scaling language modeling with pathways (2022) google scholar  fitzgerald, j., et al.: massive: a 1 m-example multilingual natural language understanding dataset with 51 typologically-diverse languages (2022) google scholar  fonseca, e., santos, l., criscuolo, m., aluisio, s.: assin: avaliacao de similaridade semantica e inferencia textual. in: 12th international conference on computational processing of the portuguese language, tomar, portugal, pp. 13–15 (2016) google scholar  gomes, j.r.s.: plue: portuguese language understanding evaluation (2020). https://github.com/jubs12/plue hoffmann, j., et al.: training compute-optimal large language models (2022) google scholar  khot, t., sabharwal, a., clark, p.: scitail: a textual entailment dataset from science question answering. in: proceedings of the aaai conference on artificial intelligence, vol. 32 (2018) google scholar  kocijan, v., lukasiewicz, t., davis, e., marcus, g., morgenstern, l.: a review of winograd schema challenge datasets and approaches. arxiv preprint arxiv:2004.13831 (2020) kwiatkowski, t., et al.: natural questions: a benchmark for question answering research. trans. assoc. comput. linguist. 7, 453–466 (2019) article  google scholar  lin, x.v., et al.: few-shot learning with multilingual language models (2022) google scholar  longpre, s., lu, y., daiber, j.: mkqa: a linguistically diverse benchmark for multilingual open domain question answering. trans. assoc. computat. linguist. 9, 1389–1406 (2021) article  google scholar  de melo, g., imaizumi, v., cozman, f.: winograd schemas in portuguese. in: anais do xvi encontro nacional de inteligência artificial e computacional, pp. 787–798. sbc (2019) google scholar  muennighoff, n., et al.: crosslingual generalization through multitask finetuning (2022) google scholar  nunes, d., primi, r., pires, r., lotufo, r., nogueira, r.: evaluating gpt-3.5 and gpt-4 models on brazilian university admission exams (2023) google scholar  openai: gpt-4 technical report (2023) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models (2023) google scholar  rajpurkar, p., zhang, j., lopyrev, k., liang, p.: squad: 100,000+ questions for machine comprehension of text. in: proceedings of the 2016 conference on empirical methods in natural language processing, pp. 2383–2392 (2016) google scholar  real, l., fonseca, e., gonçalo oliveira, h.: the assin 2 shared task: a quick overview. in: quaresma, p., vieira, r., aluísio, s., moniz, h., batista, f., gonçalves, t. (eds.) propor 2020. lncs (lnai), vol. 12037, pp. 406–412. springer, cham (2020). https://doi.org/10.1007/978-3-030-41505-1_39 chapter  google scholar  de la rosa, j., ponferrada, e.g., villegas, p., de prado salas, p.g., romero, m., grandury, m.: bertin: efficient pre-training of a spanish language model using perplexity sampling (2022) google scholar  sayama, h.f., araujo, a.v., fernandes, e.r.: faquad: reading comprehension dataset in the domain of brazilian higher education. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 443–448. ieee (2019) google scholar  silveira, i.c., mauá, d.d.: advances in automatically solving the enem. in: 2018 7th brazilian conference on intelligent systems (bracis), pp. 43–48. ieee (2018) google scholar  taori, r., et al.: stanford alpaca: an instruction-following llama model (2023). https://github.com/tatsu-lab/stanford_alpaca tiedemann, j., thottingal, s.: opus-mt building open translation services for the world. in: proceedings of the 22nd annual conference of the european association for machine translation (eamt), lisbon, portugal (2020) google scholar  touvron, h., et al.: llama: open and efficient foundation language models (2023) google scholar  wang, a., singh, a., michael, j., hill, f., levy, o., bowman, s.r.: glue: a multi-task benchmark and analysis platform for natural language understanding. in: international conference on learning representations (2019). https://openreview.net/forum?id=rj4km2r5t7 wang, b., komatsuzaki, a.: gpt-j-6b: a 6 billion parameter autoregressive language model, may 2021. https://github.com/kingoflolz/mesh-transformer-jax wei, j., et al.: chain of thought prompting elicits reasoning in large language models. in: oh, a.h., agarwal, a., belgrave, d., cho, k. (eds.) advances in neural information processing systems (2022). https://openreview.net/forum?id=_vjqlmesb_j le scao, t., et al.: bloom: a 176b-parameter open-access multilingual language model (2023) google scholar  zhang, s., et al.: opt: open pre-trained transformer language models (2022) google scholar  download references author information authors and affiliations state university of campinas (unicamp), campinas, brazil thales sales almeida, thiago laitz, giovana k. bonás & rodrigo nogueira maritaca ai, campinas, brazil thales sales almeida & rodrigo nogueira neuralmind ai, campinas, brazil thiago laitz authors thales sales almeidaview author publications search author on:pubmed google scholar thiago laitzview author publications search author on:pubmed google scholar giovana k. bonásview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar corresponding author correspondence to thales sales almeida . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi 7 appendix 7 appendix 1.1 7.1 prompt for evaluation the prompt used for all the experiments in this paper is shown in the fig. 3. fig. 3. example of prompt used in the experiments, the question was translated into english for the convenience of readers. the text in red is the expected output. (color figure online) full size image table 4. results for each model by subject in bluex.full size table 1.2 7.2 benchmark per subject table 4 provides a detailed report of each model achieved accuracy by subject. questions that were associated with more than one subject contributed to the accuracy of both scores. for example, a question related to mathematics and english will be taken into account when calculating the accuracy of both mathematics and english subjects. rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper almeida, t.s., laitz, t., bonás, g.k., nogueira, r. (2023). bluex: a benchmark based on brazilian leading universities entrance exams. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_22 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_22 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature multimodal and hybrid models for predicting scd risk in chagas cardiomyopathy | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper multimodal and hybrid models for predicting scd risk in chagas cardiomyopathy conference paper first online: 30 january 2025 pp 224–237 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) multimodal and hybrid models for predicting scd risk in chagas cardiomyopathy download book pdf download book epub isabelly p. da costa  orcid: orcid.org/0009-0003-3967-42809, bruno m. p. takazono  orcid: orcid.org/0009-0002-4579-34149, carlos h. l. cavalcante  orcid: orcid.org/0000-0001-9395-83389, joão p. v. madeiro  orcid: orcid.org/0000-0001-6511-670711 & … roberto c. pedrosa  orcid: orcid.org/0000-0002-3270-159510  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 368 accesses abstract predicting sudden cardiac death (scd) in chagas cardiomyopathy (cc) remains a challenge. an estimated 6 to 7 million people worldwide are affected by this condition. to address this challenge, we propose a novel intelligent system that utilizes multimodal and hybrid machine learning (ml) models to predict scd risk in cc patients. we evaluated different approaches to combine clinical features (tabular data) and temporal features extracted from ecg-holter exams. our results indicate that the hybrid model (rnn with mlp) achieved the highest recall (sensitivity) of 91.63% and 81.08% of auc. this study aims to develop a predictive model to support clinical decision-making in the context of scd in cc. the model can potentially aid in risk stratification, enabling early intervention and potentially saving lives. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others prediction models of sudden death from chronic chagas cardiomyopathy chapter © 2024 prediction of sudden cardiac death: looking beyond ejection fraction article 10 april 2023 multimodality imaging predictors of sudden cardiac death article 02 december 2019 1 introduction the world health organization (who) estimates that between 6 and 7 million people are infected with chagas disease (cd). cd is endemic in brazil and latin america, and its incidence is increasing steadily across north america, europe, japan and australia. cd, caused by the parasite trypanosoma cruzi, is a persistent and potentially lethal illness. it affects up to 1.5 million (30%) people worldwide who have chagas cardiomyopathy (cc). this condition significantly compromises the health of infected individuals, leading to a shortened lifespan or death due to sudden cardiac events (scd) [27]. scd is one of the leading causes of death worldwide, often presenting as an abrupt collapse with a documented loss of vital signs, which may lead to attempts to restore circulation through cardiopulmonary resuscitation (cpr) [8]. in the context of cd, scd accounts for approximately 45% of deaths among patients diagnosed with cc [19], with an occurrence rate (2.4% per year) higher than that of the general population [20]. the search for early and advanced treatments for patients with cc who are susceptible to scd has led to measures such as implantable cardioverter-defibrillators (icds), which can extend patients’ lifespans [20]. however, these interventions are limited by the unpredictability of scd, a challenge that is well-documented in the literature [12, 14, 19]. in this context, providing systems that aid in the diagnosis of scd risk in cc patients becomes a valuable tool, especially given the high prevalence of this disease in some countries and the unpredictable nature of scd. therefore, this study proposes a model architecture that allows for the incorporation of various data types (tabular and serial data) and hybrid ml models, eventually supporting clinical decision-making in the context of scd in cc. additionally, this work developed a first prototype of a user-friendly computational system to assist doctors in patient care. the system utilizes a predictive model built with a minimal feature set, focusing on patients not considered at high risk of scd. by combining the output of a recurrent neural network (rnn), which effectively captures temporal features, with tabular data, the approach leverages both sequential and non-sequential information. this combined approach ensures the model’s scalability to handle diverse input data types and tasks. consequently, it enables the exploration of potential impacts and the relevance of new features for predicting scd risk in cc patients. this study is organized into five sections. the first section provides an introduction to the research problem. the second section discusses related works that align with the themes of the present study. the third section explores the methodological aspects of the research, outlining the proposed model and system architecture. the fourth section presents the experimental results, detailing the system’s development and the model’s performance. finally, the last section offers concluding considerations of the developed model and suggestions for future work. 2 related work intelligent systems (is) with machine learning (ml) have a significant impact on assisting diagnosis in various areas of healthcare. their use can lead to safer and more precise medical practices [13]. in the specific context of cc, some studies have shown that ml algorithms can classify patients with cd with the same or higher precision than physicians [3]. despite the global significance of scd [15], its occurrence in the specific context of cc has been scarcely explored, especially regarding the use of ml computational tools by end-users. furthermore, a comprehensive review of the literature reveals a current lack of studies dedicated to addressing this knowledge gap. one study has a limited approach, as it only performed classic linear analysis and did not fully explore the potential of combining different parameters [23]. another research study utilized eight variables from 78 patients, applying heart rate variability (hrv) and heart rate turbulence (hrt) techniques to ecg-holter recordings to investigate scd in chagas heart disease (chhd). the attribute set was reduced using forward and reverse stepwise approaches. this work employed the k-nearest neighbors (knn) classifier and leave-one-out (loo) cross-validation (cv) [1]. however, the lack of sensitivity and methodology for handling imbalanced data makes reproducing this study unfeasible. a third study employed several classic ml methods in a dataset with 218 patients, using seven cardiac restitution metrics (crm) extracted from a 4-hour segment of a 24-hour ecg-holter along with clinical data. this study reported sensitivity results of 90% and identified the gaussian naive bayes algorithm as the best-performing model [4]. however, there was no exclusion of high-risk patients in this research. [18] investigated the use of action potential duration restitution (apdr) dynamics, using a cross-sectional study of 221 patients, the authors extracted apdr dynamics metrics from 4-hour ecg segments and analyzed them alongside clinical variables using a classification tree. the study identified the %qtend tendq > 1, as the most relevant predictor of scd in the model with a 0.96 auc. however, it is limited by its inclusion criteria that included patients who met the following: (1) had regular appointments at the clementino fraga filho university hospital (hucff-ufrj), rio de janeiro, brazil, (2) had resided outside an area with endemic chagas disease transmission for at least 20 years, (3) were classified as having a low-intermediate rassi risk score, (4) had their vital status checked until the end of february 2017, and (5) had an established diagnosis of chagas cardiomyopathy (cc), confirmed by at least two positive serologic tests for antibodies against trypanosoma cruzi and characteristic 12-lead electrocardiogram changes. 3 methodology about the scientific method, this work adheres to the action research methodology, aiming to propose and test solutions for problems within a specific context [17]. as for its approach, this study can be categorized as quantitative, centered on problem-solving through theory testing and the numerical quantification of variables [10]. the project was organized into developing prediction models and creating a prototype web system, aspects that are detailed below. 3.1 data collection the study utilized data from a clinical follow-up program conducted between 1992 and 2023 at clementino fraga filho university hospital of the federal university of rio de janeiro. patients with a high degree of cardiac compromise were excluded, as they already had indications for potential interventions to ensure survival, such as crt implantation or other treatments. samples meeting the following criteria were removed from the dataset: nyha class 4, severe dysfunction classification, high rassi score, 2005 classification guideline of c or d, and all patients with a teicholz ef below 0.40. after this exclusion, 120 patients remained for analysis, including 19 (16%) who experienced scd and 101 (84%) who did not. the gender distribution among these patients was 42 (35%) males and 78 (65%) females. 3.2 standardization process the dataset used consists of two subsets: the first derived from tabular data and the second from time series extracted from ecg-holter signals using the sames methods used by [16, 21]. regarding the number of attributes, the dataset consists of 49 tabular features and 27 time series features. all features are grouped into 7 categories: clinical data, echocardiogram (echo), electrocardiogram (ecg), electrocardiogram holter (ecg-holter), cardiology guideline classification (cgc), treatments, and cardiac restitution metrics (crm). notably, the crm group exclusively contains the 27 time series features, while the remaining 49 tabular features are distributed among the other four categories. the groups and features used in this work are detailed in table 1. table 1. description of the sequential and non-sequential features used in this work, grouped into 7 groups. the 49 tabular features are classified into the groups: clinical data, echo, ecg, ecg-holter, cgc, and treatments. the 27 time series features extracted from ecg-holter are exclusively in the crm.full size table initially, 14 features were excluded due to confidentiality concerns or because they contained irrelevant information for this study, which resulted in 49 final tabular attributes. among these attributes, 30 were binary, 12 were scalar, and 7 were categorical. the seven categorical attributes underwent one-hot encoding, transforming each category into a binary attribute, thereby bringing the dataset back to 63 attributes. subsequently, the data was normalized using the minmax technique. regarding the time series extracted from the ecg-holter, each series initially had 30 attributes, but two were excluded—one constant and the other the target variable indicating scd. this left 28 final attributes: 27 corresponding to crm and 1 indicating the time of each sample in the series; all of these attributes were scalar. these 27 time series features represent an initial result of an ongoing study that is processing ecg-holters from chagas disease patients. in this work, we propose an architecture that utilizes both sequential and non-sequential data for an initial analysis of the results. this architecture concatenates the output of a recurrent neural network (rnn) with tabular data as input to a multilayer perceptron (mlp). 3.3 tools the python programming language was chosen for this study due to its widespread adoption in the field of ml. the google colab platform was utilized, taking advantage of its cloud-hosted jupyter notebooks and the cpu resources provided by google, which facilitated the execution of experiments for the prediction model. for model development, two primary libraries were used: scikit-learn and pytorch. these libraries were selected for their wide range of algorithms and functionalities available for developing and evaluating ml models. the predictive model with the best performance was deployed through an application programming interface (api) using the pickle5 library. regarding the system prototype, a rest architecture with docker was defined for both the backend and frontend applications. the flask framework was employed for backend construction, integrated with a postgresql database. frontend development was implemented using next.js. 3.4 classification experiments the experiments to create the predictive model were conducted under two scenarios. in each scenario, the data were split into a 67% training set and a 33% test set for robust evaluation and all experiments were repeated 10 times to calculate the average of the metrics. synthetic data generation was not considered for either scenario due to the specific nature of the data, scope limitations, and time constraints. in the first scenario, exclusively tabular data from 120 patients was used. three models were employed: multilayer perceptron (mlp), xgboost (xgb), and random forest (rf). the selection of tree-based models was motivated by their established performance in handling tabular data, as documented in the literature [7, 25]. for the experiments in this scenario, scikit-learn was utilized. the hyperparameters for the classification algorithms were chosen from the following values: multilayer perceptron (mlp): hidden_layer_size: (200, 50, 30), (100, 50, 10), (100, 50), (200, 100), (500, 250), (20,), (50,), (100,), (10,), (200,); activation: tangencial e relu.; solver: sgd e adam; alpha: 0.0001, 0.005, 0.05; learning_rate: constant e adaptive. random forest (rf): n_estimators: 100, 300, 500, 800, 1200; max_depth: 5, 8, 15, 25, 30; min_samples_split: 2, 5, 10, 15, 100; min_samples_leaf: 1, 2, 5, 10; max_features: 1, 2, 3, 4, 5. xgboost (xgb): min_child_weight: 1, 5, 10.; gamma: 0.5, 1.0, 1.5, 2.0, 5.0; subsample: 0.6, 0.8, 1.0; colsample_bytree: 0.6, 0.8, 1.0; max_depth: 3, 4, 5, 8, 12; eta: 0.3, 0.2, 0.1, 0.05, 0.01, 0.005. for the second scenario, tabular data and time series data from each patient’s ecg-holter exams were utilized. in this scenario, we present a formal description of a composite system employing an elman rnn and a multilayer perceptron (mlp) for tasks involving time series and tabular data. the overall process can be decomposed into two stages: (1) the elman rnn, responsible for handling the inherent sequential nature of the time series data [5], and (2) the multilayer perceptron, which integrates the outputs generated by the rnn with the provided tabular data. the purpose of choosing a hybrid model using rnn + mlp is to observe, through the use of the rnn, whether the time series extracted from ecg exams add value to the scd problem in cc when concatenated with tabular data, in order to obtain a more efficient multimodal hybrid model. another point is that backpropagation ensures that both parts of the model (rnn + mlp) are adjusted in a coordinated and efficient manner, maximizing the model’s ability to capture and integrate temporal and tabular information. the output of the rnn at time step \( t \) can be mathematically described by equation below (i), where \( x_t \) represents the input vector at time \( t \), \( h_t \) is the hidden state at time \( t \), \( y_t \) is the output at time \( t \), \( b_y \) is the bias vector for output, and \( h_t \) is the hidden state. $$\begin{aligned} y_t = w_{hy} h_t + b_y \qquad (i) \end{aligned}$$ the \( h_t \) is given by the following equation (ii), where \( w_{xh} \) represents the weight matrix from input to hidden state, \( w_{hh} \) is the weight matrix from previous hidden state to current hidden state, \( w_{hy} \) is the weight matrix from hidden state to output, and \( b_h \) is the bias vector for hidden state. $$\begin{aligned} h_t = \sigma (w_{xh} x_t + w_{hh} h_{t-1} + b_h) \qquad (ii) \end{aligned}$$ assume that the vector \( z_t \) is the concat of the output of an rnn (\( y_t \)) and tabular data (\( d_t \)). the mathematical expression for processing \( z_t \) as the input to an mlp is described as follows, where $$ z_t = \left[ y_t || d_t \right] \qquad (iii) $$ the input to the mlp is \(a^{(0)} = z_t\). for an mlp with \( l \) hidden layers and an output layer (l + 1), the final output \( \hat{y} \) is showed in followed equitation (iv), where \( w^{(l)} \) represents the weight matrix of layer \( l \), \( b^{(l)} \) is the bias vector of layer \( l \), \( a^{(l)} \) is the activation vector of layer \( l \), and \( \sigma \) is the an activation function, such as relu, sigmoid, or tanh. $$ \hat{y} = a^{(l+1)} = \sigma (w^{(l+1)} a^{(l)} + b^{(l+1)}) \qquad (iv) $$ the summary of the process used in this work can described in four pass: (i) pass the time series through the rnn to obtain \( y_t \); (ii) concatenate \( y_t \) with \( d_t \) to get \( z_t \), (iii) pass \( z_t \) through the mlp to obtain the final output \( \hat{y} \) and (iv) the compute the error and propagate the gradients back through the mlp and then through the rnn to update the weights and biases. in scenario 1, the experiments were conducted using a feature selection process, considering a range of 5, 10, 15, 20, and 25 features, as well as using all 63 features. the gridsearchcv method was employed to select the top features in each configuration. all experiments were evaluated using leave-one-out (loo) and 5-fold cv. in the second scenario, two approaches were executed. in the first approach, the output of the rnn was directly concatenated with all tabular data features. in the second approach, the rnn output was concatenated only with each of the top selected features from the mlp model in scenario 1. both the rnn and mlp were implemented using pytorch. to facilitate data manipulation, two tensors were created for the tabular data: one containing the target labels and the other containing the attributes. during model training in scenario 2, the crossentropyloss loss function was automatically calculated gradients. backpropagation was carried out using the loss.backward() function to optimize the model by learning from errors and enhancing performance. the gradients were propagated through the entire network, encompassing both the rnn layer and the mlp. to handle the time series extracted from the ecg-holter exams of the 120 patients, two tensors were created as well. one tensor stored the values of the 27 crm variables, while the other stored the length of each time series. hyperparameter selection was carried out using the parametergrid function from scikit-learn. the rnn hyperparameters were chosen from the following values: recurrent neural network (rnn): hidden_layer_size: 8, 10, 16, 32, 50, 64, 100, 128; num_layers: 1; nonlinearity: relu; solven: sgd e adam.; learning_rate: 0.0001, 0.005, 0.001, 0.05 ou 0.01. 4 results 4.1 model prediction performance experiments were conducted using all features (63) and subsets (5, 10, 15, 20, and 25) of the most relevant features selected through a feature selection process. after the feature selection process was completed, the subset of features that yielded the best performance for each algorithm was selected. interestingly, certain features were frequently selected across various combinations of top features that led to the models (mlp, rf, xg) best performance. table 2 provides a detailed breakdown of the selected features for each model. table 2. selected tabular features present in the best performance of the experiments conducted with the models (mlp, rf, xgboost).full size table the results demonstrate four features (nsvt, amiodarone, “moderate classification”, and the “b2 2005 classification guideline”) were selected in all models. additionally, the features total ve and “intermediate rassi score” appear in two of the models. this consistency indicates the potential relevance of these features to the prediction of scd, which is crucial for comparing and validating these selected attributes with the state-of-the-art in medical literature. the best results from the feature selection process and from using all the features are presented for each model. in the first scenario, the best performance of the feature selection process was achieved by the multilayer perceptron (mlp) with 5 features, the random forest (rf) with 5 features, and the xgboost (xgb) with 10 features. these models were trained using only tabular data as input. in the second scenario, the rnn output was concatenated with the 5 best-performing features selected from the mlp model in scenario 1 and all 63 tabular features. to select the best performance for each tested configuration, we prioritized recall (sensitivity) and auc (area under the roc curve). this choice reflects the critical need to accurately identify patients at high risk for scd, maximizing the detection of potential cases consistent with other studies [1, 4, 6, 22, 23]. the table 3 displays the models used, the number of features employed, and the evaluation metrics including acc (accuracy), auc, recall (sensitivity), precision, and f1 score, along with their respective results and standard deviations. important to note, the feature quantity of 63 in the table is regarding the 49 features after one-hot encoding. table 3. prediction results of the xgb, mlp, and rf models in scenario 1 using only tabular data. results are presented with all 63 features and the best result with feature selection using the two types of cross-validation.(s= scenario; nf = number of features)full size table regarding only the scenario 1 with tabular data, the rf model performed better with 5 features obtained 89.33% recall and 95.38% auc. however, it presented a significant standard deviation in recall of almost 20%. in second place, the xgb model with 10 features acquired 83.85% recall and 91.44% auc, performing slightly better than the mlp with 5 features, that reached 83.68% recall and 90.51% auc. when evaluating the three models (mlp, rf, xgb), we observed that feature selection did not result in a significant performance loss; in fact, it improved the recall metric. this finding suggests that some of the features might not be relevant to the context. the high dimensionality of data can be a challenge for developing intelligent systems. however, reducing the number of features offers several advantages. these include creating systems that are interoperable for domain experts, leading to faster, more accurate decision-making and improved patient care (or other relevant outcome). in the second scenario, where the time series is added, the experiments obtained 91.63% recall with all features (27 temporal series + 63 tabular features) and 90.13% with selected features (27 temporal series + 5 tabular features). interestingly, the approach that concatenates all features (63 tabular) yielded better results for all metrics compared with concatenating only the feature selected (5 tabular). analyzing all results of both scenarios, the rnn+mlp hybrid model applied in scenario 2, using sequential and non-sequential information, obtained a recall improvement. to assess the significance of this improvement, we performed a hypothesis test with a significance level (alpha) of 0.05, using the techniques wilcoxon signed-rank test [26] and the student’s t-test [9]. the null hypothesis (h0) was defined as follows: the recall of the combined sequential and non-sequential data is not significantly higher than the recall using only non-sequential data. fig. 1. roc curves from all models (mlp, rf, xgb, rnn+mlp) in scenarios 1 and 2. full size image the p-values obtained for the rnn+mlp model with 27 temporal features and 63 tabular features were highly significant (1.95e-3 for wilcoxon and 1.47e-5 for student’s t-test), leading to a strong rejection of h0. similarly, for the model with 5 features, the p-values (1.95e-2 for wilcoxon signed-rank and 7.21e-3 for student’s t-test) also resulted in h0 rejection. these results support the conclusion that combining time series and tabular data statistically improves recall. the figure (fig. 1) shows the roc curves for all models (mlp, rf, xgb, rnn+mlp) in scenarios 1 and 2. table 4. performance of the machine learning model of studies on predicting scd risk, considering sensitivity as a reference parameter.full size table for comparative purposes, table 4 shows the performance of scd-related works in chhd and non-chhd contexts with the same parameter (recall) as this study. the results from both approaches were similar to the other works, the recall from rnn+mlp (27 temporal features + 63 tabular features) being the second highest one. despite methodological differences, our results seem relevant to the scd scenario in chhd. 4.2 user interface to empower domain experts with the ability to predict scd in cc, the mlp model with 5 features from scenario 1 was exported using the pickle5 library. this exported model served as the initial foundation for a initial web prediction system. we proposed a prototype intelligent system with six key functionalities: login, patient registration, appointment scheduling, exam registration, prediction, and model visualization. the application’s login was created using oauth authentication with the google platform, that offers secure and convenient access by leveraging existing google credentials. upon successful login, the user is redirected to the application’s main screen, depicted in fig. 2. fig. 2. home screen that provides access to all system functionalities. full size image this screen serves as a central hub for managing patients. it lists all registered patients, displaying their names, birth dates, last appointment details, and quick access buttons for requesting relevant cardiac exams (e.g., ecg, eco, ecg-holter) to aid in scd risk assessment. additionally, users can register patient data, input clinical or exam-related information, allowing the system to function as basic medical record software. fig. 3. cardiology risk prediction listing screen full size image regarding scd risk prediction functionality, users can access the dedicated screen by clicking on the ‘predict’ in top menu of the main screen, as depicted in fig. 3. this screen displays a grid summarizing all scd risk predictions made in the system. each entry in the grid shows the patient’s name, prediction date, predicted result (risk or non-risk, and probability), and a button to view the features values used for the prediction. fig. 4. add prediction screen full size image after clicking the “predict +” button, the system redirects the user to a screen for inputting new prediction data (fig. 4). this screen includes all fields used as input for the predictive model, including patient demographics (name, physician) and five selected tabular features (nsvt, amiodarone classification, rassi score). the very simple and user-friendly prototype system was developed as a web platform to facilitate easy access and use by doctors, ultimately assisting them in patient care. 5 conclusion this work proposes the use of multimodal and hybrid models to assess scd risk in non-high-risk cc patients. furthermore, a prototype is has been developed to support healthcare professionals in identifying patients with potential scd risk, aiding in early diagnosis and intervention. encouragingly, despite a limited sample size, the study achieved promising results. combining time series data with tabular data using a hybrid rnn+mlp model improved recall, precision, and f1-score performance, regardless of feature selection. it’s important to note that time series feature extraction is an ongoing area of research, and this analysis represents an initial exploration of the findings. this work serves as a foundation for future research aimed at developing even more robust models that can assist in the early diagnosis of scd risk in cc patients who are not classified as high-risk. these models have the potential to address the current limitations in accurately predicting scd in this patient population. future work includes reprocessing the models applying feature selection techniques to the temporal data, comparing the results of classical models with results of other models that have the capability to handle different types of data, applying xai techniques to enhance the interpretability of predictions for specialists, and exploring other rnn combination approaches to potentially improve model performance and generalizability. references alberto, a.c., pedrosa, r.c., zarzoso, v., nadal, j.: association between circadian holter ecg changes and sudden cardiac death in patients with chagas heart disease. physiol. meas. 41(2), 025006 (2020) article  google scholar  atallah, j., gonzalez corcia, m.c., walsh, e.p.: ventricular arrhythmia and life-threatening events in patients with repaired tetralogy of fallot. am. j. cardiol. 132, 126–132 (2020) article  google scholar  barker, j., et al.: machine learning in sudden cardiac death risk prediction: a systematic review. europace 24(11), 1777–1787 (2022) article  math  google scholar  cavalcante, c.h., et al.: sudden cardiac death multiparametric classification system for chagas heart disease’s patients based on clinical data and 24-hours ecg monitoring. math. biosci. eng. 20(5), 9159–9178 (2023) article  math  google scholar  elman, j.l.: finding structure in time. cogn. sci. 14(2), 179–211 (1990) article  math  google scholar  goldstein, b.a., chang, t.i., mitani, a.a., assimes, t.l., winkelmayer, w.c.: near-term prediction of sudden cardiac death in older hemodialysis patients using electronic health records. clin. j. am. soc. nephrol. 9(1), 82–91 (2014) article  math  google scholar  grinsztajn, l., oyallon, e., varoquaux, g.: why do tree-based models still outperform deep learning on typical tabular data? adv. neural. inf. process. syst. 35, 507–520 (2022) google scholar  hinkle, l.e., jr., thaler, h.t.: clinical classification of cardiac deaths. circulation 65(3), 457–464 (1982) article  math  google scholar  johnston, l.: student’s t-test. j. qual. technol. 2(4), 243–245 (1970) article  math  google scholar  knechtel, m.d.r.: metodologia da pesquisa em educação: uma abordagem teórico-prática dialogada. curitiba: intersaberes, 106 (2014) google scholar  lee, s., et al.: territory-wide cohort study of brugada syndrome in hong kong: predictors of long-term outcomes using random survival forests and non-negative matrix factorisation. open heart 8(1), e001505 (2021) article  mathscinet  google scholar  leoni, l., et al.: bridging implantable cardioverter-defibrillator patients undergoing radiotherapy with a wearable cardioverter-defibrillator. cardiology: open access 8(1), 24–27 (2023) google scholar  lopez-jimenez, f., et al.: artificial intelligence in cardiology: present and future. in: mayo clinic proceedings, vol. 95, pp. 1015–1039. elsevier (2020) google scholar  malik, l.h., singh, g.d., amsterdam, e.a.: chagas heart disease: an update. am. j. med. 128(11), 1251-e7 (2015) article  math  google scholar  members, a.f., priori, s.g., blomström-lundqvist, c., mazzanti, a., blom, n., borggrefe, m., camm, j., elliott, p.m., fitzsimons, d., hatala, r., et al.: 2015 esc guidelines for the management of patients with ventricular arrhythmias and the prevention of sudden cardiac death: the task force for the management of patients with ventricular arrhythmias and the prevention of sudden cardiac death of the european society of cardiology (esc) endorsed by: association for european paediatric and congenital cardiology (aepc). ep europace 17(11), 1601–1687 (2015) google scholar  nicolson, w.b., et al.: prospective evaluation of two novel ecg-based restitution biomarkers for prediction of sudden cardiac death risk in ischaemic cardiomyopathy. heart 100(23), 1878–1885 (2014) article  math  google scholar  nunes, j.m., infante, m.: pesquisa-ação: uma metodologia de consultoria. fiocruz, rio de janeiro (1996) google scholar  pedrosa, r.c., et al.: risk stratifier for sudden cardiac death beyond the left ventricular ejection fraction in chagas cardiomyopathy. pacing clini. electrophysiol. 47(2), 312–320 (2024) google scholar  pérez-molina, j.a., molina, i.: chagas disease cardiomyopathy treatment remains a challenge-authors’ reply. lancet 391(10136), 2209–2210 (2018) article  math  google scholar  rassi, a., jr., rassi, a.: another disappointing result with implantable cardioverter-defibrillator therapy in patients with chagas disease. europace 15(9), 1383 (2013) article  math  google scholar  schwartz, p., la rovere, m., vanoli, e.: autonomic nervous system and sudden cardiac death. experimental basis and clinical observations for post-myocardial infarction risk stratification. circulation 85(1 suppl), i77–91 (1992) google scholar  shakibfar, s., et al.: predicting electrical storms by remote monitoring of implantable cardioverter-defibrillator patients using machine learning. ep europace 21(2), 268–274 (2019) article  google scholar  de souza, a.c.j., et al.: development of a risk score to predict sudden death in patients with chaga’s heart disease. inter. j. cardiol. 187, 700–704 (2015) google scholar  tse, g., et al.: incorporating latent variables using nonnegative matrix factorization improves risk stratification in brugada syndrome. j. am. heart assoc. 9(22), e012714 (2020) article  google scholar  uddin, s., lu, h.: confirming the statistically significant superiority of tree-based machine learning algorithms over their counterparts for tabular data. plos one 19(4), e0301541 (2024) article  google scholar  woolson, r.f.: wilcoxon signed-rank test. encyclopedia biostatist. 8 (2005) google scholar  world health organization: chagas disease (american trypanosomiasis) (2021). https://www.who.int/news-room/fact-sheets/detail/chagas-disease-(american-trypanosomiasis) zoni-berisso, m., molini, d., viani, s., mela, g.s., delfino, l.: noninvasive prediction of sudden death and sustained ventricular tachycardia after acute myocardial infarction using a neural network algorithm. ital. heart. j. 2, 612–620 (2001) google scholar  download references acknowledgments the authors thank the federal institute of education, science, and technology of ceará (ifce) for its support through the pibic nž7/2023 and pibiti nž9/2023 calls from the prpi/reitoria-ifce. author information authors and affiliations federal institute of education and technology of ceara, maracanau, ceara, brazil isabelly p. da costa, bruno m. p. takazono & carlos h. l. cavalcante edson saad heart institute – federal university of rio de janeiro, rio de janeiro, brazil roberto c. pedrosa universidade feferal do ceará (ufc), fortaleza, ce, brazil joão p. v. madeiro authors isabelly p. da costaview author publications search author on:pubmed google scholar bruno m. p. takazonoview author publications search author on:pubmed google scholar carlos h. l. cavalcanteview author publications search author on:pubmed google scholar joão p. v. madeiroview author publications search author on:pubmed google scholar roberto c. pedrosaview author publications search author on:pubmed google scholar corresponding author correspondence to carlos h. l. cavalcante . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da costa, i.p., takazono, b.m.p., cavalcante, c.h.l., madeiro, j.p.v., pedrosa, r.c. (2025). multimodal and hybrid models for predicting scd risk in chagas cardiomyopathy. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_16 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords chagas disease sudden cardiac death machine learning recurrent neural network publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature the multi-attribute fairer cover problem | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper the multi-attribute fairer cover problem conference paper first online: 12 october 2023 pp 163–177 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) the multi-attribute fairer cover problem download book pdf download book epub ana paula s. dantas  orcid: orcid.org/0000-0002-8831-07109, gabriel bianchin de oliveira  orcid: orcid.org/0000-0002-1238-48609, helio pedrini  orcid: orcid.org/0000-0003-0125-630x9, cid c. de souza  orcid: orcid.org/0000-0002-5945-08459 & … zanoni dias  orcid: orcid.org/0000-0003-3333-68229  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 542 accesses abstract alongside the increased use of algorithms as decision making tools, there have been an increase of cases where minority classes have been harmed. this gives rise to study of algorithmic fairness that deals with how to include fairness aspects in the design of algorithms. with this in mind, we define a new problem of fair coverage called multi-attribute fairer cover, that deals with the task of selecting a subset for training that is as fair as possible. we applied our method to the age regression model using instances from the utkface dataset. we also present computational experiments for an integer linear programming model and for the age regression model. the experiments showed significant reduction on the error of the regression model when compared to a random selection. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others maximizing coverage while ensuring fairness: a tale of conflicting objectives article 07 december 2022 general procedure to measure fairness in regression problems article open access 13 february 2025 matching code and law: achieving algorithmic fairness with optimal transport article 01 november 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. algorithms computational social sciences computer ethics learning algorithms machine learning statistical learning 1 introduction in recent years, there have been a surge of cases where algorithms applications have been linked to some sort of discrimination towards a minority group of people. these cases are known as algorithmic injustice, or even algorithmic racism, when the discrimination is based on race. silva [12] has compiled a timeline showing that illustrates the increasing frequency of news reports on such cases. algorithms have been extensively used in the decision making process. the level of human participation varies in these decisions. for example, o’neil [10] reports on the case where american teachers are evaluated by a proprietary algorithms and are required to keep a certain score. although these scores can affect their careers, the teachers have no idea how the scores are assigned by the algorithm. in this example, the algorithms are helping on the decision making process, but one extreme case can be found in the report by soper [13]. the author showcases how some human-resources operations from an online retailer have supposedly been transferred to algorithms. the report discusses how the decisions made by these algorithms are affecting the lives of their collaborators, culminating even on wrongful terminations. with the increased use of these tools, the scientific community has been reacting and giving attention to the potential problems that the indiscriminate use of these tools may cause. for instance, chung [2] has presented an extensive research of cases of algorithmic racism. they pointed out among others things that we need to address the issues of fairness during the design of the algorithm, as well as explicitly including protected attributes in auditing processes in order to properly identify bias. it has been shown that including these attributes even on the training phase can improve accuracy [6, 8]. there are several approaches to insert fairness into models and mitigate the problems a biased model may cause. for example, galhotra et al. [4] investigated the selection of features in order to ensure a fairer model, paying special attention to sensitive characteristics. roh et al. [11] presented a greedy algorithm to select a fair subset during the sampling step in the batch selection. asudeh et al. [1] proposed that fairness be considered in a the coverage problem. they defined a combinatorial optimization problem that, given a universe set of colored elements and a family of subsets, aims to select k subsets such that the number of covered elements is the same for each color and the sum of the weight of these elements is maximum, showing that this problem is np-hard. based on this interpretation of the problem, dantas et al. [3] proposed the fairer cover problem, that removes restriction of having the same number of covered elements from each class and puts this in the objective function. therefore, the goal is to find a cover that is as fair as possible. both of these works consider fairness as a part of the algorithm design, instead of just optimizing models regardless of the impact in the different classes. in this work, we conjecture that a fair training subset will improve the quality of a machine learning model, specifically of a regression model. we present a new variation and application of a fair coverage problem, and we define a generalization of the fairer cover problem introduced by dantas et al. [3], which we call as multi-attribute fairer cover. we propose a new integer linear programming (ilp) model to solve instances of this problem and derive instances from the facial dataset utkfaces. we also present computational experiments for the ilp model and attest the improvement of the age regression model when using a fair subset in the training. the experiments showed significant reduction on the error of the regression model. the remainder of this paper is organized as follows. in sect. 2, we explain our methodology, starting with a few notations, followed by a description of the dataset used in the computational experiments and a detailing of our method to solve the fair coverage problem. in sect. 3, we present and discuss the results of our computational experiments. lastly, in sect. 4, we outline our conclusions and point out future work. 2 methodology in order to address the task of selecting a fairer subset as a combinatorial optimization problem, it is essential to begin by introducing a few definitions. let \(\mathcal {u}\) be the universe set of \(u_{j}\) elements. let \(\mathcal {s}\) be a family of subsets \(\mathcal {u}\). lastly, let \(\mathcal {c}\) represent a set of \(\chi \) colors and let \(c: \mathcal {u}\rightarrow \mathcal {c}\) be a function that assigns one color to each \(u_{j}\). we call a k-cover, a selection of k subsets from \(\mathcal {s}\), and we say that an element \(u_{j}\) is covered if there is a subset \(\mathcal {s}_{\ell }\) from a k-cover that contains \(u_{j}\). a k-cover is considered fair if the number of covered elements is the same for every color. we refer to as tabular a dataset that is represented as a matrix of where the rows are the samples and the columns are the attributes. we consider that the cell from the i-th row and the j-th column of the dataset contains exactly one of the possible values for the attribute described by the column j. that is, if the column j represents eye color and the possible values for eye color are either blue or brown, the cell from the i-th row and the j-th column has exactly one of these two values. we can use this type of dataset to create a combinatorial optimization problem by translating it into a family of subsets and treating the task of selecting a fair subset of size k for training as the task of finding a fair k-cover. to translate a tabular dataset into a family of subsets, we start by representing each cell of the dataset by a distinct element \(u_{j}\) and the universe set \(\mathcal {u}\) is the set containing every cell. we create subsets \(\mathcal {s}_{\ell }\) of universe \(\mathcal {u}\) based on the rows of the dataset, such that each row is represented by one subset \(\mathcal {s}_{\ell }\). for this paper, let us consider that every attribute is discrete. this way, each possible value for an attribute of the dataset can be mapped to a finite set of colors. we color the elements of \(\mathcal {u}\) based on the class of the attribute contained in the cell that it represents. note that, since a subset \(\mathcal {s}_{\ell }\) corresponds to a row of the dataset, every subset \(\mathcal {s}_{\ell }\) contains one element for each attribute (column) and they are all of different colors. we assume that each row of a tabular dataset is complete and there is no missing data. under this assumption, we have that every row of the dataset has the same number of valid attributes, as do the subsets representing them. asudeh et al. [1] defined an optimization problem called fair maximum coverage – fmc. in this problem, each element has a weight and there is a family \(\mathcal {s}\) of subsets \(\mathcal {s}_{\ell }\) of \(\mathcal {u}\). the goal of the fmc is to select k subsets \(\mathcal {s}_{\ell }\) such that the k-cover formed by these subsets is fair and has a maximum sum of the weights. as pointed out by dantas et al. [3], the interpretation of asudeh et al. may result in a limited solution pool, often leading to infeasible instances. dantas et al. [3] defined a new problem called fairer cover – fc, based on the fmc. this problem relaxes the fairness aspect, allowing for solutions to have a different number of covered elements for each color. this was achieved by removing the need to have a fair coverage as prerequisite and tackling this aspect in the objective function. the fc problem is given in definition 2.1, and its goal is to find a k-cover that is as fair as possible and covers at least s elements, with k and s being given in the instance. the fc was applied to the human protein atlas dataset obtained from the “human protein atlas single cell classification”. this dataset has a very distinct characteristic: there is a single set of attributes, which are the labels of the identified proteins. we expand the work presented in dantas et al. [3] to deal with multiple attributes using as an example the utkface dataset, which will be presented in the following subsection. by dealing with multiple classes of attributes, it becomes necessary to divide the colors that represent the values of the attributes into groups. this happens because each class of attribute may have different sizes and meanings. for example, suppose we have a tabular dataset containing the attributes eye color and age group, where the eye color can be either blue or brown and age group can be either child, teenager or adult. in addition, suppose that each row is made up of a pair of eye color and age group. each of the values blue, brown, child, teenager and adult will be represented by one of five distinct colors. to obtain a fair coverage of this dataset we would need a subset of rows where each of the five colors has the exact same number of elements. we should not aim to have the same number of covered elements representing adult and blue because they are of different types, therefore fc and fmc are not suitable for this kinds of instances. also, it is worth noting that in the case of the fmc, where the equality of the covered classes is a characteristic of the solution, the instance may become infeasible. given a list of color groups \(\mathcal {g} = \{ g_1, g_2, \dots , g_h\}\) that puts each color in exactly one group, that is, \(\bigcup _{a=1}^h g_a = \mathcal {c}\) and \(g_a \cap g_b = \emptyset \) for all \(a, b\in \{1, 2, \dots , h\}\) with \(a \ne b\). we propose a generalized version of the fc called multi-attribute fairer cover – mafc as shown in definition 2.2. this version of the problem considers the fairness among the colors of the group, that is, our goal is to have a selection that is as fair as possible when analyzed in each group of colors individually. note that, if \(h=1\), we have a problem equivalent to fc. the definition of the mafc differs from that of fc in two main points, one is the already discussed need to consider the fairness among the groups of elements and the other is the removal of parameter s that specifies the minimum number of elements to be covered. in an instance where every subset \(\mathcal {s}_{\ell }\) has the same size t, a k-cover always covers exactly \(s\times t\) elements. therefore, there is no need to specify a minimum number of covered elements. in the remainder of this section, we describe the dataset used in our experiments, as well as the proposed method for selecting a fair coverage and the model employed for the age regression task. we also present the evaluation metrics and the setup used for our experiments. 2.1 utkface dataset to evaluate our proposed method for a fair selection of samples, we use utkface dataset [14] in our experiments. this database consists of face images of people with different characteristics, which includes age (integer value ranging from 1 to 116), gender (male or female), and race (white, black, asian, indian, and other, that is, hispanic, latino, and middle eastern). it is worth noting that even though the age ranges from 1 to 116, the dataset does not contain samples for every value. in fact, there are only 104 distinct values. from the original utkface dataset, we split it into 64%, 16%, and 20% for training, validation, and test sets, respectively. table 1 presents the number of samples of each set of our split from utkface. table 1. number of images in training, validation, and test sets.full size table the frequencies of gender and race of training, validation, and test sets are presented in figs. 1a and 1b. concerning gender, there are more images of males than females in all three sets, although this difference is less accentuated. considering race, the white class is more frequent, followed by black, indian, and asian, and other (hispanic, latino, and middle eastern) have the lowest representation in the sets. it is worth noting that the most frequent class is roughly double the second most frequent class, which showcases how unbalanced the dataset is. 2.2 proposed method in this subsection, we describe the proposed method for fair age regression, divided into ilp and age regression models. the pipeline of the proposed method, with ilp and age regression models, is illustrated in fig. 2. from the original utkface images and annotations, we created instances for the mafc combinatorial problem. we then solved each instance of the mafc using an ilp model. from the ilp solution, we obtained a fair training subset considering age, gender, and race. lastly, we trained and evaluated the predictions of the age regression model. fig. 1. gender and race distribution on training, validation, and test sets. full size image fig. 2. pipeline of the proposed method. full size image ilp model. in this section we present an ilp formulation that models the mafc problem. the model is described by eqs. (1a) - (1g). the model uses three decision variables: \(x_j\), \(y_\ell \) and \(q_c\). the first two are binary and the last one is a positive integer. the variable \(x_j\) indicates whether the element \(u_{j}\) is covered and the variable \(y_\ell \) indicates whether the subset \(\mathcal {s}_{\ell }\) was selected to be a part of the k-cover. the variable \(q_c\) indicates the difference in the number of covered elements that have color c and the ideal number of covered elements that have color c. the ideal number of each color is given by the constants \(i_c\), the estimation for this value can be adjusted depending on the application. the model also uses the constants h to represent the number of color groups, m to indicate the number of elements in \(\mathcal {u}\), k to indicate the number of subsets to be selected, n to define total number of subsets, and \(\chi \) to specify the total number of colors used in the instance. we also define subsets \(c_c\), which are the subsets of the elements \(u_{j}\) that are colored with the color c. considering the concept of fairness as the equality among the classes in the cover and the objective of the mafc as it is stated in definition 2.2, we can consider the task of finding a k-cover that is as fair as possible as the minimization of the differences among the number of covered elements in each class. dantas et al. [3] modeled this directly in the objective function as the minimization of the difference in the number of color of each pair of colors. this approach might not be viable when the number of color increases, if we have for example 100 colors it would be necessary to minimize the sum of \(\frac{100\times 99}{2} = 4950\) pairs of colors. therefore, we choose to minimize the difference in the number covered elements for a given color c to its ideal value \(i_c\). the ideal value \(i_c\) can be easily estimated, since we know the number of colors in each group of colors and the number of selected subsets. in the case of a tabular dataset we can find the exact number \(i_c\) by simply dividing k by the number of colors in the group \(g_a\) to which c belongs. going back to the example given in sect. 2, the ideal number of elements covered representing the attribute blue is \(\frac{k}{2}\), since the class in which blue is contained has size two. with these definitions, the ilp model reads: $$\begin{aligned} & \min \quad \displaystyle \sum _{c\in g_1} q_c + \sum _{c\in g_2} q_c + \dots + \sum _{c\in g_h} q_c \end{aligned}$$ (1a) $$\begin{aligned} & \text {s. t.} \qquad x_j \le \sum _{\ell ~|~u_j \in \mathcal {s}_{\ell }} y_\ell ~~ \forall \, j \in \{1, 2, \dots , m\} \end{aligned}$$ (1b) $$\begin{aligned} & y_\ell \le x_j \qquad \forall \, u_j \in \mathcal {s}_{\ell }, \forall \, \mathcal {s}_{\ell } \in ~\mathcal {s} \end{aligned}$$ (1c) $$\begin{aligned} & \sum _{\ell = 1}^{n} y_\ell = k \end{aligned}$$ (1d) $$\begin{aligned} & \sum _{u_j \in c_c} x_j i_c \le q_{c} \qquad \forall \, c \in \mathcal {c} \end{aligned}$$ (1e) $$\begin{aligned} & i_c \sum _{u_j \in c_c} x_j \le q_{c} \qquad \forall \, c \in \mathcal {c} \end{aligned}$$ (1f) $$\begin{aligned} & x \in \mathbb {b}^{m}, ~y \in \mathbb {b}^{n},~q \in \mathbb {z}_+^{\chi } \end{aligned}$$ (1g) the objective function that reflects this modeling decision is given in eq. (1a). the remaining equations describe the restrictions of the model, through which we specify the characteristics of a solution. equation (1b) indicates that an element \(u_{j}\) can only be considered covered (\(x_j = 1\)), if there is at least one subset \(\mathcal {s}_{\ell }\) such that \(u_{j} \in \mathcal {s}_{\ell }\) and this subset is selected to be in the cover (i.e., \(y_\ell = 1\)). equation (1c) specifies that if a subset \(\mathcal {s}_{\ell }\) is selected to be in the cover (\(y_\ell = 1\)), then every element \(u_{j}\) of \(u_{\ell }\) is considered to be covered, along with the previous set of restrictions we guarantee the basic characteristics of a k-cover. equation (1d) restricts that exactly k subsets must be selected to form a k-cover. equations (1e) and (1f) are complementary and together they are used to define the value of the decision variable \(q_c\). note that the sums of \(x_j\) are used to indicate the number of covered elements with color c. suppose that this sum is smaller than the ideal value of \(i_c\). then, equation (1e) states that \(q_c\) must be larger than a negative value and, since \(q_c\) must be positive, this restriction is satisfied by any possible value of \(q_c\). now in the same scenario, equation (1f) establishes that \(q_c\) must be larger than a positive value. although there is no positive upper bound on the variable \(q_c\), we know that in an optimal solution this value will not be greater than the value of the difference. this is true because the objective function is a minimization on the value of \(q_c\). lastly, equation (1g) indicates the domain of the decision variables. age regression model. for evaluation of ilp model and comparison with random selection of sub-samples of the original utkface dataset, we fine-tune resnet50 architecture [5] for regression task about age prediction of each image. to do so, we remove the original output layer from imagenet database [7] and we include the regression output layer, which consists of a single neuron, in order to predict a single float value corresponding to the age of each sample. as each image from the dataset can have different lengths of pixel dimensions, that is, width and height sizes, we resize each image to have the same shape. with this preprocessing step, all images are transformed to 128 pixels of width and 128 pixels of height. 2.3 evaluation metrics to assess our proposed method and comparison with a random selection of images, we apply balanced mae during our experiments. balanced mae is derived from mean absolute error (mae), which is presented in eq. 2, where n indicates the number of samples, \(y_{i}\) and \(\hat{y}_i\) represent the ground-truth and predicted values for sample i, respectively. $$\begin{aligned} {\text {mae}} = \frac{1}{n} \times \sum _{i=1}^{n} |y_{i} \hat{y}_{i}| \end{aligned}$$ (2) based on the mae score for each age, that is, the value calculated using only the images from a specific age, we calculate balanced mae (bmae), as shown in eq. 3. in this equation, c represents the number of ages evaluated at the current stage and \({\text {mae}}_{j}\) indicates the mae score for a specific age j. $$\begin{aligned} {\text {bmae}} = \frac{1}{c} \times \sum _{j=1}^{c} {\text {mae}}_{j} \end{aligned}$$ (3) it is important to note that bmae gives the same importance for each age, that is, independently of the number of samples of a specific age. for example, the age with the majority number of samples, it has the same impact as any other age (for example, the age with the minority number of samples). since we are aiming for a model that has a fair training in several aspects, including the coverage of the ages, this score is more appropriate when compared to a simple mae, where we would have that errors in ages that are underrepresented would have a lesser impact on the overall metric. 3 computational results in this section, we present and discuss the results of our computational experiments. our experiments are divided into two phases: experimentation with the ilp model and experimentation with the age regression model. in both phases, we analyze the concepts of fairness and how they are manifested in each case. 3.1 computational setups before we present our computational results, let us first present the setups used in each phase of our experiments. ilp setup: we used the ibm cplex studio (version 12.8) to solve the combinatorial optimization instances described in the previous sections. we used the programming language c++ to implement the model described in sect. 2 with compiler g++ (version 11.3.0), flags c++19 and -o3. the experiments reported in sect. 3.2 were run in a ubuntu 22.04.2 lts desktop with intel® core™ i7-7700k and 32 gb of ram. age regression model setup: the age regression model was implemented using the programming language python and tensorflow library. to train and evaluate our model, we used google colaboratory virtual environment. during the training step, the model was trained during 200 epochs, with early stopping technique with patience of 20 epochs and reduction of learning rate by a factor of \(10^{-1}\) if the model did not improved the results on validation set after 10 epochs. for the optimization process, we employed mae as loss function and adamw optimizer [9] with initial learning rate equal to \(10^{-4}\). it is worth noting that we tuned these parameter with experiments on the full training dataset, without any fairness adjustments. we followed this configuration for both fair and random selection of images for age regression models. 3.2 ilp model results with the model implemented, we started the experiments with the different instances created from the utkface dataset. to create the instances, we considered the annotations of gender, age and race provided in the dataset to be a tabular set with three attributes and 15172 samples, that is, we considered only the images from training split of the original dataset. for each possible value of the three attributes, we assigned a distinct color and created the sets corresponding to the each sample. note that each subset of elements has size three, since it contains one element for each attribute. we specified four arbitrary values for k to use in the instances: 1000, 2000, 5000, 10000. these numbers will specify the sizes of the cover. after defining the elements, the sets, the coloring and the values for the parameter k, we needed to estimate the ideal values for each color. in a perfect scenario, the ideal value \(i_c\) for a given color c would be defined as \(\frac{k}{|g_a|}\), where \(c \in g_a\), but from analyzing the dataset, we saw that this cannot always be achieved since there were sub-represented colors in the dataset. for these cases we devised a simple procedure to compute the value \(i_c\) for each color group \(g_a\). we start the procedure by defining a temporary \(t_i\) ideal value as \(\frac{k}{|g_a|}\). for every color in \(c \in g_a\) that has a frequency smaller than \(t_i\), we set \(i_c\) as the value of the frequency of color c, next we update \(t_i = \frac{k k^\prime }{|g_a| n^\prime }\), where \(k^\prime \) is the sum of frequency of the colors that just had their ideal value \(i_c\) set and \(n^\prime \) is the number of said colors. we repeat this until every color has a frequency of at least \(t_i\). the remaining colors have their \(i_c\) value set to \(t_i\). by doing this, we set realistic ideal values for each class. that is, the classes that have a frequency lower than the ideal, have ideal value set to their respective frequencies because the solution has to respect this limitation. then, the remaining classes have a new ideal value that accounts for the elements that are underrepresented. in our initial experiments, we noticed a tendency of the model’s solutions to have a discrepancy in the covering when considering the combinations of the attributes. in fig. 3, we show the distribution of gender (fig. 3a) and race (fig. 3b) for each of the ages in the training split of the dataset. this figure considers that the parameter k is set to 1000. since our model is considering only the equality of each attribute separately, we had several cases where all the covered images of an given age were just from the class male or just the class female, as shown in fig. 3a. in fig. 3b, most ages are predominantly from a single race, in special the race other is predominant in the ages 20 through 40. fig. 3. percentage of gender and race in each of age values in the solution given by the ilp model for the instance with \(k = 1000\). full size image this showed a possible problem for our model, that points towards a necessity to consider also the pairings of the attributes. this aspect could be considered either in the model for the problem or in the instance. for example, we could add to model restrictions that decrease the difference of each pair formed by one color from group \(\mathcal {g}_a\) and one color from group \(\mathcal {g}_b\). we choose to contour this problem by inserting a new element to each set, and color it with colors from a fourth group. this new element was colored based on the three original attributes combined. that is, if the sample corresponded was annotated as \(\{\texttt {male}, \texttt {black}, \texttt {3 years old}\}\), the new element received the color c and every other sample that had this exact three characteristics also received color c in the fourth element. note that if any of the three characteristics is different, the color of the fourth element would also be different. this approach increases the size of the instance, since the new element added to each set has the potential to add 2 genders \(\times 5\) races \( \times 104 \) ages \( = 1040\) new colors. this was not the case because the dataset does not have a sample for each of possible triples formed by the attributes. the increase on the instance size (more colors and more elements) is an aspect that cannot be avoided, either we increase the instance or increase the number of restrictions in the modeling. by creating this new element we avoid modifying the model and the problem definition. we note that we could also have considered pairs of attributes to color new elements in the set, but this would result in a much larger instance. fig. 4. percentage of gender and race in each of age values in the solution given by the ilp model for the instance with \(k = 1000\) after adding the fourth element. full size image with these modifications, we executed the model with the new instances and we can see the impact of these changes in fig. 4. we now have a more even distribution of the both gender and races in each age. it is worth noting that the dataset itself, has some gaps that will reflect in the possibility of equality in our problem. for example, the samples for ages larger than 100 years are all of female subjects. over the age of 80 years, we do not have samples of the race other, which tends to skew the selection of the classes that are present in these ages. that is, despite the race white being the most frequent in every age, its presence becomes more dominant on the larger ages because there are no other options to select the sets and also maintain the aspect of fairness in the colors referent to the classes of ages. the distribution of each attribute in the solutions given by the model can be seen in fig. 5. the first two graphs show the frequency of the elements of each class that were covered in the solution. in fig. 5a, the values for the gender are very close to the ideal for each of the values of k, meaning that for this attribute the model found a near perfectly fair solution. now, in figs. 5b and 5c, we start to see small variations on the number of covered elements of each age and class of race. such variations are accentuated when k is larger, as expected, because the least common attributes will reach their limits. it is worth noting that the solutions obtained by the ilp model are optimal solutions. there was a time limit of 30 min for the executions of these experiments, but this limit was never reached. all instances finished execution within a few seconds, even after increasing the size of the instance with the addition of the fourth element to each set. fig. 5. frequency of the attributes in the fair cover given by the ilp model for each of the sizes of the parameter k. full size image 3.3 age regression results based on the fair subsets generated by ilp model, we ran each experiment for the regression task. we also ran experiments for a random selection of the images, considering the same size of each fair set. for each experiment, we executed five runs and compared the mean value of each configuration. in our first analysis, we evaluated fair and random sets for each size considering bmae metric. table 2 presents the results, showing that fair achieved the best outcomes of each size. we can highlight that fair selection outperformed random selection on smallest sets, with the difference of more than 2 years on 1000 set and more than 1 year on 2000 and 5000 sets on average for each age. table 2. comparison of random and fair selections considering bmae for each training set size.full size table the average error for each age of fair selection based on ilp model is lower than random selection, as shown in fig. 6. the fair selection achieved the lowest errors on ages with more than 80 years on sets with a limited number of images, which are the ages with less number of samples on the dataset. fig. 6. comparison of random and fair selections considering bmae per age. full size image then, we assessed ilp fair and random sets considering gender and race classes. to do so, we filtered images from the condition in analysis, e.g., images of gender male, and evaluated using bmae. table 3 presents the outcomes of each selection approaches. concerning gender, the fair set obtained the lowest errors on all the four training sizes against random selection, with the highest difference on training set with 1000 images. table 3. comparison of random and fair selections for gender and race considering bmae for each training set size.full size table considering race, fair selection also presented best outcomes compared to random selection in all the classes for each training set size for k greater than 2000, and the best results for most of the classes on k equal to 1000. for white, black, asian, and indian classes, which are the four most frequent ones, the highest difference of fair and random selections occurred on the smallest training set, that is, set with size equal to 1000, and the difference decreases considering the size of the training size. for other, which is the least frequent class, the difference tends to increase considering the size of the training set. we can conclude that this occured due to the frequency of other class, which represents less than 10% of the original training set, as well as the validation and test sets. it is worth noting that, although the classes representing race and gender are very balanced, as shown in fig. 5, indicating that we have a fair cover on these characteristics, we still have a noticeable disparity in the bmae when considering the different classes of race and gender. for example, the error for the regression of female is always greater than that of male. we also have the race black has a greater error consistently in all cases. this behavior is present in both the training with a fair subset and with a random subset. although the training with the fair subset has decreased this difference in every case. 4 conclusions and future work in this paper, we defined a generalized version of the fairer cover, called multi-attribute fairer cover. in this version of the problem, we were able to handle the selection of a training subset on a tabular dataset that has multiple attributes and applies the fairness condition to each attribute individually. we present an integer linear programming (ilp) formulation that models this problem and also detail a procedure to create a combinatorial instance from a tabular dataset. we presented experiments with our ilp model and instances derived from a real application dataset. the model achieved a good performance, returning the optimal solution to every instance tested in just a few seconds. we also analyzed the solutions given by the model, and showed how to adjust the instance in order to satisfy the fairness condition through the pairing of the attributes. we developed an age regression model to test the training with the different sizes of fair subsets obtained with the ilp model and compared these models with their random counterparts. we found that error was reduced when using the fair subsets, specially when dealing with more advanced ages. we also reduced the disparity in error between the samples of the different genders and races. as future work, we intend to investigate how the fair training subset affects a different regression models specifically tuned for fairness. since the utkface dataset has annotations on both race and genders, we also intend to experiment with our model on other tasks, such as race and gender classification, and investigate how to decrease the error disparity between gender and race classes. references asudeh, a., berger-wolf, t., dasgupta, b., sidiropoulos, a.: maximizing coverage while ensuring fairness: a tale of conflicting objectives. algorithmica 85(5), 1287–1331 (2023) article  mathscinet  math  google scholar  chung, j.: racism in. public citizen, racism out a primer on algorithmic racism (2022) google scholar  dantas., a.p.s., de oliveira., g.b., de oliveira., d.m., pedrini., h., de souza., c.c., dias., z.: algorithmic fairness applied to the multi-label classification problem. in: 18th international joint conference on computer vision, imaging and computer graphics theory and applications volume 5: visapp, pp. 737–744. scitepress (2023) google scholar  galhotra, s., shanmugam, k., sattigeri, p., varshney, k.r.: causal feature selection for algorithmic fairness. in: international conference on management of data, sigmod2022, pp. 276–285. association for computing machinery (2022) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: conference on computer vision and pattern recognition, pp. 770–778. ieee (2016) google scholar  kleinberg, j., ludwig, j., mullainathan, s., rambachan, a.: algorithmic fairness. aea papers proc. 108, 22–27 (2018) article  google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. in: advances in neural information processing systems (nips), pp. 1097–1105. curran associates, inc. (2012) google scholar  lin, y., guan, y., asudeh, a., jagadish, h.v.j.: identifying insufficient data coverage in databases with multiple relations. vldb endowment 13(12), 2229–2242 (2020) article  google scholar  loshchilov, i., hutter, f.: decoupled weight decay regularization, pp. 1–11 (2019). arxiv:1711.05101 o’neil, c.: weapons of math destruction: how big data increases inequality and threatens democracy. in: crown (2017) google scholar  roh, y., lee, k., whang, s., suh, c.: sample selection for fair and robust training. adv. neural. inf. process. syst. 34, 815–827 (2021) google scholar  silva, t.: algorithmic racism timeline (2020). https://bit.ly/3o6rjyc soper, s.: fired by bot at amazon: ‘it’s you against the machine’ (2021). https://www.bloomberg.com/news/features/2021-06-28/fired-by-bot-amazon-turns-to-machine-managers-and-workers-are-losing-out, https://bit.ly/3ivhuxy zhang, z., song, y., qi, h.: age progression/regression by conditional adversarial autoencoder. in: conference on computer vision and pattern recognition, pp. 5810–5818. ieee (2017) google scholar  download references acknowledgements the authors would like to thank the são paulo research foundation [grants #2017/12646-3, #2020/16439-5]; the national council for scientific and technological development [grants #306454/2018-1, #161015/2021-2, #302530/2022-3, #304836/2022-2]; the coordination for the improvement of higher education personnel; and santander bank, brazil. author information authors and affiliations institute of computing, university of campinas, campinas, 13083-852, brazil ana paula s. dantas, gabriel bianchin de oliveira, helio pedrini, cid c. de souza & zanoni dias authors ana paula s. dantasview author publications search author on:pubmed google scholar gabriel bianchin de oliveiraview author publications search author on:pubmed google scholar helio pedriniview author publications search author on:pubmed google scholar cid c. de souzaview author publications search author on:pubmed google scholar zanoni diasview author publications search author on:pubmed google scholar corresponding author correspondence to ana paula s. dantas . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper dantas, a.p.s., de oliveira, g.b., pedrini, h., de souza, c.c., dias, z. (2023). the multi-attribute fairer cover problem. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_11 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords algorithmic fairness fair coverage integer linear programming publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature featgenn: improving model performance for tabular data with correlation-based feature extraction | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper featgenn: improving model performance for tabular data with correlation-based feature extraction conference paper first online: 12 october 2023 pp 259–273 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) featgenn: improving model performance for tabular data with correlation-based feature extraction download book pdf download book epub sammuel ramos silva  orcid: orcid.org/0000-0002-2274-13909,10 & rodrigo silva  orcid: orcid.org/0000-0003-2547-38359  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 576 accesses 1 citation abstract automated feature engineering (autofe) has become an important task for any machine learning project, as it can help improve model performance and gain more information for statistical analysis. however, most current approaches for autofe rely on manual feature creation or use methods that can generate a large number of features, which can be computationally intensive and lead to overfitting. to address these challenges, we propose a novel convolutional method called featgenn that extracts and creates new features using correlation as a pooling function. unlike traditional pooling functions like max-pooling, correlation-based pooling considers the linear relationship between the features in the data matrix, making it more suitable for tabular data. we evaluate our method on various benchmark datasets and demonstrate that featgenn outperforms existing autofe approaches regarding model performance. our results suggest that correlation-based pooling can be a promising alternative to max-pooling for autofe in tabular data applications. the author would like to acknowledge fapemig (fundação de amparo à pesquisa do estado de minas gerais), capes (coordenação de aperfeiçoamento de pessoal de nível superior), ufop (universidade federal de ouro preto) and cloudwalk, inc, for the financial support which has been instrumental in the successful execution of our research endeavors. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others attribute selection based on correlation analysis chapter © 2018 iterative feature exclusion ranking for deep tabular learning article 16 december 2025 towards automatic complex feature engineering chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining machine learning quantitative trait skills statistical learning 1 introduction creating effective features is a crucial aspect of machine-learning projects. essentially, it involves deriving new features from existing data to train a model or extract more information for statistical analysis. discovering novel features from raw datasets is often the key to improving model performance [1]. traditionally, feature creation is a manual process that heavily relies on an analyst’s domain knowledge and programming skills. however, this approach can be limiting, as an analyst’s intuition and expertise often influence the features created. to overcome these limitations, researchers have been exploring the field of automated feature engineering (autofe). autofe aims to automate the feature creation process, enabling the discovery of more complex and effective features without relying solely on human input. automated feature engineering methods involve applying transformations to raw data to create new features. one commonly used technique is the expansion-reduction method [4], which generates a large number of features and then applies a feature selection algorithm to reduce their dimensionality. during the expansion phase, various transformations, such as logarithmic, max/min, or sum, are applied to the raw data. in the reduction phase, a feature selection method is utilized to identify the most effective set of features, which can significantly enhance a model’s performance. the possible number of transformation operations that can be performed on already-transformed features is practically infinite, which leads to an exponential increase in the feature space. this issue can cause a problem in reducing the number of feature evaluations required. to address this issue, researchers have proposed adaptive methods for autofe. for instance, khurana et al. [5] introduced a q-learning agent capable of performing feature transformation search, achieving higher performance but still generating a large number of features. in another study [6], a multi-layer perceptron (mlp) was trained to suggest the best transformations for each raw feature, resolving the problem of excessive feature generation. more recently, difer [7], a gradient-based method for differentiable autofe, has demonstrated superior performance and computational efficiency compared to other approaches, although it still requires significant computation in recent years, the use of deep neural networks (dnns) has become increasingly widespread across a range of fields, such as computer vision and natural language processing [14, 15]. typically, these models extract new features by feeding input features into the hidden layers of a dnn. while this approach is effective in capturing complex interactions between implicit and explicit features, it may not always generate useful new features due to a lack of relevant interactions in the dataset [9]. moreover, most existing works use max-pooling in the pooling layer, which may not be optimal for tabular data because it does not preserve the order and context of features in the data matrix. additionally, max-pooling is intended to identify the most significant features within an image, which may not always be relevant or effective for tabular data. to address the limitations of existing autofe methods, we propose featgenn, a convolutional approach that leverages correlation as a pooling function to extract and generate new features. featgenn first applies convolutional filters to the raw data to extract high-level representations. then, instead of using traditional pooling functions like max or average pooling, it computes the correlation between the extracted features, which helps to identify the most informative features. the selected features are then passed through a multi-layer perceptron (mlp) to create the final set of new features. preliminary results indicate that featgenn outperforms existing autofe methods in both the number of generated features and model performance, demonstrating its potential as a potent tool for creating features in machine learning. 2 related work the main goal of feature engineering is to transform raw data into new features that can better express the problem to be solved. training a model with the generated features can increase the performance of the model. however, the process of feature engineering can be limited by the expertise, programming skills, and intuition of the person working with the data. for this reason, autofe approaches have recently gained attention. the authors of [7] propose a differentiable automl model that efficiently extracts low and high-order features. the model includes three steps: initialization, optimizer training, and feature evaluation. in initialization, features are constructed randomly and evaluated using a machine-learning model in the validation set. in training the optimizer, a tree-like structure is created with an encoder, a predictor, and a decoder, called a parse tree. the encoder maps the post-order traversal string to a continuous space, the predictor is a 5-layer mlp that maps the representation to the score computed by a machine learning model, and the decoder maps the embedding to the discrete feature space. in the final step of feature evolution, the best n features are selected and optimized using a gradient-based approach. in [4], the authors present an algorithm that uses mathematical functions to generate new features for relational databases. the algorithm begins by identifying the entities that make up the database and defines a set of mathematical functions that are applied at both the entity level and the relational level. the proposed approach first enumerates all possible transformations on all features and then directly selects features based on their impact on model performance. however, due to the potentially large number of features generated, it is necessary to perform feature selection and dimensionality reduction to avoid overfitting and improve the interpretability of the model. in [6], the authors propose a novel model for feature engineering in classification tasks that can generalize the effects of different feature transformations across multiple datasets. the model uses an mlp for each transformation to predict whether it can produce more useful features than the original set. the quantile sketch array (qsa) achieves a fixed-size representation of feature values to handle features and data of different lengths. the qsa uses quantile data sketch to represent feature values associated with a class label. the authors of [11] have proposed an rnn-based approach to address the feature explosion problem in feature engineering and support higher-order transformations. their architecture uses an rnn to generate transformation rules with a maximum order for each raw feature within a fixed time limit. for datasets with multiple raw features, the authors use multiple rnns as controllers to generate transformation rules for each feature. the transformed features are evaluated using a machine learning algorithm and the controller is trained using policy gradients. the model includes two special unary transformations: “delete” and “terminate”, which remove a feature and terminate the current transformation, respectively, to determine the most appropriate transformation order. in [5] they propose a heuristic model for automating feature engineering in supervised learning problems. their model is based on a tree structure, where the raw dataset is the root, each node is a transformed dataset, and the edges represent the transformation functions. the goal is to find the node with the highest score, reducing the feature construction problem to a search problem. the authors present three exploration strategies to traverse the tree. the first is “depth-first traversal”, in which a random transformation is applied to the root and the algorithm then explores a branch until there is no further improvement. then it chooses another node with the highest score and starts the process again. the second is the “global traversal”, where a global search is performed to find the most promising node out of all the nodes explored so far. the third is “balanced traversal”, in which the algorithm chooses either an exploration or exploitation strategy at each step based on a time or node budget. to handle the explosive growth of columns, feature selection is required as they grow. cognito allows the selection of features after each transformation to clean up the dataset and ensure a manageable size. in addition, at the end of the model execution, the algorithm performs another feature selection for all columns in the dataset, including the newly created columns. autofeat is a method presented in [10] that generates and selects non-linear input features from raw inputs. the method applies a series of transformations to the raw input and combines pairs of features in an alternating multi-step process to generate new features. however, this leads to an exponential increase in the size of the feature space, so a subsampling procedure is performed before computing new features. the authors have shown that two or three steps of the feature technique are usually sufficient to generate new features. after feature engineering, the new dataset has a higher number of features than the original dataset. to reduce the dimensionality, the authors developed a feature selection procedure. first, they remove new features that are highly correlated with the original or simpler features. then they apply a wrapper method with l1-regular linear models to select the most informative and non-redundant features from the dataset. in the end, only a few dozen features are retained and used after the feature creation and selection process. in [17], autolearn is proposed, a learning model based on regression between pairs of features and aimed at discovering patterns and their variations in the data. the method selects a small number of new features to achieve the desired performance. the proposed method consists of four phases: pre-processing to reduce dimensionality, where the authors perform feature selection based on information gain (ig); mining of correlated features to define and search for pairwise correlated features, where the distance correlation [8] is calculated to determine if there is an interesting predictive relationship between a pair of features; feature generation, where regularized regression algorithms are used to search for associations between features and generate new features; and feature selection, where features that do not add new information to the dataset are discarded. the authors of [3] have proposed a novel model that achieves both memorization and generalization by simultaneously training a linear model component and a neural network component. the model consists of two components: the wide component, which is a generalized linear model of the form \(yw^txb\), where y denotes prediction, x denotes features, w denotes model parameters and b denotes bias. the input features can be either raw or transformed, the most important transformation being the cross-product transformation; and the deep component, which is a feed-forward neural network. for categorical features, an embedding is created, which is then added to the dataset and fed into the network. the authors of [2] have proposed a model for predicting ctr that can handle interactions between low and high-order features by introducing a factorization-machine (fm) based neural network. the model consists of two parts: the fm component, which generates low-order features and can generate interactions between 1st and 2nd-order features with low computational cost, and the deep component, a feed-forward neural network that learns interactions between higher-order features. the input to the network is a high-dimensional vector of sparse data containing categorical and continuous variables as well as grouped fields. the fgcnn is another approach proposed in ctr for prediction [9]. this model consists of two components, namely the feature generation and the deep classifier. the feature generation component uses the mechanisms inherent in the convolutional neural network (cnn) and the multilayer perceptron (mlp) to identify relevant local and global patterns in the data and generate new features. the deep classifier component then uses the extended feature space to learn and make predictions. our work introduces a cnn-based model with correlation-pooling for extracting high-order features and improving model performance. unlike traditional pooling functions such as max-pooling, which focus on selecting the maximum value within a pooling region, correlation-pooling considers the linear relationships between features in the data matrix. it measures the correlation coefficient between the features and aggregates them based on their correlation values to capture the interdependencies and patterns in the data. by incorporating correlation-based pooling into the feature extraction process, featgenn can effectively extract high-order features that reflect the underlying relationships among input variables. our proposed method achieves competitive results on a range of problems, suggesting that correlation-based pooling is a promising technique for working with tabular data in neural networks. 3 proposed approach in this section, we describe the proposed feature generation with evolutionary convolutional neural networks (featgenn) model in detail. 3.1 problem formulation given an dataset \(d = \left\langle f, tg \right\rangle \), where \(f = \left\{ f_1, f_2, ..., f_n \right\} \) are the raw features and tg the target vector. we denote as \(l_{e}^{m}(d,tg)\) the performance of the machine learning model m that is learned from d and measured by the evaluation metric e (e.g. accuracy). in addition, we transform a raw set of features d into \(d_{new}\) by applying a set of transformation functions \(t = \left\{ t_1, t_2, ..., t_n \right\} \). formally, the goal of the autofe is to search the optimal transformed feature set \(d^{*}\) where \(l_{e}^{m}(d^{*},tg)\) is maximized. 3.2 featgenn model fig. 1. the featgenn process. full size image in this study, we use a convolutional neural network to extract features that can improve the performance of a machine learning model (i.e., random forest). as explained earlier, using an mlp alone to generate new features would not result in a good set of new features. the reason for this is the relationship between the number of informative interactions between features and the total number of features in the feature space. also, using a cnn alone might not lead to good performance because a cnn only considers local interactions and does not consider many important global interactions between features [9]. to overcome this problem, we use an architecture that combines the mlp with the cnn. the featgenn model includes two main blocks, namely local feature extraction and global feature generation (fig. 1). the first block attempts to identify the most informative interactions between local features, while the second block generates new features from the features extracted by the local feature extraction block and combines them globally. the local feature extraction block includes two main operations, namely pooling and convolution. among these operations, the pooling operation plays a crucial role in reducing dimensionality and preserving the most informative features for subsequent layers. in previous work on feature generation for tabular data with cnn, max-pooling was mainly used. however, we found that using max-pooling for tabular data may not give the desired result because the model may not compare closely related features, thus affecting the features generated by the model. therefore, we propose the use of correlation-pooling to address this issue. in correlation pooling (fig. 2), the variant of pooling used in our local feature extraction block, uses pearson correlation [19] to group features that are highly correlated. by grouping these features, correlation-pooling can preserve the relationship between closely related features and thus improve the quality of the features extracted by the cnn model. this is in contrast to max-pooling, which preserves only the most dominant feature in a group and may ignore other relevant features that are closely related. therefore, by incorporating pearson correlation in the pooling operation, correlation-pooling can effectively circumvent the limitation of max-pooling and help generate more informative features for subsequent layers in the cnn model. the pearson correlation coefficient can be formulated as follows for our problem: $$\begin{aligned} r = \frac{n\sum xy (\sum x)(\sum y)}{\sqrt{[n\sum x^{2}-(\sum x)^{2}][n\sum y^{2}-(\sum y)^{2}]}} \end{aligned}$$ (1) where x and y represent the values of the features x and y respectively, and \(x,y \in f\), where f is the set of all features. the variable n denotes the number of samples in the dataset d. to avoid having to run the pearson algorithm twice, we have introduced an iterative calculation of the pearson coefficient. this means that at the current stage of model development, we compute the pearson coefficient r to perform the pooling operation for the subsequent evolutionary generation of the model. to reduce the computations required, we also added a threshold to limit the number of data sent to the correlation calculation, i.e., a model can only use 70% of the data to calculate the correlation value for the features. fig. 2. correlation-pooling process. full size image while the pearson correlation is a statistical measure that describes the linear relationship between two variables, it is not suitable for analyzing relationships between more than two characteristics. to overcome this limitation, we use the multivariate correlation matrix, which consists of pairwise pearson correlation coefficients between all pairs of variables. this matrix allows us to analyze relationships between multiple variables and identify the most highly correlated variables. the overall correlation value for the feature f can be formulated as follows: $$\begin{aligned} cs_f = \frac{\sum _{k}^{n}r_{fk}}{n} \end{aligned}$$ (2) where \(cs_f\) is the correlation score for the feature f, \(r_{fk}\) represent the person correlation score for the feature tuple (f,k) and n the total number of feature in the dataset. in the global feature generation block, an mlp is utilized to merge the features extracted from the local feature extraction block and generate novel features. these novel features are then appended to the original dataset and used in the machine-learning model. 3.3 evolution process in this work, we adopt an evolution process for conducting autofe, as depicted in fig. 3. this process involves three distinct steps: (1) feature selection, (2) population initialization, and (3) feature evolution. fig. 3. feature evolution process. full size image the first step of our proposed approach is to reduce the combination of uncorrelated and redundant features using a feature selection method. we used the maximum relevance-minimum redundancy (mrmr) [12] method for this purpose. by minimizing the combination of such features, we aim to reduce the introduction of noise into the model and improve the quality of the features generated by the cnn model. during the population initialization step, we generate a population of the cnn model pop that is evolved in the feature evolution step. to evaluate this initial population, we use a machine learning model on the dataset resulting from step (1) specifically, we take the set of features f* from the feature selection step and input them into the cnn model p (where \(p \in pop\)) to generate n new features f. these newly created features are concatenated with the original dataset d to create a new dataset \(d* \left\{ f \cup f \right\} \), which is then evaluated by the machine learning model \(l^m\) to obtain a score \(s_p\). in the trait evolution step, a genetic algorithm [13] is used to evolve the population and identify the most effective traits to improve the performance score obtained by \(l^m\). during each epoch of the genetic algorithm, for each model p that is not part of the elite group e (where \(e is \in pop\)), a crossover is performed between its weights and those of a second model \(p'\), which is selected using a round-robin tournament [18]. following the crossover process, the offspring generated by this operation can be subjected to mutation. the features produced by the offspring are then evaluated, as described in the initialization of the population initialization step. if the score obtained by \(l^m\) is better than the current score for p or if depreciation is allowed, the offspring replaces the current model p and the score is updated. 4 results in this section, we aim to answer the following research questions: rq1: how effective is correlation-pooling compared to max-pooling? rq2: study of the impact of the number of data on the correlation pooling computation? rq3: how effective is the proposed featgenn approach? (comparison with literature) 4.1 experimental setup to evaluate the performance of the featgenn model, on classification problems, 6 classification datasets from the uci repository, which were used in the state-of-the-art methods [7, 11], were selected. the description of each dataset in terms of the number of features and number of samples is presented in table 1. table 1. statistics of the benchmarks used to perform the evaluation of the featgenn features.full size table in our experiments, we use the f1-score as the evaluation measure, which is also commonly used in the related works [11] and [7]. the threshold for questions rq1 and rq2 was set at 80% of the available data in the dataset. to ensure robustness and reliability, we use 5-fold cross-validation, in which the dataset is divided into five subsets or folds and the evaluation is performed five times, with each fold serving once as a test set. this approach helps mitigate the effects of data variability and provides a more comprehensive assessment of the model’s performance. as for the chosen algorithm, we use random forest as the base method in all our experiments. random forest is a popular and widely used ensemble learning method known for its robustness and ability to handle different types of data. 4.2 effectiveness of correlation-pooling vs. max-pooling (rq1) in this subsection, this experiment aims to answer: can our featgenn with correlation-pooling achieve competitive results compared to the version with max-pooling? table 2 shows the comparison results in terms of f1 score. the results show that the featgenn with correlation-pooling outperforms the version with max-pooling in most datasets. the only exceptions are the megawatt1 and credit_default datasets, where the results are very similar. this result can be attributed to the fact that correlation-pooling takes into account the relationships between features when generating new features, which contributes to its relatively better performance. table 2. comparing featgenn performance with correlation-pooling and max-pooling. the * denotes the version of the featgenn that was executed with correlation-pooling. the results are the average score, and the standard deviation, after 30 runsfull size table 4.3 impact of the number of data on the correlation pooling computation (rq2) in this subsection, our experiment aims to answer the question: what is the influence of the number of available data on the correlation-pooling computation?. figure 4 shows the performance of three versions of featgenn: featgenn (using all available data), featgenn (using 60% of the data), and featgenn* (using 30% of the data). the results show that, as expected, the performance of the model varies with the amount of data used to compute the correlation-pooling. on average, the version with access to the entire dataset achieves a performance improvement of 0.76% and 1.38% compared to the featgenn and featgenn* versions, respectively. compared to the version that used 80% of the available data, the result after 30 epochs is very similar, although the version with more data performs better in fewer epochs. these results indicate that the performance of featgenn is still competitive with the original version, even though the performance decreases slightly with less available data. fig. 4. the performance of the different versions of featgenn is compared in terms of the amount of data used for computation. in the image, the symbol represents the version that used 60% of the available data, the * symbol represents the version that used 30% of the data, and the symbol \(^{\circ }\) represents the version that used 100% of the data. the featgenn without symbol stands for the version that used 80% of the available data. full size image 4.4 effectiveness of featgenn (rq3) in this subsection, this experiment aims to answer: can our featgenn with correlation-pooling achieve competitive results when compared to the state-of-the-art models?. we compare featgenn on 6 datasets with state-of-the-art methods, including (a) base: raw dataset without any transformation; (b) random: randomly apply a transformation to each raw feature; (c) dfs [4]; (d) autofeat [10]; (e) lfe [6]; (f) nfs [11]; and (g) difer [7]. table 3. comparison between featgenn with other methods from the literature, reported on [7]. * reports the average and standard deviation across 30 runs, while the featgenn column reports the maximum value across the same runs.full size table table 3 shows the comparative results of featgenn relative to existing methods (results reported in [7]). from table 3 we can observe that in the classification tasks, the comparison shows that featgenn, performs the best for the spambase, credit_default, german credit, and spectf benchmarks, the second best for the ionosphere benchmark and achieves the same result as the difer method for the megawatt1 benchmark. although difer achieves the best performance in the ionosphere benchmark, they only achieve 0.58% more than the best result obtained by our proposed method. regarding the number of features, table 4 shows that featgenn excels in producing fewer features for the megawatt1, spectf, and credit_default datasets compared to other methods. for the remaining datasets, featgenn achieves comparable results with the same number of features. compared to the performances of base and random, featgenn achieved an average improvement of 5.89% considering all datasets, which demonstrates the potential of the features generated by our proposed model. table 4. comparison between featgenn, difer, autofeat, and random (\(*\) the results reported on [7]).full size table 5 conclusion in this study, we presented a novel approach for generating new features in tabular data that combines feature selection and feature generation to improve the performance of predictive models. our proposed method uses a cnn architecture to effectively capture local features during convolution operations (local feature extraction), thereby reducing the number of combinations required in the mlp phase (global feature generation). in addition, we integrated a correlation-pooling operation as a dimensionality reduction step. our approach demonstrates efficient feature learning and achieves competitive results compared to the architecture used by max-pooling and state-of-the-art methods. as a direction for future research, we intend to explore information theory methods as possible alternatives for pooling operations. this could further increase the effectiveness of our approach to learning new features. references domingos, p.m.: a few useful things to know about machine learning. commun. acm 55, 78–87 (2012) article  google scholar  guo, h., tang, r., ye, y., li, z., he, x.: deepfm: a factorization-machine based neural network for ctr prediction. arxiv (2017). https://doi.org/10.48550/arxiv.1703.04247 cheng, h., et al.: wide & deep learning for recommender systems. arxiv (2016). https://doi.org/10.48550/arxiv.1606.07792 kanter, j.m., veeramachaneni, k.: deep feature synthesis: towards automating data science endeavors. in: 2015 ieee international conference on data science and advanced analytics (dsaa), pp. 1–10 (2015) google scholar  khurana, u., turaga, d., samulowitz, h., parthasrathy, s.: cognito: automated feature engineering for supervised learning (2016) google scholar  nargesian, f., samulowitz, h., khurana, u., khalil, e.b., turaga, d.s.: learning feature engineering for classification. in: ijcai (2017) google scholar  zhu, g., xu, z., guo, x., yuan, c., huang, y.: difer: differentiable automated feature engineering. arxiv abs/2010.08784 (2020) google scholar  székely, g.j., rizzo, m.l., bakirov, n.k.: measuring and testing dependence by correlation of distances. ann. statist. 35(6), 2769–2794 (2007). https://doi.org/10.1214/009053607000000505 article  mathscinet  math  google scholar  liu, b., tang, r., chen, y., yu, j., guo, h., zhang, y.: feature generation by convolutional neural network for click-through rate prediction. in: the world wide web conference (2019) google scholar  horn, f., pack, r.t., rieger, m.: the autofeat python library for automatic feature engineering and selection. arxiv abs/1901.07329 (2019) google scholar  chen, x., et al.: neural feature search: a neural architecture for automated feature engineering. in: 2019 ieee international conference on data mining (icdm), pp. 71–80 (2019) google scholar  peng, h., long, f., ding, c.: feature selection based on mutual information criteria of max-dependency, max-relevance, and min-redundancy. ieee trans. pattern anal. mach. intell. 27, 1226–1238 (2005) article  google scholar  katoch, s., chauhan, s.s., kumar, v.: a review on genetic algorithm: past, present, and future. multimed. tools appl. 80, 8091–8126 (2021) article  google scholar  bahdanau, d., cho, k., bengio, y.: neural machine translation by jointly learning to align and translate. corr abs/1409.0473 (2015) google scholar  iandola, f.n., moskewicz, m.w., ashraf, k., han, s., dally, w.j., keutzer, k.: squeezenet: alexnet-level accuracy with 50x fewer parameters and \(<\)1mb model size. arxiv abs/1602.07360 (2016) google scholar  cheng, h.-t., et al.: wide & deep learning for recommender systems. in: proceedings of the 1st workshop on deep learning for recommender systems (2016) google scholar  kaul, a., maheshwary, s., pudi, v.: autolearn automated feature generation and selection. in: 2017 ieee international conference on data mining (icdm), pp. 217–226 (2017). https://doi.org/10.1109/icdm.2017.31 goldberg, d.e., deb, k.: a comparative analysis of selection schemes used in genetic algorithms. in: rawlins, g.j.e. (ed.) foundations of genetic algorithms, pp. 69–93. morgan kaufmann publishers inc., san francisco (1991) google scholar  pearson, k.: note on regression and inheritance in the case of two parents. proc. r. soc. lond. 58, 240–242 (1895) article  google scholar  download references author information authors and affiliations universidade federal de ouro preto, ouro preto, 35402-163, brazil sammuel ramos silva & rodrigo silva cloudwalk, inc., são paulo, são paulo, 05425-070, brazil sammuel ramos silva authors sammuel ramos silvaview author publications search author on:pubmed google scholar rodrigo silvaview author publications search author on:pubmed google scholar corresponding author correspondence to sammuel ramos silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silva, s.r., silva, r. (2023). featgenn: improving model performance for tabular data with correlation-based feature extraction. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_17 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automated feature engineering feature creation correlation-based pooling tabular data machine learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature $$\alpha $$ -mcmp: trade-offs between probability and cost in ssps with the mcmp criterion | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper \(\alpha \)-mcmp: trade-offs between probability and cost in ssps with the mcmp criterion conference paper first online: 12 october 2023 pp 112–127 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) \(\alpha \)-mcmp: trade-offs between probability and cost in ssps with the mcmp criterion download book pdf download book epub gabriel nunes crispino  orcid: orcid.org/0000-0002-7641-84929, valdinei freire  orcid: orcid.org/0000-0003-0330-39319 & karina valdivia delgado  orcid: orcid.org/0000-0002-9120-89879  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 515 accesses 2 citations abstract in stochastic shortest path (ssp) problems, not always the requirement of having at least one policy with a probability of reaching goals (probability-to-goal) equal to 1 can be met. this is the case when dead ends, states from which the probability-to-goal is equal to 0, are unavoidable for any policy, which demands the definition of alternate methods to handle such cases. the \(\alpha \)-strong probability-to-goal priority is a property that is maintained by a criterion if a necessary condition to optimality is that the ratio between the probability-to-goal values of the optimal policy and any other policy is bound by a value of \(0 \le \alpha \le 1\). this definition is helpful when evaluating the preference of different criteria for ssps with dead ends. the min-cost given max-prob (mcmp) criterion is a method that prefers policies that minimize a well-defined cost function in the presence of unavoidable dead ends given policies that maximize probability-to-goal. however, it only guarantees \(\alpha \)-strong priority for \(\alpha = 1\). in this paper, we define \(\alpha \)-mcmp, a criterion based on mcmp with the addition of the guarantee of \(\alpha \)-strong priority for any value \(0 \le \alpha \le 1\). we also perform experiments comparing \(\alpha \)-mcmp and gubs, the only other criteria known to have \(\alpha \)-strong priority for \(0 \le \alpha \le 1\), to analyze the difference between the probability-to-goal of policies generated by each criterion. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others probability-to-goal and expected cost trade-off in stochastic shortest path chapter © 2021 masked triples chapter © 2021 the discounted cost model chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. operations research and decision theory probability and statistics in computer science public-private partnership socio-economic policy social choice and welfare utilitarianism 1 introduction markov decision processes (mdps) [11] constitute the main theoretical framework for modeling probabilistic planning problems, in which an agent interacts with an environment by applying actions with stochastic outcomes, to optimize a given objective function. in this context, stochastic shortest path (ssp) [1] problems model scenarios in which an agent needs to minimize the expected accumulated cost to goal. the solution to an ssp is a policy, a mapping between states to actions. conventionally, ssps assume that there is at least one policy that has a probability of 1 to reach goals when followed by the agent. realistically this is a strong requirement. environments commonly have dead ends, defined as states from which goals can not be reached. when these states exist and are unavoidable, the conventional requirement for solving ssps is not well-defined. this requires the definition of alternate criteria to address this particular case. several criteria were proposed in the literature in this context. the maxprob criterion [7] chooses policies that maximize the probability of reaching goals (probability-to-goal). the s3p [14] and isspude [6] criteria also prefer policies that maximize probability-to-goal, then choose the ones that minimize cost measures only considering histories that reach the goal. the mcmp criterion [16], on the other hand, chooses policies that maximize probability-to-goal, but then it minimizes a different cost measure that does not ignore histories that have dead-end states. this can lead to more natural optimal policies when compared to s3p and isspude since dead ends are taken into account. also, since it can be formulated as a linear program, efficient state-of-the-art methods can be used to solve this criterion. other criteria make trade-offs between probability-to-goal and cost measures, such that they do not only prefer policies that maximize probability-to-goal. the fsspude [6] criterion uses a finite penalty to give up, such that the agent can pay this value to exit the process at any step. a discount factor, commonly used in infinite horizon mdps [11], can also be used in ssps for making such trade-offs [15]. the gubs criterion [4] was proposed as an alternative for criteria that make trade-offs between probability-to-goal and cost measures. among other features, gubs maintains good theoretical properties, like the \(\alpha \)-strong probability-to-goal priority [2], which guarantees a lower bound on the ratio of probability-to-goal values considering pairs of policies, for a given value of \(0 \le \alpha \le 1\). between the criteria that were mentioned, gubs is the only one that guarantees \(\alpha \)-strong probability-to-goal priority for \(0 \le \alpha \le 1\). however, we can modify constraints in the mcmp linear programming formulation such that a new criterion, which also guarantees \(\alpha \)-strong probability-to-goal priority for \(0 \le \alpha \le 1\), is obtained. in this paper, we thus propose \(\alpha \)-mcmp, a criterion derived from modifying a constraint in mcmp’s linear programming formulation such that it guarantees \(\alpha \)-strong probability-to-goal priority for \(0 \le \alpha \le 1\). we also perform experiments to evaluate the differences in the preference of policies of \(\alpha \)-mcmp and gubs. the structure of this work is defined as follows: sect. 2 contains the background for this work, outlining definitions and properties of ssps and alternate criteria to solve them in the presence of unavoidable dead ends; sect. 3 introduces \(\alpha \)-mcmp and a proof that it maintains the \(\alpha \)-strong probability-to-goal priority property, as well as some differences between this new criterion and gubs; sect. 4 describes the empirical evaluation that was performed; and sect. 5 has the conclusion of the present work. 2 background the following subsections will cover definitions and properties that will be used throughout this paper. 2.1 stochastic shortest path (ssp) problems a stochastic shortest path (ssp) problem [1] is an indefinite-horizon stochastic process in which at time t the agent is at a state \(s_t\) and can choose an action \(a_t\) that will lead to \(s_{t + 1}\) given a probability distribution on \(s_t\) and a by paying a cost of \(c_t\). definition 1 an ssp [1] is a tuple \(\mathcal {m} = \langle \mathcal {s}, s_0, \mathcal {a}, p, c, \mathcal {g}\rangle \), where: \(\mathcal {s}\) is a finite set of states; \(s_0\in \mathcal {s}\) is the initial state; \(\mathcal {a}\) is a finite set of actions; \(p: \mathcal {s}\times \mathcal {a}\times \mathcal {s}\rightarrow [0, 1]\) is the transition function, such that \(p(s, a, s') = \pr (s_{t + 1} = s' \mid s_t = s, a_t = a)\); \(c: \mathcal {s}\times \mathcal {a}\rightarrow \mathbb {r}_{\ge 0}\) is the cost function, which assigns a cost c(s, a) for taking action a at state s; \(\mathcal {g}\subset \mathcal {s}\) is the set of absorbing goal states, i.e. \(c(g, a) = 0\) and \(p(g, a, g) = 1, \forall g\in \mathcal {g}\) and \(a\in \mathcal {a}\). also, \(c(s, a) > 0\) for \(s\in \mathcal {s}\setminus \mathcal {g}\). the objective in an ssp is to find a policy \(\pi \), a mapping between states to actions, that minimizes the expected cost to goal \(v^{\pi }(s) = \lim _{t\rightarrow \infty } \mathbb {e}\left[ \sum _{t=0}^{t-1}c_{t}\mid \pi , s_0 = s\right] \). the probability-to-goal of a policy \(\pi \) from state \(s\in \mathcal {s}\) is given by the function \(p^\pi _g(s) = \lim _{t\rightarrow \infty }\pr (s_{t}\in \mathcal {g}\mid \pi , s)\). we also define the maximum probability-to-goal of a state s in an ssp as the function \(p_g(s) = \max _{\pi \in \pi }p^\pi _g(s)\), in which \(\pi \) is the set of all policies. a policy \(\pi \) is proper if the probability-to-goal when following it is 1, i.e. \(p^\pi _g(s_0) = 1\). when the agent acts for multiple steps \(t \in \{0, 1, \dots , t\}\), she generates a history \(h = \{\langle s_1, a_1, c_1\rangle , \langle s_2, a_2, c_2\rangle , ..., \langle s_{t-1}, a_{t-1}, c_{t-1}\rangle , s_t \}\), where \(s_t\in \mathcal {s}\) is the state in which the agent is at step t, \(a_t\in \mathcal {a}\) is the action applied at \(s_t\), and \(c_t = c(s_t, a_t)\). we define \(\mathcal {h}=(\mathcal {s}\times \mathcal {a}\times \mathbb {r}_{>0})^{*}\times \mathcal {s}\) as the set of all histories. among methods to solve ssps, it is possible to describe an ssp as a linear program and solve it by using this program as an input to a solver [3]. in this lp, the objective is to find the values of \(x_{s, a}\), which are variables that represent the expected number of times action \(a\in \mathcal {a}\) is executed at state \(s\in \mathcal {s}\). this linear program is outlined in lp1. in(s) represents the expected flow entering s, while out(s) represents the expected flow leaving s. constraint (c4) restricts that except for \(s_0\) and goal states, the expected flow entering a state must be equal to the expected flow leaving it. (c5) indicates that the expected flow leaving \(s_0\) must be higher than the one entering \(s_0\) by a difference of 1. in other words, in expectation, the agent must leave the initial state one more time than she enters it (this does not consider the first step of the process). finally, the total expected flow reaching goal states must equal 1. the objective function then minimizes the expected total cost of reaching the goal given that the agent starts at \(s_0\), given the constraints specified. because the optimal policy \(\pi ^*\) that can be generated after solving lp1 is guaranteed to be deterministic, it can be defined as \(\pi ^*(s) = \arg \max _{a\in \mathcal {a}} x_{s, a}\). 2.2 ssps with dead ends and alternate criteria when the policy is improper, \(v^\pi \) is not well-defined and diverges. thus, when there does not exist a proper policy, the standard criterion for solving ssps is not well-defined either. this happens when dead ends, states from which no goal state can be reached, are unavoidable. figure 1 contains an example of an ssp with an unavoidable dead end. this problem has a state \(s_0\), an unavoidable dead-end state \(s_{de}\) and a goal state \(s_g\). at the state \(s_0\), action a leads to \(s_g\) deterministically with cost \(c_a\). action b has cost \(c_b\) and leads to \(s_g\) with probability p, and to \(s_{de}\) with probability \(1 p\). fig. 1. example of an ssp with an unavoidable dead-end. full size image for ssps with unavoidable dead ends, different criteria have to be used to solve these problems. one can maximize probability without optimizing cost, which is the case of the maxprob criterion [7]. instead of just maximizing probability, several criteria take the dual approach of minimizing a well-defined cost measure in the case of unavoidable dead ends, but only considering policies that maximize probability-to-goal. throughout this work, we will refer to this class of criteria as lexicographic. s3p [14] and mcmp [16] are examples of such criteria. when considering the example outlined in fig. 1, lexicographic criteria would choose a as the optimal action at \(s_0\), because it maximizes probability-to-goal. other criteria can make trade-offs between probability-to-goal and some cost measures. fsspudes [6], for example, use a finite penalty to pay when a dead end is reached. a discount factor [15] can also be used for this objective. additionally, the gubs criterion [4] has parameters that combine expected utility theory with goal prioritization to make such trade-offs. in the example in fig. 1, these criteria could either choose between actions a or b, depending on which values for their parameters are selected. sections 2.4 and 2.5 will detail the mcmp and gubs criteria, respectively. before that, the next subsection will formally define the \(\alpha \)-strong probability-to-goal priority property. 2.3 making trade-offs between probability-to-goal and cost measures when analyzing different criteria for solving ssps with unavoidable dead-ends, we might ask how to evaluate the decisions made by these criteria. one method for doing so is to take into account how each criterion maintains the \(\alpha \)-strong probability-to-goal priority property [2]. the definition for this property is given as follows: definition 2 (\(\alpha \) -strong probability-to-goal priority [2]). consider \( 0 \le \alpha \le 1\), we say that a decision criterion has \(\alpha \)-strong probability-to-goal priority if for all ssps \(\mathcal {m}\) and all pairs of policies \(\pi ,\pi '\in \pi \), the following condition is true: $$ \pi \succeq \pi ' \implies \frac{p_g^{\pi }(s_{0})}{p_g^{\pi '}(s_{0})} \ge \alpha , $$ where \(p_g^{\pi '}(s_{0}) > 0\). if \(\pi ^*\) is an optimal policy then for all \(\pi '\in \pi \) the following equation holds: $$ \pi ^* \succeq \pi ' \implies \frac{ p_g^{\pi ^*}(s_{0}) }{ p_g^{\pi '}(s_{0}) } \ge \alpha . $$ note that \(\frac{p_g^{\pi }(s_{0})}{p_g^{\pi '}(s_{0})} \ge \alpha \) is a necessary condition for \(\pi \succeq \pi '\) to be true. lexicographic criteria have 1-strong probability-to-goal priority, as it is guaranteed for these criteria that every policy preferred over another has a larger value of probability-to-goal [2]. the gubs criterion guarantees \(\alpha \)-strong priority for \(0\le \alpha \le 1\), while the fsspude and discounted cost criteria only have such guarantees for \(\alpha = 0\) (only 0-strong) [2]. 2.4 min-cost given max-prob (mcmp) the min-cost given max-prob (mcmp) criterion [16] shrinks trajectories by pruning histories in the first state that a dead-end state is reached, if that ever happens. this is formulated by the function \(\psi : \mathcal {h}\rightarrow \mathcal {h}\): $$\begin{aligned} \psi (h) = {\left\{ \begin{array}{ll} \{s_i\}, &{}\text {if } |h| = 1\text { or } p_g(s_i) = 0\\ \{\langle s_i, a_i, c_i \rangle \}\cup \psi (\{\langle s_{i + 1}, a_{i + 1}, c_{i + 1} \rangle , \dots \}), &{}\text {otherwise}, \end{array}\right. } \end{aligned}$$ (1) such that \(h = \{\langle s_i, a_i, c_i \rangle , \langle s_{i + 1}, a_{i + 1}, c_{i + 1} \rangle , \dots \}\). based on that, the value of a policy under the mcmp criterion is given by the following function: $$ \bar{c}_{mcmp}^\pi (s) = \mathbb {e}\left[ \sum _{t=0}^{|\psi (h)|}c_{t}\mid \pi ,s_{0}=s\right] . $$ finally, a policy \(\pi ^*_{mcmp}\) is optimal under mcmp if it minimizes \(\bar{c}_{mcmp}\) given that it maximizes \(p^\pi _g(s_0)\): $$\begin{aligned} \pi ^*_{mcmp} = \min _{\pi \in \pi _{mp}} \bar{c}_{mcmp}^\pi (s_0), \end{aligned}$$ such that \(\pi _{mp}\) is the set that maximizes \(p^\pi _g(s_0)\), i.e. \(\pi _{mp} = \{\pi \mid p^\pi _g(s_0) = \max _{\pi '\in \pi }p^{\pi '}_g(s_0)\}\). solving mcmp is equivalent to solving the following lp 2, a modified version of lp1. the differences between the original lp for ssps and lp2 are that the constraints (c4) and (c5) were replaced respectively by (c7) and (c8), in which equalities are replaced by inequalities, and (c6) was replaced by (c9), in which 1 is substituted by \(p_g(s_0)\). the inequality introduced in (c7) means that the expected flow when reaching a state (in(s)) is higher than the expected flow when leaving it (out(s)). this can be interpreted as an implicit give-up action with no cost that can be taken by the agent, which represents the pruning of histories that contain dead ends defined in the function \(\psi \). on a similar note, the inequality represented by (c8) means that the expected flow reaching the initial state \(s_0\) is higher than the expected flow leaving this same state. in other words, the agent always enters \(s_0\), since it is the initial state (thus \(in(s_0) = 1\)), but not always leaves without giving up (which means that \(out(s_0) >= 0\)). also note that a probabilistic markovian policy \(\pi ^*(s, a) = x_{s, a} / out(s)\) can be generated after solving lp2. to find \(p_g(s_0)\), another linear program, referred to as lp3, can be used [16]. in this lp, the objective function maximizes the total expected flow of reaching goal states, which is equivalent to the probability-to-goal from the initial state \(s_0\). since mcmp is a lexicographic criterion, it follows that it maintains 1-strong probability-to-goal priority [2]. as an example, consider the ssp displayed in the fig. 1. as mentioned before, mcmp will choose as optimal the policy that takes action a at \(s_0\). let \(\pi \) be this policy. the cost value under the mcmp criterion for \(\pi \) is \(c^\pi _{mcmp}(s_0) = c_a\). additionally, we can modify mcmp to make trade-offs between probability and cost by simply substituting \(p_g(s_0)\) in (c6) to any probability value p, and lp2 will then generate a policy \(\pi \) such that \(p^\pi _g(s_0) = p\) [8]. in the example of fig. 1, if we replace \(p_g(s_0)\) in (c9) with the value p, for instance, lp2 could return a policy that always takes b at \(s_0\), or some linear combination of actions a and b that would yield probability-to-goal of p, if the mcmp cost of doing so is less than always taking b. to illustrate this more concretely, consider the case of \(p = 0.95\), \(c_a = 2\), and \(c_b = 1\). if we replace \(p_g(s_0)\) in (c9) with \(p = 0.95\), the optimal policy returned by lp2 always takes b, yielding an expected cost of 1. however, if we replace \(p_g(s_0)\) with 0.98 in (c6) instead, the optimal policy returned in this case takes a with probability 0.6 and b with probability 0.4, leading to an expected cost of \(0.6 \times 2 + 0.4 \times 1 = 1.6\) and probability-to-goal \(0.6 \times 1 + 0.4\times 0.95 = 0.98\). note that, in this case, lp2 allows for a policy that, for example, takes a with a probability of 0.98 and takes the implicit give-up action with a probability of 0.02. this also yields a probability-to-goal of 0.98, thus fulfilling (c9), but it is not the optimal policy because its resulting expected cost is \(0.98 \times 2 = 1.96\), which is higher than 1.6. 2.5 goals with utility-based semantics (gubs) the goals with utility-based semantics (gubs) criterion [4] was proposed as an alternative method for solving ssps with unavoidable dead ends by combining goal prioritization with expected utility theory. it combines these two characteristics by its two parameters: the constant goal utility \(k_g\), and the utility function over cost u, respectively. definition 3 (gubs criterion [4]). the gubs criterion evaluates a history by the utility function u: $$\begin{aligned} u(c_t, \beta _t) = u(c_t) + k_g \beta _{t}. \end{aligned}$$ (2) an agent follows the gubs if she evaluates a policy \(\pi \) under the following value function: $$\begin{aligned} v^\pi _{gubs}(s) = \lim _{t\rightarrow \infty }\mathbb {e}[u(c_t, \beta _t) | \pi , s_0=s]. \end{aligned}$$ (3) a policy \(\pi ^*\) is optimal under the gubs criterion if it maximizes the function \(v^{\pi }_{gubs}\), i.e.: $$ v^{\pi ^{*}}_{gubs}(s) \ge v^{\pi }_{gubs}(s)\quad \forall \pi \in \pi \text { e } \forall s\in \mathcal {s}. $$ the gubs criterion is the only criterion among all other mentioned criteria that guarantees \(\alpha \)-strong probability-to-goal priority for any value \(0 \le \alpha \le 1\). corollary 1 (\(\alpha \) -strong probability-to-goal priority for gubs [2]). for an arbitrary \(0 \le \alpha \le 1\), the gubs criterion has \(\alpha \)-strong probability-to-goal priority. for \(0 \le \alpha < 1\), the property is maintained if the following condition is true: $$\begin{aligned} k_g \ge \frac{(u_{max} u_{min}) \alpha }{1 \alpha }, \end{aligned}$$ such that the values returned by u are in the set \([u_{min}, u_{max}]\). for \(\alpha = 1\), this property is true when \(u_{max} = u_{min}\) and \(k_g > 0\). the egubs criterion [5] is a specialization of gubs, in which the function u is an exponential function taken from risk-sensitive ssps [10]. definition 4 (egubs criterion [2, 5]). the egubs criterion considers the gubs criterion where the utility function u is defined as follows: $$\begin{aligned} u(c_t) = {\left\{ \begin{array}{ll} 0, \text {if } c_t = \infty \\ e^{\lambda c_t}, \text { otherwise}, \end{array}\right. } \end{aligned}$$ over some accumulated cost \(c_t\) and a risk factor \(\lambda <0\). in the gubs criterion, generated policies are non-markovian, since they need to have the state space augmented with the accumulated cost. from the properties of egubs, however, it is possible to use algorithms that guarantee that these non-markovian policies are finite, based on a maximum cost obtained from the problem structure [5]. egubs-vi [5] is an algorithm that does this by computing an optimal policy under egubs completely via value iteration, while egubs-ao* [2] is another algorithm that can compute an optimal policy by using a mix of value iteration and heuristic search. also, since for egubs \(u_{min} = 0\) and \(u_{max} = 1\), the lower bound obtained in corollary 1 can be restricted to \(k_g \ge \frac{\alpha }{1 \alpha }\) [2]. 3 \(\alpha \)-mcmp criterion in this section, we will show how to define a new criterion called \(\alpha \)-mcmp, which maintains the \(\alpha \)-strong probability-to-goal priority by leveraging the mcmp definition. mcmp is 1-strong since it is a criterion that maximizes probability-to-goal. however, we can relax the lp definition of mcmp to ensure it maintains the \(\alpha \)-strong probability-to-goal priority for \(0 \le \alpha \le 1\). this can be done by replacing the constraint (c9) with a new constraint (c10), which instead of making the total expected flow entering goal states to be equal to \(p_g(s_0)\) (the maximum probability-to-goal), it considers this flow as \(\alpha p_g(s_0)\). lp4 contains the modified version of lp2: this generates a new criterion that minimizes a similar yet different cost function than mcmp. throughout this work, we refer to this modified version of mcmp as \(\alpha \)-mcmp. it can be formally defined as the following: definition 5 given a value of \(0 \le \alpha \le 1\), a policy \(\pi ^*_{\alpha mcmp}\) is optimal under the \(\alpha \)-mcmp criterion if it minimizes \(\bar{c}_{mcmp}\) given policies with probability-to-goal equal to \(\alpha p_g(s_0)\): $$\begin{aligned} \pi ^*_{\alpha mcmp} = \min _{\pi \in \pi _{\alpha mp}} \bar{c}_{mcmp}^\pi (s_0), \end{aligned}$$ for \(\pi _{\alpha mp} = \{\pi \mid p^\pi _g(s_0) = \alpha p_g(s_0)\}\). the following theorem demonstrates that \(\alpha \)-mcmp thus maintains \(\alpha \)-strong priority for any \(0\le \alpha \le 1\). theorem 1 (\(\alpha \) -strong probability-to-goal priority for \(\alpha \) -mcmp). the \(\alpha \)-mcmp criterion has \(\alpha \)-strong probability-to-goal priority for any value \(0 \le \alpha \le 1\). proof by replacing (c9) with the constraint \(\sum \limits _{s_g\in \mathcal {g}}in(s_g) \ge \alpha p_g(s_0)\), the condition \(\pi ^*\succeq \pi \implies \frac{p^{\pi ^*}_g(s_0)}{p^{\pi }_g(s_0)}\ge \alpha \) from definition 2 holds, in which \(\pi ^*\) is the optimal policy generated by lp2 and \(\pi \) is any other policy. consider an arbitrary policy \(\pi '\) and let \(\pi \) be defined as the policy that follows \(\pi '\) with probability \(\frac{\alpha p_g(s_0)}{p^{\pi '}_g(s_0)}\) and gives up with probability \(1 \frac{\alpha p_g(s_0)}{p^{\pi '}_g(s_0)}\). the probability-to-goal value of \(\pi \) is \(p^{\pi }_g(s_0) = \frac{\alpha p_g(s_0)}{p^{\pi '}_g(s_0)}p^{\pi '}_g(s_0) + (1 \frac{\alpha p_g(s_0)}{p^{\pi '}_g(s_0)})\times 0 = \alpha p_g(s_0)\). it then follows that \(\pi \) is at least as good as \(\pi '\), because \(\bar{c}^{\pi }_{mcmp}(s_0) = \alpha p_g(s_0) \bar{c}^{\pi '}_{mcmp}(s_0) \le \bar{c}^{\pi '}_{mcmp}(s_0)\). thus, a policy with probability-to-goal \(\alpha p_g(s_0)\) will always be as good as any policy with probability-to-goal higher than \(\alpha p_g(s_0)\) when the constraint \(\sum \limits _{s_g\in \mathcal {g}}in(s_g) \ge \alpha p_g(s_0)\) is used. this means that using the constraint \(\sum \limits _{s_g\in \mathcal {g}}in(s_g) = \alpha p_g(s_0)\) (c10) is equivalent to using \(\sum \limits _{s_g\in \mathcal {g}}in(s_g) \ge \alpha p_g(s_0)\). since the restriction \(\sum \limits _{s_g\in \mathcal {g}}in(s_g) \ge \alpha p_g(s_0)\) maintains the \(\alpha \)-strong probability-to-goal priority property for \(0\le \alpha \le 1\) and (c10) is equivalent to using it, the \(\alpha \)-mcmp criterion maintains the \(\alpha \)-strong probability-to-goal priority property for \(0\le \alpha \le 1\).    \(\square \) 3.1 relationship between \(\alpha \)-mcmp and gubs by defining \(\alpha \)-mcmp, we now have a criterion that maintains the \(\alpha \)-strong probability-to-goal priority property for \(0 \le \alpha \le 1\) like gubs, but with the advantages of generating a markovian policy and having efficient solutions inherited from mcmp. although both criteria maintain the \(\alpha \)-strong priority for \(0\le \alpha \le 1\), the probability-to-goal of optimal policies in \(\alpha \)-mcmp will always equal \(\alpha p_g(s_0)\). in the gubs criterion, when choosing a value of \(k_g\) by following corollary 1, its resulting policy will not necessarily have a fixed probability-to-goal value of \(\alpha p_g(s_0)\). this value can instead be higher than \(\alpha p_g(s_0)\), if a deterministic policy with such a value exists and if the chosen utility function allows for that. for example, consider the egubs criterion and a value of \(\alpha = 0.95\). from corollary 1, a value of \(k_g \ge \frac{0.95}{0.05} = 19\) guarantees that egubs is 0.95-strong. also, consider the example in fig. 1, such that \(p = 0.95\) and the cost \(c_a\) is larger than \(c_b\) by a margin of \(\delta \), i.e., \(c_a = c_b + \delta \), for \(\delta > 0\). under this setting, egubs can choose a as the optimal action at \(s_0\) if the expected utility when following it is greater than the one when following b, i.e., \(u(c_a) + k_g > 0.95 (u(c_b) + k_g) + 0.05 u(\infty )\). considering \(k_g = 20 \ge 19\): $$\begin{aligned} u(c_a) + 20 &> 0.95 (u(c_b) + 20) + 0.05 u(\infty ) \\ e^{\lambda c_a} + 20 &> 0.95 e^{\lambda c_b} + 0.95 \times 20\\ e^{\lambda c_a} 0.95 e^{\lambda c_b} &> 0.95 \times 20 20\\ e^{\lambda c_a} 0.95 e^{\lambda c_b} &> -1 \end{aligned}$$ thus, for values of \(\lambda \), \(c_a\), and \(c_b\) such that \(e^{\lambda c_a} 0.95 e^{\lambda c_b} > -1\), action a will be optimal. for instance, if \(\delta = 1\), \(\lambda = -0.1\), \(c_b = 100\), and \(c_a = 100 + 1 = 101\), the utility of taking action a is \(e^{-0.1 \times 101} + 20 \approx 20\), which is higher than the utility of taking b that is \(0.95 (e^{-0.1 \times 100} + 20) \approx 19\). for this same setting, the optimal policy under \(\alpha \)-mcmp would be the one that chooses a with probability 0.95, and gives up with probability 0.05. this policy has an expected cost of \(0.95 \times 101 = 95.95\). in summary, for a value of \(\alpha = 0.95\) and depending on the trade-off incurred by the chosen value of \(\lambda \), the gubs criterion can choose an optimal policy \(\pi \) such that \(p^\pi _g(s_0) \ge \alpha p_g(s_0)\). \(\alpha \)-mcmp, on the other hand, will always select a policy that has a probability-to-goal value equal to \(\alpha p_g(s_0)\). table 1 summarizes some of the key differences between \(\alpha \)-mcmp and gubs. 4 experiments we have performed experiments in the navigation [12], river [4], and triangle tireworld [9] domains (figs. 2a, 2b, and 2c illustrate these domains, respectively) to evaluate \(\alpha \)-mcmp empirically and compare it to the gubs criterion. table 1. key differences between \(\alpha \)-mcmp and gubs.full size table the navigation domain is a grid world, where the agent starts at the rightmost column in the last row, and the goal is in the same column but in the first row. in the middle rows, the actions have a probability of making the agent disappear, thus going to a dead-end state. these probabilities are lower on the left and higher on the right. the river domain is similar, also being a grid world, but representing an agent at a river bank having to cross the river to get to the goal that is on the other side. actions in the river have a positive probability of making the agent fall down one row, while actions in the bank and the bridge, located in the first row, are deterministic. if the agent gets to the first row after falling down in the river, the waterfall is reached, in which no actions to other states are available, thus being a dead end. finally, in the triangle tireworld domain, the agent is represented by a car that needs to go to a goal location, always with a positive probability of getting a flat tire. certain states have available tires to change, but these are in general further away from the goal than states with no tires available. in the instances depicted in fig. 2c, for example, the black circles represent locations that have available tires. if the car gets a flat tire after getting to a state with no tire to replace, then a dead end is reached. we aimed to evaluate the different probability-to-goal values obtained from optimal policies generated from both criteria when varying their parameters. for that, we varied \(\alpha \) between the values of \(\{10^{-5}, 10^{-4}, 10^{-3},\) \(10^{-2}, 0.1, 0.5, 0.999\}\) and computed lp4 to obtain the optimal policy of \(\alpha \)-mcmp. for gubs, we used the value of \(k_g\) as in corollary 1 as a parameter for the given values of \(\alpha \) and then computed the egubs criterion for this value of \(k_g\) and different values of \(\lambda \) using the egubs-vi algorithm. the values of \(\lambda \) were \(\{-0.05,\) \(-0.06, -0.07, -0.08, -0.09, -0.1, -0.2\}\) for the navigation domain, \(\{-0.01, -0.1,\) \( -0.2,-0.3, -0.4, -0.5\}\) for the river domain, and \(\{-0.1, -0.2, -0.25,-0.3,-0.35,\) \( -0.4\}\) for the triangle tireworld domain. a single instance was used for each domain. the number of states for each of these instances is 101, 200 and 19562 for the navigation, river, and triangle tireworld domains, respectively. note that the numbers of processed states can be considerably higher for egubs because it might need to process a high number of augmented states. the domains were defined as pddlgym [13] environments, and the code of the implementation is available at https://github.com/gcrispino/ssp-deadends/tree/bracis2023-paper. fig. 2. illustration of the domains used in the experiments. full size image figures 3, 4, and 5 contain graphs that display the probability-to-goal values for each policy obtained for gubs and \(\alpha \)-mcmp given different values of \(\alpha \) (displayed in \(\log _{10}\) scale in the x-axis), respectively for domains navigation, river and triangle tireworld. from them, it can be observed that, as mentioned before, while the probability-to-goal obtained by \(\alpha \)-mcmp always equals \(\alpha p_g(s_0)\), the values obtained from egubs’ optimal policies are different. in the different linesfootnote 1 reflecting different values of the risk factor \(\lambda \), the probability-to-goal values are always higher than \(\alpha p_g(s_0)\). how higher these probabilities are, though, depends on the chosen values of \(\lambda \). for example, in the river domain, we can observe several different types of probability-to-goal curves when varying \(\lambda \). the one that results in the widest difference between the minimum and the maximum probability-to-goal values is when \(\lambda = -0.5\), for which the minimum probability-to-goal obtained was about 0.07 for \(\alpha = 10^{-5}\), and the maximum value was 1, obtained when \(\alpha \in \{10^{-3}, 10^{-2}, 0.1, 0.5, 0.999\}\). fig. 3. results for the navigation domain. full size image fig. 4. results for the river domain. full size image in fact, for all domains, we observed that the higher the absolute value of \(\lambda \) is, the higher the difference between the minimum and maximum probability-to-goal values that were obtained for this value of \(\lambda \) are. it is also interesting to note that intermediate values of \(\lambda \) can cover intermediate values of probability-to-goal that higher values of \(\lambda \) used cannot. for example, in the river domain, when \(\lambda \) is equal to \(-0.1\) and \(-0.2\), values of probability-to-goal close to 0.6 were obtained, which was not the case when the absolute value of \(\lambda \) was higher than \(-0.2\) (i.e. when \(\lambda \) is equal to \(-0.3, -0.4\), and \(-0.5\)). as another example, in the navigation domain when \(\lambda = -0.1\), the value of about 0.47 was obtained, while for different values of \(\lambda \) such as \(-0.2\) and for absolute values lower than 0.09 (i.e. when \(\lambda \) is equal to \(-0.05, -0.06, -0.07\), and \(-0.08\)), either a value close to 0 or close to the maximum probability-to-goal was obtained. fig. 5. results for the triangle tireworld domain. full size image 5 conclusion in this paper, we define \(\alpha \)-mcmp, a criterion to solve ssps with unavoidable dead ends, that is obtained from mcmp. we also show how the modifications that were done to mcmp to generate \(\alpha \)-mcmp make it guarantee the \(\alpha \)-strong probability-to-goal priority property. besides this, \(\alpha \)-mcmp also provides good advantages inherited from mcmp, such as markovian policies and efficient state-of-the-art methods to solve it. it also has some differences when compared to gubs, such as when preferring policies that might give up in the middle of the process instead of policies that do not do it. finally, we performed experiments to evaluate policies obtained by \(\alpha \)-mcmp and compared them to policies generated by the egubs criterion. the results indicate that we can make trade-offs in both criteria by choosing values of \(\alpha \) a priori. nonetheless, the way that these criteria make this trade-off is different. \(\alpha \)-mcmp always has probability-to-goal equal to \(\alpha p_g(s_0)\), while the compromise made by egubs will depend on the value of \(\lambda \) used in its utility function. this paper attempts to contribute in the general understanding of sequential decision making in the presence of unavoidable dead-end states, such as several other works in this area [2, 4,5,6,7, 14,15,16]. notes 1.note that not every line representing a value of \(\lambda \) can be seen in the figures, because the values in these lines might be very close to the values in others, which can make them get covered by these other lines. references bertsekas, d.: dynamic programming and optimal control. athena scientific, belmont, mass (1995) math  google scholar  crispino, g.n., freire, v., delgado, k.v.: gubs criterion: arbitrary trade-offs between cost and probability-to-goal in stochastic planning based on expected utility theory. artif. intell. 316, 103848 (2023) article  mathscinet  math  google scholar  d’epenoux, f.: a probabilistic production and inventory problem. manage. sci. 10(1), 98–108 (1963) article  google scholar  freire, v., delgado, k.v.: gubs: a utility-based semantic for goal-directed markov decision processes. in: proceedings of the 16th conference on autonomous agents and multiagent systems, pp. 741–749 (2017) google scholar  freire, v., delgado, k.v., reis, w.a.s.: an exact algorithm to make a trade-off between cost and probability in ssps. in: proceedings of the international conference on automated planning and scheduling, vol. 29, pp. 146–154 (2019) google scholar  kolobov, a., weld, d., et al.: a theory of goal-oriented mdps with dead ends. in: uncertainty in artificial intelligence: proceedings of the twenty-eighth conference [on uncertainty in artificial intelligence] (2012), pp. 438–447 (2012) google scholar  kolobov, a., weld, d.s., geffner, h.: heuristic search for generalized stochastic shortest path mdps. in: proceedings of the twenty-first international conference on international conference on automated planning and scheduling, pp. 130–137 (2011) google scholar  kuo, i., freire, v.: probability-to-goal and expected cost trade-off in stochastic shortest path. in: gervasi, o., murgante, b., misra, s., garau, c., blečić, i., taniar, d., apduhan, b.o., rocha, a.m.a.c., tarantino, e., torre, c.m. (eds.) iccsa 2021. lncs, vol. 12951, pp. 111–125. springer, cham (2021). https://doi.org/10.1007/978-3-030-86970-0_9 chapter  google scholar  little, i., thiebaux, s., et al.: probabilistic planning vs. replanning. in: icaps workshop on ipc: past, present and future, pp. 1–10 (2007) google scholar  patek, s.d.: on terminating markov decision processes with a risk-averse objective function. automatica 37(9), 1379–1386 (2001) article  math  google scholar  puterman, m.: markov decision processes: discrete stochastic dynamic programming. wiley, new york (1994) book  math  google scholar  sanner, s., yoon, s.: ippc results presentation. in: international conference on automated planning and scheduling (2011). http://users.cecs.anu.edu.au/ssanner/ippc_2011/ippc_2011_presentation.pdf silver, t., chitnis, r.: pddlgym: gym environments from pddl problems. in: international conference on automated planning and scheduling (icaps) prl workshop, pp. 1–6 (2020). https://github.com/tomsilver/pddlgym teichteil-königsbuch, f.: stochastic safest and shortest path problems. in: proceedings of the twenty-sixth aaai conference on artificial intelligence, pp. 1825–1831 (2012) google scholar  teichteil-königsbuch, f., vidal, v., infantes, g.: extending classical planning heuristics to probabilistic planning with dead-ends. in: proceedings of the twenty-fifth aaai conference on artificial intelligence, pp. 1017–1022 (2011) google scholar  trevizan, f.w., teichteil-königsbuch, f., thiébaux, s.: efficient solutions for stochastic shortest path problems with dead ends. in: proceedings of the thirty-third conference on uncertainty in artificial intelligence (uai) (2017), pp. 1–10 (2017) google scholar  download references acknowledgments this study was supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior (capes) finance code 001, by the são paulo research foundation (fapesp) grant \(\#\)2018/11236-9 and the center for artificial intelligence (c4ai-usp), with support by fapesp (grant #2019/07665-4) and by the ibm corporation. author information authors and affiliations universidade de são paulo, são paulo, brazil gabriel nunes crispino, valdinei freire & karina valdivia delgado authors gabriel nunes crispinoview author publications search author on:pubmed google scholar valdinei freireview author publications search author on:pubmed google scholar karina valdivia delgadoview author publications search author on:pubmed google scholar corresponding author correspondence to gabriel nunes crispino . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper crispino, g.n., freire, v., delgado, k.v. (2023). \(\alpha \)-mcmp: trade-offs between probability and cost in ssps with the mcmp criterion. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_8 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords decision making under uncertainty markov decision processes stochastic shortest path publish with us policies and ethics profiles gabriel nunes crispino view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an incremental maxsat-based model to learn interpretable and balanced classification rules | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an incremental maxsat-based model to learn interpretable and balanced classification rules conference paper first online: 12 october 2023 pp 227–242 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) an incremental maxsat-based model to learn interpretable and balanced classification rules download book pdf download book epub antônio carlos souza ferreira júnior9 & thiago alves rocha9  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 536 accesses abstract the increasing advancements in the field of machine learning have led to the development of numerous applications that effectively address a wide range of problems with accurate predictions. however, in certain cases, accuracy alone may not be sufficient. many real-world problems also demand explanations and interpretability behind the predictions. one of the most popular interpretable models that are classification rules. this work aims to propose an incremental model for learning interpretable and balanced rules based on maxsat, called imlib. this new model was based on two other approaches, one based on sat and the other on maxsat. the one based on sat limits the size of each generated rule, making it possible to balance them. we suggest that such a set of rules seem more natural to be understood compared to a mixture of large and small rules. the approach based on maxsat, called imli, presents a technique to increase performance that involves learning a set of rules by incrementally applying the model in a dataset. finally, imlib and imli are compared using diverse databases. imlib obtained results comparable to imli in terms of accuracy, generating more balanced rules with smaller sizes. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others conceptual challenges for interpretable machine learning article open access 01 march 2022 interpretable machine learning – a brief history, state-of-the-art and challenges chapter © 2020 active class incremental learning for imbalanced datasets chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization learning algorithms learning theory machine learning model theory statistical learning 1 introduction the success of machine learning (ml) in recent years has led to a growing advancement in studies in this area [2, 8, 12]. several applications have emerged with the aim of circumventing various problems and situations [4, 14, 20]. one such problem is the lack of explainability of prediction models. this directly affects the reliability of using these applications in critical situations involving, for example, finance, autonomous systems, damage to equipment, the environment, and even lives [1, 7, 23]. that said, some works seek to develop approaches that bring explainability to their predictions [13, 21, 22]. precise predictions with high levels of interpretability are often not a simple task. there are some works that try to solve this problem by balancing the accuracy of the prediction with the interpretability [5, 6, 9, 15,16,17, 24]. it can be seen that some of these works use approaches based on the boolean satisfiability problem (sat) and the maximum boolean satisfiability problem (maxsat). the choice of these approaches to solve this problem has been increasingly recurrent in recent years. the reasons can be seen in the results obtained by these models. sat-based approaches have been proposed recently [18, 19] to learn quantifier-free first-order sentences from a set of classified strings. more specifically, given a set of classified strings, the goal is to find a first-order sentence over strings of minimum size that correctly classifies all the strings. one of the approaches demonstrated is sqfsat (synthesis of quantifier-free first-order sentences over strings with sat). upon receiving a set of classified strings, this approach generates a quantifier-free first-order sentence over strings in disjunctive normal form (dnf) with a given number of terms. what makes this method stand out is the fact that we can limit both the number of terms and the number of formulas per term in the generated formula. in addition, as the approach generates formulas in dnf, each term of the formula can be seen as a rule. then, for each rule, its explanation is the conjunction of formulas in the rule, which can be interesting for their interpretability [11, 18]. on the other hand, as the model is based on the sat problem, in certain situations it may bring results that are not so interesting in terms of interpretability and efficiency, such as in cases where the set of strings is large. ghosh, b. et al. created a classification model based on maxsat called imli [6]. the approach takes a set of classified samples, represented by vectors of numerical and categorical data, and generates a set of rules expressed in dnf or in conjunctive normal form (cnf) that correctly classifies as many samples as possible. in this work, we focus on using imli for learning rules in dnf. the number of rules in the set of rules can be defined similarly to sqfsat, but imli does not consider the number of elements per rule. although imli focuses on learning a sparse set of rules, it may obtain a combination of both large and small rules. imli also takes into account the option of defining a weighting for correct classifications. as the weighting increases, the accuracy of the model improves, but at the cost of an increase in the size of the generated set of rules. the smaller the weighting, the lower the accuracy of the model, but correspondingly, the generated set of rules tends to be smaller. furthermore, imli uses an incremental approach to achieve better runtime performance. the incremental form consists of dividing the set of samples into partitions in order to generate a set of rules for each partition from the set of rules obtained in the previous partitions. in this work, we aim to create a new approach for learning interpretable rules based on maxsat that unites sqfsat with the incrementality of imli. the motivation for choosing sqfsat is the possibility of defining the number of literals per clause, allowing us to generate smaller and more balanced rules. the choice of imli is motivated by its incrementability technique, which allows the method to train on large sets of samples efficiently. in addition, we propose a technique that reduces the size of the generated rules, removing possible redundancies. this work is divided into 6 sections. in sect. 2, we define the general notions and notations. since all methods presented in this paper use boolean logic, we also define in sect. 2 how these methods binarize datasets with numerical and categorical data. in sect. 3, sqfsat and imli are presented, respectively. we present sqfsat in the context of our work where samples consist of binary vectors instead of strings, and elements of rules are not first-order sentences over strings. in sect. 4, our contribution is presented: imlib. in sect. 5, we describe the experiments conducted and the results for the comparison of our approach against imli. finally, in the last section, we present the conclusions and indicate future work. 2 preliminaries we consider the binary classification problem where we are given a set of samples and their classifications. the set of samples is represented by a binary matrix of size \(n \times m\) and their classifications by a vector of size n. we call the matrix \({\textbf {x}}\) and the vector \({\textbf {y}}\). each row of \({\textbf {x}}\) is a sample of the set and we will call it \({\textbf {x}}_i\) with \(i \in \{1,...,n\}\). to represent a specific value of \({\textbf {x}}_i\), we will use \(x_{i,j}\) with \(j \in \{1,...,m\}\). each column of \({\textbf {x}}\) has a label representing a feature and the label is symbolized by \(x^j\). to represent a specific value of \({\textbf {y}}\), we will use \(y_i\). to represent the opposite value of \(y_i\), that is, if it is 1 the opposite value is 0 and vice versa, we use \(\lnot y_i\). therefore, we will use the symbol \(\lnot {\textbf {y}}\) to represent \({\textbf {y}}\) with all opposite values. to represent the opposite value of \(x_{i,j}\), we use \(\lnot x_{i,j}\). therefore, we will use the symbol \(\lnot {\textbf {x}}_i\) to represent \({\textbf {x}}_i\) with all opposite values. each label also has its opposite label which is symbolized by \(\lnot x^j\). a partition of \({\textbf {x}}\) is represented by \({\textbf {x}}^t\) with \(t \in \{1,...,p\}\), where p is the number of partitions. therefore, the partitions of vector \({\textbf {y}}\) are represented by \({\textbf {y}}^t\). each element of \({\textbf {y}}\) is symbolized by \(y_i\) and represents the class value of sample \({\textbf {x}}_i\). we use \(\mathcal {e}^= \{x_i \mid y_i = 0, 1 \le i \le n\}\) and \(\mathcal {e}^+ = \{x_i \mid y_i = 1, 1 \le i \le n\}\). to represent the size of these sets, that is, the number of samples contained in them, we use the notations: \(|\mathcal {e}^-|\) and \(|\mathcal {e}^+|\). example 1 let \({\textbf {x}}\) be the set of samples $${\textbf {x}} = \left[ \begin{array}{ccc} 0&{}0&{}1\\ 0&{}1&{}0\\ 0&{}1&{}1\\ 1&{}0&{}0 \end{array}\right] $$ and their classifications \({\textbf {y}} = [1, 0, 0, 1]\). the samples \({\textbf {x}}_i\) are: \({\textbf {x}}_1 = [0, 0, 1], ..., {\textbf {x}}_4 = [1, 0, 0]\). the values of each sample \(x_{i,j}\) are: \(x_{1,1}=0, x_{1,2}=0, x_{1,3}=1, x_{2,1}=0, ..., x_{4,3}=0\). the class values \(y_i\) of each sample are: \(y_1=1, ..., y_4=1\). we can divide \({\textbf {x}}\) into two partitions in several different ways, one of which is: \({\textbf {x}}^1 = \left[ \begin{array}{ccc} 0&{}1&{}1\\ 0&{}1&{}0 \end{array}\right] \), \({\textbf {y}}^1 = [0, 0]\), \({\textbf {x}}^2 = \left[ \begin{array}{ccc} 1&{}0&{}0\\ 0&{}0&{}1 \end{array}\right] \) e \({\textbf {y}}^2 = [1, 1]\). example 2 let \({\textbf {x}}\) be the set of samples from example 1, then $$\lnot {\textbf {x}} = \left[ \begin{array}{ccc} 1&{}1&{}0\\ 1&{}0&{}1\\ 1&{}0&{}0\\ 0&{}1&{}1 \end{array}\right] $$ and \(\lnot {\textbf {y}} = [0, 1, 1, 0]\). the samples \(\lnot {\textbf {x}}_i\) are: \(\lnot {\textbf {x}}_1 = [1, 1, 0], ..., \lnot {\textbf {x}}_4 = [0, 1, 1]\). the values of each sample \(\lnot x_{i,j}\) are: \(\lnot x_{1,1}=1, \lnot x_{1,2}=1, \lnot x_{1,3}=0, \lnot x_{2,1}=1, ..., \lnot x_{4,3}=1\). the class values of each sample \(\lnot y_i\) are: \(\lnot y_1=0, ..., \lnot y_4=0\). we can divide \(\lnot {\textbf {x}}\) in partitions as in example 1: \(\lnot {\textbf {x}}^1 = \left[ \begin{array}{ccc} 1&{}0&{}0\\ 1&{}0&{}1 \end{array}\right] \), \(\lnot {\textbf {y}}^1 = [1, 1]\), \(\lnot {\textbf {x}}^2 = \left[ \begin{array}{ccc} 0&{}1&{}1\\ 1&{}1&{}0 \end{array}\right] \) e \(\lnot {\textbf {y}}^2 = [0, 0]\). we define a set of rules being the disjunction of rules and is represented by \({\textbf {r}}\). a rule is a conjunction of one or more features. each rule in \({\textbf {r}}\) is represented by \(r_o\) with \(o \in \{1,...,k\}\), where k is the number of rules. moreover, \({\textbf {r}}({\textbf {x}}_i)\) represents the application of \({\textbf {r}}\) to \({\textbf {x}}_i\). the notations \(|{\textbf {r}}|\) and \(|r_o|\) are used to represent the number of features in \({\textbf {r}}\) and \(r_o\), respectively. example 3 let \(x^1=\) man, \(x^2=\) smoke, \(x^3=\) hike be labels of features. let \({\textbf {r}}\) be the set of rules \({\textbf {r}} =\) (man) \(\vee \) (smoke \(\wedge \) \(\lnot \)hike). the rules \(r_o\) are: \(r_1=\) (man) and \(r_2=\) (smoke \(\wedge \) \(\lnot \)hike). the application of \({\textbf {r}}\) to \({\textbf {x}}_i\) is represented as follows: \({\textbf {r}} ({\textbf {x}}_i) = x_{i,1} \vee (x_{i,2} \wedge \lnot x_{i,3})\). for example, let \({\textbf {x}}\) be the set of samples from example 1, then: \({\textbf {r}}({\textbf {x}}_1) = x_{1,1} \vee (x_{1,2} \wedge \lnot x_{1,3}) = 0 \vee (0 \wedge 0) = 0\). moreover, we have that \(|{\textbf {r}}| = 3\), \(|r_1| = 1\) and \(|r_2| = 2\). as we assume a set of binary samples, we need to perform some preprocessing. preprocessing consists of binarizing a set of samples with numerical or categorical values. the algorithm divides the features into four types: constant, where all samples have the same value; binary, where there are only two distinct variations among all the samples for the same value; categorical, when the feature does not fit in constant and binary and its values are three or more categories; ordinal, when the feature does not fit into constant and binary and has numerical values. when the feature type is constant, the algorithm discards that feature. this happens due to the fact that a feature common to all samples makes no difference in the generated rules. when the type is binary, one of the feature variations will receive 0 and the other 1 as new values. if the type is categorical, we employ the widely recognized technique of one-hot encoding. finally, for the ordinal type feature, a quantization is performed, that is, the variations of this feature are divided into quantiles. with this, boolean values are assigned to each quantile according to the original value. we use sat and maxsat solvers to implement the methods presented in this work. a solver receives a formula in cnf, for example: \((p \vee q) \wedge (q \vee \lnot p)\). furthermore, a maxsat solver receives weights that will be assigned to each clause in the formula. a clause is the disjunction of one or more literals. the weights are represented by \(w(cl) = w\) where cl is one or more clauses and w represents the weight assigned to each one of them. a sat solver tries to assign values to the literals in such a way that all clauses are satisfied. a maxsat solver tries to assign values to the literals in a way that the sum of the weights of satisfied clauses is maximum. clauses with numerical weights are considered soft. the greater the weight, the greater the priority of the clause to be satisfied. clauses assigned a weight of \(\infty \) are considered hard and must be satisfied. 3 rule learning with sat and maxsat 3.1 sqfsat sqfsat is a sat-based approach that, given x, y, k and the number of features per rule l, tries to find a set of rules \({\textbf {r}}\) with k rules and at most l features per rule that correctly classify all samples \({\textbf {x}}_i\), that is, \({\textbf {r}}({\textbf {x}}_i) = y_i\) for all i. in general, the approach takes its parameters \({\textbf {x}}\), \({\textbf {y}}\), k and l and constructs a cnf formula to apply it to a sat solver, which returns an answer that is used to get \({\textbf {r}}\). the construction of the sat clauses is defined by propositional variables: \(u_{o,d}^j\), \(p_{o,d}\), \(u_{o,d}^*\), \(e_{o,d,i}\) and \(z_{o,i}\), for \(d \in \{1,...,l\}\). if the valuation of \(u_{o,d}^j\) is true, it means that jth feature label will be the dth feature of the rule \(r_o\). furthermore, if \(p_{o,d}\) is true, it means that the dth feature of the rule \(r_o\) will be \(x^j\), in other words, will be positive. otherwise, it will be negative: \(\lnot x^j\). if \(u_{o,d}^*\) is true, it means that the dth feature is skipped in the rule \(r_o\). in this case, we ignore \(p_{o,d}\). if \(e_{o,d,i}\) is true, then the dth feature of rule \(r_o\) contributes to the correct classification of the ith sample. if \(z_{o,i}\) is true, then the rule \(r_o\) contributes to the correct classification of the ith sample. that said, below, we will show the constraints formulated in the model for constructing the sat clauses. conjunction of clauses that guarantees that exactly one \(u_{o,d}^j\) is true for the dth feature of the rule \(r_o\): $$\begin{aligned} a = \bigwedge _{o \in \{1,...,k\} \atop d \in \{1,...,l\}} \bigvee _{j \in \{1,...,m,*\}} u_{o,d}^j \end{aligned}$$ (1) $$\begin{aligned} b = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop j,j' \in \{1,...,m,*\}, j \ne j'}} (\lnot u_{o,d}^j \vee \lnot u_{o,d}^{j'}) \end{aligned}$$ (2) conjunction of clauses that ensures that each rule has at least one feature: $$\begin{aligned} c = \bigwedge _{o \in \{1,...,k\}} \bigvee _{d \in \{1,...,l\}} \lnot u_{o,d}^* \end{aligned}$$ (3) we will use the symbol \(s_{o,d,i}^j\) to represent the value of the ith sample in the jth feature label of \({\textbf {x}}\). if this value is 1, it means that if the jth feature label is in the dth position of the rule \(r_o\), then it contributes to the correct classification of the ith sample. therefore, \(s_{o,d,i}^j = e_{o,d,i}\). otherwise, \(s_{o,d,i}^j = \lnot e_{o,d,i}\). that said, the following conjunction of formulas guarantees that \(e_{o,d,i}\) is true if the jth feature in the oth rule contributes to the correct classification of the sample \({\textbf {x}}_i\): $$\begin{aligned} d = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop {j \in \{1,...,m\} \atop i \in \{1,...,n\}}}} u_{o,d}^j \rightarrow (p_{o,d} \leftrightarrow s_{o,d,i}^j) \end{aligned}$$ (4) conjunction of formulas guaranteeing that if the dth feature of a rule is skipped, then the classification of this rule is not interfered by this feature: $$\begin{aligned} e = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop i \in \{1,...,n\}}} u_{o,d}^* \rightarrow e_{o,d,i} \end{aligned}$$ (5) conjunction of formulas indicating that \(z_{o,i}\) will be set to true if all the features of rule \(r_o\) contribute to the correct classification of sample \({\textbf {x}}_i\): $$\begin{aligned} f = \bigwedge _{o \in \{1,...,k\}} \bigwedge _{i \in \{1,...,n\}} z_{o,i} \leftrightarrow \bigwedge _{d \in \{1,...,l\}} e_{o,d,i} \end{aligned}$$ (6) conjunction of clauses that guarantees that \({\textbf {r}}\) will correctly classify all samples: $$\begin{aligned} g = \bigwedge _{i \in \mathcal {e}^+} \bigvee _{o \in \{1,...,k\}} z_{o,i} \end{aligned}$$ (7) $$\begin{aligned} h = \bigwedge _{i \in \mathcal {e}^-} \bigwedge _{o \in \{1,...,k\}} \lnot z_{o,i} \end{aligned}$$ (8) next, the formula q below is converted to cnf. then, finally, we have the sat query that is sent to the solver. $$\begin{aligned} q = a \wedge b \wedge c \wedge d \wedge e \wedge f \wedge g \wedge h \end{aligned}$$ (9) 3.2 imli imli is an incremental approach based on maxsat for learning interpretable rules. given \({\textbf {x}}\), \({\textbf {y}}\), k, and a weight \(\lambda \), the model aims to obtain the smallest set of rules \({\textbf {m}}\) in cnf that correctly classifies as many samples as possible, penalizing classification errors with \(\lambda \). in general, the method solves the optimization problem \(\min _{{\textbf {m}}} \{|{\textbf {m}}| + \lambda |\mathcal {e}_m| \mid \mathcal {e}_m = \{{\textbf {x}}_i \mid {\textbf {m}}({\textbf {x}}_i) \ne y_i\}\}\), where \(|{\textbf {m}}|\) represents the number of features in \({\textbf {m}}\) and \({\textbf {m}}({\textbf {x}}_i)\) denotes the application of the set of rules \({\textbf {m}}\) to \({\textbf {x}}_i\). therefore, the approach takes its parameters \({\textbf {x}}\), \({\textbf {y}}\), k and \(\lambda \) and constructs a maxsat query to apply it to a maxsat solver, which returns an answer that is used to generate \({\textbf {m}}\). note that imli generates set of rules in cnf, whereas our objective is to obtain sets of rules in dnf. for that, we will have to use as parameter \(\lnot {\textbf {y}}\) instead of \({\textbf {y}}\) and negate the set of rules \({\textbf {m}}\) to obtain a set of rules \({\textbf {r}}\) in dnf. the construction of the maxsat clauses is defined by propositional variables: \(b_o^v\) and \(\eta _i\), for \(v \in \{1,...,2m\}\). the v ranges from 1 to 2m, as it also considers opposite features. if the valuation of \(b_o^v\) is true and \(v \le m\), it means that feature \(x^v\) will be in the rule \(m_o\), where \(m_o\) is the oth rule of \({\textbf {m}}\). if the valuation of the \(b_o^v\) is true and \(v > m\), it means that feature \(\lnot x^{v m}\) will be in the rule \(m_o\). if the valuation of \(\eta _i\) is true, it means that sample \({\textbf {x}}_i\) is not classified correctly, that is, \({\textbf {m}}({\textbf {x}}_i) \ne y_i\). that said, below, we will show the constraints for constructing maxsat clauses. constraints that represent that the cost of a misclassification is \(\lambda \): $$\begin{aligned} a = \bigwedge _{i \in \{1,...,n\}} \lnot \eta _i, w(a) = \lambda \end{aligned}$$ (10) constraints that represent that the model tries to insert as few features as possible in \({\textbf {m}}\), taking into account the weights of all clauses: $$\begin{aligned} b = \bigwedge _{v \in \{1,...,2m\} \atop o \in \{1,...,k\}} \lnot b_o^v, w(b) = 1 \end{aligned}$$ (11) even though the constraints in 11 prioritize learning sparse rules, they do so by directing attention to the overall set of rules, i.e. in the total number of features in \({\textbf {m}}\). then, imli may generate a set of rules that comprises a combination of both large and small rules. in our approach presented in sect. 4, we address this drawback by limiting the number of features in each rule. we will use \({\textbf {l}}_o\) to represent the set of variables \(b_o^v\) of a rule \(m_o\), that is, \({\textbf {l}}_o = \{b_o^v\, | v \in \{1, ...,2m\}\}\), for \(o \in \{1,...,k\}\). to represent the concatenation of two samples, we will use the symbol \(\cup \). we also use the symbol @ to represent an operation between two vectors of the same size. the operation consists of applying a conjunction between the corresponding elements of the vectors. subsequently, a disjunction between the elements of the result is applied. the following example illustrates how these definitions will be used: example 4 let be \({\textbf {x}}_4\) as in example 1, \({\textbf {x}}_4 \cup \lnot {\textbf {x}}_4 = [1,0,0,0,1,1]\) and \({\textbf {l}}_o = [b_o^1,b_o^2,b_o^3,b_o^4,b_o^5,b_o^6]\). therefore, \(({\textbf {x}}_4 \cup \lnot {\textbf {x}}_4) @ {\textbf {l}}_o = (x_{4,1} \wedge b_o^1) \vee (x_{4,2} \wedge b_o^2) \vee ... \vee (\lnot x_{4,6} \wedge b_o^6) = (1 \wedge b_o^1) \vee (0 \wedge b_o^2) \vee ... \vee (1 \wedge b_o^6) = b_o^1 \vee b_o^5 \vee b_o^6\). the objective of this operation is to generate a disjunction of variables that indicates if any of the features associated with these variables are present in \(m_o\), then sample \({\textbf {x}}_i\) will be correctly classified by \(m_o\). now, we can show the formula that guarantees that if \(\eta _i\) is false, then \({\textbf {m}} ({\textbf {x}}_i) = y_i\): $$\begin{aligned} c = \bigwedge _{\ i\ \in \ \{1, ..., n\}} \lnot \eta _i \rightarrow (y_i \leftrightarrow \bigwedge _{o \in \{1,...,k\}} (({\textbf {x}}_i \cup \lnot {\textbf {x}}_i) @ {\textbf {l}}_o)), w(c) = \infty \end{aligned}$$ (12) we can see that c is not in cnf. therefore, formula q below must be converted to cnf. with that, finally, we have the maxsat query that is sent to the solver. $$\begin{aligned} q = a \wedge b \wedge c \end{aligned}$$ (13) the set of samples \({\textbf {x}}\), in imli, can be divided into p partitions: \({\textbf {x}}^1\), \({\textbf {x}}^2\), ..., \({\textbf {x}}^p\). each partition, but the last one, contains the same values of \(|\mathcal {e}^-|\) and \(|\mathcal {e}^+|\). also, the samples are randomly distributed across the partitions. partitioning aims to make the model perform better in generating the set of rules \({\textbf {m}}\). thus, the conjunction of clauses will be created from each partition \({\textbf {x}}^t\) in an incremental way, that is, the set of rules \({\textbf {m}}\) obtained by the current partition will be reused for the next partition. in the first partition, constraints (10), (11), (12) are created in exactly the same way as described. from the second onwards, (11) is replaced by the following constraints: $$\begin{aligned} b' = \bigwedge _{v \in \{1,...,2m\} \atop o \in \{1,...,k\}} \left\{ \begin{array}{ll} b_o^v \text {, if} \, b_o^v \, \text {is true in the previous partition};\\ \lnot b_o^v \text {, otherwise}. \end{array} \right. , w(b') = 1 \end{aligned}$$ (14) the imli also has a technique for reducing the size of the generated set of rules. the technique removes possible redundancies in ordinal features as the one in example 5. in the original implementation of the model, the technique is applied at the end of each partition. in our implementation for the experiments in sect. 5, this technique is applied only at the end of the last partition. the reason for this is training performance. example 5 let \({\textbf {r}}\) be the following set of rules with redundancy in the same rule: $$(\textit{age}>18 \wedge \textit{age}>20) \vee (\textit{height}\le 2).$$ then, the technique removes the redundancy and the following set of rules is obtained: $$(\textit{age}>20) \vee (\textit{height}\le 2).$$ 4 imlib in this section, we will present our method imlib which is an incremental version of sqfsat based on maxsat. imlib also has a technique for reducing the size of the generated set of rules. therefore, our approach partitions the set of samples \({\textbf {x}}\). moreover, our method has one more constraint and weight on all clauses. with that, our approach receives five input parameters \({\textbf {x}}\), \({\textbf {y}}\), k, l, \(\lambda \) and tries to obtain the smallest \({\textbf {r}}\) that correctly classifies as many samples as possible, penalizing classification errors with \(\lambda \), that is, \(\min _{{\textbf {r}}} \{|{\textbf {r}}| + \lambda |\mathcal {e}_r| \mid \mathcal {e}_r = \{{\textbf {x}}_i \mid {\textbf {r}} ({\textbf {x}}_i) \ne y_i\}\}\). that said, below, we will show the constraints of our approach for constructing maxsat clauses. constraints that guarantee that exactly only one \(u_{o,d}^j\) is true for the dth feature of the rule \(r_o\): $$\begin{aligned} a = \bigwedge _{o \in \{1,...,k\} \atop d \in \{1,...,l\}} \bigvee _{j \in \{1,...,m,*\}} u_{o,d}^j, w(a) = \infty \end{aligned}$$ (15) $$\begin{aligned} b = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop j,j' \in \{1,...,m,*\}, j \ne j'}} \lnot u_{o,d}^j \vee \lnot u_{o,d}^{j'}, w(b) = \infty \end{aligned}$$ (16) constraints representing that the model will try to insert as few features as possible in \({\textbf {r}}\): $$\begin{aligned} c = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop j \in \{1,...,m\}}} \lnot u_{o,d}^j \wedge \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\}}} u_{o,d}^*, w(c) = 1 \end{aligned}$$ (17) conjunction of clauses that guarantees that each rule has at least one feature: $$\begin{aligned} d = \bigwedge _{o \in \{1,...,k\}} \bigvee _{d \in \{1,...,l\}} \lnot u_{o,d}^*, w(d) = \infty \end{aligned}$$ (18) the following conjunction of formulas ensures that \(e_{o,d,i}\) is true if the jth feature label in the oth rule contributes to correctly classify sample \({\textbf {x}}_i\): $$\begin{aligned} e = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop {j \in \{1,...,m\} \atop i \in \{1,...,n\}}}} u_{o,d}^j \rightarrow (p_{o,d} \leftrightarrow s_{o,d,i}^j), w(e) = \infty \end{aligned}$$ (19) conjunction of formulas guaranteeing that the classification of a specific rule will not be interfered by skipped features in the rule: $$\begin{aligned} f = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop i \in \{1,...,n\}}} u_{o,d}^* \rightarrow e_{o,d,i}, w(f) = \infty \end{aligned}$$ (20) conjunction of formulas indicating that the model assigns true to \(z_{o,i}\) if all the features of rule \(r_o\) support the correct classification of sample \({\textbf {x}}_i\): $$\begin{aligned} g = \bigwedge _{o \in \{1,...,k\}} \bigwedge _{i \in \{1,...,n\}} z_{o,i} \leftrightarrow \bigwedge _{d \in \{1,...,l\}} e_{o,d,i}, w(g) = \infty \end{aligned}$$ (21) conjunction of clauses designed to generate a set of rules \({\textbf {r}}\) that correctly classify as many samples as possible: $$\begin{aligned} h = \bigwedge _{i \in \mathcal {e}^+} \bigvee _{o \in \{1,...,k\}} z_{o,i}, w(h) = \lambda \end{aligned}$$ (22) $$\begin{aligned} i = \bigwedge _{i \in \mathcal {e}^-} \bigwedge _{o \in \{1,...,k\}} \lnot z_{o,i}, w(i) = \lambda \end{aligned}$$ (23) finally, after converting formula q below to cnf, we have the maxsat query that is sent to the solver. $$\begin{aligned} q = a \wedge b \wedge c \wedge d \wedge e \wedge f \wedge g \wedge h \wedge i \end{aligned}$$ (24) imlib can also partition the set of samples \({\textbf {x}}\) in the same way imli. therefore, all constraints described above are applied in the first partition. starting from the second partition, the constraints in (17) are replaced by the following constraints: $$\begin{aligned} c' = \bigwedge _{o \in \{1,...,k\} \atop {d \in \{1,...,l\} \atop j \in \{1,...,m,*\}}} \left\{ \begin{array}{ll} u_{o,d}^j \text {, if} \, u_{o,d}^j \, \text {is true in the previous}\\ {partition};\\ \lnot u_{o,d}^j \text {, otherwise}. \end{array} \right. , w(c') = 1 \end{aligned}$$ (25) imlib also has a technique for reducing the size of the generated set of rules demonstrated in example 5. moreover, we added two more cases which are described in example 6 and example 7. example 6 let \({\textbf {r}}\) be the following set of rules with opposite features in the same rule: $$(\textit{age}>20) \vee (\textit{height}\le 2 \wedge \textit{height}>2) \vee (\textit{hike} \wedge \textit{not hike}).$$ therefore, the technique removes rules with opposite features in the same rule obtaining the following set of rules: $$(\textit{age}>20).$$ example 7 let \({\textbf {r}}\) be the following set of rules with the same feature occurring twice in a rule: $$(\textit{hike} \wedge \textit{hike}) \vee (\textit{age}>20).$$ accordingly, our technique for removing redundancies eliminates repetitive features, resulting in the following set of rules: $$(\textit{hike}) \vee (\textit{age}>20).$$ 5 experiments table 1. databases information.full size table in this section, we present the experiments we conducted to compare our method imlib against imli. the two models were implementedfootnote 1 with python and maxsat solver rc2 [10]. the experiments were carried out on a machine with the following configurations: intel(r) core(tm) i5-4460 3.20ghz processor, and 12gb of ram memory. ten databases from the uci repository [3] were used to compare imli with imlib. information on the datasets can be seen in table 1. databases that have more than two classes were adapted, considering that both models are designed for binary classification. for purposes of comparison, we measure the following metrics: number of rules, size of the set of rules, size of the largest rule, accuracy on test data and training time. the number of rules, size of the set of rules, and size of the largest rule can be used as interpretability metrics. for example, a set of rules with few rules and small rules is more interpretable than one with many large rules. each dataset was split into \(80\%\) for training and \(20\%\) for testing. both models were trained and evaluated using the same training and test sets, as well as the same random distribution. then, the way the experiments were conducted ensured that both models had exactly the same set of samples to learn the set of rules. for both imli and imlib, we consider parameter configurations obtained by combining values of: \(k \in \{1,2,3\}\), \(\lambda \in \{5,10\}\) and \(lp \in \{8,16\}\), where lp is the number of samples per partition. since imlib has the maximum number of features per rule l as an extra parameter, for each parameter configuration of imli and its corresponding \(\textbf{r}\), we considered l ranging from 1 to one less than the size of the largest rule in \(\textbf{r}\). thus, the value of l that resulted in the best test accuracy was chosen to be compared with imli. our objective is to evaluate whether imlib can achieve higher test accuracy compared to imli by employing smaller and more balanced rules. furthermore, it should be noted that this does not exclude the possibility of our method generating sets of rules with larger sizes than imli. for each dataset and each parameter configuration of k, \(\lambda \) and lp, we conducted ten independent realizations of this experiment. for each dataset, the parameter configuration with the best average of test accuracy for imli was chosen to be inserted in table 2. for each dataset, the parameter configuration with the best average of test accuracy for imlib was chosen to be inserted in table 3. the results presented in both tables are the average over the ten realizations. in table 2, when considering parameter configurations that favor imli, we can see that imlib stands out in the size of the generated set of rules and in the size of the largest rule in datasets. furthermore, our method achieved equal or higher accuracy compared to imli in four out of ten datasets. in datasets where imli outperformed imlib in terms of accuracy, our method exhibited a modest average performance gap of only three percentage points. besides, imli outperformed our method in terms of training time in all datasets. table 2. comparison between imli and imlib in different databases with the imli configuration that obtained the best result in terms of accuracy. the column training time represents the training time in seconds.full size table table 3. comparison between imli and imlib in different databases with the imlib configuration that obtained the best result in terms of accuracy. the column training time represents the training time in seconds.full size table in table 3, when we consider parameter configurations that favor our method, we can see that imlib continues to stand out in terms of the size of the generated set of rules and the size of the largest rule in all datasets. moreover, our method achieved equal or higher accuracy than imli in all datasets. again, imli consistently demonstrated better training time performance compared to imlib across all datasets. as an illustrative example of interpretability, we present a comparison of the sizes of rules learned by both methods in the mushroom dataset. table 4 shows the sizes of rules obtained in all ten realizations of the experiment. we can observe that imlib consistently maintains a smaller and more balanced set of rules across the different realizations. this is interesting because unbalanced rules can affect interpretability. see realization 6, for instance. the largest rule learned by imli has a size of 10, nearly double the size of the remaining rules. in contrast, imlib learned a set of rules where the size of the largest rule is 6 and the others have similar sizes. thus, interpreting three rules of size at most 6 is easier than interpreting a rule of size 10. also as illustrative examples of interpretability, we can see some sets of rules learned by imlib in table 5. table 4. comparison of the size of the rules generated in the ten realizations of the mushroom base from table 2. the configuration used was \(lp = 16\), \(k = 3\) and \(\lambda = 10\). as the value of l used in imlib varies across the realizations, column l will indicate which was the value used in each realization. in the column rules sizes, we show the size of each rule in the following format: (\(|r_1|\), \(|r_2|\), \(|r_3|\)). we have highlighted in bold the cases where the size of \(|r_o|\) is the same or smaller in our model compared to imli.full size table table 5. examples of set of rules generated by imlib in some tested databases.full size table 6 conclusion in this work, we present a new incremental model for learning interpretable and balanced rules: imlib. our method leverages the strengths of sqfsat, which effectively constrains the size of rules, while incorporating techniques from imli, such as incrementability, cost for classification errors, and minimization of the set of rules. our experiments demonstrate that the proposed approach generates smaller and more balanced rules than imli, while maintaining comparable or even superior accuracy in many cases. we argue that sets of small rules with approximately the same size seem more interpretable when compared to sets with a few large rules. as future work, we plan to develop a version of imlib that can classify sets of samples with more than two classes, enabling us to compare this approach with multiclass interpretable rules from the literature [11, 24]. notes 1.source code of imlib and the implementation of the tests performed can be found at the link: https://github.com/cacajr/decision_set_models. references biran, o., cotton, c.: explanation and justification in machine learning: a survey. in: ijcai-17 workshop on explainable ai (xai), vol. 8, pp. 8–13 (2017) google scholar  carleo, g., et al.: machine learning and the physical sciences. rev. mod. phys. 91(4), 045002 (2019) article  google scholar  dua, d., graff, c.: uci machine learning repository (2017). http://archive.ics.uci.edu/ml ghassemi, m., oakden-rayner, l., beam, a.l.: the false hope of current approaches to explainable artificial intelligence in health care. lancet digit. health 3(11), e745–e750 (2021) article  google scholar  ghosh, b., malioutov, d., meel, k.s.: efficient learning of interpretable classification rules. j. artif. intell. res. 74, 1823–1863 (2022) article  mathscinet  math  google scholar  ghosh, b., meel, k.s.: imli: an incremental framework for maxsat-based learning of interpretable classification rules. in: proceedings of the 2019 aaai/acm conference on ai, ethics, and society, pp. 203–210 (2019) google scholar  gunning, d., stefik, m., choi, j., miller, t., stumpf, s., yang, g.z.: xai-explainable artificial intelligence. sci. robot. 4(37), eaay7120 (2019) google scholar  huang, h.y., et al.: power of data in quantum machine learning. nat. commun. 12(1), 2631 (2021) article  google scholar  ignatiev, a., marques-silva, j., narodytska, n., stuckey, p.j.: reasoning-based learning of interpretable ml models. in: ijcai, pp. 4458–4465 (2021) google scholar  ignatiev, a., morgado, a., marques-silva, j.: rc2: an efficient maxsat solver. j. satisfiability boolean modeling comput. 11(1), 53–64 (2019) article  mathscinet  math  google scholar  ignatiev, a., pereira, f., narodytska, n., marques-silva, j.: a sat-based approach to learn explainable decision sets. in: galmiche, d., schulz, s., sebastiani, r. (eds.) ijcar 2018. lncs (lnai), vol. 10900, pp. 627–645. springer, cham (2018). https://doi.org/10.1007/978-3-319-94205-6_41 chapter  google scholar  janiesch, c., zschech, p., heinrich, k.: machine learning and deep learning. electron. mark. 31(3), 685–695 (2021) article  google scholar  jiménez-luna, j., grisoni, f., schneider, g.: drug discovery with explainable artificial intelligence. nat. mach. intell. 2(10), 573–584 (2020) article  google scholar  kwekha-rashid, a.s., abduljabbar, h.n., alhayani, b.: coronavirus disease (covid-19) cases analysis using machine-learning applications. appl. nanosci. 1–13 (2021) google scholar  lakkaraju, h., bach, s.h., leskovec, j.: interpretable decision sets: a joint framework for description and prediction. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 1675–1684 (2016) google scholar  maliotov, d., meel, k.s.: mlic: a maxsat-based framework for learning interpretable classification rules. in: hooker, j. (ed.) cp 2018. lncs, vol. 11008, pp. 312–327. springer, cham (2018). https://doi.org/10.1007/978-3-319-98334-9_21 chapter  google scholar  mita, g., papotti, p., filippone, m., michiardi, p.: libre: learning interpretable boolean rule ensembles. in: aistats, pp. 245–255. pmlr (2020) google scholar  rocha, t.a., martins, a.t.: synthesis of quantifier-free first-order sentences from noisy samples of strings. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 12–17. ieee (2019) google scholar  rocha, t.a., martins, a.t., ferreira, f.m.: synthesis of a dnf formula from a sample of strings using ehrenfeucht-fraïssé games. theoret. comput. sci. 805, 109–126 (2020) article  mathscinet  math  google scholar  sharma, a., jain, a., gupta, p., chowdary, v.: machine learning applications for precision agriculture: a comprehensive review. ieee access 9, 4843–4873 (2020) article  google scholar  tjoa, e., guan, c.: a survey on explainable artificial intelligence (xai): toward medical xai. ieee trans. neural netw. learn. syst. 32(11), 4793–4813 (2020) article  google scholar  vilone, g., longo, l.: notions of explainability and evaluation approaches for explainable artificial intelligence. inf. fusion 76, 89–106 (2021) article  google scholar  yan, l., et al.: an interpretable mortality prediction model for covid-19 patients. nat. mach. intell. 2(5), 283–288 (2020) article  google scholar  yu, j., ignatiev, a., stuckey, p.j., le bodic, p.: computing optimal decision sets with sat. in: simonis, h. (ed.) cp 2020. lncs, vol. 12333, pp. 952–970. springer, cham (2020). https://doi.org/10.1007/978-3-030-58475-7_55 chapter  google scholar  download references author information authors and affiliations instituto federal de educação, ciência e tecnologia do ceará (ifce), fortaleza, brazil antônio carlos souza ferreira júnior & thiago alves rocha authors antônio carlos souza ferreira júniorview author publications search author on:pubmed google scholar thiago alves rochaview author publications search author on:pubmed google scholar corresponding author correspondence to antônio carlos souza ferreira júnior . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ferreira júnior, a.c.s., rocha, t.a. (2023). an incremental maxsat-based model to learn interpretable and balanced classification rules. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_15 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords interpretable artificial intelligence explainable artificial intelligence rule learning maximum satisfiability publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature enhancing stock market predictions through the integration of convolutional and recursive lstm blocks: a cross-market analysis | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper enhancing stock market predictions through the integration of convolutional and recursive lstm blocks: a cross-market analysis conference paper first online: 12 october 2023 pp 92–106 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) enhancing stock market predictions through the integration of convolutional and recursive lstm blocks: a cross-market analysis download book pdf download book epub filipe ramos  orcid: orcid.org/0000-0002-1013-62659, guilherme silva  orcid: orcid.org/0000-0002-4964-57109, eduardo luz  orcid: orcid.org/0000-0001-5249-15599 & … pedro silva  orcid: orcid.org/0000-0002-5525-61219  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 527 accesses abstract this study explores convolutional and recursive lstm blocks within a singular architecture for forecasting stock prices. we propose a method that integrates convolutional networks, which learn to process signals through filters, with recursive lstm blocks to account for critical temporal information often overlooked in convolutional approaches. our investigation primarily revolves around two research questions: (1) can integrating convolutional and recursive lstm blocks within a singular architecture enhance prediction accuracy? and (2) what is the impact of training and testing with disparate data distributions? the latter question arises from our experiment of training the model using data drawn from the indian stock market and testing the predictions with new york stock market data, thus deviating from the traditional focus on uniform stock market distributions. our results reveal a notable improvement in prediction accuracy (mape reduction of 2.22%), strongly suggesting that pre-processing data via convolutional neural networks (cnn) benefits lstm blocks and can enhance the performance of stock market prediction methodologies. supported by universidade federal de ouro preto, fapemig (apq-01518-21), capes and cnpq (308400/2022-4). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a novel lstm-cnn architecture to forecast stock prices chapter © 2022 stock market forecasting using a novel conv-lstm deep learning model chapter © 2026 analysis and forecasting of financial time series using cnn and lstm-based deep learning models chapter © 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. capital markets financial economics machine learning market intelligence quantitative finance structure prediction 1 introduction a stock market is where companies can raise funds for their business by selling fractional shares. many individuals buy and sell these shares regularly to generate profits, a strategy commonly referred to as day trading. day trading involves purchasing and selling shares on the same day to profit from price differences. traders who use this strategy often rely on statistical models and candlestick patterns to predict future prices, even though the market may be unpredictable [6]. in contrast to traders who use statistical models or candlesticks, some investors perform fundamental analysis. these individuals base their actions on various factors within the company, such as its profit, market value, return on equity, and other indicators. according to vijh et al. [24], the most challenging aspect of predicting stock prices is the influence of external factors, such as the global economy and political conditions. despite this challenge, previous works in this field often employ classical algorithms, such as linear regression, random walk theory [11], moving average convergence/divergence [2], autoregressive moving average [22], and autoregressive integrated moving average [3] to predict stock prices. traditional machine learning techniques, such as support vector machines (svms) [14] and random forests [19], can be used to enhance stock market prediction. additionally, many researchers are exploring artificial neural networks (anns) to predict stock prices and automate trading for passive income. for example, lu et al. [17] and singh and srivastava [21] utilize anns for this purpose. currently, certain types of anns, such as recurrent neural networks (rnns) [8], are showing promising results. these networks analyze current data and use previous data to make a single prediction. in [9], the authors demonstrated that deep learning models, specifically multi-layer perceptron, recursive (rnns), and convolutional networks, can be more effective than traditional statistical methods, like arima, for stock prediction. the authors trained the models on data from a single company listed on india’s national stock exchange (nse) and evaluated the approach on five stocks listed on the new york stock exchange (nyse). many works in the literature explore machine learning using sequential models for forecasting stock prices [8, 9], often incorporating filters such as moving averages to smooth time series data. some studies employ convolutional networks that learn to process signals via filters, yet these networks tend to neglect temporal information. this observation steered us towards a research question: can the integration of convolutional and recursive lstm blocks within a singular architecture bolster the results? simultaneously, this work aims to probe a further research question: what is the impact of training and testing with different data distributions? a majority of the existing studies prioritize predicting stock trends from a more uniform data distribution, often concentrating on the same stock market. to heighten the level of challenge, we ran experiments to predict from different distributions: specifically, training with data drawn from the indian stock market and testing the predictions using data from the new york stock market. our experiments indicate a substantial improvement and strongly suggest that pre-processing data through convolutional neural networks (cnn) benefits lstm blocks for the task in question. this approach underscores the potential efficacy of incorporating convolutional layers in sequence prediction tasks, thereby contributing to the broader conversation around stock market prediction methodologies this manuscript is divided as follows. section 2 shows similar papers written by different researchers and the reported results. section 3 presents the data and models used and how they were built. section 4 addresses the experiments made and compares the results obtained and the literature results hiransha et al. [9]. finally, conclusions are presented in sect. 5. 2 related works in this section, we present several works that use machine learning and deep learning for stock prediction. besides, it is also presented the kind of data used in these works. stock prediction is a challenging and popular research area that, if successful, can yield profitable results. jiang [12] presents several papers on this topic, where most utilize standard data (open, close, high, low, and volume) to train models and compare their performance with real-market results. for example, vijh et al. [24] collected ten years of data on five usa-based companies using yahoo finance and trained a low-complexity ann model, which achieved an rmse of 0.42 and a mape of 0.77% in the best result. in addition to predicting stock prices, models can be built to determine when to buy or sell stocks, as demonstrated by santuci et al. [20], who compared svm and random forest models to predict such events. in the best case, their work yielded a profit of 17.74%. another internal topic for discussion is the accuracy of different models in predicting stock prices, which is demonstrated by huang et al. [10] using two models: feed-forward neural network and adaptive neural fuzzy inference system. pang et al. [18] used two different types of lstm to make predictions based on shanghai a-share data, achieving an mse of 0.017 in the best test. many external factors can influence the stock price, and jin et al. [13] utilized an external indicator called sentiment to indicate how investors feel about the economy or a specific stock. das et al. [4] also used sentiment data extracted from tweets on twitter to predict stock prices. these papers show that the market responds to people’s actions, and using this information can better represent the market than numbers. agrawal et al. [1] utilized stock technical indicators to predict not the stock price but the trend of the price, i.e., whether the price will increase or decrease. their work achieved an accuracy of 63.59% in the best case. despite the works presented in this section using different datasets, the objective is stock price prediction. these works were selected due to the made of traditional machine learning and different neural network architectures applied to the stock price prediction context. 3 proposed methodology s this section shows how the research was driven and the specific models definition. subsection 3.1 describes the data used and how it was collected. subsection 3.2 explains the baseline technique used for comparison purposes and also the proposed architecture, combining deep learning architectures. the metrics used to evaluate the proposed approaches are described in subsect. 3.3. 3.1 dataset the first step is data collection by downloading the stock information from the website infomoney.com. the collected raw data contains: date: the days to which data refer. this data is a string with the month with three letters, followed by the day and year. high: the highest share price on a specific day. this data is a float number. close: the share price when the stock market closes on a specific day. this data is a float number too. open: the share price when the stock market opens on a specific day. this also is a float number. low: the lowest share price on a specific day. and this is a float number. volume: the number of transactions that happen on that day. this data is a integer number, but in the case of a thousand volumes, it is represented with the number divided by 1000 followed by a letter “k”. and if is a million or more the number is divided by one million and followed by the letter “m”. change: the difference in price, in percentage, that happens on a specific day. this data wasn’t used in the model but is presented because it was collected along with others. this data is a time series, so the previous samples directly influence the following sample. the close price rarely moves from a low value to a high value (or vice versa) without the other data that has changed drastically. based on hiransha et al. [9], we chose tata motors ltd. (which uses the token tamo) stock as training data and used axis bank (axbk), maruti (mrti), and hcl (hclt) as testing and the close price as to be predicted. table 1 shows the datasets used in this work and the period collected. the data was collected respecting the date range used in [9] in order to directly compare the results. table 1. data collected of the stocks.full size table the second step consists of preprocessing and cleaning the data due to the nature of the raw data being different from the one requested by a machine learning model. the objective is to delete null data and convert the fields to float numbers. we removed the “date” and “change” columns as they are unnecessary. the “date” is due to its nature which is an index, and the “change” column because it is a linear combination of the other fields. and last, before the training, it is mounted frames with a window of several days for training and testing according to the machine learning model needs. besides, the letters “m” and “k” are also converted to “000000” and “000” respectively. for example, the string “14k” would be converted to 14,000 and “2m” would be converted to 2,000,000. finally, the data needs to be normalized due to the high differences in the features scale. by normalizing the data, we ensure that all data has the same impact on training. the normalization strategy adopted was the min-max, which can be defined as: $$\begin{aligned} x_{norm} = \frac{(x x_{min})}{x_{max} -x_{min}}, \end{aligned}$$ (1) where x is a unit of data, \(x_{max}\) is the largest value in the dataset, \(x_{min}\) is the smallest value in the dataset, and \(x_{norm}\) is the return that will replace x. 3.2 model definition and training two strategies are evaluated, one base on a traditional machine learning strategy and another based on deep learning architectures. the first model is based on the support vector regression (svr) [16]. the svr works like svm (support vector machine), but instead of making a classification based on the hyperplane division, it calculates and returns the distance between the point and the hyperplane as output. in this work, the svr model uses information from a single day to predict the close price of the next day. as in trade operations, recent data is just as important as current ones since the current stock value depends on the previous ones. rnn works well because it uses the current data and previous ones. so the input is a frame with the several days including the one right before the day that will be predicted. one example of rnn is the long short-term memory (lstm) architectures [7]. we made a sequence of empirical tests with a different number of layers to compare different lstm architectures and try to find the best fit for the problem treated. we fix the number of 75 cells in each layer, a dropout of 20% (avoid overfitting [23]), and the last layer as a dense layer with one node for the regression. the lstm model learns the time series. towards to extract more features from the time series, a convolutional neural network (cnn) [15] is applied. so, instead of the lstm learning directly from the raw data, it could learn from more complex features learned by convolution layers of a cnn. this strategy was observed in other works that use a 1d signals [5]. the cnn-lstm model was constructed by using the best-performing lstm model, adding a one-dimensional convolutional layer with 32 filters, relu activation, followed by a max-pooling layer of size 4. the values for the cnn and max-pooling layer were determined empirically. 3.3 evaluation metrics to evaluate the effectiveness of the proposed approaches, we used three different evaluation metrics, all available on the scikit-learn library. the first one is the root mean square error (rmse), which is the mean error squared to turn positive the metric, and the square root to move back to the original proportion. besides, it also focuses on emphasizing large outlier errors. the metric is defined as $$\begin{aligned} rmse = \sqrt{\frac{\sum {(y_i y_p)^2}}{n}}, \end{aligned}$$ (2) where \(y_i\) represents the right value, \(y_p\) is the value predicted by the model, and n is the number of predictions made in tests. the second is mean absolute error (mae), which is the module of the mean error and it is represented by the formula $$\begin{aligned} mae = \frac{\sum |(y_i y_p)|}{n}, \end{aligned}$$ (3) in which \(y_i\), \(y_p\) and n are the same presented for rmse metric. the last metric used was the mean average percentage error (mape), which is represented by the formula $$\begin{aligned} mape = \frac{\sum {|y_i y_p|/|y_p|}}{n} \end{aligned}$$ (4) in which \(y_i\), \(y_p\) and n are the same presented for rmse and mae metrics. 4 experiments and results this section is about showing, explaining, and discussing the results. all the training and testing were made in a computer with 128 gb ram, titan x 12 gb vram, and intel(r) core(tm) i7-5820k cpu @ 3.30 ghz. the framework used was tensorflow 2 and scikit-learn 1.1 for python3. we aim to answer the two researches questions of this work: (1) can integrating convolutional and recursive lstm blocks within a singular architecture enhance prediction accuracy? and (2) what is the impact of training and testing with disparate data distributions? to correctly answer the second question, first we propose baseline results using the svr method. to answer the first question, first we evaluate a proper lstm architecture and then, add cnn layer. all the models were trained for 10,000 epochs. the svr model has an rbf kernel and a regularization parameter of 1. the lstm and cnn-lstm models were trained with a batch size of 2,048, adam optimizer, and mean squared error as a loss. the models were trained with the tamo dataset and tested using the hclt, mrti, and axbk datasets, following the pattern of the work carried out by hiransha et al. [9]. 4.1 svr model the first model evaluated was the svr. in all three datasets tested, there is a gap between the predicted close price and the real one. this can be seen in figs. 1, 2 and 3 and the error measured in table 2. fig. 1. predictions made by svr model for the axis bank dataset. time in days. full size image fig. 2. predictions made by svr model for the hcl techonologies dataset. time in days. full size image table 2. metrics of result in svr modelfull size table it is possible to observe that the predicted close prices from svr have a smoother line compared to the real one. as the svr only uses the data from today to predict tomorrow, the information from the previous days are not used, and the model cannot follow the real close price. the greatest gap is observed in the hcl technologies dataset, which has a bigger variance as the range of values is greater. fig. 3. predictions made by svr model for the maruti dataset. time in days. full size image 4.2 lstm model we conduct an empirical study to define the best lstm architecture to reach the highest mape in the tamo dataset. different models were evaluated by varying the size of the layer. after that, the model with the best results in the tamo data was chosen. as shown in table 3, the model with the slowest result has six layers. all these layers were intercalated with dropout layers of 20% to minimize the chance of overfitting. the best lstm architecture can be seen in fig. 4. fig. 4. lstm architecture with six layers with 75 unities in each one. full size image as seen in figs. 5, 6 and 7, the lstm model could predict the closest stocks value compared to the svr one. this scenario can be seen in table 4, where the reported error are below the ones reported in table 2. mape below three shows that the distance between the predicted close price and the real one is below 3. fig. 5. predictions made by lstm model for the axis bank dataset. time in days. full size image fig. 6. predictions made by lstm model for the hcl technologies dataset. time in days. full size image table 3. reported metrics of an empirical study with lstm models. the model with smaller mape, rmse, and mae is with six layers.full size table fig. 7. predictions made by lstm model for the maruti dataset. time in days. full size image table 4. metrics of result in lstm modelfull size table compared to the svr models, the lstm models predict the closed price near the real one. one of the reasons is the number of days used for each model: while the svr uses only one day, the lstm uses 12  days. similar to the svr model, hcl technologies also presented the highest gap. the lstm model could follow the real close price curve when the variance is small, although, when it is high, the model could not follow the real curve and presented a smoother behavior. 4.3 cnn-lstm model the last model build was with a combination of cnn and lstm. this model has two 1d convolution layers and was defined with empirical tests, stacking the convolutional layers before the lstm with six layers and dropout defined in the experiments presented in table 3. the cnn-lstm architecture is seen in fig. 8. fig. 8. cnn-lstm architecture with a convolutional layer with 32 filters, kernel size equal to 8 and relu activation followed by six lstm layers with 75 unities in each one. full size image as can be seen in figs. 9, 10 and 11, the line that represents the prediction almost can not be seen, which means that the predicted stock price is very close to the actual value. some parts of the graphic have a peak, but this does not affect the result in general. the errors can be seen in table 5. table 5. metrics of result in cnn-lstm modelfull size table as tables 4 and 2 show, a smaller mape is reported compared to the one reported by hiransha et al. [9]. while hiransha et al. [9] reached the mape of 4.88, the proposed cnn-lstm model reached 1.83. so, as can be seen in fig. 12, on a large scale, all the models follow the trend of the stock market, but, in a zoomed view of the predictions, the models are not accurate. we improved the curve accuracy by adding cnn in the lstm model, which forced the curve to stay softer, following the close price. therefore, the model with cnn succeeds in extracting features from the data and improving the results. fig. 9. predictions made by cnn-lstm model in axis bank dataset. time in days. full size image fig. 10. predictions made by cnn-lstm model in hcl technologies dataset. time in days. full size image 4.4 results discussion as seen in table 6, the proposed approach overcomes the baseline model proposed by hiransha et al. [9]. the results show that it will be possible to build a model that can precisely predict the prices in a near future. but, despite the reported results, it is important to observe that the market floats with external factors, which makes the model use quite risky. fig. 11. predictions made by cnn-lstm model in maruti dataset. time in days. full size image fig. 12. prediction made by all models with a zoomed view of the last 100 days predicted in mrti dataset. full size image table 6. mape of all models trained. axbk = axis bank; hclt = hcl; mrti = maruti.full size table as table 6 shows, the cnn-lstm model has better results. the intuition of getting smaller errors maybe is the information of more days used by the cnn-lstm and the data processing made by the convolution layer. gathering more data with different prices and fluctuations can result in a more general model. comparing the much higher error of the svr to that of the lstm and cnn-lstm, it is visible how the use of only one day can be detrimental to the algorithm’s performance. as only one day is used, long-term behavior is not taken into account, which tends to generate results that are inconsistent with those expected. 5 conclusion and future works in this work, we analyzed how models work in long and short periods to predict close stock prices, a challenging task. three different architectures were evaluated: a traditional machine learning algorithm (svr) and two deep learning algorithms (lstm and cnn). as can be seen, adding a cnn layer to the lstm model increased the performance by 0.75% in the best case. this finding aligns with the primary research question addressed in this study. the results also address the second research question, once the models could generalize to a dataset different from the one a model was trained on. as the input is a time series, the lstm model overcomes the svr one. another analysis is that the convolutional layers can extract features in tabular data and increase the results when added to the lstm model. in future work, two research paths to reduce the error will be explored: (1) an ensemble model, which combines some models responses, like cnn, rnn, lstm, and transformers, and (2) gathering external information about the companies to predict the close price, such as the response of sentiment analysis about news regarding the companies and other negative or positive aspects of the company. references agrawal, m., shukla, p.k., nair, r., nayyar, a., masud, m.: stock prediction based on technical indicators using deep learning model. comput. mater. continua 70(1), 287–304 (2022) article  google scholar  aguirre, a.a.a., méndez, n.d.d., medina, r.a.r.: artificial intelligence applied to investment in variable income through the macd (moving average convergence/divergence) indicator. j. econ. financ. adm. sci. (2021) google scholar  an, z., feng, z.: a stock price forecasting method using autoregressive integrated moving average model and gated recurrent unit network. in: 2021 international conference on big data analysis and computer science (bdacs), pp. 31–34. ieee (2021) google scholar  das, s., behera, r.k., rath, s.k., et al.: real-time sentiment analysis of twitter streaming data for stock prediction. procedia comput. sci. 132, 956–964 (2018) article  google scholar  freitas, c., silva, p., moreira, g., luz, e.: rede neural convolucional e lstm para biometria baseada em eeg no modo de identificaçao. in: anais do xxii simpósio brasileiro de computação aplicada à saúde, pp. 256–267. sbc (2022) google scholar  gomes, i.d.o.: estratégias para operações de day trade na b3. master’s thesis, escola de economia de são paulo da fundação getulio vargas (2018) google scholar  goodfellow, i., bengio, y., courville, a.: deep learning. mit press, cambridge (2016) google scholar  hansun, s., young, j.c.: predicting lq45 financial sector indices using rnn-lstm. j. big data 8(1), 1–13 (2021) article  google scholar  hiransha, m., gopalakrishnan, e.a., menon, v.k., soman, k.: nse stock market prediction using deep-learning models. procedia comput. sci. 132, 1351–1362 (2018) article  google scholar  huang, y., capretz, l.f., ho, d.: neural network models for stock selection based on fundamental analysis. in: 2019 ieee canadian conference of electrical and computer engineering (ccece), pp. 1–4. ieee (2019) google scholar  jayadev, s., anupama, r., sebastian, s.: testing the market efficiency of nse: a study on random walk theory. contemp. res. financ. 34 (2021) google scholar  jiang, w.: applications of deep learning in stock market prediction: recent progress. expert syst. appl. 184, 115537 (2021) article  google scholar  jin, z., yang, y., liu, y.: stock closing price prediction based on sentiment analysis and lstm. neural comput. appl. 32(13), 9713–9729 (2020) article  google scholar  kurani, a., doshi, p., vakharia, a., shah, m.: a comprehensive comparative study of artificial neural network (ann) and support vector machines (svm) on stock forecasting. ann. data sci. 1–26 (2021) google scholar  lecun, y., bengio, y., et al.: convolutional networks for images, speech, and time series. handb. brain theory neural netw. 3361(10), 1995 (1995) google scholar  lorena, a.c., de carvalho, a.c.: uma introdução às support vector machines. revista de informática teórica e aplicada 14(2), 43–67 (2007) article  google scholar  lu, w., li, j., wang, j., qin, l.: a cnn-bilstm-am method for stock price prediction. neural comput. appl. 33(10), 4741–4753 (2021) article  google scholar  pang, x., zhou, y., wang, p., lin, w., chang, v.: an innovative neural network approach for stock market prediction. j. supercomput. 76(3), 2098–2118 (2020) article  google scholar  rakhra, m., et al.: crop price prediction using random forest and decision tree regression:-a review. mater. today proc. (2021) google scholar  santuci, a., sbruzzi, e., araújo-filho, l., leles, m.: evaluation of forex trading strategies based in random forest and support vector machines. ieee lat. am. trans. 20(9), 2146–2152 (2022) article  google scholar  singh, r., srivastava, s.: stock prediction using deep learning. multimedia tools appl. 76(18), 18569–18584 (2017) article  google scholar  singh, s., parmar, k.s., kumar, j.: soft computing model coupled with statistical models to estimate future of stock market. neural comput. appl. 33(13), 7629–7647 (2021) article  google scholar  srivastava, n., hinton, g., krizhevsky, a., sutskever, i., salakhutdinov, r.: dropout: a simple way to prevent neural networks from overfitting. j. mach. learn. res. 15(1), 1929–1958 (2014) mathscinet  math  google scholar  vijh, m., chandola, d., tikkiwal, v.a., kumar, a.: stock closing price prediction using machine learning techniques. procedia comput. sci. 167, 599–606 (2020) article  google scholar  download references author information authors and affiliations computing department, federal university of ouro preto, ouro preto, brazil filipe ramos, guilherme silva, eduardo luz & pedro silva authors filipe ramosview author publications search author on:pubmed google scholar guilherme silvaview author publications search author on:pubmed google scholar eduardo luzview author publications search author on:pubmed google scholar pedro silvaview author publications search author on:pubmed google scholar corresponding author correspondence to filipe ramos . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ramos, f., silva, g., luz, e., silva, p. (2023). enhancing stock market predictions through the integration of convolutional and recursive lstm blocks: a cross-market analysis. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_7 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords stock prince machine learning cnn lstm publish with us policies and ethics profiles eduardo luz view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature dog face recognition using vision transformer | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper dog face recognition using vision transformer conference paper first online: 12 october 2023 pp 33–47 cite this conference paper download book pdf download book epub intelligent systems (bracis 2023) dog face recognition using vision transformer download book pdf download book epub victor hugo braguim canto  orcid: orcid.org/0000-0002-0511-44759, joão renato ribeiro manesco  orcid: orcid.org/0000-0002-1617-51429, gustavo botelho de souza  orcid: orcid.org/0000-0002-4441-910810 & … aparecido nilceu marana  orcid: orcid.org/0000-0003-4861-70619  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 597 accesses abstract the demand for effective, efficient and safe methods for animal identification has been increasing significantly, due to the need for traceability, management, and control of this population, which grows at higher rates than the human population, particularly pets. motivated by the efficacy of modern human identification methods based on face biometrics features, in this paper, we propose a dog face recognition method based on vision transformers, a deep learning approach that decomposes the input image into a sequence of patches and applies self-attention to these patches to capture spatial relationships between them. results obtained on dogfacenet, a public database of dog face images, show that the proposed method, which uses the efficientformer-l1 architecture, outperforms the state-of-the-art method proposed previously in literature based on resnet, a deep convolutional neural network. download conference paper pdf similar content being viewed by others dogface detection and localization of dogface’s landmarks chapter © 2019 a deep learning approach for dog face verification and recognition chapter © 2019 deep learning techniques for dogs image dataset analysis chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition biometrics computer vision object recognition pattern vision computer imaging, vision, pattern recognition and graphics 1 introduction according to pet brazil institute [9], in 2018, there were 54.2 million dogs, 39.8 million birds, 23.9 million cats, 19.1 million fishes, and 2.3 million reptiles and small mammals, considered pets, in the homes of brazilian families. compared to the 2013 census data, the cat and dog populations have grown in households at 8.1% and 5%, respectively, with an overall average growth of 5.2%. in addition, data from the world health organization (who), published by são paulo university (usp) [14], indicate that in brazil there are approximately 30 million lost or abandoned animals, with 10 million cats and 20 million dogs. the uncontrolled proliferation and the inappropriate management of animals in society can be factors with a very negative impact on public health, as animals can transmit many diseases and parasites. therefore, it is of paramount importance to carry out monitoring, control and also traceability of animals through complete medical records and effective identification mechanisms to enable monitoring and evaluation of the animal population in the country. some countries have already approved mandatory pet identification registration, such as south korea, where all dogs must be monitored through rfid (radio frequency identification), tags on collars or other devices [10]. as occurs in human identification, animal identification is also susceptible to fraud. in the uk, for example, pet insurance fraud cases have been increasing by 400% per year, reaching 2 million, annually, in claims, according to youtalk-insurance [18]. therefore, in addition to issues of effectiveness and efficiency, animal identification methods also need to be robust enough to discourage, stop or even prevent fraud. according to kumar et al. [11], animal identification approaches can be classified into invasive or non-invasive. examples of invasive approaches are ear tags, ear tattoos and microchip implantation in the animal’s body. examples of non-invasive approaches are rfid (radio frequency identification) devices, collars with gps (global positioning system), internet of things (iot) devices, bluetooth devices, and biometric features (e.g. face, iris, retina and muzzles). among these approaches, animal identification based on biometric features has proved to be the best choice, since it is effective, efficient and does not require special devices, which would introduce more costs to the identification process, being also important to discourage or prevent fraud. among several human biometric traits, the face is one of the most popular due to its high coverage, high acceptability, easiness of capture at a distance, convenience of use and lower costs since digital cameras are cheap and popular sensors. motivated by the success of modern human face recognition methods, in this paper, we propose a dog face recognition method based on vision transformers, a deep learning approach that decomposes the input image into a sequence of patches and applies self-attention to these patches to capture spatial relationships between them. 2 biometric dog identification biometric dog identification approaches have been reported in the literature by using snout and facial features, among others [10, 12, 17, 21]. this section presents some approaches proposed so far, their identification rates, as well as their advantages and disadvantages. 2.1 identification using snout features snout is the region of an animal’s face consisting of its nose, mouth, and jaw. in many animals, this structure is called muzzle. the muzzle information can be used as a biometric feature for dog identification, as reported by jang et al. [10]. figure 1 shows, on the left, a device used to capture images of dog snouts and, on the right, a captured image, with the region of interest for biometric identification highlighted. fig. 1. device used to collect images of dog snouts (on the left) and the region of interest of a collected image, used for feature extraction (on the right) [10]. full size image jang et al. [10] have proposed the use of some feature extraction methods, such as sift (scale-invariant feature transform), surf (speeded up robust features), brisk (binary robust invariant scalable keypoints) and orb (oriented fast and rotated brief), and the method that presented the best performance was the orb with eer (equal error rate) of 0.35%, when evaluated with 55 dog muzzle pattern images acquired from 11 dogs and 990 images augmented by the image deformation (i.e., angle, illumination, noise, affine transform). however, despite this low error rate, the technique proposed by these authors has some disadvantages: even though it is not an invasive technique, it is necessary to immobilize the animal to capture the image and extract features from the snout; it is important to avoid light reflection and blur in muzzle images; it is necessary to have good hygiene of the muzzle. figure 2 shows examples of dog muzzle images that were discarded in the work by jang et al. [10] due to light reflection and blurring caused by dog movements. fig. 2. dog muzzle images with light reflection (1–2) and blurring (3–4) [10]. full size image 2.2 identification using facial features recent works on dog biometric identification based on face features have used the dogfacenet database, proposed by mougeot, li and jia [17]. this database contains 3,148 images of 485 dogs, with at least two images per individual. however, the publicly available database that was developed after the published work contains 8,363 images of 1,393 dogs. figure 3 shows four images of four dogs captured from different viewpoints that compose the dogfacenet dataset. fig. 3. samples of four dog images from the dogfacenet dataset [17]. full size image the images of the dog’s faces were all aligned, using the animal’s eyes as a reference. figure 4 shows the process of detecting the fiducial points (eyes and snout) on the face of a dog (on the left), used for the alignment of the images, and the result of the alignment and cropping processes applied on the six images of one dog from the dogfacenet dataset (on the right). the dog face images of the dogfacenet database were divided into training and test sets, with 90% and 10%, respectively. fig. 4. detection of the animal’s eyes and snout (left) and the results obtained after the aligning and cropping processes, applied to six images of one dog from the dogfacenet dataset [17]. full size image dog identification through facial features is a challenging task, given the high interclass similarity and the high intraclass variability that can occur. figure 5 shows, on the left, six pairs of images captured from distinct pairs of individuals that present faces with high similarity (false positive pairs of canine image faces according to the method proposed by mougeot, li and jia [17]), and, on the right, six pairs of images, each pair from the same individual, that present high variability (false negative pairs of canine image faces according to the method proposed by mougeot, li and jia [17]). fig. 5. examples of six false positive pairs of canine faces (left) and six false negative pairs of canine faces (right), according to the method proposed by mougeot, li and jia [17]. full size image mougeot, li and jia [17] proposed a deep vgg-like and a resnet-like convolutional neural network (cnn) for the dog face verification task containing 48 dogs from an open-set of test data of dogfacenet, and concluded that the resnet-like cnn performed better with 92% of accuracy. for the identification task, the best performance was also obtained by the resnet-like cnn, with 60.44% of accuracy for rank-1 and 92% for rank-5. yoon, so and rhee [21], based on the studies of mougeot, li and jia [17], proposed a methodology for using vector spaces to improve the performance in the dog identification process. thus, in the scenario of dog facial identification, the authors proposed, using the triplet loss, the removal of the l2 norm, for comparing feature vectors from the images of the dogs in a new space within a sphere of radius 1. with this, they obtained a mean accuracy of 88.8% with arcface+vl in the verification task. 3 background in this section, we describe the fundamental concepts used in this work, such as convolutional neural network (cnn), resnet architecture, vision transformers (vit), efficientformer-l1 architecture and arcface function. 3.1 convolutional neural networks and the resnet architecture convolutional neural networks (cnn) have been widely used in solutions for computer vision applications, in problems such as object detection, image recognition, and biometrics [1, 2]. their architectures are similar to the ones of the human visual cortex. according to chollet [1], there are some high-performance cnn architectures, such as alexnet, vgg, googlenet and resnet. convolutional neural networks are organized into layers, such as the convolutional layers, the pooling layers and the fully connected layer. in fig. 6 there is an example of the architecture of a cnn. fig. 6. simplified architecture of a convolutional neural network [13]. full size image in cnns, to limit the range of a result of a convolution layer, activation functions are used, such as the linear function relu, the sigmoid function, the hyperbolic tangent function, etc. thus, according to lecun et al. [6], the convolutional layers have the objective of performing the extraction of features through the convolution operation. furthermore, the adjustments of the backpropagation algorithm weights are performed in this layer. also according to lecun et al. [6], the pooling layer aims to reduce dimensionality by combining neighbouring pixels into a single value. consequently, there is a reduction in time and a minimization of the use of computational resources for training. the fully connected layer aims to perform image classification. for example, in a neural network for classifying dogs, the result of the fully connected layer expresses which dog is associated with the input image. in addition to conventional cnn architectures, resnet is a cnn architecture proposed by he et al. [8] and its main characteristic, as its own name indicates residual networks is the use of shortcut connections, i.e., residual information is passed from previous layers to posterior ones, which improves and, consequently, change the training process (avoiding data vanishing or attenuation as well as saving computational resources, for instance). in fig. 7, there is an example of using shortcut connections and routing to the output layer of the neural network in order to optimize its training process. fig. 7. resnet features [7]. full size image according to li et al. [15], several improvements were made to the resnet architecture to increase its performance. among them, zhang et al. [22] proposed multi-level shortcut connections and targ, almeida and lyman [19] proposed using more convolution layers and data flow between layers, among others. also, different benchmark architectures of resnet neural networks were proposed, however, one of the most used and effective for many computer vision tasks is the resnet-50 (with 50 layers). 3.2 vision transformers and the efficientformer architecture transformers were proposed by vaswani et al. [20] to solve a problem involving natural language processing (nlp) and have become state-of-the-art for many types of applications. for image processing applications, dosovitskiy et al. [5] proposed the vision transformers (vit) models, in which the input image is divided into patches of fixed size, transforming it into an input sequence of image patches. continuing through the initial patches, feature vectors are projected across an initial linear layer and position references are added directly to a standard transformer. in fig. 8 there is an example of the architecture of a vision transformer and its operations. according to [5], in the benchmarks performed, the vision transformers architecture outperformed state-of-the-art image recognition architectures, specifically with the use of convolutional neural networks. for example, in the imagenet database, the vision transformer architecture had an accuracy of 88.5%, and the others, using cnns, were below 88%. fig. 8. vision transformer architecture proposed by dosovitskiy [5]. full size image even with the great progress caused by vits in the computer vision area, models based on transformers are still very heavy and present a problem of high latency, which hinders both the learning and the use of this type of model in less capable environments. efficientformer-type networks arise to mitigate this problem through a pure transformer, with consistent data dimensionality, capable of running in more diverse environments. figure 9 shows an overview of the efficentformer proposed by lie et al. [16]. the network starts with a convolution stem as patch embedding, followed by metablock (mb). the \(mb^{4d}\) and \(mb^{3d}\) contain different token mixer configurations, that is, local pooling or global multi-head self-attention, arranged in a dimension-consistent manner. fig. 9. overview of the efficientformer  [16]. full size image 3.3 arcface in the arcface [4] error function, the objective of learning becomes jointly minimizing the angular distance between elements of the same class while maximizing the intra-class similarity through eq. 1. $$\begin{aligned} l = -\frac{1}{n} \sum \limits _{i=1}^{n} log \frac{e^{s(\cos ({\theta _{y_i}} + m))}}{e^{s(\cos ({\theta _{y_i}} + m))} +\sum _{j=1,j\ne y_i}^n e^{s \cos (\theta _j)}} \end{aligned}$$ (1) in this case, the class samples are positioned in a hypersphere of radius s, and the objective of the error function becomes to minimize the angular distance between elements of this hypersphere, in such a way that the features extracted in the last layer before the softmax, represent a sample in this space, whose elements similar to this sample always end up being positioned at an angular distance in the interval m. the error function can be extended to perform an analysis on subspaces by adding an additional parameter of subcenter when extracting features, this parameter consists of a matrix of weights of dimensions \(c \times k\), where c is the number of classes and k the number of subspaces, such that the features are mapped to the k subspaces, so that the arcface error function can be applied to each of them [3]. 4 proposed method in order to improve the state-of-the-art results in the task of dog facial recognition, in this work we propose a method that uses efficientformer-l1, a vision transformer architecture proposed by lie et al. [16]. in the proposed method, the efficientformer-l1 is coupled with the arcface error function [4], a popular loss function in biometrics which aims to minimize the interclass angular distance while the intraclass angular distance is maximized. in addition, an svm classifier will be used in the output of both architectures, since in biometrics calculation is used using distances, such as the cosine distance. figure 10 demonstrates the architecture of the proposed method with the architecture of efficientformer-l1. fig. 10. diagram of the proposed method, describing both the transformer step and the arcface technique used to position the face descriptors from the same dog near each other and the face descriptors from distinct dogs far away, in the latent space. full size image 5 results and discussion in order to assess the proposed method for dog face recognition, using visual transformers, more specifically by using the efficientformer-l1 architecture, and arcface, two experiments were carried out on the dogfacenet dataset. as the subset of data used by mougeot, li and jia [17] is different than the one publicly shared, a protocol, inspired by them and using standard practices for open-set biometric authentication was used. the results obtained were compared with a baseline method based on the resnet-50 architecture [8] proposed by mougeot, li and jia [17]. it is important to point out that the experiments are not identical to those carried out by mougeot, li and jia [17], as the publicly disclosed base is different from the original base. in addiction, in table 1 there are details of the servers that were used to run the experiments. being the implementation of the resnet architecture on server 1 and the efficientformer-l1 architecture on server 2. table 1. server settingsfull size table 5.1 experiment verification task in the verification task, two architectures, efficientformer-l1 and resnet-50, were employed in a standard facial biometrics evaluation protocol. this protocol involves using all the comparisons within the dataset in order to obtain the genuine and impostor score distributions and, then, construct the receiver operating characteristic (roc) curve. furthermore, to ensure balanced evaluation, we selected all positive comparisons (a total of 2561 pairs of dogs) and an equal number of negative comparisons. from these pairs, we computed additional evaluation metrics, including accuracy, precision, recall, and f-1 score. initially, an experiment was carried out in relation to the verification task, an efficientformer-l1 architecture was used for feature extraction combined with the arcface error function with 3 subcenters to perform the learning. the hyperparameters in relation to the arcface error function were obtained using as reference the analysis proposed by the fixed adacos method, proposed by [23], in which the margin parameter is fixed at \(m = 0.5\) and the scale parameter is obtained dynamically by \(s = \sqrt{2}\cdot \log (c-1)\), where c is the number of classes. figure 11 shows the genuine comparison score distributions and the imposter comparison score distributions obtained using resnet-50 and efficientformer-l1 features. one can observe that the efficientformer-l1 features generated more separated distributions than the resnet-50 features, leading to a lower intersection between the distributions and, consequently, lower error rates. efficientformer-l1 obtained a better auc value, equal to 0.989319, while the resnet-50 obtained an auc value equal to 0.952105, as shown in fig. 12. fig. 11. distributions of similarity scores obtained through features extracted using resnet-50 and efficientformer-l1. full size image fig. 12. roc curves obtained on dogfacenet dataset by using features extracted from the resnet-50 and efficientformer-l1. full size image table 2 presents the values of area under the roc curve (auc), equal error rate (eer), accuracy, f1-score, precision and recall, obtained in the verification task, on the dogfacenet dataset, using face features extracted from the efficientformer-l1 and resnet-50. one can observe that efficientformer-l1 features performed better than the resnet-50 features in all metrics. table 2. results obtained in the verification task, on dogfacenet dataset, using face features extracted from the efficientformer-l1 and resnet-50.full size table 5.2 experiment identification task in the identification task experiment, the two architectures, efficientformer-l1 and resnet-50, were used again. to evaluate their performance, we employed two distinct settings with two different types of data split. in the first setting, we computed the cumulative matching characteristic (cmc) curve by comparing the cosine distance between every element in the probe set and every element in the gallery set. this analysis provides insights into the model’s ability to rank the correct matches in the gallery. the data was split by using a k-fold cross-validation strategy, with k set to 10, where 90% of the open-set data was used for gallery creation, and the remaining 10% was reserved for evaluation. to further evaluate the models, we trained an svm using the k-fold cross-validations with k set to 10. table 3 shows the results obtained. table 3. accuracy results in the identification task using face features extracted from efficientformer-l1 and resnet-50 and an svm classifier in the k-fold cross-validation protocol.full size table for the second split, a random sample approach was employed, as proposed by mougeot, li and jia [17], where m samples for each class are randomly selected from the dataset and included in the gallery set, while the rest of the samples are used for evaluation. for this data split approach, a k-nn classifier was used, with the k value being chosen also as proposed by mougeot, li and jia [17]. figure 3 presents the rank-10 cumulative match characteristic (cmc) curve in each of the data separation protocols for the identification task. one can observe that the efficientformer-l1 features obtained very superior performances when compared with the resnet-50 features, in the identification task. the best rank-1 accuracy value, \(91.4\%\), was obtained by efficientformer, with \(m=1\), while the best rank-1 accuracy value obtained by resnet-50 was only \(63.3\%\), also with \(m=1\). regarding the related works presented in sect. 2.2, using the smaller initial version of the dogfacenet, mougeot, li and jia [17], obtained \(92\%\) of accuracy in the verification task, and \(60.44\%\) of rank-1 accuracy (best case) in the identification task, while yoon, so and rhee [21], using the same dataset that we have used, obtained a mean accuracy of \(88.8\%\) with arcface+vl in the verification task (fig.13). fig. 13. cmc curves obtained on dogfacenet dataset by using face features extracted from efficientformer-l1 and resnet-50. full size image 6 conclusion with the advancement of technology, especially in the scenario of smart cities and internet of things devices, there is an improvement and, consequently, sophistication in biometric applications. with this, it is possible to explore the field of animal biometrics, as there are still few applications and studies in the area applied to dogs, according to the study by mougeot, li and jia [17]. in the case of dogs specifically, biometric identification through faces has shown to be very promising, as it does not require specific hardware resources and can be used through smartphone cameras. as for the recognition techniques, it is noted that the application of architectures based on transformers, especially with efficientformer-l1 used in this work, brings excellent results compared to the state of the art in the area of pattern recognition in images that are the convolutional neural networks, in this case from resnet-50. still, the application of biometric techniques in dogs is a vast area to be explored and its results add positively to the population, because, through this, it provides ways of monitoring these animals in cities, searching for lost animals, reducing abandonment, carrying out disease control more effectively and help prevent or detect fraud. references chollet, f.: how convolutional neural networks see the world. the keras blog 30 (2016) google scholar  de souza, g.b., da silva santos, d.f., pires, r.g., marana, a.n., papa, j.p.: deep texture features for robust face spoofing detection. ieee trans. circuits syst. ii express briefs 64(12), 1397–1401 (2017) google scholar  deng, j., guo, j., liu, t., gong, m., zafeiriou, s.: sub-center arcface: boosting face recognition by large-scale noisy web faces. in: vedaldi, a., bischof, h., brox, t., frahm, j.-m. (eds.) eccv 2020. lncs, vol. 12356, pp. 741–757. springer, cham (2020). https://doi.org/10.1007/978-3-030-58621-8_43 chapter  google scholar  deng, j., guo, j., xue, n., zafeiriou, s.: arcface: additive angular margin loss for deep face recognition. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 4690–4699 (2019) google scholar  dosovitskiy, a., et al.: an image is worth 16x16 words: transformers for image recognition at scale. arxiv preprint arxiv:2010.11929 (2020) forsyth, d.a., et al.: object recognition with gradient-based learning. shape, contour and grouping in computer vision, pp. 319–345 (1999) google scholar  geeksforgeeks: residual networks (resnet) deep learning. https://www.geeksforgeeks.org/residual-networks-resnet-deep-learning/. accessed 18 june 2022 he, k., zhang, x., ren, s., sun, j.: identity mappings in deep residual networks. in: leibe, b., matas, j., sebe, n., welling, m. (eds.) eccv 2016. lncs, vol. 9908, pp. 630–645. springer, cham (2016). https://doi.org/10.1007/978-3-319-46493-0_38 chapter  google scholar  institute, p.b.: pet census (2019). https://institutopetbrasil.com/imprensa/censo-pet-1393-milhoes-de-animais-de-estimacao-no-brasil. accessed 18 june 2022 jang, d.h., kwon, k.s., kim, j.k., yang, k.y., kim, j.b.: dog identification method based on muzzle pattern image. appl. sci. 10(24), 8994 (2020) article  google scholar  kumar, s., singh, s.k.: visual animal biometrics: survey. iet. biometrics 6(3), 139–156 (2017) google scholar  lai, k., tu, x., yanushkevich, s.: dog identification using soft biometrics and neural networks. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2019) google scholar  lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  google scholar  lemos, s.: the number of adoptions and abandonment of animals in the pandemic (2021). https://jornal.usp.br/atualidades/cresce-o-numero-de-adocoes-e-de-abandono-de-animais-na-pandemia. accessed 18 june 2022 li, s., jiao, j., han, y., weissman, t.: demystifying resnet. arxiv preprint arxiv:1611.01186 (2016) li, y., yuan, g., wen, y., hu, j., evangelidis, g., tulyakov, s., wang, y., ren, j.: efficientformer: vision transformers at mobilenet speed. adv. neural. inf. process. syst. 35, 12934–12949 (2022) google scholar  mougeot, g., li, d., jia, s.: a deep learning approach for dog face verification and recognition. in: nayak, a.c., sharma, a. (eds.) pricai 2019. lncs (lnai), vol. 11672, pp. 418–430. springer, cham (2019). https://doi.org/10.1007/978-3-030-29894-4_34 chapter  google scholar  software, a.: pet insurance fraud increases (2018). https://youtalk-insurance.com/broker-news/400-rise-in-pet-insurance-fraud-highlights-need-for-new-approach. accessed 18 june 2022 targ, s., almeida, d., lyman, k.: resnet in resnet: generalizing residual architectures (2016). arxiv preprint arxiv:1603.08029 vaswani, a., et al.: attention is all you need. advances in neural information processing systems 30 (2017) google scholar  yoon, b., so, h., rhee, j.: a methodology for utilizing vector space to improve the performance of a dog face identification model. appl. sci. 11(5), 2074 (2021) article  google scholar  zhang, k., sun, m., han, t.x., yuan, x., guo, l., liu, t.: residual networks of residual networks: multilevel residual networks. ieee trans. circuits syst. video technol. 28(6), 1303–1314 (2017) article  google scholar  zhang, x., zhao, r., qiao, y., wang, x., li, h.: adacos: adaptively scaling cosine logits for effectively learning deep face representations. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 10823–10832 (2019) google scholar  download references author information authors and affiliations sao paulo state university (unesp), bauru, sao paulo, brazil victor hugo braguim canto, joão renato ribeiro manesco & aparecido nilceu marana bank of brazil brasilia, federal district, brazil gustavo botelho de souza authors victor hugo braguim cantoview author publications search author on:pubmed google scholar joão renato ribeiro manescoview author publications search author on:pubmed google scholar gustavo botelho de souzaview author publications search author on:pubmed google scholar aparecido nilceu maranaview author publications search author on:pubmed google scholar corresponding author correspondence to victor hugo braguim canto . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper canto, v.h.b., manesco, j.r.r., de souza, g.b., marana, a.n. (2023). dog face recognition using vision transformer. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_3 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords animal biometrics dog identification facial recognition convolutional neural network visual transformer arcface publish with us policies and ethics profiles joão renato ribeiro manesco view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature applying reinforcement learning for multiple functions in swarm intelligence | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper applying reinforcement learning for multiple functions in swarm intelligence conference paper first online: 12 october 2023 pp 197–212 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) applying reinforcement learning for multiple functions in swarm intelligence download book pdf download book epub andré a. v. escorel ribeiro  orcid: orcid.org/0000-0002-2209-45929, rodrigo cesar lira  orcid: orcid.org/0000-0001-6089-62149, mariana macedo  orcid: orcid.org/0000-0002-7071-379x10, hugo valadares siqueira  orcid: orcid.org/0000-0002-1278-460211 & … carmelo bastos-filho  orcid: orcid.org/0000-0002-0924-53419  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 518 accesses abstract swarm intelligence (si) algorithms have become popular due to their self-learning characteristics and adaptability to external changes. they can find reasonable solutions to complex problems without in-depth knowledge. much of the success of these algorithms comes from balancing the exploration and exploitation tasks. this work evaluates the application and performance of a reinforcement learning approach applied to a well-known swarm intelligence algorithm, particle swarm optimization (pso). we use the reinforcement learning agent proximal policy optimization (ppo) to dynamically change the swarm communication topology according to the problem. we analyze the pso’s behavior, influenced by the reinforcement learning agent, through methods such as interaction networks and fitness analysis. we show that the rl approach can transfer the knowledge learned from one function to other functions, and that dynamic changes of topology over time makes pso much more efficient than setting only one specific topology, even when using a dynamic topology. our results then suggest that changing topologies might be more efficient than having a dynamic topology, and that indeed local and global topologies have an important role in the best swarm performance. our results take a step further on explaining the performance of si and automatizing their use for non-experts. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others integrating reinforcement learning and optimization task: evaluating an agent to dynamically select pso communication topology chapter © 2023 particle swarm optimization algorithm and its applications: a systematic review article open access 19 april 2022 local communication protocols for learning complex swarm behaviors with deep reinforcement learning chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence learning algorithms multiagent systems organizational learning stochastic learning and adaptive control artificial intelligence 1 introduction swarm intelligence (si) is a branch of computational intelligence in which a collective behavior is exhibited by a group of decentralized and self-organized simple reactive agents interacting with each other and the environment. the interaction among them generates a collective adaptation to allow them to solve complex problems [7]. the simple reactive agents are represented by positions in the search space with simple historical memories that explore it while exchanging information about their experiences with other agents. based on the individual’s experience and the information received by other members of the swarm, the agents can adapt the behavior in the search space and, with enough time, find and refine reasonable solutions to the presented problems. swarm-based algorithms emerge as an alternative to classical optimization methods in high-dimensional optimization problems [1]. they are an excellent alternative for optimization problems since they considerably reduce the computational cost and do not require a complete understanding of the problem regarding the characteristics of the search space. many si algorithms are based on animal social behavior metaphors, such as ant colony optimization (aco) [2] inspired by the behavior of ant colonies, particle swarm optimization (pso) [6] inspired by a flock of birds, artificial bee colony (abc) [5] inspired by the bee hives. swarm-based meta-heuristics are applied in several problems, such as nuclear engineering [18] and diagnosing diseases [14], among others. we also find applications to solve tasks related to data science and image and signal processing [13, 16, 19]. many efforts have been made to improve the performance of swarm-based algorithms. in the case of pso, xu et al. [21], and wu et al. [20] suggested the application of reinforcement learning due to this method’s characteristic of being able to learn which actions are best for a given state. it allows the dynamic modification of the behavior of the pso through the adjustment of communication topology using a reinforcement learning agent, generating better results for complex problems and increasing the convergence speed. recently, lira et al. [8] proposed a self-adaptive metaheuristic that considers the real-time information acquired during execution. for the algorithm to adapt, a reinforcement learning agent collects information and chooses actions that modify the metaheuristic’s behavior. despite the good preliminary results presented by reinforcement learning to create advanced approaches for swarm-based algorithms, it needs to be clarified if behaviors learned in some scenarios could be adapted in other scenarios not experienced by the algorithms. this paper evaluates the transfer learning capability of a reinforcement learning strategy when different topologies can be selected along the optimization with a pso algorithm, seeking to understand how the changes influence the various observed metrics and providing information about the learning of reinforcement agents and possible patterns that can be found. we find that rl is able to transfer the knowledge from one function to the other functions, and that changing topologies can be more effective than using a dynamic topology for pso. this paper is divided as follows: sect. 2 briefly describes particle swarm optimization, proximal policy optimization, and interaction networks. section 3 describes the methodology and parameterization for the experiments. section 4 presents our findings and results, and we finish in sect. 5 with our conclusions. 2 background 2.1 particle swarm optimization in 1995, kennedy and eberhart [6] proposed particle swarm optimization (pso) after observing the social behavior of flocks of animals, such as birds. pso is one of the metaheuristics of swarm intelligence most known and used in the literature [9]. particle swarm optimization consists of a group of simple agents, called particles, that will be scattered in a search space. while a stopping criterion is not reached, the particles update their velocities and positions at each iteration, keeping the information on the best solutions found by them (\(\vec {p}_i\)) and the best solutions found by their neighbors (\(\vec {n}_i\)). this information is used to calculate its next movement within the search space. in each pso iteration, the velocity (\(v_i\)) and position (\(x_i\)) information of each particle is updated according to eqs. 1 and 2. eventually, with enough iterations, the swarm will likely return a solution approaching the optimum position in the search space. $$\begin{aligned} \vec {v}_i(t+1) = \chi \big \{\vec {v}_i(t) + c_1\epsilon _1 [\vec {p}_i(t) \vec {x}_i(t)] + {c}_2\epsilon _2[\vec {n}_i(t) \vec {x}_i(t)]\big \} \end{aligned}$$ (1) $$\begin{aligned} \vec {x}_i(t+1) = \vec {v}_i(t+1) + \vec {x}_i(t), \end{aligned}$$ (2) \(c_1\) and \(c_2\) are the acceleration coefficients, \(\epsilon _1\) and \(\epsilon _2\) are uniform random numbers, and \(\chi \) is the constriction factor defined by \( \chi = \frac{2}{{\left| {2 \varphi \sqrt{\varphi ^{2} 4\varphi } } \right| }}\), where \(\varphi = c_1 + c_2\). the communication topology defines the neighborhood in pso. it describes the relations among particles, influencing the way the swarm behaves. global (gbest) and local (lbest) are two well-known topologies for pso. global is a fully connected topology where all particles can communicate with the entire swarm, allowing the best solution found to be shared quickly in the entire swarm. the global topology causes a quick convergence and may not adequately explore the search space. for the local topology, each particle is connected to k immediate particles in the swarm. it creates a ring-like communication structure using k=2. the particles have information only from the particles next to them so that the sub-swarms can independently converge on several optimal points. local topology has slower convergence but allows a better exploration of the search space [7]. however, these two topologies are more suitable for specific problems, which led oliveira et al. [4] to develop a more balanced topology that operates adaptively. in this approach, stagnant particles look for better particles to communicate. this approach adds a new attribute to the particle, called \(p_{k}\text {-}failure\), which is incremented every iteration that the particle does not improve its fitness. if the \(p_{k}\text {-}failure\) value exceeds a threshold (\(p_{k}\text {-}failure^{t}\)), the particle looks for a new neighbor to communicate. the choice of the neighbor for the particle to communicate with is probabilistic, based on roulette wheel selection, so that particles with better fitness have more chances of being chosen. 2.2 reinforcement learning reinforcement learning (rl) refers to learning which action should be taken in a situation (i.e., state) to achieve one or more goals [14]. at each iteration, a reinforcement learning agent receives observations of the current state scenario and takes an action from a list of actions allowed in that problem. after the action, the agent receives a reward, which measures whether the action was beneficial. through trial and error, the agent maps and learns the actions that obtained the best rewards for the observed states, seeking to choose the ones that maximize the accumulative reward. in reinforcement learning, the agent’s strategy to map the relationship between the observed states and the actions that must be taken is called policy. the agent aims to find the best strategy (i.e., policy) within the environment that maximizes the reward function. therefore, the agent can learn which policy works better for a given environment. reinforcement learning has been used with optimization meta-heuristics, including pso [14], seeking to improve the algorithm’s convergence speed. for applications in continuous and complex problems where mapping the set of states and actions is difficult, it is possible to use a deep reinforcement learning approach. the name deep reinforcement comes from deep learning because of the use of deep neural networks to map the set of states and actions. proximal policy optimization (ppo) schulman et al. [17] proposed the proximal policy optimization (ppo), a policy gradient method more stable, efficient, and more straightforward than other predecessors, as trust region policy optimization (trpo). ppo works to improve a policy, performing slight modifications. its main improvements are using clipped surrogate objective, value function clipping, reward and layer scaling, orthogonal, and adam learning rate [3]. ppo performed well in multiple benchmark problems for reinforcement learning [17]. 2.3 interaction network oliveira et al. [11] proposed the interaction network (in) aiming to understand the swarm dynamics better. in is a framework that assesses the flow of information generated from the agents’ interactions. the interaction network captures the exchange of information between agents, seeking to understand how the swarm influence each other. oliveira et al. [12] have demonstrated that using interaction networks helps to compare, for instance, the balance between exploration and exploitation tasks across algorithms. in is represented using a graph where each node represents an agent, and the edges represent the interactions between agents. the edges can be modelled in many ways, here, we modelled as shown in eq. 3. in this network, we do not consider how much one particle influenced each other, but who influenced whom over time that signs the communication topology structure [10]. $$\begin{aligned} i_{{t_{{i,j}} }} = \left\{ {\begin{array}{*{20}l} 1 &{} {\textrm{if}\,j\,\epsilon \,\vec {n}_{i} } \\ 0 &{} {\text {otherwise}} \\ \end{array} } \right. \end{aligned}$$ (3) in can be evaluated individually or by the accumulation of successive networks. iterations can be accumulated using a time window (tw) to capture the social interactions among agents in a frequency of iterations. the tw allows an analysis of which agents were neighbors to each other within an interaction interval so that large time windows make the interaction networks show the interactions that are most repeated. short tws contain the most recent interactions, while tws = 1 contain instant interactions. 3 methodology we used the reinforcement learning framework for swarm intelligence, created by lira et al. [8] based on python programming language and rlllibfootnote 1. we chose the pso in our experiments since it is a widely known and deployed swarm intelligence metaheuristics in the literature [9]. the simulation runs in time steps, allowing different swarm configurations. at the beginning of each time step, the rl agent acts, selecting local, global, or dynamic topology. after a preset number of iterations, the rl agents evaluate the reward and the new simulation state before starting the same cycle until the stop criterion is reached. we used this preset number of iterations equal to 10 in this paper. at the end of the training stage, the agent should be able to recommend the best topologies for functions with similar characteristics. we used ppo [17] as the reinforcement learning agent to solve this problem. the rl agent is responsible for learning the topologies that best adapt to the tested functions and modifying the topology in the pso based on the characteristics learned during execution. we used simple and widely used functions to evaluate the algorithms in unimodal and multimodal scenarios for simplicity and first validation. yet, it is challenging to appropriately cover all needed scenarios to evaluate a methodology [15], we argue that these scenarios are well-explored in the literature in regard to performance [9]. thus, we selected schwefel 2.21, sphere, and schwefel 2.22 unimodal functions, and rastrigin, griewank, and schwefel multimodal functions. the search space chosen for these functions is based on plevris and solozano [15] (table 1). then, two instances of rl agents were trained for each type of function per scenario. we trained with rastrigin for multimodal functions, and we used schwefel 2.22 for unimodal functions. we expect that the rl learns the characteristics needed from each type of function by training only in one example. table 1. description of the benchmark functions used in the simulations.full size table a given metaheuristic may perform well on a function with few dimensions and poorly on a function with multiple dimensions. this problem is called the “curse of dimensionality” – a well-known problem in data science that refers to the phenomena that arise when analyzing and organizing information in spaces of many dimensions that do not occur when few dimensions are implemented [15]. due to this problem, two different scenarios were empirically chosen for the number of dimensions and particles, scenario 1 with 20 particles and 50 dimensions and scenario 2 with 10 particles and 25 dimensions. in both scenarios, we used 1000 iterations as the stop criterion for the pso simulations. these values were chosen to guarantee convergence for multidimensional problems [15], and we point out that the number of particles is smaller than usually used in the literature in order to make the problem more complex with a smaller dimensionality. additionally, in pso, we used the parameters \(c_{1}=c_{2}=2.05\), and \(p_{k}\text {-}failure^{t}=1\). for the evaluation, we selected three metrics: (i) fitness, which is an indication of algorithm success; (ii) the distribution of the selected topology among local, global, and dynamic, i.e., which actions the reinforcement learning agent recommended the most; (iii) interaction networks (in) [10], which will be used to observe the accumulated interactions of the agents during a defined time interval, allowing us to analyse the importance of the selected topologies over time. we execute the pso without reinforcement, seeking to observe how the topologies perform in each chosen function. this topology performance information is used to compare the results obtained with the reinforcement, so the topologies with the best performance in the tested functions might be the majority of the agent’s recommendations. 4 results we divided our results into three subsections. first, we show the fitness performance of the rl proposal compared with pso using global, local, and dynamic topologies. next, we focus on our proposal, evaluating the topologies chosen in each scenario. finally, we analyze the interactions networks in the rl approach. 4.1 evaluating the fitness in figs. 1 and 2, we present the boxplots of the best fitness values found in 50 simulations in each scenario. we see that rl applied to pso reached, in general, better performance than using the pso with a specific topology for six functions. for scenario 1 (fig. 1), using unimodal functions, we see that the rl performed as well as the best communication topology found for each function. rl could have been more efficient for the multimodal functions. in schwefel function, it reached the worst results. in scenario 2 (fig. 2), we see our approach reaching competitive results. however, the results showed again that the training in rastrigin led rl to achieve bad results in schwefel. fig. 1. boxplot of the best fitness found on 50 simulations of each algorithm in scenario 1. full size image we then compared the results using a signal-ranked wilcoxon test with a confidence rate of 99.9% in tables 2 and 3. ‘–’ indicates no statistical difference between the solutions, ‘\(\blacktriangle \)’ indicates the rl approach achieved better results than the other algorithm, and ‘\(\blacktriangledown \)’ represents that our proposal reached worse results than the algorithm compared. based on the wilcoxon test results, we can assure the rl capability for solving different functions, even when we train only one of them with similar characteristics. only in schwefel function it did not work well. fig. 2. boxplot of the best fitness found on 50 simulations of each algorithm in scenario 2. full size image table 2. results of fitness values and wilcoxon test with a confidence level of \(99.9\%\) comparing the rl with the other algorithms for scenario 1.full size table table 3. results of fitness values and wilcoxon test with a confidence level of \(99.9\%\) comparing the rl with the other algorithms for scenario 2.full size table 4.2 evaluation of the selected topologies we now analyse the distribution of the selected topologies by rl over time steps. we expect that even by training on one example of a benchmark function (schwefel 2.22 or rastrigin), the rl agent will be able to learn a good policy for solving similar functions. fig. 3. percentage of times that a topology was selected by the agent, with its respective fitness evolution on the bottom of each plot for scenario 1. full size image fig. 4. percentage of times that a topology was selected by the agent, with its respective fitness evolution on the bottom of each plot for scenario 2. full size image we plot the percentage of time that a topology was chosen over time step coupled with the best fitness evolution over iteration in figs. 3 and 4. we can observe that in the first phase, “exploration phase”, the global topology was chosen most of the time, but the “exploitation phase” varies across experiments. the dynamic topology was most chosen for the “exploitation phase” indicating that the swarm needs more diversity from the connections to improve the fitness. in the “exploration phase”, being widely connected is more important than having a diverse set of connections. therefore, regardless of being unimodal and multimodal functions, it might be true that diversity on the connections is better as the swarm starts to exploit. the fitness improvement is larger while using the global topology, but we argue that this is not due to the fact that this topology is more efficient for the swarm. actually, this might be true because of the easiness of improving in a “exploration phase”. we see that for the schwefel function the swarm did not converge, the chosen topology is the local, corroborating with the literature that this topology works better than the global topology for complex multimodal functions. fig. 5. interaction network generated from the simulation with the best fitness for each function of scenario 1. full size image 4.3 analysing the interaction network we are also interested in understanding how the agents influence each other in their movement over iterations. we use the cumulative interaction network (in) to analyze the social interactions of the best simulation for each experiment in four-time windows: (i) between 0 and 99 iterations, (ii) between 100 and 199 iterations, (iii) between 200 and 299 iterations, and (iv) between 300 and 999 iterations, shown in figs. 5 and 6. each line represents the intensity of the influence of one particle on the displacement of the other particles. therefore, strong lines (yellow-red) indicate particles that strongly influence the swarm, and strong columns represent particles that are strongly influenced by the swarm. we can identify which topology impacted the most across time windows by analyzing the networks. in sect. 4.2, we see which topologies were more frequently chosen across simulations; here, we can observe which topologies impacted the most on the movement for the best experiments. if we observe strong diagonal lines and random points, the local, global, and dynamic topology substantially affected the displacement, respectively. we observe that for the best simulations, in scenario 1 (figs. 3 and 5), for unimodal functions, the global topology was more chosen combined with the dynamic at the end of the simulation which can be observed on the networks. nevertheless, the local topology strongly affects the movement from the middle to the end of the simulation (by looking at the diagonals from sphere and schwefel 2.22). for the multimodal functions, the local topology also appears as an essential element for the best simulations, even though it was not the most chosen one for rastrigin and griewank functions. in scenario 2 (figs. 4 and 6), we observe some similarities to scenario 1, but the dynamic topology is more present on the networks. the importance of the dynamic topology is in line with its performance, depicted in table 2. in contrast to the fact that the global topology was frequently chosen for the multimodal functions, the effect of this topology could have been more substantial than the other topologies. fig. 6. interaction network generated from the simulation with the best fitness for each function of scenario 2. full size image 5 conclusions in this paper, we applied rl to the pso, allowing the swarm to change its communication topology over time. we compared the efficiency of rl when trained on two functions and tested it on two other new functions with similar characteristics. we chose two well-established functions in the literature (rastrigin and schwefel 2.22) and tested them on two other multimodal and unimodal functions, respectively. using our simulated scenario, we demonstrated that applying reinforcement learning in swarm intelligence could be efficient across functions. we observed that rl could learn how to adapt to the environment even when not trained in the same function, indicating the capability of transferring learning among functions. nevertheless, a more comprehensive set of experiments is still essential for drawing stronger conclusions. our work was a step further in understanding how to automatize the use of swarm intelligence for unknown problems and how to understand the performance and patterns from si. swarm intelligence still requires expertise in the domain, so it is not as straightforward as it can become. in our future work, we aim to understand more clearly the reason for the topologies selected, seeking to understand why some functions, such as schwefel, obtained worse results in reinforcement learning. it is also necessary to evaluate if the training in a single unimodal or multimodal function is enough for the agent to learn the characteristics presented by the functions. the rl agent may more accurately identify the characteristics of the observed functions using multiple functions with the same characteristics in the training phase. notes 1.https://docs.ray.io/en/latest/rllib/index.html. references bansal, j.c., singh, p.k., pal, n.r. (eds.): evolutionary and swarm intelligence algorithms, studies in computational intelligence, vol. 779. springer international publishing, cham (2019). https://doi.org/10.1007/978-3-319-91341-4 dorigo, m., maniezzo, v., colorni, a.: ant system: optimization by a colony of cooperating agents. ieee trans. syst. man cybern. part b (cybern.) 26(1), 29–41 (1996). https://doi.org/10.1109/3477.484436 engstrom, l., et al.: implementation matters in deep policy gradients: a case study on ppo and trpo (2020) google scholar  junior, m.a.c.o., bastos filho, c.j.a., menezes, r.: using network science to define a dynamic communication topology for particle swarm optimizers. in: menezes, r., evsukoff, a., gonzález, m. (eds.) complex networks. studies in computational intelligence, vol. 424, pp. 39–47. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-30287-9_5 karaboga, d., et al.: an idea based on honey bee swarm for numerical optimization. technical report, technical report-tr06, erciyes university, engineering faculty (2005) google scholar  kennedy, j., eberhart, r.: particle swarm optimization. in: proceedings of icnn 1995 international conference on neural network, vol. 4, pp. 1942–1948 (1995). https://doi.org/10.1109/icnn.1995.488968 kennedy, j.: swarm intelligence, pp. 187–219. springer, us, boston, ma (2006) google scholar  lira, r.c., macedo, m., siqueira, h.v., bastos-filho, c.: integrating reinforcement learning and optimization task: evaluating an agent to dynamically select pso communication topology. in: tan, y., shi, y., luo, w. (eds.) advances in swarm intelligence. icsi 2023. lncs, vol. 13969, pp. 38–48. springer, cham (2023).https://doi.org/10.1007/978-3-031-36625-3_4 macedo, m., et al.: overview on binary optimization using swarm-inspired algorithms. ieee access 9, 149814–149858 (2021). https://doi.org/10.1109/access.2021.3124710 article  google scholar  oliveira, m., bastos-filho, c.j.a., menezes, r.: towards a network-based approach to analyze particle swarm optimizers. in: 2014 ieee symposium on swarm intelligence, pp. 1–8 (2014). https://doi.org/10.1109/sis.2014.7011791 oliveira, m., bastos-filho, c.j.a., menezes, r.: using network science to assess particle swarm optimizers. soc. netw. anal. min. 5(1), 3 (2015). https://doi.org/10.1007/s13278-015-0245-5 article  google scholar  oliveira, m., pinheiro, d., andrade, b., bastos-filho, c., menezes, r.: communication diversity in particle swarm optimizers. in: dorigo, m., et al. (eds.) ants 2016. lncs, vol. 9882, pp. 77–88. springer, cham (2016). https://doi.org/10.1007/978-3-319-44427-7_7 chapter  google scholar  parpinelli, r.s., lopes, h.s.: new inspirations in swarm intelligence: a survey. int. j. bio-inspired comput. 3(1), 1–16 (2011). https://doi.org/10.1504/ijbic.2011.038700 pervaiz, s., ul-qayyum, z., bangyal, w.h., gao, l., ahmad, j.: a systematic literature review on particle swarm optimization techniques for medical diseases detection. comput. math. methods med. 2021, 1–10 (2021). https://doi.org/10.1155/2021/5990999 article  google scholar  plevris, v., solorzano, g.: a collection of 30 multidimensional functions for global optimization benchmarking. data 7(4), 46 (2022). https://doi.org/10.3390/data7040046 article  google scholar  poli, r., kennedy, j., blackwell, t.: particle swarm optimization: an overview. swarm intell. 1, 33–57 (2007) article  google scholar  schulman, j., wolski, f., dhariwal, p., radford, a., klimov, o.: proximal policy optimization algorithms (2017). https://doi.org/10.48550/arxiv.1707.06347 da silveira câmara augusto, j.p., dos santos nicolau, a., schirru, r.: pso with dynamic topology and random keys method applied to nuclear reactor reload. progr. nucl. energy. 83, 191–196 (2015). https://doi.org/10.1016/j.pnucene.2015.03.009 wauters, t., verbeeck, k., de causmaecker, p., vanden berghe, g.: boosting metaheuristic search using reinforcement learning. in: talbi, eg. (eds.) hybrid metaheuristics. studies in computational intelligence, vol 434, pp. 432–452. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-30671-6_17 wu, d., wang, g.g.: employing reinforcement learning to enhance particle swarm optimization methods. eng. optim. 54(2), 329–348 (2022). https://doi.org/10.1080/0305215x.2020.1867120 article  mathscinet  math  google scholar  xu, y., pi, d.: a reinforcement learning-based communication topology in particle swarm optimization. neural comput. appl. 32(14), 10007–10032 (2020). https://doi.org/10.1007/s00521-019-04527-9 article  google scholar  download references acknowledgements the authors thank the federal institute of pernambuco (ifpe), brazilian national council for scientific and technological development (cnpq), processes number 40558/2018-5, 315298/2020-0, and araucaria foundation, process number 51497, for their financial support. mariana macedo was supported by the artificial and natural intelligence toulouse institute (aniti) institut 3ia: anr-19-pi3a-0004. author information authors and affiliations university of pernambuco, pernambuco, brazil andré a. v. escorel ribeiro, rodrigo cesar lira & carmelo bastos-filho university of toulouse, occitania, france mariana macedo federal university of technology, paraná, brazil hugo valadares siqueira authors andré a. v. escorel ribeiroview author publications search author on:pubmed google scholar rodrigo cesar liraview author publications search author on:pubmed google scholar mariana macedoview author publications search author on:pubmed google scholar hugo valadares siqueiraview author publications search author on:pubmed google scholar carmelo bastos-filhoview author publications search author on:pubmed google scholar corresponding author correspondence to rodrigo cesar lira . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ribeiro, a.a.v.e., lira, r.c., macedo, m., siqueira, h.v., bastos-filho, c. (2023). applying reinforcement learning for multiple functions in swarm intelligence. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_14 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords proximal policy optimization particle swarm optimization reinforcement learning swarm intelligence publish with us policies and ethics profiles mariana macedo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature transformer model for fault detection from brazilian pre-salt seismic data | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper transformer model for fault detection from brazilian pre-salt seismic data conference paper first online: 12 october 2023 pp 3–17 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) transformer model for fault detection from brazilian pre-salt seismic data download book pdf download book epub letícia bomfim9,11, oton cunha11, michelle kuroda11, alexandre vidal10,11 & … helio pedrini9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 611 accesses 6 citations abstract carbonate reservoirs are known for their heterogeneity, which poses challenges in interpreting and defining geological models. the brazilian reservoirs are formed mostly in highly faulted and fractured carbonate rocks, which can increase hydrocarbon transport and storage capacity. strategies that permit the identification of these structures allow the optimization in the exploration of a reservoir. to fulfill this task, machine learning models have been able to provide an understanding of these environments through the use of data obtained by seismic methods. the use of convolutional neural networks (cnns) has shown to be able to provide excellent abstractions in the field of semantic segmentation, including its use in seismic data. however, due to the highly heterogeneous formation of this type of data, the work of extracting information from these images remains challenging. from this, we investigate the potential of using transformer models in this geological context focusing on the faults identification. as a technique to analyze this type of architecture, we use the transunet network, which combines the power of cnns with the innovation brought by transformers in a hybrid model of deep learning. to evaluate its performance, we made use of conventional cnns to compare the results achieved. the results show that transunet outperforms conventional cnns, achieving a dice metric value of 88.34%, compared to 85.99% for u-net, 83.41% for u-net++, and 83.31% for segnet, being also able to identify small structures beyond what is indicated in our target. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others seismic stratum segmentation using an encoder–decoder convolutional neural network article 12 february 2021 a deep learning approach for automating the thin-section petrography of the carbonate rocks for effective reservoir characterization article 04 december 2024 a hybrid approach for automated carbonate body detection in seismic: a case study from offshore sarawak chapter © 2026 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer vision energy informatics machine learning structural geology artificial intelligence geotechnical engineering and applied earth sciences 1 introduction understanding and analyzing geological characteristics and their influence on reservoir properties is crucial for the success of hydrocarbon exploration and production. moreover, automating these processes can result in the production of deliverables that are useful in well planning optimization, reservoir modeling, and risk analysis (zheng et al. 2019). the pre-salt region in brazil has enormous exploration potential, with reserve estimates suggesting that it contains approximately 70 to 100 billion barrels of oil. the majority of production comes from the santos basin, which is located in the southeastern part of the country and accounts for over 70% of brazilian production. therefore, geological and geophysical studies, in conjunction with technological tools, have been employed to investigate optimal exploration conditions and identify the structural and sedimentary characteristics of these complex basins that contribute to their potential for hydrocarbon accumulation. gaining a comprehensive understanding of the spatial distribution of structural, faciological, and petrophysical properties can aid in optimizing the characterization, prediction, and recovery phases of reservoirs. this is achieved by creating more reliable models. the petrophysical properties of interest include the porous medium, which comprises matrix systems, vugs, and fractures that define spaces for fluid accumulation in rock, as well as permeability, which determines the connectivity of pores and is critical for percolation and the recovery of subsurface fluids. faults and fractures are crucial features within the porous medium, as they play a vital role in engineering, geotechnical, and hydrogeological applications. they can exhibit dual behavior by either serving as pathways for fluid flow or barriers, depending on factors such as intensity, connectivity, dissolution, or cementing, which renders them impermeable. numerous oil, gas, geothermal, and water supply reservoirs are formed in fractured rock. therefore, an essential aspect of comprehending and predicting fracture behavior involves identifying and locating those that demonstrate hydraulically significant behavior (council 1996). in order to analyze the properties of data from sources such as the pre-salt, which is located in ultra-deep waters below 5000m, and to identify fault regions, it is necessary to conduct seismic studies. this type of data offers insight into the structures present throughout the entire region of interest (dondurur 2018) providing information about layer structures and their features. upon obtaining this data, segmentation techniques are employed to extract the structures that will inform subsequent analyses. our aim is to incorporate the transformer concept in the automated segmentation of seismic data. while convolutional networks have been used in the literature for this task, recent studies have highlighted the advantages of transformers in computer vision tasks. according to bi et al. (2021), transformers exhibit a lower inductive bias, resulting in better performance due to fewer assumptions about the optimal approach. to this end, we have selected the transunet (chen et al. 2021) model, which combines transformers and u-net, as a promising alternative for identifying faults through image segmentation. 2 background the analysis of seismic data is crucial for the progress of hydrocarbon exploration, and it is commonly done through the study of geological structures. various techniques have been employed to analyze seismic data, including machine learning and image processing, which aim to automate and facilitate the interpretation process, using seismic data in the form of an image. pepper and bejarano (2005), for example, presented case studies on automatic fault interpretation using only seismic attributes that highlight faults. these attributes work similarly to filtering techniques used in image processing, and two of them, dip and azimuth, showed the best results in identifying fault regions that were extracted as connected components. zhao and mukhopadhyay (2018) explored the task of fault detection in synthetic and field data by using convolutional neural networks (cnns) to develop prediction models. in particular, zhao and mukhopadhyay (2018) improved the final result by adding image processing algorithms, such as smoothing and sharpening, after the prediction step. as the need for more robust models for seismic interpretation has become evident, researchers have turned to deep machine learning models for performing these tasks. wu et al. (2019b) developed several models with a primary focus on fault prediction, including faultnet3d and faultseg3d (wu et al. 2019a). using a single cnn, the faultnet3d model aimed to estimate the probability of faults, cracks, and dips. meanwhile, the faultseg3d model focused on fault delineation, with its output being a binary mask representing the seismic data, where 1 denotes the presence of faults and 0 represents the absence of faults. research on fault identification remains crucial in the geological context, as seismic data acquisition has significantly increased and deep convolutional neural networks have been successfully applied. recent approaches, including (an et al. 2021), have created a large database labeled by experts to supplement synthetic data. a deep cnn based on edge detection has been proposed, producing a pixel-by-pixel binary classification of faults with superior results compared to commonly used cnns. 3 materials the primary material for this work is the seismic data, which will be introduced along with its acquisition. figure 1 shows the four main parts of this phase: data section, data split, data augmentation, and data normalization. initially, two seismic cubes representing the input and target will be used to generate subsamples that will be augmented to provide the model with a diverse set of inputs. fig. 1. pre-processing workflow. full size image 3.1 seismic data and acquisition the acquisition of seismic data using a marine approach is the foundation of this work, as it enables the extraction of information and characteristics from sedimentary basins, such as those in the brazilian pre-salt. in order to conduct seismic surveys, a range of computational tools and systems are employed, with compatible real-time communication (dondurur 2018). this process involves propagating elastic waves through the subsurface medium, which then reflect off interfaces and return to the surface, where they are detected by receivers. sound waves are generated by equipment called airguns, which penetrate the marine subsoil and, upon reflection, are detected by receivers equipped with vibrating coils that produce electrical signals. these receivers, such as hydrophones, are often positioned near the water surface, attached by cables, or towed by seismic vessels (known as streamers). the electrical signals are transmitted via cables to a seismograph recorder on the ship, resulting in representative images of subsurface structures. the data then undergo a careful processing step, in which they are grouped and the signal-to-noise ratio is enhanced to create images of subsurface structures. the outcome of this acquisition process is a seismic volume, which is a three-dimensional function denoted by a(x, y, z). it reflects the changes in seismic amplitude (a) along the three coordinates: x, y, and z. the seismic amplitude (a) can be expressed as a function of depth (z), a function of (x, z), or a function of (x, y, z) (alsadi 2017). the dataset utilized in this study comprises two seismic volumes: the input and the target. the dimensions of both volumes are 1401\(\times \)1481\(\times \)241 pixels, and they cover an area of approximately 240 km\(^2\), encompassing two pre-salt santos basin fields. the vertical limit of the volumes is around 2000m within the area of interest. the input volume represents the amplitude seismic values of the area, while the target volume contains the faults interpreted from the amplitude seismic. the target volume was generated from 94 interpreted faults, as depicted in fig. 2, and used to create a binary model of fault (1) and no-fault (0) scenarios, based on the proximity to faults. fig. 2. target seismic definition. full size image 3.2 pre-processing we start by converting the seismic cube and mask into a numpy array, which are structured in a three-dimensional form. then, the data is manipulated in 2d sections to generate 2d sub-images that can be used to train the model. for this purpose, we extract smaller patches with dimensions \(p \times p\) pixels from the seismic inlines, where \(p=m\). this method provides a large image dataset with samples that have a conventional square dimensionality suitable for convolutional models. to avoid repetition of information, the inline region is sectioned into subimages with a \(stride=m\), without overlapping, as shown in fig. 3. this process generates \(\lfloor (n/p)\rfloor \) subimages from each of the treated inline images. fig. 3. process of data sectioning. full size image once the seismic data, input, and target have been prepared, the entire image dataset is randomized and divided into three separate sets: training, validation, and testing. these subsets are partitioned into a ratio of 70%, 15%, and 15% accordingly. the correspondence between the original seismic and the fault mask is maintained throughout the entire process. therefore, once the data had been cropped and separated, we opted to apply data augmentation to the images that displayed fault presence. this was necessary since, in this type of task, the majority of the dataset contains images without faults, and classifying pixels one by one tends to be more unbalanced, with more pixels labeled as 0 than as 1 (non-fault/fault) (wei et al. 2022). the augmentation technique applied to the database was the flip transformation. although there are various geometric transformations that can be used for augmentation, it’s important to consider the data domain to ensure that the operations do not introduce errors in the learning process. in this case, flipping was chosen as it preserves the original orientation of faults, which is typically vertical. by rotating on the vertical axis, the flip operation doubles the number of images, as illustrated in fig. 4. fig. 4. vertical flipping transformation. full size image using the described procedure, we initially generated a database of 9369 sub-images from our seismic dataset, which had a dimension of 1401\(\times \)1481\(\times \)160 pixels (inline/xlines/crossline). this was achieved by generating \(\lfloor (n/p)\rfloor = 9\) images per inline, where \(p = 160\). the resulting images were split into 6558 for training, 1405 for validation, and 1406 for testing. to increase the training and validation sets, data augmentation was applied to all images containing faults, resulting in a 29% increase in the training data and a 36% increase in the test data. as a result, the total number of images in the respective groups became 9242 and 1914. finally, all 2d image sections referring to seismic are normalized between -1 and 1, using the following equation: $$\begin{aligned} \textit{img} = 2 \cdot \frac{\textit{img}[:,:] \min }{\max \min } -1 \end{aligned}$$ (1) where img is the 2d image resulting from the normalization, \(\min \) is the minimum image value, and \(\max \) is the maximum image value. 4 methods the transunet model, which uses transformers, is utilized in this work. given the success of this architecture in the realm of visual computing, our aim is to evaluate its efficacy for fault extraction in heterogeneous seismic fields and compare it with traditional models such as convolutional neural networks. the fig. 5 shows the training input and the models used in this process. fig. 5. methodology workflow. full size image 4.1 cnn models image classification, a fundamental problem in computer vision, involves categorizing images into predefined classes and serves as the basis for other tasks such as region localization, detection, and segmentation. the cnns are one of the most commonly used deep learning networks for this task, named after the linear mathematical operation called convolution between matrices (albawi et al. 2017). cnns are a type of feedforward neural networks, meaning that the information flows only in one direction, from the input to the output. inspired by biological neural networks (rawat and wang 2017), cnns differ from regular neural networks in that each unit in a cnn layer is a two-dimensional filter that is convolved with the input of that layer, enabling them to extract local features from images (khan et al. 2018). the convolutional layer of a cnn consists of a two-dimensional filter that convolves with the input feature map. the filter is an array of discrete numbers, where each element is a weight, learned during the training phase. in the beginning, these weights are randomly assigned until the learning process updates them. the cnn architecture includes layers for convolution, pooling to reduce the feature map size, and fully connected layers where each neuron is directly connected to the neurons in the previous and next layers. to gain an understanding of how convolutional networks can be constructed, we will explore three models from the u-net family and subsequently utilize them to draw comparisons with the transformer architecture. u-net. the original purpose of the u-net network was to perform segmentation of medical images, and its architecture was an update and extension of the fully connected network. u-net aimed to improve segmentation accuracy while minimizing the required amount of data (long et al. 2015). the u-net architecture, proposed by ronneberger et al. (2015), consists of a contraction path (left side) and an expanding path (right side) for accurate segmentation of medical images. the contraction path has a typical convolutional network architecture, comprising two 3\(\times \)3 convolutions, each followed by a rectified linear unit (relu), and a max-pooling operation with a 2\(\times \)2 filter and stride 2 for downsampling. the number of feature channels is doubled at each downsampling step. the expansive path, on the other hand, involves increasing the resolution of the feature map, followed by a 2\(\times \)2 convolution (half the number of channels), concatenation with the corresponding feature map of the contraction path, and two 3\(\times \)3 convolutions (each followed by a relu). the last layer uses a 1\(\times \)1 convolution to map the output to the desired number of classes. u-net ++ . zhou et al. (2018) developed the u-net++ architecture with the goal of improving the accuracy of medical image segmentation. this architecture is based on dense and nested skip connections, which provide a new approach to the segmentation task. the u-net++ architecture was developed to enhance the performance of medical image segmentation by addressing limitations in previous models. zhou et al. (2019) proposed an approach that employs multiple u-nets with different depths, where the encoders and decoders are connected by dense and nested skip connections. the u-nets share an encoder, while their decoders are interconnected, and deep supervision is used during to train all the constituent u-nets while benefiting from a shared image representation. the redesigned skip connections in u-net++ allow for variable-scale feature maps at a decoder node, enabling the aggregation layer to decide how attribute maps carried over the skip connections should be merged with the decoder feature maps. segnet. the segnet (badrinarayanan et al. 2017) is a convolutional neural network architecture designed for semantic pixel segmentation, comprising of an encoder network, a corresponding decoder network, and a pixel-wise classification layer. the encoder network architecture is similar to that of vgg-16 with 13 convolutional layers. the decoder network also has 13 layers, each corresponding to an encoder layer. the final output of the decoder is fed into a multiclass softmax classifier to generate class probabilities for each pixel. unlike u-net, segnet does not reuse pooling indices but transfers the entire attribute map to the corresponding decoder and concatenates them into upsampled decoder feature maps through deconvolution. 4.2 transformer models the transformer (vaswani et al. 2017; khan et al. 2022) is a novel neural network that utilizes attention operations and was originally developed for natural language processing (nlp), where it has demonstrated remarkable success (huang et al. 2020). in the field of computer vision, the transformer has been increasingly employed to replace traditional techniques, resulting in various advantages (bi et al. 2021). the transformer architecture includes an encoder and a decoder, both containing multiple attention blocks with the same architecture. the encoder produces encodings of the input, while the decoder takes these encodings and utilizes its contextual information to generate the output sequence (han et al. 2022). specifically, the transformer encoder is composed of l layers of multihead self-attention (msa) and multi-layer perceptron (mlp) blocks, alternating between the two. before each block, layer normalization (ln) is applied, and residual connections are used after every block. finally, the encoded feature representation is upsampled to full resolution to predict the dense output. the success of the transformer in nlp has encouraged researchers to explore its potential in other areas. consequently, similar models have been developed to learn useful image representations using the transformer’s concept. the vision transformer (vit) (dosovitskiy et al. 2020), for instance, has proven to be highly effective in several benchmarks, drawing inspiration from the self-attention mechanism in nlp, where word embeddings are substituted by patch embeddings (fu 2022). vit has paved the way for the development of several other models based on attention mechanisms, which have brought about significant advances in various fields of computer vision. surveys conducted by guo et al. (2022) and han et al. (2022) have shown that attention-based methods have been beneficial for tasks such as image classification, semantic segmentation, face recognition, few-shot learning, medical image processing, image resolution, 3d vision, among others. transunet (chen et al. 2021) is a model that harnesses the power of transformers for medical image segmentation. by combining cnn architectures, such as u-net, which can extract low-level visual features to preserve fine spatial details, and transformers, which excel in modeling global context, transunet creates a powerful hybrid architecture for accurate and efficient medical image segmentation. transunet architecture. transunet combines cnn and transformer architectures to leverage the spatial details of cnn features and the global context captured by transformers for medical image segmentation. the model follows a u-shape design, where transformers establish self-attention mechanisms to encode the features in a sequence-by-sequence prediction perspective. the resulting self-attentive feature is upsampled and combined with high-resolution cnn features that were skipped during encoding, enabling precise localization. transformer is used as an encoder by transforming the input image into a sequence of flattened 2d patches through tokenization. to achieve this, the input image x is reshaped into n patches of size \(p \times p\), where n is determined by the image’s height and width (h and w) and the patch size (p), such that \(n=\frac{hw}{p^{2}}\). a unique marker is assigned to each patch to preserve its positional information in the sequence, and the resulting sequence is fed as input to the encoder. to recover the spatial order during upsampling, the encoded feature size is reshaped from \(\frac{hw}{p^{2}}\) to \(\frac{h}{p} \times \frac{w}{p}\), while the number of channels is reduced to the number of classes using 1\(\times \)1 convolutions. finally, the feature map is bilinearly upsampled to the full resolution of \(h \times w\) to generate the final segmentation output. to address the issue of partial information loss resulting from using transformer solely as an encoder, transunet utilizes a hybrid cnn-transformer architecture that first leverages cnn to extract features from the input, followed by patch embedding of 1\(\times \)1 patches extracted from the cnn feature map instead of the raw images. therefore, the sequence of hidden features is reshaped to achieve full resolution from \(\frac{h}{p} \times \frac{w}{p}\) to \(h \times w\) by applying multiple cascades of upsampling blocks. each block includes a 2\(\times \) upsampling operator, a 3\(\times \)3 convolution layer, and a relu layer in sequence. this enables the aggregation of features at different resolution levels through skip connections. 4.3 evaluation metrics as the task can be seen as a semantic segmentation task, it is crucial to rigorously evaluate the system’s efficiency for it to be useful and produce effective contributions. to evaluate the effectiveness of a segmentation system used for extracting faults in images using machine learning methods, two essential metrics that assess the quality of segmentation were selected: jaccard index: also known as the intersection over union (iou), it quantifies the percentage overlap between the mask input (i.e., the target of the segmentation) and the predicted output. the jaccard index (shi et al. 2014) is calculated as the number of pixels that are common between the two images \((a\cap b)\) divided by the number of pixels resulting from the union of both \((a\cup b)\). dice coefficient: is a measure frequently utilized in the computer vision field to evaluate the similarity between two images (crum et al. 2006). it is similar to the iou and computed as twice the overlap area divided by the total number of pixels in both images. the performance assessment of the employed models can also be examined through a pixel-wise classification approach. this method employs metrics, including accuracy, recall, precision, and f1-score, to evaluate the prediction performance of positive samples. the matrix presents the predictions made for each class and provides a representation to measure the effectiveness of the classification model. in the case of fault detection applications, positive samples represent faults and their locations in a seismic volume (huang et al. 2017). 5 results and discussions the presentation and discussion of the results obtained from the trained models, seismic areas reconstruction, and performance metrics are presented using the previously described model. 5.1 experiments to assess the effectiveness of applying transformer on seismic data, we utilized the transunet model on the database outlined in sect. 3. moreover, we employed three additional models that employ the convolutional neural network approach to compare the attained results. the architectures used were parameterized with identical settings for the loss function, learning rate, and number of epochs to enable comparison of outcomes with the same initialization. an empirical value of \(1e-4\) was chosen for the learning rate based on previous evaluations of different values. for the loss function, binary cross-entropy was chosen as it is commonly used for classification purposes and semantic segmentation is a pixel-level classification task. the number of epochs was set to 100 for all models, except for transunet where the batch size was reduced to 16 due to its higher memory complexity compared to the other networks. the results obtained from the execution of all methods are presented in table 1, where it can be observed that the transunet network surpasses the other architectures (u-net, u-net++, and segnet) by 2.35%, 4.93%, and 5.03%, respectively, achieving an overall dice score of 88.34%. considering the iou metric, this difference increases to 4.86%, 6.73%, and 7.23%, with transunet obtaining a value of 84.34%. it is worth noting that the difference between the values obtained by these two metrics is mainly due to the high penalty imposed by the iou in cases where the classification results are poor. the evaluation of the other metrics confirms the superior performance of the transunet network. table 1. quantitative comparison of the segmentation performance in fault detection task.full size table the comparisons between the predictions made by the previously presented models are illustrated in fig. 6. the results indicate that u-net and transunet produced images that are more similar to the target, while segnet and u-net++ exhibit a considerable amount of noise in their outputs. when comparing u-net and transunet, a slightly more accurate border delimitation can be observed in u-net. this may be due to the greater abstraction of global context extraction in the initial layers of transunet. fig. 6. qualitative comparison of different models applied to seismic segmentation. full size image in order to conduct a more comprehensive analysis of the performance of the two networks, we reconstructed the slices that were formed by each predicted sub-image. this reconstruction was made possible by the fact that each input image to the network has a corresponding nomenclature that corresponds to the seismic inline and its cut order. consequently, the training images were predicted in sequential order and their results were concatenated, as illustrated in an example shown in fig. 7. fig. 7. image prediction and concatenation process. full size image by applying this approach, we can reconstruct the entire seismic volume and examine the output of the two models. figure 8 presents a comparison of the predictions made for two different slices, indicating that both models were able to detect the structures highlighted in the target, as well as some smaller regions, with transunet providing a more significant representation of them. fig. 8. comparison between u-net and transunet predictions. full size image upon analyzing the overall result by visualizing the seismic cube prediction, as presented in fig. 9, it is evident that both models were able to identify most of the structures present in the target. however, some additional small regions were also detected, which require detailed analysis when viewed in two dimensions, as depicted in fig. 10. the transunet prediction achieved slightly better results, with more indicated faults and greater vertical continuity of faults outside the regions present in the target binary cube. nevertheless, the overall difference between the two methods was minimal. fig. 9. comparison between u-net and transunet prediction in full 3d field. full size image fig. 10. analysis of predicted failures in two different regions, a and b. full size image 6 conclusions this work investigated the use of a hybrid model, transunet, which combines the strengths of convolutional networks and transformer’s content abstraction in the geological context. the results demonstrate the effectiveness of this approach in segmenting seismic images from a heterogeneous environment, such as the pre-salt layer, indicating potential applications of this architecture in various configurations for identifying and extracting geological structures in the field of seismic imaging. the effectiveness of transunet in fault identification on seismic data was demonstrated by comparing it to conventional state-of-the-art methods. this study concludes that the incorporation of transformer in this context has the potential to extract valuable information from seismic databases. the evaluation included both qualitative and quantitative approaches, suggesting that this new type of architecture could serve as a benchmark for other databases considering the use of transunet in the field of visual computing. references albawi, s., mohammed, t.a., al-zawi, s.: understanding of a convolutional neural network. in: international conference on engineering and technology, pp. 1–6. ieee (2017) google scholar  alsadi, h.n.: seismic hydrocarbon exploration. in: 2d and 3d techniques, seismic waves. springer (2017) google scholar  an, y., guo, j., ye, q., childs, c., walsh, j., dong, r.: deep convolutional neural network for automatic fault recognition from 3d seismic datasets. comput. geosci. 153, 104776 (2021) article  google scholar  badrinarayanan, v., kendall, a., cipolla, r.: segnet: a deep convolutional encoder-decoder architecture for image segmentation. ieee trans. pattern anal. mach. intell. 39(12), 2481–2495 (2017) article  google scholar  bi, j., zhu, z., meng, q.: transformer in computer vision. in: ieee international conference on computer science, electronic information engineering and intelligent control technology, pp. 178–188. ieee (2021) google scholar  chen, j., et al.: trans-unet: transformers make strong encoders for medical image segmentation. arxiv preprint arxiv:2102.04306 (2021) council, n.r.: rock fractures and fluid flow: contemporary understanding and applications. national academies press (1996) google scholar  crum, w.r., camara, o., hill, d.l.: generalized overlap measures for evaluation and validation in medical image analysis. ieee trans. med. imaging 25(11), 1451–1461 (2006) article  google scholar  dondurur, d.: acquisition and processing of marine seismic data. elsevier (2018) google scholar  dosovitskiy, a., et al.: an image is worth 16\(\times \)16 words: transformers for image recognition at scale (2020). arxiv preprint arxiv:2010.11929 fu, z.: vision transformer: vit and its derivatives. arxiv e-prints, pages arxiv-2205 (2022) google scholar  guo, m.-h., et al.: attention mechanisms in computer vision: a survey. comput. visual media, pp. 1–38 (2022) google scholar  han, k., et al.: a survey on vision transformer. ieee trans. pattern anal. mach. intell. (2022). abs/2012.12556:1–20 google scholar  huang, l., dong, x., clee, t.e.: a scalable deep learning platform for identifying geologic features from seismic attributes. lead. edge 36(3), 249–256 (2017) article  google scholar  huang, m., zhu, x., gao, j.: challenges in building intelligent open-domain dialog systems. acm trans. inf. syst. 38(3), 1–32 (2020) google scholar  khan, s., et al.: transformers in vision: a survey. acm comput. surv. 54(10s), 1–41 (2022) article  mathscinet  google scholar  khan, s., rahmani, h., shah, s.a.a., bennamoun, m.: a guide to convolutional neural networks for computer vision. synthesis lectures comput. vision 8(1), 1–207 (2018) article  google scholar  long, j., shelhamer, e., darrell, t.: fully convolutional networks for semantic segmentation. in: ieee conference on computer vision and pattern recognition, pp. 3431–3440 (2015) google scholar  pepper, r.e.f., bejarano, g.: ps advances in seismic fault interpretation automation. in: aapg annual convention (2005) google scholar  rawat, w., wang, z.: deep convolutional neural networks for image classification: a comprehensive review. neural comput. 29(9), 2352–2449 (2017) article  mathscinet  math  google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: international conference on medical image computing and computer-assisted intervention, pp. 234–241. springer (2015) google scholar  shi, r., ngan, k.n., li, s.: jaccard index compensation for object segmentation evaluation. in: ieee international conference on image processing, pp. 4457–4461. ieee (2014) google scholar  vaswani, a., shazeer, n., et al.: attention is all you need. in: advances in neural information processing systems, 30 (2017) google scholar  wei, x.-l., et al.: seismic fault detection using convolutional neural networks with focal loss. comput. geosci. 158, 104968 (2022) article  google scholar  wu, x., liang, l., shi, y., fomel, s.: faultseg3d: using synthetic data sets to train an end-to-end convolutional neural network for 3d seismic fault segmentation. geophysics 84(3), im35-im45 (2019a) google scholar  wu, x., shi, y., fomel, s., liang, l., zhang, q., yusifov, a.z.: faultnet3d: predicting fault probabilities, strikes, and dips with a single convolutional neural network. ieee trans. geosci. remote sens. 57(11), 9138–9155 (2019) article  google scholar  zhao, t., mukhopadhyay, p.: a fault detection workflow using deep learning and image processing. in seg international exposition and annual meeting, onepetro (2018) book  google scholar  zheng, y., zhang, q., yusifov, a., shi, y.: applications of supervised deep learning for seismic interpretation and inversion. lead. edge 38(7), 526–533 (2019) article  google scholar  zhou, z., siddiquee, m.m.r., tajbakhsh, n., liang, j.: u-net++: a nested u-net architecture for medical image segmentation. in: deep learning in medical image analysis and multimodal learning for clinical decision support, pp. 3–11. springer (2018) google scholar  zhou, z., siddiquee, m.m.r., tajbakhsh, n., liang, j.: u-net++: redesigning skip connections to exploit multiscale features in image segmentation. ieee trans. med. imaging 39(6), 1856–1867 (2019) article  google scholar  download references acknowledgments the authors would like to thank the brazilian national council for scientific and technological development (cnpq #304836/2022-2) and shell brasil petróleo ltda for their support. author information authors and affiliations institute of computing (ic), university of campinas, campinas, sp, brazil letícia bomfim & helio pedrini institute of geosciences (ig), university of campinas, campinas, sp, brazil alexandre vidal center for energy and petroleum studies (cepetro), campinas, sp, brazil letícia bomfim, oton cunha, michelle kuroda & alexandre vidal authors letícia bomfimview author publications search author on:pubmed google scholar oton cunhaview author publications search author on:pubmed google scholar michelle kurodaview author publications search author on:pubmed google scholar alexandre vidalview author publications search author on:pubmed google scholar helio pedriniview author publications search author on:pubmed google scholar corresponding author correspondence to helio pedrini . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper bomfim, l., cunha, o., kuroda, m., vidal, a., pedrini, h. (2023). transformer model for fault detection from brazilian pre-salt seismic data. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_1 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords fault detection transformer models seismic data publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature efficientdeeplab for automated trachea segmentation on medical images | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper efficientdeeplab for automated trachea segmentation on medical images conference paper first online: 12 october 2023 pp 154–166 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) efficientdeeplab for automated trachea segmentation on medical images download book pdf download book epub arthur guilherme santos fernandes9, geraldo braz junior9, joão otávio bandeira diniz9, aristófanes correa silva9 & … caio eduardo falcõ matos9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 505 accesses 2 citations abstract segmentation of organs at risk is a fundamental step during radiotherapy planning for cancer treatment. its goal is to preserve healthy tissue around the tumor and ensure that the most radiation strikes only cancer cells. physicians do this job manually, which can be slow and error-prone. thus, automatic segmentation methodologies can speed up organ delimiting during radiotherapy planning. this work designs a method, efficientdeeplab, a convolutional neural network architecture trained on ct scans for trachea segmentation, and obtained an 88.6% dice score. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others eff-unet for trachea segmentation on ct scans chapter © 2024 deep learning empowered volume delineation of whole-body organs-at-risk for accelerated radiotherapy article open access 02 november 2022 multi-organ segmentation of the head and neck area: an efficient hierarchical neural networks approach article 07 march 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. automated pattern recognition cone-beam computed tomography computer vision elispot flow cytometry speech and audio processing 1 introduction radiotherapy is one type of cancer treatment consisting of blasting high radiation doses into unhealthy cells to destroy or shrink tumors [2]. identifying the tumor location and the surrounding organs is the first step in radiotherapy planning and helps diminish the ionizing radiation effects on healthy tissue. we call these healthy organs that surround a tumor organs at risk (oar) [8, 18]. a physician does the manual segmentation of the oar using images obtained by computed tomography (ct). ct is an imaging exam that allows doctors to evaluate the body’s soft and hard tissue health. an x-ray emitting machine spins around the patient while sensors at the other end capture the radiation that trespasses the patient and convert it to a digital image [1, 14]. the result is a 3d image that can be sliced down to 2d for better visualization [10]. figure 1 shows examples of ct slices. fig. 1. 2d slices from a ct test. a) axial view; b) coronal view; c) sagital view. full size image accurate manual segmentation of oar is crucial because it provides the exact location of healthy tissue near the tumor so that specialists can prevent unnecessary x-ray exposition and minimize the side effects of radiation (inflammation, fibrosis, ulceration, and even organ failure [21]). however, it depends on the physician’s expertise and can be a slow and error-prone process. smaller organs like the trachea are hard to segment due to size, texture similarity with surrounding tissue, and lack of detail richness. because of that, recent research focused on the automatic segmentation of organs at risk. based on the importance of trachea segmentation for radiotherapy planning and current literature on automatic medical image segmentation, this work proposes utilizing image pre-processing techniques and proposes an efficientdeeplab convolutional network architecture for trachea segmentation from computed tomography images. general contributions are: (a) pixel intensity windowing for better contrast on trachea images, and (b) a method that merges efficientnet and deeplabv3+ architectures to perform trachea segmentation, increasing the convolutions field-of-view by using atrous convolutions, leading to more detailed segmentation, considerably less network parameters and efficient utilization of computational resources. 2 related work some works develop methods for trachea segmentation using fully convolutional networks or combining transformers with these networks. driven by recent advances in transformer architecture for image segmentation, [24] used it alongside u-net to perform segmentation on various oars, including the trachea. some methods use the 3d nature of tomography to extract contextual information about the ct scan, like in [22] and [23]. other methods combine u-net and another convolutional network architecture to perform oar segmentation: [9] used pyramid fusion module to merge features from different scales on the u-net and [25] applied generative adversarial networks and u-net together to segment organs from ct images. there are also some efforts to build public datasets for oar segmentation. in [17], researchers gathered ct scans from 60 patients diagnosed with lung cancer to train a u-net with 2d input. of those 60 ct scans, 40 are publicly available, composing the segthor dataset. the literature presents efforts based on methods such as u-net, but without necessarily worrying about the efficiency of networks in terms of trainable parameters. starting from this point, we propose a method where the network backbone is built using efficientnet and network decoder/encoder with deeplab. the aim is to build a model with few trainable parameters, easily applicable in small datasets, and presenting consistent results. 3 materials and methods this section presents the proposed method, its steps, and information about the image dataset used for training. figure 2 illustrates the general pipeline of this work, beginning from the image acquisition, passing to the preprocessing work, efficientdeeplab model training, and segmentation mask output. fig. 2. general workflow of the method. full size image 3.1 dataset the images in the experiment come from the segthor database [17]. there are 40 publicly available 3d ct volumes from patients affected by lung cancer. the volumes were sliced along the longitudinal body axis to generate 7420 2d images. from the total, 1987 shows trachea and the other 5433 don’t. this distribution generates an unbalanced dataset, so, for training purposes, 1987 images without trachea were randomly selected, resulting in equally distributed slices. the data also includes a physician-made manual segmentation for ground truth. figure 3 shows a trachea region slice and the correspondent demarcation. fig. 3. a) axial view of trachea slice; b) manual segmentation made by radiology professional. full size image the pixel intensity values on a ct image range from -1000 to 1000, according to the hounsfield scale [11], which is a way to represent tissue density in radiography images (the denser the tissue, the brighter the image). table 1 summarizes the hounsfield scale. table 1. hounsfield intensity values by kind of biological tissue.full size table pre-processing: for better contrast, we applied an intensity windowing method, limiting the pixel values to an interval, thus discarding unwanted information, after this process, a normalization followed by image resizing were applied. in the training phase, a random data augmentation technique was applied. intensity windowing: this technique is employed within the radiology domain to improve ct imaging. its usage varies depending on the application, allowing for enhanced visualization of particular body tissues. let \(\alpha \) and \(\beta \) represent the lower and upper boundaries of the interval \([\alpha , \beta ]\), respectively. the intensity windowing technique involves clipping the pixel values outside this interval to the closest boundary value while letting pixels within the interval remain unchanged. these intervals are called windows, and each range emphasizes different kinds of tissues from the body. the trachea is a hollow tubular organ, so the chosen intensity interval was [−1000, 60] because it enhances the contrast between the trachea muscle walls and the air inside it [6]. figure 4 shows different window ranges applied to the same image. normalization: after the intensity windowing, we also apply normalization, subtracting the pixel values from the image mean and dividing by its standard deviation. this way, the data tend to be more uniform, facilitating the learning process for the model. equation 1 describes the intensity windowing process, and eqs. 2 to 4 detail the image normalization method. resizing: after the normalization, the images were resized from 512\(\,\times \,\)512 to 256\(\,\times \,\)256. fig. 4. examples of different intensity windowing preprocessing applied to ct images. full size image $$\begin{aligned} f(x) = {\left\{ \begin{array}{ll} -1000, &{} \text {if } x < -1000 \\ 60, &{} \text {if } x > 60\\ x, &{} \text {if } x \in [-1000, 60] \end{array}\right. } \end{aligned}$$ (1) $$\begin{aligned} \mu = \frac{\sum _i^n{i_{i}}}{n} \end{aligned}$$ (2) $$\begin{aligned} \sigma = \sqrt{\frac{\sum _i^n{(i_{i} \mu )^2}}{n}} \end{aligned}$$ (3) $$\begin{aligned} f_{norm}(i) = \frac{i \mu }{\sigma } \end{aligned}$$ (4) data augmentation: this process is executed to insert variance into the training data and make the model more robust and generalist. a random rotation is set to each image, ranging from −45 to 45\(^\circ \) with a chance of 50%. 3.2 model architecture for semantic segmentation of the trachea area, we apply a neural network architecture based on deeplabv3+ [5] combined with efficientnet [20] for feature extraction. figure 5 illustrates the encoder/decoder model architecture overview. fig. 5. architectural overview of efficientdeeplab. full size image the main deeplabv3+ innovations are the atrous convolutions (also called dilatable convolutions) and atrous spatial pyramid pooling [12]. this way, we are capable of merging features from multiple scales. in the original deeplabv3+, the author uses resnet [13] architecture as a feature extraction backbone, aggregating intermediate layer feature maps, summing them, and interpolating back to the input resolution, thus obtaining a segmentation output mask. in this work, we modified the feature extraction backbone to use efficientnet, a scalable, robust, and compact deep learning model that surpasses resnet in recent benchmarks on imagenet [7] dataset. efficientnet: built with efficient resource usage in mind, it consists of a family of networks derived from a baseline model obtained with neural architecture search. the baseline network is called the b0, the other models range from b1 to b7, and the differences are the input resolution, the number of channels, and layers in each stage, such that the larger the number on the network’s name, the bigger the model (in terms of parameters). the b0 model’s main structure is summarized in table 2. table 2. efficientnet b0 architecture [20].full size table the main building blocks of the efficientnet are the mbconv blocks, introduced in mobilenetv2 [19]. they are an inverted residual bottleneck, expanding the number of channels in the intermediate layers and lately squeezing them to their original size. figure 6 illustrates the structure of the mbconv block. the number that follows the name in mbconv1 and mbconv6 is a multiplier that indicates the number of filters in the leftmost 1\(\,\times \,\)1 convolutional layer. for example, if the number of input channels is 4, then the number of filters will be 4 in the mbconv1 and 24 in the mbconv6. fig. 6. schematic diagram of mbconv. full size image two outputs from the efficientnet model are used: the last convolutional layer activations and the intermediate features from the third convolutional block. the first one feeds the encoder part, while the latter feeds the decoder part. atrous convolution: introduced by deeplabv1 [3], it is a convolution with dilatable kernels, and its goal is to achieve a larger field of view on feature extraction. the filter is filled with zeros according to a chosen dilation rate r. figure 7 exemplifies the dilation rate effect over the convolution kernel. notice that the area covered by the filter goes from 3\(\,\times \,\)3 to 5\(\,\times \,\)5, but the number of parameters is still the same. varying the dilation rate along the network convolutions helps the model to capture features on different scales. deeplabv3+ also implements an atrous version of the depthwise separable convolution, where the depthwise convolution also has a dilation in the kernel. fig. 7. atrous convolution kernel with different dilation rates. full size image atrous spatial pyramid pooling: inspired by the spatial pyramid pooling (spp) proposed in [12], the authors of deeplab introduced the atrous version of it in [4]. the method encodes features from different resolutions into fixed-size feature maps and concatenates them together to feed the rest of the network, as illustrated in fig. 8. fig. 8. architectural overview of atrous spatial pyramid pooling module. full size image encoder/decoder: following the recipe of the most successful methods for semantic image segmentation, this work uses the encoder/decoder structure for extracting features and restoring them to the original resolution. the efficientnet model serves as the feature extraction backbone alongside the aspp module, and both compose the encoder part of the model. the decoder inputs are the features from efficientnet’s third layer. they are further convoluted by one atrous separable convolution and one 1\(\,\times \,\)1 convolution layer. the result is concatenated with the encoder output to get convolved and upsampled to the original image resolution, outputting an image segmentation mask. 3.3 results we conducted experiments to validate the proposed method. the training environment was a computer with intel core ™ i5-10400f cpu @ 2.90 ghz and one nvidia ® gpu rtx-3060 12 gb of memory. for the implementation of the model, we used python programming language and the libraries: tensorflow, keras, and tensorflow advanced segmentation models [15]. for a better understanding of how large the feature extraction backbone model needs to be, we experimented with effcientnet b0, b1, and b2, estimating which one suits better for the problem. we did not use the models above b2 because the dataset is not big enough to supply those models. the pre-processed segthor 2d slices were split 70% for training and 30% for the test, per patient. it is important to emphasize that the results use at least 30% fewer data than the works on the segthor dataset due to the unavailability of 20 test case patients from the original 60 patients on the dataset. the training was executed for 100 epochs and used adam (adaptive moment estimation) to update the network weights [16]. the loss function used is binary cross-entropy loss \(+\) dice loss. the training process was repeated 5 times for each model, following the holdout validation method. table 3 summarizes the dice and jaccard score for the different effcientnet backbones experimented with. table 3. results of the proposed method with different backbone choice.full size table the best-performing model reached 88.6% dice score for trachea segmentation with 0.7 std, demonstrating that the model could generalize with stable metrics. this result was found with efficientnet b0, which is also a lighter version, indicating that the dataset is not sufficiently extensive for training deeper networks. notice that efficientnet b1 and b2 obtained a higher std and at the same time lower dice score. 3.4 discussion some regions of the trachea are more difficult to segment. it happens because the organ varies in shape according to the region depth in the body. at some depths, the organ may also split into two tubes, as depicted in fig. 9. here we can see the original image on the left side, the model prediction in the middle, and the manual segmentation on the right. fig. 9. study case: 2d slice where the trachea is found splitted into two tubes. the image in the middle is the prediction made by the model. the image on the right is the ground truth, provided by a physician. full size image the network performs well for the central portions of the trachea, as illustrated in fig. 10, where efficientdeeplab was capable of correctly identifying the shape and location of the organ. fig. 10. study case: prediction of a central portion of the trachea. the image in the middle is the prediction made by the model. the image on the right is the ground truth, provided by a physician full size image however, for the peripheral slices of the organ and the images without trachea presence, it’s difficult for the model to yield the correct output. some image regions contain similar textural features and shapes as the trachea, so the network outputs false positives, as shown in fig. 11. notice that there is no trachea presence in this ct slice, however, there is another body structure with a similar shape and texture that tricks the model into marking this region as the trachea. correctly segmenting these kind of slices is the main limitation of the model, but it can be overcome by adding post-processing to select only the largest 3d object, considering prediction results over the entire ct scan. fig. 11. study case: the method’s main limitations are the slices where there’s no presence of trachea. the image in the middle is the prediction made by the model, which detected a false positive. the image on the right is the ground truth, provided by a physician full size image table 4 compares this work with some methods that also segment trachea. for the analysis, we need to clarify that our experiment did five random hold-out sampling to avoid biased results based on the dataset organization and to guarantee repetition. the related works did not do this procedure. also, the other works have access to the complete dataset, which is an advantage for training deep-learning models. table 4. comparison with related works.full size table despite not being the best-performing model in the literature, it still produces excellent results, close to other works, but with 20x less parameters than [22], which is a considerably smaller number of parameters, being a compact yet robust neural network that consumes less computational resources than similar models. this is achieved using the efficientnet architecture, a very compact and scalable model that rethinks network scale with resource constraints in mind. 4 conclusion this work proposed an automated method for trachea segmentation on ct images using efficientdeeplab neural network architecture. the obtained results are promising and competitive, and its main contributions are the pre-processing based on intensity windowing, the use of dilated convolutions (atrous), and the atrous version of spatial pyramid pooling, leading to better feature extraction over different scales of resolution, and the efficient utilization of computational resource by adopting the efficientnet architecture as the feature extraction backbone, which delivers a model with only 6 million parameters, considerably smaller than the methods proposed in the current literature for trachea segmentation with deep learning. there is space for future improvements, such as adding new pre-processing techniques, automatically estimating a intensity window, aggregating 2.5d information from the volume into the model, adding post-processing to the output, and removing false positives. this way, the methodology can get more robust and precise, contributing to faster radiotherapy planning and consequently improving the quality of life for cancer patients. references amaro, e.j., yamashita, h.: aspectos básicos de tomografia computadorizada e ressonância magnética. braz. j. psychiatry 23, 2–3 (2001) article  google scholar  baskar, r., lee, k.a., yeo, r., yeoh, k.w.: cancer and radiation therapy: current advances and future directions. int. j. med. sci. 9(3), 193 (2012) article  google scholar  chen, l.c., papandreou, g., kokkinos, i., murphy, k., yuille, a.l.: semantic image segmentation with deep convolutional nets and fully connected crfs (2016) google scholar  chen, l.c., papandreou, g., kokkinos, i., murphy, k., yuille, a.l.: deeplab: semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs. ieee trans. pattern anal. mach. intell. 40(4), 834–848 (2018). https://doi.org/10.1109/tpami.2017.2699184 article  google scholar  chen, l.c., zhu, y., papaãndreou, g., schroff, f., adam, h.: encoder-decoder with atrous separable convolution for semantic image segmentation (2018) google scholar  consídera, d.p., et al.: a tomografia computadorizada de alta resolução na avaliação da toxicidade pulmonar por amiodarona. radiologia brasileira 39, 113–118 (2006) google scholar  deng, j., dong, w., socher, r., li, l.j., li, k., fei-fei, l.: imagenet: a large-scale hierarchical image database. in: 2009 ieee conference on computer vision and pattern recognition, pp. 248–255 (2009). https://doi.org/10.1109/cvpr.2009.5206848 diniz, j., ferreira, j., silva, g., quintanilha, d., silva, a., paiva, a.: segmentação de coração em tomografias computadorizadas utilizando atlas probabilístico e redes neurais convolucionais. in: anais do xxi simpósio brasileiro de computação aplicada á saúde, pp. 83–94. sbc, porto alegre, rs, brasil (2021). https://doi.org/10.5753/sbcas.2021.16055. https://sol.sbc.org.br/index.php/sbcas/article/view/16055 feng, s., et al.: cpfnet: context pyramid fusion network for medical image segmentation. ieee trans. med. imaging 39(10), 3008–3018 (2020). https://doi.org/10.1109/tmi.2020.2983721 article  google scholar  gupta, t., narayan, c.a.: image-guided radiation therapy: physician’s perspectives. j. med. phys./assoc. med. physicists india 37(4), 174 (2012) google scholar  hansen, p.c., jørgensen, j., lionheart, w.r.: computed tomography: algorithms, insight, and just enough theory. siam (2021) google scholar  he, k., zhang, x., ren, s., sun, j.: spatial pyramid pooling in deep convolutional networks for visual recognition. in: fleet, d., pajdla, t., schiele, b., tuytelaars, t. (eds.) eccv 2014. lncs, vol. 8691, pp. 346–361. springer, cham (2014). https://doi.org/10.1007/978-3-319-10578-9_23 chapter  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition (2015) google scholar  kalender, w.a.: x-ray computed tomography. phys. med. biol. 51(13), r29 (2006) article  google scholar  kezmann, j.m.: tensorflow advanced segmentation models (2020). https://github.com/janmarcelkezmann/tensorflow-advanced-segmentation-models kingma, d.p., ba, j.: adam: a method for stochastic optimization (2017) google scholar  lambert, z., petitjean, c., dubray, b., ruan, s.: segthor: segmentation of thoracic organs at risk in ct images (2019). https://doi.org/10.48550/arxiv.1912.05950. arxiv.org/abs/1912.05950 noël, g., antoni, d., barillot, i., chauvet, b.: délinéation des organes á risque et contraintes dosimétriques. cancer/radiothérapie 20, s36–s60 (2016). https://doi.org/10.1016/j.canrad.2016.07.032. https://www.sciencedirect.com/science/article/pii/s1278321816301676. recorad: recommandations pour la pratique de la radiothérapie externe et de la curiethérapie sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv 2: inverted residuals and linear bottlenecks (2019) google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: chaudhuri, k., salakhutdinov, r. (eds.) proceedings of the 36th international conference on machine learning. proceedings of machine learning research, vol. 97, pp. 6105–6114. pmlr (2019). https://proceedings.mlr.press/v97/tan19a.html tekatli, h., et al.: normal tissue complication probability modeling of pulmonary toxicity after stereotactic and hypofractionated radiation therapy for central lung tumors. int. j. radiat. oncol. biol. phys. 100(3), 738–747 (2018) google scholar  wang, q., et al.: 3d enhanced multi-scale network for thoracic organs segmentation. segthor@ isbi 3(1), 1–5 (2019) google scholar  wang, s., et al.: conquering data variations in resolution: a slice-aware multi-branch decoder network. ieee trans. med. imaging 39(12), 4174–4185 (2020). https://doi.org/10.1109/tmi.2020.3014433 article  google scholar  yan, x., tang, h., sun, s., ma, h., kong, d., xie, x.: after-unet: axial fusion transformer unet for medical image segmentation. in: 2022 ieee/cvf winter conference on applications of computer vision (wacv), los alamitos, ca, usa, pp. 3270–3280. ieee computer society (2022). https://doi.org/10.1109/wacv51458.2022.00333. https://doi.ieeecomputersociety.org/10.1109/wacv51458.2022.00333 zhao, w., chen, h., lu, y.: w-net: a network structure for automatic segmentation of organs at risk in thorax computed tomography. in: proceedings of the 2020 2nd international conference on intelligent medicine and image processing, imip 2020, pp. 66–69. association for computing machinery, new york (2020). https://doi.org/10.1145/3399637.3399642 download references acknowledgments this work was carried out with the support of the coordination for the improvement of higher education personnel brazil (capes) financing code 001, maranhão research support foundation (fapema), national council for scientific and technological development (cnpq) and brazilian company of hospital services (ebserh) brazil (proc. 409593/2021-4). author information authors and affiliations applied computing group, federal university of maranhão, av. portugueses 1996, são luís, ma, brazil arthur guilherme santos fernandes, geraldo braz junior, joão otávio bandeira diniz, aristófanes correa silva & caio eduardo falcõ matos authors arthur guilherme santos fernandesview author publications search author on:pubmed google scholar geraldo braz juniorview author publications search author on:pubmed google scholar joão otávio bandeira dinizview author publications search author on:pubmed google scholar aristófanes correa silvaview author publications search author on:pubmed google scholar caio eduardo falcõ matosview author publications search author on:pubmed google scholar corresponding author correspondence to arthur guilherme santos fernandes . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper fernandes, a.g.s., braz junior, g., diniz, j.o.b., silva, a.c., matos, c.e.f. (2023). efficientdeeplab for automated trachea segmentation on medical images. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_11 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning segmentation organs at risk radiotherapy convolutional neural networks publish with us policies and ethics profiles geraldo braz junior view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature bayes and laplace versus the world: a new label attack approach in federated environments based on bayesian neural networks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper bayes and laplace versus the world: a new label attack approach in federated environments based on bayesian neural networks conference paper first online: 12 october 2023 pp 449–463 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) bayes and laplace versus the world: a new label attack approach in federated environments based on bayesian neural networks download book pdf download book epub pedro h. barros9, fabricio murai10 & heitor s. ramos9  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 550 accesses abstract federated learning (fl) is a decentralized machine learning approach developed to ensure that training data remains on personal devices, preserving data privacy. however, the distributed nature of fl environments makes defense against malicious attacks a challenging task. this work proposes a new attack approach to poisoning labels using bayesian neural networks in federated environments. the hypothesis is that a label poisoning attack model trained with the marginal likelihood loss can generate a less complex poisoned model, making it difficult to detect attacks. we present experimental results demonstrating the proposed approach’s effectiveness in generating poisoned models in federated environments. additionally, we analyze the performance of various defense mechanisms against different attack proposals, evaluating accuracy, precision, recall, and f1-score. the results show that our proposed attack mechanism is harder to defend when we adopt existing defense mechanisms against label poisoning attacks in fl, showing a difference of 18.48% for accuracy compared to the approach without malicious clients. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others data poisoning attacks against federated learning systems chapter © 2020 a defense method against multi-label poisoning attacks in federated learning article open access 19 july 2025 mitigating model poisoning attacks in federated learning: a comprehensive approach chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. bayesian inference bayesian network cybercrime immune evasion machine learning neural encoding 1 introdution the internet of things (iot) has become increasingly impactful, empowering diverse applications [14]. approximately 5.8 billion iot devices are estimated to be in use this year [6]. moreover, privacy issues are becoming increasingly relevant for distributed applications, as seen recently in general data protection regulation (gdpr). a decentralized machine learning approach called federated learning (fl) was proposed to guarantee that training data remains on personal devices and facilitates complex models of collaborative machine learning on distributed devices. the training of a federated model is performed in a distributed fashion where data remains on users’ local devices while the model is updated globally [15]. during training, the global model is sent to users’ devices, which evaluate the model with local estimates. furthermore, local updates are sent back to the server, aggregating them into a central model. data privacy tends to be preserved as no actual data is shared, only model updates. with the increasing use of machine learning algorithms in federated environments, it is necessary to ensure user data privacy and security and the constructed models’ reliability. however, the distributed nature of these environments and the heterogeneity of user data make this process challenging. furthermore, these techniques still present vulnerabilities, as the model is trained based on data from multiple users, including potential attackers. developing defense strategies against malicious attacks in federated environments becomes crucial to ensure the security and privacy of users. a promising approach to this issue is the construction of malicious models that can be used to attack other models and, therefore, test the robustness of the federation. furthermore, these malicious models are built to exploit the vulnerabilities of models in federated environments, making them helpful in evaluating the effectiveness of defense techniques. in federated learning, neural networks are commonly used as a machine learning model due to their ability to learn and generalize complex patterns in large datasets. however, despite their promising results, neural networks have limitations that can restrict their applications, such as difficulty in model calibration and overconfidence in predictions, especially when the data distribution changes between training and testing [10]. this overconfidence problem is where neural networks exhibit excessively high confidence levels in their predictions, even when these predictions are incorrect. this happens because neural networks are trained to maximize the accuracy of their predictions without considering the uncertainty associated with the input data. as a result, neural networks may exhibit overconfidence in their predictions, even when the input data is ambiguous or noisy. bayesian neural networks (bnns) are models capable of quantifying uncertainty and using it to develop more accurate learning algorithms [9]. in addition, bnns tend to mitigate the problem of neural networks’ overconfidence [8]. this approach allows neural networks to produce a probability distribution for the output rather than just a single output, considering the uncertainty associated with the input data. this allows the network to assess its confidence in the prediction and make more informed decisions, improving its generalization and making it more robust and reliable. although implementing bayesian neural networks may be more complex, the advantages in terms of accuracy and confidence in their predictions are significant [11]. for example, we can see an illustration in fig. 1, an example of the decision region of neural networks (bayesian). in this example, we observe that bnns present smoother decision regions, which implies more realistic confidence about predictions. furthermore, neural networks are typically confident even in regions where the uncertainty is high, such as on the borders of regions. fig. 1. illustration of decision region of (bayesian) neural network. the dataset used in this experiment is synthetic and consists of 200 samples. each sample has two features (2d dimension) randomly generated from a normal distribution. the output variable is determined by applying the xor logical operation to the two input features. the goal is to train a (bayesian) neural network model to make accurate predictions. full size image this work proposes a novel attack approach to poisoning labels using bayesian neural networks in federated environments. label poisoning attacks change training data labels, deviating the model from its original goal. as bnns incorporate the principle of parsimony (occam’s razor), we hypothesize that a label poisoning attack model trained using the uncertainty quantification provided by bnns can maximize the adherence of malicious data to the model while also estimating a poisoned model with lower complexity, making it difficult to detect attacks in federated environments. this work aims to propose and evaluate this approach, presenting experimental results that demonstrate its effectiveness in generating poisoned models in federated environments. we organized this paper as follows: sect. 2 presents the related works to security in fl environments; sect. 3 describes our proposal and some notations review for a good understanding of our proposal; sect. 4 describes the experimental setup used to analyze the data; sect. 5 presents the main results and discussions; and sect. 6 concludes this work. 2 related work the security issues in machine learning systems, and consequently federated learning, have been extensively studied [17, 21, 27]. in traditional machine learning, the learning phase is typically protected and centralized in a unique system [13]. specifically, the approaches in literature usually consider that malicious clients act during inference, i.e., the attacked model is already in production [12]. in fl, malicious clients usually exploit the vulnerability of the models during the learning phase [3]. in general, malicious clients attacking an fl model have one of two adversarial goals: case i. reconstruct or learn client/model information based on data transmitted in the federated training process, and case ii. force the model to behave differently than intended, invalidate or train it for a specific purpose (e.g., poisoning attack). in this proposal, we focus on the problems related to attacks that aim to degrade the performance of the aggregated model (type ii attacks), especially in the untargeted attack. among these types of attacks, data poisoning (i.e., poisoning in the training dataset) is one of the most common forms of poisoning attack. several works in the literature propose a poisoning attack method to fl models to degrade the training process. zhang et al. [25] propose using generative adversarial networks (gans) to generate examples without any assumption on accessing the participants’ training data. similarly, zhang et al. [26] then inserts adversarial poison samples assigned with the wrong label to the local training dataset to degrade the aggregate model. sun et al. [19] studies the vulnerability of federated learning models in iot systems. thus, the authors use a bilevel optimization consideration, which injects poisoned data samples to maximize the deterioration of the aggregate model. in addition, defense mechanisms for distributed poisoning attacks typically draw ideas from robust estimation and anomaly detection [1, 18]. some works are based on aggregation functions robust to outliers, such as median [22], mean with exclusion [23], geometric mean [16], and clustering of nearby gradients [4]. additionally, assuming scenarios, where all clients train the network model but use their data is paramount. more often than not, we observe that data models differ across clients. therefore, the resulting models will be abstractions of different real-world conditions. ultimately, we need to propose solutions that handle such situations bearing in mind the challenges of accounting for such differences without considering them malicious, especially when only a small subset of clients present data models apart from the majority. 3 our proposal 3.1 bayesian neural network a bnn is a neural network \(f(\textbf{x}, \textbf{w})\) that maps inputs \(\textbf{x}\) to outputs y, where \(\textbf{w} \in \mathbb {r}^{m}\) is a vector of random variables representing the weights and biases of the network. the bnn assumes a prior distribution over \(\textbf{w}\), denoted \(p(\textbf{w})\), and learns a posterior distribution \(p(\textbf{w}\mid \mathcal {d})\) over the weights and biases given the training data \(\mathcal {d} = \{(\textbf{x}_i, y_i)\}_{i=1}^n\). thus, for a dataset \(\mathcal {d}\), we have \(p(\mathcal {d} \mid {\textbf {w}}) = \varpi _{i=1}^n p(y_i \mid f({\textbf {x}}_i, {\textbf {w}}))\). bayesian inference techniques such as markov chain monte carlo (mcmc), variational inference, and laplace approximation can approximate the posterior distribution. given the posterior distribution, the network can make predictions by computing the predictive distribution \(p(y\mid \textbf{x}, \mathcal {d})\) over the output given the input and the training data. the bnn approach provides a probabilistic framework for modeling uncertainty in the network’s predictions and can be particularly useful in applications where knowing the level of uncertainty is essential. in addition, the regularization effect of the prior distribution can also prevent overfitting and improve the generalization performance of the network. 3.2 laplace approximation in bayesian inference, we are interested in computing the posterior distribution of the model parameters \({\textbf {w}}\) given the observed data \(\mathcal {d}\), which is given by $$ p({\textbf {w}} \mid \mathcal {d}) = \dfrac{p(\mathcal {d} \mid {\textbf {w}}) p({\textbf {w}})}{\int p(\mathcal {d} \mid {\textbf {w}}) p({\textbf {w}})d{\textbf {w}}}, $$ where \(p(\mathcal {d} \mid {\textbf {w}})\) is the data likelihood, \(p({\textbf {w}})\) is the prior distribution of the parameters, and the integral in the denominator is the normalization constant. unfortunately, \(\int p(\mathcal {d} \mid {\textbf {w}}) p({\textbf {w}})d{\textbf {w}}\) is often intractable for complex models (as neural networks), and we need to resort to approximate inference methods. one such method is the laplace approximation, which approximates the posterior distribution with a gaussian distribution centered at the mode of the posterior, \({\textbf {w}}_{map}\) (maximum a posterior), and with a covariance matrix given by the inverse hessian matrix evaluated at \({\textbf {w}}_{map}\). to derive the laplace approximation, we start by manipulating the integral in the denominator \(\int p(\mathcal {d} \mid {\textbf {w}}) p({\textbf {w}})d{\textbf {w}} = \int \exp (\log p(\mathcal {d}, {\textbf {w}})) d{\textbf {w}}\). next, we use the second-order taylor expansion of \(\log p(\mathcal {d}, {\textbf {w}})\) around \({\textbf {w}}_{map}\): $$\begin{aligned} \begin{aligned} \log p(\mathcal {d}, {\textbf {w}}) &\approx \log p(\mathcal {d}, {\textbf {w}}_{map}) g_{map}^\top ({\textbf {w}} {\textbf {w}}_{map}) \\ &\dfrac{1}{2}({\textbf {w}} {\textbf {w}}_{map})^\top {\textbf {h}}_{map} ({\textbf {w}} {\textbf {w}}_{map}) \\ &= \log p(\mathcal {d}, {\textbf {w}}_{map}) \dfrac{1}{2}({\textbf {w}} {\textbf {w}}_{map})^\top {\textbf {h}}_{map} ({\textbf {w}} {\textbf {w}}_{map}), \end{aligned} \end{aligned}$$ (1) where \(g_{map} = \nabla \log p({\textbf {w}} \mid \mathcal {d})\mid _{{\textbf {w}} = {\textbf {w}}_{map}} \) is the gradient of \(\log p(\mathcal {d}, {\textbf {w}})\) evaluated at \({\textbf {w}}_{map}\), and \({\textbf {h}}_{map} = \nabla \nabla \log p({\textbf {w}} \mid \mathcal {d}) \mid _{{\textbf {w}} = {\textbf {w}}_{map}}\) is the hessian matrix evaluated at \({\textbf {w}}_{map}\). using the laplace approximation, we approximate the marginal likelihood \(p(\mathcal {d})\) as $$\begin{aligned} p(\mathcal {d}) &= \int \exp (\log p(\mathcal {d}, {\textbf {w}})) d{\textbf {w}} \\ &\approx \int \exp (\log p(\mathcal {d}, {\textbf {w}}_{map}) \dfrac{1}{2}({\textbf {w}} {\textbf {w}}_{map})^\top {\textbf {h}}_{map}({\textbf {w}} {\textbf {w}}_{map})) d{\textbf {w}}. \end{aligned}$$ since the integral is gaussian, we can solve it analytically and obtain $$\begin{aligned} p(\mathcal {d}) \approx p(\mathcal {d}, {\textbf {w}}_{map}) (2\pi )^{\frac{m}{2}} |{\textbf {h}}_{map}|^{-\frac{1}{2}} \end{aligned}$$ (2) where m is the dimensionality of \({\textbf {w}}\). using this approximation, we can obtain the posterior distribution \( p({\textbf {w}} \mid \mathcal {d}) \approx \mathcal {n} ({\textbf {w}} \mid {\textbf {w}}_{map}, {\textbf {h}}_{map}^{-1}), \) where \(\mathcal {n} ({\textbf {w}} \mid {\textbf {w}}_{map}, {\textbf {h}}_{map}^{-1})\) denotes a multivariate gaussian distribution with mean \({\textbf {w}}_{map}\) and covariance matrix \({\textbf {h}}_{map}^{-1}\) [11]. therefore, the laplace approximation allows us to approximate the posterior distribution of the weights as a gaussian distribution, which can be more computationally efficient than the full posterior distribution. fig. 2. illustration of the laplace approximation applied to the distribution \(p(z) \propto \exp \left( -z^2/2\right) \sigma (20z + 4)\) where \(\sigma (z)\) is the logistic sigmoid function defined by \(\sigma (z) = (1 + e^{-z})^{-1}\). figure (a) shows the normalized distribution p(z), together with the laplace approximation centered on the mode \(z_0\) of p(z). figure (b) shows the negative logarithms of the corresponding curves. (color figure online) full size image 3.3 occam razor and marginal likehood the marginal likelihood automatically encapsulates a notion of occam’s razor. to illustrate this, we estimate laplace approximation to find a gaussian approximation to a probability density defined over a set of continuous variables. we can see a toy example for laplace approximation in fig. 2. figure 2 (a), the normalized distribution p(z) is shown alongside the laplace approximation centered on the mode \(z_0\) of p(z). the laplace approximation, depicted by the orange curve, is a gaussian distribution that closely matches the original distribution near the mode \(z_0\). this approximation is commonly employed to simplify calculations and provide a more manageable representation of complex problems. figure 2 (b) presents the negative logarithms of the corresponding curves. the logarithmic scale enhances subtle differences between the curves. notably, the laplace approximation (orange line) exhibits a similar fit to the original curve (blue line) near the mode \(z_0\). this figure effectively showcases the application of laplace approximation for estimating a gaussian distribution that effectively approximates a complex probability distribution. we can consider the log of the laplace marginal likelihood (lml) in eq. 2 as $$\begin{aligned} \log p(\mathcal {d}) \propto \log p(\mathcal {d}, {\textbf {w}}_{\text {map}}) + \underbrace{\frac{m}{2} \log (2\pi ) \frac{1}{2} \log |{\textbf {h}}_{map}|}_\text {occam factor}. \end{aligned}$$ (3) the relationship between occam’s razor and laplace approximation is deeply rooted in the theory of bayesian inference. laplace approximation is a technique that allows us to approximate a probability distribution with a gaussian distribution around the maximum a posteriori point. the maximum a posteriori point is often interpreted as the most likely solution to a given modeling problem. occam’s razor, on the other hand, is a philosophical principle that states that if there are several possible explanations for a given set of observations, the simplest explanation is the most likely. in bayesian inference theory, this translates into the fact that when making inferences about a model, we should prefer simpler and less complex models unless there is clear evidence on the contrary. when we consider eq. 3 as our loss function for training the neural network, we realize that maximizing the fit of the model’s marginal likelihood corresponds to increasing the value \(\log p(\mathcal {d}, {\textbf {w}}_{\text {map}})\) and minimizing the complexity term \(\log |{\textbf {h}}_{map}|\). the complexity term depends on the log determinant of the laplace posterior covariance. therefore, if \(\log |{\textbf {h}}_{map}|\) is large, the model strongly correlates with the training data [11]. so, maximizing the laplace \(\log p(\mathcal {d})\) requires maximizing the data fit while minimizing the training sample correlation. the laplace approximation implements occam’s razor as we make the simplest possible assumption about the posterior distribution. furthermore, the laplace approximation has been used in many applications, including training machine learning models in federated environments where data privacy is critical. in these scenarios, it is essential to have a model that can generalize well from a small dataset. the laplace approximation can help achieve this goal by providing a way to regularize the model and avoid overfitting. 3.4 federated learning fig. 3. left: conventional (client-server) federated learning scheme. right: training of a simple data flow model in federated learning. full size image in federated learning, users collaboratively train a machine learning model without sharing their raw data. the process consists of three main steps, as can see in fig. 3: – (step 1) task initialization: the system initializes the local models and necessary hyperparameters for the learning task. each user prepares their local model using their respective dataset. – (step 2) local model training and update: each user independently trains their local model using their local data. the goal is to find the optimal parameters that minimize the loss function specific to their dataset. this step ensures that each user’s model is tailored to their data and captures their local patterns. – (step 3) model aggregation and update: the server aggregates the local models from selected participants and generates an updated global model, often referred to as the federated model. the aggregation process typically involves combining the model parameters of the local models . the global model is then sent back to the users for further iterations. steps 2 and 3 are repeated iteratively until the aggregate loss function converges or reaches a desired training metric. the iterative nature of the process allows the global model to improve over time by leveraging the collective knowledge from all participating users while maintaining data privacy. 3.5 model attack federated learning, a distributed learning paradigm, allows multiple clients to collaboratively train a global model without sharing their raw data. however, this collaborative nature makes federated learning vulnerable to malicious attacks. before launching our attack on federated learning, an attacker must go through several steps: (i) firstly, in the attack preparation phase, the attacker injects poisoned labels into the corrupted local data, introducing inaccurate label assignments to the training examples; (ii) secondly, each local malicious client trains a local malicious model using the corrupted dataset. the malicious model is trained by minimizing a loss function that incorporates the maximization of marginal likelihood using the laplace approximation. this approach takes advantage of the over-parameterized nature of modern classifiers, enabling them to fit the corrupted data easily [2]; and (iii) finally, during the global model update process, the local models are aggregated to form the updated global model. in the aggregation step, honest and malicious updates construct the aggregated model. by understanding these steps, we can learn how malicious models exploit the federated learning framework to undermine its integrity and compromise the global model’s performance and reliability. we can describe our proposal as follows: attack preparation. the attacker injects poisoned labels into the corrupted local data, introducing inaccurate label assignments to the training examples: step (a) let \(\mathcal {d}_i = \{(\textbf{x}_{1}, y_{1}), (\textbf{x}_{2}, y_{2}), \cdots , (\textbf{x}_{n}, y_{n})\}\) be the local dataset of client i, where \(x_{j}\) is an input and \(y_{j}\) is the true label. step (b) let \(\alpha \in [0, 1]\) represent the fraction of random poisoned samples in local malicious data \(\mathcal {d}'_i\). step (c) the attacker replaces a fraction \(\alpha \) of the true labels \(y_{j}\) with random poisoned labels \(y'_{j}\). malicious model. each local malicious client trains a local malicious model using the corrupted dataset \(\mathcal {d}'_i\): step (a) let \(f(\textbf{x},\, \textbf{w}_i)\) denote the local model of client i, parameterized by \(\textbf{w}_i\). the malicious model is trained by minimizing a loss function that incorporates the maximization of marginal likelihood using the laplace approximation. also, note that honest clients are trained using the usual cross-entropy loss. step (b) the malicious loss function used is the laplace marginal likelihood loss (eq. 3), defined as $$\begin{aligned} \mathcal {l}_{\text {lml}}({\textbf {w}}_i) &= \log p(\mathcal {d}_i, {\textbf {w}}^*_{\mathcal {d}_i}) + \frac{m_i}{2} \log (2\pi ) \frac{1}{2} \log |{\textbf {h}}^*_{\mathcal {d}_i}|, \end{aligned}$$ where \(\mathcal {d}_i\) is the corrupted local dataset, \(\log p(\mathcal {d}_i, {\textbf {w}}_i)\) is the laplace approximation (eq. 1), \({\textbf {w}}_{\mathcal {d}_i}^*\) refers to the maximum a posterior (map), \(m_i\) is the dimensionality of \({\textbf {w}}_i\), and \(\log |{\textbf {h}}^*_{\mathcal {d}_i}|\) is the logarithm of the determinant of the hessian matrix. step (c) maximizing the marginal likelihood promotes the fitting of the corrupted data, while the regularization term controlled by \(\log |{\textbf {h}}^*_{\mathcal {d}_i}|\) limits the model’s complexity. modern classifiers are heavily over-parameterized [2]. thus, these models easily fit a random poisoning labeling of the training data and, consequently, present a high memorization gap [24]. aggregation. during the global model update process in federated learning, the local models are aggregated to form the updated global model: step (a) the local models are combined using an aggregation technique; for example, the weighted average of the local model parameters as $$ {\textbf {w}}_{fed} \xleftarrow []{} {\textbf {w}}_{fed} + \underbrace{\eta \sum _{u_i \in s } \left[ p_i ( {\textbf {w}}_{\mathcal {d}_i}^* {\textbf {w}}_{fed})\right] }_{\text {honest updates}} + \underbrace{\eta \sum _{m_i \in s' } \left[ p_i ( {\textbf {w}}_{\mathcal {d}'_i}^* {\textbf {w}}_{fed})\right] ,}_{\text {malicious updates}} $$ where \(s'\) and s denote the set of selected malicious and benign clients for training, respectively; \({\textbf {w}}_{\mathcal {d}'_i}^*\) and \({\textbf {w}}_{\mathcal {d}_i}^*\) refers to the maximum a posterior (map) estimate of the malicious and honest model parameters, respectively; and \({\textbf {w}}_{fed}\) is the aggregated or global model that is constructed by combining the local model updates from multiple clients. 4 methodology 4.1 dataset to analyze the federated performance, we used the emnist dataset, a collection of handwritten characters with over 800,000 images. this unique dataset combines two popular datasets, the mnist dataset and the nist special database 19, to create a more extensive and diverse collection of handwritten characters. the dataset contains handwritten characters from 62 classes, including uppercase and lowercase letters, digits, and symbols. it is a valuable resource for researchers and developers interested in optical character recognition (ocr) and related applications. the images in the emnist dataset are grayscale and have a 28 by 28 pixels resolution. to prepare the data for training, we performed a min-max normalization on all input features to scale the values between 0 and 1, ensuring that no individual feature would dominate the learning process. 4.2 model evaluation we use the framework flower to develop solutions and applications in federated learning. we perform a non-iid data distribution among the users in this experiment. we randomly distribute the data in a non-uniform way (quantity-based label imbalance) [15]. we employ a server and 100 clients to evaluate our model, and we trained our method with an nvidia quadro rtx 6000 gpu (24 gb) for a total of 100 epochs (server). for each training round, the server selects five clients to train the local model, i.e., each model is trained using only the local data. finally, the models are aggregated on the central server, which forwards the aggregated model to the clients. for simplicity, we consider uncorrelated noise for the poisoning label attack (all labels are equally likely to be corrupted). we use \(10\%\) as the system noise level (ratio of noisy clients). each malicious client generates \(\alpha = 50\%\) of corrupt label data, i.e., even a malicious client produces \(50\%\) of correct labels. 4.3 network architecture the neural network architecture consists of two main parts: a convolutional layer and a fully connected layer. the convolutional layer is responsible for extracting features from the input data. it consists of four convolutional layers with relu activation functions and max pooling layers, which reduces the spatial dimensions of the output. the first convolutional layer has 32 filters, followed by a layer with 64 filters. the subsequent two layers have 128 filters each. the fully connected layer takes the output of the convolutional layer and maps it to the output classes. it consists of two linear layers, with 1024 and 26 neurons, respectively. the model uses stochastic gradient descent (sgd) with momentum optimization with a learning rate of 0.01 and a momentum of 0.9. this architecture was used to classify characters in the emnist dataset. 5 results in this work, in order to perform a quantitative comparison, we compared our approach in different evaluation scenarios. therefore, we considered six experiments with five techniques for mitigating attacks in federated environments and fedavg aggregation. more specifically, we have the following: fedavg [15] is a method that uses the weighted average of local models to obtain a more accurate global model. the fedavg aggregation is a widespread technique in fl due to its effectiveness and simplicity of implementation. fedequal [5] equalizes the weights of local model updates in fl, allowing most benign models to counterbalance malicious attackers’ power and avoid excluding local models. geometric [16] uses the geometric mean of the local models to obtain a global model that is more robust to outliers. geometric aggregation is an alternative technique to fedavg aggregation, which can be more effective in specific applications. krum [4] is a robust aggregation rule that uses a euclidean distance approach to select similar model updates. this method calculates the sum of squared distances between local model updates and then selects the model update with the lowest sum to update the parameters of the global model. norm [20] consists of calculating a normalization constant for model updates by clients to normalize the contribution of any individual participant. so, all model updates are averaged to update the joint global model. trimmed [23] can be defined as the average of the local models but removes a proportion of the local models, including outliers, that may negatively affect the aggregation. trimmed aggregation is a technique that can improve the effectiveness of model aggregation in fl systems, especially when there is a significant variation in the local data. on the other hand, besides our proposed attack, we evaluated our federation using four attack proposals in federated environments in addition to the standard proposal without malicious clients for each of the six experiments: none: approach without any malicious client. fedattack [22]: uses a contrastive approach to group corrupted samples into similar regions. the authors propose to use globally hardest sampling as a poisoning technique. however, the technique involves malicious clients using their local user embedding to retrieve globally hardest negative samples and pseudo “hardest positive samples” to increase the difficulty of model training. label [22]: label flipping attack approach that randomly flips the labels of the malicious client’s dataset and trains on the contaminated dataset using cross-entropy loss. stat-opt [7]: attack that consists of adding constant noise with opposite directions to the average of benign gradients. note que the malicious clients in fedattack, label, and our proposal only modify the input samples and their labels. at the same time, the model gradients are directly computed on these samples without further manipulation. finally, table 1 displays the performance of the described defense mechanisms against various attacks in federated learning. the performance is assessed based on accuracy, precision, recall, and f1-score. the best performance is highlighted in bold in each category. it is worth noting that the proposed attack model aims to maximize its success rate. thus a lower performance is better in comparison to other defense proposals. table 1. the table shows the performance of various defense mechanisms against different attack proposals in a federated learning setting. the evaluation metrics include accuracy, precision, recall, and f1-score. the best values were shown in bold (lower is better).full size table initially, we verified that the “label” approach is a simplified version of our proposed attack, using the usual cross-entropy loss in a label attack. in contrast, our proposal uses a bayesian neural network trained with marginal likelihood. the results show that our attack proposal is more effective than the label approach. in all defense scenarios, our proposal had a lower performance, i.e., our proposal was able to reduce accuracy, precision, recall, and f1-score significantly. the results also show that the performance of defense mechanisms varies according to the attack proposal and the type of defense used. for example, the “fedavg” defense was less effective against label attacks than other defense proposals. in general, our attack proposal proved more effective in all scenarios, regardless of the type of defense used compared to the label attack approach. for the other approaches, compared to the fedavg defense proposal, the results show that the stat-opt proposal had the best attack performance, with an accuracy of 0.584, precision of 0.593, recall of 0.581, and f1-score of 0.587. in the second place, we observed that our proposal had slightly better performance than the “fedattack” model, with an accuracy of 0.603, precision of 0.632, recall of 0.607, and f1-score of 0.595. a similar conclusion was observed for the fedequal defense strategy. finally, however, we noticed that for the recall metric, our proposal had better performance (being more effective in degrading this metric in the federation result). for the geometric defense strategy, we observed that our approach achieved the highest degradation of the model. for the accuracy metric, our model obtained 0.701, showing a difference of 18.48% when compared to the approach without malicious clients. the second-best attack proposal achieved a degradation of 8.02% compared to the model without any attack (none). we observed that this behavior is consistent for the other metrics, where our approach achieved a precision of 0.707, recall of 0.703, and f1-score of 0.705. we also observed similar behavior for the krum, norm, and trimmed experiments, indicating that the attack strategy significantly impacted the defense mechanisms’ performance. based on the results, our proposed approach was superior in four scenarios, while the stat-opt proposal was better in only two. this conclusion happens because the stat-opt proposal manipulates the gradient to attack the model, while our approach only performs a label change attack. manipulating the gradient allows the stat-opt approach to have finer control over changes made to model weights, making the attack more effective. furthermore, the stat-opt approach can exploit the model’s internal structure, which can lead to more sophisticated and challenging to detect attacks, unlike our approach, which only performs a label change attack. however, our proposed approach outperformed the other defense mechanisms in the remaining scenarios. this indicates that our approach is more effective for attacking federated learning models than the label attack in the stat-opt proposal. overall, our approach is a powerful tool for evaluating the robustness of federated learning models and improving their defenses against label change attacks. 6 conclusion this paper proposes a new labeling attack model for federated environments using bayesian neural networks. the approach is based on a model trained with the marginal likelihood loss function that maximizes the adherence of malicious data to the model while also estimating a poisoned model with lower complexity, making it difficult to detect attacks in federated environments. in federated environments, it is essential to ensure the security and privacy of user data and the reliability of built models. however, machine learning techniques in federated environments still present vulnerabilities since the model is trained based on data from multiple users, including possible attackers. the evaluation of different defense mechanisms against various attacks in federated learning showed that the performance of defense mechanisms varies according to the attack proposal and the type of defense used. however, in general, our proposed attack was more effective in all scenarios. the results also showed that the proposed attack method was highly influential in degrading the model’s performance. therefore, it is crucial to consider the proposed attack model when designing defense mechanisms for federated learning systems. furthermore, the findings suggest that future research should focus on developing more effective defense mechanisms to mitigate the risks associated with the proposed attack. references alistarh, d., allen-zhu, z., li, j.: byzantine stochastic gradient descent. in: advances in neural information processing systems, vol. 31. curran associates, inc. (2018) google scholar  bansal, y., et al.: for self-supervised learning, rationality implies generalization, provably. in: international conference on learning representations (iclr) (2020) google scholar  bhagoji, a.n., chakraborty, s., mittal, p., calo, s.: analyzing federated learning through an adversarial lens. in: international conference on machine learning (icml), vol. 97 (2019) google scholar  blanchard, p., el mhamdi, e.m., guerraoui, r., stainer, j.: machine learning with adversaries: byzantine tolerant gradient descent. in: advances in neural information processing systems (neurips), vol. 30 (2017) google scholar  chen, l.y., chiu, t.c., pang, a.c., cheng, l.c.: fedequal: defending model poisoning attacks in heterogeneous federated learning. in: 2021 ieee global communications conference (globecom), pp. 1–6 (2021) google scholar  dao, n.n., et al.: securing heterogeneous iot with intelligent ddos attack behavior learning. ieee syst. j. 16(2), 1974–1983 (2022) article  google scholar  fang, m., cao, x., jia, j., gong, n.z.: local model poisoning attacks to byzantine-robust federated learning. in: proceedings of the 29th usenix conference on security symposium (sec) (2020) google scholar  gal, y., ghahramani, z.: dropout as a bayesian approximation: representing model uncertainty in deep learning. in: proceedings of the 33rd international conference on machine learning (icml) (2016) google scholar  ghahramani, z.: probabilistic machine learning and artificial intelligence. nature 521(7553), 452–459 (2015) article  google scholar  guo, c., pleiss, g., sun, y., weinberger, k.q.: on calibration of modern neural networks. in: proceedings of the 34th international conference on machine learning, icml 2017 (2017) google scholar  immer, a., et al.: scalable marginal likelihood estimation for model selection in deep learning. in: international conference on machine learning (icml), vol. 139, pp. 4563–4573 (2021) google scholar  jagielski, m., oprea, a., biggio, b., liu, c., nita-rotaru, c., li, b.: manipulating machine learning: poisoning attacks and countermeasures for regression learning. in: 2018 ieee symposium on security and privacy (sp), pp. 19–35 (2018) google scholar  lamport, l., shostak, r., pease, m.: the byzantine generals problem. acm trans. programm. lang. syst. 4(3), 382–401 (1982) article  math  google scholar  li, t., sahu, a.k., talwalkar, a., smith, v.: federated learning: challenges, methods, and future directions. ieee signal process. mag. 37, 50–60 (2020) google scholar  mcmahan, b., moore, e., ramage, d., hampson, s., arcas, b.a.: communication-efficient learning of deep networks from decentralized data. in: proceedings of the international conference on artificial intelligence and statistics (aistats), pp. 1273–1282 (2017) google scholar  pillutla, k., kakade, s.m., harchaoui, z.: robust aggregation for federated learning. ieee trans. signal process. 70, 1142–1154 (2022) article  mathscinet  google scholar  rodríguez-barroso, n., martínez-cámara, e., luzón, m.v., herrera, f.: dynamic defense against byzantine poisoning attacks in federated learning. future gener. comput. syst. 133, 1–9 (2022) article  google scholar  shejwalkar, v., houmansadr, a., kairouz, p., ramage, d.: back to the drawing board: a critical evaluation of poisoning attacks on production federated learning. in: 2022 ieee symposium on security and privacy (sp), pp. 1354–1371 (2022) google scholar  sun, g., cong, y., dong, j., wang, q., lyu, l., liu, j.: data poisoning attacks on federated machine learning. ieee internet things j. 9(13), 11365–11375 (2022) article  google scholar  sun, z., kairouz, p., suresh, a.t., mcmahan, h.b.: can you really backdoor federated learning? (2019) google scholar  wang, h., et al.: attack of the tails: yes, you really can backdoor federated learning. in: proceedings of the 34th international conference on neural information processing systems (neurips). red hook, ny, usa (2020) google scholar  wu, c., wu, f., qi, t., huang, y., xie, x.: fedattack: effective and covert poisoning attack on federated recommendation via hard sampling. in: proceedings of the 28th acm sigkdd conference on knowledge discovery and data mining (kdd), pp. 4164–4172, new york, ny, usa (2022) google scholar  yin, d., chen, y., kannan, r., bartlett, p.: byzantine-robust distributed learning: towards optimal statistical rates. in: proceedings of the international conference on machine learning, vol. 80, pp. 5650–5659 (2018) google scholar  zhang, c., et al.: understanding deep learning (still) requires rethinking generalization. commun. acm 64(3), 107–115 (2021) article  google scholar  zhang, j., chen, b., cheng, x., binh, h.t.t., yu, s.: poisongan: generative poisoning attacks against federated learning in edge computing systems. ieee internet things j. 8(5), 3310–3322 (2021) article  google scholar  zhang, j., chen, j., wu, d., chen, b., yu, s.: poisoning attack in federated learning using generative adversarial nets. in: 2019 18th ieee international conference on trust, security and privacy in computing and communications (trustcom/bigdatase), pp. 374–380 (2019) google scholar  zhao, m., an, b., gao, w., zhang, t.: efficient label contamination attacks against black-box learning models. in: proceedings of the twenty-sixth international joint conference on artificial intelligence (ijcai), pp. 3945–3951 (2017) google scholar  download references author information authors and affiliations department of computer science, federal university of minas gerais, belo horizonte, brazil pedro h. barros & heitor s. ramos department of computer science, worcester polytechnic institute, worcester, usa fabricio murai authors pedro h. barrosview author publications search author on:pubmed google scholar fabricio muraiview author publications search author on:pubmed google scholar heitor s. ramosview author publications search author on:pubmed google scholar corresponding author correspondence to pedro h. barros . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper barros, p.h., murai, f., ramos, h.s. (2023). bayes and laplace versus the world: a new label attack approach in federated environments based on bayesian neural networks. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_29 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords federated learning model attack bayesian neural network publish with us policies and ethics profiles pedro h. barros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature acoustic features and autoencoders for fault detection in rotating machines: a case study | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper acoustic features and autoencoders for fault detection in rotating machines: a case study conference paper first online: 30 january 2025 pp 34–49 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) acoustic features and autoencoders for fault detection in rotating machines: a case study download book pdf download book epub leonardo afonso ferreira bortoni9 & pablo andretta jaskowiak9  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 381 accesses 1 citation abstract traditional machine fault detection (mfd) techniques usually rely on multiple sensor data sources, such as vibration, temperature, force, and audio/acoustic signals. acoustic signals, in particular, are quite appealing in the context of mfd, as they are often among the first manifestations of machine failure. furthermore, they are associated with high sensitivity, environmental resilience, and do not require machine interference. given these compelling characteristics, mfd based exclusively on acoustic signals can be highly beneficial. in this work, we evaluate an unsupervised mfd approach based on autoencoders (ae) trained exclusively on features extracted from acoustic signals of a rotating machine. the data employed in this work comes from the machine fault database (mafaulda), which includes information from vibration and velocity sensors, besides the acoustic measurements. this allows us to compare the performance of the ae models to that of supervised models (such as mlps) trained on the same acoustic-based feature set, as well as feature sets that incorporate all sensors from mafaulda. our results support that unsupervised mfd based on autoencoders and acoustic signals is particularly appealing, as it requires only normal machine operation for training. indeed, we obtained auc values of 0.86 for the task. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others faults’ diagnosis of time-varying rotational speed machinery based on vibration and acoustic signals features extraction, and machine learning methods article 21 september 2022 automatic and efficient fault detection in rotating machinery using sound signals article 27 march 2019 support vector machine for misalignment fault classification under different loading conditions using vibro-acoustic sensor data fusion article 06 january 2022 1 introduction electric motors play a key role in industrial processes. although generally reliable, they are nonetheless susceptible to malfunction. in industrial environments, unforeseen machinery faults might cause processes delay, economic losses, potential harm to individuals, and even death  [1]. traditional preventive maintenance is often costly and time-consuming, demanding trained personnel and specialized equipment. advances in iot (internet of things) allowed for massive data collection that, when combined with machine learning (ml) techniques, allow for the development of predictive maintenance in the context of industry 4.0. as the adoption of industry 4.0 expands, so does the number of works focusing on the context of machine fault detection (mfd) [19, 26]. given that faults will inevitably occur, it is crucial to detect them as early as possible to prevent further implications. usual approaches to mfd rely primarily on signal analysis of vibration, force, temperature, and audio [26]. as discussed by chacon et al. [5], traditional signal-based mfd approaches that rely solely on vibration measurements are not effective for early-stage detection. conversely, acoustic signals should be favored due to their cost-effectiveness, high sensitivity, environmental resilience, ease of deployment, and lack of interference with equipment operation. such a point of view is also emphasized by saufi et al. [21], in their discussion of failure manifestations through time, summarized in fig. 1. considering that acoustic signals are amongst the first manifestations of failure, it is not surprising that their use has been gaining attention in mfd. fig. 1. evolution of an electric rotary machine’s health subsequent to a fault. important landmarks in fault manifestation are highlighted from 1 to 6. adapted from [21]. full size image as discussed by different authors, convolutional neural networks (cnns) and autoencoders (aes) are the state-of-the-art approaches in ml for acoustic signal-based mfd  [14, 21, 26]. autoencoders are particularly appealing in this setting due to their unsupervised nature, i.e., they are trained to reconstruct actual inputs. as normal operational data from machines is easily obtainable, it can be employed to model exclusively their regular behaviour. later, faults can be identified on the basis of their (ideally high) reconstruction error, as the model is not expected to provide a reasonable reconstruction for data that deviates from standard machine operation. this eliminates the need of faulty data collection for model training, which might be difficult and costly to obtain. moreover, collecting training data to represent all fault scenarios might be impractical. in this work we present a case study aimed at investigating the suitability of autoencoders for machine fault detection built solely based on acoustic features. for that we consider the machinery fault database (mafaulda) [11], which accounts for recordings of an electrical rotating machine during normal and faulty operation at different speeds. each recording is obtained from eight sensors: six accelerometers, a tachometer, and a microphone. it is worth noticing that this dataset has already been employed for machine fault detection, but considering combined data from all input sensors, as we shall discuss later. here, we place these results in perspective by considering exclusively acoustic data and “vanilla” or standard autoencoders (aes). we compare the performance of aes to that of mlps trained on statistical and spectral attributes from all the sensors’ signals, similar to the input feature set from previous work on the mafaulda database [11, 20]. the potential of acoustic mfd based on an unsupervised approach (ae) is contrasted to that of a supervised method (mlp). this work is organized as follows. related work on the mafaulda is presented in sect. 2. a proper description of the case study is given in sect. 3. the proposed approach for acoustic-based mfd employing autoencoders is detailed in sect. 4. the experimental setup adopted for evaluation and comparison, including metrics and baselines, is discussed in sect. 5. results are presented in sect. 6. final remarks and future work directions are made in sect. 7. 2 previous works on mafaulda different strategies have already been explored in the context of mfd using mafaulda. it is worth noticing, however, that the vast majority of these works employ all signals from the database (from six accelerometers, a tachometer, and a microphone) or perform the analysis considering only vibration data. the works of ribeiro et al. [20] and marins et al. [11] perform feature extraction from all the original signals considering statistical and spectral measures. both works employ similarity-based modeling (sbm) to perform classification of the signals according to their fault type (five classes), besides the normal class. a random forest is also employed in [11]. in both works, accuracy values higher than 80% are obtained. for the best models, accuracies close to 100% were observed. it is worth noticing, once more, that all signals are employed. more recently, zhao et al. [29] employed a parallel cnn (convolutional neural network) with an attention mechanism to perform multi-class fault diagnosis considering the mafaulda database. the model was trained considering only accelerometer data (vibration data) and four out of five classes from the database, normal condition and three fault types. the proposed classification methodology achieved an accuracy of \(99.6\%\). as previously discussed, however, this particular approach has a major disadvantage, that is, by relying solely on vibration measurements, the model is not effective for early-stage fault detection. finally, saufi et al. [22] proposed a hybrid approach that employs deep sparse autoencoders to perform feature extraction from mafaulda’s raw vibration signals. afterwards, a softmax classifier is trained on this data to perform the classification considering four classes: normal condition and three fault types. their approach reached an accuracy of \(95\%\), but suffers from the same drawback previously discussed, namely, early faults cannot be detected with vibration signals. 3 case study 3.1 the mafaulda database mafaulda stands for machinery fault database. the data was acquired by the federal university of rio de janeiro from a spectraquest’s machinery fault simulator (mfs) alignment-balance vibration trainer (abvt), which is an emulator of motor dynamics [11]. the database comprises 50 khz time series recordings of eight sensors: six accelerometers, a tachometer, and a microphone. these recordings were obtained from a working rotating machine operating at different rotation speeds, in normal and faulty conditions. five fault types are presented: imbalance, horizontal misalignment, vertical misalignment, overhang and underhang bearing. the faulty scenarios were obtained taking into account several degrees of intensity. overall, the database is heavily imbalanced, with 49 recordings from normal operation and 1,902 from abnormal operation (fault scenarios). the full database, along with further information, is available at [10]. 3.2 dataset preprocessing studies on mfd that consider spectrogram-based features extracted from acoustic signals, as in this case study, are commonly carried out on signals sampled at a maximum of 16 khz [4, 7, 8, 16, 25, 27]. indeed, recently, yi et al. [27] observed that, in the case of their application, almost all relevant information was below 8 khz. an in-depth frequency analysis regarding the acoustic signals for mafaulda has not been carried out to verify if such observations also hold for this case. nevertheless, in order to reduce the final complexity of the models (and thus computational time), we down-sampled the database’s acoustic data to 16 khz, following previous works. some preliminary/investigatory experiments were carried out with models trained on spectrogram features derived from the database at 50 and 16 khz. no significant changes in effectiveness were observed between the results obtained with higher and lower frequency signals. fig. 2. graphical representation of the down-sampling procedure. a 50 khz signal is down-sampled into three 16 khz signals by taking three sub-sampling windows. full size image as the down-sampling procedure is applied, it is possible to split the original 50 khz signal into three signals of 16 khz each, by shifting the sampling window one unit per sub-sampling group, as depicted in fig. 2. as a result, three signals with unique samples are obtained. even though unique, such sub-sampled signals resemble the original one and, therefore, some care shall be taken during training and testing of the models, to allow for a fair evaluation. in brief, sub-sampled signals that stem from the same original signal are all placed together either in the train or test set, not in both. on the one hand, this prevents training and testing on signals that are too similar to each other, which would yield overoptimistic results. on the other, this procedure allows for obtaining a larger number of objects. following this approach, we ended up with 147 normal acoustic signals and 5,706 abnormal ones (failure cases). before applying any mfd method, however, feature extraction was performed, as we detail in the next section. 3.3 feature extraction in the context of machine fault detection (mfd) based on acoustic signals, spectrogram-based approaches are often employed for feature engineering, i.e., feature extraction, with mel frequency energy coefficients (mfecs) figuring as the preferred representation [14, 26]. in brief, it consists of the application of the short-time fourier transform (stft), which, in turn, corresponds to performing the fast fourier transform (fft) in overlapping windowed segments of the acoustic signal. the resulting amplitude-frequencies are then grouped into bins (a.k.a mel bins) according to triangular filter banks. finally, their log-values are taken (more details regarding the feature extraction procedure can be obtained in [3, 26]). the final spectrogram organizes the time dimension into ranges, and maps for each time range a column of bins, where each bin is an mfec, and each column is termed as a frame, as depicted in fig. 3. fig. 3. the acoustic signal (left) is converted to an mfec space (center). once the sizes of the frame stack and the frame hop are defined, frame stacks are obtained. these constitute the dataset (right) that later is provided as input to the models. in this example, frame hop and frame stack sizes are set to 3 and 4, respectively. at the center frame stack 2 is highlighted (a total of three frame stacks are obtained). full size image in this work, we adopted the stack of frames featuring approach, which is a traditional methodology to compose the objects of a dataset stemming from mel-spectrograms [26]. this strategy builds each data object on the basis of a group of frames, which is referred to as a frame stack. consecutive frame stacks are shifted from each other by a number of frames, i.e., a frame hop. note that if the size of the frame hop is smaller than that of the frame stack there will be some overlap for successive frames. each acoustic signal is processed into a batch of frame stacks. the final dataset is formed from the collection of all frame stacks (objects) extracted from all the available acoustic signal segments (recordings) for the phenomenon under study. the process is illustrated in fig. 3. considering this procedure, table 1 shows the parameter values adopted for feature extraction in this work. parameter values for short-time fourier transform (stft) were set based on the works from  [3, 17, 24]. parameter values for mel bins, frame stack size and frame hop size were empirically determined. applying this featuring engineering to the outcome from data pre-processing (sect. 3.2), we ended up with a dataset composed of 409,710 objects, 10,290 normal and 399,420 abnormal, each with 1,600 dimensions. table 1. parameters employed in feature extraction from acoustic signals.full size table 4 machine fault detection based on autoencoders in brief, autoencoders (aes) are artificial neural networks (anns) designed primarily for input reconstruction, that is, the model is trained to reproduce inputs as outputs [6]. a vanilla autoencoder architecture has three distinct elements: encoder, bottleneck, and decoder. the encoder processes the input feature vector, reducing its original dimension. the smallest representation obtained with the encoder constitutes the bottleneck (a.k.a. latent space) of the model. this representation is, in turn, the input for the decoder, which aims to reconstruct the original input from the compressed information. the relevance of aes lies not only in their input reconstruction capabilities, but also in their ability to compress the input, which allows the model to capture the fundamental aspects needed for its reconstruction. due to the constraints imposed by the bottleneck structure, aes are compelled to represent internally only fundamental characteristics regarding input reconstruction, often learning interesting properties about it during the process [6]. despite using supervised learning algorithms for training (e.g., backpropagation) aes are regarded as unsupervised methods, given that no labeled data is needed [6]. the aforementioned characteristics are appealing in the context of mfd. in brief, after being successfully trained exclusively on data regarding normal machine behaviour, an ae is capable of reconstructing new instances drawn from healthy machine operation with minor errors. however, when instances of faulty machine operation are presented to the same ae, a higher reconstruction error is expected since the model was not trained to represent such phenomena. reconstruction errors can be measured/quantified and, if they exceed some pre-defined threshold, can be deemed as faults. the whole procedure is summarized by fig. 4. besides the feature extraction procedure, which was already discussed in sect. 3.3, we are left to define reconstruction errors and the definition of the threshold to determine a fault. this is done in the following. fig. 4. setup of our anomaly detection methodology. full size image when presented with an unseen frame stack (\(\textbf{fs}\)), an autoencoder model will produce as output a reconstructed version of the input (\(\widehat{\textbf{fs}}\)). the reconstruction error of the ae (re) is computed as the average of the individual errors obtained for each pair of corresponding bins (b) from: the input frame stack (\(\textbf{fs}_{b}\)) and the reconstructed stack (\(\widehat{\textbf{fs}}_b\)). considering that these frames have the same length (\(l_{fs}\)), equation (1) provides the definition for reconstruction error (re). $$\begin{aligned} re(\textbf{fs},\widehat{\textbf{fs}}) &= \frac{1}{l_{fs}} \sum _{b = 1}^{l_{fs}} \left( \textbf{fs}_{b} \widehat{\textbf{fs}}_b\right) ^2 \end{aligned}$$ (1) here, we recall that each 16 khz acoustic signal yielded a number of frame stacks (\(n_{fs}\))—see sect. 3.3 for details. we perform fault detection at a signal level, that is, we evaluate the 16 khz signal reconstruction error score (\(s_{re}\)), computed as the average over all its frame stacks’ reconstruction errors, so that \(s_{re} = {1}/{n_{fs}} \sum _{i = 1}^{n_{fs}} re(\textbf{fs}(i),\widehat{\textbf{fs}}(i))\), where \(\textbf{fs}(i)\) is a frame stack at index i. with that in mind, a faulty behaviour can be identified if its acoustic signal yields a reconstruction error score (\(s_{re}\)) higher than a pre-defined acceptable threshold. to determine such threshold, we resort to the youden index, in the context of roc curves [28], which identifies the score with the greatest vertical distance to the chance line and has been widely employed to identify the best threshold cutoff [15]. it seeks to maximize the effectiveness of a binary model by indicating the cutoff where tpr (true positive rate) and tnr (true negative rate) are optimal. the optimal cutoff (\(c^*\)) is given by eq. (2). $$\begin{aligned} c^* = \underset{c}{argmax}\ \left( tpr(c) + tnr(c) 1\right) \end{aligned}$$ (2) it is worth noting that calibration of the cutoff with the use of youden index was performed on training and validation set, never considering the test set (that is, the cutoff was estimated as in a real application scenario). given that a 5-fold cross-validation was employed for that (details are provided sect. 5 – experimental setup) four approaches were considered to determine the actual threshold from the validation folds for posterior fault determination within the test fold, namely: minimum (min), maximum (max), median and mean. 5 experimental setup 5.1 architecture of the autoencoder we evaluate autoencoders w.r.t. to their ability to model the behaviour of normal condition frame stacks, extracted from the acoustic signals. the ae input/output size is 1,600, i.e., the frame stack length obtained with the parameters presented in table 1. we performed a grid search over the number of layers \([2,4,\ldots ,10,12]\), neurons per inner layer \([400, 800,\ldots , 3200, 3600]\) and, neurons in the bottleneck layer \([4,8,16,\ldots ,128,256,512]\). the l2 penalty value was also included in the grid search, for which we considered the following values: [0.01, 0.1, 0.5, 1, 1.3, 1.5, 1.8, 2]. the most promising set of hyper-parameters identified was: six inner layers (considering encoder and decoder), each with 1,600 input neurons, except for the bottleneck, which is strangled from 1600 to 16 neurons. the best l2 penalty value was 1.0. this is the architecture employed in our evaluation. in our architecture, the relu activation function provided the best results. a batch normalization layer was included in the input. we trained our models using the mean squared error (mse) loss function, adam optimizer, and learning rate of \(10^{-5}\). figure 5 summarizes the design. note that the input’s reconstruction results from a direct linear combination of the last layer. fig. 5. architecture of the autoencoder (ae) considered for mfd during the evaluation. the input features are normalized through the application of batch normalization. full size image 5.2 baselines several studies have been carried out on the mafaulda database. the majority of them focus on multi-class classification and employ either all eight sensors, or only accelerometer data [11,12,13, 20, 23]. in this work we focus only on the acoustic signal in order to establish if it is sufficient to detect faults in a binary fashion (i.e., we do not perform fault isolation). we compare the autoencoder approach against four baseline models, all based on mlps (multilayer perceptrons). it is worth noticing that all baselines are evaluated on the same binary task of determining if the machine is under normal or faulty operation. the different baselines arise from distinct feature spaces, as we detail in the following. the literature considers statistical and spectral features as a traditional and efficient approach to engineer features and perform machine fault detection  [18, 26]. in fact, former works with mafaulda relied on this approach, considering all eight available signals [11, 20]. a total of 13 features are extracted from each raw signal, namely: root mean square; square root of the signal’s amplitude; kurtosis value; kurtosis factor; skewness value; peak-to-peak value; crest factor; impulse factor; margin factor; shape factor; frequency center; root variance frequency and; rms frequency. for further information on these features, see [18]. the first two baselines were trained with the statistical and spectral features listed above. the first, referred to as b104, is an mlp with 104 inputs, 13 for each one of the eight signal sensors from mafaulda. the second, is an mlp with 13 inputs, referred to as b13, that is based only on the 13 features extracted from the acoustic signal. the remaining two baselines, are referred to as mlp and cmlp (constrained mlp, due to a bottleneck in its arrangement), respectively. these are trained with the very same input features as provided to the autoencoder, that is, the mfec-based feature space described in sect. 3.3. regarding architectures b104 and b13, for a given acoustic signal, statistical and spectral features are extracted and provided as inputs. once trained, each model outputs a single representative score (\(s_b\)) in the range [0, 1]. for mlp and cmlp a single score is produced either. recall, however, that these two models take a frame stack (\(\textbf{fs}\)) as input. therefore, for a given 16 khz acoustic signal with \(n_{fs}\) frame stacks, we take the average over all frame stacks’ score predictions (\(score(\textbf{fs})\)) as the signal representative score (\(s_{mlp}\)), so that \(s_{mlp} = {1}/{n_{fs}}\sum _{i=1}^{n_{fs}} score({{ \textbf{fs}}(i)})\), where \(\textbf{fs}(i)\) is a frame stack at index i. the representative scores (\(s_b\) and \(s_{mlp}\)) from all baselines were calibrated following the same procedure employed for the autoencoder, i.e., using the youden index. each baseline has its hyper-parameters individually fine-tuned with a grid-search procedure. the hyper-parameters that led to the best mfd performance for each baseline were chosen, resulting in the configuration presented in table 2. table 2. architectures for the multilayer perceptrons (mlps) employed as baselines.full size table 5.3 training of score generators in order to obtain estimates regarding the quality of all the models a stratified 5\(\,\times \,\)5 fold cross-validation was employed. in brief, it comprises two nested 5-fold cross-validation estimators. the first one separates the data for testing, with the remaining data (referred to as full-train) being subjected to another 5-fold cross-validation approach, which further divides the data into training and validation. these last two sets are employed for hyper-parameter estimation and calibration of the thresholds for the fault detectors. at the end, the full-train set is used to train each model with the best parameters found. all models, are finally evaluated with the test set, which was not employed anywhere before. the split of the dataset occurs at a signal segment level. first, all signals are arranged and distributed according to the cross-validation folds. afterwards, the dataset is built by replacing the original acoustic segment by its three respective sub-sampled acoustic signals (see sect. 3.2). finally, feature extraction is performed on those. this approach prevents a model from being trained and tested with sub-samples that stem from the same acoustic signal, which would bias the evaluation, providing overconfident results. the first two baselines (traditional feature extraction) were trained with features extracted from the original signal. 5.4 evaluation metrics in this work, following traditional convention, we consider the normal class negative and the faulty class positive. given that we are dealing with a heavily imbalanced dataset (towards the positive class) metrics such as accuracy are not appropriate for evaluation. therefore, we perform our analysis on the basis of the area under the roc curve (auc), true positive rate (tpr), true negative rate (tnr) and, g-means. the g-means metric was introduced by [9]. it is defined as \(g{\scriptstyle -}means = \sqrt{tpr*tnr} = \sqrt{sensitivity * specificity}\) and is maximized when the sensitivity and specificity are both high and balanced. conversely, a low g-means score indicates a model that has a poor performance in predicting at least one of the classes, even if it correctly classifies the other. 6 results and discussion the results regarding the evaluation of the autoencoders and the four baselines are presented in fig. 6 and table 3. these results were obtained with a 5-fold cross-validation procedure considering the following metrics:  \(g\text {-}means\), auc, tpr and, tnr. for these, we depict both averages and standard deviation. recall that for each method we have to establish a threshold in order to perform fault detection. we considered four different approaches in order to select such a threshold from those of the validation set (minimum, maximum, mean, median). fig. 6. performance of the different models w.r.t. machine fault detection (mfd). the proposed model (ae) and the baselines mlp and cmlp were built using only features extracted on the basis of mfecs from the acoustic signal. baselines b104 and b13 were built with statistical and spectral features from all signals and only the acoustic signal, respectively. the figure shows the average auc, g-means, tnr and tpr, for each threshold approach: mean, median, minimum and maximum. full size image we begin our assessment considering the auc metric. it is worth noticing that the auc evaluation is threshold agnostic, that is, no particular threshold is considered (for that reason there is only one value of auc for each method in table 3). for this particular measure, the baseline b104, the mlp built on the basis of spectral and statistical features from all eight sensors, is the most promising model. the other baseline built on the basis of traditional features considering only acoustic signals (b13), however, has a poor performance, with an auc of \(0.63\pm 0.06\), which is barely above that of a completely random classifier. the remaining methods, the ae and the remaining baselines (mlp and cmlp), also perform well in terms of auc. these three are built considering features extracted from the mfecs from the acoustic signals only. from these, the ae was actually the worst performer (around 10% worse than the two mlps). it is important to recall, however, that the ae model does not require labeled data. the performance of the models in light of the g-means metric follows a similar pattern of that from the auc analysis. the most accurate anomaly detector is still b104 (with a performance similar to that observed in previous works on mafaulda), succeeded by both mlps and the ae. in the case of b104, the four thresholds yield almost the same performance. the mlp models are lightly affected by threshold selection, whereas for the ae model, mean and median thresholds taken from the validation set are more promising. once again, the model built on the basis of statistical and spectral features of the acoustic signal alone (b13) was the worst performer among all models under evaluation. table 3. detailed result for all the methods considered in the experiments.full size table the values of tnr and tpr metrics provide some insights into the values of the g-means metric. there is a higher standard deviation for the tnr metric, which might be justified by the small amount of negative (normal behaviour) class. the ae model predominantly exhibits higher tnr values than tpr. this might indicate that the threshold selected from the validation sets is too high. indeed, for the min threshold, tpr is higher than tnr for ae. such an effect might be an indicative that the ae did not fully learn how to represent the normal signal. this, in turn, might result in reconstruction errors for abnormal signals (faults) that are not clearly distinguishable from normal ones. even though the autoencoder approach did not provide the best results, we believe that taking this result on its own would be misleading. to fully appreciate these results, we must take into account some key points, discussed next. first, these results should be appreciated in light of the fact that there is a severe imbalance in the mafaulda database towards the positive class (faulty class). this means that the aes were trained on a subset from a total of 10,290 normal objectsfootnote 1. meanwhile, the mlp baselines that employed the same feature space (mlp and cmlp) were trained on a subset from a total of 409,710 objects (10,290 normal and 399,420 faulty). in brief, the baselines had almost 40 times more data available during training, which might have helped them to better represent the phenomenon under investigation, enhancing their performance. second, it is important to highlight that the autoencoder (ae) is an unsupervised method, whereas all baselines are supervised. this has major practical implications, given that the ae model depends exclusively on normal operation data from a machine for its training. all the baselines, on the other hand, cannot be trained without labeled data (normal and faulty). faulty data might be not only difficult to obtain, but also costly. moreover, it is almost impossible to foresee all fault types and magnitudes in real-world scenarios. 7 conclusions in this work a case study in the context of fault detection for electrical rotating machines was considered. more specifically, we evaluated unsupervised fault detection, performed with autoencoders, taking into account exclusively acoustic data. our case study was based on machine fault database (mafaulda), which is publicly available [10]. following good research practices, our experiments are fully reproducible, with all source code publicly available at [2]. our proposed approach was evaluated against baselines considering two feature extraction techniques, i.e., (i) statistical and spectral features (a “traditional” way of dealing with acoustic signals) and; (ii) the very same mfec-based feature space employed by our proposed approach. we also considered baselines that took as input additional signals from the rotating machine, other than the acoustic one, to place results in perspective. results from the evaluation indicate that a multilayer perception, trained on data from all the sensors available in mafaulda (baseline referred to as b104), provides the best overall results. in fact, these are close to perfect fault detection in the application. another mlp model (also a baseline—referred to as b13), however, that considered only acoustic signals, did not provide promising results. in fact, its auc values were close to that of a chance classifier. the remaining baselines (mlp and cmlp), which are based on acoustic signals, with the same feature extraction procedure considered for the ae, provided favorable results. we argue that the results obtained in this case study are encouraging and promising, despite the fact that the autoencoder model was not the top performer. the key points to support this claim are discussed in the following. first and foremost, our experiments confirm that fault detection based exclusively on acoustic signals is viable, even for cases in which acoustic collection was not optimized. we believe that this is the case for mafaulda, given that a set of sensory inputs was collected, with no focus on acoustic signals in particular. second, the autoencoder was the only unsupervised model from the set of models under evaluation. it is important to emphasize that it was trained only with normal behaviour data which, for mafaulda, consists in the minority class. indeed, the normal class has around 40 times fewer objects than the faulty one. note that even with this severe imbalance, the autoencoder model delivered auc values of 0.86, only 12% below the supervised mlp models that were trained on both normal and faulty data. still considering the learning paradigms employed by the models, we argue that the autoencoder is preferable in real-world application scenarios, as obtaining faulty data is not only time-consuming and costly, but might also pose a risk to equipment and people’s lives. we believe that, in light of these facts, future works considering acoustic signals and unsupervised fault detection based on autoencoders are promising and worth exploring. therefore, we plan to explore autoencoder architectures besides the “vanilla” flavour explored in this work. in particular, convolutional and variational autoencoders are of particular interest, due to their compelling properties. different approaches for determining thresholds will also be explored. notes 1.note that a 5-fold cross-validation was employed for performance estimation of the models, which means that not all data was actually available, due to train/test split. references alharbi, f., et al.: a brief review of acoustic and vibration signal-based fault detection for belt conveyor idlers using machine learning models. sensors 23(4), 1902 (2023). https://doi.org/10.3390/s23041902 article  mathscinet  math  google scholar  bortoni, l.: gitlab/leobortoni (2024). https://gitlab.com/leobortoni/mafaulda-stats-classifiers coelho, g., et al.: deep autoencoders for acoustic anomaly detection: experiments with working machine and in-vehicle audio. neural comput. appl. 34(22), 19485–19499 (2022). https://doi.org/10.1007/s00521-022-07375-2 article  math  google scholar  dohi, k., et al.: mimii dg: sound dataset for malfunctioning industrial machine investigation and inspection for domain generalization task. arxiv preprint arxiv:2205.13879 (2022) ferrando chacon, j.l., et al.: a novel approach for incipient defect detection in rolling bearings using acoustic emission technique. appl. acoust. 89, 88–100 (2015). https://doi.org/10.1016/j.apacoust.2014.09.002 article  math  google scholar  goodfellow, i., bengio, y., courville, a.: deep learning. mit press (2016). http://www.deeplearningbook.org harada, n., daisuke niizumi, d.t., yasunori ohishi, m.y., saito, s.: toyadmos2: another dataset of miniature-machine operating sounds for anomalous sound detection under domain shift conditions (2021) google scholar  koizumi, y., et al.: toyadmos: a dataset of miniature-machine operating sounds for anomalous sound detection. in: 2019 ieee workshop on applications of signal processing to audio and acoustics (waspaa), pp. 313–317. ieee, new paltz, ny, usa (2019). https://doi.org/10.1109/waspaa.2019.8937164 kubát, m., matwin, s.: addressing the curse of imbalanced training sets: one-sided selection. in: international conference on machine learning (1997) google scholar  mafaulda: machinery fault database (2016). https://www02.smt.ufrj.br/~offshore/mfs/. accessed 10 may 2024 marins, m.a., et al.: improved similarity-based modeling for the classification of rotating-machine failures. j. franklin inst. 355(4), 1913–1930 (2018). https://doi.org/10.1016/j.jfranklin.2017.07.038 article  math  google scholar  martins, d.h.c.s.s., hemerly, d.o., marins, m., lima, a.a., silva, f.l., prego, t.m., ribeiro, f.m.l., netto, s.l., da silva, e.a.b.: application of machine learning to evaluate unbalance severity in rotating machines. in: cavalca, k.l., weber, h.i. (eds.) iftomm 2018. mms, vol. 61, pp. 144–160. springer, cham (2019). https://doi.org/10.1007/978-3-319-99268-6_11 chapter  math  google scholar  messaoudi, m., et al.: classification of mechanical faults in rotating machines using smote method and deep neural networks. in: iecon 2022 48th annual conference of the ieee industrial electronics society, pp. 1–6 (2022). https://doi.org/10.1109/iecon49645.2022.9968875 nunes, e.c.: anomalous sound detection with machine learning: a systematic review. arxiv preprint arxiv:2102.07820 (2021) perkins, n.j., schisterman, e.f.: the inconsistency of optimal cutpoints obtained using two criteria based on the receiver operating characteristic curve. am. j. epidemiol. 163(7), 670–675 (2006). https://doi.org/10.1093/aje/kwj063 article  math  google scholar  purohit, h., et al.: mimii dataset: sound dataset for malfunctioning industrial machine investigation and inspection. in: proceedings of the detection and classification of acoustic scenes and events workshop, pp. 209–213 (2019) google scholar  purohit, h., et al.: hierarchical conditional variational autoencoder based acoustic anomaly detection. in: 30th european signal processing conference, pp. 274–278 (2022).https://doi.org/10.23919/eusipco55093.2022.9909785 rauber, t.w., de assis boldt, f., varejao, f.m.: heterogeneous feature models and feature selection applied to bearing fault diagnosis. ieee trans. ind. electron. 62(1), 637–646 (2015). https://doi.org/10.1109/tie.2014.2327589 article  math  google scholar  ren, s., et al.: a comprehensive review of big data analytics throughout product lifecycle to support sustainable smart manufacturing: a framework, challenges and future research directions. j. cleaner prod. 210, 1343–1365 (2019). https://doi.org/10.1016/j.jclepro.2018.11.025 ribeiro, f., marins, m., netto, s., silva, e.: rotating machinery fault diagnosis using similarity-based models. in: anais de xxxv simpósio brasileiro de telecomunicações e processamento de sinais (2017). https://doi.org/10.14209/sbrt.2017.133 saufi, s.r., ahmad, z.a.b., leong, m.s., lim, m.h.: challenges and opportunities of deep learning models for machinery fault detection and diagnosis: a review. ieee access 7, 122644–122662 (2019). https://doi.org/10.1109/access.2019.2938227 saufi, s.r., isham, m.f., ahmad, z.a., hasan, m.d.a.: machinery fault diagnosis based on a modified hybrid deep sparse autoencoder using a raw vibration time-series signal. j. ambient. intell. humaniz. comput. 14(4), 3827–3838 (2023). https://doi.org/10.1007/s12652-022-04436-1 article  google scholar  souza, r.m., et al.: deep learning for diagnosis and classification of faults in industrial rotating machinery. comput. ind. eng. 153, 107060 (2021). https://doi.org/10.1016/j.cie.2020.107060 suefusa, k., et al.: anomalous sound detection based on interpolation deep neural network. in: icassp 2020 2020 ieee international conference on acoustics, speech and signal processing (icassp), pp. 271–275 (2020). https://doi.org/10.1109/icassp40776.2020.9054344 tanabe, r., et al.: mimii due: sound dataset for malfunctioning industrial machine investigation and inspection with domain shifts due to changes in operational and environmental conditions. in: 2021 ieee workshop on applications of signal processing to audio and acoustics, pp. 21–25. ieee, new paltz, ny, usa (2021). https://doi.org/10.1109/waspaa52581.2021.9632802 tang, l., tian, h., huang, h., shi, s., ji, q.: a survey of mechanical fault diagnosis based on audio signal analysis. measurement 220, 113294 (2023). https://doi.org/10.1016/j.measurement.2023.113294 article  math  google scholar  yi, w., choi, j.w., lee, j.w.: sound-based drone fault classification using multitask learning (2023) google scholar  youden, w.j.: index for rating diagnostic tests. cancer 3(1), 32–35 (1950) article  math  google scholar  zhao, z., et al.: a fault diagnosis method of rotor system based on parallel convolutional neural network architecture with attention mechanism. j. sig. process. syst. 95(8), 965–977 (2023). https://doi.org/10.1007/s11265-023-01846-y article  math  google scholar  download references acknowledgements the authors thank fapesc for the financial support. author information authors and affiliations joinville technological center (ctj), federal university of santa catarina (ufsc), joinville, brazil leonardo afonso ferreira bortoni & pablo andretta jaskowiak authors leonardo afonso ferreira bortoniview author publications search author on:pubmed google scholar pablo andretta jaskowiakview author publications search author on:pubmed google scholar corresponding author correspondence to pablo andretta jaskowiak . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper bortoni, l.a.f., jaskowiak, p.a. (2025). acoustic features and autoencoders for fault detection in rotating machines: a case study. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_3 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_3 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine fault detection acoustic audio autoencoders publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature does pre-training on brain-related tasks results in better deep-learning-based brain age biomarkers? | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper does pre-training on brain-related tasks results in better deep-learning-based brain age biomarkers? conference paper first online: 12 october 2023 pp 181–194 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) does pre-training on brain-related tasks results in better deep-learning-based brain age biomarkers? download book pdf download book epub bruno m. pacheco9, victor h. r. de oliveira9, augusto b. f. antunes10, saulo d. s. pedro11 & danilo silva9 for the alzheimer’s disease neuroimaging initiative show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 520 accesses 4 citations abstract brain age prediction using neuroimaging data has shown great potential as an indicator of overall brain health and successful aging, as well as a disease biomarker. deep learning models have been established as reliable and efficient brain age estimators, being trained to predict the chronological age of healthy subjects. in this paper, we investigate the impact of a pre-training step on deep learning models for brain age prediction. more precisely, instead of the common approach of pre-training on natural imaging classification, we propose pre-training the models on brain-related tasks, which led to state-of-the-art results in our experiments on adni data. furthermore, we validate the resulting brain age biomarker on images of patients with mild cognitive impairment and alzheimer’s disease. interestingly, our results indicate that better-performing deep learning models in terms of brain age prediction on healthy patients do not result in more reliable biomarkers. data used in preparation of this article were obtained from the alzheimer’s disease neuroimaging initiative (adni) database (adni.loni.usc.edu). as such, the investigators within the adni contributed to the design and implementation of adni and/or provided data but did not participate in analysis or writing of this report. a complete listing of adni investigators can be found at: http://adni.loni.usc.edu/wp-content/uploads/how_to_apply/adni_acknowledgement_list.pdf. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others predicting brain age using lightweight 3d cnn architecture from t1-weighted mri images chapter © 2025 improving across dataset brain age predictions using transfer learning chapter © 2021 brain age and other bodily ‘ages’: implications for neuropsychiatry article open access 11 june 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. ageing brain mapping cognitive ageing neural ageing predictive markers artificial intelligence 1 introduction as human lifespan increases, there is a growing need for reliable methods to assess brain health and age-related changes in the brain. brain age prediction is a promising technique that uses neuroimaging data to estimate the apparent age of an individual’s brain, which can serve as an indicator of overall brain health and successful aging, as well as a disease biomarker [5, 6, 11, 17, 21]. deep learning models have shown great potential in accurately predicting brain age from magnetic resonance imaging (mri) data [4, 7, 17, 27, 29]. training deep learning models for brain age prediction shares several challenges with other neuroimaging tasks, in comparison to traditional computer vision, such as the increased gpu memory used from the 3d data and the extensive pre-processing required to account for the variability in the acquisition process. in particular, available neuroimaging datasets are much smaller than existing natural imaging datasets [13, 33], and deep learning models are known to be very dependent on sample sizes. therefore, data-efficient training strategies are crucial to achieve high performance in brain age prediction. in this paper, we explore the impact of pre-training deep learning models for brain age prediction. inspired by the learning process of expert neuroradiologists, we apply transfer learning by pre-training our brain age models on a brain-related task. for comparison, we also train models without pre-training and models pre-trained on natural image classification. we investigate the performance gain from pre-training and evaluate the models’ brain age prediction as a biomarker for cognitive impairment. more specifically: we pre-train deep learning models on the brain tumor segmentation task and compare them to models without pre-training and with pre-training on the imagenet natural image classification task; we test the brain age models using data from the adni studies, and show that the models pre-trained on the brain-related task outperform the other models, achieving the state-of-the-art in brain age prediction; we evaluate the brain age prediction of all models as a biomarker for different clinical groups (healthy, mild cognitive impairment, and alzheimer’s disease); our experiments suggest that, despite the common practice, better models in terms of brain age prediction of healthy patients do not result in more reliable biomarkers; all of our results are reported on a standardized, publicly available dataset, providing an easy comparison with future research. 2 related work detecting aging features on brain mri has been an active area of research for many years [10, 18, 34]. the use of deep learning for brain age prediction has gained considerable attention in recent years [1, 8, 20, 27, 29, 30]. in this section, we provide a brief overview of related works on brain age prediction from mris using deep learning models. one of the earliest applications of deep learning to brain age prediction was presented by cole et al. [7]. the authors employed a neural network comprising a convolutional backbone and a fully connected regression head to analyze a dataset of t1-weighted mri scans from 2001 healthy subjects aged 18 to 90. the training is performed solely on images of healthy subjects, following the hypothesis that the brain age of healthy subjects is close to their actual age. their deep learning model outperformed the machine learning approach (gaussian process regressor). the authors also assessed the reliability of the predictions across individuals and acquisition methods. jonsson et al. [17] developed a deep learning model for brain age prediction using brain mri scans from 1264 healthy subjects aged 18 to 75. they explored the impact of training and testing on distinct datasets, finding that the performance of brain age prediction degraded when the target dataset differed from the training dataset. bashyam et al. [4] proposed to improve brain age prediction by utilizing a larger dataset of brain mri scans from 14,468 subjects. the dataset included data acquired from different sites following different protocols, with subjects aged 3 to 95. the authors pre-trained their neural network on the imagenet dataset, which is an even larger dataset of natural images. they found that models performing well at chronological age prediction might not be the best at providing brain age estimates that correlate to the diagnosis of diseases such as schizophrenia and alzheimer’s. peng et al. [27] proposed quality brain age prediction using a lightweight deep learning model. they used a dataset containing 14,503 subjects from the uk biobank, with ages ranging from 44 to 80. the authors showed that even though larger models perform well on natural image tasks, smaller models can perform equally well and sometimes even better on medical imaging tasks. multiple authors have reported brain age performance on mri data from adni [19, 20, 22, 26, 29]. to the best of our knowledge, neither has provided means to reproduce the dataset used for testing the models. the studies either used a random, non-disclosed split, or did not provide which images have been selected from the adni database. therefore, a direct comparison is not possible, as we cannot perfectly replicate the evaluation setting. nonetheless, we highlight that the best performance reported was a mean absolute error of 3.10 years [20]. further details on the performance of each approach can be seen in table 1. table 1. performance of brain age prediction methods on mris from adni. the authors have not disclosed from which phases of the adni study the subjects were drawn. a\(\beta \)(-) indicates that the subjects have sustained a negative amyloid beta status over 3 years.full size table overall, these studies demonstrate the potential of pre-training for brain age prediction and the need for more efficient training strategies, as acquiring medical imaging data is very laborious.footnote 1 none of them, however, took advantage of models trained for other brain-related tasks, such as brain tumor segmentation. previous works also lack a standardized dataset on which we could perform a fair comparison, either because they use private datasets or because they do not share which samples were used for training or testing. therefore, our paper stands out by comparing brain age models pre-trained on brain tumor segmentation to models without pre-training or pre-trained on natural image classification. furthermore, we experimented on a standardized and publicly available dataset, providing reproducible results. 3 materials and methods 3.1 data adni. the alzheimer’s disease neuroimaging initiative (adni) was launched in 2003 with the primary goal of testing whether serial mri, positron emission tomography (pet), other biological markers, and clinical and neuropsychological assessment can be combined to measure the progression of mild cognitive impairment (mci) and early alzheimer’s disease (ad) [28]. the adni database contains longitudinal data from clinical evaluations, cognitive tests, biological samples, and various types of imaging data, including mri, functional mri and pet. for up-to-date information, see https://adni.loni.usc.edu. the study’s data has been collected over several phases: adni-1, adni-go, adni-2, and adni-3. the dataset contains cohorts of individuals with ad, mci, and healthy controls (cognitively normal, cn). furthermore, all exams underwent quality control for the image quality (e.g., subject motion, anatomic coverage). in this paper, we employed available mprage t1-weighted mris from all phases, filtering out images deemed “unusable” by the quality control assessment. the images are available after gradient non-linearity and intensity inhomogeneity correction, when necessary. an overview of the demographics from each dataset can be seen in table 2. more detailed information on the images from adni-1 used in this work can be found in wyman et al. [35]. table 2. overview of the subjects from the adni database whose images were used in this work. age is considered at the time of the first visit.full size table brats. the brain tumor segmentation challenge (brats) [25] provides a dataset of structural brain mris along with expert annotations of tumorous regions [2]. for each subject, four mri scan modalities are available: t1-weighted, contrast-enhanced t1-weighted, t2-weighted, and t2-flair. all images are available after preprocessing (registration to a common atlas, interpolation to 1 mm\(^{3}\), and skull-stripping) [3]. in the 2020 editionfootnote 2, the dataset contained 369 images from subjects aged 18 to 86 (avg. of 61.2). 3.2 preprocessing we applied minimal preprocessing to the adni images. our major goal was to ensure all images would have the same orientation and spatial resolution, and that they would present no skull or non-brain-tissue information, i.e., only brain voxels would be present in the image. only the brain information must be available in the image, otherwise, the deep learning models could learn to predict the age based on other structures. we register all adni images to the mni152 template, interpolate to 2 mm\(^{3}\) resolution, and apply skull-stripping using hd-bet [16]. to feed the preprocessed 3d mri scans to the 2d brain age models, each volume was sliced through the axial plane. we discard the slices from the top 40 mm and the bottom 35 mm of the scan to exclude slices with little to no brain information. therefore, we extract 40 images from each 3d mri scan. with respect to the images from the brats dataset, no further (see sect. 3.1) preprocessing is performed. we feed all slices of the 3d mri scan to the deep learning models. 3.3 deep learning models we use 2d deep convolutional neural networks for brain age prediction. our proposed model can be divided into a backbone and a head. intuitively, the backbone is responsible for extracting relevant features from the input image, while the head combines these features into the final prediction. the backbone consists of several convolution filters that reduce the image size. the head is a single unit with linear activation that is fully connected to the backbone’s output. figure 1 illustrates our proposed model, highlighting both the backbone and the head. fig. 1. illustration of the overall architecture of our brain age model. the backbone is composed of several convolutional operations, extracted from the u-net or the resnet architectures. the head is a fully-connected layer applied to the vectorized values of the backbone’s output. full size image even though the architecture of the head is task-specific, the backbone’s architecture depends only on a few characteristics of the input (e.g., number of channels, minimum size). furthermore, learned features from one task can be useful for another, at least as a starting point, which is known as transfer learning [31]. this allows us to reutilize the backbone of models trained for different tasks, that is, we can extract the backbone from a model trained for some task and use it as the backbone for our model designed for brain age prediction. we use the backbones from two different architectures: resnet [12] and u-net [32]. more specifically, we use the resnet-50 architecture, available in the torchvision package [24], and the 2d u-net proposed in [14], which is designed for medical image segmentation. the u-net is composed of an encoder, a bottleneck, and a decoder. for our backbone, we use the encoder with the bottleneck. to obtain the brain age prediction of a 3d mri, we apply the model to the 40 axial slices of the image that contain brain information and take the mean of the outputs [4]. the pipeline of operations can be seen in fig. 2. fig. 2. brain age prediction of a 3d mri using a 2d deep learning model. full size image 3.4 pre-training on brain tumor segmentation to leverage the knowledge from other brain-related tasks to brain age prediction, we pre-train our backbones in the brain tumor segmentation task. we follow the brats challenge setup, with brats data, for both u-net and resnet backbones. to be able to train a resnet backbone in a segmentation task, we replace the original head of the resnet with a u-net decoder, in a resunet architecture [36]. this means that the decoder matches the resnet backbone with respect to the size of the intermediate feature maps so that the skip connections can be added in the same way as in the original u-net implementation. we follow crimi et al. [15] for training all models on the brats data. we first train the models on a random 80/20 split of the brats 2020 dataset. the models are evaluated through the dice score [25], which measures the overlap between the predicted segmentation mask \(\hat{y}\) and the ground truth y as $$ \text {dice}(\hat{y},y) = \frac{2 |\hat{y} \cap y|}{|\hat{y}| + |y|} .$$ based on the models’ performance, we fixed the number of epochs to avoid overfitting. then, the entirety of the brats data is used to train the backbones for a fixed number of epochs. 3.5 training on brain age prediction to train the deep learning models on brain age prediction, we assume, following previous work, that the brain age of healthy subjects is close to their chronological age [4, 7, 20]. thus, we train and evaluate our models solely on the images of subjects belonging to the cn group. to provide easy-to-compare results, we choose to evaluate our models on a standardized dataset. therefore, we follow the standardized split of the analysis set for adni-1 [35], and use the standard test set as our test set, and their training set as our validation set. in other words, we divide the preprocessed adni-1 t1-weighted scans from cn subjects following the standardized split to form our validation and test sets. the remaining images (i.e., those from adni-go, adni-2, and adni-3) compose our training set. detailed information on the images used in the training set can be found in our code repositoryfootnote 3. we train all models, regardless of backbone architecture or pre-training, using the adam optimizer to minimize the mean squared error between the age predicted from each slice and the true age of the cn subjects. the models are first trained on the training set. the performance of these models on the validation set is used for hyperparameter tuning and early stopping. namely, batch size and learning rate were adjusted, and a moving averagefootnote 4 of the mae on the validation set was used to determine the ideal epoch (i.e., the one with the smallest mae) for stopping the training. the models with the best performance on the validation set are then evaluated on the test set, which is unseen up to then. 3.6 evaluation the brain age models were evaluated on the error between the predicted age and the actual age of the cn subjects. more specifically, we use the mean absolute error (mae) as our standard evaluation metric. we compute the error based on the predicted age of the whole 3d mri, i.e., after averaging the predictions of all slices as described in sect. 3.5 and illustrated in fig. 2. furthermore, we evaluate the capacity of the brain age estimate in differentiating between cn, mci, and ad patients. for this, we use the brain age delta \(\vardelta _{ba} = \hat{y}_{ba} y_{ca}\), which is the difference between the predicted brain age \(\hat{y}_{ba}\) and the chronological age \(y_{ca}\) of a subject. as the progression toward alzheimer’s diagnosis is associated with aging patterns, it is expected that the \(\vardelta _{ba}\) of a subject in the ad group is greater than that of a subject in the mci group, and that the latter’s \(\vardelta _{ba}\) is still greater than the \(\vardelta _{ba}\) of a subject in the cn group. therefore, we compute the predicted \(\vardelta _{ba}\) for all images in the three groups (cn, mci, ad) of the test set and apply a pairwise mann-whitney u (mwu) test [9, 23]. the mwu test is a nonparametric version of the t-test for independent samples. in our case, the null hypothesis is that the \(\vardelta _{ba}\) from one group is not stochastically greater than the other. 4 experiments and results in our experiments, we evaluate the impact of brain-related pre-training using 3 backbones: the u-net with random initialization, resunet with random initialization, and resunet pre-trained on the imagenet. to improve the reliability of our results, 5 models are trained for each experiment, e.g., 5 u-net models with random weight initializations are evaluated without brain-related pre-training, against 5 (different) u-net models with brain-related pre-training. in other words, for each of the 3 backbones, we train 10 models: 5 with no brain-related pre-training, and 5 with brain-related pre-training. all experiments reported below were performed on a linux machine with 8 vcpus, 30 gb of ram and an nvidia t4 gpu. further details regarding the implementation of the experiments and additional results can be seen in code our repositoryfootnote 5. 4.1 pre-training on brats following the procedure described in sect. 3.4, we trained 5 u-net models and 5 resunet models with random initialization on the brain tumor segmentation task. we also used the backbone from resnets pre-trained on the imagenet’s natural image classification task, therefore, we trained 5 resunet models using the imagenet pre-trained backbone. imagenet pre-trained models are readily available in the torchvision package. using an 80-20 random split, we observed that 30 epochs were enough to achieve peak performance and avoid overfitting for the u-net models, while 50 epochs were enough for the resunet models. the average performance of these models on the random split can be seen in table 3. we highlight that the models present performance on par with state-of-the-art brain tumor segmentation models [3]. all models were then re-trained (with new initial random weights) on the entirety of the dataset for the same number of epochs. table 3. performance of the models on an 80-20 split of the brats dataset. models were trained on 80% of the data and evaluated on 20%. reported values are the average (and standard deviation) of 5 runs with random initialization, except for the backbone of the resunet that uses the backbone of the resnet pre-trained on imagenet on all runs.full size table 4.2 brain age prediction we train 5 models of each combination of backbone and pre-training available. namely: u-net backbone with random initialization or pre-trained on brats; resnet backbone with random initialization or pre-trained on brats; and resnet backbone pre-trained on imagenet or pre-trained on imagenet and then on brats. as described in sect. 3.5, we use the validation set to define the best set of hyperparameters for each backbone and pre-training combination. more specifically, we used a batch size of 64 images for all models and a learning rate of \(10^{-3}\) for the models with u-net backbone without pre-training, \(10^{-5}\) for the models with resnet backbone with imagenet and brats pre-training, and \(10^{-4}\) for all other models. we trained all models for 50 epochs, early-stopping the training when a running average of the mae on the validation set achieved the smallest value. the average performance of the models on the test set can be seen in table 4. furthermore, after hyperparameter tuning, we re-train the models on the union of the training and the validation sets. this increases the amount of data used for training and increases the similarity between the distribution of the training data and the test data, as both validation and test sets are drawn from the same study (adni-1). we defined the training budget for each backbone and pre-training combination as the average of the epochs in which the respective 5 models achieved the early-stopping criterion on the validation set. note that the test set is not considered in any step of this process, therefore, no data leakage occurs. the average test set performance of the models trained on the training and validation sets can be seen in table 4, under column “test mae (train+validation models)”. in our experiments, pre-training on the brain tumor segmentation task was consistently advantageous. even though the performance difference was not highly significant at all experiments, the models with the proposed pre-training outperformed their counterpart in all configurations except for the resnet backbone pre-trained on the imagenet when trained on the union of the train and validation sets. it is also evident, when comparing the resnet backbones, that pre-training solely on brats was consistently better than pre-training on the imagenet. our experiments show that the best brain age predictions are achieved by using a u-net backbone pre-trained on brats, even though the resnet backbones achieved a better brain tumor segmentation performance (see table 3). finally, the use of the validation set for training (after hyperparameter tuning) improved the performance of all models, as can be seen in the column “test mae (train+validation models)” of table 4. table 4. average performance (over 5 runs) of brain age models. “train+validation models” indicates that the models were trained on the union of train and validation datasets, with hyperparameters defined on previous experiments using the validation set. values in bold indicate the best performance in their respective set (column).full size table 4.3 statistical analysis of \(\vardelta _{ba}\) to assess the significance of the resulting brain age indicator, we apply the mwu test to the predicted \(\vardelta _{ba}\) of all models for subjects in the cn, mci, and ad groups of the validation and the test sets (see sect. 3.6). the tests on samples between cn and mci patients and cn and ad patients all pointed to a strong differentiation, with p-values smaller than 0.1% for all modelsfootnote 6, indicating that the \(\vardelta _{ba}\) biomarker is useful to distinguish healthy patients from those with alzheimer’s disease or mild cognitive impairment. the distinction, however, between mci and ad patients was not as significant, as can be seen in table 5. note that the images from mci and ad patients of the validation set are not used for training or hyperparameter tuning, thus, the validation set can be interpreted as an additional test set in the case of differentiating between mci and ad. table 5. significance of the mwu test on the distinction between ad and mci patients using the predicted \(\vardelta _{ba}\) values. we report the average (maximum) p-value over the 5 models trained for each combination of backbone and pre-training. “train+validation models” indicates that the models were trained on the union of train and validation datasets (cn only), with hyperparameters defined on previous experiments using the validation set (cn only). values in bold indicate high significance (p-value \(\le 5\%\)).full size table even though the models with the u-net backbones achieved more significant results in the statistical analysis of the biomarker, these results do not allow us to conclude that pre-training (with any of the tasks) had a positive impact, as was observed for brain age prediction. in fact, the exact opposite is observed. using pre-trained backbones (for both brats and imagenet pre-training) resulted, most of the times, in models that achieved a worse separation between ad and mci patients, in comparison to their counterparts without pre-training. the same effect was also observed in the use of the validation set for training, upon which no model achieved a significant distinction between ad and mci patients. 5 discussion and conclusions in this study, we investigated the transfer learning capacity of a brain-related task to the task of brain age prediction. more specifically, we pre-trained deep learning models for brain age prediction on the task of brain tumor segmentation. in comparison to pre-training on a natural image classification task or performing no pre-training at all, our results suggest that the proposed pre-training may be a better option. the only inconclusive case is on performing both pre-trainings, that is, first pre-training on imagenet and then on brain tumor segmentation, which yielded mixed results for brain age prediction. furthermore, we observed that using the validation set for training the models (after hyperparameter tuning) significantly reduced the brain age prediction error in all scenarios, even though there was only a small increment in the number of subjects in the training set (15% more subjects, as can be verified through table 2). using the u-net backbone pre-trained on brats and then trained on the union of train and validation sets showed state-of-the-art results, with mae values previously unseen on adni data (see table 1). we recall that the validation and test sets are built with images from the adni-1 study, while the training set is built from adni-go, adni-2, and adni-3. as the image acquisition protocols change between the studies, we can assume that there are different characteristics between their images. therefore, we can expect a distribution shift between the training and the test sets that does not exist between the validation and the test sets. this allows us to conclude that the use of the validation set for training resulted in better models because it decreased the difference between the training and test distributions. at the same time, by evaluating our models as biomarkers for cognitive impairment levels, we observed results that challenge the standard approach of training deep learning models for brain age. most of the modifications that improved the brain age predictive performance on healthy subjects (i.e., the chronological age prediction), resulted in models that were less reliable for distinguishing between patients with mci and ad. a similar behavior was reported by bashyam et al. [4], in which the models that achieved the lowest mae did not provide the strongest distinction between healthy subjects and subjects with ad, mci, schizophrenia or depression. our results show an inverse relationship between the performance on chronological age prediction (see table 4) and the reliability of the biomarker for cognitive impairment levels (see table 5). this is particularly true for the use of the validation set for training brain age models, which significantly reduced the reliability in all scenarios. given that reducing the distribution shift between training and testing data degraded the performance of the biomarker, we speculate that state-of-the-art models got to a point of overfitting the images of healthy subjects, thus, degrading their performance on images from ad and mci patients. therefore, we suggest that the validity of chronological age predictions as a means to develop brain age models is an important investigation topic for the development of the brain age biomarker. notes 1.in contrast to natural imaging datasets, as medical imaging requires an expensive procedure and legal authorization from each subject. 2.https://www.med.upenn.edu/cbica/brats2020/data.html. 3.https://github.com/gama-ufsc/brain-age. 4.at the end of each epoch, we compute the average over the 5 latest results, including the current one. 5.https://github.com/gama-ufsc/brain-age. 6.the results of the tests on cn-mci and cn-ad are available in our code repository https://github.com/gama-ufsc/brain-age. references armanious, k., et al.: age-net: an mri-based iterative framework for brain biological age estimation. ieee trans. med. imaging 40(7), 1778–1791 (2021). https://doi.org/10.1109/tmi.2021.3066857 bakas, s., et al.: advancing the cancer genome atlas glioma mri collections with expert segmentation labels and radiomic features. sci. data 4(1), 170117 (2017). https://doi.org/10.1038/sdata.2017.117 article  google scholar  bakas, s., et al.: identifying the best machine learning algorithms for brain tumor segmentation, progression assessment, and overall survival prediction in the brats challenge (2018). https://doi.org/10.48550/arxiv.1811.02629. publisher: arxiv version number: 3 bashyam, v.m., et al.: mri signatures of brain age and disease over the lifespan based on a deep brain network and 14,468 individuals worldwide. brain 143(7), 2312–2324 (2020). https://doi.org/10.1093/brain/awaa160 article  google scholar  cole, j.h., et al.: brain age predicts mortality. mol. psychiatry 23(5), 1385–1392 (2018). https://doi.org/10.1038/mp.2017.62 article  google scholar  cole, j.h., leech, r., sharp, d.j., for the alzheimer’s disease neuroimaging initiative: prediction of brain age suggests accelerated atrophy after traumatic brain injury. ann. neurol. 77(4), 571–581 (2015). https://doi.org/10.1002/ana.24367 cole, j.h., et al.: predicting brain age with deep learning from raw imaging data results in a reliable and heritable biomarker. neuroimage 163, 115–124 (2017). https://doi.org/10.1016/j.neuroimage.2017.07.059 article  google scholar  dinsdale, n.k., et al.: learning patterns of the ageing brain in mri using deep convolutional networks. neuroimage 224, 117401 (2021). https://doi.org/10.1016/j.neuroimage.2020.117401 article  google scholar  fay, m.p., proschan, m.a.: wilcoxon-mann-whitney or t-test? on assumptions for hypothesis tests and multiple interpretations of decision rules. stat. surv. 4, 1–39 (2010). https://doi.org/10.1214/09-ss051 franke, k., ziegler, g., klöppel, s., gaser, c.: estimating the age of healthy subjects from t1-weighted mri scans using kernel methods: exploring the influence of various parameters. neuroimage 50(3), 883–892 (2010). https://doi.org/10.1016/j.neuroimage.2010.01.005 article  google scholar  gaser, c., franke, k., klöppel, s., koutsouleris, n., sauer, h.: alzheimer’s disease neuroimaging initiative: brainage in mild cognitive impaired patients: predicting the conversion to alzheimer’s disease. plos one 8(6), e67346 (2013). https://doi.org/10.1371/journal.pone.0067346 article  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: 2016 ieee conference on computer vision and pattern recognition (cvpr), pp. 770–778 (2016). https://doi.org/10.1109/cvpr.2016.90 deng, j., berg, a.c., li, k., fei-fei, l.: what does classifying more than 10,000 image categories tell us? in: daniilidis, k., maragos, p., paragios, n. (eds.) eccv 2010. lncs, vol. 6315, pp. 71–84. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-15555-0_6 chapter  google scholar  isensee, f., jaeger, p.f., kohl, s.a.a., petersen, j., maier-hein, k.h.: nnu-net: a self-configuring method for deep learning-based biomedical image segmentation. nat. methods 18(2), 203–211 (2021). https://doi.org/10.1038/s41592-020-01008-z article  google scholar  isensee, f., jäger, p.f., full, p.m., vollmuth, p., maier-hein, k.h.: nnu-net for brain tumor segmentation. in: crimi, a., bakas, s. (eds.) brainles 2020. lncs, vol. 12659, pp. 118–132. springer, cham (2021). https://doi.org/10.1007/978-3-030-72087-2_11 chapter  google scholar  isensee, f., et al.: automated brain extraction of multisequence mri using artificial neural networks. hum. brain mapp. 40(17), 4952–4964 (2019). https://doi.org/10.1002/hbm.24750 article  google scholar  jonsson, b.a., et al.: brain age prediction using deep learning uncovers associated sequence variants. nat. commun. 10(1), 5409 (2019). https://doi.org/10.1038/s41467-019-13163-9 article  google scholar  kondo, c., et al.: an age estimation method using brain local features for t1-weighted images. in: 2015 37th annual international conference of the ieee engineering in medicine and biology society (embc), milan, pp. 666–669. ieee (2015). https://doi.org/10.1109/embc.2015.7318450 lam, p., zhu, a.h., gari, i.b., jahanshad, n., thompson, p.m.: 3d grid-attention networks for interpretable age and alzheimer’s disease prediction from structural mri (2020) google scholar  lee, j., et al.: deep learning-based brain age prediction in normal aging and dementia. nat. aging 2(5), 412–424 (2022). https://doi.org/10.1038/s43587-022-00219-7 article  google scholar  liem, f., et al.: predicting brain-age from multimodal imaging data captures cognitive impairment. neuroimage 148, 179–188 (2017). https://doi.org/10.1016/j.neuroimage.2016.11.005 article  google scholar  ly, m., et al.: improving brain age prediction models: incorporation of amyloid status in alzheimer’s disease. neurobiol. aging 87, 44–48 (2020). https://doi.org/10.1016/j.neurobiolaging.2019.11.005 article  google scholar  mann, h.b., whitney, d.r.: on a test of whether one of two random variables is stochastically larger than the other. ann. math. stat. 18(1), 50–60 (1947). https://doi.org/10.1214/aoms/1177730491 article  mathscinet  math  google scholar  marcel, s., rodriguez, y.: torchvision the machine-vision package of torch. in: proceedings of the 18th acm international conference on multimedia, firenze, italy, pp. 1485–1488. acm (2010). https://doi.org/10.1145/1873951.1874254 menze, b.h., et al.: the multimodal brain tumor image segmentation benchmark (brats). ieee trans. med. imaging 34(10), 1993–2024 (2015). https://doi.org/10.1109/tmi.2014.2377694 article  google scholar  more, s., antonopoulos, g., hoffstaedter, f., caspers, j., eickhoff, s.b., patil, k.r.: brain-age prediction: a systematic comparison of machine learning workflows. neuroimage 270, 119947 (2023). https://doi.org/10.1016/j.neuroimage.2023.119947 article  google scholar  peng, h., gong, w., beckmann, c.f., vedaldi, a., smith, s.m.: accurate brain age prediction with lightweight deep neural networks. med. image anal. 68, 101871 (2021). https://doi.org/10.1016/j.media.2020.101871 article  google scholar  petersen, r.c., et al.: alzheimer’s disease neuroimaging initiative (adni): clinical characterization. neurology 74(3), 201–209 (2010). https://doi.org/10.1212/wnl.0b013e3181cb3e25 article  google scholar  poloni, k.m., ferrari, r.j.: a deep ensemble hippocampal cnn model for brain age estimation applied to alzheimer’s diagnosis. expert syst. appl. 195, 116622 (2022). https://doi.org/10.1016/j.eswa.2022.116622 article  google scholar  popescu, s.g., glocker, b., sharp, d.j., cole, j.h.: local brain-age: a u-net model. front. aging neurosci. 13, 761954 (2021). https://doi.org/10.3389/fnagi.2021.761954 article  google scholar  raghu, m., zhang, c., kleinberg, j., bengio, s.: transfusion: understanding transfer learning for medical imaging. curran associates inc., red hook (2019). https://dl.acm.org/doi/10.5555/3454287.3454588 ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: navab, n., hornegger, j., wells, w.m., frangi, a.f. (eds.) miccai 2015. lncs, vol. 9351, pp. 234–241. springer, cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28 chapter  google scholar  russakovsky, o., et al.: imagenet large scale visual recognition challenge. int. j. comput. vision 115(3), 211–252 (2015). https://doi.org/10.1007/s11263-015-0816-y article  mathscinet  google scholar  wang, j., li, w., miao, w., dai, d., hua, j., he, h.: age estimation using cortical surface pattern combining thickness with curvatures. med. biol. eng. comput. 52(4), 331–341 (2014). https://doi.org/10.1007/s11517-013-1131-9 article  google scholar  wyman, b.t., et al.: standardization of analysis sets for reporting results from adni mri data. alzheimer’s dement. 9(3), 332–337 (2013). https://doi.org/10.1016/j.jalz.2012.06.004 article  google scholar  zhang, z., liu, q., wang, y.: road extraction by deep residual u-net. ieee geosci. remote sens. lett. 15(5), 749–753 (2018). https://doi.org/10.1109/lgrs.2018.2802944 article  google scholar  download references author information authors and affiliations federal university of santa catarina (ufsc), florianópolis, sc, brazil bruno m. pacheco, victor h. r. de oliveira & danilo silva alliar nepia, belo horizonte, mg, brazil augusto b. f. antunes 3778 healthcare, belo horizonte, mg, brazil saulo d. s. pedro authors bruno m. pachecoview author publications search author on:pubmed google scholar victor h. r. de oliveiraview author publications search author on:pubmed google scholar augusto b. f. antunesview author publications search author on:pubmed google scholar saulo d. s. pedroview author publications search author on:pubmed google scholar danilo silvaview author publications search author on:pubmed google scholar consortia for the alzheimer’s disease neuroimaging initiative corresponding author correspondence to danilo silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pacheco, b.m., de oliveira, v.h.r., antunes, a.b.f., pedro, s.d.s., silva, d., for the alzheimer’s disease neuroimaging initiative. (2023). does pre-training on brain-related tasks results in better deep-learning-based brain age biomarkers?. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_13 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords brain age deep learning adni brats mri transfer learning publish with us policies and ethics profiles danilo silva view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature the artificial intelligence as a technological resource in the application of tasks for the development of joint attention in children with autism | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper the artificial intelligence as a technological resource in the application of tasks for the development of joint attention in children with autism conference paper first online: 12 october 2023 pp 306–320 cite this conference paper intelligent systems (bracis 2023) nathália assis valentim9, fabiano azevedo dorça9, valéria peres asnis9 & … nassim chamel elias10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 632 accesses 3 citations abstract people with autism spectrum disorder (asd) may present, in addition to deficits in communication, social interaction and patterns of restricted and repetitive behaviors, also present a deficit in joint attention (ja), which refers to the response repertoire of following and/or directing an adult’s visual attention to objects or events in the environment. by having a strong relationship with the learning process, joint attention deficits can compromise a person’s learning process. in this way, the use of technology can help in the development of abilities in people with autism, such as, for example, improving joint attention, communication and social skills. in this context, the general objective of the work proposal was to develop a computational approach for intervention that allows the interaction of the student with autism, with 4 and 5 years old, with deficit in joint attention and social-communicative difficulties. artificial intelligence (ai) techniques were used to model the most appropriate sequence and level of complexity of exercises for each child. ai resources were used with the intention of providing an intelligent environment to guide the child, dynamically and adaptively, in order to promote stimuli and adequate personalization of the process. in this way, it is intended to contribute significantly to the advancement of the state of the art regarding the production of computational technologies for people with asd. supported by research support foundation of the state of minas gerais (fapemig) universal demand process: apq-00837-21. this research has an opinion embodied by the research ethics committee number 5.273.182, with caae 54880921.7.0000.5152, the proposing institution being the faculty of computing of the federal university of uberlândia. this is a preview of subscription content, log in via an institution to check access. access this chapter log in via an institution subscribe and save springer+ from $39.99 /month starting from 10 chapters or articles per month access and download chapters and articles from more than 300k books and 2,500 journals cancel anytime view plans buy now chapter usd 29.95 price excludes vat (usa) available as pdf read on any device instant download own it forever buy chapter ebook usd 69.99 price excludes vat (usa) available as epub and pdf read on any device instant download own it forever buy ebook softcover book usd 89.99 price excludes vat (usa) compact, lightweight edition dispatched in 3 to 5 business days free shipping worldwide see info buy softcover book tax calculation will be finalised at checkout purchases are for personal use only institutional subscriptions similar content being viewed by others individualizing intervention to teach joint attention, requesting, and social referencing to children with autism article 05 june 2018 investigating joint attention in children with autism spectrum disorder through virtual reality and eye-tracking: a comparative study article 04 april 2025 the role of artificial intelligence interventions to improve eye contact for children with autism spectrum disorder: a systematic review article open access 19 december 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. autism spectrum disorders computational intelligence developmental disabilities intelligence augmentation intelligence development artificial intelligence references de almeida, l.g.s.: padrões de projeto de análise para desenvolvimento de software do domínio do transtorno do espectro autista (tea). master’s thesis, universidade federal fluminense (2021) google scholar  american psychiatric association: dsm-5: manual diagnóstico e estatístico de transtornos mentais. artmed editora (2014) google scholar  barsoum: fer+ (face expression recognition plus dataset) (2017). https://github.com/microsoft/ferplus barsoum, e., zhang, c., ferrer, c.c., zhang, z.: training deep networks for facial expression recognition with crowd-sourced label distribution. in: acm international conference on multimodal interaction, pp. 279–283 (2016). https://doi.org/10.1145/2993148.2993165 bates, e., benigni, l., bretherton, i., camaioni, l., volterra, v.: the emergence of symbols: cognition and communication in infancy. academic press, new york (1979) google scholar  cardon, t.a., wilcox, m.j., campbell, p.h.: caregiver perspectives about assistive technology use with their young children with autism spectrum disorders. infants young child. 24(2), 153–173 (2011). https://doi.org/10.1097/iyc.0b013e31820eae40 article  google scholar  elias, n.c.: teorias comportamentais sobre a etiologia do autismo e uma nova proposta. uel (2019) google scholar  gera, d., balasubramanian, s.: landmark guidance independent spatio-channel attention and complementary context information based facial expression recognition. pattern recogn. lett. 145, 58–66 (2021). https://doi.org/10.1016/j.patrec.2021.01.029 article  google scholar  ghahramani, z.: unsupervised learning. in: bousquet, o., von luxburg, u., rätsch, g. (eds.) ml 2003. lncs (lnai), vol. 3176, pp. 72–112. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-28650-9_5 chapter  math  google scholar  goodfellow, i., bengio, y., courville, a., bengio, y.: deep learning, vol. 1. mit press, cambridge (2016) google scholar  jia, y., et al.: caffe: convolutional architecture for fast feature embedding. in: acm international conference on multimedia. association for computing machinery (2014). https://doi.org/10.1145/2647868.2654889 juárez-ramírez, r., navarro-almanza, r., gomez-tagle, y., licea, g., huertas, c., quinto, g.: orchestrating an adaptive intelligent tutoring system: towards integrating the user profile for learning improvement. procedia. soc. behav. sci. 106, 1986–1999 (2013). https://doi.org/10.1016/j.sbspro.2013.12.227 article  google scholar  chinea manrique de lara, a., jiménez de espinoza, c., gonzález-mora, j.: a fast automated diagnosis system for autism spectrum disorders based on eye tracking technology (2016). https://doi.org/10.13140/rg.2.2.32220.28809 mordvintsev, a., abid, k.: opencv-python tutorials documentation (2014). https://media.readthedocs.org/pdf/opencv-python-tutroals/latest/opencv-python-tutroals.pdf mundy, p., delgado, c., block, j., venezia, m., hogan, a., seibert, j.: early social communication scales (escs). university of miami, coral gables (2003) google scholar  pavlov, n.: user interface for people with autism spectrum disorders. j. softw. eng. appl. (2014). https://doi.org/10.4236/jsea.2014.72014 article  google scholar  pimenta, t.: transtorno do espectro autista ou autismo: causas e tratamento (2018). https://www.vittude.com/blog/transtorno-do-espectro-autista-ou-autismo/. accessed may 2019 sherkatghanad, z., et al.: automated detection of autism spectrum disorder using a convolutional neural network. front. neurosci. 13 (2019). https://doi.org/10.3389/fnins.2019.01325 tenório, m., vasconcelos, n.: autismo: a tecnologia como ferramenta assistiva ao processo de ensino e aprendizagem de uma criança dentro do espectro. cintedi-práticas pedagógicas direitos humanos e interculturalidade (2015) google scholar  valentim, n.a.: experiment data (2022). https://l1nk.dev/experimentdata. accessed oct 2022 vijayan, a., janmasree, s., keerthana, c., syla, l.b.: a framework for intelligent learning assistant platform based on cognitive computing for children with autism spectrum disorder. in: international cet conference on control, communication, and computing, pp. 361–365 (2018). https://doi.org/10.1109/cetic4.2018.8530940 download references acknowledgement i offer my sincerest gratitude to my right arm research support foundation of the state of minas gerais (fapemig) universal demand process: apq-00837-21. author information authors and affiliations universidade federal de uberlândia, uberlândia, minas gerais, brazil nathália assis valentim, fabiano azevedo dorça & valéria peres asnis universidade federal de são carlos, são carlos, são paulo, brazil nassim chamel elias authors nathália assis valentimview author publications search author on:pubmed google scholar fabiano azevedo dorçaview author publications search author on:pubmed google scholar valéria peres asnisview author publications search author on:pubmed google scholar nassim chamel eliasview author publications search author on:pubmed google scholar corresponding author correspondence to nathália assis valentim . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper valentim, n.a., dorça, f.a., asnis, v.p., elias, n.c. (2023). the artificial intelligence as a technological resource in the application of tasks for the development of joint attention in children with autism. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_20 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_20 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial intelligence intelligent tutoring systems autism learning process joint attention publish with us policies and ethics profiles fabiano azevedo dorça view author profile access this chapter log in via an institution subscribe and save springer+ from $39.99 /month starting from 10 chapters or articles per month access and download chapters and articles from more than 300k books and 2,500 journals cancel anytime view plans buy now chapter usd 29.95 price excludes vat (usa) available as pdf read on any device instant download own it forever buy chapter ebook usd 69.99 price excludes vat (usa) available as epub and pdf read on any device instant download own it forever buy ebook softcover book usd 89.99 price excludes vat (usa) compact, lightweight edition dispatched in 3 to 5 business days free shipping worldwide see info buy softcover book tax calculation will be finalised at checkout purchases are for personal use only institutional subscriptions search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 not affiliated © 2025 springer nature a custom bio-inspired algorithm for the molecular distance geometry problem | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a custom bio-inspired algorithm for the molecular distance geometry problem conference paper first online: 12 october 2023 pp 178–192 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a custom bio-inspired algorithm for the molecular distance geometry problem download book pdf download book epub sarah ribeiro lisboa carneiro  orcid: orcid.org/0009-0004-6877-52059, michael ferreira de souza  orcid: orcid.org/0000-0001-6751-287710, douglas o. cardoso  orcid: orcid.org/0000-0002-1932-334x11, luís tarrataca  orcid: orcid.org/0000-0001-9359-51439 & … laura s. assis  orcid: orcid.org/0000-0003-3081-97229  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 518 accesses abstract protein structure allows for an understanding of its function and enables the evaluation of possible interactions with other proteins. the molecular distance geometry problem (mdgp) regards determining a molecule’s three-dimensional (3d) structure based on the known distances between some pairs of atoms. an important application consists in finding 3d protein arrangements through data obtained by nuclear magnetic resonance (nmr). this work presents a study concerning the discretized version of the mdgp and the viability of employing genetic algorithms (gas) to look for optimal solutions. we present computational results for input instances whose sizes varied from 10 to \(10^3\) atoms. the results obtained show that approaches to solving the discrete version of the mdgp based on gas are promising. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a quantum approach to the discretizable molecular distance geometry problem article 07 july 2022 recent advances on the interval distance geometry problem article 03 january 2017 unassigned distance geometry and molecular conformation problems article 15 april 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biological structure determination molecular evolution molecular modelling molecular biophysics protein design protein structure 1 introduction an important application in molecular biology is to find 3d arrangements of proteins using the interatomic distances obtained through nuclear magnetic resonance (nmr) experiments [24]. the objective of the molecular distance geometry problem (mdgp) is to determine molecular structures from a set of interatomic distances. proteins are essential parts of living organisms. the interest in the 3d structure of proteins results from the intimate relationship between form and protein functions. the function of a molecule is determined by its chemical and geometrical structure [4]. calculating the 3d structure of proteins is an essential problem for the pharmaceutical industry [16]. the mdgp was studied in [11] where a discretizable version of the mdgp (dmdgp) with a finite search space is also presented. the work proposes a branch-and-prune algorithm by searching a binary tree (representing the search space of the dmdgp) for a feasible solution to the problem. the mdgp can be formulated as an optimization problem whose objective is to determine, according to some criteria, the best alternative in a specified universe. in general, combinatorial optimization problems can be defined as given a set of candidate solutions s, a set of constraints \(\varomega \), and an objective function f responsible for associating a cost f(s) to each candidate solution \(s \in s\). the objective is to find \(s^* \in s\) that is globally optimal and with the best cost amongst the candidate solutions [22]. optimization methods can be divided into two classes: (i) deterministic, and (ii) stochastic. in the former, it is possible to predict the trajectory according to the input, i.e., the same input always leads to the same exit. stochastic methods rely on randomness, thus the same input data can produce distinct outputs, as is the case with bio-inspired algorithms [9]. this work presents an evolutionary approach for solving the dmdgp and studies its respective viability. the dmdgp belongs to the class of np-hard problems through a reduction of the subset-sum problem [14]. our aim is to answer the following questions: (i) how effective is an evolutionary heuristic for solving the dmdgp, evaluated on real 3d instances of proteins?; (ii) for which instance sizes is it possible to find optimal solutions? in order to tackle these issues we present two stochastic approaches based on genetic algorithms (ga) for solving the dmdgp. the work is organized as follows. related works are presented in sect. 2. section 3 formalizes the problem. the solution approaches proposals are characterized in sect. 4. the computational experiments and approach comparisons are presented in sect. 5. the concluding remarks are presented in sect. 6. 2 related work there are two ways to model the mdgp, namely, a continuous and a discrete formulation. there also exist disparities concerning the instances employed. some works make use of geometric instances that were randomly generated based on a model [17]. others employ synthetic instances produced in a random fashion based on a physical model [13]. there are also works that use dense real instances [18] whilst others employ sparse real instances [11] that were taken from the protein data bank (pdb) [1]. regarding the continuous version of the mdgp, in [6] the authors considered the case where the exact distances between all pairs of atoms are known. the algorithm is based on simple geometrical relationships between atom coordinates and the distances between them. the method was tested with the protein set hiv-1 rt that is composed of 4200 atoms. the algorithm is executed in linear time. in [12] a divide and conquer (d &c) algorithm was proposed that makes use of sdp alongside gradient descent refining. a structure consisting of 13000 atoms was solved in an hour. a geometric algorithm for approximate solutions with instances ranging from 402 to 7398 atoms was presented in [25]. in [5] the 3d-as-synchronized-as-possible (3d-asap) d &c approach was proposed. protein lengths varied between 200 to 1009 atoms. the work also proposes a more agile version of 3d-asap, 3d-sp-asap, that uses a spectral partitioning algorithm to preprocess the graph into subgraphs. the results obtained showed that both are robust algorithms for sparse instances. the continuous approach most resembling this work was presented in [23] and describes the memhpg memetic algorithm (ma). the method combines swarm intelligence alongside evolutionary computation associated with a local search with the objective of optimizing the crossover operator. memhpg was tested with proteins whose lengths varied between 402 and 1003. the algorithm was able to obtain the solution to the problem. the result obtained was comparable with the then state of the art described in [25]. regarding the discrete approaches, in [10], a d &c approach (abbie) was presented that explores combinatorial characteristics of the mdgp. the idea is to reduce the global optimization problem into subproblems. abbie was able to find approximate solutions for the mdgp. in [2] a distributed algorithm was described that does not make use of “known” positions to determine the successor ones. the method divides the problem graph into subgraphs using sampling methods: (i) sdp relaxation vying to minimize the sum of errors for specified and estimated distances; and (ii) gradient descent method of error reduction. in the discrete formulations, the works most closely related to the one here presented are [21]. the authors define a discretizable variation of the geometry distance problem known as the dmdgp and propose the branch-and-prune algorithm. in [8] a method is described to explore the symmetries that are present in the search space of the dmdgp and that can be represented by a binary tree. the method was able to obtain a relevant speedup for sparse instances against the branch-and-prune. work [26] discusses a sequence of nested and overlapping dmdgp sub-problems. the results exhibited superior performance for randomly generated sparse instances. 3 presentation and modelling 3.1 dgp the distance geometry problem (dgp) consists in finding a set of points in a given geometric space of dimension k by knowing only some distances between them [15]. the dgp can be solved in linear time when all the distances are known [6]. formally, given a simple, connected and weighted graph, \(g = (v, e, d)\), with a set v of n vertices, a set e of m edges, and a weight function \(d:e\rightarrow \mathbb {r}_+\) we wish to determine a function \(x: v \rightarrow \mathbb {r}^k\) such that \(||x(u) x(v)|| = d(u, v)\) for every \(\{u, v\} \in e\). the dgp can be formulated as a global optimization problem whose objective function is described in eq. (1). $$\begin{aligned} & \min _x g(x) = \sum _{\{u,v\} \in e}{(||x(u) x(v)||^2 d(u, v)^2)^2}, \end{aligned}$$ (1) where x solves the problem if and only if \(g(x) = 0\). this approach is challenging since an elevated number of local minima exist due to the non-convergence of the associated optimization problem. 3.2 mdgp the 3d structure of a molecule is intrinsic to its function. data provided by nmr allows for the calculation of the 3d structure of a protein. this procedure only provides the distances between close atoms. the problem consists in employing this information to obtain the position of every atom in the molecule. the mdgp is a particular case of dgp where the set of vertices v corresponds to the set of atoms of a molecule, the edges e is the set of known distances between atoms and the weight function d is the distance value. figure 1(a) shows a solution in \(\mathbb {r}^2\) for a graph representing a problem instance. atoms a, b, c can be fixed in \(\mathbb {r}^2\) to eliminate solutions from rotations and translations. point d can be positioned in any of the countless points of the circle of center c and radius represented by the distance between c and d (fig. 1(b)). the represented problem has a non-enumerable quantity of solutions. if the distance between b and d is also known (in addition to the one between c and d) then the problem presents a finite number of solutions. in this case, two solutions can be found at the intersection of the circumferences of center c and b with respective radius represented by the distances between: (i) c and d; and (ii) b and d. this case is exemplified in fig. 1(c). fig. 1. solutions may be non-enumerable or finite. full size image 3.3 discrete version of the mdgp as seen, the mdgp is a constraint satisfaction problem, which is generally reformulated as a global optimization one, where the objective is to minimize a penalty function capable of measuring how much the constraints are being violated (eq. (1)). however, the penalty function of the optimization problem is strongly non-smooth. optimization methods risk getting stuck at local minima with an objective value very close to the optimal one. the dmdgp [19] is a particular case of mdgp when the geometric space is \(\mathbb {r}^3\), the set of vertices v is finite, and the set of edges e satisfies the following constraints: (i) every subset \(v_i=\{v_i, v_{i-1}, v_{i-2},v_{i-3}\}\subset v\) induces a \(4-\)clique in g; (ii) the distances in \(\{d(v_{i}, v_{j}):i,j\in v_i\}\) define non-null volume tetrahedrons. therefore, when the discretization assumptions are satisfied, the domain of the penalty function can be reduced to a tree. in [11] the branch-and-prune method was proposed where the dmdgp search space is represented by a binary tree and the procedure attempts to fix an atom with each iteration. assume that for each atom \(v_i\), for \(i > 3\), the distances between \(v_i\) and the tree immediately predecessors atoms \(v_{i-1}, v_{i-2}, v_{i-3}\) are known. assuming that these three atoms possess realizations \(x_{i-1}\), \(x_{i-2}\) e \(x_{i-3}\) in \(\mathbb {r}^3\), then it is possible to define tree spheres centered in these points and whose radius is, respectively, the distances \(d(v_i,v_{i-1})\), \(d(v_i,v_{i-2})\) and \(d(v_i,v_{i-3})\). under reasonable assumptions, the intersections of these spheres are the set \(\{x_i, x^\prime _i\}\) formed by the coordinates (branch) satisfying those three distances. one of these realizations can be chosen for vertex \(v_i\) and it is then possible to proceed to fix the next vertex \(v_{i+1}\). it may be possible that other edges (distances) exist that use vertex \(v_i\) that invalidate these coordinates. if this occurs then it becomes necessary to change the choices previously made (prune). this methodology defines a binary tree of atomic coordinates that contains every possible position for each vertex \(v_i\), \(i>3\), in the \(i^{\text {th}}\) tree layer. each dmdgp solution is associated with a path from the root to a leaf node. this behaviour is exemplified in fig. 2 where vector [0, 1, 1] represents a solution. fig. 2. solution example binary tree path. full size image the representation of the dmdgp as a binary tree evidences several symmetry properties in the search space [20]. namely, the invariance of the solution set concerning the total reflection of a solution is commonly referred to as the first symmetry. for example, if \(s = [0, 1, 1]\) in fig. 2 is a feasible solution to the problem, then the inverse of all the bits in s (\(\overline{s} = [1, 0, 0]\)) is also a feasible solution. it is important to emphasize that, in the binary representation, the first three atoms are fixed in \(\mathbb {r}^3\). these characteristics led to the approach choice proposed in this work. 4 solution methodology the solution approach developed to solve the dmdgp is based on the concepts of gas. these are optimization algorithms imbued with a certain oriented randomness that is based on an analogy with the darwinian principles of evolution and species genetics [7]. gas have applications in discrete and continuous systems [27]. in this work, we propose two methods based on gas, referred to as (i) simplified genetic algorithm (sga); (ii) feasible index genetic algorithm (figa). the main difference resides in the genetic operator employed. each stage of the gas proposed and their respective particularities are presented in the following subsections. 4.1 representation and initial population gas are population algorithms where each individual represents a candidate solution for a problem. the population can be defined as a set of candidate solutions in the search space. the dmdgp search space is a binary tree, where each of the two possibilities in a given level of the tree symbolizes a position in \(\mathbb {r}^3\). a solution can be represented by a binary uni-dimensional chromosome of size n (number of atoms). the first three atoms have their position fixed. as a result, the chromosome can be implemented with size \(n-3\). the values assumed for each position, 0 or 1, are known as alleles and indicate the path chosen in the tree. the set of genes is known as the genotype. figure 1 illustrates the coding implemented for the dmdgp. a constructive random heuristic generates the initial population. namely, due to the coding of the problem for each position a choice is made (with 50% of probability) if the allele will have value 1 or 0. table 1. chromosomal representation for the mdgp.full size table 4.2 feasibility the set of distances of the dmdgp can be divided into two distinct subsets, namely: (1) discretization distances; and (2) pruning distances. discretization distances are the ones utilized in the construction of the binary tree, and pruning distances are the remaining ones. when only the subset of discretization distances is considered then all tree paths lead to feasible solutions. however, a vertex may exist in the tree that does not obey the pruning distances constraints. this results in the solution becoming unfeasible for that tree branch. in the proposed figa algorithm, there is a method responsible for going through the tree and verifying until which chromosome position the solution is feasible, i.e., \(||x(u) x(v)|| = d(u, v)\) for each \(\{u, v\} \in e\). this position, which will be referred to as the feasible index, is used to implement crossover and mutation operations. the sga approach represents the basic implementation of a ga and therefore does not consider the feasible index in its genetic operators. this is the main difference between the previously proposed methods. 4.3 fitness function at each generation, the individuals of a population are evaluated based on their fitness values. as is typical in optimization problems, the fitness function adopted in this work is the inverse of the objective function (of) with 1 added in the denominator, as can be seen in eq. (2). $$\begin{aligned} fitness = \left( 1 + \displaystyle \sum _{\{u,v\} \in e}(||x(u) x(v)||^2 d(u, v)^2)^2 \right) ^{-1}. \end{aligned}$$ (2) the of is equal to 0 when the known distances are equal to the calculated distances. in this case, x is a feasible solution for the dmdgp and the fitness will be equal to 1. the of value is always greater or equal to zero. the fitness value tends to zero the bigger the of value. 4.4 selection at each generation, the selection process elects the pairs of individuals that will participate in the crossover. these individuals are chosen based on a roulette selection that guarantees that all the individuals have a certain probability of being chosen that is proportional to their aptitude. 4.5 crossover crossover is the process in which an offspring individual is created from the genetic information of the parents. this work employs a one-point crossover operator. in the sga approach, the crossover consists in generating a random position between \([1, n-1]\), where n is the chromosome size, and the randomly selected position is the cut point of the sga crossover. in the figa strategy, the crossover point is determined based on the feasible index (f) (subsect. 4.2). this mechanism allows for the generated offspring to always inherit the feasible part of the parent that presents that largest feasible index. the figa crossover behavior is illustrated in fig. 3. the feasible index for each parent is emphasized with the color red. the cut point is determined by the largest feasible index (dotted line). the interval position [0, c[ of the son, in which c is the cut point, will have the genetic information of the parent with the largest feasible index. the remaining values of the son (i.e., \(] c+1, n[\)) are copied from the other parent. fig. 3. crossover operator. full size image 4.6 mutation the main objective of this operator is to introduce genetic diversity in the population of the current generation. namely, each offspring generated can be mutated according to probability \(p_m\). if an offspring is selected for mutation, then the operator considers each gene individually and allows the allele to be inverted in accordance with the same probability. in the figa approach, the mutation is performed only on the infeasible part of the solution. let f represent the feasible index (chromosome position in red fig. 4) and n the size of the chromosome, then the figa mutation can occur in the position of the interval \([f, n-1]\). in the sga strategy, the mutation can happen in all the chromosome. figure 4 illustrates the behavior of the figa mutation, with the feasible index being emphasized in red color. fig. 4. mutation operator. (color figure online) full size image 4.7 reset the reset operation aims to avoid the solution getting stuck in the local minimum. namely, if the fitness value of the best individual does not improve over a certain number of generations then the reset is activated. the best individual is maintained and the remaining population is discarded. a new population of individuals is generated with the random constructive heuristic. 4.8 pseudocode the pseudocode for the figa is presented in algorithm 1. both the sga and the figa have as a stopping criterion a maximum quantity of generations (genmax) and search for a feasible solution (line 6). line 1 initializes the variable that stores the largest fitness, bestfitness. the algorithm generates in a random fashion an initial population, calculates the fitness of each individual, and their respective feasible indexes (lines 3, 5 and 5, respectively). the sga does not consider the feasible index. therefore, line 5 is only present in the figa algorithm. the second cycle (line 8) considers the number of offspring formed with each generation. this value is defined by the multiplication of the crossover rate (\(t_c\)) and the population size (\(p_s\)). the line 9 selects a pair of individuals for the crossover operation. line 10 is responsible for performing the crossover of the previously selected individuals. the newly generated offspring individual is stored in the variable offspring and can incur a mutation with probability \(p_m\). the mutation occurs in line 13. the line 15 (only available in figa) calculates the feasible index of the new individuals. as previously stated, the genetic operators (crossover and mutation) are different in the proposed algorithms. the fitness of the offspring is then compared against its respective parents. if an improvement is observed, the parent with worse fitness is replaced by its offspring (lines 16 to 24). both methods use the elitism scheme to prevent the loss of the fittest member of the population. therefore, the method “getbestfiness” returns the largest fitness of the current population, this value is part of the stop criteria, and that individual will be part of the population for the next generation. 5 results 5.1 parameter configuration due to the high-dimension search space, performing a full grid search of the optimization parameters would have a prohibitive cost. as a consequence, a smaller test was performed for synthetic instancesfootnote 1 in \(\mathbb {r}^2\). in order to define quality values for the ga parameters. the full set of results obtained is detailed in [3]. the evaluation procedure consisted of a set of instances with size \(n =\) [10, 30, 50, 100] was randomly generated taking into account the mdgp. these instances are composed of d lines, where d is the number of known distances. each line represents a graph edge, i.e., a known distance between a pair of atoms. each edge identifies the connecting atoms as well as the distance between them. for example, \([0 \ 1 \ 2.5]\) represents that atom \(v_0\) is at a distance of 2.5 measurement units from atom \(v_1\). algorithm 1 : figa pseudocode full size image table 2. genetic algorithm parameter variation in \(\mathbb {r}^2\).full size table in these first tests, both methods were executed 10 times each in an algorithm independent run for each instance with each parameter combination. the parameters and values evaluated are presented in table 2. there were 840 combinations of parameters totaling 8400 executions of the proposed algorithms for the four instance sizes considered in the study. the tests allowed for a comparison between sga and figa. for instances with only 10 atoms, it is not possible to observe which ga presents the best performance. however, as the number of atoms increases the balance shifts in favor of the figa. namely, sga is able to find a feasible solution for instances of small size (maximum 30 atoms). on the other hand, the figa is able to find the feasible solution for every instance size evaluated. by analyzing the results obtained the value were fixed in \(t_c = 0.7\), \(p_m = 0.3\), \(p_s = 200\) e \(genmax = 30\) values. 5.2 experiments in 3d space this section presents the tests performed with real instances in \(\mathbb {r}^3\). instances. the computational experiments described in this section employed real instances taken from the pdbfootnote 2 [1]. the instances selection was made randomly ensuring a linear increase in the number of atoms. the set of instances used is listed in table 3 and illustrated in fig. 5. table 3. pdb instances.full size table presentation and result analysis. this section presents the figa results for the above mentioned \(\mathbb {r}^3\) instances. the elevated computational time made it unfeasible to perform a parameter study specific to the \(\mathbb {r}^3\) instances. as a consequence, the parameter values were empirically chosen based on the tests performed for \(\mathbb {r}^2\) and some tests for \(\mathbb {r}^3\) instances. after conducting empirical tests with various values for the reset, including not performing it, the value of 25 was selected for this parameter. additional empirical tests were conducted considering values around the best parameters for \(\mathbb {r}^2\), as shown in table 2. however, since the instances in \(\mathbb {r}^3\) are much larger than the synthetic instances in \(\mathbb {r}^2\), we had to increase the population size and the number of generations. consequently, the parameter values for \(\mathbb {r}^3\) were defined as indicated in table 4. table 4. configuration of genetic algorithm parameter in \(\mathbb {r}^3\).full size table fig. 5. pdb proteins used. full size image in the graphs that follow (fig. 6), the green line depicts the largest fitness of the population, the blue line represents the average fitness and the red line corresponds to the smallest fitness. figure 6 shows the relationship between the feasible index and the number of generations for the instances with 114, 201, 300, 402, 501, and 702 atoms. when the algorithm finds a feasible solution the feasible index is equal to the chromosome size. the algorithm was able to find a feasible solution for all of the above mentioned instance sizes. however, the algorithm was unable to find a feasible solution for the remaining instances (table 3 600, 801, 900, and 1002 atoms) using the set of stipulated parameters. by analyzing the results presented in the plots it is possible to gain a better understanding of the figa evolution until a feasible solution is found. for the smallest instance employed (fig. 6(a)) the algorithm only required 6 generations to find a feasible solution. in the case of the instance using 702 atoms the algorithm was able to find a feasible solution after 350 generations. table 5 indicates for each instance what was the iteration where a feasible solution was found or indicates with a character “–” when a solution was not found. future work would try to improve the ga in order to explore the symmetry properties of the dmdgp in the crossover and mutation operators. potential improvement strategies: i) using a hierarchically structured population, ii) developing a memetic algorithm, and iii) evaluating other genetic operator types. fig. 6. figa performance in \(\mathbb {r}^3\) feasible index \(\times \) number of generations. (color figure online) full size image table 5. figa results in \(\mathbb {r}^3\).full size table 6 final considerations this work presented a viability study regarding the application of ga for solving the dmdgp. the difference between the proposed algorithms, sga and figa, resides in the partial or total feasibility of the solution and how this information affects the corresponding genetic operators. the figa evaluates a tree and verifies up until which chromosome position the solution is feasible. this index is used to perform crossover and mutation operations. the sga does not consider this information in its genetic operators. though subtle, this difference was responsible for significant improvements in the tests performed. for instances of size \(n =\) [10, 30, 50, 100] in \(\mathbb {r}^2\) the figa presented better or equal performance when compared against sga. this performance was measured in terms of fitness and also the generation in which convergence occurred. the figa was able to solve real pdb instances for sizes 114, 201, 300, 402, 501, and 700. notes 1.instance set is publicly available in: https://bit.ly/3d0ezzo. 2.the set of instances is available in https://www.rcsb.org/. references berman, h.m., et al.: the protein data bank. nucleic acids res. 28(1), 235–242 (2000) article  google scholar  biswas, p., toh, k.c., ye, y.: a distributed sdp approach for large-scale noisy anchor-free graph realization with applications to molecular conformation. siam j. sci. comput. 30(3), 1251–1277 (2008) article  mathscinet  math  google scholar  carneiro, s., souza, m., filho, n., tarrataca, l., rosa, j., assis, l.: algoritmo genético aplicado ao problema da geometria de distâncias moleculares. in: anais do lii simpósio brasileiro de pesquisa operacional (2020) google scholar  creighton, t.e.: proteins: structures and molecular properties. macmillan (1993) google scholar  cucuringu, m., singer, a., cowburn, d.: eigenvector synchronization, graph rigidity and the molecule problem. inf. infer. j. ima 1(1), 21–67 (2012) mathscinet  math  google scholar  dong, q., wu, z.: a linear-time algorithm for solving the molecular distance geometry problem with exact inter-atomic distances. j. global optim. 22(1), 365–375 (2002) article  mathscinet  math  google scholar  goldberg, d.e.: genetic algorithms in search. optimization, and machine learning (1989) google scholar  goncalves, d.s., lavor, c., liberti, l., souza, m.: a new algorithm for the \(^k\)dmdgp subclass of distance geometry problems (2020) google scholar  gong, y.j., et al.: distributed evolutionary algorithms and their models: a survey of the state-of-the-art. appl. soft comput. 34, 286–300 (2015) article  google scholar  hendrickson, b.: the molecule problem: exploiting structure in global optimization. siam j. optim. 5(4), 835–857 (1995) article  mathscinet  math  google scholar  lavor, c., liberti, l., maculan, n., mucherino, a.: the discretizable molecular distance geometry problem. comput. optim. appl. 52(1), 115–146 (2012) article  mathscinet  math  google scholar  leung, n.h.z., toh, k.c.: an sdp-based divide-and-conquer algorithm for large-scale noisy anchor-free graph realization. siam j. sci. comput. 31(6), 4351–4372 (2010) article  mathscinet  math  google scholar  liberti, l., lavor, c., maculan, n., marinelli, f.: double variable neighbourhood search with smoothing for the molecular distance geometry problem. j. global optim. 43(2–3), 207–218 (2009) article  mathscinet  math  google scholar  liberti, l., lavor, c., mucherino, a.: the discretizable molecular distance geometry problem seems easier on proteins. in: mucherino, a., lavor, c., liberti, l., maculan, n. (eds.) distance geometry, pp. 47–60. springer, new york (2013). https://doi.org/10.1007/978-1-4614-5128-0_3 liberti, l., lavor, c., mucherino, a., maculan, n.: molecular distance geometry methods: from continuous to discrete. int. trans. oper. res. 18(1), 33–51 (2011) article  mathscinet  math  google scholar  maculan filho, n., lavor, c.c., de souza, m.f., alves, r.: álgebra e geometria no cálculo de estrutura molecular. colóquio brasileiro de matemática, impa (2017) google scholar  moré, j.j., wu, z.: global continuation for distance geometry problems. siam j. optim. 7(3), 814–836 (1997) article  mathscinet  math  google scholar  moré, j.j., wu, z.: distance geometry optimization for protein structures. j. global optim. 15(3), 219–234 (1999) article  mathscinet  math  google scholar  mucherino, a., lavor, c., liberti, l.: the discretizable distance geometry problem. optim. lett. 6(8), 1671–1686 (2012) article  mathscinet  math  google scholar  mucherino, a., lavor, c., liberti, l.: exploiting symmetry properties of the discretizable molecular distance geometry problem. j. bioinform. comput. biol. 10(03), 1242009 (2012) article  math  google scholar  mucherino, a., liberti, l., lavor, c.: md-jeep: an implementation of a branch and prune algorithm for distance geometry problems. in: fukuda, k., hoeven, j., joswig, m., takayama, n. (eds.) icms 2010. lncs, vol. 6327, pp. 186–197. springer, heidelberg (2010). https://doi.org/10.1007/978-3-642-15582-6_34 chapter  google scholar  mulati, m.h., constantino, a.a., da silva, a.f.: otimização por colônia de formigas. in: lopes, h.s., de abreu rodrigues, l.c., steiner, m.t.a. (eds.) meta-heurísticas em pesquisa operacional, 1 edn., chap. 4, pp. 53–68. omnipax, curitiba, pr (2013) google scholar  nobile, m.s., citrolo, a.g., cazzaniga, p., besozzi, d., mauri, g.: a memetic hybrid method for the molecular distance geometry problem with incomplete information. in: 2014 ieee congress on evolutionary computation (cec), pp. 1014–1021. ieee (2014) google scholar  schlick, t.: molecular modeling and simulation: an interdisciplinary guide: an interdisciplinary guide, vol. 21. springer, new york (2010). https://doi.org/10.1007/978-1-4419-6351-2 sit, a., wu, z.: solving a generalized distance geometry problem for protein structure determination. bull. math. biol. 73(12), 2809–2836 (2011) article  mathscinet  math  google scholar  souza, m., gonçalves, d.s., carvalho, l.m., lavor, c., liberti, l.: a new algorithm for a class of distance geometry problems. in: 18th cologne-twente workshop on graphs and combinatorial optimization (2020) google scholar  yang, x.s.: engineering optimization: an introduction with metaheuristic applications. wiley (2010) google scholar  download references acknowledgements douglas o. cardoso acknowledges the financial support by the foundation for science and technology (fundação para a ciência e a tecnologia, fct) through grant uidb/05567/2020, and by the european social fund and programs centro 2020 and portugal 2020 through project centro-04-3559-fse-000158. author information authors and affiliations celso suckow da fonseca federal center of technological education, rio de janeiro, rj, brazil sarah ribeiro lisboa carneiro, luís tarrataca & laura s. assis department of statistics and applied mathematics, federal university of ceará, fortaleza, ceará, brazil michael ferreira de souza smart cities research center, polytechnic institute of tomar, tomar, portugal douglas o. cardoso authors sarah ribeiro lisboa carneiroview author publications search author on:pubmed google scholar michael ferreira de souzaview author publications search author on:pubmed google scholar douglas o. cardosoview author publications search author on:pubmed google scholar luís tarratacaview author publications search author on:pubmed google scholar laura s. assisview author publications search author on:pubmed google scholar corresponding author correspondence to laura s. assis . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper carneiro, s.r.l., de souza, m.f., cardoso, d.o., tarrataca, l., assis, l.s. (2023). a custom bio-inspired algorithm for the molecular distance geometry problem. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_12 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords combinatorial optimization genetic algorithm distance geometry molecular structure publish with us policies and ethics profiles michael ferreira de souza view author profile douglas o. cardoso view author profile laura s. assis view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature machine teaching: an explainable machine learning model for individualized education | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper machine teaching: an explainable machine learning model for individualized education conference paper first online: 12 october 2023 pp 321–336 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) machine teaching: an explainable machine learning model for individualized education download book pdf download book epub eduardo vargas ferreira  orcid: orcid.org/0000-0002-9241-97789,10 & ana carolina lorena  orcid: orcid.org/0000-0002-6140-571x9  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 585 accesses 2 citations abstract education is the single most important investment that people can make in their futures, and since the universal declaration of human rights in 1948, the goal of achieving universal education has been on the international agenda. in this regard, there is no doubt that the web has had a profound impact on making education both universally available and more relevant. nevertheless, online courses are based on “static” learning material (one-size-fits-all). for that reason, it is not straightforward to assess learning with a great number of learners who differ considerably in their educational background, engagement styles, and cognitive skills. in this work, we aim to address these aforementioned challenges by proposing an explainable machine learning algorithm for personalizing web-based education systems. the method has a “deep” architecture mimicking the information representation structure in human brains, and it is continuously adapted based on the signals of the students, understanding their performance through micro-steps and maximizing the learning outcome. while our methodology is general and can be applied in numerous scenarios, we demonstrate its performance by a real case study which comprises a non-mandatory, standardized exam, that evaluates high school students in brazil. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others integration of machine learning in education: challenges, issues and trends chapter © 2022 transforming classroom instruction: embracing machine learning-assisted teaching pedagogy chapter © 2024 exploring educational horizons: a comprehensive review of finding teacher and interview-based learning experiences using machine learning chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computers and education instructional theory learning and instruction learning algorithms learning theory machine learning 1 introduction education is the single-most important investment that people can make in their futures, and the best way to beat poverty, promoting more inclusive societies [40]. even in the worst circumstances, it is the basic building block to give people confidence face to future. nevertheless, the cost of attaining higher education has skyrocketed over the past few decades [39]. faced with this, the web has enabled one of the most prominent developments in higher education in recent years, not only taking classes out of the traditional classroom and making them open, accessible and dynamic [26], but also minimizing the lack of quality due to poor conditions of schools and the unavailability of adequately trained teachers. in 2011, the deployment of massive open online courses (moocs), such as edx consortium [8] and coursera [19], exploded around the world and rapidly became the current trend for online learning [25], available to all those who wish to learn regardless of educational background and individual needs. despite their importance and a high degree of interest, there are significant limitations of moving from open content towards open educational practices, as described by [12]. since online courses are based on “static” learning material, i.e. one-size-fits-all program and content [29], it is not straightforward to assess learning with those great number of students who differ considerably in their educational background, engagement styles and cognitive skills, which might merely lead to surface learning. in this spirit, machine learning plays an important role to open up the “black box of learning”, giving us deeper and fine-grained understandings of how learning actually happens, creating a relevant and transformative approach for making the education both universally available and more relevant. this paper aims to address these aforementioned challenges by proposing an explainable machine learning algorithm for personalizing web-based education systems. the key feature of the method is automatically tailoring the content to the proficiency level of the examinees by using an unsupervised deep learning algorithm, such that the next item (or set of items) selected to be administered is optimally informative for a given person. the items’ personalized recommendations are elicited by using an post hoc explanation approach. this paper is structured as follows: sect. 2 presents some related work. section 3 presents the problem formulation and describes the proposed method to deal with personalized learning. section 4 explains the scenario of the simulations and results. section 5 presents some results on the analysis of real data from a national exam for students in brazil. section 6 discusses and concludes the paper. 2 related work one of the most important latent variable models, the item response theory (irt) models [15], have been very successful in educational research [23]. in this context, latent traits refer to unobservable characteristics of individuals that may influence their performance in educational settings. by considering those abilities, it is possible to gain insights into the complex interplay between individual differences and educational outcomes (item responses), allowing for more targeted interventions and instructional approaches tailored to students’ unique needs and characteristics. considering the rich and broad corpus of findings devoted to irt models [22], the two-parameter logistic (2pl) [7] is one of the most prominent. let a response of a person j (\(j=1,\ldots ,n\)) to the item i (\(i = 1,\ldots ,i\)) be represented by a stochastic vector \(\textbf{x}_{j} = (x_{1j},x_{2j},\ldots ,x_{ij})\) with elements $$\begin{aligned} x_{ij} = {\left\{ \begin{array}{ll} 1, &{} \text {if person } j \text { gives a correct response in item } i \\ 0, &{} \text {otherwise.} \end{array}\right. } \end{aligned}$$ selecting a response alternative is the result of the complex interaction between the characteristics of the test item and the latent trait of the person, \(y_j\), which is commonly known as ‘ability’ (or proficiency) in the framework of educational and psychological measurement [23]. the probability of a correct response for an item i by individual j \((x_{ij} = 1)\), is given by: $$\begin{aligned} p(x_{ij} = 1 | y_j, \mathbf {\theta }) = \frac{e^{\left( w_i y_j a_i\right) }}{1 + e^{\left( w_i y_j a_i\right) }}, \end{aligned}$$ (1) where \(\mathbf {\theta } = \left( \textbf{w}, \textbf{a}\right) \) groups the model parameters. \(\textbf{w} = \left( w_1,\ldots ,w_i\right) \) is the vector of items discrimination parameters, which reflects the capability of a generic item i to discriminate between individuals with different levels of ability. the difficulty parameters of the items are \(d_i = a_i/w_i\). in irt, each item is assumed to have its own unique item characteristic curve (icc) and it provides valuable information about the properties of an item. the left panel of fig. 1 presents three examples of iccs with varying w-parameters and the right panel presents iccs with varying a-parameters. greater values of w represent curves with higher slopes. therefore, even individuals with almost the same latent traits will be well discriminated, because they will produce different response probabilities. on the other hand, higher values of a indicate a proportionally higher level of difficulty of the item, since greater latent traits are needed to obtain \(p(x_{ij} = 1| y_j, \mathbf {\theta })\). fig. 1. item characteristic curves for the 2pl with w-parameters 0.3, 1.0, 1.7 (left) and a-parameters \(-1,0,1\) (right), respectively. full size image while the 2pl is often easy to comprehend, interpret and can have various important measurement properties, the complexities of the brain are even more challenging. many pieces of knowledge with different “wiring” are required to identify the correct response. for example, the paper [27] describes fourth-grade reading as follows “reading includes developing a general understanding of written text, thinking about texts, and using texts for different purposes”, while [9] have identified 15 factors (dimensions) related to language ability. in recent years, the study of nervous systems, minds and other complex phenomena through artificial intelligence systems has shown a successful performance and defined state of the art results in many fields [11, 24, 32]. furthermore, there is an increasing effort to apply such methods to problems in psychometry and education [18]. however, for the greater part, this attention is focused only on predicting student performance by using classification techniques, such as support vector machines [14, 41], or by matrix factorization [36, 38], etc. furthermore, these algorithms often act as a “black box” and do not provide detailed information about the reasons for a given decision. the field of explainable ai (xai) has produced an abundance of explanation techniques, from the simplest, such as intrinsically interpretable models [10], to more sophisticated ones, which assign a local interpretation, according to the specific characteristics of the input. among them, one of the most prominent is the layer-wise relevance propagation (lrp) [4]. it belongs to the propagation-based class, which assumes that the prediction was produced by a neural network, and takes advantage of this structure to produce an explanation, starting from the last layer and backtracking to the initial [33]. that is, from the weights of the network and its activation functions, the lrp propagates the contribution of each neuron to the input layer. nevertheless, so far, research on explanation methods has mainly concentrated on supervised learning. in a technical sense, our methodology can be seen as an extension of 2pl model by considering more interaction layers between latent abilities, modeled by a deep boltzmann machine. furthermore, it is a new framework that can explain an unsupervised model, such as 2pl, rewritten as neural networks or “neuralizing it”, and then applying a post hoc explanation such as lrp. 3 proposed method over the past several years, big data methods, including but not limited to the use of machine learning (ml), have been very successful in computer science applications, and there is increasing effort to apply such methods to problems in psychometry and education [18]. this includes methods that have a “deep” architecture mimicking the information representation structure in human brains. recently, deep boltzmann machines (dbms [31]) have risen to prominence due to their capacity of learning efficient representations of seemingly complex data. dbm is a probabilistic undirected graphical model of interconnected units that learn a joint probability density over these units by adapting connections between them. formally, a dbm can be used to specify the relation between the level of a latent trait and item responses, as follows $$\begin{aligned} p\left( \textbf{x},\textbf{y}, z;\mathbf {\theta }\right) = & {} \frac{e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })}}{g},\\ g = & {} \displaystyle \int e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })} \text { d}\textbf{x} \text { d}\textbf{y} \text { d}z, \nonumber \end{aligned}$$ (2) where \(\textbf{x}\) are the item responses, \(\textbf{y}\) is the first hidden layer (m-dimensional) and z is the second hidden layer (unidimensional). the term in the denominator is called the partition function, which normalizes the probability distribution by integrating over all possible values of \(\textbf{x}\), \(\textbf{y}\) and z, and $$\begin{aligned} f(\textbf{x},\textbf{y}, z;\mathbf {\theta }) = {} & {} \frac{1}{2} \left( \textbf{y} \textbf{b}\right) ^t\mathbf {\sigma }\left( \textbf{y} \textbf{b}\right) ^t \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w} \textbf{x} \\ {} + & {} \frac{1}{2} \frac{\left( z c\right) ^2}{\sigma ^2} \frac{z}{\sigma } \textbf{v} \textbf{y} \textbf{x}^t \textbf{a}, \end{aligned}$$ where \(\mathbf {\theta } = \left( \mathbf {\sigma },\textbf{w}, \textbf{v},\textbf{a},\textbf{b}, c \right) \) groups the model parameters. \(\textbf{w} \in m\times i\) corresponds to the items discrimination parameters matrix (the slope of the surface). the parameters b and c are the bias into hidden units. \(\mathbf {\sigma }\) is a diagonal matrix of precisions. figure 2 shows how the variables are interconnected. the standard 2pl model corresponds to having one unique output \(y_1\). the dbm extends this model by allowing more output dimensions (\(y_2\) to \(y_m\)) and also by providing a combined view of those dimensions in z. fig. 2. relationship between variables in the dbm. full size image almost all models for latent traits have unidentified parameters due to scale invariance, this occurs because of overparametrization. to rule out it, we fixed the matrix of precisions to the identity matrix, as in [6]. many other impressive results fixing those parameters to unity have been published [16, 20, 30]. the probability of the latent traits is given by: $$\begin{aligned} p \left( y = y | \textbf{x}, z;\mathbf {\theta }\right) = & {} \mathcal {n}_r\left( \textbf{y} | \textbf{w}\textbf{x} + \frac{\textbf{v}^tz}{\sigma } + b, \textbf{i}_m \right) \end{aligned}$$ (3) $$\begin{aligned} p \left( z = z | \textbf{y};\mathbf {\theta }\right) = & {} \mathcal {n}\left( z|\textbf{v}\textbf{y} + c, \sigma ^2\right) \end{aligned}$$ (4) where \(\mathcal {n}(\cdot | \mu , \sigma ^2)\) denotes the gaussian probability density function with mean \(\mu \) and variance \(\sigma ^2\) and \(\mathcal {n}_r(\cdot | \mu , \sigma ^2)\) denotes the rectified gaussian distribution. thus, it is possible to demonstrate that eq. 2 is structurally similar to the expression for 2pl (eq. 1). detailed proofs of these results are given in the appendix 1. this is a first result presented in this work, that is, that dbms generalize a 2pl model. 3.1 explaining dbms parameters in contrast to their very high flexibility, in dbms it is often difficult to estimate the item parameters in high dimensional spaces and provide a comprehensive explanation along several dimensions of knowledge, limiting their broader applicability. this fact stands a huge drawback in ml models, since their nested non-linear structure makes them highly non-transparent. not surprisingly, this issue has recently received a lot of attention in the literature [4, 5, 34]. the goal of the explanation is to assign the importance of each input (item) for the output (latent trait). the multidimensional item discrimination can be interpreted through the \(\vert w_i \vert \), and can be seen as sensitivity analysis. therefore, the higher the value of \(\vert w_i \vert \), the higher the discrimination power of that item, independently from the assumed underlying latent structure. the multidimensional item difficulty can be interpreted through \(d_i = a_i/\vert w_i \vert \). a limitation of this approach is that it is an explanation of the latent trait variation rather than the value itself. in other words, it answers the question “which items lead to increase/decrease of latent trait when changed?”. the aforementioned weakness has led to the development of a more precise technique, that explains “which items contribute much to the latent trait”. to achieve this, we use the layer-wise relevance propagation (lrp) [4]. let j and k be indices for the neurons in layers l and \(l + 1\), respectively, and assume that the output of function f(x) has been propagated through from the upper layer to layer \(l+1\). the lrp defines the “message” \(r_{j\leftarrow k}\) as the redistribution of relevance \(r_k\) to the neurons in the previous layer: $$\begin{aligned} r_{j\leftarrow k} = \frac{y_j\cdot w_{jk}}{\sum _{j}y_j \cdot w_{jk}}r_{k}. \end{aligned}$$ the overall relevance of j is then obtained by calculating \(r_j = \sum _k r_{j\leftarrow k}\). note that, by definition, \(\sum _i r_i = \ldots = \sum _j r_j = \sum _k r_k = \ldots = f(x)\), that is, it satisfies the conservation (or completeness) property, since for all \(x \in \mathcal {x}\), we have \(\textbf{1}^t \mathcal {e}_f = f(x)\). the process is outlined in fig. 3. first, the dbms estimate the latent trait, f(x), based on the input. then, lrp is applied to explain the prediction, that is, to indicate which items are more or less relevant. 3.2 recommender system when a student finishes a test, there are two fundamental and commonly asked questions: “what did i do wrong?” and “how can i improve it?” the former is about identifying specific breakdowns; whereas the latter focuses on creating knowledge. we are asking these two questions to introduce our methodology. the algorithm, at each time-step, reads the input (items responses) and returns an output (student proficiency). once the current state is outputted, it is aggregated through the recommender system to define the next items to be administered, which are sorted according to items with the same difficulty parameter (vector \(\textbf{d}\)) and lesser relevance (defined by the lrp) answered by the student, which can be viewed as a content-based recommender system focusing on items where the student needs more attention/learning [3]. next, the sequence in which the items are presented is considered. the system intentionally incites experiences of like-minded students failure, grading the stimuli to be presented accordingly. in such a setting, the approach comprises collaborative filtering-based methods [35], by performing failure-based learning collaboratively. various research posits fault-stages as a strategic way to identify possible breakdowns and engender higher-order knowledge [17, 21, 37]. this, in turn, has pedagogical benefits because it allows students to challenge their existing (incomplete/inaccurate) mental models and activate cognitive mechanisms for deep learning. fig. 3. explaining predictions of an ai system using lrp. full size image formally, we have a set of latent traits \(\mathcal {z} = \{z_j\}\), \(j=1,\ldots ,n\), and a set-based responses \(\mathcal {x} = \{\textbf{x}_i\}\), \(i = 1,\ldots ,i\). the aim of the system is to create a list of the best l recommendations called stimuli, \(\mathcal {s} = \{s_k\}\), \(k = 1,\ldots ,l\), for an examinee with ability \(z_a \in \mathcal {z}\). the formulation of the algorithm is based on two steps. the first step involves selecting a set of items based on the difficulty parameter (note that the latent trait is in the same unit as the parameter). these items are chosen to be similar to those with lesser relevance (redistributed “message” given \(z_a\)) that have been answered by the student, such as: $$\begin{aligned} \mathcal {s}_a = \{i \ :\ d_i \delta \le z_a \le d_i + \delta \}, \end{aligned}$$ (5) where \(\delta \) is determined by the user and defines the homogeneity of items presented to the student. for the second step, failure is an intentional part. the items are sorted by the average of correct responses based on the k nearest neighbors of \(z_a\). consequently, we are able to control the examinee’s exposure to failure. this strategy allows better opportunities for the student deepen conceptual understanding through items suitable to their proficiency level, and increase the problem-solving capacity through failure-based learning. 4 simulation study to demonstrate our arguments and set up the subsequent discussion, in this section two simulated examples will be presented. the first example will investigate how the true values of latent traits and items are recovered in different situations, involving sample sizes and number of items. in the second example, we show how the test items are sensitive to examinee differences on a number of correlated dimensions (latent traits). the data is designed to mirror a real-life scenario where items have different characteristics and latent traits exhibit multiple dimensions with varying correlations. these initial experiments aim to show the dbm model is indeed a precise estimator of latent characteristics, generalizing the 2pl model to more latent dimensions. 4.1 example 1 the datasets were generated containing responses of \(n = 500, 2000\) and 5000 examinees to \(i = 50, 100\) and 250 items. the proficiency parameters are normally distributed, by model assumption. the values of discrimination and difficulty parameters were generated from a uniform distribution such that \(w \in (0.4,1.8)\) and \(a \in (-2.4,2.4)\). as mentioned earlier, the proposed model has a structural similarity with the two-parameter logistic (2pl) item response model [7] if we consider one hidden layer and \(m = 1\). thus, the responses were simulated using the 2pl parameter values. each implementation involved 10 replications. using these data, the parameters were estimated based on the dbm model. to carry out comparisons, the average root-mean-squared error and correlation between the true and estimated parameters for the unidimensional models were obtained, and they are summarized in table 1. it can be seen that the parameter estimates are almost similar and appear to have converged to reasonable values, with high correlation to the ground truth values and low rmse in all scenarios. table 1. average root-mean-squared error and correlation between the true (2pl’s) and estimated parameters (dbm’s) based on 10 replications.full size table 4.2 example 2 for the second example, the response datasets were produced with three levels of correlation between latent trait dimensions (\(\rho = 0.2,0.6\) and 0.8). the data is considered more unidimensional when the value of \(\rho \) is closer to 1, since all latent traits will represent the same piece of knowledge. the proficiency parameters were drawn from a multivariate normal distribution. two datasets with known characteristics are used here. dataset 1 was designed to represent more realistic conditions where the difficulty parameters were confounded with dimensionality, such that the easier items tend to have high values for \(w_1\) and more difficult items tend to have high values for \(w_2\). thus, the discrimination parameters, \(\textbf{w}_i = (w_{i1},w_{i2})\), were ranged from \(\textbf{w}_1 = (1.8, 0.4)\) to \(\textbf{w}_i =(0.4, 1.8)\) and a was ranged from \(-2.4\) (for \(w_{i1} = 0.4, w_{i2} = 1.8\)) to 2.4 (for \(w_{i1} = 1.8, w_{i2} = 0.4\)). this pattern of systematic shift in the item’s parameters is the same as those also found in [28] and [1]. dataset 2 was generated according to [2], considering three dimensions in the latent trait to obtain the simulated response. the results are presented in fig. 4. in contrast to the previous example where the dimension was set at 1, in cases involving multiple dimensions, there are additional discrimination parameters to consider. consequently, only a correlation metric was employed. overall, the model recovered the parameters precisely, particularly when the model considered was two-dimensional in dataset 1 and three-dimensional in dataset 2 (matching the number of latent traits used to generate the responses). it is worth noting that when a one-dimensional model was employed with multidimensional data, we observed poor estimates of the discrimination parameters. furthermore, the results were slightly worse when the correlation between competencies was 0.8, a quite high correlation value. fig. 4. results of correlation and rmse for the data sets 1 and 2. full size image 5 real data analysis in the following, the operation of the recommender system is illustrated by a real example which comprises the brazilian national high school examination [13]. this is a non-mandatory, standardized exam, that evaluates high school students in brazil. every year, more than 5 million students take the test, which is composed of 180 multiple-choice items, coded as zero (incorrect) or one (correct), on four main content areas referred to as natural sciences (nc), human sciences (hs), languages and codes (lc) and mathematics (mt). after 2009, its importance has increased since enem scores became accepted for admission by more than 23 universities and other institutions as well as for certification for a high school degree. this study selected a sample of 100.000.000 responses of the 2018 enem datasetfootnote 1 (among those that answered all 180 items of the test). the number of test dimensions (latent traits) was inferred from the presence of a single factor in all items and evidence of distinct clusters of local dependencies formed by other factors, based on the bayesian information criterion. the results indicate that the data set’s dimensionality is equal to 4. thus, we considered a model with four latent traits. furthermore, the generative p-value for the latent traits was 0.69 and for the overall items it was 0.66. these results suggest that the selected model (in terms of the number of dimensions) is appropriate for the data set. figure 5a illustrates the distribution of the latent trait according to each of the four dimensions of knowledge. additionally their correlation is illustrated in fig. 5b. note that the distribution of latent traits are complementary (compensatory structure), reinforcing their multidimensional nature. in other words, a high ability on one dimension can compensate for a low ability on another dimension. for instance, having a high level of proficiency in ‘reading comprehension’ can offset a lower proficiency in ‘basic mathematics’. in addition, it is possible to assemble all dimensions into one unidimensional representation as described in fig. 5c. thereby, we assess the overall latent trait to compare students. moreover, this information is used to make recommendations for suitable new items. after applying the recommendation system, many improvements emerged after applying a here called step 2, where items are recommended according to eq. 5. it achieved the personalized items based on the examinee’s neighborhood, dug further into the quality of recommendations to determine the degree of fault-stages, as shown in fig. 6(left), instead of simply choosing according to \(p \left( x_{ij} = 1 | \textbf{y}; \mathbf {\theta }\right) \), as depicted in fig. 6(right). on the left, each point expresses the proportion of items correctly answered, chosen according to their expected probability of failure (based on the k nearest neighbors responses). in contrast, on the right, the experiences of like-minded student is ignored. one would expect the box-plots to be within the horizontal dashed lines, that is, if one sets as goal an average of 0.2 (20%) of correct responses, the students responses should be around this value. this clearly happens more effectively when step 2 is used (left of fig. 6), since the boxplots presents a lower variation and are around the horizontal lines. in practice, this means being able to recommend items that challenge the student at any level, from items with a low chance of solution (e.g., expected average = 0.2) to high chance of solving (e.g., expected average = 0.8). this information is more refined than simply considering \(p \left( x_{ij} = 1 | \textbf{y}; \mathbf {\theta }\right) \), which does not yield good results, especially for items that do not challenge the student (with expected average of correct responses from 0.4 onwards). fig. 5. the distribution of the latent trait according to each dimension (panel a), its correlation (panel b), as well as the assemble unidimensional representation (panel c). full size image fig. 6. results of recommender system step 2. the left panel refers to the distribution of the observed average of correct responses from items chosen according to responses of the nearest k neighbor. the right panel shows the distribution of the observed average of correct responses from items chosen according to \(p \left( x | \textbf{y}; \mathbf {\theta }\right) \). full size image 6 conclusion and discussion in this paper, we have reformulated the two-parameter logistic model as a neural network, with the deep boltzmann machine (dbm) being defined as its extension. this makes the model explainable based on the underlying interpretation of item response theory parameters. but interpretability is further extended by the use of layer-wise relevance propagation, which propagates the prediction backwards through the neural network to the input features. based on that model, we have developed an algorithm for individualized education. the approach is continuously adapted based on substantively complex constructs and recommends new items according to their characteristics and students’ latent traits. the explainable algorithm allows grasping what made the algorithm arrive at a particular decision, playing an important role of open up the “black box of learning” of both algorithm and student. in the second case, this interpretable feedback is a valuable tool for verifying and improving the items to be recommended, as well as generating higher-order knowledge. we illustrate our procedure by analyzing two simulated data sets, where the dbm model recovered the irt parameters precisely in different situations, involving sample sizes, number of items and dimensions. furthermore, we employ the method in a real-world example which comprises the brazilian national high school examination. overall, our approach produced a better determination of the probability of answering an item correctly as compared with using simply \(p \left( x | \textbf{y}; \mathbf {\theta }\right) \). this performance arose due to the highly detailed subpopulation selected based on responses of the others examinees. however, given the student and set of items, it is not obvious what degree the fault-stages serves as catalysts for deep and constructive learning. thus, more research is needed to ascertain it. in addition, there are various possible applications beyond the one presented here, particularly in more complex domains, such as massive open online courses and e-commerce. notes 1.dataset available at http://portal.inep.gov.br/microdados. references ackerman, t.a.: unidimensional irt calibration of compensatory and noncompensatory multidimensional items. appl. psychol. meas. 13, 113–127 (1989) article  google scholar  ackerman, t.a.: multidimensional item response theory modeling. in: contemporary psychometrics: a festschrift for roderick p. mcdonald (2013) google scholar  ansari, a., essegaier, s., kohli, r.: internet recommendation systems. j. mark. res. 37, 363–375 (2000) article  google scholar  bach, s., binder, a., montavon, g., klauschen, f., müller, k.r., samek, w.: on pixel-wise explanations for non-linear classifier decisions by layer-wise relevance propagation. plos one 10, e0130140 (2015) article  google scholar  baehrens, d., schroeter, t., harmeling, s., hansen, k., mueller, k.r.: how to explain individual classification decisions. j. mach. learn. res. 11, 1803–1831 (2010) mathscinet  math  google scholar  béguin, a.a., glas, c.a.: mcmc estimation and some model-fit analysis of multidimensional irt models. psychometrika 66, 541–561 (2001). https://doi.org/10.1007/bf02296195 article  mathscinet  math  google scholar  birnbaum, a.: some latent trait models and their use in inferring an examinee’s ability. in: statistical theories of mental test scores, pp. 397–479 (1968) google scholar  breslow, l., pritchard, d.e., deboer, j., stump, g.s., ho, a.d., seaton, d.t.: studying learning in the worldwide classroom: research into edx’s first mooc. res. pract. assess. 8, 13–25 (2013) google scholar  carroll, j.b., et al.: human cognitive abilities: a survey of factor-analytic studies. vol. 1, cambridge university press (1993) google scholar  craven, m.w., shavlik, j.w.: extracting tree-structured representations of trained networks. in: proceedings of the 8th international conference on neural information processing systems, nips 1995, pp. 24–30. mit press, cambridge, ma, usa (1995) google scholar  dahlkamp, h., kaehler, a., stavens, d., thrun, s., bradski, g.: self-supervised monocular road detection in desert terrain, pp. 25–32 (2016) google scholar  ehlers, u.d.: extending the territory: from open educational resources to open educational practices. j. open flex. distance learn. 15, 1–8 (2011) google scholar  enem: exame nacional do ensino médio (2019). https://enem.inep.gov.br/ gray, g., mcguinness, c., owende, p.: an application of classification models to predict learner progression in tertiary education. in: souvenir of the 2014 ieee international advance computing conference, iacc 2014, pp. 549–554 (2014) google scholar  hambleton, r.k., swaminathan, h.: item response theory: principles and applications. springer, dordrecht (1985). https://doi.org/10.1017/cbo9781107415324.004 book  google scholar  hinton, g.e.: a practical guide to training restricted boltzmann machines. in: montavon, g., orr, g.b., müller, k.-r. (eds.) neural networks: tricks of the trade. lncs, vol. 7700, pp. 599–619. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-35289-8_32 chapter  google scholar  kapur, m., bielaczyc, k.: designing for productive failure. j. learn. sci. 21, 45–83 (2012) article  google scholar  kdd cup: educational data minding challenge (2010). http://pslcdatashop.web.cmu.edu/kddcup/downloads.jsp kizilcec, r.f., piech, c., schneider, e.: deconstructing disengagement: analyzing learner subpopulations in massive open online courses. in: proceedings of the third international conference on learning analytics and knowledge, lak 2013, pp. 170–179 (2013) google scholar  krizhevsky, a.: learning multiple layers of features from tiny images. technical report, department of computer science university of toronto (2009) google scholar  lam, r.: what students do when encountering failure in collaborative tasks. npj sci. learn. 4, 6 (2019). https://doi.org/10.1038/s41539-019-0045-1 article  google scholar  van der linden, w.j.: handbook of item response theory, three volume set. chapman and hall/crc, boca raton, fl: crc press, 2015 (february 2018). https://doi.org/10.1201/9781315119144. https://www.taylorfrancis.com/books/9781315119144 lord, f.m.: applications of item response theory to practical testing problems (2014). https://doi.org/10.4324/9780203056615 lu, c., tang, x.: surpassing human-level face verification performance on lfw with gaussianface. in: proceedings of the 29th aaai conference on artificial intelligence, pp. 3811–3819 (2015) google scholar  pappano, l.: the year of the mooc. the new york times 2(12) (2012) google scholar  pank, c.m.: online education. ajn am. j. nurs. 107(5), 74–76 (2007) article  google scholar  perie, m., grigg, w., donahue, p.: the nation’s report card: reading, 2005. nces (2006-451). technical report (2005) google scholar  reckase, m.d.: the difficulty of test items that measure more than one ability. appl. psychol. measure. 9, 401–412 (1985). https://doi.org/10.1177/014662168500900409 article  google scholar  romero, c., ventura, s., delgado, j.a., de bra, p.: personalized links recommendation based on data mining in adaptive educational hypermedia systems. in: duval, e., klamma, r., wolpers, m. (eds.) ec-tel 2007. lncs, vol. 4753, pp. 292–306. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-75195-3_21 chapter  google scholar  salakhutdinov, r.: learning deep generative models. ph.d. thesis, university of toronto (2009) google scholar  salakhutdinov, r., hinton, g.: deep boltzmann machines. in: van dyk, d., welling, m. (eds.) proceedings of the twelfth international conference on artificial intelligence and statistics. proceedings of machine learning research, hilton clearwater beach resort, clearwater beach, florida usa, 16–18 april 2009, vol. 5, pp. 448–455. pmlr (2009). https://proceedings.mlr.press/v5/salakhutdinov09a.html schölkopf, b., williamson, r., smola, a., shawe-taylor, j., platt, j.: support vector method for novelty detection. in: advances in neural information processing systems, vol. 12 (2000) google scholar  selvaraju, r.r., cogswell, m., das, a., vedantam, r., parikh, d., batra, d.: grad-cam: visual explanations from deep networks via gradient-based localization. in: 2017 ieee international conference on computer vision (iccv), pp. 618–626 (2017). https://doi.org/10.1109/iccv.2017.74 srinivasan, v., lapuschkin, s., hellge, c., muller, k.r., samek, w.: interpretable human action recognition in compressed domain. in: proceedings of the ieee international conference on acoustics, speech and signal processing (icassp), pp. 1692–1696 (2017). https://doi.org/10.1109/icassp.2017.7952445 su, x., khoshgoftaar, t.m.: a survey of collaborative filtering techniques. in: advances in artificial intelligence, pp. 115–129 (2009). https://doi.org/10.1155/2009/421425 sweeney, m., rangwala, h., lester, j., johri, a.: next-term student performance prediction: a recommender systems approach. j. educ. data min. 8, 22–51 (2016) google scholar  tawfik, a.a., rong, h., choi, i.: failing to learn: towards a unified design approach for failure-based learning. educ. technol. res. dev. 63(6), 975–994 (2015). https://doi.org/10.1007/s11423-015-9399-0 article  google scholar  thai-nghe, n., et al.: multi-relational factorization models for predicting student performance. in: proceedings of the kdd workshop on knowledge discovery in educational data (2011) google scholar  the white house: making college affordable (2016). https://obamawhitehouse.archives.gov/issues/education/higher-education/making-college-affordable unesco: global education first initiative (2016). http://www.unesco.org/new/en/gefi/about/ yu, h., lo, h., hsieh, h.: feature engineering and classifier ensemble for kdd cup 2010. j. mach. learn. res. 1, 1–16 (2010) google scholar  download references acknowledgements the authors thank the financial support of fapesp (grant 2021/06870-3) and cnpq. author information authors and affiliations aeronautics institute of technology, são josé dos campos, brazil eduardo vargas ferreira & ana carolina lorena federal university of paraná, curitiba, brazil eduardo vargas ferreira authors eduardo vargas ferreiraview author publications search author on:pubmed google scholar ana carolina lorenaview author publications search author on:pubmed google scholar corresponding author correspondence to eduardo vargas ferreira . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi appendix 1 appendix 1 1.1 6.1 proof of the equations proof of the eq. 2: $$\begin{aligned} p \left( x_i| \textbf{y};\mathbf {\theta }\right) = & {} \displaystyle \frac{e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })}}{\displaystyle \sum _x e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })}} \\ = & {} \frac{exp\left\{ \displaystyle -\frac{1}{2} \left( \textbf{y} \textbf{b}\right) ^t\mathbf {\sigma }\left( \textbf{y} \textbf{b}\right) ^t + \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w}_{.i} x_i \frac{1}{2} \frac{\left( z c\right) ^2}{\sigma ^2} + \frac{z}{\sigma } \textbf{v} \textbf{y} + x_i a_i \right\} }{\displaystyle \sum _x exp\left\{ \displaystyle -\frac{1}{2} \left( \textbf{y} \textbf{b}\right) ^t\mathbf {\sigma }\left( \textbf{y} \textbf{b}\right) ^t + \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w}_{.i} x_i \frac{1}{2} \frac{\left( z c\right) ^2}{\sigma ^2} + \frac{z}{\sigma } \textbf{v} \textbf{y} + x_i a_i \right\} }\\[15pt] = & {} \frac{exp\left\{ \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w}_{.i} x_i + x_i a_i\right\} }{\displaystyle \sum _{x_i} exp \left\{ \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w}_{.i} x_i + x_i a_i\right\} } \ (\text {the only terms that depend on x}) \end{aligned}$$ proof of the eq. 3: $$\begin{aligned} p \left( y| \textbf{x}, z;\mathbf {\theta }\right) = & {} \displaystyle \frac{e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })}}{\displaystyle \int _u e^{-f(\textbf{x},\textbf{u}, z;\mathbf {\theta })}d u} \\ = & {} \frac{exp\left\{ \displaystyle -\frac{1}{2} \left( \textbf{y} \textbf{b}\right) ^t\mathbf {\sigma }\left( \textbf{y} \textbf{b}\right) ^t + \textbf{y}\mathbf {\sigma }^{1/2} \textbf{w} \textbf{x} + \frac{z}{\sigma } \textbf{v} \textbf{y} \right\} \mathbb {i}(\textbf{y} \ge 0)}{\displaystyle \int _\textbf{u} exp\left\{ \displaystyle -\frac{1}{2} \left( \textbf{u} \textbf{b}\right) ^t\mathbf {\sigma }\left( \textbf{u} \textbf{b}\right) ^t + \textbf{u}\mathbf {\sigma }^{1/2} \textbf{w} \textbf{x} + \frac{z}{\sigma } \textbf{v} \textbf{u} \right\} \mathbb {i}(\textbf{u} \ge 0) d \textbf{u}}\\ = & {} \mathcal {n}_r\left( \textbf{y} | \textbf{w}\textbf{x} + \frac{\textbf{v}^tz}{\sigma } + b, \mathbf {\sigma } \right) \end{aligned}$$ proof of the eq. 4: $$\begin{aligned} p \left( z| \textbf{y};\mathbf {\theta }\right) = & {} \displaystyle \frac{e^{-f(\textbf{x},\textbf{y}, z;\mathbf {\theta })}}{\displaystyle \int _u e^{-f(\textbf{x},\textbf{u}, z;\mathbf {\theta })}d u} \\ = & {} \frac{exp\left\{ \displaystyle \frac{1}{2} \frac{\left( z c\right) ^2}{\sigma ^2} + \frac{z}{\sigma } \textbf{v} \textbf{y} \right\} }{\displaystyle \int _\textbf{u} exp\left\{ \displaystyle \frac{1}{2} \frac{\left( u c\right) ^2}{\sigma ^2} + \frac{u}{\sigma } \textbf{v} \textbf{y} \right\} d \textbf{u}}\\ = & {} \mathcal {n}\left( z|\textbf{v}\textbf{y} + c, \sigma ^2\right) \end{aligned}$$ rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper ferreira, e.v., lorena, a.c. (2023). machine teaching: an explainable machine learning model for individualized education. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_21 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_21 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords explainable artificial intelligence deep boltzmann machines education publish with us policies and ethics profiles ana carolina lorena view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature multilingual extractive summarization: investigating state-of-the-art methods for english and brazilian portuguese | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper multilingual extractive summarization: investigating state-of-the-art methods for english and brazilian portuguese conference paper first online: 30 january 2025 pp 212–223 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) multilingual extractive summarization: investigating state-of-the-art methods for english and brazilian portuguese download book pdf download book epub germano antonio zani jorge9, davi alves bezerra9, clarissa castellã xavier10 & … thiago alexandre salgueiro pardo9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 411 accesses 1 citation abstract automatic text summarization (ats) is a natural language processing (nlp) task essential for handling large volumes of information. ats can be classified into two main types: extractive and abstractive. extractive summarization selects sentences or phrases directly from the source text(s), while abstractive summarization generates new sentences that try to capture the original meaning of the source text(s). this paper describes our efforts to perform extractive single-document summarization in multilingual contexts. although various summarization methods, such as presumm and histruct+, have shown promising results on english corpora like cnn/dm, there is a significant gap in applying these methods to other languages, especially brazilian portuguese. additionally, these summarizers were evaluated with traditional metrics like rouge, which has limitations as it primarily measures superficial text overlap. to fill these gaps, we evaluate the effectiveness of these state-of-the-art methods on the cstnews corpus (with news texts in brazilian portuguese) employing rouge and the recent blanc metric, which measures how much the generated summary aids a pre-trained language model (like bert) in understanding the document. our contributions include the results and comparison of adapted models, the discussion of the blanc metric in contrast to rouge, and the expansion of resources available to the portuguese and multilingual nlp community. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others enhancing automatic text summarization: extractive, abstractive, and hybrid approaches chapter © 2025 an arabic multi-source news corpus: experimenting on single-document extractive summarization article 04 february 2021 an abstractive text summarization using recurrent neural network chapter © 2023 1 introduction automatic text summarization is a fundamental task in the field of natural language processing (nlp), essential for handling large amounts of information. this task aims to condense information from one or more documents, producing summaries that preserve the main points of the original content [13]. summarization can be classified into two main types: extractive and abstractive. extractive summarization selects sentences or phrases directly from the original text(s), while abstractive summarization generates new sentences that capture the original meaning of the text(s) [17]. additionally, summarization can be categorized as single-document, when summarizing an unique text, or multi-document, integrating information from multiple texts [7]. more recently, large language models (llms) have been employed for several nlp applications, including generation tasks as text summarization [1]. despite their popularity, llms present an explainability deficit and have some limitations, such as the expensive necessary computational infrastructure and the occurrence of hallucinations, which, in the case of summarization, may hinder the proposal of preserving the original meaning of the texts. on the other hand, extractive summarization methods generally do not suffer from these disadvantages. interestingly, there has been a resurgence of interest in these methods, with the proposal of novel methods that incorporate classical ideas with new perspectives and also bring new approaches to the task. various extractive methods have been proposed and tested mainly on english corpora like cnn/dm [9]. some notable methods are presumm [12], histruct+ [20], ranksum [10], memsum [8] and matchsum [22], which have demonstrated promising performances in generating summaries. however, there is a significant gap in the application of these methods to other languages and, therefore, limited evaluation on their multilingual capacity. with the web, multilingual summarization is a very relevant field, and there is a growing need for resources and techniques that support multiple languages, expanding the applicability of nlp advances to diverse linguistic communities. evaluation in summarization (whether it is multilingual or not) is crucial for advancing research in the field. traditionally, the rouge (recall-oriented understudy for gisting evaluation) metric [11] has been widely used to measure the overlap of n-grams between automatic summaries and human reference summaries. however, rouge is limited since it only computes superficial text similarity. to address these limitations, other metrics have arisen, as blanc [21]. blanc evaluates summaries by using them to help pre-trained language models (such as bert [5]) to perform the token unmasking task, with the supposition that a “more helpful” summary may improve the model success on the task. in this context, this work aims at evaluating the effectiveness of some more recent extractive methods for brazilian portuguese, trying to assess their multilingual performance. we run some selected state-of-the-art and more classical methods on news corpora for english and brazilian portuguese, using well-known reference summarization corpora. we compare the results using both rouge and blanc metrics, in order to provide a more comprehensive evaluation. next section briefly describes the main related work. the datasets that we use are detailed in sect. 3. section 4 brings our experiment setup, while discussion of results and final remarks are made in sects. 5 and 6, respectively. 2 related work several automatic extractive summarization methods have been developed in the area, especially using deep learning models and word embeddings. here, we briefly overview classical and more recent methods, including those that are considered state-of-the-art in the area. textrank [14] is a graph-based ranking algorithm used for both sentence and keyword extraction. inspired by google’s pagerank algorithm, textrank applies the idea of “voting" to determine the importance of textual units. it constructs a graph where nodes represent sentences or words, and edges indicate co-occurrence or similarity relations. the algorithm iterates over the graph until the importance scores of the nodes converge. textrank is an unsupervised method that has proven competitive in established benchmarks like duc-2002, demonstrating robustness and portability across different domains and languages. centroid-embedding [19] summarization leverages the compositional capabilities of word embeddings to capture semantic relationships between words and sentences. this method constructs a centroid vector, which represents the central theme of the document, by summing the embeddings of the most significant words, determined by their tf-idf scores. each sentence is then represented as a sum of its word embeddings, and sentences closest to the centroid vector are selected for the summary. this approach addresses the limitations of traditional bag-of-words models, which often fail to capture semantic similarities between sentences with different words but similar meanings. the centroid-based method has proven effective in both multi-document and multilingual summarization tasks, offering competitive performance compared to more complex deep learning models. its simplicity and robustness make it an attractive option for extractive summarization across various domains and languages. presumm [12], also known as bertsum, has extractive and abstractive summarization strategies. it uses the bert architecture to understand the context of sentences and select the most relevant ones for the summary. the technique includes pretraining on general language tasks and fine-tuning specific to the summarization task. presumm performs summarization in two phases: (1) an extraction phase where the most important sentences are selected, and (2) an abstractive generation phase where these sentences can be refined to improve the fluency and cohesion of the final summary. for this work, we used only the extractive strategy of presumm. histruct+ [20] incorporates hierarchical structure information into pre-trained language models like bert and roberta to improve extractive summarization. this model is designed to handle the intrinsic structure of documents, considering the hierarchy of sections and subtitles when selecting sentences for the summary. histruct+ is particularly effective for long and complex documents with clear hierarchical structures, such as scientific articles. the approach improves summarization accuracy by preserving the logical and semantic structure of the documents. other relevant methods are ranksum [10], memsum [8] and matchsum [22], but we do not address them in this paper because they have been outperformed by some of the above methods in the literature and also in some of our initial experiments (that we do not report in this paper). 3 datasets for our experiments for brazilian portuguese, we adopt the cstnews corpusfootnote 1 [2]. it is a reference and widely known corpus developed to support research in automatic summarization of journalistic texts in brazilian portuguese. the corpus comprises 50 clusters of news articles, totaling 150 texts and their respective summaries. each cluster contains 2 to 3 news articles about the same event, collected from various brazilian media sources such as folha de são paulo, estadão, o globo, jornal do brasil, and gazeta do povo, according to their impact at the time of publication. the corpus contains 2,088 sentences and 47,240 words, with an average of 2.8 texts, 41.76 sentences, and 944.8 words per cluster. in addition to the original texts, each cluster includes single-document manual summaries, multiple-document manual summaries, and automatic summaries. the corpus is also manually annotated in various ways for syntax, semantics and discourse information. cnn/daily mail [9] is a corpus widely used in automatic summarization research for english. it comprises approximately 286,817 article-summary pairs, each containing a news article and an associated summary. the articles are sourced from the cnn and daily mail news websites, and the summaries are human-generated bullet points summarizing the main points of the news articles. the corpus was created initially for the passage-based question-answering task but was adapted for summarization research by restoring the bullet points to form multi-sentence summaries. in the training set, the source documents have an average of 766 words distributed over 29.74 sentences, while the summaries consist of 53 words distributed over 3.72 sentences. the corpus is available in two versions: one with real entity names and another with anonymized entities that were replaced by document-specific ids, facilitating vocabulary reduction and experimentation with deep learning models. 4 experiment setup the corpora data was divided into 70%, 15%, and 15% for training, validation, and testing for the experiments (as some methods needed training). the original configurations of each summarizer were maintained, adapting them to the cstnews corpus. the experimental process is summarized in the following subsections. 4.1 pre-processing the nltk library was used for tokenization, stopword removal, and normalization of portuguese texts. this step is crucial to ensure that the models, originally trained in english, can operate effectively in the new language. tokenization segments the text into smaller units, such as words or subwords. stopword removal eliminates common words that do not significantly contribute to the task. normalization adjusts the text to a more consistent form, facilitating subsequent processing. for histruct+ and presumm, we performed similar pre-processing using the stanford corenlp tool. 4.2 model training and adjustment we applied the methods’ original codes with minimal modifications. we also fine-tuned the model parameters to optimize their performance on cstnews. for this, we adapted the tokenization processes and changed some hyperparameters specific to the language and style of the texts in the cstnews corpus. we aimed to replicate the original training conditions of the models as closely as possible despite our limitations. the model used in presumm was bert-base-uncased, while histruct+ employed roberta-base, as these were the models used in the original works. since we only had access to a cpu, the number of epochs and batch sizes had to be adjusted. presumm and histruct+ were trained for 7 epochs with a batch size of 14. the learning rates were the same as in the original works: 2e-3 for presumm and histruct+. 4.3 evaluation the generated summaries were evaluated using rouge and blanc. rouge measures the overlap of n-grams between the automatic and human reference summaries. the most commonly used variants are rouge-1, which measures unigram (individual words) overlap, rouge-2, which measures bigram (word pairs) overlap, and rouge-l, which measures the longest common subsequence overlap. each variant can be evaluated in terms of f1 score, precision, and recall. the f1-score is the harmonic mean of precision and recall, balancing the two. precision measures the proportion of n-grams in the generated summary that is present in the reference summary. in contrast, recall measures the proportion of n-grams in the reference summary that is present in the generated summary. although rouge is widely used for its simplicity and effectiveness, it is criticized for not adequately capturing summaries’ semantic quality and coherence, focusing mainly on superficial text similarity. conversely, blanc measures how much the generated summary aids a pre-trained language model (like bert) in understanding the document. it uses the masked token task (cloze task) to evaluate the functional utility of a summary. in blanc-help mode (which is the one we used), the summary is concatenated to each document sentence during inference, and the model’s ability to predict masked words with the summary is measured. the difference in prediction accuracy with and without the summary indicates the summary’s quality. a higher blanc score means the summary provides more contextual information that helps the model to understand the document better. unlike rouge, blanc does not require human-written reference summaries. 5 results and discussion tables 1 and 2 show the rouge results for the summarization methods on the cstnews and cnn/dm corpora. for cstnews, presumm demonstrates the highest performance in most rouge metrics, particularly in rouge-1 f1 (55.55), rouge-1 precision (67.91), and rouge-2 precision (49.64). this indicates its high effectiveness in selecting relevant sentences that closely match the reference summaries at the unigram and bigram levels. however, its lower recall scores suggest that, while maintaining high precision, it may miss some relevant content. for the cnn/dm dataset, presumm also shows the highest performance with rouge-1 score of 53.37. textrank and centroid-embedding have lower scores (30.14 and 18.12), consistent with their lower performance on the cstnews corpus too and demonstrating the need for advanced models to handle complex text summarization tasks. table 1. results on rouge metrics for cstnews datasetfull size table table 2. results on rouge metrics for cnn/dm datasetfull size table table 3. results on blanc metric for cstnews and cnn/dmfull size table table 3 presents the blanc results for the same models. centroid-embedding achieves the highest blanc score (57.74) for cstnews. despite its high rouge scores, presumm produces the lowest blanc score, which shows the importance of using complementary metrics for a more holistic evaluation. for the cnn/dm dataset, textrank achieves the highest blanc score (26.20). centroid-embedding also performs well, with score of 24.35. presumm again shows a relatively low blanc score. as illustration, table 4 shows a summary generated by presumm, allowing comparing it with the human-written summary. the summary generated by presumm accurately captured fabiana murer’s victory and the matching of the medal record, but omitted details about the competitors and the achieved marks. table 4. comparison between summary generated by presumm and the human-written summaryfull size table table 5 shows three cstnews summaries generated by presumm, the summarizer which achieved the best rouge results. these summaries were randomly chosen for a manual human evaluation. we used some of the known tac (text analysis conference) criteria [4], which include overall responsiveness and readability. the assessment of overall responsiveness examines how effectively a summary addresses the information need outlined in the topic statement, considering both the content and linguistic quality of the summary. the readability score evaluates the summary’s fluency and structure, independent of content, and is based on factors such as grammatical correctness, lack of redundancy, referential clarity, focus, structure, and coherence. these criteria are evaluated on a five-point scale: from very poor to very good. we evaluated these three random summaries as “very good" based on the above criteria once they show no redundancy, no referential problems, and very good structure and coherence. their high rouge scores can reflect the responsiveness of the summaries. once the summaries generated are close to the golden standard, that is, the summary generated by humans, they are expected to cover the document’s main content, thus, having a very good responsiveness. table 5. three cstnews summaries generated by presummfull size table table 6 shows three cstnews summaries generated by centroid-embedding, the summarizer which achieved the best blanc results. these summaries were also randomly chosen for a manual human evaluation and evaluated under the tac criteria. similarly to the previous summaries, we also evaluate these three as “very good" regarding the overall responsiveness and readability. table 6. three cstnews summaries generated by centroid-embeddingfull size table these results raise an important question: why did the centroid-embedding achieve such high blanc scores? since blanc measures how much the generated summary aids a pre-trained language model in understanding the original document, its high blanc score suggests a high amount of informational content in the summaries. however, based on these three randomly chosen summaries, we could not hypothesize why they had higher blanc scores than presumm, for example. thus, we encourage further research and new experiments with both rouge and blanc metrics. while providing valuable insights into the performance of various summarization models on the cstnews corpus, the present study is subject to several methodological limitations. one significant limitation is the size of the dataset. although a reference in the area, the cstnews corpus contains only 150 texts, and more than this limited number of texts is needed for training robust summarization models. so, we emphasize that we need more resources in languages other than english, to enhance the reliability and generalizability of summarization models. we are aware of a recent corpus, recognasumm [15], which contains over 130.000 journalistic texts in brazilian portuguese, and we plan to perform experiments with it in the future. another limitation of this study is the computational resources used. due to the unavailability of gpu resources, the experiments were conducted using cpus. the lack of gpu access limited our ability to fine-tune model parameters optimally, potentially impacting the performance of the summarizers. with the availability of gpus, future studies could perform more extensive parameter tuning, leading to better results. more efforts are also needed to refine and develop evaluation metrics beyond traditional measures like rouge. metrics such as blanc, which supposedly consider semantic coherence and fluency, should be further explored and validated. developing new metrics that can more accurately reflect the quality of summaries in different languages and contexts will be crucial for advancing the field. 6 final remarks we contributed to the investigation of multilingual summarization by evaluating state-of-the-art extractive summarization models on the brazilian portuguese cstnews corpus and the english cnn/dm corpus. our study utilized rouge and the more recently developed blanc metric, providing insights into the models’ performance. the findings revealed that presumm performed well according to rouge metrics, while centroid-embedding had good blanc scores for both languages. future work includes testing other methods for the previous languages, as well as other languages, including languages of different linguistic typologies. we also aim at using corpora of diverse genres and domains. such evaluation is necessary for determining the potential of current and future summarization methods for multilingual summarization. another relevant issue consists in exploring multilingual multi-document summarization, leveraging some previous researches and datasets that exist for brazilian portuguese, as the ones of [3] and [6]. exploring the multilingual performance of some classic single and multi-document methods that were created for portuguese may be other interesting endeavor, as it is the case of gistsumm [16] – the first summarization system for portuguese – and rsumm [18]. the interested reader may find the datasets, the source codes of the methods and other details about this work at the poetisa project website (at https://sites.google.com/icmc.usp.br/poetisa). notes 1.https://sites.icmc.usp.br/taspardo/sucinto/cstnews.html. references bommasani, r., et al.: on the opportunities and risks of foundation models (2022) google scholar  cardoso, p.c., et al.: cstnews-a discourse-annotated corpus for single and multi-document summarization of news texts in brazilian portuguese. in: proceedings of the 3rd rst brazilian meeting, pp. 88–105 (2011) google scholar  chaud, m., felippo, a.d.: exploring content selection strategies for multilingual multi-document summarization based on the universal network language (unl). revista de estudos da linguagem 26(1), 45–71 (2017) article  google scholar  dang, h.t., owczarzak, k., et al.: overview of the tac 2008 update summarization task. in: tac (2008) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2018). https://doi.org/10.48550/arxiv.1810.04805. http://arxiv.org/abs/1810.04805v2 di felippo, a., tosta, f.e.s., pardo, t.a.s.: applying lexical-conceptual knowledge for multilingual multi-document summarization. in: silva, j., ribeiro, r., quaresma, p., adami, a., branco, a. (eds.) propor 2016. lncs (lnai), vol. 9727, pp. 38–49. springer, cham (2016). https://doi.org/10.1007/978-3-319-41552-9_4 chapter  google scholar  el-kassas, w.s., salama, c.r., rafea, a.a., mohamed, h.k.: automatic text summarization: a comprehensive survey. expert syst. appl. 165, 113679 (2021). https://doi.org/10.1016/j.eswa.2020.113679. https://linkinghub.elsevier.com/retrieve/pii/s0957417420305030 gu, n., ash, e., hahnloser, r.: memsum: extractive summarization of long documents using multi-step episodic markov decision processes. in: muresan, s., nakov, p., villavicencio, a. (eds.) proceedings of the 60th annual meeting of the association for computational linguistics, vol. 1: long papers, pp. 6507–6522. association for computational linguistics, dublin (2022). https://doi.org/10.18653/v1/2022.acl-long.450. https://aclanthology.org/2022.acl-long.450 hermann, k.m., et al.: teaching machines to read and comprehend. adv. neural inf. process. syst. 28 (2015) google scholar  joshi, a., fidalgo, e., alegre, e., alaiz-rodriguez, r.: ranksum–an unsupervised extractive text summarization based on rank fusion. expert syst. appl. 200, 116846 (2022) article  google scholar  lin, c.y.: rouge: a package for automatic evaluation of summaries (2004) google scholar  liu, y., lapata, m.: text summarization with pretrained encoders. in: inui, k., jiang, j., ng, v., wan, x. (eds.) proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 3730–3740. association for computational linguistics, hong kong (2019). https://doi.org/10.18653/v1/d19-1387. https://aclanthology.org/d19-1387 mani, i.: automatic summarization. natural language processing (2001). https://doi.org/10.1075/nlp.3 mihalcea, r., tarau, p.: textrank: bringing order into text. in: proceedings of the 2004 conference on empirical methods in natural language processing, pp. 404–411 (2004) google scholar  paiola, p.h., garcia, g.l., jodas, d.s., correia, j.v.m., sugi, l.a., papa, j.p.: recognasumm: a novel brazilian summarization dataset. in: proceedings of the 16th international conference on computational processing of portuguese, pp. 575–579 (2024) google scholar  pardo, t.a.s., rino, l.h.m., nunes, m.g.v.: gistsumm: a summarization tool based on a new extractive method. in: mamede, n.j., trancoso, i., baptista, j., das graças volpe nunes, m. (eds.) propor 2003. lncs (lnai), vol. 2721, pp. 210–218. springer, heidelberg (2003). https://doi.org/10.1007/3-540-45011-4_34 chapter  math  google scholar  radev, d.r., mckeown, k., hovy, e.: introduction to the special issue on summarization (2002) google scholar  ribaldo, r., christina figueira cardoso, p., alexandre salgueiro pardo, t.: exploring the subtopic-based relationship map strategy for multi-document summarization. revista de informática teórica e aplicada 23(1), 183–211 (2016). https://doi.org/10.22456/2175-2745.59104. https://seer.ufrgs.br/index.php/rita/article/view/rita-vol23-nr1-183 rossiello, g., basile, p., semeraro, g.: centroid-based text summarization through compositionality of word embeddings. in: giannakopoulos, g., lloret, e., conroy, j.m., steinberger, j., litvak, m., rankel, p., favre, b. (eds.) proceedings of the multiling 2017 workshop on summarization and summary evaluation across source types and genres, pp. 12–21. association for computational linguistics, valencia (2017). https://doi.org/10.18653/v1/w17-1003. https://aclanthology.org/w17-1003 ruan, q., ostendorff, m., rehm, g.: histruct+: improving extractive text summarization with hierarchical structure information. in: muresan, s., nakov, p., villavicencio, a. (eds.) findings of the association for computational linguistics: acl 2022, pp. 1292–1308. association for computational linguistics, dublin (2022). https://doi.org/10.18653/v1/2022.findings-acl.102. https://aclanthology.org/2022.findings-acl.102 vasilyev, o., dharnidharka, v., bohannon, j.: fill in the blanc: human-free quality estimation of document summaries. arxiv preprint arxiv:2002.09836 (2020) zhong, m., liu, p., chen, y., wang, d., qiu, x., huang, x.: extractive summarization as text matching. in: jurafsky, d., chai, j., schluter, n., tetreault, j. (eds.) proceedings of the 58th annual meeting of the association for computational linguistics, pp. 6197–6208. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.acl-main.552. https://aclanthology.org/2020.acl-main.552 download references acknowledgments to sidi and capes (coordenação de aperfeiçoamento de pessoal de nível superior), for supporting this work. the author thiago alexandre salgueiro pardo also thanks the center for artificial intelligence of the university of são paulo (c4ai http://c4ai.inova.usp.br/), with support by the são paulo research foundation (fapesp grant #2019/07665-4) and by the ibm corporation, and the ministry of science, technology and innovation, with resources of law n. 8,248, of october 23, 1991, within the scope of ppi-softex, coordinated by softex and published as residence in tic 13, dou 01245.010222/2022-44. author information authors and affiliations núcleo interinstitucional de linguística computacional (nilc), instituto de ciências matemáticas e de computação, universidade de são paulo, são carlos, sp, brazil germano antonio zani jorge, davi alves bezerra & thiago alexandre salgueiro pardo sidi, campinas, sp, brazil clarissa castellã xavier authors germano antonio zani jorgeview author publications search author on:pubmed google scholar davi alves bezerraview author publications search author on:pubmed google scholar clarissa castellã xavierview author publications search author on:pubmed google scholar thiago alexandre salgueiro pardoview author publications search author on:pubmed google scholar corresponding author correspondence to germano antonio zani jorge . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper jorge, g.a.z., bezerra, d.a., xavier, c.c., pardo, t.a.s. (2025). multilingual extractive summarization: investigating state-of-the-art methods for english and brazilian portuguese. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_15 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords extractive summarization multilinguality natural language processing publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a multi-algorithm approach to the optimization of thermal power plants operation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a multi-algorithm approach to the optimization of thermal power plants operation conference paper first online: 12 october 2023 pp 209–223 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a multi-algorithm approach to the optimization of thermal power plants operation download book pdf download book epub gabriela t. justino  orcid: orcid.org/0000-0001-6154-35259, gabriela c. freitas  orcid: orcid.org/0000-0002-7365-01309, camilla b. batista  orcid: orcid.org/0009-0000-5363-41799, kleyton p. cotta  orcid: orcid.org/0000-0003-4332-56419, bruno deon  orcid: orcid.org/0009-0009-7181-64309, flávio l. loução jr.  orcid: orcid.org/0009-0009-1615-32839, rodrigo j. s. de almeida  orcid: orcid.org/0000-0003-4910-659710 & … carlos a. a. de araújo jr.  orcid: orcid.org/0000-0002-7211-472310  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 510 accesses abstract a new multi-algorithm approach for the daily optimization of thermal power plants and the economic dispatch was tested. for this, the dispatches were clustered in different groups based on their duration and the total power output requested. genetic algorithm, differential evolution and simulated annealing were selected for implementation and were employed according to the distinct characteristics of each dispatch. a monthly improvement of up to 2,45\(\,\times \,\)105 r$ in the gross profit of the thermal power plant with the use of the optimization tool was estimated. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an optimization approach for potential power generator outputs based on parallelized firefly algorithm chapter © 2017 exergy analysis and optimization of multi-effect distillation with thermal vapor compression system of bandar abbas thermal power plant using genetic algorithm article 25 january 2018 performance optimization of thermal power plants using metaheuristic algorithms article 09 july 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. continuous optimization discrete optimization energy management energy informatics optimization thermal process engineering 1 introduction the real-time operation of systems, especially in the generation of electricity, requires agility and assertiveness in decision-making. the availability, reliability and performance of the generating machines of a thermoelectric power plant (tpp) are critical issues to maximize the economic results of the business and to guarantee the fulfillment of the demand of the electric sector. a part of the brazilian thermoelectric park may remain idle if hydrological conditions are favorable. in most countries, combined cycle coal or gas power plants generally do not experience long-term idleness, due to operation as the backbone of the system, being dispatched almost continuously. in brazil, thermal electricity is normally used to support base generation in the electrical system: for peak generation, with daily activation or for at least a large fraction of working days [5]. system planning is centralized at the ons (national system operator) and the daily energy operation, in general, is a problem for the plants, which have availability contracts. this type of plant has high pollutant emission rates and, considering the current scenario of search for solutions to reduce greenhouse gas emissions, it is important to guarantee the maximum energy performance of complementary plants, necessary for the balance of the electrical system. in addition, given the current scenario of the electricity sector and the way in which generating agents are remunerated, the reduction of operating costs brings numerous benefits. in the context of the plant, the large number of generating units and auxiliary equipment makes its operation arduous and the maximization of performance is complex. this makes the actions taken by operators not completely efficient from an energy and economic point of view. furthermore, the simultaneous start-up of many generating units creates scenarios of operational complexity that can lead to low-performance operational conditions. currently, the plants do not have a computational tool to predict trends, identify failures and operational deviations in an automated and reliable way for engines and other large generating machines. 1.1 motivation the present work belongs to an aneel (national electric energy agency) research and development (r &d) project, which consists of developing a tool to assist the operation of a thermoelectric plant, aiming to improve its energy performance, reduce costs and mitigate process uncertainties, through the optimization of the relationship between fuel used versus energy generated. in general, the objective is to enhance the energy produced for a lower fuel consumption through the health index of the generating units. the expected result obtained by this tool comes in the format of an optimal operational configuration of the thermal power plant: which engines should be used and how much of the generating capacity of each of them should be employed. 2 method review and selection heuristic methods are groups of practical guidelines coming from experience. through trial and error, they help in finding good, but not always optimal, solutions. their use is of particular interest in the resolution of complex problems, without a clear analytical solution. from the application of heuristic rules, many meta-heuristic techniques were developed. they can be seen as a generic algorithmic framework which can be applied to different optimization problems with only a few specific modifications [13]. although they do not guarantee the optimal solution, these methods are widely used, because they provide good solutions (close to the optimal) in a reasonable computation time, enabling their implementation in real time. these methods have also been used successfully to solve high-dimensional problems in many fields of study, including the electric sector [17, 18]. the economic dispatch (ed) optimization problem is non-convex, therefore finding the optimal solution becomes difficult [15]. nevertheless, modern meta-heuristic algorithms are promising alternatives for the resolution of such complex problems [7]. genetic algorithm was used to solve the ed problem with convex and non-convex cost functions and reached satisfactory results when applied on the electrical grid of crete island [4]. the ant colony technique is also a viable option for solving ed problems. silva et al. [13] used this algorithm in conjunction with a sensitivity matrix based on the information provided by the lagrange multipliers to improve the biologically inspired search process. al-amyal et al. [1] proposed a multistage ant colony algorithm to solve the ed problem for six (6) thermal generation plants with good results. the grey wolf optimizer is another option for solving ed problems that are nonlinear, non-convex and discontinuous in nature, with numerous equality and inequality constraints. tests with model systems with up to 80 generators showed that this method is effective in solving ed problems, when compared to other similar algorithms [7]. nascimento et al. [11] used the differential evolution algorithm to solve the ed problem. their proposed methodology reduced the total fuel cost of the plants significantly. an algorithm based on bat behavior and echolocation was also successfully employed by hanafi and dalimi [6] to solve the ed problem in six (6) tpps. wang et al. [16] compared traditional and adaptative versions of the particle swarm optimization (pso) algorithm in the resolution of the ed problem. the modified version of the algorithm used an immune mechanism and adaptative weights controlled by the fitness of the particle. when applied in a system with 10 generating units, the adaptative pso performed better in terms of solution quality and stability, but the increased complexity of the algorithm also increased its execution time. the environmental and economic dispatch (eed) is also an important daily optimization task in the operation of many energy plants. it presents two conflicting objectives: fuel cost and gas emission. this problem can be formulated as a highly restricted non-linear multiobjective optimization. thus, algorithms based on concepts such as pareto dominance and crowding distance are viable tools for solving the eed problem. one such algorithm is the elitist non-dominated sorting genetic algorithm [3, 12]. the goal-attainment method is also an option to solve eed problems. basu [2] used this method in conjunction with the simulated annealing technique to solve an eed problem. also in the electric sector, the use of neural networks to solve optimization problems has been increasing. this methodology is usually employed for online optimization of the system power flow [9, 14]. all these works focused only on the use of one optimization algorithm to solve the economic dispatch problem, without considering the particularities of each dispatch regarding the power requested of the tpp in relation to its overall capacity. however, recent works in other fields started to use a combination of algorithms in order to find better solutions for complex problems. in some cases, an association of meta-heuristic algorithms was employed; e.g., the combined use of genetic algorithm and simulated annealing to solve an emergency location routing problem [10]. other works tested unions of algorithms with distinct fundamentals, such as the combination of recurrent neural network and particle swarm optimization [8]. with the objective of implementing in plant operation an optimization tool capable of finding good solutions without a high computational cost, five different meta-heuristic algorithms were tested. four of them had a mono-objective approach: adaptative particle swarm optimization (pso), genetic algorithm (ga), simulated annealing (sa) and differential evolution (ed). with a multi-objective approach, the elitist non-dominated sorting genetic algorithm (nsga-ii) was tested. 3 problem formulation 3.1 system description in this article we study the power generation process of a thermoelectric complex, which involves the operation of two identical tpps. together they have an installed capacity of 342 mw, enough energy to serve a population of about 980 thousand inhabitants. the structure of the tpp is composed of diesel engines as main equipment, generators and subsystems linked to them, guaranteeing their maintenance and operation. each tpp has 19 man/stx engines of type 18v32/40 with capacity to generate 8.76 mw and 1 man/stx engine of type 9l32/40 with capacity to generate 4.38 mw. the engines are supercharged, nonreversible, 4-stroke, with 320 mm cylinder diameter, 400 mm piston stroke and 720 rpm rotation. the 18v32/40 engines contains 18 cylinders v-shaped and the 9l32/40 engines are l-shaped with 9 cylinders. the auxiliary systems of the tpp are: intake air system: the air is filtered, compressed, cooled and finally admitted to the engine; fuel injection system: the fuel (usually hfo oil) is heated and filtered to be fed into the engine. if diesel-lfo oil is used, no heating is necessary, as its viscosity is low enough; exhaust system: the exhaust gases pass through the turbocharger turbines and exchange heat with the water in the recovery boilers to generate steam. then, they are sent to exhaustion in the chimneys; boiler system: steam from the boilers is used to heat the fuel. the recovery boilers can only be activated when there is generation. otherwise, auxiliary boilers (diesel oil and natural gas burner) are activated; lubrication system: the lubrication oil stored in the crankcase of the engines will promote the lubrication of all moving parts of the engine that come into contact with each other; cooling system: water is used for the cooling of the engine and its parts (nozzles and intake air system). this water comes from radiators and also cools heat exchangers from other systems. 3.2 economic dispatch problem the aim of this work is to solve the ed problem for tpps. the ed consists in minimizing the operational costs while fulfilling the energy demand requested from the plant. simultaneously, the plant must also obey physical and operational restrictions. the fuel consumption of a thermal power plant is responsible for approximately 60% of its operational costs [6]. the performance of the generating units regarding other factors necessary for efficient operation was also considered. using data from the air intake/exhaust, lubrication and cooling systems of each engine, a health index (hi) is calculated in real time. this parameter refers to the operational condition of each generating unit regarding need for maintenance and probability of failures and can be used to allocate the engines during operation. therefore, the objective function of the ed problem considered the fuel consumption and the health index. the specification of the dispatch requested by the ons is the main input of the algorithm. the daily schedule specifies the power demand for 48 intervals of 30 min each. the schedule is always received a day before operation starts, allowing for time to plan and optimize the operation during the following day. the initial condition of the engines is obtained from sensor data regarding active and reactive power and the hourmeter of each generating unit, as well as machine learning models regarding fuel consumption. the optimization tool outputs an optimized configuration of the tpp for the execution of the requested dispatch, in the form of the values for active and reactive power for each engine in each time interval. these are the decision variables for the model and are hereafter represented by \(p_{j,t}\) and \(p_{reactive,j,t}\) respectively. figure 1 shows the inputs for the optimization algorithm. fig. 1. structure of the optimization tool. full size image objective function: equation 1 shows the part of the objective function which represents the fuel consumption (kg) of the tpp for the whole dispatch (\(f_{fuel}\)). $$\begin{aligned} f_{fuel}=\sum _{j=1}^{n_g} \sum _{t=0}^{t} f(p_{j,t}) \end{aligned}$$ (1) where \(p_{j,t}\) is the vector corresponding to the generated active power (mw) by the j-th generating unit (among the \(n_g=40\) available ones) for each time interval t of 30 min in a dispatch of \(t=48\) intervals. the function \(f(p_{j,t})\) represents the relationship between fuel consumption and generated power. the health index (hi) of each engine was considered as a penalty in the objective function, to apply greater penalties for turning off generating units with the best hi and turning on the ones with the worst hi. considering hi is set between zero (0) and one (1) and these values respectively represent the worst and the best possible operational status for the engine, the penalty \(f_{hi}\) was elaborated as described in eq. 2: $$\begin{aligned} f_{hi}=\sum _{j=1}^{n_g} \sum _{t=0}^{t} cos^{-1}(hi_{j,t}m_{j,t})|m_{j,t}| \end{aligned}$$ (2) where \(hi_{j,t}\) is the health index of the j-th generating unit in time interval t and \(m_{j,t}\) represents the variation of status of the same engine for the same time interval according to table 1. table 1. engine status alteration indexfull size table thus, the sum of these two parts constructs the objective function \(f_{obj}\) for the mono-objective algorithms (pso, ga, sa and de). the factor \(\lambda \) is used to tune their different orders of magnitude, as described in eq. 3. for multi-objective nsga-ii, both \(f_{fuel}\) and \(f_{hi}\) were considered as individual objectives to be minimized. $$\begin{aligned} f_{obj} = \;\sum _{j=1}^{n_g} \sum _{t=0}^{t} f(p_{j,t}) + \lambda \sum _{j=1}^{n_g} \sum _{t=0}^{t} cos^{-1}(hi_{j,t}m_{j,t})|m_{j,t}| \end{aligned}$$ (3) practical operating constraints: physical constraints of the generating units, such as limits for active and reactive power, are some of the main restrictions for the ed problem. aspects relating to the internal operational rules for the power plant and the energy demand for the requested dispatch are also input for the problem restrictions. thus, the objective function is subject to: $$\begin{aligned} p_{min} \le p_{j,t} \le p_{max} \end{aligned}$$ (4) $$\begin{aligned} p_{reactive,min} \le p_{reactive,j,t} \le p_{reactive,max} \end{aligned}$$ (5) $$\begin{aligned} pf_{min} \le pf_{j,t} = \frac{p_{j,t}}{\sqrt{p_{j,t}^2 + p_{reactive,j,t}^2}} \le pf_{max} \end{aligned}$$ (6) $$\begin{aligned} p_{d,t} 5 mw \le \sum _{j=1}^{n_g} p_{j,t} \le p_{d,t} + 5 mw \end{aligned}$$ (7) $$\begin{aligned} t_{previous,j,t} = \{\forall t_{previous} \, | \, m_{j,t_{previous}} \ne 0 \, and \, t_{previous} \le t\} \end{aligned}$$ (8) $$\begin{aligned} t_{status,j,t} = t max(t_{previous}) \end{aligned}$$ (9) $$\begin{aligned} \bigg \{\begin{array}{ll} t_{status,j,t} \ge 24,\ \ if\ \ {\text {p}}_{j,t-1} > 0\,\text {mw} \\ t_{status,j,t} \ge 6,\ \ {if}\ \ {\text {p}}_{j,t-1} = 0\,\text {mw} \end{array} \end{aligned}$$ (10) $$\begin{aligned} n{_{boilers,t}} = card(\{\forall j = 2, 7, 12, 14, 22, 27, 32, 37, 39 \, | \, p_{j,t} > 0 \,\text {mw}\} ) \end{aligned}$$ (11) $$\begin{aligned} \bigg \{\begin{array}{ll} n_{boilers,t} \ge 4,\ \ if\ \ {\text {p}}_{d,t} < 200\,\text {mw} \\ n_{boilers,t} = 9,\ \ if\ \ {\text {p}}_{d,t} \ge 200\,\text {mw} \end{array} \end{aligned}$$ (12) equations 4 and 5 represent the physical limits for, respectively, active and reactive power in the j-th generating unit in the time interval t. equation 6 represents the limits for the power factor (\(pf_{j,t}\)) of each engine. equation 7 elucidates that the sum of the power generated in all engines must stay within a 5mw tolerance of the total requested power (\(p_{d,t}\)) for each time interval t, according to ons rules. equation 8 defines the set of time intervals (\(t_{previous,j,t}\)) before a specific time interval t in which an engine j changed status (turning on or off). equation 9 defines the difference (\(t_{status,j,t}\)) between a time interval t and the last change in status for an engine j. equation 10 shows the minimum operation time of a generating unit and its minimum rest time, after which it can be turned on again. both variables are represented by \(t_{status,j,t}\), depending only on whether the engine is on or off. equation 11 defines the number of heat recovery boilers active (\(n_{boilers,t}\)) at a certain time interval t, considering the boilers are associated to engines 2, 7, 12, 14, 22, 27, 32, 37 and 39. equation 12 elucidates the number of heat recovery boilers that must be operational during a certain time interval t as a function of the scheduled energy demand. 4 experimental procedure 4.1 dispatch clusterization the total power requested by ons is the main input of the optimization algorithm. evaluating data from january 1st 2018 to february 21st 2021, it was noted that the daily dispatches were diverse in their characteristics. thus, the sum of all generated power in a daily dispatch and the fraction of hours of scheduled operation in a day were used to cluster the dispatches in five (5) distinct categories employing the spectral clustering method. this division allowed the separate evaluation of the optimization algorithms for distinct dispatches. 4.2 hyperparameter tuning and algorithm selection a group of dates was randomly selected from each dispatch category to be used as model-dispatches in the estimation of the hyperparameters of the optimization algorithms and the \(\lambda \) factor of the objective function. due to the great number of possible combinations, the random search technique was applied. the following parameters were estimated for each algorithm: genetic algorithm: population size, maximum number of generations, mutation probability and elitism. elitist non-dominated sorting genetic algorithm: population size, maximum number of generations, mutation probability. differential evolution: population size, maximum number of generations, crossover probability and mutation factor. simulated annealing: initial and minimal temperatures, maximum number of iterations per temperature, cooling factor. during the evolution of the algorithm, the temperature \(t_i\) is updated according to the cooling factor \(\kappa \) and the previous temperature \(t_{i-1}\), as described in eq. 13. $$\begin{aligned} t_i = \frac{t_{i-1}}{1+\kappa t_{i-1}} \end{aligned}$$ (13) particle swarm optimization: population size, maximum number of iterations and the inferior and superior limits of the social, cognitive and inertia coefficients. these coefficients are updated according to eq. 14 for each iteration. $$\begin{aligned} c_{i,j} = c_{i,max} (c_{i,max}-c_{i,min}) \frac{j}{j_{max}} \end{aligned}$$ (14) where \(c_{i,j}\) is the value of coefficient i for iteration j, \(c_{i,max}\) and \(c_{i,min}\) are respectively the maximum and minimum values possible for coefficient i and \(j_{max}\) is the maximum number of iterations. afterwards, to compare the algorithms, one model-dispatch was randomly selected from each category. each algorithm was tested with ten (10) experiments with each model-dispatch, reaching a total of 50 experiments. the feasibility of the algorithms was tested by evaluating their execution time on a computer with intel core i7 1.80 ghz processor and 16.0 gb ram. the effectiveness of each algorithm in solving the ed problem was evaluated through the value of the objective function of the output solution. for multi-objective algorithm nsga-ii, an equivalent objective function was calculated using the final value of both optimized objectives, which correspond to the parts of the objective function described in eq. 3 and used for the mono-objective algorithms. these metrics were calculated with a 95% confidence interval. this procedure was executed independently for each distinct category of dispatch. 4.3 testing after the selection of the algorithms for implementation, the optimization tool was tested with the dispatches scheduled for the month of january 2021. the dispatches were divided into clusters as described in sect. 4.1 and optimized with the algorithms selected. the results obtained were compared to the performance of the tpp for the same period before the implementation of the optimization tool. the metric used for this comparison was the tpp’s estimated gross profit (egp), calculated according to eq. 15. during the evaluated period, the price of fuel was r$ 2,78/kg of fuel and the value received for the generated energy (cvu) was 912,28 r$/mwh. $$\begin{aligned} \begin{aligned} egp = & \;cvu\,*(generated\; energy) \\ & (kg\; of\; fuel)*(price\; of\; fuel) \end{aligned} \end{aligned}$$ (15) 5 results and discussion 5.1 dispatch clustering a total of 390 scheduled daily dispatches were analyzed. the sum of the power to be generated throughout the day and the fraction of hours per day in operation were used to clusterize the dispatches. they were separated in five (5) distinct categories, whose characteristics are displayed in fig. 2. it can be noted that the use of only the generated power is enough to classify the dispatches. table 2 shows the number of dispatches per category. table 2. distribution of dispatches per categoryfull size table fig. 2. dispatch categories: (a) kernel density estimate of dispatches per generated power; (b) cumulative distribution of dispatches per fraction of the day in operation. full size image 5.2 hyperparameter tuning and algorithm selection using model dispatches selected from each category, the best values were estimated for the \(\lambda \) factor of the objective function and the hyperparameters of each algorithm. tables 3 and 4, respectively, show these results. it can be noted that the change in the scheduled power demand significantly affects the relationship between the two parts of the objective function and thus distinct values for \(\lambda \) are needed. table 3. factor \(\lambda \) of the objective functionfull size table table 4. algorithm hyperparametersfull size table after estimating the hyperparameters, the execution time and the value of the objective function for the optimized solution were used to select the algorithms to be implemented for operation at the tpp. these results are described in tables 5a to 5e and summarized in fig. 3. to provide better comparison between the algorithms, for nsga-ii an equivalent objective function was calculated from the two objectives minimized, using the \(\lambda \) factors shown in table 3 and implemented in the other algorithms. table 5. execution time and objective function for dispatches.full size table table 6. algorithms selected for implementationfull size table fig. 3. objective function (a) and execution time (b) of the algorithms per dispatch cluster. full size image nsga-ii did not perform well regarding computational cost, with execution times significantly superior to those of the other algorithms, making unfeasible its implementation in production. for dispatches from category a, the best results for objective function were provided by sa, pso and de, with this last one also standing out for the low execution time. for category b, ga had the best results for objective function and a reasonable execution time (257 s). although sa was the fastest algorithm, its performance was much inferior regarding the objective function. for dispatches c, sa and pso achieved good results for the objective function, with sa performing better in relation to computational cost. for category d, sa achieved the best value for the objective function and the second best for execution time. the fastest algorithm (pso) was not as efficient in minimizing the objective function. for dispatches e, sa was the best algorithm for both evaluated metrics. table 6 shows the algorithms selected for implementation in operation according to the category of the daily dispatch to be optimized. these results show that the commonly used approach to solve this problem (using only one algorithm regardless of differences in the economic dispatch) may not yield good results when the power requested during dispatch changes considerably. the use of different algorithms allows for optimization methods which best fit the particularities of each problem. 5.3 testing from the results obtained by the optimization model, it was possible to estimate the gross profit of the tpp for the optimized configuration suggested by the algorithm. figure 4a and table 7 show these estimates, calculated using the methodology described in sect. 4.3, for the month of january/2021. fig. 4. tests for january/2021. full size image table 7. estimated gross profitfull size table considering an average error of 0,344 kg/mwh for the fuel consumption prediction model in the evaluated period, the use of the operational configuration suggested by the optimization tool should lead to an increase in gross profit between 1,44\(\,\times \,\)104 r$ and 2,45\(\,\times \,\)105 r$. despite the increased fuel consumption, the increment in the estimated gross profit corroborates the operational and financial benefits that using such an optimization tool can bring to thermoelectric generation. figure 4b shows a comparison between the ons requested power demand, the optimized configuration suggested and the realized operation before the implementation of the algorithms, for the scheduled dispatch on january 21st, 2021. it can be noted that the suggested optimized configuration follows the tendencies of the requested schedule and respects the ons tolerance limits of 5 mw (shown in the colored area of the graph). however, the optimized configuration is usually above the realized without use of the tool, ratifying the effectiveness of the optimization tool and explaining its superior estimated gross profit. 6 conclusion through the techniques and methodology used and the obtained results, this work provides, during thermoelectric generation, more standardized operational conditions, favoring a more efficient use of the engines. the methodology employing different algorithms for distinct categories of scheduled dispatches helps in adapting the optimization tool to the specificities of these dispatch categories. moreover, there was a significant financial gain, even if only estimated. the optimized configuration suggested by the algorithms achieved the objective of the ed problem, minimizing the fuel cost for the requested power demand and is ready for implementation in the supervisory system of the thermal power plant. this optimization tool can also be generalized to fit other thermal power plants, regardless of fuel. references al-amyal, f., al-attabi, k.j., al-khayyat, a.: multistage ant colony algorithm for economic emission dispatch problem. in: 2019 international ieee conference and workshop in óbuda on electrical and power engineering (cando-epe), pp. 161–166 (2019). https://doi.org/10.1109/cando-epe47959.2019.9111048 basu, m.: a simulated annealing-based goal-attainment method for economic emission load dispatch of fixed head hydrothermal power systems. int. j. electr. power energy syst. 27, 147–153 (2005). https://doi.org/10.1016/j.ijepes.2004.09.004 article  google scholar  basu, m.: dynamic economic emission dispatch using nondominated sorting genetic algorithm-ii. int. j. electric. power energy syst. 30, 140–149 (2008). https://doi.org/10.1016/j.ijepes.2007.06.009 damousis, i., bakirtzis, a., dokopoulos, p.: network-constrained economic dispatch using real-coded genetic algorithm. ieee trans. power syst. 18(1), 198–205 (2003). https://doi.org/10.1109/tpwrs.2002.807115 article  google scholar  fonseca, m., bezerra, u.h., brito, j.d.a., leite, j.c., nascimento, m.h.r.: pre-dispatch of load in thermoelectric power plants considering maintenance management using fuzzy logic. ieee access 6, 41379–41390 (2018). https://doi.org/10.1109/access.2018.2854612 hanafi, i.f., dalimi, i.r.: economic load dispatch optimation of thermal power plant based on merit order and bat algorithm. in: 2019 ieee international conference on innovative research and development (icird), pp. 1–5 (2019). https://doi.org/10.1109/icird47319.2019.9074734 jayabarathi, t., raghunathan, t., adarsh, b.r., suganthan, p.n.: economic dispatch using hybrid grey wolf optimizer. energy 111, 630–641 (2016). https://doi.org/10.1016/j.energy.2016.05.105 kumar, r.d., chakrapani, a., kannan, s.: design and analysis on molecular level biomedical event trigger extraction using recurrent neural network-based particle swarm optimisation for covid-19 research. int. j. comput. appl. technol. 66(3–4), 334–339 (2021). https://doi.org/10.1504/ijcat.2021.120459 article  google scholar  liu, h., shen, x., guo, q., sun, h.: a data-driven approach towards fast economic dispatch in electricity-gas coupled systems based on artificial neural network. appl. energy 286, 116480 (2021). https://doi.org/10.1016/j.apenergy.2021.116480 article  google scholar  nahavandi, b., homayounfar, m., daneshvar, a., shokouhifar, m.: hierarchical structure modelling in uncertain emergency location-routing problem using combined genetic algorithm and simulated annealing. int. j. comput. appl. technol. 68(2), 150–163 (2022). https://doi.org/10.1504/ijcat.2022.123466 article  google scholar  nascimento, m.h.r., nunes, m.v.a., rodríguez, j.l.m., leite, j.c.: a new solution to the economical load dispatch of power plants and optimization using differential evolution. electr. eng. 99, 561–571 (2017). https://doi.org/10.1007/s00202-016-0385-2 article  google scholar  e silva, m.d.a.c., klein, c.e., mariani, v.c., coelho, l.d.s.: multiobjective scatter search approach with new combination scheme applied to solve environmental/economic dispatch problem. energy 53(c), 14–21 (2013). https://doi.org/10.1016/j.energy.2013.02.045 silva, i.c. jr., do nascimento, f.r., de oliveira, e.j., marcato, a.l., de oliveira, l.w., passos filho, j.a.: programming of thermoelectric generation systems based on a heuristic composition of ant colonies. int. j. electric. power energy syst. 44, 134–145 (2013). https://doi.org/10.1016/j.ijepes.2012.07.036 sundaram, a.: multiobjective multi verse optimization algorithm to solve dynamic economic emission dispatch problem with transmission loss prediction by an artificial neural network. appl. soft comput. 124, 109021 (2022). https://doi.org/10.1016/j.asoc.2022.109021 article  google scholar  tian, j., wei, h., tan, j.: global optimization for power dispatch problems based on theory of moments. int. j. electric. power energy syst. 71, 184–194 (2015). https://doi.org/10.1016/j.ijepes.2015.02.018 wang, k., zhou, c., jia, r., he, w.: adaptive variable weights immune particle swarm optimization for economic dispatch of power system. in: 2020 asia energy and electrical engineering symposium (aeees), pp. 886–891 (2020). https://doi.org/10.1109/aeees48850.2020.9121455 wei, h., chen, s., pan, t., tao, j., zhu, m.: capacity configuration optimisation of hybrid renewable energy system using improved grey wolf optimiser. int. j. comput. appl. technol. 68(1), 1–11 (2022). https://doi.org/10.1504/ijcat.2022.123234 article  google scholar  zhang, m.j., long, d.y., li, d.d., wang, x., qin, t., yang, j.: a novel chaotic grey wolf optimisation for high-dimensional and numerical optimisation. int. j. comput. appl. technol. 67(2–3), 194–203 (2021). https://doi.org/10.1504/ijcat.2021.121524 article  google scholar  download references acknowledgments the authors would like to thank centrais elétricas da paraíba (epasa) for the financial support to pd-07236-0011-2020 optimization of energy performance of combustion engine thermal power plants with a digital twin approach, developed under the research and development program of the national electric energy agency (aneel r &d), which the engineering company carried out radix engenharia e software s/a, rio de janeiro, brazil. author information authors and affiliations research, development and innovation department (rd &i), radix engineering and software development, rio de janeiro, rj, brazil gabriela t. justino, gabriela c. freitas, camilla b. batista, kleyton p. cotta, bruno deon & flávio l. loução jr. paraíba power plants s.a. epasa, joão pessoa, pb, brazil rodrigo j. s. de almeida & carlos a. a. de araújo jr. authors gabriela t. justinoview author publications search author on:pubmed google scholar gabriela c. freitasview author publications search author on:pubmed google scholar camilla b. batistaview author publications search author on:pubmed google scholar kleyton p. cottaview author publications search author on:pubmed google scholar bruno deonview author publications search author on:pubmed google scholar flávio l. loução jr.view author publications search author on:pubmed google scholar rodrigo j. s. de almeidaview author publications search author on:pubmed google scholar carlos a. a. de araújo jr.view author publications search author on:pubmed google scholar corresponding author correspondence to gabriela t. justino . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper justino, g.t. et al. (2023). a multi-algorithm approach to the optimization of thermal power plants operation. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_14 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords optimization machine learning economic dispatch thermoelectric generation fuel consumption publish with us policies and ethics profiles gabriela c. freitas view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature hybrid multilevel explanation: a new approach for explaining regression models | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper hybrid multilevel explanation: a new approach for explaining regression models conference paper first online: 12 october 2023 pp 399–414 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) hybrid multilevel explanation: a new approach for explaining regression models download book pdf download book epub renato miranda filho  orcid: orcid.org/0000-0003-3455-58919,10 & gisele l. pappa  orcid: orcid.org/0000-0002-0349-44949  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 536 accesses abstract regression models are commonly used to model the associations between a set of features and an observed outcome, for purposes such as prediction, finding associations, and determining causal relationships. however, interpreting the outputs of these models can be challenging, especially in complex models with many features and nonlinear interactions. current methods for explaining regression models include simplification, visual, counterfactual, example-based, and attribute-based approaches. furthermore, these methods often provide only a global or local explanation. in this paper, we propose a hybrid multilevel explanation (hybrid multilevel explanation humie) method that enhances example-based explanations for regression models. in addition to a set of instances capable of representing the learned model, the humie method provides a complete understanding of why an output is obtained by explaining the reasons in terms of attribute importance and expected values in similar instances. this approach also provides intermediate explanations between global and local explanations by grouping semantically similar instances during the explanation process. the proposed method offers a new possibility of understanding complex models and proved to be able to find examples statistically equal to or better than the main competing methods and to provide a coherent explanation with the context of the explained model. this work was supported by fapemig (through the grant no. cex-ppm-00098-17), mpmg (through the project analytical capabilities), cnpq (through the grant no. 310833/2019-1), capes, mctic/rnp (through the grant no. 51119) and ifmg campus sabará. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a hybrid regression model for mixed numerical and categorical data chapter © 2020 a relational vulnerability analytic: exploring hybrid methodologies for human dimensions of climate change research in the himalayas chapter © 2020 a multivariate generalized linear model approach to mediation analysis and application of confidence ellipses article 10 may 2017 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. heterodox economics information model linear models and regression mixed methods model theory multivariate analysis 1 introduction regression models are widely used in various fields, including medicine, engineering, finance, marketing, among others. they are used to generate associations between a set of features and an observed outcome, and can be used for: (i) prediction—when, given a prediction value, we want to interpolate or extrapolate from observations to estimate the outcome, or (ii) to find associations between the independent features and the outcome and how strong these associations are. when the regression models can account for non-linear data relationships, the produced models cannot be directly interpreted by analyzing their coefficients, as it is usual in linear regression. as these models can be complex, have many features and nonlinear interactions, having methods able to understand the reasons that lead to a regression model’s output is important, especially in critical areas that can affect human life. current methods for explaining regression models provide different approaches, including simplification, visual, counterfactual, example-based, and feature-based explanations [3, 9, 16]. each of these approaches provides complementary explanations for the model’s output, but often they do not provide a complete understanding of why an output occurs. additionally, these approaches usually provide a global explanation of the overall model behavior or a local explanation about the output of a specific instance. in this paper we propose hybrid multilevel explanation (humie), a method for explaining regression models that build up on example-based explanations. example-based explanations have the advantage of being similar to the human way of explaining things by similarity to past events, but they often do not provide a complete understanding of why an output occurs. our proposed method, on the other hand, provides a complete understanding of the reasons why an output is obtained, in terms of feature importance and expected values in similar instances. for each selected example, humie explain the reasons why the output is obtained in terms of feature importance and expected values in similar instances. furthermore, the method provides intermediate explanations between the possibilities of global and local explanations, grouping semantically similar instances during the explanation process and bringing new possibilities of output understanding to the user. the remainder of this paper is organized as follows. in sect. 2, we review the existing literature on explanation methods for regression models. in sect. 3, we introduce humie. in sect. 4, we report the experimental results. finally, in sect. 5, we report our conclusions and possibilities of future work. 2 related work as the use of models created by machine learning algorithm is increasingly used to make critical decisions in our daily lives, so does the concern to understand the internal mechanisms that lead such models to obtain their outputs. machine learning models can be divided into two categories: those that create intrinsically interpretable models and those that are considered black-boxes. intrinsically interpretable models are easily understandable by humans, and include linear models, and decision trees. in black box models, in turn, a human cannot easily understand the internal mechanisms that led to the output [13]. it is fair to assume that currently the most accurate models are precisely of this type [9]. examples include support vector regressors (svr) and neural network regression (nnr). the explainable artificial intelligence (xai) literature encompasses methods generated for explaining black-box models, with an emphasis on classification tasks [12]. more recently we have seen various initiatives aimed to explain regression models, where the output is represented by a continuous value instead of a categorical one. among the main explanation approaches described in the literature, we have: i) simplification, where mathematical substitutions are made, generating more trivial algebraic expressions; ii) visual, where the model is explained through graphical elements [1, 23]; iii) counterfactual, methods demonstrate that minimal modifications made to the input data can significantly modify the output [8, 17]; iv) feature-based, analyze the characteristics of the data-including, for example, statistical summaries and feature importancethat cause a given black-box model to perform a prediction [20, 22]; and v) example-based, where representative instances of the model are created or selected from the training set (and are also called prototypes) to generate a global explanation [2, 10, 11, 15], and local explanations are based on the distance of these prototypes to the instance to be explained. in this context, we observe that example-based explanation approaches, although working in a similar to humans in the sense of justifying events based on analogies by similarity, end up not fully explaining why a similar instance resulted in a specific output. therefore, we propose a hybrid multilevel approach that will fill this gap, providing feature-based explanations that will demonstrate how their output was obtained for each selected example. one of the few works that has already developed hybrid approaches in the literature was mame (model agnostic multilevel explanations) [19]. as per mame, little attention has been given to obtaining insights at an intermediate level of model explanations. mame is a meta-method that builds a multilevel tree from a given method of local explanation. the tree is built by automatically grouping local explanations, without any hybrid character combining different types of explanation. the intermediate levels are based on this automatic grouping of the local explanations. furthermore, the local explanation methods evaluated in mame were not specifically built to work with regression problems, which can cause some damage when we start to have a continuous range of possible values for each instance output. finally, mame did not make the source code available and neither did they make available the tree generated for the only regression dataset used in their work (auto-mpg), which makes any kind of comparison between the approaches difficult. 3 hybrid multilevel explanations fig. 1. example of the multilevel tree built by humie. in this figure, the squares represent instances, and the colors of the instances the feature values used to calculate similarities. full size image the section introduces hybrid multilevel explanation (humie), an explanation method that combines both example-based and feature-base explanations. humie produces a tree where each node represents a prototype selected from the training set. after prototypes are selected, a feature-based approach is used to provide further details on the models decision. the dynamics of creating this explanation tree is exemplified in fig. 1. in the figure, each square represents an instance from the training set (t), and the colors of the squares reflect the values of the features of the instances. we calculate the similarity between instances using these features. in this example, humie first selects two prototypes as the root of the tree (r). after that, the distance from the remaining instances to each prototype is calculated, and each instance associated with its most similar prototype, generating 2 subgroups. for each of these subgroups, again two children prototypes are chosen. this process is repeated until it reaches the height (h) of the tree defined by the user. having a tree of prototypes, for each node of the tree, humie shows the most relevant features considering the prototype associated with that node, together with the range of values assumed by the instances of the subgroup represented by that prototype. finally, humie allows for a local explanation of a test instance by selecting the leaf-level prototype most similar to the instance to be explained. different algorithms can be used to select both the prototypes and to determine features importance. in this paper, we use m-peer (multiobjective prototype-based explanation for regression) [7] to select prototypes and dela (dynamic explanation by local approximation) [5] to determine feature importance, although they could be replaced by other algorithms. 3.1 prototype selection the prototype selection is done by m-peer. m-peer is based on the spea-2 evolutionary algorithm and simultaneously optimizes three metrics: fidelity of global prototypes, stability of global prototypes, and root mean squared error (rmse). the fidelity and stability measures were originally proposed in the context of classification [14, 18] and adapted in [7] for regression. the fidelity metric assesses how close the output of the chosen prototypes is from the output of the instances to be explained, and is defined in eq. 1. $$\begin{aligned} \overline{gf}(f,pr_t,t) = median_{\forall t \in t}(lf(f(t), pr(t))) \end{aligned}$$ (1) $$\begin{aligned} lf(f(t'),pr(t')) = dist(y_i,y_j) = (y_i y_j)^2 \end{aligned}$$ (2) the second metric, global stability, measures whether the characteristics of the prototypes are similar to the characteristics of the instances to be explained, as defined in eq. 3. $$\begin{aligned} \overline{gs}(t,pr_t) = median_{\forall t \in t}(ls(t, pr_{t})) \end{aligned}$$ (3) $$\begin{aligned} ls(x_i,x_j) = dist_{mink}(x_i,x_j) = (\sum \nolimits _{a=1}^{d} |{x_i}_a {x_j}_a|^{0.3})^{1/{0.3}} \end{aligned}$$ (4) where \(x_i\) and \(x_j\) are the features of instances to be compared, d is the number of dimensions, and \(dist_{mink}\) is the minkowski distance. in addition, \(\overline{gs}\) and \(\overline{gf}\) are normalized using min-max. in m-peer, the authors observed that the obtained results were better when the \(\overline{gs}\) and \(\overline{gf}\) metrics are used together through the gfs combination, shown in eq. 5. by using gfs we make the similarity take into account both the features of the instance and the output predicted by the model. $$\begin{aligned} gfs(f,pr_t',t')= \sqrt{ \left[ {\frac{1}{\log (\overline{gf}(f,pr_t',t'))}} \right] ^2 + \left[ {\frac{1}{\log (\overline{gs}(t',pr_t'))}} \right] ^2} \end{aligned}$$ (5) finally, the error is used to ensure that the selected set of prototypes is as close as possible to the error of the black-box model. 3.2 feature importance after selecting the prototypes, we want to obtain a feature-based explanation showing what motivated the output obtained for the specific instance. for this, we use the dela explanation method. dela is an improvement on the work presented by ela (explanation by local approximation) [6]. by analyzing the coefficients found in local linear regressions, this method provides a global explanation of the model, using a stacked area chart where the user can check the feature importance for different output values, and also provides local explanations for specific instances through the local importance of the features, expected value ranges, and the coefficients of the local explanation. additionally, dela considers the characteristics of the data sets used in the model training stage, chooses the most appropriate distance measure to be used, dynamically defines the number of neighbors that will be used for local explanation according to the local data density, and calculates feature importance based on the location of the test instance. dela calculates the most relevant features of a prototype using eq. 6: $$\begin{aligned} importance_{x_i} = \frac{|levelimportance_i |\times 100}{\sum _{a=1}^{d} |levelimportance_a |} \end{aligned}$$ (6) \(levelimportance_i\) is defined as: $$\begin{aligned} levelimportance_i = b_i \times mean(v_i) \end{aligned}$$ (7) where \(b_i\) is the coefficient returned for feature i and \(v_i\) the vector of values found in the neighboring examples (set of closest instances) for feature i and the minimum and maximum values of each feature, extracted considering the prototype and the instances in its subgroup. 3.3 algorithm the method is illustrated in algorithm 1. it receives as input the training (t) and test (\(t'\)) sets, together with the height h of the explanation tree to be built and the number r of root nodes present in the tree, where the root nodes represent the global explanations of the model. algorithm 1 . hybrid multilevel explanation (humie) full size image in line 1, we create the regression model to be explained. note that humie is model agnostic, meaning that it has the ability to explain any regression model. to ensure semantically valid results, humie works with continuous features only. any categorical feature should be pre-processed. next, we initialize some important variables for building the multilevel tree. in line 2, the tree is initialized as an array. this array will be filled with tuples containing three pieces of information: the selected prototype, the prototype’s depth level in the tree, and the prototype at the previous level (parent) of the selected prototype. in line 3, a dictionary is initialized to store, for each selected prototype, the minimum and maximum values of each feature found in the subset of instances from where the prototype was chosen. this information will serve in the future to show the minimum and maximum values of each feature within the subset that the prototype will be representing. in line 4, an array is initialized to store the prototypes already chosen during the tree construction process. finally, on line 5, we initialize an array to store the set of prototypes selected at the top level. as the explainer is built on a top-down fashion (i.e., starting from the root level), it has no higher-level prototype, and this variable is initialized with \(-1\). in line 6 it is defined that the number of prototypes of the root level is the value of r received as a parameter in the algorithm. after that, we carry out the selection of the prototypes that will belong to the tree in the loop between lines 7 and 29. note that the construction of this tree will start from the root, which will represent the global explanation of the model. after that, for each root level prototype selected, we expand it up to the level at limit h, defined by the user. in line 10 it is defined that for intermediate levels of the explanation tree we will have a binary division of the prototypes. in the loop of lines 11 to 13, we check, for all instances of the training set, which prototype of the previous level is the most similar to represent it. we are redistributing the training points to identify the most suitable prototypes in a subset. note that if we are at the first level this array will be an empty set (line 8). to measure the similarity between the instances we used the l2 of the normalized values of lf and ls measure (eq. 8). $$\begin{aligned} \begin{aligned} l2fs(f,pr_{t'},t')= \sqrt{ \left[ {\frac{1}{\log (\overline{lf}(f(pr_{t'}), f(t')))}} \right] ^2 + \left[ {\frac{1}{\log (\overline{ls}(t',pr_{t'}))}} \right] ^2} \end{aligned} \end{aligned}$$ (8) \(\overline{lf}\) and \(\overline{ls}\) are normalized values by the min-max of the original metrics. in the loop between lines 15 and 28 we find the prototypes that represent each of the subsets divided by the prototypes in the previous iteration. to accomplish this, in line 20, we check the indexes of valid training points, i.e., all training points explained by the same prototype in the previous tree level, excluding instances already selected as prototypes. the conditional performed on line 17 verifies whether the chosen prototypes are from the root level, if so, all instances of the training set make up the list of valid indexes. in line 20, the m-peer method is called for this subset of instances and the chosen prototypes are added to the tree in line 21. in line 22, the minimum and maximum values of all instances used to select each of the prototypes are checked, and in line 24 the set of selected prototypes is added to the array of previously selected prototypes. finally, the condition in line 25 tests whether we have reached the last level h defined by the user, which corresponds to the leaf nodes of the tree that will be later associated with test instances. the selection of the most relevant attributes and the range of expected values for each attribute of similar instances is performed in the loop between lines 31 and 33 for selected prototype points at all levels of the tree. the same procedure of feature analysis is performed in the loop between lines 35 and 38 for the set of test instances. this is done by dela. in line 36, we check which leaf node is closest to the test point to associate the two in the final view of the tree. finally, on line 39, a function is called to draw the tree, showing the most relevant features and the range of expected features for the selected prototype instance and for the test instances. 4 experiments we perform both a quantitative and a qualitative evaluation of the prototypes chosen by humie and their characteristics. 4.1 experimental setup in the quantitative evaluation, we compare the prototypes chosen by humie for local explanation with those chosen by other methods of explanation based on examples: protodash [10] and the standalone version of m-peer. protodash is an example-based explanation method that selects prototypes by comparing the distribution of training data and the distribution of candidate prototypes, i.e., it does not take into account the model output. for that, we use the 25 datasets presented in table 1. to measure the quality of the prototypes, we use the classic root mean squared error (rmse) and also the metrics gf, gs and gfs presented in sect. 3.1. we also performed a qualitative analysis on the classic auto-mpg dataset, which portrays the fuel consumption of automobiles (miles per gallon mpg) according to their construction characteristics. table 1. dataset used in the experiments of the humie explanation method.full size table 4.2 results this section first presents a quantitative experimental analysis of the explainability method proposed. the models to be explained were built using a random forest regression algorithm, implemented with sklearnfootnote 1. all parameters were used with their default values. taking as input the model built using each of the datasets shown in table 1, we run the explanation methods protodash, m-peer and humie to compare the quality of the selected prototypes. for protodash, we selected 6 global explanation prototypes and assigned each of the test instances to the most similar. for the m-peer method we selected 3 global prototypes that are used for local explanation according to the proximity of the test instances. finally, in humie we select 3 global prototypes (the same as in the m-peer method) to generate a sublevel of 6 prototypes that were then used to explain the test instances. the results obtained for each dataset are shown in table 2, where the best ones are highlighted in bold. in addition, we show the critical diagrams of each of the metrics in the graphs in fig. 2. these diagrams were generated after carrying out an adapted friedman test followed by a nemenyi post-hoc with a significance level of 0.05. as we are comparing multiple approaches, bonferroni’s correction was applied to all tests [4]. in these diagrams, the main line shows the average ranking of the methods, i.e., how well one method did when compared to the others. this ranking takes into account the absolute value obtained by each method according to the evaluated metric. the best methods are shown on the left (lowest rankings). table 2. result of the quality of the prototypes chosen by different methods in the local explanation of the test instances.full size table fig. 2. critical diagrams of the choice of prototypes for local explanation of test instances according to rmse metrics (a), global stability (b), global fidelity (c) and gfs (d). from left to right, methods are ranked according to their performance, and those connected by a bold line present no statistical difference in their results. full size image as we can see, the creation of a prototype explanation intermediate level managed to keep the same rmse as its competitors, as not method was statistically superior than the others. furthermore, humie was statistically better than the explanation with only one m-peer prototype level in relation to global stability and gfs. humie was also statistically better than all competitors in terms of the global fidelity of the chosen prototypes. next, we show a qualitative analysis for auto-mpg, which has 398 instances described by 8 features. to evaluate the model error, we divided the data into two subsets, the first being the training set with 70% of the instances (279) and the test set with the remaining 30% (119). note that, in this case, the model error is used only to identify how good the explanation model is related to the original regression model to be explained. as our focus is the explanation provided and not minimizing the error, we do not perform any further evaluation, for example, through cross-validation. the auto-mpg dataset portrays a classic regression problem whose objective is to find the consumption of a car having as features constructive characteristics of the vehicle, they are: cylinders, displacement, horsepower, weight, acceleration, model year and origin. to evaluate the explanation provided by our humie approach we build a model using all available features in a random forest algorithm. the rmse error found on the test set was 0.36. figure 3 shows the explanation provided for the model. in the explanations provided in this work we defined the height h of the tree as 2 and the total number of prototypes in the global explanation of the root as 3. two test instances, one with a low output value and another with a high output value, were previously selected for local explanation. fig. 3. hybrid tree explanation for the auto-mpg dataset. (color figure online) full size image in the explanation, each node highlighted in gray represents a prototype selected from the training set and the nodes, highlighted in red represent two test instances selected for local explanation. in addition, within each of the nodes, the actual output value of the instance (y) and the respective value predicted by the regressor model (\(y'\)) are presented in the first line. the other lines of the tree node represent the 5 main features identified for that instance with their respective value (the most relevant number of features to be shown can easily be changed if the user wants to). the bars represent the observed threshold values between all valid closest points when finding the prototype. in the case of the first upper level (root of the tree) the limit is imposed by the values of all instances of the training set. in the case of the test node the limit is imposed by all the closest features that originated the prototype local explanation leaf (leaf node of the hybrid explanation tree). the red dot represents the feature-specific value for the instance. the green range represents the allowable variation for creating a similar instance, determined by the dela method. specifically, it represents the range of feature values found among instances that deviate no more than 10% from the output predicted by the model. when hovering over one of the features, the green range is shown numerically with a highlight balloon for the user. note that the global explanation prototypes managed to cover the solution space well, with the first prototype being a high fuel consumption car (14.5 mpg), the second an intermediate consumption car (19.08 mpg) and the last one a low fuel consumption car of fuel (28.05 mpg). in addition, among the most relevant features at this level, we find those that domain experts traditionally portray as determining the consumption of a car, we can highlight: weight, displacement, horsepower and acceleration. in the second level (intermediate explanation) we observe that the prototypes really represent subgroups of the prototypes of the global explanation. this observation can be made both in terms of the output returned by the model and the range for each of the features. in the output we can observe that the values obtained do not extrapolate to values of the side prototypes of the previous level. in the range of features, we observed, for example, that in the first prototype of the second level the displacement feature varied between 260 and 455 while in its parent prototype these values varied between 68 and 455, portraying a specification of the group. finally, we can also observe that the l2fs similarity metric was adequate when associating the leaf prototypes of the explanation tree with the test instances, since both the values returned by the regressor model and the feature intervals represent the best possible association in each specific scenario. fig. 4. lime explanation for the auto-mpg dataset full size image note that an explanation provided by the m-peer for this dataset would only consist of the 3 instances represented at the root level of the explanation tree. in this context, we would only have information about the values for each of the features for these instances. we can also compare our explanation with what we could obtain from a popular feature-based explanation method called lime (local interpretable model-agnostic explanations) [21]. lime explains an observation locally by generating a linear model (considered interpretable by humans) using randomly generated instances in the neighborhood of the instance of interest. the linear model is then used to portray the feature’s contribution to the output of the specific instance based on the model’s coefficients. using the same two test instances shown in fig. 3, we have the explanations provided by lime in fig. 4. as we can observe, lime would end up showing semantically invalid interpretations to the user, such as portraying a positive contribution of the “weight” feature for cars weighing below 2250.0, without a lower limit. thus, one could interpret, for example, the possibility of a negative weight for the car. furthermore, with our hybrid multilevel approach, we have access to both similar instances in the training set and the characteristics that make these instances relevant, providing an explanation that brings new information about the model. 5 conclusions and future work humie is a method capable of creating multilevel hybrid explanations for regression problems. by hybrid we mean that the provided explanation encompasses as many elements as explanations based on examples, through the selection of prototypes from the training set, as well as the presentation of explanations based on features such as, for example, the most relevant features and the expected values of features for the region of a given instance. by multilevel, we refer to the fact that humie presents the model’s global explanations in a tree format, with local explanations for specific test instances being at the leaves of this tree, global explanations at the root but also intermediate explanations, portraying semantically similar subgroups. humie uses the m-peer approach as a auxiliar method for choosing prototypes and the dela approach for features analysis. experiments with real-world datasets quantitatively showed that the prototypes chosen by humie were better than all competitors (m-peer and protodash) in relation to the fidelity metric and better than m-peer with a single level in relation to the stability metrics and gfs. qualitatively, we showed that humie is able to diversify the choice of prototypes according to the characteristics of the presented dataset, both in terms of model output and features. in addition, humie was able to find subgroups of instances that are similar, providing an intermediate interpretation between the local and global explanations which might be useful for the user to understand the moded’s rationale. in this context, humie was able to identify which features are most important according to the subgroup the instance belongs to. as future work, we propose to explore additional metrics to evaluate humie’s performance and the robustness of explanations, in order to gain a more comprehensive view of the method’s capabilities. furthermore, combining other explanation approaches, such as counterfactuals, during the construction of the hybrid explanation could contribute to an even more thorough understanding of the model. notes 1.https://scikit-learn.org/stable/modules/generated/sklearn.ensemble.randomforestregressor.html. references adadi, a., berrada, m.: peeking inside the black-box: a survey on explainable artificial intelligence (xai). ieee access 6, 52138–52160 (2018) article  google scholar  bien, j., tibshirani, r.: prototype selection for interpretable classification. ann. appl. stat. 5(4), 2403–2424 (2011) article  mathscinet  math  google scholar  carvalho, d.v., pereira, e.m., cardoso, j.s.: machine learning interpretability: a survey on methods and metrics. electronics 8(8), 832 (2019) article  google scholar  demšar, j.: statistical comparisons of classifiers over multiple data sets. j. mach. learn. res. 7, 1–30 (2006) mathscinet  math  google scholar  filho, r.m.: explaining regression models predictions. ph.d. thesis, universidade federal de minas gerais (2023) google scholar  filho, r.m., lacerda, a., pappa, g.l.: explaining symbolic regression predictions. in: 2020 ieee congress on evolutionary computation (cec), pp. 1–8 (2020) google scholar  filho, r.m., lacerda, a.m., pappa, g.l.: explainable regression via prototypes. acm trans. evol. learn. optim. 2(4) (2023) google scholar  grari, v., lamprier, s., detyniecki, m.: adversarial learning for counterfactual fairness. mach. learn. 11, 1–23 (2022) math  google scholar  guidotti, r., monreale, a., ruggieri, s., turini, f., giannotti, f., pedreschi, d.: a survey of methods for explaining black box models. acm comput. surv. 51(5), 9:31-93:42 (2018) google scholar  gurumoorthy, k.s., dhurandhar, a., cecchi, g.a., aggarwal, c.c.: efficient data representation by selecting prototypes with importance weights. in: icdm 2019 (2019) google scholar  kim, b., khanna, r., koyejo, o.o.: examples are not enough, learn to criticize! criticism for interpretability. in: lee, d., sugiyama, m., luxburg, u., guyon, i., garnett, r. (eds.) advances in neural information processing systems, vol. 29. curran associates, inc. (2016) google scholar  letzgus, s., wagner, p., lederer, j., samek, w., muller, k.r., montavon, g.: toward explainable artificial intelligence for regression models: a methodological perspective. ieee sig. process. mag. 39(4), 40–58 (2022) article  google scholar  loyola-gonzález, o.: black-box vs. white-box: understanding their advantages and weaknesses from a practical point of view. ieee access 7, 154096–154113 (2019) article  google scholar  melis, d.a., jaakkola, t.: towards robust interpretability with self-explaining neural networks. in: neurips 2018, pp. 7775–7784 (2018) google scholar  ming, y., xu, p., qu, h., ren, l.: interpretable and steerable sequence learning via prototypes. in: kdd 2019, pp. 903–913 (2019) google scholar  molnar, c.: interpretable machine learning, 2nd edn. lulu.com (2022). https://christophm.github.io/interpretable-ml-book mothilal, r.k., sharma, a., tan, c.: explaining machine learning classifiers through diverse counterfactual explanations. in: proceedings of the 2020 conference on fairness, accountability, and transparency, fat* 2020, pp. 607–617. association for computing machinery, new york, ny, usa (2020) google scholar  plumb, g., al-shedivat, m., xing, e.p., talwalkar, a.: regularizing black-box models for improved interpretability. corr (2019) google scholar  ramamurthy, k.n., vinzamuri, b., zhang, y., dhurandhar, a.: model agnostic multilevel explanations. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) annual conference on neural information processing systems 2020, neurips 2020. advances in neural information processing systems 33, 6–12 december 2020, virtual (2020) google scholar  ribeiro, m.t., singh, s., guestrin, c.: “why should i trust you?”: explaining the predictions of any classifier. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, kdd 2016, pp. 1135–1144. acm, new york, ny, usa (2016) google scholar  ribeiro, m.t., singh, s., guestrin, c.: “why should i trust you?” explaining the predictions of any classifier. in: kdd 2016, pp. 1135–1144 (2016) google scholar  schwab, p., karlen, w.: cxplain: causal explanations for model interpretation under uncertainty. in: neurips 2019, pp. 10220–10230 (2019) google scholar  zhao, q., hastie, t.: causal interpretations of black-box models. j. bus. econ. stat. 39(1), 272–281 (2021) article  mathscinet  google scholar  download references author information authors and affiliations universidade federal de minas gerais, belo horizonte, minas gerais, brazil renato miranda filho & gisele l. pappa instituto federal de minas gerais, sabará, minas gerais, brazil renato miranda filho authors renato miranda filhoview author publications search author on:pubmed google scholar gisele l. pappaview author publications search author on:pubmed google scholar corresponding author correspondence to renato miranda filho . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper filho, r.m., pappa, g.l. (2023). hybrid multilevel explanation: a new approach for explaining regression models. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_26 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords regression explanation multilevel hybrid explanation publish with us policies and ethics profiles gisele l. pappa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature question answering with texts and tables through deep reinforcement learning | springer nature link (formerly springerlink) skip to main content advertisement menu find a journal publish with us track your research search cart home intelligent systems conference paper question answering with texts and tables through deep reinforcement learning conference paper first online: 30 january 2025 pp 339–353 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) question answering with texts and tables through deep reinforcement learning download book pdf download book epub marcos m. josé  orcid: orcid.org/0000-0003-4663-43869, flávio n. cação  orcid: orcid.org/0000-0003-4771-60099,10, maria f. ribeiro9, rafael m. cheang  orcid: orcid.org/0000-0003-2434-13049, paulo pirozelli  orcid: orcid.org/0000-0002-4714-287x9 & … fabio g. cozman  orcid: orcid.org/0000-0003-4077-49359  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 394 accesses abstract this paper proposes a novel architecture to generate multi-hop answers to open domain questions that require information from texts and tables, using the open table-and-text question answering dataset for validation and training. one of the most common ways to generate answers in this setting is to retrieve information sequentially, where a selected piece of data helps searching for the next piece. as different models can have distinct behaviors when called in this sequential information search, a challenge is how to select models at each step. our architecture employs reinforcement learning to choose between different state-of-the-art tools sequentially until, in the end, a desired answer is generated. this system achieved an f1-score of 19.03, comparable to iterative systems in the literature. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others forcing-seq2seq model: an automatic model of title generation for natural text using deep learning chapter © 2022 cross-domain transfer of generative explanations using text-to-text models chapter © 2021 learning from similarity and information extraction from structured documents article 11 june 2021 1 introduction question answering (qa) is concerned with developing programs that can respond to questions posed in natural language. those systems are widely used to assist users with their queries without the need for direct human intervention. qa datasets typically contain question-answer pairs written in natural language; e.g., squad [17]. while textual data is the most common source of information, other formats, such as graphs [1] and tabular data [6], may contribute in different ways to the task. tabular data is particularly valuable for structured information, which can be challenging to retrieve in pure textual databases. a mix of textual and tabular data, thus, offers significant potential for improvement in qa systems. this work addresses qa problems that demand retrieving and aggregating information from both tabular and textual data. our testbed is the open table-and-text question answering (ott-qa) dataset [5], an open-domain multi-hop qa dataset. in this dataset, questions are provided without supporting texts, and to answer them, one is typically required to navigate through multiple passages, textual and tabular. performance on this dataset remains limited compared to purely textual datasets like squad, where models already outperform humans. currently, the most adopted strategies to qa combine retrieval and reader modules: a retriever module fetches relevant passages, whereas a reader module turns these passages into a meaningful answer [4, 7, 12]. in multi-hop datasets like ott-qa, this retrieval process needs to be done sequentially, moving from a piece of information to another, in order to get the required data (textual or tabular). even though recent non-sequential systems have attained greater performance, a sequential decision-making formulation seems to be essential to coordinate retriever and reader schemes. in this paper we wish to explore the boundaries of performance obtained with sequential decisions applied to standard modules, where we resort to reinforcement learning (rl) to learn strategies. rl is suitable for this task: while supervised learning depends on labeled [3], static data for the action sequences, in rl the sequence of decisions is built through evolving information. we thus propose a novel deep reinforcement learning (drl) approach for this task, which iteratively decides which actions to take (retrieve text, retrieve table, generate answer), using standard modules to conduct these actions.footnote 1 the paper is organized as follows. section 2 offers some necessary background. section 3 presents our proposed rl architecture. section 4 describes our experimental setup, while sect. 5 presents the results and their associated discussion. finally, sect. 6 presents our concluding thoughts. 2 background 2.1 open table-and-text question answering numerous qa datasets and architectures have been designed to address questions that require understanding textual information, such as squad, or tabular data, as mimicsql [20]. however, there are not many resources that handle both texts and tables. to bridge this gap, hybridqa has been introduced as a closed-domain multi-hop multimodal question-answering dataset [6]. in hybridqa, each question carries a table and multiple associated passages, and to answer a question, it is necessary to combine several of these sources together. ott-qa [5] is an english dataset with 45841 qa pairs built over hybridqa. its core aim is to serve as an open-domain dataset that combines text and tabular information while simultaneously increasing the complexity of the task. the dataset contains tables and texts with answers for every question in the dataset. an example of a qa pair from ott-qa is shown in fig. 1. fig. 1. qa pair from ott-qa [5]. the question asks who is the author of the series in which nonso aznozie played the character robert. to accurately answer this question, one must locate the relevant table labeled “nonso anozie” (highlighted in pink) and identify the cell with “prime suspect” (highlighted in yellow), by referencing the role “robert” (highlighted in green). it is then possible to retrieve the associated text, which reveals that the creator of the movie was “lynda la plante” (highlighted in blue). (color figure online) full size image to assess the answers, we employ two well-known evaluation metrics (popularized by the squad dataset [17] and used by ott-qa): exact match, the percentage of predicted answers that are exactly equal to the golden answers, and macro average f1-score, the overlap between predicted and golden answers. in this context, precision is the ratio of shared words between the golden and prediction answers over the total number of prediction words; and recall is the ratio of shared words over the number of words in the golden answer. the f1-score is the harmonic mean between the precision and the recall. 2.2 open-domain and question answering in open-domain qa, the questions might be about any topic and a system must find answers based on a given question and usually within a corpus of information. qa systems typically consist of two main components: a retriever and a reader. the retriever’s task is to search the corpus for relevant texts and passages, while the reader processes this retrieved information to generate a suitable answer. some approaches leverage neural transformers for both retriever and reader tasks [7, 12], whereas others use keyword-based models specifically for the retriever [2, 4]. retriever: a widely adopted keyword-based retriever model in qa is bm25 [18]. bm25 relies on tf-idf (term frequency–inverse document frequency) by incorporating document length normalization; hence, shorter documents matching the query are prioritized over longer ones with similar keyword matches. in contrast, dense passage retrieval (dpr) [9] overcomes the limitations of keyword matching by ranking passages based on semantic similarity. dpr utilizes two bert models trained jointly to produce dense vector representations of questions and passages—similarity between these vectors is estimated by the dot product, enabling more nuanced retrieval based on meaning. reader: the best current readers typically are based on transformer neural networks, which take concatenated inputs of questions and retrieved passages to generate answers. for example, bert-based models, as seen in guu et al. [7], focus on substring prediction, while encoder-decoder architectures like bart, as in lewis et al. [12], excel in text-to-text tasks. another notable reader is the fusion-in-decoder (fid) [8]. unlike simple concatenation of question and passages followed by direct use in an encoder-decoder network, fid independently processes each passage in the encoder and combines resulting vectors as input to the decoder. this method efficiently handles a larger number of documents. similarly, fusion-in-encoder (fie) [10] integrates information from multiple passages within the encoder rather than the decoder. fie creates a unified representation and incorporates cross-sample attention over all tokens across different samples. 2.3 architectures tested on ott-qa the authors of ott-qa [5] presented several solutions, each addressing different retrieval and reading strategies. the first solution, bm25-hybrider, establishes a baseline for ott-qa. it retrieves 1 to 4 texts or tables from the corpus using the bm25 algorithm. these retrieved passages are then input to a reader, which selects the answer with the highest confidence interval. the second solution, “iterative-retrieval + cross-block reader,” employs two iterative retriever variants. sparse uses bm25 to retrieve 10 text passages and 10 table segments. for each text passage, it retrieves 5 table segments, and vice versa. dense utilizes a dpr-like model for retrieval. it starts with 8 text or table passages and iteratively expands the search. both variants feed the retrieved passages into the cross-block reader, a fine-tuned etc transformer model, to generate the answer. the last solution introduced by the authors, the “fusion retriever + cross-block reader”, was shown to outperform the previous methods. this approach combines tables and text into blocks using bm25 and enhances queries with gpt-2. the fusion retriever is a dual-encoder setup, based on dpr, that retrieves k blocks based on their similarity to the question while maintaining the same cross-block reader to generate the answer. the core (chain of reasoning) qa system employs a retriever-reader architecture but includes two intermediary components, the linker and the chainer [14]. the retriever is built on dpr and is responsible for retrieving 100 tables from the corpus. subsequently, a linker system based on bert models retrieves passages from the corpus based on the rows obtained from the retrieved tables. as the linker may provide an excessive amount of information for the reader to process, the chainer’s role is to select the top 50 chains (table row and text passage), which consist of a table row and a corresponding text passage. the chainer accomplishes this task by employing a t0 neural network that, in zero-shot manner, assesses the probabilities of the model generating a question based on specific text passages and table rows. finally, to answer the question, the system uses a fid model, which relies on the information curated by the chainer. the chain-of-skills (cos) architecture [15] currently delivers the highest performance on ott-qa. the authors introduced a (non-sequential) modularization approach that facilitates multi-task training across various open-domain qa datasets. the tasks tackled include single retrieval, expanded query retrieval, entity span proposal, entity linking, and reranking. in the inference phase, the system first identifies 100 tables through single retrieval. these tables are subsequently divided into rows, and the reranking process selects the top 200 rows. for each row, the system collects 10 passages from the expanded query retrieval and one passage from linked entities. the system then leverages the same chainer used in core to select the top 100 passages, which are subsequently input into a fine-tuned fie reader tailored for ott-qa. another significant contribution is the “multi-modal retrieval of tables and texts using tri-encoder models” [11], which extends dpr’s capabilities by incorporating three small-bert networks for encoding questions, texts, and tables individually. 2.4 reinforcement learning in question answering one of the pioneering approaches to integrate deep reinforcement learning (drl) into qa is the reinforced ranker-reader (\(r^3\)) architecture [21] introduced in 2017. this framework focuses on joint training of the retriever (referred to as the ranker) and the reader, using the reinforce algorithm for the ranker and a supervised approach for the reader. a key advantage of this approach is the unsupervised nature of retriever training, which gives rewards based solely on the similarity of the reader’s output to the golden answer. similarly, mscqa [22] employs drl to train an action selector that decides among three components: invoking the reader, selecting additional passages via the retriever, and removing potentially incorrect answers from memory. in [3], a drl agent is designed with actions for retriever, reader, and cleaner functions, in order to tackle a complex multi-hop qa dataset, where the retriever iteratively uses previously retrieved passages to find new relevant information. reinforcement learning with human feedback (rlhf) is an approach that has gained traction in the development of natural language models. rlhf involves training models through a process that combines reinforcement learning, where models learn from their actions and consequences, with human feedback, which involves human reviewers providing feedback and rankings on model-generated content. one example is chatgpt (or gpt3.5), a conversational agent built upon the foundation of gpt3. while the specifics of chatgpt’s training process are not fully disclosed, available information indicates that the rlhf training was executed through the ppo algorithm to enhance chatgpt’s alignment with users and minimizing issues related to misinformation, toxicity, and harmful sentiments. 3 proposed architecture the multi-hop and multi-model nature of ott-qa presents a significant challenge due to the sequential retrieval process it requires. to address these challenges within an explicitly sequential formulation, we propose a novel architecture. our approach involves training a deep reinforcement learning (drl) agent that selects from a set of already trained modules in the existing literature. as done for mscqa [22], we employ a drl agent as an action selector to determine which pre-trained component should be activated. as depicted in fig. 2, we have the following actions: retrieve texts, retrieve tables, and generate the answer. given that ott-qa does not specify the correct path for information retrieval, including the sequence of passages to retrieve, we have adopted a reinforcement learning approach. we incorporate a delayed reward mechanism, which compares the predicted answers to the gold standard answers at the end of each episode. fig. 2. proposed architecture. at each time step, the agent selects one of three actions: retrieve texts, retrieve tables, or generate the answer, based on the question and the information gathered so far. full size image one notable advantage of our proposed architecture is its flexibility. all components, including the reader and retrievers, can be easily replaced with newer and superior models as they become available for public use. furthermore, the system can be enhanced by incorporating additional components, such as a graph retrieval model, or employing multiple readers optimized for specific scenarios. the rl agent learns when to use each component in different situations through experiential training. 3.1 initial concepts our approach draws inspiration from the iterative-retrieval + cross-block reader architecture 5 described in sect. 2.3, being also an iterative system. for a given question, the initial retrieval step involves searching for 10 passages, which can be either textual or tabular in nature. these passages are used to create 10 different blocks. should the agent opt to perform an additional retrieval step, it concatenates the items within each block to retrieve four new passages, be they textual or tabular. the agent may undergo one more retrieval step, leading to the selection of 4 new passages for each block. to maintain the quality of retrieved information and prevent excessive noise, we limit the maximum number of retrieval steps to three. this limitation is intended to avoid an accumulation of uncertainty, as additional passages introduce more information, making it more challenging for the retrieval process to distinguish between valuable information and noise. suppose the agent decides to generate the answer or has reached the maximum number of steps. in that case, the reader is presented with the question and all non-repeated passages (note that different blocks may contain the same passage). the reader utilizes the reader input to generate a predicted answer \(a^{p}\), which is subsequently compared to the golden answer \(a^g\). in cases where the reader is invoked before any passage has been retrieved, a preliminary table retrieval is run to ensure that a minimum of 10 documents are available for processing. 3.2 reward since we lack prior knowledge of the correct sequence of actions for the agent, we cannot use supervised learning and must rely on the comparison between the expected (golden) answer and the agent’s predicted answer to construct the reward function. we started with the reward function \(r^3\), which trains the retriever indirectly, by using the final answer. however, due to the complexity of the problem, the −1 penalty for a generated answer with no intersection with the golden answer seemed excessive, especially considering that the iterative-retrieval + cross-block reader, our inspiration, achieves an f1-score of only 20.9 on the test set. therefore, we adjusted the penalty to −0.5. the following equation represents the reward at the end of the episode: $$\begin{aligned} r(a^g, a^{rc} ) = {\left\{ \begin{array}{ll} 2, \text {if } a^g = a^{rc}, \\ f_1 (a^g, a^{rc}), \text { else if } a^g \cap a^{rc} \ne \emptyset , \\ -0.5, \text {else,} \end{array}\right. } \end{aligned}$$ (1) where the reward is assigned a value of 2 when the golden answer \(a^g\) precisely matches the predicted answer \(a^{p}\). in cases of partial match, the reward is determined by the f1-score. lastly, when no match occurs, the reward defaults to −0.5, as mentioned before. we also introduced a minor penalty of \(-0.02\) for each retrieval action taken. this penalty encourages the agent to minimize retrieving unnecessary texts and tables. 3.3 actions as previously mentioned, our approach involves three distinct actions: retrieve texts (a1), where a retriever model searches for relevant textual information; retrieve tables (a2), where a retriever model searches for relevant tables; and generate answer (a3), where a reader model produces the answer using extracted textual passages and tables. next, we discuss our choices for the reader and retriever models. we opted not to retrain any models from the literature, limiting ourselves to those publicly available. retriever: the retriever module gathers information for the reader. we evaluated two types: bm25 [18] and tri-encoder [11]. this approach allowed us to assess classical sparse retrieval and neural dense retrieval methods, akin to the iterative-retrieval + cross-block reader architecture. reader: a critical component of any qa system, the reader generates answers. for ott-qa, the publicly available fusion-in-encoder (fie) model from the cos system [15] demonstrated superior performance. notably, this model handles passages individually, making it versatile for handling multiple inputs. following the authors’ recommendations, we capped the number of input passages at 50, which yielded optimal results in our validation experiments. 3.4 state/observation the observation is the information that the agent uses for the decision-making process. in our case, this information comprises the question and the passages that have been retrieved up to that point. rather than working directly with pure text, we first convert these elements into numerical data using embeddings. we employ two distinct encoders to perform this transformation into embeddings. when the retriever is the tri-encoder, we utilize its embeddings to represent the question, retrieved passages, and retrieved tables. however, in the case of bm25, we leverage the representations provided by a fine-tuned mpnet model trained for semantic search in qafootnote 2. each embedding is a vector with length of 512 for tri-encoder (same as small-bert) and 768 for the mpnet model. this information is then translated into a sequence of 11 vectors, with one vector for the question and one for each of the 10 blocks. for each block, we calculate the average of the representation values for each passage, considering the question in isolation, generating the embedding sequence. 3.5 agent in drl applications, the agent’s primary function is to process observations and generate corresponding actions. our specific observations consist of a sequence comprising 11 vectors. in our exploration, we experimented with four distinct neural network architectures: multilayer perceptron (mlp), long short-term memory (lstm), gated recurrent unit (gru), and transformer. for the mlp, we flattened the entire sequence into a single vector, which meant adding zeros in the first step to account for the lack of retrieved passages at that point. in contrast, the other networks process this information as a sequence. to train the agent, we explored two widely-used algorithms: deep q network (dqn) [16] and proximal policy optimization (ppo) [19]. our implementation is based on the stable-baselines3 python library, selected for its robustness and reliability. the training workflow of the agent is depicted in fig. 3. fig. 3. training workflow of the proposed architecture. full size image 4 experiments 4.1 baselines before training the drl agent, we explored baseline scenarios using our architecture. these baselines involved fixed sets of actions chosen independently of the question and retrieved information. each conceivable combination of actions was systematically evaluated across a validation dataset. for example, one baseline involved the agent consistently choosing to retrieve texts (action a1) twice and then invoking the reader to generate the answer. in total, we explored 14 different configurations of actions, encompassing two retriever types—bm25 and tri-encoder—resulting in 28 baselines. the primary purpose of these baseline tests was to establish a benchmark for comparison, determining if the trained drl agent could outperform or at least match the performance of fixed action paths without contextual understanding. 4.2 training the training of the drl agent involved sampling a random question from the ott-qa training set for each episode. we evaluated the impact of training both retriever solutions and compared their performance. for the tri-encoder, we trained for a total of one million timesteps, a reasonable number considering the dataset size of 41,469 training questions. each question could take up to four timesteps (between retriever and reader), allowing for revisiting the same question multiple times. due to the higher inference time of bm25 (up to 20 times more than tri-encoder), we opted to train for only 100 thousand timesteps. we implemented the dqn and ppo algorithms, employing various neural network architectures, including mlp, lstm, gru, and transformer. subsequent subsections provide detailed explanations of these implementations. dqn: for training dqn, we made a few adjustments to the dedefault hyperparameters of stable-baselines3,footnote 3 primarily to account for the number of training steps. we set the buffer size to 500k (maximum number of transitions stored in the experience replay) and learning starts (number of steps taking random actions just to store some transitions in the buffer before training) to 50,000 for the system with tri-encoder as the retriever. for bm25, we decreased the learning starts by a factor of 5, taking into account that the training has only 100,000 steps. ppo: similarly to dqn, we mostly relied on the default hyperparameters for training using the ppo implementation from stable-baselines3. we set the number of steps to run before training as 128, with a batch size of 32, and 60 as the number of epochs to optimize the surrogate loss. as ppo is an actor-critic algorithm, it employs two different networks: the actor and the critic. instead of using entirely separate networks, a common approach is to use a feature extractor to avoid redundant computations. this involves using shared layers between the networks for pre-processing the input, with these shared layers being trained jointly. networks: below are the configurations for the feature extractors we used, maintaining consistency for both training algorithms. for ppo, we employed two hidden layers with 64 and 32 neurons for both the actor and critic networks after the feature extractor: mlp: for dqn, the mlp network is a simple network with three hidden layers having 512, 128, and 64 neurons. for ppo, the feature extractor is two-layered network with hidden sizes of 512 and 128. lstm: we used 2 lstm layers with a hidden size of 512 for the tri-encoder and 768 for the bm25 architecture to match the size of the vectors of the embedding sequence e, with a dropout of 0.1. additionally, there is one hidden layer with 128 nodes. gru: similar to lstm, but with the lstm layers replaced by gru layers. transformer: for the ppo feature extractor, we opted for two encoder transformer layers with two attention heads, a feature dimension matching the input embedding size, and the same 128 hidden layer. for dqn, we used the same architecture with an extra 64 neurons last layer. 5 results and discussion 5.1 baselines the complete results of our baselines are provided in table 1. for the bm25 retriever, it is evident that retrieving tables is less effective than retrieving texts. this imbalance results in the best sequence of actions always being the retrieval of texts and never tables, yielding an f1-score of 19.03. for the tri-encoder, the results are more balanced. the optimal sequence of actions involves retrieving texts, followed by tables, and then retrieving texts again, resulting in a total f1-score of 8.24. it is notable, however, that the tri-encoder performance when retrieving texts is considerably lower compared to bm25. table 1. baseline results for the ott-qa validation dataset using our framework with bm25 and tri-encoder as the retriever. in this experiment, we assumed that the agent always takes the same sequence of actions, regardless of the question. a1 corresponds to retrieving texts, a2 is retrieving tables, and a3 calls the reader to generate the answer.full size table 5.2 results for deep reinforcement learning agent we present detailed results for the trained drl agents: bm25. the performance of trained algorithms using the bm25 retriever is detailed in table 2. among the various training algorithms, the ppo agent equipped with a transformer neural network demonstrated remarkable results. surpassing other configurations, this agent achieved the same level of performance as the best baseline, which involved consistently selecting the a1 action regardless of the question or context. tri-encoder. results by algorithms utilizing the tri-encoder retriever are presented in table 2. notably, the best-performing agent emerged from the ppo algorithm, employing the mlp network, demonstrating a f1-score of 6.44%. table 2. results for networks and training algorithms on the ott-qa dataset.full size table 5.3 discussion baseline results show a significant advantage for text retrieval over table search with bm25, even though some questions require information that is strictly only in tables. this may be due to bm25’s difficulty handling structured tables, where column presence and data types are crucial. despite its popularity in ott-qa solutions, the tri-encoder achieved f1-scores below 10%, potentially due to the lack of multi-hop training and reliance on text retrieval data from other datasets, as ott-qa lacks this information. the drl agents showed performance comparable to or slightly below the best baseline, particularly for bm25, often favoring text retrieval, which aligns with the optimal baseline strategy. table retrieval proved noisy, especially in iterative search, as tables often contain excessive data with relevant information typically in a single row. this issue may be mitigated by breaking tables into rows or smaller content sets, as suggested by other researchers [14]. the ppo algorithm generally showed slightly lower performance, with one exception: the ppo agent with a transformer network for bm25, which achieved the best performance by converging to the trivial solution of only picking the best action on average (retrieving texts). the choice of neural network architecture had minimal impact, indicating the need for further experiments. our best-performing drl agent achieved an f1-score of 19.03 using bm25, approaching the 20.7 score of the iterative-retrieval (sparse) + cross-block solution, which inspired our approach. however, our dense approach reached an f1-score of 8.24, compared to their 18.5, suggesting that the tri-encoder setting requires further refinement. our solution has a few limitations. the exploratory nature of reinforcement learning (rl) and the stochastic behavior of neural networks benefit from averaged multiple test runs. however, due to time constraints—each training execution took over a week—we were not able to perform multiple runs. additionally, we only used 100k training steps with the bm25 retriever. this number of steps is potentially insufficient given the dataset’s 41,469 training questions, each potentially requiring up to three steps. finally, on the tri-encoder’s performance, it is important to stress that this model was not trained for iterative retrieval, likely affecting its efficacy. 6 conclusion and future work in this work, we introduced a novel system designed to tackle multi-hop questions across texts and tables. our approach relies on a drl decision-maker system that iteratively selects among three distinct modules: text retriever, table retriever and reader. our best system achieved an f1-score of 19.03, closely competing with the non-drl iterative-retrieval (sparse) + cross-block system, which scored 20.7. we emphasize that our goal here is to explore sequential strategies; other (non-sequential) strategies have been explored with success—for instance, cos, which achieved an f1-score of 63.2 on the dev set. despite the comparatively lower performance, we believe there is significant value in solutions that explore sequential decisions as they have potential for modular improvement by refining various system modules. solutions that combine sequential decisions with non-sequential modules seem to be the most profitable avenue to explore in future work; here we have explored the performance boundary of sequential decisions using reinforcement learning. future work should also include implementing the joint-reranking approach [13], which uses a bert-based architecture to score and select top passages, enhancing the system’s ability to clear excess passages and tables from memory. we are also exploring alternative methods for generating observations for the agent. currently, we average the embeddings of each block, but we are considering encoding all distinct retrieved passages as a sequence. additionally, integrating our drl agent with the cos architecture, which excels in ott-qa by employing diverse retrieval strategies, could enhance performance. the drl agent would determine optimal retrieval paths based on the question and retrieved passages. notes 1.data and code available at: https://github.com/mmenonj/drl_qa_tt. 2.the mpnet model can be at https://huggingface.co/sentence-transformers/multi-qa-mpnet-base-dot-v1. 3.https://stable-baselines3.readthedocs.io. references berant, j., chou, a.k., frostig, r., liang, p.: semantic parsing on freebase from question-answer pairs. in: conference on empirical methods in natural language processing (2013) google scholar  cação, f.n., josé, m.m., oliveira, a.s., spindola, s., costa, a.h.r., cozman, f.g.: deepagé: answering questions in portuguese about the brazilian environment. in: britto, a., valdivia delgado, k. (eds.) intelligent systems, pp. 419–433. springer, cham (2021) chapter  math  google scholar  cação, f.n., reali costa, a.h.: a deep reinforcement learning approach to complex open-domain question answering. master’s thesis, univ. de são paulo (2023) google scholar  chen, d., fisch, a., weston, j., bordes, a.: reading wikipedia to answer open-domain questions. in: annual meeting of the association for computational linguistics, pp. 1870–1879. association for computational linguistics, vancouver, canada (2017). https://doi.org/10.18653/v1/p17-1171 chen, w., chang, m.w., schlinger, e., wang, w.y., cohen, w.w.: open question answering over tables and text. in: international conference on learning representations (2021) google scholar  chen, w., zha, h., chen, z., xiong, w., wang, h., wang, w.y.: hybridqa: a dataset of multi-hop question answering over tabular and textual data. in: findings of the association for computational linguistics, pp. 1026–1036. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.91 guu, k., lee, k., tung, z., pasupat, p., chang, m.w.: realm: retrieval-augmented language model pre-training. in: international conference on machine learning, partf168147-6, pp. 3887–3896 (2020) google scholar  izacard, g., grave, e.: leveraging passage retrieval with generative models for open domain question answering. in: conference of the european chapter of the association for computational linguistics, online, pp. 874–880. association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.eacl-main.74 karpukhin, v., et al.: dense passage retrieval for open-domain question answering. in: conference on empirical methods in natural language processing, pp. 6769–6781. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.emnlp-main.550 kedia, a., zaidi, m.a., lee, h.: fie: building a global probability space by leveraging early fusion in encoder for open-domain question answering. in: conference on empirical methods in natural language processing, abu dhabi, united arab emirates, pp. 4246–4260. association for computational linguistics (2022) google scholar  kostić, b., risch, j., möller, t.: multi-modal retrieval of tables and texts using tri-encoder models. in: fisch, a., et al. (eds.) 3rd workshop on machine reading for question answering, punta cana, dominican republic, november 2021, pp. 82–91. association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.mrqa-1.8 lewis, p., et al.: retrieval-augmented generation for knowledge-intensive nlp tasks. in: advances in neural information processing systems, vol. 33, pp. 9459–9474. curran associates, inc. (2020) google scholar  li, a.h., ng, p., xu, p., zhu, h., wang, z., xiang, b.: dual reader-parser on hybrid textual and tabular evidence for open domain question answering. in: annual meeting of the association for computational linguistics and the joint conference on natural language processing (volume 1), pp. 4078–4088. association for computational linguistics (2021). https://doi.org/10.18653/v1/2021.acl-long.315 ma, k., cheng, h., liu, x., nyberg, e., gao, j.: open-domain question answering via chain of reasoning over heterogeneous knowledge. in: conference on empirical methods in natural language processing, abu dhabi, united arab emirates, pp. 5360–5374. association for computational linguistics (2022) google scholar  ma, k., cheng, h., zhang, y., liu, x., nyberg, e., gao, j.: chain-of-skills: a configurable model for open-domain question answering. in: rogers, a., boyd-graber, j., okazaki, n. (eds.) annual meeting of the association for computational linguistics (volume 1: long papers), toronto, canada, pp. 1599–1618. association for computational linguistics (2023). https://doi.org/10.18653/v1/2023.acl-long.89 mnih, v., et al.: playing atari with deep reinforcement learning. corr abs/1312.5602 (2013) google scholar  rajpurkar, p., zhang, j., lopyrev, k., liang, p.: squad: 100,000+ questions for machine comprehension of text. in: conference on empirical methods in natural language processing (ii), pp. 2383–2392 (2016). https://doi.org/10.18653/v1/d16-1264 robertson, s., zaragoza, h.: the probabilistic relevance framework: bm25 and beyond, vol. 3 (2009). https://doi.org/10.1561/1500000019 schulman, j., wolski, f., dhariwal, p., radford, a., klimov, o.: proximal policy optimization algorithms. corr abs/1707.06347 (2017) google scholar  wang, p., shi, t., reddy, c.k.: text-to-sql generation for question answering on electronic medical records. in: web conference, pp. 350–361. association for computing machinery, new york (2020). https://doi.org/10.1145/3366423.3380120 wang, s., et al.: r3: reinforced ranker-reader for open-domain question answering. in: aaai conference on artificial intelligence. aaai press (2018) google scholar  wang, y., jin, h.: a deep reinforcement learning based multi-step coarse to fine question answering (mscqa) system. aaai conf. artif. intell. 33(01), 7224–7232 (2019). https://doi.org/10.1609/aaai.v33i01.33017224 article  math  google scholar  download references acknowledgments this research has been carried out with support by itaú unibanco s.a. through the scholarship program programa de bolsas itaú (pbi); it is also supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior (capes), finance code 001, brazil. the authors would like to thank the center for artificial intelligence (c4ai-usp), with support by the são paulo research foundation (fapesp) under grant number 2019/07665-4 and by the ibm corporation. paulo pirozelli is supported by the fapesp grant 2019/26762-0. any opinions, findings, and conclusions expressed in this manuscript are those of the authors and do not necessarily reflect the views, official policy or position of the itaú-unibanco, fapesp, ibm, and capes. author information authors and affiliations escola politécnica, universidade de são paulo, são paulo, brazil marcos m. josé, flávio n. cação, maria f. ribeiro, rafael m. cheang, paulo pirozelli & fabio g. cozman novelo data, são paulo, brazil flávio n. cação authors marcos m. joséview author publications search author on:pubmed google scholar flávio n. caçãoview author publications search author on:pubmed google scholar maria f. ribeiroview author publications search author on:pubmed google scholar rafael m. cheangview author publications search author on:pubmed google scholar paulo pirozelliview author publications search author on:pubmed google scholar fabio g. cozmanview author publications search author on:pubmed google scholar corresponding author correspondence to marcos m. josé . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper josé, m.m., cação, f.n., ribeiro, m.f., cheang, r.m., pirozelli, p., cozman, f.g. (2025). question answering with texts and tables through deep reinforcement learning. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_24 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords reinforcement learning large language models question answering multi-hop publish with us policies and ethics profiles paulo pirozelli view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature applying theory of mind to multi-agent systems: a systematic review | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper applying theory of mind to multi-agent systems: a systematic review conference paper first online: 12 october 2023 pp 367–381 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) applying theory of mind to multi-agent systems: a systematic review download book pdf download book epub michele rocha9, heitor henrique da silva9, analúcia schiaffino morales9, stefan sarkadi10 & … alison r. panisson9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 1035 accesses 11 citations abstract life in society requires constant communication and coordination. these abilities are efficiently achieved through sophisticated cognitive processes in which individuals are able to reason about the mental attitudes and actions of others. this ability is known as theory of mind. inspired by human intelligence, the field of artificial intelligence aims to reproduce these sophisticated cognitive processes in intelligent software agents. in the field of multi-agent systems, intelligent agents are defined not only to execute reasoning cycles inspired by human reasoning but also to work similarly to human society, including aspects of communication, coordination, and organisation. consequently, it is essential to explore the use of these sophisticated cognitive processes, such as theory of mind, in intelligent agents and multi-agent systems. in this paper, we conducted a literature review on how theory of mind has been applied to multi-agent systems, and summarise the contributions in this field. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a computational model for agents in a social context: an approach based on theory of mind chapter © 2023 an epistemic logic for modular development of multi-agent systems chapter © 2022 a computer science perspective on models of the mind chapter © 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. attribution theory emotion theory multiagent systems philosophy of mind philosophy of artificial intelligence agent-based economics 1 introduction theory of mind (tom) refers to the ability of individuals to attribute mental states to themselves and others, including beliefs, desires, intentions, and emotions. it is an important cognitive skill that enables us to understand and predict the behaviour of others. additionally, it plays a fundamental role in communication and social interaction [24]. a multi-agent system (mas) is a system composed of multiple autonomous agents that interact with each other to achieve a common goal. these agents can be humans or software, and the system’s goals are accomplished through collaboration, negotiation, and coordination among them [68]. the conceptualisation of mas takes inspiration from by human society, and considers approaches of modelling not only autonomous intelligent agents as individuals, but the aspects concerning the societies made up of these agents. furthermore, the field of mas also explores hybrid systems, where humans are considered as part of the mas, which aligns mas with emerging concepts such as hybrid intelligence [1]. although there have been fascinating achievements in the field of mas, including models for describing and implementing sophisticated reasoning mechanisms and communication methods, there has always been a continuous exploration of incorporating human cognitive capabilities, such as tom, within the field of mas. by integrating tom into autonomous intelligent agents and, consequently into mas composed of these agents, there is a promise of achieving sophisticated capabilities that were previously seen only in humans, human interactions, and human societies. this integration also opens up new possibilities for systems where software agents and humans can work together synergistically, leading to the development of systems that align with the concept of hybrid intelligence [1]. in this paper, we present a systematic review which aims to give a brief, but comprehensive, account of the state-of-the-art in applying tom in the area of mas. our goal is to identify what is lacking to achieve the ambitious vision of an ai-human society. it is crucial to evaluate current approaches and techniques in artificial and cognitive intelligence, as well as to address the limitations and challenges involved in reaching these goals. there are four sections in the paper. section 2 presents an overview of the systematic review methodology, including the identification of research gaps, the search strategies used, and the selection and exclusion criteria used for articles. in sect. 3, the findings are discussed in relation to the specific points of interest outlined earlier. finally, the paper concludes with a few final remarks and references. 2 methodology in order to conduct the systematic review, preferred reporting items for systematic reviews and meta-analyses (prisma) guidelines [41] were adhered to. the objective of the study was to fill a few research gaps regarding tom and mas employing the spider as a strategy defining the following aspects [15]: sample (s), phenomenon of interest (p), design (d), evaluation (e), and research type (r). this strategy is useful for systematic reviews involving studies of different designs and types of interventions. below, each aspect for this strategy is detailed: sample: studies focused on the tom and mas context. phenomenon of interest: theoretical and empirical research on applying tom to mas. interest/information sources: peer-reviewed articles, conference proceedings, and book chapters published in english, available online. design: qualitative and quantitative studies, theories, and case studies. evaluation: studies that examine the relationship between tom and mas, or applied tom to mas. research type: primary research studies, including experimental and non-experimental designs. and the applications areas (health, education, games, personal assistant). the following research questions were used to identify such gaps: (rq1) what theory underlies the use of the theory of mind? what is the reference work? (here, we may have multiple works grounded in the same theory, which could be from psychology, philosophy, etc.). (rq2) which mental attitudes are modelled by the work? (for example beliefs, memories, intentions, emotions, etc.). (rq3) how does the system model knowledge related to the theory of mind? what is the form of representation for information about the theory of mind? (rq4) does the work model theory of mind of human users or only computational entities (software agents)? (rq5) what is the application area of the work? (for example, health, games, personal assistants, theoretical works, etc.) (rq6) what technologies were used to develop the work? (for example, modelling and simulation platforms, algorithms, ai techniques, etc.) (rq7) what are the main results achieved? (rq8) what are the main challenges encountered? (rq9) what are the limitations pointed out by the work? 2.1 databases four databases were utilised for this study, each of which was carefully selected based on their relevance to our research topic. specifically, we focused on databases that were most closely aligned with the scope of our study, and that had a proven track record of returning a significant number of relevant papers. our aim was to ensure that we had access to the most comprehensive and up-to-date literature on the subject, in order to provide a rigorous and thorough analysis. in this regard, the databases we selected were instrumental in providing us with the necessary information to carry out our review. we examined pubmed, web of science, scopus, and ieee xplore databases. grey literature is not considered in this study. table 1. inclusion and exclusion criteria.full size table 2.2 adopted criteria and selection procedures this study employed an accurate search approach, utilising four electronic databases (see sect. 2.1) to identify relevant scientific articles on tom and mas. after the search process, it was applied a two-stage selection process. in the first stage, the researchers screened the titles, abstracts, and keywords of each article to identify potentially relevant studies. subsequently, studies that did not meet the inclusion criteria were excluded from the analysis. in cases where there was uncertainty during the initial evaluation, a second evaluation of the full text was conducted, in which we were able to evaluate if those studies were part of the scope of this literature review. the inclusion and exclusion criteria for all studies are outlined in table 1. across all databases, a string with the following keywords was investigated. note that this search string focus on searching articles that use synonyms for ‘agent’ in the context of software agents, but also including the concept of ‘theory of mind’ specifically, which is the interest of this study. 2.3 selection process the selection process began with 698 articles, including 144 from pubmed, 184 from wos, 311 from scopus database, and 58 from ieee xplore. out of the 698 articles, 126 were removed due to duplication. from the remaining 572 articles, 456 were excluded in the initial stage as they did not meet the selection criteria. five reviewers participated in the first stage, with each assigned to read the title and abstract of all 572 articles that remained after removing the duplicates. the review was conducted in a blinded manner using the rayyan toolfootnote 1 to minimise the influence of individual researchers’ decisions. subsequently, the blinding review was removed, allowing for a comparison of the researchers’ decisions. in this stage, a set of rules was established to facilitate the selection process. articles receiving three or more votes for inclusion were selected for full reading, while those receiving four or more votes for exclusion were directly excluded. in cases where conflicting votes occurred, the reviewers convened to make a final decision. applying these criteria, a total of 116 articles were selected for full reading and subsequently categorized into a separate section. with the final set of 116 articles for full reading, each researcher was tasked with answering the questions created before the reading. in this part of the research, three researchers participated. after reading the articles, 22 were excluded because they were out of scope (mostly related to the study of tom in humans), 16 were excluded because they were only published in workshops, 11 were excluded because they were short papers (extended abstracts), 4 were excluded because they were books, 2 articles were excluded because they could not be found online for free, and 1 article was excluded because it was a survey. fig. 1. prisma diagram. full size image fig. 2. papers by year of publication. full size image table 2. main references used by the papers.full size table figure 1 presents the prisma diagram of the systematic review, which illustrates the eligible investigation process. 2.4 selected articles in the end, 60 articles remained and were included in this study. the selected studies were published between 2002 and 2022. we observe an increase in the popularity of the topic from 2019 to 2022. however, there is a slight decrease in publications from 2020 to 2022, which could be attributed to external factors, such as the covid-19 pandemic. the pandemic could have limited the participation of researchers in events and access to university and research labs as a whole. figure 2 displays the distribution of the selected articles by year. from the 60 selected articles, 26 of them are from academic journals and 34 from conference proceedings. 3 results 3.1 answer to rq1 (main references) table 2 shows those works that were referenced as the theory that underlines the use of tom, or as the primary reference to the work. only those works referred to by multiple sources are included. furthermore, in 25% of the articles included, the main reference could not be identified since the authors did not provide this information explicitly. of the remaining articles (approximately 19%), while they did provide this information, none of those main references were referenced by multiple articles. table 3. mental attitudes explicitly modelled.full size table table 4. knowledge representation.full size table 3.2 answer to rq2 (modelled mental attitudes) table 3 summarises the mental attitudes (or states) modelled by the selected articles. it is notable that most of the research focuses on mental attitudes commonly present in agent architectures, including the well-known bdi (beliefs-desires-intentions) architecture [10]. additionally, some works focus on modelling memories (which are slightly stronger than beliefs), emotions, and goals. 3.3 answer to rq3 (knowledge representation) table 4 summarises the distribution of the selected articles according to how they represent the knowledge related to the tom. a formal representation is presented in approximately 65% of the works, including epistemic logic, bayesian modeling, state transition systems, and mdps. in addition, many of the works utilise symbolic representations, which are similar to agent-oriented programming languages. furthermore, some works use a representation based on vectors of characteristics, which is typically used in machine learning techniques. finally, there are works that use natural language representation, and one work differs from others by using semantic databases (ontologies) to represent tom. table 5. entities modelled.full size table table 6. application areas.full size table 3.4 answer to rq4 (modelled entities) table 5 summarises the distributions of the selected articles based on whether they modelled tom for other software agents, for humans (e.g., human users), or for both. most of the research focuses on modelling tom for other software agents and studying the diverse phenomena that can emerge from agents with this capability. some works focus on modelling tom for both software agents and humans, often introducing abstract approaches and logic that can incorporate the modelling and reasoning about tom. finally, there are some works that focus on modelling tom for humans, particularly related to personal assistants. in these works, an software agent aims to improve its decision-making and interaction by using tom. 3.5 answer to rq5 (application areas) table 6 summarises the distributions of the selected articles according to their application domains. it can be noted that most of the research focus on providing approaches without any particular application domain, providing theoretical proofs and definitions over the proposed logic or framework. regarding the works that apply the proposed approach and explore applications, there is a notable interest in the areas of games, human-agent interaction and agent-based simulations. in the area of games, most works explore how tom can provide an advantage to agents with this capability, studying whether they outperform agents without this capability. additionally, some works focus on providing a more realistic experience for players and/or improving the naturalness of non-playable characters (npcs), with some also contrasting human-agent interaction in games. in the area of human-agent interaction, most works explore how tom can improve the software agent communication by anticipating humans thoughts, intentions, etc. and remembering important information such as their beliefs, goals, and preferences. in the area of agent-based simulation, most works focus on studying social phenomena in which individuals model tom. those studies include emergent behaviours, agents profiles, and other related factors. furthermore, there are works focusing on how tom can be applied to: (i) identify false beliefs, (ii) deceive others (and similar dishonest behvaiours), (iii) train and/or implement better negotiators, (iv) achieve cooperation, (v) achieve better results in educational platforms, (vi) infer mental attitudes applied to robotics, and (vii) create better personal assistants. table 7. main technologies.full size table 3.6 answer to rq6 (technologies) table 7 presents the main technologies employed in the selected articles for this literature review. it is evident that the majority of the works concentrate on studies that necessitate a human-computer interface, enabling interaction between humans and agents, or showcasing emergent behaviours through agent-based simulations. additionally, a significant number of articles employ probabilistic approaches and machine learning as their primary technologies. a few studies employ robotics, planning, and agent-oriented programming languages as their main technologies. lastly, some works remain focused on theoretical aspects without utilising any specific technology. 3.7 answer to rq7 (main results) most of the selected works focus on providing formal properties, semantics, and definitions regarding their approaches for modelling and reasoning with tom [11, 13, 25, 35, 36, 42, 51, 54, 55, 69] or aim to provide proof of concepts [3, 9, 14, 16,17,18,19, 22, 28,29,30, 39, 44, 45, 47,48,49, 62,63,64]. for example, some studies demonstrate that tom improves the performance of agents, especially when they have access to more information. they also show that agents with tom outperform those who are not able to model and reason about tom [16, 63]. furthermore, many works aim to provide a computational model for tom, which can address various challenges, such as (i) dealing with uncertainty [21, 32, 54]; (ii) handling preferences [13]; (iii) based on machine learning models [34, 40, 50, 65]; (iv) using planning [58, 66]; (v) incorporating emotions [23, 52, 53, 59]; (vi) using ontologies [56]; among others [2, 8, 12, 20, 26, 27, 31, 38, 57, 61]. finally, one work provides a datasets as one of the main results [4]. some works provide empirical results, such as comparing the ability of humans to model tom with those modeled by agents [5], comparing the performance of agents with and without tom [33, 60], among others [50]. 3.8 answer to rq8 (main challenges) more than half of the selected papers do not point out any challenges in developing their approaches. most of the works that did mention challenges highlight the difficulty of dealing with the dynamism of real-world environments, which also creates a highly dynamic mental state of agents. for instance, in [4], the authors describe how changes in the physical world, which is very dynamic, affect agents’ beliefs, consequently the tom they have about each other, making it a challenging issue. in [18], the authors also describe the unpredictability of the environment as one of the main challenges. in [20], the authors explain the challenge of robots representing knowledge about their interactions with the environment and other agents. in [23], the authors discuss about the difficulty of capturing real-time gestures from human users and creating a more general model applicable to other domains. in [31], the authors emphasise the complexity of applying achieved results in real-world scenarios. in [12], the authors describe challenges associated with selecting the correct plan agents should use to achieve user-delegated goals, considering the dynamics of the environment and their individual state. also, in [39], the authors state that scalability is one of the main challenges, making it hard to consider more factors to make inferences related to tom. further, in [47], the authors describe the great challenge of including humans in the loop when considering tom. furthermore, in [2], the authors describe how challenging it can be for agents to decide on initial interactions when their behaviors are unknown. in [25], the authors point out the difficulty of developing cognitive models capable of making inferences about “common sense”. in [36], the authors describe the challenge of finding a balance between representing the explicit and implicit beliefs of others. in [22], the authors find it challenging to collect and use knowledge from crowdsourcing and to evaluate different narratives based on the diversity of phrases, actions, and sentiments. in [57], the authors describe the challenge of implementing complex social capabilities in humanoid robots. in [64], the authors highlight the difficulty of extending the proposed approach to real and more complex case studies, such as legal cases. in [52], the authors describe the challenge of integrating different emotional and cognitive theories into a single agent architecture. in [21], the authors describe the challenge of learning parameters from collected data and exploring different learning techniques. in [60], the authors describe the difficulty of training and measuring skills related to modeling and reasoning about tom. finally, in [11], the authors point out the inadequacy of mathematical frameworks from computer science to model tom. in [63], the authors describe that higher-order tom only gains an advantage over lower-order tom or other simple strategies when enough information is available, which is often hard to achieve in many scenarios. in [33], the authors describe the difficulty of dealing with wrong and uncertain tom. lastly, in [66], the authors describe the challenges related to debugging the system, particularly in their case, when debugging narrative planning problems. 3.9 answer to rq9 (main limitations) the works that applied subjective or quantitative evaluations supported by human users, such as [13, 28], indicated that the number of human users in the experiments was limited. also, some works have described the need to extend their approach with other mental attitudes and knowledge representations. for instance, in [19], the authors describe that they intends to extend their work to model desires and intentions in addition to beliefs, while in [66], the authors explain that their approach does not allow the representation of uncertainty, which may be necessary for certain applications. furthermore, in [58], the authors note that their approach only explored a limited number of agents. in [23, 45], the authors describe the specificity of their approaches as limitations. in [23], the authors highlight the difficulty of learning the user’s model, while in [36], the authors discuss inconsistencies that can arise from using propositional logic in their approach. additionally, in [31], the authors acknowledge that their simplified model for agents and environment could not capture all complexities of social interaction as they occur. moreover, in [39], the authors state that their approach is limited to inferring the intents of only one user and considers a limited number of intents. in [22, 30], the authors describe limitations regarding the technologies used, such as robotics and crowdsourcing. in [21], the authors describe limitations regarding the data used and consequently the generated model. in [63], the authors mention that they only explore two levels of tom, which could be a limitation of their work. in [33, 44], the authors assume perfect knowledge about other agents’ policies, which could also be a limitation of their work. in [25], the authors describe that they intend to explore other formal properties based on possible worlds. many articles also point out common limitations related to validation and evaluation methods, such as [12, 17, 28, 48, 52, 53, 60]. they often highlight limitations resulting from (i) using controlled environments for experiments that may not reflect the real world, and (ii) using a limited number of parameters and/or fixed instances of the problem, among others. finally, note that the majority of the articles do not mention limitations (their limitations, if any, could not be accessed for this literature review). 4 conclusion in this paper we have presented a short and systematic literature review of the approaches in mas used to apply tom, that is the concept from cognitive science understood as the ability of humans to model the minds of other agents. as future work, we plan to situate these approaches w.r.t. the ‘flavours’ of tom in cognitive science [24]. there seems to be a representational equivalence between the flavours of tom from cognitive science and types of ai modelling approaches. for instance theory-theory of mind (tt) would be the equivalent of symbolic ai, simulation theory of mind (st) of subsymbolic ai, and the hybrid theory of mind would be something similar to neurosymbolic ai. while tt represents the type of tom where agents already have an understanding to some extent of the other agents’ minds, e.g., a set or a knowledge base that consists of beliefs of others’ mental attitudes, st represents a process for using the already known set of others’ mental attitudes to simulate their minds in order to predict their behaviour. ht is a ‘high-level’ tom that combines tt and st in a practical reasoning process in the way its target (the other agent who’s mind is being mentalised) would. perhaps the most obvious application of ht in ai is to enable language, communication, and explanation [37], a component that is foundational for mas where agents must communicate in order to interact successfully with other artificial agents, or humans. this could enable future research directions in mas, in which simulation of intelligent agents are showed to humans, and humans are asked to estimate their mental attitudes, that means, humans building a tom about the mental attitudes of intelligent agents [43]. however, as we have seen from answering the literature review questions in this paper, there are quite a few limitations regarding the application of tom in mas, and quite a few challenges still remain to be addressed in order for tom in mas to showcase the promises regarding realistic human-like mentalisation in agent-agent interactions. furthermore, there might be even more limitations which more than half of the papers that presented applications of tom to mas did not even consider to report. notes 1.https://www.rayyan.ai/. references akata, z., et al.: a research agenda for hybrid intelligence: augmenting human intellect with collaborative, adaptive, responsible, and explainable artificial intelligence. computer 53(08), 18–28 (2020) article  google scholar  albrecht, s.v., crandall, j.w., ramamoorthy, s.: belief and truth in hypothesised behaviours. artif. intell. 235, 63–94 (2016) article  mathscinet  math  google scholar  aylett, r., et al.: werewolves, cheats, and cultural sensitivity. in: international conference on autonomous agents and multi-agent systems (2014) google scholar  bara, c.p., ch-wang, s., chai, j.: mindcraft: theory of mind modeling for situated dialogue in collaborative tasks. arxiv preprint arxiv:2109.06275 (2021) bard, n., et al.: the hanabi challenge: a new frontier for ai research. artif. intell. 280, 103216 (2020) article  mathscinet  math  google scholar  baron-cohen, s.: mindblindness: an essay on autism and theory of mind. mit press (1997) google scholar  baron-cohen, s., leslie, a.m., frith, u.: does the autistic child have a “theory of mind?’’. cognition 21(1), 37–46 (1985) article  google scholar  berthiaume, v.g., shultz, t.r., onishi, k.h.: a constructivist connectionist model of transitions on false-belief tasks. cognition 126(3), 441–458 (2013) article  google scholar  boella, g., van der torre, l., et al.: from the theory of mind to the construction of social reality. in: proceedings of the annual conference on the cognitive science society, vol. 5, pp. 298–303 (2005) google scholar  bratman, m.: intention, plans, and practical reason. harvard university press, cambridge (1987) google scholar  bringsjord, s., et al.: toward logic-based cognitively robust synthetic characters in digital environments. front. artif. intell. appl. 171, 87 (2008) google scholar  cantucci, f., falcone, r.: towards trustworthiness and transparency in social human-robot interaction. in: 2020 ieee international conference on human-machine systems (ichms), pp. 1–6. ieee (2020) google scholar  cantucci, f., falcone, r.: collaborative autonomy: human-robot interaction to the test of intelligent help. electronics 11(19), 3065 (2022) article  google scholar  chang, h.m., soo, v.w.: simulation-based story generation with a theory of mind. in: proceedings of the aaai conference on artificial intelligence and interactive digital entertainment, vol. 4(1), pp. 16–21 (2008) google scholar  cooke, a., smith, d., booth, a.: beyond pico: the spider tool for qualitative evidence synthesis. qual. health res. 22(10), 1435–1443 (2012) article  google scholar  de weerd, h., verbrugge, r., verheij, b.: higher-order theory of mind in negotiations under incomplete information. in: boella, g., elkind, e., savarimuthu, b.t.r., dignum, f., purvis, m.k. (eds.) prima 2013. lncs (lnai), vol. 8291, pp. 101–116. springer, heidelberg (2013). https://doi.org/10.1007/978-3-642-44927-7_8 chapter  math  google scholar  de weerd, h., verbrugge, r., verheij, b.: negotiating with other minds: the role of recursive theory of mind in negotiation with incomplete information. auton. agent. multi-agent syst. 31, 250–287 (2017) article  google scholar  de weerd, h., verbrugge, r., verheij, b.: higher-order theory of mind is especially useful in unpredictable negotiations. auton. agent. multi-agent syst. 36(2), 30 (2022) article  google scholar  dissing, l., bolander, t.: implementing theory of mind on a robot using dynamic epistemic logic. in: ijcai, pp. 1615–1621 (2020) google scholar  djerroud, h., chérif, a.a.: vica: a vicarious cognitive architecture environment model for navigation among movable obstacles. in: icaart, vol. 2, pp. 298–305 (2021) google scholar  doshi, p., qu, x., goodie, a.s., young, d.l.: modeling human recursive reasoning using empirically informed interactive partially observable markov decision processes. ieee trans. syst. man cybern. part a syst. hum. 42(6), 1529–1542 (2012) article  google scholar  feng, d., carstensdottir, e., el-nasr, m.s., marsella, s.: exploring improvisational approaches to social knowledge acquisition. in: 18th international conference on autonomous agents and multiagent systems, aamas 2019 (2019) google scholar  gebhard, p., schneeberger, t., baur, t., andré, e.: marssi: model of appraisal, regulation, and social signal interpretation. in: international conference on autonomous agents and multi-agent systems (2018) google scholar  goldman, a.i.: theory of mind. oxford university press, united kingdom (2012) google scholar  gouidis, f., vassiliades, a., basina, n., patkos, t.: towards a formal framework for social robots with theory of mind. in: icaart, vol. 3, pp. 689–696 (2022) google scholar  harbers, m., van den bosch, k., meyer, j.j.c.: agents with a theory of mind in virtual training. in: multi-agent systems for education and interactive entertainment: design, use and experience, pp. 172–187. igi global (2011) google scholar  hoogendoorn, m., merk, r.-j.: action selection using theory of mind: a case study in the domain of fighter pilot training. in: jiang, h., ding, w., ali, m., wu, x. (eds.) iea/aie 2012. lncs (lnai), vol. 7345, pp. 521–533. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-31087-4_54 chapter  google scholar  hoogendoorn, m., soumokil, j.: evaluation of virtual agents utilizing theory of mind in a real time action game. in: proceedings of the 9th international conference on autonomous agents and multiagent systems, vol. 1, pp. 59–66 (2010) google scholar  hu, q., lu, y., pan, z., gong, y., yang, z.: can ai artifacts influence human cognition? the effects of artificial autonomy in intelligent personal assistants. int. j. inf. manage. 56, 102250 (2021) article  google scholar  husemann, s., pöppel, j., kopp, s.: differences and biases in mentalizing about humans and robots. in: 2022 31st ieee international conference on robot and human interactive communication (ro-man), pp. 490–497. ieee (2022) google scholar  kaufmann, r., gupta, p., taylor, j.: an active inference model of collective intelligence. entropy 23(7), 830 (2021) article  google scholar  kelley, r., tavakkoli, a., king, c., nicolescu, m., nicolescu, m., bebis, g.: understanding human intentions via hidden markov models in autonomous mobile robots. in: proceedings of the 3rd acm/ieee international conference on human robot interaction, pp. 367–374 (2008) google scholar  lerer, a., hu, h., foerster, j., brown, n.: improving policies via search in cooperative partially observable games. in: proceedings of the aaai conference on artificial intelligence, vol. 34, no. 05, pp. 7187–7194 (2020) google scholar  lin, b., bouneffouf, d., cecchi, g.: predicting human decision making with lstm. in: 2022 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2022) google scholar  lorini, e.: rethinking epistemic logic with belief bases. artif. intell. 282, 103233 (2020) article  mathscinet  math  google scholar  lorini, e., jimenez, b.f.r.: decision procedures for epistemic logic exploiting belief bases. in: 18th international conference on autonomous agents and multiagent systems, aamas 2019, pp. 944–952 (2019) google scholar  malle, b.f.: the relation between language and theory of mind in development and evolution. in: the evolution of language out of pre-language, vol. 18, pp. 265–284 (2002) google scholar  marsella, s.c., pynadath, d.v.: modeling influence and theory of mind. in: virtual social agents, p. 199 (2005) google scholar  narang, s., best, a., manocha, d.: inferring user intent using bayesian theory of mind in shared avatar-agent virtual environments. ieee trans. vis. comput. graph. 25(5), 2113–2122 (2019) article  google scholar  nguyen, d., nguyen, p., le, h., do, k., venkatesh, s., tran, t.: learning theory of mind via dynamic traits attribution. arxiv preprint arxiv:2204.09047 (2022) page, m.j., et al.: the prisma 2020 statement: an updated guideline for reporting systematic reviews. bmj 372, n71 (2021) article  google scholar  panisson, a.r., et al.: on the formal semantics of theory of mind in agent communication. in: lujak, m. (ed.) at 2018. lncs (lnai), vol. 11327, pp. 18–32. springer, cham (2019). https://doi.org/10.1007/978-3-030-17294-7_2 chapter  google scholar  pantelis, p.c., et al.: agency and rationality: adopting the intentional stance toward evolved virtual agents. decision 3(1), 40 (2016) article  google scholar  persiani, m., hellström, t.: inference of the intentions of unknown agents in a theory of mind setting. in: dignum, f., corchado, j.m., de la prieta, f. (eds.) paams 2021. lncs (lnai), vol. 12946, pp. 188–200. springer, cham (2021). https://doi.org/10.1007/978-3-030-85739-4_16 chapter  google scholar  pöppel, j., kahl, s., kopp, s.: resonating minds-emergent collaboration through hierarchical active inference. cogn. comput. 14(2), 581–601 (2022) article  google scholar  premack, d., woodruff, g.: does the chimpanzee have a theory of mind? behav. brain sci. 1(4), 515–526 (1978) article  google scholar  pynadath, d.v., et al.: disaster world: decision-theoretic agents for simulating population responses to hurricanes. comput. math. organ. theor. 29(1), 84–117 (2022) article  google scholar  pynadath, d.v., marsella, s.c.: psychsim: modeling theory of mind with decision-theoretic agents. in: ijcai, vol. 5, pp. 1181–1186 (2005) google scholar  pynadath, d.v., marsella, s.c.: socio-cultural modeling through decision-theoretic agents with theory of mind. in: advances in design for cross-cultural activities, pp. 417–426 (2012) google scholar  pynadath, d.v., rosenbloom, p.s., marsella, s.c.: reinforcement learning for adaptive theory of mind in the sigma cognitive architecture. in: goertzel, b., orseau, l., snaider, j. (eds.) agi 2014. lncs (lnai), vol. 8598, pp. 143–154. springer, cham (2014). https://doi.org/10.1007/978-3-319-09274-4_14 chapter  google scholar  pynadath, d.v., si, m., marsella, s.c.: modeling theory of mind and cognitive appraisal with decision-theoretic agents. in: social emotions in nature and artifact: emotions in human and human-computer interaction, pp. 70–87 (2011) google scholar  reisenzein, r., et al.: computational modeling of emotion: toward improving the inter-and intradisciplinary exchange. ieee trans. affect. comput. 4(3), 246–266 (2013) article  google scholar  rumbell, t., barnden, j., denham, s., wennekers, t.: emotions in autonomous agents: comparative analysis of mechanisms and functions. auton. agent. multi-agent syst. 25, 1–45 (2012) article  google scholar  sarkadi, ş, panisson, a.r., bordini, r.h., mcburney, p., parsons, s.: towards an approach for modelling uncertain theory of mind in multi-agent systems. in: lujak, m. (ed.) at 2018. lncs (lnai), vol. 11327, pp. 3–17. springer, cham (2019). https://doi.org/10.1007/978-3-030-17294-7_1 chapter  google scholar  sarkadi, s., panisson, a.r., bordini, r.h., mcburney, p., parsons, s., chapman, m.: modelling deception using theory of mind in multi-agent systems. ai commun. 32(4), 287–302 (2019) article  mathscinet  math  google scholar  sarthou, g., clodic, a., alami, r.: ontologenius: a long-term semantic memory for robotic agents. in: 2019 28th ieee international conference on robot and human interactive communication (ro-man), pp. 1–8. ieee (2019) google scholar  scassellati, b.: theory of mind for a humanoid robot. auton. robot. 12, 13–24 (2002) article  math  google scholar  shum, m., kleiman-weiner, m., littman, m.l., tenenbaum, j.b.: theory of minds: understanding behavior in groups through inverse planning. in: proceedings of the aaai conference on artificial intelligence, vol. 33(1), pp. 6163–6170 (2019) google scholar  si, m., marsella, s.c., pynadath, d.v.: modeling appraisal in theory of mind reasoning. auton. agent. multi-agent syst. 20, 14–31 (2010) article  google scholar  stevens, c., de weerd, h., cnossen, f., taatgen, n.: a metacognitive agent for training negotiation skills. in: proceedings of the 14th international conference on cognitive modeling, iccm 2016 (2016) google scholar  tekülve, j., schöner, g.: neural dynamic concepts for intentional systems. in: cogsci, pp. 1090–1096 (2019) google scholar  veltman, k., de weerd, h., verbrugge, r.: training the use of theory of mind using artificial agents. j. multimodal user interfaces 13, 3–18 (2019) article  google scholar  von der osten, f.b., kirley, m., miller, t.: the minds of many: opponent modeling in a stochastic game. in: ijcai, pp. 3845–3851 (2017) google scholar  walton, d.: using argumentation schemes to find motives and intentions of a rational agent. argument computat. 10(3), 233–275 (2019) article  google scholar  wang, r.e., wu, s.a., evans, j.a., tenenbaum, j.b., parkes, d.c., kleiman-weiner, m.: too many cooks: coordinating multi-agent collaboration through inverse planning. in: proceedings of the 19th international conference on autonomous agents and multiagent systems, pp. 2032–2034 (2020) google scholar  ware, s.g., siler, c.: sabre: a narrative planner supporting intention and deep theory of mind. in: proceedings of the aaai conference on artificial intelligence and interactive digital entertainment, vol. 17, no. 1, pp. 99–106 (2021) google scholar  whiten, a., byrne, r.: natural theories of mind: evolution, development and simulation of everyday mindreading. b. blackwell oxford, uk (1991) google scholar  wooldridge, m.: an introduction to multiagent systems. wiley, hoboken (2009) google scholar  yousefi, z., heinke, d., apperly, i., siebers, p.-o.: an agent-based model for false belief tasks: belief representation systematic approach (brsa). in: thomson, r., dancy, c., hyder, a., bisgin, h. (eds.) sbp-brims 2018. lncs, vol. 10899, pp. 111–126. springer, cham (2018). https://doi.org/10.1007/978-3-319-93372-6_14 chapter  google scholar  download references author information authors and affiliations department of computing, ufsc, florianópolis, brazil michele rocha, heitor henrique da silva, analúcia schiaffino morales & alison r. panisson department of informatics, kcl, london, uk stefan sarkadi authors michele rochaview author publications search author on:pubmed google scholar heitor henrique da silvaview author publications search author on:pubmed google scholar analúcia schiaffino moralesview author publications search author on:pubmed google scholar stefan sarkadiview author publications search author on:pubmed google scholar alison r. panissonview author publications search author on:pubmed google scholar corresponding author correspondence to alison r. panisson . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper rocha, m., da silva, h.h., morales, a.s., sarkadi, s., panisson, a.r. (2023). applying theory of mind to multi-agent systems: a systematic review. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_24 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_24 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial intelligence multi-agent systems theory of mind systematic review publish with us policies and ethics profiles analúcia schiaffino morales view author profile stefan sarkadi view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature convolutional neural networks for the molecular detection of covid-19 | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper convolutional neural networks for the molecular detection of covid-19 conference paper first online: 12 october 2023 pp 51–62 cite this conference paper download book pdf download book epub intelligent systems (bracis 2023) convolutional neural networks for the molecular detection of covid-19 download book pdf download book epub anisio p. santos jr.  orcid: orcid.org/0000-0002-7823-38549, anage c. mundim filho  orcid: orcid.org/0000-0002-9334-10529, robinson sabino-silva  orcid: orcid.org/0000-0002-2104-578010 & … murillo g. carneiro  orcid: orcid.org/0000-0002-2915-89909  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 514 accesses 4 citations abstract the ongoing covid-19 pandemic caused an unprecedented overburning of healthcare systems and still represents a global health issue with the emergence of covid-19 variants. the relevance of mass testing for covid-19 in the find-test-trace-isolate-support strategy suggested by the world health organization (who) is imperative to reduce covid-19 transmission. although real-time polymerase chain reaction (rt-pcr) is considered a reference standard for covid-19 detection, it is an expensive, lengthened, and laborious process, and problems in rna extraction can reduce the sensitivity. in this context, the raman spectroscopy analysis in biofluids is a label-free method performing a suitable cost-benefit application for covid-19 detection. we propose a convolutional neural network (cnn) architecture that processes spectra images generated by the raman spectrum and returns the covid-19 diagnosis of the spectrum sample. the predictive performance of the cnn was compared against several other algorithms widely adopted in the literature. the cnn architecture discriminates covid-19 with raman spectroscopy of blood samples with 96.8% accuracy, 95.5% sensitivity, and 98.2% of specificity, representing the best results as well as a promising alternative to distinguish samples. moreover, we also present a model explanation analysis that contributes to clarifying the salient features taken into account by our cnn. download conference paper pdf similar content being viewed by others detection of covid-19-affected persons using convolutional neural network from x-rays’ images chapter © 2023 decision and feature level fusion of deep features extracted from public covid-19 data-sets article 30 october 2021 a nonclinical spectroscopic approach for diagnosing covid-19: a concise perspective article 12 september 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. molecular spectroscopy optical spectroscopy raman spectroscopy resonance raman spectroscopy spectroscopy surface-enhanced raman spectroscopy 1 introduction the covid-19 pandemic caused an unprecedented global effect on private and public healthcare systems and social and economic impacts. among the covid-19 containment measures is vaccination. however, the distribution of covid-19 vaccines continues to be applied with low vaccination coverage in underdeveloped countries, especially adapted vaccines to prevalent covid-19 variants. therefore, mass testing for covid-19 is still a strategic preventive measure to reduce its transmission. there is a need for new techniques developed to prevent it from mass propagating again and also to prevent possible pandemic scenarios. locations such as airports, schools, workplaces, and remote communities would benefit from faster and reagent-free screening techniques. vibrational spectroscopy is considered a promising method for biological analyses. it enables label-free extraction of biochemical information and images toward diagnosis and the assessment of the biochemical and molecular composition of biofluids [1]. vibrational spectroscopy techniques are bioanalytical approaches with great potential to discriminate between healthy and pathological conditions, like diabetes mellitus [4] and zika virus [15]. in addition, it allows the analysis of different bio-fluids, such as blood, urine, saliva, and tissues [2, 14, 18]. among several vibrational spectroscopy technologies, raman is known by its ability to determine molecular components of various types of materials quickly and without the need for the use of reagents. it uses inelastic light scattering based on a monochromatic laser source. the changes in the spectra represent parts of specific molecules. each raman shift is unique in each wavenumber frequency, representing a specific functional group [8]. the application of biofluids in raman spectroscopy for diagnostic or screening fields permits simple sample preparation without the insertion of reagents and thus can be considered a green technology due to a significant reduction in environmental waste [8]. furthermore, its usage has been investigated in the diagnosis of several diseases like breast, colon, lung, and prostate cancers [9]. in this paper, we are interested in the analysis of raman spectra obtained from blood serum samples for the detection of covid-19. the ongoing covid-19 pandemic caused an unprecedented overburning of healthcare systems and still represents a global health issue with the emergence of covid-19 variants. due to the relevance of the topic, there are already some related works in the literature. in [18] the authors presented a support vector machine (svm) model to detect covid-19 using raman spectrum of serum samples, and in [19] the authors extended the work of [18] by combining the svm with an extreme gradient boosting (xgb) technique. despite the raman spectroscopy data used in [18, 19] being the same considered in our research, our work goes a step further by investigating deep learning techniques, like convolutional neural networks (cnns). different from most raman spectra analysis, which adopts simple statistical analysis [6], and also of some machine learning initiatives with raman, which are focused on traditional classifiers [18, 19], we hypothesize that convolutional neural networks [2] can contribute in the detection of covid-19 from raman spectra of serum samples by efficiently learning robust representations of such a problem and also improve model explainability. despite some few works have already applied 1d cnn to classify breast cancer, minerals, and pancreatic cancer by raman spectroscopy, e.g., [11, 13, 16], the investigation of such architectures for covid-19 detection from raman spectra is yet a barely explored topic in the literature. therefore, the work most related to ours in the literature is the study presented in [5], which investigated traditional and deep learning techniques for the diagnosis of covid-19 using raman spectroscopy. in that work, the authors evaluated svm, xgb, random forest (rf), and cnn, which achieved good predictive performance over several groups of experiments. however, our work differs from this for some reasons, such as: we consider raman spectra of serum samples instead of saliva ones; we consider a publicly available raman data set instead of a private one, which benefits reproducibility aspects; and besides the predictive performance, we also are interested in the explainability analysis. the main obstacle to differentiate each class based on the raman spectrum from control and covid-19 samples is the similarity of both spectra. the complexity of the dataset is a critical feature to solve issues for discriminating samples of each class. truncation in regions with organic compounds can be performed at specific wavelength regions. the raman dataset analyses and interpretation require expertise due to the potential overlap of peaks corresponding to distinct chemical components. the raman spectra differentiation can be especially problematic in complex biofluids with thousands of molecular components, featuring variability which is an issue for discrimination. this spectral variability can impact the accuracy and discrimination associated with high-dimensional data, with a large number of wavelengths contributing to the spectrum. the management of high-dimensional data is a challenge for computational complexity, feature selection, and generalization of the classification models for spectral datasets. this paper aims to contribute to the development and evaluation of cnns for the detection of covid-19 using raman spectra of serum samples, in order to obtain more accurate models. to be specific, the study presents a one-dimensional (1d) cnn architecture to process raman spectra as well as an explainability analysis of its salient features. in summary, the main contributions of our work are listed as follows: development and evaluation of the proposed 1d cnn method for detection of covid-19 from raman spectra of serum samples; comparative evaluation of several traditional and deep learning techniques for the problem; explainability analysis regarding salient raman shifts to detect and discriminate covid-19 and healthy groups. the remainder of the paper is organized as follows. section 2 presents the raman spectroscopy data as well as the steps of data preparation and preprocessing. section 3 describes the 1d cnn proposed in this paper. section 4 shows the experimental results and explainability analysis regarding our cnn model. section 5 concludes the paper. 2 data set description and preparation the data adopted in this work was publicly available by [18]. blood samples were collected between february 10th and may 10th, 2020. blood samples were taken from covid-19 patients and suspected cases upon admission between february 10 and may 10, 2020. the serum was isolated by centrifuging at 3000 rpm for 10 min after a one-hour rest. the serum samples were stored at 4\(^\circ \)c and measured within 36 h. for measurement, approximately 0.5 ml of serum sample was prepared in cryopreservation tubes and strictly sealed for raman scan. additional spectral data were collected from cryopreservation tubes with saline solution [18]. the data were obtained from blood serum samples in a total of 309 spectra, of which 159 received a positive diagnosis for covid-19 (covid-19 group) and 150 received a negative test result for covid-19 (healthy group). the serum samples were processed by a raman spectrometer. for each sample, the equipment is responsible for generating a corresponding spectrum. figure 1 shows the average spectrum of the healthy group (a), the average spectrum of the covid-19 group (b), the average spectrum of both groups (c) and all spectra from both groups (d). one can see some interesting regions in figs. 1(c) and 1(d), in which samples from different groups seem better separated. fig. 1. (a) average of healthy group spectra, (b) average of covid-19 group spectra, (c) average of healthy and covid-19 spectra, (d) total covid-19 and healthy spectral plot full size image regarding data preparation and preprocessing, two steps were conducted in this study: savitzky-golay filtering and vector normalization: the savitzky-golay (sg) is a filter for smoothing and differentiation that optimally fits a set of data points to a polynomial in the least-squares sense [17]. this process is important to reduce noise and smooth the signal while maintaining higher-order moments of the original spectrum. sg has been used to normalize and preprocess spectral data for ftir, raman, and other spectroscopy equipments due to the inherent presence of noises during sample collection, manipulation, and so on. vector normalization is the process in which the intensity values of all spectra are normalized by the euclidean norm. 3 model description this section describes the one-dimensional cnn proposed in this study for the detection of covid-19 using raman spectra of serum samples. the proposed architecture is illustrated by fig. 2. there are three main types of layers: convolutional, pooling, and fully connected. in the following, each one of them is briefly described. fig. 2. the architecture of the one-dimensional cnn proposed. full size image convolutional layer. the convolutional layer aims to learn representative features of the input data. the convolution layer is made up of several convolution kernels, also known as filters, which are used to compute different feature maps. specifically, each neuron in a feature map is connected to a region of neighboring neurons in the previous layer. such a neighborhood is referred to as the receptive field of the neuron. mathematically, the feature value at location (i, j) referring to the kth feature map of the lth layer, \(z^l_{i,j,k}\) is calculated as: $$\begin{aligned} z^l_i = \sigma \left( w^{l^t}_{k} x^l_{i,j} + b^l_k \right) ~, \end{aligned}$$ (1) in which \(x^l_{i,j}\) is the input centered at position (i, j) referring to the l-th layer, \(w^{l^t}_{k}\) and \(b^l_k\) are respectively the weight vector and the bias term of the k-th filter of the l-th layer, and \(\sigma \) is the activation function which in cnn architecture is usually relu (rectified linear units): $$\begin{aligned} relu(x) = max\{0, x\}~. \end{aligned}$$ (2) to compute a feature map, the process can be divided into two steps: first, the input is convoluted with an already learned filter, and then a non-linear activation function is applied to the convolution results. this non-linearity is not an integral part of the convolutional layer, but in practice, almost all convolutional layers are followed by a non-linearity. these non-linearities are desirable, as they allow the detection of non-linear features [7]. pooling layer. the pooling layer aims to achieve displacement invariance, reducing the spatial resolution of feature maps. this spatial reduction is achieved through a non-unitary stride. in this way, only one activation at each n (stride size) position of the input is calculated in each spatial dimension. each feature map of a pooling layer is connected to the corresponding feature map of the previous convolutional layer. the output generated by the pooling function for the feature map \(z^l_{i,j,k}\) is represented by the expression: $$\begin{aligned} p_{m,k} = \underset{n=1}{\overset{n}{\max }} \left( z^{l}_{i-1} \times q + n \right) ~, \end{aligned}$$ (3) in which n is the cluster size; and q is the slide size, which determines the degree to which the adjacent pool windows overlap. fully connected layer and dropout. fully connected layers are usually located at the end of the network, as shown by our proposed architecture in fig. 2. in these layers, the features extracted in the previous convolution layers are used to obtain the network classification output. a fully connected layer takes all the neurons in the previous layer and connects them to every neuron in the current layer. it also establishes which high-level attributes most closely relate to the object’s class. dropout is a technique usually adopted in this layer, in which some neurons are randomly turned off, along with their connections, during training only. during test, all neurons are kept active. the reason for doing this is to avoid overfitting in training. in our architecture, we adopted a dropout rate of 20% between the dense layers. finally, after the fully connected layers and the dropouts come to the output layer. for the classification problem, it is common to use as many neurons as there are classes to predict and the output has a sigmoid activation function that can show the result of the classification. the sigmoid is represented in the following equation: $$\begin{aligned} \sigma (x) = \dfrac{1}{1+e^{-x}}~. \end{aligned}$$ (4) this function only returns values between 0 and 1. it is responsible to classify the output and is located in the final step of our model shown in fig. 2. the cnn architecture shown by fig. 2 receives a raman spectrum as input. it then extracts features of the spectrum through three one-dimensional convolutional layers. at the end of each convolutional layer, relu is adopted as the activation function. between each convolutional layer, there is a maxpooling layer. by stacking multiple convolutional and pooling layers, one can gradually extract features with a higher (complex) level of abstraction. after extracting the features of the input, the flattening step transforms the arrays into an one-dimensional vector to start the classification method. the dense layers, which are also composed by dropout functions, learn the weights to accurately adjust the classification of each group, and the sigmoid function is used to return the prediction. 4 experimental results in this section, we present the experiments and results obtained by our cnn in comparison with other techniques in the problem of covid-19 detection from raman spectra. in addition, we also present an explainability analysis regarding the salient features considered by cnn in its classification using shapley additive explanations (shap) [3]. the experiments were conducted using a k-fold cross-validation process. this method partition the dataset samples in k disjoint subsets, with k-1 adopted as a training set and the remaining one as a test set. a total of k executions are performed, which means each subset is adopted as the test set once. the predictive performance of the techniques is then obtained over k executions in which performance metrics like accuracy, sensitivity, and specificity are calculated. in this study, we considered 5-fold cross-validation. we compared our cnn technique against widely known supervised learning techniques like naive bayes (nb), random forest (rf), extreme gradient boosting (xgb), multi-layer perceptron (mlp), and support vector machine (svm) as well as against other deep learning techniques like fully convolutional network (fcn), multi-channel deep convolutional neural network (mcdcnn) and residual networks (resnet). all models were trained using the parameters recommended in [10, 19]. regarding the computational simulations, the experiments were done using python on two machines: a laptop with core i7 9th gen processor, 32 gb of ram, and geforce gtx 1660ti gpu, and a desktop with ryzen 9 processor, 32 gb of ram, and titan v gpu. 4.1 predictive results for covid-19 detection table 1 shows the predictive results of the techniques under analysis in terms of accuracy, sensitivity, and specificity. as sensitivity and specificity covers different groups, in order to support our analysis we also evaluate the arithmetic and harmonic mean between both metrics, respectively named mean(se, sp) and f1(se, sp). the best result obtained by each metric is boldfaced in the table. among traditional techniques, rf, svm and mlp achieved the best results. indeed, svm and mlp have respectively the highest sensitivity and specificity in the table. however, the best overall results were achieved by the cnn which classified samples correctly more than any other technique. on the other hand, worse results were obtained by nb and fcn. while nb is the unique linear technique under analysis, fcn is a deep learning technique essentially based on the convolutional layer, which means it does not have fully connected layers to help in the learning of complex mapping functions. table 1. predictive performance of the techniques under analysis for the covid-19 detection from raman spectra.full size table fig. 3. analysis of the predictive performance of the techniques under comparison over the 5-fold cross-validation. full size image now we analyze each one of the executions of the 5-fold cross-validation in order to better understand the performance of the techniques under analysis. figure 3 shows the predictive performance of each model in terms of accuracy, specificity, sensitivity, and mean(se, sp). one can see fcn really had troubles in some executions. one can also see that resnet, a state-of-the-art deep learning technique, had its performance oscillating among the executions. for example, in fig. 3 resnet achieved the worst (execution 1) and the best predictive result (execution 3), indicating that the model may be overfitted in some scenarios, which will be subject of our further investigations. regarding the cnn performance, it was very stable in terms of accuracy, specificity, and mean(se, sp), indicating that such a technique can contribute in the detection of covid-19 from serum raman spectroscopy samples. in order to better understand the salient features that contributed to the almost 97% of accuracy, we next investigate the explainability of the cnn model. fig. 4. explainability analysis of the cnn model considering the 5-fold cross-validation for covid-19 detection. full size image 4.2 explainability analysis the shap method is an approach to explain the output of machine learning models. it connects optimal credit allocation with local explanations using the classic shapley values from game theory and their related extensions [12]. with that in mind, to understand a bit more about how the cnn model created its inductive mechanism for covid-19 detection, some shap analysis were conducted in order to provide insights regarding the cnn most relevant features. figure 4 shows the shap features of the main raman shift capable to discriminate both groups of samples. the main raman shifts between 1605–1607 cm\(^{-1}\) can be attributed to nucleoproteins. the raman shifts between 1448–1450 cm\(^{-1}\) and 1271–1273 cm\(^{-1}\) can be attributed to (ch2/ch3) from lipids and the proline from collagen, respectively. besides, the raman shift at 831 cm\(^{-1}\) can be attributed to the amino acid tyrosine. furthermore, the raman shifts between 427–432 cm\(^{-1}\) and 402-410 cm\(^{-1}\) were attributed to phosphate and the heme deformation of proteins. lastly, the raman shift at 1657 cm\(^{-1}\) can be attributed to amide i in proteins [19] . in general, the main raman shifts used to discriminate control from covid-19 samples were related to proteins, indicating changes in the proteomic profile in the serum of covid-19-infected subjects. 5 conclusions this paper investigated deep learning solutions based on cnns for the detection of covid-19 using raman spectra of serum samples in order to obtain more accurate models. to be specific, an 1d cnn was proposed and comparatively evaluated against several traditional and deep learning techniques. experimental results showed that the cnn model achieved the highest predictive performance in comparison with other techniques, with 96.8% accuracy, 95.5% sensitivity, and 98.2% of specificity, outperforming even other robust deep learning solutions like resnet. in addition, explainability analysis regarding salient raman shifts to detect covid-19 and healthy groups were also conducted, revealing some proteins which probably are associated with the spectral differences between the groups. the investigation conducted here contributes to the raman spectroscopy field and may also impact the society in the future by providing regular, and financially accessible diagnosis for the population. the fast and accurate results achieved by the cnn method proposed can make the system suitable for use in screening and routine examinations, for example, contributing to the promotion and regular testing of the population. forthcoming works will focuses on the development of even better deep learning architectures as well as in addressing other applications based on spectroscopy technologies. references baker, m.j., et al.: using fourier transform ir spectroscopy to analyze biological materials. nat. protoc. 9(8), 1771–1791 (2014) article  google scholar  barauna, v.g., et al.: ultrarapid on-site detection of sars-cov-2 infection using simple atr-ftir spectroscopy and an analysis algorithm: high sensitivity and specificity. anal. chem. 93(5), 2950–2958 (2021) article  google scholar  van den broeck, g., lykov, a., schleich, m., suciu, d.: on the tractability of shap explanations. j. artif. intell. res. 74, 851–886 (2022) article  mathscinet  math  google scholar  caixeta, d.c., et al.: salivary atr-ftir spectroscopy coupled with support vector machine classification for screening of type 2 diabetes mellitus. diagnostics 13(8), 1396 (2023) article  google scholar  carlomagno, c., et al.: covid-19 salivary raman fingerprint: innovative approach for the detection of current and past sars-cov-2 infections. sci. rep. 11(1), 4943 (2021) article  mathscinet  google scholar  desai, s., et al.: raman spectroscopy-based detection of rna viruses in saliva: a preliminary report. j. biophotonics 13(10), e202000189 (2020) article  google scholar  géron, a.: hands-on machine learning with scikit-learn, keras, and tensorflow. “o’reilly media, inc” (2022) google scholar  giamougiannis, p., et al.: a comparative analysis of different biofluids towards ovarian cancer diagnosis using raman microspectroscopy. anal. bioanal. chem. 413, 911–922 (2021) article  google scholar  hanna, k., krzoska, e., shaaban, a.m., muirhead, d., abu-eid, r., speirs, v.: raman spectroscopy: current applications in breast cancer diagnosis, challenges and future prospects. br. j. cancer 126(8), 1125–1139 (2022) article  google scholar  ismail fawaz, h., forestier, g., weber, j., idoumghar, l., muller, p.a.: deep learning for time series classification: a review. data min. knowl. disc. 33(4), 917–963 (2019) article  mathscinet  math  google scholar  li, z., et al.: detection of pancreatic cancer by convolutional-neural-network-assisted spontaneous raman spectroscopy with critical feature visualization. neural netw. 144, 455–464 (2021) article  google scholar  lundberg, s.m., lee, s.i.: a unified approach to interpreting model predictions. in: guyon, i., luxburg, u.v., bengio, s., wallach, h., fergus, r., vishwanathan, s., garnett, r. (eds.) advances in neural information processing systems 30, pp. 4765–4774. curran associates, inc. (2017) google scholar  ma, d., shang, l., tang, j., bao, y., fu, j., yin, j.: classifying breast cancer tissue by raman spectroscopy with one-dimensional convolutional neural network. spectrochim. acta part a mol. biomol. spectrosc. 256, 119732 (2021) article  google scholar  naseer, k., ali, s., qazi, j.: atr-ftir spectroscopy as the future of diagnostics: a systematic review of the approach using bio-fluids. appl. spectrosc. rev. 56(2), 85–97 (2021) article  google scholar  oliveira, s.w., et al.: salivary detection of zika virus infection using atr-ftir spectroscopy coupled with machine learning algorithms and univariate analysis: a proof-of-concept animal study. diagnostics 13(8), 1443 (2023) article  google scholar  sang, x., zhou, r.g., li, y., xiong, s.: one-dimensional deep convolutional neural network for mineral classification from raman spectroscopy. neural processing letters, pp. 1–14 (2022) google scholar  savitzky, a., golay, m.j.: smoothing and differentiation of data by simplified least squares procedures. anal. chem. 36(8), 1627–1639 (1964) article  google scholar  yin, g., et al.: an efficient primary screening of covid-19 by serum raman spectroscopy. j. raman spectrosc. 52(5), 949–958 (2021) article  google scholar  zeng, w., wang, q., xia, z., li, z., qu, h.: application of xgboost algorithm in the detection of sars-cov-2 using raman spectroscopy. j. phys. conf. seri. 1775, 012007. iop publishing (2021) google scholar  download references acknowledgment authors thank the financial support given by google (through the 2020 and 2021 google latin america research awards), minas gerais research foundation fapemig (grants number apq-00410-21), brazilian national council for scientific and technological development cnpq (grants number 402196/2021-0 and 408216/2022-0), and national institute of science and technology in theranostics and nanobiotechnology inct-teranano (grant number cnpq-465669/2014-0). rs-s also thanks the cnpq for the productivity fellowship. we also thank nvidia corporation by the donation of a titan v gpu used in this research. author information authors and affiliations faculty of computing, federal university of uberlandia, uberlandia, mg, brazil anisio p. santos jr., anage c. mundim filho & murillo g. carneiro department of physiology, institute of biomedical sciences, federal university of uberlandia, uberlandia, mg, brazil robinson sabino-silva authors anisio p. santos jr.view author publications search author on:pubmed google scholar anage c. mundim filhoview author publications search author on:pubmed google scholar robinson sabino-silvaview author publications search author on:pubmed google scholar murillo g. carneiroview author publications search author on:pubmed google scholar corresponding author correspondence to murillo g. carneiro . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper santos, a.p., filho, a.c.m., sabino-silva, r., carneiro, m.g. (2023). convolutional neural networks for the molecular detection of covid-19. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_4 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords covid-19 raman spectroscopy convolutional neural networks deep learning publish with us policies and ethics profiles murillo g. carneiro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature likelihood estimator for multi model-based reinforcement learning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper likelihood estimator for multi model-based reinforcement learning conference paper first online: 30 january 2025 pp 184–198 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) likelihood estimator for multi model-based reinforcement learning download book pdf download book epub guilherme albarrans9 & valdinei freire9  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 361 accesses abstract efficiently learning and generalizing from interactions with the environment is a concern in reinforcement learning. model-based approaches offer promising potential to address these challenges, leveraging the underlying structure of the world to speed learning and improve generalization. by assuming a structured environment, such approaches aim to capture key patterns from limited interactions and synthesize experiences for enhanced exploitation and generalization to unexplored territories. in this paper, we test a model-based reinforcement learning framework aimed at further advancing the frontiers of efficiency and generalization. motivated by the insight that environments often exhibit distinct regions with varying dynamics, we introduce additional assumptions about the structure of the world to facilitate faster generalization. specifically, we construct a model comprising a classifier that, given a state input, selects parameters characterizing the operating regime of the environment. both the model and the regional parameters are learned from past experiences using a likelihood function. leveraging this model, we employ a rapid exploring random tree based planner to generate new real experiences, capitalizing on the identified structural nuances within the environment. to evaluate the efficacy learning a segmented model, we conduct a comparative analysis against a traditional method that employs a single general model to learn the dynamics of the entire environment. our results demonstrate the superiority of the segmented approach in terms of both efficiency and generalization, underscoring the benefits of incorporating additional assumptions about the structure of the world into model-based reinforcement learning paradigms. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others exploring ensemble error exploration for unsupervised reinforcement learning chapter © 2025 a survey on model-based reinforcement learning article 23 january 2024 learning probabilistic models for model checking: an evolutionary approach and an empirical study article 25 april 2018 1 introduction model-based reinforcement learning (rl) has a focus on efficiency and generalization through model construction. by leveraging these models, the agent can simulate possible outcomes of actions without directly interacting with the environment, or explore a state [11]. this can potentially reduce the number of interactions needed to learn an effective policy. however, model-based methods may suffer from errors or inaccuracies in the learned model, leading to sub-optimal performance [2]. the concerns of model-based rl lies in the balance between the accuracy of the learned model and the performance achieved by utilizing it to derive a policy. by incorporating learned models into the decision-making process, these methods can adapt more readily to new situations and exploit previously acquired knowledge to achieve better performance. as demonstrated by young et al. [14] the benefits of the model-based approach could go even further at the point of presenting better solutions for some problems, as in cases where the model’s error account for an unintended exploration that can have a high return payoff. we intend to delve into the nuanced advantages of model-based solutions, particularly in environments characterized by distinct regions with varying dynamics. consider a scenario where a robot operates in an environment comprising two areas with markedly different frictional properties on their floors. in one area, the effects of actions accounting for movement differ significantly from those in the other. this paper aim to explore how assumptions like the existence of distinct regions in model-based solutions can enhance its capabilities of planning. constructing multiple models as done by doya et al. [3] tailored to individual regions exhibiting stable dynamics, rather than attempting to capture the complex dynamics of both regions within a single model could yield better results. we propose an approach wherein distinct models are built for each identifiable region, and selected for use based on agent past experience. the learned models are leveraged for plan generation through the utilization of a rapidly exploring random tree (rrt) based algorithm. our planning strategy draws inspiration from the work of lin and zhang [9], which introduced the integration of rrt with reinforcement learning for planning tasks. by adopting this multi-model approach, we aim to investigate its efficacy in accurately representing the environment’s dynamics and its impact on the agent’s decision-making process. we hypothesize that by aligning models with specific regions, the agent can better understand and adapt to the unique characteristics of each area, ultimately leading to improved performance and efficiency in task execution. through empirical evaluation and analysis, we aim to provide insights into the comparative advantages and limitations of both traditional single-model and proposed segmented model approaches in addressing challenges posed by environments with heterogeneous dynamics. our findings aim to contribute to a deeper understanding of the applicability of model-based solutions in real-world scenarios characterized by spatially varying dynamics. in sect. 2 of our paper, we delve into related works that have explored the concept of leveraging multiple models to enhance generalization in rl. section 3 formalizes the segmented environment problem in terms of a markov decision problem. section 4 introduces our approach based on segmented models problem that forms the basis of our comparative analysis. moving on to sect. 5, we provide a concise description of the simple environment utilized for collecting empirical results. finally, in sect. 6, we present the results of our empirical evaluations, detailing the performance metrics and outcomes observed across different experimental conditions. 2 related works several existing works have explored the concept of constructing a segmented model of the environment’s dynamics, aiming to create a comprehensive understanding of the environment. one notable example is the work by doya et al. [3], which introduces a concept akin to a mixture of specialists. in this approach, the environment’s dynamics are modeled by a collection of smaller models, each specialized for specific states or moments where distinct behaviors are observed. the fundamental difference between observed approach [3] and our methodology lies in the mechanism for combining the outputs of these specialized models. in multiple model based reinforcemnt learning [3], the specialists contribute to the overall model through a continuous responsibility signal, allowing for a potentially blended output that may not be directly attributable to any individual specialist. in contrast, our approach adopts a structured mechanism where the output is determined by the specialist that maximizes its signal. this ensures that the output is directly linked to the most relevant specialist, facilitating clearer interpretation and utilization of the model’s predictions. an approach was proposed [5], which introduces a novel representation of the problem by augmenting the traditional markov decision process (mdp) with contextual information. this augmentation enables the inference of different underlying models based on the context provided, offering a flexible framework for adapting to diverse environmental dynamics. one notable aspect of this work is the similarity in notation used for representing context, which bears resemblance to the representation of our models. however, a significant distinction lies in the treatment of context over the course of a trajectory. unlike our approach, which allows for the possibility of encountering multiple contexts within the same episode, the contextual mdp framework [5] assumes a fixed context for the entire trajectory. the idea of dynamically identifying the context have been addressed [12] with a solution capable of learning a new local model whenever it detects a context not yet experienced. also using as a basis the prediction error of models already learned. this approach resulted in solutions that proved to be very suitable for dealing with non-stationary environments, as detailed by alegre, bazzan e silva (2021) [1] who explored the power of rapid adaptation by identifying change points with high confidence. li et al. [7, 8], have delved into hierarchical approaches, where segments of information are compounded in a hierarchical manner. this hierarchical structuring enables more efficient learning and decision-making processes, particularly in complex environments with multiple levels of abstraction. there are works [10] that have focused on incremental learning techniques, which have shown promising results in adapting to changing environments and evolving tasks. by incrementally updating models based on new data and experiences, these approaches offer flexibility and scalability in learning complex tasks over time. offline learning settings, [6] have demonstrated the effectiveness of model-based approaches, showcasing their utility in offline learning scenarios where access to real-time data may be limited or impractical. leveraging offline data, these approaches enable efficient learning and decision-making based on previously collected experiences. the utilization of a likelihood estimator in the context of building options [4], inspired our exploration into employing a similar approach for constructing models tailored to different regions within the world. the likelihood estimator framework proved instrumental in our efforts to construct region-specific models, enhancing the efficiency and effectiveness of our reinforcement learning approach in complex and heterogeneous environments as it was used in the options context. 3 k-models mdp the markov decision process (mdp) [13] framework serves as the foundational formalism for representing the core aspects of reinforcement learning. an mdp is typically characterized by the tuple \(<\mathcal {s}, \mathcal {a}, \mathcal {p}, \mathcal {r}>\), where \(\mathcal {s}\) denotes the set of states, \(\mathcal {a}\) represents the set of actions, \(\mathcal {p}\) defines the transition probabilities between states given actions, and \(\mathcal {r}\) denotes the reward function associated with each state. if we assume a know \(\mathcal {p}\) parameterized function a natural implementation for a model based approach would be to find the parameters that fit a observed experience, and if we assume that the dynamics of the world vary spatially, meaning that different regions exhibit distinct behaviors, we can define a function \(\beta : \mathcal {s} \mapsto \theta \) that give us the parameter for the model based on a given state. we are going to extend this concept now based in a assumption that the world has k regions where the parameters can be different in the context of modeling segmented dynamics of the world, we extend the mdp framework to \(<\mathcal {s}, \mathcal {a}, \mathcal {p}, \mathcal {r}, \vartheta , \beta , \mathcal {g}, \mathcal {s}_0>\) where the transition function \(\mathcal {p}\) is parameterized by \(\theta \), so a segment of the world is represented by a \(\theta \) that dictates the dynamics applied by \(\mathcal {p}\) on that region of the state space, and to define that region we introduce a new function \(\beta : \mathcal {s} \times \theta \mapsto [0,1]\), which assigns a probability of the parameters \(\theta \) based on the current state. so given a state s and a parameter \(\theta \) the \(\beta (s, \theta )\) function yields the probabilities for parameter \(\theta \) to be the one that parameterize \(\mathcal {p}\) on s. the set \(\vartheta \) represents the union of all possible parameters evolved in functions \(\mathcal {p}\) that maps a region in the environment, in a way that for a world with k distinct regions \(\vartheta = \{\theta _1, \theta _2..., \theta _k\}\). each \(\theta _i\) represents a distinct parameter for \(\mathcal {p}\) that captures the dynamics of the environment within a specific region. we define \(\mathcal {g} \subset \mathcal {s}\) as the set of goal states used by the planning module to construct plans to reach them, and \(\mathcal {s}_0 \subset \mathcal {s}\) as the set of initial states used to randomly define the initial state of each episode. 4 likelihood solution for a k-models mdp the model-based approach to solving mdps involves learning a model that describes the transitions between states in the environment. this model is acquired by observing real transitions that occur as the agent interacts with the environment. over time, as more experience is gathered, the model can be refined and adjusted to more accurately reflect the dynamics of the environment. the process of learning the model typically involves estimating transition probabilities or dynamics from observed state-action-state transitions. we bring a likelihood approach to find the best parameters of the model based on the observed experience. as the agent interacts with the environment and accumulates more experience, we can update the parameters so the learned model becomes increasingly accurate and reliable. by learning and utilizing this model, the agent can plan and make decisions more effectively, for solving the reinforcement learning problem at hand. to facilitate performance comparison, we have implemented a separate planner module capable of generating plan sequences to reach a goal state given a model of the world. this planner module can be utilized with various implementations, including the k-models approach, a standard single-model approach, or even with real-world models. by integrating these components, we create a comprehensive framework for modeling segmented dynamics, learning multiple models, and conducting performance evaluations across different approaches. this approach enables us to explore the effectiveness of our proposed method in improving sample efficiency and generalization in reinforcement learning tasks. fig. 1. modules that compose a k-model’s agent that interacts with the environment full size image figure 1 shows a x-ray of the proposed framework that brings k parameters that are updated with likelihood estimation based on experienced transitions. we can see above the dashed line the components that are activated every time step, like the memory that every interaction receives new data to be stored and the current executing plan that is driving the decisions the agent is taking every step. below the dashed line are the components that are not activated every time step like the planner, that uses the models to generate a new plan, and the estimator that relies on the memory containing new observed transitions together with past experiences, to update the parameters of the models. 4.1 likelihood estimator drawing inspiration from the likelihood methodology used to find deep options [4], we devise an estimation approach for all parameters \(\vartheta \) within our framework. for the options it was utilized the likelihood in a gradient-based expectation-maximization algorithm to identify the optimal parameterized options from observed trajectories. leveraging this insight, we develop a likelihood function tailored to our parameters \(\vartheta \). our likelihood function captures the probability of observing the transitions in the environment given the current parameters \(\vartheta \). by maximizing this likelihood function, we aim to iteratively update and refine our parameter estimates to better align with observed data. to achieve this, we employ a stochastic gradient ascent algorithm, which iteratively adjusts the parameters \(\vartheta \) based on online experience. by maximizing the log likelihood of the observed data, we can effectively update our parameter estimates in a principled manner, leveraging the information gleaned from interactions with the environment. this approach allows us to adaptively learn and refine the parameters \(\vartheta \) over time, thereby improving the accuracy and effectiveness of our multi model-based reinforcement learning system. by continuously updating our parameter estimates based on observed data, we can enhance the performance and efficiency of our algorithm in learning the dynamics of segmented environments and generating optimal plans for achieving desired goals. the likelihood function for a trajectory \(\xi = {s_1, a_1, s_2, a_2, ..., s_t}\) in a standard mdp with a model parameterized by \(\vartheta \) is calculated using the transition probabilities as follows: $$\begin{aligned} \mathcal{{l}}[\xi | \vartheta ] = \sum _{t=1}^{t-1}{ \log \; \mathcal {p}_{\vartheta }(s_{t+1}|s_t, a_t)} \end{aligned}$$ (1) to adapt this likelihood to the context of k models, we incorporate the \(\beta \) function and account for all \(\mathcal {p}_{\theta }\) as follows: $$\begin{aligned} \mathcal {p}_{\vartheta }(s_{t+1}|s_t, a_t) = \sum _{\theta \in \vartheta } \mathcal {p}_{\theta }(s_{t+1}| s_t, a_t) \beta _{\theta }(s_t) \end{aligned}$$ (2) we can use this equation in the traditional likelihood equation. by iteratively applying stochastic gradient descent on the negative log likelihood, we aim to optimize the parameters \(\theta \): $$\begin{aligned} \vartheta = \vartheta + \eta \nabla \mathcal{{l}}[\xi |\vartheta ] \alpha \vardelta \vartheta \end{aligned}$$ (3) where \(\eta \) represents the learning rate and \(\alpha \) is the decay factor for momentum. this iterative process allows us to update the parameters \(\vartheta \) in a principled manner, leveraging the observed trajectory \(\xi \) to refine our model and improve its accuracy in capturing the dynamics of the environment. 4.2 planner with the established models we have chosen a planning approach for obtaining value of the representation of the world, inspired by the implementation rrt based algorithms with rl [9] for optimizing policies to find a path to the goal state, we decides to implement our modified version of the rrt algorithm that makes use of a model to come with a plan for reaching the goal. the implementation builds a tree of state transitions based on the execution of actions on a model passed as input, so the model can generate synthetic transitions that are incorporated to the tree plan, when completed the tree starts from a start state, and have a depth of a certain max number of steps that we wish the plan to have. the algorithm 1 is responsible for building this plan tree. algorithm 1 . rrt based planner full size image once the plan tree is constructed, we can extract a path to the goal state or to the state within the tree that is closest to the goal. to accomplish this, we only require a function to calculate the relative distance between states. algorithm 2 provides a systematic method for extracting a path from the input tree to either the goal state or the closest state found using the tree.nearest function. algorithm 2 . get path full size image the tree.nearest function plays a crucial role in this process by identifying the closest state to the goal within the plan tree. in this case it is used the euclidean distance to compare the states distances. once this closest state is determined, the algorithm proceeds to construct a path from the initial state to either the goal state or the identified closest state. by leveraging algorithm 2, the agent can effectively obtain a plan of actions to take. the quality of plans generated by the algorithm is directly influenced by the accuracy and fidelity of the underlying models used to construct them. initially, when the models are far from reflecting the real transitions in the environment, the generated plans may be misguided or sub optimal. however, as the agent gathers more experience and refines its models to better reflect the true dynamics of the environment, the quality of the generated plans improves significantly. this highlights the iterative nature of model-based reinforcement learning, and works [14] have even shown that this initial misguidance of poor models could bring valuable exploration experiences in some specific problems. 5 applied environment our challenge lies in simultaneously learning the k parameters \(\theta \) and the probability distribution given by \(\beta \) across states. to address this challenge, we assume that the function \(\mathcal {p}\) is based in a gaussian parameterized, with \(\theta _i = \{\tau _i, \sigma _i\}\) representing the parameters that will be used in the transitions. to create a simulation environment reflecting practical problems characterized by segmented dynamics, we developed a two-dimensional space where states are represented by x and y coordinates specifying \(\mathcal {s} \in \mathbb {r}^2\). in this environment, agents have four possible actions corresponding to movement in the cardinal directions then giving us \(\mathcal {a} \in \{\uparrow , \leftarrow , \downarrow , \rightarrow \} = \{(0,-1), (-1, 0), (0, 1), (1,0)\}\). the uncertainty of the agents’ movements, are formulated in transitions as linear transformations of the state with respect to the chosen action with a random factor drawn from a gaussian distribution. notably, the parameters of this distribution vary across distinct regions of the state space. so if we have k regions resulting state \(s'\) after executing action a in state s would be \(s' \sim \mathcal {n}(s+a\tau _i, \sigma _i)\) with \(\tau _i\) and \(\sigma _i\) being parameters \(\theta _i\) of region \(m_i\) where \(m_i \subset \mathcal {s}; s \in m_i\) and \(s' \in \mathcal {s}\); \(a \in \mathcal {a}\); this give us a transition function that is derived from the probability density function dependent of the region i: $$\begin{aligned} \mathcal {p}_{\theta }(s'| s, a) = \frac{1}{\sigma \sqrt{2\pi }}e^{-\frac{1}{2}(\frac{s'-(s+a\tau )}{\sigma })^2} \end{aligned}$$ (4) the probability defined by eq. 4 will be used to determine the likelihood function,as elaborated in sect. 4. an illustration of such an environment can be observed in fig. 2, providing a visual representation of the two-dimensional space with segmented dynamics. this figure effectively encapsulates the concept of distinct regions within the state space, each characterized by unique parameters of the gaussian distribution governing uncertainty in transitions. fig. 2. a representation of a instance of environment,with the goal states represented by the square with g, and a set of initial states represented by the white rectangle, and the two regions of different dynamics represented with the colors yellow and blue (color figure online) full size image by implementing this approach, we effectively model environments where dynamics vary between different regions. adjusting the parameters of the gaussian distribution enables us to replicate diverse levels of uncertainty and variability within each region. consequently, agents must adapt their strategies accordingly as they navigate through the environment. this simulation framework offers a versatile platform for investigating and evaluating model-based reinforcement learning algorithms in scenarios where the state space exhibits heterogeneous dynamics. through experimentation and analysis within this environment, researchers gain valuable insights into the effectiveness and adaptability of different learning approaches in complex real-world settings. 5.1 functions aproximators in our approach, we leverage the likelihood estimators to optimize both the parameters of the \(\beta \) function and the k-models, which can also be represented by parameterized functions. for simplicity and flexibility, in our experiments, we employed a feedforward neural network with an softmax activation function in the output layer to represent the \(\beta \) functions. additionally, we assumed a known distribution function for the k-models, specifically a gaussian distribution. the parameters of the neural network, along with the means and variances of the gaussian distributions representing the k-models, collectively form the parameter set \(\vartheta \) that we aim to estimate using the likelihood-based optimization method described previously. replacing the structural assumptions for the k-model’s functions with a universal function approximation, similar to what we’ve done for the \(\beta \) function, is a natural step towards generalizing this solution for more practical problems, and for that, no changes in the likelihood estimator is necessary, since it’s agnostic to the functions structure, optimizing the parameters directly. fig. 3. a representation of an episode, showing the steps where the model is updated and a new plan is generated full size image the dynamics for updating the models based on new experience and the generation of a new plan base on the new model can be visualized on fig. 3 that show how an episode starts on \(s_0\) and a plan is generated to start guiding the agent’s initial actions, and after a certain number of steps, the model can be updated based on the new gained information, which enable the agent to come with a new plan, and this goes on, until the agent reaches the goal. there are two aspects that can be adjusted in our approach that can change the impact of the learning of the models. the first one is the memory size, since we use real past experience to update the parameters via likelihood, we have decided to specify the size of the memory, in other words, how many experiences the agent will remember, and use to recalculate the likelihood. as the size increases, the better the updates reflects the reality, but given a infinity horizon problem, this could became a problem. the second relevant aspect is the frequency in which the models are updated. we control that frequency by specifying the number of steps that the agent will be forced to exploit the models after each update. so every fixed number of steps, the experience collected are added to the memory and the parameters can be recalculated afterwards with the extended trajectory. with all that put together we have a solution that start roaming for some steps and then uses its short experience to fit a model, and then uses the model to plan the next actions generating new experiences. following this loop for some time, eventually the models are able to get precise and the plans actually leads to the goal. fig. 4. in blue is the three step tree plan generated with the models learned trough a episode, on the left is the tree generated in the beginning and in the right at the end of the episode. the red line shows the entire path taken in the episode. (color figure online) full size image the fig. 4 show us two examples of tree plan generated with the models learned trough a episode, on the left is the tree generated in the beginning and in the right at the end of the episode. we can see how in the end of the episode, the quality of the models were improved resulting in a more consistent plan. 6 results to compare the advantages of using a segmented representation of the world against the standard approach of a single unified model, we designed a series of experiments aimed at highlighting the differences in performance. first we design the experiment in which we conducted ten runs of two episodes with at most three hundred steps each, where both models interacted with the same environment. by comparing their performances in this controlled setting, we expect to have both solution being very similar in the first episode. we limited the horizon to three hundred steps. so with this experiment setup we develop two scenarios, the first one just as we described but the second one we change the environment between first and second episode. as the agent maintain the knowledge of the previous episode, we expected to see a better adaption by the k-model solution in a different environment that brings familiar dynamics. so, for the first experiment, we run the process into two episodes. in this setup, each solution retained knowledge from previous episodes, allowing us to observe how their performance evolved over time. we expected both models to improve across episodes, leveraging their accumulated experience to make increasingly informed decisions. fig. 5. the different environment presented in scenario 2 where the environment changes between episodes. full size image in the second experiment, we introduced a novel twist by presenting the agent with a different environment in the second episode. while the dynamics of this environment remained very similar to the one previously encountered, the regions within it were arranged in different shapes and positions, as depicted in fig. 5. despite this change, both models retained persistent knowledge from the previous episodes. however, we anticipated that the segmented model would exhibit superior adaptability in this scenario. this expectation stemmed from the segmented model’s ability to identify previously learned models and adjust its routing network accordingly to accommodate the new configuration of regions. fig. 6. box plot showing distribution of steps that each model spent to get to the goal across first episode of the first scenario with blue being the standard single model approach and orange being the k-model segmented approach. (color figure online) full size image the analysis presented in fig. 6 reveals interesting insights into the performance of both approaches across episodes. initially, in the first episode, both the single-model and the k-models exhibit comparable performances, as evidenced by the similar duration of the episodes with the k-model being slightly worse. this suggests that both models are capable of effectively learning to navigate through the environment in a similar fashion. however, in the subsequent episode shown in fig. 6 a notable disparity emerges between the two approaches. the k-models demonstrate a significant reduction in episode length in most of the runs, compared to the single model. this observation suggests that the k-models have a better reset power to retrieve the knowledge from the beginning of the previous episode, while the changes presented through the episode had more impact in the single model approach, resulting in a better performance compared to that observed in the first episode but not as good as the improvement observed by the k-models. the improvement became even more evident when analysing the performances in the second episode for the scenario when the environment changes. the fig. 7 show us the difference between the two approaches on this scenario. the k-models demonstrate a significant reduction in episode length in most of the runs, compared to the single model as it did in the first scenario, but this time we can’t see the same improvement in the single model approach. this observation suggests that the single model struggles to transfer knowledge from the previous episode to the next when the environment changes, resulting in performance similar to that observed in the first episode, where it had to learn the environment from scratch. in contrast, the k-models leverage the dynamics learned in the first episode to adapt and improve their performance in the second episode. by dynamically adjusting their routing network, the k-models demonstrate a remarkable ability to rapidly adapt to the new environment, resulting in shorter episode duration. fig. 7. box plot showing distribution of steps that each model spent to get to the goal across the second episode of the second scenario with blue being the standard single model approach and orange being the k-model segmented approach (color figure online) full size image overall, these findings underscore the effectiveness of the segmented k-models approach in leveraging accumulated knowledge from previous episodes to facilitate faster adaptation and improved performance in subsequent episodes. this highlights the advantages of employing a segmented representation of the world’s dynamics, particularly in scenarios where the environment exhibits dynamic changes between episodes. 7 conclusion this work aims to demonstrate the benefits of leveraging segmented dynamics in reinforcement learning problems. by recognizing that the environment’s dynamics can be effectively segmented into distinct regions, we introduce a novel approach that enhances the efficiency of reinforcement learning solutions. through the utilization of a structured framework comprised of smaller models, the agent gains the ability to discern situations where it can switch between models. this segmentation allows for improved generalization across diverse contexts and facilitates faster adaptation to familiar dynamics within different environments. by adopting this approach, our research highlights the importance of understanding and leveraging the underlying structure of the environment’s dynamics. by incorporating segmented models into the learning process, we empower agents to navigate complex environments more effectively, ultimately leading to enhanced performance and adaptability in a variety of reinforcement learning tasks. one limitation of our approach is the requirement to specify the number of models as input, which may be unknown in practical applications. this presents a challenge, as determining the optimal number of models can be non-trivial and may require prior knowledge or experimentation. to address this limitation, future research could explore methods for automatically determining the appropriate number of models based on the characteristics of the environment or through data-driven approaches. techniques such as model selection criteria, bayesian optimization, or clustering algorithms could be employed to dynamically adjust the number of models based on observed data and performance metrics. additionally, the flexibility of our approach allows for the incorporation of adaptive mechanisms that can dynamically adjust the number of models based on feedback from the learning process. by incorporating such adaptive mechanisms, our approach can become more robust and applicable to a wider range of real-world scenarios where the number of models may vary or change over time. overall, addressing the limitation of requiring a predefined number of models is an important direction for future research in order to enhance the scalability and applicability of our approach in practical reinforcement learning applications. references alegre, l.n., bazzan, a.l.c., da silva, b.c.: minimum-delay adaptation in non-stationary reinforcement learning via online high-confidence change-point detection. in: proceedings of the 20th international conference on autonomous agents and multiagent systems, aamas ’21, pp. 97–105. international foundation for autonomous agents and multiagent systems, richland (2021) google scholar  chitnis, r., silver, t., tenenbaum, j., kaelbling, l.p., lozano-pérez, t.: glib: efficient exploration for relational model-based reinforcement learning via goal-literal babbling, vol. 13b, pp. 11782–11791 (2021). https://www.scopus.com/inward/record.uri?eid=2-s2.0-85100285730&partnerid=40 &md5=4e2e2fca957818475950cdcd634527fe doya, k., samejima, k., katagiri, k.i., kawato, m.: multiple model-based reinforcement learning. neural comput. 14(6), 1347–1369 (2002). https://doi.org/10.1162/089976602753712972 fox, r., krishnan, s., stoica, i., goldberg, k.: multi-level discovery of deep options (2017) google scholar  hallak, a., castro, d., mannor, s.: contextual markov decision processes (2015) google scholar  kidambi, r., rajeswaran, a., netrapalli, p., joachims, t.: morel: model-based offline reinforcement learning. in: nips 2020. curran associates inc., red hook (2020) google scholar  li, j., tang, c., tomizuka, m., zhan, w.: hierarchical planning through goal-conditioned offline reinforcement learning. ieee rob. autom. lett. 7(4), 10216–10223 (2022). https://doi.org/10.1109/lra.2022.3190100 article  math  google scholar  li, t., lambert, n., calandra, r., meier, f., rai, a.: learning generalizable locomotion skills with hierarchical reinforcement learning. in: 2020 ieee international conference on robotics and automation (icra), pp. 413–419 (2020). https://doi.org/10.1109/icra40945.2020.9196642 lin, d., zhang, j.: a reinforcement learning based rrt algorithm with value estimation. in: 2022 3rd international conference on big data, artificial intelligence and internet of things engineering (icbaie), pp. 619–624 (2022). https://doi.org/10.1109/icbaie56435.2022.9985840 ng, j.h.a., petrick, r.p.a.: incremental learning of planning actions in model-based reinforcement learning. in: proceedings of the 28th international joint conference on artificial intelligence, ijcai 2019, pp. 3195–3201. aaai press (2019) google scholar  rybkin, o., zhu, c., nagabandi, a., daniilidis, k., mordatch, i., levine, s.: model-based reinforcement learning via latent-space collocation, vol. 139, pp. 9190–9201 (2021). https://www.scopus.com/inward/record.uri?eid=2-s2.0-85161314975&partnerid=40 &md5=0f9769abe528c97cbf983f602ecff397 da silva, b., basso, e., bazzan, a., engel, p.: dealing with non-stationary environments using context detection, vol. 2006, pp. 217–224 (2006). https://doi.org/10.1145/1143844.1143872 sutton, r., barto, a.: reinforcement learning: an introduction. mit press, cambridge (2018) math  google scholar  young, k., ramesh, a., kirsch, l., schmidhuber, j.: the benefits of model-based generalization in reinforcement learning. in: proceedings of the 40th international conference on machine learning. icml 2023. jmlr.org (2023) google scholar  download references acknowledgments this study was supported in part by the coordenação de aperfeiçoamento de pessoal de nível superior (capes) – finance code 001, and by the center for artificial intelligence (c4ai-usp), with support by fapesp (grant #2019/07665-4) and by the ibm corporation. author information authors and affiliations universidade de são paulo, são paulo, sp, brazil guilherme albarrans & valdinei freire authors guilherme albarransview author publications search author on:pubmed google scholar valdinei freireview author publications search author on:pubmed google scholar corresponding author correspondence to guilherme albarrans . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper albarrans, g., freire, v. (2025). likelihood estimator for multi model-based reinforcement learning. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_13 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_13 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords model based reinforcement learning planning likelihood estimator publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature anisotropicbreast-vit: breast cancer classification in ultrasound images using anisotropic filtering and vision transformer | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper anisotropicbreast-vit: breast cancer classification in ultrasound images using anisotropic filtering and vision transformer conference paper first online: 30 january 2025 pp 95–109 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) anisotropicbreast-vit: breast cancer classification in ultrasound images using anisotropic filtering and vision transformer download book pdf download book epub joão otávio bandeira diniz  orcid: orcid.org/0000-0003-3303-33469,10, neilson p. ribeiro  orcid: orcid.org/0009-0004-0431-94389,10, domingos a. dias jr.  orcid: orcid.org/0000-0001-5916-695111, luana b. da cruz  orcid: orcid.org/0000-0002-9150-255311, giovanni l. f. da silva  orcid: orcid.org/0000-0001-6370-939x10,12, daniel l. gomes jr  orcid: orcid.org/0000-0001-8415-39179, anselmo c. de paiva  orcid: orcid.org/0000-0003-4921-062610 & … aristófanes c. silva  orcid: orcid.org/0000-0003-0423-251410  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 379 accesses 1 citation abstract breast cancer classification in ultrasound images is a challenging and arduous task, primarily due to the quality of the images and the complexity of the lesions encountered. early diagnosis of this pathology is known to increase patient survival chances. therefore, computational methods have been developed to assist specialist physicians in this critical task. this study introduces anisotropicbreast-vit, a method that integrates anisotropic filtering, balanced data augmentation, and vision transformer to aid in the classification of breast ultrasound images. the proposed approach achieves promising results with an accuracy of 98.82%, specificity of 98.62%, sensitivity of 99.25%, precision of 97.08%, f1-score of 98.16%, and auc-roc of 0.989, surpassing current benchmarks in the field. these findings suggest that anisotropicbreast-vit has the potential to significantly improve breast cancer diagnosis, demonstrating its effectiveness and robustness in clinical applications. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others advancing breast cancer histopathology classification with vision transformers chapter © 2025 vision transformer based tokenization for enhanced breast cancer histopathological images classification chapter © 2024 revolutionizing her-2 assessment: multidimensional radiomics in breast cancer diagnosis article open access 14 february 2025 1 introduction breast cancer is one of the leading causes of mortality among women worldwide. according to the world health organization (who), approximately 15% of annual deaths are attributed to this disease [3]. early detection is crucial for effective treatment, thereby contributing to the reduction of mortality rates. ultrasonography (us) is one of the safe techniques used for the detection and diagnosis of breast cancer, as it is a non-invasive method, does not involve radiation, and provides high-resolution images in real-time [5]. however, us image interpretation requires the expertise of highly trained radiologists. even the most experienced radiologists can exhibit variations in their tumor diagnoses, leading to inconsistencies [27]. to address these challenges, computer-aided detection and diagnosis systems (cad/cadx) have emerged. these systems have the potential to automate the analysis of us images, providing support to healthcare professionals in the identification and classification of breast lesions [24]. furthermore, deep learning methods such as convolutional neural networks (cnns) have proven to be highly effective in various computer vision applications in the medical field [11, 12, 16, 17, 23]. these models are capable of identifying patterns directly from images, eliminating the need for manual feature extraction [8]. however, it is important to note that us images are complex and often require appropriate preprocessing to highlight suspicious structures, thus ensuring accurate results. considering this issue, the main goal of this work is to develop and evaluate an automated method for breast lesion classification in ultrasound images using image preprocessing techniques and deep learning. also, the use of anisotropic and mean filters to improve the classification pipeline of breast cancer in medical images. the primary contributions of the proposed method include: development of an automated classification method that uses image enhancement techniques to highlight intrinsic features (anisotropic and mean filters) of lesions in ultrasound images; creation of a balanced data augmentation method for the effective training of a vision transformer; robust method composed of techniques that optimize the results of the vision transformer, improving classification reliability. 2 related works breast cancer classification in ultrasound images is widely studied in the literature. in this section, we present the main related works to the proposed method found from the year 2020 onwards. it is worth noting that the search was also conducted considering the variability of existing techniques to identify gaps among them. in [2], the authors highlight the challenge of training deep learning models due to the scarcity of training samples in breast ultrasound (us) images. to address this problem, the authors proposed an approach called noise filter network (nf-net). this approach demonstrated an accuracy of 73%, sensitivity of 80%, and f1-score of 74%. the results suggest that nf-net is a promising approach for breast lesion classification, showing the potential to improve accuracy in scenarios with limited data. in [24], breast lesion classification in ultrasound (us) images was performed using image fusion and various cnn architectures, such as vggnet, resnet, and densenet. the authors achieved notable results, with an accuracy of 94.62%, sensitivity of 92.31%, specificity of 95.60%, precision of 90%, f1-score of 91.14%, and an area under the curve (auc) of 0.9711. these results, obtained on the busi dataset, indicate the effectiveness of image fusion techniques and various cnns in detecting breast lesions. in the study conducted by [25], a radiomics-based technique combined with a semi-supervised adversarial neural network model for data augmentation was used. the dataset, of private origin, contained 1,447 us images, with 767 benign and 680 malignant. the best results achieved were an accuracy of 90.41%, sensitivity of 87.94%, and specificity of 85.86%. these results highlight the potential of using radiomics and semi-supervised learning to improve the classification of breast lesions in the us. the authors of [31] investigated bilateral filters with various cnn architectures to determine the best training model. the study results showed an accuracy of 95.48%, sensitivity of 93.92%, specificity of 98.33%, precision of 98.11%, f1-score of 95.71%, and auc of 0.9883. these results indicate that preprocessing with bilateral filtering, combined with optimized cnn architectures, can significantly improve the accuracy of breast lesion classification in the us. in the study by [15], a hybrid method was presented that uses three cnns for feature extraction. then, the minimum redundancy maximum relevance algorithm is applied to select the most relevant features, and a support vector machine (svm) is used to train the model. this method achieved an accuracy of 95.6%, highlighting the effectiveness of combining feature extraction and selection techniques in improving the performance of classification models. in turn, [28] proposed an architecture based on inceptionv3 for breast lesion classification. to overcome the lack of data, the authors used five combined datasets. the resulting model achieved an accuracy of 81%, sensitivity of 77%, f1-score of 80%, and auc of 0.81. these results suggest that using multiple datasets can help mitigate issues related to data scarcity and improve the robustness of deep learning models. finally, [21] proposed using three combined resnet architectures for feature extraction. after extraction, three different techniques were applied to select the most relevant features, and training was performed using an svm. using a dataset with 1,236 images, the best results achieved were an accuracy of 84.9%. these results indicate that combining multiple cnn architectures and feature selection techniques can improve the performance of breast lesion classification models. the related works present various approaches for classifying breast lesions in ultrasound images, each with its advancements. techniques range from methods involving radiomics and semi-supervised learning to methods using cnns and image fusion. despite significant advancements, each study faces specific challenges, such as the scarcity of training data, variations in image quality, and the need for intensive preprocessing. the method proposed in this work aims to fill a crucial gap by employing advanced preprocessing techniques and deep learning to automate and enhance breast lesion classification in ultrasound images. by integrating image enhancement techniques and a balanced data augmentation method for training a vision transformer, our method aims to achieve higher classification accuracy and robustness. additionally, the proposed approach seeks to simplify the process for healthcare professionals, providing faster and more accurate diagnosis, which is crucial for early detection and breast cancer treatment. 3 materials and proposed method in this section, we describe the materials and the proposed method. it consists of five steps, as highlighted in fig. 1. subsequently, we detail each step. fig. 1. proposed method flow. full size image 3.1 materials (busi dataset) the dataset used for this study was the breast ultrasound images dataset (busi). the busi was collected in 2018 and includes women aged between 25 and 75 years. the dataset has 780 images with an average size of \(500\times 500\) pixels in png format. also, a lesion marking file accompanies each image and has three classes: normal, benign, and malignant [1]. only images from the benign (210 images) and malignant (454 images) classes were selected in this work. this choice is because the fact that a biopsy is only necessary when there is doubt in the diagnosis between benign and malignant. therefore, normal images were not included in the dataset, focusing on the problem of diagnosing benign and malignant lesions, which is the main objective of this study. figure 2 shows examples of each class. fig. 2. materials (busi): (a) benign images; (b) malignant images. full size image in the next step, we will describe the extraction of the region of interest (roi), which was used to enhance the model by eliminating unnecessary information from the images. 3.2 roi cropping as described earlier, the images contain markings made by specialists. it is also noticeable, as illustrated in fig. 2, that there is a significant amount of information in the image that is not essential for classification and may even confuse the classifier. many studies show improved results using roi extraction [6, 7, 9, 11]. therefore, after conducting detailed experiments, we chose roi crop. in practice, this involves selecting the largest bounding box of the lesion in the markings and, subsequently, cropping the corresponding region from the original image, generating a new image that will be used in the subsequent steps of the method. figure 3 illustrates the procedure for extracting the roi. fig. 3. roi cropping: (a) original image (top: benign lesion, bottom: malignant lesion); (b) marking with bounding box in yellow; (c) resulting image. (color figure online) full size image with the rois extracted, the next step is image preprocessing. 3.3 roi preprocessing many studies have shown that image preprocessing has a significant impact on cnn results [9, 10, 20, 26]. in this step, we applied two filters sequentially to enhance the image representation. firstly, the anisotropic filter is applied, known for preserving edges in the image while effectively removing noise [30]. this filter is particularly advantageous in ultrasound (us) images, where edges are crucial for interpreting benign and malignant lesions. it preserves the structural characteristics of the image, such as edges and details, while adaptively reducing noise. next, the mean filter is applied. this filter is effective in removing background noise, smoothing the image, and enhancing the consistency of the structures identified by the anisotropic filter [18]. these filters combination enhances the visual us quality images and prepare them efficiently for feature extraction by cnns, improving the accuracy of computer-aided diagnostic systems. figure 4 illustrates the application of these filters. fig. 4. roi preprocessing: (a) roi image; (b) anisotropic filter; (c) resulting image (anisotropic + mean). full size image we observed that the noise has been smoothed while the edges of the lesion have been preserved and enhanced. this process has proven to be effective in improving the training results of the vision transformer. 3.4 balanced data augmentation before training the model, it is necessary to prepare the data by resizing and augmenting the minority class in the training dataset. firstly, it is important to note that after the roi extraction step, the dimensions of the images in the dataset may vary in height and width. however, to optimize the training of specific cnn architectures, all images must have the same size in height and width. for this reason, we chose to resize all images to dimensions of (\(224\times {224}\)), which are suitable for training within the deep learning pipeline [19]. next, observing 210 malignant images and 454 benign images, an imbalance problem between the classes becomes evident. therefore, the database is divided into three sets: training, validation, and test. during training, data augmentation is performed based on the minority class (malignant images) to balance the number of samples between the classes. essentially, we compared the size of the minority class to that of the majority class. a set of hyperparameters is defined for data augmentation (up to a 5-degree rotation range, 10% image zoom-in, horizontal and vertical flipping). thus, while the size of the minority class is smaller than that of the majority class, these transformations can occur randomly until the size of the majority class is reached. as a result, we increase the data and its representation. after balancing the training samples, the vision transformer (vit) architecture is defined and trained. 3.5 vision transformer training after preparing the data, the next step is to classify benign and malignant lesions in breast us images using the vit-base model [13], as shown in fig. 5. first, the training and validation data were used to build the model. after constructing the model, the test dataset was employed to evaluate its robustness and performance. the vit-base architecture includes an embedding layer, multiple transformer blocks, and a multilayer perceptron (mlp). the embedding layer converts a 2d image into flattened token sequences while retaining positional information to feed into the stacked transformer blocks. the standard transformer encoder features multi-head self-attention layers interspersed with mlp blocks. additionally, each block begins with a layernorm and includes residual connections at the end. the final mlp layer classifies the samples based on the output from the stacked transformer blocks [13]. to facilitate training, pre-trained models (with the imagenet database) were used with the default settings of the keras library [4]. fig. 5. vision transformer training: architecture illustration. full size image 3.6 evaluation metrics after training, we evaluate the model predictions using validation metrics to determine the effectiveness and robustness of the proposed method. in this study, the following metrics were employed: accuracy (acc), sensitivity (sens), specificity (spec), precision (prec), f1-score, and area under the roc curve (auc-roc) [14]. 4 results and discussion now, after the method, the next step is to evaluate each technique that comprises it. in this section, we will present the results achieved by the proposed method. to develop the method and experiments, a computer with the following specifications was used: intel® core™ i7 cpu@ 2.90ghz, nvidia rtx-3060 12gb gpu, 16gb of ram, and windows 11 pro operating system. the implementation was carried out in python, using libraries such as pandas, opencv, scikit-learn, and keras. the busi dataset was divided into training (60%), validation (20%), and test (20%) sets. the training and validation sets were used to create the best model, while the test set was used for model evaluation. all hyperparameters used in vit were defaulted and available in the keras library. to evaluate each step of the method, we conducted comparisons between the inclusion and exclusion of each proposed technique. the following sections present and discuss each of the experiments. finally, we compare the results obtained with the literature presented in sect. 2. 4.1 roi cropping result two tests were conducted to evaluate the need for roi cropping in the exams. first, the vit (as described in sect. 3.5) was executed on the original images (without the roi). then, the images were subjected to roi cropping and processed again by vit. table 1 detailed roi cropping results. table 1. results of roi cropping step.full size table we can observe that by performing roi cropping, there is a significant improvement in the accuracy, specificity, precision, f1-score, and auc metrics. although the sensitivity metric shows a reduction, it is important to highlight that the accuracy and f1-score metrics, which provide an overall average of the models and a harmonic mean between precision and sensitivity, respectively, showed noteworthy improvements in results. it suggests that the model was able to enhance its classification capability. 4.2 roi preprocessing result after defining the roi cropping step, the next step involves applying image preprocessing techniques to enhance its internal characteristics. to evaluate this step, we compared the results obtained without and with preprocessing (table 2). table 2. results of roi preprocessing step.full size table the results indicate that using the preprocessing step with anisotropic and mean filters resulted in improvements across all metrics. we highlight sensitivity, which had previously decreased with the roi cropping step but has now exceeded the previous metric. additionally, the f1-score deserves attention, showing a 6% improvement with the image enhancement step. these improvements can be attributed to the ability of these techniques to enhance relevant features in the breast us images. for instance, the anisotropic filter effectively preserved edges while reducing noise, which is crucial for detecting image details. similarly, the mean filter reduced noise impact, making it easier to identify subtle patterns. therefore, the effectiveness of these techniques in improving evaluation metrics can be attributed to their ability to optimize the representation of relevant features in the images, allowing the model to more accurately focus on important aspects during the classification process. 4.3 balanced data augmentation result at this step, the training images of the malignant class undergo random data augmentation operations until they reach the same quantity as the majority class (benign class). this approach is crucial to increase the representativeness of the minority class, making the model more robust and balanced. table 3 describes the results of the proposed technique in this step, comparing it with the previous step (without using data augmentation). table 3. results of balanced data augmentation step.full size table by employing the proposed data augmentation technique, we observed a significant improvement in results. all metrics showed notable enhancements, particularly sensitivity, which reached 99.25%. this increase in sensitivity indicates the model’s ability to correctly identify images from the malignant class. furthermore, when observing the other metrics, all are above 97%. we believe that the overall performance improvement of the model is a consequence of the increased diversity in the training data provided by data augmentation. by exposing the model to a greater variety of examples, a more comprehensive understanding of relevant characteristics is obtained, making it more capable of generalizing to new examples and handling variability in real-world data. it demonstrates that achieving greater diversity in the training set contributes to robust model performance. 4.4 vit result to evaluate the performance of vit, we compared it with two other widely used cnns: resnet50 [22] and efficientnetb0 [29]. overall, we kept all the steps of the method the same, replacing only vit with the mentioned networks. it is worth noting that these networks are also pre-trained on imagenet with default settings from the keras library. table 4 describes the results achieved. table 4. results of distinct cnns.full size table it’s worth noting that all networks achieved noteworthy metrics for breast cancer classification in breast us images. it demonstrates that the steps before the cnn were crucial in improving the model’s performance. however, when analyzing the results achieved by vit, a significant improvement of 2% to 6% is observed compared to the cnns, surpassing them. transformers such as vit distinguish themselves from traditional cnns like resnet50 and efficientnetb0 by employing global attention mechanisms instead of convolution operations. it allows vit to process relationships between all parts of the image simultaneously, capturing correlations on a broader scale. the architecture also excels in capturing long-range dependencies in the data, which is crucial for tasks where subtle patterns may occur in different parts of the image and require a global approach for accurate classification [13]. these factors contributed to its superior performance in breast cancer classification tasks in us images. therefore, we emphasize that the proposed method is robust in breast cancer classification. this method, used in cad/cadx systems, can contribute to the rapid and effective diagnosis of this pathology, assisting specialist doctors in their daily clinical routine and thereby improving the quality of life for patients. 4.5 related works comparison to validate the proposed method concerning the studies found in the literature, we present table 5, listing the studies from sect. 2 along with the metrics achieved by them and by our work. it is important to note that, due to the methodological variations among the studies, the comparison is a reference for general trends and differences rather than providing an absolute measure of superiority. table 5. comparison of the proposed method with related works.full size table the proposed method excels in all evaluated metrics, surpassing the results of the related works presented. particularly, the f1-score and sensitivity metrics demonstrate the robustness of the proposed method, reflecting its ability to balance precision and sensitivity in detecting positive cases of breast cancer. these results reinforce the efficiency and reliability of the method, establishing it as a solid and effective approach for breast cancer classification in us images. it is important to note that direct comparisons can be complex, as methods are developed on different datasets and with distinct approaches. however, upon examining the techniques employed, our work stands out as the only one to propose enhancements to ultrasound images using advanced image processing techniques. additionally, we introduce automated data augmentation to improve model generalization. furthermore, we innovate by using a transformer-based cnn, an approach not addressed in related studies. thus, these approaches’ integration harmonizes classification techniques and surpasses the metrics of all other works. therefore, we believe our work represents a significant contribution to the literature, holding a prominent position. 4.6 case study in this section, we will analyze two case studies. in the first study, a lesion initially considered benign was incorrectly classified as malignant (fig. 6 (a)). in the second case, a malignant lesion was erroneously classified as benign (fig. 6 (b)). fig. 6. case study. (a) benign lesion classified as malignant; (b) malignant lesion classified as benign. full size image in fig. 6 (a), it can be observed that although the lesion is benign, its edges exhibit irregularities. this particularity may have contributed to the model’s misclassification as malignant. on the other hand, upon examining fig. 6 (b), it is noticeable that the edges are more uniform, a common characteristic in benign lesions, which may have resulted in the incorrect classification of the lesion. however, it is to emphasize that these models can complement medical expertise, offering an additional tool for analysis and diagnosis. by combining them with careful clinical assessment and other relevant data, it is possible to improve the accuracy of diagnosis and thus play a crucial role in the fight against breast cancer. 5 conclusion for this work, we propose anisotropic and mean filters, balanced data augmentation, and the vit network for breast cancer classification in ultrasound images. the filters proved effective in enhancing relevant image features, while balanced data augmentation contributed to improving the model’s robustness by balancing the representation of benign and malignant classes. furthermore, the vit network demonstrated high efficacy in classifying medical ultrasound images. the combination of these techniques resulted in promising outcomes, surpassing existing methods in the literature. additionally, all validation metrics were above 97%, confirming the effectiveness of the proposed techniques. in future work, we suggest investigating other preprocessing filters to validate the choice of anisotropic and mean filters. additionally, another improvement to the method would be the optimization of the vit network’s hyperparameters, as we used only the standard hyperparameters from the literature for this work. thus, we highlight that the proposed method is promising, potentially facilitating the clinical routine of specialist physicians and improving the chances of cure and survival for patients. references al-dhabyani, w., gomaa, m., khaled, h., fahmy, a.: dataset of breast ultrasound images. data brief 28, 104863 (2020) article  math  google scholar  cao, z., yang, g., chen, q., chen, x., lv, f.: breast tumor classification through learning from noisy labeled ultrasound images. med. phys. 47(3), 1048–1057 (2020) article  math  google scholar  cheng, h.d., shan, j., ju, w., guo, y., zhang, l.: automated breast cancer detection and classification using ultrasound images: a survey. pattern recogn. 43(1), 299–317 (2010) article  math  google scholar  chollet, f., et al.: keras: the python deep learning library. astrophysics source code library (2018) google scholar  chougrad, h., zouaki, h., alheyane, o.: deep convolutional neural networks for breast cancer screening. comput. methods programs biomed. 157, 19–30 (2018) article  google scholar  diniz, j.o., ferreira, j.l., da silva, g.l., quintanilha, d.b., silva, a.c., paiva, a.: segmentação de coração em tomografias computadorizadas utilizando atlas probabilístico e redes neurais convolucionais. in: anais do xxi simpósio brasileiro de computação aplicada à saúde, pp. 83–94. sbc (2021) google scholar  diniz, j.o.b., et al.: heart segmentation in planning ct using 2.5d u-net++ with attention gate. comput. methods biomech. biomed. eng. imag. vis. 11(3), 317–325 (2023) google scholar  diniz, j.o.b., diniz, p.h.b., valente, t.l.a., silva, a.c., de paiva, a.c., gattass, m.: detection of mass regions in mammograms by bilateral analysis adapted to breast density using similarity indexes and convolutional neural networks. comput. methods programs biomed. 156, 191–207 (2018) article  math  google scholar  diniz, j.o.b., ferreira, j.l., cortes, o.a.c., silva, a.c., de paiva, a.c.: an automatic approach for heart segmentation in ct scans through image processing techniques and concat-u-net. expert syst. appl. 196, 116632 (2022) article  google scholar  diniz, j.o.b., ferreira, j.l., diniz, p.h.b., silva, a.c., de paiva, a.c.: esophagus segmentation from planning ct images using an atlas-based deep learning approach. comput. methods programs biomed. 197, 105685 (2020) article  math  google scholar  diniz, j.o.b., ferreira, j.l., diniz, p.h.b., silva, a.c., paiva, a.c.: a deep learning method with residual blocks for automatic spinal cord segmentation in planning ct. biomed. sig. process. control 71, 103074 (2022) article  math  google scholar  diniz, j.o., quintanilha, d.b., de carvalho filho, a.o., gomes jr, d.l., silva, a.c., braz jr, g., de paiva, a.c., luz, d.d.s.: detecção de covid-19 em imagens de raio-x de tórax através de seleção automática de pré-processamento e de rede neural convolucional. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 162–173. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229576 dosovitskiy, a., et al.: an image is worth 16x16 words: transformers for image recognition at scale. arxiv preprint arxiv:2010.11929 (2020) duda, r.: pattern classification and scene analysis. wiley-interscience publication, vol. 512 (1973) google scholar  eroğlu, y., yildirim, m., çinar, a.: convolutional neural networks based classification of breast ultrasonography images by hybrid method with respect to benign, malignant, and normal using mrmr. comput. biol. med. 133, 104407 (2021) article  math  google scholar  fernandes, a.g.s., braz junior, g., diniz, j.o.b., silva, a.c., matos, c.e.f.: efficientdeeplab for automated trachea segmentation on medical images. in: brazilian conference on intelligent systems, pp. 154–166. springer (2023) google scholar  figueredo, w., et al.: abordagem computacional baseada em deep learning para o diagnóstico de endometriose profunda através de imagens de ressonância magnética. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 138–149. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229567 gonzalez, r., woods, r.: digital image processing. prentice hall, pearson (2008) math  google scholar  júnior, d.a.d., et al.: automatic method for classifying covid-19 patients based on chest x-ray images, using deep features and pso-optimized xgboost. expert syst. appl. 183, 115452 (2021) article  math  google scholar  júnior, d.a.d., da cruz, l.b., diniz, j.o., júnior, g.b., silva, a.c.: classificação automática de glóbulos brancos usando descritores de forma e textura e extreme gradient boosting. in: anais do xxi simpósio brasileiro de computação aplicada à saúde, pp. 95–106. sbc, porto alegre, rs, brasil (2021). https://doi.org/10.5753/sbcas.2021.16056 kılıçarslan, g., koç, c., özyurt, f., gül, y.: breast lesion classification using features fusion and selection of ensemble resnet method. int. j. imaging syst. technol. 33(5), 1779–1795 (2023) article  google scholar  koonce, b., koonce, b.: resnet 50. in: convolutional neural networks with swift for tensorflow: image recognition and dataset categorization, pp. 63–72 (2021) google scholar  marin, l.f., silva, a.c., bandeira, j.o.: meta-data construction for selection of breast tissue biopsy slides image classifier to identify ductal carcinoma. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 729–734. ieee (2019) google scholar  moon, w.k., lee, y.w., ke, h.h., lee, s.h., huang, c.s., chang, r.f.: computer-aided diagnosis of breast ultrasound images using ensemble learning from convolutional neural networks. comput. methods programs biomed. 190, 105361 (2020) article  math  google scholar  pang, t., wong, j.h.d., ng, w.l., chan, c.s.: semi-supervised gan-based radiomics model for data augmentation in breast ultrasound mass classification. comput. methods programs biomed. 203, 106018 (2021) article  google scholar  santos, p., brito, v., filho, a.c., sousa, a., diniz, j., luz, d.: efficientbacillus: uma arquitetura profunda para detecção dos bacilos de koch. in: anais do xxiii simpósio brasileiro de computação aplicada à saúde, pp. 198–209. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/sbcas.2023.229608 shan, j., alam, s.k., garra, b., zhang, y., ahmed, t.: computer-aided diagnosis for breast ultrasound using computerized bi-rads features and machine learning methods. ultrasound med. biol. 42(4), 980–988 (2016) article  google scholar  sirjani, n., et al.: a novel deep learning model for breast lesion classification using ultrasound images: a multicenter data evaluation. phys. med. 107, 102560 (2023) article  math  google scholar  tan, m., le, q.: efficientnet: rethinking model scaling for convolutional neural networks. in: international conference on machine learning, p. 6105. pmlr (2019) google scholar  weickert, j., et al.: anisotropic diffusion in image processing, vol. 1. teubner stuttgart (1998) google scholar  zhuang, z., yang, z., raj, a.n.j., wei, c., jin, p., zhuang, s.: breast ultrasound tumor image classification using image decomposition and fusion based on adaptive multi-model spatial feature fusion. comput. methods programs biomed. 208, 106221 (2021) article  math  google scholar  download references acknowledgments this work was supported by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, fundação de amparo à pesquisa do maranhão (fapema), conselho nacional de desenvolvimento científico e tecnológico (cnpq), and empresa brasileira de serviços hospitalares (ebserh) brazil (proc. 409593/2021-4). we also acknowledge llm use for spell-checking, grammar correction, and assistance in translating specific terms. author information authors and affiliations innovation factory – federal institute of maranhão (ifma), são luís, maranhão, brazil joão otávio bandeira diniz, neilson p. ribeiro & daniel l. gomes jr applied computing group (nca), federal university of maranhão (ufma), são luís, maranhão, brazil joão otávio bandeira diniz, neilson p. ribeiro, giovanni l. f. da silva, anselmo c. de paiva & aristófanes c. silva federal university of cariri (ufca), juazeiro do norte, ce, brazil domingos a. dias jr. & luana b. da cruz dom bosco higher education unit (undb), são luís, ma, brazil giovanni l. f. da silva authors joão otávio bandeira dinizview author publications search author on:pubmed google scholar neilson p. ribeiroview author publications search author on:pubmed google scholar domingos a. dias jr.view author publications search author on:pubmed google scholar luana b. da cruzview author publications search author on:pubmed google scholar giovanni l. f. da silvaview author publications search author on:pubmed google scholar daniel l. gomes jrview author publications search author on:pubmed google scholar anselmo c. de paivaview author publications search author on:pubmed google scholar aristófanes c. silvaview author publications search author on:pubmed google scholar corresponding author correspondence to joão otávio bandeira diniz . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper diniz, j.o.b. et al. (2025). anisotropicbreast-vit: breast cancer classification in ultrasound images using anisotropic filtering and vision transformer. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_7 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_7 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords breast cancer ultrasound vision transformer publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature interpreting convolutional neural networks for brain tumor classification: an explainable artificial intelligence approach | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper interpreting convolutional neural networks for brain tumor classification: an explainable artificial intelligence approach conference paper first online: 12 october 2023 pp 77–91 cite this conference paper download book pdf download book epub intelligent systems (bracis 2023) interpreting convolutional neural networks for brain tumor classification: an explainable artificial intelligence approach download book pdf download book epub dieine estela bernieri schiavon  orcid: orcid.org/0000-0001-8090-23869, carla diniz lopes becker  orcid: orcid.org/0000-0002-0403-49169, viviane rodrigues botelho  orcid: orcid.org/0000-0003-1864-59269 & … thatiane alves pianoski  orcid: orcid.org/0000-0001-9552-69939  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 613 accesses 10 citations abstract brain tumors pose a complex medical challenge, requiring a specific approach for accurate diagnosis and effective treatment. early detection can significantly improve outcomes and quality of life for patients with brain tumors. magnetic resonance (mri) is a powerful diagnostic tool, and convolutional neural networks (cnns) are efficient deep learning algorithms for image analysis. in this study, we explored using two cnn models for brain tumor classification and applied hyperparameter optimization and data augmentation techniques to achieve an accuracy of up to 96%. in addition, we use explainable artificial intelligence (xai) techniques to visualize and interpret the behavior of cnn models. our results show that cnn models accurately classified mri images with brain tumors. xai techniques helped us to identify the patterns and features used by the models to make predictions. this study supports the development of more reliable medical diagnoses for brain tumors using cnn models and xai techniques. the source code is available at (https://github.com/dieineb/bracis23). the repository also has images generated during the experiments. download conference paper pdf similar content being viewed by others interpreting convolutional neural networks for brain tumors: a transparent box approach chapter © 2025 classification of brain tumors from mri images utilizing a convolutional neural network chapter © 2024 tumor sight ai: brain tumor prediction system using deep learning and explainable artificial intelligence (xai) chapter © 2026 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. brain cancer computational intelligence learning algorithms machine learning neuroradiology artificial intelligence 1 introduction the central nervous system (cns) tumors are due to the abnormal growth of cells in the tissues of these locations. according to the national cancer institute, cns cancer represents 1.4% to 1.8% of all malignant tumors worldwide. about 88% of cns tumors are in the brain [1]. these tumors cause profound deficits and are fatal in patients whose initial symptoms are often not perceived. malignant-based tumors are classified by the world health organization (who) [2] into four stages (i-iv) according to aggressiveness. tumors classified in stages iii and iv are more aggressive and need immediate treatment and can lead the individual to death in less than two years. the classification of magnetic resonance imaging (mri) is an important part of the processing of medical imaging and assists physicians in making diagnoses and more accurate treatments. the mri is one of the leading imaging tools for studying brain issues. it is often the modality of choice for diagnosis and treatment planning for tumors brain due to the high resolution and significant contrast of soft tissues [3]. the first methods of characterization of cancer brains were predominantly based on modeling neuroimaging data statistics. driven by advances in computing, deep learning has become standard in image classification medical. integrated statistics and deep learning methods have recently emerged as a new direction in the automation of medical practice, unifying knowledge multidisciplinary in medicine, statistics, and artificial intelligence [4]. convolutional neural networks (cnns) are deep learning algorithms, very efficient for image analysis [5]. they are frequently used in health, mainly for classifying, locating, detecting, and segmenting tumors on magnetic resonance imaging. the architecture typical of cnn for image processing consists of a series of layers of convolution filters interspersed with a series of reduction layers or data grouping. convolutional layers in cnns serve as feature extractors, learning feature representations from input images. neurons in these layers are organized into feature maps, and filters connect them to neurons in the previous layer. the convolution filters detect relevant image features, progressively capturing more complex patterns. additionally, pooling layers are commonly employed after convolutional layers to reduce image dimensions. multiple convolutional layers are often stacked together to create a deep model, enabling the extraction of abstract feature representations [6]. the fully connected layers interpret these extracted features and perform high-level reasoning tasks. deep learning algorithms are having an increasing impact on our everyday lives. as machine learning algorithms are increasingly applied to high-impact, high-risk tasks like medical diagnoses, it is critical that researchers can explain how these algorithms arrived at their predictions [7]. however, neural network architectures have a natural propensity for opacity in understanding data processing and prediction. explainable artificial intelligence (xai), used in machine learning, is an artificial intelligence approach that aims to create transparent and interpretable models providing human-understandable explanations for their decisions or predictions. xai aims to improve the reliability of ai systems and facilitate collaboration between humans and machines. these techniques include methods for visualizing model internals, feature importance analysis, and rule extraction, among other methods. this paper focuses on the model visualization technique: feature visualization by layers and class activation maps (cams). visualization and cams techniques are used to interpret and understand the behavior of machine learning models. feature visualization generates images that activate specific neurons in a neural network, allowing researchers to understand the patterns and features that the network uses to make predictions. cams identify the important regions of an input image that contribute the most to the output of a model, which helps understand which features the model uses to make predictions and diagnose performance issues. both techniques are valuable tools for identifying areas for improvement in a model’s performance. by explaining the predictions made by the model, xai techniques increase transparency and reliability, allowing medical professionals to understand the reasoning behind the diagnosis better. this study aims to evaluate two cnn models that, after using hyperparameter optimization techniques and data augmentation, presented accuracy results greater than 96% in all stages (training, validation, and testing) to classify resonance images with brain tumors and demonstrate the characteristics of the networks considered for image classification using the xai technique. for this purpose, the databases br35h brain tumor detection [8] and the brain tumor classification (mri) by sartaj [9], both available in kaggle, will be used. this paper is structured as follows. section 2 presents related works in deep learning on medical image analysis and explainable artificial intelligence. then, sect. 3 presents the dataset, pre-processing, data augmentation, and the networks’ structure used for training. section 4 presents the results, the discussion, and the extracted xai features for enhancing interpretability. section 5 concludes the paper. 2 related works explainable artificial intelligence is a technique used to understand an ai model and explain how it arrived at its results or decisions. xai is particularly important in deep learning, as deep neural networks can be very complex, making it difficult to understand how they arrived at a particular decision or prediction. in deep learning, xai provides insight into a neural network’s internal workings. this can be achieved through various techniques, such as visualizing the intermediate layers of a network or generating heatmaps to show which parts of an input image were most important in making a particular prediction. xai techniques are becoming increasingly important in medical diagnosis using deep learning models, as they can help improve the interpretability and trustworthiness of these models and improve patient outcomes. medical diagnostic support has shown great promise with them in identifying and classifying various medical conditions based on medical imaging data, such as x-rays, mri, and computed tomography scans. however, they can be highly complex. this section reports some published research on the question of this article: are the results obtained through cnns reliable for supporting the medical diagnosis? the paper [10] discusses the need for a more scientific approach to interpretability in machine learning. they propose a framework for characterizing different types of interpretability and identify research challenges in developing a rigorous science of interpretable machine learning. the authors discuss potential future directions for research in this area, such as developing new evaluation metrics and exploring the ethical implications of interpretability. the paper emphasizes the importance of interpretability as a critical component of responsible and trustworthy machine learning systems. in [11], the authors propose a method that uses a combination of deep neural networks (dnns) and multiclass support vector machines (svms) to improve the accuracy and interpretability of brain tumor diagnosis. to improve the interpretability of their model, the authors use explainable artificial intelligence (xai) techniques, including feature visualization and salience maps, to identify the key features and regions of the input images most relevant for tumor detection. they also compare the performance of their model to other state-of-the-art brain tumor detection methods. the authors demonstrate that the proposed method achieves high accuracy in detecting brain tumors greater than 97% and that using xai techniques improves the interpretability of their model and provides information about the underlying characteristics of brain tumors. the authors tested the model using two databases (figshare and brats 2018). the paper highlights the potential of combining deep learning, machine learning, and xai techniques to improve the accuracy and interpretability of brain tumor diagnosis. in the study [12], seven cnn models are evaluated for the classification of brain tumors. the models considered include a generic cnn model and six pre-trained cnn models. the dataset used for this research is called msoud, which comprises three existing datasets: fighshare, sartaj, and br35h. this combined dataset consists of 7023 mri images. the study results indicate that the inceptionv3 model achieves the highest accuracy among all evaluated cnn models. it achieves an average accuracy of 97.12% on the msoud dataset. the authors used the three banks together (80% for training and 20% for testing). the paper focused on showing the performance of the models through metrics based on values, even though it is a recent study published in 2023. however, there has been a growing tendency to incorporate explainability techniques for greater transparency of the models, in addition to the traditional metrics in the last few years. in [13], the authors propose an explanation-driven dl model that utilizes a convolutional neural network, local interpretable model-agnostic explanation (lime), and shap for predicting discrete subtypes of brain tumors (meningioma, glioma, and pituitary) using an mri image dataset. unlike previous models, the proposed model uses a dual-input cnn approach to overcome classification challenges posed by low-quality images with noise and metal artifacts by adding gaussian noise. the cnn training results demonstrate 94.64% accuracy, outperforming other state-of-the-art methods. to ensure consistency and local accuracy for interpretation, the authors employ shapley additive explanations (shap), which examines all future predictions applying all possible combinations of inputs, and lime, which constructs sparse linear models around each prediction to illustrate how the model operates in the immediate area. the study [14] proposes a convolutional neural network (cnn) model with a new form of grad-cam called numerical grad-cam (numgrad-cam) to provide a user-friendly explainability interface for brain tumor diagnosis. the numgrad-cam-cnn model was evaluated using technical and physician-oriented (human-side) evaluations and achieved an average accuracy of 97.11% a sensitivity of 95.58% and a specificity of 96.81% for the targeted brain tumor diagnosis setup. moreover, the integration of numgrad-cam provided an accuracy of 90.11% compared to other cam variations in the same cnn model. physicians who used the numgrad-cam-cnn model provided positive feedback on its usefulness in providing an explainable and safe diagnosis decision-making perspective for brain tumors. this study demonstrates that the proposed method can effectively enhance the interpretability of deep learning models for brain tumor diagnosis, making them more trustworthy and accessible to healthcare professionals. the study [15] demonstrated the performance of the commonly used convolutional neural network (cnn) models: vgg16, resnet50, and mobilenet. these models were applied to brain magnetic resonance imaging to identify tumor cells. the reported accuracy of these models was 97%, 94.5%, and 99%, respectively. the authors aimed to develop a modified cnn model comparing the proposed model with the three known networks. the model proposed in the study achieved an overall classification accuracy of 98.5%. all models were trained and tested using only one database, the br35h. the authors did not explicitly mention the use of explainable techniques in their study. however, they expressed their intention to increase the model’s reliability by implementing similar medical imaging datasets. using a single dataset in a study, as mentioned in some papers presented in this section, may limit the generalizability of the results. robust conclusions typically require validation across multiple independent datasets. while the studies presented provide insights into the performance of specific cnn models for medical diagnosis, further research, and validation on diverse datasets are essential to establish the reliability and generalizability of these models. in addition to evaluating the performance of the models with statistical metrics and testing in more than one database, an excellent current practice is to evaluate through interpretability techniques so that the models are not seen as black boxes. these xai techniques offer different approaches to explain the decision-making processes of deep learning models. visualizing important regions, assigning feature importance values, or calculating gradients provide insight into the model’s inner workings and help users understand why specific decisions or predictions were made. these techniques contribute to the reliability, acceptance, and responsible application of deep learning models in various domains, including medical diagnosis. 3 experiments for the development and execution of codes necessary for manipulating data and implementing the model, was used google collaboratory pro. the machine learning library used was keras via the tensorflow platform. two cnn architectures were used for the experiments, a standard multi-layered deep cnn (fig. 1), conv2d developed with three convolutional layers, and a deeper one, the xception model (fig. 2) with 36 and separable convolutional layers. fig. 1. the basic model layout [own author]. full size image fig. 2. xception model layout [16] full size image datasets. this study utilized two databases for brain tumor detection and classification. the first database used was the br35h [8] brain tumor detection database, which consists of 3000 magnetic resonance images (mri) representing both images with brain tumors and normal images. the database is well-balanced, containing 1500 images depicting brain tumors and an equal number of 1500 normal images. the available interpretation categories for this database are “no” (without a brain tumor) and “yes” (with the presence of a brain tumor). the brain tumor classification (mri) (sartaj) [9] database from the kaggle platform was employed to assess the generalization of the models. this database consists of 2870 images for training and 394 for testing, divided into four classes: without tumor, meningioma, pituitary tumor, and glioma. among the test folder images, there are 105 images without tumors, 115 with meningioma, 100 with glioma, and 74 with pituitary tumors. since the research paper focuses on binary classification (with and without tumor), without classifying the tumor types, the authors decided to conduct an experiment using the meningioma class and the no-tumor class due to their balanced distribution. as a result, only 220 images were selected for this experiment: 105 files representing “no tumor” and 115 files representing “meningioma tumor”. data preprocessing and data augmentation. the images were resized to (224, 224) for the first model and (299,299) for the xception pre-trained model. data augmentation generates more training samples by augmenting the samples through some random transformations that produce believable-looking images. the goal is for the model to see the same image only once during training. the following transformations were applied in the training of both models: randomflip(“horizontal”): this layer performs random horizontal flips on the input images. it randomly mirrors images horizontally, which helps the model learn invariant features and improve its generalizability. randomrotation(0.1): this layer applies a random rotation to the input images. parameter 0.1 indicates the maximum angle of rotation in radians. in this case, the images will be rotated by a random angle within the -0.1 to 0.1 radians range. randomzoom(0.2): this layer randomly zooms in on the input images. parameter 0.2 specifies the maximum zoom range. this means that images can be enlarged or reduced by a random factor within 0.8 to 1.2. architectures. below we present the details of the two architectures used in the experiments. basic model. the basic model was developed with a lean architecture, with three convolutional layers, three max pooling layers, a dense layer with 128 neurons, and the final layer with one neuron for the binary classification problem. a dropout of 40% was added after the flatten and dense(128) layers to avoid overfitting. xception. inspired by the assumptions of inception architecture, the xception model, which means “extreme inception,” was named because, according to the author [17], this hypothesis is a stronger version of the hypothesis underlying inception architecture. the author proposed a convolutional neural network architecture based entirely on depthwise separable convolution layers. in effect, they make the following hypothesis: mapping cross-channel and spatial correlations in the feature maps of convolutional neural networks can be entirely decoupled. the xception architecture has 36 convolutional layers forming the feature extraction base of the network. the convolutional layers are structured into 14 modules with linear residual connections around them, except for the first and last modules [17]. metrics. the statistical metrics used to evaluate the models were: accuracy, precision, recall, specificity, and receiving operational characteristics (roc curve). these metrics are often employed in similar works [11, 14]. xai techniques. the feature visualization technique and gradient-weighted class activation mapping (grad-cam) were used in the basic cnn model and the cam with lime (local interpretable model-agnostic explanation) techniques in the xception model. the general idea behind visualizing cnn layers is to identify which parts of the input image activate the different filters at each layer and how these activations are combined to form the final prediction [18]. this information can be used to understand how the network is making its predictions and identify potential improvement areas. the heatmap technique [18], denominated “class activation maps” (cams), involves taking the output of the last convolutional layer and obtaining a vector feature by applying a global average pooling operation. the vector feature is then used to compute the final prediction and the class activation map, which is obtained by weighting the output of the last convolutional layer with the corresponding weights of the final prediction (eq. 1). $$\begin{aligned} m^{(c)} = \sum _k w_k^{(c)} \phi _k(x) \end{aligned}$$ (1) where \(w_{k}^{(c)}\) is the weight of the k-th feature map in the last convolutional layer for class c, and \(\phi _k(x)\) is the activation map of the k-th feature map in the last convolutional layer for the input image x. grad-cam [19] is an xai technique used in cnns to identify the most important regions of an image that contributed to a particular class prediction. it uses the gradient information flowing into the last convolutional layer of the network to generate a heatmap that highlights these regions. grad-cam is an extension of cam that can be used with any cnn architecture and produces more accurate and finer-grained heatmaps than cam. these techniques help interpret cnn predictions and identify potential biases or limitations in the model. 3.1 training, validation, and test the br35h [8] database was used in both models for training and validation using the holdout technique (80% for training and 20% for validation), and in the first step, 10% of the entire dataset, 300 samples, was saved for testing. afterward, the models were trained for 60 epochs and compiled using the adam learning rate optimizer by default 0.001. the metrics for monitoring the convergence of the models were accuracy and binary cross-entropy for loss. in the final phase, to test the models and generate the confusion matrix, 220 images from the other database [9] were used. 4 results and discussion this section will present and discuss the performance of the models in the br35h and sartaj datasets. afterward, information regarding the xai techniques applied to the models will be presented. the hyperparameters used were selected based on previous research and empirical evaluation and were considered effective for the classification task of brain tumor images. the adam optimizer was used for both models, with a learning rate of 0.001. the adam optimizer is a widely used optimization algorithm that adapts the learning rate for each parameter during training, making it suitable for various tasks. the learning rate determines the step size in each iteration during optimization. the batch size was 32 for the base model and 64 for the xception model. batch size affects the data used to update model parameters on each iteration. larger batch size can lead to more stable convergence and increase memory requirement. finally, both models were trained for 60 epochs. the learning curves shown in fig. 3 and fig. 4 demonstrate the training and validation accuracy and loss over the 60 epochs for the basic model cnn and xception model cnn, respectively. in the case of the basic model cnn, the training accuracy gradually increases and plateaus at around 95% after about 20 epochs. the validation accuracy also increases but plateaus at around 90% after about 15 epochs. the loss decreases sharply in the first few epochs, then decreases at a slower rate until it plateaus after about 30 epochs. these results suggest that the basic model cnn can achieve high accuracy on the training data but may be overfitting after 20 epochs, as the validation accuracy is a bit lower than the training accuracy. for the xception model cnn, the training accuracy increases more rapidly than the basic model cnn and plateaus at around 98% after ten epochs. the validation accuracy also increases and plateaus at around 96% after about 15 epochs. the loss decreases more smoothly and plateaus after about 40 epochs. these results suggest that the xception model cnn can achieve even higher accuracy than the basic model cnn and is less prone to overfitting. the hyperparameters used appear to have been effectively training both models. however, the xception model cnn performs better than the basic model cnn, likely due to its more complex architecture. fig. 3. learning curves of basic model cnn full size image fig. 4. learning curves of xception model cnn full size image fig. 5. roc curve basic and xception models on br35h dataset. full size image the experimental results in table 1 indicate that the basic model and xception perform well on the br35h dataset, achieving high accuracy and auc (area under the curve) values (fig. 5). however, xception outperforms the basic precision, recall, and specificity model. specifically, xception achieves a precision of 96.71%, a recall of 98%, and a specificity of 96.67%, compared to the basic model’s precision of 96.03%, recall of 96.67%, and specificity of 96%. these results suggest that xception can better distinguish between positive and negative instances in the br35h dataset. fig. 6. roc curve basic and xception models on sartaj dataset. full size image on the other hand, the test data did not perform well in the sartaj dataset. the basic model and xception achieve moderate accuracy and auc values (fig. 6), but xception performs better in precision, recall, and specificity. xception achieves a precision of 80.15%, a recall of 94.78%, and a specificity of 74.28%, compared to the basic model’s precision of 77.60%, recall of 84.35%, and specificity of 73.3%. despite these improvements, the overall performance of the sartaj dataset is still relatively low. one hypothesis for this low performance could be due to the variability of the sartaj dataset, which may contain variations in image features. it could be due to differences in imaging capture techniques, image quality, or specific characteristics of the tumors in the dataset. the models were trained on a different dataset than the sartaj dataset and may struggle to generalize effectively. the results demonstrate the importance of evaluating a model on multiple datasets to ensure its generalization performance. although the models are promising on the br35h dataset, it has yet to show good generalizability on new images. more investigations are needed to improve its performance on the sartaj and in other datasets. table 1. test results on br35h and sartaj datasetsfull size table the following topic aims to apply the xai techniques, following the interpretable evaluation of the cnns’ performance. our investigation employed a triad of xai methodologies to extract insights regarding the cnn models utilized for brain tumor classification. the first technique was visualizing images in convolutional layers of the basic model, which showed how the input image (fig. 7) was processed in each of the three convolutional layers (fig. 8). this technique aimed to provide an understanding of how the model made its predictions and identify which parts of the input image were used by the model for classification. these images allow us to observe the features learned by the model at each layer and the increasing complexity of the representations as we move deeper into the network. the grad-cam technique is applied to the basic model’s first, second, and third convolutional layers (fig. 9). for each layer, the input image is passed through the model, and the gradient information flowing into the layer generates a heat map highlighting the most important regions of the image that contributed to the classification. the heatmap is overlaid on the input image to show the image regions that contributed to the ranking. the results show that grad-cam can provide valuable information about the model’s decision-making process. for the first convolutional layer, the heatmap highlights the edges and contours of the object in the image. the second convolutional layer highlights more complex features, such as textures and patterns. the heatmap highlights even more complex features in the third convolutional layer, such as shapes and structures. the third technique was the class activation map, applied to the xception model. figure 10 illustrates a false positive output generated by the xception model, showing its prediction with 100% confidence. the true class of the image is identified as “no tumor”, while the model predicts it as “yes tumor”. in addition to correctly interpreting classified classes, it is also essential to understand the factors contributing to false positives and negatives to improve model performance, reduce classification errors, and increase confidence in model predictions. interpretability techniques offer a unique advantage, allowing humans to study models and actively look for flaws. when flaws are exposed and understood, we can better understand the potential risks associated with these models and design safeguards to mitigate them. it could ensure that deployed systems meet transparency requirements and are more reliable. visualizing the cnn layers, the cam and grad-cam techniques provide an understanding of the inner workings of the cnn models used to classify medical images, improving interpretability and consequently making the network more transparent to human eyes. these visualization techniques help explain the model’s decision-making process and can provide insights into improving model performance. 5 conclusion this paper evaluated the performance of two cnn models, the basic model and xception, for brain tumor classification on two datasets, br35h and sartaj. the accuracy values of the models reached 96.33% and 97.33% in the test set of the database in which they were trained. the results obtained here are similar to those found in recent research [11, 12, 14, 15] and relatively higher than in the study [13]. the results also indicate that xception outperforms the basic model on both datasets. however, the overall performance of the sartaj dataset remains low, suggesting the need for further investigation to improve model generalization. using different databases in research is crucial to support robust conclusions about the generalization of a model. it helps assess a model’s ability to generalize diverse data, identify biases, increase reliability and reproducibility, and validate real-world applicability. it is a common approach that researchers in deep learning and medical imaging must follow to ensure the validity and usefulness of their discoveries. furthermore, the paper introduces explainable artificial intelligence (xai) techniques, the visualization of cnn layers, cam, and grad-cam techniques to enhance the cnn models’ interpretability. the visualization of cnn layers demonstrates the learned features by the model at each layer and the increasing complexity of the representations. these visualization techniques improve the understanding of the cnn models’ decision-making process and could provide valuable insights into enhancing model performance. xai techniques have several benefits, including increased trust and reliability of the model’s output, better communication between medical professionals and patients, and potential improvements in model performance. additionally, the visualizations generated by xai techniques can assist radiologists in identifying the most important features in medical images, leading to more accurate diagnoses. however, there are also some limitations to consider. xai techniques can be computationally expensive, requiring significant processing power and time. furthermore, the visualizations generated by these techniques can be subjective. therefore, it is crucial to perform thorough validation and testing of the models using xai techniques to ensure their generalization to new data. xai techniques have demonstrated significant potential for improving the interpretability of cnn models in medical image classification tasks. their use is likely to become increasingly important in the future. nevertheless, further research and development are necessary to overcome the limitations and challenges associated with these techniques to ensure their reliability and usefulness in real-world medical applications. fig. 7. basic model the original image input is on the left, and the right is in the first convolutional layer. full size image fig. 8. basic model visual layers technique 1st, 2nd and 3rd convolutional layers showcasing the learned features and their growing complexity within the network. full size image fig. 9. grad-cam analysis for basic model (1st, 2nd, and 3rd convolutional layers) reveals distinct insights: edges and contours (1st), complex textures and patterns (2nd) and shapes and structures(3rd). full size image fig. 10. false positive output and after cam in xception model. the image on the left shows the classification of the xception model with a 100% of confidence. true class: “no tumor”; predicted class: “yes”. the heatmap is shown in the central image. in the image on the right, the area colored is those that increase the probability that the image belongs to the yes class (tumor). full size image references câncer do sistema nervoso central. instituto nacional de câncer-inca (n.d.). https://www.gov.br/inca/pt-br/assuntos/cancer/tipos/sistema-nervoso-central brain and central nervous system cancer-iarc. (n.d.).https://www.iarc.who.int/cancer-type/brain-and-central-nervous-system-cancer/. accessed 16 may 2023 magadza, t., viriri, s.: deep learning for brain tumor segmentation: a survey of state-of-the-art. j. imaging 7(2), 19 (2021). https://doi.org/10.3390/jimaging7020019 fernando, k.r.m., tsokos, c.p.: deep and statistical learning in biomedical imaging: state of the art in 3d mri brain tumor segmentation. inf. fusion 92, 450–465 (2023). https://doi.org/10.1016/j.inffus.2022.12.013 dsa, e.: chapter 43 pooling layers in convolutional neural networks. in: deep learning book, 10 december 2022. https://www.deeplearningbook.com.br/camadas-de-pooling-em-redes-neurais-convolucionais alzheimer’s disease diagnosis using deep learning techniques. int. j. eng. adv. technol. 9(3), 874–880 (2020). https://doi.org/10.35940/ijeat.c5345.029320 lipton, z.c.: the mythos of model interpretability. commun. acm 61(10), 3643 (2018). https://doi.org/10.1145/3233231 br 35h: brain tumor detection 2020 (n.d.). www.kaggle.com, https://www.kaggle.com/datasets/ahmedhamada0/brain-tumor-detection brain tumor classification (mri) (n.d.). www.kaggle.com, https://www.kaggle.com/datasets/sartajbhuvaji/brain-tumor-classification-mri doshi-velez, f., kim, b.:towards a rigorous science of interpretable machine learning. arxiv:1702.08608 [cs, stat]. https://arxiv.org/abs/1702.08608 (2017) maqsood, s., damaševičius, r., maskeliūnas, r.: multi-modal brain tumor detection using deep neural network and multiclass svm. medicina 58(8), 1090 (2022). https://doi.org/10.3390/medicina58081090 gómez-guzmán, m.a., et al.: classifying brain tumors on magnetic resonance imaging by using convolutional neural networks. electronics 12(4), 955 (2023). https://doi.org/10.3390/electronics12040955 gaur, l., bhandari, m., razdan, t., mallik, s., zhao, z.: explanation-driven deep learning model for prediction of brain tumour status using mri image data. front. genet. 13. https://doi.org/10.3389/fgene.2022.822666 marmolejo-saucedo, j.a., kose, u.: numerical grad-cam based explainable convolutional neural network for brain tumor diagnosis. mob. netw. appl. (2022). https://doi.org/10.1007/s11036-022-02021-6 article  google scholar  islam, md.a., et al.:a low parametric cnn based solution to efficiently detect brain tumor cells from ultrasound scans (2023). https://doi.org/10.1109/ccwc57344.2023.10099302 westphal, e., seitz, h.: a machine learning method for defect detection and visualization in selective laser sintering based on convolutional neural networks. additive manuf. 41, 101965 (2021). https://doi.org/10.1016/j.addma.2021.101965 chollet, f.: xception: deep learning with depthwise separable convolutions (2017). arxiv:1610.02357 [cs]. https://arxiv.org/abs/1610.02357v3 chollet, f.: deep learning with python. manning publications, new york (2017) google scholar  selvaraju, r.r., cogswell, m., das, a., vedantam, r., parikh, d., batra, d.: grad-cam: visual explanations from deep networks via gradient-based localization. int. j. comput. vision 128(2), 336–359 (2020). https://doi.org/10.1007/s11263-019-01228-7 download references author information authors and affiliations federal university of health sciences of porto alegre, porto alegre, rs, 90050-170, brazil dieine estela bernieri schiavon, carla diniz lopes becker, viviane rodrigues botelho & thatiane alves pianoski authors dieine estela bernieri schiavonview author publications search author on:pubmed google scholar carla diniz lopes beckerview author publications search author on:pubmed google scholar viviane rodrigues botelhoview author publications search author on:pubmed google scholar thatiane alves pianoskiview author publications search author on:pubmed google scholar corresponding author correspondence to dieine estela bernieri schiavon . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper schiavon, d.e.b., becker, c.d.l., botelho, v.r., pianoski, t.a. (2023). interpreting convolutional neural networks for brain tumor classification: an explainable artificial intelligence approach. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_6 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords brain tumor convolutional neural networks xai techniques publish with us policies and ethics profiles dieine estela bernieri schiavon view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature using complex networks to improve legal text hierarchical classification | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper using complex networks to improve legal text hierarchical classification conference paper first online: 30 january 2025 pp 476–490 cite this conference paper download book pdf download book epub intelligent systems (bracis 2024) using complex networks to improve legal text hierarchical classification download book pdf download book epub rilder s. pires9,10, raquel silveira11, carlos g. o. fernandes10,12, joão a. monteiro neto13 & … vasco furtado9,10,14  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 375 accesses 1 citation abstract legal topics are typically organized into trees of labels, where each branch, from the root (generic topics) to the leaf (specific topics), categorizes the vocabulary that describe lawsuits. in this article, we describe an approach, innovative in the brazilian justice system, of automatic hierarchical classification of a petition topic. the approach integrates methods based on transformers and complex networks to capture, in addition to the characteristics of the text, the relationship between the legal citations present in the petitions and the topic to which the petition refers. the validation of this approach is done through a benchmark that shows accuracy gains, as well as, a practical scenario with the implementation of a microservice on a national justice platform whose front-end implementation is already being used by a state court to automatically suggest the classification of the petitions topic in the national procedural system. download conference paper pdf similar content being viewed by others using topic modeling in classification of brazilian lawsuits chapter © 2022 towards automated classification of repetitive themes in brazilian courts with legalclass chapter © 2024 using attention methods to predict judicial outcomes article 27 december 2022 1 introduction classification of the topic addressed by a legal document is a task where artificial intelligence, specifically through natural language processing (nlp), can provide automatic inference capabilities [21]. legal topics are typically organized into hierarchical trees, where each branch, from the root (e.g., consumer law) to the leaf (e.g., moral/material damage), categorizes the vocabulary that describes lawsuits. topic classification generally occurs in the early stages of the judicial process when a petition is submitted to the court. at this point, the petitioner is required to indicate the relevant subject matter of the demand. in the brazilian context, a petitioner must choose the topic from a hierarchy encompassing over 4,000 subjects, which are part of the unified procedural tables (tpu system) maintained by the national council of justice (cnj) [9]. making the correct association within this hierarchy is not a trivial task and is often done incorrectly by the petitioner. this misclassification generates delays in the judicial process, leading to negative impacts both financially, due to rework, and socially, by fostering a sense of impunity. the scientific literature presents various methodologies for the automatic classification of judicial documents [1, 2, 4, 14, 21, 29]. the advent of textual classification via deep learning, particularly with transformer models, has significantly advanced the field of text classification [10, 17, 28, 31]. nonetheless, the intricate and extensive hierarchical structure of topics, along with the specialized nature of legal texts, poses significant challenges to traditional methods. taking a closer look at our problem, we see that it is a hierarchical text classification problem, where we typically aim to categorize a text into a set of labels organized in a structured class hierarchy. the big challenge with this type of task is to adequately model the label hierarchy, which is usually large-scale, unbalanced, and highly structured [34]. these challenges have motivated us to develop an innovative approach. our approach assumes that the numerous legal citations present within petitions are crucial for understanding the document’s content and defining its relevant subject matter. these legal citations not only help formalize the arguments presented by both petitioners and decision-makers but also serve as a bridge connecting the petition to the applicable legal provisions, such as laws, decrees, precedents, and articles. by examining the citations within each petition, we can construct a network linking legal provisions to the petition topics. this network offers supplementary information to enhance the automatic classification process, thereby significantly improving its accuracy. using this network-based approach, we developed a classifier based on bert that leverages not only the petition text but also vector information generated from the network connections between topics and legal provisions. we validated our methodology with a dataset comprising 300,000 petitions from various brazilian courts, maintained by the cnj’s codex system. the results demonstrate the validity of our approach, showcasing significant gains in accuracy, in some situations, when compared to a hierarchical classification method that do not use the network information. additionally, we show a example of application of the proposed approach in a version of a hierarchical classification model that is available on the sinapse platform of the council and is currently being used by a state court to automatically suggest topic classifications for petitions entered in the national procedural system. 2 related work the modeling of complex networks based on the characteristics of a lawsuit has its potential evidenced in several works as described in [5, 11, 13, 24, 25, 27]. classification of legal documents has been used in the forensic and law enforcement context (e.g. identifying potential criminals [32], predicting crimes [22, 23] or discovering antisocial behavior [7]). mariana et al. [21] investigated, in the brazilian context, different methods of classification and data representation to conclude that deep learning (more specifically) with word2vec trained on a domain-specific corpus, are promising, but still limited, methods for classifying many classes. luz de araujo et al. [4] presented a new dataset constructed from legal documents from the federal supreme court of brazil, containing labeled text data for two types of tasks: document type classification and assignment of themes. the authors explored and compared different methods for classification, such as bag-of-words models, cnn, rnn, boosting algorithms, and linear-chain conditional random fields. cnn with word embedding also proved to be more performing. silva et al. [30] proposed a structured model in a cnn to classify types of documents received by the federal supreme court of brazil. aguiar et al. [1] investigated the application of topic modeling to identify the subject of legal documents and evaluated the applicability in classifying brazilian legal processes. the classification model was trained on documents from the court of justice of ceará, in brazil, and the cases were classified into the five most representative classes of the national council of justice (cnj) of brazil. recently, pre-trained models based on bert has shown to be more performing to classify topics. importantly, the effectiveness of bert is mainly due to the transfer learning ability that leverages semantic and syntactic knowledge from pre-training on a large non-labeled corpus [10] aguiar et al. [2] investigate different text classification methods and different combinations of embeddings, extracted from portuguese language models, and information about legislation cited in the initial documents to classification task of lawsuits. the models were trained with a golden collection from the court of justice of the state of ceará, in brazil, whose lawsuits were classified in the five more representative cnj’s classes. the best result was obtained by the bert model. shaheen et al. [29] studied the performance of several transformer-based models (namely bert [10], roberta [17], distilbert [28] and xlnet [36]) in combination with strategies such as pre-generative training, gradual unfreezing and discriminating learning rates to achieve competitive ranking performance. chalkidis et al. [6] developed an english legal judgment prediction dataset, containing cases from the european court of human rights. a wide variety of neural models are evaluated on this dataset, establishing strong baselines that outperform previous feature-based models. wang et al. [34] presented the hierarchy-aware prompt tuning (hpt) method, aimed at addressing hierarchical text classification from the perspective of a masked language model with multiple labels. the method involves creating a dynamic virtual template and using label words that act as soft prompts, integrating knowledge of the label hierarchy. additionally, a zero-bounded multi-label cross-entropy loss is introduced to align the processes of hierarchical text classification with masked language models. the authors conducted experiments that demonstrated hpt achieves a good performance on several popular hierarchical text classification datasets. 3 the proposed approach in the proposed approach, we assume that the primary source of data is a set of textual petitions p that make up the initial piece of the judicial case. each initial petition is associated with a legal topic that the case addresses. in the text of these petitions, there are references to legal provisions that support the arguments put forward by the petitioner. our hypothesis is that the relationship of a legal topic with one or more legal provisions is relevant for automatically classifying the matter of a petition. figure 1 shows the pipeline of the proposed approach, which depends on the texts of the petitions and the citations to legislation made within them. additional background knowledge about the hierarchical structure of legal topics and provisions is used. this additional information and the citations of legal provisions within the text form a complex network of topics and legal provisions that we will model here through a weighted graph. then, the embedding of vertices in this graph allows us to generate a vector representation of the petitions. next sections will provide formalization of these concepts. fig. 1. pipeline of the proposed approach. in the upper left corner are the primary source of data namely the petitions and the legal citations made within them as well as background knowledge about the hierarchical structure of legal topics and provisions. this data is transformed in a weighted graph of legal topics and provisions that allows to generate a vector representation of the petitions. full size image 3.1 bipartite graph of legal provisions and topics given a set of petitions p, we define \(g_b\) as the graph that represents the co-occurrence relationship between the legal topics of the petitions of p and the respective legal provisions cited therein. here, we define \(g_b=(a,d,e_b,w_b)\), where \(a = \{a_1,a_2,\ldots ,a_n\}\), \(n\in \mathbb {n}\), is a finite set of vertices related to legal topics. \(d = \{d_1,d_2,\ldots ,d_m\}\), \(m\in \mathbb {n}\), is a finite set of vertices related to the legal provisions \(e_b \subset a \times d\) is a finite set of edges connecting subsets of vertices of a and d. \(w_b: e_b \rightarrow \mathbb {r}\) is a edge weight function defined on the edges of \(g_b\). in this graph, there is an edge between the vertices \(a_i\) and \(d_j\) if there is at least one petition in p whose legal topic corresponds to \(a_i\) and which makes at least one reference to legal provision \(d_j\). generally speaking, we assume that the weight \({w_b}_{i,j}\) of this edge will be a function of the number of times \(q_{i,j}\) that the legal provisions \(d_j\) is referenced by the petitions of the topic \(a_i \) to p. the \(g_b\) graph bears similarities with other graphs already studied in the literature, such as the bibliographic citations graph [19]. there, the vertices represent documents that make references to other documents. an important characteristic of this type of graph is the existence of documents with a high amount of citations [19]. these vertices with many links end up making it difficult to identify the clustering patterns existing in the graph [3]. similarly to what happens in bipartite graphs of users and objects, we define a weight function that uses a strategy based on a popular technique in information retrieval [3] called tfidf, $$\begin{aligned} {w_b}_{i,j} \equiv {w_b}(a_i,d_j)= f(a_i,d_j)\times \log \left( \frac{n}{\text {deg}(d_j)} \right) \end{aligned}$$ (1) where \(f(a_i,d_j)\) is similar to “tf” and represents the frequency that the legal provision \(d_j\) is cited in petitions of a certain topic \(a_i\), $$\begin{aligned} f(a_i,d_j) = \frac{q_{i,j}}{\sum _k^{\langle a_i \rangle } q_{i,k}}, \end{aligned}$$ and \(\langle \cdot \rangle \) means that the sum in k is done at the neighborhood of \(a_i\). the term \(\log \left( n/\text {deg}(d_j)\right) \) of eq. (1) is similar to “idf”, measuring how much the legal provision is rare or common among all legal topics. 3.2 forests of legal topics and legal provisions legal topics have taxonomic hierarchical relationships and are predefined in the unified procedural tables [9] system. the graph \(g_a = (a,e_a)\) represents the taxonomy of legal topics, where: \(a = \{a_1,a_2,\ldots ,a_n\}\), \(n\in \mathbb {n}\), is a finite set of vertices related to legal topics. \(e_a \subset a \times a\) is a finite set of directed edges that connect the vertices of \(g_a\). the intrinsic asymmetry of the relationships between the vertices of \(g_a\) makes it fall into the class of directed rooted forests. such graphs have important properties from a topological point of view. in these graphs, we can define two types of special vertices, the root vertices and the leaf vertices. considering that the connections between two vertices always occur from the most general legal subject to the most specific subject, a root vertex is defined as a vertex that only has connections “going” from it. complementary, a leaf vertex can be defined as a vertex that only has links “entering” it. from the point of view of legal topics, we can interpret leaf vertices as the most specific topics existing in \(g_a\) and root vertices as the most generic ones in \(g_a\). furthermore, the topological structure of directed rooted trees allows us to define for each leaf vertex \(a_i\) a unique corresponding root vertex \(\text {root}(a_i)\), that is, for each specific subject a unique corresponding generic subject. similarly to what happens with topics, legal provisions have hierarchical relationships that allow us to define a graph \(g_d = (d,e_d)\), where \(d = \{d_1,d_2,\ldots ,d_n\}\), \(m\in \mathbb {n}\), is a finite set of vertices describing legal provisions. \(e_d \subset d \times d\) is a finite set of directed edges that connect the vertices of \(g_d\) representing the part-of relationship between the pieces that make up a law. although semantically different from the previously described taxonomy, the asymmetry of the relationships between the vertices of \(g_d\) also classifies it as a directed rooted forest. legal provisions are hierarchically organized as a partonomy, where the vertices of the root type represent complete legal provisions, such as laws, decrees and precedents that are composed of parts such as sections, articles, paragraphs and items. for each leaf vertex \(d_i\) there is, also, a single corresponding root vertex \(\text {root}(d_i)\). 3.3 weighted graph of legal topics and provisions given a set of initial petitions p, we can define a bipartite graph of topics and legal provisions \(g_b=(a,d,e_b,w)\) as shown in sect. 3.1. furthermore, the background knowledge of legal topics allows us to build a forest of legal topics \(g_a = (a,e_a)\) as shown in sect. 3.2. thus, we can define a new bipartite graph \(g_b^\prime = (a_\text {leaf},d,e_b^\prime ,w_b^\prime )\) as the subgraph of \(g_b\), where: \(a_\text {leaf}\) is the subset of vertices that are of leaf type in \(g_a\) representing specific topics. d is the set of vertices of \(g_b\) representing the legal provisions. \(e_b^\prime \) is the subset of edges of \(g_b\) where \(w_{i,j}\ge \tau \). \(w_b^\prime : e^\prime \rightarrow \mathbb {r}\) is an edge weighting function defined in \(e^\prime \) as \({w_b^\prime }_{i ,j}={w_b}_{i,j}\). here, \(\tau \) corresponds to a threshold of tfidf below which we consider that an edge does not justify being created in \(g_b^\prime \). it is important to note that, unlike to what happens in \(g_b\),, in \(g_b^\prime \) there are only vertices related to legal topics of leaf type. now, we can define the weighted graph of legal topics and provisions \(g=(v,e,w)\), through the junction of \(g_b^\prime \) and the undirected versions of the forests \(g_a^\prime =( a,e_a^\prime )\) and \(g_d^\prime =(d,e_d^\prime )\), where: \(v = a\cup d = \{v_1,v_2,\ldots ,v_{n+m}\}\) is the set of vertices of \(g^{\prime }\), defined as the union of the vertices related to topics and legal provisions. \(e = e_b^\prime \cup e_a^\prime \cup e_d^\prime \) is the set of edges connecting the vertices in v. \(w: e \rightarrow \mathbb {r}\) is an edge weighting function defined in e as: $$ w_{i,j}=\left\{ \begin{matrix} {w_b^\prime }(v_i,v_j) & \text {case edge } (v_i,v_j) \in e_b^\prime \\ \tau & \text {otherwise}\\ \end{matrix}\right. $$ in order to the condition \((v_i,v_j) \in e_b^\prime \) to be satisfied, necessarily, the conditions \(v_i \in a\) and \(v_j \in d\) must also be satisfied, since \(g_b^\prime \) is a bipartite graph. as defined before, the graph g is a simple weighted graph. this allows us to generate vector representations for the vertices of this graph, as we will see in the next subsection. 3.4 generating embeddings in this section, we describe the process of generating the vector representation of legal citations from weighted graph of topics and legal provisions and the embeddings of petitions that, in turn, use these representations. generating vertices’ embeddings of g. in this step, we use the graph presented in the previous section to generate the embeddings of its vertices. for this, we define the mapping function $$\begin{aligned} g_{[g]} : v \rightarrow \mathbb {r}^d, \end{aligned}$$ which associates to each vertex \(v_i \in v\) a vector \(\textbf{v}_i\) of d dimensions. the intuition of this approach is that, in this space, the vector \(\textbf{v}_i\) will be close to vertices belonging to the neighborhood of \(v_i\). then, legal topics will be “placed” close to legal provisions that occur concurrently in petitions and/or next to more general topics in the topic hierarchy. the citation embeddings aforementioned can be seen as an alternative way, other than the natural language text, of representing the arguments used by the petitioner. in this sense, legal provisions are cited to represent and support the matter of a petition and are relevant to determine the legal topic the petition refers to. in order to generate the embedding for g, we used the node2vec+ algorithm implemented in pecanpy [16] library, which was chosen for its ability to capture conformal communities [15]. generation of a petition’s embeddings. from the vector representation of each legal citation made in a given petition, the petition’s embedding can be calculated. the representation \(\textbf{p}_j\) for a petition from the legal provisions it cites is defined as $$ \textbf{p}_j = \frac{\sum ^{\varomega _j}_k \textbf{v}_k}{l(\varomega _j)}, $$ where the sum is performed on the set \(\varomega _j\) of all representations corresponding to the legal provisions mentioned in the petition of the process j and \(l(\varomega _j)\) is the total number of elements in \(\varomega _j\). 3.5 the classifier the petition set p is the main input of the classifier, as illustrated in the architecture of our model in fig. 2. the text of the initial petition goes through a textual standardization phase with the objective of standardizing legal terms (for example, art. 1 of law no. 11,419 of december 19, 2006 to art. 1 of law 11,419/2006), removing excess spaces, repeated lines and special characters and extending acronyms of agencies and states of the federation. next, the legal provisions cited in the petition are identified using regular expressions and then we proceed as described in the previous section in order to generate the petition embedding. the automatic classification is done according to a hierarchy-aware prompt tuning architecture [34]. at a high level, the standardized text of the application is sent to the bert module, whose architecture is structured in a multi-layered bidirectional transformer encoder [10]. the input text is encoded in a sequence of tokens generated using wordpiece embeddings [35]. the textual features are then passed to a stack of encoder layers, after augmentation with positional encoding. positional embeddings are added to each layer of the transformer’s encoder to help the transformer learn dependencies between tokens. after applying the transformer coding layers, the bert module results in a linear pooling layer. the combination module receives as input the text resources emitted from the transformer bert model and the numerical resources coming from the “vector representation of legal citations”. from there the module generates a combined multimodal representation. we use the combination module proposed by [34], which receives as input the textual features generated from a transformers model and numerical features and generates a combined representation, through different methods that combine the different representations in their respective spaces of features. to incorporate the label hierarchy, we propose a layer-wise prompt (according to [34]). since the label hierarchy is a tree, we construct templates based on the depth of the label hierarchy. the hierarchy constraint only introduces depth of labels but lacks their connectivity. to make full use of the label hierarchy in an mlm-manner, we further inject the per-layer label hierarchy knowledge into template embedding. a k-layer stacked graph attention network (gat) [12] is adopted to model the label hierarchy. since hierarchical text classification is a multi-label classification problem, in this paper, instead of calculating the score of each label separately, we expect the scores of all target labels to be greater than those of all non-target labels. we use a multilabel cross entropy (mlce) loss [33], introducing an anchor label with a constant score of 0 in mlce and hope that the scores of the target labels and non-target labels are all greater and less than 0 respectively. thus, we form a zero-bounded multi-label cross entropy (zmlce) loss. to be consistent with the hierarchy constraint, we adopt zmlce at each label hierarchy layer for the layer-wise prediction. fig. 2. architecture of our model. the architecture is divided into two stages: (i) data preprocessing: the text of the initial petition is pre-processed to standardize legal terms and to identify the articles and laws cited in the text, and (ii) classification: combines the input data (text and bi-partite citation topic-net) and uses the architecture of hierarchy-aware prompt tuning (hpt) [32] during training. hpt transforms hierarchical text classification (htc) into a hierarchy-aware multi-label masked language model (mlm) problem, incorporing the label hierarchy knowledge. hpt transforms htc into a multi-label mlm task with a zero-bounded multi-label cross entropy loss. full size image 4 the classifier in use the performance evaluation of the model was made from its application in real data of 300,000 petitions of legal processes coming from labor courts, federal regional courts and courts of justice of the brazilian states. 4.1 the petition dataset as already stated in sect. 3.2, legal topics are categorized according to a specific table of topics called unified procedural table tpu [9] developed by the brazilian national council of justice cnj. the hierarchy present in these tables allows us to aggregate specific topics into generic ones. we used this strategy to separate our dataset into very distinct topic groups, which allowed us to test our approach at different levels of complexity. in table 1, we show the percentage distribution of initial petitions for each generic subject in the dataset of 300,000 petitions. table 1. percentage distribution of lawsuits by legal topics. the codes used in this table are defined by the tpu and are the same ones that will be used in the results section.full size table the set of 300,000 petitions was then divided following a ratio of 80% for training and 20% for testing. we took care that the dataset only contained petitions with at least 50 words and that more than 80% of valid words were validfootnote 1. 4.2 training process the graph g was created from the legal provisions identified in the set of petitions p of the training data. then, the vector representation of the petition citations was generated as described in sect. 3.4. the value \(\tau \approx 8.48\cdot 10^{-3}\) was empirically adjusted by identifying the value that best separated the topics from \(g_b\). this was done by analyzing the assortativity coefficient [20] of the one-mode projection [19] relative to the legal topic partition. a fine-tuning of the pre-trained bertimbau [31] model was done, where all parameters were tuned using the specific topics of the petitions used in the training. in the mesun_etal_2020rge module, we use the gated summation merge method in the transformer output for the petition text and the vector representation of the citation, before the final classifier layerfootnote 2. the model was trained with batch size of 16 examples, 30 epochs max and a early stopping of 5 epochs. adam optimizer was chosen with learning rate of \(10^{-5}\) and 5-fold cross-validation. random oversampling was applied in order to equalize number of examples of each class. 4.3 comparative evaluation and deployment for a detailed evaluation of our approach, we compare our model (descibed in sect. 3.5) with the original hierarchical classification method [34] (that uses only text as input). the results for each specific topic were calculated in terms of precision, recall and f1-score, obtained from the actual and predicted labels on the test dataset. in table 2, we show the results of the classifiers for the different generic topics shown in the previous section. the results in this table show that the classifier that uses information from the citations performs better in some situations. precisely, it shows better results in “864 labor law”, “899 civil right” and “1209 criminal procedural law” generic topics. we believe that, in these cases, our approach is able to better capture the relationships that exist between legal provisions and specific topics due to the existence of legal provisions that detail several specific situations in these branches of law. an example of an application where this approach is very useful occurs when a petitioner only informs a generic topic when submitting the petition. in this case, a meta-model would be capable of identifying which approach is most suitable for classifying the specific subject of that petition based on the generic topic informed. indeed, we incorporated a version of a hierarchical classification model into the cnj’s sinapses platform [8] in the form of a micro-service. this platform aims to share, among brazilian judicial institutions, models of artificial intelligence developed by the public and private sector. in fig. 3, we shows the interface of the judicial process registration system, pje, after the implementation of the leaf topic prediction functionality with the call of the classification micro-service implemented in the sinapses platform. table 2. precision, recall and f1-score results for different generic topics. here, n is the number of specific topics present in each generic topic. the metrics show the comparison between the classifier that uses only the text of the petitions and the one that uses the text and the vector representation defined by our approach.full size table fig. 3. screenshot of pje system with the feature of predicting the topic of a petition. the screenshot is numbered to show: 1) is the main topic informed by the user, 2) is the predicted specific topic and 3) is the text of the petition. full size image 5 conclusion citations to existing legal provisions in lawsuits is an important particularity of this type of document. they add additional information to the text and can be used by automatic natural language processing methods for a variety of tasks. in this article we describe how the automatic topic classification of a legal petition can be enhanced based on these citations. the approach integrates transformer-based methods with a complex network in order to capture, in addition to text features, the relationship between legal citations that are present in the petitions and the topic to which the petition refers to. this research work characterizes itself by the proposition of a novel method based on the combination of deep learning and complex networks, and in the application of these ideas in the real world. the implementation of a micro-service into the justice national platform as well as the implementation of a front-end that uses this service constitutes a novel tool to be used country wide by different institutions of the brazilian justice. it is expected bringing productivity augmentation, reduction of costs and more agility to the judicial sentences. future work involves improving accuracy of the method via fine-tuning of parameters and techniques used to manipulate the network. for instance, investigations of alternative methods for graph embedding generation for capturing temporal relations may improve the final results. also, monitoring the use of the classifier by the different institutions may shed lights on the user experience and the real benefits the approach brings to the citizen. notes 1.the verification of valid words was performed using the lexicons proposed by [18]. 2.this merge method is inspired by [26]. references aguiar, a., silveira, r., furtado, v., pinheiro, v., neto, j.a.m.: using topic modeling in classification of brazilian lawsuits. in: pinheiro, v., et al. (eds.) propor 2022. lncs (lnai), vol. 13208, pp. 233–242. springer, cham (2022). https://doi.org/10.1007/978-3-030-98305-5_22 chapter  math  google scholar  aguiar, a., silveira, r., pinheiro, v., furtado, v., neto, j.a.: text classification in legal documents extracted from lawsuits in brazilian courts. in: anais da x brazilian conference on intelligent systems. sbc, porto alegre, rs, brasil (2021). https://sol.sbc.org.br/index.php/bracis/article/view/19093 alupoaie, s., cunningham, p.: using tf-idf as an edge weighting scheme in user-object bipartite networks. arxiv preprint arxiv:1308.6118 (2013) luz de araujo, p.h., de campos, t.e., ataides braz, f., correia da silva, n.: victor: a dataset for brazilian legal documents classification. in: proceedings of the 12th language resources and evaluation conference, pp. 1449–1458. european language resources association, marseille, france (2020). https://aclanthology.org/2020.lrec-1.181 carmichael, i., wudel, j., kim, m., jushchuk, j.: examining the evolution of legal precedent through citation network analysis. ncl rev. 96, 227 (2017) google scholar  chalkidis, i., androutsopoulos, i., aletras, n.: neural legal judgment prediction in english. corr abs/1906.02059 (2019). http://arxiv.org/abs/1906.02059 cheng, j., danescu-niculescu-mizil, c., leskovec, j.: antisocial behavior in online discussion communities. corr abs/1504.00680 (2015). http://arxiv.org/abs/1504.00680 cnj: sinapses (2022). https://sinapses.ia.pje.jus.br/. accessed 9 aug 2022 cnj: sistema de gestão de tabelas processuais unificadas (2022). https://www.cnj.jus.br/sgt/consulta_publica_assuntos.php. accessed 9 aug 2022 devlin, j., chang, m., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: burstein, j., doran, c., solorio, t. (eds.) proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, naacl-hlt 2019, minneapolis, mn, usa, 2–7 june 2019, volume 1 (long and short papers), pp. 4171–4186. association for computational linguistics (2019). https://doi.org/10.18653/v1/n19-1423 fowler, j.h., johnson, t.r., spriggs, j.f., jeon, s., wahlbeck, p.j.: network analysis and the law: measuring the legal importance of precedents at the us supreme court. polit. anal. 15(3), 324–346 (2007) article  google scholar  kipf, t.n., welling, m.: semisupervised classification with graph convolutional networks. in: 5th international conference on learning representations (iclr) (2017) google scholar  koniaris, m., anagnostopoulos, i., vassiliou, y.: network analysis in the legal domain: a complex model for european union legal sources. j. complex netw. 6(2), 243–268 (2018) article  math  google scholar  limsopatham, n.: effectively leveraging bert for legal document classification. in: proceedings of the natural legal language processing workshop 2021, pp. 210–216. association for computational linguistics, punta cana, dominican republic (2021). https://doi.org/10.18653/v1/2021.nllp-1.22. https://aclanthology.org/2021.nllp-1.22 liu, r., hirn, m., krishnan, a.: accurately modeling biased random walks on weighted graphs using node2vec+. arxiv preprint arxiv:2109.08031 (2021) liu, r., krishnan, a.: pecanpy: a fast, efficient and parallelized python implementation of node2vec. bioinformatics 37(19), 3377–3379 (2021) google scholar  liu, y., et al.: roberta: a robustly optimized bert pretraining approach. corr abs/1907.11692 (2019). http://arxiv.org/abs/1907.11692 muniz, m.: a construcão de recursos linguistico-computacionais para o português do brasil: o projeto unitex-pb. dissertação (mestrado em ciências de computação e matemática computacional) instituto de ciências matemáticas e de computação, universidade de são paulo (2004). https://doi.org/10.11606/d.55.2020.tde-19022020-151305 newman, m.: networks. oxford university press (2018) google scholar  newman, m.e.: mixing patterns in networks. phys. rev. e 67(2), 026126 (2003) google scholar  noguti, m.y., vellasques, e., oliveira, l.s.: legal document classification: an application to law area prediction of petitions to public prosecution service. in: 2020 international joint conference on neural networks (ijcnn), pp. 1–8 (2020). https://doi.org/10.1109/ijcnn48605.2020.9207211 pérez-rosas, v., mihalcea, r.: experiments in open domain deception detection. in: proceedings of the 2015 conference on empirical methods in natural language processing, pp. 1120–1125. association for computational linguistics, lisbon, portugal (2015). https://doi.org/10.18653/v1/d15-1133. https://aclanthology.org/d15-1133 pinheiro, v., pequeno, t., furtado, v., nogueira, d.: information extraction from text based on semantic inferentialism. in: andreasen, t., yager, r.r., bulskov, h., christiansen, h., larsen, h.l. (eds.) fqas 2009. lncs (lnai), vol. 5822, pp. 333–344. springer, heidelberg (2009). https://doi.org/10.1007/978-3-642-04957-6_29 chapter  math  google scholar  pires, r.s., oliveira, e.a., almeida, v.f., neto, j.a.m., furtado, v.: análise da ontologia dos assuntos jurídicos e suas respectivas legislações através de redes complexas. in: anais do xi brazilian workshop on social network analysis and mining, pp. 181–191. sbc (2022) google scholar  pires, r.s., oliveira, e.a., fernandes, c.g., neto, j.a.m., furtado, v.: mapping landmark cases in the us legal system. ceur workshop proceedings (2021) google scholar  rahman, w., et al.: integrating multimodal information in large pretrained transformers. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 2359–2369. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.acl-main.214. https://aclanthology.org/2020.acl-main.214 ruhl, j., katz, d.m., bommarito, m.j.: harnessing legal complexity. science 355(6332), 1377–1378 (2017) article  google scholar  sanh, v., debut, l., chaumond, j., wolf, t.: distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. corr abs/1910.01108 (2019). http://arxiv.org/abs/1910.01108 shaheen, z., wohlgenannt, g., filtz, e.: large scale legal text classification using transformer models. corr abs/2010.12871 (2020). https://arxiv.org/abs/2010.12871 silva, n., braz, f., de campos, t.: document type classification for brazil’s supreme court using a convolutional neural network. in: 10th international conference on forensic computer science and cyber law (icofcs), sao paulo, brazil, pp. 7–11 (2018). https://doi.org/10.5769/c2018001 souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  sumner, c., byers, a., boochever, r., park, g.j.: predicting dark triad personality traits from twitter usage and a linguistic analysis of tweets. in: 2012 11th international conference on machine learning and applications, vol. 2, pp. 386–393 (2012). https://doi.org/10.1109/icmla.2012.218 sun, y., et al.: circle loss: a unified perspective of pair similarity optimization. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition (2020) google scholar  wang, z., et al.: hpt: hierarchy-aware prompt tuning for hierarchical text classification. arxiv preprint arxiv:2204.13413 (2022) wu, y., et al.: google’s neural machine translation system: bridging the gap between human and machine translation. corr abs/1609.08144 (2016). http://arxiv.org/abs/1609.08144 yang, z., dai, z., yang, y., carbonell, j.g., salakhutdinov, r., le, q.v.: xlnet: generalized autoregressive pretraining for language understanding. corr abs/1906.08237 (2019). http://arxiv.org/abs/1906.08237 download references acknowledgments we gratefully acknowledge cnpq, capes, funcap, bnb, and the edson queiroz foundation for financial support. author information authors and affiliations laboratório de ciência de dados e inteligência artificial, universidade de fortaleza, fortaleza, ceará, brazil rilder s. pires & vasco furtado programa de pós graduação em informática aplicada, universidade de fortaleza, fortaleza, ceará, brazil rilder s. pires, carlos g. o. fernandes & vasco furtado instituto federal de educação, ciência e tecnologia do ceará, tianguá, ceará, brazil raquel silveira bnb banco do nordeste do brasil s.a., fortaleza, ceará, brazil carlos g. o. fernandes centro de ciências jurídicas, universidade de fortaleza, fortaleza, ceará, brazil joão a. monteiro neto etice empresa de tecnologia da informação do ceará, fortaleza, ceará, brazil vasco furtado authors rilder s. piresview author publications search author on:pubmed google scholar raquel silveiraview author publications search author on:pubmed google scholar carlos g. o. fernandesview author publications search author on:pubmed google scholar joão a. monteiro netoview author publications search author on:pubmed google scholar vasco furtadoview author publications search author on:pubmed google scholar corresponding author correspondence to rilder s. pires . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pires, r.s., silveira, r., fernandes, c.g.o., monteiro neto, j.a., furtado, v. (2025). using complex networks to improve legal text hierarchical classification. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_33 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_33 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords text classification brazilian legal system complex networks publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature traffic forecasting using federated randomized high-order fuzzy cognitive maps | springer nature link (formerly springerlink) skip to main content advertisement menu find a journal publish with us track your research search cart home intelligent systems conference paper traffic forecasting using federated randomized high-order fuzzy cognitive maps conference paper first online: 30 january 2025 pp 445–459 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) traffic forecasting using federated randomized high-order fuzzy cognitive maps download book pdf download book epub omid orang  orcid: orcid.org/0000-0002-4077-377511, felipe a. r. da silva  orcid: orcid.org/0000-0003-4567-85049, petrônio c. l. silva  orcid: orcid.org/0000-0002-1202-255210,11, pedro h. s. s. barros  orcid: orcid.org/0000-0001-6606-013511, heitor s. ramos  orcid: orcid.org/0000-0003-4523-646611 & … frederico g. guimarães  orcid: orcid.org/0000-0001-9238-883911  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 384 accesses 2 citations abstract numerous machine learning (ml) and deep learning (dl) forecasting techniques have emerged for predicting traffic flow and traffic speed. despite their notable achievements, they encounter challenges related to preserving in-vehicle user privacy. alternatively, federated learning (fl) offers a decentralized ml strategy to strike a balance between prediction accuracy and privacy preservation. this paper proposes a novel fuzzy-based method called fl-rhfcm, which integrates the principles of randomized high-order fuzzy cognitive maps (r-hfcm) with fl. r-hfcm is akin to an echo state network (esn), where the only trainable component is the output layer using least squares (ls) minimization. the reservoir layer’s weights are initialized randomly and remain unchanged during training. thus, the ls coefficients represent the only parameters shared with the server for aggregation in our fl-rhfcm approach. fl-rhfcm introduces an efficient distributed forecasting method using fcms, significantly reducing communication costs. the effectiveness of our proposed model is assessed using two datasets, demonstrating its promise compared to some existing baseline methods. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others automated forecasting model leveraging large reservoirs of randomized fuzzy cognitive maps article 11 november 2025 dual-layer fl and blockchain empowered high accurate edge training framework chapter © 2024 federated learning analysis for vehicular traffic flow prediction: evaluation of learning algorithms and aggregation approaches article 14 january 2024 1 introduction intelligent transportation systems (its) have recently gained attention due to rising road safety and efficiency concerns. traffic prediction is crucial for route planning, optimizing vehicle dispatching, controlling traffic congestion, etc. [1, 2]. traffic prediction (travel time, flows, speeds, occupancy, and demand) utilizes a trainable function to analyze past traffic data and forecast future traffic conditions. it relies on two primary types of data: traffic flow, indicating total detected vehicles over a period, and traffic speed, representing average vehicle velocity in the same area during the same time frame [3]. this study will refer to traffic flow and speed as traffic. forecasting traffic presents a formidable challenge given the non-linear, time-dependent, and spatio-temporal properties of traffic time series [1, 4]. consequently, developing an accurate traffic prediction model is imperative. accordingly, a wide range of traffic forecasting approaches have been presented in the literature, which can be broadly categorized into conventional parametric statistical models and non-parametric machine learning (ml) and deep learning (dl)-based methods [1, 5]. autoregressive integrated moving average (arima) models and their variants, including seasonal arima (sarima) and vector arima (varima), are commonly used statistical techniques in traffic prediction, as discussed in [5, 6]. however, these models encounter challenges due to the complexity, non-stationarity, and non-linearity of traffic data. additionally, arima models generally necessitate a substantial volume of historical data to perform effectively, which may not always be accessible [5, 7]. to tackle these challenges, ml and dl methods are extensively utilized in traffic prediction tasks owing to their capacity to automatically extract crucial features from historical traffic data, thus eliminating the need for complex mathematical model design. convolutional neural networks (cnn), recurrent neural networks (rnn), and their variants, such as long short-term memory (lstm) and gated recurrent unit (gru), have shown remarkable performance [5, 8]. in addition, graph neural networks (gnns) [4] are considered state-of-the-art methods particularly well-suited for traffic forecasting challenges due to their capacity to capture spatial dependencies. as discussed in [4], several types of gnns have been developed, including graph recurrent neural networks (grnn) [9], graph-structured recurrent neural networks (gsrnn) [10], and graph lstm (glstm) [11], among others. attention mechanisms and transformers represent another category of dl techniques that have proven effective in the field of traffic prediction [5, 12, 13]. despite the considerable success and frequent usage of dl methods, they suffer from limitations such as interpretability, parsimony, and high computational costs in terms of time and resources. fuzzy time series (fts) forecasting methods offer an alternative due to their simplicity, interpretability, updatability, scalability, and capacity to handle uncertainty and complex systems [14]. fts is a methodology for time series forecasting (tsf) that involves converting the numerical time series into a linguistic time series representation using fuzzy sets. then, the transitions between the fuzzy sets in the historical data are used to find rules for forecasting. fuzzy cognitive maps (fcms), a subset of weighted fts methods, represent a specialized category in fuzzy modeling and forecasting techniques [15]. fcms, as interpretable ml models, have a good capacity for dealing with uncertainty and effectively simulating the dynamic behavior of non-linear and complex systems [15, 16]. few studies among the proposed fts methods have utilized fcm for traffic prediction. examples include interpretable deep attention fcm in [17], interpretable deep fcm (dfcm) in [18], fcm learned with an evolutionary algorithm (fcmevol) in [19]. from another perspective, existing centralized ml and dl traffic prediction methods necessitate collecting raw data for model training, leading to significant privacy risks. dl algorithms typically require vehicles to transmit raw data, including sensitive information like location, to a central server for training the proposed models in a centralized manner [20]. if the central server is compromised, the entire forecasting system is vulnerable to a single point-of-failure attack, risking severe privacy breaches for vehicles. additionally, heavy reliance on extensive data for centralized training increases communication overhead and the risk of data leakage and pollution [21]. to address these problems, federated learning (fl) [22], which shares model updates without exchanging raw data, has recently been introduced as an efficient solution [23]. fl performs local learning model calculations and then sends the local learning model parameters to the central server for global model aggregation. accordingly, it can avoid direct interaction with original data and achieve the trade-off between model performance, communication overhead, and data privacy. to the best of our knowledge, only two investigations have explored the use of fl and fcms. the authors in [24] implemented fl methods for fcms in medical applications, specifically for classifying dengue in colombian cities. a blind federated learning approach without an initial model was introduced in [25] proposing two innovative methodologies for pso-based fl of fcms to classify breast cancer and demographic features related to adult income. consequently, there are no references regarding federated fcms in forecasting applications. thus, the focal contribution of this study is to fill this gap by presenting a novel univariate forecasting method termed fl-rhfcm for the first time in the literature. fl-rhfcm is a hybrid method integrating fl and randomized high-order fcm (r-hfcm) to predict traffic flow and traffic speed. r-hfcm [26] is a new class of fcms combining fts, fcm, and echo-state network (esn), trained via least squares (ls). it functions as an esn with input, reservoir, and output layers. the reservoir contains parallel sub-reservoirs (l) with unalterable weights, a unique feature that sets r-hfcm apart from typical fcm-based methods. ls is then applied to train the output layer and determine the ls coefficients (\(\lambda _i\)). therefore, in fl-rhfcm, each user is trained locally to obtain the optimal values of \(\lambda _i\). afterward, these values are shared with the server to be aggregated using the average method to update the global model. finally, clients are updated locally, receiving updated global model parameters from the aggregator. this process is executed for 15 rounds using the flower platform. the rest of this paper is structured as follows: sect. 2 shares a brief review of traffic prediction; sect. 3 provides a comprehensive presentation of the proposed approach; sect. 4 details the experimental results and discussion; and lastly, sect. 6 encapsulates the paper’s findings and delineates potential directions for future research. 2 literature review plenty of univariate and multivariate traffic forecasting techniques have been developed in the literature [3, 27], which can be grouped into three main classes, including statistical methods, traditional ml, and dl methods. according to [27, 28], statistical methods, particularly arima and its variants such as sarima, sarimax, kohenen arima (karima), and vector arima, have been frequently employed to predict traffic. these methods are generally suitable for simpler and less complex datasets but require a substantial volume of historical data to perform effectively. this dual requirement means they face limitations when dealing with non-stationary, non-linear, spatio-temporal, and complex datasets. ml models excel in generalization and adaptability to changing traffic network conditions compared to statistical methods [27, 28]. they are typically divided into three categories: feature-based methods, gaussian process models, and state-space models, capable of handling non-linear and complex time series data [29]. they have pros and cons, as explained in [27]. artificial neural networks (ann) [30], k-nearest neighbor (knn) [31], and support vector regression (svr) are some examples of this category [32]. dl methods are extensively used in traffic prediction tasks because of their strong capacity to capture stochastic and nonlinear relationships in traffic data [27]. based on this reference, multi-layer perceptrons (mlp), autoencoder (ae), cnn, rnn, lstm, gnns, restricted boltzmann machines (rbms), deep belief networks (dbn), graph convolution network (gcn), wavelet neural network (wnn) and attention-based models have been employed to predict traffic. also, the authors in [3] reviewed 37 dl traffic forecasting models to predict spatial and/or temporal traffic flow and speed, confirming various types of dnn methods. for instance, in [33], dbn, k-means clustering, and dempster-shafer theory are utilized for traffic flow prediction, while in [34], cnn with pearson correlation-based theory is employed to predict traffic speed. graph neural networks (gnns) with dl are another traffic forecasting method elaborated in [4], including models like graph lstm (glstm), graph multi-attention network (gman), graph attention temporal convolutional network (gatcn), among others. the next sub-group of dnn forecasting methods are hybrid methods such as lstm and bidirectional lstm [35], encoder-decoder lstm and fnn-based attention module [36], encoder-decoder gru, and graph diffusion [37], encoder-decoder gru, fnn and graph attention network [38], among others. fuzzy-based forecasting models are the last category of traffic forecasting techniques that can solve some limitations regarding dl methods regarding interpretability, training time, scalability, and complexity. a new fuzzy-based cnn method in [39], an evolving fuzzy neural network (efnn) in [40], and an adaptive hybrid fuzzy rule-based system in [41] are some studies in this group. also, some researchers combined fuzzy and dl such as fuzzy deep convolution network (fdcn) in [42], spatiotemporal fuzzy-graph convolutional network model in [43], and a combination of fuzzy logic, lstm, and decision trees (dts) in [44]. despite accurately forecasting centralized methods, data privacy poses significant challenges. to address this, some researchers have utilized decentralized fl methods to make a tradeoff between prediction accuracy and privacy preservation. an integration of fl and gru in [45], a combination of fl and attention-based spatial-temporal gnn (astgnn) in [46], graph attention networks (gat), lstm, and fl in [21], clustering-based hierarchical and two-step-optimized fl in [47], federated community gcn (fcgcn) in [48], and fl with asynchronous gcn in [49] are some examples of fl-based models for traffic flow and speed forecasting. fcms, as a neuro-fuzzy method, have shown considerable success in capturing the dynamics of various complex systems and effectively handling uncertainties [26]. despite this, there is a notable gap in the literature regarding using decentralized fcms for predictive applications, such as traffic forecasting. to address this, our research introduces a novel federated fcm forecasting model in this field. fig. 1. generic structure of the proposed r-hfcm technique full size image 3 proposed fl-rhfcm method as explained in [15], fcms are composed of a set of concepts and signed directed connections, known as weights. therefore, weight matrices form the core element of each fcm, and various training methods have been developed to optimize both the weights and structure. this research introduces a novel hybrid univariate forecasting method called fl-rhfcm, a fusion of r-hfcm and fl, to predict traffic flow and speed. r-hfcm [26] is a centralized fcm-based forecasting method with a different structure from regular fcms. accordingly, this section is divided into two subsections. section 3.1 details r-hfcm, and sect. 3.2 provides information regarding our proposed fl-rhfm technique. 3.1 centralized r-hfcm method r-hfcm is a hybrid method combining the concepts of fts, fcms, and echo state networks (esn) [26]. more precisely, as shown in fig. 1, r-hfcm consists of three layers: the input layer, reservoir, and output layer. the reservoir layer is composed of a specific number of sub-reservoirs (l). each sub-reservoir employs the hfcm-fts method [50], with weights randomly initialized following the esn approach and kept constant throughout training. then, output from each sub-reservoir, generated from the defuzzification step, is fed to the output layer. finally, ls is employed to train the output layer and identify the optimal values of ls coefficients (\(\lambda _i\)). from another point of view, r-hfcm can be seen as an esn (or reservoir computing) method such that only the output layer is trainable. in r-hfcm, the ls coefficients (\(\lambda _i\)) serve as the sole trainable parameters. thus, r-hfcm is not focused on training weights among the concepts. this property distinguishes r-hfcm from other fcm-based methods and makes it much faster than methods trained via population-based techniques such as genetic algorithms (ga) or particle swarm optimization (pso). figure 2 a simple topology of the r-hfcm method with \(l=2\), \(k=5\) (concepts), and \(\varomega =2\) (order). in this case, ls trains the output layer to find ls coefficients. the number of coefficients is three (\(\lambda _0,\lambda _1\) and \(\lambda _2\)) which means that the number of ls coefficients directly depends on the number of sub-reservoirs. as such, the number of ls coefficients is equal to \(l+1\). the details of training and forecasting procedures are described in detail as follows: fig. 2. a simple example r-hfcm model with l = 2, \(\varomega = 2\) and k = 5. full size image a. training process 1) weight and bias initialization: the weights among the concepts \(c_i\) and \(c_j\) are randomly chosen from a uniform distribution in [−1,1] and scaled inspiring by esn weight initialization using: (1) where \(\mathbf {\rho _{max}}(\mathbf {w^{rand}})\) is the largest absolute eigenvalue of \(\mathbf {w^{rand}}\), and \(\mathbf {\epsilon } = 0.5\). the bias vector \(\mathbf {w^0}\) is initialized similarly: (2) where \(\textbf{s}\) is the maximum singular value of \(\mathbf {w_{rand}^0}\). 2) partitioning: firstly, the universe of discourse (uod) is determined using the formula \(uod = [\min (y_i)-d_1, \max (y_i)+d_2]\), where \(d_1 = \min (y_i)\times 0.2\) and \(d_2 = \max (y_i)\times 0.2\). then, uod is partitioned into k even-length intervals (representing concepts of fcms) using grid partitioning and the triangular membership function. 3) fuzzification: in this step, the activation state \(a_i(t)\) of each concept \(c_i \in c\), \(\forall y(t) \in y\), is computed to convert the crisp time series y into a fuzzy series a. each fuzzified sample \(a(t) \in a\) is given by \(a_i(t) = \mu _{c_i}(y(t))\), \(\forall i\in \{1, \ldots , k\}\). 4) activation: the state value of each concept within each sub-reservoir at time t+1 is updated based on the following formula: $$\begin{aligned} \textbf{a}_j(t+1) = f\left( \textbf{w}^0+ \sum _{l=1}^\varomega \textbf{w}^{l} \cdot \textbf{a}(t-l+1) \right) \end{aligned}$$ (3) 5) defuzzification: the output from each sub-reservoir is generated through this process using the below equation: $$\begin{aligned} \hat{y}_j(t+1)=\dfrac{\sum _{i=1}^k a_{ji}(t+1) \cdot mp_{i}}{\sum _{i=1}^k a_{ji}(t+1)} \end{aligned}$$ (4) where \(a_j(t+1)\) is the activation state of each concept at time \(t+1\) and \(mp_i\) represents the center of each concept \(c_i\). 6) calculating ls coefficients: from the outputs \(y_j(t+1)\) \(\forall j \in \{1,\ldots ,l\}\), and for each input sample \(y(t)\in y\), \(t=1\ldots t\), a design matrix x is formed for the linear system \(y = \lambda x\). the ls method is then used to find the coefficient vector \(\lambda \) to minimize the mean squared error. b. forecasting process 1) fuzzification: same as the third stage of the training process. 2) activation: same as the fourth stage of the training process. 3) defuzzification: first, the defuzzified value of each sub-reservoir is calculated using eq. 4. then, the linear combination of \(\hat{y}_j(t+1)\) and \(\lambda _j\) is considered to generate the final predicted value as follows: $$\begin{aligned} \hat{y}_{f}(t+1)=\lambda _0+\sum _{j=1}^{l} \lambda _{j}\cdot \hat{y}_{j}(t+1) \end{aligned}$$ (5) 3.2 decentralized r-hfcm method this section introduces fl-rhfcm as a decentralized adaptation of the r-hfcm technique, wherein fl is incorporated into r-hfcm to enhance data privacy. thus, in fl-rhfcm, data is not shared among the users, and each user is trained using the local dataset. figure 3 exhibits our proposed fl-rhfcm approach. given m users, denoted as \(\{u_1, u_2,\dots , u_m\}\), each associated with its local data denoted by \(\{y_1, y_2, \dots , y_m\} \). in fl-rhfcm, the uod for each client is calculated using \(uod = [\min (y_i)-d_1, \max (y_i)+d_2]\), as expressed earlier. fig. 3. the generic architecture of fl-rhfcm, considering m clients with m different datasets full size image in fig. 3, \(a_i\) and \(b_i\) respectively represent \(\min {(y_i)}\) and \(\max {(y_i)}\), \(\forall i \in m\). furthermore, \(w_0\), w, and \(\lambda _{avg}\) represent bias, weight, and average ls coefficients, respectively. as mentioned earlier in sect. 3.1, the only trainable parameters in the r-hfcm method are ls coefficients. accordingly, each local node is in charge of training a local model and sending the local model to the server node for aggregation. more specifically, the steps of our proposed federated method are expressed as follows: step 1 the server initializes \(w_0\), w, a, b, and \(\lambda \), then transmits them to the clients. besides, \(w_0\) and w remain fixed and are the same during all executions. step 2 each client (\(u_i\)) is trained using local time series (\(y_i\)), calculating \(a_i\), \(b_i\) and \(\lambda _i \in \{\lambda _{0i},\lambda _{1i},\ldots ,\lambda _{li}\}\), \(\forall l \in l \). step 3 the obtained \(a_i\), \(b_i\) and \(\lambda _i\) are transferred to the server for aggregation. the server updates a, b, and \(\lambda \) as follows: \( a=\min _{i=1}^{m} a_i\), \( b=\max _{i=1}^{m} b_i\) and \(\lambda =\frac{1}{m} \sum _{i=1}^{m} \lambda _i\). step 4 the updated values are shared with the users. the performance of the proposed model for each user is calculated in terms of accuracy metrics. step 5 this process is repeated in 15 rounds. it is worth noting that we deployed fl-rhfcm with three client nodes, each trained using distinct time series datasets, as explained in the following section. 4 computational experiments 4.1 case studies two different traffic datasets are exploited to validate our approach: 1. traffic flow (hourly)footnote 1: this dataset collected from sensors contains 48,120 observations of the number of vehicles in four junctions; 2. traffic speed (minutely)footnote 2: contains 6 road segments of the xueyuan road in beijing, china. fig. 4. plotting of 8,000 samples from three junctions (j1, j2, and j3) full size image table 1. summary statistics of the 6 datasets used in the experimentsfull size table fig. 5. plotting of 8,000 samples from three road segments (r3, r4, and r5) full size image this study employs datasets from three junctions (j1, j2, and j3) and three segments (r3, r4, and r5). detailed information about these time series is reported in table 1. in addition, figs. 4 and 5 display traffic flow and traffic speed time series, respectively. 4.2 experimental methodology root mean squared error (rmse) and normalized rmse (nrmse) are employed to evaluate the model’s accuracy. rmse is calculated using $$ rmse = \sqrt{ \frac{1}{n} \sum _{i=1}^n (y_i \hat{y}_i)^2 }, $$ where \(y_i\) represents the actual values, \(\hat{y}_i\) represents the predicted values, and n is the total number of samples. nrmse is then obtained by normalizing rmse as follows: $$ nrmse = \frac{rmse}{y_{max}-y_{min}}. $$ these metrics provide insights into the model’s performance. it is noteworthy that 80% of each dataset is designated for training, while the remaining 20% is for testing. the model’s python code is publicly available for replication via the provided link: https://github.com/omidufmg2019/fl-rhfcm-model. 5 results and discussion this section analyzes the accuracy of fl-rhfcm in comparison to various centralized methods, including r-hfcm, pwfts, lstm, cnn, cnn-lstm, and arima. as we mentioned, fl-rhfcm includes three client nodes. two scenarios are considered: (i) in the first scenario, each traffic speed time series (r3, r4, and r5) is directed to each node, and (ii) in the second scenario, each node receives its traffic flow time series (j1, j2, and j3). to obtain the optimized performance of the fl-hfcm technique, multiple experiments are conducted, exploring various combinations of hyperparameters (hps) including \(k \in \{3,4,\ldots ,10,20,30\}\), \(l \in \{2,3,\ldots ,10,20,40\}\), \(\varomega \in \{2,3,\ldots ,10\}\), and \(f \in \{sigmoid,tanh, relu\} \)). in scenario 1, the best performance is achieved with \(l=20\), \(k=3\), \(\varomega =5\), and \(f=tanh\), whereas \(l=8\), \(k=3\), \(\varomega =5\), and \(f=tanh\) yield the best result in scenario 2. it is worth observing that the randomized grid search is utilized to find the best hps for competing methods. table 2 showcases the experimental results of all methods to compare the performance of fl-rhfcm with other centralized techniques. the evaluation metrics used are rmse and nrmse, where the top result for each dataset is highlighted in bold and the second-best is underscored. table 2. comparison of the fl-rhfcm method with other centralized modelsfull size table the recorded results in table 2 suggest that fl-rhfcm performs better predicting traffic speed than forecasting traffic flow. in more detail, fl-rhfcm outperforms other competing methods with r3 and j3 datasets, while centralized r-hfcm is superior for r4, r5, and j2. to this end, employing the average nrmse facilitates a meticulous comparison of the accuracy of the methods. according to the table, r-hfcm demonstrates the most accurate predictor. although pwfts, cnn-lstm, and lstm outperform fl-rhfcm, their performances only marginally exceed that of fl-rhfcm. furthermore, fl-rhfcm maintains data privacy and security, as data is not shared among clients, unlike centralized methods. moreover, this method is simpler, has faster training speed, and incurs reduced communication costs compared to deep learning models. also, fig. 6 indicates that fl-rhfcm converges for each client after the second round, demonstrating its efficiency in reaching stable predictions quickly. thus, fl-rhfcm’s competitive performance and additional benefits make it a valuable method for spatio-temporal forecasting in federated learning environments. the combination of accuracy, privacy, and efficiency positions fl-rhfcm as a noteworthy approach in the landscape of forecasting methodologies. 5.1 limitations 1. the number of clients involved in the experiments is relatively low. 2. it is not compared with complex state-of-the-art techniques such as gnns in this area. 3. while federated experiments were conducted to evaluate our proposed approach, there needs to be more comparison with other federated learning approaches in the literature. fig. 6. the nrmse accuracy of the fl-rhfcm method for each client per round. full size image 6 conclusion and future works this study introduces the first proposal of a distributed framework for fcm-based forecasting methods called fl-rhfcm. fl-rhfcm integrates fl and centralized r-hfcm to predict both traffic speed and traffic flow. the experimental results illustrate that fl-rhfcm is a robust and efficient forecasting method, especially advantageous in scenarios requiring data privacy and decentralized learning. while centralized r-hfcm demonstrates the highest overall accuracy, fl-rhfcm’s competitive performance and additional benefits make it a valuable method in spatio-temporal time series forecasting scenarios. since traffic datasets are high-dimensional and non-stationary time series, future work will extend fl-rhfcm to predict multivariate time series, considering these aspects. notes 1.https://www.kaggle.com/datasets/fedesoriano/traffic-prediction-dataset/. 2.https://github.com/buaapercy/traffic-dataset-of-beijing-road. references boukerche, a., tao, y., sun, p.: artificial intelligence-based vehicular traffic flow prediction methods for supporting intelligent transportation systems. comput. netw. 182, 107484 (2020) google scholar  cunha, f., et al.: vehicular networks to intelligent transportation systems, pp. 297–315. springer singapore, singapore (2018) google scholar  tedjopurnomo, d.a., bao, z., zheng, b., choudhury, f.m., qin, a.k.: a survey on modern deep neural network for traffic prediction: trends, methods and challenges. ieee trans. knowl. data eng. 34(4), 1544–1561 (2020) google scholar  jiang, w., luo, j.: graph neural network for traffic forecasting: a survey. expert syst. appl. 207, 117921 (2022) google scholar  wang, z., sun, p., hu, y., boukerche, a.: a novel hybrid method for achieving accurate and timeliness vehicular traffic flow prediction in road networks. comput. commun. (2023) google scholar  boukerche, a., tao, y., sun, p.: artificial intelligence-based vehicular traffic flow prediction methods for supporting intelligent transportation systems. comput. netw. 182, 107484 (2020) google scholar  yin, x., wu, g., wei, j., shen, y., qi, h., yin, b.: a comprehensive survey on traffic prediction, arxiv, abs/ arxiv: 2004.08555 (2020) almeida, a., brás, s., oliveira, i., sargento, s.: vehicular traffic flow prediction using deployed traffic counters in a city. futur. gener. comput. syst. 128, 429–442 (2022) article  google scholar  wang, x., chen, c., min, y., he, j., yang, b., zhang, y.l: efficient metropolitan traffic prediction based on graph recurrent neural network, arxiv preprint, arxiv: 1811.00740 (2018) wang, b., luo, x., zhang, f., yuan, b., bertozzi, a.l., brantingham, p.j.: graph-based deep modeling and real time forecasting of sparse spatio-temporal data, arxiv preprint arxiv: 1804.00684 (2018) lu, z., lv, w., cao, y., xie, z., peng, h., du, b.: lstm variants meet graph neural networks for road speed prediction. neurocomputing 400, 34–45 (2020) article  math  google scholar  cai, l., janowicz, k., mai, g., yan, b., zhu, r.: traffic transformer: capturing the continuity and periodicity of time series for traffic forecasting. trans. gis 24(3), 736–755 (2020) article  google scholar  shi, x., qi, h., shen, y., wu, g., yin, b.: a spatial-temporal attention approach for traffic prediction. ieee trans. intell. transp. syst. 22(8), 4909–4918 (2020) article  math  google scholar  lucas, p.o., orang, o., silva, p.c., mendes, e.m., guimaraes, f.g.: a tutorial on fuzzy time series forecasting models: recent advances and challenges. learn nonlinear models 19, 29–50 (2022) article  google scholar  orang, o., silva, p.c.d.l., guimarães, f.g.: time series forecasting using fuzzy cognitive maps: a survey. artifi. intell. rev. 1–62 (2022) google scholar  kosko, b.: fuzzy cognitive maps. inter. j. man-mach. stud. 24(1), 65–75 (1986). http://www.sciencedirect.com/science/article/pii/s0020737386800402 qin, d., peng, z., wu, l.: deep attention fuzzy cognitive maps for interpretable multivariate time series prediction. knowl.-based syst. 275, 110700 (2023) google scholar  wang, j., wang, x., li, c., wu, j., et al.: deep fuzzy cognitive maps for interpretable multivariate time series prediction. ieee trans. fuzzy syst. (2020) google scholar  chmiel, w., szwed, p.: learning fuzzy cognitive map for traffic prediction using an evolutionary algorithm. in: dziech, a., leszczuk, m., baran, r. (eds.) mcss 2015. ccis, vol. 566, pp. 195–209. springer, cham (2015). https://doi.org/10.1007/978-3-319-26404-2_16 chapter  math  google scholar  lv, y., duan, y., kang, w., li, z., wang, f.-y.: traffic flow prediction with big data: a deep learning approach. ieee trans. intell. transp. syst. 16(2), 865–873 (2015) math  google scholar  huang, h., hu, z., wang, y., lu, z., wen, x., fu, b.: train a central traffic prediction model using local data: a spatio-temporal network based on federated learning. eng. appl. artif. intell. 125, 106612 (2023) google scholar  konecný, j., mcmahan, h., yu, f., richtárik, p., suresh, a., bacon, d.: federated learning: strategies for improving communication efficiency. corr, abs/ arxiv: 1610.05492 (2016) qi, y., hossain, m.s., nie, j., li, x.: privacy-preserving blockchain-based federated learning for traffic flow prediction. futur. gener. comput. syst. 117, 328–337 (2021) article  math  google scholar  hoyos, w., aguilar, j., toro, m.: federated learning approaches for fuzzy cognitive maps to support clinical decision-making in dengue. eng. appl. artif. intell. 123, 106371 (2023) google scholar  salmeron, j., arévalo, i.: blind federated learning without initial model. j. big data 11, 04 (2024) article  math  google scholar  orang, o., de lima e silva, p.c., silva, r., guimarães, f.g.: randomized high order fuzzy cognitive maps as reservoir computing models: a first introduction and applications. neurocomputing (2022) google scholar  shaygan, m., meese, c., li, w., zhao, x.g., nejad, m.: traffic prediction using artificial intelligence: review of recent advances and emerging opportunities. transportation res. part c: emerging technol. 145, 103921 (2022) google scholar  yang, x., zou, y., tang, j., liang, j., ijaz, m.: evaluation of short-term freeway speed prediction based on periodic analysis using statistical models and machine learning models. j. adv. trans. 2020, 1–16 (2020) math  google scholar  yin, x., wu, g., wei, j., shen, y., qi, h., yin, b.: deep learning on traffic prediction: methods, analysis, and future directions. ieee trans. intell. transp. syst. 23(6), 4927–4943 (2021) article  math  google scholar  dougherty, m.s., kirby, h.r., boyle, r.d.: the use of neural networks to recognise and predict traffic congestion. traffic eng. control 34(6), 311–314 (1993) math  google scholar  zhang, l., liu, q., yang, w., wei, n., dong, d.: an improved k-nearest neighbor model for short-term traffic flow prediction. procedia. soc. behav. sci. 96, 653–662 (2013) article  math  google scholar  castro-neto, m., jeong, y.-s., jeong, m.-k., han, l.d.: online-svr for short-term traffic flow prediction under typical and atypical traffic conditions. expert syst. appli. 36(3), part 2, 6164–6173 (2009) google scholar  soua, r., koesdwiady, a., karray, f.: big-data-generated traffic flow prediction using deep learning and dempster-shafer theory. in: 2016 international joint conference on neural networks (ijcnn), pp. 3195–3202. ieee (2016) google scholar  wang, j., gu, q., wu, j., liu, g., xiong, z.: traffic speed prediction and congestion source exploration: a deep learning method. in: 2016 ieee 16th international conference on data mining (icdm), pp. 499–508. ieee (2016) google scholar  cui, z., ke, r., pu, z., wang, y.: deep bidirectional and unidirectional lstm recurrent neural network for network-wide traffic speed prediction, arxiv preprint, arxiv: 1801.02143 (2018) he, z., chow, c.-y., zhang, j.-d.: stann: a spatio-temporal attentive neural network for traffic prediction. ieee access 7, 4795–4806 (2018) article  math  google scholar  li, y., yu, r., shahabi, c., liu, y.: diffusion convolutional recurrent neural network: data-driven traffic forecasting, arxiv preprint, arxiv: 1707.01926 (2017) pan, z., liang, y., wang, w., yu, y., zheng, y., zhang, j.: urban traffic prediction from spatio-temporal data using deep meta learning. in: proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining, pp. 1720–1730 (2019) google scholar  an, j., fu, l., hu, m., chen, w., zhan, j.: a novel fuzzy-based convolutional neural network method to traffic flow prediction with uncertain traffic accident information. ieee access 7, 20 708–20 722 (2019) google scholar  tang, j., liu, f., zou, y., zhang, w., wang, y.: an improved fuzzy neural network for traffic speed prediction considering periodic characteristic. ieee trans. intell. transp. syst. 18(9), 2340–2350 (2017) article  math  google scholar  dimitriou, l., tsekeris, t., stathopoulos, a.: adaptive hybrid fuzzy rule-based system approach for modeling and predicting urban traffic flow. trans. res. part c: emerging technol. 16(5), 554–573 (2008) article  math  google scholar  chen, w., an, j., li, r., fu, l., xie, g., bhuiyan, m.z.a., li, k.: a novel fuzzy deep-learning approach to traffic flow prediction with uncertain spatial-temporal data features. futur. gener. comput. syst. 89, 78–88 (2018) article  math  google scholar  zhang, s., chen, y., zhang, w.: spatiotemporal fuzzy-graph convolutional network model with dynamic feature encoding for traffic forecasting. knowl.-based syst. 231, 107403 (2021) google scholar  alkheder, s., alomair, a.: urban traffic prediction using metrological data with fuzzy logic, long short-term memory (lstm), and decision trees (dts). nat. hazards 111, 03 (2022) article  google scholar  liu, y., yu, j.j.q., kang, j., niyato, d., zhang, s.: privacy-preserving traffic flow prediction: a federated learning approach. ieee internet things j. 7(8), 7751–7763 (2020) article  math  google scholar  zhang, c., zhang, s., yu, j.j.q., yu, s.: fastgnn: a topological information protected federated learning approach for traffic speed forecasting. ieee trans. industr. inf. 17(12), 8464–8474 (2021) article  math  google scholar  zhang, c., cui, l., yu, s., james, j.: a communication-efficient federated learning scheme for iot-based traffic forecasting. ieee internet things j. 9(14), 11918–11931 (2021) article  math  google scholar  xia, m., jin, d., chen, j.: short-term traffic flow prediction based on graph convolutional networks and federated learning. ieee trans. intell. transp. syst. 24(1), 1191–1203 (2023) article  math  google scholar  qi, t., chen, l., li, g., li, y., wang, c.: fedagcn: a traffic flow prediction framework based on federated learning and asynchronous graph convolutional network. appl. soft comput. 138, 110175 (2023) google scholar  orang, o., silva, r., de lima e silva, p.c., guimarães, f.g.: solar energy forecasting with fuzzy time series using high-order fuzzy cognitive maps. in: 2020 ieee international conference on fuzzy systems (fuzz-ieee), pp. 1–8 (2020) google scholar  download references acknowledgment this work was partially funded by grant apq-00426-22, fundação de amparo a pesquisa do estado de minas gerais (fapemig), grant 2023/00721-1, são paulo research foundation (fapesp), and grant 312682/2021-2, conselho nacional de desenvolvimento científico e tecnológico (cnpq). author information authors and affiliations graduate program in electrical engineering, federal university of minas gerais, belo horizonte, brazil felipe a. r. da silva federal institute of northern minas gerais, januaria, brazil petrônio c. l. silva department of computer science, federal university of minas gerais, belo horizonte, brazil omid orang, petrônio c. l. silva, pedro h. s. s. barros, heitor s. ramos & frederico g. guimarães authors omid orangview author publications search author on:pubmed google scholar felipe a. r. da silvaview author publications search author on:pubmed google scholar petrônio c. l. silvaview author publications search author on:pubmed google scholar pedro h. s. s. barrosview author publications search author on:pubmed google scholar heitor s. ramosview author publications search author on:pubmed google scholar frederico g. guimarãesview author publications search author on:pubmed google scholar corresponding author correspondence to petrônio c. l. silva . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper orang, o., da silva, f.a.r., silva, p.c.l., barros, p.h.s.s., ramos, h.s., guimarães, f.g. (2025). traffic forecasting using federated randomized high-order fuzzy cognitive maps. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_31 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_31 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords traffic prediction federated learning randomized high-order fuzzy cognitive maps echo state network least squares publish with us policies and ethics profiles petrônio c. l. silva view author profile pedro h. s. s. barros view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an analysis of time-frequency consistency in human activity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an analysis of time-frequency consistency in human activity recognition conference paper first online: 30 january 2025 pp 66–81 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) an analysis of time-frequency consistency in human activity recognition download book pdf download book epub nícolas hecker9, otávio o. napoli9, jaime delgado9, anderson r. rocha9, levy boccato9 & … edson borin9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 385 accesses abstract this work relies upon raw data to present the time-frequency consistency (tf-c) evaluation for human activity recognition (har). the original paper utilized data for this task in the pretext stage but did not explore its application in the downstream task. an application with a modified tf-c architecture uses har data on the downstream task, reporting an accuracy of 64.08%. we propose three experiments. first, we reproduce the original experiment with the epilepsy dataset, comparing the results with the reported ones. second, we make a performance comparison test using different percentages of data from 0.1% to 100% and report the corresponding accuracy. finally, we compare the results with supervised convolutional neural networks and the supervised tf-c. this work demonstrates the feasibility of utilizing tf-c to perform har as downstream task, achieving an accuracy of 96% utilizing all data of the training dataset in fine-tuning. even with just 42 samples of the training dataset, the model achieved an accuracy of 85% and to obtain an accuracy greater than 90% it is only necessary 126 train samples. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others implementation of a convolutional recurrent fuzzy neural network model on a high-performance field-programmable gate array accelerator for real-time human activity recognition article 12 august 2025 a deep learning framework for smartphone based human activity recognition article 15 may 2023 modeling transformer architecture with attention layer for human activity recognition article 10 january 2024 1 introduction the smartphone-based human activity recognition (har) task amounts to identifying the activity performed by an individual given samples from inertial sensors, such as gyroscope and accelerometer. in recent years, approaches based on deep neural networks (dnns), including convolutional networks (cnns), have achieved good results for different datasets [4, 7, 14]. training deep models in a supervised fashion requires a significant amount of labeled data, which, in turn, depends on a labor-intensive and error-prone process of manually labelling each sample or even each time window. in this context, adopting self-supervised learning (ssl) methods to pretrain part of the network structure emerges as an interesting alternative, since it may lead to learning an effective latent representation using only unlabeled data, which are more easily collected. additionally, it may also allow the use of a relatively small labeled dataset for training the model in the target task (har, for instance), so that an adequate classifier can be obtained even with a few labeled samples. a common ssl framework involves two training steps: the pretext task and training for the downstream task. in the first step, a feature extractor, called the backbone, followed by a projection head—typically a shallow neural network—is trained to solve a supervised learning task with labels automatically derived from the original unlabeled input samples. this task, although not particularly useful in practice, is designed to encourage the model to learn meaningful data representations. the idea is that by training the structure to solve this pretext task, the backbone learns to capture general and relevant information from the input data into a latent representation, which can then be used in the downstream task. examples of pretext tasks include reconstruction tasks, jigsaw tasks, and contrastive techniques (performing augmentations and comparing similar samples), such as tf-c. in the second step, the pretrained backbone is combined with a new prediction head to create the downstream model, which is trained to solve the target task using a labeled dataset. most ssl techniques are first proposed and evaluated on image classification tasks [3, 5, 6, 9, 26]. nonetheless, several methods have also been proposed for time-series [27], such as cpc [17], tnc [23], and tf-c  [28]. tf-c is a contrastive learning technique focused on time series problems with two main characteristics: (i) the backbone is divided in two separate parts that process the raw time windows and the frequency spectra, respectively; and (ii) the pretext task encompasses three losses a contrastive time domain loss, a contrastive frequency domain loss, and a cross-domain contrastive loss. recent works analyzed tf-c in different time series scenarios, with modifications on the framework and on the architecture for backbone [8, 10, 24, 28]. however, there are few works that studied tf-c for har in the downstream task and with a cnn architecture. this work focus on tf-c for har, expanding the comparison between ssl and supervised strategies by: (i) considering the influence of the number of labeled training samples in the downstream task performance; (ii) comparing the latent spaces; and (iii) analyzing the impact of time and frequency-domain pipelines in the final performance. the obtained results indicate a similar accuracy in comparison to the state-of-the-art and an excellent class separation in latent space, even when only a few samples are employed in the training stage. the remainder of this text is organized as follows: sect. 2 presents the related work; sect. 3 provides an overview of the tf-c technique; sect. 4 discusses the experimental results; finally, sect. 5 presents the conclusions and future work. 2 related work this section presents the related work. first it provices an overview of works that employed ssl techniques to support the development of models for har. then, it lists works that evaluated tf-c on non-har tasks. finally, it discusses works that assess the performance of tf-c on har. ssl for har: several works evaluate the potential of ssl techniques on human activity recognition tasks. saeed et al.  [20] introduced an ssl technique using transformation classification as a pretext task. the model is composed of a three-layer 1d cnn backbone with multiple binary prediction heads, each identifying a single transformation such as noise addition, rescaling, rotation, negation, horizontal flipping, permutation, time warping, and channel shuffling. applied to accelerometer and gyroscope data, their approach outperforms both supervised and unsupervised methods on datasets like hhar, unimib, uci-har, wisdm, and motionsense, with improvements in performance up to 4%. haresamudram et al.  [11] proposed an ssl technique using masked autoencoders. here, n distinct time instants are randomly selected and perturbed. this input is passed to a transformer-based autoencoder that, then, go through fully connected layers aiming to reconstruct the original input, with the loss function calculated only on the perturbed part. the authors demonstrate that the model learns useful representations and achieves good results in har using the motionsense and usc-had datasets. tang et al.  [22] adapted simclr [5] for har. simclr is a popular ssl method that uses contrastive learning to learn discriminative representations by maximizing the similarity between positive pairs and minimizing it between negative pairs. positive pairs are generated from two augmentations of the same input, while negative pairs come from different inputs. in their work, the authors use eight augmentations for time-series data and reported a performance gain of over 2% with pretrained models. in a later work, tang et al.  [21] introduced selfhar, a teacher-student architecture for har that integrates semi-supervised and ssl techniques. the teacher model, a 1d cnn, generates pseudo-labels for unlabeled data, which are then used to train the student model on high-confidence samples. the student model employs a self-supervised, multi-task learning approach with transformation classification as the pretext task, similar to saeed et al.  [20]. self-supervised training utilized the fenland study data, encompassing over 12,000 participants to identify causes of obesity and type 2 diabetes. evaluated on six datasets (hhar, motionsense, mobiact, paas, unimib, uci, and wisdm), selfhar showed significant performance improvements, with a 12% gain on unimib compared to fully supervised methods, and outperformed saeed’s method. contrastive predictive coding (cpc) [17] is an ssl technique where the pretext task is to predict future information based on present context, similar to time-series forecasting tasks. it has two main components: an encoder, which maps input data to a latent representation, and an autoregressive model, which processes this latent representation to capture temporal dependencies. haresamudram et al.  [12] proposed a modified encoder architecture for cpc on har, with three one-dimensional convolutional layers, while retaining the autoregressive network as a gru with 256 states. the authors achieved strong results by using a pretrained backbone, rather than training the network from scratch, in a supervised manner. qian et al.  [18] conducted an empirical study on contrastive-based methods of ssl in har context, evaluating several properties (e.g. backbone architecture, augmentation strategies, and pair construction). comparing five ssl methods, byol, simsiam, simclr, nnclr, and ts-tcc, they made an interesting insight related to data-scarce cross-person generalization. nnclr achieved the best performance in the uci dataset, while simclr performed best with the shar dataset, highlighting that the effectiveness of ssl methods can be dataset-dependent, with nearest neighbor pair construction being particularly effective for certain datasets like uci. similarly to qian et al., in 2022 haresamudram et al.  [13] also assessed the performance of several ssl methods on har tasks. the multi-task self-supervision [20], masked reconstruction [11], cpc [12], autoencoder [20], simclr [5, 22], simsiam [6], and byol [9] techniques were evaluated under different criteria on 10 different datasets. the authors affirm the capability of these techniques for empowering har tasks. recently, yuan et al.  [25] employed a multi-task ssl approach to train a resnet backbone using the uk biobank data, a large-scale dataset with more than 700,000 person-days of data from wearable sensors. the ssl method is the same as in saaed’s work [20], however, the authors incorporated weighted sampling to improve training stability. they sampled data windows in proportion to their standard deviation, giving more weight to high-movement periods. the authors demonstrated significant performance improvements in several har datasets compared to classical algorithms and fully-supervised resnet training. tf-c: tf-c is an ssl technique that was proposed by zhang et al.  [28], in 2022. in their work, the authors evaluated the technique on a variety of time-series related tasks, such as epilepsy detection [1], sleeping disorder classification [15], and hand gesture recognition [16]. they also employed a har dataset to pretrain the tf-c backbone, however, they did not evaluate the impact of tf-c on the har task. recently, in 2024, xiao et al.  [24] employed a modified version of tf-c, known as tfcsrec, for product recommendations. this adapted version learns frequency-domain representations through a trainable frequency-domain encoder, similar to the original tf-c. however, this study does not extend its analysis to the har domain, focusing solely on recommendation systems. dong et al.  [8] introduced a another variant of the tf-c framework named simmtm, focusing on masked time-series modeling tasks. unlike traditional methods that reconstruct the original series from unmasked points, simmtm reconstructs the original series from multiple neighboring masked series. although, this study utilized tf-c and an architecture with minor modifications, retaining the convolutional structure, it did not perform pretraining and downstream tasks in the har domain. tf-c for har: guo et al.  [10] explored the tf-c method for har. the authors evaluate two training schemes: (a) pretraining in pamap2 dataset and, then, fine-tuning using the uci-har dataset, reaching an accuracy of 60.31%; and (b) pretraining in opportunity dataset and, then, fine-tuning using the uci-har dataset, reaching an accuracy of 64.04%. however, the main focus of this work is the modcl framework, similar to tf-c due to its pipeline, with contrastive learning and data augmentation. the architecture for both was a resnet. table 1 provides a summary of the work presented in this section. to the best of our knowledge, only the work of guo et al.  [10] assess the performance of tf-c on the human activity recognition task, however, as discussed previously, it does not achieve high performances with tf-c due to its backbone architecture. here we analyze the convolutional neural network as the backbone, which generates better results, and take the opportunity to discuss the quality of latent space and compares different amount of fine-tuning data. table 1. related works. tf-c: evaluated the tf-c technique. har: analyzed har as a target task. cnn: employed a cnn as the backbone.full size table 3 the time frequency consistency framework (tf-c) the time frequency consistency (tf-c) framework is a self-supervised learning approach based on contrastive learning that simultaneously explores information in both the time and frequency domains. figure 1 exhibits the main elements of tf-c and the corresponding pretext task. the backbone is composed of two encoders (\(g_t\) and \(g_f\)) and two projectors (\(r_t\) and \(r_f\)) and organized in three pipelines, one that processes the data in the time domain (top) and another that processes the data in the frequency domain (bottom), and one that projects the time and frequency embeddings into a common latent space for consistency (right). fig. 1. general scheme and elements of tf-c. full size image the time pipeline processes the original time series (\(x_i^t\)) and its augmented version (\(\tilde{x}_i^t\)) using the \(g_t\) encoder, resulting in the embeddings \(h_i^t\) and \(\tilde{h}_i^t\). meanwhile, the frequency pipeline converts the original time series into its frequency domain version (\(x_i^f\)) using a fast fourier transform (fft) algorithm. an augmented version of this frequency data (\(\tilde{x}_i^f\)) is also produced. both the original and augmented frequency data are encoded by the \(g_f\) encoder, yielding the representations \(h_i^f\) and \(\tilde{h}_i^f\). finally, the consistency pipeline (right) encode the time embeddings (i.e., \(h_i^t\) and \(\tilde{h}_i^t\)) using the time projector (\(r_t\)), producing the \(z_i^t\) and \(\tilde{z}_i^t\) embeddings, and encode the frequency embeddings (i.e., \(h_i^f\) and \(\tilde{h}_i^f\)) using the frequency projector (\(r_f\)), producing the \(z_i^f\) and \(\tilde{z}_i^f\) embeddings. the time and frequency encoders (\(g_t\) and \(g_f\)) are trained to ensure that the representations produced by the time and frequency pipelines are discriminative. additionally, the encoders and the projectors in the consistency pipeline (i.e., \(r_t\) and \(r_f\)) are trained to produce time and frequency representations that are consistent with each other. to achieve this, the tfc framework employs a loss function with three components: time loss, frequency loss, and consistency loss. the time and frequency losses ensure that the time and frequency representations of the data are discriminative and invariant to augmentations. they use contrastive loss to make the embeddings of the original and augmented versions of the data (positive pairs) similar, while ensuring embeddings from distinct samples in the batch are different. the consistency loss, also a contrastive loss, is designed to ensure that the representations produced by the projectors are consistent across the time and frequency domains. encoder’s architecture: in our experiments, we employ convolutional neural networks to implement the time (\(g_t\)) and frequency encoders (\(g_f\)). these cnns have the same architecture/layers, but with independent weights. all convolutional layers implement one-dimensional (1d) convolutions with kernel size of 8, stride of 1, padding of 4, and no bias. the first convolutional block produces 32 channels and is composed of the conv. layer, batch normalization, relu activation and max-pool 1d (kernel size of 2, stride of 2, and padding of 1). only this block includes a dropout layer with a rate of 0.35 at the output. the subsequent convolutional blocks only differ in the number of channels – 64 and 128, respectively –, and explore the same structure and hyper-parameters of the first block (except dropout). this architecture is illustrated in fig. 2a. fig. 2. architecture parts used on this work full size image projector’s architecture: the projectors (both \(r_t\) and \(r_f\)) are composed of a dense linear layer with 256 neurons, followed by batch-normalization, relu activation and another linear layer with 128 units, as indicated in fig. 2b. prediction head and downstream model: once the model components (i.e., \(g_t\), \(g_f\), \(r_t\), and \(r_f\)) are trained in the pretext task, a downstream model is build by combining with a prediction head. the prediction head is attached after the time (\(r_t\)) and frequency (\(r_f\)) projectors, receiving as input the concatenation of \(z^t\) and \(z^f\). it is composed of a linear dense layer with 64 elements, a relu activation and another linear dense layer with 6 neurons to produce the class-logits, as illustrated in fig. 2c. time-frequency consistency augmentations: in the time domain, the data is augmented by a jitter transformation with scale of 0.8. in the frequency domain some frequencies, selected when a random variable with normal distribution (mean 0.5 and standard deviation 0.5) is greater than 1. 4 experimental results this section presents the analyses of the tf-c framework. first, sect. 4.1 lists the materials and methods used in our experiments. then, sect. 4.2 presents the validation results, demonstrating that our code is competitive with the state-of-the-art. next, sect. 4.3 discusses the quantitative results, assessing the performance of the ml models under different data regimen. finally, sect. 4.4 assess the quality of the representations learned by the models. 4.1 materials and methods models: to understand the importance of each component of tf-c on har, we evaluate four models in this work: tfc: corresponds to the tf-c downstream model. it is composed of the tf-c backbone (i.e., \(g_t\), \(g_f\), \(r_t\), and \(r_f\)) and a prediction head. the backbone structures are pretrained by the tf-c pretext task and further finetuned together with the prediction head on the target (downstream) task. sup tfc: has the same architecture as the previous model (i.e., tfc), however, the backbone is not pretrained on the pretext task. it serves as a reference to measure the impact of the pretraining process. sup time: is composed of \(g_t\) concatenated with \(r_t\) and the prediction head. this model is not pretrained. in other words, this model consists of the time portion of the backbone, followed by the prediction head (which here has half the number of input features). sup frequency: is similar to sup time, but using the frequency portion of the tf-c backbone. the architecture of the sup time and sup frequency models is depicted in fig. 2d. tf-c framework modifications: we utilized the code made by zhang et al.  [28], which is publicly available on github. however, in order to conduct our experiments in har, we carried out two main modifications. first, the code employed transformer models as time (\(g_t\)) and frequency (\(g_f\)) encoders. here, we adapted the encoders in order to explore cnns, as occurs in their original paper [28]. second, the original implementation combined cross-entropy with the tf-c overall loss during the training stage for the downstream task. in our work, we simplified the downstream loss to be only the cross-entropy, since this led to similar or better results in the performed tests and fewer terms. the loss in eq. 1 was presented by tf-c authors as the fine-tune total loss, where \(\lambda \) is a hyper-parameter, \(\mathcal {l}_{\text {t}, i}\) is the temporal loss, \(\mathcal {l}_{\text {f}, i}\) is the frequency loss, \(\mathcal {l}_{\text {c}, i}\) is the consistency loss and \(\mathcal {l}_{\text {p}i}\) is a common cross entropy loss. on the other hand the loss in eq. 2 is the fine-tune loss used on this work, which is simply the \(\mathcal {l}_{\text {p}i}\). $$\begin{aligned} \mathcal {l}_{fine-original} = \lambda {(\mathcal {l}_{\text {t}, i} + \mathcal {l}_{\text {f}, i})} + {(1 \lambda )} \mathcal {l}_{\text {c}, i} + \mathcal {l}_{\text {p}i} \end{aligned}$$ (1) $$\begin{aligned} \mathcal {l}_{fine-current} = \mathcal {l}_{\text {p}i} \end{aligned}$$ (2) in summary, the changes were in the backbone and the downstream loss. datasets: we employed the sleepeeg [15], epilepsy [1], and the uci [2, 19] datasets on this work. table 2 summarizes the characteristics of these datasets, but, in particular, the uci dataset has windows of length 128 and 9 channels of the inertial signals: total acceleration, body acceleration and body gyroscope. the train set has 7352 samples and the test set has 2947 samples. the uci labels used were: walking, walking upstairs, walking downstairs, sitting, standing, and lying. table 2. dataset characteristicsfull size table hyperparameters: we used the adam optimizer with learning rate of \(3\cdot 10^{-4}\). the temperature was 0.2 and the model was trained for 40 epochs, with an early stopping based on validation loss. any other hyperparameter was the same adopted by zhang et al.  [28] 4.2 validating our implementation the first experiment mimics one of the scenarios explored by the zhang et al.  [28], and is performed in order to assess the correctness of our implementation. we pretrain the tf-c backbone using the sleepeeg dataset. once the backbone is trained, we train and evaluate the downstream model on the epilepsy dataset. finally, we select the model with the best accuracy (acc), the area under receiver operating characteristic (auroc), and area under the precision-recall curve (auprc) metrics on the test dataset. this was the process adopted for original authors, which may introduce some bias. once we need to replicate the pipeline, at this stage we use the test set, while in the next experiments the best model is selected by the loss value on the validation set and its test performance is reported. we repeat the experiment five times and report the average and standard deviation for each metric. table 3 shows the results reported by zhang et al.  [28] and our results. our reproduction of tf-c yields accuracy similar to that reported by the authors of tf-c (taking into account both uncertainties), validating our implementation. table 3. performance of tf-c on the epilepsy dataset. all values are mean and standard deviation of 5 independent runs.full size table 4.3 accuracy by percentage of har data in this experiment we evaluate the performance of tf-c on the har task. first, we pretrain the tf-c backbone using all the samples of the uci dataset, but without labels. then, we build the downstream model and fine-tune it using the uci train set. we conduct several experiments with different subsets of the train set to assess the benefits of tf-c in a low-data regime. specifically, we train the downstream model using 0.1%, 1%, 10%, and 100% of the training set. to maintain a consistent number of training iterations across experiments, we proportionally increase the number of epochs relative to the reduction in the training set size. finally, we select the best model according to the validation loss computed at the end of each epoch. this model is tested on the uci test set and the accuracy, auroc, and auprc metrics are exhibited on table 4 and fig. 3, as average of five runs. table 4. performance of the tfc and sup tfc models trained with different percentages of the training set.full size table regarding table 4, we can observe that pretrained models (tfc) have better performance when compared to the models trained from scratch (sup tfc) in low data scenarios (1% and 0.1% of the data). on the other hand, in the 10% and 100% scenarios, the difference between the models is not so significant. a similar behavior is observed in fig. 3, where we compare the accuracy of the four models evaluated in this work. we can observe that the tfc model has the best performance in almost all cases. in a low-data regime, ssl plays a crucial role in the performance of the model, as indicated by the performance differences between the tfc and the sup tfc models. despite on very low data scenarios (using 0.1% of the data) where sup frequency achieves a slightly better (but poor) performance, the pretraining seems to allow the model to achieve a better performance faster than other supervised models. in fact, only 1% of the data is necessary to achieve an accuracy above 85%, while others models need, at least, 2% of the data to reach this threshold. in the 100% scenario, the difference between the models is not so significant, which suggests that the uci training set is large enough to support the supervised training of the discussed models. fig. 3. performance of the four methods evaluated in this work with different percentages of data, in fine-tuning process representing different times necessary for training. the x-axis is in logarithmic scale. full size image a wilcoxon signed rank hypothesis test was conducted in order to analyze the influence of the ssl technique on the accuracy of the tfc, comparing it with the values of the tests of the sup tfc, sup time, and sup frequency models individually. the comparison between the 9 samples of the tf-c and the 9 samples of the sup tfc (as shown in fig. 3) yielded a p-value of 0.01953, which is below the 5% significance threshold. this result supports the alternative hypothesis, indicating an improvement with the ssl technique. similarly, when comparing the ssl technique with the temporal pipeline (sup time), a p-value of 0.01172 was obtained, further supporting the alternative hypothesis. however, when the ssl technique was compared with the frequency pipeline (sup frequency), the resulting p-value was 0.3594, indicating that we cannot conclude there is a significant difference between the results produced by the tf-c and sup frequency models. the test was conducted comparing the ssl results with complete sup tfc, separately from sup time and after sup frequency. 4.4 qualitative assessment of learned representations another perspective to consider in this analysis involves the distribution of samples in the latent spaces generated by each model. hence, we exhibit in fig. 4 the visualization in two dimensions produced by t-sne (with standard parameters) of the test samples considering: (i) the raw representation in time domain (fig. 4a); (ii) the corresponding frequency-domain representation (magnitude of fourier transform) (fig. 4b); (iii) the latent representation (embedding with 256 dimensions) associated with the backbone + projector pretrained with tf-c (fig. 4c); and (iv) the latent representation (embedding with 256 dimensions) of backbone + projector after fine-tuning with the complete train dataset (fig. 4d). fig. 4. t-sne representations of high dimensional data in different scenarios of separability full size image it is possible to draw some important observations from fig. 4. firstly, fig. 4a indicates that the raw data present a significant overlap between the activities; it is possible to recognize parts that resemble clusters, mainly for the “laying” samples, but in general the data are mixed. secondly, as indicated by fig. 4b, by simply mapping the samples to the frequency domain, we can notice a more distinctive separation between the activities, which corroborates the appeal for exploring the fourier transform in har. in fig. 4c, we can see three main groups of activities: (a) laying; (b) sitting and standing; and (c) walking, upstairs and downstairs, with a non-negligible overlap between classes within the last two groups. this suggests that by pretraining the model without exploring the activity labels, the model still was capable of recognizing features that end up discriminating certain aspects of the underlying activities. finally, as indicated by fig. 4d, when we perform a fine-tuning of the backbone and projector using labeled samples, the activities become even more separated, with well-defined clusters that could be more easily identified by machine learning methods. 5 conclusions and future work our work effectively utilizes the tf-c model from original authors [28] to test performance across various amounts of training data. the tf-c, exhibits a significant facility in solving tasks of time series in scenarios with limited labeled data and an abundance of unlabeled data. in scenarios with large labeled data, pretrained and supervised strategies perform equally well, while with small amount of data, there is a tiny difference between both, but a significant difference between only time pipeline of supervised tf-c. these conclusions are specific to the har domain.the source code can be found at https://github.com/h-iaac/kr-papers-artifacts. first, we conducted a performance comparison test using varying percentages of the dataset, ranging from 0.1% to 100%. remarkably, even with just 1% of training set (42 samples) we achieved an accuracy of \(85.9\%\). the highest accuracy achieved was \(95.6\%\) using the entire dataset, comprising 7350 samples. the minimal number of samples to obtain an accuracy greater than 90% was only 126 samples (0.2% of dataset) subsequently, we compared the results with supervised cnns, which showed a significant performance drop in scenarios with small percentages of the dataset. we can conclude that the tf-c technique with pretraining is slightly advantageous in scenarios with limited labeled data. except when compared with the complete frequency pipeline. analyzing the latent space, we conclude that tf-c has a great ability to combine frequency and temporal features and work well in different domains, but do not help a lot when the downstream and fine-tune tasks were in the same dataset. in these cases, the backbone after fine-tune produces very relevant features to the target task and separate the data in distinctive clusters based on their labels. this work demonstrates the potential of using har datasets in downstream tasks of tf-c using a pre-trained model with same kind of data, achieving an accuracy of 95.6% when utilizing the entire training dataset for fine-tuning. even with just 42 samples of the training dataset, our approach achieves an accuracy of 85.9%. this highlights the effectiveness of classifying human activity recognition data using tf-c, surpassing the performance of individual supervised cnns with same architecture. for future work, we aim at expanding the analysis of tf-c considering other har datasets, as well as the impact of tf-c pretraining for cross-dataset scenarios, in addition to investigating the effects of different architectures on tf-c references andrzejak, r.g., lehnertz, k., mormann, f., rieke, c., david, p., elger, c.e.: indications of nonlinear deterministic and finite-dimensional structures in time series of brain electrical activity: dependence on recording region and brain state. phys. rev. e 64, 061907 (2001). https://doi.org/10.1103/physreve.64.061907 article  math  google scholar  anguita, d., ghio, a., oneto, l., parra, x., reyes-ortiz, j.l., et al.: a public domain dataset for human activity recognition using smartphones. in: esann, vol. 3, p. 3 (2013) google scholar  caron, m., misra, i., mairal, j., goyal, p., bojanowski, p., joulin, a.: unsupervised learning of visual features by contrasting cluster assignments. adv. neural. inf. process. syst. 33, 9912–9924 (2020) google scholar  challa, s.k., kumar, a., semwal, v.b.: a multibranch cnn-bilstm model for human activity recognition using wearable sensor data. vis. comput. 38(12), 4095–4109 (2022) article  math  google scholar  chen, t., kornblith, s., norouzi, m., hinton, g.: simclr: a simple framework for contrastive learning of visual representations. in: international conference on learning representations, vol. 2, p. 4 (2020) google scholar  chen, x., he, k.: exploring simple siamese representation learning. in: proceedings of the ieee/cvf conference on computer vision and pattern recognition, pp. 15750–15758 (2021) google scholar  cho, h., yoon, s.m.: divide and conquer-based 1d cnn human activity recognition using test data sharpening. sensors 18(4), 1055 (2018) article  math  google scholar  dong, j., wu, h., zhang, h., zhang, l., wang, j., long, m.: simmtm: a simple pre-training framework for masked time-series modeling. adv. neural info. process. syst. 36 (2024) google scholar  grill, j.b.: bootstrap your own latent-a new approach to self-supervised learning. adv. neural. inf. process. syst. 33, 21271–21284 (2020) math  google scholar  guo, c., zhang, y., chen, y., xu, c., wang, z.: modality consistency-guided contrastive learning for wearable-based human activity recognition. ieee int. things j. 11(12), 21750–21762 (2024) article  google scholar  haresamudram, h., et al.: masked reconstruction based self-supervision for human activity recognition. in: proceedings of the 2020 acm international symposium on wearable computers, pp. 45–49 (2020) google scholar  haresamudram, h., essa, i., plötz, t.: contrastive predictive coding for human activity recognition. proc. acm interact. mob. wearable ubiquit. technol. 5(2), 1–26 (2021) article  google scholar  haresamudram, h., essa, i., plötz, t.: assessing the state of self-supervised human activity recognition using wearables. proc. acm interact. mob. wearable ubiq. technol. 6(3) (2022). https://doi.org/10.1145/3550299 ismail, w.n., alsalamah, h.a., hassan, m.m., mohamed, e.: auto-har: an adaptive human activity recognition framework using an automated cnn architecture design. heliyon 9(2) (2023) google scholar  kemp, b., zwinderman, a., tuk, b., kamphuisen, h., oberye, j.: analysis of a sleep-dependent neuronal feedback loop: the slow-wave microcontinuity of the eeg. ieee trans. biomed. eng. 47(9), 1185–1194 (2000). https://doi.org/10.1109/10.867928 article  google scholar  liu, j., zhong, l., wickramasuriya, j., vasudevan, v.: uwave: accelerometer-based personalized gesture recognition and its applications. pervasive mob. comput. 5(6), 657–675 (2009). https://doi.org/10.1016/j.pmcj.2009.07.007 article  google scholar  oord, a.v.d., li, y., vinyals, o.: representation learning with contrastive predictive coding. arxiv preprint arxiv:1807.03748 (2018) qian, h., tian, t., miao, c.: what makes good contrastive learning on small-scale wearable-based tasks? in: zhang, a., rangwala, h. (eds.) kdd ’22: the 28th acm sigkdd conference on knowledge discovery and data mining, washington, dc, usa, august 14 18, 2022, pp. 3761–3771. acm (2022). https://doi.org/10.1145/3534678.3539134 reyes-ortiz, j., anguita, d., ghio, a., oneto, l., parra, x.: human activity recognition using smartphones. uci mach. learn. repository (2012). https://doi.org/10.24432/c54s4k saeed, a., ozcelebi, t., lukkien, j.: multi-task self-supervised learning for human activity detection. proc. acm interact. mob. wearable ubiq. technol. 3(2) (2019). https://doi.org/10.1145/3328932 tang, c.i., perez-pozuelo, i., spathis, d., brage, s., wareham, n., mascolo, c.: selfhar: improving human activity recognition through self-training with unlabeled data. proc. acm interact. mob. wearable ubiq. technol. 5(1) (2021). https://doi.org/10.1145/3448112 tang, c.i., perez-pozuelo, i., spathis, d., mascolo, c.: exploring contrastive learning in human activity recognition for healthcare. arxiv preprint arxiv:2011.11542 (2020) tonekaboni, s., eytan, d., goldenberg, a.: unsupervised representation learning for time series with temporal neighborhood coding. in: international conference on learning representations (2021). https://openreview.net/forum?id=8qdwejcucn xiao, y., huang, j., yang, j.: tfcsrec: time-frequency consistency based contrastive learning for sequential recommendation. expert syst. appl. 245, 123118 (2024) article  math  google scholar  yuan, h., et al.: self-supervised learning for human activity recognition using 700,000 person-days of wearable data. npj dig. med. 7(91), 1–10 (2024). https://doi.org/10.1038/s41746-024-01062-3 article  math  google scholar  zbontar, j., jing, l., misra, i., lecun, y., deny, s.: barlow twins: self-supervised learning via redundancy reduction. in: international conference on machine learning, pp. 12310–12320. pmlr (2021) google scholar  zhang, k., et al.: self-supervised learning for time series analysis: taxonomy, progress, and prospects. ieee trans. pattern anal. mach. intell. 46(10), 6775–6794 (2024) article  math  google scholar  zhang, x., zhao, z., tsiligkaridis, t., zitnik, m.: self-supervised contrastive pre-training for time series via time-frequency consistency. adv. neural. inf. process. syst. 35, 3988–4003 (2022) math  google scholar  download references acknowledgments this project was supported by the ministry of science, technology, and innovation of brazil, with resources granted by the federal law 8.248 of october 23, 1991, under the ppi-softex [01245.003479/2024-10]. the authors also thank cnpq (315399/2023-6 and 404087/2021-3) and fapesp (2013/08293-7) for their financial support. author information authors and affiliations hub for artificial intelligence and cognitive architectures (h.iaac), university of campinas, campinas, 13083-852, brazil nícolas hecker, otávio o. napoli, jaime delgado, anderson r. rocha, levy boccato & edson borin authors nícolas heckerview author publications search author on:pubmed google scholar otávio o. napoliview author publications search author on:pubmed google scholar jaime delgadoview author publications search author on:pubmed google scholar anderson r. rochaview author publications search author on:pubmed google scholar levy boccatoview author publications search author on:pubmed google scholar edson borinview author publications search author on:pubmed google scholar corresponding author correspondence to nícolas hecker . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper hecker, n., napoli, o.o., delgado, j., rocha, a.r., boccato, l., borin, e. (2025). an analysis of time-frequency consistency in human activity recognition. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_5 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords convolutional neural network human activity recognition self-supervised learning time frequency consistency publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature comparing lime and shap global explanations for human activity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper comparing lime and shap global explanations for human activity recognition conference paper first online: 30 january 2025 pp 172–186 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) comparing lime and shap global explanations for human activity recognition download book pdf download book epub patrick alves9, jaime delgado9, luis gonzalez9, anderson r. rocha9, levy boccato9 & … edson borin9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 513 accesses 1 citation abstract the development of complex machine learning models has been increasing in recent years, and the need to understand the decisions made by these models has become essential. in this context, explainable artificial intelligence (xai) has emerged as a field of study that aims to provide explanations for the decisions made by ml models. this work presents a comparison between two state-of-the-art xai techniques, lime and shap, in the context of human activity recognition (har). as lime provides only local explanations, we present a way to compute global feature importance from lime explanations based on a global aggregation approach and use correlation metrics to compare the feature importance provided by lime and shap across different har datasets and models. the results show that using correlation metrics to compare xai techniques is not enough to conclude if the techniques are similar or are not, so we employ a feature removal and retrain approach and show that, besides some divergences in the correlation metrics, both xai techniques successfully identify the most and least important features used by the model for the task. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a comparison of global explanations given on electronic health records chapter © 2024 explainable artificial intelligence (xai): a perspective chapter © 2025 enhancing machine learning model using explainable ai chapter © 2024 1 introduction smartphone-based human activity recognition (har) consists in identifying individuals activities such as walking, running, sitting, standing, and jumping, given samples from tri-axial sensors such as accelerometers and gyroscopes, typically embedded in smartphones. these sensors provide signals capable of discriminating the activities. the significance of solving har extends to various applications, particularly in remote patient monitoring and elderly care within the healthcare sector, identifying potential fall [20], and monitoring the degree of parkinson’s disease [17]. additionally, smartphone sensor data can be useful in the security field for authentication, employing patterns as a distinctive identifier [5]. to improve the capability of the classifiers, it is often necessary to extract features from the raw data by computing handcrafted features, employing deep learning models, or dimensionality reduction techniques [8, 24, 27]. the first aforementioned perspective favors the interpretation of the so-called latent representation, as the involved features are specifically tailored and selected for the task. on the other hand, dimensionality reduction and deep learning approaches lead to representations that are difficult to understand and to assign relevance [22, 24, 28]. these representations are commonly very abstract, and the use of complex models (black-box) to classify the data makes it difficult to understand the factors that have more impact on the outcome. to address this issue, explainable artificial intelligence (xai) techniques are useful for understanding the factors of the input data that have more impact on the model outcome or to understand the model behavior by calculating the contribution of each feature, computing the feature importance. xai techniques play an important role in making systems more interpretable, helping experts in deploying and judging the model’s reliability [19]. shap and lime are state-of-the-art xai techniques that provide explanations for models based on images, tabular, and text data [13, 19]. these techniques are model-agnostic, that is, they can be applied to any machine-learning model and can pinpoint which features are more relevant for the model to make a decision. to address the explanations provided, they can be classified as local explanations, that are used to provide insights into individual outcomes based on specific input data, while global explanations aim to offer a broader understanding of the model by providing explanations across a set of samples. one of the challenges in the literature is how to measure the (dis)agreement among the explanations provided by different xai techniques and in the case of disagreement, “in what technique should we trust?”. in this paper, we aim to provide a robust comparison among lime and shap explanations, two of the most popular xai techniques, for six har datasets over four classifiers. in this context, we compute correlation metrics to evaluate the linearity and rank similarity of the explanations, measuring their consistency, and validate the quality of these metrics. this research has been motivated by the work performed on the h.iaacfootnote 1 which explores various feature learning and dimensionality reduction techniques to obtain compact latent representations for smartphone sensors’ data. in this work, we evaluate the importance of features produced by a feature extractor that combines the original temporal data collected by the smartphone accelerometer and gyroscope into a latent space with 24 features. we explore with a feature extractor that applies the fast fourier transform (fft) to project the original data into the frequency domain and then employs kpca (a dimensionality reduction technique) to combine the features and reduce them to a set of 24 features. by identifying the importance of features within this latent space, developers can gain valuable insights into the newly constructed feature space, enabling them to assess the efficiency of the feature extractor and interpret the decisions made by the classifier. the contributions of this paper are as follows: (i) we employed a new method to provide global explanations from lime based on aggregations similar to those provided by shap; (ii) we provide a robust comparison measuring the agreement between lime and shap explanations for six har datasets over four classifiers. furthermore, we show that despite some disagreements among the explanations, these xai techniques have successfully identified the least relevant features for the classification task. (iii) we analyze the effect of removing the most and least relevant features on the model performance, showing that the least relevant features have almost no impact on the model performance, while the most relevant features do not necessarily have a high impact on the model performance. our results indicate that we can leverage lime to generate global explanations, providing insights on how the model behaves over the entire dataset. to the best of our knowledge, this is one of the first works that compare the global explanations of lime and shap for the har task. this comparison can assist researchers in understanding any disagreements among the explanations. moreover, we demonstrate that, despite potential discrepancies, the explanations are complementary and can be used with confidence to understand the model’s behavior and deploy the model in real-world applications. the rest of this paper is divided into the following parts: sect. 2 presents the related work for har, xai, and how to compare the explanations. section 3 shows the theoretical background of xai and describes the xai techniques employed in this work. section 4 discusses the datasets, the preprocessing steps, and the metrics to compare the explanations. section 5 exhibits the research questions and experiment results and addresses the proposed questions. finally, sect. 6 presents the conclusions and feature works. 2 related work in the har literature, researchers have employed different approaches to explain the models, such as lime, shap, lrp, grad-cam, and others to provide the relevance of each slice of the time series to the prediction [6, 10, 21]. on the other hand, some works have extracted handcrafted features from the time and frequency domains or dimensionality reduction to represent the data and then employed lime and shap to explain the models [3, 8, 9, 24]. these are some xai approaches employed to explain har models, but these works do not provide a concrete analysis or discussions about the explanations. furthermore, they do not address the disagreement problem among the xai techniques. in the literature, the most common approach to measuring the disagreement between the explanations is to compute the correlation among explanations over individual samples [2, 7, 11, 16]. this last-mentioned approach is not a good way to validate the disagreement. since, in some cases, we have a classification problem and there are many differences between the samples of the data from different classes, so it is viable to think that the reasons for the model’s decision can be totally different. to evaluate the trustworthiness of the explanations, some researchers employed xai techniques to explain the model’s decision and compared the explanations with the previous knowledge about the problem [3, 9, 25]. another comparison is among the explanations provided by different xai techniques, aiming to verify if the explanations are coherent with each other. this approach is common in the literature, providing subjective analysis among the explanations, verifying if the set of the most important features is the same among the explanations, for example, or only discussing the differences in time consumed to compute the feature importance metric [4, 11]. in this study, we compute correlation metrics to assess the agreement between lime and shap global explanations for har models. additionally, we evaluate the quality of the feature importance results by removing the least and most relevant features according to lime and shap, then retraining and re-evaluating the models using these subsets. 3 theoretical background this section provides an overview of lime and the methodology we employ to compute global importance values from local ones. local interpretable model-agnostic explanations [19] is a model-agnostic method that explains the predictions of any classifier by learning an interpretable model locally around the prediction over a dataset from perturbed instances around the instance to be explained and then fits a linear model to the dataset. the coefficients of the linear model are used to explain the prediction. lime has been used to explain the predictions of a variety of models, including deep neural networks, random forests, and support vector machines. it is also capable of explaining the predictions of models in a variety of domains, including text, image, and tabular data. to measure the agreement among global explanations, it is necessary to calculate feature importance globally for lime to compare the xai techniques globally. van der linden et al.  [12] proposed a methodology to calculate global explanations for lime based on global aggregations of local explanations. our approach is similar to the one proposed by van der linden et al.  [12] – the key difference is that we compute explanations per class, then compute the global explanations using the classes explanations. our method computes the global average importance for each class and then sums the aggregated importance for each feature to obtain the global importance. the steps to calculate the global importance are as follows: step 1: group the samples by class and calculate the feature importance for each sample; step 2: calculate the absolute mean of the feature importance \(w_{ij}\) for each class k: $$\begin{aligned} i_{j}^{k} = \dfrac{1}{n_k}\sum _{i=1}^{n_k} |w_{ij}| \; ; \; \forall k \in \{1, \cdots , m\} \end{aligned}$$ (1) where m is the number of classes, \(n_k\) is the number of samples that belong to class k, \(w_{ij}\) is the feature importance for the feature j from the \(i-th\) sample, and \(i_{j}^{k}\) is the importance of feature j for class k; step 3: finally, the global explanation is calculated by summing the feature importance for each class: $$\begin{aligned} i_{j}^{global} = \sum _{k=1}^m i_{j}^{k} \; ; \; \forall j \in \{1, \cdots , n\} \end{aligned}$$ (2) where \(i_{j}^{global}\) is the global importance of the feature j over the entire set of samples. 4 materials and methods this section presents the datasets used in our experiments, the preprocessing steps applied to standardize the data, and the methodology employed to evaluate the explanations. the experimental setup includes the dimensionality reduction techniques employed to reduce the data to a common feature space, the classifiers used, and the metrics to evaluate the disagreement of xai techniques. 4.1 datasets as mentioned before, the selection of the datasets and the choice to employ a reducer aiming to provide a general and compact representation is based on the work of h.iaacfootnote 2 team, which selected a subset of available datasets to work with. after analyzing various datasets, the group decided to work with: kuhar (kh) [15], motionsense (ms) [14], realworld (rw) [23], wisdm [26], and the updated version of the uci-har (uci) dataset [18]. these datasets have some differences regarding the smartphone position, the metric used to record the accelerometer samples, the sampling rate, and the number of users and activities registered, but they share common activities, such as sitting, standing, walking, walking upstairs, downstairs, and running. table 1 provides a comprehensive overview of the datasets’ key characteristics. table 1. selected datasets and their main features.full size table due to many divergences among the datasets’ characteristics, we executed preprocessing operations to minimize attribute variations, ensure a meaningful comparison of evaluation results, and address disparities among the datasets. table 2 presents the set of preprocessing steps applied to generate the standardized view for each dataset. table 2. set of preprocessing steps applied to generate the standardized view for each dataset.full size table the gravity component was removed using a third-order high-pass butterworth filter with a cutoff frequency of 0.3 hz, an approach presented by anguita et al.  [1], and the re-labeling step was applied to standardize the labels for the same activities across the datasets. after these rigorous preprocessing steps on each dataset, we ensured that all datasets had the same data format, the same number of features, and that their features have the same meaning, facilitating the comparison among them. furthermore, we balanced the dataset for activities (i.e., classes), ensuring that each activity had the same number of samples, and split the data into training, validation, and test sets in 70%, 10%, and 20%, respectively, by users, ensuring that there is no intersection between samples of the same users in the training, validation, and test sets. 4.2 experimental methodology figure 1 illustrates the feature extraction pipeline used in our experiments. first, we applied the fast fourier transform (fft) to transform the data to the frequency domain. then, we explored several dimensionality reduction techniques (reducers), including principal component analysis (pca), kernel pca (kpca), independent component analysis (ica), locally linear embedding (lle), isomap, uniform manifold approximation and projection (umap). fig. 1. pipeline for the encoder model. full size image we also explored different target dimensionalities, such as 12, 18, and 24 features, all possible combinations of the sets of datasets to train the reducer varying from 1 to 3 datasets, and evaluated the performance on rf, svm, and knn classifiers ordered by the minimum accuracy achieved among them for each reducer configuration. our results indicated that applying kpca trained with motionsense and realworld-waist datasets allowed us to reduce the data to 24 features while maintaining the classifiers’ performance with minimal loss of accuracy, as shown in table 3. table 3. accuracies achieved by each ml model on each dataset. the best and worst accuracy for each dataset are in bold and underlined, respectively.full size table notice that, in general, all classifiers managed to learn the task and perform much better than a random classifierfootnote 3. after reducing the data and training the classifiers, we applied the lime and shap xai techniques to generate the global explanations using the test sets. to easily facilitate the comparison among the techniques, we normalized the relevance vector for each technique, dataset, and classifier using the \(\ell _1\)-norm. to compare the agreement among the explanations, we computed correlation coefficients among the importance scores assigned to each feature by lime and shap. we utilized three commonly employed correlation metrics: pearson, spearman rank correlation, and kendall’s tau. pearson correlation is ideal for continuous data with a linear relationship, measuring the strength and direction of this connection. spearman correlation, a non-parametric method, assesses monotonic relationships without assuming linearity or specific data distribution. it’s less affected by outliers and works well with ordinal or non-linear data. kendall’s tau, another non-parametric method, is used for both continuous and ordinal data and it evaluates the association strength based on data ranks and is robust against outliers. in summary, pearson is best suited for evaluating linear relationships, whereas spearman and kendall’s tau are more appropriate for assessing monotonic relationships or when the data does not conform to normality assumptions. to interpret the coefficient values calculated by each metric, we can refer to table 4a and table 4b. table 4. interpretation of correlation coefficientsfull size table 5 experimental results this section presents the results of our experiments, the analysis, and the answers to the research questions proposed to evaluate the disagreement between lime and shap when explaining feature importance and if the divergence in the explanations is a problem for the interpretability of the models. as addressed in the sect. 4.2, we preprocess the datasets by computing the fft and reducing the data to 24 dimensions with kpca before training the ml models. the first research question is addressed in sect. 5.1 (are lime and shap coherent when explaining feature importance?), where we analyze the coherence between the feature importance values produced by lime and shap for each ml model and dataset, computing different correlation metrics to measure how similar the explanations are. subsequently, in sect. 5.2 (is there a direct relationship between classifier discriminative capability and feature importance?), we evaluate the impact of removing the most and least important features on model performance across different ml models and datasets. this analysis helps determine how the exclusion of these features affects the overall performance. 5.1 explainability coherence the first research question aims to evaluate the coherence between the feature importance values produced by lime and shap for each ml model and dataset. answering this question is essential to understand if the explanations provided by these techniques are consistent and can be trusted. to answer rq 1, i.e., “are lime and shap coherent when explaining feature importance?”, we first train each ml model (rf, svm, knn, and dt) with each dataset, then we calculate the importance of each feature according to each explainability technique for each ml model trained. figure 2 shows the global importance of each feature according to shap and lime, when classifying the samples from the wisdm dataset using an rf, svm, knn, and a dt ml model. notice that the explainability techniques do not assign the same importance to each feature. however, the techniques tend to agree on the most relevant and least relevant features. both techniques agree that the most important features for the rf model (fig. 2a) are features 0, 1, 2, and 8, and the most important features for the dt model (fig. 2b) are features 1, 2, and 4. for the svm (fig. 2c) and knn (fig. 2d) models, the most important features are 0, 1, and 2, in this order. fig. 2. twelve most important features according to lime and shap for (a) random forest, (b) decision tree, (c) svm, and (d) knn models trained with dataset wisdm to recognize human activities. full size image the same trend can be observed when training the four ml models with other datasets, and, despite a few exceptions, the importance assigned by lime and shap to each feature has a strong correlation. figure 3 shows the pearson, spearman, and kendall’s tau correlation for lime and shap feature importance values. the results indicate a very strong correlation among lime and shap ’ feature importance values overall, with a few exceptions, mostly on dt models. however, even in these cases, the correlation was close to the determined boundary and demonstrated a strong to very strong correlation (based on tables 4a and 4b), indicating that the explanations provided by both techniques agree with each other. despite slight differences in ranking when analyzing the most important features, these differences are negligible because the techniques can indicate the set of features that are most important. fig. 3. (a) pearson, (b) spearman and (c) kendall’s tau correlation for lime and shap feature importance values. results above the horizontal dashed lines indicate strong correlations. full size image the results in figs. 3a and 3b indicate that lime and shap had the lowest spearman and kendalltau correlation when computing the feature importance values for knn and dt models on the kuhar and uci datasets. however, fig. 3c indicates that the pearson correlation for this experiment is high. this happens because there are some features to which their relevance is much bigger than the others and the pearson correlation is very sensitive to outliers. figure 2b shows the feature importance values for the dt model trained with wisdm. we can see that feature 0 has a much higher importance value than the others, which can lead to a high pearson correlation. 5.2 feature importance vs classifier performance this section presents the experimental results to address rq 2, i.e., “is there a direct relationship between classifier discriminative capability and feature importance?”. answering this question helps to understand if the least relevant features can be removed without affecting the model’s performance and if the most relevant features are essential to the model’s performance. to answer this question, we propose two new questions: rq 2-a “does the removal of features with low relevance affect the discriminative power of the models?”; rq 2-b “does the removal of highly relevant features significantly affect the discriminative power of the models?”. answering these questions will help to understand the impact of each feature on the model’s performance, aiding researchers in determining which features are essential to the model without affecting its performance. in addressing rq 2-a, we computed the feature importance for each ml model using both lime and shap. subsequently, we organized the 24 features in ascending order of relevance and retrained the ml model by eliminating the least important features. similarly, to address rq 2-b, we conducted the same analysis, focusing on removing the most important features. the performance of the models when removing the most and least important features by xai technique is depicted in figs. 4 and 5. the lines in red and gray (-best) represent the accuracy of the models when removing the most important features according to lime and shap, respectively. and, the lines in blue and black (-worst) represent the accuracy of the models when removing the least important ones according to lime and shap, respectively. fig. 4. accuracy of the models when removing the most and least important features by xai technique on kuhar, motionsense, and uci. full size image fig. 5. accuracy of the models when removing the most and least important features by xai technique on wisdm, rw-thigh, and rw-waist. full size image the results indicate that, despite some differences in their respective rankings, the xai techniques agree on the sets of features that are most and least important. it can be observed that the lines corresponding to the removal of the most important features (in red and gray) decrease together, while the lines corresponding to the removal of the least important features (in blue and black) also decrease together. furthermore, the performance of the models is generally unaffected when the least important features are removed, except for the dt model, which exhibits oscillatory behavior. conversely, removing the most important features significantly affects model performance, resulting in a noticeable decrease in accuracy. moreover, the expected behavior is that the lines corresponding to the removal of the most important features (in red and gray) decrease faster than the lines corresponding to the removal of the least important features (in blue and black). this pattern is observed in most cases. however, there are exceptions, such as the svm model on the kh, ms, and rw-thigh datasets, where the lines decrease at the same rate, and the dt model, which exhibits oscillatory behavior across almost all datasets. regarding these exceptions, several hypotheses can be proposed: (i) there might still be redundancy across features, allowing the model to adapt to the new feature space created by the removal of certain features. this adaptation could result in assigning more importance to the remaining features, potentially explaining the unexpected behavior of the svm model on the kh, ms, and rw-thigh datasets. (ii) the dt model might have identified correlations between some features, focusing primarily on one of them. when this feature is removed, the model shifts its attention to another feature, which could account for the observed oscillatory behavior. 6 conclusions and future work in this paper, we evaluated the explanations generated by two xai techniques, lime and shap, on a variety of har dataset and ml models. to do so, we employed correlation metrics to compare the explanations generated by the techniques based on feature importance. we also showed that, for random forest and svm, the techniques agree almost perfectly in ranking comparison, indicating that the explanations are very similar. despite some differences when comparing the rankings of the features, these differences are negligible because, in most cases, the techniques successfully identify the most important and least important features. this finding is important because it demonstrates that the xai techniques are capable of identifying the most and least relevant features for the model, which is their primary goal. the results of this study show that correlation metrics alone are not sufficient to evaluate how similar the explanations generated by the techniques are. furthermore, the removal of the least important features does not affect the model’s performance, which indicates that the techniques are capable of identifying the least important features. on the other hand, the removal of the most important features does not necessarily affect the model’s performance, but this does not mean that the techniques are incapable of identifying the most important features. instead, it suggests that the model may learn the same patterns from other features. as future work, we intend to investigate the similarities and differences between the explanations generated by the xai techniques in other domains, such as time and frequency domains, and to understand how the techniques behave with more interpretable forms of data. code availability all codes are open-source, licensed under mit license, and available at https://github.com/h-iaac/kr-papers-artifacts notes 1.the hub for artificial intelligence and cognitive architectures (h.iaac) aims to develop and disseminate knowledge about technologies capable of integrating different intelligence resources into mobile devices, making them skilled in decision-making. to read more about the h.iaac project, visit the website: https://hiaac.unicamp.br/en/about-the-hub/. 2.link to the page: https://hiaac.unicamp.br/en/. 3.since the dataset is balanced, the random classifier performance is estimated by dividing 100% by the number of classes. references anguita, d., et al.: a public domain dataset for human activity recognition using smartphones. in: esann, vol. 3 (2013) google scholar  belaid, m.k., bornemann, r., rabus, m., krestel, r., hüllermeier, e.: compare-xai: toward unifying functional testing methods for post-hoc xai algorithms into a multi-dimensional benchmark. in: world conference on xai, pp. 88–109. springer (2023) google scholar  bragança, h., colonna, j.g., oliveira, h.a., souto, e.: how validation methodology influences human activity recognition mobile systems. sensors 22(6), 2360 (2022) article  google scholar  duell, j., fan, x., burnett, b., aarts, g., zhou, s.m.: a comparison of explanations given by explainable artificial intelligence methods on analysing electronic health records. in: 2021 ieee embs international conference on bhi, pp. 1–4. ieee (2021) google scholar  ferreira, a., santos, g., rocha, a., goldenstein, s.: user-centric coordinates for applications leveraging 3-axis accelerometer data. ieee sens. j. 17(16), 5231–5243 (2017) article  math  google scholar  giurgiu, i., schumann, a.: explainable failure predictions with rnn classifiers based on time series data. arxiv preprint arxiv:1901.08554 (2019) gwinner, f., tomitza, c., winkelmann, a.: comparing expert systems and their explainability through similarity. in: decision support systems, p. 114248 (2024) google scholar  hassan, m.m., uddin, m.z., mohamed, a., almogren, a.: a robust human activity recognition system using smartphone sensors and deep learning. futur. gener. comput. syst. 81, 307–313 (2018) article  google scholar  jeyashree, g., padmavathi, s.: ihar-a fog-driven interpretable human activity recognition system. trans. emerg. telecommun. technol. 33(9) (2022) google scholar  kaushik, a., gurucharan, k., padmavathi, s.: enhancing human activity recognition: an exploration of machine learning models and explainable ai approaches for feature contribution analysis. in: 2023 icemce, pp. 1–6. ieee (2023) google scholar  krishna, s., et al.: the disagreement problem in explainable machine learning: a practitioner’s perspective. arxiv preprint arxiv:2202.01602 (2022) van der linden, i., haned, h., kanoulas, e.: global aggregations of local explanations for black box models. arxiv preprint arxiv:1907.03039 (2019) lundberg, s.m., lee, s.i.: a unified approach to interpreting model predictions. adv. neural inf. process. syst. 30 (2017) google scholar  malekzadeh, m., clegg, r.g., cavallaro, a., haddadi, h.: mobile sensor data anonymization. in: proceedings of the international conference on iot design and implementation, pp. 49–58 (2019) google scholar  nahid, a.a., sikder, n., rafi, i.: ku-har: an open dataset for human activity recognition. mendeley data (2021) google scholar  neely, m., schouten, s., bleeker, m., lucic, a.: order in the court: explainable ai methods prone to disagreement. arxiv preprint arxiv:2105.03287 (2021) papadopoulos, a., kyritsis, k., klingelhoefer, l., bostanjopoulou, s., chaudhuri, k.r., delopoulos, a.: detecting parkinsonian tremor from imu data collected in-the-wild using deep multiple-instance learning. ieee bhi 24(9) (2019) google scholar  reyes-ortiz, j.l., oneto, l., samà, a., parra, x., anguita, d.: transition-aware human activity recognition using smartphones. neurocomputing 171 (2016) google scholar  ribeiro, m.t., singh, s., guestrin, c.: “why should i trust you?” explaining the predictions of any classifier. in: proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining, pp. 1135–1144 (2016) google scholar  santoyo-ramón, j.a., casilari, e., cano-garcía, j.m.: a study of one-class classification algorithms for wearable fall sensors. biosensors 11(8), 284 (2021) article  math  google scholar  schlegel, u., arnout, h., el-assady, m., oelke, d., keim, d.a.: towards a rigorous evaluation of xai methods on time series. in: 2019 ieee/cvf (iccvw), pp. 4197–4201. ieee (2019) google scholar  soni, v., yadav, h., semwal, v.b., roy, b., choubey, d.k., mallick, d.k.: a novel smartphone-based human activity recognition using deep learning in health care. in: 3rd international conference on mind 2021, pp. 493–503. springer (2023) google scholar  sztyler, t., stuckenschmidt, h.: on-body localization of wearable devices: an investigation of position-aware activity recognition. in: 2016 ieee international conference on percom, pp. 1–9. ieee (2016) google scholar  uddin, m., soylu, a.: human activity recognition using wearable sensors, discriminant analysis, and long short-term memory-based neural structured learning. sci. rep. (2021) google scholar  vijayvargiya, a., singh, p., kumar, r., dey, n.: hardware implementation for lower limb surface emg measurement and analysis using explainable ai for activity recognition. ieee tim 71, 1–9 (2022) math  google scholar  weiss, g.m., yoneda, k., hayajneh, t.: smartphone and smartwatch-based biometrics using activities of daily living. ieee access 7, 133190–133202 (2019) article  google scholar  yang, j., nguyen, m.n., san, p.p., li, x., krishnaswamy, s.: deep convolutional neural networks on multichannel time series for human activity recognition. in: ijcai, vol. 15, pp. 3995–4001. buenos aires, argentina (2015) google scholar  zhang, m., sawchuk, a.a.: manifold learning and recognition of human activity using body-area sensors. in: 2011 10th icmlaw2011 10th international conference on ml and applications and workshops, vol. 2, pp. 7–13. ieee (2011) google scholar  download references acknowledgements this project was supported by the ministry of science, technology, and innovation of brazil, with resources granted by the federal law 8.248 of october 23, 1991, under the ppi-softex [01245.003479/2024-10]. the authors also thank cnpq (315399/2023-6 and 404087/2021-3) and fapesp (2013/08293-7) for their financial support. author information authors and affiliations hub for artificial intelligence and cognitive architectures (h.iaac), university of campinas, campinas, 13083-852, brazil patrick alves, jaime delgado, luis gonzalez, anderson r. rocha, levy boccato & edson borin authors patrick alvesview author publications search author on:pubmed google scholar jaime delgadoview author publications search author on:pubmed google scholar luis gonzalezview author publications search author on:pubmed google scholar anderson r. rochaview author publications search author on:pubmed google scholar levy boccatoview author publications search author on:pubmed google scholar edson borinview author publications search author on:pubmed google scholar corresponding author correspondence to patrick alves . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alves, p., delgado, j., gonzalez, l., rocha, a.r., boccato, l., borin, e. (2025). comparing lime and shap global explanations for human activity recognition. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_12 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords xai human activity recognition feature importance lime shap publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature improving short-content misinformation detection using multiple aspect trajectories classification techniques | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper improving short-content misinformation detection using multiple aspect trajectories classification techniques conference paper first online: 30 january 2025 pp 125–139 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) improving short-content misinformation detection using multiple aspect trajectories classification techniques download book pdf download book epub juan pablo chavarro sanchez  orcid: orcid.org/0009-0003-5059-28769, tarlis tortelli portela  orcid: orcid.org/0000-0001-7405-333010 & jônata tyska carvalho  orcid: orcid.org/0000-0001-9020-20769  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 381 accesses abstract the proliferation of fake news and misinformation on social media poses a significant threat to social integrity. while there are approaches such as automatic content analysis and artificial intelligence detection to mitigate this problem, these methods face challenges in classifying misinformation in short content, like that posted on microblogging platforms or instant messaging services. in this study, we present an innovative approach to misinformation detection that combines content-based detection with multi-aspect trajectory classification. this approach models information propagation by considering each shared message as a moving object within the network, enriching the propagation trajectory with different semantic aspects. we implemented this approach on a misinformation dataset collected from whatsapp and compared it with traditional content-based detection methods, and we combined these two detection orientations into a hybrid approach. our findings indicate that the proposed hybrid approach achieves an f1-score of 0.89, surpassing baseline models by 8% in misinformation detection, even in short content. these results suggest that combining content-based detection with multi-aspect trajectory classification is a novel and promising strategy for addressing misinformation on social media. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others how early can we detect? detecting misinformation on social media using user profiling and network characteristics chapter © 2023 efficient multivariate data fusion for misinformation detection during high impact events chapter © 2022 near real-time detection of misinformation on online social networks chapter © 2020 1 introduction in today’s digital era, social media has taken a pivotal role in people’s daily lives. these platforms provide users from around the world with the ability to communicate, share information, and express opinions quickly, affordably, and accessibly. according to statistica’s statistics [30], over 50% of the world’s population is active on social media, emphasizing its significance as a tool for communication, information, and opinion sharing. the growing reliance on social media for real-time news has increased the risk of misinformation, as malicious actors exploit these platforms [23, 34]. technological advancements, such as large language models, further enable the rapid spread of false information, reinforcing echo chambers and filter bubbles [3, 21]. additionally, the structure of social networks promotes the dissemination of misinformation [2], creating significant challenges for governments and society. the detection of fake news and online misinformation has been approached in various ways, including two main approaches: manual fact-checking and automatic detection. manual verification by news agencies faces limitations due to the vast volume of data, potential human bias, and impracticality for early detection [24]. automatic detection can overcome most of these limitations. these methods can be based on the context, source of the information or content. source-based detection assesses the credibility of the sources creating or sharing the information, while context-based detection analyzes user interaction patterns and information propagation. content-based detection seeks patterns in the text or images to identify the news article’s intent. despite its effectiveness and the fact that this is the most studied analysis in the state of the art [11], this approach is vulnerable to the manipulation of news style, language, and adversarial attacks [37]. a significant and noteworthy challenge in the detection of misinformation is the recognition of short texts, typical on microblogging platforms such as weibo and twitter; and instant messages services such as whatsapp or telegram. this feature is particularly relevant in the context of fake news, as the concise messages on these platforms, due to their brevity, can be more complex to evaluate and verify accurately. given the complexity and heterogeneity of data related to information on online social networks, along with the limitations of content-based approaches, we propose two novel approaches: one based on the context of misinformation propagation and user interaction on social networks, and another hybrid approach that combines content-based and trajectory-based detection. in this paper, we introduce a model that conceptualizes the propagation path of a message on an online social network, specifically an instant messaging service, as a multiple aspects trajectory (mat) [18]. this type of data provides an effective and expressive representation of the diversity of data dimensions, namely the contexts associated with online information and its propagation. to validate the feasibility of these approaches, we turned to a dataset of messages spread in public whatsapp groups focused on brazilian politics [1]. using the analogous definition of information as multiple aspects trajectories, we modeled the propagation paths of these messages based on forwarding chains as trajectories. this study proposes a novel trajectory-based approach and a hybrid approach for misinformation detection in social networks, combining content-based and trajectory-based classification. the results show that the hybrid approach outperforms the baseline method by 8%, achieving an f1 score of 0.89, especially in the classification of short texts. the main contributions of this work are: a new dataset of information trajectories, a new application domain for trajectory-based classification methods, and a new classification method with great potential for detecting misinformation in social networks. 2 related work in this section, we will analyze the related works on detecting misinformation and fake news, with a primary focus on those employing propagation-based approaches. these approaches take into careful consideration the social context in which such misleading information spreads, accounting for the ways data disseminates across online networks. researchers have approached this challenge from various angles and have employed different input representations in their models [13, 36]. the models that utilize cascade, tree, or “tree-like” structures directly represent the path of information propagation. in [4], del et al. investigate how users in facebook groups tend to share homogeneous content, leading to the formation of echo chambers. within these echo chambers, homogeneity drives the spread of misinformation. additionally, relationships between the trajectory size and the lifecycle of information are established. similarly, [32] shows that false news spreads faster, farther, and to a broader audience on twitter compared to true news, analyzing factors such as cascade depth and breadth. additionally, [5] ducci et al. introduces the “cascade-lstm” model, which detects misinformation and analyzes retweet behavior. some other propagation-based detection methods utilize structural network information, such as text data and the propagation process, to portray the behavior of users involved in the dissemination of misinformation [13]. often, these methods adopt a graph-based geometric orientation, and depending on the interaction between nodes, they can be classified as homogeneous, heterogeneous, or hierarchical. homogeneous networks involve a single type of node and a single type of edge. in [16], ma et al. propose a model based on recursive neural networks for rumor detection based on propagation trees that use user characteristics in the twitter15 and twitter16 datasets. in [10], han et al. use a homogeneous network representation to classify news using graph neural networks (gnns), highlighting the advantages of propagation-based approaches in terms of news text independence. in [29], a model is proposed to capture temporal evolution patterns of news from twitter15, twitter16, fakenewssetgen, and weibo using homogeneous dynamic graph neural networks for fake news detection. in [19], a homogeneous graph is created without considering the textual content of each tweet. instead, a tree structure is generated, and graph convolutional networks (gcns) are employed for fake news detection. heterogeneous networks involve multiple types of nodes and edges. for instance, in [27], relationships between authors, users, and news in a network called trifn are proposed, and a classifier is employed to detect fake news. in the work monti et al. [20], which employs similar dimensions to the previous studies and cnns for classification, the model is found to be potentially independent of the text and geographic location of the users involved in the network. additionally, the advantages of propagation-based approaches in terms of adversarial attacks, as these would require manipulating the social network. hierarchical networks involve various types of nodes and edges forming hierarchical relationships. this is exemplified in the work developed by shu et al. [26], where a hierarchical propagation network is constructed by evaluating different granularities to extract temporal, structural, or linguistic features. an approach for representing propagation can be based on time series, more specifically, multivariate time series. in the work of preveti et al. [22], rumor propagation is represented as the time series of a cascade, incorporating user-related and temporal aspects for classification. in the work of liu et al. [15], the propagation path is represented as a multivariate time series, with points based on user and time-related aspects. these aspects are primarily numeric or binary, and convolutional and recurrent networks are applied to detect time series associated with fake news in the twitter15, twitter16, and weibo datasets. 3 dissemination of misinformation as mat this section, divided into two parts, explores addressing misinformation in online social networks through multi-aspect trajectories. the first part covers the theoretical foundations, while the second focuses on applying these principles by converting a dataset into multi-aspect trajectories. 3.1 misinformation and mat as discussed in sect. 2, existing representations of news or information propagation paths generally use cascade structures, graphs, or time series, incorporating mainly numeric or textual features related to users, news, networks, or time. however, these representations lack semantic data. in this section, we introduce the essential definitions and concepts for our approach. we begin by presenting the definition of “multiple-aspect trajectory” as suggested in [17], and then define what a “multiple-aspect trajectory” is in the context of information in online social networks (osn), capable of capturing semantic data. definition 1 (multiple aspect trajectory). a multiple aspect trajectory is an ordered sequence of points \(t\;=\; \left\langle d_1,\;d_2,\; \dots \;,d_n\right\rangle \), with \(d_i=(x,y,t, \mathcal {a})\) being the \(i-\)th point of the trajectory at location (x, y) and timestamp t, described by the set \(\mathcal {a}=\left\{ a_1,\;a_2,\; \dots \;,a_r \right\} \) of r attributes. this type of sequential data allows for the inclusion of heterogeneous data of diverse nature and types. these trajectories can be multi-labeled, and for the classification task. according to the work by petry et al.  [17], the concept of a “trajectory classification” is defined as follows: definition 2 (trajectory classification). given a set of labels \(\mathcal {l}\) and a trajectory set defined by a set of pairs \(\mathcal {t} = \{(t_1, \text {label}(t_1)),\ldots , (t_{|\mathcal {t}|}, \text {label}(t_{|\mathcal {t}|}))\}\), where each pair contains a trajectory \(t_i\) and its class label label\((t_i)\in \mathcal {l}\). trajectory classification involves learning a prediction function (i.e., a model) f that assigns each trajectory \(t_i\) in the set \(\mathcal {t}\) to one of the class labels in \(\mathcal {l}\). kumar et al. [14] differentiate between misinformation, which is false information without intent to deceive, and disinformation, which involves deceptive intent. this distinction is supported by other researchers [12, 33], though some studies use a broader definition of misinformation to encompass all false or inaccurate information spread on social networks. this includes rumors, disinformation (intentional or unintentional), urban legends, fake news, spam, and hate speech. these phenomena, given their temporal evolution in a social network, can be represented as a sequence of multiple aspects. in line with our objective to analyze the behavior of information in a social network, it is essential to have a general concept that represents the dissemination of information. following guo et al. [9], we define information dissemination in social networks by considering the content, social, and temporal dimensions, as outlined by shu et al. [25]. let s be information disseminated in a social network n, which consists of two main sets: publisher and content. publisher \(p_{s}\) includes a set of profile features to describe the original author, such as name, domain, age, among other attributes. content \(c_{s}\) consists of a set of attributes that represent the news article and includes headline, text, image, etc. let \(e=\{e_{it}\}\) be the set of tuples to represent the process of how information spread over the time among n users \(u=\{u_1,u_2,\dots u_n\} \) and their corresponding posts \(p=\{p_1,p_2,\dots p_n\}\) on social media regarding information s. each element \(e_{it}\) is described as \(\{p_i, u_i, t\}\), indicating that a user \(u_i\) spreads a post s using \(p_i\) at time t. therefore, to consider information or news s as a trajectory, we must treat it as a sequence that passes through points of interest (pois). due to privacy concerns in social networks, it is not always possible to obtain a user’s location. thus, we initially need to define a temporal sequence of points along the trajectory it follows. in this case, each point corresponds to an interaction \(e_i\) at a time t, where a user \(u_i\) interacts with the post \(p_i\) corresponding to the poi at a specific time t. this i-th point in the temporal sequence is associated with content and context aspects provided by \(u_i\) and \(p_i\). in this regard, and considering the definition of misinformation propagation in a social network adapted to the definition of multi-aspect trajectories given in the definition 1, we present the following definition. definition 3 (multiple-aspect trajectory of information). a multi-aspect trajectory of information propagated in an online social network (osn) is an ordered sequence of points \(e= \left\langle e_1,\; e_2,\;\dots ,\;e_{n}\right\rangle \), where \(e_i=(u_i,t,\mathcal {a})\) represents the \(i-\)th point of the trajectory located at \(p_i\) at a timestamp t, described by a set of attributes \(\mathcal {a}=c_{s_{i}}\cup p_{s_{i}}\), where \(c_{s_i}\) is the set of attributes related to the content of s, and \(p_{s_{i}}\) is the set of context attributes related to s. this study addresses the problem of binary classification of trajectories, using \(\mathcal {l}=\{0,1\}\), as specified in definition 2. 3.2 dataset in this section, we start from the original dataset “fakewhatsapp.br”, which contains whatsapp messages labeled as misinformation or not. from these messages, we generate the dataset “mat fakewhatsapp.br”, which contains the multi-aspect trajectories of the messages, according to definition 3. fakewhatsapp.br. the original dataset, proposed in [1], consists of news collected from 59 public whatsapp groups that were used during the political campaigns for the 2018 presidential elections in brazil. this data collection took place from july 2, 2018, to october 29 of the same year, resulting in a total of 282,601 messages labeled in two classes: the positive class, represented as 1, corresponds to messages with misinformation, while the negative class, represented as 0, corresponds to messages without misinformation. to effectively label the dataset, the findings of vosoughi et al. [32] were considered, demonstrating that false information spreads faster, wider, and deeper. a subset of 5,284 unique messages was selected based on their virality and manually labeled following a classification protocol. the labeled subset comprises mostly brief messages, averaging 20 tokens for non-misinformative messages and 34 for misinformative messages. to automatically expand the categories assigned to the subset of messages to the entire dataset, an algorithm generated the tf-idf matrix from both the labeled subset and the complete corpus. it calculated the similarity between unlabeled and labeled messages, assigning labels to unlabeled messages when the cosine similarity exceeded 0.9. this process labeled 21,289 messages, with 6,926 being unique. cabral [1] evaluated this dataset using classic natural language processing methods, including bag of words (bow) and tf-idf features with n-gram variations. the best results, with an f1 score of 0.73, were achieved using mlp, lsvm, and sgd algorithms. excluding short messages and focusing on those with over 50 words increased the f1 score to 0.87. mat fakewhatsapp.br. taking into account the definition 3, we proceed to generate the set of multi-aspect trajectories using the aforementioned dataset. this set can be seen as a collection of trajectories passing through points of interest (pois). in the context of whatsapp, each poi represents a user who shares a message, and the moving object is the message being shared from one user to another. the semantic aspects are derived from the data obtained from the social network. if we were to construct the trajectories as temporal sequences of message repetitions (forwards), we would obtain the 6,926 trajectories corresponding to the unique labeled messages in the dataset, composed of 21,289 points. however, a more detailed examination of the dataset reveals that the vector representation and similarity calculation fail to fully capture similar records that exhibit variations. to address this limitation, we will use a representation capable of capturing the context of the text and generating a better representation through word embeddings generated by bert. the process of creating these trajectories unfolds following the detailed steps below and can be followed in the fig. 1. fig. 1. trajectory creation process. full size image i: extraction and grouping of trajectories. identification of forwards: all forwarded messages in the conversation are identified, thus establishing the beginning of a trajectory. temporal grouping: forwarded messages are grouped into trajectories ordered by time, allowing us to capture the sequence of forwards and the spread of information in the chat. ii: representation of trajectories with sentence embeddings. generation of embeddings each message in a trajectory is represented by a multilingual bert sentence embedding. this captures the semantic and lexical context of each message in a vector space. similarity calculation: we calculate cosine similarity between the embeddings of all trajectories. those that exceed a similarity threshold of 0.9 are considered candidates for merging. merging similar trajectories: trajectories or records with high similarity are candidates for merging and are combined into a single trajectory. this reflects the relationship between messages that share similar content. iii: assignment of unique identifiers (tid). temporal sorting: the forwards of each message representing a trajectory are sorted chronologically to establish their temporal sequence. tid assignment: a unique tid is assigned to each message or trajectory, ensuring that each trajectory in the dataset corresponds to a unique representative message. the process of creating the dataset involved selecting messages that were shared at least 10 times, thus representing trajectories with 10 or more points. this criterion reduced the dataset to 469 trajectories, composed of 177 labeled as non-misinformation and 292 as misinformation, reflecting the higher frequency of misinformation messages in the original dataset. these 469 trajectories span 10, 251 points and are associated with 2, 408 unique messages. the selected aspects for analysis are presented in table 1. the selection of these aspects was guided by the attributes commonly used in location check-in datasets such as foursquare [17]. the process of construction can be replicated using datasets from different messaging platforms such as telegram, or it can be modified for use on other social networks like twitter, where information is disseminated in brief texts. this representation of the spread of information on online social networks as a trajectory of multiple aspects is agnostic to the text itself, as it leverages data from the context and the propagation of information. table 1. description of dataset attributes.full size table 4 trajectory classification models in the field of multi-aspect trajectory analysis, various techniques have been developed to discover patterns and knowledge in this data, ranging from similarity analysis to clustering and classification, among others. initially, the focus was on the raw trajectories of moving objects over time, limited to extractable attributes from space-time dimensions [28]. with the advent of semantic trajectories, defined as sequences of points of interest visited by a user, the problem of “trajectory-user-linking” emerged, which aims to predict which user generated a trajectory. despite advances such as tulvae [35] and tuler [8], which have addressed these semantic trajectories, they still face limitations in other dimensions. finally, with multi-aspect trajectories, models capable of encompassing all dimensions have emerged. the movelets method [6], a pioneer in this field, extracts representative subtrajectories of classes, continually improving the quality of the extracted movelets and reducing computational costs. additionally, the marc [17] model, a recurrent neural network, can effectively handle all these dimensions. these models have found applications in a variety of datasets, including human trajectories, animal trajectories, transportation trajectories, hurricane trajectories, vessel trajectories, and storm trajectories. below, we explore in detail how these models are capable of addressing multi-aspect trajectories [28]. 4.1 marc model one of the standout models in this context is the marc model [17], which relies on attribute embeddings and recurrent neural networks (rnns). this model excels in processing trajectories defined by categorical or textual attributes. its architecture comprises several key components: encoding of trajectory points: initially, trajectory attributes are encoded using one-hot encoding. this encoded representation of our attributes is then multiplied by an embedding matrix, which has as many rows as possible values of the attribute. the rows of this embedding matrix represent the embedding of each of the attributes. finally, an aggregation function is applied to these embedded attribute points. in the marc model, three different aggregation functions can be used, which can be sum, average, or concatenation. recurrent component: the encoded trajectory points move on to a recurrent component, utilizing long short-term memory (lstm) cells. this neural network structure with lstm units is fundamental for handling sequential data like trajectory points. lstm units can capture relationships and patterns among different trajectory points and their attributes. fully connected layer and sigmoid function: the final component is a fully connected layer that receives information from the recurrent component and subsequently passes through a sigmoid function to calculate probabilities. 4.2 movelet-based models another category of models aims to identify the best trajectory or subtrajectory features for input into a classifier. various methods have been proposed for this task. these methods extract movelets [6], that are subsequences of the trajectory of any size and any combination of attributes. the movelets are evaluated by a quality score based on its discriminative power for each class, find the optimal alignment and division, and then assess its f-score, which determines its importance. the internal architecture and the way the candidate movelet is calculated differ for each model: mastermovelets [7]: calculates the distance from each candidate to all trajectories, making it the most computationally expensive method. hipermovelets [31]: extracts limited-size subtrajectories and generates movelet candidates based on their in-class frequency, currently being the fastest model. 5 experimental evaluation in this section, we describe the methodology used to evaluate our approach for misinformation classification using multi-aspect trajectory classification. we compare it with a content-based detection approach similar to cabral et al. [1], which serves as the baseline. all experiments will emphasize hyperparameter optimization through 5-fold stratified cross-validation, with the f1-score as the primary evaluation metric. 5.1 content-based detection in order to provide a comparative analysis, a classic content-based detection approach was established employing conventional natural language processing and machine learning methods. in this experiment, different textual content associated with each trajectory was utilized, meaning that if a trajectory represents the dissemination of certain information, all textual variants of that information are taken into account. for each of the 2408 different messages linked to the 469 trajectories, the same preprocessing employed by cabral et al. [1] was employed, included removing stopwords, normalization, lemmatization, retaining emojis as tokens, and normalizing urls. given their success with text vector representation using tf-idf and various n-grams, the same vectorizer was used, incorporating unigrams, bigrams, and trigrams. finally, the classification algorithms that yielded the best results with the aforementioned vector representation were selected, including logistic regression (lr), multi-layer perceptron (mlp), and linear support vector machine (lsvm). 5.2 trajectory-based detection in evaluating the proposed trajectory propagation-based approach, the trajectory classification models discussed in the previous section were used, with preprocessing and transformation detailed in sect. 3. for the marc model, experiments varied the aggregation method of the embeddings, testing different methods (add, average, concatenate). for the model based on movelets, hipermovelets was chosen for its efficiency and quality in extracting the candidate movelets matrix representing each trajectory. this matrix was used in three classification models: random forest (rf), support vector machine (svm), and multilayer perceptron (mlp). 5.3 hybrid approach as pointed out by cabral et al. [1], detecting misinformation based solely on content has limitations, especially with short texts. in addition to this limitation, it is important to consider the quantity of texts associated with an information, as content-based models employ frequency-based methods, and this can affect their performance if the frequency is low. that’s why the proposed hybrid classifier generates a classification threshold regarding the amount of words and texts associated with an information. if the text has fewer than a certain number of words and fewer than a certain number of different associated messages, the classification of this message will be determined by the classification assigned by the trajectory-based method to the trajectory to which that message belongs. otherwise, the classification will be made by a content-based classification model. for this approach, we evaluate the combination of all trajectory-based and content-based models with their optimized hyperparameters. furthermore, for the estimation of this threshold, we consider the quantity of words and different texts as hyperparameters. the models are trained using cross-validation, evaluating the hybrid model on the validation fold. after finding the best hyperparameters, the hybrid model is trained on the complete training set, and its performance is evaluated on the test set. 6 results a total of 69 experiments were conducted to evaluate different approaches to misinformation detection. nine combinations of tf-idf with unigrams, bigrams, or trigrams were explored, using multilayer perceptron (mlp), logistic regression (lr), and linear support vector machines (lsvm) classifiers for content-based detection. additionally, six trajectory-based models were evaluated: three variants of the marc model and three models based on the creation of movelets with hypermovelets, classified with multilayer perceptron (mlp), support vector machines (svm), and random forest (rf). finally, 54 experiments were implemented with the hybrid approach, combining tf-idf with unigrams, bigrams, and trigrams along with the classifier for content-based detection, with the six trajectory classification models mentioned above. table 2. results of misinformation classification using different approachesfull size table given the number of experiments, table 2 presents the top three results per approach, indicating their f1 score and accuracy. it is noteworthy that the best result is achieved when employing the hybrid approach that combines bigram+mlp+hyper+rf, which achieves the highest accuracy (0.85) and f1-score (0.89) among all evaluated models. content-based models exhibit acceptable performance, primarily when considering the use of n-grams of any order and the mlp classifier. however, these models achieve the lowest accuracy. on the other hand, trajectory-based models are led by the marc-based model, which in its three variants achieves the best results for this approach. furthermore, the best trajectory-based model (marc-avg) slightly outperforms the best content-based model in terms of f1 score, but the difference is more significant when observing the precisions achieved by these three models. table 2 presents the best results of the hybrid approach that combines trajectory-based and content-based models when determining a classification threshold. the hyperparameters of this threshold are, on average across all experiments, fewer than 40 words and fewer than 5 associated messages for an information. in such cases, classification is done by a trajectory-based method; otherwise, it is done by a content-based method. comparing the three best trajectory-based classifiers presented in table 2, we observe that, on average, they correctly classify 77.5% of short messages with few different texts associated, while the three best content-based classifiers classify, on average, 62.1%, which is 15% less. this demonstrates an advantage of trajectory-based classification: its independence from the text of messages associated with an information, as this approach only utilizes propagation data to represent a set of messages. when classifying a trajectory, all messages associated with it are classified, whereas in a content-based approach, only the unique text associated with an information is classified. this characteristic makes trajectory-based classification particularly suitable for short texts, where the information contained in the text may be limited. with the thresholds determined for each experiment, the best results are achieved when using unigrams and bigrams with the mlp classifier if the threshold condition is not satisfied, and hyper+rf or marc-cct otherwise. the best of these hybrid models achieves an f1-score of 0.89, which surpasses the best content-based method by 8%. these models also achieve superior accuracy. to better observe this, fig. 2 presents the average accuracy and f1-score per classification orientation. fig. 2. performance comparison of different approaches full size image as shown in fig. 2, trajectory-based and content-based methods deliver similar performance in terms of f1-score, with trajectory-based models achieving slightly higher accuracy but with greater variability. in contrast, the hybrid approach significantly outperforms both methods, demonstrating higher mean f1-score and accuracy with much lower variance, highlighting the advantage of combining both strategies for misinformation detection. we conducted a statistical significance analysis, starting with the shapiro-wilk test to assess normality (w = 0.95, p = 0.75), followed by a paired t-test (t = 3.51, p < 0.02). the results indicated that the hybrid model significantly outperformed contentand trajectory-based models. 7 conclusion in this article, we have introduced an innovative definition of multi-aspect trajectories to address the spread of misinformation in social networks. our approach views each shared message as a moving object within the social network and applies trajectory classification techniques, commonly used for classifying moving entities such as humans, animals, weather phenomena, among others. multi-aspect trajectory-based models have achieved comparable or even superior results to conventional misinformation detection methods, both in their trajectory-based variant and in combination with content-based models. the hybridization of both approaches has shown a significant improvement in classification capability. this research opens a new horizon in the fight against misinformation. the application of trajectory-based models, particularly those based on movelets, promises greater interpretability and accuracy in misinformation detection. references cabral, l., monteiro, j.m., da silva, j.w.f., mattos, c.l.c., mourao, p.j.c.: fakewhastapp. br: nlp and machine learning techniques for misinformation detection in brazilian portuguese whatsapp messages. in: iceis (1), pp. 63–74 (2021) google scholar  ceylan, g., anderson, i.a., wood, w.: sharing of misinformation is habitual, not just lazy or biased. proc. natl. acad. sci. 120(4), e2216614120 (2023) article  math  google scholar  choi, d., chun, s., oh, h., han, j., kwon, t.t.: rumor propagation is amplified by echo chambers in social media. sci. rep. 10(1), 310 (2020) article  google scholar  del vicario, m., et al.: the spreading of misinformation online. proc. natl. acad. sci. 113(3), 554–559 (2016) article  math  google scholar  ducci, f., kraus, m., feuerriegel, s.: cascade-lstm: a tree-structured neural classifier for detecting misinformation cascades. in: proceedings of the 26th acm sigkdd international conference on knowledge discovery & data mining, pp. 2666–2676 (2020) google scholar  ferrero, c.a., alvares, l.o., zalewski, w., bogorny, v.: movelets: exploring relevant subtrajectories for robust trajectory classification. in: proceedings of the 33rd annual acm symposium on applied computing, pp. 849–856 (2018) google scholar  ferrero, c.a., petry, l.m., alvares, l.o., da silva, c.l., zalewski, w., bogorny, v.: mastermovelets: discovering heterogeneous movelets for multiple aspect trajectory classification. data min. knowl. disc. 34(3), 652–680 (2020) article  mathscinet  math  google scholar  gao, q., zhou, f., zhang, k., trajcevski, g., luo, x., zhang, f.: identifying human mobility via trajectory embeddings. in: ijcai, vol. 17, pp. 1689–1695 (2017) google scholar  guo, b., ding, y., yao, l., liang, y., yu, z.: the future of false information detection on social media: new perspectives and trends. acm comput. surv. (csur) 53(4), 1–36 (2020) math  google scholar  han, y., karunasekera, s., leckie, c.: graph neural networks with continual learning for fake news detection from social media. arxiv preprint arxiv:2007.03316 (2020) hoy, n., koulouri, t.: a systematic review on the detection of fake news articles. arxiv preprint arxiv:2110.11240 (2021) islam, m.r., liu, s., wang, x., xu, g.: deep learning for misinformation detection on online social networks: a survey and new perspectives. soc. netw. anal. min. 10(1), 1–20 (2020). https://doi.org/10.1007/s13278-020-00696-x article  math  google scholar  kondamudi, m.r., sahoo, s.r., chouhan, l., yadav, n.: a comprehensive survey of fake news in social networks: attributes, features, and detection approaches. j. king saud univ.-comput. inf. sci. 35(6), 101571 (2023) math  google scholar  kumar, s., shah, n.: false information on web and social media: a survey. arxiv preprint arxiv:1804.08559 (2018) liu, y., wu, y.f.: early detection of fake news on social media through propagation path classification with recurrent and convolutional networks. in: proceedings of the aaai conference on artificial intelligence, vol. 32 (2018) google scholar  ma, j., gao, w., wong, k.f.: rumor detection on twitter with tree-structured recursive neural networks. in: proceedings of the 56th annual meeting of the association for computational linguistics (volume 1: long papers), pp. 1980–1989 (2018) google scholar  may petry, l., leite da silva, c., esuli, a., renso, c., bogorny, v.: marc: a robust method for multiple-aspect trajectory classification via space, time, and semantic embeddings. int. j. geogr. inf. sci. 34(7), 1428–1450 (2020) google scholar  mello, r.d.s., et al.: master: a multiple aspect view on trajectories. trans. gis 23(4), 805–822 (2019) google scholar  michail, d., kanakaris, n., varlamis, i.: detection of fake news campaigns using graph convolutional networks. int. j. inf. manag. data insights 2(2), 100104 (2022) google scholar  monti, f., frasca, f., eynard, d., mannion, d., bronstein, m.m.: fake news detection on social media using geometric deep learning. arxiv preprint arxiv:1902.06673 (2019) newsguard: the next great misinformation superspreader: how chatgpt could spread toxic misinformation at unprecedented scale (2023). https://www.newsguardtech.com/misinformation-monitor/jan-2023/ previti, m., rodriguez-fernandez, v., camacho, d., carchiolo, v., malgeri, m.: fake news detection using time series and user features classification. in: castillo, p.a., jiménez laredo, j.l., fernández de vega, f. (eds.) evoapplications 2020. lncs, vol. 12104, pp. 339–353. springer, cham (2020). https://doi.org/10.1007/978-3-030-43722-0_22 chapter  math  google scholar  reuters institute for the study of journalism: digital news report 2023 (2023). https://reutersinstitute.politics.ox.ac.uk/digital-news-report/2023 saikh, t., de, a., ekbal, a., bhattacharyya, p.: a deep learning approach for automatic detection of fake news. arxiv preprint arxiv:2005.04938 (2020) shu, k., bernard, h.r., liu, h.: studying fake news via network analysis: detection and mitigation. in: emerging research challenges and opportunities in computational social network analysis and mining, pp. 43–65 (2019) google scholar  shu, k., mahudeswaran, d., wang, s., liu, h.: hierarchical propagation networks for fake news detection: investigation and exploitation. in: proceedings of the international aaai conference on web and social media, vol. 14, pp. 626–637 (2020) google scholar  shu, k., wang, s., liu, h.: beyond news contents: the role of social context for fake news detection. in: proceedings of the twelfth acm international conference on web search and data mining, pp. 312–320 (2019) google scholar  da silva, c.l., petry, l.m., bogorny, v.: a survey and comparison of trajectory classification methods. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 788–793. ieee (2019) google scholar  song, c., shu, k., wu, b.: temporally evolving graph neural network for fake news detection. inf. process. manag. 58(6), 102712 (2021) article  math  google scholar  statista: number of social network users worldwide from 2017 to 2025 (2021). https://www.statista.com/statistics/278414/number-of-worldwide-social-network-users/ tortelli portela, t., tyska carvalho, j., bogorny, v.: hipermovelets: high-performance movelet extraction for trajectory classification. int. j. geogr. inf. sci. 36(5), 1012–1036 (2022) article  google scholar  vosoughi, s., roy, d., aral, s.: the spread of true and false news online. science 359(6380), 1146–1151 (2018) google scholar  wu, l., morstatter, f., carley, k.m., liu, h.: misinformation in social media: definition, manipulation, and detection. acm sigkdd explor. newsl. 21(2), 80–90 (2019) article  math  google scholar  zhang, x., ghorbani, a.a.: an overview of online fake news: characterization, detection, and discussion. inf. process. manag. 57(2), 102025 (2020) article  math  google scholar  zhou, f., gao, q., trajcevski, g., zhang, k., zhong, t., zhang, f.: trajectory-user linking via variational autoencoder. in: ijcai, pp. 3212–3218 (2018) google scholar  zhou, x., zafarani, r.: a survey of fake news: fundamental theories, detection methods, and opportunities. acm comput. surv. (csur) 53(5), 1–40 (2020) article  math  google scholar  zhou, z., guan, h., bhat, m.m., hsu, j.: fake news detection via nlp is vulnerable to adversarial attacks. arxiv preprint arxiv:1901.09657 (2019) download references acknowledgements this work has been partially supported by capes (finance code 001). we also thank the reviewers who contributed to the improvement of this work. the views and opinions expressed in this paper are the sole responsibility of the author. author information authors and affiliations universidade federal de santa catarina, 476 88.040-900, florianopolis, brazil juan pablo chavarro sanchez & jônata tyska carvalho instituto federal do paraná (ifpr), curitiba, brazil tarlis tortelli portela authors juan pablo chavarro sanchezview author publications search author on:pubmed google scholar tarlis tortelli portelaview author publications search author on:pubmed google scholar jônata tyska carvalhoview author publications search author on:pubmed google scholar corresponding author correspondence to juan pablo chavarro sanchez . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sanchez, j.p.c., portela, t.t., carvalho, j.t. (2025). improving short-content misinformation detection using multiple aspect trajectories classification techniques. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_9 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-aspect trajectory misinformation detection online social networks publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature sara a generative ai for legal process summarization based on chain of density prompt engineering | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper sara a generative ai for legal process summarization based on chain of density prompt engineering conference paper first online: 30 january 2025 pp 370–383 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) sara a generative ai for legal process summarization based on chain of density prompt engineering download book pdf download book epub francisco das chagas jucá bomfim  orcid: orcid.org/0000-0001-6160-78329, joao araujo monteiro neto  orcid: orcid.org/0000-0002-0690-24499, gilson bezerra filho  orcid: orcid.org/0009-0004-6868-77589, vasco furtado  orcid: orcid.org/0000-0001-8721-43089 & … vládia pinheiro  orcid: orcid.org/0000-0002-9851-83049  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 420 accesses 1 citation abstract generative ai, particularly large language models (llms), holds significant promise for enhancing judicial tasks, especially in automating the generation of legal cases during the sentencing phase. this paper introduces sara (system for analysis and summarization of legal actions), an innovative method for abstractive and multi-document summarization of legal proceedings. sara employs gpt-4o, trained exclusively through in-context learning, utilizing chain of density (cod) and co-star prompt engineering techniques. these methods, adapted from judicial procedural knowledge, significantly improve the quality of generated summaries. traditional evaluation metrics reveal effective training strategies but highlight their limitations in assessing summary quality. therefore, we propose a qualitative evaluation methodology based on expert-generated questionnaires, focusing on essential content inclusion and proper report structuring. inspired by this methodology, we trained another gpt model for large-scale summary evaluation. evaluations of summaries of fifteen first-degree court cases from the court of justice of the state of ceará show a significant advantage of in-context learning with cod, emphasizing the role of domain knowledge and report style. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others cost–benefit analysis of deploying shallow, deep learning and generative models for legal text classification article open access 11 november 2025 effectiveness in retrieving legal precedents: exploring text summarization and cutting-edge language models toward a cost-efficient approach article open access 20 february 2025 legal knowledge generation based on llm prompt engineering chapter © 2026 1 introduction generative artificial intelligence (gai), particularly large language models (llms), exhibits remarkable language generation capabilities shows great potential for various judicial tasks. recently, the federal superior court issued public call 01/2023 to develop ais tailored for these purposes [17], aligning with the goals set by the cnj (national council of justice) to enhance procedural efficiency and adhere to global judicial innovation practices [12]. a key objective is the automatic generation of the process report, crucial for ensuring judges are well-informed about essential case issues, thereby justifying their final decisions adequately. a process report must accurately narrate the facts and allegations presented, documenting the parties involved, their arguments, summaries of requests and defenses, and major events during the process. producing such a report is a complex, time-consuming task that requires domain knowledge and synthesis ability, making it well-suited for abstractive summarization. unlike extractive summarization, which merely selects phrases from the text, abstractive summarization generates a coherent synthesis that captures the essence of the case. research on using llms for abstractive summarization of legal documents has progressed, but it faces challenges, such as handling the length and volume of legal texts and evaluating summary quality, which often relies on syntactic metrics that inadequately capture semantic context [11]. notable state-of-the-art (sota) models include simcls [15] for english, which employs contrastive learning (cl), legalsumm for portuguese legal documents [13], and clsjur.br [14], which combines cl with free-reference evaluation techniques and a language model for legal documents in brazilian portuguese legalbert.pt [19]. these models, however, have limitations related to the original document size and the nature of the generated summaries, typically producing only syllabuses. traditional metrics like rouge and bleu, though useful for comparison, fall short of accurately reflecting how well a summary captures the essence of the document. in this paper, we introduce sara (system for analysis and summarization of legal actions), a novel method for abstractive and multi-document summarization of legal proceedings aimed at generating judicial reports. sara’s architecture is based on an llm (gpt-4o [1]) trained exclusively through in-context learning. this training employs two prompt engineering techniques: chain of density (cod) [2] for summarization and co-star [3] for structuring prompt content. we demonstrate that these techniques, adapted from procedural knowledge provided by judges and legal academics, significantly enhance the informative quality of the generated summaries. results using traditional model quality assessment metrics indicate which training strategies are most effective, but they also highlight the limitations of these metrics in evaluating summary quality. consequently, we propose a qualitative evaluation methodology based on expert-generated questionnaires. this methodology focuses on essential content inclusion and the appropriate structuring of process reports. beyond guiding prompt engineering and enabling human-quality analysis of summaries, this methodology inspired the training of chatgpt [6] to evaluate generated summaries, facilitating large-scale assessments. evaluations of summaries from fifteen first-degree judicial cases in a large brazilian state court, conducted by both machines and humans, reveal a significant advantage of summaries produced using in-context learning with cod, enhanced by domain knowledge and report style familiarity. 2 related work automatic summarization can be classified into two types: abstractive and extractive. abstractive summarization generates new text that captures the original content’s essence, often rephrasing it concisely. extractive summarization selects direct excerpts from the original text. summarization can also be categorized as single-document or multi-document, with the latter integrating information from multiple sources into a cohesive summary. for first-degree judicial processes, the most suitable automatic summarization is abstractive and multi-document. several works before the gpt era proposed various approaches to legal text summarization. for example, [15] employed contrastive learning for abstractive summarization, while [7] used deep reinforcement learning, combining extractive and abstractive techniques. “delsumm: incorporating domain knowledge for extractive summarization of legal case documents” [10] utilized domain-specific knowledge for extractive summarization, achieving high rouge metrics (rouge-1: 51.2) but requiring manual annotations. a similar approach was attempted by [9], who enhanced summarization by incorporating markers like \({<}\)issue\({>}\) and \({<}\)argument\({>}\), leading to improved rouge scores (rouge-1: 50.73). however, manual tagging presents a scalability limitation. for brazilian portuguese, legalsumm [13] employs transformer models [16] and contrastive learning to summarize judicial decisions from brazil’s supreme federal court, generating concise summaries. using the ruling.br [13] corpus for training and testing, results on rouge metrics demonstrated that legalsumm improved the quality of generated summaries compared to traditional methods. in [14], the authors proposed the clsjur.br model, an evolution of legalsumm, which uses the cl approach along with a new technique for generating and evaluating candidate summaries. clsjur.br is based on the mbart neural model [18] and the legalbert.pt language model [19]. the results, using the same legalsumm test set, showed an improvement in the generation of syllabuses with rouge-l from 0.35 (legalsumm) to 0.4789 (clsjur.br). the limitation of clsjur.br for generating court reports is the limitation for mono documents and the size of the input text (1024 tokens). in summary, while research on abstractive and multi-document automatic legal document summarization (alds) has made advances with simcls, legalsumm, and clsjur.br models, significant challenges remain due to the complexity and volume of legal texts. recent work consistently highlights the need to explore the potential of generative ai (gai) and large language models (llms) to improve the efficiency and accuracy of judicial report generation, contributing to a more agile and well-founded justice system. 3 sara generating the report process in this section, we introduce the architecture of sara, a gai designed for the analysis and summarization of legal cases, and the prompt engineering process that led to the final prompt for multi-document abstractive summarization of legal cases. sara specifically generates the section known as the report in the brazilian judicial system, whose main objective is to summarize the key factual and legal elements of the procedural relationship, including the names of the parties, the identification of the case with a summary of the request and objection, and a documentation of the main events that occurred during the process. a critical aspect of court reporting is the ability to summarize from multiple documents in a cohesive and accurate manner. therefore, the core of sara is a multi-document summary prompt capable of integrating information from diverse sources, including initial pleadings, responses, replies, terms of hearing, closing briefs, and other documents within the judicial process that are essential for understanding the legal demand and reaching a decision in the case. this efficient summarization process is essential for procedural speed, allowing an integrated and holistic view of cases. 3.1 sara architecture figure 1 presents sara’s basic architecture [https://l1nk.dev/hpipv] with the following modules: fig. 1. sara’s general architecture full size image ocr and nlp tools responsável por realizar o reconhecimento óptico de caracteres (ocr) e verificações como validade do documento em relação ao tamanho, linguagem válida e normalização dos múltiplos documentos de entrada; parameter and prompt selection responsável por selecionar o prompt a ser usado na sumarização e configurá-lo com base nos seguintes hiperparâmetros: lista de documentos a serem resumidos \(\texttt {<docs list>}\); modelo padrão do relatório judicial a ser gerado \(\texttt {<report template>}\); tamanho inicial do relatório \(\texttt {<t>}\); número \(\texttt {<n>}\) de etapas no processo de densificação (estratégia cod); número \(\texttt {<m>}\) de entidades ausentes; lista de informações obrigatórias que devem ser incluídas no resumo \(\texttt {<required-info>}\); summarization api rotina que envia a instrução de sumarização (prompt) para o llm junto com os documentos dos casos a serem resumidos. 3.2 versions of sara’s prompts prompt engineering is an essential skill, focusing on creating clear and precise instructions to guide llms in generating the desired outcomes [5]. to instruct sara in generating high-quality judicial process reports, an incremental approach to prompt engineering was used, focusing on enhancing the following qualitative aspects of judicial reports: form, clarity, precision, and density. in total, four versions of prompts were developed following the co-star framework [3] and the cod strategy [2]. the co-star framework offers an organized and consistent structure for prompts, dividing them into layers of form and content to facilitate the understanding and analysis of information. the form layers include context, which provides an overview of the scenario; objective, which defines the task’s purpose; style, which specifies the type of language; tone, which indicates the desired attitude or emotion; audience, which identifies the recipients; and response, which presents the expected final format. he content layers include specific details about the documents and the legal scenario, explanations of the mandatory elements in the summary, and guidelines on appropriate language and register, ensuring that the legal summaries are accurate, coherent, and well-organized to meet the needs of judges and other legal professionals involved in the case. on the other hand, the cod (chain of density) prompt engineering strategy ensures that every word in the text significantly contributes to the overall content, removing redundancies and superfluous information. this technique has proven effective in increasing the informational density of summaries, improving clarity and conciseness. the generic densification process is detailed below: step 1: identification of missing entities identify from one to \({<}\)m\({>}\) missing informative entities that are absent in the previously generated summary. repeat step 2 for \({<}\)n\({>}\) times. step 2: write a new, denser summary of similar length that covers all entities and details from the previous summary, as well as the missing entities. the final summary must contain only the essential text of the story, without any instructions, steps, or guidelines. a missing entity is: relevant to the main story; specific descriptive but concise (6 words or less); new not in the previous summary; faithful present in the original documents; located anywhere in the original documents. avoid specific terms that are not generalizable; use standardized language according to court thesauri; this section outlines the evolution of the sara prompts, with each version improving accuracy, clarity, and density in generated summaries. reports from each prompt version were qualitatively evaluated by three judges from the court of justice of ceará. the evaluation revealed that the initial use of cod and co-star techniques failed to meet judicial report requirements, such as maintaining chronological order and adhering to legal guidelines. to address this, a template was introduced in the prompt, ensuring facts were presented in the correct order and format, incorporating domain knowledge and legal style. the *prompt-cod-baseline* represents the starting point, using the cod strategy proposed in [2] to generate dense summaries, but it lacked the necessary nuances and precision to capture the complexity of legal texts [9]. the next version, called *prompt-cod-summ-zeroshot*, applied zero-shot learning techniques, allowing the model to generalize its knowledge from other domains to the specific task of legal summarization without additional training on the legal corpus. the hyperparameters for this version were defined by empirical experiments from the stf 01/2023 public call [17]. the context and objective sections specify which documents will be processed by sara and what the ai should do, outlining the mandatory information to include in the summary, such as the nature of the action, urgent relief, involved parties, legal arguments, relevant citations, and requests made by the parties. the guidelines section provides detailed instructions for executing the summary based on the chain of density (cod) strategy. the essential segments of this section are: 1) they provide guidance on maintaining the most relevant information, ensuring that it is presented clearly and comprehensibly for anyone. 2) they define the summary length (t = 1000 words), with an emphasis on clarity and informational density. 3) to ensure informational density, it is instructed that the summary be rewritten n = 5 times to improve the flow and make room for additional entities. less informative sentences should be merged, compressed, or removed to make the text more compact and meaningful. the following excerpt from the prompt, (guidelines section), is responsible for this densification task: “follow these steps: sample heading (fourth level) step 1: identify 1 to 5 missing informative entities (delimited by “;”) that are absent in the previously generated summary. repeat step 2 five times. step 2: write a new, denser summary of similar length that covers all entities and details from the previous summary, as well as the missing entities. the final summary must contain only the essential text of the story, without any instructions, steps, or guidelines.” these guidelines aim to ensure that the summaries are clear, concise, densely informative, and faithful to the original content. continuous rewriting of the summary, according to cod instructions, is essential to maintain the density and quality of the information, facilitating the understanding and use of the summarized documents. the other sections, style, tone, audience, and response, aim to guide the language model in generating formal and professional texts with an assertive and informative tone, suitable for legal professionals, and with a logical narrative free of value judgments. the qualitative evaluation of the reports generated by sara using the prompt-cod-summ-zeroshot version indicated that, despite their overall quality, they exhibited the following issues: (1) form: they do not meet legal formalities, as they do not follow the chronological order of events in the process; (2) clarity: important information is omitted, such as the minutes of the evidentiary hearing and the public prosecutor’s opinion; (3) accuracy: only one of the parties’ requests is mentioned, whereas there were multiple requests in the process. these issues are critical and can cause the nullity of the judgment. if the order of events is unclear or if there are relevant omissions, it can result in a violation of the adversarial principle. if the facts are not presented in an organized and complete manner, the decision may be considered insufficiently substantiated. additionally, if the narration of the facts is confusing and disorganized, the defense may be unable to present arguments or adequately contest the points raised. the prompt-cod-summ-domainknowledge version was designed to enhance report quality by incorporating legal domain knowledge, ensuring compliance with judicial standards, emphasizing clear citations, organized information, and the use of standardized legal language. the *prompt-cod-summ-dk-styles* version further advanced this by integrating domain knowledge with specific writing styles, introducing command and content layers, and using a report-template developed by magistrates. this version focused on accuracy and presentation, receiving the highest evaluations from magistrates for its clarity, form, and content accuracy. table 1 presents the final structure of sara’s prompt, indicating the hyperparameters to be instantiated. table 1. general structure of the sara abstractive summarization prompt prompt-cod-summ-dk-stylesfull size table 4 empirical evaluation the experimental evaluation consisted of generating reports for 15 first-degree cases from the court of justice of the state of ceará by sara, using the following versions of prompts: v0 prompt-cod-baseline; v1 prompt-cod-summ-zeroshot; v2 prompt-cod-summ-domainknowledge; v3 prompt-cod-summ-dk-styles. the reports were then compared with the reference report using traditional rouge [20] and bleu [21] metrics, as the available metrics are not yet sufficiently reliable for this purpose [4, 6]. finally, a qualitative evaluation was performed by the human experts and by llm gpt-4o using an evaluation prompt designed in this study and a questionnaire formulated by expert magistrates from the court of justice of the state of ceará. 4.1 quantitative evaluation table 2 presents the average results of the rouge (f1-score) and bleu metrics for the reports of the processes generated by sara with prompt versions v0 to v3, compared to the reference report produced by the magistrates. individual results for each report are available at https://l1nk.dev/hpipv. in summary, version v2 performed best in rouge-1 f1, showing better word accuracy. version v3 excelled in rouge-2 f1 and rouge-l f1, capturing more bigrams and preserving sentence structure and fluency. while v2 had the highest bleu score, it may have been penalized for incorrect sequences. version v0 had the lowest scores, struggling with both word accuracy and structure. despite a lower bleu score, v3 demonstrated superior structural quality and coherence. table 2. average rouge (f1-score) and bleu metrics for the different prompt versions.full size table 4.2 qualitative evaluation the quantitative metrics (see table 2) demonstrated weak correlations between the reports generated by sara and those generated by the magistrates, being insufficient to decide on one prompt strategy over the other. in this work, we conducted two qualitative evaluation strategies to capture quality criteria for the generated reports. in the first strategy, three magistrates evaluated all reports generated by the prompt versions v0 prompt-cod-baseline and v1 prompt-cod-summ-zeroshot, based on the following criteria: form, accuracy, clarity, and density. all reports were rejected on all criteria, being considered incorrect and insufficient to support judicial decisions. interestingly, version v1 achieved one of the best results in the rouge-1 f1 metric but was rejected by the experts. to address the quality issues, domain knowledge and style information (template) were incorporated into the subsequent versions v2-prompt-cod-summ-domainknowledge and v3prompt-cod-summ-dk-styles. reports were generated by these versions and evaluated by the magistrates. version v2 fully met the criteria of form, accuracy, and density, and partially met the clarity criterion due to the absence of some legal reasoning. version v3 fully met all the evaluated criteria. in a second strategy, a qualitative questionnaire consisting of 13 questions was developed (table 3) by the magistrate experts, with the responses ranging from 1 (report does not meet) to 5 (report fully meets). in order to systematize and automate a qualitative evaluation of reports on a large scale, focusing on form, clarity, accuracy, and utility, the following prompt was submitted to chatgpt 4o along with the questionnaire and five versions of reports for each case in the sample: the reference report and four summaries generated by sara using the four prompt versions v0 to v3: “you are a judicial analyst who will receive 5 versions of judicial process summaries and should evaluate them according to a provided \({<}\)questionnaire\({>}\). at the end, you should indicate the summary that best answers the questions in the evaluation questionnaire and justify your response. \({<}\)summary1\({>}\) \({<}\)summary2\({>}\) \({<}\)summary3\({>}\) \({<}\)summary4\({>}\) \({<}\)summary5\({>}\) \({<}\)questionnaire\({>}\)” table 3. qualitative evaluation questionnaire and average of answers given by chatgpt 4.0 for the questionnaire, for each report generated by sara’s prompt versions v0 to v3.full size table versions v1 and v2 were fully reproved by the human experts, even though v1 was not so bad in quantitative metrics. among the versions v2 and v3, version v3 stood out in the human evaluation on all the criteria established by the evaluation questionnaire, as well as by the gpt-4o language model in ratings, with the language model choosing version v3 based on criteria defined in the evaluation questionnaire. among the findings, we can highlight that, although all versions demonstrated clarity in identifying the involved parties, version v3 stood out by maintaining this clarity throughout the entire document. this aspect significantly facilitates the reader’s understanding, as evidenced by the consistent high ratings in all evaluations of “identification of the parties” in the analyzed versions. version v3 provided a detailed and accurate summary of both the claim and the defense, covering all essential aspects without significant omissions. for example, while versions v1 and v2 received high evaluation for clarity, both presented omissions in important specific details. version v3, on the other hand, ensured the complete inclusion of all essential information, as indicated by the consistent five-star ratings in all evaluated criteria, including “summary of the claim” and “summary of the defense.” version v3 provided a comprehensive and well-structured contextualization of facts, offering a cohesive narrative with clear and detailed accounts of procedural occurrences and evidence. while other versions performed well, they lacked the same level of detail and cohesion. version v3 also excelled in detailing the legal backgrounds in both the initial petition and the contestation, which is crucial for preparing the magistrate’s decision. additionally, v3 followed a clear and precise chronological order, ensuring fluidity and understanding. in summary, v3 was the most complete and well-structured version, meeting the evaluation criteria with the highest clarity, detail, and comprehensiveness. despite being tested in a specific context of civil courts in ceará, v3 was recognized for its superior performance. no hallucinations were detected, but human review remains essential for such tasks. 5 conclusion the integration of prompt engineering principles, such as the chain of density and the co-star framework, along with domain knowledge, has led to the creation of more comprehensible and useful summaries for legal experts. these methods emphasize the importance of essential elements in the process report and the manner, especially the order, in which the report should be generated, effectively meeting the needs of specialists in the legal field. the combination of generic in-context learning techniques with domain-specific knowledge has proven essential for automatically generating the process report. this application has significant social implications, as it reliably accelerates a crucial stage of the judicial process. although we focused on a specific type of summary, the generic nature of the process report, which involves various elements, suggests that our approach is broadly applicable to other types of legal document summaries. further investigation in this area is warranted. quantitative metrics for evaluating legal summaries remain limited. expert-created questionnaires and large-scale validations using gpt models present a promising alternative for the qualitative and quantitative evaluation of different approaches. finally, it is worth highlighting the practical viability of sara, which is already in use by magistrates in a major brazilian court, significantly contributing to the streamlining of national justice. the deployment of sara in a real-world judicial setting underscores its robustness and adaptability to the complex requirements of legal processes. by automating the generation of process reports, sara not only reduces the time and effort required by legal professionals but also enhances the consistency and accuracy of the reports, ensuring that essential details are systematically captured and presented. the successful implementation of sara demonstrates its potential to be scaled and adapted for use in other courts across different jurisdictions, further amplifying its impact on the judicial system. as courts globally face increasing caseloads and the pressure to expedite legal proceedings, tools like sara offer a viable solution to these challenges by providing a reliable, efficient, and standardized method for handling procedural documentation. as sara continues to be refined and expanded, its role in the judicial process could extend beyond report generation to encompass other aspects of legal document management and analysis. the ongoing development and integration of such ai-driven tools represent a transformative step toward a more modern, efficient, and accessible judicial system, ultimately benefiting legal professionals, litigants, and society as a whole. references openai. hello gpt-4o. disponível em: https://openai.com/index/hello-gpt-4o/. acesso em: 30 june 2024 adams, g., fabbri, a., ladhak, f., lehman, e., elhadad, n.: from sparse to dense: gpt-4 summarization with chain of density prompting. in: proceedings of the 4th new frontiers in summarization workshop. association for computational linguistics (2023). disponível em: https://aclanthology.org/2023.propor-1.33. acesso em: 1 set. 2024 gunawan, a.: how i won singapore’s gpt-4 prompt engineering competition. disponível em: https://towardsdatascience.com/how-i-won-singapores-gpt-4-prompt-engineering-competition-34c195a93d41. acesso em: 30 june 2024 dias, p., mendes, a., martins, a.f.t.: improving abstractive summarization with energy-based re-ranking. arxiv (2022). https://doi.org/10.48550/arxiv.2210.15553 ibm.: engenharia de prompts: a arte de fazer perguntas a ia. ibm (2023). disponível em: https://www.ibm.com/br-pt/topics/prompt-engineering. acesso em: 24 june 2024 liu, y., iter, d., xu, y., wang, s., xu, r., zhu, c.: g-eval: nlg evaluation using gpt-4 with better human alignment. arxiv (2023). https://doi.org/10.48550/arxiv.2303.16634 heilmann, c.m.: text summarization of legal documents using reinforcement learning: a study, pp. 403–414 (2021). https://doi.org/10.1007/978-981-19-2894-9_30 kanapala, a., pal, s., pamula, r.: text summarization from legal documents: a survey. artif. intell. rev. 51(3), 371–402 (2017). https://doi.org/10.1007/s10462-017-9566-2 article  google scholar  elaraby, m., litman, d.j.: arglegalsumm: improving abstractive summarization of legal documents with argument mining. arxiv, (2022). http://arxiv.org/abs/2209.0165010.48550/arxiv.2209.01650 bhattacharya, p., poddar, s., rudra, k., ghosh, k., ghosh, s.: incorporating domain knowledge for extractive summarization of legal case documents. arxiv (2021). disponível em: 2209.01650. acesso em: 30 june 2024 google scholar  kanapala, a., jannu, s., pamula, r.: summarization of legal judgments using gravitational search algorithm. neural comput. appl. 31 (2019) google scholar  conselho nacional de justiça (cnj): automação processual amplia eficiência de atendimentos em tribunal. disponível em: https://www.cnj.jus.br/revista-cnj-automacao-processual-amplia-eficiencia-de-atendimentos/break-em-tribunal/. acesso em: 30 jun. 2024 feijó, d.d.v.: summarizing legal rulings. universidade federal do rio grande do sul, instituto de informática, programa de pós-graduação em computação. porto alegre (2021) google scholar  lins, a.a., carvalho, c., bomfim, f.dc.j., bentes, d.d.c., pinheiro, v.: clsjur.br a model for abstractive summarization of legal documents in portuguese language based on contrastive learning. in: proceedings of the 16th international conference on computational processing of portuguese, vol. 1, santiago de compostela, galicia/espanha, março 2024. association for computational linguistics. disponível em: https://aclanthology.org/2024.propor-1.33 liu, y., liu, p.: simcls: a simple framework for contrastive learning of abstractive summarization. arxiv (2021). disponível em: 2106.01890. acesso em: 1 jul. 2024 google scholar  vaswani, a., et al.: attention is all you need. in: advances in neural information processing systems, pp. 5998–6008 (2017) google scholar  brasil. supremo tribunal federal. chamamento público para pesquisa sobre ia no stf. disponível em: https://www.stf.jus.br/arquivo/cms/noticianoticiastf/anexo/chamamentopblicoiastf.pdf. acesso em: 1 july 2024 liu, y., lapata, m.: mbart: multilingual denoising pre-training for neural machine translation. in: proceedings of the 58th annual meeting of the association for computational linguistics, pp. 7871–7880 (2020) google scholar  silveira, r., ponte, c., almeida, v., pinheiro, v., furtado, v.: legalbert-pt: a pretrained language model for the brazilian portuguese legal domain. in: naldi, m.c., bianchi, r.a.c. (eds) bracis 2023. lncs, vol. 14197, pp. 268–282. springer, cham (2023). https://doi.org/10.1007/978-3-031-45392-2_18 lin, c.-y.: rouge: a package for automatic evaluation of summaries. in: proceedings of the workshop on text summarization branches out (was 2004), barcelona, espanha, pp. 25–26. association for computational linguistics (2004). disponível em: https://aclanthology.org/w04-1013. acesso em: 1 jul. 2024 papineni, k., roukos, s., ward, t., zhu, w.-j.: bleu: a method for automatic evaluation of machine translation. in: proceedings of the 40th annual meeting of the association for computational linguistics (acl), pp. 311–318. association for computational linguistics, philadelphia (2002). disponível em: https://aclanthology.org/p02-1040. acesso em: 1 jul. 2024 download references author information authors and affiliations universidade de fortaleza, fortaleza, ce, brazil francisco das chagas jucá bomfim, joao araujo monteiro neto, gilson bezerra filho, vasco furtado & vládia pinheiro authors francisco das chagas jucá bomfimview author publications search author on:pubmed google scholar joao araujo monteiro netoview author publications search author on:pubmed google scholar gilson bezerra filhoview author publications search author on:pubmed google scholar vasco furtadoview author publications search author on:pubmed google scholar vládia pinheiroview author publications search author on:pubmed google scholar corresponding author correspondence to francisco das chagas jucá bomfim . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper bomfim, f.d.c.j., neto, j.a.m., bezerra filho, g., furtado, v., pinheiro, v. (2025). sara a generative ai for legal process summarization based on chain of density prompt engineering. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_26 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_26 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords text summarization legal documents generative ai prompt engineering publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature automated segmentation of computed tomography images for covid-19 patient evaluation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper automated segmentation of computed tomography images for covid-19 patient evaluation conference paper first online: 30 january 2025 pp 125–140 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) automated segmentation of computed tomography images for covid-19 patient evaluation download book pdf download book epub júlio vitor monteiro marques  orcid: orcid.org/0009-0006-1699-45829, clésio de araújo gonçalves  orcid: orcid.org/0000-0002-4773-80329, antonio oseas de carvalho filho  orcid: orcid.org/0000-0001-9241-12779, rodrigo de melo souza veras  orcid: orcid.org/0000-0001-8180-40329 & … romuere rodrigues veloso e silva  orcid: orcid.org/0000-0002-7163-74699  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 365 accesses abstract covid-19, caused by sars-cov-2, resulted in 774 million cases and 7 million deaths by march 2024. this study proposes an approach to detect pulmonary lesions in computed tomography scans, integrating classification, preprocessing, and segmentation. initially, a model based on lenet-5 classifies the relevant slices of the scans, eliminating the irrelevant ones. subsequently, the selected images undergo contrast adjustments, binarization, and normalization. afterwards, segmentation is performed using a u-net-based architecture, allowing for detailed segmentation. the methodology achieved 78.40% dice, 64.80% iou, 78% sensitivity, 100% specificity, 89.60% auc, and 81% precision, using only 9 million parameters. these results offer a practical and efficient solution, supporting specialists in patient treatment. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others segmentation of lung lesions caused by covid-19 in computed tomography images using deep learning chapter © 2024 tomographic identification and evaluation of pulmonary involvement due to sars-cov-2 infection using artificial intelligence and image segmentation technique chapter © 2021 a study of u-net-based architectures for segmenting covid-19 lung lesions chapter © 2024 1 introduction covid-19, caused by the novel coronavirus (sars-cov-2), is a highly infectious disease with varying symptoms that can resemble other respiratory pathologies such as viral or bacterial pneumonia [19]. while some patients experience mild clinical conditions, others suffer from severe complications such as pneumonia, respiratory failure, and organ failure [21]. as of march 2024, the global count of covid-19 cases and deaths has reached 774 million and over 7 million, respectively [25]. in this context, early diagnosis is crucial to increase patient survival rates and prevent the spread of the disease. the standard diagnosis for covid-19 involves rt-pcr with pharyngeal swabs [23], a method widely adopted worldwide [6, 11]. however, the impact of covid-19 has been significantly mitigated by advancements in vaccinations, resulting in a substantial reduction in cases and saving millions of lives globally [24]. as vaccination campaigns and effective treatments continue to evolve, proactive identification and management of associated complications, along with ongoing patient support during recovery, are essential [7]. other diagnostic and monitoring methods such as x-ray and computed tomography (ct) have been extensively studied for assessing and tracking patient conditions [15]. x-ray is widely available and convenient for assessing areas like the chest, but its sensitivity and specificity are lower than those of ct for covid-19 [22]. ct, on the other hand, is an effective and reliable method for detecting lung lesions caused by covid-19 as well as other viral and bacterial pneumonias due to its superior detail [2]. however, analyzing ct images is labor-intensive and demands significant manual effort, making the process exhaustive for specialists. this work aims to develop an automated approach for segmenting lung lesions in ct images, focusing on patient assessment and recovery. the goal is to reduce the manual effort for specialists and optimize analysis time by automatically identifying lesions, facilitating the management of infection complications. the proposed methodology includes robust preprocessing that standardizes images from different ct devices and uses data augmentation techniques to expand the sample set. the u-net neural network architecture [18] is employed to segment lesion regions. the generated segmentations are used for detailed visualizations, assisting specialists in monitoring and treating pulmonary complications in patients. additionally, we implement a method for classifying and separating ct images that show visible lung regions from those that do not, ensuring a more accurate analysis focused on areas of interest. the remainder of the work is organized as follows: sect. 2 describes related works, sect. 3 details the materials and methods used, sect. 4 presents the results obtained during the experiments, sect. 5 discusses the findings, and sect. 6 concludes the study and suggests future directions. 2 related work since the modernization of technology in the healthcare sector and the increased availability of diagnostic imaging exams, computer-aided diagnosis (cad) systems have seen significant growth, detecting and diagnosing various diseases with results comparable to those of specialists [5, 13, 30]. numerous studies employ computed tomography (ct) images to detect respiratory diseases like covid-19 using robust metrics. notable works by castiglione et al. (2021) [4], ardakani et al. (2020) [1], and tao zhou et al. (2021) [31] report sensitivity and specificity rates exceeding 90%, utilizing deep learning methods for classification and image preprocessing. however, specialists must thoroughly assess the severity of lung lesions to monitor patient recovery, making segmentation methods indispensable. the authors [29] propose a modified model of u-net, termed d2a u-net, used for lesion segmentation, achieving 72.9% dice and 70.7% recall. in the study by canu et al. [32], they proposed a u-net-based network with a tversky loss function to handle the segmentation of small lesions, achieving 83.1% dice and 18.8% hausdorff distance. duda-net, proposed by xie et al. (2021) [26], introduces a dilated convolutional attention (dca) mechanism, allowing the network to focus on subtle areas of the lesions. the method achieved 87% dice, 90.8% sensitivity, 99.5% specificity, and 96.5% auc. lcov-net, combined with the 3d u-net, was presented by qianfei et al. (2021) [28], achieving 78.6% dice. in the work of zhang et al. (2021) [27], the qc-hc u-net is proposed, based on the 3d u-net, this architecture combines residual and dense connections to form a new connection method and apply it to both encoder and decoder. the method achieved 85.3% dice, 83.6% sensitivity, and 99.9% specificity. in the study by hasanzadeh et al. (2020) [8], a review of four segmentation methods, u-net, attention u-net, r2u-net, and attention r2u-net, is proposed. the r2u-net outperformed the others, achieving 79% dice. cogseg, proposed by sang et al. (2021) [20], uses image super-resolution as an auxiliary task, achieving 89.7% iou, 83% dice, 86.9% sensitivity, and 98% specificity. previous works indicate the adoption of u-net variants with significantly complex architectures, which implies a high computational cost. this characteristic represents a substantial barrier for implementation in environments with limited computational resources. in response to this limitation, our approach utilizes a simplified version of u-net, deliberately designed to minimize computational expense without significantly compromising performance. this strategy not only makes the technology more accessible but also maintains efficacy comparable to state-of-the-art models, demonstrating that it is possible to achieve an optimal balance between architectural simplicity and analytical capability in resource-constrained scenarios. the difficulty in acquiring ct images for research is also a limitation, with many authors using private or public databases with few samples. moreover, there is a lack of preprocessing techniques to enhance network performance and treat images from different ct devices. in this work, we develop a methodology for preprocessing ct images, using data augmentation techniques to overcome the limitation of small databases. we also include a method for classifying and separating ct images with visible lung regions, ensuring a more accurate and focused analysis in areas of interest. 3 materials and methods the proposed methodology is divided into the following steps: i) image acquisition; ii) detection of lung regions; iii) image preprocessing; iv) lesion segmentation. figure 1 displays the flowchart with all the steps of the proposed method. fig. 1. flowchart illustrating all the stages within the methodology up to the final analysis of the examination. full size image 3.1 image acquisition a dominant challenge in the task of classifying or segmenting computed tomography (ct) images for covid-19 context lies in the scarcity of comprehensive public databases. in this work, we utilize two of the few available databases: covid-19-ct-seg (covidseg) [12]footnote 1, and mosmed [16]footnote 2. covidseg is publicly accessible on the zenodo platform [10]. this database includes a total of 20 chest ct scans with dimensions of \(512\,\times \,512\), \(630\,\times \,401\), and \(630\,\times \,630\), featuring lesion markings for diagnosing covid-19. each sample in the database contains infection masks created by two radiologists and verified by a third. the scans vary in the number of images due to the imaging device used; the database encompasses 3520 images. figure 2 shows part a) samples from covidseg, where it is evident that the images have different characteristics. the second database, mosmed, comprises 50 chest ct scans with dimensions of \(512\,\times \,512\). each sample contains a positive lesion marking for covid-19, totaling 2049 images. in fig. 2, part b) presents samples from mosmed. as can be observed, the images from the scans have similar characteristics and differ from those in covidseg, also featuring different dimensions. fig. 2. representations of the images from the databases. in part a), we have samples from covidseg and in part b), samples from mosmed. full size image from these two databases, it is also possible to note the difference in styles between the segmentation masks of these databases, where the markings from covidseg were manually made by radiologists and are more consistent. in contrast, the markings from mosmed are not well-defined. this underscores the need for a preprocessing methodology capable of handling images under these conditions. moreover, this reflects the real-world scenario where images are produced by various specialists and devices. 3.2 detection of the presence of lung regions the detection stage in the proposed method, which involves analyzing slices from computed tomography (ct) scans, aims to distinguish slices containing visible lung areas from those that do not. this distinction is crucial to avoid false positives in images without visible lung areas and to reduce the number of slices processed by the segmentation model. images classified as devoid of lungs automatically receive masks without lesions, while those showing visible lungs proceed to the segmentation process. to develop the classification model, we chose lenet-5, which, according to a literature review by marques et al. (2022) [14], stands out due to its low computational cost and effectiveness in terms of metrics and parameter quantity compared to more complex and computationally demanding models. the model architecture was adjusted to include two blocks of convolutional and fully connected layers. the first block consists of three convolutional layers with four feature filters each and a maxpooling layer. the second block has four convolutional layers with eight feature filters and maxpooling. a flatten layer is used to convert the feature matrices into a vector for the dense layers. to prevent overfitting, a dropout layer with a value of 0.3 is used after the flatten layer. the dense layers comprise two layers with 4 and 8 neurons, respectively, and the final dense layer has 2 neurons for the model output, using the softmax activation function. the relu activation function is applied to all layers. the model is compiled using the adam optimizer with a learning rate of 0.00001 and the binary crossentropy loss function, suitable for binary classification tasks. the training batch size is set to 8, and the number of epochs to 100. figure 3 provides an illustration of the proposed architecture. fig. 3. representation of the proposed architecture with all its layers. full size image during training, we use several callback strategies, such as early stopping, to monitor validation loss and terminate training if there is no improvement for 15 epochs. learning rate reduction is applied to overcome plateaus, adjusting the rate to 0.000001 if the validation loss remains constant for 7 epochs. additionally, a model checkpoint is used to save the best model based on the lowest validation loss. this approach ensures that only relevant slices containing lung areas are processed by the segmentation model, guaranteeing more accurate and efficient analysis. 3.3 image pre-processing image preprocessing is a crucial step in our method, applied after lung detection to optimize the effectiveness of the process. this methodology is employed to remove noise, enhance visual quality, eliminate regions irrelevant to lesion identification, and standardize images, making them invariant to the acquisition method. the rationale for conducting preprocessing after lung detection is to maximize computational efficiency and ensure that only the isolated regions of interest are refined. this approach not only enhances the robustness of the proposal but also minimizes the unnecessary processing of non-pulmonary areas, which could distort the results of segmentation and classification. as illustrated in fig. 2, the databases include chest ct scans that vary significantly from one another, reflecting the different acquisition methods used in each case. due to these variations, the images undergo a detailed preprocessing step shown in fig. 4, essential for standardizing the images before subsequent analysis. fig. 4. representation of all the preprocessing steps. full size image the first step of preprocessing is to check the average pixel value. this process helps identify images with high and low contrast. through empirical testing, we defined that high-contrast images have an average greater than 165 pixels, and low-contrast images have a value less than 165 pixels. the images are then subjected to histogram equalization contrast stretching (hecs) [9]. we apply hecs with the 2nd and 98th percentiles on high-contrast images. in low-contrast images, the applied values are the 25th and 98th percentiles. we observed that higher values might discard small lesions present in the lungs, and lower values may not work for contrast enhancement. the aim of this step is to standardize the images. the next step is to apply the otsu algorithm [17] to binarize the images. as a result, the rib cage and the pulmonary parenchyma region are highlighted, and we can remove all regions outside the rib cage. using only the region of interest, we perform histogram equalization to normalize all samples. finally, the last step was resizing to \(512\,\times \,512\) pixels. this procedure is carried out because deep learning algorithms require all inputs to be of the same size. after preprocessing, we have standardized images that are noise-free and devoid of features outside the lung region. thus, we reduce the possibility of bias in the models. the images after preprocessing highlight only the region of interest. 3.4 data augmentation it is common for ct scans to have a higher number of images without covid-19 lesions compared to those with lesions. as a result, most scans present an imbalance between classes, which can affect model learning and reduce system performance. to address this issue, we apply data augmentation techniques using the albumentations library [3]. this approach allows us to generate images and masks to balance the images present in the scans, enhancing the model’s generalization capability. to generate the images, we apply the following operations: horizontal flipping, vertical flipping, and transposition. we also create images with random contrast and brightness adjustments, with a 30% probability, and images with gaussian noise, with a 50% probability. operations such as horizontal flipping, vertical flipping, and transposition are used to generate more samples for training the model and balancing the classes. additionally, some scans contain noisy images, and the model may not be able to generalize to these images. therefore, the gaussian noise operation is used to address this issue, creating noisy images in other scans to contribute to the overall model generalization. these data augmentation techniques are crucial for improving the robustness of the model, ensuring it can handle the diversity of images found in real-world scans. implementing these operations helps create a more balanced and varied dataset, which is vital for the effective training of deep learning models. 3.5 segmentation u-net [18] is a convolutional neural network (cnn) architecture for image segmentation. upon reviewing the literature, a trend toward the use of u-net-based architectures is evident. these studies consistently report robust results in the task of segmenting covid-19 lesions. the u-net architecture comprises contraction and expansion pathways. the contraction path is similar to a typical cnn pathway, involving repeated application of two \(3\,\times \,3\) convolutions, each followed by a relu activation function, and a \(2\,\times \,2\) max-pooling operation with a stride of two for downsampling. with each downsampling step, the number of feature filters is doubled. in the expansive path, each step involves upsampling of feature filters followed by a \(2\,\times \,2\) convolution. this is coupled with concatenation with features from the contraction path and two additional \(3\,\times \,3\) convolutions, each followed by a relu activation. in the final layer, a \(1\,\times \,1\) convolution is used to map each feature vector to the desired number of classes. in this work, the implemented u-net uses input images with dimensions of \(512\,\times \,512\) pixels and starts with 32 filters. due to computational environment limitations, we used a batch size of 6 during training. the loss function employed was the dice loss. for the learning process, we utilized the adam optimizer with a learning rate of 0.0003, as determined from preliminary experiments and literature reports. the dice and iou metrics are used to monitor the model’s performance on the validation set during training. several callback strategies are applied to execute actions during model training. the first strategy is early stopping, where training is monitored, and if the validation loss does not decrease for ten epochs, training is halted. learning rate reduction on plateau is also implemented to draw the model out of a plateau. during training, if the validation loss plateaus for five epochs, the learning rate is reduced by 0.00003. finally, the model is saved based on the lowest validation loss. the number of epochs set for training the model is 100. 3.6 experiments to evaluate the efficacy of the proposed methodology, we conducted a comprehensive experiment aimed at testing the preprocessing, data augmentation, classification, and segmentation of pulmonary lesions in computed tomography (ct) images. employing cross-validation, we sought to improve performance metrics, ensuring the robustness and generalization of the model. initially, we divided the scans from the covidseg and mosmed databases into subsets. for the covidseg database, which contains 20 exams, 60% (12 exams) are used for training, 20% (4 exams) for validation, and 20% (4 exams) for testing. this division is repeated five times to ensure that all images are included in the test set at some point. in the first stage, we trained a classification model based on lenet-5 using the mosmed dataset subset, along with data augmentation techniques. this model is responsible for classifying the scan slices, identifying those that contain visible pulmonary regions. the covidseg database, which has detailed lung markings, is used to evaluate the classification model. thus, we ensure that only images with visible pulmonary regions are passed to the segmentation model. in the second stage, we trained the segmentation model using exclusively the covidseg database, which contains precise markings of lesions caused by covid-19. the training is conducted with cross-validation, following the division of 60% (12 exams) for training and 20% (4 exams) for validation, as previously described. in the third stage, we evaluate the model on the test set during each iteration of the cross-validation. the complete flow of the experiment simulates a realistic environment and is composed of the following steps: 1. initial examination processing: the original test set exam is processed with all available slices. 2. slice classification: the classification model separates the slices with and without lungs. slices without lungs automatically receive masks without lesions. 3. preprocessing: slices classified as containing lungs undergo the preprocessing process. 4. lesion segmentation: lesions are segmented using the trained model. this experiment integrates all stages of preprocessing, data augmentation, classification, and segmentation, aiming for a comprehensive and accurate analysis of the model’s efficacy in a realistic environment. by simulating a complete clinical workflow, we ensure that the model is rigorously tested for its generalization capacity and robustness under practical conditions. 4 results the classification task was performed using a model trained to detect the slices of the exams that contain lungs. after training, the model was evaluated using the covidseg dataset. the results of this evaluation are presented in table 1. table 1. results obtained for the task of detecting lungs in the slices of exams from the covidseg database.full size table the results in table 1 indicate that the lung detection model is effective, showing high accuracy, sensitivity, and auc. the kappa index further reinforces substantial agreement between the model’s predictions and the markings by specialists. while precision could be improved, the overall results demonstrate that the model is robust for detecting lungs in the slices of ct exams. table 2 presents the results for the task of segmenting covid-19 lesions in ct images. table 2. results obtained from the averages of the metrics across the folds of the covidseg, along with the standard deviation for each result from the task of lesion segmentation. da: data augmentation.full size table the original images exhibited precision and specificity (82.00% and 100%, respectively), but the dice coefficient (70.80%) and iou (55.59%) metrics suggest that there is room for improvement. the sensitivity of 63.80% indicates that the model is missing some lesions in the images. with the application of preprocessing, all metrics except precision improved, with dice increasing to 74.20%, iou rising to 60.80%, and sensitivity increasing to 71.20%, indicating an enhanced ability of the model to detect lesions. the combination of data augmentation with preprocessing resulted in the best metrics, with dice reaching 78.40%, iou achieving 64.80%, and sensitivity reaching 78.00%. the auc also showed a high value of 88.60%, indicating robust model performance in differentiating between lesions and healthy tissue. these results demonstrate that preprocessing and data augmentation are effective in improving the detection of pulmonary lesions in ct images. table 3. results obtained for the complete flow of the methodology.full size table table 3 presents the results of the complete methodology, which integrates the classification model for slice separation, the preprocessing method, and the segmentation model. we observed an increase in precision and an increase in the auc. figure 5 highlights the main contribution of this stage, which is the reduction of false positives related to noise found in images that do not display visible lung regions. additionally, fig. 6 illustrates the performance of segmentation with the complete methodology, showing that the segmentations are more detailed and closer to those of experts, compared to segmentations performed on original images without preprocessing. fig. 5. this figure illustrates the results of the proposed methodology. part (a) shows the prediction from the segmentation model, highlighting potential false positives. part (b) displays the result after using the classification model, which successfully prevents false positives. full size image fig. 6. the figure compares pulmonary lesion segmentations. panel (a) shows results from the model without preprocessing, and panel (b) from the model with full methodology. predictions are in green and expert annotations in red. the complete methodology yields more detailed, expert-like segmentations, achieving higher iou values. (color figure online) full size image the results presented indicate that the proposed methodology, integrating classification, preprocessing, and data augmentation, is effective for the detection and segmentation of pulmonary lesions in computed tomography images. the lung detection model demonstrated high accuracy, sensitivity, and auc, ensuring reliable identification of slices relevant for segmentation. improvements in dice, iou, and sensitivity metrics, especially with the combination of data augmentation and preprocessing, highlight the effectiveness of these techniques in enhancing the performance of the segmentation model. the inclusion of the classification model contributed to the reduction of false positives, further refining the accuracy of the segmentations. in summary, the integrated approach provides a more precise and efficient analysis of pulmonary lesions, optimizing support for the diagnosis and treatment of patients with covid-19. 5 discussions the proposed method for detecting and segmenting pulmonary lesions in computed tomography images presents promising results, even with the limitations of the databases and the presence of small lesions that challenge the performance of u-net models and their variations. in all experiments conducted, we observed that, without the need for a modified and complex u-net, we achieved results comparable to the state of the art. therefore, the proposed method offers a viable alternative to complex models, particularly useful in real-world scenarios with hardware limitations. the results obtained were compared with other methods in the literature. table 4 presents a comparison of the results of the proposed method with the state of the art, which uses training, validation, and test split in the exam slices. however, this comparison is unfeasible for our method due to differences in evaluation procedures. our method uses cross-validation to ensure a more robust and generalizable evaluation, whereas the methods listed in table 4 do not use this technique. this methodological difference prevents a fair and direct comparison of the results. table 4. comparison of the proposed work with the state of the art that uses training, validation, and testing splits in the slices of the exams.full size table table 5 offers a more appropriate comparison, where the analyzed methods use the division of training, validation, and testing on the exams, similar to our proposed method. the results show that the proposed method presents similar metrics, standing out especially in the iou metric and in terms of specificity (100%) and auc (89.60%). these results reinforce the effectiveness of our method in segmenting pulmonary lesions, comparable to the state of the art, despite using a significantly lower number of parameters. table 5. comparison of the proposed work with the state of the art that uses training, validation, and testing splits in the exams.full size table in addition to presenting similar results, the proposed method also uses significantly fewer parameters compared to other methods in the literature, as shown in table 6. this is particularly important in scenarios with hardware limitations, where lighter and more efficient models are preferable. our method, with 9 million parameters, offers a viable and effective alternative to more complex models that require higher computational capacity. table 6. number of parameters used by the authors in the analyzed literature compared with the proposed methodology.full size table 6 conclusions the proposed method for detecting and segmenting pulmonary lesions in computed tomography (ct) images has demonstrated promising results despite the limitations of databases, image acquisition methods, and the detection of small lesions. in all conducted experiments, the approach using the proposed u-net, combined with preprocessing and data augmentation techniques, achieved results comparable to the state of the art. the obtained metrics, especially in terms of iou, specificity, and auc, highlight the effectiveness of our method. the preprocessing methodology for ct images proved effective in standardizing exams acquired by different methods, significantly improving performance in segmentation tasks. combining this with data augmentation techniques reduces the chance of overfitting, balances classes, and generates new samples for training, resulting in superior performance on the test set. additionally, comparative analysis with other methods in the literature revealed that, despite using a smaller number of parameters, our method demonstrated competitive performance. this characteristic is particularly relevant for applications in scenarios with hardware limitations, where lighter and more efficient models are necessary. using only 9 million parameters, compared to tens of millions in other studies, underscores the feasibility and efficiency of our approach. to further enhance the proposed method, we suggest the following future directions: expand the dataset by incorporating more images from different sources and clinical conditions to improve the model’s robustness and generalization; investigate and incorporate new data augmentation techniques that can help handle the variability of ct images and improve the detection of smaller lesions; and utilize pre-trained models on large medical datasets to enhance the performance of the model on smaller and specific datasets, such as those used in this study. by following these directions, we expect not only to improve the accuracy and efficiency of the proposed method but also to facilitate its adoption in clinical settings, contributing to faster diagnosis and better patient outcomes. notes 1.https://zenodo.org/records/3757476. 2.https://www.kaggle.com/datasets/andrewmvd/mosmed-covid19-ct-scans. references ardakani, a.a., et al.: application of deep learning technique to manage covid-19 in routine clinical practice using ct images: results of 10 convolutional neural networks. comput. biol. med. 121, 103795 (2020) article  math  google scholar  axiaq, a., et al.: the role of computed tomography scan in the diagnosis of covid-19 pneumonia. curr. opin. pulm. med. 27(3), 163–168 (2021) article  google scholar  buslaev, a., iglovikov: albumentations: fast and flexible image augmentations. information 11(2) (2020) google scholar  castiglione, a., et al.: covid-19: automatic detection of the novel coronavirus disease from ct images using an optimized convolutional neural network. ieee trans. industr. inf. 17(9), 6480–6488 (2021) article  math  google scholar  chhikara, p., et al.: deep convolutional neural network with transfer learning for detecting pneumonia on chest x-rays. in: jain, l.c. (ed.) advances in bioinformatics, multimedia, and electronics circuits and signals, pp. 155–168. springer, singapore (2020) google scholar  corman, v.m., drosten, c.: authors’ response: sars-cov-2 detection by real-time rt-pcr. eurosurveillance 25(21) (2020) google scholar  george, p.m., et al.: respiratory follow-up of patients with covid-19 pneumonia. thorax 75(11), 1009–1016 (2020) article  math  google scholar  hasanzadeh, n., et al.: segmentation of covid-19 infections on ct: comparison of four unet-based networks. in: 2020 27th national and 5th international iranian conference on biomedical engineering (icbme), pp. 222–225 (2020) google scholar  jagatheeswari, p., et al.: contrast stretching recursively separated histogram equalization for brightness preservation and contrast enhancement. in: 2009 international conference on advances in computing, control, and telecommunication technologies, pp. 111–115 (2009) google scholar  jun, m., et al.: covid-19 ct lung and infection segmentation dataset. zenodo (2020) google scholar  leblanc, j.j., et al.: real-time pcr-based sars-cov-2 detection in canadian laboratories. j. clin. virol. 128, 104433 (2020) article  math  google scholar  ma, j., et al.: toward data-efficient learning: a benchmark for covid-19 ct lung and infection segmentation. med. phys. 48(3), 1197–1210 (2021) article  math  google scholar  marcomini, k.d., et al.: evaluation of a computer-aided diagnosis system in the classification of lesions in breast strain elastography imaging. bioengineering (basel) 5(3) (2018) google scholar  marques, j.v., et al.: detection of covid-19 in computed tomography images using deep learning: a literature review. revista de sistemas e computação-rsc 12(1) (2022) google scholar  martínez chamorro, e., et al.: radiologic diagnosis of patients with covid-19. radiologia 63(1), 56–73 (2021) article  math  google scholar  morozov, s., et al.: mosmeddata: chest ct scans with covid-19 related findings dataset. medrxiv (2020) google scholar  otsu, n., et al.: a threshold selection method from gray-level histograms. ieee trans. syst. man cybern. 9(1), 62–66 (1979) article  math  google scholar  ronneberger, o., et al.: u-net: convolutional networks for biomedical image segmentation. in: navab, n. (ed.) medical image computing and computer-assisted intervention—miccai 2015, pp. 234–241. springer (2015) google scholar  samir, a., et al.: covid-19 versus h1n1: challenges in radiological diagnosis–comparative study on 130 patients using chest hrct. egypt. j. radiol. nucl. med. 52(1), 77 (2021) article  math  google scholar  sang, y., et al.: super-resolution and infection edge detection co-guided learning for covid-19 ct segmentation. in: icassp 2021 2021 ieee international conference on acoustics, speech and signal processing (icassp), pp. 1665–1669 (2021) google scholar  siddiqi, h.k., mehra, m.r.: covid-19 illness in native and immunosuppressed states: a clinical-therapeutic staging proposal. j. heart lung transplant. 39(5), 405–407 (2020) article  google scholar  stephanie, s., et al.: determinants of chest radiography sensitivity for covid-19: a multi-institutional study in the united states. radiol.: cardiothoracic imaging 2(5), e200337 (2020) google scholar  wang, w., et al.: detection of sars-cov-2 in different types of clinical specimens. jama 323(18), 1843–1844 (2020) math  google scholar  watson, o.j., et al.: global impact of the first year of covid-19 vaccination: a mathematical modelling study. the lancet infectious diseases (2022) google scholar  who. covid-19 epidemiological update – 12 april 2024. https://www.who.int/publications/m/item/covid-19-epidemiological-update-edition-166. accessed 15 jan 2024 xie, f., et al.: duda-net: a double u-shaped dilated attention network for automatic infection area segmentation in covid-19 lung ct images. int. j. comput. assist. radiol. surg. 16(3) (2021) google scholar  zhang, q., et al.: segmentation of infected region in ct images of covid-19 patients based on qc-hc u-net. sci. rep. 11(1), 22854 (2021) article  google scholar  zhao, q., et al.: lcov-net: a lightweight neural network for covid-19 pneumonia lesion segmentation from 3d ct images. in: 2021 ieee 18th international symposium on biomedical imaging (isbi), pp. 42–45 (2021) google scholar  zhao, x., et al.: d2a u-net: automatic segmentation of covid-19 ct slices based on dual attention and hybrid dilated convolution. comput. biol. med. 135, 104526 (2021) article  math  google scholar  zhou, l., et al.: a rapid, accurate and machine-agnostic segmentation and quantification method for ct-based covid-19 diagnosis. ieee trans. med. imaging 39(8), 2638–2652 (2020) article  math  google scholar  zhou, t., et al.: the ensemble deep learning model for novel covid-19 on ct images. appl. soft comput. 98, 106885 (2021) article  math  google scholar  zhou, t., et al.: automatic covid-19 ct segmentation using u-net integrated spatial and channel attention mechanism. int. j. imaging syst. technol. 31(1), 16–27 (2021) article  math  google scholar  download references acknowledgments this work was carried out with the support of the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. we are also grateful for the support of fundação de amparo à pesquisa do estado do piauí (fapepi) -http://www.fapepi.pi.gov.br. author information authors and affiliations postgraduate program in computer science, federal university of piauí (ufpi), teresina, pi, brazil júlio vitor monteiro marques, clésio de araújo gonçalves, antonio oseas de carvalho filho, rodrigo de melo souza veras & romuere rodrigues veloso e silva authors júlio vitor monteiro marquesview author publications search author on:pubmed google scholar clésio de araújo gonçalvesview author publications search author on:pubmed google scholar antonio oseas de carvalho filhoview author publications search author on:pubmed google scholar rodrigo de melo souza verasview author publications search author on:pubmed google scholar romuere rodrigues veloso e silvaview author publications search author on:pubmed google scholar corresponding author correspondence to júlio vitor monteiro marques . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper marques, j.v.m., de araújo gonçalves, c., de carvalho filho, a.o., de melo souza veras, r., veloso e silva, r.r. (2025). automated segmentation of computed tomography images for covid-19 patient evaluation. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_9 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords computed tomography covid-19 medical image segmentation publish with us policies and ethics profiles rodrigo de melo souza veras view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature investigating behavior cloning from few demonstrations for autonomous driving based on bird’s-eye view in simulated cities | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper investigating behavior cloning from few demonstrations for autonomous driving based on bird’s-eye view in simulated cities conference paper first online: 30 january 2025 pp 155–168 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) investigating behavior cloning from few demonstrations for autonomous driving based on bird’s-eye view in simulated cities download book pdf download book epub eric aislan antonelo9, gustavo claudio karl couto9, christian möller10 & … pedro henrique fernandes9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 372 accesses 4 citations abstract this paper investigates the use of behavior cloning (bc) for autonomous driving from a bird’s-eye view (bev) perspective in simulated urban environments. bc uses supervised learning to mimic expert driving behaviors. previous works have applied bc in the carla simulator but did not fully address the challenges of traffic light compliance. our approach enhances bc by integrating a kernel density estimator to adjust training sample weights based on action density, thereby improving the learning of rare but critical actions such as stopping at red lights and accelerating at green lights, specially in scenarios of scarce number of expert demonstrations. using bev inputs, which provide an abstract top-down view of the driving environment, our method simplifies the policy learning process. the trained convolutional neural network (cnn) outputs steering and acceleration actions based on these bev inputs and additional state variables. experimental results in the carla simulator demonstrate that our weighted bc method significantly improves driving performance, achieving higher route completion compared to standard bc. this weighted approach proved to be crucial in learning correct driving behaviors, particularly in test environments not encountered during training, highlighting its potential for enhancing autonomous vehicle navigation. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a deep network system for simulated autonomous driving using behavioral cloning chapter © 2019 deep learning-based self-driving car simulator: development and evaluation chapter © 2026 bev-fused imitation and reinforcement learning for autonomous driving planning article 14 october 2025 1 introduction autonomous driving has traditionally been implemented using separate modules for perception, planning, and control, organized in a sequential pipeline [13, 15, 16, 21]. however, learning-based approaches have been gaining popularity as they attempt to address the complexities of autonomous driving in various scenarios, whether in simulation or real-world environments. most of these learning-based approaches rely on behavior cloning (bc), which employs supervised learning on a collection of expert demonstrations gathered offline [2, 3, 6, 7, 19]. for example, a human driver generates a set of input (observations) and corresponding desired output pairs. codevilla [6, 7] employed bc for autonomous driving in the carla simulator. he collected a large dataset of human driving and augmented it using image processing techniques to train end-to-end policies conditioned on the desired route. further improvements in bc were introduced in [4], where a large deep resnet network was used for feature extraction. this approach combined data from cameras, lidar, and radar to create comprehensive feature maps. the work also incorporated a probabilistic motion planner to account for uncertainties in the trajectory. couto [11] has shown results on behavior cloning on the carla simulator, but the agent did not have to respect traffic lights. in this work, we investigate an approach where the agent is able to navigate the city while respecting traffic lights, i.e., breaking when the light is red, and crossing the traffic light when it is green. only relying on regular bc to train an agent is not enough in a scenario with limited amount of training routes and where the agent needs to break at the red light and accelerate from stillness when light turns to green. to solve that, we employ a kernel density estimator [18] to estimate the actions density from the expert dataset. then, the training samples are weighted in the bc loss function individually by setting the weight equal to the inverse of the actions’ density. in this way, we can account for the underrepresented actions in the expert set, i.e., those happening at traffic light transitions. this weighting is needed because we are considering only eight routes as training data. instead of learning directly from raw camera pixels as in [11, 12], the agent in this work has structured image inputs that are more abstract and meaningful to the navigation task. to decouple feature representation from policy learning [8, 14] has its advantages as potentially easing up the learning task. in our context, this decoupling is accomplished by using a mid-level representation of the vehicle’s raw camera images. for example, bird’s-eye view (bev) representations of the road ahead have been used as mid-level inputs for trajectory generation and motor control networks in [2] and [20], respectively. in bev representation, the camera scene is projected onto a top-down view, segmenting areas of interest, like vehicle lanes and sidewalks, into different colors [21]. by using bev as input for the control policy, the learning process can focus on the navigation problem, leveraging a more abstract and simplified view of the road ahead. the policy here is represented by convolutional neural networks, mapping the bev image to control actuators such as steering and acceleration. experimental results in the carla simulator demonstrate that our weighted bc method significantly improves driving performance, achieving higher route completion compared to standard bc. this weighted approach proved to be essential in learning correct driving behaviors, particularly in test environments not encountered during training. in sect. 2, we present the bird’s-eye view representation employed as input to the convolutional neural network, the behavior cloning loss for the agent, and the kernel density estimation used to generate the weights per sample for the modified loss function. the agent architecture is presented in sect. 3. finally, the experiments and conclusions are shown in sects. 4 and 5. 2 methods 2.1 bird’s-eye view bev representation the bird’s-eye view of a vehicle represents its position and movement in a top-down coordinate system [2]. the vehicle’s location, heading, and speed are represented by \(p_t\), \(\theta _t\), and \(s_t\) respectively. the top-down view is defined so that the agent’s starting position is always at a fixed point within an image (the center of it). furthermore, it is represented by a set of images of size \(w \times h\) pixels, at a ground sampling resolution of \(\phi \) meters/pixel. the bev of the environment moves as the vehicle moves, allowing the agent to see a fixed range of meters in front of it. for instance, the bev representation for the vehicle whose three frontal cameras are shown in fig. 1(a) is given in fig. 1(b), where the desired route, drivable area and lane boundaries are shown as three bev channels. besides, other three bev channels for the traffic lights representing the state of the traffic lights (green or red) at different timesteps are shown in fig. 1(c). figure 1(d) shows the integrated six-channel bev image used as input to the agent. fig. 1. a) images from three frontal cameras positioned on the left, center, and right side of the vehicle. these images were taken after the initial interactions of the agent in the carla simulation environment. each camera captures a 256\(\times \)144 rgb image. the corresponding bird’s-eye view channels that our agent uses are shown in (b), (c) and (d), computed at the same moment as shown in (a). b) three bev 192\(\times \)192 channels that represent, from left to right, the desired route, drivable area, and lane boundaries. c) bev’s traffic lights channels at timesteps (–1, –9, –16), i.e., representing the past states of the traffic light channel (grey color is red light off; white color means red ligh on). d) the final image displays all three channels combined in different colors. (color figure online) full size image 2.2 behavior cloning (bc) behavior cloning is a supervised learning approach used in imitation learning, where the goal is to learn a policy that mimics the behavior of an expert. in this work, we want to learn a policy for an autonomous vehicle that outputs two actions (acceleration, steering) based on a bird-eye-view (bev) representation of the environment and additional scalar state variables as input. for this reason we need to train a cnn to output these two continuous actions based on input from the bev and the state variables. the loss function for behavior cloning is typically based on the difference between the actions predicted by the learned policy and the actions taken by the expert in the training data. let: \( \pi _{\theta }(s) \) be the policy parameterized by \(\theta \), which maps a state \( s \) to an action \( a \); and \( \mathcal {d} = \{(s_i, a_i)\}_{i=1}^n \) be the dataset of \(n\) state-action pairs collected from the expert, where \( s_i \) is a state and \( a_i \) is the corresponding action taken by the expert. although the behavior cloning loss function can be defined as the mean squared error (mse) between the predicted actions and the expert actions (or the cross-entropy loss if the actions are discrete), in this work, we model a stochastic policy and, for that reason, the policy is represented by a beta distribution. the beta distribution is suitable for modeling variables that are bounded within an interval, such as acceleration and steering actions within \([-1, 1]\). given the actions \(a_i = (a_{i}^\textrm{acc}, a_{i}^\textrm{steer})\) and the policy \(\pi _{\theta }(s_i) = (\alpha _{i}^\textrm{acc}, \beta _{i}^\textrm{acc}, \alpha _{i}^\textrm{steer},\) \(\beta _{i}^\textrm{steer})\), where \(\alpha \) and \(\beta \) are the shape parameters of the beta distribution, the loss function will be based on the negative log-likelihood of the beta distribution. the probability density function of the beta distribution [10] is: $$\begin{aligned} f(x; \alpha , \beta ) = \frac{x^{\alpha -1} (1-x)^{\beta -1}}{b(\alpha , \beta )}, \end{aligned}$$ where \(b(\alpha , \beta )\) is the beta function, a normalization constant to ensure that the total probability is 1. to transform actions in \([-1, 1]\) to \([0, 1]\), we use: $$\begin{aligned} a_i' = \frac{a_i + 1}{2} \end{aligned}$$ the negative log-likelihood loss for the beta distribution can then be written as: $$\begin{aligned} \mathcal {l}(\theta ) = -\frac{1}{n} \sum _{i=1}^n \left[ \log f\left( \frac{a_{i}^\textrm{acc} + 1}{2}; \alpha _{i}^\textrm{acc}, \beta _{i}^\textrm{acc} \right) + \log f\left( \frac{a_{i}^\textrm{steer} + 1}{2}; \alpha _{i}^\textrm{steer}, \beta _{i}^\textrm{steer} \right) \right] \end{aligned}$$ (1) this loss function accounts for the probabilistic nature of the beta distribution, ensuring that the model learns to predict parameters (\(\alpha , \beta \)) for the distribution of actions rather than the actions directly. this approach allows for modeling uncertainty and variability in the predicted actions. 2.3 kernel density estimation a kernel density estimator (kde) is a non-parametric way to estimate the probability density function (pdf) of a random variable. for each data point in the sample, the kernel function is centered at that point. the kde is then computed by summing up all these kernel functions and normalizing by the number of data points. mathematically, the kde for a point \(x\) is given by: $$\begin{aligned} \hat{f}(x) = \frac{1}{n h} \sum _{i=1}^{n} k \left( \frac{x x_i}{h} \right) , \end{aligned}$$ (2) where \( \hat{f}(x) \) is the estimated density at point \( x \), \( n \) is the number of data points, \( h \) is the bandwidth, \( x_i \) are the data points, and \( k \) is the kernel function. the gaussian kernel, often used in kde, is a type of kernel function that assigns weights to data points based on their distance from the point where the density is being estimated. the gaussian kernel is defined by the standard normal distribution, and it ensures that closer points have a higher influence on the density estimate than points further away. the mathematical form of the gaussian kernel \( k \) is: $$\begin{aligned} k(u) = \frac{1}{\sqrt{2\pi }} e^{-\frac{u^2}{2}}, \end{aligned}$$ where \( u \) is the standardized distance between the point where the density is being estimated and the data point, scaled by the bandwidth \( h \). in this work, the kde is used to weigh the training samples in the bc loss in (1). by using the inverse of the kernel density estimation of the actions as weights, we give more importance to less frequent actions. this approach can help the model learn better on underrepresented parts of the action space. the resulting weighted bc loss function can be written as: $$\begin{aligned} \mathcal {l}(\theta ) = -\frac{1}{n} \sum _{i=1}^n w_i \left[ \log f\left( \frac{a_{i}^\textrm{ac} + 1}{2}; \alpha _{i}^\textrm{ac}, \beta _{i}^\textrm{ac} \right) + \log f\left( \frac{a_{i}^\textrm{steer} + 1}{2}; \alpha _{i}^\textrm{steer}, \beta _{i}^\textrm{steer} \right) \right] \end{aligned}$$ (3) where: \( w_i = \frac{1}{\textrm{kde}(a_i)} \); and \( \textrm{kde}(a_i) \) is the kernel density estimation evaluated at the action \( a_i \). the kde is estimated for the actions in the training set. then the weights \(w_i\) are calculated as the inverse of the kde for each action. this approach ensures that the loss function accounts for the density of actions in the training set, giving higher importance to less frequent actions. 3 agent 3.1 input representation the input \( s \) to the agent’s policy is a six-channel 192\(\,\times \,\)192 image produced by a module in carla simulator that has access to the city map. this image represents the mid-level bird’s-eye view of the vehicle’s current position. in this work, the bev contains three channels, the road, the route and the lane, and also three more channels representing the history of the traffic lights at time steps –16, –9 and –1. additionally, the agent receives the current speed of the vehicle and the last values of the policy actuators (previous acceleration and steering). these additional inputs are incorporated further down in the network, specifically in the first fully connected layer. 3.2 output representation the vehicle in carla has three actuators: steering \([-1, 1]\), throttle \([0, 1]\), and brake \([0, 1]\). our agent’s action space is \(\textbf{a} \in [-1, 1]^2\), where the two components correspond to steering and acceleration. braking is implied when acceleration is negative, ensuring the agent cannot brake and accelerate simultaneously [17]. instead of using the gaussian distribution for the policy’s actions, we employ the beta distribution \(\mathcal {b}(\alpha , \beta )\). the beta distribution’s bounded support is suitable for real-world applications like autonomous driving, where actions are naturally constrained [17]. this approach avoids the need for clipping or squashing inputs to enforce constraints, allowing explicit computation of the policy loss \(\mathcal {l}_p\). the beta distribution also handles extreme driving scenarios, such as sharp turns and sudden braking, effectively. the parameters \(\alpha \) and \(\beta \), which are outputs of the policy neural network \(\pi _\theta \), control the shape of the distribution, enabling the policy to adapt to these critical behaviors. 3.3 network architecture in this work, we train a network architecture, shown in fig. 2, to learn to control an autonomous vehicle, using steering and acceleration actuators, based on the current bev and some state variables as inputs. the convolutionary neural network (cnn) maps the bev into an embedding vector of size 256. similarly, the scalar state variables are embedded into a vector of size 256 by a standard mlp (multi-layer perceptron layer). together they form the current representation of the environment the agent faces. this concatenated embedding vector is processed by another mlp to map to the desired action. fig. 2. network architecture. the bev (state) input is processed by a cnn (mlp) to create an image (state) encoding. both image and state encodings are concatenated and fed into a final mlp that outputs the actions. full size image figure 3 shows the detailed network architecture. fig. 3. detailed network architecture. the bev input consists of 6 channels of 192\(\,\times \,\)192 resolution, while the state input correponds to the last executed actions, and the current speed measurements. the final policy output corresponds to the two parameters \(\alpha \) and \(\beta \) of the beta distribution for both acceleration and steering (thus, the two heads). full size image 4 experiments in this section, the proposed bc with weigthed loss function is compared with the standard bc approach for autonomous vehicle navigation tasks in urban scenarios using the carla simulator. 4.1 dataset generation the environment and trajectories are sourced from the carla leaderboard evaluation platform [1]. specifically, the town01 environment from this platform, along with ten predefined trajectories, is used to create the expert dataset. the expert dataset is generated using a deterministic agent that navigates with a dense point trajectory and a classic pid controller [5]. the dense point trajectory offers numerous points at a high resolution, while a sparse point trajectory has significantly fewer points, giving the agent only a general direction. therefore, the dense point trajectory is employed to generate the expert’s training data, whereas the sparse point trajectory is used by the agent for broader guidance. thus, the agent has no access to the dense point trajectory. besides, the expert breaks at the detection of red traffic lights and accelerates when the light turns back to green. in fig. 4, one of the 10 routes executed by the expert to create the labeled training set of demonstrations is illustrated. the line that starts in yellow and ends in red represents the desired trajectory (not visible to the agent as it is). the sparse trajectory appears as yellow dots, generated every 50 m or when the vehicle is about to change its movement (e.g., from straight to turn and vice versa). the ten trajectories in the expert set were recorded at a rate of 10 hz, producing 10 observation-action pairs per second. for the shortest route of 1480 samples (average route of 2129 samples), this equates to 2.5 min (3.5 min) of simulated driving. altogether, the ten trajectories resulted in 21,287 labeled samples (30 gb of uncompressed data). the entire set corresponds to approximately 36 min or 8 km of driving. fig. 4. (a) the town01 environment of the agent includes one of the routes used by the expert to collect data. this highlighted path measures 740 m, featuring 20 points in the sparse trajectory (indicated by yellow dots) and 762 points in the dense point trajectory (not shown). (b) the town02 is used to evaluate the trained agent in a new environment. (color figure online) full size image 4.2 training and inference the network shown in fig. 3 is trained with 8 routes from the expert dataset of town01 environment, where the agent must also respect traffic lights. the first agent (weighted bc) weighs the samples according to the density of the actions using (3) as loss function, whereas the second agent (bc) considers all actions with same weight, i.e., it removes the higher importance of less frequent actions, using (1) as the conventional bc loss function. 4.3 evaluation although training is offline in town01 environment, we still evaluate both bc and weighted bc agents as their model’s parameters are updated. every eight training epochs, both agents are evaluated using all ten routes from the same training town (fig. 5). as four random runs are executed for each agent, the mean and standard deviation of the number of incomplete routes is plot in fig. 5. one incomplete route means that the vehicle committed an infraction before completing the route. as the total number of routes is ten for town01, both bc and weighted bc start at 10, the minimum performance. as training evolves, both agents improve the performance by committing less infractions, such as going out of the correct lane or passing a red light, but the weighted bc agent attains the minimum error faster, i.e., it can complete all routes without any infractions and respecting traffic lights. fig. 5. evaluation of both bc policies (weighted and regular) throughout its learning epochs. at every eight training epochs, both bc and weighted bc agents are evaluated in all ten routes of town01. the vertical axis presents the number of incomplete routes. four runs were executed for each agent, with mean and standard deviation shown by the solid line and the shaded area, respectively. in the beginning, both agents commit infractions in all routes and, thus, the number of incomplete routes is ten since one infraction ends the route uncompleted. the weighted bc agent is able to achieve zero infractions by the end of training, unlike the bc agent. full size image to understand what the action’s density given by kde and, consequently, the weights in the bc loss function, look like for the first route in town01 (shown in fig. 4), we record both actuators’s values, acceleration and steering, from our expert as it follows the route as well as the corresponding logarithm of the action’s density for each timestep of the route. these values are plot in fig. 6, where we can see that different actions will produce different weights in the loss function since some of them are less frequent than others in the expert dataset. the highest action density corresponds to going straight with constant acceleration and zero steering, for instance, between timesteps 470 and 900. the least frequent action happens after a traffic light turns from red to green, which corresponds to a higher, but shorter, acceleration period. fig. 6. logarithm of the density of actions, given by kde, during navigation of a route in town1, shown in fig. 4. the expert encounters three red traffic lights, during which it breaks (acceleration is \(-1\)), and subsequently accelerates as soon as it turns green. two left turns and two right turns can be observed by examining the steer actuator of the expert. (color figure online) full size image the driving performance for both agents as well as the expert is presented in table 1. the understanding of the \(\uparrow \) or \(\downarrow \) in the table is as follows. for instance, the lower (\(\downarrow \)) the red light infraction, which is #/km or the number of infractions per kilometer, the better. in the same vein, the higher the infraction penalty (or the driving score), the better is the performance. the expert has maximum driving score, being able to complete all routes with no red light infraction in both town01 and town02. the number of crossed traffic lights is just a non-performance metric, just counting the times that the vehicle cross them. an equivalent result is achieved by weighted bc in town01, the training environment, but not by regular bc, which presents a mean driving score of 81. in town02, a test environment, weighted bc performs significantly better than bc. videos with the agent controlling the vehicle in carla are available at https://sites.google.com/view/weighted-bc, comparing both bc and weighted bc agents. table 1. driving performance and infraction analysis for different agents on town01 and town02.full size table 5 conclusion in this work, we proposed a weighted behavior cloning approach that improves agent learning by giving higher weights to underrepresented actions in the expert training set of driving behaviors. the application consisted of route navigation in two simulated cities in the carla realistic simulator for autonomous vehicles, while respecting traffic lights. as the action space of the expert is not uniformly visited, and the expert training dataset is somewhat limited (only 10 routes in town01), a kernel density estimator was employed to infer the action density which, in turn, was converted to a weight per observation-action pair. in our investigated scenario, the weighted loss function for bc proved to be crucial to learn the correct behavior, showing significantly better driving performance than regular bc, specially for the test city town02 not used during training. future work include extensions to cities with pedestrians and other vehicles and more advanced behavior cloning approaches such as implicit behavioral cloning [9] to account for multimodal action distributions. references (2020). https://leaderboard.carla.org/ bansal, m., krizhevsky, a., ogale, a.: chauffeurnet: learning to drive by imitating the best and synthesizing the worst (2018). https://doi.org/10.48550/arxiv.1812.03079. https://arxiv.org/abs/1812.03079 bojarski, m., et al.: end to end learning for self-driving cars (2016). https://doi.org/10.48550/arxiv.1604.07316. https://arxiv.org/abs/1604.07316 cai, p., wang, s., sun, y., liu, m.: probabilistic end-to-end vehicle navigation in complex dynamic environments with multimodal sensor fusion. ieee rob. autom. lett. 5(3), 4218–4224 (2020). https://doi.org/10.1109/lra.2020.2994027 article  math  google scholar  chen, d., zhou, b., koltun, v., krähenbühl, p.: learning by cheating (2019) google scholar  codevilla, f., müller, m., dosovitskiy, a., lópez, a.m., koltun, v.: end-to-end driving via conditional imitation learning. in: 2018 ieee international conference on robotics and automation (icra), pp. 1–9 (2017). https://api.semanticscholar.org/corpusid:3672301 codevilla, f., santana, e., lopez, a., gaidon, a.: exploring the limitations of behavior cloning for autonomous driving. in: 2019 ieee/cvf international conference on computer vision (iccv), pp. 9328–9337 (2019). https://doi.org/10.1109/iccv.2019.00942 couto, g.c.k., antonelo, e.a.: hierarchical generative adversarial imitation learning with mid-level input generation for autonomous driving on urban environments. arxiv preprint arxiv:2302.04823 (2023) florence, p., et al.: implicit behavioral cloning. in: conference on robot learning, pp. 158–168. pmlr (2022) google scholar  gelman, a., carlin, j.b., stern, h.s., rubin, d.b.: bayesian data analysis. chapman and hall/crc, boca raton (1995) book  math  google scholar  karl couto, g.c., antonelo, e.a.: generative adversarial imitation learning for end-to-end autonomous driving on urban environments. in: 2021 ieee symposium series on computational intelligence (ssci), pp. 1–7 (2021). https://doi.org/10.1109/ssci50451.2021.9660156 liang, x., wang, t., yang, l., xing, e.: cirl: controllable imitative reinforcement learning for vision-based self-driving. in: ferrari, v., hebert, m., sminchisescu, c., weiss, y. (eds.) eccv 2018. lncs, vol. 11211, pp. 604–620. springer, cham (2018). https://doi.org/10.1007/978-3-030-01234-2_36 chapter  google scholar  montemerlo, m., et al.: junior: the stanford entry in the urban challenge. j. field rob. 25, 569–597 (2008). https://doi.org/10.1002/rob.20258 ota, k., jha, d.k., kanezaki, a.: training larger networks for deep reinforcement learning. arxiv preprint arxiv:2102.07920 (2021) paden, b., cáp, m., yong, s.z., yershov, d.s., frazzoli, e.: a survey of motion planning and control techniques for self-driving urban vehicles. ieee trans. intell. veh. 1, 33–55 (2016) article  math  google scholar  parizotto, l.m.s., antonelo, e.a.: cone detection with convolutional neural networks for an autonomous formula student race car. in: 26th abcm international congress of mechanical engineering (cobem 2021) (2021) google scholar  petrazzini, i.g., antonelo, e.a.: proximal policy optimization with continuous bounded action space via the beta distribution. in: 2021 ieee symposium series on computational intelligence (ssci), pp. 1–8. ieee (2021) google scholar  silverman, b.w.: density estimation for statistics and data analysis. routledge, abingdon (2018) google scholar  xu, h., gao, y., yu, f., darrell, t.: end-to-end learning of driving models from large-scale video datasets (2016). https://doi.org/10.48550/arxiv.1612.01079. https://arxiv.org/abs/1612.01079 zhang, z., liniger, a., dai, d., yu, f., van gool, l.: end-to-end urban driving by imitating a reinforcement learning coach. in: 2021 ieee/cvf international conference on computer vision (iccv), pp. 15202–15212 (2021). https://doi.org/10.1109/iccv48922.2021.01494 zhu, h., yuen, k.v., mihaylova, l., leung, h.: overview of environment perception for intelligent vehicles. ieee trans. intell. transp. syst. 18(10), 2584–2601 (2017). https://doi.org/10.1109/tits.2017.2658662 article  math  google scholar  download references author information authors and affiliations automation and systems engineering department, federal university of santa catarina, florianópolis, brazil eric aislan antonelo, gustavo claudio karl couto & pedro henrique fernandes faculty of science and engineering, information technology, åbo akademi university, turku, finland christian möller authors eric aislan antoneloview author publications search author on:pubmed google scholar gustavo claudio karl coutoview author publications search author on:pubmed google scholar christian möllerview author publications search author on:pubmed google scholar pedro henrique fernandesview author publications search author on:pubmed google scholar corresponding author correspondence to eric aislan antonelo . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper antonelo, e.a., couto, g.c.k., möller, c., fernandes, p.h. (2025). investigating behavior cloning from few demonstrations for autonomous driving based on bird’s-eye view in simulated cities. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_11 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_11 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords autonomous vehicles expert demonstrations imitation learning kernel density estimation carla simulator publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature explainability of covid-19 classification models using dimensionality reduction of shap values | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper explainability of covid-19 classification models using dimensionality reduction of shap values conference paper first online: 12 october 2023 pp 415–430 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) explainability of covid-19 classification models using dimensionality reduction of shap values download book pdf download book epub daniel matheus kuhn9, melina silva de loreto10, mariana recamonde-mendoza9, joão luiz dihl comba9 & … viviane pereira moreira9  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 594 accesses abstract the critical scenario in public health triggered by covid-19 intensified the demand for predictive models to assist in the diagnosis and prognosis of patients affected by this disease. this work evaluates several machine learning classifiers to predict the risk of covid-19 mortality based on information available at the time of admission. we also apply a visualization technique based on a state-of-the-art explainability approach which, combined with a dimensionality reduction technique, allows drawing insights into the relationship between the features taken into account by the classifiers in their predictions. our experiments on two real datasets showed promising results, reaching a sensitivity of up to 84% and an auroc of 92% (95% ci, [0.89–0.95]). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others prognosis patients with covid-19 using deep learning article open access 26 march 2022 improving cnn predictive accuracy in covid-19 health analytics article open access 14 august 2025 using explainable machine learning to characterise data drift and detect emergent health risks for emergency department admissions during covid-19 article open access 26 november 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data science data analytics data mining machine learning statistical learning 1 introduction the development of predictive models for the diagnosis and prognosis of patients has been recently investigated by numerous works [17]. the increasing interest in clinical predictive models may be explained by factors such as the growing capacity of data collection, the positive impact on decision-making and clinical care, and the great advances in machine learning (ml). these models aim to assist in the various challenges faced by health professionals while treating patients, optimizing clinical management processes, and allocating hospital resources, ultimately promoting better patient care. the covid-19 pandemic was a critical public health emergency worldwide, given the sudden need for many hospital beds and increased demand for medical equipment and health professionals. this scenario intensified the demand for resources and, consequently, the interest and need in using predictive models to support decision-making for diagnosis and prognosis in the context of covid-19. as discussed by previous works [2], a considerable number of papers related to covid-19 were published in a short period, including papers proposing ml applications for covid-19 prognosis prediction. among the models for prognosis prediction, we can cite predictive models that aim to anticipate the patient’s outcome (e.g., hospital discharge or death), the severity of the disease, the hospitalization length, the evolution of the patient’s clinical condition during hospitalization, the need for intensive care unit (icu) admission or interventions such as intubation, and the development of complications (e.g., cardiac problems, thrombosis, acute respiratory syndrome) [16, 17]. in this work, we are particularly interested in mortality prediction models. mortality prediction models help recognize patients with higher risks of poor outcomes at the time of admission or during hospitalization, providing support to more effective triage and treatment of ill patients and reducing death rates. although previous studies have shown that ml algorithms achieved promising results in the mortality prediction task [2, 9, 18], the lack of explainability of the features taken into account by the predictive models has been pointed out as a limiting factor for their application in the clinical routine [9]. this work aims to evaluate several ml algorithms for predicting covid-19 mortality and explore state-of-the-art explainability approaches to understand the factors considered during prediction. we developed a visualization technique based on shap values [11] combined with a dimensionality reduction method that draws insights into the relationship between the features taken into account by the predictive models and their correct and incorrect predictions. to assess our approach, we conducted a retrospective study of patients diagnosed with covid-19, analyzing data collected during hospitalization in two hospitals. we also analyze the ability of classifiers to identify patients with a high probability of dying at different time intervals. our main contributions can be summarized as follows: (i) evaluation of six ml algorithms for the mortality prediction task; (ii) proposal of a visualization technique based on shap values to assist in explainability; and (iii) experiments with real data from two hospitals. our findings showed that, in general, deceased patients were older. in addition, age, oxygen saturation, heart disease, and hemogram information also tended to have high contributions in the predictions. logistic regression, adaboost, and xgboost models recurrently achieved the highest sensitivity scores among the evaluated classifiers. 2 related work this section discusses existing works that addressed mortality prediction for covid-19 patients based on the information available at hospital admission. we considered works evaluating classical risk scores (including those designed especially for covid-19 patients and those that precede the pandemic) and ml classifiers. several techniques have been employed to address mortality prediction, such as using risk scores based on classical statistical approaches and popular classification algorithms [5, 9, 20]. covino et al.  [5] evaluated six physiological scoring systems recurrently used as early warning scores for predicting the risk of hospitalization in an icu and the risk of mortality—both tasks considering up to seven days. the rapid emergency medicine score obtained the best results with an auroc of 0.823 (95% ci, [0.778–0.863]). zhao et al.  [20] developed specific risk scores for predicting mortality and icu admission. the scores were developed based on logistic regression and reached an auroc of 0.83 (95% ci, [0.73–0.92]). another score specifically designed to predict mortality risk was the 4c mortality score [9]. it uses generalized additive models to categorize continuous feature values and pre-select predictors, followed by training a logistic regression model. the 4c score yielded an auroc of 0.767 (95% ci, [0.760–0.773]), which was outperformed by the xgboost algorithm by a small margin. still, the authors claim that their score is preferable due to the difficulty in interpreting the predictions made by xgboost. paiva et al.  [13] also evaluated classification algorithms for predicting mortality, including a transformer (fnet), convolutional neural networks (cnn), boosting algorithms (lightgbm), random-forest (rf), support vector machines (svm), and k-nearest-neighbors (knn). in addition, an ensemble meta-model was trained to combine all the aforementioned classifiers. the ensemble model achieved the best results, with an auroc of 0.871 (95% ci [0.864–0.878]). to explain the predictive models, the study contemplated the application of shapley additive explanations (shap) et al.  [11]. araújo et al.  [3] developed risk models to predict the probability of covid-19 severity. to perform the study, the authors collected blood biomarkers data from covid-19 patients. the models were developed with a lightgbm-based implementation, which achieved an average auroc of 0.91 ± 0.01 for mortality prediction. the authors of this work apply a visualization technique similar to the one proposed in our work. the main difference is that they use it only to assess the model’s ability to separate the targeted classes. conversely, in our work, we use visualization to analyze the relationship between the features taken into account by the classifiers and their correct and incorrect predictions. the datasets used in the studies typically range from a single hospital unit with a few hundred patients to datasets comprising thousands of hospitalization records from hundreds of hospital units in several countries. covino et al.  [5] and zhao et al.  [20] carried out their studies based on datasets containing 334 and 641 patients, respectively, from a single hospital unit. paiva et al.  [13] used a dataset with 5,032 patients from 36 hospitals in 17 cities in 5 brazilian states, and araújo et al.  [3] used a dataset with 6,979 patients from two brazilian hospitals. to develop the 4c mortality score, knight et al.  [9] used a large dataset with 57,824 patients collected from 260 hospitals in england, scotland, and wales. although the datasets used in the different studies do not contain the same features, there is a recurrence among the features considered most relevant by the predictive models. age is the most frequently cited variable. other features include oxygen saturation, respiratory rate, comorbidities, heart failure, lactate dehydrogenase (ldh), c-reactive protein (crp), and consciousness level metrics (e.g., glasgow coma scale). our study complements the aforementioned works by evaluating six algorithms in new datasets from two hospitals. in this work, in addition to applying the explainability approach, we developed a visualization technique that allows drawing insights into the relationship between the features taken into account by the classifiers and their correct and incorrect predictions. 3 materials and methods this work aims to predict the risk of covid-19 mortality during patient hospitalization based on information available at the time of admission. we approach this task through the use of ml classification algorithms. our method comprises five steps, depicted in fig. 1. our input dataset goes through preprocessing and cleaning, feature selection, then classification. the results of the classifiers are evaluated and submitted to the explainability approach. the next sections describe our data and each step in greater detail. fig. 1. pipeline employed in this work. full size image 3.1 patient records and preprocessing the study involved patients hospitalized in two institutions: hmv and hcpa. hmv is a private hospital. hcpa is a public tertiary care teaching hospital academically linked to a university. the project was approved by the institutional research ethics committee of both institutionsfootnote 1. informed consent was waived due to the retrospective nature of the study. the collected data did not include any identifiable information to ensure patient privacy. patients included in the study are adults (age \(\ge \) 18) diagnosed with covid-19 using the reverse transcriptase polymerase chain reaction (rt-pcr) test. the features include clinical and demographic data: age, sex, ethnicity, comorbidities (hypertension, obesity, diabetes, etc.), lab tests (arterial blood gas, lactate, blood count, platelets, c-reactive protein, d-dimer, creatine kinase (ck), creatinine, urea, sodium, potassium, bilirubin), need for icu admission, type of ventilatory support during hospitalization and icu period (o2 catheter, non-invasive ventilation, high-flow nasal cannula, invasive mechanical ventilation), start and end date of hospitalization, as well as hospitalization outcome. we performed the shapiro-wilk test on continuous features. the results showed that our features are not normally distributed (\(\alpha = 0.01\)). patients without information on the outcome of hospitalization (dependent variable) were not considered in this study. at the hcpa hospital, 318 patients had no record of outcome due to transfer to other hospitals and 6 due to withdrawal or evasion of treatment. for the hmv hospital, patients without outcome information were disregarded during data collection. variables with missing values greater than 40% were also not considered. in the hmv dataset the variable brain natriuretic peptide was discarded, as well as the variables weight, height and blood type for hcpa. the variables troponin from hmv and fibrinogen from hcpa were the only exceptions to this rule, because although they had a missing value rate of 44.67% and 41.78%, respectively, there is previous evidence regarding their importance as indicators of severity for covid-19. in hmv, 23 patients (5 deaths and 18 discharges) were disregarded because they had a missing value index greater than 30%, while in hcpa, 258 (93 deaths and 165 discharges) were not considered. in hmv and hcpa, 7 and 19 patients, respectively, had two hospital admissions (readmissions). among these, 7 patients from hmv and 13 patients from hcpa were readmitted within 30 days of the discharge from the index hospitalization. for these patients, we used the admission records collected at the index hospitalization with the readmission outcome as the label (patients who were discharged, returned within 30 days, and died, were considered deaths). for the other 6 patients from hcpa who were readmitted after 30 days, we used only the record of the index hospitalization. hmv datasets. contain the anonymized records of patients confirmed for covid-19 admitted at hmv between march 2020 and june 2021. there are 1,526 records, among which 58.78% refer to male patients. the dataset covers both ward patients and patients admitted to the icu. the data collection process was conducted by health professionals, who manually transcribed the information contained in the patients’ medical records into a form containing the features previously established for the study (clinical information available at hospital admission). patients without outcome information were not considered during the data transcription process. thus, our study did not include hospitalization records in which patients were transferred to other hospitals and/or dropout cases. automatic verification processes were applied to identify inconsistencies. we developed classifiers considering two scenarios. in the first scenario, data of all patients are considered, including ward and icu patients (\(\text {hmv}_{all}\)). in the second scenario, only patients admitted to the icu are considered (\(\text {hmv}_{cti}\)). in both cases, we focus on information available during admission. \({\textbf {hcpa}}_{\boldsymbol{cti}}\) dataset contains anonymized records of 2,269 patients with confirmed covid-19 admitted to the icu of hcpa between march 2020 and august 2021. male patients represent 54.69% of the dataset. the data was collected directly from the hospital database. the outcome of interest was mortality during hospitalization, thus, records were categorized as alive or deceased according to data provided by the hospitals. details of the datasets after the preprocessing step are shown in table 1. table 1. statistics of the datasetsfull size table 3.2 feature selection each raw instance was represented by roughly 60 and 70 features for hmv and hcpa datasets, respectively. features with more than 40% of missing values were discarded. to pre-select features with good predictive power and avoid multicollinearity among predictive features, we applied the correlation-based feature selection approach (cfs). this approach evaluates subsets of features prioritizing features highly correlated with the classes and uncorrelated with each other [7]. feature selection was performed only over the training instances to avoid data leakage. 3.3 classification we applied six classification algorithms: (i) logistic regression (lr), three tree-based algorithms, namely (ii) j48, (iii) randomforest (rf), and (iv) gradient boosting tree xgboost (xgb), (v) a boosting-based ensemble (adaboostm1), and (vi) multi layer perceptron (mlp). in supervised classification approaches, class imbalance refers to the inequality of the proportions of instances for each class. class imbalance is frequently observed in healthcare data, where the class of interest is often the minority (e.g., disease occurrence or poor diagnosis). when training classification algorithms on imbalanced data without the necessary care, these algorithms tend to present biased results, prioritizing the performance of the majority class. in our datasets, the outcome variable presents, to some degree, an imbalance between the classes (alive and deceased). the greatest imbalance is observed in the \(\text {hmv}_{all}\) dataset, with approximately 7.38 patients with an alive outcome for each patient with a deceased outcome. considering only patients admitted to the icu, the imbalance is 2.2 and 1.67 alive for each deceased, for the \(\text {hmv}_{cti}\) and \(\text {hcpa}_{cti}\) datasets, respectively. to deal with class imbalance, we applied the cost-sensitive matrix approach, in which the classification errors (false-negatives (fn) or false-positives (fp)) are weighted depending on the task to be addressed [14]. since the \(outcome=deceased\) is our positive class, we applied a cost matrix that increased the penalty for fn. the cost matrices applied considered the imbalance between classes so that the penalty for false negatives was defined as \(pen = \frac{nc\_maj}{nc\_min}\) where \(nc\_maj\) is the number of instances of the majority class and \(nc\_min\) is the number of instances of the minority class. 3.4 evaluation metrics. with the aim of evaluating the classification models, we calculated several metrics, namely: sensitivity (also known as recall or true positive rate), specificity (or true negative rate), positive predictive value (ppv or precision), negative predictive value (npv), f1 (both for the positive class (f1\(_+\)) and macro-averaged (ma-f1) for both classes), cohen’s kappa, area under receiver operating characteristic curve (auroc), and area under precision-recall curve (auprc). using different metrics is important to interpret the results from different perspectives. resampling. we used the resampling technique bootstrap to evaluate the predictive models. for each dataset, we performed 100 iterations of resampling with replacement. for each iteration, several instances—equal to the number of instances in the dataset—were randomly selected to compose the training set, and the remaining instances to constitute the test set. thus, for each of the 100 iterations, the classifiers were trained on the training set and evaluated on the test set. at the end of the process, we take the means of the evaluation metrics. 3.5 explainability of classification results this section describes the proposed approach to help explain the features taken into account by the underlying model during inference. the goal is to extract insights into the relationship between the features utilized by the predictive models and their correct and incorrect predictions. the proposed approach corresponds to step four, shown in fig. 1 and its components are shown in fig. 2. the idea is similar to supervised clustering [10], in which shap values (not raw feature values) are used to cluster instances. however, instead of clustering, we use them to generate visual representations to inspect predictions. fig. 2. steps applied in explaining and interpreting the results returned by the predictive models. full size image first, starting from a generic dataset d, which in this case, could be any of our datasets, we use dimensionality reduction to understand aspects of linear separability of the data. it is expected that in the raw data, there are no clear separability patterns. rectangles 2.a and 3.a represent the visualization of the raw data using a t-sne projection [12]. we can see that the arrangement of instances does not constitute clusters, suggesting little linearity in the data. starting from the same dataset d, the proposed approach consists of the following steps: (2.b) training a classification algorithm and (3.b) applying the shap explanation over predictions. the training process in 2.b follows standard practices, such as handling missing values, applying techniques to deal with class imbalance, model evaluation, and calibration. in 3.b, we select instances from the test set and submit them to the shap approach. we obtain, for each feature value v of the instance i, their contribution values (shap values). in our experiments, we used kernelshap to estimate shap values, but other methods could also have been used (i.e., treeshap, linear shap [11], kernel shap method to handle dependent features [1], etc.). with the shap values, we create an instance \(i'\), which consists of the calculated shap values instead of the raw feature values. in the next step (4.b), we create a dataset c constituted by the instances \(i'\). finally, dimensionality reduction is applied to the shap-transformed instances in c to generate a 2d plot that gives insights into the separability of instances. this procedure is represented by step 5.b. we applied the t-sne dimensionality reduction (dr) technique in our tests, but other dr techniques could be used. the representation in 6.b is the result of the projection into 2d generated by t-sne. we observe that the instances may form well-defined clusters in the projection. for instance, certain regions may have a prevalence of a specific class, while others may show higher uncertainty. moreover, the projected points can be colored based on the feature values, which makes it possible to obtain important insights, such as the relationship between the features taken into account by the predictive models and their correct and incorrect predictions. 4 experimental evaluation in this section, we describe the experiments performed to answer the following research questions: rq1: is it possible to identify, among patients with covid-19, who is more likely to die from the disease? rq2: which features are taken into account by the classifiers for making predictions and how do they relate to correct/incorrect classifications? rq3: is the performance obtained with multivariable classifiers superior to the performance obtained with single-variable classifiers using only age as independent variable? 4.1 experimental setup and executions data preprocessing was performed using python v3.5 and the libraries pandas and numpy. the classifiers were implemented using the weka v3.9.6 api through the wrapper python-weka-wrapper3 v0.2.9footnote 2—with the exception of the gradient boosting tree, for which used the xgboost implementationfootnote 3. xgboost can handle missing values, so no data imputation was needed. for the other algorithms, the feature values were imputed with the median value from the training set. categorical features were imputed with mode values. the weights of the cost matrix were calculated as shown in sect. 3.3, considering the imbalance rate of the training set of each bootstrap iteration. to interpret the generated models, we use the shap libraryfootnote 4. shap relies on shapley values to explain the contribution that each variable makes to the output predicted by the model. specifically, we used the kernelshap algorithm, as it is model-agnostic. the perplexity parameter for the t-sne projection considered values between 10 and 40. empirical tests were performed in multiple executions to set the appropriate perplexity value. 4.2 results for mortality prediction (rq1) table 2 displays the scores achieved by each classifier. algorithms have better/worse performances across all metrics, with no clear winner. in our analysis, we give emphasis to sensitivity results, as the consequences of not identifying a patient with a high probability of dying are much more severe than wrongly predicting a patient into the alive class when a patient has a high probability of dying. considering only sensitivity, logistic regression is the winner, leading all scenarios. only in \(\text {hcpa}_{cti}\) xgboost reaches the same top score as lr. adaboostm1 has the second highest sensitivity values for \(\text {hmv}_{all}\) and \(\text {hmv}_{cti}\), and the third highest sensitivity value for \(\text {hcpa}_{cti}\). in all cases, j48 has the lowest sensitivity values. table 2. results for mortality predictionfull size table overall, the sensitivity values we obtain demonstrate the classifier’s promising ability to predict mortality based on the information available at the time of admission (answer to rq1). the average sensitivity for the classification algorithms is higher for the \(\text {hmv}_{cti}\) dataset (0.74) when compared to the \(\text {hcpa}_{cti}\) dataset (0.65). this difference may be explained by the individual characteristics of patients admitted to each hospital and the features available in the datasets for each hospital. a similar discussion is mentioned in futoma et al.  [6], which warns about the risks of excessively seeking the development of predictive models that generalize to various hospitals with different clinical protocols and units. our best auroc is higher than the ones reported in related work [5, 9, 13, 20] (see table 2). yet a direct comparison among approaches is not possible as they were applied to different datasets. 4.3 analysis of feature importance and explainability (rq2) we use the results of applying dimensionality reduction over shap values to help understand the factors considered by classifiers during predictions and answer rq2. figure 3 presents the global view of how features contribute to mortality prediction using the lr classifier on \(\text {hmv}_{all}\). each line refers to the contribution of a feature. each dot is an instance. low values for the feature are in blue and high values are in red. the contribution of the feature value is given by the horizontal displacement. the further the instance is positioned to the right, the greater the contribution to the deceased class. the further to the left, the greater the contribution of the feature value to the prediction of the alive class. for example, high values for age are associated with increased probability for the deceased class. the ten features with the highest contribution to the predictions were age, erythrocytes, bilirubin, oxygen saturation, neutrophils, troponin, heart disease, hemoglobin, creatinine, and central nervous system disease. using the lr classifier on \(\text {hcpa}_{cti}\), the ten features with the highest contribution were age, urea, red cell distribution width, d-dimer, mean corpuscular volume (mcv), lymphocytes, platelets, creatinine, erythrocytes, and troponin. figure 4 shows the t-sne plots generated using shap values. each point corresponds to a patient. the points are colored based on the values of different features to aid in interpreting the generated projections. each row is composed of three plots. the first plot displays the correct and incorrect predictions (true positive (tp), true negative (tn), false positive (fp) and false negative(fn)). the second and third plots show points colored by different features. fig. 3. global explanations based on lr classifier using shap for \(\text {hmv}_{all}\) full size image fig. 4. visualizations of the model’s predictions based on shap values – the x-axis corresponds to t-sne dimension 1 and the y-axis to t-sne dimension 2. full size image the t-sne plots reinforce the high importance attributed by the predictive models to the age feature. when analyzing the plots with the classifications and the colored plots based on the age of the patients (figs. 4b, 4e 4h, and 4k), it is possible to observe that recurrent fp errors (purple dots) and tp occur in the regions with a high concentration of older patients. on the other hand, fn errors (red dots) and tn are in regions with younger patients. figure 4c also shows a trend of moderate and low erythrocyte values for tp and fp patients. these findings are supported by previous studies, which showed that anemia is associated with poor prognosis in critically ill patients. figures 4d, 4e, and 4f refer to lr on the \(\text {hmv}_{cti}\) dataset. in fig. 4f, we can observe that patients with heart disease tend to be older. both features predispose those patients to be classified in the deceased class. this finding is in agreement with the related literature – studies have shown that heart disease is a risk factor for patient mortality [4]. comparing the predictions for \(\text {hcpa}_{cti}\) using xgb (figs. 4g, 4h, and 4i) and lr (figs. 4j, 4k, and 4l) we notice that xgb tends to present a greater overlap between fn and fp. this may be due to the influence of the t-sne visualization method, which may favor the visualization for the lr algorithm. figures 4i and 4l, generated based on the shap values provided by the xgb and lr classifiers, show regions of high concentrations of patients with high values of d-dimers. both younger and older patients are found in those regions. the pattern is more evident in the visualization obtained with lr, in which tp and fp classifications prevail. high d-dimer values are associated with poor prognosis in patients with covid-19 infection [15], and in both cases (xgb and lr), models tend to assign these patients to the deceased class. 4.4 single vs. multiple variable classifiers (rq3) as seen in the analysis of sect. 4.3, as well as recurrently pointed out by works in the related literature, age was frequently flagged as the attribute with the greatest contribution during mortality prediction tasks. this research question aims to compare the performance between classifiers with multiple independent variables and their respective versions with only one independent variable. for this, we compare the performance of the classifiers presented in sect. 4.2. figure 5 shows the differences between the performances obtained by the classifiers trained using only age as independent variable. fig. 5. multivariable and single-variable classifiers (age) for mortality prediction full size image for most algorithms, classification models that used only age as independent variable did not show degradation in sensitivity, and in some cases, higher levels of sensitivity were found. although this positive effect on sensitivity levels was observed, the reduction in ppv and specificity levels was also recurrent. as a consequence, in most cases, no gains were observed for the metrics f1+, ma-f1, kappa, auroc and auprc. the lr algorithm, which showed the best results in the experiments in sect. 4.2, tended not to show substantial decrease in sensitivity levels. for the \(\text {hmv}_{all}\) dataset, considerable increases in sensitivity levels occur for some algorithms. in contrast, there were considerable reductions in the levels of ppv and specificity, mainly in the \(\text {hmv}_{all}\) dataset and in the \(\text {hcpa}_{cti}\) dataset. algorithm j48, which presented the lowest sensitivity scores in the experiment in sect. 4.2, was recurrently the one that obtained the highest sensitivity gains among the classifiers that used only age as an independent variable. when considering the \(\text {hmv}_{cti}\) dataset, in addition to the j48 algorithm, the rf and mlp algorithms stood out for their increased sensitivity. however, despite the increase in sensitivity, due to the losses in the levels of ppv and specificity, no gains were observed for the metrics f1+, ma-f1 and kappa. finally, j48 and mlp were the only ones that obtained auroc gains. in \(\text {hcpa}_{cti}\) (fig. 5b), only the j48 algorithm obtained an increase in the other metrics, in addition to sensitivity. in \(\text {hmv}_{cti}\) (fig. 5c), j48 and mlp achieved the highest sensitivity gains. in this scenario, both algorithms showed a smaller reduction in terms of ppv and specificity, resulting in an increase in the levels of other metrics. essentially, the classifiers that use only age as independent variable reach similar or higher sensitivity scores compared to their multivariable versions. however, these scores were achieved at the cost of lower spe and ppv levels, showing that other variables are useful in improving these metrics. the full version of table 2 with ci (rq1) and significance tests performed on rq3 can be accessed at https://github.com/danimtk/covid-19-paper. 5 conclusion this work evaluated several ml algorithms for predicting covid-19 mortality. the classifiers showed a promising ability to predict mortality based on the information available at the time of admission, reaching a sensitivity of up to 84% and an auroc of 92% (95% ci, [0.89–0.95]). overall, besides age, oxygen saturation, heart disease, hemogram information also tended to have high contributions to the predictions. we applied a visualization technique based on shap values, which helps to corroborate the high importance attributed to the age feature by the classifiers in their predictions. the visualizations also showed that some groups of patients with low erythrocytes and high d-dimer (risk factors for mortality) tend to be recognized by the classifiers with a higher risk of dying. we believe that the visualization technique employed here can also be used in other prediction tasks. one limitation of our work is that shap is relatively time-consuming when applied to datasets with a large number of features [19]. this limitation can be mitigated using optimized shap algorithms (like treeshap or [19]). although the visualizations obtained are intuitive and seem to represent already known relationships (such as the high risk for older patients, as well as those with comorbidities), it should be noted that approaches that use shapley values, as well as approximations of these values (such as shap), may generate misconceptions regarding the actual contribution of features [8]. thus, we emphasize the importance for specialists to use other evaluation techniques concomitantly to assist in the interpretation of predictive models, avoiding misinterpretations. in future work, we plan to experiment with other dimensionality reduction methods and classifiers. we also plan to investigate whether there is a set of attributes (and how many) would be needed to maintain sensitivity levels without considerable ppv losses. notes 1.ethics committee approval: hcpa 32314720.8.0000.5327, hmv 32314720.8.3001.5330. 2.https://fracpete.github.io/python-weka-wrapper3. 3.https://github.com/dmlc/xgboost. 4.https://github.com/slundberg/shap. references aas, k., jullum, m., løland, a.: explaining individual predictions when features are dependent: accurate approximations to shapley values. artif. intell. 298, 103502 (2021) article  mathscinet  math  google scholar  alballa, n., al-turaiki, i.: machine learning approaches in covid-19 diagnosis, mortality, and severity risk prediction: a review. inf. med. unlocked 24, 100564 (2021) article  google scholar  araújo, d.c., veloso, a.a., borges, k.b.g., das graças carvalho, m.: prognosing the risk of covid-19 death through a machine learning-based routine blood panel: a retrospective study in brazil. ijmedi 165, 104835 (2022) google scholar  broberg, c.s., kovacs, a.h., sadeghi, s., et al.: covid-19 in adults with congenital heart disease. jacc 77(13), 1644–1655 (2021) article  google scholar  covino, m., sandroni, c., santoro, m., et al.: predicting intensive care unit admission and death for covid-19 patients in the emergency department using early warning scores. resuscitation 156, 84–91 (2020) article  google scholar  futoma, j., simons, m., panch, t., et al.: the myth of generalisability in clinical research and machine learning in health care. lancet digit. health 2(9), 489–492 (2020) article  google scholar  hall, m.a.: correlation-based feature selection for machine learning. ph.d. thesis, the university of waikato (1999) google scholar  huang, x., marques-silva, j.: the inadequacy of shapley values for explainability. arxiv preprint arxiv:2302.08160 (2023) knight, s.r., ho, a., pius, r., et al.: risk stratification of patients admitted to hospital with covid-19 using the isaric who clinical characterisation protocol: development and validation of the 4c mortality score. bmj 370, m3339 (2020) article  google scholar  lundberg, s.m., erion, g.g., lee, s.i.: consistent individualized feature attribution for tree ensembles. arxiv preprint arxiv:1802.03888 (2018) lundberg, s.m., lee, s.i.: a unified approach to interpreting model predictions. in: advances in neural information processing systems, vol. 30 (2017) google scholar  van der maaten, l., hinton, g.: visualizing data using t-sne. jmlr 9(11), 2579–2605 (2008) math  google scholar  miranda de paiva, b.b., delfino-pereira, p., de andrade, c.m.v., et al.: effectiveness, explainability and reliability of machine meta-learning methods for predicting mortality in patients with covid-19: results of the brazilian covid-19 registry. medrxiv (2021) google scholar  qin, z., zhang, c., wang, t., zhang, s.: cost sensitive classification in data mining. in: advanced data mining and applications, pp. 1–11 (2010) google scholar  rostami, m., mansouritorghabeh, h.: d-dimer level in covid-19 infection: a systematic review. exp. rev. hematol. 13(11), 1265–1275 (2020) article  google scholar  subudhi, s., verma, a., patel, a.b.: prognostic machine learning models for covid-19 to facilitate decision making. ijcp 74(12), e13685 (2020) google scholar  wynants, l., van calster, b., collins, g.s., et al.: prediction models for diagnosis and prognosis of covid-19: systematic review and critical appraisal. bmj 369, m1328 (2020) article  google scholar  yadaw, a.s., li, y., bose, s., et al.: clinical features of covid-19 mortality: development and validation of a clinical prediction model. lancet digt. health 2(10), e516–e525 (2020) article  google scholar  yang, j.: fast treeshap: accelerating shap value computation for trees. arxiv preprint arxiv:2109.09847 (2021) zhao, z., chen, a., hou, w., et al.: prediction model and risk scores of icu admission and mortality in covid-19. plos one 15(7), e0236618 (2020) article  google scholar  download references acknowledgment this work has been financed in part by capes finance code 001 and cnpq/brazil. author information authors and affiliations institute of informatics, ufrgs, porto alegre, brazil daniel matheus kuhn, mariana recamonde-mendoza, joão luiz dihl comba & viviane pereira moreira hospital de clínicas de porto alegre, porto alegre, brazil melina silva de loreto authors daniel matheus kuhnview author publications search author on:pubmed google scholar melina silva de loretoview author publications search author on:pubmed google scholar mariana recamonde-mendozaview author publications search author on:pubmed google scholar joão luiz dihl combaview author publications search author on:pubmed google scholar viviane pereira moreiraview author publications search author on:pubmed google scholar corresponding author correspondence to daniel matheus kuhn . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper kuhn, d.m., de loreto, m.s., recamonde-mendoza, m., comba, j.l.d., moreira, v.p. (2023). explainability of covid-19 classification models using dimensionality reduction of shap values. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_27 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords mortality prediction classification models explainable ai dimensionality reduction shap values publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature on the equivalence between logic programs and bipolar argumentation frameworks | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper on the equivalence between logic programs and bipolar argumentation frameworks conference paper first online: 30 january 2025 pp 238–253 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) on the equivalence between logic programs and bipolar argumentation frameworks download book pdf download book epub renan cordeiro  orcid: orcid.org/0000-0001-9502-34139 & joão alcântara  orcid: orcid.org/0000-0002-4297-29709  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 366 accesses abstract in this paper, we extend the connection between argumentation formalisms and logic programs, by showing the equivalence between bipolar argumentation \(\beta \)-semantics and various 3-valued logic programming semantics. in particular, we show that \(\beta \)-semi-stable semantics corresponds to l-stable semantics, which has been previously shown not to be expressible by traditional attack-only argumentation frameworks. besides enhancing our understanding of how logic programs relate to argumentation with support, we provide direct translations between them and their semantics, allowing their interchangeable use. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others an abstract argumentation and logic programming comparison based on 5-valued labellings chapter © 2021 capturing bipolar argumentation in non-flat assumption-based argumentation chapter © 2017 generalizing complete semantics to bipolar argumentation frameworks chapter © 2021 1 introduction formal argumentation and logic programming are closely related paradigms [7] for representing possibly contradictory, uncertain or incomplete information. in formal argumentation, an argument represents a claim whose acceptability is determined by some criteria (referred to as semantics) that depend on how arguments interact with each other. dung introduced abstract argumentation frameworks (\( aaf \)s) in his seminal work [10], defining an \( aaf \) by a set of arguments and an attack relation between them. in \( aaf \)s, an argument is treated as an abstract entity without any internal structure, which means that semantics can only take into account the interactions between arguments. in normal logic programs (\(\textit{nlp}\)s), a claim is represented by an atom whose evaluation consists in determining whether it can be inferred from logical rules. despite expressing information in distinctive perspectives, formal argumentation and logic programming are closely connected. their connection is a very relevant line of research, as it allows one to view logic programs as argumentation frameworks and vice versa, allowing the most suitable formalism to be employed given a specific application. besides, algorithms, techniques and semantics developed for a formalism can be adapted to the other. not surprisingly, equivalences were extensively investigated. there is correspondence between complete and partial stable semantics [16], grounded and well-founded semantics [10], preferred and regular semantics [7], stable (argumentation) and stable (logic programming) semantics [10]. however, there exists a logic programming semantics (l-stable) that cannot be captured by any abstract argumentation semantics [7]. in this paper, we continue this line of research by considering instead bipolar argumentation frameworks (\(\textit{baf}\)s [5]), in which additionally there is a support relation between arguments: a positive interaction independent from the attack relation. although indirectly and not for all semantics, bipolar argumentation can be linked to logic programming by the current literature. in [9], \(\textit{baf}\)s are represented by assumption-based argumentation (\(\textit{aba}\)) for admissible, preferred and stable semantics and four interpretations of support. as \(\textit{aba}\) is linked to logic programming [8], the transitive nature of these correspondences induces an indirect connection between \(\textit{baf}\)s and \(\textit{nlp}\)s. however, the semi-stable semantics of \(\textit{aba}\) does not correspond in general to the l-stable semantics of \(\textit{nlp}\)s [8]. a common aspect of these interpretations of support is their asymmetric treatment of attacks and supports. the former are favoured, as an argument is always rejected when it is both attacked and supported by accepted arguments. we instead resort to the interpretation of support from the \(\beta \)-semantics [1], in which an argument is always accepted if it is supported by an accepted argument. several works also relate more expressive argumentation formalisms to logic programming. for instance, abstract dialectical frameworks [2, 6, 14], \(\textit{aba}\) [8], claim-augmented argumentation [13, 15], frameworks with collective attacks [3, 15] and many others [4] are connected to logic programming. the work closest to ours is [13], where the authors relate bipolar claim-augmented argumentation frameworks (\(\textit{bcaf}\)s) to \(\textit{nlp}\)s. they could establish a one-to-one correspondence between \(\textit{bcaf}\)s and \(\textit{nlp}\)s in both directions, whilst preserving the main acceptability semantics, including the l-stable semantics. the main distinction between \(\textit{bcaf}\)s and \(\textit{baf}\)s (and hence between their work and ours) is that \(\textit{bcaf}\)s have additional structure, as each argument is associated to an explicit claim, whereas arguments in a \(\textit{baf}\) are abstract: the \(\beta \)-semantics depend exclusively on the attack and support relation between arguments. in contrast, the semi-stable semantics of \(\textit{bcaf}\)s [13] take into account the conclusion of arguments, information not inferred solely from the attack and support relations. this paper is divided as follows: in sect. 2, we briefly recall the foundations of bipolar argumentation frameworks (\(\textit{baf}\)s) and their \(\beta \)-semantics, and of normal logic programs (\(\textit{nlp}\)s) and their 3-valued semantics; in the sequence, we provide a translation from \(\textit{nlp}\)s to \(\textit{baf}\)s and between their semantics; the inverse direction from \(\textit{baf}\)s to \(\textit{nlp}\)s is discussed in sect. 4; we collect our conclusions in sect. 5. 2 preliminaries 2.1 bipolar argumentation frameworks \((\textit{baf}s)\) in [5], dung’s abstract argumentation frameworks (\( aaf \)s) were extended to incorporate an explicit notion of support between arguments, independent from the attack relation. the resulting framework, called \(\textit{baf}\), is defined next: definition 1 (baf [5]). a bipolar argumentation framework (\(\textit{baf}\)) is a tuple \((\mathcal {a}, att ,\) \( sup )\), in which \(\mathcal {a}\) is a set of arguments, \( att \subseteq \mathcal {a}\times \mathcal {a}\) is the attack relation and \( sup \subseteq \mathcal {a}\times \mathcal {a}\) is the support relation. for an argument \(a \in \mathcal {a}\), we define \( att (a) = \left\{ b \in \mathcal {a}\mid (b, a) \in att \right\} \) as the set of attackers of a and \( sup (a) = \left\{ b \in \mathcal {a}\mid (b, a) \in sup \right\} \) as the set of direct supporters of a. in \( aaf \)s, the interaction between arguments is specified only by the attack relation. in \(\textit{baf}\)s, the novelty is that arguments can also support other arguments. this means that \(\textit{baf}\)s \((\mathcal {a}, att , sup )\) with \( sup = \emptyset \) amount to (standard dung) \( aaf \)s. we are interested not only in the direct supporters of an argument, but also in the indirect ones. they are indistinctly called the supporters of an argument: definition 2 (supporters [1]). let \(\mathcal {b} = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\) and \(a \in \mathcal {a}\) an argument. we define the supporters of a recursively as follows: a is a supporter of a in \(\mathcal {b}\); if \(a'\) is a supporter of a in \(\mathcal {b}\) and \((b,a') \in sup \), then b is a supporter of a in \(\mathcal {b}\). we denote the set of all supporters of a in \(\mathcal {b}\) by \(\mathfrak {sup}(a)\). an acceptance criteria, referred to as semantics, for arguments in a \(\textit{baf}\) is how we determine the accepted or rejected arguments in a framework. semantics can be equivalently characterised in terms of extensions (sets of accepted arguments) or labellings (mapping arguments to their acceptability) [1]. we will focus on labelling-based semantics as it allows a more direct comparison to logic programming semantics. definition 3 (labelling). let \(\mathcal {b} = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\). a labelling (of \(\mathcal {b}\)) is a total function \(\mathcal {l} : \mathcal {a}\rightarrow \left\{ \texttt{in}, \texttt{out}, \texttt{undec}\right\} \). when \(\mathcal {l}\) is a labelling, we write \(\texttt{in}(\mathcal {l})\) to denote the set \(\left\{ a \mid \mathcal {l}(a) = \texttt{in}\right\} \), \(\texttt{out}(\mathcal {l})\) for \(\left\{ a \mid \mathcal {l}(a) = \texttt{out}\right\} \) and \(\texttt{undec}(\mathcal {l})\) for \(\left\{ a \mid \mathcal {l}(a) = \texttt{undec}\right\} \). as a labelling defines a partition among arguments, when convenient we write \(\mathcal {l}\) as the triple \((\texttt{in}(\mathcal {l}), \texttt{out}(\mathcal {l}), \texttt{undec}(\mathcal {l}))\). intuitively, the label \(\texttt{in}\) indicates acceptance, the label \(\texttt{out}\) indicates rejection and the label \(\texttt{undec}\) indicates that the argument is undecided, i.e., neither accepted nor rejected. we can now describe the \(\textit{baf}\) semantics studied in this paper: definition 4 (labelling-based semantics for \(\textit{baf}\)s [1]). let \(\mathcal {b} = (\mathcal {a}, att ,\) \( sup )\) be a \(\textit{baf}\). a labelling \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal {b}\) if for any \(a \in \mathcal {a}\), \(\mathcal {l}(a) = \texttt{in}\) if and only if there exists \(a' \in \mathfrak {sup}(a)\) such that for every \(b \in att (a')\), it holds \(\mathcal {l}(b) = \texttt{out}\). \(\mathcal {l}(a) = \texttt{out}\) if and only if for every \(a' \in \mathfrak {sup}(a)\), there exists \(b \in att (a')\) such that \(\mathcal {l}(b) = \texttt{in}\). additionally, we say a \(\beta \)-complete labelling \(\mathcal {l}\) of \(\mathcal {b}\) is \(\beta \)-grounded (resp. \(\beta \)-preferred) if \(\texttt{in}(\mathcal {l})\) is minimal (resp. maximal) w. r. t. \(\subseteq \) among the \(\beta \)-complete labellings of \(\mathcal {b}\); \(\beta \)-stable if \(\texttt{undec}(\mathcal {l}) = \emptyset \); \(\beta \)-semi-stable if \(\texttt{undec}(\mathcal {l})\) is minimal (w. r. t. \(\subseteq \)) among the \(\beta \)-complete labellings of \(\mathcal {b}\). fig. 1. \(\textit{baf}\) full size image labelling-based semantics of \(\textit{baf}\)s preserve remarking properties of the corresponding labelling-based semantics of \( aaf \)s [1]: for every \(\textit{baf}\) there is always a uniquely defined \(\beta \)-complete labelling; every \(\beta \)-stable labelling is a \(\beta \)-preferred labelling; and if there exists \(\beta \)-stable labellings, they coincide with \(\beta \)-semi-stable and \(\beta \)-preferred labellings. example 1 consider the \(\textit{baf}\) depicted by the directed graph in fig. 1, where nodes represent arguments, solid arrows represent attacks, and dashed arrows represent the support relation. we obtain labellings as follows: \(\beta \)-complete labellings are \(\mathcal {l}_1 = (\emptyset , \emptyset , \mathcal {a})\), \(\mathcal {l}_2 = (\left\{ a_1, a_4\right\} , \{a_2\}, \{a_3, a_5\})\) and \(\mathcal {l}_3 = (\{a_2, a_3, \) \(a_4\}, \left\{ a_1\right\} ,\{a_5\})\); \(\beta \)-grounded labelling is \(\mathcal {l}_1\); \(\beta \)-preferred labellings are \(\mathcal {l}_2\) and \(\mathcal {l}_3\); there are no \(\beta \)-stable labellings; and \(\beta \)-semi-stable labelling is \(\mathcal {l}_3\). 2.2 logic programs and semantics now, we take a look at propositional normal logic programs. to delve into their definition and semantics, we will follow the presentation outlined in [7], which draws from the foundation laid out in [12]. definition 5 a rule r is an expression $$\begin{aligned} r: c \leftarrow a_1, \ldots , a_m, \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n \end{aligned}$$ (1) where (\(m, n \ge 0\)); c, each \(a_i\) (\(1 \le i \le m\)) and each \(b_j\) (\(1 \le j \le n\)) are atoms, and \(\texttt{not}\) represents negation as failure. a literal is either an atom a (positive literal) or a negated atom \(\mathtt {not\ }a\) (negative literal). given a rule r as above, c is called the head of r, which we denote as \( head (r)\), and \( body (r) = \left\{ a_1, \ldots , a_m, \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n\right\} \) is called the body of r. further, \( body (r)\) is divided in two sets \( body ^+(r) = \left\{ a_1, \ldots , a_m\right\} \) and \( body ^-(r) = \left\{ \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n\right\} \). a fact is a rule where \(m = n = 0\). a normal logic program (\(\textit{nlp}\)) or simply a logic program p is a finite set of rules. if every \(r \in p\) has \( body ^-(r) = \emptyset \), p is a positive program. the herbrand base of p is the set \( hb _{p}\) of all atoms appearing in p. a wide range of \(\textit{nlp}\) semantics are based on the 3-valued interpretations of programs [12]: definition 6 (3-valued herbrand interpretation [12]). a 3-valued herbrand interpretation \(\mathcal {i}\) (or simply interpretation) of p is a total function \(\mathcal {i}: hb _p \rightarrow \left\{ \textbf{t}, \textbf{f}, \textbf{u}\right\} \). given an interpretation \(\mathcal {i}\) of p, we write \(\textbf{t}(\mathcal {i})\) to denote the set \(\{ a \in hb _p \mid \) \(\mathcal {i}(a) = \textbf{t}\}\), \(\textbf{f}(\mathcal {i})\) for \(\left\{ a \in hb _p \mid \mathcal {i}(a) = \textbf{f}\right\} \) and \(\textbf{u}(\mathcal {i})\) for \(\{ a \in hb _p \mid \mathcal {i}(a) = \textbf{u}\}\). as an interpretation defines a partition among atoms in \( hb _p\), when convenient we write \(\mathcal {i}\) as the triple \((\textbf{t}(\mathcal {i}), \textbf{f}(\mathcal {i}), \textbf{u}(\mathcal {i}))\). intuitively, the value \(\textbf{t}\) indicates the atom is true in \(\mathcal {i}\), the value \(\textbf{f}\) indicates the atom is false in \(\mathcal {i}\), and the value \(\textbf{u}\) indicates the atom is undefined in \(\mathcal {i}\), i.e., neither true nor false. now we will consider the main semantics for \(\textit{nlp}\)s. let \(\mathcal {i}\) be a 3-valued herbrand interpretation of p; the reduct of p with respect to \(\mathcal {i}\) (written as \(p/\mathcal {i}\)) is the \(\textit{nlp}\) constructed using the following steps: 1. remove any \(a \leftarrow a_1, \ldots , a_m,\) \(\mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_n \in p\) such that \(\mathcal {i}(b_j) = \textbf{t}\) for some j (\(1 \le j \le n\)); 2. afterwards, remove any occurrence of \(\mathtt {not\ }b_j\) from p such that \(\mathcal {i}(b_j) = \textbf{f}\). 3. then, replace any occurrence of \(\mathtt {not\ }b_j\) left by a special atom \(\textbf{u}\) (\(\textbf{u} \not \in hb _p\)). in the above procedure, \(\textbf{u}\) is assumed to be an atom not in \( hb _{p}\) which is undefined in all interpretations of p (a constant). note that \(p/\mathcal {i}\) is a positive program since all negative literals have been removed. as a consequence, \(p/\mathcal {i}\) has a unique least 3-valued model [12] denoted as \(\varpsi _p(\mathcal {i})\)footnote 1 with minimal \(\textbf{t}(\varpsi _p(\mathcal {i}))\) and maximal \(\textbf{f}(\varpsi _p(\mathcal {i}))\) (w.r.t. set inclusion) such that, for every \(a \in hb _{p}\), (i) \(a \in \textbf{t}(\varpsi _p(\mathcal {i}))\) if there is a rule \(r^{\prime } \in p/\mathcal {i}\) with \( head (r^{\prime }) = a\) and \( body ^{+}(r^{\prime }) \subseteq \textbf{t}(\varpsi _p(\mathcal {i}))\); and (ii) \(a \in \textbf{f}(\varpsi _p(\mathcal {i}))\) if for every rule \(r^{\prime } \in p/\mathcal {i}\) with \( head (r^{\prime }) = a\), it holds \( body ^{+}(r^{\prime }) \cap \textbf{f}(\varpsi _p(\mathcal {i})) \ne \emptyset \). we can now describe the \(\textit{nlp}\) semantics studied in this paper: definition 7 let p be an \(\textit{nlp}\) and \(\mathcal {i}\) be an interpretation: \(\mathcal {i}\) is a partial stable model of p iff \(\varpsi _p(\mathcal {i}) = \mathcal {i}\) [12]; \(\mathcal {i}\) is a well-founded (resp. regular) model of p iff \(\varpsi _p(\mathcal {i}) = \mathcal {i}\) and \(\textbf{t}(\mathcal {i})\) is minimal (resp. maximal) w.r.t. set inclusion among all partial stable models of p [11, 12]; \(\mathcal {i}\) is a (2-valued) stable model of p iff \(\varpsi _p(\mathcal {i}) = \mathcal {i}\) and \(\textbf{u}(\mathcal {i}) = \emptyset \) [12]; \(\mathcal {i}\) is an l-stable model of p iff \(\varpsi _p(\mathcal {i}) = \mathcal {i}\) and \(\textbf{u}(\mathcal {i})\) is minimal w.r.t. set inclusion among all partial stable models of p [11]. although some of these definitions are not standard in logic programming literature, their equivalence is proved in [7]. this format helps us to relate \(\textit{nlp}\) and \(\textit{baf}\) semantics due to the structural similarities between definition 7 and definitions 3 and 4. we illustrate these semantics in the following example: example 2 consider the following \(\textit{nlp}\) p: $$ \begin{array}{llcll} r_{0}: & d \leftarrow \texttt{not}\; d & & r_{1}: & c \leftarrow \texttt{not}\; c, \texttt{not}\; d \\ r_{2}: & a \leftarrow \texttt{not}\; b & & r_{3}: & b \leftarrow \texttt{not}\; a \\ r_{4}: & c \leftarrow \texttt{not}\; c, \texttt{not}\; a & & r_{5}: & e \leftarrow \texttt{not}\; e, \texttt{not}\; b \end{array} $$ this program has partial stable models \(\mathcal m_1 = (\emptyset , \emptyset , hb _p)\), \(\mathcal m_2 = (\left\{ a \right\} , \{ b \}, \{ c,\) \(d, e \})\), \(\mathcal m_3 = (\left\{ b \right\} , \left\{ a, e \right\} , \left\{ c, d \right\} )\); well-founded model is \(\mathcal m_1\); regular models are \(\mathcal m_2\) and \(\mathcal m_3\); there are no stable models; and l-stable model is \(\mathcal m_3\). 3 from \(\textit{nlp}\)s to \(\textit{baf}\)s in this section, we devise a translation from \(\textit{nlp}\)s to \(\textit{baf}\)s. we wish to show that it preserves the main semantics considered in this paper, including the l-stable semantics. we start by creating arguments for derivations of rules of p, exactly as done in [7]. definition 8 [7] let p be an \(\textit{nlp}\) and \(\mathcal {a}_p\) be the set of arguments recursively defined below. if \(c \leftarrow \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_m\) is a rule in p, it is also an argument (say a) with \(\texttt{conc}(a) = c\), \(\texttt{rules}(a) = \{ c \leftarrow \mathtt {not\ }b_1, \ldots , \mathtt {not\ }b_m \}\), \(\texttt{vul}(a) = \{ b_1, \ldots , b_m \}\), and \(\texttt{sub}(a) = \{ a \}\). if \(c \leftarrow a_1,\ldots ,a_n, \mathtt {not\ }b_1,\ldots , \mathtt {not\ }b_m\) is a rule (say r) in p and for each \(a_i\) (\(1 \le i \le n\)) there exists an argument \(a_i\) with \(\texttt{conc}(a_i) = a_i\) and \(r \notin \texttt{rules}(a_i)\), then r derives the argument \(c \leftarrow (a_1),\ldots ,(a_n),\) \(\mathtt {not\ }b_1,\ldots ,\) \(\mathtt {not\ }b_m\) (say a) with \(\texttt{conc}(a) = c\), \(\texttt{rules}(a) = \bigcup _{i=1}^n\texttt{rules}(a_i) \cup \{ r \}\), \(\texttt{vul}(a) = \bigcup _{i=1}^n\texttt{vul}(a_i) \cup \{b_1, \ldots , b_m\}\), and \(\texttt{sub}(a) = \{ a \} \cup \bigcup _{i=1}^n\texttt{sub}(a_i)\). for any argument a, we say \(\texttt{conc}(a)\) is its conclusion; \(\texttt{rules}(a)\), its set of rules; \(\texttt{vul}(a)\), its set of vulnerabilities; and \(\texttt{sub}(a)\), its set of subarguments. we can intuitively see an argument as a tree-like structure representing a possible derivation of an atom from the rules of a program. example 3 consider the \(\textit{nlp}\) p below with rules \(\left\{ r_1, \ldots , r_8\right\} \): $$ \begin{array}{lclclcl} r_1: a & & r_2: b \leftarrow a & & r_3: c \leftarrow \mathtt {not\ }c & & r_7: c \leftarrow f, \mathtt {not\ }g \\ r_4: d \leftarrow b, \mathtt {not\ }a & & r_5: d \leftarrow \mathtt {not\ }c & & r_6: e \leftarrow b, c, \mathtt {not\ }e & & r_8: f \leftarrow c, g \\ \end{array} $$ according to definition 8, we can construct the following statements from p: $$ \begin{array}{llcllcll} a_1: & a & & a_2: & b \leftarrow (a_1) & & a_3: & c \leftarrow \texttt{not}\; c\\ a_4: & d \leftarrow (a_2), \texttt{not}\; a & & a_5: & d \leftarrow \texttt{not}\; c & & a_6: & e \leftarrow (a_2), (a_3),\texttt{not}\; e \end{array} $$ next, we give the conclusions and vulnerabilities of each argument: the vulnerabilities of an argument a are associated with the negative literals found in the derivation of a. if \(\texttt{not}\; b\) is one of them, then b is a vulnerability of a. this means that if b is derived, then \(\texttt{conc}(a)\) cannot be obtained via the derivation represented by a. however, it can still be obtained via other derivations/arguments. for instance, in the program p of example 3, the derivation of a suffices to prevent the derivation of d via argument \(a_4\) (for that reason, \(a \in \texttt{vul}(a_4)\)), but we still can derive d via \(a_5\). notice also that there are no arguments with conclusions f and g. now we are entitled to determine both the attack and support relations: definition 9 let p be an \(\textit{nlp}\) and let a and b be arguments in the sense of definition 8. we define the attack relation \( att _p\) and the support relation \( sup _p\) as follows: \((a, b) \in att _p\) iff \(\texttt{conc}(a) \in \texttt{vul}(b)\); \((a, b) \in sup _p\) iff \(a \ne b\) and \(\texttt{conc}(a) = \texttt{conc}(b)\). our definition of the set \( att _p\) of attacks obtained from an \(\textit{nlp}\) p is exactly as it has been characterised in [7]. the novelty here is the definition of \( sup _p\), which in our proposal is determined by arguments with the same conclusion (the condition \(a \ne b\) is just to avoid incorporating the redundant case of an argument supporting itself). the failure in guaranteeing the correspondence between \(\beta \)-semi-stable and l-stable semantics, as discussed in [7], lies in the presence of arguments with distinct acceptabilities representing the derivations of the same atom. as definition 9 ensures that such arguments support each other, they will have the same acceptability according to the \(\beta \)-semantics [1]. for the arguments of example 3, \(a_1\) attacks \(a_4\); \(a_3\) attacks itself, \(a_5\) and \(a_6\); \(a_6\) attacks itself. we also have \(a_4\) and \(a_5\) supporting each other. using the thus-defined concepts of arguments, attacks and supports, one can define the bipolar argumentation framework corresponding to a particular \(\textit{nlp}\): definition 10 let p be an \(\textit{nlp}\). we define its corresponding \(\textit{baf}\) as \(\mathcal {b}_p = (\mathcal {a}_p, att _p, sup _p)\), where \(\mathcal {a}_p\) is the set of arguments in the sense of definition 8 and \( att _p\) is the attack relation in the sense of definition 9. although the internal structure of arguments constructed from definition 8 is used to construct \( att _p\) and \( sup _p\), any argument of \(\mathcal b_p\) is abstract regarding \(\textit{baf}\) semantics, which depend exclusively on \( att _p\) and \( sup _p\). as an example, the corresponding \(\textit{baf}\) of the \(\textit{nlp}\) of example 3 is depicted in fig. 2. fig. 2. a \(\textit{baf}\) \(\mathcal {b}_p = (\mathcal {a}_p, att _p, sup _p)\) full size image once the \(\textit{baf}\) has been constructed, we show the equivalence between the semantics for p and their counterpart for the corresponding \(\textit{baf}\) \(\mathcal {b}_p\). we prove the equivalence results by identifying connections between fundamental notions used in the definition of the semantics for \(\textit{nlp}\)s and \(\textit{baf}\)s. for this purpose, we introduce two functions: \(\mathcal {l}2\mathcal {i}_p\) associates an interpretation to each labelling, while \(\mathcal {i}2\mathcal {l}_p\) associates a labelling to each interpretation. we then investigate the conditions under which they are each other’s inverses and employ these results to prove the equivalence between the semantics. these functions essentially permit us to treat interpretations and labellings interchangeably. definition 11 (\(\mathcal {l}2\mathcal {i}_p\) and \(\mathcal {i}2\mathcal {l}_p\) functions). let p be an \(\textit{nlp}\), \(\mathcal {b}_p = (\mathcal {a}_p,\) \( att _p, sup _p)\) be its corresponding \(\textit{baf}\), \(\mathcal {i} nt \) be the set of all the 3-valued interpretations of p and \(\mathcal {l} ab \) be the set of all labellings of \(\mathcal {b}_p\). we introduce a function \(\mathcal {l}2\mathcal {i}_p: \mathcal {l} ab \rightarrow \mathcal {i} nt \) where \(\mathcal {l}2\mathcal {i}_p(\mathcal {l}) = (t, f, hb _p (t \cup f))\), in which \(t = \{c \in hb _p \mid \exists a \in \texttt{in}(\mathcal {l})\text { with }\texttt{conc}(a) = c\}\); and \(f = \{c \in hb _p \mid \mathcal {l}(a) = \texttt{out}\textit{ for every } a \in \mathcal {a}_p \textit{ such that } \texttt{conc}(a) = c\}\). we introduce a function \(\mathcal {i}2\mathcal {l}_p: \mathcal {i} nt \rightarrow \mathcal {l} ab \) such that for any \(\mathcal i \in \mathcal {i} nt \) and any \(a \in \mathcal {a}_p\), \(\mathcal {i}2\mathcal {l}_p(\mathcal i)(a) = \texttt{in}\) if \(\mathcal i(\texttt{conc}(a)) = \textbf{t}\); \(\mathcal {i}2\mathcal {l}_p(\mathcal i)(a) = \texttt{out}\) if \(\mathcal i(\texttt{conc}(a)) = \textbf{f}\); \(\mathcal {i}2\mathcal {l}_p(\mathcal i)(a) = \texttt{undec}\) if \(\mathcal i(\texttt{conc}(a)) = \textbf{u}\). in general, \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))\) is not equal to \(\mathcal {l}\). for instance, considering the labelling \(\mathcal {l} = (\emptyset , \left\{ a_4\right\} , \left\{ a_1, a_2, a_3, a_5, a_6\right\} )\) and the \(\textit{baf}\) \(\mathcal {b}_p\) of fig. 2, we have that \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l})) = (\emptyset , \emptyset , \left\{ a_1, a_2, a_3, a_4, a_5, a_6\right\} )\). such an inequality will always occur when two arguments with the same conclusion receive distinct labels. in our example, \(a_4\) and \(a_5\) have the same conclusion d; \(\mathcal {l}(a_4) = \texttt{out}\), but \(\mathcal {l}(a_5) = \texttt{undec}\). it holds \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})(d) = \textbf{u}\), and consequently, \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))(a_4) = \mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))(a_5) = \texttt{undec}\), i.e., \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))\) is not equal to \(\mathcal {l}\). however, if \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\), arguments with the same conclusion will always have the same label: theorem 1 let \(\mathcal {b}_p = (\mathcal {a}_p, att _p, sup _p)\) be the corresponding \(\textit{baf}\) of p. if \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\) and \(\texttt{conc}(a) = \texttt{conc}(b)\), then \(\mathcal {l}(a) = \mathcal {l}(b)\). proof let \(a, b \in \mathcal {a}_p\) with \(\texttt{conc}(a) = \texttt{conc}(b)\) and \(\mathcal {l}\) be a \(\beta \)-complete labelling of \(\mathcal {b}_p\). the result holds trivially for \(a = b\). assume \(a \ne b\). by definition 9, \((a, b) \in sup _p\) and \((b, a) \in sup _p\). as \(\mathcal {l}\) is \(\beta \)-complete, \(\mathcal {l}(a) = \mathcal {l}(b)\). thus, if \(\mathcal {l}\) is a \(\beta \)-complete labelling, we have that \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l})) = \mathcal {l}\): theorem 2 let \(\mathcal {b}_p = (\mathcal {a}_p, att _p, sup _p)\) be the corresponding \(\textit{baf}\) of p. for any \(\beta \)-complete labelling \(\mathcal {l}\) of \(\mathcal {b}_p\), it holds \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l})) = \mathcal {l}\). proof let \(a \in \mathcal {a}_p\). there are 3 cases: \(\mathcal {l}(a) = \texttt{in}\) \(\rightarrow \) \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})(\texttt{conc}(a)) = \textbf{t}\) \(\rightarrow \) \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))(a) = \texttt{in}\); \(\mathcal {l}(a) = \texttt{out}\) \(\rightarrow \) (theorem 1) \(\mathcal {l}(a') = \texttt{out}\) for every \(a' \in \mathcal {a}_p\) such that \(\texttt{conc}(a') = \texttt{conc}(a)\) \(\rightarrow \) \(\mathcal {l}2\mathcal {i}_p(\mathcal l)(\texttt{conc}(a)) = \textbf{f}\) \(\rightarrow \) \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))(a) = \texttt{out}\); \(\mathcal {l}(a) = \texttt{undec}\) \(\rightarrow \) (theorem 1) \(\mathcal {l}(a') = \texttt{undec}\) for every \(a' \in \mathcal {a}_p\) such that \(\texttt{conc}(a') = \texttt{conc}(a)\) \(\rightarrow \) \(\mathcal {l}2\mathcal {i}_p(\mathcal l)(\texttt{conc}(a)) = \textbf{u}\) \(\rightarrow \) \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))(a) = \texttt{undec}\). in general, \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal {i}))\) is not equal to \(\mathcal {i}\). consider an atom \(c \in hb _p\) that is not the conclusion of any argument in \(\mathcal {a}_p\). then, \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal {i}))(c) = \textbf{f}\), but \(\mathcal {i}\) may assign any truth value to c. however, atom c is evaluated as false in any partial stable model. we obtain the following result: theorem 3 let \(\mathcal {b}_p = (\mathcal {a}_p, att _p)\) be the corresponding \(\textit{baf}\) of p and \(\mathcal m\) be a partial stable model of p. it holds that \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m)) = \mathcal m\). proof let \(\mathcal m\) be a partial stable model of p and \(c \in hb _p\). it suffices to prove the following results: \(\mathcal m(c) = \textbf{t}\) iff \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m))(c) = \textbf{t}\). (\(\rightarrow \)) assume \(\mathcal m(c) = \textbf{t}\). as \(\varpsi _p(\mathcal m) = \mathcal m\), there exists an argument a with \(\texttt{conc}(a) = c\) such that \(\texttt{vul}(a) \subseteq \textbf{f}(\mathcal m)\). this implies \(\mathcal {i}2\mathcal {l}_p(\mathcal m)(a) = \texttt{in}\) and \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m))(c) = \textbf{t}\). (\(\leftarrow \)) assume \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m))(c) = \textbf{t}\). then there exists \(a \in \mathcal {a}_p\) such that \(\texttt{conc}(a) = c\) and \(\mathcal {i}2\mathcal {l}_p(\mathcal m)(a) = \texttt{in}\). from definition 11, \(\mathcal m(c) = \textbf{t}\). \(\mathcal m(c) = \textbf{f}\) iff \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m))(c) = \textbf{f}\), which can be similarly proved. this means that when restricted to \(\beta \)-complete labellings and partial stable models, \(\mathcal {l}2\mathcal {i}_p\) and \(\mathcal {i}2\mathcal {l}_p\) are each other’s inverse. the function \(\mathcal {l}2\mathcal {i}_p\) when applied to \(\beta \)-complete labellings can be characterised as follows: $$ \mathcal {l}2\mathcal {i}_p(\mathcal {l})(c) = \left\{ \begin{array}{lcl} \textbf{t} & & \textit{if }\exists a \in \mathcal {a}_p \textit{ such that } \texttt{conc}(a) = c \textit{ and } \mathcal {l}(a) = \texttt{in}\\ \textbf{u}& & \textit{if }\exists a \in \mathcal {a}_p \textit{ such that } \texttt{conc}(a) = c \textit{ and } \mathcal {l}(a) = \texttt{undec}\\ \textbf{f}& & \textit{otherwise} \end{array} \right. $$ it is clear the relation between a \(\beta \)-complete labelling \(\mathcal {l}\) and \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})\) for those atoms \(c \in hb _p\) in which there is an argument a with \(\texttt{conc}(a) = c\). the distinction between them is for those atoms \(c' \in hb _p\) without a corresponding argument \(a'\) with \(\texttt{conc}(a') = c'\). for this case, \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})(c') = \textbf{f}\) and \(\mathcal {l}\) is not defined for any argument \(a'\) with \(\texttt{conc}(a') = c'\). from theorems 2 and 3, we can obtain the following result: theorem 4 let \(\mathcal {b}_p\) be the corresponding \(\textit{baf}\) of p. it holds \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\) iff \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})\) is a partial stable model of p; and \(\mathcal m\) is a partial stable model of p iff \(\mathcal {i}2\mathcal {l}_p(\mathcal m)\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\). proof 1. if \(\mathcal {l}\) is a \(\beta \)-complete labelling, then \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})\) is a partial stable model: let \(\mathcal m = \mathcal {l}2\mathcal {i}_p(\mathcal {l}) = (t, f, u)\). we show \(\varpsi _p(\mathcal m) = (t', f', u') = (t, f, u)\): \(c \in t\) iff there exists \(a \in \mathcal {a}_p\) such that \(\texttt{conc}(a) = c\) and \(\mathcal {l}(a) = \texttt{in}\) iff there exists \(a' \in \mathfrak {sup}(a)\) such that \(\texttt{conc}(a') = c\) and for every \(b \in att (a')\), it holds \(\mathcal {l}(b) = \texttt{out}\) iff there exists \(a' \in \mathcal {a}_p\) such that \(\texttt{conc}(a') = c\) and for every \(b \in att (a')\), it holds \(\mathcal {l}(b) = \texttt{out}\) iff there exists \(a' \in \mathcal {a}_p\) such that \(\texttt{conc}(a') = c\) and \(\texttt{vul}(a') \subseteq f\) iff \(c \in t'\). \(c \not \in f\) iff \(c \not \in f'\) can be similarly proved. 2. if \(\mathcal m\) is a partial stable model, then \(\mathcal {i}2\mathcal {l}_p(\mathcal m)\) is a complete labelling: let \(\mathcal m = (t, f, u)\) be a partial stable model of p. then \(\varpsi _p(\mathcal m) = (t, f, u)\). let \(a \in \mathcal {a}_p\). we will prove \(\mathcal {l} = \mathcal {i}2\mathcal {l}_p(\mathcal m)\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\): \(\mathcal {l}(a) = \texttt{in}\) iff \(\texttt{conc}(a) \in t\) iff there exists \(a' \in \mathcal {a}_p\) with \(\texttt{conc}(a') = \texttt{conc}(a)\) and \(\texttt{vul}(a') \subseteq f\) iff there exists \(a' \in \mathfrak {sup}(a)\) such that for every \(b \in att (a')\), it holds \(\mathcal {l}(b) = \texttt{out}\). \(\mathcal {l}(a) \ne \texttt{out}\) iff \(\texttt{conc}(a) \ne f\) iff there exists \(a' \in \mathcal {a}_p\) with \(\texttt{conc}(a') = \texttt{conc}(a)\) and \(\texttt{vul}(a') \cap t = \emptyset \) iff there exists \(a' \in \mathfrak {sup}(a)\) such that for every \(b \in att (a')\), it holds \(\mathcal {l}(b) \ne \texttt{in}\). 3. \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})\) is a partial stable model of p \(\rightarrow \) according to item 2 above, \(\mathcal {i}2\mathcal {l}_p(\mathcal {l}2\mathcal {i}_p(\mathcal {l}))\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\) \(\rightarrow \) (via theorem 2) \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\). 4. \(\mathcal {i}2\mathcal {l}_p(\mathcal m)\) is a \(\beta \)-complete labelling of \(\mathcal {b}_p\) \(\rightarrow \) according to item 1 above, \(\mathcal {l}2\mathcal {i}_p(\mathcal {i}2\mathcal {l}_p(\mathcal m))\) is a partial stable model of p \(\rightarrow \) (via theorem 3) \(\mathcal m\) is a partial stable model of p. theorem 4 is one of the main results of this paper. it plays a central role in ensuring the equivalence between the semantics for \(\textit{nlp}\)s and their counterpart for \(\textit{baf}\)s: theorem 5 let p be an \(\textit{nlp}\) and \(\mathcal {b}_p = (\mathcal {a}_p, att _p)\) be the corresponding \(\textit{baf}\). it holds \(\mathcal {l}\) is a \(\beta \)-grounded (resp. \(\beta \)-preferred, \(\beta \)-stable, \(\beta \)-semi-stable) labelling of \(\mathcal {b}_p\) iff \(\mathcal {l}2\mathcal {i}_p(\mathcal {l})\) is a well-founded (resp. regular, stable, l-stable) model of p. the following result is a direct consequence of theorems 3 and 5: corollary 1 let p be an \(\textit{nlp}\) and \(\mathcal {b}_p = (\mathcal {a}_p, att _p)\) its corresponding \(\textit{baf}\). \(\mathcal m\) is a well-founded (resp. regular, stable, l-stable) model of p iff \(\mathcal {i}2\mathcal {l}_p(\mathcal m)\) is a \(\beta \)-grounded (resp. \(\beta \)-preferred, \(\beta \)-stable, \(\beta \)-semi-stable) labelling of \(\mathcal {b}_p\). next, we consider the \(\textit{nlp}\) exploited by caminada et al. [7] as a counterexample to show that in general, l-stable models and semi-stable labellings do not coincide with each other in their translation from \(\textit{nlp}\)s to \( aaf \)s: example 4 let p and \(\mathcal {b}_p\) be the \(\textit{nlp}\) and corresponding \(\textit{baf}\) in fig. 3, where the arguments of \(\mathcal b_p\) are fig. 3. \(\textit{nlp}\) p and its corresponding \(\textit{baf}\) \(\mathcal {b}_p\) full size image note in \(\mathcal {b}_p\) that there is both a mutual attack and a mutual support between arguments \(c_1\) and \(c_2\). as expected from theorems 4 and 5, we obtain in table 1 the equivalence between \(\beta \)-semantics of \(\textit{baf}\)s and 3-valued semantics of \(\textit{nlp}\)s. we emphasise the coincidence between l-stable models and \(\beta \)-semi-stable labellings in table 1 as it does not occur in [7]. in that reference, the corresponding \( aaf \) possesses two semi-stable labellings in contrast with the unique l-stable model \(\mathcal m_3\) of p. table 1. semantics for p and \(\mathcal {b}_p\)full size table note that if the support relation was ignored in the framework of fig. 2 as was done in [7], the \(\beta \)-complete labellings of \(\mathcal {b}_p\) would be \(\mathcal {l}_1, \mathcal {l}_2', \mathcal {l}_3\), where \(\mathcal {l}_2'\!=\! ( \left\{ a \right\} ,\!\left\{ b, c_1 \right\} ,\!\left\{ c_2, d, e \right\} )\), and \(\mathcal {l}_2'\) and \(\mathcal {l}_3\) would be the semi-stable \(\beta \)-complete labellings and the correspondence with l-stable models would not sustain anymore. 4 from \(\textit{baf}\)s to \(\textit{nlp}\)s now we will provide a translation for the opposite direction, i.e., from \(\textit{baf}\)s to \(\textit{nlp}\)s. as in the previous section, this translation guarantees the equivalence between the semantics for \(\textit{nlp}\)s and their counterpart for \(\textit{baf}\)s. definition 12 (corresponding \(\textit{nlp}\)). let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\). for any \(a, b \in \mathcal {a}\), define the rule \(r_{a,b}\) such that \( head (r_{a,b}) = \mathfrak {sup}(a)\), \( body ^+(r_{a,b}) = \emptyset \) and \( body ^-(r_{a,b}) = \{\mathtt {not\ }\mathfrak {sup}(x) \mid x \in att (b)\}\). the corresponding \(\textit{nlp}\) of \(\mathcal b\) is \(p_\mathcal {b} = \left\{ r_{a,b} \mid a \in \mathcal {a}, b\in \mathfrak {sup}(a) \right\} \). each argument is encoded in the corresponding \(\textit{nlp}\) by its set of supporters, and the herbrand base of \(p_\mathcal {b}\) is \( hb _{p_\mathcal {b}} = \{\mathfrak {sup}(a) \mid a \in \mathcal {a}\}\). intuitively, with respect to any partial stable model of \(p_\mathcal {b}\), we know \(\mathfrak {sup}(a)\) is true iff there exists \(b \in \mathfrak {sup}(a)\) such that \(\mathfrak {sup}(x)\) is false for every \(x \in att (b)\). this property follows directly from definition 12, that was conceived to reflect the concept of acceptability in the \(\beta \)-complete semantics, where an argument \(a \in \mathcal {a}\) is accepted iff there exists \(b \in \mathfrak {sup}(a)\) such that x is rejected for every \(x \in att (b)\). example 5 let \(\mathcal b = (\mathcal {a}, att , sup )\) be the \(\textit{baf}\) of fig. 4a; its corresponding \(\textit{nlp}\) \(p_\mathcal {b}\) is shown in fig. 4b. fig. 4. (a) \(\textit{baf}\) \(\mathcal b\) and (b) its corresponding \(\textit{nlp}\) \(p_\mathcal {b}\) full size image observe that distinct arguments \(a, a'\) are associated to the same atom if \(\mathfrak {sup}(a) = \mathfrak {sup}(a')\). for instance, the corresponding \(\textit{nlp}\) of example 4b encodes both arguments \(b_1\) and \(b_2\) as the atom \(\mathfrak {sup}(b_1) = \mathfrak {sup}(b_2) = \left\{ b_1, b_2, d\right\} \). from the supporters of \(b_1\), we obtain the rules \(r_{b_1, b_1} = \left\{ b_1, b_2, d\right\} \leftarrow \mathtt {not\ }\left\{ a\right\} ,\) \(\mathtt {not\ }\left\{ c\right\} \); \(r_{b_1,b_2} = \left\{ b_1, b_2, d\right\} \leftarrow \mathtt {not\ }\left\{ d\right\} \); and \(r_{b_1,d} = \left\{ b_1, b_2, d\right\} \leftarrow \mathtt {not\ }\left\{ b_1, b_2, d\right\} , \mathtt {not\ }\left\{ c\right\} , \mathtt {not\ }\left\{ e\right\} \). the intuition is that \(b_1\) is accepted iff either (i) a and c are rejected or (ii) d is rejected or (iii) \(b_1, b_2, c, d, e\) are rejected. condition (iii) seems contradictory, but it is similar to what occurs when an argument attacks itself (e.g., argument e is accepted iff e is rejected). as done in sect. 3, we want to establish a bijection between (some class of) labellings and interpretations, such that \(\beta \)-complete labellings correspond to partial stable models. however, we can have more labellings than interpretations. example 6 let \(\mathcal b\) be the \(\textit{baf}\) \(\mathcal b = (\left\{ a, b, c\right\} , \left\{ (c, b)\right\} , \left\{ (a, b), (b, a)\right\} )\). its corresponding \(\textit{nlp}\) \(p_\mathcal {b}\) has 3 rules: \(\left\{ a, b\right\} \leftarrow \left\{ c\right\} \) and \(\left\{ a, b\right\} \) and \(\left\{ c\right\} \). there are \(3^{|\mathcal {a}|} = 27\) labellings of \(\mathcal b\) and \(3^{| hb _{p_\mathcal {b}}|} = 9\) interpretations of \(p_\mathcal {b}\). hence, there is no bijection between labellings of \(\mathcal b\) and interpretations of \(p_\mathcal {b}\). notice that a labelling may assign different labels to a and b, whereas an interpretation assigns only one truth value to the atom \(\{a, b\}\). although there is no bijection between labellings and interpretations for the example above, there is always a bijection if we restrict our translations to labellings that assign the same label to arguments with identical sets of supporters. definition 13 let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\). a labelling \(\mathcal {l}\) of \(\mathcal b\) respects \(\mathfrak {sup}\) iff for every \(a, b \in \mathcal {a}\) with \(\mathfrak {sup}(a) = \mathfrak {sup}(b)\) it holds \(\mathcal {l}(a) = \mathcal {l}(b)\). for the \(\textit{baf}\) \(\mathcal b\) of example 6, there are 9 labellings respecting \(\mathfrak {sup}\), since a labelling that respects \(\mathfrak {sup}\) assigns the same label to a and b. the representation of possibly many arguments by the same atom is motivated by the fact that every \(\beta \)-complete labelling satisfies the following property: proposition 1 if \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal b\), then \(\mathcal {l}\) respects \(\mathfrak {sup}\). by restricting labellings to those that respect \(\mathfrak {sup}\), the relationship between labellings and interpretations is straightforward: definition 14 (\(\mathcal {l}2\mathcal {i}_\mathcal {b}\) and \(\mathcal {i}2\mathcal {l}_\mathcal {b}\) functions). let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\) and \(p_\mathcal {b}\) be its corresponding \(\textit{nlp}\). denote the set of all labellings of \(\mathcal b\) respecting \(\mathfrak {sup}\) as \(\mathcal {l} ab ^*\) and the set of all interpretations of \(p_\mathcal {b}\) as \(\mathcal {i} nt \). we introduce the functions (i) \(\mathcal {l}2\mathcal {i}_\mathcal {b}: \mathcal {l} ab ^* \rightarrow \mathcal {i} nt \), in which for every \(\mathcal {l} \in \mathcal {l} ab ^*\), \(\mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {l})(\mathfrak {sup}(a)) = \textbf{t}\) (resp. \(\textbf{f}, \textbf{u}\)) if \(\mathcal {l}(a) = \texttt{in}\) (resp. \(\texttt{out}, \texttt{undec}\)); and (ii) \(\mathcal {i}2\mathcal {l}_\mathcal {b}: \mathcal {i} nt \rightarrow \mathcal {l} ab ^*\), in which for every \(\mathcal m \in \mathcal {i} nt \), \(\mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal m)(a) = \texttt{in}\) (resp. \(\texttt{out}, \texttt{undec}\)) if \(\mathcal m(\mathfrak {sup}(a)) = \textbf{t}\) (resp. \(\textbf{f}, \textbf{u}\)). in contrast with \(\mathcal {l}2\mathcal {i}_p\) and \(\mathcal {i}2\mathcal {l}_p\), the functions \(\mathcal {l}2\mathcal {i}_\mathcal {b}\) and \(\mathcal {i}2\mathcal {l}_\mathcal {b}\) are each other’s inverse in the general case: theorem 6 let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\) and \(p_\mathcal {b}\) be its corresponding \(\textit{nlp}\). for any labelling \(\mathcal {l}\) of \(\mathcal b\) respecting \(\mathfrak {sup}\), it holds \(\mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {l})) = \mathcal {l}\); and for any interpretation \(\mathcal {i}\) of \(p_\mathcal {b}\), it holds \(\mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal {i})) = \mathcal {i}\). the next lemma is essential for the correspondence between \(\beta \)-complete labellings of \(\mathcal b\) and partial stable models of \(p_\mathcal {b}\). its proof follows straightforwardly from definitions 12 and 14. lemma 1 let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\) and \(p_\mathcal {b}\) be its corresponding \(\textit{nlp}\). let \(\mathcal {l}\) be a labelling of \(\mathcal b\) respecting \(\mathfrak {sup}\) and \(\mathcal m\) be an interpretation of \(p_\mathcal {b}\). if \(\mathcal {l} = \mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal m)\) or \(\mathcal m = \mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {l})\), then for any \(a \in \mathcal {a}\), (i) \( att (b) \subseteq \texttt{out}(\mathcal {l})\) for some \(b \in \mathfrak {sup}(a)\) iff \(\{\mathfrak {sup}(x) \mid \mathtt {not\ }\mathfrak {sup}(x) \in body ^-(r)\} \subseteq \textbf{f}(\mathcal m)\) for some \(r \in p_\mathcal {b}\) with \( head (r) = \mathfrak {sup}(a)\); and (ii) \( att (b) \cap \texttt{in}(\mathcal {l}) \ne \emptyset \) for every \(b \in \mathfrak {sup}(a)\) iff \(\{\mathfrak {sup}(x) \mid \mathtt {not\ }\mathfrak {sup}(x) \in body ^-(r)\} \cap \textbf{t}(\mathcal m) \ne \emptyset \) for every rule \(r \in p_\mathcal {b}\) with \( head (r) = \mathfrak {sup}(a)\). from lemma 1, we obtain a similar result to theorem 4: theorem 7 let \(\mathcal b = (\mathcal {a}, att , sup )\) be a \(\textit{baf}\) with corresponding \(\textit{nlp}\) \(p_\mathcal {b}\), \(\mathcal {l}\) be a labelling of \(\mathcal b\) respecting \(\mathfrak {sup}\) and \(\mathcal m\) be an interpretation of \(p_\mathcal {b}\). \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal b\) iff \(\mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {l})\) is a partial stable model of \(p_\mathcal {b}\); and \(\mathcal m\) is a partial stable model of \(p_\mathcal {b}\) iff \(\mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal m)\) is a \(\beta \)-complete labelling of \(\mathcal b\). proof owing to space restrictions, only part of the proof is shown. assume \(\mathcal {l}\) is a \(\beta \)-complete labelling of \(\mathcal b\). it holds \(\mathfrak {sup}(a) \in \textbf{t}(\mathcal {i}) \longleftrightarrow \mathcal {l}(a) = \texttt{in}\longleftrightarrow \exists b \in \mathfrak {sup}(a): att (b) \subseteq \texttt{out}(\mathcal {l}) \overset{\text {lemma } 1}{\longleftrightarrow } \exists r \in p_\mathcal {b}: head (r) = \mathfrak {sup}(a) \wedge \{\mathfrak {sup}(x) \mid \mathtt {not\ }\mathfrak {sup}(x) \in body ^-(r)\} \subseteq \textbf{f}(\mathcal {i}) \longleftrightarrow \exists r \in p_\mathcal {b}/\mathcal {i}: head (r) = \mathfrak {sup}(a) \wedge body ^+(r) = \emptyset \longleftrightarrow \exists r \in p_\mathcal {b}/\mathcal {i}: head (r) = \mathfrak {sup}(a) \wedge body ^+(r) \subseteq \textbf{t}(\psi _{p_\mathcal {b}}(\mathcal {i})) \longleftrightarrow \mathfrak {sup}(a) \in \textbf{t}(\psi _{p_\mathcal {b}}(\mathcal {i}))\). similarly, we can prove \(\mathfrak {sup}(a) \in \textbf{f}(\mathcal {i})\) iff \(\mathfrak {sup}(a) \in \textbf{f}(\varpsi _{p\mathcal b}(\mathcal {i}))\). from theorem 7, we can ensure the equivalence between the semantics for \(\textit{baf}\)s and their counterpart for \(\textit{nlp}\)s: theorem 8 let \(\mathcal b\) be a \(\textit{baf}\) and \(p_\mathcal {b}\) be its corresponding \(\textit{nlp}\). for any labelling \(\mathcal {l}\) of \(\mathcal b\) respecting \(\mathfrak {sup}\), it holds \(\mathcal {l}\) is a \(\beta \)-grounded (resp. \(\beta \)-preferred, \(\beta \)-stable, \(\beta \)-semi-stable) labelling of \(\mathcal b\) iff \(\mathcal {l}2\mathcal {i}_\mathcal {b}(\mathcal {l})\) is a well-founded (resp. regular, stable, l-stable) model of \(p_\mathcal {b}\). the following result is a direct consequence of theorems 6 and 8: corollary 2 let \(\mathcal b\) be a \(\textit{baf}\) and \(p_\mathcal {b}\) be its corresponding \(\textit{nlp}\). \(\mathcal m\) is a well-founded (resp. regular, stable, l-stable) model of \(p_\mathcal {b}\) iff \(\mathcal {i}2\mathcal {l}_\mathcal {b}(\mathcal m)\) is a \(\beta \)-grounded (resp. \(\beta \)-preferred, \(\beta \)-stable, \(\beta \)-semi-stable) labelling of \(\mathcal b\). recalling the \(\textit{baf}\) \(\mathcal b_p\) of example 4, its corresponding \(\textit{nlp}\) \(p_\mathcal {b_p}\) is structurally identical (i.e., equal under renaming) to the \(\textit{nlp}\) p of example 4. we obtain the expected equivalence results related to their semantics (see table 2). table 2. semantics for \(\mathcal b_p\) and \(p_\mathcal {b_p}\)full size table 5 conclusions in this paper, we extend the work of caminada et al. [7] by considering argumentation frameworks with support under the \(\beta \)-semantics [1]. we observe that \(\beta \)-complete, \(\beta \)-grounded, \(\beta \)-preferred, \(\beta \)-stable and \(\beta \)-semi-stable semantics of bipolar argumentation frameworks (\(\textit{baf}\)s) respectively coincide with the partial stable, well-founded, regular, stable and l-stable semantics of normal logic programs (\(\textit{nlp}\)s). in particular, we note that the inclusion of support in argumentation allowed the connection between \(\beta \)-semi-stable and l-stable semantics, whereas semi-stable and l-stable semantics did not coincide for attack-only argumentation frameworks [7]. each formalism encodes knowledge according to its own perspective: argumentation is concerned about arguments and their interaction, whereas \(\textit{nlp}\)s are interested in deriving atoms from rules. we provide explicit translations between these formalisms and their semantics, allowing them to be used interchangeably and enhancing our understanding about how they relate to each other. future works include exploring the connection of \(\textit{nlp}\)s with other extensions of \( aaf \)s. notes 1.the above definition consists of a least fix-point of the immediate consequence operator \(\varpsi \) defined in [12], which is guaranteed to exist and be unique for positive programs. references alcântara, j., cordeiro, r.: bipolar argumentation frameworks with a dual relation between defeat and defence. j. logic comput., exae006 (2024) google scholar  alcântara, j., sá, s., acosta-guadarrama, j.: on the equivalence between abstract dialectical frameworks and logic programs. theory pract. logic program. 19(5–6), 941–956 (2019) article  mathscinet  math  google scholar  alcântara, j., cordeiro, r., sá, s.: on the equivalence between logic programming and setaf (2024), (submitted) google scholar  alfano, g., greco, s., parisi, f., trubitsyna, i.: on the semantics of abstract argumentation frameworks: a logic programming approach. theory pract. logic program. 20(5), 703–718 (2020) article  mathscinet  math  google scholar  amgoud, l., cayrol, c., lagasquie-schiex, m.c., livet, p.: on bipolarity in argumentation frameworks. int. j. intell. syst. 23(10), 1062–1093 (2008) article  math  google scholar  brewka, g., woltran, s.: abstract dialectical frameworks. in: twelfth international conf. on the principles of knowledge representation and reasoning, pp. 102–111. aaai press (2010) google scholar  caminada, m., sá, s., alcântara, j., dvořák, w.: on the equivalence between logic programming semantics and argumentation semantics. int. j. approximate reasoning 58, 87–111 (2015) article  mathscinet  math  google scholar  caminada, m., schulz, c.: on the equivalence between assumption-based argumentation and logic programming. j. artifi. intell. res. 60, 779–825 (2017) article  mathscinet  math  google scholar  čyras, k., schulz, c., toni, f.: capturing bipolar argumentation in non-flat assumption-based argumentation. in: an, b., bazzan, a., leite, j., villata, s., van der torre, l. (eds.) prima 2017. lncs (lnai), vol. 10621, pp. 386–402. springer, cham (2017). https://doi.org/10.1007/978-3-319-69131-2_23 chapter  math  google scholar  dung, p.: on the acceptability of arguments and its fundamental role in nonmonotonic reasoning, logic programming and n-person games. artif. intell. 77(2), 321–357 (1995) article  mathscinet  math  google scholar  eiter, t., leone, n., saccá, d.: on the partial semantics for disjunctive deductive databases. ann. math. artif. intell. 19(1–2), 59–96 (1997) article  mathscinet  math  google scholar  przymusinski, t.: the well-founded semantics coincides with the three-valued stable semantics. fund. inform. 13(4), 445–463 (1990) mathscinet  math  google scholar  rocha, v.h.n., cozman, f.g.: bipolar argumentation frameworks with explicit conclusions: connecting argumentation and logic programming. in: nmr, pp. 49–60 (2022) google scholar  strass, h.: approximating operators and semantics for abstract dialectical frameworks. artif. intell. 205, 39–70 (2013) article  mathscinet  math  google scholar  toni, f., et al.: just a matter of perspective. comput. models argument: proc. comma 2022 353, 212 (2022) google scholar  wu, y., caminada, m., gabbay, d.m.: complete extensions in argumentation coincide with 3-valued stable models in logic programming. stud. logica. 93(2–3), 383 (2009) article  mathscinet  math  google scholar  download references acknowledgments this work was supported by the brazilian research agencies funcap and cnpq. author information authors and affiliations department of computer science, federal university of ceará, fortaleza, brazil renan cordeiro & joão alcântara authors renan cordeiroview author publications search author on:pubmed google scholar joão alcântaraview author publications search author on:pubmed google scholar corresponding author correspondence to renan cordeiro . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cordeiro, r., alcântara, j. (2025). on the equivalence between logic programs and bipolar argumentation frameworks. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_17 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_17 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords equivalence bipolar argumentation logic programming publish with us policies and ethics profiles renan cordeiro view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature hierarchical time-aware approach for video summarization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper hierarchical time-aware approach for video summarization conference paper first online: 12 october 2023 pp 274–288 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) hierarchical time-aware approach for video summarization download book pdf download book epub leonardo vilela cardoso  orcid: orcid.org/0000-0003-0365-64039, gustavo oliveira rocha gomes  orcid: orcid.org/0009-0008-8384-62189, silvio jamil ferzoli guimarães  orcid: orcid.org/0000-0001-8522-20569 & … zenilton kleber gonçalves do patrocínio júnior  orcid: orcid.org/0000-0003-0804-17909  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 528 accesses 1 citation abstract video summarization consists of generating a concise video representation that captures all its meaningful information. however, conventional summarization techniques often fall short of capturing all the significant events in a video due to their inability to incorporate the hierarchical structure of the video content. this work proposes an unsupervised method, named hierarchical time-aware summarizer–hietasumm, that uses a hierarchical approach for that task. in this regard, hierarchical strategies for video summarization have emerged as a promising solution, in which video content is modeled as a graph to identify keyframes that represent the most relevant information. this approach enables the extraction of the frames that convey the central message of the video, resulting in a more effective and precise summary. experimental results indicate that the proposed approach has great potential. specifically, it seems to enhance coherence among different video segments, reducing frame redundancy in the generated summaries, and enhancing the diversity of selected keyframes. code available at https://github.com/imscience-ppginf-pucminas/hietasumm. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others hierarchical clustering-based video summarization chapter © 2020 hierarchical extraction algorithm of video summary based on multi-feature similarity chapter © 2021 unsupervised temporal attention summarization model for user created videos chapter © 2021 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. diversity-oriented synthesis film theory multimedia information systems multimedia journalism targeted resequencing time-lapse imaging 1 introduction video summarization is a challenging task that has gained significant attention in the computer vision and multimedia communities [1, 19]. one of the goals of video summarization is to extract essential information from a video and present it in a condensed format [9, 10, 16, 18]. this task is essential for applications such as video captioning, surveillance, synopsis of news videos [4, 5, 13], and video retrieval, among others [1]. the video summarization task involves several sub-tasks, such as keyframe extraction, object tracking, and summarization itself. the keyframe extraction step selects representative frames that capture the essence of the video, while object tracking aims to track important objects across frames [2]. the summarization step involves selecting a subset of keyframes that provide a comprehensive summary of the video while minimizing redundancy [6]. video summarization techniques can be categorized into unsupervised and supervised approaches, depending on the availability of training data. while unsupervised techniques aim to identify patterns in the video data without any prior knowledge, supervised techniques require labeled data to train the summarization model [1, 19]. video summarization is particularly useful when dealing with a video collection containing lots of repeated or redundant information spread out over many points in time [6]. in such cases, it becomes a challenge to analyze the entire video and efficiently extract useful information. video summary techniques can help identify the most important frames in the video that are likely to contain unique and relevant information. by summarizing the video, one can achieve a condensed version that retains the most relevant information while reducing the overall size of the video collection [6]. this allows us to efficiently analyze large video datasets and highlight the most important information, improving the overall effectiveness of video analysis tasks [10, 16]. fig. 1. example of the summary generated by the hietasumm method compared with the groundtruth. in this case, the hietasumm method returns 13 keyframes in contrast to two annotators, the first one (user 1) selects 11 keyframes while the second (user 4) selects 9 keyframes for video v21 of the openvideo dataset. full size image figure 1 shows that creating a single summary for a video that accurately reflects every user’s perception and preferences can be a challenging task. since groundtruth data is generated by humans, the interpretation of each user of what is essential and relevant may vary. to generate the groundtruth for the video summarization task, annotators must watch the entire video and identify the most crucial moments. however, what one annotator perceives as essential may differ from another, leading to a subjective groundtruth. hence, the subjectivity of groundtruth generated for each user is a critical aspect to consider in a machine learning method [2, 16]. regardless of the difficulties related to the subjectivity of groundtruth generated by several users, many unsupervised methods have been proposed over the years. in [11], the authors presented a platform for customizing video summaries. using clustering techniques, they proposed a method named visto, which analyzed low-level features to determine the similarity between frames. keyframe selection is done by selecting the center of each cluster then a post-processing step is responsible for analyzing and removing possible frame redundancies. in [9], the authors presented a clustering-based strategy to solve the video summarization task named vsumm. first, a sampling process is made to reduce the number of frames under analysis. then, frames represented by color histograms were grouped into similar sets by a k-means algorithm. vsumm results tended to group dispersed frames in time that may have a considerable temporal separation. in [17], the authors presented a graph-based approach for video summarization named hsumm. the proposed approach was hierarchical and comprised keyframe extraction, scene segmentation, and video summarization stages. during the keyframe extraction stage, their method selected representative frames based on image quality and diversity. in the scene segmentation stage, the video was divided into different scenes based on the visual similarity between frames. finally, keyframes were combined to generate a video summary. the proposed approach employed a hierarchical graph-based clustering that was capable of generating effective video summaries. in [15], the authors presented an unsupervised approach for summarizing a collection of videos. they developed a diversity-aware optimization method for multi-video summarization by exploring the videos’ complementarity. the video summarization landscape has evolved over the last few years, especially after the introduction of deep learning algorithms. the study in [14] focused on egocentric’ video summarization and the challenges of this task. in [3], the authors concentrated on summarization methods that are directly applied to the compressed domain. finally, the authors in [20] presented the relevant bibliography for dynamic video summarization. according to [1], in deep-learning-based video summarization methods, the video content is represented by deep feature vectors extracted by pre-trained neural networks. the extracted features are then utilized by a deep summarizer network and its output can be either a set of keyframes (i.e., a static summary) or a set of video fragments (that form a dynamic summary). the study conducted by [6] introduces a hierarchical approach for generating a fixed number of keyframes for video captioning. their approach involves generating a fixed number of keyframes per video, which serves as a representative distribution of the video content. these keyframes are then used as input for a transformer-based method. in contrast to the proposed by [6], our proposed approach takes a different direction. we propose the creation of a dynamic number of keyframes to serve as a comprehensive summary for each video. this approach recognizes the inherent variability in video content and aims to capture the most salient frames specific to each video. consequently, the number of keyframes in the summary differs across videos within the same dataset. by employing a dynamic keyframe selection process, the proposed approach enables a more discriminating representation of the video content compared to the fixed number approach suggested by [6]. this work proposes an unsupervised method for video summarization that considers changes in video content over time, named hierarchical time-aware summarizer–hietasumm. similarly to recent deep-learning-based approaches, the proposed method uses pre-trained neural networks to generate video frame descriptions. however, it does not adopt a deep summarizer network to avoid the challenges related to its training. instead, a hierarchical graph-based clustering strategy is adopted. it is worth mentioning that the proposed method assesses frame importance over time for selecting keyframes that comprise the video summary, which is different from other hierarchical approaches. the major contributions of this work are two-fold: (i) a strategy for video summarization that incorporates frame importance over time for selecting keyframes; and (ii) the identification of keyframes through a hierarchical graph-based clustering using deep-learning-based descriptors and a dynamic strategy to define summary sizes. this work is organized as follows. section 2 defines many concepts used in this work. section 3 presents the proposed method, followed by the experimental results in sect. 4. finally, sect. 5 draws some conclusions and future work proposals. 2 fundamental concepts let \({\mathbb {a}}\subset {\mathbb {n}}^2\), \({\mathbb {a}}=\{0,\ldots ,\textrm{h}-1\}\times \{0,\ldots ,\textrm{w}-1\}\), where \(\textrm{h}\) and \(\textrm{w}\) are the width and height of each frame, respectively, and, \({\mathbb {t}}\subset {\mathbb {n}}\), \({\mathbb {t}}=\{0,\ldots , n-1\}\), in which n is the number of frames of a video. a frame f is a function from \({\mathbb {a}}\) to \({\mathbb {r}}^3\), where for each spatial position (x, y) in \({\mathbb {a}}\), f(x, y) represents the color value at pixel location (x, y). a video \(\textrm{v}_{ n }^{}\), in domain \({\mathbb {a}}\times {\mathbb {t}}\), can be seen as a sequence of frames \({ f}_{}\). it can be described by \( \textrm{v}_{ n }^{}={({ f}_{})}_{t\in {\mathbb {t}}}\), where \( n \) is the number of frames contained in the video. a frame \({ f}_{}\) is usually described in terms of a global descriptor \(d({ f}_{})\). let \({ f}_{t_1}\) and \({ f}_{t_2}\) be two video frames at locations \(t_1\) and \(t_2\), respectively. the (dis)similarity between \({ f}_{t_1}\) and \({ f}_{t_2}\) can be evaluated by a distance measure \(\mathcal {d}{(d({ f}_{t_1}),d({ f}_{t_2}))}\) between their descriptors. there are several choices for \(\mathcal {d}{(d({ f}_{t_1}),d({ f}_{t_2}))}\), i.e., the distance measure between two frames depending on the global descriptor, e.g. histogram/frame difference, histogram intersection, difference of histograms means, and even the \(l_2\) norm. a time-aware frame similarity graph \(g_\delta = (v, e_\delta )\) is a weighted undirected graph. each node \(v_{t} \in v\) represents a frame \(f_{t} \in \textrm{v}_{n}^{}\). there is an edge \(e \in e_\delta \) with a weight \(w(e) = \mathcal {d}{(d({ f}_{t_1}),d({ f}_{t_2}))}\) between two nodes \(v_{t_1}\) and \(v_{t_2}\) if the difference between their time indexes falls below a specified threshold \(\delta \), i.e., $$\begin{aligned} e_\delta = \{\;(v_{t_1},v_{t_2}, \mathcal {d}{(d({ f}_{t_1}),d({ f}_{t_2}))})\; |\;v_{t_1}, v_{t_2} \in v, v_{t_1} \ne v_{t_2}, |t_2 t_1| \le \delta \}. \end{aligned}$$ (1) this constraint over the frames’ time indexes limits the connections between distant video frames, effectively allowing the proposed method to consider two frames as similar only if they are not very far in time. this is a noteworthy distinction from many other approaches in the literature, which may consider two frames as similar independently from their time occurrence. doing that permits the proposed method to assess frame importance over time for selecting it as a keyframe to form the video summary even when it seems to reoccur throughout the video. figure 2(a) illustrates a time-aware frame similarity graph with \(\delta = 4\). similar to [17], this work also constructs a hierarchy based on a minimum spanning tree (mst) of the original graph. so, we define an edge-weighted tree of frames \(t_{g_\delta }= (v, e_\delta ^*)\) is a connected acyclic subgraph of \(g_\delta \), i.e., \(e_\delta ^* \subseteq e_\delta \). the weight of \(t_{g_\delta }\) is equal to the sum of weights of all edges belonging to \(e_\delta ^*\), i.e., \(w(t_{g_\delta }) = \sum _{e \in e_\delta ^*} w(e)\). the minimum spanning tree of frames \(t^{*}_{g_\delta }\) is a tree of frames whose weight is minimal. given a finite set v, a partition of v is a set \(\textbf{p}\) of nonempty disjoint subsets of v whose union is v. any element of \(\textbf{p}\), denoted by \(\textbf{r}\), is called a region of \(\textbf{p}\). given two partitions \(\textbf{p}\) and \(\textbf{p}^\prime \) of v, \(\textbf{p}^\prime \) is said to be a (total) refinement of \(\textbf{p}\), denoted by \(\textbf{p}^{\prime } \preceq \textbf{p}\), if any region of \(\textbf{p}^\prime \) is included in a region of \(\textbf{p}\). let \(\mathcal {h}= (\textbf{p}_1, \dots , \textbf{p}_\ell )\) be a set of \(\ell \) partitions on v. \(\mathcal {h}\) is a hierarchy if \(\textbf{p}_{i-1} \preceq \textbf{p}_i\), for any \(i \in \{ 2, \dots , \ell \}\). according to [8], an mst can be utilized to represent a hierarchy, and a weighted mst of a graph can address any connected hierarchy for that graph. additionally, the work in [12] demonstrated that creating a hierarchical graph segmentation involves reweighting an mst using a dissimilarity measure between regions. thus, the proposed method utilizes an mst of frame similarity graph \(t^{*}_{g_\delta }\) to obtain a hierarchy \(\mathcal {h}\) which is then used to obtain frame clusters. finally, a hierarchical segmentation of \(g_\delta \) into k components is equivalent to the partition of a hierarchy \(\mathcal {h}\) into k regions (containing more similar elements) and can be done by removing \(k-1\) edges that present higher weights (representing greater dissimilar) from the \(t^{*}_{g_\delta }\) (since it represents \(\mathcal {h}\)). this strategy incorporates a similarity measure between clusters while partitioning the graph, providing a more comprehensive approach than traditional methods that only consider the similarity between isolated frames. 3 hierarchical time-aware video summarization figure 2 illustrates the proposed method steps. the main steps of hietasumm method are the following: (i) generation of a time-aware frame similarity graph \(g_\delta \) to represent a video; (b) computation of a minimum spanning tree \(t^{*}_{g_\delta }\) for that graph; (c) creation of a hierarchy \(\mathcal {h}\) based on the \(t^{*}_{g_\delta }\); (d) generation of subsets of frames through cuts on the hierarchy; and (e) selection of keyframes to represent each subset. fig. 2. illustration of the proposed method steps: (a) generation of a time-aware frame similarity graph \(g_\delta \) for a video; (b) computation of its minimum spanning tree \(t^{*}_{g_\delta }\); (c) creation of a hierarchy \(\mathcal {h}\) based on \(t^{*}_{g_\delta }\); (d) generation of subsets of frames through hierarchy cuts (edge removals); and (e) selection of keyframes to represent each subset. these keyframes are the result of the summarization process. full size image the hietasumm method (see algorithm 1) created and uses a frame similarity graph \(g_\delta \). each vertex represents a distinct video frame and there is an edge between two vertices if the difference between their time indexes falls below a specified threshold \(\delta _t\). equation 2 represents this constraint and is implemented at line 7 of algorithm 1. $$\begin{aligned} |t_2 t_1| < \delta _t \end{aligned}$$ (2) in which \(t_f\) and \(t_{f'}\) represent the time indexes of frames f and \(f'\), respectively. additionally, the edge weight represents the (dis)similarity between frames. algorithm 1 hierarchical time-aware video summarization full size image the proposed method employs the kruskal algorithm to obtain the mst \(t^{*}_{g_\delta }\) from \(g_\delta \), while the watershed by area [7] is used to generate a hierarchy \(\mathcal {h}\) from \(t^{*}_{g_\delta }\). once a hierarchy \(\mathcal {h}\) is constructed, a hierarchical segmentation of \(g_\delta \) generates a video summary of size k. for that, the proposed method needs only to remove the \(k-1\) edges with higher weights from \(\mathcal {h}\). instead of generating a fixed-size video summary, we adopt a strategy for identifying the moment when stability is reached during the edge removal process that is similar (but distinct) to the one used in [17]. let \(e'\) be the edge with the highest weight in the hierarchy \(\mathcal {h}\). thus, the edge \(e'\) is removed only when its weight \(w(e')\) is greater than or equal to an equilibrium measure function \(f(e')\), i.e., \(w(e') \ge \textbf{f}(e)\). in this work, the equilibrium measure function is given by eq. 3. $$\begin{aligned} \textbf{f}(e) = \gamma \sigma _w(e) \end{aligned}$$ (3) in which \(\sigma _w(e)\) represents the standard deviation of all edge weights of the connected component that contains edge e, and \(\gamma \) is a parameter related to the allowed variability. during tests, we have set \(\gamma \) empirically. finally, after dividing the hierarchy into several connected components, central frames (concerning chronological order) are selected as keyframes for the video summary. this dynamic choice of the number of components and, consequently, the size of the video summary becomes essential when it comes to videos that contain numerous very similar scenes. in such cases, employing a static number of frames for all videos can result in redundant and repetitive content in the summary. by adopting a dynamic approach, the method can infer an adequate summary size based on the specific video content and characteristics. 4 experimental results this section provides a comprehensive analysis of results obtained by the proposed approach to video summarization with a dynamic selection of video summaries. we compared hietasumm with other unsupervised video summarization methods, namely hsumm [17], vsumm1 [9], vsumm2 [9], visto [11] and open video summaries (referred to like ovsummary). these comparative assessments allow for a comprehensive review of the performance and effectiveness of hietasumm against these established approaches. 4.1 implementation details and dataset similar to [11, 17], we applied the proposed method to the same collections of videos from the openvideo dataset (referred to as the vsumm dataset in [19]). this dataset contains 50 videos of different genres. all videos are in mpeg-1 format (30 fps, 352 \(\times \) 240 pixels). the genres are distributed into documentary, educational, ephemeral, historical, and lecture. the time duration of each video varies from 01 to 04 min. the process of creating of user summary consists of the collaboration of 50 different persons. each user is dealing with the task of choosing the keyframes for 5 videos. thus, 250 were created for the dataset each video has 05 different user summaries generated manually. and, as a way to pre-process the video dataset we extracted 04 fps from all videos. for the creation of the frame similarity graph, we use resnet50 and vgg16 (both pre-trained on imagenet) to extract frame descriptors. the cosine similarity was used to assess the similarity between two frame descriptors. and, we also set \(\delta _t = 32\) (i.e., 08 s with 04 fps) and \(\gamma = 0.05\), during the experiments. the parameter \(\delta _t\) plays a crucial role in enhancing the temporal threshold and restricting vertex connections to avoid the creation of edges that span across all frames of the video. this strategy is used since, if all frames were connected, temporal dependencies may be neglected. similarly, the parameter \(\gamma \) is employed to regulate the variance amplification in feature differences. its utilization helps control the level of distinction among features, ensuring a balanced representation of the underlying data. 4.2 evaluation metrics assessing frame quality in the context of video summarization poses a distinct challenge because of the many ways in which frames can be constructed while conveying similar meanings. these variations can arise from using different analyzes of resources from different informational aspects. although humans have an intuitive understanding of this process, abstract evaluation remains an open question without a specific framework. as a result, the conventional practice involves adapting similar metrics that have been stretched to accommodate the specific requirements of the video summary task. by re-purposing and customizing these metrics, researchers, and practitioners can assess the effectiveness and fidelity of summaries generated in video summarization, despite the inherent complexities and subjectivity involved in sentence evaluation [9, 17]. to compute the improvement of the frame selection, we will evaluate the obtained results following the same approach used by the authors of [9, 17]. they reported their results using metrics widely disseminated in the literature such as cusa, cuse [9, 17], and cov [17], defined by the eqs. 4–6, respectively, to evaluate the similarity between the frames generated by their summarization method and the gt results. $$\begin{aligned} \text{ cusa } = & {} \frac{m_a}{n_u} \end{aligned}$$ (4) $$\begin{aligned} \text{ cuse } = & {} \frac{\overline{m}_a}{n_u} \end{aligned}$$ (5) in which \(m_a\) denotes the number of matching keyframes generated from the automatic summary (as), \(\overline{m}_a\) represent non-matching keyframes from as, and \(n_u\) are the number of keyframes selected for the user to represent the user summary (u) to each video. $$\begin{aligned} \text{ cov } = \frac{\sum _{u \in us}|m(as,u)|}{\sum _{u \in us} |u|} \end{aligned}$$ (6) in which m(x, y) and |.| are the maximum matching between two sets of different elements x and y, and the cardinality of a set, respectively. while those two first metrics provide valuable insights, they often fail to measure the diversity displayed in user summaries as cov does. furthermore, the calculation of averages for each user’s measurements can introduce distortions and inaccuracies. specifically, the cusa, which is commonly employed to assess user opinions, fails to effectively capture the diversity of these opinions. to illustrate, consider two users, a and b, providing summaries for the same video. let the summary of user a be \(u_a = \{x, y\}\) while the summary of user b is \(u_{b} = \{m, n, o, p, q, r, s, t, u, v\}\), in which each character denotes a single frame of video. now suppose that three distinct methods generate summaries: \(as_1 = \{x, y\}\), \(as_2 = \{m, n, o, p, q, r, s, t, u, v\}\), and \(as_3 = \{x, m, n, o, p, q\}\). despite these summaries being completely different, they provide the same accuracy rate (i.e., cusa = 0.5). this highlights the limitations of cusa in accurately assessing divergence of opinion and the need for more comprehensive assessment metrics [9, 17]. unlike cusa, cov assesses the extent to which an automatic summary covers all user-generated summaries. this measure takes into account both the diversity of opinions expressed by users and the degree of agreement among them. specifically, the cusa measure calculates the average ratio between each user’s summary and an automatic summary, thus capturing the level of agreement between the two. in contrast, cov assesses the proportion of an automatic summary that aligns with all user summaries, providing a measure of overall covering. we use cov as the first metric to compute the effectiveness of the hietasumm. the reader should refer to [9, 17] for more information about those metrics. 4.3 quantitative analysis table 1 presents the hietasumm results. we used resnet50 and vgg16 to extract frame descriptors for the construction of the frame similarity graph. during the evaluation of the results, we also used resnet50 and vgg16 to extract frame descriptors but the cosine similarity was used to verify the agreement between the groundtruth and automatic summaries. we have also used color histograms (ch) during the assessment of the results. table 1 presents the average values of all metrics for the 50 videos belonging to the dataset. the results are presented for different levels of precision (between groundtruth and automatic summaries). it is possible to notice that the use of resnet50 presents a slight improvement compared to the results with vgg16 (under a greater precision in evaluation), and the vgg16 presented better results (under a lower preciseness in evaluation). moreover, it is also possible to observe the high values of cov and cusa achieved by hietasumm method, and even under a higher precision in evaluation, the proposed method still presents competitive results. table 1. performance of hietasumm method for different levels of precision in evaluation of video summaries. cusa, cuse, and cov values were multiplied by \(10^2\) to improve readability.full size table fig. 3. comparative example of hietasumm results compared with hsumm results and with the frames selected by the user 3 and user 5 (both selected 9 frames). the video summary generated by hietasumm contains 9 frames. full size image 4.4 qualitative analysis to provide a better understanding of the results obtained and their improvements, figs. 3 and 4 present samples of summaries generated by various approaches in the literature, including hsumm [17], vsumm1 [9], vsumm2 [9], visto [11] and open video summaries (referred to like ovsummary), and the groundtruth (gt) results, alongside those generated by the hietasumm method. this comparison enables the evaluation of time awareness, similarity with the gt results, and the rate of the frames selected by the hietasumm method and others. fig. 4. comparative example of hietasumm results compared with the results of vsumm1 [9], vsumm2 [9], visto [11], ovsummary and with the frames selected by the user 2 and user 3. full size image figure 3 shows the results generated by the hietasumm method along with hsumm [17] results, and the summaries generated by two users. each frame list created for each user encapsulates a distinct selection of frames, reflecting individual preferences and perspectives. employing cosine similarity, we can quantify the degree of similarity between the gt of the users and that generated for hietasumm method. however, it is essential to recognize that similarity is subjective and may vary among observers. factors such as the weighting of different frames, the level of granularity in frame selection, and the specific context of the video all influence perceived similarity. therefore, when evaluating the cosine similarity between two lists of frames, it is crucial to consider the subjective nature of the perception and the different perspectives that individuals bring to the comparison. the result obtained for the hsumm has a much higher number of frames than the others and, due to this, they present a large number of frames with high similarity. furthermore, hsumm results may not preserve chronological order. on the other hand, hietasumm method presents a fluid and coherent result. furthermore, the select keyframes are very similar to those frames in gt. for all keyframes selected by hietasumm method, only one frame does not have another directly correlated with those selected by the two users. but, in all cases, even with different keyframes selected by the users, the automatic summary generated by hietasumm method is very close to theirs (especially for users 3 and 5 shown in fig. 3). in addition, the unrelated keyframe preserves temporal order and, when we look at the three keyframes in which the map is present, it is possible to observe that a refinement process takes place to identify the correct highlighted region, starting from a global visualization to an analysis local that identifies the region in focus as the most important point of location on the map of the region presented in the video. figure 4 also presents some subjective characteristics for the keyframes selected by users 2 and 3. considering the number of frames selected, 15 and 17 respectively, it tends to suggest the existence of a larger number of scene modifications. this variation can cause the selection of a greater number of frames returned by automatic methods, but the increase in the number of scenes can cause frames to be repeated by automatic methods. in this way, the returned summaries have a great challenge of maintaining temporal coherence, but without two highly similar frames being selected without the presence of other events. with this difficulty in mind, ovsummary presents a series of repeated frames side by side. seen displays some repeated frames, but a reduced number of frames with more similar information. vsumm1 observes more scene modification and has some information that tends to be more similar. vsumm2 tends to keep the results without redundancy but without the presence of some scenes more relevant to the user. finally, the hierarchical approach used by hietasumm tends to reduce the redundancy of information with a lot of similarity. hietasumm results has a smaller number of keyframes, but these keyframes are more related to user summaries. moreover, keyframes selected by hietasumm method keep the temporal ordering and shows that the dynamic selection of summary size helps to better capture the changing scenes more smoothly. 5 conclusion this work proposes an unsupervised method for video summarization that considers changes in video content over time, named hierarchical time-aware summarizer– hietasumm. it uses pre-trained neural networks to generate video frame descriptions with a hierarchical graph-based clustering strategy. the proposed method explores a time-aware frame similarity graph to represent video content considering changes over time. moreover, a dynamic strategy for defining summary size is adopted. experimental results indicate that the proposed approach has great potential. specifically, it seems to enhance coherence among different video segments, reducing frame redundancy in the generated summaries, and enhancing the diversity of selected keyframes. future works may explore other strategies for selecting keyframes and different hierarchies. it might also be interesting to investigate the impact of different datasets with little scene modifications. following these future research directions, we can advance the video summary field and further refine the dynamic frame selection approach to provide more accurate, informative, and user-centric video summaries. references apostolidis, e., adamantidou, e., metsai, a.i., mezaris, v., patras, i.: video summarization using deep neural networks: a survey. proc. ieee 109(11), 1838–1863 (2021) article  google scholar  asha paul, m.k., kavitha, j., jansi rani, p.a.: key-frame extraction techniques: a review. recent patents comput. sci. 11(1), 3–16 (2018) article  google scholar  basavarajaiah, m., sharma, p.: survey of compressed domain video summarization techniques. acm comput. surv. 52(6), 1–29 (2019) article  google scholar  cardoso, l.v., guimaraes, s.j.f., patrocínio, z.k.g.: enhanced-memory transformer for coherent paragraph video captioning. in: 2021 ieee 33rd international conference on tools with artificial intelligence (ictai), pp. 836–840. ieee (2021) google scholar  cardoso, l.v., guimaraes, s.j.f., patrocinio, z.k.g.: exploring adaptive attention in memory transformer applied to coherent video paragraph captioning. in: 2022 ieee eighth international conference on multimedia big data (bigmm), pp. 37–44. ieee (2022) google scholar  cardoso, l.v., guimaraes, s.j.f., patrocinio junior, z.k.g.: hierarchical time-aware summarization with an adaptive transformer for video captioning. int. j. semant. comput. (2023) google scholar  cousty, j., najman, l.: incremental algorithm for hierarchical minimum spanning forests and saliency of watershed cuts. in: soille, p., pesaresi, m., ouzounis, g.k. (eds.) ismm 2011. lncs, vol. 6671, pp. 272–283. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-21569-8_24 chapter  math  google scholar  cousty, j., najman, l., kenmochi, y., guimarães, s.: hierarchical segmentations with graphs: quasi-flat zones, minimum spanning trees, and saliency maps. j. math. imaging vis. 60(4), 479–502 (2018) article  mathscinet  math  google scholar  de avila, s.e.f., lopes, a.p.b., da luz jr., a., de albuquerque araújo, a.: vsumm: a mechanism designed to produce static video summaries and a novel evaluation method. pattern recognit. lett. 32(1), 56–68 (2011) google scholar  ejaz, n., tariq, t.b., baik, s.w.: adaptive key frame extraction for video summarization using an aggregation mechanism. j. vis. commun. image represent. 23(7), 1031–1040 (2012) article  google scholar  furini, m., geraci, f., montangero, m., pellegrini, m.: visto: visual storyboard for web video browsing. in: proceedings of the 6th acm international conference on image and video retrieval, pp. 635–642 (2007) google scholar  guimarães, s., kenmochi, y., cousty, j., patrocinio, z., najman, l.: hierarchizing graph-based image segmentation algorithms relying on region dissimilarity: the case of the felzenszwalb-huttenlocher method. math. morphol.-theory appl. 2(1), 55–75 (2017) math  google scholar  lu, g., zhou, y., li, x., yan, p.: unsupervised, efficient and scalable key-frame selection for automatic summarization of surveillance videos. multimed. tools appl. 76, 6309–6331 (2017) article  google scholar  del molino, a.g., tan, c., lim, j.h., tan, a.h.: summarization of egocentric videos: a comprehensive survey. ieee trans. hum.-mach. syst. 47(1), 65–76 (2017) google scholar  panda, r., mithun, n.c., roy-chowdhury, a.k.: diversity-aware multi-video summarization. ieee trans. image process. 26(10), 4712–4724 (2017) article  mathscinet  google scholar  pandey, s., dwivedy, p., meena, s., potnis, a.: a survey on key frame extraction methods of a mpeg video. in: 2017 international conference on computing, communication and automation (iccca), pp. 1192–1196 (2017) google scholar  dos santos belo, l., caetano jr., c.a., patrocínio jr., z.k.g., guimarães, s.j.f.: summarizing video sequence using a graph-based hierarchical approach. neurocomputing 173, 1001–1016 (2016) google scholar  song, y., vallmitjana, j., stent, a., jaimes, a.: tvsum: summarizing web videos using titles. in: proceedings of the ieee conference on computer vision and pattern recognition (cvpr), pp. 5179–5187 (2015) google scholar  tiwari, v., bhatnagar, c.: a survey of recent work on video summarization: approaches and techniques. multimed. tools appl. 80(18), 27187–27221 (2021) article  google scholar  vivekraj, v., debashis, s., balasubramanian, r.: video skimming: taxonomy and comprehensive survey. acm comput. surv. 52(5) (2019) google scholar  download references acknowledgements the authors would like to thank conselho nacional de desenvolvimento científico e tecnológico cnpq (universal 407242/2021-0 and pq 306573/2022-9), and fundação de amparo à pesquisa do estado de minas gerais fapemig (grants ppm00006-18). this study was also financed in part by puc minas and by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations image and multimedia data science laboratory (imscience), pontifícia universidade católica de minas gerais (puc minas), belo horizonte, minas gerais, brazil leonardo vilela cardoso, gustavo oliveira rocha gomes, silvio jamil ferzoli guimarães & zenilton kleber gonçalves do patrocínio júnior authors leonardo vilela cardosoview author publications search author on:pubmed google scholar gustavo oliveira rocha gomesview author publications search author on:pubmed google scholar silvio jamil ferzoli guimarãesview author publications search author on:pubmed google scholar zenilton kleber gonçalves do patrocínio júniorview author publications search author on:pubmed google scholar corresponding author correspondence to leonardo vilela cardoso . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cardoso, l.v., gomes, g.o.r., guimarães, s.j.f., do patrocínio júnior, z.k.g. (2023). hierarchical time-aware approach for video summarization. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_18 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords video summarization hierarchical graph-based clustering unsupervised learning publish with us policies and ethics profiles silvio jamil ferzoli guimarães view author profile zenilton kleber gonçalves do patrocínio júnior view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature siamese network-based prioritization for enhanced multi-document summarization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper siamese network-based prioritization for enhanced multi-document summarization conference paper first online: 30 january 2025 pp 400–415 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) siamese network-based prioritization for enhanced multi-document summarization download book pdf download book epub klaifer garcia  orcid: orcid.org/0000-0002-9983-67349 & lilian berton  orcid: orcid.org/0000-0003-1397-60059  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 369 accesses abstract methods for document summarization hold significance in numerous applications, particularly in scenarios involving extensive content, such as news and social media monitoring. nevertheless, enhancing the task of summarizing multiple documents remains an area for improvement. systems that generate summaries from multiple documents must take into account additional challenges such as document redundancies and inconsistencies. on the other hand, as multi-document databases contain multiple descriptions of the same content, they offer benefits that go beyond the availability of more content and we believe that this benefit has not been explored in the literature. different authors may write different descriptions of the same event, some more objective, others more detailed, or even with different terms or writing styles. as a way to take advantage of this availability of different descriptions, we present a new approach to evaluating documents in order to identify those that can contribute most effectively to the generation of automatic summaries. for this, we employed a siamese network, which was trained using the rouge metric observed in individual documents. additionally, we demonstrated how to apply the outcomes of document evaluation to different summarization techniques. our experiments included a comparison with sota approaches (leadsum, textrank, pacsum, bertsum) in multiple datasets encompassing news, events, wikipedia texts, and scientific publications (multi-news, wcep, wikisum, arxiv, multi-xscience) and the results indicated our approach had a relevant improvement in the production of summaries, with statistical significance in the evaluation with wilcoxon rank sum for a confidence factor of 95%. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others multi-document summarizer chapter © 2018 multi-document summarization and opinion mining using stack decoder method and neural networks chapter © 2019 topic oriented multi-document summarization using lsa, syntactic and semantic features chapter © 2019 1 introduction automatic text summarization is the process that applies computational methods to produce a condensed version of one or more input documents while preserving the most relevant information [7]. particularly in the case of summarizing multiple documents, in addition to selecting the most relevant content, there is the challenge of avoiding redundancies and inconsistencies. news summarization is an example where an ongoing event can be reported by different channels, resulting in different descriptions with different updates. according to [22], many of the multi-document summarization models combine inputs as a flat sequence and do not consider relationships between documents. we believe the combination of input documents can help provide a more accurate description of the content, and our goal is to improve summarization by leveraging this variety of input documents. more specifically, we will apply a siamese network [13] to evaluate different input documents and identify those capable of producing better summaries. the difference between documents arises, for example, from the different volumes of information and objectivity. to emphasize these differences, we will consider different extractive summarization techniques. extractive summarization aims to produce a summary selecting a representative subset of the sentences or paragraphs from the source text without making any changes in them, opposed to abstractive summarization which aims to represent the input content with new words and sentences. despite the promising results with abstractive methods, it is not possible to guarantee that the summaries produced will be consistent with the facts [32], which can be limiting in some usage scenarios. the siamese neural network is a natural choice for our approach, as these networks are efficient in comparison learning. we use datasets composed of multiple input documents and at least one reference summary. every input document will be evaluated using the rouge metric and, after that, it will be reorganized into pairs indicating which one obtained the best evaluation. this arrangement is compatible with the siamese network application, which needs to learn to recognize the qualities that enable one document to be evaluated more highly than another. we used a training dataset and a separate test dataset, demonstrating generalization to unseen data. as an additional benefit, reorganizing documents into pairs produces a larger volume of training data. our main contributions are as follows: proposal of a document comparison method, based on siamese networks, applied to the automatic summarization of multiple documents; description of changes made to a variety of summarization methods to take advantage of document prioritization; experiments validated with statistical tests pointing to relevant differences in our results; comparison of some popular summarization techniques using the same datasets and testing methodology. our document prioritizing code is made publicly availablefootnote 1. the remainder of this article is organized as follows. we review the related work in sect. 2. section 3 presents the proposed model description. section 4 describes the datasets, the baseline methods, the changes made to the methods to include document prioritization and information about the testing methodology. in sect. 5 the results and discussions are presented. finally, sect. 6 brings the conclusion and future works. 2 related work cross-document relations should be taken into account while summarizing multiple documents. these relationships are typically used by summarizing techniques to reduce repetition and identify the most important details. applying clustering techniques, such as in [8, 33], is a way to reduce redundancy. graphs are often used to represent the relationship between pieces of text, as in [31]. this work used heterogeneous vertices to represent information of larger granularity, such as sentences, or smaller granularity, such as words. the vertices contained embeddings to represent information, which were updated with graph attention networks [29]. in [2] graphsum is presented, which analyzes text fragments to select sentences. text fragments are represented as nodes and a variation of the pagerank method is proposed to favor positive correlations. the encoder-decoder framework is a widely used method in abstractive summarization approaches, including multi-document scenarios. with this method, the encoder layer aims to create a representation of the input texts, and the decoder generates new sentences using this representation. the methodology has shown good results, although there are still some challenges, especially with long documents. to mitigate this difficulty, [15] proposed the use of graphs to guide abstractive summarization. a popular datasets for evaluating multi-document approaches is presented in [9], which also proposes a summarization technique. the suggested approach is a change to the point-generator summarizer [26], applying an attention layer that includes the importance of sentences. to determine the importance of sentences, a bidirectional lstm network is applied to produce embeddings. using the sentence representations as input, a similar process is applied to produce the document representation. this outputs are used to identify the most important sentences, which occurs with a variation of the mmr [3] technique. in the works retrieved, the main motivation for extracting cross-document relations is the reduction of redundancy and identification of the most important sentences. in contrast to these approaches, our goal is to evaluate entire documents. beyond just finding documents with the most pertinent information, our method seeks to identify those that are capable of producing the best summaries. a possible application, which will be demonstrated in our experiments, is the use of an additional layer in the evaluation of sentences, prioritizing the sentences that are present in the best documents. 3 model description the suggested approach seeks to determine which articles produce the best summaries, allowing documents to be ranked in order of importance for information extraction. the hypothesis is that better summaries can be produced when more sentences are extracted from the best documents. in the datasets we consider, which are presented in sect. 4, we observe that each sample is composed of a variable number of input documents. for simplicity, we suggest a technique for evaluating pairs of documents, which allows us to rank documents by applying multiple comparisons. in this way, the model needs to be able to determine which of the two input texts has the highest likelihood of yielding a good summary, and siamese networks are useful for learning by comparison [16]. in its simplest configuration, a siamese network is composed of two attribute extraction networks, one for each input, and a comparison head. this architecture is flexible enough to allow deep-learning techniques to be applied to the feature extraction layer, which is a configuration that has achieved state-of-the-art results in different tasks [1, 14, 30]. additionally, the two feature extraction networks have identical connection weights, ensuring a similar processing method for both input data. in our model, as depicted in fig. 1, each input document is processed by a neural network and produces a representation as document embeddings. these embeddings are used in a classifier to identify the best document. during training, the error observed in the classification is used to adjust the entire network, including the two networks that produce the document embeddings. with this approach, the reduction of errors in the classifier occurs through the optimization of document representations, which requires the network to learn how to identify the most useful attributes for evaluating document quality. fig. 1. document comparison using siamese network. two documents are evaluated by neural networks with the same connection values. the output is embeddings that are input to a classifier. full size image figure 2 shows the network in more detail up until the point where the document representation stage is completed. in this figure, the input documents are used to produce word embeddings that are processed by a convolutional neural network (cnn) [17]. cnn provide adjustable configurations that enable evaluating the precision increase as a function of computing cost in addition to the possibility of applying parallel processing. furthermore, this type of network has been successfully applied to a variety of problems such as news classification [10], medical image classification [5] and energy demand prediction [25]. our network is composed of one-dimensional filters, with multiple planes and different window sizes. this means that the filters consider each word embedding as a multidimensional point, that the filters are moved along the text, and that at each iteration several tokens are evaluated simultaneously, depending on the size of the window. filter results are compiled in a max-pooling layer. during the training, we included a dropout procedure, which in our experiments helped to increase generalization. at the end, a fully connected layer is included to performs dimensionality reduction and produce the document representation. the step after producing document embeddings, not represented in this figure, is the union of two document representations in a fully connected layer and a classifier activated with a sigmoid function. fig. 2. document embedding generation. tokens are converted to chains of word-embeddings, which are evaluated by convolutional filters. outputs are reduced with max-pooling and document embeddings are produced with a dense layer. full size image to train the network, we need labeled pairs of documents that we construct using the training data. we start by evaluating all documents with the rouge metric. using this evaluation, we choose two documents, \(d_1\) and \(d_2\), and assign the label 1 if the rouge evaluation of \(d_1\) is better than \(d_2\) and 0 otherwise. we take care during this process to maintain class balance and avoid duplicating documents in order to prevent overfitting. furthermore, as pointed out by [28], differences in the lengths can interfere with the evaluation with rouge, so we are using summaries of the input documents truncated with the same number of tokens. for simplicity, summaries are produced by extracting the initial tokens from documents and we leave other approaches for future work. to apply the trained network to order sets of documents, we need a strategy using pairwise comparisons. since each sample we considered was composed of a few input documents, for the sake of simplicity, we are applying the comparison to all possible pairs. in this way, considering all possible pairs of documents, we count the number of times that a document \(d_x\) obtained a score higher than \(d_y\), according to eq. 1. in this equation, d is the set of input documents, and \(s(d_i)\) is the output of the siamese network. $$\begin{aligned} score(d_x) = \mid \{ d_y \in d \mid d_y \ne d_x,\; s(d_x) > s(d_y) \} \mid \end{aligned}$$ (1) equation 1 is repeated for all documents allowing the identification of the best ones. to simplify integration with the other methods that will be presented in sect. 4, we normalize the document scores to sum to 1. 4 materials and methods this section presents the materials used in the experiments, including the datasets and baseline methods. it also describes the changes we made to include document prioritization and some implementation details. 4.1 datasets few datasets are available for multi-document summarization, mainly because producing reference summaries is labor-intensive. some of the datasets we use are composed of news, where it is possible to find different sources describing the same event. multi-news [9]: contains articles extracted from the news aggregation newserfootnote 2. this site contains human-written news summaries that include citations to external sources of information, gathering information from over 1,500 news channels. to compose this dataset, summaries are used as references and the news is cited as sources. there are a few versions of this data and we are using the pre-processed version without truncation. wcep [11]: contains documents extracted from the wikipedia current events portalfootnote 3. following wikipedia guidelines, summaries are short, approximately 35 words, written in the present tense and avoiding opinions and sensationalism. each summary averages only 1.2 cited sources, but this number has been incremented through documents extracted from the common crawl news datasetfootnote 4. we are using the version distributed by the authors, which was truncated at a maximum of 100 documents per sample. wikisum [18]: wikipedia was used to create this dataset by providing reference summaries. the task is to create the lead, or first part, of the wikipedia page using the references listed in the article and ten google search engine results. however, due to its enormous size, some works only use a part of this data. in this work we are using the partition of [20], which contains the first 40 paragraphs selected with a logistic regression method that was trained using rouge-2-recall from the comparison of paragraphs with the reference summary. since the input data is made up of separate paragraphs that are not identified with the document from which it was taken, we are treating each paragraph as a separate document. arxiv [4]: scientific publications taken from arxivfootnote 5 are included in this dataset. since many of the datasets now in use are composed of news articles or other shorter content, the objective was to develop a dataset composed of longer documents. the extracted content contains the abstracts, which are used as reference summaries, and the article sections which are used as input. since we are interested in multi-document datasets, we are considering each section as an input document. multi-xscience [21]: it is a dataset of scientific articles, where the objective is to produce the related works section using the abstract of the article itself and a collection of reference documents. the articles were extracted from arxiv, and related documents were identified by applying a set of heuristics in microsoft academic graph [27]. table 1 presents a summary of the evaluated datasets. the “train test val” corresponds to the number of samples in each data partition. “doc” is the average number of input documents in each sample. “doc/ref len” is the average number of tokens per document and reference, calculated using the nltkfootnote 6 library. table 1. dataset attributesfull size table 4.2 baseline and integration methods in this section, we describe the methods we used to compare the results and the procedure we applied to integrate with document prioritization. leadsum: since a part of the datasets is composed of news, we are considering extracting content from the beginning of articles. this strategy is justified by the fact that in journalistic content the first sentences are usually the most important [12]. in this way, all content is merged into a single large document, and the initial content is extracted until the length defined for the summary is reached. docscore+leadsum: to use document prioritization, we are reordering the input documents, moving the most relevant ones to the beginning. this way, if the most relevant document is larger than the size of the summary, all content will be extracted from this source. otherwise, content from other documents will be used following the same order of relevance. textrank [23]: is an unsupervised extractive summarization method based on graphs. in this method, the sentences are the vertices and the similarity between the sentences is used to define connection weights. using this graph, pagerank is applied to assign a score to each sentence, which is used to rank the sentences. summaries are produced by selecting the sentences with the highest score. in our implementation, the similarity between sentences was calculated with tf-idf (term frequency–inverse document frequency). docscore+textrank: the result of processing with textrank is a score value related to each sentence. to integrate with document prioritization, we are adding the document score to the paragraph score, resulting in a greater probability of selecting the paragraphs from the best documents. in this sum, we are applying a weighting factor, according to eq. 2, where \(t(s_{d, i})\) is the score obtained with textrank for sentence i of document d, s(d) is the document score obtained with our model and \(\alpha \) is an adjustable parameter. $$\begin{aligned} score(s_{d, i}) = t(s_{d, i}) + \alpha * s(d) \end{aligned}$$ (2) pacsum [34]: this is a summarization method that represents the input contents in the form of digraphs. the authors argue that when selecting sentences to construct summaries, there are those that contain the most relevant information and those that are complementary. a simple approach is proposed to identify the most relevant sentences considering their position of occurrence in the text. in the proposed digraph, the vertices are sentence embeddings, and the direction of the connections is defined by the order of occurrence of the paragraphs in the text. the most central sentences are selected, and centrality is defined with eq. 3. in this equation, i and j are positions of sentences in the input documents, \(s_i\) is a sentence, \(e_{i,j}\) is a measure of similarity between sentences \(s_i\) and \(s_j \) and \(\lambda _1\) and \(\lambda _2\) are adjustable parameters that determine the importance of the initial and final sentences, respectively. $$\begin{aligned} centrality(s_i) = \lambda _1 \sum _{j<i}e_{i,j} + \lambda _2 \sum _{j>i}e_{i,j} \end{aligned}$$ (3) docscore+pacsum: pacsum’s adjustable parameters are calibrated through a supervised process. in particular, the parameters \(\lambda _1\) and \(\lambda _2\) are configured to prioritize the initial and final sentences. this way, the method already has a mechanism in addition to sentence similarity that we will reuse to prioritize the most important documents. to achieve this, our approach was to reorder the input documents, keeping the most relevant ones first, which allows the parameters \(\lambda _1\) and \(\lambda _2\) to be adjusted more effectively. bertsum [19]: it is a supervised method for producing extractive summaries using the bert [6] network. the network receives the content to be summarized, which has been changed to include sentence separation and produce contextualized sentence embeddings. to select sentences, some approaches are evaluated, including a simple classifier, an inter-sentence transformer, and an lstm network. as an additional resource to avoid redundancy, summaries are produced avoiding inserting sentences with repeated trigrams. docscore+bertsum: it is challenging to use bertsum when there is a large amount of input data because it processes all input sentences at once. in the bert network, input documents are truncated into 512 tokens, which in our test case can lead to the deletion of complete documents. to minimize this limitation, we reorganize the input documents in a manner similar to our approach using pacsum, allowing the best documents to participate in the summarization. oracle: inserted as an upper bound for extractive methods. we are using a greedy approach similar to the one presented in [19]. starting with an empty set, in each iteration, we include a sentence, and this sentence must be the one that increases the rouge metric the most of all sentences in all documents. furthermore, repeated sentences should not be included. the process is repeated until the size defined for the summary is reached or there is no sentence that increases the summary score. oracle lead: an additional upper bound that exclusively takes document ordering into account. input documents were evaluated with rouge and ordered best first. using the data in this sequence we apply leadsum, which is the simplest classifier. 4.3 implementation details our goal when running the experiments is to find out whether adding document prioritization improved the quality of summaries. in order to achieve this, we conducted experiments on the datasets and evaluated how well the baseline and modified versions of the algorithms performed. all training of the document prioritization model was carried out with the training data and followed up with the validation data. we terminated training when no reductions in classification error were observed in the validation set for three consecutive iterations. to determine the model settings, we performed tests on the multi-news dataset, applying the same configuration to all datasets. in our evaluation, we only considered the quality of the results, disregarding the computational cost, although in our studies the larger network did not always produce better results. in this way, 128 convolutional kernels were used, and we produced document embeddings with a length of 32 values. the input data was converted to lowercase and we are using a vocabulary with the most frequent 10,000 lemmatized tokens. the word embeddings were generated with fasttext [24], which has a sub-word mechanism capable of avoiding out-of-vocabulary occurrences. we used the distribution trained with common crawl and allowed these representations to be optimized in the training. as mentioned in sect. 3, the input documents are truncated to the same size as expected for the summary. some of the baseline methods also require defining attributes. in the modified version of textrank, we include the \(\alpha \) variable that needs to be defined. in order to achieve this, we generate summaries for various values of \(\alpha \) using the validation data. we used the first 1,000 samples from the validation set to evaluate ten \(\alpha \) values between 0 and 1.0, using the best-performing configuration to produce the summaries in the test set. for pacsum we use the optimizer provided by the authors, which selects values using the grid-search method. in this process, in the same way as in the original article, we use the validation set. we also use the version of bert that was fine-tuned by the method’s authors. we configured bertsum to select sentences with a classifier in the output layer and maintained the default settings, which include the block-trigram strategy. we train the model with the training dataset, evaluating with the validation set every 1,000 training iterations. the training was stopped after three consecutive iterations with no decrease in error on the validation set. because it is a very large dataset, when training bertsum with wikisum, we reduced the validation set to the initial 3,200 samples. we also reduced the training set to 20% of the total, which is enough to complete training in the first epoch, since in our experiments training ended approximately after using 5% of the data. according to [28], evaluation using rouge can be influenced by the length of the summaries. this way, when generating summaries, enough sentences are chosen and the final sentence is truncated to reach the number of tokens specified by the summary, producing summaries that are precisely the same length across all methods. summary lengths were chosen through a literature search. this way we are using 300 tokens for multi-news, 40 for wcep, 60 for wikisum, 220 for arxiv, and 120 for multi-xscience. all experiments were repeated 10 times, always with different seeds of random numbers. using wilcoxon rank-sum statistics, we compared the statistical significance of the results from the prioritized version with the results from the original methods. for statistical tests, it is also necessary to repeat the tests with the original versions of the methods. since in the leadsum, pacsum, and bertsum methods, the order of documents is important, we are randomly changing the order of the input documents in each experiment. with this approach, we are comparing the random reordering of input documents with the reordering using our prioritization method. the original version of textrank does not use random numbers and is not affected by the order of documents, so we use another comparison approach. in its original implementation, the rouge metric is presented along with confidence intervals, which are calculated with bootstrap for a 95% confidence factor. in this way, we performed ten repetitions of our modified version and compared the median with this confidence interval. 5 results and discussion 5.1 classification scores in this section, the classification results using the siamese network are presented. the input data are the document pairs produced as indicated in sect. 3, so these results refer to binary classification to identify the best document. document pairs were produced for the training, testing, and validation data, and we are presenting the results with the testing data after the training is complete. the results are presented in table 2. table 2. siamese network classification resultsfull size table we conclude from the results that the suggested network helps identify the best documents. despite this, especially in the multi-xscience dataset, the accuracy was not very high. we therefore conducted experiments to determine the situations in which the model made the most mistakes. in particular, we were interested in identifying the relationship between the difference between rouge values and classification performance. since each training sample is composed of a pair of documents, we consider the difference in rouge evaluations between documents, reusing the evaluation applied to construct the training data. ordering the documents by this difference, we divide the test data into sets of 500 samples. thus, the first group corresponds to documents with similar evaluations and the last group contains documents with greater differences in quality. for each group produced, we calculate the f1-score metric and the mean difference of rouge. we calculated the spearman correlation between these values and the results are in the last column of table 2, where it is possible to verify a very strong correlation. this indicates that the proposed siamese network works better when the input documents have a large difference in the evaluation and not so well when the evaluations are very similar. these results are also represented in fig. 3. to simplify visualization, this figure represents a simplified version of the correlation experiment. in this experiment, we applied the same ordering procedure by rouge difference, with the change that the results were separated into just 10 groups. this figure shows that classifier performance is not particularly significant while the difference in evaluations with rouge is close to zero, but it advances to substantially greater performance when the difference with rouge increases. fig. 3. f1-score as a function of the difference in quality of documents evaluated with rouge. full size image 5.2 summarization scores we evaluate the results with the rouge 1, 2, and l metrics. in table 3 are the results with rouge-1, and correspond to the medians after 10 repetitions of the experiments. the results with the other metrics were equivalent and are available with our code. we are highlighting in bold the best results that have statistical significance in the evaluation with wilcoxon rank-sum for a confidence factor of 95%. at textrank we highlight the best results that exceed confidence margins calculated with bootstrap and a confidence factor of 95%. when the results are not different enough to achieve statistical significance, we highlight both values in bold. evaluating these results, we can see that document prioritization had a significant impact on the results, which is evident in oracle-lead. even applying the simplest summarizer, the perfect prioritization of input documents brought the best results. although without achieving perfect ordering, the proposed model for prioritizing documents allowed important improvements. in these results presented, the median of rouge-1 increments was 2.7. the prioritization of documents brought benefits in most results, and in the worst case, it obtained results equivalent to the non-prioritized version. equivalent results were obtained with textrank on the wcep and multi-xscience datasets. the original version of textrank is not influenced by document order and prioritization gives an advantage to the best documents. in the case of wcep, considering that it is the dataset with the largest volume of input documents, it might be necessary to increase the advantage of the best documents, which could be achieved by applying higher values of \(\alpha \) in eq. 2. the multi-xscience dataset had the smallest increase in the evaluation and was also the dataset with the lowest precision presented in sect. 5.1. table 3. multi-news: results f1-score (%)full size table 6 conclusions we present a document comparison approach using a siamese network that is useful for identifying the best documents in multi-document summarization tasks. as shown in the results, document prioritization can help increase the accuracy of summaries produced with different summarization techniques. the integration process with several methods from the literature was presented, and a possible future work is integration with other methods. since each experiment was conducted with identical tools and procedures, in addition to evaluating document prioritization, it is possible to compare different summarization techniques. in pairs, the original and prioritized versions were compared with statistical tests, indicating a significant difference in the results. notes 1.https://github.com/klaifer/docscore. 2.www.newser.com. 3.https://en.wikipedia.org/wiki/wikipedia:how_the_current_events_page_works. 4.https://commoncrawl.org/2016/10/news-dataset-available/. 5.https://arxiv.org/. 6.www.nltk.org. references bandara, w.g.c., patel, v.m.: a transformer-based siamese network for change detection. in: igarss 2022 2022 ieee international geoscience and remote sensing symposium, pp. 207–210 (2022) google scholar  baralis, e., cagliero, l., mahoto, n., fiori, a.: graphsum: discovering correlations among multiple terms for graph-based summarization. inf. sci. 249, 96–109 (2013) article  mathscinet  google scholar  carbonell, j., goldstein, j.: the use of mmr, diversity-based reranking for reordering documents and producing summaries. in: proceedings of the 21st annual international acm sigir conference on research and development in information retrieval, sigir 1998, new york, ny, usa, pp. 335–336. association for computing machinery (1998) google scholar  cohan, a., et al.: a discourse-aware attention model for abstractive summarization of long documents. in: walker, m., ji, h., stent, a. (eds.) proceedings of the 2018 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 2 (short papers), new orleans, louisiana, pp. 615–621. association for computational linguistics, june 2018 google scholar  de castro santos, m.a., berton, l.: an enhanced framework for overcoming pitfalls and enabling model interpretation in pneumonia and covid-19 classification. ieee access 11, 115330–115347 (2023) article  math  google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: burstein, j., doran, c., solorio, t. (eds.) proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), minneapolis, minnesota, pp. 4171–4186. association for computational linguistics, june 2019 google scholar  el-kassas, w.s., salama, c.r., rafea, a.a., mohamed, h.k.: automatic text summarization: a comprehensive survey. expert syst. appl. 165, 113679 (2021) article  google scholar  ernst, o., et al.: proposition-level clustering for multi-document summarization. arxiv e-prints arxiv:2112.08770, december 2021 fabbri, a.r., li, i., she, t., li, s., radev, d.r.: multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model. in: acl 2019 57th annual meeting of the association for computational linguistics, proceedings of the conference, pp. 1074–1084 (2020) google scholar  garcia, k., shiguihara, p., berton, l.: breaking news: unveiling a new dataset for portuguese news classification and comparative analysis of approaches. plos one 19(1), 1–15 (2024) article  math  google scholar  ghalandari, d.g., hokamp, c., pham, n.t., glover, j., ifrim, g.: a large-scale multi-document summarization dataset from the wikipedia current events portal. arxiv preprint arxiv:2005.10070 (2020) grenander, m., dong, y., cheung, j.c.k., louis, a.: countering the effects of lead bias in news summarization via multi-stage training and auxiliary losses. in: inui, k., jiang, j., ng, v., wan, x. (eds.) proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), hong kong, china, pp. 6019–6024. association for computational linguistics, november 2019 google scholar  koch, g., zemel, r., salakhutdinov, r., et al.: siamese neural networks for one-shot image recognition. in: icml deep learning workshop, lille, vol. 2 (2015) google scholar  lan, m., zhang, j., he, f., zhang, l.: siamese network with interactive transformer for video object segmentation. in: proceedings of the aaai conference on artificial intelligence, vol. 36, no. 2, pp. 1228–1236 (2022) google scholar  li, w., xiao, x., liu, j., wu, h., wang, h., du, j.: leveraging graph to improve abstractive multi-document summarization. in: jurafsky, d., chai, j., schluter, n., tetreault, j. (eds.) proceedings of the 58th annual meeting of the association for computational linguistics, pp. 6232–6243. association for computational linguistics, online, july 2020 google scholar  li, y., chen, c.l.p., zhang, t.: a survey on siamese network: methodologies, applications, and opportunities. ieee trans. artif. intell. 3(6), 994–1014 (2022) article  math  google scholar  li, z., liu, f., yang, w., peng, s., zhou, j.: a survey of convolutional neural networks: analysis, applications, and prospects. ieee trans. neural netw. learn. syst. 33(12), 6999–7019 (2022) article  mathscinet  math  google scholar  liu, p.j., et al.: generating wikipedia by summarizing long sequences. arxiv e-prints arxiv:1801.10198, january 2018 liu, y.: fine-tune bert for extractive summarization. arxiv e-prints arxiv:1903.10318, march 2019 liu, y., lapata, m.: hierarchical transformers for multi-document summarization. in: korhonen, a., traum, d., màrquez, l. (eds.) proceedings of the 57th annual meeting of the association for computational linguistics, florence, italy, pp. 5070–5081. association for computational linguistics, july 2019 google scholar  lu, y., dong, y., charlin, l.: multi-xscience: a large-scale dataset for extreme multi-document summarization of scientific articles. in: webber, b., cohn, t., he, y., liu, y. (eds.) proceedings of the 2020 conference on empirical methods in natural language processing (emnlp), pp. 8068–8074. association for computational linguistics, online, november 2020 google scholar  ma, c., zhang, w.e., guo, m., wang, h., sheng, q.z.: multi-document summarization via deep learning techniques: a survey. acm comput. surv. 55(5) (2022) google scholar  mihalcea, r.: graph-based ranking algorithms for sentence extraction, applied to text summarization. in: proceedings of the acl 2004 on interactive poster and demonstration sessions. acldemo 2004, usa, p. 20–es. association for computational linguistics (2004) google scholar  mikolov, t., grave, e., bojanowski, p., puhrsch, c., joulin, a.: advances in pre-training distributed word representations. in: proceedings of the international conference on language resources and evaluation (lrec 2018) (2018) google scholar  rick, r., berton, l.: energy forecasting model based on cnn-lstm-ae for many time series with unequal lengths. eng. appl. artif. intell. 113, 104998 (2022) article  math  google scholar  see, a., liu, p.j., manning, c.d.: get to the point: summarization with pointer-generator networks. in: barzilay, r., kan, m.y. (eds.) proceedings of the 55th annual meeting of the association for computational linguistics (volume 1: long papers), vancouver, canada, pp. 1073–1083. association for computational linguistics, july 2017 google scholar  sinha, a., et al.: an overview of microsoft academic service (mas) and applications. in: proceedings of the 24th international conference on world wide web, www 2015 companion, new york, ny, usa, pp. 243–246. association for computing machinery (2015) google scholar  sun, s., shapira, o., dagan, i., nenkova, a.: how to compare summarizers without target length? pitfalls, solutions and re-examination of the neural summarization literature. in: bosselut, a., et al. (eds.) proceedings of the workshop on methods for optimizing and evaluating neural language generation, minneapolis, minnesota, pp. 21–29. association for computational linguistics, june 2019 google scholar  veličković, p., cucurull, g., casanova, a., romero, a., liò, p., bengio, y.: graph attention networks. arxiv e-prints arxiv:1710.10903, october 2017 viji, d., revathy, s.: a hybrid approach of weighted fine-tuned bert extraction with deep siamese bi lstm model for semantic text similarity identification. multimedia tools appl. 81(5), 6131–6157 (2022) article  math  google scholar  wang, d., liu, p., zheng, y., qiu, x., huang, x.: heterogeneous graph neural networks for extractive document summarization. in: jurafsky, d., chai, j., schluter, n., tetreault, j. (eds.) proceedings of the 58th annual meeting of the association for computational linguistics, pp. 6209–6219. association for computational linguistics, online, july 2020 google scholar  xu, s., zhang, x., wu, y., wei, f., zhou, m.: unsupervised extractive summarization by pre-training hierarchical transformers. in: proceedings of the 2020 conference on empirical methods in natural language processing: findings, pp. 1784–1795. association for computational linguistics, online, november 2020 google scholar  zhao, j., et al.: summpip: unsupervised multi-document summarization with sentence graph compression. in: proceedings of the 43rd international acm sigir conference on research and development in information retrieval, sigir 2020, new york, ny, usa, pp. 1949–1952. association for computing machinery (2020) google scholar  zheng, h., lapata, m.: sentence centrality revisited for unsupervised summarization. in: proceedings of the 57th annual meeting of the association for computational linguistics, florence, italy, pp. 6236–6247. association for computational linguistics, july 2019 google scholar  download references acknowledgments this work has been supported by the brazilian research agencies coordenação de aperfeiçoamento de pessoal de nível superior (capes) and conselho nacional de desenvolvimento científico e tecnológico (cnpq). author information authors and affiliations universidade federal de sao paulo, avenida cesare mansueto giulio lattes, nř 1201, eugênio de mello, sp, brazil klaifer garcia & lilian berton authors klaifer garciaview author publications search author on:pubmed google scholar lilian bertonview author publications search author on:pubmed google scholar corresponding author correspondence to lilian berton . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper garcia, k., berton, l. (2025). siamese network-based prioritization for enhanced multi-document summarization. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_28 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_28 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multi-document summarization siamese network nlp publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a knowledge engineering-based approach to detect gaming the system in novice programmers | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper a knowledge engineering-based approach to detect gaming the system in novice programmers conference paper first online: 30 january 2025 pp 18–33 cite this conference paper download book pdf download book epub intelligent systems (bracis 2024) a knowledge engineering-based approach to detect gaming the system in novice programmers download book pdf download book epub hemilis joyse barbosa rocha  orcid: orcid.org/0000-0001-5422-92719,11, evandro de barros costa  orcid: orcid.org/0000-0003-4663-871510 & patricia cabral de azevedo restelli tedesco  orcid: orcid.org/0000-0001-9450-92199  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 372 accesses 1 citation abstract in recent years, the behavior called “gaming the system” has received increasing attention, especially in online learning environments, where students seek to advance in the system without actually engaging in genuine learning processes. although previous studies have made progress in identifying and understanding this behavior, making use of machine learning models and knowledge engineering, previously validated detectors have not fully captured the gaming attitudes observed in our novice programming learning context. this challenge may be related to the transferability of motivations for gaming between different contexts, as well as to detection. therefore, this article adopts an approach focused on the exploration and specific detection of “gaming the system” attitudes manifested by novice programmers in interaction with a learning environment. we propose an approach to detecting this behavior, combining knowledge engineering and machine learning algorithms. we describe the procedures for generating our database, followed by the elaboration of a model based on knowledge engineering and the development of a model based on machine learning. as a result, we identified two new game attitudes and incorporated them into the knowledge-based model, using cognitive task analysis to understand how experts encode these attitudes. download conference paper pdf similar content being viewed by others an approach for detecting gaming the system behavior in programming problem-solving chapter © 2023 automated detection and analysis of gaming the system in novice programmers chapter © 2024 predicting performance in gamified cognitive executive functioning tasks for neurodiverse children using machine learning chapter © 2025 1 introduction in the literature on artificial intelligence in education, certain behaviors have received significant attention within the context of students’ interactions with online learning environments. recent research has identified two main types of behavior observed in students during these interactions: learning-oriented behavior and non-learning-oriented behavior. non-learning-oriented behavior often involves students who do not effectively utilize the help resources available in virtual learning environments [7], resulting in an unproductive search for assistance [2]. this type of behavior is considered a form of “gaming the system” [12], defined as “trying to succeed in an educational environment by exploiting the properties of the system rather than learning the material and trying to use that knowledge to respond correctly”. furthermore, this behavior can lead students to produce rushed or incomplete work [15] and has been associated with lower academic performance [6], and can even result in almost zero learning [29]. understanding these student behaviors in virtual learning environments is crucial for developing effective support and intervention strategies to improve student engagement and academic performance. more specifically, an important challenge is to seek an effective identification mechanism for gaming the system behavior, an issue that has been addressed in various proposals in the literature in different proposals but remains largely unexplored in the programming domain. given the mentioned context, previous research has predominantly focused on two different approaches for modeling “gaming the system” behaviors. the first approach, known as knowledge engineering, involves the development of explicit rules by domain experts to identify“gaming” behaviors [17, 21, 23] and machine learning, where the model designer first creates a set of features and then a supervised learning algorithm is used to select features to predict labels of human-coded “gaming” [12, 27]. additionally, a recent study compared three gaming detectors across multiple systems: a knowledge engineering model and a machine-learned model [20]. the knowledge engineering model was developed using cognitive task analysis in a tutor in the algebra domain, identifying 13 student behavior patterns. comparisons focused on the predictive performance of the models on new data sets and their interpretability. the results indicated that the knowledge engineering model achieved better generalization and interpretability compared to the machine-learned model. however, this model was developed by studying students during interaction with an algebra tutor with a specific type of task and is not generalizable to other contexts as demonstrated by [13]. despite the important contribution of previous work [13] in considering some types of tasks, the diversity of tasks is limited and does not consider other domains, such as computer programming. this behavior has been particularly observed in the area of programming, although it is a relatively recent phenomenon and little documented in the literature [25]. despite this, some recent studies have indicated that students taking introductory programming courses have reported high frequencies of this behavior [24, 28]. however, although previous studies have made progress in identifying and understanding this behavior, making use of machine learning and knowledge engineering models, previously validated detectors have not fully captured the “gaming the system” attitudes observed in our programming learning context for starters. this challenge may be related to the transferability of gaming motivations between different contexts, as well as detection. this limitation in considering domain characteristics may restrict the generalization of “gaming the system” detectors to new contexts. with the growing demand for effective and engaging learning systems, the generalizability, interpretability, and development cost of “gaming the system” detectors are becoming increasingly important. therefore, in this article we adopt an approach focused on the exploration and specific detection of “gaming the system” attitudes expressed by novice programmers interacting with a learning environment. we propose a hybrid approach to detect this behavior, combining knowledge engineering and machine learning algorithms. furthermore, aiming to contribute to a generalization, in this study, we used this approach to detect the behavior of beginning programmers in a specific demographic context: a rural school in the northeast of brazil, a region historically marked by high social inequality [19]. the research aimed to understand the attitudes of beginner programmers who exhibit “gaming the system” behavior and detecting such behavior. as a result, in addition to the hybrid detection approach, we identified two new game attitudes and incorporated them into the knowledge-based model, using cognitive task analysis to understand how experts encode these attitudes. 2 background and related work research into detecting“gaming the system” behavior generally focuses on identifying and characterizing such behavior. however, it is crucial to distinguish between “gaming the system” and cheating. while abusing the system involves taking advantage of flaws or loopholes in the system, cheating implies directly violating the established rules [6]. it is therefore essential not to simplistically label students who manifest these behaviors as “gamers” or “cheaters”. they may have complex motivations that lead them to act in an unproductive or undesirable way [6]. to mitigate this type of behavior, many authors have proposed simply redesigning the tutor to eliminate the behavior [8]. however, if the behavior is symptomatic and involves other aspects and unknown motivations, such a solution could mask the problem rather than eliminate it. therefore, when designing systems that meet the objectives, attitudes, and behaviors of students in order to positively impact learning, it is necessary to investigate the motivations and characterize the profile or profiles of students who manifest the behavior of “gaming the system”. in this sense, some authors have carried out studies seeking to understand the motivations and beliefs of students who manifest this behavior. in a study conducted by [27], the authors investigated “gaming the system” behavior through a 32-question survey. they found that students who manifested this behavior had distinct characteristics: they were less likely to own a computer at home, had lower confidence in their mathematical abilities, frequently avoided doing math homework, showed fewer problems concentrating on the computer, expressed frustration when faced with difficult tasks, preferred learning with a computer over other ways, sought to solve problems quickly, had less interest in math classes, preferred facts and data to concepts and ideas, perceived hints as more useful in understanding similar problems, prioritized getting through as many items as possible and had the goal of learning new things. these results highlight the association between “gaming the system” behavior and various student attitudes and beliefs about learning. in previous studies, it has been observed that students may adopt the behavior of “gaming the system” due to the belief that they cannot succeed otherwise [4, 6]. in addition, the relationship between this behavior and performance goals, to the detriment of learning goals [6, 16]. however, some motivations may not be transferable from one country to another, as well as help-seeking behavior. some research has shown that models of effective help-seeking do not apply uniformly across different countries, such as costa rica, the philippines, and the usa [18]. this finding is in line with the results of previous studies in various educational contexts, highlighting the influence of racial and gender interactions on help-seeking behaviors, which in turn can influence subsequent performance patterns [22]. in addition, several researchers have investigated how demographic variables can influence help-seeking behavior [14]. on the other hand, the authors studied students in a flipped cs1 course, examining survey data to identify factors that contribute to novice programmers’ engagement in three inappropriate behaviors, two of which are related to “gaming the system” behavior [28]. the authors identified that students with a constructive mindset are correlated with less “gaming the system” behavior, while those who perceive the computing course as fun have an increased risk of engaging in such behavior. in addition, discomfort with speaking and listening to english was also associated with a greater propensity to “gaming the system” [28]. these findings highlight the importance of considering not only academic aspects but also psychological and socio-emotional factors in understanding “gaming the system” behavior. however, this study found no correlation between gambling attitudes and students’ prior knowledge, contradicting previous reports in the literature [6]. 2.1 “gaming the system” behavior detectors one of the most recent works on variable discovery for detection proposes a latent variable model for detecting “gaming the system” behavior based on item response theory (irt-gd) in the algebra domain [13]. the model estimates a latent “gaming” tendency for each student, taking into account contextual factors. these factors include the frequency with which students game in less common gaming contexts compared to more common contexts, as observed by students’ action patterns and the predictability of human labels. the irt-gd is based on a previously validated knowledge engineering game detector (ke-gd), which focuses on students’ action characteristics and the predictability of human labels. however, although the study does not explicitly explain the variables considered to implement the detector, the authors claim to have used a previously validated knowledge engineering gaming detector containing 13 interpretable patterns that model behavior [20, 21]. 3 research methods in this section, we describe the materials and methods used to conduct the research presented in this article. the construction of our “gaming the system” behavior detection model follows a hybrid approach, combining knowledge engineering and machine learning algorithms. to this end, we detail the experiments carried out to generate our database, followed by the elaboration of the model based on knowledge engineering, and, finally, we explain the method used to develop the model based on machine learning. 3.1 experiment the experiment carried out in this research took place in the first semester of 2023 and involved students from two second-year technical high school classes located in the interior of northeast brazil. these students were new to programming and had recently completed an introductory course on the subject. they were currently enrolled in a second course, focused on java programming (cs2). cs2 students were chosen because they were new to programming and had some familiarity with basic concepts. the study was carried out at this time, as the students were about to start learning object-oriented programming in the cs2 course. the experiment was carried out in four sessions, with two classes each, totaling 67 participating students, 29 from the afternoon class and 38 from the morning class, who participated in all problem-solving sessions. of these participants, 35 were female and 32 were male, aged between 16 and 21 years. of these, 61% were white and 39% were of african descent. materials. the adafootnote 1 environment was created as a learning tool for introductory programming courses, following a conceptual structure composed of elements such as classes, sessions, problems, alternatives, attempts, tips, code, and subjects. each lesson is linked to student and problem-solving sessions. the problems include four hints and alternatives, and students can make multiple attempts to solve them. features like the problem solver allow students to submit partial or complete solutions, while hints are offered at different levels, from abstract suggestions to direct answers, to guide students through each stage of the problem. students have the freedom to request hints at any time while solving the problem. procedure. the problem-solving sessions were held in person in two computer labs, with one student per computer. each laboratory was organized into five vertical rows of eight computers. in addition to the students, three researchers were present in each laboratory. the experiment took place during computer programming classes (sessions) in three stages: pre-test, intervention, and post-test. in the pre-test, the first session was subdivided into 30 min for presentation and 70 min for problem-solving. during this period, the objectives of the study were explained to the students, forms of consent for participation in the experiment were distributed, following the guidelines of the ethics committee, and a comprehensive explanation of the environment’s functionalities was provided. in addition, students used ada to register and respond to the socioeconomic questionnaire. after the introductory session, participants carried out two more problem-solving sessions (100 min) using the ada environment, being observed by three researchers. through these observations, it was possible to record occurrences of “game the system” behavior in a similar way to some previous works [3, 5, 21]. therefore, field observations were carried out so that observers observed each student’s behavior several times during each class period. each observer coded the frequency of “gaming the system” behavior and its nature. table 1. gaming the system attitudesfull size table 3.2 model development based on the findings of the previous model [21], a new model for coding system abuse behavior was designed, and adapted to the type of task provided by the ada environment. in this model, the initial approach is derived from the previous proposal, which groups the student’s actions, during interaction with the virtual learning environment, into sequences called text replays, with up to five actions in each. the authors, based on these text replays, developed patterns for sequences of student actions characterized as“gaming the system”. thus, if a pattern is identified in the analysis of the data set, all actions involved are labeled as abuse. to develop the model, the authors initially created a list of coders’ interpretations of student behavior, whether related to gaming behavior or not, and these interpretations were used to create the patterns (fig. 1). fig. 1. confusion matrix: decision tree, neural network (mlp) and k-nearest neighbors (knn) full size image the development of our knowledge-based model was conducted in four stages: analysis of the previous model, study of attitudes, investigation of specialized coding, and elaboration of the new model. in the first step, the 13 standards presented in the previous model [21] were categorized into two groups: compatible and incompatible with the ada environment. standards considered compatible were maintained, while those considered incompatible were removed or adjusted. next, an analysis of the coverage of the five attitudes related to the manifestation of “gaming the system” behavior was carried out, as presented in table 1. we observed that attitudes 4 and 5 were not addressed in the standards of the previous model. therefore, similar to a previous study [21], we chose to elicit experts’ knowledge through an active participation approach [11]. applying the active participation approach [11], knowledge elicitation sessions were conducted with experts. in the first session, we conducted an initial interaction in which the elicitor observed and took separate notes for each expert as they coded and thought aloud [26] about game behavior using text replays. in the second session, the elicitor coded some excerpts while thinking aloud, allowing each expert to comment and correct their thought process. this method provided the elicitor with a deeper understanding of the process, resulting in the elaboration of the first version of the rules. next, a new knowledge elicitation session was conducted, following each expert as they coded and expressed their thoughts aloud. subsequently, an evaluation session of the rules was held by experts, we made adjustments as necessary and, finally, we presented the final version in a final session. it is worth noting that the preliminary model was substantially influenced by previous work [21]. the final version of the model will be detailed and discussed in the next session. after developing our knowledge-based model, we move on to developing the machine learning-based model. using the latter, we proceeded to annotate the data generated during the experiment. subsequently, we analyzed three stages: pre-processing, classification, and evaluation of the data. in data preprocessing, we perform two crucial tasks: data cleaning and data transformation. in data cleaning, we remove missing data to avoid inconsistencies and errors during analysis and modeling. in data transformation, we apply normalization and standardization. in normalization we adjust attribute values to a common scale, generally between 0 and 1, allowing each attribute to contribute equally to the final result. in standardization, we adjust the data so that it has a mean of zero and a standard deviation of one, useful for standardizing deviations from the mean. in the classification step, we use a technique called 15-fold cross-validation. this method involves dividing the data into 15 parts and in each iteration training the model on 14 parts and testing it on the remaining part. by repeating this process 15 times, we increase the algorithm’s generalization capacity, resulting in more accurate predictions [9]. then, we applied three algorithms: decision tree, k-nearest neighbors (knn) and neural networks using the multi-layer perceptron (mlp) [10]. this selection was based on previous studies [3]. the implementation was carried out with the scikit-learnfootnote 2 library, configuring the decision tree with the cart algorithm, the knn with 3 neighbors and the neural network with mlp. in the evaluation stage, we analyze the performance of each algorithm on the training and testing subsets. we used six metrics: area under the roc curve (auc), accuracy, precision, recall, f1 and kappa , with a significance level of \(p-value < 0.05\). auc evaluates the model’s ability to perform accurate classifications. precision, recall, and f1 provide a complete view of performance, while kappa checks whether the model determines the correct sequences for reasons that are not based on chance. 4 results and discussion table 2. gaming the system interpretationsfull size table table 3. gaming the system patternsfull size table 4.1 model based on knowledge engineering similar to the model proposed in the previous work [21], our cognitive study of identifying gaming behavior can be subdivided into two parts: interpretation of student actions and identification of game patterns. although these two parts are normally performed simultaneously by the specialist, the cognitive model is capable of executing them separately without compromising the integrity of the process. interpreting student actions. according to the functionalities presented in the ada environment, as detailed in the 3.1 section, students can request help at various levels and submit their solutions to a problem in two stages: partial solution and full solution. the partial solution involves a multiple-choice activity, while the full solution is presented in code form. to create text replays, the specialist begins by observing the pauses between each student’s actions in the environment, building a probable interpretation of the student’s mental process. in table 2, we offer a list and description of the elements of student behaviors identified during the knowledge elicitation process with the expert, in line with the findings of previous work [21]. the length of pauses before asking for help can indicate whether the student took time to think about the problem before seeking assistance. likewise, the length of the pause after receiving help indicates whether the student took time to read the help message, whether they were scanning the help message for specific information, or whether they were expressing the att2 or att3 attitudes in table 1. the pauses before trying each alternative may indicate whether the student took time to reflect before choosing another option, or whether he was expressing the att1 attitude from table 1. when a partial submission is considered incorrect, the presence of a pause after this action suggests that the student is reflecting on the error made. given that this type of action often involves expected errors, a prolonged pause before the mistake usually indicates a sincere, albeit unsuccessful, attempt to resolve the problem. in contrast, a short pause often indicates that students are trying to guess the answer by submitting several partial solutions to the problem. additionally, when a student makes some mistakes by submitting incorrect alternatives, he or she may exit the problem, quickly select another problem, and then return to the original problem to submit the correct answer, which may manifest the att4 attitude. as for the total solution, if the student makes an empty submission or presents useless code for the proposed problem, it is an indication that he may be expressing the att5 attitude. identifying “gaming the system” patterns. when identifying potential student actions when interacting with the environment, whether related to the manifestation of “gaming the system” attitudes or not, it is crucial to reference table 2. in this table, interpretations of the student’s individual actions are described, but it is essential to understand that analyzing each action in isolation is not enough to determine whether the student is cheating the system. therefore, it is necessary to find patterns of actions that serve as robust evidence of such behavior. in table 3, we present the patterns identified during the knowledge discovery process, in collaboration with experts in the field. in the first column of table 3, we assign a tag to each identified pattern. in the second column, we describe the rules derived based on the interpretations of table 2. finally, in the third column, we relate each rule to a possible student attitude during interaction with the environment. each pattern can be represented by the eq. 1, where a pattern is composed of a set of up to 5 text replays (r) associated with up to five types of “gaming the system” attitudes (a). $$\begin{aligned} p & = & (\bigcup _{j \le 5} (r), a_{1..5}) \end{aligned}$$ (1) after identifying some possible student actions when interacting with the environment, whether related to the manifestation of “gaming the system” attitudes or not, as described in table 2, the specialist interprets the combination of these actions to judge whether the student is manipulating the system. this is because analyzing each action in isolation is not enough to encode gaming behavior. therefore, it is essential to identify patterns of actions that act as significant evidence that the student is “gaming the system”. in conclusion, our knowledge elicitation process encompassed literature review, dataset analysis, and expert observations. as a result of this process, we identified eight essential patterns for encoding a clip as a game instance in the programming learning system, as presented in table 3. in the first column of the table, we assign a unique tag to each pattern. the second column highlights the rules derived based on the interpretations of table 2 from our database and interactions with experts. in the third column, we associate each rule with a specific student attitude during interaction with the environment. notably, some clips were linked to more than one pattern. table 4. results of the algorithm comparisonfull size table 4.2 machine learning based model after the development of the knowledge-based model was completed, we proceeded with the development of the machine-learning model. to do this, we used the patterns presented in table 3 to label our dataset in a two-step process, led by two expert coders. in the first stage, the coders performed the task independently, resulting in lower reliability between them, with a cohen’s kappa coefficient of 0.72 to distinguish between the classes class0 and class1. in a second round of labeling, coders identified instances of disagreement, participated in discussions to align their interpretations of the behaviors, and subsequently conducted supplemental labeling. this collaborative effort resulted in a notably higher cohen’s kappa coefficient of 0.93 for distinguishing between gaming and non-gaming behaviors in the second set of text replays. as a result, we obtained our dataset divided according to table 5. using the already labeled data, we applied three machine learning algorithms (cart, knn, and mlp) and evaluated their performance through confusion matrix analysis, along with previously mentioned metrics such as accuracy (acu), precision (pre), recall (rec), f1 score (fme), kappa coefficient (kap) and area under the roc curve (roc). additionally, we will conduct a statistical analysis to assess the significance of each algorithm’s performance. this procedure will provide us with a deeper understanding of the effectiveness of each algorithm in detecting behavior, considering the combination of experimental data. table 5. dataset distribution by classfull size table table 4 shows the results of the comparison of algorithms, considering various performance metrics during the training and testing phases. the algorithms compared are knn, cart, and mlp. during the training phase, it can be seen that the cart algorithm obtained perfect results (value of 1.00) for all the metrics evaluated, indicating excellent and consistent performance. on the other hand, knn and mlp showed slightly lower values for all metrics, with mlp showing the lowest values compared to the other two algorithms. in the test phase, the results were different. knn showed a significant drop in all metrics compared to the training phase, suggesting a possible overfitting tendency during training. cart, although it showed a slight decrease in some metrics, maintained a generally high performance close to the training results. mlp also saw a drop in its performance metrics but managed to maintain higher values compared to knn. this analysis suggests that the cart algorithm may have a superior generalization capacity compared to the other two algorithms, maintaining consistent performance during both training and testing. we perform the wilcoxon test to compare the algorithms applied to the class. the significance level used was p-value=0.05. several performance metrics were compared between the cart, knn, and mlp algorithms during the training stage. for all metrics evaluated (acu, pre, rec, fme, kap, and roc), the p-value was consistently 0.00006 when comparing cart to knn and cart to mlp, indicating a statistically significant difference. in the testing stage, the results show variation in p values between different combinations of algorithms and metrics. the comparison between cart and knn resulted in p=0.00006 for all metrics, suggesting a significant difference. comparing mlp with knn, p values ranged from 0.0067 to 0.0102, depending on the metric, indicating that there was statistical significance even in this comparison. analyzing the data from the confusion matrix of the test data, for the decision tree, 218 true negatives, 22 false negatives, 31 false positives, and 140 true positives were recorded. for knn, there were 197 true negatives, 72 false negatives, 52 false positives, and 90 true positives. for the neural network, only 196 true negatives, 68 false negatives, 53 false positives, and 94 true positives were identified. these results provide a detailed overview of each algorithm’s performance in classifying data. overall, although the neural network showed a high number of true positives, this came at the cost of a large number of false positives, indicating a lack of accuracy in its predictions. meanwhile, the decision tree and knn demonstrated a more balanced ability to correctly classify instances, with the decision tree performing slightly better at identifying true negatives and neural network achieving mixed results. ultimately, the choice of the most appropriate algorithm will depend on the specific characteristics of the data set and the objectives of the application. therefore, considering all the analyses, the cart decision tree algorithm performed best. comparing the results of this research with previous work, in previous work, the authors state that the full model was applied to the training set, where it accurately detected 340 (64.03%) game clips, and misdiagnosed 551 (7.07%) non-games. clips, and obtained a kappa coefficient of 0.430 [21]. on the test set, the model achieved an accuracy of 93 (52.54%) gaming clips, while misdiagnosing 210 (8.67%) non-gaming clips, resulting in a kappa coefficient of 0.330. this performance was comparable to the baker [3] model. in this research, the best algorithm evaluated was the cart decision tree, achieving a hit rate between 76% and 100% in all metrics evaluated. this algorithm identified 45.99% true positives, 41.82% true negatives, 6.38% false positives, and 5.78% false negatives. furthermore, our classifier performed better than previous work [21]. 5 conclusion in this study, we investigated “gaming the system” behavior among novice programmers in a public, rural school in northeast brazil, employing an experimental approach and developing a detector using a hybrid approach. we use cognitive task analysis to understand how experts encode sequences of actions in a learning environment, distinguishing between “gaming the system” or not, and implement a cognitive model of the process of encoding these behaviors. the identified patterns revealed rules for five “gaming the system” attitudes, including two new ones not present in previous models. however, our machine learning model showed mixed results, with the neural network having high number of true positives but at the cost of a high number of false positives. decision tree and knn demonstrated a more balanced classification ability. overall, the decision tree cart algorithm performed best. although our current model was built with data from a specific introductory programming context, it represents a significant contribution to understanding and detecting this behavior, given the difficulty in generalizing “gaming the system” detectors reported in the literature. as a future perspective, we plan to conduct a comparative analysis of data from public school students in rural and urban environments, looking for similar socioeconomic and demographic contexts. by replicating our study in different settings, we hope to better understand the observed behaviors and identify possible variations or distinct patterns across these contexts. this will allow us to explore how factors such as geographic location, access to educational resources, and demographic characteristics can influence the manifestation of “gaming the system” behavior. furthermore, by testing and evaluating our detector with additional datasets from different environments, we will be able to verify its effectiveness and generalizability across a variety of educational contexts. this broad, comparative approach will help us improve our understanding of “gaming the system” behavior and develop more effective strategies for detecting and addressing it in different educational settings. notes 1.https://ada-projeto.vercel.app/. 2.https://scikit-learn.org/stable/. references aleven, v., mclaren, b., roll, i., koedinger, k.: toward tutoring help seeking. in: 7th international conference on intelligent tutoring systems, pp. 227–239. springer, berlin, heidelberg (2004) google scholar  aleven, v., mclaren, b., roll, i., koedinger, k.: toward metacognitive tutoring: a model of help seeking with a cognitive tutor. int. j. artif. intell. educ. 16(2), 101–128 (2006) math  google scholar  baker, r., de carvalho, a.: labeling student behavior faster and more precisely with text replays. in: educational data mining 2008, pp. 1–10. montreal, canada (2008) google scholar  baker, r., walonoski, j., heffernan, n., roll, i., corbett, a., koedinger, k.: why students engage in gaming the system behavior in interactive learning environments. j. interact. learn. res. 19(2), 185–224 (2008) math  google scholar  baker, r.s., corbett, a.t., koedinger, k.r.: detecting student misuse of intelligent tutoring systems. in: 7th international conference on intelligent tutoring systems, pp. 531–540. springer, berlin, heidelberg (2004) google scholar  baker, r.s., corbett, a.t., koedinger, k.r., wagner, a.z.: off-task behavior in the cognitive tutor classroom: when students game the system. in: proceedings of the sigchi conference on human factors in computing systems, pp. 383–390. acm, vienna, austria (2004) google scholar  bartholomé, t., stahl, e., bromme, r.: help-seeking in interactive learning environments: effectiveness of help and learner-related factors in a dyadic setting. cogn. learn. styles 4, 81–88 (2004) math  google scholar  beck, j.e.: using response times to model student disengagement. in: proceedings of the its2004 workshop on social and emotional intelligence in learning environments, vol. 20, pp. 88–95. maceio, brazil (2004) google scholar  berrar, d.: cross-validation. in: encyclopedia of bioinformatics and computational biology, pp. 542–545. elsevier, oxford (2019) google scholar  burse, k., yadav, r.n., shrivastava, s.: channel equalization using neural networks: a review. ieee trans. syst. man cybern. part c appl. rev. 40(3), 352–357 (2010) google scholar  cooke, n.j.: varieties of knowledge elicitation techniques. int. j. hum. comput. stud. 41(6), 801–849 (1994) article  math  google scholar  baker, r.s., corbett, a.t., roll, i., koedinger, k.r.: developing a generalizable detector of when students game the system. user model. user-adap. inter. 18(3), 287–314 (2008) article  google scholar  huang, y., et al.: item response theory-based gaming detection. in: 15th international conference on educational data mining, pp. 1–10. international educational data mining society, durham, usa (2022) google scholar  karumbaiah, s., ocumpaugh, j., baker, r.s.: context matters: differing implications of motivation and help-seeking in educational technology. int. j. artif. intell. educ. 32(3), 685–724 (2022) article  math  google scholar  koch, c.j., kleinmann, m.: a stitch in time saves nine: behavioural decision-making explanations for time management problems. eur. j. work organ. psy. 11(2), 199–217 (2002) article  math  google scholar  martinez miron, e.a., du boulay, b., luckin, r.: goal achievement orientation in the design of an ile. in: proceedings of the its2004 workshop on social and emotional intelligence in learning environments, pp. 72–78. maceio, brazil (2004) google scholar  muldner, k., burleson, w., van de sande, b., vanlehn, k.: an analysis of students’ gaming behaviors in an intelligent tutoring system: predictors and impacts. user model. user-adap. inter. 21(1), 99–135 (2011) article  google scholar  ogan, a., walker, e., baker, r., rodrigo, m.m.t., soriano, j.c., castro, m.j.: towards understanding how to assess help-seeking behavior across cultures. int. j. artif. intell. educ. 25, 229–248 (2015) article  google scholar  oliveira, d.a., de jesus, a.m.c., et al.: as políticas de avaliação e responsabilização no brasil: uma análise da educação básica nos estados da região nordeste. revista ibero-americana de educação, pp. 1–20 (2020) google scholar  paquette, l., baker, r.s.: comparing machine learning to knowledge engineering for student behavior modeling: a case study in gaming the system. interact. learn. environ. 27(5–6), 585–597 (2019) article  math  google scholar  paquette, l., de carvalho, a.m., baker, r.s.: towards understanding expert coding of student disengagement in online learning. in: 36th annual conference of the cognitive science society, pp. 1–6. austin, tx (2014) google scholar  pardos, z.a., baker, r.s., san pedro, m.o., gowda, s.m.: affective states and state tests: investigating how affect and engagement during the school year predict end-of-year learning outcomes. j. learn. anal. 1(1), 107–128 (2014) article  google scholar  ryan, allison m., et al.: do gender differences in help avoidance vary by ethnicity? an examination of african american and european american students during early adolescence. dev. psychol. 45(4), 1152 (2009) google scholar  richey, j.e., et al.: gaming and confusion explain learning advantages for a math digital learning game. in: 22nd international conference on artificial intelligence in education, aied 2021, utrecht, the netherlands, june 14–18, 2021, proceedings, part i, pp. 342–355. springer, cham (2021) google scholar  rodrigo, m.m.t., et al.: affective and behavioral predictors of novice programmer achievement. in: proceedings of the 14th annual acm sigcse conference on innovation and technology in computer science education, pp. 156–160. acm, paris, france (2009) google scholar  solomon, p.: the think aloud method: a practical guide to modelling cognitive processes: mw van someren, yf barnard, and jac sandberg. knowledge based systems series, academic press, san diego (1994) math  google scholar  walonoski, j.a., heffernan, n.t.: detection and analysis of off-task gaming behavior in intelligent tutoring systems. in: international conference on intelligent tutoring systems, pp. 382–391. springer, location (2006) google scholar  solyst, j., thakur, t., dutta, m., asano, y., petersen, a., williams, j.j.: procrastination and gaming in an online homework system of an inverted cs1. in: proceedings of the 52nd acm technical symposium on computer science education, pp. 789–795 (2021) google scholar  gong, y., beck, j., heffernan, n.t., forbes-summers, e.: the impact of gaming (?) on learning at the fine-grained level. in: proceedings of the 10th international conference on intelligent tutoring systems (its2010) part, vol. 1, pp. 194–203 (2010) google scholar  download references author information authors and affiliations computer center-cin, federal university of pernambuco, recife, brazil hemilis joyse barbosa rocha & patricia cabral de azevedo restelli tedesco institute of computing-ic, federal university of alagoas, maceió, brazil evandro de barros costa federal institute of alagoas, maceió, brazil hemilis joyse barbosa rocha authors hemilis joyse barbosa rochaview author publications search author on:pubmed google scholar evandro de barros costaview author publications search author on:pubmed google scholar patricia cabral de azevedo restelli tedescoview author publications search author on:pubmed google scholar corresponding author correspondence to hemilis joyse barbosa rocha . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper rocha, h.j.b., de barros costa, e., de azevedo restelli tedesco, p.c. (2025). a knowledge engineering-based approach to detect gaming the system in novice programmers. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_2 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords gaming the system novice programmer detect publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature reinforcement learning with utility-based semantic for goals | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper reinforcement learning with utility-based semantic for goals conference paper first online: 30 january 2025 pp 354–369 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) reinforcement learning with utility-based semantic for goals download book pdf download book epub christian delgado polar  orcid: orcid.org/0000-0001-6123-64969,10, karina valdivia delgado  orcid: orcid.org/0000-0002-9120-89879 & valdinei freire  orcid: orcid.org/0000-0003-0330-39319  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 378 accesses abstract stochastic shortest path problems (ssps) are markov decision processes with goal states and the problem is to find policies to achieve the goal with the lowest possible expected cost. something common in this type of problem is the existence of states from which it is not possible to reach the goal, these states are called dead-end. in this case, it is important to have methods that consider the importance, not only of cost, but also of the probability of achieving the goal. in reinforcement learning (rl), the problem of making this trade-off between cost to goal and probability to goal has been little studied and the common strategy to deal with ssps with dead-ends is using discounts. in some works, penalties are used on exploration in ssps with dead-ends. however, using discounts and penalties can lead to errors in finding a policy with a desired trade-off. the gubs criterion (goals with utility-based semantic) considers this type of trade-off without using discounts or penalties, has good semantics based on the expected utility theory and has been used to solve ssps with dead-end states in the planning area. thus, in this work, the gubs criterion is used to propose the first two algorithms for rl to do a trade-off between probability to goal and cost to goal, without using discounts or penalties: the q-learning-gubs and q-learning-egubs+cmax. theoretical and experimental results show that the proposed algorithms make this trade-off according to the configuration of the gubs’ parameters (code enabled on https://github.com/qlearninggubs/code). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others $$\alpha $$ -mcmp: trade-offs between probability and cost in ssps with the mcmp criterion chapter © 2023 trading utility and uncertainty: applying the value of information to resolve the exploration–exploitation dilemma in reinforcement learning chapter © 2021 a linear online guided policy search algorithm chapter © 2017 1 introduction stochastic shortest path problems (ssps) are markov decision processes (mdps) with a set of goal states. in these problems, the objective is to find a policy with the lowest expected cumulative cost to achieve these goals. in many cases, these problems have states from which it is not possible to reach the goal, these states are called dead-ends. in problems with dead-ends the criterion of the lowest expected cost becomes ill-defined when there is no a proper policy (one that reaches the goal with probability 1) [6]. some works try to solve ssps with dead-end, some of them seek to find the policy with the maximum probability of reaching the goal (maxprob) [7] or give greater preference to probability and consider cost as a secondary aspect (lexicographic criteria) [6, 14]. however, it is not always recommended to find policies with the highest probability of achieving the goal as they may have high costs. sometimes it is better to make a trade-off between probability and cost, for example, decreasing the probability a little to get a large decrease in cost or increasing the cost a little to get a large increase in probability, this characteristic is called infinite-infinitesimal trade-off [1]. an example of this type of infinite-infinitesimal trade-off is illustrated in the river problem. the river problem [4] considers a \(n_x \times n_y\) grid. the agent must cross a river from one river bank starting from an initial position \(s_0\) to a destination position on the other bank. there is a bridge at position \(y = n_y\). the agent’s actions are to move in the four cardinal directions. the actions on the river’s bank (\(x = 1\) or \(x = n_x\)) and the bridge are deterministic. the actions are probabilistic within the river. in the river, there is a probability p of following down the river, with the risk of reaching the waterfall located at \(y = 1\), where, the agent becomes trapped or killed (dead-end). the optimal solution for the lexicographical criteria is to walk toward the bridge and traverse it since this policy has a probability 1 to reach the goal, even if the cost is very high. in this problem, for example, a solution with a probability 0.9999 should be accepted if we get a large decrease in cost. in reinforcement learning (rl), most of the theoretical works focus on finite-horizon and infinite-horizon or on loop-free ssp [12]. also, the problem of making trade-off has been little studied for ssps with dead-ends. additionally, a common strategy to follow with ssps with dead-end states in rl is using discounts. geibel and wysotzki [5] consider an mdp with goal and error states (undesirable or dangerous states). the proposed rl solution is based on weighting the cost and the probability of entering error states. this work uses discount factor. park [8] considers an mdp with goal and fail states. the proposed q-learning-like algorithm records past failure experiences and uses them to guide the exploration, this work [8] also uses a discount factor. using a discount factor could have an undesirable effect of biasing importance towards short-term behavior and thus diminishing the incentive to eventually achieve the goal state [1, 12]. in some works [12], penalties are used in the exploration problem for ssps with dead-ends. however, using penalties can lead to errors in finding a policy with a desired trade-off too, as was formally demonstrated by [1]. the gubs criterion (goals with utility-based semantic) [4] was proposed to make a correct infinite-infinitesimal trade-off in planning problems. this criterion allows us to find solutions for ssps with dead-end, with a parameterization with meaning based on the expected utility theory. thus, in this work, the gubs criterion is used to propose the first two algorithms for rl to do a trade-off between probability to goal and cost to goal, without using discounts or penalties, that are: the q-learning-gubs and the q-learning-egubs+cmax. additionally, to the best of our knowledge, the proposed algorithms are the first to use the expected utility theory to make this trade-off in rl, which allows more precise value criteria to be designed. 2 problem formalism and background an stochastic shortest path (ssp) is defined by a tuple \(m= \langle s,a,p,c,\mathcal {g} \rangle \), where: s is a discrete and finite set of completely observed states that model the world, a is a finite set of actions, \(p:s \times a \times s \rightarrow [0,1]\) is a probabilistic transition function, where, \(p(s,a,s') = pr(s_{t+1} = s'|s_t = s, a_t = a)\) is the probability of transition from a state \(s \in s\) to a state \(s' \in s\) after executing an action \(a \in a\), \(c: s \times a \rightarrow \mathcal {r}\) is a cost function for each action \(a \in a\) in a state \(s \in s\), \(\mathcal {g} \subset s\) is the set of goal states, i.e. \(g \in \mathcal {g}\), such that \(c(g,a)=0\), \(p(g,a, g)=1\), \(\forall a \in a\) and \(c(s,a) > 0\), \(\forall s \in s \setminus \mathcal {g}\). in a state \(s_t \in s\) of the environment at time t, the agent executes an action \(a_t \in a\) that has a cost \(c_t=c(s_t,a_t)\) and probabilistic effects generating a future state \(s_{t+1}\) as a function of \(p(s_t,a_t,s_{t+1})\). every interaction that occurs can be summarized in a history of length t represented by \(h_t=\{s_1,a_1,c_1,s_2,a_2,c_2,s_3,...,s_{t-1},a_ {t-1},c_{t-1},s_t\}\). the set of all historical possibilities is \(\mathcal {h}=(s \times a \times \mathcal {r}_{>0})^* \times s\). in an ssp, the process follows a history \(h_t\) that ends in a goal state \(g \in \mathcal {g}\). the solution for an ssp is a policy \(\pi \) that can be stationary (\(\pi : s \rightarrow a\)) where, each state s is mapped on an action a or not stationary (\(\pi : \mathcal {h} \rightarrow a\)) where, a historical \(h_t\) is mapped at time t in an action \(a_t\). this policy must be optimal \(\pi ^*\) under some criteria. a policy is called proper if the process achieves a goal state with probability 1 when it is followed, formally, \(\lim _{t\rightarrow \infty }pr(s_t \in \mathcal {g}|\ \pi ,s_0)=1\) where, \(s_0\) is the initial state of the process. the optimal value function \(v^*\) for an ssp is defined by the bellman equation \(v^{*}(s) =\min _{a \in a} \{ c(s,a) + \displaystyle \sum _{s' \in s}p(s,a,s')v^{*}(s') \}\). the optimal policy \(\pi ^*(s)\) for state s is \(\pi ^*(s) = \arg \min _{a \in a}\{c(s,a)+\displaystyle \sum _{s' \in s}p(s,a,s')v^{*}(s')\}\). a state s is called dead-end state when the probability of reaching a goal state from s is 0. formally, \(\lim _{t\rightarrow \infty }pr(s_t \in \mathcal {g}|\pi ,s_0 \in \mathcal {d})=0\). \(\mathcal {d} \subset s \) is the set of dead-end states. conventional ssps do not consider dead-end. 2.1 criteria for ssps with dead-ends in environments with dead-end, the agent must decide between finding a policy with a low expected total cost \(\overline{c}_g(s)\) from an initial state s until the goal state or a policy with the maximum probability of achieving the goal \(p_g(s)\). several criteria have been proposed to solve ssps with dead-end some of which prioritize the probability \(p_g(s)\) [7, 13,14,15]. instead of prioritizing probability, an alternative is to make a trade-off between cost to goal or probability to goal. the criterion fsspude [6] extends the original ssp with an artificial action q and a penalty d. when q is used, the process transitions to a goal state and the agent pays the cost d, therefore \(p(s,q,g) = 1\), \(\forall s \in s\) and for a goal state g; and \(c(s,q) = d\), \(\forall s \in s\). then, the objective is to find a policy that minimizes the expected accumulated cost \(\overline{c}_g(s) = \lim _{t\rightarrow \infty } e[\sum ^{t-1}_{t=0} c(s_t,a_t)| \pi ,s_0]\). thus, fsspude can give more or less importance to avoid dead-end by choosing an appropriate d. the discounted cost criterion uses the discount factor to allow policies with a probability of reaching the goal less than 1, to converge [13]. the objective is to minimize the expected accumulated cost \(\overline{c}_g(s) = \lim _{t\rightarrow \infty } e[\sum ^{t-1}_{t=0} \gamma ^t c (s_t,a_t)| \pi ,s_0]\). the penalty d used in fsspude and the discount factor used in discounted cost criterion depend on the ssp. because it is necessary to know the structure of the ssp, both values are difficult to determine to make the desired trade-off between probability and cost [1]. besides, these criteria do not have the property \(\alpha -\)strong probability-to-goal for \(\alpha > 0\) [1]. definition 1 (\(\alpha -\) strong probability-to-goal [1]). for \(0 \le \alpha \le 1\), a criterion has \(\alpha -\)strong probability-to-goal if for every ssp and for any pair of policies \(\pi , \pi ' \in \varpi \), the following condition is true: \(\pi \succ \pi ' \longrightarrow \frac{p^{\pi }_g(s_0)}{p^{\pi '}_g(s_0)} \ge \alpha \). where \(\pi \succ \pi '\) means that the decision-maker prefers \(\pi \) to \(\pi '\). 2.2 goals with utility-based semantic-gubs a criterion that solves ssps with dead-end with a clear trade-off between cost and probability of achieving the goal is the gubs [4] criterion. gubs is based on utility theory and evaluates policies based on: $$\begin{aligned} \begin{aligned} v^\pi (s) &= \lim _{t \rightarrow \infty } e[u(c_t,\beta _t)|\pi ,s_0 = s], \\ u(c_t,\beta _t) &= u(c_t) + k_g \beta _t, \end{aligned} \end{aligned}$$ (1) where, u(.) is a utility function, \(c_t = \sum ^{t-1}_{t=0}c(s_t,a_t)\) is the accumulated cost in time t, \(k_g \) is the reward for reaching a goal state, and \(\beta _t\) is 1 if a goal is reached. unlike the discounted cost criterion and fsspude, in which the choice of penalty and discount factor depend on the ssp, the value \(k_g\) do not depend on the ssp. additionally, gubs has the \(\alpha -\)strong probability-to-goal property [1]. some planning algorithms have been proposed to calculate the gubs criterion, they are: gubs-vi [4], egubs-vi [3] and egubs-ao* [1]. the gubs-vi algorithm, which is based on the value iteration (vi) algorithm solves gubs for discrete costs and considers as a parameter the maximum accumulated cost \(c_{max}\) for every state, including dead-end states, which will have a cost of \(c_{max}\). as the algorithm needs to calculate the utility based on the accumulated cost, the policy found is non-stationary. to be able to use vi, extended states must be used with the cost accumulated so far. the formulation with discrete cost and extended states for gubs is presented next. definition 2 (discrete cost [4]). an ssp with dead-end can be restated into an mdp with bounded discrete costs \(m= \langle x,a,p',c',r'_g \rangle \), where, \(x = {s} \times \mathbb {n}\) is an extended state space, which are the states with the accumulated cost so far, \(p'(x,a,x')=p(s,a,s')\) where, \(x = (s,c)\) and \(x'=(s',c+c(s,a))\), \(x,x' \in x\) and c is the accumulated cost, \(c'(x,a)=u( c+c(s,a))-u(c)\)footnote 1 for a utility function u(.), and \(r'_g = k_g\) is the terminal reward for the goal states with value \(k_g \in \mathbb {r}\). the gubs-vi algorithm uses definition 2 to calculate the q-value: $$\begin{aligned} \small q(s,c,a) \leftarrow c'(s,a) + \displaystyle \sum _{s' \in s \cup \mathcal {g}} p(s,a,s')v(s',c+c(s,a)). \end{aligned}$$ (2) a special case of the gubs criterion is the exponential gubs (egubs) criterion. in this case, the utility function is an exponential function used in risk sensitive ssps (risk sensitive ssp rs-ssp) [9], in which the risk factor \(\lambda \), that models the agent’s risk attitude, is negative. the egubs is defined in terms of the lexicographic criterion. definition 3 (risk-sensitive lexicographic criterion [1, 3]). the risk-sensitive lexicographic criterion considers: if \(p_g^{\pi }(s) > p_g^{\pi '}(s)\), then \(\pi \succ \pi '\); or if \(p_g^{\pi }(s) = p_g^{\pi '}(s)\) and \(v^{\pi }_{\lambda } > v^{\pi '}_{\lambda }\), then \(\pi \succ \pi '\). the egubs-vi algorithm [3] solves the egubs criterion exactly. egubs-vi calls the risk-sensitive-lexicographic-vi algorithm [3] which receives an ssp and a \(\lambda \) and calculates the optimal policy \(\pi ^*_l\), the optimal value \(v_ {\lambda }\) and the maximum probability for the goal \(p_g\) for the lexicographic criterion described in definition 3. we will use risk-sensitive-lexicographic-vi as part of one of the algorithms proposed in this work. the egubs-ao* algorithm optimizes the search for the solution considering a finite, acyclic hypergraph, whose nodes are augmented states (s, c) and hyperedges leading (s, c) to its successor state \(s',c'\), where \(s'\) is the successor state in the original mdp and \(c' = c + c(s,a)\) for the action performed a. egubs-ao* solves egubs by means of a heuristic search with a variant of the ao* algorithm. 2.3 reinforcement learning in rl, it is common to divide the process into episodes (represented by e). an episode can end in several ways, for example, when a goal state, a dead-end or a pre-defined time are reached. the q-learning algorithm [16] is a well-known algorithm in rl. the q-value for all \(s_t \in s\) in time t when performing action \(a_t \in a\) in time t and following the optimal policy \(\pi ^*\) is: $$\begin{aligned} \begin{aligned} q(s_t,a_t) & \leftarrow q(s_t,a_t) + \alpha _t\delta , \\ \delta &= c(s_t,a_t) + \gamma {\min }_{a' \in a}q(s_{t+1},a') q(s_t,a_t), \end{aligned} \end{aligned}$$ (3) where, \(\alpha _t\) is the learning rate (\(0\le \alpha _t \le 1\)) that determines the learning speed, \(\gamma \) is the discount factor on future values and \(s_{t+1}\) is the next state after executing action \(a_t\). theorem 1 (convergence of the q-learning [16]) let the costs in each time t be \(c_t\le {c}\), learning rates \(0 \le \alpha _t < 1 \), and \(\sum _{t=1}^{\infty }\alpha _t(s,a) = \infty ,\sum _{t=1}^{\infty }(\alpha _t(s,a))^2 < \infty \) then \(q_t(s,a)\rightarrow q^*(s,a)\) as \(t\rightarrow \infty \), for every state s and action a, with probability 1. the algorithm proposed by geibel and wysotzki [5] is one of the few methods for rl designed specifically to perform a trade-off between cost and probability of reaching an error state. although this algorithm uses the concept of probability of reaching an error state, this concept is closely related to the concept of probability to reaching the goal used in this work. let \(\varphi \) be the set of error states. the probability of reaching an error state (also called risk) from an initial state s is defined as \(\rho ^{\pi }(s) = e\left[ \sum ^{\infty }_{t=0}\bar{\gamma }^t\bar{r}_t|s_0=s\right] \), where \(\bar{\gamma }^t\) is the discount factor for the risk at time t and \(\bar{r}_{s,a}(s') = {\left\{ \begin{array}{ll} 1, & \text {if} \,s\in \varphi \, \text {and}\, s' = \eta \\ 0, & else \end{array}\right. }\), where \(\eta \) is a fictitious absorbing state where the agent transits after reaching an error state. with these concepts, a new value for the risk is defined as \(\bar{q}^{\pi }(s,a) = e\left[ \bar{r}_0 + \bar{\gamma }\rho ^{\pi }(s')\right] \), where \(s_0 = s, a_0 = a\) and \(s'\) is the next state. then a new general value is defined as \(q^{\pi }_{\xi }(s,a) = \xi q^{\pi }(s,a) \bar{q}^{\pi }(s,a)\). the trade-off between the probability of reaching an error state and the cost to goal is determined by varying the weight \(\xi \). 3 rl under the gubs criterion as shown in the previous section, the gubs criterion is the only known criterion that guarantees \(\alpha \)-strong probability-to-goal priority. in this work, rl algorithms are introduced to solve the gubs criterion. some challenges to face in the development of rl algorithms that use the gubs criterion are: (1) use extended states with cost since gubs finds non-stationary policies, something that is not the default behavior of rl algorithms; (2) define the type of the initial and final policies used in the algorithms; (3) the equations of the gubs-vi algorithm [4] must be adapted based on temporal difference without using the transition function a priori; and (4) define or compute the maximum accumulated cost \(c_{max}\) that is used by the gubs algorithms. the algorithms introduced are: q-learning-gubs and q-learning-egubs+\(c_{max}\). these algorithms do not need to know transition probabilities or costs and do not use discounts or penalties to find the solution considering the gubs criterion. they are inspired by two well-known algorithms based on temporal difference, that are: q-learning and dyna-q (a model-based algorithm). 3.1 q-learning-gubs the q-learning-gubs algorithm is based on the q-learning [16] and gubs-vi [4] algorithms. like gubs-vi, q-learning-gubs solves gubs for discrete costs, can use several utility functions (for example the exponential function and others that hold the properties defined in [1]), and is an approximate algorithm that has as a parameter the maximum accumulated cost \(c_{max}\) for every state. based on eqs. 3 and 2, the following is the modified bellman equation to solve the discrete cost problem (definition 2). $$\begin{aligned} \begin{aligned} q(x_t,a_t) &\leftarrow q(x_t,a_t) + \alpha _t \delta , \\ \delta &\leftarrow c'_{t+1}(x_t,a_t) + \max _{a' \in \mathcal {a}} q(x_{t+1},a') q(x_t,a_t), \end{aligned} \end{aligned}$$ (4) where, the q-value is calculated for each extended state \(x_t = (s_t,c_t)\) considering the utility \(c'_{t+1 }\) of definition 2. notice that eq. 4 does not consider a discount factor. furthermore, q-learning-gubs uses a random policy as an initial policy and calculates a non-stationary final policy. the q-learning-gubs algorithm (algorithm 1) first initializes the q table considering as value the differences between the minimum utility and the utility for each accumulated cost c for states that are not goal states. for the goal states, the algorithm sets the q-value as \(k_g\) (line 25). then, the algorithm iterates over a set of episodes in which it computes \(\alpha _e\) and \(\epsilon _e\) in line 4 and gets the initial extended state \(x_0\) in line 5. at each step, an action a is chosen for the current state s following an \(\epsilon _e\)-greedy policy, it acts on the environment, observes the cost of the action c(s, a) and the next state \(s'\) (lines 11 and 12). for goal states \(s \in \mathcal {g}\), \(c(s,a) = 0\) (line 14). then, with this data, the q-value is calculated (line 18) for each discrete accumulated cost \(c'' \in \{0,...,c_{max}\}\) based on eq. 4. the algorithm moves to another episode (i) when a goal state is reached, (ii) when the accumulated cost is greater than or equal to the value \(c_{max}\) or (iii) when a dead-end is found (line 19). finally, the algorithm calculates the gubs policy based on the q table, choosing the actions with the highest value for each extended state (line 25). q-learning-gubs convergence. the algorithm finds an approximate policy for the problem regardless of the utility function used. the proof of convergence is based on the following definition and theorems. definition 4 (ssps with discrete cost and maximum accumulated cost \(c_{max}\)). the ssp of definition 2 can be restated into a new ssp with the same elements but with a finite set of states \(x' \subseteq x\) where, \(x'= \{x | x = (s,c), c \le c_{max} \}\) and c is the accumulated cost. theorem 2 (q-learning-gubs convergence). let m be the ssp of definition 4 where, in the algorithm, \(0 \le \alpha _t < 1 \), \(\sum ^{n}_{i=1} \alpha _t = \infty \) and \(\sum ^{ n}_{i=1} \alpha _t^2<\infty \), and as the number of episodes \(n \rightarrow \infty \), q-learning-gubs converges to the optimal value \(q'^*(x ,a)\) for m. proof since the ssp of definition 4 has a finite number of states, it is possible to use the same proof of convergence as the q-learning algorithm (theorem 1). thus, the algorithm 1 converges to the optimal value \(q'^*(x,a)\) for the ssp of the definition 4. theorem 3 (the q-learning-gubs algorithm is approximate). let m be the ssp of definition 2 where, \(0 \le \alpha _t < 1 \), \(\sum ^{n}_{t=1} \alpha _t = \infty \) and \(\sum ^{n}_{t=1} \alpha _t^2<\infty \), and \(n \rightarrow \infty \) is the number of episodes, so: 1. q-learning-gubs converges to the value \(q'^*(x,a)\) where, \(q^*(x,a) q'^*(x,a) \ge 0\) for every extended state x and action a, and \(q^*(x,a)\) is the optimal value for the problem in definition 2; 2. \(\lim _{c_{max} \rightarrow \infty }q'^*(x,a) = q^*(x,a)\). proof the q-learning-gubs algorithm restates the original ssp m (definition 2) into an ssp \(m'\) with maximum cost \(c_{max}\) (definition 4). since \(m'\) has a set of states \(x' \subseteq x\), then \(q^*(x,a) q'^*(x,a) \ge 0\). let \(q^{\pi *}\) be the optimal policy value of the original ssp in the ssp of definition 4, then \(q^*(x,a) \ge q'^* (x,a) \ge q^{\pi *} (x,a)\). the difference between these values is given by the traces that do not reach the goal before the cost \(c_{max}\). the greater the value \(c_{max}\), the lower the probability of having these traces, therefore \(q^*(x,a) \rightarrow q^{\pi *} (x,a)\) and \( q^{\pi *} (x,a) \rightarrow q'^* (x,a)\). thus \(\lim _{c_{max} \rightarrow \infty }q'^*(x,a) = q^*(x,a)\). 3.2 q-learning-egubs+\(c_{max}\) q-learning-egubs+\(c_{max}\) is based on gubs-vi, egubs-vi, q-learning and dyna-q [11] algorithms. as egubs-vi, q-learning-egubs+\(c_{max}\) computes a lexicographic policy as an initial policy and calculates a non-stationary final policy. although q-learning-egubs+\(c_{max}\) is based on the gubs-vi algorithm that solves the gubs criterion, it calculates the lexicographic policy with the risk-sensitive lexicographic-vi algorithm ( [3]) which is based on the exponential utility function. thus, q-learning-egubs+\(c_{max}\) solves the egubs criterion. different from q-learning-gubs, q-learning-egubs+\(c_{max}\) allows to calculate the \(c_{max}\) value automatically. in the q-learning-egubs+\(c_{max}\) algorithm, a lexicographic policy \(\pi _l\) is found based on a model of the transition probability \(\hat{p}\) and the costs of actions \(\hat{c}\) in the ssp, which is calculated at each f episodes with the risk-sensitive lexicographic-vi algorithm [3]. as in [11] and [2] the model of probabilities is calculated by \(\hat{p}(s,a,s') = \frac{\sum n(s,a,s')}{\sum _{s''}n(s,a,s'')}\), and the model of cost is calculated by \( \hat{c}(s,a) = \frac{\sum c(s,a)}{\sum _{s''}n(s,a,s'')}\) where, \(n(s,a,s')\) is the number of times a state s is visited, the action a is performed and passed to the state \(s'\). to compute the \(c_{max}\) value, after calculating the lexicographic policy \(\pi _l\), the algorithm q-learning-egubs+\(c_{max}\) evaluates the expected cost to goal \(\overline{c}^{\pi _l}_g(s)\) of the policy \(\pi _l\) iteratively for each state s based on eq. 5, without considering the infinite costs of the dead-end states. $$\begin{aligned} \overline{c}^{\pi _l}_g(s) = {\left\{ \begin{array}{ll} 0 & , s\in \mathcal {g} \cup \mathcal {d} \\ \hat{c}(s,\pi _l(s)) + \displaystyle \sum _{s' \in s}\hat{p}(s,\pi _l(s),s')\overline{c}^{\pi _l}_g(s') & , otherwise \end{array}\right. } \end{aligned}$$ (5) the \(c_{max}\) value is equal to the cost of the lexicographic policy from the initial state \(s_0\) to the goal: $$\begin{aligned} c_{max} = \overline{c}^{\pi _l}_g(s_0). \end{aligned}$$ (6) since the steps for computing the lexicographic policy and \(c_{max}\) are expensive in terms of processing time cost, and this cost depends on the number of states and the type of problem, these steps are only performed in some episodes. q-learning-egubs+\(c_{max}\) (algorithm 2), in addition to the inputs of the q-learning-gubs algorithm, receives as input the minimum error \(\varepsilon _l\) used in the risk-sensitive lexicographic-vi algorithm and the amount f of episodes needed before recalculating the lexicographic policy. after initializing the values of q, n(s, a), \(n(s,a,s')\), \(\hat{c}(s,a)\), \(\hat{c}( s,a)\) and \(\hat{p}(s,a,s')\); algorithm 2 performs a set of episodes. in each episode, it performs the same operations to interact with the environment and to calculate the value q as in the q-learning-gubs algorithm. furthermore, each f episodes, algorithm 2 recalculates the model of probabilities \(\hat{p}\) and costs \(\hat{c}\) based on the counts of the visits to states n(s, a) and \(n(s,a ,s')\) and the accumulated cost of the visits to states \(\hat{c}(s,a)\) (lines 8 and 9). it uses these models to calculate the lexicographic policy (line 11) using the risk-sensitive lexicographic-vi algorithm [3]. the value \(c_{max}\) is calculated in line 12 based on eq. 6. algorithm 2 calculates the gubs value in line 25 based on eq. 4. finally, based on the gubs value the optimal gubs policy \(\pi ^*\) is calculated in line 26. the counts n(s, a) and \(n(s,a,s')\) and the accumulated cost \(\hat{c}(s,a)\) used in the model calculation, are updated in each episode (line 27). 4 experiments the experiments were carried out in two domains: the river [4] (see sect. 1) and the navigation domains. in the navigation domain [10], a robot is walking in a grid with dimensions \(n_x \times n_y\). in this domain, the robot starts from an initial position \(s_0\) and tries to go to a destination position. in the first and last lines (\(y = 1\) and \(y = n_y\)) the actions are deterministic and in the intermediate lines, the agent has a probability p(x, y) of disappearing, going to a dead-end state. the probability of disappearing is greater as the agent is closer to the goal. although these domains are simple, they have characteristics that make them difficult. these problems have uncertainty, risk and complex decision making. they were modeled as ssps with extended states, which include the position and accumulated cost. actions correspond to decisions to move and have cost 1. in the river domain the experiments were carried out in an environment of size \(n_y = 20\) and \(n_x=5\), the initial state is on a bank of the river \(x_0 = 1, y_0 = 2\), the goal on the other bank at position \(x_g = n_x, y_g = 1\) and two probability values were used \(p=0.4\) and \(p=0.6\). these are the same values of p used by [4] and they show different preferences for proper policies. in the navigation domain, the initial state is at \(x_0 = n_x, y_0 = n_y\), the goal is at position \(x_g = n_x, y_g = 1\), and two instances were used. an instance of size \(n_x = 15 \times n_y = 3\), with a minimum probability of disappearing of 0.2 and another instance of size \(n_x = 10\) \(\times \) \(n_y = 4\) with a minimum probability of disappearing of 0.1. table 1 shows the parameters used by the algorithms. a function with linear decay was used to vary \(\alpha \) and \(\epsilon \) from their respective maximum values to the minimum values of both parameters. the initial extended state was chosen randomly for each episode. table 2 shows the parameters used for the river and navigation problems. table 1. parameters for the algorithms.full size table table 2. parameters for the river and navigation problems.full size table in sects. 4.1 and 4.2, the proposed algorithms are compared with the gubs-vi algorithm. this choice is because the proposed algorithms are based on gubs-vi. the purpose of this comparison is to verify if the proposed algorithms found the same results obtained by gubs-vi. theoretically, the proposed algorithms should obtain the same result, however, since they use sampling, they could obtain a very close solution within the number of episodes used. in the figures shown in sects. 4.1 and 4.2, the x axis has the values of \(k_g\) expressed in logarithm base 2 and the y axis shows the probability and cost of reaching the goal from the initial state. each point in the figures is the average of 5 executions of algorithms. additionally, the standard deviation is shown. we also record the learning curves of these experiments. these results show that the algorithm that converges the fastest is q-learning-gubs algorithm. we do not include these curves for space reasons. in sect. 4.3, we compare the proposed algorithms with the geibel and wysotzki criterion [5]. notice that, the gubs criterion and other criteria (fsspude and discount cost) were compared by [1]. 4.1 results of the river domain figures 1a) and 1b) show the probability to goal obtained by the gubs-vi algorithm and the proposed algorithms q-learning-gubs and q-learning-egubs+\(c_{max}\) in the river problems with \(p=0.4\) and \(p=0.6\), respectively. the probabilities obtained by the algorithms are very close to the results obtained by the algorithm gubs-vi for both instances of the domain. the proper policy for the \(p=0.4\) instance is reached when \(k_g > 16\) (\(\log (k_g)=4\)), and the proposed algorithms find policies with probabilities increasingly closer to 1. for the instance with \(p=0.6\), the optimal policy when \(k_g \ge 0.5\) (\(\log (k_g)=-1\)) is a proper policy (i.e. crossing the bridge). in this case, the proposed algorithms find the proper policy. figures 1c) and 1d) compare the costs reached from the initial state by the three algorithms for the same instances of the river domain. the results are very close between the algorithms for both instances. fig. 1. probability to goal and cost to goal found by q-learning-gubs, q-learning-egubs+\(c_{max}\) and gubs-vi algorithms for different values of \(k_g\) for instance \(n_y = 20 \times n_x=5\) of the river domain. full size image 4.2 results of the navigation domain figures 2a) and 2b) show the probabilities to goal obtained by the three algorithms in the two described instances of the navigation domain. the probabilities found by the proposed algorithms are very close to the probabilities found by the gubs-vi algorithm for both instances of the problem. however, there is a larger standard deviation for values of \(k_g\) that have policies close to maxprob and correspond to the furthest route from the goal (the probability for the goal is 0.8 for the instance \(n_x = 15 \times n_y = 3\), and 0.81 for the instance \(n_x = 10 \times n_y = 4\)). the algorithms find a policy according to \(k_g\). for values \(k_g \ge 0.3\) (\(\log (k_g) = -1.7\)) in instance \(n_x = 15 \times n_y = 3\) and \(k_g \ge 0.05\) (\(\log (k_g) = -4.3\)) in the instance \(n_x = 10 \times n_y = 4\), the algorithm q-learning-gubs can find the maxprob. however, due to the parameters used, the algorithm q-learning-egubs+\(c_{max}\) can not find the maxprob policy in the instance \(n_x = 15 \times n_y = 3\) in any of the 5 executions. figures 2c) and 2d) compare the costs reached from the initial state by the three algorithms for the same two instances of the navigation domain. the policy costs calculated by the q-learning-egubs+\(c_{max}\) algorithm are very close to the policy costs calculated by the algorithm gubs-vi. fig. 2. probability to goal and cost to goal found by q-learning-gubs, q-learning-egubs+\(c_{max}\) and gubs-vi algorithms for different values of \(k_g\) for the navigation domain. full size image the q-learning-gubs algorithm finds policies with higher costs than the policies found by the gubs-vi algorithm, especially for large values of \(k_g\). next, we analyze this behavior. when \(k_g\) is large, the q-learning-gubs algorithm oscillates more especially during the learning of probabilities in states with uncertainty, it is difficult to reach the exact value of the actions since, in addition, the difference between action values are very small in this problem. then, the found policy uses actions with values close to the optimal values for some extended states, but not the optimal one. it causes the formation of cycles, in which the policy can return to the same real state but with a higher cost. thus, the total cost to the goal is increased. as far as we have analyzed, by changing the parameters, it is possible to reach the right cost, for example, by increasing the number of episodes. 4.3 comparison with the geibel and wysotzki criterion we compare the proposed algorithms with the geibel and wysotzki criterion [5]. for this comparison, we varied the parameter \(\xi \) between the values \([0.1, 1 \times 10^7]\) to evaluate if geibel and wysotzki criterion can compute the optimal policies for egubs only by choosing the parameters appropriately and, if not, how close it can get to the optimal value. the discounts \(\gamma = 0.9\) and \(\bar{\gamma } = 1\) were used. figure 3 shows the \(log_2(\xi )\) on the x axis and, on the y-axis, the values corresponding to the optimal policy found by the gubs-vi, q-learning-gubs and q-learning-egubs+cmax algorithms with \(k_g = 5\) and \(c_{max} = 100\), and the values of the optimal policies found by the geibel and wysotzki criterion evaluated with the gubs criterion for the river domain of size \(n_y = 20\) and \(n_x=5\) with \(p=0.4\). each point in the figure is the average of 5 executions of the algorithms. the values found by the algorithms q-learning-egubs+cmax, q-learning-gubs and gubs-vi are close. when \(log_2(\xi ) = 1\), the values of the policies found by the geibel and wysotzki criterion are close to the ones found by gubs-vi, q-learning-egubs+cmax and q-learning-gubs. notice that, to carry out this experiment, an indefinite set of values for \(\xi \) had to be varied, this is because there is no clear procedure to choose a certain value of \(\xi \) for this criterion, to have a guarantee of finding the resulting policy under gubs. in certain situations, it may not be feasible to carry out this type of procedure to guarantee finding certain policies. fig. 3. comparison of the value of the optimal policy for gubs-vi, q-learning-gubs and q-learning-egubs+cmax with the value of optimal policies for the geibel and wysotzki criterion for different values of \(\xi \) for the river domain \(n_y = 20\) and \(n_x=5\) with \(p=0.4\). full size image 5 conclusion in this work, we propose the first two algorithms for rl that use the gubs criterion: q-learning-gubs and q-learning-egubs+\(c_{max}\) to make a trade-off between probability to goal and cost to goal. q-learning-gubs is an approximate algorithm and can use several utility functions. q-learning-egubs+\(c_{max}\) is a model-based algorithm that calculates the \(c_{max}\) value for the particular case of the exponential utility function. experiments were carried out with the proposed algorithms with several instances corresponding to two domains. the results show that these algorithms can find policies that make a trade-off between probability and cost according to the parameter \(k_g\) and according to the utility function used. we also compare the proposed algorithms with geibel and wysotzki criterion [5]. the values of the policies found by geibel and wysotzki criterion for some values of parameters are close to those found by the proposed algorithms. different from geibel and wysotzki criterion that is based on \(\xi \) and discount factors, where the choice of parameters depends on the problem; the parameters of gubs (\(k_g\), \(u_{max}\) and \(u_{min}\)) do not depend on the problem, so, the gubs parameters of the proposed algorithms can be rationally set. the proposed algorithms could be the basis for designing more sophisticated rl algorithms that can work in more complex domains. notes 1.notice that \(c'\) is an utility over costs and the optimal policy must maximize the utility. references crispino, g.n., freire, v., delgado, k.v.: gubs criterion: arbitrary trade-offs between cost and probability-to-goal in stochastic planning based on expected utility theory. artif. intell. 316(c) (2023) google scholar  faycal, t., zito, c.: dyna-t: dyna-q and upper confidence bounds applied to trees. arxiv abs/2201.04502 (2022) google scholar  freire, v., delgado, k.v., reis, w.a.s.: an exact algorithm to make a trade-off between cost and probability in ssps. in: international conference on automated planning and scheduling 2019, vol. 29, pp. 146–154 (2019) google scholar  freire, v., delgado, k.v.: gubs: a utility-based semantic for goal-directed markov decision processes. in: international conference on autonomous agents and multiagent systems, pp. 741–749 (2017) google scholar  geibel, p., wysotzki, f.: risk-sensitive reinforcement learning applied to control under constraints. j. artif. intell. res. 24, 81–108 (2005) article  math  google scholar  kolobov, a., mausam, weld, d.s.: a theory of goal-oriented mdps with dead ends. in: conference on uncertainty in artificial intelligence, pp. 438–447. auai press, arlington, virginia, usa (2012) google scholar  kolobov, a., mausam, m., weld, d., geffner, h.: heuristic search for generalized stochastic shortest path mdps. in: international conference on automated planning and scheduling 2011, vol. 21, pp. 130–137 (2011) google scholar  park, i.w., kim, j.h., park, k.h.: accelerated q-learning for fail state and action spaces. in: 2008 ieee international conference on systems, man and cybernetics, pp. 763–767 (2008) google scholar  patek, s.d.: on terminating markov decision processes with a risk-averse objective function. automatica 37(9), 1379–1386 (2001) article  math  google scholar  sanner, s., yoon, s.: ippc results presentation in: international conference on automated planning and scheduling (2011). http://users.cecs.anu.edu.au/ssanner/ippc_2011/ippc_2011_presentation.pdf. accessed aug 2024 sutton, r.s.: integrated architectures for learning, planning, and reacting based on approximating dynamic programming. in: international conference on machine learning, pp. 216–224. morgan kaufmann (1990) google scholar  tarbouriech, j.: goal-oriented exploration for reinforcement learning. ph.d. thesis, université de lille (2022) google scholar  teichteil-konigsbuch, f., vidal, v., infantes, g.: extending classical planning heuristics to probabilistic planning with dead-ends. in: aaai conference on artificial intelligence, pp. 1017–1022. aaai press (2011) google scholar  teichteil-königsbuch, f.: stochastic safest and shortest path problems. in: aaai conference on artificial intelligence, vol. 26, pp. 1825–1831 (2012) google scholar  trevizan, f.w., teichteil-königsbuch, f., thiébaux, s.: efficient solutions for stochastic shortest path problems with dead ends. in: conference on uncertainty in artificial intelligence (2017) google scholar  watkins, c.j.c.h., dayan, p.: technical note. q-learning. mach. learn. 8(7), 279–292 (1992) article  math  google scholar  download references acknowledgments this study was supported in part by the conselho nacional de desenvolvimento científico e tecnológico (cnpq) – finance code 001, by the são paulo research foundation (fapesp) grant \(\#\)2018/11236-9 and the center for artificial intelligence (c4ai-usp), with support by fapesp (grant #2019/07665-4) and by the ibm corporation. author information authors and affiliations university of são paulo, cidade universitária, são paulo, sp, brazil christian delgado polar, karina valdivia delgado & valdinei freire school of computer science, san pablo catholic university, arequipa, peru christian delgado polar authors christian delgado polarview author publications search author on:pubmed google scholar karina valdivia delgadoview author publications search author on:pubmed google scholar valdinei freireview author publications search author on:pubmed google scholar corresponding author correspondence to christian delgado polar . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper polar, c.d., delgado, k.v., freire, v. (2025). reinforcement learning with utility-based semantic for goals. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_25 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords reinforcement learning dead-end q-learning gubs publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature d-cc integrals: generalizing cc-integrals by restricted dissimilarity functions with applications to fuzzy-rule based systems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper d-cc integrals: generalizing cc-integrals by restricted dissimilarity functions with applications to fuzzy-rule based systems conference paper first online: 12 october 2023 pp 243–258 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) d-cc integrals: generalizing cc-integrals by restricted dissimilarity functions with applications to fuzzy-rule based systems download book pdf download book epub joelson sartori  orcid: orcid.org/0000-0001-5379-82539, giancarlo lucca  orcid: orcid.org/0000-0002-3776-02609, tiago asmus  orcid: orcid.org/0000-0002-7066-715610, helida santos  orcid: orcid.org/0000-0003-2994-28629, eduardo borges  orcid: orcid.org/0000-0003-1595-76769, benjamin bedregal  orcid: orcid.org/0000-0002-6757-793411, humberto bustince  orcid: orcid.org/0000-0002-1279-619512 & … graçaliz pereira dimuro  orcid: orcid.org/0000-0001-6986-98889  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 573 accesses 6 citations abstract the discrete choquet integral (ci) and its generalizations have been successfully applied in many different fields, with particularly good results when considered in fuzzy rule-based classification systems (frbcss). one of those functions is the cc-integral, where the product operations in the expanded form of the ci are generalized by copulas. recently, some new choquet-like operators were developed by generalizing the difference operation by a restricted dissimilarity function (rdf) in either the usual or the expanded form of the original ci, also providing good results in practical applications. so, motivated by such developments, in this paper we propose the generalization of the cc-integral by means of rdfs, resulting in a function that we call d-cc-integral. we study some relevant properties of this new definition, focusing on its monotonicity-like behavior. then, we proceed to apply d-cc-integrals in a classification problem, comparing different d-cc-integrals between them. the classification acuity of the best d-cc-integral surpasses the one achieved by the best cc-integral and is statistically equivalent to the state-of-the-art in frbcss. this research was funded by fapergs/brazil (proc. 19/2551-0001660-3, 23/2551-0000126-8), cnpq/brazil (301618/2019-4, 305805/2021-5, 150160/ 2023-2). access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others cmin-integral: a choquet-like aggregation function based on the minimum t-norm for applications to fuzzy rule-based classification systems chapter © 2018 dissimilarity based choquet integrals chapter © 2020 a way to proper generalization of \(\phi \)-divergence based on choquet derivatives article 12 september 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. functions of a complex variable measure and integration real functions special functions stochastic integrals integral transforms and operational calculus 1 introduction the discrete choquet integral (ci) is an interesting aggregation operator that can capture the relationship between the aggregated data by means of a fuzzy measure (fm) [10]. it has been successfully applied in many different fields, such as decision-making [13, 29], classification [13, 28] and image processing [24]. this yielded the theoretical development of several generalizations [13] of the ci, such as the cc-integral [22] (a generalization by copulas [3]) and \(c_{f_1f_2}\)-integral [19] (a generalization by fusion functions \(f_1\) and \(f_2\) under some constraints), all of them providing at least competitive results when applied in practical problems. in particular, fuzzy rule-based classification systems (frbcss) [17], which are notable for their high interpretability while still achieving good classification results, seem to benefit from the application of the ci and their generalizations in the aggregation process that occur in the reasoning method. this can be seen in the works of lucca et al. [19], where the application of such integrals produced results that rival the state of the art in frbcss. bustince et al. [8] generalized the ci using restricted dissimilarity functions (rdf), in the form of d-integrals. following that, wieczynski et al. studied \(dc_{f}\)-integrals [28] and d-xc-integrals [29], obtaining promising results not only in decision making and classification but also in signal processing, based on motor-imagery brain-computer interface. recently, boczek and kaluszka [4] introduced the preliminary notion of the extended choquet-sugeno-like (ecs) operator, which generalizes most of the modifications of the ci known in the literature (briefly discussed above), and also generates some new ci-type operators. this inspired us to further research their properties and applications. then, the objectives of this paper are: (1) to study some important theoretical properties of an instance of the ecs-operator, introducing a generalization of cc-integrals by rdfs, called d-cc-integrals (sect. 3); (2) to apply d-cc-integrals in frbcss in an experimental study, where we compare the classification accuracy of different d-cc-integrals based on combinations of rdfs and copulas with the best results obtained by other approaches in the literature (sect. 4 and 5). additionally, sect. 2 recalls preliminary concepts and sect. 6 is the conclusion. 2 preliminaries consider \(n = \{1, \dots , n\}\), with \(n > 0\), and \(\textbf{x}=(x_1,\ldots , x_n)\). a function \(a :[0,1]^n\rightarrow [0,1]\) is an aggregation function if (a1) it is increasing, and (a2) \(a(0,\ldots , 0)=0\) and \(a(1,\ldots , 1)=1\). copulas are a special type of aggregation function that, in the context of the theory of metric spaces, link (2-dimensional) probability distribution functions to their 1-dimensional margins . a bivariate function \(c :[0,1]^2 \rightarrow [0,1]\) is a copula if, for all \(x,x',y, y' \in [0,1]\) with \( x \le x'\) and \(y \le y'\): (c1) \(c(x,y) + c(x',y') \ge c(x,y') + c(x', y)\); (c2) \(c(x,0) = c(0,x) = 0\); (c3) \(c(x,1) = c(1,x) = x\) [3]. proposition 1 [3]. for each copula c it holds that: (i) c is increasing; (ii) c satisfies the lipschitz property with constant 1, that is, for all \(x_1,x_2,y_1,y_2 \in [0,1]\), one has that \(\mid c(x_1,y_1) c(x_2,y_2) \mid \le \mid x_1 x_2 \mid + \mid y_1 y_2\mid \). table 1 shows examples of copulas, which are used in the rest of the paper. we divided them into three groups: t-norms [3], overlap functions [6], and copulas that are neither t-norms nor overlap functions. table 1. example of copulas used as the basis for the d-cc integralsfull size table a restricted dissimilarity function (rdf) [7] \(\delta : [0,1]^2 \rightarrow [0,1]\) is a function such that, for all \(x,y,z \in [0,1]\): (d1) \(\delta (x,y) = \delta (y,x)\); (d2) \(\delta (x,y) = 1\) if and only if \(\{x, y\} = \{0,1\}\); (d3) \(\delta (x,y) = 0\) if and only if \(x = y\); (d4) if \(x \le y \le z\), then \(\delta (x, y) \le \delta (x, z)\) and \(\delta (y, z) \le \delta (x, z)\). one convenient way of constructing rdfs is through automorphisms (strictly increasing bijections) of the unit interval: proposition 2 [7]. let \(\varphi _1, \varphi _2: [0,1] \rightarrow [0,1]\) be two automorphisms. then, the function \(\delta ^{\varphi _1, \varphi _2}: [0,1]^2 \rightarrow [0,1]\), given, for all \(x,y \in [0,1]\), by \(\delta ^{\varphi _1, \varphi _2}(x,y)=\varphi _1(|\varphi _2(x)-\varphi _2(y)|)\), is a restricted dissimilarity function. all functions from table 2 are examples of rdfs, constructed via proposition 2, which are considered in our experiments presented in sect. 5. table 2. examples of rdfs based on proposition 2full size table a function \(m :2^{n} \rightarrow [0,1]\) is a fuzzy measure (fm) [10] if, for all \(x,y \subseteq n\), it satisfies the following properties: (m 1) increasing: if \(x\subseteq y\), then \(m(x)\le m(y)\); (m 2) boundary conditions: \(m(\emptyset ) =0\) and \(m(n) =1\). an example of fm is the power measure \(m_{pm} :2^{n} \rightarrow [0,1]\), which is defined, for all \(x \subseteq n\), by \(m_{pm}(x) = \left( \frac{|x|}{n} \right) ^{q}\), where the exponent \(q > 0\) can be learned genetically from the data. it is the only fm considered in this paper, since it provides excellent results in classification problems, as discussed in [21]. the discrete ci [10] was generalized in many forms [4, 13], one of them, based on its expanded form, considers copulas instead of the product operation: definition 1 [22]. let \(m :2^{n} \rightarrow [0,1]\) be a fuzzy measure and \(c :[0,1]^2 \rightarrow [0,1]\) be a bivariate copula. the cc-integral is defined as a function \(\mathfrak {c}_{m}^c :[0,1]^n \rightarrow [0,1]\), given, for all \({\boldsymbol{x}} \in [0,1]^n\), by $$\begin{aligned} \mathfrak {c}_{m}^c ({\boldsymbol{x}}) = \sum _{i=1}^{n} c \left( x_{(i)}, m\left( a_{(i)} \right) \right) c \left( x_{(i-1)}, m\left( a_{(i)} \right) \right) , \end{aligned}$$ (1) where \(\left( x_{(1)}, \ldots , x_{(n)}\right) \) is an increasing permutation on the input x, that is, \(0 \le x_{(1)} \le \ldots \le x_{(n)}\), where \(x_{(0)} = 0\) and \(a_{(i)} = \{(i), \dots , (n) \}\) is the subset of indices corresponding to the \(n-i+1\) largest components of \(\boldsymbol{x}\). in another direction, the ci in its expanded form was generalized considering rdfs in the place of the subtraction operation: definition 2 [29]. the generalization of the ci expanded form by rdfs \(\delta :[0,1]^2 \rightarrow [0,1]\) with respect to an fm \(m :2^n \rightarrow [0,1]\), named d-xchoquet integral (d-xc), is a mapping \(x\mathfrak {c}_{\delta ,m} :[0,1]^2 \rightarrow [0,n]\), defined, for all \(\boldsymbol{x} \in [0,1]^n\), by: $$\begin{aligned} x\mathfrak {c}_{\delta , m} (\boldsymbol{x}) = x_{(1)} + \sum _{i=2}^{n} \delta \left( x_{(i)} \cdot m(a_{(i)}),\ x_{(i-1)} \cdot m(a_{(i)}) \right) , \end{aligned}$$ (2) where \( x_{(i)} \) and \(a_{(i)}\) are defined according to definition 1. 3 d-cc integrals in this section, we introduce the definition of d-cc-integrals, by combining the concepts of cc-integral (definition 1) and d-xc-integral (definition 2). following that, we study some aspects of d-cc-integrals that are relevant to our application in classification, such as different forms of monotonicity. definition 3 let \(c :[0,1]^2 \rightarrow [0,1]\) be a copula. the generalization of the cc-integral by rdfs \(\delta :[0,1]^2 \rightarrow [0,1]\) with respect to a fm \(m :2^n \rightarrow [0,1]\), named d-cc-integral (d-cc), is a mapping \(\mathfrak {cc}_{\delta ,m}^c :[0,1]^2 \rightarrow [0,n]\), defined by: $$\begin{aligned} \mathfrak {cc}_{\delta , m}^c (\boldsymbol{x}) = x_{(1)} + \sum _{i=2}^{n} \delta \left( c(x_{(i)},m(a_{(i)}), c(x_{(i-1)}, \ m(a_{(i)}) \right) , \end{aligned}$$ (3) for all \(\boldsymbol{x} \in [0,1]^n\), where the ordered \(x_{(i)}\), and \(m(a_{(i)})\), with \(0\le i\le n\), were stated in definition 1. from now on, we denote \(m(a_{(i)})\) simply by \(m_{(i)}\). observe that d-cc-integrals can also be obtained by generalizing x-dc-integrals (definition 2), where the product is replaced by a copula, or a restriction of \(c_{f_1f_2}\) [19], putting \(f_1=f_2 = c\). proposition 3 under the conditions given in definition 3, \(\mathfrak {cc}_{\delta ,m}^c\) is well defined, for all rdf \(\delta \), fm m and copula c. proof for \(i = 1, \ldots , n\), one has that \(0 \le x_{(i)} \le 1\), and for \(i = 2, \ldots , n\), we have that \( 0\le m_{(i)} \le 1\), \( 0\le c(x_{(i)},m_{(i)}) \le 1\) and \(0 \le \delta (c(x_{(i)},m_{(i)}),c(x_{(i-1)},m_{(i)})) \le 1\). it is immediate that, for \(\boldsymbol{x} \in [0,1]^n\), \(0 \le x\mathfrak {c}_{\delta ,m} (\boldsymbol{x}) \le n\), for any rdf \(\delta \), fm m and copula c. consider an input vector \(\boldsymbol{x} \in [0,1]^n\), for which there may be different increasing permutations, meaning that \(\boldsymbol{x} \) has repeated elements. for the sake of simplicity, but without loss of generality, consider that there exists \(r,s \in \{1, \ldots , n\}\) such that \(x_r = x_s = z \in [0,1]\) and, for all \(i \in \{1, \ldots , n\}\), with \(i \ne r,s\), it holds that \(x_i \ne x_r, x_s\). the only two possible increasing permutations are: $$\begin{aligned} (x_{(1)}, \ldots , x_{(k-1)} = x_r,& x_{(k)} = x_s, \ldots , x_{(n)}), \end{aligned}$$ (4) $$\begin{aligned} (x_{(1)}, \ldots , x_{(k-1)} = x_s,& x_{(k)} = x_r, \ldots , x_{(n)}) \end{aligned}$$ (5) denote by \(m_{(i)}^{(1)}= m^{(1)}(a_{(i)})\) and \(m_{(i)}^{(2)}=m^{(2)}(a_{(i)})\), with \(i \in \{1, \ldots , n\}\), the fuzzy measures of the subsets of \(a_{(i)}\) of indices corresponding to the \(n i + 1\) largest components of \(\boldsymbol{x}\) with respect to the permutations (4) and (5), respectively. observe that $$\begin{aligned} m_{(i)}^{(1)}& =& m_{(i)}^{(2)}, \text { for all } i \ne k, \text { and} \end{aligned}$$ (6) $$\begin{aligned} m_{(k)}^{(1)} = & {} m(\{s, (k+1), \ldots , (n)\}), \;\; m_{(k)}^{(2)} = m(\{r, (k+1), \ldots , (n)\}), \end{aligned}$$ (7) hold, meaning it may be the case that \( m_{(k)}^{(1)} \ne m_{(k)}^{(2)}\). now, denote by \(\mathfrak {cc}_{\delta , m}^c{(1)}\) and \(\mathfrak {cc}_{\delta , m}^c{(2)}\) the d-cc integrals with respect to the permutations (4) and (5), respectively, and suppose that $$\begin{aligned} \mathfrak {cc}_{\delta , m}^c {(1)}(\boldsymbol{x}) \ne \mathfrak {cc}_{\delta , m}^c{(2)}(\boldsymbol{x}) . \end{aligned}$$ (8) from eqs. (6) and (7), whenever \(k \ne 1\), it follows that: $$\begin{aligned} & \mathfrak {cc}_{\delta , m}^c{(1)} (\boldsymbol{x}) \mathfrak {cc}_{\delta , m}^c{(2)} (\boldsymbol{x}) \\ &= \delta \left( c( x_{(k)}, m_{(k)}^{(1)}),\ c(x_{(k-1)}, m_{(k)}^{(1)}) \right) \delta \left( c(x_{(k)}, m_{(k)}^{(2)}),\ c(x_{(k-1)}, m_{(k)}^{(2)}) \right) \\ &= \delta \bigl ( c(x_s, m(\{s, (k+1), \ldots , (n)\})), c(x_r, m(\{s, (k+1), \ldots , (n)\})) \bigr ) \\ & \quad \delta \bigl ( c(x_r,m(\{r, (k+1), \ldots , (n)\})), c( x_s, m (\{r, (k+1), \ldots , (n)\})) \bigr ) \\ &= \delta \bigl ( c(z, m(\{s, (k+1), {\ldots }, (n)\})), c(z, m(\{s, (k+1), {\ldots }, (n)\})) \bigr ) \\ & \delta \bigl ( c(z, m(\{r, (k+1), {\ldots }, (n)\})), c(z, m(\{r, (k+1), {\ldots }, (n)\}) \bigr )) = 0, \quad \text {by (d3),} \end{aligned}$$ which is in contradiction to eq. (8). similarly, there is also a contradiction for \(k = 1\). the result can be easily generalized for any subsets of repeated elements in the input \(\boldsymbol{x}\). the conclusion is that for any different increasing permutations of the same input \(\boldsymbol{x}\), one always obtains the same output value of \(\mathfrak {cc}_{\delta , m}^c (\boldsymbol{x})\). remark 1 note that, in \(\mathfrak {cc}_{\delta ,m}^c \) definition, the summation first element is just \(x_{(1)}\) instead of \(\delta \left( c(x_{(1)}, m_{(1)}),\ c(x_{(0)},m_{(1)} )\right) \). according to [19, 29], this can be used to avoid a discrepant behavior of non-averagingfootnote 1 functions in the initial phase of the aggregation process. example 1 it is immediate that any choices of rdf \(\delta \) from table 2 and copula c from table 1 can be combined to obtain an example of d-cc-integral \(\mathfrak {cc}_{\delta , m}^c\). in the following, we first study some general properties and then monotonicity-like properties, for the rdfs of table 2, any fuzzy measure m and copula c. 3.1 some important properties for aggregation-like processes the next proposition shows that whenever the adopted rdf is constructed using proposition 2, then, for some choices of the automorphism \(\varphi _2\), the result obtained is upper limited in relation to the dissimilarities (by \(\delta _0\)) of the inputs: proposition 4 let \(\delta \) be an rdf constructed by proposition 2, for any automorphism \(\varphi _1\) and the identity as \(\varphi _2\). let \(\mathfrak {cc}_{\delta , m}^c :[0, 1]^n \rightarrow [0,1]\) be the derived d-cc integral for any fuzzy measure \(m\) and copula c. then, for all \(\boldsymbol{x} \in [0,1]^n\), it holds that: $$\begin{aligned} \mathfrak {cc}_{\delta , m}^c(\boldsymbol{x}) \le x_{(1)} + \sum _{i=2}^{n} \varphi _1 \left( x_{(i)} x_{(i-1)} \right) = x_{(1)} + \sum _{i=2}^{n} \varphi _1 \left( \delta _0 (x_{(i)},x_{(i-1)}) \right) . \end{aligned}$$ proof let \(\delta \) be an rdf constructed by proposition 2. then, since \(\varphi _1\) is an automorphism, for any \(\boldsymbol{x} \in [0,1]\), fm \(m\) and copula c, we have that: $$\begin{aligned} \mathfrak {cc}_{\delta , m}^c(\boldsymbol{x}) &= x_{(1)} + \sum _{i=2}^{n} \delta ( c(x_{(i)}, m_{(i)}), c(x_{(i-1)}, m_{(i)})) \\ &= x_{(1)} + \sum _{i=2}^{n} \varphi _1 \left( c(x_{(i)}, m_{(i)}) c(x_{(i-1)}, m_{(i)})\right) \text {by props. 2 and 1(i)} \\ & \le x_{(1)} + \sum _{i=2}^{n} \varphi _1 \left( x_{(i)} x_{(i-1)} \right) \text { by prop. 1 (ii)} \end{aligned}$$ the following three results are immediate: proposition 5 for \(\delta \), m and c, \(\mathfrak {cc}_{\delta ,m}^c (\boldsymbol{x}) \ge \min (\boldsymbol{x})\), for all \(\boldsymbol{x} \in [0,1]\). proposition 6 \(\mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x}) \le \max (\boldsymbol{x})\) if and only if, for all \(0 \le a_1 \le \ldots \le a_n \) and fm \(m\), the rdf \(\delta \) satisfies: \( \sum _{i=2}^n \delta (c(a_i, m_{i}),\ c(a_{i-1}, m_{i})) \le a_n a_1, \) where \(m_i = m(a_{i})\), for \(a_i = \{i, \ldots , n\}\). corollary 1 \(\mathfrak {cc}_{\delta ,m}^c\) is averaging if and only if it satisfies proposition 6. proposition 7 for any rdf \(\delta \), fm m and copula c, \(\mathfrak {cc}_{\delta ,m}^c\) is idempotent. proof if \(\boldsymbol{x} = (x, \ldots , x)\), then, by (d3), one has that: $$\begin{aligned} \mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x}) = x + \sum _{i=2}^{n} \delta \left( c(x,m_{(i)}), c(x, m_{(i)})\right) = x + 0 = x. \end{aligned}$$ since the range of the dcc-integral is [0, n], we avoid the term “boundary condition” when referring to condition (a2) in this context. instead, we simply call it 0, 1-condition. the same term was also adopted in [29]. from proposition 7, the following result is immediate: corollary 2 the d-cc integral satisfies the 0, 1-condition for any m, \(\delta \) and c. 3.2 monotonicity-like properties of d-cc integrals as explained in [13], one important property of aggregation-like operators is to present some kind of “increasingness property” to guarantee that the more information is provided the higher the aggregated value is in the considered direction. from the discussions in [29], it is immediate that d-cc integrals can not be fully monotonic, in general, since they have to satisfy the following conditions: (i) for \(z_1, z_2, z_3, z_4 \in [0,1]\), with \(z_1 \le z_2 \le z_3 \le z_4\), \(w_1, w_2 \in [0,1]\), with \(w_1 \ge w_2\), we have: \(\delta (c(z_1, w_1), c(z_3, w_1)) + \delta (c(z_3, w_2), c(z_4, w_2)) \ge \delta (c(z_1,w_1), c(z_2, w_1))\) \(+ \delta (c(z_2, w_2), c(z_4, w_2))\); (ii) for all \(z_1, z_2, z_3 \in [0, 1]\), with \(z_1 \le z_2 \le z_3\), \(w \in [0,1]\), we have: \(z_2 + \delta (c(z_3, w) , c(z_2, w)) \ge z_1 + \delta (c(z_3, w), c(z_1, w)).\) considering the rdfs of table 2, for any copula c and fm m, only \(\mathfrak {cc}_{\delta _0,m}^c\) is increasing, since it satisfies both conditions. observe that \(\mathfrak {cc}_{\delta _0,m}^c\) is, in fact, a cc-integral [22], which is fully monotonic, for any copula c. when c is the product, both coincide with the d-xc integral (definition 2) for \(\delta _0\), which is the standard ci in the expanded form. nevertheless, d-cc integrals do satisfy weaker forms of monotonicity, as we discuss in this section. directional monotonicity of d-cc integrals. directional monotonicity is one of the most adopted weaker notions of monotonicity, which enlarged the scope of aggregation processes in applications (e.g., [13, 18, 24, 25]). definition 4 [5]. let \(\boldsymbol{r} = (r_1, \dots , r_n)\) be a real \(n\)-dimensional vector such that \(\boldsymbol{r} \ne \boldsymbol{0} = (0, \ldots , 0)\). a function \(f :[0,1]^n \rightarrow [0,1]\) is said to be \(\boldsymbol{r}\)-increasing if, for all \(\boldsymbol{x}= (x_1,\dots , x_n) \in [0,1]^n\) and \(c > 0\) such that \(\boldsymbol{x} + c{\textbf {r}} = (x_1 + c r_1, \dots , x_n + c r_n) \in [0,1]^n\), it holds that \(f(\boldsymbol{x} + c\boldsymbol{r}) \ge f(\boldsymbol{x})\). theorem 1 let \(m :2^n \rightarrow [0,1]\), \(\delta :[0,1]^2 \rightarrow [0,1]\) and \(c :[0,1]^2 \rightarrow [0,1]\) be an fm, an rdf, and a copula, respectively. \(\mathfrak {cc}_{\delta ,m}^c\) is \(\boldsymbol{1}\)-increasing if and only if one of the following conditions hold: (i) the rdf \(\delta \) is \(\boldsymbol{1}\)-increasing; (ii) for all \(0 \le z_1 \le \ldots \le z_n \le 1\) and \(c>0\), such that \( z_i + c \in [0,1]\), for all \(i = 1, \ldots , n\): $$\begin{aligned} &\sum _{i=2}^{n} \delta (c(z_i + c, m_{i}), c(z_{i-1}+ c, \cdot m_{i})) \ge \sum _{i=2}^{n}\delta ( c(z_i \cdot m_{i}), c(z_{i-1} \cdot m_{i})) c, \end{aligned}$$ where \(m_i = m(a_{i})\), for \(a_i = \{i, \ldots , n\}\). proof (\(\leftarrow \))(i) suppose that \(\delta \) is \(\boldsymbol{1}\)-increasing and let \(\boldsymbol{c} = (c, \ldots , c)\), \(c>0\), such that \(\boldsymbol{x}, \boldsymbol{x} + \boldsymbol{c}\in [0,1]^n\). then: $$\begin{aligned} & \mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x} + c \boldsymbol{1} ) = (x_{(1)} + c) + \sum _{i=2}^{n} \delta \left( c (x_{(i)} + c, m_{(i)}), c (x_{(i-1)} + c, m_{(i)}) \right) \\ & \qquad \qquad \quad > x_{(1)} + \sum _{i=2}^{n} \delta \left( c(x_{(i)}, m_{(i)}), c(x_{(i-1)}, m_{(i)}) \right) = \mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x}). \end{aligned}$$ now suppose that (ii) holds. then, for all \( \boldsymbol{x} \in [0,1]^n\) and \(\boldsymbol{c} = (c, \ldots , c)\), with \(c > 0\), such that \(x_i + c \in [0,1]\), \(\forall i = 1, \ldots , n\), it follows that \(\sum _{i=2}^{n} \delta (c(x_{(i)} + c, m_{(i)}), c(x_{(i-1)} + c, m_{(i)})) \ge \sum _{i=2}^{n}\delta (c(x_{(i)}, m_{(i)}), c( x_{(i-1)}, m_{(i)})) c\). this implies that \((x_{(1)} + c) + \sum _{i=2}^{n} \delta (c(x_{(i)} + c, m_{(i)}), c(x_{(i-1)} + c, m_{(i)})) \ge x_{(1)} + \sum _{i=2}^{n}\delta (c(x_{(i)}, m_{(i)}), c( x_{(i-1)}, m_{(i)}))\). then, \(\mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x} + \boldsymbol{c}) \ge \mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x})\). therefore, if (i) or (ii) holds, then \(\mathfrak {cc}_{\delta ,m}^c\) is \(\boldsymbol{1}\)-increasing. (\(\rightarrow \)) suppose that \(\mathfrak {cc}_{\delta ,m}^c\) is \(\boldsymbol{1}\)-increasing, that is \(\mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x} + \boldsymbol{c}) \ge \mathfrak {cc}^c_{\delta ,m}(\boldsymbol{x})\). then it follows that \((x_{(1)} + c) + \sum _{i=2}^{n} \delta ( c(x_{(i)} + c, m_{(i)}), c (x_{(i-1)} + c, m_{(i)})) \) \(\ge x_{(1)} + \sum _{i=2}^{n}\delta ( c(x_{(i)},m_{(i)}), c(x_{(i-1)}, m_{(i)}))\) which implies that condition (ii) holds. ordered directional monotonicity of d-cc integrals. any d-cc integral is ordered directionally monotonic [9]. such functions are monotonic along different directions according to the ordinal size of the coordinates of each input. definition 5 [9]. consider a function \(f :[0, 1]^n \rightarrow [0, 1]\) and let \(\boldsymbol{r}=(r_1,\dots ,r_n)\) be a real n-dimensional vector, \(\boldsymbol{r} \ne \boldsymbol{0}\). f is said to be ordered directionally (od) \(\boldsymbol{r}\)-increasing if, for each \(\boldsymbol{x} \in [0, 1]^n\), any permutation \(\sigma : \{1, \ldots , n\} \rightarrow \{1, \ldots , n\}\) with \(x_{\sigma (1)} \ge \ldots \ge x_{\sigma (n)}\), and \(c>0\) such that \(1 \ge x_{\sigma (1)} + cr_1 \ge \ldots \ge x_{\sigma (n)} + cr_n\), it holds that \(f(\boldsymbol{x} + c \boldsymbol{r}_{\sigma ^{-1}} ) \ge f(\boldsymbol{x})\), where \(\boldsymbol{r}_{\sigma ^{-1}} = (r_{\sigma ^{-1}(1)}, \ldots , r_{\sigma ^{-1}(n)})\). theorem 2 for any fm m, rdf \(\delta \), copula c and \(k>0\), the d-cc integral \(\mathfrak {cc}_{\delta ,m}^c\) is an (od) \((k,0, \ldots , 0)\)-increasing function. proof for all \(\boldsymbol{x} \in [0, 1]^n\) and permutation \(\sigma :\{1, \ldots , n\} \rightarrow \{1, \ldots , n\}\), with \(x_{\sigma (1)} \ge \ldots \ge x_{\sigma (n)}\), and \(c>0\) s.t. \(x_{\sigma (i)} + cr_i \in [0,1]\), for \(i \in \{1, \ldots , n\}\), and \(1 \ge x_{\sigma (1)} + cr_1 {\ge } {\ldots } {\ge } x_{\sigma (n)} + cr_n\), for \(\boldsymbol{r}_{\sigma ^{-1}} = (r_{\sigma ^{-1}(1)}, {\ldots }, r_{\sigma ^{-1}(n)})\), one has that: $$\begin{aligned} & x\mathfrak {c}_{\delta ,m}^c(\boldsymbol{x}+ c\boldsymbol{r}_{\sigma ^{-1}} )= x_{(1)} + c\cdot r_{\sigma ^{-1}(1)} \\ &\quad + \sum _{i=2}^{n-1} \delta ( c( (x_{(i)} + c\cdot r_{\sigma ^{-1}(i)}), m_{(i)}), c((x_{(i-1)} + c\cdot r_{\sigma ^{-1}(i-1)}), m_{(i)}))\\ &\quad + \delta (c( (x_{(n)} + c\cdot r_{\sigma ^{-1}(n)}) , m_{(n)}), c( (x_{(n-1)} + c\cdot r_{\sigma ^{-1}(n-1)}), m_{(n)}))\\ &= x_{(1)} + c\cdot 0 + \sum _{i=2}^{n-1} \delta (c((x_{(i)} + c\cdot 0), m_{(i)}), c((x_{(i-1)} + c\cdot 0), m_{(i)})) \\ &\quad +\delta (c( (x_{(n)} + c\cdot k) , m_{(n)}), c( (x_{(n-1)} + c\cdot 0), m_{(n)}))\\ &= x_{(1)} + \sum _{i=2}^{n-1} \delta (c(x_{(i)}, m_{(i)}), c(x_{(i-1)}, m_{(i)})) \\ &\quad +\delta (c( (x_{(n)} + c\cdot k) , m_{(n)}), c( x_{(n-1)}, m_{(n)}))\\ &\ge x_{(1)} + \sum _{i=2}^{n-1} \delta ( c(x_{(i)}, m_{(i)}), c(x_{(i-1)}, m_{(i)}) + \delta (c(x_{(n)}, m_{(n)}), c(x_{(n-1)}, m_{(n)})) \\ {} & =\mathfrak {cc}_{\delta ,m}^c(\boldsymbol{x}) \qquad \qquad \qquad \qquad \qquad \qquad \qquad \text {by prop. 1-(i) and (d4)}\\ \end{aligned}$$ 4 application of d-cc integrals in classification a classification problem consists of p training examples \(\vec {x_p} = (x_{p1},\ldots ,x_{pn})\), \(p \in \{1,\ldots ,p\}\) where \(x_{pi}\) is the value of the i-th variable of the p-th example and each example belongs to one of m classes in \(c {=} \{c_1, \ldots , c_m\}\). the goal of the learned classifier is to identify the class of new/unknown examples. a frbcs [17] is a type of classification system that is based on rules with linguistic labels, modeled by fuzzy sets. the inference process, known as the fuzzy reasoning method (frm), is determined by four sequential steps: (1) matching degree (\({a_j}\)) it measures, for the example \(x_p\) to be classified, the strength of the if-part of a rule \(r_j\). \({a_j}\) is calculated by a fuzzy conjunction operator \(\mathfrak {c}\), as follows: \({a_j}(x_p) = \mathfrak {c}(a_{j1}(x_{p1}),\cdots , a_{jn}(x_{pn}))\). (2) association degree (\(b_j^k\)) it weights the matching degree \({a_j}\) by the rule weight \(rw_j\), through a product operation: (3) example classification soundness degree for all classes (\(y_k\)) for each class \(c_k \in c\), we aggregate all the positive association degrees \(b_j^k\) that were obtained in the previous step with respect to \(c_k\), through an aggregation function a: \(y_k = a(b_j^k)\), with \(j = 1, \ldots , l\), and \(b_j^k>0\). (4) classification the final decision is made in this step. for that, a function \(f: [0,1]^m \rightarrow c\) is applied over all example classification soundness degrees calculated in the previous step: \(f(y_1,\ldots ,y_m)= \arg \max \limits _{k=1,\ldots ,m}(y_k)\). in this paper, we will apply d-cc integrals as the aggregation operator a in the third step of the frm to analyze their effect on the classification process.footnote 2 4.1 experimental framework the application of our new family of functions is based on a benchmark composed of 33 different public datasets found in keel [2] dataset repositoryfootnote 3. it is important to mention that this benchmark has been used for all the most important generalizations of the ci discussed before. the selection of the same datasets and partitions allows us to directly compare the methods and consequently provide a more complete analysis. in table 3, we provide the information of the datasets along with their identification (id), number of instances (#noi), number of attributes (#noa), and the total of classes (#toc). table 3. summary of the considered datasets in the experimental study.full size table the experimental framework lies in the same context as other works in the literature (see [19, 20, 23]), i.e. a 5-fold cross validation approach. consequently, the results provided in sect. 5 are the accuracy mean (am) related to these folds. moreover, following the same approach seen in [22], the hyperparameters used by the model are the same originally performed by farc-hd [1]. 5 analysis of the experimental results this section explores the application of d-cc integrals. so, it is presented the performance of d-cc integrals over the datasets to achieve it. then, the best function is pointed out, and we compare it against other operators found in the literature to highlight the best-performing method efficiency. 5.1 the performance of the d-cc integrals in classification this section provides the results obtained by the application of the d-cc integrals in the frm. to ease the comprehension, in table 4, we present the combination of the rdf and d-cc integral per dataset that led to the largest accuracy. moreover, we show in table 5 the mean accuracy for our approach considering the 33 different datasetsfootnote 4 in order to provide a better analysis, we also highlight, per rdf, the largest mean in boldface. table 4. the best-achieved results in a test for each dataset.full size table table 5. obtained accuracy mean in all datasets.full size table the combination of the rdf \(\delta _5\) and copula \(o_{mm}\) achieves the largest am in the study. moreover, considering \(\delta _5\), for more than half of the considered d-cc integrals, the am is superior to 80%. the rdf \(\delta _2\) is the next one to achieve satisfactory performance. in fact, only for two copulas, \(t_m\), and \(t_l\), this rdf presents an am inferior to 80%. observe that \(\delta _5\) and \(\delta _2\) also performed well in [27, 28]. additionally, both \(\delta _0\) and \(\delta _3\) presented similar ams, around 79%. on the other hand, the remaining cases (\(\delta _1\) and \(\delta _4\)) presented the worst scenarios in this study, with ams around 78%. up to this point, the superiority of \(\delta _2\) and \(\delta _5\) among the rdfs can be perceived. however, to complete the analysis we have conducted a set of statistical tests to reinforce this conclusion. considering that the conditions necessary to perform parametric tests are not fulfilled, we considered non-parametric statistical tests. hence, the analysis was made by a group comparison using the align-friedman rank test [14], and the results were analyzed in terms of the adjusted p-value (apv) computed by holm’s post hoc test [15]. the analysis was done by performing a statistical test per rdf, with a local statistical analysis among the different functions. after that, we chose the control variables of each group, the ones having the lowest obtained rank, and once again we statistically compared them. this approach allowed us to point out the best d-cc integral, used to compare against other aggregation operators. the results, in table 6, sort the methods by ranks per rdf. the last column shows a comparison among the selected control variables. if the obtained apv is smaller than 0.1 (10% of significance level), it means the methods are statistically different and, so, we underline them. among the groups of rdfs, in general, the number of statistical differences are low. the test rejected the null hypothesis when comparing the control variable against three different approaches for \(\delta _4\), two for \(\delta _0\) and \(\delta _2\), one for \(\delta _5\) and none for \(\delta _1\) and \(\delta _2\). table 6. align friedman rank test and holm’s post hoc test for the d-cc integrals.full size table the last column performs a group comparison among the control variables of each family based on an rdf. see that the d-cc integral with \(\delta _5\) and \(o_{mm}\), when compared against almost all approaches, is statically superior. but this does not hold when \(\delta _2\) and \(o_{\alpha }\) are considered. finally, \(\delta _5-o_{mm}\) provides the winning combination in this first analysis and so, it is the representing method of the class of d-cc integrals. 5.2 comparing the best d-cc integral against classical operators this subsection states the position of the d-cc integrals among the generalizations of the ci as well as known frms found in the literature. we have selected averaging and non-averaging operators: averaging operators: [wr] (winning rule [11], based on the maximum as aggregation function), [cho] (ci), [cc] (the best cc-integral [22]), [\(c_t\)] (the best \(c_t\)-integral, a ci generalization by t-norms [20]), [\(c^f_{avg}\)] (the best averaging \(c_f\)-integral [23]), [d-cf] (the best averaging d-cf integral [28]). non-averaging functions: [ac] (the additive combination [12], based on the normalized sum, farc-hd algorithm basis), [ps] (the probabilistic sum [12]), [\(c_{f1f2}\)] (the \(c_{f1f2}\)-integral [19], being the state-of-the-art), [\(c^f_{n-avg}\)] (the best on-averaging \(c_f\)-integral [23]). the obtained results are in table 7, detailed per dataset. we highlight in boldface the largest obtained am among all results. table 7. comparing the best d-cc integral with other operators in the literature.full size table we observed that the \(c_{f1f2}\)-integral is the approach having the largest am in the study. however, the d-cc integral (our new approach) is the second one, followed by the \(c_f\) and the d-cf integrals. it is also noticeable that the d-cc integral achieves a superior mean than the classical frms of ac and ps. considering the methods that present averaging characteristics and their obtained am, it is easy to see that their performance is, in general, inferior to the non-averaging ones. the accuracy in these cases is around 79% which is inferior to the best d-cc integral, with the exception of the d-cf integral. once again, we performed a set of analyses to statistically compare the approaches. therefore, we have compared the best d-cc integral (based on omm and \(\delta _5\)) against the averaging and non-averaging approaches. the results are presented in table 8, following the same structure used in the previous analysis. table 8. statistical analysis comparing the d-cc integral against classical operators.full size table regarding the results obtained when comparing the d-cc integral against the averaging approaches, we see that it is statistically superior to all other approaches, with the exception of the d-cf integral, considered equivalent. it is noteworthy that the cc-integral is among the surpassed methods, pointing out that its generalization by rfds resulted in better performances. if the non-averaging approaches are considered, it is noticeable that our new method is the only one that is able to be considered statistically equivalent to the state-of-the-art. for all the other cases, the \(c_{f1f2}\)-integral is still superior. 6 conclusions new generalizations of the discrete ci have been developed in recent years, by generalizing the product and/or the difference operations from the original expression or its expanded form. the study of those generalizations has proven to be worthwhile since they can provide promising results in applications. in this paper, we continue this trend by focusing on the generalization of the cc-integral by means of rdfs, calling them d-cc-integrals. then, we studied some relevant properties for application purposes. the final part of our work applied the d-cc-integrals in a classification problem, followed by a two-step analysis. first, we compared different d-cc-integrals and identified that the one based on the copula omm and the rdf \(\delta _5\) had the best overall performance. then, we compared this particular d-cc-integral with other methods from the literature, showing that it is statistically equivalent to the state-of-the-art \(c_{f1f2}\)-integral and surpasses most of the other considered approaches, including the cc-integral. notes 1.a function f is said to be averaging if \(min \le f \le max\). 2.for more information about this approach see [19, 20, 23]. 3.https://keel.es. 4.to analyze the particular cases and results per fold please check the folowing link https://github.com/giancarlo-lucca/d-cc-integrals-generalizing-cc-integrals-by-restricted-dissimilarity-functions. references alcala-fdez, j., alcala, r., herrera, f.: a fuzzy association rule-based classification model for high-dimensional problems with genetic rule selection and lateral tuning. ieee trans. fuzzy syst. 19(5), 857–872 (2011) article  google scholar  alcalá-fdez, j., et al.: keel: a software tool to assess evolutionary algorithms for data mining problems. soft. comput. 13(3), 307–318 (2009) article  google scholar  alsina, c., frank, m.j., schweizer, b.: associative functions: triangular norms and copulas. world scientific publishing company, singapore (2006) book  math  google scholar  boczek, m., kaluszka, m.: on the extended choquet-sugeno-like operator. int. j. approx. reason. 154, 48–55 (2023) article  mathscinet  math  google scholar  bustince, h., fernandez, j., kolesárová, a., mesiar, r.: directional monotonicity of fusion functions. eur. j. oper. res. 244(1), 300–308 (2015) article  mathscinet  math  google scholar  bustince, h., fernandez, j., mesiar, r., montero, j., orduna, r.: overlap functions. nonlinear anal. theory methods app. 72(3–4), 1488–1499 (2010) article  mathscinet  math  google scholar  bustince, h., jurio, a., pradera, a., mesiar, r., beliakov, g.: generalization of the weighted voting method using penalty functions constructed via faithful restricted dissimilarity functions. eur. j. oper. res. 225(3), 472–478 (2013) article  mathscinet  math  google scholar  bustince, h., et al.: d-choquet integrals: choquet integrals based on dissimilarities. fuzzy sets syst. (2020) google scholar  bustince, h., et al.: ordered directionally monotone functions: justification and application. ieee trans. fuzzy syst. 26(4), 2237–2250 (2018) article  google scholar  choquet, g.: theory of capacities. institut fourier 5, 131–295 (1953–1954) google scholar  cordón, o., del jesus, m.j., herrera, f.: a proposal on reasoning methods in fuzzy rule-based classification systems. int. j. approx. reason. 20(1), 21–45 (1999) article  google scholar  cordon, o., del jesus, m.j., herrera, f.: analyzing the reasoning mechanisms in fuzzy rule based classification systems. math. soft comput. 5(2–3), 321–332 (1998) google scholar  dimuro, g.p., et al.: the state-of-art of the generalizations of the choquet integral: from aggregation and pre-aggregation to ordered directionally monotone functions. inf. fusion 57, 27–43 (2020) article  google scholar  hodges, j.l., lehmann, e.l.: ranks methods for combination of independent experiments in analysis of variance. ann. math. stat. 33, 482–497 (1962) article  mathscinet  math  google scholar  holm, s.: a simple sequentially rejective multiple test procedure. scand. j. stat. 6, 65–70 (1979) mathscinet  math  google scholar  ishibuchi, h., nakashima, t.: effect of rule weights in fuzzy rule-based classification systems. fuzzy syst. ieee trans. 9(4), 506–515 (2001) article  google scholar  ishibuchi, h., nakashima, t., nii, m.: classification and modeling with linguistic information granules, advanced approaches to linguistic data mining. advanced information processing. springer, berlin, heidelberg (2005). https://doi.org/10.1007/b138232 ko, l., et al.: multimodal fuzzy fusion for enhancing the motor-imagery-based brain computer interface. ieee comput. intell. mag. 14(1), 96–106 (2019) article  google scholar  lucca, g., dimuro, g.p., fernandez, j., bustince, h., bedregal, b., sanz, j.a.: improving the performance of fuzzy rule-based classification systems based on a nonaveraging generalization of cc-integrals named \(c_{f_1f_2}\)-integrals. ieee trans. fuzzy syst. 27(1), 124–134 (2019) article  google scholar  lucca, g., sanz, j., pereira dimuro, g., bedregal, b., mesiar, r., kolesárová, a., bustince sola, h.: pre-aggregation functions: construction and an application. ieee trans. fuzzy syst. 24(2), 260–272 (2016) article  math  google scholar  lucca, g., sanz, j.a., dimuro, g.p., borges, e.n., santos, h., bustince, h.: analyzing the performance of different fuzzy measures with generalizations of the choquet integral in classification problems. in: 2019 fuzz-ieee, pp. 1–6 (2019) google scholar  lucca, g., et al.: cc-integrals: choquet-like copula-based aggregation functions and its application in fuzzy rule-based classification systems. kbs 119, 32–43 (2017) google scholar  lucca, g., sanz, j.a., dimuro, g.p., bedregal, b., bustince, h., mesiar, r.: cf-integrals: a new family of pre-aggregation functions with application to fuzzy rule-based classification systems. inf. sci. 435, 94–110 (2018) article  math  google scholar  marco-detchart, c., lucca, g., lopez-molina, c., de miguel, l., pereira dimuro, g., bustince, h.: neuro-inspired edge feature fusion using choquet integrals. inf. sci. 581, 740–754 (2021) article  google scholar  mesiar, r., kolesárová, a., bustince, h., dimuro, g., bedregal, b.: fusion functions based discrete choquet-like integrals. ejor 252(2), 601–609 (2016) article  mathscinet  math  google scholar  sanz, j.a., bernardo, d., herrera, f., bustince, h., hagras, h.: a compact evolutionary interval-valued fuzzy rule-based classification system for the modeling and prediction of real-world financial applications with imbalanced data. ieee trans. fuzzy syst. 23(4), 973–990 (2015) article  google scholar  wieczynski, j., et al.: applying d-xchoquet integrals in classification problems. in: 2022 ieee international conference on fuzzy systems (fuzz-ieee), pp. 1–7 (2022) google scholar  wieczynski, j., et al.: \(dc_{f}\)-integrals: generalizing c\(_{f}\)-integrals by means of restricted dissimilarity functions. ieee tfs 31(1), 160–173 (2023) google scholar  wieczynski, j.c., et al.: d-xc integrals: on the generalization of the expanded form of the choquet integral by restricted dissimilarity functions and their applications. ieee tfs 30(12), 5376–5389 (2022) google scholar  download references author information authors and affiliations centro de ciêcias computacionais, univ. fed. do rio grande, porto alegre, brazil joelson sartori, giancarlo lucca, helida santos, eduardo borges & graçaliz pereira dimuro instituto de matemática, estatística e física, univ. fed. do rio grande, porto alegre, brazil tiago asmus departamento de informática e matemática aplicada, univ. fed. do rio grande do norte, natal, brazil benjamin bedregal dept. estadistica, informa. y matem., univ. publica de navarra, pamplona, spain humberto bustince authors joelson sartoriview author publications search author on:pubmed google scholar giancarlo luccaview author publications search author on:pubmed google scholar tiago asmusview author publications search author on:pubmed google scholar helida santosview author publications search author on:pubmed google scholar eduardo borgesview author publications search author on:pubmed google scholar benjamin bedregalview author publications search author on:pubmed google scholar humberto bustinceview author publications search author on:pubmed google scholar graçaliz pereira dimuroview author publications search author on:pubmed google scholar corresponding author correspondence to giancarlo lucca . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper sartori, j. et al. (2023). d-cc integrals: generalizing cc-integrals by restricted dissimilarity functions with applications to fuzzy-rule based systems. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_16 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_16 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords choquet integral cc-integral x-dc-integral fuzzy-rule based classifification systems publish with us policies and ethics profiles joelson sartori view author profile giancarlo lucca view author profile helida santos view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature multi-label classification of pathologies in chest radiograph images using densenet | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper multi-label classification of pathologies in chest radiograph images using densenet conference paper first online: 12 october 2023 pp 167–180 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) multi-label classification of pathologies in chest radiograph images using densenet download book pdf download book epub alison corrêa mendes  orcid: orcid.org/0009-0004-3687-55199, alexandre césar pinto pessoa  orcid: orcid.org/0000-0003-4995-89099 & anselmo cardoso de paiva  orcid: orcid.org/0000-0003-4921-06269  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 562 accesses 1 citation abstract chest radiography exams are still one of the main methods for detecting and diagnosing certain thoracic pathologies. this study evaluates the performance of a densenet in a multi-label classification task on radiography images, using focal loss as the loss function to address the class imbalance problem. for the experiments, 14 different types of findings were considered. satisfactory results were obtained using the area under the roc curve (auc-roc) as the metric, where the average performance across all classes was 0.861. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others utilization of deep convolutional neural networks for accurate chest x-ray diagnosis and disease detection article 26 march 2023 automatic classification and reporting of multiple common thorax diseases using chest radiographs chapter © 2019 multi-label convolutional neural network for multiple diseases detection in chest radiographs chapter © 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization computer vision machine learning non-small-cell lung cancer radiology radiography 1 introduction chest radiography is one of the most commonly used medical examinations for detection and diagnosis of thoracic pathologies. in brazil, this examination is one of the oldest, cheapest, and most widely available [15]. however, there is a need for methods to assist radiology professionals since the analysis of thoracic abnormalities through radiographic images is subjective and can vary depending on the experience and perception of the physician [11]. artificial neural networks (anns) are computational systems inspired by the functioning of the human brain, where hundreds of billions of interconnected neurons process information in parallel [16]. deep neural networks (dnns) are anns that have multiple hidden layers between their input and output [12]. these networks have been widely used and have achieved good results in the classification of medical images, especially convolutional neural networks (cnns), a class of ann commonly applied to image analysis. chexnet, for instance, has 121 layers and was trained using a dataset of 100,000 frontal chest radiographs, and outperformed the performance of four radiologists [13]. deep learning-based techniques are widely employed in image classification tasks, due to their ability to automatically learn discriminative features from raw data [9]. one of the advantages of deep learning over other techniques is that it minimizes the need for preprocessing, segmentation, and feature extraction [14]. the application of hyperparameter optimization methods in deep neural networks to analyze the network configuration mechanisms by capturing specific problem characteristics, enhances the discriminative capability of the produced techniques. this work evaluated the performance of a dense convolutional neural network (densenet) using the focal loss function in a multi-label classification task on radiographic images. 2 related works the search for methods to aid in the detection and diagnosis of pathologies from x-ray images has attracted the attention of many researchers worldwide, with several published works presenting promising results. however, obtaining a highly accurate and fully automated cnn-based method for medical diagnosis is still a challenging task. wang et al. [17] presented the chestx-ray8, a dataset with 108,948 chest x-ray images from 32,717 unique patients, where each image can have multiple labels. in this study, a quantitative benchmarking of the performance of 4 pre-trained models in the classification of the 8 thoracic pathologies in this dataset was conducted, with the resnet-50-based model [6] achieving the best results. later, this dataset was expanded to chestx-ray14, including 6 additional common thoracic pathologies. yao et al. [18] utilized an architecture that learns at multiple resolutions while generating weakly supervised saliency maps. the study also parameterized the lse-lba pooling function (long-sum-exp with learnable lower-bounded adaptation). the average auc for the 14 labels in the chestx-ray14 dataset in this work was 0.761. bhusal et al. [2] implemented a model using densenet and a modified cross-entropy loss to handle class imbalance, using specific weights for each class. the highest area under the roc curve (auc) was achieved for the cardiomegaly finding, with a value of 0.896, and the lowest auc for the nodule finding with a score of 0.655. the average auc for the 14 labels in the chestx-ray14 dataset in this work was 0.762. rajpurkar et al. [13] developed chesxnet, a 121-layer cnn trained on the chestx-ray14 dataset. rajpurkar et al. [13] compared the performance of the chesxnet with four radiologists using the f1 metric, where the network outperformed the average score of the radiologists. zhao et al. [20] proposed a model called am_densenet, which employs a dense connection network with an attention module after each dense block to optimize the feature extraction capability of the model. they used focal loss to address the class imbalance problem. the average auc for the 14 pathology labels in the chestx-ray14 dataset in this work was 0.8537. in this work, the utilization of densenet-121 combined with focal loss (fl) addresses the class imbalance in the classification of thoracic pathologies in the chestx-ray14 and chexpert datasets. in contrast to some previous approaches, we opted to preserve the original densenet-121 architecture and instead performed fine-tuning and optimization of the hyperparameters of the focal loss function. experiments were conducted comparing the results obtained with fl and cross entropy loss, allowing for the evaluation of the benefits of fl and highlighting its effectiveness in improving the model’s performance in the context of this task. 3 background 3.1 densenet densenet is a network that connects each of its layers to every other layer in a feed-forward fashion. a feed-forward neural network (fnn) is an artificial neural network in which the connections between nodes do not form a cycle [19, p. 73]. unlike conventional convolutional neural networks, which have \(l\) layers and \(l\) connections (one between each layer and its subsequent direct connections), densenet has \(\frac{l(l+1)}{2}\) direct connections [7]. preserving the feed-forward nature of the network, huang et al. [7] proposes a connectivity pattern between the layers, where the \(l^{th}\) layer receives the feature maps from all previous layers, \(x_0,...,x_{l-1}\). its input can be defined as: $$ x_l = h_l([x_0,x_1,...,x_{l-1}]), $$ where \([x_0,x_1,...,x_{l-1}]\) refers to the concatenation of the feature maps produced in layers \(0,...,l-1\). the \(h_l\) in this equation represents the multiple inputs into a single tensor. figure 1 illustrates this connectivity between the layers. fig. 1. example of a dense block with 5 layers. this image illustrates how the connections between the layers occur [7]. full size image densenet-121. the densenet-121 network consists of four dense blocks, where each block corresponds to a certain number of convolutional layers. dense block 1 is composed of 6 dense layers, dense block 2 consists of 12 dense layers, dense block 3 consists of 24 dense layers, and dense block 4 consists of 16 dense layers. each convolutional layer consists of batch normalization (bn), rectified linear unit (relu) activation function, and convolutional (conv) operations for feature extraction. densenet-121 uses a transition layer between two adjacent blocks, which consists of a 1\(\,\times \,\)1 convolution, followed by 2\(\,\times \,\)2 average pooling (calculating the average for each patch of the feature map). at the end of the last dense block, there is a pooling layer to which a classifier is attached, as shown in fig. 2. fig. 2. structure of densenet-121. full size image 3.2 focal loss the focal loss (fl) is a loss function that addresses the problem of class imbalance by reformulating the cross entropy loss (ce) for binary classification [5], in order to reduce the weight of the loss assigned to well-classified examples. the ce function can be defined as: $$ ce(p, y) = \left\{ \begin{array}{ll} -\log (p); &{} \text{ if } y=1,\\ \log (1 p); &{}\text{ otherwise, } \end{array} \right. $$ where \(y\in \{\pm 1\}\), with \(p\) being the probability for the class with \(y = 1\) and \(p\in [0,1]\). \(p_t\) is defined as follows: $$ p_t = \left\{ \begin{array}{ll} p; &{} \text{ if } y=1,\\ 1 p; &{}\text{ otherwise, } \end{array} \right. $$ thus, we have \(ce(p,y) = ce(p_t) = -\log (p_t)\). fl focuses the training on a sparse set of challenging examples, preventing a large number of easy negative examples from overwhelming the model during training [10]. it proposes the addition of a modulating factor \((1 p_t)^\gamma \) to the cross-entropy loss, with an adjustable focus parameter \(y \ge 0\). formally, the focal loss is defined as follows: $$ fl(p_t) = (1 p_t)^\gamma \log (p_t). $$ a variant of this equation, called \(\alpha \)-balanced can be used. this variant yields slightly improved precision compared to the non \(\alpha \)-balanced form [10], and is defined as: $$ fl(\rho _t) = \alpha _t(1 \rho _t)^\gamma \log (\rho _t), $$ it can also be observed that when \(\gamma = 0\) and \(\alpha _t = 1\), fl is equal to ce. 4 materials and methods 4.1 dataset this work utilized two datasets: chestx-ray14 [17] and chesxpert [8]. these datasets employ a multi-label approach, which means that each image may contain multiple pathologies. this section provides a description of each dataset used in this work, including information about the number of images, labels, as well as the origin of the data. fig. 3. number of images for each pathology in the chestx-ray14 dataset. full size image chestx-ray14. the chestx-ray14 dataset contains 112,120 radiograph images from 30,805 unique patients with image labels for 14 thoracic pathologies: atelectasis, cardiomegaly, consolidation, edema, effusion, emphysema, fibrosis, hernia, infiltration, mass, nodule, pleural thickening, pneumonia, and pneumothorax. this dataset is an extension of the 8 common pathology patterns listed in the chestx-ray8 dataset [17], which are represented in fig. 4. this dataset was extracted from clinical data in the pacs (picture archiving and communication systems) at the nih clinical center (national institutes of health clinical center). the images have a size of 1024\(\,\times \,\)1024 pixels and are in png format. natural language processing (nlp) techniques were used to extract the image labels, and the accuracy for the extracted labels was over 90% [17]. figure 3 shows the number of images for each pathology present in the dataset, highlighting a significant class imbalance. fig. 4. example of 14 of the pathologies present in chestx-ray14 dataset with their respective labels. full size image chexpert. the chexpert (chest expert) dataset consists of 224,316 frontal and lateral chest radiographic images from 65,240 unique patients. similar to chestx-ray14, this dataset includes 14 observations: no finding, enlarged cardiom., cardiomegaly, lung lesion, lung opacity, edema, consolidation, pneumonia, atelectasis, pneumothorax, pleural effusion, pleural other, fracture, and support devices. these 14 observations are labeled as positive, negative, or uncertain, where the uncertain label can represent both the uncertainty of a radiologist in the diagnosis and the inherent ambiguity in the report [8]. this data was extracted from chest radiographs performed at the inpatient and outpatient centers of stanford hospital between october 2002 and july 2017, along with their associated radiological reports. fig. 5. example of 5 of the pathologies present in chexpert dataset with their respective labels. full size image 4.2 preprocessing for the chestx-ray14 dataset, the pathology information for each image was preprocessed by converting it into an array of size 14, with each position representing each pathology. in this array, ‘0’ indicates the absence of the corresponding pathology and ‘1’ indicates the presence of the pathology. then, the images in the dataset were preprocessed. the images were divided into two subsets: a training set and a validation set. for each pathology, approximately 80% of the images were separated for training, and 20% were set aside for testing. this division was performed based on the unique patient identifier, ensuring that images from the same patient do not appear in both the training and testing sets. during the image preprocessing process, to standardize the pixel levels between 0 and 1, all intensities were divided by 255 (the maximum possible value). the feature maps were standardized by dividing each value by the standard deviation of the map, assuming a mean of 0, resulting in a standard deviation of 1 for the sample. additionally, these images were resized to a resolution of 224\(\,\times \,\)224 pixels. for the chexpert dataset, information on present pathologies was also pre-processed. the uncertainty labels in this dataset, annotated as ‘−1’, were handled using the ‘u-ones’ approach presented in [8], where these uncertainty labels were converted from ‘−1’ to ‘1’. additionally, blank labels were converted to ‘0’. five pathologies were selected from this dataset based on their predominance in the validation set and their presence in the chestx-ray14 dataset (see fig. 6). fig. 6. distribution of the selected classes in the chexpert dataset. full size image the pathology information in chexpert was converted into a vector of size 5, where ‘0’ indicates the absence of pathology and ‘1’ indicates the presence of pathology. the images in this dataset were preprocessed in the same way as the chestx-ray14 dataset. like chestx-ray14, these images were also resized to a size of 224\(\,\times \,\)224 pixels. 4.3 training configuration this work utilized a transfer learning approach known as fine-tuning, where the weights of a pretrained model are preserved (frozen) in some layers and adjusted (trained) in the remaining layers. in this work, a densenet-121 network implemented in [3] was used. it was pretrained with weights from imagenet [4]. the top classification layers were excluded. on top of this network, a global average pooling layer and the final sigmoid activation layer for classification were stacked. some layers of the pretrained model were frozen to keep their weights constant during training. then, the model was compiled with the appropriate loss functions for each experiment and an adam optimizer with a learning rate defined in the hyperparameter optimization process [1]. after that, the model underwent training and testing stages using the preprocessed datasets. the model was trained for 20 epochs with a batch size of 16, using an early stop callback to monitor the loss function. the minimum delta was set to 0.001, which defines the minimum change in the monitored value to qualify as an improvement, and the patience was set to 3, representing the number of epochs without improvement after which the training will be stopped. then, the fine-tuning stage was performed. the initial layers of the model were unfrozen, and the model was retrained with a lower learning rate. from there, the trained network is used to predict the labels of the images in the validation set of each dataset used in this work. 4.4 hyperparameter optimization process the hyperparameter optimization stage utilized the tree-structured parzen estimator (tpe) optimization algorithm, implemented in [1]. three hyperparameter search spaces were defined with the goal of maximizing the auc (roc). the first search space was for the learning rate of the network. this search space involved real values in the range [0.00001, 0.01], following a logarithmic pattern in value suggestions. the second search space was for the \(\alpha \) parameter of the focal loss function. for the values in this search space, real values in the range [0.1, 1.0] were considered, also following a logarithmic pattern in value suggestions. the third search space was related to the \(\gamma \) parameter of the focal loss function. in this search space, the values were set in the range [1.0, 5.0], and like the other two cases, it followed a logarithmic pattern in value suggestions. in each optimization process, 15 trials were executed. at the end of these processes, the data from this study were saved in a csv file, including their respective hyperparameters and auc values. 5 results and discussion in the first experiment, the proposed model was used to predict the 14 pathology labels of the chestx-ray14 dataset. for this initial experiment, the binary cross-entropy loss (bce) function, implemented in [3]. the adam optimizer with a learning rate of 0.00001 was employed. it was observed that the model handled the hernia class better than the other classes, with an auc value of 0.926. on the other hand, the pneumonia class had the lowest auc value of 0.647. in this experiment, the model achieved an auc above 0.900 in three out of the fourteen classes, while five classes had auc values above 0.800, and another five were above 0.700. only for the pneumonia class, the model achieved an auc value below 0.700. a new experiment was conducted using the chestx-ray14 dataset. this time the focal loss (fl) was used as the loss function, with \(\alpha \) set to 0.173 and \(\gamma \) set to 2.89. the adam optimizer with a learning rate of 0.00001 was employed. once again, the hernia class demonstrated the best performance among the other classes, achieving an auc of 0.986. on the other hand, the pneumonia class had the lowest performance with an auc of 0.749. notably, eight out of the fourteen categories achieved auc values above 0.850, while only three categories had auc values below 0.800, as depicted in fig. 7. fig. 7. performance of the model with focal loss using auc metric for the chestx-ray14 dataset. full size image the results of the two experiments are compared in table 1. it is evident that the model using fl as the loss function achieved superior performance for all pathologies, leading to a higher average auc across the 14 classes in the fl experiment. table 1. comparison of performance between the two experiments conducted on the chestx-ray14 dataset.full size table to further validate the proposed method, new experiments were conducted using the chexpert dataset. the binary cross-entropy loss function was used, along with the adam optimizer with a learning rate of 0.0002. the pleural effusion class achieved the highest performance, with an auc of 0.874. it is also noting that the atelectasis class had the lowest performance, with an auc value of 0.698. two classes had achieved results above 0.850, two classes had auc values above 0.700, and only one class had performance below 0.700. the average auc for this experiment was 0.790. a new experiment was conducted using the chexpert dataset. this time, fl was used as the loss function, with the parameter alpha set to 0.363 and gamma to 3.303. additionally, the adam optimizer was utilized with a learning rate of approximately 0.0002. there was a slight improvement compared to the previous experiment. the class with the highest auc score was edema with an auc of 0.909 as depicted in fig. 8. two classes achieved auc values above 0.900, and all classes had a performance above 0.700. the average auc for this experiment was 0.820. fig. 8. model performance with fl using the auc metric for the chexpert database. full size image table 2 compares the results of the experiments using chexpert. similar to the experiments using chestx-ray14, it was observed that the model utilizing fl as the loss function achieved the best performance for all studied pathologies. thus, the average for the 5 classes was higher in the experiment using fl. table 2. comparison of performance between the two experiments conducted with chexpert.full size table we compared the results of our most promising experiment with the results of other studies that also used deep learning techniques to classify thoracic pathologies in the chestx-ray14 dataset. the results shown in table 3 indicate that the proposed model obtained a superior performance in terms of the auc metric for 6 labels. note that, for the emfisem class, the difference between the proposed model and the work by zhao et al. [20] is minimal, with a difference of approximately 0.002, similar to the class atelectasis with a difference of 0.007. it is important to mention that the distribution of training sets may vary between different studies. to ensure a fair comparison, it would be ideal to reproduce the experiments using the exact same distribution of datasets. table 3. performance of the proposed model compared to the literature using the auc metric.full size table in the experiments conducted using the chestx-ray14 dataset, the impact of using focal loss on the overall network performance can be observed, resulting in an average performance gain of 5.5% points. however, for pathologies with fewer positive samples, the performance was notably better. in these cases, there was a greater performance improvement for the labels pneumonia (10.2% points), mass (6.9% points), fibrosis (6.9% points), and hernia (6.0% points), which are among the pathologies with fewer samples in the dataset. on the other hand, labels that have more positive samples in the dataset showed a performance gain of less than 5.0% points, such as effusion (3.2% points), infiltration (4.2% points), and atelectasis (4.6% points). a similar behavior can also be observed in the experiments conducted using chexpert, which demonstrated an average performance gain of 3.0% points. labels with a higher number of positive samples exhibited the lowest performance improvement, such as the pleural effusion label (0.6% points). conversely, the label with the fewest positive samples in the dataset showed the highest performance gain, as seen in the case of the consolidation label (5.3% points). 6 conclusion this work evaluated the performance of a densenet for the classification of 14 thoracic pathologies in a multi-label approach. experiments were conducted with the proposed model using binary cross-entropy loss and also using focal loss to address the class imbalance issues present in the datasets. the proposed model was validated using the chestx-ray14 and chexpert datasets. the results of the model using cross-entropy and focal loss were compared, and in these experiments, focal loss outperformed cross-entropy in all cases, as it effectively addressed the class imbalance problem present in both datasets. the most promising experiment was conducted using focal loss to predict the 14 pathologies in the chestx-ray14 dataset, achieving an average auc score of 0.861, which was the highest among the researched works and outperformed the other methods in six out of the fourteen classes. one of the major challenges of this work was the significant class imbalance in the utilized datasets. as future work, we intend to explore the structure of densenet further to improve the results, as well as apply this approach to computed tomography images. it is important to emphasize that although the model produced satisfactory results, its use does not replace the diagnosis of a radiology professional. however, it is also important to highlight that the results generated by the proposed model can provide valuable references for these professionals. this work was carried out with the support of the coordination for the improvement of higher education personnel brazil (capes) financing code 001, maranhão research support foundation (fapema), national council for scientific and technological development (cnpq) and brazilian company of hospital services (ebserh) brazil (proc. 409593/2021-4). references akiba, t., sano, s., yanase, t., ohta, t., koyama, m.: optuna: a next-generation hyperparameter optimization framework. in: proceedings of the 25th acm sigkdd international conference on knowledge discovery & data mining, pp. 2623–2631 (2019) google scholar  bhusal, d., panday, d., prasad, s.: multi-label classification of thoracic diseases using dense convolutional network on chest radiographs. arxiv preprint arxiv:2202.03583 (2022) chollet, f., et al.: keras (2015). https://github.com/fchollet/keras deng, j., dong, w., socher, r., li, l.j., li, k., fei-fei, l.: imagenet: a large-scale hierarchical image database. in: 2009 ieee conference on computer vision and pattern recognition, pp. 248–255. ieee (2009) google scholar  good, i.j.: rational decisions. in: kotz, s., johnson, n.l. (eds.) breakthroughs in statistics, pp. 365–377. springer, new york (1992). https://doi.org/10.1007/978-1-4612-0919-5_24 chapter  google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 4700–4708 (2017) google scholar  irvin, j., et al.: chexpert: a large chest radiograph dataset with uncertainty labels and expert comparison. in: proceedings of the aaai conference on artificial intelligence, vol. 33, pp. 590–597 (2019) google scholar  lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  google scholar  lin, t.y., goyal, p., girshick, r., he, k., dollár, p.: focal loss for dense object detection. in: proceedings of the ieee international conference on computer vision, pp. 2980–2988 (2017) google scholar  marcos, l., bichinho, g.l., panizzi, e.a., storino, k.k.g., pinto, d.c.: classificação da doença pulmonar obstrutiva crônica pela radiografia do tórax. radiol. bras. 46, 327–332 (2013) article  google scholar  qian, y., fan, y., hu, w., soong, f.k.: on the training aspects of deep neural network (dnn) for parametric tts synthesis. in: 2014 ieee international conference on acoustics, speech and signal processing (icassp), pp. 3829–3833. ieee (2014) google scholar  rajpurkar, p., et al.: chexnet: radiologist-level pneumonia detection on chest x-rays with deep learning. arxiv preprint arxiv:1711.05225 (2017) santos, m.k., ferreira, j.r., wada, d.t., tenório, a.p.m., barbosa, m.h.n., marques, p.m.d.a.: inteligência artificial, aprendizado de máquina, diagnóstico auxiliado por computador e radiômica: avanços da imagem rumo à medicina de precisão. radiol. bras. 52, 387–396 (2019) google scholar  wada, d.t., rodrigues, j.a.h., santos, m.k.: aspectos técnicos e roteiro de análise da radiografia de tórax. medicina 52(suppl. 1), 5–15 (2019) google scholar  wang, s.c.: artificial neural network. in: wang, s.c. (ed.) interdisciplinary computing in java programming, pp. 81–100. springer, boston (2003). https://doi.org/10.1007/978-1-4615-0377-4_5 chapter  google scholar  wang, x., peng, y., lu, l., lu, z., bagheri, m., summers, r.: chestx-ray8: hospital-scale chest x-ray database and benchmarks on weakly-supervised classification and localization of common thorax diseases. in: 2017 ieee conference on computer vision and pattern recognition (cvpr), pp. 3462–3471 (2017) google scholar  yao, l., poblenz, e., dagunts, d., covington, b., bernard, d., lyman, k.: learning to diagnose from scratch by exploiting dependencies among labels. arxiv preprint arxiv:1710.10501 (2017) zell, a.: simulation neuronaler netze, vol. 1. addison-wesley bonn (1994) google scholar  zhao, j., li, m., shi, w., miao, y., jiang, z., ji, b.: a deep learning method for classification of chest x-ray images. in: journal of physics: conference series, vol. 1848, p. 012030. iop publishing (2021) google scholar  download references author information authors and affiliations núcleo de computacão aplicada, universidade federal do maranhão (ufma) caixa postal, 65.085-580, são luís, ma, brazil alison corrêa mendes, alexandre césar pinto pessoa & anselmo cardoso de paiva authors alison corrêa mendesview author publications search author on:pubmed google scholar alexandre césar pinto pessoaview author publications search author on:pubmed google scholar anselmo cardoso de paivaview author publications search author on:pubmed google scholar corresponding author correspondence to alison corrêa mendes . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper mendes, a.c., pessoa, a.c.p., de paiva, a.c. (2023). multi-label classification of pathologies in chest radiograph images using densenet. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_12 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords radiography densenet focal loss publish with us policies and ethics profiles alexandre césar pinto pessoa view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature optimizing cleanunet architecture parameters for enhancing speech denoising | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper optimizing cleanunet architecture parameters for enhancing speech denoising conference paper first online: 30 january 2025 pp 254–264 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) optimizing cleanunet architecture parameters for enhancing speech denoising download book pdf download book epub matheus vieira da silva  orcid: orcid.org/0000-0002-1673-81879, joão fernando mari  orcid: orcid.org/0000-0001-8271-380210 & andré ricardo backes  orcid: orcid.org/0000-0002-7486-425311  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 381 accesses abstract speech enhancement refers to a set of techniques aiming to recover clean speech from a corrupted signal. one way to corrupt a signal is through noise addition. noise comes in a variety of ways. suboptimal acoustic conditions can cause background noise and echo, hampering speech clarity and making denoising techniques necessary to enhance the speech. in this work, we optimized cleanunet, a convolutional neural network (cnn) architecture proposed specifically for causal speech denoising tasks. we explored alternatives for the transformer bottleneck, such as mamba architecture, capable of handling encoder outputs more efficiently with linear complexity, we also reduced the number of hidden channels in the convolutional layers. this decreases the model’s parameter count and improves training and inference speed on a single gpu, offering a streamlined approach for enhanced performance. to our understanding, this is the first attempt to incorporate mamba as a replacement for the vanilla transformer in the cleanunet architecture. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others enhancing robustness of arabic speech recognition in noisy environments using advanced feature extraction and denoising techniques based on deep learning models article 04 december 2025 effects of skip connections in cnn-based architectures for speech enhancement article 15 may 2020 cross channel interaction based eca-net using gated recurrent convolutional network for speech enhancement article 28 june 2024 1 introduction the covid-19 pandemic and lockdowns have led to significant changes in how people work and interact with technology, exemplified by zoom’s exponential growth from 10 million daily participants in december 2019 to over 300 million by april 2020 [8]. the pandemic led to a sudden shift to remote work and virtual meetings, revealing a significant issue: poor home office acoustics negatively impact videoconferencing. suboptimal acoustic conditions caused more background noise and echo, hampering speech clarity and increasing the effort required for video calls. inadequate acoustic environments at home also hurt audio quality and workforce productivity. it’s worth noting that the problem of insufficient acoustics isn’t limited to home offices during videoconferencing. challenges of background noise disrupting communication have been a concern from the beginning of the digital revolution [20]. advanced techniques like spectral subtraction [2] and wiener filtering [15] have been developed to address the challenges in speech enhancement. these methods use advanced algorithms and filters to reduce background noise, enhance speaker voices, and improve audio quality. the main drawback of these techniques is that they assume stationary noise, meaning that the background noise features are expected to remain relatively constant during the entire audio recording. in real-world scenarios, like zoom meetings, noise can often be non-stationary, fluctuating in intensity and spectral content over time. in recent years, thanks to advancements in deep learning, the efficient removal of non-stationary noise from speech has been significantly enhanced [26]. there are two primary methods for speech enhancement: the time-frequency domain and the waveform domain. some cross-domain studies explore hybrid approaches that combine both methods [21, 25]. recent advancements in deep learning, such as convolutional networks [10], generative adversarial networks [5], u-net for segmentation [19], and transformer architecture [24] have facilitated the attainment of state-of-the-art performance by strategically harnessing the merits and limitations inherent to each architectural paradigm. an architecture that attains state-of-the-art results is known as cleanunet [9]. this model draws inspiration from [17] and combines transformers and u-net to enhance noisy speech within the waveform domain. however, the cleanunet architecture presents a formidable challenge: the size of the model and its vulnerability to quadratic complexity within the transformer bottleneck. these challenges lead to three main implications: 1) constrained computational resources: the training and inference processes necessitate substantial computational resources due to the architecture’s computational intensity and its dependency on audio samples that vary with the sample rate; 2) environmental concerns: the first implication raises environmental concerns, including issues such as water and carbon footprint, which are exacerbated by the extensive computational demands when operating at scale. 3) real-time performance: the model’s adaptability to resource-limited hardware is inadequate for real-time denoising applications. in this work, we address the three primary challenges of cleanunet by optimizing its architecture. our contributions include introducing alternatives to the transformer bottleneck to reduce the model’s parameter count for efficiency and enhancing inference speed to facilitate real-time applications. our revised model enhances efficiency by replacing the transformer bottleneck with alternatives that handle encoder outputs with linear complexity o(n). additionally, we empirically optimized the number of hidden channels in the convolutional layers. these modifications reduce the model’s parameter count and significantly improve both training and inference speeds on a single gpu. the remainder of this paper is organized as follows: sect. 2 presents the related work. section 3 describes the material and methods used in our work. section 4 presents the experiments and results, while sect. 5 concludes the paper. 2 related works spectral subtraction is a simple yet effective method for reducing background noise [23]. this method immediately reduces the amplitude of noise components while preserving speech signals by estimating the average noise spectrum during silent periods and subtracting it from the input signal spectrum. however, the randomness of noise may result in negative values and distortions. based on statistical estimates of the underlying signal and noise, wiener filtering dynamically creates a linear filter capable of suppressing existing noise [18]. the signal and noise are modeled as stochastic processes with well-known spectral properties. its self-adjusting and statistically-based design make the wiener filtering very effective at removing real-world speech noise, especially in home office environments. the statistical characteristics of stationary noises are stable and do not change significantly over time. examples include constant motor hum, continuous background conversation, and consistent background noise. constant background noise can be approximated by updating silent periods, techniques like spectral subtraction work best. non-stationary noises that change rapidly are hard to estimate and suppress. non-stationary sounds have time-varying statistics, meaning their characteristics change rapidly. examples include abrupt sounds like breaking glass, sporadic construction noises, and sudden horns. one of the early deep learning methods used for speech denoising was recurrent neural networks (rnns) [14]. rnns process input sequences incrementally while preserving an internal state to capture temporal context, making them ideal for speech processing. rnn training gradually maps noisy speech inputs to clean target outputs and aims to reduce audio noise. however, rnns have limitations, such as challenging inference and high computational cost. as an alternative, convolutional neural networks (cnns) [12] have become popular. 1d cnns use local patterns and work directly on raw audio for noise reduction. 2d cnns operate using spectrogram representations. in both cases, the network captures local speech features using small convolutional filters to separate them from background noise. pooling and downsampling aggregate information over time. multiple convolutional layers learn higher-level features, recognizing speech components even in noisy conditions. one of the main advantages of cnns is weight sharing, significantly reducing parameters compared to rnns, thus resulting in more efficient training and inference. while rnns leverage temporal context, cnns employ localized filters to effectively isolate speech from noise in the time-frequency domain for effective noise reduction. ongoing research is improving the performance of cnn-based speech denoising [26]. generative adversarial networks (gans) [5] employ a generator to enhance noisy speech and a discriminator that distinguishes between enhanced and clean speech. adversarial training improves the quality of the output. cnns are commonly used in the generator and discriminator models in gan architectures. recent advancements in deep learning have incorporated hybrid solutions involving u-net [19] and transformer [24] architectures. transformers can capture long-term speech contexts and precise interactions between noisy inputs for denoising. self-attention layers prioritize important speech components over background noise. although the use of transformers for speech processing is still being investigated, early results are promising [11]. 3 materials and methods 3.1 dataset the database employed in this work was obtained from [22] and is composed of two separate datasets from 84 speakers with 28.4 h of clean and noisy speech pairs, designed for training and testing speech enhancement models that operate at 48 khz. the training set consists of 34,647 pairs of audio samples, while the test set comprises 824 pairs. due to variations in the lengths of the audio samples within each pair, a preprocessing step was undertaken to normalize their sizes. this involved concatenating all the files and subsequently cropping a new file to a specified length, denoted as l. if the last file is smaller than l, it will be padded with the last valid value. 3.2 cleanunet cleanunet [9] is a cnn architecture proposed specifically for causal speech denoising tasks. it follows the principles of the u-net architecture, a popular model in image segmentation tasks, and it is similar to the model presented in [3]. as in u-net, cleanunet consists of an encoder-decoder structure with skip connections, enabling the network to preserve fine-grained details during upsampling. each convolutional block usually contains multiple convolutional layers followed by max-pooling layers and relu activation. cleanunet adapts this architecture to the domain of audio processing on the raw waveform. it also uses self-attention blocks in the bottleneck to improve its ability to separate noise from the clean signal. each self-attention block is composed of a multi-head self-attention layer and a position-wise fully connected layer [24]. one notable aspect of cleanunet is its ability to handle varying levels and types of noise present in audio signals. it outperforms state-of-the-art models in terms of speech quality, according to several objective and subjective evaluation metrics. 3.3 evaluation metrics we employed two objective evaluation metrics to assess the performance of our trained model: the perceptual evaluation of speech quality (pesq) and the short-time objective intelligibility (stoi) [13]. pesq. this metric focuses on the perceived quality of the speech, and it involves a psychoacoustic algorithm to simulate human hearing and compare the clean speech (original) and the enhanced speech (model output) to predict perceived quality. the calculation includes steps such as level alignment, filtering, and a detailed comparison process that considers various distortions and artifacts introduced by processing. the output score ranges from -0.5 to 4.5, where higher scores indicate better quality. stoi. designed to predict the intelligibility of speech signals, the method involves calculating the short-time temporal correlation between the clean (original) and enhanced speech (model output) signals across various frequency bands. the process includes segmenting the signals into short frames, applying a fourier transform to analyze the frequency content, and then computing the correlation between the corresponding frames of the clean and enhanced signals. the stoi score ranges from 0 to 1, where 0 indicates poor intelligibility and 1 indicates excellent intelligibility. 3.4 transformer architecture first introduced by [24], the transformer architecture uses self-attention mechanisms [1] to capture the dependencies between different words or tokens in a sequence. it allows the model to weigh the importance of each token in the sequence concerning every other token. the key component of self-attention is the calculation of attention scores based on queries (q), keys (k), and values (v). the mathematical representation of self-attention is described below: $$\begin{aligned} \text {attention}(q, k, v) = \text {softmax}\left( \frac{qk^t}{\sqrt{d_k}}\right) v \end{aligned}$$ (1) where \(q, k, v\) represent the queries, keys, and values matrices derived from the input matrix. \(\sqrt{d_k}\) is a scaling factor, where \(d_k\) has the dimensionality of \(k\) and \(v\). the softmax function normalizes the attention scores, and the result is used to compute a weighted sum of the \(v\) vectors. inspired by cnns, the architecture employs a self-attention technique variant called multi-headed attention to process input sequences simultaneously. the following expression illustrates the process: $$\begin{aligned} \text {multihead}(q, k, v) = \text {concat}(\text {head}_1, \text {head}_2, \ldots , \text {head}_h)w^o \end{aligned}$$ (2) $$\begin{aligned} \text {head}_i = \text {attention}(qw_i^q, kw_i^k, vw_i^v) \end{aligned}$$ (3) where \(head_i\) is the attention score, and \(w^q_i, w^k_i, w^v_i\) and \(w^o_i\) are parameter matrices. during training, the computational complexity of computing attention scores for each token pair of results has a quadratic growth of \(o(n^2)\) with an input sequence of length n. researchers have proposed optimizations and approximations to address this challenge to reduce computational overhead [7, 16]. 3.5 longnet the longnet architecture was introduced by [4]. this new transformer variant achieves linear computational complexity and logarithmic token dependency, optimizing efficiency for extensive sequence processing. the core innovation of the dilated attention mechanism lies in its ability to expand the attentive field exponentially with increasing distance. this mechanism achieves this by dividing the input \((q,k,v)\) into equally sized segments with a predetermined segment length \(w\). each segment is then made sparse along the sequence dimension by selecting rows at specified intervals \(r\). the segments are processed in parallel, after which they are scattered and concatenated to produce the output. incorporating the concept of multi-head attention, the innovative method further enhances the dilated attention mechanism by diversifying the computation across different heads. specifically, the computation varies among heads by sparsifying distinct segments of the \((q,k,v)\) pairs. this architecture is fully compatible with transformers with an introduction of segment sizes \(r\) and dilation rates \(d\) hyperparameters. 3.6 mamba architecture mamba [6] is an innovative architecture designed to mitigate the quadratic complexity challenges faced by conventional attention models. this model addresses the scalability constraints of traditional transformer models in processing long sequences and significantly enhances efficiency and flexibility with linear-time complexity for sequence modeling. it introduces two significant state space models (ssm) innovations: a selection mechanism and a hardware-aware algorithm. a ssm can be simplified by the parameters \((\vardelta , a, b, c)\), where each parameter represents a diagonal matrix. the discretization of \((\vardelta , a, b)\) yields \((\overline{a}, \overline{b})\). this procedure translates the model from a continuous domain to a discrete one, enabling computation within a digital framework. the discretization process can be done through fixed formulas \(\overline{a} = f_a(\vardelta , a)\) and \(\overline{b} = f_b(\vardelta , a, b)\), where \((f_a, f_b)\) pair is a discretization rule such as zero-order hold (zoh). one essential property of ssms is linear time invariance (lti). in this case, all time steps in the dynamics represented by the matrices \((\vardelta , a, b, c)\) are the same. this characteristic enables parallelizing training methods, using convolutional or recursive algorithms to efficiently compute the model’s output. the mamba architecture introduces a selection mechanism similar to a gating mechanism of rnns that can focus or filter out different parts of a context allowing ssms to adapt over time and being input-dependent. however, this modification compromises the lti property, resulting in the inability to apply convolution methods as previously done. to overcome this, the authors implement a hardware-aware method to take advantage of the memory hierarchy of modern gpus. the hardware-aware algorithm incorporates three key techniques: kernel fusion, parallel scan, and recomputation. kernel fusion reduces memory i/o by combining multiple operations into a single gpu operation. parallel scan algorithms are applied to enable parallel computation despite the non-linear nature of the ssms. recomputation is used during backpropagation to minimize memory usage by recalculating intermediate states instead of storing them. 4 experiments and results experiments were performed on a pc with an amd ryzen 7 7800x3d processor, 32gb of ram, and an rtx 3090 24gb gpu running linux ubuntu 20.04 os. we modified the original cleanunetfootnote 1 code to run on pytorch 2.1.1footnote 2, cuda 12.3, and python 3.10footnote 3, along with the modifications needed to implement mamba architecture using mamba-ssmfootnote 4. we proposed and evaluated two variations of cleanunet, both designed to reduce the number of learnable parameters, reducing complexing, training, and prediction time, and looking to maintain the quality. the first variation consists of applying dilated attention mechanisms, while the second involves replacing the transformer architecture with the mamba architecture. both models were compared with cleanunet described in [9], which we considered the baseline. the baseline cleanunet [9] consists of an encoder and a decoder, both with d layers and a bottleneck composed of n self-attention blocks (bottleneck layers). each encoder layer is composed of a 1d convolution layer (conv1d) followed by a relu layer and a 1\(\times \)1 convolution (conv1\(\times \)1) followed by a gated linear unit (glu). each conv1d has kernel size k and stride \(s = k/2\). the first conv1d has h kernels, and the other layers double the number of channels. the conv1x1 doubles the number of channels while the glu reduces it by half. each decoder is composed of a 1\(\times \)1 convolution followed by a glu and a transposed 1d convolution layer (convtranspose1d). following the u-net architecture, each encoder layer is connected to one encoder layer by a skip connection in the reverse order. the bottleneck is composed of k self-attention blocks. each self-attention block comprises a multi-head self-attention layer (8 heads) and a fully connected layer (input/output size of 512 and inner-layer size of 2048) as described in [9]. for this study we considered a cleanunet model with \(d=8\), \(k=4\), and \(n=5\) [9] the proposed dilated attention base transformers follow the unet structure from cleanunet, which is considered a baseline with a bottleneck composed of dilated attention layers. for this configuration, we considered a dilated attention-based cleanunet model with \(d = 4\), \(k = 3\), and \(n = 1\) for training and prediction in gpu. the self-attention block comprises a dilated multi-head self-attention layer (32 heads) and a fully connected layer (input/output size of 1024 and inner-layer size of 2048). the list of segment sizes {10, 20, 30, 60} was provided with its respective dilation rates {2, 4, 8, 16}. the proposed cleanunet with a bottleneck composed of mamba architecture also follows the configuration of the baseline cleanunet but with changes in the number of encoder/decoder layers \(d = 10\), and hidden channels starting from 32 from the first layers to max hidden channels 256. a mamba architecture replaced the transformer bottleneck with \(n = 1\). the dimensionality of the input vector in this bottleneck was set to 512 (input/output) with 16 layers. we used a training configuration similar to [9]. the models were trained using adam optimizer with \(\beta _{1} = 0.9\) and \(\beta _{2} = 0.999\). the learning rate decreases along the training using a cosine annealing rate scheduler with a maximum learning rate of 0.00025 and a warmup ratio of 5%. in this work, we employ the random shift data augmentation technique within a range of 0 to s seconds and bandmask augmentation [3]. the cleanunet loss function described in [9] combines a multi-resolution stft (short-time fourier transform) loss with a waveform-based loss to maximize speech-denoising performance. it is composed of two parts: the first is the \(l1\) loss applied directly to the waveform, which encourages the denoised output to resemble the time domain clean speech waveform closely, and the second part is the multi-resolution stft loss, which modifies the spectrogram’s magnitude at various resolutions. table 1 presents the optimal outcomes and configurations for the speech enhancement model, highlighting the efficiency of implementing the mamba architecture as a bottleneck. this approach significantly reduces the model’s parameters, enabling increased data throughput and larger batch sizes. consequently, it facilitates training the model on a single gpu. given the linear complexity of mamba’s input handling, the model performs computations more quickly in the bottleneck. this design also supports larger vector sequences. similarly, longnet claims linear complexity, theoretically supporting longer sequences as well. according to the table results, longnet uses fewer convolutional encoders to fit within memory constraints. this efficiency allows the preservation and utilization of features that might otherwise be discarded during the convolution process, thereby enhancing the model’s performance. table 1. cleanunet model results and settingsfull size table fig. 1. comparing the models considering the number of parameters vs audio quality metrics. in the first row, we show the total number of parameters. in the last row, we show the number of parameters in the bottleneck. the first column also includes the inference time in seconds (s). full size image the results presented in table 1 are visually detailed by the scatter graphs in fig. 1, in which we show the number of parameters on the x-axis versus the audio quality metrics on the y-axis. in the first row, we consider the total number of model parameters, including encoder, decoder, and bottleneck, and in the last row, we consider only the number of parameters in the bottleneck. the proposed cleanunet with a mamba-based bottleneck significantly reduces the number of trainable parameters with a slight reduction in the audio quality indexes. while the total number of parameters is reduced by 69.59%, the reduction in the stoi, pesq-wideband, and pesq-narrowband is only 4.60%, 3.63%, and 6.02%, respectively. it is important to note that, besides replacing the bottleneck with a mamba architecture, the number of layers in the encoders/decoders was increased, however, the number of channels was reduced to enable the model to perform faster operations while saving memory. during inference using the entire test set, the proposed model performed 46.9% faster than the baseline model. 5 conclusion this paper addressed some challenges related to using cleanunet for speech denoising. its good performance comes from a deep convolutional network that combines transformers and u-net to enhance noisy speech within the waveform domain. training requires a huge amount of resources, and its architecture limits the model’s adaptability to resource-limited hardware, making it inadequate for real-time denoising applications. we explored alternatives for the transformer bottleneck, such as mamba architecture, capable of handling encoder outputs more efficiently with linear complexity, reducing the need for multiple convolutional layers. as a result, we drastically reduced the number of trainable parameters while preserving its performance, thus improving training and inference speed on a single gpu. notes 1.https://github.com/nvidia/cleanunet. 2.https://pytorch.org/. 3.https://www.python.org/. 4.https://pypi.org/project/mamba-ssm/1.1.0/. references bahdanau, d., cho, k., bengio, y.: neural machine translation by jointly learning to align and translate. arxiv preprint arxiv:1409.0473 (2014) boll, s.: suppression of acoustic noise in speech using spectral subtraction. ieee trans. acoust. speech signal process. 27(2), 113–120 (1979) article  math  google scholar  defossez, a., synnaeve, g., adi, y.: real time speech enhancement in the waveform domain (2020) google scholar  ding, j., et al.: longnet: scaling transformers to 1,000,000,000 tokens. arxiv preprint arxiv:2307.02486 (2023) goodfellow, i., et al.: generative adversarial nets. adv. neural inf. process. syst. 27 (2014) google scholar  gu, a., dao, t.: mamba: linear-time sequence modeling with selective state spaces. arxiv preprint arxiv:2312.00752 (2023) hassani, a., shi, h.: dilated neighborhood attention transformer. arxiv preprint arxiv:2209.15001 (2022) karl, k.a., peluchette, j.v., aghakhani, n.: virtual work meetings during the covid-19 pandemic: the good, bad, and ugly. small group res. 53(3), 343–365 (2022) article  google scholar  kong, z., ping, w., dantrey, a., catanzaro, b.: speech denoising in the waveform domain with self-attention. in: icassp – ieee international conference on acoustics, speech and signal processing, pp. 7867–7871. ieee (2022) google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. adv. neural inform. process. syst. 25 (2012) google scholar  latif, s., zaidi, a., cuayahuitl, h., shamshad, f., shoukat, m., qadir, j.: transformers in speech processing: a survey. arxiv preprint arxiv:2303.11607 (2023) lecun, y., bottou, l., bengio, y., haffner, p.: gradient-based learning applied to document recognition. proc. ieee 86(11), 2278–2324 (1998) article  math  google scholar  loizou, p.c.: speech enhancement: theory and practice. crc press (2013) google scholar  maas, a., le, q.v., o’neil, t.m., vinyals, o., nguyen, p., ng, a.y.: recurrent neural networks for noise reduction in robust asr (2012) google scholar  mcaulay, r., malpass, m.: speech enhancement using a soft-decision noise suppression filter. ieee trans. acoust. speech signal process. 28(2), 137–145 (1980) article  math  google scholar  pagliardini, m., paliotta, d., jaggi, m., fleuret, f.: faster causal attention over large sequences through sparse flash attention. arxiv preprint arxiv:2306.01160 (2023) petit, o., thome, n., rambour, c., themyr, l., collins, t., soler, l.: u-net transformer: self and cross attention for medical image segmentation. in: lian, c., cao, x., rekik, i., xu, x., yan, p. (eds.) mlmi 2021. lncs, vol. 12966, pp. 267–276. springer, cham (2021). https://doi.org/10.1007/978-3-030-87589-3_28 chapter  google scholar  robinson, e.a., treitel, s.: principles of digital wiener filtering. geophys. prospect. 15(3), 311–332 (1967) article  math  google scholar  ronneberger, o., fischer, p., brox, t.: u-net: convolutional networks for biomedical image segmentation. in: navab, n., hornegger, j., wells, w.m., frangi, a.f. (eds.) miccai 2015. lncs, vol. 9351, pp. 234–241. springer, cham (2015). https://doi.org/10.1007/978-3-319-24574-4_28 chapter  math  google scholar  shannon, c.e.: a mathematical theory of communication. bell syst. technical j. 27(3), 379–423 (1948) article  mathscinet  math  google scholar  tang, c., luo, c., zhao, z., xie, w., zeng, w.: joint time-frequency and time domain learning for speech enhancement. in: 29th international conference on international joint conferences on artificial intelligence, pp. 3816–3822 (2021) google scholar  valentini-botinhao, c., et al.: noisy speech database for training speech enhancement algorithms and tts models. university of edinburgh. school of informatics, centre for speech technology research (2017) google scholar  vaseghi, s.v., vaseghi, s.v.: spectral subtraction. adv. signal process. digital noise reduct., 242–260 (1996) google scholar  vaswani, a., et al.: attention is all you need. adv. neural inform. process. syst. 30 (2017) google scholar  wang, h., wang, d.: cross-domain speech enhancement with a neural cascade architecture. in: icassp – ieee international conference on acoustics, speech and signal processing, pp. 7862–7866. ieee (2022) google scholar  yuliani, a.r., amri, m.f., suryawati, e., ramdan, a., pardede, h.f.: speech enhancement using deep learning methods: a review. jurnal elektronika dan telekomunikasi 21(1), 19–26 (2021) article  google scholar  download references acknowledgments andré r. backes gratefully acknowledges the financial support of cnpq (national council for scientific and technological development, brazil) (grant #307100/2021-9). this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brazil (capes) finance code 001. author information authors and affiliations school of computer science, federal university of uberlandia, uberlandia, brazil matheus vieira da silva institute of exacts and technological sciences, federal university of viçosa, rio paranaíba, brazil joão fernando mari departament of computing, federal university of são carlos, são carlos, brazil andré ricardo backes authors matheus vieira da silvaview author publications search author on:pubmed google scholar joão fernando mariview author publications search author on:pubmed google scholar andré ricardo backesview author publications search author on:pubmed google scholar corresponding author correspondence to andré ricardo backes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, m.v., mari, j.f., backes, a.r. (2025). optimizing cleanunet architecture parameters for enhancing speech denoising. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_18 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_18 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords cleanunet audio processing mamba architecture convolutional neural network publish with us policies and ethics profiles joão fernando mari view author profile andré ricardo backes view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature clinical oncology textual notes analysis using machine learning and deep learning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper clinical oncology textual notes analysis using machine learning and deep learning conference paper first online: 12 october 2023 pp 140–153 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) clinical oncology textual notes analysis using machine learning and deep learning download book pdf download book epub diego pinheiro da silva  orcid: orcid.org/0000-0003-2106-644x9, william da rosa fröhlich  orcid: orcid.org/0000-0003-3551-262310, marco antonio schwertner9 & … sandro josé rigo  orcid: orcid.org/0000-0001-8140-56219  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 502 accesses abstract advances in textual classification can foster quality in existing clinical systems. our research explored experimentally text classification methods applied in non-synthetic oncology clinical notes corpora. the experiments were performed in a dataset with 3,308 medical notes. experiments evaluated the following machine learning and deep learning classification methods: multilayer perceptron neural network, logistic regression, decision tree classifier, random forest classifier, k-nearest neighbors classifier, and long-short term memory. an experiment evaluated the influence of the corpora preprocessing step on the results, allowing us to identify that the classifier’s mean accuracy was leveraged from 26.1% to 86.7% with the per-clinical-event corpus and 93.9% with the per-patient corpus. the best-performing classifier was the multilayer perceptron, which achieved 93.90% accuracy, a macro f1 score of 93.61%, and a weighted f1 score of 93.99%. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others ensemble of classifiers for multilabel clinical text categorization in portuguese chapter © 2023 machine learning performance analysis for classification of medical specialties chapter © 2023 a clinical text classification paradigm using weak supervision and deep representation article open access 07 january 2019 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization clinical epigenetics learning algorithms machine learning oncology statistical learning 1 introduction today’s world is moving toward changing healthcare from reactive and hospital-centered to preventive and personalized treatment approaches [1]. therefore, physicians must be updated with these new treatments and patients’ health information. furthermore, decision-making is affected by a broad range of parameters. for instance, different information sources are used to define oncology diagnosis and treatment options [2]. in this context, the information accumulated in a health institution’s database can prove valuable to help physicians decide on the appropriate course of action in specific cases [3]. nevertheless, it is not feasible for health professionals to manually evaluate and analyze these resources due to the enormous effort it will demand, considering the size of these databases [4], and the exponential growth in medical information [5]. due to the rapid growth of electronic information, most health information about patients is found as unstructured data, i.e., narrative text or free text [3]. therefore, the unstructured format of the enormous amount of medical data is the primary motive of recent research initiatives aiming to extract information from medical notes [6,7,8,9]. considering these challenges, tools are needed to support healthcare professionals in their activities beyond a healthcare system’s restructuring. clinical decision support systems (cdss) provide additional assistance, synthesizing and integrating patient-specific information, performing complex evaluations, and presenting the results to clinicians in an adequate time [10]. in addition, cdss offers assistance to overcome the difficulties in dealing with massive amounts of information [11,12,13]. this work focuses on the text classification task, covering the steps to create and process a clinical corpus for the experiments and comparing evaluations with machine learning and deep learning approaches. an oncology clinical notes corpus was created, preprocessed, and transformed to be used by machine learning and deep learning methods. this corpus was obtained from a real-world oncology clinic, de-identified to preserve the patient and professional’s identification, and is entirely composed of brazilian portuguese language texts. in addition, this work implemented several machine learning and deep learning methods for text classification, compared their performance, and evaluated their results. related works about text classification and information extraction methods in oncology clinical notes were studied to understand how to better deal with unstructured data. the papers used one or a combination of natural language processing (nlp), machine learning, and deep learning methods. the related works study indicates text classification and information extraction as essential tasks to deal with unstructured data in healthcare. several related works used domain-specific corpora. however, none used a corpus with non-synthetic medical notes from the oncology healthcare-specific domain in the brazilian portuguese language. furthermore, most papers used machine learning or deep learning methods to consider the text classification task. just a few of them presented a comparison between several machine learning and deep learning methods. therefore, the main objective of this work consists of applying text classification approaches to support healthcare professionals regarding diagnosis decisions. the survey provided insight into crucial aspects of the area, such as that patients’ clinical data are often inputted in a free-text format. there was a lack of structured data repositories on this topic. this topic was described as an essential problem in most studied papers considering that cdss needs data in a structured format. the corpora creation is an essential element in this research and constitutes one of the contributions of this work because it was created from non-synthetic data with specific preprocessing needs. other contributions presented in this work are the corpus de-identification and enrichment process, described in the following sections. the evaluation and performance comparison of several machine learning and deep learning classifiers also brings contributions from this work. 2 material and methods this section presents the methodological aspects adopted in this work. this work aims to evaluate text classification techniques to support health professionals’ needs regarding diagnosis decisions. therefore, text classification experiments were conducted using machine learning and deep learning approaches. these techniques were applied in brazilian portuguese clinical notes corpora obtained from an ehr system in the oncology domain. figure 1 shows the general view containing the elements of the proposed approach for this work. this context is derived from real-world observation, considering actual medical clinics, and expresses health professional needs. fig. 1. general view of the approach full size image the overall process starts with the creation of a record of a medical note by the healthcare professional. this situation generates textual medical records and structured clinical information in the real-world observed cases. in this step, the healthcare professionals use the oncology ehr system [14] to input their observations about the patient, which are recorded in the system’s database. these observations can be composed of free-text and structured data, and both data types are used together to achieve better results. the upper part of fig. 1 describes the primary steps for exploring possible answers to the research question in this work. first, developing a flow of operations that starts with the health professional’s assistance was evaluated, as identified in the flow “load new medical textual records” used together with a support service, applying the classification models studied. the application of the models will generate support for determining a response with suggestions for framing and similar contexts, as represented by the “diagnosis and treatment recommendation” flow that the physician will use. the generation of new information about each patient’s clinical event is stored in the oncology ehr system’s database, as indicated in the “record medical notes” flow. this initial flow consists of a vision of the future use of the system by physicians and healthcare professionals. the other items in fig. 1 were studied and experimented. the first necessary step to generate the corpora and to use the proposed approach in the future was to anonymize and export these records. first, a de-identification process was applied to anonymize the data, preventing the personal identification of each patient and professional. after de-identification, the records were exported, creating the corpus used in text classification approaches. in this step, the textual data was exported along with some structured clinical information. the corpus was annotated, taking advantage of free-text data labels generated by the medical staff when using the ehr system and stored in the database. therefore, this corpus could be considered for some of the classification approaches tasks, such as the training tasks. after the corpus creation and annotation, it was enriched with some structured clinical information. as mentioned previously, the ehr system stores the data as a free-text type or structured type. some of these structured data are the diagnosis and treatment information used to enrich the corpus. the final step involves all the preprocessing steps necessary to train the evaluated classification algorithms. before training the artificial neural network model, the corpus was preprocessed as described in sect. 2. in this step, the corpus annotations and enriched data were assessed. when necessary, textual features were extracted. after that, the corpus and its features were used as input to train the following artificial neural network (ann) models: multilayer perceptron (mlp) and long short-term memory (lstm). part of the corpus was reserved for testing to evaluate each ann model’s performance. we developed experiments combining approaches such as corpus format, new corpus with updated information, document or paragraph processing levels, and different machine learning and deep learning classifiers (as described in sect. 3). the experiments’ results are evaluated in sect. 3.3. as a result of the work overviewed above, creating a corpus with non-synthetic oncological medical notes and implementing a de-identification and enrichment process of the corpus are highlighted. in addition, the evaluation and performance comparison of a machine learning and a deep learning classifier are also highlighted as contributions of this work. in the following sections, each component or step involved in the general approach of fig. 1 is detailed. the machine learning and deep learning architectures will be described, along with some parametrization aspects, complementary information on the feature engineering, and the general view of the planned experiments set. fig. 2. overview of the model applied in this research full size image the additional corpora preprocessing, the feature extraction details, and the model specifications are described in the following subsections. corpus preprocessing. after the corpora creation process, it was necessary to preprocess them to normalize the text. the same preprocessing was performed for both per-clinical-event and per-patient corpora types. the text analysis and preprocessing techniques applied are described below. fig. 3. a sample of a corpus full size image before the text preprocessing, it was possible to observe that a small group of diagnoses concentrates on the most frequent occurrences. hence the diagnoses with less than 50 occurrences were joined into a single group called “outros” (“others”). furthermore, to evaluate the neural network’s performance according to the dataset sparsity, a new version of the dataset was created with the 12 most occurring diagnoses. the following tasks were performed: tokenization: split the text into tokens that correspond to words; stop-words filtering: removal of most common words in the brazilian portuguese language, punctuation, and special characters; case folding: conversion all words to lowercase. an additional manual analysis was done on the corpus in the per-clinical-events corpora. the text was assessed to understand how it could be transformed to improve ann algorithms. repeated medical notes were removed from several annotation labels. these notes could weaken the representation strength of the corpus. a complementary experiment was performed to evaluate how this step leveraged the results of the classifiers. as described in sect. 3.1, a significant improvement was achieved by applying this step. feature extraction. the text from medical notes must be transformed into a structure that the classifiers could use. for that reason, the bag-of-words (bow) method was used, which is a representation that turns arbitrary text into fixed-length vectors by counting how many times each word appears. this representation is useful to be used by the classifier algorithms. this work used the medical notes bag-of-words (bow) to extract the features to be used in machine learning and deep learning training. in the per-clinical-event corpora, the bow was generated for each medical note. likewise, the per-patient corpora were generated for each patient with all their medical notes. before creating the medical notes bow, the text was normalized as described in the preprocessing sect. 2. this preprocessing step aims to reduce the number of useless words, special characters, and punctuations, which would not make a difference in the classifier model’s training. it also helped to reduce the computational effort to create the bow. the bow applied to the medical notes resulted in a sparse representation, i.e., the vector sequence of numbers representing each word contained too many zeros. the principal component analysis (pca) technique was applied to reduce the data sparsity. the pca technique converts a set of observations of possibly correlated features into values of linearly uncorrelated features. the pca with 500 features was used. machine learning and deep learning architectures. this work applied machine learning and deep learning methods for the text classification task, comparing their results. several machine learning classification algorithms were applied to evaluate which one had the best performance. furthermore, an lstm deep learning algorithm was also applied to compare traditional machine learning and a deep learning recurrent neural network. the following machine learning algorithms were evaluated: multilayer perceptron (mlp) neural network [15]; logistic regression [16]; decision tree classifier [17]; random forest classifier [18]; k-nearest neighbors (knn) classifier [19]. furthermore, a long-short term memory (lstm) deep learning experiment was also performed. the machine learning classification and the deep learning architecture used are described, regarding their algorithms and their theoretical background, in the following papers [20,21,22]. the datasets were divided into two groups: one with 80% of the data to train the models and 20% of the data to test the models. the data were shuffled to keep the categories’ proportion, and then they were divided as aforementioned. the machine learning algorithms were implemented using scikit-learnfootnote 1, and the deep learning lstm was performed using keras libraryfootnote 2, both were implemented in python. in the first set of experiments, seven tests were performed, with the following architecture details: an mlp with one hidden layer with 500 neurons; an mlp with two hidden layers with 800 and 500 neurons; a logistic regression classifier; a decision tree with a maximum of twenty levels and three samples by leaf; a random forest with a maximum of twenty levels and three samples by leaf; an extra trees with a maximum of twenty levels and three samples by leaf; a knn classifier with a unitary k. the second set of experiments was performed this time with the 12 most occurring diagnoses in the dataset. for this experiment, the best-performing machine learning was selected to compare with an lstm deep learning recurrent neural network. the machine learning algorithm had the following architecture: an mlp with two hidden layers with 800 and 500 neurons. the deep learning algorithm had the following architecture: an lstm with the library keras built on top of tensorflow in a python implementation, in which the parametrization used was composed of batch size 128, the dropout rate of 0.2, validation split of 0.2, optimizer with adam, loss measure was categorical cross-entropy. also, to prevent overfitting, earlystopping was used. standard values for the parameters were used in these experiments, as described in the literature. all the described models were evaluated using standard metrics indicated in the literature, such as accuracy and macro and weighted f1 scores. 3 experiments and results several machine learning classifiers and a deep learning recurrent neural network were applied in this work’s experiments. the main objective of these experiments was to address the main research question and identify a possible workflow to use the dataset and text classification algorithms to evaluate potential support for healthcare professionals. the dataset used was composed of an arrangement of the available options, using the per-clinical-event and the per-patient versions. the complete dataset with several machine learning text classification algorithms was used in the first step. both datasets (per clinical event and per-patient) were used in this experiment. in the second step, a new experiment was carried out involving the per-patient dataset and the algorithms mlp and lstm. the per-patient dataset was chosen in the second step because all patient’s clinical notes were joined into a single record, which would perform better considering the lstm’s ability to process entire sequences of data. all methods were executed on a partition, then repartitioned and reran all methods to get the mean accuracy. therefore, two main experiments were performed: a) machine learning several machine learning classifiers have been experimented with and their performance compared (described in sect. 3.1); b) deep learning an experiment with a deep learning recurrent neural network was performed (described in sect. 3.2). to measure performance in the experiments, we use different metrics in this study: accuracy, macro, and weighted f1 score [20]. the predicted output as true positive (tp) indicates text classified as correct, true negative (tn) when classified incorrectly. false positive (fp) if a text correctly indicates not belonging to the class. similarly, a false negative (fn) is the text is classified incorrectly. the accuracy describes the overall performance of the classifier and mathematical accuracy expressed as below in eq. (1): $$\begin{aligned} \begin{array}{rcl} \text {accuracy} & = & \frac{ tp + tn }{tp + tn + fp + fn} \end{array} \end{aligned}$$ (1) the f1-score it is a harmonic measure between precision and sensitivity, expressed in the eq. (2): $$\begin{aligned} \begin{array}{rcl} \text {f1-score} & = & \frac{ 2*tp}{2 * tp + fn + fp} \end{array} \end{aligned}$$ (2) the macro f1-score is defined as the mean of class-wise/label-wise f1-scores, in eq. (3), where i is the class/label index and n the number of classes/labels: $$\begin{aligned} \begin{array}{rcl} \text {macro f1-score} & = & \frac{1}{n} \sum _{i=0}^{n} {\text {f1-score}_i} \end{array} \end{aligned}$$ (3) in weighted-average f1-score, we weight the f1-score of each class by the number of samples from that class, such as in eq. (4), where i is the class index and n the number of classes and s the number of elements in the class. $$\begin{aligned} \begin{array}{rcl} \text {weighted f1-score} & = & \frac{1}{n} \sum _{i=0}^{n} {\text {f1-score}_i * s_i} \end{array} \end{aligned}$$ (4) 3.1 machine learning experiments the per-clinical-event corpus (described in sect. 2) of the smallest clinic in the preprocessed and raw versions were used to perform the classifiers. this is the clinic database and contains 3.308 clinical notes, and 397 distinct patients. the preprocessed dataset was used first with the machine learning classifiers, described in sect. 2. table 1. machine learning classifiers experiments results.full size table to perform the experiments, the dataset was randomly divided into two parts: 80% for training and 20% for testing. a shuffle method was used to generate these two parts, and a different set of training and testing datasets were created each time it was performed. hence different classifiers’ metrics were obtained, but it always kept the performance order. the mean accuracy, macro f1 score, and weighted f1 score of each classifier are presented in table 1. these experiments were performed to evaluate which machine learning classifier had the best performance. according to table 1, the mlp 2 classifier achieved the best accuracy, macro f1, and weighted f1 scores. these results are evaluated in sect. 3.3. an additional experiment evaluated how the dataset’s structure and preprocessing step (sect. 2) leveraged the classifiers’ performance. this experiment used the same dataset from the clinic with the smalest dataset. the preprocessed and raw versions of the per-clinical-event dataset, plus the preprocessed per-patient dataset, were used with the best performance classifier. according to table 1, the mlp 2 classifier had the best performance and was used in this experiment. table 2 presents the mean accuracy of the mlp 2 classifier with preprocessed and raw versions of the per-clinical-event dataset, plus the preprocessed per-patient dataset. table 2. comparison of the mlp 2 classifier’s performance with the per-clinical-event dataset in raw and preprocessed versions, plus the per-patient preprocessed dataset.full size table these results are evaluated in sect. 3.3, exploring the improvements obtained with the integration of the clinical events in a more complete view of the patient history. as it can be observed in table 2, the mean accuracy improves with more complete patient data. 3.2 deep learning experiments in this set of experiments, the following machine learning and deep learning classifiers were tested: the machine learning classifier that best performed (the mlp 2), according to sect. 3.1; an lstm (long-short term memory) deep learning recurrent neural network. the per-patient corpus (described in sect. 2) of the smallest clinic in the preprocessed version was used to perform the classifiers. the per-patient corpus was chosen because all patient’s clinical notes were joined into a single registry, which would perform better considering the lstm ability to process entire sequences of data. table 3. mlp 2 and lstm classifiers performed with its performance.full size table table 3 presents the mean accuracy, macro f1 score and weighted f1 score of the mlp 2 and the lstm classifiers. these results are evaluated in sect. 3.3. 3.3 results evaluation several experiments were performed to understand the behavior of the selected machine learning and deep learning classifiers with the corpora created and preprocessed (sect. 2). first, a set of experiments with seven machine learning classifiers were performed with the per-clinical-event corpus, according to sect. 3.1. considering the mean accuracy, macro f1, and weighted f1 scores, the classifier that best performed was the mlp 2, as seen in fig. 4. fig. 4. machine learning classifiers’ performance chart. full size image preparing the corpus to be used in machine learning and deep learning classifiers was an important step. an experiment was performed with the mlp 2 classifier, the preprocessed and raw versions of the per-clinical-event dataset, and the preprocessed per-patient dataset (as described in sect. 2). the performance of each is presented in table 2. this table shows a significant improvement in the mlp 2 classifier performance with the preprocessing of the dataset (sect. 2), in both per-clinical-event and per-patient datasets. furthermore, the per-patient corpus performed better than the per-clinical-events corpus. for that reason, the next experiments used the per-patient corpus. fig. 5. the performance chart of the best machine learning classifier and deep learning classifier. full size image after the evaluation of the machine learning classifier that best performed, the mlp 2 classifier was selected. a new experiment was performed, to compare the mlp 2 classifier with an lstm recurrence neural network, using the preprocessed per-patient corpus (as described in sect. 3.2). figure 5 shows that the mlp 2 performed better than the lstm classifier, even though the latter is a more recent neural network. this result can be associated with the fact that this experiment used the smallest per-patient corpus, and deep learning algorithms perform better on large datasets. 4 conclusion with a large amount of information generated daily in healthcare, it is unsuitable for humans to process it manually. furthermore, much of this information is recorded as unstructured data, transforming it into a hard task. it is necessary to develop tools to help healthcare professionals to deal with it, automate the classification and extraction of information from medical notes, and enable this information to be processed by machines. this work created the corpus based on non-synthetic oncological medical notes from an oncology ehr system database. this corpus creation process allowed the identification of various preprocessing steps (as described in sect. 2), and specific term treatment to improve overall results. the experiments achieved good accuracy, especially mlp machine learning and lstm deep learning methods, showing possibilities of using these resources for medical notes text classification. therefore, this set of tasks provided strong support indicating resources and processes to text classification in the specific context of health professional support. artificial intelligence, special machine learning, and deep learning algorithms have been widely applied in several industries, sometimes surpassing human accuracy. in the healthcare industry, several processes can be improved by ai, leveraging healthcare professionals. in the oncology area, diagnosis and treatment decision-making is one of these complex processes that ai algorithms can aid. considering this work’s development, the following list of future steps is suggested: create larger corpus with the medical notes from several oncology clinic databases and conduct new experiments; enhance the corpus preprocessing step by removing low-frequency words, spell checker, replacing acronyms and abbreviations by its standard word; create a domain-specific word embedding from the corpora and apply it with the classifiers. another option is to use general-purpose word embeddings and fine-tune it with the corpora; improve the enrichment process with more structured data available from the oncology ehr system, such as prescribed medications, patient’s problems, and allergies; tune the implemented classifiers and try different versions, such as a bidirectional lstm (bi-lstm); integrate the implemented classifiers with the oncology ehr system to obtain feedback from the healthcare professionals about their accuracy, and suggest the diagnosis based on the patient’s clinical history. notes 1.https://scikit-learn.org/. 2.https://keras.io/. references sabra, s., alobaidi, m., malik, k.m., sabeeh, v.: performance evaluation for semantic-based risk factors extraction from clinical narratives. in: ieee 8th annual computing and communication workshop and conference (ccwc). las vegas, nv, usa, vol. 2018, pp. 695–701 (2018). https://doi.org/10.1109/ccwc.2018.8301742 glatzer, m., panje, c.m., sirén, c., cihoric, n., putora, p.m.: decision making criteria in oncology. oncology 98(6), 370–378 (2020). epub 2018 sep 18. pmid: 30227426. https://doi.org/10.1159/000492272 reyes-ortiz, j.a., gonzález-beltrán, b.a., gallardo-lópez, l.: clinical decision support systems: a survey of nlp-based approaches from unstructured data. in: 26th international workshop on database and expert systems applications (dexa). valencia, spain vol. 2015, pp. 163–167 (2015). https://doi.org/10.1109/dexa.2015.47 alemzadeh, h., devarakonda, m.: an nlp-based cognitive system for disease status identification in electronic health records. in: 2017 ieee embs international conference on biomedical & health informatics (bhi), orlando, fl, usa, pp. 89–92 (2017). https://doi.org/10.1109/bhi.2017.7897212 meskó, b.: the guide to the future of medicine: technology and the human touch. in: webicina kft (2014) google scholar  zhang, r., ma, s., shanahan, l., munroe, j., horn, s., speedie, s.: automatic methods to extract new york heart association classification from clinical notes. in: 2017 ieee international conference on bioinformatics and biomedicine (bibm), kansas city, mo, usa, pp. 1296–1299 (2017). https://doi.org/10.1109/bibm.2017.8217848 chen, x., xie, h., wang, f., et al.: a bibliometric analysis of natural language processing in medical research. bmc med. inform. decis. mak. 18(suppl 1), 14 (2018). https://doi.org/10.1186/s12911-018-0594-x article  google scholar  shickel, b., et al.: deep ehr: a survey of recent advances in deep learning techniques for electronic health record (ehr) analysis. ieee j. biomed. health inform. 22(5), 1589–1604 (2017) article  google scholar  kreimeyer, k., et al.: natural language processing systems for capturing and standardizing unstructured clinical information: a systematic review. j. biomed. inform. 73, 14–29 (2017). epub 2017 jul 17. pmid: 28729030; pmcid: pmc6864736. https://doi.org/10.1016/j.jbi.2017.07.012 hunt, d.l., haynes, r.b., hanna, s.e., smith, k.: effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. jama 280(15), 1339–1346 (1998). pmid: 9794315. https://doi.org/10.1001/jama.280.15.1339 bucur, a., van leeuwen, j., cirstea, t.c., graf, n.: clinical decision support framework for validation of multiscale models and personalization of treatment in oncology. in: 13th ieee international conference on bioinformatics and bioengineering, chania, greece, pp. 1–4 (2013). https://doi.org/10.1109/bibe.2013.6701695 polpinij, j.: the cancerology ontology: designed to support the search of evidence-based oncology from biomedical literatures. in: 24th international symposium on computer-based medical systems (cbms). bristol, uk, pp. 1–6 (2011). https://doi.org/10.1109/cbms.2011.5999168 wang, y., et al.: clinical information extraction applications: a literature review. j. biomed. inform. 77, 34–49 (2018). issn 1532–0464. https://doi.org/10.1016/j.jbi.2017.11.011 interprocess: interprocess gemed oncology oncological management system (2019). www.interprocess.com.br/en/gemed-oncology/ naraei, p., abhari, a., sadeghian, a.: application of multilayer perceptron neural networks and support vector machines in classification of healthcare data. in: future technologies conference (ftc). san francisco, ca, usa, vol. 2016, pp. 848–852 (2016). https://doi.org/10.1109/ftc.2016.7821702 lemon, s.c., roy, j., clark, m.a., friedmann, p.d., rakowski, w.: classification and regression tree analysis in public health: methodological review and comparison with logistic regression. ann. behav. med. 26(3), 172–181 (2003). pmid: 14644693. https://doi.org/10.1207/s15324796abm2603_02 lavanya, d., rani, k.u.: ensemble decision tree classifier for breast cancer data. int. j. inf. technol. convergence serv., 2(1), 17 (2012) google scholar  dubrava, s., et al.: using random forest models to identify correlates of a diabetic peripheral neuropathy diagnosis from electronic health record data. pain med. 18(1), 107–115 (2017). pmid: 27252307. https://doi.org/10.1093/pm/pnw096 tayeb, s., et al.: toward predicting medical conditions using k-nearest neighbors. in: 2017 ieee international conference on big data (big data), boston, ma, usa, pp. 3897–3903 (2017). https://doi.org/10.1109/bigdata.2017.8258395 ul haq, a., li, j.p., memon, m.h., nazir, s., sun, r.: a hybrid intelligent system framework for the prediction of heart disease using machine learning algorithms. hindawi mobile inf. syst. (2018). https://doi.org/10.1155/2018/3860146 haq, a.u., et al.: intelligent machine learning approach for effective recognition of diabetes in e-healthcare using clinical data. sensors 20, 2649 (2020). https://doi.org/10.3390/s20092649 tai, k.s., socher, r., manning, c.d.: language processing, improved semantic representations from tree structured long short-term memory networks. in: proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing of the asian federation of natural, pp. 1556–1566 (2015) google scholar  download references author information authors and affiliations universidade do vale do rio dos sinos, rio grande do sul, brazil diego pinheiro da silva, marco antonio schwertner & sandro josé rigo pontifícia universidade católica do rio grande do sul, rio grande do sul, brazil william da rosa fröhlich authors diego pinheiro da silvaview author publications search author on:pubmed google scholar william da rosa fröhlichview author publications search author on:pubmed google scholar marco antonio schwertnerview author publications search author on:pubmed google scholar sandro josé rigoview author publications search author on:pubmed google scholar corresponding author correspondence to diego pinheiro da silva . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da silva, d.p., fröhlich, w.d.r., schwertner, m.a., rigo, s.j. (2023). clinical oncology textual notes analysis using machine learning and deep learning. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_10 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords artificial intelligence natural language processing oncology deep learning machine learning healthcare publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature deep reinforcement learning for voltage control in power systems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper deep reinforcement learning for voltage control in power systems conference paper first online: 12 october 2023 pp 213–227 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) deep reinforcement learning for voltage control in power systems download book pdf download book epub mauricio w. barg  orcid: orcid.org/0000-0003-2172-16279,10, barbara s. rodrigues  orcid: orcid.org/0009-0003-9540-666x9, gabriela t. justino  orcid: orcid.org/0000-0001-6154-35259, kleyton pontes cotta  orcid: orcid.org/0000-0003-4332-56419, hugo r. v. portuita  orcid: orcid.org/0009-0004-2205-12879, flávio l. loução jr.  orcid: orcid.org/0009-0009-1615-32839, iran pereira abreu  orcid: orcid.org/0009-0007-5909-955011 & … antônio carlos pigossi jr.  orcid: orcid.org/0009-0005-7690-012211  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 532 accesses abstract the great complexity and size of electrical power systems makes their operation and control a challenging task. maintaining a stable voltage profile to assure the security and stability of the system is one of many tasks that must be conducted daily by power system operators and its automatic control equipment. this work proposes a deep reinforcement learning framework for controlling the equipment responsible for keeping the voltages across the system buses within their limits. more specifically, a smart agent that is capable of deciding the best course of action in order to keep the system’s voltages within a specified range while taking into account system’s conditions is proposed. besides the traditional deep reinforcement learning approach, three novel reinforcement learning variations named windowed, ensemble and windowed ensemble q-learning, which alter the agent’s learning process for voltage control, are presented and tested on ieee 13, 37 and 123 bus systems, simulated on opendss. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others active voltage control method of distributed photovoltaic based on safe deep reinforcement learning chapter © 2025 reinforcement learning for decision-making and control in power systems chapter © 2023 modeling and deep reinforcement learning based control parameter tuning for voltage source converter in a renewable energy generation system article 16 february 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. energy grids and networks energy management smart infrastructure stochastic learning and adaptive control artificial intelligence electrical power engineering 1 introduction electric power systems (eps) are complex systems with both physical and digital resources that must work together in order to deliver electricity safely and efficiently to all kinds of customers be them industrial, commercial or residential [17]. over the last decades the demand for energy is constantly increasing as people wish for better quality of life [22]. with this growth, maintaining the quality of service (qos) and system safety becomes harder and new tools and resources must be included into the process of operating and controlling the system. mainly, the operation of epss is executed by several trained professionals called system operators (so) who must constantly observe the system’s condition and conduct different tasks such as dispatch of generators, frequency control, fault mitigation, among others [29]. one of these tasks known as voltage control regards keeping voltage on all system buses between certain limits which are usually defined by regulatory associations and take into account the systems’ correct functioning. as the demand fluctuates through the day, the voltages on system’s buses also change and operators must maneuver and switch several equipment in order to keep it under control. as most other tasks, voltage control must be executed by the sos in real time who usually must react in a short time frame as the system’s resources may be at stake. voltage can also be controlled by automatic control equipment that react to voltage fluctuations according to some embedded logic using control mechanisms that monitor the voltage at certain system’s points and issues control commands that keep it in between the predefined limits [3]. due to the need for fast reaction, operators usually do not have time for detailed analysis of their actions. they rely on their experience and on actions that have worked in the past, but this may not be optimal because the system varies due to various unknown factors. automatic control equipment also has fixed logic that does not consider the constantly changing dynamics of the system. over the years several different techniques have been proposed in order to address the voltage control problem. as it is formally known, volt-var control has seen applications of a plethora of different methods, such as classical optimization [24, 25], meta-heuristics [20, 21], classic control [1, 2], neural networks [8], fuzzy logic [15, 16], etc. also, it has been successfully applied both to distribution and transmission systems. there are many different approaches to voltage control since it can be seen as both a planning and a real-time operation problem [6, 7, 13, 14] [13]. online control uses different techniques and methodologies [11, 12, 23, 32], from classical control to reinforcement learning. usually, equipment such as shunt capacitor banks and oltc transformers are controlled, but other digital technologies and distributed generation resources are also used. recent studies have increasingly explored the use of reinforcement learning in energy systems, with different training techniques and architectures [30], q-learning [4, 28], and deep reinforcement learning [5, 31]. bus voltages and branch power flows are used as state representations. most voltage control works focus on offline scenarios and use a variety of methods, mainly capacitor banks and transformer taps. when controlled online, adaptations are necessary, and reinforcement learning is a well-suited technique. this work proposes a framework that uses reinforcement learning to control the system’s voltage in real-time, taking into account equipment constraints and different system parameters. three new methodologies that consider system complexity are also proposed. the methodologies are tested on three ieee distribution circuits with 13, 37, and 123 buses. 2 background in this section, we will provide a brief description of the voltage control problem and discuss the application of reinforcement learning as a solution to the problem. 2.1 voltage control the power system experiences fluctuations in consumption and generation, causing voltage to vary and potentially violate safety limits set by authorities. power system operators have to make quick decisions to keep voltage within limits by maneuvering equipment or shedding loads, but this can sacrifice assertiveness. due to the complexity of the power system, operators rely on previous experiences when controlling voltage, causing some equipment to be overused. however, operators are not fully responsible for controlling voltages. there are equipment that have automatic control mechanisms and monitor voltages at specific points in the system, but these usually rely on simple feedbacks and do not always provide the best control decisions. furthermore, as the system changes along the day, the parameter configuration of these controls would be better if adjusted, which most of the time, doesn’t happen. with this in mind, the tool proposed in this work aims to be an automatic system operator that is capable of replacing both human operators and equipment control by deciding and taking the best course of action in order to keep voltages inside its limits. in later sections, the voltage control problem will be described further with more technical details. 2.2 deep reinforcement learning according to [27], reinforcement learning is a machine learning paradigm based on interactions between an agent and an environment. the agent interacts with the environment through actions and observations, evaluating the effect of its actions on the environment and seeking the actions that lead to the most desired outcomes. essential elements for reinforcement learning include a state-action map, a policy to guide actions, a reward signal to evaluate the effect of actions on the environment, and intrinsic values for the states of the environment that represent the benefit of being in those states. by repeating the process of interaction, observation, and adaptation, it is expected that the agent will learn the best actions in an activity or process (fig. 1). fig. 1. reinforcement learning process [27]. full size image in environments where the number of states are small, this method of exploring and learning is very effective. however, as the number of states grow in an environment, experiencing all states becomes unfeasible. in this context, the deep reinforcement learning methodology arises [9, 10, 18, 19]. this variation allows the agent to generalize experiences for states it has not seen before by using a deep neural network in further sections, the application of reinforcement learning to the voltage control problem will be discussed. 3 problem definition as indirectly explained on previous sections: voltage control is the act of operating and configuring different control equipment in the right moments in order to keep bus’ voltages within specified limits, which are determined by physical and commercial aspects. more formally, the problem can be written as shown in eq. 1. $$\begin{aligned} \begin{array}{rrrrrr} \min \sum ^t_{t = 0}\sum _{b \in b} distance(v_{bt}, \overline{v}) \\ \text {subject to} \\ v_{-} \le v_{bt} \le v_{+} &{} (a) \\ v_{bt} = f(p, q, [e_{0t}, e_{1t}, ..., e_{nt}]) &{} (b) \\ \ [e_{il}] \le [e_{it}] \le [e_{iu}], i=0...n &{} (c) \\ \ [-1] \le [e_{it+1} e_{it}] \le [+1], i=0...n &{} (d) \\ \end{array} \end{aligned}$$ (1) where t represents the current time step and t the maximum possible time step and n is the total number of equipment. in the real world this time would be continuous as voltage is controlled all the time. although in order to simulate it, the operation period must be discretized. a certain system bus is represented by b and b is the set of all system buses. \(v_{bt}\) is the voltage at bus b on time t and \(\overline{v}\) is the voltage target. for the constraints, in constraint (a), called the voltage limit constraint, \(v_{-}\) and \(v_{+}\) represent the lower and upper voltage limits across all buses and in constraint (b), \(v_{bt}\) is represented as a function of the system’s active (p) and reactive (q) powers and all equipment set-points at time t, which is how the voltage is essentially controlled (by changing equipment set-points). in constraint (c), also known as the equipment set-point constraint, since control equipment have different set-points restrictions, \(e_{il}\) and \(e_{iu}\) represent the minimum and maximum set-points that an equipment \(e_i\) can have at a certain time (t). finally, constraint (d) called the maximum set-point change, represents how much the set-point of an equipment can change from one time step to another. this is to account for delays that an equipment may have in order to change its set-point. it is important to note that the data needed for the optimization (i.e. the voltage on each bus) is obtained online and in real time. that means that \(v_{bt}\) can only be obtained at time step t and not before that. therefore, the optimization process is conducted at each time step, as shown in fig. 2. fig. 2. the optimization process flow. full size image in order to control the voltage, the existing equipment in the network must be manipulated. in electrical networks, there are many kinds of equipment that are capable of regulating voltage: capacitors, reactors, transformers, synchronous condensers, facts equipment, etc. although there is no clear advantage of using one equipment over another, different actions may have different short and long term impacts on the system. mainly, voltage can be controlled by two means: automatic control equipment, which monitor the network and adjust themselves accordingly and by network operators, who choose a certain action (based on experience and/or intuition) which they think is best for the current scenario. the automatic control equipment usually have fixed control logic which is certainly not suitable for every scenario. furthermore, because there’s usually no time to conduct detailed studies, operators tend to rely purely on their own knowledge and past experiences increasing the bias towards certain actions and overusing individual equipment. 4 proposed solution because power systems are composed by multiple discrete and continuous variables, the amount of possible configurations it can attain is quasi-infinite, rendering the classical tabular reinforcement learning not feasible. therefore, a deep reinforcement learning technique is developed to deal with this characteristic. besides the usual deep reinforcement learning (drl) approach, three different methodologies are proposed. these propositions involve slight modifications to the reinforcement learning technique that intend to adapt the procedure to specific characteristics of power system operation. many different kinds of reinforcement learning solutions exist depending on the type of problem that is being solved. for the voltage control problem, q-learning is a well suited method. besides being very simple to implement, it is a model-free method, meaning that the model of the environment does not need to be fully known (which would be very hard for power systems). furthermore, it can account for the effect that an action may have on a future state, that is not immediately achieved after taking said action. although there are many suitable methods to perform reinforcement learning, most of them have limitations, including q-learning, due to the need to create a finite map of state to actions. discretizing the state can be infeasible or lead to inaccuracies if there are many continuous variables involved. deep reinforcement learning overcomes these limitations, allowing the agent to generalize to states never seen before through a deep neural network. in deep q-learning, the neural network’s inputs are every variable that represents the system’s state and its outputs are the “worth” or also called q-values for each action. during the training process, as the agent explores the environment, the neural network is trained with information from the system’s state, the actions taken by the agent and their effects. during operation, the system’s state is simply fed as an input to the network and the action with the highest q-value is chosen. over the years, many advancements have been made in deep q-learning. in this work, two of them are considered: double q-learning and experience replay. both these techniques are better described in [10]. in addition, two new different modifications are proposed to the deep q-learning framework. these modifications do not modify how the learning process works, bur rather change how the techniques are applied to the problem at hand. the first technique will be named windowed q-learning. in problems where episodes are too long, the agent may take longer to learn optimal actions for the entire length of the episode. also, by being long, the episode may have very distinct behaviors on its course. that is, in a certain problem with a constant number of steps per episode, steps i to j may be very different in behavior from steps m to n. the proposed windowed q-learning methodology attempts to address this characteristics by dividing the episode into windows fig. 3(a) and for each window a different agent is trained. that way, each agent only sees states relative to a certain interval and can learn the specificities of each window better when training. during operation, depending on which window the episode falls into, a different agent is consulted for making decisions. fig. 3. proposals q-learning. full size image the second technique will be called ensemble q-learning and takes inspiration from the way that humans learn. because when subjected to different experiences people learn things differently, this technique proposes that multiple agents are trained for the same problem fig. 3(b). these agents can either be equal and rely on the variability of the environment in order to experience things differently or can perceive the environment differently for example by having different reward functions. when operating, every agent is consulted when deciding which action to take. the decision can be made by following several criteria such as averaging every agent’s value for the actions and taking the highest average (eq. 2) or by taking the action with the maximum value over every agent (eq. 3). $$\begin{aligned} a = \text {argmax} \begin{pmatrix}\text {avg}(\phi _{11}, \phi _{12}, \phi _{1j})\\ \text {avg}(\phi _{21}, \phi _{22}, \phi _{2j})\\ \text {avg}(\phi _{i1}, \phi _{i2}, \phi _{ij})\end{pmatrix} \end{aligned}$$ (2) $$\begin{aligned} a = \text {argmax} (\max \phi _{ij}) \; \forall \; i, j \end{aligned}$$ (3) where \(\phi _{ij}\) is the value given to action i by agent j. the third and final technique is a combination of the other two and is called windowed ensemble q-learning. in addition to splitting the episode into windows, for each window multiple agents are trained fig. 3(c). this allows for the combination of both methods’ advantages. one disadvantage may be increased training times. during operation, the actions are also chosen by combining both methods: when the episode moment falls into a certain window, all agents trained for that window are consulted when choosing the action. 4.1 voltage control as a drl problem in this section, the voltage control problem in an electrical system is modeled. episodes have a fixed length of 1440, corresponding to each minute in a 24-hour period, and the system loads are updated at each step. the agent has the option to execute actions in the system, such as increasing or decreasing the stage of a capacitor bank, increasing or decreasing a transformer tap, or doing nothing. states are determined by the total active and reactive loads of the system, the current minute of the simulation, and the states of the transformers and capacitor banks. the state representation has a direct impact on the agent’s learning, and rewards are defined to lead the voltage at all system buses closer to a certain target and keep them within certain operational limits. the rewards are defined following algorithm 1. all learning parameters are detailed in the case study section. algorithm 33 rewards full size image 5 case study in this chapter the proposed techniques will be tested on simulated power systems. the utilized systems are the 13, 37 and 123 bus ieee test circuits [26]. for each system, four models were trained: a pure reinforcement learning model, a windowed model, a ensemble model and a windowed ensemble model. additionally, for comparison effect the system was run without any form of control and also with the control capabilities available at the simulation software (opendss). for every system, the parameters used during training were very similar with the exception of the number of episodes which was 100 for both the 13 and 123 bus systems and 200 for the 37 bus system. for the double deep q-learning with experience replay, the parameters are as follows: the used memory size n was 1024, the minimum number of samples to start training was 512 and the batch size, 256. the discount factor \(\gamma \) was 0.9. for epsilon, instead of keeping it static, a decay technique was used. that way, epsilon decays linearly with the episodes from 1.0 to a minimum of 0.3. fig. 4. neural network structure. full size image the neural network structure is shown on fig. 4 and its learning rate \(\alpha \) was 0.001. the chosen architecture is quite simple when considering its size and activation functions. this is due to the nature of deep reinforcement learning which does not require complex structures in order to predict the action values and also because of the tested systems’ sizes. its input and output layer sizes depends on the system’s characteristics, that is the size of its state and the number of available actions, both of which depends on the number of capacitors and tap changing transformers that can be controlled. this data is shown on table 1. table 1. test circuits’ state and action space sizes.full size table finally, for the rewards as shown in algorithm 1, the values used (\(v_1 = -1\), \(v_2 = 0.7\), \(v_3 = -0.8\), \(v_4 = -0.8\), \(v_5 = 1\), \(v_6 = -1\) and \(v_7 = 0.2\)). the upper and lower voltage limits considered are 1.05 and 0.92 p.u. while the target is 1.0 p.u. an exception is the ensemble approach where two alternative agents with more rigorous limits are used. the first alternative agent uses 1.03 and 0.95 p.u. for the upper and lower limits while the second uses 1.02 and 0.98 p.u. also, for the ensemble agents the “vote” on the best action is conducted by following eq. 2. 6 results and discussion after the training procedure, the agents were tested on the same systems on different days. the days were simulated by randomly choosing a load profile and introducing some random noise into the loads. as stated in the previous section, the agents’ performance is compared to the systems’ without any form of control, with the control present on the system itself and also with a “pure” deep reinforcement learning approach as proposed by [9, 10, 18, 19] and implemented, with slight variations, by [4, 5, 28, 30, 31]. in order to obtain the results, a total of 50 days for each technique and system were simulated. a few metrics are used to show the achieved results: the total number of violations, that is, the number of times the voltage surpassed the limits of 1.05 and 0.92 p.u., the maximum and minimum voltage achieved across the 50 days, the average real power loss across the 50 days and for the reinforcement learning methods, the number of actions the agents took (tables 2a, 2b and 2c). table 2. system results.full size table for the number of actions, the values are only available for the reinforcement learning techniques, since for the no control no actions are taken and for the system control method, it is not possible to obtain this value from the simulation. furthermore, a separate day is simulated in order to closely demonstrate the results and the average system voltage across the day is shown, along with the distribution of the voltages during the day (fig. 5 and 6) and a diagram showing the voltage during the day at each system bus (fig. 7, 8 and 9). fig. 5. average system voltage during the day results. full size image fig. 6. voltage distribution during the day results. full size image regarding the training process, all agents were satisfactorily trained. the training times were varied but the windowed methodology has shown a reduced training time even when compared to the traditional reinforcement learning approach. the results indicate that the proposed techniques are capable of controlling the voltage on power systems. for the 13-bus system, because it’s already a fairly balanced and small system, the results were marginal. nevertheless, the windowed methodology was capable of removing all voltage violations when compared to the control already present on the system while keeping the number of required actions low and not affecting the real power losses significantly. most importantly, fig. 5(a), 6(a), and 7 show also that the system’s voltage profile was improved with the voltages getting closer to the target. fig. 7. ieee 13-bus single-day voltage profile. full size image for the 37-bus system, while the pure reinforcement learning approach actually increased the number of voltage violations, the other three proposed methodologies completely eliminated them. the improvements are clear on the results shown on fig. 5(b), 6(b), and 8. it is possible to see that the voltages are much closer to the target of 1 p.u. also, the voltage profile in general is much closer to \(\pm 1\%\) of the target green areas on fig. 8. in this case, the windowed ensemble methodology has executed the task in the least number of actions. for all the methodologies, the average losses were kept fairly close to its original values. fig. 8. ieee 37-bus single-day voltage profile. full size image fig. 9. ieee 123-bus single-day voltage profile. full size image finally, for the 123-bus system, while there were no violations on the system both with and without control, the proposed methodologies were capable of significantly improving its voltage profile. for this system, the average losses for three of the four methodologies were increased by a significant margin. the ensemble technique, has shown a better performance overall, controlling the voltage satisfactorily while also keeping the losses at a better acceptable value and the number of actions at a minimum. 7 conclusion reinforcement learning has been around for some time and has shown great results across many different scientific and real-world problems. when combined with the power of deep neural networks, deep q-learning can tackle a plethora of problems. in this work, a deep q-learning methodology was proposed to solve the voltage control problem on electrical power systems. besides the regular reinforcement learning approach, three other novel methodologies were proposed with the goal of improving the technique’s performance on this specific problem. the results show that the application of the techniques was successful and has shown great value when compared to the traditional drl approach and also with the systems’ own control. the trained intelligent agents are capable of controlling the system voltage in a completely autonomous way while keeping the number of actions taken low and having little effect on the real power losses. references baran, m.: ming-yung hsu: volt/var control at distribution substations. ieee trans. power syst. 14(1), 312–318 (1999). https://doi.org/10.1109/59.744549. https://ieeexplore.ieee.org/document/744549/ borozan, v., baran, m., novosel, d.: integrated volt/var control in distribution systems. in: 2001 ieee power engineering society winter meeting. conference proceedings (cat. no. 01ch37194), columbus, oh, usa, vol. 3, pp. 1485–1490. ieee (2001). https://doi.org/10.1109/pesw.2001.917328. http://ieeexplore.ieee.org/document/917328/ corsi, s.: voltage control and protection in electrical power systems: from system components to wide-area control. springer, london (2015). https://doi.org/10.1007/978-1-4471-6636-8 book  google scholar  custódio, g., ochoa, l., trindade, f., alpcan, t.: using q-learning for oltc voltage regulation in pv-rich distribution networks (2020) google scholar  diao, r., wang, z., shi, d., chang, q., duan, j., zhang, x.: autonomous voltage control for grid operation using deep reinforcement learning (2019). http://arxiv.org/abs/1904.10597 franco, j.f., rider, m.j., lavorato, m., romero, r.: a mixed-integer lp model for the optimal allocation of voltage regulators and capacitors in radial distribution systems. int. j. electr. power energy syst. 48, 123–130 (2013). https://doi.org/10.1016/j.ijepes.2012.11.027. https://linkinghub.elsevier.com/retrieve/pii/s0142061512006801 gallego, r., monticelli, a., romero, r.: optimal capacitor placement in radial distribution networks. ieee trans. power syst. 16(4), 630–637 (2001). https://doi.org/10.1109/59.962407. https://ieeexplore.ieee.org/document/962407/ gu, z., rizy, d.: neural networks for combined control of capacitor banks and voltage regulators in distribution systems. ieee trans. power deliv. 11(4), 1921–1928 (1996). https://doi.org/10.1109/61.544277. https://ieeexplore.ieee.org/document/544277/ van hasselt, h., guez, a., silver, d.: deep reinforcement learning with double q-learning (2016). http://arxiv.org/abs/1509.06461 hessel, m., et al.: rainbow: combining improvements in deep reinforcement learning. in: proceedings of the aaai conference on artificial intelligence, vol. 32 (2018) google scholar  homaee, o., zakariazadeh, a., jadid, s.: real-time voltage control algorithm with switched capacitors in smart distribution system in presence of renewable generations. int. j. electr. power energy syst. 54, 187–197 (2014). https://doi.org/10.1016/j.ijepes.2013.07.010. http://linkinghub.elsevier.com/retrieve/pii/s0142061513003074 hu, z., wang, x., chen, h., taylor, g.: volt/var control in distribution systems using a time-interval based approach. iee proc.-gener. transm. distrib. 150(5), 548 (2003). https://doi.org/10.1049/ip-gtd:20030562 article  google scholar  khanabadi, m., ghasemi, h., doostizadeh, m.: optimal transmission switching considering voltage security and n-1 contingency analysis. ieee trans. power syst. 28(1), 542–550 (2013). https://doi.org/10.1109/tpwrs.2012.2207464. https://ieeexplore.ieee.org/document/6253283/ levitin, g., kalyuzhny, a., shenkman, a., chertkov, m.: optimal capacitor allocation in distribution systems using a genetic algorithm and a fast energy loss computation technique. ieee trans. power deliv. 15(2), 623–628 (2000). https://doi.org/10.1109/61.852995. https://ieeexplore.ieee.org/document/852995/ liang, r.h., chen, y.k., chen, y.t.: volt/var control in a distribution system by a fuzzy optimization approach. int. j. electr. power energy syst. 33(2), 278–287 (2011). https://doi.org/10.1016/j.ijepes.2010.08.023. https://linkinghub.elsevier.com/retrieve/pii/s0142061510001596 liu, y., zhang, p., qiu, x.: optimal volt/var control in distribution systems. int. j. electr. power energy syst. 24(4), 271–276 (2002). https://doi.org/10.1016/s0142-0615(01)00032-1. https://linkinghub.elsevier.com/retrieve/pii/s0142061501000321 meier, a.v.: electric power systems: a conceptual introduction. wiley survival guides in engineering and science, ieee press: wiley-interscience, hoboken (2006). oclc: ocm62616191 google scholar  mnih, v., et al.: playing atari with deep reinforcement learning. arxiv preprint arxiv:1312.5602 (2013) mnih, v., et al.: human-level control through deep reinforcement learning. nature 518(7540), 529–533 (2015) google scholar  niknam, t.: a new approach based on ant colony optimization for daily volt/var control in distribution networks considering distributed generators. energy convers. manag. 49(12), 3417–3424 (2008). https://doi.org/10.1016/j.enconman.2008.08.015. https://linkinghub.elsevier.com/retrieve/pii/s0196890408003087 niknam, t., firouzi, b.b., ostadi, a.: a new fuzzy adaptive particle swarm optimization for daily volt/var control in distribution networks considering distributed generators. appl. energy 87(6), 1919–1928 (2010). https://doi.org/10.1016/j.apenergy.2010.01.003. https://linkinghub.elsevier.com/retrieve/pii/s030626191000005x rashid, m.h. (ed.): electric renewable energy systems. elsevier/ap, academic press is an imprint of elsevier, amsterdam (2016) google scholar  ribeiro, l.c., schumann minami, j.p.o., bonatto, b.d., ribeiro, p.f., de souza, a.c.z.: voltage control simulations in distribution systems with high penetration of pvs using the opendss. in: 2018 simposio brasileiro de sistemas eletricos (sbse), pp. 1–6. ieee. https://doi.org/10.1109/sbse.2018.8395639. https://ieeexplore.ieee.org/document/8395639/ roytelman, i., wee, b., lugtu, r.: volt/var control algorithm for modern distribution management system. ieee trans. power syst. 10(3), 1454–1460 (1995). https://doi.org/10.1109/59.466504. https://ieeexplore.ieee.org/document/466504/ saric, a.t., stankovic, a.m.: a robust algorithm for volt/var control. in: 2009 ieee/pes power systems conference and exposition, seattle, wa, usa, pp. 1–8. ieee (2009). https://doi.org/10.1109/psce.2009.4840211. https://ieeexplore.ieee.org/document/4840211/ schneider, k.p., et al.: analytic considerations and design basis for the ieee distribution test feeders. ieee trans. power syst. 33(3), 3181–3188 (2018). https://doi.org/10.1109/tpwrs.2017.2760011. https://ieeexplore.ieee.org/document/8063903/ sutton, r.s., barto, a.g.: reinforcement learning: an introduction. adaptive computation and machine learning series, 2nd edn. the mit press, cambridge (2018) google scholar  vlachogiannis, j., hatziargyriou, n.: reinforcement learning for reactive power control. ieee trans. power syst. 19(3), 1317–1325 (2004). https://doi.org/10.1109/tpwrs.2004.831259. https://ieeexplore.ieee.org/document/1318666/ wood, a.j., wollenberg, b.f., sheblé, g.b.: power generation, operation, and control, 3rd edn. hoboken, wiley-ieee (2013) google scholar  xu, h., domínguez-garcía, a.d., sauer, p.w.: optimal tap setting of voltage regulation transformers using batch reinforcement learning. ieee trans. power syst. 35(3), 1990–2001 (2019). https://doi.org/10.1109/tpwrs.2019.2948132. https://arxiv.org/abs/1807.10997 yang, q., wang, g., sadeghi, a., giannakis, g.b., sun, j.: two-timescale voltage control in distribution grids using deep reinforcement learning. arxiv:1904.09374 (2019) zhang, w., liu, y.: multi-objective reactive power and voltage control based on fuzzy optimization strategy and fuzzy adaptive particle swarm. ieee trans. power syst. 30(9), 525–532 (2019). https://doi.org/10.1016/j.ijepes.2008.04.005. https://linkinghub.elsevier.com/retrieve/pii/s0142061508000380 download references acknowledgments the authors would like to thank companhia de transmissão de energia elétrica paulista (isa-cteep) for the financial support to project pd-00068-0044-2019 intelligent real-time decision support tool for transmission operations centers, developed under the research and development program of the national electric energy agency (aneel r &d), which the engineering company carried out radix engenharia e software s/a, rio de janeiro, brazil. author information authors and affiliations research, development and innovation dept (rd &i), radix engineering and software development, rio de janeiro, rj, brazil mauricio w. barg, barbara s. rodrigues, gabriela t. justino, kleyton pontes cotta, hugo r. v. portuita & flávio l. loução jr. informatics department, puc-rj, rio de janeiro, rj, brazil mauricio w. barg real time operation, isa-cteep, jundiaí, sp, brazil iran pereira abreu & antônio carlos pigossi jr. authors mauricio w. bargview author publications search author on:pubmed google scholar barbara s. rodriguesview author publications search author on:pubmed google scholar gabriela t. justinoview author publications search author on:pubmed google scholar kleyton pontes cottaview author publications search author on:pubmed google scholar hugo r. v. portuitaview author publications search author on:pubmed google scholar flávio l. loução jr.view author publications search author on:pubmed google scholar iran pereira abreuview author publications search author on:pubmed google scholar antônio carlos pigossi jr.view author publications search author on:pubmed google scholar corresponding author correspondence to gabriela t. justino . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper barg, m.w. et al. (2023). deep reinforcement learning for voltage control in power systems. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_15 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_15 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords voltage control deep reinforcement learning ensemble learning publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature dog face recognition using deep features embeddings | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper dog face recognition using deep features embeddings conference paper first online: 12 october 2023 pp 125–139 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) dog face recognition using deep features embeddings download book pdf download book epub joão p. b. andrade9, leonardo f. costa9, lucas s. fernandes9, paulo a. l. rego9 & … josé g. r. maia10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 589 accesses abstract over 470 million dogs are kept as pets around the world. dogs are owned at an average number of \(1.6\%\) per household. the us has the most dog pets, where about \(68\%\) of households own at least one pet. lost and missing dogs are a severe source of suffering and problems for their families. so, this paper addresses the problem of facial dog identification. this technology can benefit many applications, such as handling the missing pet problem, granting pets access to their houses, more intelligent zoonosis control, pet health care, and tracking stray pets. we evaluate a residual convolutional neural network, specifically resnet-34, for facial identification in dogs. we tested in dogfacenet and flickr-dog datasets with and without two face preprocessing techniques: a central crop and an aligned facial extraction. experimental results show promising results surpassing the state-of-the-art: \(97.6\%\) and \(82.8\%\) accuracies for dogfacenet and flickr-dog, respectively. moreover, we also provide recall metrics for the best models. supported by organization funcap. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a deep learning approach for dog face verification and recognition chapter © 2019 dogface detection and localization of dogface’s landmarks chapter © 2019 where is my puppy? retrieving lost dogs by facial features article 03 september 2016 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. biometrics computer vision machine learning neural decoding object recognition artificial intelligence 1 introduction effective identification and tracking of pets is a valuable technology for modern society. for example, many countries have more dogs than children, according to recent official data releases by [15]. stray animals, in general, could pose a serious hazard to other species and human health in urban areas, especially to children. on the other hand, missing pets is a frequent problem whose conditions of occurrence are minimal. since dogs are autonomous animals, it only takes a little carelessness to make them run away from home, especially when there are festive events with fireworks. on top of that, the human ability to discern animals by their traits is reasonably limited; consequently, it is difficult to identify and track animals effectively with the naked eye [4]. in [7], the authors sought to understand the process of coping with grief experienced by people who experienced the loss of their pets. they concluded that these processes have characteristics similar to those present in mourning processes arising from the loss of significant people. that said, canine recognition can be an attractive alternative for smart cities to deal with complex problems related to stray or lost animals, access control for pets, dog catching, zoonosis control, and others. as evidence of the interest in identifying dogs, some work seeks to identify dogs using the dog’s nose print, an even more challenging problem with the images further reduced to just the dog’s nose. [13] reaches auc of \(88.81\%\) with a model with the dual global descriptor, which can exploit the multi-scaled features of image working in a dataset with 6000 dogs with 20, 000 photographs of their nose prints. a recent study, [23], moved by the problem that in sri lanka, the authors’ country, a lost dog being found is an infrequent occurrence, presents a model of convolutional neural network (cnn) for classification and recognition of images for facial recognition of dogs. the cnn used consists of a customized model vgg-16, obtaining about \(94\%\) accuracy in the dog recognition task. it is noteworthy that facial recognition is one of the least invasive options and is less subject to operational problems when the animal does not cooperate with the use of devices or collars [15]. of course, in the case of dogs, the effectiveness of this approach may vary if the animal is not cared for similarly after it is lost (hair and other characteristics may change, for example). there are algorithms in the literature that deal with this problem, but several of them are arguably overly complex or may present a still limited performance for such a challenging but significant problem. thus, the present work seeks to consolidate results in this area by investigating the use of deep feature embedding vectors for dog face recognition, focusing on the facial identification task. these are the main contributions of this paper: we used resnet-34, a residual cnn for the face identification task in dogs, on two datasets in which each dataset was evaluated in its standard form and applied two preprocessing techniques for each dataset. we evaluate the use of this computation-intensive method, its benefits, and whether they are justifiable, taking its accuracy into account. we compare three dog face preprocessing methods using two public datasets [15, 16] found in similar works. while most papers only provide performance analysis considering accuracy, this paper also includes recall. we obtained state-of-the-art performance using a fast approach that outperforms previous methods, which usually resort to more complex, computationally expensive models. the remainder of this paper is organized as follows. in sect. 2, we presented the related works. the datasets that address the problem of facial identification in dogs are described in sect. 3. in sect. 4, we described the approach presented in this work. in sect. 5, we present the methodology for the experimental evaluation of our approach. in sect. 6, we presented the results. moreover, finally, in sect. 7, we presented final remarks and future works. 2 related work it is worth mentioning that most of the works found in the literature are related to the task of dog breed classification. this is a challenging problem due to the diversity of the existing native breeds and the breeds resulting from crossbreeding and genetic modifications, where many of these are visually similar to each other. [14] presented a method for fine-grained image classification based on using key regions during the feature extraction process to improve the performance of the classification. this method was evaluated on the dog breed classification problem using a dataset of images downloaded from google, image-net, and flickr, which comprises 133 dog breeds and 8, 351 images. the proposed method achieved a recognition rate of \(67\%\) and used conventional tools such as sift and color histograms of the animal’s face region, excluding everything else in the image. [17] used dense-sift to extract features over the input image. these features are split and combined before feeding a cnn classifier. this method resulted in \(94.2\%\) accuracy for 121 dog breeds, using a database obtained by crawl image data methods on the web. similarly, [4] created a (closed) dataset for dog breed and wild wolf classification. these authors compared different cnn architectures using transfer learning and obtained \(92.48\%\) accuracy. [19] proposed another method based on deep learning, using supervised clustering based on a multi-part-convolutional neural network and expectation-maximization. the method was tested on many datasets, including the snapshot serengeti dataset [20], which contains approximately 78, 000 labeled images of animals in africa, obtaining an accuracy of \(98.4\%\), and the authors reinforce the idea that using cnns with supervised training works very well in extracting features from images in such a type of problem. other works relate to using different techniques to track and find lost pets, whereas many approaches resort to installing gps and other kinds of sensors. [21] proposed bokk meow, a mobile application whose goal is to help users locate their animals using gps once the animals might have this type of sensor installed on their collars. the authors argue that their application contributes to lesser stray/lost pets. however, bokk meow demands an active wi-fi connection, and its tracking features do not work appropriately in dense urban building areas. [1] proposed a methodology for noseprint-based dog recognition using a deep, siamese convolutional network called dnnet. this cnn is composed of feature extraction and (expensive) self-attention modules. their method seems promising, presenting an accuracy above \(98.97\%\) and resorts to affordable equipment for image acquisition. however, such an approach sounds hard to automate. moreover, human operators should handle dogs physically, which may cause risks or discomfort to the operator and the dog when there is no collaboration from the animal. the work of [22] presents a smartphone application for dog face detection. the system uses the yolo deep convolutional neural network to simultaneously predict face bounding boxes and class confidences. the doggie smile app helps to take pictures of the dogs as they face the camera. the proposed mobile app can simultaneously assess the gaze directions of three dogs in a scene more than 13 times per second, measured on the iphone xr. the reported average accuracy of the dog face detection system is \(92\%\). 3 dog face recognition datasets and methods there are datasets for many tasks related to fine-grained dog image classification, such as columbia dogs and stanford dogs, that are used in [9, 14], respectively. those datasets were used in previous works for augmenting data and for fine-tuning pre-trained models before the dog recognition task was carried out. for example, [16] also resorted to these datasets to train and evaluate fog face detectors and aligners. 3.1 dog face recognition datasets table 1 summarizes the datasets known in the literature for the face identification task in dogs, their main characteristics, and which ones we used in our experiments. our only motivation for choosing these datasets was that they were public and freely available for download. table 1. datasets found in the literature for the task of facial identification in dogs.full size table the dogfacenet dataset [16], contains 8, 363 images from 1, 393 dog identities. each dog has at least two jpeg images with \(224\times 224\) rgb pixels. despite the previous works of [15, 16] claim that dogfacenet is the first public dataset for dog face recognition and verification. examples of the dogfacenetfootnote 1 are shown at the top of fig. 1, with 3 identities with 5 images each. it is essential to highlight that the images were cropped and aligned beforehand. however, this process introduces black borders in the images. moreover, there are many samples where the dog appears in profile or their face is partially occluded. the flickr-dog dataset [15] is 22.3 times smaller than dogfacenet, containing only 374 images of dogs from two breeds: pugs and huskies. the dataset includes 21 identities for each dog breed, totaling 42 for individual dogs and thus 5 images per identity. this means that flickr-dog provides 33.1 times fewer identities than dogfacenet, so the task of face recognition is also related to the challenge of discerning from multiple dogs of the same breed. each image in the dataset has \(250\times 250\) rgb pixels containing a horizontally aligned face of a dog. also, the images are normalized such that all images are cropped, (loosely) aligned, and resized. five sample images are shown at the bottom of fig. 1 for three individuals: lighting conditions vary significantly, there are artifacts, and also black borders are evident. fig. 1. public dog face recognition datasets: dogfacenet (top) and flickr-dog (bottom). many faces are misaligned and/or appear at challenging poses in both datasets. full size image 3.2 dog face recognition methods in the work of [23], a vgg16-based [18] cnn is trained to classify dogs in 5 categories using transfer learning. then, 128-dimensional embeddings are extracted from this cnn and used for identification by training an unsupervised model for distance calculation. according to the authors, experimental evaluation resulted in more than \(90\%\) accuracy using their dataset where dogs appear in various poses and sizes, but the work is missing evaluation details of how dog faces are processed. moreover, the dataset is private. [15] proposed a pioneer work in dog face recognition. the authors evaluated four classic approaches (eigenfaces, fisherfaces, lbph, and sparse) against two cnns, bark and woof, for retrieving lost dogs based on facial features. these cnns significantly outperformed the other methods, obtaining an accuracy of \(89.4\%\) and \(81.1\%\) for woof and bark, respectively, for the single-breed snoopybook dataset, which is not publicly available. moreover, the accuracy is considerably lower considering the evaluation on the flickr-dog dataset: \(66.9\%\) and \(67.6\%\) for woof and bark, respectively. in [3], in order to identify animals, the authors compared two different cnn architectures (vgg-16 and inception-v3) originally applied in object detection. these two architectures are applied to the flickr-dog and dogsvscats datasets. the accuracy achieved by vgg-16 on flickr-dog was \(95\%\) and \(98\%\) on dogsvscats, while inception-v3 reached \(87\%\) on flickr-dog and \(85\%\) on the dogsvscats dataset. dogsvscats dataset is a dataset with a different purpose, which includes images of cats, so it was not considered among the datasets in this work. [16] proposed a deep learning approach for dog face verification and introduced the dogfacenet dataset. they evaluated two models based on vgg and resnet using \(224\times 224\) rgb images as input. deep feature embedding vectors were generated using triplet loss with soft and hard triplet mining, generated both online and offline. these authors obtained a verification accuracy of \(86\%\), considered poor compared to the typical performance of similar approaches for human face recognition. they stated that errors arose from frequent occlusions in the test dataset, dog’s pose, similarity of different dogs of the same breed, and light exposure. [12] proposed the use of soft biometrics for improving identification. based on a fast r-cnn, their method performs a coarse-to-fine identification, i.e., starting from the breed filtering, then carrying out the identification. their results suggest that breed information improved accuracy for different dog breeds, and overall accuracy is \(76.53\%\) for the flickr-dog using an xception back-end [5]. the authors improved this result to \(84.94\%\) by using a likelihood-adjusted decision network. [2] proposed and evaluated a deep learning-based approach to pet detection and recognition, named pet2netid, based on a dataset provided by pet2network, a social network for pets and their owners. they achieved \(94.59\%\) accuracy by combining transfer learning and object detection approaches with inception v3 and ssd inception v2 on the provided pet2net dataset, and \(77.19\%\) was achieved when training with the pet2net dataset on the flickr-dog dataset. the approach proposes to work with wild images without any pre-processing, including other objects besides the pets themselves, and the identification uses information from the whole animal, not only the face. in general, as seen in table 2, these works use deep learning methods without preprocessing experimentation (e.g., face detection) and do not provide recall results, only accuracy. we observed this and included it in our experiments, described in the following sections, developing a unique approach to the problem with a more robust evaluation than the related works in this section. table 2. comparison between related works.full size table 4 proposed dog face recognition method since many works in the literature already resorted to transferring learning and fine-tuning existing models, our investigation prioritized following a different strategy, so we trained models from scratch. as stated before, the incoming images are subjected to optional preprocessing, which can occur in two ways. however, only cropping is allowed in this step, so no further filtering or image enhancement is applied in favor of an honest comparison against other methods. 4.1 preprocessing and training pipeline the pipeline of evaluation using the methods is illustrated in fig. 2. we took three different preprocessing paths and evaluated them separately, these being: (a) the dataset in its pure form; (b) a version where each image was cropped in the center, which we call central crop (cc); and (c) another path using facial detection (fd). the use of each technique is detailed below. a centralized crop is applied to assume the images are satisfactorily aligned, i.e., the dog’s eyes lie near the center of the image, but assuming there is a border whose existence is questionable for facial recognition purposes. the cc preprocessing algorithm extracts the central pixels from a given dimension, thus reducing the input image size. fd includes alignment and performs as follows. first, the dog face is detected using a cnn-based object detector using dlib’s [10] max-margin object detection (mmod) loss layer that implements the object detection scheme as described in [11]. despite using a concise representation, i.e., this cnn’s weights require only about 700 kb. however, mmod inference is the most time-consuming operation in this pipeline. second, six facial landmarks are then estimated using an ensemble of regression trees [8] with cascade depth \(=20\) and tree depth \(=5\). finally, these points (shown in fig. 3) are converted into another layout before they are used to extract aligned dog faces with padding \(=0.2\) since mapping this 5-point into an aligned box is less prone to errors. such conversion is based on the proportions of the eye and nose landmarks. fig. 2. in order to study the impact of using different preprocessing techniques, we performed tests without preprocessing (a), with alignment and face detection preprocessing (b), and with central crop (c) before using residual cnn for feature generation. full size image fig. 3. dog face landmark detection returns 6 points (left), which are converted to the 5-point layout (right) before alignment and cropping. full size image 4.2 preprocessing the datasets once we have clarified the two preprocessing techniques, in this subsection, we detail the generation of datasets’ versions with each preprocessing and the details that occurred in each technique and dataset. dogfacenet cc results from a central crop of \(180\times 180\) pixels of each image in dogfacenet, so the total individuals and images are kept. dogfacenet fd corresponds to the dogfacenet dataset after face and alignment. we then used the same \(80-20\) split used in the original dogfacenet and dogfacenet cc, but some images are missing due to face detection errors. this resulted in 1, 108 identities for training, corresponding to 5, 978 images. there are 276 identities for testing, corresponding to 1, 694 images. this process lost 9 identities corresponding to \(0.65\%\) of individuals (ids 44, 67, 287, 449, 494, 948, 1131, 1274, and 1310). on the other hand, the 691 images lost correspond to \(8.26\%\) of the images from the original dataset. it is worth highlighting that missing images cannot be included in the training-test split. flickr-dog cc corresponds to the flickr-dog dataset after a central crop for the \(200\times 200\) inner pixels. in turn, flickr-dog fd corresponds to flickr-dog after face detection and alignment. this process resulted in 41 of the 42 identities, totaling 218 images. this process lost one identity corresponding to \(2.38\%\) of individuals since the face of “wilco” could not be detected in any of its images (first row of fig. 1). on the other hand, 156 images lost correspond to \(41.71\%\) of the images from the original dataset. such a fact is quite a loss in terms of images. however, it is essential to remember that all the 218 remaining images will be used for model evaluation. in fig. 4, there are some examples of removed images where the algorithm could not detect the face and examples of dogs that lost all images and consequently lost their identity for both datasets. the final numbers for the datasets are described in table 3. fig. 4. examples of images removed from datasets because the algorithm could not detect the dog’s face full size image table 3. number of identities and images of each of the dog face datasets.full size table 5 experimental evaluation in order to evaluate the use of the residual network in the dataset and each of the generated versions, we performed the identification experiments evaluating different metrics of interest in this context. we carried out the experiments on a computer running linux operating system (ubuntu 20.04.2 lts), equipped with an intel(r) core(tm) i7-9700k cpu @ 3.60 ghz, 32 gb ram, a 1 tb sdd, and an nvidia geforce rtx 2080ti graphics card. the resnet-34 convolutional neural network was used for the experiments, as it is a good trade-off between performance and computational cost. it is a residual neural network with about 21, 282, 000 parameters, requiring about 4 gflops for activation. figure 5 shows the resnet-34 architecture. the implementation of resnet-34 present in the machine learning dlib c++ library was used [10]. the split training and testing procedures were explained individually in specific topics for each dataset. we defined that all training sessions would have 80, 000 steps, to standardize all procedures for each dataset. we chose this number because it is the average value where the loss curve is already relatively stabilized. fig. 5.(adapted from [6]). resnet34 architecture full size image 5.1 performance metrics the metrics used in this work are accuracy and recall. both are defined in terms of true positives (tp), false positives (fp), true negative (tn) and false negative (fn) samples. accuracy is defined in eq. 1 and recall in eq. 2. $$\begin{aligned} accuracy = \frac{tp+tn}{tp+tn+fp+fn} \end{aligned}$$ (1) $$\begin{aligned} recall = \frac{tp}{tp+fn} \end{aligned}$$ (2) precision and f1-score sound virtually meaningless for our experiments since the evaluation considers an upper triangular matrix where most elements in that matrix represent different individuals. in other words, we would be roughly counting the main diagonal versus the rest of the matrix in the limit, so these performance metrics are influenced mainly by the model’s ability to discard dissimilar individuals. 5.2 preparation of the datasets notice that the baseline work [16] adopts a \(90-10\) split, i.e., only 697 images (8.33% of the dataset) are used for testing. to better generalize the model used here, we adopted the \(80-20\) split, making the problem even more challenging by having fewer images for training and providing a more extensive test set. for the dogfacenet case, 5, 978 images for training in each case (cc and fd), excluding all the fd algorithm images, did not detect a face, and 1, 794 test images for original dogfacnenet, 1, 704 images in cc, and 1, 520 in dogfacenet fd. the experiment performed with flickr-dog proceeded as follows. since this is a small-scale dataset, the method heavily depends on some data augmentation strategy or external data. the results are inferior without training with external data and only with data augmentation: an accuracy limited to 76.55% and recall limited to 65.68%, also performed using the \(80-20\) split. we decided to use the model trained with the different versions of the dogfacenet dataset (original, cc, and fd) and test on a range corresponding to 20% of the flickr-dog dataset, which was also submitted and generated versions with cc and fd preprocessing. for this case, the original with 65 images, flickr-dog cc with 69, and fd with 45 images. 6 results preliminarily, we demonstrated the loss for each of the training variations of the dogfacenet dataset in fig. 6 and noticed that the curves are very similar, reaching very close values at the end of the training. the curve of the fd training stabilized faster, showing that it would not require pervasive training for this case, unlike the other two pieces of training that stabilized closer to the end of the training. still regarding the training, in fig. 7, we showed through the embeddings extracted from each dataset that resnet-34 produced an effective separation between classes, where classes are the “same dog” and “different dogs”, with euclidean distance and relative frequency as dimensions. it is also possible to see in these distinction curves that the overlap area of the cc is visually smaller than the other two pieces of training. fig. 6. loss curves for dogfacenet. full size image fig. 7. distance distribution curves for dogfacenet. full size image the results are presented in table 4, containing each dataset variation, accuracy, and recall of the method. the best result in the dogfacenet dataset is 98.43% of accuracy with dogfacenet with facial detection for training and testing. in the central cropped dataset version, the method achieves 85.64% of recall without preprocessing in training. furthermore, the best result in the flickr-dog is 91.53% of accuracy with facial detection in training and testing. overall, the facial alignment fd allowed for achieving the best accuracy for both datasets used in this work, which probably makes the network more objective, not paying attention to details that are not part of the dog’s face but are in the image. table 4. results of accuracy and recall using resnet-34 modelfull size table table 5. comparison with other methodsfull size table to compare more fairly, we do not use our results obtained in tests on the fd version with the other works in the literature. we consider that the modification made and the loss of images generate an impact, making the data set different. using the cc technique with no image loss, we demonstrated better results than all the other works by using fd for training and cc for testing, as shown in table 5. 7 conclusion this work investigated the problem of dog face recognition. we compared three strategies to preprocess the input images: no preprocessing, central crop, and face detection with alignment. we explored the performance of a robust baseline model, resnet-34, showing that it can be competitive or even outperform the results found in the literature. moreover, this paper also includes recall as an essential performance metric regarding the verification that is missing in most related works, especially considering the method’s ability not to repudiate the dog. experimental results show that face alignment does not play an important role for improving recall in dog face recognition, where the results on datasets with preprocessing were slightly better than the others. finally, it is worth emphasizing that the alignment process pays off on flickr-dog since (1) this strategy improved the state-of-the-art considering both accuracy and recall, and (2) our experiments evaluated the entire flickr-dog since the model was trained with external data. the latter observation is consistent with the hypothesis found in previous works [23] that augmentation of dog breeds yields better accuracy when the literature focus on classification and lacks a proper analysis for identification. future work includes extending the existing dog identification datasets in terms of both new identities and additional annotations, adding support for cat face recognition, investigating effective approaches for animal identification and re-identification, and the assessment of other techniques for the problem of animal biometric recognition. notes 1.https://github.com/guillaumemougeot/dogfacenet. references bae, h.b., pak, d., lee, s.: dog nose-print identification using deep neural networks. ieee access 9, 49141–49153 (2021) article  google scholar  batic, d., culibrk, d.: identifying individual dogs in social media images. arxiv preprint arxiv:2003.06705 (2020) capone, v., figueiredo, c., valle, e., andaló, f.: crowdpet: deep learning applied to the detection of dogs in the wild. multimedia tools appl. 76(14), 15325–15340 (2017) google scholar  chaturvedi, k.: wolf and dog breed image classification using deep learning techniques. ph.d. thesis, dublin, national college of ireland (2020) google scholar  chollet, f.: xception: deep learning with depthwise separable convolutions. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1251–1258 (2017) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 770–778 (2016) google scholar  karasu, s., alkar, ö., et al.: a qualitative investigation on the mourning period that occurs after the loss of pets. veteriner hekimler derneği dergisi/j. turk. vet. med. soc. 91(2), 86–97 (2020) article  google scholar  kazemi, v., sullivan, j.: one millisecond face alignment with an ensemble of regression trees. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 1867–1874 (2014) google scholar  khosla, a., jayadevaprakash, n., yao, b., li, f.f.: novel dataset for fine-grained image categorization: stanford dogs. in: proceedings of cvpr workshop on fine-grained visual categorization (fgvc), vol. 2. citeseer (2011) google scholar  king, d.e.: dlib-ml: a machine learning toolkit. j. mach. learn. res. 10, 1755–1758 (2009) google scholar  king, d.e.: max-margin object detection. corr abs/1502.00046 (2015). http://arxiv.org/abs/1502.00046 lai, k., tu, x., yanushkevich, s.: dog identification using soft biometrics and neural networks. in: 2019 international joint conference on neural networks (ijcnn), pp. 1–8. ieee (2019) google scholar  li, b., wang, z., wu, n., shi, s., ma, q.: dog nose print matching with dual global descriptor based on contrastive learning. arxiv preprint arxiv:2206.00580 (2022) liu, j., kanazawa, a., jacobs, d., belhumeur, p.: dog breed classification using part localization. in: fitzgibbon, a., lazebnik, s., perona, p., sato, y., schmid, c. (eds.) eccv 2012. lncs, vol. 7572, pp. 172–185. springer, heidelberg (2012). https://doi.org/10.1007/978-3-642-33718-5_13 chapter  google scholar  moreira, t.p., perez, m.l., de oliveira werneck, r., valle, e.: where is my puppy? retrieving lost dogs by facial features. multimedia tools appl. 76(14), 15325–15340 (2017) article  google scholar  mougeot, g., li, d., jia, s.: a deep learning approach for dog face verification and recognition. in: nayak, a.c., sharma, a. (eds.) pricai 2019. lncs (lnai), vol. 11672, pp. 418–430. springer, cham (2019). https://doi.org/10.1007/978-3-030-29894-4_34 chapter  google scholar  ouyang, j., he, h., he, y., tang, h.: dog recognition in public places based on convolutional neural network. int. j. distrib. sens. netw. 15(5), 1550147719829675 (2019) article  google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv preprint arxiv:1409.1556 (2014) sundaram, d.m., loganathan, a.: a new supervised clustering framework using multi discriminative parts and expectation-maximization approach for a fine-grained animal breed classification (sc-mpem). neural process. lett. 52(1), 727–766 (2020) article  google scholar  swanson, a., kosmala, m., lintott, c., simpson, r., smith, a., packer, c.: data from: snapshot serengeti, high-frequency annotated camera trap images of 40 mammalian species in an african savanna (2015). https://doi.org/10.5061/dryad.5pt92 tangsripairoj, s., kittirattanaviwat, p., koophiran, k., raksaithong, l.: bokk meow: a mobile application for finding and tracking pets. in: 2018 15th international joint conference on computer science and software engineering (jcsse), pp. 1–6. ieee (2018) google scholar  turečková, a., holík, t., komínková oplatková, z.: dog face detection using yolo network. in: mendel. brno university of technology (2020) google scholar  weerasekara, d., gamage, m., kulasooriya, k.: combined approach of supervised and unsupervised learning for dog face recognition. in: 2021 6th international conference for convergence in technology (i2ct), pp. 1–5. ieee (2021) google scholar  yoon, b., so, h., rhee, j.: a methodology for utilizing vector space to improve the performance of a dog face identification model. appl. sci. 11(5), 2074 (2021). https://doi.org/10.3390/app11052074 article  google scholar  download references acknowledgements the authors would like to thank the ceará state foundation for the support of scientific and technological development (funcap) for the financial support (6945087/2019). author information authors and affiliations mdcc, universidade federal do ceará, campus do pici bloco 910, fortaleza, ce, brazil joão p. b. andrade, leonardo f. costa, lucas s. fernandes & paulo a. l. rego instituto ufc virtual universidade federal do ceará, campus do pici bloco 901, fortaleza, ce, brazil josé g. r. maia authors joão p. b. andradeview author publications search author on:pubmed google scholar leonardo f. costaview author publications search author on:pubmed google scholar lucas s. fernandesview author publications search author on:pubmed google scholar paulo a. l. regoview author publications search author on:pubmed google scholar josé g. r. maiaview author publications search author on:pubmed google scholar corresponding author correspondence to joão p. b. andrade . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper andrade, j.p.b., costa, l.f., fernandes, l.s., rego, p.a.l., maia, j.g.r. (2023). dog face recognition using deep features embeddings. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_9 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_9 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords residual neural networks deep features facial identification dogs publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature analyzing college student dropout risk prediction in real data using walk-forward validation | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper analyzing college student dropout risk prediction in real data using walk-forward validation conference paper first online: 12 october 2023 pp 291–305 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) analyzing college student dropout risk prediction in real data using walk-forward validation download book pdf download book epub rodolfo sanches santos9, moacir antonelli ponti9 & kamila rios rodrigues9  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 584 accesses 1 citation abstract college dropout is a concern for educational institutions since it directly impacts educational management and academic results, as well as being directly related to social problems. therefore, there is significant incentive for studies that use data to support decisions by predicting risk of dropout so that institutions can attempt to prevent such cases. although machine learning techniques were shown to have potential for this task, there are many steps involved when it comes to the use of real data, which comes from scattered systems and present issues such as need for data cleaning and preparation, high dimensionality of the data requiring adequate feature selection, as well as class imbalance. in this paper, we used data from 32.892 students enrolled between 2008 and 2020 from all courses offered by a public high-education institution. a protocol for data preparation is proposed and found to be more important than designing complex classifiers. we present guidelines when modelling a college dropout classification task using a public university data and experiments using walk-forward validation that showed the predictive capacity for the first years. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others predicting student drop-out rates using data mining techniques: a case study chapter © 2018 predicting student retention in higher education using machine learning chapter © 2024 predicting student performance from online engagement activities using novel statistical features article 17 january 2022 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. data science data analytics data mining educational research machine learning statistical learning 1 introduction students of higher education are exposed to several positive and negative events in college. the successes often outweigh the efforts from the enrolment until the student effectively earns a degree, however adversities may lead to dropout [36, 39]. this phenomenon also related to college attrition, is common to both public and private higher education, directly interfering in their management and in the results of education quality [29, 35], which consequently generates a necessary concern, as the students’ departure from the study cycle induces several consequences [18, 31]. the costs associated with college attrition include hindering of future job prospects with impacts in the countries’ economy, the personal and professional costs for the former student, waste of institutional and federal resources, potential damage to university reputation and demoralization of students [5, 13]. several factors may influence dropout [3], the main ones reported in the literature related to financial and family reasons, unfulfilled expectations and lack of motivation [5]. xenos, pierrakeas and pintelas [40] stated that the identification of such specific factors is essential to provide special assistance to students, and categorizes them as related to: (i) internal or students’ perception; (ii) course and professors and; (iii) student demographic characteristics. additionally, many studies point out a high prevalence of mental health problems in college students, which are even most common in those exposed to relational stressors and low social support [9, 22]. in a survey conducted by eisenberg et al. [9] in an university in the us, the prevalence of depressive or anxiety disorders was 15.6% for undergraduate students and 13.0% for graduate students, as well as presence of suicidal ideation reported by 2% of students. according to leonhardt and sahil [18], in north american higher education, in 2018, about one in three students who enrol in college never earn a degree. they looked at data from 368 colleges arranged by what they would expect their graduation rates to be, based on the average for colleges with similar student bodies. they found higher success rates in those that remove hurdles for students. also, such success was correlated with students having better connection to the college community, having less financial issues, and when the university offered good infrastructure and support. in brazil, a special committee on dropout studies was established in 1995, from ministry of education ordinance, with the purpose of evaluating the performance of federal higher education institutions (fhei; ifes, in portuguese). in 2002, this became a major concern with the significant increase in the number of places offered by ifes in brazil [39]. in this context, data mining and machine learning techniques have been explored to investigate this problem. previous studies often to use data to identify student’s risk of dropout [1, 7, 15, 16, 21, 25, 32, 34]. some of those focus on specific patterns such as student’s personal profile [8, 23], others on behavior, e.g. procrastination [11] and others in social issues [6]. in this paper, we use data gathered from different information systems at the federal university of são carlos (ufscar), in the interior of the state of são paulo brazil, to predict the risk of dropout in groups of students. the objective is to integrate it to the university solutions for decision support. when using real world data is not trivial. thus, we present guidelines to evaluate and deploy predictive models and the necessary preparation steps, including data cleaning. in addition to evaluate the model using walk-forward validation evaluation, we propose four experiments using different subset of data and tests on data not seen during the training of the learning model. our results represent a milestone in terms of making it possible, in practice, to identify students in higher risk and direct them to services that address issues related to emotional distress, financial problems, among others. the paper is structured as follows: sect. 2 describes related works; sect. 3 describes the main steps of data exploration, as well as what information was collected and processed, the data preparation and cleaning, experiments applied, and the results obtained and, finally, sect. 4 points out the final remarks. 2 related work previous studies found that college attrition depend on the type of course [8, 40], student’s year at college [8, 18, 19, 36], social issues such as parental background [2, 8], class-cultural discontinuities [17] and economic profile [28]; as well as quantitative academic data [25]. in particular, lozano et al. [19] found the first and second years to be crucial when it comes to retention, as it allows for more numerous and more intensive interventions. according to casanova et al. [8], the first years are particularly difficult for students who enrolled in the course as a second option. the authors suggest improving motivation in students who have to adapt to this situation. a study carried out in catalan universities used academic data, face-to-face interviews and telephone interviews, concluding that main factors for dropout risk were: lack of motivation; work-related reasons; unfulfilled expectations; time-table mismatch; family reasons; financial reasons and other opportunities [38]. however, the need for individual interviews at the dropout risk assessment stage can hamper scalability. interventions may represent a positive approach to improve retention [30]. this was shown at a brazilian public university, in which dropout decreased 12.3% in 2018 when compared to 2017 [4]. this was due to institutional interventions that increased the number of registered students, accounting for new admissions and the re-inclusion of students who had taken a time off from college. data mining and data science techniques are also relevant in the educational context [12]. in da silva et al. [33], the authors used an ensemble of regressors to predict dropout rate percentage of higher-education institutions. pal [26], in turn, used a bayesian machine learning (bml) classification method with high school data, income, family position, type of admission, gender, and other school data. their strategy was to compare the student profile with the classification provided by the model [26]. manhaes et al. [20] studied academic data of a brazilian university and found dropout patterns related to the first year, indicating failure in early courses are present in all dropouts. a study carried out by tontini and walter [37] analyzed the effectiveness of an institutional intervention, which consisted of the institution’s contact with the student in order to influence the student to give up dropping out. they observed a decrease of 18% in the semester following the interventions. they also employed descriptive quantitative research with an online questionnaire with the aim of exploring the perception regarding the retention attributes. to predict dropouts using the gathered data, the authors used radial basis function (rbf) neural networks and cluster analysis to group new students by similarity. although the author had positive results, the techniques and steps for replicating the study were not detailed. hippel and hofflinger [10] conducted a study to identify students at risk of dropping out at 8 chilean public universities using logistic regression (lr). the authors used both personal/family data, e.g. parents’ education, high school grades, and entrance exam scores, as well as academic data collected during college. they studied the effect of programs focused on helping students adapt to university life, develop study skills and manage anxiety. the assisting programs dictated a reduction of 30–40% in the chances of dropping out in 2 of 4 universities where such programs existed. they also found the dropout risk to be inferior for older students, and for those receiving scholarships. this makes it evident there is no clear consensus on the causes of dropout, and also the variety of factors protecting against it. in this paper, we use data as available at most brazilian higher education institutions, including personal and academic data. we show the main steps to be considered when designing predictive models to provide educational managers a decision-making support, as well as important insight for similar initiatives in other institutions. in particular, we show there is a significant gap between analyzing historical data and future data, and specific features play a major role in this type of problem. as far as we know, this is the first paper to address a complete pipeline for a data-driven dropout risk assessment tool, including data preparation, feature selection and proper evaluation protocol, as well as feature importance analysis. therefore, we intend this paper to shed light on the application, and serve as guideline for other higher-education institutions. 3 data-driven dropout risk predictive model this section describes the exploration of data and the assessment of a predictive model which includes data collection from information systems, pre-processing, experiments using different subsets of the data for training and testing models using walk-forward validation, as summarized in fig. 1. fig. 1. overall flow for the design of our dropout prediction model. first, the maximum values of training data variables are computed for scaling testing data. then the data undergoes preprocessing, always fitting on training and applying on testing. finally, the training data is used for training the model, while the preprocessed testing data is used for evaluation. full size image formally, let s be a dataset where an instance \(\textbf{s} \in s \) represents a student enrolled in a course, in a given semester, by attributes or variables with \( \gamma \) dimensions, and \( y = \left\{ 0, 1 \right\} \) represents the label describing the student’s s situation: studying/regular (0 or negative) or avoidant (1 or positive). first, we sought to obtain a subset of the original data to pre-process, transform and obtain x that has \(\gamma \) original characteristics, and \(x' \in r^{m}\) with attributes such that \(m \le \gamma \). from the preprocessed dataset, \(x'\) supervised learning methods are used to infer \(f: x' \rightarrow y\). then, s was evaluated in different subsets grouped by the current semester of the student, with the test data being used only to test the generalization of the models for a group of students of a certain semester not seen during the training. 3.1 university data exploration data description. the ufscar has several computational information systems for managing institutional processes. those are separated by purpose such as: undergraduate studies; postgraduate studies; university restaurant; library, among others. the main systems used to manage academic and student data from which we gathered data were: integrated academic management system: responsible for the academic processes related to the student, lecturers and the main activities, such as course enrolments, registration of grades and frequencies, teaching plans, subjects, courses, among other activities; integrated university management support system: general management of the university including registration of people, permissions, and applications student cards; dean of extension: management of outreach and extension courses, as well as postgraduate courses lato sensu e stricto sensu. in order to collect and analyze the students’ academic, economic and personal data, in addition to the approval of the ufscar ethics committee (caae number: 34343920.5.0000.5504), a formal request was made to the department responsible for the database, which was stored in a single database management system. to maintain data secure, the stored files were encrypted and the access to the computer and the database was granted only for authorized users. data gathering. the ufscar information technology team defined business rules related to undergraduate students in order to define the data to be gathered and integrated into a single dataset s. in this way, the source database schema was simplified by querying the target data into just two schemas, both of them totalling 222 tables and 15 views. we follow the empirical evidence found by pal [26], hippel and hofflinger [10]; and de souza [35] for filtering relevant data to be used. those authors indicate that personal data, academic information prior to college enrolment, academic information collected during college, as well as economic information, are sufficient to investigate college attrition and student dropout. in this paper we included: personal information: city, genre, age, color/race declared and marital status; academic information: undergraduation course, grades and frequency for each course taken, current semester, grade obtained in the national college admission test (acronym: enem, in portuguese). economic and social information: category of admission (which is defined by lower income levels, if previous education was on public schools, ethnic/racial information, special needs); and also whether self-claimed to below income during registration at the college admission test. note that, brazilian federal universities have affirmative actions, so that the category of admission define quotas of student places related to social/economic status of the student as well as race. our analysis obtained data from students who entered the ufscar between 2008 and 2020, from all undergraduate courses offered by the institution during this period. in each year, between 2000 and 3000 students are admitted. in total, information was collected from 32.892 students, separated by course, year and period/semester. guidelines: include personal, academic and economic/social data available in the information systems, taking into account opinions from the information technology team, as well as the team of the dean of undergraduate students. in principle, sensible personal data should not be included, and be encoded if needed. also, the resulting data should be encrypted and not given broad access. 3.2 data preparation as we show later in results, such step significantly improves the quality of dropout prediction. below, we describe the data preparation/cleaning and standardization in detail. some important characteristics of the source data to be observed when designing a machine learning model are: 1. students in the first period still have no grades and attendance therefore cannot be used to train the model; 2. there are missing values in particular for personal data, in our case for the attributes: marital status, lower income and declared color/race; 3. data for active students includes status from different academic periods (semester) of all courses, while data for those that concluded the degrees are complete, and those that abandoned are partial; 4. class imbalance: around 70% of students are regularly enrolled or earned a degree, and around 30% are considered dropouts – value referring to all data collected (all years and all academic periods, including students who are out of regular period); 5. academic periods: most courses have a regular curriculum of 4 years and 8 academic periods, except for a few courses that have a curriculum of 5 years. data preparation and cleaning. below we list the cleaning procedures carried out only when those showed to improve results in validation sets within a 10-fold cross-validation experiment using the training data only: instances removed: students without grade or attendance information recorded; attributes removed: start year, registration number; city of birth of the student; name of the country; start period; has lower income were removed because they are not related to the classification task, possible insertion of bias, because they have missing values and/or have only one value for all students; obtaining the binary label: the original database contains different status for the student which will represent the positive label. following the university’s guidelines, all status indicating the student left college were defined as evader (positive), i.e.: change of area, cancelled, loss of vacancy, retired and external transfer. on the other hand: studying, undergraduate candidate, internal transfer, in appeal, graduated and graduating were considered studying (negative), in which the student still maintains a relationship with the university. in addition to cleaning, we filled the null values for the features: civil status, declared color/race, type of school in high school, gender, type of admission were filled using the value not informed, while the features course, course regime, course city, marital status, declared color/race, type of high school, nationality type, course shift, degree, admission type, admission form and sexuality attributes were one-hot-encoded. the decision for one-hot-encoding transforming was also made based on cross-validation results. the min-max normalization of the numerical data was applied in order to maintain the range from 0 to 1 of all values and, consequently, to avoid that larger scale variables were considered of greater importance during the training of the algorithms. the following transformations were performed: a) expected values between 0 and 10 were divided by 10; b) expected values between 0 and 100 were divided by 100; c) values of profile; total number of courses enrolled, cancelled, approved and disapproved; income coefficient; workload completed and classification obtained in enem were divided by the maximum value found in the database. in this case, the maximum values obtained in the training of the classifiers were stored in memory and used for data normalization in the inference (test) step. the target variable status was designed as binary. however, the original dataset has 17 distinct status, so we grouped those into class “dropout” (positive) for the statuses area change, canceled, loss of vacancy, temporary academic leave and external transfer). the class “studying” (negative) included studying, under graduation candidate, internal transfer, in appeal and forming. guidelines: remove instances that do not provide representative information, and study cases of missing values, filling those if those account for a significant number of instances. convert attributes using one-hot-encoding approaches depending on their nature (categorical). normalize the numeric data to eliminate larger scales from the values. 4 experiments and results three classification algorithms were used in the experiments: support vector classification, decision tree (cart) and lightgbm. we do not aim at comparing which is the best classifier, but instead use their results to understand the problem in terms of the available data and the features. the svm offers a proxy measure for the linear separability of the classes, therefore its performance allows us to draw conclusions about the difficulty of the problem in terms of the given variables [24]. on the other hand, the decision tree allows understanding if it is possible to infer a single set of rules based on the features that allow to estimate dropout for all students. finally, lightgbm [14] would be the attempt to combine several decision trees and, the more its results surpass the other methods, it means different students require different decision trees, needing different ways of shattering the feature space [27]. the following hyperparameters were used: support vector classification: linear kernel, cost 1 and maximum of 100 iterations; decision tree: gini criterion, best divisor and 2 minimum samples to split; lightgbm: binary target and 100 iterations. the experiments consisted of training the models and applying the tests using the techniques of wfv (walk-forward validation), in which the methodology is applied to data that have time intervals with the possibility of making good forecasts in each period. we sought to use this methodology to simulate the real scenario of application of an am algorithm at the university, in which it will learn temporally according to the years and also, validate the behavior of the algorithms with the training of different time windows. table 1 shows the number of instances per year and class (1st to 5th year) of the dataset used in the experiments, and fig. 2 illustrates all the experiments performed. table 1. number of instances per year and class – positive (evading) and negative (studying).full size table fig. 2. experiments carried out using the wfv methodology. full size image the first experiment aimed to evaluate whether the use of data from students from all previous years allows for the prediction of dropout, simulating learning from year to year successively according to the results in table 2. the second experiment verified whether the use of data from students from the immediately previous year is sufficient to classify those from the current year, without the accumulation of all previous years, with results in table 3. the third experiment followed the methodology of the second experiment, but with the use of instances with up to 2 years prior to the test, as shown in table 3. finally, we sought to assess whether the first years would provide enough signal to allow predicting dropout in instances beyond the 2nd and 3rd year, see table 5 (table 4). table 2. classification metrics results from experiment 1, training with all pre-test years in wfv for all test sets (sn) and classifiers (cl).full size table table 3. classification metrics results from experiment 2, training with the year immediately preceding the test in wfv for all test sets (sn) and classifiers (cl).full size table table 4. classification metrics results from experiment 3, training with (up to) two years immediately prior to testing in wfv for all test sets (sn) and classifiers (cl).full size table table 5. classification metrics results from experiment 4, training with data from years 1, 2, 1+2 for test sets (sn) 4 and 5, and all classifiers (cl).full size table because the second test of the first experiment had the best result in general, a list of the most relevant features was obtained according to the lightgbm classifier. thus, it was possible to observe that the relevant features are, for the most part, academic data and have a direct relationship with the student’s performance in the subjects and in the course. table 6 displays the 10 features. table 6. 10 most relevant features, in order from most to least important, according to the lightgbm classifier modelled during s2 of experiment 1. we show the absolute number of times the feature was used by the model in training, its relative percentage against all features and the cumulative percentage.full size table 4.1 discussion data engineering and processing: as in every real-world problem, and unlike databases benchmark prepared and cleaned, knowledge of the data available in the databases is necessary to extract and prepare them. the variables have different ranges, many are categorical, data have missing or inconsistencies values, and is unbalanced. thus, it is necessary to make decisions to treat the data in order to allow the subsequent modeling; training regime: if it is necessary to obtain a model capable of predicting whether a student is a potential dropout, the training regime must consider the temporal aspect, always using past data to predict a current instance. experiments show that accumulating all past history is beneficial for up to the third year, but for the fourth year, using only the previous year is better; predictive ability: the predictive ability is higher for students in the first and second years. from the third stage, the capacity is degraded, making it more difficult to differentiate evaders from non-evaders. for the best f1-score result in the third and fourth years, a lower recall than precision was observed, indicating a greater presence of false negatives. thus, a specific study may be necessary to define the training set for each year; classification method: the decision tree ensemble lightgbm showed the best results. the results with svc indicate the difficulty of finding a linear separation with the available features, in particular for the third and fourth years, confirming the conclusions about the predictive capacity. a single decision tree is also insufficient to solve the problem, so that there is no single set of rules that is suitable for all students, needing ensembles as to better tackle the problem; features: in terms of importance of features, considering the result for s2, the enem grade and classification is one of the most important, indicating the need for the university to have specific programs for students with grades lower and, possibly, have entered advanced calls and with the initial semester already started. the importance of the dispersion of grades and attendance (measured by the standard deviation) and the mean of attendance, indicates that the student with very different results for some subjects may be less motivated to continue. next, it was observed the coefficient of performance, higher grade and the features that relate to how much of the course is already completed (subjects, workload and percentage), which may indicate that the greater the student’s success and the shorter the time to complete, the less likely you are to drop out. age in years also appears to be important, showing that younger people are less likely to drop out. finally, it is worth noting that the difficulty in a single (or a few) subjects can induce dropout as indicated by the lower grade, lower frequency and the features related to the cancellation of subjects. about the importance of the variables and factors for dropout, hippel and hofflinger [10] mentioned that enrolment variables (high school grades, entrance exam grades) are not sufficient for the prediction. they also suggested the use of variables like family income, high school, first, second or third option course and financial assistance to influence the prediction. in this study, we confirm that actual academic data, directly related to the student experience during undergraduation is needed, including grades, attendance, admission modality and workload. although we did not obtain some of the attributes related to financial aid and course option (as in [10]), as these data do not exist in the ufscar systems. 5 conclusions this paper offered guidelines for the design of a college dropout risk prediction model. we used real data and proposed a protocol for data preparation and a study of classification models by modelling the task as a binary classification problem. it was possible to show the difficulties of processing the data and its challenges. by using the real data available at the college database, the best results for second and third year students (which is still early allowing for interventions), had an accuracy of 0.83 and f1-score of 0.80 (s2) and 0.83 (s3) in a realistic scenario. this represents a milestone in terms of making it viable to use as a tool in practice. we can also interpret the results of classifiers in such context: the svc method is comparable to the decision tree in particular for early years, which indicates there is some degree of linear separability but not sufficient to solve the problem. the lightgbm was in general better than using a single decision tree, showing that some instances might have conflicting patterns and that a single set of rules based on the available features is not sufficient to classify the students according to their risk of dropout. because the systems and databases studied in this paper are often available in institutions, our proposal is scalable for other educational institution that wishes to improve its college attrition management. it allows providing statistical reports for university management, as well as defining dropout risk thresholds so that to apply interventions, anticipating and avoiding dropouts. future work may investigate the role of fees from academic centers and evasion by type of admission. also, we believe there is space for improvement by segmenting the data to study it in a more fine-grained level, such as into different course areas and specific degrees. references abu-oda, g.s., el-halees, a.m.: data mining in higher education: university student dropout case study. int. j. data min. knowl. manage. process 5(1), 15 (2015) article  google scholar  aina, c.: parental background and university dropout in italy. high. educ. 65(4), 437–456 (2013) article  google scholar  aina, c., baici, e., casalone, g., pastore, f.: the determinants of university dropout: a review of the socio-economic literature. socioecon. plann. sci. 79, 101102 (2022) article  google scholar  advisory at the federal university of alagoas, c.: ufal comemora a redução do índice de evasão de estudantes de graduação. technical report (2019). https://ufal.br/ufal/noticias/2019/10/ufal-comemora-a-reducao-do-indice-de-evasao-de-estudantes-de-graduacao ataíde, j., lima, l., de oliveira alves, e.: a repetência e o abandono escolar no curso de licenciatura em física: um estudo de caso. physicae 6, 21–32 (2006) google scholar  bayer, j., bydzovská, h., géryk, j., obsivac, t., popelinsky, l.: predicting drop-out from social behaviour of students. int. educ. data min. soc. (2012) google scholar  burgos, c., campanario, m.l., de la peña, d., lara, j.a., lizcano, d., martínez, m.a.: data mining for modeling students’ performance: a tutoring action plan to prevent academic dropout. comput. electr. eng. 66, 541–556 (2018) article  google scholar  casanova, j.r., cervero fernández-castañón, a., núñez pérez, j.c., almeida, l.s., bernardo gutiérrez, a.b., et al.: factors that determine the persistence and dropout of university students. psicothema 30 (2018) google scholar  eisenberg, d., gollust, s., golberstein, e., hefner, j.: prevalence and correlates of depression, anxiety, and suicidality among university students. am. j. orthopsychiatry 77, 534–542 (2007) article  google scholar  hippel, p.t.v., hofflinger, a.: the data revolution comes to higher education: identifying students at risk of dropout in chile. j. high. educ. policy manage. 1–22 (2020) google scholar  hooshyar, d., pedaste, m., yang, y.: mining educational data to predict students’ performance through procrastination behavior. entropy 22(1), 12 (2020) article  google scholar  huo, h., et al.: predicting dropout for nontraditional undergraduate students: a machine learning approach. j. coll. student retent.: res. theory pract. 24(4), 1054–1077 (2023) article  google scholar  ivankova, n.v., stick, s.l.: students’ persistence in a distributed doctoral program in educational leadership in higher education: a mixed methods study. res. high. educ. 48(1), 93–135 (2007) article  google scholar  ke, g., et al.: lightgbm: a highly efficient gradient boosting decision tree. adv. neural inf. process. syst. 30 (2017) google scholar  kelly, j.d.o., menezes, a.g., de carvalho, a.b., montesco, c.a.: supervised learning in the context of educational data mining to avoid university students dropout. in: 2019 ieee 19th international conference on advanced learning technologies (icalt), vol. 2161, pp. 207–208. ieee (2019) google scholar  kotsiantis, s.: educational data mining: a case study for predicting dropout-prone students. int. j. knowl. eng. soft data paradigms 1(2), 101–111 (2009) article  google scholar  lehmann, w.: “i just didn’t feel like i fit in”: the role of habitus in university dropout decisions. can. j. high. educ. 37(2) (2007) google scholar  leonhardt, d., chinoy, s.: the college dropout crisis. the new york times (2019). https://www.nytimes.com/interactive/2019/05/23/opinion/sunday/college-graduation-rates-ranking.html lozano, j.m., rua vieites, a., bilbao-calabuig, p., casadesús-fa, m.: university student retention: best time and data to identify undergraduate students at risk of dropout. innov. educ. teach. int. 57, 1–12 (2018) article  google scholar  manhaes, l., manhães, b., cruz, s., costa, m., zavaleta, j., silva, g.: identificação dos fatores que influenciam a evasão em cursos de graduação através de sistemas baseados em mineração de dados: uma abordagem quantitativa. in: viii simpósio brasileiro de sistemas de informação (2012) google scholar  martins, l.c.b., carvalho, r.n., carvalho, r.s., victorino, m.c., holanda, m.: early prediction of college attrition using data mining. in: 2017 16th ieee international conference on machine learning and applications (icmla), pp. 1075–1078. ieee (2017) google scholar  da matta, k.w.: evasão universitária estudantil: precursores psicológicos do trancamento de matrícula por motivo de saúde mental. master’s thesis, universidade de brasília (2011) google scholar  meedech, p., iam-on, n., boongoen, t.: prediction of student dropout using personal profile and data mining approach. in: lavangnananda, k., phon-amnuaisuk, s., engchuan, w., chan, j. (eds.) intelligent and evolutionary systems. proceedings in adaptation, learning and optimization, vol. 5, pp. 143–155. springer, cham (2016). https://doi.org/10.1007/978-3-319-27000-5_12 chapter  google scholar  mello, r.f., ponti, m.a.: machine learning: a practical approach on the statistical learning theory. springer, heidelberg (2018). https://doi.org/10.1007/978-3-319-94989-5 book  math  google scholar  nistor, n., neubauer, k.: from participation to dropout: quantitative participation patterns in online university courses. comput. educ. 55(2), 663–672 (2010) article  google scholar  pal, s.: mining educational data using classification to decrease dropout rate of students. int. j. multidisc. sci. eng. 3, 35–39 (2012) google scholar  ponti, m.: combining classifiers: from the creation of ensembles to the decision fusion. in: 2011 24th sibgrapi conference on graphics, patterns, and images tutorials, pp. 1–10. ieee (2011) google scholar  powdthavee, n., vignoles, a.: the socio-economic gap in university dropout. be j. econ. anal. policy 9(1) (2009) google scholar  ribeiro, m.: o projeto profissional familiar como determinante da evasão universitária: um estudo preliminar. rev. brasileira orientacao prof. 6, 55–70 (2005) google scholar  santos, r.s., ponti, m.a., rodrigues, k.r.h.: evasão universitária e estratégias para retenção de alunos com base em intervenções remotas. in: anais estendidos do simpósio brasileiro de fatores humanos em sistemas computacionais (ihc) (2022) google scholar  dos santos baggi, c.a., lopes, d.a.: evasão e avaliação institucional no ensino superior: uma discussão bibliográfica. avaliação: rev. avaliação educação superior (campinas) 16, 355–374 (2011) google scholar  sarra, a., fontanella, l., di zio, s.: identifying students at risk of academic failure within the educational data mining framework. soc. indic. res. 146(1), 41–60 (2019) article  google scholar  da silva, p.m., lima, m.n., soares, w.l., silva, i.r., roberta, a.d.a., de souza, f.f.: ensemble regression models applied to dropout in higher education. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 120–125. ieee (2019) google scholar  solís, m., moreira, t., gonzalez, r., fernandez, t., hernandez, m.: perspectives to predict dropout in university students with machine learning. in: 2018 ieee international work conference on bioinspired intelligence (iwobi), pp. 1–6. ieee (2018) google scholar  de souza, a.m.: machine learning e a evasão escolar: análise preditiva no suporte à tomada de decisão. master’s thesis, faculdade de ciências empresariais (2020). https://repositorio.fumec.br/xmlui/handle/123456789/420 stein, c.: the push for higher education: college attrition rates. pa times org (2018). https://patimes.org/the-push-for-higher-education-college-attrition-rates/ tontini, g., walter, s.: pode-se identificar a propensão e reduzir a evasão de alunos? ações estratégicas e resultados táticos para instituições de ensino superior. avaliação rev. avaliação educação superior (campinas) 19, 89–110 (2014) google scholar  triado, x., sallán, j., feixas, m., figuera, p., chueca, p., fonseca, m.: student dropout rates in catalan universities: profile and motives for disengagement. qual. high. educ. 20, 165–182 (2014) article  google scholar  veloso, t.c.m.a., de almeida, e.p.: evasão nos cursos de graduação da universidade federal de mato grosso, campus universitário de cuiabá um processo de exclusão. série-estudos perioódico mestrado educação ucdb (13), 133–148 (2002) google scholar  xenos, m., pierrakeas, c., pintelas, p.: a survey on student dropout rates and dropout causes concerning the students in the course of informatics of the hellenic open university. comput. educ. 39(4), 361–377 (2002) article  google scholar  download references author information authors and affiliations instituto de ciências matemáticas e de computação, universidade de são paulo, av. trabalhador são carlense, 400, são carlos, sp, brazil rodolfo sanches santos, moacir antonelli ponti & kamila rios rodrigues authors rodolfo sanches santosview author publications search author on:pubmed google scholar moacir antonelli pontiview author publications search author on:pubmed google scholar kamila rios rodriguesview author publications search author on:pubmed google scholar corresponding author correspondence to kamila rios rodrigues . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper santos, r.s., ponti, m.a., rodrigues, k.r. (2023). analyzing college student dropout risk prediction in real data using walk-forward validation. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_19 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_19 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords higher education college dropout machine learning publish with us policies and ethics profiles kamila rios rodrigues view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature towards generating p-contrastive explanations for goal selection in extended-bdi agents | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper towards generating p-contrastive explanations for goal selection in extended-bdi agents conference paper first online: 12 october 2023 pp 351–366 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) towards generating p-contrastive explanations for goal selection in extended-bdi agents download book pdf download book epub henrique jasinski  orcid: orcid.org/0000-0003-3219-30739, mariela morveli-espinoza  orcid: orcid.org/0000-0002-7376-22719 & cesar augusto tacla  orcid: orcid.org/0000-0002-8244-89709  part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 531 accesses abstract as agent-based systems have been growing, more and more the general public has access to them and is influenced by decisions taken by these systems. this increases the necessity for such systems to be capable of explaining themselves to a user. the beliefs-desires-intentions (bdi) is a commonly used agent model that has an two-phase internal goal selection process (desires and intentions) to decide what goals to pursue. belief-based goal processing (bbgp) model is an extended-bdi model whose selection process consists of four phases. this more-grained behavior may have relevant consequences for the analysis of what an intention is and better explain how an intention becomes what it is. contrastive explanations are commonly employed by people and can bring benefits to the explanation exchange process. a property-contrast (p-contrast) explanation is a type of contrastive explanation that compares the properties of an explanation object. thus, we can take a goal as the object about which we want an explanation and its status in the selection process (that is, how much it has advanced) as a property. this work tackles the problem of generating p-contrast explanations and proposes a method to construct them. the method consists of two phases, which in turn consist of a set of steps. the first step of the second phase returns a set of beliefs that constitute a future explanation. thus, this work focuses on the first phase and the first step of the second phase. we use a scenario of the cleaner world in order to illustrate the performance of our proposal. supported by organization capes/brazil and cnpq proc. 409523/2021-6. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others quantitative verification and strategy synthesis for bdi agents chapter © 2023 human-agent collaboration: a goal-based bdi approach chapter © 2019 an argumentation-based approach for explaining goals selection in intelligent agents chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. logical analysis formal reasoning logic motivated behavior reasoning symbolic ai 1 introduction in intelligent agents, practical reasoning is the process of figuring out what to do. according to wooldridge [18], practical reasoning involves two phases: deliberation and means-ends reasoning. deliberation is decomposed in two parts: (i) firstly, the agent generates a set of possible goals and (ii) then, the agent chooses which goals he will be committed to bring about. this process is known as goal selection. in order to make this decision, a goal may have to pass from more than one stage depending on the practical reasoning model. bdi model [1] is a very known and used practical model that has two stages: (i) desires represent some end state that the agent deems as desirable and (ii) intentions are desires that the agent is committed to, with their respective plans. bbgp model [2] can be seen an extended model of bdi that consider four stages for goal processing: (i) activation, (ii) evaluation, (iii) deliberation, and (iv) checking. consequently, four different statuses for a goal are defined: (i)active, (ii)pursuable, (iii)chosen, and (iv)executive. figure 1 shows a parallel between these models. the schema presents the goal reasoning stages and the status of a goal after passing each stage. in [2], the authors claim that chosen goals can be seen as future-directed intentions and executive ones as present-directed intentions. thus, chosen and executive goals are intentions from a bdi perspective. fig. 1. schema of the goal reasoning stages in bdi and bbgp models. this figure was extracted from [10]. full size image contrastive explanations are commonly employed by people and can bring benefits to the exchange process during the explanation. contrastive questions provide an insight into the questioner’s mental model, allowing to have a better understanding of what they do not know and contrastive explanations usually are more straightforward, more feasible, and less demanding both for the questioner and explainer [8]. in [16], the authors distinguish three types of contrastive questions: (p-contrast) why does object o have property p, rather than property \(p'\)? (o-contrast) why does object o have property p, while object \(o'\) has property \(p'\)? (t-contrast) why does object o have property p at time t, but property \(p'\) at time \(t'\)? in this work, we focus on generating explanations for p-contrast questions. we consider goals as the object and the status of goals as properties. interest in explainable artificial intelligence (xai) grounded on social and cognitive sciences studies has grown recently [9] as well as contrastive explanations [15]. despite such growth, there are relatively few works directed to the end user with such grounding (e.g., [5, 7, 14]). since the explanation process is an exchange of information most likely involving a person, it should follow the rules of conversation. grice [4] presents the cooperation principle with four categories: quantity, quality, relation, and manner. each with a set of maxims that two entities in a cooperative dialogue are expected to follow. the “quantity” category is of particular interest in this work. its maxims state that one should be as informative as necessary, but not more than that. thus, we will take into account this category when generating the conditions for future explanations. regarding explainable agents focused on goal selection, to the best of our knowledge, no contrastive explanation generation approach was found. however, there are some related approaches for generating ontological and causal explanations (e.g., [10, 11, 17]). against this background, we can formulate our research questions: 1. how to generate p-contrastive explanations for goal selection process? 2. considering social and cognitive sciences works, what information is required for that generation? in order to address the first question, we propose a method for generating p-contrastive explanations, which are constructed from the agent’s execution log. the method does not present a final user-ready explanation. it provides a set of conditions (beliefs) that are required to attribute a causal relation to the p-contrastive question and that are the base for future explanations. since a goal can advance or go back during goal selection, we generate explanations for each case and also consider the case when a goal changes its status due to a preference order, which happens when there are conflicts between goals and preference is used to determine the status change. regarding the second question, the method is based on grice’s cooperation principles as requirements for the quality of explanations, specifically on the quantity category. in this sense, the resultant set of conditions includes a concise set of beliefs that are directly related to the status change of goals. the remainder of this paper is structured as follows. in sect. 2, we present method for generating explanations. in sect. 3, we focus on the generation of conditions for p-contrastive explanations. some related work is presented in sect. 5. finally, sect. 6 is devoted to conclusions and future work. 2 explanation generation method this section presents the agent architecture and the method for generating explanations. 2.1 building blocks we begin by presenting the agent architecture. an agent in the context of this work is an artificial entity that has a set of beliefs, a set of goals, and a set of rules. the temporal aspect of the agent execution is captured by the goals reasoning cycles. a cycle is considered to be a single execution of the selection process. first of all we present the logical language that will be used. let \(\mathcal {l}\) be a first order logical (fol) language which will be used to represent the goals, beliefs, and rules of an agent. the following symbols \(\wedge , \vee , \rightarrow \), and \(\lnot \) denote the logical connectives conjunction, disjunction, implication, and negation, and \(\vdash \) stands for the inference. definition 1 (agent). an agent \(\texttt{agt}\) is a tuple \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \), where \(\mathcal {k}\mathcal {b}= \mathcal {b}\cup \mathcal {r}\) is the knowledge base, which is composed of a finite set of beliefs \(\mathcal {b}\) and a set of finite rules \(\mathcal {r}\), \(\mathcal {g}\) is a finite set of goals, and \(\mathcal {p}\) is a set of plans. a goal \(g \in \mathcal {g}\) is a tuple \(g=\langle gid, st\rangle \) where gid is a unique atom representing goal g and st is the current status of goal g. a rule \(r \in \mathcal {r}\) is an expression of the form \(r=\bigwedge b \rightarrow h\), where \(b \in \mathcal {b}\) and \(h\in \mathcal {b}\) or \(h\in \mathcal {g}\). let \(p \in \mathcal {p}\) be a plan such that \(p=\langle g, gd, act \rangle \), where \(g \in \mathcal {g}\), gd is the set of guard clauses that act as a precondition for the plan execution, and act is an ordered list of actions that allows the agent to achieve the goal g. \(\mathcal {b}\) and \(\mathcal {g}\) are subsets of literalsfootnote 1 from the language \(\mathcal {l}\). it also holds that \(\mathcal {b}\) and \(\mathcal {g}\) are pairwise disjoint. for this method, the nature of the agent’s plan library and the actions list are not relevant. in the case of plans, only the goal and the guard clause are required information. a guard clause is a literal in language \(\mathcal {l}\). an explanation can be described as the assignment of causal responsibility, as it presents possible causes for what is being explained [6]. thus, by using the agent’s beliefs and the causal relationship between them and their goals, contrastive explanations can be constructed. since there is a causal relation between the beliefs and goals, the rules and plans of the agent knowledge base can be used to define it. thus, a cause is a relation between a set of beliefs and a single event, which is either a belief or a goal. definition 2 (cause). let \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \) be an agent and \(\texttt{e}=\mathcal {b}\cup \mathcal {g}\) be a finite set of events. a cause c is a tuple \(c=\langle \texttt{e}, \texttt{cond} \rangle \), where \(\texttt{e} \in \texttt{e}\) is an event and \(\texttt{cond} \subset \mathcal {b}\) is a set of beliefs called conditions. if \(\mathcal {k}\mathcal {b}\vdash \texttt{cond}\) then event \(\texttt{e}\) is expected to happen. let \(\texttt{c}\) be the set of causes. given a cause \(c=\langle \texttt{e}, \texttt{cond} \rangle \), we say that it is activated if \(\forall b \in \texttt{cond}\) \(\mathcal {k}\mathcal {b}\vdash b\) otherwise, if \(\exists b \in \texttt{cond}\) such that \(\mathcal {k}\mathcal {b}\nvdash b\) then the cause c is deactivated. when two events share a piece of causal history, they are said to be causally related. besides, the causal history of an event is called a causal chain. formally, let \(c=\langle \texttt{e}, \texttt{cond} \rangle \), \(c'=\langle \texttt{e}', \texttt{cond}' \rangle \), and \(c''=\langle \texttt{e}'', \texttt{cond}'' \rangle \) be three causes s.t. \(c, c', c'' \in \texttt{c}\): c’ and c are in a causal chain if \(\texttt{e}' \in \texttt{cond}\) or if \(\texttt{e}'' \in \texttt{cond} \wedge \texttt{e}' \in \texttt{cond}''\) c and c’ are causally related if \(\exists c'' \in \texttt{c}\), such that c” is in a causal chain with c and c’. these relations are denoted by \(\texttt{related}(c,c')\in \{true, false\}\). causes describes a relation of a set of beliefs with another belief or goal. this relation can be extracted from rules and plans. thus, we convert rules and plans to causes as follows: plan to cause conversion: given a plan \(p=\langle g, gd, act \rangle \), a cause c from p is \(c=\langle g, gd \rangle \). rule to cause conversion:. given a rule \(r=\bigwedge b \rightarrow h\), a cause c from r is \(c=\langle h, \bigwedge b \rangle \). 2.2 the proposed method in order to generate a contrastive explanations about a past decision during goal selection, it is necessary to reproduce the agent’s knowledge base configuration when the explained events happened. thus, a memory of the execution needs to be kept. we propose a method that consists of two phases: interface and explanation generator. figure 2 shows these phases and the required precedures in each phase. in the interface, four elements are required: i) the execution history is a log-like ordered list of events, ii) the preference function encodes a preference relationship between two events, iii) the causal function is responsible for mapping events to the set of conditions that are causally related to it, and iv) the conflict function is responsible for identifying if two given goals have any sort of incompatibilities between them. the explanation generator has four domain-independent elements: i) the presumptions functions that, given a set of beliefs, return the subset containing only the presumptions (that is, beliefs that need no support); ii) the generate related causes procedure is responsible for receiving the posed question, retrieving the required information using the functions defined in interface and the presumption functions, and returns the set of related conditions; iii) the generate possible explanations procedure takes as input the set of related conditions and outputs sets of explanations, where each possible explication is a subset of the related conditions; and iv) the select an explanation procedure, which takes the set of possible explanations and returns one considered the “most adequate” according to the context. these last elements are out of the scope of this work. thus, we focus on generating a set of related conditions. fig. 2. schema of the explanation generation method. full size image 2.2.1 interface each entry in the execution history needs to encapsulate, for each reasoning cycle, the set of goals that changed their status and the set of beliefs that were added or removed from the agent’s knowledge base. note that the goals changes happened during the deliberation, but the beliefs changes are updated after the deliberation has finished. thus, for a cycle i, the reconstructed knowledge base needs to apply the changes only up to cycle \(i-1\). definition 3 (execution history entry). let \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \) be an agent and \(\texttt{stats}\) be a list of possible statuses of a goal. an entry for the execution history is a tuple \(hist=\langle i, \mathcal {g}_h, \mathcal {b}_h\rangle \), where i is the cycle identifier, \(\mathcal {g}_h\) is the set of changed goals s.t. a changed goal \(g_h\) is a tuple \(g_h=\langle g, st\rangle \) where \(g \in \mathcal {g}\) and \(st \in \texttt{stats}\) is the new status of goal g, and \(\mathcal {b}_h\) is the set of changed beliefs s.t. a changed belief \(b_h\) is a tuple \(b_h=\langle b,v\rangle \) where \(b \in \mathcal {b}\) and \(v\in \{\texttt{add}, \texttt{rem}\}\) represents the beliefs addition or removal, respectively. we now define the functions employed in the interface: definition 4 (causal, preference, and conflict functions). let \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \) be an agent, \(\texttt{e}=\mathcal {b}\cup \mathcal {g}\) be the set of events, and \(\texttt{c}\) the set of causes. causal function: \(\texttt{causes}: \texttt{e} \longrightarrow 2^{\texttt{c}}\) maps an event \(e \in \texttt{e}\) to the set of every cause related to it. preference function: \(\texttt{preferred}: \mathcal {g}^2 \times \mathbb {n} \longrightarrow \{-1, 0, 1\}\). this function takes as input two goals and the cycle index where they are conflicting and returns the preference relation between both goals. thus, \(-1\) represents that the first goal of the pair is less preferred than the second one, 0 represents that both are equally preferred, and 1 denotes that the first goal of the pair is more preferred than the second one. conflict function: \(\texttt{conflict}: \mathcal {g}^2 \longrightarrow \{true, false\}\). this function takes as input two goals and returns if they are conflicting or not. we have that true denotes that there is a conflict between both goals of the pair and false denotes that there is no conflict. 2.2.2 explanation generator different patterns can be observed when analyzing how an agent answers questions about its goals. such patterns represent how the reasoning mechanism deliberates about the selected goal in each step of the goal selection. the three observed patterns are: a) positive filter beliefs, which are beliefs that lead to the advance of the targeted goal to its next status; b) negative filter beliefs, which are beliefs that impede the advance of a given goal to its next status; and c) preference filters, which express the preference relationship between conflicting goals. the agent’s goals always have some associated status. the set of possible statuses is dependent on the agent model. for example, the bdi model defines two statuses: desire and intention; in turn, the bbgp model defines four: active, pursuable, chosen, and executive. the possible statuses of a goal are ordered. considering that order, a status can be regarded as subsequent to another if it comes afterward in the list. thus, given a finite ordered list of possible statuses \(\texttt{stats}=\{st_1, ..., st_n\}\) and \(st_j, st_k \in \texttt{stats}\) two possible statuses. we say that: \(st_k\) is directly subsequent to \(st_j\) if \(k=j+1\). \(st_k\) is subsequent to \(st_j\) if \(k>j\). the expanded knowledge base, denoted by \(\mathcal {k}\mathcal {b}^+\), is the knowledge base \(\mathcal {k}\mathcal {b}\) of the agent and everything that entails from it. it contains everything that the agent considers as true. definition 5 (expanded knowledge base). let \(\mathcal {k}\mathcal {b}\) be the knowledge base of the agent, the expanded knowledge base \(\mathcal {k}\mathcal {b}^+=\{\varphi \mid \mathcal {k}\mathcal {b}\vdash \varphi \}\), and \(\not \exists \varphi \in \mathcal {k}\mathcal {b}^+ \mid \mathcal {k}\mathcal {b}\vdash \varphi \), where \(\varphi \not \in \mathcal {k}\mathcal {b}^+\). the explanations are built using the most fundamental beliefs of the agent, called presumptions, as they need no support to be considered true. thus, \(\forall r \in \mathcal {r}\), if \(r: \emptyset \rightarrow h\) then h is a presumption. there are two types of presumption extraction functions, one when the desired outcome is an activated causal rule and the other when the desired outcome is that a deactivated causal rule. the function \(\mathtt {d\_pre}\) maps a set of causes to the corresponding set of presumptions used in them. the following disjunctive form of the function is required for answers that need to deactivate a cause, as it represents the idea that if a single cause were removed, the event would not have happened. definition 6 (disjunctive presumptions). let \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \) be an agent, \(\texttt{e}\) be a set of events, \(e \in \texttt{e}\) a given event, and \(\texttt{c}=\{c_1, ..., c_n\}\) be a set of causes. \(\mathtt {d\_pre}\) is defined as follows: $$\begin{aligned} \mathtt {d\_pre}(\texttt{c}) = \left\{ \begin{array}{ll} \bigcup _{i=1}^n\mathtt {d\_pre}(\mathtt {c_i}) &{}if \mid \texttt{c} \mid >1\\ h &{} if\; \texttt{c}=\{\emptyset \rightarrow h\} \wedge h\in \mathcal {b}\\ \bigcup _{j=1}^m \mathtt {d\_pre}(\texttt{causes}(b_j)) &{} if\; \texttt{c}= b_1, ..., b_m \rightarrow e \end{array} \right. \end{aligned}$$ (1) let us now focus on conjunctive presumptions. every presumption in a cause of event e forms a single conjunctive formula. there is one formula for each different cause of e. this conjunctive form is used for answers that need to activate a cause, as it represents the idea that if all the beliefs that are missing were to become true, the event would have happened. definition 7 (conjunctive presumptions). let \(\texttt{agt}=\langle \mathcal {k}\mathcal {b}, \mathcal {g}, \mathcal {p}\rangle \) be an agent, \(\texttt{e}\) be a set of events, \(e \in \texttt{e}\) a given event, and \(\texttt{c}=\{c_1, ..., c_n\}\) be a set of causes. \(\mathtt {c\_pre}\) is defined as follows: $$\begin{aligned} \mathtt {c\_pre}(\texttt{c}) = \left\{ \begin{array}{ll} \bigcup _{i=1}^n\mathtt {c\_pre}(\mathtt {c_i}) &{}if \mid \texttt{c} \mid >1\\ h &{} if\; \texttt{c}=\{\emptyset \rightarrow h\} \wedge h\in \mathcal {b}\\ \bigwedge _{j=1}^m \mathtt {c\_pre}(\texttt{causes}(b_j)) &{} if\; \texttt{c}= b_1, ..., b_m \rightarrow e \end{array} \right. \end{aligned}$$ (2) figure 3 shows an example of applying the presumption functions on the causes for \(\lnot mop(x,y)\). rectangles denote the event, ellipses denote causes, dashed lines denote that the causes are directly related to the event, solid lines denote that the causes are in a causal chain. if two solid lines are joined by a line, it denotes that their causes are connected by logic conjunction. in (a) it is possible to see the conjunctive function, where the presumptions for each cause form a single formula. in (b) it is possible to see that the causes are disjoint. let us recall that there is a preference relation between goals. this relation is especially important between conflicting goals since it is the reason for selecting one over the other. this preference is denoted by \(g > g'\), which means that g is more preferred than \(g'\). sometimes two sets of goals need to be compared, especially when the goal being explained has a low overall preference and other compatible goals help it to become selected. note that both sets of goals must be internally compatible. this relation is denoted by \(\mathcal {g}' > \mathcal {g}''\), which means that \(\mathcal {g}'\) is preferred over \(\mathcal {g}''\). sometimes a single goal has a very high priority, that is, it is preferred over a set of other goals. this is important when the high priority of such goal can explain its selection over any other goals. we denote it by \(g \gg \mathcal {g}'\), which means that g is maximally preferred over a set of goals \(\mathcal {g}' \subset \mathcal {g}\). the procedure generate related causes is tackled in the next section because the output depends on the type of contrast explanation. fig. 3. example of presumption functions for goal \(\lnot mop(x,y)\). (a) shows a conjunction example and (b) shows a disjunction example. full size image 3 generating conditions for p-contrast explanations in this section, we focus on generating a set of conditions for p-contrast explanations. p-contrast questions are of the form: “why is goal g in status st rather than in status st’?”. the answer is “a set of absent conditions \(\texttt{rc}\) that would ensure that g achieves status \(st'\). two sub-cases can be defined considering the sequence of the statuses: 1. \(st' \succ st\), that is, “why has not g advanced in the goal reasoning stages? 2. \(st\succ st'\), that is, “why has not g gone back in the goal reasoning stages? (case 1). for \(st' \succ st\), each deliberative pattern is defined as follows: positive: the conditions for the explanation needs to present the set of presumptions that, if present, would have ensured the goal’s progress. this set can be obtained in the following manner: \(\texttt{rc}=\mathtt {c\_pre}(\texttt{causes}(g))-\mathcal {k}\mathcal {b}\). negative: the explanation needs to present a set of presumptions that, if were made absent, would have allowed the goal to progress freely. the set is obtained in the following manner: \(\texttt{rc}=\mathtt {d\_pre}(\texttt{causes}(\lnot g))\cap \mathcal {k}\mathcal {b}\). preference: although no presumption was missing or impeding the goal progression, two cases may occur there was a conflict between the goal that is object of the explanation and another goal and the lack of preference made the goal not to be selected to advance. thus, the conditions of an explanation are assertions about the preference, identifying why g was not selected: \(\texttt{rc}=\texttt{conflict}(g,g')=true, \texttt{preferred}(g',g,i)=true\). (case 2). for \(\texttt{st} \succ \texttt{st}'\), each deliberative pattern is defined as follows: positive: the conditions for an explanation need to present the set of presumptions that, if present, would have ensured the goal regression. this set can be obtained in the following manner: \(\texttt{rc}=\mathtt {d\_pre}(\texttt{causes}(\lnot g))\cap \mathcal {k}\mathcal {b}\). negative: the conditions for an explanation need to present a set of presumptions that, if are in the \(\mathcal {k}\mathcal {b}\), would have impeded the goal to stay with status st. the set is obtained in the following manner: \(\texttt{rc}=\mathtt {c\_pre}(\texttt{causes}(g)) -\mathcal {k}\mathcal {b}\). preference: although no presumptions forced the goal regression, a conflict of goals was expected, but the preference made the goal to be selected to continue with the status st. thus, the conditions of an explanation are assertions about the preference, identifying why g was not selected: \(\texttt{rc}=\texttt{conflict}(g,g')=true, \texttt{preferred}(g,g',i)=true\). 4 case study: cleaner world scenario in this section, a case study is presented. we show the mental state of the agent and show how the conditions fora p-contrast explanation are generated. in the cleaner world scenario, the agent is a robot in charge of cleaning a certain area. in the robot’s representation, the area is divided into cells referred to by a coordinate system (x, y). the robot can move along the cells, clean solid and liquid dirt by sweeping and mopping, respectively, recharging itself when necessary, replacing some broken parts, and disposing of the dirt in its internal storage. lastly, for the robot not to disrupt the passage when there are no jobs at the moment, there is a command for it to rest. there are two special locations for the robot, the workshop and the dumpster. in the workshop, the robot can find replacement parts, recharge, and is where it is assigned to rest. the dumpster is where the dirt is disposed of. to do the mopping, the robot needs a functional mop part, and to sweep, the robot needs a functional vacuum. the mop is a part that the robot can replace by itself, given that there is a replacement available. on a given day, the following series of events were observed. at time \(t_0\), three cells were dirty, where (2,2) and (3,2) had liquid dirt and (3,4) solid dirt. at \(t_1\), the robot cleaned (2,2). followed by cell (3,4) at \(t_2\). the robot proceed to go back to the workshop at \(t_3\). 4.1 mental states the agent’s knowledge base, goals, and plans for the scenario are described next. let us begin describing the agent’s starting beliefs, that is, the beliefs it had at \(t_0\) before the first deliberation cycle. thus, we have \(\mathcal {b}=\{workshop(1,1)\), dumpster(4, 1), \(\lnot available(mop)\), at(1, 1), have(battery, 75), have(cargo, 65), \(solid\_dirt(3,4)\), \(liquid\_dirt(3,2)\), \(liquid\_dirt(2,2)\}\). the goals are: \(\mathcal {g}=\{(replace(x), 0)\), (mop(x, y), 0), (sweep(x, y), 0), (recharge, 0), (rest, 0), \((dispose,0)\}\), where 0 means that the goal do not have a status yet. table 1 shows the rules for the scenario. in the bbgp model, there are four types of rules, each type for each stage. only the activation and evaluation rules are depicted because the rules for deliberation and checking stages are fixed since they are domain-independentfootnote 2. table 1. activation and evaluation rules.full size table the preference order of goals is exactly the activation rules order, in decreasing order. that is, mop(x, y) has the highest preference and recharge has the lowest one. let us recall that the first phase of the method, that is, interface, requires four elements to be implemented: the execution history, the causal function, the preference function, and the conflict function. for our scenario, the agent’s execution history is depicted in table 2, where each row is an entry of the log. the log allows for the reconstruction of the agent knowledge base and tracks the goal status changes. table 2. execution history.full size table the causal function: the activation and evaluation rules can be easily converted into causes by attributing the consequent (the goal) as e and the set of beliefs in the antecedent as \(\texttt{cond}\). table 3 shows the mapping generated. the bbgp model has two more stages: deliberation and checking. the deliberation has no rules to be converted, as it evaluates conflicts and preferences among goals, it will be encoded by the conflict and preference functions. table 3. mapping of the activation and evaluation rules to causes.full size table the preference function: in the example, the model defines a preference, which was described with the scenario, as such, using the execution history, the total (decreasing) preference relation for the scenario is as follows: \( mop(2,2)> mop(3,2)> sweep(3,4)> recharge\). the conflict function: given the nature of the scenario and the defined goals, every goal is incompatible with each other. this is the case because the goals are localized in space, and the robot needs to be at the specified place to perform the actions to achieve the goal. 4.2 the conditions for a p-contrast explanation we can now present how the conditions for generating a p-contrast explanation are constructed. let us start by posing a question: why mop(3, 2) is active, instead of executive? the rationale of this question is very straightforward: why a goal is not being executed? first, let us identify the information in the question: goal: mop(3, 2) status: active foil status: pursuable (from executive) cycle id: 2 notice that in the posed question, the contrasting statuses are active and executive. in this case, executive is converted to pursuable, a previous case from executive and directly subsequent to active because the goal did not even became pursuable so we have to know why this happened. since the question requires a single time frame, which is cycle 2, the knowledge base for that cycle can be reconstituted using the execution history by adding and removing the beliefs for each cycle to the initial beliefs, until the desired cycle. we have that the relevant beliefs are: workshop(1, 1) dumpster(4, 1) \(\lnot available(mop)\) at(2, 2) have(battery, 35) \(solid\_dirt(3,4)\) \(liquid\_dirt(3,2)\) broken(mop) related causes procedure: first, it is necessary to evaluate the relation of the statuses. be \(st=active\) the factual status and \(st'=pursuable\) the foil status such that \(st \prec st'\), as such the relevant patterns are for “not advanced its status”. for an active goal become pursuable, it is assessed in the evaluation stage, that is, the second stage of the bbgp model. in the case of evaluation stage, it uses only negative filters and the way of calculating this filter is: \(\mathtt {d\_pre}(\texttt{causes} (\lnot mop(3,2))) \cap \mathcal {k}\mathcal {b}^+\). for this scenario \(\mathcal {k}\mathcal {b}^+=\mathcal {k}\mathcal {b}\) because there are no rules with only beliefs that may infer another beliefs. thus, the calculation is as follows for the negative pattern: first, the set of causes for (mop(3, 2) is calculated. for simplicity, we only write the set of beliefs that are part of \(\texttt{cond}\): $$\begin{aligned} \texttt{causes}(mop(3,2))={} & {} \{\{\lnot have(battery, 40), liquid\_dirt(3,2)\},\\ {} & {} \,\,\,\{have(cargo,80), liquid\_dirt(3,2)\}\\ {} & {} \,\,\,\{broken(mop), liquid\_dirt(3,2)\}\} \end{aligned}$$ then, the disjunctive presumptions are calculated: $$\begin{aligned}\mathtt {d\_pre}(\texttt{causes}(mop(3,2)))= {} & {} \{\lnot have(battery, 40),liquid\_dirt(3,2), \\ {} & {} \,\,\, have(cargo,80),broken(mop)\} \end{aligned}$$ finally, we obtain the set of conditions: $$\begin{aligned}\texttt{rc}= \mathtt {d\_pre}(\texttt{causes}(mop(3,2))) \cap \mathcal {k}\mathcal {b}^+= {} & {} \{\lnot have(battery, 40), liquid\_dirt(3,2),\\ {} & {} \,\,\, have(cargo,80),broken(mop)\} \end{aligned}$$ this concludes with the procedure whose output is the set of conditions \(\texttt{rc}\), which constitute future explanations. 5 related work in this section, some articles about contrastive explanations generation and explanations for goal selections are presented. besides, we compare them with our proposal. to the best of our knowledge, there are few works where contrastive explanations are used in the context of intelligent agents. in [13], the authors propose an approach for generating contrastive explanations for questions of the for “why p and not q?”, where p and q are stand-ins for the current robot plan and the foil one, respectively. an algorithm for generating minimal explanations for users of different levels of expertise is presented. the result is that agents can explain its ongoing or planned behavior in a way that is both tailored to the user’s background and is designed to reduce cognitive burden on the user’s end. in [3], the authors propose an approach for generating contrastive explanations about agents’ preferences in the context of reinforcement learning. the explanations identify the state feature values that certain agents prefers compared to the other agent. in [12], the work focus on generating contrastive explanations in planning, more specifically, for questions such as “why action a instead of action b?”. thus, the contrastive explanation that compares properties of the original plan containing a against the contrastive plan containing b. we can notice that the contrastive questions tackled in these works have a different nature compared with the p-contrast question of our approach. in our approach, we consider an object of the explanation and a property about it, which gives one more dimension to the generated explanation. regarding explanation generation for goal selection. to the best of our knowledge, only [11] and [10] present an explanation method for goal selection. their approach also considers bbgp-based agents and uses the formal argumentation base for generating explanations. however, there is no a social sciences grounding and it does not account for contrastive explanations. we can see our approach as a complement to these works. 6 conclusions and future work this article presented a method for generating p-contrastive explanations about the bbgp-based goal selection process. a set of beliefs – called conditions – is returned to the questioner and is the base for future explanations. although, we present the phases and all the steps for generating a final explanation, the final procedure was not developed; however, all the building blocks for generating an explanations were established. we illustrate how our method works by applying it to the scenario of the cleaner world. let us recall that our method used grice’s cooperation principles as a guide for how the agent should present its explanations, specifically, we base on the quantity category. this category is concerned with the amount of information provided: “be as informative as required, and no more than that”. it is subjective how informative a piece of information is. not just in the sense that it depends on who receives the information, but it is also not measurable. in a sense, this is also part of the selection problem: how to select the most adequate answer?. we could say that the “most informative” explanations could be a solution to the problem. the method addresses this requirement by avoiding repetitive information, that is, if a condition \(c_1\) can answer all the required events, there is no need to include a second condition. as mentioned above, an important steps of the method were not tackled in this work. thus, the immediate future work is to generate the set of possible explanations. besides, we plan to study the selection problem, which focuses on deciding which single explanation will be returned as final answer. as presented in introduction, there are other types of contrastive question and explanations. we aim to generate the set of conditions and explanations for these types of explanations. finally, another possible improvement is to include special questions that can better explain internal behaviors of the agent that are beyond the standard bbgp model. a good example would be a special question type to explain trust deliberation in an agent. notes 1.a literal is either an atomic formula or the negation of an atomic formula. when a literal is an atomic formula, we say that it is a positive literal, and when a literal is the negation of an atomic formula, we say it is a negative literal. 2.for more details about the bbgp model, the reader is referred to [10]. references bratman, m.: intention, plans, and practical reason (1987) google scholar  castelfranchi, c., paglieri, f.: the role of beliefs in goal dynamics: prolegomena to a constructive theory of intentions. synthese 155, 237–263 (2007) article  mathscinet  google scholar  gajcin, j., nair, r., pedapati, t., marinescu, r., daly, e., dusparic, i.: contrastive explanations for comparing preferences of reinforcement learning. in: aaai conference on artificial intelligence (2022) google scholar  grice, h.p.: logic and conversation. in: [cole, p., morgan, j.l. (eds.) syntax and semantics, vol. 3, speech acts (1975)] google scholar  harbers, m., van den bosch, k., meyer, j.j.: design and evaluation of explainable bdi agents. in: 2010 ieee/wic/acm international conference on web intelligence and intelligent agent technology, vol. 2, pp. 125–132. ieee (2010) google scholar  josephson, j.r., josephson, s.g.: abductive inference: computation, philosophy, technology. cambridge university press (1996) google scholar  kaptein, f., broekens, j., hindriks, k., neerincx, m.: the role of emotion in self-explanations by cognitive agents. in: 2017 seventh international conference on affective computing and intelligent interaction workshops and demos (aciiw), pp. 88–93. ieee (2017) google scholar  miller, t.: contrastive explanation: a structural-model approach. knowl. eng. rev. 36, e14 (2021) article  google scholar  miller, t., howe, p., sonenberg, l.: explainable ai: beware of inmates running the asylum or: how i learnt to stop worrying and love the social and behavioural sciences. arxiv preprint arxiv:1712.00547 (2017) morveli-espinoza, m., nieves, j.c., tacla, c.a., jasinski, h.m.: an argumentation-based approach for goal reasoning and explanations generation. j. logic comput. 105, 1–26 (2022) math  google scholar  morveli-espinoza, m., possebom, a.t., tacla, c.a.: argumentation-based agents that explain their decisions. in: 2019 8th brazilian conference on intelligent systems (bracis), pp. 467–472. ieee (2019) google scholar  sarwar, m., ray, r., banerjee, a.: a contrastive plan explanation framework for hybrid system models. acm trans. embed. comput. syst. 22(2), 1–51 (2023) article  google scholar  sreedharan, s., srivastava, s., kambhampati, s.: using state abstractions to compute personalized contrastive explanations for ai agent behavior. artif. intell. 301, 103570 (2021) article  mathscinet  math  google scholar  stange, s., kopp, s.: effects of a social robot’s self-explanations on how humans understand and evaluate its behavior. in: proceedings of the 2020 acm/ieee international conference on human-robot interaction, pp. 619–627 (2020) google scholar  stepin, i., alonso, j.m., catala, a., pereira-fariña, m.: a survey of contrastive and counterfactual explanation generation methods for explainable artificial intelligence. ieee access 9, 11974–12001 (2021) article  google scholar  van bouwel, j., weber, e.: remote causes, bad explanations? j. theor. soc. behav. 32(4), 437–449 (2002) article  google scholar  winikoff, m.: debugging agent programs with “why?” questions (2017) google scholar  wooldridge, m.: reasoning about rational agents. mit press (2003) google scholar  download references author information authors and affiliations graduate program in electrical and computer engineering (cpgei), federal university of technology paraná (utfpr), curitiba, brazil henrique jasinski, mariela morveli-espinoza & cesar augusto tacla authors henrique jasinskiview author publications search author on:pubmed google scholar mariela morveli-espinozaview author publications search author on:pubmed google scholar cesar augusto taclaview author publications search author on:pubmed google scholar corresponding author correspondence to mariela morveli-espinoza . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper jasinski, h., morveli-espinoza, m., tacla, c.a. (2023). towards generating p-contrastive explanations for goal selection in extended-bdi agents. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_23 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords goal selection contrastive explanations bdi agents bbgp agents publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature surveysum: a dataset for summarizing multiple scientific articles into a survey section | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper surveysum: a dataset for summarizing multiple scientific articles into a survey section conference paper first online: 30 january 2025 pp 431–444 cite this conference paper download book pdf download book epub intelligent systems (bracis 2024) surveysum: a dataset for summarizing multiple scientific articles into a survey section download book pdf download book epub leandro carísio fernandes  orcid: orcid.org/0000-0002-4114-23349, gustavo bartz guedes  orcid: orcid.org/0000-0002-3278-996110, thiago soares laitz  orcid: orcid.org/0000-0001-7205-209410,11, thales sales almeida  orcid: orcid.org/0009-0006-9568-933110,11, rodrigo nogueira  orcid: orcid.org/0000-0002-2600-603510,11, roberto lotufo  orcid: orcid.org/0000-0002-5652-085210,12 & … jayr pereira  orcid: orcid.org/0000-0001-5478-438x10,13  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 384 accesses abstract document summarization is a task to shorten texts into concise and informative summaries. this paper introduces a novel dataset designed for summarizing multiple scientific articles into a section of a survey. our contributions are: (1) surveysum, a new dataset addressing the gap in domain-specific summarization tools; (2) two specific pipelines to summarize scientific articles into a section of a survey; and (3) the evaluation of these pipelines using multiple metrics to compare their performance. our results highlight the importance of high-quality retrieval stages and the impact of different configurations on the quality of generated summaries. l. c. fernandes, g. b. guedes, t. s. laitz and t. s. almeida—equal contribution. download conference paper pdf similar content being viewed by others an empirical assessment of citation information in scientific summarization chapter © 2016 deep learning-based scientific document summarization considering citation article 19 march 2025 metadata enrichment of multi-disciplinary digital library: a semantic-based approach chapter © 2018 1 introduction document summarization is a field of artificial intelligence aimed at creating concise and informative summaries from extensive texts [13]. this process allows users to quickly grasp the main information of a document without needing to read the entire original content. summarization can be performed in two main ways: extractive and abstractive [13]. extractive summarization selects and copies key parts of the original text to compose the summary, preserving the verbatim of the source text. on the other hand, abstractive summarization rewrites the content, generating a summary that may include paraphrasing and synthesizing information more creatively and flexibly. multi-document summarization (mds) extends these concepts to handle multiple texts simultaneously [13]. the goal is to generate a single summary that incorporates information from a set of documents, eliminating redundancies and highlighting essential cross-referenced information. this technique is useful in situations where information on a specific topic is spread across various documents. mds datasets advance mds research by providing essential data for training, testing, and validating models. these datasets support the development of algorithms capable of handling the complexities of summarizing multiple documents, enabling researchers to compare different summarization methods effectively [8]. essentially, an mds dataset is composed of summaries generated from multiple source documents. the composition of this dataset can be done in different ways, such as by selecting a set of documents and asking humans to write summaries, or by using existing summaries and their source documents. the latter is the case for the multi-news dataset [4], which uses news articles and their summaries to create an mds dataset. a comprehensive survey is a review that provides a broad overview of a specific topic, covering the most relevant and recent research in the field. these surveys are essential for researchers to understand the state-of-the-art and identify gaps in the literature. the automatic generation of a survey’s text can be considered a mds task, where a set of scientific articles is summarized to produce a cohesive text. this task is challenging due to the complexity of scientific literature and the need to maintain the technical accuracy of the original content. however, most available summarization datasets are designed with a general-purpose approach, covering a broad range of topics and document types. there is a lack of datasets dedicated to the task of summarizing multiple scientific articles to generate a survey’s text, which poses a substantial barrier to advancements in this specialized area of research. this type of dataset assists the development and benchmarking of summarization models focusing on the generation of scientific surveys. this paper introduces surveysumfootnote 1, a dataset designed for the task of summarizing multiple scientific articles into survey sections. our contribution includes: (1) the surveysum dataset and the methodology to create it, which aims to address the lack of in domain-specific summarization tools; (2) two specific pipelines to summarize scientific articles into a section of a survey; and (3) evaluation of these pipelines in respect to consistency, quality, and relevance of the generated summaries. 2 related work multi-document summarization (mds) addresses the problem of extracting information that is spread across multiple documents, making it more challenging than single-document summarization. it is still an evolving field, with no single approach to solve this problem. the development of this field relies on datasets that serve as benchmarks for evaluating and comparing different summarization methods. the scientific literature includes some examples of such datasets, each varying in domain, structure, size, and summarization objective. these datasets are important for advancing research and improving summarization models. outside the scientific domain, multi-news [4] presents a large volume of news articles and summaries, focusing on the abstractive summarization of multiple documents in the journalistic context. also in this context, ghalandari et al. [5] proposed a large-scale dataset for multi-document summarization that contains concise human-written summaries of news events. in the legal field, multi-lexsum tackles the summarization of legal cases, presenting a set of civil rights litigation summaries with multiple granularities. the summarization of instructional content is the focus of the howsumm dataset, derived from wikihow articles [1]. the findsum dataset focuses on the challenge of summarizing long text and multiple tables [10]. in the scientific domain, the need to manage the vast amount of literature has generated interest in datasets specific to the summarization of scientific articles. multi-xscience [12] is a dataset focused on the generation of “related work” sections by summarizing multiple scientific articles. bigsurvey [9] was designed to create structured summaries of academic articles, focusing on consolidating literature reviews. sumpubmed [6] and scisummnet [19] aim the summarization of single scientific articles. several other datasets can be used for multi-document summarization. koh et al. [8] work is an extensive survey on long document summarization, including ten other mds datasets. however, to the best of our knowledge, there is no dataset designed to summarize a set of scientific articles in order to generate a section of a scientific survey. the closest dataset, multi-xscience [12], focuses on writing the “related work” section of a paper based on the abstract of the paper and the abstract of the articles it references. it does not consider the text of the referenced articles. our dataset, on the other hand, adresses this gap: it considers the full text of the referenced articles to write a specific section on a subject. consequently, our proposed dataset is truly a multi-document summarization dataset, capable of handling lengthy documents. 3 method in this section, we describe the methodology to create surveysum. first, we describe our approach in selecting existing surveys from the scientific literature. later, we detail the text and citations extraction process. 3.1 survey selection the selection of surveys was based on the authors’ expertise in the field of artificial intelligence, natural language processing, and machine learning. the surveys were chosen to cover a broad range of topics within these areas, ensuring the dataset’s diversity and applicability. for surveys to be eligible for inclusion in the dataset, they had to attend to the following criteria: (1) be a comprehensive survey in the field of artificial intelligence, natural language processing, or machine learning; (2) be divided into sections, each with citations of scientific articles; (3) be freely available online; and (4) be written in english. first, we conducted a search on google scholar using the keywords comprehensive, survey, artificial intelligence, natural language processing, and machine learning. we then filtered the results to include only surveys published in the last five years and that are freely available. the surveys were manually reviewed to ensure they met the eligibility criteria, resulting in the selection of six surveys. usually, surveys are structured into sections to organize the content and facilitate reading and understanding. this makes each section a self-contained summary of a specific aspect of the survey, while the cited papers are the source documents. thus, each section is a summary of the cited papers, making it an ideal target for the mds task. finally, surveysum samples are comprised by the sections texts of surveys with the corresponding citation papers. this ensures that the dataset can be used to evaluate the summarization of specific topic areas within the surveys. 3.2 dataset creation in this section, we describe the methodology to create the dataset, illustrated in fig. 1. this process is applied to all the surveys selected in the method described in the previous section. fig. 1. dataset creation process. full size image initially, we parsed each survey with s2orc-doc2jsonfootnote 2, extracting the sections along with the corresponding citations. s2orc-doc2json converts scientific articles from various formats, such as pdfs and latex, to javascript object notation (json) format. additionally, s2orc-doc2json extracts the citations from scientific papers, indicating both the position of the citation within the text and the referenced paper. the next step is to retrieve the full text of the cited papers. for this, we conducted searches, utilizing available apis (e.g., arxiv search api), across online repositories including arxivfootnote 3 and semantic scholarfootnote 4 supplemented by a crawler for google scholarfootnote 5. finally, we manually verified each extracted citation against the original surveys. this step was necessary because s2orc-doc2json is a machine learning based process, thus susceptible to inconsistencies. also, some corresponding papers were not retrieved using the apis or the crawler, so a manual search and retrieval was necessary. using this approach, the surveysum was created. data was extracted from 79 sections across 6 surveys. each section is related to an average of 7.38 articles, with a standard deviation of 4.95 articles (median is 7 articles). thus, the task is to generate a survey’s section text given its title and a collection of papers. 4 pipelines 4.1 general pipeline the text generation of a survey’s section is a multi-document summarization problem. starting from an input topic, the goal is to find papers related to the topic and summarize them into a single cohesive, coherent, and technically correct text. we present a general pipeline, divided in three stage, to automatically generate a survey’s section text from reference papers, as shown in fig. 2 and detailed as follows: 1. stage 1: definition of the title of the survey, the title of the sections, and related papers; 2. stage 2: split the relevant papers texts into chunks and use a search algorithm to retrieve the relevant chunks according to the title of the survey’s section; 3. stage 3: generate the text for the section with a large language model (llm). fig. 2. three stages to generate a section of a survey. full size image the first stage can be done manually, with the researcher choosing a topic/title for the subject and providing a set of pre-selected articles. or, it can be automated: starting from a topic, one can use a scientific article search engine (e.g. semantic scholar [18]) to find related papers and then refine the results. the effectiveness of the next stages of the pipeline relies on the quality of the retrieved papers. if the selection is manual, it is assumed that the researcher is aware of which papers are relevant. if the selection is automatic, the quality of the result will be closely tied to the retriever. the second stage receives the data from the previous stage and selects excerpts from the texts to be consolidated into a single text. this typically includes splitting the text into smaller units, called chunks, storing them in a database, and retrieving the most relevant ones. creating the chunks can be done with different strategies. the text can be split by characters, sentences, or paragraphs. it can involve some degree of overlap or not. the database where the chunks will be stored can be either textual or vector-based. if it’s a textual database, the chunks can be enriched using some method (e.g. doc2query [15]). as for a vector database, it’s necessary to choose a model to generate the chunk embeddings. then, for retrieval, one can use a variety of methods – for example, a lexical search using bm25 [17] followed by one or more reranking steps. finally, the third stage takes as input the titles of the survey, the section (from stage 1) and the relevant chunks (from stage 2) to summarize using a llm. to achieve this, it’s essential to select the modelfootnote 6 that will be used. 4.2 pipeline 1 in this pipeline, it is assumed that the subject and title of the section are given to the system, chosen by the researcher. the researcher has the option to provide the papers for each section or let the system search online for relevant papers, in which case the selected articles are downloaded in pdf format and then have their text extracted. in the second stage, the text from each article are segmented in chunks of 150 words, with an overlap of 50%. each chunk is then judged by a neural ranker model, monot5-3b, which demonstrated a strong zero-shot performance in numerous works [14]. with the score monot5-3b gives to each chunk, we select the top n chunks and provide them to the llm. figure 3 shows the prompt template used for generating the content of a section. fig. 3. prompt used to write the section of the survey in pipeline 1. full size image 4.3 pipeline 2 in this pipeline, the task is to generate the text for a survey section given the corresponding cited papers associated with it in surveysum. in the second stage of the general pipeline, each paper’s contents are segmented into chunks of 7 sentences with an overlap of 2 sentences to preserve the surrounding context. then, the embeddings for each chunk are generated using the specter2 model [3] and stored in a faiss vector database [7]. specter2 was trained to generate embeddings for scientific articles considering the title of the paper and its abstract. in this pipeline, each chunk was encoded considering the title and the chunk. the retriever selects the 20 most relevant chunks using the section title as the query term. each chunk is sent to gpt [2], which provides a score between 0 and 5 using the prompt shown in fig. 4. fig. 4. prompt used to rerank the chunks in pipeline 2. full size image fig. 5. prompt used to write the section of the survey in pipeline 2. full size image after this step, all chunks that received a score of 0 are discarded. the remaining ones are reordered by the score, and only the top n best-scored are selected and sent to the third stage. these chunks are inserted into the prompt shown in fig. 5 and sent to gpt to generate the section text. 4.4 evaluation metrics references f1 score when using llms, particularly in the context of retrieval-augmented generation (rag) systems, it’s important to assess the quality of the generated content and the model’s effectiveness in using the provided references to generate text. in scientific literature, it is common for each claim to be supported by multiple references. therefore, one approach is to request the model to produce a summary along with the references it used. this enables the calculation of the f1 score for the references used by the model against the references provided in the input, serving as the ground truth. this method allows for an evaluation of which model or prompt more effectively uses the given information. g-eval g-eval is a metric for evaluation content generated by llms [11]. it measures the quality by comparing the generated text to a reference text. the final score is calculated by sampling a list of scores provided by an llm with decoding temperature set to 1, top-p sampling set to 1 and number of samples set to 20. it is defined as follows: $$\begin{aligned} score = \frac{1}{n} \sum _{i=1}^{n} p(s_i) \times s_i \end{aligned}$$ (1) where \(n\) is the number of samples, \(p(s_i)\) is the probability of the \(i\)-th sample, and \(s_i\) is the score of the \(i\)-th sample. the score is generated using the prompt shown in fig. 6. fig. 6. prompt used for g-eval. full size image check-eval we also used the check-eval metric to evaluate the generated text. this metric is based on the methodology proposed by pereira et al. [16]. the check-eval metric evaluates the consistency between the candidate text and the reference text using an evaluation checklist. the checklist is generated by a llm model, which is then used to evaluate the candidate text. the presence of the checklist elements in the candidate text is quantified to determine its information quality. the metric calculates a score reflecting the consistency between the candidate and reference text. for example, if the evaluation checklist consists of ten elements and the candidate text includes eight of these, the check-eval score will be 0.8, i.e., the metric suggests that the candidate text maintains an 80% consistency level with the reference text. 5 results this section presents the results of generating the texts of each section of surveysum using the two pipelines described in sect. 4 with different setups. pipeline 1 was executed using the monot5-3bfootnote 7 model to retrieve the chunks and the gpt-3.5-turbo-0125 model to generate the final text. we have evaluated different setups for this pipeline as follows: 1.1: section generation with 5 chunks, with the corresponding cited articles from surveysum; 1.2: section generation with 10 chunks, with the corresponding cited articles from surveysum; 1.3: section generation with 10 chunks, with articles retrieved from the semantic scholar api. pipeline 2 was executed with six different setups after the chunks retriever stage. after selecting the top 20 chunks, a gpt model was used to rerank the chunks and to write the section. each configuration differs in the gpt model used and the number of chunks considered to summarize the documents. in all of them, only the articles from each section of the dataset were considered as input for the models: 2.1: gpt-3.5-turbo-0125 model. section written with a single chunk; 2.2: gpt-3.5-turbo-0125 model. section written with up to 5 chunks; 2.3: gpt-3.5-turbo-0125 model. section written with up to 10 chunks; 2.4: gpt-4-0125-preview model. section written with a single chunk; 2.5: gpt-4-0125-preview model. section written with up to 5 chunks; 2.6: gpt-4-0125-preview model. section written with up to 10 chunks; table 1 shows the metrics for nine configurations evaluated. regarding the metrics used (g-eval and check-eval), both metrics generally agreed with each other. however, unlike the evaluation checklist, g-eval tends to concentrate the results around certain numbers. table 1. results of pipelines. all checklist evaluations were performed using gpt-3-5-turbo and all g-eval scores were generated by gpt-4.full size table pipeline 1.3 had the worst results in both check-eval and g-eval scores among all pipelines. this was the only configuration tested that used articles retrieved from a search engine instead of articles from surveysum. this result shows a strong dependence of text summarization on the retriever stage. for pipelines of the same type ([1.1 and 1.2], [2.1, 2.2, and 2.3], [2.4, 2.5, and 2.6]), the more chunks used, the higher the f1 score tends to be, as there is a higher probability of selecting chunks from different articles. however, it cannot be inferred that increasing the number of chunks will necessarily improve text quality. although the experiments indicated this average behavior for this number of chunks, there are samples where increasing the number of chunks worsened the result. the quality of the chunk—directly influenced by the retriever stage—is very important. regardless of the metric considered in table 1, the results for pipelines 2.4, 2.5, and 2.6 (using the gpt-4-0125-preview model) were better than the results for pipelines 2.1, 2.2, and 2.3 (equivalent pipelines using gpt-3.5-turbo-0125). this suggests that the llm used is relevant to the quality of result. the g-eval and check-eval scores are high correlated (76%), indicating consistency in their evaluation of summary quality. however, g-eval tends to produce results with lower variability compared to the check-eval score. the check-eval score, being more granular and item-specific, captures a broader range of evaluation aspects. these insights underscore the importance of employing multiple evaluation metrics to obtain a comprehensive understanding of the performance of summarization models. 6 conclusion in this paper, we introduced surveysum, a dataset designed for summarizing sections within scientific surveys in artificial intelligence, natural language processing, and machine learning. our contributions include the creation of the dataset, two pipelines to summarize scientific articles, and their evaluation using various metrics. as with various other summarization challenges, the summarization of articles for the generation of a survey section does not yield a single definitive answer. different researchers may use the same set of papers to produce a section on a given topic, each covering different aspects of that topic. nevertheless, it is important to establish parameters for comparison. surveysum addresses a gap in document summarization by providing a dataset that assists the development and benchmarking of models focused on scientific surveys. our results highlight the dependence on the quality of the retriever, as the effectiveness of the summarization pipelines was influenced by the accuracy and relevance of the documents retrieved during the initial stages. high-quality retrieval may ensure that the most relevant information was available for summarization. notes 1.https://huggingface.co/datasets/unicamp-dl/surveysum. 2.https://github.com/allenai/s2orc-doc2json. 3.https://info.arxiv.org/help/api/index.html. 4.https://www.semanticscholar.org/product/api. 5.https://scholar.google.com.br. 6.at this stage, there are no restrictions on which llm to use, but for the purposes of this paper, we experimented only with those from openai. 7.https://huggingface.co/castorini/monot5-3b-msmarco. references boni, o., feigenblat, g., lev, g., shmueli-scheuer, m., sznajder, b., konopnicki, d.: howsumm: a multi-document summarization dataset derived from wikihow articles (2021) google scholar  brown, t.b., et al.: language models are few-shot learners (2020) google scholar  cohan, a., feldman, s., beltagy, i., downey, d., weld, d.s.: specter: document-level representation learning using citation-informed transformers. in: acl (2020) google scholar  fabbri, a.r., li, i., she, t., li, s., radevm, d.r.: multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model (2019) google scholar  ghalandari, d.g., hokamp, c., pham, n.t., glover, j., ifrim, g.: a large-scale multi-document summarization dataset from the wikipedia current events portal (2020) google scholar  gupta, v., bharti, p., nokhiz, p., karnick, h.: sumpubmed: summarization dataset of pubmed scientific articles. in: kabbara, j., lin, h., paullada, a., vamvas, j. (eds.) proceedings of the 59th annual meeting of the association for computational linguistics and the 11th international joint conference on natural language processing: student research workshop, online, august 2021, pp. 292–303. association for computational linguistics (2021) google scholar  johnson, j., douze, m., jégou, h.: billion-scale similarity search with gpus. ieee trans. big data 7(3), 535–547 (2019) article  google scholar  koh, h.y., ju, j., liu, m., pan, s.: an empirical survey on long document summarization: datasets, models, and metrics. acm comput. surv. 55(8), 1–35 (2022) article  math  google scholar  liu, s., cao, j., yang, r., wen, z.: generating a structured summary of numerous academic papers: dataset and method. in: proceedings of the thirty-first international joint conference on artificial intelligence, ijcai-2022. international joint conferences on artificial intelligence organization, july 2022 google scholar  liu, s., cao, j., yang, r., wen, z.: long text and multi-table summarization: dataset and method (2023) google scholar  liu, y., iter, d., xu, y., wang, s., xu, r., zhu, c.: g-eval: nlg evaluation using gpt-4 with better human alignment. in: conference on empirical methods in natural language processing (2023) google scholar  yao, l., dong, y., charlin, l.: multi-xscience: a large-scale dataset for extreme multi-document summarization of scientific articles (2020) google scholar  nenkova, a., mckeown, k.: automatic summarization. found. trendső inf. retr. 5(2–3), 103–233 (2011) article  math  google scholar  nogueira, r., jiang, z., pradeep, r., lin, j.: document ranking with a pretrained sequence-to-sequence model. in: cohn, t., he, y., liu, y. (eds.) findings of the association for computational linguistics: emnlp 2020, pp. 708–718. association for computational linguistics, online, november 2020 google scholar  nogueira, r., yang, w., lin, j., cho, k.: document expansion by query prediction (2019) google scholar  pereira, j., lotufo, r.: check-eval: a checklist-based approach for evaluating text quality (2024) google scholar  robertson, s.e., walker, s., jones, s., hancock-beaulieu, m., gatford, m.: okapi at trec-3. in: harman, d.k. (ed.) proceedings of the third text retrieval conference, trec 1994, gaithersburg, maryland, usa, 2–4 november 1994, volume 500-225 of nist special publication, pp. 109–126. national institute of standards and technology (nist) (1994) google scholar  semantic scholar. https://www.semanticscholar.org/ yasunaga, m., et al.: scisummnet: a large annotated corpus and content-impact models for scientific paper summarization with citation networks. in: proceedings of aaai 2019 (2019) google scholar  download references author information authors and affiliations instituto de pesquisa para economia digital, brasília, df, brazil leandro carísio fernandes unicamp, campinas, sp, brazil gustavo bartz guedes, thiago soares laitz, thales sales almeida, rodrigo nogueira, roberto lotufo & jayr pereira maritaca ai, campinas, sp, brazil thiago soares laitz, thales sales almeida & rodrigo nogueira neuralmind.ai, campinas, sp, brazil roberto lotufo universidade federal do cariri, juazeiro do norte, ce, brazil jayr pereira authors leandro carísio fernandesview author publications search author on:pubmed google scholar gustavo bartz guedesview author publications search author on:pubmed google scholar thiago soares laitzview author publications search author on:pubmed google scholar thales sales almeidaview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar roberto lotufoview author publications search author on:pubmed google scholar jayr pereiraview author publications search author on:pubmed google scholar corresponding author correspondence to leandro carísio fernandes . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper fernandes, l.c. et al. (2025). surveysum: a dataset for summarizing multiple scientific articles into a survey section. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_30 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords dataset multi-document summarization document summarization text summarization scientific writing publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature investigating universal adversarial attacks against transformers-based automatic essay scoring systems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper investigating universal adversarial attacks against transformers-based automatic essay scoring systems conference paper first online: 30 january 2025 pp 169–183 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) investigating universal adversarial attacks against transformers-based automatic essay scoring systems download book pdf download book epub igor cataneo silveira  orcid: orcid.org/0000-0003-1599-83479, andré barbosa  orcid: orcid.org/0009-0005-9526-28409, daniel silva lopes da costa  orcid: orcid.org/0009-0008-8358-131x9 & … denis deratani mauá  orcid: orcid.org/0000-0003-2297-63499  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 387 accesses 3 citations abstract automatic essay scoring promises to scale up student feedback on written input, addressing the excessive cost and time demand associated with human grading. state-of-the-art automatic scorers are based on transformers-based neural networks. while such models have shown impressive results in reasoning tasks, learned models often produce answers that arise from statistical clues in datasets and are misaligned with human objectives. such systems are thus potentially fragile for scenarios where users are incentivized to deceive the system, as in a classroom setting. in this work, we evaluate the susceptibility of state-of-the-art automatic scorers to attacks made by non-expert users, such as students interacting with an automatic grader. we develop a methodology to simulate such student attacks and test them against scorers based on bert, phi-3 and gemini models. our findings suggest that (i) a bert-based grader can be deceived using simple feature-based attacks; (ii) although google’s gemini has a solid agreement with graders, it can assign undeservedly high grades for small sentences; (iii) a phi-3-based grader was less susceptible than bert, but it still assigned relatively high grades to some of our attacks. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others cheating automatic short answer grading with the adversarial usage of adjectives and adverbs article open access 26 july 2023 automated grading of open-ended questions in higher education using genai models article open access 19 october 2025 automatic grading system using sentence-bert network chapter © 2020 1 introduction automatic essay scoring (aes) systems ease the burden on teachers while enhancing student learning by providing meaningful, timely and personalized feedback [32]. the technology is currently mature enough to be employed in high-stakes standardized exams such as the gre and toefl [3, 5]. like many machine learning based solutions, aes systems are prone to malicious usage and are sensitive to spurious correlations in the training data [22]. this is particularly relevant in educational and scoring settings, where users might be motivated to exploit the system’s vulnerability, and where a correct prediction for the wrong reason might be as harmful as an incorrect prediction. unintended uses of predictive systems are known in the machine learning literature as adversarial attacks [21]. the more recent literature cares specifically about attacks that are carried out by changing an existing input in a human imperceptible way and such that the prediction shifts towards or away from some desired target value [13, 44]. for example, in aes that might amount to determining words or expressions in an essay that, if changed, maintain the text’s overall characteristics as perceived by a human while leading the predictive model to raise the score significantly. such attacks usually require knowledge and access to the system’s characteristics. thus, they are usually difficult to be performed by regular users of aes systems. a more straightforward approach is the universal attack, which consists of finding input-unrelated rules that cause the predictive model to increase scores. for example, it has been noticed that essay length is a good predictor of essay quality [26]. a malicious user can exploit this fact by appending unrelated text to an essay to increase its score. such an attack does not require deep knowledge of the system and can be formulated by repeated use. designers of aes systems have long been aware of the pitfalls of malicious usage. for instance, authors have discussed whether to remove certain predictive features such as text length in order to improve the system’s robustness against attacks [42]. the matter is complicated by the fact that textual features are often correlated, so that removing a feature might hurt performance without actually making the system less susceptible to attacks. following the trend in natural language processing, state-of-the-art aes systems are based on transformer deep neural networks [27]. while in principle such models can learn rich representations and enable logical and commonsense reasoning, there is evidence that learning algorithms and spurious correlations in data often lead to predictions being based on simple statistical cues [43]; hence such systems might be as exposed to attacks as traditional feature-based systems. in this work, we investigate whether transformers-based aes systems are susceptible to universal adversarial attacks that might occur in a classroom setting. in particular, we consider a fictitious (but realistic) case where users (i.e., students) interact with the system by submitting an essay (input) and receiving a score (the output), while having no knowledge of the predictive model being used (including parameters, derivatives, etc.). the hypothetical system is non-adaptive (i.e., no learning after deployment) and students use the system repeatedly (e.g. grading different essays along the academic term or being able to submit an improved version) and share information about usage among themselves. given that cheating in the classroom is prevalent [20, 41], some students are expected to exploit any vulnerability to improve their grade without necessarily improving the quality of their texts. to simulate how such non-expert users might learn vulnerabilities, we train an interpretable predictive model (a linear regressor) based on handcrafted features and select the most predictive features to compose attacks, such as inserting lists of adjectives of adverbs, employing more adjectives and adverbs than usual, and repeating the same sentence multiple times. the rationale is that by repeated interaction and information sharing, users might develop a simplified model of the system’s inner work and use such a model to come up with such simple attacking strategies. the effectiveness of the attacks is evaluated with respect to bert, phi-3 and gemini-based aes systems on a benchmark of human annotated (brazilian) portuguese essays [34]. we show that all models are vulnerable to simple attacks.footnote 1 for example, on a scale of 0 to 1000, where higher numbers are better, the bert and phi-3 based models assigned, respectively, 800 and 560 points to a universal attack (i.e., a fabricated essay) based on repeating a generic conclusion employing many adverbs and adjectives. the gemini-based model showed resilience against attacks based on repetition, but for instance assigned a relatively high score of 640 points to an essay consisting of only a small list of adjectives. in the rest of the document, we present related work in sect. 2, discuss our methodology in sect. 3, and show the experimental results in sect. 4. we conclude with a summary and final remarks in sect. 5. 2 related work the evolution of aes systems mimics that of other nlp tasks. the initial systems were based on handcrafted features, which had limited predictive power and require extensive expertise to be designed [32]. those systems were incrementally superseded by systems that learned representations directly from data. first, by approaches that extracted shallow representations such as word embeddings [37]. more recently, by approaches that learn representation by deep models such as transformer neural networks [27]. several aes systems for (brazilian) portuguese have been proposed, mostly focusing on grading essays in the format required for the brazilian national entrance exam (enem) [2, 4, 12, 15, 28, 34]. an early work used words as features of a naïve bayes [4]. subsequent approaches ranged from defining handcrafted features to combining word-embedding with recurrent neural networks [2, 15, 29]. our study intersects with the existing literature by evaluating the performance of previously unused models, specifically the phi-3 and gemini-based graders. our goal is not pursuing high agreement, but presenting how susceptible these novel models are to adversarial attacks. a recent work compared several (brazilian) bert-based graders [34]. we incorporate these models into our study, alongside new state-of-the-art models, for comparative analysis. different aes systems were assessed based not on their qwk performance, but rather on their oversensibility and overstability [21]. the authors investigated how perturbations in the input affected the models’ output. the asap challenge dataset [18] was used to test five models, ranging from a feature-based to a bert-based grader. they found that, except for the feature-based and one of the word-embedding-based, all models were overstable. moreover, increasing the disturbance percentage lead to increased standard deviation in the grades. their work investigated whether adding, removing or changing content of an essay caused it to be graded differently. this can be seen as students rewriting their essays after grading. our scope is different, for they do not consider universal attacks, and we consider ill-intended attacks that could be done by students with minimal effort. without any scenario restrictions, there are possibly infinite adversarial attack setups. we rule out possible attacks by restricting ourselves to a very specific scenario. firstly, students do not have access to any parts of the model, so they are not allowed to attack the embedding space [16], the gradient [11], nor the calculated features, to name a few. secondly, they do not know the underlying model, which prevents them from using the pre-trained version of the model to generate attacks, such as [25]. lastly, the students are not too invested in breaching the system, so attacks such as [14, 40] are not applicable. the first requires (possibly) testing combinations of the whole vocabulary used by the tokenizer of the underlying model. this is ruled out of our scenario as they do not have access to this information, some models do not have a vocabulary, and gemini’s vocabulary has 256k tokens, so it would require at least this many attacks. similarly, the second attack requires changing the whole essay to have the same word, which results in a massive number of needed attacks. thus, the attack we devise here is much closer to what one could expect to happen at a classroom level and has not been used yet in the literature. the task of automatic short answer grading (asag) is very similar to aes, the difference is that, instead of (long) essays, its input is a relatively small text. adjectives and adverbs have already shown to be useful to cheat asag systems based on transformers, such as bert and t5 [13]. the scenario of their study is similar to ours. the differences are: they are concerned with generating answers that cannot be identified by humans, and that their labels are: correct, incorrect or contradictory. by inserting adjectives and adverbs in the sentences, they could fool the models to change the answer from incorrect to correct between 8 and 22% of the cases. showing that they are strong candidates to fool the model. additionally, their investigation found that humans did not find the adversarial examples suspicious, but they were not natural either. to evaluate the robustness of available models against such adversarial attacks, we rely on the lmsys benchmark [9] to choose candidates. lmsys is a comprehensive benchmarking platform designed to evaluate the performance of large language models across a wide array of tasks, including natural language understanding, generation, and interaction. by providing standardized evaluation metrics and diverse benchmarks, it enables a consistent comparison of model capabilities, thus serving as an invaluable resource for researchers and developers. as of the current date, the 10 best-ranked models available there are proprietary and cannot be easily fine-tuned due to their restricted access. among the available ones, nemotron [31] and the top-performing models from the llama family [39] are notably expensive to run, limiting their practicality for widespread use. in contrast to this trend, there is a growing interest in “small" language models, such as llama3-8b and the phi-family [17]. the phi-family has recently released the phi-3 models [1], which include three distinct model sizes: 3.8b (phi-3-mini), 7b (phi-3-small), and 14b (phi-3-medium). these models, despite their smaller size, are claimed by the authors to outperform some proprietary models, such as gpt-3.5, in various tasks. we chose the phi-3 model for our research due to its open-source nature, which allows for easier access and adaptability. additionally, the phi-3 model was trained exclusively on english data and has not been extensively tested in other languages. our study aims to evaluate the performance of the phi-3 model using portuguese-only data, contributing to the understanding of its capabilities and limitations in multilingual contexts. the gpt family (davinci, 3.5 turbo and 4.0) have been tested in aes and asag. the authors of [30] showed that the davinci model could achieve 0.38 qwk in the toefl dataset, while linguistic features could achieve almost 0.6 qwk. gpt 3.5 and 4.0 were used to grade short answers in finnish [7]. the conclusion was that the models could not be directly employed and that they are more lenient than human graders, assigning fewer failing grades. while openai models are the most famous, their cost is prohibitive, thus we explore here the usage of gemini’s free api and how resilient it is against adversarial attacks. 3 methodology we start by selecting the enem-aes dataset and pre-trained models [34], which are easily accessible through hugging face api. essays in this dataset are evaluated according to five difference criteria, or competencies, and separated predictors have been trained for each specific criterion. we use the bert-base ordinal model available for each competence, as they are relatively small and achieve high performances. additionally, we used the splits available using the “propor2024” parameter. we also used the scikit-learn package [33] to train, for each individual competence, a linear regressor that uses 72 features calculated by the open api of nilc-metrix [24]. these features are interesting because they calculate various linguistic levels of written language, such as the ratio of a certain part of speech, how ambiguous the words are, text length, flesch score and others. although they might be far from what the enem guiding book states that are being evaluated, they might relate closer to what the neural models are evaluating. for this grader, as we did not optimize any hyperparameters, we joined the training and validation split to increase the number of training samples. as this model outputs a real number and the enem grades are in steps of 40, we round it to the closest allowed grade. for fine-tuning a large language model, we chose to fine-tune a phi-3-medium model, balancing model size with our available hardware. although smaller than some other models, it still demands significant computational power for training. we had access to an nvidia gpu rtx a6000, and we applied several optimizations for fine-tuning [35], including gradient checkpointing, flash attention v2 [10], and lora [19]. the available model is instruction fine-tuned.footnote 2 during our tests, we found it necessary to pass an instruction to the system role before prompting a given essay. the prompt structure is shown in fig. 1. fig. 1. prompt-structure used for each concept fine-tuning through phi-3. full size image in this structure, concept_system is a placeholder for the competence being fine-tuned (e.g., enem competences that vary from 1 to 5), and essay_example represents the essay we are interested in fine-tuning. to define the concepts, we copied the definitions directly from the enem 2023 student’s reference manual, which provides clear instructions on how each grade is applied. following these optimizations, we added a sequence classification head to the existing model and fine-tuned it according to the hyperparameters outlined in table 1. it is noteworthy that these models were instruction fine-tuned through supervised fine-tuning and direct preference optimization for english conversations. fine-tuning capabilities for languages other than english were not investigated prior to the completion of this study [1]. we found it interesting to notice an ability of the final phi-3 to generalize and outperform models that were trained solely on portuguese data—the prior model based on bertimbau [36]. table 1. phi-3 fine tuning hyperparametersfull size table finally, we employ google’s gemini chatbot. we chose gemini over chatgpt because gemini offers a free api. the current model available is the gemini 1.5 flash [38]. its size and training set are not publicly disclosed, so we can only speculate about its dimensions. according to the lmsys leaderboard [9], it is ranked near the llama3 70b model, suggesting it has at least 70 billion parameters. we do not train this model; we use it exclusively through its api. additionally, we did not employ any prompt engineering techniques, instead applying a zero-shot learning process [6, 8, 23], as we only asked it to grade the essays using the enem criteria. however, the model produces grades that do not follow the 40-point step, so we round the numbers according to the same criteria as the linear regressor. using these graders, we test them—except for the bert one, as its performance has already been published in [34]—and compare them against the baselines. then, we use the weights of the linear regression to identify the relevant features. finally, we use them to create possible universal adversarial attacks that students could create. there are no limits to the number of adversarial attacks that can be created. in our investigation, we restricted ourselves to a scenario where (1) the students do not know the underlying aes system; (2) they do not have access to any part of the system except for an interface where they can type the essay and see their final score; (3) students can share their intuitions with each other; (4) they do not want to properly write the essay, but still they want the best possible grade. having these restrictions, we devise some attacks and test them against all graders. 4 experiments we now discuss the empirical findings of our investigation. 4.1 testing the new models the first step before devising our adversarial attacks is comparing the performance of an interpretable model against the neural ones. we train the models presented in the previous section, namely a linear regressor (lr), a phi-3 and a gemini-based grader. their performances, measured in quadratic weighted kappa (qwk), are presented in table 2 and compared against the berts available and presented in [34]. table 2. performance comparison of different automatic graders, as measured by qwk.full size table the first thing to notice is that their sizes have different orders of magnitude, but the performances are still not that distant. the linear regressor was expected to have the worst performance across all competences, but it happened only in two out of the five. when it had the lowest performance, it was only by a margin of 0.07 and 0.04 compared to the bert baseline. the similar performance to the others may point out that the neural models are exploring simple features of the essays. the phi-3 model performs best in three competences, with a surprising 0.61 in the fifth competence. notably, this model loses twice to the linear regression model, but it is always at least competitive with the bert baseline. finally, gemini is the largest model, with an unknown size and training set. it drew with the linear regressor for the best performance in competence 2 and had the worst performance in competence 3. in general, this model is the most stable one, with performances between 0.41 and 0.35. its relatively poor performance must be seen with a grain of salt, as it is the only model that was not fine-tuned to actually grade the essays. having shown that the performances between the different transformers-based graders are not that different from the linear regressor, we check which features receive the largest weights. the names of the five more important (positive) features are presented in table 3. although named with a different pattern, adverbs and verbs are features that measure the ratio of such parts of speech. content words and function words also account for the ratio of words. the former aggregates nouns, verbs, adjectives, adverbs, and discourse markers. the latter accounts for articles, conjunctions, interjections, numerals, pronouns, prepositions, conjunctive and subordinating adverbs. related to the previous two, content density is the ratio of content words to functional words. finally, cau neg conn ratio is the ratio of negative causal connectives. competences 1 to 3 have the same set in the same order, which does not match what is expected from the official grading, as each competence is supposed to evaluate a different aspect of the text. competence 5 has the same set as the previous three but in a different order. competence 4 is supposed to evaluate the proper usage of connectives, so having function words and cau neg conn ratio between the top 5 makes sense. table 3. the four most important features for each competence according to the linear regressor.full size table with this information, we can assume that students will come up with intuitions like (1) it is important to have a lot of adverbs, as it is the most important feature in 3 competencies, it is part of the top feature in competence 4 and is the third most important in competence 5; (2) it is important to have a lot of adjectives, as it is the most important in one competence, the second most important in three, and part of the top feature in the other. 4.2 defining the (universal) adversarial attacks bearing the previous intuitions in mind, we can devise three goals: increase the number of adverbs, increase the number of adjectives, and increase both. we consider that although the feature is the ratio of such classes, it is not trivial to identify that the importance is the ratio and not the total number. to increase the number of one of the classes, we can consider some strategies, such as (a) writing a list of words belonging to this class, (b) copying and pasting the previous list so that it becomes similar to the four paragraphs structure; (c) similar to the previous, but with more copying and pasting and using a memorized sentence that uses many of the desired part of speech. we have 3 goals, and each goal can be reached by 3 strategies, so we can define 9 adversarial attacks: attack 1a:: we devise a list of adverbs, such as: “well, badly, enormously, smally, certainly, wrongly, rapidly, slowly, fairly, unfairly". attack 1b:: we repeat the previous list four times, one in each paragraph. attack 1c:: we create a sentence that uses more adverbs than usual, such as: “undeniably, progressing slowly, leisurely, carefully, silently while deeply breathing and thinking intensively about the given problem", copy it 10 times in each paragraph, in 4 paragraphs. attack 2a:: we devise a list of adjectives, such as: “good, bad, big, small, best, worst, right, wrong, last, first, fair, unfair". attack 2b:: we repeat the previous list four times, one in each paragraph. attack 2c:: we create a sentence that uses more adjectives than usual, such as: “the constant and innovative development of modern technology has brought significant and deep changes to this issue, creating diverse and thrilling opportunities for a more promising and sustainable future." then we copy it 10 times in each paragraph, in 4 paragraphs. attack 3a:: we concatenate the sentences of 1a and 2a. attack 3b:: we repeat the previous list four times, one in each paragraph. attack 3c:: we create a sentence that uses more adjectives and adverbs than usual, such as: “consequently and undeniably, the constant innovative development slowly and progressively brought some significant and deep changes to constantly needed problems", then we copy it 10 times in each paragraph, in 4 paragraphs. 4.3 testing the attacks we now use the sentences created for each attack and submit them to the models. the grades assigned by each model are displayed in table 4. schools in brazil usually require students to get at least 6 or 7 to be approved, so we can assume that an adversarial student would be happy with getting a 600 or higher with the lowest possible effort, and this will count as fooling the model. table 4. grades assigned by all models, in all competences, to each attack.full size table we can see that our simplest attacks, the ones with suffix a, are sufficient to fool a simple linear regressor on all but the competence 5. this is not surprising, as the attacks were defined according to most correlated predictor variables. changing from one sentence to four paragraphs (i.e., changing an attack from type a to type b) did not increase the score, while changing from type b to type c did improve the score. we note that either changing from type a to type b or from type b to c leads to a significant increase of the text length. the number of words and paragraphs are features available to the model and have a positive coefficient, so increasing them naturally increases the grade. additionally, as this linear regressor has a limit of 2k words, it is easy to get a full grade by increasing certain word frequencies. more interestingly, the bert grader was not completely fooled by the simplest attacks; it assigned attacks 1a and 2a with 360 points, and attack 3a with 400 points. this suggests that bert-based predictors are either sensitive to diversity in parts of speech or that their inner representation considers word frequencies. changing from attack type a to b increased scores, which could point to the fact that bert is sensitive to the four paragraph structure—this could be further emphasized by competence 3’s scores going up. but since bert does not have a token for line break, it is not clear whether that effect is due to the presence of longer sentences or due to the increased word frequency. finally, we see that going from attacks of type b to c resulted in an increase of 200 points. this was the only attack that received a score above zero in competence 5 with this model. gemini, on the other hand, seems to recognize attacks that increase essay length. the simplest attacks 1a and 2a were assigned scores of 560 and 640 points, respectively, by this model, which are relatively high and way higher than expected. however, the relatively longer essays produced by attacks of type b received lower scores, showing that the gemini is sensitive to repetition. this is further evidenced by the fact that is assigns only 40 points to attack 3c, which is the one with most repetitions. this model was the only one to identify one attack (2c) and it almost identified attack 3c. finally, the phi-3-based model was the most resilient to the simplest attacks od type a. even though attacks 1a and 2a are very similar, the model was more sensitive to the latter one. attack 3a, which somehow combines attack 1a and 2a) received an intermediary score, suggesting that the model is not composing the representation to produce scores. phi-3’s tokenizer has a token for the line break, which can make it sensitive to the four paragraphs structure. this appears to be the case, since attacks of type a received smaller scores than attacks of type b. finally, the attacks of type c, which are based on repetition of sentences were not as effective as the attacks of type b which are based of repetition of lists. we highlight that phi-3 achieved the highest qwk score for competence 5, across all models and competences, and was not fooled by any attack—that is, it scored all attacks with a zero. seeing that competence 5 is the one that most often receives a zero, it is possible to imagine that students would try to target it. the fifth competence is responsible for evaluating the conclusion. accordingly, we devised the attack 4 based on the previous results: a sentence that resembles a conclusion using adjectives and adverbs, namely: “consequently, it is up to the fair and democratic federal government to rapidly approve laws that rapidly reduce the occurrence of these horrendous problems. following, the dear brazilian population must abide by the undeniable laws, and the fast police must arrest those that committed any inhuman crime”, copied in 7 paragraphs. the grades assigned by each model for this attack are presented in table 5. table 5. grades assigned by each model when grading the attack 4.full size table surprisingly, the linear regressor was the only grader not deceived in the fifth competence. nonetheless, the model as a whole assigned 800 points, a high score. the grade can be further boosted by adding more repetitions and thus achieving a perfect score. as intended, the bert grader assigned a high grade in competence 5, demonstrating that it is not only using word frequency. still, this attack uses a lot of words, which could cause lead to other competences receiving high scores. it is noteworthy that when trained, this grader did not have access to many training instances of perfect grade, which causes it to rarely (if ever) assign 200 points in a competence. thus, 800 points is the highest possible grade in this method, which means that it was completely deceived by this attack. if gemini is sensitive to repetition and we repeat the same sentence 7 times, then we would expect a low grade assigned by this model. that is exactly what happens. the zero assigned in competence 3 is rounded from 20. in our rounding we always round down, but maybe systems used in classrooms would round up to motivate students. if we had rounded it up, this attack would have been the attack based on repetition that fooled gemini the most. as an additional test made only for this grader, we tested asking it to grade only the first paragraph of attack 4; it assigned a total of 440 points. finally, this attack was the one in which phi-3 assigned the highest grades, for the first time assigning points in the fifth competence. overall, phi-3 was more resilient than bert and the linear regressor but less than gemini. this might suggest that being an order of magnitude pays off by having higher-level features. nonetheless, the grade assigned by this model was surprisingly high. 5 conclusion in this work we investigated whether transformers-based automatic essay scorers are susceptible to universal adversarial attacks that might occur in a classroom setting. to simulate realistic student behavior, we trained a linear regressor based on standard nlp features to predict scores, then used the most important features to craft simple essays that should receive low scores according to grading guidelines yet might fool statistical aes systems. such essays varied from simple list of adjectives and adverbs to repetitions of template texts unrelated to the required prompt. we found that the bert-based grader is susceptible to both long sentences and to the presence of adjectives and adverbs, similarly to reported in the literature [13]. the gemini-based grader was the only model to completely identify an attack, meaning that it assigned 0 to such essays (as a human grader would). the more repetition an essay had, the lower was the score assigned by that model. this contrasts with the bert-based model, which assigned higher scores to longer (and more repetitive) texts. despite that relative improvement, this type of model still assigned undeserving high grades to most of the attacks. this is aligned with the findings of the chatgpt-based grader [7], which found that the model was more lenient than a human-grader. it is however interesting to note that this model assigned the highest score to essays consisting only of a list of adjectives or adverbs. the phi-3-based model, which is the best performing model for most competences, performed similarly to the other models. in particular, it was more susceptible than gemini to an attack that included a unrelated but sensible template conclusion to essay. this is in line with findings that fine-tuned models improve performance at the expense of being more susceptible to attacks [40]. interestingly, although the model was originally trained only texts in english and fine-tuned with texts in portuguese, it achieved the best performances, even when compared to models trained exclusively with portuguese texts. this work shows that even sophisticated aes systems can be fooled by simple attacks, which should be of great concern to real-world systems deployed. future work should better investigate whether standard techniques such as data augmentation and loss function shaping can mitigate such an effect. notes 1.the models are available at: https://github.com/kamel-usp/aes_enem/. 2.specifically, microsoft/phi-3-medium-128k-instruct on the hugging face hub: https://huggingface.co/microsoft/phi-3-medium-128k-instruct. references abdin, m., jacobs, s.a., awan, a.a., aneja, j., awadallah, a., et al.: phi-3 technical report: a highly capable language model locally on your phone (2024) google scholar  amorim, e., veloso, a.: a multi-aspect analysis of automatic essay scoring for brazilian portuguese. in: proceedings of the student research workshop at the 15th conference of the european chapter of the association for computational linguistics, pp. 94–102 (2017) google scholar  attali, y., burstein, j.: automated essay scoring with e-rater v.2. j. technol. learn. assess. 4(3) (2006) google scholar  bazelato, b.s., amorim, e.: a bayesian classifier to automatic correction of portuguese essays. in: conferência internacional sobre informática na educação (tise), vol. 18, pp. 779–782 (2013) google scholar  beigman klebanov, b., madnani, n.: automated evaluation of writing – 50 years and counting. in: proceedings of the 58th annual meeting of the association for computational linguistics. pp. 7796–7810 (2020) google scholar  brown, t., mann, b., ryder, n., subbiah, m., kaplan, j.d., et al.: language models are few-shot learners. in: advances in neural information processing systems, vol. 33, pp. 1877–1901. curran associates, inc. (2020) google scholar  chang, l.h., ginter, f.: automatic short answer grading for finnish with chatgpt. in: proceedings of the aaai conference on artificial intelligence, vol. 38, pp. 23173–23181 (2024) google scholar  chang, m.w., ratinov, l., roth, d., srikumar, v.: importance of semantic representation: dataless classification. in: proceedings of the 23rd national conference on artificial intelligence, vol. 2, pp. 830–835 (2008) google scholar  chiang, w.l., et al.: chatbot arena: an open platform for evaluating llms by human preference (2024) google scholar  dao, t.: flashattention-2: faster attention with better parallelism and work partitioning (2023) google scholar  ebrahimi, j., rao, a., lowd, d., dou, d.: hotflip: white-box adversarial examples for text classification. in: proceedings of the 56th annual meeting of the association for computational linguistics, vol. 2: short papers, pp. 31–36 (2018) google scholar  filho, a.h., concatto, f., do prado, h.a., ferneda, e.: comparing feature engineering and deep learning methods for automated essay scoring of brazilian national high school examination. in: international conference on enterprise information systems (2021) google scholar  filighera, a., ochs, s., steuer, t., tregel, t.: cheating automatic short answer grading with the adversarial usage of adjectives and adverbs. int. j. artif. intell. educ. 34(2), 616–646 (2024) article  google scholar  filighera, a., steuer, t., rensing, c.: fooling automatic short answer grading systems. in: bittencourt, i.i., cukurova, m., muldner, k., luckin, r., millán, e. (eds.) aied 2020. lncs (lnai), vol. 12163, pp. 177–190. springer, cham (2020). https://doi.org/10.1007/978-3-030-52237-7_15 chapter  math  google scholar  fonseca, e.r., medeiros, i., kamikawachi, d., bokan, a.: automatically grading brazilian student essays. in: proceedings of international conference on computational processing of the portuguese language, pp. 170–179 (2018) google scholar  gao, h., oates, t.: universal adversarial perturbation for text classification. arxiv preprint arxiv:1910.04618 (2019) gunasekar, s., et al.: textbooks are all you need (2023) google scholar  hamner, b., morgan, j., lynnvandev, m., ark, t.: the hewlett foundation: automated essay scoring. kaggle (2012) google scholar  hu, e.j., et al.: lora: low-rank adaptation of large language models (2021) google scholar  klein, h., levenburg, n., mckendall, m., mothersell, w.: cheating during the college years: how do business students compare? j. bus. ethics 72, 197–206 (2007) article  google scholar  kumar, y., bhatia, m., kabra, a., li, j.j., jin, d., shah, r.r.: calling out bluff: attacking the robustness of automatic scoring systems with simple adversarial testing. corr arxiv:2007.06796 (2020) kumar, y., parekh, s., singh, s., li, j.j., shah, r.r., chen, c.: automatic essay scoring systems are both overstable and oversensitive: explaining why and proposing defenses. dial. discour. 14(1), 1–33 (2023) article  google scholar  larochelle, h., erhan, d., bengio, y.: zero-data learning of new tasks. in: proceedings of the 23rd national conference on artificial intelligence, vol. 2, pp. 646–651 (2008) google scholar  leal, s.e., duran, m.s., scarton, c.e., hartmann, n.s., aluísio, s.m.: nilc-metrix: assessing the complexity of written and spoken language in brazilian portuguese. lang. res. eval. 58(1), 73–110 (2024) article  google scholar  li, l., ma, r., guo, q., xue, x., qiu, x.: bert-attack: adversarial attack against bert using bert. in: proceedings of the 2020 conference on empirical methods in natural language processing, pp. 6193–6202 (2020) google scholar  lim, c.t., bong, c.h., wong, w.s., lee, n.k.: a comprehensive review of automated essay scoring (aes) research and development. pertanika j. sci. technol. 29(3), 1875–1899 (2021) article  math  google scholar  liu, j., xu, y., zhu, y.: automated essay scoring based on two-stage learning (2019) google scholar  marinho, j., anchiêta, r., moura, r.: essay-br: a brazilian corpus of essays. in: anais do iii dataset showcase workshop, pp. 53–64 (2021) google scholar  marinho, j., cordeiro, f., anchiêta, r., moura, r.: automated essay scoring: an approach based on enem competencies. in: anais do xix encontro nacional de inteligência artificial e computacional, pp. 49–60 (2022) google scholar  mizumoto, a., eguchi, m.: exploring the potential of using an ai language model for automated essay scoring. res. methods appl. linguist. 2(2), 100050 (2023) article  math  google scholar  nvidia, adler, b., agarwal, n., aithal, a., anh, d.h., et al.: nemotron-4 340b technical report (2024) google scholar  page, e.b.: the imminence of... grading essays by computer. in: the phi delta kappan, pp. 238–243 (1966) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) mathscinet  math  google scholar  silveira, i.c., barbosa, a., mauá, d.d.: a new benchmark for automatic essay scoring in portuguese. in: proceedings of the 16th international conference on computational processing of portuguese, vol. 1. pp. 228–237 (2024) google scholar  singh, a., pandey, n., shirgaonkar, a., manoj, p., aski, v.: a study of optimizations for fine-tuning large language models (2024) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12319, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  taghipour, k., ng, h.t.: a neural approach to automated essay scoring. in: proceedings of the 2016 conference on empirical methods in natural language processing, pp. 1882–1891. association for computational linguistics (2016) google scholar  team, g., georgiev, p., lei, v.i., burnell, r., bai, l., et al.: gemini 1.5: unlocking multimodal understanding across millions of tokens of context (2024) google scholar  touvron, h., et al.: llama: open and efficient foundation language models (2023) google scholar  wangkriangkri, p., viboonlarp, c., rutherford, a.t., chuangsuwanich, e.: a comparative study of pretrained language models for automated essay scoring with adversarial inputs. in: ieee region 10 international conference tencon, pp. 875–880 (2020) google scholar  whitley, b.e.: factors associated with cheating among college students: a review. res. high. educ. 39(3), 235–274 (1998) article  math  google scholar  woods, b., adamson, d., miel, s., mayfield, e.: formative essay feedback using predictive scoring models. in: proceedings of the 23rd acm sigkdd international conference on knowledge discovery and data mining, pp. 2071–2080 (2017) google scholar  zhang, h., li, l.h., meng, t., chang, k.w., van den broeck, g.: on the paradox of learning to reason from data. in: proceedings of the thirty-second international joint conference on artificial intelligence, pp. 3365–3373 (2023) google scholar  zhang, j., wang, j., luo, x., ma, b., xiong, n.: imperceptible and reliable adversarial attack. in: cao, c., zhang, y., hong, y., wang, d. (eds.) fcs 2021. ccis, vol. 1558, pp. 49–62. springer, singapore (2022). https://doi.org/10.1007/978-981-19-0523-0_4 chapter  math  google scholar  download references acknowledgments the authors of this work would like to thank the center for artificial intelligence (c4ai-usp) and the support from the são paulo research foundation (fapesp grants #2019/07665-4 and 2022/02937-9) and from the ibm corporation. this work was also supported by cnpq grant no. 305136/2022-4 and capes finance code 001. author information authors and affiliations universidade de são paulo, são paulo, brazil igor cataneo silveira, andré barbosa, daniel silva lopes da costa & denis deratani mauá authors igor cataneo silveiraview author publications search author on:pubmed google scholar andré barbosaview author publications search author on:pubmed google scholar daniel silva lopes da costaview author publications search author on:pubmed google scholar denis deratani mauáview author publications search author on:pubmed google scholar corresponding author correspondence to igor cataneo silveira . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper silveira, i.c., barbosa, a., da costa, d.s.l., mauá, d.d. (2025). investigating universal adversarial attacks against transformers-based automatic essay scoring systems. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_12 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_12 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords automatic essay scoring adversarial attack natural language processing publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ensemble architectures and efficient fusion techniques for convolutional neural networks: an analysis on resource optimization strategies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ensemble architectures and efficient fusion techniques for convolutional neural networks: an analysis on resource optimization strategies conference paper first online: 12 october 2023 pp 107–121 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) ensemble architectures and efficient fusion techniques for convolutional neural networks: an analysis on resource optimization strategies download book pdf download book epub cícero l. costa  orcid: orcid.org/0000-0003-3789-50749,10, danielli a. lima  orcid: orcid.org/0000-0003-0324-66909, celia a. zorzo barcelos10 & … bruno a. n. travençolo  orcid: orcid.org/0000-0001-7690-301x10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 533 accesses 2 citations abstract the human gastrointestinal tract is prone to various abnormalities, including lethal diseases such as cancer, necessitating better endoscopic performance and standardized screening. endoscopic scoring systems lack generalizability, emphasizing the need for artificial intelligence-based solutions. using the hyperkvasir dataset, we employed deep learning, specifically convolutional neural networks, or shortly cnns, to analyze endoscopic images and videos. our study focused on improving the classification of gastrointestinal tract diseases by proposing various cnn ensembles and fusion techniques. through the use of seven cnn models and effective merging techniques, we achieved enhanced performance. validation involved literature review and experiments. densenet-161 influenced the merger process, and integrating resnet152 and vgg further enhanced effectiveness. resource analysis included gpu model, ram usage, and execution time. results demonstrated comparable performance to the previous model, with f1-score of 0.910 and matthews correlation coefficient, mcc for short, of 0.902, using 10 gb gpu ram (compared to 15.8 gb). with 24.7 gb gpu ram, f1-score of 0.913 and mcc of 0.905 were achieved. these findings advance our understanding of ensemble architectures and fusion techniques. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others evaluating efficientnet architectures for pathology detection in endoscopic gastrointestinal tract images article 02 may 2025 empirical study of attention-based models for automatic classification of gastrointestinal endoscopy images chapter © 2023 exploring the effectiveness of deep learning architectures for colorectal polyp detection: performance analysis and insights article 18 april 2024 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational intelligence computer vision gastroenterology gastrointestinal system machine learning artificial intelligence 1 introduction the human gastrointestinal (gi) tract is susceptible to several abnormal mucosal findings, including life-threatening diseases [2]. gi cancer alone accounts for millions of new cases annually, emphasizing the need for improved endoscopic performance and systematic screening [10]. gastrointestinal exams and colonoscopy are essential procedures to investigate the human gi tract [9]. these tests play a vital role in the diagnosis and management of gastrointestinal conditions, contributing to the early detection, treatment and prevention of serious complications [2, 5, 7]. however, current endoscopic scoring systems lack standardization and are subjective [2, 7]. in this context, artificial intelligence (ai)-enabled computer-assisted diagnostic systems, particularly machine learning, have shown promise in healthcare, but the scarcity of medical data impedes progress [2, 8]. to solve this, we used a database (db), called hyperkvasir, a large dataset of gastrointestinal images and videos collected during real exams [2]. the dataset contains over \(1.1 \times 10^5\) images and 374 videos and represents anatomical landmarks as well as pathological and normal findings [2]. over the years, machine learning has evolved into deep learning algorithms, relying primarily on the deep neural network (dnn). convolutional neural networks (cnn), a type of dnn, have emerged as a powerful tool for image analysis and classification, including medical imaging tasks. cnn ensemble architectures have been widely employed to improve forecast accuracy by combining the outputs of various models. these sets leverage the diversity of individual cnn models to improve overall performance. in addition, fusion techniques are employed to effectively integrate predictions from multiple cnn models [11, 16]. in this work, our main objective is to propose a new ensemble architecture and efficient fusion techniques for cnns in the classification of gastrointestinal tract diseases using the hyperkvasir dataset, aiming to obtain better results than in the literature and to optimize computational resources. to achieve this, we performed a thorough literature review to identify relevant studies on the use of deep learning methods in similar health domains. in addition, we performed several experiments to evaluate the effectiveness of our proposed approach. 2 theoretical foundation since the emergence of computers, there have been research efforts to make them reproduce biological characteristics; these are known as bioinspired research. among the bioinspired research, there is one that seeks to simulate the functioning of the brain through artificial neural networks. these networks have undergone many transformations, as reported in the papers [4, 11,12,13, 16,17,18]. this section provides an overview of the hyperkvasir database [2], the dataset utilized in this study. we reviewed the literature on deep learning in digital imaging (di) and consider borgli et al.’s general model [2] as a reference for our research. our objective is to establish a robust foundation by analyzing the dataset and surveying related studies. 2.1 hyperkvasir dataset the hyperkvasir datasetfootnote 1 is composed of images and videos. the dataset content was collected between 2008 and 2016, in a hospital in norway. in this work, 10639 labels available in the dataset are used. the images are separated into 23 classes, in fig. 1 it is possible to see examples of images contained in the dataset. class labels are z-line, pylorus, retroflex stomach, barrett’s, short segment, esophagitis grade a, esophagitis grade b-d, cecum, retroflex rectum, terminal ileum, polyps, ulcerative colitis grade 0–1, ulcerative colitis grade 1, ulcerative colitis grade 1–2, ulcerative colitis grade 2, ulcerative colitis grade \(2-3\), ulcerative colitis grade 3, hemorrhoids, dyed lifted polyps, dyed resection margins, bbps 0–1, bbps 2–3, impacted stool. the dataset offers a file (.csv) with the division of classes studied by borgli [2]. fig. 1. example of images present in the gastrointestinal disease image database. these images are not sequential. full size image 2.2 related works in our study, we started by establishing a solid foundation using the reference article [2]. expanding upon this work, we created a comprehensive graph, as illustrated in fig. 2, to visually illustrate the interconnectedness of relevant papers in our research field. this graph provides a valuable representation of the network of related literature, with a specific focus on the hyperkvasir image and video dataset for gastrointestinal endoscopy, as discussed by borgli et al. [2]. fig. 2. graph for connected papers with hyperkvasir image and video dataset for gastrointestinal endoscopy, borgli et al. [2]. full size image upon analyzing the graph, we identified a total of 41 studies connected to the article [2], resulting in a set of 42 relevant studies for our research. however, we established inclusion criteria, considering only studies published after 2019, that is, after the publication date of the base article. additionally, we excluded systematic literature reviews or survey studies from our analysis. the 10 remaining studies were evaluated for their degree of similarity to the base article, represented by the similarity index sbp (similar based-paper), ranging from 12% to 100%. the higher the sbpfootnote 2 value, the greater the similarity between the article and the base work (previous paper [2]), which is relevant to the results obtained in our research. to gather related works for our paper, each article was thoroughly reviewed based on the following parameters: study name and year, task performed, cnn architecture used, methodology approach, dataset and (sbp) similarity on the base paper. these parameters were used to assess and categorize the papers, ensuring that they align with the focus and objectives of our research. by analyzing each article based on these criteria, we were able to identify and select relevant works that contribute to our study. table 1. state of the art summarization considering gi dataset for medical diagnoses.full size table table 1 shows several studies in the context of gastrointestinal endoscopy. the studies cover a range of tasks such as polyp classification, segmentation, detection, localization, and abnormality identification. various deep learning architectures, including cnns like resnet-152, densenet-161, u-net, pix2pix and hganet are utilized in these studies. different methodologies and techniques such as fuzzy c-means clustering, resunet, conditional random fields (crf), test-time augmentation (tta), and adversarial training are also employed. multiple datasets are used for evaluation, including the hyperkvasir and the ead2019 datasets. the achieved similar based-paper (sbp) rank is used as a performance metric, with higher values indicating better results, which means is most similar to paper [2]. these studies provided valuable insights and advancements in leveraging deep learning and clustering techniques for gastrointestinal endoscopy. they contribute to the development of automated systems for polyp classification, segmentation, and abnormality detection, which can improve the efficiency and accuracy of medical diagnoses. besides that, these papers served as the foundation for our study, as they utilized various cnns for different approaches to gi problems. 2.3 background model in borgli et al. [2], the authors proposed a classification model represented in fig. 3. the model is composed of pre-trained cnns, densenet161 and resnet152. each cnn model is a function, m, composed of a set of subfunctions (convolution, pooling, batch normalization, softmax, optimize, etc.) which, in this case, given an input image \(\overrightarrow{\chi }\), a learning rate value \(\eta \) and the number of epochs e, returns an output \(\overrightarrow{p}\), according to the function: $$\begin{aligned} \overrightarrow{p}_{i} = m(\overrightarrow{\chi }, \eta , e) \end{aligned}$$ (1) fig. 3. previous model [2] for cnn models training and fusion process. full size image the output, \(\overrightarrow{p}\), is a probability vector that indicates the probability that \(\overrightarrow{v}_i\) belongs to one of the classes of the problem. the vector \(\overrightarrow{p} =[c_1, c_2, c_3, \cdots , c_{23}]\), where \(c \in c\) for each gi class and \(|c| = 23\). given a dataset with 10639 gastrointestinal images, \(\overrightarrow{\chi }\), divided into two splits, one set with 5315 images and another set with 5324 images. the model proposed in [2] alternates sets for training and validation. the final response of the model is the average of the results of the two splits. for each split, the authors trained the cnns models \(m_{1}\) and \(m_{2}\), respectively, densenet161 and resnet152, using \(\eta = 0.001\), \(e = 50\) and optimize sgd, stochastic gradient descent; \(m_{1}\) and \(m_{2}\) generated the responses \(\overrightarrow{p}_{1}\) and \(\overrightarrow{p}_{2}\), respectively. after training, the best weight set \(\overrightarrow{w}\) for each model was found and the best-trained models \(m_{1}^{b}\) and \(m_{2}^{b}\) are saved. using the trained models, a model \(m^{\nu }\) is created: $$\begin{aligned} \overrightarrow{p}_{i}^{\nu } = m^{\nu }(\overrightarrow{\chi }, \eta , e, m_{1}^{b}, m_{2}^{b}), \end{aligned}$$ (2) since \(\overrightarrow{p}_{i}^{\nu } = \frac{(\overrightarrow{p}_{1}^{b} + \overrightarrow{p}_{2}^{b})}{2}\), \(\overrightarrow{p}_{1}^{b}\) and \(\overrightarrow{p}_{2}^{b}\) are output of \(m_{1}^{b}\) and \(m_{2}^{b}\), respectively. for each split, the cnn model \(m^{\nu }\) was trained using \(\eta = 0.001\) and \(e = 50\). 3 proposal in this paper, we have two proposals. the first proposal sought to overcome the literature results using fewer gpu resources than the second proposal, which is an expansion of what the authors propose in [2]. the second proposal sought to verify whether other cnn fusions, even using more gpu resources, outperformed the literature results. the cnns models used are accessible through the pytorch framework. 3.1 fusion and ensemble processes our first proposal was to individually train a set of cnns models \(\{m_{1}, m_{2},\) \(m_{3}, ..., m_{n}\}\) and get their respective answers \(\{\overrightarrow{p}_{1}, \overrightarrow{p}_{2}, \cdots , \overrightarrow{p}_{n}\}\), for \(n = 7\). the cnns were densenet121, densenet161, densenet201, efficientnet_b0, mobilenetv2, resnet152 e vgg16. each model was trained adopting the following values for \(\eta = \{0.0001,\) 0.0003,  0.0005,  0.001,  0.003,  \(0.005\}\), \(e = 50\) and optimize sgd, as shown in fig. 4. after training, each model was tested and the responses were fused. each network could participate or not in the fusion. fusions occur between trained models with the same learning rate, the total of fusions were \(2^{n} \times 6\). fig. 4. our proposal for ensemble architecture for training and fusion process. full size image for this proposal, two fusion alternatives were analyzed, by average and by voting. in average fusion, each trained model, \(m_{i}\), generates an output \(\overrightarrow{p}_{i}\) and the fusion is given by \(\overrightarrow{p}_{\tau } = \frac{1}{n}\sum _{i=1}^{n} \overrightarrow{p}_{i}\), where n is the number of models of cnns and \(\overrightarrow{p}_{\tau }\) is the average output of the models that make up the fusion, according to fig. 5(a). fig. 5. different fusion schemes for combining models. full size image in fusion by voting, considering the values in fig. 5(a), each network that makes up the fusion votes in the class that receives the highest percentage of probability, as shown in fig. 5(b). in case of a tie, the first tiebreaker considered the number of times the class was in first and second place in the voting. if the tie remains, among the classes that met the first tiebreaker criterion, the one with the highest percentage of probability is chosen. in our second proposal, we created models with different fusions of pre-trained cnns, so \(m^{\nu }\) was the composition of best models \(\{m_{1}^{b}\), \(m_{2}^{b}\), \(\cdots , m_{n}^{b}\}\), for different values of \(2 \le n \le 7\). the results of the proposals are presented in sect. 4. in the first proposal, with trained models, it is possible to perform fusions between models without having to train them again. in the second proposal, each new fusion generates a new model that needs to be trained. 3.2 evaluation methodology our approach involved the utilization of four metrics precision, recall, f1-score, and matthews correlation coefficient (mcc) – to evaluate the performance of our model and gain valuable insights. additionally, we employed both macro and micro averages to further analyze the overall performance of our model [15]. in classification tasks, true positive (tp), true negative (tn), false positive (fp), and false negative (fn) are commonly used terms that represent the outcomes of the predictions made by a model. tp refers to correct predictions of the positive class, where the model identifies positive instances correctly. tn represents correct predictions of the negative class, where the model identifies negative instances correctly. fp refers to incorrect predictions of the positive class, where the model identifies negative instances as positive. fn represents incorrect predictions of the negative class, where the model identifies positive instances as negative [15]. matthews correlation coefficient, which is a measure of the quality of binary (two-class) classification models. it takes into account tp, tn, fp and fn to provide a balanced assessment of the model’s performance, as shown in eq. 3. the mcc ranges from \([-1,+1]\), where a value of \((+1)\) indicates a perfect classification, (0) indicates a random classification, and \((-1)\) indicates a completely wrong classification. mcc values closer to \((+1)\) indicate better performance of the classification model [3]. $$\begin{aligned} mcc = \frac{tp \times tn fp \times fn}{\sqrt{(tp+fp)(tp+fn)(tn+fp)(tn+fn)}} \end{aligned}$$ (3) micro-average is a method of aggregating the performance metrics across all classes in a multi-class classification problem. the formulas for micro-average precision (mip) (eq. 4), recall (mir) (eq. 5), and f1-score (mif1) (eq. 6) are as follows [19]: macro-average, on the other hand, calculates the performance metrics for each class individually and then takes the average across all classes. the formulas for macro-average precision (map) (eq. 7), recall (mar) (eq. 8), and f1-score (maf1) (eq. 9) are as follows [19]: where precision\(_i\), recall\(_i\), and f1-score\(_i\) represent the precision, recall, and f1-score of class i, and n is the total number of classes. by using micro and macro averaging, we can gain insights into the overall performance of the classification model, considering both the individual class performance and the overall performance across all classes. 4 results and discussion in this section, we present a comprehensive analysis of various aspects related to convolutional neural networks (cnns) and their fusion configurations. firstly, we discuss the analysis for individual cnns. next, we delve into the evaluation of fusion configurations and cnn performance metrics. furthermore, we investigate the fusion with optimal training cnn models. lastly, we conduct a comprehensive analysis of resource consumption for the cnn models. by examining these four aspects, we gain a comprehensive understanding of the individual and fused cnn models, their performance metrics, optimal training configurations, and resource requirements. this knowledge enables us to make informed decisions and design more effective and resource-efficient cnn-based systems. 4.1 analysis for individual convolutional neural networks in this section, we evaluate the individual performance of the cnn models and their effectiveness in tackling the given task. this analysis provides insights into the strengths and weaknesses of each individual model. table 2 present differents cnns and configurations, such as learning rate (lr), in addition, performance metrics, including matthews correlation coefficient (mcc), precision, recall, and f1-score. each row represents a different cnn model, denoted by \(m_{1}\) to \(m_{7}\). analyzing the results, it can be observed that different cnn models achieve varying levels of performance across the evaluated metrics. among the models, \(m_{2}\) (densenet161) stands out with the highest precision, recall, f1-score, and mcc values. on the other hand, \(m_{5}\) (mobilenetv2) and \(m_{7}\) (vgg16) exhibit slightly lower performance in terms of precision, recall, f1-score, and mcc. table 2. individual convolutional neural networks results.full size table 4.2 analysis of fusion configurations and cnn performance metrics in this section, we explore different fusion techniques and assess their impact on the overall performance of the system. table 3 presents the results obtained from the fusion of multiple convolutional neural networks (cnns), ensembles, using the average method. each row in the table represents a different fusion scenario, denoted by \(f_{i}\), where multiple cnn models (\(m_{1}\) to \(m_{7}\)) are combined, where (y) indicates the presence of a model in the fusion configuration, while (n) denotes the absence of that model. the table also includes the lr used for each fusion scenario. the evaluation metrics used to assess the performance of the fusion approach are precision, recall, f1-score, and mcc. analyzing the results, it can be seen that different ensemble architectures and their fusion responses generate varying levels of performance in the evaluated metrics. table 3. results obtained with fusion of cnn by average.full size table among the merger scenarios, \(f_3\) displays the highest f1-score value for macro-average. this scenario combines specific cnn models (\(m_{2}\), \(m_{4}\), \(m_{6}\) and \(m_{7}\)) and achieves remarkable performance in correctly classifying positive and negative instances. these scenarios show the effectiveness of ensemble methods to improve classification accuracy. fusion \(f_1\) stands out, achieving relatively high values of accuracy, recall, f1-score and mcc, for micro average. this suggests that the combination of models \(m_2\), \(m_4\), \(m_6\) and \(m_7\) with an lr of \(\eta = 0.003\) leads to successful predictions with high accuracy and completeness. considering the cnn models that appeared more frequently in the fusion experiments, the models \(m_2\), \(m_6\) and \(m_7\) were used in a greater number of experiments. this suggests that these models have a greater impact on the overall performance of the ensemble architectures. table 4 shows the results of the cnn merger using a voting mechanism where each model in the ensemble makes an independent prediction, and the final prediction is based on the highest number of votes. table 4. results obtained with cnn fusion by vote.full size table upon analyzing the results, it is evident that the performance of the fusion models varies depending on the specific combination of cnn models used. \(f_2\) stands out as it achieves the highest values in terms of f1-score for macro (0.61582), f1-score for micro average (0.90892) and mcc (0.90115). this combination includes models \(m_1\), \(m_2\), \(m_3\), \(m_4\), and \(m_7\), indicating that these models contribute significantly to the overall performance of the fusion model. 4.3 fusion with optimal training cnn models in this section, we explore the integration of specific cnn models to further enhance the system’s performance and achieve superior results. table 5 presents the results of the fusion of trained cnn models using different combinations. in our experiments, in \(f_4^b\), considering the amount of cnns that make up the fusion, we had the best performance compared to other combinations, considering both micro-average, f1-score (0.913) and mcc (0.905). overall, the analysis of the fusion results indicates that the combinations involving \(m_2^b\) (densenet161), \(m_4^b\) (efficientnet_b0), \(m_6^b\) (resnet152), and \(m_7^b\) (vgg16) generally lead to better performance, with higher f1-scores and mcc values. the presence of \(m_1^b\) and \(m_5^b\) does not contribute significantly to the overall performance improvement. table 5. fusion with trained cnn models.full size table 4.4 resource consumption analysis for cnn models in this section, we present table 6, which provides details such as the cnn model name, gpu model used for execution, gpu ram capacity, execution time in minutes, and the number of parameters for each model. to measure the execution time and gpu consumption, the timeit module and the psutil library were used, respectively. all network models, \(m_1\) to \(m_7\), utilize the tesla v100-sxm2-16 gb gpu model. additionally, the \(f_1^{b*}\) model [2] employs the tesla v100-smx2-16 gb, while fusion models \(f_2^b\) to \(f_7^b\), for table 5 and table 6, utilize the nvidia a100-sxm-40 gb gpu model. these data allow us to analyze the computational cost associated with achieving the results presented in table 3, table 4 and table 5. in table 3, the \(f_{1}\) result, including \(m_{2}\), \(m_{4}\), \(m_{6}\), and \(m_{7}\), achieved the highest f1-score (0.91014) with the least number of models used for micro-average. the models were executed individually on the gpu, resulting in a total gpu consumption equal to the highest consumption among the individual models, which is 10 gb for \(m_{2}\). thus, the proposed ensemble \(f_{1}\) with cnn model averaging has a maximum gpu consumption of 10 gb. in table 4, using the technique of fusion by vote, the ensemble \(f_{2}\) achieved the highest f1-score of 0.90892 for micro and f1-score of 0.61582 for macro. the set \(f_{2}\) consisted of models \(m_{1}\), \(m_{2}\), \(m_{3}\), \(m_{4}\) and \(m_{7}\). the gpu consumption for the set \(f_{1}\) corresponds to that of the model \(m_{2}\), which is 10 gb. table 6. resource consumption for network models.full size table in fusion with optimal training cnn models, as shown in table 5, for \(f_1^{b*}\), the f1-score is 0.910, which matches our proposal in table 3. the approach in [2], \(f_1^{b*}\), requires 15.8 gb of gpu, as the best models \(m_{2}^b\) and \(m_{6}^b\) are trained together. building upon the combination of models proposed in [2], we introduce ensembles \(f_{2}^b\) to \(f_{7}^b\), with \(f_{4}^b\) achieving the best result. figure 6 depicts a bubble chart illustrating that we have achieved comparable results (indicated by the blue and red bubbles) when compared to the fusion model \(f_1^{b*}\) in table 5 (green bubble), as presented in [2]. our proposal \(f_{1}\) in table 3 attained the same results while utilizing 10 gb of gpu, which is \(36.7\%\) less than the consumption of [2] with 15.8 gb gpu. the purple bubbles demonstrate that our ensemble architectures using fusion with optimal training cnn models obtain better results than [2], albeit at a higher gpu cost. fig. 6. bubble chart for resource consumption analysis and comparative results. the diameter of the bubbles is associated with gpu consumption. (color figure online) full size image 5 conclusions in conclusion, to aid in the diagnosis of diseases of the human gastrointestinal tract, through image classification, our objective was to propose ensemble architectures and efficient fusion techniques for cnns, aiming to obtain better results than in the literature and to optimize computational resources. a literature review and extensive experiments were conducted to validate the effectiveness of the proposed approach. the findings of this study highlight the significant contribution of densenet161 and resnet152 to the fusion process in all experiments. furthermore, our findings demonstrate a similar level of performance compared to the previous model, as indicated by an f1-score of 0.910 and mcc of 0.902. remarkably, our approach achieves this performance using just 10 gb of gpu ram, in contrast to the previous model’s requirement of 15.8 gb. we were able to achieve a minimally higher f1 value of 0.913 and mcc of 0.905, but using 24.7 gb of gpu ram. these findings contribute to our understanding of individual model performance, fusion techniques, and resource utilization, paving the way for the design and implementation of more efficient and effective deep learning-based systems in endoscopic scoring. the main challenge was accessing sufficient computational resources. future research should explore model compression techniques to reduce computational requirements. additionally, further experiments and analysis are needed to identify optimal model combinations and refine the fusion process. evaluating fusion alternatives on different datasets is crucial for assessing performance and generalizability. notes 1.available at: https://datasets.simula.no/hyper-kvasir. 2.available at: https://www.connectedpapers.com/main. references ali, s., et al.: an objective comparison of detection and segmentation algorithms for artefacts in clinical endoscopy. sci. rep. 10(1), 2748 (2020) article  google scholar  borgli, h., et al.: hyperkvasir, a comprehensive multi-class image and video dataset for gastrointestinal endoscopy. sci. data 7(1), 283 (2020) article  google scholar  chicco, d., jurman, g.: the advantages of the matthews correlation coefficient (mcc) over f1 score and accuracy in binary classification evaluation. bmc genom. 21, 1–13 (2020) article  google scholar  fukushima, k.: neocognitron: a self-organizing neural network model for a mechanism of pattern recognition unaffected by shift in position. biol. cybern. 36(4), 193–202 (1980) google scholar  hicks, s.a., jha, d., thambawita, v., halvorsen, p., hammer, h.l., riegler, m.a.: the endotect 2020 challenge: evaluation and comparison of classification, segmentation and inference time for endoscopy. in: del bimbo, a., et al. (eds.) icpr 2021. lncs, vol. 12668, pp. 263–274. springer, cham (2021). https://doi.org/10.1007/978-3-030-68793-9_18 chapter  google scholar  iqbal, i., walayat, k., kakar, m.u., ma, j.: automated identification of human gastrointestinal tract abnormalities based on deep convolutional neural network with endoscopic images. intell. syst. appl. 16, 200149 (2022) google scholar  jha, d., et al.: real-time polyp detection, localization and segmentation in colonoscopy using deep learning. ieee access 9, 40496–40510 (2021) article  google scholar  jha, d., et al.: medico multimedia task at mediaeval 2020: automatic polyp segmentation. arxiv preprint arxiv:2012.15244 (2020) jha, d., et al.: a comprehensive study on colorectal polyp segmentation with resunet++, conditional random field and test-time augmentation. ieee j. biomed. health inform. 25(6), 2029–2040 (2021) article  google scholar  jha, d., et al.: kvasir-seg: a segmented polyp dataset. in: ro, y.m., et al. (eds.) mmm 2020. lncs, vol. 11962, pp. 451–462. springer, cham (2020). https://doi.org/10.1007/978-3-030-37734-2_37 chapter  google scholar  krizhevsky, a., sutskever, i., hinton, g.e.: imagenet classification with deep convolutional neural networks. in: advances in neural information processing systems, vol. 25, pp. 1097–1105 (2012) google scholar  lecun, y., bengio, y., hinton, g.: deep learning. nature 521(7553), 436–444 (2015) article  google scholar  lecun, y., et al.: backpropagation applied to handwritten zip code recognition. neural comput. 1(4), 541–551 (1989) article  google scholar  naess, e., thambawita, v., hicks, s.a., riegler, m.a., halvorsen, p.: pyramidal segmentation of medical images using adversarial training. in: proceedings of the 2021 workshop on intelligent cross-data analysis and retrieval, pp. 33–38 (2021) google scholar  sarkar, d., bali, r., sharma, t.: practical machine learning with python (2018). https://doi.org/10.1007/978-1-4842-3207-1 sze, v., chen, y.h., yang, t.j., emer, j.s.: efficient processing of deep neural networks: a tutorial and survey. proc. ieee 105(12), 2295–2329 (2017) article  google scholar  sze, v., chen, y.h., yang, t.j., emer, j.s.: efficient processing of deep neural networks. synth. lect. comput. archit. 15(2), 1–341 (2020) math  google scholar  szegedy, c., ioffe, s., vanhoucke, v., alemi, a.a.: inception-v4, inception-resnet and the impact of residual connections on learning. in: thirty-first aaai conference on artificial intelligence (2017) google scholar  takahashi, k., yamamoto, k., kuchiba, a., koyama, t.: confidence interval for micro-averaged f 1 and macro-averaged f 1 scores. appl. intell. 52(5), 4961–4972 (2022) article  google scholar  thambawita, v., et al.: an extensive study on cross-dataset bias and evaluation metrics interpretation for machine learning applied to gastrointestinal tract abnormality classification. acm trans. comput. healthc. 1(3), 1–29 (2020) article  google scholar  download references acknowledgements this study was financed in part by coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001* and conselho nacional de desenvolvimento científico e tecnológico (grant 306436/2022-1). in addition, it had the support of the instituto federal do triângulo mineiro e universidade federal de uberlândia. author information authors and affiliations federal institute of triângulo mineiro (iftm) campus patrocínio, uberaba, mg, brazil cícero l. costa & danielli a. lima federal university of uberlândia (ufu) campus santa mônica, uberlândia, mg, brazil cícero l. costa, celia a. zorzo barcelos & bruno a. n. travençolo authors cícero l. costaview author publications search author on:pubmed google scholar danielli a. limaview author publications search author on:pubmed google scholar celia a. zorzo barcelosview author publications search author on:pubmed google scholar bruno a. n. travençoloview author publications search author on:pubmed google scholar corresponding author correspondence to cícero l. costa . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper costa, c.l., lima, d.a., zorzo barcelos, c.a., travençolo, b.a.n. (2023). ensemble architectures and efficient fusion techniques for convolutional neural networks: an analysis on resource optimization strategies. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_8 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords convolutional neural networks gastrointestinal abnormalities ensembles performance image classification performance fusion techniques resource consumption publish with us policies and ethics profiles danielli a. lima view author profile bruno a. n. travençolo view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature inranker: distilled rankers for zero-shot information retrieval | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper inranker: distilled rankers for zero-shot information retrieval conference paper first online: 30 january 2025 pp 140–154 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) inranker: distilled rankers for zero-shot information retrieval download book pdf download book epub thiago soares laitz  orcid: orcid.org/0000-0001-7205-20949,10, konstantinos papakostas  orcid: orcid.org/0000-0002-2096-293111, roberto lotufo  orcid: orcid.org/0000-0002-5652-08529,12 & … rodrigo nogueira  orcid: orcid.org/0000-0002-2600-60359,10,11  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 406 accesses 2 citations abstract despite multi-billion parameter neural rankers being common components of state-of-the-art information retrieval pipelines, they are rarely used in production due to the enormous amount of compute required for inference. in this work, we propose a method for distilling large rankers into their smaller versions focusing on out-of-domain effectiveness. we introduce inranker, a version of monot5 [25] distilled from monot5-3b with increased effectiveness on out-of-domain scenarios. our key insight is to use language models and rerankers to generate as much as possible synthetic “in-domain” training data, i.e., data that closely resembles the data that will be seen at retrieval time. the pipeline consists of two distillation phases that do not require additional user queries or manual annotations: (1) training on existing supervised soft teacher labels, and (2) training on teacher soft labels for synthetic queries generated using a large language model. consequently, models like monot5-60m and monot5-220m improved their effectiveness by using the teacher’s knowledge, despite being 50x and 13x smaller, respectively. furthermore, we show that it is possible to transfer knowledge from english models to portuguese fine-tuned models. models and code are available at https://github.com/unicamp-dl/inranker. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others efficient knowledge distillation: empowering small language models with teacher model insights chapter © 2024 multi-task feature self-distillation for semi-supervised machine translation chapter © 2024 metadistiller: network self-boosting via meta-learned top-down distillation chapter © 2020 1 introduction it is well known that the effectiveness of ir pipelines increases with larger models [2, 22, 24, 25, 28]. for instance, multi-billion parameter rankers and dense models achieve top positions on leaderboards of ir benchmarks and competitions [10,11,12]. these large models leverage increased representation capacity, enabling them to encode features that might elude smaller models. however, deploying these large models is not without its challenges. the computational costs are substantial, often requiring specialized hardware such as gpus or tpus to operate in latency-critical applications. the high cost is directly related to the large number of parameters that these models contain, as they require hardware with high memory and compute capacity and because the latency scales almost linearly with the number of parameters. in a production environment, this means higher operating costs and reduced scalability. to address these challenges, there have been efforts to create more efficient models without significantly reducing effectiveness. one such approach is model distillation [17]. distilled models, such as minilm [32], use a teacher or an ensemble of larger models to transfer knowledge to a smaller student model. rosa et al. [30] show that minilm surpassed the zero-shot effectiveness of monot5-base, which is a seq2seq model trained for binary classification, in ir tasks despite being an order of magnitude smaller in size. this has shown that knowledge transfer via model distillation is not only feasible but also effective. however, most distillation techniques have been geared towards optimizing effectiveness on specific benchmark tasks and do not focus on out-of-domain effectiveness. rosa et al. also show that while smaller models are capable of achieving high in-domain results, similar to their larger counterparts, the disparity in effectiveness becomes evident in out-of-domain scenarios. as the concept of out-of-domain is subjective, we define it as a test distribution that is significantly different from the training distribution. a straighforward example of a out-of-domain scenario is when a model is trained on chemistry-related data and tested on legal data. however, we recognize that this distinction blurs in many scenarios. fig. 1. effectiveness on the beir benchmark [31]. all models are based on monot5 [25], applying different fine-tuning methods. full size image usually, training a retrieval model requires human-annotated hard labels informing which passage is relevant for each query. however, with the advance of large language models (llms), it has become possible to generate synthetic queries for passages, providing a feasible approach for data augmentation [1, 2, 4, 19, 26]. our work introduces a method for the generation of synthetic data specifically designed for distilling rankers that increases their out-of-domain effectiveness. we present inranker, a distilled model derived from monot5-3b [25], that uses the predictions of the teacher directly with both synthetic, generated from an out-of-domain corpus, and real query-document pairs. effectively, this approach converts any corpus to be in-domain, since the model will be trained using queries from the target domain. as a result, this approach leads to reduced model sizes while maintaining improved out-of-domain effectiveness as presented in fig. 1. the methodology, results, and ablation experiments are presented in detail in the following sections. fig. 2. pipeline for generating the synthetic triples <query, passage, soft label> for the inranker model. full size image 2 related work the research community has been using llms in a variety of tasks aimed at increasing the availability of data and improving the effectiveness of existing systems. magister et al. [20] employed synthetic text generated by palm 540b [8] and gpt-3 175b [5] to transfer knowledge to smaller models such as t5. fu et al. [15] successfully specialized student models in multi-step reasoning using flant5 [9] and code-davinci-002 as teachers. however, all these works rely on training the student models using synthetic text rather than directly using the soft labels. furthermore, muhamed et al. [21] distilled cross-attention scores of a language model for click-through-rate prediction, achieving better results when exposed to contextual features such as tabular data. wang et al. [32] distilled the self-attention module, which is a crucial part of transformers, and successfully transferred knowledge to a variety of tasks. previous studies have also explored training a student from soft labels produced by a teacher: hofstätter et al. [18] proposed a cross-architecture knowledge distillation approach using the marginmse loss. similarly, formal et al. [14] used the marginmse loss to distill knowledge to sparse neural models. finally, hashemi et al. [16] proposed a method for generating synthetic data for domain adaptation of dense passage retrievers. this approach involves creating new queries and a target collection, along with pseudo-labels extracted using a bert cross-encoder. however, they did not evaluate the model’s effectiveness on datasets to which it was not domain-adapted. the existing research has mainly focused on in-domain evaluation, where the goal has been to increase the effectiveness of the student model on test datasets whose domain is similar to the datasets it was trained on. our study also focuses on the robustness of the student and its ability to perform well even in out-of-domain scenarios, similar to the abilities of the larger teacher model. 3 methodology our proposed method consists of two key phases of distillation, each designed with specific objectives to maximize the model’s zero-shot effectiveness. the first phase uses real-world data to familiarize the student model with the ranking task, while the second phase uses synthetic data designed to improve zero-shot generalization and improve the model’s effectiveness on a specific dataset. the dataset used to distill inranker consists of {query, passage, logits} triplets, where the logits (soft labels) originate from a teacher model that has been trained for the relevance task. for the first stage, we chose to use query-document pairs from the ms marco [23] dataset, given their variety, the large number of annotated pairs, and its demonstrated effectiveness in enhancing retrieval effectiveness [29]. next, we source the synthetic queries from inpars [2], which used an llm to generate queries for the datasets in beir in a few-shot manner. distilling rerankers involves using the mean squared error (mse) loss to match the logits of the teacher and the student, as part of a two-phase pipeline illustrated in fig. 2. the first phase consists of two steps: (1) generating the teacher logits given a query and either a positive (relevant) or a negative (non-relevant) passage, where the negatives are randomly sampled using bm25 on the top-\(k=1000\) candidates, and the positives are sampled from the human-annotated pairs; and (2) training inranker given the queries and passages as input using the mse loss to match the student logits to those of the teacher, who remains frozen during training. this approach can be beneficial as it removes the need for making hard decisions about a passage’s relevance, i.e. determining a threshold to obtain binary relevance labels, and instead focuses on a soft target objective aimed at aligning the student’s perception of relevance with that of the teacher. the second phase, with a focus on zero-shot effectiveness, uses the same two steps. however, instead of employing real queries sourced from a costly human-annotation process, it uses synthetic queries generated by an llm based on randomly sampled documents from the corpus. in this scenario, the positive document is the one used to create the query, and the negatives are collected using the same top-k sampling approach as before. we also perform zero-mean normalization on the teacher logits for each query-document pair, independent of the overall dataset distribution. this approach intends to make the data distribution symmetric for each query-document pair, thereby minimizing the bias that inranker is required to learn. formally: $$\begin{aligned} \begin{aligned} {l'}_\textrm{true} &= l_\textrm{true} \frac{l_\textrm{true} + l_\textrm{false}}{2} \\ {l'}_\textrm{false} &= l_\textrm{false} \frac{l_\textrm{true} + l_\textrm{false}}{2} \end{aligned} \end{aligned}$$ (1) with \(l_\textrm{true}\) and \(l_\textrm{false}\) denoting the teacher’s logits for the relevant and non-relevant classes, respectively, and \(l'\) being the normalized values. this results in the following loss for each training example: $$\begin{aligned} \mathcal {l}_\textrm{mse} = ([y_\textrm{true} {l'}_\textrm{true}]^2 + [y_\textrm{false} {l'}_\textrm{false}]^2) \end{aligned}$$ (2) with \(y_\textrm{true}\) and \(y_\textrm{false}\) representing the logits of the student. due to the training objective described in eq. (2), the model no longer determines the relevance of passages and instead focuses on replicating the teacher’s output, thus eliminating the need for tunning a relevance threshold that would be needed to produce a binary label. with this approach, we can easily expand the out-of-domain knowledge of distilled models by generating new queries for documents using an llm and fine-tuning the distilled model using the teacher’s logits. in the experiments section, we demonstrate the effectiveness of this approach in enhancing the student model’s effectiveness across 16 datasets of beir simultaneously. we present the hyperparameters used for training and the dataset curation in appendix a, and we discuss variations of the training loss in appendix c. 4 experiments 4.1 english knowledge distillation results we distilled monot5-3b to models with parameters ranging from 60m to 3b, using combinations of the following configurations: human hard: representing the common approach for training rankers with human-annotated hard (i.e., binary) labels from the ms marco passage ranking dataset. in this case, a vanilla cross-entropy loss is used: $$\begin{aligned} \mathcal {l}_\textrm{ce} = -\log p_\textrm{relevant} \log p_\mathrm {non-relevant} \end{aligned}$$ (3) where \(p_\textrm{relevant}\) and \(p_\mathrm {non-relevant}\) are the probabilities assigned by the model to the relevant and non-relevant query-document pair, respectively. non-relevant pairs are sampled from the top-1000 retrieved by bm25. human soft: representing a distillation step for matching the logits of a teacher and a student model, using real (human-generated) queries from the ranking dataset as inputs, but without the binary relevance judgments for targets. synthetic soft: representing a distillation step for matching the logits of the two models, similar to the previous configuration, but using exclusively synthetic queries generated from the corresponding beir corpora with inpars [2, 19]. table 1. distillation results (ndcg@10) on 16 beir datasets. the model marked with * represents the teacher model. we did not train inranker-3b on human soft labels due to computational constraints.full size table from table 1, we see that both distillation steps were essential for improving the average ndcg@10 score compared to the model trained solely using human hard labels from ms marco.footnote 1 as a result, inranker-60m (row 3) and inranker-220m (row 6), despite being 50x and 13x smaller than the teacher model, were able to improve their effectiveness on the beir benchmark significantly. moreover, models trained exclusively on ms marco soft labels (rows 2 & 5) saw an increase in effectiveness in comparison to training on solely hard labels (rows 1 & 4), corroborating findings from previous studies regarding the effectiveness of soft labels [14, 16,17,18]. furthermore, we observed an increase in the effectiveness even in self-distillation training (row 8), where the student learns soft labels generated by itself. we hypothesize that the improvement stems from the extra knowledge provided by the language model used to generate the synthetic queries. we did not provide results for the 3b model trained on both human soft and synthetic soft due to computational costs. furthermore, in table 2, we present a effectiveness comparison between inranker, promptagator [13], and rankt5 [33]. although we used monot5-3b as a teacher for our experiments, which has a lower effectiveness on average when compared to promptagator or rankt5-3b, our method is model-agnostic and thus one could use a stronger teacher model and anticipate even stronger results. nonetheless, inranker remains competitive in both model groups of 220m and 3b parameters, outperforming the other two baselines in 6 out of the 10 evaluated datasets, despite the average score not reflecting this due to promptagator and rankt5 attaining a significantly higher score in two datasets: arguana and touché. table 2. comparison of the effectiveness for various reranking models, measured by ndcg@10 on the beir benchmark. the model marked with * represents the teacher model used for training inranker. bolded scores correspond to the best effectiveness on a specific dataset for a given model size, while underlined scores indicate the best effectiveness overall.full size table 4.2 portuguese knowledge distillation results to further assess the efficacy of the technique in different languages, we evaluated inranker on a portuguese dataset for information retrieval: quati [6]. instead of using the same t5 model, we started from ptt5, a portuguese fine-tuned version of t5 [7, 27]. we used the same two-step training approach as before, but with a strategy that allowed us to distill a portuguese model using an english teacher (monot5-3b). given the availability of a translated version of ms marco in portuguese [3], the first step (human soft) involved training the model using the portuguese text, while matching the soft labels generated by the teacher using the original english text. this approach enabled us to leverage a stronger model that is not available in portuguese for the distillation process. in the second step, involving synthetic soft labels from beir, we trained using the english text, as there is no translated version of beir available in portuguese. table 3. inranker results on quati, a portuguese evaluation dataset for information retrieval using ptt5-v2 [27]. all synthetic soft labels were generated using the beir datasets.full size table the results of this evaluation are presented in table 3. we conclude that, similarly to english, the distillation process was able to improve the effectiveness of models in a zero-shot manner, as the models were trained using only real data from ms marco and synthetic data from beir. remarkably, inranker-740m surpassed the effectiveness of the mt5-3.7b on quati. note that for the quati evaluation we used the same prompt presented in the paper to annotate all unjudged documents using gpt-4-turbo. therefore all results are presented with a jugded@10 of 100%. further results using synthetic data from quati and mixed training data are presented in appendix e. 4.3 ablations in this section, we present our ablation experiments aimed at validating the best configuration for distilling monot5-3b into smaller t5-based models, as well as assessing their zero-shot capabilities. the initial experiments we conducted focused on evaluating how distillation would affect the model’s effectiveness on novel dataset distributions that were not seen during training, i.e., we did not generate synthetic queries for them. to achieve this, we created two subsets, each containing 8 randomly selected datasets from 16 datasets of beirfootnote 2, which we named sample sets 1 and 2 and used only one set for training per experiment. the datasets that were used for training are designated as the “in-domain” category, while the remaining datasets, i.e. the other 8 datasets that are not part of the training set, represent the “out-of-domain” (o.o.d.) category. table 4. comparison of the in-domain vs out-of-domain effectiveness of our method, measured by ndcg@10. the model marked with * represents the teacher model used for the knowledge distillation process.full size table impact of soft knowledge distillation on o.o.d. effectiveness. our first ablation experiment focused on evaluating the initial distillation process using the ms marco dataset with soft labels. to accomplish this, we generated logits with monot5-3b and trained both t5-base and t5-small models for 10 epochs. as shown in table 4, rows 1–2 & 5–6, both models demonstrated an improvement in their ndcg@10 scores compared to the baseline, which was trained using the hard labels from ms marco. remarkably, the overall score increased in both scenarios, even though the models were not exposed to any beir passages during this phase.footnote 3 adding soft synthetic targets as a second distillation phase. for the next experiment, we applied a second distillation step with synthetic soft labels on top of the model that we acquired from the last phase (monot5 w/ soft human labels). for that, we used the 100k synthetic queries generated by inpars for each dataset indicated as “in-domain” and trained for 10 epochs. as shown in table 4, rows 3 & 7, while it was expected that the in-domain datasets would have an increase in their ndcg@10 scores, we observe that the out-of-domain datasets also had improvements, suggesting that the model’s generalization capabilities were enhanced. using hard human targets for the first distillation phase. finally, we investigated the impact of skipping the first phase of distillation on ms marco logits, and instead starting from a model that was trained on hard human labels (monot5-small and monot5-base) and directly training using the synthetic soft beir targets. as we can see in table 4, rows 3–4 & 7–8, when comparing with the model that was trained using the soft human targets, the overall effectiveness was reduced. from this, we conclude that the distillation step that includes the soft human targets on ms marco is beneficial, as it improves the model’s effectiveness in both in-domain and out-of-domain scenarios. upper bound for soft distillation. to estimate the upper bound of the effectiveness that these models could attain through distillation, we repeated the process using real queries from beir, (i.e., the validation queries) instead of the synthetic ones. results presented in table 5 show that for both model sizes, there was an increase in effectiveness for the in-domain datasets, as the model was exposed to the evaluation queries during training. however, we also observed an increase in effectiveness for out-of-domain datasets, indicating that the synthetic queries used for training could be improved. table 5. upper bound effectiveness (ndcg@10) using real queries from beir for the distillation datasets. bold indicates the best between using synthetic and real queries.full size table 5 conclusion this paper introduces a method for distilling the knowledge of information retrieval models and improve upon previous work how to better use synthetic data, aimed at improving the out-of-domain effectiveness of students. the study reveals that, through this knowledge distillation process, smaller models can achieve results comparable to the teacher, even in the context of multilingual transfer knowledge. this approach is particularly significant for applications where computational resources are limited, in production environments, or for languages with a lack of available models that can serve as a teacher. the methodology involves two steps of distillation: (1) using a human-curated corpus, and (2) using synthetic data generated by an llm. consequently, our work shows that it is possible to improve a reranker’s capabilities in specific domains without requiring additional human-annotated labels. finally, we observe that synthetic query generation could be improved since the real queries achieved a better out-of-domain effectiveness compared to the model trained solely on synthetic ones. however, the presented method has limitations. specifically, it is not clear how to adapt the proposed loss to train dense retrievers, which tipically use a contrastive loss. notes 1.results per dataset are shown in appendix d. 2.we list their exact composition in appendix b. 3.the individual results for each dataset are presented in appendix d. references alaofi, m., gallagher, l., sanderson, m., scholer, f., thomas, p.: can generative llms create query variants for test collections? an exploratory study. in: proceedings of the 46th international acm sigir conference on research and development in information retrieval, pp. 1869–1873. sigir 2023. association for computing machinery, new york (2023). https://doi.org/10.1145/3539618.3591960 bonifacio, l., abonizio, h., fadaee, m., nogueira, r.: inpars: unsupervised dataset generation for information retrieval. in: proceedings of the 45th international acm sigir conference on research and development in information retrieval, pp. 2387–2392. sigir 2022. association for computing machinery, new york (2022). https://doi.org/10.1145/3477495.3531863 bonifacio, l., campiotti, i., de alencar lotufo, r., nogueira, r.f.: mmarco: a multilingual version of ms marco passage ranking dataset. corr abs/2108.13897 (2021). https://arxiv.org/abs/2108.13897 boytsov, l., et al.: inpars-light: cost-effective unsupervised training of efficient rankers (2023) google scholar  brown, t., et al.: language models are few-shot learners. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 1877–1901. curran associates, inc. (2020). https://proceedings.neurips.cc/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-paper.pdf bueno, m., de oliveira, e.s., nogueira, r., lotufo, r.a., pereira, j.a.: quati: a brazilian portuguese information retrieval dataset from native speakers (2024) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data (2020) google scholar  chowdhery, a., narang, s., devlin, j., et. al: palm: scaling language modeling with pathways (2022) google scholar  chung, h.w., et al.: scaling instruction-finetuned language models (2022) google scholar  craswell, n., mitra, b., yilmaz, e., campos, d.: overview of the trec 2020 deep learning track (2021) google scholar  craswell, n., mitra, b., yilmaz, e., campos, d., lin, j.: overview of the trec 2021 deep learning track. in: soboroff, i., ellis, a. (eds.) proceedings of the thirtieth text retrieval conference, trec 2021, online, 15–19 november 2021. nist special publication, vol. 500-335. national institute of standards and technology (nist) (2021). https://trec.nist.gov/pubs/trec30/papers/overview-dl.pdf craswell, n., et al.: overview of the trec 2022 deep learning track. in: soboroff, i., ellis, a. (eds.) proceedings of the thirty-first text retrieval conference, trec 2022, online, 15–19 november 2022. nist special publication, vol. 500-338. national institute of standards and technology (nist) (2022). https://trec.nist.gov/pubs/trec31/papers/overview_deep.pdf dai, z., et al.: promptagator: few-shot dense retrieval from 8 examples. in: the eleventh international conference on learning representations (2023). https://openreview.net/forum?id=gml46ympu2j formal, t., lassance, c., piwowarski, b., clinchant, s.: from distillation to hard negative sampling: making sparse neural ir models more effective. in: proceedings of the 45th international acm sigir conference on research and development in information retrieval, pp. 2353–2359. sigir 2022. association for computing machinery, new york (2022). https://doi.org/10.1145/3477495.3531857 fu, y., peng, h., ou, l., sabharwal, a., khot, t.: specializing smaller language models towards multi-step reasoning. in: krause, a., brunskill, e., cho, k., engelhardt, b., sabato, s., scarlett, j. (eds.) proceedings of the 40th international conference on machine learning. proceedings of machine learning research, vol. 202, pp. 10421–10430. pmlr (2023). https://proceedings.mlr.press/v202/fu23d.html hashemi, h., zhuang, y., kothur, s.s.r., prasad, s., meij, e., croft, w.b.: dense retrieval adaptation using target domain description. in: proceedings of the 2023 acm sigir international conference on theory of information retrieval, pp. 95–104. ictir 2023. association for computing machinery, new york (2023). https://doi.org/10.1145/3578337.3605127 hinton, g., vinyals, o., dean, j.: distilling the knowledge in a neural network (2015) google scholar  hofstätter, s., althammer, s., schröder, m., sertkan, m., hanbury, a.: improving efficient neural ranking models with cross-architecture knowledge distillation (2020) google scholar  jeronymo, v., et al.: inpars-v2: large language models as efficient dataset generators for information retrieval (2023) google scholar  magister, l.c., mallinson, j., adamek, j., malmi, e., severyn, a.: teaching small language models to reason. in: rogers, a., boyd-graber, j., okazaki, n. (eds.) proceedings of the 61st annual meeting of the association for computational linguistics (volume 2: short papers), pp. 1773–1781. association for computational linguistics, toronto, canada (2023). https://doi.org/10.18653/v1/2023.acl-short.151. https://aclanthology.org/2023.acl-short.151 muhamed, a., et al.: ctr-bert: cost-effective knowledge distillation for billion-parameter teacher models. in: neurips efficient natural language and speech processing workshop (2021) google scholar  neelakantan, a., et al.: text and code embeddings by contrastive pre-training (2022) google scholar  nguyen, t., et al.: ms marco: a human generated machine reading comprehension dataset (2016) google scholar  ni, j., et al.: large dual encoders are generalizable retrievers. in: goldberg, y., kozareva, z., zhang, y. (eds.) proceedings of the 2022 conference on empirical methods in natural language processing, pp. 9844–9855. association for computational linguistics, abu dhabi, united arab emirates (2022). https://doi.org/10.18653/v1/2022.emnlp-main.669. https://aclanthology.org/2022.emnlp-main.669 nogueira, r., jiang, z., pradeep, r., lin, j.: document ranking with a pretrained sequence-to-sequence model. in: cohn, t., he, y., liu, y. (eds.) findings of the association for computational linguistics: emnlp 2020, pp. 708–718. association for computational linguistics, online (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.63. https://aclanthology.org/2020.findings-emnlp.63 penha, g., palumbo, e., aziz, m., wang, a., bouchard, h.: improving content retrievability in search with controllable query generation. in: proceedings of the acm web conference 2023, pp. 3182–3192. www 2023. association for computing machinery, new york (2023). https://doi.org/10.1145/3543507.3583261 piau, m., lotufo, r., nogueira, r.: ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language (2024) google scholar  pradeep, r., nogueira, r., lin, j.: the expando-mono-duo design pattern for text ranking with pretrained sequence-to-sequence models (2021) google scholar  ren, r., et al.: a thorough examination on zero-shot dense retrieval. in: bouamor, h., pino, j., bali, k. (eds.) findings of the association for computational linguistics: emnlp 2023, pp. 15783–15796. association for computational linguistics, singapore (2023). https://doi.org/10.18653/v1/2023.findings-emnlp.1057. https://aclanthology.org/2023.findings-emnlp.1057 rosa, g.m., et al.: no parameter left behind: how distillation and model size affect zero-shot retrieval (2022) google scholar  thakur, n., reimers, n., rücklé, a., srivastava, a., gurevych, i.: beir: a heterogeneous benchmark for zero-shot evaluation of information retrieval models. in: thirty-fifth conference on neural information processing systems datasets and benchmarks track (round 2) (2021). https://openreview.net/forum?id=wcu6t5xfjej wang, w., wei, f., dong, l., bao, h., yang, n., zhou, m.: minilm: deep self-attention distillation for task-agnostic compression of pre-trained transformers. in: larochelle, h., ranzato, m., hadsell, r., balcan, m., lin, h. (eds.) advances in neural information processing systems, vol. 33, pp. 5776–5788. curran associates, inc. (2020) google scholar  zhuang, h., et al.: rankt5: fine-tuning t5 for text ranking with ranking losses. in: proceedings of the 46th international acm sigir conference on research and development in information retrieval, pp. 2308–2313. sigir 2023. association for computing machinery, new york (2023). https://doi.org/10.1145/3539618.3592047 download references author information authors and affiliations school of electrical and computing engineering, state university of campinas (unicamp), campinas, brazil thiago soares laitz, roberto lotufo & rodrigo nogueira maritaca ai, campinas, brazil thiago soares laitz & rodrigo nogueira zeta alpha, amsterdam, netherlands konstantinos papakostas & rodrigo nogueira neuralmind, campinas, brazil roberto lotufo authors thiago soares laitzview author publications search author on:pubmed google scholar konstantinos papakostasview author publications search author on:pubmed google scholar roberto lotufoview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar corresponding author correspondence to thiago soares laitz . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. appendices a training details this appendix presents the parameters used for training the models using an a100 gpu with 80gb of vram. all experiments were conducted using the same learning rate of 7e-5 and the adamw optimizer with its default hyperparameters in huggingface. the batch size was set to 32. for the 3b model, we used gradient checkpointing and gradient accumulation (to achieve an effective batch size of \(2 \times 16\)) due to memory constraints. during the generation of soft labels using the teacher model, we sampled 9 non-relevant passages passage for each relevant passage, leading to 10 pairs of logits per query. differently from inpars and promptagator, which train a separate model for each dataset, inranker is a single model trained on all 16 datasets from beir. b datasets used in ablations this section shows the datasets that were randomly chosen for inclusion in each sample set, resulting in the use of 12 out of the 16 beir datasets (as some were not used for training at all). sample set 1 includes the following datasets from the beir benchmark: nfcorpus, nq, hotpotqa, dbpedia, quora, scidocs, fiqa-2018, and signal-1m. sample set 2 comprises trec-covid, bioasq, nq, hotpotqa, robust04, scidocs, scifact, and fiqa-2018. climate-fever, trec-news, arguana, and touché-2020 are not included in either sample set. c loss function ablation we tested different loss functions, including the kl divergence and mse, to match the logits of the two models. the results indicates that kl divergence was slightly worse for t5-small and that using only the true label in mse as opposed to using both true and false labels also reduced the effectiveness. for a model with 60m parameters, the mse with normalized logits yielded a result of 0.4807, while using the true logits only resulted in 0.4748. the kl divergence for the same model size was 0.4712. for a larger model with 220m parameters, the mse with normalized logits produced a result of 0.5008, and the kl divergence was 0.5012. these results reflect the average ndcg@10 on 16 datasets of the beir benchmark with the varying loss functions. d complete results on beir table 6 presents the results obtained after distilling the models using soft labels from ms marco and beir. we can observe the impact of both proposed distillation steps, namely using soft human labels and soft synthetic labels, which bring significant effectiveness improvements over the base models. in particular, using logits from ms marco leads to an average of a 2-point ndcg@10 improvement for each model, while the subsequent fine-tuning phase with the synthetic beir queries further enhances their effectiveness by 4.5 points for t5-small and approximately 1.4 points for t5-base. table 6. ndcg@10 values for each dataset after two steps of distillation.full size table e complete results on quati table 7 shows the results obtained by fine-tuning ptt5-v1 and ptt5-v2 [27] using different training sets. in particular, ptt5-v2 has a better overall ndgg@10, showing that even though both versions have the same number of parameters, a better pre-training process can improve downstream tasks such as information retrieval. for the quati soft labels generation we used the mt5-3.7b as a teacher instead of t5-3b, since this model has a better performance on portuguese text. table 7. complete results (ndcg@10) on quati. all training datasets were used to distill knowledge from a teacher model.full size table rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper laitz, t.s., papakostas, k., lotufo, r., nogueira, r. (2025). inranker: distilled rankers for zero-shot information retrieval. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_10 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning language models information retrieval publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature hierarchical graph convolutional networks for image classification | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper hierarchical graph convolutional networks for image classification conference paper first online: 12 october 2023 pp 63–76 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) hierarchical graph convolutional networks for image classification download book pdf download book epub joão pedro oliveira batisteli  orcid: orcid.org/0009-0001-7655-95789, silvio jamil ferzoli guimarães  orcid: orcid.org/0000-0001-8522-20569 & zenilton kleber gonçalves do patrocínio júnior  orcid: orcid.org/0000-0003-0804-17909  part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 562 accesses 5 citations abstract graph-based image representation is a promising research direction that can capture the structure and semantics of images. however, existing methods for converting images to graphs often fail to preserve the hierarchical information of the image elements and produce sub-optimal or poor regions. to address these limitations, we propose a novel approach that uses a hierarchical image segmentation technique to generate graphs at multiple segmentation scales, capturing the hierarchical relationships between image elements. we also propose and train a hierarchical graph convolutional network for image classification (hgcic) model that leverages the hierarchical information with three different adjacency setups on the cifar-10 database. experimental results show that the proposed approach can achieve competitive or superior performance compared to other state-of-the-art methods while using smaller graphs. code is available at: https://github.com/imscience-ppginf-pucminas/hgcic. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others hierarchical image segmentation based on multi-feature fusion and graph cut optimization chapter © 2018 image classification using graph-based representations and graph neural networks chapter © 2021 graph approach in image segmentation chapter © 2018 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computer vision graph theory graph theory in probability graphemics image processing visual sociology 1 introduction graph-based image representation is an emerging research area that leverages the spatial relationships between image elements to model image content more effectively [16]. graph-based approaches can enhance the understanding of image semantics and context by incorporating domain-specific knowledge into the learning process. moreover, these approaches can provide multi-scale representations of the same image [17, 22, 23], capturing local and global information about its structure. these advantages make graph-based image representation appealing for various image analysis tasks. applying machine learning algorithms to graph data presents unique challenges due to its irregularity, variable sizes, and diverse neighbor relationships [24]. graph neural networks (gnns) have emerged to address these challenges, adapting neural network architectures to process graph-structured data effectively. gnns capture the intricate relationships and dependencies between vertices in a graph, leveraging information from neighboring vertices and edges to encode local and global structural information. this approach enables accurate predictions and improved generalization, making gnns ideal for tackling various graph-based machine-learning problems. one of the main applications of gnns is graph-based image analysis, which requires representing images as graphs. however, this is a challenging task. one common possibility in literature is to apply image segmentation methods for partitioning the image into regions and representing each region by a vertex in a graph [2, 9, 17, 19, 22, 23]. despite the simplicity of this approach, it has two important drawbacks: (i) the dependency on the quality and quantity of the regions produced by the segmentation algorithm; and (ii) the hierarchical relationship between image elements are not captured. here we argue that this property is essential for effective image reasoning, which a graph representation must capture. it is worth mentioning that hierarchical information could be seen as a multi-scale representation. figure 1 illustrates failures in image classification when the hierarchical structure is disregarded (all cases were classified as airplanes, using [2] representation). fig. 1. examples of images classified as airplanes, according to [2]. full size image to overcome the limitations of existing methods, we propose a novel approach that leverages hierarchical segmentation techniques to generate graph vertices for image classification from graph representation. hierarchical segmentation is a well-established technique in computer vision and image processing [6, 7, 13], enabling the identification of objects and regions within an image based on their visual characteristics. additionally, our method allows incorporating hierarchical relationships between image elements into the resulting graph, facilitating image analysis tasks. more specifically, the idea of hierarchical segmentation is similar to generating a set of image segmentations at different levels of detail with respect to the principles of multi-scale image analysis [12]. by adopting this approach, we argue that we can effectively capture the rich structural information in the image and encode it in the resulting graph. thus, our proposal for image classification uses a hierarchical segmentation method to capture hierarchical information for representing image data features. however, in some cases, while working on raw images, noise may produce poor results in image segmentation, for that, in this work, instead of using raw images, we generate superpixels from them to produce homogeneous and concise regions in conjunction with good object delineation. from these superpixels, region adjacency graphs are computed for representing superpixel images. to evaluate the effectiveness of our proposed representation, we conducted experiments using images from the cifar-10 dataset, leveraging our hierarchical segmentation technique from the superpixel images to generate graphs. we also introduce a novel model, the hierarchical graph convolutional network for image classification (hgcic), that can effectively learn from the hierarchical graphs and classify images. experiments demonstrated promising results, outperforming some state-of-the-art methods, highlighting the model and the graph representation effectiveness. furthermore, the graphs used to train our model have a much smaller number of edges and vertices than those utilized in other works, reducing the computational complexity of gnns and speeding up the learning process. we can briefly describe the two major contributions of this work to graph-based image analysis: (i) the proposition of a novel graph representation method that leverages hierarchical image segmentation to capture hierarchical representations of the underlying image structure; and (ii) the introduction of a novel graph convolutional network (gcn) architecture that can extract and use the essential information from our new graph representation. this work is organized as follows. section 2 presents some related works. section 3 describes the most important concepts needed for understanding the proposed method. section 4 presents a hierarchical graph convolutional network for classifying images based on superpixel graphs. section 5 describes the experiments and some comparative analysis of the proposed approach to the state-of-the-art methods. and, finally, in sect. 6, we have drawn some conclusions and present some further work. 2 related works superpixel image segmentation could be seen as a fundamental task in image processing that divides an image into homogeneous regions. this process can reduce the complexity of an image and provide a more efficient representation for further analysis. by treating these regions or segments as vertices, we can transform the image into a graph structure, which can facilitate the use of graph-based algorithms. a common strategy in literature for representing images as graphs is based on considering vertices as superpixels [2, 9, 11, 17, 20, 22]. superpixels are groups of pixels into perceptually meaningful regions based on similarity criteria like color or location. this approach leverages the flexibility of graph neural networks, which can handle irregular graphs with different shapes and sizes. other methods include splitting the images into fixed patches viewed as vertices [14] or interpreting each pixel as a vertex [8, 20]. some works also explore multiscale graph (from image) representations to achieve better graph classification performance. multiscale graph-image representations can capture different levels of details and features from the image. for instance, in [22], the authors proposed a novel superpixel algorithm that produces segments with a wide size distribution, allowing a more flexible representation of an image, as it can capture fine and coarse details. the authors in [23] model hyperspectral images as multiple graphs with different neighborhood scales and propose a dynamic graph convolution operation that updates the similarity measures among vertices by fusing feature embeddings. to the best of our knowledge, the work in [17] is the only one that has used a multiscale graph segmentation, in which superpixels are obtained at several scales and different types of relations between vertices are explored to improve the model expressiveness. the edges of the graph are also crucial for constructing the graph representation, as they enable the message-passing mechanism of gnns. the message-passing mechanism updates the feature representations of vertices by exchanging information with neighboring vertices through edges. one natural way to build adjacency is using a region adjacency graph (rag), in which edges are created between spatial neighbors [2, 22]. another strategy is constructing k-nearest adjacency based on the vertex’s spatial and/or feature distances, as used in [9, 14]. both methods have shown promising results, and their choice mainly depends on the problem requirements and the properties of the data. 3 theoretical background 3.1 graph neural networks graph neural networks (gnns) can be divided into two types: spatial or spectral. spatial gnns work directly on the graph structure and compute vertex (and maybe edge) representations using information from their neighbors. spectral gnns, on the other hand, function on the graph spectral domain. they utilize the eigenvectors of the graph laplacian matrix as the foundation for representing the graph data. this work focuses solely on spatial gnns. the input of each gnn layer are the vertex feature vectors \( \{h_{u}\in \mathbb {r}^{d}\;|\; u \in \mathcal {v}\}\), the set of edges \(\mathcal {e}\), and optionally edge feature vectors (maybe seen as weights) \( \{w_{uv}\in \mathbb {r}^{d}\;|\; (u,v) \in \mathcal {e}\}\). the result of each layer is a new vertex representation \(\{h_{u}^{'}\in \mathbb {r}^{d^{'}}\;|\; u \in \mathcal {v}\}\), in which the same parametric function is applied to each vertex given its neighbors \(\mathcal {n}_{u}=\{v \in \mathcal {v}\;|\; (u,v)\in \mathcal {e})\}\) and on the edges incident to it, generically given by: $$\begin{aligned} h'_{u}=f_{\theta }\left( h_{u}, aggregate(h_{v}, w'_{uv}\;|\;v\in \mathcal {n}_{u})\right) \end{aligned}$$ (1) in which aggregate is a permutation invariant function (like max, min, sum), \(f_{\theta }\) is the parametric function, and \(w'_{uv}\) is the updated edge feature defined by: $$\begin{aligned} w'_{uv} = g_{\theta }(h_{u}, h_{v}, w_{uv}) \end{aligned}$$ (2) in which \(g_{\theta }\) is a distinct parametric function. each vertex update step is also called the message passing step since vertices send information to their neighbors. 3.2 residual gated graph convolutional network according to [4], gatedgcn is a fusion of the vanilla gcn and edge gating mechanism. in [9], the authors suggested modifications to the gatedgcn architecture by introducing residual connections and batch normalization [15]. the vertex update is given by: $$\begin{aligned} h'_{u} = h_{u} + \textrm{relu}(\textrm{bn}(u^{\ell }h_{u} + \sum _{v \in n_{v}}\alpha _{uv}\odot v^{\ell }h_{v})) \end{aligned}$$ (3) in which \(u^{\ell }\), \(v^{\ell }\) are linear transformations, \(\odot \) denotes hadamard product, \(\textrm{relu}\) stands for rectified linear unit, \(\textrm{bn}\) represents batch normalization, and \(\alpha _{uv}\) are the edge gates defined by: $$\begin{aligned} \alpha _{uv} = \frac{\sigma (\hat{w}_{uv})}{\sum _{{v}' \in n_{u}}\sigma (\hat{w}_{u{v}'}) + \varepsilon } \end{aligned}$$ (4) $$\begin{aligned} \hat{w}_{uv} = a^{\ell }h_{u}^{\ell }+b^{\ell }h_{v}^{\ell } + c^{\ell } {w}_{uv} \end{aligned}$$ (5) in which \(a^{\ell }\), \(b^{\ell }\), \(c^{\ell }\) are linear transformations, \(\sigma \) is the sigmoid function, and \(\varepsilon \) is a small-fixed constant for numerical stability. the edge gate in eq. 4 works as a soft attention mechanism [9], allowing the model to learn the importance of different vertices in a neighborhood. finally, the edge features are updated as follows: $$\begin{aligned} {w}^{'}_{uv} = {w}_{uv} + \textrm{relu}(\textrm{bn}(\hat{w}_{uv})) \end{aligned}$$ (6) 3.3 hierarchical segmentation hierarchical image segmentation is a set of image segmentations at different detail levels [13]. the segmentations with lower levels of detail can be created by merging regions from segmentations at higher levels of detail. hierarchical approaches must obey the principles of multi-scale image analysis. these principles ensure that the segmentation is consistent across different levels of detail. the causality principle defines that a contour presented at a scale \(k_{1}\) should be present at any scale \(k_{2} < k_{1}\). the location principle defines that contours should be stable because they neither move nor deform from one scale to another [12]. hierarchical image segmentation organizes image segments into a tree structure where each vertex represents a different level of detail or abstraction. the highest level of the tree represents the entire image, while lower levels correspond to smaller and more specific sub-regions or sub-segments. this structure provides a way to represent the image at different levels of resolution, allowing for a better understanding of the image’s contents. fig. 2. pipeline for computing a hierarchy from the original image. full size image 4 hierarchical graph convolutional networks by using hierarchy of superpixels given a finite set v, a partition of v is a set \(\textbf{p}\) of nonempty disjoint subsets of v whose union is v. any element of \(\textbf{p}\), denoted by \(\textbf{r}\), is called a region of \(\textbf{p}\). given two partitions \(\textbf{p}\) and \(\textbf{p}^\prime \) of v, \(\textbf{p}^\prime \) is said to be a (total) refinement of \(\textbf{p}\), denoted by \(\textbf{p}^{\prime } \preceq \textbf{p}\), if any region of \(\textbf{p}^\prime \) is included in a region of \(\textbf{p}\). let \(\mathcal {h}= (\textbf{p}_1, \dots , \textbf{p}_\ell )\) be a set of \(\ell \) partitions on v. \(\mathcal {h}\) is a hierarchy if \(\textbf{p}_{i-1} \preceq \textbf{p}_i\), for any \(i \in \{ 2, \dots , \ell \}\). 4.1 graph construction let \(g=(v,e)\) be a rag computed from the superpixels in which the set v represents the superpixels and the set e the adjacency relation between the superpixels. let \(\mathcal {h}= (\textbf{p}_1, \dots , \textbf{p}_\ell )\) be a hierarchy computed from the graph g. let \(\mathcal {r}_j\) be the set of regions in the partition \(\textbf{p}_j\) of the hierarchy \(\mathcal {h}\). let \(\mathcal {r}\) be set containing all regions belonging to all partition \(\textbf{p}_j \in \mathcal {h}\). figure 2 illustrates our proposal for computing the hierarchy from the original image. in the following, we describe how to compute three different graphs from a given hierarchy. these graphs will be used as input in the learning step of our method. hierarchy-based graph. this graph, denoted by \(g_h = (v_h,e_h)\), is a graph computed from the hierarchical structure \(\mathcal {h}\) in which the set of vertices \(v_h\) is equal to the \(\mathcal {r}\). the set of edges \(e_h\) is defined by \(e_h=\{(r_i,r_j),(r_j,r_i)~|~r_i, r_j \in \mathcal {r},~r_i \ne ~r_j,~\textrm{in which}~ r_j ~\textrm{is the smallest region that contains}~ r_i \}\). knn-based graph. this graph, denoted by \(g_k = (v_k,e_k)\), is a graph computed from the k-nearest neighborhood of regions in the hierarchical structure \(\mathcal {h}\) in the feature space, in which the set of vertices \(v_k\) is equal to the \(\mathcal {r}\). let \(w(v) = \{w(v),~\forall v \in v\}\) be the set of feature vectors related to the vertex set v. the set of edges \(e_k\) is defined by \(e_k=\{(r_i,r_j)\;|\;w(r_j)~\mathrm{is one of the k-nn of~} w(r_i)\}\). complete-based graph. this graph, denoted by \(g_c = (v_c,e_c)\), is a graph computed from hierarchical structure \(\mathcal {h}\), in which the set of vertices \(v_c\) is equal to the \(\mathcal {r}\). let \(w(v) = \{w(v),~\forall v \in v\}\) be the set of feature vectors related to the vertex set v. the set of edges \(e_c\) is defined by \(e_c=\{(r_i,r_j)~|~r_i, r_j \in \mathcal {r},~r_i \ne ~r_j\}\). it is important to mention that each region is represented by the following set of features: color (color channels mean and 2-bin color histogram), texture (contrast, dissimilarity, homogeneity, energy, correlation, and angular second moment), region (orientation, bounding box area, solidity, area, eccentricity, convex area, perimeter, mean intensity, euler number, and hu moments), position (x and y mean position) and the vertex altitude in the segmentation tree. the extracted features were then used to create a vertice feature vector \(h_{u}\in \mathbb {r}^{104\times 1}\) for each \(u \in {v}\). 4.2 architecture figure 3 shows the proposed gcn architecture for the image classification task. to embed the input edge and vertices features, two linear layers were applied to produce d-dimensional embeddings. the dimension of the edge and vertices embeddings remained the same across all layers. inspired by [5], this work adopted \(\mathcal {m}+1\) convolutions, in which \(\mathcal {m}\) is chosen at inference time. all \(\mathcal {m}\) convolutional layers share the same weights, which improved the results and made the proposed architecture parameter-efficient [5]. fig. 3. model architecture. the input edges and vertices features are h and w, respectively. full size image an adaptive architecture that adjusts its depth can capture important features and patterns that lead to better accuracy and performance. however, balancing model capacity and complexity is crucial to avoid sub-optimal results. if the model is too shallow, it may not be able to capture complex patterns, while a model that is too deep may suffer from overfitting or be computationally expensive. after the graph convolution layers, we employ a readout layer that generates a fixed-size vector representation from the graph features. the output of the readout layer is then fed into a multi-layer perceptron (mlp) that learns to make class predictions based on the graph features. by combining the adaptive depth graph convolution layers, readout layer, and mlp, the proposed model can effectively extract and learn hierarchical representations of the input graphs, leading to accurate and robust classification results. 5 experimental results to analyze our proposal for image segmentation, we have applied our strategy to a well-known database. we have considered the three different graphs computed from the hierarchy. also, we have trained all the models for 1,000 epochs with an initial learning rate of \(10^{-3}\), which is reduced by half if the validation accuracy does not improve after ten epochs until reaching a stopping learning rate of \(10^{-5}\), cross entropy loss, batch size of 64 and adam optimizer with \( \beta _{1} = 0.9 \), \(\beta _{2} = 0.98\), and \(\epsilon = 10^{-9} \). we saved the weights on the epoch with the best validation accuracy. we evaluated the proposed model in the cifar-10 database [18], comprising 60,000 32 \(\times \) 32 color images across ten classes, with 6,000 images each. the database is split into 45,000 training images, 5,000 validation images, and 10,000 test images. it is important to observe that we have followed the procedure described in [9] and randomly sampled 5,000 images from the training set for validation. the same splits were used for all experiments. 5.1 implementation details graph construction. we adopt the slic [1] as the superpixel segmentation method since it is simple, fast, and memory efficient and to make a fair comparison since almost all work in our comparative analysis uses it. the target number of superpixels is typically 20 but may vary for each image. we have used the watershed by area [6] as the hierarchical segmentation method, which outperformed other methods based on different attributes in our experiments. thus, the hierarchy-based, knn-based, and complete-based are constructed from the hierarchy computed using watershed by area, which is applied to the rag of the superpixels obtained by the slic. the number of nearest neighbors k was set to 8 on the knn-based graph setup. architecture. to implement our model, we use pytorch geometric [10], which batches multiple graphs into a single graph with multiple subgraphs for mini-batch training. following [5], we could set the variable depth \(\mathcal {m}\) as half of the number of vertices in each graph. however, this is not feasible for graphs with different sizes in the same batch. instead, we use \(\mathcal {m} = \left\lfloor max(|\mathcal {v}|_{minibatch}) /2 \right\rfloor \), in which \(max(|\mathcal {v}|_{minibatch})\) is the maximum number of vertices among the graphs in the batch. we use gatedgcn as the graph convolution since it preserves and updates the edge features \(w_{uv}\) between vertices u and v at each layer [9], and the soft attention mechanism in eq. 5 enables the model to learn how important each neighbor v is for vertex u. furthermore, we changed eqs. 3–6 by replacing batch normalization with layer normalization and adding nonlinearity and normalization before calculating the attention coefficients, resulting in eqs. 7–10. $$\begin{aligned} h'_{u} = h_{u} + \textrm{relu}(\textrm{ln}(u^{\ell }h_{u} + \sum _{v \in n_{v}}\alpha _{uv}\odot v^{\ell }h_{v})) \end{aligned}$$ (7) $$\begin{aligned} \alpha _{uv} = \frac{\sigma (\hat{w}_{uv})}{\sum _{{v}' \in n_{u}}\sigma (\hat{w}_{u{v}'}) + \varepsilon } \end{aligned}$$ (8) $$\begin{aligned} \hat{w}_{uv} = \textrm{relu}(\textrm{ln}(a^{\ell }h_{u}^{\ell }+b^{\ell }h_{v}^{\ell } + c^{\ell } {w}_{uv})) \end{aligned}$$ (9) $$\begin{aligned} {w}^{'}_{uv} = {w}_{uv} + \textrm{relu}(\textrm{ln}(\hat{w}_{uv})) \end{aligned}$$ (10) distinct from other gcn models, this work used a readout layer that concatenates the global mean and the max pooling, capturing the average and maximum values of the vertex features from the entire graph. combining two permutation invariant functions was motivated by the better results obtained in the initial experiments compared to using only one. it is worth mentioning that the mlp includes two linear layers with layer normalization that help improves the model’s training stability and generalization performance. table 1 shows the details of each layer in our proposed architecture. table 1. hgsic architecture detailsfull size table table 2. accuracy of the proposed model and state-of-art methods in the cifar-10 dataset. * means the target number of nodes since the authors do not report the average value, and the cells with – refer to data that have not been reported.full size table 5.2 quantitative analysis table 2 shows the results for the hgcic model and other state-of-the-art methods. the hgcic model was trained using three different graphs: knn-based (hgcic\(_{k}\)), hierarchy-based (hgcic\(_{h}\)), and complete-based (hgcic\(_{c}\)). interestingly, the hgcic\(_{h}\) model, based on the hierarchy structure, has a much smaller number of edges and outperforms the other two graphs. this result is noteworthy since the number of edges in a graph directly impacts the message-passing step of graph neural networks, which are designed to learn from the graph’s structural information. however, our findings suggest that the relationships captured by edges in the graph are more critical than the graph’s number of edges. this is consistent with prior work showing that incorporating hierarchical relationships between vertices and edges can help improve gnn performance [17]. overall, test results demonstrate the effectiveness of the proposed hierarchical adjacency method in enhancing our model’s performance. despite the superior performance of the hgcic\(_{h}\) model, our work did not achieve the best accuracy in image classification. however, we achieved a competitive result without resorting to complex strategies that other methods used, such as recurrent neural networks [5], multiple relations [17], positional embeddings [21], or vector fields to define the directions of information propagation [3]. these techniques could also be incorporated into our model in the future to boost its performance further. another factor that could affect our results is the size of our graphs, which were the smallest among the compared methods. the only model with fewer parameters than ours was the one proposed by [2] but with a lower result compared to the proposed method. therefore, our work demonstrates a promising approach for graph-based image analysis and shows the potential of hierarchical segmentation for creating effective graph representations of images. a drawback of the methods proposed in [17] is that they require computing the eigenvalues and eigenvectors of the graph laplacian matrix. this step is essential for learning filters that depend on the laplacian eigenbasis and capture the graph structural information, but it can be costly. furthermore, since the laplacian eigenbasis is specific to each graph, models trained on one graph may not generalize well to others. this limitation can restrict the scalability and applicability of these methods to a broader range of graph data. using spatial gnns, our model can capture the geometric features of the graph without relying on the laplacian eigenbasis and avoid the problem of generalization to new graphs. table 3. examples of predictions of the proposed models. images have been enlarged for easy viewing.full size table 5.3 qualitative analysis in table 3, we present the qualitative results of our models. we observe that the model trained with the hierarchy-based adjacency can classify not only simple images but also images that are challenging for humans to classify due to their low resolution (only \(32\times 32\)). 5.4 ablation study we conducted an ablation study to assess the performance impact of various components in our proposed method. we compared our proposed architecture with a baseline model based on [9], which has four gatedgcn layers, a global average pooling for the readout layer, and an mlp similar to ours, but without normalization as in the original paper. additionally, we tested the proposed architecture and graph creation method with a simple feature vector as used in [2, 9, 17], which consists of the concatenation of the average value for each color channel and the geometric centroid, to show that a more expressive feature set can enhance the performance of the task. table 4 shows the result of our ablation study. all modified models showed degradation in accuracy compared to their originals, proving the benefits of both the proposed architecture and used features. table 4. performance of the ablation study to assess how changes in the proposed method affect performance. hgcic\(_{\bullet }^{5f}\) are the models with the proposed architecture trained with the simpler 5d feature vector, and g4gcn\(_{\bullet }\) are trained with the full set of features and four gatedgcn layers.full size table out of all the models tested, the ones using architecture proposed in [9] showed the most significant decrease in accuracy. this was due to the limited capacity of the architecture to learn complex features from the data. specifically, the four layers in this architecture were deemed insufficient for the graph-learned embeddings to converge, resulting in poor representations in the final gnn layer. among the models trained with the simpler 5d feature vector, the one with the highest accuracy was the model trained with complete graph adjacency. the superior performance of this model was because the distance between vertices is not a reliable indicator of segment differences. as a result, the model gave similar weightage to all neighbors in the message-passing, consequently prioritizing more neighbors to perform the feature aggregation. similar behavior also occurred in experiments with the [9] architecture, where the hierarchy adjacency caused relevant vertices to be farther away from others. again, the four layers are insufficient for the information to propagate throughout the graph. 6 conclusion this work introduces a new approach for constructing graphs from images using hierarchical segmentation methods. additionally, it presents a new model called hgcic, which has been trained using graphs obtained from hierarchical segmentation in three different adjacency setups. the proposed model has demonstrated remarkable results by incorporating variable depth, hierarchical relationships through edges, and well-defined features. the implications of this research are exciting and could have a far-reaching impact on various applications that rely on graph-image analysis. although the results were slightly inferior, they still showed promise. the proposed approach utilizes significantly smaller graphs than those in previous works, and the proposed gcn architecture contains fewer parameters yet still delivers promising results. moreover, the ablative study confirmed the hypothesis that the choice of architecture and features positively impacted the model’s overall performance. these findings highlight the potential of the proposed approach as a more efficient and effective means of graph construction and analysis. in future works, we plan to investigate the impact of attention mechanisms on our approach. additionally, we aim to conduct a more in-depth analysis of the relationship between the number of vertices and model performance while exploring multiple graph relations. references achanta, r., shaji, a., smith, k., lucchi, a., fua, p., süsstrunk, s.: slic superpixels compared to state-of-the-art superpixel methods. ieee trans. pattern anal. mach. intell. 34(11), 2274–2282 (2012) article  google scholar  avelar, p.h., tavares, a.r., da silveira, t.l., jung, c.r., lamb, l.c.: superpixel image classification with graph attention networks. in: 2020 33rd sibgrapi conference on graphics, patterns and images (sibgrapi), pp. 203–209. ieee (2020) google scholar  beaini, d., passaro, s., létourneau, v., hamilton, w., corso, g., liò, p.: directional graph networks. in: international conference on machine learning, pp. 748–758. pmlr (2021) google scholar  bresson, x., laurent, t.: residual gated graph convnets. arxiv preprint arxiv:1711.07553 (2017) corso, g., cavalleri, l., beaini, d., liò, p., veličković, p.: principal neighbourhood aggregation for graph nets. in: advances in neural information processing systems, vol. 33, pp. 13260–13271 (2020) google scholar  cousty, j., najman, l.: incremental algorithm for hierarchical minimum spanning forests and saliency of watershed cuts. in: soille, p., pesaresi, m., ouzounis, g.k. (eds.) ismm 2011. lncs, vol. 6671, pp. 272–283. springer, heidelberg (2011). https://doi.org/10.1007/978-3-642-21569-8_24 chapter  math  google scholar  cousty, j., najman, l., kenmochi, y., guimarães, s.: hierarchical segmentations with graphs: quasi-flat zones, minimum spanning trees, and saliency maps. j. math. imaging vis. 60(4), 479–502 (2018) google scholar  defferrard, m., bresson, x., vandergheynst, p.: convolutional neural networks on graphs with fast localized spectral filtering. in: advances in neural information processing systems, vol. 29 (2016) google scholar  dwivedi, v.p., joshi, c.k., luu, a.t., laurent, t., bengio, y., bresson, x.: benchmarking graph neural networks. j. mach. learn. res. 24(43), 1–48 (2023) mathscinet  google scholar  fey, m., lenssen, j.e.: fast graph representation learning with pytorch geometric. in: iclr workshop on representation learning on graphs and manifolds (2019) google scholar  fey, m., lenssen, j.e., weichert, f., müller, h.: splinecnn: fast geometric deep learning with continuous b-spline kernels. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 869–877 (2018) google scholar  guigues, l., cocquerez, j.p., le men, h.: scale-sets image analysis. int. j. comput. vision 68(3), 289–317 (2006) article  math  google scholar  guimarães, s., kenmochi, y., cousty, j., patrocinio, z., najman, l.: hierarchizing graph-based image segmentation algorithms relying on region dissimilarity. math. morphol.-theory appl. 2(1), 55–75 (2017) math  google scholar  han, k., wang, y., guo, j., tang, y., wu, e.: vision gnn: an image is worth graph of nodes. in: neurips (2022) google scholar  ioffe, s., szegedy, c.: batch normalization: accelerating deep network training by reducing internal covariate shift. in: international conference on machine learning, pp. 448–456. pmlr (2015) google scholar  johnson, j., et al.: image retrieval using scene graphs. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 3668–3678 (2015) google scholar  knyazev, b., lin, x., amer, m., taylor, g.: image classification with hierarchical multigraph networks. in: sidorov, k., hicks, y. (eds.) proceedings of the british machine vision conference (bmvc), pp. 223.1-223.13. bmva press (2019). https://doi.org/10.5244/c.33.223 krizhevsky, a., hinton, g., et al.: learning multiple layers of features from tiny images (2009) google scholar  liu, q., xiao, l., yang, j., wei, z.: cnn-enhanced graph convolutional network with pixel-and superpixel-level feature fusion for hyperspectral image classification. ieee trans. geosci. remote sens. 59(10), 8657–8671 (2020) article  google scholar  monti, f., boscaini, d., masci, j., rodola, e., svoboda, j., bronstein, m.m.: geometric deep learning on graphs and manifolds using mixture model cnns. in: proceedings of the ieee conference on computer vision and pattern recognition, pp. 5115–5124 (2017) google scholar  rampášek, l., galkin, m., dwivedi, v.p., luu, a.t., wolf, g., beaini, d.: recipe for a general, powerful, scalable graph transformer. in: advances in neural information processing systems, vol. 35, pp. 14501–14515 (2022) google scholar  vasudevan, v., bassenne, m., islam, m.t., xing, l.: image classification using graph neural network and multiscale wavelet superpixels. pattern recogn. lett. (2023) google scholar  wan, s., gong, c., zhong, p., du, b., zhang, l., yang, j.: multiscale dynamic graph convolutional network for hyperspectral image classification. ieee trans. geosci. remote sens. 58(5), 3162–3177 (2019) article  google scholar  wu, z., pan, s., chen, f., long, g., zhang, c., yu, p.: a comprehensive survey on graph neural networks. ieee trans. neural netw. learn. syst. (2020) google scholar  download references acknowledgements the authors would like to thank conselho nacional de desenvolvimento científico e tecnológico cnpq (universal 407242/2021-0 and pq 306573/2022-9), and fundação de amparo à pesquisa do estado de minas gerais fapemig (grants ppm00006-18). this study was also financed in part by puc minas and by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations image and multimedia data science laboratory (imscience), pontifícia universidade católica de minas gerais (puc minas) belo horizonte, minas gerais, brazil joão pedro oliveira batisteli, silvio jamil ferzoli guimarães & zenilton kleber gonçalves do patrocínio júnior authors joão pedro oliveira batisteliview author publications search author on:pubmed google scholar silvio jamil ferzoli guimarãesview author publications search author on:pubmed google scholar zenilton kleber gonçalves do patrocínio júniorview author publications search author on:pubmed google scholar corresponding author correspondence to joão pedro oliveira batisteli . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper batisteli, j.p.o., guimarães, s.j.f., do patrocínio júnior, z.k.g. (2023). hierarchical graph convolutional networks for image classification. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_5 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_5 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning graph neural networks image classification publish with us policies and ethics profiles silvio jamil ferzoli guimarães view author profile zenilton kleber gonçalves do patrocínio júnior view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature mat-tree: a tree-based method for multiple aspect trajectory clustering | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper mat-tree: a tree-based method for multiple aspect trajectory clustering conference paper first online: 12 october 2023 pp 464–478 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) mat-tree: a tree-based method for multiple aspect trajectory clustering download book pdf download book epub yuri santos  orcid: orcid.org/0000-0001-9371-427x9,10, ricardo giuliani9, vania bogorny  orcid: orcid.org/0000-0002-0159-464310, mateus grellert  orcid: orcid.org/0000-0003-0600-70549,10 & … jônata tyska carvalho  orcid: orcid.org/0000-0001-9020-20769,10  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 570 accesses 1 citation abstract multiple aspect trajectory is a relevant concept that enables mining interesting patterns and behaviors of moving objects for different applications. this new way of looking at trajectories includes a semantic dimension, which presents the notion of aspects that are relevant facts of the real world that add more meaning to spatio-temporal data. given the inherent complexity of this new type of data, the development of new data mining methods is needed. despite some works have already focused on multiple aspect trajectory classification, few have focused on clustering. although the literature presents several raw trajectory clustering algorithms, they do not deal with the heterogeneity of the semantic dimension. in this paper, we propose a novel hierarchical clustering algorithm for multiple aspect trajectories using a decision tree structure that chooses the best aspect to branch and group the most similar trajectories according to different criteria. we ran experiments using a well-known benchmark dataset extracted from a location-based social network and compared our clustering results with a state-of-the-art clustering approach over different internal and external validation metrics. as a result, we show that the proposed method outperformed the baseline, where it revealed a formation of more cohesive and homogeneous clusters in 88% of the clusters, being five times more precise according to the external metrics. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others mastermovelets: discovering heterogeneous movelets for multiple aspect trajectory classification article 30 january 2020 a method for summarizing trajectories with multiple aspects chapter © 2022 pmdtc: progressive multi-task deep trajectory clustering article 08 november 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. categorization data mining functional clustering multivariate analysis sphragistics theory and algorithms for application domains 1 introduction mining hidden and useful patterns from data is an activity that has been strongly performed and researched for at least three decades [3]. the development of techniques for analyzing trajectory data is growing rapidly [8, 24, 27], leveraged by the popularization of gps-equipped devices which is allowing the collection of spatial data from moving objects in unprecedented volumes. moreover, the advancement of the internet of things has allowed the extraction of numerous data other than spatio-temporal called aspects. the aspects contribute to enriching the semantic dimensions of mobility data. furthermore, the enriched dimension can be associated with a moving object, the entire trajectory, or a single trajectory point. besides, this enriched dimension can contain any data format, from simple labels to sophisticated objects. for example, a certain point of a trajectory that is found in a restaurant may contain, in addition to the aspect that defines the type of place visited, aspects such as price range, user evaluation, and opening hours, among others. this complex type of trajectory is called multiple aspect trajectory (mat) [13]. regarding the mobility data domain, several data mining techniques have been proposed for trajectory clustering [2, 7, 16, 20, 30, 32]. such technique allows us to extract patterns [15], and detect common and outlying moving objects behaviors [14]. furthermore, clustering can be used to find how moving objects are similar to each other with respect to their spatio-temporal trajectories, or trajectories of the same user are similar to each other [9]. some clustering applications can include carpooling services based on common trajectories (e.g., uber), profiling of users for transportation and route planning, and collaborative filtering and other association rules techniques to find which locations are often associated by their trajectories [18]. trajectory clustering methods for space [7], spatio-temporal [31], or semantic dimensions [10] have achieved very solid results so far. meanwhile, few works have focusing on clustering mats [25], which is a challenging task since it requires the capability of dealing with heterogeneous data and an even bigger data volume. this means that there are opportunities for the development of new techniques capable of grouping and describing multiple semantic aspects together. thus, it is important that clustering methods for mats adopt multidimensional similarity metrics or other strategies that allow capturing such data heterogeneity [29]. we propose a divisive hierarchical clustering algorithm that considers the three dimensions of trajectories (space, time, and semantics) by using a decision tree-based approach. regarding the frequency at which each of the aspects appears in the trajectories of a given dataset, the proposal seeks to iteratively select the ideal aspect to divide the set of trajectories in two by using a threshold (mean or median of aspect frequency). it leads to the formation of hierarchical trajectory clusters that are naturally more similar to each other, i.e., clusters that present a higher frequency for certain aspects and a lower frequency for others. it is noteworthy that since the proposed method uses a tree approach, there is a hierarchy between the formed clusters. therefore, in the bottom of the tree, at the leaf nodes, we have more specific and detailed clusters, while closer to the top of the tree, clusters have more general information, being the root node the original dataset. thus, we seek to provide a new method that supports answering questions related to trajectory similarities regarding different aspects and at different levels of abstraction. for validation purposes, the proposed method is tested on the foursquare dataset [29], the same data used by state-of-the-art for mat clustering [25]. this dataset contains user trajectories (spatio-temporal) enriched with other characteristics such as: the type of place visited, rating, price tier, and weather condition. we use internal and external validation metrics to evaluate clustering results. the experimental result shows that our proposal is quantitatively and qualitatively better than the previous state-of-the-art method [25]. the clusters generated by our method were 88% more cohesive and more separable than the baseline method [25]. regarding the external validation metrics, the proposed method was five times more precise than the baseline. from the qualitative point of view, our proposal provides different options for clustering and visualizations, being a valuable tool for data analysts performing exploratory analysis on multiple aspect trajectory data. the remainder of this paper is organized as follows. in sect. 2, we present the basic concepts of multiple aspect trajectories and related works. in sect. 3 we describe the proposed approach to cluster mat by decision trees. in sect. 4 we discuss the experimental results and evaluation of the proposed method. finally, the conclusion and further research directions of our work are presented in sect. 5. 2 basic concepts and related works in this section we present the basic concepts to guide the reader throughout this paper and a brief review of mat clustering approaches. 2.1 multiple aspect trajectory multiple aspect trajectories are defined by their three-dimensional nature, i.e., the sequences of points composed by space and time, in addition to the semantic dimension [13]. the concept of semantic dimension is the representation of any context information or relevant meanings that are of fundamental importance for understanding the data obtained in a trajectory [21]. the first approach that brought semantic data enrichment to trajectory data was stops and moves [1]. moves are made of sample locations between stops which could also be at the beginning or end of a trajectory. stops are groups of sample points close in space and time that reflect interesting spatial places known as point of interest (poi). besides, every stop has a beginning and ending time, a spatial position and a minimum duration. more recently, the semantic dimension started to represent the vast set of characteristics that each point of a trajectory can present, which is called aspect, thus bringing the idea of multiple aspect trajectories [13]. an aspect can be described as a real-world fact relevant to the analysis of moving object data. figure 1 illustrates several points on a trajectory that can contain many aspects. it shows the trajectory of a given person where the pois are represented in circles while different data are collect, such as: weather information, heart rate, emotional status while working at the office, the ticket price, the genre of the film and its rating in a cinema session, and a restaurant with aspects representing its reviews, opening hours, price range and the restaurant type. thus, the multiple aspect trajectories can present numerous aspects that enrich the semantic dimension. fig. 1. example of a multiple aspect trajectory. full size image 2.2 related works many works focus on raw trajectory analysis, and clustering due to the high availability of gps tracking devices, enabling the tracking of moving objects such as vehicles, planes, and humans. besides, when using clustering for mat, it may extract useful patterns or detect interesting outliers. nevertheless, the majority of trajectory clustering works in the literature [20, 25, 26, 28, 30, 32] take into account the space or spatio-temporal dimensions, which employ classical clustering approaches (e.g., k-means [2] and dbscan [22]) with adapted measures from these dimensions. in the work of hung et al. [7], the group proposed a framework to explore the spatio dimension by clustering and aggregating clues of trajectories to find routes. nanni and pedreschi [16] proposed a modified density-based clustering algorithm to explore the temporal dimension to improve the quality of trajectory clustering. wang et al. [26] proposed a trajectory clustering method based on hdbscan, which adaptively clusters trajectories with their shape characteristics using the hausdorff distance on the spatio dimension to compute similarities. chen et al. [2] used the dbscan method for clustering trajectories. it first divide the trajectories into a set of subtrajectories considering the spatial dimension, then it computes the similarity between trajectories using the hausdorff distance, and finally the dbscan clustering algorithm is applied for clustering. sun and wang [22] used the dbscan method combined with minimum bounding rectangle and buffer similarity over the ship trajectories using the spatio dimension to improve trajectories clustering and reduce computation time. yuan et al. [31] proposed a density-based clustering algorithm that employs an index tree for spatio-temporal trajectories. it works by partitioning the trajectories into trajectory segments, storing them in the index tree and then the segments are clustered based on the density strategy. yao et al. [30] proposed an rnn-based auto-encoder model to encode the spatio-temporal movement pattern of trajectory to improve similarity computation by learning the low-dimensional representation and then applied the classic k-means algorithm. liu et al. [11] proposed a new clustering method by extending the k-means algorithm with a time layer to take into account both spatio and temporal proprieties for clustering flight trajectories. liu and guo [10] proposed a semantic trajectory clustering method to capture global relationships among trajectories based on community detection from the perspective of the network. xuhao et al. [28] proposed a semantic-based trajectory clustering method for arrival aircraft based on k-means and dbscan. in the work of santos et al. [20], the researches proposed a co-clustering method for mining semantic trajectories by clustering trajectories without to test all their attributes by focusing on high frequent ones. different of the presented works, varlamis et al. [25] proposed a mat similarity measure with hierarchical clustering by including a multi-vector representation of mats that enables performing cluster analysis on them. the vector representation embeds all trajectory dimensions (space, time, and semantic) into a low-dimension vector which allows clustering multiple aspect trajectories. the authors compared their method with other mat similarity measures using traditional clustering methods and the results outperformed the baseline methods. note that such comparison is performed given the gaps in this recent multiple aspect trajectory clustering task. thus, there is a lack of new methods enabling trajectory cluster analysis using semantic dimension with its multiple aspects and the spatio-temporal dimension simultaneously. we aim to contribute to the reduction of this gap in the literature by proposing a novel mat clustering method. 3 the multiple aspect trajectory tree approach in this section we present a new method named mat-tree (multiple aspect trajectory tree) for finding clusters in a multiple aspect trajectory dataset by using a hierarchical strategy. the main idea is that the mat-tree groups similar trajectories in the same cluster based on aspects that occurs frequently. mat-tree aims to identify the most relevant aspects while clustering the trajectories. figure 2 illustrates an example of a tree generated by the clustering algorithm using sankey diagram representation. the vertical bars indicate the clusters generated by the division through the chosen aspect, noting that the leftmost bar represents the complete dataset with all trajectories and the rightmost bars represent the leaf nodes. figure 2 illustrates tree levels with aspect rain as the most relevant to start the splitting where other six aspects contributed to identify four clusters (leaf nodes). this result can support analysts to identify moving patterns, for instance, during raining days there are more people that go to indoor events (e.g. movie theaters and comedies) using public transportation than night parties with stops at restaurants. fig. 2. example of multiple aspect trajectory clustering tree. full size image the detailed description of mat-tree can be seen on algorithm 1. the main parts and characteristics of the algorithm are: (i) the construction of the frequency matrix; (ii) the criterion chosen for splitting the dataset at each node iteration; (iii) the criterion chosen for evaluating the quality of the split tested; (iv) the stop criterion. mat-tree receives as input the multiple aspect trajectories mat of dataset d, the statistical metric that will be used for dividing the trajectories stat_met, the criterion for choosing the aspect asp_criterion, the maximum height desired for the tree max_tl and the minimum number of trajectories for a leaf node min_t. algorithm 1 . mat-tree full size image the first node of the tree is the root and then mat-tree executes all the process recursively. mat-tree starts by creating the tree’s node structure (line 2) that will store the trajectories received as a parameter of the function (line 3). next, both child nodes are initialized empty, called left_node (line 4) and right_node (line 5), which may contain the trajectories resulting from the division process. mat-tree creates the clustering tree by exploring frequency matrices regarding the occurrences of the multiple aspects of each trajectory. it means that, for each new node generated in the tree, the occurrence of each aspect in the trajectories is used to build a frequency matrix for this new node (line 6). table 1 depicts an example of a frequency matrix showing the number of occurrences of four aspects from three mats. table 1. frequency matrix of three trajectories with four aspects.full size table we start by selecting a split criterion that is used to identify which aspect is better for partitioning the set of trajectories and how such partitioning will occur. regarding this partitioning task, mat-tree uses a statistical metric stat_met, e.g. average, to identify the split point. in this scenario, mat-tree computes the average occurrence of each aspect in the set of trajectories (lines 8–11). after an aspect has been chosen for the division (line 24), the trajectories that present a frequency higher than the stat_met in the aspect are grouped together, i.e. node.right_node (lines 31–35), while the trajectories that present a frequency lower than the stat_met are grouped in another cluster, node.left_node (lines 26–30). it is noteworthy that other statistical metrics, such as the median instead of the average, can be used in this step. we propose four evaluation criteria (\(asp\_crit\)) to select the best aspect, namely: (i) binary division (bd), (ii) minimum variance (mv), (iii) maximum variance reduction (mvr) and (iv) maximum variance reduction considering the largest reduction average among all the aspects set (lra). the idea is to provide different approaches that better fit the data distribution and the application at hand. therefore, mat-tree tests the trajectory partition for all aspects according to the chosen evaluation criterion to identify which aspect is the best choice (line 24). we provide details about each aspect evaluation criterion in the following. bd will analyze the absolute difference in the number of trajectories between the subsets formed after partitioning. the aspect that generates the smallest difference value is selected as the best. mv evaluates the mean variance of an aspect in both trajectory partitions, selecting the one with the minimum mean variance. in addition, mvr also considers the mean variance of the aspect in both trajectory partitions. however, it evaluates the variance reduction considering the previous aspect value, that is, the variance of the parent node. the selected aspect using mvr is the one that generates the maximum reduction value compared to the previous node. last but not least, lra is similar to mvr, but instead of considering the max reduction of the aspects individually, lra considers all aspects to compute the average reduction. thus, lra selects the aspect that generates the largest reduction average of variance regarding the whole aspects set. we highlight that the variance represents the dispersion of the aspects trajectory distribution, that is, how far the trajectories are from the mean. considering a low variance, we are interested in more homogeneous trajectory partitions regarding the frequency of the aspects. the proposed hierarchical tree algorithm identifies more detailed clusters regarding the tree depth, i.e., the last levels of the tree characterize clusters with more aspects on its path. it should be noted that, by design, neighboring nodes will have more similar trajectories, while distant nodes will have less similar trajectories. in conclusion, mat-tree stops and returns the generated tree (line 36) when the conditions of the minimum number of trajectories or maximum tree level (or both) are satisfied. thus, these hyperparameters may vary, and they depend on the application domain and the data distribution at hand. 4 experimental evaluation we carried out the experimentsfootnote 1 using the foursquare ny dataset [17] to evaluate the performance of different multiple aspect trajectory clustering of several users. regarding the main hyperparameters, we use the average occurrence of the aspects frequency as the split-point strategy, while the stop criterion with the minimum number of trajectories was set to 25, where it is application-dependent. the minimum number of trajectories needs to be defined empirically for an eventual optimal value. for a fair comparison with the baseline method, we employed the default hyperparameters and set the number of clusters to be equal to the number of different users (i.e. 193 users). moreover, we run every clustering algorithm 10 times and reports the average of internal and external evaluation scores. we performed the experiments in a machine with a processor intel i7-7700 3.6 ghz, 16 gb of memory, and os windows 10 64 bits. petry et al. [17] enriched the original foursquare datase [29] with semantic information to evaluate multiple aspect trajectory similarity measures. the dataset contains the trajectories of 193 different users, who accounted for a total of 66962 points in 3079 trajectories with an average length of 22 points (check-ins) per trajectory and an average of 16 trajectories per user. the aspects contained in the dataset after the enrichment are: i) the geographic coordinates (latitude and longitude, being a numerical attribute); ii) the time (numeric) at which the user checked in; iii) the day of the week (nominal); iv) the point of interest (poi), which can be understood as the name of the establishment such as starbucks or mcdonald’s (nominal); v) the type of poi (nominal), such as coffee house or fast food; vi) check-in category (nominal), called the root type (for example, food, outdoors & recreation); vii) the rating assigned by the user in the application (ordinal) and viii) the weather condition (nominal) at the time of check-in. clustering evaluation is well-known in the literature due to the fact that clustering is an unsupervised method and we do not usually have a ground truth to compare with. however, in the trajectory clustering application, we can assume that trajectories of the same user are likely to belong to the same cluster, as indicated by gonzalez et al. [5] and already used in the state-of-the-art [12]. therefore, the external evaluation of the clustering method is based on this ground truth. for the internal clustering validity metrics, we assume that the best clusters are those that are well separated and compact, as described by rendón et al. [19]. the external (supervised) and internal (unsupervised) clustering validity metrics [6] comprise: i) homogeneity score (external); ii) completeness score (external); iii) v-measure score (external); iv) adjusted mutual info (mut. info) score (external); v) adjusted rand (adj. rand) score (external); vi) fowlkes mallows (fm) score (external); vii) silhouette (s) score (internal); viii) calinski harabaz (ch) score (internal); ix) davies-bouldin (db) index (separation) (internal). we performed the experiments using the most strongly correlated aspects as similar as the experiments conducted by valarmis et al. [25] for comparison purpose. according to this, the aspects used are the weather, day of the week, and poi category. however, mat-tree is very flexible and it allows any other combination of aspects. in addition, the aspect combination also depends on the application and what kind of patterns the analyst want to mine. considering the aspect evaluation criterion, mat-tree builds four trees, one tree for each criteria. the result shows numerical similarities in terms of number of nodes, leaf nodes and height. regarding the number of nodes, on average (rounded-down), the trees generated 388 nodes, while in terms of leaf nodes the average was 194. in addition, the heights of the trees vary between 8 and 14, as shown in table 2. besides, we noted that the trees structure are different among them, it indicates that the method can identify different trajectory behavior depending on the chosen criteria. table 2. the avg number of generated groups (leaf-nodes) and the height of the modeled tree for each aspect selection criterion.full size table figure 3 shows the clustering results from two different aspect selection criterion using the dendogram to visualize them. figure 3(a) shows the tree generated by the mv aspect choice criterion, while fig. 3(b) shows the result using the mvr criterion. comparing both dendograms, it is noted that the clustering on fig. 3(a) captured an atypical behavior of the trajectories as can be seen in the extreme right branch. in this experiment, mat-tree selected the event aspect of the root_type attribute as the root node, and grouped all the trajectories of the users who checked-in for this type of aspect. this group (c1) contains 20 users out of 193 with a total of 25 trajectories. the event aspect occurs 26 times in the universe of 66962 check-ins, which can reveal a trajectory behavior practiced only in this group. fig. 3. clustering results generated from different aspect selection criterion. full size image figure 4 illustrates the exploratory frequency analysis from the aforementioned group c1. the bar graphs show the relative frequency of aspects for root_type and day category, respectively. it means the number of occurrences of aspects in this specific cluster considering the whole set of trajectories. furthermore, it can be seen in fig. 4(a) that the relative frequency for event was equivalent to its absolute frequency. that is, all trajectories that checked in the event aspect belong to the same cluster. regarding the days of the week, fig. 4(b) shows that saturday was the most frequent day in cluster c1. this behavior is expected because the events usually take place on weekends. regarding the other aspects in fig. 4, we noted that there are few points referring to college & university, which may indicate that the majority of users are not part of the university community. on the other hand, the users in this cluster are more related to activities involving art & entertainment and nightlife, with friday and saturday more frequent than the other days. fig. 4. exploratory frequency analysis of cluster c1. full size image we quantitatively evaluate the clustering result by employing internal and external validation metrics to verify the goodness of a clustering structure. furthermore, we used the msm [4] and muitas [17] in mat-tree as similarity metrics to build the similarity matrix between trajectories. we compare trafos [25] results with mat-tree because it is the state-of-the-art for mat clustering, and trafos outperformed the baselines on its experiments. table 3 shows the clustering evaluation results where mat-tree obtained promising results. we highlight the best result in bold, while the second one is underscored for each evaluation metric. it can be seen in table 3 that, in general, the internal and external validation scores for mat-tree are better than trafos, indicating that mat-tree identified better cluster structures. additionally, the best approach for selecting an aspect to partition the set of trajectories was the maximum reduction of variance (mvr) using muitas. table 3. internal and external clustering validation results.full size table trafos tested the hierarchical agglomerative clusteting with single (sl) and average (al) linkage using the binary partition. in addition, trafos evaluated three different variations for each clustering strategy such as: i) the first considers the average similarity (trafos\(_{mean}\)), ii) the second takes the maximum similarity (trafos\(_{max}\)) and iii) the third sets a threshold on the average similarity (trafos\(_{thr}\)). comparing the silhouette scores, trafos obtained −0.94 and −0.95 respectively, while mat-tree obtained −0.284 and −0.289 respectively. it is important to note that the negative value in all cases is due to the use of similarity matrix instead of distance [25]. this means that the trajectories are better clustered together, i.e., clusters are more cohesive than those found in the previous method. regarding external validation, in general, our results are better than the baseline, denoting that the resulting clusters are more homogeneous and complete, as shown in table 2 and table 3. thus, mat-tree obtained a result of 88% better than trafos looking at silhouette and muitas, while for v-measure the mat-tree was five times more precise. 5 conclusion and future works the volume and the variety of the big data era we live in require a high computational power which evidences the necessity of new approaches for analyzing complex data. multiple aspect trajectories bring a lot of opportunities in the data mining domain, where the nature of a sequence, the high dimensionality, heterogeneity, and data volume pose new challenges. even though a number of methods have been proposed for mat classification only a few works focus on mat clustering. regarding this, we proposed mat-tree, a novel method for multiple aspect trajectory clustering based on the frequency of occurrence of the aspects of the trajectories, and using a decision tree-based approach. the proposed method presented a result of 88% better than the baseline considering internal clustering evaluation metrics and five times more precise. the main contribution of mat-tree is a new approach that allows clustering trajectories considering all their dimensions and semantic aspects. it is noteworthy that studies on the semantic dimension of trajectories are recent and tailored. mat-tree results outperformed the state-of-the-art for multiple aspect trajectory clustering, it indicates that mat-tree can identify more cohesive, compact, and connected clusters. furthermore, mat-tree offers different options for clustering and visualizations, providing a flexible tool for exploratory data analysis and applications that can adapt to the task at hand. thus, once clustering is a data mining task that is inherently highly application-dependent and exploratory [23, chapter 1], the flexibility of mat-tree is an important characteristic. as future works, it would be interesting to examine other aspect selection strategies that allow mat-tree to adapt automatically to different applications. the investigation of different split aspect strategies and evaluation criteria is a promising direction because each criterion shows to be suitable for different applications and analyses. we noted that the bd evaluation criterion is well-suited for generating clusters almost of the same size. nonetheless, the mv criterion seems to easily find outliers trajectories (i.e., trajectories presenting aspects that rarely appear on other trajectories). therefore, experiments designed specifically for validating these characteristics and uses of these criteria are desirable. notes 1.https://github.com/bigdata-ufsc/mat_tree. references alvares, l.o., bogorny, v., kuijpers, b., de macedo, j.a.f., moelans, b., vaisman, a.: a model for enriching trajectories with semantic geographical information. in: proceedings of the 15th annual acm international symposium on advances in geographic information systems, pp. 1–8 (2007) google scholar  chen, j., wang, r., liu, l., song, j.: clustering of trajectories based on hausdorff distance. in: 2011 international conference on electronics, communications and control (icecc), pp. 1940–1944 (2011) google scholar  fayyad, u.m., piatetsky-shapiro, g., smyth, p., et al.: knowledge discovery and data mining: towards a unifying framework. in: kdd, vol. 96, pp. 82–88 (1996) google scholar  furtado, a.s., kopanaki, d., alvares, l.o., bogorny, v.: multidimensional similarity measuring for semantic trajectories. trans. gis 20(2), 280–298 (2016) article  google scholar  gonzalez, m.c., hidalgo, c.a., barabasi, a.-l.: understanding individual human mobility patterns. nature 453(7196), 779–782 (2008) article  google scholar  halkidi, m., batistakis, y., vazirgiannis, m.: on clustering validation techniques. j. intell. inf. syst. 17, 107–145 (2001) article  math  google scholar  hung, c.-c., peng, w.-c., lee, w.-c.: clustering and aggregating clues of trajectories for mining trajectory patterns and routes. vldb j. 24, 169–192 (2015) article  google scholar  zheng, k., zheng, y., yuan, n.j., shang, s.: on discovery of gathering patterns from trajectories. in: proceedings of the ieee international conference on data engineering, washington, dc. ieee (2013) google scholar  khoroshevsky, f., lerner, b.: human mobility-pattern discovery and next-place prediction from gps data. in: schwenker, f., scherer, s. (eds.) mprss 2016. lncs (lnai), vol. 10183, pp. 24–35. springer, cham (2017). https://doi.org/10.1007/978-3-319-59259-6_3 chapter  google scholar  liu, c., guo, c.: stccd: semantic trajectory clustering based on community detection in networks. expert syst. appl. 162, 113689 (2020) article  google scholar  liu, g., fan, y., zhang, j., wen, p., lyu, z., yuan, x.: deep flight track clustering based on spatial-temporal distance and denoising auto-encoding. expert syst. appl. 198, 116733 (2022) article  google scholar  may petry, l., leite da silva, c., esuli, a., renso, c., bogorny, v.: marc: a robust method for multiple-aspect trajectory classification via space, time, and semantic embeddings. int. j. geogr. inf. sci. 34(7), 1428–1450 (2020) google scholar  mello, r.d.s., et al.: master: a multiple aspect view on trajectories. trans. gis 23(4), 805–822 (2019) article  google scholar  meng, f., yuan, g., lv, s., wang, z., xia, s.: an overview on trajectory outlier detection. artif. intell. rev. 52, 2437–2456 (2019) article  google scholar  morris, b., trivedi, m.: learning trajectory patterns by clustering: experimental studies and comparative evaluation. in: 2009 ieee conference on computer vision and pattern recognition, pp. 312–319 (2009) google scholar  nanni, m., pedreschi, d.: time-focused clustering of trajectories of moving objects. j. intell. inf. syst. 27(3), 267–290 (2006) article  google scholar  petry, l.m., ferrero, c.a., alvares, l.o., renso, c., bogorny, v.: towards semantic-aware multiple-aspect trajectory similarity measuring. trans. gis 23(5), 960–975 (2019) article  google scholar  poushter, j., et al.: smartphone ownership and internet usage continues to climb in emerging economies. pew res. center 22(1), 1–44 (2016) google scholar  rendón, e., abundez, i., arizmendi, a., quiroz, e.m.: internal versus external cluster validation indexes. int. j. comput. commun. 5(1), 27–34 (2011) google scholar  santos, y., carvalho, j.t., bogorny, v.: ss-ococlus: a contiguous order-aware method for semantic trajectory co-clustering. in: 2022 23rd ieee international conference on mobile data management (mdm), pp. 198–207 (2022) google scholar  spaccapietra, s., parent, c., damiani, m.l., de macedo, j.a., porto, f., vangenot, c.: a conceptual view on trajectories. data knowl. eng. 65(1), 126–146 (2008) article  google scholar  sun, m., wang, j.: an approach of ship trajectory clustering based on minimum bounding rectangle and buffer similarity. in: iop conference series: earth and environmental science, vol. 769 (2021) google scholar  tan, p.n., steinbach, m., kumar, v.: introduction to data mining. pearson, london (2018) math  google scholar  tortelli portela, t., tyska carvalho, j., bogorny, v.: hipermovelets: high-performance movelet extraction for trajectory classification. int. j. geograph. inf. sci. 36(5), 1012–1036 (2022) article  google scholar  varlamis, i., et al.: a novel similarity measure for multiple aspect trajectory clustering. in: proceedings of the 36th annual acm symposium on applied computing, pp. 551–558 (2021) google scholar  wang, l., chen, p., chen, l., mou, j.: ship ais trajectory clustering: an hdbscan-based approach. j. marine sci. eng. 9(6), 566 (2021) article  google scholar  wu, s.x., wu, z., zhu, w., yang, x., li, y.: mining trajectory patterns with point-of-interest and behavior-of-interest. in: 2021 ieee international conference on communications workshops (icc workshops), pp. 1–6. ieee (2021) google scholar  xuhao, g., junfeng, z., zihan, p.: trajectory clustering for arrival aircraft via new trajectory representation. j. syst. eng. electron. 32(2), 473–486 (2021) article  google scholar  yang, d., zhang, d., zheng, v.w., yu, z.: modeling user activity preference by leveraging user spatial temporal characteristics in lbsns. ieee trans. syst. man cybern.: syst. 45(1), 129–142 (2014) article  google scholar  yao, d., zhang, c., zhu, z., huang, j., bi, j.: trajectory clustering via deep representation learning. in: 2017 international joint conference on neural networks (ijcnn), pp. 3880–3887. ieee (2017) google scholar  yuan, g., xia, s., zhang, l., zhou, y., ji, c.: an efficient trajectory-clustering algorithm based on an index tree. trans. inst. meas. control. 34(7), 850–861 (2012) article  google scholar  yuan, g., sun, p., zhao, j., li, d., wang, c.: a review of moving object trajectory clustering algorithms. artif. intell. rev. 47, 123–144 (2017) article  google scholar  download references acknowledgements this work has been partially supported by capes (finance code 001), cnpq, fapesc (project match co-financing of h2020 projects grant 2018tr 1266), and the european union’s horizon 2020 research and innovation programme under ga n. 777695 (master). we also thank the reviewers who contributed to the improvement of this work. the views and opinions expressed in this paper are the sole responsibility of the author and do not necessarily reflect the views of the european commission. author information authors and affiliations universidade federal de santa catarina, florianópolis, brazil yuri santos, ricardo giuliani, mateus grellert & jônata tyska carvalho ine, programa de pós-graduação em ciência da computação, florianópolis, brazil yuri santos, vania bogorny, mateus grellert & jônata tyska carvalho authors yuri santosview author publications search author on:pubmed google scholar ricardo giulianiview author publications search author on:pubmed google scholar vania bogornyview author publications search author on:pubmed google scholar mateus grellertview author publications search author on:pubmed google scholar jônata tyska carvalhoview author publications search author on:pubmed google scholar corresponding author correspondence to yuri santos . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper santos, y., giuliani, r., bogorny, v., grellert, m., carvalho, j.t. (2023). mat-tree: a tree-based method for multiple aspect trajectory clustering. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_30 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_30 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords multiple aspect trajectory trajectory clustering tree approach publish with us policies and ethics profiles yuri santos view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a spin-off version of jason for iot and embedded multi-agent systems | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper a spin-off version of jason for iot and embedded multi-agent systems conference paper first online: 12 october 2023 pp 382–396 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) a spin-off version of jason for iot and embedded multi-agent systems download book pdf download book epub carlos eduardo pantoja  orcid: orcid.org/0000-0002-7099-49749,10, vinicius souza de jesus  orcid: orcid.org/0000-0002-4534-60789, nilson mori lazarin  orcid: orcid.org/0000-0002-4240-39979,10 & … josé viterbo  orcid: orcid.org/0000-0002-0339-66249  show authors part of the book series: lecture notes in computer science ((lnai,volume 14195)) included in the following conference series: brazilian conference on intelligent systems 531 accesses 16 citations 1 altmetric abstract embedded artificial intelligence in iot devices is presented as an option to reduce connectivity dependence, allowing decision-making directly at the edge computing layer. the multi-agent systems (mas) embedded into iot devices enables, in addition to the ability to perceive and act in the environment, new characteristics like pro-activity, deliberation, and collaboration capabilities to these devices. a few new frameworks and extensions enable the construction of agent-based iot devices. however, no framework allows constructing them with hardware control, adaptability, and fault tolerance, besides agents’ communicability and mobility. this work presents an extension of the jason framework for developing embedded mas with bdi agents capable of controlling hardware, communicating, and moving between iot devices capable of dealing with fault tolerance. a case study of an iot solution with a smart home, a monitoring center, and an autonomous vehicle is presented to demonstrate the framework’s applicability. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others agent-based modelling approach for decision making in an iot framework chapter © 2021 iot environment based on multi agent system in smart home chapter © 2024 multiagent system for controlling a digital home connected based on internet of things chapter © 2020 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. adas, connectivity, e/e embedded systems internet of things multiagent systems smart infrastructure agent-based economics 1 introduction the internet of things (iot) promotes the use of devices that sense physical environments in various application domains. however, they generate a considerable amount of raw data stream that needs to be transmitted to be processed [2]. pervasive computing allows the development of distributed cognitive devices capable of extracting relevant information and making decisions directly at the edge computing layer, reducing the dependence on connectivity [10]. one of the fields of distributed artificial intelligence (dai) is multi-agent systems (mas), which are composed of multiple autonomous and proactive entities with decision-making capacity and social abilities that can collaboratively interact to achieve a common goal for the system [28]. in this research area, an integration of hardware and software that allow agents to sense and act in a real-world environment using sensors and actuators is named embedded mas [3]. these systems can also contribute to reduce dependency on connectivity since a cognitive agent embedded in an iot device can process the raw data received from sensors and act immediately, thus accelerating decision-making [9]. one of the most well-known cognitive agent architecture, the belief-desire-intention (bdi) model [7], is based on the knowledge that an agent can have from the environment (perceptions), other agents, or itself [21]. when applied embedded in devices, it provides decision-making at edge level by using perceptions and beliefs captured from the real world and other devices [1, 5, 9]. we performed a mapping review, where we found some works that present a framework [8, 12, 15, 25] or provide some features to construct iot agent-based devices using bdi frameworks, such as: extension to provide interoperable between cyber-physical and iot systems using fuzzy logic [16]; approaches to reconfiguring agent’s goals on the fly [11] and developing mas with iot objects [6], or architecture to ambient intelligence with iot [23]. however, none of these solutions simultaneously meets all the needs of an agent-based iot device, such as the hardware control, the communicability with other mas or devices, the mobility of the agents between different mas or devices, fault tolerance, and adaptability. this paper presents the jason embedded – a spin-off extended version of the jason framework [4] – that provides autonomy, pro-activity, social ability, adaptability, and fault-tolerance to iot devices. it allows programming agents dedicated to exchanging kqml messages [14] between different devices and agents dedicated to controlling sensors and actuators and capable of deciding when to perceive the environment or yet can define a strategy to gather perceptions from sensors. besides, allow the programming of open mas [26], where agents can move from one system to another using an iot network. for this, the reasoning cycle of a standard jason agent was modified to create two novel specific types of bdi agents to program embedded mas. in the first, the agent can perceive or act in the environment and monitor the connection status with wired microcontrollers, allowing fault tolerance and adaptability. in the second, agents can exchange messages with other mas and handle agents’ arrival and departure into its mas. then, we integrated and adapted the ad-hoc solutions [17, 19, 24, 27] in a single distribution. despite the modifications, all agents maintain the original features of jason’s agents. jason embedded allows the designer to abstract some aspects of hardware interfacing and communication, focusing only on programming the mas. the contribution of this work is an extended framework to allow hardware controlling, communicability, and agent mobility to be integrated into a single framework. to demonstrate this, we build and present a case study that implemented an iot solution for home monitoring integrated with a central control and an unmanned autonomous vehicle. this paper is structured as follows: an analysis of related works is presented in sect. 2; in sect. 3 we present the jason embedded and the newly available behavior of agents; we demonstrate an embedded mas in a case study in sect. 4; finally, a discussion and future work are presented in sect. 5. 2 related works a mapping review was conducted to find works that use the bdi framework for the development of iot systems based on cognitive agents. we followed these steps: first was to define the search stringfootnote 1 for google scholar which returned 246 results; next, the results were filtered, ignoring duplicity and results that were not a thesis or a paper, remaining 215; the third step considered only works published after 2017, remaining 171 works; the fourth step ignored surveys or reviews, remaining 153 works. after this, 56 works with the words framework, tool, architecture, library, hardware, iot, cyber-physical, things, or ubiquitous in the title were considered for the next step. finally, it was rejected the works that were not directly related to iot and bdi (i.e., simulation, data mining, and others were discarded); finally, 11 works remained. three works [1, 5, 9] only present an iot implementation based on cognitive agents. four works [6, 11, 16, 23] feature an extension to a bdi framework, however, they do not meet all the needs of an agent-based iot device, such as the hardware control, the communicability with other mas or devices, the mobility of the agents between different mas or devices, fault tolerance, and adaptability. thus, only the below four works [8, 12, 15, 25] present a framework for programming iot agents. the first [8] try to provide connectivity to the web of things using an agent-oriented visual programming ide using a framework endowed with a rest endpoint. the ide is a web-based solution for reducing that expertise in adopting the bdi. the solution allows agents to discover and connect to things using internal structures, but it does not offer embedded solutions to direct control hardware, being dependent on the iot infrastructure. our solution employs specialized agents to control hardware and access iot gateways in an extended version of jason. the hardware control is independent of the iot infrastructure, which is used for the communicability and mobility of agents. similarly, the second [12] provides a declarative language in a bdi-like style for programming mas for the iot using microcontrollers in python. however, microcontrollers are limited in processing capacity and memory and need other components to be connected to any iot gateway. withal allows high-level agents to connect and control these microcontrollers if only they are available and connected to the iot. the phidias is an improved version of profeta, and it presents ways of developing intelligence in devices, including iot, but does not allow agent mobility between different mas or environments. phidia allows communication using http. the communication between microcontrollers and agents is performed using wireless devices. the microcontrollers, sensors, and actuators are not part of the mas. besides, one agent is responsible for each edge device. the third [25] try reduce the gap between theory and practical applications in cybersecurity using bdi agents and mas. in their framework, agents can control iot devices (remotely or hosted in the device), utilizing api to connect to iot gateways or simulated software. in both, the dependence on iot gateways avoid agents from properly working even if embedded. the proposed in this paper allows agents to directly control hardware using serial communication (without network connection) and access the iot network to exchange information with other mas. the fourth [15] try to bring together academic development and industry by offering ways of developing mas using recent technologies (e.g., node.js). similarly, our approach aims to provide and facilitate ways of developing embedded and iot systems to be adopted in domains such as education, academia, and industry. in addition to the mapping review, the spade 3 [22], in its most recent version, uses xmpp and has an extension for bdi agents. however, it does not allow agent mobility between different mas. the authors even argue that mobility between different physical or logical nodes is an interesting and desired functionality, however not contemplated by spade. as previously presented, none of the works embed a mas in iot devices with bdi agents capable of communicating and moving between different embedded mas and also adapting to some faults that may arise in hardware malfunctioning, for example. therefore, our spin-off version of the jason framework presented in this work has customized architectures of bdi agents able to control hardware, communicate, and move between mas embedded in devices using an iot network. besides, the agents that interface hardware can be fault tolerant if the hardware becomes unavailable and adapt themselves to find another way to comply with their goals. 3 jason embedded jason embeddedfootnote 2 is an extended version of jason [4] that supports the development of embedded mas to provide autonomy and communicability to iot devices and mobility and adaptability to agents. in this paper, we define autonomy as a characteristic an embedded mas provides to a device that does not need external architectures to control and manage it. communicability is the agent’s ability to communicate with other agents in its mas or another mas. we define mobility as the agents’ ability to move from one mas to another. finally, fault tolerance and adaptability are defined as the ability of an agent to identify if a physical resource is absent or not answering commands and to overcome this situation by modifying its goals. jason embedded also provides new types of agents to control hardware devices and to communicate with agents hosted in other mas. besides, our extended version allows agent mobility between embedded mas, where one agent or a group of agents can move from one system to another. then, it specializes the mas by creating agents dedicated to certain functionalities without modifying the core of jason, maintaining all of its original functionalities. jason embedded is an initiative employed by our research group in embedded solutions to facilitate the development and teaching of practical mas. the overall picture of our framework is depicted in fig. 1. fig. 1. the overall picture of the framework architecture. full size image in this approach, the developer can employ standard, argo, and communicator agents to develop the embedded mas. each type of agent has specific abilities to deal with hardware interfacing, communicability, and mobility using an iot network. argo agents use hardware interfacing to capture perceptions from the exogenous environment and send commands to activate actuators. the communicator agent has an iot client middleware for connecting to an iot gateway to send messages or agents from one iot device to another. standard agents can communicate and exchange information only with agents from their system. for example, in a scenario concerning three different iot devices hosting embedded mas, one is responsible for managing resources in a house, one is responsible for a monitoring center, and the last is an emergency vehicle. an argo agent monitors the house’s sensors, and if something is detected, it informs the house’s communicator agent to send an alert to the monitoring center. consequently, the communicator agent from the monitoring center searches for an available vehicle and sends a standard agent to help the argo agent to drive the vehicle to attend the house’s emergency. the three devices are distributed, autonomous, and controlled by an embedded mas. they can communicate by sending messages using an iot gateway available, and argo agents perform all sensing in each device. finally, agents can move from one system to another. the new types of agents are based on the standard jason’s agent reasoning cycle. their reasoning cycle is modified at specific points to create new behaviors without modifying any existing function. in this way, every extended agent is still a jason agent. the agent’s reasoning cycle comprises ten steps that complete a bdi decision loop [4]. in the first step, the agent senses the simulated environment using the perceive method to gather all available perceptions. at this point, the agent can only perceive a simulated environment if it exists, and if it needs to interface with real sensors, this environment must be properly coded. the last step executes an intention by performing actions from a selected plan. these two steps represent how the agent can perceive and modify the environment, and agents could use them to interface sensors and actuators. then, the first step could allow agents to gather perceptions directly from sensors and process them as beliefs. since this mechanism works as passive perception, the agent is aware of all available information, even if it does not need all of them to accomplish a goal. to overcome this issue, the agent could also decide whether or not to gather available perceptions and filter undesired perceptions at runtime. it allows the agent to indirectly perform active perception since it can decide when and what to perceive, but it always does it in the first step of its reasoning cycle. similarly, the last step could allow agents to act upon actuators by adopting a protocol that runs over the serial port by sending activation commands [24]. the agent does not need to use jason’s simulated environment—the endogenous representation of the system’s environment—since it can directly activate or deactivate actuators in the real world—the exogenous environment. this approach reduces the use of abstractions and layers to reflect actions in the device’s hardware where the embedded mas is hosted. in the third step of the reasoning cycle, an agent checks its mail, looking for messages that other agents from its mas could have sent to it. since jason embedded aims to provide a framework for programming autonomous and embedded solutions using agents, it is worthwhile to think of prototypes communicating with other prototypes. considering that each prototype has an embedded mas, some agents could work as a communicator, receiving and sending messages to other communicator agents. otherwise, the embedded mas will be autonomous and proactive but without social ability. then, these agents could check an alternative mailbox with messages from other embedded mas. in the last step, agents could send messages to agents within and out of its system by addressing other communicators using an iot middleware, for instance. as sending a message occurs in the last step, it could also allow specific agents to move agents from one embedded mas to another. in embedded mas, agent mobility is based on bio-inspired protocols [27]. the protocols simulate natural behaviors that could be explored in collaborative tasks using iot devices. one or more agents or even the whole mas could move from one device to another to take control of the destination device (predation) or to use it as a temporary non-hazardous relationship (mutualism and inquilinism) if both parties accept the protocol. for communicability and mobility, agents must connect to an iot middleware to redirect the messages and agents to an agent of the target embedded mas. the description of all new internal actions, their behaviors, and requested parameters can be seen in table 1. table 1. the jason embedded’s internal actionsfull size table below are described the new characteristics allowed by the framework presented in this work. jason embedded offers two types of extended agents: one for interfacing hardware named argo and another for communication named communicator, responsible for all communications and mobility from outside its mas. besides, jason embedded maintains the standard agents present in the jason. it is important to remark that the option for specializing agents leads to well-defined responsibilities, allowing agents to focus on their purpose to minimize some drawbacks. for example, when interfacing hardware, it is interesting that the agent can deliberate, considering the perceptions available at that moment. suppose the agent is compromised to some other task as communication or trying to achieve another intention. so, it could affect the agent’s reaction in some applications and domains where the response time is essential—i.e., object deviation. 3.1 controlling hardware argo is a customized agent architecture built on the jason framework that extends jason’s standard agents by adding the ability to control microcontrollers [24]. argo agents interact with the physical environment, capturing information from the environment using sensors and acting by sending commands to the microcontroller to activate the actuators. sensors’ information is automatically processed as perceptions in the agent’s belief base. argo agents have four internal actions to control microcontrollers: port, limit, percepts, act, and filter. these actions define which port the agent is accessing, a time limit for accessing them, if they open or close the flow of perceptions, activate actuators, or filter the incoming percepts, respectively. argo architecture was modified to work properly along with all existing architectures since it was initially designed to be an ad-hoc solution. furthermore, the behavior of swapping resources [17] at runtime was added to avoid stopping the embedded mas during its execution. 3.2 communicability the communicator agent is another customized agent architecture built on jason’s standard agent by adding the ability to communicate with agents from other mas, which also have communicator agents [23]. it allow agents from different mas to interact, exchange knowledge and even collaborate between them. the communicator agent can interact by exchanging knowledge and even collaborating between them using kqml performatives [14]. to use these new capabilities, the communicator agent has specific internal actions: connect, disconnect, and sendout. these actions connects and disconnects from the iot gateway, and send a message to another embedded mas. the communicator agent is a client instance of an iot middleware [13] that needs an iot gateway to communicate. the agent can send a message to another mas using this iot middleware even if the target is disconnected. once it connects, the message is redirected. in this case, the original communicator architecture was totally modified to allow the connection in the iot gateway generically and work with argo agents in the same embedded mas. the connect and disconnect are new functionalities once the agent sometimes can deliberate whether to be offline. 3.3 mobility subsequently, the communicator architecture was extended with bioinspired protocols [27] following ecological relations concepts that allow agents to be transferred between different mas. they can be used to preserve the mas knowledge if the physical device is damaged since it is subject to the unpredictability of the real world. the bioinspired protocols currently have three ecological relations implemented: predation, inquilinism, and mutualism: in predation, all agents are transferred to the target mas to prey and dominate. it eliminates all agents of the target and the origin mas (after moving them) to prevent unwanted access to any residue of its knowledge; in the inquilinism, it sends all its agents to the target mas to preserve its knowledge, but they do not interfere in its activities and existence. however, similar to the predation protocol, the origin mas is eliminated to prevent unwanted access to any residue of knowledge; the mutualism sends an agent, a group of agents, or the entire mas to interact, learn, and transmit knowledge to a target mas. agents using this protocol can return to the origin mas or move to any other if allowed. the communicator has the moveout internal action for the mobility behavior. it uses the iot middleware instance to move agents from one system to another. in this version, mutualism was modified to send all agents except the communicator since the agents need a way to return to the origin mas. in the inquilinism, we modified the protocol to drop all desires in the target mas of the moving agents since they cannot interfere with the mas functioning. we modified the protocols to guarantee the departure and arrival of agents to avoid communication problems and that agents do not get lost during the agents’ transference. 4 case study in the case study, we consider a scenario, shown in fig. 2, that integrates a smart home, a central control, and an unmanned vehicle, all controlled by multi-agent systems. the mas running in the central control has a communicator to forward service calls and two jason agents for decision-making in case of calls. the vehicle has an argo agent for driving and a communicator agent for receiving agents to use the vehicle. the mas running in the house has a communicator agent responsible for interacting with the central and an argo agent controlling the sensors and actuators. fig. 2. the scenario of the case study proposed. full size image in this scenario, when the argo agent in the house’s embedded mas perceives a change in a specific sensor, it will inform the communicator agent to send a message to central control. in the central control, the message is forwarded to a coordinator agent, who will send an agent to the house to analyze it. after completion, the sent agent will send a message to the coordinator requesting a patrol. the coordinator agent will migrate to an unmanned vehicle to execute the mission. when in the vehicle, the coordinator will inform the driving agent about the path. to fulfill the proposed scenario, we built three embedded mas: the first – batcave project – represents the house, hosted in a raspberry pi with an arduino board and has an ldr (light dependent resistor) sensor and a led (light-emitting diode) as the actuator; the second – waynemansion project – represents the control center, hosted in a raspberry pi; the third – batmobile project – represents the unmanned vehicle, hosted in a chonbot 2wd [18] prototype. when the argo agent in the house mas (batcave.mas2j) notices a change in the environment (batsignal(true)) it activates an actuator (led(red)) and requests the communicator agent to forward an alert to the center control mas (waynemansion.mas2j). the fig. 3 shown the code of the batcave project. fig. 3. the implementation of the batcave project. (color figure online) full size image when this alert arrives at the destination, it is forwarded to the jason agent responsible for handling occurrences (bruce.asl); when analyzing the alert, the agent bruce decides to designate other jason agent (alfred.asl) to verify the situation (at mas batcave). the agent alfred requests the communicator agent of waynemansion mas to transfer itself to batcave mas; after this, the communicator of batcave mas receives the agent alfred in the system. alfred requests the argo agent for a new check of sensors and to change the alert state to yellow (led(yellow)); finally, alfred requests to communicator of batcave mas to send a message to agent bruce. the fig. 4 shown the code of the waynemansion project. fig. 4. the implementation of the waynemansion project. (color figure online) full size image the communicator of waynemansion mas forwards the message to bruce, who decides to patrol near the house. in this way, it requests the communicator to transport itself to the autonomous unmanned vehicle (batmobile.mas2j); once it arrives in the batmobile mas, the agent bruce requests to argo agent to pilot the vehicle (goahead). figure 5 shown the implementation of the batmobile project. finally, all projects’ argo and communicator agents have standard plans, beliefs, and intentions shown in fig. 6. fig. 5. the implementation of the batmobile project. full size image fig. 6. the common source of the argo and communicator agents. full size image aiming to enable the reproducibility of what is presented in this work, the framework and its source code, the implementation of the case study, and a demonstration video using prototypes representing the house, the control center, and the unmanned vehicle of the proposed scenario are availablefootnote 3. 5 discussion this paper presented an extended version of the jason framework named jason embedded to program autonomous devices using bdi agents. it provides two new types of agents for allowing hardware interfacing, iot-based communication, and mobility between different embedded mas. besides, jason’s agents can deal with the mas’s internal issues where it lives. we also presented a study case to cover the new functionalities provided by the extended framework. using an embedded mas could bring advantages compared to adopting just one agent to interface all prototype’s functionalities. one agent could be overloaded depending on how much information it gathers or which goals it is pursuing. the decision to use a framework that allows specialized agents to deal with certain functionality leads to an internal distributed solution to optimize the response to stimuli. if one agent has to deal with both perception gathering and external mail checking, it can eventually get overloaded. a prototype embedded with a mas is autonomous and proactive by its capabilities of perceiving and acting upon the real world. however, when dealing with prototypes, it is important that these autonomous devices could collaborate somehow. jason embedded allows these prototypes to exchange messages using the communicator agents. then, devices with different embedded mas can negotiate, coordinate and exchange information without being part of the same physical architecture or network. they only need the internet to connect to the iot server. every embedded mas is an open mas if it adopts at least one communicator agent. an open mas allows agents to enter and leave the system whenever necessary. so, in our extended version, the communicator is responsible for moving agents (and itself) to other systems by invoking bio-inspired protocols. depending on the domain, agents can move and dominate the destiny system or co-exist within the new one. in some cases, the agents can come and go from a system (mutualism). these two contributions now allow communicability and mobility in bdi agents using jason by using some new internal actions since all background technologies are abstracted from the designer. jason embedded could run on any platform or ide that the designer chooses. but to exploit the most of it, it can work along the chonos and chonide, which are an os distribution and an ide, to support the embedding process and develop a mas using jason embedded. the os comprises all the necessary technological dependencies in a single distribution, including the ide to program the firmware and the mas, start and stop the embedded mas, verify the prototype’s outputs (logs), and inspect the mind of any agent. we remark that it has been an initiative of our research group to develop embedded solutions during the past years. in this paper, we consolidate several solutions that used to work alone into a single framework to facilitate their use and adoption. in future works, we will provide an alternative to program artifacts that interfaces hardware using a serial interface and cartago [20]. then, the designer can choose between argo agents and artifacts. the former is one agent dedicated to interface sensors and actuators, while the latter could be available to all agents existing in the system. furthermore, the communicator agent needs a different format for identifying itself in the system since it uses a non-intuitive hexadecimal number. one possibility would be to use an email address to identify the communicator agent. besides, two new protocols could help in the development process of embedded mas: cloning could be used to send a copy of one or more agents to another system without killing these agents in origin; the cryogenics could be used to dump all agents into files and then restart them in the future. from the technology point of view, we intend to match the jason embedded version with the most recent jason framework. this version uses version 1.4.1 since some single-board computers only run java 8, and the recent jason framework uses java 17. another point is to exploit alternative communication infrastructures such as smtp/pop3 or xmpp to offer more reliability and privacy since contextnet uses udp, and the message is transmitted without cryptography. notes 1.(“embedded” or “embodied”) and (“multiagent system” or “multi-agent system”) and “belief-desire-intention” and “framework” and “internet of things”. 2.https://jasonembedded.chon.group/. 3.http://bracis2023.chon.group/. references akhtar, s.m., nazir, m., saleem, k., mahfooz, h., hussain, i.: an ontology-driven iot based healthcare formalism. int. j. adv. comput. sci. appl. 11(2) (2020). https://doi.org/10.14569/ijacsa.2020.0110261 baccour, e., et al.: pervasive ai for iot applications: a survey on resource-efficient distributed artificial intelligence. ieee commun. surv. tut. 24(4), 2366–2418 (2022). https://doi.org/10.1109/comst.2022.3200740 article  google scholar  barnier, c., aktouf, o.e.k., mercier, a., jamont, j.p.: toward an embedded multi-agent system methodology and positioning on testing. in: 2017 ieee international symposium on software reliability engineering workshops (issrew), pp. 239–244 (2017). https://doi.org/10.1109/issrew.2017.57 bordini, r.h., hübner, j.f., wooldridge, m.: programming multi-agent systems in agentspeak using jason. wiley series in agent technology. wiley, hoboken (2007). https://dl.acm.org/doi/10.5555/1197104 brandao, f., nunes, p., de jesus, v.s., pantoja, c.e., viterbo, j.: managing natural resources in a smart bathroom using a ubiquitous multi-agent system. in: proceedings of the 11th workshop-school on agents, environments, and applications, wesaac 2017, pp. 101–112. furg, são paulo (2017) google scholar  brandão, f.c., lima, m.a.t., pantoja, c.e., zahn, j., viterbo, j.: engineering approaches for programming agent-based iot objects using the resource management architecture. sensors 21(23), 8110 (2021). https://doi.org/10.3390/s21238110 article  google scholar  bratman, m.: intention, plans, and practical reason. harvard university press, cambridge, ma (1987) google scholar  burattini, s., et al.: agent-oriented visual programming for the web of things (2022). https://www.alexandria.unisg.ch/handle/20.500.14171/109205 souza de castro, l.f., et al.: integrating embedded multiagent systems with urban simulation tools and iot applications. rita 29(1), 81–90 (2022). https://doi.org/10.22456/2175-2745.110837 chander, b., pal, s., de, d., buyya, r.: artificial intelligence-based internet of things for industry 5.0. in: pal, s., de, d., buyya, r. (eds.) artificial intelligence-based internet of things systems. internet of things. springer, cham (2022). https://doi.org/10.1007/978-3-030-87059-1_1 ciortea, a., mayer, s., michahelles, f.: repurposing manufacturing lines on the fly with multi-agent systems for the web of things. in: proceedings of the 17th international conference on autonomous agents and multiagent systems, aamas 2018, pp. 813–822. international foundation for autonomous agents and multiagent systems, richland, sc (2018). https://dl.acm.org/doi/10.5555/3237383.3237504 d’urso, f., longo, c.f., santoro, c.: programming intelligent iot systems with a python-based declarative tool. in: proceedings of the 1st workshop on artificial intelligence and internet of things co-located with the 18th international conference of the italian association for artificial intelligence, ai*ia 2019. ceur workshop proceedings, rende (cs), italy, vol. 2502, pp. 68–81. ceur-ws.org (2019) google scholar  endler, m., et al.: contextnet: context reasoning and sharing middleware for large-scale pervasive collaboration and social networking. in: proceedings of the workshop on posters and demos track. pdt 2011, association for computing machinery, new york, ny, usa (2011). https://doi.org/10.1145/2088960.2088962 finin, t., fritzson, r., mckay, d., mcentire, r.: kqml as an agent communication language. in: proceedings of the third international conference on information and knowledge management, cikm 1994, pp. 456–463. association for computing machinery, new york, ny, usa (1994). https://doi.org/10.1145/191246.191322 kampik, t., nieves, j.c.: js-son a lean, extensible javascript agent programming library. in: dennis, l.a., bordini, r.h., lespérance, y. (eds.) engineering multi-agent systems, vol. 12058, pp. 215–234. springer, cham (2020). https://doi.org/10.1007/978-3-030-51417-4_11 karaduman, b., tezel, b.t., challenger, m.: enhancing bdi agents using fuzzy logic for cps and iot interoperability using the jaca platform. symmetry 14(7) (2022). https://doi.org/10.3390/sym14071447 lazarin., n., pantoja., c., viterbo., j.: swapping physical resources at runtime in embedded multiagent systems. in: proceedings of the 15th international conference on agents and artificial intelligence volume 1: icaart, pp. 93–104. insticc, scitepress (2023). https://doi.org/10.5220/0011750700003393 lazarin, n., pantoja, c., viterbo, j.: towards a toolkit for teaching ai supported by robotic-agents: proposal and first impressions. in: anais do xxxi workshop sobre educação em computação, pp. 20–29. sbc, porto alegre, rs, brasil (2023). https://doi.org/10.5753/wei.2023.229753 lazarin, n.m., pantoja, c.e.: a robotic-agent platform for embedding software agents using raspberry pi and arduino boards. in: proceedings of the 11th workshop-school on agents, environments, and applications, wesaac 2015, pp. 13–20. niteroi (2015) google scholar  manoel, f., pantoja, c.e., samyn, l., de jesus, v.s.: physical artifacts for agents in a cyber-physical system: a case study in oil & gas scenario (eeas). in: the 32nd international conference on software engineering and knowledge engineering, seke 2020, pp. 55–60. ksi research inc. (2020) google scholar  michel, f., ferber, j., drogoul, a.: multi-agent systems and simulation: a survey from the agents community’s perspective. in: danny weyns, a.u. (ed.) multi-agent systems: simulation and applications, p. 47. computational analysis, synthesis, and design of dynamic systems, crc press taylor & francis, may 2009 google scholar  palanca, j., terrasa, a., julian, v., carrascosa, c.: spade 3: supporting the new generation of multi-agent systems. ieee access 8, 182537–182549 (2020). https://doi.org/10.1109/access.2020.3027357 article  google scholar  pantoja, c., soares, h.d., viterbo, j., seghrouchni, a.e.f.: an architecture for the development of ambient intelligence systems managed by embedded agents. in: the 30th international conference on software engineering & knowledge engineering, pp. 215–249. ksi research inc., san francisco bay, july 2018. https://doi.org/10.18293/seke2018-110 pantoja, c.e., stabile, m.f., lazarin, n.m., sichman, j.s.: argo: an extended jason architecture that facilitates embedded robotic agents programming. in: baldoni, m., müller, j.p., nunes, i., zalila-wenkstern, r. (eds.) emas 2016. lncs (lnai), vol. 10093, pp. 136–155. springer, cham (2016). https://doi.org/10.1007/978-3-319-50983-9_8 chapter  google scholar  rafferty, l.: agent-based modeling framework for adaptive cyber defence of the internet of things. ph.d. thesis, faculty of business and it, university of ontario institute of technology, oshawa, ontario, canada (2022) google scholar  da rocha costa, a.c., hübner, j.f., bordini, r.h.: on entering an open society. in: xi brazilian symposium on artificial intelligence, vol. 535, p. 546 (1994) google scholar  souza de jesus., v., pantoja., c.e., manoel., f., alves., g.v., viterbo., j., bezerra., e.: bio-inspired protocols for embodied multi-agent systems. in: proceedings of the 13th international conference on agents and artificial intelligence volume 1: icaart, pp. 312–320. insticc, scitepress (2021). https://doi.org/10.5220/0010257803120320 wooldridge, m.: an introduction to multiagent systems. wiley, hoboken (2009) google scholar  download references author information authors and affiliations institute of computing, fluminense federal university (uff), niterói, rj, brazil carlos eduardo pantoja, vinicius souza de jesus, nilson mori lazarin & josé viterbo federal center for technological education celso suckow da fonseca (cefet/rj), rio de janeiro, rj, brazil carlos eduardo pantoja & nilson mori lazarin authors carlos eduardo pantojaview author publications search author on:pubmed google scholar vinicius souza de jesusview author publications search author on:pubmed google scholar nilson mori lazarinview author publications search author on:pubmed google scholar josé viterboview author publications search author on:pubmed google scholar corresponding author correspondence to carlos eduardo pantoja . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper pantoja, c.e., jesus, v.s.d., lazarin, n.m., viterbo, j. (2023). a spin-off version of jason for iot and embedded multi-agent systems. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14195. springer, cham. https://doi.org/10.1007/978-3-031-45368-7_25 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45368-7_25 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45367-0 online isbn: 978-3-031-45368-7 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords internet of things multi-agent systems edge computing publish with us policies and ethics profiles carlos eduardo pantoja view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature evaluating recent legal rhetorical role labeling approaches supported by transformer encoders | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper evaluating recent legal rhetorical role labeling approaches supported by transformer encoders conference paper first online: 12 october 2023 pp 18–32 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2023) evaluating recent legal rhetorical role labeling approaches supported by transformer encoders download book pdf download book epub alexandre gomes de lima  orcid: orcid.org/0000-0001-7251-00089,10, josé g. moreno  orcid: orcid.org/0000-0002-8852-579711, taoufiq dkaki11, eduardo henrique da s. aranha  orcid: orcid.org/0000-0002-8446-638x10 & … mohand boughanem11  show authors part of the book series: lecture notes in computer science ((lnai,volume 14196)) included in the following conference series: brazilian conference on intelligent systems 510 accesses 1 citation abstract pre-trained transformer models have been used to improve the results of several nlp tasks, which includes the legal rhetorical role labeling (legal rrl) one. this task assigns semantic functions, such as fact and argument, to sentences from judgment documents. several legal rrl works exploit pre-trained transformers to encode sentences but only a few employ approaches other than fine-tuning to improve the performance of models. in this work, we implement three of such approaches and evaluate them over the same datasets to achieve a better perception of their impacts. in our experiments, approaches based on data augmentation and positional encoders do not provide performance gains to our models. conversely, the models based on the dfcsc approach overcome the appropriate baselines, and they do remarkably well as the lowest and highest improvements respectively are 5.9% and 10.4%. this work was supported by npad/ufrn, by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes, finance code 001) and by the lawbot project (anr-20-ce38-0013), granted by anr the french agence nationale de la recherche. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others deeprhole: deep learning for rhetorical role labeling of sentences in legal case documents article 13 november 2021 marro: multi-headed attention for rhetorical role labeling in legal documents article 18 march 2025 rhetorical role prediction in legal documents: a comparison of machine learning algorithms chapter © 2025 explore related subjects discover the latest articles, books and news in related subjects, suggested using machine learning. computational linguistics corpus linguistics literature and law natural language processing (nlp) research methods in language and linguistics speech act theory 1 introduction one of the main goals of legal ai, the application of artificial intelligence (ai) in the legal domain, is task automation, which increases the productivity of legal professionals and makes the law more accessible. several works related to legal ai are about tasks concerning judgments such as judgment prediction [14, 24], and legal summarization [7, 9], which rely on judgment documents as the main source of data. a judgment document is a record of a lawsuit authored by the court that sets forth its decision about the case. the document includes such as statements of the facts, analysis of relevant law, rights and liabilities of the parties, and rulings, among others. specific information is required or can be exploited to boost legal ai tasks: facts for fact-based search; facts and violated statutes for judgment prediction; facts, arguments, and judicial reasoning for legal summarization. the legal rhetoric role labeling (legal rrl) task assigns semantic functions to sentences in judgment documents (e.g., fact, argument, ruling). thus, legal rrl is a way of finding information that is useful in itself as well as for downstream tasks. the legal rrl task is not trivial, even for humans. one of the limiting factors is the heterogeneity in the structure of documents. it is common for a lawsuit to be judged by different courts (e.g., lower courts, supreme court), which may adopt different standards or guidelines to write judgment documents. another issue is the subjectivity related to the interpretation of different rhetorical roles [3, 15]. inter-annotator agreements reported from works about the development of legal rrl datasets are good evidence of such subjectivity. in [15], the authors report a low inter-agreement score (kappa index is 0.65) among three annotators on judgments from the income tax domain over 1,600 sentences. similarly, in [8] a low inter-agreement score (kappa index is 0.59) is reported for 40,305 sentences and 30 annotators. legal rrl automation can be implemented as a sentence classification task, where sentences are fed into a machine learning model that assigns a label to each sentence. the model consists of two core components, a sentence encoder and a classifier. the sentence encoder is responsible to produce numeric representations (i.e., vectors), commonly known as sentence embeddings, of text sentences while the classifier assigns labels to each representation as illustrated in fig. 1. fig. 1. sentence classification task. full size image the representation vectors directly impact the performance of the model since richer representations lead to more precise classifications [21]. the exploitation of pre-trained transformer models [20] is the current state-of-the-art in many nlp tasks and this includes the sentence encoding one. several pre-trained transformer models are publicly available, such as bert [5] and roberta [13]. when generating the vector representation of a word w in a sentence, a transformer model takes w and its neighbor words into account. therefore, it yields different vector representations for w when it occurs in different sentences or different positions of the same sentence. this context exploitation ability allows the model to yield rich text representations. fine-tuning is a common procedure for exploiting pre-trained transformers that adjusts the weights of a pre-trained model for a specific task and dataset. this is a general procedure and it can be employed to tackle the legal rrl task [1, 3, 8, 18]. some recent works [10,11,12, 15] propose additional procedures as a way of improving the performance of models handling the legal rrl task but, despite their potential, such works employ different datasets what hinders a fair comparison. in this context, the goal of this work is to perform an equitable comparison among approaches proposed in recent works [10,11,12, 15]. to achieve this, we implement models based on three of such approaches and evaluate them over two datasets. only the dfcsc-based models overcome the appropriate baselines, and they do so remarkably well: the lowest and highest improvements respectively are 5.9% and 10.4%. 2 recent approaches to handle legal rrl pre-trained transformer models have achieved remarkable results in many nlp tasks. thus, it is not a surprise the publication of legal rrl works that rely on such models [1, 3, 8, 18]. fine-tuning is a common training procedure for one working with pre-trained models, whose goal is the adjustment of the model weights to the task at hand. a pre-trained transformer model can be employed out-of-the-box (i.e., without fine-tuning) but since the task at hand is mostly different from the task used in its pre-training, the fine-tuning improves the performance of the model. other training procedures can be employed together with the fine-tuning one as a way of improving the performance of a model. regarding the legal rrl task, there are a few recent works that follow this strand. mixup [23], a data augmentation technique, is exploited in [10] to augment the training set and hence improve model performance. mixup relies on the weighted interpolation of two inputs to generate a synthetic vector as defined by the following equations: $$\begin{aligned} \mathbf {\tilde{x}}& =\lambda \textbf{x}_{i}+(1-\lambda )\textbf{x}_{j}, {} & {} \textbf{x}_{i}, \textbf{x}_{j}\in \mathcal {r}^e \\ \mathbf {\tilde{y}}& =\lambda \textbf{y}_{i}+(1-\lambda )\textbf{y}_{j}, {} & {} \textbf{y}_{i}, \textbf{y}_{j} \in \mathcal {r}^{|l|}, \end{aligned}$$ where \(\textbf{x}_{i}\) and \(\textbf{x}_{j}\) are sentence embedding vectors, e is the embedding dimension, \(\textbf{y}_{i}\) and \(\textbf{y}_{j}\) are one-hot label encodings, and |l| is the number of labels or classes. \((\textbf{x}_{i}, \textbf{y}_{i})\) and \((\textbf{x}_{j}, \textbf{y}_{j})\) are two examples drawn at random from the training data and \(\lambda \) is a value in the interval [0, 1] drawn at random from a beta\((\alpha ,\alpha )\) distribution, where \(\alpha > 0\). in their setup [10], the authors exploit sentence bert [17] as an encoder without fine-tuning. the exploited dataset comprises 60 judgments, 10,024 sentences, and seven rhetorical roles. as a result, the mixup-based models overcome the respective baselines in terms of f1 score. for example, a linear classifier achieves a gain of 4.3% in the worst case and 6.8% in the best case. another work [12] leverages positional information in addition to encoded sentences. each sentence is represented by a sentence embedding and a positional embedding, the latter indicating the position a sentence occupies in its source document. positional embeddings are computed by the same sinusoidal positional encoding method exploited in the original transformer architecture [20]: $$\begin{aligned} {\begin{matrix} pe_{(pos,2k)} &{} = \sin (pos/m^{2k/e}) \\ pe_{(pos,2k+1)} &{} = \cos (pos/m^{2k/e}) \end{matrix}} \end{aligned}$$ where pos is an integer indicating the position of a sentence, k is a dimension of a \(\textbf{pe}\) vector where \(0 \le k < (e-1)/2\), e is the embedding dimension, and m is the max number of sentences in a document. the sentence and positional embeddings must be combined before being fed to the models’ classifier and to do this the authors employ vector sum and vector concatenation. in their setup [12], they experiment with four pre-trained transformers as sentence encoders. the exploited dataset comprises 369 judgments, 54,244 sentences, and two rhetorical roles. the authors report gains from 3.3% to 12.6% achieved by fine-tuned models. the approaches presented in [10, 12] rely on the single sentence classification paradigm: a sentence is encoded in isolation and so its context (neighbor sentences) is ignored. although, transformers are notable for exploiting context to encode sentences and thus the work of [11] leverages this ability to produce richer sentence embeddings as a way to achieve better results. the encoder is fed with chunks of sentences and a special character is employed to signal sentences’ limits. the authors propose a chunk layout and an assembly procedure, named dfcsc (dynamic-filled contextualized sentence chunk), that avoids the use of padding tokens, the truncation of core sentences, and that defines at run time the number of core sentences and edge tokens in each chunk. core sentences are the sentences taken into account by the classifier and edge tokens are tokens inserted at the edges of the chunks as a way to provide context for the first and last core sentences (see fig. 2). the procedure requires two hyperparameters: \(c_{len}\) (the number of tokens in a chunk) and \(m_{edges}\) (the minimum desired number of tokens in the edges of a chunk). in their setup [11], the authors exploit a dataset with 277 judgments, 31,865 sentences, and 13 rhetorical roles. they experiment with four pre-trained transformers and two additional approaches based on sentence chunks besides the dfcsc one. dfcsc-based models perform the best and the reported gains are respectively of 3.76% and 8.71% in the best and worst cases. fig. 2. illustration of a dfcsc. remark that the classifier takes into account only the tokens chosen as core sentence representations. \(s_i\) means the i-th sentence, \(y_i\) is the label assigned to \(s_i\), h represents a token embedding, t represents a token, \(|t_i|\) is the number of tokens in \(s_i\), \( \texttt {<s>}\) and \(\texttt {</s>}\) are special tokens. full size image 3 experimental setup 3.1 datasets we employ two datasets to evaluate the models which we refer to as 7-roles and 4-roles. both are derivations of the original dataset developed in [15]footnote 1 that comprises 100 judgment documents written in english from the indian legal system split into training set (80 documents and 16,845 sentences), validation set (10 documents and 2,142 sentences), and test set (10 documents and 2,197 sentences). a team of legal experts annotated the documents and each sentence is labeled with one of the following 13 rhetorical roles: fact (fac), issues (iss), argument petitioner (arg-p), argument respondent (arg-r), statute (sta), dissent (dis), precedent relied upon (pre-r), precedent not relied upon (pre-nr), precedent overruled (pre-o), ruling by lower court (rlc), ruling by present court (rpc), ratio of the decision (rod), and none (non). we follow the same procedure from [15] to create the 7-roles dataset. the issue labels are renamed as fact, the argument petitioner and argument respondent labels are renamed as argument (arg), and the precedent relied upon, precedent not relied upon and precedent overruled (pre-o) labels are renamed as precedent (pre). also, the dissent and none sentences are ignored for classification purposes. although, some models rely on document-level data to produce feature vectors and discarding such sentences would result in a data loss degree. regarding chunk-based models, dissent and none sentences are part of the model input but their predicted labels are discarded when we compute the loss function value. regarding pe-based models, such sentences are not part of the model input but they are exploited when computing the positional embeddings to preserve the actual sentence positions. by discarding the dissent and none sentences, the 7-roles dataset is composed of 16,264, 2,109, and 2,094 sentences in the training, development, and test sets, respectively. to be effective, pe-based models require some degree of correlation between sentence positions and rhetorical roles [12]. thus, we designed the 4-roles dataset to perform a fairer evaluation of pe-based models. to achieve this goal, we modified the original dataset from [15] by maintaining the roles occurring often at the same positions (i.e., the fact, ruling by present court and ratioofthedecision) and by renaming the remaining roles as other. we do not discard sentences in this process and hence the 4-roles dataset has the same number of sentences as the original one. the original dataset is highly unbalanced and this is a characteristic of the 7-roles and 4-roles datasets as well. the distribution of sentences by rhetorical role in the three datasets is shown in fig. 3. fig. 3. number of sentences per rhetorical role in the original, 7-roles, and 4-roles datasets. full size image 3.2 models we implement models relying on the approaches from [10, 12], and [11]. for each approach, we experiment with two or three transformer encoders. details of our results and source code are available at https://github.com/alexlimatds/bracis_2023. mixup-a: these are models whose training includes mixup vectors as proposed by [10]. at first, a transformer encoder is fine-tuned on the training dataset in the same fashion as the singlesc approach described ahead (step a in fig. 4). in the following, the fine-tuned transformer is employed to yield sentence embeddings which serve as source data of the mixup procedure (step b in fig. 4). finally, the mixup vectors and the sentence embeddings are employed to train a mixup-a model (step c in fig. 4). the training of a mixup-a model updates only the weights of its classification layer. sentences are represented by the last embedding vector of the token marking the beginning of the input ([cls] or \(\texttt {<s>}\)). we employ incaselaw [16] and roberta-base [13] as sentence encoders. incaselaw is a bert model pre-trained with corpora from the indian legal system. the generation of mixup vectors is oriented to selected labels and an augmentation rate \(\gamma \). for each selected label \(l_i\), we generate \(n_{l_i} \cdot \gamma \) mixup vectors, where \(n_{l_i}\) means the number of training sentences labeled with \(l_i\). a mixup vector is generated from a sentence embedding labeled with \(l_i\) and a sentence embedding labeled with \(l_j\) where \(i \ne j\) (the embeddings are selected at random). when experimenting with the 7-labels dataset, we set \(\gamma =0.5\), \(\alpha =0.1\), the selected labels are argument, statute, precedent, rulingbylowercourt, and rulingbypresentcourt which results in 3,881 synthetic vectors. for the 4-labels dataset, we set \(\gamma =0.5\), \(\alpha =0.1\), the selected labels are fact, ratio of the decision, and ruling by present court which results in 4,408 synthetic vectors. fig. 4. steps performed to train a mixup-a model. step a: fine-tuning of the transformer encoder by training a singlesc model. step b: generation of mixup synthetic vectors from sentence embeddings yielded by the fine-tuned transformer encoder. step c: employing sentence embeddings and mixup vectors to train the classifier of the mixup-a model. full size image mixup-b: these are models similar to the mixup-a ones but without a prior encoder fine-tuning step. the training of a mixup-b model updates the weights of the encoder and of the classification layer (i.e., the transformer is fine-tuned during the training of the model). since the sentence embeddings change during the fine-tuning, the mixup procedure yields a new set of synthetic vectors at each training epoch as shown in fig. 5. as before, we exploit incaselaw and roberta as sentence encoders. when experimenting with the 7-labels dataset, we set \(\gamma =1.0\), \(\alpha =0.1\), and select all seven labels, which results in about 14,800 synthetic vectors per training epoch. for the 4-labels dataset, we set \(\gamma =1.0\), \(\alpha =1.0\), the selected labels are fact, ratio of the decision, and ruling by present court which results in about 8,470 per training epoch. fig. 5. training procedure of a mixup-b model. full size image pe-s: these are models that rely on the sum of sentence embeddings and positional embeddings as proposed in [12]. we exploit incaselaw and roberta as sentence encoders and sentences are encoded in isolation. pe-c: like pe-s models, but we employ concatenation instead of sum to combine sentence embeddings and positional embeddings. dfcsc-sep: these are models based on the dfcsc approach [11]. we employ incaselaw, roberta-base, and longformer-base [2] as sentence encoders. longformer is a model that relies on a sparse self-attention mechanism, which allows it to handle longer sentences than the bert-based models. in all models, core sentences are represented by the last hidden states of the respective separator tokens (the [sep] or \(\texttt {</s>} \)token to the right of a core sentence). we set \(m_{edges}=250\) for the three models, \(c_{len}=512\) for the models based on incaselaw and roberta, and \(c_{len}=1,024\) for the model based on longformer. dfcsc-cls: like dfcsc-sep models, but each core sentence is represented by the concatenation of the last hidden states regarding the special token marking the beginning of the chunk ([cls] or \(\texttt {<s>}\)) and the corresponding separator token ([sep] or \(\texttt {</s>}\)). we also implement the following baselines: singlesc: these are single sentence classification models, i.e. models that do not rely on chunks. each sentence is fed to the model one by one and is represented by the hidden state of the token that marks the beginning of the input (\(\texttt {<s>}\) or [cls]). singlesc models are the reference baselines of mixup and pe models, since such approaches do not consider context to encode sentences. the exploited pre-trained encoders are incaselaw and roberta-base. cohan: these are models that follows the chunk design from [4], that is, chunks that do not share sentences or tokens. cohan models are the reference baselines of dfcsc models since such approaches rely on chunks to encode sentences. the exploited pre-trained encoders are incaselaw, roberta, and longformer. models based on incaselaw and roberta have the following hyperparameter values: 85 as the maximum sentence length, 512 as the chunk length, and 7 as the maximum number of sentences in a chunk. the model based on longformer has the following hyperparameter values: 85 as the maximum sentence length, 1,024 as the chunk length, and 14 as the maximum number of sentences in a chunk. each sentence is represented by the hidden state of the following separator token. all models use a single fully-connected layer as their classifier. except for the mixup-a models, the encoder is fine-tuned together with the classifier during the training of the respective model. for each mixup-a model, the transformer encoder is tuned with the same hyperparameters as its singlesc counterpart (e.g., singlesc-incaselaw and mixup-a-incaselaw). for their classifiers, we adopt 200 training epochs, \(10^{-2}\) as initial learning rate, and 0.2 as dropout rate. the remaining models are fine-tuned for four epochs, and we adopt \(10^{-5}\) as initial learning rate and 0.2 as dropout rate of the classification layer. the pre-trained transformer models are used with default parameters, except for the longformer, whose global attention is set to consider \(\texttt {<s>}\) and \(\texttt {</s>}\) tokens. for all models we employ the cross entropy loss function and the adam optimizer (\(\beta _1=0.9\), \(\beta _2=0.999\), \(\epsilon =10^{-8}\), \(weight\_decay=10^{-3}\)). the learning rate is scheduled linearly from an initial value to zero, and there are no warm-up steps. we adopt a batch size of 4 for longformer-based models and a batch size of 16 for the other models. python 3.8.13, pytorch 1.13.0, and hugging face transformer 4.28.1 are the programming language and main libraries we use to develop all models. hugging face is mainly used to load pre-trained transformer models. a single 16gb tesla v100 sxm2 gpu is used in all experiments. 3.3 evaluation all models are trained on the training set and evaluated on the test set. for each model, the training and evaluation procedure is repeated five times with a different random seed for each repetition (the same five seeds for all models). the evaluation metric is the macro f1 score. to compare the results obtained by the implemented models, we use the almost stochastic order (aso) test [6] with bonferroni correction as implemented by [19]. aso is a statistical significance test designed to compare two score distributions produced by deep learning models. the test analyzes two distributions and quantifies the amount of stochastic order violation by computing a value \(\epsilon _{min}\). given the scores over multiple random seeds from two models a and b, a is declared superior if \(\epsilon _{min} < \tau \), where \(\tau \) is 0.5 or less (or more precisely, we confirm that a is stochastically dominant over b in more cases than vice versa). we can also interpret \(\epsilon _{min}\) as a confidence score: the lower it is, the stronger the evidence that a is superior to b. in this context, and following the recommendation of \(\tau =0.2\) for multiple comparisons, we formulate our hypotheses as $$\begin{aligned} & h_0: \epsilon _{min} \ge 0.2 {} & {} \text {({ a} is not superior to { b})} \\ & h_1: \epsilon _{min} < 0.2 {} & {} \text {({ a} is superior to { b})} \end{aligned}$$ in our experiments, we compare all pairs of models based on five random seeds each using aso with a confidence level of \(\alpha = 0.05\) (before adjusting for all pairwise comparisons using the bonferroni correction). we only compare scores obtained on the same dataset. table 1. results achieved by the baseline models and the models of interest. the f1 scores are averages of five evaluations and refer to the best fine-tuning epoch (i.e., the epoch that gives the highest f1 score). the epoch column reports the best fine-tuning epoch (e.g., 1/4 indicates the 1st epoch out of a total of 4 epochs). for mixup-a models, we report the encoder fine-tuning epoch followed by the mixup-a classifier training epochs. \(\sigma \) denotes the standard deviation. the scores in bold are the best for each dataset.full size table 4 results and discussion table 1 presents the results achieved by the models described in sect. 3 and by the mtl-lsp-bilstm-bert model reported in [15]. we confirm the statistical power of the samples (scores) by employing bootstrap power analysis [22]. baselines: all cohan models achieve higher scores than singlesc ones and the differences are statistically significant in both datasets (\(\epsilon _{min} \le 0.05\) in all comparisons). this is expected since cohan models exploit more contextual data to yield sentence embeddings. the differences between singlesc-incaselaw and singlesc-roberta models are significant in both datasets as well (\(\epsilon _{min} \le 0.08\)). we also have significant differences for cohan-longformer vs. cohan-incaselaw and cohan-longformer vs. cohan-roberta (\(\epsilon _{min}=0.01\) in both comparisons) for the 7-roles dataset, and for cohan-incaselaw vs cohan-roberta (\(\epsilon _{min}=0.04\)) and cohan-incaselaw vs cohan-longformer (\(\epsilon _{min}=0.00\)) for the 4-roles dataset. the mtl-lsp-bilstm-bert model remarkably outperforms all baselines. although we cannot perform aso tests among this model and our baselines because the scores of mtl-lsp-bilstm-bert are not available, the differences are likely significant because the differences are large and the reported standard deviation is small. mixup models: regarding the two datasets, the score differences between each mixup-a or mixup-b model and its singlesc counterpart (e.g., mixup-a-incaselaw vs. singlesc-incaselaw) are really small, and according to the aso tests, these differences do not have statistical significance. hence, the mixup approaches do not harm or improve the base singlesc models. also, there are no significant differences among mixup-a and mixup-b models, considering both datasets and the same encoder. previous work [10] achieves performance gains by exploiting mixup vectors: the f1 score achieved by a linear classifier improves by 4.3% in the worst case and by 6.8% in the best case. so why do our mixup-based models not outperform their baselines? this may be due to the size of the datasets. in [10], there is little data, so the additional data provided by mixup is beneficial. conversely, the dataset of the current work is larger, leaving no room for improvements based on data augmentation. pe models: concerning the 7-roles dataset, the pe-s models perform worse than their respective singlesc counterparts, but the difference is only significant for singlesc-roberta vs. pe-s-roberta (\(\epsilon _{min}=0.03\)). this means that the pe-s approach harms the base singlesc-roberta model instead of improving it. for the 4-roles dataset, the pe-s models outperform their respective singlesc counterparts, but the differences are not significant. also, there are no significant differences among the pe-c models and their respective singlesc counterparts for the two datasets. we expected poor results (i.e., no gain or decrease in performance) from the pe models with respect to the 7-roles dataset, since most of its labels do not have a clear correlation with sentence positions. in the case of performance degradation, pe vectors should be working as noise data. conversely, we expected pe vectors to provide gains in the experiments with the 4-roles dataset, which does not happen. we believe that the positional embeddings employed do not represent positions in a way that can enrich the input data of the models. dfcsc models: regarding the 7-roles dataset, all dfcsc models outperform their respective cohan counterparts. the differences are significant with \(\epsilon _{min}=0.14\) for dfcsc-cls-incaselaw vs. cohan-incaselaw, and \(\epsilon _{min}=0.01\) for the remaining comparisons. for the 4-roles dataset, only dfcsc-sep-incaselaw and dfcsc-cls-incaselaw do not outperform their respective cohan counterparts. the aso test results in \(\epsilon _{min}=0.23\) for cohan-incaselaw vs. dfcsc-sep-incaselaw, and in \(\epsilon _{min}=0.00\) for cohan-incaselaw vs. dfcsc-cls-incaselaw, meaning that only the latter comparison is significant. the dfcsc models using roberta and longformer encoders achieve remarkable improvements: the lowest and highest gains are 5.9% and 10.4%, respectively. this behavior is also reported in [11], and thus we consider this additional evidence that the roberta and longformer encoders exploit contextual data in a better way than the incaselaw one (roberta and longformer rely on the same training procedure, which is different from the one used for incaselaw). comparing the performance of the dfcsc models using table 2, we come to some conclusions. first, regarding the 7-roles dataset, we see that all longformer-based models are superior to those based on roberta and incaselaw; that all roberta-based models are superior to those based on incaselaw; and that there is no significant difference between the dfcsc-sep and dfcsc-cls approaches when we compare models that employ the same encoder. regarding the 4-roles dataset, we find that dfcsc-sep-longformer is superior to all models using roberta and incaselaw; that dfcsc-cls-longformer is superior to dfcsc-cls-roberta, dfcsc-sep-incaselaw, and dfcsc-cls-incaselaw; that the two roberta-based models are superior to those based on incaselaw; and that there is no significant difference between the dfcsc-sep and dfcsc-cls approaches when we compare models that employ the same encoder. when comparing dfcsc and mtl-lsp-bilstm-bert models, we find that roberta-based models perform similarly and that longformer-based models perform better. table 2. confidence scores (\(\epsilon _{min}\)) from aso tests comparing dfcsc models. comparisons follow the row-to-column direction, and significant differences (i.e., \(\epsilon _{min} < 0.2\)) are indicated by underlined values.full size table 5 final remarks in this work, we compare approaches from recent proposals to handle the legalrrl task. all approaches rely on pre-trained transformers to encode sentences and employ different strategies to improve reference baselines. models based on the mixup and pe approaches are unable to overcome the singlesc baselines. for the pe models, we conclude that the 7-role dataset does not have a sufficient correlation between labels and sentence positions, which nullifies the usefulness of the models. for the 4-role dataset, we believe that the positional embeddings employed are not powerful enough to enrich the input data of the models. models based on the dfcsc approach, on the other hand, show remarkable performance. most of them overcome strong baselines based on sentence chunks. dfcsc models that exploit roberta and longformer encoders perform similarly or better than a model based on the mtl-lsp approach proposed in [15]. from the obtained results, we propose a future work that combines mtl-lsp and dfcsc approaches, since the mtl-lsp-bilstm-bert model does not rely on sentence chunks to encode sentences with bert. such work would provide interesting evidence about the ability of lsp signals to boost sentence chunk-based models. we also say that the sentence classification approaches evaluated in this work do not rely on knowledge specific to the legal domain. therefore, more studies should be conducted to evaluate the efficiency of such approaches in other domains. despite this, we believe that incorporating domain-specific knowledge into models based on pre-trained transformers is a valuable research direction. notes 1.available at https://github.com/exploration-lab/rhetorical-roles. references aragy, r., fernandes, e.r., caceres, e.n.: rhetorical role identification for portuguese legal documents. in: britto, a., valdivia delgado, k. (eds.) bracis 2021. lncs (lnai), vol. 13074, pp. 557–571. springer, cham (2021). https://doi.org/10.1007/978-3-030-91699-2_38 chapter  google scholar  beltagy, i., peters, m.e., cohan, a.: longformer: the long-document transformer. corr abs/2004.05150 (2020). https://arxiv.org/abs/2004.05150 bhattacharya, p., paul, s., ghosh, k., ghosh, s., wyner, a.: deeprhole: deep learning for rhetorical role labeling of sentences in legal case documents. artificial intelligence and law, pp. 1–38 (2021) google scholar  cohan, a., beltagy, i., king, d., dalvi, b., weld, d.: pretrained language models for sequential sentence classification. in: proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (emnlp-ijcnlp), pp. 3693–3699. association for computational linguistics, hong kong, november 2019. https://doi.org/10.18653/v1/d19-1383. https://aclanthology.org/d19-1383 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of the 2019 conference of the north american chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pp. 4171–4186. association for computational linguistics, minneapolis, minnesota, june 2019. https://doi.org/10.18653/v1/n19-1423. https://aclanthology.org/n19-1423 dror, r., shlomov, s., reichart, r.: deep dominance how to properly compare deep neural models. in: korhonen, a., traum, d.r., màrquez, l. (eds.) proceedings of the 57th conference of the association for computational linguistics, acl 2019, florence, italy, july 28–august 2, 2019, volume 1: long papers, pp. 2773–2785. association for computational linguistics (2019). https://doi.org/10.18653/v1/p19-1266. https://doi.org/10.18653/v1/p19-1266 feijo, d., moreira, v.: summarizing legal rulings: comparative experiments. in: proceedings of the international conference on recent advances in natural language processing (ranlp 2019), pp. 313–322. incoma ltd., varna, bulgaria, september 2019. https://doi.org/10.26615/978-954-452-056-4_036. http://aclanthology.org/r19-1036 kalamkar, p., et al.: corpus for automatic structuring of legal documents. in: proceedings of the thirteenth language resources and evaluation conference, pp. 4420–4429. european language resources association, marseille, france, june 2022. https://aclanthology.org/2022.lrec-1.470 li, d., yang, k., zhang, l., yin, d., peng, d.: class: a novel method for chinese legal judgments summarization. in: proceedings of the 5th international conference on computer science and application engineering. csae 2021. association for computing machinery, new york (2021). https://doi.org/10.1145/3487075.3487161 de lima, a.g., boughanem, m., da s. aranha, e.h., dkaki, t., moreno, j.g.: exploring sbert and mixup data augmentation in rhetorical role labeling of indian legal sentences. in: tamine, l., amigó, e., mothe, j. (eds.) proceedings of the 2nd joint conference of the information retrieval communities in europe (circle 2022), samatan, gers, france, july 4–7, 2022. ceur workshop proceedings, vol. 3178. ceur-ws.org (2022). https://ceur-ws.org/vol-3178/circle_2022_paper_29.pdf de lima, a.g., moreno, j.g., da s. aranha, e.h.: irit_iris_a at semeval-2023 task 6: legal rhetorical role labeling supported by dynamic-filled contextualized sentence chunks. in: proceedings of the 17th international workshop on semantic evaluation (semeval-2023). association for computational linguistics, toronto, canada, july 2023 google scholar  de lima, a.g., moreno, j.g., boughanem, m., dkaki, t., da s. aranha, e.h.: leveraging positional encoding to improve fact identification in legal documents. in: first international workshop on legal information retrieval (legalir) at ecir 2023, pp. 11–13 (2023). https://tmr.liacs.nl/legalir/legalir2023_proceedings.pdf liu, y., et al.: roberta: a robustly optimized bert pretraining approach. corr abs/1907.11692 (2019). https://arxiv.org/abs/1907.11692 ma, l., zhang, y., wang, t., liu, x., ye, w., sun, c., zhang, s.: legal judgment prediction with multi-stage case representation learning in the real court setting. in: proceedings of the 44th international acm sigir conference on research and development in information retrieval, pp. 993–1002. sigir ’21. association for computing machinery, new york (2021). https://doi.org/10.1145/3404835.3462945 malik, v., sanjay, r., guha, s.k., hazarika, a., nigam, s., bhattacharya, a., modi, a.: semantic segmentation of legal documents via rhetorical roles. in: proceedings of the natural legal language processing workshop 2022, pp. 153–171. association for computational linguistics, abu dhabi, united arab emirates (hybrid), december 2022. https://aclanthology.org/2022.nllp-1.13 paul, s., mandal, a., goyal, p., ghosh, s.: pre-training transformers on indian legal text. corr abs/2209.06049 (2022). https://doi.org/10.48550/arxiv.2209.06049 reimers, n., gurevych, i.: sentence-bert: sentence embeddings using siamese bert-networks. in: inui, k., jiang, j., ng, v., wan, x. (eds.) proceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing, emnlp-ijcnlp 2019, hong kong, china, november 3–7, 2019, pp. 3980–3990. association for computational linguistics (2019). https://doi.org/10.18653/v1/d19-1410 savelka, j., westermann, h., benyekhlef, k.: cross-domain generalization and knowledge transfer in transformers trained on legal data. in: ashley, k.d., atkinson, k., branting, l.k., francesconi, e., grabmair, m., walker, v.r., waltl, b., wyner, a.z. (eds.) proceedings of the fourth workshop on automated semantic analysis of information in legal text held online in conjunction with the 33rd international conference on legal knowledge and information systems, asail@jurix 2020, december 9, 2020. ceur workshop proceedings, vol. 2764. ceur-ws.org (2020). https://ceur-ws.org/vol-2764/paper5.pdf ulmer, d., hardmeier, c., frellsen, j.: deep-significance-easy and meaningful statistical significance testing in the age of neural networks. arxiv preprint arxiv:2204.06815 (2022) vaswani, a., et al.: attention is all you need. in: guyon, i., von luxburg, u., bengio, s., wallach, h.m., fergus, r., vishwanathan, s.v.n., garnett, r. (eds.) advances in neural information processing systems 30: annual conference on neural information processing systems 2017, december 4–9, 2017, long beach, ca, usa, pp. 5998–6008 (2017). https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-abstract.html wang, c., nulty, p., lillis, d.: a comparative study on word embeddings in deep learning for text classification. in: proceedings of the 4th international conference on natural language processing and information retrieval, nlpir 2020, pp. 37–46. association for computing machinery, new york(2021). https://doi.org/10.1145/3443279.3443304 yuan, k.h., hayashi, k.: bootstrap approach to inference and power analysis based on three test statistics for covariance structure models. br. j. math. stat. psychol. 56(1), 93–110 (2003). https://doi.org/10.1348/000711003321645368. https://bpspsychub.onlinelibrary.wiley.com/doi/abs/10.1348/000711003321645368 zhang, h., cissé, m., dauphin, y.n., lopez-paz, d.: mixup: beyond empirical risk minimization. in: 6th international conference on learning representations, iclr 2018, vancouver, bc, canada, april 30 may 3, 2018, conference track proceedings. openreview.net (2018). https://openreview.net/forum?id=r1ddp1-rb zhong, h., xiao, c., tu, c., zhang, t., liu, z., sun, m.: how does nlp benefit legal system: a summary of legal artificial intelligence. in: proceedings of the 58th annual meeting of the association for computational linguistics. pp. 5218–5230. association for computational linguistics, online, july 2020. https://doi.org/10.18653/v1/2020.acl-main.466. https://aclanthology.org/2020.acl-main.466 download references author information authors and affiliations instituto federal do rio grande do norte, natal, brazil alexandre gomes de lima universidade federal do rio grande do norte, natal, brazil alexandre gomes de lima & eduardo henrique da s. aranha institut de recherche en informatique de toulouse, umr 5505, cnrs, 31000, toulouse, france josé g. moreno, taoufiq dkaki & mohand boughanem authors alexandre gomes de limaview author publications search author on:pubmed google scholar josé g. morenoview author publications search author on:pubmed google scholar taoufiq dkakiview author publications search author on:pubmed google scholar eduardo henrique da s. aranhaview author publications search author on:pubmed google scholar mohand boughanemview author publications search author on:pubmed google scholar corresponding author correspondence to alexandre gomes de lima . editor information editors and affiliations federal university of são carlos, são carlos, brazil murilo c. naldi centro universitario da fei, são bernardo do campo, brazil reinaldo a. c. bianchi rights and permissions reprints and permissions copyright information © 2023 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper de lima, a.g., moreno, j.g., dkaki, t., da s. aranha, e.h., boughanem, m. (2023). evaluating recent legal rhetorical role labeling approaches supported by transformer encoders. in: naldi, m.c., bianchi, r.a.c. (eds) intelligent systems. bracis 2023. lecture notes in computer science(), vol 14196. springer, cham. https://doi.org/10.1007/978-3-031-45389-2_2 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-45389-2_2 published: 12 october 2023 publisher name: springer, cham print isbn: 978-3-031-45388-5 online isbn: 978-3-031-45389-2 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords indian legal system sentence classification sentence encoding dfcsc publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature llm-driven chest x-ray report generation with a modular, reduced-size architecture | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper llm-driven chest x-ray report generation with a modular, reduced-size architecture conference paper first online: 30 january 2025 pp 199–211 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) llm-driven chest x-ray report generation with a modular, reduced-size architecture download book pdf download book epub talles viana vargas9, helio pedrini9 & andré santanchè9  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 452 accesses abstract large language models (llms) have been widely employed in various text processing tasks. in computer vision, these models have found application in generating captions and text from natural images, as well as in visual question answering (vqa) systems. in the field of medical imaging, there are studies based on text generation proposing automated diagnoses of x-rays, magnetic resonance imaging scans, computed tomography scans, and other modalities. few initiatives seek to apply and harness the potential of llms in medical text generation; they use models with tens of billions of parameters and are thus computationally expensive. this work addresses this gap by evaluating the use of frozen pre-trained models (cxas u-net and biogpt) for chest x-ray report generation. we adapt the blip-2 modular architecture where only a cross-modal alignment module must be trained in order to generate text from images. we were able to achieve competitive scores over clinical efficacy (ce) metrics compared to some state-of-the-art (sota) methods, while obtaining lower scores for natural language generation (nlg) metrics. our findings suggest that nlg metrics may not serve as suitable proxies for evaluating models in the chest x-ray generation task. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a novel transformer-based framework for chest x-ray captioning with clinical sign detection article 04 june 2025 continually tuning a large language model for multi-domain radiology report generation chapter © 2024 a vision–language foundation model for the generation of realistic chest x-ray images article 26 august 2024 1 introduction chest x-rays are a fundamental diagnostic tool in healthcare, containing valuable information that can assist healthcare professionals in diagnosing various pulmonary, cardiac or traumatic conditions [1]. they provide a snapshot of a patient’s internal chest structure, revealing critical details that can guide medical practitioners in their decision-making processes. however, interpreting and extracting the findings from chest x-rays is a complex, repetitive and often time-consuming task. the analysis of these images traditionally relies on the expertise of radiologists, who interpret the visual information contained within the x-rays and then transcribe their findings into textual reports. this manual process is not only labor-intensive but is also prone to variations in interpretation and reporting. it is in this context that innovative solutions are needed to enhance the efficiency, accuracy, and accessibility of chest x-ray analysis [12]. in recent years, the advancements in artificial intelligence (ai) and natural language processing (nlp) are revolutionizing the way humans interact with the machines [20, 31]. some large language models (llms) applications, such as chatgpt, are showing off their exceptional text generation capabilities, being able to output texts with impressive syntax and semantic quality. they also show a good performance while performing tasks they were not previously trained for, for example, one can prompt them to generate a clinical diagnostic, or write a code to solve a given problem. several works are harnessing the developments in ai and nlp to automate and streamline the process of transforming visual information from chest x-rays into informative textual descriptions, however, only a few have capitalized on the potential of llms. these existing methods predominantly rely on resource-intensive models that present challenges in accessibility for those with limited computational resources. therefore, to leverage the capabilities of llms while minimizing the resources required for training and deploying a chest x-ray report generation system, this study proposes adapting the blip-2 modular architecture [16] to utilize frozen in-domain pre-trained models for both vision encoding and text generation. we employ biogpt [18] as our text generation model, which is based on gpt-2 with 354m parameters, offering significantly lower resource requirements compared to models used in other studies [6, 30, 36] and widely-used llms currently used deep learning area [20, 31]. by adopting this modular architecture, where the image encoder and text decoder weights are frozen, only a vision-text alignment module needs to be trained, reducing training complexity and computational costs compared to having to train a large llm. by harnessing the power of llms while seeking computational efficiency, this research contributes to the advancement of ai-powered medical image analysis. this advancement has the potential to reduce radiologists’ workload by serving as a decision support, increasing throughput, minimizing diagnosis delays, and allowing them to focus on complex analyses where their expertise is most valuable [11]. 2 background the literature review focuses on three areas: image captioning, medical captioning, and language models. in the first two areas, we make an comprehensive exploration focusing on different approaches, architectures, benchmarks and applicable metrics for our task. lastly, we investigate some language models, aiming to identify their sizes and number of parameters. 2.1 image captioning the image captioning task targets the automatic generation of description of natural images. in the literature, we find many methods designed to achieve this goal [16, 17, 23, 33, 37, 39]. these works vary based on: their selection of vision encoder, language model, and their approach to cross-domain alignment, which involves aligning features between the image and text domains. vinyals et al. [33] and rennie et al. [23] used cnn models to extract the features from images and lstm network to regressively generate the desired text. the cnn features provide a simple and compact representation of an image, but they can hinder further fine-grained description due to this compression. the lstm network, due to its sequential nature, can be very slow to output the text. zhou et al. [37] and li et al. [17] leveraged the potential of the transformer [32] along region image features extracted by fast-rcnn [9], an widely used cnn for feature extraction. zhou et al. [37], for example, used a shared multi-layer transformer responsible for both vision encoding and text decoding steps, hence this unique module is responsible for aligning the image features and generating the caption. very recent works [16, 39] take advantage of the few shot potential of llms [3]. zhu et al. [39] introduce minigpt-4, which combines a vision transformer image encoder with an open-source llm based on llama [31], aligning both image and text domain using solely a single projection layer. the authors named it minigpt-4 due to its similar capability on description generation compared to gpt-4. built on pre-trained models, li et al. [16] introduced the blip-2 framework, which requires training only an cross-modal alignment module called q-former to bridge the gap between image and text representations. this approach leverages pre-trained image encoders and large language models (llms) by freezing their weights (i.e., not training them). the core component, q-former, is a bert-based module responsible for aligning the features extracted from images and text. by training only the q-former, blip-2 reduces training complexity and computational costs. 2.2 medical captioning medical captioning involves the automatic generation of descriptive and informative captions for medical images, such as x-rays, mri scans, ct scans, and more. the goal is to provide accurate textual descriptions that convey relevant medical information, possibly helping medical professionals make faster and more accurate diagnoses, facilitating research and training, and ultimately improving patient care. considerable progress has been made in this field, with numerous proposed architectures and approaches [4,5,6, 26, 30, 35, 36]. some works, particularly those focused on chest x-ray captioning, employ cnns as image encoders and transformers as text decoders [4, 5]. chen et al. [4] introduced the use of a novel relational memory module alongside the transformer to enhance the caption generation, while chen et al. [5] focused on the alignment of image and text domain, proposing the use of a cross-modal memory network to facilitate the interactions across modalities (image and text). on the other hand, a recent study [35] adopts a vision transformer as its image encoder and introduces expert tokens, which are designed to interact with the extracted images patches and with one another. as a result, each token can focus on a different part of the image. recently, other papers have started to explore the potential of llms for medical caption generation [6, 30, 36]. yang et al. [36] adopted the blip-2 architecture, although they do not follow the original training process, instead, they directly use a large vision encoder and a llm decoder, and fine-tune the q-former and the llm to the imageclef task, which involves caption prediction for general x-ray images. they compared their performance solely based on the task competition ranking. thawkar et al. [30] used the vision transformer from medclip [34] as the vision encoder, applied a simple linear transformation layer to align features, and employed an llm based on llama [31] as the text decoder. this study does not compare their approach with others. danu et al. [6] took a different approach by first multi-classifying diseases on chest x-ray images with bounding boxes, and then using these abnormality features as input for the llm. med-palm [26, 27] is a large-scale generalist biomedical ai system, capable of interpreting various biomedical data modalities, including tasks like chest x-ray report generation and medical visual question answering. it employs a vit-based vision encoder and palm as the llm decoder. building on prior work, nicolson et al. [19] investigated the impact of combining different pre-trained models for chest x-ray report generation. they experimented with various combinations of vision encoders and text decoders, both from the medical domain (in-domain) and general domains. interestingly, their findings revealed that a combination of an out-of-domain vision encoder, cvt-21, and a general-domain text decoder, distilgpt-2, achieved superior performance on chest x-ray report generation benchmarks after fine-tuning. this finding suggests that leveraging out-of-domain models, when carefully chosen and fine-tuned, can lead to state-of-the-art (sota) results. 2.3 language models in recent years, significant advancements have been made in the development of language models. a significant milestone was the introduction of bert [8] (bidirectional encoder representations from transformers), which has 110 million parameters in its base version and 340 million in its large version. bert’s bidirectional training approach on masked language modeling tasks enabled it to achieve state-of-the-art results on a variety of nlp benchmarks. the transformer-based architectures have been further scaled to develop models with even larger parameter sizes. for instance, openai’s gpt-2 [21], with 1.5 billion parameters on its larger model, demonstrated the potential of unsupervised language models to generate coherent and contextually relevant text. gpt-2’s architecture paved the way for its successor, gpt-3 [3], which boasts an unprecedented 175 billion parameters, showcasing remarkable capabilities in few-shot learning and diverse nlp tasks without requiring task-specific fine-tuning. despite their impressive performance, these large-scale models necessitate substantial computational resources for training and inference, limiting their accessibility. this challenge has driven the development of more resource-efficient models. distilbert [24], with 66 million parameters, and albert [14], with 12 million parameters for its base version, offer more compact alternatives that maintain competitive performance levels. in the biomedical domain, specialized language models have been developed to address the unique challenges posed by medical texts. biobert [15], based on bert with approximately 110 million parameters, and clinicalbert [2], also based on bert, have been fine-tuned on large corpora of biomedical literature and clinical notes, resulting in improved performance on domain-specific tasks. following the llms line, biogpt [18], a model derived from gpt-2 with 354 million parameters, specifically tailored for biomedical text generation, provides a balanced approach, offering the extensive capabilities of larger models while maintaining lower resource requirements. this efficiency makes biogpt a practical choice for potentially generating high-quality biomedical text, ensuring accessibility for a broader range of researchers. wrapping up, we observe that many works in medical captioning, specifically chest x-ray captioning, are focused on enhancing performance metrics. furthermore, those concentrating on harnessing the power of llms do not appear to prioritize the search for reduced-size architectures that would enable simple computers with limited computational resources to run these models. 3 materials this work adopts the mimic-cxr dataset [13], a widely-used dataset for generating captions for chest x-rays. it is currently the largest publicfootnote 1 chest x-ray dataset, containing 377,100 radiograms and 227,835 free-text reports. the dataset has an official data split allowing fair comparison of our models against literature methods and different techniques. most related works also utilize the iu xray dataset [7]. however, in this study, we have chosen to use only the mimic-cxr due to its larger size. in the future, we plan to include additional datasets to assess generalization. since mimic-cxr includes sample reports with multiple views, such as frontal and lateral x-rays, we employed the following strategy for inference: for reports with two images: we randomly selected two images for analysis. for reports with only one image: we duplicated the single image to ensure the pipeline could process it consistently. following this selection process, all images were resized to a standard dimension of 512\(\times \)512 pixels. additionally, normalization was applied using the mean and variance values specific to the chosen vision encoder. 4 method this section presents the proposed model architecture, the training stages, and the metrics used during the evaluation process. model architecture. this works leverages the architecture and training pipeline presented by blip-2 [16], a captioning model for natural images that utilizes frozen pre-trained models. the key advantage of this architecture is its modularity. this allows for the integration of different vision encoders and text decoders, with only the alignment module (q-former) requiring specific training. the source code was based and adapted from blip-2 implementation on lavis repositoryfootnote 2. image encoder stage. the architecture, as illustrated in fig. 1, starts with the image encoder, responsible for extracting pertinent features from chest x-ray images. fig. 1. proposed architecture for chest x-ray report generation. full size image this work adopted a u-net [25] formerly pre-trained on a chest x-ray segmentation dataset, therefore we can leverage its prior in-domain knowledge. this choice offers a significant advantage, as u-net has nearly 15 times fewer parameters compared to the vision transformer (vit-g/14) employed in the original blip-2 work. at this point, we utilized the feature map output of the first upsampling layer in the network, flattened the height and width dimensions, and then switched channels dimension with this flattened spatial dimension. this allows having features that are ready to be processed by the next neural network. regarding the image feature extraction, the study utilizes two images for every sample as aforementioned, hence we followed other studies [4, 35], sequentially extracting features from both images using the selected image encoder, and averaging those features. q-former alignment module. the cornerstone of the architecture, the q-former is responsible for aligning extracted images features with the text decoder input. as the q-former is a bert-based model, we leveraged available bioclinicalbert [2] pre-trained weights, which was trained on pubmed abstracts and also on mimic iii notes. initializing language models with in-domain pre-trained weights, instead of using the standard bert initialization, potentially boosts the capacity of feature alignment of the model; behavior seen in other medical tasks [15]. we used the same configuration from the base source code. text decoder module. the text decoder is the module responsible for generating the chest x-ray descriptions given the features aligned by the q-former. similar to blip-2 architecture, we adopted a large language model as our text decoder, due to its text generation capability. generally, an llm receives text prompts on its input which work as an start point, from where the model generates the text output. however, in this work, as the llm inputs come from the q-former aligned features, these inputs can be considered as soft prompts, i.e., they are not explicit text embeddings, but a feature vector that must help and guide the llm decoder on its text generation task. this work utilizes biogpt [18], an llm based on gpt-2 [21] architecture, since it was specifically pre-trained on biomedical domain using pubmed abstracts. it is a lightweight model compared to the llm from the original blip-2, containing 345 million parameters instead of 2.7 billion. 4.1 training stages the training process is divided into two distinct stages, each serving a specific purpose in enhancing the capabilities of a portion of our architecture, as illustrated in fig. 1. representation learning. in this initial training stage, the q-former is trained in conjunction with a frozen pre-trained image encoder. the primary objective is to enable the q-former to serve as a bridge between image and text domains. this step prepares the module to align, understand, and also, generate meaningful captions, enhancing the performance of the architecture as a whole. during this phase, the q-former outputs are optimized using three distinct methods (cost functions), as outlined in the blip-2 work [16]. we adopted adamw optimizer with weight decay of 0.05. the learning rate followed a cosine decay schedule during 10 epochs, starting at 1e-4 and ending at 1e-5. a warm-up period of one epoch was employed with learning rate set to 1e-6. also, the best model checkpoint was selected based on the lowest loss overall. generative learning. in the second training stage, our whole architecture is assembled. the previously trained q-former module is integrated with the frozen pre-trained biogpt. subsequently, the q-former is fine-tuned by optimizing the outputs generated by the llm decoder. during this fine-tuning process, the q-former acts as a guide, influencing biogpt’s text generation based on the input chest x-ray images. we adopted adamw optimizer with weight decay of 0.05. the learning rate followed a cosine decay schedule during 10 epochs, starting at 1e-5 and ending at 1e-6. a warm-up period of two epochs was employed with learning rate set to 1e-8. at this point, we observed better results by unfreezing the biogpt and freezing all the other modules. same as before, the final model checkpoint was selected based on the lowest loss value. all training stages were performed on aws cloud using a ml.g5.xlarge instance containing a 24gb gpu and 16gb memory. 4.2 metrics and evaluation following chen et al. [4, 35, 38], our evaluation aimed to assess both the quality of the generated text descriptions and their potential clinical usefulness. we adopted two approaches, leveraging established natural language generation (nlg) metrics (bleu, meteor and rouge-l) and additionally, we incorporated clinical efficacy (ce), where we employed the chexbert system [28] to automatically label the generated reports within 14 categories. by comparing these labels with the ground truth diagnoses, we calculated precision, recall, and f1-score, providing insights into the model’s ability to generate clinically accurate reports. this work presents both micro and macro averages of precision, recall and f1-score [29]. 5 experimental results this section presents the findings of our evaluation on the effectiveness of the proposed method for generating clinical text reports. we assessed performance using a combination of natural language generation (nlg) metrics and clinical efficacy (ce) metrics over the mimic-cxr dataset. our architecture consists of approximately 450 million parameters. in comparison with other techniques that also adopt llm in the decoder stage, xraygpt [30] has around 7 billion parameters solely in its large language model (llm). similarly, yang et al. [36] employs the chatglm-6b llm model, which comprises approximately 6 billion parameters. med-palm [26], on the other hand, incorporates 540 billion parameters in its palm llm. unfortunately, these studies do not evaluate their models using commonly used datasets, thus our comparison is limited to the number of parameters. in order to quantitatively compare our approach with other studies in the area, we evaluated the clinical efficacy of the reports using chexbert labels. precision, recall, and f1-score were calculated for each chexbert finding. table 1 summarizes the overall ce performance compared to existing methods. our method achieved the highest precision (0.504) among the evaluated methods, indicating a low rate of false positive findings. however, the recall score (0.346) suggests it may overlook some relevant findings compared to cvt21-2distilgpt2 [19]. table 1. comparison of the performance of different methods using precision, recall, and f1 metrics.full size table table 2 presents a comparison of nlg metrics between other techniques and our model. our model has low scores if compared to other techniques. this could be explained due to the fact the those nlg metrics were developed to mainly evaluate machine translation, which is clearly not the task here. since our core model is a pre-trained gpt-2 model, our predictions may not align as closely with the ground truth. consequently, this misalignment could account for the observed low scores in our proposed model’s metrics. table 2. comparison of the nlg metrics between our model and other techniques.full size table a further analysis revealed an interesting nuance between clinical efficacy (ce) metrics and natural language generation (nlg) metrics in the context of chest x-ray report generation. specifically, within a subset of high-performing predictions based on ce metrics—where predictions effectively conveyed observations from the x-rays—the corresponding nlg metrics displayed lower scores (see table 3). this observation suggests that nlg metrics alone may not be the most reliable indicator of model quality for this task. table 3. nlg metric scores (e.g., bleu, rouge-l) for a subset of model predictions that achieved the highest precision, recall, and f1-score in the clinical efficacy (ce) evaluation. low nlg scores on these high-performing predictions suggest that nlg metrics alone may not be the most reliable indicator of model quality for chest x-ray report generation.full size table table 4. clinical efficacy (ce) metrics for each observation within the mimic-cxr test set. the positive cases and negative cases columns indicate the distribution of positive and negative labels in the ground truth data, allowing for verification of class imbalance.full size table even predictions which are clinically accurate by ce metrics might not receive high nlg scores. this highlights the limitations of nlg metrics in capturing the full spectrum of factors that contribute to good clinical reports, such as factual accuracy, nuanced phrasing, and adherence to reporting conventions. finally, as shown in table 4, the model performs well on identifying findings such as support devices (0.683 f1) and cardiomegaly (0.521 f1). however, it struggles with less prevalent findings like atelectasis (0.330 f1) and lung opacity (0.229 f1). this may be due to limitations in the training data or the model’s capacity to capture subtle variations in these specific findings. these results demonstrate the potential of our proposed method for generating clinically relevant and accurate text reports. the high precision ensures a low false positive rate, while further investigation is needed to improve recall, particularly for less frequent findings. 6 conclusions in conclusion, our work presented an approach for generating clinical text reports from chest x-ray images. the selection of relative small-sized models, and the modular characteristic of blip-2 has made the training process less-hardware intensive, and we hope it foster future investigations of this type of architecture. our method achieved the highest precision among the evaluated techniques. however, our evaluations also reveals challenges to achieve a high recall, particularly for less prevalent observations. hence, the model needs further investigations to improve model’s capacity. furthermore, our analysis of the nlg metrics highlights the limitations of using machine translation metrics for our specific task, which do not capture the actual quality of the report. future efforts should focus on developing custom evaluation methods that align with our specific goals. overall, our results demonstrate the potential of using in-domain, pre-trained, reduced-size models to generate clinically relevant and accurate text reports. however, ongoing research and development efforts are still needed to improve model’s recall, to refine to overall’s capability of the model. with this improvements, this approach could become a valuable tool, contributing to more efficient and accurate diagnosis. as future directions, we aim to assess this framework with other datasets, such as brax [22] and iu xray [7], to validate model’s generalization. notes 1.mimic-cxr webpage: https://physionet.org/. 2.lavis repository: https://github.com/salesforce/lavis. references al-dasuqi, k., johnson, m.h., cavallo, j.j.: use of artificial intelligence in emergency radiology: an overview of current applications, challenges, and opportunities. clin. imaging 89, 61–67 (2022) article  math  google scholar  alsentzer, e., murphy, j.r., boag, w., weng, w.-h., jin, d., naumann, t., mcdermott, m.: publicly available clinical bert embeddings. arxiv preprint arxiv:1904.03323 (2019) brown, t., et al.: language models are few-shot learners. adv. neural. inf. process. syst. 33, 1877–1901 (2020) math  google scholar  chen, z., song, y., chang, t.-h., wan, x.: generating radiology reports via memory-driven transformer. arxiv preprint arxiv:2010.16056, 2020 chen, z., shen, y., song, y., wan, x.: cross-modal memory networks for radiology report generation. arxiv preprint arxiv:2204.13258, 2022 danu, m.d., et al.: generation of radiology findings in chest x-ray by leveraging collaborative knowledge. arxiv preprint arxiv:2306.10448 (2023) demner-fushman, d., et al.: preparing a collection of radiology examinations for distribution and retrieval. j. am. med. inf. assoc. 23(2), 304–310 (2016) article  google scholar  devlin, j., chang, m.-w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. arxiv preprint arxiv:1810.04805 (2018) girshick, r.: fast r-cnn. in: ieee international conference on computer vision, pp. 1440–1448 (2015) google scholar  gu, t., liu, d., li, z., cai, w.: complex organ mask guided radiology report generation. in: ieee/cvf winter conference on applications of computer vision, pp. 7995–8004 (2024) google scholar  irmici, g., et al.: chest x-ray in emergency radiology: what artificial intelligence applications are available? diagnostics 13(2), 216 (2023) article  google scholar  jalal, s., parker, w., ferguson, d., nicolaou, s.: exploring the role of artificial intelligence in an emergency and trauma radiology department. can. assoc. radiol. j. 72(1), 167–174 (2021) article  google scholar  johnson, a.e., et al.: mimic-cxr-jpg, a large publicly available database of labeled chest radiographs. arxiv preprint arxiv:1901.07042 (2019) lan, z., et al.: albert: a lite bert for self-supervised learning of language representations. arxiv preprint arxiv:1909.11942 (2019) lee, j., et al.: biobert: a pre-trained biomedical language representation model for biomedical text mining. bioinformatics 36(4), 1234–1240 (2020) article  mathscinet  math  google scholar  li, j., li, d., savarese, s., hoi, s.: blip-2: bootstrapping language-image pre-training with frozen image encoders and large language models. arxiv preprint arxiv:2301.12597 (2023) li, x., et al.: oscar: object-semantics aligned pre-training for vision-language tasks. in: vedaldi, a., bischof, h., brox, t., frahm, j.-m. (eds.) eccv 2020. lncs, vol. 12375, pp. 121–137. springer, cham (2020). https://doi.org/10.1007/978-3-030-58577-8_8 chapter  math  google scholar  luo, r., et al.: biogpt: generative pre-trained transformer for biomedical text generation and mining. brief. bioinform. 23(6), 409–421 (2022) article  math  google scholar  nicolson, a., dowling, j., koopman, b.: improving chest x-ray report generation by leveraging warm starting. artif. intell. med. 144, 102633 (2023) article  google scholar  openai. gpt-4 technical report (2023) google scholar  radford, a., wu, j., child, r., luan, d., amodei, d., sutskever, i., et al.: language models are unsupervised multitask learners. openai blog 1(8), 9 (2019) google scholar  reis, e.p., et al.: brax, brazilian labeled chest x-ray dataset. sci. data 9(1), 487 (2022) article  math  google scholar  rennie, s.j., marcheret, e., mroueh, y., ross, j., goel, v.: self-critical sequence training for image captioning. in: ieee conference on computer vision and pattern recognition (2017) google scholar  sanh, v., debut, l., chaumond, j., wolf, t.: distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arxiv preprint arxiv:1910.01108 (2019) seibold, c., et al.: accurate fine-grained segmentation of human anatomy in radiographs via volumetric pseudo-labeling. arxiv preprint arxiv:2306.03934 (2023) singhal, k., et al.: large language models encode clinical knowledge. arxiv preprint arxiv:2212.13138 (2022) singhal, k., et al.: towards expert-level medical question answering with large language models. arxiv preprint arxiv:2305.09617 (2023) smit, a., et al.: chexbert: combining automatic labelers and expert annotations for accurate radiology report labeling using bert. arxiv preprint arxiv:2004.09167 (2020) sorower, m.s.: a literature survey on algorithms for multi-label learning. oregon state university, corvallis 18(1), 25 (2010) google scholar  thawkar, o., et al.: xraygpt: chest radiographs summarization using medical vision-language models. arxiv preprint arxiv:2306.07971 (2023) touvron, h., et al.: llama 2: open foundation and fine-tuned chat models. arxiv preprint arxiv:2307.09288 (2023) vaswani, a., et al.: attention is all you need. adv. neural inf. process. syst. 30 (2017) google scholar  vinyals, o., toshev, a., bengio, s., erhan, d.: show and tell: a neural image caption generator. in: ieee conference on computer vision and pattern recognition, pp. 3156–3164 (2015) google scholar  wang, z., wu, z., agarwal, d., sun, j.: medclip: contrastive learning from unpaired medical images and text. arxiv preprint arxiv:2210.10163 (2022) wang, z., liu, l., wang, l., zhou, l.: metransformer: radiology report generation by transformer with multiple learnable expert tokens. in: ieee/cvf conference on computer vision and pattern recognition, pp. 11558–11567 (2023) google scholar  yang, b., raza, a., zou, y., zhang, t.: customizing general-purpose foundation models for medical report generation. arxiv preprint arxiv:2306.05642 (2023) zhou, l., palangi, h., zhang, l., hu, h., corso, j., gao, j.: unified vision-language pre-training for image captioning and vqa. in: aaai conference on artificial intelligence, vol. 34, pp. 13041–13049 (2020) google scholar  zhou, z., shi, m., wei, m., alabi, o., yue, z., vercauteren, t.: large model driven radiology report generation with clinical quality reinforcement learning. arxiv preprint arxiv:2403.06728 (2024) zhu, d., chen, j., shen, x., li, x., elhoseiny, m.: minigpt-4: enhancing vision-language understanding with advanced large language models. arxiv preprint arxiv:2304.10592 (2023) download references author information authors and affiliations institute of computing (ic), university of campinas, campinas, sp, brazil talles viana vargas, helio pedrini & andré santanchè authors talles viana vargasview author publications search author on:pubmed google scholar helio pedriniview author publications search author on:pubmed google scholar andré santanchèview author publications search author on:pubmed google scholar corresponding author correspondence to helio pedrini . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper viana vargas, t., pedrini, h., santanchè, a. (2025). llm-driven chest x-ray report generation with a modular, reduced-size architecture. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_14 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_14 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords radiology report generation language models lightweight training chest x-rays publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature anomalies diagnostic in endoscopic images using deep learning ensemble models | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper anomalies diagnostic in endoscopic images using deep learning ensemble models conference paper first online: 30 january 2025 pp 110–124 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) anomalies diagnostic in endoscopic images using deep learning ensemble models download book pdf download book epub pedro da s. viana  orcid: orcid.org/0009-0001-3104-56379, luana b. da cruz  orcid: orcid.org/0000-0002-9150-25539, domingos a. dias jr.  orcid: orcid.org/0000-0001-5916-69519 & … joão otávio bandeira diniz  orcid: orcid.org/0000-0003-3303-334610  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 384 accesses 2 citations abstract automated anomaly detection in medical images greatly benefits the medical community by assisting in the diagnosis and identification of clinical cases. this work focuses specifically on medical images obtained from endoscopies. we categorized the images into two classes: normal (including normal-cecum, normal-pylorus, normal-z-line) and abnormal (including dyed-lifted-polyps, dyed-resection-margins, esophagitis, polyps, ulcerative colitis). we propose data augmentation to balance the classes and deep learning model techniques for classification. we employed an ensemble voting method with deep learning models to enhance the accuracy of the process. our results are robust, achieving an accuracy of 98.62%, a precision of 96.47%, a sensitivity of 99.64%, and an f1-score of 98.03%. this approach proves to be an effective tool for identifying anomalies in endoscopies. we believe our method can contribute to significant advancements in ai-assisted medical diagnostics. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others deep learning-based endoscopic anatomy classification: an accelerated approach for data preparation and model validation article 29 september 2021 custom descriptors applications for endoscopy images classification chapter © 2025 deep learning on special processed video colonoscopy datasets chapter © 2022 1 introduction the digestive system is responsible for nutrient absorption, involving physical and chemical processes from chewing to waste excretion. however, abnormalities can occur, impacting its function. these abnormalities can range from relatively benign conditions to severe diseases that require medical intervention. two notable examples are polyps and ulcerative colitis [3, 20]. polyps are small abnormal growths that resemble mushrooms and appear in various regions of the body, such as the anus and intestine. although considered benign tumors, identifying them is crucial, as their presence can indicate cancer in the area where they are found [18]. ulcerative colitis is an intestinal inflammation present in the colon, which can increase the chances of developing cancer. early identification of these abnormalities is beneficial, as it can be essential for the early diagnosis of stomach cancer [20]. stomach cancer ranks as one of the most frequent malignant neoplasms worldwide, holding the third position in terms of incidence. this aggressive form of cancer primarily affects individuals over the age of 50 and those with obesity. early detection is crucial for effective treatment and improved patient prognosis. physicians use endoscopic procedures to identify abnormalities in the digestive tract, allowing detailed visualization and early intervention in suspicious lesions [14]. to identify these abnormalities, endoscopists utilize two main procedures: upper gastrointestinal endoscopy (ugie) and lower gastrointestinal endoscopy (lgie). in ugie, the physician introduces an endoscope through the patient’s mouth, allowing direct visualization of the esophagus, stomach, and duodenum mucosa via images. lgie follows the same principle, but the endoscope is inserted through the patient’s anus to visualize the lower portion of the intestine, such as the colon [16]. one of the primary challenges encountered by endoscopists is the substantial volume of data generated during ugie. this influx of information can be effectively managed using computational systems like computer-aided detection (cad) and computer-aided diagnosis (cadx). these systems facilitate disease analysis, thereby accelerating and enhancing the diagnostic process [1]. the literature documents various computational methods developed for disease detection and diagnosis using medical images [2, 4,5,6, 8, 15]. based on the problem described previously, the proposed method aims to identify specific abnormalities, such as polyps, ulcerative colitis, esophagitis, dyed-lifted-polyps, and dyed-resection-margins, in ugie and lgie exams. this method is based on convolutional neural networks (cnns), deep learning techniques, and region of interest (roi) approach. the contributions of the proposed method include: ensemble voting of multiple neural networks, which results in superior performance across evaluated metrics; implementation of an enhanced roi approach, which improves accuracy in the abnormalities classification. 2 related works in this section, we present relevant articles that discuss methods similar to the one presented in this study. in the work of [17], they focus on ensemble voting methods with rgb channel extraction for bleeding area segmentation. this approach aims to improve the classification of the bleeding, healthy, and ulcer classes. additionally, the article tests the ensemble voting method with several ensemble methods, including f-tree, l-disc, l-svm, c-svm, m-g-svm, and f-knn. the objective is to differentiate among eight distinct classes: dyed-lifted-polyp (dlp), dyed-resection-margins (drm), esophagitis (eso), normal-cecum (nc), normal-pylorus (np), normal-z-line (nzl), polyps (p), and ulcerative-colitis (uce). the results obtained by this multiclass model on the kvasir v1 dataset [19] were remarkable, achieving an accuracy of 86.6%, precision of 87.8%, f1-score of 86.8%, specificity of 98.08%, and sensitivity of 86.62%. in the study proposed by [22], the emphasis is on pooling methods aimed at improving accuracy in the binary classification between the polyp and non-polyp categories. the pooling method that yielded the best results was dwt-pool, achieving an accuracy, recall, precision, and f1-score of 93.55%. this result was obtained using the kvasir v1 dataset. additionally, data augmentation techniques were employed to augment the image dataset. the method proposed by [12] presents a method using a dataset with 14 distinct classes. the primary strategy was to categorize these classes into two main groups: normal and abnormal. this method proposes performance metric improvements by simplifying the problem from multiclass to binary classification. the results were promising considering the number of classes, with an accuracy of 76.02%, sensitivity of 78.9%, precision of 56.2%, and f1-score of 59.3%. it is noteworthy that these results were achieved using the kvasir capsule dataset [24] and the resnet-50 architecture as the foundation for the machine learning model. in the referenced work [26], researchers utilized cnn models vgg-16, googlenet, and resnet-18 to classify various categories: dlp, drm, eso, nc, np, nzl, p, and uce. they conducted an efficacy comparison among these neural networks. particularly, the vgg-16 network achieved superior results, with an accuracy of 96.33%, sensitivity of 96.37%, precision of 96.5%, and an f1-score of 96.5%. these results were obtained using the kavasir v2 image dataset [19] and data augmentation techniques, demonstrating the effectiveness of cnns in medical image classification. in [10], the study focuses on the kvasir v2 dataset, which includes 8000 images across categories dlp, drm, eso, nc, np, nzl, p, and uce. the methodology involved applying data augmentation techniques alongside traditional cnns and a fine-tuning process. the vgg-16 network yielded the most significant results, achieving an accuracy of 98.20%, recall of 92.75%, specificity of 99%, matthews correlation coefficient of 91.73%, precision of 92.86%, and an f1-score of 92.76%. binary classification between normal and abnormal findings in medical examinations such as endoscopy is crucial for optimizing screening processes and early intervention. by focusing on quickly identifying patients with abnormal results, we can allocate medical resources more efficiently, ensuring these individuals receive timely assessment and treatment. it improves patient health outcomes by enabling early interventions that can reduce future complications and helps alleviate the burden on healthcare systems by prioritizing cases needing immediate attention. however, there is a noticeable scarcity of binary classification approaches for this problem in the literature. addressing this need, we propose a binary method that distinguishes between normal and abnormal conditions. our proposed method includes edge cropping to remove unnecessary information, uses data augmentation techniques to balance classes, and compensates for the dataset’s limited image count. furthermore, we introduce an ensemble approach based on voting with multiple cnns to improve classification performance. this method aims to introduce a novel approach not extensively explored in the literature, offering an automated system to aid specialist physicians in diagnosing these abnormalities. 3 materials and proposed method the proposed method comprises four stages, validated using the database presented in sect. 3.1. we remove all image edges through an roi process in the first step. in the second step, we apply data augmentation techniques to diversify and expand the database. the third stage involves training several cnns and combining through an ensemble voting process. in the fourth stage, we construct a predictive model using ensemble voting, and subsequently, in the fifth stage, we calculate validation metrics to measure the method’s robustness. figure 1 provides a described process visual illustration. fig. 1. proposed method illustration. full size image 3.1 image database in this step, we gathered data from the public kvasir v1 database on kaggle [19]. this dataset contains 4000 images spread across eight distinct classes: dlp, drm, eso, nc, np, nzl, p, and uce, with each class comprising 500 images. the images vary in dimensions from 730\(\,\times \,\)576 to 1920\(\,\times \,\)1072 pixels, all in jpg format. however, our method distinguishes between normal and abnormal classes. the normal includes nc, np, and nzl, totaling 1500 images. the abnormal classes consist of dlp, drm, eso, p, and uce, totaling 2500 images. figure 2 illustrates the separation of classes in the image database. fig. 2. image database separation classes into categories: (a) abnormal and (b) normal. full size image 3.2 roi extraction some images in the kvasir dataset have black borders used for annotations by specialists. however, these areas are not essential for the classifier and can even confuse it. therefore, we opted to extract the roi from the image. in practice, this involves image cropping, leaving only a square region without black pixels in the central areas of each side of the square. figure 3 illustrates the roi extraction presented. fig. 3. roi extraction: (a) original image; (b) marking with a red bounding box; and (c) resulting image. (color figure online) full size image 3.3 data augmentation and class balancing in this section, the process of class balancing and data augmentation used in the proposed method will be demonstrated. balancing the classes was necessary due to the disparity between them, with the normal class having 1500 images and the abnormal class having 2500 images. balancing is important because when a class is a majority, the classifier can develop a bias towards it, reducing precision. initially, the image base was randomly divided into training and test sets. subsequently, a difference of 742 images was observed between the two classes in the training set. to correct this imbalance, a data augmentation process was performed to generate 742 additional images in the minority class, thus balancing the classes in the training set. in addition, to reduce overfitting, a data augmentation process was performed, which includes a rotation of a maximum of 10% to the right or left and a vertical or horizontal mirroring, generating 1000 images in each class. figure 4 provides visual examples of these data augmentation techniques applied to the test dataset. 3.4 ensemble voting we chose the hard voting ensemble [7] for this study, which involves training multiple models and then conducting a voting process to determine the normal or abnormal class based on the majority of votes. for example, if we train five models and three of them predict that a particular image belongs to the normal class, then the image will be classified as normal. fig. 4. samples of data augmentation. full size image in this context, we individually trained the following cnns, each chosen for their unique characteristics and strengths: vgg-19 [23]: known for its deep architecture with 19 layers, vgg-19 achieved one of the highest results in sensitivity among the tested networks. despite this, it showed lower precision for one of the classes, indicating that while it is effective at general classification tasks, it might struggle with fine-grained distinctions. mobilenetv2 [21]: this network is designed to be both efficient and powerful, employing depthwise separable convolutions to reduce the number of parameters and computational costs while maintaining performance. mobilenetv2 significantly improved roi extraction, achieving results comparable to heavier networks. resnet-50 [11]: famous for its use of residual blocks to combat the vanishing gradient problem. this characteristic makes it particularly adept at learning intricate features, leading to fewer false positives in the classification task. densenet-121 [13]: with its dense connections that encourage feature reuse throughout the network, densenet-121 notably enhanced its performance with roi extraction and data augmentation. it showed the most substantial improvement, particularly in accuracy and sensitivity, making it excellent at detecting true positives. efficientnetb0 [25]: this model is designed to achieve better performance with fewer parameters by scaling depth, width, and resolution in a balanced manner. after roi extraction and data augmentation, efficientnetb0 demonstrated a good trade-off between accuracy, precision, and sensitivity. each of these networks contributes uniquely to our study, providing a better understanding of how different architectures respond to techniques like roi extraction and data augmentation. we trained all models using same randomly selected images with identical seed and data augmentation parameters. figure 5 visually illustrates the described ensemble voting method. fig. 5. sample of ensemble voting. full size image after training, the models make predictions based on voting in the test set. the normal or abnormal class is then determined by the majority of the models’ votes. this procedure ensures greater robustness and accuracy in classification, leveraging the diversity and complementarity of the models used. 3.5 validation metrics at the end of the process, we present the results using validation metrics to assess the method’s effectiveness. in this study, we adopted metrics such as accuracy (which measures the overall correct predictions considering all observations), sensitivity (which measures the proportion of true positives among all actual positives), precision (which measures the proportion of true positives among all positive predictions), and f1-score (which measures the harmonic mean of precision and sensitivity) [9]. 4 results and discussion in this section, we will explore the experiments conducted at each stage of the proposed method, the configurations used, and a case study demonstrating the evolution of each stage of the method employed. this detailed analysis will provide a deeper understanding of the process and the tools involved. 4.1 experimental setup the models were trained in a virtual environment on google colab using a jupyter notebook. the hardware specifications included nvidia a100 gpu and nvidia t4 tensor core gpu. development was carried out in python, primarily utilizing the libraries opencv, pandas, scikit-learn, and keras. 4.2 method experimental results we performed several experiments to validate each stage of the proposed method. it’s important to note that we opted to conduct each experiment using only densenet-121, taking into account the training cost of all architectures. the implementation adheres to keras library conventions, with specific modifications in the final layers: initially, a flatten layer was added to connect the convolutional layers to the fully connected layers. subsequently, a dense layer with 256 neurons and rectified linear unit (relu) activation was incorporated, followed by a 50% dropout layer to mitigate overfitting. finally, a dense layer with one neuron and sigmoid activation was introduced to produce the output. the hyperparameters utilized included an adam optimizer with a 0.001 learning rate and early stopping after 50 epochs based on accuracy. all other models followed the same standards and hyperparameters. furthermore, the dataset was splitted into three subsets: training (70%), validation (10%), and testing (20%). the training and validation subsets were utilized for model training, while the testing subset was reserved for method validation. experiment with and without roi extraction. in this step, we conducted two experiments to validate the roi extraction step. the first experiment used only the original images without any modifications or data augmentation. in the second experiment, we applied roi extraction to the images to validate this technique. table 1 summarizes the results. table 1. roi extration results.full size table the results unequivocally indicate significant improvements across all metrics when employing roi extraction. this improvement suggests that the removed areas likely contained irrelevant information that could potentially confuse the classifier. by eliminating these unnecessary areas, the model achieved superior generalization capability, resulting in notably enhanced accuracy and efficiency in the endoscopic classification images. this validation underscores the critical role of roi extraction in optimizing model performance and ensuring robustness in medical image analysis applications. data augmentation experiment with class balancing. in sect. 3.3, we discuss the addition of the data augmentation technique used in this study. to validate our approach, we conducted two experiments: the first experiment did not employ data augmentation, while the second experiment utilized these techniques. the experiments were performed using the densenet-121 model. the results of these experiments are summarized in table 2. table 2. data augmentation results.full size table as evidenced by the results in table 2, the implementation of data augmentation resulted in substantial improvements across all evaluated metrics. these enhancements underscore the crucial role of data augmentation in improving the robustness of our proposed method. by enriching the dataset’s diversity and enhancing the model’s capacity to generalize intricate patterns in endoscopic images, this technique has proven instrumental. it effectively mitigates overfitting while facilitating more accurate and reliable classification outcomes. thus, integrating data augmentation emerges not only as a methodological imperative but also as a strategic enhancement that fortifies the overall efficacy of our approach in clinical applications. individual cnns vs ensemble voting experiment. finally, we validated the ensemble method using the cnns vgg-19, mobilenetv2, resnet-50, densenet-121 and efficientnetb0. in general, all steps of the method were maintained, just replacing densenet-121 with the mentioned cnns. initially, we present the individual results of these cnns, as well as the performance of the ensemble voting method that combines these networks (table 3). subsequently, we present all experiments conducted for the final method validation (using densenet-121), demonstrating its robustness and highlighting the contribution of each stage to its effectiveness (table 4). table 3. results of the individual cnns and ensemble voting.full size table analyzing the table 3 we can see that ensemble voting significantly outperforms each cnn in all aspects evaluated. the increase in accuracy, precision, sensitivity, and f1-score indicates that the combination of multiple cnns allows us to capture the characteristics of the anomalies better, taking advantage of the strengths of each model and compensating for their weaknesses. table 4. results of the evolution of the proposed method.full size table table 4 presents the evolution of the proposed method showing the results of each of the steps using densenet-121. the first step was extracting the roi, which significantly improved all metrics. following this experiment, data augmentation was employed to balance both classes and generate more images, greatly enhancing the metrics for all models. finally, ensemble voting was performed, combining features from various distinct cnns to achieve a more robust model. in this method, densenet-121 provided the highest accuracy among all models; resnet-50 achieved the best precision; densenet-121 and mobilenetv2 showed exponential improvements in metrics with roi extraction and data augmentation contributing to enhanced sensitivity; and efficientnetb0 yielded balanced results, improving the f1-score. each cnn showcased unique characteristics amplified with roi extraction and data augmentation, culminating in a robust ensemble method that outperforms individual model performance. 4.3 case study in this section, we present two case studies analyzing the misclassification of stomach images. the first involves a normal image erroneously classified as abnormal (fig. 6 (a)), while the second deals with an abnormal image classified as normal (fig. 6 (b)). fig. 6. case study. (a) normal image, classified as abnormal by the method; and (b) abnormal image, classified as normal by the method. full size image by analyzing fig. 6 (a), which shows an nc image, we observe that despite being a normal image, it contains some yellowish residues that may have confused the classifier. these residues are likely food remnants or bile that visually resemble pathological features. in contrast, fig. 6 (b) depicts a uce image incorrectly classified as normal. the classifier might have been misled by the absence of mucus and bleeding, which are typically present in uce images, resulting in the incorrect classification as normal. despite these misclassifications, we consider the model to be robust. when combined with medical expertise, it can play a crucial role in detecting stomach abnormalities. 4.4 related works comparison this section proposes a comparative analysis with the studies discussed in sect. 2. it is important to note that the methodologies employed in various studies are significantly different from each other, making this comparison not absolute but rather an attempt to establish a comparative parameter. table 5 provides a summary of the metrics used for this comparison. table 5. comparison of the proposed method with related works.full size table several studies in the field employ cnn to detect abnormalities in endoscopic examinations. for instance, researchers such as [26] and [10] extensively test various architectures to identify those that yield superior performance metrics, aiming to integrate these architectures into their methodologies. while this approach is valuable, it’s important to note that not all architectures consistently outperform others; different architectures often possess unique strengths that can complement each other, as demonstrated in ensemble methods. the work by [17] utilizes an ensemble method to combine multiple classifiers and extract features from each. however, this method is constrained by its reliance solely on the dataset and does not utilize any data augmentation step. in contrast, the proposed method presented in this study demonstrates significant improvements in model performance by leveraging data augmentation techniques. an exemplary study illustrating the benefits of data augmentation is [12], which employs this approach to balance classes in a binary classification (normal and abnormal). this research underscores that addressing problems in binary classification can enhance performance metrics, utilizing cnns for testing and highlighting the strengths of each network. if these studies were to adopt an ensemble method similar to the one presented, they could effectively leverage the positive attributes of multiple models. additionally, the study by [22] focuses on a binary approach for polyp detection, achieving an accuracy of 93.55%. in contrast, the proposed method achieves an accuracy of 98.52%, even with the inclusion of four additional abnormalities. a comparison with existing literature reveals that while cnn-based approaches have demonstrated effectiveness in detecting gastrointestinal abnormalities, there remains ample opportunity for improvement, particularly in integrating diverse architectures and optimizing data augmentation strategies. this study contributes significantly by integrating ensemble methods with class balancing, data augmentation, and roi extraction, resulting in a more robust and precise model. consequently, combining this approach with medical expertise holds promise for enhancing the early detection of gastrointestinal abnormalities. 5 conclusion the proposed method combines roi extraction with data augmentation and class balancing. furthermore, it employs an ensemble voting of cnns applying a binary approach to the problem that remains underexplored in the literature. ensemble voting integrates decisions from multiple cnn models, enhancing the diagnostic capability of the method. the impressive results, with an accuracy of 98.62%, precision of 96.47%, and an f1-score of 98.03%, highlight the method’s effectiveness in detecting abnormalities in ugie and lgie exams. for future work, we suggest validating the method across diverse datasets, automating the optimization of network hyperparameters, and further exploring a broader range of cnns and ensemble techniques. these steps aim to enhance diagnostic accuracy and improve the robustness of results. references alagappan, m., brown, j.r.g., mori, y., berzin, t.m.: artificial intelligence in gastrointestinal endoscopy: the future is almost here. national library of medicine (2018) google scholar  araújo, j.d.l., et al.: liver segmentation from computed tomography images using cascade deep learning. comput. biol. med. 140, 105095 (2022) article  google scholar  chiras, d.d.: human body systems: structure, function, and environment. jones & bartlett publishers (2013) google scholar  da cruz, l.b., et al.: kidney segmentation from computed tomography images using deep neural network. comput. biol. med. 123, 103906 (2020) article  google scholar  da cruz, l.b., et al.: kidney tumor segmentation from computed tomography images using deeplabv3+ 2.5d model. expert syst. appl. 192, 116270 (2022) google scholar  da cruz, l.b., et al.: interferometer eye image classification for dry eye categorization using phylogenetic diversity indexes for texture analysis. comput. methods programs biomed. 188, 105269 (2020) article  math  google scholar  diniz, j.o., dias jr, d.a., da cruz, l.b., gomes jr, d.l., cortês, o.a., de carvalho filho, a.o.: efficientensemble: diagnóstico de câncer de mama em imagens de ultrassom utilizando processamento de imagens e ensemble de efficientnets. in: simpósio brasileiro de computação aplicada à saúde (sbcas), pp. 202–213. sbc (2024) google scholar  diniz, j.o.b., et al.: heart segmentation in planning ct using 2.5d u-net++ with attention gate. comput. methods biomech. biomed. eng. imag. vis. 11(3), 317–325 (2023) google scholar  duda, r.: pattern classification and scene analysis. tronto a wiley-interscience publication, new-york, london, sydney (1973) math  google scholar  escobar, j., sanchez, k., hinojosa, c., arguello, h., castillo, s.: accurate deep learning-based gastrointestinal disease classification via transfer learning strategy. department of electrical engineering (ieee) (2021) google scholar  he, k., zhang, x., ren, s., sun, j.: deep residual learning for image recognition. arxiv (2015) google scholar  hollstensson, m.: detecting gastrointestinal abnormalities with binary classification of the kvasir-capsule dataset. bachelor degree project (2022) google scholar  huang, g., liu, z., van der maaten, l., weinberger, k.q.: densely connected convolutional networks. arxiv (2016) google scholar  ilic, m., ilic, i.: epidemiology of stomach cancer. world j. gastroenterol. 28(12), 1187 (2022) article  math  google scholar  júnior, d.a.d., et al.: automatic method for classifying covid-19 patients based on chest x-ray images, using deep features and pso-optimized xgboost. expert syst. appl. 183, 115452 (2021) article  math  google scholar  koh, p., cole, n., evans, h.m., mcfarlane, j., roberts, a.j.: diagnostic utility of upper and lower gastrointestinal endoscopy for the diagnosis of acute graft-versus-host disease in children following stem cell transplantation: a 12-year experience. pediatr. transplant. 25(7), e14046 (2021) article  google scholar  naz1, j., khan1, m.a., alhaisoni, m., song, o.y., tariq, u., kadry, s.: segmentation and classification of stomach abnormalities using deep learning. tech science press (2021) google scholar  noffsinger, a.e.: serrated polyps and colorectal cancer: new pathway to malignancy, mechanisms of disease. ann. rev. pathol. (2008) google scholar  pogorelov, k., et al.: kvasir: a multi-class image dataset for computer aided gastrointestinal disease detection (2017). https://doi.org/10.1145/3083187.3083212 rogler, g.: chronic ulcerative colitis and colorectal cancer. cancer letters (2014) google scholar  sandler, m., howard, a., zhu, m., zhmoginov, a., chen, l.c.: mobilenetv2: inverted residuals and linear bottlenecks. arxiv (2018) google scholar  sharma, p., das, d., gautam, a., balabantaray, b.k.: lpnet: a lightweight cnn with discrete wavelet pooling strategies for colon polyps classification. wiley (2022) google scholar  simonyan, k., zisserman, a.: very deep convolutional networks for large-scale image recognition. arxiv (2014) google scholar  smedsrud, p.h., et al.: kvasir-capsule, a video capsule endoscopy dataset (2021). https://doi.org/10.1038/s41597-021-00920-z tan, m., le, q.v.: efficientnet: rethinking model scaling for convolutional neural networks. arxiv (2019) google scholar  yogapriya, j., chandran, v., sumithra, m.g., anitha, p., jenopaul, p., suresh gnana dhas, c.: gastrointestinal tract disease classification from wireless endoscopy images using pretrained deep learning model. hindawi (2021) google scholar  download references acknowledgments this work was supported by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001, fundação de amparo à pesquisa do maranhão (fapema), and conselho nacional de desenvolvimento científico e tecnológico (cnpq). we also acknowledge llm use for spell-checking, grammar correction, and assistance in translating specific terms. author information authors and affiliations applied computational intelligence laboratory (lica), federal university of cariri (ufca), juazeiro do norte, ce, brazil pedro da s. viana, luana b. da cruz & domingos a. dias jr. innovation factory, federal institute of maranhão (ifma), grajaú, ma, brazil joão otávio bandeira diniz authors pedro da s. vianaview author publications search author on:pubmed google scholar luana b. da cruzview author publications search author on:pubmed google scholar domingos a. dias jr.view author publications search author on:pubmed google scholar joão otávio bandeira dinizview author publications search author on:pubmed google scholar corresponding author correspondence to pedro da s. viana . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da s. viana, p., da cruz, l.b., dias jr., d.a., diniz, j.o.b. (2025). anomalies diagnostic in endoscopic images using deep learning ensemble models. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_8 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_8 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords deep learning endoscopy imaging ensemble publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature an evaluation of temporal neighborhood coding variants in smartphone-based human activity recognition | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper an evaluation of temporal neighborhood coding variants in smartphone-based human activity recognition conference paper first online: 30 january 2025 pp 82–94 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) an evaluation of temporal neighborhood coding variants in smartphone-based human activity recognition download book pdf download book epub gustavo p. c. p. da luz9, otávio o. napoli9, j. v. delgado9, anderson r. rocha9, levy boccato9 & … edson borin9  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 377 accesses abstract self-supervised learning (ssl) has emerged as a powerful tool for learning valuable representations from large amounts of unlabeled data. human activity recognition (har) is a field that may benefit from ssl techniques, as there is a large amount of unlabeled data, and labeling is time-consuming and costly. temporal neighborhood coding (tnc) is a ssl technique that shows promise in extracting meaningful features automatically. however, the effectiveness of different tnc variations for har data has yet to be comprehensively evaluated. current research focuses on specific contexts, leaving a gap in understanding how these variations perform across the same dataset. additionally, it is necessary to assess the impact of applying tnc on raw data instead of on handcrafted features. this paper systematically evaluates different variations of tnc for har, investigating their performances in a standardized quantitative and qualitative approach, focusing on raw data but comparing with featured engineered data of the same dataset. our findings show that using the dilated convolution encoder proposed by ts2vec is currently the best alternative in terms of performance, achieving 95% accuracy in the uci dataset with raw data. additionally, we verified that by replacing the augmented dickey-fuller (adf) statistical test with the cosine similarity to select neighboring windows makes training time seven to nine times faster with little performance impact. finally, we found that learning from handcrafted features from uci dataset is easier, but advanced versions of tnc can effectively learn robust features from raw data, achieving performance comparable to models trained on the handcrafted features. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others comparing self-supervised learning techniques for wearable human activity recognition article 30 march 2025 an active semi-supervised deep learning model for human activity recognition article open access 20 march 2022 a resource conscious human action recognition framework using 26-layered deep convolutional neural network article 01 august 2020 1 introduction self-supervised learning (ssl) has the community’s attention, being a powerful tool to learn valuable representations from large amounts of unlabeled data [2, 18]. essentially, ssl approaches conceive a pretext task in which an encoder, coupled with a projection head, is encouraged to learn efficient data representations. the pretext labels are directly derived from the own input samples (originally unlabeled). after training, the encoder is detached from the projection head, and the knowledge is transferred to the downstream task coupled with a prediction head to solve the desired task. this is especially useful to learn representations of data that can be explored in multiple tasks, also saving annotation time, which is time-consuming and expensive [4]. applying ssl to human activity recognition (har) data is a trend [13], due to limited size of labeled datasets, the high data requirements of models, and the possibility to collect data on a large scale [14]. this scenario motivated this study to evaluate the technique focused on time series data known as temporal neighborhood coding technique (tnc) [10] for the har task, predicting what activity is being performed based on inertial sensor data. ssl in time series can be divided into three main categories [6, 16] : generative-based, contrastive-based, and adversarial-based methods. the contrastive-based methods learn representation through the sample comparison of positive and negative pairs. according to zhang et al.  [16], the contrastive methods can be classified into five subcategories: prediction contrast, augmentation contrast, prototype contrast, expert knowledge contrast, and sampling contrast. in prediction contrast, the pretext task is to predict the representations for future segments/windows from previous excerpts of the time series, whereas in augmentation contrast, the representations created for augmented versions of a time window are forced to be similar, but different from those related to other distant time windows. the prototype contrast makes the clustering of garnered representations pairs. the expert knowledge contrast incorporates expert prior knowledge to choose the correct positive or negative sampling during the training. the sampling contrast uses time series segment windows and generated augmentations to compare each to check the similarity. tnc [10] is a ssl technique based on constrastive learning paradigm. this approach aims to learn representations by comparing pairs of similar and dissimilar samples. unlike traditional contrastive methods that use augmentations to generate positive and negative pairs, tnc defines these pairs based on the concept of temporal neighborhoods. the original version uses the augmented dickey-fuller (adf) statistical test [7] in order to identify the stationarity of a signal, and to define positive and negative pairs. the technique was originally tested with a bidirectional single-layer recurrent neural network (rnn) encoder. after the original work of tonekaboni et al.  [10], modified versions of tnc have been considered, such as tnc-sim [11], which proposed replacing the adf statistical test by the cosine similarity metric. additionally, the work of retrieval-based reconstruction (rebar) [12] uses tnc with a different encoder, exploring dilated convolutions proposed in ts2vec [15] instead of a rnn. available research works implement some of these alternatives in specific contexts, but do not establish a detailed comparison between them in the same dataset. in this study, we assess the performance of various tnc variations. the main contributions of this work are: 1. we evaluated the different tnc variations under identical conditions, utilizing the same code base and uci dataset with raw data and showed that the ts2vec encoder significantly outperforms the rnn encoder for this task, achieving accuracies that are 15 to 17% points higher. we also show that replacing the adf statistical test by the cosine similarity has little impact on model accuracy, however, it reduces the training times from \(7 \times \) to \(9 \times \). 2. we also evaluated the impact of applying a tnc encoder to both raw data and data represented by handcrafted features (produced and provided by the authors of the uci [8] dataset). our findings indicate that learning from handcrafted features is easier. still, the more advanced versions of tnc can effectively learn robust features from the raw data, achieving performance comparable to models trained on handcrafted features. the rest of this paper is organized as follows: sect. 2 presents the related works. section 3 provides an overview of the tnc technique and its variants. section 4 describes the material and methods and discusses the experimental results. finally, sect. 5 provides the main conclusions and possible future work. 2 related works this section describes the related works, defining the datasets used at this paper, followed by a comparison between works that evaluated tnc performance under different circumstances such as datasets, encoders, and similarity functions. before discussing the tnc-related works, it is important to distinguish two variants of the uci dataset for har, a dataset that is frequently used to evaluate machine learning (ml) models on har tasks. the uci dataset, provided by the university of california, irvine (uci) [8], contains data extracted from smartphones’ accelerometers and gyroscope and is annotated with human activity labels (e.g., walking, laying, etc.). the first version consisted of a preprocessed dataset in which authors extracted 561 handcrafted features from the raw accelerometer and gyroscope data. we refer to this version of the uci dataset as uci-fe (feature engineered). later, the authors disclosed an updated version that contains the raw signal from the smartphone’s accelerometers and gyroscopes. we refer to this version as uci-raw. the temporal neighborhood coding (tnc) technique was first introduced by tonekaboni et al.  [10], in 2021. their work employed a bidirectional single layer rnn as the encoder and the adf test to determine the interval around a time window where the signal can be approximately considered as stationary. they evaluated their technique on the uci-fe dataset and reported an accuracy of 88.3%. in 2023, wang et al.  [11] explored a variation of the tnc technique using the feature similarity of the time series measured by the cosine similarity between time windows to establish the range of a temporal neighborhood instead of adf, selecting the most similar windows as neighbors instead of using the statistical test. they used the same encoder as tonekaboni et al.  [10], i.e., an rnn, and compared the proposed method with adf on the sleep stages classification task. their results suggest that the tnc-sim performs 2.81% better than tnc-adf in time domain with three classes and 2.51% with five classes. in 2023, xu et al.  [12] evaluated a variation at the tnc technique employing a dilated convolution encoder on the uci-raw dataset and verified large performance gains when compared to the default rnn, reaching an accuracy of 94.3% in uci-raw. this is significant as the dataset in this case does not contain handcrafted features and the encoder is learning to extract knowledge better than after feature engineering. the chosen encoder structure was the one employed by yue et al.  [15], that introduced the ts2vec technique. yue et al.  [15] report significant performance improvement for tnc using their dilated convolution encoder, stating that this happens due to its architecture being more adaptable to different scales of datasets due to the receptive fields of dilated convolution. other works also compared the performance of rnns against dilated convolutions. franceschi et al.  [3] use deep neural networks with exponentially dilated causal convolutions that capture long-range dependencies better than full convolutions with triplet loss employing time-based negative sampling at variable length multivariate time series, presenting the encoder as more efficient and scalable than using an rnn. bai et al.  [1] also supports this assumption, presenting results in which dilated convolutions outperformed rnn in terms of efficiency and predictive performance. the aforementioned works evaluated tnc with different encoders (rnn and ts2vec) and test functions (adf and cosine similarity). however, these works evaluate tnc under different, specific circumstances (ml task, dataset, etc.). moreover, albeit some comparisons between ts2vec and rnn are provided, such works do not carry out experiments focused on analyzing the impact of these alternatives in a clear and standard manner. hence, in this work, we evaluate all these combinations with the same code base and on the same setup, i.e., human activity recognition using the uci-raw dataset. table 1 summarizes the tnc-based works presented in this section according to the dataset, encoder and test functions evaluated in each work. table 1. tnc related worksfull size table 3 tnc technique and its variants the tnc technique is based on the principle that neighboring windows in a time series are likely to belong to the same class and, therefore, should have similar representations. conversely, distant (non-neighboring) windows are more likely to belong to different classes and should have distinct representations. to achieve this, the tnc technique trains an encoder (enc) that encodes the time series windows and a discriminator (d) that determines whether a pair of windows are neighbors (i.e., close to each other). figure 1 illustrates the tnc technique. first, given a query window (\(w_q\)) that is randomly selected, and a window selector, \(w_s\) selects two extra windows: a close one (\(w_c\)), and a distant one (\(w_d\)). then, \(w_q\), \(w_c\), and \(w_d\) are encoded using enc into \(z_q\), \(z_c\), and \(z_d\), respectively. finally, the discriminator d is used to discriminate whether the pairs (\(z_q\), \(z_c\)) and (\(z_q\), \(z_d\)) are neighbors (close) or not. fig. 1. technique of tnc. the window selector \(w_s\) selects the neighbor \(w_c\) and non-neighbor window \(w_d\) for each query window \(w_q\). the encoder enc learns the data representation and feed the samples \(z_c\) and \(z_d\) into a discriminator d that predicts the probability of the windows being neighbors to \(w_q\). full size image the window selector method \(w_s\) tests whether the selected windows (i.e., \(w_c\) and \(w_d\)) are indeed similar to or different from \(w_q\). the test for \(w_c\) can be either the adf statistical test that determines the gaussian distribution parameters, enabling a selection of windows that follow that distribution, or to use the window with the highest cosine similarity from \(w_q\). for \(w_d\), windows are sampled randomly, ensuring a sufficient distance in the time series from \(w_q\) to be considered non-neighbors. the work of wang et al.  [11] presented the cosine similarity as a possibility for constructing non-neighborhoods, but in our case, it was just considered when forming \(w_c\). the encoder (enc) can be either a bidirectional rnn with one layer and 100 hidden units, as presented by tonekaboni et al.  [10], or the ts2vec dilated convolution encoder with ten residual blocks, each block containing two one-dimensional convolutional layers with a dilation parameter, as in the works of xu et al.  [12] and yue et al.  [15]. the discriminator d is a binary classifier that receives the encoded versions of a pair of windows (e.g., \(z_q\) and \(z_c\)) and predicts if they are close or not. it is composed by a fully connected layer followed by a relu activation, a dropout layer, and another fully connected layer, where the ts2vec discriminator includes an additional max pooling operation. losses are calculated for close (pair \(w_q\) and \(w_c\)) and distant windows (pair \(w_q\) and \(w_d\)), including a weighting term w, inspired by positive-unlabeled (pu) learning, where each neighbor sample is treated as a positive example, and each non-neighbor sample is a combination of a positive example with a weight and a negative example as a complementary weight. in short, this parameter represents the probability of an unlabeled sample being a positive sample (neighbor) and can be determined by prior knowledge or learned by a hyperparameter search. the combined loss is minimized to train both the discriminator and the encoder, ensuring the model differentiates between temporally close and distant samples. after training the encoder in a self-supervised manner, the weights of the encoder are stored and can be used for the downstream task. in this work, we evaluate different tnc variants, i.e., using the rnn and the ts2vec encoders and using the adf and the cosine similarity test functions at the window selector. to distinguish between the fours variants, we will employ the following acronyms: tnc-rnn-adf, tnc-rnn-sim, tnc-ts2vec-adf, and tnc-ts2vec-sim. this is the first work that evaluates all this variations of tnc using the same dataset and encoder, allowing a fair comparison between the different configurations. we also experimented with two values for w, which provides an exploration of different configurations of tnc in regards of the encoder, window selector method and weighting term. 4 experimental results and discussion this section presents our methodology and the experimental results. initially, sect. 4.1 describes the materials and methods used to evaluate the tnc technique. then, sect. 4.2 presents quantitative analysis, discussing the performance of each variant on the uci-raw dataset. finally, sect. 4.3 provides a qualitative analysis of the learned representations. 4.1 materials and methods the analysis is divided into two parts. first, we replicate two of the previous works: the work of tonekaboni et al.  [10] following the code made available at githubfootnote 1, in which we evaluate the tnc-rnn-adf variant using the uci-fe dataset; and the work of xu et al.  [12], in which we evaluate the tnc-ts2vec-sim variant using the uci-raw dataset, following the exact architecture proposed by the authors. this allows us to validate our code base, ensuring we execute the variants with models and parameters compatible with the related works. we also want to assess the performance impact the two variants of the uci dataset (i.e., uci-fe and uci-raw) have on tnc-rnn-adf, which has not been reported on the literature yet. after the replication of the original results, we proceeded to the performance analysis of the four tnc variants in the case of the uci-raw dataset. we decided to focus on uci-raw because most of the publicly available har datasets are made available in this format and the majority of previous work on representation learning focus on learning representations from raw data. the code used in this analysis was based on the code made available by xu et al.  [12] at githubfootnote 2. notice that this is the same dataset of uci, but xu et al.  [12] at uci-raw and tonekaboni et al.  [10] at uci-fe uses different pre-processing steps and dataset partitions, which might contain modifications that introduce bias to the results. the original uci dataset contains six classes: walking, walking upstairs, walking downstairs, sitting, standing, and laying. later, it was extended to include postural transitions (hapt [9]). in this work we employ only the original six classes. the uci-raw dataset consists of records of 6 channels, being three axes of accelerometer and three of gyroscope, sampled at 50 hz concatenated by the user and segmented in windows of 2.56 s with no overlap, corresponding to 128-time points, following the data processing steps of [12]. the selection of the window size \(\delta \) depends on prior knowledge of the signals, selected in a manner that contains just enough information of the current state of the signal. the 4,600 windows of the dataset were split, leaving 70% for training, 15% for validation, and 15% for testing, using the same partitions in all experiments. we ensured that all windows from the same user always fall into the same subset. the weight parameter w was chosen based on previous work, being set as 0.05 ( [10]) and 0.2 ( [12]). the encoding size was kept constant at 320, a batch size of 16, a learning rate of 0.00001, and 100 training epochs. to evaluate the discriminative performance of each method at the downstream task, we used the linear readout protocol, applying logistic regression on top of the frozen representations learned by the encoder. this is made to assess the capability of the features learned by the encoder to distinguish between the classes using the metrics accuracy, area under the precision-recall curve (auprc), balanced accuracy, and \(f_1\)-score. training time was measured in a machine with a 16gb intel(r) core(tm) i5-8500 cpu @ 3.00ghz and an 8gb nvidia gtx 1080 gpu. we also compared the variations of tnc in a qualitative manner by visualizing the generated representations in the latent space on a 2d chart with the aid of t-sne [5]. the goal was to verify whether the dataset samples are clustered according to their classes. 4.2 quantitative analysis table 2 presents the results reported by tonekaboni et al.  [10] and xu et al.  [12], along with the results produced with our code (named as “repro”) using the same parameters they used. notice that the accuracy and auprc results are very similar to the ones reported by tonekaboni et al. and xu et al. table 2. results in percentage for replication of original worksfull size table table 3 shows the results for the four tnc variants with parameter w equal to 0.05 and 0.20. the first four rows contain the variants with the rnn encoder, while the last ones contain the ones with the ts2vec encoder. the first observation is that the ts2vec encoder significantly outperforms the rnn encoder for this task, achieving accuracies that are 15 to 17% points higher. table 3. results in percentage with mean and standard deviation for different implementations of tnc on uci dataset raw.full size table the tnc-ts2vec-sim variant worked best with w = 0.20 while the other alternatives worked best with w = 0.05. however, the impact on performance is very small (\(\le \)1% on accuracy). as for using a statistical test or cosine similarity, the second one showed little impact on the metrics, which indicates that the method to select the neighborhood did not show major differences in the results for har data. however, one positive impact of using cosine similarity is the training time (last column), which showed to be from \(7 \times \) to \(9 \times \) faster for both encoders and weight parameters, being even faster when using rnn as the encoder. training time is relevant, especially for works that need a fast implementation to evaluate different ssl strategies, as tnc was presented as a very time-consuming and less competitive method in works such as [12, 17], which considered tnc with rnn as the encoder as being 250 times slower than the ts2vec framework for time series representation learning. by analysing the second row of table 2 (tnc-rnn-adf/\(w=0.05\) on uci-fe) and the first row of table 3 (tnc-rnn-adf/\(w=0.05\) on uci-raw) it is possible to see that it was easier for the tnc-rnn-adf to learn from the uci-fe (accuracy=88.0%) dataset than the uci-raw (accuracy=78.7%). this might be attributed to the handcrafted features. nonetheless, tnc-ts2vec-sim and tnc-ts2vec-adf were capable of learning good features from the uci-raw dataset, achieving 95% of accuracy. this result suggests that the tnc technique combined with the ts2vec encoder is capable of automatically learning very discriminative features, reaching the best performance for uci-raw in our experiments. to validate our findings, we performed a statistical analysis after running each one of the rows of table  3 experiment eight times, forming a sample of 64 runs, 32 for each encoder (rnn and ts2vec). we first assessed the normality of the data for each encoder, using shapiro-wilk test, that presented a p-value below 0.05 for the rnn, indicating a significant deviation from a normal distribution. given that, we proceeded to the non-parametric paired wilcoxon signed-rank test. the test resulted in a p-value below 0.05, which indicates that the two encoder means are statistically significant different. 4.3 qualitative analysis this section provides a qualitative analysis of the learned representations. first, we reproduce the t-sne plots presented by tonekaboni et al.  [10]. figure 2 shows the t-sne plot for the test set of the uci-fe dataset (left) and for its samples encoded by the tnc-rnn-adf variant (right). it is possible to notice that the handcrafted features already splits the data in three major clusters: (laying), (walking downstairs; walking upstairs), and (sitting; standing; walking). however, there is still some confusion among classes inside these clusters. the tcn-rnn-adf encoder, on the other hand, was capable of improving the representation by separating the classes on six, more separable, clusters, in line with the results reported by tonekaboni et al.  [10] at appendix a.7.2. fig. 2. t-sne of tnc-rnn-adf encoder on data with features full size image now, we turn our attention to the representations learned by the four variants of tnc on the uci-raw dataset. we start by analyzing the t-sne chart for the raw test set, showed in fig. 3. notice that, despite the existence of two clusters, there is no clear separation between most classes. this explains why it is easier to learn from the uci-fe dataset than from the uci-raw one. fig. 3. t-sne chart for the uci-raw dataset full size image figure 4 shows the t-sne charts for the representations learned by the four variants of tnc. since the w parameter had little impact on the result, we decided to report only the best result for each one of the variants in terms of accuracy. notice that the variants with the ts2vec encoder (bottom) perform better than the ones with the rnn encoder (top), managing to provide a clear separation between most classes. also, the choice of using adf or cosine similarity to select the neighborhood had little impact on the clustering. fig. 4. comparison between t-sne on different tnc variations full size image 5 conclusions and future work in this work, we evaluated different variants of tnc using the feature engineered (uci-fe) and the raw (uci-raw) versions of the uci har dataset. uci-fe was the version used by the original work of tnc [10], while uci-raw is more often used to evaluate machine learning techniques. regarding the tnc, we evaluate the impact of two different encoders (rnn and ts2vec) and two different neighborhood selection functions (adf and cos. sim.) have on the performance of the model when trained with the uci-raw dataset. our evaluation of different tnc variants using the uci-raw dataset demonstrated that the ts2vec encoder significantly outperforms the rnn encoder for this task, achieving accuracies that are 15 to 17% points higher. additionally, we found that replacing the adf statistical test with cosine similarity has a minimal impact on model accuracy but reduces training times from \(7 \times \) to \(9 \times \). furthermore, our assessment of the impact of applying a tnc encoder to both raw data (uci-raw) and data represented by handcrafted features (uci-fe) indicates that, while learning from handcrafted features is easier, the more advanced versions of tnc can effectively learn robust features from raw data, achieving performance comparable to models trained on handcrafted features, i.e., 95% of accuracy. as future work, an ablation study of different hyperparameters can be added to the discussion, specially the weight parameter and window size and the performance of the fully supervised training using the encoders with a linear classification layer can be added. also, an analysis of the model’s performance on different training sets at the downstream task can also be included. other har datasets or encoders can be used in the same analysis to assess their impact at the tnc variations. also, other methods can be tested with different encoders in the same data to enrich the comparison. other tasks can also be used in the intention of to evaluate tnc not in a specific downstream task such as har. reproducibility statement. sect. 4 defines materials and methods used in our evaluations. the code used is released publicly at githubfootnote 3. the datasets used are publicly available, and a script containing data processing is also available. notes 1.https://github.com/sanatonek/tnc_representation_learning. 2.https://github.com/maxxu05/rebar. 3.https://github.com/h-iaac/kr-papers-artifacts. references bai, s., kolter, j.z., koltun, v.: an empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arxiv preprint arxiv:1803.01271 (2018) ericsson, l., gouk, h., loy, c.c., hospedales, t.m.: self-supervised representation learning: introduction, advances, and challenges. ieee sig. process. mag. 39(3), 42–62 (2022) article  math  google scholar  franceschi, j.y., dieuleveut, a., jaggi, m.: unsupervised scalable representation learning for multivariate time series. in: advances in neural information processing systems (2019) google scholar  fredriksson, t., bosch, j., olsson, h.h.: machine learning models for automatic labeling: a systematic literature review. in: icsoft, pp. 552–561 (2020) google scholar  van der maaten, l., hinton, g.: visualizing data using t-sne. j. mach. learn. res. 9(11) (2008) google scholar  meng, q., qian, h., liu, y., xu, y., shen, z., cui, l.: unsupervised representation learning for time series: a review. arxiv preprint arxiv:2308.01578 (2023) mushtaq, r.: augmented dickey fuller test (2011) google scholar  reyes-ortiz, j.l., anguita, d., ghio, a., oneto, l., parra, x.: human activity recognition using smartphones in: ismsit (2012) google scholar  reyes-ortiz, j.l., oneto, l., ghio, a., samá, a., anguita, d., parra, x.: human activity recognition on smartphones with awareness of basic activities and postural transitions. in: artificial neural networks and machine learning–icann 2014: 24th international conference on artificial neural networks, hamburg, germany, september 15-19, 2014. proceedings 24, pp. 177–184. springer (2014) google scholar  tonekaboni, s., eytan, d., goldenberg, a.: unsupervised representation learning for time series with temporal neighborhood coding. arxiv preprint arxiv:2106.00750 (2021) wang, y., liang, h., zhai, b.: temporal neighborhood based self-supervised pre-training model for sleep stages classification. in: proceedings of the 2023 15th international conference on bioinformatics and biomedical technology, pp. 149–155 (2023) google scholar  xu, m., moreno, a., wei, h., marlin, b., rehg, j.m.: retrieval-based reconstruction for time-series contrastive learning. in: the twelfth international conference on learning representations (2023) google scholar  yin, y., xie, l., jiang, z., xiao, f., cao, j., lu, s.: a systematic review of human activity recognition based on mobile devices: overview, progress and trends. ieee commun. surv. tut. 26(2) (2024) google scholar  yuan, h., et al.: self-supervised learning for human activity recognition using 700,000 person-days of wearable data. npj dig. med. 7(1), 91 (2024) article  math  google scholar  yue, z., et al.: ts2vec: towards universal representation of time series. in: proceedings of the aaai conference on artificial intelligence, vol. 36, pp. 8980–8987 (2022) google scholar  zhang, k., et al.: self-supervised learning for time series analysis: taxonomy, progress, and prospects. ieee trans. pattern anal. mach. intell. 46(10), 6775–6794 (2024) article  math  google scholar  zhang, x., zhao, z., tsiligkaridis, t., zitnik, m.: self-supervised contrastive pre-training for time series via time-frequency consistency. adv. neural. inf. process. syst. 35, 3988–4003 (2022) math  google scholar  zhao, z., alzubaidi, l., zhang, j., duan, y., gu, y.: a comparison review of transfer learning and self-supervised learning: definitions, applications, advantages and limitations. expert syst. appl., p. 122807 (2023) google scholar  download references acknowledgments this project was supported by the ministry of science, technology, and innovation of brazil, with resources granted by the federal law 8.248 of october 23, 1991, under the ppi-softex [01245.003479/2024-10]. the authors also thank cnpq (315399/2023-6, 404087/2021-3 and 88887.999360/2024-00) and fapesp (2013/08293-7) for their financial support. author information authors and affiliations hub for artificial intelligence and cognitive architectures (h.iaac), university of campinas, campinas, 13083-852, brazil gustavo p. c. p. da luz, otávio o. napoli, j. v. delgado, anderson r. rocha, levy boccato & edson borin authors gustavo p. c. p. da luzview author publications search author on:pubmed google scholar otávio o. napoliview author publications search author on:pubmed google scholar j. v. delgadoview author publications search author on:pubmed google scholar anderson r. rochaview author publications search author on:pubmed google scholar levy boccatoview author publications search author on:pubmed google scholar edson borinview author publications search author on:pubmed google scholar corresponding author correspondence to gustavo p. c. p. da luz . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper da luz, g.p.c.p., napoli, o.o., delgado, j.v., rocha, a.r., boccato, l., borin, e. (2025). an evaluation of temporal neighborhood coding variants in smartphone-based human activity recognition. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_6 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_6 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords human activity recognition representation learning self supervised learning temporal neighborhood coding publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature combining clustering and genetic algorithms for portfolio optimization: a case study with b3 companies | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper combining clustering and genetic algorithms for portfolio optimization: a case study with b3 companies conference paper first online: 30 january 2025 pp 141–156 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) combining clustering and genetic algorithms for portfolio optimization: a case study with b3 companies download book pdf download book epub edsson israel andonaegui alonso  orcid: orcid.org/0009-0001-0689-80219, karina valdivia delgado  orcid: orcid.org/0000-0002-9120-89879 & francisco carlos b. dos santos  orcid: orcid.org/0000-0003-3003-12229  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 380 accesses abstract portfolio optimization has been addressed by optimization algorithms and due to the numerous local maxima, genetic algorithms have shown promise. recent studies use these algorithms to optimize stock portfolios, whether as a multi-objective problem that maximize return and minimize risk or just as one objective problem, such as maximizing the sharpe ratio. in order to adapt these algorithms to the financial market, a new approach was developed that includes penalty and cardinality constraints. this study proposes an algorithm with two models of clustering of b3 companies with penalties and cardinality constraints. one model uses the clustering of companies by economic sector and the other model uses the clustering generated by the kmeans-dtw time series clustering technique. then the algorithm selects one asset per cluster of companies with the aim of diversifying the portfolio and reducing the correlation between assets, thus reducing the risk. finally, the generated portfolios went through an optimization stage using a memetic genetic algorithm, which finds the best possible distribution that maximizes return and reduces risk. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others a genetic algorithm approach for portfolio optimization chapter © 2024 solution to the unconstrained portfolio optimisation problem using a genetic algorithm chapter © 2023 different length genetic algorithm-based clustering of indian stocks for portfolio optimization chapter © 2019 1 introduction the problem of optimizing a stock portfolio consists in finding the best arrangement of weights that maximize the expected return and minimize risk. one of the most used approaches is genetic algorithms due to their proven effectiveness in solving complex search problems [4]. this optimization problem can be modeled as a multi-objective problem (maximizing return and minimizing risk) with constraints to make it more realistic, such as cardinality constraints (limiting the maximum amount of assets that can make up the portfolio), penalty systems and transaction costs [3, 7]. the addition of these mechanisms makes optimization models more complex and challenging computationally [1]. in [11], evolutionary algorithms were used with cardinality, border and entire batch constraints to select portfolios, however the memetic version did not bring significant improvements. the sharpe ratio is a popular measure to evaluate portfolios, maximizing the risk-return ratio. this measure is an alternative that can be used instead of modeling the problem as multi-objective problem (maximizing return and minimizing risk). in the financial literature, genetic algorithms have been proposed to maximize the sharpe ratio [1, 8], however, in general, it has not received much attention in the past, perhaps due to the extremely heavy load on the calculations [5]. in [2], a markowitz medium-variance portfolio selection model was proposed that groups assets by economic sectors, but without cardinality constraints (i.e., there is no limit on the number of assets within the portfolio) associated with the optimization made by genetic algorithms. however, there is a lack of works in the literature that bring all these factors together. therefore, the objective of this work is to propose an algorithm that (i) selects assets in a completely independent way (without human supervision), containing asset allocation mechanisms that take into account the economic sector to reduce volatility, and (ii) maximizes the portfolio’s shape ratio via genetic memetic algorithm, using cardinality restrictions and containing a penalty mechanism for weight distributions to make it more realistic. as a case study, the stock portfolio of b3 companies is used, aiming to outperform the bovespa index in the long term. the proposed algorithm, first groups the b3 companies to form clusters, from which assets are chosen for the final portfolio, assigning weights via genetic memetic algorithm. among the possible ways used by the proposed algorithm to group the b3 companies are (i) to use clusters that b3 itself makes available online, in which each cluster represents a subsector of the economy; and (ii) to use the k means dtw [9] technique of clustering, which is used to calculate a measure of distance between two time series that are not aligned. 2 theoretical background the asset allocation algorithm proposed in this study uses three main theories: the sharpe ratio [10], the markowitz asset diversification theory [6] and the capital asset pricing model (capm). these theories are detailed in this section. finally, in sect. 2.6, the k-means algorithm with the dtw distance is described, which will be used in this work to create groups of assets. 2.1 sharpe ratio the sharpe ratio [10] is widely used to evaluate assets or portfolios, measuring the risk-return ratio in comparison to the risk-free return: $$\begin{aligned} sharpe ratio=(r_p r_f)/\sigma _p \end{aligned}$$ (1) where \(r_p\) is the expected return of the asset or portfolio, \(r_f\) is the risk-free return, (for example, the selic rate in brazil) and \(\sigma _p\) is the volatility of the asset or portfolio. 2.2 calculation of expected return and risk (volatility) of an asset the expected daily return \(r_p\) of an asset is the average of the daily returns observed over a period: $$\begin{aligned} r_p= \frac{\sum _{i=1}^{n} r_i}{n} \end{aligned}$$ (2) where \(r_i\) is the daily return on the day i and n are the numbers of days. the risk or volatility \(\sigma _p\) is the standard deviation of the historical returns: $$\begin{aligned} \sigma _p = \sqrt{\frac{\sum _{i=1}^{n} (r_i r_p)^2}{n}} \end{aligned}$$ (3) in this work, the daily logarithmic return is used, defined as the natural logarithm of the ratio between the current price (\(a_i\)) and the previous day’s price (\(a_{i-1}\)): $$\begin{aligned} {r_{i}}= \ln {\frac{a_i}{a_{i-1}}} \end{aligned}$$ (4) these calculations are used to determine the expected return and the risk/volatility of each candidate asset to make up the final portfolio. 2.3 calculation of expected return and portfolio risk (volatility) the calculation of the expected return of a portfolio, according to markowitz, is the weighted sum of the expected returns of the assets that compose it. for a portfolio with n assets, with expected returns \(r_1\), \(r_2\), ..., \(r_n\) and weights \(x_1\), \(x_2\), ..., \(x_n\) respectively, the expected return \(r_p\) is given by: $$\begin{aligned} r_p=\sum _{i=1}^{n} x_ir_i \end{aligned}$$ (5) markowitz defined a portfolio’s risk as its variance, which includes the individual asset variances weighted by the squares of their weights, plus twice the sum of the covariances between all pairs of assets, weighted by the product of their weights. for a portfolio with two assets a and b, with weights \(w_a\) and \(w_b\), the risk is given by: $$\begin{aligned} \sigma ^2_p = w^2_a\sigma ^2_a + w^2_b\sigma ^2_b + 2w_aw_b\sigma _{a,b} \end{aligned}$$ (6) generalizing for a portfolio with n assets: $$\begin{aligned} \sigma ^2_p = \sum _{i=1}^{n}\sum _{j=1}^{n} w_iw_j\sigma _{i,j} \end{aligned}$$ (7) where \(\sigma _{i,j}\) is the covariance between the returns on assets \(a_i\) and \(a_j\). 2.4 asset diversification markowitz’s diversification theory states that adding new assets to a portfolio reduces the total risk due to covariance between assets [6]. as more assets are added, the risk decreases until it reaches an asymptote, where adding new assets no longer reduces risk. this reduction occurs mainly due to the negative covariance between assets. markowitz’s risk formula demonstrates that the risk of a portfolio can be reduced with assets that have low covariance with each other. however, there is a limit to this reduction, called unsystematic risk, specific to the characteristics of companies or sectors. systemic risk, such as political crises and pandemics, cannot be reduced by diversification. this work proposes the use of two clustering ways and evaluates which one best separates companies into groups with lower covariance, aiming to reduce unsystematic risk. 2.5 capital asset pricing model (capm) theory the capital asset pricing model (capm), proposed by sharpe (1964), lintner (1965) and mossin (1966), calculates the expected return of an investment based on its risk compared to the market (ibovespa). the capm formula is: $$\begin{aligned} e(r_i) = r_f + \beta _i(r_m r_f) \end{aligned}$$ (8) where \(e(r_i)\) is the expected return on the investment, \(r_f\) is the risk-free return (selic rate), \(\beta _i\) is the correlation of the asset’s risk with the market risk (bovespa index) and \(r_m\) is the expected market return (bovespa index). 2.6 k-means algorithm for time series and dtw distance between time series k-means algorithm for time series groups them based on their format. the process begins with randomly selecting k time series as centroids, then calculating the distance between each time series and the centroids, and continually adjusting the centroids until they represent the center of mass of the clusters. the algorithm is run several times to improve representativeness. to calculate the distance between time series, they must all have the same size and be normalized. a similarity metric that can be used is dynamic time warping (dtw), which is more suitable than euclidean distance for time series with time-shifted correlations [9]. dtw is used to measure the similarity between two time sequences that do not exactly align in time, allowing one time series to be stretched or compressed to better match another time series, offering an advantage over traditional methods such as euclidean distance. 3 modeling the portfolio weight assignment problem using memetic genetic algorithms genetic algorithms (gas) constitute a search and optimization technique, inspired by the darwinian principle of natural selection and genetic reproduction, in which the fittest individuals have a greater chance of reproducing. in the proposed modeling, a chromosome represents a portfolio of assets, and the weights of each asset are the genes. chromosomes are then subjected to an evolutionary process that involves the steps of evaluation using a fitness function, selection, sexual recombination (crossing-over) and mutation. after several cycles of evolution, the population should contain fitter individuals. below, each of these mechanisms that make up the genetic algorithms for the case study is described. 3.1 fitness function the fitness function is the function that will evaluate how well-adapted each individual in the population is in relation to the objective they want to achieve. in this study, the objective is to maximize the portfolio’s sharpe ratio, therefore, the input to the fitness function is a portfolio (an individual) from the population and the output is its respective sharpe ratio, which can be calculated by eq. 1. the proposed ga has a penalty system, this mechanism is implemented within the fitness function and is described below. penalty mechanism. a penalty system is used to leave the final output of the ga (portfolio closest to the global maximum) with a more realistic weight distribution. unrealistic weight distributions will be penalized in such a way that their sharpe ratio value will be decreased on purpose, thus this individual will have less chance of continuing to the next generation. portfolios with excessively high or low weights in certain assets are considered unrealistic solutions. the allocation of 20% or more in a single asset is rarely seen in the real world, as it is very risky. the same solutions often allocated excessively low weights to some assets in such a way that their variation (high or low) had no impact on the total variation of the portfolio. for this reason, solutions that allocated very low weights (less than 0.5%) to assets were also penalized. 3.2 selection algorithm the selection algorithm is divided into two parts: elitist selection and window selection. elitist selection ensures that the fittest individuals are retained for the next generation, while window selection chooses portfolios randomly within a window that decreases throughout interactions. in this way, the fittest individuals will have a greater chance of being chosen because they are within the selection window more often, while the less fit individuals will have less chance of being chosen because they will be within the window less often. 3.3 crossing-over and mutation algorithm position based crossover (pbx) algorithm was used as crossing-over algorithm. firstly, pbx randomly selects 40% of the parents’ genes. child 1 receives the randomly selected genes from parent 2, that is, child 1 already has 40% of the genes completed, the other 60% comes from parent 1 completing 100%. similarly, child 2 receives randomly selected genes from parent 1 and the other 60% comes from parent 2. figure 1 shows an example. after the pbx function completes and generates 2 new individuals (children), the weights of the new portfolio must be adjusted so that the sum of all weights is always equal to 1. fig. 1. position based crossover (pbx) full size image 3.4 mutation algorithm aiming to generate genetic variability and consequently increase the chance of leaving local maxima, three mutation functions are used: order based mutation: the element in position i is exchanged with the element in position j. position based mutation: the element in position i is removed and inserted into position j. scramble mutation: a randomly selected sublist is shuffled. 3.5 local search for nearest neighbors (memetic step) memetic step of the algorithm involves a local search for the nearest neighbors, generating new individuals similar to those selected, and increasing genetic variability. this search involves small adjustments to the weights of the assets and normalization so that the sum of the weights is equal to 1. a nearest neighbor of a solution is a solution that contains 1% more in 2 weights. 4 proposed asset allocation algorithm this section will detail how the asset allocation algorithm works. the proposed algorithm will select between 6 and 25 companies among 462 companies, which are all companies listed on b3 on 10/1/2023. the range between 6 and 25 assets was defined by the following reasons: this is the range seen in most brazilian stock portfolios managed by stock funds. the smaller the number of assets within the portfolio, the greater the risk will be, and the greater the number of assets that make up the portfolio, the tendency is for the risk to decrease. limiting the maximum value to 25 is due to 2 factors: firstly, managing a portfolio with more than 25 assets is quite costly, and secondly, a value above that can dilute the profitability obtained, because as the number of assets increases, the tendency is for the weight allocated to each asset to decrease together. limiting the minimum value to 6 assets is due to the fact that, if a portfolio has only 5 assets or less, it is guaranteed that at least one of the assets will have 20% of the entire weight; this greatly increases the portfolio risk (the penalty mechanism proposed in sect. 3.1 will attempt to handle this scenario in the cases where the number of assets is greater than 6). in short, the proposed algorithm first obtains the clusters of companies and then selects an asset from each cluster of companies by analyzing the sharpe ratio to obtain the final portfolio. the memetic genetic algorithm is applied to this portfolio several times to obtain the weights in the end. these steps are described below. 4.1 get company clusters to reduce unsystematic risk (risk related to the company or sector), the asset allocation algorithm first obtains clusters of companies and selects only one asset from each existing cluster. selecting companies that belong to different clusters tends to select assets in which the correlation between them is low. selecting assets that have low correlation tends to reduce the risk of the entire portfolio as the number of assets within the portfolio increases and the correlations between them are weak. to this end, the method for calculating the risk of a stock portfolio proposed by markowitz (1952) [6] is followed, in which the risk of a set of assets (a portfolio) is a function that takes into account the covariances between assets, calculated two by two. therefore, this work proposes to design two asset allocation algorithms in which the only difference between them will be the technique used to generate these clusters. the first algorithm which will be referred to as the model b3, uses the clusters that b3 itself makes available online, in which each cluster represents a subsector of the economy. the b3 cluster is a grouping of all companies listed on the brazilian stock exchange, which divides all 463 companies present (as of november 1, 2023) into 41 different economic sectors such as agriculture, electricity, mining, telecommunications, and others. the second algorithm uses the k-means dtw clustering technique, which is used to calculate a distance measure between two times series that are not aligned. this algorithm will be referred to hereafter as the k-means dtw model. after the cluster creation step, both algorithms will follow the same subsequent steps, which are the selection of an asset from each existing cluster (described in sect. 4.2), the creation of the final portfolio (sect. 4.3) and obtaining weights from the final portfolio using genetic algorithm (sect. 4.4). 4.2 selection of an asset from each company cluster when selecting an asset from each cluster, the following criteria are used: 1. the asset must have the highest individual sharpe ratio within the respective cluster and this sharpe ratio must be greater than zero. 2. the individual asset sharpe ratio must surpass the ibovespa sharpe, which is the main benchmark. 3. the asset must have a return greater than the expected return calculated by capm formula. as previously defined, if the minimum acceptable amount returned according to the capm formula is lower than the selic rate for the training period, the amount returned by the formula will be the selic annual return rate itself. in other words, we are defining that the algorithm will not accept assets that have an expected return lower than the safe return for the same period. 4. the asset must exist in full throughout the entire training period. the asset can only be chosen if it simultaneously meets the four requirements above. if no asset within the cluster meets all four at the same time, no asset will be chosen within the respective cluster. this implies the possibility that the number of assets selected to compose the stock portfolio will not necessarily be the same as the number of clusters created by clustering techniques. 4.3 creation of the final portfolio after the stage of selecting an asset from each cluster of companies, a list of candidate companies to make up the final portfolio is generated. this list of assets is organized in descending order by the individual sharpe ratio, and is inserted in order, one by one, in the final portfolio. in other words, the assets with the highest individual sharpe ratios will be added to the final portfolio first. insertion will only be allowed if it meets the following criteria: 1. the insertion of the candidate asset must reduce the total risk/volatility of the portfolio, that is, it must respect the asset diversification proposed by markowitz [6]. if the asset does not meet the first condition, in other words, if the inclusion of the candidate asset increases the total risk of the portfolio, it will only be accepted if the sharpe ratio of the portfolio as a whole increases. in this scenario, what generally happens is that the portfolio’s risk increases, but the portfolio’s expected return increases even more, that is, the percentage increase in expected return is greater than the percentage increase in risk. 2. the insertion of the asset is only allowed if the portfolio’s expected return is greater than the minimum acceptable return calculated by the capm formula. in other words, if, when inserting the asset, it reduces the expected return of the portfolio in such a way that it is below or at the minimum acceptable (according to capm theory) it will not be selected. it is worth highlighting that the calculation of the portfolio’s return and risk will be made considering that the assets have homogeneous weights within the portfolio. if the candidate asset does not fit into any of the two situations above, it will not be chosen to make up the final portfolio. after this step, we will have a portfolio with several assets and it will go through a simple pruning/cutting mechanism, to maintain the defined number of 6–25 assets. if the number of assets is less than 6, the algorithm discards the portfolio and does not return any portfolio recommendations, if the number of assets is less than or equal to 25, the algorithm returns the portfolio recommended to the user, and if the number of assets exceeds 25, the assets with the lowest individual sharpe are removed until the maximum quantity of 25 is reached. 4.4 obtaining final portfolio weights using genetic algorithm on the portfolio assembled in the previous step, the memetic genetic algorithm is applied to obtain the best possible configuration of weights within the portfolio which returns the highest sharpe ratio. after the stage of maximizing the sharpe ratio via genetic algorithm, we have the final portfolio, the assets and their respective weights. finally, the performance of the final portfolio is calculated, that is, the return obtained one year after the training period is calculated. 4.5 asset allocation algorithm pseudocode the assetallocation algorithm (algorithm 1) shows the complete pseudocode of the proposed asset allocation algorithm. this algorithm receives a set of companies, the clustering that will be used (which can be k-means or b3), the training year and the number of weight vectors that will be used to calculate the final weight vector. after obtaining the clusters (lines 2–6), the selectcompaniesfromclusters method is applied, performing the procedure of selecting an asset from each cluster of companies (described in sect. 4.2), and creating the final portfolio (described in sect. 4.3), then returning the final portfolio (line 7). as this step involves analyzing the sharpe ratio, it is necessary to define a training time interval. on the final portfolio, the memetic genetic algorithm is applied several times (specifically quantity_vec_weights times), and in each call to the genetic algorithm, a possible configuration (the one that returns the highest sharpe ratio) of weights is obtained (lines 8 to 10). in line 11, these various weights settings found are averaged to obtain the final weights. finally, the performance of the final portfolio is calculated, which means that the return obtained one year after the training period is calculated (line 12). 5 results this section presents the results obtained using the k-means dtw model and the b3 model (model that uses clustering by economic sectors). for both models, a training period of 1 year was used, that is, to generate the portfolio for year n, year \(n-1\) was used for training. in the experiments, the income of the portfolios generated by the two proposed asset allocation models is compared with (i) the annual income of the bovespa index, both annual and accumulated (sect. 5.2); and (ii) the accumulated income of the 20 best investment funds in multimarket assets of b3 companies, to analyze whether the portfolio generated by the algorithm is competitive with other portfolios (funds) on the market (sect. 5.3). before discussing the results achieved (portfolios generated and their incomes), the results of k-means dtw clustering technique are initially presented. algorithm 1 assetallocation function full size image 5.1 k-means dtw clustering to perform the clustering, the tslearn library was used, being a python package that provides machine learning tools for time series analysis. from the tslearn library, the (untrained) model clustering.timeserieskmeans was imported, which is the model that receives 4 inputs: (i) k which is the number of centroids, (ii) n which is the number of times that k-means algorithm will be executed with different centroid seeds (the final results will be the best result of n executions), n = 2 was used in the experiments, (iii) the metric that will be used to calculate the distance between two time series, dtw was used in the experiments, (iv) the number of iterations for the process of calculating the barycenter of each of the k clusters, 10 iterations were used in the experiments, and (v) the random state field, which is the generator used to start the first centroids. in the experiments, to compare the different time series, they were standardized so that they were all on the same amplitude scale. to find the best value of k (number of clusters/centroids), the elbow method was used. the elbow method is based on the idea that as you increase the number of clusters, the variation within each cluster decreases, but at some point the improvement becomes insignificant. this point, on a graph, is what is called an elbow and indicates the ideal number of clusters for your data. the sum of squared distances (ssd) was used as the criterion for evaluating variation within each cluster. each of the seven portfolios (one portfolio per year) used 4 years in the past to create their clusters. for example, a 2019 portfolio used time series data from jan/2015 to dec/2018. the 4-year interval was reached after realizing that portfolios that used clusters generated by shorter intervals ended up generating very unbalanced clusters (many companies concentrated in a few clusters and many clusters with few companies). very unbalanced clusters ended up generating portfolios with five or fewer assets, which goes against the selection mechanism described in sect. 4. using 4 years had consequences on the number of companies analyzed in each year, as the k-means dtw time series clustering algorithm, for its proper functioning, requires that all companies exist in those 4 years (i.e., the time series must exist entirely in this interval). for example, to generate the 2019 cluster, only companies that have been on the stock exchange at least since 01/01/2015 were used. companies that entered the stock market after that date were ignored to generate the 2019 clusters. in table 1, there are the numbers of clusters per year obtained by the k-means dtw algorithm, that is, the optimal value for k found using the elbow method and the number of companies distributed in the k clusters per year. table 1. number of clusters and number of companies distributed in the k cluster per year obtained by the k-means dtw algorithmfull size table 5.2 results of b3 and k-means dtw models table 2 shows the monthly incomes of the b3, k-means dtw and bovespa index models, along with the annual and historical accumulated incomes. we can see that both the b3 and k-means dtw models had a much higher accumulated income (in the long term) than bovespa. the accumulated incomes are 310,86%, 453,65% and 119,01%, respectively. both the b3 and k-means dtw models also proved to be safer than bovespa. they do not have a negative income during this period in any year, while bovespa showed a drop of −11,59% in 2021. table 2. monthly, annual and accumulated incomes; and annual risk of the b3, k-means dtw and bovespa index models from 2017 to 2023.full size table figure 2 shows the evolution of the accumulated income of the b3 and k-means dtw models compared to bovespa. it can be noted that the annual evolution of the portfolios generally has the same shape as the bovespa index (they vary together), and, in most cases, the proposed models have a higher income. fig. 2. evolution of the accumulated income of the b3 and k-means dtw models compared with bovespa. full size image the year with the highest income, 2019, obtained by the k-means dtw model, with an income of 85,56% is analyzed. figure 3 (left) shows the evolution of the risk of the portfolio from 1 to 15 assets (the assets have ids from 0 to 14). the final portfolio has 15 assets for this year. figure 3 (right) shows the evolution of the shape ratio of the best portfolio for 2019 (better distribution of weights that maximizes the sharpe ratio) found by the genetic algorithm in each generation. finally, fig. 3 shows the evolution of the accumulated monthly income of the respective portfolio compared to the bovespa index (fig. 4). fig. 3. evolution of risk (left) and evolution of sharpe ratio (right) of the 2019 portfolio obtained by the k-means dtw model. full size image fig. 4. accumulated monthly income of the 2019 portfolio obtained by the k-means dtw model. full size image three points are worth highlighting: (i) for most portfolios, an evolutionary asymptote was reached, close to generation 35 of the genetic algorithm; (ii) the evolution of the risk of the 2019 portfolio showed an increase when the fifth asset was added, going from a risk of 19% to 33%, and this was only accepted (increase in risk) because the increase in the portfolio’s expected return was even greater, going from an expected return of 91% to an expected return of 165%; and (iii) at the end of 2019, there was a dizzying increase in income, which can be explained by the significant appreciation of euca3, logn3 and brge8 assets. euca3 assets went from an initial value of r$8.24 in jan/2019 to r$55.70 at the end of dec/2019. logn3 assets went from an initial value of r$8.13 in jan/2019 to r$20.90 at the end of dec/2019. similarly, brge8 assets went from an initial value of r$3.22 in jan/2019 to r$9.37 at the end of dec/2019. table 3. stock fundsfull size table 5.3 proposed models compared to the market in this section, the performances of the two proposed models with real stock funds are compared. the following criteria were used to select the funds: (i) the fund must be composed integrally or mainly of assets; (ii) all assets must be from brazilian assets; and (iii) the fund must exist within the entire analyzed period, from 01/01/2017 to 12/31/2023. among the funds that meet the three requirements, the 20 funds with the highest accumulated incomes over that period were selected. table 3 shows these funds and what would be the relative position of the two models proposed in this study, taking into account their accumulated incomes over the same period. table 3 shows that both models achieved excellent positions within the top 20 funds with the highest incomes. k-means dtw model was in ninth place and b3 model was in twentieth place. the fund incomes data was obtained from the mais retorno website, which is a website that compares and ranks the best investment funds per year. on the website, it is possible to make the cut within the desired period (2017–2023) and obtain the income of each fund within that period. 6 conclusion based on the results presented, it can be concluded that both models in the long term are extremely competitive with the current stock funds present in the brazilian financial market. in other words, the algorithm’s main selection strategy, which is the selection of the most promising asset (an asset that presents the highest sharpe ratio) from each cluster of companies, proved to be promising for both models. as observed in the results section, the k-means dtw model proved to be more profitable than the b3 model, since the k-means dtw model presented an accumulated income of 453,65% and b3 model had an accumulated income of 310,86%. additionally, both models performed well in pandemic years (crisis years). b3 model had an income of 33,9% and 26,05% and dtw model had an income of 26,04% and 10,5%, in 2020 and 2021, respectively. it is worth noting that the total income in both models may be even higher, since, in this study, the income received as dividends was not taken into account. references doering, j., kizys, r., juan, a.a., fito, a., polat, o.: metaheuristics for rich portfolio optimisation and risk management: current state and future trends. oper. res. perspect. 6, 100121 (2019) mathscinet  math  google scholar  elton, e.j., gruber, m.j., brown, s.j., goetzmann, w.n.: modern portfolio theory and investment analysis, 5th edn. wiley, new york (2009) math  google scholar  ertenlice, o., kalayci, c.b.: a survey of swarm intelligence for portfolio optimization: algorithms and applications. swarm evol. comput. 39, 36–52 (2018) article  math  google scholar  kalaycı, c.b., ertenlice, ö., akyer, h., aygören, h.: a review on the current applications of genetic algorithms in mean-variance portfolio optimization. pamukkale univ. j. eng. sci. 23, 470–476 (2017) article  google scholar  liu, y., zhou, g., zhu, y.: maximizing the sharpe ratio: a genetic programming approach. (2020). ssrn 3726609 google scholar  markowitz, h.: portfolio selection. j. financ. 7(1), 77–91 (1952) math  google scholar  metaxiotis, k., liagkouras, k.: multiobjective evolutionary algorithms for portfolio management: a comprehensive literature review. expert syst. appl. 39(14), 11685–11698 (2012) article  math  google scholar  neely, c., weller, p., dittmar, r.: is technical analysis in the foreign exchange market profitable? a genetic programming approach. j. financ. quant. anal. 32(4), 405–426 (1997) article  math  google scholar  serra, j., arcos, j.l.: an empirical evaluation of similarity measures for time series classification. knowl.-based syst. 67, 305–314 (2014) article  math  google scholar  sharpe, w.f.a.g.j.b.j.v.: investments, 6th edn. prentice-hall international editions. prentice-hall, new york (1998) google scholar  streichert, f., ulmer, h., zell, a.: evolutionary algorithms and the cardinality constrained portfolio optimization problem. in: operations research proceedings, pp. 253–260. springer (2004) google scholar  download references author information authors and affiliations school of arts, sciences and humanities, university of são paulo, são paulo, brazil edsson israel andonaegui alonso, karina valdivia delgado & francisco carlos b. dos santos authors edsson israel andonaegui alonsoview author publications search author on:pubmed google scholar karina valdivia delgadoview author publications search author on:pubmed google scholar francisco carlos b. dos santosview author publications search author on:pubmed google scholar corresponding author correspondence to edsson israel andonaegui alonso . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alonso, e.i.a., delgado, k.v., dos santos, f.c.b. (2025). combining clustering and genetic algorithms for portfolio optimization: a case study with b3 companies. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_10 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_10 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords memetic genetic algorithms stock portfolio optimization k-means dtw publish with us policies and ethics profiles edsson israel andonaegui alonso view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature standing on the shoulders of giants | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper standing on the shoulders of giants conference paper first online: 30 january 2025 pp 416–430 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) standing on the shoulders of giants download book pdf download book epub lucas f. f. cardoso  orcid: orcid.org/0000-0003-3838-32149, josé de sousa ribeiro filho  orcid: orcid.org/0000-0002-8836-418810, vitor c. a. santos  orcid: orcid.org/0000-0002-7960-307911, regiane s. kawasaki francês  orcid: orcid.org/0000-0003-3958-064x9 & … ronnie c. o. alves  orcid: orcid.org/0000-0003-4139-056211  show authors part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 379 accesses 2 citations abstract although fundamental to the advancement of machine learning, the classic evaluation metrics extracted from the confusion matrix, such as precision and f1, are limited. such metrics only offer a quantitative view of the models’ performance, without considering the complexity of the data or the quality of the hit. to overcome these limitations, recent research has introduced the use of psychometric metrics such as item response theory (irt), which allows an assessment at the level of latent characteristics of instances. this work investigates how irt concepts can enrich a confusion matrix in order to identify which model is the most appropriate among options with similar performance. in the study carried out, irt does not replace, but complements classical metrics by offering a new layer of evaluation and observation of the fine behavior of models in specific instances. it was also observed that there is 97% confidence that the score from the irt has different contributions from 66% of the classical metrics analyzed. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others decoding machine learning benchmarks chapter © 2020 use and interpretation of item response theory applied to machine learning chapter © 2022 comparing irt properties among different category numbers: a case from attitudinal measurement on physics education research article open access 30 november 2023 1 introduction artificial intelligence (ai) systems are present in all spheres of society. despite the different types of ai that exist, machine learning (ml) techniques are the most commonly used, especially for more objective tasks such as classification problems [7]. the increasing advancement of its applications requires models with increasingly higher success rates and a low chance of error [8]. the binary classification task is a classic ml task, in which the model is challenged with a dataset that has only two classes, normally translated into class 1 (positive) and 0 (negative). this format is used because in binary problems a class tends to be the objective of classification, e.g., in a dataset that has information on patients in a hospital who do or do not have a certain disease, the objective of a classifier would be to identify whether a new patient has the disease (positive) or does not (negative) [18]. with this standard, it is possible to evaluate the model’s performance based on its confusion matrix and extract the most classic evaluation metrics used in ml, such as accuracy, f1 score and recall [18]. despite being widely used, classical metrics can only evaluate in a quantitative way, as they only count errors and successes and then evaluate the model over the entire test set, without differentiating the instances that are classified [11]. recent work aims to overcome this limitation by applying other forms of evaluation within the ml context. an interesting approach that has been gaining ground is the use of concepts from psychometrics to evaluate models. works such as [1, 6, 19] use item response theory (irt), to more accurately measure the ability of models, due to its main characteristic of being instance-level. with irt it is possible to evaluate the performance of a model against a specific instance of the test set, in a way that allows the model and data relationship to be further explored. the application of irt in the context of ml is not to replace classical metrics, but rather to add more information to prior knowledge. given the above, this work investigates how irt can be used to open the box of the classical confusion matrix. the confusion matrix typically delivers its evaluation in a batch of instances format, therefore, it does not allow observing small fluctuations that the instances cause in the generalization of the models, in their learning. it is observed that irt is capable of bringing an extra evaluative layer by standing on the shoulders of the confusion matrix and thus reinforcing the assessment of the models’ ability, enabling a more assertive decision regarding the comparison between models that eventually perform in a similar way. a new performance metric was also introduced that evaluates intrinsic aspects of learning, not yet explicitly covered by classical performance metrics. the remainder of the paper is organized as follows: sect. 2 presents the necessary theoretical framework of the article, and explains the concepts of using irt applied to ml; sect. 3 deals with the methodology used to carry out the experiments proposed in the article; sect. 4 presents the results obtained by the experiments and discusses what was observed; sect. 5 concludes the article and presents future work. 2 background 2.1 item response theory the objective of this section is to explain the psychometric concepts of irt, how it works and how it can be used in machine learning. as already mentioned in sect. 1, irt emerged within psychometrics as a new way of evaluating individuals’ performance on the same test. just like in classic ml metrics, the classic way of evaluating people is done through a test where performance is measured based on the individual’s mistakes or successes in the test, so that 1 is added for each correct answer and 0 for each error. although it is still widely used, this format is not capable of measuring an individual’s real ability, as it does not consider the complexity of the different items that exist in the test [3]. the irt then emerges as a new assessment approach that is centered on the items and not on the test itself. its objective is to calculate the probability of success of a given respondent j on a specific item i considering the ability \(\theta \) that this individual has and the parameters that describe item i. there are 3 main parameters: (\(a_{i}\)) discrimination, whose value represents how well a given item is able to distinguish respondents with high ability from respondents with low ability, the higher its value, the more discriminating the item is; (\(b_{i}\)) difficulty, restates how difficult an item is to answer correctly, the higher its value, the more difficult an item is; (\(c_{i}\)) guessing, the parameter that indicates the minimum chance that an item has of being answered correctly or the chance of a low-skill respondent getting the item right at random [4]. irt has different logistic models that are used to calculate the probability of a correct answer, determined by the number of item parameters they use. the logistic model that uses the three item parameters is called the three-parameter logistic model (3pl) for dichotomous items, i.e., items where only whether it was answered correctly or not is considered. the 3pl logistic model is defined by eq. 1, where \(p(u_{ij} = 1\vert \theta _{j})\) is the probability of the respondent \(\theta _{j}\) correctly answering the dichotomous item \(u_{ij}\) [3, 4]. $$\begin{aligned} p(u_{ij} = 1\vert \theta _{j}) = c_{i} + (1 c_{i})\frac{1}{1+ e^{-a_{i}(\theta _{j}-b_{i})}} \end{aligned}$$ (1) despite being item-centered, irt is also capable of generating a final score on an individual’s performance. to do this, you can use the concept of true score [9], which generates a score \(trues_{j}\) for each respondent based on the sum of the probabilities of a correct answer \(p(u_{ij} = 1\vert \theta _{j})\) calculated for each test item, the true score is defined by eq. 2. $$\begin{aligned} trues_{j} = \sum _{i=1}^{n} p(u_{ij} = 1\vert \theta _{j}) \end{aligned}$$ (2) item characteristic curve. one of the main products of the irt is the item characteristic curve (icc), which allows analyzing the relationship between item parameters and the probability of a correct answer given the variation in ability. figure 1 presents an example of the icc. it is possible to visualize the behavior of the probability of a correct answer increasing as the ability \(\theta \) also increases. in the icc the guessing parameter c determines the lowest probability of a correct answer that the curve will have, while the discrimination a determines the slope of the curve, so that the higher the value of a the steeper the curve becomes. the difficulty b and the ability \(\theta \) are calculated on the same scale, so that if \(\theta =b\) the chance of a correct answer is equal to 50%. the difficulty determines the point on the curve where this happens [3]. fig. 1. example of item characteristic curve. full size image 3 methodology in this section, the methodology used to carry out the experiments with the application of irt in ml will be presented. as mentioned in sect. 1, this work aims to explore how irt can be useful for evaluating ml models considering the confusion matrix obtained in a classical classification problemfootnote 1. although its use is aimed at evaluating individuals, as explained in sect. 2.1, a simple analogy is enough to apply irt in the context of ml when considering respondents as models while items are test instances. to calculate the probability of a correct answer, the 3pl logistic model was used, due to the fact that it considers the probability of a random hit, as it is possible for the models to get the instances right by guessing. to estimate the item parameters of the dataset instances and the ability of the models, the birnbaum method [10] was used, where the item parameters are first estimated by considering only the responses of all models and subsequently the ability is estimated from the item parameters obtained and the response vector of each model. as a case study, the heart-statlogfootnote 2 dataset was chosen. the heart-statlog is a heart disease dataset, which is composed of 270 instances with 13 features. figure 2 illustrates the methodology used to calculate the irt estimators, which is separated into 6 main steps. fig. 2. flowchart of the methodology. full size image 1. the heart-satolog dataset is separated into training and testing. the necessary pre-processing of the data is also carried out at this stage, such as coding categorical features. 2. next, 200 random models are generated from different classification algorithms that will be trained with the training dataset. at this stage, 10 fine-tuned models are also trained, whose performance will be evaluated later. 3. the trained random models must then classify the instances of the test dataset to generate the response matrix from their predictions. the fitted models also classify the test dataset. the response matrix is a matrix where the columns are the instances and the rows are the dichotomous response vectors for each instance, with 1 being correct and 0 being incorrect. 4. the response matrix that was obtained is used to estimate the item parameters of the test dataset. 5. with the item parameters calculated, it is possible to estimate the ability of the fine-tuned models and then calculate the probability of a correct answer and the irt scores. 6. the results obtained are then divided according to the confusion matrix to analyze the performance of the models. to estimate accurate values of item parameters, 200 random classifiers from 10 different learning algorithms were used decision tree (dt), random forest (rf), ada boost (ada), gradient boosting (gb), bagging (bag), multilayer perceptron (mlp), k-nearest neighbors (knn), support vector machine (svm), linear support vector machine (lsvm) and linear discriminant analysis (lda). to achieve this, 20 models were generated with randomly chosen hyperparameters for each of the 10 algorithms used. this way, it is possible to generate a diversity of answers, as if there were different students of different levels answering a test. the scikit-learn library [14] was used to generate all the classification algorithms. for performance analysis, 10 models were generated, one for each algorithm used. these 10 models are trained and hyper-parameterized using the grid search strategy with cross-validation provided by scikit learn, with 5-folds for cross-validation. the r package ltm [16] was used to estimate the irt item parameters from the response matrix. subsequently, the ability of the fitted models was estimated using the catsim [12] python package, this same package was used to calculate the probability of a correct answer and the icc’s of the test instances. although irt already has the true-score, presented in sect. 2.1, it is understood that it is important to also consider the error for a fairer evaluation of the models. for this, the concept of total-score, presented in [19], was used, where the probability of error is subtracted from the score for instances that were incorrectly classified. the total-score is defined by the eq. 3, where \(i'\) corresponds to items that were answered correctly while \(i''\) corresponds to items answered incorrectly. $$\begin{aligned} totals_{j} = \sum _{i=1}^{i'} p(u_{ij} = 1\vert \theta _{j}) \sum _{i=1}^{i''} 1 p(u_{ij} = 1\vert \theta _{j}) \end{aligned}$$ (3) 4 results and discussion in this section, the results obtained from the experiments carried out in this research are presented and discussedfootnote 3. as presented in sect. 3, the heart-statlog dataset was chosen as a case study because it is a known and easy-to-understand dataset. the heart-statlog is a dataset that contains data from patients who may or may not have heart disease. it has 270 instances with 13 patient diagnostic features, with 120 instances of patients who have heart disease (positive class) and 150 instances of patients who do not have heart disease (negative class), so the dataset has a balance of 55.55% for the negative class (majority) and 44.45% for the positive class (minority). in this work, the division 70% was used for training and 30% for testing. fig. 3. heart-statlog item parameter histograms. the orange bars are the instances of the minority class while the blue ones are the majority class. (color figure online) full size image to understand the performance of models, it is important to first understand the complexity of the dataset. figure 3 generally shows the item parameters estimated for the heart-statlog, in which the instances were organized into a histogram for each item parameter separating the majority and minority class instances. according to irt concepts, the dataset can be considered not very difficult due to the negative average of difficulty with −2.67 and suitable for comparing respondents due to the positive average of discrimination with 3.13. however, despite the positive average discrimination, it is important to highlight that the dataset presents around 24.7% of instances with negative discrimination. while instances with high discrimination values mean that they are good at certifying the models’ ability, instances with negative values mean that they are not well formed or even defined, so if a model hits many instances with these characteristics it means that this model may not be reliable. it is also noted that there is a balance between the frequency of majority and minority instances present in the 5 bins for the difficulty and discrimination histograms, this was already expected given the balance of 55%–45% between the classes. for the guessing parameter, the minority class presented the highest frequency of instances with a high probability of random success, despite balancing. previous research has already highlighted that models tend to hit instances of the minority class by chance more frequently [5]. fig. 4. heart-statlog instances arranged by item parameters. full size image this distribution of instances by the guessing parameter becomes even more evident when analyzing fig. 4. although the largest number of instances of the majority class have 0 guessing, which means that chance does not tend to influence the correct answer, there are still 9 instances (11%) that have a greater chance of being classified by guessing. from the figure it is also evident that some instances of the minority class have difficulty outliers, where there are three instances that are considered much easier than the others because they have very negative values. it is understood that in a dataset it is expected that there will be determining examples of a class distribution. once you understand the behavior of the data, you can then look at the classifiers. table 1 presents the results of classic ml metrics for the 10 fine-tuned models with their respective ranking positions. when observing only the classic metrics, the gb model presents the highest score achieved in 5 of the 6 metrics considered. if the objective of this experiment was to choose the best model, with a brief analysis it is possible to indicate that the gb would be ideal. however, gb only has the 6th best score if only the recall is considered and in this case this is very pertinent information. simply put, the recall metric represents how well a model was able to correctly classify true positives instances, and in the case of a context-sensitive dataset that aims to correctly identify patients who have heart disease, the recall metric has great relevance. if only recall is considered, the model that presents the best performance is lsvm. table 1. scores obtained by each model for classic ml metrics.full size table although lsvm has the best recall score, the model only has the 5th best average performance among all classic metrics. rf has the second best average performance, behind only gb, and has the second best recall score, a condition that would qualify it as a candidate to be chosen as the best model. the precision and recall metrics tend to have inverse performances, i.e. to increase one the other tends to decrease, and this occurs in this example where the rf only presents the 5th precision score. if you also consider the high variation in the ranking as an elimination criterion, it is the mlp that has the 3rd best performance on average and also has the lowest variation in position among the rankings, with high values in both precision and recall. in this scenario, what would be the best model to choose? how else can models be evaluated appropriately? to try to answer these questions, the performance of the models will be analyzed from the perspective of irt. the first step was to calculate the irt scores proposed in sect. 3. for the true score, which only considers the respondents’ probability of being correct, it was the bag model that appears first, and it is important to highlight that the difference between true scores is only 0.0002 for the second (gb). for the total score that considers whether there was an error or a correct answer, gb appears again with the highest score, this helps to reaffirm its position as the best model in this scenario (see table 2). table 2. true and total scores for the models.full size table fig. 5. nemenyi test heatmap for evaluation metrics. full size image to assess whether there is a gain in using irt scores in conjunction with classical metrics, the friedman [15] and nemenyi [13] tests were performed on the results of all metrics to confirm whether there are different distributions. the friedman test is a non-parametric test used to check whether there are different distributions among sampled data. its result is a p-value, which if it is less than a certain limit, then it is considered that there are different distributions in the data. the nemenyi test is complementary and is used to check which distributions are different when performing pairwise comparisons. the friedman test resulted in a p-value of approximately 1.9e-09, when considering a p-value threshold of 0.05 it can be stated that there are different distributions between scores. next, a nemenyi test was performed to verify which evaluation metrics are different. figure 5 presents the result of the nemenyi test, where it is possible to see that the p-value values of the true score do not show any difference from the classic metrics. on the other hand, interesting results are observed for the total score, where the f1 and recall metrics are the only ones that do not present a p-value smaller than 0.05. thus, applying the confidence measure, there is 52.25% confidence that f1 is statistically different from the total score, while there is 24.84% confidence that the recall is statistically different, for the other metrics the lowest confidence is 97.73%. such results point to the total score as a metric capable of giving visibility to new nuances in the models’ performance. to confirm the result obtained, the instances were then opened using the concepts of the irt item characteristic curves. an ideal confusion matrix should present values only on its main diagonal. with this in mind, the icc’s of each instance were separated according to the models’ confusion matrix, called the item characteristic confusion matrix curve (iccmc). figure 6 shows the iccmc of the gb model, in which it is possible to observe that the model performs well for true positives and true negatives instances. therefore, all tp instances have high discrimination values and are considered good instances to attest to the model’s ability. furthermore, only one well-formed instance appears among the fp, it is expected that only instances with negative discrimination appear between the fp and fn, as can be observed in the set of fn instances of the gb model. the point of greatest attention is the tn set, where 7 (16.67%) are instances that are not suitable for evaluating the model’s performance (red lines). this can also be observed by the average information value of the instances, where the set of tps presents a high average information value with 0.320, while the tns present a low average information value with 0.043. fig. 6. iccmc of the gb model, where a is the average discrimination value, b is the average difficulty, c is the average guessing, totals is the total score value, inf is the average information of the instances and n is the number of instances. the blue lines represent instances with positive discrimination, the red lines are instances with negative discrimination, and the black line is the average icc of the instances. (color figure online) full size image fig. 7. iccmc of the rf model. full size image in direct comparison with the iccmc of the rf, it is highlighted that the rf has a lower performance compared to the gb (see fig. 7). this can be confirmed by analyzing the sets of tp and tn instances, where the gb total score was higher in both cases. it is also noted that, although rf hits more tp instances (28) than gb (25), 4 instances classified by rf present negative discrimination. in addition, 3 well-formed instances appear in the fp and fn sets. although the recall of rf is greater than gb, the set of tp instances of gb is composed only of well-formed items while rf presents instances that are not suitable for evaluating the model. another interesting detail is the high guessing value for the instances of the tp set seen in all iccmc presented, as previously observed in fig. 6, the minority class has a high chance of getting it right by chance. this raises the question of whether the models have actually learned how to classify this class or are just getting lucky. a possible future work would be to investigate more incisively the impact of the guessing parameter and whether these instances are really well formed. fig. 8. iccmc of the svm model. (color figure online) full size image other models with interesting results are svm and lda, where the best configuration of icc curves of the instances arranged in the confusion matrix sets is noted. as can be seen in fig. 8 for the svm there is only one instance considered not suitable for evaluation in the tp set and only instances with negative discrimination in the fp and fn sets as expected in an ideal scenario. for the tn set, there are only 2 malformed instances, but it is interesting that among them there is an instance that is repeated as tn in all models. for lda, the tp set is made up of instances suitable for evaluation, while there are three instances with negative discrimination (red lines) in the tn set. in the work [17] the authors indicate that not all instances of a dataset are equally good for learning, it is common knowledge that models need quality data to be trained. if the same thinking is applied to the evaluation, where instances with negative discrimination are removed from the analysis because they are considered bad by the irt, the result would be that lda and svm would present the best performance in all classical ml metrics (see table 3). furthermore, when analyzing the iccs within the confusion matrix, lda would be preferable because it presents the most reliable tp set for the heart-statlog scenario. furthermore, another point that makes lda preferable is its explanatory nature [2]. in a context-sensitive scenario such as the one analyzed, a simpler model to explain its predictions is also more suitable. which model is ideal for a given dataset will depend on the context in which it is inserted and determining the ideal model should only be done after a more thorough analysis of performance considering the complexity of the data. a deeper analysis of the relationship between instances at the feature level would be interesting to further explore the use of icc’s in the confusion matrix. table 3. ranking of scores obtained for classic metrics without negative discrimination.full size table 5 final considerations the empirical evaluation of ml models remains the most common way to analyze the performance of a classifier on a dataset. this work was developed with the aim of deepening the classic form of evaluation considering the performance of models at the instance level using irt. a case study was presented where the aim was to choose the best model for a given heart disease classification problem with the heart-satlog dataset. irt allowed a deeper understanding of the relationship between model and data, by combining the traditional way of evaluating the confusion matrix with the psychometric concepts of irt, represented by the idea of the item characteristic confusion matrix curve (iccmc). furthermore, the direct relationship between irt scores and classic metrics was explored and it was shown that the total score has statistically significant differences in 66% of the evaluated metrics, with a confidence of 97%. an experiment was carried out with 10 models from 10 different classification algorithms, while the classic metrics pointed to gb as the most suitable model, the irt estimators allowed us to understand the data and how the models behaved in each specific instance. in a scenario where the entire test dataset is used for evaluation, irt allowed us to point out with greater confidence that the gb would be the most suitable, but in another scenario with a filter of the evaluated instances, the lda model would be the most suitable. if it was possible to see further in models’ abilities it is by standing on the shoulders of confusion matrix. as future work, it is interesting to explore more datasets of different types and contexts, and evaluate whether it would be possible to derive a new metric that would allow evaluating models directly considering the complexity of the data in relation to its features. another aspect to be observed is the limitations of the methodology used. it is noted that the results obtained by the irt are conditioned by the population of respondents and items selected for evaluation. thus, the results may be different if another set of models and items is selected. the irt itself already has tools to perform the evaluation considering different populations; it is interesting to explore this in future work. notes 1.link to the source code of the methodology: https://github.com/lucasferrarocardoso/irt_confusion_matrix. 2.heart-statlog link: https://www.openml.org/search?type=data&status=active&id=53. 3.all results generated are available at the link: https://github.com/lucasferrarocardoso/irt_confusion_matrix/tree/main/results. references araujo santos, v.c., cardoso, l., alves, r.: the quest for the reliability of machine learning models in binary classification on tabular data. sci. rep. 13(1), 18464 (2023) article  math  google scholar  arrieta, a.b., et al.: explainable artificial intelligence (xai): concepts, taxonomies, opportunities and challenges toward responsible ai. inf. fusion 58, 82–115 (2020) article  math  google scholar  baker, f.b.: the basics of item response theory. eric, college park (2001) google scholar  cai, l., choi, k., hansen, m., harrell, l.: item response theory. ann. rev. stat. appl. 3, 297–321 (2016) article  math  google scholar  cardoso, l.f., et al.: explanation-by-example based on item response theory. in: xavier-junior, j.c., rios, r.a. (eds.) bracis 2022. lncs, vol. 13653, pp. 283–297. springer, cham (2022). https://doi.org/10.1007/978-3-031-21686-2_20 chapter  math  google scholar  cardoso, l.f.f., santos, v.c.a., francês, r.s.k., prudêncio, r.b.c., alves, r.c.o.: decoding machine learning benchmarks. in: cerri, r., prati, r.c. (eds.) bracis 2020. lncs (lnai), vol. 12320, pp. 412–425. springer, cham (2020). https://doi.org/10.1007/978-3-030-61380-8_28 chapter  math  google scholar  lee, r.s.: artificial intelligence in daily life. springer, singapore (2020). https://doi.org/10.1007/978-981-15-7695-9 book  math  google scholar  linardatos, p., papastefanopoulos, v., kotsiantis, s.: explainable ai: a review of machine learning interpretability methods. entropy 23(1), 18 (2020) article  math  google scholar  lord, f.m., wingersky, m.s.: comparison of irt true-score and equipercentile observed-score “equatings’’. appl. psychol. meas. 8(4), 453–461 (1984) article  math  google scholar  martínez-plumed, f., prudêncio, r.b., martínez-usó, a., hernández-orallo, j.: making sense of item response theory in machine learning. in: proceedings of the twenty-second european conference on artificial intelligence, pp. 1140–1148 (2016) google scholar  martínez-plumed, f., prudêncio, r.b., martínez-usó, a., hernández-orallo, j.: item response theory in ai: analysing machine learning classifiers at the instance level. artif. intell. 271, 18–42 (2019) article  mathscinet  math  google scholar  meneghetti, d.d.r., junior, p.t.a.: application and simulation of computerized adaptive tests through the package catsim. arxiv preprint arxiv:1707.03012 (2017) nemenyi, p.: distribution-free multiple comparisons. in: biometrics, vol. 18, p. 263. international biometric soc 1441 i st, nw, suite 700, washington, dc 20005-2210 (1962) google scholar  pedregosa, f., et al.: scikit-learn: machine learning in python. j. mach. learn. res. 12, 2825–2830 (2011) google scholar  pereira, d.g., afonso, a., medeiros, f.m.: overview of friedmans-test and post-hoc analysis. commun. stat.-simul. comput. 44(10), 2636–2653 (2015) article  mathscinet  math  google scholar  rizopoulos, d.: ltm: an r package for latent variable modeling and item response theory analyses. j. stat. softw. 17(5), 1–25 (2006) article  math  google scholar  smith, m.r., martinez, t.: reducing the effects of detrimental instances. in: 2014 13th international conference on machine learning and applications, pp. 183–188. ieee (2014) google scholar  sokolova, m., lapalme, g.: a systematic analysis of performance measures for classification tasks. inf. process. manag. 45(4), 427–437 (2009) article  math  google scholar  de sousa ribeiro filho, j., cardoso, l.f.f., da silva, r.l.s., carneiro, n.j.s., santos, v.c.a., de oliveira alves, r.c.: explanations based on item response theory (exirt): a model-specific method to explain tree-ensemble model in trust perspective. expert syst. appl. 244, 122986 (2024) google scholar  download references author information authors and affiliations federal university of pará ufpa, belém, brazil lucas f. f. cardoso & regiane s. kawasaki francês federal institute of pará ifpa, belém, brazil josé de sousa ribeiro filho vale technological institute itv, belém, brazil vitor c. a. santos & ronnie c. o. alves authors lucas f. f. cardosoview author publications search author on:pubmed google scholar josé de sousa ribeiro filhoview author publications search author on:pubmed google scholar vitor c. a. santosview author publications search author on:pubmed google scholar regiane s. kawasaki francêsview author publications search author on:pubmed google scholar ronnie c. o. alvesview author publications search author on:pubmed google scholar corresponding author correspondence to lucas f. f. cardoso . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri ethics declarations disclosure of interests the authors have no competing interests to declare that are relevant to the content of this article. rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper cardoso, l.f.f., ribeiro filho, j.d.s., santos, v.c.a., kawasaki francês, r.s., alves, r.c.o. (2025). standing on the shoulders of giants. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_29 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_29 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords machine learning classification item response theory publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language conference paper first online: 30 january 2025 pp 324–338 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language download book pdf download book epub marcos piau  orcid: orcid.org/0009-0001-1490-34769, roberto lotufo  orcid: orcid.org/0000-0002-5652-08529,10 & rodrigo nogueira  orcid: orcid.org/0000-0002-2600-60359,11  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 392 accesses 10 citations abstract despite advancements in natural language processing (nlp) and the growing availability of pretrained models, the english language remains the primary focus of model development. continued pretraining on language-specific corpora provides a practical solution for adapting models to other languages. however, the impact of different pretraining settings on downstream tasks remains underexplored. this work introduces ptt5-v2, investigating the continued pretraining of t5 models for portuguese. we first develop a baseline set of settings and pretrain models with sizes up to 3b parameters. finetuning on three portuguese downstream tasks (assin2 sts, assin2 rte, and tweetsentbr) yields sota results on the latter two. we then explore the effects of different pretraining configurations, including quality filters, optimization strategies, and multi-epoch pretraining. perhaps surprisingly, their impact remains subtle compared to our baseline. we release ptt5-v2 pretrained checkpoints and finetuned monot5 rerankers on huggingface in their respective collections at https://huggingface.co/unicamp-dl. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others sabiá: portuguese large language models chapter © 2023 pre-trained models for representation learning chapter © 2023 evaluating the effectiveness of pre-trained language models in predicting the helpfulness of online product reviews chapter © 2024 1 introduction transformer-based pretrained language models have established themselves as the core paradigm in the field of natural language processing (nlp). starting with the advent of bert [12], which popularized the “pretrain, then fine-tune” approach and the use of the transformer architecture itself, these models acquire a general-purpose language representation by unsupervised pretraining on extensive corpora of unlabeled text. the dynamics of the pretraining process had been studied in depth by many works, like raffel et al. [34], that introduced t5, and scaled up models to billions of parameters and set new sotas across many tasks. the trend towards increasing model sizes and datasets to improve performance motivated studies like kaplan et al. [23] on scaling laws and hoffmann et al. [18], who demonstrated the importance of training data size relative to model size for compute-optimal training regimes; more recently work from gadre et al. [14] specifically examined the influence of extended pretraining on downstream task performance. despite the extensive study of pretraining dynamics, the focus has predominantly been on english, leaving non-english languages less explored. continued pretraining presents a strategic approach to adapting these models to additional languages and domains using significantly less data and computational resources than training from scratch. this method involves further pretraining on language-specific corpora, which has been shown to substantially enhance model performance on downstream tasks in the target language [5, 6, 11, 25, 32, 42]. however, there is a lack of detailed investigations into how different settings during the continued pretraining phase influence on downstream tasks performance, with most studies merely aiming for benchmark-leading results without a thorough examination of the underlying factors. in this work, we study the continued pretraining of t5 models for the portuguese language, analyzing the impact of various settings on downstream task performance. rather than solely focusing on achieving state-of-the-art results, our study also investigates how factors like model size, optimization schedules, and the application of quality filters in the pretraining dataset affect the performance. we continue the pretraning of google’s t5 with up to 3 billion parameters on portuguese texts. by experimenting with different configurations in the pretraining stage, we observe nuanced effects on downstream tasks, with some settings only marginally outperforming the baselines. our findings also suggest that while continued pretraining enhances model capabilities, the increments in performance diminish as model size increases. t5 models [34] demonstrate adaptability across various natural language processing (nlp) tasks due to their encoder-decoder architecture. this structure enables them to process text for both understanding and generation, providing an advantage over encoder-only models like bert. while not the focus of this work, t5’s adaptability to instruction-based fine-tuning, as seen in flan-t5 [8], also enables effective zero-shot and few-shot applications. these factors, combined with the scarcity of portuguese pretrained encoder-decoder models, motivate our choice to continue the investigation the t5 architecture in this study. 2 related work the t5 model [34] is an encoder-decoder transformer, and one of its main innovations was to cast all tasks into a text-to-text format, allowing for a unified approach; scaling up models to 11b parameters, they consolidated the transfer learning approach, setting new sotas for glue [46], superglue [45], cnn/daily mail [17] benchmarks. it was pretrained using the “span corruption” objective over the c4 (“colossal clean crawled corpus”) dataset, where random consecutive spans in the input are replaced by special mask tokens, and the model is trained to predict these corrupted tokens (models with up to 11b parameters). building upon this foundation, mt5 [48] extended the t5 framework to multilingual settings, having been pretrained on mc4, a multilingual dataset covering 101 languages (models up to 13b parameters). ptt5 [6] further adapted t5 for portuguese by continuing the pretraining of t5 models on the brwac dataset [44]. this approach led to significant improvements in downstream portuguese language tasks, which were further enhanced by a portuguese tokenizer. for clarity, we’ll refer to the work of carmo et al. as ptt5-v1. other notable international adaptations of t5/mt5 include it5 [40] (italian), afriteva [22] (low-resource african languages), arat5 [29] (arabic), and plt5 [7] (polish). bertimbau [42], a popular adaptation of the bert encoder model, remains influential within portuguese language modeling. others exploring encoder architectures include albertina [37], debertinha [5], the work of gomes et al. [16] (which pretrains a roberta model), and de morais et al. [28]. reflecting a broader trend, numerous recent portuguese models prioritize decoder-only architectures, such as sabiá [32], glória [27], bode [15], cabrita [25], and gervásio [39]. in the encoder-decoder space, the work of carmo et al. (ptt5-v1) explored adapting t5 models for portuguese. beyond generic portuguese models, several works specialize in custom domains: de barros et al. [2] and bertabaporu [10] were designed for portuguese social media data, while bertaú [13] focuses on financial language. 3 methodology this section describes the methodology for pretraining and evaluating our key experiments, covering the pretraining dataset, language-specific vocabulary, model architectures, optimization strategies, and finetuning and validation processes for downstream tasks. 3.1 unsupervised continued pretraining as the pretraining data, we utilized the portuguese segment of the mc4 dataset (hereafter referred to as mc4-pt), comprising approximately 524 gb of uncompressed text across 169 million documents. this dataset is significantly larger than the one used for the pretraining of ptt5-v1 models, which originated from the brwac dataset [44] and consisted of around 15 gb of text from 7.4 million documents after preprocessing. we adopted the portuguese language vocabulary from ptt5-v1. this sentencepiece unigram tokenizer [24], comprising 32,000 tokens, was trained over a corpus of 2 million documents from the portuguese wikipedia. this vocabulary shares the same number of tokens and control tokens as t5, facilitating the direct use of google’s model checkpoints. as the pretraining objective, the span corruption task was employed, utilizing batches of 128 sequences of 512 tokens (65,536 tokens) a methodology consistent with the baseline experiment by raffel et al. [34]. adafactor optimizer [41] with a constant learning rate of 0.001 and cross-entropy as loss was used during the entire pretraining process. using these experimental settings, we started from google’s original checkpoints with sizes from t5-small (60m parameters) up to t5-3b (3b parameters), and performed a complete epoch of continued pretraining over the mc4-pt dataset. considering these settings, a single epoch over the mc4-pt dataset comprises approximately 1,764,515 training steps and 116 billion training tokens. additional pretraining experiments are detailed in sect. 5.1. both pretraining and finetuning experiments utilized tpuv2-8 and tpuv3-8 devices, leveraging t5 [34] and seqio [36] frameworks. 3.2 supervised finetuning on downstream tasks we assess the impact of our pretraining on three portuguese language downstream tasks: assin2 rte, assin2 sts, and tweetsentbr. the assin2 dataset [31] provides two tasks: rte (recognizing textual entailment), which involves determining whether one sentence entails another, and sts (semantic textual similarity), which quantifies the semantic similarity between sentence pairs on a 1–5 scale. the tweetsentbr dataset [4] is a sentiment analysis task for brazilian portuguese tweets, classifying them as positive, negative, or neutral. tables 1 and 2 show further details and examples for each task. table 1. downstream tasks datasets.full size table table 2. downstream tasks inputs and targetsfull size table we finetuned the pretrained models over 100 epochs with batches of 128 sequences and a maximum length of 512 tokens, using adafactor as the optimizer with a constant learning rate of 0.001. the model checkpoint yielding the best performance on the validation set was selected for testing, and greedy decoding was utilized as the decoding method. because tweetsentbr lacks a validation set, we reserved 10% of the training data for validation and used the remaining 90% for training. all tasks were approached using a text-to-text format. specifically for the assin2 sts task, which involves the prediction of continuous values in the range between 1 to 5, we adopted the strategy from raffel et al. [34], by rounding the target scores to the nearest 0.2 increment and converting these to strings, thus framing it as a multiclass classification problem compatible with the text-to-text format. to compare the quality of the new checkpoints against existing alternatives, we also use the same finetuning procedure on google t5 and mt5 models. 3.3 monoptt5 rerankers to evaluate the adaptability of the ptt5-v2 models for information retrieval tasks, specifically passage reranking, we trained monot5 rerankers [30] using checkpoints generated as described in sect. 3.1. we named these models monoptt5. monot5 rerankers are used for passage reranking, a two-stage process: first, a less computationally expensive method like bm25 generates an initial set of relevant documents for a given query; the reranker model then reranks a subset of these documents to improve relevance ordering. during training, the model learns in a supervised text-to-text manner to generate tokens corresponding to relevant and non-relevant labels. for inference, we greedy decode a single token and calculate the softmax over the logits of the two possible tokens, using the probability of the positive class as the relevance score. we adapted the input and target format for the portuguese language to the structure "pergunta: {query} documento: {document} relevante:", assigning the tokens “sim” (relevant) and “não” (non-relevant). this format is applied during both training and inference, regardless of the input language. the training data originated from the mmarco dataset [3], a translated version of the ms marco passage retrieval dataset [1], originally in english, to 13 languages, including portuguese. the training subset consists of triples (query, relevant passage, non-relevant passage), which we split into two training example pairs, with each pair containing the query matched to one passage – either relevant or non-relevant – thus creating one example for each label. we created a bilingual portuguese-english training dataset by randomly assigning one of the two languages to each training triplet. this “translate-train” approach [9, 20, 48] leverages synthetic data augmentation by integrating machine translations with original text data to substantially expand the available training material in the target language. prior research [3, 9, 38] has shown the effectiveness of this bilingual training strategy, motivating our adoption of this method. the models were trained for 100k steps with batch sizes of 128 sequences and a maximum length of 512 tokens, utilizing adafactor with a constant learning rate of 0.001 as the optimizer. instances exceeding the maximum token length were excluded from the training dataset; these constituted approximately 0.01% of the training samples and were predominantly attributed to noisy translation data. given the significant computational resources required for training these rerankers, we focused exclusively on models based on the main ptt5-v2 checkpoints. to evaluate the rerankers, we first used bm25 to generate an initial set of relevant documentsfootnote 1 and then rerank the top-1000 documents. retrieval metrics are calculated by comparing this ordered list with the relevance judgments from each dataset. we consider two retrieval scenarios: in-domain (using the “small dev” set of 6,980 queries from the mmarco-pt dataset) and zero-shot (using the portuguese subset of 249 annotated queries from mrobust [21]). due to the longer document length in mrobust, we segmented documents into sliding sentence windows using a spacy [19] sentencizer pipeline, with a maximum length of 8 and a stride of 4 sentences to mitigate truncation during reranking. table 3. main results after finetuning. for supervised sotas, model sizes are as follows: [38] (335m), [37] (900m), [2] (110m), [38], and [21] (580m). npm results exclude retrieval tasks.full size table 4 main results table 3 shows the results on the downstream tasks considered. in the assin2 rte task, our 3b sets a new sota, surpassing the current one by 0.61 f1-macro points. for the tweetsentbr dataset, we achieved better performance than current finetuned sotas with ptt5-v2-large and ptt5-v2-3b, by 0.52 and 1.54 f1-macro points, respectively, but our results are worse when comparing to gpt-4. we highlight that our ptt5-v2 were trained exclusively on each task training data using the text-to-text framework without any data augmentation or adaptation to the model’s architecture, unlike the works of [2] and [38], which held the sota for tweetsentbr and assin2 rte. in the assin2 sts task, our models did not surpass the current sota; regardless, ptt5-v2 still shows better performance than mt5 and t5 models with approximate sizes, and this is also the only task where a smaller ptt5-v2 model (ptt5-v2-large) has better performance of a large one ptt5-v2-3b. in addition to using individual metrics for each task, we also incorporate the normalized preferred metric (npm) [43] to facilitate the evaluation of the overall performance of a pretrained model across multiple tasks. the npm normalizes a task’s preferred metric (e.g., f1-macro for assin2 rte), assigning a value of 0 to represent random performance and 100 to denote maximum performance. below is the equation used to calculating the npm for a given model and set n of tasks: $$\begin{aligned} {\text {npm}} = \frac{100}{n} \sum _{i=1}^{n} \frac{{\text {[raw preferred metric]}}_i {\text {[random score]}}_i}{{\text {[maximum score]}}_i -{ \text {[random score]}}_i} \end{aligned}$$ (1) given that monoptt5 rerankers were exclusively trained starting from ptt5-v2 pretrained checkpoints, retrieval tasks were excluded from this evaluation. therefore, we only considered assin2 rte, assin2 sts, and tweetsentbr tasks. for each model, we calculate its aggregate performance by first determining the npm for each task and then computing the average of these values. our ptt5-v2 models have higher npm values than t5 and mt5 models with considerably more parameters: for example, ptt5-v2-base is only surpassed by t5-3b (\(\sim \)13.6x larger) and t5-xl (\(\sim \)16.81x larger); this performance gap, however, is most pronounced in smaller models, narrowing as model size increases. a similar result was also observed by xue et al. [48], which analyzed the performance of t5 and mt5 models performance on the squad benchmark [35], observing a performance gap between t5-small and t5-base vs mt5 models of equivalent sizes, which is diminished starting from t5-large. this performance gap observed on smaller model sizes is advantageous when we consider environments constrained by computational resources, increasing the maximum attainable level of performance; additionally, a language-specific tokenizer reduces text splitting into fewer tokens, leading to lower latency and the potential to accommodate more text within the same maximum token context window. interestingly, mt5 models tend to show lower npm values, except in the 3 billion parameter range, where they slightly outperform t5 models. for the retrieval tasks, our monoptt5 rerankers were able to set new sotas for both mmarco-pt and mrobust-pt. for the mmarco-pt dataset, models starting from t5-base size were able to surpass the current sota; the 3 billion parameters reranker obtained a gain of +0.026 points of mrr@10. in the mrobust-pt, our large and 3b rerankers surpassed the current sota by +0.071 and +0.121 in terms of ndgc@20, respectively. a more in-depth analysis of the retrieval tasks is conducted on sect. 5.2. 5 ablations 5.1 additional pretraining experiments this sections includes pretraining experiments additional to those described in sect. 3.1. comparison with ptt5-v1: given the title of our work, a pertinent question arises: how do ptt5-v2 models compare with the work in ptt5-v1? a few key differences exist in the pretraining of ptt5-v1. notably, it utilized brwac, a considerably smaller dataset, and a slightly different pretraining objective (denoising, where some input tokens are masked and the model is trained to predict the original text, rather than span corruption). additionally, ptt5-v1 employed models ranging from t5-small to t5-large with an adafactor optimizer using a learning rate of 0.003 (threefold larger than our setting). ptt5-v1 also explored both t5’s original english vocabulary and a portuguese language-specific tokenizer. in contrast, ptt5-v2 exclusively uses the latter. to ensure a fair comparison, we finetuned the ptt5-v1 checkpoints following the methodology outlined in sect. 3.2. figure 1 presents the npm values for both ptt5-v1 and ptt5-v2, alongside comparisons to mt5 and t5 models. the data corroborates the enhancements achievable through monolingual pretraining on the target language, which is further augmented by employing a dedicated tokenizer. in comparison between the two ptt5 iterations, a performance disparity favoring ptt5-v2 is apparent for the small and large sizes, with the largest gap observed for large; however, models of the base size exhibit marginally superior performance in the ptt5-v1 variant. surprisingly, mt5’s performance lags behind all other models, including the monolingual english t5, except in the 3b parameter range, where it closely matches ptt5-v2 and surpasses t5. fig. 1. npm vs parameters for varying t5 configurations. full size image quality filters: in our primary experiments, we utilized the entire mc4-pt dataset, containing approximately 116 billion training tokens, in the pretraining phase. in this additional experiment, we consider the massivetext quality filters [33] to investigate the impact of this filtering process on downstream tasks. applying these filters to mc4-pt reduces the number of training tokens to approximately 82 billion, a reduction of about 30%. this experiment was restricted to models of t5-base size, keeping the same batch size and optimization strategy used in the main experiments. figure 2 shows the effect of using massivetext’s quality filters on downstream task performance, measured in terms of npm. pretraining with the filtered dataset shows an upward trend in performance, which continues without saturation up to the duration of one mc4-pt epoch, which is the last point in the plot. it’s important to note that one epoch in the filtered dataset has fewer steps than the full dataset, so at one full dataset epoch, data is repeated for the filtered dataset. despite the upward trend favoring the pretraining with the filtered dataset, the difference in terms of npm is small at the one mc4-pt epoch mark. fig. 2. effect of using quality filters on mc4-pt. vertical lines indicate the number of pretraining tokens for each dataset considered. full size image pretraining optimization strategy: in our exploration of optimization strategies, we initially employed adafactor with a constant learning rate. this ablation extends our investigation to the “inverse square root” learning rate schedule, as utilized by [34] in their final pretraining experiments. this learning rate schedule computes the rate as \(\frac{1}{\sqrt{\max (n,k)}}\), where n represents the current step, and k is the number of warm-up steps. raffel et al. [34] used \(k=10,000\), which sets the learning rate of 0.01 for the initial 10k steps, subsequently decreasing exponentially. the learning rate at the end of the pretraining, which consisted of around 1 million steps, was close to 0.001, the same one used during the finetuning. figure 3 illustrates the difference between these optimization strategies. fig. 3. learning rate schedules: constant vs. inverse square root scheduler as a function of training steps. full size image fig. 4. effect of scheduler used during pretraining. epochs are relative to mc4-pt. full size image attempting to closely mirror the t5 pretraining recipe, we applied this identical schedule in our preliminary experiments. however, we observed a rapid overshoot in the training losses for t5-large t5-3b within hours; adjusting n only delayed the overshoot without averting it. transitioning to a constant learning rate of 0.001 solved the overshoot issue, leading to stable pretraining loss across all model sizes and simplifying our experimental setup. because overshoot was observed only in the larger models, and also given the computational costs associated with pretraining the larger models, we performed additional pretraining experiments with “inverse square root” scheduler for t5-small and t5-base models only. number of pretraining epochs: in the primary set of experiments, the mc4-pt dataset was fully utilized for pretraining over one epoch. to explore the influence of the number of pretraining epochs on downstream task performance, we conducted experiments over various epochs, including partial epochs (0.25, 0.5, and 0.75 of an epoch). considering the significant time and computational resources required for extended pretraining, especially with larger models, we limited the pretraining of the t5-large model to two epochs and the t5-3b model to a single epoch. the reduced epoch duration for the t5-small and t5-base models enabled more extensive pretraining periods for these configurations. in fig. 4, the npm values for t5-base are shown with the constant and the inverse square root scheduler, across a varying number of pretraining epochs. it is observed that there is a difference between these two optimization strategies: the inverse square root scheduler has the advantage for up to two epochs; afterwards, the constant learning rate takes the upper hand, and by the last epoch considered, they reach the same value. an increasing trend in the values of npm is also noted for more epochs. 5.2 monoptt5 rerankers the information retrieval tasks reported in 3 represents the performance of our monoptt5 experiments, developed with the methodology describe in sect. 3.3; in this section we also report the results for other approaches, including bm25, and dense retrieval using multilingual-e5 [47] models. the dense models are used as a single stage retrieval system without reranking; dense indexing and retrieval is performed with a100 and v100 gpus on google colab, leveraging the pyserini framework. for mrobust-pt, which contain longer documents, we mitigate document truncation by using the same splitting strategy described in the sect. 3.3, using the maximum score among the document segments as the document score. figures 5 and 6 are used to illustrate the discussion presented in this section. results in table 3 only shows the effectiveness in each retrieval task of our monoptt5 models and the sota competitors. for the in-domain retrieval task, the mmarco-pt dataset, we note that bm25 is easily surpassed by all alternatives considered, and the effectiveness figures for monoptt5 rerankers and multilingual-e5 models are similar when consider the size range in common, and monoptt5 rerankers effectiveness is above sota starting from models of size t5-base. for mrobust-pt, representing a zero-shot setting, bm25 is only surpassed by the mt5 reranker from jeronymo et al. [21], and monoptt5 models starting from t5-base size. fig. 5. retrieval results on mmarco-pt. mcolbert and mt5 values are from bonifacio et al. [3]. total size excludes embedding parameters. full size image fig. 6. retrieval results on mrobust-pt. mcolbert and mt5 values are from jeronymo et al. [21]. total size excludes embedding parameters. full size image 6 conclusion in this study, we introduced ptt5-v2, exploring the continued pretraining of t5 models for the portuguese language. we pretrained t5 models using a portuguese language tokenizer, over a portuguese language corpus. the finetuned models achieved sota on assin2 rte and tweetsentbr datasets, two of the three downstream tasks considered. additionally, we applied these pretrained checkpoints to develop monot5 rerankers customized for the portuguese language, achieving top performance on the mmarco-pt and mrobust-pt datasets. our main results supports the evidence of a performance gap favoring monolingual models over english-focused and multilingual models, a gap that narrows as model capacity increases. this underscores the importance of language-specific pretraining, and our analysis of pretraining settings suggests that while data filtering, optimization strategies, and pretraining duration can offer incremental improvements, the overall effects were limited in comparison to our baseline settings and the core pretraining recipe remained robust. notes 1.all applications of bm25 in this work use pyserini’s implementation [26] with default parameters \(k_1=0.9\) and \(b=0.4\). references bajaj, p., et al.: ms marco: a human generated machine reading comprehension dataset (2018) google scholar  de barros, t.m., pedrini, h., dias, z.: leveraging emoji to improve sentiment classification of tweets. in: proceedings of the 36th annual acm symposium on applied computing, sac 2021, pp. 845–852. association for computing machinery, new york, ny, usa (2021). https://doi.org/10.1145/3412841.3441960 bonifacio, l., et al.: mmarco: a multilingual version of the ms marco passage ranking dataset (2022) google scholar  brum, h., volpe nunes, m.d.g.: building a sentiment corpus of tweets in brazilian portuguese. in: calzolari, n., et al. (eds.) proceedings of the eleventh international conference on language resources and evaluation, lrec 2018, may 2018. european language resources association (elra), miyazaki, japan (2018) google scholar  campiotti, i., rodrigues, m., albuquerque, y., azevedo, r., andrade, a.: debertinha: a multistep approach to adapt debertav3 xsmall for brazilian portuguese natural language processing task (2023) google scholar  carmo, d., piau, m., campiotti, i., nogueira, r., lotufo, r.: ptt5: pretraining and validating the t5 model on brazilian portuguese data (2020) google scholar  chrabrowa, a., et al.: evaluation of transfer learning for polish with a text-to-text model (2022) google scholar  chung, h.w., et al.: scaling instruction-finetuned language models (2022) google scholar  conneau, a., et al.: unsupervised cross-lingual representation learning at scale (2020) google scholar  costa, p.b., pavan, m.c., santos, w.r., silva, s.c., paraboni, i.: bertabaporu: assessing a genre-specific language model for portuguese nlp. in: mitkov, r., angelova, g. (eds.) proceedings of the 14th international conference on recent advances in natural language processing, september 2023, pp. 217–223. incoma ltd., shoumen, bulgaria, varna, bulgaria (2023) google scholar  cui, y., yang, z., yao, x.: efficient and effective text encoding for chinese llama and alpaca (2024) google scholar  devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding (2019) google scholar  finardi, p., viegas, j.d., ferreira, g.t., mansano, a.f., caridá, v.f.: bertaú: itaú bert for digital customer service (2021) google scholar  gadre, s.y., et al.: language models scale reliably with over-training and on downstream tasks (2024) google scholar  garcia, g.l., et al.: introducing bode: a fine-tuned large language model for portuguese prompt-based task (2024) google scholar  gomes, j.r.s., et al.: deep learning brasil at absapt 2022: portuguese transformer ensemble approaches (2023) google scholar  hermann, k.m., et al.: teaching machines to read and comprehend (2015) google scholar  hoffmann, j., et al.: training compute-optimal large language models (2022) google scholar  honnibal, m., montani, i., landeghem, s.v., boyd, a.: spacy: industrial-strength natural language processing in python (2020, to appear). https://doi.org/10.5281/zenodo.1212303 hu, j., ruder, s., siddhant, a., neubig, g., firat, o., johnson, m.: xtreme: a massively multilingual multi-task benchmark for evaluating cross-lingual generalization (2020) google scholar  jeronymo, v., nascimento, m., lotufo, r., nogueira, r.: mrobust04: a multilingual version of the trec robust 2004 benchmark (2022) google scholar  jude ogundepo, o., oladipo, a., adeyemi, m., ogueji, k., lin, j.: afriteva: extending “small data” pretraining approaches to sequence-to-sequence models. in: cherry, c., (eds.) proceedings of the third workshop on deep learning for low-resource natural language processing, july 2022, pp. 126–135. association for computational linguistics, hybrid (2022). https://doi.org/10.18653/v1/2022.deeplo-1.14 kaplan, j., mccandlish, s., et al.: scaling laws for neural language models (2020) google scholar  kudo, t., richardson, j.: sentencepiece: a simple and language independent subword tokenizer and detokenizer for neural text processing. arxiv preprint arxiv:1808.06226 (2018) larcher, c., piau, m., finardi, p., gengo, p., esposito, p., caridá, v.: cabrita: closing the gap for foreign languages (2023) google scholar  lin, j., ma, x., lin, s.c., yang, j.h., pradeep, r., nogueira, r.: pyserini: a python toolkit for reproducible information retrieval research with sparse and dense representations. in: proceedings of the 44th annual international acm sigir conference on research and development in information retrieval, sigir 2021, pp. 2356–2362 (2021) google scholar  lopes, r., magalhães, j., semedo, d.: glória – a generative and open large language model for portuguese (2024) google scholar  de morais, l.p., da silva soares, a., da cm borges, v., da silva, n.f.f., pereira, f.s.: sub-language sentiment analysis in whatsapp domain with deep learning approaches. revista de sistemas de informaçao da fsma 1(31), 32–47 (2023) google scholar  nagoudi, e.m.b., elmadany, a., abdul-mageed, m.: arat5: text-to-text transformers for arabic language generation (2022) google scholar  nogueira, r., jiang, z., pradeep, r., lin, j.: document ranking with a pretrained sequence-to-sequence model. in: cohn, t., he, y., liu, y. (eds.) findings of the association for computational linguistics, emnlp 2020, online, november 2020, pp. 708–718. association for computational linguistics (2020). https://doi.org/10.18653/v1/2020.findings-emnlp.63 oliveira, h.g., real, l., fonseca, e. (eds.): proceedings of the assin 2 shared task: evaluating semantic textual similarity and textual entailment in portuguese, extended semantic web conference, no. 2583 in ceur workshop proceedings (2020) google scholar  pires, r., abonizio, h., almeida, t.s., nogueira, r.: sabiá: portuguese large language models, pp. 226–240. springer, heidelberg (2023). https://doi.org/10.1007/978-3-031-45392-2_15 rae, j.w., et al.: scaling language models: methods, analysis & insights from training gopher (2022) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. arxiv preprint arxiv:1910.10683 (2019) rajpurkar, p., zhang, j., lopyrev, k., liang, p.: squad: 100,000+ questions for machine comprehension of text (2016) google scholar  roberts, a., et al.: scaling up models and data with t5x and seqio (2022) google scholar  rodrigues, j., et al.: advancing neural encoding of portuguese with transformer albertina pt-* (2023) google scholar  rosa, g.m., bonifacio, l.h., de souza, l.r., lotufo, r., nogueira, r.: a cost-benefit analysis of cross-lingual transfer methods (2021) google scholar  santos, r., silva, j., gomes, l., rodrigues, j., branco, a.: advancing generative ai for portuguese with open decoder gervásio pt* (2024) google scholar  sarti, g., nissim, m.: it5: large-scale text-to-text pretraining for italian language understanding and generation (2022) google scholar  shazeer, n., stern, m.: adafactor: adaptive learning rates with sublinear memory cost (2018) google scholar  souza, f., nogueira, r., lotufo, r.: bertimbau: pretrained bert models for brazilian portuguese. in: cerri, r., prati, r.c. (eds.) lncs, pp. 403–417. springer, cham (2020). https://doi.org/10.1007/978-3-030-61377-8_28 chapter  google scholar  srivastava, a., et al.: beyond the imitation game: quantifying and extrapolating the capabilities of language models (2023) google scholar  wagner filho, j.a., wilkens, r., idiart, m., villavicencio, a.: the brwac corpus: a new open resource for brazilian portuguese. in: proceedings of the eleventh international conference on language resources and evaluation, lrec 2018 (2018) google scholar  wang, a., et al.: superglue: a stickier benchmark for general-purpose language understanding systems (2020) google scholar  wang, a., singh, a., michael, j., hill, f., levy, o., bowman, s.r.: glue: a multi-task benchmark and analysis platform for natural language understanding (2019) google scholar  wang, l., yang, n., huang, x., yang, l., majumder, r., wei, f.: multilingual e5 text embeddings: a technical report (2024) google scholar  xue, l., et al.: mt5: a massively multilingual pre-trained text-to-text transformer (2021) google scholar  download references acknowledgements we thank google for the tpu grant through the trc program. r lotufo is partially supported by cnpq (the brazilian national council for scientific and technological development) under grant 313047/2022-7. author information authors and affiliations school of electrical and computing engineering, state university of campinas (unicamp), campinas, brazil marcos piau, roberto lotufo & rodrigo nogueira neuralmind, campinas, brazil roberto lotufo maritaca ai, campinas, brazil rodrigo nogueira authors marcos piauview author publications search author on:pubmed google scholar roberto lotufoview author publications search author on:pubmed google scholar rodrigo nogueiraview author publications search author on:pubmed google scholar corresponding author correspondence to marcos piau . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper piau, m., lotufo, r., nogueira, r. (2025). ptt5-v2: a closer look at continued pretraining of t5 models for the portuguese language. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_23 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_23 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords portuguese t5 continued pretraining information retrieval publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature affective states in novice programmers: automatically detecting and analyzing the impact on learning | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper affective states in novice programmers: automatically detecting and analyzing the impact on learning conference paper first online: 30 january 2025 pp 50–65 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) affective states in novice programmers: automatically detecting and analyzing the impact on learning download book pdf download book epub hemilis joyse barbosa rocha  orcid: orcid.org/0000-0001-5422-927111, bruno almeida pimentel  orcid: orcid.org/0000-0001-5771-878310, evandro de barros costa  orcid: orcid.org/0000-0003-4663-871510 & … patricia cabral de azevedo restelli tedesco  orcid: orcid.org/0000-0001-9450-92199  show authors part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 384 accesses abstract programming is a complex cognitive activity that often poses a significant challenge for beginners. beyond the cognitive dimension, understanding the role of affective states, such as emotions, attitudes, and motivation, in the learning process can provide valuable insights for educators to improve teaching methodologies and curriculum design. despite this importance, few studies relate the cognitive aspects of beginning programmers to their affective states in the context of problem-solving. in this article, we explore the affective states reported by novice programmers in introductory programming courses, relating them to programming concepts and success rates during problem-solving activities. furthermore, we apply machine learning algorithms to detect affective states in novice programmers, considering both student and task characteristics. our initial results allow us to understand the affective states most prevalent in problems associated with each programming concept, as well as to reveal a significant association between affective states and the performance of novice programmers. positive affective states, including pleasure, motivation, and challenge, were positively correlated with better performance outcomes. on the other hand, negative affective states such as frustration, anxiety, and boredom impaired performance. additionally, our algorithms demonstrated good performance in detecting affective states. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others investigating the impact of emotions on the quality of novice programmers’ code chapter © 2024 analysis of permanence time in emotional states: a case study using educational software chapter © 2018 an architectural design and evaluation of an affective tutoring system for novice programmers article open access 24 october 2018 1 introduction programming problem-solving is widely recognized as a challenging endeavor for beginners, often leading to high rates of academic failure among students [28]. sheard et al. [5] further underscored the complexities of introductory coding, describing it as a subject difficult to both learn and teach. over the past four decades, research in computing education has extensively documented the hurdles students face when introduced to programming concepts [22, 23, 25]. these studies predominantly explore conceptual and cognitive barriers encountered during the program construction process, alongside challenges posed by specific programming constructs and control structures [21]. these learning challenges are not solely cognitive; they are often intertwined with affective states, particularly frustration. learning to code is frequently accompanied by feelings of frustration and lack of motivation, as coding is perceived as complex and daunting [3]. as students engage in early programming experiences to grasp fundamental concepts, the influence of affective states, notably negative ones like frustration, boredom, and anxiety, on novice programmers becomes apparent [18]. negative emotions, such as confusion and boredom, have been associated with poor performance in introductory programming courses [25]. these emotions have been shown to impede students’ working memory, consequently affecting their overall performance [29]. conversely, positive emotions, like concentration and motivation, are pivotal in enhancing learning outcomes [30]. this interplay between affective states and learning outcomes highlights the need for a comprehensive understanding of their relationship in the context of programming education [9]. recent research has focused on developing learning environments that recognize and respond to students’ affect [20, 24]. while these systems aim to improve students’ affective states during tutoring sessions, they may not foster long-term learning or develop students’ meta-affective capacity [7]. therefore, it is imperative to integrate affective awareness and self-regulation into programming education to support students’ holistic development [6]. despite this, few studies have explored the relationship between cognitive and affective aspects of beginning programmers during problem-solving activities [26]. investigating students’ emotional behavior when learning challenging programming concepts provides valuable insights into their affective responses and learning outcomes. by understanding how affective states and programming concepts influence problem-solving activities, educators can better support novice programmers’ learning journeys. in this article, we address the following research questions: rq 01: what is the relationship between programming concepts, success rate, and affective states reported by novice programmers during problem-solving? rq 02: has an effective automatic detector been developed to identify instances of affective states among novice programmers? through these questions, we intend to identify and detect the most prevalent affective states considering programming concepts and some implications for problem-solving. to achieve this, we conducted a study in three main stages: (i) data collection through an experiment with novice programmers; (ii) analysis of the collected data; and (iii) development of the detection model. in summary, the main contributions of this work are as follows: (i) a deeper understanding of the relationship between affective states, programming concepts, and the performance of programming students; and (ii) the development of an effective automatic detection model to identify affective states, exploring three relevant machine learning algorithms: decision tree, k-nearest neighbors, and multilayer perceptron neural network, with the decision tree model showing the best performance. 2 background and relatedwork the investigation of emotions within the learning process, particularly in tutoring systems, represents a continually evolving area of research. extensive efforts have been made to understand the intricate relationship between cognitive aspects and emotional states [37]. in the domain of computer programming, novice programmers commonly encounter both cognitive and emotional challenges [9, 21]. on the cognitive front, beginners often struggle with problem-solving tasks, including the formulation, execution, and evaluation of solutions [13]. novice programmers frequently have difficulties grasping basic programming structures such as specific constructs, control structures, variables, data structures, looping mechanisms, and logical flow with conditionals and boolean logic [21]. additionally, challenges arise when analyzing and designing mathematical and logical expressions, assigning variable names, and selecting appropriate data types and structures [33]. misinterpretation of conditional structures, particularly the or operator, further compounds these challenges [34]. however, the learning hurdles in programming extend beyond the cognitive realm and permeate the affective dimension. emotions have garnered significant attention in academic contexts, with boredom, confusion, pleasure, and frustration being prominent focal points [35]. consequently, a novice programmer’s problem-solving efforts can be significantly influenced by a spectrum of affective and mental states. for instance, unexpected feedback can induce confusion among students [36]. moreover, shifts in affective states often manifest in cognitive performance, with studies linking affective states to novice programmers’ final performance [10, 17]. high-arousal states like pleasure or fear have been associated with enhanced programming performance [19]. furthermore, a study conducted during the covid-19 pandemic shed light on the affective state of motivation among programmers [3]. in a study by bosch and d’mello [27], the prevailing emotions among novice programming students were investigated, alongside their relationship with behavior and academic performance. participants assessed various emotions during the programming process, ranging from basic emotions to learning-related ones. the results indicated that students commonly experienced a state of flow/engagement, followed by confusion, frustration, and boredom. similarly, chetty and van der westhuizen [26] explored students’ emotional experiences when grappling with challenging concepts. emotions such as anxiety, fear, confusion, and despair were identified as typical responses to difficult concepts. recognizing these emotions is crucial for educators in providing appropriate support during challenging learning moments. against this backdrop, numerous researchers have endeavored to develop solutions for identifying novice programmers’ emotional states during programming activities. vea et al. [2] developed a model to detect negative emotional states, particularly boredom, confusion, and frustration, among c++ learners. the detection relied on participants’ typing patterns and mouse movements, although detecting reduced frustration levels proved challenging. another study employed machine learning algorithms to predict emotional states among university students engaged in programming tasks of varying difficulty levels [14]. features derived from keyboard and mouse logs were utilized for emotional state classification, although some researchers caution against relying solely on log measures to accurately capture emotional processes [12]. self-reports have also been explored as an alternative approach, with harley et al. [20] reporting a 75% agreement rate between self-reports and video-based emotion detection. therefore, this article considers the emotional states reported by students within the learning environment during its utilization. 3 methodology the research described in this article is part of a project conducted in four distinct stages: the development of the virtual learning environment, approval of the project by the ethics board, data collection, and data analysis. section 3.1 will detail the developed learning environment. in this environment, programming students can record their affective states as they solve the proposed problems. these states include negative affective states (frustration, boredom, and anxiety), positive affective states (pleasure, motivation, confusion, and challenge), and neutral states. notably, half of these emotional states were identified as significant during the learning process [31], influencing many other learning environments to adopt portions of this set in their approaches [32]. additionally, it is important to highlight that, given the involvement of collecting and analyzing data from student interactions, the project was submitted to the local ethics committee and received approval before proceeding to the subsequent stages of data collection and analysis. the following subsections will detail the materials and methods used in this research, including information about the study participants, the data collection instruments, and the data processing techniques. 3.1 materials the material resources used in this study included a technological tool (virtual learning environmentfootnote 1) that captures problem-solving attempts and the affective states reported by students. hints: the hints in ada are organized into four levels, each providing increasingly specific guidance. the initial hints are abstract, while the final level almost provides the complete answer, requiring the student only to input the solution. students can request hints at any time, receiving progressively detailed assistance as needed. problem level assessment: after submitting a partial solution, the system asks the student to evaluate the difficulty level of the problem, with options ranging from very easy to very difficult. additionally, all problems are associated with the programming concepts presented in table 1. affective state: while solving programming problems, students can self-report their affective states using the top bar of ada. they can select from negative affective states (frustration, boredom, and anxiety), positive affective states (pleasure, motivation, confusion, and challenge), and neutral states. table 1. programming conceptsfull size table 3.2 data collect in this study, students actively participated as part of their regular programming curriculum at two technical high schools in northeastern brazil. the research took place during the first semester of the 2023 academic year and involved four programming classes: two cs1 classes and two cs2 classes. the cs2 course builds upon the foundation laid by cs1, with both courses utilizing java as the primary programming language. the selection of cs2 students was deliberate, as they were in the early stages of their computer science program, making them novices in the field. however, they had already acquired a fundamental understanding of programming and were reviewing programming concepts. this timing allowed us to conduct our study as these students were about to learn object-oriented programming within the cs2 curriculum. throughout the study, participants were provided with snacks during all problem-solving sessions. these sessions were organized into four blocks, totaling 16 problem-solving sessions, with four sessions dedicated to each class. the study successfully concluded with the active participation of 131 students who attended all problem-solving sessions. the breakdown of student participation across the session blocks is as follows: 220 students participated in the first block (averaging 55 students per class), 201 in the second block (averaging 50.25 students per class), 182 in the third block (averaging 45.5 students per class), and a final group of 131 students in the last block (averaging 42.5 students per class). among the participants, 74 were female, and 57 were male, with an average age of 17 (sd = 0.76). regarding their living arrangements, all students resided with their families. notably, 68% of the students did not live in the same town as the school, and 27% lived in rural areas. while all participants had access to the internet, only 27% owned a personal computer, 14% had a tablet, and the majority, 54%, possessed a smartphone. each student engaged with an average of 41.3 problems (sd = 25.0) during the study, and they tackled each problem an average of 4.3 times (sd = 2.7) to arrive at solutions. each problem-solving session was conducted face-to-face at the school using the tool presented in sect. 3.1. the first session lasted an average of 140 min, and the remaining sessions lasted 125 min each. during the first session of each class, the initial 15 min were dedicated to explaining the study’s objectives to the students, distributing the consent forms for participation, and explaining the functionalities of the ada environment, particularly the meaning of each affective state. the participants then engaged in problem-solving for 100 min. at the end of each session, interviews were conducted with the students for 25 min. regardless of whether the students finished the programming portion, the last 25 min were reserved for the researcher to ask questions about the behaviors observed during the programming task. the other sessions were divided into 100 min for problem-solving and 25 min for interviews. two researchers conducted all 143 studies. due to space constraints, this article focuses only on the programming part of our study, excluding the interview portion. in each problem-solving session, students could make several attempts to solve the problems. for each attempt, students had to make two submissions: an alternative solution and the code corresponding to the problem being solved. dataset. by running the experiment, we acquired two sets of data: the first comprising 5,902 problem-solving attempts and the second comprising 6,789 self-declarations of affective states. after merging these datasets based on trial numbers, we created a consolidated dataset consisting of 13,534 instances of student behavior. the variables present in this dataset include: problem identifier, difficulty level, problem concept, answer to the problem (alternative), affective state identifier, student identifier, number of errors, grade per problem, overall grade, number of attempts to solve the problem, and number of declarations of each affective state. 3.3 classification models we analyzed three stages: pre-processing, classification, and evaluation of the data. in the data pre-processing stage, we performed two crucial tasks: data cleaning and data transformation. during data cleaning, we removed missing data to avoid inconsistencies and errors during analysis and modeling. for data transformation, we applied normalization and standardization. in normalization, attribute values were adjusted to a common scale, generally between 0 and 1, allowing each attribute to contribute equally to the final result. in standardization, data were adjusted to have a mean of zero and a standard deviation of one, which is useful for standardizing deviations from the mean. in the classification step, we used a technique called 15-fold cross-validation. this method involves dividing the data into 15 parts and, in each iteration, training the model on 14 parts and testing it on the remaining part. by repeating this process 15 times, we increased the algorithm’s generalization capacity, resulting in more accurate predictions [15]. we applied three algorithms for classification: decision tree (using the cart algorithm), k-nearest neighbors (knn) with 3 neighbors, and neural networks using multi-layer perceptron (mlp). this selection was based on previous studies, and the implementation was carried out with the scikit-learnfootnote 2 library. in the evaluation stage, we analyzed the performance of each algorithm on the training and testing subsets using six metrics: area under the roc curve (auc), accuracy, precision, recall, f1 score, and kappa. auc evaluates the model’s ability to perform accurate classifications. precision, recall, and f1 score provide a comprehensive view of performance, while kappa checks whether the model determines the correct sequences for reasons that are not based on chance. a significance level of \(p-value < 0.05\) was used. 4 results and discussion rq 01: what is the relationship between programming concepts, success rate, and affective states reported by novice programmers during problem-solving? to address this research question, we investigated the students’ behavior during problem-solving using two main observation criteria: 1) we calculated the average grades assigned to each affective state for each concept. 2) we determined the average number of attempts required to solve the problems for each affective state associated with each concept. consequently, we conducted a detailed analysis of each concept and affective state. it is important to note that, to evaluate the results, we applied specific statistical tests, such as the wilcoxon test for comparing two samples and the friedman test for comparing more than two samples, to determine the significance level of the results. programming concepts. we calculated the average grades of students based on affective state and programming concept, while table 3 provides the average number of attempts to solve the same problems, categorized by affective state and programming concept. next, we will analyze each concept or group of concepts individually. in the context of concept \(c_{01}\), students reported significantly lower average marks when they experienced positive affective states of motivation (0.67, \(p=0.093\)) and challenge (0.69, \(p=0.159\)), as well as a lower average number of attempts while feeling motivated (1.37, \(p=0.097\)). in contrast, students who reported feeling pleasure achieved higher average marks (0.81, \(p=0.087\)), although there was no significant increase in the number of attempts (1.42, \(p=0.009\)). the negative affective state of anxiety also correlated with lower scores (0.69, \(p=0.097\)). in concept \(c_{02}\), there was no significant difference in mean scores related to different affective states (f=9.89, \(p=0.071\)). however, there were significantly more attempts when students felt challenged (4.34, \(p=0.088\)). for concepts \(c_{03}\), \(c_{04}\), and \(c_{05}\), related to the declaration and manipulation of variables, students achieved the highest mean scores in concept \(c_{05}\) when they experienced positive affective states (0.87, \(p=0.091\)), while they underperformed in negative affective states (0.72, \(p=0.077\)). there was no significant difference in the mean scores for concepts \(c_{03}\) and \(c_{04}\) (f=7.09, \(p=0.082\)). in concept \(c_{06}\), students who reported feeling confused and anxious achieved the highest average score of 0.85, although there was no significant difference between these averages (\(p=0.001\)). for concepts \(c_{07}\) and \(c_{08}\), which deal with increment and decrement of variables, students who felt pleasure achieved a significantly higher average score of 0.98 for concept \(c_{08}\) (\(p=0.061\)). additionally, students obtained significantly lower grades when they felt bored in the same concept. considering concepts \(c_{09}\), \(c_{10}\), and \(c_{11}\), which involve operators (boolean, relational, and logical), students performed significantly better (\(p=0.078\)) when they felt positive states. however, students made significantly more attempts when solving problems associated with \(c_{10}\) (\(p=0.072\)). concepts \(c_{12}\) and \(c_{13}\), which deal with conditional structures, were associated with higher average scores when students were motivated and lower scores when they felt frustrated. consequently, students achieved higher average grades in positive affective states (\(p=0.072\)). finally, for concepts \(c_{14}\) and \(c_{15}\), related to repetition structures and vectors, students who felt pleasure, motivation, and frustration when solving problems related to concept \(c_{15}\), and motivation in concept \(c_{14}\), achieved higher average grades. conversely, students who experienced frustration and boredom achieved higher averages in \(c_{15}\) and anxiety in \(c_{14}\). affective state. looking first at the positive affective states, the average values for concept \(c_{05}\) are significantly higher compared to the other concepts (\(p=0.070\)). this indicates that students experiencing pleasure in this concept had a higher average grade. in table 3, the average number of attempts for concept \(c_{05}\) is also lower compared to the other concepts, suggesting that students are more motivated and engaged with this concept. on the other hand, there are no significant differences in the average motivation values between the concepts in the two tables. the standard deviations are also similar (\(p=0.001\)). concepts \(c_{02}\) and \(c_{08}\) show significantly higher mean scores for challenge in the table compared to the other concepts (\(p=0.081\)). this may suggest that these concepts are perceived as more challenging. however, in table 3, the average number of attempts for these concepts is not the highest, indicating that students are not necessarily avoiding these challenging concepts (table 2). table 2. correlações de spearman nas taxas de sucesso dos alunos para cada conceitofull size table table 3. average number of attempts by students for each affective statefull size table table 4. number of students declaring each affective state per programming conceptfull size table regarding negative states such as anxiety, concepts \(c_{08}\), \(c_{10}\), and \(c_{11}\) seem to have significantly higher average scores compared to the other concepts (\(p=0.081\)). this may suggest that these concepts are precedents for better scores among anxious programmers. however, in table 3, the average number of attempts for these concepts is not the highest. to conclude the analysis of this research question, table 4 shows that the number of students declaring each affective state varies widely between affective states and concepts. for example, the number of students declaring affective states such as "boredom" or "anxiety" is lower compared to other affective states. rq 02: has the automatic detector been effectively developed to identify instances of affective states among novice programmers? table 5. results obtained when applying the knn algorithm to affective states (test)full size table table 6. results obtained when applying the mlp algorithm to affective states (test)full size table table 7. results obtained when applying the decision tree algorithm to affective states (test)full size table table 8. wilcoxon test p-value for classifiersfull size table the results of the knn algorithm, shown in table 5, indicate solid overall performance in classifying affective states. the accuracy (acu) varies from 0.85 to 0.91, with confusion presenting the best accuracy (0.91 ± 0.05). precision (pre) ranges from 0.78 to 0.87, and recall (rec) ranges from 0.69 to 0.74, indicating that knn is relatively effective in correctly identifying affective states. the f-measure (fme) and the kappa coefficient (kap) also present moderate to high values, with confusion once again standing out (fme = 0.79 ± 0.04, kap = 0.65 ± 0.03). the area under the roc curve (roc) is consistently high, ranging from 0.81 to 0.89, reinforcing the effectiveness of knn in distinguishing between different affective states (table 7). mlp, as shown in table 6, performs slightly worse compared to knn. the accuracy (acu) varies from 0.79 to 0.85, with confusion achieving the best performance (0.85 ± 0.04). precision (pre) ranges from 0.72 to 0.81, while recall (rec) ranges from 0.61 to 0.67, indicating that mlp may have difficulty capturing all positive examples. the f-measure (fme) and the kappa coefficient (kap) are at moderate levels, with lower values compared to knn. the area under the roc curve (roc) varies from 0.75 to 0.83, highlighting that, despite overall lower performance, mlp is still effective in distinguishing affective states. the decision tree algorithm, shown in table 8, demonstrated the best overall results among the three algorithms. the accuracy (acu) varies from 0.87 to 0.92, with confusion again achieving the best accuracy (0.92 ± 0.05). precision (pre) ranges from 0.80 to 0.88, and recall (rec) ranges from 0.71 to 0.75, showing a solid balance between correct identification and capturing positive examples. the f-measure (fme) and the kappa coefficient (kap) also present the highest values, indicating the consistency of the decision tree model. the area under the roc curve (roc) ranges from 0.83 to 0.90, reinforcing the algorithm’s ability to effectively distinguish between different affective states. the wilcoxon test results reinforce these conclusions, indicating statistically significant differences between the algorithms in several metrics. for each affective state, p-values are less than 0.05 in most cases, showing that the observed differences are not due to chance. in particular: for the pleasure state, all metrics showed significant p-values (<0.001), except fme (0.001), indicating a substantial difference between the algorithms. for the motivation state, all metrics presented p-values <0.001, suggesting a clear difference between the algorithms’ performances. for the challenge state, most metrics showed p-values <0.001, with a few exceptions (pre: 0.005, kap: 0.003). in the case of the confusion state, most p-values are <0.001, with some metrics slightly above this value (acu: 0.012, pre: 0.007, fme: 0.001). for the states of frustration, boredom, and anxiety, all metrics presented p-values <0.001, indicating significant differences between the algorithms. these results indicate that the decision tree, followed by knn and, finally, mlp, presents the best performance in detecting affective states. the statistical robustness provided by the wilcoxon test further validates the decision tree as the most effective algorithm for this context. 5 conclusion this paper explores the connection between cognitive aspects and the affective dimension of novice programmers’ learning experiences, investigating the possible relationships between programming concepts, affective states, and success rates in problem-solving activities. our research questions focused on the prevalence of specific affective states associated with various programming concepts, the interaction between these concepts and affective states, the success rates, and the influence of affective states on novice programmers’ performance. the analysis of research question 2 (rq02) revealed that the relationship between programming concepts, success rates, and affective states reported by novice programmers during problem-solving is complex and varies among different concepts and affective states. when examining individual programming concepts, we found that the relationship between affective states and success rates was inconsistent across all concepts. some concepts exhibited significant differences in success rates based on affective states, while others did not. for example, in concept \( c_{05} \) (related to variables), students who experienced pleasure achieved a significantly higher average score compared to those in negative affective states. conversely, for concept \( c_{02} \) (data types), there were no significant differences in mean scores based on affective states. these results suggest that the impact of affective states on programming performance may depend on the specific programming concept being addressed. positive affective states, such as pleasure, motivation, and challenge, were generally correlated with higher average scores, while negative affective states, such as frustration, boredom, and anxiety, were associated with lower average scores. however, these trends were inconsistent across all concepts, indicating that the relationship between affective states and programming performance is context-dependent. the analysis of the number of attempts made by students also varied based on affective states and concepts. in some cases, students in positive affective states made more attempts to solve problems, reflecting higher engagement and persistence. consequently, the relationship between programming concepts, affective states, and success rates among novice programmers is multifaceted and influenced by several factors. although positive affective states generally seem to impact programming performance positively, the specific effects vary depending on the context. notes 1.https://ada-projeto.vercel.app/. 2.https://scikit-learn.org/stable/. references grover, s., pea, r., cooper, s.: designing for deeper learning in a blended computer science course for middle school students. comput. sci. educ. 25(2), 199–237 (2015) article  math  google scholar  vea, f.f., blanco, m.e., bosch, n.: modeling negative emotions for novice programmers. in: proceedings of the ieee/acm 39th international conference on software engineering education and training (csee &t), pp. 1–5 (2017) google scholar  tek, o.e., ertekin, ş: implicit theories of intelligence and attitudes toward computer programming: a study with undergraduate students. educ. inf. technol. 23(6), 2685–2703 (2018) math  google scholar  sutter, t.r., beier, m.e.: students’ emotions during classroom learning: a comprehensive examination of multiple academic emotions. learn. individ. differ. 68, 108–120 (2018) math  google scholar  sheard, j., simon, b., carbone, a., butler, m.: a quantitative analysis of the effect of learning to program on problem-solving abilities. in: proceedings of the 14th annual sigcse conference on innovation and technology in computer science education, pp. 160–164 (2009) google scholar  richey, j.e., garas, n., brown, n.c.: more than a feeling: investigating the feasibility of using an affective learning companion to support project-based learning in computer science. in: proceedings of the 24th acm conference on innovation and technology in computer science education, pp. 60–66 (2019) google scholar  rebolledo-mendez, g., bassi, b., o’rourke, e.: meta-affective capacity: a theoretical framework for understanding students’ self-awareness and self-regulation in virtual learning environments. comput. educ. 181, 104401 (2022) math  google scholar  pekrun, r., linnenbrink-garcia, l., turner, j.e.: emotions and academic performance: introduction to the special issue. learn. instr. 46, 1–3 (2016) google scholar  pardos, z.a., heffernan, n.t.: affective states and state tests: investigating how affect throughout the school year predicts end of year learning outcomes. j. learn. anal. 1(1), 107–128 (2014) article  google scholar  ovans, a.:. how to beat boredom. harvard business review. https://hbr.org/2013/04/how-to-beat-boredom namkung, j.m., peng, p., lin, x.: the relation between positive and negative emotions and students’ academic self-regulation: a multi-method approach. contemp. educ. psychol. 58, 158–172 (2019) google scholar  munshi, a., hussain, a.: modeling student emotions for learning analytics: the role of fine-grained data. in: european conference on technology enhanced learning, pp. 457–462. springer, cham (2018) google scholar  medeiros, m.g., nunes, r., prates, r.o.: systematic mapping on difficulties faced by novice programmers: prevalent models and solutions. in: proceedings of the international conference on software engineering and knowledge engineering, pp. 463–468 (2018) google scholar  liu, y., liu, y., yao, y., li, h.: predicting students’ emotions in programming based on multimodal data. in: 2018 ieee/wic/acm international conference on web intelligence (wi), pp. 324–331. ieee (2018) google scholar  berrar, d.: cross-validation. in: encyclopedia of bioinformatics and computational biology, pp. 542–545. elsevier, oxford (2019) google scholar  kyrilov, a., prates, r. o.: students’ problem-solving strategies in introductory programming. in: proceedings of the 17th international conference on computer systems and technologies, pp. 95–102 (2016) google scholar  kumar, a., pal, s.: the rise of the angry programmer. acm sigsoft softw. eng. notes 33(5), 1–4 (2008) math  google scholar  kinnunen, p., malmi, l., salakoski, t.: through the eyes of a novice: professional vs. novice software developers’ viewpoints on program comprehension. in: proceedings of the 4th international workshop on computing education research, pp. 59–70 (2007) google scholar  khan, g.f., siddiqi, i.: mood, memory, and motivation: an empirical test of mood enhancement effect on memory performance. pak. j. psychol. res. 22(3–4), 21–30 (2007) math  google scholar  harley, j.m., bouchet, f., hussain, m.s., azevedo, r., calvo, r.a.: a multi-componential analysis of emotions during complex learning with an intelligent multi-agent system. comput. hum. behav. 29(1), 293–302 (2013) google scholar  grover, s., basu, s.: measuring problem-solving skills in introductory programming. acm trans. comput. educ. 17(2), 1–26 (2017) math  google scholar  ebrahimi, a., sheard, j., smith, e.: a novice programmer’s learning experience in the procedural and object-oriented paradigms. in: proceedings of the 25th sigcse technical symposium on computer science education, pp. 24–28 (1994) google scholar  du, b.l., wagner, e.e.: some thoughts on teaching novice programmers. acm sigcse bull. 18(2), 53–56 (1986) math  google scholar  defalco, j.a., et al.: detecting frustration and mitigating its effects on learning in serious games. user model. user-adap. inter. 28(1), 65–110 (2018) math  google scholar  coto, m. s., hernandez, a. v., pardo, a.: measuring programming students’ emotions using eeg. in: companion proceedings of the 2022 acm conference on innovation and technology in computer science education, pp. 62–68 (2022) google scholar  chetty, y., van der westhuizen, d.: hate, frustration, and learning programming. s. afr. j. high. educ. 27(3), 740–758 (2013) math  google scholar  bosch, n., d’mello, s.k.: detecting learning-centered affective states in the wild. ieee trans. affect. comput. 4(3), 298–310 (2013) math  google scholar  bennedsen, j., caspersen, m.e.: the ultimate barrier to teaching programming: teachers who do not know how to program. acm trans. comput. educ. 19(1), 1–28 (2019) math  google scholar  ashcraft, m.h., kirk, e.p.: the relationships among working memory, math anxiety, and performance. j. exp. psychol. gen. 130(2), 224 (2001) article  math  google scholar  ahmed, w., van der werf, g., kuyper, h., minnaert, a.: emotions, self-regulated learning, and achievement in mathematics: a growth curve analysis. j. educ. psychol. 105(1), 150 (2013) article  math  google scholar  craig, s.d., graesser, a.c., sullins, j., gholson, b.: affect and learning: an exploratory look into the role of affect in learning with autotutor. j. educ. media 29(3), 241–250 (2004) article  math  google scholar  karumbaiah, s., hafferjee, a., dillenbourg, p.: a case for contextualized affect sensing and feedback in computer-based learning environments. in: proceedings of the 9th international conference on learning analytics and knowledge, pp. 414–423 (2019) google scholar  gobil, a.r.m., shukor, z., mohtar, i.a.: novice difficulties in selection structure. in: 2009 international conference on electrical engineering and informatics. ieee, pp. 351–356 (2009) google scholar  herman, g.l.: describing novice programmers’ mental models: an investigation of descriptions of variables and assignment statements. j. educ. comput. res 46(2), 229–250 (2012) math  google scholar  pekrun, r., linnenbrink-garcia, l., turner, j.e.: emotions and academic performance: introduction to the special issue. learn. instr. 46, 1–3 (2016) google scholar  yang, y.t., newby, t.j., bill, r.l.: using socratic questioning to promote critical thinking skills through asynchronous discussion forums in distance learning environments. am. j. dist. educ. 30(1), 5–21 (2016) google scholar  barrett, l.f.: variety is the spice of life: a psychological construction approach to understanding variability in emotion. cogn. emot. 23(7), 1284–1306 (2009) article  math  google scholar  shneiderman, b., mayer, r.: syntactic/semantic interactions in programmer behavior: a model and experimental results. int. j. comput. info. sci. 8(1979), 219-238. liu2013sequences google scholar  liu, z., et al.: sequences of frustration and confusion, and learning. educational data mining (2013) google scholar  eckerdal, a., et al.: putting threshold concepts into context in computer science education. acm sigcse bull. 38(3), 103–107 (2006) article  google scholar  burleson, w., picard, r.w.: affective agents: sustaining motivation to learn through failure and a state of stuck. in: workshop on social and emotional intelligence in learning environments (2004) google scholar  download references author information authors and affiliations computer center-cin, federal university of pernambuco, recife, brazil patricia cabral de azevedo restelli tedesco institute of computing-ic, federal university of alagoas, maceió, brazil bruno almeida pimentel & evandro de barros costa federal institute of alagoas, maceió, brazil hemilis joyse barbosa rocha authors hemilis joyse barbosa rochaview author publications search author on:pubmed google scholar bruno almeida pimentelview author publications search author on:pubmed google scholar evandro de barros costaview author publications search author on:pubmed google scholar patricia cabral de azevedo restelli tedescoview author publications search author on:pubmed google scholar corresponding author correspondence to hemilis joyse barbosa rocha . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper rocha, h.j.b., pimentel, b.a., costa, e.d.b., tedesco, p.c.d.a.r. (2025). affective states in novice programmers: automatically detecting and analyzing the impact on learning. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_4 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_4 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords affective states novice programmer detect publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature a case study on water demand forecasting in a coastal tourist city | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper a case study on water demand forecasting in a coastal tourist city conference paper first online: 30 january 2025 pp 3–17 cite this conference paper download book pdf download book epub intelligent systems (bracis 2024) a case study on water demand forecasting in a coastal tourist city download book pdf download book epub antoniel kleber stefaniak9, pablo andretta jaskowiak9 & lucas weihmann9  part of the book series: lecture notes in computer science ((lnai,volume 15414)) included in the following conference series: brazilian conference on intelligent systems 377 accesses abstract urban water-demand (uwd) forecasting is crucial for efficient water management, improving distribution, and supporting environmental sustainability. in tourist destinations with significant seasonal variations in number of inhabitants (water consumers), accurate water-demand forecasting becomes particularly important. this work evaluates two statistical models for short-term uwd forecasting, namely, autoregressive integrated moving average (arima) and seasonal autoregressive integrated moving average (sarima). two different strategies for model deployment and comparison are considered: (i) a sliding window (sw) approach with one-year (1y) and two-year (2y) windows for training and; (ii) a expanding window (ew) approach. the arima model ployed with a sliding window (sw) with a two-year (2y) resolution achieved the best overall results, followed by sarima considering expanding window (ew) model. to place these outcomes in perspective, we performed a comparison with results from related work that took into account machine learning methods for regression for the same data. this comparison suggests that statistical methods provide results that are both competitive and robust in terms of quality for short-term forecasts. download conference paper pdf similar content being viewed by others uncertainty analysis of reservoir operation based on stochastic optimization approach using the generalized likelihood uncertainty estimation method article 01 july 2021 the role of constructed wetlands in creating water sensitive cities chapter © 2018 modelling and predicting annual rainfall over the vietnamese mekong delta (vmd) using sarima article open access 20 june 2024 1 context water scarcity is a pressing issue in various regions worldwide. even in areas not currently facing water shortage issues, efficient water management is crucial for environmental preservation and improved life quality of inhabitants. securing proper water usage is an intricate and multifaceted challenge, with significant efforts being undertaken from both public authorities and private initiatives focusing on urban water-demand (uwd). alongside improvements in treatment plants and distribution lines, which might include, for instance, the replacement of deteriorated pipework and proper maintenance of equipment, process monitoring, ranging from treatment to final consumption, starts to assume a key role. in this scenario, collected data might be employed, for example, to provide forecasts regarding water demand. if accurate, these forecasts can ultimately contribute to greater efficiency in water supply facilities. predictions for uwd can be performed on different horizons, depending on the requirements. there is no clear consensus in the literature regarding the actual definition and designation of the forecasting horizons used for uwd. short-term and long-term forecasting are commonly employed with different meanings. some authors, also include a third categorization, i.e., mid-term forecasting. this study adopts the framework proposed by ghiassi et al. [3], for which short-term predictions encompass hourly and daily forecasts, whereas year-based and decade-based forecasts are deemed long-term. monthly-based forecasts, considering up to 24 months, are categorized as mid-term forecasts by some authors. in terms of their applicability, long-term urban water-demand (uwd) forecasts can integrate economic, demographic, and climatic variations. this integration supports the planning and design of water supply infrastructure [11]. on the other hand, short-term forecasts can be used to optimize water treatment and supply management. according to adamowski et al. [1], short-term forecasting can be employed to enhance understanding of factors influencing water consumption, optimize pump and reservoir management, reduce water loss through leakage by controlling pipeline pressure, and indicate the need for emergency water rationing policies. these measures can directly and indirectly contribute to water conservation and operational cost reduction, obtained from reduced energy consumption in pumping operations, for instance. for uwd forecasting, predominant approaches are based on data-driven methodologies [7]. the review presented by niknam et al. [9] identifies neural networks, support vector machines, traditional time series methods (including arima and sarima), regression models, random forests, and dynamic systems as the primary data-driven approaches for this purpose. the authors also point out that there is no consensus in the literature regarding the prevalence of one method over others, given the diverse specific conditions of each case under study. however, according to the same authors, researchers in general agree that the methods employed for forecasting uwd should be adapted to the available data and the intended application. in this context, due to their unique characteristics, tourist cities with high seasonal population influx pose a challenge for uwd forecasting, requiring properly adjusted models to ensure reliable results. guaratuba is a coastal tourist city located in the state of paraná, brazil, that exhibits the aforementioned characteristics. every year, during the summer season, which coincides with the scholar recess period in brazil, the city receives thousands of tourists looking to enjoy the beach. as a result, the city, with approximately 42,000 inhabitants according to the latest census held in 2022, received over a million tourists in a single year, mainly concentrated in late december, january, and february. one of the challenges associated with such a sudden seasonal increase in population is managing its water supply facilities. in this context, the present study aims to evaluate statistical methods for short-term urban water demand (uwd) forecasting in guaratuba. specifically, we compare classical methods such as arima and sarima to predict a one-week (7-day) horizon considering a daily granularity. results are compared with those presented in the work of vieira et al. [12], for which machine learning methods were evaluated on the same data, with a regression oriented approach. 2 related work in this section, relevant literature and studies exploring similar topics are reviewed. these provide a foundation for discussing the methodology adopted in this work and corresponding findings, presented in subsequent sections. in the study conducted by kavya et al. [7], the authors aimed to provide short-term water demand forecasts for smart water management in the city of hubli (hubballi), india. utilizing 10-minute interval flow meter readings collected over two years (2020–2021), they compared nine machine learning and deep learning models. the study employed both univariate models, using only historical water consumption data, and multivariate models, incorporating climatic parameters and calendar inputs. the results indicate that deep learning models outperformed traditional machine learning models, with the long short-term memory (lstm) model achieving the best prediction performance. although arima presented one of the worst performances in this particular study, it is important to note that no information whatsoever is provided by the authors regarding the hyper-parameters employed for this particular method. in a comparative study of predictive models for shortand long-term urban water demand forecasting in milan, italy, hao et al. [5] considered several machine learning models and sarimax to obtain short-term forecasting horizons of 1 and 7 days ahead. additionally, they investigated whether coupling a wavelet data-driven forecasting framework (wddff) could improve predictive capacity. results showed that for the 1-day ahead forecast in the non-wavelet setting, the artificial neural network (ann) model and sarimax performed better, whereas in the wavelet setting, wa-ann and wa-lstm outperformed other models. for the 7-day ahead forecast, wa-lstm significantly outperformed all other models, with sarimax showing the worst result. the authors state that arima models can achieve satisfactory results for one-step-ahead prediction, but they have limited capacity for multi-step prediction tasks. again, no information was found on the hyper-parameters employed by the authors to train the sarimax model in order to support these conclusions. another work in the area of water consumption forecasting, was developed by gil-gamboa et al. [4]. they aimed to predict next quarter demand for each costumer of a company in a specific area of seville, spain. in their work, a deep feed-forward neural network (dffnn) was proposed and compared with other predictions methods, including isotonic regression (ir), polynomial regression (pr), random forest (rf), gradient boost tree (gbt) and long short-term memory (lstm). the results showed that the dffnn produced the smallest errors compared w.r.t the other models, outperforming even lstm, which if commonly employed for short-term water demand forecasting. sahoo et al. [10] explored the application of several deep learning models for multi-step ahead urban water demand forecasting in london, canada. the study focused on water demand for 1-day, 7-day, and 15-day horizons. among the methods under investigation, the hybrid cnn-bilstm model (bidirectional lstm and cnn architecture) demonstrated superior performance, accurately capturing daily water demand patterns and outperforming the other models across all forecast horizons. the effectiveness of the model was attributed to its powerful feature extraction capabilities combined with the bidirectional learning approach of bilstm. the study highlights the cnn-bilstm model as a promising technique for accurate urban water demand forecasting. the results presented by vieira et al. [12] are of particular interest to our work, as they rely on the same dataset we employ here. the main objective of their work was to predict the water demand in guaratuba, a coastal city in the state of paraná, brazil. a forecasting horizon of 7 days, with daily forecast frequency was considered by the authors. they employed four machine learning algorithms: linear regression (lr), k-nearest neighbors (knn), support vector regression (svr), and multilayer perceptron (mlp). no statistical model, such as arima or sarima, was tested for comparative purposes. a validation approach based on sliding and expanding windows was employed, incorporating climate and calendar variables to predict water demand. the mlp model, coupled with expanding windows outperformed all other methods. 3 materials and methods this section presents the materials and methods employed in this study. sections 3.1 and 3.2 describe the dataset, including its structure and source, as well as the analysis carried out. sections 3.3 to 3.6 cover the predictive models under comparison, the validation approach, alongside metrics and model selection. 3.1 dataset the dataset used in this work was obtained from sanepar (companhia de saneamento do paraná), the company responsible for providing water supply and sanitation services to guaratuba. this dataset has already been preprocessed, therefore, issues related to missing data, noise, and outliers have already been addressed. the dataset includes the daily volume of water produced at the water treatment plants (wtps), along with additional variables related to meteorological and calendar data. it is important to make clear that only the historical data from the wtps was analyzed and used as input for the predictive models developed in this work. that is, all models are univariate, taking as input past values for the target feature (volume of water produced by treatment). the data is presented on a daily frequency, covering a period of four years, from 2016 to 2019. the first two years (2016 and 2017) are set apart for training the models, the third year (2018) is allocated for validation (including hyper-parameter tuning), and the final year (2019) is used exclusively for testing. 3.2 exploratory analysis for all analyses, including the exploratory, the last year (2019) is excluded, in order to prevent any contamination or influence in subsequent evaluation steps. the first step undertaken in the exploratory analysis aimed to understand the behavior of the time series itself. an initial examination of daily volume of water produced at the water treatment plants (wtps) yielded insights on how to further proceed. to identify useful patterns for the models, a classical time series decomposition was applied, employing both additive and multiplicative forms. in addition, the autocorrelation function (acf) and partial autocorrelation function (pacf) were employed to identify patterns that were not readily identified through the decomposition process. these functions were also used to determine which lags could be relevant for posterior model optimization. 3.3 statistical models arima and sarima, well-known statistical time series models, were used in this work. these will be briefly discussed next. for further information, see [6]. arima(p, d, q) is one of the most commonly used methods for time series forecasting, aiming primarily to describe the autocorrelation within the data. it combines differencing order (i) with autoregression (ar) and moving average (ma) models. the ar(p) parameter specifies the number of lag observations included in the model, indicating how many previous values influence the current value of the series. differencing order (d) denotes the number of times raw observations are differenced to achieve a stationary time series. similarly to ar(p), ma(q) indicates the number of lagged forecast errors in the model [6]. the sarima model incorporates the seasonal component into the arima model and is defined as sarima(p, d, q)(p, d, q)s. in addition to the original arima hyper-parameters, the sarima model includes the seasonal hyper-parameters p, d, and q, which represent the seasonal autoregressive, differencing, and moving average orders, respectively. the remaining hyper-parameter of the sarima model (s) determines the length of the seasonal cycle [6]. 3.4 model evaluation all models were evaluated employing a backtesting, similar to that employed by [12]. in brief, the evaluation process applies a cross-validation logic without disrupting the temporal order of the data. two distinct methodologies were employed for backtesting the models, namely, sliding window (sw) and expanding window (ew). these are illustrated in fig. 1, considering five iterations of the procedure, for each case. details of each methodology are provided next. in the sw methodology (left of the figure), a model is trained with a fixed-size window (constant during all process) in each iteration of the evaluation procedure. after each training and evaluation is performed, the training window is moved forward by a specified step size. the step size also determines the number of objects that will be dropped from the beginning of the previous training set. that is, the same amount of new data points added to the new training window is also removed, considering the oldest data points available. fig. 1. sliding and expanding window procedures, respectively. the left schematic depicts a sliding window methodology, whereas the right one depicts a expanding window methodology. five evaluation iterations are depicted for both cases, with training data highlighted in blue and test data in red. the figure is adapted from [12] (color figure online). full size image in the ew methodology (right of the figure), the size of the training window increases by each iteration of the procedure. no data from the training window is ever dropped, with new data points being added after each step of evaluation procedure (comprised of training and testing). in brief, the training set dynamically increases in size over time, whereas the test set size remains constant. note that these two methodologies simulate a real application scenario. moreover, it is clear that these procedures might be more or less appropriate depending on how data behaves over time. if there is no clear trend (upwards or downwards), an expanding window would, theoretically, be preferred, given that more training data would be available as time progresses. conversely, if there is a distinguishable trend over time, a sliding window might be preferable, as past data would not reflect the phenomena under study with the passage of time. the backtesting used in the study has three configurations, summarized in table 1. one configuration uses ew with a dynamic window size, while the other two configurations employ sw with fixed window sizes of 365 days (1 year sw1y) and 730 days (2 years sw2y) for training. the step size and forecasting horizon for all three configurations is 7 days, with a forecasting frequency of one day, that is, for each iteration seven daily forecasts are made. table 1. backtesting configurations.full size table 3.5 model selection for model selection a grid search was conducted on a subset of predefined hyper-parameters. these values were identified through the exploratory analysis of the dataset, outlined in sects. 4.1 and 4.2. table 2 provides the search space for the models. root mean squared error (rmse) employed as the evaluation metric. table 2. hyperparameter optimizationfull size table the hyper-parameter optimization was carried out on the validation set and only the best values found are utilized in the final models. we performed hypre-parameter optimization considering each one of the backtesting configurations (ew, sw1y and sw2y). only the “best” models are evaluated on the test sets. 3.6 evaluation metrics the best models, as identified during hyper-parameter optimization, are applied to the test set and their results are evaluated using three different metrics: root mean squared error (rmse), mean absolute error (mae), and mean absolute percentage error (mape). these are given by eqs. (1), (2) and, (3), respectively. for all the aforementioned equations, \(y_i\) represents the actual value (target) and \(\hat{y}_i\) accounts for the forecast value for the \(i^{th}\) value of the time-series, considering that it has a total of n values under evaluation. $$\begin{aligned} rmse &= \sqrt{\frac{1}{n} \sum _{i=1}^{n} (y_i \hat{y}_i)^2}\end{aligned}$$ (1) $$\begin{aligned} mae &= \frac{1}{n} \sum _{i=1}^{n} |y_i \hat{y}_i|\end{aligned}$$ (2) $$\begin{aligned} mape &= \frac{1}{n} \sum _{i=1}^{n} \left| \frac{y_i \hat{y}_i}{y_i} \right| \times 100 \end{aligned}$$ (3) we use different metrics given that none of them provide a perfect and unbiased evaluation [8, 13]. the rmse metric, for instance is often regarded as sensitive to outliers [2]. all metrics are minimization measures, which means that the lower the value of the metric, the better is the model. 4 results this section presents the results of the proposed evaluation. sections 4.1 and 4.2 detail the dataset analysis. sections 4.3 and 4.4 cover the selection of the “best” model for each backtesting window, considering the grid-search procedure. finally, in sect. 4.5, a comparison between the statistical models proposed in this study with those from machine learning from vieira et al. [12] is carried out. fig. 2. multiplicative decomposition. time series, trend, seasonality, and noise. full size image 4.1 time series decomposition the decomposition of the water demand time series from guaratuba was quite similar for both additive and multiplicative methods. however, the multiplicative decomposition yielded slightly better results and is utilized in this analysis. figure 2 shows the plot of the time series data from 2016 to 2018, highlighting trend, seasonality, and noise, from top to bottom in the figure. the trend component shows some growth over across subsequent years, but the most prominent feature in the plots is the seasonal component. using the median seasonality obtained from the decomposition, it is apparent that peak demand occurs on the first day of the year. therefore, to address the non-stationarity observed in the trend and seasonal components, a first-order differencing (i(d) = 1) is a reasonable choice and should be among the evaluated hyper-parameters tested in both the arima and sarima models. the irregular component (noise) can help understand how external factors influence the behavior of the series. as depicted in fig. 2, shortly after the beginning of the years 2017 and 2018, there is a decline in the volume of produced water, followed by sudden growth before stabilization, indicating an anomalous event. this, however, corresponds to the carnival season, which usually lasts 4 to 7 days in brazil and has a significant impact on water demand (tourist influx). 4.2 acf and pacf both acf and pacf functions are visual tools used to assess the linear dependence of a time series on its past values (lags). figure 3 displays the acf and pacf plots, where each lag is represented by a vertical line with a marker. values outside the shaded light blue area are statistically significant and indicate potential correlation between a data point and its lagged values fig. 3. plots for acf and pacf for the time series considering the training set. full size image the acf indicates a significant correlation between the current value of the time series and its values up to lag 38, a term that might be appropriate for the ma(q) in arima and sarima models. furthermore, the presence of a 7-day seasonality is evident in the data. this seasonality can be incorporated into the sarima model by specifying a seasonal period of 7 in the s hyperparameter. the pacf isolates the influence of individual past values on the current one, excluding the effects of intermediate lags. as shown in fig. 3, it suggests significant partial correlations at lags 1 to 3 (correlation of 3), lags 7 and 8 or lags 10 and 11 (correlation of 2), and lags 20 to 27 (correlation of 8). note that the hyper-parameter values defined for the arima and sarima methods’ search space were previously presented in table 2. 4.3 model selection after completing the grid search w.r.t. hyper-parameters, the best models, coupled with the corresponding window type were identified. here, we would like to emphasize the importance of proper hyper-parameter selection for both arima and sarima, in contrast with previous studies, highlighted in sect. 2. for that, we report not only the best hyper-parameter combination obtained from the validation set, but also the worst and overall average across all combinations explored within the search procedure. these results are provided in table 3, considering the average of the three evaluation metrics and corresponding standard deviation (\(\sigma \)). all values are w.r.t. the validation set. table 3. results of the evaluation metrics for all the methods on the validation set.full size table the arima (8,0,38) model in the context of a 2 year sliding window (sw2y) exhibited the best overall results, achieving an rmse of 1425.35 with a standard deviation of 918.88. the worst results were obtained with sarima models (2,1,3),(2,1,4,7) and (3,1,3),(3,1,1,7), using expanding window (ew) and 1 year sliding window (sw1y), respectively. the highest overall rmse was 2053.38, a value 44% higher/worse than that of the best model. the selection of hyper-parameters significantly impacts the performance of time series models like arima and sarima. choosing sub-optimal hyper-parameters can lead to poor predictive performance, which could bias comparisons with other methods, such as those from the machine learning toolbox. therefore, we argue that it is crucial to conduct a comprehensive analysis and tuning of parameters to ensure accurate and reliable results. it is important to note that, at this point, the test data has not been seen by the models yet. the validation data was used solely to select the best hyper-parameters, which will be incorporated into the arima and sarima models and evaluated on the test data, as presented in the next section. 4.4 model evaluation all models with their optimal hyper-parameters were evaluated on the test data w.r.t. rmse, mae, and mape. table 4 presents the metrics for each one of the models on the test set, with their standard deviations (\(\sigma \)). additionally, the best results obtained for each window evaluation methodology (ew, sw1y, and sw2y) from the study conducted by vieira et al. [12] are also included. these will be be discussed in more detail in sect. 4.5. it is important to emphasize that we fully reproduced the experiments from [12], attaining the same results from the authors. therefore, all methods were evaluated equally. table 4. results of the evaluation metrics for the best methods regarding the test set.full size table the best evaluation metrics for each model tested in this study (arima and sarima) are highlighted in bold in table 4. analysing only the statistical models, the arima model with a 2 year sliding window configuration (sw2y) obtained the best overall performance, with an mae of 1, 419.40. this result indicates that the daily water demand prediction error is, on average, 1, 419.40m\(^3\), which represents the difference between the actual and predicted values (to place this result in perspective, in the test set, values of produced water range from  5, 000 to 26, 000 cubic meters). for the sarima method, the best results were obtained using the ew configuration, achieving an rmse of 1, 870.03. a more detailed analysis involves observing the performance of the best arima and sarima models across different months, as depicted in fig. 4. the box-plot indicates that for both models, the best performance was observed in september, while the worst results were attained in december. indeed, december is the month with greater variability in the data (justified by tourism from the holidays), which poses difficulties to the models. fig. 4. monthly absolute error for the best arima/sarima models w.r.t. the test. full size image fig. 5. forecasts and target variable on test set. full size image an overview of the forecasts made by the arima-sw2y and sarima-ew (the best models from table 4) and the actual observed time series (target) is provided by fig. 5. relevant months in terms of water demand forecasting are zoomed in to highlight the performance of both models in fig. 6. in addition to the best and worst months in terms of water demand forecasts previously discussed, january and march are emphasized due to their peculiar behavior compared to the other months of the year, as discussed next. january and december are vacation months for most people in brazil, which could account for the distinct trends observed in the graphs, posing challenges for effective predictions by the models. in early march, the carnival holiday can extend up to a week, leading to increased travel to the city and higher water demand, possibly explaining the spike in the graph. conversely, september is not a holiday season, resulting in fewer people traveling to the coast and less variation in water demand. the “stable” data observed during this month may contribute to the better performance of both models. fig. 6. forecasts and target variable on test set for selected months. full size image 4.5 comparison with related study as previously mentioned, the work from vieira et al. [12] serves as a baseline for this study. both studies had the same primary objective and employed identical backtesting configurations, with a 7-day prediction horizon and daily frequency for forecasts. therefore, both studies can be fairly compared between each other. results to support such a comparison were already presented in table 4, where the models from the present study are placed in perspective w.r.t. the best model from the work of vieira et al., for each backtesting configuration. in the ew configuration, mlp outperformed both time series methods. for the sw1y training window size, knn exhibited superior performance in previous research but was surpassed by the arima model in this study. similarly, in the sw2y training window size configuration, the arima model also outperformed the machine learning methods under evaluation by vieira and colleagues. while previous research indicated superior results with machine learning methods using the ew approach, the current study demonstrates that the statistical models performed better with the sw method. one plausible explanation for this is that as time progresses, older data lose significance and no longer affects current forecasts as much. this might arise due to population increase and/or greater tourism influx in the recent years, in comparison to previous ones. in summary, both the mlp with the ew and the arima with the sw produced comparable results. this study underscores the importance of effectively exploring the full capabilities of statistical methods in time series problems, conducting thorough data analysis, and optimizing their hyperparameters. in this work, relying solely on historical data, the statistical model (which is univariate) even achieved better rmse and mape metrics than the machine learning models, which also incorporated calendar and meteorological variables. 5 conclusions this research presented a case study conducted in guaratuba pr brazil, aimed at forecasting its daily water demand over a 7-day horizon using statistical methods. in a previous investigation [12], the authors pursued the same objective with a regression oriented approach employing machine learning models. both approaches employed backtesting for model validation and assessment. a comparison between the results obtained from the two studies revealed similar outcomes, with machine learning methods performing better when coupled with an expanding window approach. conversely, statistical models performed better with sliding windows. overall, the mlp models with expanding window (mlp-ew) and arima with a 2 year sliding window training window size (arima-sw2y) achieved the best performances, with arima-sw2y showing an advantage in rmse and mape metrics. this study demonstrated the utility of statistical models, specifically arima and sarima, in forecasting urban water demand, even over a 7-day horizon. in general, there is a widely held view among researchers that statistical models are particularly suitable for short-term forecasts in urban water demand (uwd), typically one day ahead. in our case, we extended this to a 7-day horizon and achieved results comparable to those obtained using machine learning. based on our study, we can conclude that traditional arima and sarima models continue to serve as a robust baseline for comparison with more complex machine learning models. however, achieving an appropriate model requires comprehensive data analysis, proper tuning of hyper-parameters, and a fair backtesting approach. in our work, we identified that many studies do not conduct such rigorous analyses when using statistical methods as baselines, potentially resulting in unfair comparisons and sub-optimal model performance. it is important to note that this study addresses a specific problem, and the findings may not be applicable or extendable to other contexts without further evaluations. further work is in progress, in order to evaluate long short-term memory (lstm) models in this context. references adamowski, j., et al.: comparison of multiple linear and nonlinear regression, autoregressive integrated moving average, artificial neural network, and wavelet artificial neural network methods for urban water demand forecasting in montreal, canada. water resour. res. 48(1) (2012) google scholar  chai, t., draxler, r.r.: root mean square error (rmse) or mean absolute error (mae)? arguments against avoiding rmse in the literature. geosci. model dev. 7(3), 1247–1250 (2014) article  google scholar  ghiassi, m., et al.: urban water demand forecasting with a dynamic artificial neural network model. j. water resour. plann. manag. 134(2), 138–146. american society of civil engineers (2008) google scholar  gil-gamboa, a., paneque, p., trull, o., troncoso, a.: medium-term water consumption forecasting based on deep neural networks. expert syst. appl. 247, 123234 (2024) article  google scholar  hao, w., cominola, a., castelletti, a.: comparing predictive machine learning models for shortand long-term urban water demand forecasting in milan, italy. ifac-papersonline 55(33), 92–98. 2nd ifac workshop on control methods for water resource systems cmwrs 2022 (2022) google scholar  hyndman, r., athanasopoulos, g.: forecasting: principles and practice, 3rd edn. otexts, australia (2021) math  google scholar  kavya, m., et al.: short term water demand forecast modelling using artificial intelligence for smart water management. sustain. cities soc. 95 (2023) google scholar  kim, s., kim, h.: a new metric of absolute percentage error for intermittent demand forecasts. int. j. forecast. 32(3), 669–679 (2016) article  mathscinet  math  google scholar  niknam, a., et al.: a critical review of short-term water demand forecasting tools—what method should i use? sustainability 14(9), 5412. multidisciplinary digital publishing institute google scholar  sahoo, b., panigrahi, b., nanda, t., tiwari, m.k., sankalp, s.: multi-step ahead urban water demand forecasting using deep learning models. sn comput. sci. 4 (2023) google scholar  tiwari, m.k., adamowski, j.: urban water demand forecasting and uncertainty assessment using ensemble wavelet-bootstrap-neural network models. water resour. res. 49(10), 6486–6507 (2013) article  math  google scholar  vieira, j., et al.: machine learning for water demand forecasting: case study in a brazilian coastal city. water pract. technol. 19(5), 1586–1602 (2024) google scholar  willmott, c.j., matsuura, k.: advantages of the mean absolute error (mae) over the root mean square error (rmse) in assessing average model performance. climate res. 30(1), 79–82 (2005) article  math  google scholar  download references acknowledgments the authors would like to thank capes for financial support. author information authors and affiliations joinville techonological center (ctj) federal university of santa catarina (ufsc), joinville, brazil antoniel kleber stefaniak, pablo andretta jaskowiak & lucas weihmann authors antoniel kleber stefaniakview author publications search author on:pubmed google scholar pablo andretta jaskowiakview author publications search author on:pubmed google scholar lucas weihmannview author publications search author on:pubmed google scholar corresponding author correspondence to pablo andretta jaskowiak . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper stefaniak, a.k., jaskowiak, p.a., weihmann, l. (2025). a case study on water demand forecasting in a coastal tourist city. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15414. springer, cham. https://doi.org/10.1007/978-3-031-79035-5_1 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79035-5_1 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79034-8 online isbn: 978-3-031-79035-5 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords urban water-demand forecasting arima sarima publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature unsupervised statistical keyword extraction pipeline: is llm all you need? | springer nature link (formerly springerlink) skip to main content advertisement account menu find a journal publish with us track your research search cart home intelligent systems conference paper unsupervised statistical keyword extraction pipeline: is llm all you need? conference paper first online: 30 january 2025 pp 460–475 cite this conference paper download book pdf download book epub intelligent systems (bracis 2024) unsupervised statistical keyword extraction pipeline: is llm all you need? download book pdf download book epub fernando rezende zagatti9, daniel lucrédio9 & helena de medeiros caseli9  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 425 accesses abstract keyword extraction is an important step for text interpretation, serving to identify and highlight the most significant words or phrases within a text. this step is essential for various applications such as summarization, indexing, and information retrieval. this paper presents a custom-built keyword extraction pipeline named uske (unsupervised statistical keyword extraction) and compares its performance to large language models (llms). uske is able to deliver fast and simple results based in statistical methods even when dealing with large datasets. our evaluation demonstrates that although llms can achieve good results in single sentences with minimal context, they require a lot of post-processing and may output inconsistent answers, while uske excels in efficiency and scalability. download conference paper pdf similar content being viewed by others probing llms on optimization problems: can they recall and interpret problem features? chapter © 2025 unsupervised text style transfer via llms and mask-filling with multi-way interactions chapter © 2026 extractive-abstractive: a two-stage model for long text summarization chapter © 2022 1 introduction in the era of big data, with vast amounts of information being generated continuously, the ability to quickly and accurately identify the main topics, keywords, and terms from a large volume of texts has become highly desirable [5, 6]. this capability is essential for efficiently managing and leveraging information in various domains, from academia to industry, where rapid insights can drive decision-making and innovation. in this way, keyword extraction plays a crucial role in various applications for natural language processing (nlp), such as information retrieval [22] and text summarization [9]. traditionally, this task has been approached using classical methods, mostly unsupervised and statistical ones, which often involve extensive preprocessing and the use of domain-specific features. however, with the advent of large language models (llms) such as bidirectional encoder representations for transformers (bert) [4] and generative pre-trained transformer (gpt) [16], new opportunities have emerged to increase the accuracy and efficiency of keyword extraction. these new text generation methods leverage deep learning techniques to capture contextual and semantic nuances, offering greater accuracy over traditional methods, but their computational complexity and dependence on large training datasets present practical challenges [10]. in this way, this study not only compares the performance of these advanced models with traditional statistical methods but also introduces an unsupervised pipeline to perform keyword extraction on textual data. our open-source proposed system, unsupervised statistical keyword extraction (uske)footnote 1, employs a series of nlp pipeline steps and statistical techniques to identify and extract relevant keywords from texts. by integrating text preprocessing methods with statistical techniques, we create a versatile and efficient keyword extraction that does not demand any specialized knowledge in llm or ai to be used. in this way, this paper aims to compare classical methods against modern llm-based approaches to assess their respective strengths and weaknesses. while traditional methods often struggle with large-scale data and multilingual contexts, llms can be resource-intensive and less feasible for real-time applications. thus, through this comparative analysis, we seek to provide insights into whether llms are truly indispensable for keyword extraction or if traditional methods still hold significant value. our work seeks to offer a direct comparison between classical keyword extraction methods and llm-based techniques, focusing on their respective strengths and weaknesses in a more generalizable context. unlike methods like keybert, which are resource-intensive, uske provides a lightweight, open-source alternative that is both versatile and efficient by integrating statistical techniques within a streamlined nlp pipeline, demonstrating that traditional methods can still yield competitive results, particularly when used in conjunction with domain expertise. this study demonstrates the effectiveness of our proposed approach in datasets of reviews and titles of the portuguese language, confirming that traditional methods can still obtain competitive results for this specific task, and can obtain satisfactory results when used alongside experts to make annotations to serve as input for the algorithm. this work is organized as follows. section 2 provides a comprehensive review of related works, highlighting the current state of the art in unsupervised keyword extraction. in sect. 3 we present our proposed method, detailing the algorithms and techniques used. section 4 describes the experimental setup, including the datasets and prompt tuning process. in sect. 5 we discuss the results and provide an analysis of the performance of our method when compared to llm. finally, sect. 6 concludes the paper with a summary of our findings and outlines potential directions for future research. 2 related works in recent years, keyword extraction has garnered significant attention within the field of nlp, leading to the development of various methodologies that aim to improve the accuracy and efficiency of this task [7, 12, 21]. these methods, while effective, often require extensive preprocessing and the use of external resources, which can limit their applicability across diverse languages and domains. keybert [7] is a keyword extraction method that leverages transformer-based models to identify the most relevant keywords within a text. in this way, the bert-embeddings are applied into the document and the word embeddings are extracted in n-grams; then, cosine similarity is used to find the most relevant words (or keywords) in the document. despite its advantages, keybert has some limitations: (i) its dependence on large datasets, which may not perform well on texts from certain domains or on languages less represented in the training data, and (ii) the method can be computationally expensive, as it requires generating embeddings for both the document and numerous candidate keywords, making it less feasible for real-time or large-scale applications. yake (yet another keyword extractor) [2], unlike traditional machine learning based models, does not require a training phase, making it a versatile and scalable solution that operates on single documents, enabling quick adaptation to various scenarios without the need for linguistic tools or external databases. this method identifies relevant keywords based on statistical features, like by analyzing the frequency and co-occurrence of word sequences within a document, offering a language and domain-independent solution. the effectiveness of yake has been demonstrated through extensive testing across multiple datasets and languages [1, 18]. however, it is important to highlight that the larger the analyzed dataset and the greater the number of keywords to be extracted, the time required to obtain the keywords can increase considerably. gupta et al. (2024) [8] made an integration of bert [4] and yake to analyze sustainability reports from indian it companies. the study uses bert tokenization and embedding to capture the contextual and semantic nuances of the text, understanding the importance of each word within the broader context of the document, while yake is employed to extract keywords from the text. the study acknowledges some limitations and areas for future improvement, specifying that the study only uses sustainability reports from it organizations, reinforcing that to improve the generalizability, reports from additional sectors should be included. furthermore, the study analyzes point-in-time reports, while conducting a time series study could provide insights into how sustainability practices into organizations evolve over time. aiming to perform a comparative analysis, nadim et al. (2023) [13] evaluates the performance of statistical-based, graph-based, and embedding-based keyword extraction tools, highlighting their strengths and weaknesses in terms of both performance and execution time. in this way, keybert achieved better performance of f1-score across three of the four datasets, though it has higher execution times for longer texts, while rapid automatic keyword extraction (rake) [21] and yake are highlighted for their fast execution times. among all the works, several keyword extraction tools and some benchmarks between the tools were seen. however, no studies were found that aims to compare classic pipelines and new approaches with llms. therefore, our work seeks to perform a comparison between classical and llm approaches, at the same time as providing an open-source tool for extracting keywords in a simple and quick way using statistical methods. 3 unsupervised statistical keyword extraction we propose a system, referred to as uske, that employs an unsupervised pipeline with statistical techniques to identify and extract relevant keywords in the text. this section describes how the proposed system was developed, the techniques used and the relationship between its steps, as illustrated by fig. 1. fig. 1. pipeline for uske system full size image as can be seen, there are 3 main stages for the system, namely: (i) pre-processing pipeline, (ii) candidate keyword identification, and (iii) keyword extraction. 3.1 pre-processing pipeline the preprocessing stage aims to prepare and clean text for the use of nlp techniques and extraction of the keywords. at this stage, the following techniques are applied: (i) document lowercase transformation, (ii) removal of all special characters (e.g. @, #, $) and standardization of word accentuation (e.g. á, â, ã -> a), (iii) an optional stopword removal, and (iv) simple tokenization by split. 3.2 candidate keyword identification this stage performs the identification of keywords relevant to the corpus employing statistical methods, namely: yake (yet another keyword extractor), tf-idf (term frequency-inverse document frequency), pmi (pointwise mutual information), dice coefficient, mle (maximum likelihood estimation), and n-grams count. each approach will be detailed below: yakefootnote 2: this is a method that extracts keywords or key phrases from text documents by preprocessing and identifying candidates based on term frequency (tf) within the document, inverse document frequency (idf) across a larger corpus, word co-occurrence patterns, position of the candidate within the document, and the length of the candidate phrase. the candidates are scored and ranked using a weighted combination of these features, with the highest-scoring candidates being selected as the most relevant keywords or key phrases. tf-idf: identifies keywords that are both frequently occurring within a specific document and relatively rare across the entire corpus. in this way, this approach assigns a numerical weight to each word and performs a ranking based on these scores. $$\begin{aligned} \texttt {tf-idf(t, d)} = \texttt {tf(t, d)} \times \texttt {idf(t)} \end{aligned}$$ (1) $$\begin{aligned} \texttt {idf(t)} = log \left[ \frac{(1 + \texttt {n})}{(1 + \texttt {df(t)})} \right] + 1 \end{aligned}$$ (2) where: tf-idf(t, d) : is the “term frequency-inverse document frequency” of word t in document d. idf(t) : is the “inverse document frequency” which measures how common a word (t) is in all documents. tf(t, d) : computes “term frequency” of the word t in a document d as the number of times t appears in d. n : is the total number of documents. df(t) : is the number of documents in which the word t appears. pmi: this is a statistical measure used to determine the association or co-occurrence between two events (or words) within a text corpus. a high pmi score indicates a strong association, implying that the occurrence of one word makes the presence of the other more likely, while a low or negative pmi suggests independence or repulsion; its calculation can be visualized through eq. 3, eq. 4, and eq. 5, $$\begin{aligned} pmi(w_1 \ldots w_n) = \log _2\left( \frac{p(w_1 \ldots w_n)}{\prod _{i=1}^{n}p(w_i)}\right) \end{aligned}$$ (3) $$\begin{aligned} p(w_{i}) = \frac{count(w_{i})}{totalwordcount} \end{aligned}$$ (4) $$\begin{aligned} p(w_1 \ldots w_n) = \frac{count(w_1 \ldots w_n)}{totalwordcount} \end{aligned}$$ (5) where: \(p(w_i)\) is the individual probabilities of each word; \(p(w_1 \ldots w_n)\) is the joint probability of the n-gram; n is the number of words of the keyword. \(count(w_{i})\) is the number of times that word appears in the text; \(count(w_1 \ldots w_n)\) is the number of times that n-gram appears in the text; totalwordcount is the total number of words in the text; dice: this is a measure used to quantify the degree of overlap or similarity between two sets. the score ranges from 0 to 1, with higher values indicating greater similarity. $$\begin{aligned} dice(w_1 \ldots w_n) = \frac{n * count(w_1 \ldots w_n)}{\sum _{i=1}^{n}count(w_i)} \end{aligned}$$ (6) mle: this is a technique to estimate the probability distribution of words (or phrases) in a corpus, assuming that the probability of a word occurring in a specific context is proportional to the number of times that word appears within the training corpus. $$\begin{aligned} mle(w_1 \ldots w_n)= {\left\{ \begin{array}{ll} \frac{count(w_1)}{totalwordcount}, & \text {if } n = 1\\ \frac{count(w_1 \ldots w_n)}{count(w_1 \ldots w_{n-1})}, & \text {otherwise} \end{array}\right. } \end{aligned}$$ (7) n-grams count: these are contiguous sequences of n items, which can be characters, words, or even more extended units like phrases. in this context, it is checked how many times each n-gram appears in the text. 3.3 keyword extractor this stage extracts keywords from the text based on a list of identified candidate keywords from the previous stage (or a list of keywords provided by the user). to illustrate the process, we will demonstrate the outputs of each function using the example of searching [‘electric fryer’, ‘deep fryer’] within the text “eggkitpo electric deep fryer”. this stage involves the following functions: filter tokens: in this function, the text is separated into tokens and each token is checked if it is present in the list of identified keywords. this function returns a list containing the words that passed the filter. after filter: [‘electric’, ‘deep’, ‘fryer’] generate combinations: in this function, from the list outputted by the previous step, a new sorted list (by order of appearance in the input text) is generated with all possible combinations of words. this step is important when keywords candidates have 2 or more tokens. after combinations, considering up to 2 tokens: [‘electric’, ‘deep’, ‘fryer’, ‘electric deep’, ‘electric fryer’, ‘deep fryer’] filter candidates: this function selects only those word combinations that are present in the list of identified keywords. after new filter: [‘electric fryer’, ‘deep fryer’] this separation into tokens and making combinations instead of simply searching for the sequence of words is very useful for identifying keywords that are not directly found in the text. table 1 exemplifies this problem. table 1. example of token combination efficiency versus direct keyword searchfull size table 4 experimental setup the experiments were performed with a processor intel core i7-10700k, a high-performance graphics processing unit (gpu) nvidia geforce rtx 4070 ti, and 32 gb of random access memory (ram). we compared the performance of uske, keybert and 5 llms that were available on huggingfacefootnote 3 to evaluate the performance of some language models, which can be easily found, with some classical pipelines. the models were selected based on the trending models of huggingface, selecting textual generation models that use english and/or portuguese and that allow their use with the langchain api. among the selected models, except for phi-bode, all were trained with most of the data being in english and a few in portuguese. the selected models are described below: vlt5-base-keywordsfootnote 4: a keyword generation model based on the encoder-decoder architecture presented by google. it was trained on a corpus composed of scientific articles, to predict a given set of keyword phrases that describe the article content based only on the abstract [15]. among the selected models, this is the only one specifically constructed for keyword extraction. flan-t5-xxlfootnote 5: a google model trained using 540b-parameters, performing fine-tuning with the addition of more than 1000 new tasks and covering more languages than its predecessor, t5 [3, 17]. it can generate coherent and contextually relevant text based on a given prompt, making it useful for creative writing, chatbots and content creation. bloomfootnote 6: a model trained to continue text from a prompt, which is capable of producing text in 46 languages and 13 programming languages. this model seeks to assist with text and code generation, and characteristics exploration. blenderbot-400m-distillfootnote 7: a lightweight and distilled version of facebook’s blenderbot 2.7b, designed for open-domain conversational ai. it is based on a 400 million parameter transformer model that has been fine-tuned on diverse conversational datasets to improve its ability to engage in multi-turn dialogue [20]. phi-bodefootnote 8: a language model fine-tuned from microsoft’s phi-2b base model, refined using the translated alpaca dataset for fine-tuning, aimed at adapting the model to the nuances of the portuguese language [14]. among the selected models, this is the only one specifically tuned for portuguese. 4.1 datasets two datasets composed of reviews of products or services were selected to perform the tests, namely: (i) b2w-reviews01 [19] and (ii) aspect-based sentiment analysis in portuguese (absapt) 2022. the datasets are described below: b2w-reviews01: this dataset constitutes an open corpus of product reviews, comprising more than 130 thousand reviews from e-commerce customers, collected from the americanas.com website between january and may 2018. this corpus provides product title texts in portuguese; however, there are no specialized annotations, which can make certain tasks difficult. absapt 2022: this dataset contains a corpus of reviews about hosting service companies written in portuguese. the specialists annotated the aspects and their respective polarities, totalling more than 800 annotated reviews. the aspects were used as the target keywords in the experiments described in this paper. it is important to mention that due to the challenges associated with making inferences from llms on extensive datasets, we reduced the dataset sizes for our experiments: for the b2w-reviews01, only the first 100 instances were used; while for the absapt 2022, we used the test dataset which comprises 184 instances. 4.2 prompt tuning prompt tuning only updates the input values of the prompts while the llm parameters remain unchanged. this tuning method allows one to guide the models to produce more relevant and accurate outputs without necessarily having to retrain them [11]. in this study, we experimented with a few prompt structures (tables 2, 3 and 4) to evaluate their effectiveness across different llms. below, we detail the tuning process and provide examples of the tested prompts along with their results. note that all prompts have {text}, which is a variable where the text from which we want to extract the keywords will be prompted. for the example results, we will use the following {text}: copo acrílico com canudo 500ml rocie. table 2. prompt #1full size table table 3. prompt #2full size table from prompt #1 we see that vlt5-base-keywords and flan-t5-xxl were able to extract words in a simpler way, while bloom had some difficulty and blenderbot seems not to have understood our request. furthermore, it is possible to notice that all models generated some words that did not exist in the input text, such as “acríli”, “cie”, and “n1056”. after some testing, it was found that adding “question:” and “answer:” at the beginning and end of the prompt helped some models generate more concise answers (table 3). bloom in this case only replicated the input text, but did not correctly answer our request to generate the keywords found in the text. table 4. prompt #3full size table finally, as bloom is an llm that continues texts from the prompt, “the list is” was added at the “answer” part (table 4), which helped to obtain more assertive answers and did not affect the answers of the other models. thus, prompt #3 was chosen for the experiments. for experiments containing custom lists, prompt #3 was used changing the “based in the following phrase, return a list of the most relevant words:” to “based in this list of words {list}, return only the list of words that are in the following sentence:”. as the phi-bode model was trained with texts in portuguese, versions of the same prompts were used, but translated to portuguese. 4.3 llms post-processing the outputs generated by the llms did not consistently adhere to a standard format in the reference keyword lists used in our experiments, requiring additional post-processing steps. llms output often contained special characters, word duplication, and lacked consistent token separation, which required thorough cleaning before any automatic evaluation could be performed. table 5 brings an example of the result obtained when searching for keywords for the input text {text}. table 5. example of llms outputs for the input text {text}full size table since the outputs did not follow a well-known pattern, we removed any special characters from the outputs to prevent them from interfering with evaluation metric process. for instance, outputs like “: localização, café-da-manhã, diárias” were cleaned to remove “:”, “-”, and accents, resulting in “localizacao cafe da manha diarias”. this step was performed to avoid discrepancies between the evaluations, where the word “localização” could be considered different from “localizacao”. after cleaning, we separated the output into individual tokens. this step is important for transforming textual data (strings) into lists with the words contained in the texts, allowing word-by-word comparisons with the experts’ annotations. finally, we removed all duplicate tokens from the lists to ensure that each keyword was unique. for example, if a list was “[“localizacao”, “hotel”, “hotel”, “cafe”]”, the duplicate “hotel” was removed, resulting in “[“locali-zacao”, “hotel”, “cafe”]”. this deduplication process was necessary to accurately reflect the diversity of keywords generated by the llms. these post-processing steps were critical in preparing the llm outputs for fair and accurate comparison with the annotated data. they ensured that the evaluation metrics were based on clean and standardized outputs, thus providing a more reliable assessment of the llms’ performance for this specific task. 4.4 evaluation metrics the assessment of the outputs was made based on the metrics commonly used in the literature: precision, recall, and f1-score. precision is the proportion of instances classified as positive that are actually positive. recall is the proportion of actual positive instances that are correctly classified as positive. f1-score is the harmonic mean of precision and recall, providing a balanced measure between the two metrics. our experiments also considered two similarity measures: cosine similarity and jaccard similarity . similarity is a quantitative measure used to determine the degree of similarity between objects, and is often used to compare two distinct vectors. these metrics are fundamental in information retrieval, nlp, and machine learning, where identifying patterns and benchmarking are essential. the cosine similarity calculates the cosine of the angle between two vectors. the closer the cosine value is to 1, the greater the similarity between the entities jaccard similarity is defined as the ratio between the size of the intersection of the sets and the size of their union, ranging from 0 to 1, where 1 indicates that the sets are identical and 0 indicates that there are no elements in common. 5 results and analysis we applied the 5 previously presented llm models to the 2 datasets and compared them with our tool – for uske, the outputs of each technique were merged to comprehensively evaluate the tool. the results were evaluated differently, since the absapt 2022 dataset had a prior annotation, enabling direct analysis, while b2w-reviews01 did not. 5.1 absapt 2022 results this dataset has a prior annotation of aspects for the aspect-based sentiment analysis (absa) task and these aspects were considered our reference keywords. so, two specific tests were applied: (i) extraction of keywords without the list of annotated words and (ii) direct extraction of keywords from the annotations. table 6 presents the results from the first case . in this specific dataset, since annotated data were available, jaccard and cosine similarity metrics were directly applied to the annotated data to evaluate the similarity between the model outputs and the annotated data. table 6. absapt 2022 without custom listfull size table the results demonstrate that uske obtained a precision close to the vlt5-base and flan-t5 models and was the third-best method in recall and f1-score. on the other hand, blenderbot did not get any significant results, as it was trained with a conversational objective and could not generate direct answers for this specific task, and although keybert achieves considerable similarity values, it does not have competitive results in terms of precision and recall. furthermore, phi-bode and bloom obtained significantly better results in precision, recall, and f1-score, but the similarity values were low. this happened because the llms generated more texts than expected, reducing similarity, but since most of the time the expected outputs were between the generated text, this led to increased precision and recall. in contrast, table 7 shows the test results with the list of words we wanted to extract from the text, where we evaluate the results like the previous table. table 7. absapt 2022 with custom listfull size table as our pipeline was developed to extract keywords directly from the text, the experiments using a predefined list lead to high values in all the metrics evaluated. llm models were unable to consistently extract the required words. although only the vlt5-base and flan-t5 models managed to obtain a slightly higher f1-score than the previous test, all models (except for blenderbot and phi-bode) obtained superior results in recall. however, it is important to mention that the precision values decreased in all cases, and the execution time also increased. finally, keybert was developed only to find keywords in an unsupervised way, and it is not able to deal with a predefined list. furthermore, processing time became longer in all llms (except for blenderbot), since the prompts were larger due to the lists that were sent together with the texts. 5.2 b2w-reviews01 results although this dataset does not have prior annotations, it is possible to evaluate whether the results are meaningful using similarity tests. as the task is to search for explicit words in the text, the models’ responses need to have some degree of similarity with the input text. in this way, if the similarity is equal to or close to 0, it is possible to say that the language models were not able to answer the request. the results can be seen in table 8. table 8. b2w-reviews01 resultsfull size table phi-bode obtained the highest similarity values, demonstrating that its outputs were most often related to the input text. uske also achieved respectable scores with a jaccard similarity of 0.38 and a cosine similarity of 0.55, staying at the average of llms. keybert achieved similarity values very close to uske. in terms of processing time, uske outperformed the others, completing the task in just 0.31 s, which is significantly faster than the llms. keybert also had a fast processing time of 1.81 s. however, phi-bode had one of the longest processing times, reaching 1390.15 s. 5.3 critical analysis the analysis reveals critical insights regarding the performance of various models in keyword extraction tasks, particularly emphasizing the effectiveness of models trained in portuguese, the potential for prompt tuning, and the tendency to generate answers that do not adhere to the request and input text. phi-bode, trained specifically in portuguese, demonstrated more satisfactory results in methods without pre-defined lists. this model achieved the highest similarity values, highlighting the importance of training models in the specific language of the dataset to enhance performance in language-specific tasks. furthermore, the possibility of improving prompts can lead to better answers – prompt tuning must be carried out carefully, as even small variations and adjustments can lead to huge differences in performance. experiments with different prompt structures can significantly impact the effectiveness of the llms. lastly, we reinforce that some models, such as blenderbot, are not ideal for keyword extraction activities due to their optimization for conversational tasks. its performance, with low similarity values and long processing times, highlights this limitation. preventing models from generating irrelevant or incorrect information remains a challenge, emphasizing the need for careful model selection and optimization for targeted applications. in this way, we demonstrate that the use of stricter pipelines is still extremely useful for developers, and can save a lot of time choosing models, prompt tuning, and post-processing. 6 conclusions and future work our study reveals that both our proposed pipeline and llms have distinct advantages and disadvantages for extracting relevant keywords in textual data. uske demonstrated significant advantages in processing speed and simplicity, particularly when handling large datasets with minimal computational resources. furthermore, as it is a traditional and more controlled pipeline, the outputs will always be related to the input and guaranteed to follow the desired format. traditional keyword extraction pipelines can be efficient but the lack of a broader context (multiple sentences) and poorly optimized preprocessing can lead to less accurate keywords being extracted, compromising the quality of the results. on the other hand, llms can handle single sentences or small paragraphs effectively without needing deep context, but have difficulty maintaining performance in environments that require handling a high flow of data, demanding substantial computational power and specialized hardware (e.g. gpus). therefore, whether llms are necessary depends on the specific requirements and constraints of the application. in future work, we could conduct a more qualitative benchmarking, where specialists could evaluate and compare the results of the outputs generated by the different methods. this kind of analysis would allow for a deeper understanding of the relevance and accuracy of the extracted keywords, as well as provide valuable insights that cannot be captured by quantitative metrics alone. additionally, exploring the integration of more advanced nlp techniques, such as contextualized word embeddings, with experiments on datasets from other domains, could potentially increase the quality of keyword extractions. notes 1.available at: https://github.com/fernandozagatti/unsupervised-statistical-keyword-extraction 2.available at: https://github.com/liaad/yake. 3.available at: https://huggingface.co/. 4.available at: https://huggingface.co/voicelab/vlt5-base-keywords. 5.available at: https://huggingface.co/google/flan-t5-xxl. 6.available at: https://huggingface.co/bigscience/bloom. 7.available at: https://huggingface.co/facebook/blenderbot-400m-distill. 8.available at: https://huggingface.co/recogna-nlp/phi-bode. references campos, r., jatowt, a., jorge, a.: text mining and visualization of political party programs using keyword extraction methods: the case of portuguese legislative elections. in: sserwanga, i., et al. (eds.) international conference on information, pp. 340–349. springer, cham (2023). https://doi.org/10.1007/978-3-031-28035-1_24 campos, r., mangaravite, v., pasquali, a., jorge, a., nunes, c., jatowt, a.: yake! keyword extraction from single documents using multiple local features. inf. sci. 509, 257–289 (2020) article  google scholar  chung, h.w., et al.: scaling instruction-finetuned language models (2022). https://doi.org/10.48550/arxiv.2210.11416, https://arxiv.org/abs/2210.11416 devlin, j., chang, m.w., lee, k., toutanova, k.: bert: pre-training of deep bidirectional transformers for language understanding. in: proceedings of naacl-hlt 2019, p. 41714186. minneapolis, minnesota (2019) google scholar  firoozeh, n., nazarenko, a., alizon, f., daille, b.: keyword extraction: issues and methods. nat. lang. eng. 26(3), 259–291 (2020) article  google scholar  gopan, e., rajesh, s., vishnu, g., thushara, m., et al.: comparative study on different approaches in keyword extraction. in: 4th international conference on computing methodologies and communication (iccmc), pp. 70–74. ieee (2020) google scholar  grootendorst, m.: keybert: minimal keyword extraction with bert (2020).https://doi.org/10.5281/zenodo.4461265 gupta, a., chadha, a., tewari, v.: a natural language processing model on bert and yake technique for keyword extraction on sustainability reports. ieee access 12, 7942–7951 (2024) article  math  google scholar  hernández-castañeda, á., garcía-hernández, r.a., ledeneva, y., millán-hernández, c.e.: language-independent extractive automatic text summarization based on automatic keyword extraction. comput. speech lang. 71, 101267 (2022) google scholar  li, j., tang, t., zhao, w.x., nie, j.y., wen, j.r.: pre-trained language models for text generation: a survey. acm comput. surv. 56(9), 1–39 (2024) article  math  google scholar  liu, x., et al.: p-tuning v2: prompt tuning can be comparable to fine-tuning universally across scales and tasks. arxiv preprint arxiv:2110.07602 (2021) mihalcea, r., tarau, p.: textrank: bringing order into text. in: proceedings of the 2004 emnlp, pp. 404–411 (2004) google scholar  nadim, m., akopian, d., matamoros, a.: a comparative assessment of unsupervised keyword extraction tools. ieee access 11, 144778–144798 (2023) google scholar  paiola, p.h., garcia, g.l., papa, j.p.: phi-bode (2024). https://doi.org/10.57967/hf/1880, https://huggingface.co/recogna-nlp/phi-bode pęzik, p., mikołajczyk, a., wawrzyński, a., żarnecki, f., nitoń, b., ogrodniczuk, m.: transferable keyword extraction and generation with text-to-text language models. in: international conference on computational science, pp. 398–405. springer, cham (2023). https://doi.org/10.1007/978-3-031-36021-3_42 radford, a., narasimhan, k., salimans, t., sutskever, i., et al.: improving language understanding by generative pre-training (2018) google scholar  raffel, c., et al.: exploring the limits of transfer learning with a unified text-to-text transformer. j. mach. learn. res. 21(140), 1–67 (2020). http://jmlr.org/papers/v21/20-074.html ramachandran, r., mohan, m.k., sara, s.k.: document clustering using keyword extraction. in: 2022 ieee 3rd global conference for advancement in technology (gcat), pp. 01–06. ieee (2022) google scholar  real, l., oshiro, m., mafra, a.: b2w-reviews01 a brazilian portuguese reviews corpus (2019). https://opencor.gitlab.io/corpora/real19b2wreviews01/ roller, s., et al.: recipes for building an open-domain chatbot. arxiv preprint arxiv:2004.13637 (2020) rose, s., engel, d., cramer, n., cowley, w.: automatic keyword extraction from individual documents. in: text mining: applications and theory, pp. 1–20 (2010) google scholar  yang, z., yu, h., tang, j., liu, h.: toward keyword extraction in constrained information retrieval in vehicle social network. ieee trans. veh. technol. 68(5), 4285–4294 (2019) article  google scholar  download references acknowledgments this study was financed in part by the coordenação de aperfeiçoamento de pessoal de nível superior brasil (capes) finance code 001. author information authors and affiliations department of computing, federal university of são carlos, são carlos, brazil fernando rezende zagatti, daniel lucrédio & helena de medeiros caseli authors fernando rezende zagattiview author publications search author on:pubmed google scholar daniel lucrédioview author publications search author on:pubmed google scholar helena de medeiros caseliview author publications search author on:pubmed google scholar corresponding author correspondence to fernando rezende zagatti . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper zagatti, f.r., lucrédio, d., caseli, h.d.m. (2025). unsupervised statistical keyword extraction pipeline: is llm all you need?. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_32 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_32 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords keyword extraction natural language processing large language models publish with us policies and ethics search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here not affiliated © 2025 springer nature semi-supervised predictive clustering trees for multi-label protein subcellular localization | springer nature link (formerly springerlink) skip to main content advertisement log in menu find a journal publish with us track your research search cart home intelligent systems conference paper semi-supervised predictive clustering trees for multi-label protein subcellular localization conference paper first online: 30 january 2025 pp 384–399 cite this conference paper access provided by university of notre dame hesburgh library download book pdf download book epub intelligent systems (bracis 2024) semi-supervised predictive clustering trees for multi-label protein subcellular localization download book pdf download book epub leonardo u. alcantara9, isaac triguero  orcid: orcid.org/0000-0002-0150-065110 & ricardo cerri  orcid: orcid.org/0000-0002-2582-169511  part of the book series: lecture notes in computer science ((lnai,volume 15413)) included in the following conference series: brazilian conference on intelligent systems 417 accesses 1 citation abstract protein subcellular localization is an important classification task because the location of proteins in a cell is directly linked to their functions. since a protein can act at two or more locations simultaneously, multi-label classification algorithms are necessary. the currently used algorithms are usually based on supervised learning, which presents some disadvantages such as (i) a need for a large amount of labeled instances for training; (ii) a waste of valuable information that labeled instances can provide; and (iii) a high cost involved in obtaining labeled instances for training. to overcome these disadvantages, semi-supervised learning can be applied, where classifiers exploit both labeled and unlabeled data. thus, in this paper, we propose a new semi-supervised algorithm for multi-label protein subcellular localization. our proposal is based on decision tree classifiers induced using predictive clustering trees. we investigate many semi-supervised protein subcellular localization scenarios to test whether unlabeled instances can improve the multi-label classification process. our results show that the proposal can achieve competitive or better results when compared to the pure supervised version of the predictive clustering trees. access provided by university of notre dame hesburgh library. download conference paper pdf similar content being viewed by others human protein subcellular localization with integrated source and multi-label ensemble classifier article open access 21 june 2016 multiple protein subcellular locations prediction based on deep convolutional neural networks with self-attention mechanism article 23 january 2022 predicting the subcellular localization of multi-site protein based on fusion feature and multi-label deep forest model chapter © 2022 1 introduction determining the protein subcellular localization (psl) is a crucial process for deducing and understanding protein functions and for genome annotation [26]. psl is especially important in the medical field because it is fundamental for discovering and analyzing new kinds of medicines [21]. it can also help to identify some diseases, such as cancer or alzheimer, due to abnormal psl being correlated to these diseases [9]. although experimental approaches are the most reliable methods to determine psl, they are very costly and labor-intensive. in recent years, with the completion of many large-scale genomics and proteomics sequencing projects, a vast number of protein sequences have been discovered, making experimental approaches even more unfeasible. thus, machine learning (ml) computational predictors have been used, since they can handle this problem quickly and accurately [1, 27]. in addition, proteins can move between two or more locations within a cell or reside at more than one location simultaneously. thus, multi-label classification methods have also been developed to deal with psl problems [12, 29, 33, 36]. all these methods use training and test datasets normally acquired only from proteins whose localizations were experimentally defined, referred to as labeled proteins (lps) [6]. although using only lps is a well-established method in the literature, it presents some disadvantages: (i) lps are hard to obtain because they need to be annotated with experimental approaches, which, are labor-intensive and time-consuming; (ii) these experimental protein sequences are just a small part of the overall sequences in the knowledge bases [6, 35]. about only 5.5% of all instances in uniprotkb/swiss-prot are lps; and (iii) normally not all lps retrieved from the knowledge bases are used because some of them are homologous or similar proteins to each other. this may cause overfitting, by the use of models or procedures that include more terms than are necessary or use more complicated approaches than are necessary [14]. this problem is undesirable because it leads to models that adjust themselves really well to a particular set of data but fail to fit additional data or predict future observations reliably, therefore creating weaker classifiers [6, 35]. usually, most works in the literature use supervised ml to deal with psl. therefore, these works disregard proteins whose localizations are missing, referred to as not labeled proteins (nlps). however, the small number of available labeled proteins can lead to datasets with an insufficient number of instances. as an example, this is the case of the virus benchmark dataset used in the works of wan et al. [33] and shen and chou [23], which has only 207 proteins, only 8 of them located in the “viral capsid”. supervised methods ignore the fact that nlps can provide valuable information for the classification. for example, they contain information about the joint probability distribution over sequence elements [7, 35]. in situations where there is a large amount of unlabeled data, and the economical commitment necessary to label them all is not affordable, semi-supervised learning (ssl) [30] provides strategies to overcome these limitations. ssl is a learning paradigm focused on learning from both labeled and unlabeled data, mainly centered on addressing problems where labeled data is hard to obtain, and unlabeled data is abundant. the main advantage of using learning strategies that take advantage of unlabeled data for psl prediction is that we can construct reliable classifiers to predict the psl of organisms or groups of organisms with very few labeled instances. this is the case, for example, of domains such as archaea and bacteria, groups of organisms such as alveolata and amoebozoa and viruses. as an example, table 1 presents the number of lp and nlp proteins of some domains and groups of organisms in the uniprotkb/swiss-prot in the 2021_03 releasefootnote 1. table 1. comparison between lps and nlps in uniprotkb/swiss-protfull size table in this paper, we propose a semi-supervised multi-label method based on the concept of predictive clustering trees (pct), a decision tree induction algorithm well-established in the multi-label classification literature [32]. we investigate many semi-supervised protein subcellular localization scenarios in datasets related to fungus, viruses, and plants in order to test whether unlabeled instances can improve the multi-label prediction of protein subcellular localization. our results show that the proposal can achieve competitive or better results when compared to the original pure supervised version of the predictive clustering trees algorithm. the remainder of this paper is organized as follows. section 2 presents the main theoretical concepts needed for the development of our proposal. section 3 presents our proposed semi-supervised pct. our methodology is presented in sect. 4, while our experiments are presented and discussed in sect. 5. finally, sect. 6 presents our conclusions and future work. 2 theoretical foundation this section presents the main concepts necessary to develop our proposal. 2.1 multi-label classification traditional classification algorithms associate a single label/class with each instance in a dataset. however, there are many problems where we need to associate an instance with two or more labels simultaneously. in multi-label classification, an instance can be classified into two or more labels simultaneously. documents, images, videos, music, emotions, and proteins are examples of multi-label data given \(l = \{l_j: j = 1, ..., q\}\) a finite label set, multi-label algorithms associate to each instance \(x_i\) of a dataset, a set of labels \(y_i\) such that \(y_i \subseteq l\) [31]. multi-label classification methods can be characterized into three approaches [18]: i) problem transformation, which transforms multi-label problems into single-label problems and then employs traditional classification algorithms; ii) algorithm adaptation, which creates algorithms to handle multiple labels simultaneously; and iii) ensemble, which combines a set of multi-label classifiers to solve the problem. in this work, our proposed semi-supervised predictive clustering tree is based on the algorithm adaptation approach since the literature has already shown the advantages of algorithm adaptation over problem transformation, such as considering label correlations [4, 32]. 2.2 semi-supervised classification the main goal of semi-supervised classification (ssc) is to construct a model that uses labeled and unlabeled data, obtaining a better prediction performance than the one trained only with labeled data. thus, ssc learning can be seen as a midway between supervised learning, in which we only learn from labeled instances, and unsupervised learning, in which we only learn from unlabeled instances [22]. ssc can be helpful in situations where (i) the labels of the instances are hard to obtain, requiring human specialists, special devices, slow experiments, etc., and (ii) there is a large amount of unlabeled data available. in these cases, ssl requires less human effort and can potentially improve the accuracy of the classifiers. many domains struggle with the abovementioned problems, such as speech recognition, bioinformatics, text processing, and image categorization [19, 22]. ssc can be divided into two categories: transductive learning and inductive learning. while inductive learning focuses on building a good classifier capable of predicting unseen data accurately, transductive learning aims to predict the labels from the unlabeled instances given in the training set [30]. in this work, our proposed semi-supervised predictive clustering tree is based on inductive learning. 2.3 predictive clustering trees predictive clustering trees (pcts) [3] are a generalization of decision trees that can be used for clustering and classification. in regular decision trees, the leaves contain the classes, and the branches from the root to the leaves contain conditions for the classification. pcts view a decision tree as a hierarchy of clusters in which the root of a pct represents a cluster containing all the data and is recursively partitioned into smaller clusters using a criterion to select the best split. the leaves of a pct correspond to the lowest level of the cluster hierarchy, and each leaf is labeled with a cluster prototype (multi-label prediction) [16]. pcts can be constructed using a conventional top-down induction of decision trees heuristic (tdidt) [5]. the algorithm receives a set of examples (e) as input and outputs the final pct. to do this, it uses a heuristic (h) to select the best test (\(t^*\)) between the tests (t), also called possible splits. the heuristic used in pct is the variance reduction caused by partitioning the instances from a node into many data partitions (child nodes) according to the best test (\(t^*\)). to find the best test at each node, the besttest procedure searches for the best acceptable test by calculating the variance reduction for each possible test. the procedure returns the best test found, the heuristic value, and the new partitions of the node data that will be used as child nodes. this process is repeated for every child node until no acceptable test can be found, which means that no test significantly reduces the variance of the node. then, the algorithm creates a leaf and computes the prototype of the instances belonging to that leaf to create the predicted labels. this whole pct induction process is shown in algorithm 1. maximizing the variance reduction when splitting each node ensures that the corresponding clusters’ homogeneity increases with the depth of the tree, improving the prediction performance in general [3]. the main difference between pcts and regular decision trees is that pcts treat the variance and prototype functions as hyperparameters that are instantiated based on the learning task. thus, pcts can be used in many tasks, such as clustering [3, 25], multi-output classification and regression [2, 24], time series data analysis [10], interaction prediction [20], and hierarchical multi-label classification [32]. in our experiments, we used the clus implementation of the pcts [16, 32]. 3 semi-supervised predictive clustering trees our proposal consists of adapting the clus induction procedure to create a semi-supervised predictive clustering tree (ssl-pct) able to induce a clustering decision tree using labeled and unlabeled data simultaneously. our method preserves the appealing characteristics of using a decision tree, such as being fast to learn and being readily interpretable (unlike the models learned by most semi-supervised algorithms), while making use of unlabeled data. some essential modifications are needed to adapt the clus algorithm to make it semi-supervised. the first one is related to the initial set of instances (e) since now we have to modify clus to accept labeled and unlabeled instances as input. our dataset is now represented as \(e = e_l \cup e_u \), where \(e_l\) is the labeled part of the training set and \(e_u\) is the unlabeled part. the second modification is related to the impurity calculation in the besttest procedure. we must create a new heuristic that considers labeled and unlabeled data in a node. our proposed impurity comprises two parts, a supervised and an unsupervised. a \(\omega \) hyperparameter controls the influence of each part. the proposed impurity is presented in eq. 1. $$\begin{aligned} \text {impurity}_{\text {ssl}}({e}) = \underbrace{ \frac{\omega }{t} . \sum _{i=1}^{ t} \text {impurity}(e_l, y_i)}_{\text {supervised part}} + \underbrace{ \frac{1-\omega }{d} . \sum _{i=1}^{ d} \text { impurity(}e, x_i)}_{\text {unsupervised part}} \end{aligned}$$ (1) in eq. 1, e is the set of instances, \(y_i\) represents each of the t target features (labels), \(x_i\) represents each of the d descriptive features, and \(\omega \) is a value between 0 and 1 that weights the supervision and unsupervision used in the method. if \(\omega = 1\), the method works using only the labeled part of the dataset, behaving as a supervised learning algorithm. on the other hand, if \(\omega = 0\), the algorithm behaves as an unsupervised learning algorithm, not using the labels to define the splits. any value between 0 and 1 means the algorithm uses labeled and unlabeled data. recall that in eq. 1, the whole dataset e is considered an unsupervised dataset since, regardless of the instances being labeled or not, the unsupervised part of the equation uses only descriptive features to search for the best split. what changes is the weight of the supervision employed during induction. one of our method’s main advantages is the controlled level of supervision. when performing semi-supervised learning, the unlabeled data can always negatively affect prediction performance. by adapting the level of supervision, we can guarantee that our ssl-pct achieves better or equal prediction performance compared to its supervised counterpart. the main impurity measure used in the original pct algorithm is the gini impurity (gi). gi is a measure used in decision trees to calculate how much a set of instances in a node is pure to find the best split. gi can be calculated for a set of data e and a specific label y according to eq. 2. $$\begin{aligned} \text {gini(}e, y) = 1 \sum _{i=1}^{ c} p_{i}^{2} \end{aligned}$$ (2) in eq. 2, c is the number of possible values for the label y. as an example, in a binary classification, \(c = 2\). also, \(p_i\) is the a priori probability of class \(c_i\). since we now have labeled and unlabeled instances, we need to define the impurity calculation in each case. the impurity over the target features (labels) is calculated using only the labeled instances (\(e_l\)). it is referred to as a labeled impurity. in contrast, the impurity over the descriptive features is calculated using all the labeled and unlabeled instances (e) and is referred to as an unlabeled impurity. equation 3 presents the labeled impurity calculation for a node of our ssl-pct. in the equation, \(y_i\) is a label, \(e_{l}^{train}\) represents the whole labeled set of instances before the root node split, and \(e_l\) is a subset of instances resulting after the root node split. $$\begin{aligned} \text {impurity(}e_l, y_i\text {)} = \frac{\text {gini}(e_l, y_i)}{\text {gini}(e_l^{train}, y_i)} \end{aligned}$$ (3) the unlabeled impurity is calculated over the descriptive features, which, differently from the labels, can assume numeric or categorical values. thus, we need to define an impurity for each case. the unlabeled impurities for a dataset e over categorical and numeric descriptive feature \(x_i\) are calculated according to eqs. 4 and 5. $$\begin{aligned} \underbrace{\text {impurity(}e, x_i\text {)} = \frac{\text {gini}(e, x_i)}{\text {gini}(e^{train}, x_i)}}_{\text {for categorical features}} \end{aligned}$$ (4) $$\begin{aligned} \underbrace{\text {impurity(}e, x_i\text {)} = \frac{\text {var}(e, x_i)}{\text {var}(e^{train}, x_i)}}_{\text {for numeric features}} \end{aligned}$$ (5) the variance of the \(i^{th}\) feature in the set e over the descriptive feature \(x_i\) is calculated according to eq. 6. $$\begin{aligned} \text {var}(e, x_i) = \frac{\sum _{j=1}^{ n}(x_{i}^{j})^{2} \frac{1}{n} . (\sum _{j=1}^{ n} x_{i}^{j})^{2}}{n} \end{aligned}$$ (6) after modifying the impurity calculations, our proposed ssl-pct can now handle datasets containing labeled and unlabeled instances, exploiting them according to the \(\omega \) weight hyperparameter. 4 methodology this section presents experiments, software, datasets, and evaluation measures. table 2. characteristics of the datasets used in the experimentsfull size table 4.1 clus clus is an open-source decision tree and rules learning system that implements the predictive clustering framework [3]. it generalizes the decision tree concept by learning trees interpreted as cluster hierarchies, called pcts. to create a cluster during induction, clus can be optimized using different heuristics to choose the best split in each internal node, improving prediction performance. in this work, we used the implementation provided by vens et al. [32], modifying it to use our proposed semi-supervised impurity function. all implementations and experiments were performed using the java program language. 4.2 datasets uniprot knowledge base (uniprotkb) [28] is an expertly curated database, acting as a central access point for integrated protein information with cross-references to multiple sources. the knowledge base consists of uniprotkb/swissprot and uniprotkb/trembl. the former contains manual and high-quality annotations with information extracted from literature and curator-evaluated computational analysis. the latter contains computationally analyzed records enriched with automatic annotation and classification. we extracted three uniprotkb datasets: viruses, plants, and fungi. we used data extracted from the uniprotkb/swiss-prot 2021_03 version. we used gene ontology (go) [13] terms as features for each protein. we extracted go terms associated with each protein from the uniprot-goa [15] database. the go is a set of defined vocabularies and terms representing gene functions and products across different species. two strategies were used to construct feature vectors: 1 and 0 values to indicate the presence or absence of go terms and the frequency (tf) of each go term for each protein. therefore, we constructed six protein datasets, three with categorical descriptive features (1/0) and three with numeric descriptive features (tf). table 2 describes the characteristics of the datasets used in our experiments. card refers to cardinality, the average number of labels per instance in a multi-label dataset. dens refers to density, which is a dimensionless level of the cardinality. equations 7 and 8 show how these two measures are calculated. in the equations, \(y_i\) represents a label of the dataset d. in table 2, meanir refers to the mean imbalance ratio of the datasets, which is a value that represents the average level of imbalance in a multi-label dataset [8]. this measure is essential because most of the multi-label datasets suffer from a high level of imbalance, making the learning task more complicated. 4.3 evaluation measures to evaluate the performance of our proposed ssl-pct, we used the multi-label evaluation measures defined in the work of godbole and sarawagi [11]. they are presented in eqs. 9, 10, 11 and 12. consider that each multi-label instance is represented by \((\textbf{x}_i, y_i)\) in which \(i = 1 \dots m\) (with m the total number of instances). in the equations, \(y_i \subseteq l\) is the set of true labels of instance \(\textbf{x}_i\), and \(l = \{\lambda _j : j = 1 \dots q\}\) is the set of all labels in the problem. given a specific instance \(\textbf{x}_i\), the set of labels predicted by a classifier is represented by \(z_i\). the f1 measure in eq. 12 is the harmonic mean of precision and recall. 5 experiments and discussion we executed our experiments using a 5-fold cross-validation procedure, splitting the data into five partitions containing 20% of the original data. for each fold, the algorithm is trained using the remaining four folds, referred to as a training set, and then used to label the instances in the fold, referred to as the test set. the test set is kept aside from the training data to evaluate the inductive performance of our method. the transductive performance is also evaluated, labeling the instances of the four folds used as the training set. the final result is the average of the evaluation measures calculated for each fold. each training set is then divided into labeled and unlabeled portions of instances. following the recommendation of wang et al. [34], we do not keep class proportion in both labeled and unlabeled sets since the objective of semi-supervised learning is to exploit the use of unlabeled data to improve prediction performance. the selection of instances to be labeled is performed randomly until a specific criterion is met, and the remaining instances are used as unlabeled. in this case, the labels of the instances are removed and replaced with the symbol “?” so that our algorithm recognizes these instances as unlabeled. we ensure that every labeled portion of the training set has at least one representative instance for each class. we used different ratios when dividing the training set to test many semi-supervised learning scenarios for each dataset. we performed as follows. consider c as the number of classes in a dataset e. for every dataset, the labeled portion of the training set was constructed with \(c \times \alpha \) instances. the \(\alpha \) hyperparameter is a value that controls the number of instances in the labeled portion. we used three values for \(\alpha \): 5, 10 and 20. this strategy was performed for every dataset in table 2. considering the \(\omega \) hyperparameter, which defines the contribution of supervised and unsupervised parts in the calculation of the impurity value (eq. 1), we tested six different values: 0.0, 0.2, 0.4, 0.6, 0.8 and 1.0. thus, we can evaluate if the unlabeled instances improved the prediction performance. it is worth noting that our proposed ssl-pct can present different performance on different datasets. this happens because semi-supervised methods are known to be domain dependent [17]. table 3 shows the accuracy and f1 values for all the experiments. acc i and f1 i stand for accuracy and f1 inductive performances, while acc t and f1 t stand for accuracy and f1 transductive performances. the best results for each dataset are highlighted in boldface to see the best performance among the variations of the \(\omega \) hyperparameter. each value is the accuracy and f1 average values across 5-fold cross-validation. the rows containing the symbol “-” represent cases in which our heuristic could not find any acceptable test to put in the nodes of the decision trees. thus, no tree was created in these cases. this only happened for some experiments with the numeric versions of the datasets when the \(\omega \) hyperparameter was set to 0.0, i.e., when only unlabeled instances were used to calculate the impurity. table 3. results in the virus, plant and fungus datasets.full size table table 4. results in the virus, plant and fungus datasets without located_in termsfull size table as can be seen from our results, in most cases, the best results were obtained when the maximum supervision level was used (\(\omega = 1\)), indicating that the algorithm could not improve the classification with the unlabeled data. it is possible to see that for numeric datasets, semi-supervision slightly improved the result for some cases compared to the purely supervised version. even in these cases, the increase was not very significant. in some configurations where \(\omega = 0.8\), a semi-supervised strategy could increase the performance by more than 1%. an unusual point of these results is that the evaluation metrics returned very high values even with very few labeled instances. this is even more unusual when we see that in the experiments with \(\alpha = 5\), we have approximately 30 virus labeled instances (4.65% of the total), 60 plant labeled instances (1.14% of the total), and 50 fungus labeled instances (0.91% of the total). a hypothesis raised to explain this point is that we have several types of go terms, each one representing a conceptfootnote 2. as an example, we have the type involved_in, which tells if a protein is involved in some biological process, or the type enables, which tells whether a protein allows for some biological process to occur. a possible problem may be related to our features since in the cellular components’ ontology, we have annotations with the term located_in. this term appears in a protein when we have evidence that it is present in any cellular component, i.e., in this case, we have a descriptive feature representing almost the same concept of the target feature: subcellular localization. this can explain why the evaluation measures achieved high results, even with very few proteins, given that these located_in features will always be great candidates to be the best features to partition the dataset. new experiments were executed to confirm this hypothesis, removing all go terms of the type located_in. table 4 shows the results of these new experiments. again, the best results are highlighted in boldface: acc i and f1 i stand for accuracy and f1 inductive performances, and acc t and f1 t stand for accuracy and f1 transductive performances. from these new results, we can see that our hypothesis seems to be correct since the decision trees trained in the datasets without the located_in terms obtained much lower values for the evaluation measures. however, these values better represent the reality of the taxonomies with few proteins with annotated subcellular localizations. 6 conclusions and future work in this paper, we proposed a semi-supervised predictive clustering tree (ssl-pct) algorithm capable of classifying multi-label instances exploiting labeled and unlabeled data. our proposal modifies the clus algorithm [32], introducing a new impurity measure for the decision tree heuristic. our new impurity uses labeled and unlabeled instances to find the best split in a decision tree node. our proposed algorithm is also flexible, as we have an \(\omega \) hyperparameter that controls the tree’s supervision level. if \(\omega = 1\), the tree acts precisely like a supervised pct. if \(\omega = 0\), the algorithm only uses the descriptive features from the labeled and unlabeled instances to construct the tree, without the information about the labels. we applied our method to different protein subcellular localization datasets. we also compared many values of the \(\omega \) hyperparameter, varying its value from 0.0 to 1.0 in steps of 0.2. we also investigated the performance of ssl-pct when used with different amounts of labeled instances. the results show that our new method can exploit unlabeled data to help the classification in many scenarios. our experiments showed that the best results typically relied only on the labeled data. however, the experiments also showed that the unsupervised part of the algorithm can help improve the classification results. in future work, we plan to expand the experiments in various ways, such as investigating more well-known multi-label datasets, better investigating the variation in the \(\omega \) hyperparameter to analyze the influence of the level of supervision in the datasets, evaluating the prediction performance with more multi-label evaluation measures, and perform experiments with more variations in the proportions of labeled and unlabeled data in our datasets. we will also study the possibility of creating semi-supervised learning random forests within our ssl-pct. random forests are one of the best classification algorithms, and their ability to generate a feature ranking can help define the best features in multi-label datasets. we also plan to apply some pre-processing to the protein datasets to improve the results using unlabeled data. since, in general, only a few features are used to construct the trees, there is an indication that we may remove some less important features in the used protein datasets. notes 1.http://ftp.uniprot.org/pub/databases/uniprot/previous_releases/release-2021_03/. 2.http://geneontology.org/docs/go-annotation-file-gaf-format-2.2/. references almagro armenteros, j.j., sønderby, c.k., sønderby, s.k., nielsen, h., winther, o.: deeploc: prediction of protein subcellular localization using deep learning. bioinformatics 33(21), 3387–3395 (2017) article  math  google scholar  basgalupp, m., cerri, r., schietgat, l., triguero, i., vens, c.: beyond global and local multi-target learning. inf. sci. 579, 508–524 (2021) article  mathscinet  math  google scholar  blockeel, h., raedt, l.d., ramon, j.: top-down induction of clustering trees. in: proceedings of the fifteenth international conference on machine learning, icml 1998, pp. 55–63. morgan kaufmann publishers inc. (1998) google scholar  bogatinovski, j., todorovski, l., džeroski, s., kocev, d.: comprehensive comparative study of multi-label classification methods. expert syst. appl. 203, 117215 (2022) article  math  google scholar  breiman, l., friedman, j., olshen, r., stone, c.: classification and regression trees. wadsworth and brooks, monterey (1984) math  google scholar  cao, j., liu, w., he, j., gu, h.: mining proteins with non-experimental annotations based on an active sample selection strategy for predicting protein subcellular localization. plos one 8, e67343 (2013) google scholar  caragea, c., caragea, d., silvescu, a., honavar, v.: semi-supervised prediction of protein subcellular localization using abstraction augmented markov models. bmc bioinform. 11(suppl 8), s6 (2010) article  math  google scholar  charte, f., rivera, a.j., del jesus, m.j., herrera, f.: addressing imbalance in multilabel classification: measures and random resampling algorithms. neurocomputing 163, 3–16 (2015) article  google scholar  cui, q., jiangemail, t., liu, b., ma, s.: esub8: a novel tool to predict protein subcellular localizations in eukaryotic organisms. bmc bioinform. 5, 66 (2004) article  google scholar  džeroski, s., gjorgjioski, v., slavkov, i., struyf, j.: analysis of time series data with predictive clustering trees. in: džeroski, s., struyf, j. (eds.) kdid 2006. lncs, vol. 4747, pp. 63–80. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-75549-4_5 chapter  math  google scholar  godbole, s., sarawagi, s.: discriminative methods for multi-labeled classification. in: dai, h., srikant, r., zhang, c. (eds.) pakdd 2004. lncs (lnai), vol. 3056, pp. 22–30. springer, heidelberg (2004). https://doi.org/10.1007/978-3-540-24775-3_5 chapter  math  google scholar  guo, x., liu, f., ju, y., wang, z., wang, c.: human protein subcellular localization with integrated source and multi-label ensemble classifier. sci. rep. 6, 28087 (2016) google scholar  harris, m.a., clark, j., ireland, a.: the gene ontology (go) database and informatics resource. nucl. acids res 32, d258-61 (2004) google scholar  hawkins, d.m.: the problem of overfitting. j. chem. inf. comput. sci. 44, 1–12 (2004) article  math  google scholar  huntley, r.p., et al.: the goa database: gene ontology annotation updates for 2015. nucl. acids res. 43, d1057–d1063 (2015) article  math  google scholar  kocev, d., slavkov, i., dzeroski, s.: feature ranking for multi-label classification using predictive clustering trees. in: proceedings of companion publication of the european conference on machine learning and knowledge discovery in databases (2013) google scholar  levatić, j., ceci, m., kocev, d., dźeroski, s.: semi-supervised classification trees. j. intell. inf. syst. 49(3), 461–486 (2017) article  math  google scholar  madjarov, g., kocev, d., gjorgjevikj, d., džeroski, s.: an extensive experimental comparison of methods for multi-label learning. pattern recogn. 45(9), 3084–3104 (2012) article  math  google scholar  pise, n.n., kulkarni, p.: a survey of semi-supervised learning methods. in: 2008 international conference on computational intelligence and security, vol. 2, pp. 30–34 (2008) google scholar  pliakos, k., vens, c.: drug-target interaction prediction with tree-ensemble learning and output space reconstruction. bmc bioinform. 21 (2020) google scholar  rey, s., gardy, j.l., brinkman, f.s.: assessing the precision of high-throughput computational and laboratory approaches for the genome-wide identification of protein subcellular localization in bacteria. bmc genom. 6, 162 (2005) article  math  google scholar  sadarangani, a., jivani, a.: a survey of semi-supervised learning. int. j. eng. sci. res. technol. 5(10), 138–143 (2016) math  google scholar  shen, h., chou, k.: virus-mploc: a fusion classifier for viral protein subcellular location prediction by incorporating multiple sites. j. biomol. struct. dyn. 28(2), 175–186 (2010) article  math  google scholar  struyf, j., džeroski, s.: constraint based induction of multi-objective regression trees. in: bonchi, f., boulicaut, j.-f. (eds.) kdid 2005. lncs, vol. 3933, pp. 222–233. springer, heidelberg (2006). https://doi.org/10.1007/11733492_13 chapter  math  google scholar  struyf, j., džeroski, s.: clustering trees with instance level constraints. in: kok, j.n., koronacki, j., mantaras, r.l., matwin, s., mladenič, d., skowron, a. (eds.) ecml 2007. lncs (lnai), vol. 4701, pp. 359–370. springer, heidelberg (2007). https://doi.org/10.1007/978-3-540-74958-5_34 chapter  math  google scholar  su, e., chiu, h., lo, a., hwang, j., sung, t., hsu, w.: protein subcellular localization prediction based on compartment-specific features and structure conservation. bmc bioinform. 8, 330 (2007) article  math  google scholar  su, r., he, l., liu, t., liu, x., wei, l.: protein subcellular localization based on deep image features and criterion learning strategy. brief. bioinform. 22(4), bbaa313 (2020) google scholar  the uniprot consortium: uniprot: the universal protein knowledgebase. nucl. acids res. 45(issue d1), d158–d169 (2017) google scholar  thumuluri, v., almagro armenteros, j.j., johansen, a., nielsen, h., winther, o.: deeploc 2.0: multi-label subcellular localization prediction using protein language models. nucl. acids res. 50(w1), w228–w234 (2022) google scholar  triguero, i., garcia, s., herrera, f.: self-labeled techniques for semi-supervised learning: taxonomy, software and empirical study. knowl. inf. syst. 42(2), 245–284 (2015) article  math  google scholar  tsoumakas, g., katakis, i., vlahavas, i.: mining multi-label data. in: maimon, o., rokach, l. (eds.) data mining and knowledge discovery handbook, pp. 667–685. springer, boston (2009). https://doi.org/10.1007/978-0-387-09823-4_34 chapter  google scholar  vens, c., struyf, j., schietgat, l., džeroski, s., blockeel, h.: decision trees for hierarchical multi-label classification. mach. learn. 73, 185–214 (2008) article  math  google scholar  wan, s., mak, m.w., kung, s.y.: mgoasvm: multi-label protein subcellular localization based on gene ontology and support vector machines. bmc bioinform. 13, 290 (2012) article  math  google scholar  wang, y., xu, x., zhao, h., hua, z.: semi-supervised learning based on nearest neighbor rule and cut edges. knowl.-based syst. 23(6), 547–554 (2010) article  math  google scholar  xu, q., hu, d., xue, h., yu, w., yang, q.: semi-supervised protein subcellular localization. bmc bioinform. 10(suppl 1), s47 (2009) article  math  google scholar  zhang, q., et al.: accurate prediction of multi-label protein subcellular localization through multi-view feature learning with rbrl classifier. brief. bioinform. 22(5) (2021) google scholar  download references acknowledgment this study was financed by the são paulo research foundation (fapesp) grants #2016/25220-1, #2017/24807-1 and #2022/02981-8. i. triguero is funded by a maria zambrano senior fellowship at the university of granada. i. triguero’s work is also supported by pid2023-149128nb-i00. author information authors and affiliations department of computing, federal university of são carlos, são carlos, sp, brazil leonardo u. alcantara department of computer science and artificial intelligence, university of granada, granada, spain isaac triguero institute of mathematics and computer science, university of são paulo, são carlos, sp, brazil ricardo cerri authors leonardo u. alcantaraview author publications search author on:pubmed google scholar isaac trigueroview author publications search author on:pubmed google scholar ricardo cerriview author publications search author on:pubmed google scholar corresponding author correspondence to ricardo cerri . editor information editors and affiliations universidade federal fluminense, niterói, brazil aline paes instituto tecnológico de aeronáutica, são josé dos campos, brazil filipe a. n. verri rights and permissions reprints and permissions copyright information © 2025 the author(s), under exclusive license to springer nature switzerland ag about this paper cite this paper alcantara, l.u., triguero, i., cerri, r. (2025). semi-supervised predictive clustering trees for multi-label protein subcellular localization. in: paes, a., verri, f.a.n. (eds) intelligent systems. bracis 2024. lecture notes in computer science(), vol 15413. springer, cham. https://doi.org/10.1007/978-3-031-79032-4_27 download citation .ris .enw .bib doi: https://doi.org/10.1007/978-3-031-79032-4_27 published: 30 january 2025 publisher name: springer, cham print isbn: 978-3-031-79031-7 online isbn: 978-3-031-79032-4 ebook packages: computer sciencecomputer science (r0) share this paper anyone you share the following link with will be able to read this content: get shareable linksorry, a shareable link is not currently available for this article. copy shareable link to clipboard provided by the springer nature sharedit content-sharing initiative keywords predictive clustering trees protein subcellular localization multi-label classification semi-supervised learning publish with us policies and ethics profiles isaac triguero view author profile ricardo cerri view author profile search search by keyword or author search navigation find a journal publish with us track your research discover content journals a-z books a-z publish with us journal finder publish your research language editing open access publishing products and services our products librarians societies partners and advertisers our brands springer nature portfolio bmc palgrave macmillan apress discover your privacy choices/manage cookies your us state privacy rights accessibility statement terms and conditions privacy policy help and support legal notice cancel contracts here 129.74.145.123 hesburgh library er unit (3005732405) northeast research libraries (nerl) (8200828607) nerl ta account (3006206169) university of notre dame hesburgh library (3000184373) © 2025 springer nature